2024-03-19T11:16:15.401 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-19T11:16:15.401 DEBUG:teuthology.run:Teuthology command: teuthology --name rfriedma-2024-03-19_08:22:49-rados:thrash-wip-rf-nextobj-distro-default-smithi --archive /home/teuthworker/archive/rfriedma-2024-03-19_08:22:49-rados:thrash-wip-rf-nextobj-distro-default-smithi/7610739 --verbose --owner scheduled_rfriedma@teuthology --description rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/dedup-io-snaps} -- /home/teuthworker/archive/rfriedma-2024-03-19_08:22:49-rados:thrash-wip-rf-nextobj-distro-default-smithi/7610739/orig.config.yaml 2024-03-19T11:16:15.444 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-19T11:16:15.537 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rfriedma-2024-03-19_08:22:49-rados:thrash-wip-rf-nextobj-distro-default-smithi/7610739 branch: wip-rf-nextobj description: rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/dedup-io-snaps} email: null first_in_suite: false job_id: '7610739' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: rfriedma-2024-03-19_08:22:49-rados:thrash-wip-rf-nextobj-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 30 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-rf-nextobj ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 1 ms inject socket failures: 5000 ms type: async osd_pool_default_min_size: 2 osd_pool_default_size: 2 osd_recovery_max_active: 10 osd_recovery_max_single_start: 10 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.3 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd heartbeat use min delay socket: true osd inject bad map crc probability: 0.1 osd max backfills: 3 osd max markdown count: 1000 osd max scrubs: 5 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(REQUEST_STUCK\) - failed to encode map - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - \(POOL_APP_NOT_ENABLED\) sha1: 9ef43722ad94ea688fa7f70daab6dd41fac6ee53 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 9ef43722ad94ea688fa7f70daab6dd41fac6ee53 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-rf-nextobj sha1: 9ef43722ad94ea688fa7f70daab6dd41fac6ee53 owner: scheduled_rfriedma@teuthology priority: 95 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 2525 sha1: 9ef43722ad94ea688fa7f70daab6dd41fac6ee53 sleep_before_teardown: 0 subset: 4/400 suite: rados:thrash suite_branch: wip-rf-nextobj suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 9ef43722ad94ea688fa7f70daab6dd41fac6ee53 targets: smithi038.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN12b76+6OZUqzRu5kZgY1R3xXUct6XvKeRMHOD6LkRGOzsW2C5qGHLKLNc/hTgPYE3wvjLymEenOFRKGhA/ouk= smithi172.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOkhYhhIXg4DTniTbJjE1J7at0/UswGbMkAbBGurn57KL5GO5auYOgugBa786l9o0o5uMhI6NPM/z44KCkJT+3g= tasks: - install: null - ceph: conf: osd: debug monc: 20 - exec: mon.a: - ceph config set mgr mgr/balancer/log_level debug - ceph balancer status - ceph osd set-require-min-compat-client reef - ceph balancer mode upmap-read - ceph balancer on - ceph balancer status - ceph balancer status detail - thrashosds: aggressive_pg_num_changes: false chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - exec: client.0: - sudo ceph osd pool create low_tier 4 - rados: clients: - client.0 dedup_chunk_algo: fastcdc dedup_chunk_size: '131072' enable_dedup: true low_tier_pool: low_tier objects: 50 op_weights: read: 100 rollback: 10 set_chunk: 30 snap_create: 10 snap_remove: 10 tier_evict: 10 tier_flush: 5 tier_promote: 10 write: 50 ops: 1500 set_chunk: true teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-19_08:22:49 tube: smithi user: rfriedma verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-19T11:16:15.537 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa; will attempt to use it 2024-03-19T11:16:15.538 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks 2024-03-19T11:16:15.538 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-19T11:16:15.539 INFO:teuthology.task.internal:Checking packages... 2024-03-19T11:16:15.562 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '9ef43722ad94ea688fa7f70daab6dd41fac6ee53' 2024-03-19T11:16:15.562 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-19T11:16:15.562 INFO:teuthology.packaging:ref: None 2024-03-19T11:16:15.563 INFO:teuthology.packaging:tag: None 2024-03-19T11:16:15.563 INFO:teuthology.packaging:branch: wip-rf-nextobj 2024-03-19T11:16:15.563 INFO:teuthology.packaging:sha1: 9ef43722ad94ea688fa7f70daab6dd41fac6ee53 2024-03-19T11:16:15.563 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-rf-nextobj 2024-03-19T11:16:15.812 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2159-g9ef43722-1jammy 2024-03-19T11:16:15.814 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-19T11:16:15.821 INFO:teuthology.task.internal:no buildpackages task found 2024-03-19T11:16:15.821 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-19T11:16:15.861 INFO:teuthology.task.internal:Saving configuration 2024-03-19T11:16:15.877 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-19T11:16:15.938 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-19T11:16:15.962 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi038.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rfriedma-2024-03-19_08:22:49-rados:thrash-wip-rf-nextobj-distro-default-smithi/7610739', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-19 11:10:37.930425', 'locked_by': 'scheduled_rfriedma@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICCTZCEyaDg9hWbICXcvJUE3tpKJIpvdkCSF2USExsAI'} 2024-03-19T11:16:15.984 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi172.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rfriedma-2024-03-19_08:22:49-rados:thrash-wip-rf-nextobj-distro-default-smithi/7610739', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-19 11:10:37.928766', 'locked_by': 'scheduled_rfriedma@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPnV98cT+tlJhzBR0spJyfU9rzVefi+RCltMjPJSzkKN'} 2024-03-19T11:16:15.984 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-19T11:16:15.990 INFO:teuthology.task.internal:roles: ubuntu@smithi038.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-19T11:16:15.990 INFO:teuthology.task.internal:roles: ubuntu@smithi172.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-19T11:16:15.991 INFO:teuthology.run_tasks:Running task console_log... 2024-03-19T11:16:16.076 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f06792990d0>, signals=[15]) 2024-03-19T11:16:16.076 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-19T11:16:16.084 INFO:teuthology.task.internal:Opening connections... 2024-03-19T11:16:16.084 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi038.front.sepia.ceph.com 2024-03-19T11:16:16.086 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T11:16:16.169 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi172.front.sepia.ceph.com 2024-03-19T11:16:16.170 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T11:16:16.247 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-19T11:16:16.254 DEBUG:teuthology.orchestra.run.smithi038:> uname -m 2024-03-19T11:16:16.259 INFO:teuthology.orchestra.run.smithi038.stdout:x86_64 2024-03-19T11:16:16.260 DEBUG:teuthology.orchestra.run.smithi038:> cat /etc/os-release 2024-03-19T11:16:16.305 INFO:teuthology.orchestra.run.smithi038.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-03-19T11:16:16.305 INFO:teuthology.orchestra.run.smithi038.stdout:NAME="Ubuntu" 2024-03-19T11:16:16.305 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION_ID="22.04" 2024-03-19T11:16:16.305 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-03-19T11:16:16.306 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION_CODENAME=jammy 2024-03-19T11:16:16.306 INFO:teuthology.orchestra.run.smithi038.stdout:ID=ubuntu 2024-03-19T11:16:16.306 INFO:teuthology.orchestra.run.smithi038.stdout:ID_LIKE=debian 2024-03-19T11:16:16.306 INFO:teuthology.orchestra.run.smithi038.stdout:HOME_URL="https://www.ubuntu.com/" 2024-03-19T11:16:16.306 INFO:teuthology.orchestra.run.smithi038.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-03-19T11:16:16.306 INFO:teuthology.orchestra.run.smithi038.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-03-19T11:16:16.306 INFO:teuthology.orchestra.run.smithi038.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-03-19T11:16:16.306 INFO:teuthology.orchestra.run.smithi038.stdout:UBUNTU_CODENAME=jammy 2024-03-19T11:16:16.307 INFO:teuthology.lock.ops:Updating smithi038.front.sepia.ceph.com on lock server 2024-03-19T11:16:16.336 DEBUG:teuthology.orchestra.run.smithi172:> uname -m 2024-03-19T11:16:16.341 INFO:teuthology.orchestra.run.smithi172.stdout:x86_64 2024-03-19T11:16:16.341 DEBUG:teuthology.orchestra.run.smithi172:> cat /etc/os-release 2024-03-19T11:16:16.386 INFO:teuthology.orchestra.run.smithi172.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-03-19T11:16:16.386 INFO:teuthology.orchestra.run.smithi172.stdout:NAME="Ubuntu" 2024-03-19T11:16:16.387 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION_ID="22.04" 2024-03-19T11:16:16.387 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-03-19T11:16:16.387 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION_CODENAME=jammy 2024-03-19T11:16:16.387 INFO:teuthology.orchestra.run.smithi172.stdout:ID=ubuntu 2024-03-19T11:16:16.387 INFO:teuthology.orchestra.run.smithi172.stdout:ID_LIKE=debian 2024-03-19T11:16:16.387 INFO:teuthology.orchestra.run.smithi172.stdout:HOME_URL="https://www.ubuntu.com/" 2024-03-19T11:16:16.387 INFO:teuthology.orchestra.run.smithi172.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-03-19T11:16:16.387 INFO:teuthology.orchestra.run.smithi172.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-03-19T11:16:16.387 INFO:teuthology.orchestra.run.smithi172.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-03-19T11:16:16.387 INFO:teuthology.orchestra.run.smithi172.stdout:UBUNTU_CODENAME=jammy 2024-03-19T11:16:16.388 INFO:teuthology.lock.ops:Updating smithi172.front.sepia.ceph.com on lock server 2024-03-19T11:16:16.410 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-19T11:16:16.421 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-19T11:16:16.435 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-19T11:16:16.436 DEBUG:teuthology.orchestra.run.smithi038:> test '!' -e /home/ubuntu/cephtest 2024-03-19T11:16:16.438 DEBUG:teuthology.orchestra.run.smithi172:> test '!' -e /home/ubuntu/cephtest 2024-03-19T11:16:16.441 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-19T11:16:16.453 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-19T11:16:16.453 DEBUG:teuthology.orchestra.run.smithi038:> test -z $(ls -A /var/lib/ceph) 2024-03-19T11:16:16.483 DEBUG:teuthology.orchestra.run.smithi172:> test -z $(ls -A /var/lib/ceph) 2024-03-19T11:16:16.489 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-19T11:16:16.544 INFO:teuthology.run_tasks:Running task kernel... 2024-03-19T11:16:16.558 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-19T11:16:16.559 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-19T11:16:16.559 DEBUG:teuthology.orchestra.run.smithi038:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-19T11:16:16.559 DEBUG:teuthology.orchestra.run.smithi172:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-19T11:16:16.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T11:16:16.564 DEBUG:teuthology.orchestra.run.smithi172:> uname -r 2024-03-19T11:16:16.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T11:16:16.565 DEBUG:teuthology.orchestra.run.smithi038:> uname -r 2024-03-19T11:16:16.611 INFO:teuthology.orchestra.run.smithi172.stdout:5.15.0-56-generic 2024-03-19T11:16:16.611 INFO:teuthology.task.kernel:Running kernel on smithi172: 5.15.0-56-generic 2024-03-19T11:16:16.611 DEBUG:teuthology.orchestra.run.smithi172:> sudo apt-get clean 2024-03-19T11:16:16.614 INFO:teuthology.orchestra.run.smithi038.stdout:5.15.0-56-generic 2024-03-19T11:16:16.614 INFO:teuthology.task.kernel:Running kernel on smithi038: 5.15.0-56-generic 2024-03-19T11:16:16.614 DEBUG:teuthology.orchestra.run.smithi038:> sudo apt-get clean 2024-03-19T11:16:16.745 DEBUG:teuthology.orchestra.run.smithi172:> sudo apt-get update 2024-03-19T11:16:16.774 DEBUG:teuthology.orchestra.run.smithi038:> sudo apt-get update 2024-03-19T11:16:16.896 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-19T11:16:16.934 INFO:teuthology.orchestra.run.smithi038.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-03-19T11:16:16.937 INFO:teuthology.orchestra.run.smithi172.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-03-19T11:16:16.987 INFO:teuthology.orchestra.run.smithi172.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-03-19T11:16:17.022 INFO:teuthology.orchestra.run.smithi038.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-19T11:16:17.071 INFO:teuthology.orchestra.run.smithi172.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-03-19T11:16:17.092 INFO:teuthology.orchestra.run.smithi038.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-03-19T11:16:17.157 INFO:teuthology.orchestra.run.smithi172.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [593 kB] 2024-03-19T11:16:17.310 INFO:teuthology.orchestra.run.smithi038.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,272 kB] 2024-03-19T11:16:17.347 INFO:teuthology.orchestra.run.smithi172.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,492 kB] 2024-03-19T11:16:17.424 INFO:teuthology.orchestra.run.smithi172.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [426 kB] 2024-03-19T11:16:17.453 INFO:teuthology.orchestra.run.smithi038.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-03-19T11:16:17.508 INFO:teuthology.orchestra.run.smithi172.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [288 kB] 2024-03-19T11:16:17.525 INFO:teuthology.orchestra.run.smithi172.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-03-19T11:16:17.527 INFO:teuthology.orchestra.run.smithi172.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [35.3 kB] 2024-03-19T11:16:17.534 INFO:teuthology.orchestra.run.smithi172.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,605 kB] 2024-03-19T11:16:17.597 INFO:teuthology.orchestra.run.smithi038.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,492 kB] 2024-03-19T11:16:17.671 INFO:teuthology.orchestra.run.smithi172.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [268 kB] 2024-03-19T11:16:17.676 INFO:teuthology.orchestra.run.smithi172.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-03-19T11:16:17.677 INFO:teuthology.orchestra.run.smithi172.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,058 kB] 2024-03-19T11:16:17.695 INFO:teuthology.orchestra.run.smithi172.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [696 kB] 2024-03-19T11:16:17.725 INFO:teuthology.orchestra.run.smithi038.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [426 kB] 2024-03-19T11:16:17.727 INFO:teuthology.orchestra.run.smithi172.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [239 kB] 2024-03-19T11:16:17.727 INFO:teuthology.orchestra.run.smithi172.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,272 kB] 2024-03-19T11:16:17.731 INFO:teuthology.orchestra.run.smithi172.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-03-19T11:16:17.731 INFO:teuthology.orchestra.run.smithi172.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,160 B] 2024-03-19T11:16:17.732 INFO:teuthology.orchestra.run.smithi172.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.1 kB] 2024-03-19T11:16:17.733 INFO:teuthology.orchestra.run.smithi172.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.1 kB] 2024-03-19T11:16:17.733 INFO:teuthology.orchestra.run.smithi172.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-03-19T11:16:17.737 INFO:teuthology.orchestra.run.smithi038.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [228 kB] 2024-03-19T11:16:17.746 INFO:teuthology.orchestra.run.smithi172.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-03-19T11:16:17.747 INFO:teuthology.orchestra.run.smithi172.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-03-19T11:16:17.748 INFO:teuthology.orchestra.run.smithi172.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-03-19T11:16:17.759 INFO:teuthology.orchestra.run.smithi038.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-03-19T11:16:17.762 INFO:teuthology.orchestra.run.smithi038.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [33.7 kB] 2024-03-19T11:16:17.762 INFO:teuthology.orchestra.run.smithi038.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,572 kB] 2024-03-19T11:16:17.768 INFO:teuthology.orchestra.run.smithi172.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-03-19T11:16:17.768 INFO:teuthology.orchestra.run.smithi172.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-03-19T11:16:17.768 INFO:teuthology.orchestra.run.smithi172.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-03-19T11:16:17.769 INFO:teuthology.orchestra.run.smithi172.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-03-19T11:16:17.769 INFO:teuthology.orchestra.run.smithi172.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-03-19T11:16:17.859 INFO:teuthology.orchestra.run.smithi038.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [263 kB] 2024-03-19T11:16:17.873 INFO:teuthology.orchestra.run.smithi038.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-03-19T11:16:17.874 INFO:teuthology.orchestra.run.smithi038.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [599 kB] 2024-03-19T11:16:17.874 INFO:teuthology.orchestra.run.smithi172.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [228 kB] 2024-03-19T11:16:17.887 INFO:teuthology.orchestra.run.smithi172.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-03-19T11:16:17.888 INFO:teuthology.orchestra.run.smithi172.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [33.7 kB] 2024-03-19T11:16:17.890 INFO:teuthology.orchestra.run.smithi172.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,572 kB] 2024-03-19T11:16:17.898 INFO:teuthology.orchestra.run.smithi038.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [593 kB] 2024-03-19T11:16:17.899 INFO:teuthology.orchestra.run.smithi038.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [851 kB] 2024-03-19T11:16:17.922 INFO:teuthology.orchestra.run.smithi038.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [162 kB] 2024-03-19T11:16:17.924 INFO:teuthology.orchestra.run.smithi038.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-03-19T11:16:17.936 INFO:teuthology.orchestra.run.smithi038.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-03-19T11:16:17.937 INFO:teuthology.orchestra.run.smithi038.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.1 kB] 2024-03-19T11:16:17.937 INFO:teuthology.orchestra.run.smithi038.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,476 B] 2024-03-19T11:16:17.938 INFO:teuthology.orchestra.run.smithi038.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-03-19T11:16:17.938 INFO:teuthology.orchestra.run.smithi038.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [288 kB] 2024-03-19T11:16:17.954 INFO:teuthology.orchestra.run.smithi038.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-03-19T11:16:17.954 INFO:teuthology.orchestra.run.smithi038.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,605 kB] 2024-03-19T11:16:17.972 INFO:teuthology.orchestra.run.smithi172.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [263 kB] 2024-03-19T11:16:17.981 INFO:teuthology.orchestra.run.smithi172.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-03-19T11:16:17.981 INFO:teuthology.orchestra.run.smithi172.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [599 kB] 2024-03-19T11:16:17.998 INFO:teuthology.orchestra.run.smithi172.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [851 kB] 2024-03-19T11:16:18.015 INFO:teuthology.orchestra.run.smithi038.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [35.3 kB] 2024-03-19T11:16:18.016 INFO:teuthology.orchestra.run.smithi038.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [268 kB] 2024-03-19T11:16:18.023 INFO:teuthology.orchestra.run.smithi038.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-03-19T11:16:18.188 INFO:teuthology.orchestra.run.smithi172.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [162 kB] 2024-03-19T11:16:18.191 INFO:teuthology.orchestra.run.smithi038.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,058 kB] 2024-03-19T11:16:18.592 INFO:teuthology.orchestra.run.smithi172.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-03-19T11:16:18.598 INFO:teuthology.orchestra.run.smithi172.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-03-19T11:16:18.598 INFO:teuthology.orchestra.run.smithi172.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.1 kB] 2024-03-19T11:16:18.613 INFO:teuthology.orchestra.run.smithi172.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,476 B] 2024-03-19T11:16:18.615 INFO:teuthology.orchestra.run.smithi172.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-03-19T11:16:18.824 INFO:teuthology.orchestra.run.smithi038.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [696 kB] 2024-03-19T11:16:18.863 INFO:teuthology.orchestra.run.smithi038.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [239 kB] 2024-03-19T11:16:18.877 INFO:teuthology.orchestra.run.smithi038.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-03-19T11:16:18.878 INFO:teuthology.orchestra.run.smithi038.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.1 kB] 2024-03-19T11:16:18.880 INFO:teuthology.orchestra.run.smithi038.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,160 B] 2024-03-19T11:16:18.881 INFO:teuthology.orchestra.run.smithi038.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.1 kB] 2024-03-19T11:16:18.881 INFO:teuthology.orchestra.run.smithi038.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-03-19T11:16:18.882 INFO:teuthology.orchestra.run.smithi038.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-03-19T11:16:18.907 INFO:teuthology.orchestra.run.smithi038.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-03-19T11:16:18.910 INFO:teuthology.orchestra.run.smithi038.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-03-19T11:16:19.074 INFO:teuthology.orchestra.run.smithi038.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-03-19T11:16:19.157 INFO:teuthology.orchestra.run.smithi038.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-03-19T11:16:19.264 INFO:teuthology.orchestra.run.smithi038.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-03-19T11:16:19.322 INFO:teuthology.orchestra.run.smithi038.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-03-19T11:16:19.356 INFO:teuthology.orchestra.run.smithi038.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-03-19T11:16:23.131 INFO:teuthology.orchestra.run.smithi172.stdout:Fetched 12.4 MB in 2s (6,227 kB/s) 2024-03-19T11:16:23.847 INFO:teuthology.orchestra.run.smithi038.stdout:Fetched 12.4 MB in 3s (4,897 kB/s) 2024-03-19T11:16:24.768 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2024-03-19T11:16:24.786 DEBUG:teuthology.orchestra.run.smithi172:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-19T11:16:24.840 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2024-03-19T11:16:25.035 INFO:teuthology.orchestra.run.smithi172.stdout:Building dependency tree... 2024-03-19T11:16:25.036 INFO:teuthology.orchestra.run.smithi172.stdout:Reading state information... 2024-03-19T11:16:25.191 INFO:teuthology.orchestra.run.smithi172.stdout:The following packages were automatically installed and are no longer required: 2024-03-19T11:16:25.191 INFO:teuthology.orchestra.run.smithi172.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-19T11:16:25.191 INFO:teuthology.orchestra.run.smithi172.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-19T11:16:25.191 INFO:teuthology.orchestra.run.smithi172.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-19T11:16:25.191 INFO:teuthology.orchestra.run.smithi172.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-03-19T11:16:25.192 INFO:teuthology.orchestra.run.smithi172.stdout: python2.7-minimal 2024-03-19T11:16:25.192 INFO:teuthology.orchestra.run.smithi172.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-19T11:16:25.193 INFO:teuthology.orchestra.run.smithi172.stdout:The following additional packages will be installed: 2024-03-19T11:16:25.193 INFO:teuthology.orchestra.run.smithi172.stdout: linux-generic linux-headers-5.15.0-101 linux-headers-5.15.0-101-generic 2024-03-19T11:16:25.193 INFO:teuthology.orchestra.run.smithi172.stdout: linux-headers-generic linux-image-5.15.0-101-generic 2024-03-19T11:16:25.193 INFO:teuthology.orchestra.run.smithi172.stdout: linux-modules-5.15.0-101-generic linux-modules-extra-5.15.0-101-generic 2024-03-19T11:16:25.194 INFO:teuthology.orchestra.run.smithi172.stdout:Suggested packages: 2024-03-19T11:16:25.194 INFO:teuthology.orchestra.run.smithi172.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-03-19T11:16:25.194 INFO:teuthology.orchestra.run.smithi172.stdout:Recommended packages: 2024-03-19T11:16:25.194 INFO:teuthology.orchestra.run.smithi172.stdout: thermald 2024-03-19T11:16:25.237 INFO:teuthology.orchestra.run.smithi172.stdout:The following NEW packages will be installed: 2024-03-19T11:16:25.237 INFO:teuthology.orchestra.run.smithi172.stdout: linux-headers-5.15.0-101 linux-headers-5.15.0-101-generic 2024-03-19T11:16:25.237 INFO:teuthology.orchestra.run.smithi172.stdout: linux-image-5.15.0-101-generic linux-modules-5.15.0-101-generic 2024-03-19T11:16:25.238 INFO:teuthology.orchestra.run.smithi172.stdout: linux-modules-extra-5.15.0-101-generic 2024-03-19T11:16:25.238 INFO:teuthology.orchestra.run.smithi172.stdout:The following packages will be upgraded: 2024-03-19T11:16:25.239 INFO:teuthology.orchestra.run.smithi172.stdout: linux-generic linux-headers-generic linux-image-generic 2024-03-19T11:16:25.265 INFO:teuthology.orchestra.run.smithi038.stdout:Reading package lists... 2024-03-19T11:16:25.284 DEBUG:teuthology.orchestra.run.smithi038:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-19T11:16:25.305 INFO:teuthology.orchestra.run.smithi172.stdout:3 upgraded, 5 newly installed, 0 to remove and 298 not upgraded. 2024-03-19T11:16:25.306 INFO:teuthology.orchestra.run.smithi172.stdout:Need to get 113 MB of archives. 2024-03-19T11:16:25.306 INFO:teuthology.orchestra.run.smithi172.stdout:After this operation, 582 MB of additional disk space will be used. 2024-03-19T11:16:25.306 INFO:teuthology.orchestra.run.smithi172.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-101-generic amd64 5.15.0-101.111 [22.6 MB] 2024-03-19T11:16:25.338 INFO:teuthology.orchestra.run.smithi038.stdout:Reading package lists... 2024-03-19T11:16:25.539 INFO:teuthology.orchestra.run.smithi038.stdout:Building dependency tree... 2024-03-19T11:16:25.539 INFO:teuthology.orchestra.run.smithi038.stdout:Reading state information... 2024-03-19T11:16:25.715 INFO:teuthology.orchestra.run.smithi038.stdout:The following packages were automatically installed and are no longer required: 2024-03-19T11:16:25.715 INFO:teuthology.orchestra.run.smithi038.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-19T11:16:25.715 INFO:teuthology.orchestra.run.smithi038.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-19T11:16:25.716 INFO:teuthology.orchestra.run.smithi038.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-19T11:16:25.716 INFO:teuthology.orchestra.run.smithi038.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-03-19T11:16:25.716 INFO:teuthology.orchestra.run.smithi038.stdout: python2.7-minimal 2024-03-19T11:16:25.716 INFO:teuthology.orchestra.run.smithi038.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-19T11:16:25.717 INFO:teuthology.orchestra.run.smithi038.stdout:The following additional packages will be installed: 2024-03-19T11:16:25.717 INFO:teuthology.orchestra.run.smithi038.stdout: linux-generic linux-headers-5.15.0-101 linux-headers-5.15.0-101-generic 2024-03-19T11:16:25.717 INFO:teuthology.orchestra.run.smithi038.stdout: linux-headers-generic linux-image-5.15.0-101-generic 2024-03-19T11:16:25.717 INFO:teuthology.orchestra.run.smithi038.stdout: linux-modules-5.15.0-101-generic linux-modules-extra-5.15.0-101-generic 2024-03-19T11:16:25.719 INFO:teuthology.orchestra.run.smithi038.stdout:Suggested packages: 2024-03-19T11:16:25.719 INFO:teuthology.orchestra.run.smithi038.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-03-19T11:16:25.719 INFO:teuthology.orchestra.run.smithi038.stdout:Recommended packages: 2024-03-19T11:16:25.719 INFO:teuthology.orchestra.run.smithi038.stdout: thermald 2024-03-19T11:16:25.757 INFO:teuthology.orchestra.run.smithi038.stdout:The following NEW packages will be installed: 2024-03-19T11:16:25.758 INFO:teuthology.orchestra.run.smithi038.stdout: linux-headers-5.15.0-101 linux-headers-5.15.0-101-generic 2024-03-19T11:16:25.758 INFO:teuthology.orchestra.run.smithi038.stdout: linux-image-5.15.0-101-generic linux-modules-5.15.0-101-generic 2024-03-19T11:16:25.758 INFO:teuthology.orchestra.run.smithi038.stdout: linux-modules-extra-5.15.0-101-generic 2024-03-19T11:16:25.759 INFO:teuthology.orchestra.run.smithi038.stdout:The following packages will be upgraded: 2024-03-19T11:16:25.760 INFO:teuthology.orchestra.run.smithi038.stdout: linux-generic linux-headers-generic linux-image-generic 2024-03-19T11:16:25.864 INFO:teuthology.orchestra.run.smithi038.stdout:3 upgraded, 5 newly installed, 0 to remove and 298 not upgraded. 2024-03-19T11:16:25.864 INFO:teuthology.orchestra.run.smithi038.stdout:Need to get 113 MB of archives. 2024-03-19T11:16:25.864 INFO:teuthology.orchestra.run.smithi038.stdout:After this operation, 582 MB of additional disk space will be used. 2024-03-19T11:16:25.865 INFO:teuthology.orchestra.run.smithi038.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-101-generic amd64 5.15.0-101.111 [22.6 MB] 2024-03-19T11:16:26.027 INFO:teuthology.orchestra.run.smithi172.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-101-generic amd64 5.15.0-101.111 [11.5 MB] 2024-03-19T11:16:26.275 INFO:teuthology.orchestra.run.smithi172.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-101-generic amd64 5.15.0-101.111 [63.9 MB] 2024-03-19T11:16:26.855 INFO:teuthology.orchestra.run.smithi038.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-101-generic amd64 5.15.0-101.111 [11.5 MB] 2024-03-19T11:16:27.149 INFO:teuthology.orchestra.run.smithi038.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-101-generic amd64 5.15.0-101.111 [63.9 MB] 2024-03-19T11:16:27.953 INFO:teuthology.orchestra.run.smithi172.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.101.98 [1,698 B] 2024-03-19T11:16:27.953 INFO:teuthology.orchestra.run.smithi172.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.101.98 [2,486 B] 2024-03-19T11:16:27.953 INFO:teuthology.orchestra.run.smithi172.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-101 all 5.15.0-101.111 [12.3 MB] 2024-03-19T11:16:28.191 INFO:teuthology.orchestra.run.smithi172.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-101-generic amd64 5.15.0-101.111 [2,876 kB] 2024-03-19T11:16:28.251 INFO:teuthology.orchestra.run.smithi172.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.101.98 [2,332 B] 2024-03-19T11:16:28.599 INFO:teuthology.orchestra.run.smithi038.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.101.98 [1,698 B] 2024-03-19T11:16:28.599 INFO:teuthology.orchestra.run.smithi038.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.101.98 [2,486 B] 2024-03-19T11:16:28.600 INFO:teuthology.orchestra.run.smithi038.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-101 all 5.15.0-101.111 [12.3 MB] 2024-03-19T11:16:28.651 INFO:teuthology.orchestra.run.smithi172.stdout:Fetched 113 MB in 3s (37.6 MB/s) 2024-03-19T11:16:28.779 INFO:teuthology.orchestra.run.smithi038.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-101-generic amd64 5.15.0-101.111 [2,876 kB] 2024-03-19T11:16:28.818 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package linux-modules-5.15.0-101-generic. 2024-03-19T11:16:28.841 INFO:teuthology.orchestra.run.smithi038.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.101.98 [2,332 B] 2024-03-19T11:16:29.226 INFO:teuthology.orchestra.run.smithi038.stdout:Fetched 113 MB in 3s (36.8 MB/s) 2024-03-19T11:16:29.385 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package linux-modules-5.15.0-101-generic. 2024-03-19T11:16:30.845 INFO:teuthology.orchestra.run.smithi172.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-03-19T11:16:30.849 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../0-linux-modules-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-19T11:16:30.948 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-modules-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T11:16:31.392 INFO:teuthology.orchestra.run.smithi038.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-03-19T11:16:31.396 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../0-linux-modules-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-19T11:16:31.473 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking linux-modules-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T11:16:33.049 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package linux-image-5.15.0-101-generic. 2024-03-19T11:16:33.070 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../1-linux-image-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-19T11:16:33.159 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T11:16:33.466 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package linux-image-5.15.0-101-generic. 2024-03-19T11:16:33.487 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../1-linux-image-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-19T11:16:33.574 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T11:16:33.960 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-101-generic. 2024-03-19T11:16:33.980 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-19T11:16:34.009 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-modules-extra-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T11:16:34.019 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-101-generic. 2024-03-19T11:16:34.040 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-19T11:16:34.077 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking linux-modules-extra-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T11:16:40.246 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../3-linux-generic_5.15.0.101.98_amd64.deb ... 2024-03-19T11:16:40.371 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking linux-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-19T11:16:40.393 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../3-linux-generic_5.15.0.101.98_amd64.deb ... 2024-03-19T11:16:40.534 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-19T11:16:40.687 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.101.98_amd64.deb ... 2024-03-19T11:16:40.807 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking linux-image-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-19T11:16:40.816 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.101.98_amd64.deb ... 2024-03-19T11:16:40.939 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-image-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-19T11:16:41.102 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package linux-headers-5.15.0-101. 2024-03-19T11:16:41.113 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../5-linux-headers-5.15.0-101_5.15.0-101.111_all.deb ... 2024-03-19T11:16:41.143 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking linux-headers-5.15.0-101 (5.15.0-101.111) ... 2024-03-19T11:16:41.242 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package linux-headers-5.15.0-101. 2024-03-19T11:16:41.253 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../5-linux-headers-5.15.0-101_5.15.0-101.111_all.deb ... 2024-03-19T11:16:41.292 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-headers-5.15.0-101 (5.15.0-101.111) ... 2024-03-19T11:16:46.784 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package linux-headers-5.15.0-101-generic. 2024-03-19T11:16:46.821 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../6-linux-headers-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-19T11:16:46.858 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-headers-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T11:16:46.952 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package linux-headers-5.15.0-101-generic. 2024-03-19T11:16:46.973 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../6-linux-headers-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-19T11:16:47.010 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking linux-headers-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T11:16:49.655 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.101.98_amd64.deb ... 2024-03-19T11:16:49.780 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking linux-headers-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-19T11:16:50.009 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.101.98_amd64.deb ... 2024-03-19T11:16:50.089 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up linux-headers-5.15.0-101 (5.15.0-101.111) ... 2024-03-19T11:16:50.135 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-headers-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-19T11:16:50.218 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up linux-headers-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T11:16:50.344 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up linux-headers-generic (5.15.0.101.98) ... 2024-03-19T11:16:50.473 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T11:16:50.502 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-headers-5.15.0-101 (5.15.0-101.111) ... 2024-03-19T11:16:50.623 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-headers-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T11:16:50.749 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-headers-generic (5.15.0.101.98) ... 2024-03-19T11:16:50.853 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T11:16:51.869 INFO:teuthology.orchestra.run.smithi038.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-03-19T11:16:51.869 INFO:teuthology.orchestra.run.smithi038.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-03-19T11:16:51.869 INFO:teuthology.orchestra.run.smithi038.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-101-generic 2024-03-19T11:16:51.869 INFO:teuthology.orchestra.run.smithi038.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-101-generic 2024-03-19T11:16:52.034 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up linux-modules-extra-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T11:16:52.159 INFO:teuthology.orchestra.run.smithi172.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-03-19T11:16:52.159 INFO:teuthology.orchestra.run.smithi172.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-03-19T11:16:52.159 INFO:teuthology.orchestra.run.smithi172.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-101-generic 2024-03-19T11:16:52.159 INFO:teuthology.orchestra.run.smithi172.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-101-generic 2024-03-19T11:16:52.313 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-modules-extra-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T11:16:53.476 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up linux-modules-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T11:16:53.714 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-modules-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T11:16:54.878 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up linux-image-generic (5.15.0.101.98) ... 2024-03-19T11:16:54.987 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up linux-generic (5.15.0.101.98) ... 2024-03-19T11:16:55.121 INFO:teuthology.orchestra.run.smithi038.stdout:Processing triggers for linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T11:16:55.132 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-image-generic (5.15.0.101.98) ... 2024-03-19T11:16:55.182 INFO:teuthology.orchestra.run.smithi038.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-03-19T11:16:55.182 INFO:teuthology.orchestra.run.smithi038.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-101-generic 2024-03-19T11:16:55.257 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-generic (5.15.0.101.98) ... 2024-03-19T11:16:55.367 INFO:teuthology.orchestra.run.smithi172.stdout:Processing triggers for linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-19T11:16:55.422 INFO:teuthology.orchestra.run.smithi172.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-03-19T11:16:55.422 INFO:teuthology.orchestra.run.smithi172.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-101-generic 2024-03-19T11:17:09.564 INFO:teuthology.orchestra.run.smithi172.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-03-19T11:17:09.565 INFO:teuthology.orchestra.run.smithi172.stdout:Sourcing file `/etc/default/grub' 2024-03-19T11:17:09.591 INFO:teuthology.orchestra.run.smithi172.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-19T11:17:09.616 INFO:teuthology.orchestra.run.smithi172.stdout:Generating grub configuration file ... 2024-03-19T11:17:09.631 INFO:teuthology.orchestra.run.smithi038.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-03-19T11:17:09.631 INFO:teuthology.orchestra.run.smithi038.stdout:Sourcing file `/etc/default/grub' 2024-03-19T11:17:09.649 INFO:teuthology.orchestra.run.smithi038.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-19T11:17:09.673 INFO:teuthology.orchestra.run.smithi038.stdout:Generating grub configuration file ... 2024-03-19T11:17:09.894 INFO:teuthology.orchestra.run.smithi172.stdout:Found linux image: /boot/vmlinuz-5.15.0-101-generic 2024-03-19T11:17:09.918 INFO:teuthology.orchestra.run.smithi172.stdout:Found initrd image: /boot/initrd.img-5.15.0-101-generic 2024-03-19T11:17:10.010 INFO:teuthology.orchestra.run.smithi038.stdout:Found linux image: /boot/vmlinuz-5.15.0-101-generic 2024-03-19T11:17:10.040 INFO:teuthology.orchestra.run.smithi038.stdout:Found initrd image: /boot/initrd.img-5.15.0-101-generic 2024-03-19T11:17:10.204 INFO:teuthology.orchestra.run.smithi172.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-19T11:17:10.210 INFO:teuthology.orchestra.run.smithi172.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-19T11:17:10.285 INFO:teuthology.orchestra.run.smithi172.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-19T11:17:10.291 INFO:teuthology.orchestra.run.smithi172.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-19T11:17:10.401 INFO:teuthology.orchestra.run.smithi038.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-19T11:17:10.407 INFO:teuthology.orchestra.run.smithi038.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-19T11:17:10.434 INFO:teuthology.orchestra.run.smithi172.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-19T11:17:10.434 INFO:teuthology.orchestra.run.smithi172.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-03-19T11:17:10.434 INFO:teuthology.orchestra.run.smithi172.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-19T11:17:10.473 INFO:teuthology.orchestra.run.smithi038.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-19T11:17:10.479 INFO:teuthology.orchestra.run.smithi038.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-19T11:17:10.479 INFO:teuthology.orchestra.run.smithi172.stdout:done 2024-03-19T11:17:10.633 INFO:teuthology.orchestra.run.smithi038.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-19T11:17:10.633 INFO:teuthology.orchestra.run.smithi038.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-03-19T11:17:10.633 INFO:teuthology.orchestra.run.smithi038.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-19T11:17:10.675 INFO:teuthology.orchestra.run.smithi038.stdout:done 2024-03-19T11:17:10.962 DEBUG:teuthology.orchestra.run.smithi172:> dpkg -s linux-image-generic 2024-03-19T11:17:10.981 INFO:teuthology.orchestra.run.smithi172.stdout:Package: linux-image-generic 2024-03-19T11:17:10.981 INFO:teuthology.orchestra.run.smithi172.stdout:Status: install ok installed 2024-03-19T11:17:10.981 INFO:teuthology.orchestra.run.smithi172.stdout:Priority: optional 2024-03-19T11:17:10.981 INFO:teuthology.orchestra.run.smithi172.stdout:Section: kernel 2024-03-19T11:17:10.981 INFO:teuthology.orchestra.run.smithi172.stdout:Installed-Size: 21 2024-03-19T11:17:10.981 INFO:teuthology.orchestra.run.smithi172.stdout:Maintainer: Ubuntu Kernel Team 2024-03-19T11:17:10.982 INFO:teuthology.orchestra.run.smithi172.stdout:Architecture: amd64 2024-03-19T11:17:10.982 INFO:teuthology.orchestra.run.smithi172.stdout:Source: linux-meta 2024-03-19T11:17:10.982 INFO:teuthology.orchestra.run.smithi172.stdout:Version: 5.15.0.101.98 2024-03-19T11:17:10.982 INFO:teuthology.orchestra.run.smithi172.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-101), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-03-19T11:17:10.982 INFO:teuthology.orchestra.run.smithi172.stdout:Depends: linux-image-5.15.0-101-generic, linux-modules-extra-5.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-19T11:17:10.982 INFO:teuthology.orchestra.run.smithi172.stdout:Recommends: thermald 2024-03-19T11:17:10.982 INFO:teuthology.orchestra.run.smithi172.stdout:Description: Generic Linux kernel image 2024-03-19T11:17:10.982 INFO:teuthology.orchestra.run.smithi172.stdout: This package will always depend on the latest generic kernel image 2024-03-19T11:17:10.982 INFO:teuthology.orchestra.run.smithi172.stdout: available. 2024-03-19T11:17:10.982 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-101-generic 2024-03-19T11:17:10.983 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-19T11:17:10.983 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-19T11:17:10.983 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-19T11:17:10.983 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi172.front.sepia.ceph.com, path=None, version=distro) 2024-03-19T11:17:10.983 DEBUG:teuthology.orchestra.run.smithi172:> sudo apt-get clean 2024-03-19T11:17:11.131 DEBUG:teuthology.orchestra.run.smithi172:> sudo apt-get update 2024-03-19T11:17:11.132 DEBUG:teuthology.orchestra.run.smithi038:> dpkg -s linux-image-generic 2024-03-19T11:17:11.156 INFO:teuthology.orchestra.run.smithi038.stdout:Package: linux-image-generic 2024-03-19T11:17:11.156 INFO:teuthology.orchestra.run.smithi038.stdout:Status: install ok installed 2024-03-19T11:17:11.156 INFO:teuthology.orchestra.run.smithi038.stdout:Priority: optional 2024-03-19T11:17:11.157 INFO:teuthology.orchestra.run.smithi038.stdout:Section: kernel 2024-03-19T11:17:11.157 INFO:teuthology.orchestra.run.smithi038.stdout:Installed-Size: 21 2024-03-19T11:17:11.157 INFO:teuthology.orchestra.run.smithi038.stdout:Maintainer: Ubuntu Kernel Team 2024-03-19T11:17:11.157 INFO:teuthology.orchestra.run.smithi038.stdout:Architecture: amd64 2024-03-19T11:17:11.157 INFO:teuthology.orchestra.run.smithi038.stdout:Source: linux-meta 2024-03-19T11:17:11.157 INFO:teuthology.orchestra.run.smithi038.stdout:Version: 5.15.0.101.98 2024-03-19T11:17:11.157 INFO:teuthology.orchestra.run.smithi038.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-101), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-03-19T11:17:11.157 INFO:teuthology.orchestra.run.smithi038.stdout:Depends: linux-image-5.15.0-101-generic, linux-modules-extra-5.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-19T11:17:11.157 INFO:teuthology.orchestra.run.smithi038.stdout:Recommends: thermald 2024-03-19T11:17:11.157 INFO:teuthology.orchestra.run.smithi038.stdout:Description: Generic Linux kernel image 2024-03-19T11:17:11.157 INFO:teuthology.orchestra.run.smithi038.stdout: This package will always depend on the latest generic kernel image 2024-03-19T11:17:11.157 INFO:teuthology.orchestra.run.smithi038.stdout: available. 2024-03-19T11:17:11.157 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-101-generic 2024-03-19T11:17:11.157 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-19T11:17:11.158 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-19T11:17:11.158 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-19T11:17:11.158 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi038.front.sepia.ceph.com, path=None, version=distro) 2024-03-19T11:17:11.158 DEBUG:teuthology.orchestra.run.smithi038:> sudo apt-get clean 2024-03-19T11:17:11.299 DEBUG:teuthology.orchestra.run.smithi038:> sudo apt-get update 2024-03-19T11:17:11.467 INFO:teuthology.orchestra.run.smithi038.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-19T11:17:11.584 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-19T11:17:11.820 INFO:teuthology.orchestra.run.smithi038.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-19T11:17:11.840 INFO:teuthology.orchestra.run.smithi038.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-19T11:17:11.861 INFO:teuthology.orchestra.run.smithi038.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-19T11:17:12.710 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-19T11:17:12.729 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-19T11:17:12.750 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-19T11:17:13.142 INFO:teuthology.orchestra.run.smithi038.stdout:Reading package lists... 2024-03-19T11:17:13.159 DEBUG:teuthology.orchestra.run.smithi038:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-19T11:17:13.213 INFO:teuthology.orchestra.run.smithi038.stdout:Reading package lists... 2024-03-19T11:17:13.392 INFO:teuthology.orchestra.run.smithi038.stdout:Building dependency tree... 2024-03-19T11:17:13.392 INFO:teuthology.orchestra.run.smithi038.stdout:Reading state information... 2024-03-19T11:17:13.553 INFO:teuthology.orchestra.run.smithi038.stdout:linux-image-generic is already the newest version (5.15.0.101.98). 2024-03-19T11:17:13.553 INFO:teuthology.orchestra.run.smithi038.stdout:The following packages were automatically installed and are no longer required: 2024-03-19T11:17:13.553 INFO:teuthology.orchestra.run.smithi038.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-19T11:17:13.553 INFO:teuthology.orchestra.run.smithi038.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-19T11:17:13.553 INFO:teuthology.orchestra.run.smithi038.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-19T11:17:13.553 INFO:teuthology.orchestra.run.smithi038.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-19T11:17:13.554 INFO:teuthology.orchestra.run.smithi038.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-19T11:17:13.554 INFO:teuthology.orchestra.run.smithi038.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-19T11:17:13.593 INFO:teuthology.orchestra.run.smithi038.stdout:0 upgraded, 0 newly installed, 0 to remove and 298 not upgraded. 2024-03-19T11:17:13.595 DEBUG:teuthology.orchestra.run.smithi038:> dpkg -s linux-image-generic 2024-03-19T11:17:13.648 INFO:teuthology.orchestra.run.smithi038.stdout:Package: linux-image-generic 2024-03-19T11:17:13.648 INFO:teuthology.orchestra.run.smithi038.stdout:Status: install ok installed 2024-03-19T11:17:13.648 INFO:teuthology.orchestra.run.smithi038.stdout:Priority: optional 2024-03-19T11:17:13.648 INFO:teuthology.orchestra.run.smithi038.stdout:Section: kernel 2024-03-19T11:17:13.648 INFO:teuthology.orchestra.run.smithi038.stdout:Installed-Size: 21 2024-03-19T11:17:13.649 INFO:teuthology.orchestra.run.smithi038.stdout:Maintainer: Ubuntu Kernel Team 2024-03-19T11:17:13.649 INFO:teuthology.orchestra.run.smithi038.stdout:Architecture: amd64 2024-03-19T11:17:13.649 INFO:teuthology.orchestra.run.smithi038.stdout:Source: linux-meta 2024-03-19T11:17:13.649 INFO:teuthology.orchestra.run.smithi038.stdout:Version: 5.15.0.101.98 2024-03-19T11:17:13.649 INFO:teuthology.orchestra.run.smithi038.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-101), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-03-19T11:17:13.649 INFO:teuthology.orchestra.run.smithi038.stdout:Depends: linux-image-5.15.0-101-generic, linux-modules-extra-5.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-19T11:17:13.649 INFO:teuthology.orchestra.run.smithi038.stdout:Recommends: thermald 2024-03-19T11:17:13.649 INFO:teuthology.orchestra.run.smithi038.stdout:Description: Generic Linux kernel image 2024-03-19T11:17:13.649 INFO:teuthology.orchestra.run.smithi038.stdout: This package will always depend on the latest generic kernel image 2024-03-19T11:17:13.649 INFO:teuthology.orchestra.run.smithi038.stdout: available. 2024-03-19T11:17:13.649 DEBUG:teuthology.orchestra.run.smithi038:> mktemp 2024-03-19T11:17:13.696 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/tmp.6SUReyy114 2024-03-19T11:17:13.696 DEBUG:teuthology.orchestra.run.smithi038:> sudo cp /boot/grub/grub.cfg /tmp/tmp.6SUReyy114 2024-03-19T11:17:13.750 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 0666 /tmp/tmp.6SUReyy114 2024-03-19T11:17:13.867 DEBUG:teuthology.orchestra.remote:smithi038:/tmp/tmp.6SUReyy114 is 10KB 2024-03-19T11:17:13.880 DEBUG:teuthology.orchestra.run.smithi038:> rm -fr /tmp/tmp.6SUReyy114 2024-03-19T11:17:13.885 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-03-19T11:17:13.943 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-19T11:17:13.961 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-03-19T11:17:13.961 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-03-19T11:17:13.975 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2024-03-19T11:17:13.993 DEBUG:teuthology.orchestra.run.smithi172:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-19T11:17:14.047 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-101-generic 2024-03-19T11:17:14.047 DEBUG:teuthology.orchestra.run.smithi038:> sudo update-grub 2024-03-19T11:17:14.059 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2024-03-19T11:17:14.254 INFO:teuthology.orchestra.run.smithi172.stdout:Building dependency tree... 2024-03-19T11:17:14.255 INFO:teuthology.orchestra.run.smithi172.stdout:Reading state information... 2024-03-19T11:17:14.426 INFO:teuthology.orchestra.run.smithi172.stdout:linux-image-generic is already the newest version (5.15.0.101.98). 2024-03-19T11:17:14.426 INFO:teuthology.orchestra.run.smithi172.stdout:The following packages were automatically installed and are no longer required: 2024-03-19T11:17:14.426 INFO:teuthology.orchestra.run.smithi172.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-19T11:17:14.426 INFO:teuthology.orchestra.run.smithi172.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-19T11:17:14.426 INFO:teuthology.orchestra.run.smithi172.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-19T11:17:14.427 INFO:teuthology.orchestra.run.smithi172.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-19T11:17:14.427 INFO:teuthology.orchestra.run.smithi172.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-19T11:17:14.427 INFO:teuthology.orchestra.run.smithi172.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-19T11:17:14.469 INFO:teuthology.orchestra.run.smithi172.stdout:0 upgraded, 0 newly installed, 0 to remove and 298 not upgraded. 2024-03-19T11:17:14.470 DEBUG:teuthology.orchestra.run.smithi172:> dpkg -s linux-image-generic 2024-03-19T11:17:14.486 INFO:teuthology.orchestra.run.smithi172.stdout:Package: linux-image-generic 2024-03-19T11:17:14.486 INFO:teuthology.orchestra.run.smithi172.stdout:Status: install ok installed 2024-03-19T11:17:14.486 INFO:teuthology.orchestra.run.smithi172.stdout:Priority: optional 2024-03-19T11:17:14.486 INFO:teuthology.orchestra.run.smithi172.stdout:Section: kernel 2024-03-19T11:17:14.486 INFO:teuthology.orchestra.run.smithi172.stdout:Installed-Size: 21 2024-03-19T11:17:14.486 INFO:teuthology.orchestra.run.smithi172.stdout:Maintainer: Ubuntu Kernel Team 2024-03-19T11:17:14.486 INFO:teuthology.orchestra.run.smithi172.stdout:Architecture: amd64 2024-03-19T11:17:14.486 INFO:teuthology.orchestra.run.smithi172.stdout:Source: linux-meta 2024-03-19T11:17:14.487 INFO:teuthology.orchestra.run.smithi172.stdout:Version: 5.15.0.101.98 2024-03-19T11:17:14.487 INFO:teuthology.orchestra.run.smithi172.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-101), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-03-19T11:17:14.487 INFO:teuthology.orchestra.run.smithi172.stdout:Depends: linux-image-5.15.0-101-generic, linux-modules-extra-5.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-19T11:17:14.487 INFO:teuthology.orchestra.run.smithi172.stdout:Recommends: thermald 2024-03-19T11:17:14.487 INFO:teuthology.orchestra.run.smithi172.stdout:Description: Generic Linux kernel image 2024-03-19T11:17:14.487 INFO:teuthology.orchestra.run.smithi172.stdout: This package will always depend on the latest generic kernel image 2024-03-19T11:17:14.487 INFO:teuthology.orchestra.run.smithi172.stdout: available. 2024-03-19T11:17:14.487 DEBUG:teuthology.orchestra.run.smithi172:> mktemp 2024-03-19T11:17:14.533 INFO:teuthology.orchestra.run.smithi172.stdout:/tmp/tmp.XfAiIlTLLs 2024-03-19T11:17:14.534 DEBUG:teuthology.orchestra.run.smithi172:> sudo cp /boot/grub/grub.cfg /tmp/tmp.XfAiIlTLLs 2024-03-19T11:17:14.589 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0666 /tmp/tmp.XfAiIlTLLs 2024-03-19T11:17:14.709 DEBUG:teuthology.orchestra.remote:smithi172:/tmp/tmp.XfAiIlTLLs is 10KB 2024-03-19T11:17:14.722 DEBUG:teuthology.orchestra.run.smithi172:> rm -fr /tmp/tmp.XfAiIlTLLs 2024-03-19T11:17:14.728 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-03-19T11:17:14.781 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-19T11:17:14.781 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-03-19T11:17:14.781 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-03-19T11:17:14.843 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-101-generic 2024-03-19T11:17:14.843 DEBUG:teuthology.orchestra.run.smithi172:> sudo update-grub 2024-03-19T11:17:15.226 INFO:teuthology.orchestra.run.smithi038.stderr:Sourcing file `/etc/default/grub' 2024-03-19T11:17:15.227 INFO:teuthology.orchestra.run.smithi038.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-19T11:17:15.230 INFO:teuthology.orchestra.run.smithi038.stderr:Generating grub configuration file ... 2024-03-19T11:17:15.478 INFO:teuthology.orchestra.run.smithi038.stderr:Found linux image: /boot/vmlinuz-5.15.0-101-generic 2024-03-19T11:17:15.483 INFO:teuthology.orchestra.run.smithi038.stderr:Found initrd image: /boot/initrd.img-5.15.0-101-generic 2024-03-19T11:17:15.773 INFO:teuthology.orchestra.run.smithi038.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-19T11:17:15.778 INFO:teuthology.orchestra.run.smithi038.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-19T11:17:15.820 INFO:teuthology.orchestra.run.smithi038.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-19T11:17:15.826 INFO:teuthology.orchestra.run.smithi038.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-19T11:17:15.960 INFO:teuthology.orchestra.run.smithi038.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-19T11:17:15.960 INFO:teuthology.orchestra.run.smithi038.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-03-19T11:17:15.960 INFO:teuthology.orchestra.run.smithi038.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-19T11:17:15.971 INFO:teuthology.orchestra.run.smithi038.stderr:done 2024-03-19T11:17:15.987 DEBUG:teuthology.orchestra.run.smithi038:> sudo shutdown -r now 2024-03-19T11:17:16.010 INFO:teuthology.orchestra.run.smithi172.stderr:Sourcing file `/etc/default/grub' 2024-03-19T11:17:16.011 INFO:teuthology.orchestra.run.smithi172.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-19T11:17:16.013 INFO:teuthology.orchestra.run.smithi172.stderr:Generating grub configuration file ... 2024-03-19T11:17:16.197 INFO:teuthology.orchestra.run.smithi172.stderr:Found linux image: /boot/vmlinuz-5.15.0-101-generic 2024-03-19T11:17:16.203 INFO:teuthology.orchestra.run.smithi172.stderr:Found initrd image: /boot/initrd.img-5.15.0-101-generic 2024-03-19T11:17:16.429 INFO:teuthology.orchestra.run.smithi172.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-19T11:17:16.434 INFO:teuthology.orchestra.run.smithi172.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-19T11:17:16.477 INFO:teuthology.orchestra.run.smithi172.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-19T11:17:16.483 INFO:teuthology.orchestra.run.smithi172.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-19T11:17:16.606 INFO:teuthology.orchestra.run.smithi172.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-19T11:17:16.606 INFO:teuthology.orchestra.run.smithi172.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-03-19T11:17:16.606 INFO:teuthology.orchestra.run.smithi172.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-19T11:17:16.628 INFO:teuthology.orchestra.run.smithi172.stderr:done 2024-03-19T11:17:16.637 DEBUG:teuthology.orchestra.run.smithi172:> sudo shutdown -r now 2024-03-19T11:17:46.018 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-19T11:17:46.019 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-03-19T11:17:46.020 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T11:17:46.641 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-19T11:17:46.641 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-03-19T11:17:46.641 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T11:18:04.579 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.38 2024-03-19T11:18:05.087 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.172 2024-03-19T11:18:13.586 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-03-19T11:18:13.587 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T11:18:14.088 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-03-19T11:18:14.089 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T11:18:16.638 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.38 2024-03-19T11:18:17.154 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.172 2024-03-19T11:18:28.650 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-03-19T11:18:28.651 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T11:18:29.156 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-03-19T11:18:29.157 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T11:19:28.711 DEBUG:teuthology.orchestra.remote:timed out 2024-03-19T11:19:29.159 DEBUG:teuthology.orchestra.remote:timed out 2024-03-19T11:19:43.726 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-03-19T11:19:43.728 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T11:19:43.934 DEBUG:teuthology.orchestra.run.smithi038:> true 2024-03-19T11:19:44.159 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-03-19T11:19:44.160 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T11:19:44.451 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-03-19T11:19:44.451 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-101-generic"... 2024-03-19T11:19:44.451 DEBUG:teuthology.orchestra.run.smithi038:> uname -r 2024-03-19T11:19:44.495 DEBUG:teuthology.orchestra.run.smithi172:> true 2024-03-19T11:19:44.497 INFO:teuthology.orchestra.run.smithi038.stdout:5.15.0-101-generic 2024-03-19T11:19:44.497 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-101-generic vs 5.15.0-101-generic 2024-03-19T11:19:44.497 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-19T11:19:44.497 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-03-19T11:19:45.091 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-03-19T11:19:45.091 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-101-generic"... 2024-03-19T11:19:45.091 DEBUG:teuthology.orchestra.run.smithi172:> uname -r 2024-03-19T11:19:45.138 INFO:teuthology.orchestra.run.smithi172.stdout:5.15.0-101-generic 2024-03-19T11:19:45.138 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-101-generic vs 5.15.0-101-generic 2024-03-19T11:19:45.138 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-19T11:19:45.138 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-03-19T11:19:45.498 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-19T11:19:45.498 DEBUG:teuthology.orchestra.run.smithi038:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-19T11:19:45.578 INFO:teuthology.orchestra.run.smithi038.stdout:ttyS1 2024-03-19T11:19:45.595 DEBUG:teuthology.parallel:result is None 2024-03-19T11:19:46.139 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-19T11:19:46.140 DEBUG:teuthology.orchestra.run.smithi172:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-19T11:19:46.490 INFO:teuthology.orchestra.run.smithi172.stdout:ttyS1 2024-03-19T11:19:46.523 DEBUG:teuthology.parallel:result is None 2024-03-19T11:19:46.523 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-19T11:19:46.530 INFO:teuthology.task.internal:Creating test directory... 2024-03-19T11:19:46.531 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-19T11:19:46.533 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-19T11:19:46.538 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-19T11:19:46.576 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-19T11:19:46.612 INFO:teuthology.task.internal:Creating archive directory... 2024-03-19T11:19:46.613 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-19T11:19:46.616 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-19T11:19:46.638 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-19T11:19:46.645 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-19T11:19:46.645 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-19T11:19:46.679 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-19T11:19:46.696 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-19T11:19:46.703 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-19T11:19:46.706 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-19T11:19:46.711 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-19T11:19:46.714 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-19T11:19:46.722 INFO:teuthology.task.internal:Configuring sudo... 2024-03-19T11:19:46.722 DEBUG:teuthology.orchestra.run.smithi038:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-19T11:19:46.754 DEBUG:teuthology.orchestra.run.smithi172:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-19T11:19:46.769 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-19T11:19:46.778 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-19T11:19:46.779 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-19T11:19:46.810 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-19T11:19:46.818 DEBUG:teuthology.orchestra.run.smithi038:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-19T11:19:46.861 DEBUG:teuthology.orchestra.run.smithi038:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-19T11:19:46.909 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-19T11:19:46.909 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-19T11:19:46.965 DEBUG:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-19T11:19:46.971 DEBUG:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-19T11:19:47.018 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-19T11:19:47.018 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-19T11:19:47.088 DEBUG:teuthology.orchestra.run.smithi038:> sudo service rsyslog restart 2024-03-19T11:19:47.090 DEBUG:teuthology.orchestra.run.smithi172:> sudo service rsyslog restart 2024-03-19T11:19:47.162 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-19T11:19:47.170 INFO:teuthology.task.internal:Starting timer... 2024-03-19T11:19:47.170 INFO:teuthology.run_tasks:Running task pcp... 2024-03-19T11:19:47.197 INFO:teuthology.run_tasks:Running task selinux... 2024-03-19T11:19:47.230 DEBUG:teuthology.task.selinux:Excluding smithi038: OS 'ubuntu' does not support SELinux 2024-03-19T11:19:47.230 DEBUG:teuthology.task.selinux:Excluding smithi172: OS 'ubuntu' does not support SELinux 2024-03-19T11:19:47.230 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-19T11:19:47.230 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-03-19T11:19:47.230 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-19T11:19:47.230 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-19T11:19:47.262 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-19T11:19:47.267 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-19T11:19:47.383 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-19T11:19:47.402 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-19T11:19:47.403 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi038.front.sepia.ceph.com,smithi172.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-19T11:26:50.721 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi038.front.sepia.ceph.com'), Remote(name='ubuntu@smithi172.front.sepia.ceph.com')] 2024-03-19T11:26:50.723 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-03-19T11:26:50.725 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T11:26:50.807 DEBUG:teuthology.orchestra.run.smithi038:> true 2024-03-19T11:26:50.886 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-03-19T11:26:50.887 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-03-19T11:26:50.887 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T11:26:50.966 DEBUG:teuthology.orchestra.run.smithi172:> true 2024-03-19T11:26:51.046 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-03-19T11:26:51.046 INFO:teuthology.run_tasks:Running task clock... 2024-03-19T11:26:51.057 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-19T11:26:51.057 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-19T11:26:51.057 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-19T11:26:51.060 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-19T11:26:51.060 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-19T11:26:51.087 INFO:teuthology.orchestra.run.smithi038.stdout:19 Mar 11:26:51 ntpd[17100]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-03-19T11:26:51.087 INFO:teuthology.orchestra.run.smithi038.stdout:19 Mar 11:26:51 ntpd[17100]: Command line: ntpd -gq 2024-03-19T11:26:51.087 INFO:teuthology.orchestra.run.smithi038.stdout:19 Mar 11:26:51 ntpd[17100]: ---------------------------------------------------- 2024-03-19T11:26:51.087 INFO:teuthology.orchestra.run.smithi038.stdout:19 Mar 11:26:51 ntpd[17100]: ntp-4 is maintained by Network Time Foundation, 2024-03-19T11:26:51.087 INFO:teuthology.orchestra.run.smithi038.stdout:19 Mar 11:26:51 ntpd[17100]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-03-19T11:26:51.087 INFO:teuthology.orchestra.run.smithi038.stdout:19 Mar 11:26:51 ntpd[17100]: corporation. Support and training for ntp-4 are 2024-03-19T11:26:51.087 INFO:teuthology.orchestra.run.smithi038.stdout:19 Mar 11:26:51 ntpd[17100]: available at https://www.nwtime.org/support 2024-03-19T11:26:51.087 INFO:teuthology.orchestra.run.smithi038.stdout:19 Mar 11:26:51 ntpd[17100]: ---------------------------------------------------- 2024-03-19T11:26:51.088 INFO:teuthology.orchestra.run.smithi038.stdout:19 Mar 11:26:51 ntpd[17100]: proto: precision = 0.067 usec (-24) 2024-03-19T11:26:51.088 INFO:teuthology.orchestra.run.smithi038.stdout:19 Mar 11:26:51 ntpd[17100]: basedate set to 2022-02-04 2024-03-19T11:26:51.088 INFO:teuthology.orchestra.run.smithi038.stdout:19 Mar 11:26:51 ntpd[17100]: gps base set to 2022-02-06 (week 2196) 2024-03-19T11:26:51.088 INFO:teuthology.orchestra.run.smithi038.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-19T11:26:51.089 INFO:teuthology.orchestra.run.smithi038.stdout:19 Mar 11:26:51 ntpd[17100]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-19T11:26:51.089 INFO:teuthology.orchestra.run.smithi038.stdout:19 Mar 11:26:51 ntpd[17100]: restrict ::: KOD does nothing without LIMITED. 2024-03-19T11:26:51.089 INFO:teuthology.orchestra.run.smithi038.stderr:restrict ::: KOD does nothing without LIMITED. 2024-03-19T11:26:51.090 INFO:teuthology.orchestra.run.smithi038.stdout:19 Mar 11:26:51 ntpd[17100]: Listen and drop on 0 v6wildcard [::]:123 2024-03-19T11:26:51.090 INFO:teuthology.orchestra.run.smithi038.stdout:19 Mar 11:26:51 ntpd[17100]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-03-19T11:26:51.090 INFO:teuthology.orchestra.run.smithi038.stdout:19 Mar 11:26:51 ntpd[17100]: Listen normally on 2 lo 127.0.0.1:123 2024-03-19T11:26:51.090 INFO:teuthology.orchestra.run.smithi038.stdout:19 Mar 11:26:51 ntpd[17100]: Listen normally on 3 ens1f0 172.21.15.38:123 2024-03-19T11:26:51.090 INFO:teuthology.orchestra.run.smithi038.stdout:19 Mar 11:26:51 ntpd[17100]: Listen normally on 4 lo [::1]:123 2024-03-19T11:26:51.090 INFO:teuthology.orchestra.run.smithi038.stdout:19 Mar 11:26:51 ntpd[17100]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:15fc%4]:123 2024-03-19T11:26:51.090 INFO:teuthology.orchestra.run.smithi038.stdout:19 Mar 11:26:51 ntpd[17100]: Listening on routing socket on fd #22 for interface updates 2024-03-19T11:26:51.117 INFO:teuthology.orchestra.run.smithi172.stdout:19 Mar 11:26:51 ntpd[17226]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-03-19T11:26:51.117 INFO:teuthology.orchestra.run.smithi172.stdout:19 Mar 11:26:51 ntpd[17226]: Command line: ntpd -gq 2024-03-19T11:26:51.117 INFO:teuthology.orchestra.run.smithi172.stdout:19 Mar 11:26:51 ntpd[17226]: ---------------------------------------------------- 2024-03-19T11:26:51.117 INFO:teuthology.orchestra.run.smithi172.stdout:19 Mar 11:26:51 ntpd[17226]: ntp-4 is maintained by Network Time Foundation, 2024-03-19T11:26:51.117 INFO:teuthology.orchestra.run.smithi172.stdout:19 Mar 11:26:51 ntpd[17226]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-03-19T11:26:51.117 INFO:teuthology.orchestra.run.smithi172.stdout:19 Mar 11:26:51 ntpd[17226]: corporation. Support and training for ntp-4 are 2024-03-19T11:26:51.117 INFO:teuthology.orchestra.run.smithi172.stdout:19 Mar 11:26:51 ntpd[17226]: available at https://www.nwtime.org/support 2024-03-19T11:26:51.117 INFO:teuthology.orchestra.run.smithi172.stdout:19 Mar 11:26:51 ntpd[17226]: ---------------------------------------------------- 2024-03-19T11:26:51.118 INFO:teuthology.orchestra.run.smithi172.stdout:19 Mar 11:26:51 ntpd[17226]: proto: precision = 0.045 usec (-24) 2024-03-19T11:26:51.118 INFO:teuthology.orchestra.run.smithi172.stdout:19 Mar 11:26:51 ntpd[17226]: basedate set to 2022-02-04 2024-03-19T11:26:51.118 INFO:teuthology.orchestra.run.smithi172.stdout:19 Mar 11:26:51 ntpd[17226]: gps base set to 2022-02-06 (week 2196) 2024-03-19T11:26:51.118 INFO:teuthology.orchestra.run.smithi172.stdout:19 Mar 11:26:51 ntpd[17226]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-19T11:26:51.118 INFO:teuthology.orchestra.run.smithi172.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-19T11:26:51.119 INFO:teuthology.orchestra.run.smithi172.stderr:restrict ::: KOD does nothing without LIMITED. 2024-03-19T11:26:51.120 INFO:teuthology.orchestra.run.smithi172.stdout:19 Mar 11:26:51 ntpd[17226]: restrict ::: KOD does nothing without LIMITED. 2024-03-19T11:26:51.120 INFO:teuthology.orchestra.run.smithi172.stdout:19 Mar 11:26:51 ntpd[17226]: Listen and drop on 0 v6wildcard [::]:123 2024-03-19T11:26:51.120 INFO:teuthology.orchestra.run.smithi172.stdout:19 Mar 11:26:51 ntpd[17226]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-03-19T11:26:51.120 INFO:teuthology.orchestra.run.smithi172.stdout:19 Mar 11:26:51 ntpd[17226]: Listen normally on 2 lo 127.0.0.1:123 2024-03-19T11:26:51.120 INFO:teuthology.orchestra.run.smithi172.stdout:19 Mar 11:26:51 ntpd[17226]: Listen normally on 3 enp3s0f1 172.21.15.172:123 2024-03-19T11:26:51.120 INFO:teuthology.orchestra.run.smithi172.stdout:19 Mar 11:26:51 ntpd[17226]: Listen normally on 4 lo [::1]:123 2024-03-19T11:26:51.120 INFO:teuthology.orchestra.run.smithi172.stdout:19 Mar 11:26:51 ntpd[17226]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7f91%5]:123 2024-03-19T11:26:51.120 INFO:teuthology.orchestra.run.smithi172.stdout:19 Mar 11:26:51 ntpd[17226]: Listening on routing socket on fd #22 for interface updates 2024-03-19T11:26:52.087 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:52 ntpd[17100]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T11:26:52.088 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:52 ntpd[17100]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T11:26:52.088 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:52 ntpd[17100]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T11:26:52.088 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:52 ntpd[17100]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T11:26:52.088 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:52 ntpd[17100]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T11:26:52.088 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:52 ntpd[17100]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T11:26:53.118 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:53 ntpd[17226]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T11:26:53.118 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:53 ntpd[17226]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T11:26:53.118 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:53 ntpd[17226]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T11:26:53.118 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:53 ntpd[17226]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T11:26:53.118 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:53 ntpd[17226]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T11:26:53.118 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:53 ntpd[17226]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T11:26:53.119 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:53 ntpd[17226]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T11:26:53.119 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:53 ntpd[17226]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T11:26:53.119 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:53 ntpd[17226]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T11:26:53.119 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:53 ntpd[17226]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T11:26:53.119 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:53 ntpd[17226]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T11:26:53.119 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:53 ntpd[17226]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T11:26:54.088 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:54 ntpd[17100]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T11:26:54.088 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:54 ntpd[17100]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T11:26:54.088 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:54 ntpd[17100]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T11:26:54.088 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:54 ntpd[17100]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T11:26:54.088 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:54 ntpd[17100]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T11:26:54.088 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:54 ntpd[17100]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T11:26:54.088 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:54 ntpd[17100]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T11:26:54.088 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:54 ntpd[17100]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T11:26:54.117 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:54 ntpd[17226]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T11:26:54.117 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:54 ntpd[17226]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T11:26:55.088 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:55 ntpd[17100]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T11:26:55.088 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:55 ntpd[17100]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T11:26:55.088 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:55 ntpd[17100]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T11:26:55.088 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:55 ntpd[17100]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T11:26:55.088 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:55 ntpd[17100]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T11:26:55.089 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:55 ntpd[17100]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T11:26:55.089 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:55 ntpd[17100]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T11:26:55.089 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:55 ntpd[17100]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T11:26:55.089 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:55 ntpd[17100]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T11:26:55.089 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:55 ntpd[17100]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T11:26:55.089 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:55 ntpd[17100]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T11:26:55.089 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:55 ntpd[17100]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T11:26:55.118 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:55 ntpd[17226]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T11:26:55.118 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:55 ntpd[17226]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T11:26:55.118 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:55 ntpd[17226]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T11:26:55.118 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:55 ntpd[17226]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T11:26:55.118 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:55 ntpd[17226]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T11:26:55.118 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:55 ntpd[17226]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T11:26:55.118 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:55 ntpd[17226]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T11:26:55.119 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:55 ntpd[17226]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T11:26:55.119 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:55 ntpd[17226]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T11:26:55.119 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:55 ntpd[17226]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T11:26:55.119 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:55 ntpd[17226]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T11:26:55.119 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:55 ntpd[17226]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T11:26:56.087 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:56 ntpd[17100]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T11:26:56.088 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:56 ntpd[17100]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T11:26:56.088 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:56 ntpd[17100]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T11:26:56.088 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:56 ntpd[17100]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T11:26:56.088 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:56 ntpd[17100]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T11:26:56.088 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:56 ntpd[17100]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T11:26:56.117 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:56 ntpd[17226]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T11:26:56.117 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:56 ntpd[17226]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T11:26:56.117 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:56 ntpd[17226]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T11:26:56.117 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:56 ntpd[17226]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T11:26:56.118 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:56 ntpd[17226]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T11:26:56.118 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:56 ntpd[17226]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T11:26:57.088 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:57 ntpd[17100]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T11:26:57.088 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:57 ntpd[17100]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T11:26:57.088 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:57 ntpd[17100]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T11:26:57.088 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:57 ntpd[17100]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T11:26:57.088 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:57 ntpd[17100]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T11:26:57.088 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:57 ntpd[17100]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T11:26:57.089 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:57 ntpd[17100]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T11:26:57.089 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:57 ntpd[17100]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T11:26:57.089 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:57 ntpd[17100]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T11:26:57.089 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:57 ntpd[17100]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T11:26:57.089 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:57 ntpd[17100]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T11:26:57.089 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:57 ntpd[17100]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T11:26:57.120 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:57 ntpd[17226]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T11:26:57.120 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:57 ntpd[17226]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T11:26:57.120 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:57 ntpd[17226]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T11:26:57.120 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:57 ntpd[17226]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T11:26:57.120 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:57 ntpd[17226]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T11:26:57.120 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:57 ntpd[17226]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T11:26:57.120 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:57 ntpd[17226]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T11:26:57.120 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:57 ntpd[17226]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T11:26:57.120 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:57 ntpd[17226]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T11:26:57.121 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:57 ntpd[17226]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T11:26:57.121 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:57 ntpd[17226]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T11:26:57.121 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:57 ntpd[17226]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T11:26:58.087 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:58 ntpd[17100]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T11:26:58.088 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:58 ntpd[17100]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T11:26:58.088 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:58 ntpd[17100]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T11:26:58.088 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:58 ntpd[17100]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T11:26:58.088 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:58 ntpd[17100]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T11:26:58.088 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:58 ntpd[17100]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T11:26:58.088 INFO:teuthology.orchestra.run.smithi038.stdout:19 Mar 11:26:58 ntpd[17100]: ntpd: time slew +0.000187 s 2024-03-19T11:26:58.088 INFO:teuthology.orchestra.run.smithi038.stdout:ntpd: time slew +0.000187s 2024-03-19T11:26:58.089 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:58 ntpd[17100]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-03-19T11:26:58.089 INFO:teuthology.orchestra.run.smithi038.stderr:19 Mar 11:26:58 ntpd[17100]: can't open /var/log/ntpstats/loopstats.20240319: Permission denied 2024-03-19T11:26:58.117 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:58 ntpd[17226]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T11:26:58.117 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:58 ntpd[17226]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T11:26:58.117 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:58 ntpd[17226]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T11:26:58.117 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:58 ntpd[17226]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T11:26:58.117 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:58 ntpd[17226]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T11:26:58.118 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:58 ntpd[17226]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T11:26:58.140 INFO:teuthology.orchestra.run.smithi038.stdout: remote refid st t when poll reach delay offset jitter 2024-03-19T11:26:58.140 INFO:teuthology.orchestra.run.smithi038.stdout:============================================================================== 2024-03-19T11:26:58.140 INFO:teuthology.orchestra.run.smithi038.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-19T11:26:58.140 INFO:teuthology.orchestra.run.smithi038.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-19T11:26:58.140 INFO:teuthology.orchestra.run.smithi038.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-19T11:26:58.140 INFO:teuthology.orchestra.run.smithi038.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-19T11:26:59.118 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:59 ntpd[17226]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T11:26:59.118 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:59 ntpd[17226]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T11:26:59.118 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:59 ntpd[17226]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T11:26:59.118 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:59 ntpd[17226]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T11:26:59.118 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:59 ntpd[17226]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-19T11:26:59.118 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:59 ntpd[17226]: can't open /var/log/ntpstats/rawstats.20240319: Permission denied 2024-03-19T11:26:59.118 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:59 ntpd[17226]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-19T11:26:59.118 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:59 ntpd[17226]: can't open /var/log/ntpstats/peerstats.20240319: Permission denied 2024-03-19T11:26:59.119 INFO:teuthology.orchestra.run.smithi172.stdout:19 Mar 11:26:59 ntpd[17226]: ntpd: time slew +0.000792 s 2024-03-19T11:26:59.119 INFO:teuthology.orchestra.run.smithi172.stdout:ntpd: time slew +0.000792s 2024-03-19T11:26:59.119 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:59 ntpd[17226]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-03-19T11:26:59.119 INFO:teuthology.orchestra.run.smithi172.stderr:19 Mar 11:26:59 ntpd[17226]: can't open /var/log/ntpstats/loopstats.20240319: Permission denied 2024-03-19T11:26:59.166 INFO:teuthology.orchestra.run.smithi172.stdout: remote refid st t when poll reach delay offset jitter 2024-03-19T11:26:59.166 INFO:teuthology.orchestra.run.smithi172.stdout:============================================================================== 2024-03-19T11:26:59.166 INFO:teuthology.orchestra.run.smithi172.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-19T11:26:59.166 INFO:teuthology.orchestra.run.smithi172.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-19T11:26:59.166 INFO:teuthology.orchestra.run.smithi172.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-19T11:26:59.166 INFO:teuthology.orchestra.run.smithi172.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-19T11:26:59.167 INFO:teuthology.run_tasks:Running task install... 2024-03-19T11:26:59.179 DEBUG:teuthology.task.install:project ceph 2024-03-19T11:26:59.179 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '9ef43722ad94ea688fa7f70daab6dd41fac6ee53'}} 2024-03-19T11:26:59.179 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '9ef43722ad94ea688fa7f70daab6dd41fac6ee53'} 2024-03-19T11:26:59.179 INFO:teuthology.task.install:Using flavor: default 2024-03-19T11:26:59.185 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-19T11:26:59.185 INFO:teuthology.task.install:extra packages: [] 2024-03-19T11:26:59.185 DEBUG:teuthology.orchestra.run.smithi038:> sudo apt-key list | grep Ceph 2024-03-19T11:26:59.186 DEBUG:teuthology.orchestra.run.smithi172:> sudo apt-key list | grep Ceph 2024-03-19T11:26:59.251 INFO:teuthology.orchestra.run.smithi038.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-03-19T11:26:59.266 INFO:teuthology.orchestra.run.smithi172.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-03-19T11:26:59.282 INFO:teuthology.orchestra.run.smithi038.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-03-19T11:26:59.283 INFO:teuthology.orchestra.run.smithi038.stdout:uid [ unknown] Ceph.com (release key) 2024-03-19T11:26:59.284 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-03-19T11:26:59.285 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=9ef43722ad94ea688fa7f70daab6dd41fac6ee53 2024-03-19T11:26:59.294 INFO:teuthology.orchestra.run.smithi172.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-03-19T11:26:59.294 INFO:teuthology.orchestra.run.smithi172.stdout:uid [ unknown] Ceph.com (release key) 2024-03-19T11:26:59.295 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-03-19T11:26:59.295 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=9ef43722ad94ea688fa7f70daab6dd41fac6ee53 2024-03-19T11:26:59.438 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default/ 2024-03-19T11:26:59.438 INFO:teuthology.task.install.deb:Package version is 19.0.0-2159-g9ef43722-1jammy 2024-03-19T11:26:59.548 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-19T11:26:59.548 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-03-19T11:26:59.562 DEBUG:teuthology.orchestra.run.smithi172:> sudo apt-get update 2024-03-19T11:26:59.618 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default/ 2024-03-19T11:26:59.618 INFO:teuthology.task.install.deb:Package version is 19.0.0-2159-g9ef43722-1jammy 2024-03-19T11:26:59.728 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-19T11:26:59.728 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-03-19T11:26:59.743 DEBUG:teuthology.orchestra.run.smithi038:> sudo apt-get update 2024-03-19T11:26:59.783 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-19T11:26:59.813 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-19T11:26:59.814 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-19T11:26:59.858 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-19T11:26:59.935 INFO:teuthology.orchestra.run.smithi172.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy InRelease 2024-03-19T11:26:59.964 INFO:teuthology.orchestra.run.smithi038.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-19T11:26:59.965 INFO:teuthology.orchestra.run.smithi038.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-19T11:26:59.991 INFO:teuthology.orchestra.run.smithi172.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-03-19T11:26:59.998 INFO:teuthology.orchestra.run.smithi038.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-19T11:27:00.052 INFO:teuthology.orchestra.run.smithi038.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-19T11:27:00.062 INFO:teuthology.orchestra.run.smithi172.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy Release.gpg 2024-03-19T11:27:00.122 INFO:teuthology.orchestra.run.smithi172.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-03-19T11:27:00.126 INFO:teuthology.orchestra.run.smithi038.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy InRelease 2024-03-19T11:27:00.181 INFO:teuthology.orchestra.run.smithi038.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-03-19T11:27:00.237 INFO:teuthology.orchestra.run.smithi038.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy Release.gpg 2024-03-19T11:27:00.239 INFO:teuthology.orchestra.run.smithi172.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,417 B] 2024-03-19T11:27:00.295 INFO:teuthology.orchestra.run.smithi038.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-03-19T11:27:00.331 INFO:teuthology.orchestra.run.smithi172.stdout:Fetched 29.0 kB in 1s (50.7 kB/s) 2024-03-19T11:27:00.356 INFO:teuthology.orchestra.run.smithi038.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,417 B] 2024-03-19T11:27:00.438 INFO:teuthology.orchestra.run.smithi038.stdout:Fetched 29.0 kB in 1s (57.3 kB/s) 2024-03-19T11:27:01.278 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2024-03-19T11:27:01.293 DEBUG:teuthology.orchestra.run.smithi172:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2159-g9ef43722-1jammy cephadm=19.0.0-2159-g9ef43722-1jammy ceph-mds=19.0.0-2159-g9ef43722-1jammy ceph-mgr=19.0.0-2159-g9ef43722-1jammy ceph-common=19.0.0-2159-g9ef43722-1jammy ceph-fuse=19.0.0-2159-g9ef43722-1jammy ceph-test=19.0.0-2159-g9ef43722-1jammy ceph-volume=19.0.0-2159-g9ef43722-1jammy radosgw=19.0.0-2159-g9ef43722-1jammy python3-rados=19.0.0-2159-g9ef43722-1jammy python3-rgw=19.0.0-2159-g9ef43722-1jammy python3-cephfs=19.0.0-2159-g9ef43722-1jammy python3-rbd=19.0.0-2159-g9ef43722-1jammy libcephfs2=19.0.0-2159-g9ef43722-1jammy libcephfs-dev=19.0.0-2159-g9ef43722-1jammy librados2=19.0.0-2159-g9ef43722-1jammy librbd1=19.0.0-2159-g9ef43722-1jammy rbd-fuse=19.0.0-2159-g9ef43722-1jammy 2024-03-19T11:27:01.344 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2024-03-19T11:27:01.429 INFO:teuthology.orchestra.run.smithi038.stdout:Reading package lists... 2024-03-19T11:27:01.447 DEBUG:teuthology.orchestra.run.smithi038:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2159-g9ef43722-1jammy cephadm=19.0.0-2159-g9ef43722-1jammy ceph-mds=19.0.0-2159-g9ef43722-1jammy ceph-mgr=19.0.0-2159-g9ef43722-1jammy ceph-common=19.0.0-2159-g9ef43722-1jammy ceph-fuse=19.0.0-2159-g9ef43722-1jammy ceph-test=19.0.0-2159-g9ef43722-1jammy ceph-volume=19.0.0-2159-g9ef43722-1jammy radosgw=19.0.0-2159-g9ef43722-1jammy python3-rados=19.0.0-2159-g9ef43722-1jammy python3-rgw=19.0.0-2159-g9ef43722-1jammy python3-cephfs=19.0.0-2159-g9ef43722-1jammy python3-rbd=19.0.0-2159-g9ef43722-1jammy libcephfs2=19.0.0-2159-g9ef43722-1jammy libcephfs-dev=19.0.0-2159-g9ef43722-1jammy librados2=19.0.0-2159-g9ef43722-1jammy librbd1=19.0.0-2159-g9ef43722-1jammy rbd-fuse=19.0.0-2159-g9ef43722-1jammy 2024-03-19T11:27:01.506 INFO:teuthology.orchestra.run.smithi038.stdout:Reading package lists... 2024-03-19T11:27:01.518 INFO:teuthology.orchestra.run.smithi172.stdout:Building dependency tree... 2024-03-19T11:27:01.518 INFO:teuthology.orchestra.run.smithi172.stdout:Reading state information... 2024-03-19T11:27:01.679 INFO:teuthology.orchestra.run.smithi172.stdout:The following packages were automatically installed and are no longer required: 2024-03-19T11:27:01.679 INFO:teuthology.orchestra.run.smithi172.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-19T11:27:01.679 INFO:teuthology.orchestra.run.smithi172.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-19T11:27:01.679 INFO:teuthology.orchestra.run.smithi172.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-19T11:27:01.679 INFO:teuthology.orchestra.run.smithi172.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-19T11:27:01.682 INFO:teuthology.orchestra.run.smithi172.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-19T11:27:01.682 INFO:teuthology.orchestra.run.smithi172.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-19T11:27:01.683 INFO:teuthology.orchestra.run.smithi172.stdout:The following additional packages will be installed: 2024-03-19T11:27:01.683 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-03-19T11:27:01.683 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-03-19T11:27:01.683 INFO:teuthology.orchestra.run.smithi172.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-03-19T11:27:01.683 INFO:teuthology.orchestra.run.smithi172.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-03-19T11:27:01.683 INFO:teuthology.orchestra.run.smithi172.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-03-19T11:27:01.683 INFO:teuthology.orchestra.run.smithi172.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-03-19T11:27:01.683 INFO:teuthology.orchestra.run.smithi172.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-03-19T11:27:01.683 INFO:teuthology.orchestra.run.smithi172.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-03-19T11:27:01.683 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-03-19T11:27:01.683 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-03-19T11:27:01.683 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-19T11:27:01.683 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-19T11:27:01.683 INFO:teuthology.orchestra.run.smithi172.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-19T11:27:01.684 INFO:teuthology.orchestra.run.smithi172.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-19T11:27:01.684 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-19T11:27:01.684 INFO:teuthology.orchestra.run.smithi172.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-19T11:27:01.684 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-03-19T11:27:01.684 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-03-19T11:27:01.684 INFO:teuthology.orchestra.run.smithi172.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-03-19T11:27:01.684 INFO:teuthology.orchestra.run.smithi172.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-03-19T11:27:01.684 INFO:teuthology.orchestra.run.smithi172.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-03-19T11:27:01.684 INFO:teuthology.orchestra.run.smithi172.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-03-19T11:27:01.684 INFO:teuthology.orchestra.run.smithi172.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-03-19T11:27:01.684 INFO:teuthology.orchestra.run.smithi172.stdout: xmlstarlet zip 2024-03-19T11:27:01.684 INFO:teuthology.orchestra.run.smithi172.stdout:Suggested packages: 2024-03-19T11:27:01.684 INFO:teuthology.orchestra.run.smithi172.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-03-19T11:27:01.684 INFO:teuthology.orchestra.run.smithi172.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-03-19T11:27:01.685 INFO:teuthology.orchestra.run.smithi172.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-03-19T11:27:01.685 INFO:teuthology.orchestra.run.smithi172.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-03-19T11:27:01.685 INFO:teuthology.orchestra.run.smithi172.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-03-19T11:27:01.685 INFO:teuthology.orchestra.run.smithi172.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-03-19T11:27:01.685 INFO:teuthology.orchestra.run.smithi172.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-03-19T11:27:01.685 INFO:teuthology.orchestra.run.smithi172.stdout:Recommended packages: 2024-03-19T11:27:01.685 INFO:teuthology.orchestra.run.smithi172.stdout: btrfs-tools 2024-03-19T11:27:01.703 INFO:teuthology.orchestra.run.smithi038.stdout:Building dependency tree... 2024-03-19T11:27:01.704 INFO:teuthology.orchestra.run.smithi038.stdout:Reading state information... 2024-03-19T11:27:01.784 INFO:teuthology.orchestra.run.smithi172.stdout:The following NEW packages will be installed: 2024-03-19T11:27:01.785 INFO:teuthology.orchestra.run.smithi172.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-03-19T11:27:01.785 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-03-19T11:27:01.785 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-03-19T11:27:01.785 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-03-19T11:27:01.785 INFO:teuthology.orchestra.run.smithi172.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-03-19T11:27:01.785 INFO:teuthology.orchestra.run.smithi172.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-03-19T11:27:01.785 INFO:teuthology.orchestra.run.smithi172.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-03-19T11:27:01.785 INFO:teuthology.orchestra.run.smithi172.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-03-19T11:27:01.785 INFO:teuthology.orchestra.run.smithi172.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-03-19T11:27:01.785 INFO:teuthology.orchestra.run.smithi172.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-03-19T11:27:01.786 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-03-19T11:27:01.786 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-03-19T11:27:01.786 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-19T11:27:01.786 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-19T11:27:01.786 INFO:teuthology.orchestra.run.smithi172.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-19T11:27:01.786 INFO:teuthology.orchestra.run.smithi172.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-19T11:27:01.786 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-19T11:27:01.786 INFO:teuthology.orchestra.run.smithi172.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-19T11:27:01.786 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-03-19T11:27:01.786 INFO:teuthology.orchestra.run.smithi172.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-03-19T11:27:01.786 INFO:teuthology.orchestra.run.smithi172.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-03-19T11:27:01.786 INFO:teuthology.orchestra.run.smithi172.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-03-19T11:27:01.787 INFO:teuthology.orchestra.run.smithi172.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-03-19T11:27:01.787 INFO:teuthology.orchestra.run.smithi172.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-03-19T11:27:01.787 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-03-19T11:27:01.787 INFO:teuthology.orchestra.run.smithi172.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-03-19T11:27:01.889 INFO:teuthology.orchestra.run.smithi038.stdout:The following packages were automatically installed and are no longer required: 2024-03-19T11:27:01.889 INFO:teuthology.orchestra.run.smithi038.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-19T11:27:01.889 INFO:teuthology.orchestra.run.smithi038.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-19T11:27:01.889 INFO:teuthology.orchestra.run.smithi038.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-19T11:27:01.890 INFO:teuthology.orchestra.run.smithi038.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-19T11:27:01.890 INFO:teuthology.orchestra.run.smithi038.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-19T11:27:01.890 INFO:teuthology.orchestra.run.smithi038.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-19T11:27:01.892 INFO:teuthology.orchestra.run.smithi038.stdout:The following additional packages will be installed: 2024-03-19T11:27:01.892 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-03-19T11:27:01.892 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-03-19T11:27:01.892 INFO:teuthology.orchestra.run.smithi038.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-03-19T11:27:01.892 INFO:teuthology.orchestra.run.smithi038.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-03-19T11:27:01.892 INFO:teuthology.orchestra.run.smithi038.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-03-19T11:27:01.892 INFO:teuthology.orchestra.run.smithi038.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-03-19T11:27:01.892 INFO:teuthology.orchestra.run.smithi038.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-03-19T11:27:01.892 INFO:teuthology.orchestra.run.smithi038.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-03-19T11:27:01.892 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-03-19T11:27:01.893 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-03-19T11:27:01.893 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-19T11:27:01.893 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-19T11:27:01.893 INFO:teuthology.orchestra.run.smithi038.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-19T11:27:01.893 INFO:teuthology.orchestra.run.smithi038.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-19T11:27:01.893 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-19T11:27:01.893 INFO:teuthology.orchestra.run.smithi038.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-19T11:27:01.893 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-03-19T11:27:01.893 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-03-19T11:27:01.893 INFO:teuthology.orchestra.run.smithi038.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-03-19T11:27:01.893 INFO:teuthology.orchestra.run.smithi038.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-03-19T11:27:01.893 INFO:teuthology.orchestra.run.smithi038.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-03-19T11:27:01.893 INFO:teuthology.orchestra.run.smithi038.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-03-19T11:27:01.894 INFO:teuthology.orchestra.run.smithi038.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-03-19T11:27:01.894 INFO:teuthology.orchestra.run.smithi038.stdout: xmlstarlet zip 2024-03-19T11:27:01.894 INFO:teuthology.orchestra.run.smithi038.stdout:Suggested packages: 2024-03-19T11:27:01.894 INFO:teuthology.orchestra.run.smithi038.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-03-19T11:27:01.894 INFO:teuthology.orchestra.run.smithi038.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-03-19T11:27:01.894 INFO:teuthology.orchestra.run.smithi038.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-03-19T11:27:01.894 INFO:teuthology.orchestra.run.smithi038.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-03-19T11:27:01.894 INFO:teuthology.orchestra.run.smithi038.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-03-19T11:27:01.894 INFO:teuthology.orchestra.run.smithi038.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-03-19T11:27:01.895 INFO:teuthology.orchestra.run.smithi038.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-03-19T11:27:01.895 INFO:teuthology.orchestra.run.smithi038.stdout:Recommended packages: 2024-03-19T11:27:01.895 INFO:teuthology.orchestra.run.smithi038.stdout: btrfs-tools 2024-03-19T11:27:01.971 INFO:teuthology.orchestra.run.smithi172.stdout:0 upgraded, 118 newly installed, 0 to remove and 285 not upgraded. 2024-03-19T11:27:01.971 INFO:teuthology.orchestra.run.smithi172.stdout:Need to get 178 MB of archives. 2024-03-19T11:27:01.972 INFO:teuthology.orchestra.run.smithi172.stdout:After this operation, 736 MB of additional disk space will be used. 2024-03-19T11:27:01.972 INFO:teuthology.orchestra.run.smithi172.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-03-19T11:27:02.000 INFO:teuthology.orchestra.run.smithi038.stdout:The following NEW packages will be installed: 2024-03-19T11:27:02.001 INFO:teuthology.orchestra.run.smithi038.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-03-19T11:27:02.001 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-03-19T11:27:02.001 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-03-19T11:27:02.001 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-03-19T11:27:02.001 INFO:teuthology.orchestra.run.smithi038.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-03-19T11:27:02.001 INFO:teuthology.orchestra.run.smithi038.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-03-19T11:27:02.002 INFO:teuthology.orchestra.run.smithi038.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-03-19T11:27:02.002 INFO:teuthology.orchestra.run.smithi038.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-03-19T11:27:02.002 INFO:teuthology.orchestra.run.smithi038.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-03-19T11:27:02.003 INFO:teuthology.orchestra.run.smithi038.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-03-19T11:27:02.003 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-03-19T11:27:02.003 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-03-19T11:27:02.003 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-19T11:27:02.003 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-19T11:27:02.003 INFO:teuthology.orchestra.run.smithi038.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-19T11:27:02.003 INFO:teuthology.orchestra.run.smithi038.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-19T11:27:02.003 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-19T11:27:02.003 INFO:teuthology.orchestra.run.smithi038.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-19T11:27:02.003 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-03-19T11:27:02.003 INFO:teuthology.orchestra.run.smithi038.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-03-19T11:27:02.003 INFO:teuthology.orchestra.run.smithi038.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-03-19T11:27:02.003 INFO:teuthology.orchestra.run.smithi038.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-03-19T11:27:02.003 INFO:teuthology.orchestra.run.smithi038.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-03-19T11:27:02.003 INFO:teuthology.orchestra.run.smithi038.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-03-19T11:27:02.004 INFO:teuthology.orchestra.run.smithi038.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-03-19T11:27:02.004 INFO:teuthology.orchestra.run.smithi038.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-03-19T11:27:02.010 INFO:teuthology.orchestra.run.smithi172.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2159-g9ef43722-1jammy [3,556 kB] 2024-03-19T11:27:02.217 INFO:teuthology.orchestra.run.smithi038.stdout:0 upgraded, 118 newly installed, 0 to remove and 285 not upgraded. 2024-03-19T11:27:02.217 INFO:teuthology.orchestra.run.smithi038.stdout:Need to get 178 MB of archives. 2024-03-19T11:27:02.217 INFO:teuthology.orchestra.run.smithi038.stdout:After this operation, 736 MB of additional disk space will be used. 2024-03-19T11:27:02.217 INFO:teuthology.orchestra.run.smithi038.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-03-19T11:27:02.255 INFO:teuthology.orchestra.run.smithi038.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2159-g9ef43722-1jammy [3,556 kB] 2024-03-19T11:27:02.394 INFO:teuthology.orchestra.run.smithi172.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-03-19T11:27:02.406 INFO:teuthology.orchestra.run.smithi172.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-03-19T11:27:02.488 INFO:teuthology.orchestra.run.smithi172.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-03-19T11:27:02.576 INFO:teuthology.orchestra.run.smithi172.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2159-g9ef43722-1jammy [3,223 kB] 2024-03-19T11:27:02.629 INFO:teuthology.orchestra.run.smithi038.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2159-g9ef43722-1jammy [3,223 kB] 2024-03-19T11:27:02.637 INFO:teuthology.orchestra.run.smithi038.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-03-19T11:27:02.649 INFO:teuthology.orchestra.run.smithi038.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-03-19T11:27:02.731 INFO:teuthology.orchestra.run.smithi038.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-03-19T11:27:02.738 INFO:teuthology.orchestra.run.smithi172.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-03-19T11:27:02.749 INFO:teuthology.orchestra.run.smithi172.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-03-19T11:27:02.786 INFO:teuthology.orchestra.run.smithi172.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-03-19T11:27:02.794 INFO:teuthology.orchestra.run.smithi172.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-03-19T11:27:02.795 INFO:teuthology.orchestra.run.smithi172.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-03-19T11:27:02.796 INFO:teuthology.orchestra.run.smithi172.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-03-19T11:27:02.810 INFO:teuthology.orchestra.run.smithi038.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2159-g9ef43722-1jammy [712 kB] 2024-03-19T11:27:02.816 INFO:teuthology.orchestra.run.smithi172.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-03-19T11:27:02.840 INFO:teuthology.orchestra.run.smithi038.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2159-g9ef43722-1jammy [356 kB] 2024-03-19T11:27:02.850 INFO:teuthology.orchestra.run.smithi172.stdout:Get:14 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2159-g9ef43722-1jammy [712 kB] 2024-03-19T11:27:02.855 INFO:teuthology.orchestra.run.smithi038.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2159-g9ef43722-1jammy [32.8 kB] 2024-03-19T11:27:02.856 INFO:teuthology.orchestra.run.smithi038.stdout:Get:10 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2159-g9ef43722-1jammy [180 kB] 2024-03-19T11:27:02.861 INFO:teuthology.orchestra.run.smithi038.stdout:Get:11 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2159-g9ef43722-1jammy [64.6 kB] 2024-03-19T11:27:02.864 INFO:teuthology.orchestra.run.smithi038.stdout:Get:12 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2159-g9ef43722-1jammy [330 kB] 2024-03-19T11:27:02.878 INFO:teuthology.orchestra.run.smithi038.stdout:Get:13 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2159-g9ef43722-1jammy [6,659 kB] 2024-03-19T11:27:02.889 INFO:teuthology.orchestra.run.smithi172.stdout:Get:15 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2159-g9ef43722-1jammy [356 kB] 2024-03-19T11:27:02.933 INFO:teuthology.orchestra.run.smithi172.stdout:Get:16 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2159-g9ef43722-1jammy [32.8 kB] 2024-03-19T11:27:02.934 INFO:teuthology.orchestra.run.smithi172.stdout:Get:17 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2159-g9ef43722-1jammy [180 kB] 2024-03-19T11:27:02.940 INFO:teuthology.orchestra.run.smithi172.stdout:Get:18 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2159-g9ef43722-1jammy [64.6 kB] 2024-03-19T11:27:02.942 INFO:teuthology.orchestra.run.smithi172.stdout:Get:19 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2159-g9ef43722-1jammy [330 kB] 2024-03-19T11:27:02.960 INFO:teuthology.orchestra.run.smithi038.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-03-19T11:27:02.971 INFO:teuthology.orchestra.run.smithi038.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-03-19T11:27:02.985 INFO:teuthology.orchestra.run.smithi172.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-03-19T11:27:02.986 INFO:teuthology.orchestra.run.smithi172.stdout:Get:21 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2159-g9ef43722-1jammy [6,659 kB] 2024-03-19T11:27:03.006 INFO:teuthology.orchestra.run.smithi038.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-03-19T11:27:03.013 INFO:teuthology.orchestra.run.smithi038.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-03-19T11:27:03.014 INFO:teuthology.orchestra.run.smithi038.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-03-19T11:27:03.016 INFO:teuthology.orchestra.run.smithi038.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-03-19T11:27:03.033 INFO:teuthology.orchestra.run.smithi038.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-03-19T11:27:03.204 INFO:teuthology.orchestra.run.smithi038.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-03-19T11:27:03.258 INFO:teuthology.orchestra.run.smithi038.stdout:Get:22 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2159-g9ef43722-1jammy [112 kB] 2024-03-19T11:27:03.261 INFO:teuthology.orchestra.run.smithi038.stdout:Get:23 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2159-g9ef43722-1jammy [466 kB] 2024-03-19T11:27:03.270 INFO:teuthology.orchestra.run.smithi038.stdout:Get:24 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2159-g9ef43722-1jammy [25.0 MB] 2024-03-19T11:27:03.400 INFO:teuthology.orchestra.run.smithi172.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-03-19T11:27:03.434 INFO:teuthology.orchestra.run.smithi172.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-03-19T11:27:03.437 INFO:teuthology.orchestra.run.smithi172.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-03-19T11:27:03.516 INFO:teuthology.orchestra.run.smithi172.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-03-19T11:27:03.542 INFO:teuthology.orchestra.run.smithi172.stdout:Get:26 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2159-g9ef43722-1jammy [112 kB] 2024-03-19T11:27:03.545 INFO:teuthology.orchestra.run.smithi172.stdout:Get:27 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2159-g9ef43722-1jammy [466 kB] 2024-03-19T11:27:03.567 INFO:teuthology.orchestra.run.smithi172.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-03-19T11:27:03.590 INFO:teuthology.orchestra.run.smithi172.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-03-19T11:27:03.596 INFO:teuthology.orchestra.run.smithi172.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-03-19T11:27:03.597 INFO:teuthology.orchestra.run.smithi172.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-03-19T11:27:03.609 INFO:teuthology.orchestra.run.smithi172.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-03-19T11:27:03.610 INFO:teuthology.orchestra.run.smithi172.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-03-19T11:27:03.617 INFO:teuthology.orchestra.run.smithi038.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-03-19T11:27:03.618 INFO:teuthology.orchestra.run.smithi172.stdout:Get:34 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2159-g9ef43722-1jammy [25.0 MB] 2024-03-19T11:27:03.653 INFO:teuthology.orchestra.run.smithi038.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-03-19T11:27:03.655 INFO:teuthology.orchestra.run.smithi038.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-03-19T11:27:03.733 INFO:teuthology.orchestra.run.smithi038.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-03-19T11:27:03.775 INFO:teuthology.orchestra.run.smithi172.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-03-19T11:27:03.785 INFO:teuthology.orchestra.run.smithi038.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-03-19T11:27:03.807 INFO:teuthology.orchestra.run.smithi038.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-03-19T11:27:03.813 INFO:teuthology.orchestra.run.smithi038.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-03-19T11:27:03.814 INFO:teuthology.orchestra.run.smithi038.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-03-19T11:27:03.826 INFO:teuthology.orchestra.run.smithi038.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-03-19T11:27:03.826 INFO:teuthology.orchestra.run.smithi038.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-03-19T11:27:03.858 INFO:teuthology.orchestra.run.smithi172.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-03-19T11:27:03.886 INFO:teuthology.orchestra.run.smithi172.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-03-19T11:27:03.903 INFO:teuthology.orchestra.run.smithi172.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-03-19T11:27:03.948 INFO:teuthology.orchestra.run.smithi172.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-03-19T11:27:03.992 INFO:teuthology.orchestra.run.smithi038.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-03-19T11:27:04.075 INFO:teuthology.orchestra.run.smithi038.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-03-19T11:27:04.104 INFO:teuthology.orchestra.run.smithi038.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-03-19T11:27:04.121 INFO:teuthology.orchestra.run.smithi038.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-03-19T11:27:04.162 INFO:teuthology.orchestra.run.smithi172.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-03-19T11:27:04.164 INFO:teuthology.orchestra.run.smithi038.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-03-19T11:27:04.177 INFO:teuthology.orchestra.run.smithi172.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-03-19T11:27:04.184 INFO:teuthology.orchestra.run.smithi172.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-03-19T11:27:04.189 INFO:teuthology.orchestra.run.smithi172.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-03-19T11:27:04.210 INFO:teuthology.orchestra.run.smithi172.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-03-19T11:27:04.214 INFO:teuthology.orchestra.run.smithi172.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-03-19T11:27:04.375 INFO:teuthology.orchestra.run.smithi038.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-03-19T11:27:04.380 INFO:teuthology.orchestra.run.smithi172.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-03-19T11:27:04.390 INFO:teuthology.orchestra.run.smithi038.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-03-19T11:27:04.398 INFO:teuthology.orchestra.run.smithi038.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-03-19T11:27:04.403 INFO:teuthology.orchestra.run.smithi038.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-03-19T11:27:04.424 INFO:teuthology.orchestra.run.smithi038.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-03-19T11:27:04.427 INFO:teuthology.orchestra.run.smithi038.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-03-19T11:27:04.594 INFO:teuthology.orchestra.run.smithi038.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-03-19T11:27:04.714 INFO:teuthology.orchestra.run.smithi172.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-03-19T11:27:04.746 INFO:teuthology.orchestra.run.smithi172.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-03-19T11:27:04.756 INFO:teuthology.orchestra.run.smithi172.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-03-19T11:27:04.757 INFO:teuthology.orchestra.run.smithi038.stdout:Get:47 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2159-g9ef43722-1jammy [5,084 kB] 2024-03-19T11:27:04.925 INFO:teuthology.orchestra.run.smithi038.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-03-19T11:27:04.951 INFO:teuthology.orchestra.run.smithi038.stdout:Get:49 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2159-g9ef43722-1jammy [244 kB] 2024-03-19T11:27:04.959 INFO:teuthology.orchestra.run.smithi172.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-03-19T11:27:04.960 INFO:teuthology.orchestra.run.smithi172.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-03-19T11:27:04.964 INFO:teuthology.orchestra.run.smithi038.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-03-19T11:27:04.967 INFO:teuthology.orchestra.run.smithi172.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-03-19T11:27:04.971 INFO:teuthology.orchestra.run.smithi038.stdout:Get:51 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2159-g9ef43722-1jammy [124 kB] 2024-03-19T11:27:04.973 INFO:teuthology.orchestra.run.smithi172.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-03-19T11:27:04.976 INFO:teuthology.orchestra.run.smithi038.stdout:Get:52 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2159-g9ef43722-1jammy [1,525 kB] 2024-03-19T11:27:04.981 INFO:teuthology.orchestra.run.smithi038.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-03-19T11:27:04.988 INFO:teuthology.orchestra.run.smithi172.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-03-19T11:27:05.009 INFO:teuthology.orchestra.run.smithi172.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-03-19T11:27:05.015 INFO:teuthology.orchestra.run.smithi172.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-03-19T11:27:05.075 INFO:teuthology.orchestra.run.smithi038.stdout:Get:54 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2159-g9ef43722-1jammy [6,253 kB] 2024-03-19T11:27:05.191 INFO:teuthology.orchestra.run.smithi172.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-03-19T11:27:05.193 INFO:teuthology.orchestra.run.smithi038.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-03-19T11:27:05.194 INFO:teuthology.orchestra.run.smithi038.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-03-19T11:27:05.202 INFO:teuthology.orchestra.run.smithi038.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-03-19T11:27:05.205 INFO:teuthology.orchestra.run.smithi038.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-03-19T11:27:05.220 INFO:teuthology.orchestra.run.smithi038.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-03-19T11:27:05.241 INFO:teuthology.orchestra.run.smithi038.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-03-19T11:27:05.247 INFO:teuthology.orchestra.run.smithi038.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-03-19T11:27:05.401 INFO:teuthology.orchestra.run.smithi038.stdout:Get:62 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2159-g9ef43722-1jammy [22.8 MB] 2024-03-19T11:27:05.424 INFO:teuthology.orchestra.run.smithi038.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-03-19T11:27:05.445 INFO:teuthology.orchestra.run.smithi172.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-03-19T11:27:05.689 INFO:teuthology.orchestra.run.smithi172.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-03-19T11:27:05.690 INFO:teuthology.orchestra.run.smithi038.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-03-19T11:27:05.690 INFO:teuthology.orchestra.run.smithi172.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-03-19T11:27:05.733 INFO:teuthology.orchestra.run.smithi172.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-03-19T11:27:05.733 INFO:teuthology.orchestra.run.smithi172.stdout:Get:62 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2159-g9ef43722-1jammy [5,084 kB] 2024-03-19T11:27:05.737 INFO:teuthology.orchestra.run.smithi172.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-03-19T11:27:05.868 INFO:teuthology.orchestra.run.smithi038.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-03-19T11:27:05.882 INFO:teuthology.orchestra.run.smithi038.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-03-19T11:27:05.901 INFO:teuthology.orchestra.run.smithi038.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-03-19T11:27:05.958 INFO:teuthology.orchestra.run.smithi172.stdout:Get:64 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2159-g9ef43722-1jammy [244 kB] 2024-03-19T11:27:05.966 INFO:teuthology.orchestra.run.smithi038.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-03-19T11:27:05.969 INFO:teuthology.orchestra.run.smithi172.stdout:Get:65 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2159-g9ef43722-1jammy [124 kB] 2024-03-19T11:27:05.973 INFO:teuthology.orchestra.run.smithi172.stdout:Get:66 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2159-g9ef43722-1jammy [1,525 kB] 2024-03-19T11:27:06.051 INFO:teuthology.orchestra.run.smithi172.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-03-19T11:27:06.053 INFO:teuthology.orchestra.run.smithi172.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-03-19T11:27:06.055 INFO:teuthology.orchestra.run.smithi172.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-03-19T11:27:06.056 INFO:teuthology.orchestra.run.smithi172.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-03-19T11:27:06.058 INFO:teuthology.orchestra.run.smithi172.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-03-19T11:27:06.070 INFO:teuthology.orchestra.run.smithi172.stdout:Get:72 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2159-g9ef43722-1jammy [6,253 kB] 2024-03-19T11:27:06.262 INFO:teuthology.orchestra.run.smithi038.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-03-19T11:27:06.264 INFO:teuthology.orchestra.run.smithi038.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-03-19T11:27:06.266 INFO:teuthology.orchestra.run.smithi038.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-03-19T11:27:06.266 INFO:teuthology.orchestra.run.smithi038.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-03-19T11:27:06.268 INFO:teuthology.orchestra.run.smithi038.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-03-19T11:27:06.285 INFO:teuthology.orchestra.run.smithi172.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-03-19T11:27:06.490 INFO:teuthology.orchestra.run.smithi172.stdout:Get:74 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2159-g9ef43722-1jammy [22.8 MB] 2024-03-19T11:27:06.498 INFO:teuthology.orchestra.run.smithi038.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-03-19T11:27:06.722 INFO:teuthology.orchestra.run.smithi172.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-03-19T11:27:06.728 INFO:teuthology.orchestra.run.smithi172.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-03-19T11:27:06.930 INFO:teuthology.orchestra.run.smithi038.stdout:Get:75 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2159-g9ef43722-1jammy [14.0 kB] 2024-03-19T11:27:06.931 INFO:teuthology.orchestra.run.smithi038.stdout:Get:76 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2159-g9ef43722-1jammy [870 kB] 2024-03-19T11:27:06.948 INFO:teuthology.orchestra.run.smithi038.stdout:Get:77 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2159-g9ef43722-1jammy [2,385 kB] 2024-03-19T11:27:06.967 INFO:teuthology.orchestra.run.smithi038.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-03-19T11:27:06.973 INFO:teuthology.orchestra.run.smithi038.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-03-19T11:27:07.019 INFO:teuthology.orchestra.run.smithi172.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-03-19T11:27:07.019 INFO:teuthology.orchestra.run.smithi172.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-03-19T11:27:07.021 INFO:teuthology.orchestra.run.smithi172.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-03-19T11:27:07.023 INFO:teuthology.orchestra.run.smithi172.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-03-19T11:27:07.024 INFO:teuthology.orchestra.run.smithi172.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-03-19T11:27:07.026 INFO:teuthology.orchestra.run.smithi172.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-03-19T11:27:07.027 INFO:teuthology.orchestra.run.smithi172.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-03-19T11:27:07.029 INFO:teuthology.orchestra.run.smithi172.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-03-19T11:27:07.061 INFO:teuthology.orchestra.run.smithi038.stdout:Get:80 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2159-g9ef43722-1jammy [777 kB] 2024-03-19T11:27:07.102 INFO:teuthology.orchestra.run.smithi038.stdout:Get:81 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2159-g9ef43722-1jammy [144 kB] 2024-03-19T11:27:07.106 INFO:teuthology.orchestra.run.smithi038.stdout:Get:82 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2159-g9ef43722-1jammy [1,936 kB] 2024-03-19T11:27:07.211 INFO:teuthology.orchestra.run.smithi172.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-03-19T11:27:07.236 INFO:teuthology.orchestra.run.smithi038.stdout:Get:83 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2159-g9ef43722-1jammy [8,625 kB] 2024-03-19T11:27:07.269 INFO:teuthology.orchestra.run.smithi038.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-03-19T11:27:07.269 INFO:teuthology.orchestra.run.smithi038.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-03-19T11:27:07.271 INFO:teuthology.orchestra.run.smithi038.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-03-19T11:27:07.273 INFO:teuthology.orchestra.run.smithi038.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-03-19T11:27:07.274 INFO:teuthology.orchestra.run.smithi038.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-03-19T11:27:07.276 INFO:teuthology.orchestra.run.smithi038.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-03-19T11:27:07.277 INFO:teuthology.orchestra.run.smithi038.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-03-19T11:27:07.279 INFO:teuthology.orchestra.run.smithi038.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-03-19T11:27:07.460 INFO:teuthology.orchestra.run.smithi038.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-03-19T11:27:07.621 INFO:teuthology.orchestra.run.smithi172.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-03-19T11:27:07.668 INFO:teuthology.orchestra.run.smithi172.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-03-19T11:27:07.702 INFO:teuthology.orchestra.run.smithi172.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-03-19T11:27:07.800 INFO:teuthology.orchestra.run.smithi172.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-03-19T11:27:07.814 INFO:teuthology.orchestra.run.smithi038.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2159-g9ef43722-1jammy [14.2 kB] 2024-03-19T11:27:07.815 INFO:teuthology.orchestra.run.smithi038.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2159-g9ef43722-1jammy [50.2 MB] 2024-03-19T11:27:07.840 INFO:teuthology.orchestra.run.smithi172.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-03-19T11:27:07.852 INFO:teuthology.orchestra.run.smithi172.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-03-19T11:27:07.857 INFO:teuthology.orchestra.run.smithi172.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-03-19T11:27:07.871 INFO:teuthology.orchestra.run.smithi038.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-03-19T11:27:07.909 INFO:teuthology.orchestra.run.smithi172.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-03-19T11:27:07.912 INFO:teuthology.orchestra.run.smithi172.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-03-19T11:27:07.919 INFO:teuthology.orchestra.run.smithi038.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-03-19T11:27:07.933 INFO:teuthology.orchestra.run.smithi172.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-03-19T11:27:07.952 INFO:teuthology.orchestra.run.smithi038.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-03-19T11:27:08.031 INFO:teuthology.orchestra.run.smithi172.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2159-g9ef43722-1jammy [14.0 kB] 2024-03-19T11:27:08.032 INFO:teuthology.orchestra.run.smithi172.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2159-g9ef43722-1jammy [870 kB] 2024-03-19T11:27:08.048 INFO:teuthology.orchestra.run.smithi038.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-03-19T11:27:08.049 INFO:teuthology.orchestra.run.smithi172.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2159-g9ef43722-1jammy [2,385 kB] 2024-03-19T11:27:08.086 INFO:teuthology.orchestra.run.smithi038.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-03-19T11:27:08.097 INFO:teuthology.orchestra.run.smithi172.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-03-19T11:27:08.097 INFO:teuthology.orchestra.run.smithi038.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-03-19T11:27:08.103 INFO:teuthology.orchestra.run.smithi038.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-03-19T11:27:08.151 INFO:teuthology.orchestra.run.smithi038.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-03-19T11:27:08.154 INFO:teuthology.orchestra.run.smithi038.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-03-19T11:27:08.175 INFO:teuthology.orchestra.run.smithi038.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-03-19T11:27:08.194 INFO:teuthology.orchestra.run.smithi172.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2159-g9ef43722-1jammy [777 kB] 2024-03-19T11:27:08.250 INFO:teuthology.orchestra.run.smithi172.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2159-g9ef43722-1jammy [144 kB] 2024-03-19T11:27:08.253 INFO:teuthology.orchestra.run.smithi172.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2159-g9ef43722-1jammy [1,936 kB] 2024-03-19T11:27:08.340 INFO:teuthology.orchestra.run.smithi038.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-03-19T11:27:08.362 INFO:teuthology.orchestra.run.smithi172.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-03-19T11:27:08.379 INFO:teuthology.orchestra.run.smithi172.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-03-19T11:27:08.389 INFO:teuthology.orchestra.run.smithi172.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2159-g9ef43722-1jammy [8,625 kB] 2024-03-19T11:27:08.533 INFO:teuthology.orchestra.run.smithi172.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-03-19T11:27:08.561 INFO:teuthology.orchestra.run.smithi172.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-03-19T11:27:08.615 INFO:teuthology.orchestra.run.smithi038.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-03-19T11:27:08.632 INFO:teuthology.orchestra.run.smithi038.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-03-19T11:27:08.719 INFO:teuthology.orchestra.run.smithi172.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-03-19T11:27:08.721 INFO:teuthology.orchestra.run.smithi172.stdout:Get:109 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-03-19T11:27:08.723 INFO:teuthology.orchestra.run.smithi172.stdout:Get:110 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-03-19T11:27:08.746 INFO:teuthology.orchestra.run.smithi172.stdout:Get:111 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-03-19T11:27:08.751 INFO:teuthology.orchestra.run.smithi172.stdout:Get:112 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-03-19T11:27:08.786 INFO:teuthology.orchestra.run.smithi038.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-03-19T11:27:08.816 INFO:teuthology.orchestra.run.smithi038.stdout:Get:109 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-03-19T11:27:08.934 INFO:teuthology.orchestra.run.smithi172.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2159-g9ef43722-1jammy [14.2 kB] 2024-03-19T11:27:08.935 INFO:teuthology.orchestra.run.smithi172.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2159-g9ef43722-1jammy [50.2 MB] 2024-03-19T11:27:08.973 INFO:teuthology.orchestra.run.smithi038.stdout:Get:110 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-03-19T11:27:08.976 INFO:teuthology.orchestra.run.smithi038.stdout:Get:111 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-03-19T11:27:08.977 INFO:teuthology.orchestra.run.smithi038.stdout:Get:112 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-03-19T11:27:09.000 INFO:teuthology.orchestra.run.smithi038.stdout:Get:113 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-03-19T11:27:09.005 INFO:teuthology.orchestra.run.smithi038.stdout:Get:114 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-03-19T11:27:11.873 INFO:teuthology.orchestra.run.smithi038.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2159-g9ef43722-1jammy [121 kB] 2024-03-19T11:27:11.875 INFO:teuthology.orchestra.run.smithi038.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2159-g9ef43722-1jammy [38.5 kB] 2024-03-19T11:27:11.877 INFO:teuthology.orchestra.run.smithi038.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2159-g9ef43722-1jammy [13.2 MB] 2024-03-19T11:27:11.949 INFO:teuthology.orchestra.run.smithi172.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2159-g9ef43722-1jammy [121 kB] 2024-03-19T11:27:11.951 INFO:teuthology.orchestra.run.smithi172.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2159-g9ef43722-1jammy [38.5 kB] 2024-03-19T11:27:11.952 INFO:teuthology.orchestra.run.smithi172.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2159-g9ef43722-1jammy [13.2 MB] 2024-03-19T11:27:12.503 INFO:teuthology.orchestra.run.smithi172.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2159-g9ef43722-1jammy [91.7 kB] 2024-03-19T11:27:12.636 INFO:teuthology.orchestra.run.smithi038.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/wip-rf-nextobj/9ef43722ad94ea688fa7f70daab6dd41fac6ee53/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2159-g9ef43722-1jammy [91.7 kB] 2024-03-19T11:27:13.062 INFO:teuthology.orchestra.run.smithi172.stdout:Fetched 178 MB in 11s (16.6 MB/s) 2024-03-19T11:27:13.217 INFO:teuthology.orchestra.run.smithi038.stdout:Fetched 178 MB in 11s (16.7 MB/s) 2024-03-19T11:27:13.222 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-03-19T11:27:13.295 INFO:teuthology.orchestra.run.smithi172.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165997 files and directories currently installed.) 2024-03-19T11:27:13.301 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-03-19T11:27:13.383 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-03-19T11:27:13.387 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-19T11:27:13.486 INFO:teuthology.orchestra.run.smithi038.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165997 files and directories currently installed.) 2024-03-19T11:27:13.492 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-03-19T11:27:13.573 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-19T11:27:13.749 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-03-19T11:27:13.776 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-03-19T11:27:13.816 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-19T11:27:13.919 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-03-19T11:27:13.945 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-03-19T11:27:13.996 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-19T11:27:14.119 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-03-19T11:27:14.136 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-03-19T11:27:14.168 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-19T11:27:14.504 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-03-19T11:27:14.530 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-03-19T11:27:14.614 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-03-19T11:27:14.640 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-19T11:27:14.689 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-19T11:27:14.724 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-19T11:27:15.100 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-03-19T11:27:15.127 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-19T11:27:15.167 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-19T11:27:15.194 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-03-19T11:27:15.221 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-19T11:27:15.269 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-19T11:27:15.445 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-03-19T11:27:15.461 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-19T11:27:15.495 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-19T11:27:15.689 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-03-19T11:27:15.716 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-19T11:27:15.756 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-19T11:27:15.848 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-03-19T11:27:15.875 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-03-19T11:27:15.915 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-19T11:27:16.042 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-03-19T11:27:16.069 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-19T11:27:16.100 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-19T11:27:16.251 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package librados2. 2024-03-19T11:27:16.277 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../007-librados2_19.0.0-2159-g9ef43722-1jammy_amd64.deb ... 2024-03-19T11:27:16.317 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking librados2 (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:16.454 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-03-19T11:27:16.480 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-03-19T11:27:16.521 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-19T11:27:16.762 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package librbd1. 2024-03-19T11:27:16.789 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../008-librbd1_19.0.0-2159-g9ef43722-1jammy_amd64.deb ... 2024-03-19T11:27:16.820 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking librbd1 (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:16.840 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package librados2. 2024-03-19T11:27:16.867 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../007-librados2_19.0.0-2159-g9ef43722-1jammy_amd64.deb ... 2024-03-19T11:27:16.907 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking librados2 (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:17.240 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libcephfs2. 2024-03-19T11:27:17.266 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-2159-g9ef43722-1jammy_amd64.deb ... 2024-03-19T11:27:17.306 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libcephfs2 (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:17.335 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package librbd1. 2024-03-19T11:27:17.361 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../008-librbd1_19.0.0-2159-g9ef43722-1jammy_amd64.deb ... 2024-03-19T11:27:17.401 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking librbd1 (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:17.567 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-rados. 2024-03-19T11:27:17.583 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../010-python3-rados_19.0.0-2159-g9ef43722-1jammy_amd64.deb ... 2024-03-19T11:27:17.625 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-rados (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:17.838 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package libcephfs2. 2024-03-19T11:27:17.865 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-2159-g9ef43722-1jammy_amd64.deb ... 2024-03-19T11:27:17.904 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking libcephfs2 (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:17.953 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-03-19T11:27:17.980 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-2159-g9ef43722-1jammy_all.deb ... 2024-03-19T11:27:18.020 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-ceph-argparse (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:18.182 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-rados. 2024-03-19T11:27:18.198 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../010-python3-rados_19.0.0-2159-g9ef43722-1jammy_amd64.deb ... 2024-03-19T11:27:18.240 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-rados (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:18.281 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-cephfs. 2024-03-19T11:27:18.307 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-2159-g9ef43722-1jammy_amd64.deb ... 2024-03-19T11:27:18.356 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-cephfs (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:18.568 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-03-19T11:27:18.595 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-2159-g9ef43722-1jammy_all.deb ... 2024-03-19T11:27:18.625 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-ceph-common. 2024-03-19T11:27:18.635 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-ceph-argparse (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:18.641 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-2159-g9ef43722-1jammy_all.deb ... 2024-03-19T11:27:18.675 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-ceph-common (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:18.862 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-cephfs. 2024-03-19T11:27:18.878 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-2159-g9ef43722-1jammy_amd64.deb ... 2024-03-19T11:27:18.912 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-cephfs (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:18.978 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-wcwidth. 2024-03-19T11:27:19.004 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-03-19T11:27:19.045 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-19T11:27:19.232 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-ceph-common. 2024-03-19T11:27:19.259 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-2159-g9ef43722-1jammy_all.deb ... 2024-03-19T11:27:19.299 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-ceph-common (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:19.493 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-prettytable. 2024-03-19T11:27:19.519 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-03-19T11:27:19.623 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-03-19T11:27:19.926 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-rbd. 2024-03-19T11:27:19.927 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-wcwidth. 2024-03-19T11:27:19.953 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-2159-g9ef43722-1jammy_amd64.deb ... 2024-03-19T11:27:19.954 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-03-19T11:27:19.994 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-19T11:27:20.000 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-rbd (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:20.322 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-prettytable. 2024-03-19T11:27:20.349 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-03-19T11:27:20.362 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-03-19T11:27:20.388 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-03-19T11:27:20.389 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-03-19T11:27:20.428 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-19T11:27:20.641 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-rbd. 2024-03-19T11:27:20.657 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-2159-g9ef43722-1jammy_amd64.deb ... 2024-03-19T11:27:20.699 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-rbd (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:20.706 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-03-19T11:27:20.733 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-03-19T11:27:20.773 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-03-19T11:27:21.052 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-03-19T11:27:21.079 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-03-19T11:27:21.109 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-03-19T11:27:21.119 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-19T11:27:21.125 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-03-19T11:27:21.176 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-19T11:27:21.397 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-03-19T11:27:21.424 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-03-19T11:27:21.464 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-03-19T11:27:21.562 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package lua5.1. 2024-03-19T11:27:21.590 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-03-19T11:27:21.637 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-03-19T11:27:21.767 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-03-19T11:27:21.783 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-03-19T11:27:21.816 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-19T11:27:21.965 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package lua-any. 2024-03-19T11:27:21.992 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-03-19T11:27:22.032 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking lua-any (27ubuntu1) ... 2024-03-19T11:27:22.162 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package lua5.1. 2024-03-19T11:27:22.188 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-03-19T11:27:22.228 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-03-19T11:27:22.310 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package zip. 2024-03-19T11:27:22.336 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-03-19T11:27:22.384 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking zip (3.0-12build2) ... 2024-03-19T11:27:22.540 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package lua-any. 2024-03-19T11:27:22.567 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-03-19T11:27:22.607 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking lua-any (27ubuntu1) ... 2024-03-19T11:27:22.754 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package unzip. 2024-03-19T11:27:22.781 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-03-19T11:27:22.829 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-03-19T11:27:22.842 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package zip. 2024-03-19T11:27:22.858 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-03-19T11:27:22.892 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking zip (3.0-12build2) ... 2024-03-19T11:27:23.229 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package unzip. 2024-03-19T11:27:23.241 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package luarocks. 2024-03-19T11:27:23.256 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-03-19T11:27:23.268 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-03-19T11:27:23.296 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-03-19T11:27:23.308 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-03-19T11:27:23.683 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package luarocks. 2024-03-19T11:27:23.710 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-03-19T11:27:23.733 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package librgw2. 2024-03-19T11:27:23.750 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-03-19T11:27:23.759 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../025-librgw2_19.0.0-2159-g9ef43722-1jammy_amd64.deb ... 2024-03-19T11:27:23.799 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking librgw2 (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:24.142 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package librgw2. 2024-03-19T11:27:24.168 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../025-librgw2_19.0.0-2159-g9ef43722-1jammy_amd64.deb ... 2024-03-19T11:27:24.208 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking librgw2 (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:24.343 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-rgw. 2024-03-19T11:27:24.359 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-2159-g9ef43722-1jammy_amd64.deb ... 2024-03-19T11:27:24.401 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-rgw (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:24.746 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package liboath0:amd64. 2024-03-19T11:27:24.773 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-03-19T11:27:24.813 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-03-19T11:27:24.905 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-rgw. 2024-03-19T11:27:24.932 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-2159-g9ef43722-1jammy_amd64.deb ... 2024-03-19T11:27:25.010 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-rgw (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:25.091 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libradosstriper1. 2024-03-19T11:27:25.118 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-2159-g9ef43722-1jammy_amd64.deb ... 2024-03-19T11:27:25.157 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libradosstriper1 (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:25.339 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package liboath0:amd64. 2024-03-19T11:27:25.366 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-03-19T11:27:25.405 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-03-19T11:27:25.594 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-common. 2024-03-19T11:27:25.621 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../029-ceph-common_19.0.0-2159-g9ef43722-1jammy_amd64.deb ... 2024-03-19T11:27:25.661 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-common (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:25.725 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package libradosstriper1. 2024-03-19T11:27:25.752 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-2159-g9ef43722-1jammy_amd64.deb ... 2024-03-19T11:27:25.792 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking libradosstriper1 (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:26.220 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package ceph-common. 2024-03-19T11:27:26.247 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../029-ceph-common_19.0.0-2159-g9ef43722-1jammy_amd64.deb ... 2024-03-19T11:27:26.286 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking ceph-common (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:26.943 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-base. 2024-03-19T11:27:26.969 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../030-ceph-base_19.0.0-2159-g9ef43722-1jammy_amd64.deb ... 2024-03-19T11:27:27.010 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-base (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:27.581 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-03-19T11:27:27.607 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-03-19T11:27:27.609 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package ceph-base. 2024-03-19T11:27:27.636 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../030-ceph-base_19.0.0-2159-g9ef43722-1jammy_amd64.deb ... 2024-03-19T11:27:27.647 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-03-19T11:27:27.685 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking ceph-base (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:27.958 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-cheroot. 2024-03-19T11:27:27.985 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-03-19T11:27:28.025 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-19T11:27:28.280 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-03-19T11:27:28.307 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-03-19T11:27:28.347 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-03-19T11:27:28.353 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-03-19T11:27:28.380 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-03-19T11:27:28.419 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-03-19T11:27:28.667 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-cheroot. 2024-03-19T11:27:28.694 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-03-19T11:27:28.706 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-jaraco.text. 2024-03-19T11:27:28.732 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-03-19T11:27:28.733 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-19T11:27:28.764 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-03-19T11:27:29.034 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-03-19T11:27:29.061 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-03-19T11:27:29.061 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-03-19T11:27:29.085 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-03-19T11:27:29.100 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-03-19T11:27:29.128 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-03-19T11:27:29.389 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-jaraco.text. 2024-03-19T11:27:29.403 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-tempora. 2024-03-19T11:27:29.416 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-03-19T11:27:29.430 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-03-19T11:27:29.456 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-03-19T11:27:29.470 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-03-19T11:27:29.759 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-03-19T11:27:29.772 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-portend. 2024-03-19T11:27:29.786 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-03-19T11:27:29.799 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-03-19T11:27:29.825 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-03-19T11:27:29.839 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-03-19T11:27:30.117 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-03-19T11:27:30.128 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-tempora. 2024-03-19T11:27:30.144 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-03-19T11:27:30.144 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-03-19T11:27:30.184 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-03-19T11:27:30.186 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-03-19T11:27:30.448 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-portend. 2024-03-19T11:27:30.465 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-03-19T11:27:30.487 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-cherrypy3. 2024-03-19T11:27:30.498 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-03-19T11:27:30.513 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-03-19T11:27:30.553 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-03-19T11:27:30.801 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-03-19T11:27:30.828 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-03-19T11:27:30.867 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-03-19T11:27:30.873 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-natsort. 2024-03-19T11:27:30.900 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-03-19T11:27:30.931 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-03-19T11:27:31.171 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-cherrypy3. 2024-03-19T11:27:31.198 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-03-19T11:27:31.235 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-logutils. 2024-03-19T11:27:31.237 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-03-19T11:27:31.261 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-03-19T11:27:31.301 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-03-19T11:27:31.599 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-natsort. 2024-03-19T11:27:31.604 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-markupsafe. 2024-03-19T11:27:31.626 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-03-19T11:27:31.691 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-03-19T11:27:31.691 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-03-19T11:27:31.691 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-03-19T11:27:31.927 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-logutils. 2024-03-19T11:27:31.954 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-03-19T11:27:31.985 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-03-19T11:27:31.990 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-mako. 2024-03-19T11:27:32.017 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-03-19T11:27:32.057 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-19T11:27:32.264 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-markupsafe. 2024-03-19T11:27:32.290 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-03-19T11:27:32.339 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-03-19T11:27:32.360 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-simplegeneric. 2024-03-19T11:27:32.387 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-03-19T11:27:32.418 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-03-19T11:27:32.642 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-mako. 2024-03-19T11:27:32.669 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-03-19T11:27:32.679 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-singledispatch. 2024-03-19T11:27:32.706 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-03-19T11:27:32.708 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-19T11:27:32.754 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-03-19T11:27:33.037 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-simplegeneric. 2024-03-19T11:27:33.057 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-webob. 2024-03-19T11:27:33.063 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-03-19T11:27:33.084 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-03-19T11:27:33.103 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-03-19T11:27:33.123 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-03-19T11:27:33.415 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-singledispatch. 2024-03-19T11:27:33.441 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-03-19T11:27:33.481 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-03-19T11:27:33.502 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-waitress. 2024-03-19T11:27:33.528 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-03-19T11:27:33.572 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-19T11:27:33.784 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-webob. 2024-03-19T11:27:33.811 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-03-19T11:27:33.850 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-03-19T11:27:33.880 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-tempita. 2024-03-19T11:27:33.906 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-03-19T11:27:33.946 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-03-19T11:27:34.249 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-paste. 2024-03-19T11:27:34.276 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-03-19T11:27:34.316 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-03-19T11:27:34.653 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-03-19T11:27:34.679 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-03-19T11:27:34.719 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-03-19T11:27:34.846 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-waitress. 2024-03-19T11:27:34.873 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-03-19T11:27:34.908 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-19T11:27:35.374 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-tempita. 2024-03-19T11:27:35.401 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-03-19T11:27:35.440 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-03-19T11:27:35.727 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-paste. 2024-03-19T11:27:35.754 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-03-19T11:27:35.793 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-03-19T11:27:35.823 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pastedeploy. 2024-03-19T11:27:35.850 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-03-19T11:27:35.889 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-03-19T11:27:36.088 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-03-19T11:27:36.101 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-03-19T11:27:36.138 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-03-19T11:27:36.192 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-webtest. 2024-03-19T11:27:36.219 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-03-19T11:27:36.259 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-03-19T11:27:36.433 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-pastedeploy. 2024-03-19T11:27:36.460 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-03-19T11:27:36.524 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-03-19T11:27:36.553 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pecan. 2024-03-19T11:27:36.580 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-03-19T11:27:36.620 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-03-19T11:27:36.828 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-webtest. 2024-03-19T11:27:36.855 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-03-19T11:27:36.902 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-03-19T11:27:37.008 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-certifi. 2024-03-19T11:27:37.035 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-03-19T11:27:37.074 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-03-19T11:27:37.214 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-pecan. 2024-03-19T11:27:37.241 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-03-19T11:27:37.289 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-03-19T11:27:37.403 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-idna. 2024-03-19T11:27:37.429 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-03-19T11:27:37.469 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-idna (3.3-1) ... 2024-03-19T11:27:37.652 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-certifi. 2024-03-19T11:27:37.680 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-03-19T11:27:37.719 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-03-19T11:27:37.755 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-urllib3. 2024-03-19T11:27:37.783 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-03-19T11:27:37.822 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-19T11:27:37.997 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-idna. 2024-03-19T11:27:38.024 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-03-19T11:27:38.063 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-idna (3.3-1) ... 2024-03-19T11:27:38.125 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-requests. 2024-03-19T11:27:38.152 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-03-19T11:27:38.192 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-19T11:27:38.383 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-urllib3. 2024-03-19T11:27:38.411 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-03-19T11:27:38.450 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-19T11:27:38.461 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-werkzeug. 2024-03-19T11:27:38.478 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-03-19T11:27:38.519 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-19T11:27:38.778 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-requests. 2024-03-19T11:27:38.797 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-03-19T11:27:38.805 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-03-19T11:27:38.813 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-2159-g9ef43722-1jammy_all.deb ... 2024-03-19T11:27:38.845 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-19T11:27:38.847 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:39.131 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-werkzeug. 2024-03-19T11:27:39.158 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-03-19T11:27:39.189 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-19T11:27:39.244 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-03-19T11:27:39.271 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-2159-g9ef43722-1jammy_amd64.deb ... 2024-03-19T11:27:39.310 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:39.476 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-03-19T11:27:39.492 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-2159-g9ef43722-1jammy_all.deb ... 2024-03-19T11:27:39.534 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:39.622 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-bcrypt. 2024-03-19T11:27:39.649 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-03-19T11:27:39.688 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-03-19T11:27:39.923 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-03-19T11:27:39.950 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-2159-g9ef43722-1jammy_amd64.deb ... 2024-03-19T11:27:39.989 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:40.284 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-bcrypt. 2024-03-19T11:27:40.314 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-03-19T11:27:40.316 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-openssl. 2024-03-19T11:27:40.343 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-03-19T11:27:40.350 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-03-19T11:27:40.374 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-03-19T11:27:40.711 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mgr. 2024-03-19T11:27:40.738 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-2159-g9ef43722-1jammy_amd64.deb ... 2024-03-19T11:27:40.777 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mgr (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:40.812 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-openssl. 2024-03-19T11:27:40.839 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-03-19T11:27:40.878 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-03-19T11:27:41.172 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mon. 2024-03-19T11:27:41.199 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-2159-g9ef43722-1jammy_amd64.deb ... 2024-03-19T11:27:41.238 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mon (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:41.240 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package ceph-mgr. 2024-03-19T11:27:41.268 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-2159-g9ef43722-1jammy_amd64.deb ... 2024-03-19T11:27:41.307 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking ceph-mgr (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:41.710 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package ceph-mon. 2024-03-19T11:27:41.737 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-2159-g9ef43722-1jammy_amd64.deb ... 2024-03-19T11:27:41.776 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking ceph-mon (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:41.800 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-osd. 2024-03-19T11:27:41.828 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-2159-g9ef43722-1jammy_amd64.deb ... 2024-03-19T11:27:41.859 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-osd (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:42.455 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package ceph-osd. 2024-03-19T11:27:42.483 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-2159-g9ef43722-1jammy_amd64.deb ... 2024-03-19T11:27:42.522 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking ceph-osd (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:42.904 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph. 2024-03-19T11:27:42.930 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../066-ceph_19.0.0-2159-g9ef43722-1jammy_amd64.deb ... 2024-03-19T11:27:42.970 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:43.298 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-fuse. 2024-03-19T11:27:43.324 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-2159-g9ef43722-1jammy_amd64.deb ... 2024-03-19T11:27:43.364 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-fuse (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:43.592 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package ceph. 2024-03-19T11:27:43.608 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../066-ceph_19.0.0-2159-g9ef43722-1jammy_amd64.deb ... 2024-03-19T11:27:43.650 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking ceph (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:43.709 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mds. 2024-03-19T11:27:43.737 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-2159-g9ef43722-1jammy_amd64.deb ... 2024-03-19T11:27:43.767 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mds (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:43.970 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package ceph-fuse. 2024-03-19T11:27:43.997 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-2159-g9ef43722-1jammy_amd64.deb ... 2024-03-19T11:27:44.036 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking ceph-fuse (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:44.137 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package cephadm. 2024-03-19T11:27:44.163 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../069-cephadm_19.0.0-2159-g9ef43722-1jammy_amd64.deb ... 2024-03-19T11:27:44.206 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking cephadm (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:44.364 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package ceph-mds. 2024-03-19T11:27:44.381 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-2159-g9ef43722-1jammy_amd64.deb ... 2024-03-19T11:27:44.414 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking ceph-mds (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:44.523 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-asyncssh. 2024-03-19T11:27:44.550 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-03-19T11:27:44.589 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-03-19T11:27:44.809 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package cephadm. 2024-03-19T11:27:44.837 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../069-cephadm_19.0.0-2159-g9ef43722-1jammy_amd64.deb ... 2024-03-19T11:27:44.875 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking cephadm (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:44.918 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python-babel-localedata. 2024-03-19T11:27:44.944 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-03-19T11:27:44.984 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-19T11:27:45.237 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-asyncssh. 2024-03-19T11:27:45.265 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-03-19T11:27:45.303 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-03-19T11:27:45.765 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python-babel-localedata. 2024-03-19T11:27:45.792 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-03-19T11:27:45.831 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-19T11:27:46.073 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-babel. 2024-03-19T11:27:46.100 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-03-19T11:27:46.139 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-03-19T11:27:46.461 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-jinja2. 2024-03-19T11:27:46.486 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-03-19T11:27:46.525 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-19T11:27:46.870 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-babel. 2024-03-19T11:27:46.870 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-03-19T11:27:46.897 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-2159-g9ef43722-1jammy_all.deb ... 2024-03-19T11:27:46.898 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-03-19T11:27:46.936 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-03-19T11:27:46.957 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:47.239 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-jinja2. 2024-03-19T11:27:47.267 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-03-19T11:27:47.297 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-19T11:27:47.298 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-repoze.lru. 2024-03-19T11:27:47.325 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-03-19T11:27:47.364 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-03-19T11:27:47.617 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-03-19T11:27:47.642 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-2159-g9ef43722-1jammy_all.deb ... 2024-03-19T11:27:47.668 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-routes. 2024-03-19T11:27:47.676 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:47.695 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-03-19T11:27:47.734 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-03-19T11:27:48.029 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-repoze.lru. 2024-03-19T11:27:48.057 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-03-19T11:27:48.070 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-03-19T11:27:48.095 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-03-19T11:27:48.097 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-2159-g9ef43722-1jammy_all.deb ... 2024-03-19T11:27:48.137 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:48.365 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-routes. 2024-03-19T11:27:48.382 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-03-19T11:27:48.415 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-03-19T11:27:48.549 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-03-19T11:27:48.576 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-03-19T11:27:48.615 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-19T11:27:48.710 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-03-19T11:27:48.727 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-2159-g9ef43722-1jammy_all.deb ... 2024-03-19T11:27:48.760 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:49.061 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-joblib. 2024-03-19T11:27:49.087 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-03-19T11:27:49.119 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-03-19T11:27:49.239 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-03-19T11:27:49.266 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-03-19T11:27:49.305 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-19T11:27:49.481 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-03-19T11:27:49.507 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-03-19T11:27:49.547 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-03-19T11:27:49.834 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-joblib. 2024-03-19T11:27:49.850 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-sklearn. 2024-03-19T11:27:49.861 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-03-19T11:27:49.877 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-03-19T11:27:49.909 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-03-19T11:27:49.916 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-19T11:27:50.279 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-03-19T11:27:50.306 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-03-19T11:27:50.345 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-03-19T11:27:50.625 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-03-19T11:27:50.653 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-2159-g9ef43722-1jammy_all.deb ... 2024-03-19T11:27:50.657 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-sklearn. 2024-03-19T11:27:50.684 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-03-19T11:27:50.732 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-19T11:27:50.777 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:51.435 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-03-19T11:27:51.463 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-2159-g9ef43722-1jammy_all.deb ... 2024-03-19T11:27:51.579 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:51.772 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-cachetools. 2024-03-19T11:27:51.799 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-03-19T11:27:51.839 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-03-19T11:27:52.125 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pyasn1. 2024-03-19T11:27:52.152 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-03-19T11:27:52.191 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-03-19T11:27:52.532 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-cachetools. 2024-03-19T11:27:52.545 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-03-19T11:27:52.559 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-03-19T11:27:52.572 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-03-19T11:27:52.598 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-03-19T11:27:52.611 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-03-19T11:27:52.869 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-pyasn1. 2024-03-19T11:27:52.896 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-03-19T11:27:52.923 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-rsa. 2024-03-19T11:27:52.926 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-03-19T11:27:52.950 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-03-19T11:27:52.989 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-rsa (4.8-1) ... 2024-03-19T11:27:53.260 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-google-auth. 2024-03-19T11:27:53.263 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-03-19T11:27:53.287 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-03-19T11:27:53.291 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-03-19T11:27:53.318 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-03-19T11:27:53.321 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-03-19T11:27:53.625 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-rsa. 2024-03-19T11:27:53.646 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-03-19T11:27:53.652 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-03-19T11:27:53.673 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-03-19T11:27:53.691 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-rsa (4.8-1) ... 2024-03-19T11:27:53.713 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-19T11:27:53.996 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-google-auth. 2024-03-19T11:27:54.023 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-03-19T11:27:54.024 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-websocket. 2024-03-19T11:27:54.051 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-03-19T11:27:54.062 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-03-19T11:27:54.090 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-03-19T11:27:54.411 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-kubernetes. 2024-03-19T11:27:54.423 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-03-19T11:27:54.438 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-03-19T11:27:54.451 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-03-19T11:27:54.477 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-19T11:27:54.490 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-19T11:27:54.802 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-websocket. 2024-03-19T11:27:54.829 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-03-19T11:27:54.868 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-03-19T11:27:55.158 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-03-19T11:27:55.174 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-2159-g9ef43722-1jammy_all.deb ... 2024-03-19T11:27:55.180 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-kubernetes. 2024-03-19T11:27:55.207 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:55.208 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-03-19T11:27:55.246 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-19T11:27:55.811 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libonig5:amd64. 2024-03-19T11:27:55.838 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-03-19T11:27:55.877 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-19T11:27:56.213 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libjq1:amd64. 2024-03-19T11:27:56.241 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-03-19T11:27:56.278 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-03-19T11:27:56.280 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-19T11:27:56.306 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-2159-g9ef43722-1jammy_all.deb ... 2024-03-19T11:27:56.344 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:56.549 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package jq. 2024-03-19T11:27:56.565 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-03-19T11:27:56.607 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-03-19T11:27:56.655 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package libonig5:amd64. 2024-03-19T11:27:56.683 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-03-19T11:27:56.722 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-19T11:27:56.843 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package socat. 2024-03-19T11:27:56.859 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-03-19T11:27:56.893 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-03-19T11:27:57.058 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package libjq1:amd64. 2024-03-19T11:27:57.086 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-03-19T11:27:57.133 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-19T11:27:57.188 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package xmlstarlet. 2024-03-19T11:27:57.213 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-03-19T11:27:57.255 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-03-19T11:27:57.419 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package jq. 2024-03-19T11:27:57.436 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-03-19T11:27:57.469 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-03-19T11:27:57.601 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-test. 2024-03-19T11:27:57.617 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../097-ceph-test_19.0.0-2159-g9ef43722-1jammy_amd64.deb ... 2024-03-19T11:27:57.659 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-test (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:57.663 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package socat. 2024-03-19T11:27:57.680 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-03-19T11:27:57.722 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-03-19T11:27:58.025 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package xmlstarlet. 2024-03-19T11:27:58.042 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-03-19T11:27:58.083 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-03-19T11:27:58.455 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package ceph-test. 2024-03-19T11:27:58.483 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../097-ceph-test_19.0.0-2159-g9ef43722-1jammy_amd64.deb ... 2024-03-19T11:27:58.521 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking ceph-test (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:27:59.922 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-volume. 2024-03-19T11:27:59.949 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-2159-g9ef43722-1jammy_all.deb ... 2024-03-19T11:27:59.988 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-volume (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:00.300 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libcephfs-dev. 2024-03-19T11:28:00.328 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-2159-g9ef43722-1jammy_amd64.deb ... 2024-03-19T11:28:00.367 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libcephfs-dev (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:00.695 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package lua-socket:amd64. 2024-03-19T11:28:00.722 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-03-19T11:28:00.795 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-19T11:28:00.867 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package ceph-volume. 2024-03-19T11:28:00.895 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-2159-g9ef43722-1jammy_all.deb ... 2024-03-19T11:28:00.933 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking ceph-volume (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:01.175 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package lua-sec:amd64. 2024-03-19T11:28:01.191 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-03-19T11:28:01.238 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package libcephfs-dev. 2024-03-19T11:28:01.241 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-03-19T11:28:01.254 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-2159-g9ef43722-1jammy_amd64.deb ... 2024-03-19T11:28:01.296 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking libcephfs-dev (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:01.612 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package nvme-cli. 2024-03-19T11:28:01.632 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package lua-socket:amd64. 2024-03-19T11:28:01.639 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-03-19T11:28:01.660 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-03-19T11:28:01.678 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-03-19T11:28:01.699 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-19T11:28:02.029 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package lua-sec:amd64. 2024-03-19T11:28:02.057 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-03-19T11:28:02.075 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package pkg-config. 2024-03-19T11:28:02.095 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-03-19T11:28:02.103 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-03-19T11:28:02.141 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-03-19T11:28:02.403 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-03-19T11:28:02.416 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package nvme-cli. 2024-03-19T11:28:02.431 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-03-19T11:28:02.444 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-03-19T11:28:02.469 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-03-19T11:28:02.482 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-03-19T11:28:02.863 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package pkg-config. 2024-03-19T11:28:02.872 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-iniconfig. 2024-03-19T11:28:02.891 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-03-19T11:28:02.900 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-03-19T11:28:02.921 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-03-19T11:28:02.939 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-03-19T11:28:03.157 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-03-19T11:28:03.174 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-03-19T11:28:03.215 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-03-19T11:28:03.242 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pastescript. 2024-03-19T11:28:03.269 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-03-19T11:28:03.308 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-03-19T11:28:03.594 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-iniconfig. 2024-03-19T11:28:03.622 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-03-19T11:28:03.645 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pluggy. 2024-03-19T11:28:03.660 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-03-19T11:28:03.673 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-03-19T11:28:03.712 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-03-19T11:28:03.974 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-psutil. 2024-03-19T11:28:03.974 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-pastescript. 2024-03-19T11:28:03.999 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-03-19T11:28:04.000 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-03-19T11:28:04.031 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-03-19T11:28:04.038 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-03-19T11:28:04.351 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-py. 2024-03-19T11:28:04.358 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-pluggy. 2024-03-19T11:28:04.378 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-03-19T11:28:04.386 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-03-19T11:28:04.425 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-03-19T11:28:04.425 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-py (1.10.0-1) ... 2024-03-19T11:28:04.704 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pygments. 2024-03-19T11:28:04.720 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-psutil. 2024-03-19T11:28:04.720 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-03-19T11:28:04.748 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-03-19T11:28:04.754 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-03-19T11:28:04.786 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-03-19T11:28:05.106 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-py. 2024-03-19T11:28:05.134 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-03-19T11:28:05.172 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-py (1.10.0-1) ... 2024-03-19T11:28:05.199 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pyinotify. 2024-03-19T11:28:05.227 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-03-19T11:28:05.266 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-03-19T11:28:05.514 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-pygments. 2024-03-19T11:28:05.542 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-03-19T11:28:05.571 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-toml. 2024-03-19T11:28:05.584 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-03-19T11:28:05.598 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-03-19T11:28:05.637 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-03-19T11:28:05.940 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pytest. 2024-03-19T11:28:05.968 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-03-19T11:28:05.997 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-pyinotify. 2024-03-19T11:28:06.006 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-03-19T11:28:06.013 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-03-19T11:28:06.046 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-03-19T11:28:06.333 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-toml. 2024-03-19T11:28:06.343 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-simplejson. 2024-03-19T11:28:06.361 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-03-19T11:28:06.371 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-03-19T11:28:06.399 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-03-19T11:28:06.409 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-03-19T11:28:06.679 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-03-19T11:28:06.707 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-03-19T11:28:06.710 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-pytest. 2024-03-19T11:28:06.739 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-03-19T11:28:06.745 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-03-19T11:28:06.777 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-03-19T11:28:07.114 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package python3-simplejson. 2024-03-19T11:28:07.142 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-03-19T11:28:07.180 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-03-19T11:28:07.441 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-03-19T11:28:07.458 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-03-19T11:28:07.500 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-03-19T11:28:07.525 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package radosgw. 2024-03-19T11:28:07.553 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../116-radosgw_19.0.0-2159-g9ef43722-1jammy_amd64.deb ... 2024-03-19T11:28:07.583 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking radosgw (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:08.246 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package radosgw. 2024-03-19T11:28:08.274 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../116-radosgw_19.0.0-2159-g9ef43722-1jammy_amd64.deb ... 2024-03-19T11:28:08.312 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking radosgw (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:08.386 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package rbd-fuse. 2024-03-19T11:28:08.413 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-2159-g9ef43722-1jammy_amd64.deb ... 2024-03-19T11:28:08.452 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking rbd-fuse (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:08.757 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-03-19T11:28:09.016 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-19T11:28:09.133 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-03-19T11:28:09.207 INFO:teuthology.orchestra.run.smithi038.stdout:Selecting previously unselected package rbd-fuse. 2024-03-19T11:28:09.224 INFO:teuthology.orchestra.run.smithi038.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-2159-g9ef43722-1jammy_amd64.deb ... 2024-03-19T11:28:09.265 INFO:teuthology.orchestra.run.smithi038.stdout:Unpacking rbd-fuse (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:09.365 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-03-19T11:28:09.562 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-03-19T11:28:09.678 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-03-19T11:28:09.837 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-19T11:28:09.963 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-03-19T11:28:10.120 INFO:teuthology.orchestra.run.smithi172.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-03-19T11:28:10.176 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up cephadm (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:10.232 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-03-19T11:28:10.582 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-03-19T11:28:10.760 INFO:teuthology.orchestra.run.smithi172.stdout:Adding system user cephadm....done 2024-03-19T11:28:10.916 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-19T11:28:11.014 INFO:teuthology.orchestra.run.smithi038.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-03-19T11:28:11.082 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up cephadm (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:11.267 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-03-19T11:28:11.502 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-03-19T11:28:11.629 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-03-19T11:28:11.879 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-03-19T11:28:11.998 INFO:teuthology.orchestra.run.smithi038.stdout:Adding system user cephadm....done 2024-03-19T11:28:12.155 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-19T11:28:12.155 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-03-19T11:28:12.281 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-py (1.10.0-1) ... 2024-03-19T11:28:12.431 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-03-19T11:28:12.591 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-03-19T11:28:12.699 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-03-19T11:28:12.800 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-03-19T11:28:12.927 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-03-19T11:28:13.135 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-03-19T11:28:13.186 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-03-19T11:28:13.403 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-03-19T11:28:13.454 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-03-19T11:28:13.529 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-py (1.10.0-1) ... 2024-03-19T11:28:13.739 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-03-19T11:28:13.831 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-03-19T11:28:14.015 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-03-19T11:28:14.208 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-03-19T11:28:14.274 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-03-19T11:28:14.459 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-03-19T11:28:14.517 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-ceph-argparse (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:14.711 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-03-19T11:28:14.768 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-19T11:28:14.902 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-19T11:28:14.945 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-03-19T11:28:15.028 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-03-19T11:28:15.154 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-03-19T11:28:15.205 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-03-19T11:28:15.414 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-03-19T11:28:15.514 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-03-19T11:28:15.796 INFO:teuthology.orchestra.run.smithi172.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-03-19T11:28:15.828 INFO:teuthology.orchestra.run.smithi172.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-03-19T11:28:15.840 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-ceph-argparse (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:15.884 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-19T11:28:15.985 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-03-19T11:28:16.092 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-19T11:28:16.201 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-19T11:28:16.310 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-03-19T11:28:16.352 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-19T11:28:16.420 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-03-19T11:28:16.478 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-03-19T11:28:16.679 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-03-19T11:28:16.746 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-03-19T11:28:17.014 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-03-19T11:28:17.063 INFO:teuthology.orchestra.run.smithi038.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-03-19T11:28:17.094 INFO:teuthology.orchestra.run.smithi038.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-03-19T11:28:17.149 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-19T11:28:17.250 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-03-19T11:28:17.315 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up zip (3.0-12build2) ... 2024-03-19T11:28:17.433 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-03-19T11:28:17.610 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-19T11:28:17.711 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-03-19T11:28:18.004 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-03-19T11:28:18.084 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-03-19T11:28:18.272 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-03-19T11:28:18.336 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-03-19T11:28:18.462 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-03-19T11:28:18.581 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up zip (3.0-12build2) ... 2024-03-19T11:28:18.588 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-19T11:28:18.708 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-03-19T11:28:18.880 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-03-19T11:28:19.256 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-03-19T11:28:19.393 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-03-19T11:28:19.491 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-03-19T11:28:19.644 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-03-19T11:28:19.770 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-03-19T11:28:19.851 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-19T11:28:19.896 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-19T11:28:20.152 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-19T11:28:20.197 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-03-19T11:28:20.520 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-03-19T11:28:20.574 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-03-19T11:28:20.771 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-03-19T11:28:20.825 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-03-19T11:28:20.914 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-ceph-common (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:21.215 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-idna (3.3-1) ... 2024-03-19T11:28:21.227 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-19T11:28:21.525 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-19T11:28:21.528 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-19T11:28:21.908 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-03-19T11:28:22.172 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-03-19T11:28:22.298 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-ceph-common (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:22.608 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-idna (3.3-1) ... 2024-03-19T11:28:22.638 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-03-19T11:28:22.890 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-19T11:28:22.934 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-19T11:28:23.008 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-toml (0.10.2-1) ... 2024-03-19T11:28:23.242 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-19T11:28:23.368 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-03-19T11:28:23.503 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-19T11:28:23.821 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-03-19T11:28:24.080 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-03-19T11:28:24.149 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-03-19T11:28:24.348 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-19T11:28:24.383 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-19T11:28:24.457 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-03-19T11:28:24.518 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-toml (0.10.2-1) ... 2024-03-19T11:28:24.750 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-03-19T11:28:24.786 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-19T11:28:24.912 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-03-19T11:28:25.001 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-03-19T11:28:25.038 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-19T11:28:25.286 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-03-19T11:28:25.356 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-03-19T11:28:25.537 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-03-19T11:28:25.616 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-03-19T11:28:25.788 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-03-19T11:28:25.867 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-19T11:28:26.018 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-03-19T11:28:26.064 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-19T11:28:26.190 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-03-19T11:28:26.361 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-03-19T11:28:26.458 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-19T11:28:26.559 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-19T11:28:26.621 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-03-19T11:28:26.852 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-03-19T11:28:26.905 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-03-19T11:28:27.156 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-03-19T11:28:27.178 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-03-19T11:28:27.407 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-03-19T11:28:27.429 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-19T11:28:27.555 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-03-19T11:28:27.681 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-03-19T11:28:27.683 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-19T11:28:27.771 INFO:teuthology.orchestra.run.smithi172.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-03-19T11:28:27.793 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-03-19T11:28:27.999 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-19T11:28:28.061 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-19T11:28:28.100 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-03-19T11:28:28.162 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-19T11:28:28.438 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-03-19T11:28:28.476 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-03-19T11:28:28.722 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-03-19T11:28:28.753 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up lua-any (27ubuntu1) ... 2024-03-19T11:28:28.879 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-portend (3.0.0-1) ... 2024-03-19T11:28:28.982 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-19T11:28:29.108 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-03-19T11:28:29.146 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-19T11:28:29.234 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-03-19T11:28:29.272 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-19T11:28:29.324 INFO:teuthology.orchestra.run.smithi038.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-03-19T11:28:29.578 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-19T11:28:29.590 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-03-19T11:28:29.704 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-03-19T11:28:29.883 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-19T11:28:30.134 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-03-19T11:28:30.138 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-03-19T11:28:30.285 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-rsa (4.8-1) ... 2024-03-19T11:28:30.389 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up lua-any (27ubuntu1) ... 2024-03-19T11:28:30.516 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-portend (3.0.0-1) ... 2024-03-19T11:28:30.553 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-03-19T11:28:30.775 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-19T11:28:30.829 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-19T11:28:30.884 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-19T11:28:31.174 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-03-19T11:28:31.202 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-03-19T11:28:31.495 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-19T11:28:31.559 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-03-19T11:28:31.755 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-03-19T11:28:31.844 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-03-19T11:28:31.881 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-rsa (4.8-1) ... 2024-03-19T11:28:32.155 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-19T11:28:32.281 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up librados2 (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:32.390 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-03-19T11:28:32.398 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:32.499 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-03-19T11:28:32.708 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-19T11:28:32.811 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libcephfs2 (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:32.937 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libradosstriper1 (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:33.062 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-03-19T11:28:33.063 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-03-19T11:28:33.339 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up librbd1 (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:33.422 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-03-19T11:28:33.465 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mgr-modules-core (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:33.633 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-fuse (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:33.715 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-03-19T11:28:33.848 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-19T11:28:33.848 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-19T11:28:34.051 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-19T11:28:34.210 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up librados2 (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:34.320 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:34.393 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libcephfs-dev (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:34.421 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-03-19T11:28:34.519 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-rados (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:34.741 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up libcephfs2 (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:34.867 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up libradosstriper1 (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:34.954 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up librgw2 (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:34.993 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-03-19T11:28:35.080 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-19T11:28:35.270 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up librbd1 (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:35.396 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up ceph-mgr-modules-core (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:35.522 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up ceph-fuse (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:35.746 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-19T11:28:35.746 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-19T11:28:36.225 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-rbd (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:36.231 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up libcephfs-dev (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:36.334 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up rbd-fuse (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:36.358 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-rados (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:36.444 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-rgw (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:36.484 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up librgw2 (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:36.553 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-cephfs (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:36.585 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-19T11:28:36.679 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-common (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:37.247 INFO:teuthology.orchestra.run.smithi172.stdout:Adding group ceph....done 2024-03-19T11:28:37.607 INFO:teuthology.orchestra.run.smithi172.stdout:Adding system user ceph....done 2024-03-19T11:28:37.767 INFO:teuthology.orchestra.run.smithi172.stdout:Setting system user ceph properties....done 2024-03-19T11:28:37.785 INFO:teuthology.orchestra.run.smithi172.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-03-19T11:28:37.864 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-rbd (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:37.915 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-03-19T11:28:37.990 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up rbd-fuse (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:38.116 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-rgw (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:38.242 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up python3-cephfs (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:38.249 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-03-19T11:28:38.410 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up ceph-common (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:38.714 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-test (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:38.840 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up radosgw (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:38.930 INFO:teuthology.orchestra.run.smithi038.stdout:Adding group ceph....done 2024-03-19T11:28:39.289 INFO:teuthology.orchestra.run.smithi038.stdout:Adding system user ceph....done 2024-03-19T11:28:39.408 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-19T11:28:39.408 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-19T11:28:39.458 INFO:teuthology.orchestra.run.smithi038.stdout:Setting system user ceph properties....done 2024-03-19T11:28:39.477 INFO:teuthology.orchestra.run.smithi038.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-03-19T11:28:39.625 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-03-19T11:28:39.883 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-base (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:39.966 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-03-19T11:28:40.218 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-03-19T11:28:40.447 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up ceph-test (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:40.573 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up radosgw (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:40.685 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mds (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:40.899 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-19T11:28:40.899 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-19T11:28:41.062 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-19T11:28:41.062 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-19T11:28:41.395 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mgr (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:41.525 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up ceph-base (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:41.665 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-19T11:28:41.665 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-19T11:28:41.893 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-03-19T11:28:42.360 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up ceph-mds (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:42.370 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-osd (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:42.621 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-19T11:28:42.622 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-19T11:28:42.729 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-19T11:28:42.729 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-19T11:28:43.078 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up ceph-mgr (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:43.233 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:43.314 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-19T11:28:43.314 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-19T11:28:43.334 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:43.493 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mon (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:43.732 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-19T11:28:43.733 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-19T11:28:43.780 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up ceph-osd (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:44.081 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-19T11:28:44.081 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-19T11:28:44.236 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mgr-cephadm (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:44.379 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:44.505 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mgr-dashboard (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:44.590 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:44.656 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-volume (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:44.716 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:44.867 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up ceph-mon (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:44.983 INFO:teuthology.orchestra.run.smithi172.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-03-19T11:28:45.068 INFO:teuthology.orchestra.run.smithi172.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-19T11:28:45.084 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-19T11:28:45.084 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-19T11:28:45.229 INFO:teuthology.orchestra.run.smithi172.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-19T11:28:45.551 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up ceph-mgr-cephadm (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:45.677 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up ceph (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:45.778 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up ceph-mgr-dashboard (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:45.913 INFO:teuthology.orchestra.run.smithi038.stdout:Setting up ceph-volume (19.0.0-2159-g9ef43722-1jammy) ... 2024-03-19T11:28:45.962 INFO:teuthology.orchestra.run.smithi172.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-03-19T11:28:46.274 INFO:teuthology.orchestra.run.smithi038.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-03-19T11:28:46.383 INFO:teuthology.orchestra.run.smithi038.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-19T11:28:46.544 INFO:teuthology.orchestra.run.smithi038.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-19T11:28:46.641 INFO:teuthology.orchestra.run.smithi172.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-19T11:28:46.647 DEBUG:teuthology.parallel:result is None 2024-03-19T11:28:47.637 INFO:teuthology.orchestra.run.smithi038.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-03-19T11:28:48.353 INFO:teuthology.orchestra.run.smithi038.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-19T11:28:48.358 DEBUG:teuthology.parallel:result is None 2024-03-19T11:28:48.358 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=9ef43722ad94ea688fa7f70daab6dd41fac6ee53 2024-03-19T11:28:48.500 DEBUG:teuthology.orchestra.run.smithi038:> dpkg-query -W -f '${Version}' ceph 2024-03-19T11:28:48.520 INFO:teuthology.orchestra.run.smithi038.stdout:19.0.0-2159-g9ef43722-1jammy 2024-03-19T11:28:48.520 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2159-g9ef43722-1jammy 2024-03-19T11:28:48.520 INFO:teuthology.task.install:The correct ceph version 19.0.0-2159-g9ef43722-1jammy is installed. 2024-03-19T11:28:48.522 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=9ef43722ad94ea688fa7f70daab6dd41fac6ee53 2024-03-19T11:28:48.738 DEBUG:teuthology.orchestra.run.smithi172:> dpkg-query -W -f '${Version}' ceph 2024-03-19T11:28:48.759 INFO:teuthology.orchestra.run.smithi172.stdout:19.0.0-2159-g9ef43722-1jammy 2024-03-19T11:28:48.760 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2159-g9ef43722-1jammy 2024-03-19T11:28:48.760 INFO:teuthology.task.install:The correct ceph version 19.0.0-2159-g9ef43722-1jammy is installed. 2024-03-19T11:28:48.762 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-19T11:28:48.762 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-19T11:28:48.762 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-19T11:28:48.778 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-19T11:28:48.778 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-19T11:28:48.815 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-19T11:28:48.815 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-19T11:28:48.815 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/daemon-helper 2024-03-19T11:28:48.833 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-19T11:28:48.890 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-19T11:28:48.890 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/daemon-helper 2024-03-19T11:28:48.904 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-19T11:28:48.959 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-19T11:28:48.959 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-19T11:28:48.959 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-19T11:28:48.973 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-19T11:28:49.028 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-19T11:28:49.028 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-19T11:28:49.042 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-19T11:28:49.097 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-19T11:28:49.098 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-19T11:28:49.098 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/stdin-killer 2024-03-19T11:28:49.112 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-19T11:28:49.167 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-19T11:28:49.167 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/stdin-killer 2024-03-19T11:28:49.180 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-19T11:28:49.233 INFO:teuthology.run_tasks:Running task ceph... 2024-03-19T11:28:49.316 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-19T11:28:49.316 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 777 /var/log/ceph 2024-03-19T11:28:49.318 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 777 /var/log/ceph 2024-03-19T11:28:49.332 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-19T11:28:49.332 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-19T11:28:49.377 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-19T11:28:49.389 INFO:tasks.ceph:Creating extra log directories... 2024-03-19T11:28:49.389 DEBUG:teuthology.orchestra.run.smithi038:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-19T11:28:49.433 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-19T11:28:49.459 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-19T11:28:49.459 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd heartbeat use min delay socket': True, 'osd inject bad map crc probability': 0.1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd max scrubs': 5, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2, 'osd_recovery_max_active': 10, 'osd_recovery_max_single_start': 10}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', 'failed to encode map', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-19T11:28:49.459 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rfriedma-2024-03-19_08:22:49-rados:thrash-wip-rf-nextobj-distro-default-smithi/7610739', 'branch': 'wip-rf-nextobj', 'description': 'rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/dedup-io-snaps}', 'email': None, 'first_in_suite': False, 'job_id': '7610739', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'rfriedma-2024-03-19_08:22:49-rados:thrash-wip-rf-nextobj-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-rf-nextobj'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2, 'osd_recovery_max_active': 10, 'osd_recovery_max_single_start': 10}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd heartbeat use min delay socket': True, 'osd inject bad map crc probability': 0.1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd max scrubs': 5, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', 'failed to encode map', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '9ef43722ad94ea688fa7f70daab6dd41fac6ee53'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '9ef43722ad94ea688fa7f70daab6dd41fac6ee53'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-rf-nextobj', 'sha1': '9ef43722ad94ea688fa7f70daab6dd41fac6ee53'}}, 'owner': 'scheduled_rfriedma@teuthology', 'priority': 95, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 2525, 'sha1': '9ef43722ad94ea688fa7f70daab6dd41fac6ee53', 'sleep_before_teardown': 0, 'subset': '4/400', 'suite': 'rados:thrash', 'suite_branch': 'wip-rf-nextobj', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '9ef43722ad94ea688fa7f70daab6dd41fac6ee53', 'targets': {'smithi038.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN12b76+6OZUqzRu5kZgY1R3xXUct6XvKeRMHOD6LkRGOzsW2C5qGHLKLNc/hTgPYE3wvjLymEenOFRKGhA/ouk=', 'smithi172.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOkhYhhIXg4DTniTbJjE1J7at0/UswGbMkAbBGurn57KL5GO5auYOgugBa786l9o0o5uMhI6NPM/z44KCkJT+3g='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd heartbeat use min delay socket': True, 'osd inject bad map crc probability': 0.1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd max scrubs': 5, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2, 'osd_recovery_max_active': 10, 'osd_recovery_max_single_start': 10}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', 'failed to encode map', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '9ef43722ad94ea688fa7f70daab6dd41fac6ee53', 'cluster': 'ceph'}}, {'exec': {'mon.a': ['ceph config set mgr mgr/balancer/log_level debug', 'ceph balancer status', 'ceph osd set-require-min-compat-client reef', 'ceph balancer mode upmap-read', 'ceph balancer on', 'ceph balancer status', 'ceph balancer status detail']}}, {'thrashosds': {'aggressive_pg_num_changes': False, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'exec': {'client.0': ['sudo ceph osd pool create low_tier 4']}}, {'rados': {'clients': ['client.0'], 'dedup_chunk_algo': 'fastcdc', 'dedup_chunk_size': '131072', 'enable_dedup': True, 'low_tier_pool': 'low_tier', 'objects': 50, 'op_weights': {'read': 100, 'rollback': 10, 'set_chunk': 30, 'snap_create': 10, 'snap_remove': 10, 'tier_evict': 10, 'tier_flush': 5, 'tier_promote': 10, 'write': 50}, 'ops': 1500, 'set_chunk': True}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-19_08:22:49', 'tube': 'smithi', 'user': 'rfriedma', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-19T11:28:49.460 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-19T11:28:49.488 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-19T11:28:49.494 DEBUG:teuthology.orchestra.run.smithi038:> sudo install -d -m0777 -- /var/run/ceph 2024-03-19T11:28:49.537 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -d -m0777 -- /var/run/ceph 2024-03-19T11:28:49.550 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-19T11:28:49.550 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/scratch_devs of=/dev/stdout 2024-03-19T11:28:49.599 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-19T11:28:49.600 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_1 2024-03-19T11:28:49.648 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-19T11:28:49.648 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T11:28:49.648 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 856 Links: 1 2024-03-19T11:28:49.648 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T11:28:49.648 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-03-19 11:23:34.411520739 +0000 2024-03-19T11:28:49.648 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-03-19 11:23:34.135523201 +0000 2024-03-19T11:28:49.648 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-03-19 11:23:34.135523201 +0000 2024-03-19T11:28:49.648 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: - 2024-03-19T11:28:49.649 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-19T11:28:49.702 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-03-19T11:28:49.703 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-03-19T11:28:49.703 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000591566 s, 865 kB/s 2024-03-19T11:28:49.704 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-19T11:28:49.754 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_2 2024-03-19T11:28:49.800 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-19T11:28:49.800 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T11:28:49.800 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 873 Links: 1 2024-03-19T11:28:49.800 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T11:28:49.801 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-03-19 11:23:34.931516099 +0000 2024-03-19T11:28:49.801 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-03-19 11:23:34.663518491 +0000 2024-03-19T11:28:49.801 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-03-19 11:23:34.663518491 +0000 2024-03-19T11:28:49.801 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: - 2024-03-19T11:28:49.801 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-19T11:28:49.854 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-03-19T11:28:49.854 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-03-19T11:28:49.855 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000504417 s, 1.0 MB/s 2024-03-19T11:28:49.856 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-19T11:28:49.905 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_3 2024-03-19T11:28:49.952 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-19T11:28:49.953 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T11:28:49.953 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 890 Links: 1 2024-03-19T11:28:49.953 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T11:28:49.953 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-03-19 11:23:35.439511568 +0000 2024-03-19T11:28:49.953 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-03-19 11:23:35.179513887 +0000 2024-03-19T11:28:49.953 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-03-19 11:23:35.179513887 +0000 2024-03-19T11:28:49.953 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: - 2024-03-19T11:28:49.953 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-19T11:28:50.006 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-03-19T11:28:50.007 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-03-19T11:28:50.007 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000509652 s, 1.0 MB/s 2024-03-19T11:28:50.008 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-19T11:28:50.061 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_4 2024-03-19T11:28:50.108 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-19T11:28:50.108 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T11:28:50.108 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 912 Links: 1 2024-03-19T11:28:50.108 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T11:28:50.108 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-03-19 11:23:35.699509249 +0000 2024-03-19T11:28:50.108 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-03-19 11:23:35.695509285 +0000 2024-03-19T11:28:50.108 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-03-19 11:23:35.695509285 +0000 2024-03-19T11:28:50.108 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: - 2024-03-19T11:28:50.109 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-19T11:28:50.162 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-03-19T11:28:50.163 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-03-19T11:28:50.163 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000565354 s, 906 kB/s 2024-03-19T11:28:50.164 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-19T11:28:50.213 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-19T11:28:50.213 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-19T11:28:50.213 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/scratch_devs of=/dev/stdout 2024-03-19T11:28:50.219 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-19T11:28:50.219 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_1 2024-03-19T11:28:50.267 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-19T11:28:50.267 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T11:28:50.267 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 878 Links: 1 2024-03-19T11:28:50.267 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T11:28:50.267 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-03-19 11:23:27.771779771 +0000 2024-03-19T11:28:50.267 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-03-19 11:23:27.767779808 +0000 2024-03-19T11:28:50.267 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-03-19 11:23:27.767779808 +0000 2024-03-19T11:28:50.267 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2024-03-19T11:28:50.268 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-19T11:28:50.321 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-03-19T11:28:50.321 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-03-19T11:28:50.321 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000368964 s, 1.4 MB/s 2024-03-19T11:28:50.322 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-19T11:28:50.371 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_2 2024-03-19T11:28:50.418 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-19T11:28:50.419 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T11:28:50.419 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 895 Links: 1 2024-03-19T11:28:50.419 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T11:28:50.419 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-03-19 11:23:28.579772555 +0000 2024-03-19T11:28:50.419 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-03-19 11:23:28.295775091 +0000 2024-03-19T11:28:50.419 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-03-19 11:23:28.295775091 +0000 2024-03-19T11:28:50.419 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2024-03-19T11:28:50.419 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-19T11:28:50.472 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-03-19T11:28:50.473 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-03-19T11:28:50.473 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000436056 s, 1.2 MB/s 2024-03-19T11:28:50.474 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-19T11:28:50.524 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_3 2024-03-19T11:28:50.574 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-19T11:28:50.575 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T11:28:50.575 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 916 Links: 1 2024-03-19T11:28:50.575 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T11:28:50.575 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-03-19 11:23:29.075768125 +0000 2024-03-19T11:28:50.575 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-03-19 11:23:28.811770483 +0000 2024-03-19T11:28:50.575 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-03-19 11:23:28.811770483 +0000 2024-03-19T11:28:50.575 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2024-03-19T11:28:50.575 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-19T11:28:50.632 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-03-19T11:28:50.633 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-03-19T11:28:50.633 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000426399 s, 1.2 MB/s 2024-03-19T11:28:50.634 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-19T11:28:50.683 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_4 2024-03-19T11:28:50.730 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-19T11:28:50.730 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T11:28:50.731 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 928 Links: 1 2024-03-19T11:28:50.731 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T11:28:50.731 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-03-19 11:23:29.595763480 +0000 2024-03-19T11:28:50.731 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-03-19 11:23:29.303766087 +0000 2024-03-19T11:28:50.731 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-03-19 11:23:29.303766087 +0000 2024-03-19T11:28:50.731 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2024-03-19T11:28:50.731 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-19T11:28:50.784 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-03-19T11:28:50.785 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-03-19T11:28:50.785 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000362216 s, 1.4 MB/s 2024-03-19T11:28:50.786 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-19T11:28:50.836 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-19T11:28:50.836 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi038.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi172.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-19T11:28:50.836 INFO:tasks.ceph:Generating config... 2024-03-19T11:28:50.837 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-19T11:28:50.837 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-19T11:28:50.837 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2024-03-19T11:28:50.837 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-19T11:28:50.837 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-19T11:28:50.838 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-19T11:28:50.838 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-19T11:28:50.838 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-19T11:28:50.838 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-19T11:28:50.838 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-19T11:28:50.838 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-19T11:28:50.838 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-19T11:28:50.838 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-19T11:28:50.838 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-19T11:28:50.838 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-19T11:28:50.838 INFO:tasks.ceph:[osd] osd debug reject backfill probability = 0.3 2024-03-19T11:28:50.838 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-19T11:28:50.838 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-19T11:28:50.838 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-19T11:28:50.838 INFO:tasks.ceph:[osd] osd heartbeat use min delay socket = True 2024-03-19T11:28:50.839 INFO:tasks.ceph:[osd] osd inject bad map crc probability = 0.1 2024-03-19T11:28:50.839 INFO:tasks.ceph:[osd] osd max backfills = 3 2024-03-19T11:28:50.839 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-19T11:28:50.839 INFO:tasks.ceph:[osd] osd max scrubs = 5 2024-03-19T11:28:50.839 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-19T11:28:50.839 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-19T11:28:50.839 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-19T11:28:50.839 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-19T11:28:50.839 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-19T11:28:50.839 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2024-03-19T11:28:50.839 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2024-03-19T11:28:50.839 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-19T11:28:50.839 INFO:tasks.ceph:[osd] osd snap trim sleep = 2 2024-03-19T11:28:50.839 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-19T11:28:50.839 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-19T11:28:50.840 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-19T11:28:50.840 INFO:tasks.ceph:[global] ms type = async 2024-03-19T11:28:50.840 INFO:tasks.ceph:[global] osd_pool_default_min_size = 2 2024-03-19T11:28:50.840 INFO:tasks.ceph:[global] osd_pool_default_size = 2 2024-03-19T11:28:50.840 INFO:tasks.ceph:[global] osd_recovery_max_active = 10 2024-03-19T11:28:50.840 INFO:tasks.ceph:[global] osd_recovery_max_single_start = 10 2024-03-19T11:28:50.840 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-19T11:28:50.840 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-19T11:28:50.840 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-19T11:28:50.840 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-19T11:28:50.840 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-19T11:28:50.840 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-19T11:28:50.840 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-19T11:28:50.840 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-19T11:28:50.840 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-19T11:28:50.841 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-19T11:28:50.841 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-19T11:28:50.841 INFO:tasks.ceph:Setting up mon.a... 2024-03-19T11:28:50.841 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-19T11:28:50.876 INFO:teuthology.orchestra.run.smithi038.stdout:creating /etc/ceph/ceph.keyring 2024-03-19T11:28:50.889 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-19T11:28:50.970 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-19T11:28:51.028 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.38'), ('mon.c', '[v2:172.21.15.38:3301,v1:172.21.15.38:6790]'), ('mon.b', '172.21.15.172')] 2024-03-19T11:28:51.028 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.38,[v2:172.21.15.38:3301,v1:172.21.15.38:6790],172.21.15.172', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2, 'osd_recovery_max_active': 10, 'osd_recovery_max_single_start': 10}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd heartbeat use min delay socket': True, 'osd inject bad map crc probability': 0.1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd max scrubs': 5, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-19T11:28:51.029 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-19T11:28:51.029 DEBUG:teuthology.orchestra.run.smithi038:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-19T11:28:51.076 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.38 --addv c '[v2:172.21.15.38:3301,v1:172.21.15.38:6790]' --add b 172.21.15.172 --print /home/ubuntu/cephtest/ceph.monmap 2024-03-19T11:28:51.162 INFO:teuthology.orchestra.run.smithi038.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-19T11:28:51.163 INFO:teuthology.orchestra.run.smithi038.stdout:monmaptool: generated fsid 246bcf68-c431-4aa7-b787-5e9b2d9519d6 2024-03-19T11:28:51.163 INFO:teuthology.orchestra.run.smithi038.stdout:setting min_mon_release = quincy 2024-03-19T11:28:51.163 INFO:teuthology.orchestra.run.smithi038.stdout:epoch 0 2024-03-19T11:28:51.163 INFO:teuthology.orchestra.run.smithi038.stdout:fsid 246bcf68-c431-4aa7-b787-5e9b2d9519d6 2024-03-19T11:28:51.163 INFO:teuthology.orchestra.run.smithi038.stdout:last_changed 2024-03-19T11:28:51.175628+0000 2024-03-19T11:28:51.163 INFO:teuthology.orchestra.run.smithi038.stdout:created 2024-03-19T11:28:51.175628+0000 2024-03-19T11:28:51.163 INFO:teuthology.orchestra.run.smithi038.stdout:min_mon_release 17 (quincy) 2024-03-19T11:28:51.163 INFO:teuthology.orchestra.run.smithi038.stdout:election_strategy: 1 2024-03-19T11:28:51.163 INFO:teuthology.orchestra.run.smithi038.stdout:0: [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon.a 2024-03-19T11:28:51.163 INFO:teuthology.orchestra.run.smithi038.stdout:1: [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] mon.b 2024-03-19T11:28:51.163 INFO:teuthology.orchestra.run.smithi038.stdout:2: [v2:172.21.15.38:3301/0,v1:172.21.15.38:6790/0] mon.c 2024-03-19T11:28:51.163 INFO:teuthology.orchestra.run.smithi038.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-19T11:28:51.164 DEBUG:teuthology.orchestra.run.smithi038:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-19T11:28:51.212 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 246bcf68-c431-4aa7-b787-5e9b2d9519d6... 2024-03-19T11:28:51.213 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-19T11:28:51.257 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-19T11:28:51.285 INFO:teuthology.orchestra.run.smithi038.stdout:[global] 2024-03-19T11:28:51.285 INFO:teuthology.orchestra.run.smithi038.stdout: chdir = "" 2024-03-19T11:28:51.285 INFO:teuthology.orchestra.run.smithi038.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-19T11:28:51.285 INFO:teuthology.orchestra.run.smithi038.stdout: auth supported = cephx 2024-03-19T11:28:51.285 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:28:51.285 INFO:teuthology.orchestra.run.smithi038.stdout: filestore xattr use omap = true 2024-03-19T11:28:51.285 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:28:51.285 INFO:teuthology.orchestra.run.smithi038.stdout: mon clock drift allowed = 1.000 2024-03-19T11:28:51.286 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:28:51.286 INFO:teuthology.orchestra.run.smithi038.stdout: osd crush chooseleaf type = 0 2024-03-19T11:28:51.286 INFO:teuthology.orchestra.run.smithi038.stdout: auth debug = true 2024-03-19T11:28:51.286 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:28:51.286 INFO:teuthology.orchestra.run.smithi038.stdout: ms die on old message = true 2024-03-19T11:28:51.286 INFO:teuthology.orchestra.run.smithi038.stdout: ms die on bug = true 2024-03-19T11:28:51.286 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:28:51.286 INFO:teuthology.orchestra.run.smithi038.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-19T11:28:51.286 INFO:teuthology.orchestra.run.smithi038.stdout: mon pg warn max object skew = 0 2024-03-19T11:28:51.286 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:28:51.286 INFO:teuthology.orchestra.run.smithi038.stdout: # disable pg_autoscaler by default for new pools 2024-03-19T11:28:51.286 INFO:teuthology.orchestra.run.smithi038.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-19T11:28:51.287 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:28:51.287 INFO:teuthology.orchestra.run.smithi038.stdout: osd pool default size = 2 2024-03-19T11:28:51.287 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:28:51.287 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd allow primary affinity = true 2024-03-19T11:28:51.287 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd allow pg remap = true 2024-03-19T11:28:51.287 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on legacy crush tunables = false 2024-03-19T11:28:51.287 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on crush straw calc version zero = false 2024-03-19T11:28:51.287 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on no sortbitwise = false 2024-03-19T11:28:51.287 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on osd down out interval zero = false 2024-03-19T11:28:51.287 INFO:teuthology.orchestra.run.smithi038.stdout: mon warn on too few osds = false 2024-03-19T11:28:51.287 INFO:teuthology.orchestra.run.smithi038.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-19T11:28:51.287 INFO:teuthology.orchestra.run.smithi038.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-19T11:28:51.287 INFO:teuthology.orchestra.run.smithi038.stdout: mon_allow_pool_size_one = true 2024-03-19T11:28:51.287 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:28:51.287 INFO:teuthology.orchestra.run.smithi038.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-19T11:28:51.288 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:28:51.288 INFO:teuthology.orchestra.run.smithi038.stdout: osd default data pool replay window = 5 2024-03-19T11:28:51.288 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:28:51.288 INFO:teuthology.orchestra.run.smithi038.stdout: mon allow pool delete = true 2024-03-19T11:28:51.288 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:28:51.288 INFO:teuthology.orchestra.run.smithi038.stdout: mon cluster log file level = debug 2024-03-19T11:28:51.288 INFO:teuthology.orchestra.run.smithi038.stdout: debug asserts on shutdown = true 2024-03-19T11:28:51.288 INFO:teuthology.orchestra.run.smithi038.stdout: mon health detail to clog = false 2024-03-19T11:28:51.288 INFO:teuthology.orchestra.run.smithi038.stdout: mon host = "172.21.15.38,[v2:172.21.15.38:3301,v1:172.21.15.38:6790],172.21.15.172" 2024-03-19T11:28:51.288 INFO:teuthology.orchestra.run.smithi038.stdout: mon client directed command retry = 5 2024-03-19T11:28:51.288 INFO:teuthology.orchestra.run.smithi038.stdout: mon election default strategy = 1 2024-03-19T11:28:51.288 INFO:teuthology.orchestra.run.smithi038.stdout: ms inject socket failures = 5000 2024-03-19T11:28:51.288 INFO:teuthology.orchestra.run.smithi038.stdout: ms type = async 2024-03-19T11:28:51.288 INFO:teuthology.orchestra.run.smithi038.stdout: osd_pool_default_min_size = 2 2024-03-19T11:28:51.289 INFO:teuthology.orchestra.run.smithi038.stdout: osd_pool_default_size = 2 2024-03-19T11:28:51.289 INFO:teuthology.orchestra.run.smithi038.stdout: osd_recovery_max_active = 10 2024-03-19T11:28:51.289 INFO:teuthology.orchestra.run.smithi038.stdout: osd_recovery_max_single_start = 10 2024-03-19T11:28:51.289 INFO:teuthology.orchestra.run.smithi038.stdout: fsid = 246bcf68-c431-4aa7-b787-5e9b2d9519d6 2024-03-19T11:28:51.289 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:28:51.289 INFO:teuthology.orchestra.run.smithi038.stdout:[osd] 2024-03-19T11:28:51.289 INFO:teuthology.orchestra.run.smithi038.stdout: osd journal size = 100 2024-03-19T11:28:51.289 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:28:51.289 INFO:teuthology.orchestra.run.smithi038.stdout: osd scrub load threshold = 5.0 2024-03-19T11:28:51.289 INFO:teuthology.orchestra.run.smithi038.stdout: osd scrub max interval = 120 2024-03-19T11:28:51.289 INFO:teuthology.orchestra.run.smithi038.stdout: osd mclock profile = high_recovery_ops 2024-03-19T11:28:51.289 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:28:51.289 INFO:teuthology.orchestra.run.smithi038.stdout: osd recover clone overlap = true 2024-03-19T11:28:51.289 INFO:teuthology.orchestra.run.smithi038.stdout: osd recovery max chunk = 1048576 2024-03-19T11:28:51.290 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:28:51.290 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug shutdown = true 2024-03-19T11:28:51.290 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug op order = true 2024-03-19T11:28:51.290 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug verify stray on activate = true 2024-03-19T11:28:51.290 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug trim objects = true 2024-03-19T11:28:51.290 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:28:51.290 INFO:teuthology.orchestra.run.smithi038.stdout: osd open classes on start = true 2024-03-19T11:28:51.290 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug pg log writeout = true 2024-03-19T11:28:51.290 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:28:51.290 INFO:teuthology.orchestra.run.smithi038.stdout: osd deep scrub update digest min age = 30 2024-03-19T11:28:51.290 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:28:51.290 INFO:teuthology.orchestra.run.smithi038.stdout: osd map max advance = 10 2024-03-19T11:28:51.290 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:28:51.290 INFO:teuthology.orchestra.run.smithi038.stdout: journal zero on create = true 2024-03-19T11:28:51.291 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:28:51.291 INFO:teuthology.orchestra.run.smithi038.stdout: filestore ondisk finisher threads = 3 2024-03-19T11:28:51.291 INFO:teuthology.orchestra.run.smithi038.stdout: filestore apply finisher threads = 3 2024-03-19T11:28:51.291 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:28:51.291 INFO:teuthology.orchestra.run.smithi038.stdout: bdev debug aio = true 2024-03-19T11:28:51.291 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug misdirected ops = true 2024-03-19T11:28:51.291 INFO:teuthology.orchestra.run.smithi038.stdout: debug monc = 20 2024-03-19T11:28:51.291 INFO:teuthology.orchestra.run.smithi038.stdout: bluestore block size = 96636764160 2024-03-19T11:28:51.291 INFO:teuthology.orchestra.run.smithi038.stdout: bluestore compression algorithm = zstd 2024-03-19T11:28:51.291 INFO:teuthology.orchestra.run.smithi038.stdout: bluestore compression mode = aggressive 2024-03-19T11:28:51.291 INFO:teuthology.orchestra.run.smithi038.stdout: bluestore fsck on mount = True 2024-03-19T11:28:51.291 INFO:teuthology.orchestra.run.smithi038.stdout: bluestore zero block detection = True 2024-03-19T11:28:51.291 INFO:teuthology.orchestra.run.smithi038.stdout: debug bluefs = 20 2024-03-19T11:28:51.291 INFO:teuthology.orchestra.run.smithi038.stdout: debug bluestore = 20 2024-03-19T11:28:51.292 INFO:teuthology.orchestra.run.smithi038.stdout: debug ms = 1 2024-03-19T11:28:51.292 INFO:teuthology.orchestra.run.smithi038.stdout: debug osd = 20 2024-03-19T11:28:51.292 INFO:teuthology.orchestra.run.smithi038.stdout: debug rocksdb = 10 2024-03-19T11:28:51.292 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd backfillfull_ratio = 0.85 2024-03-19T11:28:51.292 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd full ratio = 0.9 2024-03-19T11:28:51.292 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd nearfull ratio = 0.8 2024-03-19T11:28:51.292 INFO:teuthology.orchestra.run.smithi038.stdout: osd blocked scrub grace period = 3600 2024-03-19T11:28:51.292 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug reject backfill probability = 0.3 2024-03-19T11:28:51.292 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug verify cached snaps = True 2024-03-19T11:28:51.292 INFO:teuthology.orchestra.run.smithi038.stdout: osd debug verify missing on start = True 2024-03-19T11:28:51.292 INFO:teuthology.orchestra.run.smithi038.stdout: osd failsafe full ratio = 0.95 2024-03-19T11:28:51.292 INFO:teuthology.orchestra.run.smithi038.stdout: osd heartbeat use min delay socket = True 2024-03-19T11:28:51.292 INFO:teuthology.orchestra.run.smithi038.stdout: osd inject bad map crc probability = 0.1 2024-03-19T11:28:51.292 INFO:teuthology.orchestra.run.smithi038.stdout: osd max backfills = 3 2024-03-19T11:28:51.292 INFO:teuthology.orchestra.run.smithi038.stdout: osd max markdown count = 1000 2024-03-19T11:28:51.293 INFO:teuthology.orchestra.run.smithi038.stdout: osd max scrubs = 5 2024-03-19T11:28:51.293 INFO:teuthology.orchestra.run.smithi038.stdout: osd mclock override recovery settings = True 2024-03-19T11:28:51.293 INFO:teuthology.orchestra.run.smithi038.stdout: osd objectstore = bluestore 2024-03-19T11:28:51.293 INFO:teuthology.orchestra.run.smithi038.stdout: osd op queue = debug_random 2024-03-19T11:28:51.293 INFO:teuthology.orchestra.run.smithi038.stdout: osd op queue cut off = debug_random 2024-03-19T11:28:51.293 INFO:teuthology.orchestra.run.smithi038.stdout: osd scrub min interval = 60 2024-03-19T11:28:51.293 INFO:teuthology.orchestra.run.smithi038.stdout: osd shutdown pgref assert = True 2024-03-19T11:28:51.293 INFO:teuthology.orchestra.run.smithi038.stdout: osd snap trim sleep = 2 2024-03-19T11:28:51.293 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:28:51.293 INFO:teuthology.orchestra.run.smithi038.stdout:[mgr] 2024-03-19T11:28:51.293 INFO:teuthology.orchestra.run.smithi038.stdout: debug ms = 1 2024-03-19T11:28:51.293 INFO:teuthology.orchestra.run.smithi038.stdout: debug mgr = 20 2024-03-19T11:28:51.293 INFO:teuthology.orchestra.run.smithi038.stdout: debug mon = 20 2024-03-19T11:28:51.293 INFO:teuthology.orchestra.run.smithi038.stdout: debug auth = 20 2024-03-19T11:28:51.294 INFO:teuthology.orchestra.run.smithi038.stdout: mon reweight min pgs per osd = 4 2024-03-19T11:28:51.294 INFO:teuthology.orchestra.run.smithi038.stdout: mon reweight min bytes per osd = 10 2024-03-19T11:28:51.294 INFO:teuthology.orchestra.run.smithi038.stdout: mgr/telemetry/nag = false 2024-03-19T11:28:51.294 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:28:51.294 INFO:teuthology.orchestra.run.smithi038.stdout:[mon] 2024-03-19T11:28:51.294 INFO:teuthology.orchestra.run.smithi038.stdout: debug ms = 1 2024-03-19T11:28:51.294 INFO:teuthology.orchestra.run.smithi038.stdout: debug mon = 20 2024-03-19T11:28:51.294 INFO:teuthology.orchestra.run.smithi038.stdout: debug paxos = 20 2024-03-19T11:28:51.294 INFO:teuthology.orchestra.run.smithi038.stdout: debug auth = 20 2024-03-19T11:28:51.294 INFO:teuthology.orchestra.run.smithi038.stdout: mon data avail warn = 5 2024-03-19T11:28:51.294 INFO:teuthology.orchestra.run.smithi038.stdout: mon mgr mkfs grace = 240 2024-03-19T11:28:51.294 INFO:teuthology.orchestra.run.smithi038.stdout: mon reweight min pgs per osd = 4 2024-03-19T11:28:51.294 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd reporter subtree level = osd 2024-03-19T11:28:51.295 INFO:teuthology.orchestra.run.smithi038.stdout: mon osd prime pg temp = true 2024-03-19T11:28:51.295 INFO:teuthology.orchestra.run.smithi038.stdout: mon reweight min bytes per osd = 10 2024-03-19T11:28:51.295 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:28:51.295 INFO:teuthology.orchestra.run.smithi038.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-19T11:28:51.295 INFO:teuthology.orchestra.run.smithi038.stdout: auth mon ticket ttl = 660 # 11m 2024-03-19T11:28:51.295 INFO:teuthology.orchestra.run.smithi038.stdout: auth service ticket ttl = 240 # 4m 2024-03-19T11:28:51.295 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:28:51.295 INFO:teuthology.orchestra.run.smithi038.stdout: # don't complain about insecure global_id in the test suite 2024-03-19T11:28:51.295 INFO:teuthology.orchestra.run.smithi038.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-19T11:28:51.295 INFO:teuthology.orchestra.run.smithi038.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-19T11:28:51.295 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:28:51.295 INFO:teuthology.orchestra.run.smithi038.stdout: # 1m isn't quite enough 2024-03-19T11:28:51.295 INFO:teuthology.orchestra.run.smithi038.stdout: mon_down_mkfs_grace = 2m 2024-03-19T11:28:51.295 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:28:51.295 INFO:teuthology.orchestra.run.smithi038.stdout: mon_warn_on_filestore_osds = false 2024-03-19T11:28:51.296 INFO:teuthology.orchestra.run.smithi038.stdout: mon min osdmap epochs = 50 2024-03-19T11:28:51.296 INFO:teuthology.orchestra.run.smithi038.stdout: mon osdmap full prune interval = 2 2024-03-19T11:28:51.296 INFO:teuthology.orchestra.run.smithi038.stdout: mon osdmap full prune min = 15 2024-03-19T11:28:51.296 INFO:teuthology.orchestra.run.smithi038.stdout: mon osdmap full prune txsize = 2 2024-03-19T11:28:51.296 INFO:teuthology.orchestra.run.smithi038.stdout: mon scrub interval = 300 2024-03-19T11:28:51.296 INFO:teuthology.orchestra.run.smithi038.stdout: paxos service trim min = 10 2024-03-19T11:28:51.296 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:28:51.296 INFO:teuthology.orchestra.run.smithi038.stdout:[client] 2024-03-19T11:28:51.296 INFO:teuthology.orchestra.run.smithi038.stdout: rgw cache enabled = true 2024-03-19T11:28:51.296 INFO:teuthology.orchestra.run.smithi038.stdout: rgw enable ops log = true 2024-03-19T11:28:51.296 INFO:teuthology.orchestra.run.smithi038.stdout: rgw enable usage log = true 2024-03-19T11:28:51.296 INFO:teuthology.orchestra.run.smithi038.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-19T11:28:51.296 INFO:teuthology.orchestra.run.smithi038.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-19T11:28:51.296 INFO:teuthology.orchestra.run.smithi038.stdout:[mon.a] 2024-03-19T11:28:51.296 INFO:teuthology.orchestra.run.smithi038.stdout:[mon.c] 2024-03-19T11:28:51.296 INFO:teuthology.orchestra.run.smithi038.stdout:[mon.b] 2024-03-19T11:28:51.298 INFO:teuthology.orchestra.run.smithi172.stdout:[global] 2024-03-19T11:28:51.298 INFO:teuthology.orchestra.run.smithi172.stdout: chdir = "" 2024-03-19T11:28:51.298 INFO:teuthology.orchestra.run.smithi172.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-19T11:28:51.298 INFO:teuthology.orchestra.run.smithi172.stdout: auth supported = cephx 2024-03-19T11:28:51.298 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-19T11:28:51.298 INFO:teuthology.orchestra.run.smithi172.stdout: filestore xattr use omap = true 2024-03-19T11:28:51.298 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-19T11:28:51.298 INFO:teuthology.orchestra.run.smithi172.stdout: mon clock drift allowed = 1.000 2024-03-19T11:28:51.299 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-19T11:28:51.299 INFO:teuthology.orchestra.run.smithi172.stdout: osd crush chooseleaf type = 0 2024-03-19T11:28:51.299 INFO:teuthology.orchestra.run.smithi172.stdout: auth debug = true 2024-03-19T11:28:51.299 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-19T11:28:51.299 INFO:teuthology.orchestra.run.smithi172.stdout: ms die on old message = true 2024-03-19T11:28:51.299 INFO:teuthology.orchestra.run.smithi172.stdout: ms die on bug = true 2024-03-19T11:28:51.299 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-19T11:28:51.299 INFO:teuthology.orchestra.run.smithi172.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-19T11:28:51.299 INFO:teuthology.orchestra.run.smithi172.stdout: mon pg warn max object skew = 0 2024-03-19T11:28:51.299 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-19T11:28:51.299 INFO:teuthology.orchestra.run.smithi172.stdout: # disable pg_autoscaler by default for new pools 2024-03-19T11:28:51.299 INFO:teuthology.orchestra.run.smithi172.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-19T11:28:51.299 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-19T11:28:51.299 INFO:teuthology.orchestra.run.smithi172.stdout: osd pool default size = 2 2024-03-19T11:28:51.299 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-19T11:28:51.300 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd allow primary affinity = true 2024-03-19T11:28:51.300 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd allow pg remap = true 2024-03-19T11:28:51.300 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on legacy crush tunables = false 2024-03-19T11:28:51.300 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on crush straw calc version zero = false 2024-03-19T11:28:51.300 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on no sortbitwise = false 2024-03-19T11:28:51.300 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on osd down out interval zero = false 2024-03-19T11:28:51.300 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on too few osds = false 2024-03-19T11:28:51.300 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-19T11:28:51.300 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-19T11:28:51.300 INFO:teuthology.orchestra.run.smithi172.stdout: mon_allow_pool_size_one = true 2024-03-19T11:28:51.300 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-19T11:28:51.300 INFO:teuthology.orchestra.run.smithi172.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-19T11:28:51.300 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-19T11:28:51.300 INFO:teuthology.orchestra.run.smithi172.stdout: osd default data pool replay window = 5 2024-03-19T11:28:51.300 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-19T11:28:51.301 INFO:teuthology.orchestra.run.smithi172.stdout: mon allow pool delete = true 2024-03-19T11:28:51.301 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-19T11:28:51.301 INFO:teuthology.orchestra.run.smithi172.stdout: mon cluster log file level = debug 2024-03-19T11:28:51.301 INFO:teuthology.orchestra.run.smithi172.stdout: debug asserts on shutdown = true 2024-03-19T11:28:51.301 INFO:teuthology.orchestra.run.smithi172.stdout: mon health detail to clog = false 2024-03-19T11:28:51.301 INFO:teuthology.orchestra.run.smithi172.stdout: mon host = "172.21.15.38,[v2:172.21.15.38:3301,v1:172.21.15.38:6790],172.21.15.172" 2024-03-19T11:28:51.301 INFO:teuthology.orchestra.run.smithi172.stdout: mon client directed command retry = 5 2024-03-19T11:28:51.301 INFO:teuthology.orchestra.run.smithi172.stdout: mon election default strategy = 1 2024-03-19T11:28:51.301 INFO:teuthology.orchestra.run.smithi172.stdout: ms inject socket failures = 5000 2024-03-19T11:28:51.301 INFO:teuthology.orchestra.run.smithi172.stdout: ms type = async 2024-03-19T11:28:51.301 INFO:teuthology.orchestra.run.smithi172.stdout: osd_pool_default_min_size = 2 2024-03-19T11:28:51.301 INFO:teuthology.orchestra.run.smithi172.stdout: osd_pool_default_size = 2 2024-03-19T11:28:51.301 INFO:teuthology.orchestra.run.smithi172.stdout: osd_recovery_max_active = 10 2024-03-19T11:28:51.301 INFO:teuthology.orchestra.run.smithi172.stdout: osd_recovery_max_single_start = 10 2024-03-19T11:28:51.301 INFO:teuthology.orchestra.run.smithi172.stdout: fsid = 246bcf68-c431-4aa7-b787-5e9b2d9519d6 2024-03-19T11:28:51.301 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-19T11:28:51.302 INFO:teuthology.orchestra.run.smithi172.stdout:[osd] 2024-03-19T11:28:51.302 INFO:teuthology.orchestra.run.smithi172.stdout: osd journal size = 100 2024-03-19T11:28:51.302 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-19T11:28:51.302 INFO:teuthology.orchestra.run.smithi172.stdout: osd scrub load threshold = 5.0 2024-03-19T11:28:51.302 INFO:teuthology.orchestra.run.smithi172.stdout: osd scrub max interval = 120 2024-03-19T11:28:51.302 INFO:teuthology.orchestra.run.smithi172.stdout: osd mclock profile = high_recovery_ops 2024-03-19T11:28:51.302 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-19T11:28:51.302 INFO:teuthology.orchestra.run.smithi172.stdout: osd recover clone overlap = true 2024-03-19T11:28:51.302 INFO:teuthology.orchestra.run.smithi172.stdout: osd recovery max chunk = 1048576 2024-03-19T11:28:51.302 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-19T11:28:51.302 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug shutdown = true 2024-03-19T11:28:51.302 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug op order = true 2024-03-19T11:28:51.302 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug verify stray on activate = true 2024-03-19T11:28:51.302 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug trim objects = true 2024-03-19T11:28:51.303 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-19T11:28:51.303 INFO:teuthology.orchestra.run.smithi172.stdout: osd open classes on start = true 2024-03-19T11:28:51.303 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug pg log writeout = true 2024-03-19T11:28:51.303 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-19T11:28:51.303 INFO:teuthology.orchestra.run.smithi172.stdout: osd deep scrub update digest min age = 30 2024-03-19T11:28:51.303 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-19T11:28:51.303 INFO:teuthology.orchestra.run.smithi172.stdout: osd map max advance = 10 2024-03-19T11:28:51.303 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-19T11:28:51.303 INFO:teuthology.orchestra.run.smithi172.stdout: journal zero on create = true 2024-03-19T11:28:51.303 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-19T11:28:51.303 INFO:teuthology.orchestra.run.smithi172.stdout: filestore ondisk finisher threads = 3 2024-03-19T11:28:51.303 INFO:teuthology.orchestra.run.smithi172.stdout: filestore apply finisher threads = 3 2024-03-19T11:28:51.303 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-19T11:28:51.303 INFO:teuthology.orchestra.run.smithi172.stdout: bdev debug aio = true 2024-03-19T11:28:51.304 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug misdirected ops = true 2024-03-19T11:28:51.304 INFO:teuthology.orchestra.run.smithi172.stdout: debug monc = 20 2024-03-19T11:28:51.304 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore block size = 96636764160 2024-03-19T11:28:51.304 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore compression algorithm = zstd 2024-03-19T11:28:51.304 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore compression mode = aggressive 2024-03-19T11:28:51.304 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore fsck on mount = True 2024-03-19T11:28:51.304 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore zero block detection = True 2024-03-19T11:28:51.304 INFO:teuthology.orchestra.run.smithi172.stdout: debug bluefs = 20 2024-03-19T11:28:51.304 INFO:teuthology.orchestra.run.smithi172.stdout: debug bluestore = 20 2024-03-19T11:28:51.304 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2024-03-19T11:28:51.304 INFO:teuthology.orchestra.run.smithi172.stdout: debug osd = 20 2024-03-19T11:28:51.304 INFO:teuthology.orchestra.run.smithi172.stdout: debug rocksdb = 10 2024-03-19T11:28:51.304 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd backfillfull_ratio = 0.85 2024-03-19T11:28:51.304 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd full ratio = 0.9 2024-03-19T11:28:51.304 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd nearfull ratio = 0.8 2024-03-19T11:28:51.304 INFO:teuthology.orchestra.run.smithi172.stdout: osd blocked scrub grace period = 3600 2024-03-19T11:28:51.305 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug reject backfill probability = 0.3 2024-03-19T11:28:51.305 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug verify cached snaps = True 2024-03-19T11:28:51.305 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug verify missing on start = True 2024-03-19T11:28:51.305 INFO:teuthology.orchestra.run.smithi172.stdout: osd failsafe full ratio = 0.95 2024-03-19T11:28:51.305 INFO:teuthology.orchestra.run.smithi172.stdout: osd heartbeat use min delay socket = True 2024-03-19T11:28:51.305 INFO:teuthology.orchestra.run.smithi172.stdout: osd inject bad map crc probability = 0.1 2024-03-19T11:28:51.305 INFO:teuthology.orchestra.run.smithi172.stdout: osd max backfills = 3 2024-03-19T11:28:51.305 INFO:teuthology.orchestra.run.smithi172.stdout: osd max markdown count = 1000 2024-03-19T11:28:51.305 INFO:teuthology.orchestra.run.smithi172.stdout: osd max scrubs = 5 2024-03-19T11:28:51.305 INFO:teuthology.orchestra.run.smithi172.stdout: osd mclock override recovery settings = True 2024-03-19T11:28:51.305 INFO:teuthology.orchestra.run.smithi172.stdout: osd objectstore = bluestore 2024-03-19T11:28:51.305 INFO:teuthology.orchestra.run.smithi172.stdout: osd op queue = debug_random 2024-03-19T11:28:51.305 INFO:teuthology.orchestra.run.smithi172.stdout: osd op queue cut off = debug_random 2024-03-19T11:28:51.305 INFO:teuthology.orchestra.run.smithi172.stdout: osd scrub min interval = 60 2024-03-19T11:28:51.306 INFO:teuthology.orchestra.run.smithi172.stdout: osd shutdown pgref assert = True 2024-03-19T11:28:51.306 INFO:teuthology.orchestra.run.smithi172.stdout: osd snap trim sleep = 2 2024-03-19T11:28:51.306 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-19T11:28:51.306 INFO:teuthology.orchestra.run.smithi172.stdout:[mgr] 2024-03-19T11:28:51.306 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2024-03-19T11:28:51.306 INFO:teuthology.orchestra.run.smithi172.stdout: debug mgr = 20 2024-03-19T11:28:51.306 INFO:teuthology.orchestra.run.smithi172.stdout: debug mon = 20 2024-03-19T11:28:51.306 INFO:teuthology.orchestra.run.smithi172.stdout: debug auth = 20 2024-03-19T11:28:51.306 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min pgs per osd = 4 2024-03-19T11:28:51.306 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min bytes per osd = 10 2024-03-19T11:28:51.306 INFO:teuthology.orchestra.run.smithi172.stdout: mgr/telemetry/nag = false 2024-03-19T11:28:51.306 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-19T11:28:51.306 INFO:teuthology.orchestra.run.smithi172.stdout:[mon] 2024-03-19T11:28:51.306 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2024-03-19T11:28:51.306 INFO:teuthology.orchestra.run.smithi172.stdout: debug mon = 20 2024-03-19T11:28:51.307 INFO:teuthology.orchestra.run.smithi172.stdout: debug paxos = 20 2024-03-19T11:28:51.307 INFO:teuthology.orchestra.run.smithi172.stdout: debug auth = 20 2024-03-19T11:28:51.307 INFO:teuthology.orchestra.run.smithi172.stdout: mon data avail warn = 5 2024-03-19T11:28:51.307 INFO:teuthology.orchestra.run.smithi172.stdout: mon mgr mkfs grace = 240 2024-03-19T11:28:51.307 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min pgs per osd = 4 2024-03-19T11:28:51.307 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd reporter subtree level = osd 2024-03-19T11:28:51.307 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd prime pg temp = true 2024-03-19T11:28:51.307 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min bytes per osd = 10 2024-03-19T11:28:51.307 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-19T11:28:51.307 INFO:teuthology.orchestra.run.smithi172.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-19T11:28:51.307 INFO:teuthology.orchestra.run.smithi172.stdout: auth mon ticket ttl = 660 # 11m 2024-03-19T11:28:51.307 INFO:teuthology.orchestra.run.smithi172.stdout: auth service ticket ttl = 240 # 4m 2024-03-19T11:28:51.307 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-19T11:28:51.307 INFO:teuthology.orchestra.run.smithi172.stdout: # don't complain about insecure global_id in the test suite 2024-03-19T11:28:51.307 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-19T11:28:51.308 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-19T11:28:51.308 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-19T11:28:51.308 INFO:teuthology.orchestra.run.smithi172.stdout: # 1m isn't quite enough 2024-03-19T11:28:51.308 INFO:teuthology.orchestra.run.smithi172.stdout: mon_down_mkfs_grace = 2m 2024-03-19T11:28:51.308 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-19T11:28:51.308 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_filestore_osds = false 2024-03-19T11:28:51.308 INFO:teuthology.orchestra.run.smithi172.stdout: mon min osdmap epochs = 50 2024-03-19T11:28:51.308 INFO:teuthology.orchestra.run.smithi172.stdout: mon osdmap full prune interval = 2 2024-03-19T11:28:51.308 INFO:teuthology.orchestra.run.smithi172.stdout: mon osdmap full prune min = 15 2024-03-19T11:28:51.308 INFO:teuthology.orchestra.run.smithi172.stdout: mon osdmap full prune txsize = 2 2024-03-19T11:28:51.308 INFO:teuthology.orchestra.run.smithi172.stdout: mon scrub interval = 300 2024-03-19T11:28:51.308 INFO:teuthology.orchestra.run.smithi172.stdout: paxos service trim min = 10 2024-03-19T11:28:51.308 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-19T11:28:51.309 INFO:teuthology.orchestra.run.smithi172.stdout:[client] 2024-03-19T11:28:51.309 INFO:teuthology.orchestra.run.smithi172.stdout: rgw cache enabled = true 2024-03-19T11:28:51.309 INFO:teuthology.orchestra.run.smithi172.stdout: rgw enable ops log = true 2024-03-19T11:28:51.309 INFO:teuthology.orchestra.run.smithi172.stdout: rgw enable usage log = true 2024-03-19T11:28:51.309 INFO:teuthology.orchestra.run.smithi172.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-19T11:28:51.309 INFO:teuthology.orchestra.run.smithi172.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-19T11:28:51.309 INFO:teuthology.orchestra.run.smithi172.stdout:[mon.a] 2024-03-19T11:28:51.309 INFO:teuthology.orchestra.run.smithi172.stdout:[mon.c] 2024-03-19T11:28:51.309 INFO:teuthology.orchestra.run.smithi172.stdout:[mon.b] 2024-03-19T11:28:51.309 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-19T11:28:51.309 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-19T11:28:51.384 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-19T11:28:51.384 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-19T11:28:51.384 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-19T11:28:51.390 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-19T11:28:51.390 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-19T11:28:51.436 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi038.front.sepia.ceph.com 2024-03-19T11:28:51.436 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-19T11:28:51.436 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-19T11:28:51.436 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-19T11:28:51.501 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-19T11:28:51.501 DEBUG:teuthology.orchestra.run.smithi038:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-19T11:28:51.548 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi172.front.sepia.ceph.com 2024-03-19T11:28:51.549 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-19T11:28:51.549 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-19T11:28:51.549 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-19T11:28:51.571 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-19T11:28:51.571 DEBUG:teuthology.orchestra.run.smithi172:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-19T11:28:51.619 INFO:tasks.ceph:Setting up mon nodes... 2024-03-19T11:28:51.619 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-19T11:28:51.619 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-19T11:28:51.664 INFO:teuthology.orchestra.run.smithi038.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-19T11:28:51.667 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-19T11:28:51.710 INFO:teuthology.orchestra.run.smithi172.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-19T11:28:51.728 INFO:tasks.ceph:Setting up mds nodes... 2024-03-19T11:28:51.728 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-19T11:28:51.728 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-19T11:28:51.764 INFO:teuthology.orchestra.run.smithi038.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-19T11:28:51.776 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-19T11:28:51.810 INFO:teuthology.orchestra.run.smithi172.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-19T11:28:51.821 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-19T11:28:51.821 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi038.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi172.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-19T11:28:51.821 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-19T11:28:51.834 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-19T11:28:51.834 INFO:tasks.ceph:role: osd.0 2024-03-19T11:28:51.835 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi038.front.sepia.ceph.com 2024-03-19T11:28:51.835 DEBUG:teuthology.orchestra.run.smithi038:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-19T11:28:51.919 INFO:teuthology.orchestra.run.smithi038.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-19T11:28:51.920 INFO:teuthology.orchestra.run.smithi038.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-19T11:28:51.920 INFO:teuthology.orchestra.run.smithi038.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-19T11:28:51.920 INFO:teuthology.orchestra.run.smithi038.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-19T11:28:51.920 INFO:teuthology.orchestra.run.smithi038.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-19T11:28:51.920 INFO:teuthology.orchestra.run.smithi038.stdout: = sunit=0 swidth=0 blks 2024-03-19T11:28:51.920 INFO:teuthology.orchestra.run.smithi038.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-19T11:28:51.920 INFO:teuthology.orchestra.run.smithi038.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-19T11:28:51.920 INFO:teuthology.orchestra.run.smithi038.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-19T11:28:51.920 INFO:teuthology.orchestra.run.smithi038.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-19T11:28:53.222 INFO:teuthology.orchestra.run.smithi038.stdout:Discarding blocks...Done. 2024-03-19T11:28:53.224 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi038.front.sepia.ceph.com -o noatime 2024-03-19T11:28:53.224 DEBUG:teuthology.orchestra.run.smithi038:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-19T11:28:53.286 DEBUG:teuthology.orchestra.run.smithi038:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-19T11:28:53.342 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: /sbin/restorecon: command not found 2024-03-19T11:28:53.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T11:28:53.343 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-19T11:28:53.399 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-19T11:28:53.400 INFO:tasks.ceph:role: osd.1 2024-03-19T11:28:53.400 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi038.front.sepia.ceph.com 2024-03-19T11:28:53.400 DEBUG:teuthology.orchestra.run.smithi038:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-19T11:28:53.484 INFO:teuthology.orchestra.run.smithi038.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-19T11:28:53.484 INFO:teuthology.orchestra.run.smithi038.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-19T11:28:53.484 INFO:teuthology.orchestra.run.smithi038.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-19T11:28:53.484 INFO:teuthology.orchestra.run.smithi038.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-19T11:28:53.484 INFO:teuthology.orchestra.run.smithi038.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-19T11:28:53.485 INFO:teuthology.orchestra.run.smithi038.stdout: = sunit=0 swidth=0 blks 2024-03-19T11:28:53.485 INFO:teuthology.orchestra.run.smithi038.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-19T11:28:53.485 INFO:teuthology.orchestra.run.smithi038.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-19T11:28:53.485 INFO:teuthology.orchestra.run.smithi038.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-19T11:28:53.485 INFO:teuthology.orchestra.run.smithi038.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-19T11:28:54.793 INFO:teuthology.orchestra.run.smithi038.stdout:Discarding blocks...Done. 2024-03-19T11:28:54.794 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi038.front.sepia.ceph.com -o noatime 2024-03-19T11:28:54.795 DEBUG:teuthology.orchestra.run.smithi038:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-19T11:28:54.857 DEBUG:teuthology.orchestra.run.smithi038:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-19T11:28:54.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T11:28:54.910 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: /sbin/restorecon: command not found 2024-03-19T11:28:54.911 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-19T11:28:54.968 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-19T11:28:54.968 INFO:tasks.ceph:role: osd.2 2024-03-19T11:28:54.968 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi038.front.sepia.ceph.com 2024-03-19T11:28:54.968 DEBUG:teuthology.orchestra.run.smithi038:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-19T11:28:55.054 INFO:teuthology.orchestra.run.smithi038.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-19T11:28:55.055 INFO:teuthology.orchestra.run.smithi038.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-19T11:28:55.055 INFO:teuthology.orchestra.run.smithi038.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-19T11:28:55.055 INFO:teuthology.orchestra.run.smithi038.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-19T11:28:55.055 INFO:teuthology.orchestra.run.smithi038.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-19T11:28:55.055 INFO:teuthology.orchestra.run.smithi038.stdout: = sunit=0 swidth=0 blks 2024-03-19T11:28:55.055 INFO:teuthology.orchestra.run.smithi038.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-19T11:28:55.055 INFO:teuthology.orchestra.run.smithi038.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-19T11:28:55.055 INFO:teuthology.orchestra.run.smithi038.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-19T11:28:55.055 INFO:teuthology.orchestra.run.smithi038.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-19T11:28:56.353 INFO:teuthology.orchestra.run.smithi038.stdout:Discarding blocks...Done. 2024-03-19T11:28:56.356 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi038.front.sepia.ceph.com -o noatime 2024-03-19T11:28:56.356 DEBUG:teuthology.orchestra.run.smithi038:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-19T11:28:56.422 DEBUG:teuthology.orchestra.run.smithi038:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-19T11:28:56.471 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: /sbin/restorecon: command not found 2024-03-19T11:28:56.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T11:28:56.473 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-19T11:28:56.528 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-19T11:28:56.528 INFO:tasks.ceph:role: osd.3 2024-03-19T11:28:56.528 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi038.front.sepia.ceph.com 2024-03-19T11:28:56.528 DEBUG:teuthology.orchestra.run.smithi038:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-19T11:28:56.613 INFO:teuthology.orchestra.run.smithi038.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-19T11:28:56.613 INFO:teuthology.orchestra.run.smithi038.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-19T11:28:56.613 INFO:teuthology.orchestra.run.smithi038.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-19T11:28:56.613 INFO:teuthology.orchestra.run.smithi038.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-19T11:28:56.613 INFO:teuthology.orchestra.run.smithi038.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-19T11:28:56.613 INFO:teuthology.orchestra.run.smithi038.stdout: = sunit=0 swidth=0 blks 2024-03-19T11:28:56.613 INFO:teuthology.orchestra.run.smithi038.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-19T11:28:56.613 INFO:teuthology.orchestra.run.smithi038.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-19T11:28:56.614 INFO:teuthology.orchestra.run.smithi038.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-19T11:28:56.614 INFO:teuthology.orchestra.run.smithi038.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-19T11:28:57.906 INFO:teuthology.orchestra.run.smithi038.stdout:Discarding blocks...Done. 2024-03-19T11:28:57.908 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi038.front.sepia.ceph.com -o noatime 2024-03-19T11:28:57.908 DEBUG:teuthology.orchestra.run.smithi038:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-19T11:28:57.971 DEBUG:teuthology.orchestra.run.smithi038:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-19T11:28:58.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T11:28:58.025 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: /sbin/restorecon: command not found 2024-03-19T11:28:58.025 DEBUG:teuthology.orchestra.run.smithi038:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-19T11:28:58.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-19T11:28:58.141+0000 7f07ebfff7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-19T11:28:58.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-19T11:28:58.145+0000 7f07ebfff7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-19T11:28:58.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-19T11:28:58.145+0000 7f07ebfff7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-19T11:28:58.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-19T11:28:58.145+0000 7f07ebfff7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-19T11:28:58.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-19T11:28:58.145+0000 7f07ebfff7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-19T11:28:58.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-19T11:28:58.145+0000 7f07ebfff7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-19T11:29:01.166 DEBUG:teuthology.orchestra.run.smithi038:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-19T11:29:01.181 DEBUG:teuthology.orchestra.run.smithi038:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-19T11:29:01.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-19T11:29:01.269+0000 7ffa18dcf7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-19T11:29:01.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-19T11:29:01.273+0000 7ffa18dcf7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-19T11:29:01.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-19T11:29:01.273+0000 7ffa18dcf7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-19T11:29:01.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-19T11:29:01.273+0000 7ffa18dcf7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-19T11:29:01.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-19T11:29:01.273+0000 7ffa18dcf7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-19T11:29:01.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-19T11:29:01.273+0000 7ffa18dcf7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-19T11:29:04.263 DEBUG:teuthology.orchestra.run.smithi038:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-19T11:29:04.325 DEBUG:teuthology.orchestra.run.smithi038:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-19T11:29:04.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-19T11:29:04.417+0000 7fd94011f7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-19T11:29:04.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-19T11:29:04.417+0000 7fd94011f7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-19T11:29:04.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-19T11:29:04.417+0000 7fd94011f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-19T11:29:04.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-19T11:29:04.417+0000 7fd94011f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-19T11:29:04.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-19T11:29:04.417+0000 7fd94011f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-19T11:29:04.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-19T11:29:04.417+0000 7fd94011f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-19T11:29:07.387 DEBUG:teuthology.orchestra.run.smithi038:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-19T11:29:07.443 DEBUG:teuthology.orchestra.run.smithi038:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-19T11:29:07.528 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-19T11:29:07.537+0000 7fdab6eb87c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-19T11:29:07.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-19T11:29:07.537+0000 7fdab6eb87c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-19T11:29:07.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-19T11:29:07.537+0000 7fdab6eb87c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-19T11:29:07.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-19T11:29:07.537+0000 7fdab6eb87c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-19T11:29:07.529 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-19T11:29:07.537+0000 7fdab6eb87c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-19T11:29:07.530 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-19T11:29:07.537+0000 7fdab6eb87c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-19T11:29:10.527 DEBUG:teuthology.orchestra.run.smithi038:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-19T11:29:10.584 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-19T11:29:10.601 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-19T11:29:10.601 INFO:tasks.ceph:role: osd.4 2024-03-19T11:29:10.601 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi172.front.sepia.ceph.com 2024-03-19T11:29:10.601 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-19T11:29:10.661 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-19T11:29:10.661 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-19T11:29:10.661 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-19T11:29:10.661 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-19T11:29:10.661 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-19T11:29:10.662 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2024-03-19T11:29:10.662 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-19T11:29:10.662 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-19T11:29:10.662 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-19T11:29:10.662 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-19T11:29:11.106 INFO:teuthology.orchestra.run.smithi172.stdout:Discarding blocks...Done. 2024-03-19T11:29:11.108 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2024-03-19T11:29:11.108 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-19T11:29:11.196 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-19T11:29:11.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T11:29:11.207 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: /sbin/restorecon: command not found 2024-03-19T11:29:11.208 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-19T11:29:11.260 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-19T11:29:11.260 INFO:tasks.ceph:role: osd.5 2024-03-19T11:29:11.260 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi172.front.sepia.ceph.com 2024-03-19T11:29:11.260 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-19T11:29:11.321 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-19T11:29:11.321 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-19T11:29:11.322 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-19T11:29:11.322 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-19T11:29:11.322 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-19T11:29:11.322 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2024-03-19T11:29:11.322 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-19T11:29:11.322 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-19T11:29:11.322 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-19T11:29:11.322 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-19T11:29:11.789 INFO:teuthology.orchestra.run.smithi172.stdout:Discarding blocks...Done. 2024-03-19T11:29:11.790 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2024-03-19T11:29:11.790 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-19T11:29:11.944 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-19T11:29:11.955 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: /sbin/restorecon: command not found 2024-03-19T11:29:11.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T11:29:11.956 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-19T11:29:12.009 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-19T11:29:12.009 INFO:tasks.ceph:role: osd.6 2024-03-19T11:29:12.009 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi172.front.sepia.ceph.com 2024-03-19T11:29:12.009 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-19T11:29:12.073 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-19T11:29:12.074 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-19T11:29:12.074 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-19T11:29:12.074 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-19T11:29:12.074 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-19T11:29:12.074 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2024-03-19T11:29:12.074 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-19T11:29:12.075 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-19T11:29:12.075 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-19T11:29:12.075 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-19T11:29:12.550 INFO:teuthology.orchestra.run.smithi172.stdout:Discarding blocks...Done. 2024-03-19T11:29:12.552 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2024-03-19T11:29:12.552 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-19T11:29:12.698 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-19T11:29:12.709 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: /sbin/restorecon: command not found 2024-03-19T11:29:12.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T11:29:12.710 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-19T11:29:12.765 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-19T11:29:12.766 INFO:tasks.ceph:role: osd.7 2024-03-19T11:29:12.766 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi172.front.sepia.ceph.com 2024-03-19T11:29:12.766 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-19T11:29:12.828 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-19T11:29:12.828 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-19T11:29:12.828 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-19T11:29:12.828 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-19T11:29:12.828 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-19T11:29:12.828 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2024-03-19T11:29:12.828 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-19T11:29:12.828 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-19T11:29:12.829 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-19T11:29:12.829 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-19T11:29:13.251 INFO:teuthology.orchestra.run.smithi172.stdout:Discarding blocks...Done. 2024-03-19T11:29:13.253 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2024-03-19T11:29:13.254 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-19T11:29:13.351 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-19T11:29:13.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T11:29:13.362 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: /sbin/restorecon: command not found 2024-03-19T11:29:13.362 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-19T11:29:13.480 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-19T11:29:13.488+0000 7ff3c1f6b7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-19T11:29:13.485 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-19T11:29:13.492+0000 7ff3c1f6b7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-19T11:29:13.485 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-19T11:29:13.492+0000 7ff3c1f6b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-19T11:29:13.485 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-19T11:29:13.492+0000 7ff3c1f6b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-19T11:29:13.485 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-19T11:29:13.492+0000 7ff3c1f6b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-19T11:29:13.487 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-19T11:29:13.496+0000 7ff3c1f6b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-19T11:29:16.465 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-19T11:29:16.479 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-19T11:29:16.560 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-19T11:29:16.568+0000 7f50a49d87c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-19T11:29:16.561 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-19T11:29:16.568+0000 7f50a49d87c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-19T11:29:16.561 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-19T11:29:16.568+0000 7f50a49d87c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-19T11:29:16.561 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-19T11:29:16.568+0000 7f50a49d87c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-19T11:29:16.561 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-19T11:29:16.568+0000 7f50a49d87c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-19T11:29:16.561 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-19T11:29:16.572+0000 7f50a49d87c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-19T11:29:19.544 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-19T11:29:19.598 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-19T11:29:19.680 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-19T11:29:19.688+0000 7f1d873347c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-19T11:29:19.680 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-19T11:29:19.688+0000 7f1d873347c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-19T11:29:19.681 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-19T11:29:19.688+0000 7f1d873347c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-19T11:29:19.681 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-19T11:29:19.688+0000 7f1d873347c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-19T11:29:19.681 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-19T11:29:19.688+0000 7f1d873347c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-19T11:29:19.681 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-19T11:29:19.692+0000 7f1d873347c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-19T11:29:22.653 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-19T11:29:22.710 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-19T11:29:22.791 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-19T11:29:22.800+0000 7fef63a287c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-19T11:29:22.793 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-19T11:29:22.800+0000 7fef63a287c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-19T11:29:22.793 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-19T11:29:22.800+0000 7fef63a287c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-19T11:29:22.793 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-19T11:29:22.800+0000 7fef63a287c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-19T11:29:22.793 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-19T11:29:22.800+0000 7fef63a287c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-19T11:29:22.794 INFO:teuthology.orchestra.run.smithi172.stderr:2024-03-19T11:29:22.800+0000 7fef63a287c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-19T11:29:25.776 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-19T11:29:25.830 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-19T11:29:25.830 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-19T11:29:25.830 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-19T11:29:25.844 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-19T11:29:25.844 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-19T11:29:25.899 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-19T11:29:25.900 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-19T11:29:25.955 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-19T11:29:25.955 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-19T11:29:26.011 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-19T11:29:26.011 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-19T11:29:26.067 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-19T11:29:26.067 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-19T11:29:26.080 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-19T11:29:26.081 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-19T11:29:26.137 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-19T11:29:26.137 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-19T11:29:26.193 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-19T11:29:26.193 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-19T11:29:26.248 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-19T11:29:26.248 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-19T11:29:26.300 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-19T11:29:26.301 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-19T11:29:26.306 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-19T11:29:26.306 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-19T11:29:26.350 INFO:tasks.ceph:Adding keys to all mons... 2024-03-19T11:29:26.350 DEBUG:teuthology.orchestra.run.smithi038:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-19T11:29:26.352 DEBUG:teuthology.orchestra.run.smithi172:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-19T11:29:26.395 INFO:teuthology.orchestra.run.smithi038.stdout:[mgr.y] 2024-03-19T11:29:26.396 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQDzdvllZ25kKBAA4l4PErSB1USxL9a8SkrQaQ== 2024-03-19T11:29:26.396 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.0] 2024-03-19T11:29:26.396 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQD6dvllfgzICBAAvWpznrhysQfwQ2WUwpXuYA== 2024-03-19T11:29:26.396 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.1] 2024-03-19T11:29:26.396 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQD9dvllZ2dzEBAAYk7JfL7iz76HHOhPcA0luQ== 2024-03-19T11:29:26.396 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.2] 2024-03-19T11:29:26.396 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQAAd/llrMUcGRAA5PlupJANy9ey4Ly3EHEECg== 2024-03-19T11:29:26.396 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.3] 2024-03-19T11:29:26.396 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQADd/llo0ZPIBAAjhUlNL/e5toIE3ClYV14sg== 2024-03-19T11:29:26.396 INFO:teuthology.orchestra.run.smithi038.stdout:[mgr.x] 2024-03-19T11:29:26.396 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQDzdvllPgAqKxAAwKNJfhTfgA4VdJvEh7Criw== 2024-03-19T11:29:26.396 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.4] 2024-03-19T11:29:26.396 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQAJd/llJgR5HRAAJAk2D9o3cQ069HKu0gyLCg== 2024-03-19T11:29:26.396 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.5] 2024-03-19T11:29:26.397 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQAMd/lleQQ9IhAA2CpwWjwCneFIUyqtUlMEQg== 2024-03-19T11:29:26.397 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.6] 2024-03-19T11:29:26.397 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQAPd/llHLhjKRAAoGBbvwgUTJUFjdXg6FC/qQ== 2024-03-19T11:29:26.397 INFO:teuthology.orchestra.run.smithi038.stdout:[osd.7] 2024-03-19T11:29:26.397 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQASd/ll+moLMBAAwcd/it8TGPr1J+JWJYYr0g== 2024-03-19T11:29:26.397 INFO:teuthology.orchestra.run.smithi038.stdout:[client.0] 2024-03-19T11:29:26.397 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQDzdvllbYVWLhAApTZlsO+OkCSYUZR9fYE8gw== 2024-03-19T11:29:26.397 INFO:teuthology.orchestra.run.smithi038.stdout:[client.1] 2024-03-19T11:29:26.397 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQDzdvllgzojMRAAEihLhJ7Nn/CGhSsxUA99DA== 2024-03-19T11:29:26.403 INFO:teuthology.orchestra.run.smithi172.stdout:[mgr.y] 2024-03-19T11:29:26.403 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQDzdvllZ25kKBAA4l4PErSB1USxL9a8SkrQaQ== 2024-03-19T11:29:26.403 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.0] 2024-03-19T11:29:26.403 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQD6dvllfgzICBAAvWpznrhysQfwQ2WUwpXuYA== 2024-03-19T11:29:26.403 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.1] 2024-03-19T11:29:26.404 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQD9dvllZ2dzEBAAYk7JfL7iz76HHOhPcA0luQ== 2024-03-19T11:29:26.404 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.2] 2024-03-19T11:29:26.404 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQAAd/llrMUcGRAA5PlupJANy9ey4Ly3EHEECg== 2024-03-19T11:29:26.404 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.3] 2024-03-19T11:29:26.404 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQADd/llo0ZPIBAAjhUlNL/e5toIE3ClYV14sg== 2024-03-19T11:29:26.404 INFO:teuthology.orchestra.run.smithi172.stdout:[mgr.x] 2024-03-19T11:29:26.404 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQDzdvllPgAqKxAAwKNJfhTfgA4VdJvEh7Criw== 2024-03-19T11:29:26.404 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.4] 2024-03-19T11:29:26.404 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQAJd/llJgR5HRAAJAk2D9o3cQ069HKu0gyLCg== 2024-03-19T11:29:26.404 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.5] 2024-03-19T11:29:26.404 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQAMd/lleQQ9IhAA2CpwWjwCneFIUyqtUlMEQg== 2024-03-19T11:29:26.404 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.6] 2024-03-19T11:29:26.404 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQAPd/llHLhjKRAAoGBbvwgUTJUFjdXg6FC/qQ== 2024-03-19T11:29:26.404 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.7] 2024-03-19T11:29:26.405 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQASd/ll+moLMBAAwcd/it8TGPr1J+JWJYYr0g== 2024-03-19T11:29:26.405 INFO:teuthology.orchestra.run.smithi172.stdout:[client.0] 2024-03-19T11:29:26.405 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQDzdvllbYVWLhAApTZlsO+OkCSYUZR9fYE8gw== 2024-03-19T11:29:26.405 INFO:teuthology.orchestra.run.smithi172.stdout:[client.1] 2024-03-19T11:29:26.405 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQDzdvllgzojMRAAEihLhJ7Nn/CGhSsxUA99DA== 2024-03-19T11:29:26.406 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-19T11:29:26.444 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-19T11:29:26.493 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T11:29:26.536 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T11:29:26.578 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T11:29:26.620 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T11:29:26.660 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T11:29:26.704 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T11:29:26.748 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T11:29:26.750 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T11:29:26.825 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-19T11:29:26.836 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-19T11:29:26.907 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T11:29:26.909 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T11:29:26.991 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T11:29:26.994 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T11:29:27.073 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T11:29:27.080 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T11:29:27.156 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T11:29:27.164 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-19T11:29:27.242 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-19T11:29:27.244 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-19T11:29:27.331 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-19T11:29:27.333 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-19T11:29:27.420 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-19T11:29:27.420 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-19T11:29:27.433 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-19T11:29:27.550 DEBUG:teuthology.orchestra.run.smithi038:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-19T11:29:27.563 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-19T11:29:27.619 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-19T11:29:27.740 DEBUG:teuthology.orchestra.run.smithi038:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-19T11:29:27.751 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-19T11:29:27.766 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-19T11:29:27.858 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-19T11:29:27.868 DEBUG:teuthology.orchestra.run.smithi038:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-19T11:29:27.870 DEBUG:teuthology.orchestra.run.smithi172:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-19T11:29:27.913 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-19T11:29:27.913 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-19T11:29:27.913 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-19T11:29:27.916 INFO:tasks.ceph.mon.a:Started 2024-03-19T11:29:27.916 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-19T11:29:27.916 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-19T11:29:27.919 INFO:tasks.ceph.mon.c:Started 2024-03-19T11:29:27.919 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-19T11:29:27.919 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-19T11:29:27.958 INFO:tasks.ceph.mon.b:Started 2024-03-19T11:29:27.958 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-19T11:29:27.959 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-19T11:29:27.959 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-19T11:29:27.961 INFO:tasks.ceph.mgr.y:Started 2024-03-19T11:29:27.961 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-19T11:29:27.961 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-19T11:29:27.964 INFO:tasks.ceph.mgr.x:Started 2024-03-19T11:29:27.964 INFO:tasks.ceph:Setting crush tunables to default 2024-03-19T11:29:27.964 DEBUG:teuthology.orchestra.run.smithi038:> sudo ceph --cluster ceph osd crush tunables default 2024-03-19T11:29:33.426 INFO:teuthology.orchestra.run.smithi038.stderr:adjusted tunables profile to default 2024-03-19T11:29:33.446 INFO:tasks.ceph:check_enable_crimson: False 2024-03-19T11:29:33.446 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-19T11:29:33.446 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-19T11:29:33.446 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-19T11:29:33.458 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-19T11:29:33.459 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-19T11:29:33.514 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-19T11:29:33.514 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-19T11:29:33.562 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-03-19T11:29:33.569+0000 7f0fb4eef180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-19T11:29:33.574 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-19T11:29:33.574 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-19T11:29:33.634 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-19T11:29:33.634 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-19T11:29:33.639 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-19T11:29:33.648+0000 7f10c7c32180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-19T11:29:33.648 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-19T11:29:33.649 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-19T11:29:33.705 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-19T11:29:33.705 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-19T11:29:33.761 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-19T11:29:33.761 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-19T11:29:33.817 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new 3bb2d2be-5253-4da0-acf3-85a114dc3801 0 2024-03-19T11:29:34.079 INFO:teuthology.orchestra.run.smithi172.stdout:0 2024-03-19T11:29:34.098 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new e1c926b2-e0be-4ccc-b8a5-d80d0b584ead 1 2024-03-19T11:29:34.106 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-03-19T11:29:34.113+0000 7f0fb4eef180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-19T11:29:34.160 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-19T11:29:34.168+0000 7f10c7c32180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-19T11:29:34.293 INFO:teuthology.orchestra.run.smithi172.stdout:1 2024-03-19T11:29:34.309 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new 61262219-4f8a-4733-af9d-b5208a17c0ea 2 2024-03-19T11:29:34.319 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-03-19T11:29:34.329+0000 7f0fb4eef180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-19T11:29:34.374 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-19T11:29:34.384+0000 7f10c7c32180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-19T11:29:34.511 INFO:teuthology.orchestra.run.smithi172.stdout:2 2024-03-19T11:29:34.526 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new a31f8d1b-4e38-4ca3-b251-0877002d7235 3 2024-03-19T11:29:34.720 INFO:teuthology.orchestra.run.smithi172.stdout:3 2024-03-19T11:29:34.737 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new 6746723d-f59c-40ba-857e-d8c954d2675f 4 2024-03-19T11:29:34.888 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-03-19T11:29:34.897+0000 7f0fb4eef180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-19T11:29:34.928 INFO:teuthology.orchestra.run.smithi172.stdout:4 2024-03-19T11:29:34.944 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new f59afdf8-2277-4583-af68-78136f1c5d62 5 2024-03-19T11:29:34.945 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-03-19T11:29:34.953+0000 7f0fb4eef180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-19T11:29:34.961 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-19T11:29:34.968+0000 7f10c7c32180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-19T11:29:35.000 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-03-19T11:29:35.009+0000 7f0fb4eef180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-19T11:29:35.017 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-19T11:29:35.024+0000 7f10c7c32180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-19T11:29:35.070 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-19T11:29:35.080+0000 7f10c7c32180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-19T11:29:35.081 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-03-19T11:29:35.089+0000 7f0fb4eef180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-19T11:29:35.135 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-03-19T11:29:35.145+0000 7f0fb4eef180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-19T11:29:35.151 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-19T11:29:35.160+0000 7f10c7c32180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-19T11:29:35.156 INFO:teuthology.orchestra.run.smithi172.stdout:5 2024-03-19T11:29:35.173 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new f91d8c47-4e10-40c6-a02d-66d6b72a657b 6 2024-03-19T11:29:35.206 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-19T11:29:35.216+0000 7f10c7c32180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-19T11:29:35.216 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-03-19T11:29:35.225+0000 7f0fb4eef180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-19T11:29:35.271 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-03-19T11:29:35.281+0000 7f0fb4eef180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-19T11:29:35.287 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-19T11:29:35.296+0000 7f10c7c32180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-19T11:29:35.342 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-19T11:29:35.352+0000 7f10c7c32180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-19T11:29:35.371 INFO:teuthology.orchestra.run.smithi172.stdout:6 2024-03-19T11:29:35.387 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster ceph osd new 46fc052a-5708-42ff-9d5a-bed7a0fc457a 7 2024-03-19T11:29:35.432 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-03-19T11:29:35.441+0000 7f0fb4eef180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-19T11:29:35.496 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-19T11:29:35.504+0000 7f10c7c32180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-19T11:29:35.497 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-03-19T11:29:35.505+0000 7f0fb4eef180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-19T11:29:35.560 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-19T11:29:35.568+0000 7f10c7c32180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-19T11:29:35.626 INFO:teuthology.orchestra.run.smithi172.stdout:7 2024-03-19T11:29:35.643 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-19T11:29:35.644 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-19T11:29:35.648 INFO:tasks.ceph.osd.0:Started 2024-03-19T11:29:35.648 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-19T11:29:35.648 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-19T11:29:35.651 INFO:tasks.ceph.osd.1:Started 2024-03-19T11:29:35.651 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-19T11:29:35.651 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-19T11:29:35.654 INFO:tasks.ceph.osd.2:Started 2024-03-19T11:29:35.654 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-19T11:29:35.654 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-19T11:29:35.657 INFO:tasks.ceph.osd.3:Started 2024-03-19T11:29:35.657 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-19T11:29:35.657 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-19T11:29:35.659 INFO:tasks.ceph.osd.4:Started 2024-03-19T11:29:35.660 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-19T11:29:35.660 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-19T11:29:35.662 INFO:tasks.ceph.osd.5:Started 2024-03-19T11:29:35.662 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-19T11:29:35.662 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-19T11:29:35.664 INFO:tasks.ceph.osd.6:Started 2024-03-19T11:29:35.664 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-19T11:29:35.664 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-19T11:29:35.666 INFO:tasks.ceph.osd.7:Started 2024-03-19T11:29:35.667 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:29:35.902 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:29:35.903 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":10,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:29:35.632327+0000","last_up_change":"0.000000","last_in_change":"2024-03-19T11:29:35.632327+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:29:35.918 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-19T11:29:35.918 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-19T11:29:36.161 INFO:tasks.ceph.mgr.y.smithi038.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-03-19T11:29:36.162 INFO:tasks.ceph.mgr.y.smithi038.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-03-19T11:29:36.162 INFO:tasks.ceph.mgr.y.smithi038.stderr: from numpy import show_config as show_numpy_config 2024-03-19T11:29:36.164 INFO:tasks.ceph.mgr.x.smithi172.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-03-19T11:29:36.165 INFO:tasks.ceph.mgr.x.smithi172.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-03-19T11:29:36.165 INFO:tasks.ceph.mgr.x.smithi172.stderr: from numpy import show_config as show_numpy_config 2024-03-19T11:29:36.202 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-03-19T11:29:36.209+0000 7f0fb4eef180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-19T11:29:36.246 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-19T11:29:36.256+0000 7f10c7c32180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-19T11:29:36.266 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:29:36.276+0000 7f1843a817c0 -1 Falling back to public interface 2024-03-19T11:29:36.292 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:29:36.301+0000 7f671f60c7c0 -1 Falling back to public interface 2024-03-19T11:29:36.292 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:29:36.301+0000 7fa5ecf557c0 -1 Falling back to public interface 2024-03-19T11:29:36.292 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:29:36.301+0000 7f70d36597c0 -1 Falling back to public interface 2024-03-19T11:29:36.298 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:29:36.308+0000 7fb18e3f47c0 -1 Falling back to public interface 2024-03-19T11:29:36.298 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:29:36.308+0000 7f1449f327c0 -1 Falling back to public interface 2024-03-19T11:29:36.305 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:29:36.316+0000 7f92500a27c0 -1 Falling back to public interface 2024-03-19T11:29:36.308 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:29:36.317+0000 7f6a9005c7c0 -1 Falling back to public interface 2024-03-19T11:29:36.354 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-03-19T11:29:36.361+0000 7f0fb4eef180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-19T11:29:36.426 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-19T11:29:36.436+0000 7f10c7c32180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-19T11:29:36.513 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-03-19T11:29:36.521+0000 7f0fb4eef180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-19T11:29:36.578 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-03-19T11:29:36.589+0000 7f0fb4eef180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-19T11:29:36.579 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-19T11:29:36.588+0000 7f10c7c32180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-19T11:29:36.642 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-19T11:29:36.652+0000 7f10c7c32180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-19T11:29:36.719 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-03-19T11:29:36.729+0000 7f0fb4eef180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-19T11:29:36.724 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-19T11:29:36.775 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-19T11:29:36.784+0000 7f10c7c32180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-19T11:29:36.902 INFO:teuthology.misc.health.smithi038.stdout: 2024-03-19T11:29:36.902 INFO:teuthology.misc.health.smithi038.stdout:{"epoch":10,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:29:35.632327+0000","last_up_change":"0.000000","last_in_change":"2024-03-19T11:29:35.632327+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:29:36.921 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-19T11:29:37.380 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-03-19T11:29:37.389+0000 7f0fb4eef180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-19T11:29:37.399 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-19T11:29:37.408+0000 7f10c7c32180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-19T11:29:37.458 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-03-19T11:29:37.465+0000 7f0fb4eef180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-19T11:29:37.474 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-19T11:29:37.484+0000 7f10c7c32180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-19T11:29:37.647 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-03-19T11:29:37.656+0000 7f0fb4eef180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-19T11:29:37.655 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-19T11:29:37.664+0000 7f10c7c32180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-19T11:29:37.724 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-03-19T11:29:37.732+0000 7f0fb4eef180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-19T11:29:37.728 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-19T11:29:37.736+0000 7f10c7c32180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-19T11:29:38.016 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-19T11:29:38.024+0000 7f10c7c32180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-19T11:29:38.027 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-03-19T11:29:38.036+0000 7f0fb4eef180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-19T11:29:38.077 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-19T11:29:38.088+0000 7f10c7c32180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-19T11:29:38.092 INFO:tasks.ceph.mgr.y.smithi038.stderr:2024-03-19T11:29:38.100+0000 7f0fb4eef180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-19T11:29:39.122 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-19T11:29:39.132+0000 7f106f079640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-19T11:29:39.123 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-19T11:29:39.132+0000 7f106f079640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-19T11:29:39.917 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:29:39.928+0000 7fb18e3f47c0 -1 osd.4 0 log_to_monitors true 2024-03-19T11:29:40.070 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:29:40.080+0000 7f1843a817c0 -1 osd.6 0 log_to_monitors true 2024-03-19T11:29:40.083 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:29:40.092+0000 7f70d36597c0 -1 osd.0 0 log_to_monitors true 2024-03-19T11:29:40.201 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:29:40.212+0000 7f92500a27c0 -1 osd.7 0 log_to_monitors true 2024-03-19T11:29:40.202 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:29:40.208+0000 7f671f60c7c0 -1 osd.2 0 log_to_monitors true 2024-03-19T11:29:40.207 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:29:40.216+0000 7f1449f327c0 -1 osd.5 0 log_to_monitors true 2024-03-19T11:29:40.221 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:29:40.228+0000 7f6a9005c7c0 -1 osd.1 0 log_to_monitors true 2024-03-19T11:29:40.240 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:29:40.248+0000 7fa5ecf557c0 -1 osd.3 0 log_to_monitors true 2024-03-19T11:29:41.147 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:29:41.156+0000 7f183fa2a640 -1 osd.6 0 waiting for initial osdmap 2024-03-19T11:29:41.153 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:29:41.160+0000 7f70cf602640 -1 osd.0 0 waiting for initial osdmap 2024-03-19T11:29:41.163 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:29:41.172+0000 7f70cb43f640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T11:29:41.174 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:29:41.184+0000 7f183b867640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T11:29:41.842 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:29:41.852+0000 7fb18a39d640 -1 osd.4 0 waiting for initial osdmap 2024-03-19T11:29:41.843 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:29:41.852+0000 7fb17dd72640 -1 osd.4 0 handle_osd_map injecting map crc failure 2024-03-19T11:29:41.862 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:29:41.872+0000 7fb1821d2640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T11:29:42.154 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:29:42.164+0000 7f1445edb640 -1 osd.5 0 waiting for initial osdmap 2024-03-19T11:29:42.155 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:29:42.164+0000 7fa5e8efe640 -1 osd.3 0 waiting for initial osdmap 2024-03-19T11:29:42.159 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:29:42.168+0000 7fa5e08db640 -1 osd.3 0 handle_osd_map injecting map crc failure 2024-03-19T11:29:42.160 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:29:42.168+0000 7f143d8b8640 -1 osd.5 0 handle_osd_map injecting map crc failure 2024-03-19T11:29:42.160 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:29:42.168+0000 7f671b5b5640 -1 osd.2 0 waiting for initial osdmap 2024-03-19T11:29:42.161 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:29:42.168+0000 7f6712f92640 -1 osd.2 0 handle_osd_map injecting map crc failure 2024-03-19T11:29:42.174 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:29:42.184+0000 7f1441d18640 -1 osd.5 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T11:29:42.179 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:29:42.188+0000 7f67173f2640 -1 osd.2 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T11:29:42.191 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:29:42.200+0000 7fa5e4d3b640 -1 osd.3 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T11:29:42.780 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:29:42.788+0000 7f6a8c005640 -1 osd.1 0 waiting for initial osdmap 2024-03-19T11:29:42.781 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:29:42.788+0000 7f6a7f9da640 -1 osd.1 0 handle_osd_map injecting map crc failure 2024-03-19T11:29:42.807 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:29:42.816+0000 7f6a83e3a640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T11:29:42.856 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:29:42.864+0000 7f924c04b640 -1 osd.7 0 waiting for initial osdmap 2024-03-19T11:29:42.856 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:29:42.864+0000 7f923fa20640 -1 osd.7 0 handle_osd_map injecting map crc failure 2024-03-19T11:29:42.882 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:29:42.892+0000 7f9243e80640 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T11:29:43.729 DEBUG:teuthology.orchestra.run.smithi038:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-19T11:29:44.044 INFO:teuthology.misc.health.smithi038.stdout: 2024-03-19T11:29:44.044 INFO:teuthology.misc.health.smithi038.stdout:{"epoch":14,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:29:43.166618+0000","last_up_change":"2024-03-19T11:29:43.166618+0000","last_in_change":"2024-03-19T11:29:35.632327+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists","up"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:29:41.144572+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:29:40.978689+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:29:41.083084+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:29:44.062 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-19T11:29:44.062 INFO:tasks.ceph:Creating RBD pool 2024-03-19T11:29:44.062 DEBUG:teuthology.orchestra.run.smithi038:> sudo ceph --cluster ceph osd pool create rbd 8 2024-03-19T11:29:45.183 INFO:teuthology.orchestra.run.smithi038.stderr:pool 'rbd' created 2024-03-19T11:29:45.215 DEBUG:teuthology.orchestra.run.smithi038:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-03-19T11:29:46.294 INFO:teuthology.orchestra.run.smithi038.stderr:enabled application 'rbd' on pool 'rbd' 2024-03-19T11:29:46.319 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-19T11:29:46.320 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-19T11:29:46.320 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-19T11:29:46.320 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-19T11:29:46.320 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-19T11:29:46.409 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:29:46.420+0000 7f923fa20640 -1 osd.7 16 handle_osd_map injecting map crc failure 2024-03-19T11:29:46.669 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:29:46.688 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":4,"flags":0,"active_gid":4105,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":745929961},{"type":"v1","addr":"172.21.15.172:6833","nonce":745929961}]},"active_addr":"172.21.15.172:6833/745929961","active_change":"2024-03-19T11:29:38.113645+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4100,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.172:0","nonce":3321583434}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.172:0","nonce":3214331818}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.172:0","nonce":2930087180}]}]} 2024-03-19T11:29:46.695 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-19T11:29:46.695 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-19T11:29:46.696 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:29:47.000 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:29:47.000 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":17,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:29:46.303392+0000","last_up_change":"2024-03-19T11:29:43.166618+0000","last_in_change":"2024-03-19T11:29:35.632327+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists","up"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:29:41.144572+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:29:41.282904+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:29:41.176062+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:29:41.269303+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:29:40.978689+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:29:41.198288+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:29:41.083084+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:29:41.265929+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:29:47.021 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-19T11:29:47.021 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:29:47.327 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:29:47.328 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":18,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:29:47.307435+0000","last_up_change":"2024-03-19T11:29:43.166618+0000","last_in_change":"2024-03-19T11:29:35.632327+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists","up"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:29:41.144572+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:29:41.282904+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:29:41.176062+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:29:41.269303+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:29:40.978689+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:29:41.198288+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:29:41.083084+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:29:41.265929+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:29:47.347 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-19T11:29:47.347 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-19T11:29:47.348 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-19T11:29:47.348 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-19T11:29:47.348 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-19T11:29:47.349 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-19T11:29:47.349 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-19T11:29:47.349 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-19T11:29:47.629 INFO:teuthology.orchestra.run.smithi038.stdout:60129542148 2024-03-19T11:29:47.629 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-19T11:29:47.630 INFO:teuthology.orchestra.run.smithi038.stdout:60129542147 2024-03-19T11:29:47.630 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-19T11:29:47.631 INFO:teuthology.orchestra.run.smithi038.stdout:60129542148 2024-03-19T11:29:47.632 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-19T11:29:47.633 INFO:teuthology.orchestra.run.smithi038.stdout:60129542148 2024-03-19T11:29:47.633 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-19T11:29:47.637 INFO:teuthology.orchestra.run.smithi038.stdout:55834574852 2024-03-19T11:29:47.638 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-19T11:29:47.640 INFO:teuthology.orchestra.run.smithi038.stdout:60129542147 2024-03-19T11:29:47.640 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-19T11:29:47.642 INFO:teuthology.orchestra.run.smithi038.stdout:55834574851 2024-03-19T11:29:47.642 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-19T11:29:47.646 INFO:teuthology.orchestra.run.smithi038.stdout:55834574851 2024-03-19T11:29:47.646 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-19T11:29:48.127 INFO:teuthology.orchestra.run.smithi038.stdout:60129542146 2024-03-19T11:29:48.131 INFO:teuthology.orchestra.run.smithi038.stdout:60129542147 2024-03-19T11:29:48.155 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542147 got 60129542146 for osd.2 2024-03-19T11:29:48.162 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542147 for osd.7 2024-03-19T11:29:48.164 INFO:teuthology.orchestra.run.smithi038.stdout:60129542147 2024-03-19T11:29:48.171 INFO:teuthology.orchestra.run.smithi038.stdout:55834574850 2024-03-19T11:29:48.178 INFO:teuthology.orchestra.run.smithi038.stdout:60129542146 2024-03-19T11:29:48.186 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542147 for osd.3 2024-03-19T11:29:48.195 INFO:teuthology.orchestra.run.smithi038.stdout:55834574850 2024-03-19T11:29:48.195 INFO:teuthology.orchestra.run.smithi038.stdout:60129542147 2024-03-19T11:29:48.204 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574850 for osd.0 2024-03-19T11:29:48.209 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542147 got 60129542146 for osd.1 2024-03-19T11:29:48.213 INFO:teuthology.orchestra.run.smithi038.stdout:55834574850 2024-03-19T11:29:48.217 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574850 for osd.6 2024-03-19T11:29:48.220 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542147 for osd.5 2024-03-19T11:29:48.230 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574850 for osd.4 2024-03-19T11:29:49.156 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-19T11:29:49.163 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-19T11:29:49.187 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-19T11:29:49.205 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-19T11:29:49.210 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-19T11:29:49.218 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-19T11:29:49.221 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-19T11:29:49.231 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-19T11:29:49.705 INFO:teuthology.orchestra.run.smithi038.stdout:60129542147 2024-03-19T11:29:49.728 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542147 got 60129542147 for osd.2 2024-03-19T11:29:49.728 DEBUG:teuthology.parallel:result is None 2024-03-19T11:29:49.729 INFO:teuthology.orchestra.run.smithi038.stdout:60129542148 2024-03-19T11:29:49.738 INFO:teuthology.orchestra.run.smithi038.stdout:55834574851 2024-03-19T11:29:49.756 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542148 for osd.3 2024-03-19T11:29:49.757 DEBUG:teuthology.parallel:result is None 2024-03-19T11:29:49.762 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.4 2024-03-19T11:29:49.762 DEBUG:teuthology.parallel:result is None 2024-03-19T11:29:49.763 INFO:teuthology.orchestra.run.smithi038.stdout:60129542148 2024-03-19T11:29:49.769 INFO:teuthology.orchestra.run.smithi038.stdout:55834574852 2024-03-19T11:29:49.783 INFO:teuthology.orchestra.run.smithi038.stdout:55834574851 2024-03-19T11:29:49.788 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.0 2024-03-19T11:29:49.789 DEBUG:teuthology.parallel:result is None 2024-03-19T11:29:49.790 INFO:teuthology.orchestra.run.smithi038.stdout:60129542147 2024-03-19T11:29:49.792 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542148 for osd.7 2024-03-19T11:29:49.792 DEBUG:teuthology.parallel:result is None 2024-03-19T11:29:49.804 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.6 2024-03-19T11:29:49.804 INFO:teuthology.orchestra.run.smithi038.stdout:60129542148 2024-03-19T11:29:49.804 DEBUG:teuthology.parallel:result is None 2024-03-19T11:29:49.810 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542147 got 60129542147 for osd.1 2024-03-19T11:29:49.810 DEBUG:teuthology.parallel:result is None 2024-03-19T11:29:49.820 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542148 for osd.5 2024-03-19T11:29:49.820 DEBUG:teuthology.parallel:result is None 2024-03-19T11:29:49.820 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-19T11:29:49.821 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T11:29:50.161 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:29:50.161 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-03-19T11:29:50.178 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-03-19T11:29:50.130035+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":221532,"kb_used_data":1184,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754753188,"statfs":{"total":773094113280,"available":772867264512,"internally_reserved":0,"allocated":1212416,"data_stored":1463311,"data_compressed":12128,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":12751,"internal_metadata":225496625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000641"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T11:29:47.318721+0000","last_change":"2024-03-19T11:29:46.435120+0000","last_active":"2024-03-19T11:29:47.318721+0000","last_peered":"2024-03-19T11:29:47.318721+0000","last_clean":"2024-03-19T11:29:47.318721+0000","last_became_active":"2024-03-19T11:29:46.434588+0000","last_became_peered":"2024-03-19T11:29:46.434588+0000","last_unstale":"2024-03-19T11:29:47.318721+0000","last_undegraded":"2024-03-19T11:29:47.318721+0000","last_fullsized":"2024-03-19T11:29:47.318721+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:03.134473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T11:29:48.191220+0000","last_change":"2024-03-19T11:29:46.423438+0000","last_active":"2024-03-19T11:29:48.191220+0000","last_peered":"2024-03-19T11:29:48.191220+0000","last_clean":"2024-03-19T11:29:48.191220+0000","last_became_active":"2024-03-19T11:29:46.423027+0000","last_became_peered":"2024-03-19T11:29:46.423027+0000","last_unstale":"2024-03-19T11:29:48.191220+0000","last_undegraded":"2024-03-19T11:29:48.191220+0000","last_fullsized":"2024-03-19T11:29:48.191220+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:55.305083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T11:29:48.193184+0000","last_change":"2024-03-19T11:29:46.542897+0000","last_active":"2024-03-19T11:29:48.193184+0000","last_peered":"2024-03-19T11:29:48.193184+0000","last_clean":"2024-03-19T11:29:48.193184+0000","last_became_active":"2024-03-19T11:29:46.542092+0000","last_became_peered":"2024-03-19T11:29:46.542092+0000","last_unstale":"2024-03-19T11:29:48.193184+0000","last_undegraded":"2024-03-19T11:29:48.193184+0000","last_fullsized":"2024-03-19T11:29:48.193184+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:46.318698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T11:29:48.191202+0000","last_change":"2024-03-19T11:29:46.541233+0000","last_active":"2024-03-19T11:29:48.191202+0000","last_peered":"2024-03-19T11:29:48.191202+0000","last_clean":"2024-03-19T11:29:48.191202+0000","last_became_active":"2024-03-19T11:29:46.540939+0000","last_became_peered":"2024-03-19T11:29:46.540939+0000","last_unstale":"2024-03-19T11:29:48.191202+0000","last_undegraded":"2024-03-19T11:29:48.191202+0000","last_fullsized":"2024-03-19T11:29:48.191202+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:54.761098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T11:29:48.193708+0000","last_change":"2024-03-19T11:29:46.321255+0000","last_active":"2024-03-19T11:29:48.193708+0000","last_peered":"2024-03-19T11:29:48.193708+0000","last_clean":"2024-03-19T11:29:48.193708+0000","last_became_active":"2024-03-19T11:29:46.320679+0000","last_became_peered":"2024-03-19T11:29:46.320679+0000","last_unstale":"2024-03-19T11:29:48.193708+0000","last_undegraded":"2024-03-19T11:29:48.193708+0000","last_fullsized":"2024-03-19T11:29:48.193708+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:14.793563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T11:29:47.322737+0000","last_change":"2024-03-19T11:29:46.319811+0000","last_active":"2024-03-19T11:29:47.322737+0000","last_peered":"2024-03-19T11:29:47.322737+0000","last_clean":"2024-03-19T11:29:47.322737+0000","last_became_active":"2024-03-19T11:29:46.319334+0000","last_became_peered":"2024-03-19T11:29:46.319334+0000","last_unstale":"2024-03-19T11:29:47.322737+0000","last_undegraded":"2024-03-19T11:29:47.322737+0000","last_fullsized":"2024-03-19T11:29:47.322737+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:04.320463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T11:29:48.193072+0000","last_change":"2024-03-19T11:29:46.436253+0000","last_active":"2024-03-19T11:29:48.193072+0000","last_peered":"2024-03-19T11:29:48.193072+0000","last_clean":"2024-03-19T11:29:48.193072+0000","last_became_active":"2024-03-19T11:29:46.435783+0000","last_became_peered":"2024-03-19T11:29:46.435783+0000","last_unstale":"2024-03-19T11:29:48.193072+0000","last_undegraded":"2024-03-19T11:29:48.193072+0000","last_fullsized":"2024-03-19T11:29:48.193072+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:59.997466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T11:29:48.193767+0000","last_change":"2024-03-19T11:29:46.322491+0000","last_active":"2024-03-19T11:29:48.193767+0000","last_peered":"2024-03-19T11:29:48.193767+0000","last_clean":"2024-03-19T11:29:48.193767+0000","last_became_active":"2024-03-19T11:29:46.321760+0000","last_became_peered":"2024-03-19T11:29:46.321760+0000","last_unstale":"2024-03-19T11:29:48.193767+0000","last_undegraded":"2024-03-19T11:29:48.193767+0000","last_fullsized":"2024-03-19T11:29:48.193767+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:04.561915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":166,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T11:29:48.193062+0000","last_change":"2024-03-19T11:29:45.008018+0000","last_active":"2024-03-19T11:29:48.193062+0000","last_peered":"2024-03-19T11:29:48.193062+0000","last_clean":"2024-03-19T11:29:48.193062+0000","last_became_active":"2024-03-19T11:29:45.005482+0000","last_became_peered":"2024-03-19T11:29:45.005482+0000","last_unstale":"2024-03-19T11:29:48.193062+0000","last_undegraded":"2024-03-19T11:29:48.193062+0000","last_fullsized":"2024-03-19T11:29:48.193062+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:43.166618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:43.166618+0000","last_clean_scrub_stamp":"2024-03-19T11:29:43.166618+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:02.385897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":12128,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542149,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":184320,"data_stored":579370,"data_compressed":6064,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542149,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":143360,"data_stored":51624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27144,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344696,"statfs":{"total":96636764160,"available":96608968704,"internally_reserved":0,"allocated":122880,"data_stored":49255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542148,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27144,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344696,"statfs":{"total":96636764160,"available":96608968704,"internally_reserved":0,"allocated":122880,"data_stored":49255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542148,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31316,"kb_used_data":132,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340524,"statfs":{"total":96636764160,"available":96604696576,"internally_reserved":0,"allocated":135168,"data_stored":51064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27180,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344660,"statfs":{"total":96636764160,"available":96608931840,"internally_reserved":0,"allocated":163840,"data_stored":51435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27152,"kb_used_data":124,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":126976,"data_stored":51567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":212992,"data_stored":579741,"data_compressed":6064,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6064,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6064,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T11:29:50.179 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T11:29:50.466 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:29:50.467 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-03-19T11:29:50.483 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-03-19T11:29:50.130035+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":221532,"kb_used_data":1184,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754753188,"statfs":{"total":773094113280,"available":772867264512,"internally_reserved":0,"allocated":1212416,"data_stored":1463311,"data_compressed":12128,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":12751,"internal_metadata":225496625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000641"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T11:29:47.318721+0000","last_change":"2024-03-19T11:29:46.435120+0000","last_active":"2024-03-19T11:29:47.318721+0000","last_peered":"2024-03-19T11:29:47.318721+0000","last_clean":"2024-03-19T11:29:47.318721+0000","last_became_active":"2024-03-19T11:29:46.434588+0000","last_became_peered":"2024-03-19T11:29:46.434588+0000","last_unstale":"2024-03-19T11:29:47.318721+0000","last_undegraded":"2024-03-19T11:29:47.318721+0000","last_fullsized":"2024-03-19T11:29:47.318721+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:03.134473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T11:29:48.191220+0000","last_change":"2024-03-19T11:29:46.423438+0000","last_active":"2024-03-19T11:29:48.191220+0000","last_peered":"2024-03-19T11:29:48.191220+0000","last_clean":"2024-03-19T11:29:48.191220+0000","last_became_active":"2024-03-19T11:29:46.423027+0000","last_became_peered":"2024-03-19T11:29:46.423027+0000","last_unstale":"2024-03-19T11:29:48.191220+0000","last_undegraded":"2024-03-19T11:29:48.191220+0000","last_fullsized":"2024-03-19T11:29:48.191220+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:55.305083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T11:29:48.193184+0000","last_change":"2024-03-19T11:29:46.542897+0000","last_active":"2024-03-19T11:29:48.193184+0000","last_peered":"2024-03-19T11:29:48.193184+0000","last_clean":"2024-03-19T11:29:48.193184+0000","last_became_active":"2024-03-19T11:29:46.542092+0000","last_became_peered":"2024-03-19T11:29:46.542092+0000","last_unstale":"2024-03-19T11:29:48.193184+0000","last_undegraded":"2024-03-19T11:29:48.193184+0000","last_fullsized":"2024-03-19T11:29:48.193184+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:46.318698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T11:29:48.191202+0000","last_change":"2024-03-19T11:29:46.541233+0000","last_active":"2024-03-19T11:29:48.191202+0000","last_peered":"2024-03-19T11:29:48.191202+0000","last_clean":"2024-03-19T11:29:48.191202+0000","last_became_active":"2024-03-19T11:29:46.540939+0000","last_became_peered":"2024-03-19T11:29:46.540939+0000","last_unstale":"2024-03-19T11:29:48.191202+0000","last_undegraded":"2024-03-19T11:29:48.191202+0000","last_fullsized":"2024-03-19T11:29:48.191202+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:54.761098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T11:29:48.193708+0000","last_change":"2024-03-19T11:29:46.321255+0000","last_active":"2024-03-19T11:29:48.193708+0000","last_peered":"2024-03-19T11:29:48.193708+0000","last_clean":"2024-03-19T11:29:48.193708+0000","last_became_active":"2024-03-19T11:29:46.320679+0000","last_became_peered":"2024-03-19T11:29:46.320679+0000","last_unstale":"2024-03-19T11:29:48.193708+0000","last_undegraded":"2024-03-19T11:29:48.193708+0000","last_fullsized":"2024-03-19T11:29:48.193708+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:14.793563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T11:29:47.322737+0000","last_change":"2024-03-19T11:29:46.319811+0000","last_active":"2024-03-19T11:29:47.322737+0000","last_peered":"2024-03-19T11:29:47.322737+0000","last_clean":"2024-03-19T11:29:47.322737+0000","last_became_active":"2024-03-19T11:29:46.319334+0000","last_became_peered":"2024-03-19T11:29:46.319334+0000","last_unstale":"2024-03-19T11:29:47.322737+0000","last_undegraded":"2024-03-19T11:29:47.322737+0000","last_fullsized":"2024-03-19T11:29:47.322737+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:04.320463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T11:29:48.193072+0000","last_change":"2024-03-19T11:29:46.436253+0000","last_active":"2024-03-19T11:29:48.193072+0000","last_peered":"2024-03-19T11:29:48.193072+0000","last_clean":"2024-03-19T11:29:48.193072+0000","last_became_active":"2024-03-19T11:29:46.435783+0000","last_became_peered":"2024-03-19T11:29:46.435783+0000","last_unstale":"2024-03-19T11:29:48.193072+0000","last_undegraded":"2024-03-19T11:29:48.193072+0000","last_fullsized":"2024-03-19T11:29:48.193072+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:59.997466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T11:29:48.193767+0000","last_change":"2024-03-19T11:29:46.322491+0000","last_active":"2024-03-19T11:29:48.193767+0000","last_peered":"2024-03-19T11:29:48.193767+0000","last_clean":"2024-03-19T11:29:48.193767+0000","last_became_active":"2024-03-19T11:29:46.321760+0000","last_became_peered":"2024-03-19T11:29:46.321760+0000","last_unstale":"2024-03-19T11:29:48.193767+0000","last_undegraded":"2024-03-19T11:29:48.193767+0000","last_fullsized":"2024-03-19T11:29:48.193767+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:04.561915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":166,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T11:29:48.193062+0000","last_change":"2024-03-19T11:29:45.008018+0000","last_active":"2024-03-19T11:29:48.193062+0000","last_peered":"2024-03-19T11:29:48.193062+0000","last_clean":"2024-03-19T11:29:48.193062+0000","last_became_active":"2024-03-19T11:29:45.005482+0000","last_became_peered":"2024-03-19T11:29:45.005482+0000","last_unstale":"2024-03-19T11:29:48.193062+0000","last_undegraded":"2024-03-19T11:29:48.193062+0000","last_fullsized":"2024-03-19T11:29:48.193062+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:43.166618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:43.166618+0000","last_clean_scrub_stamp":"2024-03-19T11:29:43.166618+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:02.385897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":12128,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542149,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":184320,"data_stored":579370,"data_compressed":6064,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542149,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":143360,"data_stored":51624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27144,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344696,"statfs":{"total":96636764160,"available":96608968704,"internally_reserved":0,"allocated":122880,"data_stored":49255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542148,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27144,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344696,"statfs":{"total":96636764160,"available":96608968704,"internally_reserved":0,"allocated":122880,"data_stored":49255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542148,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31316,"kb_used_data":132,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340524,"statfs":{"total":96636764160,"available":96604696576,"internally_reserved":0,"allocated":135168,"data_stored":51064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27180,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344660,"statfs":{"total":96636764160,"available":96608931840,"internally_reserved":0,"allocated":163840,"data_stored":51435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27152,"kb_used_data":124,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":126976,"data_stored":51567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":212992,"data_stored":579741,"data_compressed":6064,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6064,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6064,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T11:29:50.484 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-19T11:29:50.484 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-19T11:29:50.484 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-19T11:29:50.484 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-19T11:29:50.801 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:29:50.801 INFO:teuthology.orchestra.run.smithi038.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-19T11:29:50.819 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-19T11:29:50.819 INFO:teuthology.run_tasks:Running task exec... 2024-03-19T11:29:50.829 INFO:teuthology.task.exec:Executing custom commands... 2024-03-19T11:29:50.829 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi038.front.sepia.ceph.com 2024-03-19T11:29:50.830 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/balancer/log_level debug' 2024-03-19T11:29:51.151 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph balancer status' 2024-03-19T11:29:51.435 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:29:51.435 INFO:teuthology.orchestra.run.smithi038.stdout: "active": true, 2024-03-19T11:29:51.435 INFO:teuthology.orchestra.run.smithi038.stdout: "last_optimize_duration": "0:00:00.000295", 2024-03-19T11:29:51.435 INFO:teuthology.orchestra.run.smithi038.stdout: "last_optimize_started": "Tue Mar 19 11:29:38 2024", 2024-03-19T11:29:51.435 INFO:teuthology.orchestra.run.smithi038.stdout: "mode": "upmap", 2024-03-19T11:29:51.435 INFO:teuthology.orchestra.run.smithi038.stdout: "no_optimization_needed": false, 2024-03-19T11:29:51.435 INFO:teuthology.orchestra.run.smithi038.stdout: "optimize_result": "No pools available", 2024-03-19T11:29:51.435 INFO:teuthology.orchestra.run.smithi038.stdout: "plans": [] 2024-03-19T11:29:51.435 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:29:51.453 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph osd set-require-min-compat-client reef' 2024-03-19T11:29:52.469 INFO:teuthology.orchestra.run.smithi038.stderr:set require_min_compat_client to reef 2024-03-19T11:29:52.474 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:29:52.480+0000 7f6a7f9da640 -1 osd.1 18 handle_osd_map injecting map crc failure 2024-03-19T11:29:52.498 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph balancer mode upmap-read' 2024-03-19T11:29:52.852 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph balancer on' 2024-03-19T11:29:53.160 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph balancer status' 2024-03-19T11:29:53.466 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:29:53.466 INFO:teuthology.orchestra.run.smithi038.stdout: "active": true, 2024-03-19T11:29:53.466 INFO:teuthology.orchestra.run.smithi038.stdout: "last_optimize_duration": "0:00:00.001890", 2024-03-19T11:29:53.467 INFO:teuthology.orchestra.run.smithi038.stdout: "last_optimize_started": "Tue Mar 19 11:29:53 2024", 2024-03-19T11:29:53.467 INFO:teuthology.orchestra.run.smithi038.stdout: "mode": "upmap-read", 2024-03-19T11:29:53.467 INFO:teuthology.orchestra.run.smithi038.stdout: "no_optimization_needed": true, 2024-03-19T11:29:53.467 INFO:teuthology.orchestra.run.smithi038.stdout: "optimize_result": "Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect", 2024-03-19T11:29:53.467 INFO:teuthology.orchestra.run.smithi038.stdout: "plans": [] 2024-03-19T11:29:53.467 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:29:53.485 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph balancer status detail' 2024-03-19T11:29:53.777 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:29:53.777 INFO:teuthology.orchestra.run.smithi038.stdout: "active": true, 2024-03-19T11:29:53.777 INFO:teuthology.orchestra.run.smithi038.stdout: "last_optimize_duration": "0:00:00.001890", 2024-03-19T11:29:53.777 INFO:teuthology.orchestra.run.smithi038.stdout: "last_optimize_started": "Tue Mar 19 11:29:53 2024", 2024-03-19T11:29:53.777 INFO:teuthology.orchestra.run.smithi038.stdout: "mode": "upmap-read", 2024-03-19T11:29:53.777 INFO:teuthology.orchestra.run.smithi038.stdout: "no_optimization_needed": true, 2024-03-19T11:29:53.777 INFO:teuthology.orchestra.run.smithi038.stdout: "optimize_result": "Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect", 2024-03-19T11:29:53.777 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_upmap_items_added": [], 2024-03-19T11:29:53.778 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_upmap_items_removed": [], 2024-03-19T11:29:53.778 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_upmap_primaries_added": [], 2024-03-19T11:29:53.778 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_upmap_primaries_removed": [], 2024-03-19T11:29:53.778 INFO:teuthology.orchestra.run.smithi038.stdout: "plans": [] 2024-03-19T11:29:53.778 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:29:53.794 INFO:teuthology.run_tasks:Running task thrashosds... 2024-03-19T11:29:53.805 INFO:tasks.thrashosds:config is {'aggressive_pg_num_changes': False, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-03-19T11:29:53.806 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-rf-nextobj'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2, 'osd_recovery_max_active': 10, 'osd_recovery_max_single_start': 10}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd heartbeat use min delay socket': True, 'osd inject bad map crc probability': 0.1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd max scrubs': 5, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', 'failed to encode map', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '9ef43722ad94ea688fa7f70daab6dd41fac6ee53'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '9ef43722ad94ea688fa7f70daab6dd41fac6ee53'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-rf-nextobj', 'sha1': '9ef43722ad94ea688fa7f70daab6dd41fac6ee53'}} 2024-03-19T11:29:53.806 INFO:tasks.thrashosds:config is {'aggressive_pg_num_changes': False, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2024-03-19T11:29:53.806 INFO:tasks.thrashosds:Beginning thrashosds... 2024-03-19T11:29:53.806 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-19T11:29:53.807 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T11:29:54.089 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:29:54.089 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-03-19T11:29:54.105 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-03-19T11:29:52.485656+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":221548,"kb_used_data":1196,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754753172,"statfs":{"total":773094113280,"available":772867248128,"internally_reserved":0,"allocated":1224704,"data_stored":1469059,"data_compressed":12128,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":12751,"internal_metadata":225496625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.356262"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T11:29:47.318721+0000","last_change":"2024-03-19T11:29:46.435120+0000","last_active":"2024-03-19T11:29:47.318721+0000","last_peered":"2024-03-19T11:29:47.318721+0000","last_clean":"2024-03-19T11:29:47.318721+0000","last_became_active":"2024-03-19T11:29:46.434588+0000","last_became_peered":"2024-03-19T11:29:46.434588+0000","last_unstale":"2024-03-19T11:29:47.318721+0000","last_undegraded":"2024-03-19T11:29:47.318721+0000","last_fullsized":"2024-03-19T11:29:47.318721+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:03.134473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T11:29:48.191220+0000","last_change":"2024-03-19T11:29:46.423438+0000","last_active":"2024-03-19T11:29:48.191220+0000","last_peered":"2024-03-19T11:29:48.191220+0000","last_clean":"2024-03-19T11:29:48.191220+0000","last_became_active":"2024-03-19T11:29:46.423027+0000","last_became_peered":"2024-03-19T11:29:46.423027+0000","last_unstale":"2024-03-19T11:29:48.191220+0000","last_undegraded":"2024-03-19T11:29:48.191220+0000","last_fullsized":"2024-03-19T11:29:48.191220+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:55.305083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T11:29:48.193184+0000","last_change":"2024-03-19T11:29:46.542897+0000","last_active":"2024-03-19T11:29:48.193184+0000","last_peered":"2024-03-19T11:29:48.193184+0000","last_clean":"2024-03-19T11:29:48.193184+0000","last_became_active":"2024-03-19T11:29:46.542092+0000","last_became_peered":"2024-03-19T11:29:46.542092+0000","last_unstale":"2024-03-19T11:29:48.193184+0000","last_undegraded":"2024-03-19T11:29:48.193184+0000","last_fullsized":"2024-03-19T11:29:48.193184+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:46.318698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T11:29:48.191202+0000","last_change":"2024-03-19T11:29:46.541233+0000","last_active":"2024-03-19T11:29:48.191202+0000","last_peered":"2024-03-19T11:29:48.191202+0000","last_clean":"2024-03-19T11:29:48.191202+0000","last_became_active":"2024-03-19T11:29:46.540939+0000","last_became_peered":"2024-03-19T11:29:46.540939+0000","last_unstale":"2024-03-19T11:29:48.191202+0000","last_undegraded":"2024-03-19T11:29:48.191202+0000","last_fullsized":"2024-03-19T11:29:48.191202+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:54.761098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T11:29:48.193708+0000","last_change":"2024-03-19T11:29:46.321255+0000","last_active":"2024-03-19T11:29:48.193708+0000","last_peered":"2024-03-19T11:29:48.193708+0000","last_clean":"2024-03-19T11:29:48.193708+0000","last_became_active":"2024-03-19T11:29:46.320679+0000","last_became_peered":"2024-03-19T11:29:46.320679+0000","last_unstale":"2024-03-19T11:29:48.193708+0000","last_undegraded":"2024-03-19T11:29:48.193708+0000","last_fullsized":"2024-03-19T11:29:48.193708+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:14.793563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T11:29:47.322737+0000","last_change":"2024-03-19T11:29:46.319811+0000","last_active":"2024-03-19T11:29:47.322737+0000","last_peered":"2024-03-19T11:29:47.322737+0000","last_clean":"2024-03-19T11:29:47.322737+0000","last_became_active":"2024-03-19T11:29:46.319334+0000","last_became_peered":"2024-03-19T11:29:46.319334+0000","last_unstale":"2024-03-19T11:29:47.322737+0000","last_undegraded":"2024-03-19T11:29:47.322737+0000","last_fullsized":"2024-03-19T11:29:47.322737+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:04.320463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T11:29:48.193072+0000","last_change":"2024-03-19T11:29:46.436253+0000","last_active":"2024-03-19T11:29:48.193072+0000","last_peered":"2024-03-19T11:29:48.193072+0000","last_clean":"2024-03-19T11:29:48.193072+0000","last_became_active":"2024-03-19T11:29:46.435783+0000","last_became_peered":"2024-03-19T11:29:46.435783+0000","last_unstale":"2024-03-19T11:29:48.193072+0000","last_undegraded":"2024-03-19T11:29:48.193072+0000","last_fullsized":"2024-03-19T11:29:48.193072+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:59.997466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T11:29:48.193767+0000","last_change":"2024-03-19T11:29:46.322491+0000","last_active":"2024-03-19T11:29:48.193767+0000","last_peered":"2024-03-19T11:29:48.193767+0000","last_clean":"2024-03-19T11:29:48.193767+0000","last_became_active":"2024-03-19T11:29:46.321760+0000","last_became_peered":"2024-03-19T11:29:46.321760+0000","last_unstale":"2024-03-19T11:29:48.193767+0000","last_undegraded":"2024-03-19T11:29:48.193767+0000","last_fullsized":"2024-03-19T11:29:48.193767+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:04.561915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":166,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T11:29:48.193062+0000","last_change":"2024-03-19T11:29:45.008018+0000","last_active":"2024-03-19T11:29:48.193062+0000","last_peered":"2024-03-19T11:29:48.193062+0000","last_clean":"2024-03-19T11:29:48.193062+0000","last_became_active":"2024-03-19T11:29:45.005482+0000","last_became_peered":"2024-03-19T11:29:45.005482+0000","last_unstale":"2024-03-19T11:29:48.193062+0000","last_undegraded":"2024-03-19T11:29:48.193062+0000","last_fullsized":"2024-03-19T11:29:48.193062+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:43.166618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:43.166618+0000","last_clean_scrub_stamp":"2024-03-19T11:29:43.166618+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:02.385897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":12128,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542149,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":184320,"data_stored":579370,"data_compressed":6064,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542149,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":143360,"data_stored":51624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27144,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344696,"statfs":{"total":96636764160,"available":96608968704,"internally_reserved":0,"allocated":122880,"data_stored":49255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542148,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27144,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344696,"statfs":{"total":96636764160,"available":96608968704,"internally_reserved":0,"allocated":122880,"data_stored":49255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542148,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31316,"kb_used_data":132,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340524,"statfs":{"total":96636764160,"available":96604696576,"internally_reserved":0,"allocated":135168,"data_stored":51064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27180,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344660,"statfs":{"total":96636764160,"available":96608931840,"internally_reserved":0,"allocated":163840,"data_stored":51435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27152,"kb_used_data":124,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":126976,"data_stored":51567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":225280,"data_stored":585489,"data_compressed":6064,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6064,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6064,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T11:29:54.106 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T11:29:54.405 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:29:54.406 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-03-19T11:29:54.424 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-03-19T11:29:54.131147+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":221568,"kb_used_data":1220,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754753152,"statfs":{"total":773094113280,"available":772867227648,"internally_reserved":0,"allocated":1249280,"data_stored":1480217,"data_compressed":12128,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":12751,"internal_metadata":225496625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001753"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T11:29:47.318721+0000","last_change":"2024-03-19T11:29:46.435120+0000","last_active":"2024-03-19T11:29:47.318721+0000","last_peered":"2024-03-19T11:29:47.318721+0000","last_clean":"2024-03-19T11:29:47.318721+0000","last_became_active":"2024-03-19T11:29:46.434588+0000","last_became_peered":"2024-03-19T11:29:46.434588+0000","last_unstale":"2024-03-19T11:29:47.318721+0000","last_undegraded":"2024-03-19T11:29:47.318721+0000","last_fullsized":"2024-03-19T11:29:47.318721+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:03.134473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T11:29:48.191220+0000","last_change":"2024-03-19T11:29:46.423438+0000","last_active":"2024-03-19T11:29:48.191220+0000","last_peered":"2024-03-19T11:29:48.191220+0000","last_clean":"2024-03-19T11:29:48.191220+0000","last_became_active":"2024-03-19T11:29:46.423027+0000","last_became_peered":"2024-03-19T11:29:46.423027+0000","last_unstale":"2024-03-19T11:29:48.191220+0000","last_undegraded":"2024-03-19T11:29:48.191220+0000","last_fullsized":"2024-03-19T11:29:48.191220+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:55.305083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T11:29:48.193184+0000","last_change":"2024-03-19T11:29:46.542897+0000","last_active":"2024-03-19T11:29:48.193184+0000","last_peered":"2024-03-19T11:29:48.193184+0000","last_clean":"2024-03-19T11:29:48.193184+0000","last_became_active":"2024-03-19T11:29:46.542092+0000","last_became_peered":"2024-03-19T11:29:46.542092+0000","last_unstale":"2024-03-19T11:29:48.193184+0000","last_undegraded":"2024-03-19T11:29:48.193184+0000","last_fullsized":"2024-03-19T11:29:48.193184+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:46.318698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T11:29:48.191202+0000","last_change":"2024-03-19T11:29:46.541233+0000","last_active":"2024-03-19T11:29:48.191202+0000","last_peered":"2024-03-19T11:29:48.191202+0000","last_clean":"2024-03-19T11:29:48.191202+0000","last_became_active":"2024-03-19T11:29:46.540939+0000","last_became_peered":"2024-03-19T11:29:46.540939+0000","last_unstale":"2024-03-19T11:29:48.191202+0000","last_undegraded":"2024-03-19T11:29:48.191202+0000","last_fullsized":"2024-03-19T11:29:48.191202+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:54.761098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T11:29:48.193708+0000","last_change":"2024-03-19T11:29:46.321255+0000","last_active":"2024-03-19T11:29:48.193708+0000","last_peered":"2024-03-19T11:29:48.193708+0000","last_clean":"2024-03-19T11:29:48.193708+0000","last_became_active":"2024-03-19T11:29:46.320679+0000","last_became_peered":"2024-03-19T11:29:46.320679+0000","last_unstale":"2024-03-19T11:29:48.193708+0000","last_undegraded":"2024-03-19T11:29:48.193708+0000","last_fullsized":"2024-03-19T11:29:48.193708+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:14.793563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T11:29:47.322737+0000","last_change":"2024-03-19T11:29:46.319811+0000","last_active":"2024-03-19T11:29:47.322737+0000","last_peered":"2024-03-19T11:29:47.322737+0000","last_clean":"2024-03-19T11:29:47.322737+0000","last_became_active":"2024-03-19T11:29:46.319334+0000","last_became_peered":"2024-03-19T11:29:46.319334+0000","last_unstale":"2024-03-19T11:29:47.322737+0000","last_undegraded":"2024-03-19T11:29:47.322737+0000","last_fullsized":"2024-03-19T11:29:47.322737+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:04.320463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T11:29:48.193072+0000","last_change":"2024-03-19T11:29:46.436253+0000","last_active":"2024-03-19T11:29:48.193072+0000","last_peered":"2024-03-19T11:29:48.193072+0000","last_clean":"2024-03-19T11:29:48.193072+0000","last_became_active":"2024-03-19T11:29:46.435783+0000","last_became_peered":"2024-03-19T11:29:46.435783+0000","last_unstale":"2024-03-19T11:29:48.193072+0000","last_undegraded":"2024-03-19T11:29:48.193072+0000","last_fullsized":"2024-03-19T11:29:48.193072+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:59.997466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T11:29:48.193767+0000","last_change":"2024-03-19T11:29:46.322491+0000","last_active":"2024-03-19T11:29:48.193767+0000","last_peered":"2024-03-19T11:29:48.193767+0000","last_clean":"2024-03-19T11:29:48.193767+0000","last_became_active":"2024-03-19T11:29:46.321760+0000","last_became_peered":"2024-03-19T11:29:46.321760+0000","last_unstale":"2024-03-19T11:29:48.193767+0000","last_undegraded":"2024-03-19T11:29:48.193767+0000","last_fullsized":"2024-03-19T11:29:48.193767+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:04.561915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":166,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-03-19T11:29:48.193062+0000","last_change":"2024-03-19T11:29:45.008018+0000","last_active":"2024-03-19T11:29:48.193062+0000","last_peered":"2024-03-19T11:29:48.193062+0000","last_clean":"2024-03-19T11:29:48.193062+0000","last_became_active":"2024-03-19T11:29:45.005482+0000","last_became_peered":"2024-03-19T11:29:45.005482+0000","last_unstale":"2024-03-19T11:29:48.193062+0000","last_undegraded":"2024-03-19T11:29:48.193062+0000","last_fullsized":"2024-03-19T11:29:48.193062+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:43.166618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:43.166618+0000","last_clean_scrub_stamp":"2024-03-19T11:29:43.166618+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:02.385897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":12128,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542149,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":184320,"data_stored":579370,"data_compressed":6064,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542149,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":143360,"data_stored":51624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27144,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344696,"statfs":{"total":96636764160,"available":96608968704,"internally_reserved":0,"allocated":122880,"data_stored":49255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542148,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27144,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344696,"statfs":{"total":96636764160,"available":96608968704,"internally_reserved":0,"allocated":122880,"data_stored":49255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542148,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31316,"kb_used_data":132,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340524,"statfs":{"total":96636764160,"available":96604696576,"internally_reserved":0,"allocated":135168,"data_stored":51064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":176128,"data_stored":57183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27156,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":139264,"data_stored":56977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":225280,"data_stored":585489,"data_compressed":6064,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6064,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6064,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T11:29:54.425 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-19T11:29:54.426 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-03-19T11:29:54.715 INFO:teuthology.orchestra.run.smithi038.stdout:epoch 19 2024-03-19T11:29:54.716 INFO:teuthology.orchestra.run.smithi038.stdout:fsid 246bcf68-c431-4aa7-b787-5e9b2d9519d6 2024-03-19T11:29:54.716 INFO:teuthology.orchestra.run.smithi038.stdout:created 2024-03-19T11:29:33.327073+0000 2024-03-19T11:29:54.716 INFO:teuthology.orchestra.run.smithi038.stdout:modified 2024-03-19T11:29:52.460922+0000 2024-03-19T11:29:54.716 INFO:teuthology.orchestra.run.smithi038.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-19T11:29:54.716 INFO:teuthology.orchestra.run.smithi038.stdout:crush_version 5 2024-03-19T11:29:54.716 INFO:teuthology.orchestra.run.smithi038.stdout:full_ratio 0.95 2024-03-19T11:29:54.716 INFO:teuthology.orchestra.run.smithi038.stdout:backfillfull_ratio 0.9 2024-03-19T11:29:54.716 INFO:teuthology.orchestra.run.smithi038.stdout:nearfull_ratio 0.85 2024-03-19T11:29:54.716 INFO:teuthology.orchestra.run.smithi038.stdout:require_min_compat_client reef 2024-03-19T11:29:54.717 INFO:teuthology.orchestra.run.smithi038.stdout:min_compat_client jewel 2024-03-19T11:29:54.717 INFO:teuthology.orchestra.run.smithi038.stdout:require_osd_release squid 2024-03-19T11:29:54.717 INFO:teuthology.orchestra.run.smithi038.stdout:stretch_mode_enabled false 2024-03-19T11:29:54.717 INFO:teuthology.orchestra.run.smithi038.stdout:pool 1 '.mgr' replicated size 2 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 16 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 8.00 2024-03-19T11:29:54.717 INFO:teuthology.orchestra.run.smithi038.stdout:pool 2 'rbd' replicated size 2 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 18 flags hashpspool stripe_width 0 application rbd read_balance_score 1.99 2024-03-19T11:29:54.717 INFO:teuthology.orchestra.run.smithi038.stdout:max_osd 8 2024-03-19T11:29:54.717 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.38:6808/3148574891,v1:172.21.15.38:6811/3148574891] [v2:172.21.15.38:6813/3148574891,v1:172.21.15.38:6815/3148574891] exists,up 3bb2d2be-5253-4da0-acf3-85a114dc3801 2024-03-19T11:29:54.717 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1 up in weight 1 up_from 14 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:172.21.15.38:6824/675027622,v1:172.21.15.38:6825/675027622] [v2:172.21.15.38:6826/675027622,v1:172.21.15.38:6827/675027622] exists,up e1c926b2-e0be-4ccc-b8a5-d80d0b584ead 2024-03-19T11:29:54.717 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2 up in weight 1 up_from 14 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:172.21.15.38:6800/58279616,v1:172.21.15.38:6801/58279616] [v2:172.21.15.38:6802/58279616,v1:172.21.15.38:6804/58279616] exists,up 61262219-4f8a-4733-af9d-b5208a17c0ea 2024-03-19T11:29:54.717 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.38:6803/2942163952,v1:172.21.15.38:6805/2942163952] [v2:172.21.15.38:6806/2942163952,v1:172.21.15.38:6809/2942163952] exists,up a31f8d1b-4e38-4ca3-b251-0877002d7235 2024-03-19T11:29:54.717 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.172:6808/1353919713,v1:172.21.15.172:6810/1353919713] [v2:172.21.15.172:6812/1353919713,v1:172.21.15.172:6814/1353919713] exists,up 6746723d-f59c-40ba-857e-d8c954d2675f 2024-03-19T11:29:54.717 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5 up in weight 1 up_from 14 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:172.21.15.172:6809/2733060387,v1:172.21.15.172:6811/2733060387] [v2:172.21.15.172:6813/2733060387,v1:172.21.15.172:6815/2733060387] exists,up f59afdf8-2277-4583-af68-78136f1c5d62 2024-03-19T11:29:54.717 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6 up in weight 1 up_from 13 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:172.21.15.172:6800/337743699,v1:172.21.15.172:6801/337743699] [v2:172.21.15.172:6802/337743699,v1:172.21.15.172:6803/337743699] exists,up f91d8c47-4e10-40c6-a02d-66d6b72a657b 2024-03-19T11:29:54.718 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7 up in weight 1 up_from 14 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:172.21.15.172:6824/4145361100,v1:172.21.15.172:6825/4145361100] [v2:172.21.15.172:6826/4145361100,v1:172.21.15.172:6827/4145361100] exists,up 46fc052a-5708-42ff-9d5a-bed7a0fc457a 2024-03-19T11:29:54.733 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.38:6808/3148574891,v1:172.21.15.38:6811/3148574891] [v2:172.21.15.38:6813/3148574891,v1:172.21.15.38:6815/3148574891] exists,up 3bb2d2be-5253-4da0-acf3-85a114dc3801', 'osd.1 up in weight 1 up_from 14 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:172.21.15.38:6824/675027622,v1:172.21.15.38:6825/675027622] [v2:172.21.15.38:6826/675027622,v1:172.21.15.38:6827/675027622] exists,up e1c926b2-e0be-4ccc-b8a5-d80d0b584ead', 'osd.2 up in weight 1 up_from 14 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:172.21.15.38:6800/58279616,v1:172.21.15.38:6801/58279616] [v2:172.21.15.38:6802/58279616,v1:172.21.15.38:6804/58279616] exists,up 61262219-4f8a-4733-af9d-b5208a17c0ea', 'osd.3 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.38:6803/2942163952,v1:172.21.15.38:6805/2942163952] [v2:172.21.15.38:6806/2942163952,v1:172.21.15.38:6809/2942163952] exists,up a31f8d1b-4e38-4ca3-b251-0877002d7235', 'osd.4 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.172:6808/1353919713,v1:172.21.15.172:6810/1353919713] [v2:172.21.15.172:6812/1353919713,v1:172.21.15.172:6814/1353919713] exists,up 6746723d-f59c-40ba-857e-d8c954d2675f', 'osd.5 up in weight 1 up_from 14 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:172.21.15.172:6809/2733060387,v1:172.21.15.172:6811/2733060387] [v2:172.21.15.172:6813/2733060387,v1:172.21.15.172:6815/2733060387] exists,up f59afdf8-2277-4583-af68-78136f1c5d62', 'osd.6 up in weight 1 up_from 13 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:172.21.15.172:6800/337743699,v1:172.21.15.172:6801/337743699] [v2:172.21.15.172:6802/337743699,v1:172.21.15.172:6803/337743699] exists,up f91d8c47-4e10-40c6-a02d-66d6b72a657b', 'osd.7 up in weight 1 up_from 14 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:172.21.15.172:6824/4145361100,v1:172.21.15.172:6825/4145361100] [v2:172.21.15.172:6826/4145361100,v1:172.21.15.172:6827/4145361100] exists,up 46fc052a-5708-42ff-9d5a-bed7a0fc457a'] 2024-03-19T11:29:54.734 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-03-19T11:29:54.851 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:29:54.851 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mon.a", 2024-03-19T11:29:54.851 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster": "ceph", 2024-03-19T11:29:54.851 INFO:teuthology.orchestra.run.smithi038.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-03-19T11:29:54.851 INFO:teuthology.orchestra.run.smithi038.stdout: "admin_socket_mode": "", 2024-03-19T11:29:54.851 INFO:teuthology.orchestra.run.smithi038.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-03-19T11:29:54.852 INFO:teuthology.orchestra.run.smithi038.stdout: "auth_client_required": "cephx, none", 2024-03-19T11:29:54.852 INFO:teuthology.orchestra.run.smithi038.stdout: "auth_cluster_required": "cephx", 2024-03-19T11:29:54.852 INFO:teuthology.orchestra.run.smithi038.stdout: "auth_debug": "true", 2024-03-19T11:29:54.852 INFO:teuthology.orchestra.run.smithi038.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-03-19T11:29:54.852 INFO:teuthology.orchestra.run.smithi038.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-03-19T11:29:54.852 INFO:teuthology.orchestra.run.smithi038.stdout: "auth_service_required": "cephx", 2024-03-19T11:29:54.852 INFO:teuthology.orchestra.run.smithi038.stdout: "auth_service_ticket_ttl": "240.000000", 2024-03-19T11:29:54.852 INFO:teuthology.orchestra.run.smithi038.stdout: "auth_supported": "cephx", 2024-03-19T11:29:54.852 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_aio": "true", 2024-03-19T11:29:54.852 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_aio_max_queue_depth": "1024", 2024-03-19T11:29:54.852 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_aio_poll_ms": "250", 2024-03-19T11:29:54.852 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_aio_reap_max": "16", 2024-03-19T11:29:54.852 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_async_discard_threads": "0", 2024-03-19T11:29:54.853 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_block_size": "4096", 2024-03-19T11:29:54.853 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_debug_aio": "false", 2024-03-19T11:29:54.853 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-03-19T11:29:54.853 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-03-19T11:29:54.853 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_debug_inflight_ios": "false", 2024-03-19T11:29:54.853 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_enable_discard": "false", 2024-03-19T11:29:54.853 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_flock_retry": "3", 2024-03-19T11:29:54.853 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_flock_retry_interval": "0.100000", 2024-03-19T11:29:54.853 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_inject_crash": "0", 2024-03-19T11:29:54.853 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_inject_crash_flush_delay": "2", 2024-03-19T11:29:54.853 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_ioring": "false", 2024-03-19T11:29:54.853 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_ioring_hipri": "false", 2024-03-19T11:29:54.853 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_ioring_sqthread_poll": "false", 2024-03-19T11:29:54.853 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-03-19T11:29:54.853 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_read_buffer_alignment": "4096", 2024-03-19T11:29:54.854 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-03-19T11:29:54.854 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_type": "", 2024-03-19T11:29:54.854 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_alloc_size": "1048576", 2024-03-19T11:29:54.854 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_allocator": "hybrid", 2024-03-19T11:29:54.854 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_buffered_io": "true", 2024-03-19T11:29:54.854 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_check_for_zeros": "false", 2024-03-19T11:29:54.854 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_check_volume_selector_often": "false", 2024-03-19T11:29:54.854 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-03-19T11:29:54.854 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_compact_log_sync": "false", 2024-03-19T11:29:54.854 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-03-19T11:29:54.854 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-03-19T11:29:54.855 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_log_compact_min_size": "16777216", 2024-03-19T11:29:54.855 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_log_replay_check_allocations": "true", 2024-03-19T11:29:54.855 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_max_log_runway": "4194304", 2024-03-19T11:29:54.855 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_max_prefetch": "1048576", 2024-03-19T11:29:54.855 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_min_flush_size": "524288", 2024-03-19T11:29:54.855 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_min_log_runway": "1048576", 2024-03-19T11:29:54.855 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_replay_recovery": "false", 2024-03-19T11:29:54.855 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-03-19T11:29:54.855 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_shared_alloc_size": "65536", 2024-03-19T11:29:54.855 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_sync_write": "false", 2024-03-19T11:29:54.855 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-03-19T11:29:54.855 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-03-19T11:29:54.855 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-03-19T11:29:54.855 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_allocation_from_file": "true", 2024-03-19T11:29:54.856 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_allocator": "hybrid", 2024-03-19T11:29:54.856 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-03-19T11:29:54.856 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-03-19T11:29:54.856 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-03-19T11:29:54.856 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-03-19T11:29:54.856 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-03-19T11:29:54.856 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-03-19T11:29:54.856 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blobid_prealloc": "10240", 2024-03-19T11:29:54.856 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_block_create": "true", 2024-03-19T11:29:54.856 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_block_db_create": "false", 2024-03-19T11:29:54.856 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_block_db_path": "", 2024-03-19T11:29:54.856 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_block_db_size": "0", 2024-03-19T11:29:54.857 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_block_path": "", 2024-03-19T11:29:54.857 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_block_preallocate_file": "false", 2024-03-19T11:29:54.857 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_block_size": "107374182400", 2024-03-19T11:29:54.857 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_block_wal_create": "false", 2024-03-19T11:29:54.857 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_block_wal_path": "", 2024-03-19T11:29:54.857 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_block_wal_size": "100663296", 2024-03-19T11:29:54.857 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_bluefs": "true", 2024-03-19T11:29:54.857 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-03-19T11:29:54.857 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_bluefs_env_mirror": "false", 2024-03-19T11:29:54.857 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-03-19T11:29:54.858 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-03-19T11:29:54.858 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-03-19T11:29:54.858 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-03-19T11:29:54.858 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-03-19T11:29:54.858 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-03-19T11:29:54.858 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_autotune": "true", 2024-03-19T11:29:54.858 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-03-19T11:29:54.858 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-03-19T11:29:54.858 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-03-19T11:29:54.858 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-03-19T11:29:54.858 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_size": "0", 2024-03-19T11:29:54.858 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-03-19T11:29:54.858 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-03-19T11:29:54.859 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-03-19T11:29:54.859 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-03-19T11:29:54.859 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_type": "2q", 2024-03-19T11:29:54.859 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-03-19T11:29:54.859 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_clone_cow": "true", 2024-03-19T11:29:54.859 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_compression_algorithm": "snappy", 2024-03-19T11:29:54.859 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_compression_max_blob_size": "0", 2024-03-19T11:29:54.859 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-03-19T11:29:54.859 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-03-19T11:29:54.859 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_compression_min_blob_size": "0", 2024-03-19T11:29:54.859 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-03-19T11:29:54.859 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-03-19T11:29:54.859 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_compression_mode": "none", 2024-03-19T11:29:54.859 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-03-19T11:29:54.859 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_csum_type": "crc32c", 2024-03-19T11:29:54.860 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-03-19T11:29:54.860 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_enforce_settings": "default", 2024-03-19T11:29:54.860 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_freelist": "false", 2024-03-19T11:29:54.860 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_fsck_abort": "false", 2024-03-19T11:29:54.860 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-03-19T11:29:54.860 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-03-19T11:29:54.860 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_inject_read_err": "false", 2024-03-19T11:29:54.860 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_legacy_omap": "false", 2024-03-19T11:29:54.860 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-03-19T11:29:54.860 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-03-19T11:29:54.860 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-03-19T11:29:54.860 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-03-19T11:29:54.860 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_prefill": "0.000000", 2024-03-19T11:29:54.860 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-03-19T11:29:54.861 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-03-19T11:29:54.861 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-03-19T11:29:54.861 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_small_allocations": "0", 2024-03-19T11:29:54.861 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-03-19T11:29:54.861 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_default_buffered_read": "true", 2024-03-19T11:29:54.861 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_default_buffered_write": "false", 2024-03-19T11:29:54.861 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_deferred_batch_ops": "0", 2024-03-19T11:29:54.861 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-03-19T11:29:54.861 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-03-19T11:29:54.861 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_elastic_shared_blobs": "true", 2024-03-19T11:29:54.861 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-03-19T11:29:54.861 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-03-19T11:29:54.861 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-03-19T11:29:54.861 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-03-19T11:29:54.861 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-03-19T11:29:54.862 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fail_eio": "false", 2024-03-19T11:29:54.862 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-03-19T11:29:54.862 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-03-19T11:29:54.862 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-03-19T11:29:54.862 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-03-19T11:29:54.862 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_on_mkfs": "true", 2024-03-19T11:29:54.862 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-03-19T11:29:54.862 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_on_mount": "false", 2024-03-19T11:29:54.862 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-03-19T11:29:54.862 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_on_umount": "false", 2024-03-19T11:29:54.863 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-03-19T11:29:54.863 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-03-19T11:29:54.863 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-03-19T11:29:54.863 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-03-19T11:29:54.863 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-03-19T11:29:54.863 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-03-19T11:29:54.863 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-03-19T11:29:54.863 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-03-19T11:29:54.863 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_ignore_data_csum": "false", 2024-03-19T11:29:54.863 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-03-19T11:29:54.863 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_kvbackend": "rocksdb", 2024-03-19T11:29:54.863 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-03-19T11:29:54.863 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-03-19T11:29:54.863 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_log_op_age": "5.000000", 2024-03-19T11:29:54.863 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_max_alloc_size": "0", 2024-03-19T11:29:54.864 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_max_blob_size": "0", 2024-03-19T11:29:54.864 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-03-19T11:29:54.864 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-03-19T11:29:54.864 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_max_defer_interval": "3.000000", 2024-03-19T11:29:54.864 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_max_deferred_txc": "32", 2024-03-19T11:29:54.864 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_min_alloc_size": "0", 2024-03-19T11:29:54.864 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-03-19T11:29:54.864 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-03-19T11:29:54.864 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_nid_prealloc": "1024", 2024-03-19T11:29:54.864 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_prefer_deferred_size": "0", 2024-03-19T11:29:54.864 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-03-19T11:29:54.865 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-03-19T11:29:54.865 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_qfsck_on_mount": "true", 2024-03-19T11:29:54.865 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_retry_disk_reads": "3", 2024-03-19T11:29:54.865 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_rocksdb_cf": "true", 2024-03-19T11:29:54.865 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-03-19T11:29:54.865 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-03-19T11:29:54.865 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_rocksdb_options_annex": "", 2024-03-19T11:29:54.865 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_spdk_coremask": "0x1", 2024-03-19T11:29:54.866 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_spdk_io_sleep": "5", 2024-03-19T11:29:54.866 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_spdk_max_io_completion": "0", 2024-03-19T11:29:54.866 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_spdk_mem": "512", 2024-03-19T11:29:54.866 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_sync_submit_transaction": "false", 2024-03-19T11:29:54.866 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_throttle_bytes": "67108864", 2024-03-19T11:29:54.866 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_throttle_cost_per_io": "0", 2024-03-19T11:29:54.866 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-03-19T11:29:54.866 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-03-19T11:29:54.866 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-03-19T11:29:54.866 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-03-19T11:29:54.866 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_tracing": "false", 2024-03-19T11:29:54.866 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-03-19T11:29:54.866 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-03-19T11:29:54.866 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_volume_selection_reserved": "0", 2024-03-19T11:29:54.867 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-03-19T11:29:54.867 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-03-19T11:29:54.867 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-03-19T11:29:54.867 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-03-19T11:29:54.867 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-03-19T11:29:54.867 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-03-19T11:29:54.867 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_zero_block_detection": "false", 2024-03-19T11:29:54.867 INFO:teuthology.orchestra.run.smithi038.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-03-19T11:29:54.867 INFO:teuthology.orchestra.run.smithi038.stdout: "cephfs_mirror_action_update_interval": "2", 2024-03-19T11:29:54.867 INFO:teuthology.orchestra.run.smithi038.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-03-19T11:29:54.867 INFO:teuthology.orchestra.run.smithi038.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-03-19T11:29:54.867 INFO:teuthology.orchestra.run.smithi038.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-03-19T11:29:54.867 INFO:teuthology.orchestra.run.smithi038.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-03-19T11:29:54.867 INFO:teuthology.orchestra.run.smithi038.stdout: "cephfs_mirror_mount_timeout": "10", 2024-03-19T11:29:54.868 INFO:teuthology.orchestra.run.smithi038.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-03-19T11:29:54.868 INFO:teuthology.orchestra.run.smithi038.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-03-19T11:29:54.868 INFO:teuthology.orchestra.run.smithi038.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-03-19T11:29:54.868 INFO:teuthology.orchestra.run.smithi038.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-03-19T11:29:54.868 INFO:teuthology.orchestra.run.smithi038.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-03-19T11:29:54.868 INFO:teuthology.orchestra.run.smithi038.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-03-19T11:29:54.868 INFO:teuthology.orchestra.run.smithi038.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-03-19T11:29:54.868 INFO:teuthology.orchestra.run.smithi038.stdout: "cephx_cluster_require_signatures": "false", 2024-03-19T11:29:54.868 INFO:teuthology.orchestra.run.smithi038.stdout: "cephx_cluster_require_version": "2", 2024-03-19T11:29:54.868 INFO:teuthology.orchestra.run.smithi038.stdout: "cephx_require_signatures": "false", 2024-03-19T11:29:54.868 INFO:teuthology.orchestra.run.smithi038.stdout: "cephx_require_version": "2", 2024-03-19T11:29:54.869 INFO:teuthology.orchestra.run.smithi038.stdout: "cephx_service_require_signatures": "false", 2024-03-19T11:29:54.869 INFO:teuthology.orchestra.run.smithi038.stdout: "cephx_service_require_version": "2", 2024-03-19T11:29:54.869 INFO:teuthology.orchestra.run.smithi038.stdout: "cephx_sign_messages": "true", 2024-03-19T11:29:54.869 INFO:teuthology.orchestra.run.smithi038.stdout: "chdir": "", 2024-03-19T11:29:54.869 INFO:teuthology.orchestra.run.smithi038.stdout: "client_acl_type": "", 2024-03-19T11:29:54.869 INFO:teuthology.orchestra.run.smithi038.stdout: "client_asio_thread_count": "2", 2024-03-19T11:29:54.869 INFO:teuthology.orchestra.run.smithi038.stdout: "client_cache_mid": "0.750000", 2024-03-19T11:29:54.869 INFO:teuthology.orchestra.run.smithi038.stdout: "client_cache_size": "16384", 2024-03-19T11:29:54.870 INFO:teuthology.orchestra.run.smithi038.stdout: "client_caps_release_delay": "5", 2024-03-19T11:29:54.870 INFO:teuthology.orchestra.run.smithi038.stdout: "client_check_pool_perm": "true", 2024-03-19T11:29:54.870 INFO:teuthology.orchestra.run.smithi038.stdout: "client_collect_and_send_global_metrics": "false", 2024-03-19T11:29:54.870 INFO:teuthology.orchestra.run.smithi038.stdout: "client_debug_force_sync_read": "false", 2024-03-19T11:29:54.870 INFO:teuthology.orchestra.run.smithi038.stdout: "client_debug_getattr_caps": "false", 2024-03-19T11:29:54.870 INFO:teuthology.orchestra.run.smithi038.stdout: "client_debug_inject_tick_delay": "0", 2024-03-19T11:29:54.870 INFO:teuthology.orchestra.run.smithi038.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-03-19T11:29:54.871 INFO:teuthology.orchestra.run.smithi038.stdout: "client_die_on_failed_remount": "false", 2024-03-19T11:29:54.871 INFO:teuthology.orchestra.run.smithi038.stdout: "client_dirsize_rbytes": "true", 2024-03-19T11:29:54.871 INFO:teuthology.orchestra.run.smithi038.stdout: "client_force_lazyio": "false", 2024-03-19T11:29:54.871 INFO:teuthology.orchestra.run.smithi038.stdout: "client_fs": "", 2024-03-19T11:29:54.871 INFO:teuthology.orchestra.run.smithi038.stdout: "client_inject_fixed_oldest_tid": "false", 2024-03-19T11:29:54.872 INFO:teuthology.orchestra.run.smithi038.stdout: "client_inject_release_failure": "false", 2024-03-19T11:29:54.872 INFO:teuthology.orchestra.run.smithi038.stdout: "client_max_inline_size": "4096", 2024-03-19T11:29:54.872 INFO:teuthology.orchestra.run.smithi038.stdout: "client_max_retries_on_remount_failure": "5", 2024-03-19T11:29:54.872 INFO:teuthology.orchestra.run.smithi038.stdout: "client_mds_namespace": "", 2024-03-19T11:29:54.872 INFO:teuthology.orchestra.run.smithi038.stdout: "client_metadata": "", 2024-03-19T11:29:54.872 INFO:teuthology.orchestra.run.smithi038.stdout: "client_mount_gid": "-1", 2024-03-19T11:29:54.872 INFO:teuthology.orchestra.run.smithi038.stdout: "client_mount_timeout": "300", 2024-03-19T11:29:54.872 INFO:teuthology.orchestra.run.smithi038.stdout: "client_mount_uid": "-1", 2024-03-19T11:29:54.872 INFO:teuthology.orchestra.run.smithi038.stdout: "client_mountpoint": "/", 2024-03-19T11:29:54.872 INFO:teuthology.orchestra.run.smithi038.stdout: "client_notify_timeout": "10", 2024-03-19T11:29:54.872 INFO:teuthology.orchestra.run.smithi038.stdout: "client_oc": "true", 2024-03-19T11:29:54.873 INFO:teuthology.orchestra.run.smithi038.stdout: "client_oc_max_dirty": "104857600", 2024-03-19T11:29:54.873 INFO:teuthology.orchestra.run.smithi038.stdout: "client_oc_max_dirty_age": "5.000000", 2024-03-19T11:29:54.873 INFO:teuthology.orchestra.run.smithi038.stdout: "client_oc_max_objects": "1000", 2024-03-19T11:29:54.873 INFO:teuthology.orchestra.run.smithi038.stdout: "client_oc_size": "209715200", 2024-03-19T11:29:54.873 INFO:teuthology.orchestra.run.smithi038.stdout: "client_oc_target_dirty": "8388608", 2024-03-19T11:29:54.874 INFO:teuthology.orchestra.run.smithi038.stdout: "client_permissions": "true", 2024-03-19T11:29:54.874 INFO:teuthology.orchestra.run.smithi038.stdout: "client_quota": "true", 2024-03-19T11:29:54.874 INFO:teuthology.orchestra.run.smithi038.stdout: "client_quota_df": "true", 2024-03-19T11:29:54.874 INFO:teuthology.orchestra.run.smithi038.stdout: "client_readahead_max_bytes": "0", 2024-03-19T11:29:54.875 INFO:teuthology.orchestra.run.smithi038.stdout: "client_readahead_max_periods": "4", 2024-03-19T11:29:54.875 INFO:teuthology.orchestra.run.smithi038.stdout: "client_readahead_min": "131072", 2024-03-19T11:29:54.875 INFO:teuthology.orchestra.run.smithi038.stdout: "client_reconnect_stale": "false", 2024-03-19T11:29:54.875 INFO:teuthology.orchestra.run.smithi038.stdout: "client_shutdown_timeout": "30", 2024-03-19T11:29:54.875 INFO:teuthology.orchestra.run.smithi038.stdout: "client_snapdir": ".snap", 2024-03-19T11:29:54.875 INFO:teuthology.orchestra.run.smithi038.stdout: "client_tick_interval": "1", 2024-03-19T11:29:54.875 INFO:teuthology.orchestra.run.smithi038.stdout: "client_trace": "", 2024-03-19T11:29:54.875 INFO:teuthology.orchestra.run.smithi038.stdout: "client_try_dentry_invalidate": "false", 2024-03-19T11:29:54.875 INFO:teuthology.orchestra.run.smithi038.stdout: "client_use_faked_inos": "false", 2024-03-19T11:29:54.875 INFO:teuthology.orchestra.run.smithi038.stdout: "client_use_random_mds": "false", 2024-03-19T11:29:54.875 INFO:teuthology.orchestra.run.smithi038.stdout: "clog_to_graylog": "false", 2024-03-19T11:29:54.876 INFO:teuthology.orchestra.run.smithi038.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-03-19T11:29:54.876 INFO:teuthology.orchestra.run.smithi038.stdout: "clog_to_graylog_port": "12201", 2024-03-19T11:29:54.876 INFO:teuthology.orchestra.run.smithi038.stdout: "clog_to_monitors": "default=true", 2024-03-19T11:29:54.876 INFO:teuthology.orchestra.run.smithi038.stdout: "clog_to_syslog": "false", 2024-03-19T11:29:54.876 INFO:teuthology.orchestra.run.smithi038.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-03-19T11:29:54.876 INFO:teuthology.orchestra.run.smithi038.stdout: "clog_to_syslog_level": "info", 2024-03-19T11:29:54.876 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "-", 2024-03-19T11:29:54.876 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_network": "", 2024-03-19T11:29:54.876 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_network_interface": "", 2024-03-19T11:29:54.876 INFO:teuthology.orchestra.run.smithi038.stdout: "compressor_zlib_isal": "false", 2024-03-19T11:29:54.877 INFO:teuthology.orchestra.run.smithi038.stdout: "compressor_zlib_level": "5", 2024-03-19T11:29:54.877 INFO:teuthology.orchestra.run.smithi038.stdout: "compressor_zlib_winsize": "-15", 2024-03-19T11:29:54.877 INFO:teuthology.orchestra.run.smithi038.stdout: "compressor_zstd_level": "1", 2024-03-19T11:29:54.877 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-03-19T11:29:54.877 INFO:teuthology.orchestra.run.smithi038.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-03-19T11:29:54.877 INFO:teuthology.orchestra.run.smithi038.stdout: "crimson_alien_op_num_threads": "6", 2024-03-19T11:29:54.877 INFO:teuthology.orchestra.run.smithi038.stdout: "crimson_alien_thread_cpu_cores": "", 2024-03-19T11:29:54.877 INFO:teuthology.orchestra.run.smithi038.stdout: "crimson_osd_obc_lru_size": "512", 2024-03-19T11:29:54.877 INFO:teuthology.orchestra.run.smithi038.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-03-19T11:29:54.877 INFO:teuthology.orchestra.run.smithi038.stdout: "crimson_seastar_cpu_cores": "", 2024-03-19T11:29:54.877 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_location": "", 2024-03-19T11:29:54.877 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_location_hook": "", 2024-03-19T11:29:54.877 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_location_hook_timeout": "10", 2024-03-19T11:29:54.878 INFO:teuthology.orchestra.run.smithi038.stdout: "daemonize": "false", 2024-03-19T11:29:54.878 INFO:teuthology.orchestra.run.smithi038.stdout: "daos_pool": "tank", 2024-03-19T11:29:54.878 INFO:teuthology.orchestra.run.smithi038.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-03-19T11:29:54.878 INFO:teuthology.orchestra.run.smithi038.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-03-19T11:29:54.878 INFO:teuthology.orchestra.run.smithi038.stdout: "dbstore_db_name_prefix": "dbstore", 2024-03-19T11:29:54.878 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_alienstore": "0/5", 2024-03-19T11:29:54.878 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_allow_any_pool_priority": "false", 2024-03-19T11:29:54.878 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_asok": "1/5", 2024-03-19T11:29:54.878 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_asok_assert_abort": "false", 2024-03-19T11:29:54.878 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_asserts_on_shutdown": "true", 2024-03-19T11:29:54.878 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_auth": "20/20", 2024-03-19T11:29:54.878 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_bdev": "1/3", 2024-03-19T11:29:54.878 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_bluefs": "1/5", 2024-03-19T11:29:54.879 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_bluestore": "1/5", 2024-03-19T11:29:54.879 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_buffer": "0/1", 2024-03-19T11:29:54.879 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_ceph_exporter": "1/5", 2024-03-19T11:29:54.879 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_cephfs_mirror": "0/5", 2024-03-19T11:29:54.879 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_cephsqlite": "0/5", 2024-03-19T11:29:54.879 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_client": "0/5", 2024-03-19T11:29:54.879 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_compressor": "1/5", 2024-03-19T11:29:54.879 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_context": "0/1", 2024-03-19T11:29:54.879 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_crimson_interrupt": "0/5", 2024-03-19T11:29:54.879 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_crush": "1/1", 2024-03-19T11:29:54.879 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_crypto": "1/5", 2024-03-19T11:29:54.880 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_cyanstore": "0/5", 2024-03-19T11:29:54.880 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_deliberately_leak_memory": "false", 2024-03-19T11:29:54.880 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_disable_randomized_ping": "false", 2024-03-19T11:29:54.880 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_dpdk": "1/5", 2024-03-19T11:29:54.880 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_eventtrace": "1/5", 2024-03-19T11:29:54.880 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_filer": "0/1", 2024-03-19T11:29:54.880 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_filestore": "1/3", 2024-03-19T11:29:54.880 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_finisher": "1/1", 2024-03-19T11:29:54.880 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_fuse": "1/5", 2024-03-19T11:29:54.880 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_heartbeat_testing_span": "0", 2024-03-19T11:29:54.880 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_heartbeatmap": "1/5", 2024-03-19T11:29:54.880 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_immutable_obj_cache": "0/5", 2024-03-19T11:29:54.880 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_javaclient": "1/5", 2024-03-19T11:29:54.880 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_journal": "1/3", 2024-03-19T11:29:54.881 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_journaler": "0/5", 2024-03-19T11:29:54.881 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_kstore": "1/5", 2024-03-19T11:29:54.881 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_lockdep": "0/1", 2024-03-19T11:29:54.881 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_mclock": "1/5", 2024-03-19T11:29:54.881 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_mds": "1/5", 2024-03-19T11:29:54.881 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_mds_balancer": "1/5", 2024-03-19T11:29:54.881 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_mds_locker": "1/5", 2024-03-19T11:29:54.881 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_mds_log": "1/5", 2024-03-19T11:29:54.881 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_mds_log_expire": "1/5", 2024-03-19T11:29:54.881 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_mds_migrator": "1/5", 2024-03-19T11:29:54.881 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_mds_quiesce": "3/5", 2024-03-19T11:29:54.881 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_memstore": "1/5", 2024-03-19T11:29:54.881 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_mgr": "2/5", 2024-03-19T11:29:54.881 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_mgrc": "1/5", 2024-03-19T11:29:54.882 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_mon": "20/20", 2024-03-19T11:29:54.882 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_monc": "0/10", 2024-03-19T11:29:54.882 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_ms": "1/1", 2024-03-19T11:29:54.882 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_none": "0/5", 2024-03-19T11:29:54.882 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_objclass": "0/5", 2024-03-19T11:29:54.882 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_objectcacher": "0/5", 2024-03-19T11:29:54.882 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_objecter": "0/1", 2024-03-19T11:29:54.882 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_optracker": "0/5", 2024-03-19T11:29:54.882 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_osd": "1/5", 2024-03-19T11:29:54.882 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_paxos": "20/20", 2024-03-19T11:29:54.882 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_perfcounter": "1/5", 2024-03-19T11:29:54.882 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_prioritycache": "1/5", 2024-03-19T11:29:54.882 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rados": "0/5", 2024-03-19T11:29:54.882 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rbd": "0/5", 2024-03-19T11:29:54.883 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rbd_mirror": "0/5", 2024-03-19T11:29:54.883 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rbd_pwl": "0/5", 2024-03-19T11:29:54.883 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rbd_replay": "0/5", 2024-03-19T11:29:54.883 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_refs": "0/0", 2024-03-19T11:29:54.883 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_reserver": "1/1", 2024-03-19T11:29:54.883 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rgw": "1/5", 2024-03-19T11:29:54.883 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rgw_access": "1/5", 2024-03-19T11:29:54.883 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rgw_datacache": "1/5", 2024-03-19T11:29:54.883 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rgw_dbstore": "1/5", 2024-03-19T11:29:54.883 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rgw_flight": "1/5", 2024-03-19T11:29:54.883 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rgw_lifecycle": "1/5", 2024-03-19T11:29:54.883 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rgw_sync": "1/5", 2024-03-19T11:29:54.883 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rocksdb": "4/5", 2024-03-19T11:29:54.883 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore": "0/5", 2024-03-19T11:29:54.884 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_backref": "0/5", 2024-03-19T11:29:54.884 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_cache": "0/5", 2024-03-19T11:29:54.884 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_cleaner": "0/5", 2024-03-19T11:29:54.884 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_device": "0/5", 2024-03-19T11:29:54.884 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_epm": "0/5", 2024-03-19T11:29:54.884 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-03-19T11:29:54.884 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_journal": "0/5", 2024-03-19T11:29:54.884 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_lba": "0/5", 2024-03-19T11:29:54.884 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_odata": "0/5", 2024-03-19T11:29:54.884 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_omap": "0/5", 2024-03-19T11:29:54.884 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_onode": "0/5", 2024-03-19T11:29:54.884 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_t": "0/5", 2024-03-19T11:29:54.884 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_tm": "0/5", 2024-03-19T11:29:54.885 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_striper": "0/1", 2024-03-19T11:29:54.885 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_test": "0/5", 2024-03-19T11:29:54.885 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_throttle": "1/1", 2024-03-19T11:29:54.885 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_timer": "0/1", 2024-03-19T11:29:54.885 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_tp": "0/5", 2024-03-19T11:29:54.885 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_trace": "1/5", 2024-03-19T11:29:54.885 INFO:teuthology.orchestra.run.smithi038.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-03-19T11:29:54.885 INFO:teuthology.orchestra.run.smithi038.stdout: "device_failure_prediction_mode": "none", 2024-03-19T11:29:54.885 INFO:teuthology.orchestra.run.smithi038.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-03-19T11:29:54.886 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_dir": "/usr/lib/ceph/erasure-code", 2024-03-19T11:29:54.886 INFO:teuthology.orchestra.run.smithi038.stdout: "err_to_graylog": "false", 2024-03-19T11:29:54.886 INFO:teuthology.orchestra.run.smithi038.stdout: "err_to_journald": "false", 2024-03-19T11:29:54.886 INFO:teuthology.orchestra.run.smithi038.stdout: "err_to_stderr": "true", 2024-03-19T11:29:54.886 INFO:teuthology.orchestra.run.smithi038.stdout: "err_to_syslog": "false", 2024-03-19T11:29:54.886 INFO:teuthology.orchestra.run.smithi038.stdout: "event_tracing": "false", 2024-03-19T11:29:54.886 INFO:teuthology.orchestra.run.smithi038.stdout: "exporter_addr": "0.0.0.0", 2024-03-19T11:29:54.886 INFO:teuthology.orchestra.run.smithi038.stdout: "exporter_http_port": "9926", 2024-03-19T11:29:54.886 INFO:teuthology.orchestra.run.smithi038.stdout: "exporter_prio_limit": "5", 2024-03-19T11:29:54.887 INFO:teuthology.orchestra.run.smithi038.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-03-19T11:29:54.887 INFO:teuthology.orchestra.run.smithi038.stdout: "exporter_sort_metrics": "true", 2024-03-19T11:29:54.887 INFO:teuthology.orchestra.run.smithi038.stdout: "exporter_stats_period": "5", 2024-03-19T11:29:54.887 INFO:teuthology.orchestra.run.smithi038.stdout: "fake_statfs_for_testing": "0", 2024-03-19T11:29:54.887 INFO:teuthology.orchestra.run.smithi038.stdout: "fatal_signal_handlers": "true", 2024-03-19T11:29:54.887 INFO:teuthology.orchestra.run.smithi038.stdout: "filer_max_purge_ops": "10", 2024-03-19T11:29:54.887 INFO:teuthology.orchestra.run.smithi038.stdout: "filer_max_truncate_ops": "128", 2024-03-19T11:29:54.887 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_apply_finisher_threads": "1", 2024-03-19T11:29:54.887 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_blackhole": "false", 2024-03-19T11:29:54.887 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_btrfs_clone_range": "true", 2024-03-19T11:29:54.887 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_btrfs_snap": "true", 2024-03-19T11:29:54.887 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_caller_concurrency": "10", 2024-03-19T11:29:54.887 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_collect_device_partition_information": "true", 2024-03-19T11:29:54.887 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_commit_timeout": "600.000000", 2024-03-19T11:29:54.888 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_debug_inject_read_err": "false", 2024-03-19T11:29:54.888 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_debug_omap_check": "false", 2024-03-19T11:29:54.888 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_debug_random_read_err": "0.000000", 2024-03-19T11:29:54.888 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_debug_verify_split": "false", 2024-03-19T11:29:54.888 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_dump_file": "", 2024-03-19T11:29:54.888 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-03-19T11:29:54.888 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-03-19T11:29:54.888 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_fadvise": "true", 2024-03-19T11:29:54.888 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_fail_eio": "true", 2024-03-19T11:29:54.888 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_fd_cache_shards": "16", 2024-03-19T11:29:54.888 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_fd_cache_size": "128", 2024-03-19T11:29:54.888 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_fiemap": "false", 2024-03-19T11:29:54.888 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_fiemap_threshold": "4096", 2024-03-19T11:29:54.888 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-03-19T11:29:54.889 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_index_retry_probability": "0.000000", 2024-03-19T11:29:54.889 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_inject_stall": "0", 2024-03-19T11:29:54.889 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_journal_parallel": "false", 2024-03-19T11:29:54.889 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_journal_trailing": "false", 2024-03-19T11:29:54.889 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_journal_writeahead": "false", 2024-03-19T11:29:54.889 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_kill_at": "0", 2024-03-19T11:29:54.889 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-03-19T11:29:54.889 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_inline_xattr_size": "0", 2024-03-19T11:29:54.889 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-03-19T11:29:54.889 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-03-19T11:29:54.889 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-03-19T11:29:54.889 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_inline_xattrs": "0", 2024-03-19T11:29:54.889 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-03-19T11:29:54.890 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_inline_xattrs_other": "2", 2024-03-19T11:29:54.890 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-03-19T11:29:54.890 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_sync_interval": "5.000000", 2024-03-19T11:29:54.890 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_xattr_value_size": "0", 2024-03-19T11:29:54.890 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-03-19T11:29:54.890 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-03-19T11:29:54.890 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-03-19T11:29:54.890 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_merge_threshold": "-10", 2024-03-19T11:29:54.890 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_min_sync_interval": "0.010000", 2024-03-19T11:29:54.890 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_odsync_write": "false", 2024-03-19T11:29:54.890 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_omap_backend": "rocksdb", 2024-03-19T11:29:54.890 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_omap_backend_path": "", 2024-03-19T11:29:54.890 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_omap_header_cache_size": "1024", 2024-03-19T11:29:54.890 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_ondisk_finisher_threads": "1", 2024-03-19T11:29:54.891 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-03-19T11:29:54.891 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_op_thread_timeout": "60", 2024-03-19T11:29:54.891 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_op_threads": "2", 2024-03-19T11:29:54.891 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_punch_hole": "false", 2024-03-19T11:29:54.891 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-03-19T11:29:54.891 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-03-19T11:29:54.891 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-03-19T11:29:54.891 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-03-19T11:29:54.891 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-03-19T11:29:54.891 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_queue_max_bytes": "104857600", 2024-03-19T11:29:54.891 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-03-19T11:29:54.891 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-03-19T11:29:54.891 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-03-19T11:29:54.892 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_queue_max_ops": "50", 2024-03-19T11:29:54.892 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-03-19T11:29:54.892 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_seek_data_hole": "false", 2024-03-19T11:29:54.892 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_sloppy_crc": "false", 2024-03-19T11:29:54.892 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-03-19T11:29:54.892 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_splice": "false", 2024-03-19T11:29:54.892 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_split_multiple": "2", 2024-03-19T11:29:54.892 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_split_rand_factor": "20", 2024-03-19T11:29:54.892 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_update_to": "1000", 2024-03-19T11:29:54.892 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-03-19T11:29:54.892 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-03-19T11:29:54.892 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-03-19T11:29:54.892 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-03-19T11:29:54.892 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-03-19T11:29:54.893 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-03-19T11:29:54.893 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_enable": "true", 2024-03-19T11:29:54.893 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-03-19T11:29:54.893 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-03-19T11:29:54.893 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-03-19T11:29:54.893 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-03-19T11:29:54.893 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-03-19T11:29:54.893 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-03-19T11:29:54.893 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_xfs_extsize": "false", 2024-03-19T11:29:54.893 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_zfs_snap": "false", 2024-03-19T11:29:54.893 INFO:teuthology.orchestra.run.smithi038.stdout: "fio_dir": "/tmp/fio", 2024-03-19T11:29:54.893 INFO:teuthology.orchestra.run.smithi038.stdout: "fsid": "246bcf68-c431-4aa7-b787-5e9b2d9519d6", 2024-03-19T11:29:54.894 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_allow_other": "true", 2024-03-19T11:29:54.894 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_atomic_o_trunc": "true", 2024-03-19T11:29:54.894 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_big_writes": "true", 2024-03-19T11:29:54.894 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_debug": "false", 2024-03-19T11:29:54.894 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_default_permissions": "false", 2024-03-19T11:29:54.894 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_disable_pagecache": "false", 2024-03-19T11:29:54.894 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_max_write": "0", 2024-03-19T11:29:54.894 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_multithreaded": "true", 2024-03-19T11:29:54.894 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_require_active_mds": "true", 2024-03-19T11:29:54.894 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_set_user_groups": "true", 2024-03-19T11:29:54.894 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_splice_move": "true", 2024-03-19T11:29:54.894 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_splice_read": "true", 2024-03-19T11:29:54.894 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_splice_write": "true", 2024-03-19T11:29:54.894 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_syncfs_on_mksnap": "true", 2024-03-19T11:29:54.895 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_use_invalidate_cb": "true", 2024-03-19T11:29:54.895 INFO:teuthology.orchestra.run.smithi038.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-03-19T11:29:54.895 INFO:teuthology.orchestra.run.smithi038.stdout: "gss_target_name": "ceph", 2024-03-19T11:29:54.895 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_file": "", 2024-03-19T11:29:54.895 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_inject_failure": "0", 2024-03-19T11:29:54.895 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_interval": "5", 2024-03-19T11:29:54.895 INFO:teuthology.orchestra.run.smithi038.stdout: "host": "smithi038", 2024-03-19T11:29:54.895 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-03-19T11:29:54.895 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-03-19T11:29:54.895 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_max_size": "1073741824", 2024-03-19T11:29:54.895 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-03-19T11:29:54.895 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-03-19T11:29:54.895 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-03-19T11:29:54.895 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-03-19T11:29:54.896 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-03-19T11:29:54.896 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-03-19T11:29:54.896 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-03-19T11:29:54.896 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-03-19T11:29:54.896 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-03-19T11:29:54.896 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_watermark": "0.900000", 2024-03-19T11:29:54.896 INFO:teuthology.orchestra.run.smithi038.stdout: "inject_early_sigterm": "false", 2024-03-19T11:29:54.896 INFO:teuthology.orchestra.run.smithi038.stdout: "jaeger_agent_port": "6799", 2024-03-19T11:29:54.896 INFO:teuthology.orchestra.run.smithi038.stdout: "jaeger_tracing_enable": "false", 2024-03-19T11:29:54.896 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_aio": "true", 2024-03-19T11:29:54.896 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_align_min_size": "65536", 2024-03-19T11:29:54.896 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_block_align": "true", 2024-03-19T11:29:54.896 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_block_size": "4096", 2024-03-19T11:29:54.896 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_dio": "true", 2024-03-19T11:29:54.897 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_discard": "false", 2024-03-19T11:29:54.897 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_force_aio": "false", 2024-03-19T11:29:54.897 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_ignore_corruption": "false", 2024-03-19T11:29:54.897 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_max_write_bytes": "10485760", 2024-03-19T11:29:54.897 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_max_write_entries": "100", 2024-03-19T11:29:54.897 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_replay_from": "0", 2024-03-19T11:29:54.897 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_throttle_high_multiple": "0.000000", 2024-03-19T11:29:54.897 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-03-19T11:29:54.897 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-03-19T11:29:54.897 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_throttle_max_multiple": "0.000000", 2024-03-19T11:29:54.897 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_write_header_frequency": "0", 2024-03-19T11:29:54.897 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_zero_on_create": "false", 2024-03-19T11:29:54.898 INFO:teuthology.orchestra.run.smithi038.stdout: "journaler_prefetch_periods": "10", 2024-03-19T11:29:54.898 INFO:teuthology.orchestra.run.smithi038.stdout: "journaler_prezero_periods": "5", 2024-03-19T11:29:54.898 INFO:teuthology.orchestra.run.smithi038.stdout: "journaler_write_head_interval": "15", 2024-03-19T11:29:54.898 INFO:teuthology.orchestra.run.smithi038.stdout: "key": "", 2024-03-19T11:29:54.898 INFO:teuthology.orchestra.run.smithi038.stdout: "keyfile": "", 2024-03-19T11:29:54.898 INFO:teuthology.orchestra.run.smithi038.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-03-19T11:29:54.898 INFO:teuthology.orchestra.run.smithi038.stdout: "kstore_backend": "rocksdb", 2024-03-19T11:29:54.898 INFO:teuthology.orchestra.run.smithi038.stdout: "kstore_default_stripe_size": "65536", 2024-03-19T11:29:54.898 INFO:teuthology.orchestra.run.smithi038.stdout: "kstore_fsck_on_mount": "false", 2024-03-19T11:29:54.898 INFO:teuthology.orchestra.run.smithi038.stdout: "kstore_fsck_on_mount_deep": "true", 2024-03-19T11:29:54.898 INFO:teuthology.orchestra.run.smithi038.stdout: "kstore_max_bytes": "67108864", 2024-03-19T11:29:54.898 INFO:teuthology.orchestra.run.smithi038.stdout: "kstore_max_ops": "512", 2024-03-19T11:29:54.898 INFO:teuthology.orchestra.run.smithi038.stdout: "kstore_nid_prealloc": "1024", 2024-03-19T11:29:54.898 INFO:teuthology.orchestra.run.smithi038.stdout: "kstore_onode_map_size": "1024", 2024-03-19T11:29:54.899 INFO:teuthology.orchestra.run.smithi038.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-03-19T11:29:54.899 INFO:teuthology.orchestra.run.smithi038.stdout: "kstore_sync_submit_transaction": "false", 2024-03-19T11:29:54.899 INFO:teuthology.orchestra.run.smithi038.stdout: "kstore_sync_transaction": "false", 2024-03-19T11:29:54.899 INFO:teuthology.orchestra.run.smithi038.stdout: "librados_thread_count": "2", 2024-03-19T11:29:54.899 INFO:teuthology.orchestra.run.smithi038.stdout: "lockdep": "false", 2024-03-19T11:29:54.899 INFO:teuthology.orchestra.run.smithi038.stdout: "lockdep_force_backtrace": "false", 2024-03-19T11:29:54.899 INFO:teuthology.orchestra.run.smithi038.stdout: "log_coarse_timestamps": "true", 2024-03-19T11:29:54.899 INFO:teuthology.orchestra.run.smithi038.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-03-19T11:29:54.899 INFO:teuthology.orchestra.run.smithi038.stdout: "log_flush_on_exit": "false", 2024-03-19T11:29:54.899 INFO:teuthology.orchestra.run.smithi038.stdout: "log_graylog_host": "127.0.0.1", 2024-03-19T11:29:54.899 INFO:teuthology.orchestra.run.smithi038.stdout: "log_graylog_port": "12201", 2024-03-19T11:29:54.899 INFO:teuthology.orchestra.run.smithi038.stdout: "log_max_new": "1000", 2024-03-19T11:29:54.899 INFO:teuthology.orchestra.run.smithi038.stdout: "log_max_recent": "10000", 2024-03-19T11:29:54.899 INFO:teuthology.orchestra.run.smithi038.stdout: "log_stderr_prefix": "", 2024-03-19T11:29:54.900 INFO:teuthology.orchestra.run.smithi038.stdout: "log_stop_at_utilization": "0.970000", 2024-03-19T11:29:54.900 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": "true", 2024-03-19T11:29:54.900 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_graylog": "false", 2024-03-19T11:29:54.900 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_journald": "false", 2024-03-19T11:29:54.900 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_stderr": "false", 2024-03-19T11:29:54.900 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_syslog": "false", 2024-03-19T11:29:54.900 INFO:teuthology.orchestra.run.smithi038.stdout: "max_rotating_auth_attempts": "10", 2024-03-19T11:29:54.900 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-03-19T11:29:54.900 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_action_on_write_error": "1", 2024-03-19T11:29:54.900 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_alternate_name_max": "8192", 2024-03-19T11:29:54.900 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_asio_thread_count": "2", 2024-03-19T11:29:54.900 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_export_pin": "true", 2024-03-19T11:29:54.901 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_fragment_dirs": "true", 2024-03-19T11:29:54.901 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-03-19T11:29:54.901 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_fragment_interval": "5", 2024-03-19T11:29:54.901 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_fragment_size_max": "100000", 2024-03-19T11:29:54.901 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_idle_threshold": "0.000000", 2024-03-19T11:29:54.901 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_interval": "10", 2024-03-19T11:29:54.901 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_max": "-1", 2024-03-19T11:29:54.901 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_max_until": "-1", 2024-03-19T11:29:54.901 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_merge_size": "50", 2024-03-19T11:29:54.901 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_midchunk": "0.300000", 2024-03-19T11:29:54.901 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_min_rebalance": "0.100000", 2024-03-19T11:29:54.901 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_min_start": "0.200000", 2024-03-19T11:29:54.902 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_minchunk": "0.001000", 2024-03-19T11:29:54.902 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_mode": "0", 2024-03-19T11:29:54.902 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_need_max": "1.200000", 2024-03-19T11:29:54.902 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_need_min": "0.800000", 2024-03-19T11:29:54.902 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_overload_epochs": "2", 2024-03-19T11:29:54.902 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-03-19T11:29:54.902 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_sample_interval": "3.000000", 2024-03-19T11:29:54.902 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_split_bits": "3", 2024-03-19T11:29:54.902 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_split_rd": "25000.000000", 2024-03-19T11:29:54.902 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_split_size": "10000", 2024-03-19T11:29:54.902 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_split_wr": "10000.000000", 2024-03-19T11:29:54.903 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_target_decay": "10.000000", 2024-03-19T11:29:54.903 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-03-19T11:29:54.903 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_beacon_grace": "15.000000", 2024-03-19T11:29:54.903 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_beacon_interval": "4.000000", 2024-03-19T11:29:54.903 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_beacon_mon_down_grace": "60", 2024-03-19T11:29:54.903 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cache_memory_limit": "4294967296", 2024-03-19T11:29:54.903 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cache_mid": "0.700000", 2024-03-19T11:29:54.903 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cache_release_free_interval": "10", 2024-03-19T11:29:54.903 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cache_reservation": "0.050000", 2024-03-19T11:29:54.903 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-03-19T11:29:54.903 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cache_trim_interval": "1", 2024-03-19T11:29:54.903 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cache_trim_threshold": "262144", 2024-03-19T11:29:54.904 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-03-19T11:29:54.904 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-03-19T11:29:54.904 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_client_delegate_inos_pct": "50", 2024-03-19T11:29:54.904 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_client_prealloc_inos": "1000", 2024-03-19T11:29:54.904 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-03-19T11:29:54.904 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_connect_bootstrapping": "false", 2024-03-19T11:29:54.904 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_damage_table_max_entries": "10000", 2024-03-19T11:29:54.904 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-03-19T11:29:54.904 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_debug_auth_pins": "false", 2024-03-19T11:29:54.904 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_debug_frag": "false", 2024-03-19T11:29:54.904 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_debug_scatterstat": "false", 2024-03-19T11:29:54.904 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_debug_subtrees": "false", 2024-03-19T11:29:54.904 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_decay_halflife": "5.000000", 2024-03-19T11:29:54.904 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_default_dir_hash": "2", 2024-03-19T11:29:54.904 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_defer_session_stale": "true", 2024-03-19T11:29:54.905 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_deny_all_reconnect": "false", 2024-03-19T11:29:54.905 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_dir_keys_per_op": "16384", 2024-03-19T11:29:54.905 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_dir_max_commit_size": "10", 2024-03-19T11:29:54.905 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_dir_max_entries": "0", 2024-03-19T11:29:54.905 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_dir_prefetch": "true", 2024-03-19T11:29:54.905 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_dirstat_min_interval": "1.000000", 2024-03-19T11:29:54.905 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_dump_cache_after_rejoin": "false", 2024-03-19T11:29:54.905 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_dump_cache_on_map": "false", 2024-03-19T11:29:54.905 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_dump_cache_threshold_file": "0", 2024-03-19T11:29:54.905 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-03-19T11:29:54.905 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_early_reply": "true", 2024-03-19T11:29:54.905 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_enable_op_tracker": "true", 2024-03-19T11:29:54.906 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_enforce_unique_name": "true", 2024-03-19T11:29:54.906 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_export_ephemeral_distributed": "true", 2024-03-19T11:29:54.906 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-03-19T11:29:54.906 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_export_ephemeral_random": "true", 2024-03-19T11:29:54.906 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-03-19T11:29:54.906 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-03-19T11:29:54.906 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_forward_all_requests_to_auth": "false", 2024-03-19T11:29:54.906 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-03-19T11:29:54.906 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-03-19T11:29:54.906 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-03-19T11:29:54.906 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-03-19T11:29:54.906 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_health_cache_threshold": "1.500000", 2024-03-19T11:29:54.906 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_health_summarize_threshold": "10", 2024-03-19T11:29:54.906 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_heartbeat_grace": "15.000000", 2024-03-19T11:29:54.907 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_heartbeat_reset_grace": "1000", 2024-03-19T11:29:54.907 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_inject_health_dummy": "false", 2024-03-19T11:29:54.907 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-03-19T11:29:54.907 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_inject_migrator_session_race": "false", 2024-03-19T11:29:54.907 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-03-19T11:29:54.907 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-03-19T11:29:54.907 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-03-19T11:29:54.907 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_join_fs": "", 2024-03-19T11:29:54.907 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_journal_format": "1", 2024-03-19T11:29:54.907 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_kill_create_at": "0", 2024-03-19T11:29:54.907 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_kill_export_at": "0", 2024-03-19T11:29:54.907 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_kill_import_at": "0", 2024-03-19T11:29:54.907 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_kill_journal_at": "0", 2024-03-19T11:29:54.908 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_kill_journal_expire_at": "0", 2024-03-19T11:29:54.908 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_kill_journal_replay_at": "0", 2024-03-19T11:29:54.908 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_kill_link_at": "0", 2024-03-19T11:29:54.908 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_kill_mdstable_at": "0", 2024-03-19T11:29:54.908 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_kill_openc_at": "0", 2024-03-19T11:29:54.908 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_kill_rename_at": "0", 2024-03-19T11:29:54.908 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_kill_shutdown_at": "0", 2024-03-19T11:29:54.908 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-03-19T11:29:54.908 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_event_large_threshold": "524288", 2024-03-19T11:29:54.908 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_events_per_segment": "1024", 2024-03-19T11:29:54.908 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_major_segment_event_ratio": "12", 2024-03-19T11:29:54.908 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_max_events": "-1", 2024-03-19T11:29:54.908 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_max_segments": "128", 2024-03-19T11:29:54.909 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_pause": "false", 2024-03-19T11:29:54.909 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_segment_size": "0", 2024-03-19T11:29:54.909 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_skip_corrupt_events": "false", 2024-03-19T11:29:54.909 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_skip_unbounded_events": "false", 2024-03-19T11:29:54.909 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-03-19T11:29:54.909 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_trim_threshold": "128", 2024-03-19T11:29:54.909 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-03-19T11:29:54.909 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_warn_factor": "2.000000", 2024-03-19T11:29:54.909 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_max_caps_per_client": "1048576", 2024-03-19T11:29:54.909 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_max_completed_flushes": "100000", 2024-03-19T11:29:54.909 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_max_completed_requests": "100000", 2024-03-19T11:29:54.909 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_max_export_size": "20971520", 2024-03-19T11:29:54.909 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_max_file_recover": "32", 2024-03-19T11:29:54.909 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_max_purge_files": "64", 2024-03-19T11:29:54.910 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_max_purge_ops": "8192", 2024-03-19T11:29:54.910 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-03-19T11:29:54.910 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-03-19T11:29:54.910 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_max_snaps_per_dir": "100", 2024-03-19T11:29:54.910 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_metrics_update_interval": "2", 2024-03-19T11:29:54.910 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_min_caps_per_client": "100", 2024-03-19T11:29:54.910 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_min_caps_working_set": "10000", 2024-03-19T11:29:54.910 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-03-19T11:29:54.910 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_numa_node": "-1", 2024-03-19T11:29:54.910 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-03-19T11:29:54.910 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_op_complaint_time": "30.000000", 2024-03-19T11:29:54.910 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_op_history_duration": "600", 2024-03-19T11:29:54.910 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_op_history_size": "20", 2024-03-19T11:29:54.910 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_op_history_slow_op_size": "20", 2024-03-19T11:29:54.911 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_op_history_slow_op_threshold": "10", 2024-03-19T11:29:54.911 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_op_log_threshold": "5", 2024-03-19T11:29:54.911 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_ping_grace": "15", 2024-03-19T11:29:54.911 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_ping_interval": "5", 2024-03-19T11:29:54.911 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-03-19T11:29:54.911 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-03-19T11:29:54.911 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_recall_max_caps": "30000", 2024-03-19T11:29:54.911 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-03-19T11:29:54.911 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_recall_max_decay_threshold": "131072", 2024-03-19T11:29:54.911 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-03-19T11:29:54.911 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_recall_warning_threshold": "262144", 2024-03-19T11:29:54.911 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_reconnect_timeout": "45.000000", 2024-03-19T11:29:54.911 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_replay_interval": "1.000000", 2024-03-19T11:29:54.912 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-03-19T11:29:54.912 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-03-19T11:29:54.912 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_root_ino_gid": "0", 2024-03-19T11:29:54.912 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_root_ino_uid": "0", 2024-03-19T11:29:54.912 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-03-19T11:29:54.912 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_session_blocklist_on_evict": "true", 2024-03-19T11:29:54.912 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_session_blocklist_on_timeout": "true", 2024-03-19T11:29:54.912 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-03-19T11:29:54.912 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-03-19T11:29:54.912 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-03-19T11:29:54.912 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-03-19T11:29:54.912 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-03-19T11:29:54.913 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_session_metadata_threshold": "16777216", 2024-03-19T11:29:54.913 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-03-19T11:29:54.913 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_shutdown_check": "0", 2024-03-19T11:29:54.913 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_skip_ino": "0", 2024-03-19T11:29:54.913 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_sleep_rank_change": "0.000000", 2024-03-19T11:29:54.913 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_snap_max_uid": "4294967294", 2024-03-19T11:29:54.913 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_snap_min_uid": "0", 2024-03-19T11:29:54.913 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_snap_rstat": "false", 2024-03-19T11:29:54.913 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_standby_replay_damaged": "false", 2024-03-19T11:29:54.913 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_symlink_recovery": "true", 2024-03-19T11:29:54.913 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_task_status_update_interval": "2.000000", 2024-03-19T11:29:54.913 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_thrash_exports": "0", 2024-03-19T11:29:54.913 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_thrash_fragments": "0", 2024-03-19T11:29:54.914 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_tick_interval": "5.000000", 2024-03-19T11:29:54.914 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_valgrind_exit": "false", 2024-03-19T11:29:54.914 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_verify_backtrace": "1", 2024-03-19T11:29:54.914 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_verify_scatter": "false", 2024-03-19T11:29:54.914 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_wipe_ino_prealloc": "false", 2024-03-19T11:29:54.914 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_wipe_sessions": "false", 2024-03-19T11:29:54.914 INFO:teuthology.orchestra.run.smithi038.stdout: "mempool_debug": "false", 2024-03-19T11:29:54.914 INFO:teuthology.orchestra.run.smithi038.stdout: "memstore_debug_omit_block_device_write": "false", 2024-03-19T11:29:54.914 INFO:teuthology.orchestra.run.smithi038.stdout: "memstore_device_bytes": "1073741824", 2024-03-19T11:29:54.914 INFO:teuthology.orchestra.run.smithi038.stdout: "memstore_page_set": "false", 2024-03-19T11:29:54.914 INFO:teuthology.orchestra.run.smithi038.stdout: "memstore_page_size": "65536", 2024-03-19T11:29:54.914 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_client_bytes": "134217728", 2024-03-19T11:29:54.914 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_client_messages": "512", 2024-03-19T11:29:54.914 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-03-19T11:29:54.915 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_connect_retry_interval": "1.000000", 2024-03-19T11:29:54.915 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-03-19T11:29:54.915 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-03-19T11:29:54.915 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_disabled_modules": "", 2024-03-19T11:29:54.915 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-03-19T11:29:54.915 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_max_pg_num_change": "128", 2024-03-19T11:29:54.915 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_mds_bytes": "134217728", 2024-03-19T11:29:54.915 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_mds_messages": "128", 2024-03-19T11:29:54.915 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-03-19T11:29:54.915 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_mon_bytes": "134217728", 2024-03-19T11:29:54.915 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_mon_messages": "128", 2024-03-19T11:29:54.915 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_osd_bytes": "536870912", 2024-03-19T11:29:54.915 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_osd_messages": "8192", 2024-03-19T11:29:54.915 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_pool": "true", 2024-03-19T11:29:54.916 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_service_beacon_grace": "60.000000", 2024-03-19T11:29:54.916 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_standby_modules": "true", 2024-03-19T11:29:54.916 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_stats_period": "5", 2024-03-19T11:29:54.916 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_stats_threshold": "5", 2024-03-19T11:29:54.916 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_tick_period": "2", 2024-03-19T11:29:54.916 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-03-19T11:29:54.916 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_accept_timeout_factor": "2.000000", 2024-03-19T11:29:54.916 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_allow_pool_delete": "true", 2024-03-19T11:29:54.916 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_allow_pool_size_one": "true", 2024-03-19T11:29:54.916 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_auth_validate_all_caps": "true", 2024-03-19T11:29:54.916 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-03-19T11:29:54.916 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-03-19T11:29:54.916 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_client_bytes": "104857600", 2024-03-19T11:29:54.916 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_client_directed_command_retry": "5", 2024-03-19T11:29:54.916 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_client_hunt_interval": "3.000000", 2024-03-19T11:29:54.917 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-03-19T11:29:54.917 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-03-19T11:29:54.917 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-03-19T11:29:54.917 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_client_hunt_parallel": "3", 2024-03-19T11:29:54.917 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_client_log_interval": "1.000000", 2024-03-19T11:29:54.917 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-03-19T11:29:54.917 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_client_ping_interval": "10.000000", 2024-03-19T11:29:54.917 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_client_ping_timeout": "30.000000", 2024-03-19T11:29:54.917 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_client_target_rank": "-1", 2024-03-19T11:29:54.917 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_clock_drift_allowed": "1.000000", 2024-03-19T11:29:54.917 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-03-19T11:29:54.917 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-03-19T11:29:54.917 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cluster_log_file_level": "debug", 2024-03-19T11:29:54.918 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cluster_log_to_file": "true", 2024-03-19T11:29:54.918 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cluster_log_to_graylog": "false", 2024-03-19T11:29:54.918 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-03-19T11:29:54.918 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-03-19T11:29:54.918 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cluster_log_to_journald": "false", 2024-03-19T11:29:54.918 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cluster_log_to_stderr": "false", 2024-03-19T11:29:54.918 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-03-19T11:29:54.918 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-03-19T11:29:54.918 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-03-19T11:29:54.918 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_compact_on_bootstrap": "false", 2024-03-19T11:29:54.918 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_compact_on_start": "false", 2024-03-19T11:29:54.918 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_compact_on_trim": "true", 2024-03-19T11:29:54.918 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_con_tracker_persist_interval": "10", 2024-03-19T11:29:54.919 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_con_tracker_score_halflife": "43200", 2024-03-19T11:29:54.919 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_config_key_max_entry_size": "65536", 2024-03-19T11:29:54.919 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cpu_threads": "4", 2024-03-19T11:29:54.919 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_crush_min_required_version": "hammer", 2024-03-19T11:29:54.919 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_daemon_bytes": "419430400", 2024-03-19T11:29:54.919 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-03-19T11:29:54.919 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_data_avail_crit": "5", 2024-03-19T11:29:54.919 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_data_avail_warn": "5", 2024-03-19T11:29:54.919 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_data_size_warn": "16106127360", 2024-03-19T11:29:54.919 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_debug_block_osdmap_trim": "false", 2024-03-19T11:29:54.919 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-03-19T11:29:54.919 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_debug_dump_json": "false", 2024-03-19T11:29:54.920 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-03-19T11:29:54.920 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_debug_dump_transactions": "false", 2024-03-19T11:29:54.920 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_debug_extra_checks": "false", 2024-03-19T11:29:54.920 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-03-19T11:29:54.920 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-03-19T11:29:54.920 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_debug_no_require_reef": "false", 2024-03-19T11:29:54.920 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_debug_no_require_squid": "false", 2024-03-19T11:29:54.920 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-03-19T11:29:54.920 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_delta_reset_interval": "10.000000", 2024-03-19T11:29:54.920 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_dns_srv_name": "ceph-mon", 2024-03-19T11:29:54.920 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_down_mkfs_grace": "120", 2024-03-19T11:29:54.921 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_election_default_strategy": "1", 2024-03-19T11:29:54.921 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_election_timeout": "5.000000", 2024-03-19T11:29:54.921 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-03-19T11:29:54.921 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_elector_ping_divisor": "2", 2024-03-19T11:29:54.921 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_elector_ping_timeout": "2.000000", 2024-03-19T11:29:54.921 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_enable_op_tracker": "true", 2024-03-19T11:29:54.921 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_fake_pool_delete": "false", 2024-03-19T11:29:54.921 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_force_quorum_join": "false", 2024-03-19T11:29:54.921 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_globalid_prealloc": "10000", 2024-03-19T11:29:54.921 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_health_detail_to_clog": "false", 2024-03-19T11:29:54.921 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_health_log_update_period": "5", 2024-03-19T11:29:54.921 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_health_max_detail": "50", 2024-03-19T11:29:54.921 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_health_to_clog": "true", 2024-03-19T11:29:54.922 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_health_to_clog_interval": "600", 2024-03-19T11:29:54.922 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-03-19T11:29:54.922 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_host": "172.21.15.38,[v2:172.21.15.38:3301,v1:172.21.15.38:6790],172.21.15.172", 2024-03-19T11:29:54.922 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_host_override": "", 2024-03-19T11:29:54.922 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_initial_members": "", 2024-03-19T11:29:54.922 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-03-19T11:29:54.922 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-03-19T11:29:54.922 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-03-19T11:29:54.922 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-03-19T11:29:54.922 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_keyvaluedb": "rocksdb", 2024-03-19T11:29:54.922 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_lease": "5.000000", 2024-03-19T11:29:54.922 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-03-19T11:29:54.922 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-03-19T11:29:54.922 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_log_full_interval": "50", 2024-03-19T11:29:54.923 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_log_max": "10000", 2024-03-19T11:29:54.923 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_log_max_summary": "50", 2024-03-19T11:29:54.923 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_max_log_entries_per_event": "4096", 2024-03-19T11:29:54.923 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_max_log_epochs": "500", 2024-03-19T11:29:54.923 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_max_mdsmap_epochs": "500", 2024-03-19T11:29:54.923 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_max_mgrmap_epochs": "500", 2024-03-19T11:29:54.923 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_max_osd": "10000", 2024-03-19T11:29:54.923 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_max_pg_per_osd": "10000", 2024-03-19T11:29:54.923 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_max_pool_pg_num": "65536", 2024-03-19T11:29:54.923 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-03-19T11:29:54.923 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-03-19T11:29:54.924 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_mds_force_trim_to": "0", 2024-03-19T11:29:54.924 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_mds_skip_sanity": "false", 2024-03-19T11:29:54.924 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_memory_autotune": "true", 2024-03-19T11:29:54.924 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_memory_target": "2147483648", 2024-03-19T11:29:54.924 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_mgr_beacon_grace": "30", 2024-03-19T11:29:54.924 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-03-19T11:29:54.924 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_mgr_digest_period": "5", 2024-03-19T11:29:54.924 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_mgr_inactive_grace": "60", 2024-03-19T11:29:54.924 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_mgr_mkfs_grace": "240", 2024-03-19T11:29:54.924 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-03-19T11:29:54.924 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_min_osdmap_epochs": "50", 2024-03-19T11:29:54.924 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_op_complaint_time": "30", 2024-03-19T11:29:54.924 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_op_history_duration": "600", 2024-03-19T11:29:54.925 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_op_history_size": "20", 2024-03-19T11:29:54.925 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_op_history_slow_op_size": "20", 2024-03-19T11:29:54.925 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_op_history_slow_op_threshold": "10", 2024-03-19T11:29:54.925 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_op_log_threshold": "5", 2024-03-19T11:29:54.925 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-03-19T11:29:54.925 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-03-19T11:29:54.925 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-03-19T11:29:54.925 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_auto_mark_in": "false", 2024-03-19T11:29:54.925 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_auto_mark_new_in": "true", 2024-03-19T11:29:54.925 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-03-19T11:29:54.925 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-03-19T11:29:54.925 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_cache_size": "500", 2024-03-19T11:29:54.926 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_cache_size_min": "134217728", 2024-03-19T11:29:54.926 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_crush_smoke_test": "true", 2024-03-19T11:29:54.926 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_destroyed_out_interval": "600", 2024-03-19T11:29:54.926 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_down_out_interval": "600", 2024-03-19T11:29:54.926 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-03-19T11:29:54.926 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-03-19T11:29:54.926 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_force_trim_to": "0", 2024-03-19T11:29:54.926 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_full_ratio": "0.950000", 2024-03-19T11:29:54.926 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-03-19T11:29:54.926 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_laggy_halflife": "3600", 2024-03-19T11:29:54.926 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_laggy_max_interval": "300", 2024-03-19T11:29:54.926 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_laggy_weight": "0.300000", 2024-03-19T11:29:54.927 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-03-19T11:29:54.927 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_max_creating_pgs": "1024", 2024-03-19T11:29:54.927 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_max_initial_pgs": "1024", 2024-03-19T11:29:54.927 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_min_down_reporters": "2", 2024-03-19T11:29:54.927 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-03-19T11:29:54.927 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-03-19T11:29:54.927 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-03-19T11:29:54.927 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_prime_pg_temp": "true", 2024-03-19T11:29:54.927 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-03-19T11:29:54.927 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-03-19T11:29:54.927 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_report_timeout": "900", 2024-03-19T11:29:54.927 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-03-19T11:29:54.927 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-03-19T11:29:54.927 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_warn_num_repaired": "10", 2024-03-19T11:29:54.928 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_warn_op_age": "32.000000", 2024-03-19T11:29:54.928 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-03-19T11:29:54.928 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osdmap_full_prune_interval": "2", 2024-03-19T11:29:54.928 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osdmap_full_prune_min": "15", 2024-03-19T11:29:54.928 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-03-19T11:29:54.928 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-03-19T11:29:54.928 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_pg_stuck_threshold": "60", 2024-03-19T11:29:54.928 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-03-19T11:29:54.928 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_pg_warn_min_objects": "10000", 2024-03-19T11:29:54.928 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_pg_warn_min_per_osd": "0", 2024-03-19T11:29:54.928 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-03-19T11:29:54.928 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_pool_quota_crit_threshold": "0", 2024-03-19T11:29:54.928 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_pool_quota_warn_threshold": "0", 2024-03-19T11:29:54.929 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_probe_timeout": "2.000000", 2024-03-19T11:29:54.929 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_reweight_max_change": "0.050000", 2024-03-19T11:29:54.929 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_reweight_max_osds": "4", 2024-03-19T11:29:54.929 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-03-19T11:29:54.929 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-03-19T11:29:54.929 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-03-19T11:29:54.929 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-03-19T11:29:54.929 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-03-19T11:29:54.929 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_scrub_interval": "300", 2024-03-19T11:29:54.929 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_scrub_max_keys": "100", 2024-03-19T11:29:54.929 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_scrub_timeout": "300", 2024-03-19T11:29:54.929 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_session_timeout": "300", 2024-03-19T11:29:54.929 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_smart_report_timeout": "5", 2024-03-19T11:29:54.930 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_stat_smooth_intervals": "6", 2024-03-19T11:29:54.930 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-03-19T11:29:54.930 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_stretch_pool_min_size": "2", 2024-03-19T11:29:54.930 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_stretch_pool_size": "4", 2024-03-19T11:29:54.930 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-03-19T11:29:54.930 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_subscribe_interval": "86400.000000", 2024-03-19T11:29:54.930 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_sync_debug": "false", 2024-03-19T11:29:54.930 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_sync_max_payload_keys": "2000", 2024-03-19T11:29:54.930 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_sync_max_payload_size": "1048576", 2024-03-19T11:29:54.930 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_sync_provider_kill_at": "0", 2024-03-19T11:29:54.930 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_sync_requester_kill_at": "0", 2024-03-19T11:29:54.930 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_sync_timeout": "60.000000", 2024-03-19T11:29:54.930 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_target_pg_per_osd": "100", 2024-03-19T11:29:54.931 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_tick_interval": "5", 2024-03-19T11:29:54.931 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_timecheck_interval": "300.000000", 2024-03-19T11:29:54.931 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-03-19T11:29:54.931 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_use_min_delay_socket": "false", 2024-03-19T11:29:54.931 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_older_version_delay": "604800", 2024-03-19T11:29:54.931 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-03-19T11:29:54.931 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-03-19T11:29:54.931 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-03-19T11:29:54.931 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_filestore_osds": "false", 2024-03-19T11:29:54.931 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-03-19T11:29:54.931 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-03-19T11:29:54.931 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-03-19T11:29:54.931 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_misplaced": "false", 2024-03-19T11:29:54.931 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-03-19T11:29:54.932 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_older_version": "true", 2024-03-19T11:29:54.932 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-03-19T11:29:54.932 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_pool_no_app": "true", 2024-03-19T11:29:54.932 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-03-19T11:29:54.932 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-03-19T11:29:54.932 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-03-19T11:29:54.932 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-03-19T11:29:54.932 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_too_few_osds": "false", 2024-03-19T11:29:54.932 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-03-19T11:29:54.932 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-03-19T11:29:54.932 INFO:teuthology.orchestra.run.smithi038.stdout: "monmap": "", 2024-03-19T11:29:54.932 INFO:teuthology.orchestra.run.smithi038.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-03-19T11:29:54.932 INFO:teuthology.orchestra.run.smithi038.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-03-19T11:29:54.932 INFO:teuthology.orchestra.run.smithi038.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-03-19T11:29:54.933 INFO:teuthology.orchestra.run.smithi038.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-03-19T11:29:54.933 INFO:teuthology.orchestra.run.smithi038.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-03-19T11:29:54.933 INFO:teuthology.orchestra.run.smithi038.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-03-19T11:29:54.933 INFO:teuthology.orchestra.run.smithi038.stdout: "motr_tracing_enabled": "false", 2024-03-19T11:29:54.933 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_op_threads": "3", 2024-03-19T11:29:54.933 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_buffer_size": "131072", 2024-03-19T11:29:54.933 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_cm": "false", 2024-03-19T11:29:54.933 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_device_name": "", 2024-03-19T11:29:54.933 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_dscp": "96", 2024-03-19T11:29:54.933 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-03-19T11:29:54.933 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_gid_idx": "0", 2024-03-19T11:29:54.933 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_local_gid": "", 2024-03-19T11:29:54.934 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_polling_us": "1000", 2024-03-19T11:29:54.934 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_port_num": "1", 2024-03-19T11:29:54.934 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-03-19T11:29:54.934 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-03-19T11:29:54.934 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_roce_ver": "1", 2024-03-19T11:29:54.934 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_send_buffers": "1024", 2024-03-19T11:29:54.934 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_sl": "3", 2024-03-19T11:29:54.934 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_support_srq": "true", 2024-03-19T11:29:54.934 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_type": "ib", 2024-03-19T11:29:54.934 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_reap_threshold": "5", 2024-03-19T11:29:54.934 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_bind_before_connect": "false", 2024-03-19T11:29:54.934 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_bind_ipv4": "true", 2024-03-19T11:29:54.934 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_bind_ipv6": "false", 2024-03-19T11:29:54.935 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_bind_msgr1": "true", 2024-03-19T11:29:54.935 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_bind_msgr2": "true", 2024-03-19T11:29:54.935 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_bind_port_max": "7568", 2024-03-19T11:29:54.935 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_bind_port_min": "6800", 2024-03-19T11:29:54.935 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_bind_prefer_ipv4": "false", 2024-03-19T11:29:54.935 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_bind_retry_count": "3", 2024-03-19T11:29:54.935 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_bind_retry_delay": "5", 2024-03-19T11:29:54.935 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_blackhole_client": "false", 2024-03-19T11:29:54.935 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_blackhole_mds": "false", 2024-03-19T11:29:54.935 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_blackhole_mgr": "false", 2024-03-19T11:29:54.935 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_blackhole_mon": "false", 2024-03-19T11:29:54.935 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_blackhole_osd": "false", 2024-03-19T11:29:54.935 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_client_mode": "crc secure", 2024-03-19T11:29:54.935 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_cluster_mode": "crc secure", 2024-03-19T11:29:54.936 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_cluster_type": "", 2024-03-19T11:29:54.936 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_compress_secure": "false", 2024-03-19T11:29:54.936 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_connection_idle_timeout": "900", 2024-03-19T11:29:54.936 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_connection_ready_timeout": "10", 2024-03-19T11:29:54.936 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_crc_data": "true", 2024-03-19T11:29:54.936 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_crc_header": "true", 2024-03-19T11:29:54.936 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_die_on_bad_msg": "false", 2024-03-19T11:29:54.936 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_die_on_bug": "true", 2024-03-19T11:29:54.936 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_die_on_old_message": "true", 2024-03-19T11:29:54.936 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_die_on_skipped_message": "false", 2024-03-19T11:29:54.936 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_die_on_unhandled_msg": "false", 2024-03-19T11:29:54.936 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-03-19T11:29:54.936 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_coremask": "0xF", 2024-03-19T11:29:54.937 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-03-19T11:29:54.937 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_devs_allowlist": "", 2024-03-19T11:29:54.937 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_enable_tso": "true", 2024-03-19T11:29:54.937 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-03-19T11:29:54.937 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-03-19T11:29:54.937 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_hugepages": "", 2024-03-19T11:29:54.937 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_hw_flow_control": "true", 2024-03-19T11:29:54.937 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-03-19T11:29:54.937 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_lro": "true", 2024-03-19T11:29:54.937 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_memory_channel": "4", 2024-03-19T11:29:54.937 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-03-19T11:29:54.937 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_pmd": "", 2024-03-19T11:29:54.937 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_port_id": "0", 2024-03-19T11:29:54.937 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-03-19T11:29:54.938 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dump_corrupt_message_level": "1", 2024-03-19T11:29:54.938 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dump_on_send": "false", 2024-03-19T11:29:54.938 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_initial_backoff": "0.200000", 2024-03-19T11:29:54.938 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_inject_delay_max": "1.000000", 2024-03-19T11:29:54.938 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_inject_delay_probability": "0.000000", 2024-03-19T11:29:54.938 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_inject_delay_type": "", 2024-03-19T11:29:54.938 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_inject_internal_delays": "0.000000", 2024-03-19T11:29:54.938 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_inject_network_congestion": "0", 2024-03-19T11:29:54.938 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_inject_socket_failures": "5000", 2024-03-19T11:29:54.938 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_learn_addr_from_peer": "true", 2024-03-19T11:29:54.938 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_max_accept_failures": "4", 2024-03-19T11:29:54.938 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_max_backoff": "15.000000", 2024-03-19T11:29:54.938 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_mon_client_mode": "secure crc", 2024-03-19T11:29:54.939 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_mon_cluster_mode": "secure crc", 2024-03-19T11:29:54.939 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_mon_service_mode": "secure crc", 2024-03-19T11:29:54.939 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_osd_compress_min_size": "1024", 2024-03-19T11:29:54.939 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_osd_compress_mode": "none", 2024-03-19T11:29:54.939 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_osd_compression_algorithm": "snappy", 2024-03-19T11:29:54.939 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-03-19T11:29:54.939 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_pq_min_cost": "65536", 2024-03-19T11:29:54.939 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_public_type": "", 2024-03-19T11:29:54.939 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_service_mode": "crc secure", 2024-03-19T11:29:54.939 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_tcp_listen_backlog": "512", 2024-03-19T11:29:54.939 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_tcp_nodelay": "true", 2024-03-19T11:29:54.939 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-03-19T11:29:54.940 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_tcp_rcvbuf": "0", 2024-03-19T11:29:54.940 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_type": "async", 2024-03-19T11:29:54.940 INFO:teuthology.orchestra.run.smithi038.stdout: "no_config_file": "false", 2024-03-19T11:29:54.940 INFO:teuthology.orchestra.run.smithi038.stdout: "objecter_completion_locks_per_session": "32", 2024-03-19T11:29:54.940 INFO:teuthology.orchestra.run.smithi038.stdout: "objecter_debug_inject_relock_delay": "false", 2024-03-19T11:29:54.940 INFO:teuthology.orchestra.run.smithi038.stdout: "objecter_inflight_op_bytes": "104857600", 2024-03-19T11:29:54.940 INFO:teuthology.orchestra.run.smithi038.stdout: "objecter_inflight_ops": "1024", 2024-03-19T11:29:54.940 INFO:teuthology.orchestra.run.smithi038.stdout: "objecter_inject_no_watch_ping": "false", 2024-03-19T11:29:54.940 INFO:teuthology.orchestra.run.smithi038.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-03-19T11:29:54.940 INFO:teuthology.orchestra.run.smithi038.stdout: "objecter_tick_interval": "5.000000", 2024-03-19T11:29:54.940 INFO:teuthology.orchestra.run.smithi038.stdout: "objecter_timeout": "10.000000", 2024-03-19T11:29:54.940 INFO:teuthology.orchestra.run.smithi038.stdout: "objectstore_blackhole": "false", 2024-03-19T11:29:54.940 INFO:teuthology.orchestra.run.smithi038.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-03-19T11:29:54.941 INFO:teuthology.orchestra.run.smithi038.stdout: "openssl_engine_opts": "", 2024-03-19T11:29:54.941 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_agent_delay_time": "5.000000", 2024-03-19T11:29:54.941 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_agent_hist_halflife": "1000", 2024-03-19T11:29:54.941 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_agent_max_low_ops": "2", 2024-03-19T11:29:54.941 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_agent_max_ops": "4", 2024-03-19T11:29:54.941 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-03-19T11:29:54.941 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_agent_quantize_effort": "0.100000", 2024-03-19T11:29:54.941 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_agent_slop": "0.020000", 2024-03-19T11:29:54.941 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-03-19T11:29:54.941 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_allow_recovery_below_min_size": "true", 2024-03-19T11:29:54.941 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_asio_thread_count": "2", 2024-03-19T11:29:54.941 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_async_recovery_min_cost": "100", 2024-03-19T11:29:54.941 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_auto_mark_unfound_lost": "false", 2024-03-19T11:29:54.941 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_backfill_retry_interval": "30.000000", 2024-03-19T11:29:54.941 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_backfill_scan_max": "512", 2024-03-19T11:29:54.941 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_backfill_scan_min": "64", 2024-03-19T11:29:54.942 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_backoff_on_degraded": "false", 2024-03-19T11:29:54.942 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_backoff_on_peering": "false", 2024-03-19T11:29:54.942 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_backoff_on_unfound": "true", 2024-03-19T11:29:54.942 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_beacon_report_interval": "300", 2024-03-19T11:29:54.942 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_bench_duration": "30", 2024-03-19T11:29:54.942 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-03-19T11:29:54.942 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_bench_max_block_size": "67108864", 2024-03-19T11:29:54.942 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_bench_small_size_max_iops": "100", 2024-03-19T11:29:54.942 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_blkin_trace_all": "false", 2024-03-19T11:29:54.942 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_blocked_scrub_grace_period": "120", 2024-03-19T11:29:54.942 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-03-19T11:29:54.942 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-03-19T11:29:54.942 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-03-19T11:29:54.942 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_check_for_log_corruption": "false", 2024-03-19T11:29:54.943 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-03-19T11:29:54.943 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-03-19T11:29:54.943 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_class_dir": "/usr/lib/rados-classes", 2024-03-19T11:29:54.943 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-03-19T11:29:54.943 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_class_update_on_start": "true", 2024-03-19T11:29:54.943 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_client_message_cap": "256", 2024-03-19T11:29:54.943 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_client_message_size_cap": "524288000", 2024-03-19T11:29:54.943 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_client_op_priority": "63", 2024-03-19T11:29:54.943 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_client_watch_timeout": "30", 2024-03-19T11:29:54.943 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_command_max_records": "256", 2024-03-19T11:29:54.943 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_compact_on_start": "false", 2024-03-19T11:29:54.943 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-03-19T11:29:54.943 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_crush_chooseleaf_type": "0", 2024-03-19T11:29:54.943 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_crush_initial_weight": "-1.000000", 2024-03-19T11:29:54.943 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_crush_update_on_start": "true", 2024-03-19T11:29:54.944 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_crush_update_weight_set": "true", 2024-03-19T11:29:54.944 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-03-19T11:29:54.944 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-03-19T11:29:54.944 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-03-19T11:29:54.944 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_drop_ping_duration": "0", 2024-03-19T11:29:54.944 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-03-19T11:29:54.944 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_feed_pullee": "-1", 2024-03-19T11:29:54.944 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-03-19T11:29:54.944 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-03-19T11:29:54.944 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-03-19T11:29:54.944 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_misdirected_ops": "false", 2024-03-19T11:29:54.944 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_no_acting_change": "false", 2024-03-19T11:29:54.944 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_no_purge_strays": "false", 2024-03-19T11:29:54.944 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_op_order": "false", 2024-03-19T11:29:54.945 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_pg_log_writeout": "false", 2024-03-19T11:29:54.945 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_pretend_recovery_active": "false", 2024-03-19T11:29:54.945 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-03-19T11:29:54.945 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-03-19T11:29:54.945 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_shutdown": "false", 2024-03-19T11:29:54.945 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-03-19T11:29:54.945 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-03-19T11:29:54.945 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_trim_objects": "false", 2024-03-19T11:29:54.945 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_verify_cached_snaps": "false", 2024-03-19T11:29:54.945 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_verify_missing_on_start": "false", 2024-03-19T11:29:54.945 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_verify_snaps": "false", 2024-03-19T11:29:54.945 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-03-19T11:29:54.945 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-03-19T11:29:54.945 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_deep_scrub_keys": "1024", 2024-03-19T11:29:54.946 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-03-19T11:29:54.946 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-03-19T11:29:54.946 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-03-19T11:29:54.946 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_deep_scrub_stride": "524288", 2024-03-19T11:29:54.946 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-03-19T11:29:54.946 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_default_data_pool_replay_window": "5", 2024-03-19T11:29:54.946 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_default_notify_timeout": "30", 2024-03-19T11:29:54.946 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_delete_sleep": "0.000000", 2024-03-19T11:29:54.946 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-03-19T11:29:54.946 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-03-19T11:29:54.946 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-03-19T11:29:54.947 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_discard_disconnected_ops": "true", 2024-03-19T11:29:54.947 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_enable_op_tracker": "true", 2024-03-19T11:29:54.947 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-03-19T11:29:54.947 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_extblkdev_plugins": "vdo", 2024-03-19T11:29:54.947 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-03-19T11:29:54.947 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-03-19T11:29:54.947 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_fast_info": "true", 2024-03-19T11:29:54.947 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_fast_shutdown": "true", 2024-03-19T11:29:54.947 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-03-19T11:29:54.947 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_fast_shutdown_timeout": "15", 2024-03-19T11:29:54.948 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-03-19T11:29:54.948 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-03-19T11:29:54.948 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-03-19T11:29:54.948 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_function_tracing": "false", 2024-03-19T11:29:54.948 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_heartbeat_grace": "20", 2024-03-19T11:29:54.948 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_heartbeat_interval": "6", 2024-03-19T11:29:54.948 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-03-19T11:29:54.948 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_heartbeat_min_peers": "10", 2024-03-19T11:29:54.948 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_heartbeat_min_size": "2000", 2024-03-19T11:29:54.948 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_heartbeat_stale": "600", 2024-03-19T11:29:54.948 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-03-19T11:29:54.948 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_hit_set_max_size": "100000", 2024-03-19T11:29:54.948 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_hit_set_min_size": "1000", 2024-03-19T11:29:54.949 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-03-19T11:29:54.949 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-03-19T11:29:54.949 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-03-19T11:29:54.949 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-03-19T11:29:54.949 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-03-19T11:29:54.949 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_journal_flush_on_shutdown": "true", 2024-03-19T11:29:54.949 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_journal_size": "5120", 2024-03-19T11:29:54.949 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_kill_backfill_at": "0", 2024-03-19T11:29:54.949 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_loop_before_reset_tphandle": "64", 2024-03-19T11:29:54.949 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_map_cache_size": "50", 2024-03-19T11:29:54.949 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_map_dedup": "true", 2024-03-19T11:29:54.949 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_map_message_max": "40", 2024-03-19T11:29:54.949 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_map_message_max_bytes": "10485760", 2024-03-19T11:29:54.950 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_map_share_max_epochs": "40", 2024-03-19T11:29:54.950 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_attr_name_len": "100", 2024-03-19T11:29:54.950 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_attr_size": "0", 2024-03-19T11:29:54.950 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_backfills": "1", 2024-03-19T11:29:54.950 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_markdown_count": "5", 2024-03-19T11:29:54.950 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_markdown_period": "600", 2024-03-19T11:29:54.950 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_object_name_len": "2048", 2024-03-19T11:29:54.950 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_object_namespace_len": "256", 2024-03-19T11:29:54.950 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_object_size": "134217728", 2024-03-19T11:29:54.950 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-03-19T11:29:54.950 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_omap_entries_per_request": "1024", 2024-03-19T11:29:54.951 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_pg_blocked_by": "16", 2024-03-19T11:29:54.951 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_pg_log_entries": "10000", 2024-03-19T11:29:54.951 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-03-19T11:29:54.951 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_pgls": "1024", 2024-03-19T11:29:54.951 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_push_cost": "8388608", 2024-03-19T11:29:54.951 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_push_objects": "10", 2024-03-19T11:29:54.951 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_scrubs": "3", 2024-03-19T11:29:54.951 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-03-19T11:29:54.951 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_trimming_pgs": "2", 2024-03-19T11:29:54.951 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_write_op_reply_len": "64", 2024-03-19T11:29:54.951 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_write_size": "90", 2024-03-19T11:29:54.951 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-03-19T11:29:54.952 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-03-19T11:29:54.952 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-03-19T11:29:54.952 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-03-19T11:29:54.952 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-03-19T11:29:54.952 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-03-19T11:29:54.952 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-03-19T11:29:54.952 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_override_recovery_settings": "false", 2024-03-19T11:29:54.952 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_profile": "balanced", 2024-03-19T11:29:54.952 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-03-19T11:29:54.952 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-03-19T11:29:54.952 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-03-19T11:29:54.952 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-03-19T11:29:54.953 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-03-19T11:29:54.953 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-03-19T11:29:54.953 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-03-19T11:29:54.953 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-03-19T11:29:54.953 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-03-19T11:29:54.953 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-03-19T11:29:54.953 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_skip_benchmark": "false", 2024-03-19T11:29:54.953 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_memory_base": "805306368", 2024-03-19T11:29:54.953 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_memory_cache_min": "134217728", 2024-03-19T11:29:54.953 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-03-19T11:29:54.953 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-03-19T11:29:54.953 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_memory_target": "4294967296", 2024-03-19T11:29:54.954 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_memory_target_autotune": "false", 2024-03-19T11:29:54.954 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-03-19T11:29:54.954 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_min_pg_log_entries": "250", 2024-03-19T11:29:54.954 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_min_recovery_priority": "0", 2024-03-19T11:29:54.954 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mon_heartbeat_interval": "30", 2024-03-19T11:29:54.954 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-03-19T11:29:54.954 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mon_report_interval": "5", 2024-03-19T11:29:54.954 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mon_report_max_in_flight": "2", 2024-03-19T11:29:54.954 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-03-19T11:29:54.954 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_num_cache_shards": "32", 2024-03-19T11:29:54.954 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_num_op_tracker_shard": "32", 2024-03-19T11:29:54.954 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_numa_auto_affinity": "true", 2024-03-19T11:29:54.954 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_numa_node": "-1", 2024-03-19T11:29:54.954 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_numa_prefer_iface": "true", 2024-03-19T11:29:54.955 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-03-19T11:29:54.955 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_objecter_finishers": "1", 2024-03-19T11:29:54.955 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_objectstore": "bluestore", 2024-03-19T11:29:54.955 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_objectstore_fuse": "false", 2024-03-19T11:29:54.955 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_objectstore_tracing": "false", 2024-03-19T11:29:54.955 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_complaint_time": "30.000000", 2024-03-19T11:29:54.955 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_history_duration": "600", 2024-03-19T11:29:54.955 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_history_size": "20", 2024-03-19T11:29:54.955 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_history_slow_op_size": "20", 2024-03-19T11:29:54.955 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-03-19T11:29:54.955 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_log_threshold": "5", 2024-03-19T11:29:54.955 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_num_shards": "0", 2024-03-19T11:29:54.956 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_num_shards_hdd": "5", 2024-03-19T11:29:54.956 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_num_shards_ssd": "8", 2024-03-19T11:29:54.956 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_num_threads_per_shard": "0", 2024-03-19T11:29:54.956 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-03-19T11:29:54.956 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-03-19T11:29:54.956 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-03-19T11:29:54.956 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_pq_min_cost": "65536", 2024-03-19T11:29:54.956 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_queue": "mclock_scheduler", 2024-03-19T11:29:54.956 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_queue_cut_off": "high", 2024-03-19T11:29:54.956 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_thread_suicide_timeout": "150", 2024-03-19T11:29:54.956 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_thread_timeout": "15", 2024-03-19T11:29:54.956 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_open_classes_on_start": "true", 2024-03-19T11:29:54.956 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_os_flags": "0", 2024-03-19T11:29:54.956 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_peering_op_priority": "255", 2024-03-19T11:29:54.957 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pg_delete_cost": "1048576", 2024-03-19T11:29:54.957 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pg_delete_priority": "5", 2024-03-19T11:29:54.957 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-03-19T11:29:54.957 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-03-19T11:29:54.957 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pg_log_dups_tracked": "3000", 2024-03-19T11:29:54.957 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pg_log_trim_max": "10000", 2024-03-19T11:29:54.957 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pg_log_trim_min": "100", 2024-03-19T11:29:54.957 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-03-19T11:29:54.957 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pg_object_context_cache_count": "64", 2024-03-19T11:29:54.957 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-03-19T11:29:54.957 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-03-19T11:29:54.957 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-03-19T11:29:54.957 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-03-19T11:29:54.957 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-03-19T11:29:54.957 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-03-19T11:29:54.958 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-03-19T11:29:54.958 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-03-19T11:29:54.958 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_crimson": "false", 2024-03-19T11:29:54.958 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_crush_rule": "-1", 2024-03-19T11:29:54.958 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_ec_fast_read": "false", 2024-03-19T11:29:54.958 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-03-19T11:29:54.958 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_flag_bulk": "false", 2024-03-19T11:29:54.958 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-03-19T11:29:54.958 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_flag_nodelete": "false", 2024-03-19T11:29:54.958 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-03-19T11:29:54.958 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-03-19T11:29:54.958 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_flags": "0", 2024-03-19T11:29:54.958 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-03-19T11:29:54.959 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_min_size": "2", 2024-03-19T11:29:54.959 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-03-19T11:29:54.959 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_pg_num": "32", 2024-03-19T11:29:54.959 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_pgp_num": "0", 2024-03-19T11:29:54.959 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-03-19T11:29:54.959 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_read_ratio": "70", 2024-03-19T11:29:54.959 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_size": "2", 2024-03-19T11:29:54.959 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_type": "replicated", 2024-03-19T11:29:54.959 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-03-19T11:29:54.959 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_use_gmt_hitset": "true", 2024-03-19T11:29:54.959 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_push_per_object_cost": "1000", 2024-03-19T11:29:54.959 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_read_ec_check_for_errors": "false", 2024-03-19T11:29:54.959 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recover_clone_overlap": "true", 2024-03-19T11:29:54.959 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recover_clone_overlap_limit": "10", 2024-03-19T11:29:54.960 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_cost": "20971520", 2024-03-19T11:29:54.960 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_delay_start": "0.000000", 2024-03-19T11:29:54.960 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_max_active": "10", 2024-03-19T11:29:54.960 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_max_active_hdd": "3", 2024-03-19T11:29:54.960 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_max_active_ssd": "10", 2024-03-19T11:29:54.960 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_max_chunk": "8388608", 2024-03-19T11:29:54.960 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-03-19T11:29:54.960 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_max_single_start": "10", 2024-03-19T11:29:54.960 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_op_priority": "3", 2024-03-19T11:29:54.960 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_op_warn_multiple": "16", 2024-03-19T11:29:54.960 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_priority": "5", 2024-03-19T11:29:54.960 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_retry_interval": "30.000000", 2024-03-19T11:29:54.960 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_sleep": "0.000000", 2024-03-19T11:29:54.960 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-03-19T11:29:54.961 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-03-19T11:29:54.961 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-03-19T11:29:54.961 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_repair_during_recovery": "false", 2024-03-19T11:29:54.961 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_requested_scrub_priority": "120", 2024-03-19T11:29:54.961 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-03-19T11:29:54.961 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_rollback_to_cluster_snap": "", 2024-03-19T11:29:54.961 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_auto_repair": "false", 2024-03-19T11:29:54.961 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-03-19T11:29:54.961 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-03-19T11:29:54.961 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_begin_hour": "0", 2024-03-19T11:29:54.961 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_begin_week_day": "0", 2024-03-19T11:29:54.962 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_chunk_max": "25", 2024-03-19T11:29:54.962 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_chunk_min": "5", 2024-03-19T11:29:54.962 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_cost": "52428800", 2024-03-19T11:29:54.962 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-03-19T11:29:54.962 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_during_recovery": "false", 2024-03-19T11:29:54.962 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_end_hour": "0", 2024-03-19T11:29:54.962 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_end_week_day": "0", 2024-03-19T11:29:54.962 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_event_cost": "4096", 2024-03-19T11:29:54.962 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-03-19T11:29:54.962 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-03-19T11:29:54.962 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_invalid_stats": "true", 2024-03-19T11:29:54.962 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_load_threshold": "0.500000", 2024-03-19T11:29:54.962 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_max_interval": "604800.000000", 2024-03-19T11:29:54.962 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_max_preemptions": "5", 2024-03-19T11:29:54.963 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_min_interval": "86400.000000", 2024-03-19T11:29:54.963 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_priority": "5", 2024-03-19T11:29:54.963 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_reservation_timeout": "300000", 2024-03-19T11:29:54.963 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_sleep": "0.000000", 2024-03-19T11:29:54.963 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_slow_reservation_response": "30000", 2024-03-19T11:29:54.963 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-03-19T11:29:54.963 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-03-19T11:29:54.963 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_shutdown_pgref_assert": "false", 2024-03-19T11:29:54.963 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-03-19T11:29:54.963 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_skip_data_digest": "false", 2024-03-19T11:29:54.963 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_smart_report_timeout": "5", 2024-03-19T11:29:54.963 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_snap_trim_cost": "1048576", 2024-03-19T11:29:54.963 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_snap_trim_priority": "5", 2024-03-19T11:29:54.963 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_snap_trim_sleep": "0.000000", 2024-03-19T11:29:54.964 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-03-19T11:29:54.964 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-03-19T11:29:54.964 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-03-19T11:29:54.964 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-03-19T11:29:54.964 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_stats_update_period_scrubbing": "15", 2024-03-19T11:29:54.964 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-03-19T11:29:54.964 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_target_transaction_size": "30", 2024-03-19T11:29:54.964 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-03-19T11:29:54.964 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-03-19T11:29:54.964 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-03-19T11:29:54.964 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-03-19T11:29:54.964 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-03-19T11:29:54.964 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-03-19T11:29:54.964 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-03-19T11:29:54.965 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_tier_default_cache_mode": "writeback", 2024-03-19T11:29:54.965 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-03-19T11:29:54.965 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-03-19T11:29:54.965 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_tracing": "false", 2024-03-19T11:29:54.965 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_use_stale_snap": "false", 2024-03-19T11:29:54.965 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-03-19T11:29:54.965 INFO:teuthology.orchestra.run.smithi038.stdout: "osdc_blkin_trace_all": "false", 2024-03-19T11:29:54.965 INFO:teuthology.orchestra.run.smithi038.stdout: "paxos_kill_at": "0", 2024-03-19T11:29:54.965 INFO:teuthology.orchestra.run.smithi038.stdout: "paxos_max_join_drift": "10", 2024-03-19T11:29:54.965 INFO:teuthology.orchestra.run.smithi038.stdout: "paxos_min": "500", 2024-03-19T11:29:54.965 INFO:teuthology.orchestra.run.smithi038.stdout: "paxos_min_wait": "0.050000", 2024-03-19T11:29:54.965 INFO:teuthology.orchestra.run.smithi038.stdout: "paxos_propose_interval": "1.000000", 2024-03-19T11:29:54.966 INFO:teuthology.orchestra.run.smithi038.stdout: "paxos_service_trim_max": "500", 2024-03-19T11:29:54.966 INFO:teuthology.orchestra.run.smithi038.stdout: "paxos_service_trim_max_multiplier": "20", 2024-03-19T11:29:54.966 INFO:teuthology.orchestra.run.smithi038.stdout: "paxos_service_trim_min": "10", 2024-03-19T11:29:54.966 INFO:teuthology.orchestra.run.smithi038.stdout: "paxos_stash_full_interval": "25", 2024-03-19T11:29:54.966 INFO:teuthology.orchestra.run.smithi038.stdout: "paxos_trim_max": "500", 2024-03-19T11:29:54.966 INFO:teuthology.orchestra.run.smithi038.stdout: "paxos_trim_min": "250", 2024-03-19T11:29:54.966 INFO:teuthology.orchestra.run.smithi038.stdout: "perf": "true", 2024-03-19T11:29:54.966 INFO:teuthology.orchestra.run.smithi038.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-03-19T11:29:54.966 INFO:teuthology.orchestra.run.smithi038.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-03-19T11:29:54.966 INFO:teuthology.orchestra.run.smithi038.stdout: "plugin_dir": "/usr/lib/ceph", 2024-03-19T11:29:54.966 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "-", 2024-03-19T11:29:54.966 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrv": "", 2024-03-19T11:29:54.967 INFO:teuthology.orchestra.run.smithi038.stdout: "public_bind_addr": "-", 2024-03-19T11:29:54.967 INFO:teuthology.orchestra.run.smithi038.stdout: "public_network": "", 2024-03-19T11:29:54.967 INFO:teuthology.orchestra.run.smithi038.stdout: "public_network_interface": "", 2024-03-19T11:29:54.967 INFO:teuthology.orchestra.run.smithi038.stdout: "qat_compressor_enabled": "false", 2024-03-19T11:29:54.967 INFO:teuthology.orchestra.run.smithi038.stdout: "qat_compressor_session_max_number": "256", 2024-03-19T11:29:54.967 INFO:teuthology.orchestra.run.smithi038.stdout: "rados_mon_op_timeout": "0", 2024-03-19T11:29:54.967 INFO:teuthology.orchestra.run.smithi038.stdout: "rados_osd_op_timeout": "0", 2024-03-19T11:29:54.968 INFO:teuthology.orchestra.run.smithi038.stdout: "rados_tracing": "false", 2024-03-19T11:29:54.968 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_atime_update_interval": "60", 2024-03-19T11:29:54.968 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-03-19T11:29:54.968 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_balance_parent_reads": "false", 2024-03-19T11:29:54.968 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_balance_snap_reads": "false", 2024-03-19T11:29:54.968 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_blkin_trace_all": "false", 2024-03-19T11:29:54.968 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_blocklist_expire_seconds": "0", 2024-03-19T11:29:54.968 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_blocklist_on_break_lock": "true", 2024-03-19T11:29:54.968 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_cache": "true", 2024-03-19T11:29:54.968 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_cache_block_writes_upfront": "false", 2024-03-19T11:29:54.968 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_cache_max_dirty": "25165824", 2024-03-19T11:29:54.968 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-03-19T11:29:54.969 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_cache_max_dirty_object": "0", 2024-03-19T11:29:54.969 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_cache_policy": "writearound", 2024-03-19T11:29:54.969 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_cache_size": "33554432", 2024-03-19T11:29:54.969 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_cache_target_dirty": "16777216", 2024-03-19T11:29:54.969 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-03-19T11:29:54.969 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_clone_copy_on_read": "false", 2024-03-19T11:29:54.969 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_compression_hint": "none", 2024-03-19T11:29:54.969 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_concurrent_management_ops": "10", 2024-03-19T11:29:54.969 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-03-19T11:29:54.969 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_default_clone_format": "auto", 2024-03-19T11:29:54.969 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_default_data_pool": "", 2024-03-19T11:29:54.969 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_default_features": "61", 2024-03-19T11:29:54.970 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_default_format": "2", 2024-03-19T11:29:54.970 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_default_map_options": "", 2024-03-19T11:29:54.970 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_default_order": "22", 2024-03-19T11:29:54.970 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_default_pool": "rbd", 2024-03-19T11:29:54.970 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-03-19T11:29:54.970 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_default_stripe_count": "0", 2024-03-19T11:29:54.970 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_default_stripe_unit": "0", 2024-03-19T11:29:54.970 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_disable_zero_copy_writes": "true", 2024-03-19T11:29:54.970 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_discard_granularity_bytes": "65536", 2024-03-19T11:29:54.970 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-03-19T11:29:54.970 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_enable_alloc_hint": "true", 2024-03-19T11:29:54.970 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-03-19T11:29:54.970 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_io_scheduler": "simple", 2024-03-19T11:29:54.970 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-03-19T11:29:54.971 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_journal_commit_age": "5.000000", 2024-03-19T11:29:54.971 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-03-19T11:29:54.971 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-03-19T11:29:54.971 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-03-19T11:29:54.971 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-03-19T11:29:54.971 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_journal_object_flush_interval": "0", 2024-03-19T11:29:54.971 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-03-19T11:29:54.971 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-03-19T11:29:54.971 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_journal_order": "24", 2024-03-19T11:29:54.971 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_journal_pool": "", 2024-03-19T11:29:54.971 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_journal_splay_width": "4", 2024-03-19T11:29:54.971 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_localize_parent_reads": "false", 2024-03-19T11:29:54.971 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_localize_snap_reads": "false", 2024-03-19T11:29:54.971 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-03-19T11:29:54.972 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-03-19T11:29:54.972 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-03-19T11:29:54.972 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-03-19T11:29:54.972 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-03-19T11:29:54.972 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-03-19T11:29:54.972 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_image_policy_type": "simple", 2024-03-19T11:29:54.972 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-03-19T11:29:54.972 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-03-19T11:29:54.972 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-03-19T11:29:54.972 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-03-19T11:29:54.972 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-03-19T11:29:54.972 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-03-19T11:29:54.972 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-03-19T11:29:54.973 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_memory_autotune": "true", 2024-03-19T11:29:54.973 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_memory_base": "805306368", 2024-03-19T11:29:54.973 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-03-19T11:29:54.973 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-03-19T11:29:54.973 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-03-19T11:29:54.973 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-03-19T11:29:54.973 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_memory_target": "4294967296", 2024-03-19T11:29:54.973 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-03-19T11:29:54.973 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-03-19T11:29:54.973 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-03-19T11:29:54.974 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirroring_delete_delay": "0", 2024-03-19T11:29:54.974 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-03-19T11:29:54.974 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirroring_replay_delay": "0", 2024-03-19T11:29:54.974 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-03-19T11:29:54.974 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-03-19T11:29:54.974 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_move_to_trash_on_remove": "false", 2024-03-19T11:29:54.974 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-03-19T11:29:54.974 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mtime_update_interval": "60", 2024-03-19T11:29:54.974 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_non_blocking_aio": "true", 2024-03-19T11:29:54.974 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_op_thread_timeout": "60", 2024-03-19T11:29:54.974 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_op_threads": "1", 2024-03-19T11:29:54.975 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_parent_cache_enabled": "false", 2024-03-19T11:29:54.975 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_persistent_cache_mode": "disabled", 2024-03-19T11:29:54.975 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_persistent_cache_path": "/tmp", 2024-03-19T11:29:54.975 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_persistent_cache_size": "1073741824", 2024-03-19T11:29:54.975 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_plugins": "", 2024-03-19T11:29:54.975 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_bps_burst": "0", 2024-03-19T11:29:54.975 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-03-19T11:29:54.975 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_bps_limit": "0", 2024-03-19T11:29:54.975 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_exclude_ops": "0", 2024-03-19T11:29:54.975 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_iops_burst": "0", 2024-03-19T11:29:54.975 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-03-19T11:29:54.975 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_iops_limit": "0", 2024-03-19T11:29:54.975 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_read_bps_burst": "0", 2024-03-19T11:29:54.975 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-03-19T11:29:54.976 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_read_bps_limit": "0", 2024-03-19T11:29:54.976 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_read_iops_burst": "0", 2024-03-19T11:29:54.976 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-03-19T11:29:54.976 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_read_iops_limit": "0", 2024-03-19T11:29:54.976 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_schedule_tick_min": "50", 2024-03-19T11:29:54.976 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_write_bps_burst": "0", 2024-03-19T11:29:54.976 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-03-19T11:29:54.976 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_write_bps_limit": "0", 2024-03-19T11:29:54.976 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_write_iops_burst": "0", 2024-03-19T11:29:54.976 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-03-19T11:29:54.976 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_write_iops_limit": "0", 2024-03-19T11:29:54.976 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_quiesce_notification_attempts": "10", 2024-03-19T11:29:54.976 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_read_from_replica_policy": "default", 2024-03-19T11:29:54.976 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-03-19T11:29:54.977 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_readahead_max_bytes": "524288", 2024-03-19T11:29:54.977 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_readahead_trigger_requests": "10", 2024-03-19T11:29:54.978 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_request_timed_out_seconds": "30", 2024-03-19T11:29:54.978 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_skip_partial_discard": "true", 2024-03-19T11:29:54.978 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-03-19T11:29:54.978 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_tracing": "false", 2024-03-19T11:29:54.978 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_validate_names": "true", 2024-03-19T11:29:54.978 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_validate_pool": "true", 2024-03-19T11:29:54.978 INFO:teuthology.orchestra.run.smithi038.stdout: "restapi_base_url": "", 2024-03-19T11:29:54.978 INFO:teuthology.orchestra.run.smithi038.stdout: "restapi_log_level": "", 2024-03-19T11:29:54.978 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_acl_grants_max_num": "100", 2024-03-19T11:29:54.978 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_admin_entry": "admin", 2024-03-19T11:29:54.978 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-03-19T11:29:54.979 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_backend_store": "rados", 2024-03-19T11:29:54.979 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_barbican_url": "", 2024-03-19T11:29:54.979 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_beast_enable_async": "true", 2024-03-19T11:29:54.979 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_bucket_counters_cache": "false", 2024-03-19T11:29:54.979 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-03-19T11:29:54.979 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-03-19T11:29:54.979 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-03-19T11:29:54.979 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_bucket_index_max_aio": "128", 2024-03-19T11:29:54.979 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-03-19T11:29:54.979 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-03-19T11:29:54.979 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_bucket_quota_ttl": "600", 2024-03-19T11:29:54.979 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-03-19T11:29:54.979 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_cache_enabled": "true", 2024-03-19T11:29:54.979 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_cache_expiry_interval": "900", 2024-03-19T11:29:54.979 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_cache_lru_size": "10000", 2024-03-19T11:29:54.980 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_config_store": "rados", 2024-03-19T11:29:54.980 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_content_length_compat": "false", 2024-03-19T11:29:54.980 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_copy_obj_progress": "true", 2024-03-19T11:29:54.980 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-03-19T11:29:54.980 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_cors_rules_max_num": "100", 2024-03-19T11:29:54.980 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_cross_domain_policy": "", 2024-03-19T11:29:54.980 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_default_encryption_key": "", 2024-03-19T11:29:54.980 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_kmip_addr": "", 2024-03-19T11:29:54.980 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_kmip_ca_path": "", 2024-03-19T11:29:54.980 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_kmip_client_cert": "", 2024-03-19T11:29:54.980 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_kmip_client_key": "", 2024-03-19T11:29:54.981 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-03-19T11:29:54.981 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_kmip_password": "", 2024-03-19T11:29:54.981 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-03-19T11:29:54.981 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_kmip_username": "", 2024-03-19T11:29:54.981 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_require_ssl": "true", 2024-03-19T11:29:54.981 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-03-19T11:29:54.981 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-03-19T11:29:54.981 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-03-19T11:29:54.981 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-03-19T11:29:54.981 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-03-19T11:29:54.981 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-03-19T11:29:54.981 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-03-19T11:29:54.981 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-03-19T11:29:54.981 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-03-19T11:29:54.981 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-03-19T11:29:54.982 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-03-19T11:29:54.982 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-03-19T11:29:54.982 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-03-19T11:29:54.982 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-03-19T11:29:54.982 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_suppress_logs": "true", 2024-03-19T11:29:54.982 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_vault_addr": "", 2024-03-19T11:29:54.982 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_vault_auth": "token", 2024-03-19T11:29:54.982 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_vault_namespace": "", 2024-03-19T11:29:54.982 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_vault_prefix": "", 2024-03-19T11:29:54.982 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-03-19T11:29:54.982 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-03-19T11:29:54.982 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-03-19T11:29:54.982 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-03-19T11:29:54.982 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_vault_token_file": "", 2024-03-19T11:29:54.982 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-03-19T11:29:54.983 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_curl_buffersize": "524288", 2024-03-19T11:29:54.983 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_curl_low_speed_limit": "1024", 2024-03-19T11:29:54.983 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_curl_low_speed_time": "300", 2024-03-19T11:29:54.983 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_curl_tcp_keepalive": "0", 2024-03-19T11:29:54.983 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-03-19T11:29:54.983 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-03-19T11:29:54.983 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-03-19T11:29:54.983 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-03-19T11:29:54.983 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-03-19T11:29:54.983 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_d3n_l1_fadvise": "4", 2024-03-19T11:29:54.983 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-03-19T11:29:54.983 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-03-19T11:29:54.983 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-03-19T11:29:54.983 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_d4n_host": "127.0.0.1", 2024-03-19T11:29:54.983 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_d4n_port": "6379", 2024-03-19T11:29:54.984 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-03-19T11:29:54.984 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_data_log_changes_size": "1000", 2024-03-19T11:29:54.984 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_data_log_num_shards": "128", 2024-03-19T11:29:54.984 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-03-19T11:29:54.984 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_data_log_window": "30", 2024-03-19T11:29:54.984 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_data_notify_interval_msec": "0", 2024-03-19T11:29:54.984 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_data_sync_poll_interval": "20", 2024-03-19T11:29:54.984 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_data_sync_spawn_window": "20", 2024-03-19T11:29:54.984 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-03-19T11:29:54.984 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-03-19T11:29:54.984 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-03-19T11:29:54.984 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_default_data_log_backing": "fifo", 2024-03-19T11:29:54.984 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-03-19T11:29:54.984 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_default_region_info_oid": "default.region", 2024-03-19T11:29:54.984 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-03-19T11:29:54.985 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-03-19T11:29:54.985 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_defer_to_bucket_acls": "", 2024-03-19T11:29:54.985 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-03-19T11:29:54.985 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-03-19T11:29:54.985 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-03-19T11:29:54.985 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-03-19T11:29:54.985 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-03-19T11:29:54.985 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-03-19T11:29:54.985 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-03-19T11:29:54.985 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-03-19T11:29:54.985 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dmclock_data_res": "500.000000", 2024-03-19T11:29:54.985 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-03-19T11:29:54.985 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-03-19T11:29:54.985 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-03-19T11:29:54.986 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-03-19T11:29:54.986 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dns_name": "", 2024-03-19T11:29:54.986 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dns_s3website_name": "", 2024-03-19T11:29:54.986 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dynamic_resharding": "true", 2024-03-19T11:29:54.986 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-03-19T11:29:54.986 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_enable_gc_threads": "true", 2024-03-19T11:29:54.986 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_enable_lc_threads": "true", 2024-03-19T11:29:54.986 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_enable_ops_log": "false", 2024-03-19T11:29:54.986 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_enable_quota_threads": "true", 2024-03-19T11:29:54.986 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_enable_static_website": "false", 2024-03-19T11:29:54.986 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_enable_usage_log": "false", 2024-03-19T11:29:54.986 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_enforce_swift_acls": "true", 2024-03-19T11:29:54.986 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_exit_timeout_secs": "120", 2024-03-19T11:29:54.986 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_expose_bucket": "false", 2024-03-19T11:29:54.987 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_extended_http_attrs": "", 2024-03-19T11:29:54.987 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_filter": "none", 2024-03-19T11:29:54.987 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-03-19T11:29:54.987 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_frontends": "beast port=7480", 2024-03-19T11:29:54.987 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_gc_max_concurrent_io": "10", 2024-03-19T11:29:54.987 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_gc_max_deferred": "50", 2024-03-19T11:29:54.987 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-03-19T11:29:54.987 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_gc_max_objs": "32", 2024-03-19T11:29:54.987 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_gc_max_queue_size": "134213632", 2024-03-19T11:29:54.987 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_gc_max_trim_chunk": "16", 2024-03-19T11:29:54.987 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_gc_obj_min_wait": "7200", 2024-03-19T11:29:54.987 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_gc_processor_max_time": "3600", 2024-03-19T11:29:54.987 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_gc_processor_period": "3600", 2024-03-19T11:29:54.987 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-03-19T11:29:54.987 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_get_obj_window_size": "16777216", 2024-03-19T11:29:54.988 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_healthcheck_disabling_path": "", 2024-03-19T11:29:54.988 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_ignore_get_invalid_range": "false", 2024-03-19T11:29:54.988 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_init_timeout": "300", 2024-03-19T11:29:54.988 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-03-19T11:29:54.988 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-03-19T11:29:54.988 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_kafka_connection_idle": "300", 2024-03-19T11:29:54.988 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_kafka_message_timeout": "5000", 2024-03-19T11:29:54.988 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_kafka_sleep_timeout": "10", 2024-03-19T11:29:54.988 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-03-19T11:29:54.988 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-03-19T11:29:54.988 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-03-19T11:29:54.988 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_admin_domain": "", 2024-03-19T11:29:54.988 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_admin_password": "", 2024-03-19T11:29:54.988 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_admin_password_path": "", 2024-03-19T11:29:54.988 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_admin_project": "", 2024-03-19T11:29:54.989 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_admin_tenant": "", 2024-03-19T11:29:54.989 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_admin_token": "", 2024-03-19T11:29:54.989 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_admin_token_path": "", 2024-03-19T11:29:54.989 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_admin_user": "", 2024-03-19T11:29:54.989 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_api_version": "2", 2024-03-19T11:29:54.989 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_barbican_domain": "", 2024-03-19T11:29:54.989 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_barbican_password": "", 2024-03-19T11:29:54.989 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_barbican_project": "", 2024-03-19T11:29:54.989 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_barbican_tenant": "", 2024-03-19T11:29:54.989 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_barbican_user": "", 2024-03-19T11:29:54.989 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-03-19T11:29:54.989 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_implicit_tenants": "false", 2024-03-19T11:29:54.989 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-03-19T11:29:54.989 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_service_token_enabled": "false", 2024-03-19T11:29:54.990 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_token_cache_size": "10000", 2024-03-19T11:29:54.990 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_url": "", 2024-03-19T11:29:54.990 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_verify_ssl": "true", 2024-03-19T11:29:54.990 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_lc_debug_interval": "-1", 2024-03-19T11:29:54.990 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_lc_lock_max_time": "90", 2024-03-19T11:29:54.990 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_lc_max_objs": "32", 2024-03-19T11:29:54.990 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_lc_max_rules": "1000", 2024-03-19T11:29:54.990 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_lc_max_worker": "3", 2024-03-19T11:29:54.990 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_lc_max_wp_worker": "3", 2024-03-19T11:29:54.990 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_lc_thread_delay": "0", 2024-03-19T11:29:54.990 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-03-19T11:29:54.990 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_ldap_dnattr": "uid", 2024-03-19T11:29:54.990 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-03-19T11:29:54.990 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_ldap_searchfilter": "", 2024-03-19T11:29:54.991 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-03-19T11:29:54.991 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_ldap_uri": "ldaps://", 2024-03-19T11:29:54.991 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-03-19T11:29:54.991 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-03-19T11:29:54.991 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-03-19T11:29:54.991 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_log_http_headers": "", 2024-03-19T11:29:54.991 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_log_nonexistent_bucket": "false", 2024-03-19T11:29:54.991 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-03-19T11:29:54.991 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_log_object_name_utc": "false", 2024-03-19T11:29:54.991 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-03-19T11:29:54.991 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-03-19T11:29:54.991 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_attr_name_len": "0", 2024-03-19T11:29:54.991 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_attr_size": "0", 2024-03-19T11:29:54.991 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_attrs_num_in_req": "0", 2024-03-19T11:29:54.992 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_chunk_size": "4194304", 2024-03-19T11:29:54.992 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_concurrent_requests": "1024", 2024-03-19T11:29:54.992 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-03-19T11:29:54.992 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_dynamic_shards": "1999", 2024-03-19T11:29:54.992 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_listing_results": "1000", 2024-03-19T11:29:54.992 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_notify_retries": "10", 2024-03-19T11:29:54.992 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_objs_per_shard": "100000", 2024-03-19T11:29:54.992 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_put_param_size": "1048576", 2024-03-19T11:29:54.992 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_put_size": "5368709120", 2024-03-19T11:29:54.992 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_slo_entries": "1000", 2024-03-19T11:29:54.992 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_md_log_max_shards": "64", 2024-03-19T11:29:54.992 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_md_notify_interval_msec": "200", 2024-03-19T11:29:54.992 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_meta_sync_poll_interval": "20", 2024-03-19T11:29:54.992 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_meta_sync_spawn_window": "20", 2024-03-19T11:29:54.992 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-03-19T11:29:54.993 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_mp_lock_max_time": "600", 2024-03-19T11:29:54.993 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-03-19T11:29:54.993 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_multipart_min_part_size": "5242880", 2024-03-19T11:29:54.993 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-03-19T11:29:54.993 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-03-19T11:29:54.993 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_fhcache_size": "2017", 2024-03-19T11:29:54.993 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-03-19T11:29:54.993 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-03-19T11:29:54.993 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_lru_lanes": "5", 2024-03-19T11:29:54.993 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_max_gc": "300", 2024-03-19T11:29:54.993 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-03-19T11:29:54.993 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_run_gc_threads": "false", 2024-03-19T11:29:54.993 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_run_lc_threads": "false", 2024-03-19T11:29:54.994 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_run_quota_threads": "false", 2024-03-19T11:29:54.994 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_run_sync_thread": "false", 2024-03-19T11:29:54.994 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-03-19T11:29:54.994 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-03-19T11:29:54.994 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_num_async_rados_threads": "32", 2024-03-19T11:29:54.994 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_num_control_oids": "8", 2024-03-19T11:29:54.994 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_numa_node": "-1", 2024-03-19T11:29:54.994 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_obj_stripe_size": "4194304", 2024-03-19T11:29:54.994 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-03-19T11:29:54.994 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_objexp_chunk_size": "100", 2024-03-19T11:29:54.994 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_objexp_gc_interval": "600", 2024-03-19T11:29:54.994 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_objexp_hints_num_shards": "127", 2024-03-19T11:29:54.994 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-03-19T11:29:54.994 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-03-19T11:29:54.995 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_op_thread_timeout": "600", 2024-03-19T11:29:54.995 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_op_tracing": "false", 2024-03-19T11:29:54.995 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_opa_token": "", 2024-03-19T11:29:54.995 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_opa_url": "", 2024-03-19T11:29:54.995 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_opa_verify_ssl": "true", 2024-03-19T11:29:54.995 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-03-19T11:29:54.995 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-03-19T11:29:54.995 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_ops_log_rados": "false", 2024-03-19T11:29:54.995 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_ops_log_socket_path": "", 2024-03-19T11:29:54.995 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-03-19T11:29:54.995 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-03-19T11:29:54.995 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-03-19T11:29:54.995 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_period_push_interval": "2.000000", 2024-03-19T11:29:54.995 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_period_push_interval_max": "30.000000", 2024-03-19T11:29:54.996 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_period_root_pool": ".rgw.root", 2024-03-19T11:29:54.996 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-03-19T11:29:54.996 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-03-19T11:29:54.996 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_posix_cache_lanes": "3", 2024-03-19T11:29:54.996 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-03-19T11:29:54.996 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_posix_cache_max_buckets": "100", 2024-03-19T11:29:54.996 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_posix_cache_partitions": "3", 2024-03-19T11:29:54.996 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-03-19T11:29:54.996 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_print_continue": "true", 2024-03-19T11:29:54.996 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_print_prohibited_content_length": "false", 2024-03-19T11:29:54.996 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-03-19T11:29:54.996 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-03-19T11:29:54.996 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-03-19T11:29:54.996 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-03-19T11:29:54.996 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_rados_tracing": "false", 2024-03-19T11:29:54.997 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_realm": "", 2024-03-19T11:29:54.997 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_realm_id": "", 2024-03-19T11:29:54.997 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-03-19T11:29:54.997 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_region": "", 2024-03-19T11:29:54.997 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_region_root_pool": ".rgw.root", 2024-03-19T11:29:54.997 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_relaxed_region_enforcement": "false", 2024-03-19T11:29:54.997 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-03-19T11:29:54.997 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-03-19T11:29:54.997 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_request_uri": "", 2024-03-19T11:29:54.997 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_reshard_batch_size": "64", 2024-03-19T11:29:54.997 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-03-19T11:29:54.997 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_reshard_max_aio": "128", 2024-03-19T11:29:54.997 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_reshard_num_logs": "16", 2024-03-19T11:29:54.997 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_reshard_thread_interval": "600", 2024-03-19T11:29:54.998 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_resolve_cname": "false", 2024-03-19T11:29:54.998 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_rest_getusage_op_compat": "false", 2024-03-19T11:29:54.998 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_run_sync_thread": "true", 2024-03-19T11:29:54.998 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-03-19T11:29:54.998 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_s3_auth_use_keystone": "false", 2024-03-19T11:29:54.998 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_s3_auth_use_ldap": "false", 2024-03-19T11:29:54.998 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_s3_auth_use_rados": "true", 2024-03-19T11:29:54.998 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_s3_auth_use_sts": "false", 2024-03-19T11:29:54.998 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-03-19T11:29:54.998 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_s3_success_create_obj_status": "0", 2024-03-19T11:29:54.998 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-03-19T11:29:54.998 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_scheduler_type": "throttler", 2024-03-19T11:29:54.998 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_script_uri": "", 2024-03-19T11:29:54.998 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_service_provider_name": "", 2024-03-19T11:29:54.999 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-03-19T11:29:54.999 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sts_client_id": "", 2024-03-19T11:29:54.999 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sts_client_secret": "", 2024-03-19T11:29:54.999 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sts_entry": "sts", 2024-03-19T11:29:54.999 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sts_key": "sts", 2024-03-19T11:29:54.999 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sts_max_session_duration": "43200", 2024-03-19T11:29:54.999 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sts_min_session_duration": "900", 2024-03-19T11:29:54.999 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sts_token_introspection_url": "", 2024-03-19T11:29:54.999 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_swift_account_in_url": "false", 2024-03-19T11:29:54.999 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_swift_auth_entry": "auth", 2024-03-19T11:29:54.999 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_swift_auth_url": "", 2024-03-19T11:29:54.999 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_swift_custom_header": "", 2024-03-19T11:29:54.999 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_swift_enforce_content_length": "false", 2024-03-19T11:29:54.999 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_swift_need_stats": "true", 2024-03-19T11:29:54.999 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_swift_tenant_name": "", 2024-03-19T11:29:55.000 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_swift_token_expiration": "86400", 2024-03-19T11:29:55.000 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_swift_url": "", 2024-03-19T11:29:55.000 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_swift_url_prefix": "swift", 2024-03-19T11:29:55.000 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_swift_versioning_enabled": "false", 2024-03-19T11:29:55.000 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-03-19T11:29:55.000 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-03-19T11:29:55.000 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sync_lease_period": "120", 2024-03-19T11:29:55.000 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-03-19T11:29:55.000 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sync_log_trim_interval": "1200", 2024-03-19T11:29:55.000 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-03-19T11:29:55.000 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-03-19T11:29:55.001 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-03-19T11:29:55.001 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sync_obj_etag_verify": "false", 2024-03-19T11:29:55.001 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sync_trace_history_size": "4096", 2024-03-19T11:29:55.001 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-03-19T11:29:55.001 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-03-19T11:29:55.001 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_thread_pool_size": "512", 2024-03-19T11:29:55.001 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_topic_persistency_max_retries": "0", 2024-03-19T11:29:55.001 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-03-19T11:29:55.001 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-03-19T11:29:55.001 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_topic_require_publish_policy": "false", 2024-03-19T11:29:55.001 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_torrent_comment": "", 2024-03-19T11:29:55.001 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_torrent_createby": "", 2024-03-19T11:29:55.001 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_torrent_encoding": "", 2024-03-19T11:29:55.002 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_torrent_flag": "false", 2024-03-19T11:29:55.002 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_torrent_max_size": "5368709120", 2024-03-19T11:29:55.002 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_torrent_origin": "", 2024-03-19T11:29:55.002 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_torrent_sha_unit": "524288", 2024-03-19T11:29:55.002 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_torrent_tracker": "", 2024-03-19T11:29:55.002 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_trust_forwarded_https": "false", 2024-03-19T11:29:55.002 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-03-19T11:29:55.002 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_usage_log_tick_interval": "30", 2024-03-19T11:29:55.002 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_usage_max_shards": "32", 2024-03-19T11:29:55.002 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_usage_max_user_shards": "1", 2024-03-19T11:29:55.002 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_use_opa_authz": "false", 2024-03-19T11:29:55.002 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_user_counters_cache": "false", 2024-03-19T11:29:55.002 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_user_counters_cache_size": "10000", 2024-03-19T11:29:55.002 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-03-19T11:29:55.003 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_user_default_quota_max_size": "-1", 2024-03-19T11:29:55.003 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_user_max_buckets": "1000", 2024-03-19T11:29:55.003 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_user_policies_max_num": "100", 2024-03-19T11:29:55.003 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-03-19T11:29:55.003 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-03-19T11:29:55.003 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_user_quota_sync_interval": "86400", 2024-03-19T11:29:55.003 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-03-19T11:29:55.003 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_user_unique_email": "true", 2024-03-19T11:29:55.003 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_verify_ssl": "true", 2024-03-19T11:29:55.003 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_website_routing_rules_max_num": "50", 2024-03-19T11:29:55.003 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_zone": "", 2024-03-19T11:29:55.003 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_zone_id": "", 2024-03-19T11:29:55.003 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-03-19T11:29:55.004 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_zonegroup": "", 2024-03-19T11:29:55.004 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_zonegroup_id": "", 2024-03-19T11:29:55.004 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-03-19T11:29:55.004 INFO:teuthology.orchestra.run.smithi038.stdout: "rgwlc_auto_session_clear": "true", 2024-03-19T11:29:55.004 INFO:teuthology.orchestra.run.smithi038.stdout: "rgwlc_skip_bucket_step": "false", 2024-03-19T11:29:55.004 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_block_size": "4096", 2024-03-19T11:29:55.004 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-03-19T11:29:55.004 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-03-19T11:29:55.004 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-03-19T11:29:55.004 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-03-19T11:29:55.004 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_cache_shard_bits": "4", 2024-03-19T11:29:55.004 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_cache_size": "536870912", 2024-03-19T11:29:55.004 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_cache_type": "binned_lru", 2024-03-19T11:29:55.004 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-03-19T11:29:55.005 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-03-19T11:29:55.005 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-03-19T11:29:55.005 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_collect_compaction_stats": "false", 2024-03-19T11:29:55.005 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_collect_extended_stats": "false", 2024-03-19T11:29:55.005 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_collect_memory_stats": "false", 2024-03-19T11:29:55.005 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-03-19T11:29:55.005 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_index_type": "binary_search", 2024-03-19T11:29:55.005 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_log_to_ceph_log": "true", 2024-03-19T11:29:55.005 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_metadata_block_size": "4096", 2024-03-19T11:29:55.005 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_partition_filters": "false", 2024-03-19T11:29:55.005 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_perf": "false", 2024-03-19T11:29:55.005 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-03-19T11:29:55.005 INFO:teuthology.orchestra.run.smithi038.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-03-19T11:29:55.005 INFO:teuthology.orchestra.run.smithi038.stdout: "rotating_keys_renewal_timeout": "10", 2024-03-19T11:29:55.006 INFO:teuthology.orchestra.run.smithi038.stdout: "run_dir": "/var/run/ceph", 2024-03-19T11:29:55.006 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_block_create": "true", 2024-03-19T11:29:55.006 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_cache_lru_size": "67108864", 2024-03-19T11:29:55.006 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_cbjournal_size": "5368709120", 2024-03-19T11:29:55.006 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_data_delta_based_overwrite": "0", 2024-03-19T11:29:55.006 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_default_max_object_size": "16777216", 2024-03-19T11:29:55.006 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-03-19T11:29:55.006 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_device_size": "53687091200", 2024-03-19T11:29:55.006 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_journal_batch_capacity": "16", 2024-03-19T11:29:55.006 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-03-19T11:29:55.006 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-03-19T11:29:55.006 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_journal_iodepth_limit": "5", 2024-03-19T11:29:55.006 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_main_device_type": "SSD", 2024-03-19T11:29:55.006 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_max_concurrent_transactions": "8", 2024-03-19T11:29:55.007 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-03-19T11:29:55.007 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-03-19T11:29:55.007 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-03-19T11:29:55.007 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-03-19T11:29:55.007 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_segment_size": "67108864", 2024-03-19T11:29:55.007 INFO:teuthology.orchestra.run.smithi038.stdout: "set_keepcaps": "false", 2024-03-19T11:29:55.007 INFO:teuthology.orchestra.run.smithi038.stdout: "setgroup": "", 2024-03-19T11:29:55.007 INFO:teuthology.orchestra.run.smithi038.stdout: "setuser": "", 2024-03-19T11:29:55.007 INFO:teuthology.orchestra.run.smithi038.stdout: "setuser_match_path": "", 2024-03-19T11:29:55.007 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_misplaced_ratio": "0.050000", 2024-03-19T11:29:55.007 INFO:teuthology.orchestra.run.smithi038.stdout: "thp": "false", 2024-03-19T11:29:55.007 INFO:teuthology.orchestra.run.smithi038.stdout: "threadpool_default_timeout": "60", 2024-03-19T11:29:55.007 INFO:teuthology.orchestra.run.smithi038.stdout: "threadpool_empty_queue_max_wait": "2", 2024-03-19T11:29:55.007 INFO:teuthology.orchestra.run.smithi038.stdout: "throttler_perf_counter": "true" 2024-03-19T11:29:55.007 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:29:55.010 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-03-19T11:29:55.155 INFO:teuthology.orchestra.run.smithi038.stdout:mon.a: {} 2024-03-19T11:29:55.155 INFO:teuthology.orchestra.run.smithi038.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-19T11:29:55.168 INFO:teuthology.orchestra.run.smithi038.stdout:mon.b: {} 2024-03-19T11:29:55.168 INFO:teuthology.orchestra.run.smithi038.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-19T11:29:55.177 INFO:teuthology.orchestra.run.smithi038.stdout:mon.c: {} 2024-03-19T11:29:55.178 INFO:teuthology.orchestra.run.smithi038.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-19T11:29:55.194 DEBUG:teuthology.orchestra.run.smithi038:> type ceph-objectstore-tool 2024-03-19T11:29:55.199 INFO:teuthology.orchestra.run.smithi038.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-03-19T11:29:55.199 DEBUG:teuthology.orchestra.run.smithi172:> type ceph-objectstore-tool 2024-03-19T11:29:55.204 INFO:teuthology.orchestra.run.smithi172.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-03-19T11:29:55.205 INFO:teuthology.run_tasks:Running task exec... 2024-03-19T11:29:55.215 INFO:teuthology.task.exec:Executing custom commands... 2024-03-19T11:29:55.215 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi038.front.sepia.ceph.com 2024-03-19T11:29:55.215 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool create low_tier 4' 2024-03-19T11:29:55.216 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-03-19T11:29:55.216 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-19T11:29:55.216 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-19T11:29:55.216 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-03-19T11:29:55.216 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-03-19T11:29:55.216 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-19T11:29:55.217 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-03-19T11:29:55.217 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-19T11:29:55.217 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-03-19T11:29:55.218 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-03-19T11:29:55.247 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:29:55.256+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:29:55.319 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:29:55.328+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:29:55.367 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:29:55.367 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:29:55.367 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:29:55.367 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:29:55.383 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-19T11:29:55.386 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:29:55.386 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:29:55.420 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:29:55.420 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:29:55.420 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:29:55.428+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:29:55.436 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:29:55.436 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:29:55.452 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:29:55.453 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:29:55.473 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:29:55.473 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:29:55.494 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:29:55.495 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:29:55.496 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:29:55.496 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:29:55.511 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-19T11:29:55.515 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:29:55.516 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:29:55.522 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:29:55.532+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:29:55.536 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:29:55.536 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:29:55.622 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:29:55.628+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:29:55.626 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:29:55.626 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:29:55.639 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-19T11:29:55.726 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:29:55.736+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:29:55.804 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:29:55.804 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:29:55.818 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-19T11:29:55.867 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:29:55.876+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:29:55.924 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:29:55.936+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:29:55.992 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:29:55.992 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:29:56.006 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-19T11:29:56.050 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:29:56.060+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:29:56.127 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:29:56.136+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:29:56.128 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:29:56.128 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:29:56.141 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-19T11:29:56.228 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:29:56.236+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:29:56.297 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:29:56.297 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:29:56.310 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-19T11:29:56.329 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:29:56.340+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:29:56.430 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:29:56.440+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:29:56.479 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:29:56.479 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:29:56.493 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-19T11:29:56.515 INFO:teuthology.orchestra.run.smithi038.stderr:pool 'low_tier' created 2024-03-19T11:29:56.515 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-03-19T11:29:56.543 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:29:56.552+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:29:56.545 INFO:teuthology.run_tasks:Running task rados... 2024-03-19T11:29:56.564 INFO:tasks.rados:Beginning rados... 2024-03-19T11:29:56.564 INFO:tasks.rados:config is {'clients': ['client.0'], 'dedup_chunk_algo': 'fastcdc', 'dedup_chunk_size': '131072', 'enable_dedup': True, 'low_tier_pool': 'low_tier', 'objects': 50, 'op_weights': {'read': 100, 'rollback': 10, 'set_chunk': 30, 'snap_create': 10, 'snap_remove': 10, 'tier_evict': 10, 'tier_flush': 5, 'tier_promote': 10, 'write': 50}, 'ops': 1500, 'set_chunk': True} 2024-03-19T11:29:56.564 INFO:tasks.rados:overrides is {'admin_socket': {'branch': 'wip-rf-nextobj'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2, 'osd_recovery_max_active': 10, 'osd_recovery_max_single_start': 10}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd heartbeat use min delay socket': True, 'osd inject bad map crc probability': 0.1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd max scrubs': 5, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', 'failed to encode map', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '9ef43722ad94ea688fa7f70daab6dd41fac6ee53'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '9ef43722ad94ea688fa7f70daab6dd41fac6ee53'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-rf-nextobj', 'sha1': '9ef43722ad94ea688fa7f70daab6dd41fac6ee53'}} 2024-03-19T11:29:56.564 INFO:tasks.rados:config is {'clients': ['client.0'], 'dedup_chunk_algo': 'fastcdc', 'dedup_chunk_size': '131072', 'enable_dedup': True, 'low_tier_pool': 'low_tier', 'objects': 50, 'op_weights': {'read': 100, 'rollback': 10, 'set_chunk': 30, 'snap_create': 10, 'snap_remove': 10, 'tier_evict': 10, 'tier_flush': 5, 'tier_promote': 10, 'write': 50}, 'ops': 1500, 'set_chunk': True} 2024-03-19T11:29:56.565 DEBUG:teuthology.run_tasks:Unwinding manager rados 2024-03-19T11:29:56.600 INFO:tasks.rados:joining rados 2024-03-19T11:29:56.600 INFO:tasks.rados:clients are ['client.0', 'client.1'] 2024-03-19T11:29:56.600 INFO:tasks.rados:starting run 0 out of 1 2024-03-19T11:29:56.600 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2024-03-19T11:29:56.600 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 2024-03-19T11:29:56.632 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:29:56.640+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:29:56.646 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:29:56.646 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:29:56.662 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-19T11:29:56.732 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:29:56.744+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:29:56.824 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:29:56.825 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:29:56.833 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:29:56.848+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:29:56.838 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-19T11:29:56.934 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:29:56.948+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:29:56.961 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:29:56.962 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:29:56.975 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-19T11:29:57.035 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:29:57.048+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:29:57.102 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:29:57.102 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:29:57.118 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-19T11:29:57.136 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:29:57.144+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:29:57.237 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:29:57.244+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:29:57.250 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:29:57.250 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:29:57.264 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-19T11:29:57.338 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:29:57.348+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:29:57.386 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:29:57.386 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:29:57.401 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-19T11:29:57.439 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:29:57.448+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:29:57.519 INFO:teuthology.orchestra.run.smithi038.stderr:pool 'unique_pool_0' created 2024-03-19T11:29:57.523 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:29:57.532+0000 7f6712f92640 -1 osd.2 20 handle_osd_map injecting map crc failure 2024-03-19T11:29:57.525 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:29:57.525 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:29:57.528 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:29:57.536+0000 7f70c6fdf640 -1 osd.0 20 handle_osd_map injecting map crc failure 2024-03-19T11:29:57.540 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:29:57.548+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:29:57.543 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-19T11:29:57.551 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-03-19T11:29:57.553 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-19T11:29:57.641 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:29:57.652+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:29:57.663 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:29:57.663 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:29:57.678 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-19T11:29:57.737 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:29:57.737 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:29:57.765 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:29:57.776+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:29:57.765 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:29:57.765 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:29:57.781 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:29:57.781 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:29:57.799 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:29:57.799 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:29:57.819 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:29:57.819 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:29:57.840 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:29:57.841 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:29:57.842 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:29:57.852+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:29:57.846 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:29:57.846 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:29:57.846 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:29:57.846 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:29:57.846 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:29:57.860 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-19T11:29:57.861 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:29:57.861 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:29:57.897 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:29:57.897 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:29:57.962 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:29:57.972+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:29:58.026 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:29:58.026 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:29:58.026 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:29:58.026 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [] 2024-03-19T11:29:58.026 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:29:58.039 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-19T11:29:58.044 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:29:58.052+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:29:58.144 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:29:58.156+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:29:58.157 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:29:58.157 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:29:58.157 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:29:58.157 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:29:58.171 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-19T11:29:58.260 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:29:58.272+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:29:58.339 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:29:58.339 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:29:58.339 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:29:58.339 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:29:58.339 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:29:58.346 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:29:58.356+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:29:58.352 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-19T11:29:58.448 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:29:58.456+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:29:58.521 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:29:58.521 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:29:58.521 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:29:58.521 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [] 2024-03-19T11:29:58.521 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:29:58.522 INFO:teuthology.orchestra.run.smithi038.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-03-19T11:29:58.526 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:29:58.536+0000 7fb17dd72640 -1 osd.4 21 handle_osd_map injecting map crc failure 2024-03-19T11:29:58.540 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-19T11:29:58.544 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-03-19T11:29:58.584 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:29:58.596+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:29:58.649 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:29:58.660+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:29:58.658 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:29:58.658 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:29:58.658 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:29:58.658 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:29:58.672 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-19T11:29:58.751 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:29:58.760+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:29:58.839 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:29:58.839 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:29:58.839 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:29:58.839 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:29:58.839 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:29:58.852 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:29:58.860+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:29:58.853 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-19T11:29:58.953 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:29:58.960+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:29:59.020 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:29:59.020 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:29:59.020 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:29:59.020 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:29:59.020 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.020 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:29:59.021 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:44.200920+0000", 2024-03-19T11:29:59.021 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.829512767000001, 2024-03-19T11:29:59.021 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.80881896499999995, 2024-03-19T11:29:59.021 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:29:59.021 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:29:59.021 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:29:59.021 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:29:59.021 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:29:59.021 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1 2024-03-19T11:29:59.021 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.021 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:29:59.021 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.021 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:29:59.021 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200920+0000", 2024-03-19T11:29:59.021 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:29:59.022 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.022 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.022 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:29:59.022 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200920+0000", 2024-03-19T11:29:59.022 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:29:59.022 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.022 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.022 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:29:59.022 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200923+0000", 2024-03-19T11:29:59.022 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6469999999999998e-06 2024-03-19T11:29:59.022 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.022 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.022 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:29:59.022 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200937+0000", 2024-03-19T11:29:59.023 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.399e-05 2024-03-19T11:29:59.023 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.023 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.023 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:29:59.023 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200960+0000", 2024-03-19T11:29:59.023 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2603e-05 2024-03-19T11:29:59.023 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.023 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.023 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:29:59.023 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200978+0000", 2024-03-19T11:29:59.023 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.791e-05 2024-03-19T11:29:59.023 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.023 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.023 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:29:59.024 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.201081+0000", 2024-03-19T11:29:59.024 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010344900000000001 2024-03-19T11:29:59.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.024 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:29:59.024 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.201096+0000", 2024-03-19T11:29:59.024 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4443e-05 2024-03-19T11:29:59.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.024 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:29:59.024 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.005978+0000", 2024-03-19T11:29:59.024 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.80488281500000003 2024-03-19T11:29:59.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.025 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:29:59.025 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.006267+0000", 2024-03-19T11:29:59.025 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000288688 2024-03-19T11:29:59.025 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.025 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.025 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:29:59.025 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.006534+0000", 2024-03-19T11:29:59.025 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026725799999999998 2024-03-19T11:29:59.025 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.025 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.025 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:29:59.026 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.008457+0000", 2024-03-19T11:29:59.026 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0019227820000000001 2024-03-19T11:29:59.026 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.026 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.026 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:29:59.026 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.009570+0000", 2024-03-19T11:29:59.026 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011131750000000001 2024-03-19T11:29:59.026 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.026 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.026 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:29:59.026 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.009670+0000", 2024-03-19T11:29:59.026 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9458999999999998e-05 2024-03-19T11:29:59.026 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.026 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.027 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:29:59.027 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.009739+0000", 2024-03-19T11:29:59.027 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9746000000000006e-05 2024-03-19T11:29:59.027 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:29:59.027 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:29:59.027 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:29:59.027 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.027 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.027 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:29:59.027 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.071728+0000", 2024-03-19T11:29:59.027 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 13.958705330000001, 2024-03-19T11:29:59.027 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013714690999999999, 2024-03-19T11:29:59.027 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:29:59.027 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:29:59.028 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:29:59.028 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:29:59.028 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:29:59.028 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 34 2024-03-19T11:29:59.028 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.028 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:29:59.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.028 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:29:59.028 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071728+0000", 2024-03-19T11:29:59.028 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:29:59.028 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.028 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:29:59.028 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071728+0000", 2024-03-19T11:29:59.028 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:29:59.029 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.029 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.029 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:29:59.029 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071729+0000", 2024-03-19T11:29:59.029 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.47e-07 2024-03-19T11:29:59.030 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.031 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.031 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:29:59.031 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071735+0000", 2024-03-19T11:29:59.031 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3799999999999999e-06 2024-03-19T11:29:59.031 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.031 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.031 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:29:59.031 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071748+0000", 2024-03-19T11:29:59.031 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2806000000000001e-05 2024-03-19T11:29:59.031 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.031 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.031 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:29:59.031 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071756+0000", 2024-03-19T11:29:59.032 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1240000000000005e-06 2024-03-19T11:29:59.032 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.032 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.032 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:29:59.032 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.077570+0000", 2024-03-19T11:29:59.032 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.005813342 2024-03-19T11:29:59.032 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.032 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.032 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:29:59.032 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.077793+0000", 2024-03-19T11:29:59.032 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022391500000000001 2024-03-19T11:29:59.032 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.032 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.033 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:29:59.033 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.078146+0000", 2024-03-19T11:29:59.033 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00035219 2024-03-19T11:29:59.033 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.033 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.033 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:29:59.033 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.083390+0000", 2024-03-19T11:29:59.033 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052445590000000002 2024-03-19T11:29:59.033 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.033 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.033 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:29:59.033 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085288+0000", 2024-03-19T11:29:59.033 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0018977060000000001 2024-03-19T11:29:59.033 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.033 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.034 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:29:59.034 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085377+0000", 2024-03-19T11:29:59.034 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9068999999999997e-05 2024-03-19T11:29:59.034 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.034 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.034 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:29:59.034 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085443+0000", 2024-03-19T11:29:59.034 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5653000000000003e-05 2024-03-19T11:29:59.034 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:29:59.034 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:29:59.034 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:29:59.034 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.034 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.034 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:29:59.035 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.071777+0000", 2024-03-19T11:29:59.035 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 13.95865622, 2024-03-19T11:29:59.035 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013962445, 2024-03-19T11:29:59.035 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:29:59.035 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:29:59.035 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:29:59.035 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:29:59.035 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:29:59.035 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 35 2024-03-19T11:29:59.035 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.035 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:29:59.035 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.035 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:29:59.035 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071777+0000", 2024-03-19T11:29:59.036 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:29:59.036 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.036 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:29:59.036 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071777+0000", 2024-03-19T11:29:59.036 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:29:59.036 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.036 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:29:59.036 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071778+0000", 2024-03-19T11:29:59.036 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5000000000000001e-07 2024-03-19T11:29:59.036 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.036 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:29:59.036 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071917+0000", 2024-03-19T11:29:59.037 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013880199999999999 2024-03-19T11:29:59.037 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.037 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.037 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:29:59.037 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071931+0000", 2024-03-19T11:29:59.037 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4080000000000001e-05 2024-03-19T11:29:59.037 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.037 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.038 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:29:59.038 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071940+0000", 2024-03-19T11:29:59.038 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.363e-06 2024-03-19T11:29:59.038 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.038 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.038 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:29:59.038 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.080462+0000", 2024-03-19T11:29:59.038 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085221810000000002 2024-03-19T11:29:59.038 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.038 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.038 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:29:59.038 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.080582+0000", 2024-03-19T11:29:59.038 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011935299999999999 2024-03-19T11:29:59.038 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.039 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:29:59.039 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.080782+0000", 2024-03-19T11:29:59.039 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020077100000000001 2024-03-19T11:29:59.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.039 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:29:59.039 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.083401+0000", 2024-03-19T11:29:59.039 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002618333 2024-03-19T11:29:59.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.039 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:29:59.039 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085571+0000", 2024-03-19T11:29:59.039 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021702829999999998 2024-03-19T11:29:59.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.040 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.040 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:29:59.040 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085672+0000", 2024-03-19T11:29:59.040 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100876 2024-03-19T11:29:59.040 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.040 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.040 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:29:59.040 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085739+0000", 2024-03-19T11:29:59.040 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7552999999999995e-05 2024-03-19T11:29:59.040 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:29:59.040 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:29:59.040 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:29:59.040 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.040 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.040 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:29:59.041 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.071957+0000", 2024-03-19T11:29:59.041 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 13.958476032, 2024-03-19T11:29:59.041 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013989822000000001, 2024-03-19T11:29:59.041 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:29:59.041 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:29:59.041 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:29:59.041 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:29:59.041 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:29:59.041 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 36 2024-03-19T11:29:59.041 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.041 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:29:59.041 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.041 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:29:59.041 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071957+0000", 2024-03-19T11:29:59.041 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:29:59.042 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.042 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.042 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:29:59.042 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071957+0000", 2024-03-19T11:29:59.042 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:29:59.042 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.042 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.042 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:29:59.042 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071958+0000", 2024-03-19T11:29:59.042 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.06e-06 2024-03-19T11:29:59.042 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.042 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.042 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:29:59.042 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071965+0000", 2024-03-19T11:29:59.043 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8929999999999999e-06 2024-03-19T11:29:59.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.043 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:29:59.043 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071976+0000", 2024-03-19T11:29:59.043 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1046999999999999e-05 2024-03-19T11:29:59.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.043 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:29:59.043 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071984+0000", 2024-03-19T11:29:59.043 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7170000000000008e-06 2024-03-19T11:29:59.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.043 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:29:59.043 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.081761+0000", 2024-03-19T11:29:59.044 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0097769310000000009 2024-03-19T11:29:59.044 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.044 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.044 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:29:59.044 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.081928+0000", 2024-03-19T11:29:59.044 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000167219 2024-03-19T11:29:59.044 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.044 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.044 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:29:59.044 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.082131+0000", 2024-03-19T11:29:59.044 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020244499999999999 2024-03-19T11:29:59.045 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.045 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.045 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:29:59.045 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.083455+0000", 2024-03-19T11:29:59.045 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001324073 2024-03-19T11:29:59.045 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.045 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.045 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:29:59.045 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085816+0000", 2024-03-19T11:29:59.045 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0023613309999999999 2024-03-19T11:29:59.045 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.045 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.045 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:29:59.045 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085892+0000", 2024-03-19T11:29:59.045 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6154000000000003e-05 2024-03-19T11:29:59.046 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.046 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.046 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:29:59.046 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085947+0000", 2024-03-19T11:29:59.046 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4951999999999998e-05 2024-03-19T11:29:59.046 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:29:59.046 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:29:59.046 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:29:59.046 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.046 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.046 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:29:59.046 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.072000+0000", 2024-03-19T11:29:59.046 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 13.958433282, 2024-03-19T11:29:59.046 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014418268999999999, 2024-03-19T11:29:59.047 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:29:59.047 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:29:59.047 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:29:59.047 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:29:59.047 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:29:59.047 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 37 2024-03-19T11:29:59.047 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.047 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:29:59.047 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.047 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:29:59.047 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072000+0000", 2024-03-19T11:29:59.047 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:29:59.047 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.047 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.047 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:29:59.048 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072000+0000", 2024-03-19T11:29:59.048 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:29:59.048 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.048 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.048 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:29:59.048 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072001+0000", 2024-03-19T11:29:59.048 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0699999999999996e-07 2024-03-19T11:29:59.048 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.048 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.048 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:29:59.048 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072007+0000", 2024-03-19T11:29:59.048 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2060000000000004e-06 2024-03-19T11:29:59.048 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.048 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.049 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:29:59.049 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072023+0000", 2024-03-19T11:29:59.049 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6617e-05 2024-03-19T11:29:59.049 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.049 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.049 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:29:59.049 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072031+0000", 2024-03-19T11:29:59.049 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8900000000000007e-06 2024-03-19T11:29:59.049 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.049 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.049 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:29:59.049 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.082682+0000", 2024-03-19T11:29:59.049 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010650868000000001 2024-03-19T11:29:59.049 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.049 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.050 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:29:59.050 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.082804+0000", 2024-03-19T11:29:59.050 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012202899999999999 2024-03-19T11:29:59.050 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.050 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.050 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:29:59.050 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.083006+0000", 2024-03-19T11:29:59.050 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020127799999999999 2024-03-19T11:29:59.050 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.050 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.050 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:29:59.050 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.084145+0000", 2024-03-19T11:29:59.050 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001139791 2024-03-19T11:29:59.050 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.051 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.051 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:29:59.051 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.086292+0000", 2024-03-19T11:29:59.051 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021461969999999999 2024-03-19T11:29:59.051 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.051 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.051 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:29:59.051 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.086364+0000", 2024-03-19T11:29:59.051 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2439000000000002e-05 2024-03-19T11:29:59.051 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.051 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.051 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:29:59.051 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.086418+0000", 2024-03-19T11:29:59.051 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4246999999999997e-05 2024-03-19T11:29:59.051 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:29:59.052 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:29:59.052 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:29:59.052 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.052 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.052 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:29:59.052 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.339200+0000", 2024-03-19T11:29:59.052 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 13.691233567999999, 2024-03-19T11:29:59.052 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011947332, 2024-03-19T11:29:59.052 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:29:59.052 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:29:59.052 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:29:59.052 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:29:59.052 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:29:59.052 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 109 2024-03-19T11:29:59.052 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.053 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:29:59.053 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.053 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:29:59.053 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339200+0000", 2024-03-19T11:29:59.053 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:29:59.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.053 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.053 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:29:59.053 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339200+0000", 2024-03-19T11:29:59.053 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:29:59.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.053 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.053 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:29:59.053 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339201+0000", 2024-03-19T11:29:59.054 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.697e-06 2024-03-19T11:29:59.054 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.054 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:29:59.054 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339328+0000", 2024-03-19T11:29:59.054 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012660300000000001 2024-03-19T11:29:59.054 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.054 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:29:59.054 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339362+0000", 2024-03-19T11:29:59.054 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4351999999999999e-05 2024-03-19T11:29:59.054 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.054 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:29:59.054 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339382+0000", 2024-03-19T11:29:59.055 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9983000000000002e-05 2024-03-19T11:29:59.055 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.055 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.055 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:29:59.055 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.345946+0000", 2024-03-19T11:29:59.055 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0065633330000000002 2024-03-19T11:29:59.055 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.055 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.055 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:29:59.055 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.346073+0000", 2024-03-19T11:29:59.055 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000127799 2024-03-19T11:29:59.055 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.055 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.055 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:29:59.055 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.346269+0000", 2024-03-19T11:29:59.056 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000195875 2024-03-19T11:29:59.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.056 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:29:59.056 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.350756+0000", 2024-03-19T11:29:59.056 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044864190000000002 2024-03-19T11:29:59.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.056 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:29:59.056 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.350878+0000", 2024-03-19T11:29:59.056 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012271600000000001 2024-03-19T11:29:59.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.056 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:29:59.056 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.351030+0000", 2024-03-19T11:29:59.057 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015188199999999999 2024-03-19T11:29:59.057 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.057 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:29:59.057 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.351147+0000", 2024-03-19T11:29:59.057 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000116673 2024-03-19T11:29:59.057 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:29:59.057 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:29:59.057 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:29:59.057 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.057 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:29:59.057 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.339422+0000", 2024-03-19T11:29:59.057 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 13.691010916, 2024-03-19T11:29:59.058 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012041094, 2024-03-19T11:29:59.058 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:29:59.058 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:29:59.058 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:29:59.058 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:29:59.058 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:29:59.058 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 110 2024-03-19T11:29:59.058 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.058 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:29:59.058 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.058 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:29:59.058 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339422+0000", 2024-03-19T11:29:59.058 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:29:59.058 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.058 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.059 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:29:59.059 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339422+0000", 2024-03-19T11:29:59.059 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:29:59.059 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.059 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.059 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:29:59.059 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339424+0000", 2024-03-19T11:29:59.059 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.697e-06 2024-03-19T11:29:59.059 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.059 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.059 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:29:59.059 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339440+0000", 2024-03-19T11:29:59.059 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5642999999999999e-05 2024-03-19T11:29:59.059 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.059 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.060 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:29:59.060 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339462+0000", 2024-03-19T11:29:59.060 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2167e-05 2024-03-19T11:29:59.060 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.060 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.060 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:29:59.060 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339477+0000", 2024-03-19T11:29:59.060 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5189e-05 2024-03-19T11:29:59.060 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.060 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.060 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:29:59.060 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.347327+0000", 2024-03-19T11:29:59.060 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0078502559999999999 2024-03-19T11:29:59.060 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.061 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:29:59.061 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.347450+0000", 2024-03-19T11:29:59.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012261600000000001 2024-03-19T11:29:59.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.061 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:29:59.061 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.347644+0000", 2024-03-19T11:29:59.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019408500000000001 2024-03-19T11:29:59.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.061 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:29:59.061 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.351304+0000", 2024-03-19T11:29:59.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003660118 2024-03-19T11:29:59.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:29:59.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.351328+0000", 2024-03-19T11:29:59.062 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3734e-05 2024-03-19T11:29:59.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:29:59.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.351402+0000", 2024-03-19T11:29:59.062 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3708999999999995e-05 2024-03-19T11:29:59.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:29:59.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.351463+0000", 2024-03-19T11:29:59.063 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1879999999999997e-05 2024-03-19T11:29:59.063 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:29:59.063 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:29:59.063 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:29:59.063 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.063 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.063 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:29:59.063 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.339510+0000", 2024-03-19T11:29:59.063 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 13.690923434, 2024-03-19T11:29:59.063 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012358388999999999, 2024-03-19T11:29:59.063 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:29:59.063 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:29:59.063 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:29:59.063 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:29:59.063 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:29:59.064 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 111 2024-03-19T11:29:59.064 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.064 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:29:59.064 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.064 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:29:59.064 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339510+0000", 2024-03-19T11:29:59.064 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:29:59.064 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.064 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.064 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:29:59.064 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339510+0000", 2024-03-19T11:29:59.064 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:29:59.064 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.064 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.064 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:29:59.065 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339511+0000", 2024-03-19T11:29:59.065 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4399999999999998e-07 2024-03-19T11:29:59.065 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.065 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.065 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:29:59.065 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339518+0000", 2024-03-19T11:29:59.065 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9999999999999999e-06 2024-03-19T11:29:59.065 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.065 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.065 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:29:59.065 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339535+0000", 2024-03-19T11:29:59.065 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7552999999999999e-05 2024-03-19T11:29:59.065 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.065 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.066 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:29:59.066 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339550+0000", 2024-03-19T11:29:59.066 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5003e-05 2024-03-19T11:29:59.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.066 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.066 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:29:59.066 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.348403+0000", 2024-03-19T11:29:59.066 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0088530810000000005 2024-03-19T11:29:59.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.066 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.066 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:29:59.066 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.348629+0000", 2024-03-19T11:29:59.066 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022534599999999999 2024-03-19T11:29:59.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.066 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.067 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:29:59.067 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.348976+0000", 2024-03-19T11:29:59.067 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00034710699999999998 2024-03-19T11:29:59.067 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.067 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.067 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:29:59.067 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.351479+0000", 2024-03-19T11:29:59.067 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0025035309999999998 2024-03-19T11:29:59.067 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.067 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.067 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:29:59.067 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.351602+0000", 2024-03-19T11:29:59.067 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000122912 2024-03-19T11:29:59.067 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.067 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.068 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:29:59.068 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.351766+0000", 2024-03-19T11:29:59.068 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016410599999999999 2024-03-19T11:29:59.068 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.068 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.068 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:29:59.068 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.351868+0000", 2024-03-19T11:29:59.068 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010180599999999999 2024-03-19T11:29:59.068 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:29:59.068 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:29:59.068 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:29:59.068 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.068 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.068 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:29:59.068 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.339576+0000", 2024-03-19T11:29:59.069 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 13.69085701, 2024-03-19T11:29:59.069 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012548515999999999, 2024-03-19T11:29:59.069 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:29:59.069 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:29:59.069 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:29:59.069 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:29:59.069 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:29:59.069 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 112 2024-03-19T11:29:59.069 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.069 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:29:59.069 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.069 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:29:59.069 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339576+0000", 2024-03-19T11:29:59.069 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:29:59.069 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.070 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.070 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:29:59.070 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339576+0000", 2024-03-19T11:29:59.070 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:29:59.070 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.070 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.070 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:29:59.070 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339577+0000", 2024-03-19T11:29:59.070 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2529999999999999e-06 2024-03-19T11:29:59.070 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.070 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.070 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:29:59.070 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339813+0000", 2024-03-19T11:29:59.070 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000235811 2024-03-19T11:29:59.070 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.071 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.071 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:29:59.071 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339848+0000", 2024-03-19T11:29:59.071 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4437000000000002e-05 2024-03-19T11:29:59.071 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.071 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.071 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:29:59.071 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339867+0000", 2024-03-19T11:29:59.071 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9046e-05 2024-03-19T11:29:59.071 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.071 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.071 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:29:59.071 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.349815+0000", 2024-03-19T11:29:59.071 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0099484029999999998 2024-03-19T11:29:59.071 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.072 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.072 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:29:59.072 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.349955+0000", 2024-03-19T11:29:59.072 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000140106 2024-03-19T11:29:59.072 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.072 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.072 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:29:59.072 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.350163+0000", 2024-03-19T11:29:59.072 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020803799999999999 2024-03-19T11:29:59.072 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.072 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.072 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:29:59.072 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.351984+0000", 2024-03-19T11:29:59.072 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0018202159999999999 2024-03-19T11:29:59.072 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:29:59.073 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352002+0000", 2024-03-19T11:29:59.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8430000000000001e-05 2024-03-19T11:29:59.073 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:29:59.073 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352070+0000", 2024-03-19T11:29:59.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7623e-05 2024-03-19T11:29:59.073 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:29:59.073 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352125+0000", 2024-03-19T11:29:59.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5152999999999999e-05 2024-03-19T11:29:59.073 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:29:59.073 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:29:59.074 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:29:59.074 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.074 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.074 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:29:59.074 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.339898+0000", 2024-03-19T11:29:59.074 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 13.69053566, 2024-03-19T11:29:59.074 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019501793, 2024-03-19T11:29:59.074 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:29:59.074 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:29:59.074 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:29:59.074 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:29:59.074 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:29:59.074 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113 2024-03-19T11:29:59.074 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.074 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:29:59.075 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.075 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:29:59.075 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339898+0000", 2024-03-19T11:29:59.075 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:29:59.075 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.075 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.075 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:29:59.075 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339898+0000", 2024-03-19T11:29:59.075 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:29:59.075 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.075 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.075 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:29:59.075 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339899+0000", 2024-03-19T11:29:59.075 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.353e-06 2024-03-19T11:29:59.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.076 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:29:59.076 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339907+0000", 2024-03-19T11:29:59.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3000000000000002e-06 2024-03-19T11:29:59.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.076 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:29:59.076 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339924+0000", 2024-03-19T11:29:59.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6906999999999999e-05 2024-03-19T11:29:59.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.076 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:29:59.076 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339933+0000", 2024-03-19T11:29:59.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8969999999999997e-06 2024-03-19T11:29:59.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.077 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:29:59.077 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352239+0000", 2024-03-19T11:29:59.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012305814999999999 2024-03-19T11:29:59.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.077 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:29:59.077 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352466+0000", 2024-03-19T11:29:59.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022684100000000001 2024-03-19T11:29:59.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.077 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:29:59.077 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352816+0000", 2024-03-19T11:29:59.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00035030499999999997 2024-03-19T11:29:59.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.078 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:29:59.078 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.358956+0000", 2024-03-19T11:29:59.078 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061400079999999998 2024-03-19T11:29:59.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.078 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:29:59.078 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359118+0000", 2024-03-19T11:29:59.078 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016237599999999999 2024-03-19T11:29:59.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.078 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:29:59.078 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359287+0000", 2024-03-19T11:29:59.078 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000168488 2024-03-19T11:29:59.079 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.079 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.079 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:29:59.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359399+0000", 2024-03-19T11:29:59.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000112503 2024-03-19T11:29:59.079 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:29:59.079 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:29:59.079 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:29:59.079 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.079 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.079 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:29:59.079 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.339947+0000", 2024-03-19T11:29:59.079 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 13.690485893, 2024-03-19T11:29:59.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021923069999999999, 2024-03-19T11:29:59.079 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:29:59.080 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:29:59.080 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:29:59.080 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:29:59.080 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:29:59.080 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 114 2024-03-19T11:29:59.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.080 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:29:59.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.080 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:29:59.080 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339947+0000", 2024-03-19T11:29:59.080 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:29:59.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.080 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:29:59.080 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339947+0000", 2024-03-19T11:29:59.081 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:29:59.081 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.081 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.081 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:29:59.081 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339948+0000", 2024-03-19T11:29:59.081 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6000000000000003e-07 2024-03-19T11:29:59.081 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.081 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.081 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:29:59.081 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339956+0000", 2024-03-19T11:29:59.081 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1559999999999995e-06 2024-03-19T11:29:59.081 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.081 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.081 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:29:59.081 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339967+0000", 2024-03-19T11:29:59.082 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0669999999999999e-05 2024-03-19T11:29:59.082 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.082 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.082 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:29:59.082 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339975+0000", 2024-03-19T11:29:59.082 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0099999999999995e-06 2024-03-19T11:29:59.082 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.082 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.082 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:29:59.082 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.353672+0000", 2024-03-19T11:29:59.082 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013696583999999999 2024-03-19T11:29:59.082 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.082 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.082 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:29:59.083 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.353808+0000", 2024-03-19T11:29:59.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013652299999999999 2024-03-19T11:29:59.083 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.083 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:29:59.083 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.354036+0000", 2024-03-19T11:29:59.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022744799999999999 2024-03-19T11:29:59.083 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.083 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:29:59.083 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.360480+0000", 2024-03-19T11:29:59.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0064441630000000001 2024-03-19T11:29:59.083 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.083 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:29:59.083 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361722+0000", 2024-03-19T11:29:59.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001242754 2024-03-19T11:29:59.084 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.084 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.084 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:29:59.084 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361807+0000", 2024-03-19T11:29:59.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4605999999999996e-05 2024-03-19T11:29:59.084 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.084 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.084 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:29:59.084 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361870+0000", 2024-03-19T11:29:59.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3396000000000001e-05 2024-03-19T11:29:59.084 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:29:59.084 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:29:59.084 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:29:59.084 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.085 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:29:59.085 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.340017+0000", 2024-03-19T11:29:59.085 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 13.690416577000001, 2024-03-19T11:29:59.085 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022169355000000002, 2024-03-19T11:29:59.085 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:29:59.085 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:29:59.085 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:29:59.085 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:29:59.085 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:29:59.085 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 115 2024-03-19T11:29:59.085 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.085 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:29:59.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.086 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:29:59.086 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340017+0000", 2024-03-19T11:29:59.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:29:59.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.086 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.086 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:29:59.086 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340017+0000", 2024-03-19T11:29:59.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:29:59.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.086 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.086 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:29:59.086 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340018+0000", 2024-03-19T11:29:59.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.043e-06 2024-03-19T11:29:59.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.086 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:29:59.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340139+0000", 2024-03-19T11:29:59.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121502 2024-03-19T11:29:59.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:29:59.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340157+0000", 2024-03-19T11:29:59.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7382999999999999e-05 2024-03-19T11:29:59.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:29:59.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340166+0000", 2024-03-19T11:29:59.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2469999999999997e-06 2024-03-19T11:29:59.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.088 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:29:59.088 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.357470+0000", 2024-03-19T11:29:59.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017304119999999999 2024-03-19T11:29:59.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.088 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:29:59.088 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.357671+0000", 2024-03-19T11:29:59.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000200798 2024-03-19T11:29:59.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.088 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:29:59.088 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.358002+0000", 2024-03-19T11:29:59.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033090099999999998 2024-03-19T11:29:59.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.089 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:29:59.089 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361892+0000", 2024-03-19T11:29:59.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00389018 2024-03-19T11:29:59.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.089 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:29:59.089 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361983+0000", 2024-03-19T11:29:59.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0976000000000004e-05 2024-03-19T11:29:59.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:29:59.090 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362071+0000", 2024-03-19T11:29:59.090 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8022999999999994e-05 2024-03-19T11:29:59.090 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:29:59.090 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362186+0000", 2024-03-19T11:29:59.090 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115182 2024-03-19T11:29:59.090 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:29:59.090 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:29:59.090 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:29:59.090 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.090 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:29:59.091 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.340183+0000", 2024-03-19T11:29:59.091 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 13.690249744999999, 2024-03-19T11:29:59.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022279841000000002, 2024-03-19T11:29:59.091 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:29:59.091 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:29:59.091 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:29:59.091 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:29:59.091 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:29:59.091 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 116 2024-03-19T11:29:59.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.091 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:29:59.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.091 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:29:59.091 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340183+0000", 2024-03-19T11:29:59.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:29:59.092 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.092 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:29:59.092 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340183+0000", 2024-03-19T11:29:59.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:29:59.092 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.092 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:29:59.092 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340184+0000", 2024-03-19T11:29:59.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9000000000000005e-07 2024-03-19T11:29:59.092 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.092 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:29:59.092 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340191+0000", 2024-03-19T11:29:59.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0360000000000003e-06 2024-03-19T11:29:59.093 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.093 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:29:59.093 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340201+0000", 2024-03-19T11:29:59.093 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0844e-05 2024-03-19T11:29:59.093 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.093 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:29:59.093 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340209+0000", 2024-03-19T11:29:59.093 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7360000000000001e-06 2024-03-19T11:29:59.093 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.093 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:29:59.093 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359535+0000", 2024-03-19T11:29:59.093 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019325546999999998 2024-03-19T11:29:59.094 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.094 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:29:59.094 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359672+0000", 2024-03-19T11:29:59.094 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013706600000000001 2024-03-19T11:29:59.094 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.094 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:29:59.094 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359905+0000", 2024-03-19T11:29:59.094 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023334500000000001 2024-03-19T11:29:59.094 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.094 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:29:59.094 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362306+0000", 2024-03-19T11:29:59.094 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002401041 2024-03-19T11:29:59.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.095 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:29:59.095 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362329+0000", 2024-03-19T11:29:59.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3283999999999999e-05 2024-03-19T11:29:59.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.095 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:29:59.095 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362402+0000", 2024-03-19T11:29:59.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2459000000000005e-05 2024-03-19T11:29:59.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.095 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:29:59.095 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362463+0000", 2024-03-19T11:29:59.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1493000000000002e-05 2024-03-19T11:29:59.096 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:29:59.096 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:29:59.096 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:29:59.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.096 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:29:59.096 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.340222+0000", 2024-03-19T11:29:59.096 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 13.690211162000001, 2024-03-19T11:29:59.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022500218999999998, 2024-03-19T11:29:59.096 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:29:59.096 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:29:59.096 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:29:59.096 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:29:59.096 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:29:59.097 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 117 2024-03-19T11:29:59.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.097 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:29:59.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:29:59.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340222+0000", 2024-03-19T11:29:59.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:29:59.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:29:59.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340222+0000", 2024-03-19T11:29:59.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:29:59.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:29:59.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340223+0000", 2024-03-19T11:29:59.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7999999999999995e-07 2024-03-19T11:29:59.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:29:59.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340229+0000", 2024-03-19T11:29:59.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0399999999999998e-06 2024-03-19T11:29:59.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:29:59.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340239+0000", 2024-03-19T11:29:59.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0190000000000001e-05 2024-03-19T11:29:59.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:29:59.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340250+0000", 2024-03-19T11:29:59.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0720000000000001e-05 2024-03-19T11:29:59.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.099 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:29:59.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.360587+0000", 2024-03-19T11:29:59.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020337542 2024-03-19T11:29:59.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.099 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:29:59.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.360818+0000", 2024-03-19T11:29:59.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023120500000000001 2024-03-19T11:29:59.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.099 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:29:59.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361218+0000", 2024-03-19T11:29:59.100 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000399637 2024-03-19T11:29:59.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.100 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:29:59.100 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362483+0000", 2024-03-19T11:29:59.100 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012649200000000001 2024-03-19T11:29:59.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.100 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:29:59.100 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362538+0000", 2024-03-19T11:29:59.100 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4490000000000002e-05 2024-03-19T11:29:59.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.100 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:29:59.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362617+0000", 2024-03-19T11:29:59.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9839000000000006e-05 2024-03-19T11:29:59.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.101 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:29:59.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362722+0000", 2024-03-19T11:29:59.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104956 2024-03-19T11:29:59.101 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:29:59.101 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:29:59.101 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:29:59.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.101 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:29:59.101 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.806124+0000", 2024-03-19T11:29:59.101 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 13.224309011000001, 2024-03-19T11:29:59.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010639073000000001, 2024-03-19T11:29:59.102 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:29:59.102 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:29:59.102 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:29:59.102 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:29:59.102 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:29:59.102 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 136 2024-03-19T11:29:59.102 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.102 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:29:59.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:29:59.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806124+0000", 2024-03-19T11:29:59.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:29:59.102 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.103 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:29:59.103 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806124+0000", 2024-03-19T11:29:59.103 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:29:59.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.103 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:29:59.103 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806125+0000", 2024-03-19T11:29:59.103 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.13e-07 2024-03-19T11:29:59.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.103 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:29:59.103 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806130+0000", 2024-03-19T11:29:59.103 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4199999999999998e-06 2024-03-19T11:29:59.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.104 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:29:59.104 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806138+0000", 2024-03-19T11:29:59.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7479999999999993e-06 2024-03-19T11:29:59.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.104 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:29:59.104 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806145+0000", 2024-03-19T11:29:59.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2269999999999996e-06 2024-03-19T11:29:59.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.104 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:29:59.104 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.808827+0000", 2024-03-19T11:29:59.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026814260000000002 2024-03-19T11:29:59.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.105 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:29:59.105 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.808880+0000", 2024-03-19T11:29:59.105 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3427999999999997e-05 2024-03-19T11:29:59.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.105 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:29:59.105 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.808969+0000", 2024-03-19T11:29:59.105 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8525999999999999e-05 2024-03-19T11:29:59.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.105 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:29:59.105 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.811343+0000", 2024-03-19T11:29:59.105 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002374101 2024-03-19T11:29:59.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:29:59.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.816680+0000", 2024-03-19T11:29:59.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053375990000000002 2024-03-19T11:29:59.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:29:59.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.816716+0000", 2024-03-19T11:29:59.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5781999999999997e-05 2024-03-19T11:29:59.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:29:59.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.816763+0000", 2024-03-19T11:29:59.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7302999999999998e-05 2024-03-19T11:29:59.106 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:29:59.106 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:29:59.107 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:29:59.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.107 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:29:59.107 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.806171+0000", 2024-03-19T11:29:59.107 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 13.224261789, 2024-03-19T11:29:59.107 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010681563999999999, 2024-03-19T11:29:59.107 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:29:59.107 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:29:59.107 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:29:59.107 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:29:59.107 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:29:59.107 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 137 2024-03-19T11:29:59.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.107 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:29:59.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.108 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:29:59.108 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806171+0000", 2024-03-19T11:29:59.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:29:59.108 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.108 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:29:59.108 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806171+0000", 2024-03-19T11:29:59.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:29:59.108 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.108 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:29:59.108 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806172+0000", 2024-03-19T11:29:59.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9100000000000003e-07 2024-03-19T11:29:59.108 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.109 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:29:59.109 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806243+0000", 2024-03-19T11:29:59.109 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0563999999999997e-05 2024-03-19T11:29:59.109 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.109 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.109 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:29:59.109 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806252+0000", 2024-03-19T11:29:59.109 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7499999999999998e-06 2024-03-19T11:29:59.109 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.109 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.109 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:29:59.109 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806263+0000", 2024-03-19T11:29:59.109 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0205e-05 2024-03-19T11:29:59.109 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.109 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.110 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:29:59.110 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.809226+0000", 2024-03-19T11:29:59.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029634679999999999 2024-03-19T11:29:59.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.110 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.110 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:29:59.110 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.809278+0000", 2024-03-19T11:29:59.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1811000000000003e-05 2024-03-19T11:29:59.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.110 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.110 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:29:59.110 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.809363+0000", 2024-03-19T11:29:59.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4752000000000004e-05 2024-03-19T11:29:59.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.110 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.111 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:29:59.111 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.812567+0000", 2024-03-19T11:29:59.111 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032038710000000001 2024-03-19T11:29:59.111 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.111 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.111 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:29:59.111 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.816801+0000", 2024-03-19T11:29:59.111 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0042343040000000004 2024-03-19T11:29:59.111 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.111 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.111 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:29:59.111 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.816833+0000", 2024-03-19T11:29:59.111 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2289000000000002e-05 2024-03-19T11:29:59.111 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.111 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.112 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:29:59.112 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.816853+0000", 2024-03-19T11:29:59.112 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9859000000000001e-05 2024-03-19T11:29:59.112 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:29:59.112 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:29:59.112 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:29:59.112 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.112 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.112 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:29:59.112 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.806275+0000", 2024-03-19T11:29:59.112 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 13.224158452999999, 2024-03-19T11:29:59.112 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010695962, 2024-03-19T11:29:59.112 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:29:59.112 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:29:59.112 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:29:59.112 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:29:59.113 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:29:59.113 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 138 2024-03-19T11:29:59.113 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.113 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:29:59.113 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.113 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:29:59.113 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806275+0000", 2024-03-19T11:29:59.113 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:29:59.113 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.113 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.113 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:29:59.113 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806275+0000", 2024-03-19T11:29:59.113 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:29:59.113 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.114 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.114 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:29:59.114 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806275+0000", 2024-03-19T11:29:59.114 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1099999999999995e-07 2024-03-19T11:29:59.114 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.114 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.114 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:29:59.114 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806280+0000", 2024-03-19T11:29:59.114 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.42e-06 2024-03-19T11:29:59.114 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.114 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.114 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:29:59.114 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806288+0000", 2024-03-19T11:29:59.114 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9589999999999995e-06 2024-03-19T11:29:59.114 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.115 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.115 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:29:59.115 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806296+0000", 2024-03-19T11:29:59.115 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6980000000000004e-06 2024-03-19T11:29:59.115 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.115 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.115 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:29:59.115 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.810916+0000", 2024-03-19T11:29:59.115 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0046194579999999999 2024-03-19T11:29:59.115 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.115 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.115 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:29:59.115 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.810975+0000", 2024-03-19T11:29:59.115 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8767999999999997e-05 2024-03-19T11:29:59.115 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.116 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.116 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:29:59.116 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.811066+0000", 2024-03-19T11:29:59.116 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1543000000000001e-05 2024-03-19T11:29:59.116 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.116 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.116 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:29:59.116 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.813111+0000", 2024-03-19T11:29:59.116 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0020449650000000002 2024-03-19T11:29:59.116 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.116 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.116 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:29:59.116 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.816889+0000", 2024-03-19T11:29:59.116 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037775270000000001 2024-03-19T11:29:59.116 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.117 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.117 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:29:59.117 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.816925+0000", 2024-03-19T11:29:59.117 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6539999999999999e-05 2024-03-19T11:29:59.117 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.117 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.117 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:29:59.117 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.816971+0000", 2024-03-19T11:29:59.117 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5473000000000003e-05 2024-03-19T11:29:59.117 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:29:59.117 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:29:59.117 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:29:59.117 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.117 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.117 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:29:59.117 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.806306+0000", 2024-03-19T11:29:59.118 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 13.224126862, 2024-03-19T11:29:59.118 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010748485, 2024-03-19T11:29:59.118 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:29:59.118 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:29:59.118 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:29:59.118 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:29:59.118 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:29:59.118 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 139 2024-03-19T11:29:59.118 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.118 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:29:59.118 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.118 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:29:59.118 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806306+0000", 2024-03-19T11:29:59.118 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:29:59.118 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.119 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.119 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:29:59.119 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806306+0000", 2024-03-19T11:29:59.119 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:29:59.119 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.119 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.119 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:29:59.119 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806307+0000", 2024-03-19T11:29:59.119 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7199999999999999e-07 2024-03-19T11:29:59.119 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.119 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.119 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:29:59.119 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806311+0000", 2024-03-19T11:29:59.119 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1989999999999999e-06 2024-03-19T11:29:59.119 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.120 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.120 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:29:59.120 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806323+0000", 2024-03-19T11:29:59.120 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2075e-05 2024-03-19T11:29:59.120 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.120 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.120 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:29:59.120 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806331+0000", 2024-03-19T11:29:59.120 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4959999999999999e-06 2024-03-19T11:29:59.120 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.120 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.120 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:29:59.120 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.811381+0000", 2024-03-19T11:29:59.120 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.005050232 2024-03-19T11:29:59.120 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.121 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:29:59.121 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.811437+0000", 2024-03-19T11:29:59.121 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6116999999999998e-05 2024-03-19T11:29:59.121 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.121 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:29:59.121 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.811525+0000", 2024-03-19T11:29:59.121 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7757999999999996e-05 2024-03-19T11:29:59.121 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.121 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:29:59.121 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.813119+0000", 2024-03-19T11:29:59.121 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0015946739999999999 2024-03-19T11:29:59.121 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.122 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.122 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:29:59.122 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.817005+0000", 2024-03-19T11:29:59.122 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038853080000000001 2024-03-19T11:29:59.122 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.122 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.122 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:29:59.122 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.817034+0000", 2024-03-19T11:29:59.122 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9297000000000001e-05 2024-03-19T11:29:59.122 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.122 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.122 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:29:59.122 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.817055+0000", 2024-03-19T11:29:59.122 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0857e-05 2024-03-19T11:29:59.122 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:29:59.123 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:29:59.123 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:29:59.123 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.123 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.123 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:29:59.123 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.806342+0000", 2024-03-19T11:29:59.123 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 13.224090855, 2024-03-19T11:29:59.123 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010826809999999999, 2024-03-19T11:29:59.123 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:29:59.123 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:29:59.123 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:29:59.123 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:29:59.123 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:29:59.123 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 140 2024-03-19T11:29:59.123 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.124 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:29:59.124 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.124 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:29:59.124 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806342+0000", 2024-03-19T11:29:59.124 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:29:59.124 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.124 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.124 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:29:59.124 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806342+0000", 2024-03-19T11:29:59.124 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:29:59.124 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.124 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.124 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:29:59.124 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806343+0000", 2024-03-19T11:29:59.124 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6600000000000002e-07 2024-03-19T11:29:59.124 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.125 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.125 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:29:59.125 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806422+0000", 2024-03-19T11:29:59.125 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9536000000000006e-05 2024-03-19T11:29:59.125 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.125 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.125 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:29:59.125 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806439+0000", 2024-03-19T11:29:59.125 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7098000000000001e-05 2024-03-19T11:29:59.125 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.125 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.125 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:29:59.125 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806447+0000", 2024-03-19T11:29:59.125 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1300000000000003e-06 2024-03-19T11:29:59.125 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.126 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.126 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:29:59.126 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.811917+0000", 2024-03-19T11:29:59.126 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054699019999999996 2024-03-19T11:29:59.126 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.126 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.126 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:29:59.126 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.811973+0000", 2024-03-19T11:29:59.126 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6001000000000001e-05 2024-03-19T11:29:59.126 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.126 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.126 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:29:59.126 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.812055+0000", 2024-03-19T11:29:59.126 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2655000000000001e-05 2024-03-19T11:29:59.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.127 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:29:59.127 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.813256+0000", 2024-03-19T11:29:59.127 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012011999999999999 2024-03-19T11:29:59.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.127 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:29:59.127 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.817091+0000", 2024-03-19T11:29:59.127 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038344659999999999 2024-03-19T11:29:59.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.127 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:29:59.127 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.817125+0000", 2024-03-19T11:29:59.127 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4653000000000002e-05 2024-03-19T11:29:59.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.128 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.128 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:29:59.128 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.817169+0000", 2024-03-19T11:29:59.128 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3702999999999998e-05 2024-03-19T11:29:59.128 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:29:59.128 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:29:59.128 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:29:59.128 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.128 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.128 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:29:59.128 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.806461+0000", 2024-03-19T11:29:59.128 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 13.223972711, 2024-03-19T11:29:59.128 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010789451, 2024-03-19T11:29:59.128 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:29:59.128 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:29:59.129 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:29:59.129 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:29:59.129 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:29:59.129 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 141 2024-03-19T11:29:59.129 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.129 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:29:59.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.129 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:29:59.129 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806461+0000", 2024-03-19T11:29:59.129 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:29:59.129 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.129 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:29:59.129 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806461+0000", 2024-03-19T11:29:59.129 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:29:59.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.130 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:29:59.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806461+0000", 2024-03-19T11:29:59.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5499999999999997e-07 2024-03-19T11:29:59.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.130 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:29:59.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806466+0000", 2024-03-19T11:29:59.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9049999999999996e-06 2024-03-19T11:29:59.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.130 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:29:59.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806474+0000", 2024-03-19T11:29:59.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6009999999999997e-06 2024-03-19T11:29:59.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.131 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:29:59.131 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806480+0000", 2024-03-19T11:29:59.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.207e-06 2024-03-19T11:29:59.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.131 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:29:59.131 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.812605+0000", 2024-03-19T11:29:59.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.006125096 2024-03-19T11:29:59.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.131 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:29:59.131 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.812662+0000", 2024-03-19T11:29:59.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7095999999999997e-05 2024-03-19T11:29:59.132 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.132 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:29:59.132 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.812751+0000", 2024-03-19T11:29:59.132 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9115000000000003e-05 2024-03-19T11:29:59.132 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.132 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:29:59.132 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.813361+0000", 2024-03-19T11:29:59.132 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00060919399999999997 2024-03-19T11:29:59.132 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.132 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:29:59.132 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.817201+0000", 2024-03-19T11:29:59.133 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038408510000000002 2024-03-19T11:29:59.133 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.133 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.133 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:29:59.133 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.817230+0000", 2024-03-19T11:29:59.133 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8963999999999999e-05 2024-03-19T11:29:59.133 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:29:59.133 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:29:59.133 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:29:59.133 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.817250+0000", 2024-03-19T11:29:59.133 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9667e-05 2024-03-19T11:29:59.133 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:29:59.133 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:29:59.133 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:29:59.133 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:29:59.134 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:29:59.134 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:29:59.135 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-19T11:29:59.136 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:29:59.144+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:29:59.236 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:29:59.244+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:29:59.272 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:29:59.272 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:29:59.272 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:29:59.272 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:29:59.286 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-19T11:29:59.378 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:29:59.388+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:29:59.438 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:29:59.452+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:29:59.459 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:29:59.459 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:29:59.459 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:29:59.459 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:29:59.459 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:29:59.473 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-19T11:29:59.552 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-03-19T11:29:59.552 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:29:59.560+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:29:59.557 DEBUG:teuthology.orchestra.run.smithi038:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_chunk --enable_dedup --low_tier_pool low_tier --dedup_chunk_size 131072 --dedup_chunk_algo fastcdc --max-ops 1500 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 25 --op delete 10 --op snap_create 10 --op snap_remove 10 --op rollback 10 --op set_chunk 30 --op tier_promote 10 --op tier_evict 10 --op tier_flush 5 --op write_excl 25 --pool unique_pool_0 2024-03-19T11:29:59.575 INFO:tasks.rados.rados.0.smithi038.stdout:adding op weight read -> 100 2024-03-19T11:29:59.576 INFO:tasks.rados.rados.0.smithi038.stdout:adding op weight write -> 25 2024-03-19T11:29:59.576 INFO:tasks.rados.rados.0.smithi038.stdout:adding op weight delete -> 10 2024-03-19T11:29:59.576 INFO:tasks.rados.rados.0.smithi038.stdout:adding op weight snap_create -> 10 2024-03-19T11:29:59.576 INFO:tasks.rados.rados.0.smithi038.stdout:adding op weight snap_remove -> 10 2024-03-19T11:29:59.576 INFO:tasks.rados.rados.0.smithi038.stdout:adding op weight rollback -> 10 2024-03-19T11:29:59.576 INFO:tasks.rados.rados.0.smithi038.stdout:adding op weight set_chunk -> 30 2024-03-19T11:29:59.576 INFO:tasks.rados.rados.0.smithi038.stdout:adding op weight tier_promote -> 10 2024-03-19T11:29:59.576 INFO:tasks.rados.rados.0.smithi038.stdout:adding op weight tier_evict -> 10 2024-03-19T11:29:59.576 INFO:tasks.rados.rados.0.smithi038.stdout:adding op weight tier_flush -> 5 2024-03-19T11:29:59.576 INFO:tasks.rados.rados.0.smithi038.stdout:adding op weight write_excl -> 25 2024-03-19T11:29:59.576 INFO:tasks.rados.rados.0.smithi038.stdout:ceph version 19.0.0-2159-g9ef43722 (9ef43722ad94ea688fa7f70daab6dd41fac6ee53) squid (dev) 2024-03-19T11:29:59.576 INFO:tasks.rados.rados.0.smithi038.stdout:Configuration: 2024-03-19T11:29:59.576 INFO:tasks.rados.rados.0.smithi038.stdout: Number of operations: 1500 2024-03-19T11:29:59.576 INFO:tasks.rados.rados.0.smithi038.stdout: Number of objects: 50 2024-03-19T11:29:59.576 INFO:tasks.rados.rados.0.smithi038.stdout: Max in flight operations: 16 2024-03-19T11:29:59.577 INFO:tasks.rados.rados.0.smithi038.stdout: Object size (in bytes): 4000000 2024-03-19T11:29:59.577 INFO:tasks.rados.rados.0.smithi038.stdout: Write stride min: 400000 2024-03-19T11:29:59.577 INFO:tasks.rados.rados.0.smithi038.stdout: Write stride max: 800000 2024-03-19T11:29:59.638 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:29:59.638 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:29:59.639 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:29:59.639 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:29:59.639 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.639 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:34 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'20, mlcod=15'16)", 2024-03-19T11:29:59.639 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.077542+0000", 2024-03-19T11:29:59.639 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 14.573426951, 2024-03-19T11:29:59.639 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0064305179999999997, 2024-03-19T11:29:59.639 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:29:59.639 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:29:59.639 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:29:59.639 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.639 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:29:59.639 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.077542+0000", 2024-03-19T11:29:59.640 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:29:59.640 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.640 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.640 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:29:59.640 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.077542+0000", 2024-03-19T11:29:59.640 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:29:59.640 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.640 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.640 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:29:59.640 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.077543+0000", 2024-03-19T11:29:59.640 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.387e-06 2024-03-19T11:29:59.640 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.640 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.640 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:29:59.641 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.077555+0000", 2024-03-19T11:29:59.641 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1759999999999999e-05 2024-03-19T11:29:59.641 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.641 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.641 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:29:59.641 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.077580+0000", 2024-03-19T11:29:59.641 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.529e-05 2024-03-19T11:29:59.641 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.641 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.641 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:29:59.641 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.077596+0000", 2024-03-19T11:29:59.641 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5466000000000001e-05 2024-03-19T11:29:59.641 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.641 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.641 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:29:59.642 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.077753+0000", 2024-03-19T11:29:59.642 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000157026 2024-03-19T11:29:59.642 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.642 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.642 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:29:59.642 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.077789+0000", 2024-03-19T11:29:59.642 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6259e-05 2024-03-19T11:29:59.642 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.642 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.642 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:29:59.642 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.083910+0000", 2024-03-19T11:29:59.642 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0061212080000000004 2024-03-19T11:29:59.642 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.642 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.643 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:29:59.643 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.083972+0000", 2024-03-19T11:29:59.643 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2121999999999999e-05 2024-03-19T11:29:59.643 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:29:59.643 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:29:59.643 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:29:59.643 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.643 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.643 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:35 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'21, mlcod=15'17)", 2024-03-19T11:29:59.643 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.080212+0000", 2024-03-19T11:29:59.643 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 14.570756826, 2024-03-19T11:29:59.643 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003806023, 2024-03-19T11:29:59.643 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:29:59.643 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:29:59.644 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:29:59.644 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.644 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:29:59.644 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.080212+0000", 2024-03-19T11:29:59.644 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:29:59.644 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.644 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.644 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:29:59.644 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.080212+0000", 2024-03-19T11:29:59.644 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:29:59.644 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.644 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.644 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:29:59.644 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.080213+0000", 2024-03-19T11:29:59.644 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2330000000000001e-06 2024-03-19T11:29:59.645 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.645 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.645 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:29:59.645 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.080266+0000", 2024-03-19T11:29:59.645 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2460000000000003e-05 2024-03-19T11:29:59.645 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.645 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.645 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:29:59.645 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.080288+0000", 2024-03-19T11:29:59.645 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1892999999999999e-05 2024-03-19T11:29:59.645 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.645 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.645 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:29:59.645 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.080323+0000", 2024-03-19T11:29:59.645 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5107000000000001e-05 2024-03-19T11:29:59.646 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.646 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.646 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:29:59.646 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.082276+0000", 2024-03-19T11:29:59.646 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0019535170000000001 2024-03-19T11:29:59.646 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.646 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.646 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:29:59.646 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.082315+0000", 2024-03-19T11:29:59.646 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8760000000000002e-05 2024-03-19T11:29:59.646 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.646 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.646 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:29:59.646 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.083977+0000", 2024-03-19T11:29:59.646 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001662483 2024-03-19T11:29:59.647 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.647 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.647 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:29:59.647 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.084018+0000", 2024-03-19T11:29:59.647 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.057e-05 2024-03-19T11:29:59.647 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:29:59.647 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:29:59.648 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:29:59.648 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.648 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.648 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:39 1.0 e15/14 1:30306672:devicehealth::main.db.0000000000000000:head v 15'25, mlcod=15'22)", 2024-03-19T11:29:59.648 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.088045+0000", 2024-03-19T11:29:59.648 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 14.562924036, 2024-03-19T11:29:59.648 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0036692769999999999, 2024-03-19T11:29:59.648 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:29:59.648 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:29:59.648 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:29:59.648 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.648 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:29:59.648 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.088045+0000", 2024-03-19T11:29:59.648 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:29:59.649 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.649 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.649 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:29:59.649 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.088045+0000", 2024-03-19T11:29:59.649 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:29:59.649 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.649 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.649 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:29:59.649 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.088046+0000", 2024-03-19T11:29:59.649 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1230000000000001e-06 2024-03-19T11:29:59.649 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.649 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.649 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:29:59.649 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.088109+0000", 2024-03-19T11:29:59.649 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3326999999999999e-05 2024-03-19T11:29:59.650 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.650 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.650 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:29:59.650 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.088133+0000", 2024-03-19T11:29:59.650 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3356000000000001e-05 2024-03-19T11:29:59.650 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.650 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.650 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:29:59.650 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.088149+0000", 2024-03-19T11:29:59.650 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6192999999999999e-05 2024-03-19T11:29:59.650 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.650 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.650 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:29:59.650 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.088312+0000", 2024-03-19T11:29:59.651 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000162922 2024-03-19T11:29:59.651 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.651 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.651 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:29:59.651 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.088354+0000", 2024-03-19T11:29:59.651 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2722999999999997e-05 2024-03-19T11:29:59.651 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.651 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.651 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:29:59.651 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.091621+0000", 2024-03-19T11:29:59.651 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0032666169999999999 2024-03-19T11:29:59.651 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.651 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.651 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:29:59.651 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.091714+0000", 2024-03-19T11:29:59.652 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3016000000000005e-05 2024-03-19T11:29:59.652 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:29:59.652 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:29:59.652 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:29:59.652 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.652 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.652 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:40 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'26, mlcod=15'24)", 2024-03-19T11:29:59.652 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.093333+0000", 2024-03-19T11:29:59.652 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 14.557636102, 2024-03-19T11:29:59.652 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0037043660000000002, 2024-03-19T11:29:59.652 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:29:59.652 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:29:59.652 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:29:59.652 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.652 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:29:59.653 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.093333+0000", 2024-03-19T11:29:59.653 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:29:59.653 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.653 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.653 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:29:59.653 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.093333+0000", 2024-03-19T11:29:59.653 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:29:59.653 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.653 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.653 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:29:59.653 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.093334+0000", 2024-03-19T11:29:59.653 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.33e-06 2024-03-19T11:29:59.653 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.653 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.654 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:29:59.654 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.093343+0000", 2024-03-19T11:29:59.654 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3659999999999998e-06 2024-03-19T11:29:59.654 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.654 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.654 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:29:59.654 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.093368+0000", 2024-03-19T11:29:59.654 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4967e-05 2024-03-19T11:29:59.654 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.654 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.654 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:29:59.654 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.093385+0000", 2024-03-19T11:29:59.654 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6773e-05 2024-03-19T11:29:59.654 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.655 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.655 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:29:59.655 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.093533+0000", 2024-03-19T11:29:59.655 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014780199999999999 2024-03-19T11:29:59.655 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.655 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.655 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:29:59.655 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.093571+0000", 2024-03-19T11:29:59.655 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7929999999999998e-05 2024-03-19T11:29:59.655 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.655 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.655 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:29:59.655 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.096945+0000", 2024-03-19T11:29:59.655 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033740829999999999 2024-03-19T11:29:59.656 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.656 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.656 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:29:59.656 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.097037+0000", 2024-03-19T11:29:59.656 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2114999999999995e-05 2024-03-19T11:29:59.656 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:29:59.656 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:29:59.656 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:29:59.656 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.656 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.656 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:114 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'76, mlcod=16'73)", 2024-03-19T11:29:59.656 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.353321+0000", 2024-03-19T11:29:59.656 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 14.297648098, 2024-03-19T11:29:59.656 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050714999999999996, 2024-03-19T11:29:59.656 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:29:59.657 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:29:59.657 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:29:59.657 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.657 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:29:59.657 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.353321+0000", 2024-03-19T11:29:59.657 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:29:59.657 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.657 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.657 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:29:59.657 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.353321+0000", 2024-03-19T11:29:59.657 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:29:59.657 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.657 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.657 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:29:59.657 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.353321+0000", 2024-03-19T11:29:59.658 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3399999999999998e-07 2024-03-19T11:29:59.658 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.658 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.658 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:29:59.658 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.353336+0000", 2024-03-19T11:29:59.658 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4154999999999999e-05 2024-03-19T11:29:59.658 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.658 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.658 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:29:59.658 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.353349+0000", 2024-03-19T11:29:59.658 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3644e-05 2024-03-19T11:29:59.658 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.658 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.658 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:29:59.659 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.353357+0000", 2024-03-19T11:29:59.659 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5109999999999997e-06 2024-03-19T11:29:59.659 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.659 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.659 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:29:59.659 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.353409+0000", 2024-03-19T11:29:59.659 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2723999999999999e-05 2024-03-19T11:29:59.659 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.659 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.659 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:29:59.659 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.353428+0000", 2024-03-19T11:29:59.659 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8216e-05 2024-03-19T11:29:59.659 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.659 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.659 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:29:59.660 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.358359+0000", 2024-03-19T11:29:59.660 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0049314550000000004 2024-03-19T11:29:59.662 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.662 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.662 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:29:59.662 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.358392+0000", 2024-03-19T11:29:59.662 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3061000000000001e-05 2024-03-19T11:29:59.662 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:29:59.662 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:29:59.662 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:29:59.662 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.662 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.662 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:119 1.0 e16/14 1:30306672:devicehealth::main.db.0000000000000000:head v 16'81, mlcod=16'78)", 2024-03-19T11:29:59.663 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.364626+0000", 2024-03-19T11:29:59.663 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 14.286343137999999, 2024-03-19T11:29:59.663 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046261449999999999, 2024-03-19T11:29:59.663 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:29:59.663 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:29:59.663 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:29:59.663 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.663 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:29:59.663 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.364626+0000", 2024-03-19T11:29:59.663 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:29:59.663 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.663 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.663 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:29:59.663 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.364626+0000", 2024-03-19T11:29:59.663 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:29:59.664 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.664 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.664 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:29:59.664 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.364626+0000", 2024-03-19T11:29:59.664 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1399999999999996e-07 2024-03-19T11:29:59.664 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.664 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.664 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:29:59.664 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.364724+0000", 2024-03-19T11:29:59.664 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7972000000000003e-05 2024-03-19T11:29:59.664 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.664 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.664 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:29:59.664 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.364737+0000", 2024-03-19T11:29:59.665 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2585000000000001e-05 2024-03-19T11:29:59.665 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.665 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.665 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:29:59.665 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.364744+0000", 2024-03-19T11:29:59.665 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7780000000000002e-06 2024-03-19T11:29:59.665 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.665 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.665 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:29:59.665 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.364780+0000", 2024-03-19T11:29:59.665 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5992999999999999e-05 2024-03-19T11:29:59.665 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.665 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.665 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:29:59.665 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.364795+0000", 2024-03-19T11:29:59.666 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4939e-05 2024-03-19T11:29:59.666 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.666 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.666 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:29:59.666 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.369211+0000", 2024-03-19T11:29:59.666 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0044161139999999996 2024-03-19T11:29:59.666 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.666 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.666 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:29:59.666 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.369252+0000", 2024-03-19T11:29:59.666 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1050000000000002e-05 2024-03-19T11:29:59.666 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:29:59.666 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:29:59.666 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:29:59.667 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.667 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.667 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:120 1.0 e16/14 1:30306672:devicehealth::main.db.0000000000000000:head v 16'82, mlcod=16'78)", 2024-03-19T11:29:59.667 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.365811+0000", 2024-03-19T11:29:59.667 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 14.285158181, 2024-03-19T11:29:59.667 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0040843659999999999, 2024-03-19T11:29:59.667 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:29:59.667 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:29:59.667 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:29:59.667 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.667 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:29:59.667 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.365811+0000", 2024-03-19T11:29:59.667 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:29:59.667 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.667 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.668 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:29:59.668 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.365811+0000", 2024-03-19T11:29:59.668 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:29:59.668 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.668 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.668 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:29:59.668 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.365811+0000", 2024-03-19T11:29:59.668 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4199999999999995e-07 2024-03-19T11:29:59.668 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.668 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.668 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:29:59.668 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.365884+0000", 2024-03-19T11:29:59.668 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2527999999999994e-05 2024-03-19T11:29:59.668 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.669 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.669 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:29:59.669 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.365896+0000", 2024-03-19T11:29:59.669 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1912999999999999e-05 2024-03-19T11:29:59.669 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.669 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.669 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:29:59.669 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.365902+0000", 2024-03-19T11:29:59.669 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3130000000000004e-06 2024-03-19T11:29:59.669 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.669 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.669 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:29:59.669 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.365952+0000", 2024-03-19T11:29:59.669 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0283e-05 2024-03-19T11:29:59.669 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.670 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.670 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:29:59.670 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.365966+0000", 2024-03-19T11:29:59.670 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.349e-05 2024-03-19T11:29:59.670 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.670 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.670 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:29:59.670 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.369869+0000", 2024-03-19T11:29:59.670 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0039034989999999999 2024-03-19T11:29:59.670 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.670 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.670 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:29:59.670 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.369895+0000", 2024-03-19T11:29:59.670 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5598000000000001e-05 2024-03-19T11:29:59.671 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:29:59.671 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:29:59.671 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:29:59.671 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.671 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.671 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:129 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'87, mlcod=16'85)", 2024-03-19T11:29:59.671 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.805130+0000", 2024-03-19T11:29:59.671 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 13.845838541999999, 2024-03-19T11:29:59.671 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003788769, 2024-03-19T11:29:59.671 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:29:59.671 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:29:59.671 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:29:59.671 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.671 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:29:59.672 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.805130+0000", 2024-03-19T11:29:59.672 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:29:59.672 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.672 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.672 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:29:59.672 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.805130+0000", 2024-03-19T11:29:59.672 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:29:59.672 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.672 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.672 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:29:59.672 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.805132+0000", 2024-03-19T11:29:59.672 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.733e-06 2024-03-19T11:29:59.672 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.672 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.672 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:29:59.673 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.805145+0000", 2024-03-19T11:29:59.673 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2830000000000001e-05 2024-03-19T11:29:59.673 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.673 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.673 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:29:59.673 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.805173+0000", 2024-03-19T11:29:59.673 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.853e-05 2024-03-19T11:29:59.673 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.673 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.673 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:29:59.673 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.805187+0000", 2024-03-19T11:29:59.673 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3183000000000001e-05 2024-03-19T11:29:59.673 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.673 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.673 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:29:59.674 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.805329+0000", 2024-03-19T11:29:59.674 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000142578 2024-03-19T11:29:59.674 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.674 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.674 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:29:59.674 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.805366+0000", 2024-03-19T11:29:59.674 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6390000000000002e-05 2024-03-19T11:29:59.674 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.674 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.674 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:29:59.674 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808820+0000", 2024-03-19T11:29:59.674 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034547459999999999 2024-03-19T11:29:59.674 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.674 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.674 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:29:59.675 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808919+0000", 2024-03-19T11:29:59.675 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8778999999999997e-05 2024-03-19T11:29:59.675 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:29:59.675 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:29:59.675 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:29:59.675 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.675 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.675 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:130 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'88, mlcod=16'85)", 2024-03-19T11:29:59.675 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.805586+0000", 2024-03-19T11:29:59.675 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 13.845383113, 2024-03-19T11:29:59.675 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0054104690000000002, 2024-03-19T11:29:59.676 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:29:59.676 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:29:59.676 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:29:59.676 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.676 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:29:59.676 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.805586+0000", 2024-03-19T11:29:59.676 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:29:59.676 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.676 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.676 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:29:59.676 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.805586+0000", 2024-03-19T11:29:59.676 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:29:59.676 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.676 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.677 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:29:59.677 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.805587+0000", 2024-03-19T11:29:59.677 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7269999999999999e-06 2024-03-19T11:29:59.677 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.677 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.677 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:29:59.677 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.805600+0000", 2024-03-19T11:29:59.677 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2293000000000001e-05 2024-03-19T11:29:59.677 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.677 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.677 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:29:59.677 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.805623+0000", 2024-03-19T11:29:59.677 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3592999999999999e-05 2024-03-19T11:29:59.677 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.677 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.678 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:29:59.678 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.805636+0000", 2024-03-19T11:29:59.678 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2660000000000001e-05 2024-03-19T11:29:59.678 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.678 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.678 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:29:59.678 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.805838+0000", 2024-03-19T11:29:59.678 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020228800000000001 2024-03-19T11:29:59.678 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.678 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.678 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:29:59.678 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.805878+0000", 2024-03-19T11:29:59.678 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.9586999999999999e-05 2024-03-19T11:29:59.679 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.679 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.679 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:29:59.679 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.810908+0000", 2024-03-19T11:29:59.679 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050303090000000002 2024-03-19T11:29:59.679 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.679 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.679 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:29:59.679 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.810996+0000", 2024-03-19T11:29:59.679 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8011999999999997e-05 2024-03-19T11:29:59.679 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:29:59.679 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:29:59.679 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:29:59.679 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.679 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.680 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:131 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'89, mlcod=16'85)", 2024-03-19T11:29:59.680 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.806096+0000", 2024-03-19T11:29:59.680 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 13.844872464, 2024-03-19T11:29:59.680 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0049616699999999996, 2024-03-19T11:29:59.680 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:29:59.680 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:29:59.680 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:29:59.680 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.680 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:29:59.680 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806096+0000", 2024-03-19T11:29:59.680 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:29:59.680 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.680 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.680 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:29:59.680 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806096+0000", 2024-03-19T11:29:59.681 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:29:59.681 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.681 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.681 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:29:59.681 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806098+0000", 2024-03-19T11:29:59.681 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.734e-06 2024-03-19T11:29:59.681 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.681 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.681 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:29:59.681 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806267+0000", 2024-03-19T11:29:59.681 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016936800000000001 2024-03-19T11:29:59.681 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.681 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.681 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:29:59.681 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806296+0000", 2024-03-19T11:29:59.682 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8653000000000002e-05 2024-03-19T11:29:59.682 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.682 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.682 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:29:59.682 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806315+0000", 2024-03-19T11:29:59.682 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9049999999999999e-05 2024-03-19T11:29:59.682 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.682 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.682 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:29:59.682 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806481+0000", 2024-03-19T11:29:59.682 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016540599999999999 2024-03-19T11:29:59.682 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.682 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.683 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:29:59.683 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806511+0000", 2024-03-19T11:29:59.683 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0626000000000001e-05 2024-03-19T11:29:59.683 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.683 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.683 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:29:59.683 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.811002+0000", 2024-03-19T11:29:59.683 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0044912399999999996 2024-03-19T11:29:59.683 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.683 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.683 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:29:59.683 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.811058+0000", 2024-03-19T11:29:59.683 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5593000000000001e-05 2024-03-19T11:29:59.683 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:29:59.684 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:29:59.684 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:29:59.684 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.684 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.684 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:132 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'90, mlcod=16'85)", 2024-03-19T11:29:59.684 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.806470+0000", 2024-03-19T11:29:59.684 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 13.844498973, 2024-03-19T11:29:59.684 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0066812390000000003, 2024-03-19T11:29:59.684 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:29:59.684 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:29:59.684 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:29:59.684 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.684 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:29:59.684 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806470+0000", 2024-03-19T11:29:59.684 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:29:59.685 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.685 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.685 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:29:59.685 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806470+0000", 2024-03-19T11:29:59.685 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:29:59.685 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.685 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.685 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:29:59.685 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806471+0000", 2024-03-19T11:29:59.685 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.486e-06 2024-03-19T11:29:59.685 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.685 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.685 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:29:59.685 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806485+0000", 2024-03-19T11:29:59.685 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3380000000000001e-05 2024-03-19T11:29:59.686 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.686 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.686 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:29:59.686 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806509+0000", 2024-03-19T11:29:59.686 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4617000000000002e-05 2024-03-19T11:29:59.686 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.686 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.686 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:29:59.686 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806524+0000", 2024-03-19T11:29:59.686 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4688999999999999e-05 2024-03-19T11:29:59.686 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.686 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.687 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:29:59.687 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807070+0000", 2024-03-19T11:29:59.687 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00054601300000000003 2024-03-19T11:29:59.687 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.687 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.687 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:29:59.687 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807104+0000", 2024-03-19T11:29:59.687 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3575999999999998e-05 2024-03-19T11:29:59.687 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.687 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.687 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:29:59.687 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813077+0000", 2024-03-19T11:29:59.687 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0059733590000000001 2024-03-19T11:29:59.687 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.688 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.688 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:29:59.688 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813151+0000", 2024-03-19T11:29:59.688 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4119e-05 2024-03-19T11:29:59.688 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:29:59.688 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:29:59.688 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:29:59.688 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.688 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.688 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:133 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'91, mlcod=16'85)", 2024-03-19T11:29:59.688 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.806875+0000", 2024-03-19T11:29:59.688 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 13.84409331, 2024-03-19T11:29:59.688 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0063196260000000001, 2024-03-19T11:29:59.688 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:29:59.688 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:29:59.689 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:29:59.689 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.689 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:29:59.689 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806875+0000", 2024-03-19T11:29:59.689 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:29:59.689 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.689 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.689 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:29:59.689 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806875+0000", 2024-03-19T11:29:59.689 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:29:59.689 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.689 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.689 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:29:59.690 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806877+0000", 2024-03-19T11:29:59.690 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8899999999999999e-06 2024-03-19T11:29:59.690 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.690 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.690 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:29:59.690 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806895+0000", 2024-03-19T11:29:59.690 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7980000000000001e-05 2024-03-19T11:29:59.690 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.690 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.690 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:29:59.690 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806926+0000", 2024-03-19T11:29:59.690 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0363000000000001e-05 2024-03-19T11:29:59.690 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.691 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.691 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:29:59.691 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806942+0000", 2024-03-19T11:29:59.691 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6422999999999998e-05 2024-03-19T11:29:59.691 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.691 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.691 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:29:59.692 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807659+0000", 2024-03-19T11:29:59.692 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00071724699999999996 2024-03-19T11:29:59.692 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.692 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.692 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:29:59.692 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807688+0000", 2024-03-19T11:29:59.692 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8940000000000001e-05 2024-03-19T11:29:59.692 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.692 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.692 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:29:59.692 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813157+0000", 2024-03-19T11:29:59.692 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0054689930000000001 2024-03-19T11:29:59.692 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.692 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.692 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:29:59.693 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813195+0000", 2024-03-19T11:29:59.693 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7790000000000002e-05 2024-03-19T11:29:59.693 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:29:59.693 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:29:59.693 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:29:59.693 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.693 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.693 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:134 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'92, mlcod=16'85)", 2024-03-19T11:29:59.693 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.807418+0000", 2024-03-19T11:29:59.693 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 13.843550788, 2024-03-19T11:29:59.693 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0058240560000000002, 2024-03-19T11:29:59.693 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:29:59.693 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:29:59.693 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:29:59.694 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.694 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:29:59.694 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807418+0000", 2024-03-19T11:29:59.694 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:29:59.694 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.694 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.694 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:29:59.694 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807418+0000", 2024-03-19T11:29:59.694 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:29:59.694 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.694 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.694 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:29:59.694 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807420+0000", 2024-03-19T11:29:59.694 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5600000000000001e-06 2024-03-19T11:29:59.694 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.695 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.695 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:29:59.695 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807514+0000", 2024-03-19T11:29:59.695 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4256e-05 2024-03-19T11:29:59.695 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.695 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.695 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:29:59.695 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807542+0000", 2024-03-19T11:29:59.695 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8283e-05 2024-03-19T11:29:59.695 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.695 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.695 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:29:59.695 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807562+0000", 2024-03-19T11:29:59.695 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0126999999999999e-05 2024-03-19T11:29:59.696 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.696 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.696 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:29:59.696 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808272+0000", 2024-03-19T11:29:59.696 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00071012400000000002 2024-03-19T11:29:59.696 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.696 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.696 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:29:59.696 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808308+0000", 2024-03-19T11:29:59.696 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5722999999999997e-05 2024-03-19T11:29:59.696 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.696 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.696 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:29:59.696 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813200+0000", 2024-03-19T11:29:59.697 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0048915010000000004 2024-03-19T11:29:59.697 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.697 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.697 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:29:59.697 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813242+0000", 2024-03-19T11:29:59.697 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2481999999999997e-05 2024-03-19T11:29:59.697 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:29:59.697 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:29:59.697 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:29:59.697 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.697 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.697 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:135 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'93, mlcod=16'85)", 2024-03-19T11:29:59.697 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.807817+0000", 2024-03-19T11:29:59.697 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 13.843151551, 2024-03-19T11:29:59.697 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0054653059999999996, 2024-03-19T11:29:59.698 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:29:59.698 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:29:59.698 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:29:59.698 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.698 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:29:59.698 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807817+0000", 2024-03-19T11:29:59.698 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:29:59.698 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.698 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.698 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:29:59.698 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807817+0000", 2024-03-19T11:29:59.698 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:29:59.698 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.698 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.699 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:29:59.699 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807819+0000", 2024-03-19T11:29:59.699 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7999999999999999e-06 2024-03-19T11:29:59.699 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.699 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.699 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:29:59.699 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807833+0000", 2024-03-19T11:29:59.699 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3546999999999999e-05 2024-03-19T11:29:59.699 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.699 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.699 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:29:59.699 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807858+0000", 2024-03-19T11:29:59.699 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5119000000000001e-05 2024-03-19T11:29:59.699 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.700 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.700 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:29:59.700 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807871+0000", 2024-03-19T11:29:59.700 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.287e-05 2024-03-19T11:29:59.700 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.700 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.700 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:29:59.700 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808989+0000", 2024-03-19T11:29:59.700 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001118841 2024-03-19T11:29:59.700 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.700 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.700 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:29:59.700 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.809022+0000", 2024-03-19T11:29:59.700 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3000000000000003e-05 2024-03-19T11:29:59.700 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.701 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.701 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:29:59.701 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813249+0000", 2024-03-19T11:29:59.701 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0042268260000000004 2024-03-19T11:29:59.701 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.701 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.701 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:29:59.701 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813283+0000", 2024-03-19T11:29:59.701 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3303000000000003e-05 2024-03-19T11:29:59.701 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:29:59.701 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:29:59.701 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:29:59.701 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.701 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.701 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:136 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'94, mlcod=16'85)", 2024-03-19T11:29:59.702 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.808219+0000", 2024-03-19T11:29:59.702 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 13.842750271, 2024-03-19T11:29:59.702 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005105086, 2024-03-19T11:29:59.702 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:29:59.702 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:29:59.702 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:29:59.702 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.702 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:29:59.702 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808219+0000", 2024-03-19T11:29:59.702 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:29:59.702 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.702 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.702 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:29:59.703 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808219+0000", 2024-03-19T11:29:59.703 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:29:59.703 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.703 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.703 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:29:59.703 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808220+0000", 2024-03-19T11:29:59.703 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7600000000000001e-06 2024-03-19T11:29:59.703 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.703 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.703 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:29:59.703 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808233+0000", 2024-03-19T11:29:59.703 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2916e-05 2024-03-19T11:29:59.703 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.703 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.704 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:29:59.704 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808294+0000", 2024-03-19T11:29:59.704 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1199999999999997e-05 2024-03-19T11:29:59.704 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.704 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.704 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:29:59.704 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808310+0000", 2024-03-19T11:29:59.704 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5996999999999999e-05 2024-03-19T11:29:59.704 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.704 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.704 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:29:59.704 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.809445+0000", 2024-03-19T11:29:59.704 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001134227 2024-03-19T11:29:59.704 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.704 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.705 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:29:59.705 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.809473+0000", 2024-03-19T11:29:59.705 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8603e-05 2024-03-19T11:29:59.705 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.705 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.705 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:29:59.705 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813291+0000", 2024-03-19T11:29:59.705 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038174799999999998 2024-03-19T11:29:59.705 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.705 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.705 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:29:59.705 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813324+0000", 2024-03-19T11:29:59.705 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2903e-05 2024-03-19T11:29:59.705 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:29:59.705 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:29:59.706 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:29:59.706 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.706 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.706 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:137 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'95, mlcod=16'85)", 2024-03-19T11:29:59.706 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.808663+0000", 2024-03-19T11:29:59.706 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 13.842306118, 2024-03-19T11:29:59.706 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0047004350000000002, 2024-03-19T11:29:59.706 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:29:59.706 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:29:59.706 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:29:59.706 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.706 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:29:59.706 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808663+0000", 2024-03-19T11:29:59.706 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:29:59.707 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.707 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.707 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:29:59.707 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808663+0000", 2024-03-19T11:29:59.707 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:29:59.707 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.707 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.707 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:29:59.707 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808665+0000", 2024-03-19T11:29:59.707 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9300000000000002e-06 2024-03-19T11:29:59.707 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.707 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.707 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:29:59.707 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808768+0000", 2024-03-19T11:29:59.707 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000103009 2024-03-19T11:29:59.708 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.708 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.708 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:29:59.708 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808792+0000", 2024-03-19T11:29:59.708 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3972999999999999e-05 2024-03-19T11:29:59.708 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.708 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.708 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:29:59.708 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808807+0000", 2024-03-19T11:29:59.708 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5614e-05 2024-03-19T11:29:59.708 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.708 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.708 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:29:59.708 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.809896+0000", 2024-03-19T11:29:59.708 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0010891270000000001 2024-03-19T11:29:59.709 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.709 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.709 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:29:59.709 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.809930+0000", 2024-03-19T11:29:59.709 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3444e-05 2024-03-19T11:29:59.709 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.709 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.709 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:29:59.709 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813330+0000", 2024-03-19T11:29:59.709 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003399839 2024-03-19T11:29:59.709 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.709 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.709 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:29:59.709 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813363+0000", 2024-03-19T11:29:59.710 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3498999999999999e-05 2024-03-19T11:29:59.710 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:29:59.710 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:29:59.710 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:29:59.710 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.710 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.710 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:140 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'98, mlcod=16'86)", 2024-03-19T11:29:59.710 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.811388+0000", 2024-03-19T11:29:59.710 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 13.839580387, 2024-03-19T11:29:59.710 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033436199999999998, 2024-03-19T11:29:59.710 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:29:59.710 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:29:59.710 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:29:59.710 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.711 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:29:59.711 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.811388+0000", 2024-03-19T11:29:59.711 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:29:59.711 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.711 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.711 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:29:59.711 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.811388+0000", 2024-03-19T11:29:59.711 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:29:59.711 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.711 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.711 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:29:59.711 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.811396+0000", 2024-03-19T11:29:59.711 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5769999999999996e-06 2024-03-19T11:29:59.711 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.712 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.712 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:29:59.712 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.811462+0000", 2024-03-19T11:29:59.712 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5820000000000003e-05 2024-03-19T11:29:59.712 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.712 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.712 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:29:59.712 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.811491+0000", 2024-03-19T11:29:59.712 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8979000000000002e-05 2024-03-19T11:29:59.712 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.712 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.712 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:29:59.712 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.811508+0000", 2024-03-19T11:29:59.712 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.721e-05 2024-03-19T11:29:59.712 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.713 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.713 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:29:59.713 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.812088+0000", 2024-03-19T11:29:59.713 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00057991599999999998 2024-03-19T11:29:59.713 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.713 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.713 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:29:59.713 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.812121+0000", 2024-03-19T11:29:59.713 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2836000000000002e-05 2024-03-19T11:29:59.713 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.713 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.713 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:29:59.713 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.814640+0000", 2024-03-19T11:29:59.713 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025190329999999999 2024-03-19T11:29:59.714 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.714 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.714 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:29:59.714 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.814732+0000", 2024-03-19T11:29:59.714 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2249000000000004e-05 2024-03-19T11:29:59.714 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:29:59.714 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:29:59.714 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:29:59.714 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.714 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.714 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:141 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'99, mlcod=16'86)", 2024-03-19T11:29:59.714 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.812015+0000", 2024-03-19T11:29:59.714 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 13.838954092, 2024-03-19T11:29:59.714 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003849239, 2024-03-19T11:29:59.714 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:29:59.715 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:29:59.715 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:29:59.715 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.715 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:29:59.715 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.812015+0000", 2024-03-19T11:29:59.715 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:29:59.715 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.715 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.715 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:29:59.715 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.812015+0000", 2024-03-19T11:29:59.715 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:29:59.715 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.715 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.715 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:29:59.715 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.812017+0000", 2024-03-19T11:29:59.716 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8470000000000001e-06 2024-03-19T11:29:59.716 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.716 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.716 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:29:59.716 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.812030+0000", 2024-03-19T11:29:59.716 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3867e-05 2024-03-19T11:29:59.716 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.716 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.716 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:29:59.716 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.812058+0000", 2024-03-19T11:29:59.716 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8005999999999999e-05 2024-03-19T11:29:59.716 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.716 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.716 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:29:59.716 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.812070+0000", 2024-03-19T11:29:59.717 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1207e-05 2024-03-19T11:29:59.717 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.717 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.717 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:29:59.717 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813509+0000", 2024-03-19T11:29:59.717 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014389909999999999 2024-03-19T11:29:59.717 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.717 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.717 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:29:59.717 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813564+0000", 2024-03-19T11:29:59.717 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5420000000000001e-05 2024-03-19T11:29:59.717 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.717 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.717 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:29:59.718 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.815782+0000", 2024-03-19T11:29:59.718 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022183419999999999 2024-03-19T11:29:59.718 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.718 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.718 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:29:59.718 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.815864+0000", 2024-03-19T11:29:59.718 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1558999999999996e-05 2024-03-19T11:29:59.718 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:29:59.718 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:29:59.718 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:29:59.718 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.718 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.718 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:142 1.0 e16/14 1:30306672:devicehealth::main.db.0000000000000000:head v 16'100, mlcod=16'98)", 2024-03-19T11:29:59.718 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.817085+0000", 2024-03-19T11:29:59.718 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 13.833883742999999, 2024-03-19T11:29:59.719 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0035837740000000001, 2024-03-19T11:29:59.719 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:29:59.719 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:29:59.719 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:29:59.719 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.719 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:29:59.719 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.817085+0000", 2024-03-19T11:29:59.719 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:29:59.719 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.719 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.719 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:29:59.719 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.817085+0000", 2024-03-19T11:29:59.719 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:29:59.719 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.720 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.720 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:29:59.720 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.817092+0000", 2024-03-19T11:29:59.720 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.46e-06 2024-03-19T11:29:59.720 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.720 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.720 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:29:59.720 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.817365+0000", 2024-03-19T11:29:59.720 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00027311099999999998 2024-03-19T11:29:59.720 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.720 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.720 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:29:59.720 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.817390+0000", 2024-03-19T11:29:59.720 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.55e-05 2024-03-19T11:29:59.720 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.721 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.721 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:29:59.721 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.817403+0000", 2024-03-19T11:29:59.721 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2680000000000001e-05 2024-03-19T11:29:59.721 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.721 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.721 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:29:59.721 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.817521+0000", 2024-03-19T11:29:59.721 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000117862 2024-03-19T11:29:59.721 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.721 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.721 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:29:59.721 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.817558+0000", 2024-03-19T11:29:59.721 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7280000000000002e-05 2024-03-19T11:29:59.721 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.722 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.722 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:29:59.722 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.820599+0000", 2024-03-19T11:29:59.722 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030407780000000001 2024-03-19T11:29:59.722 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.722 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.722 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:29:59.722 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.820669+0000", 2024-03-19T11:29:59.722 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.0103000000000003e-05 2024-03-19T11:29:59.722 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:29:59.722 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:29:59.722 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:29:59.722 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.722 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.723 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:143 1.0 e16/14 1:30306672:devicehealth::main.db.0000000000000000:head v 16'101, mlcod=16'98)", 2024-03-19T11:29:59.723 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.817553+0000", 2024-03-19T11:29:59.723 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 13.833416292999999, 2024-03-19T11:29:59.723 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031569139999999998, 2024-03-19T11:29:59.723 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:29:59.723 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:29:59.723 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:29:59.723 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.723 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:29:59.723 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.817553+0000", 2024-03-19T11:29:59.723 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:29:59.723 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.723 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.723 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:29:59.724 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.817553+0000", 2024-03-19T11:29:59.724 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:29:59.724 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.724 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.724 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:29:59.724 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.817554+0000", 2024-03-19T11:29:59.724 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.026e-06 2024-03-19T11:29:59.724 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.724 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.724 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:29:59.724 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.817691+0000", 2024-03-19T11:29:59.724 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013721600000000001 2024-03-19T11:29:59.724 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.724 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.724 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:29:59.725 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.817712+0000", 2024-03-19T11:29:59.725 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0913000000000001e-05 2024-03-19T11:29:59.725 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.725 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.725 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:29:59.725 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.817724+0000", 2024-03-19T11:29:59.725 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2363e-05 2024-03-19T11:29:59.725 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.725 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.725 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:29:59.725 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.818276+0000", 2024-03-19T11:29:59.725 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00055174200000000003 2024-03-19T11:29:59.725 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.725 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.725 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:29:59.726 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.818314+0000", 2024-03-19T11:29:59.726 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8367e-05 2024-03-19T11:29:59.726 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.726 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.726 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:29:59.726 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.820673+0000", 2024-03-19T11:29:59.726 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023588670000000002 2024-03-19T11:29:59.726 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:29:59.726 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:29:59.726 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:29:59.726 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.820709+0000", 2024-03-19T11:29:59.726 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.642e-05 2024-03-19T11:29:59.726 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:29:59.726 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:29:59.726 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:29:59.727 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:29:59.727 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:29:59.727 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:29:59.728 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-19T11:29:59.729 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:29:59.740+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:29:59.828 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:29:59.836+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:29:59.903 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:29:59.904 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:29:59.904 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:29:59.904 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:29:59.919 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-19T11:29:59.921 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-19T11:29:59.963 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:29:59.972+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:00.031 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:00.040+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:00.088 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:00.088 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:30:00.088 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:30:00.088 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:30:00.088 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:00.103 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-19T11:30:00.107 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:30:00.108 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:30:00.124 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:30:00.125 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:30:00.132 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:00.140+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:00.141 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:30:00.142 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:30:00.158 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:30:00.158 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:30:00.179 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:30:00.180 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:30:00.200 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:30:00.200 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:30:00.217 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-19T11:30:00.217 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-19T11:30:00.217 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-19T11:30:00.217 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-03-19T11:30:00.222 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:30:00.222 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:30:00.224 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:00.224 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:00.233 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:00.240+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:00.240 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-19T11:30:00.245 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:30:00.245 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:30:00.334 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:00.344+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:00.376 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:00.376 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:00.391 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-19T11:30:00.438 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:00.448+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:00.509 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:00.510 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:00.525 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-19T11:30:00.529 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:30:00.529 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:00.529 INFO:teuthology.orchestra.run.smithi038.stdout: "epoch": 23, 2024-03-19T11:30:00.529 INFO:teuthology.orchestra.run.smithi038.stdout: "fsid": "246bcf68-c431-4aa7-b787-5e9b2d9519d6", 2024-03-19T11:30:00.529 INFO:teuthology.orchestra.run.smithi038.stdout: "created": "2024-03-19T11:29:33.327073+0000", 2024-03-19T11:30:00.529 INFO:teuthology.orchestra.run.smithi038.stdout: "modified": "2024-03-19T11:29:59.551091+0000", 2024-03-19T11:30:00.529 INFO:teuthology.orchestra.run.smithi038.stdout: "last_up_change": "2024-03-19T11:29:43.166618+0000", 2024-03-19T11:30:00.529 INFO:teuthology.orchestra.run.smithi038.stdout: "last_in_change": "2024-03-19T11:29:35.632327+0000", 2024-03-19T11:30:00.529 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-19T11:30:00.529 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_num": 5806080, 2024-03-19T11:30:00.530 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_set": [ 2024-03-19T11:30:00.530 INFO:teuthology.orchestra.run.smithi038.stdout: "nodeep-scrub", 2024-03-19T11:30:00.530 INFO:teuthology.orchestra.run.smithi038.stdout: "noscrub", 2024-03-19T11:30:00.530 INFO:teuthology.orchestra.run.smithi038.stdout: "pglog_hardlimit", 2024-03-19T11:30:00.530 INFO:teuthology.orchestra.run.smithi038.stdout: "purged_snapdirs", 2024-03-19T11:30:00.530 INFO:teuthology.orchestra.run.smithi038.stdout: "recovery_deletes", 2024-03-19T11:30:00.530 INFO:teuthology.orchestra.run.smithi038.stdout: "sortbitwise" 2024-03-19T11:30:00.530 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-19T11:30:00.530 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_version": 5, 2024-03-19T11:30:00.530 INFO:teuthology.orchestra.run.smithi038.stdout: "full_ratio": 0.94999998807907104, 2024-03-19T11:30:00.530 INFO:teuthology.orchestra.run.smithi038.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-19T11:30:00.530 INFO:teuthology.orchestra.run.smithi038.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-19T11:30:00.530 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_snapshot": "", 2024-03-19T11:30:00.531 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_max": 4, 2024-03-19T11:30:00.531 INFO:teuthology.orchestra.run.smithi038.stdout: "max_osd": 8, 2024-03-19T11:30:00.531 INFO:teuthology.orchestra.run.smithi038.stdout: "require_min_compat_client": "reef", 2024-03-19T11:30:00.531 INFO:teuthology.orchestra.run.smithi038.stdout: "min_compat_client": "jewel", 2024-03-19T11:30:00.531 INFO:teuthology.orchestra.run.smithi038.stdout: "require_osd_release": "squid", 2024-03-19T11:30:00.531 INFO:teuthology.orchestra.run.smithi038.stdout: "allow_crimson": false, 2024-03-19T11:30:00.531 INFO:teuthology.orchestra.run.smithi038.stdout: "pools": [ 2024-03-19T11:30:00.531 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.531 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 1, 2024-03-19T11:30:00.531 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": ".mgr", 2024-03-19T11:30:00.531 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-19T11:29:42.219654+0000", 2024-03-19T11:30:00.531 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-19T11:30:00.531 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-03-19T11:30:00.531 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-19T11:30:00.532 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 2, 2024-03-19T11:30:00.532 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-19T11:30:00.532 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-19T11:30:00.532 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-19T11:30:00.532 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-19T11:30:00.532 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-19T11:30:00.532 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-19T11:30:00.532 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-19T11:30:00.532 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-19T11:30:00.532 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 1, 2024-03-19T11:30:00.532 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 1, 2024-03-19T11:30:00.532 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 1, 2024-03-19T11:30:00.532 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 1, 2024-03-19T11:30:00.532 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 1, 2024-03-19T11:30:00.533 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-19T11:30:00.533 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-19T11:30:00.533 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-19T11:30:00.533 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-19T11:30:00.533 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-19T11:30:00.533 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-19T11:30:00.533 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-19T11:30:00.533 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.533 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "16", 2024-03-19T11:30:00.533 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-19T11:30:00.533 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-19T11:30:00.533 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-19T11:30:00.533 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-19T11:30:00.533 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-19T11:30:00.533 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-03-19T11:30:00.534 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-03-19T11:30:00.534 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-19T11:30:00.534 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-19T11:30:00.534 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-19T11:30:00.534 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-19T11:30:00.534 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-19T11:30:00.534 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-19T11:30:00.534 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-19T11:30:00.534 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-19T11:30:00.535 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-19T11:30:00.535 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-19T11:30:00.535 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-19T11:30:00.535 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-19T11:30:00.535 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-19T11:30:00.535 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-19T11:30:00.535 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-19T11:30:00.535 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-19T11:30:00.535 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-19T11:30:00.535 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-19T11:30:00.535 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-19T11:30:00.535 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.536 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-19T11:30:00.536 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-19T11:30:00.536 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-19T11:30:00.536 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-19T11:30:00.536 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-19T11:30:00.536 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-19T11:30:00.536 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-19T11:30:00.536 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-19T11:30:00.536 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-19T11:30:00.536 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-19T11:30:00.536 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-19T11:30:00.536 INFO:teuthology.orchestra.run.smithi038.stdout: "options": { 2024-03-19T11:30:00.536 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_max": 32, 2024-03-19T11:30:00.536 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_min": 1 2024-03-19T11:30:00.537 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.537 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-19T11:30:00.537 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr": {} 2024-03-19T11:30:00.537 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.537 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-19T11:30:00.537 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-19T11:30:00.537 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 8, 2024-03-19T11:30:00.537 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 8, 2024-03-19T11:30:00.537 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.25, 2024-03-19T11:30:00.537 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 2, 2024-03-19T11:30:00.537 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 2, 2024-03-19T11:30:00.537 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 1, 2024-03-19T11:30:00.537 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 1, 2024-03-19T11:30:00.537 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 1 2024-03-19T11:30:00.537 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:00.538 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.538 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.538 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 2, 2024-03-19T11:30:00.538 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": "rbd", 2024-03-19T11:30:00.538 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-19T11:29:44.428406+0000", 2024-03-19T11:30:00.538 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-19T11:30:00.538 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-03-19T11:30:00.538 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-19T11:30:00.538 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 2, 2024-03-19T11:30:00.538 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-19T11:30:00.538 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-19T11:30:00.538 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-19T11:30:00.538 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-19T11:30:00.538 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-19T11:30:00.539 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-19T11:30:00.539 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-19T11:30:00.539 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-19T11:30:00.539 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 8, 2024-03-19T11:30:00.539 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 8, 2024-03-19T11:30:00.539 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 8, 2024-03-19T11:30:00.539 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 8, 2024-03-19T11:30:00.539 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 8, 2024-03-19T11:30:00.539 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-19T11:30:00.539 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-19T11:30:00.539 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-19T11:30:00.539 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-19T11:30:00.539 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-19T11:30:00.539 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-19T11:30:00.540 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-19T11:30:00.540 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.540 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "18", 2024-03-19T11:30:00.540 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-19T11:30:00.540 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-19T11:30:00.540 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-19T11:30:00.540 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-19T11:30:00.540 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-19T11:30:00.540 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-03-19T11:30:00.540 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-03-19T11:30:00.540 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-19T11:30:00.540 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-19T11:30:00.540 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-19T11:30:00.540 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-19T11:30:00.541 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-19T11:30:00.541 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-19T11:30:00.541 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-19T11:30:00.541 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-19T11:30:00.541 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-19T11:30:00.541 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-19T11:30:00.541 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-19T11:30:00.541 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-19T11:30:00.541 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-19T11:30:00.541 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-19T11:30:00.541 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-19T11:30:00.541 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-19T11:30:00.541 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-19T11:30:00.541 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-19T11:30:00.541 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-19T11:30:00.542 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.542 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-19T11:30:00.542 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-19T11:30:00.542 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-19T11:30:00.542 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-19T11:30:00.542 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-19T11:30:00.542 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-19T11:30:00.542 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-19T11:30:00.542 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-19T11:30:00.542 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-19T11:30:00.542 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-19T11:30:00.542 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-19T11:30:00.542 INFO:teuthology.orchestra.run.smithi038.stdout: "options": {}, 2024-03-19T11:30:00.542 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-19T11:30:00.543 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd": {} 2024-03-19T11:30:00.543 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.543 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-19T11:30:00.543 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-19T11:30:00.543 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 1.9900000095367432, 2024-03-19T11:30:00.543 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 1.9900000095367432, 2024-03-19T11:30:00.543 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.87999999523162842, 2024-03-19T11:30:00.543 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 1.75, 2024-03-19T11:30:00.543 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 1.75, 2024-03-19T11:30:00.543 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 1, 2024-03-19T11:30:00.543 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 1, 2024-03-19T11:30:00.543 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 1 2024-03-19T11:30:00.543 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:00.543 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.544 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.544 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 3, 2024-03-19T11:30:00.544 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": "low_tier", 2024-03-19T11:30:00.544 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-19T11:29:55.608699+0000", 2024-03-19T11:30:00.544 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-19T11:30:00.544 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-03-19T11:30:00.544 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-19T11:30:00.544 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 2, 2024-03-19T11:30:00.544 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-19T11:30:00.544 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-19T11:30:00.544 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-19T11:30:00.544 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-19T11:30:00.544 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-19T11:30:00.545 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-19T11:30:00.545 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-19T11:30:00.545 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-19T11:30:00.545 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 4, 2024-03-19T11:30:00.545 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 4, 2024-03-19T11:30:00.546 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 4, 2024-03-19T11:30:00.546 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 4, 2024-03-19T11:30:00.546 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 4, 2024-03-19T11:30:00.546 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-19T11:30:00.546 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-19T11:30:00.546 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-19T11:30:00.547 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-19T11:30:00.547 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-19T11:30:00.547 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-19T11:30:00.547 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-19T11:30:00.547 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.547 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "22", 2024-03-19T11:30:00.548 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-19T11:30:00.548 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-19T11:30:00.548 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-19T11:30:00.548 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-19T11:30:00.548 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-19T11:30:00.548 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-03-19T11:30:00.548 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-03-19T11:30:00.548 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-19T11:30:00.548 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-19T11:30:00.548 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-19T11:30:00.548 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-19T11:30:00.549 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-19T11:30:00.549 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-19T11:30:00.549 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-19T11:30:00.549 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-19T11:30:00.549 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-19T11:30:00.549 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-19T11:30:00.549 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-19T11:30:00.549 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-19T11:30:00.549 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-19T11:30:00.549 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-19T11:30:00.549 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-19T11:30:00.549 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-19T11:30:00.549 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-19T11:30:00.549 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-19T11:30:00.550 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-19T11:30:00.550 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.550 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-19T11:30:00.550 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-19T11:30:00.550 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-19T11:30:00.550 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-19T11:30:00.550 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-19T11:30:00.550 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-19T11:30:00.550 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-19T11:30:00.550 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-19T11:30:00.550 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-19T11:30:00.550 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-19T11:30:00.550 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-19T11:30:00.550 INFO:teuthology.orchestra.run.smithi038.stdout: "options": {}, 2024-03-19T11:30:00.551 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": {}, 2024-03-19T11:30:00.551 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-19T11:30:00.551 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-19T11:30:00.551 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 2, 2024-03-19T11:30:00.551 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 2, 2024-03-19T11:30:00.551 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.75, 2024-03-19T11:30:00.551 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 1.5, 2024-03-19T11:30:00.551 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 1.5, 2024-03-19T11:30:00.551 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 1, 2024-03-19T11:30:00.551 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 1, 2024-03-19T11:30:00.551 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 1 2024-03-19T11:30:00.551 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:00.551 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.552 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.552 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 4, 2024-03-19T11:30:00.552 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": "unique_pool_0", 2024-03-19T11:30:00.552 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-19T11:29:56.945423+0000", 2024-03-19T11:30:00.552 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-19T11:30:00.552 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-03-19T11:30:00.552 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-19T11:30:00.552 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 2, 2024-03-19T11:30:00.552 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-19T11:30:00.552 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-19T11:30:00.552 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-19T11:30:00.552 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-19T11:30:00.552 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-19T11:30:00.552 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-19T11:30:00.553 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-19T11:30:00.553 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-19T11:30:00.553 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 16, 2024-03-19T11:30:00.553 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 16, 2024-03-19T11:30:00.553 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 16, 2024-03-19T11:30:00.553 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 16, 2024-03-19T11:30:00.553 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 16, 2024-03-19T11:30:00.553 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-19T11:30:00.553 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-19T11:30:00.553 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-19T11:30:00.553 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-19T11:30:00.553 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-19T11:30:00.553 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-19T11:30:00.553 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-19T11:30:00.553 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.554 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "23", 2024-03-19T11:30:00.554 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-19T11:30:00.554 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-19T11:30:00.554 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-19T11:30:00.554 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-19T11:30:00.554 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-19T11:30:00.554 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-03-19T11:30:00.554 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-03-19T11:30:00.554 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-19T11:30:00.554 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-19T11:30:00.554 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-19T11:30:00.554 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-19T11:30:00.554 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-19T11:30:00.555 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-19T11:30:00.555 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-19T11:30:00.555 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-19T11:30:00.555 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-19T11:30:00.555 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-19T11:30:00.555 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-19T11:30:00.555 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-19T11:30:00.555 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-19T11:30:00.555 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-19T11:30:00.555 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-19T11:30:00.555 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-19T11:30:00.556 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-19T11:30:00.556 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-19T11:30:00.556 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-19T11:30:00.556 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.556 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-19T11:30:00.556 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-19T11:30:00.556 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-19T11:30:00.556 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-19T11:30:00.556 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-19T11:30:00.556 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-19T11:30:00.556 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-19T11:30:00.556 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-19T11:30:00.556 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-19T11:30:00.556 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-19T11:30:00.557 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-19T11:30:00.557 INFO:teuthology.orchestra.run.smithi038.stdout: "options": {}, 2024-03-19T11:30:00.557 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-19T11:30:00.557 INFO:teuthology.orchestra.run.smithi038.stdout: "rados": {} 2024-03-19T11:30:00.557 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.557 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-19T11:30:00.557 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-19T11:30:00.557 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 2.5, 2024-03-19T11:30:00.557 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 2.5, 2024-03-19T11:30:00.557 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 1, 2024-03-19T11:30:00.557 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 2.5, 2024-03-19T11:30:00.557 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 2.5, 2024-03-19T11:30:00.557 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 1, 2024-03-19T11:30:00.557 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 1, 2024-03-19T11:30:00.558 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 1 2024-03-19T11:30:00.558 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:00.558 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:00.558 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-19T11:30:00.558 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-03-19T11:30:00.558 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.558 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 0, 2024-03-19T11:30:00.558 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "3bb2d2be-5253-4da0-acf3-85a114dc3801", 2024-03-19T11:30:00.558 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-19T11:30:00.558 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-19T11:30:00.558 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-19T11:30:00.558 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-19T11:30:00.558 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-19T11:30:00.558 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-19T11:30:00.559 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 13, 2024-03-19T11:30:00.559 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 21, 2024-03-19T11:30:00.559 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-19T11:30:00.559 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-19T11:30:00.559 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-19T11:30:00.559 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:00.559 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.559 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:00.559 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6808", 2024-03-19T11:30:00.559 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3148574891 2024-03-19T11:30:00.559 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.559 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.559 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:00.559 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6811", 2024-03-19T11:30:00.560 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3148574891 2024-03-19T11:30:00.560 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:00.560 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:00.560 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.560 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-19T11:30:00.560 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:00.560 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.560 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:00.560 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6813", 2024-03-19T11:30:00.560 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3148574891 2024-03-19T11:30:00.560 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.560 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.560 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:00.560 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6815", 2024-03-19T11:30:00.561 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3148574891 2024-03-19T11:30:00.561 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:00.561 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:00.561 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.561 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-19T11:30:00.561 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:00.561 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.561 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:00.561 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6822", 2024-03-19T11:30:00.561 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3148574891 2024-03-19T11:30:00.561 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.561 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.561 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:00.561 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6823", 2024-03-19T11:30:00.562 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3148574891 2024-03-19T11:30:00.562 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:00.562 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:00.562 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.562 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-19T11:30:00.562 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:00.562 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.562 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:00.562 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6818", 2024-03-19T11:30:00.562 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3148574891 2024-03-19T11:30:00.562 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.562 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.562 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:00.562 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6821", 2024-03-19T11:30:00.563 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3148574891 2024-03-19T11:30:00.563 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:00.563 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:00.563 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.563 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6811/3148574891", 2024-03-19T11:30:00.563 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6815/3148574891", 2024-03-19T11:30:00.563 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6823/3148574891", 2024-03-19T11:30:00.563 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6821/3148574891", 2024-03-19T11:30:00.563 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-19T11:30:00.563 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-19T11:30:00.563 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-19T11:30:00.563 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:00.563 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.563 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.564 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 1, 2024-03-19T11:30:00.564 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "e1c926b2-e0be-4ccc-b8a5-d80d0b584ead", 2024-03-19T11:30:00.564 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-19T11:30:00.564 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-19T11:30:00.564 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-19T11:30:00.564 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-19T11:30:00.564 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-19T11:30:00.564 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-19T11:30:00.564 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 14, 2024-03-19T11:30:00.564 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 21, 2024-03-19T11:30:00.564 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-19T11:30:00.564 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-19T11:30:00.564 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-19T11:30:00.564 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:00.564 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.565 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:00.565 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6824", 2024-03-19T11:30:00.565 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 675027622 2024-03-19T11:30:00.565 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.565 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.565 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:00.565 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6825", 2024-03-19T11:30:00.565 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 675027622 2024-03-19T11:30:00.565 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:00.565 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:00.565 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.565 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-19T11:30:00.565 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:00.565 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.566 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:00.566 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6826", 2024-03-19T11:30:00.566 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 675027622 2024-03-19T11:30:00.566 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.566 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.566 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:00.566 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6827", 2024-03-19T11:30:00.566 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 675027622 2024-03-19T11:30:00.566 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:00.566 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:00.566 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.566 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-19T11:30:00.566 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:00.566 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.567 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:00.567 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6830", 2024-03-19T11:30:00.567 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 675027622 2024-03-19T11:30:00.567 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.567 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.567 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:00.567 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6831", 2024-03-19T11:30:00.567 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 675027622 2024-03-19T11:30:00.567 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:00.567 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:00.567 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.567 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-19T11:30:00.567 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:00.567 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.568 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:00.568 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6828", 2024-03-19T11:30:00.568 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 675027622 2024-03-19T11:30:00.568 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.568 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.568 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:00.568 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6829", 2024-03-19T11:30:00.568 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 675027622 2024-03-19T11:30:00.568 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:00.568 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:00.568 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.568 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6825/675027622", 2024-03-19T11:30:00.568 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6827/675027622", 2024-03-19T11:30:00.568 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6831/675027622", 2024-03-19T11:30:00.569 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6829/675027622", 2024-03-19T11:30:00.569 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-19T11:30:00.569 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-19T11:30:00.569 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-19T11:30:00.569 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:00.569 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.569 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.569 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 2, 2024-03-19T11:30:00.569 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "61262219-4f8a-4733-af9d-b5208a17c0ea", 2024-03-19T11:30:00.569 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-19T11:30:00.569 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-19T11:30:00.569 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-19T11:30:00.569 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-19T11:30:00.569 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-19T11:30:00.569 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-19T11:30:00.570 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 14, 2024-03-19T11:30:00.570 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 16, 2024-03-19T11:30:00.570 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-19T11:30:00.570 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-19T11:30:00.570 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-19T11:30:00.570 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:00.570 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.570 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:00.570 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6800", 2024-03-19T11:30:00.570 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 58279616 2024-03-19T11:30:00.570 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.570 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.570 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:00.570 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6801", 2024-03-19T11:30:00.571 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 58279616 2024-03-19T11:30:00.571 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:00.571 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:00.571 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.571 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-19T11:30:00.571 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:00.571 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.571 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:00.571 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6802", 2024-03-19T11:30:00.571 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 58279616 2024-03-19T11:30:00.571 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.571 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.571 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:00.571 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6804", 2024-03-19T11:30:00.571 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 58279616 2024-03-19T11:30:00.572 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:00.572 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:00.572 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.572 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-19T11:30:00.572 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:00.572 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.572 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:00.572 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6814", 2024-03-19T11:30:00.572 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 58279616 2024-03-19T11:30:00.572 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.572 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.572 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:00.572 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6817", 2024-03-19T11:30:00.572 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 58279616 2024-03-19T11:30:00.573 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:00.573 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:00.573 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.573 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-19T11:30:00.573 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:00.573 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.573 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:00.573 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6807", 2024-03-19T11:30:00.573 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 58279616 2024-03-19T11:30:00.573 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.573 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.573 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:00.573 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6810", 2024-03-19T11:30:00.573 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 58279616 2024-03-19T11:30:00.574 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:00.574 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:00.574 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.574 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6801/58279616", 2024-03-19T11:30:00.574 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6804/58279616", 2024-03-19T11:30:00.574 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6817/58279616", 2024-03-19T11:30:00.574 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6810/58279616", 2024-03-19T11:30:00.574 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-19T11:30:00.574 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-19T11:30:00.574 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-19T11:30:00.574 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:00.574 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.574 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.574 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 3, 2024-03-19T11:30:00.574 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "a31f8d1b-4e38-4ca3-b251-0877002d7235", 2024-03-19T11:30:00.575 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-19T11:30:00.575 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-19T11:30:00.575 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-19T11:30:00.575 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-19T11:30:00.575 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-19T11:30:00.575 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-19T11:30:00.575 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 14, 2024-03-19T11:30:00.575 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 21, 2024-03-19T11:30:00.575 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-19T11:30:00.575 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-19T11:30:00.575 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-19T11:30:00.575 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:00.575 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.575 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:00.576 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6803", 2024-03-19T11:30:00.576 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2942163952 2024-03-19T11:30:00.576 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.576 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.576 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:00.576 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6805", 2024-03-19T11:30:00.576 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2942163952 2024-03-19T11:30:00.576 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:00.576 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:00.576 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.576 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-19T11:30:00.576 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:00.576 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.576 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:00.576 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6806", 2024-03-19T11:30:00.577 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2942163952 2024-03-19T11:30:00.577 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.577 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.577 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:00.577 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6809", 2024-03-19T11:30:00.577 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2942163952 2024-03-19T11:30:00.577 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:00.577 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:00.577 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.577 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-19T11:30:00.577 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:00.577 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.577 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:00.577 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6819", 2024-03-19T11:30:00.578 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2942163952 2024-03-19T11:30:00.578 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.578 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.578 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:00.578 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6820", 2024-03-19T11:30:00.578 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2942163952 2024-03-19T11:30:00.578 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:00.578 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:00.578 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.578 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-19T11:30:00.578 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:00.578 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.578 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:00.579 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6812", 2024-03-19T11:30:00.579 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2942163952 2024-03-19T11:30:00.579 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.579 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.579 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:00.579 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6816", 2024-03-19T11:30:00.579 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2942163952 2024-03-19T11:30:00.579 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:00.579 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:00.579 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.579 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6805/2942163952", 2024-03-19T11:30:00.579 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6809/2942163952", 2024-03-19T11:30:00.579 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6820/2942163952", 2024-03-19T11:30:00.580 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6816/2942163952", 2024-03-19T11:30:00.580 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-19T11:30:00.580 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-19T11:30:00.580 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-19T11:30:00.580 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:00.580 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.580 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.580 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 4, 2024-03-19T11:30:00.580 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "6746723d-f59c-40ba-857e-d8c954d2675f", 2024-03-19T11:30:00.580 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-19T11:30:00.580 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-19T11:30:00.580 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-19T11:30:00.580 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-19T11:30:00.580 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-19T11:30:00.581 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-19T11:30:00.581 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 13, 2024-03-19T11:30:00.581 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 21, 2024-03-19T11:30:00.581 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-19T11:30:00.581 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-19T11:30:00.581 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-19T11:30:00.581 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:00.581 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.581 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:00.581 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6808", 2024-03-19T11:30:00.581 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1353919713 2024-03-19T11:30:00.581 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.581 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.581 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:00.581 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6810", 2024-03-19T11:30:00.582 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1353919713 2024-03-19T11:30:00.582 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:00.582 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:00.582 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.582 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-19T11:30:00.582 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:00.582 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.582 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:00.582 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6812", 2024-03-19T11:30:00.582 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1353919713 2024-03-19T11:30:00.582 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.582 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.582 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:00.582 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6814", 2024-03-19T11:30:00.583 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1353919713 2024-03-19T11:30:00.583 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:00.583 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:00.583 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.583 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-19T11:30:00.583 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:00.583 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.583 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:00.583 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6819", 2024-03-19T11:30:00.583 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1353919713 2024-03-19T11:30:00.583 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.583 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.583 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:00.583 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6821", 2024-03-19T11:30:00.584 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1353919713 2024-03-19T11:30:00.584 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:00.584 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:00.584 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.584 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-19T11:30:00.584 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:00.584 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.584 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:00.584 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6816", 2024-03-19T11:30:00.584 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1353919713 2024-03-19T11:30:00.584 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.584 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.584 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:00.584 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6818", 2024-03-19T11:30:00.585 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1353919713 2024-03-19T11:30:00.585 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:00.585 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:00.585 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.585 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.172:6810/1353919713", 2024-03-19T11:30:00.585 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.172:6814/1353919713", 2024-03-19T11:30:00.585 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.172:6821/1353919713", 2024-03-19T11:30:00.585 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.172:6818/1353919713", 2024-03-19T11:30:00.585 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-19T11:30:00.585 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-19T11:30:00.585 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-19T11:30:00.585 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:00.585 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.585 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.586 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 5, 2024-03-19T11:30:00.586 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "f59afdf8-2277-4583-af68-78136f1c5d62", 2024-03-19T11:30:00.586 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-19T11:30:00.586 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-19T11:30:00.586 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-19T11:30:00.586 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-19T11:30:00.586 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-19T11:30:00.586 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-19T11:30:00.586 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 14, 2024-03-19T11:30:00.586 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 21, 2024-03-19T11:30:00.586 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-19T11:30:00.586 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-19T11:30:00.586 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-19T11:30:00.586 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:00.587 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.587 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:00.587 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6809", 2024-03-19T11:30:00.587 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2733060387 2024-03-19T11:30:00.587 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.587 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.587 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:00.587 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6811", 2024-03-19T11:30:00.587 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2733060387 2024-03-19T11:30:00.587 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:00.587 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:00.587 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.587 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-19T11:30:00.587 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:00.588 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.588 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:00.588 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6813", 2024-03-19T11:30:00.588 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2733060387 2024-03-19T11:30:00.588 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.588 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.588 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:00.588 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6815", 2024-03-19T11:30:00.588 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2733060387 2024-03-19T11:30:00.588 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:00.588 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:00.588 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.588 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-19T11:30:00.588 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:00.588 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.589 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:00.589 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6822", 2024-03-19T11:30:00.589 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2733060387 2024-03-19T11:30:00.589 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.589 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.589 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:00.589 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6823", 2024-03-19T11:30:00.589 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2733060387 2024-03-19T11:30:00.589 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:00.589 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:00.589 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.589 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-19T11:30:00.589 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:00.589 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.590 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:00.590 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6817", 2024-03-19T11:30:00.590 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2733060387 2024-03-19T11:30:00.590 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.590 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.590 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:00.590 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6820", 2024-03-19T11:30:00.590 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2733060387 2024-03-19T11:30:00.590 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:00.590 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:00.590 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.590 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.172:6811/2733060387", 2024-03-19T11:30:00.590 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.172:6815/2733060387", 2024-03-19T11:30:00.590 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.172:6823/2733060387", 2024-03-19T11:30:00.591 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.172:6820/2733060387", 2024-03-19T11:30:00.591 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-19T11:30:00.591 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-19T11:30:00.591 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-19T11:30:00.591 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:00.591 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.591 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.591 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 6, 2024-03-19T11:30:00.591 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "f91d8c47-4e10-40c6-a02d-66d6b72a657b", 2024-03-19T11:30:00.591 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-19T11:30:00.591 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-19T11:30:00.591 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-19T11:30:00.591 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-19T11:30:00.591 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-19T11:30:00.591 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-19T11:30:00.592 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 13, 2024-03-19T11:30:00.592 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 21, 2024-03-19T11:30:00.592 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-19T11:30:00.592 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-19T11:30:00.592 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-19T11:30:00.592 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:00.592 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.592 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:00.592 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6800", 2024-03-19T11:30:00.592 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 337743699 2024-03-19T11:30:00.592 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.592 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.592 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:00.592 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6801", 2024-03-19T11:30:00.593 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 337743699 2024-03-19T11:30:00.593 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:00.593 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:00.593 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.593 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-19T11:30:00.593 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:00.593 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.593 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:00.593 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6802", 2024-03-19T11:30:00.593 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 337743699 2024-03-19T11:30:00.593 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.593 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.593 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:00.593 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6803", 2024-03-19T11:30:00.593 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 337743699 2024-03-19T11:30:00.594 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:00.594 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:00.594 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.594 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-19T11:30:00.594 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:00.594 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.594 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:00.594 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6806", 2024-03-19T11:30:00.594 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 337743699 2024-03-19T11:30:00.594 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.594 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.594 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:00.594 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6807", 2024-03-19T11:30:00.595 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 337743699 2024-03-19T11:30:00.595 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:00.595 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:00.595 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.595 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-19T11:30:00.595 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:00.595 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.595 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:00.595 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6804", 2024-03-19T11:30:00.595 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 337743699 2024-03-19T11:30:00.595 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.595 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.595 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:00.595 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6805", 2024-03-19T11:30:00.595 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 337743699 2024-03-19T11:30:00.596 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:00.596 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:00.596 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.596 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.172:6801/337743699", 2024-03-19T11:30:00.596 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.172:6803/337743699", 2024-03-19T11:30:00.596 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.172:6807/337743699", 2024-03-19T11:30:00.596 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.172:6805/337743699", 2024-03-19T11:30:00.596 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-19T11:30:00.596 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-19T11:30:00.596 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-19T11:30:00.596 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:00.596 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.596 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.596 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 7, 2024-03-19T11:30:00.597 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "46fc052a-5708-42ff-9d5a-bed7a0fc457a", 2024-03-19T11:30:00.597 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-19T11:30:00.597 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-19T11:30:00.597 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-19T11:30:00.597 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-19T11:30:00.597 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-19T11:30:00.597 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-19T11:30:00.597 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 14, 2024-03-19T11:30:00.597 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 16, 2024-03-19T11:30:00.597 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-19T11:30:00.597 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-19T11:30:00.597 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-19T11:30:00.597 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:00.597 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.597 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:00.598 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6824", 2024-03-19T11:30:00.598 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 4145361100 2024-03-19T11:30:00.598 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.598 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.598 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:00.598 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6825", 2024-03-19T11:30:00.598 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 4145361100 2024-03-19T11:30:00.598 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:00.598 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:00.598 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.598 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-19T11:30:00.598 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:00.598 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.598 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:00.599 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6826", 2024-03-19T11:30:00.599 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 4145361100 2024-03-19T11:30:00.599 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.599 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.599 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:00.599 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6827", 2024-03-19T11:30:00.599 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 4145361100 2024-03-19T11:30:00.599 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:00.599 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:00.599 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.599 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-19T11:30:00.599 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:00.599 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.599 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:00.600 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6830", 2024-03-19T11:30:00.600 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 4145361100 2024-03-19T11:30:00.600 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.600 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.600 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:00.600 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6831", 2024-03-19T11:30:00.600 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 4145361100 2024-03-19T11:30:00.600 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:00.600 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:00.600 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.600 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-19T11:30:00.600 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:00.600 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.600 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:00.600 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6828", 2024-03-19T11:30:00.601 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 4145361100 2024-03-19T11:30:00.601 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.601 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.601 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:00.601 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6829", 2024-03-19T11:30:00.601 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 4145361100 2024-03-19T11:30:00.601 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:00.601 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:00.601 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.601 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.172:6825/4145361100", 2024-03-19T11:30:00.601 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.172:6827/4145361100", 2024-03-19T11:30:00.601 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.172:6831/4145361100", 2024-03-19T11:30:00.601 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.172:6829/4145361100", 2024-03-19T11:30:00.602 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-19T11:30:00.602 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-19T11:30:00.602 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-19T11:30:00.602 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:00.602 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:00.602 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-19T11:30:00.602 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_xinfo": [ 2024-03-19T11:30:00.602 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.602 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 0, 2024-03-19T11:30:00.602 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-19T11:30:00.602 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-19T11:30:00.602 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-19T11:30:00.602 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-19T11:30:00.602 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-19T11:30:00.603 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-19T11:29:41.144572+0000", 2024-03-19T11:30:00.603 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-19T11:30:00.603 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.603 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.603 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 1, 2024-03-19T11:30:00.603 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-19T11:30:00.603 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-19T11:30:00.603 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-19T11:30:00.603 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-19T11:30:00.603 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-19T11:30:00.603 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-19T11:29:41.282904+0000", 2024-03-19T11:30:00.603 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-19T11:30:00.603 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.603 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.603 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 2, 2024-03-19T11:30:00.604 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-19T11:30:00.604 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-19T11:30:00.604 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-19T11:30:00.604 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-19T11:30:00.604 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-19T11:30:00.604 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-19T11:29:41.176062+0000", 2024-03-19T11:30:00.604 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-19T11:30:00.604 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.604 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.604 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 3, 2024-03-19T11:30:00.604 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-19T11:30:00.604 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-19T11:30:00.604 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-19T11:30:00.604 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-19T11:30:00.605 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-19T11:30:00.605 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-19T11:29:41.269303+0000", 2024-03-19T11:30:00.605 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-19T11:30:00.605 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.605 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.605 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 4, 2024-03-19T11:30:00.605 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-19T11:30:00.605 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-19T11:30:00.605 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-19T11:30:00.605 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-19T11:30:00.605 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-19T11:30:00.605 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-19T11:29:40.978689+0000", 2024-03-19T11:30:00.605 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-19T11:30:00.605 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.605 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.606 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 5, 2024-03-19T11:30:00.606 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-19T11:30:00.606 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-19T11:30:00.606 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-19T11:30:00.606 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-19T11:30:00.606 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-19T11:30:00.606 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-19T11:29:41.198288+0000", 2024-03-19T11:30:00.606 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-19T11:30:00.606 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.606 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.606 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 6, 2024-03-19T11:30:00.606 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-19T11:30:00.606 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-19T11:30:00.606 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-19T11:30:00.607 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-19T11:30:00.607 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-19T11:30:00.607 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-19T11:29:41.083084+0000", 2024-03-19T11:30:00.607 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-19T11:30:00.607 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.607 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:00.607 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 7, 2024-03-19T11:30:00.607 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-19T11:30:00.607 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-19T11:30:00.607 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-19T11:30:00.607 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-19T11:30:00.607 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-19T11:30:00.607 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-19T11:29:41.265929+0000", 2024-03-19T11:30:00.607 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-19T11:30:00.608 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:00.608 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-19T11:30:00.608 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_upmap": [], 2024-03-19T11:30:00.608 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_upmap_items": [], 2024-03-19T11:30:00.608 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_upmap_primaries": [], 2024-03-19T11:30:00.608 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_temp": [], 2024-03-19T11:30:00.608 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_temp": [], 2024-03-19T11:30:00.608 INFO:teuthology.orchestra.run.smithi038.stdout: "blocklist": {}, 2024-03-19T11:30:00.608 INFO:teuthology.orchestra.run.smithi038.stdout: "range_blocklist": {}, 2024-03-19T11:30:00.608 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profiles": { 2024-03-19T11:30:00.608 INFO:teuthology.orchestra.run.smithi038.stdout: "default": { 2024-03-19T11:30:00.608 INFO:teuthology.orchestra.run.smithi038.stdout: "crush-failure-domain": "osd", 2024-03-19T11:30:00.608 INFO:teuthology.orchestra.run.smithi038.stdout: "k": "2", 2024-03-19T11:30:00.608 INFO:teuthology.orchestra.run.smithi038.stdout: "m": "1", 2024-03-19T11:30:00.608 INFO:teuthology.orchestra.run.smithi038.stdout: "plugin": "jerasure", 2024-03-19T11:30:00.609 INFO:teuthology.orchestra.run.smithi038.stdout: "technique": "reed_sol_van" 2024-03-19T11:30:00.609 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:00.609 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:00.609 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps_queue": [], 2024-03-19T11:30:00.609 INFO:teuthology.orchestra.run.smithi038.stdout: "new_removed_snaps": [], 2024-03-19T11:30:00.609 INFO:teuthology.orchestra.run.smithi038.stdout: "new_purged_snaps": [], 2024-03-19T11:30:00.609 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_node_flags": {}, 2024-03-19T11:30:00.609 INFO:teuthology.orchestra.run.smithi038.stdout: "device_class_flags": {}, 2024-03-19T11:30:00.609 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode": { 2024-03-19T11:30:00.609 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode_enabled": false, 2024-03-19T11:30:00.609 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_bucket_count": 0, 2024-03-19T11:30:00.609 INFO:teuthology.orchestra.run.smithi038.stdout: "degraded_stretch_mode": 0, 2024-03-19T11:30:00.609 INFO:teuthology.orchestra.run.smithi038.stdout: "recovering_stretch_mode": 0, 2024-03-19T11:30:00.609 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode_bucket": 0 2024-03-19T11:30:00.610 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:00.610 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:00.612 INFO:tasks.thrashosds.thrasher:j is {'epoch': 23, 'fsid': '246bcf68-c431-4aa7-b787-5e9b2d9519d6', 'created': '2024-03-19T11:29:33.327073+0000', 'modified': '2024-03-19T11:29:59.551091+0000', 'last_up_change': '2024-03-19T11:29:43.166618+0000', 'last_in_change': '2024-03-19T11:29:35.632327+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 8, 'require_min_compat_client': 'reef', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T11:29:42.219654+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-03-19T11:29:44.428406+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'low_tier', 'create_time': '2024-03-19T11:29:55.608699+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 0.75, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'unique_pool_0', 'create_time': '2024-03-19T11:29:56.945423+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '3bb2d2be-5253-4da0-acf3-85a114dc3801', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 21, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6808', 'nonce': 3148574891}, {'type': 'v1', 'addr': '172.21.15.38:6811', 'nonce': 3148574891}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6813', 'nonce': 3148574891}, {'type': 'v1', 'addr': '172.21.15.38:6815', 'nonce': 3148574891}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6822', 'nonce': 3148574891}, {'type': 'v1', 'addr': '172.21.15.38:6823', 'nonce': 3148574891}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6818', 'nonce': 3148574891}, {'type': 'v1', 'addr': '172.21.15.38:6821', 'nonce': 3148574891}]}, 'public_addr': '172.21.15.38:6811/3148574891', 'cluster_addr': '172.21.15.38:6815/3148574891', 'heartbeat_back_addr': '172.21.15.38:6823/3148574891', 'heartbeat_front_addr': '172.21.15.38:6821/3148574891', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'e1c926b2-e0be-4ccc-b8a5-d80d0b584ead', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 21, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6824', 'nonce': 675027622}, {'type': 'v1', 'addr': '172.21.15.38:6825', 'nonce': 675027622}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6826', 'nonce': 675027622}, {'type': 'v1', 'addr': '172.21.15.38:6827', 'nonce': 675027622}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6830', 'nonce': 675027622}, {'type': 'v1', 'addr': '172.21.15.38:6831', 'nonce': 675027622}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6828', 'nonce': 675027622}, {'type': 'v1', 'addr': '172.21.15.38:6829', 'nonce': 675027622}]}, 'public_addr': '172.21.15.38:6825/675027622', 'cluster_addr': '172.21.15.38:6827/675027622', 'heartbeat_back_addr': '172.21.15.38:6831/675027622', 'heartbeat_front_addr': '172.21.15.38:6829/675027622', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '61262219-4f8a-4733-af9d-b5208a17c0ea', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 16, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6800', 'nonce': 58279616}, {'type': 'v1', 'addr': '172.21.15.38:6801', 'nonce': 58279616}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6802', 'nonce': 58279616}, {'type': 'v1', 'addr': '172.21.15.38:6804', 'nonce': 58279616}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6814', 'nonce': 58279616}, {'type': 'v1', 'addr': '172.21.15.38:6817', 'nonce': 58279616}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6807', 'nonce': 58279616}, {'type': 'v1', 'addr': '172.21.15.38:6810', 'nonce': 58279616}]}, 'public_addr': '172.21.15.38:6801/58279616', 'cluster_addr': '172.21.15.38:6804/58279616', 'heartbeat_back_addr': '172.21.15.38:6817/58279616', 'heartbeat_front_addr': '172.21.15.38:6810/58279616', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'a31f8d1b-4e38-4ca3-b251-0877002d7235', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 21, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6803', 'nonce': 2942163952}, {'type': 'v1', 'addr': '172.21.15.38:6805', 'nonce': 2942163952}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6806', 'nonce': 2942163952}, {'type': 'v1', 'addr': '172.21.15.38:6809', 'nonce': 2942163952}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6819', 'nonce': 2942163952}, {'type': 'v1', 'addr': '172.21.15.38:6820', 'nonce': 2942163952}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6812', 'nonce': 2942163952}, {'type': 'v1', 'addr': '172.21.15.38:6816', 'nonce': 2942163952}]}, 'public_addr': '172.21.15.38:6805/2942163952', 'cluster_addr': '172.21.15.38:6809/2942163952', 'heartbeat_back_addr': '172.21.15.38:6820/2942163952', 'heartbeat_front_addr': '172.21.15.38:6816/2942163952', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6746723d-f59c-40ba-857e-d8c954d2675f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 21, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6808', 'nonce': 1353919713}, {'type': 'v1', 'addr': '172.21.15.172:6810', 'nonce': 1353919713}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6812', 'nonce': 1353919713}, {'type': 'v1', 'addr': '172.21.15.172:6814', 'nonce': 1353919713}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6819', 'nonce': 1353919713}, {'type': 'v1', 'addr': '172.21.15.172:6821', 'nonce': 1353919713}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6816', 'nonce': 1353919713}, {'type': 'v1', 'addr': '172.21.15.172:6818', 'nonce': 1353919713}]}, 'public_addr': '172.21.15.172:6810/1353919713', 'cluster_addr': '172.21.15.172:6814/1353919713', 'heartbeat_back_addr': '172.21.15.172:6821/1353919713', 'heartbeat_front_addr': '172.21.15.172:6818/1353919713', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'f59afdf8-2277-4583-af68-78136f1c5d62', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 21, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6809', 'nonce': 2733060387}, {'type': 'v1', 'addr': '172.21.15.172:6811', 'nonce': 2733060387}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6813', 'nonce': 2733060387}, {'type': 'v1', 'addr': '172.21.15.172:6815', 'nonce': 2733060387}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6822', 'nonce': 2733060387}, {'type': 'v1', 'addr': '172.21.15.172:6823', 'nonce': 2733060387}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6817', 'nonce': 2733060387}, {'type': 'v1', 'addr': '172.21.15.172:6820', 'nonce': 2733060387}]}, 'public_addr': '172.21.15.172:6811/2733060387', 'cluster_addr': '172.21.15.172:6815/2733060387', 'heartbeat_back_addr': '172.21.15.172:6823/2733060387', 'heartbeat_front_addr': '172.21.15.172:6820/2733060387', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'f91d8c47-4e10-40c6-a02d-66d6b72a657b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 21, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6800', 'nonce': 337743699}, {'type': 'v1', 'addr': '172.21.15.172:6801', 'nonce': 337743699}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6802', 'nonce': 337743699}, {'type': 'v1', 'addr': '172.21.15.172:6803', 'nonce': 337743699}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6806', 'nonce': 337743699}, {'type': 'v1', 'addr': '172.21.15.172:6807', 'nonce': 337743699}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6804', 'nonce': 337743699}, {'type': 'v1', 'addr': '172.21.15.172:6805', 'nonce': 337743699}]}, 'public_addr': '172.21.15.172:6801/337743699', 'cluster_addr': '172.21.15.172:6803/337743699', 'heartbeat_back_addr': '172.21.15.172:6807/337743699', 'heartbeat_front_addr': '172.21.15.172:6805/337743699', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '46fc052a-5708-42ff-9d5a-bed7a0fc457a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 16, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6824', 'nonce': 4145361100}, {'type': 'v1', 'addr': '172.21.15.172:6825', 'nonce': 4145361100}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6826', 'nonce': 4145361100}, {'type': 'v1', 'addr': '172.21.15.172:6827', 'nonce': 4145361100}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6830', 'nonce': 4145361100}, {'type': 'v1', 'addr': '172.21.15.172:6831', 'nonce': 4145361100}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6828', 'nonce': 4145361100}, {'type': 'v1', 'addr': '172.21.15.172:6829', 'nonce': 4145361100}]}, 'public_addr': '172.21.15.172:6825/4145361100', 'cluster_addr': '172.21.15.172:6827/4145361100', 'heartbeat_back_addr': '172.21.15.172:6831/4145361100', 'heartbeat_front_addr': '172.21.15.172:6829/4145361100', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-19T11:29:41.144572+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-19T11:29:41.282904+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-19T11:29:41.176062+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-19T11:29:41.269303+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-19T11:29:40.978689+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-19T11:29:41.198288+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-19T11:29:41.083084+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-19T11:29:41.265929+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-19T11:30:00.612 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T11:30:00.613 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:00.624+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:00.712 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:00.720+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:00.740 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:00.740 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:00.755 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-19T11:30:00.814 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:00.824+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:00.914 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:00.915 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:00.915 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:00.924+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:00.929 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-19T11:30:00.954 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:30:00.955 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-03-19T11:30:00.974 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-03-19T11:30:00.578685+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":52,"acting":52,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":26,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":221812,"kb_used_data":1472,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754752908,"statfs":{"total":773094113280,"available":772866977792,"internally_reserved":0,"allocated":1507328,"data_stored":1611163,"data_compressed":12128,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":12751,"internal_metadata":225496625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.047201"},"pg_stats":[{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-19T11:29:58.556487+0000","last_change":"2024-03-19T11:29:58.551869+0000","last_active":"2024-03-19T11:29:58.556487+0000","last_peered":"2024-03-19T11:29:58.556487+0000","last_clean":"2024-03-19T11:29:58.556487+0000","last_became_active":"2024-03-19T11:29:58.551505+0000","last_became_peered":"2024-03-19T11:29:58.551505+0000","last_unstale":"2024-03-19T11:29:58.556487+0000","last_undegraded":"2024-03-19T11:29:58.556487+0000","last_fullsized":"2024-03-19T11:29:58.556487+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:12.912735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1,"reported_epoch":21,"state":"creating+peering","last_fresh":"2024-03-19T11:29:57.544156+0000","last_change":"2024-03-19T11:29:57.544156+0000","last_active":"2024-03-19T11:29:57.528796+0000","last_peered":"2024-03-19T11:29:57.528796+0000","last_clean":"2024-03-19T11:29:57.528796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T11:29:57.544156+0000","last_undegraded":"2024-03-19T11:29:57.544156+0000","last_fullsized":"2024-03-19T11:29:57.544156+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1,"reported_epoch":21,"state":"creating+peering","last_fresh":"2024-03-19T11:29:57.544496+0000","last_change":"2024-03-19T11:29:57.544496+0000","last_active":"2024-03-19T11:29:57.528796+0000","last_peered":"2024-03-19T11:29:57.528796+0000","last_clean":"2024-03-19T11:29:57.528796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T11:29:57.544496+0000","last_undegraded":"2024-03-19T11:29:57.544496+0000","last_fullsized":"2024-03-19T11:29:57.544496+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1,"reported_epoch":21,"state":"creating+peering","last_fresh":"2024-03-19T11:29:57.544964+0000","last_change":"2024-03-19T11:29:57.544964+0000","last_active":"2024-03-19T11:29:57.528796+0000","last_peered":"2024-03-19T11:29:57.528796+0000","last_clean":"2024-03-19T11:29:57.528796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T11:29:57.544964+0000","last_undegraded":"2024-03-19T11:29:57.544964+0000","last_fullsized":"2024-03-19T11:29:57.544964+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-19T11:29:57.528796+0000","last_change":"2024-03-19T11:29:57.528796+0000","last_active":"2024-03-19T11:29:57.528796+0000","last_peered":"2024-03-19T11:29:57.528796+0000","last_clean":"2024-03-19T11:29:57.528796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T11:29:57.528796+0000","last_undegraded":"2024-03-19T11:29:57.528796+0000","last_fullsized":"2024-03-19T11:29:57.528796+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1,"reported_epoch":21,"state":"creating+peering","last_fresh":"2024-03-19T11:29:57.549711+0000","last_change":"2024-03-19T11:29:57.549711+0000","last_active":"2024-03-19T11:29:57.528796+0000","last_peered":"2024-03-19T11:29:57.528796+0000","last_clean":"2024-03-19T11:29:57.528796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T11:29:57.549711+0000","last_undegraded":"2024-03-19T11:29:57.549711+0000","last_fullsized":"2024-03-19T11:29:57.549711+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1,"reported_epoch":21,"state":"creating+peering","last_fresh":"2024-03-19T11:29:57.545738+0000","last_change":"2024-03-19T11:29:57.545738+0000","last_active":"2024-03-19T11:29:57.528796+0000","last_peered":"2024-03-19T11:29:57.528796+0000","last_clean":"2024-03-19T11:29:57.528796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T11:29:57.545738+0000","last_undegraded":"2024-03-19T11:29:57.545738+0000","last_fullsized":"2024-03-19T11:29:57.545738+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-19T11:29:58.643310+0000","last_change":"2024-03-19T11:29:58.642222+0000","last_active":"2024-03-19T11:29:58.643310+0000","last_peered":"2024-03-19T11:29:58.643310+0000","last_clean":"2024-03-19T11:29:58.643310+0000","last_became_active":"2024-03-19T11:29:58.641526+0000","last_became_peered":"2024-03-19T11:29:58.641526+0000","last_unstale":"2024-03-19T11:29:58.643310+0000","last_undegraded":"2024-03-19T11:29:58.643310+0000","last_fullsized":"2024-03-19T11:29:58.643310+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:13.287011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1,"reported_epoch":21,"state":"creating+peering","last_fresh":"2024-03-19T11:29:57.548636+0000","last_change":"2024-03-19T11:29:57.548636+0000","last_active":"2024-03-19T11:29:57.528796+0000","last_peered":"2024-03-19T11:29:57.528796+0000","last_clean":"2024-03-19T11:29:57.528796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T11:29:57.548636+0000","last_undegraded":"2024-03-19T11:29:57.548636+0000","last_fullsized":"2024-03-19T11:29:57.548636+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":174,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-19T11:29:58.641020+0000","last_change":"2024-03-19T11:29:45.008018+0000","last_active":"2024-03-19T11:29:58.641020+0000","last_peered":"2024-03-19T11:29:58.641020+0000","last_clean":"2024-03-19T11:29:58.641020+0000","last_became_active":"2024-03-19T11:29:45.005482+0000","last_became_peered":"2024-03-19T11:29:45.005482+0000","last_unstale":"2024-03-19T11:29:58.641020+0000","last_undegraded":"2024-03-19T11:29:58.641020+0000","last_fullsized":"2024-03-19T11:29:58.641020+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:43.166618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:43.166618+0000","last_clean_scrub_stamp":"2024-03-19T11:29:43.166618+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:02.385897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-19T11:29:58.548290+0000","last_change":"2024-03-19T11:29:46.322491+0000","last_active":"2024-03-19T11:29:58.548290+0000","last_peered":"2024-03-19T11:29:58.548290+0000","last_clean":"2024-03-19T11:29:58.548290+0000","last_became_active":"2024-03-19T11:29:46.321760+0000","last_became_peered":"2024-03-19T11:29:46.321760+0000","last_unstale":"2024-03-19T11:29:58.548290+0000","last_undegraded":"2024-03-19T11:29:58.548290+0000","last_fullsized":"2024-03-19T11:29:58.548290+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:04.561915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-19T11:29:58.543220+0000","last_change":"2024-03-19T11:29:57.549277+0000","last_active":"2024-03-19T11:29:58.543220+0000","last_peered":"2024-03-19T11:29:58.543220+0000","last_clean":"2024-03-19T11:29:58.543220+0000","last_became_active":"2024-03-19T11:29:57.548946+0000","last_became_peered":"2024-03-19T11:29:57.548946+0000","last_unstale":"2024-03-19T11:29:58.543220+0000","last_undegraded":"2024-03-19T11:29:58.543220+0000","last_fullsized":"2024-03-19T11:29:58.543220+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:08.380405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-19T11:29:57.528796+0000","last_change":"2024-03-19T11:29:57.528796+0000","last_active":"2024-03-19T11:29:57.528796+0000","last_peered":"2024-03-19T11:29:57.528796+0000","last_clean":"2024-03-19T11:29:57.528796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T11:29:57.528796+0000","last_undegraded":"2024-03-19T11:29:57.528796+0000","last_fullsized":"2024-03-19T11:29:57.528796+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-19T11:29:58.640766+0000","last_change":"2024-03-19T11:29:46.436253+0000","last_active":"2024-03-19T11:29:58.640766+0000","last_peered":"2024-03-19T11:29:58.640766+0000","last_clean":"2024-03-19T11:29:58.640766+0000","last_became_active":"2024-03-19T11:29:46.435783+0000","last_became_peered":"2024-03-19T11:29:46.435783+0000","last_unstale":"2024-03-19T11:29:58.640766+0000","last_undegraded":"2024-03-19T11:29:58.640766+0000","last_fullsized":"2024-03-19T11:29:58.640766+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:59.997466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1,"reported_epoch":20,"state":"creating+peering","last_fresh":"2024-03-19T11:29:56.627171+0000","last_change":"2024-03-19T11:29:56.627171+0000","last_active":"2024-03-19T11:29:56.503774+0000","last_peered":"2024-03-19T11:29:56.503774+0000","last_clean":"2024-03-19T11:29:56.503774+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T11:29:56.627171+0000","last_undegraded":"2024-03-19T11:29:56.627171+0000","last_fullsized":"2024-03-19T11:29:56.627171+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-19T11:29:58.559713+0000","last_change":"2024-03-19T11:29:58.556838+0000","last_active":"2024-03-19T11:29:58.559713+0000","last_peered":"2024-03-19T11:29:58.559713+0000","last_clean":"2024-03-19T11:29:58.559713+0000","last_became_active":"2024-03-19T11:29:58.555355+0000","last_became_peered":"2024-03-19T11:29:58.555355+0000","last_unstale":"2024-03-19T11:29:58.559713+0000","last_undegraded":"2024-03-19T11:29:58.559713+0000","last_fullsized":"2024-03-19T11:29:58.559713+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:22.021410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-19T11:29:58.545153+0000","last_change":"2024-03-19T11:29:46.319811+0000","last_active":"2024-03-19T11:29:58.545153+0000","last_peered":"2024-03-19T11:29:58.545153+0000","last_clean":"2024-03-19T11:29:58.545153+0000","last_became_active":"2024-03-19T11:29:46.319334+0000","last_became_peered":"2024-03-19T11:29:46.319334+0000","last_unstale":"2024-03-19T11:29:58.545153+0000","last_undegraded":"2024-03-19T11:29:58.545153+0000","last_fullsized":"2024-03-19T11:29:58.545153+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:04.320463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-19T11:29:58.546199+0000","last_change":"2024-03-19T11:29:57.550320+0000","last_active":"2024-03-19T11:29:58.546199+0000","last_peered":"2024-03-19T11:29:58.546199+0000","last_clean":"2024-03-19T11:29:58.546199+0000","last_became_active":"2024-03-19T11:29:57.549928+0000","last_became_peered":"2024-03-19T11:29:57.549928+0000","last_unstale":"2024-03-19T11:29:58.546199+0000","last_undegraded":"2024-03-19T11:29:58.546199+0000","last_fullsized":"2024-03-19T11:29:58.546199+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:01.987302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-19T11:29:58.559669+0000","last_change":"2024-03-19T11:29:58.556426+0000","last_active":"2024-03-19T11:29:58.559669+0000","last_peered":"2024-03-19T11:29:58.559669+0000","last_clean":"2024-03-19T11:29:58.559669+0000","last_became_active":"2024-03-19T11:29:58.555204+0000","last_became_peered":"2024-03-19T11:29:58.555204+0000","last_unstale":"2024-03-19T11:29:58.559669+0000","last_undegraded":"2024-03-19T11:29:58.559669+0000","last_fullsized":"2024-03-19T11:29:58.559669+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:14.476183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-19T11:29:58.548395+0000","last_change":"2024-03-19T11:29:46.321255+0000","last_active":"2024-03-19T11:29:58.548395+0000","last_peered":"2024-03-19T11:29:58.548395+0000","last_clean":"2024-03-19T11:29:58.548395+0000","last_became_active":"2024-03-19T11:29:46.320679+0000","last_became_peered":"2024-03-19T11:29:46.320679+0000","last_unstale":"2024-03-19T11:29:58.548395+0000","last_undegraded":"2024-03-19T11:29:58.548395+0000","last_fullsized":"2024-03-19T11:29:58.548395+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:14.793563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T11:29:57.560348+0000","last_change":"2024-03-19T11:29:57.554995+0000","last_active":"2024-03-19T11:29:57.560348+0000","last_peered":"2024-03-19T11:29:57.560348+0000","last_clean":"2024-03-19T11:29:57.560348+0000","last_became_active":"2024-03-19T11:29:57.554481+0000","last_became_peered":"2024-03-19T11:29:57.554481+0000","last_unstale":"2024-03-19T11:29:57.560348+0000","last_undegraded":"2024-03-19T11:29:57.560348+0000","last_fullsized":"2024-03-19T11:29:57.560348+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:58.244883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-19T11:29:58.559579+0000","last_change":"2024-03-19T11:29:58.555412+0000","last_active":"2024-03-19T11:29:58.559579+0000","last_peered":"2024-03-19T11:29:58.559579+0000","last_clean":"2024-03-19T11:29:58.559579+0000","last_became_active":"2024-03-19T11:29:58.555026+0000","last_became_peered":"2024-03-19T11:29:58.555026+0000","last_unstale":"2024-03-19T11:29:58.559579+0000","last_undegraded":"2024-03-19T11:29:58.559579+0000","last_fullsized":"2024-03-19T11:29:58.559579+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:17.432215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-19T11:29:58.547101+0000","last_change":"2024-03-19T11:29:46.541233+0000","last_active":"2024-03-19T11:29:58.547101+0000","last_peered":"2024-03-19T11:29:58.547101+0000","last_clean":"2024-03-19T11:29:58.547101+0000","last_became_active":"2024-03-19T11:29:46.540939+0000","last_became_peered":"2024-03-19T11:29:46.540939+0000","last_unstale":"2024-03-19T11:29:58.547101+0000","last_undegraded":"2024-03-19T11:29:58.547101+0000","last_fullsized":"2024-03-19T11:29:58.547101+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:54.761098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-19T11:29:57.528796+0000","last_change":"2024-03-19T11:29:57.528796+0000","last_active":"2024-03-19T11:29:57.528796+0000","last_peered":"2024-03-19T11:29:57.528796+0000","last_clean":"2024-03-19T11:29:57.528796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T11:29:57.528796+0000","last_undegraded":"2024-03-19T11:29:57.528796+0000","last_fullsized":"2024-03-19T11:29:57.528796+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-19T11:29:58.640812+0000","last_change":"2024-03-19T11:29:46.542897+0000","last_active":"2024-03-19T11:29:58.640812+0000","last_peered":"2024-03-19T11:29:58.640812+0000","last_clean":"2024-03-19T11:29:58.640812+0000","last_became_active":"2024-03-19T11:29:46.542092+0000","last_became_peered":"2024-03-19T11:29:46.542092+0000","last_unstale":"2024-03-19T11:29:58.640812+0000","last_undegraded":"2024-03-19T11:29:58.640812+0000","last_fullsized":"2024-03-19T11:29:58.640812+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:46.318698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-19T11:29:58.644265+0000","last_change":"2024-03-19T11:29:58.643130+0000","last_active":"2024-03-19T11:29:58.644265+0000","last_peered":"2024-03-19T11:29:58.644265+0000","last_clean":"2024-03-19T11:29:58.644265+0000","last_became_active":"2024-03-19T11:29:58.642619+0000","last_became_peered":"2024-03-19T11:29:58.642619+0000","last_unstale":"2024-03-19T11:29:58.644265+0000","last_undegraded":"2024-03-19T11:29:58.644265+0000","last_fullsized":"2024-03-19T11:29:58.644265+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:06.477736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-03-19T11:29:58.546461+0000","last_change":"2024-03-19T11:29:46.423438+0000","last_active":"2024-03-19T11:29:58.546461+0000","last_peered":"2024-03-19T11:29:58.546461+0000","last_clean":"2024-03-19T11:29:58.546461+0000","last_became_active":"2024-03-19T11:29:46.423027+0000","last_became_peered":"2024-03-19T11:29:46.423027+0000","last_unstale":"2024-03-19T11:29:58.546461+0000","last_undegraded":"2024-03-19T11:29:58.546461+0000","last_fullsized":"2024-03-19T11:29:58.546461+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:55.305083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1,"reported_epoch":21,"state":"creating+peering","last_fresh":"2024-03-19T11:29:57.544539+0000","last_change":"2024-03-19T11:29:57.544539+0000","last_active":"2024-03-19T11:29:57.528796+0000","last_peered":"2024-03-19T11:29:57.528796+0000","last_clean":"2024-03-19T11:29:57.528796+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T11:29:57.544539+0000","last_undegraded":"2024-03-19T11:29:57.544539+0000","last_fullsized":"2024-03-19T11:29:57.544539+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-19T11:29:57.548216+0000","last_change":"2024-03-19T11:29:46.435120+0000","last_active":"2024-03-19T11:29:57.548216+0000","last_peered":"2024-03-19T11:29:57.548216+0000","last_clean":"2024-03-19T11:29:57.548216+0000","last_became_active":"2024-03-19T11:29:46.434588+0000","last_became_peered":"2024-03-19T11:29:46.434588+0000","last_unstale":"2024-03-19T11:29:57.548216+0000","last_undegraded":"2024-03-19T11:29:57.548216+0000","last_fullsized":"2024-03-19T11:29:57.548216+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:03.134473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":26,"acting":26,"num_store_stats":7},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":12128,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542151,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":603295,"data_compressed":6064,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542151,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":180224,"data_stored":69671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542151,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27180,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344660,"statfs":{"total":96636764160,"available":96608931840,"internally_reserved":0,"allocated":159744,"data_stored":67302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542150,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":66674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542150,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31344,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340496,"statfs":{"total":96636764160,"available":96604667904,"internally_reserved":0,"allocated":167936,"data_stored":68867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":200704,"data_stored":68717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27180,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344660,"statfs":{"total":96636764160,"available":96608931840,"internally_reserved":0,"allocated":163840,"data_stored":69614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":249856,"data_stored":597023,"data_compressed":6064,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6064,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6064,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T11:30:00.976 INFO:tasks.thrashosds.thrasher:Setting 2.7 to [0, 5, 6, 2] 2024-03-19T11:30:00.976 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.7', '0', '5', '6', '2'] 2024-03-19T11:30:00.976 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 2.7 0 5 6 2 2024-03-19T11:30:01.018 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:01.028+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:01.067 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:01.068 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:01.081 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-19T11:30:01.126 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:01.136+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:01.194 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:01.194 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:01.209 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-19T11:30:01.253 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:01.264+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:01.318 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:01.328+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:01.340 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:01.340 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:01.354 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-19T11:30:01.420 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:01.428+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:01.489 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:01.496+0000 7f70c4fdb640 -1 osd.0 23 handle_osd_map injecting map crc failure 2024-03-19T11:30:01.491 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:01.500+0000 7f70c4fdb640 -1 osd.0 23 handle_osd_map injecting map crc failure 2024-03-19T11:30:01.520 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:01.521 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:01.521 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:01.528+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:01.534 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-19T11:30:01.574 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-03-19T11:30:01.604 INFO:teuthology.orchestra.run.smithi038.stderr:set 2.7 pg_upmap_items mapping to [0->5,6->2] 2024-03-19T11:30:01.621 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:01.628+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:01.659 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:01.660 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:01.673 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-19T11:30:01.722 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:01.732+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:01.791 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:01.791 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:01.805 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-19T11:30:01.849 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:01.860+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:01.924 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:01.932+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:01.972 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:01.973 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:01.987 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-19T11:30:02.024 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:02.032+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:02.125 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:02.136+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:02.155 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:02.155 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:02.169 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-19T11:30:02.226 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:02.236+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:02.267 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-19T11:30:02.327 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:02.336+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:02.338 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:02.338 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:02.353 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-19T11:30:02.428 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:02.436+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:02.429 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:30:02.429 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:30:02.445 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:30:02.445 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:30:02.461 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:30:02.461 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:30:02.475 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:02.475 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:02.477 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:30:02.477 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:30:02.489 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-19T11:30:02.496 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:30:02.497 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:30:02.517 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:30:02.517 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:30:02.529 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:02.536+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:02.537 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:30:02.537 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:30:02.556 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:30:02.556 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:30:02.605 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-03-19T11:30:02.610 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:02.620+0000 7fa5e08db640 -1 osd.3 25 handle_osd_map injecting map crc failure 2024-03-19T11:30:02.615 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:02.615 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:30:02.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:30:02.615 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [] 2024-03-19T11:30:02.615 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:02.629 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-19T11:30:02.630 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:02.640+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:02.731 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:02.740+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:02.750 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:02.751 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:02.751 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:30:02.751 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:02.764 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-19T11:30:02.832 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:02.840+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:02.887 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:02.887 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:02.887 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:30:02.888 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:30:02.888 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:02.901 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-19T11:30:02.948 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:02.960+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:03.020 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:03.020 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:30:03.020 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:30:03.020 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:30:03.020 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.020 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:30:03.020 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:44.200920+0000", 2024-03-19T11:30:03.021 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.832075465999999, 2024-03-19T11:30:03.021 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.80881896499999995, 2024-03-19T11:30:03.021 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:03.021 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:03.021 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:03.021 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:03.021 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:03.021 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1 2024-03-19T11:30:03.021 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.021 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:03.021 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.021 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:03.021 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200920+0000", 2024-03-19T11:30:03.021 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:03.021 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.022 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.022 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:03.022 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200920+0000", 2024-03-19T11:30:03.022 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:03.022 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.022 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.022 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:03.022 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200923+0000", 2024-03-19T11:30:03.022 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6469999999999998e-06 2024-03-19T11:30:03.022 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.022 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.022 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:03.023 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200937+0000", 2024-03-19T11:30:03.023 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.399e-05 2024-03-19T11:30:03.023 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.023 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.023 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:03.023 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200960+0000", 2024-03-19T11:30:03.023 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2603e-05 2024-03-19T11:30:03.023 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.023 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.023 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:03.023 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200978+0000", 2024-03-19T11:30:03.023 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.791e-05 2024-03-19T11:30:03.023 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.023 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.023 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:03.024 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.201081+0000", 2024-03-19T11:30:03.024 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010344900000000001 2024-03-19T11:30:03.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.024 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:30:03.024 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.201096+0000", 2024-03-19T11:30:03.024 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4443e-05 2024-03-19T11:30:03.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.024 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:03.024 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.005978+0000", 2024-03-19T11:30:03.024 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.80488281500000003 2024-03-19T11:30:03.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.025 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:03.025 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.006267+0000", 2024-03-19T11:30:03.025 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000288688 2024-03-19T11:30:03.025 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.025 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.025 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:03.025 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.006534+0000", 2024-03-19T11:30:03.025 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026725799999999998 2024-03-19T11:30:03.025 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.025 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.025 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:03.025 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.008457+0000", 2024-03-19T11:30:03.025 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0019227820000000001 2024-03-19T11:30:03.025 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.025 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.026 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:03.026 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.009570+0000", 2024-03-19T11:30:03.026 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011131750000000001 2024-03-19T11:30:03.026 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.026 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.026 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:03.026 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.009670+0000", 2024-03-19T11:30:03.026 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9458999999999998e-05 2024-03-19T11:30:03.026 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.026 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.026 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:03.026 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.009739+0000", 2024-03-19T11:30:03.026 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9746000000000006e-05 2024-03-19T11:30:03.026 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:03.027 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:03.027 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:03.027 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.027 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.027 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:30:03.027 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.071728+0000", 2024-03-19T11:30:03.027 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 17.961268028999999, 2024-03-19T11:30:03.027 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013714690999999999, 2024-03-19T11:30:03.027 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:03.027 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:03.027 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:03.027 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:03.027 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:03.027 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 34 2024-03-19T11:30:03.027 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.028 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:03.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.028 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:03.028 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071728+0000", 2024-03-19T11:30:03.028 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:03.028 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.028 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:03.028 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071728+0000", 2024-03-19T11:30:03.029 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:03.029 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.029 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.029 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:03.029 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071729+0000", 2024-03-19T11:30:03.029 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.47e-07 2024-03-19T11:30:03.029 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.029 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.029 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:03.029 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071735+0000", 2024-03-19T11:30:03.029 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3799999999999999e-06 2024-03-19T11:30:03.029 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.029 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.029 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:03.029 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071748+0000", 2024-03-19T11:30:03.030 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2806000000000001e-05 2024-03-19T11:30:03.030 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.030 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.030 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:03.030 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071756+0000", 2024-03-19T11:30:03.030 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1240000000000005e-06 2024-03-19T11:30:03.030 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.030 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.030 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:03.030 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.077570+0000", 2024-03-19T11:30:03.030 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.005813342 2024-03-19T11:30:03.030 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.030 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.030 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:03.031 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.077793+0000", 2024-03-19T11:30:03.031 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022391500000000001 2024-03-19T11:30:03.031 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.031 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.031 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:03.031 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.078146+0000", 2024-03-19T11:30:03.031 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00035219 2024-03-19T11:30:03.031 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.031 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.031 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:03.031 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.083390+0000", 2024-03-19T11:30:03.031 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052445590000000002 2024-03-19T11:30:03.031 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.031 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.032 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:03.032 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085288+0000", 2024-03-19T11:30:03.032 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0018977060000000001 2024-03-19T11:30:03.032 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.032 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.032 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:03.032 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085377+0000", 2024-03-19T11:30:03.032 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9068999999999997e-05 2024-03-19T11:30:03.032 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.032 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.032 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:03.032 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085443+0000", 2024-03-19T11:30:03.032 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5653000000000003e-05 2024-03-19T11:30:03.032 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:03.032 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:03.033 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:03.033 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.033 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.033 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:30:03.033 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.071777+0000", 2024-03-19T11:30:03.033 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 17.961218919, 2024-03-19T11:30:03.033 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013962445, 2024-03-19T11:30:03.033 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:03.033 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:03.033 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:03.033 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:03.033 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:03.033 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 35 2024-03-19T11:30:03.033 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.034 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:03.034 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.034 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:03.034 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071777+0000", 2024-03-19T11:30:03.034 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:03.034 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.034 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.034 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:03.034 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071777+0000", 2024-03-19T11:30:03.034 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:03.034 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.034 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.034 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:03.035 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071778+0000", 2024-03-19T11:30:03.035 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5000000000000001e-07 2024-03-19T11:30:03.035 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.035 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.035 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:03.035 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071917+0000", 2024-03-19T11:30:03.035 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013880199999999999 2024-03-19T11:30:03.035 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.035 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.035 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:03.035 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071931+0000", 2024-03-19T11:30:03.035 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4080000000000001e-05 2024-03-19T11:30:03.035 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.035 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.035 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:03.036 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071940+0000", 2024-03-19T11:30:03.036 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.363e-06 2024-03-19T11:30:03.036 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.036 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:03.036 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.080462+0000", 2024-03-19T11:30:03.036 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085221810000000002 2024-03-19T11:30:03.036 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.036 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:03.036 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.080582+0000", 2024-03-19T11:30:03.036 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011935299999999999 2024-03-19T11:30:03.036 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.036 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:03.037 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.080782+0000", 2024-03-19T11:30:03.037 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020077100000000001 2024-03-19T11:30:03.037 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.037 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.037 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:03.037 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.083401+0000", 2024-03-19T11:30:03.037 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002618333 2024-03-19T11:30:03.037 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.037 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.037 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:03.037 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085571+0000", 2024-03-19T11:30:03.037 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021702829999999998 2024-03-19T11:30:03.037 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.037 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.037 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:03.038 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085672+0000", 2024-03-19T11:30:03.038 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100876 2024-03-19T11:30:03.038 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.038 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.038 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:03.038 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085739+0000", 2024-03-19T11:30:03.038 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7552999999999995e-05 2024-03-19T11:30:03.038 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:03.038 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:03.039 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:03.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.039 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:30:03.039 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.071957+0000", 2024-03-19T11:30:03.039 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 17.961038730999999, 2024-03-19T11:30:03.039 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013989822000000001, 2024-03-19T11:30:03.039 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:03.039 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:03.039 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:03.039 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:03.039 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:03.039 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 36 2024-03-19T11:30:03.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.040 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:03.040 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.040 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:03.040 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071957+0000", 2024-03-19T11:30:03.040 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:03.040 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.040 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.040 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:03.040 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071957+0000", 2024-03-19T11:30:03.041 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:03.041 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.041 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.041 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:03.041 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071958+0000", 2024-03-19T11:30:03.041 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.06e-06 2024-03-19T11:30:03.041 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.041 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.041 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:03.041 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071965+0000", 2024-03-19T11:30:03.041 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8929999999999999e-06 2024-03-19T11:30:03.041 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.041 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.041 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:03.041 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071976+0000", 2024-03-19T11:30:03.042 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1046999999999999e-05 2024-03-19T11:30:03.042 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.042 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.042 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:03.042 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071984+0000", 2024-03-19T11:30:03.042 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7170000000000008e-06 2024-03-19T11:30:03.042 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.042 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.042 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:03.042 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.081761+0000", 2024-03-19T11:30:03.042 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0097769310000000009 2024-03-19T11:30:03.042 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.042 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.042 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:03.043 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.081928+0000", 2024-03-19T11:30:03.043 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000167219 2024-03-19T11:30:03.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.043 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:03.043 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.082131+0000", 2024-03-19T11:30:03.043 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020244499999999999 2024-03-19T11:30:03.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.043 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:03.043 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.083455+0000", 2024-03-19T11:30:03.043 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001324073 2024-03-19T11:30:03.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.043 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:03.044 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085816+0000", 2024-03-19T11:30:03.044 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0023613309999999999 2024-03-19T11:30:03.044 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.044 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.044 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:03.044 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085892+0000", 2024-03-19T11:30:03.044 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6154000000000003e-05 2024-03-19T11:30:03.044 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.044 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.044 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:03.044 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085947+0000", 2024-03-19T11:30:03.044 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4951999999999998e-05 2024-03-19T11:30:03.044 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:03.044 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:03.044 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:03.045 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.045 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.045 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:30:03.045 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.072000+0000", 2024-03-19T11:30:03.045 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 17.960995981, 2024-03-19T11:30:03.045 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014418268999999999, 2024-03-19T11:30:03.045 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:03.045 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:03.045 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:03.045 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:03.046 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:03.046 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 37 2024-03-19T11:30:03.046 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.046 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:03.046 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.046 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:03.046 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072000+0000", 2024-03-19T11:30:03.046 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:03.046 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.046 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.046 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:03.046 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072000+0000", 2024-03-19T11:30:03.046 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:03.047 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.047 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.047 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:03.047 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072001+0000", 2024-03-19T11:30:03.047 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0699999999999996e-07 2024-03-19T11:30:03.047 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.047 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.048 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:03.048 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072007+0000", 2024-03-19T11:30:03.048 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2060000000000004e-06 2024-03-19T11:30:03.048 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.048 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.048 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:03.048 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072023+0000", 2024-03-19T11:30:03.048 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6617e-05 2024-03-19T11:30:03.048 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.048 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.048 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:03.048 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072031+0000", 2024-03-19T11:30:03.049 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8900000000000007e-06 2024-03-19T11:30:03.049 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.049 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.049 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:03.049 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.082682+0000", 2024-03-19T11:30:03.049 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010650868000000001 2024-03-19T11:30:03.049 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.049 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.049 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:03.049 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.082804+0000", 2024-03-19T11:30:03.050 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012202899999999999 2024-03-19T11:30:03.050 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.050 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.050 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:03.050 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.083006+0000", 2024-03-19T11:30:03.050 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020127799999999999 2024-03-19T11:30:03.050 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.050 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.050 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:03.050 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.084145+0000", 2024-03-19T11:30:03.050 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001139791 2024-03-19T11:30:03.051 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.051 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.051 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:03.051 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.086292+0000", 2024-03-19T11:30:03.051 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021461969999999999 2024-03-19T11:30:03.051 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.051 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.051 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:03.051 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.086364+0000", 2024-03-19T11:30:03.051 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2439000000000002e-05 2024-03-19T11:30:03.051 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.051 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.052 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:03.052 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.086418+0000", 2024-03-19T11:30:03.052 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4246999999999997e-05 2024-03-19T11:30:03.052 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:03.052 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:03.052 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:03.052 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.052 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.052 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:30:03.052 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.339200+0000", 2024-03-19T11:30:03.052 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 17.693796267, 2024-03-19T11:30:03.052 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011947332, 2024-03-19T11:30:03.052 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:03.052 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:03.053 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:03.053 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:03.053 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:03.053 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 109 2024-03-19T11:30:03.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.053 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:03.053 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.053 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:03.053 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339200+0000", 2024-03-19T11:30:03.053 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:03.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.053 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.053 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:03.053 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339200+0000", 2024-03-19T11:30:03.053 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:03.054 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.054 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:03.054 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339201+0000", 2024-03-19T11:30:03.054 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.697e-06 2024-03-19T11:30:03.054 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.054 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:03.054 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339328+0000", 2024-03-19T11:30:03.054 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012660300000000001 2024-03-19T11:30:03.054 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.054 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:03.054 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339362+0000", 2024-03-19T11:30:03.055 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4351999999999999e-05 2024-03-19T11:30:03.055 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.055 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.055 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:03.055 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339382+0000", 2024-03-19T11:30:03.055 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9983000000000002e-05 2024-03-19T11:30:03.055 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.055 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.055 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:03.055 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.345946+0000", 2024-03-19T11:30:03.055 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0065633330000000002 2024-03-19T11:30:03.055 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.055 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.055 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:03.055 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.346073+0000", 2024-03-19T11:30:03.056 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000127799 2024-03-19T11:30:03.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.056 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:03.056 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.346269+0000", 2024-03-19T11:30:03.056 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000195875 2024-03-19T11:30:03.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.056 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:03.056 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.350756+0000", 2024-03-19T11:30:03.056 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044864190000000002 2024-03-19T11:30:03.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.056 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:03.057 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.350878+0000", 2024-03-19T11:30:03.057 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012271600000000001 2024-03-19T11:30:03.057 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.057 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:03.057 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.351030+0000", 2024-03-19T11:30:03.057 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015188199999999999 2024-03-19T11:30:03.057 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.057 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:03.057 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.351147+0000", 2024-03-19T11:30:03.057 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000116673 2024-03-19T11:30:03.058 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:03.058 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:03.058 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:03.058 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.058 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.058 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:30:03.058 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.339422+0000", 2024-03-19T11:30:03.058 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 17.693573614999998, 2024-03-19T11:30:03.058 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012041094, 2024-03-19T11:30:03.058 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:03.058 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:03.058 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:03.058 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:03.059 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:03.059 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 110 2024-03-19T11:30:03.059 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.059 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:03.059 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.059 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:03.059 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339422+0000", 2024-03-19T11:30:03.059 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:03.059 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.060 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.060 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:03.060 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339422+0000", 2024-03-19T11:30:03.060 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:03.060 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.060 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.060 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:03.060 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339424+0000", 2024-03-19T11:30:03.060 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.697e-06 2024-03-19T11:30:03.060 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.060 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.060 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:03.060 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339440+0000", 2024-03-19T11:30:03.060 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5642999999999999e-05 2024-03-19T11:30:03.060 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.061 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:03.061 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339462+0000", 2024-03-19T11:30:03.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2167e-05 2024-03-19T11:30:03.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.061 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:03.061 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339477+0000", 2024-03-19T11:30:03.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5189e-05 2024-03-19T11:30:03.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.061 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:03.061 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.347327+0000", 2024-03-19T11:30:03.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0078502559999999999 2024-03-19T11:30:03.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:03.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.347450+0000", 2024-03-19T11:30:03.062 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012261600000000001 2024-03-19T11:30:03.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:03.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.347644+0000", 2024-03-19T11:30:03.062 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019408500000000001 2024-03-19T11:30:03.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:03.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.351304+0000", 2024-03-19T11:30:03.063 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003660118 2024-03-19T11:30:03.063 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.063 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.063 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:03.063 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.351328+0000", 2024-03-19T11:30:03.063 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3734e-05 2024-03-19T11:30:03.063 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.063 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.063 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:03.063 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.351402+0000", 2024-03-19T11:30:03.063 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3708999999999995e-05 2024-03-19T11:30:03.063 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.063 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.063 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:03.064 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.351463+0000", 2024-03-19T11:30:03.064 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1879999999999997e-05 2024-03-19T11:30:03.064 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:03.064 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:03.064 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:03.064 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.064 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.064 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:30:03.064 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.339510+0000", 2024-03-19T11:30:03.064 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 17.693486133, 2024-03-19T11:30:03.064 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012358388999999999, 2024-03-19T11:30:03.064 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:03.065 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:03.065 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:03.065 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:03.065 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:03.065 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 111 2024-03-19T11:30:03.065 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.065 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:03.065 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.065 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:03.065 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339510+0000", 2024-03-19T11:30:03.065 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:03.065 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.065 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.065 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:03.065 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339510+0000", 2024-03-19T11:30:03.066 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:03.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.066 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.066 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:03.066 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339511+0000", 2024-03-19T11:30:03.066 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4399999999999998e-07 2024-03-19T11:30:03.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.066 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.066 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:03.066 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339518+0000", 2024-03-19T11:30:03.066 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9999999999999999e-06 2024-03-19T11:30:03.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.066 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.067 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:03.067 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339535+0000", 2024-03-19T11:30:03.067 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7552999999999999e-05 2024-03-19T11:30:03.067 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.067 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.067 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:03.067 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339550+0000", 2024-03-19T11:30:03.067 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5003e-05 2024-03-19T11:30:03.067 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.067 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.067 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:03.067 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.348403+0000", 2024-03-19T11:30:03.067 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0088530810000000005 2024-03-19T11:30:03.068 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.068 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.068 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:03.068 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.348629+0000", 2024-03-19T11:30:03.068 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022534599999999999 2024-03-19T11:30:03.068 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.068 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.068 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:03.068 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.348976+0000", 2024-03-19T11:30:03.068 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00034710699999999998 2024-03-19T11:30:03.068 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.068 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.068 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:03.068 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.351479+0000", 2024-03-19T11:30:03.069 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0025035309999999998 2024-03-19T11:30:03.069 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.069 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.069 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:03.069 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.351602+0000", 2024-03-19T11:30:03.069 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000122912 2024-03-19T11:30:03.069 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.069 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.069 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:03.069 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.351766+0000", 2024-03-19T11:30:03.069 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016410599999999999 2024-03-19T11:30:03.069 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.069 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.069 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:03.069 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.351868+0000", 2024-03-19T11:30:03.070 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010180599999999999 2024-03-19T11:30:03.070 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:03.070 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:03.070 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:03.070 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.070 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.070 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:30:03.070 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.339576+0000", 2024-03-19T11:30:03.070 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 17.693419709, 2024-03-19T11:30:03.070 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012548515999999999, 2024-03-19T11:30:03.070 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:03.070 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:03.070 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:03.070 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:03.071 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:03.071 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 112 2024-03-19T11:30:03.071 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.071 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:03.071 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.071 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:03.071 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339576+0000", 2024-03-19T11:30:03.071 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:03.071 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.071 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.071 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:03.071 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339576+0000", 2024-03-19T11:30:03.071 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:03.071 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.072 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.072 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:03.072 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339577+0000", 2024-03-19T11:30:03.072 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2529999999999999e-06 2024-03-19T11:30:03.072 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.072 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.072 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:03.072 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339813+0000", 2024-03-19T11:30:03.072 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000235811 2024-03-19T11:30:03.072 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.072 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.072 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:03.072 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339848+0000", 2024-03-19T11:30:03.072 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4437000000000002e-05 2024-03-19T11:30:03.073 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:03.073 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339867+0000", 2024-03-19T11:30:03.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9046e-05 2024-03-19T11:30:03.073 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:03.073 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.349815+0000", 2024-03-19T11:30:03.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0099484029999999998 2024-03-19T11:30:03.073 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:03.073 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.349955+0000", 2024-03-19T11:30:03.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000140106 2024-03-19T11:30:03.074 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.074 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.074 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:03.074 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.350163+0000", 2024-03-19T11:30:03.074 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020803799999999999 2024-03-19T11:30:03.074 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.074 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.074 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:03.074 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.351984+0000", 2024-03-19T11:30:03.074 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0018202159999999999 2024-03-19T11:30:03.074 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.074 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.074 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:03.074 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352002+0000", 2024-03-19T11:30:03.075 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8430000000000001e-05 2024-03-19T11:30:03.075 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.075 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.075 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:03.075 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352070+0000", 2024-03-19T11:30:03.075 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7623e-05 2024-03-19T11:30:03.075 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.075 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.075 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:03.075 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352125+0000", 2024-03-19T11:30:03.075 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5152999999999999e-05 2024-03-19T11:30:03.075 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:03.075 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:03.075 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:03.075 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.075 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.076 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:30:03.076 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.339898+0000", 2024-03-19T11:30:03.076 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 17.693098359, 2024-03-19T11:30:03.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019501793, 2024-03-19T11:30:03.076 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:03.076 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:03.076 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:03.076 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:03.076 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:03.076 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113 2024-03-19T11:30:03.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.076 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:03.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.076 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:03.076 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339898+0000", 2024-03-19T11:30:03.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:03.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.077 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:03.077 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339898+0000", 2024-03-19T11:30:03.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:03.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.077 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:03.077 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339899+0000", 2024-03-19T11:30:03.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.353e-06 2024-03-19T11:30:03.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.078 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:03.078 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339907+0000", 2024-03-19T11:30:03.078 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3000000000000002e-06 2024-03-19T11:30:03.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.078 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:03.078 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339924+0000", 2024-03-19T11:30:03.078 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6906999999999999e-05 2024-03-19T11:30:03.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.079 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:03.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339933+0000", 2024-03-19T11:30:03.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8969999999999997e-06 2024-03-19T11:30:03.079 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.079 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.079 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:03.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352239+0000", 2024-03-19T11:30:03.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012305814999999999 2024-03-19T11:30:03.079 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.079 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.079 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:03.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352466+0000", 2024-03-19T11:30:03.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022684100000000001 2024-03-19T11:30:03.079 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.080 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:03.080 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352816+0000", 2024-03-19T11:30:03.080 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00035030499999999997 2024-03-19T11:30:03.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.080 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:03.080 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.358956+0000", 2024-03-19T11:30:03.080 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061400079999999998 2024-03-19T11:30:03.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.080 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:03.080 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359118+0000", 2024-03-19T11:30:03.080 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016237599999999999 2024-03-19T11:30:03.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.081 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.081 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:03.081 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359287+0000", 2024-03-19T11:30:03.081 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000168488 2024-03-19T11:30:03.081 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.081 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.081 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:03.081 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359399+0000", 2024-03-19T11:30:03.081 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000112503 2024-03-19T11:30:03.081 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:03.082 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:03.082 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:03.082 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.082 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.082 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:30:03.082 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.339947+0000", 2024-03-19T11:30:03.082 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 17.693048592, 2024-03-19T11:30:03.082 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021923069999999999, 2024-03-19T11:30:03.082 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:03.082 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:03.082 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:03.083 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:03.083 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:03.083 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 114 2024-03-19T11:30:03.083 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.083 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:03.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.083 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:03.083 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339947+0000", 2024-03-19T11:30:03.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:03.083 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.083 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:03.083 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339947+0000", 2024-03-19T11:30:03.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:03.084 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.084 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.084 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:03.084 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339948+0000", 2024-03-19T11:30:03.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6000000000000003e-07 2024-03-19T11:30:03.084 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.084 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.084 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:03.084 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339956+0000", 2024-03-19T11:30:03.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1559999999999995e-06 2024-03-19T11:30:03.084 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.084 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.084 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:03.084 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339967+0000", 2024-03-19T11:30:03.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0669999999999999e-05 2024-03-19T11:30:03.085 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.085 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:03.085 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339975+0000", 2024-03-19T11:30:03.085 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0099999999999995e-06 2024-03-19T11:30:03.085 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.085 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:03.085 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.353672+0000", 2024-03-19T11:30:03.085 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013696583999999999 2024-03-19T11:30:03.085 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.085 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:03.085 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.353808+0000", 2024-03-19T11:30:03.085 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013652299999999999 2024-03-19T11:30:03.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.086 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.086 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:03.086 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.354036+0000", 2024-03-19T11:30:03.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022744799999999999 2024-03-19T11:30:03.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.086 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.086 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:03.086 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.360480+0000", 2024-03-19T11:30:03.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0064441630000000001 2024-03-19T11:30:03.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.086 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.086 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:03.086 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361722+0000", 2024-03-19T11:30:03.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001242754 2024-03-19T11:30:03.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:03.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361807+0000", 2024-03-19T11:30:03.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4605999999999996e-05 2024-03-19T11:30:03.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:03.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361870+0000", 2024-03-19T11:30:03.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3396000000000001e-05 2024-03-19T11:30:03.087 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:03.087 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:03.087 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:03.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.088 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:30:03.088 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.340017+0000", 2024-03-19T11:30:03.088 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 17.692979275999999, 2024-03-19T11:30:03.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022169355000000002, 2024-03-19T11:30:03.088 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:03.088 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:03.088 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:03.088 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:03.088 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:03.088 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 115 2024-03-19T11:30:03.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.088 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:03.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.088 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:03.089 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340017+0000", 2024-03-19T11:30:03.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:03.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.089 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:03.089 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340017+0000", 2024-03-19T11:30:03.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:03.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.089 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:03.089 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340018+0000", 2024-03-19T11:30:03.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.043e-06 2024-03-19T11:30:03.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.089 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:03.090 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340139+0000", 2024-03-19T11:30:03.090 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121502 2024-03-19T11:30:03.090 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:03.090 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340157+0000", 2024-03-19T11:30:03.090 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7382999999999999e-05 2024-03-19T11:30:03.090 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:03.090 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340166+0000", 2024-03-19T11:30:03.090 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2469999999999997e-06 2024-03-19T11:30:03.090 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.091 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:03.091 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.357470+0000", 2024-03-19T11:30:03.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017304119999999999 2024-03-19T11:30:03.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.091 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:03.091 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.357671+0000", 2024-03-19T11:30:03.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000200798 2024-03-19T11:30:03.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.091 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:03.091 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.358002+0000", 2024-03-19T11:30:03.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033090099999999998 2024-03-19T11:30:03.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.092 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:03.092 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361892+0000", 2024-03-19T11:30:03.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00389018 2024-03-19T11:30:03.092 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.092 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:03.092 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361983+0000", 2024-03-19T11:30:03.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0976000000000004e-05 2024-03-19T11:30:03.092 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.092 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:03.092 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362071+0000", 2024-03-19T11:30:03.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8022999999999994e-05 2024-03-19T11:30:03.093 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.093 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:03.093 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362186+0000", 2024-03-19T11:30:03.093 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115182 2024-03-19T11:30:03.093 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:03.093 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:03.093 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:03.093 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.093 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:30:03.093 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.340183+0000", 2024-03-19T11:30:03.093 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 17.692812444000001, 2024-03-19T11:30:03.093 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022279841000000002, 2024-03-19T11:30:03.093 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:03.094 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:03.094 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:03.094 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:03.094 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:03.094 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 116 2024-03-19T11:30:03.094 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.094 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:03.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.094 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:03.094 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340183+0000", 2024-03-19T11:30:03.094 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:03.094 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.095 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:03.095 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340183+0000", 2024-03-19T11:30:03.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:03.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.095 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:03.095 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340184+0000", 2024-03-19T11:30:03.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9000000000000005e-07 2024-03-19T11:30:03.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.095 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:03.095 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340191+0000", 2024-03-19T11:30:03.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0360000000000003e-06 2024-03-19T11:30:03.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.096 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:03.096 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340201+0000", 2024-03-19T11:30:03.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0844e-05 2024-03-19T11:30:03.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.096 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:03.096 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340209+0000", 2024-03-19T11:30:03.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7360000000000001e-06 2024-03-19T11:30:03.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.096 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:03.096 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359535+0000", 2024-03-19T11:30:03.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019325546999999998 2024-03-19T11:30:03.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:03.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359672+0000", 2024-03-19T11:30:03.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013706600000000001 2024-03-19T11:30:03.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:03.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359905+0000", 2024-03-19T11:30:03.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023334500000000001 2024-03-19T11:30:03.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:03.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362306+0000", 2024-03-19T11:30:03.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002401041 2024-03-19T11:30:03.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:03.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362329+0000", 2024-03-19T11:30:03.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3283999999999999e-05 2024-03-19T11:30:03.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:03.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362402+0000", 2024-03-19T11:30:03.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2459000000000005e-05 2024-03-19T11:30:03.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:03.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362463+0000", 2024-03-19T11:30:03.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1493000000000002e-05 2024-03-19T11:30:03.099 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:03.099 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:03.099 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:03.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.099 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:30:03.099 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.340222+0000", 2024-03-19T11:30:03.099 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 17.692773860999999, 2024-03-19T11:30:03.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022500218999999998, 2024-03-19T11:30:03.099 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:03.099 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:03.099 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:03.099 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:03.099 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:03.099 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 117 2024-03-19T11:30:03.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.100 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:03.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.100 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:03.100 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340222+0000", 2024-03-19T11:30:03.100 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:03.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.100 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:03.100 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340222+0000", 2024-03-19T11:30:03.100 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:03.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.100 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:03.100 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340223+0000", 2024-03-19T11:30:03.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7999999999999995e-07 2024-03-19T11:30:03.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.101 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:03.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340229+0000", 2024-03-19T11:30:03.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0399999999999998e-06 2024-03-19T11:30:03.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.101 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:03.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340239+0000", 2024-03-19T11:30:03.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0190000000000001e-05 2024-03-19T11:30:03.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.101 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:03.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340250+0000", 2024-03-19T11:30:03.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0720000000000001e-05 2024-03-19T11:30:03.102 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:03.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.360587+0000", 2024-03-19T11:30:03.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020337542 2024-03-19T11:30:03.102 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:03.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.360818+0000", 2024-03-19T11:30:03.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023120500000000001 2024-03-19T11:30:03.102 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:03.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361218+0000", 2024-03-19T11:30:03.103 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000399637 2024-03-19T11:30:03.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.103 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:03.103 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362483+0000", 2024-03-19T11:30:03.103 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012649200000000001 2024-03-19T11:30:03.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.103 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:03.103 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362538+0000", 2024-03-19T11:30:03.103 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4490000000000002e-05 2024-03-19T11:30:03.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.103 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:03.104 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362617+0000", 2024-03-19T11:30:03.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9839000000000006e-05 2024-03-19T11:30:03.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.104 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:03.104 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362722+0000", 2024-03-19T11:30:03.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104956 2024-03-19T11:30:03.104 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:03.104 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:03.104 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:03.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.104 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:30:03.104 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.806124+0000", 2024-03-19T11:30:03.104 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 17.226871710000001, 2024-03-19T11:30:03.105 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010639073000000001, 2024-03-19T11:30:03.105 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:03.105 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:03.105 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:03.105 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:03.105 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:03.105 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 136 2024-03-19T11:30:03.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.105 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:03.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.105 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:03.105 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806124+0000", 2024-03-19T11:30:03.105 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:03.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:03.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806124+0000", 2024-03-19T11:30:03.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:03.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:03.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806125+0000", 2024-03-19T11:30:03.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.13e-07 2024-03-19T11:30:03.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:03.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806130+0000", 2024-03-19T11:30:03.107 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4199999999999998e-06 2024-03-19T11:30:03.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.107 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:03.107 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806138+0000", 2024-03-19T11:30:03.107 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7479999999999993e-06 2024-03-19T11:30:03.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.107 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:03.107 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806145+0000", 2024-03-19T11:30:03.107 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2269999999999996e-06 2024-03-19T11:30:03.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.107 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:03.107 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.808827+0000", 2024-03-19T11:30:03.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026814260000000002 2024-03-19T11:30:03.108 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.108 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:03.108 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.808880+0000", 2024-03-19T11:30:03.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3427999999999997e-05 2024-03-19T11:30:03.108 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.108 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:03.108 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.808969+0000", 2024-03-19T11:30:03.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8525999999999999e-05 2024-03-19T11:30:03.108 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.108 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:03.108 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.811343+0000", 2024-03-19T11:30:03.109 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002374101 2024-03-19T11:30:03.109 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.109 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.109 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:03.109 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.816680+0000", 2024-03-19T11:30:03.109 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053375990000000002 2024-03-19T11:30:03.109 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.109 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.109 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:03.109 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.816716+0000", 2024-03-19T11:30:03.109 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5781999999999997e-05 2024-03-19T11:30:03.109 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.109 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.109 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:03.109 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.816763+0000", 2024-03-19T11:30:03.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7302999999999998e-05 2024-03-19T11:30:03.110 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:03.110 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:03.110 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:03.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.110 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.110 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:30:03.110 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.806171+0000", 2024-03-19T11:30:03.110 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 17.226824487999998, 2024-03-19T11:30:03.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010681563999999999, 2024-03-19T11:30:03.110 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:03.110 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:03.110 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:03.110 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:03.110 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:03.111 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 137 2024-03-19T11:30:03.111 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.111 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:03.111 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.111 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:03.111 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806171+0000", 2024-03-19T11:30:03.111 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:03.111 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.111 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.111 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:03.111 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806171+0000", 2024-03-19T11:30:03.111 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:03.111 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.111 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.111 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:03.112 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806172+0000", 2024-03-19T11:30:03.112 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9100000000000003e-07 2024-03-19T11:30:03.112 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.112 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.112 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:03.112 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806243+0000", 2024-03-19T11:30:03.112 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0563999999999997e-05 2024-03-19T11:30:03.112 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.112 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.112 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:03.112 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806252+0000", 2024-03-19T11:30:03.112 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7499999999999998e-06 2024-03-19T11:30:03.112 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.112 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.112 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:03.113 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806263+0000", 2024-03-19T11:30:03.113 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0205e-05 2024-03-19T11:30:03.113 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.113 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.113 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:03.113 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.809226+0000", 2024-03-19T11:30:03.113 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029634679999999999 2024-03-19T11:30:03.113 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.113 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.113 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:03.113 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.809278+0000", 2024-03-19T11:30:03.113 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1811000000000003e-05 2024-03-19T11:30:03.113 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.113 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.113 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:03.114 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.809363+0000", 2024-03-19T11:30:03.114 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4752000000000004e-05 2024-03-19T11:30:03.114 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.114 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.114 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:03.114 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.812567+0000", 2024-03-19T11:30:03.114 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032038710000000001 2024-03-19T11:30:03.114 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.114 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.114 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:03.114 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.816801+0000", 2024-03-19T11:30:03.114 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0042343040000000004 2024-03-19T11:30:03.114 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.114 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.114 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:03.115 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.816833+0000", 2024-03-19T11:30:03.115 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2289000000000002e-05 2024-03-19T11:30:03.115 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.115 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.115 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:03.115 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.816853+0000", 2024-03-19T11:30:03.115 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9859000000000001e-05 2024-03-19T11:30:03.115 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:03.115 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:03.115 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:03.115 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.116 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.116 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:30:03.116 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.806275+0000", 2024-03-19T11:30:03.116 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 17.226721152, 2024-03-19T11:30:03.116 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010695962, 2024-03-19T11:30:03.116 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:03.116 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:03.116 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:03.116 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:03.116 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:03.116 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 138 2024-03-19T11:30:03.116 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.116 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:03.116 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.116 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:03.117 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806275+0000", 2024-03-19T11:30:03.117 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:03.117 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.117 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.117 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:03.117 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806275+0000", 2024-03-19T11:30:03.117 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:03.117 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.117 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.117 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:03.117 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806275+0000", 2024-03-19T11:30:03.117 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1099999999999995e-07 2024-03-19T11:30:03.117 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.117 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.118 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:03.118 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806280+0000", 2024-03-19T11:30:03.118 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.42e-06 2024-03-19T11:30:03.118 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.118 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.118 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:03.118 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806288+0000", 2024-03-19T11:30:03.118 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9589999999999995e-06 2024-03-19T11:30:03.118 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.118 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.118 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:03.118 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806296+0000", 2024-03-19T11:30:03.118 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6980000000000004e-06 2024-03-19T11:30:03.118 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.119 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.119 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:03.119 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.810916+0000", 2024-03-19T11:30:03.119 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0046194579999999999 2024-03-19T11:30:03.119 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.119 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.119 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:03.119 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.810975+0000", 2024-03-19T11:30:03.119 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8767999999999997e-05 2024-03-19T11:30:03.119 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.119 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.119 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:03.119 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.811066+0000", 2024-03-19T11:30:03.119 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1543000000000001e-05 2024-03-19T11:30:03.119 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.120 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.120 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:03.120 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.813111+0000", 2024-03-19T11:30:03.120 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0020449650000000002 2024-03-19T11:30:03.120 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.120 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.120 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:03.120 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.816889+0000", 2024-03-19T11:30:03.120 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037775270000000001 2024-03-19T11:30:03.120 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.120 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.120 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:03.120 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.816925+0000", 2024-03-19T11:30:03.120 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6539999999999999e-05 2024-03-19T11:30:03.120 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.121 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:03.121 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.816971+0000", 2024-03-19T11:30:03.121 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5473000000000003e-05 2024-03-19T11:30:03.121 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:03.121 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:03.121 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:03.121 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.121 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:30:03.121 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.806306+0000", 2024-03-19T11:30:03.121 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 17.226689561000001, 2024-03-19T11:30:03.121 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010748485, 2024-03-19T11:30:03.121 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:03.122 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:03.122 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:03.122 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:03.122 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:03.122 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 139 2024-03-19T11:30:03.122 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.122 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:03.122 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.122 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:03.122 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806306+0000", 2024-03-19T11:30:03.122 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:03.122 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.122 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.122 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:03.122 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806306+0000", 2024-03-19T11:30:03.123 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:03.123 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.123 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.123 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:03.123 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806307+0000", 2024-03-19T11:30:03.123 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7199999999999999e-07 2024-03-19T11:30:03.123 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.123 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.123 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:03.123 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806311+0000", 2024-03-19T11:30:03.123 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1989999999999999e-06 2024-03-19T11:30:03.123 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.123 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.123 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:03.123 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806323+0000", 2024-03-19T11:30:03.124 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2075e-05 2024-03-19T11:30:03.124 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.124 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.124 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:03.124 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806331+0000", 2024-03-19T11:30:03.124 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4959999999999999e-06 2024-03-19T11:30:03.124 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.124 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.124 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:03.124 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.811381+0000", 2024-03-19T11:30:03.124 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.005050232 2024-03-19T11:30:03.124 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.124 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.124 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:03.124 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.811437+0000", 2024-03-19T11:30:03.125 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6116999999999998e-05 2024-03-19T11:30:03.125 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.125 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.125 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:03.125 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.811525+0000", 2024-03-19T11:30:03.125 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7757999999999996e-05 2024-03-19T11:30:03.125 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.125 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.125 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:03.125 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.813119+0000", 2024-03-19T11:30:03.125 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0015946739999999999 2024-03-19T11:30:03.125 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.125 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.125 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:03.125 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.817005+0000", 2024-03-19T11:30:03.126 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038853080000000001 2024-03-19T11:30:03.126 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.126 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.126 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:03.126 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.817034+0000", 2024-03-19T11:30:03.126 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9297000000000001e-05 2024-03-19T11:30:03.126 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.126 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.126 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:03.126 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.817055+0000", 2024-03-19T11:30:03.126 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0857e-05 2024-03-19T11:30:03.126 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:03.126 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:03.126 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:03.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.127 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:30:03.127 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.806342+0000", 2024-03-19T11:30:03.127 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 17.226653553999999, 2024-03-19T11:30:03.127 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010826809999999999, 2024-03-19T11:30:03.127 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:03.127 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:03.127 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:03.127 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:03.127 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:03.127 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 140 2024-03-19T11:30:03.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.127 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:03.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.128 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:03.128 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806342+0000", 2024-03-19T11:30:03.128 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:03.128 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.128 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.128 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:03.128 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806342+0000", 2024-03-19T11:30:03.128 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:03.128 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.128 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.128 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:03.128 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806343+0000", 2024-03-19T11:30:03.128 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6600000000000002e-07 2024-03-19T11:30:03.128 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.128 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.128 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:03.129 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806422+0000", 2024-03-19T11:30:03.129 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9536000000000006e-05 2024-03-19T11:30:03.129 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.129 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:03.129 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806439+0000", 2024-03-19T11:30:03.129 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7098000000000001e-05 2024-03-19T11:30:03.129 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.129 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:03.129 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806447+0000", 2024-03-19T11:30:03.129 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1300000000000003e-06 2024-03-19T11:30:03.129 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.129 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:03.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.811917+0000", 2024-03-19T11:30:03.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054699019999999996 2024-03-19T11:30:03.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.130 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:03.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.811973+0000", 2024-03-19T11:30:03.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6001000000000001e-05 2024-03-19T11:30:03.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.130 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:03.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.812055+0000", 2024-03-19T11:30:03.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2655000000000001e-05 2024-03-19T11:30:03.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.131 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:03.131 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.813256+0000", 2024-03-19T11:30:03.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012011999999999999 2024-03-19T11:30:03.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.131 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:03.131 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.817091+0000", 2024-03-19T11:30:03.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038344659999999999 2024-03-19T11:30:03.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.131 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:03.131 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.817125+0000", 2024-03-19T11:30:03.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4653000000000002e-05 2024-03-19T11:30:03.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.132 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:03.132 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.817169+0000", 2024-03-19T11:30:03.132 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3702999999999998e-05 2024-03-19T11:30:03.132 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:03.132 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:03.132 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:03.132 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.132 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:30:03.132 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.806461+0000", 2024-03-19T11:30:03.132 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 17.22653541, 2024-03-19T11:30:03.132 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010789451, 2024-03-19T11:30:03.132 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:03.132 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:03.132 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:03.133 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:03.133 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:03.133 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 141 2024-03-19T11:30:03.133 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.133 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:03.133 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.133 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:03.133 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806461+0000", 2024-03-19T11:30:03.133 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:03.133 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.133 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.133 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:03.133 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806461+0000", 2024-03-19T11:30:03.133 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:03.133 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.134 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.134 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:03.134 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806461+0000", 2024-03-19T11:30:03.134 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5499999999999997e-07 2024-03-19T11:30:03.134 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.134 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.134 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:03.134 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806466+0000", 2024-03-19T11:30:03.134 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9049999999999996e-06 2024-03-19T11:30:03.135 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.135 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.135 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:03.135 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806474+0000", 2024-03-19T11:30:03.135 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6009999999999997e-06 2024-03-19T11:30:03.135 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.135 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.135 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:03.135 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806480+0000", 2024-03-19T11:30:03.135 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.207e-06 2024-03-19T11:30:03.135 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.135 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.135 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:03.135 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.812605+0000", 2024-03-19T11:30:03.136 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.006125096 2024-03-19T11:30:03.136 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.136 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.136 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:03.136 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.812662+0000", 2024-03-19T11:30:03.136 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7095999999999997e-05 2024-03-19T11:30:03.136 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.136 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.136 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:03.136 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.812751+0000", 2024-03-19T11:30:03.136 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9115000000000003e-05 2024-03-19T11:30:03.136 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.137 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.137 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:03.137 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.813361+0000", 2024-03-19T11:30:03.137 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00060919399999999997 2024-03-19T11:30:03.137 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.137 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.137 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:03.137 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.817201+0000", 2024-03-19T11:30:03.137 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038408510000000002 2024-03-19T11:30:03.137 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.137 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.137 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:03.137 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.817230+0000", 2024-03-19T11:30:03.137 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8963999999999999e-05 2024-03-19T11:30:03.137 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:03.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:03.138 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:03.138 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.817250+0000", 2024-03-19T11:30:03.138 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9667e-05 2024-03-19T11:30:03.138 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:03.138 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:03.138 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:03.138 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:03.138 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:03.138 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:03.140 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-19T11:30:03.140 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:03.148+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:03.241 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:03.248+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:03.255 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:03.268+0000 7f923fa20640 -1 osd.7 25 handle_osd_map injecting map crc failure 2024-03-19T11:30:03.276 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:03.276 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:30:03.276 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:30:03.276 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:03.291 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-19T11:30:03.342 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:03.352+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:03.443 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:03.452+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:03.459 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:03.459 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:30:03.459 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:30:03.459 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:30:03.459 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:03.473 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-19T11:30:03.558 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:03.568+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:03.647 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:03.647 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:30:03.647 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:30:03.647 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:30:03.647 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.647 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:34 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'20, mlcod=15'16)", 2024-03-19T11:30:03.647 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.077542+0000", 2024-03-19T11:30:03.647 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.580056106000001, 2024-03-19T11:30:03.648 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0064305179999999997, 2024-03-19T11:30:03.648 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:03.648 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:03.648 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:03.648 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.648 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:03.648 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.077542+0000", 2024-03-19T11:30:03.648 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:03.648 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.648 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.648 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:03.648 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.077542+0000", 2024-03-19T11:30:03.648 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:03.648 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.648 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.649 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:03.649 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.077543+0000", 2024-03-19T11:30:03.649 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.387e-06 2024-03-19T11:30:03.649 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.649 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.649 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:03.649 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.077555+0000", 2024-03-19T11:30:03.649 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1759999999999999e-05 2024-03-19T11:30:03.649 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.649 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.649 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:03.649 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.077580+0000", 2024-03-19T11:30:03.649 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.529e-05 2024-03-19T11:30:03.650 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.650 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.650 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:03.650 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.077596+0000", 2024-03-19T11:30:03.650 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5466000000000001e-05 2024-03-19T11:30:03.650 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.650 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.650 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:03.650 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.077753+0000", 2024-03-19T11:30:03.650 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000157026 2024-03-19T11:30:03.650 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.650 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.650 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:03.650 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.077789+0000", 2024-03-19T11:30:03.651 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6259e-05 2024-03-19T11:30:03.651 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.651 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.651 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:03.651 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.083910+0000", 2024-03-19T11:30:03.652 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0061212080000000004 2024-03-19T11:30:03.652 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.652 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.652 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:03.652 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.083972+0000", 2024-03-19T11:30:03.652 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2121999999999999e-05 2024-03-19T11:30:03.652 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:03.652 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:03.652 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:03.652 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.652 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.652 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:35 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'21, mlcod=15'17)", 2024-03-19T11:30:03.652 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.080212+0000", 2024-03-19T11:30:03.653 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.577385980999999, 2024-03-19T11:30:03.653 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003806023, 2024-03-19T11:30:03.653 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:03.653 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:03.653 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:03.653 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.653 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:03.653 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.080212+0000", 2024-03-19T11:30:03.653 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:03.653 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.653 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.654 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:03.654 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.080212+0000", 2024-03-19T11:30:03.654 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:03.654 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.654 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.654 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:03.654 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.080213+0000", 2024-03-19T11:30:03.654 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2330000000000001e-06 2024-03-19T11:30:03.654 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.654 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.654 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:03.654 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.080266+0000", 2024-03-19T11:30:03.654 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2460000000000003e-05 2024-03-19T11:30:03.654 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.655 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.655 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:03.655 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.080288+0000", 2024-03-19T11:30:03.655 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1892999999999999e-05 2024-03-19T11:30:03.655 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.655 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.655 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:03.655 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.080323+0000", 2024-03-19T11:30:03.655 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5107000000000001e-05 2024-03-19T11:30:03.655 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.655 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.656 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:03.656 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.082276+0000", 2024-03-19T11:30:03.656 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0019535170000000001 2024-03-19T11:30:03.656 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.656 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.656 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:03.656 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.082315+0000", 2024-03-19T11:30:03.656 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8760000000000002e-05 2024-03-19T11:30:03.656 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.656 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.656 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:03.656 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.083977+0000", 2024-03-19T11:30:03.656 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001662483 2024-03-19T11:30:03.656 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.657 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.657 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:03.657 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.084018+0000", 2024-03-19T11:30:03.657 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.057e-05 2024-03-19T11:30:03.657 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:03.657 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:03.657 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:03.657 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.657 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.657 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:39 1.0 e15/14 1:30306672:devicehealth::main.db.0000000000000000:head v 15'25, mlcod=15'22)", 2024-03-19T11:30:03.657 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.088045+0000", 2024-03-19T11:30:03.658 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.569553191000001, 2024-03-19T11:30:03.658 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0036692769999999999, 2024-03-19T11:30:03.658 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:03.658 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:03.658 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:03.658 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.658 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:03.658 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.088045+0000", 2024-03-19T11:30:03.658 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:03.658 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.658 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.658 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:03.658 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.088045+0000", 2024-03-19T11:30:03.659 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:03.659 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.659 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.659 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:03.659 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.088046+0000", 2024-03-19T11:30:03.659 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1230000000000001e-06 2024-03-19T11:30:03.659 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.659 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.659 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:03.659 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.088109+0000", 2024-03-19T11:30:03.659 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3326999999999999e-05 2024-03-19T11:30:03.660 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.660 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.660 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:03.660 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.088133+0000", 2024-03-19T11:30:03.660 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3356000000000001e-05 2024-03-19T11:30:03.660 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.660 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.660 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:03.660 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.088149+0000", 2024-03-19T11:30:03.660 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6192999999999999e-05 2024-03-19T11:30:03.660 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.660 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.660 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:03.660 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.088312+0000", 2024-03-19T11:30:03.661 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000162922 2024-03-19T11:30:03.661 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.661 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.661 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:03.661 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.088354+0000", 2024-03-19T11:30:03.661 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2722999999999997e-05 2024-03-19T11:30:03.661 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.662 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.662 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:03.662 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.091621+0000", 2024-03-19T11:30:03.662 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0032666169999999999 2024-03-19T11:30:03.662 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.662 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.662 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:03.662 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.091714+0000", 2024-03-19T11:30:03.662 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3016000000000005e-05 2024-03-19T11:30:03.662 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:03.662 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:03.662 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:03.662 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.662 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.663 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:40 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'26, mlcod=15'24)", 2024-03-19T11:30:03.663 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.093333+0000", 2024-03-19T11:30:03.663 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.564265256999999, 2024-03-19T11:30:03.663 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0037043660000000002, 2024-03-19T11:30:03.663 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:03.663 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:03.663 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:03.663 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.663 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:03.664 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.093333+0000", 2024-03-19T11:30:03.664 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:03.664 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.664 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.664 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:03.664 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.093333+0000", 2024-03-19T11:30:03.664 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:03.664 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.664 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.664 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:03.664 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.093334+0000", 2024-03-19T11:30:03.664 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.33e-06 2024-03-19T11:30:03.664 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.664 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.665 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:03.665 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.093343+0000", 2024-03-19T11:30:03.665 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3659999999999998e-06 2024-03-19T11:30:03.665 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.665 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.665 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:03.665 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.093368+0000", 2024-03-19T11:30:03.665 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4967e-05 2024-03-19T11:30:03.665 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.665 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.665 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:03.665 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.093385+0000", 2024-03-19T11:30:03.665 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6773e-05 2024-03-19T11:30:03.665 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.665 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.666 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:03.666 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.093533+0000", 2024-03-19T11:30:03.666 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014780199999999999 2024-03-19T11:30:03.666 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.666 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.666 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:03.666 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.093571+0000", 2024-03-19T11:30:03.666 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7929999999999998e-05 2024-03-19T11:30:03.666 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.666 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.666 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:03.666 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.096945+0000", 2024-03-19T11:30:03.666 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033740829999999999 2024-03-19T11:30:03.667 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.667 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.667 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:03.667 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.097037+0000", 2024-03-19T11:30:03.667 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2114999999999995e-05 2024-03-19T11:30:03.667 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:03.667 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:03.667 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:03.667 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.667 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.667 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:114 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'76, mlcod=16'73)", 2024-03-19T11:30:03.668 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.353321+0000", 2024-03-19T11:30:03.668 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.304277252999999, 2024-03-19T11:30:03.668 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050714999999999996, 2024-03-19T11:30:03.668 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:03.668 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:03.668 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:03.668 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.668 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:03.668 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.353321+0000", 2024-03-19T11:30:03.668 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:03.668 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.668 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.668 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:03.668 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.353321+0000", 2024-03-19T11:30:03.669 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:03.669 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.669 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.669 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:03.669 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.353321+0000", 2024-03-19T11:30:03.669 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3399999999999998e-07 2024-03-19T11:30:03.669 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.669 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.669 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:03.669 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.353336+0000", 2024-03-19T11:30:03.669 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4154999999999999e-05 2024-03-19T11:30:03.669 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.669 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.669 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:03.669 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.353349+0000", 2024-03-19T11:30:03.670 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3644e-05 2024-03-19T11:30:03.670 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.670 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.670 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:03.670 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.353357+0000", 2024-03-19T11:30:03.670 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5109999999999997e-06 2024-03-19T11:30:03.670 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.670 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.670 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:03.670 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.353409+0000", 2024-03-19T11:30:03.670 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2723999999999999e-05 2024-03-19T11:30:03.670 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.670 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.670 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:03.671 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.353428+0000", 2024-03-19T11:30:03.671 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8216e-05 2024-03-19T11:30:03.671 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.671 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.671 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:03.671 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.358359+0000", 2024-03-19T11:30:03.671 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0049314550000000004 2024-03-19T11:30:03.671 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.671 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.671 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:03.671 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.358392+0000", 2024-03-19T11:30:03.671 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3061000000000001e-05 2024-03-19T11:30:03.671 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:03.671 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:03.672 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:03.672 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.672 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.672 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:119 1.0 e16/14 1:30306672:devicehealth::main.db.0000000000000000:head v 16'81, mlcod=16'78)", 2024-03-19T11:30:03.672 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.364626+0000", 2024-03-19T11:30:03.672 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.292972292999998, 2024-03-19T11:30:03.672 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046261449999999999, 2024-03-19T11:30:03.672 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:03.672 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:03.672 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:03.672 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.672 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:03.672 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.364626+0000", 2024-03-19T11:30:03.672 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:03.673 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.673 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.673 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:03.673 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.364626+0000", 2024-03-19T11:30:03.673 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:03.673 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.673 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.673 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:03.673 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.364626+0000", 2024-03-19T11:30:03.673 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1399999999999996e-07 2024-03-19T11:30:03.673 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.673 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.673 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:03.673 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.364724+0000", 2024-03-19T11:30:03.674 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7972000000000003e-05 2024-03-19T11:30:03.674 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.674 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.674 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:03.674 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.364737+0000", 2024-03-19T11:30:03.674 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2585000000000001e-05 2024-03-19T11:30:03.674 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.674 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.674 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:03.674 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.364744+0000", 2024-03-19T11:30:03.674 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7780000000000002e-06 2024-03-19T11:30:03.674 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.674 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.674 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:03.675 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.364780+0000", 2024-03-19T11:30:03.675 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5992999999999999e-05 2024-03-19T11:30:03.675 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.675 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.675 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:03.675 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.364795+0000", 2024-03-19T11:30:03.675 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4939e-05 2024-03-19T11:30:03.675 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.675 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.675 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:03.675 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.369211+0000", 2024-03-19T11:30:03.675 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0044161139999999996 2024-03-19T11:30:03.675 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.675 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.675 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:03.676 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.369252+0000", 2024-03-19T11:30:03.676 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1050000000000002e-05 2024-03-19T11:30:03.676 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:03.676 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:03.676 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:03.676 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.676 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.676 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:120 1.0 e16/14 1:30306672:devicehealth::main.db.0000000000000000:head v 16'82, mlcod=16'78)", 2024-03-19T11:30:03.676 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.365811+0000", 2024-03-19T11:30:03.676 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.291787335999999, 2024-03-19T11:30:03.676 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0040843659999999999, 2024-03-19T11:30:03.676 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:03.676 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:03.676 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:03.676 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.677 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:03.677 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.365811+0000", 2024-03-19T11:30:03.677 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:03.677 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.677 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.677 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:03.677 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.365811+0000", 2024-03-19T11:30:03.677 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:03.677 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.677 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.677 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:03.677 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.365811+0000", 2024-03-19T11:30:03.677 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4199999999999995e-07 2024-03-19T11:30:03.677 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.677 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.678 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:03.678 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.365884+0000", 2024-03-19T11:30:03.678 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2527999999999994e-05 2024-03-19T11:30:03.678 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.678 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.678 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:03.678 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.365896+0000", 2024-03-19T11:30:03.678 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1912999999999999e-05 2024-03-19T11:30:03.678 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.678 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.678 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:03.678 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.365902+0000", 2024-03-19T11:30:03.679 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3130000000000004e-06 2024-03-19T11:30:03.679 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.679 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.679 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:03.679 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.365952+0000", 2024-03-19T11:30:03.679 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0283e-05 2024-03-19T11:30:03.679 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.679 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.679 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:03.679 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.365966+0000", 2024-03-19T11:30:03.679 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.349e-05 2024-03-19T11:30:03.679 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.679 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.679 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:03.679 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.369869+0000", 2024-03-19T11:30:03.680 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0039034989999999999 2024-03-19T11:30:03.680 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.680 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.680 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:03.680 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.369895+0000", 2024-03-19T11:30:03.680 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5598000000000001e-05 2024-03-19T11:30:03.680 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:03.680 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:03.680 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:03.680 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.680 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.680 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:129 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'87, mlcod=16'85)", 2024-03-19T11:30:03.680 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.805130+0000", 2024-03-19T11:30:03.680 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 17.852467697000002, 2024-03-19T11:30:03.681 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003788769, 2024-03-19T11:30:03.681 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:03.681 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:03.681 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:03.681 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.681 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:03.681 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.805130+0000", 2024-03-19T11:30:03.681 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:03.681 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.681 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.681 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:03.681 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.805130+0000", 2024-03-19T11:30:03.681 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:03.682 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.682 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.682 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:03.682 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.805132+0000", 2024-03-19T11:30:03.682 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.733e-06 2024-03-19T11:30:03.682 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.682 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.682 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:03.682 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.805145+0000", 2024-03-19T11:30:03.682 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2830000000000001e-05 2024-03-19T11:30:03.682 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.682 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.682 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:03.682 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.805173+0000", 2024-03-19T11:30:03.683 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.853e-05 2024-03-19T11:30:03.683 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.683 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.683 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:03.683 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.805187+0000", 2024-03-19T11:30:03.683 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3183000000000001e-05 2024-03-19T11:30:03.683 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.683 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.683 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:03.683 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.805329+0000", 2024-03-19T11:30:03.683 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000142578 2024-03-19T11:30:03.683 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.683 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.683 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:03.684 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.805366+0000", 2024-03-19T11:30:03.684 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6390000000000002e-05 2024-03-19T11:30:03.684 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.684 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.684 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:03.684 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808820+0000", 2024-03-19T11:30:03.684 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034547459999999999 2024-03-19T11:30:03.684 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.684 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.684 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:03.684 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808919+0000", 2024-03-19T11:30:03.684 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8778999999999997e-05 2024-03-19T11:30:03.684 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:03.684 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:03.685 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:03.685 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.685 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.685 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:130 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'88, mlcod=16'85)", 2024-03-19T11:30:03.685 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.805586+0000", 2024-03-19T11:30:03.685 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 17.852012267999999, 2024-03-19T11:30:03.685 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0054104690000000002, 2024-03-19T11:30:03.685 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:03.685 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:03.685 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:03.685 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.685 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:03.685 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.805586+0000", 2024-03-19T11:30:03.686 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:03.686 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.686 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.686 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:03.686 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.805586+0000", 2024-03-19T11:30:03.686 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:03.686 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.686 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.686 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:03.686 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.805587+0000", 2024-03-19T11:30:03.686 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7269999999999999e-06 2024-03-19T11:30:03.686 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.686 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.686 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:03.687 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.805600+0000", 2024-03-19T11:30:03.687 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2293000000000001e-05 2024-03-19T11:30:03.687 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.687 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.687 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:03.687 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.805623+0000", 2024-03-19T11:30:03.687 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3592999999999999e-05 2024-03-19T11:30:03.687 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.687 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.687 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:03.687 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.805636+0000", 2024-03-19T11:30:03.687 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2660000000000001e-05 2024-03-19T11:30:03.687 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.687 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.688 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:03.688 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.805838+0000", 2024-03-19T11:30:03.688 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020228800000000001 2024-03-19T11:30:03.688 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.688 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.688 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:03.688 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.805878+0000", 2024-03-19T11:30:03.688 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.9586999999999999e-05 2024-03-19T11:30:03.688 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.688 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.688 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:03.688 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.810908+0000", 2024-03-19T11:30:03.688 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050303090000000002 2024-03-19T11:30:03.688 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.689 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.689 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:03.689 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.810996+0000", 2024-03-19T11:30:03.689 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8011999999999997e-05 2024-03-19T11:30:03.689 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:03.689 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:03.689 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:03.689 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.689 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.689 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:131 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'89, mlcod=16'85)", 2024-03-19T11:30:03.689 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.806096+0000", 2024-03-19T11:30:03.689 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 17.851501619, 2024-03-19T11:30:03.690 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0049616699999999996, 2024-03-19T11:30:03.690 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:03.690 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:03.690 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:03.690 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.690 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:03.690 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806096+0000", 2024-03-19T11:30:03.690 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:03.690 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.690 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.691 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:03.691 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806096+0000", 2024-03-19T11:30:03.691 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:03.691 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.691 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.691 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:03.691 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806098+0000", 2024-03-19T11:30:03.691 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.734e-06 2024-03-19T11:30:03.691 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.691 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.691 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:03.691 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806267+0000", 2024-03-19T11:30:03.691 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016936800000000001 2024-03-19T11:30:03.691 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.692 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.692 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:03.692 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806296+0000", 2024-03-19T11:30:03.692 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8653000000000002e-05 2024-03-19T11:30:03.692 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.692 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.692 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:03.692 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806315+0000", 2024-03-19T11:30:03.692 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9049999999999999e-05 2024-03-19T11:30:03.692 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.692 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.692 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:03.692 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806481+0000", 2024-03-19T11:30:03.692 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016540599999999999 2024-03-19T11:30:03.693 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.693 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.693 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:03.693 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806511+0000", 2024-03-19T11:30:03.693 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0626000000000001e-05 2024-03-19T11:30:03.693 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.693 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.693 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:03.693 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.811002+0000", 2024-03-19T11:30:03.693 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0044912399999999996 2024-03-19T11:30:03.693 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.693 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.693 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:03.694 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.811058+0000", 2024-03-19T11:30:03.694 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5593000000000001e-05 2024-03-19T11:30:03.694 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:03.694 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:03.694 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:03.694 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.694 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.694 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:132 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'90, mlcod=16'85)", 2024-03-19T11:30:03.694 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.806470+0000", 2024-03-19T11:30:03.694 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 17.851128127999999, 2024-03-19T11:30:03.694 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0066812390000000003, 2024-03-19T11:30:03.694 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:03.694 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:03.694 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:03.695 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.695 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:03.695 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806470+0000", 2024-03-19T11:30:03.695 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:03.695 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.695 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.695 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:03.695 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806470+0000", 2024-03-19T11:30:03.695 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:03.695 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.695 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.695 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:03.695 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806471+0000", 2024-03-19T11:30:03.695 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.486e-06 2024-03-19T11:30:03.695 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.696 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.696 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:03.696 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806485+0000", 2024-03-19T11:30:03.696 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3380000000000001e-05 2024-03-19T11:30:03.696 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.696 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.696 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:03.696 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806509+0000", 2024-03-19T11:30:03.696 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4617000000000002e-05 2024-03-19T11:30:03.696 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.696 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.696 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:03.696 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806524+0000", 2024-03-19T11:30:03.697 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4688999999999999e-05 2024-03-19T11:30:03.697 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.697 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.697 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:03.697 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807070+0000", 2024-03-19T11:30:03.697 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00054601300000000003 2024-03-19T11:30:03.697 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.697 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.697 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:03.697 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807104+0000", 2024-03-19T11:30:03.697 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3575999999999998e-05 2024-03-19T11:30:03.698 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.698 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.698 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:03.698 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813077+0000", 2024-03-19T11:30:03.698 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0059733590000000001 2024-03-19T11:30:03.698 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.698 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.698 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:03.698 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813151+0000", 2024-03-19T11:30:03.698 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4119e-05 2024-03-19T11:30:03.698 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:03.698 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:03.698 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:03.698 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.699 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.699 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:133 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'91, mlcod=16'85)", 2024-03-19T11:30:03.699 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.806875+0000", 2024-03-19T11:30:03.699 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 17.850722465, 2024-03-19T11:30:03.699 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0063196260000000001, 2024-03-19T11:30:03.699 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:03.699 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:03.699 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:03.699 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.699 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:03.699 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806875+0000", 2024-03-19T11:30:03.699 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:03.699 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.699 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.700 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:03.700 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806875+0000", 2024-03-19T11:30:03.700 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:03.700 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.700 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.700 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:03.700 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806877+0000", 2024-03-19T11:30:03.700 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8899999999999999e-06 2024-03-19T11:30:03.700 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.700 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.700 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:03.700 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806895+0000", 2024-03-19T11:30:03.700 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7980000000000001e-05 2024-03-19T11:30:03.700 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.700 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.701 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:03.701 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806926+0000", 2024-03-19T11:30:03.701 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0363000000000001e-05 2024-03-19T11:30:03.701 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.701 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.701 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:03.701 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806942+0000", 2024-03-19T11:30:03.701 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6422999999999998e-05 2024-03-19T11:30:03.701 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.701 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.701 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:03.701 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807659+0000", 2024-03-19T11:30:03.701 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00071724699999999996 2024-03-19T11:30:03.701 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.702 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.702 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:03.702 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807688+0000", 2024-03-19T11:30:03.702 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8940000000000001e-05 2024-03-19T11:30:03.702 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.702 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.702 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:03.702 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813157+0000", 2024-03-19T11:30:03.702 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0054689930000000001 2024-03-19T11:30:03.702 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.702 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.702 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:03.702 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813195+0000", 2024-03-19T11:30:03.702 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7790000000000002e-05 2024-03-19T11:30:03.703 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:03.703 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:03.703 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:03.703 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.703 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.703 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:134 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'92, mlcod=16'85)", 2024-03-19T11:30:03.703 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.807418+0000", 2024-03-19T11:30:03.703 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 17.850179943000001, 2024-03-19T11:30:03.703 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0058240560000000002, 2024-03-19T11:30:03.703 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:03.703 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:03.703 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:03.703 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.703 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:03.703 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807418+0000", 2024-03-19T11:30:03.704 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:03.704 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.704 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.704 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:03.704 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807418+0000", 2024-03-19T11:30:03.704 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:03.704 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.704 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.704 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:03.704 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807420+0000", 2024-03-19T11:30:03.704 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5600000000000001e-06 2024-03-19T11:30:03.704 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.704 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.704 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:03.704 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807514+0000", 2024-03-19T11:30:03.705 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4256e-05 2024-03-19T11:30:03.705 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.705 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.705 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:03.705 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807542+0000", 2024-03-19T11:30:03.705 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8283e-05 2024-03-19T11:30:03.705 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.705 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.705 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:03.705 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807562+0000", 2024-03-19T11:30:03.705 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0126999999999999e-05 2024-03-19T11:30:03.705 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.705 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.705 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:03.705 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808272+0000", 2024-03-19T11:30:03.706 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00071012400000000002 2024-03-19T11:30:03.706 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.706 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.706 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:03.706 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808308+0000", 2024-03-19T11:30:03.706 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5722999999999997e-05 2024-03-19T11:30:03.706 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.706 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.706 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:03.706 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813200+0000", 2024-03-19T11:30:03.706 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0048915010000000004 2024-03-19T11:30:03.706 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.706 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.706 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:03.707 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813242+0000", 2024-03-19T11:30:03.707 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2481999999999997e-05 2024-03-19T11:30:03.707 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:03.707 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:03.707 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:03.707 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.707 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.707 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:135 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'93, mlcod=16'85)", 2024-03-19T11:30:03.707 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.807817+0000", 2024-03-19T11:30:03.707 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 17.849780706000001, 2024-03-19T11:30:03.707 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0054653059999999996, 2024-03-19T11:30:03.707 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:03.707 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:03.707 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:03.707 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.708 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:03.708 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807817+0000", 2024-03-19T11:30:03.708 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:03.708 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.708 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.708 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:03.708 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807817+0000", 2024-03-19T11:30:03.708 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:03.708 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.708 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.708 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:03.708 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807819+0000", 2024-03-19T11:30:03.708 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7999999999999999e-06 2024-03-19T11:30:03.709 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.709 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.709 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:03.709 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807833+0000", 2024-03-19T11:30:03.709 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3546999999999999e-05 2024-03-19T11:30:03.709 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.709 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.709 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:03.709 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807858+0000", 2024-03-19T11:30:03.709 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5119000000000001e-05 2024-03-19T11:30:03.709 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.709 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.709 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:03.709 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807871+0000", 2024-03-19T11:30:03.709 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.287e-05 2024-03-19T11:30:03.710 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.710 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.710 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:03.710 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808989+0000", 2024-03-19T11:30:03.710 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001118841 2024-03-19T11:30:03.710 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.710 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.710 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:03.710 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.809022+0000", 2024-03-19T11:30:03.710 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3000000000000003e-05 2024-03-19T11:30:03.710 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.710 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.710 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:03.710 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813249+0000", 2024-03-19T11:30:03.711 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0042268260000000004 2024-03-19T11:30:03.711 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.711 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.711 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:03.711 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813283+0000", 2024-03-19T11:30:03.711 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3303000000000003e-05 2024-03-19T11:30:03.711 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:03.711 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:03.711 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:03.711 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.711 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.711 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:136 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'94, mlcod=16'85)", 2024-03-19T11:30:03.711 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.808219+0000", 2024-03-19T11:30:03.711 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 17.849379425999999, 2024-03-19T11:30:03.712 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005105086, 2024-03-19T11:30:03.712 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:03.712 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:03.712 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:03.712 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.712 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:03.712 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808219+0000", 2024-03-19T11:30:03.712 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:03.712 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.712 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.712 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:03.712 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808219+0000", 2024-03-19T11:30:03.712 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:03.712 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.713 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.713 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:03.713 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808220+0000", 2024-03-19T11:30:03.713 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7600000000000001e-06 2024-03-19T11:30:03.713 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.713 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.713 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:03.713 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808233+0000", 2024-03-19T11:30:03.713 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2916e-05 2024-03-19T11:30:03.713 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.713 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.713 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:03.713 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808294+0000", 2024-03-19T11:30:03.713 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1199999999999997e-05 2024-03-19T11:30:03.713 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.714 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.714 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:03.714 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808310+0000", 2024-03-19T11:30:03.714 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5996999999999999e-05 2024-03-19T11:30:03.714 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.714 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.714 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:03.714 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.809445+0000", 2024-03-19T11:30:03.714 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001134227 2024-03-19T11:30:03.714 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.714 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.714 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:03.714 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.809473+0000", 2024-03-19T11:30:03.714 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8603e-05 2024-03-19T11:30:03.715 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.715 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.715 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:03.715 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813291+0000", 2024-03-19T11:30:03.715 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038174799999999998 2024-03-19T11:30:03.715 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.715 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.715 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:03.715 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813324+0000", 2024-03-19T11:30:03.715 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2903e-05 2024-03-19T11:30:03.715 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:03.715 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:03.715 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:03.715 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.716 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.716 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:137 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'95, mlcod=16'85)", 2024-03-19T11:30:03.716 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.808663+0000", 2024-03-19T11:30:03.716 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 17.848935272999999, 2024-03-19T11:30:03.716 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0047004350000000002, 2024-03-19T11:30:03.716 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:03.716 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:03.716 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:03.716 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.716 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:03.716 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808663+0000", 2024-03-19T11:30:03.716 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:03.716 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.716 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.716 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:03.716 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808663+0000", 2024-03-19T11:30:03.717 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:03.717 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.717 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.717 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:03.717 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808665+0000", 2024-03-19T11:30:03.717 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9300000000000002e-06 2024-03-19T11:30:03.717 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.717 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.717 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:03.717 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808768+0000", 2024-03-19T11:30:03.717 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000103009 2024-03-19T11:30:03.717 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.717 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.717 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:03.717 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808792+0000", 2024-03-19T11:30:03.718 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3972999999999999e-05 2024-03-19T11:30:03.718 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.718 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.718 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:03.718 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808807+0000", 2024-03-19T11:30:03.718 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5614e-05 2024-03-19T11:30:03.718 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.718 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.718 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:03.718 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.809896+0000", 2024-03-19T11:30:03.718 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0010891270000000001 2024-03-19T11:30:03.718 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.718 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.718 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:03.718 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.809930+0000", 2024-03-19T11:30:03.719 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3444e-05 2024-03-19T11:30:03.719 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.719 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.719 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:03.719 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813330+0000", 2024-03-19T11:30:03.719 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003399839 2024-03-19T11:30:03.719 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.719 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.719 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:03.719 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813363+0000", 2024-03-19T11:30:03.719 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3498999999999999e-05 2024-03-19T11:30:03.719 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:03.719 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:03.719 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:03.720 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.720 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.720 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:140 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'98, mlcod=16'86)", 2024-03-19T11:30:03.720 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.811388+0000", 2024-03-19T11:30:03.720 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 17.846209542, 2024-03-19T11:30:03.720 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033436199999999998, 2024-03-19T11:30:03.720 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:03.720 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:03.720 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:03.720 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.720 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:03.720 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.811388+0000", 2024-03-19T11:30:03.720 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:03.720 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.720 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.721 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:03.721 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.811388+0000", 2024-03-19T11:30:03.721 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:03.721 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.721 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.721 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:03.721 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.811396+0000", 2024-03-19T11:30:03.721 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5769999999999996e-06 2024-03-19T11:30:03.721 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.721 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.721 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:03.721 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.811462+0000", 2024-03-19T11:30:03.721 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5820000000000003e-05 2024-03-19T11:30:03.722 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.722 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.722 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:03.722 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.811491+0000", 2024-03-19T11:30:03.722 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8979000000000002e-05 2024-03-19T11:30:03.722 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.722 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.722 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:03.722 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.811508+0000", 2024-03-19T11:30:03.722 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.721e-05 2024-03-19T11:30:03.722 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.722 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.722 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:03.722 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.812088+0000", 2024-03-19T11:30:03.723 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00057991599999999998 2024-03-19T11:30:03.723 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.723 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.723 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:03.723 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.812121+0000", 2024-03-19T11:30:03.723 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2836000000000002e-05 2024-03-19T11:30:03.723 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.723 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.723 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:03.723 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.814640+0000", 2024-03-19T11:30:03.723 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025190329999999999 2024-03-19T11:30:03.723 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.723 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.723 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:03.724 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.814732+0000", 2024-03-19T11:30:03.724 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2249000000000004e-05 2024-03-19T11:30:03.724 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:03.724 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:03.724 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:03.724 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.724 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.724 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:141 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'99, mlcod=16'86)", 2024-03-19T11:30:03.724 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.812015+0000", 2024-03-19T11:30:03.724 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 17.845583247, 2024-03-19T11:30:03.724 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003849239, 2024-03-19T11:30:03.724 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:03.724 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:03.724 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:03.724 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.725 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:03.725 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.812015+0000", 2024-03-19T11:30:03.725 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:03.725 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.725 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.725 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:03.725 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.812015+0000", 2024-03-19T11:30:03.725 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:03.725 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.725 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.725 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:03.725 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.812017+0000", 2024-03-19T11:30:03.725 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8470000000000001e-06 2024-03-19T11:30:03.725 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.725 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.726 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:03.726 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.812030+0000", 2024-03-19T11:30:03.726 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3867e-05 2024-03-19T11:30:03.726 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.726 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.726 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:03.726 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.812058+0000", 2024-03-19T11:30:03.726 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8005999999999999e-05 2024-03-19T11:30:03.726 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.726 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.726 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:03.726 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.812070+0000", 2024-03-19T11:30:03.726 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1207e-05 2024-03-19T11:30:03.726 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.726 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.727 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:03.727 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813509+0000", 2024-03-19T11:30:03.727 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014389909999999999 2024-03-19T11:30:03.727 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.727 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.727 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:03.727 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813564+0000", 2024-03-19T11:30:03.727 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5420000000000001e-05 2024-03-19T11:30:03.727 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.727 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.727 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:03.727 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.815782+0000", 2024-03-19T11:30:03.727 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022183419999999999 2024-03-19T11:30:03.727 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.727 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.728 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:03.728 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.815864+0000", 2024-03-19T11:30:03.728 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1558999999999996e-05 2024-03-19T11:30:03.728 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:03.728 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:03.728 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:03.728 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.728 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.728 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:142 1.0 e16/14 1:30306672:devicehealth::main.db.0000000000000000:head v 16'100, mlcod=16'98)", 2024-03-19T11:30:03.728 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.817085+0000", 2024-03-19T11:30:03.728 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 17.840512898, 2024-03-19T11:30:03.728 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0035837740000000001, 2024-03-19T11:30:03.728 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:03.728 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:03.729 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:03.729 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.729 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:03.729 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.817085+0000", 2024-03-19T11:30:03.729 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:03.729 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.729 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.729 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:03.729 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.817085+0000", 2024-03-19T11:30:03.729 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:03.729 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.729 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.729 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:03.729 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.817092+0000", 2024-03-19T11:30:03.729 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.46e-06 2024-03-19T11:30:03.730 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.730 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.730 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:03.730 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.817365+0000", 2024-03-19T11:30:03.730 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00027311099999999998 2024-03-19T11:30:03.730 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.730 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.730 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:03.730 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.817390+0000", 2024-03-19T11:30:03.730 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.55e-05 2024-03-19T11:30:03.730 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.731 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.731 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:03.731 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.817403+0000", 2024-03-19T11:30:03.731 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2680000000000001e-05 2024-03-19T11:30:03.731 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.731 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.731 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:03.731 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.817521+0000", 2024-03-19T11:30:03.731 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000117862 2024-03-19T11:30:03.731 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.731 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.731 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:03.731 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.817558+0000", 2024-03-19T11:30:03.731 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7280000000000002e-05 2024-03-19T11:30:03.731 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.732 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.732 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:03.732 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.820599+0000", 2024-03-19T11:30:03.732 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030407780000000001 2024-03-19T11:30:03.732 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.732 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.732 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:03.732 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.820669+0000", 2024-03-19T11:30:03.732 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.0103000000000003e-05 2024-03-19T11:30:03.732 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:03.732 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:03.732 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:03.732 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.732 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.732 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:143 1.0 e16/14 1:30306672:devicehealth::main.db.0000000000000000:head v 16'101, mlcod=16'98)", 2024-03-19T11:30:03.733 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.817553+0000", 2024-03-19T11:30:03.733 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 17.840045448000001, 2024-03-19T11:30:03.733 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031569139999999998, 2024-03-19T11:30:03.733 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:03.733 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:03.733 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:03.733 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.733 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:03.733 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.817553+0000", 2024-03-19T11:30:03.733 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:03.733 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.733 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.733 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:03.733 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.817553+0000", 2024-03-19T11:30:03.734 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:03.734 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.734 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.734 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:03.734 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.817554+0000", 2024-03-19T11:30:03.734 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.026e-06 2024-03-19T11:30:03.734 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.734 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.734 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:03.734 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.817691+0000", 2024-03-19T11:30:03.734 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013721600000000001 2024-03-19T11:30:03.735 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.735 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.735 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:03.735 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.817712+0000", 2024-03-19T11:30:03.735 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0913000000000001e-05 2024-03-19T11:30:03.735 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.735 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.735 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:03.735 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.817724+0000", 2024-03-19T11:30:03.735 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2363e-05 2024-03-19T11:30:03.735 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.735 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.735 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:03.735 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.818276+0000", 2024-03-19T11:30:03.735 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00055174200000000003 2024-03-19T11:30:03.736 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.736 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.736 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:03.736 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.818314+0000", 2024-03-19T11:30:03.736 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8367e-05 2024-03-19T11:30:03.736 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.736 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.736 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:03.736 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.820673+0000", 2024-03-19T11:30:03.736 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023588670000000002 2024-03-19T11:30:03.736 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:03.736 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:03.736 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:03.737 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.820709+0000", 2024-03-19T11:30:03.737 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.642e-05 2024-03-19T11:30:03.737 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:03.737 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:03.737 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:03.737 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:03.737 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:03.737 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:03.738 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:03.656+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:03.739 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-19T11:30:03.748 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:03.760+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:03.849 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:03.860+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:03.912 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:03.912 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:30:03.912 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:30:03.912 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:03.927 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-19T11:30:03.974 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:03.984+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:04.052 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:04.060+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:04.052 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:04.053 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:30:04.053 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:30:04.053 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:30:04.053 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:04.067 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-19T11:30:04.154 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:04.167+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:04.238 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:04.238 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:30:04.238 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:30:04.238 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-03-19T11:30:04.238 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:04.253 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-19T11:30:04.254 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:04.264+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:04.356 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:04.364+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:04.376 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:04.376 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:30:04.376 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:30:04.376 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:04.390 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-19T11:30:04.457 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:04.471+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:04.558 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:04.567+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:04.563 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:04.564 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:30:04.564 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:30:04.564 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:30:04.564 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:04.573 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-19T11:30:04.578 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-19T11:30:04.625 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-03-19T11:30:04.648 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:04.656+0000 7f6712f92640 -1 osd.2 27 handle_osd_map injecting map crc failure 2024-03-19T11:30:04.650 INFO:tasks.rados.rados.0.smithi038.stdout:1: write initial oid 1 2024-03-19T11:30:04.650 INFO:tasks.rados.rados.0.smithi038.stdout:1: seq_num 0 ranges {416807=475249,1542129=474678} 2024-03-19T11:30:04.657 INFO:tasks.rados.rados.0.smithi038.stdout:1: writing smithi03829856-1 from 416807 to 892056 tid 1 2024-03-19T11:30:04.658 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:04.667+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:04.666 INFO:tasks.rados.rados.0.smithi038.stdout:1: writing smithi03829856-1 from 1542129 to 2016807 tid 2 2024-03-19T11:30:04.666 INFO:tasks.rados.rados.0.smithi038.stdout:2: write initial oid 2 2024-03-19T11:30:04.666 INFO:tasks.rados.rados.0.smithi038.stdout:2: seq_num 1 ranges {416807=475249,1542129=474678} 2024-03-19T11:30:04.673 INFO:tasks.rados.rados.0.smithi038.stdout:2: writing smithi03829856-2 from 416807 to 892056 tid 1 2024-03-19T11:30:04.682 INFO:tasks.rados.rados.0.smithi038.stdout:2: writing smithi03829856-2 from 1542129 to 2016807 tid 2 2024-03-19T11:30:04.682 INFO:tasks.rados.rados.0.smithi038.stdout:3: write initial oid 3 2024-03-19T11:30:04.682 INFO:tasks.rados.rados.0.smithi038.stdout:3: seq_num 2 ranges {433614=550498,1600611=433003} 2024-03-19T11:30:04.689 INFO:tasks.rados.rados.0.smithi038.stdout:3: writing smithi03829856-3 from 433614 to 984112 tid 1 2024-03-19T11:30:04.697 INFO:tasks.rados.rados.0.smithi038.stdout:3: writing smithi03829856-3 from 1600611 to 2033614 tid 2 2024-03-19T11:30:04.697 INFO:tasks.rados.rados.0.smithi038.stdout:4: write initial oid 4 2024-03-19T11:30:04.697 INFO:tasks.rados.rados.0.smithi038.stdout:4: seq_num 3 ranges {450421=625747,1659093=391328} 2024-03-19T11:30:04.705 INFO:tasks.rados.rados.0.smithi038.stdout:4: writing smithi03829856-4 from 450421 to 1076168 tid 1 2024-03-19T11:30:04.711 INFO:tasks.rados.rados.0.smithi038.stdout:4: writing smithi03829856-4 from 1659093 to 2050421 tid 2 2024-03-19T11:30:04.712 INFO:tasks.rados.rados.0.smithi038.stdout:1: finishing write tid 1 to smithi03829856-1 2024-03-19T11:30:04.712 INFO:tasks.rados.rados.0.smithi038.stdout:1: finishing write tid 2 to smithi03829856-1 2024-03-19T11:30:04.712 INFO:tasks.rados.rados.0.smithi038.stdout:1: finishing write tid 3 to smithi03829856-1 2024-03-19T11:30:04.712 INFO:tasks.rados.rados.0.smithi038.stdout:1: finishing write tid 4 to smithi03829856-1 2024-03-19T11:30:04.712 INFO:tasks.rados.rados.0.smithi038.stdout:1: oid 1 updating version 0 to 1 2024-03-19T11:30:04.712 INFO:tasks.rados.rados.0.smithi038.stdout:1: oid 1 updating version 1 to 2 2024-03-19T11:30:04.712 INFO:tasks.rados.rados.0.smithi038.stdout:1: oid 1 updating version 2 to 3 2024-03-19T11:30:04.712 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-03-19T11:30:04.712 INFO:tasks.rados.rados.0.smithi038.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2024-03-19T11:30:04.712 INFO:tasks.rados.rados.0.smithi038.stdout:2: finishing write tid 1 to smithi03829856-2 2024-03-19T11:30:04.712 INFO:tasks.rados.rados.0.smithi038.stdout:2: finishing write tid 2 to smithi03829856-2 2024-03-19T11:30:04.712 INFO:tasks.rados.rados.0.smithi038.stdout:2: finishing write tid 3 to smithi03829856-2 2024-03-19T11:30:04.712 INFO:tasks.rados.rados.0.smithi038.stdout:2: finishing write tid 4 to smithi03829856-2 2024-03-19T11:30:04.713 INFO:tasks.rados.rados.0.smithi038.stdout:2: oid 2 updating version 0 to 3 2024-03-19T11:30:04.713 INFO:tasks.rados.rados.0.smithi038.stdout:2: oid 2 version 3 is already newer than 2 2024-03-19T11:30:04.713 INFO:tasks.rados.rados.0.smithi038.stdout:2: oid 2 version 3 is already newer than 1 2024-03-19T11:30:04.713 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 2 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-03-19T11:30:04.713 INFO:tasks.rados.rados.0.smithi038.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2024-03-19T11:30:04.713 INFO:tasks.rados.rados.0.smithi038.stdout:3: finishing write tid 1 to smithi03829856-3 2024-03-19T11:30:04.713 INFO:tasks.rados.rados.0.smithi038.stdout:3: finishing write tid 2 to smithi03829856-3 2024-03-19T11:30:04.713 INFO:tasks.rados.rados.0.smithi038.stdout:3: finishing write tid 3 to smithi03829856-3 2024-03-19T11:30:04.713 INFO:tasks.rados.rados.0.smithi038.stdout:3: finishing write tid 4 to smithi03829856-3 2024-03-19T11:30:04.713 INFO:tasks.rados.rados.0.smithi038.stdout:3: oid 3 updating version 0 to 2 2024-03-19T11:30:04.713 INFO:tasks.rados.rados.0.smithi038.stdout:3: oid 3 updating version 2 to 3 2024-03-19T11:30:04.713 INFO:tasks.rados.rados.0.smithi038.stdout:3: oid 3 version 3 is already newer than 1 2024-03-19T11:30:04.713 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 3 v 3 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-03-19T11:30:04.713 INFO:tasks.rados.rados.0.smithi038.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2024-03-19T11:30:04.714 INFO:tasks.rados.rados.0.smithi038.stdout:1: done (3 left) 2024-03-19T11:30:04.714 INFO:tasks.rados.rados.0.smithi038.stdout:2: done (2 left) 2024-03-19T11:30:04.714 INFO:tasks.rados.rados.0.smithi038.stdout:3: done (1 left) 2024-03-19T11:30:04.714 INFO:tasks.rados.rados.0.smithi038.stdout:5: write initial oid 5 2024-03-19T11:30:04.714 INFO:tasks.rados.rados.0.smithi038.stdout:5: seq_num 4 ranges {467228=700996,1717575=349653} 2024-03-19T11:30:04.719 INFO:tasks.rados.rados.0.smithi038.stdout:5: writing smithi03829856-5 from 467228 to 1168224 tid 1 2024-03-19T11:30:04.725 INFO:tasks.rados.rados.0.smithi038.stdout:5: writing smithi03829856-5 from 1717575 to 2067228 tid 2 2024-03-19T11:30:04.725 INFO:tasks.rados.rados.0.smithi038.stdout:6: write initial oid 6 2024-03-19T11:30:04.725 INFO:tasks.rados.rados.0.smithi038.stdout:6: seq_num 5 ranges {484035=776245,2059704=24331} 2024-03-19T11:30:04.732 INFO:tasks.rados.rados.0.smithi038.stdout:6: writing smithi03829856-6 from 484035 to 1260280 tid 1 2024-03-19T11:30:04.737 INFO:tasks.rados.rados.0.smithi038.stdout:6: writing smithi03829856-6 from 2059704 to 2084035 tid 2 2024-03-19T11:30:04.737 INFO:tasks.rados.rados.0.smithi038.stdout:4: finishing write tid 1 to smithi03829856-4 2024-03-19T11:30:04.737 INFO:tasks.rados.rados.0.smithi038.stdout:4: finishing write tid 2 to smithi03829856-4 2024-03-19T11:30:04.737 INFO:tasks.rados.rados.0.smithi038.stdout:4: finishing write tid 3 to smithi03829856-4 2024-03-19T11:30:04.737 INFO:tasks.rados.rados.0.smithi038.stdout:4: finishing write tid 4 to smithi03829856-4 2024-03-19T11:30:04.737 INFO:tasks.rados.rados.0.smithi038.stdout:4: oid 4 updating version 0 to 1 2024-03-19T11:30:04.738 INFO:tasks.rados.rados.0.smithi038.stdout:4: oid 4 updating version 1 to 2 2024-03-19T11:30:04.738 INFO:tasks.rados.rados.0.smithi038.stdout:4: oid 4 updating version 2 to 3 2024-03-19T11:30:04.738 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 4 v 3 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-03-19T11:30:04.738 INFO:tasks.rados.rados.0.smithi038.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2024-03-19T11:30:04.738 INFO:tasks.rados.rados.0.smithi038.stdout:5: finishing write tid 1 to smithi03829856-5 2024-03-19T11:30:04.738 INFO:tasks.rados.rados.0.smithi038.stdout:5: finishing write tid 2 to smithi03829856-5 2024-03-19T11:30:04.738 INFO:tasks.rados.rados.0.smithi038.stdout:5: finishing write tid 3 to smithi03829856-5 2024-03-19T11:30:04.738 INFO:tasks.rados.rados.0.smithi038.stdout:5: finishing write tid 4 to smithi03829856-5 2024-03-19T11:30:04.738 INFO:tasks.rados.rados.0.smithi038.stdout:5: oid 5 updating version 0 to 2 2024-03-19T11:30:04.738 INFO:tasks.rados.rados.0.smithi038.stdout:5: oid 5 updating version 2 to 3 2024-03-19T11:30:04.738 INFO:tasks.rados.rados.0.smithi038.stdout:5: oid 5 version 3 is already newer than 1 2024-03-19T11:30:04.738 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 5 v 3 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-03-19T11:30:04.738 INFO:tasks.rados.rados.0.smithi038.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2024-03-19T11:30:04.738 INFO:tasks.rados.rados.0.smithi038.stdout:4: done (2 left) 2024-03-19T11:30:04.739 INFO:tasks.rados.rados.0.smithi038.stdout:5: done (1 left) 2024-03-19T11:30:04.739 INFO:tasks.rados.rados.0.smithi038.stdout:7: write initial oid 7 2024-03-19T11:30:04.739 INFO:tasks.rados.rados.0.smithi038.stdout:7: seq_num 6 ranges {500842=451494,1718186=382656} 2024-03-19T11:30:04.743 INFO:tasks.rados.rados.0.smithi038.stdout:7: writing smithi03829856-7 from 500842 to 952336 tid 1 2024-03-19T11:30:04.750 INFO:tasks.rados.rados.0.smithi038.stdout:7: writing smithi03829856-7 from 1718186 to 2100842 tid 2 2024-03-19T11:30:04.750 INFO:tasks.rados.rados.0.smithi038.stdout:8: write initial oid 8 2024-03-19T11:30:04.750 INFO:tasks.rados.rados.0.smithi038.stdout:8: seq_num 7 ranges {517649=526743,1776668=340981} 2024-03-19T11:30:04.754 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:30:04.754 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:30:04.756 INFO:tasks.rados.rados.0.smithi038.stdout:8: writing smithi03829856-8 from 517649 to 1044392 tid 1 2024-03-19T11:30:04.759 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:04.767+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:04.762 INFO:tasks.rados.rados.0.smithi038.stdout:8: writing smithi03829856-8 from 1776668 to 2117649 tid 2 2024-03-19T11:30:04.762 INFO:tasks.rados.rados.0.smithi038.stdout:9: write initial oid 9 2024-03-19T11:30:04.763 INFO:tasks.rados.rados.0.smithi038.stdout:9: seq_num 8 ranges {534456=718345,1951503=182953} 2024-03-19T11:30:04.769 INFO:tasks.rados.rados.0.smithi038.stdout:9: writing smithi03829856-9 from 534456 to 1252801 tid 1 2024-03-19T11:30:04.775 INFO:tasks.rados.rados.0.smithi038.stdout:9: writing smithi03829856-9 from 1951503 to 2134456 tid 2 2024-03-19T11:30:04.775 INFO:tasks.rados.rados.0.smithi038.stdout:6: finishing write tid 1 to smithi03829856-6 2024-03-19T11:30:04.775 INFO:tasks.rados.rados.0.smithi038.stdout:6: finishing write tid 2 to smithi03829856-6 2024-03-19T11:30:04.775 INFO:tasks.rados.rados.0.smithi038.stdout:6: finishing write tid 3 to smithi03829856-6 2024-03-19T11:30:04.775 INFO:tasks.rados.rados.0.smithi038.stdout:6: finishing write tid 4 to smithi03829856-6 2024-03-19T11:30:04.775 INFO:tasks.rados.rados.0.smithi038.stdout:6: oid 6 updating version 0 to 3 2024-03-19T11:30:04.775 INFO:tasks.rados.rados.0.smithi038.stdout:6: oid 6 version 3 is already newer than 1 2024-03-19T11:30:04.775 INFO:tasks.rados.rados.0.smithi038.stdout:6: oid 6 version 3 is already newer than 2 2024-03-19T11:30:04.775 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 6 v 3 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-03-19T11:30:04.775 INFO:tasks.rados.rados.0.smithi038.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2024-03-19T11:30:04.775 INFO:tasks.rados.rados.0.smithi038.stdout:7: finishing write tid 1 to smithi03829856-7 2024-03-19T11:30:04.775 INFO:tasks.rados.rados.0.smithi038.stdout:7: finishing write tid 2 to smithi03829856-7 2024-03-19T11:30:04.775 INFO:tasks.rados.rados.0.smithi038.stdout:7: finishing write tid 3 to smithi03829856-7 2024-03-19T11:30:04.775 INFO:tasks.rados.rados.0.smithi038.stdout:7: finishing write tid 4 to smithi03829856-7 2024-03-19T11:30:04.776 INFO:tasks.rados.rados.0.smithi038.stdout:7: oid 7 updating version 0 to 3 2024-03-19T11:30:04.776 INFO:tasks.rados.rados.0.smithi038.stdout:7: oid 7 version 3 is already newer than 2 2024-03-19T11:30:04.776 INFO:tasks.rados.rados.0.smithi038.stdout:7: oid 7 version 3 is already newer than 1 2024-03-19T11:30:04.776 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 7 v 3 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-03-19T11:30:04.776 INFO:tasks.rados.rados.0.smithi038.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2024-03-19T11:30:04.776 INFO:tasks.rados.rados.0.smithi038.stdout:8: finishing write tid 1 to smithi03829856-8 2024-03-19T11:30:04.776 INFO:tasks.rados.rados.0.smithi038.stdout:6: done (3 left) 2024-03-19T11:30:04.776 INFO:tasks.rados.rados.0.smithi038.stdout:7: done (2 left) 2024-03-19T11:30:04.776 INFO:tasks.rados.rados.0.smithi038.stdout:10: write initial oid 10 2024-03-19T11:30:04.776 INFO:tasks.rados.rados.0.smithi038.stdout:10: seq_num 9 ranges {551263=793594,1893632=257631} 2024-03-19T11:30:04.782 INFO:tasks.rados.rados.0.smithi038.stdout:10: writing smithi03829856-10 from 551263 to 1344857 tid 1 2024-03-19T11:30:04.787 INFO:tasks.rados.rados.0.smithi038.stdout:10: writing smithi03829856-10 from 1893632 to 2151263 tid 2 2024-03-19T11:30:04.788 INFO:tasks.rados.rados.0.smithi038.stdout:11: write initial oid 11 2024-03-19T11:30:04.788 INFO:tasks.rados.rados.0.smithi038.stdout:11: seq_num 10 ranges {568070=468843,1552114=615956} 2024-03-19T11:30:04.788 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:04.788 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:30:04.788 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:30:04.788 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [] 2024-03-19T11:30:04.789 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:04.789 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:30:04.789 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:30:04.794 INFO:tasks.rados.rados.0.smithi038.stdout:11: writing smithi03829856-11 from 568070 to 1036913 tid 1 2024-03-19T11:30:04.800 INFO:tasks.rados.rados.0.smithi038.stdout:11: writing smithi03829856-11 from 1552114 to 2168070 tid 2 2024-03-19T11:30:04.800 INFO:tasks.rados.rados.0.smithi038.stdout:12: write initial oid 12 2024-03-19T11:30:04.800 INFO:tasks.rados.rados.0.smithi038.stdout:12: seq_num 11 ranges {584877=544092,1610596=562003} 2024-03-19T11:30:04.807 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-19T11:30:04.808 INFO:tasks.rados.rados.0.smithi038.stdout:12: writing smithi03829856-12 from 584877 to 1128969 tid 1 2024-03-19T11:30:04.813 INFO:tasks.rados.rados.0.smithi038.stdout:12: writing smithi03829856-12 from 1610596 to 2172599 tid 2 2024-03-19T11:30:04.813 INFO:tasks.rados.rados.0.smithi038.stdout:13: write initial oid 13 2024-03-19T11:30:04.813 INFO:tasks.rados.rados.0.smithi038.stdout:13: seq_num 12 ranges {601684=619341,1669078=532606} 2024-03-19T11:30:04.819 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:30:04.819 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:30:04.820 INFO:tasks.rados.rados.0.smithi038.stdout:13: writing smithi03829856-13 from 601684 to 1221025 tid 1 2024-03-19T11:30:04.826 INFO:tasks.rados.rados.0.smithi038.stdout:13: writing smithi03829856-13 from 1669078 to 2201684 tid 2 2024-03-19T11:30:04.826 INFO:tasks.rados.rados.0.smithi038.stdout:14: write initial oid 14 2024-03-19T11:30:04.826 INFO:tasks.rados.rados.0.smithi038.stdout:14: seq_num 13 ranges {618491=694590,2011207=207284} 2024-03-19T11:30:04.833 INFO:tasks.rados.rados.0.smithi038.stdout:14: writing smithi03829856-14 from 618491 to 1313081 tid 1 2024-03-19T11:30:04.838 INFO:tasks.rados.rados.0.smithi038.stdout:14: writing smithi03829856-14 from 2011207 to 2218491 tid 2 2024-03-19T11:30:04.839 INFO:tasks.rados.rados.0.smithi038.stdout:15: write initial oid 15 2024-03-19T11:30:04.839 INFO:tasks.rados.rados.0.smithi038.stdout:15: seq_num 14 ranges {635298=769839,2069689=165609} 2024-03-19T11:30:04.846 INFO:tasks.rados.rados.0.smithi038.stdout:15: writing smithi03829856-15 from 635298 to 1405137 tid 1 2024-03-19T11:30:04.849 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:30:04.849 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:30:04.851 INFO:tasks.rados.rados.0.smithi038.stdout:15: writing smithi03829856-15 from 2069689 to 2235298 tid 2 2024-03-19T11:30:04.851 INFO:tasks.rados.rados.0.smithi038.stdout:16: write initial oid 16 2024-03-19T11:30:04.851 INFO:tasks.rados.rados.0.smithi038.stdout:16: seq_num 15 ranges {652105=445088,1728171=452988} 2024-03-19T11:30:04.858 INFO:tasks.rados.rados.0.smithi038.stdout:16: writing smithi03829856-16 from 652105 to 1097193 tid 1 2024-03-19T11:30:04.860 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:04.868+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:04.863 INFO:tasks.rados.rados.0.smithi038.stdout:16: writing smithi03829856-16 from 1728171 to 2181159 tid 2 2024-03-19T11:30:04.864 INFO:tasks.rados.rados.0.smithi038.stdout:17: write initial oid 17 2024-03-19T11:30:04.864 INFO:tasks.rados.rados.0.smithi038.stdout:17: seq_num 16 ranges {668912=636690,1903006=365906} 2024-03-19T11:30:04.871 INFO:tasks.rados.rados.0.smithi038.stdout:17: writing smithi03829856-17 from 668912 to 1305602 tid 1 2024-03-19T11:30:04.877 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:30:04.878 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:30:04.878 INFO:tasks.rados.rados.0.smithi038.stdout:17: writing smithi03829856-17 from 1903006 to 2268912 tid 2 2024-03-19T11:30:04.878 INFO:tasks.rados.rados.0.smithi038.stdout:18: write initial oid 18 2024-03-19T11:30:04.879 INFO:tasks.rados.rados.0.smithi038.stdout:18: seq_num 17 ranges {685719=711939,1845135=440584} 2024-03-19T11:30:04.884 INFO:tasks.rados.rados.0.smithi038.stdout:18: writing smithi03829856-18 from 685719 to 1397658 tid 1 2024-03-19T11:30:04.889 INFO:tasks.rados.rados.0.smithi038.stdout:18: writing smithi03829856-18 from 1845135 to 2285719 tid 2 2024-03-19T11:30:04.890 INFO:tasks.rados.rados.0.smithi038.stdout:19: write initial oid 19 2024-03-19T11:30:04.890 INFO:tasks.rados.rados.0.smithi038.stdout:19: seq_num 18 ranges {702526=787188,1903617=398909} 2024-03-19T11:30:04.897 INFO:tasks.rados.rados.0.smithi038.stdout:19: writing smithi03829856-19 from 702526 to 1489714 tid 1 2024-03-19T11:30:04.899 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:30:04.899 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:30:04.902 INFO:tasks.rados.rados.0.smithi038.stdout:19: writing smithi03829856-19 from 1903617 to 2302526 tid 2 2024-03-19T11:30:04.902 INFO:tasks.rados.rados.0.smithi038.stdout:20: write initial oid 20 2024-03-19T11:30:04.903 INFO:tasks.rados.rados.0.smithi038.stdout:20: seq_num 19 ranges {719333=462437,1962099=357234} 2024-03-19T11:30:04.909 INFO:tasks.rados.rados.0.smithi038.stdout:20: writing smithi03829856-20 from 719333 to 1181770 tid 1 2024-03-19T11:30:04.915 INFO:tasks.rados.rados.0.smithi038.stdout:20: writing smithi03829856-20 from 1962099 to 2319333 tid 2 2024-03-19T11:30:04.916 INFO:tasks.rados.rados.0.smithi038.stdout:21: write initial oid 21 2024-03-19T11:30:04.916 INFO:tasks.rados.rados.0.smithi038.stdout:21: seq_num 20 ranges {736140=537686,2020581=315559} 2024-03-19T11:30:04.923 INFO:tasks.rados.rados.0.smithi038.stdout:21: writing smithi03829856-21 from 736140 to 1273826 tid 1 2024-03-19T11:30:04.924 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:30:04.924 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:30:04.929 INFO:tasks.rados.rados.0.smithi038.stdout:21: writing smithi03829856-21 from 2020581 to 2336140 tid 2 2024-03-19T11:30:04.929 INFO:tasks.rados.rados.0.smithi038.stdout:22: write initial oid 22 2024-03-19T11:30:04.929 INFO:tasks.rados.rados.0.smithi038.stdout:22: seq_num 21 ranges {752947=612935,1962710=390237} 2024-03-19T11:30:04.936 INFO:tasks.rados.rados.0.smithi038.stdout:22: writing smithi03829856-22 from 752947 to 1365882 tid 1 2024-03-19T11:30:04.940 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:04.940 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:04.942 INFO:tasks.rados.rados.0.smithi038.stdout:22: writing smithi03829856-22 from 1962710 to 2352947 tid 2 2024-03-19T11:30:04.942 INFO:tasks.rados.rados.0.smithi038.stdout:23: write initial oid 23 2024-03-19T11:30:04.942 INFO:tasks.rados.rados.0.smithi038.stdout:23: seq_num 22 ranges {769754=688184,2021192=348562} 2024-03-19T11:30:04.946 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:30:04.946 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:30:04.950 INFO:tasks.rados.rados.0.smithi038.stdout:23: writing smithi03829856-23 from 769754 to 1457938 tid 1 2024-03-19T11:30:04.957 INFO:tasks.rados.rados.0.smithi038.stdout:23: writing smithi03829856-23 from 2021192 to 2369754 tid 2 2024-03-19T11:30:04.957 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:04.957 INFO:tasks.rados.rados.0.smithi038.stdout:8: finishing write tid 2 to smithi03829856-8 2024-03-19T11:30:04.957 INFO:tasks.rados.rados.0.smithi038.stdout:8: finishing write tid 3 to smithi03829856-8 2024-03-19T11:30:04.957 INFO:tasks.rados.rados.0.smithi038.stdout:9: finishing write tid 1 to smithi03829856-9 2024-03-19T11:30:04.958 INFO:tasks.rados.rados.0.smithi038.stdout:8: finishing write tid 4 to smithi03829856-8 2024-03-19T11:30:04.958 INFO:tasks.rados.rados.0.smithi038.stdout:8: oid 8 updating version 0 to 1 2024-03-19T11:30:04.958 INFO:tasks.rados.rados.0.smithi038.stdout:8: oid 8 updating version 1 to 3 2024-03-19T11:30:04.958 INFO:tasks.rados.rados.0.smithi038.stdout:8: oid 8 version 3 is already newer than 2 2024-03-19T11:30:04.958 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 8 v 3 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-03-19T11:30:04.958 INFO:tasks.rados.rados.0.smithi038.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2024-03-19T11:30:04.958 INFO:tasks.rados.rados.0.smithi038.stdout:9: finishing write tid 2 to smithi03829856-9 2024-03-19T11:30:04.958 INFO:tasks.rados.rados.0.smithi038.stdout:9: finishing write tid 3 to smithi03829856-9 2024-03-19T11:30:04.958 INFO:tasks.rados.rados.0.smithi038.stdout:9: finishing write tid 4 to smithi03829856-9 2024-03-19T11:30:04.958 INFO:tasks.rados.rados.0.smithi038.stdout:9: oid 9 updating version 0 to 5 2024-03-19T11:30:04.958 INFO:tasks.rados.rados.0.smithi038.stdout:9: oid 9 version 5 is already newer than 4 2024-03-19T11:30:04.958 INFO:tasks.rados.rados.0.smithi038.stdout:9: oid 9 updating version 5 to 6 2024-03-19T11:30:04.959 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 9 v 6 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-03-19T11:30:04.959 INFO:tasks.rados.rados.0.smithi038.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2024-03-19T11:30:04.959 INFO:tasks.rados.rados.0.smithi038.stdout:10: finishing write tid 1 to smithi03829856-10 2024-03-19T11:30:04.959 INFO:tasks.rados.rados.0.smithi038.stdout:10: finishing write tid 2 to smithi03829856-10 2024-03-19T11:30:04.959 INFO:tasks.rados.rados.0.smithi038.stdout:10: finishing write tid 3 to smithi03829856-10 2024-03-19T11:30:04.959 INFO:tasks.rados.rados.0.smithi038.stdout:8: done (15 left) 2024-03-19T11:30:04.959 INFO:tasks.rados.rados.0.smithi038.stdout:9: done (14 left) 2024-03-19T11:30:04.959 INFO:tasks.rados.rados.0.smithi038.stdout:24: write initial oid 24 2024-03-19T11:30:04.959 INFO:tasks.rados.rados.0.smithi038.stdout:24: seq_num 23 ranges {786561=479786,1796027=467664} 2024-03-19T11:30:04.960 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-19T11:30:04.962 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:04.971+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:04.963 INFO:tasks.rados.rados.0.smithi038.stdout:24: writing smithi03829856-24 from 786561 to 1266347 tid 1 2024-03-19T11:30:04.969 INFO:tasks.rados.rados.0.smithi038.stdout:24: writing smithi03829856-24 from 1796027 to 2263691 tid 2 2024-03-19T11:30:04.969 INFO:tasks.rados.rados.0.smithi038.stdout:25: write initial oid 25 2024-03-19T11:30:04.969 INFO:tasks.rados.rados.0.smithi038.stdout:25: seq_num 24 ranges {403368=555035,1454509=727675} 2024-03-19T11:30:04.975 INFO:tasks.rados.rados.0.smithi038.stdout:25: writing smithi03829856-25 from 403368 to 958403 tid 1 2024-03-19T11:30:04.982 INFO:tasks.rados.rados.0.smithi038.stdout:25: writing smithi03829856-25 from 1454509 to 2182184 tid 2 2024-03-19T11:30:04.982 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:04.982 INFO:tasks.rados.rados.0.smithi038.stdout:10: finishing write tid 4 to smithi03829856-10 2024-03-19T11:30:04.982 INFO:tasks.rados.rados.0.smithi038.stdout:10: oid 10 updating version 0 to 6 2024-03-19T11:30:04.982 INFO:tasks.rados.rados.0.smithi038.stdout:10: oid 10 version 6 is already newer than 5 2024-03-19T11:30:04.982 INFO:tasks.rados.rados.0.smithi038.stdout:10: oid 10 version 6 is already newer than 4 2024-03-19T11:30:04.982 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 10 v 6 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-03-19T11:30:04.983 INFO:tasks.rados.rados.0.smithi038.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2024-03-19T11:30:04.983 INFO:tasks.rados.rados.0.smithi038.stdout:11: finishing write tid 1 to smithi03829856-11 2024-03-19T11:30:04.983 INFO:tasks.rados.rados.0.smithi038.stdout:11: finishing write tid 2 to smithi03829856-11 2024-03-19T11:30:04.983 INFO:tasks.rados.rados.0.smithi038.stdout:10: done (15 left) 2024-03-19T11:30:04.983 INFO:tasks.rados.rados.0.smithi038.stdout:26: write initial oid 26 2024-03-19T11:30:04.983 INFO:tasks.rados.rados.0.smithi038.stdout:26: seq_num 25 ranges {420175=630284,1796638=471333} 2024-03-19T11:30:04.987 INFO:tasks.rados.rados.0.smithi038.stdout:26: writing smithi03829856-26 from 420175 to 1050459 tid 1 2024-03-19T11:30:04.995 INFO:tasks.rados.rados.0.smithi038.stdout:26: writing smithi03829856-26 from 1796638 to 2267971 tid 2 2024-03-19T11:30:04.995 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:04.995 INFO:tasks.rados.rados.0.smithi038.stdout:11: finishing write tid 3 to smithi03829856-11 2024-03-19T11:30:04.995 INFO:tasks.rados.rados.0.smithi038.stdout:11: finishing write tid 4 to smithi03829856-11 2024-03-19T11:30:04.995 INFO:tasks.rados.rados.0.smithi038.stdout:11: oid 11 updating version 0 to 3 2024-03-19T11:30:04.995 INFO:tasks.rados.rados.0.smithi038.stdout:11: oid 11 version 3 is already newer than 1 2024-03-19T11:30:04.995 INFO:tasks.rados.rados.0.smithi038.stdout:11: oid 11 version 3 is already newer than 2 2024-03-19T11:30:04.995 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 11 v 3 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-03-19T11:30:04.995 INFO:tasks.rados.rados.0.smithi038.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2024-03-19T11:30:04.995 INFO:tasks.rados.rados.0.smithi038.stdout:12: finishing write tid 1 to smithi03829856-12 2024-03-19T11:30:04.995 INFO:tasks.rados.rados.0.smithi038.stdout:12: finishing write tid 2 to smithi03829856-12 2024-03-19T11:30:04.996 INFO:tasks.rados.rados.0.smithi038.stdout:11: done (15 left) 2024-03-19T11:30:04.996 INFO:tasks.rados.rados.0.smithi038.stdout:27: write initial oid 27 2024-03-19T11:30:04.996 INFO:tasks.rados.rados.0.smithi038.stdout:27: seq_num 26 ranges {436982=705533,1855120=581862} 2024-03-19T11:30:05.001 INFO:tasks.rados.rados.0.smithi038.stdout:27: writing smithi03829856-27 from 436982 to 1142515 tid 1 2024-03-19T11:30:05.009 INFO:tasks.rados.rados.0.smithi038.stdout:27: writing smithi03829856-27 from 1855120 to 2436982 tid 2 2024-03-19T11:30:05.010 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.010 INFO:tasks.rados.rados.0.smithi038.stdout:12: finishing write tid 3 to smithi03829856-12 2024-03-19T11:30:05.010 INFO:tasks.rados.rados.0.smithi038.stdout:12: finishing write tid 4 to smithi03829856-12 2024-03-19T11:30:05.010 INFO:tasks.rados.rados.0.smithi038.stdout:12: oid 12 updating version 0 to 1 2024-03-19T11:30:05.010 INFO:tasks.rados.rados.0.smithi038.stdout:12: oid 12 updating version 1 to 3 2024-03-19T11:30:05.010 INFO:tasks.rados.rados.0.smithi038.stdout:12: oid 12 version 3 is already newer than 2 2024-03-19T11:30:05.010 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 12 v 3 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-03-19T11:30:05.010 INFO:tasks.rados.rados.0.smithi038.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2024-03-19T11:30:05.010 INFO:tasks.rados.rados.0.smithi038.stdout:13: finishing write tid 1 to smithi03829856-13 2024-03-19T11:30:05.010 INFO:tasks.rados.rados.0.smithi038.stdout:13: finishing write tid 2 to smithi03829856-13 2024-03-19T11:30:05.010 INFO:tasks.rados.rados.0.smithi038.stdout:13: finishing write tid 3 to smithi03829856-13 2024-03-19T11:30:05.010 INFO:tasks.rados.rados.0.smithi038.stdout:13: finishing write tid 4 to smithi03829856-13 2024-03-19T11:30:05.010 INFO:tasks.rados.rados.0.smithi038.stdout:13: oid 13 updating version 0 to 5 2024-03-19T11:30:05.011 INFO:tasks.rados.rados.0.smithi038.stdout:13: oid 13 version 5 is already newer than 4 2024-03-19T11:30:05.011 INFO:tasks.rados.rados.0.smithi038.stdout:13: oid 13 updating version 5 to 6 2024-03-19T11:30:05.011 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 13 v 6 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-03-19T11:30:05.011 INFO:tasks.rados.rados.0.smithi038.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2024-03-19T11:30:05.011 INFO:tasks.rados.rados.0.smithi038.stdout:14: finishing write tid 1 to smithi03829856-14 2024-03-19T11:30:05.011 INFO:tasks.rados.rados.0.smithi038.stdout:14: finishing write tid 2 to smithi03829856-14 2024-03-19T11:30:05.011 INFO:tasks.rados.rados.0.smithi038.stdout:12: done (15 left) 2024-03-19T11:30:05.011 INFO:tasks.rados.rados.0.smithi038.stdout:13: done (14 left) 2024-03-19T11:30:05.011 INFO:tasks.rados.rados.0.smithi038.stdout:28: write initial oid 28 2024-03-19T11:30:05.011 INFO:tasks.rados.rados.0.smithi038.stdout:28: seq_num 27 ranges {453789=780782,1913602=475002} 2024-03-19T11:30:05.016 INFO:tasks.rados.rados.0.smithi038.stdout:28: writing smithi03829856-28 from 453789 to 1234571 tid 1 2024-03-19T11:30:05.023 INFO:tasks.rados.rados.0.smithi038.stdout:28: writing smithi03829856-28 from 1913602 to 2388604 tid 2 2024-03-19T11:30:05.024 INFO:tasks.rados.rados.0.smithi038.stdout:29: write initial oid 29 2024-03-19T11:30:05.024 INFO:tasks.rados.rados.0.smithi038.stdout:29: seq_num 28 ranges {470596=456031,1572084=618660} 2024-03-19T11:30:05.029 INFO:tasks.rados.rados.0.smithi038.stdout:29: writing smithi03829856-29 from 470596 to 926627 tid 1 2024-03-19T11:30:05.036 INFO:tasks.rados.rados.0.smithi038.stdout:29: writing smithi03829856-29 from 1572084 to 2190744 tid 2 2024-03-19T11:30:05.037 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.037 INFO:tasks.rados.rados.0.smithi038.stdout:14: finishing write tid 3 to smithi03829856-14 2024-03-19T11:30:05.037 INFO:tasks.rados.rados.0.smithi038.stdout:14: finishing write tid 4 to smithi03829856-14 2024-03-19T11:30:05.037 INFO:tasks.rados.rados.0.smithi038.stdout:14: oid 14 updating version 0 to 2 2024-03-19T11:30:05.037 INFO:tasks.rados.rados.0.smithi038.stdout:14: oid 14 version 2 is already newer than 1 2024-03-19T11:30:05.037 INFO:tasks.rados.rados.0.smithi038.stdout:14: oid 14 updating version 2 to 3 2024-03-19T11:30:05.037 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 14 v 3 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-03-19T11:30:05.037 INFO:tasks.rados.rados.0.smithi038.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2024-03-19T11:30:05.037 INFO:tasks.rados.rados.0.smithi038.stdout:15: finishing write tid 1 to smithi03829856-15 2024-03-19T11:30:05.037 INFO:tasks.rados.rados.0.smithi038.stdout:15: finishing write tid 2 to smithi03829856-15 2024-03-19T11:30:05.037 INFO:tasks.rados.rados.0.smithi038.stdout:15: finishing write tid 3 to smithi03829856-15 2024-03-19T11:30:05.037 INFO:tasks.rados.rados.0.smithi038.stdout:15: finishing write tid 4 to smithi03829856-15 2024-03-19T11:30:05.037 INFO:tasks.rados.rados.0.smithi038.stdout:15: oid 15 updating version 0 to 4 2024-03-19T11:30:05.037 INFO:tasks.rados.rados.0.smithi038.stdout:15: oid 15 updating version 4 to 5 2024-03-19T11:30:05.038 INFO:tasks.rados.rados.0.smithi038.stdout:15: oid 15 updating version 5 to 6 2024-03-19T11:30:05.038 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 15 v 6 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-03-19T11:30:05.038 INFO:tasks.rados.rados.0.smithi038.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2024-03-19T11:30:05.038 INFO:tasks.rados.rados.0.smithi038.stdout:16: finishing write tid 1 to smithi03829856-16 2024-03-19T11:30:05.038 INFO:tasks.rados.rados.0.smithi038.stdout:16: finishing write tid 2 to smithi03829856-16 2024-03-19T11:30:05.038 INFO:tasks.rados.rados.0.smithi038.stdout:16: finishing write tid 3 to smithi03829856-16 2024-03-19T11:30:05.040 INFO:tasks.rados.rados.0.smithi038.stdout:16: finishing write tid 4 to smithi03829856-16 2024-03-19T11:30:05.040 INFO:tasks.rados.rados.0.smithi038.stdout:16: oid 16 updating version 0 to 5 2024-03-19T11:30:05.040 INFO:tasks.rados.rados.0.smithi038.stdout:16: oid 16 updating version 5 to 6 2024-03-19T11:30:05.040 INFO:tasks.rados.rados.0.smithi038.stdout:16: oid 16 version 6 is already newer than 4 2024-03-19T11:30:05.040 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 16 v 6 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-03-19T11:30:05.040 INFO:tasks.rados.rados.0.smithi038.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2024-03-19T11:30:05.041 INFO:tasks.rados.rados.0.smithi038.stdout:17: finishing write tid 1 to smithi03829856-17 2024-03-19T11:30:05.041 INFO:tasks.rados.rados.0.smithi038.stdout:17: finishing write tid 2 to smithi03829856-17 2024-03-19T11:30:05.041 INFO:tasks.rados.rados.0.smithi038.stdout:14: done (15 left) 2024-03-19T11:30:05.041 INFO:tasks.rados.rados.0.smithi038.stdout:15: done (14 left) 2024-03-19T11:30:05.041 INFO:tasks.rados.rados.0.smithi038.stdout:16: done (13 left) 2024-03-19T11:30:05.041 INFO:tasks.rados.rados.0.smithi038.stdout:30: write initial oid 30 2024-03-19T11:30:05.041 INFO:tasks.rados.rados.0.smithi038.stdout:30: seq_num 29 ranges {487403=531280,1514213=478671} 2024-03-19T11:30:05.042 INFO:tasks.rados.rados.0.smithi038.stdout:30: writing smithi03829856-30 from 487403 to 1018683 tid 1 2024-03-19T11:30:05.048 INFO:tasks.rados.rados.0.smithi038.stdout:30: writing smithi03829856-30 from 1514213 to 1992884 tid 2 2024-03-19T11:30:05.048 INFO:tasks.rados.rados.0.smithi038.stdout:31: write initial oid 31 2024-03-19T11:30:05.049 INFO:tasks.rados.rados.0.smithi038.stdout:31: seq_num 30 ranges {504210=606529,1572695=622329} 2024-03-19T11:30:05.054 INFO:tasks.rados.rados.0.smithi038.stdout:31: writing smithi03829856-31 from 504210 to 1110739 tid 1 2024-03-19T11:30:05.060 INFO:tasks.rados.rados.0.smithi038.stdout:31: writing smithi03829856-31 from 1572695 to 2195024 tid 2 2024-03-19T11:30:05.060 INFO:tasks.rados.rados.0.smithi038.stdout:32: write initial oid 32 2024-03-19T11:30:05.060 INFO:tasks.rados.rados.0.smithi038.stdout:32: seq_num 31 ranges {521017=798131,1747530=482340} 2024-03-19T11:30:05.065 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:05.072+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:05.067 INFO:tasks.rados.rados.0.smithi038.stdout:32: writing smithi03829856-32 from 521017 to 1319148 tid 1 2024-03-19T11:30:05.073 INFO:tasks.rados.rados.0.smithi038.stdout:32: writing smithi03829856-32 from 1747530 to 2229870 tid 2 2024-03-19T11:30:05.074 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.074 INFO:tasks.rados.rados.0.smithi038.stdout:17: finishing write tid 3 to smithi03829856-17 2024-03-19T11:30:05.074 INFO:tasks.rados.rados.0.smithi038.stdout:17: finishing write tid 4 to smithi03829856-17 2024-03-19T11:30:05.074 INFO:tasks.rados.rados.0.smithi038.stdout:17: oid 17 updating version 0 to 9 2024-03-19T11:30:05.074 INFO:tasks.rados.rados.0.smithi038.stdout:17: oid 17 version 9 is already newer than 7 2024-03-19T11:30:05.074 INFO:tasks.rados.rados.0.smithi038.stdout:17: oid 17 version 9 is already newer than 8 2024-03-19T11:30:05.074 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 17 v 9 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-03-19T11:30:05.074 INFO:tasks.rados.rados.0.smithi038.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2024-03-19T11:30:05.074 INFO:tasks.rados.rados.0.smithi038.stdout:18: finishing write tid 1 to smithi03829856-18 2024-03-19T11:30:05.074 INFO:tasks.rados.rados.0.smithi038.stdout:18: finishing write tid 2 to smithi03829856-18 2024-03-19T11:30:05.074 INFO:tasks.rados.rados.0.smithi038.stdout:17: done (15 left) 2024-03-19T11:30:05.074 INFO:tasks.rados.rados.0.smithi038.stdout:33: write initial oid 33 2024-03-19T11:30:05.075 INFO:tasks.rados.rados.0.smithi038.stdout:33: seq_num 32 ranges {537824=473380,1806012=625998} 2024-03-19T11:30:05.079 INFO:tasks.rados.rados.0.smithi038.stdout:33: writing smithi03829856-33 from 537824 to 1011204 tid 1 2024-03-19T11:30:05.083 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:05.083 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:05.089 INFO:tasks.rados.rados.0.smithi038.stdout:33: writing smithi03829856-33 from 1806012 to 2432010 tid 2 2024-03-19T11:30:05.089 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.089 INFO:tasks.rados.rados.0.smithi038.stdout:18: finishing write tid 3 to smithi03829856-18 2024-03-19T11:30:05.089 INFO:tasks.rados.rados.0.smithi038.stdout:18: finishing write tid 4 to smithi03829856-18 2024-03-19T11:30:05.089 INFO:tasks.rados.rados.0.smithi038.stdout:18: oid 18 updating version 0 to 5 2024-03-19T11:30:05.089 INFO:tasks.rados.rados.0.smithi038.stdout:18: oid 18 version 5 is already newer than 4 2024-03-19T11:30:05.089 INFO:tasks.rados.rados.0.smithi038.stdout:18: oid 18 updating version 5 to 6 2024-03-19T11:30:05.090 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 18 v 6 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-03-19T11:30:05.090 INFO:tasks.rados.rados.0.smithi038.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2024-03-19T11:30:05.090 INFO:tasks.rados.rados.0.smithi038.stdout:19: finishing write tid 1 to smithi03829856-19 2024-03-19T11:30:05.090 INFO:tasks.rados.rados.0.smithi038.stdout:19: finishing write tid 2 to smithi03829856-19 2024-03-19T11:30:05.090 INFO:tasks.rados.rados.0.smithi038.stdout:19: finishing write tid 3 to smithi03829856-19 2024-03-19T11:30:05.090 INFO:tasks.rados.rados.0.smithi038.stdout:19: finishing write tid 4 to smithi03829856-19 2024-03-19T11:30:05.090 INFO:tasks.rados.rados.0.smithi038.stdout:19: oid 19 updating version 0 to 3 2024-03-19T11:30:05.090 INFO:tasks.rados.rados.0.smithi038.stdout:19: oid 19 version 3 is already newer than 1 2024-03-19T11:30:05.090 INFO:tasks.rados.rados.0.smithi038.stdout:19: oid 19 version 3 is already newer than 2 2024-03-19T11:30:05.090 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 19 v 3 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-03-19T11:30:05.090 INFO:tasks.rados.rados.0.smithi038.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2024-03-19T11:30:05.090 INFO:tasks.rados.rados.0.smithi038.stdout:20: finishing write tid 1 to smithi03829856-20 2024-03-19T11:30:05.090 INFO:tasks.rados.rados.0.smithi038.stdout:20: finishing write tid 2 to smithi03829856-20 2024-03-19T11:30:05.090 INFO:tasks.rados.rados.0.smithi038.stdout:20: finishing write tid 3 to smithi03829856-20 2024-03-19T11:30:05.091 INFO:tasks.rados.rados.0.smithi038.stdout:20: finishing write tid 4 to smithi03829856-20 2024-03-19T11:30:05.091 INFO:tasks.rados.rados.0.smithi038.stdout:20: oid 20 updating version 0 to 11 2024-03-19T11:30:05.091 INFO:tasks.rados.rados.0.smithi038.stdout:20: oid 20 version 11 is already newer than 10 2024-03-19T11:30:05.091 INFO:tasks.rados.rados.0.smithi038.stdout:20: oid 20 updating version 11 to 12 2024-03-19T11:30:05.091 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 20 v 12 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-03-19T11:30:05.091 INFO:tasks.rados.rados.0.smithi038.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2024-03-19T11:30:05.091 INFO:tasks.rados.rados.0.smithi038.stdout:21: finishing write tid 1 to smithi03829856-21 2024-03-19T11:30:05.091 INFO:tasks.rados.rados.0.smithi038.stdout:21: finishing write tid 2 to smithi03829856-21 2024-03-19T11:30:05.091 INFO:tasks.rados.rados.0.smithi038.stdout:21: finishing write tid 3 to smithi03829856-21 2024-03-19T11:30:05.091 INFO:tasks.rados.rados.0.smithi038.stdout:21: finishing write tid 4 to smithi03829856-21 2024-03-19T11:30:05.091 INFO:tasks.rados.rados.0.smithi038.stdout:21: oid 21 updating version 0 to 14 2024-03-19T11:30:05.091 INFO:tasks.rados.rados.0.smithi038.stdout:21: oid 21 version 14 is already newer than 13 2024-03-19T11:30:05.091 INFO:tasks.rados.rados.0.smithi038.stdout:21: oid 21 updating version 14 to 15 2024-03-19T11:30:05.091 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 21 v 15 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-03-19T11:30:05.091 INFO:tasks.rados.rados.0.smithi038.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2024-03-19T11:30:05.092 INFO:tasks.rados.rados.0.smithi038.stdout:18: done (15 left) 2024-03-19T11:30:05.092 INFO:tasks.rados.rados.0.smithi038.stdout:19: done (14 left) 2024-03-19T11:30:05.092 INFO:tasks.rados.rados.0.smithi038.stdout:20: done (13 left) 2024-03-19T11:30:05.092 INFO:tasks.rados.rados.0.smithi038.stdout:21: done (12 left) 2024-03-19T11:30:05.092 INFO:tasks.rados.rados.0.smithi038.stdout:34: write initial oid 34 2024-03-19T11:30:05.092 INFO:tasks.rados.rados.0.smithi038.stdout:34: seq_num 33 ranges {554631=548629,1748141=486009} 2024-03-19T11:30:05.094 INFO:tasks.rados.rados.0.smithi038.stdout:34: writing smithi03829856-34 from 554631 to 1103260 tid 1 2024-03-19T11:30:05.098 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-19T11:30:05.101 INFO:tasks.rados.rados.0.smithi038.stdout:34: writing smithi03829856-34 from 1748141 to 2234150 tid 2 2024-03-19T11:30:05.101 INFO:tasks.rados.rados.0.smithi038.stdout:35: write initial oid 35 2024-03-19T11:30:05.101 INFO:tasks.rados.rados.0.smithi038.stdout:35: seq_num 34 ranges {571438=623878,1806623=629667} 2024-03-19T11:30:05.107 INFO:tasks.rados.rados.0.smithi038.stdout:35: writing smithi03829856-35 from 571438 to 1195316 tid 1 2024-03-19T11:30:05.114 INFO:tasks.rados.rados.0.smithi038.stdout:35: writing smithi03829856-35 from 1806623 to 2436290 tid 2 2024-03-19T11:30:05.115 INFO:tasks.rados.rados.0.smithi038.stdout:36: write initial oid 36 2024-03-19T11:30:05.115 INFO:tasks.rados.rados.0.smithi038.stdout:36: seq_num 35 ranges {588245=699127,1865105=489678} 2024-03-19T11:30:05.122 INFO:tasks.rados.rados.0.smithi038.stdout:36: writing smithi03829856-36 from 588245 to 1287372 tid 1 2024-03-19T11:30:05.128 INFO:tasks.rados.rados.0.smithi038.stdout:36: writing smithi03829856-36 from 1865105 to 2354783 tid 2 2024-03-19T11:30:05.128 INFO:tasks.rados.rados.0.smithi038.stdout:37: write initial oid 37 2024-03-19T11:30:05.128 INFO:tasks.rados.rados.0.smithi038.stdout:37: seq_num 36 ranges {605052=774376,1923587=633336} 2024-03-19T11:30:05.136 INFO:tasks.rados.rados.0.smithi038.stdout:37: writing smithi03829856-37 from 605052 to 1379428 tid 1 2024-03-19T11:30:05.143 INFO:tasks.rados.rados.0.smithi038.stdout:37: writing smithi03829856-37 from 1923587 to 2556923 tid 2 2024-03-19T11:30:05.143 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.143 INFO:tasks.rados.rados.0.smithi038.stdout:22: finishing write tid 1 to smithi03829856-22 2024-03-19T11:30:05.143 INFO:tasks.rados.rados.0.smithi038.stdout:22: finishing write tid 2 to smithi03829856-22 2024-03-19T11:30:05.143 INFO:tasks.rados.rados.0.smithi038.stdout:22: finishing write tid 3 to smithi03829856-22 2024-03-19T11:30:05.143 INFO:tasks.rados.rados.0.smithi038.stdout:22: finishing write tid 4 to smithi03829856-22 2024-03-19T11:30:05.143 INFO:tasks.rados.rados.0.smithi038.stdout:22: oid 22 updating version 0 to 2 2024-03-19T11:30:05.143 INFO:tasks.rados.rados.0.smithi038.stdout:22: oid 22 version 2 is already newer than 1 2024-03-19T11:30:05.143 INFO:tasks.rados.rados.0.smithi038.stdout:22: oid 22 updating version 2 to 3 2024-03-19T11:30:05.144 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 22 v 3 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-03-19T11:30:05.144 INFO:tasks.rados.rados.0.smithi038.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2024-03-19T11:30:05.144 INFO:tasks.rados.rados.0.smithi038.stdout:23: finishing write tid 1 to smithi03829856-23 2024-03-19T11:30:05.144 INFO:tasks.rados.rados.0.smithi038.stdout:23: finishing write tid 2 to smithi03829856-23 2024-03-19T11:30:05.144 INFO:tasks.rados.rados.0.smithi038.stdout:22: done (15 left) 2024-03-19T11:30:05.144 INFO:tasks.rados.rados.0.smithi038.stdout:38: write initial oid 38 2024-03-19T11:30:05.144 INFO:tasks.rados.rados.0.smithi038.stdout:38: seq_num 37 ranges {621859=449625,1865716=756143} 2024-03-19T11:30:05.148 INFO:tasks.rados.rados.0.smithi038.stdout:38: writing smithi03829856-38 from 621859 to 1071484 tid 1 2024-03-19T11:30:05.158 INFO:tasks.rados.rados.0.smithi038.stdout:38: writing smithi03829856-38 from 1865716 to 2621859 tid 2 2024-03-19T11:30:05.158 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.158 INFO:tasks.rados.rados.0.smithi038.stdout:23: finishing write tid 3 to smithi03829856-23 2024-03-19T11:30:05.158 INFO:tasks.rados.rados.0.smithi038.stdout:23: finishing write tid 4 to smithi03829856-23 2024-03-19T11:30:05.158 INFO:tasks.rados.rados.0.smithi038.stdout:23: oid 23 updating version 0 to 8 2024-03-19T11:30:05.158 INFO:tasks.rados.rados.0.smithi038.stdout:23: oid 23 version 8 is already newer than 7 2024-03-19T11:30:05.158 INFO:tasks.rados.rados.0.smithi038.stdout:23: oid 23 updating version 8 to 9 2024-03-19T11:30:05.158 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 23 v 9 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-03-19T11:30:05.158 INFO:tasks.rados.rados.0.smithi038.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2024-03-19T11:30:05.159 INFO:tasks.rados.rados.0.smithi038.stdout:24: finishing write tid 1 to smithi03829856-24 2024-03-19T11:30:05.159 INFO:tasks.rados.rados.0.smithi038.stdout:24: finishing write tid 2 to smithi03829856-24 2024-03-19T11:30:05.159 INFO:tasks.rados.rados.0.smithi038.stdout:24: finishing write tid 3 to smithi03829856-24 2024-03-19T11:30:05.159 INFO:tasks.rados.rados.0.smithi038.stdout:24: finishing write tid 4 to smithi03829856-24 2024-03-19T11:30:05.159 INFO:tasks.rados.rados.0.smithi038.stdout:24: oid 24 updating version 0 to 7 2024-03-19T11:30:05.159 INFO:tasks.rados.rados.0.smithi038.stdout:24: oid 24 updating version 7 to 8 2024-03-19T11:30:05.159 INFO:tasks.rados.rados.0.smithi038.stdout:24: oid 24 updating version 8 to 9 2024-03-19T11:30:05.159 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 24 v 9 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-03-19T11:30:05.159 INFO:tasks.rados.rados.0.smithi038.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2024-03-19T11:30:05.159 INFO:tasks.rados.rados.0.smithi038.stdout:25: finishing write tid 1 to smithi03829856-25 2024-03-19T11:30:05.159 INFO:tasks.rados.rados.0.smithi038.stdout:25: finishing write tid 2 to smithi03829856-25 2024-03-19T11:30:05.159 INFO:tasks.rados.rados.0.smithi038.stdout:25: finishing write tid 3 to smithi03829856-25 2024-03-19T11:30:05.160 INFO:tasks.rados.rados.0.smithi038.stdout:25: finishing write tid 4 to smithi03829856-25 2024-03-19T11:30:05.160 INFO:tasks.rados.rados.0.smithi038.stdout:25: oid 25 updating version 0 to 12 2024-03-19T11:30:05.160 INFO:tasks.rados.rados.0.smithi038.stdout:25: oid 25 version 12 is already newer than 10 2024-03-19T11:30:05.160 INFO:tasks.rados.rados.0.smithi038.stdout:25: oid 25 version 12 is already newer than 11 2024-03-19T11:30:05.160 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 25 v 12 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-03-19T11:30:05.160 INFO:tasks.rados.rados.0.smithi038.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2024-03-19T11:30:05.160 INFO:tasks.rados.rados.0.smithi038.stdout:26: finishing write tid 1 to smithi03829856-26 2024-03-19T11:30:05.160 INFO:tasks.rados.rados.0.smithi038.stdout:26: finishing write tid 2 to smithi03829856-26 2024-03-19T11:30:05.160 INFO:tasks.rados.rados.0.smithi038.stdout:26: finishing write tid 3 to smithi03829856-26 2024-03-19T11:30:05.160 INFO:tasks.rados.rados.0.smithi038.stdout:23: done (15 left) 2024-03-19T11:30:05.160 INFO:tasks.rados.rados.0.smithi038.stdout:24: done (14 left) 2024-03-19T11:30:05.160 INFO:tasks.rados.rados.0.smithi038.stdout:25: done (13 left) 2024-03-19T11:30:05.160 INFO:tasks.rados.rados.0.smithi038.stdout:39: write initial oid 39 2024-03-19T11:30:05.161 INFO:tasks.rados.rados.0.smithi038.stdout:39: seq_num 38 ranges {638666=524874,1924198=637005} 2024-03-19T11:30:05.164 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:05.171+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:05.164 INFO:tasks.rados.rados.0.smithi038.stdout:39: writing smithi03829856-39 from 638666 to 1163540 tid 1 2024-03-19T11:30:05.173 INFO:tasks.rados.rados.0.smithi038.stdout:39: writing smithi03829856-39 from 1924198 to 2561203 tid 2 2024-03-19T11:30:05.174 INFO:tasks.rados.rados.0.smithi038.stdout:40: write initial oid 40 2024-03-19T11:30:05.174 INFO:tasks.rados.rados.0.smithi038.stdout:26: finishing write tid 4 to smithi03829856-26 2024-03-19T11:30:05.174 INFO:tasks.rados.rados.0.smithi038.stdout:26: oid 26 updating version 0 to 8 2024-03-19T11:30:05.174 INFO:tasks.rados.rados.0.smithi038.stdout:26: oid 26 version 8 is already newer than 7 2024-03-19T11:30:05.174 INFO:tasks.rados.rados.0.smithi038.stdout:26: oid 26 updating version 8 to 9 2024-03-19T11:30:05.174 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 26 v 9 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-03-19T11:30:05.174 INFO:tasks.rados.rados.0.smithi038.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2024-03-19T11:30:05.174 INFO:tasks.rados.rados.0.smithi038.stdout:27: finishing write tid 1 to smithi03829856-27 2024-03-19T11:30:05.174 INFO:tasks.rados.rados.0.smithi038.stdout:27: finishing write tid 2 to smithi03829856-27 2024-03-19T11:30:05.174 INFO:tasks.rados.rados.0.smithi038.stdout:27: finishing write tid 3 to smithi03829856-27 2024-03-19T11:30:05.174 INFO:tasks.rados.rados.0.smithi038.stdout:27: finishing write tid 4 to smithi03829856-27 2024-03-19T11:30:05.174 INFO:tasks.rados.rados.0.smithi038.stdout:27: oid 27 updating version 0 to 5 2024-03-19T11:30:05.174 INFO:tasks.rados.rados.0.smithi038.stdout:27: oid 27 version 5 is already newer than 4 2024-03-19T11:30:05.174 INFO:tasks.rados.rados.0.smithi038.stdout:27: oid 27 updating version 5 to 6 2024-03-19T11:30:05.175 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 27 v 6 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-03-19T11:30:05.175 INFO:tasks.rados.rados.0.smithi038.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2024-03-19T11:30:05.175 INFO:tasks.rados.rados.0.smithi038.stdout:28: finishing write tid 1 to smithi03829856-28 2024-03-19T11:30:05.175 INFO:tasks.rados.rados.0.smithi038.stdout:28: finishing write tid 2 to smithi03829856-28 2024-03-19T11:30:05.175 INFO:tasks.rados.rados.0.smithi038.stdout:28: finishing write tid 3 to smithi03829856-28 2024-03-19T11:30:05.175 INFO:tasks.rados.rados.0.smithi038.stdout:28: finishing write tid 4 to smithi03829856-28 2024-03-19T11:30:05.175 INFO:tasks.rados.rados.0.smithi038.stdout:28: oid 28 updating version 0 to 6 2024-03-19T11:30:05.175 INFO:tasks.rados.rados.0.smithi038.stdout:28: oid 28 version 6 is already newer than 5 2024-03-19T11:30:05.175 INFO:tasks.rados.rados.0.smithi038.stdout:28: oid 28 version 6 is already newer than 4 2024-03-19T11:30:05.175 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 28 v 6 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-03-19T11:30:05.175 INFO:tasks.rados.rados.0.smithi038.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2024-03-19T11:30:05.175 INFO:tasks.rados.rados.0.smithi038.stdout:29: finishing write tid 1 to smithi03829856-29 2024-03-19T11:30:05.175 INFO:tasks.rados.rados.0.smithi038.stdout:29: finishing write tid 2 to smithi03829856-29 2024-03-19T11:30:05.175 INFO:tasks.rados.rados.0.smithi038.stdout:29: finishing write tid 3 to smithi03829856-29 2024-03-19T11:30:05.176 INFO:tasks.rados.rados.0.smithi038.stdout:29: finishing write tid 4 to smithi03829856-29 2024-03-19T11:30:05.176 INFO:tasks.rados.rados.0.smithi038.stdout:29: oid 29 updating version 0 to 2 2024-03-19T11:30:05.176 INFO:tasks.rados.rados.0.smithi038.stdout:29: oid 29 updating version 2 to 3 2024-03-19T11:30:05.176 INFO:tasks.rados.rados.0.smithi038.stdout:29: oid 29 version 3 is already newer than 1 2024-03-19T11:30:05.176 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 29 v 3 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-03-19T11:30:05.176 INFO:tasks.rados.rados.0.smithi038.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2024-03-19T11:30:05.176 INFO:tasks.rados.rados.0.smithi038.stdout:30: finishing write tid 1 to smithi03829856-30 2024-03-19T11:30:05.176 INFO:tasks.rados.rados.0.smithi038.stdout:30: finishing write tid 2 to smithi03829856-30 2024-03-19T11:30:05.176 INFO:tasks.rados.rados.0.smithi038.stdout:30: finishing write tid 3 to smithi03829856-30 2024-03-19T11:30:05.176 INFO:tasks.rados.rados.0.smithi038.stdout:30: finishing write tid 4 to smithi03829856-30 2024-03-19T11:30:05.176 INFO:tasks.rados.rados.0.smithi038.stdout:30: oid 30 updating version 0 to 6 2024-03-19T11:30:05.176 INFO:tasks.rados.rados.0.smithi038.stdout:30: oid 30 version 6 is already newer than 4 2024-03-19T11:30:05.176 INFO:tasks.rados.rados.0.smithi038.stdout:30: oid 30 version 6 is already newer than 5 2024-03-19T11:30:05.176 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 30 v 6 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-03-19T11:30:05.176 INFO:tasks.rados.rados.0.smithi038.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2024-03-19T11:30:05.176 INFO:tasks.rados.rados.0.smithi038.stdout:31: finishing write tid 1 to smithi03829856-31 2024-03-19T11:30:05.177 INFO:tasks.rados.rados.0.smithi038.stdout:32: finishing write tid 1 to smithi03829856-32 2024-03-19T11:30:05.177 INFO:tasks.rados.rados.0.smithi038.stdout:31: finishing write tid 2 to smithi03829856-31 2024-03-19T11:30:05.177 INFO:tasks.rados.rados.0.smithi038.stdout:31: finishing write tid 3 to smithi03829856-31 2024-03-19T11:30:05.177 INFO:tasks.rados.rados.0.smithi038.stdout:31: finishing write tid 4 to smithi03829856-31 2024-03-19T11:30:05.177 INFO:tasks.rados.rados.0.smithi038.stdout:31: oid 31 updating version 0 to 4 2024-03-19T11:30:05.177 INFO:tasks.rados.rados.0.smithi038.stdout:31: oid 31 updating version 4 to 5 2024-03-19T11:30:05.177 INFO:tasks.rados.rados.0.smithi038.stdout:31: oid 31 updating version 5 to 6 2024-03-19T11:30:05.177 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 31 v 6 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-03-19T11:30:05.177 INFO:tasks.rados.rados.0.smithi038.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2024-03-19T11:30:05.177 INFO:tasks.rados.rados.0.smithi038.stdout:32: finishing write tid 2 to smithi03829856-32 2024-03-19T11:30:05.177 INFO:tasks.rados.rados.0.smithi038.stdout:32: finishing write tid 3 to smithi03829856-32 2024-03-19T11:30:05.177 INFO:tasks.rados.rados.0.smithi038.stdout:32: finishing write tid 4 to smithi03829856-32 2024-03-19T11:30:05.177 INFO:tasks.rados.rados.0.smithi038.stdout:32: oid 32 updating version 0 to 5 2024-03-19T11:30:05.178 INFO:tasks.rados.rados.0.smithi038.stdout:32: oid 32 updating version 5 to 6 2024-03-19T11:30:05.178 INFO:tasks.rados.rados.0.smithi038.stdout:32: oid 32 version 6 is already newer than 4 2024-03-19T11:30:05.178 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 32 v 6 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-03-19T11:30:05.178 INFO:tasks.rados.rados.0.smithi038.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2024-03-19T11:30:05.178 INFO:tasks.rados.rados.0.smithi038.stdout:33: finishing write tid 1 to smithi03829856-33 2024-03-19T11:30:05.178 INFO:tasks.rados.rados.0.smithi038.stdout:33: finishing write tid 2 to smithi03829856-33 2024-03-19T11:30:05.178 INFO:tasks.rados.rados.0.smithi038.stdout:33: finishing write tid 3 to smithi03829856-33 2024-03-19T11:30:05.178 INFO:tasks.rados.rados.0.smithi038.stdout:33: finishing write tid 4 to smithi03829856-33 2024-03-19T11:30:05.178 INFO:tasks.rados.rados.0.smithi038.stdout:33: oid 33 updating version 0 to 7 2024-03-19T11:30:05.178 INFO:tasks.rados.rados.0.smithi038.stdout:33: oid 33 updating version 7 to 9 2024-03-19T11:30:05.178 INFO:tasks.rados.rados.0.smithi038.stdout:33: oid 33 version 9 is already newer than 8 2024-03-19T11:30:05.178 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 33 v 9 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-03-19T11:30:05.178 INFO:tasks.rados.rados.0.smithi038.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2024-03-19T11:30:05.178 INFO:tasks.rados.rados.0.smithi038.stdout:34: finishing write tid 1 to smithi03829856-34 2024-03-19T11:30:05.178 INFO:tasks.rados.rados.0.smithi038.stdout:34: finishing write tid 2 to smithi03829856-34 2024-03-19T11:30:05.179 INFO:tasks.rados.rados.0.smithi038.stdout:34: finishing write tid 3 to smithi03829856-34 2024-03-19T11:30:05.179 INFO:tasks.rados.rados.0.smithi038.stdout:34: finishing write tid 4 to smithi03829856-34 2024-03-19T11:30:05.179 INFO:tasks.rados.rados.0.smithi038.stdout:34: oid 34 updating version 0 to 3 2024-03-19T11:30:05.179 INFO:tasks.rados.rados.0.smithi038.stdout:34: oid 34 version 3 is already newer than 1 2024-03-19T11:30:05.179 INFO:tasks.rados.rados.0.smithi038.stdout:34: oid 34 version 3 is already newer than 2 2024-03-19T11:30:05.179 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 34 v 3 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-03-19T11:30:05.179 INFO:tasks.rados.rados.0.smithi038.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2024-03-19T11:30:05.179 INFO:tasks.rados.rados.0.smithi038.stdout:35: finishing write tid 1 to smithi03829856-35 2024-03-19T11:30:05.179 INFO:tasks.rados.rados.0.smithi038.stdout:35: finishing write tid 2 to smithi03829856-35 2024-03-19T11:30:05.179 INFO:tasks.rados.rados.0.smithi038.stdout:35: finishing write tid 3 to smithi03829856-35 2024-03-19T11:30:05.179 INFO:tasks.rados.rados.0.smithi038.stdout:35: finishing write tid 4 to smithi03829856-35 2024-03-19T11:30:05.179 INFO:tasks.rados.rados.0.smithi038.stdout:35: oid 35 updating version 0 to 17 2024-03-19T11:30:05.179 INFO:tasks.rados.rados.0.smithi038.stdout:35: oid 35 updating version 17 to 18 2024-03-19T11:30:05.180 INFO:tasks.rados.rados.0.smithi038.stdout:35: oid 35 version 18 is already newer than 16 2024-03-19T11:30:05.180 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 35 v 18 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-03-19T11:30:05.180 INFO:tasks.rados.rados.0.smithi038.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2024-03-19T11:30:05.180 INFO:tasks.rados.rados.0.smithi038.stdout:36: finishing write tid 1 to smithi03829856-36 2024-03-19T11:30:05.180 INFO:tasks.rados.rados.0.smithi038.stdout:36: finishing write tid 2 to smithi03829856-36 2024-03-19T11:30:05.180 INFO:tasks.rados.rados.0.smithi038.stdout:36: finishing write tid 3 to smithi03829856-36 2024-03-19T11:30:05.180 INFO:tasks.rados.rados.0.smithi038.stdout:36: finishing write tid 4 to smithi03829856-36 2024-03-19T11:30:05.180 INFO:tasks.rados.rados.0.smithi038.stdout:36: oid 36 updating version 0 to 9 2024-03-19T11:30:05.180 INFO:tasks.rados.rados.0.smithi038.stdout:36: oid 36 version 9 is already newer than 8 2024-03-19T11:30:05.180 INFO:tasks.rados.rados.0.smithi038.stdout:36: oid 36 version 9 is already newer than 7 2024-03-19T11:30:05.180 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 36 v 9 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-03-19T11:30:05.180 INFO:tasks.rados.rados.0.smithi038.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2024-03-19T11:30:05.181 INFO:tasks.rados.rados.0.smithi038.stdout:40: seq_num 39 ranges {655473=716476,2099033=556440} 2024-03-19T11:30:05.181 INFO:tasks.rados.rados.0.smithi038.stdout:40: writing smithi03829856-40 from 655473 to 1371949 tid 1 2024-03-19T11:30:05.187 INFO:tasks.rados.rados.0.smithi038.stdout:40: writing smithi03829856-40 from 2099033 to 2655473 tid 2 2024-03-19T11:30:05.187 INFO:tasks.rados.rados.0.smithi038.stdout:26: done (14 left) 2024-03-19T11:30:05.187 INFO:tasks.rados.rados.0.smithi038.stdout:27: done (13 left) 2024-03-19T11:30:05.187 INFO:tasks.rados.rados.0.smithi038.stdout:28: done (12 left) 2024-03-19T11:30:05.187 INFO:tasks.rados.rados.0.smithi038.stdout:29: done (11 left) 2024-03-19T11:30:05.187 INFO:tasks.rados.rados.0.smithi038.stdout:30: done (10 left) 2024-03-19T11:30:05.187 INFO:tasks.rados.rados.0.smithi038.stdout:31: done (9 left) 2024-03-19T11:30:05.187 INFO:tasks.rados.rados.0.smithi038.stdout:32: done (8 left) 2024-03-19T11:30:05.187 INFO:tasks.rados.rados.0.smithi038.stdout:33: done (7 left) 2024-03-19T11:30:05.187 INFO:tasks.rados.rados.0.smithi038.stdout:34: done (6 left) 2024-03-19T11:30:05.187 INFO:tasks.rados.rados.0.smithi038.stdout:35: done (5 left) 2024-03-19T11:30:05.187 INFO:tasks.rados.rados.0.smithi038.stdout:36: done (4 left) 2024-03-19T11:30:05.187 INFO:tasks.rados.rados.0.smithi038.stdout:41: write initial oid 41 2024-03-19T11:30:05.187 INFO:tasks.rados.rados.0.smithi038.stdout:41: seq_num 40 ranges {672280=791725,2157515=514765} 2024-03-19T11:30:05.194 INFO:tasks.rados.rados.0.smithi038.stdout:41: writing smithi03829856-41 from 672280 to 1464005 tid 1 2024-03-19T11:30:05.201 INFO:tasks.rados.rados.0.smithi038.stdout:41: writing smithi03829856-41 from 2157515 to 2672280 tid 2 2024-03-19T11:30:05.201 INFO:tasks.rados.rados.0.smithi038.stdout:42: write initial oid 42 2024-03-19T11:30:05.201 INFO:tasks.rados.rados.0.smithi038.stdout:42: seq_num 41 ranges {689087=466974,1699644=784332} 2024-03-19T11:30:05.208 INFO:tasks.rados.rados.0.smithi038.stdout:42: writing smithi03829856-42 from 689087 to 1156061 tid 1 2024-03-19T11:30:05.215 INFO:tasks.rados.rados.0.smithi038.stdout:42: writing smithi03829856-42 from 1699644 to 2483976 tid 2 2024-03-19T11:30:05.215 INFO:tasks.rados.rados.0.smithi038.stdout:43: write initial oid 43 2024-03-19T11:30:05.215 INFO:tasks.rados.rados.0.smithi038.stdout:43: seq_num 42 ranges {705894=542223,1758126=644343} 2024-03-19T11:30:05.222 INFO:tasks.rados.rados.0.smithi038.stdout:43: writing smithi03829856-43 from 705894 to 1248117 tid 1 2024-03-19T11:30:05.226 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:05.226 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:05.229 INFO:tasks.rados.rados.0.smithi038.stdout:43: writing smithi03829856-43 from 1758126 to 2402469 tid 2 2024-03-19T11:30:05.229 INFO:tasks.rados.rados.0.smithi038.stdout:44: write initial oid 44 2024-03-19T11:30:05.229 INFO:tasks.rados.rados.0.smithi038.stdout:44: seq_num 43 ranges {722701=617472,1816608=788001} 2024-03-19T11:30:05.236 INFO:tasks.rados.rados.0.smithi038.stdout:44: writing smithi03829856-44 from 722701 to 1340173 tid 1 2024-03-19T11:30:05.239 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-19T11:30:05.243 INFO:tasks.rados.rados.0.smithi038.stdout:44: writing smithi03829856-44 from 1816608 to 2604609 tid 2 2024-03-19T11:30:05.243 INFO:tasks.rados.rados.0.smithi038.stdout:45: write initial oid 45 2024-03-19T11:30:05.244 INFO:tasks.rados.rados.0.smithi038.stdout:45: seq_num 44 ranges {739508=692721,1875090=648012} 2024-03-19T11:30:05.251 INFO:tasks.rados.rados.0.smithi038.stdout:45: writing smithi03829856-45 from 739508 to 1432229 tid 1 2024-03-19T11:30:05.258 INFO:tasks.rados.rados.0.smithi038.stdout:45: writing smithi03829856-45 from 1875090 to 2523102 tid 2 2024-03-19T11:30:05.258 INFO:tasks.rados.rados.0.smithi038.stdout:46: write initial oid 46 2024-03-19T11:30:05.258 INFO:tasks.rados.rados.0.smithi038.stdout:46: seq_num 45 ranges {756315=767970,1933572=791670} 2024-03-19T11:30:05.264 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:05.272+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:05.266 INFO:tasks.rados.rados.0.smithi038.stdout:46: writing smithi03829856-46 from 756315 to 1524285 tid 1 2024-03-19T11:30:05.273 INFO:tasks.rados.rados.0.smithi038.stdout:46: writing smithi03829856-46 from 1933572 to 2725242 tid 2 2024-03-19T11:30:05.273 INFO:tasks.rados.rados.0.smithi038.stdout:47: write initial oid 47 2024-03-19T11:30:05.273 INFO:tasks.rados.rados.0.smithi038.stdout:47: seq_num 46 ranges {773122=559572,1992054=651681} 2024-03-19T11:30:05.280 INFO:tasks.rados.rados.0.smithi038.stdout:47: writing smithi03829856-47 from 773122 to 1332694 tid 1 2024-03-19T11:30:05.288 INFO:tasks.rados.rados.0.smithi038.stdout:47: writing smithi03829856-47 from 1992054 to 2643735 tid 2 2024-03-19T11:30:05.288 INFO:tasks.rados.rados.0.smithi038.stdout:48: write initial oid 48 2024-03-19T11:30:05.288 INFO:tasks.rados.rados.0.smithi038.stdout:48: seq_num 47 ranges {789929=634821,2050536=739393} 2024-03-19T11:30:05.295 INFO:tasks.rados.rados.0.smithi038.stdout:48: writing smithi03829856-48 from 789929 to 1424750 tid 1 2024-03-19T11:30:05.303 INFO:tasks.rados.rados.0.smithi038.stdout:48: writing smithi03829856-48 from 2050536 to 2789929 tid 2 2024-03-19T11:30:05.303 INFO:tasks.rados.rados.0.smithi038.stdout:49: write initial oid 49 2024-03-19T11:30:05.303 INFO:tasks.rados.rados.0.smithi038.stdout:49: seq_num 48 ranges {406736=710070,1709018=655350} 2024-03-19T11:30:05.309 INFO:tasks.rados.rados.0.smithi038.stdout:49: writing smithi03829856-49 from 406736 to 1116806 tid 1 2024-03-19T11:30:05.317 INFO:tasks.rados.rados.0.smithi038.stdout:49: writing smithi03829856-49 from 1709018 to 2364368 tid 2 2024-03-19T11:30:05.317 INFO:tasks.rados.rados.0.smithi038.stdout:50: write initial oid 50 2024-03-19T11:30:05.317 INFO:tasks.rados.rados.0.smithi038.stdout:50: seq_num 49 ranges {423543=785319,1767500=799008} 2024-03-19T11:30:05.323 INFO:tasks.rados.rados.0.smithi038.stdout:50: writing smithi03829856-50 from 423543 to 1208862 tid 1 2024-03-19T11:30:05.334 INFO:tasks.rados.rados.0.smithi038.stdout:50: writing smithi03829856-50 from 1767500 to 2566508 tid 2 2024-03-19T11:30:05.334 INFO:tasks.rados.rados.0.smithi038.stdout:51: copy oid 2 target oid 2 low_tier 2024-03-19T11:30:05.334 INFO:tasks.rados.rados.0.smithi038.stdout:copy op oid 2 to 2 low_tier tgt_pool_name low_tier 2024-03-19T11:30:05.334 INFO:tasks.rados.rados.0.smithi038.stdout:52: copy oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.334 INFO:tasks.rados.rados.0.smithi038.stdout:copy op oid 3 to 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-03-19T11:30:05.334 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.334 INFO:tasks.rados.rados.0.smithi038.stdout:37: finishing write tid 1 to smithi03829856-37 2024-03-19T11:30:05.334 INFO:tasks.rados.rados.0.smithi038.stdout:37: finishing write tid 2 to smithi03829856-37 2024-03-19T11:30:05.334 INFO:tasks.rados.rados.0.smithi038.stdout:37: finishing write tid 3 to smithi03829856-37 2024-03-19T11:30:05.335 INFO:tasks.rados.rados.0.smithi038.stdout:37: finishing write tid 4 to smithi03829856-37 2024-03-19T11:30:05.335 INFO:tasks.rados.rados.0.smithi038.stdout:37: oid 37 updating version 0 to 20 2024-03-19T11:30:05.335 INFO:tasks.rados.rados.0.smithi038.stdout:37: oid 37 version 20 is already newer than 19 2024-03-19T11:30:05.335 INFO:tasks.rados.rados.0.smithi038.stdout:37: oid 37 updating version 20 to 21 2024-03-19T11:30:05.335 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 37 v 21 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-03-19T11:30:05.335 INFO:tasks.rados.rados.0.smithi038.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2024-03-19T11:30:05.335 INFO:tasks.rados.rados.0.smithi038.stdout:38: finishing write tid 1 to smithi03829856-38 2024-03-19T11:30:05.335 INFO:tasks.rados.rados.0.smithi038.stdout:38: finishing write tid 2 to smithi03829856-38 2024-03-19T11:30:05.335 INFO:tasks.rados.rados.0.smithi038.stdout:38: finishing write tid 3 to smithi03829856-38 2024-03-19T11:30:05.335 INFO:tasks.rados.rados.0.smithi038.stdout:38: finishing write tid 4 to smithi03829856-38 2024-03-19T11:30:05.335 INFO:tasks.rados.rados.0.smithi038.stdout:38: oid 38 updating version 0 to 11 2024-03-19T11:30:05.335 INFO:tasks.rados.rados.0.smithi038.stdout:38: oid 38 updating version 11 to 12 2024-03-19T11:30:05.335 INFO:tasks.rados.rados.0.smithi038.stdout:38: oid 38 version 12 is already newer than 10 2024-03-19T11:30:05.335 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 38 v 12 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-03-19T11:30:05.335 INFO:tasks.rados.rados.0.smithi038.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2024-03-19T11:30:05.336 INFO:tasks.rados.rados.0.smithi038.stdout:39: finishing write tid 1 to smithi03829856-39 2024-03-19T11:30:05.336 INFO:tasks.rados.rados.0.smithi038.stdout:39: finishing write tid 2 to smithi03829856-39 2024-03-19T11:30:05.336 INFO:tasks.rados.rados.0.smithi038.stdout:39: finishing write tid 3 to smithi03829856-39 2024-03-19T11:30:05.336 INFO:tasks.rados.rados.0.smithi038.stdout:37: done (15 left) 2024-03-19T11:30:05.336 INFO:tasks.rados.rados.0.smithi038.stdout:38: done (14 left) 2024-03-19T11:30:05.336 INFO:tasks.rados.rados.0.smithi038.stdout:53: copy oid 4 target oid 4 low_tier 2024-03-19T11:30:05.336 INFO:tasks.rados.rados.0.smithi038.stdout:copy op oid 4 to 4 low_tier tgt_pool_name low_tier 2024-03-19T11:30:05.336 INFO:tasks.rados.rados.0.smithi038.stdout:54: copy oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.336 INFO:tasks.rados.rados.0.smithi038.stdout:copy op oid 5 to 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-03-19T11:30:05.336 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.336 INFO:tasks.rados.rados.0.smithi038.stdout:39: finishing write tid 4 to smithi03829856-39 2024-03-19T11:30:05.336 INFO:tasks.rados.rados.0.smithi038.stdout:39: oid 39 updating version 0 to 6 2024-03-19T11:30:05.336 INFO:tasks.rados.rados.0.smithi038.stdout:39: oid 39 version 6 is already newer than 5 2024-03-19T11:30:05.336 INFO:tasks.rados.rados.0.smithi038.stdout:39: oid 39 version 6 is already newer than 4 2024-03-19T11:30:05.336 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 39 v 6 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-03-19T11:30:05.337 INFO:tasks.rados.rados.0.smithi038.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2024-03-19T11:30:05.337 INFO:tasks.rados.rados.0.smithi038.stdout:40: finishing write tid 1 to smithi03829856-40 2024-03-19T11:30:05.337 INFO:tasks.rados.rados.0.smithi038.stdout:40: finishing write tid 2 to smithi03829856-40 2024-03-19T11:30:05.337 INFO:tasks.rados.rados.0.smithi038.stdout:40: finishing write tid 3 to smithi03829856-40 2024-03-19T11:30:05.337 INFO:tasks.rados.rados.0.smithi038.stdout:40: finishing write tid 4 to smithi03829856-40 2024-03-19T11:30:05.337 INFO:tasks.rados.rados.0.smithi038.stdout:40: oid 40 updating version 0 to 7 2024-03-19T11:30:05.337 INFO:tasks.rados.rados.0.smithi038.stdout:40: oid 40 updating version 7 to 8 2024-03-19T11:30:05.337 INFO:tasks.rados.rados.0.smithi038.stdout:40: oid 40 updating version 8 to 9 2024-03-19T11:30:05.337 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 40 v 9 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-03-19T11:30:05.337 INFO:tasks.rados.rados.0.smithi038.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2024-03-19T11:30:05.337 INFO:tasks.rados.rados.0.smithi038.stdout:41: finishing write tid 1 to smithi03829856-41 2024-03-19T11:30:05.337 INFO:tasks.rados.rados.0.smithi038.stdout:41: finishing write tid 2 to smithi03829856-41 2024-03-19T11:30:05.337 INFO:tasks.rados.rados.0.smithi038.stdout:41: finishing write tid 3 to smithi03829856-41 2024-03-19T11:30:05.337 INFO:tasks.rados.rados.0.smithi038.stdout:41: finishing write tid 4 to smithi03829856-41 2024-03-19T11:30:05.337 INFO:tasks.rados.rados.0.smithi038.stdout:41: oid 41 updating version 0 to 8 2024-03-19T11:30:05.338 INFO:tasks.rados.rados.0.smithi038.stdout:41: oid 41 updating version 8 to 9 2024-03-19T11:30:05.338 INFO:tasks.rados.rados.0.smithi038.stdout:41: oid 41 version 9 is already newer than 7 2024-03-19T11:30:05.338 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 41 v 9 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-03-19T11:30:05.338 INFO:tasks.rados.rados.0.smithi038.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2024-03-19T11:30:05.338 INFO:tasks.rados.rados.0.smithi038.stdout:42: finishing write tid 1 to smithi03829856-42 2024-03-19T11:30:05.338 INFO:tasks.rados.rados.0.smithi038.stdout:42: finishing write tid 2 to smithi03829856-42 2024-03-19T11:30:05.338 INFO:tasks.rados.rados.0.smithi038.stdout:42: finishing write tid 3 to smithi03829856-42 2024-03-19T11:30:05.338 INFO:tasks.rados.rados.0.smithi038.stdout:42: finishing write tid 4 to smithi03829856-42 2024-03-19T11:30:05.338 INFO:tasks.rados.rados.0.smithi038.stdout:42: oid 42 updating version 0 to 10 2024-03-19T11:30:05.338 INFO:tasks.rados.rados.0.smithi038.stdout:42: oid 42 updating version 10 to 12 2024-03-19T11:30:05.338 INFO:tasks.rados.rados.0.smithi038.stdout:42: oid 42 version 12 is already newer than 11 2024-03-19T11:30:05.338 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 42 v 12 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-03-19T11:30:05.338 INFO:tasks.rados.rados.0.smithi038.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2024-03-19T11:30:05.338 INFO:tasks.rados.rados.0.smithi038.stdout:43: finishing write tid 1 to smithi03829856-43 2024-03-19T11:30:05.338 INFO:tasks.rados.rados.0.smithi038.stdout:43: finishing write tid 2 to smithi03829856-43 2024-03-19T11:30:05.339 INFO:tasks.rados.rados.0.smithi038.stdout:43: finishing write tid 3 to smithi03829856-43 2024-03-19T11:30:05.339 INFO:tasks.rados.rados.0.smithi038.stdout:43: finishing write tid 4 to smithi03829856-43 2024-03-19T11:30:05.339 INFO:tasks.rados.rados.0.smithi038.stdout:43: oid 43 updating version 0 to 9 2024-03-19T11:30:05.339 INFO:tasks.rados.rados.0.smithi038.stdout:43: oid 43 version 9 is already newer than 7 2024-03-19T11:30:05.339 INFO:tasks.rados.rados.0.smithi038.stdout:43: oid 43 version 9 is already newer than 8 2024-03-19T11:30:05.339 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 43 v 9 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-03-19T11:30:05.339 INFO:tasks.rados.rados.0.smithi038.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2024-03-19T11:30:05.339 INFO:tasks.rados.rados.0.smithi038.stdout:44: finishing write tid 1 to smithi03829856-44 2024-03-19T11:30:05.339 INFO:tasks.rados.rados.0.smithi038.stdout:44: finishing write tid 2 to smithi03829856-44 2024-03-19T11:30:05.339 INFO:tasks.rados.rados.0.smithi038.stdout:44: finishing write tid 3 to smithi03829856-44 2024-03-19T11:30:05.339 INFO:tasks.rados.rados.0.smithi038.stdout:44: finishing write tid 4 to smithi03829856-44 2024-03-19T11:30:05.339 INFO:tasks.rados.rados.0.smithi038.stdout:44: oid 44 updating version 0 to 9 2024-03-19T11:30:05.339 INFO:tasks.rados.rados.0.smithi038.stdout:44: oid 44 version 9 is already newer than 7 2024-03-19T11:30:05.339 INFO:tasks.rados.rados.0.smithi038.stdout:44: oid 44 version 9 is already newer than 8 2024-03-19T11:30:05.339 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 44 v 9 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-03-19T11:30:05.340 INFO:tasks.rados.rados.0.smithi038.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2024-03-19T11:30:05.340 INFO:tasks.rados.rados.0.smithi038.stdout:45: finishing write tid 1 to smithi03829856-45 2024-03-19T11:30:05.340 INFO:tasks.rados.rados.0.smithi038.stdout:45: finishing write tid 2 to smithi03829856-45 2024-03-19T11:30:05.340 INFO:tasks.rados.rados.0.smithi038.stdout:45: finishing write tid 3 to smithi03829856-45 2024-03-19T11:30:05.340 INFO:tasks.rados.rados.0.smithi038.stdout:45: finishing write tid 4 to smithi03829856-45 2024-03-19T11:30:05.340 INFO:tasks.rados.rados.0.smithi038.stdout:45: oid 45 updating version 0 to 10 2024-03-19T11:30:05.340 INFO:tasks.rados.rados.0.smithi038.stdout:45: oid 45 updating version 10 to 11 2024-03-19T11:30:05.340 INFO:tasks.rados.rados.0.smithi038.stdout:45: oid 45 updating version 11 to 12 2024-03-19T11:30:05.340 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 45 v 12 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-03-19T11:30:05.340 INFO:tasks.rados.rados.0.smithi038.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2024-03-19T11:30:05.340 INFO:tasks.rados.rados.0.smithi038.stdout:46: finishing write tid 1 to smithi03829856-46 2024-03-19T11:30:05.340 INFO:tasks.rados.rados.0.smithi038.stdout:46: finishing write tid 2 to smithi03829856-46 2024-03-19T11:30:05.340 INFO:tasks.rados.rados.0.smithi038.stdout:46: finishing write tid 3 to smithi03829856-46 2024-03-19T11:30:05.340 INFO:tasks.rados.rados.0.smithi038.stdout:46: finishing write tid 4 to smithi03829856-46 2024-03-19T11:30:05.340 INFO:tasks.rados.rados.0.smithi038.stdout:46: oid 46 updating version 0 to 12 2024-03-19T11:30:05.340 INFO:tasks.rados.rados.0.smithi038.stdout:46: oid 46 version 12 is already newer than 10 2024-03-19T11:30:05.341 INFO:tasks.rados.rados.0.smithi038.stdout:46: oid 46 version 12 is already newer than 11 2024-03-19T11:30:05.341 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 46 v 12 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-03-19T11:30:05.341 INFO:tasks.rados.rados.0.smithi038.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2024-03-19T11:30:05.341 INFO:tasks.rados.rados.0.smithi038.stdout:47: finishing write tid 1 to smithi03829856-47 2024-03-19T11:30:05.341 INFO:tasks.rados.rados.0.smithi038.stdout:47: finishing write tid 2 to smithi03829856-47 2024-03-19T11:30:05.341 INFO:tasks.rados.rados.0.smithi038.stdout:47: finishing write tid 3 to smithi03829856-47 2024-03-19T11:30:05.341 INFO:tasks.rados.rados.0.smithi038.stdout:47: finishing write tid 4 to smithi03829856-47 2024-03-19T11:30:05.341 INFO:tasks.rados.rados.0.smithi038.stdout:47: oid 47 updating version 0 to 13 2024-03-19T11:30:05.341 INFO:tasks.rados.rados.0.smithi038.stdout:47: oid 47 updating version 13 to 14 2024-03-19T11:30:05.341 INFO:tasks.rados.rados.0.smithi038.stdout:47: oid 47 updating version 14 to 15 2024-03-19T11:30:05.341 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 47 v 15 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-03-19T11:30:05.341 INFO:tasks.rados.rados.0.smithi038.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2024-03-19T11:30:05.341 INFO:tasks.rados.rados.0.smithi038.stdout:48: finishing write tid 1 to smithi03829856-48 2024-03-19T11:30:05.341 INFO:tasks.rados.rados.0.smithi038.stdout:48: finishing write tid 2 to smithi03829856-48 2024-03-19T11:30:05.341 INFO:tasks.rados.rados.0.smithi038.stdout:48: finishing write tid 3 to smithi03829856-48 2024-03-19T11:30:05.342 INFO:tasks.rados.rados.0.smithi038.stdout:48: finishing write tid 4 to smithi03829856-48 2024-03-19T11:30:05.342 INFO:tasks.rados.rados.0.smithi038.stdout:48: oid 48 updating version 0 to 10 2024-03-19T11:30:05.342 INFO:tasks.rados.rados.0.smithi038.stdout:48: oid 48 updating version 10 to 11 2024-03-19T11:30:05.342 INFO:tasks.rados.rados.0.smithi038.stdout:48: oid 48 updating version 11 to 12 2024-03-19T11:30:05.342 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 48 v 12 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-03-19T11:30:05.342 INFO:tasks.rados.rados.0.smithi038.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2024-03-19T11:30:05.342 INFO:tasks.rados.rados.0.smithi038.stdout:49: finishing write tid 1 to smithi03829856-49 2024-03-19T11:30:05.342 INFO:tasks.rados.rados.0.smithi038.stdout:49: finishing write tid 2 to smithi03829856-49 2024-03-19T11:30:05.342 INFO:tasks.rados.rados.0.smithi038.stdout:49: finishing write tid 3 to smithi03829856-49 2024-03-19T11:30:05.342 INFO:tasks.rados.rados.0.smithi038.stdout:49: finishing write tid 4 to smithi03829856-49 2024-03-19T11:30:05.342 INFO:tasks.rados.rados.0.smithi038.stdout:49: oid 49 updating version 0 to 8 2024-03-19T11:30:05.342 INFO:tasks.rados.rados.0.smithi038.stdout:49: oid 49 version 8 is already newer than 7 2024-03-19T11:30:05.342 INFO:tasks.rados.rados.0.smithi038.stdout:49: oid 49 updating version 8 to 9 2024-03-19T11:30:05.343 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 49 v 9 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-03-19T11:30:05.343 INFO:tasks.rados.rados.0.smithi038.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2024-03-19T11:30:05.343 INFO:tasks.rados.rados.0.smithi038.stdout:50: finishing write tid 1 to smithi03829856-50 2024-03-19T11:30:05.343 INFO:tasks.rados.rados.0.smithi038.stdout:39: done (15 left) 2024-03-19T11:30:05.343 INFO:tasks.rados.rados.0.smithi038.stdout:40: done (14 left) 2024-03-19T11:30:05.343 INFO:tasks.rados.rados.0.smithi038.stdout:41: done (13 left) 2024-03-19T11:30:05.343 INFO:tasks.rados.rados.0.smithi038.stdout:42: done (12 left) 2024-03-19T11:30:05.343 INFO:tasks.rados.rados.0.smithi038.stdout:43: done (11 left) 2024-03-19T11:30:05.343 INFO:tasks.rados.rados.0.smithi038.stdout:44: done (10 left) 2024-03-19T11:30:05.343 INFO:tasks.rados.rados.0.smithi038.stdout:45: done (9 left) 2024-03-19T11:30:05.344 INFO:tasks.rados.rados.0.smithi038.stdout:46: done (8 left) 2024-03-19T11:30:05.344 INFO:tasks.rados.rados.0.smithi038.stdout:47: done (7 left) 2024-03-19T11:30:05.344 INFO:tasks.rados.rados.0.smithi038.stdout:48: done (6 left) 2024-03-19T11:30:05.344 INFO:tasks.rados.rados.0.smithi038.stdout:49: done (5 left) 2024-03-19T11:30:05.344 INFO:tasks.rados.rados.0.smithi038.stdout:55: copy oid 6 target oid 6 low_tier 2024-03-19T11:30:05.344 INFO:tasks.rados.rados.0.smithi038.stdout:copy op oid 6 to 6 low_tier tgt_pool_name low_tier 2024-03-19T11:30:05.344 INFO:tasks.rados.rados.0.smithi038.stdout:56: copy oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.344 INFO:tasks.rados.rados.0.smithi038.stdout:copy op oid 7 to 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-03-19T11:30:05.344 INFO:tasks.rados.rados.0.smithi038.stdout:57: copy oid 8 target oid 8 low_tier 2024-03-19T11:30:05.344 INFO:tasks.rados.rados.0.smithi038.stdout:copy op oid 8 to 8 low_tier tgt_pool_name low_tier 2024-03-19T11:30:05.344 INFO:tasks.rados.rados.0.smithi038.stdout:58: copy oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.344 INFO:tasks.rados.rados.0.smithi038.stdout:copy op oid 9 to 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-03-19T11:30:05.344 INFO:tasks.rados.rados.0.smithi038.stdout:59: copy oid 10 target oid 10 low_tier 2024-03-19T11:30:05.344 INFO:tasks.rados.rados.0.smithi038.stdout:copy op oid 10 to 10 low_tier tgt_pool_name low_tier 2024-03-19T11:30:05.345 INFO:tasks.rados.rados.0.smithi038.stdout:60: copy oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.345 INFO:tasks.rados.rados.0.smithi038.stdout:copy op oid 11 to 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-03-19T11:30:05.345 INFO:tasks.rados.rados.0.smithi038.stdout:61: copy oid 12 target oid 12 low_tier 2024-03-19T11:30:05.345 INFO:tasks.rados.rados.0.smithi038.stdout:copy op oid 12 to 12 low_tier tgt_pool_name low_tier 2024-03-19T11:30:05.345 INFO:tasks.rados.rados.0.smithi038.stdout:62: copy oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.345 INFO:tasks.rados.rados.0.smithi038.stdout:copy op oid 13 to 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-03-19T11:30:05.345 INFO:tasks.rados.rados.0.smithi038.stdout:63: copy oid 14 target oid 14 low_tier 2024-03-19T11:30:05.345 INFO:tasks.rados.rados.0.smithi038.stdout:copy op oid 14 to 14 low_tier tgt_pool_name low_tier 2024-03-19T11:30:05.345 INFO:tasks.rados.rados.0.smithi038.stdout:64: copy oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.345 INFO:tasks.rados.rados.0.smithi038.stdout:copy op oid 15 to 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-03-19T11:30:05.345 INFO:tasks.rados.rados.0.smithi038.stdout:65: copy oid 16 target oid 16 low_tier 2024-03-19T11:30:05.345 INFO:tasks.rados.rados.0.smithi038.stdout:copy op oid 16 to 16 low_tier tgt_pool_name low_tier 2024-03-19T11:30:05.345 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.346 INFO:tasks.rados.rados.0.smithi038.stdout:50: finishing write tid 2 to smithi03829856-50 2024-03-19T11:30:05.358 INFO:tasks.rados.rados.0.smithi038.stdout:50: finishing write tid 3 to smithi03829856-50 2024-03-19T11:30:05.358 INFO:tasks.rados.rados.0.smithi038.stdout:50: finishing write tid 4 to smithi03829856-50 2024-03-19T11:30:05.358 INFO:tasks.rados.rados.0.smithi038.stdout:50: oid 50 updating version 0 to 13 2024-03-19T11:30:05.358 INFO:tasks.rados.rados.0.smithi038.stdout:50: oid 50 updating version 13 to 14 2024-03-19T11:30:05.358 INFO:tasks.rados.rados.0.smithi038.stdout:50: oid 50 updating version 14 to 15 2024-03-19T11:30:05.358 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 50 v 15 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-03-19T11:30:05.358 INFO:tasks.rados.rados.0.smithi038.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2024-03-19T11:30:05.358 INFO:tasks.rados.rados.0.smithi038.stdout:50: done (15 left) 2024-03-19T11:30:05.358 INFO:tasks.rados.rados.0.smithi038.stdout:66: copy oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.358 INFO:tasks.rados.rados.0.smithi038.stdout:copy op oid 17 to 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-03-19T11:30:05.358 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.367 INFO:tasks.rados.rados.0.smithi038.stdout:64: finishing copy op to oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.367 INFO:tasks.rados.rados.0.smithi038.stdout:64: done (15 left) 2024-03-19T11:30:05.367 INFO:tasks.rados.rados.0.smithi038.stdout:67: copy oid 18 target oid 18 low_tier 2024-03-19T11:30:05.367 INFO:tasks.rados.rados.0.smithi038.stdout:copy op oid 18 to 18 low_tier tgt_pool_name low_tier 2024-03-19T11:30:05.367 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.367 INFO:tasks.rados.rados.0.smithi038.stdout:57: finishing copy op to oid 8 low_tier 2024-03-19T11:30:05.368 INFO:tasks.rados.rados.0.smithi038.stdout:57: done (15 left) 2024-03-19T11:30:05.368 INFO:tasks.rados.rados.0.smithi038.stdout:68: copy oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.368 INFO:tasks.rados.rados.0.smithi038.stdout:copy op oid 19 to 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-03-19T11:30:05.368 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.368 INFO:tasks.rados.rados.0.smithi038.stdout:61: finishing copy op to oid 12 low_tier 2024-03-19T11:30:05.368 INFO:tasks.rados.rados.0.smithi038.stdout:61: done (15 left) 2024-03-19T11:30:05.368 INFO:tasks.rados.rados.0.smithi038.stdout:69: copy oid 20 target oid 20 low_tier 2024-03-19T11:30:05.368 INFO:tasks.rados.rados.0.smithi038.stdout:copy op oid 20 to 20 low_tier tgt_pool_name low_tier 2024-03-19T11:30:05.368 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.368 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:05.372+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:05.369 INFO:tasks.rados.rados.0.smithi038.stdout:55: finishing copy op to oid 6 low_tier 2024-03-19T11:30:05.369 INFO:tasks.rados.rados.0.smithi038.stdout:55: done (15 left) 2024-03-19T11:30:05.369 INFO:tasks.rados.rados.0.smithi038.stdout:70: copy oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.370 INFO:tasks.rados.rados.0.smithi038.stdout:copy op oid 21 to 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-03-19T11:30:05.370 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.370 INFO:tasks.rados.rados.0.smithi038.stdout:54: finishing copy op to oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.370 INFO:tasks.rados.rados.0.smithi038.stdout:54: done (15 left) 2024-03-19T11:30:05.370 INFO:tasks.rados.rados.0.smithi038.stdout:71: copy oid 22 target oid 22 low_tier 2024-03-19T11:30:05.370 INFO:tasks.rados.rados.0.smithi038.stdout:copy op oid 22 to 22 low_tier tgt_pool_name low_tier 2024-03-19T11:30:05.370 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.378 INFO:tasks.rados.rados.0.smithi038.stdout:53: finishing copy op to oid 4 low_tier 2024-03-19T11:30:05.379 INFO:tasks.rados.rados.0.smithi038.stdout:66: finishing copy op to oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.380 INFO:tasks.rados.rados.0.smithi038.stdout:71: finishing copy op to oid 22 low_tier 2024-03-19T11:30:05.380 INFO:tasks.rados.rados.0.smithi038.stdout:53: done (15 left) 2024-03-19T11:30:05.380 INFO:tasks.rados.rados.0.smithi038.stdout:66: done (14 left) 2024-03-19T11:30:05.380 INFO:tasks.rados.rados.0.smithi038.stdout:71: done (13 left) 2024-03-19T11:30:05.380 INFO:tasks.rados.rados.0.smithi038.stdout:72: copy oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.380 INFO:tasks.rados.rados.0.smithi038.stdout:copy op oid 23 to 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-03-19T11:30:05.380 INFO:tasks.rados.rados.0.smithi038.stdout:73: copy oid 24 target oid 24 low_tier 2024-03-19T11:30:05.380 INFO:tasks.rados.rados.0.smithi038.stdout:copy op oid 24 to 24 low_tier tgt_pool_name low_tier 2024-03-19T11:30:05.380 INFO:tasks.rados.rados.0.smithi038.stdout:74: copy oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.380 INFO:tasks.rados.rados.0.smithi038.stdout:copy op oid 25 to 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-03-19T11:30:05.380 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.381 INFO:tasks.rados.rados.0.smithi038.stdout:69: finishing copy op to oid 20 low_tier 2024-03-19T11:30:05.381 INFO:tasks.rados.rados.0.smithi038.stdout:69: done (15 left) 2024-03-19T11:30:05.381 INFO:tasks.rados.rados.0.smithi038.stdout:75: copy oid 26 target oid 26 low_tier 2024-03-19T11:30:05.381 INFO:tasks.rados.rados.0.smithi038.stdout:copy op oid 26 to 26 low_tier tgt_pool_name low_tier 2024-03-19T11:30:05.381 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.383 INFO:tasks.rados.rados.0.smithi038.stdout:63: finishing copy op to oid 14 low_tier 2024-03-19T11:30:05.383 INFO:tasks.rados.rados.0.smithi038.stdout:63: done (15 left) 2024-03-19T11:30:05.383 INFO:tasks.rados.rados.0.smithi038.stdout:76: copy oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.383 INFO:tasks.rados.rados.0.smithi038.stdout:copy op oid 27 to 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-03-19T11:30:05.383 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.383 INFO:tasks.rados.rados.0.smithi038.stdout:51: finishing copy op to oid 2 low_tier 2024-03-19T11:30:05.383 INFO:tasks.rados.rados.0.smithi038.stdout:51: done (15 left) 2024-03-19T11:30:05.383 INFO:tasks.rados.rados.0.smithi038.stdout:77: copy oid 28 target oid 28 low_tier 2024-03-19T11:30:05.383 INFO:tasks.rados.rados.0.smithi038.stdout:copy op oid 28 to 28 low_tier tgt_pool_name low_tier 2024-03-19T11:30:05.383 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.387 INFO:tasks.rados.rados.0.smithi038.stdout:60: finishing copy op to oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.387 INFO:tasks.rados.rados.0.smithi038.stdout:56: finishing copy op to oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.387 INFO:tasks.rados.rados.0.smithi038.stdout:56: done (15 left) 2024-03-19T11:30:05.387 INFO:tasks.rados.rados.0.smithi038.stdout:60: done (14 left) 2024-03-19T11:30:05.387 INFO:tasks.rados.rados.0.smithi038.stdout:78: copy oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.387 INFO:tasks.rados.rados.0.smithi038.stdout:copy op oid 29 to 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-03-19T11:30:05.387 INFO:tasks.rados.rados.0.smithi038.stdout:79: copy oid 30 target oid 30 low_tier 2024-03-19T11:30:05.387 INFO:tasks.rados.rados.0.smithi038.stdout:copy op oid 30 to 30 low_tier tgt_pool_name low_tier 2024-03-19T11:30:05.387 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.387 INFO:tasks.rados.rados.0.smithi038.stdout:65: finishing copy op to oid 16 low_tier 2024-03-19T11:30:05.387 INFO:tasks.rados.rados.0.smithi038.stdout:65: done (15 left) 2024-03-19T11:30:05.388 INFO:tasks.rados.rados.0.smithi038.stdout:80: copy oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.388 INFO:tasks.rados.rados.0.smithi038.stdout:copy op oid 31 to 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-03-19T11:30:05.388 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.388 INFO:tasks.rados.rados.0.smithi038.stdout:59: finishing copy op to oid 10 low_tier 2024-03-19T11:30:05.388 INFO:tasks.rados.rados.0.smithi038.stdout:59: done (15 left) 2024-03-19T11:30:05.389 INFO:tasks.rados.rados.0.smithi038.stdout:81: copy oid 32 target oid 32 low_tier 2024-03-19T11:30:05.389 INFO:tasks.rados.rados.0.smithi038.stdout:copy op oid 32 to 32 low_tier tgt_pool_name low_tier 2024-03-19T11:30:05.389 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.399 INFO:tasks.rados.rados.0.smithi038.stdout:62: finishing copy op to oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.399 INFO:tasks.rados.rados.0.smithi038.stdout:62: done (15 left) 2024-03-19T11:30:05.399 INFO:tasks.rados.rados.0.smithi038.stdout:82: copy oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.399 INFO:tasks.rados.rados.0.smithi038.stdout:copy op oid 33 to 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-03-19T11:30:05.400 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.402 INFO:tasks.rados.rados.0.smithi038.stdout:73: finishing copy op to oid 24 low_tier 2024-03-19T11:30:05.402 INFO:tasks.rados.rados.0.smithi038.stdout:73: done (15 left) 2024-03-19T11:30:05.402 INFO:tasks.rados.rados.0.smithi038.stdout:83: copy oid 34 target oid 34 low_tier 2024-03-19T11:30:05.402 INFO:tasks.rados.rados.0.smithi038.stdout:copy op oid 34 to 34 low_tier tgt_pool_name low_tier 2024-03-19T11:30:05.402 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.406 INFO:tasks.rados.rados.0.smithi038.stdout:70: finishing copy op to oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.406 INFO:tasks.rados.rados.0.smithi038.stdout:70: done (15 left) 2024-03-19T11:30:05.407 INFO:tasks.rados.rados.0.smithi038.stdout:84: copy oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.407 INFO:tasks.rados.rados.0.smithi038.stdout:copy op oid 35 to 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-03-19T11:30:05.407 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.409 INFO:tasks.rados.rados.0.smithi038.stdout:75: finishing copy op to oid 26 low_tier 2024-03-19T11:30:05.409 INFO:tasks.rados.rados.0.smithi038.stdout:75: done (15 left) 2024-03-19T11:30:05.409 INFO:tasks.rados.rados.0.smithi038.stdout:85: copy oid 36 target oid 36 low_tier 2024-03-19T11:30:05.409 INFO:tasks.rados.rados.0.smithi038.stdout:copy op oid 36 to 36 low_tier tgt_pool_name low_tier 2024-03-19T11:30:05.409 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.410 INFO:tasks.rados.rados.0.smithi038.stdout:78: finishing copy op to oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.410 INFO:tasks.rados.rados.0.smithi038.stdout:78: done (15 left) 2024-03-19T11:30:05.411 INFO:tasks.rados.rados.0.smithi038.stdout:86: copy oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.411 INFO:tasks.rados.rados.0.smithi038.stdout:copy op oid 37 to 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-03-19T11:30:05.411 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.411 INFO:tasks.rados.rados.0.smithi038.stdout:81: finishing copy op to oid 32 low_tier 2024-03-19T11:30:05.411 INFO:tasks.rados.rados.0.smithi038.stdout:81: done (15 left) 2024-03-19T11:30:05.411 INFO:tasks.rados.rados.0.smithi038.stdout:87: copy oid 38 target oid 38 low_tier 2024-03-19T11:30:05.411 INFO:tasks.rados.rados.0.smithi038.stdout:copy op oid 38 to 38 low_tier tgt_pool_name low_tier 2024-03-19T11:30:05.411 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.412 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:05.412 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:05.417 INFO:tasks.rados.rados.0.smithi038.stdout:74: finishing copy op to oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.417 INFO:tasks.rados.rados.0.smithi038.stdout:74: done (15 left) 2024-03-19T11:30:05.417 INFO:tasks.rados.rados.0.smithi038.stdout:88: copy oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.417 INFO:tasks.rados.rados.0.smithi038.stdout:copy op oid 39 to 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-03-19T11:30:05.417 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.417 INFO:tasks.rados.rados.0.smithi038.stdout:52: finishing copy op to oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.417 INFO:tasks.rados.rados.0.smithi038.stdout:52: done (15 left) 2024-03-19T11:30:05.418 INFO:tasks.rados.rados.0.smithi038.stdout:89: copy oid 40 target oid 40 low_tier 2024-03-19T11:30:05.418 INFO:tasks.rados.rados.0.smithi038.stdout:copy op oid 40 to 40 low_tier tgt_pool_name low_tier 2024-03-19T11:30:05.418 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.418 INFO:tasks.rados.rados.0.smithi038.stdout:58: finishing copy op to oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.418 INFO:tasks.rados.rados.0.smithi038.stdout:58: done (15 left) 2024-03-19T11:30:05.418 INFO:tasks.rados.rados.0.smithi038.stdout:90: copy oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.418 INFO:tasks.rados.rados.0.smithi038.stdout:copy op oid 41 to 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-03-19T11:30:05.418 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.419 INFO:tasks.rados.rados.0.smithi038.stdout:76: finishing copy op to oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.419 INFO:tasks.rados.rados.0.smithi038.stdout:76: done (15 left) 2024-03-19T11:30:05.419 INFO:tasks.rados.rados.0.smithi038.stdout:91: copy oid 42 target oid 42 low_tier 2024-03-19T11:30:05.419 INFO:tasks.rados.rados.0.smithi038.stdout:copy op oid 42 to 42 low_tier tgt_pool_name low_tier 2024-03-19T11:30:05.419 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.423 INFO:tasks.rados.rados.0.smithi038.stdout:82: finishing copy op to oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.423 INFO:tasks.rados.rados.0.smithi038.stdout:82: done (15 left) 2024-03-19T11:30:05.423 INFO:tasks.rados.rados.0.smithi038.stdout:92: copy oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.423 INFO:tasks.rados.rados.0.smithi038.stdout:copy op oid 43 to 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-03-19T11:30:05.423 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.428 INFO:tasks.rados.rados.0.smithi038.stdout:83: finishing copy op to oid 34 low_tier 2024-03-19T11:30:05.429 INFO:tasks.rados.rados.0.smithi038.stdout:83: done (15 left) 2024-03-19T11:30:05.429 INFO:tasks.rados.rados.0.smithi038.stdout:93: copy oid 44 target oid 44 low_tier 2024-03-19T11:30:05.429 INFO:tasks.rados.rados.0.smithi038.stdout:copy op oid 44 to 44 low_tier tgt_pool_name low_tier 2024-03-19T11:30:05.429 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.431 INFO:tasks.rados.rados.0.smithi038.stdout:77: finishing copy op to oid 28 low_tier 2024-03-19T11:30:05.431 INFO:tasks.rados.rados.0.smithi038.stdout:72: finishing copy op to oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.431 INFO:tasks.rados.rados.0.smithi038.stdout:72: done (15 left) 2024-03-19T11:30:05.431 INFO:tasks.rados.rados.0.smithi038.stdout:77: done (14 left) 2024-03-19T11:30:05.432 INFO:tasks.rados.rados.0.smithi038.stdout:94: copy oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.432 INFO:tasks.rados.rados.0.smithi038.stdout:copy op oid 45 to 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-03-19T11:30:05.432 INFO:tasks.rados.rados.0.smithi038.stdout:95: copy oid 46 target oid 46 low_tier 2024-03-19T11:30:05.432 INFO:tasks.rados.rados.0.smithi038.stdout:copy op oid 46 to 46 low_tier tgt_pool_name low_tier 2024-03-19T11:30:05.432 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.434 INFO:tasks.rados.rados.0.smithi038.stdout:86: finishing copy op to oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.434 INFO:tasks.rados.rados.0.smithi038.stdout:86: done (15 left) 2024-03-19T11:30:05.434 INFO:tasks.rados.rados.0.smithi038.stdout:96: copy oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.434 INFO:tasks.rados.rados.0.smithi038.stdout:copy op oid 47 to 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-03-19T11:30:05.434 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.436 INFO:tasks.rados.rados.0.smithi038.stdout:80: finishing copy op to oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.436 INFO:tasks.rados.rados.0.smithi038.stdout:80: done (15 left) 2024-03-19T11:30:05.437 INFO:tasks.rados.rados.0.smithi038.stdout:97: copy oid 48 target oid 48 low_tier 2024-03-19T11:30:05.437 INFO:tasks.rados.rados.0.smithi038.stdout:copy op oid 48 to 48 low_tier tgt_pool_name low_tier 2024-03-19T11:30:05.437 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.437 INFO:tasks.rados.rados.0.smithi038.stdout:84: finishing copy op to oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.437 INFO:tasks.rados.rados.0.smithi038.stdout:84: done (15 left) 2024-03-19T11:30:05.437 INFO:tasks.rados.rados.0.smithi038.stdout:98: copy oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.437 INFO:tasks.rados.rados.0.smithi038.stdout:copy op oid 49 to 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-03-19T11:30:05.437 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.438 INFO:tasks.rados.rados.0.smithi038.stdout:91: finishing copy op to oid 42 low_tier 2024-03-19T11:30:05.438 INFO:tasks.rados.rados.0.smithi038.stdout:91: done (15 left) 2024-03-19T11:30:05.438 INFO:tasks.rados.rados.0.smithi038.stdout:99: copy oid 50 target oid 50 low_tier 2024-03-19T11:30:05.438 INFO:tasks.rados.rados.0.smithi038.stdout:copy op oid 50 to 50 low_tier tgt_pool_name low_tier 2024-03-19T11:30:05.438 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.439 INFO:tasks.rados.rados.0.smithi038.stdout:90: finishing copy op to oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.439 INFO:tasks.rados.rados.0.smithi038.stdout:90: done (15 left) 2024-03-19T11:30:05.439 INFO:tasks.rados.rados.0.smithi038.stdout:100: copy oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.439 INFO:tasks.rados.rados.0.smithi038.stdout:copy op oid 1 to 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-03-19T11:30:05.439 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.444 INFO:tasks.rados.rados.0.smithi038.stdout:92: finishing copy op to oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.444 INFO:tasks.rados.rados.0.smithi038.stdout:92: done (15 left) 2024-03-19T11:30:05.444 INFO:tasks.rados.rados.0.smithi038.stdout:101: set_chunk oid 2 target oid 2 low_tier 2024-03-19T11:30:05.444 INFO:tasks.rados.rados.0.smithi038.stdout:101: set_chunk oid 2 offset: 321536 length: 1537024 target oid smithi03829856-2 low_tier offset: 321536 snap -1 2024-03-19T11:30:05.444 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.447 INFO:tasks.rados.rados.0.smithi038.stdout:68: finishing copy op to oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.447 INFO:tasks.rados.rados.0.smithi038.stdout:68: done (15 left) 2024-03-19T11:30:05.447 INFO:tasks.rados.rados.0.smithi038.stdout:102: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.447 INFO:tasks.rados.rados.0.smithi038.stdout:102: set_chunk oid 3 offset: 899584 length: 936960 target oid smithi03829856-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 899584 snap -1 2024-03-19T11:30:05.447 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.447 INFO:tasks.rados.rados.0.smithi038.stdout:79: finishing copy op to oid 30 low_tier 2024-03-19T11:30:05.447 INFO:tasks.rados.rados.0.smithi038.stdout:67: finishing copy op to oid 18 low_tier 2024-03-19T11:30:05.447 INFO:tasks.rados.rados.0.smithi038.stdout:67: done (15 left) 2024-03-19T11:30:05.447 INFO:tasks.rados.rados.0.smithi038.stdout:79: done (14 left) 2024-03-19T11:30:05.447 INFO:tasks.rados.rados.0.smithi038.stdout:103: set_chunk oid 4 target oid 4 low_tier 2024-03-19T11:30:05.447 INFO:tasks.rados.rados.0.smithi038.stdout:103: set_chunk oid 4 offset: 229888 length: 1472000 target oid smithi03829856-4 low_tier offset: 229888 snap -1 2024-03-19T11:30:05.447 INFO:tasks.rados.rados.0.smithi038.stdout:104: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.448 INFO:tasks.rados.rados.0.smithi038.stdout:104: set_chunk oid 5 offset: 299520 length: 1748480 target oid smithi03829856-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 299520 snap -1 2024-03-19T11:30:05.448 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.449 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-19T11:30:05.451 INFO:tasks.rados.rados.0.smithi038.stdout:88: finishing copy op to oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.451 INFO:tasks.rados.rados.0.smithi038.stdout:88: done (15 left) 2024-03-19T11:30:05.451 INFO:tasks.rados.rados.0.smithi038.stdout:105: set_chunk oid 6 target oid 6 low_tier 2024-03-19T11:30:05.451 INFO:tasks.rados.rados.0.smithi038.stdout:105: set_chunk oid 6 offset: 1276416 length: 60928 target oid smithi03829856-6 low_tier offset: 1276416 snap -1 2024-03-19T11:30:05.451 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.453 INFO:tasks.rados.rados.0.smithi038.stdout:95: finishing copy op to oid 46 low_tier 2024-03-19T11:30:05.453 INFO:tasks.rados.rados.0.smithi038.stdout:95: done (15 left) 2024-03-19T11:30:05.453 INFO:tasks.rados.rados.0.smithi038.stdout:106: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.453 INFO:tasks.rados.rados.0.smithi038.stdout:106: set_chunk oid 7 offset: 823296 length: 735232 target oid smithi03829856-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 823296 snap -1 2024-03-19T11:30:05.453 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.454 INFO:tasks.rados.rados.0.smithi038.stdout:97: finishing copy op to oid 48 low_tier 2024-03-19T11:30:05.454 INFO:tasks.rados.rados.0.smithi038.stdout:97: done (15 left) 2024-03-19T11:30:05.454 INFO:tasks.rados.rados.0.smithi038.stdout:107: set_chunk oid 8 target oid 8 low_tier 2024-03-19T11:30:05.454 INFO:tasks.rados.rados.0.smithi038.stdout:107: set_chunk oid 8 offset: 529920 length: 869888 target oid smithi03829856-8 low_tier offset: 529920 snap -1 2024-03-19T11:30:05.455 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.460 INFO:tasks.rados.rados.0.smithi038.stdout:100: finishing copy op to oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.460 INFO:tasks.rados.rados.0.smithi038.stdout:100: done (15 left) 2024-03-19T11:30:05.460 INFO:tasks.rados.rados.0.smithi038.stdout:108: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.460 INFO:tasks.rados.rados.0.smithi038.stdout:108: set_chunk oid 9 offset: 958464 length: 208384 target oid smithi03829856-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 958464 snap -1 2024-03-19T11:30:05.460 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.462 INFO:tasks.rados.rados.0.smithi038.stdout:87: finishing copy op to oid 38 low_tier 2024-03-19T11:30:05.462 INFO:tasks.rados.rados.0.smithi038.stdout:87: done (15 left) 2024-03-19T11:30:05.462 INFO:tasks.rados.rados.0.smithi038.stdout:109: set_chunk oid 10 target oid 10 low_tier 2024-03-19T11:30:05.462 INFO:tasks.rados.rados.0.smithi038.stdout:109: set_chunk oid 10 offset: 780800 length: 539648 target oid smithi03829856-10 low_tier offset: 780800 snap -1 2024-03-19T11:30:05.462 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.464 INFO:tasks.rados.rados.0.smithi038.stdout:107: finishing set_chunk to oid 8 2024-03-19T11:30:05.464 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 8 v 3 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-03-19T11:30:05.464 INFO:tasks.rados.rados.0.smithi038.stdout:107: done (15 left) 2024-03-19T11:30:05.464 INFO:tasks.rados.rados.0.smithi038.stdout:110: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.464 INFO:tasks.rados.rados.0.smithi038.stdout:110: set_chunk oid 11 offset: 540672 length: 1238016 target oid smithi03829856-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 540672 snap -1 2024-03-19T11:30:05.464 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.464 INFO:tasks.rados.rados.0.smithi038.stdout:93: finishing copy op to oid 44 low_tier 2024-03-19T11:30:05.464 INFO:tasks.rados.rados.0.smithi038.stdout:93: done (15 left) 2024-03-19T11:30:05.464 INFO:tasks.rados.rados.0.smithi038.stdout:111: set_chunk oid 12 target oid 12 low_tier 2024-03-19T11:30:05.465 INFO:tasks.rados.rados.0.smithi038.stdout:111: set_chunk oid 12 offset: 240128 length: 332288 target oid smithi03829856-12 low_tier offset: 240128 snap -1 2024-03-19T11:30:05.465 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.465 INFO:tasks.rados.rados.0.smithi038.stdout:104: finishing set_chunk to oid 5 2024-03-19T11:30:05.465 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 5 v 3 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-03-19T11:30:05.465 INFO:tasks.rados.rados.0.smithi038.stdout:104: done (15 left) 2024-03-19T11:30:05.465 INFO:tasks.rados.rados.0.smithi038.stdout:112: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.465 INFO:tasks.rados.rados.0.smithi038.stdout:112: set_chunk oid 13 offset: 72704 length: 1255424 target oid smithi03829856-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 72704 snap -1 2024-03-19T11:30:05.465 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.467 INFO:tasks.rados.rados.0.smithi038.stdout:98: finishing copy op to oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.468 INFO:tasks.rados.rados.0.smithi038.stdout:98: done (15 left) 2024-03-19T11:30:05.468 INFO:tasks.rados.rados.0.smithi038.stdout:113: set_chunk oid 14 target oid 14 low_tier 2024-03-19T11:30:05.468 INFO:tasks.rados.rados.0.smithi038.stdout:113: set_chunk oid 14 offset: 1027072 length: 599552 target oid smithi03829856-14 low_tier offset: 1027072 snap -1 2024-03-19T11:30:05.468 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.468 INFO:tasks.rados.rados.0.smithi038.stdout:105: finishing set_chunk to oid 6 2024-03-19T11:30:05.468 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 6 v 3 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-03-19T11:30:05.468 INFO:tasks.rados.rados.0.smithi038.stdout:105: done (15 left) 2024-03-19T11:30:05.468 INFO:tasks.rados.rados.0.smithi038.stdout:114: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.468 INFO:tasks.rados.rados.0.smithi038.stdout:114: set_chunk oid 15 offset: 540672 length: 1126912 target oid smithi03829856-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 540672 snap -1 2024-03-19T11:30:05.468 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.468 INFO:tasks.rados.rados.0.smithi038.stdout:89: finishing copy op to oid 40 low_tier 2024-03-19T11:30:05.468 INFO:tasks.rados.rados.0.smithi038.stdout:89: done (15 left) 2024-03-19T11:30:05.468 INFO:tasks.rados.rados.0.smithi038.stdout:115: set_chunk oid 16 target oid 16 low_tier 2024-03-19T11:30:05.468 INFO:tasks.rados.rados.0.smithi038.stdout:115: set_chunk oid 16 offset: 413184 length: 1648128 target oid smithi03829856-16 low_tier offset: 413184 snap -1 2024-03-19T11:30:05.468 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.469 INFO:tasks.rados.rados.0.smithi038.stdout:111: finishing set_chunk to oid 12 2024-03-19T11:30:05.469 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 12 v 3 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-03-19T11:30:05.469 INFO:tasks.rados.rados.0.smithi038.stdout:96: finishing copy op to oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.469 INFO:tasks.rados.rados.0.smithi038.stdout:96: done (15 left) 2024-03-19T11:30:05.469 INFO:tasks.rados.rados.0.smithi038.stdout:111: done (14 left) 2024-03-19T11:30:05.469 INFO:tasks.rados.rados.0.smithi038.stdout:116: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.469 INFO:tasks.rados.rados.0.smithi038.stdout:116: set_chunk oid 17 offset: 285696 length: 769536 target oid smithi03829856-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 285696 snap -1 2024-03-19T11:30:05.469 INFO:tasks.rados.rados.0.smithi038.stdout:117: set_chunk oid 18 target oid 18 low_tier 2024-03-19T11:30:05.469 INFO:tasks.rados.rados.0.smithi038.stdout:117: set_chunk oid 18 offset: 30720 length: 1831424 target oid smithi03829856-18 low_tier offset: 30720 snap -1 2024-03-19T11:30:05.469 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.472 INFO:tasks.rados.rados.0.smithi038.stdout:99: finishing copy op to oid 50 low_tier 2024-03-19T11:30:05.472 INFO:tasks.rados.rados.0.smithi038.stdout:99: done (15 left) 2024-03-19T11:30:05.472 INFO:tasks.rados.rados.0.smithi038.stdout:118: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.472 INFO:tasks.rados.rados.0.smithi038.stdout:118: set_chunk oid 19 offset: 1094656 length: 1051136 target oid smithi03829856-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1094656 snap -1 2024-03-19T11:30:05.472 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.472 INFO:tasks.rados.rados.0.smithi038.stdout:101: finishing set_chunk to oid 2 2024-03-19T11:30:05.472 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 2 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-03-19T11:30:05.472 INFO:tasks.rados.rados.0.smithi038.stdout:101: done (15 left) 2024-03-19T11:30:05.472 INFO:tasks.rados.rados.0.smithi038.stdout:119: set_chunk oid 20 target oid 20 low_tier 2024-03-19T11:30:05.472 INFO:tasks.rados.rados.0.smithi038.stdout:119: set_chunk oid 20 offset: 59904 length: 173568 target oid smithi03829856-20 low_tier offset: 59904 snap -1 2024-03-19T11:30:05.472 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.473 INFO:tasks.rados.rados.0.smithi038.stdout:103: finishing set_chunk to oid 4 2024-03-19T11:30:05.473 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 4 v 3 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-03-19T11:30:05.473 INFO:tasks.rados.rados.0.smithi038.stdout:103: done (15 left) 2024-03-19T11:30:05.473 INFO:tasks.rados.rados.0.smithi038.stdout:120: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.473 INFO:tasks.rados.rados.0.smithi038.stdout:120: set_chunk oid 21 offset: 960000 length: 273408 target oid smithi03829856-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 960000 snap -1 2024-03-19T11:30:05.473 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.473 INFO:tasks.rados.rados.0.smithi038.stdout:113: finishing set_chunk to oid 14 2024-03-19T11:30:05.473 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 14 v 3 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-03-19T11:30:05.473 INFO:tasks.rados.rados.0.smithi038.stdout:113: done (15 left) 2024-03-19T11:30:05.473 INFO:tasks.rados.rados.0.smithi038.stdout:121: set_chunk oid 22 target oid 22 low_tier 2024-03-19T11:30:05.473 INFO:tasks.rados.rados.0.smithi038.stdout:121: set_chunk oid 22 offset: 269824 length: 1462784 target oid smithi03829856-22 low_tier offset: 269824 snap -1 2024-03-19T11:30:05.473 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.473 INFO:tasks.rados.rados.0.smithi038.stdout:112: finishing set_chunk to oid 13 2024-03-19T11:30:05.473 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 13 v 6 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-03-19T11:30:05.473 INFO:tasks.rados.rados.0.smithi038.stdout:112: done (15 left) 2024-03-19T11:30:05.474 INFO:tasks.rados.rados.0.smithi038.stdout:122: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.474 INFO:tasks.rados.rados.0.smithi038.stdout:122: set_chunk oid 23 offset: 1157632 length: 1017856 target oid smithi03829856-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1157632 snap -1 2024-03-19T11:30:05.474 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.474 INFO:tasks.rados.rados.0.smithi038.stdout:85: finishing copy op to oid 36 low_tier 2024-03-19T11:30:05.474 INFO:tasks.rados.rados.0.smithi038.stdout:85: done (15 left) 2024-03-19T11:30:05.474 INFO:tasks.rados.rados.0.smithi038.stdout:123: set_chunk oid 24 target oid 24 low_tier 2024-03-19T11:30:05.474 INFO:tasks.rados.rados.0.smithi038.stdout:123: set_chunk oid 24 offset: 147968 length: 1257984 target oid smithi03829856-24 low_tier offset: 147968 snap -1 2024-03-19T11:30:05.474 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.474 INFO:tasks.rados.rados.0.smithi038.stdout:116: finishing set_chunk to oid 17 2024-03-19T11:30:05.474 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 17 v 9 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-03-19T11:30:05.474 INFO:tasks.rados.rados.0.smithi038.stdout:116: done (15 left) 2024-03-19T11:30:05.474 INFO:tasks.rados.rados.0.smithi038.stdout:124: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.474 INFO:tasks.rados.rados.0.smithi038.stdout:124: set_chunk oid 25 offset: 1724416 length: 110592 target oid smithi03829856-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1724416 snap -1 2024-03-19T11:30:05.474 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.474 INFO:tasks.rados.rados.0.smithi038.stdout:115: finishing set_chunk to oid 16 2024-03-19T11:30:05.475 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 16 v 6 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-03-19T11:30:05.475 INFO:tasks.rados.rados.0.smithi038.stdout:115: done (15 left) 2024-03-19T11:30:05.475 INFO:tasks.rados.rados.0.smithi038.stdout:125: set_chunk oid 26 target oid 26 low_tier 2024-03-19T11:30:05.475 INFO:tasks.rados.rados.0.smithi038.stdout:125: set_chunk oid 26 offset: 102400 length: 1281024 target oid smithi03829856-26 low_tier offset: 102400 snap -1 2024-03-19T11:30:05.475 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.477 INFO:tasks.rados.rados.0.smithi038.stdout:94: finishing copy op to oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.478 INFO:tasks.rados.rados.0.smithi038.stdout:94: done (15 left) 2024-03-19T11:30:05.478 INFO:tasks.rados.rados.0.smithi038.stdout:126: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.478 INFO:tasks.rados.rados.0.smithi038.stdout:126: set_chunk oid 27 offset: 593920 length: 748544 target oid smithi03829856-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 593920 snap -1 2024-03-19T11:30:05.478 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.478 INFO:tasks.rados.rados.0.smithi038.stdout:102: finishing set_chunk to oid 3 2024-03-19T11:30:05.478 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 3 v 3 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-03-19T11:30:05.478 INFO:tasks.rados.rados.0.smithi038.stdout:102: done (15 left) 2024-03-19T11:30:05.478 INFO:tasks.rados.rados.0.smithi038.stdout:127: set_chunk oid 28 target oid 28 low_tier 2024-03-19T11:30:05.478 INFO:tasks.rados.rados.0.smithi038.stdout:127: set_chunk oid 28 offset: 116224 length: 1920512 target oid smithi03829856-28 low_tier offset: 116224 snap -1 2024-03-19T11:30:05.478 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.478 INFO:tasks.rados.rados.0.smithi038.stdout:108: finishing set_chunk to oid 9 2024-03-19T11:30:05.478 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 9 v 6 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-03-19T11:30:05.478 INFO:tasks.rados.rados.0.smithi038.stdout:119: finishing set_chunk to oid 20 2024-03-19T11:30:05.478 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 20 v 12 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-03-19T11:30:05.479 INFO:tasks.rados.rados.0.smithi038.stdout:108: done (15 left) 2024-03-19T11:30:05.479 INFO:tasks.rados.rados.0.smithi038.stdout:119: done (14 left) 2024-03-19T11:30:05.479 INFO:tasks.rados.rados.0.smithi038.stdout:128: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.479 INFO:tasks.rados.rados.0.smithi038.stdout:128: set_chunk oid 29 offset: 2334208 length: 88576 target oid smithi03829856-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 2334208 snap -1 2024-03-19T11:30:05.479 INFO:tasks.rados.rados.0.smithi038.stdout:129: set_chunk oid 30 target oid 30 low_tier 2024-03-19T11:30:05.479 INFO:tasks.rados.rados.0.smithi038.stdout:129: set_chunk oid 30 offset: 404480 length: 701440 target oid smithi03829856-30 low_tier offset: 404480 snap -1 2024-03-19T11:30:05.479 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.479 INFO:tasks.rados.rados.0.smithi038.stdout:120: finishing set_chunk to oid 21 2024-03-19T11:30:05.479 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 21 v 15 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-03-19T11:30:05.479 INFO:tasks.rados.rados.0.smithi038.stdout:120: done (15 left) 2024-03-19T11:30:05.479 INFO:tasks.rados.rados.0.smithi038.stdout:130: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.479 INFO:tasks.rados.rados.0.smithi038.stdout:130: set_chunk oid 31 offset: 1084416 length: 105984 target oid smithi03829856-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1084416 snap -1 2024-03-19T11:30:05.479 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.479 INFO:tasks.rados.rados.0.smithi038.stdout:109: finishing set_chunk to oid 10 2024-03-19T11:30:05.479 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 10 v 6 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-03-19T11:30:05.480 INFO:tasks.rados.rados.0.smithi038.stdout:109: done (15 left) 2024-03-19T11:30:05.480 INFO:tasks.rados.rados.0.smithi038.stdout:131: set_chunk oid 32 target oid 32 low_tier 2024-03-19T11:30:05.480 INFO:tasks.rados.rados.0.smithi038.stdout:131: set_chunk oid 32 offset: 1563136 length: 434176 target oid smithi03829856-32 low_tier offset: 1563136 snap -1 2024-03-19T11:30:05.480 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.480 INFO:tasks.rados.rados.0.smithi038.stdout:106: finishing set_chunk to oid 7 2024-03-19T11:30:05.480 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 7 v 3 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-03-19T11:30:05.480 INFO:tasks.rados.rados.0.smithi038.stdout:106: done (15 left) 2024-03-19T11:30:05.480 INFO:tasks.rados.rados.0.smithi038.stdout:132: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.480 INFO:tasks.rados.rados.0.smithi038.stdout:132: set_chunk oid 33 offset: 2023424 length: 164864 target oid smithi03829856-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 2023424 snap -1 2024-03-19T11:30:05.480 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.480 INFO:tasks.rados.rados.0.smithi038.stdout:118: finishing set_chunk to oid 19 2024-03-19T11:30:05.480 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 19 v 3 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-03-19T11:30:05.480 INFO:tasks.rados.rados.0.smithi038.stdout:110: finishing set_chunk to oid 11 2024-03-19T11:30:05.480 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 11 v 3 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-03-19T11:30:05.480 INFO:tasks.rados.rados.0.smithi038.stdout:110: done (15 left) 2024-03-19T11:30:05.481 INFO:tasks.rados.rados.0.smithi038.stdout:118: done (14 left) 2024-03-19T11:30:05.481 INFO:tasks.rados.rados.0.smithi038.stdout:133: set_chunk oid 34 target oid 34 low_tier 2024-03-19T11:30:05.481 INFO:tasks.rados.rados.0.smithi038.stdout:133: set_chunk oid 34 offset: 1714176 length: 172032 target oid smithi03829856-34 low_tier offset: 1714176 snap -1 2024-03-19T11:30:05.481 INFO:tasks.rados.rados.0.smithi038.stdout:134: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.481 INFO:tasks.rados.rados.0.smithi038.stdout:134: set_chunk oid 35 offset: 1638400 length: 928256 target oid smithi03829856-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1638400 snap -1 2024-03-19T11:30:05.481 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.481 INFO:tasks.rados.rados.0.smithi038.stdout:117: finishing set_chunk to oid 18 2024-03-19T11:30:05.481 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 18 v 6 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-03-19T11:30:05.481 INFO:tasks.rados.rados.0.smithi038.stdout:117: done (15 left) 2024-03-19T11:30:05.481 INFO:tasks.rados.rados.0.smithi038.stdout:135: set_chunk oid 36 target oid 36 low_tier 2024-03-19T11:30:05.481 INFO:tasks.rados.rados.0.smithi038.stdout:135: set_chunk oid 36 offset: 2084352 length: 335872 target oid smithi03829856-36 low_tier offset: 2084352 snap -1 2024-03-19T11:30:05.481 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.481 INFO:tasks.rados.rados.0.smithi038.stdout:114: finishing set_chunk to oid 15 2024-03-19T11:30:05.481 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 15 v 6 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-03-19T11:30:05.481 INFO:tasks.rados.rados.0.smithi038.stdout:114: done (15 left) 2024-03-19T11:30:05.482 INFO:tasks.rados.rados.0.smithi038.stdout:136: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.482 INFO:tasks.rados.rados.0.smithi038.stdout:136: set_chunk oid 37 offset: 590336 length: 972800 target oid smithi03829856-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 590336 snap -1 2024-03-19T11:30:05.482 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.482 INFO:tasks.rados.rados.0.smithi038.stdout:123: finishing set_chunk to oid 24 2024-03-19T11:30:05.482 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 24 v 9 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-03-19T11:30:05.482 INFO:tasks.rados.rados.0.smithi038.stdout:123: done (15 left) 2024-03-19T11:30:05.482 INFO:tasks.rados.rados.0.smithi038.stdout:137: set_chunk oid 38 target oid 38 low_tier 2024-03-19T11:30:05.482 INFO:tasks.rados.rados.0.smithi038.stdout:137: set_chunk oid 38 offset: 1394176 length: 438784 target oid smithi03829856-38 low_tier offset: 1394176 snap -1 2024-03-19T11:30:05.482 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.496 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:05.503+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:05.499 INFO:tasks.rados.rados.0.smithi038.stdout:122: finishing set_chunk to oid 23 2024-03-19T11:30:05.499 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 23 v 9 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-03-19T11:30:05.499 INFO:tasks.rados.rados.0.smithi038.stdout:122: done (15 left) 2024-03-19T11:30:05.499 INFO:tasks.rados.rados.0.smithi038.stdout:138: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.499 INFO:tasks.rados.rados.0.smithi038.stdout:138: set_chunk oid 39 offset: 1296896 length: 1088512 target oid smithi03829856-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1296896 snap -1 2024-03-19T11:30:05.499 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.499 INFO:tasks.rados.rados.0.smithi038.stdout:125: finishing set_chunk to oid 26 2024-03-19T11:30:05.499 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 26 v 9 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-03-19T11:30:05.499 INFO:tasks.rados.rados.0.smithi038.stdout:125: done (15 left) 2024-03-19T11:30:05.499 INFO:tasks.rados.rados.0.smithi038.stdout:139: set_chunk oid 40 target oid 40 low_tier 2024-03-19T11:30:05.500 INFO:tasks.rados.rados.0.smithi038.stdout:139: set_chunk oid 40 offset: 755200 length: 198656 target oid smithi03829856-40 low_tier offset: 755200 snap -1 2024-03-19T11:30:05.500 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.500 INFO:tasks.rados.rados.0.smithi038.stdout:127: finishing set_chunk to oid 28 2024-03-19T11:30:05.500 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 28 v 6 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-03-19T11:30:05.500 INFO:tasks.rados.rados.0.smithi038.stdout:127: done (15 left) 2024-03-19T11:30:05.500 INFO:tasks.rados.rados.0.smithi038.stdout:140: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.500 INFO:tasks.rados.rados.0.smithi038.stdout:140: set_chunk oid 41 offset: 622592 length: 1903616 target oid smithi03829856-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 622592 snap -1 2024-03-19T11:30:05.500 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.500 INFO:tasks.rados.rados.0.smithi038.stdout:131: finishing set_chunk to oid 32 2024-03-19T11:30:05.500 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 32 v 6 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-03-19T11:30:05.500 INFO:tasks.rados.rados.0.smithi038.stdout:131: done (15 left) 2024-03-19T11:30:05.500 INFO:tasks.rados.rados.0.smithi038.stdout:141: set_chunk oid 42 target oid 42 low_tier 2024-03-19T11:30:05.500 INFO:tasks.rados.rados.0.smithi038.stdout:141: set_chunk oid 42 offset: 399872 length: 1828864 target oid smithi03829856-42 low_tier offset: 399872 snap -1 2024-03-19T11:30:05.500 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.501 INFO:tasks.rados.rados.0.smithi038.stdout:128: finishing set_chunk to oid 29 2024-03-19T11:30:05.501 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 29 v 3 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-03-19T11:30:05.501 INFO:tasks.rados.rados.0.smithi038.stdout:128: done (15 left) 2024-03-19T11:30:05.501 INFO:tasks.rados.rados.0.smithi038.stdout:142: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.501 INFO:tasks.rados.rados.0.smithi038.stdout:142: set_chunk oid 43 offset: 148992 length: 508928 target oid smithi03829856-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 148992 snap -1 2024-03-19T11:30:05.501 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.501 INFO:tasks.rados.rados.0.smithi038.stdout:126: finishing set_chunk to oid 27 2024-03-19T11:30:05.501 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 27 v 6 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-03-19T11:30:05.501 INFO:tasks.rados.rados.0.smithi038.stdout:126: done (15 left) 2024-03-19T11:30:05.501 INFO:tasks.rados.rados.0.smithi038.stdout:143: set_chunk oid 44 target oid 44 low_tier 2024-03-19T11:30:05.501 INFO:tasks.rados.rados.0.smithi038.stdout:143: set_chunk oid 44 offset: 1948160 length: 57344 target oid smithi03829856-44 low_tier offset: 1948160 snap -1 2024-03-19T11:30:05.501 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.501 INFO:tasks.rados.rados.0.smithi038.stdout:124: finishing set_chunk to oid 25 2024-03-19T11:30:05.501 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 25 v 12 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-03-19T11:30:05.501 INFO:tasks.rados.rados.0.smithi038.stdout:124: done (15 left) 2024-03-19T11:30:05.502 INFO:tasks.rados.rados.0.smithi038.stdout:144: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.502 INFO:tasks.rados.rados.0.smithi038.stdout:144: set_chunk oid 45 offset: 372224 length: 80384 target oid smithi03829856-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 372224 snap -1 2024-03-19T11:30:05.502 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.502 INFO:tasks.rados.rados.0.smithi038.stdout:133: finishing set_chunk to oid 34 2024-03-19T11:30:05.502 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 34 v 3 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-03-19T11:30:05.502 INFO:tasks.rados.rados.0.smithi038.stdout:133: done (15 left) 2024-03-19T11:30:05.502 INFO:tasks.rados.rados.0.smithi038.stdout:145: set_chunk oid 46 target oid 46 low_tier 2024-03-19T11:30:05.502 INFO:tasks.rados.rados.0.smithi038.stdout:145: set_chunk oid 46 offset: 2311680 length: 395776 target oid smithi03829856-46 low_tier offset: 2311680 snap -1 2024-03-19T11:30:05.502 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.502 INFO:tasks.rados.rados.0.smithi038.stdout:121: finishing set_chunk to oid 22 2024-03-19T11:30:05.502 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 22 v 3 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-03-19T11:30:05.502 INFO:tasks.rados.rados.0.smithi038.stdout:121: done (15 left) 2024-03-19T11:30:05.502 INFO:tasks.rados.rados.0.smithi038.stdout:146: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.502 INFO:tasks.rados.rados.0.smithi038.stdout:146: set_chunk oid 47 offset: 555520 length: 697344 target oid smithi03829856-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 555520 snap -1 2024-03-19T11:30:05.503 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.503 INFO:tasks.rados.rados.0.smithi038.stdout:136: finishing set_chunk to oid 37 2024-03-19T11:30:05.503 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 37 v 21 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-03-19T11:30:05.503 INFO:tasks.rados.rados.0.smithi038.stdout:136: done (15 left) 2024-03-19T11:30:05.503 INFO:tasks.rados.rados.0.smithi038.stdout:147: set_chunk oid 48 target oid 48 low_tier 2024-03-19T11:30:05.503 INFO:tasks.rados.rados.0.smithi038.stdout:147: set_chunk oid 48 offset: 399872 length: 798720 target oid smithi03829856-48 low_tier offset: 399872 snap -1 2024-03-19T11:30:05.503 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.503 INFO:tasks.rados.rados.0.smithi038.stdout:130: finishing set_chunk to oid 31 2024-03-19T11:30:05.503 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 31 v 6 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-03-19T11:30:05.503 INFO:tasks.rados.rados.0.smithi038.stdout:130: done (15 left) 2024-03-19T11:30:05.503 INFO:tasks.rados.rados.0.smithi038.stdout:148: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.503 INFO:tasks.rados.rados.0.smithi038.stdout:148: set_chunk oid 49 offset: 1097728 length: 671232 target oid smithi03829856-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1097728 snap -1 2024-03-19T11:30:05.503 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.503 INFO:tasks.rados.rados.0.smithi038.stdout:132: finishing set_chunk to oid 33 2024-03-19T11:30:05.504 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 33 v 9 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-03-19T11:30:05.504 INFO:tasks.rados.rados.0.smithi038.stdout:132: done (15 left) 2024-03-19T11:30:05.504 INFO:tasks.rados.rados.0.smithi038.stdout:149: set_chunk oid 50 target oid 50 low_tier 2024-03-19T11:30:05.504 INFO:tasks.rados.rados.0.smithi038.stdout:149: set_chunk oid 50 offset: 1017344 length: 836096 target oid smithi03829856-50 low_tier offset: 1017344 snap -1 2024-03-19T11:30:05.504 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.504 INFO:tasks.rados.rados.0.smithi038.stdout:134: finishing set_chunk to oid 35 2024-03-19T11:30:05.504 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 35 v 18 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-03-19T11:30:05.504 INFO:tasks.rados.rados.0.smithi038.stdout:129: finishing set_chunk to oid 30 2024-03-19T11:30:05.504 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 30 v 6 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-03-19T11:30:05.504 INFO:tasks.rados.rados.0.smithi038.stdout:140: finishing set_chunk to oid 41 2024-03-19T11:30:05.504 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 41 v 9 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-03-19T11:30:05.504 INFO:tasks.rados.rados.0.smithi038.stdout:129: done (15 left) 2024-03-19T11:30:05.504 INFO:tasks.rados.rados.0.smithi038.stdout:134: done (14 left) 2024-03-19T11:30:05.504 INFO:tasks.rados.rados.0.smithi038.stdout:140: done (13 left) 2024-03-19T11:30:05.504 INFO:tasks.rados.rados.0.smithi038.stdout:150: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.505 INFO:tasks.rados.rados.0.smithi038.stdout:150: set_chunk oid 1 offset: 1390080 length: 171008 target oid smithi03829856-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1390080 snap -1 2024-03-19T11:30:05.505 INFO:tasks.rados.rados.0.smithi038.stdout:151: set_chunk oid 2 target oid 2 low_tier 2024-03-19T11:30:05.505 INFO:tasks.rados.rados.0.smithi038.stdout:151: set_chunk oid 2 offset: 134144 length: 1303040 target oid smithi03829856-2 low_tier offset: 134144 snap -1 2024-03-19T11:30:05.505 INFO:tasks.rados.rados.0.smithi038.stdout:152: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.505 INFO:tasks.rados.rados.0.smithi038.stdout:152: set_chunk oid 3 offset: 1446400 length: 188416 target oid smithi03829856-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1446400 snap -1 2024-03-19T11:30:05.505 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.505 INFO:tasks.rados.rados.0.smithi038.stdout:135: finishing set_chunk to oid 36 2024-03-19T11:30:05.505 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 36 v 9 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-03-19T11:30:05.505 INFO:tasks.rados.rados.0.smithi038.stdout:135: done (15 left) 2024-03-19T11:30:05.505 INFO:tasks.rados.rados.0.smithi038.stdout:153: set_chunk oid 4 target oid 4 low_tier 2024-03-19T11:30:05.505 INFO:tasks.rados.rados.0.smithi038.stdout:153: set_chunk oid 4 offset: 990208 length: 670720 target oid smithi03829856-4 low_tier offset: 990208 snap -1 2024-03-19T11:30:05.505 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.505 INFO:tasks.rados.rados.0.smithi038.stdout:141: finishing set_chunk to oid 42 2024-03-19T11:30:05.505 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 42 v 12 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-03-19T11:30:05.506 INFO:tasks.rados.rados.0.smithi038.stdout:141: done (15 left) 2024-03-19T11:30:05.506 INFO:tasks.rados.rados.0.smithi038.stdout:154: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.506 INFO:tasks.rados.rados.0.smithi038.stdout:154: set_chunk oid 5 offset: 1771520 length: 294400 target oid smithi03829856-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1771520 snap -1 2024-03-19T11:30:05.506 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.506 INFO:tasks.rados.rados.0.smithi038.stdout:151: finishing set_chunk to oid 2 2024-03-19T11:30:05.506 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 2 set_chunk 2 low_tier returned error code -95 offset: 134144 length: 1303040 2024-03-19T11:30:05.506 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-03-19T11:30:05.506 INFO:tasks.rados.rados.0.smithi038.stdout:152: finishing set_chunk to oid 3 2024-03-19T11:30:05.506 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 3 set_chunk 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1446400 length: 188416 2024-03-19T11:30:05.506 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 3 v 0 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-03-19T11:30:05.506 INFO:tasks.rados.rados.0.smithi038.stdout:151: done (15 left) 2024-03-19T11:30:05.506 INFO:tasks.rados.rados.0.smithi038.stdout:152: done (14 left) 2024-03-19T11:30:05.506 INFO:tasks.rados.rados.0.smithi038.stdout:155: set_chunk oid 6 target oid 6 low_tier 2024-03-19T11:30:05.506 INFO:tasks.rados.rados.0.smithi038.stdout:155: set_chunk oid 6 offset: 1612288 length: 374272 target oid smithi03829856-6 low_tier offset: 1612288 snap -1 2024-03-19T11:30:05.507 INFO:tasks.rados.rados.0.smithi038.stdout:156: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.507 INFO:tasks.rados.rados.0.smithi038.stdout:156: set_chunk oid 7 offset: 1106944 length: 610816 target oid smithi03829856-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1106944 snap -1 2024-03-19T11:30:05.507 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.507 INFO:tasks.rados.rados.0.smithi038.stdout:154: finishing set_chunk to oid 5 2024-03-19T11:30:05.507 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 5 set_chunk 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1771520 length: 294400 2024-03-19T11:30:05.507 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-03-19T11:30:05.507 INFO:tasks.rados.rados.0.smithi038.stdout:153: finishing set_chunk to oid 4 2024-03-19T11:30:05.507 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 4 set_chunk 4 low_tier returned error code -95 offset: 990208 length: 670720 2024-03-19T11:30:05.507 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-03-19T11:30:05.507 INFO:tasks.rados.rados.0.smithi038.stdout:153: done (15 left) 2024-03-19T11:30:05.507 INFO:tasks.rados.rados.0.smithi038.stdout:154: done (14 left) 2024-03-19T11:30:05.507 INFO:tasks.rados.rados.0.smithi038.stdout:157: set_chunk oid 8 target oid 8 low_tier 2024-03-19T11:30:05.507 INFO:tasks.rados.rados.0.smithi038.stdout:157: set_chunk oid 8 offset: 122880 length: 76800 target oid smithi03829856-8 low_tier offset: 122880 snap -1 2024-03-19T11:30:05.507 INFO:tasks.rados.rados.0.smithi038.stdout:158: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.507 INFO:tasks.rados.rados.0.smithi038.stdout:158: set_chunk oid 9 offset: 952320 length: 857600 target oid smithi03829856-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 952320 snap -1 2024-03-19T11:30:05.508 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.508 INFO:tasks.rados.rados.0.smithi038.stdout:137: finishing set_chunk to oid 38 2024-03-19T11:30:05.508 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 38 v 12 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-03-19T11:30:05.508 INFO:tasks.rados.rados.0.smithi038.stdout:137: done (15 left) 2024-03-19T11:30:05.508 INFO:tasks.rados.rados.0.smithi038.stdout:159: set_chunk oid 10 target oid 10 low_tier 2024-03-19T11:30:05.508 INFO:tasks.rados.rados.0.smithi038.stdout:159: set_chunk oid 10 offset: 1437184 length: 71680 target oid smithi03829856-10 low_tier offset: 1437184 snap -1 2024-03-19T11:30:05.508 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.508 INFO:tasks.rados.rados.0.smithi038.stdout:156: finishing set_chunk to oid 7 2024-03-19T11:30:05.508 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 7 set_chunk 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1106944 length: 610816 2024-03-19T11:30:05.508 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-03-19T11:30:05.508 INFO:tasks.rados.rados.0.smithi038.stdout:156: done (15 left) 2024-03-19T11:30:05.508 INFO:tasks.rados.rados.0.smithi038.stdout:160: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.508 INFO:tasks.rados.rados.0.smithi038.stdout:160: set_chunk oid 11 offset: 500224 length: 842752 target oid smithi03829856-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 500224 snap -1 2024-03-19T11:30:05.508 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.509 INFO:tasks.rados.rados.0.smithi038.stdout:158: finishing set_chunk to oid 9 2024-03-19T11:30:05.509 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 9 set_chunk 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 952320 length: 857600 2024-03-19T11:30:05.509 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 9 v 0 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-03-19T11:30:05.509 INFO:tasks.rados.rados.0.smithi038.stdout:158: done (15 left) 2024-03-19T11:30:05.509 INFO:tasks.rados.rados.0.smithi038.stdout:161: set_chunk oid 12 target oid 12 low_tier 2024-03-19T11:30:05.509 INFO:tasks.rados.rados.0.smithi038.stdout:161: set_chunk oid 12 offset: 1477120 length: 281088 target oid smithi03829856-12 low_tier offset: 1477120 snap -1 2024-03-19T11:30:05.509 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.509 INFO:tasks.rados.rados.0.smithi038.stdout:160: finishing set_chunk to oid 11 2024-03-19T11:30:05.509 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 11 set_chunk 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 500224 length: 842752 2024-03-19T11:30:05.509 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 11 v 0 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-03-19T11:30:05.509 INFO:tasks.rados.rados.0.smithi038.stdout:160: done (15 left) 2024-03-19T11:30:05.509 INFO:tasks.rados.rados.0.smithi038.stdout:162: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.509 INFO:tasks.rados.rados.0.smithi038.stdout:162: set_chunk oid 13 offset: 15360 length: 166912 target oid smithi03829856-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 15360 snap -1 2024-03-19T11:30:05.509 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.510 INFO:tasks.rados.rados.0.smithi038.stdout:145: finishing set_chunk to oid 46 2024-03-19T11:30:05.510 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 46 v 12 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-03-19T11:30:05.510 INFO:tasks.rados.rados.0.smithi038.stdout:145: done (15 left) 2024-03-19T11:30:05.510 INFO:tasks.rados.rados.0.smithi038.stdout:163: set_chunk oid 14 target oid 14 low_tier 2024-03-19T11:30:05.510 INFO:tasks.rados.rados.0.smithi038.stdout:163: set_chunk oid 14 offset: 179712 length: 1016320 target oid smithi03829856-14 low_tier offset: 179712 snap -1 2024-03-19T11:30:05.510 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.510 INFO:tasks.rados.rados.0.smithi038.stdout:162: finishing set_chunk to oid 13 2024-03-19T11:30:05.510 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 13 set_chunk 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 15360 length: 166912 2024-03-19T11:30:05.510 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 13 v 0 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-03-19T11:30:05.510 INFO:tasks.rados.rados.0.smithi038.stdout:162: done (15 left) 2024-03-19T11:30:05.510 INFO:tasks.rados.rados.0.smithi038.stdout:164: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.510 INFO:tasks.rados.rados.0.smithi038.stdout:164: set_chunk oid 15 offset: 703488 length: 577536 target oid smithi03829856-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 703488 snap -1 2024-03-19T11:30:05.510 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.510 INFO:tasks.rados.rados.0.smithi038.stdout:163: finishing set_chunk to oid 14 2024-03-19T11:30:05.510 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 14 set_chunk 14 low_tier returned error code -95 offset: 179712 length: 1016320 2024-03-19T11:30:05.511 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 14 v 0 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-03-19T11:30:05.511 INFO:tasks.rados.rados.0.smithi038.stdout:146: finishing set_chunk to oid 47 2024-03-19T11:30:05.511 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 47 v 15 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-03-19T11:30:05.511 INFO:tasks.rados.rados.0.smithi038.stdout:146: done (15 left) 2024-03-19T11:30:05.511 INFO:tasks.rados.rados.0.smithi038.stdout:163: done (14 left) 2024-03-19T11:30:05.511 INFO:tasks.rados.rados.0.smithi038.stdout:165: set_chunk oid 16 target oid 16 low_tier 2024-03-19T11:30:05.511 INFO:tasks.rados.rados.0.smithi038.stdout:165: set_chunk oid 16 offset: 225280 length: 1419264 target oid smithi03829856-16 low_tier offset: 225280 snap -1 2024-03-19T11:30:05.511 INFO:tasks.rados.rados.0.smithi038.stdout:166: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.511 INFO:tasks.rados.rados.0.smithi038.stdout:166: set_chunk oid 17 offset: 627712 length: 1497600 target oid smithi03829856-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 627712 snap -1 2024-03-19T11:30:05.511 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.511 INFO:tasks.rados.rados.0.smithi038.stdout:164: finishing set_chunk to oid 15 2024-03-19T11:30:05.511 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 703488 length: 577536 2024-03-19T11:30:05.511 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-03-19T11:30:05.511 INFO:tasks.rados.rados.0.smithi038.stdout:165: finishing set_chunk to oid 16 2024-03-19T11:30:05.512 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 16 set_chunk 16 low_tier returned error code -95 offset: 225280 length: 1419264 2024-03-19T11:30:05.512 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-03-19T11:30:05.512 INFO:tasks.rados.rados.0.smithi038.stdout:166: finishing set_chunk to oid 17 2024-03-19T11:30:05.512 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 17 set_chunk 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 627712 length: 1497600 2024-03-19T11:30:05.512 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 17 v 0 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-03-19T11:30:05.512 INFO:tasks.rados.rados.0.smithi038.stdout:164: done (15 left) 2024-03-19T11:30:05.512 INFO:tasks.rados.rados.0.smithi038.stdout:165: done (14 left) 2024-03-19T11:30:05.512 INFO:tasks.rados.rados.0.smithi038.stdout:166: done (13 left) 2024-03-19T11:30:05.512 INFO:tasks.rados.rados.0.smithi038.stdout:167: set_chunk oid 18 target oid 18 low_tier 2024-03-19T11:30:05.512 INFO:tasks.rados.rados.0.smithi038.stdout:167: set_chunk oid 18 offset: 94720 length: 727040 target oid smithi03829856-18 low_tier offset: 94720 snap -1 2024-03-19T11:30:05.512 INFO:tasks.rados.rados.0.smithi038.stdout:168: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.512 INFO:tasks.rados.rados.0.smithi038.stdout:168: set_chunk oid 19 offset: 1049088 length: 811008 target oid smithi03829856-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1049088 snap -1 2024-03-19T11:30:05.512 INFO:tasks.rados.rados.0.smithi038.stdout:169: set_chunk oid 20 target oid 20 low_tier 2024-03-19T11:30:05.512 INFO:tasks.rados.rados.0.smithi038.stdout:169: set_chunk oid 20 offset: 1520128 length: 79872 target oid smithi03829856-20 low_tier offset: 1520128 snap -1 2024-03-19T11:30:05.512 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.513 INFO:tasks.rados.rados.0.smithi038.stdout:167: finishing set_chunk to oid 18 2024-03-19T11:30:05.513 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 18 set_chunk 18 low_tier returned error code -95 offset: 94720 length: 727040 2024-03-19T11:30:05.513 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 18 v 0 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-03-19T11:30:05.513 INFO:tasks.rados.rados.0.smithi038.stdout:168: finishing set_chunk to oid 19 2024-03-19T11:30:05.513 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 19 set_chunk 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1049088 length: 811008 2024-03-19T11:30:05.513 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 19 v 0 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-03-19T11:30:05.513 INFO:tasks.rados.rados.0.smithi038.stdout:167: done (15 left) 2024-03-19T11:30:05.513 INFO:tasks.rados.rados.0.smithi038.stdout:168: done (14 left) 2024-03-19T11:30:05.513 INFO:tasks.rados.rados.0.smithi038.stdout:170: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.513 INFO:tasks.rados.rados.0.smithi038.stdout:170: set_chunk oid 21 offset: 847872 length: 87040 target oid smithi03829856-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 847872 snap -1 2024-03-19T11:30:05.513 INFO:tasks.rados.rados.0.smithi038.stdout:171: set_chunk oid 22 target oid 22 low_tier 2024-03-19T11:30:05.513 INFO:tasks.rados.rados.0.smithi038.stdout:171: set_chunk oid 22 offset: 22016 length: 1375744 target oid smithi03829856-22 low_tier offset: 22016 snap -1 2024-03-19T11:30:05.513 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.513 INFO:tasks.rados.rados.0.smithi038.stdout:171: finishing set_chunk to oid 22 2024-03-19T11:30:05.513 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 22 set_chunk 22 low_tier returned error code -95 offset: 22016 length: 1375744 2024-03-19T11:30:05.514 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-03-19T11:30:05.514 INFO:tasks.rados.rados.0.smithi038.stdout:171: done (15 left) 2024-03-19T11:30:05.514 INFO:tasks.rados.rados.0.smithi038.stdout:172: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.514 INFO:tasks.rados.rados.0.smithi038.stdout:172: set_chunk oid 23 offset: 1230848 length: 877568 target oid smithi03829856-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1230848 snap -1 2024-03-19T11:30:05.514 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.514 INFO:tasks.rados.rados.0.smithi038.stdout:172: finishing set_chunk to oid 23 2024-03-19T11:30:05.514 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 23 set_chunk 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1230848 length: 877568 2024-03-19T11:30:05.514 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 23 v 0 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-03-19T11:30:05.514 INFO:tasks.rados.rados.0.smithi038.stdout:172: done (15 left) 2024-03-19T11:30:05.514 INFO:tasks.rados.rados.0.smithi038.stdout:173: set_chunk oid 24 target oid 24 low_tier 2024-03-19T11:30:05.514 INFO:tasks.rados.rados.0.smithi038.stdout:173: set_chunk oid 24 offset: 383488 length: 97280 target oid smithi03829856-24 low_tier offset: 383488 snap -1 2024-03-19T11:30:05.514 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.514 INFO:tasks.rados.rados.0.smithi038.stdout:173: finishing set_chunk to oid 24 2024-03-19T11:30:05.514 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 24 set_chunk 24 low_tier returned error code -95 offset: 383488 length: 97280 2024-03-19T11:30:05.515 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-03-19T11:30:05.515 INFO:tasks.rados.rados.0.smithi038.stdout:173: done (15 left) 2024-03-19T11:30:05.515 INFO:tasks.rados.rados.0.smithi038.stdout:174: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.515 INFO:tasks.rados.rados.0.smithi038.stdout:174: set_chunk oid 25 offset: 1356800 length: 955392 target oid smithi03829856-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1356800 snap -1 2024-03-19T11:30:05.515 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.515 INFO:tasks.rados.rados.0.smithi038.stdout:174: finishing set_chunk to oid 25 2024-03-19T11:30:05.515 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 25 set_chunk 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1356800 length: 955392 2024-03-19T11:30:05.515 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-03-19T11:30:05.515 INFO:tasks.rados.rados.0.smithi038.stdout:174: done (15 left) 2024-03-19T11:30:05.515 INFO:tasks.rados.rados.0.smithi038.stdout:175: set_chunk oid 26 target oid 26 low_tier 2024-03-19T11:30:05.515 INFO:tasks.rados.rados.0.smithi038.stdout:175: set_chunk oid 26 offset: 711680 length: 179712 target oid smithi03829856-26 low_tier offset: 711680 snap -1 2024-03-19T11:30:05.515 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.515 INFO:tasks.rados.rados.0.smithi038.stdout:150: finishing set_chunk to oid 1 2024-03-19T11:30:05.515 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-03-19T11:30:05.515 INFO:tasks.rados.rados.0.smithi038.stdout:150: done (15 left) 2024-03-19T11:30:05.516 INFO:tasks.rados.rados.0.smithi038.stdout:176: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.516 INFO:tasks.rados.rados.0.smithi038.stdout:176: set_chunk oid 27 offset: 344064 length: 1554432 target oid smithi03829856-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 344064 snap -1 2024-03-19T11:30:05.516 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.516 INFO:tasks.rados.rados.0.smithi038.stdout:147: finishing set_chunk to oid 48 2024-03-19T11:30:05.516 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 48 v 12 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-03-19T11:30:05.516 INFO:tasks.rados.rados.0.smithi038.stdout:147: done (15 left) 2024-03-19T11:30:05.516 INFO:tasks.rados.rados.0.smithi038.stdout:177: set_chunk oid 28 target oid 28 low_tier 2024-03-19T11:30:05.516 INFO:tasks.rados.rados.0.smithi038.stdout:177: set_chunk oid 28 offset: 546816 length: 1159680 target oid smithi03829856-28 low_tier offset: 546816 snap -1 2024-03-19T11:30:05.516 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.516 INFO:tasks.rados.rados.0.smithi038.stdout:143: finishing set_chunk to oid 44 2024-03-19T11:30:05.516 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 44 v 9 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-03-19T11:30:05.516 INFO:tasks.rados.rados.0.smithi038.stdout:161: finishing set_chunk to oid 12 2024-03-19T11:30:05.516 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 12 v 3 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-03-19T11:30:05.516 INFO:tasks.rados.rados.0.smithi038.stdout:143: done (15 left) 2024-03-19T11:30:05.516 INFO:tasks.rados.rados.0.smithi038.stdout:161: done (14 left) 2024-03-19T11:30:05.517 INFO:tasks.rados.rados.0.smithi038.stdout:178: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.517 INFO:tasks.rados.rados.0.smithi038.stdout:178: set_chunk oid 29 offset: 1436672 length: 820736 target oid smithi03829856-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1436672 snap -1 2024-03-19T11:30:05.517 INFO:tasks.rados.rados.0.smithi038.stdout:179: set_chunk oid 30 target oid 30 low_tier 2024-03-19T11:30:05.517 INFO:tasks.rados.rados.0.smithi038.stdout:179: set_chunk oid 30 offset: 1064448 length: 1181696 target oid smithi03829856-30 low_tier offset: 1064448 snap -1 2024-03-19T11:30:05.517 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.517 INFO:tasks.rados.rados.0.smithi038.stdout:138: finishing set_chunk to oid 39 2024-03-19T11:30:05.517 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 39 v 6 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-03-19T11:30:05.517 INFO:tasks.rados.rados.0.smithi038.stdout:175: finishing set_chunk to oid 26 2024-03-19T11:30:05.517 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 26 set_chunk 26 low_tier returned error code -95 offset: 711680 length: 179712 2024-03-19T11:30:05.517 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-03-19T11:30:05.517 INFO:tasks.rados.rados.0.smithi038.stdout:138: done (15 left) 2024-03-19T11:30:05.517 INFO:tasks.rados.rados.0.smithi038.stdout:175: done (14 left) 2024-03-19T11:30:05.517 INFO:tasks.rados.rados.0.smithi038.stdout:180: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.517 INFO:tasks.rados.rados.0.smithi038.stdout:180: set_chunk oid 31 offset: 350208 length: 1075712 target oid smithi03829856-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 350208 snap -1 2024-03-19T11:30:05.517 INFO:tasks.rados.rados.0.smithi038.stdout:181: set_chunk oid 32 target oid 32 low_tier 2024-03-19T11:30:05.518 INFO:tasks.rados.rados.0.smithi038.stdout:181: set_chunk oid 32 offset: 1096192 length: 1304064 target oid smithi03829856-32 low_tier offset: 1096192 snap -1 2024-03-19T11:30:05.518 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.518 INFO:tasks.rados.rados.0.smithi038.stdout:176: finishing set_chunk to oid 27 2024-03-19T11:30:05.518 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 27 set_chunk 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 344064 length: 1554432 2024-03-19T11:30:05.518 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-03-19T11:30:05.518 INFO:tasks.rados.rados.0.smithi038.stdout:176: done (15 left) 2024-03-19T11:30:05.518 INFO:tasks.rados.rados.0.smithi038.stdout:182: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.518 INFO:tasks.rados.rados.0.smithi038.stdout:182: set_chunk oid 33 offset: 1183232 length: 531456 target oid smithi03829856-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1183232 snap -1 2024-03-19T11:30:05.518 INFO:tasks.rados.rados.0.smithi038.stdout:177: finishing set_chunk to oid 28 2024-03-19T11:30:05.518 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 28 set_chunk 28 low_tier returned error code -95 offset: 546816 length: 1159680 2024-03-19T11:30:05.518 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 28 v 0 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-03-19T11:30:05.518 INFO:tasks.rados.rados.0.smithi038.stdout:157: finishing set_chunk to oid 8 2024-03-19T11:30:05.518 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 8 v 3 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-03-19T11:30:05.519 INFO:tasks.rados.rados.0.smithi038.stdout:157: done (15 left) 2024-03-19T11:30:05.519 INFO:tasks.rados.rados.0.smithi038.stdout:177: done (14 left) 2024-03-19T11:30:05.519 INFO:tasks.rados.rados.0.smithi038.stdout:183: set_chunk oid 34 target oid 34 low_tier 2024-03-19T11:30:05.519 INFO:tasks.rados.rados.0.smithi038.stdout:183: set_chunk oid 34 offset: 843776 length: 1169408 target oid smithi03829856-34 low_tier offset: 843776 snap -1 2024-03-19T11:30:05.519 INFO:tasks.rados.rados.0.smithi038.stdout:184: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.519 INFO:tasks.rados.rados.0.smithi038.stdout:184: set_chunk oid 35 offset: 1203712 length: 497664 target oid smithi03829856-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1203712 snap -1 2024-03-19T11:30:05.519 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.519 INFO:tasks.rados.rados.0.smithi038.stdout:139: finishing set_chunk to oid 40 2024-03-19T11:30:05.519 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 40 v 9 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-03-19T11:30:05.519 INFO:tasks.rados.rados.0.smithi038.stdout:142: finishing set_chunk to oid 43 2024-03-19T11:30:05.519 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 43 v 9 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-03-19T11:30:05.519 INFO:tasks.rados.rados.0.smithi038.stdout:144: finishing set_chunk to oid 45 2024-03-19T11:30:05.519 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 45 v 12 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-03-19T11:30:05.519 INFO:tasks.rados.rados.0.smithi038.stdout:139: done (15 left) 2024-03-19T11:30:05.519 INFO:tasks.rados.rados.0.smithi038.stdout:142: done (14 left) 2024-03-19T11:30:05.520 INFO:tasks.rados.rados.0.smithi038.stdout:144: done (13 left) 2024-03-19T11:30:05.520 INFO:tasks.rados.rados.0.smithi038.stdout:185: set_chunk oid 36 target oid 36 low_tier 2024-03-19T11:30:05.520 INFO:tasks.rados.rados.0.smithi038.stdout:185: set_chunk oid 36 offset: 639488 length: 237056 target oid smithi03829856-36 low_tier offset: 639488 snap -1 2024-03-19T11:30:05.520 INFO:tasks.rados.rados.0.smithi038.stdout:186: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.520 INFO:tasks.rados.rados.0.smithi038.stdout:186: set_chunk oid 37 offset: 1783808 length: 756736 target oid smithi03829856-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1783808 snap -1 2024-03-19T11:30:05.520 INFO:tasks.rados.rados.0.smithi038.stdout:187: set_chunk oid 38 target oid 38 low_tier 2024-03-19T11:30:05.520 INFO:tasks.rados.rados.0.smithi038.stdout:187: set_chunk oid 38 offset: 1240576 length: 75776 target oid smithi03829856-38 low_tier offset: 1240576 snap -1 2024-03-19T11:30:05.520 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.520 INFO:tasks.rados.rados.0.smithi038.stdout:184: finishing set_chunk to oid 35 2024-03-19T11:30:05.520 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 35 set_chunk 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1203712 length: 497664 2024-03-19T11:30:05.520 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 35 v 0 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-03-19T11:30:05.520 INFO:tasks.rados.rados.0.smithi038.stdout:183: finishing set_chunk to oid 34 2024-03-19T11:30:05.520 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 34 set_chunk 34 low_tier returned error code -95 offset: 843776 length: 1169408 2024-03-19T11:30:05.520 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 34 v 0 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-03-19T11:30:05.520 INFO:tasks.rados.rados.0.smithi038.stdout:183: done (15 left) 2024-03-19T11:30:05.521 INFO:tasks.rados.rados.0.smithi038.stdout:184: done (14 left) 2024-03-19T11:30:05.521 INFO:tasks.rados.rados.0.smithi038.stdout:188: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.521 INFO:tasks.rados.rados.0.smithi038.stdout:188: set_chunk oid 39 offset: 1272832 length: 780288 target oid smithi03829856-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1272832 snap -1 2024-03-19T11:30:05.521 INFO:tasks.rados.rados.0.smithi038.stdout:189: set_chunk oid 40 target oid 40 low_tier 2024-03-19T11:30:05.521 INFO:tasks.rados.rados.0.smithi038.stdout:189: set_chunk oid 40 offset: 693248 length: 455680 target oid smithi03829856-40 low_tier offset: 693248 snap -1 2024-03-19T11:30:05.521 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.521 INFO:tasks.rados.rados.0.smithi038.stdout:181: finishing set_chunk to oid 32 2024-03-19T11:30:05.521 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 32 set_chunk 32 low_tier returned error code -95 offset: 1096192 length: 1304064 2024-03-19T11:30:05.521 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-03-19T11:30:05.521 INFO:tasks.rados.rados.0.smithi038.stdout:181: done (15 left) 2024-03-19T11:30:05.521 INFO:tasks.rados.rados.0.smithi038.stdout:190: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.521 INFO:tasks.rados.rados.0.smithi038.stdout:190: set_chunk oid 41 offset: 432128 length: 456192 target oid smithi03829856-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 432128 snap -1 2024-03-19T11:30:05.521 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.521 INFO:tasks.rados.rados.0.smithi038.stdout:180: finishing set_chunk to oid 31 2024-03-19T11:30:05.522 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 31 set_chunk 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 350208 length: 1075712 2024-03-19T11:30:05.522 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 31 v 0 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-03-19T11:30:05.522 INFO:tasks.rados.rados.0.smithi038.stdout:179: finishing set_chunk to oid 30 2024-03-19T11:30:05.522 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 30 set_chunk 30 low_tier returned error code -95 offset: 1064448 length: 1181696 2024-03-19T11:30:05.522 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 30 v 0 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-03-19T11:30:05.522 INFO:tasks.rados.rados.0.smithi038.stdout:179: done (15 left) 2024-03-19T11:30:05.522 INFO:tasks.rados.rados.0.smithi038.stdout:180: done (14 left) 2024-03-19T11:30:05.522 INFO:tasks.rados.rados.0.smithi038.stdout:191: set_chunk oid 42 target oid 42 low_tier 2024-03-19T11:30:05.522 INFO:tasks.rados.rados.0.smithi038.stdout:191: set_chunk oid 42 offset: 615936 length: 352256 target oid smithi03829856-42 low_tier offset: 615936 snap -1 2024-03-19T11:30:05.522 INFO:tasks.rados.rados.0.smithi038.stdout:192: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.522 INFO:tasks.rados.rados.0.smithi038.stdout:192: set_chunk oid 43 offset: 846848 length: 1231360 target oid smithi03829856-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 846848 snap -1 2024-03-19T11:30:05.522 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.522 INFO:tasks.rados.rados.0.smithi038.stdout:148: finishing set_chunk to oid 49 2024-03-19T11:30:05.522 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 49 v 9 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-03-19T11:30:05.523 INFO:tasks.rados.rados.0.smithi038.stdout:148: done (15 left) 2024-03-19T11:30:05.523 INFO:tasks.rados.rados.0.smithi038.stdout:193: set_chunk oid 44 target oid 44 low_tier 2024-03-19T11:30:05.523 INFO:tasks.rados.rados.0.smithi038.stdout:193: set_chunk oid 44 offset: 1280512 length: 450048 target oid smithi03829856-44 low_tier offset: 1280512 snap -1 2024-03-19T11:30:05.523 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.523 INFO:tasks.rados.rados.0.smithi038.stdout:188: finishing set_chunk to oid 39 2024-03-19T11:30:05.523 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 39 set_chunk 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1272832 length: 780288 2024-03-19T11:30:05.523 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 39 v 0 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-03-19T11:30:05.523 INFO:tasks.rados.rados.0.smithi038.stdout:190: finishing set_chunk to oid 41 2024-03-19T11:30:05.523 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 41 set_chunk 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 432128 length: 456192 2024-03-19T11:30:05.523 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 41 v 0 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-03-19T11:30:05.523 INFO:tasks.rados.rados.0.smithi038.stdout:188: done (15 left) 2024-03-19T11:30:05.523 INFO:tasks.rados.rados.0.smithi038.stdout:190: done (14 left) 2024-03-19T11:30:05.523 INFO:tasks.rados.rados.0.smithi038.stdout:194: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.523 INFO:tasks.rados.rados.0.smithi038.stdout:194: set_chunk oid 45 offset: 1366528 length: 206848 target oid smithi03829856-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1366528 snap -1 2024-03-19T11:30:05.524 INFO:tasks.rados.rados.0.smithi038.stdout:195: set_chunk oid 46 target oid 46 low_tier 2024-03-19T11:30:05.524 INFO:tasks.rados.rados.0.smithi038.stdout:195: set_chunk oid 46 offset: 1302528 length: 765952 target oid smithi03829856-46 low_tier offset: 1302528 snap -1 2024-03-19T11:30:05.524 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.524 INFO:tasks.rados.rados.0.smithi038.stdout:189: finishing set_chunk to oid 40 2024-03-19T11:30:05.524 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 40 set_chunk 40 low_tier returned error code -95 offset: 693248 length: 455680 2024-03-19T11:30:05.524 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 40 v 0 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-03-19T11:30:05.524 INFO:tasks.rados.rados.0.smithi038.stdout:189: done (15 left) 2024-03-19T11:30:05.524 INFO:tasks.rados.rados.0.smithi038.stdout:196: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.524 INFO:tasks.rados.rados.0.smithi038.stdout:196: set_chunk oid 47 offset: 1432064 length: 918528 target oid smithi03829856-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1432064 snap -1 2024-03-19T11:30:05.524 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.524 INFO:tasks.rados.rados.0.smithi038.stdout:191: finishing set_chunk to oid 42 2024-03-19T11:30:05.524 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 42 set_chunk 42 low_tier returned error code -95 offset: 615936 length: 352256 2024-03-19T11:30:05.524 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-03-19T11:30:05.524 INFO:tasks.rados.rados.0.smithi038.stdout:191: done (15 left) 2024-03-19T11:30:05.524 INFO:tasks.rados.rados.0.smithi038.stdout:197: set_chunk oid 48 target oid 48 low_tier 2024-03-19T11:30:05.525 INFO:tasks.rados.rados.0.smithi038.stdout:197: set_chunk oid 48 offset: 1644544 length: 1076736 target oid smithi03829856-48 low_tier offset: 1644544 snap -1 2024-03-19T11:30:05.525 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.525 INFO:tasks.rados.rados.0.smithi038.stdout:155: finishing set_chunk to oid 6 2024-03-19T11:30:05.525 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 6 v 3 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-03-19T11:30:05.525 INFO:tasks.rados.rados.0.smithi038.stdout:155: done (15 left) 2024-03-19T11:30:05.525 INFO:tasks.rados.rados.0.smithi038.stdout:198: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.525 INFO:tasks.rados.rados.0.smithi038.stdout:198: set_chunk oid 49 offset: 122880 length: 1648128 target oid smithi03829856-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 122880 snap -1 2024-03-19T11:30:05.525 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.525 INFO:tasks.rados.rados.0.smithi038.stdout:169: finishing set_chunk to oid 20 2024-03-19T11:30:05.525 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 20 v 12 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-03-19T11:30:05.525 INFO:tasks.rados.rados.0.smithi038.stdout:169: done (15 left) 2024-03-19T11:30:05.525 INFO:tasks.rados.rados.0.smithi038.stdout:198 retry set_chunk ! 2024-03-19T11:30:05.525 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 15 2024-03-19T11:30:05.525 INFO:tasks.rados.rados.0.smithi038.stdout:159: finishing set_chunk to oid 10 2024-03-19T11:30:05.525 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 10 v 6 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-03-19T11:30:05.526 INFO:tasks.rados.rados.0.smithi038.stdout:159: done (14 left) 2024-03-19T11:30:05.526 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 14 2024-03-19T11:30:05.526 INFO:tasks.rados.rados.0.smithi038.stdout:198: finishing set_chunk to oid 49 2024-03-19T11:30:05.526 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 49 set_chunk 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 122880 length: 1648128 2024-03-19T11:30:05.526 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-03-19T11:30:05.526 INFO:tasks.rados.rados.0.smithi038.stdout:198: done (13 left) 2024-03-19T11:30:05.526 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 13 2024-03-19T11:30:05.526 INFO:tasks.rados.rados.0.smithi038.stdout:149: finishing set_chunk to oid 50 2024-03-19T11:30:05.526 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 50 v 15 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-03-19T11:30:05.526 INFO:tasks.rados.rados.0.smithi038.stdout:149: done (12 left) 2024-03-19T11:30:05.526 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 12 2024-03-19T11:30:05.526 INFO:tasks.rados.rados.0.smithi038.stdout:195: finishing set_chunk to oid 46 2024-03-19T11:30:05.526 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 46 v 12 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-03-19T11:30:05.526 INFO:tasks.rados.rados.0.smithi038.stdout:195: done (11 left) 2024-03-19T11:30:05.526 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 11 2024-03-19T11:30:05.527 INFO:tasks.rados.rados.0.smithi038.stdout:170: finishing set_chunk to oid 21 2024-03-19T11:30:05.527 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 21 v 15 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-03-19T11:30:05.527 INFO:tasks.rados.rados.0.smithi038.stdout:170: done (10 left) 2024-03-19T11:30:05.527 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 10 2024-03-19T11:30:05.527 INFO:tasks.rados.rados.0.smithi038.stdout:187: finishing set_chunk to oid 38 2024-03-19T11:30:05.527 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 38 v 12 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-03-19T11:30:05.527 INFO:tasks.rados.rados.0.smithi038.stdout:187: done (9 left) 2024-03-19T11:30:05.527 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 9 2024-03-19T11:30:05.527 INFO:tasks.rados.rados.0.smithi038.stdout:185: finishing set_chunk to oid 36 2024-03-19T11:30:05.527 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 36 v 9 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-03-19T11:30:05.527 INFO:tasks.rados.rados.0.smithi038.stdout:185: done (8 left) 2024-03-19T11:30:05.527 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 8 2024-03-19T11:30:05.527 INFO:tasks.rados.rados.0.smithi038.stdout:197: finishing set_chunk to oid 48 2024-03-19T11:30:05.527 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 48 v 12 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-03-19T11:30:05.528 INFO:tasks.rados.rados.0.smithi038.stdout:197: done (7 left) 2024-03-19T11:30:05.528 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 7 2024-03-19T11:30:05.528 INFO:tasks.rados.rados.0.smithi038.stdout:182: finishing set_chunk to oid 33 2024-03-19T11:30:05.528 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 33 v 9 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-03-19T11:30:05.528 INFO:tasks.rados.rados.0.smithi038.stdout:182: done (6 left) 2024-03-19T11:30:05.528 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 6 2024-03-19T11:30:05.528 INFO:tasks.rados.rados.0.smithi038.stdout:192: finishing set_chunk to oid 43 2024-03-19T11:30:05.528 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 43 v 9 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-03-19T11:30:05.528 INFO:tasks.rados.rados.0.smithi038.stdout:192: done (5 left) 2024-03-19T11:30:05.528 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 5 2024-03-19T11:30:05.528 INFO:tasks.rados.rados.0.smithi038.stdout:178: finishing set_chunk to oid 29 2024-03-19T11:30:05.528 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 29 v 3 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-03-19T11:30:05.528 INFO:tasks.rados.rados.0.smithi038.stdout:178: done (4 left) 2024-03-19T11:30:05.528 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 4 2024-03-19T11:30:05.528 INFO:tasks.rados.rados.0.smithi038.stdout:186: finishing set_chunk to oid 37 2024-03-19T11:30:05.528 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 37 v 21 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-03-19T11:30:05.529 INFO:tasks.rados.rados.0.smithi038.stdout:186: done (3 left) 2024-03-19T11:30:05.529 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 3 2024-03-19T11:30:05.575 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:05.579+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:05.602 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:05.602 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:05.602 INFO:tasks.rados.rados.0.smithi038.stdout:196: finishing set_chunk to oid 47 2024-03-19T11:30:05.603 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 47 v 15 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-03-19T11:30:05.603 INFO:tasks.rados.rados.0.smithi038.stdout:196: done (2 left) 2024-03-19T11:30:05.603 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 2 2024-03-19T11:30:05.603 INFO:tasks.rados.rados.0.smithi038.stdout:193: finishing set_chunk to oid 44 2024-03-19T11:30:05.603 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 44 v 9 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-03-19T11:30:05.603 INFO:tasks.rados.rados.0.smithi038.stdout:193: done (1 left) 2024-03-19T11:30:05.604 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 1 2024-03-19T11:30:05.604 INFO:tasks.rados.rados.0.smithi038.stdout:194: finishing set_chunk to oid 45 2024-03-19T11:30:05.604 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 45 v 12 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-03-19T11:30:05.604 INFO:tasks.rados.rados.0.smithi038.stdout:194: done (0 left) 2024-03-19T11:30:05.604 INFO:tasks.rados.rados.0.smithi038.stdout:199: set_chunk oid 50 target oid 50 low_tier 2024-03-19T11:30:05.604 INFO:tasks.rados.rados.0.smithi038.stdout:199: set_chunk oid 50 offset: 960000 length: 1705472 target oid smithi03829856-50 low_tier offset: 960000 snap -1 2024-03-19T11:30:05.604 INFO:tasks.rados.rados.0.smithi038.stdout:200: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.605 INFO:tasks.rados.rados.0.smithi038.stdout:200: set_chunk oid 1 offset: 437760 length: 1145856 target oid smithi03829856-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 437760 snap -1 2024-03-19T11:30:05.605 INFO:tasks.rados.rados.0.smithi038.stdout:201: set_chunk oid 2 target oid 2 low_tier 2024-03-19T11:30:05.605 INFO:tasks.rados.rados.0.smithi038.stdout:201: set_chunk oid 2 offset: 162816 length: 97280 target oid smithi03829856-2 low_tier offset: 162816 snap -1 2024-03-19T11:30:05.605 INFO:tasks.rados.rados.0.smithi038.stdout:202: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.605 INFO:tasks.rados.rados.0.smithi038.stdout:202: set_chunk oid 3 offset: 321536 length: 1463808 target oid smithi03829856-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 321536 snap -1 2024-03-19T11:30:05.605 INFO:tasks.rados.rados.0.smithi038.stdout:203: set_chunk oid 4 target oid 4 low_tier 2024-03-19T11:30:05.605 INFO:tasks.rados.rados.0.smithi038.stdout:203: set_chunk oid 4 offset: 1232896 length: 353792 target oid smithi03829856-4 low_tier offset: 1232896 snap -1 2024-03-19T11:30:05.605 INFO:tasks.rados.rados.0.smithi038.stdout:204: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.605 INFO:tasks.rados.rados.0.smithi038.stdout:204: set_chunk oid 5 offset: 1699328 length: 305152 target oid smithi03829856-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1699328 snap -1 2024-03-19T11:30:05.605 INFO:tasks.rados.rados.0.smithi038.stdout:205: set_chunk oid 6 target oid 6 low_tier 2024-03-19T11:30:05.605 INFO:tasks.rados.rados.0.smithi038.stdout:205: set_chunk oid 6 offset: 511488 length: 581632 target oid smithi03829856-6 low_tier offset: 511488 snap -1 2024-03-19T11:30:05.605 INFO:tasks.rados.rados.0.smithi038.stdout:206: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.605 INFO:tasks.rados.rados.0.smithi038.stdout:206: set_chunk oid 7 offset: 72704 length: 1282048 target oid smithi03829856-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 72704 snap -1 2024-03-19T11:30:05.605 INFO:tasks.rados.rados.0.smithi038.stdout:207: set_chunk oid 8 target oid 8 low_tier 2024-03-19T11:30:05.606 INFO:tasks.rados.rados.0.smithi038.stdout:207: set_chunk oid 8 offset: 1497088 length: 138240 target oid smithi03829856-8 low_tier offset: 1497088 snap -1 2024-03-19T11:30:05.606 INFO:tasks.rados.rados.0.smithi038.stdout:208: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.606 INFO:tasks.rados.rados.0.smithi038.stdout:208: set_chunk oid 9 offset: 1745408 length: 87552 target oid smithi03829856-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1745408 snap -1 2024-03-19T11:30:05.606 INFO:tasks.rados.rados.0.smithi038.stdout:209: set_chunk oid 10 target oid 10 low_tier 2024-03-19T11:30:05.606 INFO:tasks.rados.rados.0.smithi038.stdout:209: set_chunk oid 10 offset: 568320 length: 1214976 target oid smithi03829856-10 low_tier offset: 568320 snap -1 2024-03-19T11:30:05.606 INFO:tasks.rados.rados.0.smithi038.stdout:210: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.606 INFO:tasks.rados.rados.0.smithi038.stdout:210: set_chunk oid 11 offset: 558592 length: 1562112 target oid smithi03829856-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 558592 snap -1 2024-03-19T11:30:05.606 INFO:tasks.rados.rados.0.smithi038.stdout:211: set_chunk oid 12 target oid 12 low_tier 2024-03-19T11:30:05.606 INFO:tasks.rados.rados.0.smithi038.stdout:211: set_chunk oid 12 offset: 584704 length: 898048 target oid smithi03829856-12 low_tier offset: 584704 snap -1 2024-03-19T11:30:05.606 INFO:tasks.rados.rados.0.smithi038.stdout:212: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.606 INFO:tasks.rados.rados.0.smithi038.stdout:212: set_chunk oid 13 offset: 55296 length: 139264 target oid smithi03829856-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 55296 snap -1 2024-03-19T11:30:05.606 INFO:tasks.rados.rados.0.smithi038.stdout:213: set_chunk oid 14 target oid 14 low_tier 2024-03-19T11:30:05.606 INFO:tasks.rados.rados.0.smithi038.stdout:213: set_chunk oid 14 offset: 561152 length: 743936 target oid smithi03829856-14 low_tier offset: 561152 snap -1 2024-03-19T11:30:05.606 INFO:tasks.rados.rados.0.smithi038.stdout:214: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.607 INFO:tasks.rados.rados.0.smithi038.stdout:214: set_chunk oid 15 offset: 360960 length: 1664512 target oid smithi03829856-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 360960 snap -1 2024-03-19T11:30:05.607 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.607 INFO:tasks.rados.rados.0.smithi038.stdout:199: finishing set_chunk to oid 50 2024-03-19T11:30:05.607 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 50 set_chunk 50 low_tier returned error code -95 offset: 960000 length: 1705472 2024-03-19T11:30:05.607 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 50 v 0 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-03-19T11:30:05.607 INFO:tasks.rados.rados.0.smithi038.stdout:199: done (15 left) 2024-03-19T11:30:05.607 INFO:tasks.rados.rados.0.smithi038.stdout:215: set_chunk oid 16 target oid 16 low_tier 2024-03-19T11:30:05.607 INFO:tasks.rados.rados.0.smithi038.stdout:215: set_chunk oid 16 offset: 1528832 length: 547840 target oid smithi03829856-16 low_tier offset: 1528832 snap -1 2024-03-19T11:30:05.607 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.607 INFO:tasks.rados.rados.0.smithi038.stdout:202: finishing set_chunk to oid 3 2024-03-19T11:30:05.607 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 3 set_chunk 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 321536 length: 1463808 2024-03-19T11:30:05.607 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 3 v 0 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-03-19T11:30:05.607 INFO:tasks.rados.rados.0.smithi038.stdout:204: finishing set_chunk to oid 5 2024-03-19T11:30:05.607 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 5 set_chunk 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1699328 length: 305152 2024-03-19T11:30:05.608 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-03-19T11:30:05.608 INFO:tasks.rados.rados.0.smithi038.stdout:202: done (15 left) 2024-03-19T11:30:05.608 INFO:tasks.rados.rados.0.smithi038.stdout:204: done (14 left) 2024-03-19T11:30:05.608 INFO:tasks.rados.rados.0.smithi038.stdout:216: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.608 INFO:tasks.rados.rados.0.smithi038.stdout:216: set_chunk oid 17 offset: 1620992 length: 194560 target oid smithi03829856-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1620992 snap -1 2024-03-19T11:30:05.608 INFO:tasks.rados.rados.0.smithi038.stdout:217: set_chunk oid 18 target oid 18 low_tier 2024-03-19T11:30:05.608 INFO:tasks.rados.rados.0.smithi038.stdout:217: set_chunk oid 18 offset: 1187840 length: 83968 target oid smithi03829856-18 low_tier offset: 1187840 snap -1 2024-03-19T11:30:05.608 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.608 INFO:tasks.rados.rados.0.smithi038.stdout:206: finishing set_chunk to oid 7 2024-03-19T11:30:05.608 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 7 set_chunk 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 72704 length: 1282048 2024-03-19T11:30:05.608 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-03-19T11:30:05.608 INFO:tasks.rados.rados.0.smithi038.stdout:209: finishing set_chunk to oid 10 2024-03-19T11:30:05.609 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 10 set_chunk 10 low_tier returned error code -95 offset: 568320 length: 1214976 2024-03-19T11:30:05.609 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 10 v 0 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-03-19T11:30:05.609 INFO:tasks.rados.rados.0.smithi038.stdout:211: finishing set_chunk to oid 12 2024-03-19T11:30:05.609 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 12 set_chunk 12 low_tier returned error code -95 offset: 584704 length: 898048 2024-03-19T11:30:05.609 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 12 v 0 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-03-19T11:30:05.609 INFO:tasks.rados.rados.0.smithi038.stdout:212: finishing set_chunk to oid 13 2024-03-19T11:30:05.609 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 13 set_chunk 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 55296 length: 139264 2024-03-19T11:30:05.609 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 13 v 0 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-03-19T11:30:05.609 INFO:tasks.rados.rados.0.smithi038.stdout:214: finishing set_chunk to oid 15 2024-03-19T11:30:05.609 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 360960 length: 1664512 2024-03-19T11:30:05.609 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-03-19T11:30:05.609 INFO:tasks.rados.rados.0.smithi038.stdout:210: finishing set_chunk to oid 11 2024-03-19T11:30:05.609 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 11 set_chunk 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 558592 length: 1562112 2024-03-19T11:30:05.609 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 11 v 0 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-03-19T11:30:05.609 INFO:tasks.rados.rados.0.smithi038.stdout:200: finishing set_chunk to oid 1 2024-03-19T11:30:05.610 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 1 set_chunk 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 437760 length: 1145856 2024-03-19T11:30:05.610 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 1 v 0 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-03-19T11:30:05.610 INFO:tasks.rados.rados.0.smithi038.stdout:203: finishing set_chunk to oid 4 2024-03-19T11:30:05.610 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 4 set_chunk 4 low_tier returned error code -95 offset: 1232896 length: 353792 2024-03-19T11:30:05.610 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-03-19T11:30:05.610 INFO:tasks.rados.rados.0.smithi038.stdout:213: finishing set_chunk to oid 14 2024-03-19T11:30:05.610 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 14 set_chunk 14 low_tier returned error code -95 offset: 561152 length: 743936 2024-03-19T11:30:05.610 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 14 v 0 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-03-19T11:30:05.610 INFO:tasks.rados.rados.0.smithi038.stdout:200: done (15 left) 2024-03-19T11:30:05.610 INFO:tasks.rados.rados.0.smithi038.stdout:203: done (14 left) 2024-03-19T11:30:05.610 INFO:tasks.rados.rados.0.smithi038.stdout:206: done (13 left) 2024-03-19T11:30:05.610 INFO:tasks.rados.rados.0.smithi038.stdout:209: done (12 left) 2024-03-19T11:30:05.610 INFO:tasks.rados.rados.0.smithi038.stdout:210: done (11 left) 2024-03-19T11:30:05.610 INFO:tasks.rados.rados.0.smithi038.stdout:211: done (10 left) 2024-03-19T11:30:05.610 INFO:tasks.rados.rados.0.smithi038.stdout:212: done (9 left) 2024-03-19T11:30:05.611 INFO:tasks.rados.rados.0.smithi038.stdout:213: done (8 left) 2024-03-19T11:30:05.611 INFO:tasks.rados.rados.0.smithi038.stdout:214: done (7 left) 2024-03-19T11:30:05.611 INFO:tasks.rados.rados.0.smithi038.stdout:218: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.611 INFO:tasks.rados.rados.0.smithi038.stdout:218: set_chunk oid 19 offset: 920064 length: 253440 target oid smithi03829856-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 920064 snap -1 2024-03-19T11:30:05.611 INFO:tasks.rados.rados.0.smithi038.stdout:219: set_chunk oid 20 target oid 20 low_tier 2024-03-19T11:30:05.611 INFO:tasks.rados.rados.0.smithi038.stdout:219: set_chunk oid 20 offset: 1453568 length: 837632 target oid smithi03829856-20 low_tier offset: 1453568 snap -1 2024-03-19T11:30:05.611 INFO:tasks.rados.rados.0.smithi038.stdout:220: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.611 INFO:tasks.rados.rados.0.smithi038.stdout:220: set_chunk oid 21 offset: 577024 length: 1006592 target oid smithi03829856-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 577024 snap -1 2024-03-19T11:30:05.611 INFO:tasks.rados.rados.0.smithi038.stdout:221: set_chunk oid 22 target oid 22 low_tier 2024-03-19T11:30:05.611 INFO:tasks.rados.rados.0.smithi038.stdout:221: set_chunk oid 22 offset: 980992 length: 249856 target oid smithi03829856-22 low_tier offset: 980992 snap -1 2024-03-19T11:30:05.611 INFO:tasks.rados.rados.0.smithi038.stdout:222: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.611 INFO:tasks.rados.rados.0.smithi038.stdout:222: set_chunk oid 23 offset: 614400 length: 1725952 target oid smithi03829856-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 614400 snap -1 2024-03-19T11:30:05.611 INFO:tasks.rados.rados.0.smithi038.stdout:223: set_chunk oid 24 target oid 24 low_tier 2024-03-19T11:30:05.612 INFO:tasks.rados.rados.0.smithi038.stdout:223: set_chunk oid 24 offset: 349696 length: 381952 target oid smithi03829856-24 low_tier offset: 349696 snap -1 2024-03-19T11:30:05.612 INFO:tasks.rados.rados.0.smithi038.stdout:224: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.612 INFO:tasks.rados.rados.0.smithi038.stdout:224: set_chunk oid 25 offset: 202752 length: 564224 target oid smithi03829856-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 202752 snap -1 2024-03-19T11:30:05.612 INFO:tasks.rados.rados.0.smithi038.stdout:225: set_chunk oid 26 target oid 26 low_tier 2024-03-19T11:30:05.612 INFO:tasks.rados.rados.0.smithi038.stdout:225: set_chunk oid 26 offset: 1004032 length: 1111552 target oid smithi03829856-26 low_tier offset: 1004032 snap -1 2024-03-19T11:30:05.612 INFO:tasks.rados.rados.0.smithi038.stdout:226: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.612 INFO:tasks.rados.rados.0.smithi038.stdout:226: set_chunk oid 27 offset: 673280 length: 1387520 target oid smithi03829856-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 673280 snap -1 2024-03-19T11:30:05.612 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.612 INFO:tasks.rados.rados.0.smithi038.stdout:215: finishing set_chunk to oid 16 2024-03-19T11:30:05.612 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 16 set_chunk 16 low_tier returned error code -95 offset: 1528832 length: 547840 2024-03-19T11:30:05.612 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-03-19T11:30:05.612 INFO:tasks.rados.rados.0.smithi038.stdout:217: finishing set_chunk to oid 18 2024-03-19T11:30:05.612 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 18 set_chunk 18 low_tier returned error code -95 offset: 1187840 length: 83968 2024-03-19T11:30:05.612 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 18 v 0 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-03-19T11:30:05.612 INFO:tasks.rados.rados.0.smithi038.stdout:215: done (15 left) 2024-03-19T11:30:05.613 INFO:tasks.rados.rados.0.smithi038.stdout:217: done (14 left) 2024-03-19T11:30:05.613 INFO:tasks.rados.rados.0.smithi038.stdout:227: set_chunk oid 28 target oid 28 low_tier 2024-03-19T11:30:05.613 INFO:tasks.rados.rados.0.smithi038.stdout:227: set_chunk oid 28 offset: 67584 length: 1520640 target oid smithi03829856-28 low_tier offset: 67584 snap -1 2024-03-19T11:30:05.613 INFO:tasks.rados.rados.0.smithi038.stdout:228: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.613 INFO:tasks.rados.rados.0.smithi038.stdout:228: set_chunk oid 29 offset: 653312 length: 1058304 target oid smithi03829856-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 653312 snap -1 2024-03-19T11:30:05.613 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.613 INFO:tasks.rados.rados.0.smithi038.stdout:218: finishing set_chunk to oid 19 2024-03-19T11:30:05.613 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 19 set_chunk 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 920064 length: 253440 2024-03-19T11:30:05.613 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 19 v 0 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-03-19T11:30:05.613 INFO:tasks.rados.rados.0.smithi038.stdout:218: done (15 left) 2024-03-19T11:30:05.613 INFO:tasks.rados.rados.0.smithi038.stdout:229: set_chunk oid 30 target oid 30 low_tier 2024-03-19T11:30:05.613 INFO:tasks.rados.rados.0.smithi038.stdout:229: set_chunk oid 30 offset: 240640 length: 580096 target oid smithi03829856-30 low_tier offset: 240640 snap -1 2024-03-19T11:30:05.613 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.613 INFO:tasks.rados.rados.0.smithi038.stdout:221: finishing set_chunk to oid 22 2024-03-19T11:30:05.613 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 22 set_chunk 22 low_tier returned error code -95 offset: 980992 length: 249856 2024-03-19T11:30:05.614 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-03-19T11:30:05.614 INFO:tasks.rados.rados.0.smithi038.stdout:222: finishing set_chunk to oid 23 2024-03-19T11:30:05.614 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 23 set_chunk 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 614400 length: 1725952 2024-03-19T11:30:05.614 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 23 v 0 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-03-19T11:30:05.614 INFO:tasks.rados.rados.0.smithi038.stdout:221: done (15 left) 2024-03-19T11:30:05.614 INFO:tasks.rados.rados.0.smithi038.stdout:222: done (14 left) 2024-03-19T11:30:05.614 INFO:tasks.rados.rados.0.smithi038.stdout:230: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.614 INFO:tasks.rados.rados.0.smithi038.stdout:230: set_chunk oid 31 offset: 2022400 length: 342528 target oid smithi03829856-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 2022400 snap -1 2024-03-19T11:30:05.614 INFO:tasks.rados.rados.0.smithi038.stdout:231: set_chunk oid 32 target oid 32 low_tier 2024-03-19T11:30:05.614 INFO:tasks.rados.rados.0.smithi038.stdout:231: set_chunk oid 32 offset: 476160 length: 98304 target oid smithi03829856-32 low_tier offset: 476160 snap -1 2024-03-19T11:30:05.614 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.614 INFO:tasks.rados.rados.0.smithi038.stdout:219: finishing set_chunk to oid 20 2024-03-19T11:30:05.614 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 20 set_chunk 20 low_tier returned error code -95 offset: 1453568 length: 837632 2024-03-19T11:30:05.614 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 20 v 0 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-03-19T11:30:05.614 INFO:tasks.rados.rados.0.smithi038.stdout:226: finishing set_chunk to oid 27 2024-03-19T11:30:05.615 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 27 set_chunk 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 673280 length: 1387520 2024-03-19T11:30:05.615 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-03-19T11:30:05.615 INFO:tasks.rados.rados.0.smithi038.stdout:219: done (15 left) 2024-03-19T11:30:05.615 INFO:tasks.rados.rados.0.smithi038.stdout:226: done (14 left) 2024-03-19T11:30:05.615 INFO:tasks.rados.rados.0.smithi038.stdout:232: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.615 INFO:tasks.rados.rados.0.smithi038.stdout:232: set_chunk oid 33 offset: 13312 length: 1454080 target oid smithi03829856-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 13312 snap -1 2024-03-19T11:30:05.615 INFO:tasks.rados.rados.0.smithi038.stdout:233: set_chunk oid 34 target oid 34 low_tier 2024-03-19T11:30:05.615 INFO:tasks.rados.rados.0.smithi038.stdout:233: set_chunk oid 34 offset: 2421760 length: 72192 target oid smithi03829856-34 low_tier offset: 2421760 snap -1 2024-03-19T11:30:05.615 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.615 INFO:tasks.rados.rados.0.smithi038.stdout:223: finishing set_chunk to oid 24 2024-03-19T11:30:05.615 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 24 set_chunk 24 low_tier returned error code -95 offset: 349696 length: 381952 2024-03-19T11:30:05.615 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-03-19T11:30:05.615 INFO:tasks.rados.rados.0.smithi038.stdout:220: finishing set_chunk to oid 21 2024-03-19T11:30:05.615 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 21 set_chunk 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 577024 length: 1006592 2024-03-19T11:30:05.615 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-03-19T11:30:05.616 INFO:tasks.rados.rados.0.smithi038.stdout:220: done (15 left) 2024-03-19T11:30:05.616 INFO:tasks.rados.rados.0.smithi038.stdout:223: done (14 left) 2024-03-19T11:30:05.616 INFO:tasks.rados.rados.0.smithi038.stdout:234: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.616 INFO:tasks.rados.rados.0.smithi038.stdout:234: set_chunk oid 35 offset: 1333248 length: 553984 target oid smithi03829856-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1333248 snap -1 2024-03-19T11:30:05.616 INFO:tasks.rados.rados.0.smithi038.stdout:235: set_chunk oid 36 target oid 36 low_tier 2024-03-19T11:30:05.616 INFO:tasks.rados.rados.0.smithi038.stdout:235: set_chunk oid 36 offset: 80384 length: 1646592 target oid smithi03829856-36 low_tier offset: 80384 snap -1 2024-03-19T11:30:05.616 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.616 INFO:tasks.rados.rados.0.smithi038.stdout:225: finishing set_chunk to oid 26 2024-03-19T11:30:05.616 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 26 set_chunk 26 low_tier returned error code -95 offset: 1004032 length: 1111552 2024-03-19T11:30:05.616 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-03-19T11:30:05.616 INFO:tasks.rados.rados.0.smithi038.stdout:228: finishing set_chunk to oid 29 2024-03-19T11:30:05.616 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 29 set_chunk 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 653312 length: 1058304 2024-03-19T11:30:05.617 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 29 v 0 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-03-19T11:30:05.617 INFO:tasks.rados.rados.0.smithi038.stdout:229: finishing set_chunk to oid 30 2024-03-19T11:30:05.617 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 30 set_chunk 30 low_tier returned error code -95 offset: 240640 length: 580096 2024-03-19T11:30:05.617 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 30 v 0 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-03-19T11:30:05.617 INFO:tasks.rados.rados.0.smithi038.stdout:232: finishing set_chunk to oid 33 2024-03-19T11:30:05.617 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 33 set_chunk 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 13312 length: 1454080 2024-03-19T11:30:05.617 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-03-19T11:30:05.617 INFO:tasks.rados.rados.0.smithi038.stdout:225: done (15 left) 2024-03-19T11:30:05.617 INFO:tasks.rados.rados.0.smithi038.stdout:228: done (14 left) 2024-03-19T11:30:05.617 INFO:tasks.rados.rados.0.smithi038.stdout:229: done (13 left) 2024-03-19T11:30:05.617 INFO:tasks.rados.rados.0.smithi038.stdout:232: done (12 left) 2024-03-19T11:30:05.617 INFO:tasks.rados.rados.0.smithi038.stdout:236: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.617 INFO:tasks.rados.rados.0.smithi038.stdout:236: set_chunk oid 37 offset: 1971200 length: 93696 target oid smithi03829856-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1971200 snap -1 2024-03-19T11:30:05.617 INFO:tasks.rados.rados.0.smithi038.stdout:237: set_chunk oid 38 target oid 38 low_tier 2024-03-19T11:30:05.618 INFO:tasks.rados.rados.0.smithi038.stdout:237: set_chunk oid 38 offset: 1397760 length: 284672 target oid smithi03829856-38 low_tier offset: 1397760 snap -1 2024-03-19T11:30:05.618 INFO:tasks.rados.rados.0.smithi038.stdout:238: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.618 INFO:tasks.rados.rados.0.smithi038.stdout:238: set_chunk oid 39 offset: 872960 length: 1259008 target oid smithi03829856-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 872960 snap -1 2024-03-19T11:30:05.618 INFO:tasks.rados.rados.0.smithi038.stdout:239: set_chunk oid 40 target oid 40 low_tier 2024-03-19T11:30:05.618 INFO:tasks.rados.rados.0.smithi038.stdout:239: set_chunk oid 40 offset: 863232 length: 851968 target oid smithi03829856-40 low_tier offset: 863232 snap -1 2024-03-19T11:30:05.618 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.618 INFO:tasks.rados.rados.0.smithi038.stdout:234: finishing set_chunk to oid 35 2024-03-19T11:30:05.618 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 35 set_chunk 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1333248 length: 553984 2024-03-19T11:30:05.618 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 35 v 0 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-03-19T11:30:05.618 INFO:tasks.rados.rados.0.smithi038.stdout:234: done (15 left) 2024-03-19T11:30:05.618 INFO:tasks.rados.rados.0.smithi038.stdout:240: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.618 INFO:tasks.rados.rados.0.smithi038.stdout:240: set_chunk oid 41 offset: 59904 length: 877568 target oid smithi03829856-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 59904 snap -1 2024-03-19T11:30:05.618 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.618 INFO:tasks.rados.rados.0.smithi038.stdout:235: finishing set_chunk to oid 36 2024-03-19T11:30:05.619 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 36 set_chunk 36 low_tier returned error code -95 offset: 80384 length: 1646592 2024-03-19T11:30:05.619 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-03-19T11:30:05.619 INFO:tasks.rados.rados.0.smithi038.stdout:235: done (15 left) 2024-03-19T11:30:05.619 INFO:tasks.rados.rados.0.smithi038.stdout:241: set_chunk oid 42 target oid 42 low_tier 2024-03-19T11:30:05.619 INFO:tasks.rados.rados.0.smithi038.stdout:241: set_chunk oid 42 offset: 2555392 length: 39424 target oid smithi03829856-42 low_tier offset: 2555392 snap -1 2024-03-19T11:30:05.619 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.619 INFO:tasks.rados.rados.0.smithi038.stdout:238: finishing set_chunk to oid 39 2024-03-19T11:30:05.619 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 39 set_chunk 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 872960 length: 1259008 2024-03-19T11:30:05.619 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 39 v 0 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-03-19T11:30:05.619 INFO:tasks.rados.rados.0.smithi038.stdout:227: finishing set_chunk to oid 28 2024-03-19T11:30:05.619 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 28 set_chunk 28 low_tier returned error code -95 offset: 67584 length: 1520640 2024-03-19T11:30:05.619 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 28 v 0 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-03-19T11:30:05.619 INFO:tasks.rados.rados.0.smithi038.stdout:237: finishing set_chunk to oid 38 2024-03-19T11:30:05.619 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 38 set_chunk 38 low_tier returned error code -95 offset: 1397760 length: 284672 2024-03-19T11:30:05.619 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 38 v 0 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-03-19T11:30:05.620 INFO:tasks.rados.rados.0.smithi038.stdout:227: done (15 left) 2024-03-19T11:30:05.620 INFO:tasks.rados.rados.0.smithi038.stdout:237: done (14 left) 2024-03-19T11:30:05.620 INFO:tasks.rados.rados.0.smithi038.stdout:238: done (13 left) 2024-03-19T11:30:05.620 INFO:tasks.rados.rados.0.smithi038.stdout:242: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.620 INFO:tasks.rados.rados.0.smithi038.stdout:242: set_chunk oid 43 offset: 1357824 length: 927232 target oid smithi03829856-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1357824 snap -1 2024-03-19T11:30:05.620 INFO:tasks.rados.rados.0.smithi038.stdout:243: set_chunk oid 44 target oid 44 low_tier 2024-03-19T11:30:05.620 INFO:tasks.rados.rados.0.smithi038.stdout:243: set_chunk oid 44 offset: 1565184 length: 443392 target oid smithi03829856-44 low_tier offset: 1565184 snap -1 2024-03-19T11:30:05.620 INFO:tasks.rados.rados.0.smithi038.stdout:244: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.620 INFO:tasks.rados.rados.0.smithi038.stdout:244: set_chunk oid 45 offset: 588288 length: 1318912 target oid smithi03829856-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 588288 snap -1 2024-03-19T11:30:05.620 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.620 INFO:tasks.rados.rados.0.smithi038.stdout:240: finishing set_chunk to oid 41 2024-03-19T11:30:05.620 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 41 set_chunk 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 59904 length: 877568 2024-03-19T11:30:05.620 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 41 v 0 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-03-19T11:30:05.621 INFO:tasks.rados.rados.0.smithi038.stdout:239: finishing set_chunk to oid 40 2024-03-19T11:30:05.621 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 40 set_chunk 40 low_tier returned error code -95 offset: 863232 length: 851968 2024-03-19T11:30:05.621 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 40 v 0 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-03-19T11:30:05.621 INFO:tasks.rados.rados.0.smithi038.stdout:239: done (15 left) 2024-03-19T11:30:05.621 INFO:tasks.rados.rados.0.smithi038.stdout:240: done (14 left) 2024-03-19T11:30:05.621 INFO:tasks.rados.rados.0.smithi038.stdout:245: set_chunk oid 46 target oid 46 low_tier 2024-03-19T11:30:05.621 INFO:tasks.rados.rados.0.smithi038.stdout:245: set_chunk oid 46 offset: 890368 length: 1423360 target oid smithi03829856-46 low_tier offset: 890368 snap -1 2024-03-19T11:30:05.621 INFO:tasks.rados.rados.0.smithi038.stdout:246: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.621 INFO:tasks.rados.rados.0.smithi038.stdout:246: set_chunk oid 47 offset: 156160 length: 1734144 target oid smithi03829856-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 156160 snap -1 2024-03-19T11:30:05.621 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.621 INFO:tasks.rados.rados.0.smithi038.stdout:236: finishing set_chunk to oid 37 2024-03-19T11:30:05.621 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 37 set_chunk 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1971200 length: 93696 2024-03-19T11:30:05.621 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 37 v 0 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-03-19T11:30:05.621 INFO:tasks.rados.rados.0.smithi038.stdout:236: done (15 left) 2024-03-19T11:30:05.622 INFO:tasks.rados.rados.0.smithi038.stdout:247: set_chunk oid 48 target oid 48 low_tier 2024-03-19T11:30:05.622 INFO:tasks.rados.rados.0.smithi038.stdout:247: set_chunk oid 48 offset: 820224 length: 532480 target oid smithi03829856-48 low_tier offset: 820224 snap -1 2024-03-19T11:30:05.622 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.622 INFO:tasks.rados.rados.0.smithi038.stdout:242: finishing set_chunk to oid 43 2024-03-19T11:30:05.622 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 43 set_chunk 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1357824 length: 927232 2024-03-19T11:30:05.622 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 43 v 0 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-03-19T11:30:05.622 INFO:tasks.rados.rados.0.smithi038.stdout:245: finishing set_chunk to oid 46 2024-03-19T11:30:05.622 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 46 set_chunk 46 low_tier returned error code -95 offset: 890368 length: 1423360 2024-03-19T11:30:05.622 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-03-19T11:30:05.622 INFO:tasks.rados.rados.0.smithi038.stdout:242: done (15 left) 2024-03-19T11:30:05.622 INFO:tasks.rados.rados.0.smithi038.stdout:245: done (14 left) 2024-03-19T11:30:05.622 INFO:tasks.rados.rados.0.smithi038.stdout:248: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.622 INFO:tasks.rados.rados.0.smithi038.stdout:248: set_chunk oid 49 offset: 1823232 length: 187392 target oid smithi03829856-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1823232 snap -1 2024-03-19T11:30:05.622 INFO:tasks.rados.rados.0.smithi038.stdout:249: set_chunk oid 50 target oid 50 low_tier 2024-03-19T11:30:05.622 INFO:tasks.rados.rados.0.smithi038.stdout:249: set_chunk oid 50 offset: 2479104 length: 4096 target oid smithi03829856-50 low_tier offset: 2479104 snap -1 2024-03-19T11:30:05.623 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.623 INFO:tasks.rados.rados.0.smithi038.stdout:243: finishing set_chunk to oid 44 2024-03-19T11:30:05.623 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 44 set_chunk 44 low_tier returned error code -95 offset: 1565184 length: 443392 2024-03-19T11:30:05.623 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 44 v 0 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-03-19T11:30:05.623 INFO:tasks.rados.rados.0.smithi038.stdout:244: finishing set_chunk to oid 45 2024-03-19T11:30:05.623 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 45 set_chunk 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 588288 length: 1318912 2024-03-19T11:30:05.623 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 45 v 0 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-03-19T11:30:05.623 INFO:tasks.rados.rados.0.smithi038.stdout:247: finishing set_chunk to oid 48 2024-03-19T11:30:05.623 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 48 set_chunk 48 low_tier returned error code -95 offset: 820224 length: 532480 2024-03-19T11:30:05.623 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 48 v 0 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-03-19T11:30:05.623 INFO:tasks.rados.rados.0.smithi038.stdout:246: finishing set_chunk to oid 47 2024-03-19T11:30:05.623 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 47 set_chunk 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 156160 length: 1734144 2024-03-19T11:30:05.623 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-03-19T11:30:05.623 INFO:tasks.rados.rados.0.smithi038.stdout:243: done (15 left) 2024-03-19T11:30:05.623 INFO:tasks.rados.rados.0.smithi038.stdout:244: done (14 left) 2024-03-19T11:30:05.624 INFO:tasks.rados.rados.0.smithi038.stdout:246: done (13 left) 2024-03-19T11:30:05.624 INFO:tasks.rados.rados.0.smithi038.stdout:247: done (12 left) 2024-03-19T11:30:05.624 INFO:tasks.rados.rados.0.smithi038.stdout:250: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.624 INFO:tasks.rados.rados.0.smithi038.stdout:250: set_chunk oid 1 offset: 1216000 length: 103424 target oid smithi03829856-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1216000 snap -1 2024-03-19T11:30:05.624 INFO:tasks.rados.rados.0.smithi038.stdout:250 retry set_chunk ! 2024-03-19T11:30:05.624 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 13 2024-03-19T11:30:05.624 INFO:tasks.rados.rados.0.smithi038.stdout:207: finishing set_chunk to oid 8 2024-03-19T11:30:05.624 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 8 v 3 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-03-19T11:30:05.624 INFO:tasks.rados.rados.0.smithi038.stdout:207: done (12 left) 2024-03-19T11:30:05.624 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 12 2024-03-19T11:30:05.624 INFO:tasks.rados.rados.0.smithi038.stdout:201: finishing set_chunk to oid 2 2024-03-19T11:30:05.624 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 2 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-03-19T11:30:05.624 INFO:tasks.rados.rados.0.smithi038.stdout:205: finishing set_chunk to oid 6 2024-03-19T11:30:05.624 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 6 v 3 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-03-19T11:30:05.624 INFO:tasks.rados.rados.0.smithi038.stdout:201: done (11 left) 2024-03-19T11:30:05.624 INFO:tasks.rados.rados.0.smithi038.stdout:205: done (10 left) 2024-03-19T11:30:05.625 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 10 2024-03-19T11:30:05.625 INFO:tasks.rados.rados.0.smithi038.stdout:241: finishing set_chunk to oid 42 2024-03-19T11:30:05.625 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 42 v 12 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-03-19T11:30:05.625 INFO:tasks.rados.rados.0.smithi038.stdout:241: done (9 left) 2024-03-19T11:30:05.625 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 9 2024-03-19T11:30:05.625 INFO:tasks.rados.rados.0.smithi038.stdout:231: finishing set_chunk to oid 32 2024-03-19T11:30:05.625 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 32 v 6 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-03-19T11:30:05.625 INFO:tasks.rados.rados.0.smithi038.stdout:231: done (8 left) 2024-03-19T11:30:05.625 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 8 2024-03-19T11:30:05.625 INFO:tasks.rados.rados.0.smithi038.stdout:208: finishing set_chunk to oid 9 2024-03-19T11:30:05.625 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 9 v 6 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-03-19T11:30:05.625 INFO:tasks.rados.rados.0.smithi038.stdout:208: done (7 left) 2024-03-19T11:30:05.625 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 7 2024-03-19T11:30:05.625 INFO:tasks.rados.rados.0.smithi038.stdout:216: finishing set_chunk to oid 17 2024-03-19T11:30:05.625 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 17 v 9 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-03-19T11:30:05.626 INFO:tasks.rados.rados.0.smithi038.stdout:216: done (6 left) 2024-03-19T11:30:05.626 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 6 2024-03-19T11:30:05.626 INFO:tasks.rados.rados.0.smithi038.stdout:250: finishing set_chunk to oid 1 2024-03-19T11:30:05.626 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-03-19T11:30:05.626 INFO:tasks.rados.rados.0.smithi038.stdout:250: done (5 left) 2024-03-19T11:30:05.626 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 5 2024-03-19T11:30:05.626 INFO:tasks.rados.rados.0.smithi038.stdout:224: finishing set_chunk to oid 25 2024-03-19T11:30:05.626 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 25 v 12 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-03-19T11:30:05.626 INFO:tasks.rados.rados.0.smithi038.stdout:224: done (4 left) 2024-03-19T11:30:05.626 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 4 2024-03-19T11:30:05.626 INFO:tasks.rados.rados.0.smithi038.stdout:233: finishing set_chunk to oid 34 2024-03-19T11:30:05.626 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 34 v 3 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-03-19T11:30:05.626 INFO:tasks.rados.rados.0.smithi038.stdout:233: done (3 left) 2024-03-19T11:30:05.626 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 3 2024-03-19T11:30:05.627 INFO:tasks.rados.rados.0.smithi038.stdout:230: finishing set_chunk to oid 31 2024-03-19T11:30:05.627 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 31 v 6 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-03-19T11:30:05.627 INFO:tasks.rados.rados.0.smithi038.stdout:230: done (2 left) 2024-03-19T11:30:05.627 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 2 2024-03-19T11:30:05.627 INFO:tasks.rados.rados.0.smithi038.stdout:249: finishing set_chunk to oid 50 2024-03-19T11:30:05.627 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 50 v 15 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-03-19T11:30:05.627 INFO:tasks.rados.rados.0.smithi038.stdout:249: done (1 left) 2024-03-19T11:30:05.627 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 1 2024-03-19T11:30:05.627 INFO:tasks.rados.rados.0.smithi038.stdout:248: finishing set_chunk to oid 49 2024-03-19T11:30:05.627 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 49 v 9 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-03-19T11:30:05.627 INFO:tasks.rados.rados.0.smithi038.stdout:248: done (0 left) 2024-03-19T11:30:05.627 INFO:tasks.rados.rados.0.smithi038.stdout:251: set_chunk oid 2 target oid 2 low_tier 2024-03-19T11:30:05.627 INFO:tasks.rados.rados.0.smithi038.stdout:251: set_chunk oid 2 offset: 388608 length: 1506304 target oid smithi03829856-2 low_tier offset: 388608 snap -1 2024-03-19T11:30:05.627 INFO:tasks.rados.rados.0.smithi038.stdout:252: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.627 INFO:tasks.rados.rados.0.smithi038.stdout:252: set_chunk oid 3 offset: 752640 length: 64000 target oid smithi03829856-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 752640 snap -1 2024-03-19T11:30:05.628 INFO:tasks.rados.rados.0.smithi038.stdout:253: set_chunk oid 4 target oid 4 low_tier 2024-03-19T11:30:05.628 INFO:tasks.rados.rados.0.smithi038.stdout:253: set_chunk oid 4 offset: 260096 length: 829952 target oid smithi03829856-4 low_tier offset: 260096 snap -1 2024-03-19T11:30:05.628 INFO:tasks.rados.rados.0.smithi038.stdout:254: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.628 INFO:tasks.rados.rados.0.smithi038.stdout:254: set_chunk oid 5 offset: 410624 length: 1355776 target oid smithi03829856-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 410624 snap -1 2024-03-19T11:30:05.628 INFO:tasks.rados.rados.0.smithi038.stdout:255: set_chunk oid 6 target oid 6 low_tier 2024-03-19T11:30:05.628 INFO:tasks.rados.rados.0.smithi038.stdout:255: set_chunk oid 6 offset: 455168 length: 617984 target oid smithi03829856-6 low_tier offset: 455168 snap -1 2024-03-19T11:30:05.628 INFO:tasks.rados.rados.0.smithi038.stdout:256: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.628 INFO:tasks.rados.rados.0.smithi038.stdout:256: set_chunk oid 7 offset: 93184 length: 1929216 target oid smithi03829856-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 93184 snap -1 2024-03-19T11:30:05.628 INFO:tasks.rados.rados.0.smithi038.stdout:257: set_chunk oid 8 target oid 8 low_tier 2024-03-19T11:30:05.628 INFO:tasks.rados.rados.0.smithi038.stdout:257: set_chunk oid 8 offset: 29184 length: 564224 target oid smithi03829856-8 low_tier offset: 29184 snap -1 2024-03-19T11:30:05.628 INFO:tasks.rados.rados.0.smithi038.stdout:258: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.628 INFO:tasks.rados.rados.0.smithi038.stdout:258: set_chunk oid 9 offset: 298496 length: 913920 target oid smithi03829856-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 298496 snap -1 2024-03-19T11:30:05.628 INFO:tasks.rados.rados.0.smithi038.stdout:259: set_chunk oid 10 target oid 10 low_tier 2024-03-19T11:30:05.628 INFO:tasks.rados.rados.0.smithi038.stdout:259: set_chunk oid 10 offset: 742400 length: 1257472 target oid smithi03829856-10 low_tier offset: 742400 snap -1 2024-03-19T11:30:05.628 INFO:tasks.rados.rados.0.smithi038.stdout:260: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.629 INFO:tasks.rados.rados.0.smithi038.stdout:260: set_chunk oid 11 offset: 404992 length: 829440 target oid smithi03829856-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 404992 snap -1 2024-03-19T11:30:05.629 INFO:tasks.rados.rados.0.smithi038.stdout:261: set_chunk oid 12 target oid 12 low_tier 2024-03-19T11:30:05.629 INFO:tasks.rados.rados.0.smithi038.stdout:261: set_chunk oid 12 offset: 93696 length: 320000 target oid smithi03829856-12 low_tier offset: 93696 snap -1 2024-03-19T11:30:05.629 INFO:tasks.rados.rados.0.smithi038.stdout:262: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.629 INFO:tasks.rados.rados.0.smithi038.stdout:262: set_chunk oid 13 offset: 85504 length: 1895424 target oid smithi03829856-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 85504 snap -1 2024-03-19T11:30:05.629 INFO:tasks.rados.rados.0.smithi038.stdout:263: set_chunk oid 14 target oid 14 low_tier 2024-03-19T11:30:05.629 INFO:tasks.rados.rados.0.smithi038.stdout:263: set_chunk oid 14 offset: 313856 length: 1835008 target oid smithi03829856-14 low_tier offset: 313856 snap -1 2024-03-19T11:30:05.629 INFO:tasks.rados.rados.0.smithi038.stdout:264: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.629 INFO:tasks.rados.rados.0.smithi038.stdout:264: set_chunk oid 15 offset: 310784 length: 1069568 target oid smithi03829856-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 310784 snap -1 2024-03-19T11:30:05.629 INFO:tasks.rados.rados.0.smithi038.stdout:265: set_chunk oid 16 target oid 16 low_tier 2024-03-19T11:30:05.629 INFO:tasks.rados.rados.0.smithi038.stdout:265: set_chunk oid 16 offset: 1035264 length: 1168896 target oid smithi03829856-16 low_tier offset: 1035264 snap -1 2024-03-19T11:30:05.629 INFO:tasks.rados.rados.0.smithi038.stdout:266: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.629 INFO:tasks.rados.rados.0.smithi038.stdout:266: set_chunk oid 17 offset: 296448 length: 668672 target oid smithi03829856-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 296448 snap -1 2024-03-19T11:30:05.629 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.629 INFO:tasks.rados.rados.0.smithi038.stdout:251: finishing set_chunk to oid 2 2024-03-19T11:30:05.629 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 2 set_chunk 2 low_tier returned error code -95 offset: 388608 length: 1506304 2024-03-19T11:30:05.630 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-03-19T11:30:05.630 INFO:tasks.rados.rados.0.smithi038.stdout:251: done (15 left) 2024-03-19T11:30:05.630 INFO:tasks.rados.rados.0.smithi038.stdout:267: set_chunk oid 18 target oid 18 low_tier 2024-03-19T11:30:05.630 INFO:tasks.rados.rados.0.smithi038.stdout:267: set_chunk oid 18 offset: 1817088 length: 124416 target oid smithi03829856-18 low_tier offset: 1817088 snap -1 2024-03-19T11:30:05.630 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.630 INFO:tasks.rados.rados.0.smithi038.stdout:253: finishing set_chunk to oid 4 2024-03-19T11:30:05.630 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 4 set_chunk 4 low_tier returned error code -95 offset: 260096 length: 829952 2024-03-19T11:30:05.630 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-03-19T11:30:05.630 INFO:tasks.rados.rados.0.smithi038.stdout:254: finishing set_chunk to oid 5 2024-03-19T11:30:05.630 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 5 set_chunk 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 410624 length: 1355776 2024-03-19T11:30:05.630 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-03-19T11:30:05.630 INFO:tasks.rados.rados.0.smithi038.stdout:255: finishing set_chunk to oid 6 2024-03-19T11:30:05.630 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 6 set_chunk 6 low_tier returned error code -95 offset: 455168 length: 617984 2024-03-19T11:30:05.631 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-03-19T11:30:05.631 INFO:tasks.rados.rados.0.smithi038.stdout:259: finishing set_chunk to oid 10 2024-03-19T11:30:05.631 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 10 set_chunk 10 low_tier returned error code -95 offset: 742400 length: 1257472 2024-03-19T11:30:05.631 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 10 v 0 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-03-19T11:30:05.631 INFO:tasks.rados.rados.0.smithi038.stdout:263: finishing set_chunk to oid 14 2024-03-19T11:30:05.631 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 14 set_chunk 14 low_tier returned error code -95 offset: 313856 length: 1835008 2024-03-19T11:30:05.631 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 14 v 0 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-03-19T11:30:05.631 INFO:tasks.rados.rados.0.smithi038.stdout:253: done (15 left) 2024-03-19T11:30:05.631 INFO:tasks.rados.rados.0.smithi038.stdout:254: done (14 left) 2024-03-19T11:30:05.631 INFO:tasks.rados.rados.0.smithi038.stdout:255: done (13 left) 2024-03-19T11:30:05.631 INFO:tasks.rados.rados.0.smithi038.stdout:259: done (12 left) 2024-03-19T11:30:05.631 INFO:tasks.rados.rados.0.smithi038.stdout:263: done (11 left) 2024-03-19T11:30:05.631 INFO:tasks.rados.rados.0.smithi038.stdout:268: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.631 INFO:tasks.rados.rados.0.smithi038.stdout:268: set_chunk oid 19 offset: 1007104 length: 858624 target oid smithi03829856-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1007104 snap -1 2024-03-19T11:30:05.632 INFO:tasks.rados.rados.0.smithi038.stdout:269: set_chunk oid 20 target oid 20 low_tier 2024-03-19T11:30:05.632 INFO:tasks.rados.rados.0.smithi038.stdout:269: set_chunk oid 20 offset: 1647616 length: 226816 target oid smithi03829856-20 low_tier offset: 1647616 snap -1 2024-03-19T11:30:05.632 INFO:tasks.rados.rados.0.smithi038.stdout:270: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.632 INFO:tasks.rados.rados.0.smithi038.stdout:270: set_chunk oid 21 offset: 2006528 length: 55808 target oid smithi03829856-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 2006528 snap -1 2024-03-19T11:30:05.632 INFO:tasks.rados.rados.0.smithi038.stdout:271: set_chunk oid 22 target oid 22 low_tier 2024-03-19T11:30:05.632 INFO:tasks.rados.rados.0.smithi038.stdout:271: set_chunk oid 22 offset: 2213888 length: 71680 target oid smithi03829856-22 low_tier offset: 2213888 snap -1 2024-03-19T11:30:05.632 INFO:tasks.rados.rados.0.smithi038.stdout:272: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.632 INFO:tasks.rados.rados.0.smithi038.stdout:272: set_chunk oid 23 offset: 1084416 length: 1009664 target oid smithi03829856-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1084416 snap -1 2024-03-19T11:30:05.632 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.632 INFO:tasks.rados.rados.0.smithi038.stdout:257: finishing set_chunk to oid 8 2024-03-19T11:30:05.632 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 8 set_chunk 8 low_tier returned error code -95 offset: 29184 length: 564224 2024-03-19T11:30:05.632 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-03-19T11:30:05.632 INFO:tasks.rados.rados.0.smithi038.stdout:261: finishing set_chunk to oid 12 2024-03-19T11:30:05.632 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 12 set_chunk 12 low_tier returned error code -95 offset: 93696 length: 320000 2024-03-19T11:30:05.632 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 12 v 0 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-03-19T11:30:05.633 INFO:tasks.rados.rados.0.smithi038.stdout:256: finishing set_chunk to oid 7 2024-03-19T11:30:05.633 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 7 set_chunk 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 93184 length: 1929216 2024-03-19T11:30:05.633 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-03-19T11:30:05.633 INFO:tasks.rados.rados.0.smithi038.stdout:260: finishing set_chunk to oid 11 2024-03-19T11:30:05.633 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 11 set_chunk 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 404992 length: 829440 2024-03-19T11:30:05.633 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 11 v 0 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-03-19T11:30:05.633 INFO:tasks.rados.rados.0.smithi038.stdout:258: finishing set_chunk to oid 9 2024-03-19T11:30:05.633 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 9 set_chunk 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 298496 length: 913920 2024-03-19T11:30:05.633 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 9 v 0 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-03-19T11:30:05.633 INFO:tasks.rados.rados.0.smithi038.stdout:256: done (15 left) 2024-03-19T11:30:05.633 INFO:tasks.rados.rados.0.smithi038.stdout:257: done (14 left) 2024-03-19T11:30:05.633 INFO:tasks.rados.rados.0.smithi038.stdout:258: done (13 left) 2024-03-19T11:30:05.633 INFO:tasks.rados.rados.0.smithi038.stdout:260: done (12 left) 2024-03-19T11:30:05.633 INFO:tasks.rados.rados.0.smithi038.stdout:261: done (11 left) 2024-03-19T11:30:05.633 INFO:tasks.rados.rados.0.smithi038.stdout:273: set_chunk oid 24 target oid 24 low_tier 2024-03-19T11:30:05.634 INFO:tasks.rados.rados.0.smithi038.stdout:273: set_chunk oid 24 offset: 722944 length: 1345536 target oid smithi03829856-24 low_tier offset: 722944 snap -1 2024-03-19T11:30:05.634 INFO:tasks.rados.rados.0.smithi038.stdout:274: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.634 INFO:tasks.rados.rados.0.smithi038.stdout:267: finishing set_chunk to oid 18 2024-03-19T11:30:05.634 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 18 set_chunk 18 low_tier returned error code -95 offset: 1817088 length: 124416 2024-03-19T11:30:05.634 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 18 v 0 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-03-19T11:30:05.634 INFO:tasks.rados.rados.0.smithi038.stdout:274: set_chunk oid 25 offset: 501248 length: 1596416 target oid smithi03829856-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 501248 snap -1 2024-03-19T11:30:05.634 INFO:tasks.rados.rados.0.smithi038.stdout:267: done (12 left) 2024-03-19T11:30:05.634 INFO:tasks.rados.rados.0.smithi038.stdout:275: set_chunk oid 26 target oid 26 low_tier 2024-03-19T11:30:05.634 INFO:tasks.rados.rados.0.smithi038.stdout:275: set_chunk oid 26 offset: 396800 length: 1566720 target oid smithi03829856-26 low_tier offset: 396800 snap -1 2024-03-19T11:30:05.634 INFO:tasks.rados.rados.0.smithi038.stdout:276: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.634 INFO:tasks.rados.rados.0.smithi038.stdout:276: set_chunk oid 27 offset: 625152 length: 299008 target oid smithi03829856-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 625152 snap -1 2024-03-19T11:30:05.634 INFO:tasks.rados.rados.0.smithi038.stdout:277: set_chunk oid 28 target oid 28 low_tier 2024-03-19T11:30:05.634 INFO:tasks.rados.rados.0.smithi038.stdout:277: set_chunk oid 28 offset: 1204736 length: 34816 target oid smithi03829856-28 low_tier offset: 1204736 snap -1 2024-03-19T11:30:05.635 INFO:tasks.rados.rados.0.smithi038.stdout:278: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.635 INFO:tasks.rados.rados.0.smithi038.stdout:278: set_chunk oid 29 offset: 189440 length: 2083840 target oid smithi03829856-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 189440 snap -1 2024-03-19T11:30:05.635 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.635 INFO:tasks.rados.rados.0.smithi038.stdout:265: finishing set_chunk to oid 16 2024-03-19T11:30:05.635 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 16 set_chunk 16 low_tier returned error code -95 offset: 1035264 length: 1168896 2024-03-19T11:30:05.635 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-03-19T11:30:05.635 INFO:tasks.rados.rados.0.smithi038.stdout:264: finishing set_chunk to oid 15 2024-03-19T11:30:05.635 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 310784 length: 1069568 2024-03-19T11:30:05.635 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-03-19T11:30:05.635 INFO:tasks.rados.rados.0.smithi038.stdout:262: finishing set_chunk to oid 13 2024-03-19T11:30:05.635 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 13 set_chunk 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 85504 length: 1895424 2024-03-19T11:30:05.635 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 13 v 0 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-03-19T11:30:05.635 INFO:tasks.rados.rados.0.smithi038.stdout:266: finishing set_chunk to oid 17 2024-03-19T11:30:05.635 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 17 set_chunk 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 296448 length: 668672 2024-03-19T11:30:05.636 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 17 v 0 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-03-19T11:30:05.636 INFO:tasks.rados.rados.0.smithi038.stdout:268: finishing set_chunk to oid 19 2024-03-19T11:30:05.636 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 19 set_chunk 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1007104 length: 858624 2024-03-19T11:30:05.636 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 19 v 0 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-03-19T11:30:05.636 INFO:tasks.rados.rados.0.smithi038.stdout:262: done (15 left) 2024-03-19T11:30:05.636 INFO:tasks.rados.rados.0.smithi038.stdout:264: done (14 left) 2024-03-19T11:30:05.636 INFO:tasks.rados.rados.0.smithi038.stdout:265: done (13 left) 2024-03-19T11:30:05.636 INFO:tasks.rados.rados.0.smithi038.stdout:266: done (12 left) 2024-03-19T11:30:05.636 INFO:tasks.rados.rados.0.smithi038.stdout:268: done (11 left) 2024-03-19T11:30:05.636 INFO:tasks.rados.rados.0.smithi038.stdout:279: set_chunk oid 30 target oid 30 low_tier 2024-03-19T11:30:05.636 INFO:tasks.rados.rados.0.smithi038.stdout:279: set_chunk oid 30 offset: 922112 length: 816128 target oid smithi03829856-30 low_tier offset: 922112 snap -1 2024-03-19T11:30:05.636 INFO:tasks.rados.rados.0.smithi038.stdout:280: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.636 INFO:tasks.rados.rados.0.smithi038.stdout:280: set_chunk oid 31 offset: 148992 length: 1935360 target oid smithi03829856-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 148992 snap -1 2024-03-19T11:30:05.636 INFO:tasks.rados.rados.0.smithi038.stdout:281: set_chunk oid 32 target oid 32 low_tier 2024-03-19T11:30:05.636 INFO:tasks.rados.rados.0.smithi038.stdout:281: set_chunk oid 32 offset: 887808 length: 178176 target oid smithi03829856-32 low_tier offset: 887808 snap -1 2024-03-19T11:30:05.637 INFO:tasks.rados.rados.0.smithi038.stdout:282: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.637 INFO:tasks.rados.rados.0.smithi038.stdout:282: set_chunk oid 33 offset: 257024 length: 2101248 target oid smithi03829856-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 257024 snap -1 2024-03-19T11:30:05.637 INFO:tasks.rados.rados.0.smithi038.stdout:283: set_chunk oid 34 target oid 34 low_tier 2024-03-19T11:30:05.637 INFO:tasks.rados.rados.0.smithi038.stdout:283: set_chunk oid 34 offset: 36864 length: 1334784 target oid smithi03829856-34 low_tier offset: 36864 snap -1 2024-03-19T11:30:05.637 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.637 INFO:tasks.rados.rados.0.smithi038.stdout:273: finishing set_chunk to oid 24 2024-03-19T11:30:05.637 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 24 set_chunk 24 low_tier returned error code -95 offset: 722944 length: 1345536 2024-03-19T11:30:05.637 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-03-19T11:30:05.637 INFO:tasks.rados.rados.0.smithi038.stdout:272: finishing set_chunk to oid 23 2024-03-19T11:30:05.637 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 23 set_chunk 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1084416 length: 1009664 2024-03-19T11:30:05.637 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 23 v 0 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-03-19T11:30:05.637 INFO:tasks.rados.rados.0.smithi038.stdout:274: finishing set_chunk to oid 25 2024-03-19T11:30:05.637 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 25 set_chunk 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 501248 length: 1596416 2024-03-19T11:30:05.637 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-03-19T11:30:05.637 INFO:tasks.rados.rados.0.smithi038.stdout:277: finishing set_chunk to oid 28 2024-03-19T11:30:05.638 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 28 set_chunk 28 low_tier returned error code -95 offset: 1204736 length: 34816 2024-03-19T11:30:05.638 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 28 v 0 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-03-19T11:30:05.638 INFO:tasks.rados.rados.0.smithi038.stdout:276: finishing set_chunk to oid 27 2024-03-19T11:30:05.638 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 27 set_chunk 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 625152 length: 299008 2024-03-19T11:30:05.638 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-03-19T11:30:05.638 INFO:tasks.rados.rados.0.smithi038.stdout:272: done (15 left) 2024-03-19T11:30:05.638 INFO:tasks.rados.rados.0.smithi038.stdout:273: done (14 left) 2024-03-19T11:30:05.638 INFO:tasks.rados.rados.0.smithi038.stdout:274: done (13 left) 2024-03-19T11:30:05.638 INFO:tasks.rados.rados.0.smithi038.stdout:276: done (12 left) 2024-03-19T11:30:05.638 INFO:tasks.rados.rados.0.smithi038.stdout:277: done (11 left) 2024-03-19T11:30:05.638 INFO:tasks.rados.rados.0.smithi038.stdout:284: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.638 INFO:tasks.rados.rados.0.smithi038.stdout:284: set_chunk oid 35 offset: 1541632 length: 345600 target oid smithi03829856-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1541632 snap -1 2024-03-19T11:30:05.638 INFO:tasks.rados.rados.0.smithi038.stdout:285: set_chunk oid 36 target oid 36 low_tier 2024-03-19T11:30:05.638 INFO:tasks.rados.rados.0.smithi038.stdout:285: set_chunk oid 36 offset: 373760 length: 1160704 target oid smithi03829856-36 low_tier offset: 373760 snap -1 2024-03-19T11:30:05.639 INFO:tasks.rados.rados.0.smithi038.stdout:286: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.639 INFO:tasks.rados.rados.0.smithi038.stdout:286: set_chunk oid 37 offset: 518144 length: 938496 target oid smithi03829856-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 518144 snap -1 2024-03-19T11:30:05.639 INFO:tasks.rados.rados.0.smithi038.stdout:287: set_chunk oid 38 target oid 38 low_tier 2024-03-19T11:30:05.639 INFO:tasks.rados.rados.0.smithi038.stdout:287: set_chunk oid 38 offset: 1545216 length: 290304 target oid smithi03829856-38 low_tier offset: 1545216 snap -1 2024-03-19T11:30:05.639 INFO:tasks.rados.rados.0.smithi038.stdout:288: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.639 INFO:tasks.rados.rados.0.smithi038.stdout:288: set_chunk oid 39 offset: 196096 length: 1280512 target oid smithi03829856-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 196096 snap -1 2024-03-19T11:30:05.639 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.639 INFO:tasks.rados.rados.0.smithi038.stdout:284: finishing set_chunk to oid 35 2024-03-19T11:30:05.639 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 35 set_chunk 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1541632 length: 345600 2024-03-19T11:30:05.639 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 35 v 0 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-03-19T11:30:05.639 INFO:tasks.rados.rados.0.smithi038.stdout:287: finishing set_chunk to oid 38 2024-03-19T11:30:05.639 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 38 set_chunk 38 low_tier returned error code -95 offset: 1545216 length: 290304 2024-03-19T11:30:05.639 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 38 v 0 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-03-19T11:30:05.639 INFO:tasks.rados.rados.0.smithi038.stdout:286: finishing set_chunk to oid 37 2024-03-19T11:30:05.639 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 37 set_chunk 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 518144 length: 938496 2024-03-19T11:30:05.640 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 37 v 0 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-03-19T11:30:05.640 INFO:tasks.rados.rados.0.smithi038.stdout:284: done (15 left) 2024-03-19T11:30:05.640 INFO:tasks.rados.rados.0.smithi038.stdout:286: done (14 left) 2024-03-19T11:30:05.640 INFO:tasks.rados.rados.0.smithi038.stdout:287: done (13 left) 2024-03-19T11:30:05.640 INFO:tasks.rados.rados.0.smithi038.stdout:289: set_chunk oid 40 target oid 40 low_tier 2024-03-19T11:30:05.640 INFO:tasks.rados.rados.0.smithi038.stdout:289: set_chunk oid 40 offset: 916480 length: 1295360 target oid smithi03829856-40 low_tier offset: 916480 snap -1 2024-03-19T11:30:05.640 INFO:tasks.rados.rados.0.smithi038.stdout:290: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.640 INFO:tasks.rados.rados.0.smithi038.stdout:290: set_chunk oid 41 offset: 31744 length: 499712 target oid smithi03829856-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 31744 snap -1 2024-03-19T11:30:05.640 INFO:tasks.rados.rados.0.smithi038.stdout:291: set_chunk oid 42 target oid 42 low_tier 2024-03-19T11:30:05.640 INFO:tasks.rados.rados.0.smithi038.stdout:291: set_chunk oid 42 offset: 1702400 length: 652288 target oid smithi03829856-42 low_tier offset: 1702400 snap -1 2024-03-19T11:30:05.640 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.640 INFO:tasks.rados.rados.0.smithi038.stdout:275: finishing set_chunk to oid 26 2024-03-19T11:30:05.640 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 26 set_chunk 26 low_tier returned error code -95 offset: 396800 length: 1566720 2024-03-19T11:30:05.640 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-03-19T11:30:05.641 INFO:tasks.rados.rados.0.smithi038.stdout:278: finishing set_chunk to oid 29 2024-03-19T11:30:05.641 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 29 set_chunk 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 189440 length: 2083840 2024-03-19T11:30:05.641 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 29 v 0 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-03-19T11:30:05.641 INFO:tasks.rados.rados.0.smithi038.stdout:279: finishing set_chunk to oid 30 2024-03-19T11:30:05.641 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 30 set_chunk 30 low_tier returned error code -95 offset: 922112 length: 816128 2024-03-19T11:30:05.641 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 30 v 0 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-03-19T11:30:05.641 INFO:tasks.rados.rados.0.smithi038.stdout:288: finishing set_chunk to oid 39 2024-03-19T11:30:05.641 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 39 set_chunk 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 196096 length: 1280512 2024-03-19T11:30:05.641 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 39 v 0 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-03-19T11:30:05.641 INFO:tasks.rados.rados.0.smithi038.stdout:280: finishing set_chunk to oid 31 2024-03-19T11:30:05.641 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 31 set_chunk 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 148992 length: 1935360 2024-03-19T11:30:05.641 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 31 v 0 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-03-19T11:30:05.641 INFO:tasks.rados.rados.0.smithi038.stdout:282: finishing set_chunk to oid 33 2024-03-19T11:30:05.641 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 33 set_chunk 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 257024 length: 2101248 2024-03-19T11:30:05.641 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-03-19T11:30:05.641 INFO:tasks.rados.rados.0.smithi038.stdout:275: done (15 left) 2024-03-19T11:30:05.642 INFO:tasks.rados.rados.0.smithi038.stdout:278: done (14 left) 2024-03-19T11:30:05.642 INFO:tasks.rados.rados.0.smithi038.stdout:279: done (13 left) 2024-03-19T11:30:05.642 INFO:tasks.rados.rados.0.smithi038.stdout:280: done (12 left) 2024-03-19T11:30:05.642 INFO:tasks.rados.rados.0.smithi038.stdout:282: done (11 left) 2024-03-19T11:30:05.642 INFO:tasks.rados.rados.0.smithi038.stdout:288: done (10 left) 2024-03-19T11:30:05.642 INFO:tasks.rados.rados.0.smithi038.stdout:292: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.642 INFO:tasks.rados.rados.0.smithi038.stdout:292: set_chunk oid 43 offset: 377856 length: 1418752 target oid smithi03829856-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 377856 snap -1 2024-03-19T11:30:05.642 INFO:tasks.rados.rados.0.smithi038.stdout:293: set_chunk oid 44 target oid 44 low_tier 2024-03-19T11:30:05.642 INFO:tasks.rados.rados.0.smithi038.stdout:293: set_chunk oid 44 offset: 529920 length: 2157056 target oid smithi03829856-44 low_tier offset: 529920 snap -1 2024-03-19T11:30:05.642 INFO:tasks.rados.rados.0.smithi038.stdout:294: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.642 INFO:tasks.rados.rados.0.smithi038.stdout:294: set_chunk oid 45 offset: 514560 length: 1787392 target oid smithi03829856-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 514560 snap -1 2024-03-19T11:30:05.643 INFO:tasks.rados.rados.0.smithi038.stdout:295: set_chunk oid 46 target oid 46 low_tier 2024-03-19T11:30:05.643 INFO:tasks.rados.rados.0.smithi038.stdout:295: set_chunk oid 46 offset: 315904 length: 1901056 target oid smithi03829856-46 low_tier offset: 315904 snap -1 2024-03-19T11:30:05.643 INFO:tasks.rados.rados.0.smithi038.stdout:296: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.643 INFO:tasks.rados.rados.0.smithi038.stdout:296: set_chunk oid 47 offset: 1105408 length: 1230848 target oid smithi03829856-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1105408 snap -1 2024-03-19T11:30:05.643 INFO:tasks.rados.rados.0.smithi038.stdout:297: set_chunk oid 48 target oid 48 low_tier 2024-03-19T11:30:05.643 INFO:tasks.rados.rados.0.smithi038.stdout:297: set_chunk oid 48 offset: 893952 length: 1033216 target oid smithi03829856-48 low_tier offset: 893952 snap -1 2024-03-19T11:30:05.643 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.643 INFO:tasks.rados.rados.0.smithi038.stdout:289: finishing set_chunk to oid 40 2024-03-19T11:30:05.643 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 40 set_chunk 40 low_tier returned error code -95 offset: 916480 length: 1295360 2024-03-19T11:30:05.643 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 40 v 0 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-03-19T11:30:05.643 INFO:tasks.rados.rados.0.smithi038.stdout:289: done (15 left) 2024-03-19T11:30:05.643 INFO:tasks.rados.rados.0.smithi038.stdout:298: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.643 INFO:tasks.rados.rados.0.smithi038.stdout:298: set_chunk oid 49 offset: 1542144 length: 512 target oid smithi03829856-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1542144 snap -1 2024-03-19T11:30:05.643 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.644 INFO:tasks.rados.rados.0.smithi038.stdout:293: finishing set_chunk to oid 44 2024-03-19T11:30:05.644 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 44 set_chunk 44 low_tier returned error code -95 offset: 529920 length: 2157056 2024-03-19T11:30:05.644 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 44 v 0 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-03-19T11:30:05.644 INFO:tasks.rados.rados.0.smithi038.stdout:293: done (15 left) 2024-03-19T11:30:05.644 INFO:tasks.rados.rados.0.smithi038.stdout:299: set_chunk oid 50 target oid 50 low_tier 2024-03-19T11:30:05.644 INFO:tasks.rados.rados.0.smithi038.stdout:299: set_chunk oid 50 offset: 770560 length: 1370112 target oid smithi03829856-50 low_tier offset: 770560 snap -1 2024-03-19T11:30:05.644 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.644 INFO:tasks.rados.rados.0.smithi038.stdout:297: finishing set_chunk to oid 48 2024-03-19T11:30:05.644 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 48 set_chunk 48 low_tier returned error code -95 offset: 893952 length: 1033216 2024-03-19T11:30:05.644 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 48 v 0 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-03-19T11:30:05.644 INFO:tasks.rados.rados.0.smithi038.stdout:285: finishing set_chunk to oid 36 2024-03-19T11:30:05.644 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 36 set_chunk 36 low_tier returned error code -95 offset: 373760 length: 1160704 2024-03-19T11:30:05.644 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-03-19T11:30:05.644 INFO:tasks.rados.rados.0.smithi038.stdout:285: done (15 left) 2024-03-19T11:30:05.645 INFO:tasks.rados.rados.0.smithi038.stdout:297: done (14 left) 2024-03-19T11:30:05.645 INFO:tasks.rados.rados.0.smithi038.stdout:300: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.645 INFO:tasks.rados.rados.0.smithi038.stdout:300: set_chunk oid 1 offset: 574976 length: 1050624 target oid smithi03829856-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 574976 snap -1 2024-03-19T11:30:05.645 INFO:tasks.rados.rados.0.smithi038.stdout:301: set_chunk oid 2 target oid 2 low_tier 2024-03-19T11:30:05.645 INFO:tasks.rados.rados.0.smithi038.stdout:301: set_chunk oid 2 offset: 41984 length: 749056 target oid smithi03829856-2 low_tier offset: 41984 snap -1 2024-03-19T11:30:05.645 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.645 INFO:tasks.rados.rados.0.smithi038.stdout:295: finishing set_chunk to oid 46 2024-03-19T11:30:05.645 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 46 set_chunk 46 low_tier returned error code -95 offset: 315904 length: 1901056 2024-03-19T11:30:05.645 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-03-19T11:30:05.645 INFO:tasks.rados.rados.0.smithi038.stdout:296: finishing set_chunk to oid 47 2024-03-19T11:30:05.645 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 47 set_chunk 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1105408 length: 1230848 2024-03-19T11:30:05.645 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-03-19T11:30:05.645 INFO:tasks.rados.rados.0.smithi038.stdout:292: finishing set_chunk to oid 43 2024-03-19T11:30:05.645 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 43 set_chunk 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 377856 length: 1418752 2024-03-19T11:30:05.645 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 43 v 0 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-03-19T11:30:05.646 INFO:tasks.rados.rados.0.smithi038.stdout:292: done (15 left) 2024-03-19T11:30:05.646 INFO:tasks.rados.rados.0.smithi038.stdout:295: done (14 left) 2024-03-19T11:30:05.646 INFO:tasks.rados.rados.0.smithi038.stdout:296: done (13 left) 2024-03-19T11:30:05.646 INFO:tasks.rados.rados.0.smithi038.stdout:301 retry set_chunk ! 2024-03-19T11:30:05.646 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 13 2024-03-19T11:30:05.646 INFO:tasks.rados.rados.0.smithi038.stdout:298: finishing set_chunk to oid 49 2024-03-19T11:30:05.646 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 49 set_chunk 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1542144 length: 512 2024-03-19T11:30:05.646 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-03-19T11:30:05.646 INFO:tasks.rados.rados.0.smithi038.stdout:298: done (12 left) 2024-03-19T11:30:05.646 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 12 2024-03-19T11:30:05.646 INFO:tasks.rados.rados.0.smithi038.stdout:294: finishing set_chunk to oid 45 2024-03-19T11:30:05.646 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 45 set_chunk 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 514560 length: 1787392 2024-03-19T11:30:05.646 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 45 v 0 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-03-19T11:30:05.646 INFO:tasks.rados.rados.0.smithi038.stdout:294: done (11 left) 2024-03-19T11:30:05.646 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 11 2024-03-19T11:30:05.647 INFO:tasks.rados.rados.0.smithi038.stdout:291: finishing set_chunk to oid 42 2024-03-19T11:30:05.647 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 42 set_chunk 42 low_tier returned error code -95 offset: 1702400 length: 652288 2024-03-19T11:30:05.647 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-03-19T11:30:05.647 INFO:tasks.rados.rados.0.smithi038.stdout:301: finishing set_chunk to oid 2 2024-03-19T11:30:05.647 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 2 set_chunk 2 low_tier returned error code -95 offset: 41984 length: 749056 2024-03-19T11:30:05.647 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-03-19T11:30:05.647 INFO:tasks.rados.rados.0.smithi038.stdout:291: done (10 left) 2024-03-19T11:30:05.647 INFO:tasks.rados.rados.0.smithi038.stdout:301: done (9 left) 2024-03-19T11:30:05.647 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 9 2024-03-19T11:30:05.647 INFO:tasks.rados.rados.0.smithi038.stdout:300: finishing set_chunk to oid 1 2024-03-19T11:30:05.647 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 1 set_chunk 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 574976 length: 1050624 2024-03-19T11:30:05.647 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 1 v 0 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-03-19T11:30:05.647 INFO:tasks.rados.rados.0.smithi038.stdout:299: finishing set_chunk to oid 50 2024-03-19T11:30:05.647 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 50 set_chunk 50 low_tier returned error code -95 offset: 770560 length: 1370112 2024-03-19T11:30:05.647 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 50 v 0 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-03-19T11:30:05.648 INFO:tasks.rados.rados.0.smithi038.stdout:299: done (8 left) 2024-03-19T11:30:05.648 INFO:tasks.rados.rados.0.smithi038.stdout:300: done (7 left) 2024-03-19T11:30:05.648 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 7 2024-03-19T11:30:05.648 INFO:tasks.rados.rados.0.smithi038.stdout:252: finishing set_chunk to oid 3 2024-03-19T11:30:05.648 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 3 v 3 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-03-19T11:30:05.648 INFO:tasks.rados.rados.0.smithi038.stdout:252: done (6 left) 2024-03-19T11:30:05.648 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 6 2024-03-19T11:30:05.648 INFO:tasks.rados.rados.0.smithi038.stdout:270: finishing set_chunk to oid 21 2024-03-19T11:30:05.648 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 21 v 15 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-03-19T11:30:05.648 INFO:tasks.rados.rados.0.smithi038.stdout:270: done (5 left) 2024-03-19T11:30:05.648 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 5 2024-03-19T11:30:05.648 INFO:tasks.rados.rados.0.smithi038.stdout:283: finishing set_chunk to oid 34 2024-03-19T11:30:05.648 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 34 v 3 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-03-19T11:30:05.648 INFO:tasks.rados.rados.0.smithi038.stdout:283: done (4 left) 2024-03-19T11:30:05.649 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 4 2024-03-19T11:30:05.649 INFO:tasks.rados.rados.0.smithi038.stdout:269: finishing set_chunk to oid 20 2024-03-19T11:30:05.649 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 20 v 12 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-03-19T11:30:05.649 INFO:tasks.rados.rados.0.smithi038.stdout:269: done (3 left) 2024-03-19T11:30:05.649 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 3 2024-03-19T11:30:05.649 INFO:tasks.rados.rados.0.smithi038.stdout:271: finishing set_chunk to oid 22 2024-03-19T11:30:05.649 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 22 v 3 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-03-19T11:30:05.649 INFO:tasks.rados.rados.0.smithi038.stdout:271: done (2 left) 2024-03-19T11:30:05.649 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 2 2024-03-19T11:30:05.649 INFO:tasks.rados.rados.0.smithi038.stdout:290: finishing set_chunk to oid 41 2024-03-19T11:30:05.649 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 41 v 9 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-03-19T11:30:05.649 INFO:tasks.rados.rados.0.smithi038.stdout:290: done (1 left) 2024-03-19T11:30:05.649 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 1 2024-03-19T11:30:05.649 INFO:tasks.rados.rados.0.smithi038.stdout:281: finishing set_chunk to oid 32 2024-03-19T11:30:05.650 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 32 v 6 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-03-19T11:30:05.650 INFO:tasks.rados.rados.0.smithi038.stdout:281: done (0 left) 2024-03-19T11:30:05.650 INFO:tasks.rados.rados.0.smithi038.stdout:302: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.650 INFO:tasks.rados.rados.0.smithi038.stdout:302: set_chunk oid 3 offset: 694784 length: 851456 target oid smithi03829856-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 694784 snap -1 2024-03-19T11:30:05.650 INFO:tasks.rados.rados.0.smithi038.stdout:303: set_chunk oid 4 target oid 4 low_tier 2024-03-19T11:30:05.650 INFO:tasks.rados.rados.0.smithi038.stdout:303: set_chunk oid 4 offset: 149504 length: 494080 target oid smithi03829856-4 low_tier offset: 149504 snap -1 2024-03-19T11:30:05.650 INFO:tasks.rados.rados.0.smithi038.stdout:304: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.650 INFO:tasks.rados.rados.0.smithi038.stdout:304: set_chunk oid 5 offset: 511488 length: 593920 target oid smithi03829856-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 511488 snap -1 2024-03-19T11:30:05.650 INFO:tasks.rados.rados.0.smithi038.stdout:305: set_chunk oid 6 target oid 6 low_tier 2024-03-19T11:30:05.650 INFO:tasks.rados.rados.0.smithi038.stdout:305: set_chunk oid 6 offset: 47616 length: 1697792 target oid smithi03829856-6 low_tier offset: 47616 snap -1 2024-03-19T11:30:05.650 INFO:tasks.rados.rados.0.smithi038.stdout:306: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.650 INFO:tasks.rados.rados.0.smithi038.stdout:306: set_chunk oid 7 offset: 996864 length: 693760 target oid smithi03829856-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 996864 snap -1 2024-03-19T11:30:05.651 INFO:tasks.rados.rados.0.smithi038.stdout:307: set_chunk oid 8 target oid 8 low_tier 2024-03-19T11:30:05.651 INFO:tasks.rados.rados.0.smithi038.stdout:307: set_chunk oid 8 offset: 646656 length: 816128 target oid smithi03829856-8 low_tier offset: 646656 snap -1 2024-03-19T11:30:05.651 INFO:tasks.rados.rados.0.smithi038.stdout:308: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.651 INFO:tasks.rados.rados.0.smithi038.stdout:308: set_chunk oid 9 offset: 1081856 length: 14336 target oid smithi03829856-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1081856 snap -1 2024-03-19T11:30:05.651 INFO:tasks.rados.rados.0.smithi038.stdout:309: set_chunk oid 10 target oid 10 low_tier 2024-03-19T11:30:05.651 INFO:tasks.rados.rados.0.smithi038.stdout:309: set_chunk oid 10 offset: 626688 length: 1123840 target oid smithi03829856-10 low_tier offset: 626688 snap -1 2024-03-19T11:30:05.651 INFO:tasks.rados.rados.0.smithi038.stdout:310: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.651 INFO:tasks.rados.rados.0.smithi038.stdout:310: set_chunk oid 11 offset: 406528 length: 1094656 target oid smithi03829856-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 406528 snap -1 2024-03-19T11:30:05.651 INFO:tasks.rados.rados.0.smithi038.stdout:311: set_chunk oid 12 target oid 12 low_tier 2024-03-19T11:30:05.651 INFO:tasks.rados.rados.0.smithi038.stdout:311: set_chunk oid 12 offset: 755200 length: 282624 target oid smithi03829856-12 low_tier offset: 755200 snap -1 2024-03-19T11:30:05.651 INFO:tasks.rados.rados.0.smithi038.stdout:312: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.651 INFO:tasks.rados.rados.0.smithi038.stdout:312: set_chunk oid 13 offset: 695808 length: 1494528 target oid smithi03829856-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 695808 snap -1 2024-03-19T11:30:05.651 INFO:tasks.rados.rados.0.smithi038.stdout:313: set_chunk oid 14 target oid 14 low_tier 2024-03-19T11:30:05.651 INFO:tasks.rados.rados.0.smithi038.stdout:313: set_chunk oid 14 offset: 1744384 length: 124928 target oid smithi03829856-14 low_tier offset: 1744384 snap -1 2024-03-19T11:30:05.651 INFO:tasks.rados.rados.0.smithi038.stdout:314: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.652 INFO:tasks.rados.rados.0.smithi038.stdout:314: set_chunk oid 15 offset: 379392 length: 1563136 target oid smithi03829856-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 379392 snap -1 2024-03-19T11:30:05.652 INFO:tasks.rados.rados.0.smithi038.stdout:315: set_chunk oid 16 target oid 16 low_tier 2024-03-19T11:30:05.652 INFO:tasks.rados.rados.0.smithi038.stdout:315: set_chunk oid 16 offset: 393728 length: 1673728 target oid smithi03829856-16 low_tier offset: 393728 snap -1 2024-03-19T11:30:05.652 INFO:tasks.rados.rados.0.smithi038.stdout:316: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.652 INFO:tasks.rados.rados.0.smithi038.stdout:316: set_chunk oid 17 offset: 542208 length: 756224 target oid smithi03829856-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 542208 snap -1 2024-03-19T11:30:05.652 INFO:tasks.rados.rados.0.smithi038.stdout:317: set_chunk oid 18 target oid 18 low_tier 2024-03-19T11:30:05.652 INFO:tasks.rados.rados.0.smithi038.stdout:317: set_chunk oid 18 offset: 1625088 length: 215040 target oid smithi03829856-18 low_tier offset: 1625088 snap -1 2024-03-19T11:30:05.652 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.652 INFO:tasks.rados.rados.0.smithi038.stdout:303: finishing set_chunk to oid 4 2024-03-19T11:30:05.652 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 4 set_chunk 4 low_tier returned error code -95 offset: 149504 length: 494080 2024-03-19T11:30:05.652 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-03-19T11:30:05.652 INFO:tasks.rados.rados.0.smithi038.stdout:307: finishing set_chunk to oid 8 2024-03-19T11:30:05.652 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 8 set_chunk 8 low_tier returned error code -95 offset: 646656 length: 816128 2024-03-19T11:30:05.652 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-03-19T11:30:05.653 INFO:tasks.rados.rados.0.smithi038.stdout:302: finishing set_chunk to oid 3 2024-03-19T11:30:05.653 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 3 set_chunk 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 694784 length: 851456 2024-03-19T11:30:05.653 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 3 v 0 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-03-19T11:30:05.653 INFO:tasks.rados.rados.0.smithi038.stdout:304: finishing set_chunk to oid 5 2024-03-19T11:30:05.653 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 5 set_chunk 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 511488 length: 593920 2024-03-19T11:30:05.653 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-03-19T11:30:05.653 INFO:tasks.rados.rados.0.smithi038.stdout:309: finishing set_chunk to oid 10 2024-03-19T11:30:05.653 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 10 set_chunk 10 low_tier returned error code -95 offset: 626688 length: 1123840 2024-03-19T11:30:05.653 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 10 v 0 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-03-19T11:30:05.653 INFO:tasks.rados.rados.0.smithi038.stdout:305: finishing set_chunk to oid 6 2024-03-19T11:30:05.653 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 6 set_chunk 6 low_tier returned error code -95 offset: 47616 length: 1697792 2024-03-19T11:30:05.653 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-03-19T11:30:05.653 INFO:tasks.rados.rados.0.smithi038.stdout:306: finishing set_chunk to oid 7 2024-03-19T11:30:05.653 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 7 set_chunk 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 996864 length: 693760 2024-03-19T11:30:05.653 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-03-19T11:30:05.654 INFO:tasks.rados.rados.0.smithi038.stdout:302: done (15 left) 2024-03-19T11:30:05.654 INFO:tasks.rados.rados.0.smithi038.stdout:303: done (14 left) 2024-03-19T11:30:05.654 INFO:tasks.rados.rados.0.smithi038.stdout:304: done (13 left) 2024-03-19T11:30:05.654 INFO:tasks.rados.rados.0.smithi038.stdout:305: done (12 left) 2024-03-19T11:30:05.654 INFO:tasks.rados.rados.0.smithi038.stdout:306: done (11 left) 2024-03-19T11:30:05.654 INFO:tasks.rados.rados.0.smithi038.stdout:307: done (10 left) 2024-03-19T11:30:05.654 INFO:tasks.rados.rados.0.smithi038.stdout:309: done (9 left) 2024-03-19T11:30:05.654 INFO:tasks.rados.rados.0.smithi038.stdout:318: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.654 INFO:tasks.rados.rados.0.smithi038.stdout:318: set_chunk oid 19 offset: 755712 length: 983552 target oid smithi03829856-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 755712 snap -1 2024-03-19T11:30:05.654 INFO:tasks.rados.rados.0.smithi038.stdout:319: set_chunk oid 20 target oid 20 low_tier 2024-03-19T11:30:05.654 INFO:tasks.rados.rados.0.smithi038.stdout:308: finishing set_chunk to oid 9 2024-03-19T11:30:05.654 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 9 set_chunk 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1081856 length: 14336 2024-03-19T11:30:05.654 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 9 v 0 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-03-19T11:30:05.654 INFO:tasks.rados.rados.0.smithi038.stdout:319: set_chunk oid 20 offset: 706048 length: 222720 target oid smithi03829856-20 low_tier offset: 706048 snap -1 2024-03-19T11:30:05.655 INFO:tasks.rados.rados.0.smithi038.stdout:308: done (10 left) 2024-03-19T11:30:05.655 INFO:tasks.rados.rados.0.smithi038.stdout:320: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.655 INFO:tasks.rados.rados.0.smithi038.stdout:320: set_chunk oid 21 offset: 268800 length: 1433600 target oid smithi03829856-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 268800 snap -1 2024-03-19T11:30:05.655 INFO:tasks.rados.rados.0.smithi038.stdout:321: set_chunk oid 22 target oid 22 low_tier 2024-03-19T11:30:05.655 INFO:tasks.rados.rados.0.smithi038.stdout:321: set_chunk oid 22 offset: 270336 length: 48128 target oid smithi03829856-22 low_tier offset: 270336 snap -1 2024-03-19T11:30:05.655 INFO:tasks.rados.rados.0.smithi038.stdout:322: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.655 INFO:tasks.rados.rados.0.smithi038.stdout:312: finishing set_chunk to oid 13 2024-03-19T11:30:05.655 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 13 set_chunk 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 695808 length: 1494528 2024-03-19T11:30:05.655 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 13 v 0 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-03-19T11:30:05.655 INFO:tasks.rados.rados.0.smithi038.stdout:310: finishing set_chunk to oid 11 2024-03-19T11:30:05.655 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 11 set_chunk 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 406528 length: 1094656 2024-03-19T11:30:05.655 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 11 v 0 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-03-19T11:30:05.655 INFO:tasks.rados.rados.0.smithi038.stdout:314: finishing set_chunk to oid 15 2024-03-19T11:30:05.655 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 379392 length: 1563136 2024-03-19T11:30:05.655 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-03-19T11:30:05.656 INFO:tasks.rados.rados.0.smithi038.stdout:316: finishing set_chunk to oid 17 2024-03-19T11:30:05.656 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 17 set_chunk 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 542208 length: 756224 2024-03-19T11:30:05.656 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 17 v 0 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-03-19T11:30:05.656 INFO:tasks.rados.rados.0.smithi038.stdout:315: finishing set_chunk to oid 16 2024-03-19T11:30:05.656 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 16 set_chunk 16 low_tier returned error code -95 offset: 393728 length: 1673728 2024-03-19T11:30:05.656 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-03-19T11:30:05.656 INFO:tasks.rados.rados.0.smithi038.stdout:317: finishing set_chunk to oid 18 2024-03-19T11:30:05.656 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 18 set_chunk 18 low_tier returned error code -95 offset: 1625088 length: 215040 2024-03-19T11:30:05.656 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 18 v 0 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-03-19T11:30:05.656 INFO:tasks.rados.rados.0.smithi038.stdout:322: set_chunk oid 23 offset: 503296 length: 371200 target oid smithi03829856-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 503296 snap -1 2024-03-19T11:30:05.656 INFO:tasks.rados.rados.0.smithi038.stdout:310: done (12 left) 2024-03-19T11:30:05.656 INFO:tasks.rados.rados.0.smithi038.stdout:312: done (11 left) 2024-03-19T11:30:05.656 INFO:tasks.rados.rados.0.smithi038.stdout:314: done (10 left) 2024-03-19T11:30:05.656 INFO:tasks.rados.rados.0.smithi038.stdout:315: done (9 left) 2024-03-19T11:30:05.656 INFO:tasks.rados.rados.0.smithi038.stdout:316: done (8 left) 2024-03-19T11:30:05.657 INFO:tasks.rados.rados.0.smithi038.stdout:317: done (7 left) 2024-03-19T11:30:05.657 INFO:tasks.rados.rados.0.smithi038.stdout:323: set_chunk oid 24 target oid 24 low_tier 2024-03-19T11:30:05.657 INFO:tasks.rados.rados.0.smithi038.stdout:323: set_chunk oid 24 offset: 525312 length: 884224 target oid smithi03829856-24 low_tier offset: 525312 snap -1 2024-03-19T11:30:05.657 INFO:tasks.rados.rados.0.smithi038.stdout:324: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.657 INFO:tasks.rados.rados.0.smithi038.stdout:324: set_chunk oid 25 offset: 660480 length: 1635328 target oid smithi03829856-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 660480 snap -1 2024-03-19T11:30:05.657 INFO:tasks.rados.rados.0.smithi038.stdout:325: set_chunk oid 26 target oid 26 low_tier 2024-03-19T11:30:05.657 INFO:tasks.rados.rados.0.smithi038.stdout:325: set_chunk oid 26 offset: 901120 length: 664576 target oid smithi03829856-26 low_tier offset: 901120 snap -1 2024-03-19T11:30:05.657 INFO:tasks.rados.rados.0.smithi038.stdout:326: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.657 INFO:tasks.rados.rados.0.smithi038.stdout:318: finishing set_chunk to oid 19 2024-03-19T11:30:05.657 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 19 set_chunk 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 755712 length: 983552 2024-03-19T11:30:05.657 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 19 v 0 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-03-19T11:30:05.657 INFO:tasks.rados.rados.0.smithi038.stdout:321: finishing set_chunk to oid 22 2024-03-19T11:30:05.657 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 22 set_chunk 22 low_tier returned error code -95 offset: 270336 length: 48128 2024-03-19T11:30:05.657 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-03-19T11:30:05.658 INFO:tasks.rados.rados.0.smithi038.stdout:320: finishing set_chunk to oid 21 2024-03-19T11:30:05.658 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 21 set_chunk 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 268800 length: 1433600 2024-03-19T11:30:05.658 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-03-19T11:30:05.658 INFO:tasks.rados.rados.0.smithi038.stdout:326: set_chunk oid 27 offset: 30208 length: 1783808 target oid smithi03829856-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 30208 snap -1 2024-03-19T11:30:05.658 INFO:tasks.rados.rados.0.smithi038.stdout:318: done (10 left) 2024-03-19T11:30:05.658 INFO:tasks.rados.rados.0.smithi038.stdout:320: done (9 left) 2024-03-19T11:30:05.658 INFO:tasks.rados.rados.0.smithi038.stdout:321: done (8 left) 2024-03-19T11:30:05.658 INFO:tasks.rados.rados.0.smithi038.stdout:327: set_chunk oid 28 target oid 28 low_tier 2024-03-19T11:30:05.658 INFO:tasks.rados.rados.0.smithi038.stdout:327: set_chunk oid 28 offset: 628224 length: 1310720 target oid smithi03829856-28 low_tier offset: 628224 snap -1 2024-03-19T11:30:05.658 INFO:tasks.rados.rados.0.smithi038.stdout:328: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.658 INFO:tasks.rados.rados.0.smithi038.stdout:328: set_chunk oid 29 offset: 1689600 length: 150528 target oid smithi03829856-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1689600 snap -1 2024-03-19T11:30:05.658 INFO:tasks.rados.rados.0.smithi038.stdout:329: set_chunk oid 30 target oid 30 low_tier 2024-03-19T11:30:05.658 INFO:tasks.rados.rados.0.smithi038.stdout:329: set_chunk oid 30 offset: 420352 length: 1935360 target oid smithi03829856-30 low_tier offset: 420352 snap -1 2024-03-19T11:30:05.658 INFO:tasks.rados.rados.0.smithi038.stdout:330: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.659 INFO:tasks.rados.rados.0.smithi038.stdout:330: set_chunk oid 31 offset: 358400 length: 1653248 target oid smithi03829856-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 358400 snap -1 2024-03-19T11:30:05.659 INFO:tasks.rados.rados.0.smithi038.stdout:331: set_chunk oid 32 target oid 32 low_tier 2024-03-19T11:30:05.659 INFO:tasks.rados.rados.0.smithi038.stdout:331: set_chunk oid 32 offset: 1675264 length: 558080 target oid smithi03829856-32 low_tier offset: 1675264 snap -1 2024-03-19T11:30:05.659 INFO:tasks.rados.rados.0.smithi038.stdout:332: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.659 INFO:tasks.rados.rados.0.smithi038.stdout:332: set_chunk oid 33 offset: 1190400 length: 835584 target oid smithi03829856-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1190400 snap -1 2024-03-19T11:30:05.659 INFO:tasks.rados.rados.0.smithi038.stdout:333: set_chunk oid 34 target oid 34 low_tier 2024-03-19T11:30:05.659 INFO:tasks.rados.rados.0.smithi038.stdout:333: set_chunk oid 34 offset: 415232 length: 588800 target oid smithi03829856-34 low_tier offset: 415232 snap -1 2024-03-19T11:30:05.659 INFO:tasks.rados.rados.0.smithi038.stdout:334: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.659 INFO:tasks.rados.rados.0.smithi038.stdout:334: set_chunk oid 35 offset: 1892864 length: 515072 target oid smithi03829856-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1892864 snap -1 2024-03-19T11:30:05.659 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.659 INFO:tasks.rados.rados.0.smithi038.stdout:324: finishing set_chunk to oid 25 2024-03-19T11:30:05.659 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 25 set_chunk 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 660480 length: 1635328 2024-03-19T11:30:05.659 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-03-19T11:30:05.659 INFO:tasks.rados.rados.0.smithi038.stdout:323: finishing set_chunk to oid 24 2024-03-19T11:30:05.660 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 24 set_chunk 24 low_tier returned error code -95 offset: 525312 length: 884224 2024-03-19T11:30:05.660 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-03-19T11:30:05.660 INFO:tasks.rados.rados.0.smithi038.stdout:323: done (15 left) 2024-03-19T11:30:05.660 INFO:tasks.rados.rados.0.smithi038.stdout:324: done (14 left) 2024-03-19T11:30:05.660 INFO:tasks.rados.rados.0.smithi038.stdout:335: set_chunk oid 36 target oid 36 low_tier 2024-03-19T11:30:05.660 INFO:tasks.rados.rados.0.smithi038.stdout:335: set_chunk oid 36 offset: 1249792 length: 1193472 target oid smithi03829856-36 low_tier offset: 1249792 snap -1 2024-03-19T11:30:05.660 INFO:tasks.rados.rados.0.smithi038.stdout:336: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.660 INFO:tasks.rados.rados.0.smithi038.stdout:336: set_chunk oid 37 offset: 379392 length: 693760 target oid smithi03829856-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 379392 snap -1 2024-03-19T11:30:05.660 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.660 INFO:tasks.rados.rados.0.smithi038.stdout:325: finishing set_chunk to oid 26 2024-03-19T11:30:05.660 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 26 set_chunk 26 low_tier returned error code -95 offset: 901120 length: 664576 2024-03-19T11:30:05.660 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-03-19T11:30:05.660 INFO:tasks.rados.rados.0.smithi038.stdout:326: finishing set_chunk to oid 27 2024-03-19T11:30:05.660 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 27 set_chunk 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 30208 length: 1783808 2024-03-19T11:30:05.660 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-03-19T11:30:05.661 INFO:tasks.rados.rados.0.smithi038.stdout:325: done (15 left) 2024-03-19T11:30:05.661 INFO:tasks.rados.rados.0.smithi038.stdout:326: done (14 left) 2024-03-19T11:30:05.661 INFO:tasks.rados.rados.0.smithi038.stdout:337: set_chunk oid 38 target oid 38 low_tier 2024-03-19T11:30:05.661 INFO:tasks.rados.rados.0.smithi038.stdout:337: set_chunk oid 38 offset: 336384 length: 173568 target oid smithi03829856-38 low_tier offset: 336384 snap -1 2024-03-19T11:30:05.661 INFO:tasks.rados.rados.0.smithi038.stdout:338: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.661 INFO:tasks.rados.rados.0.smithi038.stdout:338: set_chunk oid 39 offset: 86016 length: 391680 target oid smithi03829856-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 86016 snap -1 2024-03-19T11:30:05.661 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.661 INFO:tasks.rados.rados.0.smithi038.stdout:334: finishing set_chunk to oid 35 2024-03-19T11:30:05.661 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 35 set_chunk 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1892864 length: 515072 2024-03-19T11:30:05.661 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 35 v 0 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-03-19T11:30:05.661 INFO:tasks.rados.rados.0.smithi038.stdout:331: finishing set_chunk to oid 32 2024-03-19T11:30:05.661 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 32 set_chunk 32 low_tier returned error code -95 offset: 1675264 length: 558080 2024-03-19T11:30:05.661 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-03-19T11:30:05.661 INFO:tasks.rados.rados.0.smithi038.stdout:331: done (15 left) 2024-03-19T11:30:05.661 INFO:tasks.rados.rados.0.smithi038.stdout:334: done (14 left) 2024-03-19T11:30:05.662 INFO:tasks.rados.rados.0.smithi038.stdout:339: set_chunk oid 40 target oid 40 low_tier 2024-03-19T11:30:05.662 INFO:tasks.rados.rados.0.smithi038.stdout:339: set_chunk oid 40 offset: 842752 length: 1175040 target oid smithi03829856-40 low_tier offset: 842752 snap -1 2024-03-19T11:30:05.662 INFO:tasks.rados.rados.0.smithi038.stdout:340: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.662 INFO:tasks.rados.rados.0.smithi038.stdout:340: set_chunk oid 41 offset: 1194496 length: 1052672 target oid smithi03829856-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1194496 snap -1 2024-03-19T11:30:05.662 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.662 INFO:tasks.rados.rados.0.smithi038.stdout:327: finishing set_chunk to oid 28 2024-03-19T11:30:05.662 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 28 set_chunk 28 low_tier returned error code -95 offset: 628224 length: 1310720 2024-03-19T11:30:05.662 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 28 v 0 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-03-19T11:30:05.662 INFO:tasks.rados.rados.0.smithi038.stdout:336: finishing set_chunk to oid 37 2024-03-19T11:30:05.662 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 37 set_chunk 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 379392 length: 693760 2024-03-19T11:30:05.662 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 37 v 0 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-03-19T11:30:05.662 INFO:tasks.rados.rados.0.smithi038.stdout:328: finishing set_chunk to oid 29 2024-03-19T11:30:05.662 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 29 set_chunk 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1689600 length: 150528 2024-03-19T11:30:05.662 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 29 v 0 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-03-19T11:30:05.663 INFO:tasks.rados.rados.0.smithi038.stdout:327: done (15 left) 2024-03-19T11:30:05.663 INFO:tasks.rados.rados.0.smithi038.stdout:328: done (14 left) 2024-03-19T11:30:05.663 INFO:tasks.rados.rados.0.smithi038.stdout:336: done (13 left) 2024-03-19T11:30:05.663 INFO:tasks.rados.rados.0.smithi038.stdout:341: set_chunk oid 42 target oid 42 low_tier 2024-03-19T11:30:05.663 INFO:tasks.rados.rados.0.smithi038.stdout:341: set_chunk oid 42 offset: 1507328 length: 964608 target oid smithi03829856-42 low_tier offset: 1507328 snap -1 2024-03-19T11:30:05.663 INFO:tasks.rados.rados.0.smithi038.stdout:342: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.663 INFO:tasks.rados.rados.0.smithi038.stdout:342: set_chunk oid 43 offset: 811520 length: 1878016 target oid smithi03829856-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 811520 snap -1 2024-03-19T11:30:05.663 INFO:tasks.rados.rados.0.smithi038.stdout:343: set_chunk oid 44 target oid 44 low_tier 2024-03-19T11:30:05.663 INFO:tasks.rados.rados.0.smithi038.stdout:343: set_chunk oid 44 offset: 2405376 length: 177152 target oid smithi03829856-44 low_tier offset: 2405376 snap -1 2024-03-19T11:30:05.663 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.663 INFO:tasks.rados.rados.0.smithi038.stdout:333: finishing set_chunk to oid 34 2024-03-19T11:30:05.663 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 34 set_chunk 34 low_tier returned error code -95 offset: 415232 length: 588800 2024-03-19T11:30:05.663 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 34 v 0 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-03-19T11:30:05.663 INFO:tasks.rados.rados.0.smithi038.stdout:329: finishing set_chunk to oid 30 2024-03-19T11:30:05.663 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 30 set_chunk 30 low_tier returned error code -95 offset: 420352 length: 1935360 2024-03-19T11:30:05.663 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 30 v 0 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-03-19T11:30:05.664 INFO:tasks.rados.rados.0.smithi038.stdout:330: finishing set_chunk to oid 31 2024-03-19T11:30:05.664 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 31 set_chunk 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 358400 length: 1653248 2024-03-19T11:30:05.664 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 31 v 0 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-03-19T11:30:05.664 INFO:tasks.rados.rados.0.smithi038.stdout:329: done (15 left) 2024-03-19T11:30:05.664 INFO:tasks.rados.rados.0.smithi038.stdout:330: done (14 left) 2024-03-19T11:30:05.664 INFO:tasks.rados.rados.0.smithi038.stdout:333: done (13 left) 2024-03-19T11:30:05.664 INFO:tasks.rados.rados.0.smithi038.stdout:344: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.664 INFO:tasks.rados.rados.0.smithi038.stdout:344: set_chunk oid 45 offset: 1108480 length: 366592 target oid smithi03829856-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1108480 snap -1 2024-03-19T11:30:05.664 INFO:tasks.rados.rados.0.smithi038.stdout:345: set_chunk oid 46 target oid 46 low_tier 2024-03-19T11:30:05.664 INFO:tasks.rados.rados.0.smithi038.stdout:345: set_chunk oid 46 offset: 81920 length: 2663936 target oid smithi03829856-46 low_tier offset: 81920 snap -1 2024-03-19T11:30:05.664 INFO:tasks.rados.rados.0.smithi038.stdout:346: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.664 INFO:tasks.rados.rados.0.smithi038.stdout:346: set_chunk oid 47 offset: 1827328 length: 321024 target oid smithi03829856-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1827328 snap -1 2024-03-19T11:30:05.664 INFO:tasks.rados.rados.0.smithi038.stdout:332: finishing set_chunk to oid 33 2024-03-19T11:30:05.664 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 33 set_chunk 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1190400 length: 835584 2024-03-19T11:30:05.665 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-03-19T11:30:05.665 INFO:tasks.rados.rados.0.smithi038.stdout:332: done (15 left) 2024-03-19T11:30:05.665 INFO:tasks.rados.rados.0.smithi038.stdout:347: set_chunk oid 48 target oid 48 low_tier 2024-03-19T11:30:05.665 INFO:tasks.rados.rados.0.smithi038.stdout:347: set_chunk oid 48 offset: 785408 length: 1983488 target oid smithi03829856-48 low_tier offset: 785408 snap -1 2024-03-19T11:30:05.665 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.665 INFO:tasks.rados.rados.0.smithi038.stdout:340: finishing set_chunk to oid 41 2024-03-19T11:30:05.665 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 41 set_chunk 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1194496 length: 1052672 2024-03-19T11:30:05.665 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 41 v 0 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-03-19T11:30:05.665 INFO:tasks.rados.rados.0.smithi038.stdout:340: done (15 left) 2024-03-19T11:30:05.665 INFO:tasks.rados.rados.0.smithi038.stdout:348: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.665 INFO:tasks.rados.rados.0.smithi038.stdout:348: set_chunk oid 49 offset: 839168 length: 286208 target oid smithi03829856-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 839168 snap -1 2024-03-19T11:30:05.665 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.665 INFO:tasks.rados.rados.0.smithi038.stdout:335: finishing set_chunk to oid 36 2024-03-19T11:30:05.665 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 36 set_chunk 36 low_tier returned error code -95 offset: 1249792 length: 1193472 2024-03-19T11:30:05.665 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-03-19T11:30:05.666 INFO:tasks.rados.rados.0.smithi038.stdout:335: done (15 left) 2024-03-19T11:30:05.666 INFO:tasks.rados.rados.0.smithi038.stdout:349: set_chunk oid 50 target oid 50 low_tier 2024-03-19T11:30:05.666 INFO:tasks.rados.rados.0.smithi038.stdout:349: set_chunk oid 50 offset: 483328 length: 103424 target oid smithi03829856-50 low_tier offset: 483328 snap -1 2024-03-19T11:30:05.666 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.666 INFO:tasks.rados.rados.0.smithi038.stdout:339: finishing set_chunk to oid 40 2024-03-19T11:30:05.666 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 40 set_chunk 40 low_tier returned error code -95 offset: 842752 length: 1175040 2024-03-19T11:30:05.666 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 40 v 0 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-03-19T11:30:05.666 INFO:tasks.rados.rados.0.smithi038.stdout:342: finishing set_chunk to oid 43 2024-03-19T11:30:05.666 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 43 set_chunk 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 811520 length: 1878016 2024-03-19T11:30:05.666 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 43 v 0 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-03-19T11:30:05.666 INFO:tasks.rados.rados.0.smithi038.stdout:339: done (15 left) 2024-03-19T11:30:05.666 INFO:tasks.rados.rados.0.smithi038.stdout:342: done (14 left) 2024-03-19T11:30:05.666 INFO:tasks.rados.rados.0.smithi038.stdout:350: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.666 INFO:tasks.rados.rados.0.smithi038.stdout:350: set_chunk oid 1 offset: 1074688 length: 589824 target oid smithi03829856-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1074688 snap -1 2024-03-19T11:30:05.667 INFO:tasks.rados.rados.0.smithi038.stdout:351: set_chunk oid 2 target oid 2 low_tier 2024-03-19T11:30:05.667 INFO:tasks.rados.rados.0.smithi038.stdout:351: set_chunk oid 2 offset: 101376 length: 55296 target oid smithi03829856-2 low_tier offset: 101376 snap -1 2024-03-19T11:30:05.667 INFO:tasks.rados.rados.0.smithi038.stdout:345: finishing set_chunk to oid 46 2024-03-19T11:30:05.667 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 46 set_chunk 46 low_tier returned error code -95 offset: 81920 length: 2663936 2024-03-19T11:30:05.667 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-03-19T11:30:05.667 INFO:tasks.rados.rados.0.smithi038.stdout:348: finishing set_chunk to oid 49 2024-03-19T11:30:05.667 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 49 set_chunk 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 839168 length: 286208 2024-03-19T11:30:05.667 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-03-19T11:30:05.667 INFO:tasks.rados.rados.0.smithi038.stdout:345: done (15 left) 2024-03-19T11:30:05.667 INFO:tasks.rados.rados.0.smithi038.stdout:348: done (14 left) 2024-03-19T11:30:05.667 INFO:tasks.rados.rados.0.smithi038.stdout:352: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.667 INFO:tasks.rados.rados.0.smithi038.stdout:352: set_chunk oid 3 offset: 1124864 length: 623104 target oid smithi03829856-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1124864 snap -1 2024-03-19T11:30:05.667 INFO:tasks.rados.rados.0.smithi038.stdout:353: set_chunk oid 4 target oid 4 low_tier 2024-03-19T11:30:05.667 INFO:tasks.rados.rados.0.smithi038.stdout:353: set_chunk oid 4 offset: 1732608 length: 512 target oid smithi03829856-4 low_tier offset: 1732608 snap -1 2024-03-19T11:30:05.668 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.668 INFO:tasks.rados.rados.0.smithi038.stdout:347: finishing set_chunk to oid 48 2024-03-19T11:30:05.668 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 48 set_chunk 48 low_tier returned error code -95 offset: 785408 length: 1983488 2024-03-19T11:30:05.668 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 48 v 0 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-03-19T11:30:05.668 INFO:tasks.rados.rados.0.smithi038.stdout:347: done (15 left) 2024-03-19T11:30:05.668 INFO:tasks.rados.rados.0.smithi038.stdout:354: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.668 INFO:tasks.rados.rados.0.smithi038.stdout:354: set_chunk oid 5 offset: 402944 length: 1075712 target oid smithi03829856-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 402944 snap -1 2024-03-19T11:30:05.668 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.668 INFO:tasks.rados.rados.0.smithi038.stdout:344: finishing set_chunk to oid 45 2024-03-19T11:30:05.668 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 45 set_chunk 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1108480 length: 366592 2024-03-19T11:30:05.668 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 45 v 0 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-03-19T11:30:05.668 INFO:tasks.rados.rados.0.smithi038.stdout:344: done (15 left) 2024-03-19T11:30:05.668 INFO:tasks.rados.rados.0.smithi038.stdout:355: set_chunk oid 6 target oid 6 low_tier 2024-03-19T11:30:05.668 INFO:tasks.rados.rados.0.smithi038.stdout:355: set_chunk oid 6 offset: 1653248 length: 417280 target oid smithi03829856-6 low_tier offset: 1653248 snap -1 2024-03-19T11:30:05.668 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.669 INFO:tasks.rados.rados.0.smithi038.stdout:346: finishing set_chunk to oid 47 2024-03-19T11:30:05.669 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 47 set_chunk 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1827328 length: 321024 2024-03-19T11:30:05.669 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-03-19T11:30:05.669 INFO:tasks.rados.rados.0.smithi038.stdout:350: finishing set_chunk to oid 1 2024-03-19T11:30:05.669 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 1 set_chunk 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1074688 length: 589824 2024-03-19T11:30:05.669 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 1 v 0 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-03-19T11:30:05.669 INFO:tasks.rados.rados.0.smithi038.stdout:346: done (15 left) 2024-03-19T11:30:05.669 INFO:tasks.rados.rados.0.smithi038.stdout:350: done (14 left) 2024-03-19T11:30:05.669 INFO:tasks.rados.rados.0.smithi038.stdout:356: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.669 INFO:tasks.rados.rados.0.smithi038.stdout:356: set_chunk oid 7 offset: 227840 length: 755200 target oid smithi03829856-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 227840 snap -1 2024-03-19T11:30:05.669 INFO:tasks.rados.rados.0.smithi038.stdout:357: set_chunk oid 8 target oid 8 low_tier 2024-03-19T11:30:05.669 INFO:tasks.rados.rados.0.smithi038.stdout:357: set_chunk oid 8 offset: 39424 length: 1591808 target oid smithi03829856-8 low_tier offset: 39424 snap -1 2024-03-19T11:30:05.669 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.669 INFO:tasks.rados.rados.0.smithi038.stdout:341: finishing set_chunk to oid 42 2024-03-19T11:30:05.670 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 42 set_chunk 42 low_tier returned error code -95 offset: 1507328 length: 964608 2024-03-19T11:30:05.670 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-03-19T11:30:05.670 INFO:tasks.rados.rados.0.smithi038.stdout:341: done (15 left) 2024-03-19T11:30:05.670 INFO:tasks.rados.rados.0.smithi038.stdout:358: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.670 INFO:tasks.rados.rados.0.smithi038.stdout:358: set_chunk oid 9 offset: 224256 length: 309760 target oid smithi03829856-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 224256 snap -1 2024-03-19T11:30:05.670 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.670 INFO:tasks.rados.rados.0.smithi038.stdout:352: finishing set_chunk to oid 3 2024-03-19T11:30:05.670 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 3 set_chunk 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1124864 length: 623104 2024-03-19T11:30:05.670 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 3 v 0 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-03-19T11:30:05.670 INFO:tasks.rados.rados.0.smithi038.stdout:352: done (15 left) 2024-03-19T11:30:05.670 INFO:tasks.rados.rados.0.smithi038.stdout:359: set_chunk oid 10 target oid 10 low_tier 2024-03-19T11:30:05.670 INFO:tasks.rados.rados.0.smithi038.stdout:359: set_chunk oid 10 offset: 360960 length: 1586176 target oid smithi03829856-10 low_tier offset: 360960 snap -1 2024-03-19T11:30:05.670 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.670 INFO:tasks.rados.rados.0.smithi038.stdout:311: finishing set_chunk to oid 12 2024-03-19T11:30:05.670 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 12 v 3 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-03-19T11:30:05.671 INFO:tasks.rados.rados.0.smithi038.stdout:311: done (15 left) 2024-03-19T11:30:05.671 INFO:tasks.rados.rados.0.smithi038.stdout:360: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.671 INFO:tasks.rados.rados.0.smithi038.stdout:360: set_chunk oid 11 offset: 206848 length: 677888 target oid smithi03829856-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 206848 snap -1 2024-03-19T11:30:05.671 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.671 INFO:tasks.rados.rados.0.smithi038.stdout:356: finishing set_chunk to oid 7 2024-03-19T11:30:05.671 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 7 set_chunk 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 227840 length: 755200 2024-03-19T11:30:05.671 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-03-19T11:30:05.671 INFO:tasks.rados.rados.0.smithi038.stdout:356: done (15 left) 2024-03-19T11:30:05.671 INFO:tasks.rados.rados.0.smithi038.stdout:361: set_chunk oid 12 target oid 12 low_tier 2024-03-19T11:30:05.671 INFO:tasks.rados.rados.0.smithi038.stdout:361: set_chunk oid 12 offset: 132608 length: 1261056 target oid smithi03829856-12 low_tier offset: 132608 snap -1 2024-03-19T11:30:05.671 INFO:tasks.rados.rados.0.smithi038.stdout:354: finishing set_chunk to oid 5 2024-03-19T11:30:05.671 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 5 set_chunk 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 402944 length: 1075712 2024-03-19T11:30:05.672 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-03-19T11:30:05.672 INFO:tasks.rados.rados.0.smithi038.stdout:354: done (15 left) 2024-03-19T11:30:05.672 INFO:tasks.rados.rados.0.smithi038.stdout:362: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.672 INFO:tasks.rados.rados.0.smithi038.stdout:362: set_chunk oid 13 offset: 1097216 length: 466944 target oid smithi03829856-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1097216 snap -1 2024-03-19T11:30:05.672 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.672 INFO:tasks.rados.rados.0.smithi038.stdout:355: finishing set_chunk to oid 6 2024-03-19T11:30:05.672 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 6 set_chunk 6 low_tier returned error code -95 offset: 1653248 length: 417280 2024-03-19T11:30:05.672 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-03-19T11:30:05.672 INFO:tasks.rados.rados.0.smithi038.stdout:355: done (15 left) 2024-03-19T11:30:05.672 INFO:tasks.rados.rados.0.smithi038.stdout:362 retry set_chunk ! 2024-03-19T11:30:05.672 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 15 2024-03-19T11:30:05.672 INFO:tasks.rados.rados.0.smithi038.stdout:359: finishing set_chunk to oid 10 2024-03-19T11:30:05.672 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 10 set_chunk 10 low_tier returned error code -95 offset: 360960 length: 1586176 2024-03-19T11:30:05.672 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 10 v 0 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-03-19T11:30:05.672 INFO:tasks.rados.rados.0.smithi038.stdout:359: done (14 left) 2024-03-19T11:30:05.673 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 14 2024-03-19T11:30:05.673 INFO:tasks.rados.rados.0.smithi038.stdout:360: finishing set_chunk to oid 11 2024-03-19T11:30:05.673 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 11 set_chunk 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 206848 length: 677888 2024-03-19T11:30:05.673 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 11 v 0 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-03-19T11:30:05.673 INFO:tasks.rados.rados.0.smithi038.stdout:357: finishing set_chunk to oid 8 2024-03-19T11:30:05.673 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 8 set_chunk 8 low_tier returned error code -95 offset: 39424 length: 1591808 2024-03-19T11:30:05.673 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-03-19T11:30:05.673 INFO:tasks.rados.rados.0.smithi038.stdout:357: done (13 left) 2024-03-19T11:30:05.673 INFO:tasks.rados.rados.0.smithi038.stdout:360: done (12 left) 2024-03-19T11:30:05.673 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 12 2024-03-19T11:30:05.673 INFO:tasks.rados.rados.0.smithi038.stdout:361: finishing set_chunk to oid 12 2024-03-19T11:30:05.673 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 12 set_chunk 12 low_tier returned error code -95 offset: 132608 length: 1261056 2024-03-19T11:30:05.673 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 12 v 0 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-03-19T11:30:05.673 INFO:tasks.rados.rados.0.smithi038.stdout:362: finishing set_chunk to oid 13 2024-03-19T11:30:05.673 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 13 set_chunk 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1097216 length: 466944 2024-03-19T11:30:05.674 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 13 v 0 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-03-19T11:30:05.674 INFO:tasks.rados.rados.0.smithi038.stdout:361: done (11 left) 2024-03-19T11:30:05.674 INFO:tasks.rados.rados.0.smithi038.stdout:362: done (10 left) 2024-03-19T11:30:05.674 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 10 2024-03-19T11:30:05.674 INFO:tasks.rados.rados.0.smithi038.stdout:313: finishing set_chunk to oid 14 2024-03-19T11:30:05.674 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 14 v 3 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-03-19T11:30:05.674 INFO:tasks.rados.rados.0.smithi038.stdout:313: done (9 left) 2024-03-19T11:30:05.674 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 9 2024-03-19T11:30:05.674 INFO:tasks.rados.rados.0.smithi038.stdout:319: finishing set_chunk to oid 20 2024-03-19T11:30:05.674 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 20 v 12 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-03-19T11:30:05.674 INFO:tasks.rados.rados.0.smithi038.stdout:319: done (8 left) 2024-03-19T11:30:05.674 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 8 2024-03-19T11:30:05.674 INFO:tasks.rados.rados.0.smithi038.stdout:337: finishing set_chunk to oid 38 2024-03-19T11:30:05.674 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 38 v 12 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-03-19T11:30:05.674 INFO:tasks.rados.rados.0.smithi038.stdout:337: done (7 left) 2024-03-19T11:30:05.675 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 7 2024-03-19T11:30:05.675 INFO:tasks.rados.rados.0.smithi038.stdout:353: finishing set_chunk to oid 4 2024-03-19T11:30:05.675 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 4 v 3 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-03-19T11:30:05.675 INFO:tasks.rados.rados.0.smithi038.stdout:353: done (6 left) 2024-03-19T11:30:05.675 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 6 2024-03-19T11:30:05.675 INFO:tasks.rados.rados.0.smithi038.stdout:351: finishing set_chunk to oid 2 2024-03-19T11:30:05.675 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 2 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-03-19T11:30:05.675 INFO:tasks.rados.rados.0.smithi038.stdout:351: done (5 left) 2024-03-19T11:30:05.675 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 5 2024-03-19T11:30:05.675 INFO:tasks.rados.rados.0.smithi038.stdout:322: finishing set_chunk to oid 23 2024-03-19T11:30:05.675 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 23 v 9 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-03-19T11:30:05.675 INFO:tasks.rados.rados.0.smithi038.stdout:322: done (4 left) 2024-03-19T11:30:05.675 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 4 2024-03-19T11:30:05.675 INFO:tasks.rados.rados.0.smithi038.stdout:338: finishing set_chunk to oid 39 2024-03-19T11:30:05.676 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 39 v 6 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-03-19T11:30:05.676 INFO:tasks.rados.rados.0.smithi038.stdout:338: done (3 left) 2024-03-19T11:30:05.676 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 3 2024-03-19T11:30:05.676 INFO:tasks.rados.rados.0.smithi038.stdout:358: finishing set_chunk to oid 9 2024-03-19T11:30:05.676 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 9 v 6 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-03-19T11:30:05.676 INFO:tasks.rados.rados.0.smithi038.stdout:349: finishing set_chunk to oid 50 2024-03-19T11:30:05.676 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 50 v 15 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-03-19T11:30:05.676 INFO:tasks.rados.rados.0.smithi038.stdout:349: done (2 left) 2024-03-19T11:30:05.676 INFO:tasks.rados.rados.0.smithi038.stdout:358: done (1 left) 2024-03-19T11:30:05.676 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 1 2024-03-19T11:30:05.676 INFO:tasks.rados.rados.0.smithi038.stdout:343: finishing set_chunk to oid 44 2024-03-19T11:30:05.676 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 44 v 9 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-03-19T11:30:05.676 INFO:tasks.rados.rados.0.smithi038.stdout:343: done (0 left) 2024-03-19T11:30:05.676 INFO:tasks.rados.rados.0.smithi038.stdout:363: set_chunk oid 14 target oid 14 low_tier 2024-03-19T11:30:05.676 INFO:tasks.rados.rados.0.smithi038.stdout:363: set_chunk oid 14 offset: 592896 length: 782848 target oid smithi03829856-14 low_tier offset: 592896 snap -1 2024-03-19T11:30:05.677 INFO:tasks.rados.rados.0.smithi038.stdout:364: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.677 INFO:tasks.rados.rados.0.smithi038.stdout:364: set_chunk oid 15 offset: 143360 length: 665088 target oid smithi03829856-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 143360 snap -1 2024-03-19T11:30:05.677 INFO:tasks.rados.rados.0.smithi038.stdout:365: set_chunk oid 16 target oid 16 low_tier 2024-03-19T11:30:05.677 INFO:tasks.rados.rados.0.smithi038.stdout:365: set_chunk oid 16 offset: 1637888 length: 471552 target oid smithi03829856-16 low_tier offset: 1637888 snap -1 2024-03-19T11:30:05.677 INFO:tasks.rados.rados.0.smithi038.stdout:366: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.677 INFO:tasks.rados.rados.0.smithi038.stdout:366: set_chunk oid 17 offset: 1156608 length: 654336 target oid smithi03829856-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1156608 snap -1 2024-03-19T11:30:05.677 INFO:tasks.rados.rados.0.smithi038.stdout:367: set_chunk oid 18 target oid 18 low_tier 2024-03-19T11:30:05.677 INFO:tasks.rados.rados.0.smithi038.stdout:367: set_chunk oid 18 offset: 390656 length: 795136 target oid smithi03829856-18 low_tier offset: 390656 snap -1 2024-03-19T11:30:05.677 INFO:tasks.rados.rados.0.smithi038.stdout:368: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.677 INFO:tasks.rados.rados.0.smithi038.stdout:368: set_chunk oid 19 offset: 65024 length: 1584128 target oid smithi03829856-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 65024 snap -1 2024-03-19T11:30:05.677 INFO:tasks.rados.rados.0.smithi038.stdout:369: set_chunk oid 20 target oid 20 low_tier 2024-03-19T11:30:05.677 INFO:tasks.rados.rados.0.smithi038.stdout:369: set_chunk oid 20 offset: 495104 length: 908288 target oid smithi03829856-20 low_tier offset: 495104 snap -1 2024-03-19T11:30:05.677 INFO:tasks.rados.rados.0.smithi038.stdout:370: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.677 INFO:tasks.rados.rados.0.smithi038.stdout:370: set_chunk oid 21 offset: 497152 length: 1476608 target oid smithi03829856-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 497152 snap -1 2024-03-19T11:30:05.677 INFO:tasks.rados.rados.0.smithi038.stdout:371: set_chunk oid 22 target oid 22 low_tier 2024-03-19T11:30:05.678 INFO:tasks.rados.rados.0.smithi038.stdout:371: set_chunk oid 22 offset: 50176 length: 176640 target oid smithi03829856-22 low_tier offset: 50176 snap -1 2024-03-19T11:30:05.678 INFO:tasks.rados.rados.0.smithi038.stdout:372: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.678 INFO:tasks.rados.rados.0.smithi038.stdout:372: set_chunk oid 23 offset: 1773568 length: 528384 target oid smithi03829856-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1773568 snap -1 2024-03-19T11:30:05.678 INFO:tasks.rados.rados.0.smithi038.stdout:373: set_chunk oid 24 target oid 24 low_tier 2024-03-19T11:30:05.678 INFO:tasks.rados.rados.0.smithi038.stdout:373: set_chunk oid 24 offset: 20992 length: 675840 target oid smithi03829856-24 low_tier offset: 20992 snap -1 2024-03-19T11:30:05.678 INFO:tasks.rados.rados.0.smithi038.stdout:374: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.678 INFO:tasks.rados.rados.0.smithi038.stdout:374: set_chunk oid 25 offset: 691712 length: 1484288 target oid smithi03829856-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 691712 snap -1 2024-03-19T11:30:05.678 INFO:tasks.rados.rados.0.smithi038.stdout:375: set_chunk oid 26 target oid 26 low_tier 2024-03-19T11:30:05.678 INFO:tasks.rados.rados.0.smithi038.stdout:375: set_chunk oid 26 offset: 772096 length: 1305600 target oid smithi03829856-26 low_tier offset: 772096 snap -1 2024-03-19T11:30:05.678 INFO:tasks.rados.rados.0.smithi038.stdout:376: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.678 INFO:tasks.rados.rados.0.smithi038.stdout:376: set_chunk oid 27 offset: 1374720 length: 19456 target oid smithi03829856-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1374720 snap -1 2024-03-19T11:30:05.678 INFO:tasks.rados.rados.0.smithi038.stdout:377: set_chunk oid 28 target oid 28 low_tier 2024-03-19T11:30:05.678 INFO:tasks.rados.rados.0.smithi038.stdout:377: set_chunk oid 28 offset: 1591808 length: 315904 target oid smithi03829856-28 low_tier offset: 1591808 snap -1 2024-03-19T11:30:05.679 INFO:tasks.rados.rados.0.smithi038.stdout:378: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.679 INFO:tasks.rados.rados.0.smithi038.stdout:378: set_chunk oid 29 offset: 1117184 length: 140800 target oid smithi03829856-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1117184 snap -1 2024-03-19T11:30:05.679 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.679 INFO:tasks.rados.rados.0.smithi038.stdout:363: finishing set_chunk to oid 14 2024-03-19T11:30:05.679 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 14 set_chunk 14 low_tier returned error code -95 offset: 592896 length: 782848 2024-03-19T11:30:05.679 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 14 v 0 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-03-19T11:30:05.679 INFO:tasks.rados.rados.0.smithi038.stdout:365: finishing set_chunk to oid 16 2024-03-19T11:30:05.679 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 16 set_chunk 16 low_tier returned error code -95 offset: 1637888 length: 471552 2024-03-19T11:30:05.679 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-03-19T11:30:05.679 INFO:tasks.rados.rados.0.smithi038.stdout:364: finishing set_chunk to oid 15 2024-03-19T11:30:05.679 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 143360 length: 665088 2024-03-19T11:30:05.680 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-03-19T11:30:05.680 INFO:tasks.rados.rados.0.smithi038.stdout:366: finishing set_chunk to oid 17 2024-03-19T11:30:05.680 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 17 set_chunk 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1156608 length: 654336 2024-03-19T11:30:05.680 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 17 v 0 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-03-19T11:30:05.680 INFO:tasks.rados.rados.0.smithi038.stdout:367: finishing set_chunk to oid 18 2024-03-19T11:30:05.680 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 18 set_chunk 18 low_tier returned error code -95 offset: 390656 length: 795136 2024-03-19T11:30:05.680 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 18 v 0 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-03-19T11:30:05.680 INFO:tasks.rados.rados.0.smithi038.stdout:369: finishing set_chunk to oid 20 2024-03-19T11:30:05.680 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 20 set_chunk 20 low_tier returned error code -95 offset: 495104 length: 908288 2024-03-19T11:30:05.680 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 20 v 0 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-03-19T11:30:05.680 INFO:tasks.rados.rados.0.smithi038.stdout:370: finishing set_chunk to oid 21 2024-03-19T11:30:05.680 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 21 set_chunk 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 497152 length: 1476608 2024-03-19T11:30:05.680 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-03-19T11:30:05.680 INFO:tasks.rados.rados.0.smithi038.stdout:372: finishing set_chunk to oid 23 2024-03-19T11:30:05.680 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 23 set_chunk 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1773568 length: 528384 2024-03-19T11:30:05.681 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 23 v 0 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-03-19T11:30:05.681 INFO:tasks.rados.rados.0.smithi038.stdout:373: finishing set_chunk to oid 24 2024-03-19T11:30:05.681 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 24 set_chunk 24 low_tier returned error code -95 offset: 20992 length: 675840 2024-03-19T11:30:05.681 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-03-19T11:30:05.681 INFO:tasks.rados.rados.0.smithi038.stdout:368: finishing set_chunk to oid 19 2024-03-19T11:30:05.681 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 19 set_chunk 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 65024 length: 1584128 2024-03-19T11:30:05.681 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 19 v 0 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-03-19T11:30:05.681 INFO:tasks.rados.rados.0.smithi038.stdout:363: done (15 left) 2024-03-19T11:30:05.681 INFO:tasks.rados.rados.0.smithi038.stdout:364: done (14 left) 2024-03-19T11:30:05.681 INFO:tasks.rados.rados.0.smithi038.stdout:365: done (13 left) 2024-03-19T11:30:05.681 INFO:tasks.rados.rados.0.smithi038.stdout:366: done (12 left) 2024-03-19T11:30:05.681 INFO:tasks.rados.rados.0.smithi038.stdout:367: done (11 left) 2024-03-19T11:30:05.681 INFO:tasks.rados.rados.0.smithi038.stdout:368: done (10 left) 2024-03-19T11:30:05.681 INFO:tasks.rados.rados.0.smithi038.stdout:369: done (9 left) 2024-03-19T11:30:05.681 INFO:tasks.rados.rados.0.smithi038.stdout:370: done (8 left) 2024-03-19T11:30:05.682 INFO:tasks.rados.rados.0.smithi038.stdout:372: done (7 left) 2024-03-19T11:30:05.682 INFO:tasks.rados.rados.0.smithi038.stdout:373: done (6 left) 2024-03-19T11:30:05.682 INFO:tasks.rados.rados.0.smithi038.stdout:379: set_chunk oid 30 target oid 30 low_tier 2024-03-19T11:30:05.682 INFO:tasks.rados.rados.0.smithi038.stdout:379: set_chunk oid 30 offset: 1887232 length: 37376 target oid smithi03829856-30 low_tier offset: 1887232 snap -1 2024-03-19T11:30:05.682 INFO:tasks.rados.rados.0.smithi038.stdout:380: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.682 INFO:tasks.rados.rados.0.smithi038.stdout:380: set_chunk oid 31 offset: 72704 length: 626688 target oid smithi03829856-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 72704 snap -1 2024-03-19T11:30:05.682 INFO:tasks.rados.rados.0.smithi038.stdout:381: set_chunk oid 32 target oid 32 low_tier 2024-03-19T11:30:05.682 INFO:tasks.rados.rados.0.smithi038.stdout:381: set_chunk oid 32 offset: 850944 length: 987648 target oid smithi03829856-32 low_tier offset: 850944 snap -1 2024-03-19T11:30:05.682 INFO:tasks.rados.rados.0.smithi038.stdout:382: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.682 INFO:tasks.rados.rados.0.smithi038.stdout:382: set_chunk oid 33 offset: 1763840 length: 465408 target oid smithi03829856-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1763840 snap -1 2024-03-19T11:30:05.682 INFO:tasks.rados.rados.0.smithi038.stdout:383: set_chunk oid 34 target oid 34 low_tier 2024-03-19T11:30:05.682 INFO:tasks.rados.rados.0.smithi038.stdout:383: set_chunk oid 34 offset: 440320 length: 1363968 target oid smithi03829856-34 low_tier offset: 440320 snap -1 2024-03-19T11:30:05.682 INFO:tasks.rados.rados.0.smithi038.stdout:384: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.682 INFO:tasks.rados.rados.0.smithi038.stdout:384: set_chunk oid 35 offset: 88064 length: 217088 target oid smithi03829856-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 88064 snap -1 2024-03-19T11:30:05.682 INFO:tasks.rados.rados.0.smithi038.stdout:385: set_chunk oid 36 target oid 36 low_tier 2024-03-19T11:30:05.683 INFO:tasks.rados.rados.0.smithi038.stdout:385: set_chunk oid 36 offset: 775168 length: 696832 target oid smithi03829856-36 low_tier offset: 775168 snap -1 2024-03-19T11:30:05.683 INFO:tasks.rados.rados.0.smithi038.stdout:386: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.683 INFO:tasks.rados.rados.0.smithi038.stdout:386: set_chunk oid 37 offset: 456192 length: 1123328 target oid smithi03829856-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 456192 snap -1 2024-03-19T11:30:05.683 INFO:tasks.rados.rados.0.smithi038.stdout:387: set_chunk oid 38 target oid 38 low_tier 2024-03-19T11:30:05.683 INFO:tasks.rados.rados.0.smithi038.stdout:374: finishing set_chunk to oid 25 2024-03-19T11:30:05.683 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 25 set_chunk 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 691712 length: 1484288 2024-03-19T11:30:05.683 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-03-19T11:30:05.683 INFO:tasks.rados.rados.0.smithi038.stdout:387: set_chunk oid 38 offset: 249856 length: 424960 target oid smithi03829856-38 low_tier offset: 249856 snap -1 2024-03-19T11:30:05.683 INFO:tasks.rados.rados.0.smithi038.stdout:374: done (14 left) 2024-03-19T11:30:05.683 INFO:tasks.rados.rados.0.smithi038.stdout:388: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.683 INFO:tasks.rados.rados.0.smithi038.stdout:388: set_chunk oid 39 offset: 365568 length: 1124864 target oid smithi03829856-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 365568 snap -1 2024-03-19T11:30:05.683 INFO:tasks.rados.rados.0.smithi038.stdout:389: set_chunk oid 40 target oid 40 low_tier 2024-03-19T11:30:05.683 INFO:tasks.rados.rados.0.smithi038.stdout:389: set_chunk oid 40 offset: 254976 length: 1890304 target oid smithi03829856-40 low_tier offset: 254976 snap -1 2024-03-19T11:30:05.684 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.684 INFO:tasks.rados.rados.0.smithi038.stdout:377: finishing set_chunk to oid 28 2024-03-19T11:30:05.684 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 28 set_chunk 28 low_tier returned error code -95 offset: 1591808 length: 315904 2024-03-19T11:30:05.684 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 28 v 0 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-03-19T11:30:05.684 INFO:tasks.rados.rados.0.smithi038.stdout:375: finishing set_chunk to oid 26 2024-03-19T11:30:05.684 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 26 set_chunk 26 low_tier returned error code -95 offset: 772096 length: 1305600 2024-03-19T11:30:05.684 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-03-19T11:30:05.684 INFO:tasks.rados.rados.0.smithi038.stdout:375: done (15 left) 2024-03-19T11:30:05.684 INFO:tasks.rados.rados.0.smithi038.stdout:377: done (14 left) 2024-03-19T11:30:05.684 INFO:tasks.rados.rados.0.smithi038.stdout:390: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.684 INFO:tasks.rados.rados.0.smithi038.stdout:390: set_chunk oid 41 offset: 691712 length: 1924608 target oid smithi03829856-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 691712 snap -1 2024-03-19T11:30:05.684 INFO:tasks.rados.rados.0.smithi038.stdout:391: set_chunk oid 42 target oid 42 low_tier 2024-03-19T11:30:05.684 INFO:tasks.rados.rados.0.smithi038.stdout:391: set_chunk oid 42 offset: 1087488 length: 531456 target oid smithi03829856-42 low_tier offset: 1087488 snap -1 2024-03-19T11:30:05.684 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.684 INFO:tasks.rados.rados.0.smithi038.stdout:381: finishing set_chunk to oid 32 2024-03-19T11:30:05.685 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 32 set_chunk 32 low_tier returned error code -95 offset: 850944 length: 987648 2024-03-19T11:30:05.685 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-03-19T11:30:05.685 INFO:tasks.rados.rados.0.smithi038.stdout:385: finishing set_chunk to oid 36 2024-03-19T11:30:05.685 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 36 set_chunk 36 low_tier returned error code -95 offset: 775168 length: 696832 2024-03-19T11:30:05.685 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-03-19T11:30:05.685 INFO:tasks.rados.rados.0.smithi038.stdout:387: finishing set_chunk to oid 38 2024-03-19T11:30:05.685 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 38 set_chunk 38 low_tier returned error code -95 offset: 249856 length: 424960 2024-03-19T11:30:05.685 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 38 v 0 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-03-19T11:30:05.685 INFO:tasks.rados.rados.0.smithi038.stdout:391: finishing set_chunk to oid 42 2024-03-19T11:30:05.685 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 42 set_chunk 42 low_tier returned error code -95 offset: 1087488 length: 531456 2024-03-19T11:30:05.685 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-03-19T11:30:05.685 INFO:tasks.rados.rados.0.smithi038.stdout:386: finishing set_chunk to oid 37 2024-03-19T11:30:05.685 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 37 set_chunk 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 456192 length: 1123328 2024-03-19T11:30:05.685 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 37 v 0 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-03-19T11:30:05.685 INFO:tasks.rados.rados.0.smithi038.stdout:390: finishing set_chunk to oid 41 2024-03-19T11:30:05.686 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 41 set_chunk 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 691712 length: 1924608 2024-03-19T11:30:05.686 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 41 v 0 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-03-19T11:30:05.686 INFO:tasks.rados.rados.0.smithi038.stdout:383: finishing set_chunk to oid 34 2024-03-19T11:30:05.686 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 34 set_chunk 34 low_tier returned error code -95 offset: 440320 length: 1363968 2024-03-19T11:30:05.686 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 34 v 0 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-03-19T11:30:05.686 INFO:tasks.rados.rados.0.smithi038.stdout:382: finishing set_chunk to oid 33 2024-03-19T11:30:05.686 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 33 set_chunk 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1763840 length: 465408 2024-03-19T11:30:05.686 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-03-19T11:30:05.686 INFO:tasks.rados.rados.0.smithi038.stdout:388: finishing set_chunk to oid 39 2024-03-19T11:30:05.686 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 39 set_chunk 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 365568 length: 1124864 2024-03-19T11:30:05.686 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 39 v 0 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-03-19T11:30:05.686 INFO:tasks.rados.rados.0.smithi038.stdout:389: finishing set_chunk to oid 40 2024-03-19T11:30:05.687 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 40 set_chunk 40 low_tier returned error code -95 offset: 254976 length: 1890304 2024-03-19T11:30:05.687 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 40 v 0 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-03-19T11:30:05.687 INFO:tasks.rados.rados.0.smithi038.stdout:381: done (15 left) 2024-03-19T11:30:05.687 INFO:tasks.rados.rados.0.smithi038.stdout:382: done (14 left) 2024-03-19T11:30:05.687 INFO:tasks.rados.rados.0.smithi038.stdout:383: done (13 left) 2024-03-19T11:30:05.687 INFO:tasks.rados.rados.0.smithi038.stdout:385: done (12 left) 2024-03-19T11:30:05.687 INFO:tasks.rados.rados.0.smithi038.stdout:386: done (11 left) 2024-03-19T11:30:05.687 INFO:tasks.rados.rados.0.smithi038.stdout:387: done (10 left) 2024-03-19T11:30:05.687 INFO:tasks.rados.rados.0.smithi038.stdout:388: done (9 left) 2024-03-19T11:30:05.687 INFO:tasks.rados.rados.0.smithi038.stdout:389: done (8 left) 2024-03-19T11:30:05.687 INFO:tasks.rados.rados.0.smithi038.stdout:390: done (7 left) 2024-03-19T11:30:05.687 INFO:tasks.rados.rados.0.smithi038.stdout:391: done (6 left) 2024-03-19T11:30:05.687 INFO:tasks.rados.rados.0.smithi038.stdout:392: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.687 INFO:tasks.rados.rados.0.smithi038.stdout:392: set_chunk oid 43 offset: 1156608 length: 149504 target oid smithi03829856-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1156608 snap -1 2024-03-19T11:30:05.687 INFO:tasks.rados.rados.0.smithi038.stdout:393: set_chunk oid 44 target oid 44 low_tier 2024-03-19T11:30:05.688 INFO:tasks.rados.rados.0.smithi038.stdout:393: set_chunk oid 44 offset: 458240 length: 572416 target oid smithi03829856-44 low_tier offset: 458240 snap -1 2024-03-19T11:30:05.688 INFO:tasks.rados.rados.0.smithi038.stdout:394: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.688 INFO:tasks.rados.rados.0.smithi038.stdout:394: set_chunk oid 45 offset: 246784 length: 2156544 target oid smithi03829856-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 246784 snap -1 2024-03-19T11:30:05.688 INFO:tasks.rados.rados.0.smithi038.stdout:395: set_chunk oid 46 target oid 46 low_tier 2024-03-19T11:30:05.688 INFO:tasks.rados.rados.0.smithi038.stdout:395: set_chunk oid 46 offset: 44544 length: 870912 target oid smithi03829856-46 low_tier offset: 44544 snap -1 2024-03-19T11:30:05.688 INFO:tasks.rados.rados.0.smithi038.stdout:396: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.688 INFO:tasks.rados.rados.0.smithi038.stdout:396: set_chunk oid 47 offset: 921600 length: 1547776 target oid smithi03829856-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 921600 snap -1 2024-03-19T11:30:05.688 INFO:tasks.rados.rados.0.smithi038.stdout:397: set_chunk oid 48 target oid 48 low_tier 2024-03-19T11:30:05.688 INFO:tasks.rados.rados.0.smithi038.stdout:397: set_chunk oid 48 offset: 1384448 length: 844288 target oid smithi03829856-48 low_tier offset: 1384448 snap -1 2024-03-19T11:30:05.688 INFO:tasks.rados.rados.0.smithi038.stdout:398: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.688 INFO:tasks.rados.rados.0.smithi038.stdout:398: set_chunk oid 49 offset: 95232 length: 2338816 target oid smithi03829856-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 95232 snap -1 2024-03-19T11:30:05.688 INFO:tasks.rados.rados.0.smithi038.stdout:399: set_chunk oid 50 target oid 50 low_tier 2024-03-19T11:30:05.688 INFO:tasks.rados.rados.0.smithi038.stdout:399: set_chunk oid 50 offset: 1558528 length: 1056768 target oid smithi03829856-50 low_tier offset: 1558528 snap -1 2024-03-19T11:30:05.689 INFO:tasks.rados.rados.0.smithi038.stdout:400: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.689 INFO:tasks.rados.rados.0.smithi038.stdout:400: set_chunk oid 1 offset: 70144 length: 1682944 target oid smithi03829856-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 70144 snap -1 2024-03-19T11:30:05.689 INFO:tasks.rados.rados.0.smithi038.stdout:401: set_chunk oid 2 target oid 2 low_tier 2024-03-19T11:30:05.689 INFO:tasks.rados.rados.0.smithi038.stdout:392: finishing set_chunk to oid 43 2024-03-19T11:30:05.689 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 43 set_chunk 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1156608 length: 149504 2024-03-19T11:30:05.689 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 43 v 0 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-03-19T11:30:05.689 INFO:tasks.rados.rados.0.smithi038.stdout:401: set_chunk oid 2 offset: 582144 length: 454656 target oid smithi03829856-2 low_tier offset: 582144 snap -1 2024-03-19T11:30:05.689 INFO:tasks.rados.rados.0.smithi038.stdout:392: done (15 left) 2024-03-19T11:30:05.689 INFO:tasks.rados.rados.0.smithi038.stdout:402: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.689 INFO:tasks.rados.rados.0.smithi038.stdout:402: set_chunk oid 3 offset: 498688 length: 1405952 target oid smithi03829856-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 498688 snap -1 2024-03-19T11:30:05.689 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.689 INFO:tasks.rados.rados.0.smithi038.stdout:396: finishing set_chunk to oid 47 2024-03-19T11:30:05.689 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 47 set_chunk 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 921600 length: 1547776 2024-03-19T11:30:05.689 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-03-19T11:30:05.689 INFO:tasks.rados.rados.0.smithi038.stdout:394: finishing set_chunk to oid 45 2024-03-19T11:30:05.690 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 45 set_chunk 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 246784 length: 2156544 2024-03-19T11:30:05.690 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 45 v 0 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-03-19T11:30:05.690 INFO:tasks.rados.rados.0.smithi038.stdout:394: done (15 left) 2024-03-19T11:30:05.690 INFO:tasks.rados.rados.0.smithi038.stdout:396: done (14 left) 2024-03-19T11:30:05.690 INFO:tasks.rados.rados.0.smithi038.stdout:403: set_chunk oid 4 target oid 4 low_tier 2024-03-19T11:30:05.690 INFO:tasks.rados.rados.0.smithi038.stdout:403: set_chunk oid 4 offset: 1094656 length: 723456 target oid smithi03829856-4 low_tier offset: 1094656 snap -1 2024-03-19T11:30:05.690 INFO:tasks.rados.rados.0.smithi038.stdout:404: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.690 INFO:tasks.rados.rados.0.smithi038.stdout:404: set_chunk oid 5 offset: 596480 length: 1009664 target oid smithi03829856-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 596480 snap -1 2024-03-19T11:30:05.690 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.690 INFO:tasks.rados.rados.0.smithi038.stdout:398: finishing set_chunk to oid 49 2024-03-19T11:30:05.690 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 49 set_chunk 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 95232 length: 2338816 2024-03-19T11:30:05.690 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-03-19T11:30:05.690 INFO:tasks.rados.rados.0.smithi038.stdout:397: finishing set_chunk to oid 48 2024-03-19T11:30:05.690 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 48 set_chunk 48 low_tier returned error code -95 offset: 1384448 length: 844288 2024-03-19T11:30:05.690 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 48 v 0 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-03-19T11:30:05.691 INFO:tasks.rados.rados.0.smithi038.stdout:399: finishing set_chunk to oid 50 2024-03-19T11:30:05.691 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 50 set_chunk 50 low_tier returned error code -95 offset: 1558528 length: 1056768 2024-03-19T11:30:05.691 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 50 v 0 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-03-19T11:30:05.691 INFO:tasks.rados.rados.0.smithi038.stdout:400: finishing set_chunk to oid 1 2024-03-19T11:30:05.691 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 1 set_chunk 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 70144 length: 1682944 2024-03-19T11:30:05.691 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 1 v 0 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-03-19T11:30:05.691 INFO:tasks.rados.rados.0.smithi038.stdout:397: done (15 left) 2024-03-19T11:30:05.691 INFO:tasks.rados.rados.0.smithi038.stdout:398: done (14 left) 2024-03-19T11:30:05.691 INFO:tasks.rados.rados.0.smithi038.stdout:399: done (13 left) 2024-03-19T11:30:05.691 INFO:tasks.rados.rados.0.smithi038.stdout:400: done (12 left) 2024-03-19T11:30:05.691 INFO:tasks.rados.rados.0.smithi038.stdout:405: set_chunk oid 6 target oid 6 low_tier 2024-03-19T11:30:05.691 INFO:tasks.rados.rados.0.smithi038.stdout:405: set_chunk oid 6 offset: 989184 length: 489472 target oid smithi03829856-6 low_tier offset: 989184 snap -1 2024-03-19T11:30:05.691 INFO:tasks.rados.rados.0.smithi038.stdout:406: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.691 INFO:tasks.rados.rados.0.smithi038.stdout:406: set_chunk oid 7 offset: 1093632 length: 934400 target oid smithi03829856-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1093632 snap -1 2024-03-19T11:30:05.692 INFO:tasks.rados.rados.0.smithi038.stdout:407: set_chunk oid 8 target oid 8 low_tier 2024-03-19T11:30:05.692 INFO:tasks.rados.rados.0.smithi038.stdout:407: set_chunk oid 8 offset: 1734656 length: 136192 target oid smithi03829856-8 low_tier offset: 1734656 snap -1 2024-03-19T11:30:05.692 INFO:tasks.rados.rados.0.smithi038.stdout:408: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.692 INFO:tasks.rados.rados.0.smithi038.stdout:408: set_chunk oid 9 offset: 215552 length: 117248 target oid smithi03829856-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 215552 snap -1 2024-03-19T11:30:05.692 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.692 INFO:tasks.rados.rados.0.smithi038.stdout:402: finishing set_chunk to oid 3 2024-03-19T11:30:05.692 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 3 set_chunk 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 498688 length: 1405952 2024-03-19T11:30:05.692 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 3 v 0 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-03-19T11:30:05.692 INFO:tasks.rados.rados.0.smithi038.stdout:371: finishing set_chunk to oid 22 2024-03-19T11:30:05.692 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 22 v 3 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-03-19T11:30:05.692 INFO:tasks.rados.rados.0.smithi038.stdout:371: done (15 left) 2024-03-19T11:30:05.692 INFO:tasks.rados.rados.0.smithi038.stdout:402: done (14 left) 2024-03-19T11:30:05.692 INFO:tasks.rados.rados.0.smithi038.stdout:409: set_chunk oid 10 target oid 10 low_tier 2024-03-19T11:30:05.692 INFO:tasks.rados.rados.0.smithi038.stdout:409: set_chunk oid 10 offset: 836608 length: 725504 target oid smithi03829856-10 low_tier offset: 836608 snap -1 2024-03-19T11:30:05.692 INFO:tasks.rados.rados.0.smithi038.stdout:410: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.693 INFO:tasks.rados.rados.0.smithi038.stdout:410: set_chunk oid 11 offset: 228352 length: 192000 target oid smithi03829856-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 228352 snap -1 2024-03-19T11:30:05.693 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.693 INFO:tasks.rados.rados.0.smithi038.stdout:404: finishing set_chunk to oid 5 2024-03-19T11:30:05.693 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 5 set_chunk 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 596480 length: 1009664 2024-03-19T11:30:05.693 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-03-19T11:30:05.693 INFO:tasks.rados.rados.0.smithi038.stdout:403: finishing set_chunk to oid 4 2024-03-19T11:30:05.693 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 4 set_chunk 4 low_tier returned error code -95 offset: 1094656 length: 723456 2024-03-19T11:30:05.693 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-03-19T11:30:05.693 INFO:tasks.rados.rados.0.smithi038.stdout:403: done (15 left) 2024-03-19T11:30:05.693 INFO:tasks.rados.rados.0.smithi038.stdout:404: done (14 left) 2024-03-19T11:30:05.693 INFO:tasks.rados.rados.0.smithi038.stdout:411: set_chunk oid 12 target oid 12 low_tier 2024-03-19T11:30:05.693 INFO:tasks.rados.rados.0.smithi038.stdout:411: set_chunk oid 12 offset: 1676288 length: 491520 target oid smithi03829856-12 low_tier offset: 1676288 snap -1 2024-03-19T11:30:05.693 INFO:tasks.rados.rados.0.smithi038.stdout:412: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.693 INFO:tasks.rados.rados.0.smithi038.stdout:412: set_chunk oid 13 offset: 226304 length: 1228288 target oid smithi03829856-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 226304 snap -1 2024-03-19T11:30:05.693 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.694 INFO:tasks.rados.rados.0.smithi038.stdout:401: finishing set_chunk to oid 2 2024-03-19T11:30:05.694 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 2 set_chunk 2 low_tier returned error code -95 offset: 582144 length: 454656 2024-03-19T11:30:05.694 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-03-19T11:30:05.694 INFO:tasks.rados.rados.0.smithi038.stdout:408: finishing set_chunk to oid 9 2024-03-19T11:30:05.694 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 9 set_chunk 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 215552 length: 117248 2024-03-19T11:30:05.694 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 9 v 0 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-03-19T11:30:05.694 INFO:tasks.rados.rados.0.smithi038.stdout:409: finishing set_chunk to oid 10 2024-03-19T11:30:05.694 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 10 set_chunk 10 low_tier returned error code -95 offset: 836608 length: 725504 2024-03-19T11:30:05.694 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 10 v 0 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-03-19T11:30:05.695 INFO:tasks.rados.rados.0.smithi038.stdout:405: finishing set_chunk to oid 6 2024-03-19T11:30:05.695 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 6 set_chunk 6 low_tier returned error code -95 offset: 989184 length: 489472 2024-03-19T11:30:05.695 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-03-19T11:30:05.696 INFO:tasks.rados.rados.0.smithi038.stdout:406: finishing set_chunk to oid 7 2024-03-19T11:30:05.696 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 7 set_chunk 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1093632 length: 934400 2024-03-19T11:30:05.696 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-03-19T11:30:05.696 INFO:tasks.rados.rados.0.smithi038.stdout:411: finishing set_chunk to oid 12 2024-03-19T11:30:05.696 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 12 set_chunk 12 low_tier returned error code -95 offset: 1676288 length: 491520 2024-03-19T11:30:05.696 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 12 v 0 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-03-19T11:30:05.696 INFO:tasks.rados.rados.0.smithi038.stdout:401: done (15 left) 2024-03-19T11:30:05.696 INFO:tasks.rados.rados.0.smithi038.stdout:405: done (14 left) 2024-03-19T11:30:05.696 INFO:tasks.rados.rados.0.smithi038.stdout:406: done (13 left) 2024-03-19T11:30:05.696 INFO:tasks.rados.rados.0.smithi038.stdout:408: done (12 left) 2024-03-19T11:30:05.696 INFO:tasks.rados.rados.0.smithi038.stdout:409: done (11 left) 2024-03-19T11:30:05.696 INFO:tasks.rados.rados.0.smithi038.stdout:411: done (10 left) 2024-03-19T11:30:05.696 INFO:tasks.rados.rados.0.smithi038.stdout:413: set_chunk oid 14 target oid 14 low_tier 2024-03-19T11:30:05.696 INFO:tasks.rados.rados.0.smithi038.stdout:413: set_chunk oid 14 offset: 91648 length: 509952 target oid smithi03829856-14 low_tier offset: 91648 snap -1 2024-03-19T11:30:05.697 INFO:tasks.rados.rados.0.smithi038.stdout:414: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.697 INFO:tasks.rados.rados.0.smithi038.stdout:414: set_chunk oid 15 offset: 18944 length: 1864704 target oid smithi03829856-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 18944 snap -1 2024-03-19T11:30:05.697 INFO:tasks.rados.rados.0.smithi038.stdout:415: set_chunk oid 16 target oid 16 low_tier 2024-03-19T11:30:05.697 INFO:tasks.rados.rados.0.smithi038.stdout:415: set_chunk oid 16 offset: 9728 length: 338944 target oid smithi03829856-16 low_tier offset: 9728 snap -1 2024-03-19T11:30:05.697 INFO:tasks.rados.rados.0.smithi038.stdout:416: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.697 INFO:tasks.rados.rados.0.smithi038.stdout:416: set_chunk oid 17 offset: 928768 length: 877056 target oid smithi03829856-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 928768 snap -1 2024-03-19T11:30:05.697 INFO:tasks.rados.rados.0.smithi038.stdout:417: set_chunk oid 18 target oid 18 low_tier 2024-03-19T11:30:05.697 INFO:tasks.rados.rados.0.smithi038.stdout:417: set_chunk oid 18 offset: 838656 length: 902144 target oid smithi03829856-18 low_tier offset: 838656 snap -1 2024-03-19T11:30:05.697 INFO:tasks.rados.rados.0.smithi038.stdout:418: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.697 INFO:tasks.rados.rados.0.smithi038.stdout:418: set_chunk oid 19 offset: 182272 length: 1555456 target oid smithi03829856-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 182272 snap -1 2024-03-19T11:30:05.697 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.697 INFO:tasks.rados.rados.0.smithi038.stdout:414: finishing set_chunk to oid 15 2024-03-19T11:30:05.697 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 18944 length: 1864704 2024-03-19T11:30:05.697 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-03-19T11:30:05.698 INFO:tasks.rados.rados.0.smithi038.stdout:414: done (15 left) 2024-03-19T11:30:05.698 INFO:tasks.rados.rados.0.smithi038.stdout:419: set_chunk oid 20 target oid 20 low_tier 2024-03-19T11:30:05.698 INFO:tasks.rados.rados.0.smithi038.stdout:419: set_chunk oid 20 offset: 917504 length: 275456 target oid smithi03829856-20 low_tier offset: 917504 snap -1 2024-03-19T11:30:05.698 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.698 INFO:tasks.rados.rados.0.smithi038.stdout:417: finishing set_chunk to oid 18 2024-03-19T11:30:05.698 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 18 set_chunk 18 low_tier returned error code -95 offset: 838656 length: 902144 2024-03-19T11:30:05.698 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 18 v 0 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-03-19T11:30:05.698 INFO:tasks.rados.rados.0.smithi038.stdout:412: finishing set_chunk to oid 13 2024-03-19T11:30:05.698 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 13 set_chunk 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 226304 length: 1228288 2024-03-19T11:30:05.698 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 13 v 0 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-03-19T11:30:05.698 INFO:tasks.rados.rados.0.smithi038.stdout:418: finishing set_chunk to oid 19 2024-03-19T11:30:05.698 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 19 set_chunk 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 182272 length: 1555456 2024-03-19T11:30:05.698 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 19 v 0 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-03-19T11:30:05.698 INFO:tasks.rados.rados.0.smithi038.stdout:416: finishing set_chunk to oid 17 2024-03-19T11:30:05.699 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 17 set_chunk 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 928768 length: 877056 2024-03-19T11:30:05.699 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 17 v 0 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-03-19T11:30:05.699 INFO:tasks.rados.rados.0.smithi038.stdout:412: done (15 left) 2024-03-19T11:30:05.699 INFO:tasks.rados.rados.0.smithi038.stdout:416: done (14 left) 2024-03-19T11:30:05.699 INFO:tasks.rados.rados.0.smithi038.stdout:417: done (13 left) 2024-03-19T11:30:05.699 INFO:tasks.rados.rados.0.smithi038.stdout:418: done (12 left) 2024-03-19T11:30:05.699 INFO:tasks.rados.rados.0.smithi038.stdout:420: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.699 INFO:tasks.rados.rados.0.smithi038.stdout:420: set_chunk oid 21 offset: 769536 length: 260608 target oid smithi03829856-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 769536 snap -1 2024-03-19T11:30:05.699 INFO:tasks.rados.rados.0.smithi038.stdout:419: finishing set_chunk to oid 20 2024-03-19T11:30:05.699 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 20 set_chunk 20 low_tier returned error code -95 offset: 917504 length: 275456 2024-03-19T11:30:05.699 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 20 v 0 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-03-19T11:30:05.699 INFO:tasks.rados.rados.0.smithi038.stdout:419: done (12 left) 2024-03-19T11:30:05.699 INFO:tasks.rados.rados.0.smithi038.stdout:421: set_chunk oid 22 target oid 22 low_tier 2024-03-19T11:30:05.700 INFO:tasks.rados.rados.0.smithi038.stdout:421: set_chunk oid 22 offset: 663552 length: 1048064 target oid smithi03829856-22 low_tier offset: 663552 snap -1 2024-03-19T11:30:05.700 INFO:tasks.rados.rados.0.smithi038.stdout:422: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.700 INFO:tasks.rados.rados.0.smithi038.stdout:422: set_chunk oid 23 offset: 1251328 length: 968192 target oid smithi03829856-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1251328 snap -1 2024-03-19T11:30:05.700 INFO:tasks.rados.rados.0.smithi038.stdout:423: set_chunk oid 24 target oid 24 low_tier 2024-03-19T11:30:05.700 INFO:tasks.rados.rados.0.smithi038.stdout:423: set_chunk oid 24 offset: 414720 length: 773120 target oid smithi03829856-24 low_tier offset: 414720 snap -1 2024-03-19T11:30:05.700 INFO:tasks.rados.rados.0.smithi038.stdout:424: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.700 INFO:tasks.rados.rados.0.smithi038.stdout:424: set_chunk oid 25 offset: 585216 length: 351232 target oid smithi03829856-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 585216 snap -1 2024-03-19T11:30:05.700 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.700 INFO:tasks.rados.rados.0.smithi038.stdout:378: finishing set_chunk to oid 29 2024-03-19T11:30:05.700 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 29 v 3 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-03-19T11:30:05.700 INFO:tasks.rados.rados.0.smithi038.stdout:421: finishing set_chunk to oid 22 2024-03-19T11:30:05.700 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 22 set_chunk 22 low_tier returned error code -95 offset: 663552 length: 1048064 2024-03-19T11:30:05.700 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-03-19T11:30:05.700 INFO:tasks.rados.rados.0.smithi038.stdout:420: finishing set_chunk to oid 21 2024-03-19T11:30:05.701 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 21 set_chunk 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 769536 length: 260608 2024-03-19T11:30:05.701 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-03-19T11:30:05.701 INFO:tasks.rados.rados.0.smithi038.stdout:378: done (15 left) 2024-03-19T11:30:05.701 INFO:tasks.rados.rados.0.smithi038.stdout:420: done (14 left) 2024-03-19T11:30:05.701 INFO:tasks.rados.rados.0.smithi038.stdout:421: done (13 left) 2024-03-19T11:30:05.701 INFO:tasks.rados.rados.0.smithi038.stdout:425: set_chunk oid 26 target oid 26 low_tier 2024-03-19T11:30:05.701 INFO:tasks.rados.rados.0.smithi038.stdout:425: set_chunk oid 26 offset: 552448 length: 211968 target oid smithi03829856-26 low_tier offset: 552448 snap -1 2024-03-19T11:30:05.701 INFO:tasks.rados.rados.0.smithi038.stdout:425 retry set_chunk ! 2024-03-19T11:30:05.701 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 14 2024-03-19T11:30:05.701 INFO:tasks.rados.rados.0.smithi038.stdout:423: finishing set_chunk to oid 24 2024-03-19T11:30:05.701 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 24 set_chunk 24 low_tier returned error code -95 offset: 414720 length: 773120 2024-03-19T11:30:05.701 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-03-19T11:30:05.701 INFO:tasks.rados.rados.0.smithi038.stdout:423: done (13 left) 2024-03-19T11:30:05.701 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 13 2024-03-19T11:30:05.702 INFO:tasks.rados.rados.0.smithi038.stdout:395: finishing set_chunk to oid 46 2024-03-19T11:30:05.702 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 46 v 12 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-03-19T11:30:05.702 INFO:tasks.rados.rados.0.smithi038.stdout:395: done (12 left) 2024-03-19T11:30:05.702 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 12 2024-03-19T11:30:05.702 INFO:tasks.rados.rados.0.smithi038.stdout:422: finishing set_chunk to oid 23 2024-03-19T11:30:05.702 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 23 set_chunk 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1251328 length: 968192 2024-03-19T11:30:05.702 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 23 v 0 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-03-19T11:30:05.702 INFO:tasks.rados.rados.0.smithi038.stdout:424: finishing set_chunk to oid 25 2024-03-19T11:30:05.702 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 25 set_chunk 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 585216 length: 351232 2024-03-19T11:30:05.702 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-03-19T11:30:05.702 INFO:tasks.rados.rados.0.smithi038.stdout:422: done (11 left) 2024-03-19T11:30:05.702 INFO:tasks.rados.rados.0.smithi038.stdout:424: done (10 left) 2024-03-19T11:30:05.702 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 10 2024-03-19T11:30:05.702 INFO:tasks.rados.rados.0.smithi038.stdout:384: finishing set_chunk to oid 35 2024-03-19T11:30:05.702 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 35 v 18 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-03-19T11:30:05.703 INFO:tasks.rados.rados.0.smithi038.stdout:384: done (9 left) 2024-03-19T11:30:05.703 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 9 2024-03-19T11:30:05.703 INFO:tasks.rados.rados.0.smithi038.stdout:425: finishing set_chunk to oid 26 2024-03-19T11:30:05.703 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 26 set_chunk 26 low_tier returned error code -95 offset: 552448 length: 211968 2024-03-19T11:30:05.703 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-03-19T11:30:05.703 INFO:tasks.rados.rados.0.smithi038.stdout:425: done (8 left) 2024-03-19T11:30:05.703 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 8 2024-03-19T11:30:05.703 INFO:tasks.rados.rados.0.smithi038.stdout:376: finishing set_chunk to oid 27 2024-03-19T11:30:05.703 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 27 v 6 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-03-19T11:30:05.703 INFO:tasks.rados.rados.0.smithi038.stdout:376: done (7 left) 2024-03-19T11:30:05.703 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 7 2024-03-19T11:30:05.703 INFO:tasks.rados.rados.0.smithi038.stdout:379: finishing set_chunk to oid 30 2024-03-19T11:30:05.703 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 30 v 6 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-03-19T11:30:05.703 INFO:tasks.rados.rados.0.smithi038.stdout:379: done (6 left) 2024-03-19T11:30:05.703 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 6 2024-03-19T11:30:05.704 INFO:tasks.rados.rados.0.smithi038.stdout:393: finishing set_chunk to oid 44 2024-03-19T11:30:05.704 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 44 v 9 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-03-19T11:30:05.704 INFO:tasks.rados.rados.0.smithi038.stdout:393: done (5 left) 2024-03-19T11:30:05.704 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 5 2024-03-19T11:30:05.704 INFO:tasks.rados.rados.0.smithi038.stdout:380: finishing set_chunk to oid 31 2024-03-19T11:30:05.704 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 31 v 6 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-03-19T11:30:05.704 INFO:tasks.rados.rados.0.smithi038.stdout:380: done (4 left) 2024-03-19T11:30:05.704 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 4 2024-03-19T11:30:05.704 INFO:tasks.rados.rados.0.smithi038.stdout:410: finishing set_chunk to oid 11 2024-03-19T11:30:05.704 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 11 v 3 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-03-19T11:30:05.704 INFO:tasks.rados.rados.0.smithi038.stdout:410: done (3 left) 2024-03-19T11:30:05.704 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 3 2024-03-19T11:30:05.704 INFO:tasks.rados.rados.0.smithi038.stdout:407: finishing set_chunk to oid 8 2024-03-19T11:30:05.704 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 8 v 3 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-03-19T11:30:05.704 INFO:tasks.rados.rados.0.smithi038.stdout:407: done (2 left) 2024-03-19T11:30:05.705 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 2 2024-03-19T11:30:05.705 INFO:tasks.rados.rados.0.smithi038.stdout:413: finishing set_chunk to oid 14 2024-03-19T11:30:05.705 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 14 v 3 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-03-19T11:30:05.705 INFO:tasks.rados.rados.0.smithi038.stdout:413: done (1 left) 2024-03-19T11:30:05.705 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 1 2024-03-19T11:30:05.705 INFO:tasks.rados.rados.0.smithi038.stdout:415: finishing set_chunk to oid 16 2024-03-19T11:30:05.705 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 16 v 6 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-03-19T11:30:05.705 INFO:tasks.rados.rados.0.smithi038.stdout:415: done (0 left) 2024-03-19T11:30:05.705 INFO:tasks.rados.rados.0.smithi038.stdout:426: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.705 INFO:tasks.rados.rados.0.smithi038.stdout:426: set_chunk oid 27 offset: 391680 length: 351744 target oid smithi03829856-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 391680 snap -1 2024-03-19T11:30:05.705 INFO:tasks.rados.rados.0.smithi038.stdout:427: set_chunk oid 28 target oid 28 low_tier 2024-03-19T11:30:05.705 INFO:tasks.rados.rados.0.smithi038.stdout:427: set_chunk oid 28 offset: 779264 length: 1333248 target oid smithi03829856-28 low_tier offset: 779264 snap -1 2024-03-19T11:30:05.705 INFO:tasks.rados.rados.0.smithi038.stdout:428: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.705 INFO:tasks.rados.rados.0.smithi038.stdout:428: set_chunk oid 29 offset: 2085376 length: 214528 target oid smithi03829856-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 2085376 snap -1 2024-03-19T11:30:05.705 INFO:tasks.rados.rados.0.smithi038.stdout:429: set_chunk oid 30 target oid 30 low_tier 2024-03-19T11:30:05.706 INFO:tasks.rados.rados.0.smithi038.stdout:429: set_chunk oid 30 offset: 726016 length: 657920 target oid smithi03829856-30 low_tier offset: 726016 snap -1 2024-03-19T11:30:05.706 INFO:tasks.rados.rados.0.smithi038.stdout:430: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.706 INFO:tasks.rados.rados.0.smithi038.stdout:430: set_chunk oid 31 offset: 380416 length: 1284608 target oid smithi03829856-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 380416 snap -1 2024-03-19T11:30:05.706 INFO:tasks.rados.rados.0.smithi038.stdout:431: set_chunk oid 32 target oid 32 low_tier 2024-03-19T11:30:05.706 INFO:tasks.rados.rados.0.smithi038.stdout:431: set_chunk oid 32 offset: 980480 length: 1033728 target oid smithi03829856-32 low_tier offset: 980480 snap -1 2024-03-19T11:30:05.706 INFO:tasks.rados.rados.0.smithi038.stdout:432: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.706 INFO:tasks.rados.rados.0.smithi038.stdout:432: set_chunk oid 33 offset: 142848 length: 1862144 target oid smithi03829856-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 142848 snap -1 2024-03-19T11:30:05.706 INFO:tasks.rados.rados.0.smithi038.stdout:433: set_chunk oid 34 target oid 34 low_tier 2024-03-19T11:30:05.706 INFO:tasks.rados.rados.0.smithi038.stdout:433: set_chunk oid 34 offset: 1022976 length: 26112 target oid smithi03829856-34 low_tier offset: 1022976 snap -1 2024-03-19T11:30:05.706 INFO:tasks.rados.rados.0.smithi038.stdout:434: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.706 INFO:tasks.rados.rados.0.smithi038.stdout:434: set_chunk oid 35 offset: 1101312 length: 599552 target oid smithi03829856-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1101312 snap -1 2024-03-19T11:30:05.706 INFO:tasks.rados.rados.0.smithi038.stdout:435: set_chunk oid 36 target oid 36 low_tier 2024-03-19T11:30:05.706 INFO:tasks.rados.rados.0.smithi038.stdout:435: set_chunk oid 36 offset: 2266624 length: 51200 target oid smithi03829856-36 low_tier offset: 2266624 snap -1 2024-03-19T11:30:05.706 INFO:tasks.rados.rados.0.smithi038.stdout:436: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.707 INFO:tasks.rados.rados.0.smithi038.stdout:436: set_chunk oid 37 offset: 2560 length: 1021952 target oid smithi03829856-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 2560 snap -1 2024-03-19T11:30:05.707 INFO:tasks.rados.rados.0.smithi038.stdout:437: set_chunk oid 38 target oid 38 low_tier 2024-03-19T11:30:05.707 INFO:tasks.rados.rados.0.smithi038.stdout:437: set_chunk oid 38 offset: 2048 length: 2169856 target oid smithi03829856-38 low_tier offset: 2048 snap -1 2024-03-19T11:30:05.707 INFO:tasks.rados.rados.0.smithi038.stdout:438: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.707 INFO:tasks.rados.rados.0.smithi038.stdout:438: set_chunk oid 39 offset: 94208 length: 2073600 target oid smithi03829856-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 94208 snap -1 2024-03-19T11:30:05.707 INFO:tasks.rados.rados.0.smithi038.stdout:439: set_chunk oid 40 target oid 40 low_tier 2024-03-19T11:30:05.707 INFO:tasks.rados.rados.0.smithi038.stdout:439: set_chunk oid 40 offset: 663552 length: 1711616 target oid smithi03829856-40 low_tier offset: 663552 snap -1 2024-03-19T11:30:05.707 INFO:tasks.rados.rados.0.smithi038.stdout:440: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.707 INFO:tasks.rados.rados.0.smithi038.stdout:440: set_chunk oid 41 offset: 606720 length: 1874944 target oid smithi03829856-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 606720 snap -1 2024-03-19T11:30:05.707 INFO:tasks.rados.rados.0.smithi038.stdout:441: set_chunk oid 42 target oid 42 low_tier 2024-03-19T11:30:05.707 INFO:tasks.rados.rados.0.smithi038.stdout:441: set_chunk oid 42 offset: 1140736 length: 1138688 target oid smithi03829856-42 low_tier offset: 1140736 snap -1 2024-03-19T11:30:05.707 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.707 INFO:tasks.rados.rados.0.smithi038.stdout:431: finishing set_chunk to oid 32 2024-03-19T11:30:05.707 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 32 set_chunk 32 low_tier returned error code -95 offset: 980480 length: 1033728 2024-03-19T11:30:05.708 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-03-19T11:30:05.708 INFO:tasks.rados.rados.0.smithi038.stdout:431: done (15 left) 2024-03-19T11:30:05.708 INFO:tasks.rados.rados.0.smithi038.stdout:442: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.708 INFO:tasks.rados.rados.0.smithi038.stdout:442: set_chunk oid 43 offset: 1137152 length: 1016832 target oid smithi03829856-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1137152 snap -1 2024-03-19T11:30:05.708 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.708 INFO:tasks.rados.rados.0.smithi038.stdout:427: finishing set_chunk to oid 28 2024-03-19T11:30:05.708 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 28 set_chunk 28 low_tier returned error code -95 offset: 779264 length: 1333248 2024-03-19T11:30:05.708 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 28 v 0 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-03-19T11:30:05.708 INFO:tasks.rados.rados.0.smithi038.stdout:437: finishing set_chunk to oid 38 2024-03-19T11:30:05.708 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 38 set_chunk 38 low_tier returned error code -95 offset: 2048 length: 2169856 2024-03-19T11:30:05.708 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 38 v 0 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-03-19T11:30:05.708 INFO:tasks.rados.rados.0.smithi038.stdout:426: finishing set_chunk to oid 27 2024-03-19T11:30:05.708 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 27 set_chunk 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 391680 length: 351744 2024-03-19T11:30:05.708 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-03-19T11:30:05.708 INFO:tasks.rados.rados.0.smithi038.stdout:434: finishing set_chunk to oid 35 2024-03-19T11:30:05.709 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 35 set_chunk 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1101312 length: 599552 2024-03-19T11:30:05.709 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 35 v 0 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-03-19T11:30:05.709 INFO:tasks.rados.rados.0.smithi038.stdout:429: finishing set_chunk to oid 30 2024-03-19T11:30:05.709 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 30 set_chunk 30 low_tier returned error code -95 offset: 726016 length: 657920 2024-03-19T11:30:05.709 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 30 v 0 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-03-19T11:30:05.709 INFO:tasks.rados.rados.0.smithi038.stdout:435: finishing set_chunk to oid 36 2024-03-19T11:30:05.709 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 36 set_chunk 36 low_tier returned error code -95 offset: 2266624 length: 51200 2024-03-19T11:30:05.709 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-03-19T11:30:05.710 INFO:tasks.rados.rados.0.smithi038.stdout:440: finishing set_chunk to oid 41 2024-03-19T11:30:05.710 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 41 set_chunk 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 606720 length: 1874944 2024-03-19T11:30:05.710 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 41 v 0 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-03-19T11:30:05.710 INFO:tasks.rados.rados.0.smithi038.stdout:433: finishing set_chunk to oid 34 2024-03-19T11:30:05.710 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 34 set_chunk 34 low_tier returned error code -95 offset: 1022976 length: 26112 2024-03-19T11:30:05.710 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 34 v 0 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-03-19T11:30:05.710 INFO:tasks.rados.rados.0.smithi038.stdout:438: finishing set_chunk to oid 39 2024-03-19T11:30:05.710 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 39 set_chunk 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 94208 length: 2073600 2024-03-19T11:30:05.710 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 39 v 0 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-03-19T11:30:05.710 INFO:tasks.rados.rados.0.smithi038.stdout:426: done (15 left) 2024-03-19T11:30:05.710 INFO:tasks.rados.rados.0.smithi038.stdout:427: done (14 left) 2024-03-19T11:30:05.710 INFO:tasks.rados.rados.0.smithi038.stdout:429: done (13 left) 2024-03-19T11:30:05.710 INFO:tasks.rados.rados.0.smithi038.stdout:433: done (12 left) 2024-03-19T11:30:05.710 INFO:tasks.rados.rados.0.smithi038.stdout:434: done (11 left) 2024-03-19T11:30:05.710 INFO:tasks.rados.rados.0.smithi038.stdout:435: done (10 left) 2024-03-19T11:30:05.711 INFO:tasks.rados.rados.0.smithi038.stdout:437: done (9 left) 2024-03-19T11:30:05.711 INFO:tasks.rados.rados.0.smithi038.stdout:438: done (8 left) 2024-03-19T11:30:05.711 INFO:tasks.rados.rados.0.smithi038.stdout:440: done (7 left) 2024-03-19T11:30:05.711 INFO:tasks.rados.rados.0.smithi038.stdout:443: set_chunk oid 44 target oid 44 low_tier 2024-03-19T11:30:05.711 INFO:tasks.rados.rados.0.smithi038.stdout:443: set_chunk oid 44 offset: 834048 length: 1708032 target oid smithi03829856-44 low_tier offset: 834048 snap -1 2024-03-19T11:30:05.711 INFO:tasks.rados.rados.0.smithi038.stdout:444: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.711 INFO:tasks.rados.rados.0.smithi038.stdout:444: set_chunk oid 45 offset: 886272 length: 123392 target oid smithi03829856-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 886272 snap -1 2024-03-19T11:30:05.711 INFO:tasks.rados.rados.0.smithi038.stdout:445: set_chunk oid 46 target oid 46 low_tier 2024-03-19T11:30:05.711 INFO:tasks.rados.rados.0.smithi038.stdout:445: set_chunk oid 46 offset: 109056 length: 210432 target oid smithi03829856-46 low_tier offset: 109056 snap -1 2024-03-19T11:30:05.711 INFO:tasks.rados.rados.0.smithi038.stdout:446: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.711 INFO:tasks.rados.rados.0.smithi038.stdout:446: set_chunk oid 47 offset: 737792 length: 1857536 target oid smithi03829856-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 737792 snap -1 2024-03-19T11:30:05.711 INFO:tasks.rados.rados.0.smithi038.stdout:447: set_chunk oid 48 target oid 48 low_tier 2024-03-19T11:30:05.711 INFO:tasks.rados.rados.0.smithi038.stdout:447: set_chunk oid 48 offset: 463360 length: 1422336 target oid smithi03829856-48 low_tier offset: 463360 snap -1 2024-03-19T11:30:05.711 INFO:tasks.rados.rados.0.smithi038.stdout:448: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.712 INFO:tasks.rados.rados.0.smithi038.stdout:448: set_chunk oid 49 offset: 59904 length: 827392 target oid smithi03829856-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 59904 snap -1 2024-03-19T11:30:05.712 INFO:tasks.rados.rados.0.smithi038.stdout:449: set_chunk oid 50 target oid 50 low_tier 2024-03-19T11:30:05.712 INFO:tasks.rados.rados.0.smithi038.stdout:449: set_chunk oid 50 offset: 1018368 length: 834560 target oid smithi03829856-50 low_tier offset: 1018368 snap -1 2024-03-19T11:30:05.712 INFO:tasks.rados.rados.0.smithi038.stdout:450: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.712 INFO:tasks.rados.rados.0.smithi038.stdout:450: set_chunk oid 1 offset: 73728 length: 430080 target oid smithi03829856-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 73728 snap -1 2024-03-19T11:30:05.712 INFO:tasks.rados.rados.0.smithi038.stdout:451: set_chunk oid 2 target oid 2 low_tier 2024-03-19T11:30:05.712 INFO:tasks.rados.rados.0.smithi038.stdout:428: finishing set_chunk to oid 29 2024-03-19T11:30:05.712 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 29 set_chunk 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 2085376 length: 214528 2024-03-19T11:30:05.712 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 29 v 0 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-03-19T11:30:05.712 INFO:tasks.rados.rados.0.smithi038.stdout:439: finishing set_chunk to oid 40 2024-03-19T11:30:05.712 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 40 set_chunk 40 low_tier returned error code -95 offset: 663552 length: 1711616 2024-03-19T11:30:05.712 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 40 v 0 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-03-19T11:30:05.712 INFO:tasks.rados.rados.0.smithi038.stdout:441: finishing set_chunk to oid 42 2024-03-19T11:30:05.712 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 42 set_chunk 42 low_tier returned error code -95 offset: 1140736 length: 1138688 2024-03-19T11:30:05.713 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-03-19T11:30:05.713 INFO:tasks.rados.rados.0.smithi038.stdout:436: finishing set_chunk to oid 37 2024-03-19T11:30:05.713 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 37 set_chunk 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 2560 length: 1021952 2024-03-19T11:30:05.713 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 37 v 0 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-03-19T11:30:05.713 INFO:tasks.rados.rados.0.smithi038.stdout:432: finishing set_chunk to oid 33 2024-03-19T11:30:05.713 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 33 set_chunk 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 142848 length: 1862144 2024-03-19T11:30:05.713 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-03-19T11:30:05.713 INFO:tasks.rados.rados.0.smithi038.stdout:430: finishing set_chunk to oid 31 2024-03-19T11:30:05.713 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 31 set_chunk 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 380416 length: 1284608 2024-03-19T11:30:05.713 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 31 v 0 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-03-19T11:30:05.713 INFO:tasks.rados.rados.0.smithi038.stdout:442: finishing set_chunk to oid 43 2024-03-19T11:30:05.713 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 43 set_chunk 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1137152 length: 1016832 2024-03-19T11:30:05.713 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 43 v 0 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-03-19T11:30:05.713 INFO:tasks.rados.rados.0.smithi038.stdout:451: set_chunk oid 2 offset: 578560 length: 976384 target oid smithi03829856-2 low_tier offset: 578560 snap -1 2024-03-19T11:30:05.714 INFO:tasks.rados.rados.0.smithi038.stdout:428: done (15 left) 2024-03-19T11:30:05.714 INFO:tasks.rados.rados.0.smithi038.stdout:430: done (14 left) 2024-03-19T11:30:05.714 INFO:tasks.rados.rados.0.smithi038.stdout:432: done (13 left) 2024-03-19T11:30:05.714 INFO:tasks.rados.rados.0.smithi038.stdout:436: done (12 left) 2024-03-19T11:30:05.714 INFO:tasks.rados.rados.0.smithi038.stdout:439: done (11 left) 2024-03-19T11:30:05.714 INFO:tasks.rados.rados.0.smithi038.stdout:441: done (10 left) 2024-03-19T11:30:05.714 INFO:tasks.rados.rados.0.smithi038.stdout:442: done (9 left) 2024-03-19T11:30:05.714 INFO:tasks.rados.rados.0.smithi038.stdout:452: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.714 INFO:tasks.rados.rados.0.smithi038.stdout:452: set_chunk oid 3 offset: 1466368 length: 263168 target oid smithi03829856-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1466368 snap -1 2024-03-19T11:30:05.714 INFO:tasks.rados.rados.0.smithi038.stdout:453: set_chunk oid 4 target oid 4 low_tier 2024-03-19T11:30:05.714 INFO:tasks.rados.rados.0.smithi038.stdout:453: set_chunk oid 4 offset: 365568 length: 885248 target oid smithi03829856-4 low_tier offset: 365568 snap -1 2024-03-19T11:30:05.714 INFO:tasks.rados.rados.0.smithi038.stdout:454: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.714 INFO:tasks.rados.rados.0.smithi038.stdout:454: set_chunk oid 5 offset: 35328 length: 1195008 target oid smithi03829856-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 35328 snap -1 2024-03-19T11:30:05.714 INFO:tasks.rados.rados.0.smithi038.stdout:455: set_chunk oid 6 target oid 6 low_tier 2024-03-19T11:30:05.714 INFO:tasks.rados.rados.0.smithi038.stdout:455: set_chunk oid 6 offset: 1081344 length: 325632 target oid smithi03829856-6 low_tier offset: 1081344 snap -1 2024-03-19T11:30:05.715 INFO:tasks.rados.rados.0.smithi038.stdout:445: finishing set_chunk to oid 46 2024-03-19T11:30:05.715 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 46 set_chunk 46 low_tier returned error code -95 offset: 109056 length: 210432 2024-03-19T11:30:05.715 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-03-19T11:30:05.715 INFO:tasks.rados.rados.0.smithi038.stdout:443: finishing set_chunk to oid 44 2024-03-19T11:30:05.715 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 44 set_chunk 44 low_tier returned error code -95 offset: 834048 length: 1708032 2024-03-19T11:30:05.715 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 44 v 0 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-03-19T11:30:05.715 INFO:tasks.rados.rados.0.smithi038.stdout:443: done (12 left) 2024-03-19T11:30:05.715 INFO:tasks.rados.rados.0.smithi038.stdout:445: done (11 left) 2024-03-19T11:30:05.715 INFO:tasks.rados.rados.0.smithi038.stdout:456: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.715 INFO:tasks.rados.rados.0.smithi038.stdout:456: set_chunk oid 7 offset: 1068544 length: 391680 target oid smithi03829856-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1068544 snap -1 2024-03-19T11:30:05.715 INFO:tasks.rados.rados.0.smithi038.stdout:457: set_chunk oid 8 target oid 8 low_tier 2024-03-19T11:30:05.715 INFO:tasks.rados.rados.0.smithi038.stdout:457: set_chunk oid 8 offset: 22528 length: 497664 target oid smithi03829856-8 low_tier offset: 22528 snap -1 2024-03-19T11:30:05.715 INFO:tasks.rados.rados.0.smithi038.stdout:458: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.715 INFO:tasks.rados.rados.0.smithi038.stdout:458: set_chunk oid 9 offset: 1081344 length: 212480 target oid smithi03829856-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1081344 snap -1 2024-03-19T11:30:05.716 INFO:tasks.rados.rados.0.smithi038.stdout:446: finishing set_chunk to oid 47 2024-03-19T11:30:05.716 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 47 set_chunk 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 737792 length: 1857536 2024-03-19T11:30:05.716 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-03-19T11:30:05.716 INFO:tasks.rados.rados.0.smithi038.stdout:447: finishing set_chunk to oid 48 2024-03-19T11:30:05.716 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 48 set_chunk 48 low_tier returned error code -95 offset: 463360 length: 1422336 2024-03-19T11:30:05.716 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 48 v 0 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-03-19T11:30:05.716 INFO:tasks.rados.rados.0.smithi038.stdout:446: done (13 left) 2024-03-19T11:30:05.716 INFO:tasks.rados.rados.0.smithi038.stdout:447: done (12 left) 2024-03-19T11:30:05.716 INFO:tasks.rados.rados.0.smithi038.stdout:459: set_chunk oid 10 target oid 10 low_tier 2024-03-19T11:30:05.716 INFO:tasks.rados.rados.0.smithi038.stdout:459: set_chunk oid 10 offset: 23040 length: 1396224 target oid smithi03829856-10 low_tier offset: 23040 snap -1 2024-03-19T11:30:05.716 INFO:tasks.rados.rados.0.smithi038.stdout:449: finishing set_chunk to oid 50 2024-03-19T11:30:05.716 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 50 set_chunk 50 low_tier returned error code -95 offset: 1018368 length: 834560 2024-03-19T11:30:05.716 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 50 v 0 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-03-19T11:30:05.716 INFO:tasks.rados.rados.0.smithi038.stdout:451: finishing set_chunk to oid 2 2024-03-19T11:30:05.716 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 2 set_chunk 2 low_tier returned error code -95 offset: 578560 length: 976384 2024-03-19T11:30:05.717 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-03-19T11:30:05.717 INFO:tasks.rados.rados.0.smithi038.stdout:449: done (12 left) 2024-03-19T11:30:05.717 INFO:tasks.rados.rados.0.smithi038.stdout:451: done (11 left) 2024-03-19T11:30:05.717 INFO:tasks.rados.rados.0.smithi038.stdout:460: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.717 INFO:tasks.rados.rados.0.smithi038.stdout:460: set_chunk oid 11 offset: 225280 length: 800768 target oid smithi03829856-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 225280 snap -1 2024-03-19T11:30:05.717 INFO:tasks.rados.rados.0.smithi038.stdout:461: set_chunk oid 12 target oid 12 low_tier 2024-03-19T11:30:05.717 INFO:tasks.rados.rados.0.smithi038.stdout:452: finishing set_chunk to oid 3 2024-03-19T11:30:05.717 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 3 set_chunk 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1466368 length: 263168 2024-03-19T11:30:05.717 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 3 v 0 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-03-19T11:30:05.717 INFO:tasks.rados.rados.0.smithi038.stdout:461: set_chunk oid 12 offset: 1813504 length: 150528 target oid smithi03829856-12 low_tier offset: 1813504 snap -1 2024-03-19T11:30:05.717 INFO:tasks.rados.rados.0.smithi038.stdout:452: done (12 left) 2024-03-19T11:30:05.717 INFO:tasks.rados.rados.0.smithi038.stdout:462: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.717 INFO:tasks.rados.rados.0.smithi038.stdout:454: finishing set_chunk to oid 5 2024-03-19T11:30:05.717 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 5 set_chunk 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 35328 length: 1195008 2024-03-19T11:30:05.718 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-03-19T11:30:05.718 INFO:tasks.rados.rados.0.smithi038.stdout:458: finishing set_chunk to oid 9 2024-03-19T11:30:05.718 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 9 set_chunk 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1081344 length: 212480 2024-03-19T11:30:05.718 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 9 v 0 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-03-19T11:30:05.718 INFO:tasks.rados.rados.0.smithi038.stdout:462: set_chunk oid 13 offset: 2074624 length: 73216 target oid smithi03829856-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 2074624 snap -1 2024-03-19T11:30:05.718 INFO:tasks.rados.rados.0.smithi038.stdout:454: done (12 left) 2024-03-19T11:30:05.718 INFO:tasks.rados.rados.0.smithi038.stdout:458: done (11 left) 2024-03-19T11:30:05.718 INFO:tasks.rados.rados.0.smithi038.stdout:463: set_chunk oid 14 target oid 14 low_tier 2024-03-19T11:30:05.718 INFO:tasks.rados.rados.0.smithi038.stdout:463: set_chunk oid 14 offset: 68608 length: 506368 target oid smithi03829856-14 low_tier offset: 68608 snap -1 2024-03-19T11:30:05.718 INFO:tasks.rados.rados.0.smithi038.stdout:464: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.718 INFO:tasks.rados.rados.0.smithi038.stdout:464: set_chunk oid 15 offset: 191488 length: 1819136 target oid smithi03829856-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 191488 snap -1 2024-03-19T11:30:05.718 INFO:tasks.rados.rados.0.smithi038.stdout:465: set_chunk oid 16 target oid 16 low_tier 2024-03-19T11:30:05.718 INFO:tasks.rados.rados.0.smithi038.stdout:460: finishing set_chunk to oid 11 2024-03-19T11:30:05.718 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 11 set_chunk 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 225280 length: 800768 2024-03-19T11:30:05.718 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 11 v 0 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-03-19T11:30:05.719 INFO:tasks.rados.rados.0.smithi038.stdout:465: set_chunk oid 16 offset: 340992 length: 677376 target oid smithi03829856-16 low_tier offset: 340992 snap -1 2024-03-19T11:30:05.719 INFO:tasks.rados.rados.0.smithi038.stdout:460: done (13 left) 2024-03-19T11:30:05.719 INFO:tasks.rados.rados.0.smithi038.stdout:466: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.719 INFO:tasks.rados.rados.0.smithi038.stdout:466: set_chunk oid 17 offset: 1362944 length: 361472 target oid smithi03829856-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1362944 snap -1 2024-03-19T11:30:05.719 INFO:tasks.rados.rados.0.smithi038.stdout:467: set_chunk oid 18 target oid 18 low_tier 2024-03-19T11:30:05.719 INFO:tasks.rados.rados.0.smithi038.stdout:467: set_chunk oid 18 offset: 1001472 length: 896512 target oid smithi03829856-18 low_tier offset: 1001472 snap -1 2024-03-19T11:30:05.719 INFO:tasks.rados.rados.0.smithi038.stdout:468: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.719 INFO:tasks.rados.rados.0.smithi038.stdout:468: set_chunk oid 19 offset: 410624 length: 1316352 target oid smithi03829856-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 410624 snap -1 2024-03-19T11:30:05.719 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.719 INFO:tasks.rados.rados.0.smithi038.stdout:453: finishing set_chunk to oid 4 2024-03-19T11:30:05.719 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 4 set_chunk 4 low_tier returned error code -95 offset: 365568 length: 885248 2024-03-19T11:30:05.719 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-03-19T11:30:05.719 INFO:tasks.rados.rados.0.smithi038.stdout:453: done (15 left) 2024-03-19T11:30:05.719 INFO:tasks.rados.rados.0.smithi038.stdout:469: set_chunk oid 20 target oid 20 low_tier 2024-03-19T11:30:05.720 INFO:tasks.rados.rados.0.smithi038.stdout:469: set_chunk oid 20 offset: 1988608 length: 222720 target oid smithi03829856-20 low_tier offset: 1988608 snap -1 2024-03-19T11:30:05.720 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.720 INFO:tasks.rados.rados.0.smithi038.stdout:455: finishing set_chunk to oid 6 2024-03-19T11:30:05.720 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 6 set_chunk 6 low_tier returned error code -95 offset: 1081344 length: 325632 2024-03-19T11:30:05.720 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-03-19T11:30:05.720 INFO:tasks.rados.rados.0.smithi038.stdout:456: finishing set_chunk to oid 7 2024-03-19T11:30:05.720 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 7 set_chunk 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1068544 length: 391680 2024-03-19T11:30:05.720 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-03-19T11:30:05.720 INFO:tasks.rados.rados.0.smithi038.stdout:459: finishing set_chunk to oid 10 2024-03-19T11:30:05.720 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 10 set_chunk 10 low_tier returned error code -95 offset: 23040 length: 1396224 2024-03-19T11:30:05.720 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 10 v 0 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-03-19T11:30:05.720 INFO:tasks.rados.rados.0.smithi038.stdout:455: done (15 left) 2024-03-19T11:30:05.720 INFO:tasks.rados.rados.0.smithi038.stdout:456: done (14 left) 2024-03-19T11:30:05.720 INFO:tasks.rados.rados.0.smithi038.stdout:459: done (13 left) 2024-03-19T11:30:05.721 INFO:tasks.rados.rados.0.smithi038.stdout:470: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.721 INFO:tasks.rados.rados.0.smithi038.stdout:457: finishing set_chunk to oid 8 2024-03-19T11:30:05.721 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 8 set_chunk 8 low_tier returned error code -95 offset: 22528 length: 497664 2024-03-19T11:30:05.721 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-03-19T11:30:05.721 INFO:tasks.rados.rados.0.smithi038.stdout:470: set_chunk oid 21 offset: 404992 length: 1629696 target oid smithi03829856-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 404992 snap -1 2024-03-19T11:30:05.721 INFO:tasks.rados.rados.0.smithi038.stdout:457: done (13 left) 2024-03-19T11:30:05.721 INFO:tasks.rados.rados.0.smithi038.stdout:471: set_chunk oid 22 target oid 22 low_tier 2024-03-19T11:30:05.721 INFO:tasks.rados.rados.0.smithi038.stdout:471: set_chunk oid 22 offset: 220672 length: 1533440 target oid smithi03829856-22 low_tier offset: 220672 snap -1 2024-03-19T11:30:05.721 INFO:tasks.rados.rados.0.smithi038.stdout:472: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.721 INFO:tasks.rados.rados.0.smithi038.stdout:472: set_chunk oid 23 offset: 67584 length: 328704 target oid smithi03829856-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 67584 snap -1 2024-03-19T11:30:05.721 INFO:tasks.rados.rados.0.smithi038.stdout:473: set_chunk oid 24 target oid 24 low_tier 2024-03-19T11:30:05.721 INFO:tasks.rados.rados.0.smithi038.stdout:473: set_chunk oid 24 offset: 731136 length: 30720 target oid smithi03829856-24 low_tier offset: 731136 snap -1 2024-03-19T11:30:05.721 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.721 INFO:tasks.rados.rados.0.smithi038.stdout:467: finishing set_chunk to oid 18 2024-03-19T11:30:05.721 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 18 set_chunk 18 low_tier returned error code -95 offset: 1001472 length: 896512 2024-03-19T11:30:05.722 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 18 v 0 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-03-19T11:30:05.722 INFO:tasks.rados.rados.0.smithi038.stdout:467: done (15 left) 2024-03-19T11:30:05.722 INFO:tasks.rados.rados.0.smithi038.stdout:474: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.722 INFO:tasks.rados.rados.0.smithi038.stdout:474: set_chunk oid 25 offset: 1807872 length: 75264 target oid smithi03829856-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1807872 snap -1 2024-03-19T11:30:05.722 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.722 INFO:tasks.rados.rados.0.smithi038.stdout:468: finishing set_chunk to oid 19 2024-03-19T11:30:05.722 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 19 set_chunk 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 410624 length: 1316352 2024-03-19T11:30:05.722 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 19 v 0 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-03-19T11:30:05.722 INFO:tasks.rados.rados.0.smithi038.stdout:468: done (15 left) 2024-03-19T11:30:05.722 INFO:tasks.rados.rados.0.smithi038.stdout:475: set_chunk oid 26 target oid 26 low_tier 2024-03-19T11:30:05.722 INFO:tasks.rados.rados.0.smithi038.stdout:475: set_chunk oid 26 offset: 1496576 length: 917504 target oid smithi03829856-26 low_tier offset: 1496576 snap -1 2024-03-19T11:30:05.722 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.722 INFO:tasks.rados.rados.0.smithi038.stdout:466: finishing set_chunk to oid 17 2024-03-19T11:30:05.722 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 17 set_chunk 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1362944 length: 361472 2024-03-19T11:30:05.723 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 17 v 0 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-03-19T11:30:05.723 INFO:tasks.rados.rados.0.smithi038.stdout:464: finishing set_chunk to oid 15 2024-03-19T11:30:05.723 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 191488 length: 1819136 2024-03-19T11:30:05.723 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-03-19T11:30:05.723 INFO:tasks.rados.rados.0.smithi038.stdout:473: finishing set_chunk to oid 24 2024-03-19T11:30:05.723 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 24 set_chunk 24 low_tier returned error code -95 offset: 731136 length: 30720 2024-03-19T11:30:05.723 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-03-19T11:30:05.723 INFO:tasks.rados.rados.0.smithi038.stdout:474: finishing set_chunk to oid 25 2024-03-19T11:30:05.723 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 25 set_chunk 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1807872 length: 75264 2024-03-19T11:30:05.723 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-03-19T11:30:05.723 INFO:tasks.rados.rados.0.smithi038.stdout:463: finishing set_chunk to oid 14 2024-03-19T11:30:05.723 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 14 set_chunk 14 low_tier returned error code -95 offset: 68608 length: 506368 2024-03-19T11:30:05.723 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 14 v 0 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-03-19T11:30:05.723 INFO:tasks.rados.rados.0.smithi038.stdout:465: finishing set_chunk to oid 16 2024-03-19T11:30:05.723 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 16 set_chunk 16 low_tier returned error code -95 offset: 340992 length: 677376 2024-03-19T11:30:05.724 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-03-19T11:30:05.724 INFO:tasks.rados.rados.0.smithi038.stdout:471: finishing set_chunk to oid 22 2024-03-19T11:30:05.724 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 22 set_chunk 22 low_tier returned error code -95 offset: 220672 length: 1533440 2024-03-19T11:30:05.724 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-03-19T11:30:05.724 INFO:tasks.rados.rados.0.smithi038.stdout:470: finishing set_chunk to oid 21 2024-03-19T11:30:05.724 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 21 set_chunk 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 404992 length: 1629696 2024-03-19T11:30:05.724 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-03-19T11:30:05.724 INFO:tasks.rados.rados.0.smithi038.stdout:463: done (15 left) 2024-03-19T11:30:05.724 INFO:tasks.rados.rados.0.smithi038.stdout:464: done (14 left) 2024-03-19T11:30:05.724 INFO:tasks.rados.rados.0.smithi038.stdout:465: done (13 left) 2024-03-19T11:30:05.724 INFO:tasks.rados.rados.0.smithi038.stdout:466: done (12 left) 2024-03-19T11:30:05.724 INFO:tasks.rados.rados.0.smithi038.stdout:470: done (11 left) 2024-03-19T11:30:05.724 INFO:tasks.rados.rados.0.smithi038.stdout:471: done (10 left) 2024-03-19T11:30:05.724 INFO:tasks.rados.rados.0.smithi038.stdout:473: done (9 left) 2024-03-19T11:30:05.724 INFO:tasks.rados.rados.0.smithi038.stdout:474: done (8 left) 2024-03-19T11:30:05.725 INFO:tasks.rados.rados.0.smithi038.stdout:476: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.725 INFO:tasks.rados.rados.0.smithi038.stdout:476: set_chunk oid 27 offset: 636928 length: 1056256 target oid smithi03829856-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 636928 snap -1 2024-03-19T11:30:05.725 INFO:tasks.rados.rados.0.smithi038.stdout:477: set_chunk oid 28 target oid 28 low_tier 2024-03-19T11:30:05.725 INFO:tasks.rados.rados.0.smithi038.stdout:477: set_chunk oid 28 offset: 41984 length: 1259008 target oid smithi03829856-28 low_tier offset: 41984 snap -1 2024-03-19T11:30:05.725 INFO:tasks.rados.rados.0.smithi038.stdout:478: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.725 INFO:tasks.rados.rados.0.smithi038.stdout:478: set_chunk oid 29 offset: 2018816 length: 76288 target oid smithi03829856-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 2018816 snap -1 2024-03-19T11:30:05.725 INFO:tasks.rados.rados.0.smithi038.stdout:479: set_chunk oid 30 target oid 30 low_tier 2024-03-19T11:30:05.725 INFO:tasks.rados.rados.0.smithi038.stdout:479: set_chunk oid 30 offset: 1262080 length: 125440 target oid smithi03829856-30 low_tier offset: 1262080 snap -1 2024-03-19T11:30:05.725 INFO:tasks.rados.rados.0.smithi038.stdout:480: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.725 INFO:tasks.rados.rados.0.smithi038.stdout:480: set_chunk oid 31 offset: 842752 length: 760320 target oid smithi03829856-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 842752 snap -1 2024-03-19T11:30:05.725 INFO:tasks.rados.rados.0.smithi038.stdout:481: set_chunk oid 32 target oid 32 low_tier 2024-03-19T11:30:05.725 INFO:tasks.rados.rados.0.smithi038.stdout:481: set_chunk oid 32 offset: 910848 length: 1211904 target oid smithi03829856-32 low_tier offset: 910848 snap -1 2024-03-19T11:30:05.725 INFO:tasks.rados.rados.0.smithi038.stdout:482: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.725 INFO:tasks.rados.rados.0.smithi038.stdout:482: set_chunk oid 33 offset: 38912 length: 1144320 target oid smithi03829856-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 38912 snap -1 2024-03-19T11:30:05.725 INFO:tasks.rados.rados.0.smithi038.stdout:483: set_chunk oid 34 target oid 34 low_tier 2024-03-19T11:30:05.726 INFO:tasks.rados.rados.0.smithi038.stdout:483: set_chunk oid 34 offset: 2015744 length: 15360 target oid smithi03829856-34 low_tier offset: 2015744 snap -1 2024-03-19T11:30:05.726 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.726 INFO:tasks.rados.rados.0.smithi038.stdout:476: finishing set_chunk to oid 27 2024-03-19T11:30:05.726 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 27 set_chunk 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 636928 length: 1056256 2024-03-19T11:30:05.726 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-03-19T11:30:05.726 INFO:tasks.rados.rados.0.smithi038.stdout:477: finishing set_chunk to oid 28 2024-03-19T11:30:05.726 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 28 set_chunk 28 low_tier returned error code -95 offset: 41984 length: 1259008 2024-03-19T11:30:05.726 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 28 v 0 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-03-19T11:30:05.726 INFO:tasks.rados.rados.0.smithi038.stdout:478: finishing set_chunk to oid 29 2024-03-19T11:30:05.726 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 29 set_chunk 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 2018816 length: 76288 2024-03-19T11:30:05.726 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 29 v 0 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-03-19T11:30:05.726 INFO:tasks.rados.rados.0.smithi038.stdout:476: done (15 left) 2024-03-19T11:30:05.726 INFO:tasks.rados.rados.0.smithi038.stdout:477: done (14 left) 2024-03-19T11:30:05.727 INFO:tasks.rados.rados.0.smithi038.stdout:478: done (13 left) 2024-03-19T11:30:05.727 INFO:tasks.rados.rados.0.smithi038.stdout:484: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.727 INFO:tasks.rados.rados.0.smithi038.stdout:484: set_chunk oid 35 offset: 1944576 length: 199168 target oid smithi03829856-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1944576 snap -1 2024-03-19T11:30:05.727 INFO:tasks.rados.rados.0.smithi038.stdout:485: set_chunk oid 36 target oid 36 low_tier 2024-03-19T11:30:05.727 INFO:tasks.rados.rados.0.smithi038.stdout:485: set_chunk oid 36 offset: 65024 length: 515584 target oid smithi03829856-36 low_tier offset: 65024 snap -1 2024-03-19T11:30:05.727 INFO:tasks.rados.rados.0.smithi038.stdout:486: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.727 INFO:tasks.rados.rados.0.smithi038.stdout:486: set_chunk oid 37 offset: 301568 length: 996864 target oid smithi03829856-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 301568 snap -1 2024-03-19T11:30:05.727 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.727 INFO:tasks.rados.rados.0.smithi038.stdout:480: finishing set_chunk to oid 31 2024-03-19T11:30:05.727 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 31 set_chunk 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 842752 length: 760320 2024-03-19T11:30:05.727 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 31 v 0 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-03-19T11:30:05.727 INFO:tasks.rados.rados.0.smithi038.stdout:481: finishing set_chunk to oid 32 2024-03-19T11:30:05.727 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 32 set_chunk 32 low_tier returned error code -95 offset: 910848 length: 1211904 2024-03-19T11:30:05.728 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-03-19T11:30:05.728 INFO:tasks.rados.rados.0.smithi038.stdout:480: done (15 left) 2024-03-19T11:30:05.728 INFO:tasks.rados.rados.0.smithi038.stdout:481: done (14 left) 2024-03-19T11:30:05.728 INFO:tasks.rados.rados.0.smithi038.stdout:487: set_chunk oid 38 target oid 38 low_tier 2024-03-19T11:30:05.728 INFO:tasks.rados.rados.0.smithi038.stdout:487: set_chunk oid 38 offset: 1338368 length: 813056 target oid smithi03829856-38 low_tier offset: 1338368 snap -1 2024-03-19T11:30:05.728 INFO:tasks.rados.rados.0.smithi038.stdout:488: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.728 INFO:tasks.rados.rados.0.smithi038.stdout:488: set_chunk oid 39 offset: 333312 length: 722944 target oid smithi03829856-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 333312 snap -1 2024-03-19T11:30:05.728 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.728 INFO:tasks.rados.rados.0.smithi038.stdout:484: finishing set_chunk to oid 35 2024-03-19T11:30:05.728 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 35 set_chunk 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1944576 length: 199168 2024-03-19T11:30:05.728 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 35 v 0 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-03-19T11:30:05.728 INFO:tasks.rados.rados.0.smithi038.stdout:484: done (15 left) 2024-03-19T11:30:05.728 INFO:tasks.rados.rados.0.smithi038.stdout:489: set_chunk oid 40 target oid 40 low_tier 2024-03-19T11:30:05.728 INFO:tasks.rados.rados.0.smithi038.stdout:489: set_chunk oid 40 offset: 442368 length: 1960448 target oid smithi03829856-40 low_tier offset: 442368 snap -1 2024-03-19T11:30:05.728 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.729 INFO:tasks.rados.rados.0.smithi038.stdout:486: finishing set_chunk to oid 37 2024-03-19T11:30:05.729 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 37 set_chunk 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 301568 length: 996864 2024-03-19T11:30:05.729 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 37 v 0 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-03-19T11:30:05.729 INFO:tasks.rados.rados.0.smithi038.stdout:486: done (15 left) 2024-03-19T11:30:05.729 INFO:tasks.rados.rados.0.smithi038.stdout:490: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.729 INFO:tasks.rados.rados.0.smithi038.stdout:490: set_chunk oid 41 offset: 367104 length: 706048 target oid smithi03829856-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 367104 snap -1 2024-03-19T11:30:05.729 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.729 INFO:tasks.rados.rados.0.smithi038.stdout:487: finishing set_chunk to oid 38 2024-03-19T11:30:05.729 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 38 set_chunk 38 low_tier returned error code -95 offset: 1338368 length: 813056 2024-03-19T11:30:05.729 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 38 v 0 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-03-19T11:30:05.729 INFO:tasks.rados.rados.0.smithi038.stdout:488: finishing set_chunk to oid 39 2024-03-19T11:30:05.729 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 39 set_chunk 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 333312 length: 722944 2024-03-19T11:30:05.729 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 39 v 0 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-03-19T11:30:05.729 INFO:tasks.rados.rados.0.smithi038.stdout:487: done (15 left) 2024-03-19T11:30:05.729 INFO:tasks.rados.rados.0.smithi038.stdout:488: done (14 left) 2024-03-19T11:30:05.730 INFO:tasks.rados.rados.0.smithi038.stdout:491: set_chunk oid 42 target oid 42 low_tier 2024-03-19T11:30:05.730 INFO:tasks.rados.rados.0.smithi038.stdout:491: set_chunk oid 42 offset: 13824 length: 864256 target oid smithi03829856-42 low_tier offset: 13824 snap -1 2024-03-19T11:30:05.730 INFO:tasks.rados.rados.0.smithi038.stdout:492: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.730 INFO:tasks.rados.rados.0.smithi038.stdout:492: set_chunk oid 43 offset: 590336 length: 672768 target oid smithi03829856-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 590336 snap -1 2024-03-19T11:30:05.730 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.730 INFO:tasks.rados.rados.0.smithi038.stdout:490: finishing set_chunk to oid 41 2024-03-19T11:30:05.730 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 41 set_chunk 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 367104 length: 706048 2024-03-19T11:30:05.730 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 41 v 0 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-03-19T11:30:05.730 INFO:tasks.rados.rados.0.smithi038.stdout:490: done (15 left) 2024-03-19T11:30:05.730 INFO:tasks.rados.rados.0.smithi038.stdout:493: set_chunk oid 44 target oid 44 low_tier 2024-03-19T11:30:05.730 INFO:tasks.rados.rados.0.smithi038.stdout:493: set_chunk oid 44 offset: 1178624 length: 1290752 target oid smithi03829856-44 low_tier offset: 1178624 snap -1 2024-03-19T11:30:05.730 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.730 INFO:tasks.rados.rados.0.smithi038.stdout:489: finishing set_chunk to oid 40 2024-03-19T11:30:05.730 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 40 set_chunk 40 low_tier returned error code -95 offset: 442368 length: 1960448 2024-03-19T11:30:05.731 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 40 v 0 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-03-19T11:30:05.731 INFO:tasks.rados.rados.0.smithi038.stdout:493: finishing set_chunk to oid 44 2024-03-19T11:30:05.731 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 44 set_chunk 44 low_tier returned error code -95 offset: 1178624 length: 1290752 2024-03-19T11:30:05.731 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 44 v 0 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-03-19T11:30:05.731 INFO:tasks.rados.rados.0.smithi038.stdout:491: finishing set_chunk to oid 42 2024-03-19T11:30:05.731 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 42 set_chunk 42 low_tier returned error code -95 offset: 13824 length: 864256 2024-03-19T11:30:05.731 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-03-19T11:30:05.731 INFO:tasks.rados.rados.0.smithi038.stdout:489: done (15 left) 2024-03-19T11:30:05.731 INFO:tasks.rados.rados.0.smithi038.stdout:491: done (14 left) 2024-03-19T11:30:05.731 INFO:tasks.rados.rados.0.smithi038.stdout:493: done (13 left) 2024-03-19T11:30:05.731 INFO:tasks.rados.rados.0.smithi038.stdout:493 retry set_chunk ! 2024-03-19T11:30:05.731 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 13 2024-03-19T11:30:05.731 INFO:tasks.rados.rados.0.smithi038.stdout:450: finishing set_chunk to oid 1 2024-03-19T11:30:05.731 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-03-19T11:30:05.731 INFO:tasks.rados.rados.0.smithi038.stdout:450: done (12 left) 2024-03-19T11:30:05.732 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 12 2024-03-19T11:30:05.732 INFO:tasks.rados.rados.0.smithi038.stdout:492: finishing set_chunk to oid 43 2024-03-19T11:30:05.732 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 43 set_chunk 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 590336 length: 672768 2024-03-19T11:30:05.732 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 43 v 0 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-03-19T11:30:05.732 INFO:tasks.rados.rados.0.smithi038.stdout:492: done (11 left) 2024-03-19T11:30:05.732 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 11 2024-03-19T11:30:05.732 INFO:tasks.rados.rados.0.smithi038.stdout:448: finishing set_chunk to oid 49 2024-03-19T11:30:05.732 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 49 v 9 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-03-19T11:30:05.732 INFO:tasks.rados.rados.0.smithi038.stdout:448: done (10 left) 2024-03-19T11:30:05.732 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 10 2024-03-19T11:30:05.732 INFO:tasks.rados.rados.0.smithi038.stdout:461: finishing set_chunk to oid 12 2024-03-19T11:30:05.732 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 12 v 3 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-03-19T11:30:05.732 INFO:tasks.rados.rados.0.smithi038.stdout:461: done (9 left) 2024-03-19T11:30:05.732 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 9 2024-03-19T11:30:05.732 INFO:tasks.rados.rados.0.smithi038.stdout:444: finishing set_chunk to oid 45 2024-03-19T11:30:05.733 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 45 v 12 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-03-19T11:30:05.733 INFO:tasks.rados.rados.0.smithi038.stdout:444: done (8 left) 2024-03-19T11:30:05.733 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 8 2024-03-19T11:30:05.733 INFO:tasks.rados.rados.0.smithi038.stdout:475: finishing set_chunk to oid 26 2024-03-19T11:30:05.733 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 26 v 9 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-03-19T11:30:05.733 INFO:tasks.rados.rados.0.smithi038.stdout:475: done (7 left) 2024-03-19T11:30:05.733 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 7 2024-03-19T11:30:05.733 INFO:tasks.rados.rados.0.smithi038.stdout:462: finishing set_chunk to oid 13 2024-03-19T11:30:05.733 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 13 v 6 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-03-19T11:30:05.733 INFO:tasks.rados.rados.0.smithi038.stdout:462: done (6 left) 2024-03-19T11:30:05.733 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 6 2024-03-19T11:30:05.733 INFO:tasks.rados.rados.0.smithi038.stdout:469: finishing set_chunk to oid 20 2024-03-19T11:30:05.733 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 20 v 12 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-03-19T11:30:05.733 INFO:tasks.rados.rados.0.smithi038.stdout:469: done (5 left) 2024-03-19T11:30:05.733 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 5 2024-03-19T11:30:05.734 INFO:tasks.rados.rados.0.smithi038.stdout:485: finishing set_chunk to oid 36 2024-03-19T11:30:05.734 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 36 v 9 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-03-19T11:30:05.734 INFO:tasks.rados.rados.0.smithi038.stdout:479: finishing set_chunk to oid 30 2024-03-19T11:30:05.734 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 30 v 6 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-03-19T11:30:05.734 INFO:tasks.rados.rados.0.smithi038.stdout:479: done (4 left) 2024-03-19T11:30:05.734 INFO:tasks.rados.rados.0.smithi038.stdout:485: done (3 left) 2024-03-19T11:30:05.734 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 3 2024-03-19T11:30:05.734 INFO:tasks.rados.rados.0.smithi038.stdout:483: finishing set_chunk to oid 34 2024-03-19T11:30:05.734 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 34 v 3 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-03-19T11:30:05.734 INFO:tasks.rados.rados.0.smithi038.stdout:483: done (2 left) 2024-03-19T11:30:05.734 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 2 2024-03-19T11:30:05.734 INFO:tasks.rados.rados.0.smithi038.stdout:472: finishing set_chunk to oid 23 2024-03-19T11:30:05.734 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 23 v 9 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-03-19T11:30:05.734 INFO:tasks.rados.rados.0.smithi038.stdout:482: finishing set_chunk to oid 33 2024-03-19T11:30:05.734 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 33 v 9 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-03-19T11:30:05.735 INFO:tasks.rados.rados.0.smithi038.stdout:472: done (1 left) 2024-03-19T11:30:05.735 INFO:tasks.rados.rados.0.smithi038.stdout:482: done (0 left) 2024-03-19T11:30:05.735 INFO:tasks.rados.rados.0.smithi038.stdout:494: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.735 INFO:tasks.rados.rados.0.smithi038.stdout:494: set_chunk oid 45 offset: 265728 length: 2460160 target oid smithi03829856-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 265728 snap -1 2024-03-19T11:30:05.735 INFO:tasks.rados.rados.0.smithi038.stdout:495: set_chunk oid 46 target oid 46 low_tier 2024-03-19T11:30:05.735 INFO:tasks.rados.rados.0.smithi038.stdout:495: set_chunk oid 46 offset: 1816064 length: 277504 target oid smithi03829856-46 low_tier offset: 1816064 snap -1 2024-03-19T11:30:05.735 INFO:tasks.rados.rados.0.smithi038.stdout:496: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.735 INFO:tasks.rados.rados.0.smithi038.stdout:496: set_chunk oid 47 offset: 759296 length: 702976 target oid smithi03829856-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 759296 snap -1 2024-03-19T11:30:05.735 INFO:tasks.rados.rados.0.smithi038.stdout:497: set_chunk oid 48 target oid 48 low_tier 2024-03-19T11:30:05.735 INFO:tasks.rados.rados.0.smithi038.stdout:497: set_chunk oid 48 offset: 1771520 length: 91136 target oid smithi03829856-48 low_tier offset: 1771520 snap -1 2024-03-19T11:30:05.736 INFO:tasks.rados.rados.0.smithi038.stdout:498: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.736 INFO:tasks.rados.rados.0.smithi038.stdout:498: set_chunk oid 49 offset: 389632 length: 2092032 target oid smithi03829856-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 389632 snap -1 2024-03-19T11:30:05.736 INFO:tasks.rados.rados.0.smithi038.stdout:499: set_chunk oid 50 target oid 50 low_tier 2024-03-19T11:30:05.736 INFO:tasks.rados.rados.0.smithi038.stdout:499: set_chunk oid 50 offset: 457728 length: 2082816 target oid smithi03829856-50 low_tier offset: 457728 snap -1 2024-03-19T11:30:05.736 INFO:tasks.rados.rados.0.smithi038.stdout:500: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.736 INFO:tasks.rados.rados.0.smithi038.stdout:500: set_chunk oid 1 offset: 46592 length: 1263104 target oid smithi03829856-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 46592 snap -1 2024-03-19T11:30:05.736 INFO:tasks.rados.rados.0.smithi038.stdout:501: set_chunk oid 2 target oid 2 low_tier 2024-03-19T11:30:05.736 INFO:tasks.rados.rados.0.smithi038.stdout:501: set_chunk oid 2 offset: 850944 length: 349184 target oid smithi03829856-2 low_tier offset: 850944 snap -1 2024-03-19T11:30:05.736 INFO:tasks.rados.rados.0.smithi038.stdout:502: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.736 INFO:tasks.rados.rados.0.smithi038.stdout:502: set_chunk oid 3 offset: 409088 length: 1217024 target oid smithi03829856-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 409088 snap -1 2024-03-19T11:30:05.736 INFO:tasks.rados.rados.0.smithi038.stdout:503: set_chunk oid 4 target oid 4 low_tier 2024-03-19T11:30:05.736 INFO:tasks.rados.rados.0.smithi038.stdout:503: set_chunk oid 4 offset: 254976 length: 1136640 target oid smithi03829856-4 low_tier offset: 254976 snap -1 2024-03-19T11:30:05.736 INFO:tasks.rados.rados.0.smithi038.stdout:504: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.736 INFO:tasks.rados.rados.0.smithi038.stdout:504: set_chunk oid 5 offset: 904704 length: 1131520 target oid smithi03829856-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 904704 snap -1 2024-03-19T11:30:05.737 INFO:tasks.rados.rados.0.smithi038.stdout:505: set_chunk oid 6 target oid 6 low_tier 2024-03-19T11:30:05.737 INFO:tasks.rados.rados.0.smithi038.stdout:505: set_chunk oid 6 offset: 860160 length: 683520 target oid smithi03829856-6 low_tier offset: 860160 snap -1 2024-03-19T11:30:05.737 INFO:tasks.rados.rados.0.smithi038.stdout:506: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.737 INFO:tasks.rados.rados.0.smithi038.stdout:506: set_chunk oid 7 offset: 1243648 length: 51712 target oid smithi03829856-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1243648 snap -1 2024-03-19T11:30:05.737 INFO:tasks.rados.rados.0.smithi038.stdout:507: set_chunk oid 8 target oid 8 low_tier 2024-03-19T11:30:05.737 INFO:tasks.rados.rados.0.smithi038.stdout:507: set_chunk oid 8 offset: 147968 length: 359936 target oid smithi03829856-8 low_tier offset: 147968 snap -1 2024-03-19T11:30:05.737 INFO:tasks.rados.rados.0.smithi038.stdout:508: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.737 INFO:tasks.rados.rados.0.smithi038.stdout:508: set_chunk oid 9 offset: 772608 length: 1256960 target oid smithi03829856-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 772608 snap -1 2024-03-19T11:30:05.737 INFO:tasks.rados.rados.0.smithi038.stdout:509: set_chunk oid 10 target oid 10 low_tier 2024-03-19T11:30:05.737 INFO:tasks.rados.rados.0.smithi038.stdout:509: set_chunk oid 10 offset: 111616 length: 685568 target oid smithi03829856-10 low_tier offset: 111616 snap -1 2024-03-19T11:30:05.737 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.737 INFO:tasks.rados.rados.0.smithi038.stdout:495: finishing set_chunk to oid 46 2024-03-19T11:30:05.737 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 46 set_chunk 46 low_tier returned error code -95 offset: 1816064 length: 277504 2024-03-19T11:30:05.737 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-03-19T11:30:05.737 INFO:tasks.rados.rados.0.smithi038.stdout:495: done (15 left) 2024-03-19T11:30:05.738 INFO:tasks.rados.rados.0.smithi038.stdout:510: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.738 INFO:tasks.rados.rados.0.smithi038.stdout:510: set_chunk oid 11 offset: 1276928 length: 32768 target oid smithi03829856-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1276928 snap -1 2024-03-19T11:30:05.738 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.738 INFO:tasks.rados.rados.0.smithi038.stdout:496: finishing set_chunk to oid 47 2024-03-19T11:30:05.738 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 47 set_chunk 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 759296 length: 702976 2024-03-19T11:30:05.738 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-03-19T11:30:05.738 INFO:tasks.rados.rados.0.smithi038.stdout:498: finishing set_chunk to oid 49 2024-03-19T11:30:05.738 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 49 set_chunk 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 389632 length: 2092032 2024-03-19T11:30:05.738 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-03-19T11:30:05.738 INFO:tasks.rados.rados.0.smithi038.stdout:501: finishing set_chunk to oid 2 2024-03-19T11:30:05.738 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 2 set_chunk 2 low_tier returned error code -95 offset: 850944 length: 349184 2024-03-19T11:30:05.738 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-03-19T11:30:05.738 INFO:tasks.rados.rados.0.smithi038.stdout:503: finishing set_chunk to oid 4 2024-03-19T11:30:05.739 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 4 set_chunk 4 low_tier returned error code -95 offset: 254976 length: 1136640 2024-03-19T11:30:05.739 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-03-19T11:30:05.739 INFO:tasks.rados.rados.0.smithi038.stdout:506: finishing set_chunk to oid 7 2024-03-19T11:30:05.739 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 7 set_chunk 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1243648 length: 51712 2024-03-19T11:30:05.739 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-03-19T11:30:05.739 INFO:tasks.rados.rados.0.smithi038.stdout:505: finishing set_chunk to oid 6 2024-03-19T11:30:05.739 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 6 set_chunk 6 low_tier returned error code -95 offset: 860160 length: 683520 2024-03-19T11:30:05.739 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-03-19T11:30:05.739 INFO:tasks.rados.rados.0.smithi038.stdout:509: finishing set_chunk to oid 10 2024-03-19T11:30:05.739 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 10 set_chunk 10 low_tier returned error code -95 offset: 111616 length: 685568 2024-03-19T11:30:05.739 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 10 v 0 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-03-19T11:30:05.739 INFO:tasks.rados.rados.0.smithi038.stdout:500: finishing set_chunk to oid 1 2024-03-19T11:30:05.739 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 1 set_chunk 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 46592 length: 1263104 2024-03-19T11:30:05.739 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 1 v 0 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-03-19T11:30:05.739 INFO:tasks.rados.rados.0.smithi038.stdout:497: finishing set_chunk to oid 48 2024-03-19T11:30:05.740 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 48 set_chunk 48 low_tier returned error code -95 offset: 1771520 length: 91136 2024-03-19T11:30:05.740 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 48 v 0 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-03-19T11:30:05.740 INFO:tasks.rados.rados.0.smithi038.stdout:507: finishing set_chunk to oid 8 2024-03-19T11:30:05.740 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 8 set_chunk 8 low_tier returned error code -95 offset: 147968 length: 359936 2024-03-19T11:30:05.740 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-03-19T11:30:05.740 INFO:tasks.rados.rados.0.smithi038.stdout:494: finishing set_chunk to oid 45 2024-03-19T11:30:05.740 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 45 set_chunk 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 265728 length: 2460160 2024-03-19T11:30:05.740 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 45 v 0 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-03-19T11:30:05.740 INFO:tasks.rados.rados.0.smithi038.stdout:502: finishing set_chunk to oid 3 2024-03-19T11:30:05.740 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 3 set_chunk 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 409088 length: 1217024 2024-03-19T11:30:05.740 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 3 v 0 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-03-19T11:30:05.740 INFO:tasks.rados.rados.0.smithi038.stdout:508: finishing set_chunk to oid 9 2024-03-19T11:30:05.741 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 9 set_chunk 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 772608 length: 1256960 2024-03-19T11:30:05.741 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 9 v 0 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-03-19T11:30:05.741 INFO:tasks.rados.rados.0.smithi038.stdout:499: finishing set_chunk to oid 50 2024-03-19T11:30:05.741 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 50 set_chunk 50 low_tier returned error code -95 offset: 457728 length: 2082816 2024-03-19T11:30:05.741 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 50 v 0 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-03-19T11:30:05.741 INFO:tasks.rados.rados.0.smithi038.stdout:494: done (15 left) 2024-03-19T11:30:05.741 INFO:tasks.rados.rados.0.smithi038.stdout:496: done (14 left) 2024-03-19T11:30:05.741 INFO:tasks.rados.rados.0.smithi038.stdout:497: done (13 left) 2024-03-19T11:30:05.741 INFO:tasks.rados.rados.0.smithi038.stdout:498: done (12 left) 2024-03-19T11:30:05.741 INFO:tasks.rados.rados.0.smithi038.stdout:499: done (11 left) 2024-03-19T11:30:05.741 INFO:tasks.rados.rados.0.smithi038.stdout:500: done (10 left) 2024-03-19T11:30:05.741 INFO:tasks.rados.rados.0.smithi038.stdout:501: done (9 left) 2024-03-19T11:30:05.741 INFO:tasks.rados.rados.0.smithi038.stdout:502: done (8 left) 2024-03-19T11:30:05.742 INFO:tasks.rados.rados.0.smithi038.stdout:503: done (7 left) 2024-03-19T11:30:05.742 INFO:tasks.rados.rados.0.smithi038.stdout:505: done (6 left) 2024-03-19T11:30:05.742 INFO:tasks.rados.rados.0.smithi038.stdout:506: done (5 left) 2024-03-19T11:30:05.742 INFO:tasks.rados.rados.0.smithi038.stdout:507: done (4 left) 2024-03-19T11:30:05.742 INFO:tasks.rados.rados.0.smithi038.stdout:508: done (3 left) 2024-03-19T11:30:05.742 INFO:tasks.rados.rados.0.smithi038.stdout:509: done (2 left) 2024-03-19T11:30:05.742 INFO:tasks.rados.rados.0.smithi038.stdout:511: set_chunk oid 12 target oid 12 low_tier 2024-03-19T11:30:05.742 INFO:tasks.rados.rados.0.smithi038.stdout:511: set_chunk oid 12 offset: 1209856 length: 315904 target oid smithi03829856-12 low_tier offset: 1209856 snap -1 2024-03-19T11:30:05.742 INFO:tasks.rados.rados.0.smithi038.stdout:512: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.742 INFO:tasks.rados.rados.0.smithi038.stdout:512: set_chunk oid 13 offset: 167424 length: 796160 target oid smithi03829856-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 167424 snap -1 2024-03-19T11:30:05.742 INFO:tasks.rados.rados.0.smithi038.stdout:513: set_chunk oid 14 target oid 14 low_tier 2024-03-19T11:30:05.742 INFO:tasks.rados.rados.0.smithi038.stdout:513: set_chunk oid 14 offset: 360960 length: 595968 target oid smithi03829856-14 low_tier offset: 360960 snap -1 2024-03-19T11:30:05.743 INFO:tasks.rados.rados.0.smithi038.stdout:514: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.743 INFO:tasks.rados.rados.0.smithi038.stdout:514: set_chunk oid 15 offset: 464896 length: 561152 target oid smithi03829856-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 464896 snap -1 2024-03-19T11:30:05.743 INFO:tasks.rados.rados.0.smithi038.stdout:515: set_chunk oid 16 target oid 16 low_tier 2024-03-19T11:30:05.743 INFO:tasks.rados.rados.0.smithi038.stdout:515: set_chunk oid 16 offset: 1762304 length: 432128 target oid smithi03829856-16 low_tier offset: 1762304 snap -1 2024-03-19T11:30:05.743 INFO:tasks.rados.rados.0.smithi038.stdout:516: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.743 INFO:tasks.rados.rados.0.smithi038.stdout:516: set_chunk oid 17 offset: 1187840 length: 119808 target oid smithi03829856-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1187840 snap -1 2024-03-19T11:30:05.743 INFO:tasks.rados.rados.0.smithi038.stdout:517: set_chunk oid 18 target oid 18 low_tier 2024-03-19T11:30:05.743 INFO:tasks.rados.rados.0.smithi038.stdout:517: set_chunk oid 18 offset: 157184 length: 1966592 target oid smithi03829856-18 low_tier offset: 157184 snap -1 2024-03-19T11:30:05.743 INFO:tasks.rados.rados.0.smithi038.stdout:518: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.743 INFO:tasks.rados.rados.0.smithi038.stdout:518: set_chunk oid 19 offset: 1655296 length: 282112 target oid smithi03829856-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1655296 snap -1 2024-03-19T11:30:05.743 INFO:tasks.rados.rados.0.smithi038.stdout:519: set_chunk oid 20 target oid 20 low_tier 2024-03-19T11:30:05.743 INFO:tasks.rados.rados.0.smithi038.stdout:519: set_chunk oid 20 offset: 1236992 length: 283136 target oid smithi03829856-20 low_tier offset: 1236992 snap -1 2024-03-19T11:30:05.744 INFO:tasks.rados.rados.0.smithi038.stdout:520: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.744 INFO:tasks.rados.rados.0.smithi038.stdout:520: set_chunk oid 21 offset: 201216 length: 496640 target oid smithi03829856-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 201216 snap -1 2024-03-19T11:30:05.744 INFO:tasks.rados.rados.0.smithi038.stdout:521: set_chunk oid 22 target oid 22 low_tier 2024-03-19T11:30:05.744 INFO:tasks.rados.rados.0.smithi038.stdout:521: set_chunk oid 22 offset: 688640 length: 1560064 target oid smithi03829856-22 low_tier offset: 688640 snap -1 2024-03-19T11:30:05.744 INFO:tasks.rados.rados.0.smithi038.stdout:522: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.744 INFO:tasks.rados.rados.0.smithi038.stdout:522: set_chunk oid 23 offset: 870400 length: 667136 target oid smithi03829856-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 870400 snap -1 2024-03-19T11:30:05.744 INFO:tasks.rados.rados.0.smithi038.stdout:523: set_chunk oid 24 target oid 24 low_tier 2024-03-19T11:30:05.744 INFO:tasks.rados.rados.0.smithi038.stdout:523: set_chunk oid 24 offset: 740352 length: 1109504 target oid smithi03829856-24 low_tier offset: 740352 snap -1 2024-03-19T11:30:05.744 INFO:tasks.rados.rados.0.smithi038.stdout:524: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.744 INFO:tasks.rados.rados.0.smithi038.stdout:524: set_chunk oid 25 offset: 758784 length: 236032 target oid smithi03829856-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 758784 snap -1 2024-03-19T11:30:05.744 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.744 INFO:tasks.rados.rados.0.smithi038.stdout:504: finishing set_chunk to oid 5 2024-03-19T11:30:05.744 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 5 set_chunk 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 904704 length: 1131520 2024-03-19T11:30:05.744 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-03-19T11:30:05.744 INFO:tasks.rados.rados.0.smithi038.stdout:510: finishing set_chunk to oid 11 2024-03-19T11:30:05.745 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 11 set_chunk 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1276928 length: 32768 2024-03-19T11:30:05.745 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 11 v 0 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-03-19T11:30:05.745 INFO:tasks.rados.rados.0.smithi038.stdout:504: done (15 left) 2024-03-19T11:30:05.745 INFO:tasks.rados.rados.0.smithi038.stdout:510: done (14 left) 2024-03-19T11:30:05.745 INFO:tasks.rados.rados.0.smithi038.stdout:525: set_chunk oid 26 target oid 26 low_tier 2024-03-19T11:30:05.745 INFO:tasks.rados.rados.0.smithi038.stdout:525: set_chunk oid 26 offset: 591872 length: 1509888 target oid smithi03829856-26 low_tier offset: 591872 snap -1 2024-03-19T11:30:05.745 INFO:tasks.rados.rados.0.smithi038.stdout:526: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.745 INFO:tasks.rados.rados.0.smithi038.stdout:526: set_chunk oid 27 offset: 331776 length: 1209344 target oid smithi03829856-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 331776 snap -1 2024-03-19T11:30:05.745 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.745 INFO:tasks.rados.rados.0.smithi038.stdout:517: finishing set_chunk to oid 18 2024-03-19T11:30:05.745 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 18 set_chunk 18 low_tier returned error code -95 offset: 157184 length: 1966592 2024-03-19T11:30:05.745 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 18 v 0 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-03-19T11:30:05.745 INFO:tasks.rados.rados.0.smithi038.stdout:511: finishing set_chunk to oid 12 2024-03-19T11:30:05.746 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 12 set_chunk 12 low_tier returned error code -95 offset: 1209856 length: 315904 2024-03-19T11:30:05.746 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 12 v 0 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-03-19T11:30:05.746 INFO:tasks.rados.rados.0.smithi038.stdout:512: finishing set_chunk to oid 13 2024-03-19T11:30:05.746 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 13 set_chunk 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 167424 length: 796160 2024-03-19T11:30:05.746 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 13 v 0 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-03-19T11:30:05.746 INFO:tasks.rados.rados.0.smithi038.stdout:514: finishing set_chunk to oid 15 2024-03-19T11:30:05.746 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 464896 length: 561152 2024-03-19T11:30:05.746 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-03-19T11:30:05.746 INFO:tasks.rados.rados.0.smithi038.stdout:518: finishing set_chunk to oid 19 2024-03-19T11:30:05.746 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 19 set_chunk 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1655296 length: 282112 2024-03-19T11:30:05.746 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 19 v 0 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-03-19T11:30:05.746 INFO:tasks.rados.rados.0.smithi038.stdout:511: done (15 left) 2024-03-19T11:30:05.746 INFO:tasks.rados.rados.0.smithi038.stdout:512: done (14 left) 2024-03-19T11:30:05.747 INFO:tasks.rados.rados.0.smithi038.stdout:514: done (13 left) 2024-03-19T11:30:05.747 INFO:tasks.rados.rados.0.smithi038.stdout:517: done (12 left) 2024-03-19T11:30:05.747 INFO:tasks.rados.rados.0.smithi038.stdout:518: done (11 left) 2024-03-19T11:30:05.747 INFO:tasks.rados.rados.0.smithi038.stdout:527: set_chunk oid 28 target oid 28 low_tier 2024-03-19T11:30:05.747 INFO:tasks.rados.rados.0.smithi038.stdout:527: set_chunk oid 28 offset: 827904 length: 1399296 target oid smithi03829856-28 low_tier offset: 827904 snap -1 2024-03-19T11:30:05.747 INFO:tasks.rados.rados.0.smithi038.stdout:528: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.747 INFO:tasks.rados.rados.0.smithi038.stdout:528: set_chunk oid 29 offset: 183808 length: 1294336 target oid smithi03829856-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 183808 snap -1 2024-03-19T11:30:05.747 INFO:tasks.rados.rados.0.smithi038.stdout:529: set_chunk oid 30 target oid 30 low_tier 2024-03-19T11:30:05.747 INFO:tasks.rados.rados.0.smithi038.stdout:529: set_chunk oid 30 offset: 865792 length: 1133568 target oid smithi03829856-30 low_tier offset: 865792 snap -1 2024-03-19T11:30:05.747 INFO:tasks.rados.rados.0.smithi038.stdout:523: finishing set_chunk to oid 24 2024-03-19T11:30:05.747 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 24 set_chunk 24 low_tier returned error code -95 offset: 740352 length: 1109504 2024-03-19T11:30:05.747 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-03-19T11:30:05.747 INFO:tasks.rados.rados.0.smithi038.stdout:523: done (13 left) 2024-03-19T11:30:05.747 INFO:tasks.rados.rados.0.smithi038.stdout:530: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.747 INFO:tasks.rados.rados.0.smithi038.stdout:530: set_chunk oid 31 offset: 416768 length: 491008 target oid smithi03829856-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 416768 snap -1 2024-03-19T11:30:05.748 INFO:tasks.rados.rados.0.smithi038.stdout:531: set_chunk oid 32 target oid 32 low_tier 2024-03-19T11:30:05.748 INFO:tasks.rados.rados.0.smithi038.stdout:531: set_chunk oid 32 offset: 1110016 length: 589824 target oid smithi03829856-32 low_tier offset: 1110016 snap -1 2024-03-19T11:30:05.748 INFO:tasks.rados.rados.0.smithi038.stdout:532: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.748 INFO:tasks.rados.rados.0.smithi038.stdout:532: set_chunk oid 33 offset: 2159616 length: 87552 target oid smithi03829856-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 2159616 snap -1 2024-03-19T11:30:05.748 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.748 INFO:tasks.rados.rados.0.smithi038.stdout:522: finishing set_chunk to oid 23 2024-03-19T11:30:05.748 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 23 set_chunk 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 870400 length: 667136 2024-03-19T11:30:05.748 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 23 v 0 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-03-19T11:30:05.748 INFO:tasks.rados.rados.0.smithi038.stdout:513: finishing set_chunk to oid 14 2024-03-19T11:30:05.748 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 14 set_chunk 14 low_tier returned error code -95 offset: 360960 length: 595968 2024-03-19T11:30:05.748 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 14 v 0 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-03-19T11:30:05.748 INFO:tasks.rados.rados.0.smithi038.stdout:515: finishing set_chunk to oid 16 2024-03-19T11:30:05.748 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 16 set_chunk 16 low_tier returned error code -95 offset: 1762304 length: 432128 2024-03-19T11:30:05.748 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-03-19T11:30:05.748 INFO:tasks.rados.rados.0.smithi038.stdout:521: finishing set_chunk to oid 22 2024-03-19T11:30:05.749 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 22 set_chunk 22 low_tier returned error code -95 offset: 688640 length: 1560064 2024-03-19T11:30:05.749 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-03-19T11:30:05.749 INFO:tasks.rados.rados.0.smithi038.stdout:526: finishing set_chunk to oid 27 2024-03-19T11:30:05.749 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 27 set_chunk 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 331776 length: 1209344 2024-03-19T11:30:05.749 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-03-19T11:30:05.749 INFO:tasks.rados.rados.0.smithi038.stdout:524: finishing set_chunk to oid 25 2024-03-19T11:30:05.749 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 25 set_chunk 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 758784 length: 236032 2024-03-19T11:30:05.749 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-03-19T11:30:05.749 INFO:tasks.rados.rados.0.smithi038.stdout:527: finishing set_chunk to oid 28 2024-03-19T11:30:05.749 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 28 set_chunk 28 low_tier returned error code -95 offset: 827904 length: 1399296 2024-03-19T11:30:05.749 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 28 v 0 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-03-19T11:30:05.749 INFO:tasks.rados.rados.0.smithi038.stdout:513: done (15 left) 2024-03-19T11:30:05.749 INFO:tasks.rados.rados.0.smithi038.stdout:515: done (14 left) 2024-03-19T11:30:05.750 INFO:tasks.rados.rados.0.smithi038.stdout:521: done (13 left) 2024-03-19T11:30:05.750 INFO:tasks.rados.rados.0.smithi038.stdout:522: done (12 left) 2024-03-19T11:30:05.750 INFO:tasks.rados.rados.0.smithi038.stdout:524: done (11 left) 2024-03-19T11:30:05.750 INFO:tasks.rados.rados.0.smithi038.stdout:526: done (10 left) 2024-03-19T11:30:05.750 INFO:tasks.rados.rados.0.smithi038.stdout:527: done (9 left) 2024-03-19T11:30:05.750 INFO:tasks.rados.rados.0.smithi038.stdout:533: set_chunk oid 34 target oid 34 low_tier 2024-03-19T11:30:05.750 INFO:tasks.rados.rados.0.smithi038.stdout:533: set_chunk oid 34 offset: 793600 length: 889344 target oid smithi03829856-34 low_tier offset: 793600 snap -1 2024-03-19T11:30:05.750 INFO:tasks.rados.rados.0.smithi038.stdout:534: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.750 INFO:tasks.rados.rados.0.smithi038.stdout:534: set_chunk oid 35 offset: 1013760 length: 603648 target oid smithi03829856-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1013760 snap -1 2024-03-19T11:30:05.750 INFO:tasks.rados.rados.0.smithi038.stdout:535: set_chunk oid 36 target oid 36 low_tier 2024-03-19T11:30:05.750 INFO:tasks.rados.rados.0.smithi038.stdout:535: set_chunk oid 36 offset: 1957376 length: 426496 target oid smithi03829856-36 low_tier offset: 1957376 snap -1 2024-03-19T11:30:05.750 INFO:tasks.rados.rados.0.smithi038.stdout:536: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.750 INFO:tasks.rados.rados.0.smithi038.stdout:536: set_chunk oid 37 offset: 24064 length: 196608 target oid smithi03829856-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 24064 snap -1 2024-03-19T11:30:05.750 INFO:tasks.rados.rados.0.smithi038.stdout:537: set_chunk oid 38 target oid 38 low_tier 2024-03-19T11:30:05.751 INFO:tasks.rados.rados.0.smithi038.stdout:537: set_chunk oid 38 offset: 1679360 length: 911360 target oid smithi03829856-38 low_tier offset: 1679360 snap -1 2024-03-19T11:30:05.751 INFO:tasks.rados.rados.0.smithi038.stdout:531: finishing set_chunk to oid 32 2024-03-19T11:30:05.751 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 32 set_chunk 32 low_tier returned error code -95 offset: 1110016 length: 589824 2024-03-19T11:30:05.751 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-03-19T11:30:05.751 INFO:tasks.rados.rados.0.smithi038.stdout:531: done (13 left) 2024-03-19T11:30:05.751 INFO:tasks.rados.rados.0.smithi038.stdout:538: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.751 INFO:tasks.rados.rados.0.smithi038.stdout:538: set_chunk oid 39 offset: 245760 length: 485888 target oid smithi03829856-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 245760 snap -1 2024-03-19T11:30:05.751 INFO:tasks.rados.rados.0.smithi038.stdout:539: set_chunk oid 40 target oid 40 low_tier 2024-03-19T11:30:05.751 INFO:tasks.rados.rados.0.smithi038.stdout:539: set_chunk oid 40 offset: 300032 length: 1547776 target oid smithi03829856-40 low_tier offset: 300032 snap -1 2024-03-19T11:30:05.751 INFO:tasks.rados.rados.0.smithi038.stdout:540: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.751 INFO:tasks.rados.rados.0.smithi038.stdout:540: set_chunk oid 41 offset: 242176 length: 38912 target oid smithi03829856-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 242176 snap -1 2024-03-19T11:30:05.751 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.751 INFO:tasks.rados.rados.0.smithi038.stdout:535: finishing set_chunk to oid 36 2024-03-19T11:30:05.752 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 36 set_chunk 36 low_tier returned error code -95 offset: 1957376 length: 426496 2024-03-19T11:30:05.752 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-03-19T11:30:05.752 INFO:tasks.rados.rados.0.smithi038.stdout:537: finishing set_chunk to oid 38 2024-03-19T11:30:05.752 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 38 set_chunk 38 low_tier returned error code -95 offset: 1679360 length: 911360 2024-03-19T11:30:05.752 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 38 v 0 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-03-19T11:30:05.752 INFO:tasks.rados.rados.0.smithi038.stdout:535: done (15 left) 2024-03-19T11:30:05.752 INFO:tasks.rados.rados.0.smithi038.stdout:537: done (14 left) 2024-03-19T11:30:05.752 INFO:tasks.rados.rados.0.smithi038.stdout:541: set_chunk oid 42 target oid 42 low_tier 2024-03-19T11:30:05.752 INFO:tasks.rados.rados.0.smithi038.stdout:541: set_chunk oid 42 offset: 2199552 length: 445440 target oid smithi03829856-42 low_tier offset: 2199552 snap -1 2024-03-19T11:30:05.752 INFO:tasks.rados.rados.0.smithi038.stdout:542: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.752 INFO:tasks.rados.rados.0.smithi038.stdout:542: set_chunk oid 43 offset: 2035200 length: 120320 target oid smithi03829856-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 2035200 snap -1 2024-03-19T11:30:05.752 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.752 INFO:tasks.rados.rados.0.smithi038.stdout:541: finishing set_chunk to oid 42 2024-03-19T11:30:05.752 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 42 set_chunk 42 low_tier returned error code -95 offset: 2199552 length: 445440 2024-03-19T11:30:05.753 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-03-19T11:30:05.753 INFO:tasks.rados.rados.0.smithi038.stdout:541: done (15 left) 2024-03-19T11:30:05.753 INFO:tasks.rados.rados.0.smithi038.stdout:543: set_chunk oid 44 target oid 44 low_tier 2024-03-19T11:30:05.753 INFO:tasks.rados.rados.0.smithi038.stdout:543: set_chunk oid 44 offset: 2150912 length: 400896 target oid smithi03829856-44 low_tier offset: 2150912 snap -1 2024-03-19T11:30:05.753 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.753 INFO:tasks.rados.rados.0.smithi038.stdout:540: finishing set_chunk to oid 41 2024-03-19T11:30:05.753 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 41 set_chunk 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 242176 length: 38912 2024-03-19T11:30:05.753 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 41 v 0 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-03-19T11:30:05.753 INFO:tasks.rados.rados.0.smithi038.stdout:540: done (15 left) 2024-03-19T11:30:05.753 INFO:tasks.rados.rados.0.smithi038.stdout:544: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.753 INFO:tasks.rados.rados.0.smithi038.stdout:544: set_chunk oid 45 offset: 327168 length: 597504 target oid smithi03829856-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 327168 snap -1 2024-03-19T11:30:05.753 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.753 INFO:tasks.rados.rados.0.smithi038.stdout:525: finishing set_chunk to oid 26 2024-03-19T11:30:05.754 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 26 set_chunk 26 low_tier returned error code -95 offset: 591872 length: 1509888 2024-03-19T11:30:05.754 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-03-19T11:30:05.754 INFO:tasks.rados.rados.0.smithi038.stdout:529: finishing set_chunk to oid 30 2024-03-19T11:30:05.754 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 30 set_chunk 30 low_tier returned error code -95 offset: 865792 length: 1133568 2024-03-19T11:30:05.754 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 30 v 0 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-03-19T11:30:05.754 INFO:tasks.rados.rados.0.smithi038.stdout:533: finishing set_chunk to oid 34 2024-03-19T11:30:05.754 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 34 set_chunk 34 low_tier returned error code -95 offset: 793600 length: 889344 2024-03-19T11:30:05.754 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 34 v 0 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-03-19T11:30:05.754 INFO:tasks.rados.rados.0.smithi038.stdout:538: finishing set_chunk to oid 39 2024-03-19T11:30:05.754 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 39 set_chunk 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 245760 length: 485888 2024-03-19T11:30:05.754 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 39 v 0 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-03-19T11:30:05.754 INFO:tasks.rados.rados.0.smithi038.stdout:528: finishing set_chunk to oid 29 2024-03-19T11:30:05.755 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 29 set_chunk 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 183808 length: 1294336 2024-03-19T11:30:05.755 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 29 v 0 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-03-19T11:30:05.755 INFO:tasks.rados.rados.0.smithi038.stdout:539: finishing set_chunk to oid 40 2024-03-19T11:30:05.755 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 40 set_chunk 40 low_tier returned error code -95 offset: 300032 length: 1547776 2024-03-19T11:30:05.755 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 40 v 0 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-03-19T11:30:05.755 INFO:tasks.rados.rados.0.smithi038.stdout:525: done (15 left) 2024-03-19T11:30:05.755 INFO:tasks.rados.rados.0.smithi038.stdout:528: done (14 left) 2024-03-19T11:30:05.755 INFO:tasks.rados.rados.0.smithi038.stdout:529: done (13 left) 2024-03-19T11:30:05.755 INFO:tasks.rados.rados.0.smithi038.stdout:533: done (12 left) 2024-03-19T11:30:05.755 INFO:tasks.rados.rados.0.smithi038.stdout:538: done (11 left) 2024-03-19T11:30:05.755 INFO:tasks.rados.rados.0.smithi038.stdout:539: done (10 left) 2024-03-19T11:30:05.755 INFO:tasks.rados.rados.0.smithi038.stdout:545: set_chunk oid 46 target oid 46 low_tier 2024-03-19T11:30:05.755 INFO:tasks.rados.rados.0.smithi038.stdout:545: set_chunk oid 46 offset: 652288 length: 1944576 target oid smithi03829856-46 low_tier offset: 652288 snap -1 2024-03-19T11:30:05.756 INFO:tasks.rados.rados.0.smithi038.stdout:546: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.756 INFO:tasks.rados.rados.0.smithi038.stdout:546: set_chunk oid 47 offset: 1368576 length: 292352 target oid smithi03829856-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1368576 snap -1 2024-03-19T11:30:05.756 INFO:tasks.rados.rados.0.smithi038.stdout:532: finishing set_chunk to oid 33 2024-03-19T11:30:05.756 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 33 set_chunk 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 2159616 length: 87552 2024-03-19T11:30:05.756 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-03-19T11:30:05.756 INFO:tasks.rados.rados.0.smithi038.stdout:532: done (11 left) 2024-03-19T11:30:05.756 INFO:tasks.rados.rados.0.smithi038.stdout:547: set_chunk oid 48 target oid 48 low_tier 2024-03-19T11:30:05.756 INFO:tasks.rados.rados.0.smithi038.stdout:547: set_chunk oid 48 offset: 1048576 length: 860160 target oid smithi03829856-48 low_tier offset: 1048576 snap -1 2024-03-19T11:30:05.756 INFO:tasks.rados.rados.0.smithi038.stdout:548: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.756 INFO:tasks.rados.rados.0.smithi038.stdout:548: set_chunk oid 49 offset: 269312 length: 1552384 target oid smithi03829856-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 269312 snap -1 2024-03-19T11:30:05.756 INFO:tasks.rados.rados.0.smithi038.stdout:549: set_chunk oid 50 target oid 50 low_tier 2024-03-19T11:30:05.756 INFO:tasks.rados.rados.0.smithi038.stdout:549: set_chunk oid 50 offset: 544768 length: 1070080 target oid smithi03829856-50 low_tier offset: 544768 snap -1 2024-03-19T11:30:05.756 INFO:tasks.rados.rados.0.smithi038.stdout:550: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.756 INFO:tasks.rados.rados.0.smithi038.stdout:530: finishing set_chunk to oid 31 2024-03-19T11:30:05.757 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 31 set_chunk 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 416768 length: 491008 2024-03-19T11:30:05.757 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 31 v 0 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-03-19T11:30:05.757 INFO:tasks.rados.rados.0.smithi038.stdout:550: set_chunk oid 1 offset: 556032 length: 1328640 target oid smithi03829856-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 556032 snap -1 2024-03-19T11:30:05.757 INFO:tasks.rados.rados.0.smithi038.stdout:530: done (14 left) 2024-03-19T11:30:05.757 INFO:tasks.rados.rados.0.smithi038.stdout:551: set_chunk oid 2 target oid 2 low_tier 2024-03-19T11:30:05.757 INFO:tasks.rados.rados.0.smithi038.stdout:551: set_chunk oid 2 offset: 581120 length: 812544 target oid smithi03829856-2 low_tier offset: 581120 snap -1 2024-03-19T11:30:05.757 INFO:tasks.rados.rados.0.smithi038.stdout:552: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.757 INFO:tasks.rados.rados.0.smithi038.stdout:552: set_chunk oid 3 offset: 667648 length: 227840 target oid smithi03829856-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 667648 snap -1 2024-03-19T11:30:05.757 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.757 INFO:tasks.rados.rados.0.smithi038.stdout:544: finishing set_chunk to oid 45 2024-03-19T11:30:05.757 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 45 set_chunk 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 327168 length: 597504 2024-03-19T11:30:05.757 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 45 v 0 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-03-19T11:30:05.757 INFO:tasks.rados.rados.0.smithi038.stdout:545: finishing set_chunk to oid 46 2024-03-19T11:30:05.758 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 46 set_chunk 46 low_tier returned error code -95 offset: 652288 length: 1944576 2024-03-19T11:30:05.758 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-03-19T11:30:05.758 INFO:tasks.rados.rados.0.smithi038.stdout:542: finishing set_chunk to oid 43 2024-03-19T11:30:05.758 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 43 set_chunk 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 2035200 length: 120320 2024-03-19T11:30:05.758 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 43 v 0 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-03-19T11:30:05.758 INFO:tasks.rados.rados.0.smithi038.stdout:546: finishing set_chunk to oid 47 2024-03-19T11:30:05.758 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 47 set_chunk 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1368576 length: 292352 2024-03-19T11:30:05.758 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-03-19T11:30:05.758 INFO:tasks.rados.rados.0.smithi038.stdout:550: finishing set_chunk to oid 1 2024-03-19T11:30:05.758 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 1 set_chunk 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 556032 length: 1328640 2024-03-19T11:30:05.758 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 1 v 0 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-03-19T11:30:05.759 INFO:tasks.rados.rados.0.smithi038.stdout:542: done (15 left) 2024-03-19T11:30:05.759 INFO:tasks.rados.rados.0.smithi038.stdout:544: done (14 left) 2024-03-19T11:30:05.759 INFO:tasks.rados.rados.0.smithi038.stdout:545: done (13 left) 2024-03-19T11:30:05.759 INFO:tasks.rados.rados.0.smithi038.stdout:546: done (12 left) 2024-03-19T11:30:05.759 INFO:tasks.rados.rados.0.smithi038.stdout:550: done (11 left) 2024-03-19T11:30:05.759 INFO:tasks.rados.rados.0.smithi038.stdout:553: set_chunk oid 4 target oid 4 low_tier 2024-03-19T11:30:05.759 INFO:tasks.rados.rados.0.smithi038.stdout:553: set_chunk oid 4 offset: 966144 length: 1057280 target oid smithi03829856-4 low_tier offset: 966144 snap -1 2024-03-19T11:30:05.759 INFO:tasks.rados.rados.0.smithi038.stdout:554: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.759 INFO:tasks.rados.rados.0.smithi038.stdout:554: set_chunk oid 5 offset: 586240 length: 875008 target oid smithi03829856-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 586240 snap -1 2024-03-19T11:30:05.759 INFO:tasks.rados.rados.0.smithi038.stdout:555: set_chunk oid 6 target oid 6 low_tier 2024-03-19T11:30:05.759 INFO:tasks.rados.rados.0.smithi038.stdout:549: finishing set_chunk to oid 50 2024-03-19T11:30:05.759 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 50 set_chunk 50 low_tier returned error code -95 offset: 544768 length: 1070080 2024-03-19T11:30:05.760 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 50 v 0 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-03-19T11:30:05.760 INFO:tasks.rados.rados.0.smithi038.stdout:552: finishing set_chunk to oid 3 2024-03-19T11:30:05.760 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 3 set_chunk 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 667648 length: 227840 2024-03-19T11:30:05.760 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 3 v 0 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-03-19T11:30:05.760 INFO:tasks.rados.rados.0.smithi038.stdout:548: finishing set_chunk to oid 49 2024-03-19T11:30:05.760 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 49 set_chunk 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 269312 length: 1552384 2024-03-19T11:30:05.760 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-03-19T11:30:05.760 INFO:tasks.rados.rados.0.smithi038.stdout:551: finishing set_chunk to oid 2 2024-03-19T11:30:05.760 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 2 set_chunk 2 low_tier returned error code -95 offset: 581120 length: 812544 2024-03-19T11:30:05.760 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-03-19T11:30:05.761 INFO:tasks.rados.rados.0.smithi038.stdout:555: set_chunk oid 6 offset: 1026560 length: 443392 target oid smithi03829856-6 low_tier offset: 1026560 snap -1 2024-03-19T11:30:05.761 INFO:tasks.rados.rados.0.smithi038.stdout:548: done (13 left) 2024-03-19T11:30:05.761 INFO:tasks.rados.rados.0.smithi038.stdout:549: done (12 left) 2024-03-19T11:30:05.761 INFO:tasks.rados.rados.0.smithi038.stdout:551: done (11 left) 2024-03-19T11:30:05.761 INFO:tasks.rados.rados.0.smithi038.stdout:552: done (10 left) 2024-03-19T11:30:05.761 INFO:tasks.rados.rados.0.smithi038.stdout:556: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.761 INFO:tasks.rados.rados.0.smithi038.stdout:556: set_chunk oid 7 offset: 1483264 length: 4608 target oid smithi03829856-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1483264 snap -1 2024-03-19T11:30:05.761 INFO:tasks.rados.rados.0.smithi038.stdout:557: set_chunk oid 8 target oid 8 low_tier 2024-03-19T11:30:05.761 INFO:tasks.rados.rados.0.smithi038.stdout:557: set_chunk oid 8 offset: 1609728 length: 332800 target oid smithi03829856-8 low_tier offset: 1609728 snap -1 2024-03-19T11:30:05.761 INFO:tasks.rados.rados.0.smithi038.stdout:558: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.761 INFO:tasks.rados.rados.0.smithi038.stdout:558: set_chunk oid 9 offset: 1660416 length: 17408 target oid smithi03829856-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1660416 snap -1 2024-03-19T11:30:05.761 INFO:tasks.rados.rados.0.smithi038.stdout:559: set_chunk oid 10 target oid 10 low_tier 2024-03-19T11:30:05.761 INFO:tasks.rados.rados.0.smithi038.stdout:559: set_chunk oid 10 offset: 823808 length: 1261056 target oid smithi03829856-10 low_tier offset: 823808 snap -1 2024-03-19T11:30:05.761 INFO:tasks.rados.rados.0.smithi038.stdout:560: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.762 INFO:tasks.rados.rados.0.smithi038.stdout:560: set_chunk oid 11 offset: 1270272 length: 878080 target oid smithi03829856-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1270272 snap -1 2024-03-19T11:30:05.762 INFO:tasks.rados.rados.0.smithi038.stdout:561: set_chunk oid 12 target oid 12 low_tier 2024-03-19T11:30:05.762 INFO:tasks.rados.rados.0.smithi038.stdout:561: set_chunk oid 12 offset: 1320448 length: 485888 target oid smithi03829856-12 low_tier offset: 1320448 snap -1 2024-03-19T11:30:05.762 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.762 INFO:tasks.rados.rados.0.smithi038.stdout:543: finishing set_chunk to oid 44 2024-03-19T11:30:05.762 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 44 set_chunk 44 low_tier returned error code -95 offset: 2150912 length: 400896 2024-03-19T11:30:05.762 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 44 v 0 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-03-19T11:30:05.762 INFO:tasks.rados.rados.0.smithi038.stdout:547: finishing set_chunk to oid 48 2024-03-19T11:30:05.762 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 48 set_chunk 48 low_tier returned error code -95 offset: 1048576 length: 860160 2024-03-19T11:30:05.762 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 48 v 0 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-03-19T11:30:05.762 INFO:tasks.rados.rados.0.smithi038.stdout:543: done (15 left) 2024-03-19T11:30:05.762 INFO:tasks.rados.rados.0.smithi038.stdout:547: done (14 left) 2024-03-19T11:30:05.762 INFO:tasks.rados.rados.0.smithi038.stdout:562: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.763 INFO:tasks.rados.rados.0.smithi038.stdout:562: set_chunk oid 13 offset: 940544 length: 351744 target oid smithi03829856-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 940544 snap -1 2024-03-19T11:30:05.763 INFO:tasks.rados.rados.0.smithi038.stdout:563: set_chunk oid 14 target oid 14 low_tier 2024-03-19T11:30:05.763 INFO:tasks.rados.rados.0.smithi038.stdout:563: set_chunk oid 14 offset: 345088 length: 1595904 target oid smithi03829856-14 low_tier offset: 345088 snap -1 2024-03-19T11:30:05.763 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.763 INFO:tasks.rados.rados.0.smithi038.stdout:553: finishing set_chunk to oid 4 2024-03-19T11:30:05.763 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 4 set_chunk 4 low_tier returned error code -95 offset: 966144 length: 1057280 2024-03-19T11:30:05.763 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-03-19T11:30:05.763 INFO:tasks.rados.rados.0.smithi038.stdout:555: finishing set_chunk to oid 6 2024-03-19T11:30:05.763 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 6 set_chunk 6 low_tier returned error code -95 offset: 1026560 length: 443392 2024-03-19T11:30:05.763 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-03-19T11:30:05.763 INFO:tasks.rados.rados.0.smithi038.stdout:556: finishing set_chunk to oid 7 2024-03-19T11:30:05.763 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 7 set_chunk 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1483264 length: 4608 2024-03-19T11:30:05.763 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-03-19T11:30:05.763 INFO:tasks.rados.rados.0.smithi038.stdout:559: finishing set_chunk to oid 10 2024-03-19T11:30:05.763 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 10 set_chunk 10 low_tier returned error code -95 offset: 823808 length: 1261056 2024-03-19T11:30:05.764 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 10 v 0 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-03-19T11:30:05.764 INFO:tasks.rados.rados.0.smithi038.stdout:553: done (15 left) 2024-03-19T11:30:05.764 INFO:tasks.rados.rados.0.smithi038.stdout:555: done (14 left) 2024-03-19T11:30:05.764 INFO:tasks.rados.rados.0.smithi038.stdout:556: done (13 left) 2024-03-19T11:30:05.764 INFO:tasks.rados.rados.0.smithi038.stdout:559: done (12 left) 2024-03-19T11:30:05.764 INFO:tasks.rados.rados.0.smithi038.stdout:564: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.764 INFO:tasks.rados.rados.0.smithi038.stdout:564: set_chunk oid 15 offset: 308224 length: 1129472 target oid smithi03829856-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 308224 snap -1 2024-03-19T11:30:05.764 INFO:tasks.rados.rados.0.smithi038.stdout:565: set_chunk oid 16 target oid 16 low_tier 2024-03-19T11:30:05.764 INFO:tasks.rados.rados.0.smithi038.stdout:565: set_chunk oid 16 offset: 1613312 length: 315392 target oid smithi03829856-16 low_tier offset: 1613312 snap -1 2024-03-19T11:30:05.764 INFO:tasks.rados.rados.0.smithi038.stdout:565 retry set_chunk ! 2024-03-19T11:30:05.764 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 14 2024-03-19T11:30:05.764 INFO:tasks.rados.rados.0.smithi038.stdout:560: finishing set_chunk to oid 11 2024-03-19T11:30:05.764 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 11 set_chunk 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1270272 length: 878080 2024-03-19T11:30:05.764 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 11 v 0 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-03-19T11:30:05.764 INFO:tasks.rados.rados.0.smithi038.stdout:563: finishing set_chunk to oid 14 2024-03-19T11:30:05.765 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 14 set_chunk 14 low_tier returned error code -95 offset: 345088 length: 1595904 2024-03-19T11:30:05.765 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 14 v 0 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-03-19T11:30:05.765 INFO:tasks.rados.rados.0.smithi038.stdout:560: done (13 left) 2024-03-19T11:30:05.765 INFO:tasks.rados.rados.0.smithi038.stdout:563: done (12 left) 2024-03-19T11:30:05.765 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 12 2024-03-19T11:30:05.765 INFO:tasks.rados.rados.0.smithi038.stdout:554: finishing set_chunk to oid 5 2024-03-19T11:30:05.765 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 5 set_chunk 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 586240 length: 875008 2024-03-19T11:30:05.765 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-03-19T11:30:05.765 INFO:tasks.rados.rados.0.smithi038.stdout:554: done (11 left) 2024-03-19T11:30:05.765 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 11 2024-03-19T11:30:05.765 INFO:tasks.rados.rados.0.smithi038.stdout:564: finishing set_chunk to oid 15 2024-03-19T11:30:05.765 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 308224 length: 1129472 2024-03-19T11:30:05.765 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-03-19T11:30:05.765 INFO:tasks.rados.rados.0.smithi038.stdout:564: done (10 left) 2024-03-19T11:30:05.766 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 10 2024-03-19T11:30:05.766 INFO:tasks.rados.rados.0.smithi038.stdout:565: finishing set_chunk to oid 16 2024-03-19T11:30:05.766 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 16 set_chunk 16 low_tier returned error code -95 offset: 1613312 length: 315392 2024-03-19T11:30:05.766 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-03-19T11:30:05.766 INFO:tasks.rados.rados.0.smithi038.stdout:565: done (9 left) 2024-03-19T11:30:05.766 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 9 2024-03-19T11:30:05.766 INFO:tasks.rados.rados.0.smithi038.stdout:516: finishing set_chunk to oid 17 2024-03-19T11:30:05.766 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 17 v 9 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-03-19T11:30:05.766 INFO:tasks.rados.rados.0.smithi038.stdout:561: finishing set_chunk to oid 12 2024-03-19T11:30:05.766 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 12 set_chunk 12 low_tier returned error code -95 offset: 1320448 length: 485888 2024-03-19T11:30:05.766 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 12 v 0 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-03-19T11:30:05.766 INFO:tasks.rados.rados.0.smithi038.stdout:557: finishing set_chunk to oid 8 2024-03-19T11:30:05.766 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 8 set_chunk 8 low_tier returned error code -95 offset: 1609728 length: 332800 2024-03-19T11:30:05.766 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-03-19T11:30:05.767 INFO:tasks.rados.rados.0.smithi038.stdout:516: done (8 left) 2024-03-19T11:30:05.767 INFO:tasks.rados.rados.0.smithi038.stdout:557: done (7 left) 2024-03-19T11:30:05.767 INFO:tasks.rados.rados.0.smithi038.stdout:561: done (6 left) 2024-03-19T11:30:05.767 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 6 2024-03-19T11:30:05.767 INFO:tasks.rados.rados.0.smithi038.stdout:562: finishing set_chunk to oid 13 2024-03-19T11:30:05.767 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 13 set_chunk 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 940544 length: 351744 2024-03-19T11:30:05.767 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 13 v 0 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-03-19T11:30:05.767 INFO:tasks.rados.rados.0.smithi038.stdout:562: done (5 left) 2024-03-19T11:30:05.767 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 5 2024-03-19T11:30:05.767 INFO:tasks.rados.rados.0.smithi038.stdout:520: finishing set_chunk to oid 21 2024-03-19T11:30:05.767 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 21 v 15 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-03-19T11:30:05.767 INFO:tasks.rados.rados.0.smithi038.stdout:520: done (4 left) 2024-03-19T11:30:05.767 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 4 2024-03-19T11:30:05.767 INFO:tasks.rados.rados.0.smithi038.stdout:519: finishing set_chunk to oid 20 2024-03-19T11:30:05.767 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 20 v 12 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-03-19T11:30:05.768 INFO:tasks.rados.rados.0.smithi038.stdout:519: done (3 left) 2024-03-19T11:30:05.768 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 3 2024-03-19T11:30:05.768 INFO:tasks.rados.rados.0.smithi038.stdout:536: finishing set_chunk to oid 37 2024-03-19T11:30:05.768 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 37 v 21 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-03-19T11:30:05.768 INFO:tasks.rados.rados.0.smithi038.stdout:536: done (2 left) 2024-03-19T11:30:05.768 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 2 2024-03-19T11:30:05.768 INFO:tasks.rados.rados.0.smithi038.stdout:534: finishing set_chunk to oid 35 2024-03-19T11:30:05.768 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 35 v 18 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-03-19T11:30:05.768 INFO:tasks.rados.rados.0.smithi038.stdout:534: done (1 left) 2024-03-19T11:30:05.768 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 1 2024-03-19T11:30:05.768 INFO:tasks.rados.rados.0.smithi038.stdout:558: finishing set_chunk to oid 9 2024-03-19T11:30:05.768 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 9 v 6 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-03-19T11:30:05.768 INFO:tasks.rados.rados.0.smithi038.stdout:558: done (0 left) 2024-03-19T11:30:05.768 INFO:tasks.rados.rados.0.smithi038.stdout:566: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.768 INFO:tasks.rados.rados.0.smithi038.stdout:566: set_chunk oid 17 offset: 1441792 length: 512000 target oid smithi03829856-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1441792 snap -1 2024-03-19T11:30:05.769 INFO:tasks.rados.rados.0.smithi038.stdout:567: set_chunk oid 18 target oid 18 low_tier 2024-03-19T11:30:05.769 INFO:tasks.rados.rados.0.smithi038.stdout:567: set_chunk oid 18 offset: 342528 length: 1383424 target oid smithi03829856-18 low_tier offset: 342528 snap -1 2024-03-19T11:30:05.769 INFO:tasks.rados.rados.0.smithi038.stdout:568: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.769 INFO:tasks.rados.rados.0.smithi038.stdout:568: set_chunk oid 19 offset: 1696256 length: 550400 target oid smithi03829856-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1696256 snap -1 2024-03-19T11:30:05.769 INFO:tasks.rados.rados.0.smithi038.stdout:569: set_chunk oid 20 target oid 20 low_tier 2024-03-19T11:30:05.769 INFO:tasks.rados.rados.0.smithi038.stdout:569: set_chunk oid 20 offset: 1801728 length: 394240 target oid smithi03829856-20 low_tier offset: 1801728 snap -1 2024-03-19T11:30:05.769 INFO:tasks.rados.rados.0.smithi038.stdout:570: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.769 INFO:tasks.rados.rados.0.smithi038.stdout:570: set_chunk oid 21 offset: 525824 length: 651264 target oid smithi03829856-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 525824 snap -1 2024-03-19T11:30:05.769 INFO:tasks.rados.rados.0.smithi038.stdout:571: set_chunk oid 22 target oid 22 low_tier 2024-03-19T11:30:05.769 INFO:tasks.rados.rados.0.smithi038.stdout:571: set_chunk oid 22 offset: 1526784 length: 262144 target oid smithi03829856-22 low_tier offset: 1526784 snap -1 2024-03-19T11:30:05.769 INFO:tasks.rados.rados.0.smithi038.stdout:572: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.769 INFO:tasks.rados.rados.0.smithi038.stdout:572: set_chunk oid 23 offset: 76800 length: 100864 target oid smithi03829856-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 76800 snap -1 2024-03-19T11:30:05.769 INFO:tasks.rados.rados.0.smithi038.stdout:573: set_chunk oid 24 target oid 24 low_tier 2024-03-19T11:30:05.769 INFO:tasks.rados.rados.0.smithi038.stdout:573: set_chunk oid 24 offset: 588800 length: 1712640 target oid smithi03829856-24 low_tier offset: 588800 snap -1 2024-03-19T11:30:05.770 INFO:tasks.rados.rados.0.smithi038.stdout:574: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.770 INFO:tasks.rados.rados.0.smithi038.stdout:574: set_chunk oid 25 offset: 513024 length: 819712 target oid smithi03829856-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 513024 snap -1 2024-03-19T11:30:05.770 INFO:tasks.rados.rados.0.smithi038.stdout:575: set_chunk oid 26 target oid 26 low_tier 2024-03-19T11:30:05.770 INFO:tasks.rados.rados.0.smithi038.stdout:575: set_chunk oid 26 offset: 1654272 length: 36352 target oid smithi03829856-26 low_tier offset: 1654272 snap -1 2024-03-19T11:30:05.770 INFO:tasks.rados.rados.0.smithi038.stdout:576: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.770 INFO:tasks.rados.rados.0.smithi038.stdout:576: set_chunk oid 27 offset: 320512 length: 1479168 target oid smithi03829856-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 320512 snap -1 2024-03-19T11:30:05.770 INFO:tasks.rados.rados.0.smithi038.stdout:577: set_chunk oid 28 target oid 28 low_tier 2024-03-19T11:30:05.770 INFO:tasks.rados.rados.0.smithi038.stdout:577: set_chunk oid 28 offset: 142848 length: 96768 target oid smithi03829856-28 low_tier offset: 142848 snap -1 2024-03-19T11:30:05.770 INFO:tasks.rados.rados.0.smithi038.stdout:578: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.770 INFO:tasks.rados.rados.0.smithi038.stdout:578: set_chunk oid 29 offset: 1206784 length: 887296 target oid smithi03829856-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1206784 snap -1 2024-03-19T11:30:05.770 INFO:tasks.rados.rados.0.smithi038.stdout:579: set_chunk oid 30 target oid 30 low_tier 2024-03-19T11:30:05.770 INFO:tasks.rados.rados.0.smithi038.stdout:579: set_chunk oid 30 offset: 662528 length: 1465856 target oid smithi03829856-30 low_tier offset: 662528 snap -1 2024-03-19T11:30:05.770 INFO:tasks.rados.rados.0.smithi038.stdout:580: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.770 INFO:tasks.rados.rados.0.smithi038.stdout:580: set_chunk oid 31 offset: 88064 length: 9728 target oid smithi03829856-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 88064 snap -1 2024-03-19T11:30:05.771 INFO:tasks.rados.rados.0.smithi038.stdout:581: set_chunk oid 32 target oid 32 low_tier 2024-03-19T11:30:05.771 INFO:tasks.rados.rados.0.smithi038.stdout:581: set_chunk oid 32 offset: 375808 length: 1124352 target oid smithi03829856-32 low_tier offset: 375808 snap -1 2024-03-19T11:30:05.771 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.771 INFO:tasks.rados.rados.0.smithi038.stdout:568: finishing set_chunk to oid 19 2024-03-19T11:30:05.771 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 19 set_chunk 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1696256 length: 550400 2024-03-19T11:30:05.771 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 19 v 0 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-03-19T11:30:05.771 INFO:tasks.rados.rados.0.smithi038.stdout:571: finishing set_chunk to oid 22 2024-03-19T11:30:05.771 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 22 set_chunk 22 low_tier returned error code -95 offset: 1526784 length: 262144 2024-03-19T11:30:05.771 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-03-19T11:30:05.771 INFO:tasks.rados.rados.0.smithi038.stdout:573: finishing set_chunk to oid 24 2024-03-19T11:30:05.771 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 24 set_chunk 24 low_tier returned error code -95 offset: 588800 length: 1712640 2024-03-19T11:30:05.771 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-03-19T11:30:05.771 INFO:tasks.rados.rados.0.smithi038.stdout:566: finishing set_chunk to oid 17 2024-03-19T11:30:05.771 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 17 set_chunk 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1441792 length: 512000 2024-03-19T11:30:05.772 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 17 v 0 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-03-19T11:30:05.772 INFO:tasks.rados.rados.0.smithi038.stdout:566: done (15 left) 2024-03-19T11:30:05.772 INFO:tasks.rados.rados.0.smithi038.stdout:568: done (14 left) 2024-03-19T11:30:05.772 INFO:tasks.rados.rados.0.smithi038.stdout:571: done (13 left) 2024-03-19T11:30:05.772 INFO:tasks.rados.rados.0.smithi038.stdout:573: done (12 left) 2024-03-19T11:30:05.772 INFO:tasks.rados.rados.0.smithi038.stdout:582: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.772 INFO:tasks.rados.rados.0.smithi038.stdout:582: set_chunk oid 33 offset: 512000 length: 1224704 target oid smithi03829856-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 512000 snap -1 2024-03-19T11:30:05.772 INFO:tasks.rados.rados.0.smithi038.stdout:583: set_chunk oid 34 target oid 34 low_tier 2024-03-19T11:30:05.772 INFO:tasks.rados.rados.0.smithi038.stdout:583: set_chunk oid 34 offset: 752128 length: 891904 target oid smithi03829856-34 low_tier offset: 752128 snap -1 2024-03-19T11:30:05.772 INFO:tasks.rados.rados.0.smithi038.stdout:584: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.772 INFO:tasks.rados.rados.0.smithi038.stdout:584: set_chunk oid 35 offset: 732672 length: 1589760 target oid smithi03829856-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 732672 snap -1 2024-03-19T11:30:05.772 INFO:tasks.rados.rados.0.smithi038.stdout:585: set_chunk oid 36 target oid 36 low_tier 2024-03-19T11:30:05.772 INFO:tasks.rados.rados.0.smithi038.stdout:585: set_chunk oid 36 offset: 1046528 length: 50688 target oid smithi03829856-36 low_tier offset: 1046528 snap -1 2024-03-19T11:30:05.772 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.773 INFO:tasks.rados.rados.0.smithi038.stdout:577: finishing set_chunk to oid 28 2024-03-19T11:30:05.773 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 28 set_chunk 28 low_tier returned error code -95 offset: 142848 length: 96768 2024-03-19T11:30:05.773 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 28 v 0 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-03-19T11:30:05.773 INFO:tasks.rados.rados.0.smithi038.stdout:567: finishing set_chunk to oid 18 2024-03-19T11:30:05.773 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 18 set_chunk 18 low_tier returned error code -95 offset: 342528 length: 1383424 2024-03-19T11:30:05.773 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 18 v 0 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-03-19T11:30:05.773 INFO:tasks.rados.rados.0.smithi038.stdout:576: finishing set_chunk to oid 27 2024-03-19T11:30:05.773 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 27 set_chunk 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 320512 length: 1479168 2024-03-19T11:30:05.773 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-03-19T11:30:05.773 INFO:tasks.rados.rados.0.smithi038.stdout:569: finishing set_chunk to oid 20 2024-03-19T11:30:05.773 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 20 set_chunk 20 low_tier returned error code -95 offset: 1801728 length: 394240 2024-03-19T11:30:05.773 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 20 v 0 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-03-19T11:30:05.773 INFO:tasks.rados.rados.0.smithi038.stdout:572: finishing set_chunk to oid 23 2024-03-19T11:30:05.773 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 23 set_chunk 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 76800 length: 100864 2024-03-19T11:30:05.773 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 23 v 0 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-03-19T11:30:05.774 INFO:tasks.rados.rados.0.smithi038.stdout:567: done (15 left) 2024-03-19T11:30:05.774 INFO:tasks.rados.rados.0.smithi038.stdout:569: done (14 left) 2024-03-19T11:30:05.774 INFO:tasks.rados.rados.0.smithi038.stdout:572: done (13 left) 2024-03-19T11:30:05.774 INFO:tasks.rados.rados.0.smithi038.stdout:576: done (12 left) 2024-03-19T11:30:05.774 INFO:tasks.rados.rados.0.smithi038.stdout:577: done (11 left) 2024-03-19T11:30:05.774 INFO:tasks.rados.rados.0.smithi038.stdout:586: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.774 INFO:tasks.rados.rados.0.smithi038.stdout:586: set_chunk oid 37 offset: 860160 length: 1038848 target oid smithi03829856-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 860160 snap -1 2024-03-19T11:30:05.774 INFO:tasks.rados.rados.0.smithi038.stdout:587: set_chunk oid 38 target oid 38 low_tier 2024-03-19T11:30:05.774 INFO:tasks.rados.rados.0.smithi038.stdout:587: set_chunk oid 38 offset: 865280 length: 739840 target oid smithi03829856-38 low_tier offset: 865280 snap -1 2024-03-19T11:30:05.774 INFO:tasks.rados.rados.0.smithi038.stdout:588: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.774 INFO:tasks.rados.rados.0.smithi038.stdout:588: set_chunk oid 39 offset: 1808896 length: 190976 target oid smithi03829856-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1808896 snap -1 2024-03-19T11:30:05.774 INFO:tasks.rados.rados.0.smithi038.stdout:589: set_chunk oid 40 target oid 40 low_tier 2024-03-19T11:30:05.774 INFO:tasks.rados.rados.0.smithi038.stdout:589: set_chunk oid 40 offset: 2166784 length: 283648 target oid smithi03829856-40 low_tier offset: 2166784 snap -1 2024-03-19T11:30:05.775 INFO:tasks.rados.rados.0.smithi038.stdout:590: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.775 INFO:tasks.rados.rados.0.smithi038.stdout:590: set_chunk oid 41 offset: 726016 length: 39424 target oid smithi03829856-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 726016 snap -1 2024-03-19T11:30:05.775 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.775 INFO:tasks.rados.rados.0.smithi038.stdout:581: finishing set_chunk to oid 32 2024-03-19T11:30:05.775 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 32 set_chunk 32 low_tier returned error code -95 offset: 375808 length: 1124352 2024-03-19T11:30:05.775 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-03-19T11:30:05.775 INFO:tasks.rados.rados.0.smithi038.stdout:581: done (15 left) 2024-03-19T11:30:05.775 INFO:tasks.rados.rados.0.smithi038.stdout:591: set_chunk oid 42 target oid 42 low_tier 2024-03-19T11:30:05.775 INFO:tasks.rados.rados.0.smithi038.stdout:591: set_chunk oid 42 offset: 19968 length: 665088 target oid smithi03829856-42 low_tier offset: 19968 snap -1 2024-03-19T11:30:05.775 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.775 INFO:tasks.rados.rados.0.smithi038.stdout:574: finishing set_chunk to oid 25 2024-03-19T11:30:05.775 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 25 set_chunk 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 513024 length: 819712 2024-03-19T11:30:05.775 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-03-19T11:30:05.775 INFO:tasks.rados.rados.0.smithi038.stdout:570: finishing set_chunk to oid 21 2024-03-19T11:30:05.775 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 21 set_chunk 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 525824 length: 651264 2024-03-19T11:30:05.776 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-03-19T11:30:05.776 INFO:tasks.rados.rados.0.smithi038.stdout:570: done (15 left) 2024-03-19T11:30:05.776 INFO:tasks.rados.rados.0.smithi038.stdout:574: done (14 left) 2024-03-19T11:30:05.776 INFO:tasks.rados.rados.0.smithi038.stdout:592: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.776 INFO:tasks.rados.rados.0.smithi038.stdout:592: set_chunk oid 43 offset: 218624 length: 500736 target oid smithi03829856-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 218624 snap -1 2024-03-19T11:30:05.776 INFO:tasks.rados.rados.0.smithi038.stdout:593: set_chunk oid 44 target oid 44 low_tier 2024-03-19T11:30:05.776 INFO:tasks.rados.rados.0.smithi038.stdout:593: set_chunk oid 44 offset: 525312 length: 1613824 target oid smithi03829856-44 low_tier offset: 525312 snap -1 2024-03-19T11:30:05.776 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.776 INFO:tasks.rados.rados.0.smithi038.stdout:575: finishing set_chunk to oid 26 2024-03-19T11:30:05.776 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 26 set_chunk 26 low_tier returned error code -95 offset: 1654272 length: 36352 2024-03-19T11:30:05.776 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-03-19T11:30:05.776 INFO:tasks.rados.rados.0.smithi038.stdout:578: finishing set_chunk to oid 29 2024-03-19T11:30:05.777 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 29 set_chunk 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1206784 length: 887296 2024-03-19T11:30:05.777 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 29 v 0 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-03-19T11:30:05.777 INFO:tasks.rados.rados.0.smithi038.stdout:579: finishing set_chunk to oid 30 2024-03-19T11:30:05.777 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 30 set_chunk 30 low_tier returned error code -95 offset: 662528 length: 1465856 2024-03-19T11:30:05.777 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 30 v 0 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-03-19T11:30:05.777 INFO:tasks.rados.rados.0.smithi038.stdout:587: finishing set_chunk to oid 38 2024-03-19T11:30:05.777 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 38 set_chunk 38 low_tier returned error code -95 offset: 865280 length: 739840 2024-03-19T11:30:05.777 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 38 v 0 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-03-19T11:30:05.777 INFO:tasks.rados.rados.0.smithi038.stdout:575: done (15 left) 2024-03-19T11:30:05.777 INFO:tasks.rados.rados.0.smithi038.stdout:578: done (14 left) 2024-03-19T11:30:05.777 INFO:tasks.rados.rados.0.smithi038.stdout:579: done (13 left) 2024-03-19T11:30:05.777 INFO:tasks.rados.rados.0.smithi038.stdout:587: done (12 left) 2024-03-19T11:30:05.777 INFO:tasks.rados.rados.0.smithi038.stdout:594: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.777 INFO:tasks.rados.rados.0.smithi038.stdout:594: set_chunk oid 45 offset: 54272 length: 1777152 target oid smithi03829856-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 54272 snap -1 2024-03-19T11:30:05.777 INFO:tasks.rados.rados.0.smithi038.stdout:595: set_chunk oid 46 target oid 46 low_tier 2024-03-19T11:30:05.778 INFO:tasks.rados.rados.0.smithi038.stdout:595: set_chunk oid 46 offset: 1346048 length: 853504 target oid smithi03829856-46 low_tier offset: 1346048 snap -1 2024-03-19T11:30:05.778 INFO:tasks.rados.rados.0.smithi038.stdout:596: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.778 INFO:tasks.rados.rados.0.smithi038.stdout:596: set_chunk oid 47 offset: 1767424 length: 167936 target oid smithi03829856-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1767424 snap -1 2024-03-19T11:30:05.778 INFO:tasks.rados.rados.0.smithi038.stdout:597: set_chunk oid 48 target oid 48 low_tier 2024-03-19T11:30:05.778 INFO:tasks.rados.rados.0.smithi038.stdout:597: set_chunk oid 48 offset: 712192 length: 1425408 target oid smithi03829856-48 low_tier offset: 712192 snap -1 2024-03-19T11:30:05.778 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:05.778 INFO:tasks.rados.rados.0.smithi038.stdout:590: finishing set_chunk to oid 41 2024-03-19T11:30:05.778 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 41 set_chunk 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 726016 length: 39424 2024-03-19T11:30:05.778 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 41 v 0 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-03-19T11:30:05.778 INFO:tasks.rados.rados.0.smithi038.stdout:582: finishing set_chunk to oid 33 2024-03-19T11:30:05.778 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 33 set_chunk 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 512000 length: 1224704 2024-03-19T11:30:05.778 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-03-19T11:30:05.778 INFO:tasks.rados.rados.0.smithi038.stdout:580: finishing set_chunk to oid 31 2024-03-19T11:30:05.779 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 31 set_chunk 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 88064 length: 9728 2024-03-19T11:30:05.779 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 31 v 0 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-03-19T11:30:05.779 INFO:tasks.rados.rados.0.smithi038.stdout:588: finishing set_chunk to oid 39 2024-03-19T11:30:05.779 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 39 set_chunk 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1808896 length: 190976 2024-03-19T11:30:05.779 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 39 v 0 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-03-19T11:30:05.779 INFO:tasks.rados.rados.0.smithi038.stdout:580: done (15 left) 2024-03-19T11:30:05.779 INFO:tasks.rados.rados.0.smithi038.stdout:582: done (14 left) 2024-03-19T11:30:05.779 INFO:tasks.rados.rados.0.smithi038.stdout:588: done (13 left) 2024-03-19T11:30:05.779 INFO:tasks.rados.rados.0.smithi038.stdout:590: done (12 left) 2024-03-19T11:30:05.779 INFO:tasks.rados.rados.0.smithi038.stdout:598: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.779 INFO:tasks.rados.rados.0.smithi038.stdout:598: set_chunk oid 49 offset: 1260032 length: 758272 target oid smithi03829856-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1260032 snap -1 2024-03-19T11:30:05.780 INFO:tasks.rados.rados.0.smithi038.stdout:599: set_chunk oid 50 target oid 50 low_tier 2024-03-19T11:30:05.780 INFO:tasks.rados.rados.0.smithi038.stdout:599: set_chunk oid 50 offset: 1341952 length: 647168 target oid smithi03829856-50 low_tier offset: 1341952 snap -1 2024-03-19T11:30:05.780 INFO:tasks.rados.rados.0.smithi038.stdout:591: finishing set_chunk to oid 42 2024-03-19T11:30:05.780 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 42 set_chunk 42 low_tier returned error code -95 offset: 19968 length: 665088 2024-03-19T11:30:05.780 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-03-19T11:30:05.780 INFO:tasks.rados.rados.0.smithi038.stdout:584: finishing set_chunk to oid 35 2024-03-19T11:30:05.780 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 35 set_chunk 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 732672 length: 1589760 2024-03-19T11:30:05.780 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 35 v 0 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-03-19T11:30:05.780 INFO:tasks.rados.rados.0.smithi038.stdout:584: done (13 left) 2024-03-19T11:30:05.780 INFO:tasks.rados.rados.0.smithi038.stdout:591: done (12 left) 2024-03-19T11:30:05.780 INFO:tasks.rados.rados.0.smithi038.stdout:600: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.780 INFO:tasks.rados.rados.0.smithi038.stdout:600: set_chunk oid 1 offset: 274944 length: 1252352 target oid smithi03829856-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 274944 snap -1 2024-03-19T11:30:05.780 INFO:tasks.rados.rados.0.smithi038.stdout:601 oid_not_in_use 37 oid_redirect_not_in_use 0 2024-03-19T11:30:05.780 INFO:tasks.rados.rados.0.smithi038.stdout:600 wait for manifest initialization 2024-03-19T11:30:05.781 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 13 2024-03-19T11:30:05.781 INFO:tasks.rados.rados.0.smithi038.stdout:595: finishing set_chunk to oid 46 2024-03-19T11:30:05.781 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 46 set_chunk 46 low_tier returned error code -95 offset: 1346048 length: 853504 2024-03-19T11:30:05.781 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-03-19T11:30:05.781 INFO:tasks.rados.rados.0.smithi038.stdout:593: finishing set_chunk to oid 44 2024-03-19T11:30:05.781 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 44 set_chunk 44 low_tier returned error code -95 offset: 525312 length: 1613824 2024-03-19T11:30:05.781 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 44 v 0 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-03-19T11:30:05.781 INFO:tasks.rados.rados.0.smithi038.stdout:586: finishing set_chunk to oid 37 2024-03-19T11:30:05.781 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 37 set_chunk 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 860160 length: 1038848 2024-03-19T11:30:05.781 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 37 v 0 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-03-19T11:30:05.781 INFO:tasks.rados.rados.0.smithi038.stdout:597: finishing set_chunk to oid 48 2024-03-19T11:30:05.781 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 48 set_chunk 48 low_tier returned error code -95 offset: 712192 length: 1425408 2024-03-19T11:30:05.781 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 48 v 0 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-03-19T11:30:05.781 INFO:tasks.rados.rados.0.smithi038.stdout:586: done (12 left) 2024-03-19T11:30:05.781 INFO:tasks.rados.rados.0.smithi038.stdout:593: done (11 left) 2024-03-19T11:30:05.782 INFO:tasks.rados.rados.0.smithi038.stdout:595: done (10 left) 2024-03-19T11:30:05.782 INFO:tasks.rados.rados.0.smithi038.stdout:597: done (9 left) 2024-03-19T11:30:05.782 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 9 2024-03-19T11:30:05.782 INFO:tasks.rados.rados.0.smithi038.stdout:598: finishing set_chunk to oid 49 2024-03-19T11:30:05.782 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 49 set_chunk 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1260032 length: 758272 2024-03-19T11:30:05.782 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-03-19T11:30:05.782 INFO:tasks.rados.rados.0.smithi038.stdout:598: done (8 left) 2024-03-19T11:30:05.782 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 8 2024-03-19T11:30:05.782 INFO:tasks.rados.rados.0.smithi038.stdout:596: finishing set_chunk to oid 47 2024-03-19T11:30:05.782 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 47 set_chunk 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1767424 length: 167936 2024-03-19T11:30:05.782 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-03-19T11:30:05.782 INFO:tasks.rados.rados.0.smithi038.stdout:596: done (7 left) 2024-03-19T11:30:05.783 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 7 2024-03-19T11:30:05.783 INFO:tasks.rados.rados.0.smithi038.stdout:583: finishing set_chunk to oid 34 2024-03-19T11:30:05.783 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 34 set_chunk 34 low_tier returned error code -95 offset: 752128 length: 891904 2024-03-19T11:30:05.783 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 34 v 0 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-03-19T11:30:05.783 INFO:tasks.rados.rados.0.smithi038.stdout:592: finishing set_chunk to oid 43 2024-03-19T11:30:05.783 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 43 set_chunk 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 218624 length: 500736 2024-03-19T11:30:05.783 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 43 v 0 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-03-19T11:30:05.783 INFO:tasks.rados.rados.0.smithi038.stdout:583: done (6 left) 2024-03-19T11:30:05.783 INFO:tasks.rados.rados.0.smithi038.stdout:592: done (5 left) 2024-03-19T11:30:05.783 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 5 2024-03-19T11:30:05.785 INFO:tasks.rados.rados.0.smithi038.stdout:600: finishing set_chunk to oid 1 2024-03-19T11:30:05.785 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 1 set_chunk 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 274944 length: 1252352 2024-03-19T11:30:05.785 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 1 v 0 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-03-19T11:30:05.785 INFO:tasks.rados.rados.0.smithi038.stdout:600: done (4 left) 2024-03-19T11:30:05.785 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 4 2024-03-19T11:30:05.785 INFO:tasks.rados.rados.0.smithi038.stdout:594: finishing set_chunk to oid 45 2024-03-19T11:30:05.786 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 45 set_chunk 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 54272 length: 1777152 2024-03-19T11:30:05.786 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 45 v 0 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-03-19T11:30:05.786 INFO:tasks.rados.rados.0.smithi038.stdout:594: done (3 left) 2024-03-19T11:30:05.786 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 3 2024-03-19T11:30:05.786 INFO:tasks.rados.rados.0.smithi038.stdout:599: finishing set_chunk to oid 50 2024-03-19T11:30:05.786 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 50 set_chunk 50 low_tier returned error code -95 offset: 1341952 length: 647168 2024-03-19T11:30:05.786 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 50 v 0 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-03-19T11:30:05.786 INFO:tasks.rados.rados.0.smithi038.stdout:599: done (2 left) 2024-03-19T11:30:05.786 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 2 2024-03-19T11:30:05.786 INFO:tasks.rados.rados.0.smithi038.stdout:585: finishing set_chunk to oid 36 2024-03-19T11:30:05.786 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 36 v 9 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-03-19T11:30:05.786 INFO:tasks.rados.rados.0.smithi038.stdout:585: done (1 left) 2024-03-19T11:30:05.786 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 1 2024-03-19T11:30:05.787 INFO:tasks.rados.rados.0.smithi038.stdout:589: finishing set_chunk to oid 40 2024-03-19T11:30:05.787 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 40 v 9 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-03-19T11:30:05.787 INFO:tasks.rados.rados.0.smithi038.stdout:589: done (0 left) 2024-03-19T11:30:05.787 INFO:tasks.rados.rados.0.smithi038.stdout:601 oid_not_in_use 50 oid_redirect_not_in_use 0 2024-03-19T11:30:05.787 INFO:tasks.rados.rados.0.smithi038.stdout: redirect_not_in_use: 51 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.787 INFO:tasks.rados.rados.0.smithi038.stdout: redirect_not_in_use: 52 low_tier 2024-03-19T11:30:05.787 INFO:tasks.rados.rados.0.smithi038.stdout: redirect_not_in_use: 53 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.787 INFO:tasks.rados.rados.0.smithi038.stdout: redirect_not_in_use: 54 low_tier 2024-03-19T11:30:05.787 INFO:tasks.rados.rados.0.smithi038.stdout: redirect_not_in_use: 55 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.788 INFO:tasks.rados.rados.0.smithi038.stdout: redirect_not_in_use: 56 low_tier 2024-03-19T11:30:05.788 INFO:tasks.rados.rados.0.smithi038.stdout: redirect_not_in_use: 57 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.788 INFO:tasks.rados.rados.0.smithi038.stdout: redirect_not_in_use: 58 low_tier 2024-03-19T11:30:05.788 INFO:tasks.rados.rados.0.smithi038.stdout: redirect_not_in_use: 59 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.788 INFO:tasks.rados.rados.0.smithi038.stdout: redirect_not_in_use: 60 low_tier 2024-03-19T11:30:05.788 INFO:tasks.rados.rados.0.smithi038.stdout: redirect_not_in_use: 61 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.788 INFO:tasks.rados.rados.0.smithi038.stdout: redirect_not_in_use: 62 low_tier 2024-03-19T11:30:05.788 INFO:tasks.rados.rados.0.smithi038.stdout: redirect_not_in_use: 63 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.788 INFO:tasks.rados.rados.0.smithi038.stdout: redirect_not_in_use: 64 low_tier 2024-03-19T11:30:05.788 INFO:tasks.rados.rados.0.smithi038.stdout: redirect_not_in_use: 65 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.788 INFO:tasks.rados.rados.0.smithi038.stdout: redirect_not_in_use: 66 low_tier 2024-03-19T11:30:05.789 INFO:tasks.rados.rados.0.smithi038.stdout: redirect_not_in_use: 67 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.789 INFO:tasks.rados.rados.0.smithi038.stdout: redirect_not_in_use: 68 low_tier 2024-03-19T11:30:05.789 INFO:tasks.rados.rados.0.smithi038.stdout: redirect_not_in_use: 69 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.789 INFO:tasks.rados.rados.0.smithi038.stdout: redirect_not_in_use: 70 low_tier 2024-03-19T11:30:05.789 INFO:tasks.rados.rados.0.smithi038.stdout: redirect_not_in_use: 71 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.789 INFO:tasks.rados.rados.0.smithi038.stdout: redirect_not_in_use: 72 low_tier 2024-03-19T11:30:05.789 INFO:tasks.rados.rados.0.smithi038.stdout: redirect_not_in_use: 73 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.789 INFO:tasks.rados.rados.0.smithi038.stdout: redirect_not_in_use: 74 low_tier 2024-03-19T11:30:05.789 INFO:tasks.rados.rados.0.smithi038.stdout: redirect_not_in_use: 75 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.789 INFO:tasks.rados.rados.0.smithi038.stdout: redirect_not_in_use: 76 low_tier 2024-03-19T11:30:05.790 INFO:tasks.rados.rados.0.smithi038.stdout: redirect_not_in_use: 77 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.790 INFO:tasks.rados.rados.0.smithi038.stdout: redirect_not_in_use: 78 low_tier 2024-03-19T11:30:05.790 INFO:tasks.rados.rados.0.smithi038.stdout: redirect_not_in_use: 79 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.790 INFO:tasks.rados.rados.0.smithi038.stdout: redirect_not_in_use: 80 low_tier 2024-03-19T11:30:05.790 INFO:tasks.rados.rados.0.smithi038.stdout: redirect_not_in_use: 81 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.790 INFO:tasks.rados.rados.0.smithi038.stdout: redirect_not_in_use: 82 low_tier 2024-03-19T11:30:05.790 INFO:tasks.rados.rados.0.smithi038.stdout: redirect_not_in_use: 83 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.790 INFO:tasks.rados.rados.0.smithi038.stdout: redirect_not_in_use: 84 low_tier 2024-03-19T11:30:05.790 INFO:tasks.rados.rados.0.smithi038.stdout: redirect_not_in_use: 85 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.790 INFO:tasks.rados.rados.0.smithi038.stdout: redirect_not_in_use: 86 low_tier 2024-03-19T11:30:05.790 INFO:tasks.rados.rados.0.smithi038.stdout: redirect_not_in_use: 87 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.790 INFO:tasks.rados.rados.0.smithi038.stdout: redirect_not_in_use: 88 low_tier 2024-03-19T11:30:05.790 INFO:tasks.rados.rados.0.smithi038.stdout: redirect_not_in_use: 89 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.790 INFO:tasks.rados.rados.0.smithi038.stdout: redirect_not_in_use: 90 low_tier 2024-03-19T11:30:05.791 INFO:tasks.rados.rados.0.smithi038.stdout: redirect_not_in_use: 91 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.791 INFO:tasks.rados.rados.0.smithi038.stdout: redirect_not_in_use: 92 low_tier 2024-03-19T11:30:05.791 INFO:tasks.rados.rados.0.smithi038.stdout: redirect_not_in_use: 93 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.791 INFO:tasks.rados.rados.0.smithi038.stdout: redirect_not_in_use: 94 low_tier 2024-03-19T11:30:05.791 INFO:tasks.rados.rados.0.smithi038.stdout: redirect_not_in_use: 95 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.791 INFO:tasks.rados.rados.0.smithi038.stdout: redirect_not_in_use: 96 low_tier 2024-03-19T11:30:05.791 INFO:tasks.rados.rados.0.smithi038.stdout: redirect_not_in_use: 97 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.791 INFO:tasks.rados.rados.0.smithi038.stdout: redirect_not_in_use: 98 low_tier 2024-03-19T11:30:05.791 INFO:tasks.rados.rados.0.smithi038.stdout: redirect_not_in_use: 99 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-03-19T11:30:05.791 INFO:tasks.rados.rados.0.smithi038.stdout: redirect_not_in_use: 100 low_tier 2024-03-19T11:30:05.791 INFO:tasks.rados.rados.0.smithi038.stdout:601: write oid 43 current snap is 0 2024-03-19T11:30:05.791 INFO:tasks.rados.rados.0.smithi038.stdout:601: seq_num 50 ranges {440350=460568,1309629=542666,2323973=516377} 2024-03-19T11:30:05.791 INFO:tasks.rados.rados.0.smithi038.stdout:601: writing smithi03829856-43 from 440350 to 900918 tid 1 2024-03-19T11:30:05.791 INFO:tasks.rados.rados.0.smithi038.stdout:601: writing smithi03829856-43 from 1309629 to 1852295 tid 2 2024-03-19T11:30:05.792 INFO:tasks.rados.rados.0.smithi038.stdout:601: writing smithi03829856-43 from 2323973 to 2840350 tid 3 2024-03-19T11:30:05.792 INFO:tasks.rados.rados.0.smithi038.stdout:602: read oid 22 snap -1 2024-03-19T11:30:05.792 INFO:tasks.rados.rados.0.smithi038.stdout:602: expect (ObjNum 21 snap 0 seq_num 21) 2024-03-19T11:30:05.792 INFO:tasks.rados.rados.0.smithi038.stdout:603: tier_evict oid 25 2024-03-19T11:30:05.792 INFO:tasks.rados.rados.0.smithi038.stdout:603: tier_evict oid 25 snap -1 2024-03-19T11:30:05.792 INFO:tasks.rados.rados.0.smithi038.stdout:601: finishing write tid 1 to smithi03829856-43 2024-03-19T11:30:05.792 INFO:tasks.rados.rados.0.smithi038.stdout:604: write oid 1 current snap is 0 2024-03-19T11:30:05.792 INFO:tasks.rados.rados.0.smithi038.stdout:604: seq_num 51 ranges {457157=535817,1768111=402677} 2024-03-19T11:30:05.792 INFO:tasks.rados.rados.0.smithi038.stdout:604: writing smithi03829856-1 from 457157 to 992974 tid 1 2024-03-19T11:30:05.792 INFO:tasks.rados.rados.0.smithi038.stdout:604: writing smithi03829856-1 from 1768111 to 2170788 tid 2 2024-03-19T11:30:05.792 INFO:tasks.rados.rados.0.smithi038.stdout:605: write (excl) oid 42 current snap is 0 2024-03-19T11:30:05.792 INFO:tasks.rados.rados.0.smithi038.stdout:605: seq_num 52 ranges {473964=611066,1826593=546335,2778819=95145} 2024-03-19T11:30:05.792 INFO:tasks.rados.rados.0.smithi038.stdout:605: writing smithi03829856-42 from 473964 to 1085030 tid 1 2024-03-19T11:30:05.795 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-19T11:30:05.795 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:05.807+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:05.796 INFO:tasks.rados.rados.0.smithi038.stdout:605: writing smithi03829856-42 from 1826593 to 2372928 tid 2 2024-03-19T11:30:05.796 INFO:tasks.rados.rados.0.smithi038.stdout:605: writing smithi03829856-42 from 2778819 to 2873964 tid 3 2024-03-19T11:30:05.796 INFO:tasks.rados.rados.0.smithi038.stdout:606: tier_evict oid 29 2024-03-19T11:30:05.796 INFO:tasks.rados.rados.0.smithi038.stdout:606: tier_evict oid 29 snap -1 2024-03-19T11:30:05.796 INFO:tasks.rados.rados.0.smithi038.stdout:601: finishing write tid 2 to smithi03829856-43 2024-03-19T11:30:05.797 INFO:tasks.rados.rados.0.smithi038.stdout:607: set_chunk oid 20 target oid 2024-03-19T11:30:05.797 INFO:tasks.rados.rados.0.smithi038.stdout:607: set_chunk oid 20 offset: 844288 length: 533504 target oid e3a26bf9230b1b95b8cf1a4cb01909236bf2bf3fd59b083b2291d278ab98264d offset: 0 snap -1 2024-03-19T11:30:05.797 INFO:tasks.rados.rados.0.smithi038.stdout:608: write oid 6 current snap is 0 2024-03-19T11:30:05.797 INFO:tasks.rados.rados.0.smithi038.stdout:608: seq_num 53 ranges {490771=686315,1885075=406346} 2024-03-19T11:30:05.797 INFO:tasks.rados.rados.0.smithi038.stdout:608: writing smithi03829856-6 from 490771 to 1177086 tid 1 2024-03-19T11:30:05.797 INFO:tasks.rados.rados.0.smithi038.stdout:608: writing smithi03829856-6 from 1885075 to 2291421 tid 2 2024-03-19T11:30:05.797 INFO:tasks.rados.rados.0.smithi038.stdout:609: rollback oid 16 current snap is 0 2024-03-19T11:30:05.797 INFO:tasks.rados.rados.0.smithi038.stdout:609: done (8 left) 2024-03-19T11:30:05.797 INFO:tasks.rados.rados.0.smithi038.stdout:610: read oid 8 snap -1 2024-03-19T11:30:05.797 INFO:tasks.rados.rados.0.smithi038.stdout:610: expect (ObjNum 7 snap 0 seq_num 7) 2024-03-19T11:30:05.797 INFO:tasks.rados.rados.0.smithi038.stdout:611: read oid 30 snap -1 2024-03-19T11:30:05.798 INFO:tasks.rados.rados.0.smithi038.stdout:611: expect (ObjNum 29 snap 0 seq_num 29) 2024-03-19T11:30:05.798 INFO:tasks.rados.rados.0.smithi038.stdout:612: read oid 50 snap -1 2024-03-19T11:30:05.798 INFO:tasks.rados.rados.0.smithi038.stdout:612: expect (ObjNum 49 snap 0 seq_num 49) 2024-03-19T11:30:05.798 INFO:tasks.rados.rados.0.smithi038.stdout:613: rollback oid 31 current snap is 0 2024-03-19T11:30:05.798 INFO:tasks.rados.rados.0.smithi038.stdout:613: done (11 left) 2024-03-19T11:30:05.798 INFO:tasks.rados.rados.0.smithi038.stdout:614: set_chunk oid 26 target oid 2024-03-19T11:30:05.798 INFO:tasks.rados.rados.0.smithi038.stdout:614: set_chunk oid 26 offset: 290816 length: 1352704 target oid 5bb2431930b721582011e31c3f26cdf2ee6d128770b3d28569cebfd0aec3c4a5 offset: 0 snap -1 2024-03-19T11:30:05.798 INFO:tasks.rados.rados.0.smithi038.stdout:603: got (22) Invalid argument 2024-03-19T11:30:05.798 INFO:tasks.rados.rados.0.smithi038.stdout:603: done (11 left) 2024-03-19T11:30:05.798 INFO:tasks.rados.rados.0.smithi038.stdout:615: read oid 31 snap -1 2024-03-19T11:30:05.798 INFO:tasks.rados.rados.0.smithi038.stdout:615: expect (ObjNum 30 snap 0 seq_num 30) 2024-03-19T11:30:05.798 INFO:tasks.rados.rados.0.smithi038.stdout:616: snap_create 2024-03-19T11:30:05.798 INFO:tasks.rados.rados.0.smithi038.stdout:602: expect (ObjNum 21 snap 0 seq_num 21) 2024-03-19T11:30:05.799 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-03-19T11:30:05.800 INFO:tasks.rados.rados.0.smithi038.stdout:601: finishing write tid 3 to smithi03829856-43 2024-03-19T11:30:05.800 INFO:tasks.rados.rados.0.smithi038.stdout:601: finishing write tid 4 to smithi03829856-43 2024-03-19T11:30:05.800 INFO:tasks.rados.rados.0.smithi038.stdout:601: finishing write tid 5 to smithi03829856-43 2024-03-19T11:30:05.800 INFO:tasks.rados.rados.0.smithi038.stdout:601: oid 43 updating version 0 to 20 2024-03-19T11:30:05.800 INFO:tasks.rados.rados.0.smithi038.stdout:601: oid 43 version 20 is already newer than 18 2024-03-19T11:30:05.800 INFO:tasks.rados.rados.0.smithi038.stdout:601: oid 43 updating version 20 to 21 2024-03-19T11:30:05.800 INFO:tasks.rados.rados.0.smithi038.stdout:601: oid 43 version 21 is already newer than 19 2024-03-19T11:30:05.801 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 43 v 21 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-03-19T11:30:05.801 INFO:tasks.rados.rados.0.smithi038.stdout:601: left oid 43 (ObjNum 50 snap 0 seq_num 50) 2024-03-19T11:30:05.801 INFO:tasks.rados.rados.0.smithi038.stdout:604: finishing write tid 1 to smithi03829856-1 2024-03-19T11:30:05.801 INFO:tasks.rados.rados.0.smithi038.stdout:604: finishing write tid 2 to smithi03829856-1 2024-03-19T11:30:05.801 INFO:tasks.rados.rados.0.smithi038.stdout:604: finishing write tid 3 to smithi03829856-1 2024-03-19T11:30:05.801 INFO:tasks.rados.rados.0.smithi038.stdout:604: finishing write tid 4 to smithi03829856-1 2024-03-19T11:30:05.801 INFO:tasks.rados.rados.0.smithi038.stdout:604: oid 1 updating version 0 to 24 2024-03-19T11:30:05.801 INFO:tasks.rados.rados.0.smithi038.stdout:604: oid 1 version 24 is already newer than 23 2024-03-19T11:30:05.801 INFO:tasks.rados.rados.0.smithi038.stdout:604: oid 1 version 24 is already newer than 22 2024-03-19T11:30:05.801 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 1 v 24 (ObjNum 51 snap 0 seq_num 51) dirty exists 2024-03-19T11:30:05.801 INFO:tasks.rados.rados.0.smithi038.stdout:604: left oid 1 (ObjNum 51 snap 0 seq_num 51) 2024-03-19T11:30:05.801 INFO:tasks.rados.rados.0.smithi038.stdout:605: finishing write tid 1 to smithi03829856-42 2024-03-19T11:30:05.801 INFO:tasks.rados.rados.0.smithi038.stdout:606: got (22) Invalid argument 2024-03-19T11:30:05.801 INFO:tasks.rados.rados.0.smithi038.stdout:605: finishing write tid 2 to smithi03829856-42 2024-03-19T11:30:05.802 INFO:tasks.rados.rados.0.smithi038.stdout:605: finishing write tid 3 to smithi03829856-42 2024-03-19T11:30:05.802 INFO:tasks.rados.rados.0.smithi038.stdout:605: finishing write tid 4 to smithi03829856-42 2024-03-19T11:30:05.802 INFO:tasks.rados.rados.0.smithi038.stdout:605: finishing write tid 5 to smithi03829856-42 2024-03-19T11:30:05.802 INFO:tasks.rados.rados.0.smithi038.stdout:605: oid 42 updating version 0 to 26 2024-03-19T11:30:05.802 INFO:tasks.rados.rados.0.smithi038.stdout:605: oid 42 version 26 is already newer than 25 2024-03-19T11:30:05.802 INFO:tasks.rados.rados.0.smithi038.stdout:605: oid 42 updating version 26 to 27 2024-03-19T11:30:05.802 INFO:tasks.rados.rados.0.smithi038.stdout:605: oid 42 updating version 27 to 28 2024-03-19T11:30:05.802 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 42 v 28 (ObjNum 52 snap 0 seq_num 52) dirty exists 2024-03-19T11:30:05.802 INFO:tasks.rados.rados.0.smithi038.stdout:605: left oid 42 (ObjNum 52 snap 0 seq_num 52) 2024-03-19T11:30:05.802 INFO:tasks.rados.rados.0.smithi038.stdout:607: finishing set_chunk to oid 20 2024-03-19T11:30:05.802 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 20 set_chunk returned error code -95 offset: 844288 length: 533504 2024-03-19T11:30:05.802 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 20 v 0 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-03-19T11:30:05.803 INFO:tasks.rados.rados.0.smithi038.stdout:608: finishing write tid 1 to smithi03829856-6 2024-03-19T11:30:05.803 INFO:tasks.rados.rados.0.smithi038.stdout:608: finishing write tid 2 to smithi03829856-6 2024-03-19T11:30:05.803 INFO:tasks.rados.rados.0.smithi038.stdout:608: finishing write tid 3 to smithi03829856-6 2024-03-19T11:30:05.803 INFO:tasks.rados.rados.0.smithi038.stdout:608: finishing write tid 4 to smithi03829856-6 2024-03-19T11:30:05.803 INFO:tasks.rados.rados.0.smithi038.stdout:608: oid 6 updating version 0 to 29 2024-03-19T11:30:05.803 INFO:tasks.rados.rados.0.smithi038.stdout:608: oid 6 updating version 29 to 30 2024-03-19T11:30:05.803 INFO:tasks.rados.rados.0.smithi038.stdout:608: oid 6 updating version 30 to 31 2024-03-19T11:30:05.803 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 6 v 31 (ObjNum 53 snap 0 seq_num 53) dirty exists 2024-03-19T11:30:05.803 INFO:tasks.rados.rados.0.smithi038.stdout:608: left oid 6 (ObjNum 53 snap 0 seq_num 53) 2024-03-19T11:30:05.803 INFO:tasks.rados.rados.0.smithi038.stdout:611: expect (ObjNum 29 snap 0 seq_num 29) 2024-03-19T11:30:05.845 INFO:tasks.rados.rados.0.smithi038.stdout:610: expect (ObjNum 7 snap 0 seq_num 7) 2024-03-19T11:30:05.895 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:05.903+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:05.904 INFO:tasks.rados.rados.0.smithi038.stdout:612: expect (ObjNum 49 snap 0 seq_num 49) 2024-03-19T11:30:05.977 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:05.977 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:05.989 INFO:tasks.rados.rados.0.smithi038.stdout:614: finishing set_chunk to oid 26 2024-03-19T11:30:05.990 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 26 set_chunk returned error code -95 offset: 290816 length: 1352704 2024-03-19T11:30:05.990 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-03-19T11:30:05.990 INFO:tasks.rados.rados.0.smithi038.stdout:615: expect (ObjNum 30 snap 0 seq_num 30) 2024-03-19T11:30:05.991 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-19T11:30:06.033 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:06.043+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:06.065 INFO:tasks.rados.rados.0.smithi038.stdout:601: done (12 left) 2024-03-19T11:30:06.065 INFO:tasks.rados.rados.0.smithi038.stdout:602: done (11 left) 2024-03-19T11:30:06.065 INFO:tasks.rados.rados.0.smithi038.stdout:604: done (10 left) 2024-03-19T11:30:06.065 INFO:tasks.rados.rados.0.smithi038.stdout:605: done (9 left) 2024-03-19T11:30:06.066 INFO:tasks.rados.rados.0.smithi038.stdout:606: done (8 left) 2024-03-19T11:30:06.066 INFO:tasks.rados.rados.0.smithi038.stdout:607: done (7 left) 2024-03-19T11:30:06.066 INFO:tasks.rados.rados.0.smithi038.stdout:608: done (6 left) 2024-03-19T11:30:06.066 INFO:tasks.rados.rados.0.smithi038.stdout:610: done (5 left) 2024-03-19T11:30:06.066 INFO:tasks.rados.rados.0.smithi038.stdout:611: done (4 left) 2024-03-19T11:30:06.066 INFO:tasks.rados.rados.0.smithi038.stdout:612: done (3 left) 2024-03-19T11:30:06.066 INFO:tasks.rados.rados.0.smithi038.stdout:614: done (2 left) 2024-03-19T11:30:06.066 INFO:tasks.rados.rados.0.smithi038.stdout:615: done (1 left) 2024-03-19T11:30:06.066 INFO:tasks.rados.rados.0.smithi038.stdout:616: done (0 left) 2024-03-19T11:30:06.066 INFO:tasks.rados.rados.0.smithi038.stdout:617: set_chunk oid 48 target oid 2024-03-19T11:30:06.067 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:06.075+0000 7f1837407640 -1 osd.6 28 handle_osd_map injecting map crc failure 2024-03-19T11:30:06.097 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:06.107+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:06.100 INFO:tasks.rados.rados.0.smithi038.stdout:617: set_chunk oid 48 offset: 486912 length: 2098176 target oid b1fb925e39ab40886d36b70ff1c8c4f867ef6cb2ef150fbe90de57d057dfec36 offset: 0 snap -1 2024-03-19T11:30:06.100 INFO:tasks.rados.rados.0.smithi038.stdout:618: read oid 49 snap -1 2024-03-19T11:30:06.100 INFO:tasks.rados.rados.0.smithi038.stdout:618: expect (ObjNum 48 snap 0 seq_num 48) 2024-03-19T11:30:06.100 INFO:tasks.rados.rados.0.smithi038.stdout:619: write oid 28 current snap is 1 2024-03-19T11:30:06.100 INFO:tasks.rados.rados.0.smithi038.stdout:619: seq_num 54 ranges {507578=477917,1543557=550004,2833665=73913} 2024-03-19T11:30:06.116 INFO:tasks.rados.rados.0.smithi038.stdout:619: writing smithi03829856-28 from 507578 to 985495 tid 1 2024-03-19T11:30:06.136 INFO:tasks.rados.rados.0.smithi038.stdout:619: writing smithi03829856-28 from 1543557 to 2093561 tid 2 2024-03-19T11:30:06.147 INFO:tasks.rados.rados.0.smithi038.stdout:619: writing smithi03829856-28 from 2833665 to 2907578 tid 3 2024-03-19T11:30:06.147 INFO:tasks.rados.rados.0.smithi038.stdout:620: write oid 10 current snap is 1 2024-03-19T11:30:06.147 INFO:tasks.rados.rados.0.smithi038.stdout:620: seq_num 55 ranges {524385=553166,1602039=410015,2577441=346944} 2024-03-19T11:30:06.160 INFO:tasks.rados.rados.0.smithi038.stdout:620: writing smithi03829856-10 from 524385 to 1077551 tid 1 2024-03-19T11:30:06.163 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:06.163 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:06.170 INFO:tasks.rados.rados.0.smithi038.stdout:620: writing smithi03829856-10 from 1602039 to 2012054 tid 2 2024-03-19T11:30:06.178 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-19T11:30:06.179 INFO:tasks.rados.rados.0.smithi038.stdout:620: writing smithi03829856-10 from 2577441 to 2924385 tid 3 2024-03-19T11:30:06.180 INFO:tasks.rados.rados.0.smithi038.stdout:621: write (excl) oid 31 current snap is 1 2024-03-19T11:30:06.180 INFO:tasks.rados.rados.0.smithi038.stdout:621: seq_num 56 ranges {541192=628415,1660521=553673,2888511=52681} 2024-03-19T11:30:06.189 INFO:tasks.rados.rados.0.smithi038.stdout:621: writing smithi03829856-31 from 541192 to 1169607 tid 1 2024-03-19T11:30:06.197 INFO:tasks.rados.rados.0.smithi038.stdout:621: writing smithi03829856-31 from 1660521 to 2214194 tid 2 2024-03-19T11:30:06.200 INFO:tasks.rados.rados.0.smithi038.stdout:621: writing smithi03829856-31 from 2888511 to 2941192 tid 3 2024-03-19T11:30:06.200 INFO:tasks.rados.rados.0.smithi038.stdout:622: set_chunk oid 35 target oid 2024-03-19T11:30:06.221 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:06.231+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:06.224 INFO:tasks.rados.rados.0.smithi038.stdout:622: set_chunk oid 35 offset: 386560 length: 2058752 target oid da5b2d5e0c07b826bee6bf639955e5681011e91ae1a36e413838428871f3e757 offset: 0 snap -1 2024-03-19T11:30:06.224 INFO:tasks.rados.rados.0.smithi038.stdout:623: read oid 39 snap -1 2024-03-19T11:30:06.224 INFO:tasks.rados.rados.0.smithi038.stdout:623: expect (ObjNum 38 snap 0 seq_num 38) 2024-03-19T11:30:06.224 INFO:tasks.rados.rados.0.smithi038.stdout:624: tier_promote oid 38 2024-03-19T11:30:06.224 INFO:tasks.rados.rados.0.smithi038.stdout:617: finishing set_chunk to oid 48 2024-03-19T11:30:06.224 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 48 set_chunk returned error code -95 offset: 486912 length: 2098176 2024-03-19T11:30:06.224 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 48 v 0 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-03-19T11:30:06.224 INFO:tasks.rados.rados.0.smithi038.stdout:618: expect (ObjNum 48 snap 0 seq_num 48) 2024-03-19T11:30:06.299 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:06.307+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:06.346 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:06.346 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:06.360 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-19T11:30:06.372 INFO:tasks.rados.rados.0.smithi038.stdout:619: finishing write tid 1 to smithi03829856-28 2024-03-19T11:30:06.372 INFO:tasks.rados.rados.0.smithi038.stdout:619: finishing write tid 2 to smithi03829856-28 2024-03-19T11:30:06.372 INFO:tasks.rados.rados.0.smithi038.stdout:619: finishing write tid 3 to smithi03829856-28 2024-03-19T11:30:06.372 INFO:tasks.rados.rados.0.smithi038.stdout:619: finishing write tid 4 to smithi03829856-28 2024-03-19T11:30:06.372 INFO:tasks.rados.rados.0.smithi038.stdout:619: finishing write tid 5 to smithi03829856-28 2024-03-19T11:30:06.372 INFO:tasks.rados.rados.0.smithi038.stdout:619: oid 28 updating version 0 to 11 2024-03-19T11:30:06.372 INFO:tasks.rados.rados.0.smithi038.stdout:619: oid 28 updating version 11 to 13 2024-03-19T11:30:06.373 INFO:tasks.rados.rados.0.smithi038.stdout:619: oid 28 version 13 is already newer than 10 2024-03-19T11:30:06.373 INFO:tasks.rados.rados.0.smithi038.stdout:619: oid 28 version 13 is already newer than 12 2024-03-19T11:30:06.373 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 28 v 13 (ObjNum 54 snap 1 seq_num 54) dirty exists 2024-03-19T11:30:06.373 INFO:tasks.rados.rados.0.smithi038.stdout:619: left oid 28 (ObjNum 54 snap 1 seq_num 54) 2024-03-19T11:30:06.373 INFO:tasks.rados.rados.0.smithi038.stdout:620: finishing write tid 1 to smithi03829856-10 2024-03-19T11:30:06.373 INFO:tasks.rados.rados.0.smithi038.stdout:620: finishing write tid 2 to smithi03829856-10 2024-03-19T11:30:06.373 INFO:tasks.rados.rados.0.smithi038.stdout:620: finishing write tid 3 to smithi03829856-10 2024-03-19T11:30:06.373 INFO:tasks.rados.rados.0.smithi038.stdout:620: finishing write tid 4 to smithi03829856-10 2024-03-19T11:30:06.373 INFO:tasks.rados.rados.0.smithi038.stdout:620: finishing write tid 5 to smithi03829856-10 2024-03-19T11:30:06.373 INFO:tasks.rados.rados.0.smithi038.stdout:620: oid 10 updating version 0 to 25 2024-03-19T11:30:06.373 INFO:tasks.rados.rados.0.smithi038.stdout:620: oid 10 updating version 25 to 26 2024-03-19T11:30:06.373 INFO:tasks.rados.rados.0.smithi038.stdout:620: oid 10 version 26 is already newer than 23 2024-03-19T11:30:06.373 INFO:tasks.rados.rados.0.smithi038.stdout:620: oid 10 version 26 is already newer than 24 2024-03-19T11:30:06.373 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 10 v 26 (ObjNum 55 snap 1 seq_num 55) dirty exists 2024-03-19T11:30:06.373 INFO:tasks.rados.rados.0.smithi038.stdout:620: left oid 10 (ObjNum 55 snap 1 seq_num 55) 2024-03-19T11:30:06.374 INFO:tasks.rados.rados.0.smithi038.stdout:621: finishing write tid 1 to smithi03829856-31 2024-03-19T11:30:06.374 INFO:tasks.rados.rados.0.smithi038.stdout:621: finishing write tid 2 to smithi03829856-31 2024-03-19T11:30:06.374 INFO:tasks.rados.rados.0.smithi038.stdout:621: finishing write tid 3 to smithi03829856-31 2024-03-19T11:30:06.374 INFO:tasks.rados.rados.0.smithi038.stdout:621: finishing write tid 4 to smithi03829856-31 2024-03-19T11:30:06.374 INFO:tasks.rados.rados.0.smithi038.stdout:621: finishing write tid 5 to smithi03829856-31 2024-03-19T11:30:06.374 INFO:tasks.rados.rados.0.smithi038.stdout:621: oid 31 updating version 0 to 25 2024-03-19T11:30:06.374 INFO:tasks.rados.rados.0.smithi038.stdout:621: oid 31 version 25 is already newer than 24 2024-03-19T11:30:06.374 INFO:tasks.rados.rados.0.smithi038.stdout:621: oid 31 updating version 25 to 26 2024-03-19T11:30:06.374 INFO:tasks.rados.rados.0.smithi038.stdout:621: oid 31 version 26 is already newer than 23 2024-03-19T11:30:06.374 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 31 v 26 (ObjNum 56 snap 1 seq_num 56) dirty exists 2024-03-19T11:30:06.374 INFO:tasks.rados.rados.0.smithi038.stdout:621: left oid 31 (ObjNum 56 snap 1 seq_num 56) 2024-03-19T11:30:06.374 INFO:tasks.rados.rados.0.smithi038.stdout:622: finishing set_chunk to oid 35 2024-03-19T11:30:06.374 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 35 set_chunk returned error code -95 offset: 386560 length: 2058752 2024-03-19T11:30:06.374 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 35 v 0 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-03-19T11:30:06.374 INFO:tasks.rados.rados.0.smithi038.stdout:624: got (0) Success 2024-03-19T11:30:06.375 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 38 v 12 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-03-19T11:30:06.375 INFO:tasks.rados.rados.0.smithi038.stdout:623: expect (ObjNum 38 snap 0 seq_num 38) 2024-03-19T11:30:06.405 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:06.415+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:06.458 INFO:tasks.rados.rados.0.smithi038.stdout:617: done (7 left) 2024-03-19T11:30:06.458 INFO:tasks.rados.rados.0.smithi038.stdout:618: done (6 left) 2024-03-19T11:30:06.458 INFO:tasks.rados.rados.0.smithi038.stdout:619: done (5 left) 2024-03-19T11:30:06.458 INFO:tasks.rados.rados.0.smithi038.stdout:620: done (4 left) 2024-03-19T11:30:06.458 INFO:tasks.rados.rados.0.smithi038.stdout:621: done (3 left) 2024-03-19T11:30:06.458 INFO:tasks.rados.rados.0.smithi038.stdout:622: done (2 left) 2024-03-19T11:30:06.458 INFO:tasks.rados.rados.0.smithi038.stdout:623: done (1 left) 2024-03-19T11:30:06.458 INFO:tasks.rados.rados.0.smithi038.stdout:624: done (0 left) 2024-03-19T11:30:06.458 INFO:tasks.rados.rados.0.smithi038.stdout:625: read oid 37 snap -1 2024-03-19T11:30:06.458 INFO:tasks.rados.rados.0.smithi038.stdout:625: expect (ObjNum 36 snap 0 seq_num 36) 2024-03-19T11:30:06.459 INFO:tasks.rados.rados.0.smithi038.stdout:626: delete oid 18 current snap is 1 2024-03-19T11:30:06.462 INFO:tasks.rados.rados.0.smithi038.stdout:626: done (1 left) 2024-03-19T11:30:06.462 INFO:tasks.rados.rados.0.smithi038.stdout:627: read oid 29 snap -1 2024-03-19T11:30:06.462 INFO:tasks.rados.rados.0.smithi038.stdout:627: expect (ObjNum 28 snap 0 seq_num 28) 2024-03-19T11:30:06.462 INFO:tasks.rados.rados.0.smithi038.stdout:628: read oid 26 snap -1 2024-03-19T11:30:06.462 INFO:tasks.rados.rados.0.smithi038.stdout:628: expect (ObjNum 25 snap 0 seq_num 25) 2024-03-19T11:30:06.462 INFO:tasks.rados.rados.0.smithi038.stdout:629: read oid 42 snap -1 2024-03-19T11:30:06.462 INFO:tasks.rados.rados.0.smithi038.stdout:629: expect (ObjNum 52 snap 0 seq_num 52) 2024-03-19T11:30:06.462 INFO:tasks.rados.rados.0.smithi038.stdout:630: read oid 13 snap 0 2024-03-19T11:30:06.462 INFO:tasks.rados.rados.0.smithi038.stdout:630: expect (ObjNum 12 snap 0 seq_num 12) 2024-03-19T11:30:06.463 INFO:tasks.rados.rados.0.smithi038.stdout:631: tier_promote oid 4 2024-03-19T11:30:06.463 INFO:tasks.rados.rados.0.smithi038.stdout:632: snap_create 2024-03-19T11:30:06.466 INFO:tasks.rados.rados.0.smithi038.stdout:625: expect (ObjNum 36 snap 0 seq_num 36) 2024-03-19T11:30:06.502 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:06.511+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:06.528 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:06.528 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:06.542 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-19T11:30:06.549 INFO:tasks.rados.rados.0.smithi038.stdout:627: expect (ObjNum 28 snap 0 seq_num 28) 2024-03-19T11:30:06.603 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:06.612+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:06.627 INFO:tasks.rados.rados.0.smithi038.stdout:629: expect (ObjNum 52 snap 0 seq_num 52) 2024-03-19T11:30:06.627 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-19T11:30:06.627 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-19T11:30:06.628 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-03-19T11:30:06.628 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-19T11:30:06.628 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-19T11:30:06.628 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3 2024-03-19T11:30:06.670 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:06.679+0000 7fb17dd72640 -1 osd.4 29 handle_osd_map injecting map crc failure 2024-03-19T11:30:06.707 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:06.707 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:06.707 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:06.715+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:06.718 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-19T11:30:06.751 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:06.751 INFO:teuthology.orchestra.run.smithi038.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-19T11:30:06.751 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:06.759 INFO:tasks.rados.rados.0.smithi038.stdout:628: expect (ObjNum 25 snap 0 seq_num 25) 2024-03-19T11:30:06.808 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:06.819+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:06.834 INFO:tasks.rados.rados.0.smithi038.stdout:630: expect (ObjNum 12 snap 0 seq_num 12) 2024-03-19T11:30:06.883 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:06.883 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:06.897 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-19T11:30:06.901 INFO:tasks.rados.rados.0.smithi038.stdout:631: got (0) Success 2024-03-19T11:30:06.901 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 4 v 3 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-03-19T11:30:06.901 INFO:tasks.rados.rados.0.smithi038.stdout:625: done (6 left) 2024-03-19T11:30:06.901 INFO:tasks.rados.rados.0.smithi038.stdout:627: done (5 left) 2024-03-19T11:30:06.901 INFO:tasks.rados.rados.0.smithi038.stdout:628: done (4 left) 2024-03-19T11:30:06.902 INFO:tasks.rados.rados.0.smithi038.stdout:629: done (3 left) 2024-03-19T11:30:06.902 INFO:tasks.rados.rados.0.smithi038.stdout:630: done (2 left) 2024-03-19T11:30:06.902 INFO:tasks.rados.rados.0.smithi038.stdout:631: done (1 left) 2024-03-19T11:30:06.902 INFO:tasks.rados.rados.0.smithi038.stdout:632: done (0 left) 2024-03-19T11:30:06.902 INFO:tasks.rados.rados.0.smithi038.stdout:633: snap_create 2024-03-19T11:30:06.908 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:06.916+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:06.966 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-19T11:30:07.010 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:07.019+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:07.065 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:07.065 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:07.080 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-19T11:30:07.110 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:07.120+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:07.117 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:30:07.117 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:30:07.133 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:30:07.133 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:30:07.151 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:30:07.151 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:30:07.168 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:30:07.169 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:30:07.187 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:30:07.187 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:30:07.207 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:30:07.207 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:30:07.210 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:07.220+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:07.229 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:30:07.230 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:30:07.250 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:30:07.251 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:30:07.251 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:07.251 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:07.251 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:30:07.251 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:07.265 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-19T11:30:07.312 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:07.320+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:07.388 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:07.388 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:07.388 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:30:07.388 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:30:07.388 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:07.402 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-19T11:30:07.448 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:07.459+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:07.514 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:07.523+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:07.522 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:07.523 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:30:07.523 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:30:07.523 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:30:07.523 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.523 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:30:07.523 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:44.200920+0000", 2024-03-19T11:30:07.523 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.333614118, 2024-03-19T11:30:07.523 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.80881896499999995, 2024-03-19T11:30:07.524 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:07.524 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:07.524 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:07.524 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:07.524 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:07.524 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1 2024-03-19T11:30:07.524 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.524 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:07.525 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.525 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:07.525 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200920+0000", 2024-03-19T11:30:07.525 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:07.525 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.525 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.525 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:07.525 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200920+0000", 2024-03-19T11:30:07.526 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:07.526 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.526 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.526 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:07.526 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200923+0000", 2024-03-19T11:30:07.526 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6469999999999998e-06 2024-03-19T11:30:07.526 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.526 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.526 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:07.527 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200937+0000", 2024-03-19T11:30:07.527 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.399e-05 2024-03-19T11:30:07.527 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.527 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.527 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:07.527 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200960+0000", 2024-03-19T11:30:07.527 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2603e-05 2024-03-19T11:30:07.528 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.528 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.528 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:07.528 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200978+0000", 2024-03-19T11:30:07.528 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.791e-05 2024-03-19T11:30:07.528 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.528 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.528 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:07.528 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.201081+0000", 2024-03-19T11:30:07.529 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010344900000000001 2024-03-19T11:30:07.529 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.529 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.529 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:30:07.529 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.201096+0000", 2024-03-19T11:30:07.529 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4443e-05 2024-03-19T11:30:07.529 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.529 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.529 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:07.530 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.005978+0000", 2024-03-19T11:30:07.530 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.80488281500000003 2024-03-19T11:30:07.530 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.530 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.530 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:07.530 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.006267+0000", 2024-03-19T11:30:07.530 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000288688 2024-03-19T11:30:07.531 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.531 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.531 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:07.531 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.006534+0000", 2024-03-19T11:30:07.531 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026725799999999998 2024-03-19T11:30:07.531 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.532 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.532 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:07.532 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.008457+0000", 2024-03-19T11:30:07.532 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0019227820000000001 2024-03-19T11:30:07.532 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.532 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.533 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:07.533 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.009570+0000", 2024-03-19T11:30:07.533 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011131750000000001 2024-03-19T11:30:07.533 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.533 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.533 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:07.533 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.009670+0000", 2024-03-19T11:30:07.533 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9458999999999998e-05 2024-03-19T11:30:07.534 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.534 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.534 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:07.534 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.009739+0000", 2024-03-19T11:30:07.534 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9746000000000006e-05 2024-03-19T11:30:07.534 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:07.534 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:07.534 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:07.534 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.534 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.535 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:30:07.535 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.071728+0000", 2024-03-19T11:30:07.535 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.462806681, 2024-03-19T11:30:07.535 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013714690999999999, 2024-03-19T11:30:07.535 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:07.535 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:07.535 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:07.535 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:07.535 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:07.535 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 34 2024-03-19T11:30:07.536 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.536 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:07.536 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.536 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:07.536 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071728+0000", 2024-03-19T11:30:07.536 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:07.536 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.536 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.536 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:07.536 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071728+0000", 2024-03-19T11:30:07.536 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:07.536 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.536 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.537 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:07.537 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071729+0000", 2024-03-19T11:30:07.537 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.47e-07 2024-03-19T11:30:07.537 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.537 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.537 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:07.537 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071735+0000", 2024-03-19T11:30:07.537 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3799999999999999e-06 2024-03-19T11:30:07.537 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.537 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.537 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:07.537 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071748+0000", 2024-03-19T11:30:07.537 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2806000000000001e-05 2024-03-19T11:30:07.538 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.538 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.538 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:07.538 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071756+0000", 2024-03-19T11:30:07.538 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1240000000000005e-06 2024-03-19T11:30:07.538 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.538 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.538 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:07.538 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.077570+0000", 2024-03-19T11:30:07.538 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.005813342 2024-03-19T11:30:07.538 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.538 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.538 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:07.539 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.077793+0000", 2024-03-19T11:30:07.539 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022391500000000001 2024-03-19T11:30:07.539 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.539 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.539 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:07.539 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.078146+0000", 2024-03-19T11:30:07.539 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00035219 2024-03-19T11:30:07.539 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.539 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.539 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:07.539 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.083390+0000", 2024-03-19T11:30:07.539 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052445590000000002 2024-03-19T11:30:07.539 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.539 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.540 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:07.540 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085288+0000", 2024-03-19T11:30:07.540 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0018977060000000001 2024-03-19T11:30:07.540 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.540 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.540 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:07.540 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085377+0000", 2024-03-19T11:30:07.540 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9068999999999997e-05 2024-03-19T11:30:07.540 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.540 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.540 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:07.540 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085443+0000", 2024-03-19T11:30:07.540 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5653000000000003e-05 2024-03-19T11:30:07.540 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:07.541 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:07.541 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:07.541 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.541 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.541 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:30:07.541 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.071777+0000", 2024-03-19T11:30:07.541 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.462757571000001, 2024-03-19T11:30:07.541 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013962445, 2024-03-19T11:30:07.541 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:07.541 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:07.541 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:07.541 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:07.541 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:07.541 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 35 2024-03-19T11:30:07.542 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.542 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:07.542 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.542 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:07.542 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071777+0000", 2024-03-19T11:30:07.542 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:07.542 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.542 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.542 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:07.542 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071777+0000", 2024-03-19T11:30:07.542 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:07.542 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.542 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.542 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:07.543 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071778+0000", 2024-03-19T11:30:07.543 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5000000000000001e-07 2024-03-19T11:30:07.543 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.543 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.543 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:07.543 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071917+0000", 2024-03-19T11:30:07.543 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013880199999999999 2024-03-19T11:30:07.543 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.543 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.543 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:07.543 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071931+0000", 2024-03-19T11:30:07.543 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4080000000000001e-05 2024-03-19T11:30:07.543 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.543 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.543 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:07.544 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071940+0000", 2024-03-19T11:30:07.544 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.363e-06 2024-03-19T11:30:07.544 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.544 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.544 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:07.544 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.080462+0000", 2024-03-19T11:30:07.544 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085221810000000002 2024-03-19T11:30:07.544 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.544 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.544 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:07.544 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.080582+0000", 2024-03-19T11:30:07.544 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011935299999999999 2024-03-19T11:30:07.544 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.544 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.545 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:07.545 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.080782+0000", 2024-03-19T11:30:07.545 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020077100000000001 2024-03-19T11:30:07.545 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.545 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.545 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:07.545 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.083401+0000", 2024-03-19T11:30:07.545 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002618333 2024-03-19T11:30:07.545 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.545 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.545 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:07.545 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085571+0000", 2024-03-19T11:30:07.545 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021702829999999998 2024-03-19T11:30:07.545 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.546 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.546 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:07.546 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085672+0000", 2024-03-19T11:30:07.546 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100876 2024-03-19T11:30:07.546 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.546 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.546 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:07.546 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085739+0000", 2024-03-19T11:30:07.546 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7552999999999995e-05 2024-03-19T11:30:07.546 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:07.546 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:07.546 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:07.546 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.546 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.547 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:30:07.547 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.071957+0000", 2024-03-19T11:30:07.547 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.462577382999999, 2024-03-19T11:30:07.547 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013989822000000001, 2024-03-19T11:30:07.547 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:07.547 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:07.547 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:07.547 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:07.547 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:07.547 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 36 2024-03-19T11:30:07.547 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.547 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:07.547 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.547 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:07.547 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071957+0000", 2024-03-19T11:30:07.548 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:07.548 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.548 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.548 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:07.548 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071957+0000", 2024-03-19T11:30:07.548 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:07.548 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.548 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.548 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:07.548 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071958+0000", 2024-03-19T11:30:07.548 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.06e-06 2024-03-19T11:30:07.548 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.548 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.548 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:07.549 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071965+0000", 2024-03-19T11:30:07.549 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8929999999999999e-06 2024-03-19T11:30:07.549 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.549 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.549 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:07.549 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071976+0000", 2024-03-19T11:30:07.549 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1046999999999999e-05 2024-03-19T11:30:07.549 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.549 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.549 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:07.549 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071984+0000", 2024-03-19T11:30:07.549 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7170000000000008e-06 2024-03-19T11:30:07.549 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.549 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.549 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:07.550 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.081761+0000", 2024-03-19T11:30:07.550 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0097769310000000009 2024-03-19T11:30:07.550 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.550 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.550 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:07.550 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.081928+0000", 2024-03-19T11:30:07.550 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000167219 2024-03-19T11:30:07.550 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.550 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.550 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:07.550 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.082131+0000", 2024-03-19T11:30:07.550 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020244499999999999 2024-03-19T11:30:07.550 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.550 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.551 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:07.551 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.083455+0000", 2024-03-19T11:30:07.551 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001324073 2024-03-19T11:30:07.551 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.551 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.551 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:07.551 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085816+0000", 2024-03-19T11:30:07.551 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0023613309999999999 2024-03-19T11:30:07.551 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.551 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.551 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:07.551 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085892+0000", 2024-03-19T11:30:07.551 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6154000000000003e-05 2024-03-19T11:30:07.551 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.551 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.552 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:07.552 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085947+0000", 2024-03-19T11:30:07.552 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4951999999999998e-05 2024-03-19T11:30:07.552 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:07.552 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:07.552 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:07.552 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.552 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.552 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:30:07.552 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.072000+0000", 2024-03-19T11:30:07.552 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.462534633000001, 2024-03-19T11:30:07.552 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014418268999999999, 2024-03-19T11:30:07.552 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:07.552 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:07.552 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:07.552 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:07.553 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:07.553 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 37 2024-03-19T11:30:07.553 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.553 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:07.553 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.553 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:07.553 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072000+0000", 2024-03-19T11:30:07.553 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:07.553 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.553 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.553 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:07.553 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072000+0000", 2024-03-19T11:30:07.553 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:07.553 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.553 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.554 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:07.554 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072001+0000", 2024-03-19T11:30:07.554 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0699999999999996e-07 2024-03-19T11:30:07.554 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.554 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.554 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:07.554 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072007+0000", 2024-03-19T11:30:07.554 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2060000000000004e-06 2024-03-19T11:30:07.554 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.554 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.554 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:07.554 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072023+0000", 2024-03-19T11:30:07.554 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6617e-05 2024-03-19T11:30:07.554 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.555 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.555 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:07.555 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072031+0000", 2024-03-19T11:30:07.555 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8900000000000007e-06 2024-03-19T11:30:07.555 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.555 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.555 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:07.555 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.082682+0000", 2024-03-19T11:30:07.555 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010650868000000001 2024-03-19T11:30:07.555 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.555 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.555 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:07.555 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.082804+0000", 2024-03-19T11:30:07.555 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012202899999999999 2024-03-19T11:30:07.555 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.556 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.556 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:07.556 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.083006+0000", 2024-03-19T11:30:07.556 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020127799999999999 2024-03-19T11:30:07.556 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.556 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.556 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:07.556 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.084145+0000", 2024-03-19T11:30:07.556 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001139791 2024-03-19T11:30:07.556 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.556 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.556 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:07.556 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.086292+0000", 2024-03-19T11:30:07.556 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021461969999999999 2024-03-19T11:30:07.556 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.557 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.557 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:07.557 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.086364+0000", 2024-03-19T11:30:07.557 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2439000000000002e-05 2024-03-19T11:30:07.557 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.557 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.557 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:07.557 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.086418+0000", 2024-03-19T11:30:07.557 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4246999999999997e-05 2024-03-19T11:30:07.557 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:07.557 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:07.557 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:07.557 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.557 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.557 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:30:07.558 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.339200+0000", 2024-03-19T11:30:07.558 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.195334919, 2024-03-19T11:30:07.558 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011947332, 2024-03-19T11:30:07.558 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:07.558 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:07.558 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:07.558 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:07.558 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:07.559 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 109 2024-03-19T11:30:07.560 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.560 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:07.560 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.560 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:07.560 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339200+0000", 2024-03-19T11:30:07.560 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:07.560 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.560 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.560 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:07.560 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339200+0000", 2024-03-19T11:30:07.560 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:07.560 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.560 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.560 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:07.561 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339201+0000", 2024-03-19T11:30:07.561 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.697e-06 2024-03-19T11:30:07.561 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.561 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.561 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:07.561 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339328+0000", 2024-03-19T11:30:07.561 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012660300000000001 2024-03-19T11:30:07.561 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.561 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.561 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:07.561 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339362+0000", 2024-03-19T11:30:07.561 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4351999999999999e-05 2024-03-19T11:30:07.561 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.561 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.562 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:07.562 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339382+0000", 2024-03-19T11:30:07.562 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9983000000000002e-05 2024-03-19T11:30:07.562 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.562 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.562 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:07.562 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.345946+0000", 2024-03-19T11:30:07.562 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0065633330000000002 2024-03-19T11:30:07.562 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.562 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.562 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:07.562 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.346073+0000", 2024-03-19T11:30:07.562 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000127799 2024-03-19T11:30:07.562 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.563 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.563 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:07.563 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.346269+0000", 2024-03-19T11:30:07.563 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000195875 2024-03-19T11:30:07.563 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.563 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.563 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:07.563 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.350756+0000", 2024-03-19T11:30:07.563 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044864190000000002 2024-03-19T11:30:07.563 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.563 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.563 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:07.563 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.350878+0000", 2024-03-19T11:30:07.564 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012271600000000001 2024-03-19T11:30:07.564 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.564 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.564 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:07.564 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.351030+0000", 2024-03-19T11:30:07.564 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015188199999999999 2024-03-19T11:30:07.564 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.564 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.564 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:07.564 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.351147+0000", 2024-03-19T11:30:07.564 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000116673 2024-03-19T11:30:07.564 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:07.564 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:07.564 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:07.564 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.565 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.565 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:30:07.565 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.339422+0000", 2024-03-19T11:30:07.565 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.195112266999999, 2024-03-19T11:30:07.565 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012041094, 2024-03-19T11:30:07.565 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:07.565 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:07.565 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:07.565 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:07.565 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:07.565 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 110 2024-03-19T11:30:07.565 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.565 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:07.565 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.565 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:07.566 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339422+0000", 2024-03-19T11:30:07.566 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:07.566 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.566 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.566 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:07.566 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339422+0000", 2024-03-19T11:30:07.566 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:07.566 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.566 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.566 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:07.566 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339424+0000", 2024-03-19T11:30:07.566 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.697e-06 2024-03-19T11:30:07.566 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.566 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.566 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:07.567 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339440+0000", 2024-03-19T11:30:07.567 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5642999999999999e-05 2024-03-19T11:30:07.567 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.567 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.567 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:07.567 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339462+0000", 2024-03-19T11:30:07.567 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2167e-05 2024-03-19T11:30:07.567 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.567 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.567 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:07.567 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339477+0000", 2024-03-19T11:30:07.567 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5189e-05 2024-03-19T11:30:07.567 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.567 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.568 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:07.568 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.347327+0000", 2024-03-19T11:30:07.568 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0078502559999999999 2024-03-19T11:30:07.568 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.568 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.568 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:07.568 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.347450+0000", 2024-03-19T11:30:07.568 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012261600000000001 2024-03-19T11:30:07.568 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.568 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.568 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:07.568 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.347644+0000", 2024-03-19T11:30:07.568 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019408500000000001 2024-03-19T11:30:07.568 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.569 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.569 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:07.569 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.351304+0000", 2024-03-19T11:30:07.569 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003660118 2024-03-19T11:30:07.569 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.569 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.569 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:07.569 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.351328+0000", 2024-03-19T11:30:07.569 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3734e-05 2024-03-19T11:30:07.569 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.569 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.569 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:07.569 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.351402+0000", 2024-03-19T11:30:07.569 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3708999999999995e-05 2024-03-19T11:30:07.570 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.570 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.570 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:07.570 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.351463+0000", 2024-03-19T11:30:07.570 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1879999999999997e-05 2024-03-19T11:30:07.570 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:07.570 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:07.570 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:07.570 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.570 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.570 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:30:07.570 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.339510+0000", 2024-03-19T11:30:07.570 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.195024785000001, 2024-03-19T11:30:07.571 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012358388999999999, 2024-03-19T11:30:07.571 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:07.571 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:07.571 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:07.571 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:07.571 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:07.571 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 111 2024-03-19T11:30:07.571 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.571 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:07.571 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.571 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:07.571 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339510+0000", 2024-03-19T11:30:07.571 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:07.572 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.572 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.572 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:07.572 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339510+0000", 2024-03-19T11:30:07.572 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:07.572 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.572 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.572 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:07.572 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339511+0000", 2024-03-19T11:30:07.572 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4399999999999998e-07 2024-03-19T11:30:07.572 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.572 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.572 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:07.573 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339518+0000", 2024-03-19T11:30:07.573 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9999999999999999e-06 2024-03-19T11:30:07.573 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.573 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.573 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:07.573 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339535+0000", 2024-03-19T11:30:07.573 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7552999999999999e-05 2024-03-19T11:30:07.573 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.573 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.573 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:07.573 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339550+0000", 2024-03-19T11:30:07.573 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5003e-05 2024-03-19T11:30:07.573 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.573 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.574 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:07.574 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.348403+0000", 2024-03-19T11:30:07.574 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0088530810000000005 2024-03-19T11:30:07.574 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.574 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.574 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:07.574 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.348629+0000", 2024-03-19T11:30:07.574 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022534599999999999 2024-03-19T11:30:07.574 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.574 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.574 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:07.574 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.348976+0000", 2024-03-19T11:30:07.574 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00034710699999999998 2024-03-19T11:30:07.574 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.575 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.575 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:07.575 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.351479+0000", 2024-03-19T11:30:07.575 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0025035309999999998 2024-03-19T11:30:07.575 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.575 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.575 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:07.575 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.351602+0000", 2024-03-19T11:30:07.575 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000122912 2024-03-19T11:30:07.575 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.575 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.575 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:07.575 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.351766+0000", 2024-03-19T11:30:07.575 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016410599999999999 2024-03-19T11:30:07.575 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.576 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.576 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:07.576 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.351868+0000", 2024-03-19T11:30:07.576 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010180599999999999 2024-03-19T11:30:07.576 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:07.576 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:07.576 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:07.576 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.576 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.576 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:30:07.576 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.339576+0000", 2024-03-19T11:30:07.576 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.194958361000001, 2024-03-19T11:30:07.576 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012548515999999999, 2024-03-19T11:30:07.576 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:07.577 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:07.577 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:07.577 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:07.577 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:07.577 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 112 2024-03-19T11:30:07.577 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.577 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:07.577 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.577 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:07.577 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339576+0000", 2024-03-19T11:30:07.577 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:07.577 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.577 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.577 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:07.577 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339576+0000", 2024-03-19T11:30:07.578 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:07.578 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.578 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.578 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:07.578 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339577+0000", 2024-03-19T11:30:07.578 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2529999999999999e-06 2024-03-19T11:30:07.578 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.578 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.578 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:07.578 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339813+0000", 2024-03-19T11:30:07.578 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000235811 2024-03-19T11:30:07.578 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.578 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.578 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:07.579 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339848+0000", 2024-03-19T11:30:07.579 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4437000000000002e-05 2024-03-19T11:30:07.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.579 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:07.579 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339867+0000", 2024-03-19T11:30:07.579 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9046e-05 2024-03-19T11:30:07.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.579 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:07.579 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.349815+0000", 2024-03-19T11:30:07.579 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0099484029999999998 2024-03-19T11:30:07.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.580 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:07.580 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.349955+0000", 2024-03-19T11:30:07.580 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000140106 2024-03-19T11:30:07.580 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.580 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.580 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:07.580 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.350163+0000", 2024-03-19T11:30:07.580 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020803799999999999 2024-03-19T11:30:07.580 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.580 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.580 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:07.580 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.351984+0000", 2024-03-19T11:30:07.580 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0018202159999999999 2024-03-19T11:30:07.580 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.581 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:07.581 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352002+0000", 2024-03-19T11:30:07.581 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8430000000000001e-05 2024-03-19T11:30:07.581 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.581 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:07.581 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352070+0000", 2024-03-19T11:30:07.581 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7623e-05 2024-03-19T11:30:07.581 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.581 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:07.581 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352125+0000", 2024-03-19T11:30:07.581 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5152999999999999e-05 2024-03-19T11:30:07.582 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:07.582 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:07.582 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:07.582 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.582 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:30:07.582 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.339898+0000", 2024-03-19T11:30:07.582 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.194637011000001, 2024-03-19T11:30:07.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019501793, 2024-03-19T11:30:07.582 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:07.582 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:07.583 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:07.583 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:07.583 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:07.583 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113 2024-03-19T11:30:07.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.583 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:07.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.583 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:07.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339898+0000", 2024-03-19T11:30:07.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:07.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.583 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:07.584 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339898+0000", 2024-03-19T11:30:07.584 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:07.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.584 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:07.584 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339899+0000", 2024-03-19T11:30:07.584 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.353e-06 2024-03-19T11:30:07.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.584 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:07.584 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339907+0000", 2024-03-19T11:30:07.584 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3000000000000002e-06 2024-03-19T11:30:07.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.584 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:07.585 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339924+0000", 2024-03-19T11:30:07.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6906999999999999e-05 2024-03-19T11:30:07.585 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.585 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:07.585 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339933+0000", 2024-03-19T11:30:07.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8969999999999997e-06 2024-03-19T11:30:07.585 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.585 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:07.585 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352239+0000", 2024-03-19T11:30:07.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012305814999999999 2024-03-19T11:30:07.585 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.586 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:07.586 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352466+0000", 2024-03-19T11:30:07.586 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022684100000000001 2024-03-19T11:30:07.586 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.586 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:07.586 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352816+0000", 2024-03-19T11:30:07.586 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00035030499999999997 2024-03-19T11:30:07.586 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.586 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:07.586 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.358956+0000", 2024-03-19T11:30:07.586 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061400079999999998 2024-03-19T11:30:07.586 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.587 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:07.587 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359118+0000", 2024-03-19T11:30:07.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016237599999999999 2024-03-19T11:30:07.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.587 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:07.587 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359287+0000", 2024-03-19T11:30:07.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000168488 2024-03-19T11:30:07.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.587 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:07.587 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359399+0000", 2024-03-19T11:30:07.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000112503 2024-03-19T11:30:07.587 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:07.588 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:07.588 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:07.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.588 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.588 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:30:07.588 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.339947+0000", 2024-03-19T11:30:07.588 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.194587244000001, 2024-03-19T11:30:07.588 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021923069999999999, 2024-03-19T11:30:07.588 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:07.588 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:07.588 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:07.588 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:07.588 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:07.588 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 114 2024-03-19T11:30:07.589 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.589 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:07.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.589 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:07.589 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339947+0000", 2024-03-19T11:30:07.589 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:07.589 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.589 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:07.589 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339947+0000", 2024-03-19T11:30:07.589 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:07.589 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.589 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:07.590 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339948+0000", 2024-03-19T11:30:07.590 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6000000000000003e-07 2024-03-19T11:30:07.590 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.590 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.590 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:07.590 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339956+0000", 2024-03-19T11:30:07.590 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1559999999999995e-06 2024-03-19T11:30:07.590 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.590 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.590 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:07.590 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339967+0000", 2024-03-19T11:30:07.590 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0669999999999999e-05 2024-03-19T11:30:07.590 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.591 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:07.591 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339975+0000", 2024-03-19T11:30:07.591 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0099999999999995e-06 2024-03-19T11:30:07.591 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.591 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:07.591 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.353672+0000", 2024-03-19T11:30:07.591 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013696583999999999 2024-03-19T11:30:07.591 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.591 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:07.591 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.353808+0000", 2024-03-19T11:30:07.591 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013652299999999999 2024-03-19T11:30:07.591 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.592 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.592 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:07.592 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.354036+0000", 2024-03-19T11:30:07.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022744799999999999 2024-03-19T11:30:07.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.592 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.592 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:07.592 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.360480+0000", 2024-03-19T11:30:07.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0064441630000000001 2024-03-19T11:30:07.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.592 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.592 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:07.592 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361722+0000", 2024-03-19T11:30:07.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001242754 2024-03-19T11:30:07.593 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.593 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.593 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:07.593 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361807+0000", 2024-03-19T11:30:07.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4605999999999996e-05 2024-03-19T11:30:07.593 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.593 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.593 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:07.593 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361870+0000", 2024-03-19T11:30:07.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3396000000000001e-05 2024-03-19T11:30:07.593 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:07.593 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:07.593 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:07.593 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.593 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.594 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:30:07.594 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.340017+0000", 2024-03-19T11:30:07.595 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.194517928, 2024-03-19T11:30:07.595 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022169355000000002, 2024-03-19T11:30:07.595 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:07.595 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:07.595 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:07.595 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:07.595 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:07.595 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 115 2024-03-19T11:30:07.595 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.595 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:07.595 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:07.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340017+0000", 2024-03-19T11:30:07.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:07.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:07.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340017+0000", 2024-03-19T11:30:07.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:07.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:07.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340018+0000", 2024-03-19T11:30:07.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.043e-06 2024-03-19T11:30:07.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.597 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:07.597 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340139+0000", 2024-03-19T11:30:07.597 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121502 2024-03-19T11:30:07.597 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.597 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:07.597 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340157+0000", 2024-03-19T11:30:07.597 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7382999999999999e-05 2024-03-19T11:30:07.597 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.597 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:07.597 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340166+0000", 2024-03-19T11:30:07.597 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2469999999999997e-06 2024-03-19T11:30:07.598 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.598 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.598 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:07.598 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.357470+0000", 2024-03-19T11:30:07.598 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017304119999999999 2024-03-19T11:30:07.598 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.598 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.598 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:07.598 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.357671+0000", 2024-03-19T11:30:07.598 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000200798 2024-03-19T11:30:07.598 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.598 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.598 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:07.598 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.358002+0000", 2024-03-19T11:30:07.599 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033090099999999998 2024-03-19T11:30:07.599 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.599 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.599 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:07.599 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361892+0000", 2024-03-19T11:30:07.599 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00389018 2024-03-19T11:30:07.599 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.599 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.599 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:07.599 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361983+0000", 2024-03-19T11:30:07.599 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0976000000000004e-05 2024-03-19T11:30:07.599 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.599 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.599 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:07.599 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362071+0000", 2024-03-19T11:30:07.600 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8022999999999994e-05 2024-03-19T11:30:07.600 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.600 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.600 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:07.600 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362186+0000", 2024-03-19T11:30:07.600 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115182 2024-03-19T11:30:07.600 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:07.600 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:07.600 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:07.600 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.600 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.600 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:30:07.600 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.340183+0000", 2024-03-19T11:30:07.600 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.194351095999998, 2024-03-19T11:30:07.601 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022279841000000002, 2024-03-19T11:30:07.601 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:07.601 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:07.601 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:07.601 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:07.601 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:07.601 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 116 2024-03-19T11:30:07.601 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.601 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:07.601 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.601 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:07.601 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340183+0000", 2024-03-19T11:30:07.601 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:07.601 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.601 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.602 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:07.602 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340183+0000", 2024-03-19T11:30:07.602 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:07.602 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.602 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.602 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:07.602 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340184+0000", 2024-03-19T11:30:07.602 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9000000000000005e-07 2024-03-19T11:30:07.602 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.602 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.602 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:07.602 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340191+0000", 2024-03-19T11:30:07.602 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0360000000000003e-06 2024-03-19T11:30:07.602 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.602 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.603 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:07.603 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340201+0000", 2024-03-19T11:30:07.603 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0844e-05 2024-03-19T11:30:07.603 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.603 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.603 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:07.603 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340209+0000", 2024-03-19T11:30:07.603 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7360000000000001e-06 2024-03-19T11:30:07.603 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.603 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.603 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:07.603 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359535+0000", 2024-03-19T11:30:07.603 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019325546999999998 2024-03-19T11:30:07.603 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.603 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.604 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:07.604 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359672+0000", 2024-03-19T11:30:07.604 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013706600000000001 2024-03-19T11:30:07.604 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.604 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.604 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:07.604 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359905+0000", 2024-03-19T11:30:07.604 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023334500000000001 2024-03-19T11:30:07.604 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.604 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.604 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:07.604 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362306+0000", 2024-03-19T11:30:07.604 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002401041 2024-03-19T11:30:07.604 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.604 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.605 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:07.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362329+0000", 2024-03-19T11:30:07.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3283999999999999e-05 2024-03-19T11:30:07.605 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.605 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:07.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362402+0000", 2024-03-19T11:30:07.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2459000000000005e-05 2024-03-19T11:30:07.605 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.605 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:07.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362463+0000", 2024-03-19T11:30:07.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1493000000000002e-05 2024-03-19T11:30:07.605 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:07.605 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:07.606 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:07.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.606 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:30:07.606 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.340222+0000", 2024-03-19T11:30:07.606 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.194312513, 2024-03-19T11:30:07.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022500218999999998, 2024-03-19T11:30:07.606 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:07.606 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:07.606 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:07.606 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:07.606 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:07.606 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 117 2024-03-19T11:30:07.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.607 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:07.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:07.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340222+0000", 2024-03-19T11:30:07.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:07.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:07.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340222+0000", 2024-03-19T11:30:07.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:07.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:07.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340223+0000", 2024-03-19T11:30:07.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7999999999999995e-07 2024-03-19T11:30:07.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.608 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:07.608 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340229+0000", 2024-03-19T11:30:07.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0399999999999998e-06 2024-03-19T11:30:07.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.608 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:07.608 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340239+0000", 2024-03-19T11:30:07.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0190000000000001e-05 2024-03-19T11:30:07.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.608 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:07.608 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340250+0000", 2024-03-19T11:30:07.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0720000000000001e-05 2024-03-19T11:30:07.609 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.609 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:07.609 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.360587+0000", 2024-03-19T11:30:07.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020337542 2024-03-19T11:30:07.609 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.609 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:07.609 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.360818+0000", 2024-03-19T11:30:07.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023120500000000001 2024-03-19T11:30:07.609 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.610 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:07.610 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361218+0000", 2024-03-19T11:30:07.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000399637 2024-03-19T11:30:07.610 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.610 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:07.610 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362483+0000", 2024-03-19T11:30:07.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012649200000000001 2024-03-19T11:30:07.610 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.610 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:07.610 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362538+0000", 2024-03-19T11:30:07.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4490000000000002e-05 2024-03-19T11:30:07.610 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:07.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362617+0000", 2024-03-19T11:30:07.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9839000000000006e-05 2024-03-19T11:30:07.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:07.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362722+0000", 2024-03-19T11:30:07.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104956 2024-03-19T11:30:07.611 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:07.611 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:07.611 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:07.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.611 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:30:07.612 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.806124+0000", 2024-03-19T11:30:07.612 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 21.728410361999998, 2024-03-19T11:30:07.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010639073000000001, 2024-03-19T11:30:07.612 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:07.612 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:07.612 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:07.612 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:07.612 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:07.612 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 136 2024-03-19T11:30:07.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.612 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:07.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:07.613 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806124+0000", 2024-03-19T11:30:07.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:07.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.613 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:07.613 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806124+0000", 2024-03-19T11:30:07.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:07.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.613 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:07.613 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806125+0000", 2024-03-19T11:30:07.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.13e-07 2024-03-19T11:30:07.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.613 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:07.614 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806130+0000", 2024-03-19T11:30:07.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4199999999999998e-06 2024-03-19T11:30:07.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.614 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:07.614 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806138+0000", 2024-03-19T11:30:07.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7479999999999993e-06 2024-03-19T11:30:07.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.614 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:07.614 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806145+0000", 2024-03-19T11:30:07.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2269999999999996e-06 2024-03-19T11:30:07.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:07.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.808827+0000", 2024-03-19T11:30:07.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026814260000000002 2024-03-19T11:30:07.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:07.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.808880+0000", 2024-03-19T11:30:07.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3427999999999997e-05 2024-03-19T11:30:07.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:07.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.808969+0000", 2024-03-19T11:30:07.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8525999999999999e-05 2024-03-19T11:30:07.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:07.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.811343+0000", 2024-03-19T11:30:07.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002374101 2024-03-19T11:30:07.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:07.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.816680+0000", 2024-03-19T11:30:07.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053375990000000002 2024-03-19T11:30:07.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:07.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.816716+0000", 2024-03-19T11:30:07.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5781999999999997e-05 2024-03-19T11:30:07.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.617 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:07.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.816763+0000", 2024-03-19T11:30:07.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7302999999999998e-05 2024-03-19T11:30:07.617 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:07.617 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:07.617 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:07.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.617 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:30:07.617 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.806171+0000", 2024-03-19T11:30:07.617 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 21.728363139999999, 2024-03-19T11:30:07.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010681563999999999, 2024-03-19T11:30:07.617 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:07.617 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:07.618 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:07.618 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:07.618 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:07.618 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 137 2024-03-19T11:30:07.618 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.618 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:07.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.618 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:07.618 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806171+0000", 2024-03-19T11:30:07.618 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:07.618 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:07.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806171+0000", 2024-03-19T11:30:07.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:07.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:07.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806172+0000", 2024-03-19T11:30:07.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9100000000000003e-07 2024-03-19T11:30:07.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:07.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806243+0000", 2024-03-19T11:30:07.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0563999999999997e-05 2024-03-19T11:30:07.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:07.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806252+0000", 2024-03-19T11:30:07.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7499999999999998e-06 2024-03-19T11:30:07.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:07.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806263+0000", 2024-03-19T11:30:07.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0205e-05 2024-03-19T11:30:07.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:07.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.809226+0000", 2024-03-19T11:30:07.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029634679999999999 2024-03-19T11:30:07.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:07.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.809278+0000", 2024-03-19T11:30:07.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1811000000000003e-05 2024-03-19T11:30:07.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:07.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.809363+0000", 2024-03-19T11:30:07.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4752000000000004e-05 2024-03-19T11:30:07.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:07.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.812567+0000", 2024-03-19T11:30:07.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032038710000000001 2024-03-19T11:30:07.622 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.622 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.622 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:07.622 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.816801+0000", 2024-03-19T11:30:07.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0042343040000000004 2024-03-19T11:30:07.622 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.622 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.622 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:07.622 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.816833+0000", 2024-03-19T11:30:07.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2289000000000002e-05 2024-03-19T11:30:07.622 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.622 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.622 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:07.622 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.816853+0000", 2024-03-19T11:30:07.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9859000000000001e-05 2024-03-19T11:30:07.623 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:07.623 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:07.623 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:07.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.623 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:30:07.623 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.806275+0000", 2024-03-19T11:30:07.623 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 21.728259804, 2024-03-19T11:30:07.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010695962, 2024-03-19T11:30:07.623 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:07.623 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:07.623 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:07.623 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:07.624 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:07.624 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 138 2024-03-19T11:30:07.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.624 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:07.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:07.624 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806275+0000", 2024-03-19T11:30:07.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:07.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:07.624 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806275+0000", 2024-03-19T11:30:07.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:07.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:07.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806275+0000", 2024-03-19T11:30:07.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1099999999999995e-07 2024-03-19T11:30:07.625 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:07.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806280+0000", 2024-03-19T11:30:07.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.42e-06 2024-03-19T11:30:07.625 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:07.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806288+0000", 2024-03-19T11:30:07.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9589999999999995e-06 2024-03-19T11:30:07.626 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.626 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:07.626 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806296+0000", 2024-03-19T11:30:07.626 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6980000000000004e-06 2024-03-19T11:30:07.626 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.626 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:07.626 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.810916+0000", 2024-03-19T11:30:07.626 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0046194579999999999 2024-03-19T11:30:07.626 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.626 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:07.626 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.810975+0000", 2024-03-19T11:30:07.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8767999999999997e-05 2024-03-19T11:30:07.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:07.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.811066+0000", 2024-03-19T11:30:07.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1543000000000001e-05 2024-03-19T11:30:07.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:07.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.813111+0000", 2024-03-19T11:30:07.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0020449650000000002 2024-03-19T11:30:07.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:07.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.816889+0000", 2024-03-19T11:30:07.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037775270000000001 2024-03-19T11:30:07.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:07.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.816925+0000", 2024-03-19T11:30:07.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6539999999999999e-05 2024-03-19T11:30:07.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:07.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.816971+0000", 2024-03-19T11:30:07.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5473000000000003e-05 2024-03-19T11:30:07.628 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:07.628 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:07.628 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:07.629 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.629 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.629 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:30:07.629 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.806306+0000", 2024-03-19T11:30:07.629 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 21.728228213000001, 2024-03-19T11:30:07.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010748485, 2024-03-19T11:30:07.629 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:07.629 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:07.629 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:07.629 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:07.629 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:07.629 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 139 2024-03-19T11:30:07.629 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.629 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:07.629 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.630 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:07.630 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806306+0000", 2024-03-19T11:30:07.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:07.630 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.630 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:07.630 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806306+0000", 2024-03-19T11:30:07.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:07.630 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.630 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:07.630 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806307+0000", 2024-03-19T11:30:07.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7199999999999999e-07 2024-03-19T11:30:07.630 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.631 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:07.631 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806311+0000", 2024-03-19T11:30:07.631 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1989999999999999e-06 2024-03-19T11:30:07.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.631 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:07.631 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806323+0000", 2024-03-19T11:30:07.631 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2075e-05 2024-03-19T11:30:07.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.631 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:07.631 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806331+0000", 2024-03-19T11:30:07.631 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4959999999999999e-06 2024-03-19T11:30:07.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:07.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.811381+0000", 2024-03-19T11:30:07.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.005050232 2024-03-19T11:30:07.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:07.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.811437+0000", 2024-03-19T11:30:07.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6116999999999998e-05 2024-03-19T11:30:07.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:07.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.811525+0000", 2024-03-19T11:30:07.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7757999999999996e-05 2024-03-19T11:30:07.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:07.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.813119+0000", 2024-03-19T11:30:07.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0015946739999999999 2024-03-19T11:30:07.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:07.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.817005+0000", 2024-03-19T11:30:07.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038853080000000001 2024-03-19T11:30:07.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:07.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.817034+0000", 2024-03-19T11:30:07.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9297000000000001e-05 2024-03-19T11:30:07.634 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.634 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.634 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:07.634 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.817055+0000", 2024-03-19T11:30:07.634 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0857e-05 2024-03-19T11:30:07.634 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:07.634 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:07.634 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:07.634 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.634 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.634 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:30:07.634 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.806342+0000", 2024-03-19T11:30:07.634 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 21.728192205999999, 2024-03-19T11:30:07.634 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010826809999999999, 2024-03-19T11:30:07.635 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:07.635 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:07.635 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:07.635 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:07.635 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:07.635 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 140 2024-03-19T11:30:07.635 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.635 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:07.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.635 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:07.635 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806342+0000", 2024-03-19T11:30:07.635 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:07.635 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.635 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:07.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806342+0000", 2024-03-19T11:30:07.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:07.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:07.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806343+0000", 2024-03-19T11:30:07.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6600000000000002e-07 2024-03-19T11:30:07.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:07.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806422+0000", 2024-03-19T11:30:07.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9536000000000006e-05 2024-03-19T11:30:07.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.637 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:07.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806439+0000", 2024-03-19T11:30:07.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7098000000000001e-05 2024-03-19T11:30:07.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.637 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:07.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806447+0000", 2024-03-19T11:30:07.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1300000000000003e-06 2024-03-19T11:30:07.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.637 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:07.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.811917+0000", 2024-03-19T11:30:07.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054699019999999996 2024-03-19T11:30:07.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.638 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.638 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:07.638 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.811973+0000", 2024-03-19T11:30:07.638 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6001000000000001e-05 2024-03-19T11:30:07.638 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.638 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.638 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:07.638 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.812055+0000", 2024-03-19T11:30:07.638 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2655000000000001e-05 2024-03-19T11:30:07.638 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.638 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.638 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:07.638 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.813256+0000", 2024-03-19T11:30:07.638 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012011999999999999 2024-03-19T11:30:07.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.639 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:07.639 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.817091+0000", 2024-03-19T11:30:07.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038344659999999999 2024-03-19T11:30:07.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.639 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:07.639 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.817125+0000", 2024-03-19T11:30:07.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4653000000000002e-05 2024-03-19T11:30:07.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.639 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:07.639 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.817169+0000", 2024-03-19T11:30:07.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.3702999999999998e-05 2024-03-19T11:30:07.640 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:07.640 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:07.640 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:07.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.640 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:30:07.640 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.806461+0000", 2024-03-19T11:30:07.640 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 21.728074062000001, 2024-03-19T11:30:07.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010789451, 2024-03-19T11:30:07.640 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:07.640 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:07.640 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:07.640 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:07.640 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:07.641 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 141 2024-03-19T11:30:07.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.641 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:07.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.641 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:07.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806461+0000", 2024-03-19T11:30:07.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:07.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.641 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:07.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806461+0000", 2024-03-19T11:30:07.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:07.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.641 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:07.642 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806461+0000", 2024-03-19T11:30:07.642 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5499999999999997e-07 2024-03-19T11:30:07.642 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.642 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.642 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:07.642 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806466+0000", 2024-03-19T11:30:07.642 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9049999999999996e-06 2024-03-19T11:30:07.642 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.642 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.642 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:07.642 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806474+0000", 2024-03-19T11:30:07.642 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6009999999999997e-06 2024-03-19T11:30:07.642 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.642 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.643 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:07.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.806480+0000", 2024-03-19T11:30:07.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.207e-06 2024-03-19T11:30:07.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.643 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:07.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.812605+0000", 2024-03-19T11:30:07.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.006125096 2024-03-19T11:30:07.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.643 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:07.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.812662+0000", 2024-03-19T11:30:07.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.7095999999999997e-05 2024-03-19T11:30:07.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.644 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:07.644 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.812751+0000", 2024-03-19T11:30:07.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9115000000000003e-05 2024-03-19T11:30:07.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.644 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:07.644 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.813361+0000", 2024-03-19T11:30:07.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00060919399999999997 2024-03-19T11:30:07.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.644 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:07.644 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.817201+0000", 2024-03-19T11:30:07.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0038408510000000002 2024-03-19T11:30:07.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:07.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.817230+0000", 2024-03-19T11:30:07.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8963999999999999e-05 2024-03-19T11:30:07.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:07.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:07.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:07.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.817250+0000", 2024-03-19T11:30:07.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9667e-05 2024-03-19T11:30:07.645 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:07.645 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:07.645 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:07.645 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:07.645 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:07.645 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:07.647 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:07.655+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:07.647 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-19T11:30:07.702 INFO:tasks.rados.rados.0.smithi038.stdout:633: done (0 left) 2024-03-19T11:30:07.702 INFO:tasks.rados.rados.0.smithi038.stdout:634: read oid 30 snap 0 2024-03-19T11:30:07.702 INFO:tasks.rados.rados.0.smithi038.stdout:634: expect (ObjNum 29 snap 0 seq_num 29) 2024-03-19T11:30:07.702 INFO:tasks.rados.rados.0.smithi038.stdout:635: snap_remove snap 2 2024-03-19T11:30:07.748 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:07.759+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:07.785 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:07.785 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:30:07.785 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:30:07.785 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:07.799 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-19T11:30:07.800 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-03-19T11:30:07.849 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:07.856+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:07.931 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:07.931 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:30:07.931 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:30:07.931 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:30:07.931 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:07.945 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-19T11:30:07.986 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:07.996+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:08.051 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:08.060+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:08.105 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:08.105 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:30:08.105 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:30:08.105 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:30:08.105 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.105 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:34 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'20, mlcod=15'16)", 2024-03-19T11:30:08.106 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.077542+0000", 2024-03-19T11:30:08.106 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 23.037850602999999, 2024-03-19T11:30:08.106 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0064305179999999997, 2024-03-19T11:30:08.106 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:08.106 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:08.106 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:08.106 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.106 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:08.107 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.077542+0000", 2024-03-19T11:30:08.107 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.107 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.107 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.107 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:08.107 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.077542+0000", 2024-03-19T11:30:08.107 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.107 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.107 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.107 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:08.107 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.077543+0000", 2024-03-19T11:30:08.107 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.387e-06 2024-03-19T11:30:08.108 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.108 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.108 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:08.108 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.077555+0000", 2024-03-19T11:30:08.108 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1759999999999999e-05 2024-03-19T11:30:08.108 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.108 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.108 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:08.108 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.077580+0000", 2024-03-19T11:30:08.108 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.529e-05 2024-03-19T11:30:08.108 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.108 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.108 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:08.109 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.077596+0000", 2024-03-19T11:30:08.109 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5466000000000001e-05 2024-03-19T11:30:08.109 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.109 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.109 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:08.109 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.077753+0000", 2024-03-19T11:30:08.109 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000157026 2024-03-19T11:30:08.109 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.109 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.109 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:08.109 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.077789+0000", 2024-03-19T11:30:08.109 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6259e-05 2024-03-19T11:30:08.109 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.109 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.109 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:08.110 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.083910+0000", 2024-03-19T11:30:08.110 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0061212080000000004 2024-03-19T11:30:08.110 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.110 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.110 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:08.110 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.083972+0000", 2024-03-19T11:30:08.110 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2121999999999999e-05 2024-03-19T11:30:08.110 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.110 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:08.110 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.110 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.110 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.110 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:35 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'21, mlcod=15'17)", 2024-03-19T11:30:08.111 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.080212+0000", 2024-03-19T11:30:08.111 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 23.035180478000001, 2024-03-19T11:30:08.111 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003806023, 2024-03-19T11:30:08.111 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:08.111 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:08.111 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:08.111 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.111 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:08.111 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.080212+0000", 2024-03-19T11:30:08.111 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.111 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.111 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.111 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:08.111 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.080212+0000", 2024-03-19T11:30:08.111 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.112 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.112 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.112 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:08.112 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.080213+0000", 2024-03-19T11:30:08.112 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2330000000000001e-06 2024-03-19T11:30:08.112 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.112 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.112 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:08.112 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.080266+0000", 2024-03-19T11:30:08.112 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2460000000000003e-05 2024-03-19T11:30:08.112 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.112 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.112 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:08.112 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.080288+0000", 2024-03-19T11:30:08.113 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1892999999999999e-05 2024-03-19T11:30:08.113 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.113 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.113 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:08.113 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.080323+0000", 2024-03-19T11:30:08.113 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5107000000000001e-05 2024-03-19T11:30:08.113 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.113 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.113 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:08.113 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.082276+0000", 2024-03-19T11:30:08.113 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0019535170000000001 2024-03-19T11:30:08.113 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.113 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.113 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:08.113 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.082315+0000", 2024-03-19T11:30:08.114 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8760000000000002e-05 2024-03-19T11:30:08.114 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.114 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.114 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:08.114 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.083977+0000", 2024-03-19T11:30:08.114 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001662483 2024-03-19T11:30:08.114 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.114 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.114 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:08.114 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.084018+0000", 2024-03-19T11:30:08.114 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.057e-05 2024-03-19T11:30:08.114 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.114 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:08.114 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.115 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.115 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.115 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:114 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'76, mlcod=16'73)", 2024-03-19T11:30:08.115 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.353321+0000", 2024-03-19T11:30:08.115 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 22.76207175, 2024-03-19T11:30:08.115 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050714999999999996, 2024-03-19T11:30:08.115 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:08.115 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:08.115 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:08.115 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.115 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:08.115 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.353321+0000", 2024-03-19T11:30:08.115 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.115 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.115 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.115 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:08.116 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.353321+0000", 2024-03-19T11:30:08.116 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.116 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.116 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.116 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:08.116 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.353321+0000", 2024-03-19T11:30:08.116 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3399999999999998e-07 2024-03-19T11:30:08.116 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.116 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.116 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:08.116 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.353336+0000", 2024-03-19T11:30:08.117 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4154999999999999e-05 2024-03-19T11:30:08.117 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.117 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.117 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:08.117 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.353349+0000", 2024-03-19T11:30:08.117 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3644e-05 2024-03-19T11:30:08.117 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.117 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.117 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:08.117 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.353357+0000", 2024-03-19T11:30:08.117 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5109999999999997e-06 2024-03-19T11:30:08.117 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.117 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.117 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:08.117 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.353409+0000", 2024-03-19T11:30:08.118 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2723999999999999e-05 2024-03-19T11:30:08.118 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.118 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.118 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:08.118 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.353428+0000", 2024-03-19T11:30:08.118 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8216e-05 2024-03-19T11:30:08.118 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.118 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.118 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:08.118 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.358359+0000", 2024-03-19T11:30:08.118 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0049314550000000004 2024-03-19T11:30:08.118 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.118 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.119 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:08.119 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.358392+0000", 2024-03-19T11:30:08.119 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3061000000000001e-05 2024-03-19T11:30:08.119 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.119 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:08.119 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.119 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.119 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.119 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:119 1.0 e16/14 1:30306672:devicehealth::main.db.0000000000000000:head v 16'81, mlcod=16'78)", 2024-03-19T11:30:08.119 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.364626+0000", 2024-03-19T11:30:08.119 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 22.75076679, 2024-03-19T11:30:08.119 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046261449999999999, 2024-03-19T11:30:08.119 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:08.119 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:08.120 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:08.120 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.120 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:08.120 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.364626+0000", 2024-03-19T11:30:08.120 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.120 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.120 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.120 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:08.120 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.364626+0000", 2024-03-19T11:30:08.120 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.120 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.120 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.121 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:08.121 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.364626+0000", 2024-03-19T11:30:08.121 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1399999999999996e-07 2024-03-19T11:30:08.121 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.121 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.121 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:08.121 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.364724+0000", 2024-03-19T11:30:08.121 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7972000000000003e-05 2024-03-19T11:30:08.121 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.121 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.121 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:08.121 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.364737+0000", 2024-03-19T11:30:08.121 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2585000000000001e-05 2024-03-19T11:30:08.121 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.122 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:08.122 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.364744+0000", 2024-03-19T11:30:08.122 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7780000000000002e-06 2024-03-19T11:30:08.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.122 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:08.122 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.364780+0000", 2024-03-19T11:30:08.122 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5992999999999999e-05 2024-03-19T11:30:08.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.122 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:08.122 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.364795+0000", 2024-03-19T11:30:08.123 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4939e-05 2024-03-19T11:30:08.123 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.123 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.123 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:08.123 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.369211+0000", 2024-03-19T11:30:08.123 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0044161139999999996 2024-03-19T11:30:08.123 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.123 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.123 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:08.123 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.369252+0000", 2024-03-19T11:30:08.123 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1050000000000002e-05 2024-03-19T11:30:08.123 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.123 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:08.123 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.124 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.124 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.124 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:120 1.0 e16/14 1:30306672:devicehealth::main.db.0000000000000000:head v 16'82, mlcod=16'78)", 2024-03-19T11:30:08.124 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.365811+0000", 2024-03-19T11:30:08.124 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 22.749581833000001, 2024-03-19T11:30:08.124 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0040843659999999999, 2024-03-19T11:30:08.124 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:08.124 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:08.124 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:08.124 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.124 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:08.124 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.365811+0000", 2024-03-19T11:30:08.124 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.124 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.124 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.125 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:08.125 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.365811+0000", 2024-03-19T11:30:08.125 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.125 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.125 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.125 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:08.125 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.365811+0000", 2024-03-19T11:30:08.125 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4199999999999995e-07 2024-03-19T11:30:08.125 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.125 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.125 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:08.125 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.365884+0000", 2024-03-19T11:30:08.125 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2527999999999994e-05 2024-03-19T11:30:08.125 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.125 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.126 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:08.126 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.365896+0000", 2024-03-19T11:30:08.126 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1912999999999999e-05 2024-03-19T11:30:08.126 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.126 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.126 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:08.126 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.365902+0000", 2024-03-19T11:30:08.126 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3130000000000004e-06 2024-03-19T11:30:08.126 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.126 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.126 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:08.126 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.365952+0000", 2024-03-19T11:30:08.126 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0283e-05 2024-03-19T11:30:08.127 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.127 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.127 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:08.127 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.365966+0000", 2024-03-19T11:30:08.127 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.349e-05 2024-03-19T11:30:08.127 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.127 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.127 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:08.127 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.369869+0000", 2024-03-19T11:30:08.127 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0039034989999999999 2024-03-19T11:30:08.127 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.128 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.128 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:08.128 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.369895+0000", 2024-03-19T11:30:08.128 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5598000000000001e-05 2024-03-19T11:30:08.128 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.128 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:08.128 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.128 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.128 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.128 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:130 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'88, mlcod=16'85)", 2024-03-19T11:30:08.128 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.805586+0000", 2024-03-19T11:30:08.128 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 22.309806765000001, 2024-03-19T11:30:08.128 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0054104690000000002, 2024-03-19T11:30:08.129 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:08.129 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:08.129 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:08.129 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.129 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:08.129 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.805586+0000", 2024-03-19T11:30:08.129 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.129 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.129 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.129 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:08.129 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.805586+0000", 2024-03-19T11:30:08.129 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.130 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.130 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.130 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:08.130 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.805587+0000", 2024-03-19T11:30:08.130 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7269999999999999e-06 2024-03-19T11:30:08.130 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.130 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.130 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:08.130 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.805600+0000", 2024-03-19T11:30:08.130 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2293000000000001e-05 2024-03-19T11:30:08.130 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.130 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.131 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:08.131 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.805623+0000", 2024-03-19T11:30:08.131 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3592999999999999e-05 2024-03-19T11:30:08.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.131 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.131 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:08.131 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.805636+0000", 2024-03-19T11:30:08.131 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2660000000000001e-05 2024-03-19T11:30:08.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.131 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.131 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:08.131 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.805838+0000", 2024-03-19T11:30:08.131 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020228800000000001 2024-03-19T11:30:08.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.131 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.132 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:08.132 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.805878+0000", 2024-03-19T11:30:08.132 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.9586999999999999e-05 2024-03-19T11:30:08.132 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.132 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.132 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:08.132 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.810908+0000", 2024-03-19T11:30:08.132 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050303090000000002 2024-03-19T11:30:08.132 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.132 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.132 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:08.132 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.810996+0000", 2024-03-19T11:30:08.132 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8011999999999997e-05 2024-03-19T11:30:08.132 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.132 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:08.133 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.133 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.133 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.133 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:131 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'89, mlcod=16'85)", 2024-03-19T11:30:08.133 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.806096+0000", 2024-03-19T11:30:08.133 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 22.309296115999999, 2024-03-19T11:30:08.133 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0049616699999999996, 2024-03-19T11:30:08.133 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:08.133 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:08.133 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:08.133 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.133 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:08.133 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806096+0000", 2024-03-19T11:30:08.133 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.134 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.134 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.134 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:08.134 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806096+0000", 2024-03-19T11:30:08.134 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.134 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.134 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.134 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:08.134 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806098+0000", 2024-03-19T11:30:08.134 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.734e-06 2024-03-19T11:30:08.134 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.134 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.134 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:08.134 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806267+0000", 2024-03-19T11:30:08.135 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016936800000000001 2024-03-19T11:30:08.135 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.135 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.135 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:08.135 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806296+0000", 2024-03-19T11:30:08.135 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8653000000000002e-05 2024-03-19T11:30:08.135 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.135 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.135 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:08.135 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806315+0000", 2024-03-19T11:30:08.136 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9049999999999999e-05 2024-03-19T11:30:08.136 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.136 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.136 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:08.136 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806481+0000", 2024-03-19T11:30:08.136 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016540599999999999 2024-03-19T11:30:08.136 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.136 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.136 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:08.136 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806511+0000", 2024-03-19T11:30:08.136 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0626000000000001e-05 2024-03-19T11:30:08.137 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.137 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.137 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:08.137 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.811002+0000", 2024-03-19T11:30:08.137 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0044912399999999996 2024-03-19T11:30:08.137 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.137 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.137 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:08.137 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.811058+0000", 2024-03-19T11:30:08.137 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5593000000000001e-05 2024-03-19T11:30:08.137 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.137 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:08.137 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.138 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.138 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.138 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:132 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'90, mlcod=16'85)", 2024-03-19T11:30:08.138 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.806470+0000", 2024-03-19T11:30:08.138 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 22.308922625000001, 2024-03-19T11:30:08.138 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0066812390000000003, 2024-03-19T11:30:08.138 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:08.138 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:08.138 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:08.138 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.138 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:08.139 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806470+0000", 2024-03-19T11:30:08.139 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.139 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.139 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.139 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:08.139 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806470+0000", 2024-03-19T11:30:08.139 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.139 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.139 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.139 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:08.139 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806471+0000", 2024-03-19T11:30:08.139 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.486e-06 2024-03-19T11:30:08.139 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.139 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.140 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:08.140 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806485+0000", 2024-03-19T11:30:08.140 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3380000000000001e-05 2024-03-19T11:30:08.140 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.140 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.140 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:08.140 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806509+0000", 2024-03-19T11:30:08.140 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4617000000000002e-05 2024-03-19T11:30:08.140 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.140 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.140 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:08.141 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806524+0000", 2024-03-19T11:30:08.141 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4688999999999999e-05 2024-03-19T11:30:08.141 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.141 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.141 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:08.141 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807070+0000", 2024-03-19T11:30:08.141 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00054601300000000003 2024-03-19T11:30:08.141 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.141 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.141 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:08.141 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807104+0000", 2024-03-19T11:30:08.141 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3575999999999998e-05 2024-03-19T11:30:08.141 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.141 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.142 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:08.142 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813077+0000", 2024-03-19T11:30:08.142 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0059733590000000001 2024-03-19T11:30:08.142 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.142 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.142 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:08.142 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813151+0000", 2024-03-19T11:30:08.142 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4119e-05 2024-03-19T11:30:08.142 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.142 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:08.142 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.142 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.142 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.142 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:133 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'91, mlcod=16'85)", 2024-03-19T11:30:08.142 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.806875+0000", 2024-03-19T11:30:08.143 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 22.308516961999999, 2024-03-19T11:30:08.143 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0063196260000000001, 2024-03-19T11:30:08.143 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:08.143 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:08.143 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:08.143 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.143 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:08.143 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806875+0000", 2024-03-19T11:30:08.143 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.143 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.143 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.143 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:08.143 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806875+0000", 2024-03-19T11:30:08.143 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.143 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.144 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.144 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:08.144 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806877+0000", 2024-03-19T11:30:08.144 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8899999999999999e-06 2024-03-19T11:30:08.144 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.144 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.144 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:08.144 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806895+0000", 2024-03-19T11:30:08.144 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7980000000000001e-05 2024-03-19T11:30:08.144 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.144 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.144 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:08.144 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806926+0000", 2024-03-19T11:30:08.144 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0363000000000001e-05 2024-03-19T11:30:08.144 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.145 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.145 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:08.145 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806942+0000", 2024-03-19T11:30:08.145 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6422999999999998e-05 2024-03-19T11:30:08.145 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.145 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.145 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:08.145 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807659+0000", 2024-03-19T11:30:08.145 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00071724699999999996 2024-03-19T11:30:08.145 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.145 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.145 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:08.145 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807688+0000", 2024-03-19T11:30:08.145 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8940000000000001e-05 2024-03-19T11:30:08.145 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.146 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.146 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:08.146 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813157+0000", 2024-03-19T11:30:08.146 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0054689930000000001 2024-03-19T11:30:08.146 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.146 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.146 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:08.146 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813195+0000", 2024-03-19T11:30:08.146 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7790000000000002e-05 2024-03-19T11:30:08.146 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.146 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:08.146 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.146 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.146 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.147 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:134 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'92, mlcod=16'85)", 2024-03-19T11:30:08.147 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.807418+0000", 2024-03-19T11:30:08.147 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 22.307974439999999, 2024-03-19T11:30:08.147 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0058240560000000002, 2024-03-19T11:30:08.147 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:08.147 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:08.147 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:08.147 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.147 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:08.147 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807418+0000", 2024-03-19T11:30:08.147 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.147 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.147 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.147 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:08.147 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807418+0000", 2024-03-19T11:30:08.147 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.148 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.148 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.148 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:08.148 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807420+0000", 2024-03-19T11:30:08.148 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5600000000000001e-06 2024-03-19T11:30:08.148 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.148 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.148 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:08.148 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807514+0000", 2024-03-19T11:30:08.148 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4256e-05 2024-03-19T11:30:08.148 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.148 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.148 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:08.148 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807542+0000", 2024-03-19T11:30:08.149 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8283e-05 2024-03-19T11:30:08.149 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.149 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.149 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:08.149 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807562+0000", 2024-03-19T11:30:08.149 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0126999999999999e-05 2024-03-19T11:30:08.149 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.149 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.149 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:08.149 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808272+0000", 2024-03-19T11:30:08.149 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00071012400000000002 2024-03-19T11:30:08.149 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.149 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.149 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:08.149 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808308+0000", 2024-03-19T11:30:08.149 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5722999999999997e-05 2024-03-19T11:30:08.150 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.150 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.150 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:08.150 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813200+0000", 2024-03-19T11:30:08.150 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0048915010000000004 2024-03-19T11:30:08.150 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.150 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.150 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:08.150 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813242+0000", 2024-03-19T11:30:08.150 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2481999999999997e-05 2024-03-19T11:30:08.150 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.150 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:08.150 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.150 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.151 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.151 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:135 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'93, mlcod=16'85)", 2024-03-19T11:30:08.151 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.807817+0000", 2024-03-19T11:30:08.151 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 22.307575202999999, 2024-03-19T11:30:08.151 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0054653059999999996, 2024-03-19T11:30:08.151 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:08.151 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:08.151 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:08.151 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.151 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:08.151 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807817+0000", 2024-03-19T11:30:08.151 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.151 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.151 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.151 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:08.152 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807817+0000", 2024-03-19T11:30:08.152 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.152 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.152 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.152 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:08.152 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807819+0000", 2024-03-19T11:30:08.152 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7999999999999999e-06 2024-03-19T11:30:08.152 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.152 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.152 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:08.152 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807833+0000", 2024-03-19T11:30:08.152 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3546999999999999e-05 2024-03-19T11:30:08.152 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.152 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.152 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:08.153 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807858+0000", 2024-03-19T11:30:08.153 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5119000000000001e-05 2024-03-19T11:30:08.153 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.153 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.153 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:08.153 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807871+0000", 2024-03-19T11:30:08.153 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.287e-05 2024-03-19T11:30:08.153 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.153 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.153 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:08.153 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808989+0000", 2024-03-19T11:30:08.153 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001118841 2024-03-19T11:30:08.153 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.154 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.154 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:08.154 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.809022+0000", 2024-03-19T11:30:08.154 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3000000000000003e-05 2024-03-19T11:30:08.154 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.154 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.154 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:08.154 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813249+0000", 2024-03-19T11:30:08.154 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0042268260000000004 2024-03-19T11:30:08.154 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.154 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.154 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:08.154 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813283+0000", 2024-03-19T11:30:08.154 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3303000000000003e-05 2024-03-19T11:30:08.154 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.155 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:08.155 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.155 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.155 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.155 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:136 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'94, mlcod=16'85)", 2024-03-19T11:30:08.155 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.808219+0000", 2024-03-19T11:30:08.155 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 22.307173923000001, 2024-03-19T11:30:08.155 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005105086, 2024-03-19T11:30:08.155 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:08.155 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:08.155 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:08.155 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.156 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:08.156 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808219+0000", 2024-03-19T11:30:08.156 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.156 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.156 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.156 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:08.156 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808219+0000", 2024-03-19T11:30:08.156 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.156 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.156 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.156 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:08.156 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808220+0000", 2024-03-19T11:30:08.156 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7600000000000001e-06 2024-03-19T11:30:08.157 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.157 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.157 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:08.157 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808233+0000", 2024-03-19T11:30:08.157 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2916e-05 2024-03-19T11:30:08.157 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.157 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.157 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:08.157 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808294+0000", 2024-03-19T11:30:08.157 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1199999999999997e-05 2024-03-19T11:30:08.157 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.157 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.157 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:08.157 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808310+0000", 2024-03-19T11:30:08.158 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5996999999999999e-05 2024-03-19T11:30:08.158 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.158 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.158 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:08.158 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.809445+0000", 2024-03-19T11:30:08.158 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001134227 2024-03-19T11:30:08.158 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.158 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.158 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:08.158 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.809473+0000", 2024-03-19T11:30:08.158 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8603e-05 2024-03-19T11:30:08.158 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.158 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.158 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:08.159 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813291+0000", 2024-03-19T11:30:08.159 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038174799999999998 2024-03-19T11:30:08.159 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.159 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.159 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:08.159 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813324+0000", 2024-03-19T11:30:08.159 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2903e-05 2024-03-19T11:30:08.159 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.159 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:08.159 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.159 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.159 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.159 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:137 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'95, mlcod=16'85)", 2024-03-19T11:30:08.159 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.808663+0000", 2024-03-19T11:30:08.159 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 22.30672977, 2024-03-19T11:30:08.159 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0047004350000000002, 2024-03-19T11:30:08.160 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:08.160 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:08.160 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:08.160 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.160 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:08.160 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808663+0000", 2024-03-19T11:30:08.160 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.160 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.160 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.160 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:08.160 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808663+0000", 2024-03-19T11:30:08.160 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.160 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.160 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.160 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:08.161 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808665+0000", 2024-03-19T11:30:08.161 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9300000000000002e-06 2024-03-19T11:30:08.161 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.161 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.161 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:08.161 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808768+0000", 2024-03-19T11:30:08.161 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000103009 2024-03-19T11:30:08.161 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.161 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.161 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:08.161 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808792+0000", 2024-03-19T11:30:08.162 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3972999999999999e-05 2024-03-19T11:30:08.162 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.162 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.162 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:08.162 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808807+0000", 2024-03-19T11:30:08.162 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5614e-05 2024-03-19T11:30:08.162 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.162 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.162 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:08.162 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.809896+0000", 2024-03-19T11:30:08.163 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0010891270000000001 2024-03-19T11:30:08.163 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.163 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.163 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:08.163 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.809930+0000", 2024-03-19T11:30:08.163 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3444e-05 2024-03-19T11:30:08.163 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.163 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.163 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:08.164 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813330+0000", 2024-03-19T11:30:08.164 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003399839 2024-03-19T11:30:08.164 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.164 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.165 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:08.165 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813363+0000", 2024-03-19T11:30:08.165 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3498999999999999e-05 2024-03-19T11:30:08.165 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.165 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:08.165 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.165 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.165 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.165 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:141 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'99, mlcod=16'86)", 2024-03-19T11:30:08.165 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.812015+0000", 2024-03-19T11:30:08.166 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 22.303377743999999, 2024-03-19T11:30:08.166 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003849239, 2024-03-19T11:30:08.166 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:08.166 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:08.166 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:08.166 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.166 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:08.166 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.812015+0000", 2024-03-19T11:30:08.166 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.166 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.166 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.166 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:08.166 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.812015+0000", 2024-03-19T11:30:08.167 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.167 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.167 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.167 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:08.167 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.812017+0000", 2024-03-19T11:30:08.167 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8470000000000001e-06 2024-03-19T11:30:08.167 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.167 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.167 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:08.167 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.812030+0000", 2024-03-19T11:30:08.167 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3867e-05 2024-03-19T11:30:08.167 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.167 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.168 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:08.168 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.812058+0000", 2024-03-19T11:30:08.168 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8005999999999999e-05 2024-03-19T11:30:08.168 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.168 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.168 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:08.168 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.812070+0000", 2024-03-19T11:30:08.168 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1207e-05 2024-03-19T11:30:08.168 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.168 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.169 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:08.169 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813509+0000", 2024-03-19T11:30:08.169 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014389909999999999 2024-03-19T11:30:08.169 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.169 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.169 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:08.169 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813564+0000", 2024-03-19T11:30:08.169 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5420000000000001e-05 2024-03-19T11:30:08.169 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.169 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.169 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:08.169 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.815782+0000", 2024-03-19T11:30:08.169 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022183419999999999 2024-03-19T11:30:08.169 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.170 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.170 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:08.170 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.815864+0000", 2024-03-19T11:30:08.170 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1558999999999996e-05 2024-03-19T11:30:08.170 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.170 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:08.170 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.170 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.170 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.170 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:5 4.6 4:626acd4d:::smithi03829856-2:head [write 416807~475249,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-03-19T11:30:08.170 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.688442+0000", 2024-03-19T11:30:08.170 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.426950111, 2024-03-19T11:30:08.171 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.004734121, 2024-03-19T11:30:08.171 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:08.171 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:08.171 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:08.171 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:08.171 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:08.171 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 5 2024-03-19T11:30:08.171 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.171 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:08.171 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.171 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:08.171 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.688442+0000", 2024-03-19T11:30:08.172 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.172 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.173 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.173 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:08.173 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.688442+0000", 2024-03-19T11:30:08.174 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.174 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.174 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.174 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:08.174 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.688445+0000", 2024-03-19T11:30:08.174 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1629999999999999e-06 2024-03-19T11:30:08.174 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.174 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.174 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:08.174 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.688905+0000", 2024-03-19T11:30:08.174 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000460536 2024-03-19T11:30:08.174 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.174 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.174 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:08.175 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.688925+0000", 2024-03-19T11:30:08.175 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0404999999999999e-05 2024-03-19T11:30:08.175 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.175 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.175 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:08.175 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.688940+0000", 2024-03-19T11:30:08.175 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4792e-05 2024-03-19T11:30:08.175 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.175 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.175 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:08.175 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.689031+0000", 2024-03-19T11:30:08.175 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0618000000000006e-05 2024-03-19T11:30:08.175 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.175 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.175 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:08.176 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.689209+0000", 2024-03-19T11:30:08.176 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000177868 2024-03-19T11:30:08.176 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.176 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.176 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:08.176 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.689380+0000", 2024-03-19T11:30:08.176 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017090700000000001 2024-03-19T11:30:08.176 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.176 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.176 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:08.176 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.692293+0000", 2024-03-19T11:30:08.176 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002913728 2024-03-19T11:30:08.176 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.176 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.176 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:08.176 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.693042+0000", 2024-03-19T11:30:08.177 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00074911300000000004 2024-03-19T11:30:08.177 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.177 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.177 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:08.177 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.693112+0000", 2024-03-19T11:30:08.177 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9846999999999997e-05 2024-03-19T11:30:08.177 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.177 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.177 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:08.177 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.693176+0000", 2024-03-19T11:30:08.177 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4144000000000001e-05 2024-03-19T11:30:08.177 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.177 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:08.177 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.177 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.178 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.178 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:6 4.6 4:626acd4d:::smithi03829856-2:head [write 1542129~474678] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-03-19T11:30:08.178 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.696120+0000", 2024-03-19T11:30:08.178 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.4192726159999998, 2024-03-19T11:30:08.178 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0082896359999999995, 2024-03-19T11:30:08.178 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:08.178 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:08.178 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:08.178 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:08.178 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:08.178 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 6 2024-03-19T11:30:08.178 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.178 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:08.178 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.178 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:08.179 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696120+0000", 2024-03-19T11:30:08.179 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.179 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.179 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.179 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:08.179 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696120+0000", 2024-03-19T11:30:08.179 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.179 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.179 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.179 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:08.179 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696123+0000", 2024-03-19T11:30:08.179 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.661e-06 2024-03-19T11:30:08.179 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.179 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.179 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:08.180 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696282+0000", 2024-03-19T11:30:08.180 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015960400000000001 2024-03-19T11:30:08.180 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.180 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.180 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:08.180 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696307+0000", 2024-03-19T11:30:08.180 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5352999999999999e-05 2024-03-19T11:30:08.180 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.180 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.181 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:08.181 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696320+0000", 2024-03-19T11:30:08.181 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2847000000000001e-05 2024-03-19T11:30:08.181 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.181 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.181 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:08.181 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696403+0000", 2024-03-19T11:30:08.181 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2507000000000001e-05 2024-03-19T11:30:08.181 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.181 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.181 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:08.181 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696585+0000", 2024-03-19T11:30:08.182 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00018234999999999999 2024-03-19T11:30:08.182 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.182 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.182 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:08.182 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696770+0000", 2024-03-19T11:30:08.182 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000184377 2024-03-19T11:30:08.182 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.182 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.182 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:08.182 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.699608+0000", 2024-03-19T11:30:08.182 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00283797 2024-03-19T11:30:08.182 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.182 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.182 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:08.183 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.704252+0000", 2024-03-19T11:30:08.183 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046443980000000001 2024-03-19T11:30:08.183 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.183 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.183 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:08.183 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.704349+0000", 2024-03-19T11:30:08.183 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7388999999999999e-05 2024-03-19T11:30:08.183 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.183 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.183 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:08.183 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.704409+0000", 2024-03-19T11:30:08.183 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0180000000000003e-05 2024-03-19T11:30:08.183 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.183 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:08.183 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.184 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.184 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.184 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:7 4.6 4:626acd4d:::smithi03829856-2:head [setxattr _header (55) in=7b,truncate 2016807] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-03-19T11:30:08.184 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.696352+0000", 2024-03-19T11:30:08.184 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.4190404760000002, 2024-03-19T11:30:08.184 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0084077100000000005, 2024-03-19T11:30:08.184 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:08.184 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:08.184 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:08.184 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:08.185 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:08.185 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 7 2024-03-19T11:30:08.185 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.185 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:08.185 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.185 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:08.185 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696352+0000", 2024-03-19T11:30:08.185 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.185 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.185 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.185 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:08.186 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696352+0000", 2024-03-19T11:30:08.186 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.186 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.186 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.186 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:08.186 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696353+0000", 2024-03-19T11:30:08.186 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1399999999999996e-07 2024-03-19T11:30:08.186 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.186 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.186 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:08.186 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696358+0000", 2024-03-19T11:30:08.186 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5300000000000004e-06 2024-03-19T11:30:08.186 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.186 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.187 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:08.187 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696368+0000", 2024-03-19T11:30:08.187 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6209999999999999e-06 2024-03-19T11:30:08.187 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.187 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.187 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:08.187 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696374+0000", 2024-03-19T11:30:08.187 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5599999999999999e-06 2024-03-19T11:30:08.187 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.187 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.187 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:08.187 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.698823+0000", 2024-03-19T11:30:08.187 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0024489899999999998 2024-03-19T11:30:08.187 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.187 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.188 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:08.188 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.698925+0000", 2024-03-19T11:30:08.188 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010166399999999999 2024-03-19T11:30:08.188 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.188 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.188 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:08.188 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.699071+0000", 2024-03-19T11:30:08.188 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000146014 2024-03-19T11:30:08.188 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.188 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.188 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:08.188 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.703885+0000", 2024-03-19T11:30:08.188 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.004813866 2024-03-19T11:30:08.189 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.189 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.189 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:08.189 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.704622+0000", 2024-03-19T11:30:08.189 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00073729600000000002 2024-03-19T11:30:08.189 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.189 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.189 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:08.189 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.704688+0000", 2024-03-19T11:30:08.189 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6192000000000006e-05 2024-03-19T11:30:08.189 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.189 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.189 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:08.189 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.704760+0000", 2024-03-19T11:30:08.189 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1262999999999999e-05 2024-03-19T11:30:08.190 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.190 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:08.190 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.190 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.190 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.190 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:8 4.6 4:626acd4d:::smithi03829856-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-03-19T11:30:08.190 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.696385+0000", 2024-03-19T11:30:08.190 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.4190071770000001, 2024-03-19T11:30:08.190 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0086443730000000003, 2024-03-19T11:30:08.190 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:08.190 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:08.190 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:08.190 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:08.190 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:08.191 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 8 2024-03-19T11:30:08.191 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.191 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:08.191 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.191 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:08.191 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696385+0000", 2024-03-19T11:30:08.191 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.191 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.191 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.191 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:08.191 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696385+0000", 2024-03-19T11:30:08.191 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.191 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.191 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.191 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:08.192 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696386+0000", 2024-03-19T11:30:08.192 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3e-07 2024-03-19T11:30:08.192 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.192 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.192 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:08.192 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696390+0000", 2024-03-19T11:30:08.192 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0130000000000004e-06 2024-03-19T11:30:08.192 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.192 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.192 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:08.192 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696399+0000", 2024-03-19T11:30:08.192 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9139999999999997e-06 2024-03-19T11:30:08.192 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.192 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.192 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:08.193 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696405+0000", 2024-03-19T11:30:08.193 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4699999999999999e-06 2024-03-19T11:30:08.193 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.193 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.193 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:08.193 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.699495+0000", 2024-03-19T11:30:08.193 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030890969999999998 2024-03-19T11:30:08.193 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.193 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.193 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-03-19T11:30:08.193 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.699562+0000", 2024-03-19T11:30:08.193 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7980999999999998e-05 2024-03-19T11:30:08.193 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.193 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.193 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:08.194 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.704809+0000", 2024-03-19T11:30:08.194 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0052468769999999996 2024-03-19T11:30:08.194 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.194 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.194 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:08.194 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.704909+0000", 2024-03-19T11:30:08.194 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9189000000000002e-05 2024-03-19T11:30:08.194 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.194 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.194 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:08.194 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.705030+0000", 2024-03-19T11:30:08.194 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012110199999999999 2024-03-19T11:30:08.194 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.195 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:08.195 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.195 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.195 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.195 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:21 4.b 4:db322469:::smithi03829856-6:head [write 484035~776245,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:30:08.195 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.746924+0000", 2024-03-19T11:30:08.195 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.3684684329999999, 2024-03-19T11:30:08.195 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074997270000000003, 2024-03-19T11:30:08.195 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:08.195 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:08.195 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:08.195 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:08.195 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:08.195 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 21 2024-03-19T11:30:08.195 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.196 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:08.196 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.196 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:08.196 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.746924+0000", 2024-03-19T11:30:08.196 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.196 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.196 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.196 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:08.196 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.746924+0000", 2024-03-19T11:30:08.196 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.196 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.196 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.196 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:08.196 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.746927+0000", 2024-03-19T11:30:08.196 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4820000000000001e-06 2024-03-19T11:30:08.197 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.197 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.197 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:08.197 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.747304+0000", 2024-03-19T11:30:08.197 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037742299999999999 2024-03-19T11:30:08.197 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.197 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.197 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:08.197 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.747326+0000", 2024-03-19T11:30:08.197 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1648999999999999e-05 2024-03-19T11:30:08.197 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.197 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.197 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:08.197 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.747336+0000", 2024-03-19T11:30:08.198 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0288e-05 2024-03-19T11:30:08.198 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.198 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.198 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:08.198 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.747414+0000", 2024-03-19T11:30:08.198 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8085999999999997e-05 2024-03-19T11:30:08.198 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.198 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.198 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-03-19T11:30:08.198 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.747421+0000", 2024-03-19T11:30:08.198 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9609999999999999e-06 2024-03-19T11:30:08.198 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.198 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.199 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:08.199 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.749216+0000", 2024-03-19T11:30:08.199 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001795172 2024-03-19T11:30:08.199 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.199 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.199 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:08.199 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.749355+0000", 2024-03-19T11:30:08.199 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013904799999999999 2024-03-19T11:30:08.199 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.199 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.199 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:08.199 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.749513+0000", 2024-03-19T11:30:08.199 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000157453 2024-03-19T11:30:08.199 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.199 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.200 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:08.200 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.754081+0000", 2024-03-19T11:30:08.200 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0045681230000000003 2024-03-19T11:30:08.200 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.200 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.200 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:08.200 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.754323+0000", 2024-03-19T11:30:08.200 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00024241900000000001 2024-03-19T11:30:08.200 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.200 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.200 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:08.200 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.754373+0000", 2024-03-19T11:30:08.200 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9632999999999997e-05 2024-03-19T11:30:08.200 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.201 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.201 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:08.201 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.754424+0000", 2024-03-19T11:30:08.201 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0989999999999998e-05 2024-03-19T11:30:08.201 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.201 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:08.201 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.201 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.201 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.201 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:25 4.3 4:cef44a0d:::smithi03829856-7:head [write 500842~451494,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:30:08.201 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.757843+0000", 2024-03-19T11:30:08.201 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.357549364, 2024-03-19T11:30:08.201 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0059188269999999998, 2024-03-19T11:30:08.201 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:08.201 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:08.202 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:08.202 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:08.202 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:08.202 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 25 2024-03-19T11:30:08.202 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.202 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:08.202 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.202 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:08.202 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.757843+0000", 2024-03-19T11:30:08.202 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.202 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.202 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.202 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:08.202 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.757843+0000", 2024-03-19T11:30:08.202 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.203 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.203 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.203 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:08.203 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.757845+0000", 2024-03-19T11:30:08.203 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.424e-06 2024-03-19T11:30:08.203 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.203 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.203 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:08.203 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.757987+0000", 2024-03-19T11:30:08.203 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014264400000000001 2024-03-19T11:30:08.203 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.203 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.203 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:08.203 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.758005+0000", 2024-03-19T11:30:08.203 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7629e-05 2024-03-19T11:30:08.204 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.204 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.204 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:08.204 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.758014+0000", 2024-03-19T11:30:08.204 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9500000000000007e-06 2024-03-19T11:30:08.204 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.204 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.204 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:08.204 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.758080+0000", 2024-03-19T11:30:08.204 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6122000000000002e-05 2024-03-19T11:30:08.204 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.204 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.204 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:08.204 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.758217+0000", 2024-03-19T11:30:08.204 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013761000000000001 2024-03-19T11:30:08.204 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.205 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.205 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:08.205 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.758354+0000", 2024-03-19T11:30:08.205 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000136706 2024-03-19T11:30:08.205 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.205 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.205 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:08.205 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.760460+0000", 2024-03-19T11:30:08.205 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002106199 2024-03-19T11:30:08.205 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.205 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.205 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:08.205 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.763649+0000", 2024-03-19T11:30:08.205 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031888630000000001 2024-03-19T11:30:08.205 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.205 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.206 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:08.206 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.763707+0000", 2024-03-19T11:30:08.206 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7773999999999999e-05 2024-03-19T11:30:08.206 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.206 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.206 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:08.206 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.763762+0000", 2024-03-19T11:30:08.206 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4905999999999999e-05 2024-03-19T11:30:08.206 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.206 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:08.206 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.206 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.206 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:08.206 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:08.208 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-19T11:30:08.208 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:08.216+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:08.309 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:08.319+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:08.382 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:08.382 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:30:08.382 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:30:08.383 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:08.398 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-19T11:30:08.443 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:08.452+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:08.511 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:08.523+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:08.524 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:08.524 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:30:08.525 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:30:08.525 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:30:08.525 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:08.538 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-19T11:30:08.611 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:08.619+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:08.712 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:08.720+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:08.716 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:08.716 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:30:08.716 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:30:08.716 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:30:08.716 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.716 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:5 4.6 e27/21 4:626acd4d:::smithi03829856-2:head v 27'1, mlcod=0'0)", 2024-03-19T11:30:08.717 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.689822+0000", 2024-03-19T11:30:08.717 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.0366976709999998, 2024-03-19T11:30:08.717 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030572360000000001, 2024-03-19T11:30:08.717 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:08.717 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:08.717 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:08.717 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.717 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:08.717 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.689822+0000", 2024-03-19T11:30:08.717 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.718 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.718 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.718 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:08.718 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.689822+0000", 2024-03-19T11:30:08.718 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.718 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.718 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.718 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:08.718 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.689823+0000", 2024-03-19T11:30:08.719 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.39e-07 2024-03-19T11:30:08.719 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.719 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.719 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:08.719 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.690184+0000", 2024-03-19T11:30:08.719 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00036105399999999998 2024-03-19T11:30:08.719 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.719 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.719 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:08.719 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.690201+0000", 2024-03-19T11:30:08.719 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7317999999999999e-05 2024-03-19T11:30:08.720 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.720 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.720 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:08.720 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.690211+0000", 2024-03-19T11:30:08.720 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9529999999999994e-06 2024-03-19T11:30:08.720 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.720 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.720 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:08.720 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.690288+0000", 2024-03-19T11:30:08.720 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6933000000000004e-05 2024-03-19T11:30:08.720 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.721 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.721 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:08.721 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.690324+0000", 2024-03-19T11:30:08.721 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6038000000000003e-05 2024-03-19T11:30:08.721 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.721 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.721 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:08.721 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.692797+0000", 2024-03-19T11:30:08.721 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0024729230000000001 2024-03-19T11:30:08.722 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.722 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.722 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:08.722 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.692880+0000", 2024-03-19T11:30:08.722 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2677999999999997e-05 2024-03-19T11:30:08.722 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.722 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:08.722 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.722 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.722 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.722 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:6 4.6 e27/21 4:626acd4d:::smithi03829856-2:head v 27'2, mlcod=0'0)", 2024-03-19T11:30:08.723 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.696903+0000", 2024-03-19T11:30:08.723 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.029617537, 2024-03-19T11:30:08.723 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0071029129999999998, 2024-03-19T11:30:08.723 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:08.723 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:08.723 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:08.723 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.723 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:08.723 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696903+0000", 2024-03-19T11:30:08.723 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.723 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.723 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.724 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:08.724 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696903+0000", 2024-03-19T11:30:08.724 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.724 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.724 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.724 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:08.724 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696904+0000", 2024-03-19T11:30:08.724 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4270000000000001e-06 2024-03-19T11:30:08.724 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.724 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.724 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:08.724 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.697064+0000", 2024-03-19T11:30:08.725 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015956100000000001 2024-03-19T11:30:08.725 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.725 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.725 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:08.725 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.697082+0000", 2024-03-19T11:30:08.725 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8868999999999998e-05 2024-03-19T11:30:08.725 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.725 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.725 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:08.725 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.697096+0000", 2024-03-19T11:30:08.725 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3155e-05 2024-03-19T11:30:08.725 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.726 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.726 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:08.726 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.697169+0000", 2024-03-19T11:30:08.726 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3226999999999995e-05 2024-03-19T11:30:08.726 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.726 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.726 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:08.726 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.697193+0000", 2024-03-19T11:30:08.726 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4153000000000001e-05 2024-03-19T11:30:08.726 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.726 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.726 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:08.727 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.703936+0000", 2024-03-19T11:30:08.727 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0067430570000000002 2024-03-19T11:30:08.727 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.727 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.727 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:08.727 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.704006+0000", 2024-03-19T11:30:08.727 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9463999999999998e-05 2024-03-19T11:30:08.727 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.727 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:08.728 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.728 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.728 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.728 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:7 4.6 e27/21 4:626acd4d:::smithi03829856-2:head v 27'3, mlcod=0'0)", 2024-03-19T11:30:08.728 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.699209+0000", 2024-03-19T11:30:08.728 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.0273111239999997, 2024-03-19T11:30:08.728 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00524114, 2024-03-19T11:30:08.728 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:08.728 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:08.728 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:08.728 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.728 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:08.728 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.699209+0000", 2024-03-19T11:30:08.729 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.729 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.729 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.729 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:08.729 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.699209+0000", 2024-03-19T11:30:08.729 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.729 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.729 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.729 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:08.729 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.699211+0000", 2024-03-19T11:30:08.729 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7230000000000001e-06 2024-03-19T11:30:08.729 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.730 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.730 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:08.730 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.699221+0000", 2024-03-19T11:30:08.730 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0698e-05 2024-03-19T11:30:08.730 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.730 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.730 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:08.730 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.699237+0000", 2024-03-19T11:30:08.730 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5665999999999999e-05 2024-03-19T11:30:08.730 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.730 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.730 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:08.731 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.699247+0000", 2024-03-19T11:30:08.731 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7880000000000001e-06 2024-03-19T11:30:08.731 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.731 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.731 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:08.731 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.699304+0000", 2024-03-19T11:30:08.731 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7120999999999997e-05 2024-03-19T11:30:08.731 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.731 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.731 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:08.731 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.699327+0000", 2024-03-19T11:30:08.732 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2795000000000001e-05 2024-03-19T11:30:08.732 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.732 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.732 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:08.732 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.704397+0000", 2024-03-19T11:30:08.732 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050705790000000004 2024-03-19T11:30:08.732 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.732 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.732 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:08.732 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.704450+0000", 2024-03-19T11:30:08.732 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2769999999999998e-05 2024-03-19T11:30:08.732 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.732 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:08.732 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.733 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.733 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.733 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:13 4.4 4:21eac9b2:::smithi03829856-4:head [write 450421~625747,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:30:08.733 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.720170+0000", 2024-03-19T11:30:08.733 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.0063502230000001, 2024-03-19T11:30:08.733 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0090522840000000007, 2024-03-19T11:30:08.733 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:08.733 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:08.733 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:08.733 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:08.733 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:08.733 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 13 2024-03-19T11:30:08.733 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.733 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:08.734 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.734 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:08.734 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.720170+0000", 2024-03-19T11:30:08.734 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.734 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.734 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.734 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:08.734 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.720170+0000", 2024-03-19T11:30:08.734 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.734 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.735 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.735 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:08.735 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.720172+0000", 2024-03-19T11:30:08.735 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9709999999999998e-06 2024-03-19T11:30:08.735 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.735 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.735 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:08.736 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.720601+0000", 2024-03-19T11:30:08.736 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00042897999999999998 2024-03-19T11:30:08.736 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.736 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.736 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:08.736 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.720615+0000", 2024-03-19T11:30:08.736 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4617000000000001e-05 2024-03-19T11:30:08.736 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.736 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.737 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:08.737 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.720627+0000", 2024-03-19T11:30:08.737 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1513e-05 2024-03-19T11:30:08.737 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.737 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.737 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:08.737 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.720705+0000", 2024-03-19T11:30:08.738 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7880000000000007e-05 2024-03-19T11:30:08.738 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.738 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.738 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-03-19T11:30:08.738 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.720709+0000", 2024-03-19T11:30:08.738 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5549999999999996e-06 2024-03-19T11:30:08.738 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.738 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.739 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:08.739 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.722748+0000", 2024-03-19T11:30:08.739 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0020388630000000001 2024-03-19T11:30:08.739 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.739 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.739 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:08.739 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.722891+0000", 2024-03-19T11:30:08.739 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014286 2024-03-19T11:30:08.739 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.740 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.740 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 2", 2024-03-19T11:30:08.740 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723057+0000", 2024-03-19T11:30:08.740 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000166286 2024-03-19T11:30:08.740 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.740 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.740 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:08.740 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.725297+0000", 2024-03-19T11:30:08.741 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022397770000000001 2024-03-19T11:30:08.741 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.741 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.741 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:08.741 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.729110+0000", 2024-03-19T11:30:08.741 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038130109999999998 2024-03-19T11:30:08.741 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.741 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.741 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:08.741 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.729169+0000", 2024-03-19T11:30:08.742 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8616000000000003e-05 2024-03-19T11:30:08.742 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.742 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.742 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:08.742 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.729222+0000", 2024-03-19T11:30:08.742 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3355e-05 2024-03-19T11:30:08.742 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.742 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:08.742 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.742 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.742 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.742 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:14 4.4 4:21eac9b2:::smithi03829856-4:head [write 1659093~391328] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:30:08.743 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.725543+0000", 2024-03-19T11:30:08.743 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.0009773979999999, 2024-03-19T11:30:08.743 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038586969999999999, 2024-03-19T11:30:08.743 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:08.743 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:08.743 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:08.743 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:08.743 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:08.743 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 14 2024-03-19T11:30:08.743 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.743 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:08.744 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.744 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:08.744 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.725543+0000", 2024-03-19T11:30:08.744 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.744 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.744 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.744 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:08.744 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.725543+0000", 2024-03-19T11:30:08.744 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.744 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.744 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.744 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:08.745 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.725545+0000", 2024-03-19T11:30:08.745 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.844e-06 2024-03-19T11:30:08.745 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.745 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.745 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:08.745 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.725702+0000", 2024-03-19T11:30:08.745 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015712500000000001 2024-03-19T11:30:08.745 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.745 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.745 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:08.745 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.725717+0000", 2024-03-19T11:30:08.746 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.508e-05 2024-03-19T11:30:08.746 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.746 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.746 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:08.746 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.725728+0000", 2024-03-19T11:30:08.746 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1106e-05 2024-03-19T11:30:08.746 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.746 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.746 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:08.746 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.725794+0000", 2024-03-19T11:30:08.746 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6045999999999998e-05 2024-03-19T11:30:08.747 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.747 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.747 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:08.747 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.725912+0000", 2024-03-19T11:30:08.747 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000118497 2024-03-19T11:30:08.747 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.747 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.747 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 2", 2024-03-19T11:30:08.747 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.726004+0000", 2024-03-19T11:30:08.747 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1321000000000002e-05 2024-03-19T11:30:08.747 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.747 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.748 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:08.748 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.729239+0000", 2024-03-19T11:30:08.748 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0032350080000000002 2024-03-19T11:30:08.748 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.748 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.748 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:08.748 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.729333+0000", 2024-03-19T11:30:08.748 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4653000000000003e-05 2024-03-19T11:30:08.748 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.748 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.748 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:08.749 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.729365+0000", 2024-03-19T11:30:08.749 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1720999999999997e-05 2024-03-19T11:30:08.749 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.749 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.749 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:08.749 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.729401+0000", 2024-03-19T11:30:08.749 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6295999999999999e-05 2024-03-19T11:30:08.749 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.749 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:08.749 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.749 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.749 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.750 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:15 4.4 4:21eac9b2:::smithi03829856-4:head [setxattr _header (55) in=7b,truncate 2050421] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:30:08.750 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.725770+0000", 2024-03-19T11:30:08.750 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.0007501200000002, 2024-03-19T11:30:08.750 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.004453622, 2024-03-19T11:30:08.750 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:08.750 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:08.750 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:08.750 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:08.750 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:08.750 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 15 2024-03-19T11:30:08.750 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.750 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:08.751 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.751 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:08.751 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.725770+0000", 2024-03-19T11:30:08.751 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.751 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.751 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.751 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:08.751 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.725770+0000", 2024-03-19T11:30:08.751 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.751 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.751 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.751 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:08.751 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.725771+0000", 2024-03-19T11:30:08.752 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4300000000000001e-07 2024-03-19T11:30:08.752 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.752 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.752 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:08.752 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.725776+0000", 2024-03-19T11:30:08.752 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5289999999999999e-06 2024-03-19T11:30:08.752 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.753 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.753 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:08.753 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.725785+0000", 2024-03-19T11:30:08.753 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8310000000000007e-06 2024-03-19T11:30:08.753 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.753 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.753 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:08.753 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.725793+0000", 2024-03-19T11:30:08.753 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1669999999999999e-06 2024-03-19T11:30:08.753 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.753 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.753 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:08.754 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.728614+0000", 2024-03-19T11:30:08.754 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0028204269999999999 2024-03-19T11:30:08.754 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.754 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.754 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:08.754 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.728681+0000", 2024-03-19T11:30:08.754 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6903999999999995e-05 2024-03-19T11:30:08.754 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.754 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.755 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 2", 2024-03-19T11:30:08.755 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.728809+0000", 2024-03-19T11:30:08.755 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000128603 2024-03-19T11:30:08.755 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.755 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.755 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:08.755 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.729577+0000", 2024-03-19T11:30:08.755 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00076769900000000005 2024-03-19T11:30:08.755 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.755 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.756 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:08.756 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.730105+0000", 2024-03-19T11:30:08.756 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00052791999999999997 2024-03-19T11:30:08.756 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.756 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.756 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:08.756 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.730164+0000", 2024-03-19T11:30:08.756 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8584e-05 2024-03-19T11:30:08.756 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.756 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.757 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:08.757 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.730224+0000", 2024-03-19T11:30:08.757 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0015000000000001e-05 2024-03-19T11:30:08.757 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.757 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:08.757 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.757 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.757 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.757 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:16 4.4 4:21eac9b2:::smithi03829856-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:30:08.757 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.725805+0000", 2024-03-19T11:30:08.758 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.0007154380000003, 2024-03-19T11:30:08.758 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0045749199999999997, 2024-03-19T11:30:08.758 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:08.758 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:08.758 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:08.758 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:08.758 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:08.758 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 16 2024-03-19T11:30:08.758 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.759 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:08.759 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.759 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:08.759 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.725805+0000", 2024-03-19T11:30:08.759 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.759 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.759 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.759 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:08.759 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.725805+0000", 2024-03-19T11:30:08.759 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.760 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.760 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.760 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:08.760 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.725805+0000", 2024-03-19T11:30:08.760 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1900000000000003e-07 2024-03-19T11:30:08.760 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.760 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.760 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:08.760 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.725808+0000", 2024-03-19T11:30:08.760 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0189999999999998e-06 2024-03-19T11:30:08.760 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.760 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.760 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:08.761 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.725816+0000", 2024-03-19T11:30:08.761 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6210000000000004e-06 2024-03-19T11:30:08.761 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.761 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.761 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:08.761 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.725821+0000", 2024-03-19T11:30:08.761 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7609999999999998e-06 2024-03-19T11:30:08.761 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.761 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.761 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:08.761 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.729002+0000", 2024-03-19T11:30:08.761 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003181377 2024-03-19T11:30:08.762 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.762 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.762 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-03-19T11:30:08.762 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.729069+0000", 2024-03-19T11:30:08.762 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6755999999999996e-05 2024-03-19T11:30:08.762 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.762 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.762 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:08.762 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.730268+0000", 2024-03-19T11:30:08.762 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011995490000000001 2024-03-19T11:30:08.762 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.762 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.763 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:08.763 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.730314+0000", 2024-03-19T11:30:08.763 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5534999999999997e-05 2024-03-19T11:30:08.763 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.763 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.763 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:08.763 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.730380+0000", 2024-03-19T11:30:08.763 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5783000000000004e-05 2024-03-19T11:30:08.763 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.763 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:08.763 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.764 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.764 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.764 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop_reply(client.4415.0:13 4.4 e28/21 ondisk, result = 0)", 2024-03-19T11:30:08.764 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.728211+0000", 2024-03-19T11:30:08.764 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.9983088879999999, 2024-03-19T11:30:08.764 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0010288039999999999, 2024-03-19T11:30:08.764 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:08.764 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:08.764 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:08.764 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.765 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:08.765 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.728211+0000", 2024-03-19T11:30:08.765 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.765 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.765 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.765 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:08.765 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.728211+0000", 2024-03-19T11:30:08.765 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.766 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.766 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.766 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:08.766 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.728212+0000", 2024-03-19T11:30:08.766 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2440000000000001e-06 2024-03-19T11:30:08.766 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.766 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.766 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:08.766 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.728219+0000", 2024-03-19T11:30:08.767 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2689999999999996e-06 2024-03-19T11:30:08.767 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.767 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.767 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:08.767 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.728230+0000", 2024-03-19T11:30:08.767 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1422e-05 2024-03-19T11:30:08.767 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.767 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.768 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:08.768 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.728239+0000", 2024-03-19T11:30:08.768 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.422e-06 2024-03-19T11:30:08.768 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.768 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.768 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:08.768 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.729098+0000", 2024-03-19T11:30:08.768 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00085901300000000005 2024-03-19T11:30:08.769 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.769 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.769 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:08.769 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.729105+0000", 2024-03-19T11:30:08.769 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1690000000000003e-06 2024-03-19T11:30:08.769 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.769 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.769 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:08.769 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.729240+0000", 2024-03-19T11:30:08.769 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000135265 2024-03-19T11:30:08.769 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.769 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:08.769 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.770 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.770 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.770 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop_reply(client.4415.0:14 4.4 e28/21 ondisk, result = 0)", 2024-03-19T11:30:08.770 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.728985+0000", 2024-03-19T11:30:08.770 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.997534828, 2024-03-19T11:30:08.770 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00042438000000000003, 2024-03-19T11:30:08.770 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:08.770 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:08.770 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:08.770 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.770 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:08.770 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.728985+0000", 2024-03-19T11:30:08.771 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.771 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.771 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.771 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:08.771 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.728985+0000", 2024-03-19T11:30:08.771 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.771 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.771 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.771 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:08.771 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.728986+0000", 2024-03-19T11:30:08.771 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2900000000000004e-07 2024-03-19T11:30:08.771 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.772 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.772 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:08.772 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.728990+0000", 2024-03-19T11:30:08.772 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4139999999999996e-06 2024-03-19T11:30:08.772 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.772 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.772 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:08.772 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.729000+0000", 2024-03-19T11:30:08.772 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6160000000000008e-06 2024-03-19T11:30:08.772 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.772 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.772 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:08.772 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.729007+0000", 2024-03-19T11:30:08.773 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.686e-06 2024-03-19T11:30:08.773 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.773 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.773 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:08.773 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.729326+0000", 2024-03-19T11:30:08.773 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00031969300000000002 2024-03-19T11:30:08.773 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.773 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.773 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:08.773 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.729330+0000", 2024-03-19T11:30:08.773 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0960000000000003e-06 2024-03-19T11:30:08.773 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.774 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.774 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:08.774 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.729410+0000", 2024-03-19T11:30:08.774 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9345999999999996e-05 2024-03-19T11:30:08.774 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.774 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:08.774 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.774 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.774 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.774 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop_reply(client.4415.0:15 4.4 e28/21 ondisk, result = 0)", 2024-03-19T11:30:08.774 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.730019+0000", 2024-03-19T11:30:08.774 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.9965010400000001, 2024-03-19T11:30:08.774 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000214422, 2024-03-19T11:30:08.775 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:08.775 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:08.775 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:08.775 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.775 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:08.775 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.730019+0000", 2024-03-19T11:30:08.775 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.775 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.775 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.775 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:08.775 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.730019+0000", 2024-03-19T11:30:08.775 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.776 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.776 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.776 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:08.776 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.730020+0000", 2024-03-19T11:30:08.776 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1230000000000001e-06 2024-03-19T11:30:08.776 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.776 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.776 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:08.776 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.730024+0000", 2024-03-19T11:30:08.776 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0080000000000004e-06 2024-03-19T11:30:08.776 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.776 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.777 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:08.777 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.730036+0000", 2024-03-19T11:30:08.777 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1783999999999999e-05 2024-03-19T11:30:08.777 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.777 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.777 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:08.777 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.730043+0000", 2024-03-19T11:30:08.777 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3679999999999996e-06 2024-03-19T11:30:08.777 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.777 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.777 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:08.777 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.730092+0000", 2024-03-19T11:30:08.777 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8757000000000001e-05 2024-03-19T11:30:08.778 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.778 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.778 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:08.778 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.730100+0000", 2024-03-19T11:30:08.778 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4429999999999997e-06 2024-03-19T11:30:08.778 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.778 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.778 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:08.778 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.730234+0000", 2024-03-19T11:30:08.778 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013393899999999999 2024-03-19T11:30:08.778 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.778 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:08.779 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.779 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.779 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.779 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:17 4.9 e28/21 4:9ef41243:::smithi03829856-5:head v 28'1, mlcod=0'0)", 2024-03-19T11:30:08.779 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.736136+0000", 2024-03-19T11:30:08.779 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.9903840270000002, 2024-03-19T11:30:08.779 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0039361250000000004, 2024-03-19T11:30:08.779 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:08.779 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:08.779 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:08.779 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.779 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:08.779 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.736136+0000", 2024-03-19T11:30:08.780 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.780 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.780 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.780 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:08.780 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.736136+0000", 2024-03-19T11:30:08.780 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.780 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.780 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.780 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:08.780 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.736136+0000", 2024-03-19T11:30:08.780 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6899999999999999e-07 2024-03-19T11:30:08.780 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.780 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.781 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:08.781 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.737146+0000", 2024-03-19T11:30:08.781 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001010171 2024-03-19T11:30:08.781 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.781 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.781 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:08.781 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.737160+0000", 2024-03-19T11:30:08.781 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3998e-05 2024-03-19T11:30:08.781 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.781 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.781 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:08.781 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.737173+0000", 2024-03-19T11:30:08.782 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2865e-05 2024-03-19T11:30:08.782 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.782 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.782 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:08.782 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.737281+0000", 2024-03-19T11:30:08.782 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000108136 2024-03-19T11:30:08.782 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.782 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.782 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:08.782 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.737324+0000", 2024-03-19T11:30:08.782 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2821000000000002e-05 2024-03-19T11:30:08.782 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.783 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.783 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:08.783 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.740021+0000", 2024-03-19T11:30:08.783 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0026964609999999998 2024-03-19T11:30:08.783 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.783 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.783 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:08.783 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.740072+0000", 2024-03-19T11:30:08.783 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1504e-05 2024-03-19T11:30:08.783 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.783 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:08.783 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.783 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.784 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.784 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:18 4.9 e28/21 4:9ef41243:::smithi03829856-5:head v 28'2, mlcod=0'0)", 2024-03-19T11:30:08.784 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.740772+0000", 2024-03-19T11:30:08.784 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.9857479840000001, 2024-03-19T11:30:08.784 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00224741, 2024-03-19T11:30:08.784 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:08.784 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:08.784 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:08.784 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.784 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:08.784 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.740772+0000", 2024-03-19T11:30:08.784 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.784 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.785 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.785 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:08.785 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.740772+0000", 2024-03-19T11:30:08.785 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.785 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.785 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.785 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:08.785 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.740773+0000", 2024-03-19T11:30:08.785 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7400000000000003e-07 2024-03-19T11:30:08.785 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.785 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.786 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:08.786 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.741152+0000", 2024-03-19T11:30:08.786 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037904300000000002 2024-03-19T11:30:08.786 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.786 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.786 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:08.786 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.741164+0000", 2024-03-19T11:30:08.786 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2063e-05 2024-03-19T11:30:08.786 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.786 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.786 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:08.787 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.741175+0000", 2024-03-19T11:30:08.787 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0584000000000001e-05 2024-03-19T11:30:08.787 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.787 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.787 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:08.787 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.741236+0000", 2024-03-19T11:30:08.787 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1768999999999998e-05 2024-03-19T11:30:08.787 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.787 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.787 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:08.787 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.741262+0000", 2024-03-19T11:30:08.788 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5561000000000001e-05 2024-03-19T11:30:08.788 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.788 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.788 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:08.788 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.742953+0000", 2024-03-19T11:30:08.788 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0016912839999999999 2024-03-19T11:30:08.788 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.788 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.788 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:08.788 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.743020+0000", 2024-03-19T11:30:08.789 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6432000000000004e-05 2024-03-19T11:30:08.789 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.789 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:08.789 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.789 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.789 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.789 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:19 4.9 e28/21 4:9ef41243:::smithi03829856-5:head v 28'3, mlcod=0'0)", 2024-03-19T11:30:08.789 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.743669+0000", 2024-03-19T11:30:08.789 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.9828508029999998, 2024-03-19T11:30:08.789 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0026100580000000002, 2024-03-19T11:30:08.789 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:08.789 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:08.790 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:08.790 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.790 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:08.790 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.743669+0000", 2024-03-19T11:30:08.790 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.790 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.790 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.791 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:08.791 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.743669+0000", 2024-03-19T11:30:08.791 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.791 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.791 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.791 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:08.791 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.743671+0000", 2024-03-19T11:30:08.791 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3990000000000001e-06 2024-03-19T11:30:08.791 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.791 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.791 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:08.791 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.743680+0000", 2024-03-19T11:30:08.792 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2750000000000001e-06 2024-03-19T11:30:08.792 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.792 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.792 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:08.792 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.743695+0000", 2024-03-19T11:30:08.792 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5415999999999999e-05 2024-03-19T11:30:08.792 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.792 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.792 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:08.792 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.743707+0000", 2024-03-19T11:30:08.792 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1447000000000001e-05 2024-03-19T11:30:08.792 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.793 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.793 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:08.793 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.743768+0000", 2024-03-19T11:30:08.793 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1393e-05 2024-03-19T11:30:08.793 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.793 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.793 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:08.793 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.743795+0000", 2024-03-19T11:30:08.793 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.7002e-05 2024-03-19T11:30:08.794 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.794 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.794 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:08.794 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.746234+0000", 2024-03-19T11:30:08.794 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0024388000000000001 2024-03-19T11:30:08.794 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.794 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.794 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:08.794 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.746279+0000", 2024-03-19T11:30:08.794 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5325999999999999e-05 2024-03-19T11:30:08.794 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.795 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:08.795 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.795 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.795 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.795 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:21 4.b e28/21 4:db322469:::smithi03829856-6:head v 28'1, mlcod=0'0)", 2024-03-19T11:30:08.795 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.749685+0000", 2024-03-19T11:30:08.795 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.9768351499999999, 2024-03-19T11:30:08.795 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034416389999999998, 2024-03-19T11:30:08.795 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:08.796 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:08.796 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:08.796 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.796 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:08.796 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.749685+0000", 2024-03-19T11:30:08.796 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.796 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.796 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.796 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:08.797 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.749685+0000", 2024-03-19T11:30:08.797 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.797 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.797 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.797 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:08.797 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.749685+0000", 2024-03-19T11:30:08.797 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9200000000000002e-07 2024-03-19T11:30:08.797 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.797 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.797 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:08.798 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.750142+0000", 2024-03-19T11:30:08.798 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00045645899999999999 2024-03-19T11:30:08.798 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.798 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.798 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:08.798 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.750162+0000", 2024-03-19T11:30:08.798 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0564000000000002e-05 2024-03-19T11:30:08.798 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.798 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.799 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:08.799 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.750173+0000", 2024-03-19T11:30:08.799 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0373999999999999e-05 2024-03-19T11:30:08.799 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.799 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.799 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:08.799 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.750260+0000", 2024-03-19T11:30:08.799 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7427000000000002e-05 2024-03-19T11:30:08.799 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.799 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.799 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:08.800 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.750284+0000", 2024-03-19T11:30:08.800 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4114999999999999e-05 2024-03-19T11:30:08.800 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.800 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.800 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:08.800 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.753081+0000", 2024-03-19T11:30:08.800 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0027968559999999999 2024-03-19T11:30:08.801 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.801 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.801 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:08.801 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.753127+0000", 2024-03-19T11:30:08.801 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5552e-05 2024-03-19T11:30:08.801 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.801 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:08.802 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.802 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.802 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.802 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:22 4.b e28/21 4:db322469:::smithi03829856-6:head v 28'2, mlcod=0'0)", 2024-03-19T11:30:08.802 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.752663+0000", 2024-03-19T11:30:08.802 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.9738566479999999, 2024-03-19T11:30:08.802 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00123269, 2024-03-19T11:30:08.802 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:08.803 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:08.803 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:08.803 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.803 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:08.803 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.752663+0000", 2024-03-19T11:30:08.803 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.803 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.803 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.803 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:08.803 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.752663+0000", 2024-03-19T11:30:08.804 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.804 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.804 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.804 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:08.804 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.752665+0000", 2024-03-19T11:30:08.804 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0279999999999999e-06 2024-03-19T11:30:08.804 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.804 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.804 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:08.804 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.752682+0000", 2024-03-19T11:30:08.804 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7235e-05 2024-03-19T11:30:08.804 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.804 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.805 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:08.805 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.752694+0000", 2024-03-19T11:30:08.805 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1800999999999999e-05 2024-03-19T11:30:08.805 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.805 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.805 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:08.805 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.752701+0000", 2024-03-19T11:30:08.805 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7610000000000007e-06 2024-03-19T11:30:08.805 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.805 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.805 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:08.805 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.752769+0000", 2024-03-19T11:30:08.806 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8137000000000002e-05 2024-03-19T11:30:08.806 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.806 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.806 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:08.806 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.752793+0000", 2024-03-19T11:30:08.806 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3892000000000001e-05 2024-03-19T11:30:08.806 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.806 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.806 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:08.806 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.753835+0000", 2024-03-19T11:30:08.806 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001042056 2024-03-19T11:30:08.806 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.807 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.807 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:08.807 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.753896+0000", 2024-03-19T11:30:08.807 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0779999999999997e-05 2024-03-19T11:30:08.807 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.807 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:08.807 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.807 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.807 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.807 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:23 4.b e28/21 4:db322469:::smithi03829856-6:head v 28'3, mlcod=0'0)", 2024-03-19T11:30:08.807 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.753396+0000", 2024-03-19T11:30:08.807 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.9731244829999999, 2024-03-19T11:30:08.808 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00074388600000000003, 2024-03-19T11:30:08.808 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:08.808 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:08.808 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:08.808 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.808 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:08.808 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.753396+0000", 2024-03-19T11:30:08.808 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.808 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.808 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.808 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:08.808 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.753396+0000", 2024-03-19T11:30:08.808 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.809 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.809 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.809 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:08.809 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.753396+0000", 2024-03-19T11:30:08.809 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.13e-07 2024-03-19T11:30:08.809 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.809 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.809 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:08.809 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.753402+0000", 2024-03-19T11:30:08.809 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8279999999999999e-06 2024-03-19T11:30:08.810 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.810 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.810 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:08.810 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.753413+0000", 2024-03-19T11:30:08.810 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0569e-05 2024-03-19T11:30:08.810 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.810 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.810 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:08.810 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.753420+0000", 2024-03-19T11:30:08.810 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4440000000000002e-06 2024-03-19T11:30:08.810 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.810 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.811 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:08.811 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.753469+0000", 2024-03-19T11:30:08.811 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8974000000000002e-05 2024-03-19T11:30:08.811 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.811 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.811 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:08.811 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.753487+0000", 2024-03-19T11:30:08.811 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8264999999999999e-05 2024-03-19T11:30:08.811 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.811 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.811 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:08.811 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.754102+0000", 2024-03-19T11:30:08.811 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00061429300000000002 2024-03-19T11:30:08.812 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.812 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.812 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:08.812 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.754140+0000", 2024-03-19T11:30:08.812 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.79e-05 2024-03-19T11:30:08.812 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.812 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:08.812 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.812 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.812 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.812 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:37 4.6 e28/21 4:6868af91:::smithi03829856-10:head v 28'4, mlcod=27'2)", 2024-03-19T11:30:08.813 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.798395+0000", 2024-03-19T11:30:08.813 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.9281247769999998, 2024-03-19T11:30:08.813 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002825036, 2024-03-19T11:30:08.813 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:08.813 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:08.813 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:08.813 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.813 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:08.813 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.798395+0000", 2024-03-19T11:30:08.813 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.813 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.814 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.814 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:08.814 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.798395+0000", 2024-03-19T11:30:08.814 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:08.814 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.814 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.814 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:08.814 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.798396+0000", 2024-03-19T11:30:08.814 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.091e-06 2024-03-19T11:30:08.814 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.814 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.815 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:08.815 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.798791+0000", 2024-03-19T11:30:08.815 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00039492299999999999 2024-03-19T11:30:08.815 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.815 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.815 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:08.815 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.798804+0000", 2024-03-19T11:30:08.815 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2573e-05 2024-03-19T11:30:08.815 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.815 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.815 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:08.815 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.798813+0000", 2024-03-19T11:30:08.815 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1609999999999992e-06 2024-03-19T11:30:08.816 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.816 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.816 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:08.816 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.798888+0000", 2024-03-19T11:30:08.816 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4419999999999996e-05 2024-03-19T11:30:08.816 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.816 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.816 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:08.816 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.798913+0000", 2024-03-19T11:30:08.816 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5554999999999998e-05 2024-03-19T11:30:08.816 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.816 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.816 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:08.817 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.801165+0000", 2024-03-19T11:30:08.817 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022522409999999999 2024-03-19T11:30:08.817 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:08.817 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:08.817 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:08.817 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.801220+0000", 2024-03-19T11:30:08.817 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5071999999999997e-05 2024-03-19T11:30:08.817 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.817 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:08.817 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.817 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:08.818 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:08.818 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:08.819 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:08.731+0000 7f1837407640 -1 osd.6 31 handle_osd_map injecting map crc failure 2024-03-19T11:30:08.819 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-03-19T11:30:08.819 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:08.724+0000 7f6a7f9da640 -1 osd.1 31 handle_osd_map injecting map crc failure 2024-03-19T11:30:08.820 INFO:tasks.rados.rados.0.smithi038.stdout:635: done (1 left) 2024-03-19T11:30:08.820 INFO:tasks.rados.rados.0.smithi038.stdout:636: write oid 14 current snap is 3 2024-03-19T11:30:08.820 INFO:tasks.rados.rados.0.smithi038.stdout:636: seq_num 58 ranges {574806=778913,2061132=557342} 2024-03-19T11:30:08.820 INFO:tasks.rados.rados.0.smithi038.stdout:636: writing smithi03829856-14 from 574806 to 1353719 tid 1 2024-03-19T11:30:08.821 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-19T11:30:08.822 INFO:tasks.rados.rados.0.smithi038.stdout:636: writing smithi03829856-14 from 2061132 to 2618474 tid 2 2024-03-19T11:30:08.822 INFO:tasks.rados.rados.0.smithi038.stdout:637: read oid 31 snap -1 2024-03-19T11:30:08.822 INFO:tasks.rados.rados.0.smithi038.stdout:637: expect (ObjNum 56 snap 1 seq_num 56) 2024-03-19T11:30:08.823 INFO:tasks.rados.rados.0.smithi038.stdout:638: write (excl) oid 45 current snap is 3 2024-03-19T11:30:08.823 INFO:tasks.rados.rados.0.smithi038.stdout:638: seq_num 59 ranges {591613=454162,1719614=417353,2570780=420833} 2024-03-19T11:30:08.824 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:08.827+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:08.825 INFO:tasks.rados.rados.0.smithi038.stdout:638: writing smithi03829856-45 from 591613 to 1045775 tid 1 2024-03-19T11:30:08.825 INFO:tasks.rados.rados.0.smithi038.stdout:638: writing smithi03829856-45 from 1719614 to 2136967 tid 2 2024-03-19T11:30:08.825 INFO:tasks.rados.rados.0.smithi038.stdout:638: writing smithi03829856-45 from 2570780 to 2991613 tid 3 2024-03-19T11:30:08.826 INFO:tasks.rados.rados.0.smithi038.stdout:639: write oid 8 current snap is 3 2024-03-19T11:30:08.826 INFO:tasks.rados.rados.0.smithi038.stdout:639: seq_num 60 ranges {608420=529411,1778096=561011,2881850=126570} 2024-03-19T11:30:08.826 INFO:tasks.rados.rados.0.smithi038.stdout:639: writing smithi03829856-8 from 608420 to 1137831 tid 1 2024-03-19T11:30:08.826 INFO:tasks.rados.rados.0.smithi038.stdout:639: writing smithi03829856-8 from 1778096 to 2339107 tid 2 2024-03-19T11:30:08.826 INFO:tasks.rados.rados.0.smithi038.stdout:639: writing smithi03829856-8 from 2881850 to 3008420 tid 3 2024-03-19T11:30:08.826 INFO:tasks.rados.rados.0.smithi038.stdout:640: delete oid 48 current snap is 3 2024-03-19T11:30:08.826 INFO:tasks.rados.rados.0.smithi038.stdout:634: expect (ObjNum 29 snap 0 seq_num 29) 2024-03-19T11:30:08.858 INFO:tasks.rados.rados.0.smithi038.stdout:636: finishing write tid 1 to smithi03829856-14 2024-03-19T11:30:08.858 INFO:tasks.rados.rados.0.smithi038.stdout:636: finishing write tid 2 to smithi03829856-14 2024-03-19T11:30:08.858 INFO:tasks.rados.rados.0.smithi038.stdout:636: finishing write tid 3 to smithi03829856-14 2024-03-19T11:30:08.858 INFO:tasks.rados.rados.0.smithi038.stdout:636: finishing write tid 4 to smithi03829856-14 2024-03-19T11:30:08.858 INFO:tasks.rados.rados.0.smithi038.stdout:636: oid 14 updating version 0 to 10 2024-03-19T11:30:08.859 INFO:tasks.rados.rados.0.smithi038.stdout:636: oid 14 version 10 is already newer than 8 2024-03-19T11:30:08.859 INFO:tasks.rados.rados.0.smithi038.stdout:636: oid 14 version 10 is already newer than 9 2024-03-19T11:30:08.859 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 14 v 10 (ObjNum 58 snap 3 seq_num 58) dirty exists 2024-03-19T11:30:08.859 INFO:tasks.rados.rados.0.smithi038.stdout:636: left oid 14 (ObjNum 58 snap 3 seq_num 58) 2024-03-19T11:30:08.859 INFO:tasks.rados.rados.0.smithi038.stdout:637: expect (ObjNum 56 snap 1 seq_num 56) 2024-03-19T11:30:08.920 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:08.931+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:08.980 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:08.981 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:30:08.981 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:30:08.981 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:08.995 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-19T11:30:08.996 INFO:tasks.rados.rados.0.smithi038.stdout:638: finishing write tid 1 to smithi03829856-45 2024-03-19T11:30:08.996 INFO:tasks.rados.rados.0.smithi038.stdout:638: finishing write tid 2 to smithi03829856-45 2024-03-19T11:30:08.996 INFO:tasks.rados.rados.0.smithi038.stdout:639: finishing write tid 1 to smithi03829856-8 2024-03-19T11:30:08.997 INFO:tasks.rados.rados.0.smithi038.stdout:638: finishing write tid 3 to smithi03829856-45 2024-03-19T11:30:08.997 INFO:tasks.rados.rados.0.smithi038.stdout:638: finishing write tid 4 to smithi03829856-45 2024-03-19T11:30:08.997 INFO:tasks.rados.rados.0.smithi038.stdout:634: done (5 left) 2024-03-19T11:30:08.997 INFO:tasks.rados.rados.0.smithi038.stdout:636: done (4 left) 2024-03-19T11:30:08.997 INFO:tasks.rados.rados.0.smithi038.stdout:637: done (3 left) 2024-03-19T11:30:08.997 INFO:tasks.rados.rados.0.smithi038.stdout:640: done (2 left) 2024-03-19T11:30:08.997 INFO:tasks.rados.rados.0.smithi038.stdout:641: read oid 35 snap -1 2024-03-19T11:30:08.998 INFO:tasks.rados.rados.0.smithi038.stdout:641: expect (ObjNum 34 snap 0 seq_num 34) 2024-03-19T11:30:08.998 INFO:tasks.rados.rados.0.smithi038.stdout:642: set_chunk oid 12 target oid 2024-03-19T11:30:09.005 INFO:tasks.rados.rados.0.smithi038.stdout:642: set_chunk oid 12 offset: 1011712 length: 462848 target oid 626f73f06d73fccc02b43398389ff991d56d81381f5e170c36fb9439a43cbbc0 offset: 0 snap -1 2024-03-19T11:30:09.005 INFO:tasks.rados.rados.0.smithi038.stdout:643: read oid 31 snap -1 2024-03-19T11:30:09.005 INFO:tasks.rados.rados.0.smithi038.stdout:643: expect (ObjNum 56 snap 1 seq_num 56) 2024-03-19T11:30:09.005 INFO:tasks.rados.rados.0.smithi038.stdout:644: write (excl) oid 3 current snap is 3 2024-03-19T11:30:09.005 INFO:tasks.rados.rados.0.smithi038.stdout:638: finishing write tid 5 to smithi03829856-45 2024-03-19T11:30:09.006 INFO:tasks.rados.rados.0.smithi038.stdout:638: oid 45 updating version 0 to 30 2024-03-19T11:30:09.006 INFO:tasks.rados.rados.0.smithi038.stdout:638: oid 45 updating version 30 to 31 2024-03-19T11:30:09.006 INFO:tasks.rados.rados.0.smithi038.stdout:638: oid 45 updating version 31 to 33 2024-03-19T11:30:09.006 INFO:tasks.rados.rados.0.smithi038.stdout:638: oid 45 version 33 is already newer than 32 2024-03-19T11:30:09.006 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 45 v 33 (ObjNum 59 snap 3 seq_num 59) dirty exists 2024-03-19T11:30:09.006 INFO:tasks.rados.rados.0.smithi038.stdout:638: left oid 45 (ObjNum 59 snap 3 seq_num 59) 2024-03-19T11:30:09.006 INFO:tasks.rados.rados.0.smithi038.stdout:639: finishing write tid 2 to smithi03829856-8 2024-03-19T11:30:09.006 INFO:tasks.rados.rados.0.smithi038.stdout:644: seq_num 62 ranges {642034=796262,1895060=564680} 2024-03-19T11:30:09.012 INFO:tasks.rados.rados.0.smithi038.stdout:644: writing smithi03829856-3 from 642034 to 1438296 tid 1 2024-03-19T11:30:09.019 INFO:tasks.rados.rados.0.smithi038.stdout:644: writing smithi03829856-3 from 1895060 to 2459740 tid 2 2024-03-19T11:30:09.019 INFO:tasks.rados.rados.0.smithi038.stdout:638: done (5 left) 2024-03-19T11:30:09.019 INFO:tasks.rados.rados.0.smithi038.stdout:645: write oid 34 current snap is 3 2024-03-19T11:30:09.020 INFO:tasks.rados.rados.0.smithi038.stdout:639: finishing write tid 3 to smithi03829856-8 2024-03-19T11:30:09.020 INFO:tasks.rados.rados.0.smithi038.stdout:639: finishing write tid 4 to smithi03829856-8 2024-03-19T11:30:09.020 INFO:tasks.rados.rados.0.smithi038.stdout:639: finishing write tid 5 to smithi03829856-8 2024-03-19T11:30:09.020 INFO:tasks.rados.rados.0.smithi038.stdout:639: oid 8 updating version 0 to 48 2024-03-19T11:30:09.020 INFO:tasks.rados.rados.0.smithi038.stdout:639: oid 8 updating version 48 to 50 2024-03-19T11:30:09.020 INFO:tasks.rados.rados.0.smithi038.stdout:639: oid 8 updating version 50 to 51 2024-03-19T11:30:09.020 INFO:tasks.rados.rados.0.smithi038.stdout:639: oid 8 version 51 is already newer than 49 2024-03-19T11:30:09.020 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 8 v 51 (ObjNum 60 snap 3 seq_num 60) dirty exists 2024-03-19T11:30:09.020 INFO:tasks.rados.rados.0.smithi038.stdout:639: left oid 8 (ObjNum 60 snap 3 seq_num 60) 2024-03-19T11:30:09.020 INFO:tasks.rados.rados.0.smithi038.stdout:645: seq_num 63 ranges {658841=471511,1553542=708338,2964119=94722} 2024-03-19T11:30:09.022 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:09.031+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:09.025 INFO:tasks.rados.rados.0.smithi038.stdout:645: writing smithi03829856-34 from 658841 to 1130352 tid 1 2024-03-19T11:30:09.032 INFO:tasks.rados.rados.0.smithi038.stdout:645: writing smithi03829856-34 from 1553542 to 2261880 tid 2 2024-03-19T11:30:09.037 INFO:tasks.rados.rados.0.smithi038.stdout:645: writing smithi03829856-34 from 2964119 to 3058841 tid 3 2024-03-19T11:30:09.037 INFO:tasks.rados.rados.0.smithi038.stdout:639: done (5 left) 2024-03-19T11:30:09.037 INFO:tasks.rados.rados.0.smithi038.stdout:646: tier_evict oid 39 2024-03-19T11:30:09.037 INFO:tasks.rados.rados.0.smithi038.stdout:646: tier_evict oid 39 snap -1 2024-03-19T11:30:09.037 INFO:tasks.rados.rados.0.smithi038.stdout:641: expect (ObjNum 34 snap 0 seq_num 34) 2024-03-19T11:30:09.119 INFO:tasks.rados.rados.0.smithi038.stdout:642: finishing set_chunk to oid 12 2024-03-19T11:30:09.119 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 12 set_chunk returned error code -95 offset: 1011712 length: 462848 2024-03-19T11:30:09.119 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 12 v 0 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-03-19T11:30:09.119 INFO:tasks.rados.rados.0.smithi038.stdout:643: expect (ObjNum 56 snap 1 seq_num 56) 2024-03-19T11:30:09.123 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:09.131+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:09.130 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:09.130 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:30:09.130 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:30:09.130 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:30:09.130 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:09.144 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-19T11:30:09.224 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:09.235+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:09.255 INFO:tasks.rados.rados.0.smithi038.stdout:644: finishing write tid 1 to smithi03829856-3 2024-03-19T11:30:09.255 INFO:tasks.rados.rados.0.smithi038.stdout:644: finishing write tid 2 to smithi03829856-3 2024-03-19T11:30:09.255 INFO:tasks.rados.rados.0.smithi038.stdout:644: finishing write tid 3 to smithi03829856-3 2024-03-19T11:30:09.256 INFO:tasks.rados.rados.0.smithi038.stdout:644: finishing write tid 4 to smithi03829856-3 2024-03-19T11:30:09.256 INFO:tasks.rados.rados.0.smithi038.stdout:644: oid 3 updating version 0 to 19 2024-03-19T11:30:09.256 INFO:tasks.rados.rados.0.smithi038.stdout:644: oid 3 updating version 19 to 21 2024-03-19T11:30:09.256 INFO:tasks.rados.rados.0.smithi038.stdout:644: oid 3 version 21 is already newer than 20 2024-03-19T11:30:09.256 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 3 v 21 (ObjNum 62 snap 3 seq_num 62) dirty exists 2024-03-19T11:30:09.256 INFO:tasks.rados.rados.0.smithi038.stdout:644: left oid 3 (ObjNum 62 snap 3 seq_num 62) 2024-03-19T11:30:09.256 INFO:tasks.rados.rados.0.smithi038.stdout:645: finishing write tid 1 to smithi03829856-34 2024-03-19T11:30:09.256 INFO:tasks.rados.rados.0.smithi038.stdout:646: got (22) Invalid argument 2024-03-19T11:30:09.256 INFO:tasks.rados.rados.0.smithi038.stdout:645: finishing write tid 2 to smithi03829856-34 2024-03-19T11:30:09.256 INFO:tasks.rados.rados.0.smithi038.stdout:645: finishing write tid 3 to smithi03829856-34 2024-03-19T11:30:09.256 INFO:tasks.rados.rados.0.smithi038.stdout:645: finishing write tid 4 to smithi03829856-34 2024-03-19T11:30:09.256 INFO:tasks.rados.rados.0.smithi038.stdout:645: finishing write tid 5 to smithi03829856-34 2024-03-19T11:30:09.256 INFO:tasks.rados.rados.0.smithi038.stdout:645: oid 34 updating version 0 to 9 2024-03-19T11:30:09.256 INFO:tasks.rados.rados.0.smithi038.stdout:645: oid 34 updating version 9 to 11 2024-03-19T11:30:09.256 INFO:tasks.rados.rados.0.smithi038.stdout:645: oid 34 updating version 11 to 12 2024-03-19T11:30:09.257 INFO:tasks.rados.rados.0.smithi038.stdout:645: oid 34 version 12 is already newer than 10 2024-03-19T11:30:09.257 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 34 v 12 (ObjNum 63 snap 3 seq_num 63) dirty exists 2024-03-19T11:30:09.257 INFO:tasks.rados.rados.0.smithi038.stdout:645: left oid 34 (ObjNum 63 snap 3 seq_num 63) 2024-03-19T11:30:09.257 INFO:tasks.rados.rados.0.smithi038.stdout:641: done (5 left) 2024-03-19T11:30:09.257 INFO:tasks.rados.rados.0.smithi038.stdout:642: done (4 left) 2024-03-19T11:30:09.257 INFO:tasks.rados.rados.0.smithi038.stdout:643: done (3 left) 2024-03-19T11:30:09.257 INFO:tasks.rados.rados.0.smithi038.stdout:644: done (2 left) 2024-03-19T11:30:09.257 INFO:tasks.rados.rados.0.smithi038.stdout:645: done (1 left) 2024-03-19T11:30:09.257 INFO:tasks.rados.rados.0.smithi038.stdout:646: done (0 left) 2024-03-19T11:30:09.257 INFO:tasks.rados.rados.0.smithi038.stdout:647: read oid 30 snap -1 2024-03-19T11:30:09.257 INFO:tasks.rados.rados.0.smithi038.stdout:647: expect (ObjNum 29 snap 0 seq_num 29) 2024-03-19T11:30:09.257 INFO:tasks.rados.rados.0.smithi038.stdout:648: read oid 21 snap -1 2024-03-19T11:30:09.257 INFO:tasks.rados.rados.0.smithi038.stdout:648: expect (ObjNum 20 snap 0 seq_num 20) 2024-03-19T11:30:09.257 INFO:tasks.rados.rados.0.smithi038.stdout:649: tier_evict oid 40 2024-03-19T11:30:09.257 INFO:tasks.rados.rados.0.smithi038.stdout:649: tier_evict oid 40 snap -1 2024-03-19T11:30:09.258 INFO:tasks.rados.rados.0.smithi038.stdout:650: set_chunk oid 49 target oid 2024-03-19T11:30:09.268 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-19T11:30:09.271 INFO:tasks.rados.rados.0.smithi038.stdout:650: set_chunk oid 49 offset: 1284608 length: 781312 target oid 895a63407b6109b3cca9a6f5ef827134d56679bef42daadc5892269682c428ed offset: 0 snap -1 2024-03-19T11:30:09.271 INFO:tasks.rados.rados.0.smithi038.stdout:651: tier_flush oid 20 2024-03-19T11:30:09.271 INFO:tasks.rados.rados.0.smithi038.stdout:651: tier_flush oid 20 snap -1 2024-03-19T11:30:09.271 INFO:tasks.rados.rados.0.smithi038.stdout:649: got (22) Invalid argument 2024-03-19T11:30:09.271 INFO:tasks.rados.rados.0.smithi038.stdout:649: done (4 left) 2024-03-19T11:30:09.272 INFO:tasks.rados.rados.0.smithi038.stdout:652: set_chunk oid 29 target oid 2024-03-19T11:30:09.279 INFO:tasks.rados.rados.0.smithi038.stdout:652: set_chunk oid 29 offset: 557056 length: 448000 target oid c8c8170dc54fef3b4e25ae28106dacc8accf0657c6d6457bff8aa63e56c9678d offset: 0 snap -1 2024-03-19T11:30:09.280 INFO:tasks.rados.rados.0.smithi038.stdout:647: expect (ObjNum 29 snap 0 seq_num 29) 2024-03-19T11:30:09.317 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:09.317 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:30:09.317 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:30:09.317 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:30:09.317 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.317 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:9 4.d e28/21 4:b4ef97e0:::smithi03829856-3:head v 28'1, mlcod=0'0)", 2024-03-19T11:30:09.317 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.707728+0000", 2024-03-19T11:30:09.317 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.622224245, 2024-03-19T11:30:09.318 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0037487950000000001, 2024-03-19T11:30:09.318 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:09.318 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:09.318 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:09.318 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.318 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:09.318 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.707728+0000", 2024-03-19T11:30:09.318 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:09.318 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.318 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.318 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:09.318 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.707728+0000", 2024-03-19T11:30:09.318 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:09.319 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.319 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.319 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:09.319 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.707729+0000", 2024-03-19T11:30:09.319 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6899999999999998e-07 2024-03-19T11:30:09.319 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.319 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.319 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:09.319 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.708625+0000", 2024-03-19T11:30:09.319 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00089603899999999995 2024-03-19T11:30:09.320 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.320 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.320 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:09.320 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.708645+0000", 2024-03-19T11:30:09.320 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9629000000000001e-05 2024-03-19T11:30:09.320 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.320 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.320 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:09.320 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.708657+0000", 2024-03-19T11:30:09.320 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2537999999999999e-05 2024-03-19T11:30:09.320 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.320 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.320 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:09.320 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.708740+0000", 2024-03-19T11:30:09.321 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2490999999999993e-05 2024-03-19T11:30:09.321 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.321 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.321 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:09.321 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.708774+0000", 2024-03-19T11:30:09.321 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4362000000000001e-05 2024-03-19T11:30:09.321 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.321 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.321 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:09.321 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.711419+0000", 2024-03-19T11:30:09.322 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0026446830000000001 2024-03-19T11:30:09.322 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.322 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.322 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:09.322 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.711477+0000", 2024-03-19T11:30:09.322 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8684000000000003e-05 2024-03-19T11:30:09.322 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:09.322 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:09.322 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:09.322 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.322 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.322 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:10 4.d e28/21 4:b4ef97e0:::smithi03829856-3:head v 28'2, mlcod=0'0)", 2024-03-19T11:30:09.323 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.713599+0000", 2024-03-19T11:30:09.323 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.6163537869999995, 2024-03-19T11:30:09.323 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002599905, 2024-03-19T11:30:09.323 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:09.323 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:09.323 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:09.323 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.323 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:09.323 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.713599+0000", 2024-03-19T11:30:09.323 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:09.323 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.324 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.324 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:09.324 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.713599+0000", 2024-03-19T11:30:09.324 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:09.324 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.324 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.324 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:09.324 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.713619+0000", 2024-03-19T11:30:09.324 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0398999999999999e-05 2024-03-19T11:30:09.324 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.324 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.324 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:09.324 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.713955+0000", 2024-03-19T11:30:09.324 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000335691 2024-03-19T11:30:09.324 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.325 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.325 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:09.325 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.713976+0000", 2024-03-19T11:30:09.325 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0866999999999999e-05 2024-03-19T11:30:09.325 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.325 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.325 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:09.325 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.713988+0000", 2024-03-19T11:30:09.325 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2228e-05 2024-03-19T11:30:09.325 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.325 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.325 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:09.325 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.714058+0000", 2024-03-19T11:30:09.325 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9868000000000003e-05 2024-03-19T11:30:09.325 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.326 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.326 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:09.326 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.714086+0000", 2024-03-19T11:30:09.326 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8124999999999999e-05 2024-03-19T11:30:09.326 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.326 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.326 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:09.326 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.716138+0000", 2024-03-19T11:30:09.326 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0020517759999999999 2024-03-19T11:30:09.326 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.326 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.326 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:09.326 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.716199+0000", 2024-03-19T11:30:09.326 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0951e-05 2024-03-19T11:30:09.327 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:09.327 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:09.327 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:09.327 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.327 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.327 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:11 4.d e28/21 4:b4ef97e0:::smithi03829856-3:head v 28'3, mlcod=0'0)", 2024-03-19T11:30:09.327 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.717405+0000", 2024-03-19T11:30:09.327 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.6125473909999997, 2024-03-19T11:30:09.327 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00079985699999999996, 2024-03-19T11:30:09.327 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:09.328 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:09.328 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:09.328 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.328 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:09.328 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.717405+0000", 2024-03-19T11:30:09.328 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:09.328 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.328 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.328 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:09.328 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.717405+0000", 2024-03-19T11:30:09.328 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:09.329 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.329 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.329 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:09.329 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.717406+0000", 2024-03-19T11:30:09.329 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0950000000000001e-06 2024-03-19T11:30:09.329 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.329 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.329 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:09.330 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.717417+0000", 2024-03-19T11:30:09.330 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0400999999999999e-05 2024-03-19T11:30:09.330 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.330 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.330 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:09.330 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.717439+0000", 2024-03-19T11:30:09.330 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2554000000000001e-05 2024-03-19T11:30:09.330 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.330 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.330 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:09.331 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.717451+0000", 2024-03-19T11:30:09.331 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1515000000000001e-05 2024-03-19T11:30:09.331 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.331 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.331 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:09.331 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.717524+0000", 2024-03-19T11:30:09.331 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2892000000000006e-05 2024-03-19T11:30:09.331 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.331 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.331 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:09.331 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.717551+0000", 2024-03-19T11:30:09.331 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.7183999999999999e-05 2024-03-19T11:30:09.331 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.332 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.332 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:09.332 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.718177+0000", 2024-03-19T11:30:09.332 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00062564100000000002 2024-03-19T11:30:09.332 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.332 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.332 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:09.332 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.718205+0000", 2024-03-19T11:30:09.332 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8575e-05 2024-03-19T11:30:09.332 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:09.332 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:09.332 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:09.332 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.332 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.333 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:13 4.4 e28/21 4:21eac9b2:::smithi03829856-4:head v 28'1, mlcod=0'0)", 2024-03-19T11:30:09.333 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.723258+0000", 2024-03-19T11:30:09.333 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.6066942500000003, 2024-03-19T11:30:09.333 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0048979840000000002, 2024-03-19T11:30:09.333 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:09.333 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:09.333 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:09.333 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.333 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:09.333 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723258+0000", 2024-03-19T11:30:09.333 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:09.333 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.333 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.333 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:09.334 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723258+0000", 2024-03-19T11:30:09.334 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:09.334 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.334 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.334 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:09.334 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723259+0000", 2024-03-19T11:30:09.334 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.7399999999999999e-07 2024-03-19T11:30:09.334 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.334 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.334 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:09.334 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723812+0000", 2024-03-19T11:30:09.334 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00055371000000000001 2024-03-19T11:30:09.335 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.335 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.335 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:09.335 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723833+0000", 2024-03-19T11:30:09.335 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0990000000000001e-05 2024-03-19T11:30:09.335 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.335 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.335 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:09.335 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723847+0000", 2024-03-19T11:30:09.335 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3774000000000001e-05 2024-03-19T11:30:09.335 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.335 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.335 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:09.335 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723912+0000", 2024-03-19T11:30:09.336 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4758000000000006e-05 2024-03-19T11:30:09.336 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.336 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.336 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:09.336 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723933+0000", 2024-03-19T11:30:09.336 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1046000000000001e-05 2024-03-19T11:30:09.336 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.336 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.336 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:09.336 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.728103+0000", 2024-03-19T11:30:09.336 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0041695079999999997 2024-03-19T11:30:09.336 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.336 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.336 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:09.336 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.728156+0000", 2024-03-19T11:30:09.337 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3924000000000001e-05 2024-03-19T11:30:09.337 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:09.337 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:09.337 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:09.337 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.337 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.337 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:14 4.4 e28/21 4:21eac9b2:::smithi03829856-4:head v 28'2, mlcod=0'0)", 2024-03-19T11:30:09.337 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.726103+0000", 2024-03-19T11:30:09.337 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.6038493999999996, 2024-03-19T11:30:09.337 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025805440000000002, 2024-03-19T11:30:09.337 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:09.337 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:09.337 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:09.337 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.337 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:09.338 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.726103+0000", 2024-03-19T11:30:09.338 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:09.338 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.338 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.338 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:09.338 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.726103+0000", 2024-03-19T11:30:09.338 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:09.338 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.338 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.338 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:09.338 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.726104+0000", 2024-03-19T11:30:09.338 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0920000000000001e-06 2024-03-19T11:30:09.338 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.338 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.339 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:09.339 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.726321+0000", 2024-03-19T11:30:09.339 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00021668499999999999 2024-03-19T11:30:09.339 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.339 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.339 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:09.339 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.726338+0000", 2024-03-19T11:30:09.339 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6382999999999999e-05 2024-03-19T11:30:09.339 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.339 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.339 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:09.339 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.726347+0000", 2024-03-19T11:30:09.339 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7399999999999999e-06 2024-03-19T11:30:09.339 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.339 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.340 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:09.340 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.726422+0000", 2024-03-19T11:30:09.340 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4365999999999999e-05 2024-03-19T11:30:09.340 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.340 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.340 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:09.340 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.726451+0000", 2024-03-19T11:30:09.340 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9085e-05 2024-03-19T11:30:09.340 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.340 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.340 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:09.340 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.728643+0000", 2024-03-19T11:30:09.340 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0021926620000000002 2024-03-19T11:30:09.340 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.340 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.341 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:09.341 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.728684+0000", 2024-03-19T11:30:09.341 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0531000000000002e-05 2024-03-19T11:30:09.341 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:09.341 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:09.341 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:09.341 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.341 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.341 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:15 4.4 e28/21 4:21eac9b2:::smithi03829856-4:head v 28'3, mlcod=0'0)", 2024-03-19T11:30:09.341 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.729075+0000", 2024-03-19T11:30:09.341 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.6008775350000004, 2024-03-19T11:30:09.341 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000891773, 2024-03-19T11:30:09.341 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:09.341 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:09.342 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:09.342 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.342 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:09.342 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.729075+0000", 2024-03-19T11:30:09.342 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:09.342 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.342 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.342 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:09.342 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.729075+0000", 2024-03-19T11:30:09.342 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:09.342 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.342 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.342 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:09.343 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.729076+0000", 2024-03-19T11:30:09.343 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.046e-06 2024-03-19T11:30:09.343 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.343 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.343 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:09.343 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.729085+0000", 2024-03-19T11:30:09.343 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6419999999999996e-06 2024-03-19T11:30:09.343 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.343 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.343 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:09.343 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.729099+0000", 2024-03-19T11:30:09.343 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3886000000000001e-05 2024-03-19T11:30:09.343 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.343 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.343 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:09.344 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.729108+0000", 2024-03-19T11:30:09.344 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8510000000000005e-06 2024-03-19T11:30:09.344 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.344 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.344 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:09.344 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.729170+0000", 2024-03-19T11:30:09.344 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2156999999999995e-05 2024-03-19T11:30:09.344 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.344 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.344 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:09.344 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.729193+0000", 2024-03-19T11:30:09.344 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3685000000000001e-05 2024-03-19T11:30:09.344 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.344 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.344 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:09.345 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.729922+0000", 2024-03-19T11:30:09.345 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00072842299999999996 2024-03-19T11:30:09.345 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.345 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.345 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:09.345 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.729967+0000", 2024-03-19T11:30:09.345 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5083000000000002e-05 2024-03-19T11:30:09.345 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:09.345 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:09.345 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:09.345 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.345 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.345 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:33 4.d e28/21 4:b65e6658:::smithi03829856-9:head v 28'4, mlcod=28'2)", 2024-03-19T11:30:09.345 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.785780+0000", 2024-03-19T11:30:09.345 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.5441724790000002, 2024-03-19T11:30:09.346 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0032708569999999998, 2024-03-19T11:30:09.346 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:09.346 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:09.346 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:09.346 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.346 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:09.346 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.785780+0000", 2024-03-19T11:30:09.346 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:09.346 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.346 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.346 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:09.346 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.785780+0000", 2024-03-19T11:30:09.346 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:09.347 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.347 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.347 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:09.347 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.785783+0000", 2024-03-19T11:30:09.347 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4150000000000002e-06 2024-03-19T11:30:09.347 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.347 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.347 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:09.347 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.786482+0000", 2024-03-19T11:30:09.347 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00069957299999999999 2024-03-19T11:30:09.347 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.347 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.347 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:09.348 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.786507+0000", 2024-03-19T11:30:09.348 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5160999999999998e-05 2024-03-19T11:30:09.348 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.348 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.348 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:09.348 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.786521+0000", 2024-03-19T11:30:09.348 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3301e-05 2024-03-19T11:30:09.348 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.348 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.348 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:09.348 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.786599+0000", 2024-03-19T11:30:09.348 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7905999999999996e-05 2024-03-19T11:30:09.348 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.348 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.349 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:09.349 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.786630+0000", 2024-03-19T11:30:09.349 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1636e-05 2024-03-19T11:30:09.349 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.349 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.349 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:09.349 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.789001+0000", 2024-03-19T11:30:09.349 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023710530000000001 2024-03-19T11:30:09.349 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.349 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.349 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:09.349 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.789051+0000", 2024-03-19T11:30:09.349 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9811999999999997e-05 2024-03-19T11:30:09.349 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:09.349 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:09.350 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:09.350 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.350 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.350 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:34 4.d e28/21 4:b65e6658:::smithi03829856-9:head v 28'5, mlcod=28'2)", 2024-03-19T11:30:09.350 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.790321+0000", 2024-03-19T11:30:09.350 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.5396312759999997, 2024-03-19T11:30:09.350 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001391556, 2024-03-19T11:30:09.350 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:09.350 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:09.350 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:09.350 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.350 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:09.351 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.790321+0000", 2024-03-19T11:30:09.351 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:09.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.351 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.351 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:09.351 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.790321+0000", 2024-03-19T11:30:09.351 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:09.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.351 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.351 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:09.351 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.790322+0000", 2024-03-19T11:30:09.351 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0270000000000001e-06 2024-03-19T11:30:09.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.351 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.352 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:09.352 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.790505+0000", 2024-03-19T11:30:09.352 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00018226900000000001 2024-03-19T11:30:09.352 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.352 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.352 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:09.352 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.790518+0000", 2024-03-19T11:30:09.352 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3711e-05 2024-03-19T11:30:09.352 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.352 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.353 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:09.353 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.790527+0000", 2024-03-19T11:30:09.353 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9139999999999997e-06 2024-03-19T11:30:09.353 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.353 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.353 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:09.353 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.790582+0000", 2024-03-19T11:30:09.353 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4678000000000001e-05 2024-03-19T11:30:09.353 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.353 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.353 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:09.353 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.790603+0000", 2024-03-19T11:30:09.353 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0562000000000001e-05 2024-03-19T11:30:09.354 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.354 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.354 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:09.354 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.791674+0000", 2024-03-19T11:30:09.354 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0010713090000000001 2024-03-19T11:30:09.354 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.354 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.354 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:09.354 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.791713+0000", 2024-03-19T11:30:09.354 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.9085999999999998e-05 2024-03-19T11:30:09.354 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:09.354 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:09.355 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:09.355 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.355 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.355 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:35 4.d e28/21 4:b65e6658:::smithi03829856-9:head v 28'6, mlcod=28'2)", 2024-03-19T11:30:09.355 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.792441+0000", 2024-03-19T11:30:09.355 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.5375121199999997, 2024-03-19T11:30:09.355 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00057869099999999999, 2024-03-19T11:30:09.355 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:09.355 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:09.355 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:09.355 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.355 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:09.355 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.792441+0000", 2024-03-19T11:30:09.356 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:09.356 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.356 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.356 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:09.356 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.792441+0000", 2024-03-19T11:30:09.356 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:09.356 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.356 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.356 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:09.356 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.792442+0000", 2024-03-19T11:30:09.356 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1999999999999998e-07 2024-03-19T11:30:09.356 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.357 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.357 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:09.357 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.792450+0000", 2024-03-19T11:30:09.357 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0979999999999994e-06 2024-03-19T11:30:09.357 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.357 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.357 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:09.357 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.792466+0000", 2024-03-19T11:30:09.357 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6473999999999998e-05 2024-03-19T11:30:09.357 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.357 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.357 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:09.357 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.792474+0000", 2024-03-19T11:30:09.357 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9820000000000008e-06 2024-03-19T11:30:09.358 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.358 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.358 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:09.358 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.792539+0000", 2024-03-19T11:30:09.358 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5035000000000003e-05 2024-03-19T11:30:09.358 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.358 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.358 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:09.358 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.792554+0000", 2024-03-19T11:30:09.358 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4945e-05 2024-03-19T11:30:09.359 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.359 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.359 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:09.359 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.792988+0000", 2024-03-19T11:30:09.359 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00043427599999999999 2024-03-19T11:30:09.359 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.359 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.359 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:09.359 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.793019+0000", 2024-03-19T11:30:09.359 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0960999999999997e-05 2024-03-19T11:30:09.359 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:09.359 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:09.359 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:09.359 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.360 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.360 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:871 4.d e32/21 4:b4ef97e0:::smithi03829856-3:head v 32'19, mlcod=28'16)", 2024-03-19T11:30:09.360 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.029331+0000", 2024-03-19T11:30:09.360 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.30062157699999997, 2024-03-19T11:30:09.360 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057816339999999999, 2024-03-19T11:30:09.360 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:09.360 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:09.360 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:09.360 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.361 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:09.361 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.029331+0000", 2024-03-19T11:30:09.361 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:09.361 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.361 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.361 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:09.361 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.029331+0000", 2024-03-19T11:30:09.361 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:09.361 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.361 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.362 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:09.362 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.029331+0000", 2024-03-19T11:30:09.362 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0400000000000002e-07 2024-03-19T11:30:09.362 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.362 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.362 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:09.362 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.030605+0000", 2024-03-19T11:30:09.362 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0012733899999999999 2024-03-19T11:30:09.362 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.362 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.362 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:09.363 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.030620+0000", 2024-03-19T11:30:09.363 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5557999999999999e-05 2024-03-19T11:30:09.363 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.363 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.363 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:09.363 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.030630+0000", 2024-03-19T11:30:09.363 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2660000000000007e-06 2024-03-19T11:30:09.363 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.363 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.364 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:09.364 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.030727+0000", 2024-03-19T11:30:09.364 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7452000000000001e-05 2024-03-19T11:30:09.364 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.364 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.364 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:09.364 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.030757+0000", 2024-03-19T11:30:09.364 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0018999999999998e-05 2024-03-19T11:30:09.364 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.364 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.365 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:09.365 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.035063+0000", 2024-03-19T11:30:09.365 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0043055350000000001 2024-03-19T11:30:09.365 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.365 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.365 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:09.365 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.035113+0000", 2024-03-19T11:30:09.365 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.011e-05 2024-03-19T11:30:09.365 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:09.365 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:09.365 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:09.365 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.365 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.366 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:872 4.d e32/21 4:b4ef97e0:::smithi03829856-3:head v 32'20, mlcod=28'16)", 2024-03-19T11:30:09.366 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.036163+0000", 2024-03-19T11:30:09.366 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.29378935099999998, 2024-03-19T11:30:09.366 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029333250000000001, 2024-03-19T11:30:09.366 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:09.366 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:09.366 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:09.366 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.366 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:09.366 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.036163+0000", 2024-03-19T11:30:09.366 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:09.366 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.366 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.367 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:09.367 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.036163+0000", 2024-03-19T11:30:09.367 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:09.367 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.367 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.367 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:09.367 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.036164+0000", 2024-03-19T11:30:09.367 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7000000000000003e-07 2024-03-19T11:30:09.367 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.367 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.367 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:09.367 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.036721+0000", 2024-03-19T11:30:09.368 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00055646700000000005 2024-03-19T11:30:09.368 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.368 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.368 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:09.368 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.036732+0000", 2024-03-19T11:30:09.368 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1649e-05 2024-03-19T11:30:09.368 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.368 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.368 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:09.368 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.036739+0000", 2024-03-19T11:30:09.368 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6760000000000001e-06 2024-03-19T11:30:09.368 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.368 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.368 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:09.369 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.036785+0000", 2024-03-19T11:30:09.369 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6177000000000002e-05 2024-03-19T11:30:09.369 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.369 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.369 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:09.369 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.036808+0000", 2024-03-19T11:30:09.369 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2654999999999999e-05 2024-03-19T11:30:09.369 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.369 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.369 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:09.369 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.039022+0000", 2024-03-19T11:30:09.369 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022145530000000002 2024-03-19T11:30:09.369 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.369 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.369 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:09.370 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.039097+0000", 2024-03-19T11:30:09.370 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4277999999999996e-05 2024-03-19T11:30:09.370 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:09.370 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:09.370 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:09.370 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.370 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.370 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:873 4.d e32/21 4:b4ef97e0:::smithi03829856-3:head v 32'21, mlcod=28'16)", 2024-03-19T11:30:09.370 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.040151+0000", 2024-03-19T11:30:09.370 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.28980165400000002, 2024-03-19T11:30:09.370 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000691641, 2024-03-19T11:30:09.371 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:09.371 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:09.371 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:09.371 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.371 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:09.371 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.040151+0000", 2024-03-19T11:30:09.371 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:09.371 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.371 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.371 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:09.371 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.040151+0000", 2024-03-19T11:30:09.371 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:09.371 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.372 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.372 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:09.372 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.040152+0000", 2024-03-19T11:30:09.372 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3999999999999999e-06 2024-03-19T11:30:09.372 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.372 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.372 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:09.372 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.040169+0000", 2024-03-19T11:30:09.372 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6093999999999999e-05 2024-03-19T11:30:09.372 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.372 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.372 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:09.372 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.040199+0000", 2024-03-19T11:30:09.372 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0565999999999998e-05 2024-03-19T11:30:09.373 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.373 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.373 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:09.373 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.040218+0000", 2024-03-19T11:30:09.373 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8620000000000001e-05 2024-03-19T11:30:09.373 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.373 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.373 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:09.373 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.040297+0000", 2024-03-19T11:30:09.373 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9105999999999998e-05 2024-03-19T11:30:09.373 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.373 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.373 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:09.373 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.040329+0000", 2024-03-19T11:30:09.374 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2165999999999997e-05 2024-03-19T11:30:09.374 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.374 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.374 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:09.374 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.040808+0000", 2024-03-19T11:30:09.374 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00047866299999999999 2024-03-19T11:30:09.374 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:09.374 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:09.374 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:09.374 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.040843+0000", 2024-03-19T11:30:09.374 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5026e-05 2024-03-19T11:30:09.374 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:09.374 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:09.374 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:09.374 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:09.375 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:09.375 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:09.376 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-19T11:30:09.377 INFO:tasks.rados.rados.0.smithi038.stdout:648: expect (ObjNum 20 snap 0 seq_num 20) 2024-03-19T11:30:09.377 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:09.387+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:09.425 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:30:09.425 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:30:09.441 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:30:09.441 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:30:09.450 INFO:tasks.rados.rados.0.smithi038.stdout:650: finishing set_chunk to oid 49 2024-03-19T11:30:09.450 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 49 set_chunk returned error code -95 offset: 1284608 length: 781312 2024-03-19T11:30:09.450 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-03-19T11:30:09.450 INFO:tasks.rados.rados.0.smithi038.stdout:652: finishing set_chunk to oid 29 2024-03-19T11:30:09.451 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 29 v 3 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-03-19T11:30:09.451 INFO:tasks.rados.rados.0.smithi038.stdout:651: got (0) Success 2024-03-19T11:30:09.451 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 20 v 0 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-03-19T11:30:09.451 INFO:tasks.rados.rados.0.smithi038.stdout:653: read oid 15 snap -1 2024-03-19T11:30:09.451 INFO:tasks.rados.rados.0.smithi038.stdout:653: expect (ObjNum 14 snap 0 seq_num 14) 2024-03-19T11:30:09.451 INFO:tasks.rados.rados.0.smithi038.stdout:647: done (5 left) 2024-03-19T11:30:09.451 INFO:tasks.rados.rados.0.smithi038.stdout:648: done (4 left) 2024-03-19T11:30:09.451 INFO:tasks.rados.rados.0.smithi038.stdout:650: done (3 left) 2024-03-19T11:30:09.451 INFO:tasks.rados.rados.0.smithi038.stdout:651: done (2 left) 2024-03-19T11:30:09.451 INFO:tasks.rados.rados.0.smithi038.stdout:652: done (1 left) 2024-03-19T11:30:09.451 INFO:tasks.rados.rados.0.smithi038.stdout:654: read oid 38 snap -1 2024-03-19T11:30:09.451 INFO:tasks.rados.rados.0.smithi038.stdout:654: expect (ObjNum 37 snap 0 seq_num 37) 2024-03-19T11:30:09.451 INFO:tasks.rados.rados.0.smithi038.stdout:655: delete oid 26 current snap is 3 2024-03-19T11:30:09.454 INFO:tasks.rados.rados.0.smithi038.stdout:655: done (2 left) 2024-03-19T11:30:09.455 INFO:tasks.rados.rados.0.smithi038.stdout:656: set_chunk oid 30 target oid 2024-03-19T11:30:09.460 INFO:tasks.rados.rados.0.smithi038.stdout:656: set_chunk oid 30 offset: 943616 length: 202240 target oid 8892b7b523af2cde6be0d91d0468f8be86a651cb052f82daf3e2c6278805eb6e offset: 0 snap -1 2024-03-19T11:30:09.461 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:30:09.461 INFO:tasks.rados.rados.0.smithi038.stdout:657: read oid 1 snap -1 2024-03-19T11:30:09.461 INFO:tasks.rados.rados.0.smithi038.stdout:657: expect (ObjNum 51 snap 0 seq_num 51) 2024-03-19T11:30:09.461 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:30:09.462 INFO:tasks.rados.rados.0.smithi038.stdout:658: write (excl) oid 16 current snap is 3 2024-03-19T11:30:09.462 INFO:tasks.rados.rados.0.smithi038.stdout:658: seq_num 65 ranges {692455=622009,2070506=712007} 2024-03-19T11:30:09.468 INFO:tasks.rados.rados.0.smithi038.stdout:658: writing smithi03829856-16 from 692455 to 1314464 tid 1 2024-03-19T11:30:09.478 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:09.484+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:09.478 INFO:tasks.rados.rados.0.smithi038.stdout:658: writing smithi03829856-16 from 2070506 to 2782513 tid 2 2024-03-19T11:30:09.478 INFO:tasks.rados.rados.0.smithi038.stdout:653: expect (ObjNum 14 snap 0 seq_num 14) 2024-03-19T11:30:09.481 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:30:09.481 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:30:09.502 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:30:09.502 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:30:09.522 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:30:09.522 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:30:09.542 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:30:09.542 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:30:09.555 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:09.555 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:09.562 INFO:tasks.rados.rados.0.smithi038.stdout:654: expect (ObjNum 37 snap 0 seq_num 37) 2024-03-19T11:30:09.562 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:30:09.563 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:30:09.570 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-19T11:30:09.579 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:09.588+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:09.647 INFO:tasks.rados.rados.0.smithi038.stdout:656: finishing set_chunk to oid 30 2024-03-19T11:30:09.648 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 30 set_chunk returned error code -95 offset: 943616 length: 202240 2024-03-19T11:30:09.648 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 30 v 0 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-03-19T11:30:09.648 INFO:tasks.rados.rados.0.smithi038.stdout:657: expect (ObjNum 51 snap 0 seq_num 51) 2024-03-19T11:30:09.678 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:09.687+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:09.744 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:09.745 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:09.759 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-19T11:30:09.777 INFO:tasks.rados.rados.0.smithi038.stdout:658: finishing write tid 1 to smithi03829856-16 2024-03-19T11:30:09.777 INFO:tasks.rados.rados.0.smithi038.stdout:658: finishing write tid 2 to smithi03829856-16 2024-03-19T11:30:09.778 INFO:tasks.rados.rados.0.smithi038.stdout:658: finishing write tid 3 to smithi03829856-16 2024-03-19T11:30:09.778 INFO:tasks.rados.rados.0.smithi038.stdout:658: finishing write tid 4 to smithi03829856-16 2024-03-19T11:30:09.778 INFO:tasks.rados.rados.0.smithi038.stdout:658: oid 16 updating version 0 to 31 2024-03-19T11:30:09.778 INFO:tasks.rados.rados.0.smithi038.stdout:658: oid 16 version 31 is already newer than 30 2024-03-19T11:30:09.778 INFO:tasks.rados.rados.0.smithi038.stdout:658: oid 16 updating version 31 to 32 2024-03-19T11:30:09.778 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 16 v 32 (ObjNum 65 snap 3 seq_num 65) dirty exists 2024-03-19T11:30:09.778 INFO:tasks.rados.rados.0.smithi038.stdout:658: left oid 16 (ObjNum 65 snap 3 seq_num 65) 2024-03-19T11:30:09.778 INFO:tasks.rados.rados.0.smithi038.stdout:659: read oid 6 snap 0 2024-03-19T11:30:09.778 INFO:tasks.rados.rados.0.smithi038.stdout:659: expect (ObjNum 53 snap 0 seq_num 53) 2024-03-19T11:30:09.778 INFO:tasks.rados.rados.0.smithi038.stdout:653: done (5 left) 2024-03-19T11:30:09.778 INFO:tasks.rados.rados.0.smithi038.stdout:654: done (4 left) 2024-03-19T11:30:09.778 INFO:tasks.rados.rados.0.smithi038.stdout:656: done (3 left) 2024-03-19T11:30:09.778 INFO:tasks.rados.rados.0.smithi038.stdout:657: done (2 left) 2024-03-19T11:30:09.778 INFO:tasks.rados.rados.0.smithi038.stdout:658: done (1 left) 2024-03-19T11:30:09.779 INFO:tasks.rados.rados.0.smithi038.stdout:660: write oid 48 current snap is 3 2024-03-19T11:30:09.779 INFO:tasks.rados.rados.0.smithi038.stdout:660: seq_num 66 ranges {709262=697258,2012635=572018,3046388=62874} 2024-03-19T11:30:09.780 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:09.791+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:09.785 INFO:tasks.rados.rados.0.smithi038.stdout:660: writing smithi03829856-48 from 709262 to 1406520 tid 1 2024-03-19T11:30:09.792 INFO:tasks.rados.rados.0.smithi038.stdout:660: writing smithi03829856-48 from 2012635 to 2584653 tid 2 2024-03-19T11:30:09.796 INFO:tasks.rados.rados.0.smithi038.stdout:660: writing smithi03829856-48 from 3046388 to 3109262 tid 3 2024-03-19T11:30:09.796 INFO:tasks.rados.rados.0.smithi038.stdout:661: read oid 43 snap 0 2024-03-19T11:30:09.796 INFO:tasks.rados.rados.0.smithi038.stdout:661: expect (ObjNum 50 snap 0 seq_num 50) 2024-03-19T11:30:09.796 INFO:tasks.rados.rados.0.smithi038.stdout:662: read oid 40 snap -1 2024-03-19T11:30:09.797 INFO:tasks.rados.rados.0.smithi038.stdout:662: expect (ObjNum 39 snap 0 seq_num 39) 2024-03-19T11:30:09.797 INFO:tasks.rados.rados.0.smithi038.stdout:663: snap_remove snap 1 2024-03-19T11:30:09.881 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:09.888+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:09.894 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:09.894 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:09.908 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-19T11:30:09.982 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:09.992+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:10.040 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:10.040 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:10.054 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-19T11:30:10.100 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:10.111+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:10.172 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:10.172 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:10.184 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:10.195+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:10.186 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-19T11:30:10.286 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:10.296+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:10.354 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:10.354 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:10.368 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-19T11:30:10.412 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:10.423+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:10.491 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:10.499+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:10.492 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:10.492 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:10.506 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-19T11:30:10.592 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:10.604+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:10.669 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:10.669 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:10.683 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-19T11:30:10.717 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:10.724+0000 7fa5e08db640 -1 osd.3 32 handle_osd_map injecting map crc failure 2024-03-19T11:30:10.718 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:10.728+0000 7f6712f92640 -1 osd.2 32 handle_osd_map injecting map crc failure 2024-03-19T11:30:10.728 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:10.739+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:10.729 INFO:tasks.rados.rados.0.smithi038.stdout:659: expect (ObjNum 53 snap 0 seq_num 53) 2024-03-19T11:30:10.795 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:10.803+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:10.808 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:10.808 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:10.824 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-19T11:30:10.824 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-03-19T11:30:10.862 INFO:tasks.rados.rados.0.smithi038.stdout:660: finishing write tid 1 to smithi03829856-48 2024-03-19T11:30:10.863 INFO:tasks.rados.rados.0.smithi038.stdout:659: done (4 left) 2024-03-19T11:30:10.863 INFO:tasks.rados.rados.0.smithi038.stdout:663: done (3 left) 2024-03-19T11:30:10.863 INFO:tasks.rados.rados.0.smithi038.stdout:664: read oid 35 snap -1 2024-03-19T11:30:10.863 INFO:tasks.rados.rados.0.smithi038.stdout:664: expect (ObjNum 34 snap 0 seq_num 34) 2024-03-19T11:30:10.863 INFO:tasks.rados.rados.0.smithi038.stdout:665: write oid 7 current snap is 3 2024-03-19T11:30:10.863 INFO:tasks.rados.rados.0.smithi038.stdout:660: finishing write tid 2 to smithi03829856-48 2024-03-19T11:30:10.863 INFO:tasks.rados.rados.0.smithi038.stdout:660: finishing write tid 3 to smithi03829856-48 2024-03-19T11:30:10.863 INFO:tasks.rados.rados.0.smithi038.stdout:665: seq_num 67 ranges {726069=772507,2071117=715676} 2024-03-19T11:30:10.864 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:10.871+0000 7f1837407640 -1 osd.6 32 handle_osd_map injecting map crc failure 2024-03-19T11:30:10.872 INFO:tasks.rados.rados.0.smithi038.stdout:665: writing smithi03829856-7 from 726069 to 1498576 tid 1 2024-03-19T11:30:10.881 INFO:tasks.rados.rados.0.smithi038.stdout:665: writing smithi03829856-7 from 2071117 to 2786793 tid 2 2024-03-19T11:30:10.881 INFO:tasks.rados.rados.0.smithi038.stdout:666: rollback oid 29 current snap is 3 2024-03-19T11:30:10.881 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 29 to 0 2024-03-19T11:30:10.881 INFO:tasks.rados.rados.0.smithi038.stdout:660: finishing write tid 4 to smithi03829856-48 2024-03-19T11:30:10.882 INFO:tasks.rados.rados.0.smithi038.stdout:662: expect (ObjNum 39 snap 0 seq_num 39) 2024-03-19T11:30:10.908 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:10.919+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:10.967 INFO:tasks.rados.rados.0.smithi038.stdout:660: finishing write tid 5 to smithi03829856-48 2024-03-19T11:30:10.968 INFO:tasks.rados.rados.0.smithi038.stdout:660: oid 48 updating version 0 to 28 2024-03-19T11:30:10.968 INFO:tasks.rados.rados.0.smithi038.stdout:660: oid 48 updating version 28 to 29 2024-03-19T11:30:10.968 INFO:tasks.rados.rados.0.smithi038.stdout:660: oid 48 updating version 29 to 30 2024-03-19T11:30:10.968 INFO:tasks.rados.rados.0.smithi038.stdout:660: oid 48 version 30 is already newer than 27 2024-03-19T11:30:10.968 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 48 v 30 (ObjNum 66 snap 3 seq_num 66) dirty exists 2024-03-19T11:30:10.968 INFO:tasks.rados.rados.0.smithi038.stdout:660: left oid 48 (ObjNum 66 snap 3 seq_num 66) 2024-03-19T11:30:10.968 INFO:tasks.rados.rados.0.smithi038.stdout:661: expect (ObjNum 50 snap 0 seq_num 50) 2024-03-19T11:30:10.992 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:10.992 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:10.996 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:11.003+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:11.006 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-19T11:30:11.085 INFO:tasks.rados.rados.0.smithi038.stdout:664: expect (ObjNum 34 snap 0 seq_num 34) 2024-03-19T11:30:11.097 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:11.104+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:11.164 INFO:tasks.rados.rados.0.smithi038.stdout:665: finishing write tid 1 to smithi03829856-7 2024-03-19T11:30:11.164 INFO:tasks.rados.rados.0.smithi038.stdout:666: finishing rollback tid 0 to smithi03829856-29 2024-03-19T11:30:11.165 INFO:tasks.rados.rados.0.smithi038.stdout:666: finishing rollback tid 1 to smithi03829856-29 2024-03-19T11:30:11.165 INFO:tasks.rados.rados.0.smithi038.stdout:665: finishing write tid 2 to smithi03829856-7 2024-03-19T11:30:11.165 INFO:tasks.rados.rados.0.smithi038.stdout:665: finishing write tid 3 to smithi03829856-7 2024-03-19T11:30:11.165 INFO:tasks.rados.rados.0.smithi038.stdout:666: finishing rollback tid 2 to smithi03829856-29 2024-03-19T11:30:11.165 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 29 v 31 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-03-19T11:30:11.166 INFO:tasks.rados.rados.0.smithi038.stdout:665: finishing write tid 4 to smithi03829856-7 2024-03-19T11:30:11.166 INFO:tasks.rados.rados.0.smithi038.stdout:665: oid 7 updating version 0 to 18 2024-03-19T11:30:11.166 INFO:tasks.rados.rados.0.smithi038.stdout:665: oid 7 version 18 is already newer than 17 2024-03-19T11:30:11.166 INFO:tasks.rados.rados.0.smithi038.stdout:665: oid 7 version 18 is already newer than 16 2024-03-19T11:30:11.166 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 7 v 18 (ObjNum 67 snap 3 seq_num 67) dirty exists 2024-03-19T11:30:11.167 INFO:tasks.rados.rados.0.smithi038.stdout:665: left oid 7 (ObjNum 67 snap 3 seq_num 67) 2024-03-19T11:30:11.167 INFO:tasks.rados.rados.0.smithi038.stdout:660: done (5 left) 2024-03-19T11:30:11.167 INFO:tasks.rados.rados.0.smithi038.stdout:661: done (4 left) 2024-03-19T11:30:11.167 INFO:tasks.rados.rados.0.smithi038.stdout:662: done (3 left) 2024-03-19T11:30:11.167 INFO:tasks.rados.rados.0.smithi038.stdout:664: done (2 left) 2024-03-19T11:30:11.167 INFO:tasks.rados.rados.0.smithi038.stdout:665: done (1 left) 2024-03-19T11:30:11.168 INFO:tasks.rados.rados.0.smithi038.stdout:666: done (0 left) 2024-03-19T11:30:11.168 INFO:tasks.rados.rados.0.smithi038.stdout:667: write oid 38 current snap is 3 2024-03-19T11:30:11.168 INFO:tasks.rados.rados.0.smithi038.stdout:667: seq_num 68 ranges {742876=447756,1729599=575687,3101234=41642} 2024-03-19T11:30:11.170 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:11.170 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:11.183 INFO:tasks.rados.rados.0.smithi038.stdout:667: writing smithi03829856-38 from 742876 to 1190632 tid 1 2024-03-19T11:30:11.184 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-19T11:30:11.197 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:11.204+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:11.201 INFO:tasks.rados.rados.0.smithi038.stdout:667: writing smithi03829856-38 from 1729599 to 2305286 tid 2 2024-03-19T11:30:11.215 INFO:tasks.rados.rados.0.smithi038.stdout:667: writing smithi03829856-38 from 3101234 to 3142876 tid 3 2024-03-19T11:30:11.215 INFO:tasks.rados.rados.0.smithi038.stdout:668: read oid 44 snap -1 2024-03-19T11:30:11.215 INFO:tasks.rados.rados.0.smithi038.stdout:668: expect (ObjNum 43 snap 0 seq_num 43) 2024-03-19T11:30:11.215 INFO:tasks.rados.rados.0.smithi038.stdout:669: read oid 4 snap -1 2024-03-19T11:30:11.216 INFO:tasks.rados.rados.0.smithi038.stdout:669: expect (ObjNum 3 snap 0 seq_num 3) 2024-03-19T11:30:11.216 INFO:tasks.rados.rados.0.smithi038.stdout:670: write oid 18 current snap is 3 2024-03-19T11:30:11.216 INFO:tasks.rados.rados.0.smithi038.stdout:667: finishing write tid 1 to smithi03829856-38 2024-03-19T11:30:11.216 INFO:tasks.rados.rados.0.smithi038.stdout:667: finishing write tid 2 to smithi03829856-38 2024-03-19T11:30:11.216 INFO:tasks.rados.rados.0.smithi038.stdout:670: seq_num 69 ranges {759683=639358,1904434=719345,3128657=31026} 2024-03-19T11:30:11.236 INFO:tasks.rados.rados.0.smithi038.stdout:670: writing smithi03829856-18 from 759683 to 1399041 tid 1 2024-03-19T11:30:11.252 INFO:tasks.rados.rados.0.smithi038.stdout:670: writing smithi03829856-18 from 1904434 to 2623779 tid 2 2024-03-19T11:30:11.259 INFO:tasks.rados.rados.0.smithi038.stdout:670: writing smithi03829856-18 from 3128657 to 3159683 tid 3 2024-03-19T11:30:11.259 INFO:tasks.rados.rados.0.smithi038.stdout:671: set_chunk oid 43 target oid 2024-03-19T11:30:11.281 INFO:tasks.rados.rados.0.smithi038.stdout:671: set_chunk oid 43 offset: 611328 length: 1606656 target oid a3639ef4c215b429a637f1406db2398074b33464b996fe20e20d8b66c6bd720b offset: 0 snap -1 2024-03-19T11:30:11.282 INFO:tasks.rados.rados.0.smithi038.stdout:672: read oid 17 snap -1 2024-03-19T11:30:11.282 INFO:tasks.rados.rados.0.smithi038.stdout:672: expect (ObjNum 16 snap 0 seq_num 16) 2024-03-19T11:30:11.282 INFO:tasks.rados.rados.0.smithi038.stdout:673: snap_create 2024-03-19T11:30:11.282 INFO:tasks.rados.rados.0.smithi038.stdout:669: expect (ObjNum 3 snap 0 seq_num 3) 2024-03-19T11:30:11.298 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:11.304+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:11.310 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:11.310 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:11.324 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-19T11:30:11.361 INFO:tasks.rados.rados.0.smithi038.stdout:668: expect (ObjNum 43 snap 0 seq_num 43) 2024-03-19T11:30:11.398 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:11.407+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:11.446 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:11.446 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:11.446 INFO:tasks.rados.rados.0.smithi038.stdout:667: finishing write tid 3 to smithi03829856-38 2024-03-19T11:30:11.447 INFO:tasks.rados.rados.0.smithi038.stdout:667: finishing write tid 4 to smithi03829856-38 2024-03-19T11:30:11.447 INFO:tasks.rados.rados.0.smithi038.stdout:667: finishing write tid 5 to smithi03829856-38 2024-03-19T11:30:11.447 INFO:tasks.rados.rados.0.smithi038.stdout:667: oid 38 updating version 0 to 32 2024-03-19T11:30:11.447 INFO:tasks.rados.rados.0.smithi038.stdout:667: oid 38 updating version 32 to 33 2024-03-19T11:30:11.447 INFO:tasks.rados.rados.0.smithi038.stdout:667: oid 38 version 33 is already newer than 30 2024-03-19T11:30:11.447 INFO:tasks.rados.rados.0.smithi038.stdout:667: oid 38 version 33 is already newer than 31 2024-03-19T11:30:11.447 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 38 v 33 (ObjNum 68 snap 3 seq_num 68) dirty exists 2024-03-19T11:30:11.447 INFO:tasks.rados.rados.0.smithi038.stdout:667: left oid 38 (ObjNum 68 snap 3 seq_num 68) 2024-03-19T11:30:11.447 INFO:tasks.rados.rados.0.smithi038.stdout:670: finishing write tid 1 to smithi03829856-18 2024-03-19T11:30:11.447 INFO:tasks.rados.rados.0.smithi038.stdout:670: finishing write tid 2 to smithi03829856-18 2024-03-19T11:30:11.447 INFO:tasks.rados.rados.0.smithi038.stdout:670: finishing write tid 3 to smithi03829856-18 2024-03-19T11:30:11.447 INFO:tasks.rados.rados.0.smithi038.stdout:670: finishing write tid 4 to smithi03829856-18 2024-03-19T11:30:11.447 INFO:tasks.rados.rados.0.smithi038.stdout:670: finishing write tid 5 to smithi03829856-18 2024-03-19T11:30:11.447 INFO:tasks.rados.rados.0.smithi038.stdout:670: oid 18 updating version 0 to 29 2024-03-19T11:30:11.447 INFO:tasks.rados.rados.0.smithi038.stdout:670: oid 18 version 29 is already newer than 27 2024-03-19T11:30:11.448 INFO:tasks.rados.rados.0.smithi038.stdout:670: oid 18 updating version 29 to 30 2024-03-19T11:30:11.448 INFO:tasks.rados.rados.0.smithi038.stdout:670: oid 18 version 30 is already newer than 28 2024-03-19T11:30:11.448 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 18 v 30 (ObjNum 69 snap 3 seq_num 69) dirty exists 2024-03-19T11:30:11.448 INFO:tasks.rados.rados.0.smithi038.stdout:670: left oid 18 (ObjNum 69 snap 3 seq_num 69) 2024-03-19T11:30:11.448 INFO:tasks.rados.rados.0.smithi038.stdout:671: finishing set_chunk to oid 43 2024-03-19T11:30:11.448 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 43 v 21 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-03-19T11:30:11.448 INFO:tasks.rados.rados.0.smithi038.stdout:672: expect (ObjNum 16 snap 0 seq_num 16) 2024-03-19T11:30:11.461 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-19T11:30:11.499 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:11.508+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:11.580 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-19T11:30:11.600 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:11.611+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:11.624 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:11.624 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:11.638 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-19T11:30:11.700 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:11.708+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:11.717 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-03-19T11:30:11.723 INFO:tasks.rados.rados.0.smithi038.stdout:667: done (6 left) 2024-03-19T11:30:11.723 INFO:tasks.rados.rados.0.smithi038.stdout:668: done (5 left) 2024-03-19T11:30:11.723 INFO:tasks.rados.rados.0.smithi038.stdout:669: done (4 left) 2024-03-19T11:30:11.723 INFO:tasks.rados.rados.0.smithi038.stdout:670: done (3 left) 2024-03-19T11:30:11.723 INFO:tasks.rados.rados.0.smithi038.stdout:671: done (2 left) 2024-03-19T11:30:11.723 INFO:tasks.rados.rados.0.smithi038.stdout:672: done (1 left) 2024-03-19T11:30:11.723 INFO:tasks.rados.rados.0.smithi038.stdout:673: done (0 left) 2024-03-19T11:30:11.724 INFO:tasks.rados.rados.0.smithi038.stdout:674: read oid 33 snap -1 2024-03-19T11:30:11.724 INFO:tasks.rados.rados.0.smithi038.stdout:674: expect (ObjNum 32 snap 0 seq_num 32) 2024-03-19T11:30:11.724 INFO:tasks.rados.rados.0.smithi038.stdout:675: read oid 2 snap -1 2024-03-19T11:30:11.724 INFO:tasks.rados.rados.0.smithi038.stdout:675: expect (ObjNum 1 snap 0 seq_num 1) 2024-03-19T11:30:11.724 INFO:tasks.rados.rados.0.smithi038.stdout:676: read oid 12 snap -1 2024-03-19T11:30:11.724 INFO:tasks.rados.rados.0.smithi038.stdout:676: expect (ObjNum 11 snap 0 seq_num 11) 2024-03-19T11:30:11.724 INFO:tasks.rados.rados.0.smithi038.stdout:677: read oid 41 snap -1 2024-03-19T11:30:11.724 INFO:tasks.rados.rados.0.smithi038.stdout:677: expect (ObjNum 40 snap 0 seq_num 40) 2024-03-19T11:30:11.724 INFO:tasks.rados.rados.0.smithi038.stdout:678: read oid 16 snap 3 2024-03-19T11:30:11.724 INFO:tasks.rados.rados.0.smithi038.stdout:678: expect (ObjNum 65 snap 3 seq_num 65) 2024-03-19T11:30:11.724 INFO:tasks.rados.rados.0.smithi038.stdout:679: set_chunk oid 35 target oid 2024-03-19T11:30:11.724 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:11.732+0000 7f70c6fdf640 -1 osd.0 33 handle_osd_map injecting map crc failure 2024-03-19T11:30:11.739 INFO:tasks.rados.rados.0.smithi038.stdout:679: set_chunk oid 35 offset: 1611264 length: 352256 target oid 224af4be3148b0e1d1488e9a61d61df0d18c2aaf99226706579b57650bfaf388 offset: 0 snap -1 2024-03-19T11:30:11.739 INFO:tasks.rados.rados.0.smithi038.stdout:680: set_chunk oid 30 target oid 2024-03-19T11:30:11.744 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:30:11.745 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:30:11.749 INFO:tasks.rados.rados.0.smithi038.stdout:680: set_chunk oid 30 offset: 571904 length: 689152 target oid 06a0957e01d2e04d1b09d0c04f111819905dcf8372f2c6bee5014c22d5b575ef offset: 0 snap -1 2024-03-19T11:30:11.749 INFO:tasks.rados.rados.0.smithi038.stdout:681: set_chunk oid 24 target oid 2024-03-19T11:30:11.753 INFO:tasks.rados.rados.0.smithi038.stdout:681: set_chunk oid 24 offset: 1684480 length: 301056 target oid 3a5754b7623e7ca5a5d75294c93fff32541fd7e390d130b21bd43d84137ffa31 offset: 0 snap 0 2024-03-19T11:30:11.753 INFO:tasks.rados.rados.0.smithi038.stdout:682: read oid 42 snap -1 2024-03-19T11:30:11.754 INFO:tasks.rados.rados.0.smithi038.stdout:682: expect (ObjNum 52 snap 0 seq_num 52) 2024-03-19T11:30:11.754 INFO:tasks.rados.rados.0.smithi038.stdout:683: read oid 34 snap -1 2024-03-19T11:30:11.754 INFO:tasks.rados.rados.0.smithi038.stdout:683: expect (ObjNum 63 snap 3 seq_num 63) 2024-03-19T11:30:11.754 INFO:tasks.rados.rados.0.smithi038.stdout:674: expect (ObjNum 32 snap 0 seq_num 32) 2024-03-19T11:30:11.763 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:30:11.763 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:30:11.764 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-19T11:30:11.765 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-19T11:30:11.765 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-19T11:30:11.765 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-03-19T11:30:11.780 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:30:11.780 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:30:11.796 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:30:11.796 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:30:11.801 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:11.808+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:11.809 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:11.809 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:11.817 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:30:11.817 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:30:11.824 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-19T11:30:11.837 INFO:tasks.rados.rados.0.smithi038.stdout:676: expect (ObjNum 11 snap 0 seq_num 11) 2024-03-19T11:30:11.838 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:30:11.838 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:30:11.856 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:30:11.856 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:30:11.876 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:30:11.876 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:30:11.902 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:11.911+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:11.909 INFO:tasks.rados.rados.0.smithi038.stdout:675: expect (ObjNum 1 snap 0 seq_num 1) 2024-03-19T11:30:11.945 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:11.945 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:30:11.946 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:30:11.946 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:11.959 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-19T11:30:11.973 INFO:tasks.rados.rados.0.smithi038.stdout:677: expect (ObjNum 40 snap 0 seq_num 40) 2024-03-19T11:30:12.003 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:12.015+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:12.058 INFO:tasks.rados.rados.0.smithi038.stdout:679: finishing set_chunk to oid 35 2024-03-19T11:30:12.058 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 35 set_chunk returned error code -95 offset: 1611264 length: 352256 2024-03-19T11:30:12.058 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 35 v 0 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-03-19T11:30:12.058 INFO:tasks.rados.rados.0.smithi038.stdout:678: expect (ObjNum 65 snap 3 seq_num 65) 2024-03-19T11:30:12.067 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:30:12.067 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:12.068 INFO:teuthology.orchestra.run.smithi038.stdout: "epoch": 34, 2024-03-19T11:30:12.068 INFO:teuthology.orchestra.run.smithi038.stdout: "fsid": "246bcf68-c431-4aa7-b787-5e9b2d9519d6", 2024-03-19T11:30:12.068 INFO:teuthology.orchestra.run.smithi038.stdout: "created": "2024-03-19T11:29:33.327073+0000", 2024-03-19T11:30:12.068 INFO:teuthology.orchestra.run.smithi038.stdout: "modified": "2024-03-19T11:30:11.730797+0000", 2024-03-19T11:30:12.068 INFO:teuthology.orchestra.run.smithi038.stdout: "last_up_change": "2024-03-19T11:29:43.166618+0000", 2024-03-19T11:30:12.068 INFO:teuthology.orchestra.run.smithi038.stdout: "last_in_change": "2024-03-19T11:29:35.632327+0000", 2024-03-19T11:30:12.068 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-19T11:30:12.068 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_num": 5806080, 2024-03-19T11:30:12.068 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_set": [ 2024-03-19T11:30:12.068 INFO:teuthology.orchestra.run.smithi038.stdout: "nodeep-scrub", 2024-03-19T11:30:12.068 INFO:teuthology.orchestra.run.smithi038.stdout: "noscrub", 2024-03-19T11:30:12.068 INFO:teuthology.orchestra.run.smithi038.stdout: "pglog_hardlimit", 2024-03-19T11:30:12.068 INFO:teuthology.orchestra.run.smithi038.stdout: "purged_snapdirs", 2024-03-19T11:30:12.068 INFO:teuthology.orchestra.run.smithi038.stdout: "recovery_deletes", 2024-03-19T11:30:12.068 INFO:teuthology.orchestra.run.smithi038.stdout: "sortbitwise" 2024-03-19T11:30:12.069 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-19T11:30:12.069 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_version": 5, 2024-03-19T11:30:12.069 INFO:teuthology.orchestra.run.smithi038.stdout: "full_ratio": 0.94999998807907104, 2024-03-19T11:30:12.069 INFO:teuthology.orchestra.run.smithi038.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-19T11:30:12.069 INFO:teuthology.orchestra.run.smithi038.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-19T11:30:12.069 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_snapshot": "", 2024-03-19T11:30:12.069 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_max": 4, 2024-03-19T11:30:12.069 INFO:teuthology.orchestra.run.smithi038.stdout: "max_osd": 8, 2024-03-19T11:30:12.069 INFO:teuthology.orchestra.run.smithi038.stdout: "require_min_compat_client": "reef", 2024-03-19T11:30:12.069 INFO:teuthology.orchestra.run.smithi038.stdout: "min_compat_client": "luminous", 2024-03-19T11:30:12.069 INFO:teuthology.orchestra.run.smithi038.stdout: "require_osd_release": "squid", 2024-03-19T11:30:12.069 INFO:teuthology.orchestra.run.smithi038.stdout: "allow_crimson": false, 2024-03-19T11:30:12.069 INFO:teuthology.orchestra.run.smithi038.stdout: "pools": [ 2024-03-19T11:30:12.069 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.069 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 1, 2024-03-19T11:30:12.070 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": ".mgr", 2024-03-19T11:30:12.070 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-19T11:29:42.219654+0000", 2024-03-19T11:30:12.070 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-19T11:30:12.070 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-03-19T11:30:12.070 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-19T11:30:12.070 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 2, 2024-03-19T11:30:12.070 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-19T11:30:12.070 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-19T11:30:12.070 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-19T11:30:12.070 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-19T11:30:12.070 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-19T11:30:12.070 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-19T11:30:12.070 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-19T11:30:12.071 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-19T11:30:12.071 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 1, 2024-03-19T11:30:12.071 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 1, 2024-03-19T11:30:12.071 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 1, 2024-03-19T11:30:12.071 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 1, 2024-03-19T11:30:12.071 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 1, 2024-03-19T11:30:12.071 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-19T11:30:12.071 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-19T11:30:12.071 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-19T11:30:12.071 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-19T11:30:12.072 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-19T11:30:12.072 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-19T11:30:12.072 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-19T11:30:12.072 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.072 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "16", 2024-03-19T11:30:12.072 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-19T11:30:12.072 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-19T11:30:12.072 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-19T11:30:12.072 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-19T11:30:12.072 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-19T11:30:12.072 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-03-19T11:30:12.072 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-03-19T11:30:12.072 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-19T11:30:12.073 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-19T11:30:12.073 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-19T11:30:12.073 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-19T11:30:12.073 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-19T11:30:12.073 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-19T11:30:12.073 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-19T11:30:12.073 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-19T11:30:12.073 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-19T11:30:12.073 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-19T11:30:12.073 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-19T11:30:12.073 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-19T11:30:12.073 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-19T11:30:12.073 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-19T11:30:12.073 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-19T11:30:12.074 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-19T11:30:12.074 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-19T11:30:12.074 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-19T11:30:12.074 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-19T11:30:12.074 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.074 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-19T11:30:12.074 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-19T11:30:12.074 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-19T11:30:12.074 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-19T11:30:12.074 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-19T11:30:12.074 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-19T11:30:12.074 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-19T11:30:12.074 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-19T11:30:12.075 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-19T11:30:12.075 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-19T11:30:12.075 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-19T11:30:12.075 INFO:teuthology.orchestra.run.smithi038.stdout: "options": { 2024-03-19T11:30:12.075 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_max": 32, 2024-03-19T11:30:12.075 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_min": 1 2024-03-19T11:30:12.075 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.075 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-19T11:30:12.075 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr": {} 2024-03-19T11:30:12.075 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.075 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-19T11:30:12.075 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-19T11:30:12.075 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 8, 2024-03-19T11:30:12.075 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 8, 2024-03-19T11:30:12.076 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.25, 2024-03-19T11:30:12.076 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 2, 2024-03-19T11:30:12.076 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 2, 2024-03-19T11:30:12.076 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 1, 2024-03-19T11:30:12.076 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 1, 2024-03-19T11:30:12.076 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 1 2024-03-19T11:30:12.076 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.076 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.076 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.076 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 2, 2024-03-19T11:30:12.076 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": "rbd", 2024-03-19T11:30:12.076 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-19T11:29:44.428406+0000", 2024-03-19T11:30:12.076 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-19T11:30:12.076 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-03-19T11:30:12.077 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-19T11:30:12.077 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 2, 2024-03-19T11:30:12.077 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-19T11:30:12.077 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-19T11:30:12.077 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-19T11:30:12.077 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-19T11:30:12.077 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-19T11:30:12.077 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-19T11:30:12.077 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-19T11:30:12.077 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-19T11:30:12.077 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 8, 2024-03-19T11:30:12.077 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 8, 2024-03-19T11:30:12.077 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 8, 2024-03-19T11:30:12.077 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 8, 2024-03-19T11:30:12.077 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 8, 2024-03-19T11:30:12.078 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-19T11:30:12.078 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-19T11:30:12.078 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-19T11:30:12.078 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-19T11:30:12.078 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-19T11:30:12.078 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-19T11:30:12.078 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-19T11:30:12.078 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.078 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "18", 2024-03-19T11:30:12.078 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-19T11:30:12.078 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-19T11:30:12.078 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-19T11:30:12.078 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-19T11:30:12.078 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-19T11:30:12.078 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-03-19T11:30:12.079 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-03-19T11:30:12.079 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-19T11:30:12.079 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-19T11:30:12.079 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-19T11:30:12.079 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-19T11:30:12.079 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-19T11:30:12.079 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-19T11:30:12.079 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-19T11:30:12.079 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-19T11:30:12.079 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-19T11:30:12.079 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-19T11:30:12.079 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-19T11:30:12.079 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-19T11:30:12.079 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-19T11:30:12.079 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-19T11:30:12.080 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-19T11:30:12.080 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-19T11:30:12.080 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-19T11:30:12.080 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-19T11:30:12.080 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-19T11:30:12.080 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.080 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-19T11:30:12.080 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-19T11:30:12.080 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-19T11:30:12.080 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-19T11:30:12.080 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-19T11:30:12.080 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-19T11:30:12.081 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-19T11:30:12.081 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-19T11:30:12.081 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-19T11:30:12.081 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-19T11:30:12.081 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-19T11:30:12.081 INFO:teuthology.orchestra.run.smithi038.stdout: "options": {}, 2024-03-19T11:30:12.081 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-19T11:30:12.081 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd": {} 2024-03-19T11:30:12.081 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.081 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-19T11:30:12.081 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-19T11:30:12.081 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 1.9900000095367432, 2024-03-19T11:30:12.081 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 1.9900000095367432, 2024-03-19T11:30:12.081 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.87999999523162842, 2024-03-19T11:30:12.082 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 1.75, 2024-03-19T11:30:12.082 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 1.75, 2024-03-19T11:30:12.082 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 1, 2024-03-19T11:30:12.082 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 1, 2024-03-19T11:30:12.082 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 1 2024-03-19T11:30:12.082 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.082 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.082 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.082 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 3, 2024-03-19T11:30:12.082 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": "low_tier", 2024-03-19T11:30:12.082 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-19T11:29:55.608699+0000", 2024-03-19T11:30:12.082 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-19T11:30:12.082 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-03-19T11:30:12.082 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-19T11:30:12.083 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 2, 2024-03-19T11:30:12.083 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-19T11:30:12.083 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-19T11:30:12.083 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-19T11:30:12.083 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-19T11:30:12.083 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-19T11:30:12.083 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-19T11:30:12.083 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-19T11:30:12.083 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-19T11:30:12.083 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 4, 2024-03-19T11:30:12.083 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 4, 2024-03-19T11:30:12.083 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 4, 2024-03-19T11:30:12.083 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 4, 2024-03-19T11:30:12.083 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 4, 2024-03-19T11:30:12.083 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-19T11:30:12.084 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-19T11:30:12.084 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-19T11:30:12.084 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-19T11:30:12.084 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-19T11:30:12.084 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-19T11:30:12.084 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-19T11:30:12.084 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.084 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "22", 2024-03-19T11:30:12.084 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-19T11:30:12.084 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-19T11:30:12.084 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-19T11:30:12.084 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-19T11:30:12.085 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-19T11:30:12.085 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-03-19T11:30:12.085 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-03-19T11:30:12.085 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-19T11:30:12.085 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-19T11:30:12.085 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-19T11:30:12.085 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-19T11:30:12.085 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-19T11:30:12.085 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-19T11:30:12.085 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-19T11:30:12.085 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-19T11:30:12.085 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-19T11:30:12.085 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-19T11:30:12.086 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-19T11:30:12.086 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-19T11:30:12.086 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-19T11:30:12.086 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-19T11:30:12.086 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-19T11:30:12.086 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-19T11:30:12.086 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-19T11:30:12.086 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-19T11:30:12.086 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-19T11:30:12.086 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.086 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-19T11:30:12.086 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-19T11:30:12.086 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-19T11:30:12.086 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-19T11:30:12.087 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-19T11:30:12.087 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-19T11:30:12.087 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-19T11:30:12.087 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-19T11:30:12.087 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-19T11:30:12.087 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-19T11:30:12.087 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-19T11:30:12.087 INFO:teuthology.orchestra.run.smithi038.stdout: "options": {}, 2024-03-19T11:30:12.087 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": {}, 2024-03-19T11:30:12.087 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-19T11:30:12.087 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-19T11:30:12.087 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 2, 2024-03-19T11:30:12.087 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 2, 2024-03-19T11:30:12.087 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.75, 2024-03-19T11:30:12.087 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 1.5, 2024-03-19T11:30:12.088 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 1.5, 2024-03-19T11:30:12.088 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 1, 2024-03-19T11:30:12.088 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 1, 2024-03-19T11:30:12.088 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 1 2024-03-19T11:30:12.088 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.088 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.088 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.088 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 4, 2024-03-19T11:30:12.088 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": "unique_pool_0", 2024-03-19T11:30:12.088 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-19T11:29:56.945423+0000", 2024-03-19T11:30:12.088 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 8193, 2024-03-19T11:30:12.088 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-19T11:30:12.088 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-19T11:30:12.088 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 2, 2024-03-19T11:30:12.089 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-19T11:30:12.089 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-19T11:30:12.089 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-19T11:30:12.089 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-19T11:30:12.089 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-19T11:30:12.089 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-19T11:30:12.089 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-19T11:30:12.089 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-19T11:30:12.089 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 16, 2024-03-19T11:30:12.089 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 16, 2024-03-19T11:30:12.089 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 16, 2024-03-19T11:30:12.089 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 16, 2024-03-19T11:30:12.089 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 16, 2024-03-19T11:30:12.089 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-19T11:30:12.090 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-19T11:30:12.090 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-19T11:30:12.090 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-19T11:30:12.090 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-19T11:30:12.090 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-19T11:30:12.090 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-19T11:30:12.090 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.090 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "34", 2024-03-19T11:30:12.090 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-19T11:30:12.090 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-19T11:30:12.090 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-19T11:30:12.090 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-19T11:30:12.090 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-19T11:30:12.091 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 5, 2024-03-19T11:30:12.091 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 34, 2024-03-19T11:30:12.091 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-19T11:30:12.091 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-19T11:30:12.091 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-19T11:30:12.091 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-19T11:30:12.091 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-19T11:30:12.091 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-19T11:30:12.091 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-19T11:30:12.091 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-19T11:30:12.092 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-19T11:30:12.092 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-19T11:30:12.092 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-19T11:30:12.092 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-19T11:30:12.092 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-19T11:30:12.092 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-19T11:30:12.092 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-19T11:30:12.092 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-19T11:30:12.092 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-19T11:30:12.092 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-19T11:30:12.092 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-19T11:30:12.092 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.092 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-19T11:30:12.092 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-19T11:30:12.093 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-19T11:30:12.093 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-19T11:30:12.093 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-19T11:30:12.093 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-19T11:30:12.093 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-19T11:30:12.093 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-19T11:30:12.093 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-19T11:30:12.093 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-19T11:30:12.093 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-19T11:30:12.093 INFO:teuthology.orchestra.run.smithi038.stdout: "options": { 2024-03-19T11:30:12.093 INFO:teuthology.orchestra.run.smithi038.stdout: "dedup_cdc_chunk_size": 131072, 2024-03-19T11:30:12.094 INFO:teuthology.orchestra.run.smithi038.stdout: "dedup_chunk_algorithm": "fastcdc", 2024-03-19T11:30:12.094 INFO:teuthology.orchestra.run.smithi038.stdout: "dedup_tier": 3, 2024-03-19T11:30:12.094 INFO:teuthology.orchestra.run.smithi038.stdout: "fingerprint_algorithm": "sha256" 2024-03-19T11:30:12.094 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.094 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-19T11:30:12.094 INFO:teuthology.orchestra.run.smithi038.stdout: "rados": {} 2024-03-19T11:30:12.094 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.094 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-19T11:30:12.094 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-19T11:30:12.094 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 2.5, 2024-03-19T11:30:12.094 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 2.5, 2024-03-19T11:30:12.094 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 1, 2024-03-19T11:30:12.094 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 2.5, 2024-03-19T11:30:12.095 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 2.5, 2024-03-19T11:30:12.095 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 1, 2024-03-19T11:30:12.095 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 1, 2024-03-19T11:30:12.095 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 1 2024-03-19T11:30:12.095 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.095 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.095 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-19T11:30:12.095 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-03-19T11:30:12.095 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.095 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 0, 2024-03-19T11:30:12.095 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "3bb2d2be-5253-4da0-acf3-85a114dc3801", 2024-03-19T11:30:12.095 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-19T11:30:12.095 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-19T11:30:12.096 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-19T11:30:12.096 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-19T11:30:12.096 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-19T11:30:12.096 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-19T11:30:12.096 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 13, 2024-03-19T11:30:12.096 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 21, 2024-03-19T11:30:12.096 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-19T11:30:12.096 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-19T11:30:12.096 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-19T11:30:12.096 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:12.096 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.096 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:12.096 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6808", 2024-03-19T11:30:12.096 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3148574891 2024-03-19T11:30:12.096 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.097 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.097 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:12.097 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6811", 2024-03-19T11:30:12.097 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3148574891 2024-03-19T11:30:12.097 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.097 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.097 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.097 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-19T11:30:12.097 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:12.097 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.098 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:12.098 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6813", 2024-03-19T11:30:12.098 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3148574891 2024-03-19T11:30:12.098 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.098 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.098 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:12.098 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6815", 2024-03-19T11:30:12.098 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3148574891 2024-03-19T11:30:12.098 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.098 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.098 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.098 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-19T11:30:12.098 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:12.098 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.098 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:12.099 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6822", 2024-03-19T11:30:12.099 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3148574891 2024-03-19T11:30:12.099 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.099 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.099 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:12.099 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6823", 2024-03-19T11:30:12.099 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3148574891 2024-03-19T11:30:12.099 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.099 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.099 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.099 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-19T11:30:12.099 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:12.099 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.099 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:12.099 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6818", 2024-03-19T11:30:12.100 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3148574891 2024-03-19T11:30:12.100 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.100 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.100 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:12.100 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6821", 2024-03-19T11:30:12.100 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3148574891 2024-03-19T11:30:12.100 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.100 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.100 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.100 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6811/3148574891", 2024-03-19T11:30:12.100 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6815/3148574891", 2024-03-19T11:30:12.101 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6823/3148574891", 2024-03-19T11:30:12.101 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6821/3148574891", 2024-03-19T11:30:12.101 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-19T11:30:12.101 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-19T11:30:12.101 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-19T11:30:12.101 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.101 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.101 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.101 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 1, 2024-03-19T11:30:12.101 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "e1c926b2-e0be-4ccc-b8a5-d80d0b584ead", 2024-03-19T11:30:12.101 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-19T11:30:12.101 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-19T11:30:12.101 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-19T11:30:12.101 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-19T11:30:12.101 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-19T11:30:12.102 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-19T11:30:12.102 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 14, 2024-03-19T11:30:12.102 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 21, 2024-03-19T11:30:12.102 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-19T11:30:12.102 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-19T11:30:12.102 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-19T11:30:12.102 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:12.102 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.102 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:12.102 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6824", 2024-03-19T11:30:12.102 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 675027622 2024-03-19T11:30:12.102 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.102 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.102 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:12.103 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6825", 2024-03-19T11:30:12.103 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 675027622 2024-03-19T11:30:12.103 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.103 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.103 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.103 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-19T11:30:12.103 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:12.103 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.103 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:12.103 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6826", 2024-03-19T11:30:12.103 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 675027622 2024-03-19T11:30:12.103 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.103 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.103 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:12.103 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6827", 2024-03-19T11:30:12.104 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 675027622 2024-03-19T11:30:12.104 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.104 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.104 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.104 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-19T11:30:12.104 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:12.104 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.104 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:12.104 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6830", 2024-03-19T11:30:12.104 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 675027622 2024-03-19T11:30:12.104 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.104 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.104 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:12.104 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6831", 2024-03-19T11:30:12.104 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 675027622 2024-03-19T11:30:12.105 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.105 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.105 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.105 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-19T11:30:12.105 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:12.105 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.105 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:12.105 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6828", 2024-03-19T11:30:12.105 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 675027622 2024-03-19T11:30:12.105 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.105 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.105 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:12.105 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6829", 2024-03-19T11:30:12.105 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 675027622 2024-03-19T11:30:12.105 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.106 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.106 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.106 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6825/675027622", 2024-03-19T11:30:12.106 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6827/675027622", 2024-03-19T11:30:12.106 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6831/675027622", 2024-03-19T11:30:12.106 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6829/675027622", 2024-03-19T11:30:12.106 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-19T11:30:12.106 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-19T11:30:12.106 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-19T11:30:12.106 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.106 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.106 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.106 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 2, 2024-03-19T11:30:12.106 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "61262219-4f8a-4733-af9d-b5208a17c0ea", 2024-03-19T11:30:12.107 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-19T11:30:12.107 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-19T11:30:12.107 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-19T11:30:12.107 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-19T11:30:12.107 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-19T11:30:12.107 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-19T11:30:12.107 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 14, 2024-03-19T11:30:12.107 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 25, 2024-03-19T11:30:12.107 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-19T11:30:12.107 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-19T11:30:12.107 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-19T11:30:12.107 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:12.107 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.107 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:12.107 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6800", 2024-03-19T11:30:12.108 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 58279616 2024-03-19T11:30:12.108 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.108 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.108 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:12.108 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6801", 2024-03-19T11:30:12.108 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 58279616 2024-03-19T11:30:12.108 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.108 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.108 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.108 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-19T11:30:12.108 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:12.108 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.108 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:12.108 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6802", 2024-03-19T11:30:12.108 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 58279616 2024-03-19T11:30:12.108 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.109 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.109 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:12.109 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6804", 2024-03-19T11:30:12.109 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 58279616 2024-03-19T11:30:12.109 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.109 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.109 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.109 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-19T11:30:12.109 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:12.109 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.109 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:12.109 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6814", 2024-03-19T11:30:12.109 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 58279616 2024-03-19T11:30:12.109 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.110 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.110 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:12.110 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6817", 2024-03-19T11:30:12.110 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 58279616 2024-03-19T11:30:12.110 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.110 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.110 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.110 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-19T11:30:12.110 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:12.110 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.110 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:12.110 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6807", 2024-03-19T11:30:12.110 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 58279616 2024-03-19T11:30:12.110 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.111 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.111 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:12.111 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6810", 2024-03-19T11:30:12.111 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 58279616 2024-03-19T11:30:12.111 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.111 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.111 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.111 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6801/58279616", 2024-03-19T11:30:12.111 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6804/58279616", 2024-03-19T11:30:12.111 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6817/58279616", 2024-03-19T11:30:12.111 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6810/58279616", 2024-03-19T11:30:12.111 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-19T11:30:12.111 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-19T11:30:12.111 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-19T11:30:12.111 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.112 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.112 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.112 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 3, 2024-03-19T11:30:12.112 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "a31f8d1b-4e38-4ca3-b251-0877002d7235", 2024-03-19T11:30:12.112 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-19T11:30:12.112 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-19T11:30:12.112 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-19T11:30:12.112 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-19T11:30:12.112 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-19T11:30:12.112 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-19T11:30:12.112 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 14, 2024-03-19T11:30:12.112 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 21, 2024-03-19T11:30:12.112 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-19T11:30:12.112 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-19T11:30:12.112 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-19T11:30:12.113 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:12.113 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.113 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:12.113 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6803", 2024-03-19T11:30:12.113 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2942163952 2024-03-19T11:30:12.113 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.113 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.113 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:12.113 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6805", 2024-03-19T11:30:12.113 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2942163952 2024-03-19T11:30:12.113 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.113 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.114 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.114 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-19T11:30:12.114 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:12.114 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.114 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:12.114 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6806", 2024-03-19T11:30:12.114 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2942163952 2024-03-19T11:30:12.114 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.114 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.114 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:12.114 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6809", 2024-03-19T11:30:12.114 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2942163952 2024-03-19T11:30:12.114 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.114 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.114 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.115 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-19T11:30:12.115 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:12.115 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.115 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:12.115 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6819", 2024-03-19T11:30:12.115 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2942163952 2024-03-19T11:30:12.115 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.115 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.115 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:12.115 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6820", 2024-03-19T11:30:12.115 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2942163952 2024-03-19T11:30:12.115 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.115 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.115 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.116 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-19T11:30:12.116 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:12.116 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.116 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:12.116 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6812", 2024-03-19T11:30:12.116 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2942163952 2024-03-19T11:30:12.116 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.116 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.116 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:12.116 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6816", 2024-03-19T11:30:12.116 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2942163952 2024-03-19T11:30:12.116 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.116 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.116 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.116 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6805/2942163952", 2024-03-19T11:30:12.116 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6809/2942163952", 2024-03-19T11:30:12.117 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6820/2942163952", 2024-03-19T11:30:12.117 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6816/2942163952", 2024-03-19T11:30:12.117 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-19T11:30:12.117 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-19T11:30:12.117 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-19T11:30:12.117 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.117 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.117 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.117 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 4, 2024-03-19T11:30:12.117 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "6746723d-f59c-40ba-857e-d8c954d2675f", 2024-03-19T11:30:12.117 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-19T11:30:12.117 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-19T11:30:12.117 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-19T11:30:12.117 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-19T11:30:12.118 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-19T11:30:12.118 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-19T11:30:12.118 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 13, 2024-03-19T11:30:12.118 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 21, 2024-03-19T11:30:12.118 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-19T11:30:12.118 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-19T11:30:12.118 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-19T11:30:12.118 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:12.118 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.118 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:12.118 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6808", 2024-03-19T11:30:12.118 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1353919713 2024-03-19T11:30:12.118 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.118 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.118 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:12.119 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6810", 2024-03-19T11:30:12.119 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1353919713 2024-03-19T11:30:12.119 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.119 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.119 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.119 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-19T11:30:12.119 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:12.119 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.119 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:12.119 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6812", 2024-03-19T11:30:12.121 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1353919713 2024-03-19T11:30:12.121 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.121 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.121 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:12.121 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6814", 2024-03-19T11:30:12.121 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1353919713 2024-03-19T11:30:12.121 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.121 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.121 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.122 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-19T11:30:12.122 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:12.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.122 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:12.122 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6819", 2024-03-19T11:30:12.122 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1353919713 2024-03-19T11:30:12.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.122 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:12.122 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6821", 2024-03-19T11:30:12.122 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1353919713 2024-03-19T11:30:12.123 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.123 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.123 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.123 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-19T11:30:12.123 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:12.123 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.123 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:12.123 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6816", 2024-03-19T11:30:12.123 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1353919713 2024-03-19T11:30:12.123 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.123 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.123 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:12.123 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6818", 2024-03-19T11:30:12.123 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1353919713 2024-03-19T11:30:12.123 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.124 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.124 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.124 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.172:6810/1353919713", 2024-03-19T11:30:12.124 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.172:6814/1353919713", 2024-03-19T11:30:12.124 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.172:6821/1353919713", 2024-03-19T11:30:12.124 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.172:6818/1353919713", 2024-03-19T11:30:12.124 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-19T11:30:12.124 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-19T11:30:12.124 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-19T11:30:12.124 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.124 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.124 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.124 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 5, 2024-03-19T11:30:12.124 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "f59afdf8-2277-4583-af68-78136f1c5d62", 2024-03-19T11:30:12.125 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-19T11:30:12.125 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-19T11:30:12.125 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-19T11:30:12.125 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-19T11:30:12.125 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-19T11:30:12.125 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-19T11:30:12.125 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 14, 2024-03-19T11:30:12.125 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 21, 2024-03-19T11:30:12.125 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-19T11:30:12.125 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-19T11:30:12.125 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-19T11:30:12.125 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:12.125 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.125 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:12.126 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6809", 2024-03-19T11:30:12.126 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2733060387 2024-03-19T11:30:12.126 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.126 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.126 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:12.126 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6811", 2024-03-19T11:30:12.126 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2733060387 2024-03-19T11:30:12.126 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.126 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.126 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.126 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-19T11:30:12.126 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:12.126 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.126 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:12.127 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6813", 2024-03-19T11:30:12.127 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2733060387 2024-03-19T11:30:12.127 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.127 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.127 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:12.127 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6815", 2024-03-19T11:30:12.127 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2733060387 2024-03-19T11:30:12.127 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.127 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.127 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.127 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-19T11:30:12.127 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:12.127 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.127 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:12.128 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6822", 2024-03-19T11:30:12.128 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2733060387 2024-03-19T11:30:12.128 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.128 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.128 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:12.128 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6823", 2024-03-19T11:30:12.128 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2733060387 2024-03-19T11:30:12.128 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.128 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.128 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.128 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-19T11:30:12.128 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:12.128 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.128 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:12.129 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6817", 2024-03-19T11:30:12.129 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2733060387 2024-03-19T11:30:12.129 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.129 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.129 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:12.129 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6820", 2024-03-19T11:30:12.129 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2733060387 2024-03-19T11:30:12.129 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.129 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.129 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.129 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.172:6811/2733060387", 2024-03-19T11:30:12.129 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.172:6815/2733060387", 2024-03-19T11:30:12.129 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.172:6823/2733060387", 2024-03-19T11:30:12.129 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.172:6820/2733060387", 2024-03-19T11:30:12.129 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-19T11:30:12.130 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-19T11:30:12.130 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-19T11:30:12.130 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.130 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.130 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.130 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 6, 2024-03-19T11:30:12.130 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "f91d8c47-4e10-40c6-a02d-66d6b72a657b", 2024-03-19T11:30:12.130 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-19T11:30:12.130 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-19T11:30:12.130 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-19T11:30:12.130 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-19T11:30:12.130 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-19T11:30:12.130 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-19T11:30:12.130 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 13, 2024-03-19T11:30:12.131 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 21, 2024-03-19T11:30:12.131 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-19T11:30:12.131 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-19T11:30:12.131 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-19T11:30:12.131 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:12.131 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.131 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:12.131 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6800", 2024-03-19T11:30:12.131 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 337743699 2024-03-19T11:30:12.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.131 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.131 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:12.131 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6801", 2024-03-19T11:30:12.131 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 337743699 2024-03-19T11:30:12.132 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.132 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.132 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.132 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-19T11:30:12.132 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:12.132 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.132 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:12.132 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6802", 2024-03-19T11:30:12.132 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 337743699 2024-03-19T11:30:12.132 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.132 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.132 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:12.132 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6803", 2024-03-19T11:30:12.133 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 337743699 2024-03-19T11:30:12.133 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.133 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.133 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.133 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-19T11:30:12.133 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:12.133 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.133 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:12.133 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6806", 2024-03-19T11:30:12.133 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 337743699 2024-03-19T11:30:12.133 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.133 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.133 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:12.133 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6807", 2024-03-19T11:30:12.134 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 337743699 2024-03-19T11:30:12.134 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.134 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.134 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.134 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-19T11:30:12.134 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:12.134 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.134 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:12.134 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6804", 2024-03-19T11:30:12.134 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 337743699 2024-03-19T11:30:12.134 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.134 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.134 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:12.134 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6805", 2024-03-19T11:30:12.134 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 337743699 2024-03-19T11:30:12.135 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.135 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.135 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.135 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.172:6801/337743699", 2024-03-19T11:30:12.135 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.172:6803/337743699", 2024-03-19T11:30:12.135 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.172:6807/337743699", 2024-03-19T11:30:12.135 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.172:6805/337743699", 2024-03-19T11:30:12.135 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-19T11:30:12.135 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-19T11:30:12.135 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-19T11:30:12.135 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.135 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.135 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.135 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 7, 2024-03-19T11:30:12.135 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "46fc052a-5708-42ff-9d5a-bed7a0fc457a", 2024-03-19T11:30:12.136 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-19T11:30:12.136 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-19T11:30:12.136 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-19T11:30:12.136 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-19T11:30:12.136 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-19T11:30:12.136 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-19T11:30:12.136 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 14, 2024-03-19T11:30:12.136 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 16, 2024-03-19T11:30:12.136 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-19T11:30:12.136 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-19T11:30:12.136 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-19T11:30:12.136 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:12.136 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.136 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:12.137 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6824", 2024-03-19T11:30:12.137 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 4145361100 2024-03-19T11:30:12.137 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.137 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.137 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:12.137 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6825", 2024-03-19T11:30:12.137 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 4145361100 2024-03-19T11:30:12.137 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.137 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.137 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.137 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-19T11:30:12.137 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:12.137 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.137 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:12.138 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6826", 2024-03-19T11:30:12.138 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 4145361100 2024-03-19T11:30:12.138 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.138 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.138 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:12.138 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6827", 2024-03-19T11:30:12.138 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 4145361100 2024-03-19T11:30:12.138 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.138 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.138 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.138 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-19T11:30:12.138 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:12.138 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.138 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:12.139 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6830", 2024-03-19T11:30:12.139 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 4145361100 2024-03-19T11:30:12.139 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.139 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.139 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:12.139 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6831", 2024-03-19T11:30:12.139 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 4145361100 2024-03-19T11:30:12.139 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.139 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.139 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.139 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-19T11:30:12.139 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:30:12.139 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.140 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:30:12.140 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6828", 2024-03-19T11:30:12.140 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 4145361100 2024-03-19T11:30:12.140 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.140 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.140 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:30:12.140 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6829", 2024-03-19T11:30:12.140 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 4145361100 2024-03-19T11:30:12.140 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.140 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.140 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.140 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.172:6825/4145361100", 2024-03-19T11:30:12.140 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.172:6827/4145361100", 2024-03-19T11:30:12.140 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.172:6831/4145361100", 2024-03-19T11:30:12.140 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.172:6829/4145361100", 2024-03-19T11:30:12.141 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-19T11:30:12.141 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-19T11:30:12.141 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-19T11:30:12.141 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.141 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.141 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-19T11:30:12.141 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_xinfo": [ 2024-03-19T11:30:12.141 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.141 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 0, 2024-03-19T11:30:12.141 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-19T11:30:12.141 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-19T11:30:12.141 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-19T11:30:12.141 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-19T11:30:12.141 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-19T11:30:12.142 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-19T11:29:41.144572+0000", 2024-03-19T11:30:12.142 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-19T11:30:12.142 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.142 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.142 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 1, 2024-03-19T11:30:12.142 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-19T11:30:12.142 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-19T11:30:12.142 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-19T11:30:12.142 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-19T11:30:12.142 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-19T11:30:12.142 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-19T11:29:41.282904+0000", 2024-03-19T11:30:12.142 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-19T11:30:12.142 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.142 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.143 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 2, 2024-03-19T11:30:12.143 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-19T11:30:12.143 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-19T11:30:12.143 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-19T11:30:12.143 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-19T11:30:12.143 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-19T11:30:12.143 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-19T11:29:41.176062+0000", 2024-03-19T11:30:12.143 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-19T11:30:12.143 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.143 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.143 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 3, 2024-03-19T11:30:12.143 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-19T11:30:12.143 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-19T11:30:12.143 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-19T11:30:12.143 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-19T11:30:12.144 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-19T11:30:12.144 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-19T11:29:41.269303+0000", 2024-03-19T11:30:12.144 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-19T11:30:12.144 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.144 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.144 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 4, 2024-03-19T11:30:12.144 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-19T11:30:12.144 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-19T11:30:12.144 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-19T11:30:12.144 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-19T11:30:12.144 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-19T11:30:12.144 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-19T11:29:40.978689+0000", 2024-03-19T11:30:12.144 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-19T11:30:12.145 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.145 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.145 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 5, 2024-03-19T11:30:12.145 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-19T11:30:12.145 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-19T11:30:12.145 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-19T11:30:12.145 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-19T11:30:12.145 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-19T11:30:12.145 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-19T11:29:41.198288+0000", 2024-03-19T11:30:12.145 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-19T11:30:12.145 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.145 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.145 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 6, 2024-03-19T11:30:12.146 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-19T11:30:12.146 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-19T11:30:12.146 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-19T11:30:12.146 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-19T11:30:12.146 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-19T11:30:12.146 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-19T11:29:41.083084+0000", 2024-03-19T11:30:12.146 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-19T11:30:12.146 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.146 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.146 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 7, 2024-03-19T11:30:12.146 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-19T11:30:12.146 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-19T11:30:12.146 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-19T11:30:12.146 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-19T11:30:12.147 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-19T11:30:12.147 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-19T11:29:41.265929+0000", 2024-03-19T11:30:12.147 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-19T11:30:12.147 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.147 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-19T11:30:12.147 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_upmap": [], 2024-03-19T11:30:12.147 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_upmap_items": [ 2024-03-19T11:30:12.147 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.147 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "2.7", 2024-03-19T11:30:12.147 INFO:teuthology.orchestra.run.smithi038.stdout: "mappings": [ 2024-03-19T11:30:12.147 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.147 INFO:teuthology.orchestra.run.smithi038.stdout: "from": 6, 2024-03-19T11:30:12.147 INFO:teuthology.orchestra.run.smithi038.stdout: "to": 2 2024-03-19T11:30:12.147 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.147 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.148 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.148 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-19T11:30:12.148 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_upmap_primaries": [], 2024-03-19T11:30:12.148 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_temp": [], 2024-03-19T11:30:12.148 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_temp": [], 2024-03-19T11:30:12.148 INFO:teuthology.orchestra.run.smithi038.stdout: "blocklist": {}, 2024-03-19T11:30:12.148 INFO:teuthology.orchestra.run.smithi038.stdout: "range_blocklist": {}, 2024-03-19T11:30:12.148 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profiles": { 2024-03-19T11:30:12.148 INFO:teuthology.orchestra.run.smithi038.stdout: "default": { 2024-03-19T11:30:12.148 INFO:teuthology.orchestra.run.smithi038.stdout: "crush-failure-domain": "osd", 2024-03-19T11:30:12.148 INFO:teuthology.orchestra.run.smithi038.stdout: "k": "2", 2024-03-19T11:30:12.148 INFO:teuthology.orchestra.run.smithi038.stdout: "m": "1", 2024-03-19T11:30:12.148 INFO:teuthology.orchestra.run.smithi038.stdout: "plugin": "jerasure", 2024-03-19T11:30:12.148 INFO:teuthology.orchestra.run.smithi038.stdout: "technique": "reed_sol_van" 2024-03-19T11:30:12.148 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.149 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.149 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps_queue": [ 2024-03-19T11:30:12.149 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.149 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 4, 2024-03-19T11:30:12.149 INFO:teuthology.orchestra.run.smithi038.stdout: "snaps": [ 2024-03-19T11:30:12.149 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.149 INFO:teuthology.orchestra.run.smithi038.stdout: "begin": 3, 2024-03-19T11:30:12.149 INFO:teuthology.orchestra.run.smithi038.stdout: "length": 2 2024-03-19T11:30:12.149 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.149 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.149 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.149 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-19T11:30:12.149 INFO:teuthology.orchestra.run.smithi038.stdout: "new_removed_snaps": [], 2024-03-19T11:30:12.149 INFO:teuthology.orchestra.run.smithi038.stdout: "new_purged_snaps": [], 2024-03-19T11:30:12.149 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_node_flags": {}, 2024-03-19T11:30:12.150 INFO:teuthology.orchestra.run.smithi038.stdout: "device_class_flags": {}, 2024-03-19T11:30:12.150 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode": { 2024-03-19T11:30:12.150 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode_enabled": false, 2024-03-19T11:30:12.150 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_bucket_count": 0, 2024-03-19T11:30:12.150 INFO:teuthology.orchestra.run.smithi038.stdout: "degraded_stretch_mode": 0, 2024-03-19T11:30:12.150 INFO:teuthology.orchestra.run.smithi038.stdout: "recovering_stretch_mode": 0, 2024-03-19T11:30:12.150 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode_bucket": 0 2024-03-19T11:30:12.150 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.150 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:12.152 INFO:tasks.thrashosds.thrasher:j is {'epoch': 34, 'fsid': '246bcf68-c431-4aa7-b787-5e9b2d9519d6', 'created': '2024-03-19T11:29:33.327073+0000', 'modified': '2024-03-19T11:30:11.730797+0000', 'last_up_change': '2024-03-19T11:29:43.166618+0000', 'last_in_change': '2024-03-19T11:29:35.632327+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 8, 'require_min_compat_client': 'reef', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T11:29:42.219654+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-03-19T11:29:44.428406+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'low_tier', 'create_time': '2024-03-19T11:29:55.608699+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 0.75, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'unique_pool_0', 'create_time': '2024-03-19T11:29:56.945423+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 5, 'snap_epoch': 34, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'dedup_cdc_chunk_size': 131072, 'dedup_chunk_algorithm': 'fastcdc', 'dedup_tier': 3, 'fingerprint_algorithm': 'sha256'}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '3bb2d2be-5253-4da0-acf3-85a114dc3801', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 21, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6808', 'nonce': 3148574891}, {'type': 'v1', 'addr': '172.21.15.38:6811', 'nonce': 3148574891}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6813', 'nonce': 3148574891}, {'type': 'v1', 'addr': '172.21.15.38:6815', 'nonce': 3148574891}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6822', 'nonce': 3148574891}, {'type': 'v1', 'addr': '172.21.15.38:6823', 'nonce': 3148574891}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6818', 'nonce': 3148574891}, {'type': 'v1', 'addr': '172.21.15.38:6821', 'nonce': 3148574891}]}, 'public_addr': '172.21.15.38:6811/3148574891', 'cluster_addr': '172.21.15.38:6815/3148574891', 'heartbeat_back_addr': '172.21.15.38:6823/3148574891', 'heartbeat_front_addr': '172.21.15.38:6821/3148574891', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'e1c926b2-e0be-4ccc-b8a5-d80d0b584ead', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 21, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6824', 'nonce': 675027622}, {'type': 'v1', 'addr': '172.21.15.38:6825', 'nonce': 675027622}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6826', 'nonce': 675027622}, {'type': 'v1', 'addr': '172.21.15.38:6827', 'nonce': 675027622}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6830', 'nonce': 675027622}, {'type': 'v1', 'addr': '172.21.15.38:6831', 'nonce': 675027622}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6828', 'nonce': 675027622}, {'type': 'v1', 'addr': '172.21.15.38:6829', 'nonce': 675027622}]}, 'public_addr': '172.21.15.38:6825/675027622', 'cluster_addr': '172.21.15.38:6827/675027622', 'heartbeat_back_addr': '172.21.15.38:6831/675027622', 'heartbeat_front_addr': '172.21.15.38:6829/675027622', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '61262219-4f8a-4733-af9d-b5208a17c0ea', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 25, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6800', 'nonce': 58279616}, {'type': 'v1', 'addr': '172.21.15.38:6801', 'nonce': 58279616}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6802', 'nonce': 58279616}, {'type': 'v1', 'addr': '172.21.15.38:6804', 'nonce': 58279616}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6814', 'nonce': 58279616}, {'type': 'v1', 'addr': '172.21.15.38:6817', 'nonce': 58279616}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6807', 'nonce': 58279616}, {'type': 'v1', 'addr': '172.21.15.38:6810', 'nonce': 58279616}]}, 'public_addr': '172.21.15.38:6801/58279616', 'cluster_addr': '172.21.15.38:6804/58279616', 'heartbeat_back_addr': '172.21.15.38:6817/58279616', 'heartbeat_front_addr': '172.21.15.38:6810/58279616', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'a31f8d1b-4e38-4ca3-b251-0877002d7235', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 21, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6803', 'nonce': 2942163952}, {'type': 'v1', 'addr': '172.21.15.38:6805', 'nonce': 2942163952}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6806', 'nonce': 2942163952}, {'type': 'v1', 'addr': '172.21.15.38:6809', 'nonce': 2942163952}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6819', 'nonce': 2942163952}, {'type': 'v1', 'addr': '172.21.15.38:6820', 'nonce': 2942163952}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6812', 'nonce': 2942163952}, {'type': 'v1', 'addr': '172.21.15.38:6816', 'nonce': 2942163952}]}, 'public_addr': '172.21.15.38:6805/2942163952', 'cluster_addr': '172.21.15.38:6809/2942163952', 'heartbeat_back_addr': '172.21.15.38:6820/2942163952', 'heartbeat_front_addr': '172.21.15.38:6816/2942163952', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6746723d-f59c-40ba-857e-d8c954d2675f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 21, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6808', 'nonce': 1353919713}, {'type': 'v1', 'addr': '172.21.15.172:6810', 'nonce': 1353919713}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6812', 'nonce': 1353919713}, {'type': 'v1', 'addr': '172.21.15.172:6814', 'nonce': 1353919713}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6819', 'nonce': 1353919713}, {'type': 'v1', 'addr': '172.21.15.172:6821', 'nonce': 1353919713}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6816', 'nonce': 1353919713}, {'type': 'v1', 'addr': '172.21.15.172:6818', 'nonce': 1353919713}]}, 'public_addr': '172.21.15.172:6810/1353919713', 'cluster_addr': '172.21.15.172:6814/1353919713', 'heartbeat_back_addr': '172.21.15.172:6821/1353919713', 'heartbeat_front_addr': '172.21.15.172:6818/1353919713', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'f59afdf8-2277-4583-af68-78136f1c5d62', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 21, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6809', 'nonce': 2733060387}, {'type': 'v1', 'addr': '172.21.15.172:6811', 'nonce': 2733060387}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6813', 'nonce': 2733060387}, {'type': 'v1', 'addr': '172.21.15.172:6815', 'nonce': 2733060387}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6822', 'nonce': 2733060387}, {'type': 'v1', 'addr': '172.21.15.172:6823', 'nonce': 2733060387}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6817', 'nonce': 2733060387}, {'type': 'v1', 'addr': '172.21.15.172:6820', 'nonce': 2733060387}]}, 'public_addr': '172.21.15.172:6811/2733060387', 'cluster_addr': '172.21.15.172:6815/2733060387', 'heartbeat_back_addr': '172.21.15.172:6823/2733060387', 'heartbeat_front_addr': '172.21.15.172:6820/2733060387', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'f91d8c47-4e10-40c6-a02d-66d6b72a657b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 21, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6800', 'nonce': 337743699}, {'type': 'v1', 'addr': '172.21.15.172:6801', 'nonce': 337743699}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6802', 'nonce': 337743699}, {'type': 'v1', 'addr': '172.21.15.172:6803', 'nonce': 337743699}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6806', 'nonce': 337743699}, {'type': 'v1', 'addr': '172.21.15.172:6807', 'nonce': 337743699}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6804', 'nonce': 337743699}, {'type': 'v1', 'addr': '172.21.15.172:6805', 'nonce': 337743699}]}, 'public_addr': '172.21.15.172:6801/337743699', 'cluster_addr': '172.21.15.172:6803/337743699', 'heartbeat_back_addr': '172.21.15.172:6807/337743699', 'heartbeat_front_addr': '172.21.15.172:6805/337743699', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '46fc052a-5708-42ff-9d5a-bed7a0fc457a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 16, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6824', 'nonce': 4145361100}, {'type': 'v1', 'addr': '172.21.15.172:6825', 'nonce': 4145361100}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6826', 'nonce': 4145361100}, {'type': 'v1', 'addr': '172.21.15.172:6827', 'nonce': 4145361100}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6830', 'nonce': 4145361100}, {'type': 'v1', 'addr': '172.21.15.172:6831', 'nonce': 4145361100}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6828', 'nonce': 4145361100}, {'type': 'v1', 'addr': '172.21.15.172:6829', 'nonce': 4145361100}]}, 'public_addr': '172.21.15.172:6825/4145361100', 'cluster_addr': '172.21.15.172:6827/4145361100', 'heartbeat_back_addr': '172.21.15.172:6831/4145361100', 'heartbeat_front_addr': '172.21.15.172:6829/4145361100', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-19T11:29:41.144572+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-19T11:29:41.282904+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-19T11:29:41.176062+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-19T11:29:41.269303+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-19T11:29:40.978689+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-19T11:29:41.198288+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-19T11:29:41.083084+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-19T11:29:41.265929+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.7', 'mappings': [{'from': 6, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 4, 'snaps': [{'begin': 3, 'length': 2}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-19T11:30:12.153 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T11:30:12.154 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:12.154 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:30:12.154 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:30:12.154 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:30:12.154 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:12.154 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-19T11:30:12.155 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:12.159+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:12.201 INFO:tasks.rados.rados.0.smithi038.stdout:680: finishing set_chunk to oid 30 2024-03-19T11:30:12.201 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 30 set_chunk returned error code -95 offset: 571904 length: 689152 2024-03-19T11:30:12.201 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 30 v 0 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-03-19T11:30:12.201 INFO:tasks.rados.rados.0.smithi038.stdout:681: finishing set_chunk to oid 24 2024-03-19T11:30:12.201 INFO:tasks.rados.rados.0.smithi038.stdout:682: expect (ObjNum 52 snap 0 seq_num 52) 2024-03-19T11:30:12.252 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:12.260+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:12.326 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:12.326 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:30:12.326 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:30:12.326 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:30:12.326 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.327 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:34 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'20, mlcod=15'16)", 2024-03-19T11:30:12.327 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.077542+0000", 2024-03-19T11:30:12.327 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 27.257562930999999, 2024-03-19T11:30:12.327 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0064305179999999997, 2024-03-19T11:30:12.327 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:12.327 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:12.327 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:12.327 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.327 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:12.327 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.077542+0000", 2024-03-19T11:30:12.327 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.327 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.327 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.327 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:12.327 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.077542+0000", 2024-03-19T11:30:12.328 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.328 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.328 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.328 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:12.328 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.077543+0000", 2024-03-19T11:30:12.328 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.387e-06 2024-03-19T11:30:12.328 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.328 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.328 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:12.328 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.077555+0000", 2024-03-19T11:30:12.328 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1759999999999999e-05 2024-03-19T11:30:12.328 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.328 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.328 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:12.328 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.077580+0000", 2024-03-19T11:30:12.329 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.529e-05 2024-03-19T11:30:12.329 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.329 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.329 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:12.329 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.077596+0000", 2024-03-19T11:30:12.329 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5466000000000001e-05 2024-03-19T11:30:12.329 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.329 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.329 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:12.329 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.077753+0000", 2024-03-19T11:30:12.329 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000157026 2024-03-19T11:30:12.329 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.329 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.329 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:12.330 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.077789+0000", 2024-03-19T11:30:12.330 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6259e-05 2024-03-19T11:30:12.330 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.330 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.330 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:12.330 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.083910+0000", 2024-03-19T11:30:12.330 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0061212080000000004 2024-03-19T11:30:12.330 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.330 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.330 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:12.330 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.083972+0000", 2024-03-19T11:30:12.330 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2121999999999999e-05 2024-03-19T11:30:12.330 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.330 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.331 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.331 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.331 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.331 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:114 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'76, mlcod=16'73)", 2024-03-19T11:30:12.331 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.353321+0000", 2024-03-19T11:30:12.331 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 26.981784078, 2024-03-19T11:30:12.331 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050714999999999996, 2024-03-19T11:30:12.331 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:12.331 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:12.331 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:12.331 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.331 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:12.331 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.353321+0000", 2024-03-19T11:30:12.331 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.331 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.332 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.332 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:12.332 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.353321+0000", 2024-03-19T11:30:12.332 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.332 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.332 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.332 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:12.332 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.353321+0000", 2024-03-19T11:30:12.332 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3399999999999998e-07 2024-03-19T11:30:12.332 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.332 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.332 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:12.332 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.353336+0000", 2024-03-19T11:30:12.332 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4154999999999999e-05 2024-03-19T11:30:12.332 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.333 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.333 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:12.333 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.353349+0000", 2024-03-19T11:30:12.333 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3644e-05 2024-03-19T11:30:12.333 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.333 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.333 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:12.333 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.353357+0000", 2024-03-19T11:30:12.333 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5109999999999997e-06 2024-03-19T11:30:12.333 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.333 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.333 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:12.333 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.353409+0000", 2024-03-19T11:30:12.333 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2723999999999999e-05 2024-03-19T11:30:12.333 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.334 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.334 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:12.334 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.353428+0000", 2024-03-19T11:30:12.334 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8216e-05 2024-03-19T11:30:12.334 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.334 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.334 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:12.334 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.358359+0000", 2024-03-19T11:30:12.334 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0049314550000000004 2024-03-19T11:30:12.334 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.334 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.334 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:12.334 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.358392+0000", 2024-03-19T11:30:12.335 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3061000000000001e-05 2024-03-19T11:30:12.335 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.335 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.335 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.335 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.335 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.335 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:119 1.0 e16/14 1:30306672:devicehealth::main.db.0000000000000000:head v 16'81, mlcod=16'78)", 2024-03-19T11:30:12.335 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.364626+0000", 2024-03-19T11:30:12.335 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 26.970479118, 2024-03-19T11:30:12.335 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046261449999999999, 2024-03-19T11:30:12.335 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:12.335 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:12.335 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:12.335 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.336 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:12.336 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.364626+0000", 2024-03-19T11:30:12.336 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.336 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.336 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.336 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:12.336 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.364626+0000", 2024-03-19T11:30:12.336 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.336 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.336 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.336 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:12.336 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.364626+0000", 2024-03-19T11:30:12.336 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1399999999999996e-07 2024-03-19T11:30:12.336 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.336 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.337 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:12.337 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.364724+0000", 2024-03-19T11:30:12.337 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7972000000000003e-05 2024-03-19T11:30:12.337 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.337 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.337 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:12.337 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.364737+0000", 2024-03-19T11:30:12.337 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2585000000000001e-05 2024-03-19T11:30:12.337 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.337 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.337 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:12.337 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.364744+0000", 2024-03-19T11:30:12.337 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7780000000000002e-06 2024-03-19T11:30:12.337 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.338 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.338 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:12.338 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.364780+0000", 2024-03-19T11:30:12.338 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5992999999999999e-05 2024-03-19T11:30:12.338 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.338 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.338 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:12.338 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.364795+0000", 2024-03-19T11:30:12.338 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4939e-05 2024-03-19T11:30:12.338 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.339 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.339 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:12.339 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.369211+0000", 2024-03-19T11:30:12.339 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0044161139999999996 2024-03-19T11:30:12.339 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.339 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.339 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:12.339 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.369252+0000", 2024-03-19T11:30:12.339 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1050000000000002e-05 2024-03-19T11:30:12.339 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.339 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.339 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.340 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.340 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.340 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:130 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'88, mlcod=16'85)", 2024-03-19T11:30:12.340 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.805586+0000", 2024-03-19T11:30:12.340 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 26.529519093000001, 2024-03-19T11:30:12.340 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0054104690000000002, 2024-03-19T11:30:12.340 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:12.340 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:12.340 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:12.341 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.341 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:12.341 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.805586+0000", 2024-03-19T11:30:12.341 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.341 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.341 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.341 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:12.341 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.805586+0000", 2024-03-19T11:30:12.341 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.341 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.341 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.341 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:12.341 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.805587+0000", 2024-03-19T11:30:12.342 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7269999999999999e-06 2024-03-19T11:30:12.342 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.342 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.342 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:12.342 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.805600+0000", 2024-03-19T11:30:12.342 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2293000000000001e-05 2024-03-19T11:30:12.342 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.342 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.342 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:12.342 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.805623+0000", 2024-03-19T11:30:12.343 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3592999999999999e-05 2024-03-19T11:30:12.343 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.343 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.343 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:12.343 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.805636+0000", 2024-03-19T11:30:12.343 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2660000000000001e-05 2024-03-19T11:30:12.343 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.343 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.343 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:12.343 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.805838+0000", 2024-03-19T11:30:12.344 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020228800000000001 2024-03-19T11:30:12.344 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.344 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.344 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:12.344 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.805878+0000", 2024-03-19T11:30:12.344 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.9586999999999999e-05 2024-03-19T11:30:12.344 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.344 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.344 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:12.344 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.810908+0000", 2024-03-19T11:30:12.344 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050303090000000002 2024-03-19T11:30:12.345 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.345 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.345 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:12.345 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.810996+0000", 2024-03-19T11:30:12.345 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8011999999999997e-05 2024-03-19T11:30:12.345 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.345 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.345 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.345 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.345 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.345 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:131 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'89, mlcod=16'85)", 2024-03-19T11:30:12.345 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.806096+0000", 2024-03-19T11:30:12.345 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 26.529008443999999, 2024-03-19T11:30:12.346 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0049616699999999996, 2024-03-19T11:30:12.346 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:12.346 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:12.346 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:12.346 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.346 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:12.346 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806096+0000", 2024-03-19T11:30:12.346 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.346 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.346 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.347 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:12.347 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806096+0000", 2024-03-19T11:30:12.347 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.347 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.347 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.347 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:12.347 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806098+0000", 2024-03-19T11:30:12.347 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.734e-06 2024-03-19T11:30:12.347 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.347 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.348 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:12.348 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806267+0000", 2024-03-19T11:30:12.348 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016936800000000001 2024-03-19T11:30:12.348 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.348 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.348 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:12.348 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806296+0000", 2024-03-19T11:30:12.348 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8653000000000002e-05 2024-03-19T11:30:12.348 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.348 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.349 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:12.349 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806315+0000", 2024-03-19T11:30:12.349 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9049999999999999e-05 2024-03-19T11:30:12.349 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.349 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.349 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:12.349 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806481+0000", 2024-03-19T11:30:12.349 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016540599999999999 2024-03-19T11:30:12.349 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.349 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.349 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:12.350 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806511+0000", 2024-03-19T11:30:12.350 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0626000000000001e-05 2024-03-19T11:30:12.350 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.350 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.350 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:12.350 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.811002+0000", 2024-03-19T11:30:12.350 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0044912399999999996 2024-03-19T11:30:12.350 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.350 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.351 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:12.351 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.811058+0000", 2024-03-19T11:30:12.351 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5593000000000001e-05 2024-03-19T11:30:12.351 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.351 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.351 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.351 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.351 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:132 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'90, mlcod=16'85)", 2024-03-19T11:30:12.351 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.806470+0000", 2024-03-19T11:30:12.351 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 26.528634953000001, 2024-03-19T11:30:12.351 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0066812390000000003, 2024-03-19T11:30:12.351 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:12.351 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:12.352 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:12.352 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.352 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:12.352 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806470+0000", 2024-03-19T11:30:12.352 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.352 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.352 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.352 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:12.352 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806470+0000", 2024-03-19T11:30:12.352 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.352 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.352 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.353 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:12.353 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806471+0000", 2024-03-19T11:30:12.353 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.486e-06 2024-03-19T11:30:12.353 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.353 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.353 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:12.353 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806485+0000", 2024-03-19T11:30:12.353 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3380000000000001e-05 2024-03-19T11:30:12.353 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.353 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.353 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:12.353 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806509+0000", 2024-03-19T11:30:12.353 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4617000000000002e-05 2024-03-19T11:30:12.353 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.354 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.354 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:12.354 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806524+0000", 2024-03-19T11:30:12.354 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4688999999999999e-05 2024-03-19T11:30:12.354 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.354 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.354 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:12.354 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807070+0000", 2024-03-19T11:30:12.354 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00054601300000000003 2024-03-19T11:30:12.354 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.354 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.354 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:12.354 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807104+0000", 2024-03-19T11:30:12.354 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3575999999999998e-05 2024-03-19T11:30:12.355 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.355 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.355 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:12.355 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813077+0000", 2024-03-19T11:30:12.355 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0059733590000000001 2024-03-19T11:30:12.355 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.355 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.355 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:12.355 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813151+0000", 2024-03-19T11:30:12.355 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4119e-05 2024-03-19T11:30:12.355 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.355 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.355 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.355 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.355 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.355 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:133 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'91, mlcod=16'85)", 2024-03-19T11:30:12.356 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.806875+0000", 2024-03-19T11:30:12.356 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 26.528229289999999, 2024-03-19T11:30:12.356 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0063196260000000001, 2024-03-19T11:30:12.356 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:12.356 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:12.356 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:12.356 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.356 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:12.356 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806875+0000", 2024-03-19T11:30:12.356 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.356 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.356 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.356 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:12.357 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806875+0000", 2024-03-19T11:30:12.357 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.357 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.357 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.357 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:12.357 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806877+0000", 2024-03-19T11:30:12.357 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8899999999999999e-06 2024-03-19T11:30:12.357 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.357 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.357 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:12.357 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806895+0000", 2024-03-19T11:30:12.357 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7980000000000001e-05 2024-03-19T11:30:12.357 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.357 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.357 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:12.358 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806926+0000", 2024-03-19T11:30:12.358 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0363000000000001e-05 2024-03-19T11:30:12.358 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.358 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.358 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:12.358 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806942+0000", 2024-03-19T11:30:12.358 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6422999999999998e-05 2024-03-19T11:30:12.358 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.358 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.358 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:12.358 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807659+0000", 2024-03-19T11:30:12.358 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00071724699999999996 2024-03-19T11:30:12.358 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.358 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.359 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:12.359 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807688+0000", 2024-03-19T11:30:12.359 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8940000000000001e-05 2024-03-19T11:30:12.359 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.359 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.359 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:12.359 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813157+0000", 2024-03-19T11:30:12.359 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0054689930000000001 2024-03-19T11:30:12.359 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.359 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.359 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:12.359 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813195+0000", 2024-03-19T11:30:12.359 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7790000000000002e-05 2024-03-19T11:30:12.360 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.360 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.360 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.360 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.360 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.360 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:134 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'92, mlcod=16'85)", 2024-03-19T11:30:12.360 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.807418+0000", 2024-03-19T11:30:12.360 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 26.527686767999999, 2024-03-19T11:30:12.360 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0058240560000000002, 2024-03-19T11:30:12.360 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:12.360 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:12.360 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:12.360 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.360 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:12.360 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807418+0000", 2024-03-19T11:30:12.361 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.361 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.361 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.361 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:12.361 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807418+0000", 2024-03-19T11:30:12.361 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.361 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.361 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.361 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:12.361 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807420+0000", 2024-03-19T11:30:12.361 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5600000000000001e-06 2024-03-19T11:30:12.361 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.361 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.362 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:12.362 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807514+0000", 2024-03-19T11:30:12.362 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4256e-05 2024-03-19T11:30:12.362 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.362 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.362 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:12.362 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807542+0000", 2024-03-19T11:30:12.362 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8283e-05 2024-03-19T11:30:12.362 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.362 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.362 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:12.362 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807562+0000", 2024-03-19T11:30:12.362 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0126999999999999e-05 2024-03-19T11:30:12.362 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.362 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.363 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:12.363 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808272+0000", 2024-03-19T11:30:12.363 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00071012400000000002 2024-03-19T11:30:12.363 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.363 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.363 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:12.363 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808308+0000", 2024-03-19T11:30:12.363 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5722999999999997e-05 2024-03-19T11:30:12.363 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.363 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.363 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:12.363 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813200+0000", 2024-03-19T11:30:12.363 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0048915010000000004 2024-03-19T11:30:12.363 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.363 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.364 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:12.364 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813242+0000", 2024-03-19T11:30:12.364 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2481999999999997e-05 2024-03-19T11:30:12.364 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.364 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.364 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.364 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.364 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.364 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:135 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'93, mlcod=16'85)", 2024-03-19T11:30:12.364 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.807817+0000", 2024-03-19T11:30:12.364 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 26.527287530999999, 2024-03-19T11:30:12.364 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0054653059999999996, 2024-03-19T11:30:12.364 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:12.364 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:12.364 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:12.365 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.365 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:12.365 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807817+0000", 2024-03-19T11:30:12.365 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.365 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.365 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.365 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:12.365 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807817+0000", 2024-03-19T11:30:12.365 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.365 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.365 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.365 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:12.365 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807819+0000", 2024-03-19T11:30:12.365 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7999999999999999e-06 2024-03-19T11:30:12.365 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.366 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.366 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:12.366 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807833+0000", 2024-03-19T11:30:12.366 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3546999999999999e-05 2024-03-19T11:30:12.366 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.366 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.366 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:12.366 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807858+0000", 2024-03-19T11:30:12.366 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5119000000000001e-05 2024-03-19T11:30:12.366 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.366 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.366 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:12.366 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807871+0000", 2024-03-19T11:30:12.366 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.287e-05 2024-03-19T11:30:12.367 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.367 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.367 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:12.367 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808989+0000", 2024-03-19T11:30:12.367 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001118841 2024-03-19T11:30:12.367 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.367 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.367 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:12.367 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.809022+0000", 2024-03-19T11:30:12.367 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3000000000000003e-05 2024-03-19T11:30:12.367 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.367 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.367 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:12.367 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813249+0000", 2024-03-19T11:30:12.367 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0042268260000000004 2024-03-19T11:30:12.367 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.368 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.368 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:12.368 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813283+0000", 2024-03-19T11:30:12.368 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3303000000000003e-05 2024-03-19T11:30:12.368 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.368 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.368 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.368 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.368 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.368 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:136 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'94, mlcod=16'85)", 2024-03-19T11:30:12.368 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.808219+0000", 2024-03-19T11:30:12.368 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 26.526886251000001, 2024-03-19T11:30:12.368 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005105086, 2024-03-19T11:30:12.368 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:12.368 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:12.369 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:12.369 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.369 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:12.369 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808219+0000", 2024-03-19T11:30:12.369 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.369 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.369 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.369 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:12.369 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808219+0000", 2024-03-19T11:30:12.369 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.369 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.369 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.369 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:12.369 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808220+0000", 2024-03-19T11:30:12.369 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7600000000000001e-06 2024-03-19T11:30:12.370 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.370 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.370 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:12.370 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808233+0000", 2024-03-19T11:30:12.370 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2916e-05 2024-03-19T11:30:12.370 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.370 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.370 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:12.370 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808294+0000", 2024-03-19T11:30:12.370 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1199999999999997e-05 2024-03-19T11:30:12.370 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.370 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.370 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:12.370 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808310+0000", 2024-03-19T11:30:12.371 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5996999999999999e-05 2024-03-19T11:30:12.371 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.371 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.371 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:12.371 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.809445+0000", 2024-03-19T11:30:12.371 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001134227 2024-03-19T11:30:12.371 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.371 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.371 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:12.371 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.809473+0000", 2024-03-19T11:30:12.371 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8603e-05 2024-03-19T11:30:12.371 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.371 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.371 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:12.372 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813291+0000", 2024-03-19T11:30:12.372 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038174799999999998 2024-03-19T11:30:12.372 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.372 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.372 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:12.372 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813324+0000", 2024-03-19T11:30:12.372 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2903e-05 2024-03-19T11:30:12.372 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.372 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.372 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.373 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.373 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.373 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:137 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'95, mlcod=16'85)", 2024-03-19T11:30:12.373 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.808663+0000", 2024-03-19T11:30:12.373 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 26.526442098, 2024-03-19T11:30:12.373 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0047004350000000002, 2024-03-19T11:30:12.373 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:12.373 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:12.373 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:12.373 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.373 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:12.373 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808663+0000", 2024-03-19T11:30:12.373 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.373 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.373 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.374 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:12.374 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808663+0000", 2024-03-19T11:30:12.374 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.374 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.374 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.374 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:12.374 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808665+0000", 2024-03-19T11:30:12.374 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9300000000000002e-06 2024-03-19T11:30:12.374 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.374 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.375 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:12.375 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808768+0000", 2024-03-19T11:30:12.375 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000103009 2024-03-19T11:30:12.375 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.375 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.375 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:12.375 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808792+0000", 2024-03-19T11:30:12.375 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3972999999999999e-05 2024-03-19T11:30:12.375 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.375 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.375 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:12.375 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.808807+0000", 2024-03-19T11:30:12.376 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5614e-05 2024-03-19T11:30:12.376 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.376 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.376 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:12.376 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.809896+0000", 2024-03-19T11:30:12.376 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0010891270000000001 2024-03-19T11:30:12.376 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.376 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.376 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:12.376 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.809930+0000", 2024-03-19T11:30:12.376 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3444e-05 2024-03-19T11:30:12.377 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.377 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.377 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:12.377 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813330+0000", 2024-03-19T11:30:12.377 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003399839 2024-03-19T11:30:12.377 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.377 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.377 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:12.377 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813363+0000", 2024-03-19T11:30:12.377 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3498999999999999e-05 2024-03-19T11:30:12.377 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.377 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.377 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.378 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.378 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.378 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:5 4.6 4:626acd4d:::smithi03829856-2:head [write 416807~475249,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-03-19T11:30:12.378 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.688442+0000", 2024-03-19T11:30:12.378 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 7.646662439, 2024-03-19T11:30:12.378 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.004734121, 2024-03-19T11:30:12.378 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:12.378 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:12.378 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:12.378 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:12.378 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:12.379 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 5 2024-03-19T11:30:12.379 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.379 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:12.379 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.379 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:12.379 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.688442+0000", 2024-03-19T11:30:12.379 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.379 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.379 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.379 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:12.379 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.688442+0000", 2024-03-19T11:30:12.379 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.379 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.379 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.380 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:12.380 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.688445+0000", 2024-03-19T11:30:12.380 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1629999999999999e-06 2024-03-19T11:30:12.380 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.380 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.380 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:12.380 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.688905+0000", 2024-03-19T11:30:12.380 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000460536 2024-03-19T11:30:12.380 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.380 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.380 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:12.381 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.688925+0000", 2024-03-19T11:30:12.381 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0404999999999999e-05 2024-03-19T11:30:12.381 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.381 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.381 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:12.381 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.688940+0000", 2024-03-19T11:30:12.381 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4792e-05 2024-03-19T11:30:12.381 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.381 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.381 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:12.382 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.689031+0000", 2024-03-19T11:30:12.382 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0618000000000006e-05 2024-03-19T11:30:12.382 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.382 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.382 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:12.382 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.689209+0000", 2024-03-19T11:30:12.382 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000177868 2024-03-19T11:30:12.382 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.382 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.382 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:12.382 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.689380+0000", 2024-03-19T11:30:12.382 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017090700000000001 2024-03-19T11:30:12.383 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.383 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.383 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:12.383 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.692293+0000", 2024-03-19T11:30:12.383 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002913728 2024-03-19T11:30:12.383 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.383 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.383 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:12.383 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.693042+0000", 2024-03-19T11:30:12.383 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00074911300000000004 2024-03-19T11:30:12.383 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.384 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.384 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:12.384 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.693112+0000", 2024-03-19T11:30:12.384 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9846999999999997e-05 2024-03-19T11:30:12.384 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.384 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.384 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:12.384 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.693176+0000", 2024-03-19T11:30:12.384 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4144000000000001e-05 2024-03-19T11:30:12.384 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.385 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.385 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.385 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.385 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.385 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:6 4.6 4:626acd4d:::smithi03829856-2:head [write 1542129~474678] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-03-19T11:30:12.385 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.696120+0000", 2024-03-19T11:30:12.385 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 7.6389849439999997, 2024-03-19T11:30:12.385 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0082896359999999995, 2024-03-19T11:30:12.385 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:12.385 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:12.385 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:12.385 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:12.385 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:12.385 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 6 2024-03-19T11:30:12.385 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.386 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:12.386 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.386 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:12.386 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696120+0000", 2024-03-19T11:30:12.386 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.386 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.386 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.386 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:12.386 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696120+0000", 2024-03-19T11:30:12.386 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.386 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.386 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.386 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:12.386 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696123+0000", 2024-03-19T11:30:12.387 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.661e-06 2024-03-19T11:30:12.387 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.387 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.387 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:12.387 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696282+0000", 2024-03-19T11:30:12.387 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015960400000000001 2024-03-19T11:30:12.387 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.387 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.387 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:12.387 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696307+0000", 2024-03-19T11:30:12.387 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5352999999999999e-05 2024-03-19T11:30:12.387 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.387 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.387 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:12.387 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696320+0000", 2024-03-19T11:30:12.388 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2847000000000001e-05 2024-03-19T11:30:12.388 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.388 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.388 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:12.388 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696403+0000", 2024-03-19T11:30:12.388 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2507000000000001e-05 2024-03-19T11:30:12.388 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.388 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.388 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:12.388 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696585+0000", 2024-03-19T11:30:12.388 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00018234999999999999 2024-03-19T11:30:12.388 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.388 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.388 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:12.388 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696770+0000", 2024-03-19T11:30:12.389 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000184377 2024-03-19T11:30:12.389 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.389 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.389 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:12.389 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.699608+0000", 2024-03-19T11:30:12.389 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00283797 2024-03-19T11:30:12.389 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.389 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.389 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:12.389 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.704252+0000", 2024-03-19T11:30:12.389 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046443980000000001 2024-03-19T11:30:12.389 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.389 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.389 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:12.390 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.704349+0000", 2024-03-19T11:30:12.390 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7388999999999999e-05 2024-03-19T11:30:12.390 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.390 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.390 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:12.390 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.704409+0000", 2024-03-19T11:30:12.390 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0180000000000003e-05 2024-03-19T11:30:12.390 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.390 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.390 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.390 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.390 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.390 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:7 4.6 4:626acd4d:::smithi03829856-2:head [setxattr _header (55) in=7b,truncate 2016807] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-03-19T11:30:12.390 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.696352+0000", 2024-03-19T11:30:12.391 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 7.6387528040000001, 2024-03-19T11:30:12.391 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0084077100000000005, 2024-03-19T11:30:12.391 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:12.391 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:12.391 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:12.391 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:12.391 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:12.391 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 7 2024-03-19T11:30:12.391 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.391 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:12.391 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.391 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:12.391 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696352+0000", 2024-03-19T11:30:12.391 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.392 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.392 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.392 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:12.392 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696352+0000", 2024-03-19T11:30:12.392 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.392 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.392 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.392 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:12.392 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696353+0000", 2024-03-19T11:30:12.392 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1399999999999996e-07 2024-03-19T11:30:12.392 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.392 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.392 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:12.392 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696358+0000", 2024-03-19T11:30:12.392 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5300000000000004e-06 2024-03-19T11:30:12.393 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.393 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.393 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:12.393 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696368+0000", 2024-03-19T11:30:12.393 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6209999999999999e-06 2024-03-19T11:30:12.393 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.393 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.393 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:12.393 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696374+0000", 2024-03-19T11:30:12.393 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5599999999999999e-06 2024-03-19T11:30:12.393 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.393 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.393 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:12.393 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.698823+0000", 2024-03-19T11:30:12.393 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0024489899999999998 2024-03-19T11:30:12.394 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.394 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.394 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:12.394 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.698925+0000", 2024-03-19T11:30:12.394 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010166399999999999 2024-03-19T11:30:12.394 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.394 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.394 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:12.394 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.699071+0000", 2024-03-19T11:30:12.394 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000146014 2024-03-19T11:30:12.394 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.394 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.394 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:12.394 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.703885+0000", 2024-03-19T11:30:12.395 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.004813866 2024-03-19T11:30:12.395 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.395 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.395 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:12.395 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.704622+0000", 2024-03-19T11:30:12.395 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00073729600000000002 2024-03-19T11:30:12.395 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.395 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.395 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:12.395 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.704688+0000", 2024-03-19T11:30:12.395 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6192000000000006e-05 2024-03-19T11:30:12.395 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.395 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.395 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:12.395 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.704760+0000", 2024-03-19T11:30:12.396 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1262999999999999e-05 2024-03-19T11:30:12.396 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.396 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.396 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.396 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.396 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.396 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:8 4.6 4:626acd4d:::smithi03829856-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-03-19T11:30:12.396 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.696385+0000", 2024-03-19T11:30:12.396 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 7.6387195050000001, 2024-03-19T11:30:12.396 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0086443730000000003, 2024-03-19T11:30:12.396 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:12.396 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:12.396 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:12.396 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:12.396 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:12.397 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 8 2024-03-19T11:30:12.397 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.397 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:12.397 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.397 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:12.397 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696385+0000", 2024-03-19T11:30:12.397 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.397 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.397 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.397 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:12.397 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696385+0000", 2024-03-19T11:30:12.397 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.397 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.397 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.397 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:12.398 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696386+0000", 2024-03-19T11:30:12.398 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3e-07 2024-03-19T11:30:12.398 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.398 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.398 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:12.398 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696390+0000", 2024-03-19T11:30:12.398 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0130000000000004e-06 2024-03-19T11:30:12.398 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.398 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.398 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:12.398 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696399+0000", 2024-03-19T11:30:12.398 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9139999999999997e-06 2024-03-19T11:30:12.398 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.398 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.398 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:12.399 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696405+0000", 2024-03-19T11:30:12.399 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4699999999999999e-06 2024-03-19T11:30:12.399 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.399 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.399 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:12.399 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.699495+0000", 2024-03-19T11:30:12.399 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030890969999999998 2024-03-19T11:30:12.399 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.399 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.399 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-03-19T11:30:12.399 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.699562+0000", 2024-03-19T11:30:12.399 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7980999999999998e-05 2024-03-19T11:30:12.399 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.399 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.400 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:12.400 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.704809+0000", 2024-03-19T11:30:12.400 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0052468769999999996 2024-03-19T11:30:12.400 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.400 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.400 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:12.400 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.704909+0000", 2024-03-19T11:30:12.400 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9189000000000002e-05 2024-03-19T11:30:12.400 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.400 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.400 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:12.400 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.705030+0000", 2024-03-19T11:30:12.400 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012110199999999999 2024-03-19T11:30:12.400 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.400 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.401 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.401 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.401 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.401 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:21 4.b 4:db322469:::smithi03829856-6:head [write 484035~776245,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:30:12.401 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.746924+0000", 2024-03-19T11:30:12.401 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 7.5881807610000003, 2024-03-19T11:30:12.401 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074997270000000003, 2024-03-19T11:30:12.401 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:12.401 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:12.401 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:12.401 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:12.401 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:12.401 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 21 2024-03-19T11:30:12.401 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.402 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:12.402 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.402 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:12.402 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.746924+0000", 2024-03-19T11:30:12.402 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.402 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.402 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.402 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:12.402 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.746924+0000", 2024-03-19T11:30:12.402 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.402 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.402 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.402 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:12.402 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.746927+0000", 2024-03-19T11:30:12.403 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4820000000000001e-06 2024-03-19T11:30:12.403 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.403 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.403 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:12.403 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.747304+0000", 2024-03-19T11:30:12.403 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037742299999999999 2024-03-19T11:30:12.403 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.403 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.403 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:12.403 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.747326+0000", 2024-03-19T11:30:12.403 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1648999999999999e-05 2024-03-19T11:30:12.403 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.403 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.403 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:12.403 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.747336+0000", 2024-03-19T11:30:12.404 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0288e-05 2024-03-19T11:30:12.404 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.404 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.404 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:12.404 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.747414+0000", 2024-03-19T11:30:12.404 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8085999999999997e-05 2024-03-19T11:30:12.404 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.404 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.404 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-03-19T11:30:12.404 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.747421+0000", 2024-03-19T11:30:12.404 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9609999999999999e-06 2024-03-19T11:30:12.404 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.404 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.404 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:12.405 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.749216+0000", 2024-03-19T11:30:12.405 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001795172 2024-03-19T11:30:12.405 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.405 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.405 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:12.405 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.749355+0000", 2024-03-19T11:30:12.405 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013904799999999999 2024-03-19T11:30:12.405 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.405 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.405 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:12.405 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.749513+0000", 2024-03-19T11:30:12.405 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000157453 2024-03-19T11:30:12.405 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.405 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.405 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:12.406 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.754081+0000", 2024-03-19T11:30:12.406 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0045681230000000003 2024-03-19T11:30:12.406 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.406 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.406 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:12.406 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.754323+0000", 2024-03-19T11:30:12.406 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00024241900000000001 2024-03-19T11:30:12.406 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.406 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.406 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:12.407 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.754373+0000", 2024-03-19T11:30:12.407 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9632999999999997e-05 2024-03-19T11:30:12.407 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.407 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.407 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:12.407 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.754424+0000", 2024-03-19T11:30:12.407 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0989999999999998e-05 2024-03-19T11:30:12.407 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.407 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.407 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.407 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.407 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.407 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:25 4.3 4:cef44a0d:::smithi03829856-7:head [write 500842~451494,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:30:12.408 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.757843+0000", 2024-03-19T11:30:12.408 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 7.5772616920000004, 2024-03-19T11:30:12.408 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0059188269999999998, 2024-03-19T11:30:12.408 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:12.408 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:12.408 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:12.408 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:12.408 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:12.408 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 25 2024-03-19T11:30:12.408 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.408 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:12.408 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.408 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:12.408 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.757843+0000", 2024-03-19T11:30:12.408 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.409 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.409 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.409 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:12.409 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.757843+0000", 2024-03-19T11:30:12.409 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.409 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.409 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.409 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:12.409 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.757845+0000", 2024-03-19T11:30:12.409 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.424e-06 2024-03-19T11:30:12.409 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.409 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.409 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:12.409 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.757987+0000", 2024-03-19T11:30:12.409 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014264400000000001 2024-03-19T11:30:12.410 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.410 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.410 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:12.410 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.758005+0000", 2024-03-19T11:30:12.410 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7629e-05 2024-03-19T11:30:12.410 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.410 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.410 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:12.410 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.758014+0000", 2024-03-19T11:30:12.410 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9500000000000007e-06 2024-03-19T11:30:12.410 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.410 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.410 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:12.410 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.758080+0000", 2024-03-19T11:30:12.410 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6122000000000002e-05 2024-03-19T11:30:12.411 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.411 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.411 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:12.411 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.758217+0000", 2024-03-19T11:30:12.411 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013761000000000001 2024-03-19T11:30:12.411 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.411 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.411 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:12.411 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.758354+0000", 2024-03-19T11:30:12.412 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000136706 2024-03-19T11:30:12.412 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.412 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.412 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:12.412 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.760460+0000", 2024-03-19T11:30:12.412 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002106199 2024-03-19T11:30:12.412 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.412 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.412 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:12.412 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.763649+0000", 2024-03-19T11:30:12.412 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031888630000000001 2024-03-19T11:30:12.412 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.412 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.412 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:12.412 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.763707+0000", 2024-03-19T11:30:12.413 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7773999999999999e-05 2024-03-19T11:30:12.413 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.413 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.413 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:12.413 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.763762+0000", 2024-03-19T11:30:12.413 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4905999999999999e-05 2024-03-19T11:30:12.413 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.413 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.413 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.413 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.414 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.414 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:889 3.1 3:99a49d1b:::895a63407b6109b3cca9a6f5ef827134d56679bef42daadc5892269682c428ed:head [write 0~781312] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:12.414 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.277573+0000", 2024-03-19T11:30:12.414 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.0575318870000001, 2024-03-19T11:30:12.414 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074918249999999997, 2024-03-19T11:30:12.414 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:12.414 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:12.414 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:12.414 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:12.414 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:12.414 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 889 2024-03-19T11:30:12.414 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.414 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:12.414 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.415 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:12.415 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.277573+0000", 2024-03-19T11:30:12.415 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.415 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.415 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.415 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:12.415 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.277573+0000", 2024-03-19T11:30:12.415 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.415 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.415 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.416 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:12.416 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.277576+0000", 2024-03-19T11:30:12.416 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0869999999999998e-06 2024-03-19T11:30:12.416 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.416 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.416 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:12.416 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.278553+0000", 2024-03-19T11:30:12.416 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0009769990000000001 2024-03-19T11:30:12.416 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.416 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.416 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:12.416 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.278604+0000", 2024-03-19T11:30:12.417 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0679e-05 2024-03-19T11:30:12.417 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.417 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.417 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:12.417 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.278629+0000", 2024-03-19T11:30:12.417 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5673e-05 2024-03-19T11:30:12.417 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.417 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.417 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:12.417 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.278741+0000", 2024-03-19T11:30:12.417 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000112151 2024-03-19T11:30:12.417 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.417 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.418 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:12.418 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.278925+0000", 2024-03-19T11:30:12.418 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000183673 2024-03-19T11:30:12.418 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.418 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.418 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4", 2024-03-19T11:30:12.418 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.279096+0000", 2024-03-19T11:30:12.418 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017097 2024-03-19T11:30:12.418 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.418 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.418 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:12.419 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.281583+0000", 2024-03-19T11:30:12.419 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0024863720000000001 2024-03-19T11:30:12.419 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.419 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.419 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:12.419 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.284892+0000", 2024-03-19T11:30:12.419 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033099359999999999 2024-03-19T11:30:12.419 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.420 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.420 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:12.420 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.284988+0000", 2024-03-19T11:30:12.420 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.5159000000000002e-05 2024-03-19T11:30:12.420 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.420 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.420 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:12.420 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.285065+0000", 2024-03-19T11:30:12.420 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7126000000000007e-05 2024-03-19T11:30:12.421 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.421 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.421 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.421 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.421 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.421 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:985 4.b 4:d0dfa670:::smithi03829856-42:head [sparse-read 0~1561574] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e34)", 2024-03-19T11:30:12.421 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:11.768218+0000", 2024-03-19T11:30:12.421 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.56688715700000003, 2024-03-19T11:30:12.421 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0043794639999999996, 2024-03-19T11:30:12.421 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:12.421 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:12.421 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:12.422 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:12.422 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:12.422 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 985 2024-03-19T11:30:12.422 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.422 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:12.422 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.422 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:12.422 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:11.768218+0000", 2024-03-19T11:30:12.422 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.422 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.422 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.422 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:12.422 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:11.768218+0000", 2024-03-19T11:30:12.422 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.423 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.423 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.423 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:12.423 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:11.768218+0000", 2024-03-19T11:30:12.423 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4799999999999999e-07 2024-03-19T11:30:12.423 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.423 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.423 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:12.423 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:11.768221+0000", 2024-03-19T11:30:12.423 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5110000000000002e-06 2024-03-19T11:30:12.423 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.423 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.423 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:12.423 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:11.768227+0000", 2024-03-19T11:30:12.424 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6889999999999998e-06 2024-03-19T11:30:12.424 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.424 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.424 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:12.424 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:11.768233+0000", 2024-03-19T11:30:12.424 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3519999999999997e-06 2024-03-19T11:30:12.424 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.424 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.424 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:12.424 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:11.772156+0000", 2024-03-19T11:30:12.424 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0039229039999999996 2024-03-19T11:30:12.424 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.424 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.424 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:12.424 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:11.772243+0000", 2024-03-19T11:30:12.425 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7142e-05 2024-03-19T11:30:12.425 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.425 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.425 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:12.425 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:11.772597+0000", 2024-03-19T11:30:12.425 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00035441800000000001 2024-03-19T11:30:12.425 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.425 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.425 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.425 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.425 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.425 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:986 4.b 4:d0dfa670:::smithi03829856-42:head [sparse-read 0~1561574] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e34)", 2024-03-19T11:30:12.425 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:11.768241+0000", 2024-03-19T11:30:12.425 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.56686370200000002, 2024-03-19T11:30:12.425 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0047664430000000004, 2024-03-19T11:30:12.426 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:12.426 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:12.426 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:12.426 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:12.426 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:12.426 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 986 2024-03-19T11:30:12.426 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.426 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:12.426 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.426 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:12.426 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:11.768241+0000", 2024-03-19T11:30:12.426 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.426 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.426 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.427 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:12.427 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:11.768241+0000", 2024-03-19T11:30:12.427 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.427 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.427 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.427 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:12.427 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:11.768241+0000", 2024-03-19T11:30:12.427 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0999999999999999e-07 2024-03-19T11:30:12.427 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.427 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.427 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:12.427 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:11.768244+0000", 2024-03-19T11:30:12.427 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3329999999999999e-06 2024-03-19T11:30:12.427 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.427 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.428 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:12.428 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:11.768250+0000", 2024-03-19T11:30:12.428 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2570000000000004e-06 2024-03-19T11:30:12.428 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.428 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.428 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:12.428 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:11.768255+0000", 2024-03-19T11:30:12.428 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7620000000000003e-06 2024-03-19T11:30:12.428 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.428 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.428 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:12.428 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:11.772640+0000", 2024-03-19T11:30:12.428 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0043846620000000001 2024-03-19T11:30:12.428 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.428 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.429 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:12.429 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:11.772728+0000", 2024-03-19T11:30:12.429 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8098999999999998e-05 2024-03-19T11:30:12.429 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.429 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.429 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:12.429 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:11.773008+0000", 2024-03-19T11:30:12.429 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00027992000000000002 2024-03-19T11:30:12.429 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.429 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.429 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.429 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.429 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.429 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:12.431 INFO:tasks.rados.rados.0.smithi038.stdout:674: done (9 left) 2024-03-19T11:30:12.431 INFO:tasks.rados.rados.0.smithi038.stdout:675: done (8 left) 2024-03-19T11:30:12.431 INFO:tasks.rados.rados.0.smithi038.stdout:676: done (7 left) 2024-03-19T11:30:12.431 INFO:tasks.rados.rados.0.smithi038.stdout:677: done (6 left) 2024-03-19T11:30:12.431 INFO:tasks.rados.rados.0.smithi038.stdout:678: done (5 left) 2024-03-19T11:30:12.431 INFO:tasks.rados.rados.0.smithi038.stdout:679: done (4 left) 2024-03-19T11:30:12.431 INFO:tasks.rados.rados.0.smithi038.stdout:680: done (3 left) 2024-03-19T11:30:12.431 INFO:tasks.rados.rados.0.smithi038.stdout:681: done (2 left) 2024-03-19T11:30:12.431 INFO:tasks.rados.rados.0.smithi038.stdout:682: done (1 left) 2024-03-19T11:30:12.432 INFO:tasks.rados.rados.0.smithi038.stdout:684: read oid 8 snap -1 2024-03-19T11:30:12.432 INFO:tasks.rados.rados.0.smithi038.stdout:684: expect (ObjNum 60 snap 3 seq_num 60) 2024-03-19T11:30:12.432 INFO:tasks.rados.rados.0.smithi038.stdout:685: write oid 39 current snap is 4 2024-03-19T11:30:12.432 INFO:tasks.rados.rados.0.smithi038.stdout:685: seq_num 70 ranges {776490=714607,2246563=579356} 2024-03-19T11:30:12.433 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-19T11:30:12.434 INFO:tasks.rados.rados.0.smithi038.stdout:685: writing smithi03829856-39 from 776490 to 1491097 tid 1 2024-03-19T11:30:12.434 INFO:tasks.rados.rados.0.smithi038.stdout:685: writing smithi03829856-39 from 2246563 to 2825919 tid 2 2024-03-19T11:30:12.434 INFO:tasks.rados.rados.0.smithi038.stdout:686: rollback oid 46 current snap is 4 2024-03-19T11:30:12.434 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 46 to 0 2024-03-19T11:30:12.434 INFO:tasks.rados.rados.0.smithi038.stdout:687: read oid 48 snap -1 2024-03-19T11:30:12.434 INFO:tasks.rados.rados.0.smithi038.stdout:687: expect (ObjNum 66 snap 3 seq_num 66) 2024-03-19T11:30:12.434 INFO:tasks.rados.rados.0.smithi038.stdout:688: write (excl) oid 41 current snap is 4 2024-03-19T11:30:12.434 INFO:tasks.rados.rados.0.smithi038.stdout:688: seq_num 71 ranges {793297=789856,2305045=723014} 2024-03-19T11:30:12.434 INFO:tasks.rados.rados.0.smithi038.stdout:688: writing smithi03829856-41 from 793297 to 1583153 tid 1 2024-03-19T11:30:12.434 INFO:tasks.rados.rados.0.smithi038.stdout:688: writing smithi03829856-41 from 2305045 to 3028059 tid 2 2024-03-19T11:30:12.435 INFO:tasks.rados.rados.0.smithi038.stdout:689: snap_create 2024-03-19T11:30:12.435 INFO:tasks.rados.rados.0.smithi038.stdout:684: expect (ObjNum 60 snap 3 seq_num 60) 2024-03-19T11:30:12.435 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:12.439+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:12.505 INFO:tasks.rados.rados.0.smithi038.stdout:686: finishing rollback tid 0 to smithi03829856-46 2024-03-19T11:30:12.505 INFO:tasks.rados.rados.0.smithi038.stdout:683: expect (ObjNum 63 snap 3 seq_num 63) 2024-03-19T11:30:12.510 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:30:12.510 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-03-19T11:30:12.532 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:12.543+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:12.533 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2024-03-19T11:30:12.137078+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":253634463,"num_objects":110,"num_object_clones":5,"num_object_copies":220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":110,"num_whiteouts":1,"num_read":484,"num_read_kb":129293,"num_write":592,"num_write_kb":192750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":51,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":626,"ondisk_log_size":626,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":464884,"kb_used_data":240572,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754509836,"statfs":{"total":773094113280,"available":772618072064,"internally_reserved":0,"allocated":246345728,"data_stored":259044355,"data_compressed":9276788,"data_compressed_allocated":10027008,"data_compressed_original":22667264,"omap_allocated":12751,"internal_metadata":225496625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":191816124,"num_objects":82,"num_object_clones":4,"num_object_copies":164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":1,"num_read":288,"num_read_kb":102601,"num_write":324,"num_write_kb":142995,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":39,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.427695"},"pg_stats":[{"pgid":"4.f","version":"0'0","reported_seq":41,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-19T11:30:08.740659+0000","last_change":"2024-03-19T11:30:08.741099+0000","last_active":"2024-03-19T11:30:08.740659+0000","last_peered":"2024-03-19T11:30:08.740659+0000","last_clean":"2024-03-19T11:30:08.740659+0000","last_became_active":"2024-03-19T11:29:58.551505+0000","last_became_peered":"2024-03-19T11:29:58.551505+0000","last_unstale":"2024-03-19T11:30:08.740659+0000","last_undegraded":"2024-03-19T11:30:08.740659+0000","last_fullsized":"2024-03-19T11:30:08.740659+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:04.221530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065722299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"4.e","version":"28'15","reported_seq":60,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-19T11:30:07.721021+0000","last_change":"2024-03-19T11:29:58.555092+0000","last_active":"2024-03-19T11:30:07.721021+0000","last_peered":"2024-03-19T11:30:07.721021+0000","last_clean":"2024-03-19T11:30:07.721021+0000","last_became_active":"2024-03-19T11:29:58.554727+0000","last_became_peered":"2024-03-19T11:29:58.554727+0000","last_unstale":"2024-03-19T11:30:07.721021+0000","last_undegraded":"2024-03-19T11:30:07.721021+0000","last_fullsized":"2024-03-19T11:30:07.721021+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:14.332076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7462209,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":15,"num_read_kb":11450,"num_write":12,"num_write_kb":3440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"28'17","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-19T11:30:07.720995+0000","last_change":"2024-03-19T11:29:58.554934+0000","last_active":"2024-03-19T11:30:07.720995+0000","last_peered":"2024-03-19T11:30:07.720995+0000","last_clean":"2024-03-19T11:30:07.720995+0000","last_became_active":"2024-03-19T11:29:58.554560+0000","last_became_peered":"2024-03-19T11:29:58.554560+0000","last_unstale":"2024-03-19T11:30:07.720995+0000","last_undegraded":"2024-03-19T11:30:07.720995+0000","last_fullsized":"2024-03-19T11:30:07.720995+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:25.304925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6588245,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":11860,"num_write":12,"num_write_kb":2919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"28'7","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-19T11:30:07.720857+0000","last_change":"2024-03-19T11:29:58.553395+0000","last_active":"2024-03-19T11:30:07.720857+0000","last_peered":"2024-03-19T11:30:07.720857+0000","last_clean":"2024-03-19T11:30:07.720857+0000","last_became_active":"2024-03-19T11:29:58.552036+0000","last_became_peered":"2024-03-19T11:29:58.552036+0000","last_unstale":"2024-03-19T11:30:07.720857+0000","last_undegraded":"2024-03-19T11:30:07.720857+0000","last_fullsized":"2024-03-19T11:30:07.720857+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:25.740176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2554631,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"28'31","reported_seq":77,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T11:30:06.705521+0000","last_change":"2024-03-19T11:29:58.558896+0000","last_active":"2024-03-19T11:30:06.705521+0000","last_peered":"2024-03-19T11:30:06.705521+0000","last_clean":"2024-03-19T11:30:06.705521+0000","last_became_active":"2024-03-19T11:29:58.555569+0000","last_became_peered":"2024-03-19T11:29:58.555569+0000","last_unstale":"2024-03-19T11:30:06.705521+0000","last_undegraded":"2024-03-19T11:30:06.705521+0000","last_fullsized":"2024-03-19T11:30:06.705521+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:26.428285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10873997,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":19,"num_read_kb":8674,"num_write":25,"num_write_kb":6712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"28'24","reported_seq":69,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-19T11:30:07.711941+0000","last_change":"2024-03-19T11:29:58.554004+0000","last_active":"2024-03-19T11:30:07.711941+0000","last_peered":"2024-03-19T11:30:07.711941+0000","last_clean":"2024-03-19T11:30:07.711941+0000","last_became_active":"2024-03-19T11:29:58.553514+0000","last_became_peered":"2024-03-19T11:29:58.553514+0000","last_unstale":"2024-03-19T11:30:07.711941+0000","last_undegraded":"2024-03-19T11:30:07.711941+0000","last_fullsized":"2024-03-19T11:30:07.711941+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:26.439842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10134489,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":9,"num_read_kb":2053,"num_write":16,"num_write_kb":5055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"28'28","reported_seq":83,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-19T11:30:07.725828+0000","last_change":"2024-03-19T11:29:58.553226+0000","last_active":"2024-03-19T11:30:07.725828+0000","last_peered":"2024-03-19T11:30:07.725828+0000","last_clean":"2024-03-19T11:30:07.725828+0000","last_became_active":"2024-03-19T11:29:58.552840+0000","last_became_peered":"2024-03-19T11:29:58.552840+0000","last_unstale":"2024-03-19T11:30:07.725828+0000","last_undegraded":"2024-03-19T11:30:07.725828+0000","last_fullsized":"2024-03-19T11:30:07.725828+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:04.461177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12655506,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":30138,"num_write":20,"num_write_kb":5947,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"29'26","reported_seq":73,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-19T11:30:08.755339+0000","last_change":"2024-03-19T11:30:08.755851+0000","last_active":"2024-03-19T11:30:08.755339+0000","last_peered":"2024-03-19T11:30:08.755339+0000","last_clean":"2024-03-19T11:30:08.755339+0000","last_became_active":"2024-03-19T11:29:58.641526+0000","last_became_peered":"2024-03-19T11:29:58.641526+0000","last_unstale":"2024-03-19T11:30:08.755339+0000","last_undegraded":"2024-03-19T11:30:08.755339+0000","last_fullsized":"2024-03-19T11:30:08.755339+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:15.965143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089404700000000003,"stat_sum":{"num_bytes":9915998,"num_objects":5,"num_object_clones":1,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":10,"num_read_kb":5,"num_write":21,"num_write_kb":5242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"4.5","version":"28'46","reported_seq":107,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-19T11:30:07.711942+0000","last_change":"2024-03-19T11:29:58.550651+0000","last_active":"2024-03-19T11:30:07.711942+0000","last_peered":"2024-03-19T11:30:07.711942+0000","last_clean":"2024-03-19T11:30:07.711942+0000","last_became_active":"2024-03-19T11:29:58.550423+0000","last_became_peered":"2024-03-19T11:29:58.550423+0000","last_unstale":"2024-03-19T11:30:07.711942+0000","last_undegraded":"2024-03-19T11:30:07.711942+0000","last_fullsized":"2024-03-19T11:30:07.711942+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:09.999580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16420208,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":29242,"num_write":28,"num_write_kb":7191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":7,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":193,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-19T11:30:09.097945+0000","last_change":"2024-03-19T11:29:45.008018+0000","last_active":"2024-03-19T11:30:09.097945+0000","last_peered":"2024-03-19T11:30:09.097945+0000","last_clean":"2024-03-19T11:30:09.097945+0000","last_became_active":"2024-03-19T11:29:45.005482+0000","last_became_peered":"2024-03-19T11:29:45.005482+0000","last_unstale":"2024-03-19T11:30:09.097945+0000","last_undegraded":"2024-03-19T11:30:09.097945+0000","last_fullsized":"2024-03-19T11:30:09.097945+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:43.166618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:43.166618+0000","last_clean_scrub_stamp":"2024-03-19T11:29:43.166618+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:02.385897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-19T11:30:08.754874+0000","last_change":"2024-03-19T11:29:46.322491+0000","last_active":"2024-03-19T11:30:08.754874+0000","last_peered":"2024-03-19T11:30:08.754874+0000","last_clean":"2024-03-19T11:30:08.754874+0000","last_became_active":"2024-03-19T11:29:46.321760+0000","last_became_peered":"2024-03-19T11:29:46.321760+0000","last_unstale":"2024-03-19T11:30:08.754874+0000","last_undegraded":"2024-03-19T11:30:08.754874+0000","last_fullsized":"2024-03-19T11:30:08.754874+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:04.561915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"29'44","reported_seq":76,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-19T11:30:09.043904+0000","last_change":"2024-03-19T11:29:57.549277+0000","last_active":"2024-03-19T11:30:09.043904+0000","last_peered":"2024-03-19T11:30:09.043904+0000","last_clean":"2024-03-19T11:30:09.043904+0000","last_became_active":"2024-03-19T11:29:57.548946+0000","last_became_peered":"2024-03-19T11:29:57.548946+0000","last_unstale":"2024-03-19T11:30:09.043904+0000","last_undegraded":"2024-03-19T11:30:09.043904+0000","last_fullsized":"2024-03-19T11:30:09.043904+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:08.380405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25974783,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":44,"num_write_kb":28014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"29'28","reported_seq":69,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T11:30:06.705520+0000","last_change":"2024-03-19T11:29:58.552682+0000","last_active":"2024-03-19T11:30:06.705520+0000","last_peered":"2024-03-19T11:30:06.705520+0000","last_clean":"2024-03-19T11:30:06.705520+0000","last_became_active":"2024-03-19T11:29:58.552259+0000","last_became_peered":"2024-03-19T11:29:58.552259+0000","last_unstale":"2024-03-19T11:30:06.705520+0000","last_undegraded":"2024-03-19T11:30:06.705520+0000","last_fullsized":"2024-03-19T11:30:06.705520+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:12.887021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10912793,"num_objects":5,"num_object_clones":1,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":5,"num_write":24,"num_write_kb":6015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-19T11:30:09.097921+0000","last_change":"2024-03-19T11:29:46.436253+0000","last_active":"2024-03-19T11:30:09.097921+0000","last_peered":"2024-03-19T11:30:09.097921+0000","last_clean":"2024-03-19T11:30:09.097921+0000","last_became_active":"2024-03-19T11:29:46.435783+0000","last_became_peered":"2024-03-19T11:29:46.435783+0000","last_unstale":"2024-03-19T11:30:09.097921+0000","last_undegraded":"2024-03-19T11:30:09.097921+0000","last_fullsized":"2024-03-19T11:30:09.097921+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:59.997466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"28'48","reported_seq":78,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T11:30:06.705315+0000","last_change":"2024-03-19T11:29:57.555187+0000","last_active":"2024-03-19T11:30:06.705315+0000","last_peered":"2024-03-19T11:30:06.705315+0000","last_clean":"2024-03-19T11:30:06.705315+0000","last_became_active":"2024-03-19T11:29:57.554825+0000","last_became_peered":"2024-03-19T11:29:57.554825+0000","last_unstale":"2024-03-19T11:30:06.705315+0000","last_undegraded":"2024-03-19T11:30:06.705315+0000","last_fullsized":"2024-03-19T11:30:06.705315+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:58.878342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32298305,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":36,"num_read_kb":1460,"num_write":48,"num_write_kb":31549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"32'10","reported_seq":53,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-19T11:30:08.765847+0000","last_change":"2024-03-19T11:30:08.741399+0000","last_active":"2024-03-19T11:30:08.765847+0000","last_peered":"2024-03-19T11:30:08.765847+0000","last_clean":"2024-03-19T11:30:08.765847+0000","last_became_active":"2024-03-19T11:29:58.555355+0000","last_became_peered":"2024-03-19T11:29:58.555355+0000","last_unstale":"2024-03-19T11:30:08.765847+0000","last_undegraded":"2024-03-19T11:30:08.765847+0000","last_fullsized":"2024-03-19T11:30:08.765847+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:22.115208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064704200000000002,"stat_sum":{"num_bytes":3911078,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":8,"num_write_kb":2188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":40,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-19T11:30:08.737820+0000","last_change":"2024-03-19T11:29:46.319811+0000","last_active":"2024-03-19T11:30:08.737820+0000","last_peered":"2024-03-19T11:30:08.737820+0000","last_clean":"2024-03-19T11:30:08.737820+0000","last_became_active":"2024-03-19T11:29:46.319334+0000","last_became_peered":"2024-03-19T11:29:46.319334+0000","last_unstale":"2024-03-19T11:30:08.737820+0000","last_undegraded":"2024-03-19T11:30:08.737820+0000","last_fullsized":"2024-03-19T11:30:08.737820+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:04.320463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"28'51","reported_seq":82,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-19T11:30:08.738184+0000","last_change":"2024-03-19T11:29:57.550320+0000","last_active":"2024-03-19T11:30:08.738184+0000","last_peered":"2024-03-19T11:30:08.738184+0000","last_clean":"2024-03-19T11:30:08.738184+0000","last_became_active":"2024-03-19T11:29:57.549928+0000","last_became_peered":"2024-03-19T11:29:57.549928+0000","last_unstale":"2024-03-19T11:30:08.738184+0000","last_undegraded":"2024-03-19T11:30:08.738184+0000","last_fullsized":"2024-03-19T11:30:08.738184+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:01.987302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30226734,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":38,"num_read_kb":26,"num_write":51,"num_write_kb":29523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"29'28","reported_seq":79,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-19T11:30:08.739545+0000","last_change":"2024-03-19T11:30:08.739914+0000","last_active":"2024-03-19T11:30:08.739545+0000","last_peered":"2024-03-19T11:30:08.739545+0000","last_clean":"2024-03-19T11:30:08.739545+0000","last_became_active":"2024-03-19T11:29:58.555204+0000","last_became_peered":"2024-03-19T11:29:58.555204+0000","last_unstale":"2024-03-19T11:30:08.739545+0000","last_undegraded":"2024-03-19T11:30:08.739545+0000","last_fullsized":"2024-03-19T11:30:08.739545+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:12.571983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057545199999999995,"stat_sum":{"num_bytes":12504243,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":22,"num_write_kb":7297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":38,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-19T11:30:08.755311+0000","last_change":"2024-03-19T11:29:46.321255+0000","last_active":"2024-03-19T11:30:08.755311+0000","last_peered":"2024-03-19T11:30:08.755311+0000","last_clean":"2024-03-19T11:30:08.755311+0000","last_became_active":"2024-03-19T11:29:46.320679+0000","last_became_peered":"2024-03-19T11:29:46.320679+0000","last_unstale":"2024-03-19T11:30:08.755311+0000","last_undegraded":"2024-03-19T11:30:08.755311+0000","last_fullsized":"2024-03-19T11:30:08.755311+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:14.793563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"29'47","reported_seq":80,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-19T11:30:07.720733+0000","last_change":"2024-03-19T11:29:57.554995+0000","last_active":"2024-03-19T11:30:07.720733+0000","last_peered":"2024-03-19T11:30:07.720733+0000","last_clean":"2024-03-19T11:30:07.720733+0000","last_became_active":"2024-03-19T11:29:57.554481+0000","last_became_peered":"2024-03-19T11:29:57.554481+0000","last_unstale":"2024-03-19T11:30:07.720733+0000","last_undegraded":"2024-03-19T11:30:07.720733+0000","last_fullsized":"2024-03-19T11:30:07.720733+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:12.584961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35442802,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":35,"num_read_kb":691,"num_write":47,"num_write_kb":34619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"28'6","reported_seq":52,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-19T11:30:08.739168+0000","last_change":"2024-03-19T11:30:08.739465+0000","last_active":"2024-03-19T11:30:08.739168+0000","last_peered":"2024-03-19T11:30:08.739168+0000","last_clean":"2024-03-19T11:30:08.739168+0000","last_became_active":"2024-03-19T11:29:58.555026+0000","last_became_peered":"2024-03-19T11:29:58.555026+0000","last_unstale":"2024-03-19T11:30:08.739168+0000","last_undegraded":"2024-03-19T11:30:08.739168+0000","last_fullsized":"2024-03-19T11:30:08.739168+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:23.324332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067483699999999996,"stat_sum":{"num_bytes":2352947,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":6561,"num_write":4,"num_write_kb":981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-19T11:30:08.739533+0000","last_change":"2024-03-19T11:29:46.541233+0000","last_active":"2024-03-19T11:30:08.739533+0000","last_peered":"2024-03-19T11:30:08.739533+0000","last_clean":"2024-03-19T11:30:08.739533+0000","last_became_active":"2024-03-19T11:29:46.540939+0000","last_became_peered":"2024-03-19T11:29:46.540939+0000","last_unstale":"2024-03-19T11:30:08.739533+0000","last_undegraded":"2024-03-19T11:30:08.739533+0000","last_fullsized":"2024-03-19T11:30:08.739533+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:54.761098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"28'14","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-19T11:30:06.705566+0000","last_change":"2024-03-19T11:29:58.555974+0000","last_active":"2024-03-19T11:30:06.705566+0000","last_peered":"2024-03-19T11:30:06.705566+0000","last_clean":"2024-03-19T11:30:06.705566+0000","last_became_active":"2024-03-19T11:29:58.555039+0000","last_became_peered":"2024-03-19T11:29:58.555039+0000","last_unstale":"2024-03-19T11:30:06.705566+0000","last_undegraded":"2024-03-19T11:30:06.705566+0000","last_fullsized":"2024-03-19T11:30:06.705566+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:14.495409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7142876,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":16453,"num_write":12,"num_write_kb":3063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-19T11:30:09.097563+0000","last_change":"2024-03-19T11:29:46.542897+0000","last_active":"2024-03-19T11:30:09.097563+0000","last_peered":"2024-03-19T11:30:09.097563+0000","last_clean":"2024-03-19T11:30:09.097563+0000","last_became_active":"2024-03-19T11:29:46.542092+0000","last_became_peered":"2024-03-19T11:29:46.542092+0000","last_unstale":"2024-03-19T11:30:09.097563+0000","last_undegraded":"2024-03-19T11:30:09.097563+0000","last_fullsized":"2024-03-19T11:30:09.097563+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:46.318698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"29'13","reported_seq":61,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-19T11:30:09.044127+0000","last_change":"2024-03-19T11:30:09.044498+0000","last_active":"2024-03-19T11:30:09.044127+0000","last_peered":"2024-03-19T11:30:09.044127+0000","last_clean":"2024-03-19T11:30:09.044127+0000","last_became_active":"2024-03-19T11:29:58.642619+0000","last_became_peered":"2024-03-19T11:29:58.642619+0000","last_unstale":"2024-03-19T11:30:09.044127+0000","last_undegraded":"2024-03-19T11:30:09.044127+0000","last_fullsized":"2024-03-19T11:30:09.044127+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:23.783217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054984999999999995,"stat_sum":{"num_bytes":6238025,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":13,"num_write_kb":3464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":39,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-19T11:30:08.740166+0000","last_change":"2024-03-19T11:29:46.423438+0000","last_active":"2024-03-19T11:30:08.740166+0000","last_peered":"2024-03-19T11:30:08.740166+0000","last_clean":"2024-03-19T11:30:08.740166+0000","last_became_active":"2024-03-19T11:29:46.423027+0000","last_became_peered":"2024-03-19T11:29:46.423027+0000","last_unstale":"2024-03-19T11:30:08.740166+0000","last_undegraded":"2024-03-19T11:30:08.740166+0000","last_fullsized":"2024-03-19T11:30:08.740166+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:55.305083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"29'26","reported_seq":75,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-19T11:30:07.721020+0000","last_change":"2024-03-19T11:29:58.554149+0000","last_active":"2024-03-19T11:30:07.721020+0000","last_peered":"2024-03-19T11:30:07.721020+0000","last_clean":"2024-03-19T11:30:07.721020+0000","last_became_active":"2024-03-19T11:29:58.552749+0000","last_became_peered":"2024-03-19T11:29:58.552749+0000","last_unstale":"2024-03-19T11:30:07.721020+0000","last_undegraded":"2024-03-19T11:30:07.721020+0000","last_fullsized":"2024-03-19T11:30:07.721020+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:13.312961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9434226,"num_objects":4,"num_object_clones":1,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":10542,"num_write":22,"num_write_kb":6104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":22,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-19T11:30:08.738060+0000","last_change":"2024-03-19T11:30:03.276845+0000","last_active":"2024-03-19T11:30:08.738060+0000","last_peered":"2024-03-19T11:30:08.738060+0000","last_clean":"2024-03-19T11:30:08.738060+0000","last_became_active":"2024-03-19T11:30:03.276220+0000","last_became_peered":"2024-03-19T11:30:03.276220+0000","last_unstale":"2024-03-19T11:30:08.738060+0000","last_undegraded":"2024-03-19T11:30:08.738060+0000","last_fullsized":"2024-03-19T11:30:08.738060+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:01.829767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":129101471,"num_objects":55,"num_object_clones":5,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":1,"num_read":227,"num_read_kb":126995,"num_write":243,"num_write_kb":66629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":51,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131489792,"data_stored":134422633,"data_compressed":236030,"data_compressed_allocated":245760,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},"log_size":319,"ondisk_log_size":319,"up":32,"acting":32,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":123942624,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":141,"num_read_kb":2198,"num_write":190,"num_write_kb":123705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":117800960,"data_stored":129209358,"data_compressed":9028630,"data_compressed_allocated":9715712,"data_compressed_original":21250048,"omap_allocated":0,"internal_metadata":0},"log_size":190,"ondisk_log_size":190,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":12128,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542153,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36100,"kb_used_data":8408,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94335740,"statfs":{"total":96636764160,"available":96599797760,"internally_reserved":0,"allocated":8609792,"data_stored":9473628,"data_compressed":6064,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542153,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55384,"kb_used_data":27332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94316456,"statfs":{"total":96636764160,"available":96580050944,"internally_reserved":0,"allocated":27987968,"data_stored":29088664,"data_compressed":1107341,"data_compressed_allocated":1183744,"data_compressed_original":2433024,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542153,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44800,"kb_used_data":17776,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94327040,"statfs":{"total":96636764160,"available":96590888960,"internally_reserved":0,"allocated":18202624,"data_stored":19935570,"data_compressed":1107341,"data_compressed_allocated":1183744,"data_compressed_original":2433024,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542152,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49996,"kb_used_data":22976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94321844,"statfs":{"total":96636764160,"available":96585568256,"internally_reserved":0,"allocated":23527424,"data_stored":24772882,"data_compressed":1072404,"data_compressed_allocated":1155072,"data_compressed_original":2621440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542152,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75336,"kb_used_data":43496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94296504,"statfs":{"total":96636764160,"available":96559620096,"internally_reserved":0,"allocated":44539904,"data_stored":46037024,"data_compressed":1072404,"data_compressed_allocated":1155072,"data_compressed_original":2621440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43128,"kb_used_data":16100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94328712,"statfs":{"total":96636764160,"available":96592601088,"internally_reserved":0,"allocated":16486400,"data_stored":16183156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85632,"kb_used_data":58612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94286208,"statfs":{"total":96636764160,"available":96549076992,"internally_reserved":0,"allocated":60018688,"data_stored":62891711,"data_compressed":2452585,"data_compressed_allocated":2641920,"data_compressed_original":5767168,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74508,"kb_used_data":45872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94297332,"statfs":{"total":96636764160,"available":96560467968,"internally_reserved":0,"allocated":46972928,"data_stored":50661720,"data_compressed":2458649,"data_compressed_allocated":2674688,"data_compressed_original":6279168,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6064,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6064,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32694272,"data_stored":35780601,"data_compressed":2452585,"data_compressed_allocated":2641920,"data_compressed_original":5767168,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14204928,"data_stored":15582936,"data_compressed":954389,"data_compressed_allocated":1032192,"data_compressed_original":2424832,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14204928,"data_stored":15582936,"data_compressed":954389,"data_compressed_allocated":1032192,"data_compressed_original":2424832,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12001280,"data_stored":13241142,"data_compressed":1107341,"data_compressed_allocated":1183744,"data_compressed_original":2433024,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":32694272,"data_stored":35780601,"data_compressed":2452585,"data_compressed_allocated":2641920,"data_compressed_original":5767168,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12001280,"data_stored":13241142,"data_compressed":1107341,"data_compressed_allocated":1183744,"data_compressed_original":2433024,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13914112,"data_stored":14219469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32694272,"data_stored":33770341,"data_compressed":118015,"data_compressed_allocated":122880,"data_compressed_original":196608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10420224,"data_stored":10921739,"data_compressed":118015,"data_compressed_allocated":122880,"data_compressed_original":196608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":5910528,"data_stored":6555042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":27058176,"data_stored":26983740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":17039360,"data_stored":17104663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16166912,"data_stored":16049211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8286208,"data_stored":8818428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T11:30:12.534 INFO:tasks.thrashosds.thrasher:Setting 4.c to [3, 4, 0, 1] 2024-03-19T11:30:12.535 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '4.c', '3', '4', '0', '1'] 2024-03-19T11:30:12.535 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 4.c 3 4 0 1 2024-03-19T11:30:12.602 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:12.603 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:30:12.603 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:30:12.603 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:12.617 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-19T11:30:12.643 INFO:tasks.rados.rados.0.smithi038.stdout:686: finishing rollback tid 1 to smithi03829856-46 2024-03-19T11:30:12.643 INFO:tasks.rados.rados.0.smithi038.stdout:686: finishing rollback tid 2 to smithi03829856-46 2024-03-19T11:30:12.643 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 46 v 38 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-03-19T11:30:12.643 INFO:tasks.rados.rados.0.smithi038.stdout:687: expect (ObjNum 66 snap 3 seq_num 66) 2024-03-19T11:30:12.644 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:12.652+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:12.734 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:12.744+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:12.747 INFO:tasks.rados.rados.0.smithi038.stdout:685: finishing write tid 1 to smithi03829856-39 2024-03-19T11:30:12.747 INFO:tasks.rados.rados.0.smithi038.stdout:685: finishing write tid 2 to smithi03829856-39 2024-03-19T11:30:12.747 INFO:tasks.rados.rados.0.smithi038.stdout:685: finishing write tid 3 to smithi03829856-39 2024-03-19T11:30:12.747 INFO:tasks.rados.rados.0.smithi038.stdout:685: finishing write tid 4 to smithi03829856-39 2024-03-19T11:30:12.747 INFO:tasks.rados.rados.0.smithi038.stdout:685: oid 39 updating version 0 to 17 2024-03-19T11:30:12.747 INFO:tasks.rados.rados.0.smithi038.stdout:685: oid 39 updating version 17 to 19 2024-03-19T11:30:12.747 INFO:tasks.rados.rados.0.smithi038.stdout:685: oid 39 version 19 is already newer than 18 2024-03-19T11:30:12.747 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 39 v 19 (ObjNum 70 snap 4 seq_num 70) dirty exists 2024-03-19T11:30:12.747 INFO:tasks.rados.rados.0.smithi038.stdout:685: left oid 39 (ObjNum 70 snap 4 seq_num 70) 2024-03-19T11:30:12.747 INFO:tasks.rados.rados.0.smithi038.stdout:688: finishing write tid 1 to smithi03829856-41 2024-03-19T11:30:12.748 INFO:tasks.rados.rados.0.smithi038.stdout:688: finishing write tid 2 to smithi03829856-41 2024-03-19T11:30:12.748 INFO:tasks.rados.rados.0.smithi038.stdout:683: done (6 left) 2024-03-19T11:30:12.748 INFO:tasks.rados.rados.0.smithi038.stdout:684: done (5 left) 2024-03-19T11:30:12.748 INFO:tasks.rados.rados.0.smithi038.stdout:685: done (4 left) 2024-03-19T11:30:12.748 INFO:tasks.rados.rados.0.smithi038.stdout:686: done (3 left) 2024-03-19T11:30:12.748 INFO:tasks.rados.rados.0.smithi038.stdout:687: done (2 left) 2024-03-19T11:30:12.748 INFO:tasks.rados.rados.0.smithi038.stdout:689: done (1 left) 2024-03-19T11:30:12.748 INFO:tasks.rados.rados.0.smithi038.stdout:690: tier_promote oid 28 2024-03-19T11:30:12.748 INFO:tasks.rados.rados.0.smithi038.stdout:688: finishing write tid 3 to smithi03829856-41 2024-03-19T11:30:12.748 INFO:tasks.rados.rados.0.smithi038.stdout:688: finishing write tid 4 to smithi03829856-41 2024-03-19T11:30:12.749 INFO:tasks.rados.rados.0.smithi038.stdout:688: oid 41 updating version 0 to 41 2024-03-19T11:30:12.749 INFO:tasks.rados.rados.0.smithi038.stdout:688: oid 41 updating version 41 to 42 2024-03-19T11:30:12.749 INFO:tasks.rados.rados.0.smithi038.stdout:688: oid 41 version 42 is already newer than 40 2024-03-19T11:30:12.749 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 41 v 42 (ObjNum 71 snap 4 seq_num 71) dirty exists 2024-03-19T11:30:12.749 INFO:tasks.rados.rados.0.smithi038.stdout:688: left oid 41 (ObjNum 71 snap 4 seq_num 71) 2024-03-19T11:30:12.749 INFO:tasks.rados.rados.0.smithi038.stdout:688: done (1 left) 2024-03-19T11:30:12.749 INFO:tasks.rados.rados.0.smithi038.stdout:691: read oid 10 snap 4 2024-03-19T11:30:12.749 INFO:tasks.rados.rados.0.smithi038.stdout:691: expect (ObjNum 55 snap 1 seq_num 55) 2024-03-19T11:30:12.749 INFO:tasks.rados.rados.0.smithi038.stdout:692: read oid 42 snap -1 2024-03-19T11:30:12.749 INFO:tasks.rados.rados.0.smithi038.stdout:692: expect (ObjNum 52 snap 0 seq_num 52) 2024-03-19T11:30:12.749 INFO:tasks.rados.rados.0.smithi038.stdout:693: tier_promote oid 20 2024-03-19T11:30:12.750 INFO:tasks.rados.rados.0.smithi038.stdout:694: read oid 12 snap -1 2024-03-19T11:30:12.750 INFO:tasks.rados.rados.0.smithi038.stdout:694: expect (ObjNum 11 snap 0 seq_num 11) 2024-03-19T11:30:12.750 INFO:tasks.rados.rados.0.smithi038.stdout:695: set_chunk oid 2 target oid 2024-03-19T11:30:12.750 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:12.756+0000 7fa5e08db640 -1 osd.3 34 handle_osd_map injecting map crc failure 2024-03-19T11:30:12.763 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:12.763 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:30:12.763 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:30:12.763 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:30:12.763 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:12.766 INFO:tasks.rados.rados.0.smithi038.stdout:695: set_chunk oid 2 offset: 601600 length: 207872 target oid 24df73eb14d5465f466b1889d4952f96785c09a615e42a99c9b768b4b3347e65 offset: 0 snap -1 2024-03-19T11:30:12.766 INFO:tasks.rados.rados.0.smithi038.stdout:696: write oid 34 current snap is 5 2024-03-19T11:30:12.766 INFO:tasks.rados.rados.0.smithi038.stdout:690: got (0) Success 2024-03-19T11:30:12.766 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 28 v 13 (ObjNum 54 snap 1 seq_num 54) dirty exists 2024-03-19T11:30:12.766 INFO:tasks.rados.rados.0.smithi038.stdout:696: seq_num 72 ranges {410104=465105,1563527=583025,2810926=399178} 2024-03-19T11:30:12.771 INFO:tasks.rados.rados.0.smithi038.stdout:696: writing smithi03829856-34 from 410104 to 875209 tid 1 2024-03-19T11:30:12.777 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-19T11:30:12.778 INFO:tasks.rados.rados.0.smithi038.stdout:696: writing smithi03829856-34 from 1563527 to 2146552 tid 2 2024-03-19T11:30:12.785 INFO:tasks.rados.rados.0.smithi038.stdout:696: writing smithi03829856-34 from 2810926 to 3210104 tid 3 2024-03-19T11:30:12.785 INFO:tasks.rados.rados.0.smithi038.stdout:690: done (6 left) 2024-03-19T11:30:12.785 INFO:tasks.rados.rados.0.smithi038.stdout:697: read oid 29 snap -1 2024-03-19T11:30:12.785 INFO:tasks.rados.rados.0.smithi038.stdout:697: expect (ObjNum 28 snap 0 seq_num 28) 2024-03-19T11:30:12.785 INFO:tasks.rados.rados.0.smithi038.stdout:698: delete oid 28 current snap is 5 2024-03-19T11:30:12.785 INFO:tasks.rados.rados.0.smithi038.stdout:692: expect (ObjNum 52 snap 0 seq_num 52) 2024-03-19T11:30:12.835 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:12.843+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:12.892 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:12.892 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:30:12.892 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:30:12.892 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:30:12.892 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.892 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:6 4.6 e27/21 4:626acd4d:::smithi03829856-2:head v 27'2, mlcod=0'0)", 2024-03-19T11:30:12.893 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.696903+0000", 2024-03-19T11:30:12.893 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.2056423580000004, 2024-03-19T11:30:12.893 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0071029129999999998, 2024-03-19T11:30:12.893 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:12.893 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:12.893 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:12.893 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.893 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:12.893 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696903+0000", 2024-03-19T11:30:12.893 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.893 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.893 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.893 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:12.893 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696903+0000", 2024-03-19T11:30:12.894 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.894 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.894 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.894 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:12.894 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696904+0000", 2024-03-19T11:30:12.894 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4270000000000001e-06 2024-03-19T11:30:12.894 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.894 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.894 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:12.894 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.697064+0000", 2024-03-19T11:30:12.894 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015956100000000001 2024-03-19T11:30:12.894 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.895 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.895 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:12.895 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.697082+0000", 2024-03-19T11:30:12.895 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8868999999999998e-05 2024-03-19T11:30:12.895 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.895 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.895 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:12.895 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.697096+0000", 2024-03-19T11:30:12.895 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3155e-05 2024-03-19T11:30:12.895 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.895 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.895 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:12.895 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.697169+0000", 2024-03-19T11:30:12.895 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3226999999999995e-05 2024-03-19T11:30:12.896 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.896 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.896 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:12.896 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.697193+0000", 2024-03-19T11:30:12.896 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4153000000000001e-05 2024-03-19T11:30:12.896 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.896 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.896 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:12.896 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.703936+0000", 2024-03-19T11:30:12.896 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0067430570000000002 2024-03-19T11:30:12.896 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.896 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.896 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:12.896 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.704006+0000", 2024-03-19T11:30:12.897 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9463999999999998e-05 2024-03-19T11:30:12.897 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.897 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.897 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.897 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.897 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.897 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:7 4.6 e27/21 4:626acd4d:::smithi03829856-2:head v 27'3, mlcod=0'0)", 2024-03-19T11:30:12.897 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.699209+0000", 2024-03-19T11:30:12.897 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.2033359449999992, 2024-03-19T11:30:12.897 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00524114, 2024-03-19T11:30:12.897 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:12.897 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:12.897 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:12.897 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.898 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:12.898 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.699209+0000", 2024-03-19T11:30:12.898 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.898 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.898 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.898 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:12.898 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.699209+0000", 2024-03-19T11:30:12.898 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.898 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.898 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.898 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:12.898 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.699211+0000", 2024-03-19T11:30:12.898 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7230000000000001e-06 2024-03-19T11:30:12.898 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.899 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.899 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:12.899 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.699221+0000", 2024-03-19T11:30:12.899 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0698e-05 2024-03-19T11:30:12.899 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.899 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.899 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:12.899 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.699237+0000", 2024-03-19T11:30:12.899 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5665999999999999e-05 2024-03-19T11:30:12.899 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.899 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.899 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:12.899 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.699247+0000", 2024-03-19T11:30:12.899 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7880000000000001e-06 2024-03-19T11:30:12.899 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.900 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.900 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:12.900 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.699304+0000", 2024-03-19T11:30:12.900 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7120999999999997e-05 2024-03-19T11:30:12.900 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.900 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.900 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:12.900 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.699327+0000", 2024-03-19T11:30:12.900 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2795000000000001e-05 2024-03-19T11:30:12.900 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.900 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.900 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:12.900 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.704397+0000", 2024-03-19T11:30:12.900 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050705790000000004 2024-03-19T11:30:12.901 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.901 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.901 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:12.901 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.704450+0000", 2024-03-19T11:30:12.901 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2769999999999998e-05 2024-03-19T11:30:12.901 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.901 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.901 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.901 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.901 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.901 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:13 4.4 4:21eac9b2:::smithi03829856-4:head [write 450421~625747,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:30:12.901 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.720170+0000", 2024-03-19T11:30:12.901 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.1823750440000005, 2024-03-19T11:30:12.901 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0090522840000000007, 2024-03-19T11:30:12.901 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:12.902 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:12.902 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:12.902 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:12.902 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:12.902 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 13 2024-03-19T11:30:12.902 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.902 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:12.902 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.902 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:12.902 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.720170+0000", 2024-03-19T11:30:12.902 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.902 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.902 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.902 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:12.903 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.720170+0000", 2024-03-19T11:30:12.903 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.903 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.903 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.903 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:12.903 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.720172+0000", 2024-03-19T11:30:12.903 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9709999999999998e-06 2024-03-19T11:30:12.903 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.903 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.903 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:12.903 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.720601+0000", 2024-03-19T11:30:12.903 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00042897999999999998 2024-03-19T11:30:12.903 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.903 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.904 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:12.904 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.720615+0000", 2024-03-19T11:30:12.904 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4617000000000001e-05 2024-03-19T11:30:12.904 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.904 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.904 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:12.904 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.720627+0000", 2024-03-19T11:30:12.904 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1513e-05 2024-03-19T11:30:12.904 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.904 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.904 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:12.904 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.720705+0000", 2024-03-19T11:30:12.904 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7880000000000007e-05 2024-03-19T11:30:12.904 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.904 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.905 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-03-19T11:30:12.905 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.720709+0000", 2024-03-19T11:30:12.905 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5549999999999996e-06 2024-03-19T11:30:12.905 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.905 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.905 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:12.905 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.722748+0000", 2024-03-19T11:30:12.905 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0020388630000000001 2024-03-19T11:30:12.905 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.905 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.905 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:12.905 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.722891+0000", 2024-03-19T11:30:12.905 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014286 2024-03-19T11:30:12.905 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.905 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.906 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 2", 2024-03-19T11:30:12.906 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723057+0000", 2024-03-19T11:30:12.906 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000166286 2024-03-19T11:30:12.906 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.906 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.906 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:12.906 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.725297+0000", 2024-03-19T11:30:12.906 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022397770000000001 2024-03-19T11:30:12.906 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.906 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.906 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:12.906 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.729110+0000", 2024-03-19T11:30:12.906 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038130109999999998 2024-03-19T11:30:12.907 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.907 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.907 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:12.907 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.729169+0000", 2024-03-19T11:30:12.907 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8616000000000003e-05 2024-03-19T11:30:12.907 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.907 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.907 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:12.907 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.729222+0000", 2024-03-19T11:30:12.907 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3355e-05 2024-03-19T11:30:12.907 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.907 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.907 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.907 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.907 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.908 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:15 4.4 4:21eac9b2:::smithi03829856-4:head [setxattr _header (55) in=7b,truncate 2050421] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:30:12.908 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.725770+0000", 2024-03-19T11:30:12.908 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.1767749409999997, 2024-03-19T11:30:12.908 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.004453622, 2024-03-19T11:30:12.908 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:12.908 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:12.908 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:12.908 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:12.908 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:12.908 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 15 2024-03-19T11:30:12.908 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.908 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:12.908 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.908 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:12.908 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.725770+0000", 2024-03-19T11:30:12.909 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.909 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.909 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.909 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:12.909 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.725770+0000", 2024-03-19T11:30:12.909 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.909 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.909 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.909 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:12.909 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.725771+0000", 2024-03-19T11:30:12.909 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4300000000000001e-07 2024-03-19T11:30:12.909 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.909 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.909 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:12.910 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.725776+0000", 2024-03-19T11:30:12.910 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5289999999999999e-06 2024-03-19T11:30:12.910 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.910 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.910 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:12.910 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.725785+0000", 2024-03-19T11:30:12.910 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8310000000000007e-06 2024-03-19T11:30:12.910 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.910 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.910 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:12.910 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.725793+0000", 2024-03-19T11:30:12.910 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1669999999999999e-06 2024-03-19T11:30:12.910 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.910 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.911 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:12.911 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.728614+0000", 2024-03-19T11:30:12.911 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0028204269999999999 2024-03-19T11:30:12.911 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.911 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.911 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:12.911 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.728681+0000", 2024-03-19T11:30:12.911 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6903999999999995e-05 2024-03-19T11:30:12.911 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.911 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.911 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 2", 2024-03-19T11:30:12.911 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.728809+0000", 2024-03-19T11:30:12.911 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000128603 2024-03-19T11:30:12.912 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.912 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.912 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:12.912 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.729577+0000", 2024-03-19T11:30:12.912 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00076769900000000005 2024-03-19T11:30:12.912 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.912 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.912 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:12.912 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.730105+0000", 2024-03-19T11:30:12.912 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00052791999999999997 2024-03-19T11:30:12.912 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.912 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.912 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:12.912 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.730164+0000", 2024-03-19T11:30:12.913 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8584e-05 2024-03-19T11:30:12.913 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.913 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.913 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:12.913 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.730224+0000", 2024-03-19T11:30:12.913 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0015000000000001e-05 2024-03-19T11:30:12.913 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.913 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.913 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.913 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.913 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.913 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:16 4.4 4:21eac9b2:::smithi03829856-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:30:12.913 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.725805+0000", 2024-03-19T11:30:12.913 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.1767402590000007, 2024-03-19T11:30:12.913 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0045749199999999997, 2024-03-19T11:30:12.914 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:12.914 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:12.914 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:12.914 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:12.914 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:12.914 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 16 2024-03-19T11:30:12.914 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.914 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:12.914 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.914 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:12.914 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.725805+0000", 2024-03-19T11:30:12.914 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.914 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.914 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.915 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:12.915 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.725805+0000", 2024-03-19T11:30:12.915 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.915 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.915 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.915 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:12.915 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.725805+0000", 2024-03-19T11:30:12.915 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1900000000000003e-07 2024-03-19T11:30:12.915 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.915 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.915 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:12.915 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.725808+0000", 2024-03-19T11:30:12.915 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0189999999999998e-06 2024-03-19T11:30:12.915 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.916 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.916 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:12.916 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.725816+0000", 2024-03-19T11:30:12.916 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6210000000000004e-06 2024-03-19T11:30:12.916 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.916 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.916 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:12.916 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.725821+0000", 2024-03-19T11:30:12.916 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7609999999999998e-06 2024-03-19T11:30:12.916 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.916 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.916 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:12.916 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.729002+0000", 2024-03-19T11:30:12.916 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003181377 2024-03-19T11:30:12.916 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.917 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.917 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-03-19T11:30:12.917 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.729069+0000", 2024-03-19T11:30:12.917 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6755999999999996e-05 2024-03-19T11:30:12.917 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.917 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.917 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:12.917 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.730268+0000", 2024-03-19T11:30:12.917 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011995490000000001 2024-03-19T11:30:12.917 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.917 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.917 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:12.917 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.730314+0000", 2024-03-19T11:30:12.917 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5534999999999997e-05 2024-03-19T11:30:12.918 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.918 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.918 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:12.918 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.730380+0000", 2024-03-19T11:30:12.918 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5783000000000004e-05 2024-03-19T11:30:12.918 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.918 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.918 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.918 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.918 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.918 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:844 4.7 4:e41d47fb:::smithi03829856-14:head [write 574806~778913] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:12.918 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:08.749074+0000", 2024-03-19T11:30:12.918 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.1534712630000001, 2024-03-19T11:30:12.918 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013996124, 2024-03-19T11:30:12.919 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:12.919 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:12.919 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:12.919 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:12.919 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:12.919 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 844 2024-03-19T11:30:12.919 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.919 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:12.919 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.919 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:12.920 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.749074+0000", 2024-03-19T11:30:12.920 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.920 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.920 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.920 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:12.920 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.749074+0000", 2024-03-19T11:30:12.920 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.920 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.920 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.920 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:12.920 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.749076+0000", 2024-03-19T11:30:12.920 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.588e-06 2024-03-19T11:30:12.920 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.921 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.921 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:12.921 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.749714+0000", 2024-03-19T11:30:12.921 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00063805999999999997 2024-03-19T11:30:12.921 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.921 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.921 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:12.921 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.749750+0000", 2024-03-19T11:30:12.921 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5709e-05 2024-03-19T11:30:12.921 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.921 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.921 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:12.921 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.749767+0000", 2024-03-19T11:30:12.921 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7127000000000001e-05 2024-03-19T11:30:12.922 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.922 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.922 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:12.922 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.749860+0000", 2024-03-19T11:30:12.922 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2875999999999996e-05 2024-03-19T11:30:12.922 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.922 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.922 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:12.922 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.749943+0000", 2024-03-19T11:30:12.922 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2915000000000002e-05 2024-03-19T11:30:12.922 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.922 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.922 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:12.922 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.750083+0000", 2024-03-19T11:30:12.923 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013953300000000001 2024-03-19T11:30:12.923 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.923 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.923 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:12.923 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.753673+0000", 2024-03-19T11:30:12.923 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0035904470000000001 2024-03-19T11:30:12.923 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.923 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.923 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:12.923 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.762965+0000", 2024-03-19T11:30:12.923 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0092923390000000002 2024-03-19T11:30:12.923 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.923 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.923 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:12.924 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.763001+0000", 2024-03-19T11:30:12.924 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6087000000000002e-05 2024-03-19T11:30:12.924 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.924 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.924 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:12.924 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.763070+0000", 2024-03-19T11:30:12.924 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8442999999999995e-05 2024-03-19T11:30:12.924 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.924 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.924 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.924 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.924 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.924 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:845 4.7 4:e41d47fb:::smithi03829856-14:head [write 2061132~557342] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:12.924 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:08.756359+0000", 2024-03-19T11:30:12.925 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.1461863049999996, 2024-03-19T11:30:12.925 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074752910000000002, 2024-03-19T11:30:12.925 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:12.925 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:12.925 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:12.925 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:12.925 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:12.925 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 845 2024-03-19T11:30:12.925 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.925 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:12.925 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.925 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:12.925 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756359+0000", 2024-03-19T11:30:12.925 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.926 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.926 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.926 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:12.926 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756359+0000", 2024-03-19T11:30:12.926 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.926 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.926 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.926 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:12.926 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756361+0000", 2024-03-19T11:30:12.926 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5280000000000002e-06 2024-03-19T11:30:12.926 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.926 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.926 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:12.926 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756535+0000", 2024-03-19T11:30:12.927 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000173697 2024-03-19T11:30:12.927 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.927 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.927 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:12.927 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756552+0000", 2024-03-19T11:30:12.927 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7397e-05 2024-03-19T11:30:12.927 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.927 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.927 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:12.927 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756561+0000", 2024-03-19T11:30:12.927 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8769999999999999e-06 2024-03-19T11:30:12.927 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.927 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.927 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:12.927 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756614+0000", 2024-03-19T11:30:12.928 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2425e-05 2024-03-19T11:30:12.928 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.928 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.928 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:12.928 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756693+0000", 2024-03-19T11:30:12.928 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9338999999999994e-05 2024-03-19T11:30:12.928 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.928 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.928 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:12.928 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756801+0000", 2024-03-19T11:30:12.928 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000108175 2024-03-19T11:30:12.928 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.928 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.928 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:12.929 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.758827+0000", 2024-03-19T11:30:12.929 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0020263379999999999 2024-03-19T11:30:12.929 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.929 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.929 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:12.929 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.763778+0000", 2024-03-19T11:30:12.929 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0049505740000000001 2024-03-19T11:30:12.929 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.929 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.929 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:12.929 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.763807+0000", 2024-03-19T11:30:12.929 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8702999999999999e-05 2024-03-19T11:30:12.929 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.929 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.929 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:12.930 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.763834+0000", 2024-03-19T11:30:12.930 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.7237999999999999e-05 2024-03-19T11:30:12.930 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.930 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.930 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.930 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.930 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.930 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:846 4.7 4:e41d47fb:::smithi03829856-14:head [setxattr _header (56) in=7b,truncate 2974806] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:12.930 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:08.756597+0000", 2024-03-19T11:30:12.930 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.1459474949999997, 2024-03-19T11:30:12.930 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0092954909999999995, 2024-03-19T11:30:12.930 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:12.930 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:12.930 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:12.931 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:12.931 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:12.931 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 846 2024-03-19T11:30:12.931 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.931 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:12.931 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.931 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:12.931 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756597+0000", 2024-03-19T11:30:12.931 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.931 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.931 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.931 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:12.931 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756597+0000", 2024-03-19T11:30:12.931 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.932 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.932 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.932 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:12.932 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756598+0000", 2024-03-19T11:30:12.932 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4600000000000004e-07 2024-03-19T11:30:12.932 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.932 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.932 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:12.932 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756604+0000", 2024-03-19T11:30:12.932 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5010000000000004e-06 2024-03-19T11:30:12.932 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.932 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.932 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:12.932 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756613+0000", 2024-03-19T11:30:12.933 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3999999999999998e-06 2024-03-19T11:30:12.933 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.933 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.933 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:12.933 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756620+0000", 2024-03-19T11:30:12.933 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.968e-06 2024-03-19T11:30:12.933 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.933 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.933 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:12.933 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.758321+0000", 2024-03-19T11:30:12.933 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001701337 2024-03-19T11:30:12.933 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.933 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.934 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:12.934 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.758414+0000", 2024-03-19T11:30:12.934 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2375999999999998e-05 2024-03-19T11:30:12.934 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.934 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.934 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:12.934 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.758553+0000", 2024-03-19T11:30:12.934 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013910300000000001 2024-03-19T11:30:12.934 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.934 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.934 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:12.934 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.758987+0000", 2024-03-19T11:30:12.934 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00043375100000000002 2024-03-19T11:30:12.935 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.935 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.935 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:12.935 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.765803+0000", 2024-03-19T11:30:12.935 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0068161089999999999 2024-03-19T11:30:12.935 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.935 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.935 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:12.935 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.765846+0000", 2024-03-19T11:30:12.935 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3226000000000003e-05 2024-03-19T11:30:12.935 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.935 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.935 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:12.935 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.765893+0000", 2024-03-19T11:30:12.936 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6974000000000001e-05 2024-03-19T11:30:12.936 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.936 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.936 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.936 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.936 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.936 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:847 4.7 4:e41d47fb:::smithi03829856-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:12.936 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:08.756635+0000", 2024-03-19T11:30:12.936 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.1459096329999996, 2024-03-19T11:30:12.936 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093966859999999996, 2024-03-19T11:30:12.936 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:12.936 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:12.936 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:12.936 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:12.937 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:12.937 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 847 2024-03-19T11:30:12.937 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.937 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:12.937 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.937 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:12.937 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756635+0000", 2024-03-19T11:30:12.937 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.937 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.937 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.937 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:12.937 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756635+0000", 2024-03-19T11:30:12.937 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.937 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.938 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.938 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:12.938 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756636+0000", 2024-03-19T11:30:12.938 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9500000000000002e-07 2024-03-19T11:30:12.938 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.938 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.938 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:12.938 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756640+0000", 2024-03-19T11:30:12.938 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7349999999999998e-06 2024-03-19T11:30:12.938 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.938 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.938 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:12.938 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756647+0000", 2024-03-19T11:30:12.938 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7130000000000005e-06 2024-03-19T11:30:12.939 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.939 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.939 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:12.939 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756653+0000", 2024-03-19T11:30:12.939 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3229999999999997e-06 2024-03-19T11:30:12.939 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.939 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.939 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:12.939 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.758756+0000", 2024-03-19T11:30:12.939 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0021037170000000002 2024-03-19T11:30:12.939 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.939 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.939 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-03-19T11:30:12.941 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.758814+0000", 2024-03-19T11:30:12.941 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7253999999999998e-05 2024-03-19T11:30:12.941 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.941 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.941 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:12.941 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.765926+0000", 2024-03-19T11:30:12.941 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0071118350000000004 2024-03-19T11:30:12.941 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.941 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.942 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:12.942 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.765970+0000", 2024-03-19T11:30:12.942 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4761000000000001e-05 2024-03-19T11:30:12.942 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.942 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.942 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:12.942 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.766032+0000", 2024-03-19T11:30:12.942 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1753000000000003e-05 2024-03-19T11:30:12.942 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.942 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.942 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.942 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.942 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.942 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:851 4.9 e32/21 4:953e720b:::smithi03829856-45:head v 32'30, mlcod=28'27)", 2024-03-19T11:30:12.943 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:08.774266+0000", 2024-03-19T11:30:12.943 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.1282787580000004, 2024-03-19T11:30:12.943 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0045649519999999997, 2024-03-19T11:30:12.943 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:12.943 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:12.943 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:12.943 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.943 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:12.943 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.774266+0000", 2024-03-19T11:30:12.943 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.943 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.943 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.943 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:12.943 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.774266+0000", 2024-03-19T11:30:12.943 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.944 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.944 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.944 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:12.944 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.774266+0000", 2024-03-19T11:30:12.944 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1199999999999999e-07 2024-03-19T11:30:12.944 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.944 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.944 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:12.944 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.775443+0000", 2024-03-19T11:30:12.944 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001177041 2024-03-19T11:30:12.944 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.944 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.944 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:12.944 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.775456+0000", 2024-03-19T11:30:12.945 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2527999999999999e-05 2024-03-19T11:30:12.945 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.945 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.945 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:12.945 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.775462+0000", 2024-03-19T11:30:12.945 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6840000000000001e-06 2024-03-19T11:30:12.945 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.945 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.945 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:12.945 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.775538+0000", 2024-03-19T11:30:12.945 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6068000000000004e-05 2024-03-19T11:30:12.945 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.945 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.945 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:12.946 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.775559+0000", 2024-03-19T11:30:12.946 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1362999999999999e-05 2024-03-19T11:30:12.946 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.946 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.946 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:12.946 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.778746+0000", 2024-03-19T11:30:12.946 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031867169999999999 2024-03-19T11:30:12.946 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.946 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.946 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:12.946 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.778831+0000", 2024-03-19T11:30:12.946 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5339000000000004e-05 2024-03-19T11:30:12.947 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.947 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.947 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.947 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.947 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.947 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:39 3.0 3:3dcd2456:::039914993b6f4082c204b24ae0ff925ebffccc22519067b64afa9eff107cf533:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:12.947 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.307505+0000", 2024-03-19T11:30:12.947 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.5950403419999999, 2024-03-19T11:30:12.947 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0052126840000000004, 2024-03-19T11:30:12.947 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:12.948 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:12.948 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:12.948 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.948 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:12.948 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.307505+0000", 2024-03-19T11:30:12.948 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.948 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.948 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.948 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:12.948 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.307505+0000", 2024-03-19T11:30:12.948 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.948 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.948 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.949 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:12.949 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.307507+0000", 2024-03-19T11:30:12.949 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1330000000000002e-06 2024-03-19T11:30:12.949 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.949 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.949 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:12.949 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308070+0000", 2024-03-19T11:30:12.949 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00056291999999999996 2024-03-19T11:30:12.949 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.949 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.949 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:12.949 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308093+0000", 2024-03-19T11:30:12.949 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3189000000000001e-05 2024-03-19T11:30:12.949 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.950 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.950 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:12.950 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308105+0000", 2024-03-19T11:30:12.950 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2398999999999999e-05 2024-03-19T11:30:12.950 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.950 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.950 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:12.950 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308223+0000", 2024-03-19T11:30:12.950 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000117895 2024-03-19T11:30:12.950 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.950 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.950 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:12.951 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308408+0000", 2024-03-19T11:30:12.951 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00018471199999999999 2024-03-19T11:30:12.951 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.951 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.951 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 2", 2024-03-19T11:30:12.951 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308691+0000", 2024-03-19T11:30:12.951 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00028322800000000001 2024-03-19T11:30:12.951 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.951 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.951 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:12.951 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.311665+0000", 2024-03-19T11:30:12.952 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029735460000000001 2024-03-19T11:30:12.952 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.952 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.952 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:12.952 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.312602+0000", 2024-03-19T11:30:12.952 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00093760600000000001 2024-03-19T11:30:12.952 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.952 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.952 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:12.952 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.312665+0000", 2024-03-19T11:30:12.952 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2472999999999996e-05 2024-03-19T11:30:12.952 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.952 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.952 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:12.953 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.312717+0000", 2024-03-19T11:30:12.953 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2583000000000001e-05 2024-03-19T11:30:12.953 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.953 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.953 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.953 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.953 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.953 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:40 3.0 3:0f3835fc:::79445f5816960fdb5fb7d7699594b1dfc30a0a44639e804aee69d6cb400c02a5:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:12.953 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.308283+0000", 2024-03-19T11:30:12.953 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.5942617070000002, 2024-03-19T11:30:12.953 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0048911880000000003, 2024-03-19T11:30:12.953 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:12.953 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:12.954 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:12.954 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.954 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:12.954 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308283+0000", 2024-03-19T11:30:12.954 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.954 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.954 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.954 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:12.954 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308283+0000", 2024-03-19T11:30:12.954 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.954 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.954 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.954 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:12.954 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308284+0000", 2024-03-19T11:30:12.955 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.147e-06 2024-03-19T11:30:12.955 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.955 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.955 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:12.955 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308524+0000", 2024-03-19T11:30:12.955 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023912400000000001 2024-03-19T11:30:12.955 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.955 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.955 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:12.955 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308537+0000", 2024-03-19T11:30:12.955 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3062e-05 2024-03-19T11:30:12.955 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.955 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.955 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:12.956 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308544+0000", 2024-03-19T11:30:12.956 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.841e-06 2024-03-19T11:30:12.956 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.956 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.956 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:12.956 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309919+0000", 2024-03-19T11:30:12.956 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0013743620000000001 2024-03-19T11:30:12.956 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.956 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.956 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:12.956 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310086+0000", 2024-03-19T11:30:12.956 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000167064 2024-03-19T11:30:12.956 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.956 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.956 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 2", 2024-03-19T11:30:12.957 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310466+0000", 2024-03-19T11:30:12.957 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00038026599999999998 2024-03-19T11:30:12.957 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.957 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.957 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:12.957 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.312788+0000", 2024-03-19T11:30:12.957 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023222899999999999 2024-03-19T11:30:12.957 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.957 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.957 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:12.957 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.313083+0000", 2024-03-19T11:30:12.957 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00029421699999999999 2024-03-19T11:30:12.957 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.958 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.958 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:12.958 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.313126+0000", 2024-03-19T11:30:12.958 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.354e-05 2024-03-19T11:30:12.958 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.958 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.958 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:12.958 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.313174+0000", 2024-03-19T11:30:12.958 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8275000000000001e-05 2024-03-19T11:30:12.958 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.958 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.958 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.958 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.958 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.959 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:51 3.0 3:0dfaac1c:::smithi03829856-20 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:12.959 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.322851+0000", 2024-03-19T11:30:12.959 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.5796943260000003, 2024-03-19T11:30:12.959 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0048459289999999997, 2024-03-19T11:30:12.959 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:12.959 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:12.959 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:12.959 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.959 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:12.959 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.322851+0000", 2024-03-19T11:30:12.959 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.959 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.959 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.959 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:12.959 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.322851+0000", 2024-03-19T11:30:12.960 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.960 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.960 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.960 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:12.960 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.322852+0000", 2024-03-19T11:30:12.960 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8299999999999995e-07 2024-03-19T11:30:12.960 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.960 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.960 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:12.960 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.322858+0000", 2024-03-19T11:30:12.960 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3999999999999997e-06 2024-03-19T11:30:12.960 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.960 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.960 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:12.961 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.322872+0000", 2024-03-19T11:30:12.961 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4487e-05 2024-03-19T11:30:12.961 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.961 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.961 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:12.961 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.322884+0000", 2024-03-19T11:30:12.961 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1195999999999999e-05 2024-03-19T11:30:12.961 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.961 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.961 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:12.961 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.325728+0000", 2024-03-19T11:30:12.961 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0028444149999999999 2024-03-19T11:30:12.961 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.962 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.962 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:12.962 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.325789+0000", 2024-03-19T11:30:12.962 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1029999999999997e-05 2024-03-19T11:30:12.962 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.962 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.962 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 2", 2024-03-19T11:30:12.962 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.325910+0000", 2024-03-19T11:30:12.962 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000120952 2024-03-19T11:30:12.962 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.962 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.962 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:12.962 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.326969+0000", 2024-03-19T11:30:12.963 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0010589060000000001 2024-03-19T11:30:12.963 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.963 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.963 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:12.963 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.327593+0000", 2024-03-19T11:30:12.963 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00062405700000000002 2024-03-19T11:30:12.963 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.963 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.963 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:12.963 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.327653+0000", 2024-03-19T11:30:12.963 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0368000000000002e-05 2024-03-19T11:30:12.963 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.963 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.963 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:12.964 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.327697+0000", 2024-03-19T11:30:12.964 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3235000000000002e-05 2024-03-19T11:30:12.964 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.964 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.964 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.964 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.964 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.964 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:997 4.4 4:27909373:::smithi03829856-46:head [append 0~0] snapc 5=[5,2] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-03-19T11:30:12.964 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.368093+0000", 2024-03-19T11:30:12.964 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.53445244199999997, 2024-03-19T11:30:12.964 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0048149320000000001, 2024-03-19T11:30:12.964 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:12.964 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:12.964 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:12.965 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:12.965 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:12.965 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 997 2024-03-19T11:30:12.965 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.965 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:12.965 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.965 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:12.965 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368093+0000", 2024-03-19T11:30:12.965 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.965 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.965 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.965 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:12.965 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368093+0000", 2024-03-19T11:30:12.965 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.965 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.966 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.966 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:12.966 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368095+0000", 2024-03-19T11:30:12.966 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8669999999999998e-06 2024-03-19T11:30:12.966 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.966 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.966 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:12.966 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368114+0000", 2024-03-19T11:30:12.966 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.853e-05 2024-03-19T11:30:12.966 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.966 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.966 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:12.966 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368146+0000", 2024-03-19T11:30:12.966 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2435999999999999e-05 2024-03-19T11:30:12.967 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.967 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.967 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:12.967 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368160+0000", 2024-03-19T11:30:12.967 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4025999999999999e-05 2024-03-19T11:30:12.967 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.967 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.967 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:12.967 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368245+0000", 2024-03-19T11:30:12.967 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4748999999999998e-05 2024-03-19T11:30:12.967 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.967 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.967 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:12.967 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368345+0000", 2024-03-19T11:30:12.968 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000100095 2024-03-19T11:30:12.968 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.968 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.968 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 2", 2024-03-19T11:30:12.968 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368493+0000", 2024-03-19T11:30:12.968 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014750700000000001 2024-03-19T11:30:12.968 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.968 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.968 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:12.968 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372784+0000", 2024-03-19T11:30:12.968 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0042915510000000002 2024-03-19T11:30:12.968 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.968 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.968 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:12.969 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372798+0000", 2024-03-19T11:30:12.969 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4204e-05 2024-03-19T11:30:12.969 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.969 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.969 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:12.969 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372866+0000", 2024-03-19T11:30:12.969 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7445000000000002e-05 2024-03-19T11:30:12.969 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.969 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.969 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:12.969 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372907+0000", 2024-03-19T11:30:12.969 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1522e-05 2024-03-19T11:30:12.969 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.969 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.970 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.970 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.970 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.970 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:998 4.4 4:27909373:::smithi03829856-46:head [rollback 2] snapc 5=[5,2] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-03-19T11:30:12.970 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.368196+0000", 2024-03-19T11:30:12.970 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.53434876899999995, 2024-03-19T11:30:12.970 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0079626200000000001, 2024-03-19T11:30:12.970 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:12.970 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:12.970 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:12.970 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:12.970 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:12.970 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 998 2024-03-19T11:30:12.970 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.971 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:12.971 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.971 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:12.971 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368196+0000", 2024-03-19T11:30:12.971 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.971 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.971 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.971 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:12.971 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368196+0000", 2024-03-19T11:30:12.971 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.971 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.971 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.971 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:12.971 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368197+0000", 2024-03-19T11:30:12.972 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6899999999999998e-07 2024-03-19T11:30:12.972 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.972 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.972 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:12.972 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368200+0000", 2024-03-19T11:30:12.972 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1389999999999999e-06 2024-03-19T11:30:12.972 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.972 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.972 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:12.972 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368207+0000", 2024-03-19T11:30:12.972 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9870000000000002e-06 2024-03-19T11:30:12.972 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.972 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.972 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:12.972 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368212+0000", 2024-03-19T11:30:12.973 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3820000000000003e-06 2024-03-19T11:30:12.973 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.973 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.973 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:12.973 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.369920+0000", 2024-03-19T11:30:12.973 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017078 2024-03-19T11:30:12.973 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.973 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.973 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:12.973 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370087+0000", 2024-03-19T11:30:12.973 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016742800000000001 2024-03-19T11:30:12.973 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.974 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.974 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 2", 2024-03-19T11:30:12.974 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370522+0000", 2024-03-19T11:30:12.974 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00043494300000000003 2024-03-19T11:30:12.974 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.974 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.974 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:12.974 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.375478+0000", 2024-03-19T11:30:12.974 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0049562240000000004 2024-03-19T11:30:12.974 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.974 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.974 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:12.974 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.375918+0000", 2024-03-19T11:30:12.974 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00043914699999999997 2024-03-19T11:30:12.975 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.975 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.975 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:12.975 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.376083+0000", 2024-03-19T11:30:12.975 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0001649 2024-03-19T11:30:12.975 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.975 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.975 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:12.975 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.376159+0000", 2024-03-19T11:30:12.975 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6301e-05 2024-03-19T11:30:12.975 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.975 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.975 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.975 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.976 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.976 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:999 4.4 4:27909373:::smithi03829856-46:head [append 0~0] snapc 5=[5,2] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-03-19T11:30:12.976 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.368233+0000", 2024-03-19T11:30:12.976 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.53431179399999995, 2024-03-19T11:30:12.976 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0081209219999999992, 2024-03-19T11:30:12.976 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:12.976 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:12.976 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:12.976 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:12.976 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:12.976 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 999 2024-03-19T11:30:12.976 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.976 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:12.976 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.977 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:12.977 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368233+0000", 2024-03-19T11:30:12.977 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.977 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.977 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.977 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:12.977 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368233+0000", 2024-03-19T11:30:12.977 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.977 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.977 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.977 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:12.977 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368234+0000", 2024-03-19T11:30:12.977 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5400000000000002e-07 2024-03-19T11:30:12.977 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.978 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.978 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:12.978 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368236+0000", 2024-03-19T11:30:12.978 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.7010000000000001e-06 2024-03-19T11:30:12.978 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.978 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.978 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:12.978 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368244+0000", 2024-03-19T11:30:12.978 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.289e-06 2024-03-19T11:30:12.978 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.978 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.978 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:12.978 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368248+0000", 2024-03-19T11:30:12.978 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9429999999999999e-06 2024-03-19T11:30:12.978 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.979 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.979 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:12.979 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372084+0000", 2024-03-19T11:30:12.979 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038354769999999999 2024-03-19T11:30:12.979 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.979 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.979 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:12.979 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372185+0000", 2024-03-19T11:30:12.979 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010088400000000001 2024-03-19T11:30:12.979 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.979 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.979 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 2", 2024-03-19T11:30:12.979 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372376+0000", 2024-03-19T11:30:12.979 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000191545 2024-03-19T11:30:12.980 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.980 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.980 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:12.980 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.376187+0000", 2024-03-19T11:30:12.980 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038108159999999999 2024-03-19T11:30:12.980 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.980 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.980 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:12.980 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.376243+0000", 2024-03-19T11:30:12.980 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6032000000000002e-05 2024-03-19T11:30:12.980 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.980 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.980 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:12.980 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.376304+0000", 2024-03-19T11:30:12.981 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0834000000000001e-05 2024-03-19T11:30:12.981 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.981 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.981 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:12.981 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.376354+0000", 2024-03-19T11:30:12.981 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0046999999999997e-05 2024-03-19T11:30:12.981 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.981 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.981 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.981 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.981 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.981 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1003 4.4 4:22709789:::smithi03829856-41:head [write 793297~789856,stat] snapc 5=[5,2] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-03-19T11:30:12.981 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.377573+0000", 2024-03-19T11:30:12.981 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.52497166799999995, 2024-03-19T11:30:12.981 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0060098649999999997, 2024-03-19T11:30:12.982 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:12.982 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:12.982 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:12.982 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:12.982 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:12.982 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1003 2024-03-19T11:30:12.982 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.982 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:12.982 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.982 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:12.982 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.377573+0000", 2024-03-19T11:30:12.982 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.982 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.982 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.983 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:12.983 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.377573+0000", 2024-03-19T11:30:12.983 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.983 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.983 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.983 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:12.983 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.377574+0000", 2024-03-19T11:30:12.983 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.192e-06 2024-03-19T11:30:12.983 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.983 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.983 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:12.983 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.377951+0000", 2024-03-19T11:30:12.983 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037638 2024-03-19T11:30:12.983 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.984 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.984 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:12.984 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.377965+0000", 2024-03-19T11:30:12.984 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4177e-05 2024-03-19T11:30:12.984 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.984 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.984 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:12.984 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.377972+0000", 2024-03-19T11:30:12.984 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5889999999999999e-06 2024-03-19T11:30:12.984 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.984 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.984 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:12.984 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.378047+0000", 2024-03-19T11:30:12.984 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5810999999999996e-05 2024-03-19T11:30:12.985 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.985 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.985 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:12.985 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.378181+0000", 2024-03-19T11:30:12.985 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000133485 2024-03-19T11:30:12.985 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.985 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.985 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 2", 2024-03-19T11:30:12.985 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.378419+0000", 2024-03-19T11:30:12.985 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023761100000000001 2024-03-19T11:30:12.985 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.985 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.985 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:12.985 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.382891+0000", 2024-03-19T11:30:12.986 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0044720180000000003 2024-03-19T11:30:12.986 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.986 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.986 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:12.986 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.383452+0000", 2024-03-19T11:30:12.986 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00056113599999999995 2024-03-19T11:30:12.986 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.986 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.986 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:12.986 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.383513+0000", 2024-03-19T11:30:12.986 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0850000000000002e-05 2024-03-19T11:30:12.986 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.986 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.986 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:12.987 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.383583+0000", 2024-03-19T11:30:12.987 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.0616000000000003e-05 2024-03-19T11:30:12.987 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.987 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.987 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.987 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.987 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.987 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1004 4.4 4:22709789:::smithi03829856-41:head [write 2305045~723014] snapc 5=[5,2] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-03-19T11:30:12.987 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.385935+0000", 2024-03-19T11:30:12.987 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.51661015700000001, 2024-03-19T11:30:12.987 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0051009560000000002, 2024-03-19T11:30:12.987 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:12.987 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:12.987 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:12.987 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:12.988 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:12.988 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1004 2024-03-19T11:30:12.988 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.988 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:12.988 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.988 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:12.988 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.385935+0000", 2024-03-19T11:30:12.988 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.988 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.988 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.988 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:12.988 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.385935+0000", 2024-03-19T11:30:12.988 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.989 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.989 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.989 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:12.989 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.385936+0000", 2024-03-19T11:30:12.989 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1620000000000001e-06 2024-03-19T11:30:12.989 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.989 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.989 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:12.989 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.386144+0000", 2024-03-19T11:30:12.989 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020755900000000001 2024-03-19T11:30:12.989 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.989 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.989 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:12.989 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.386161+0000", 2024-03-19T11:30:12.990 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7679000000000001e-05 2024-03-19T11:30:12.990 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.990 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.990 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:12.990 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.386171+0000", 2024-03-19T11:30:12.990 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.5480000000000007e-06 2024-03-19T11:30:12.990 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.990 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.990 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:12.990 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.386387+0000", 2024-03-19T11:30:12.990 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000216632 2024-03-19T11:30:12.990 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.990 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.990 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:12.991 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.386525+0000", 2024-03-19T11:30:12.991 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013713199999999999 2024-03-19T11:30:12.991 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.991 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.991 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 2", 2024-03-19T11:30:12.991 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.386728+0000", 2024-03-19T11:30:12.991 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020348799999999999 2024-03-19T11:30:12.991 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.991 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.991 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:12.991 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.390190+0000", 2024-03-19T11:30:12.991 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003462442 2024-03-19T11:30:12.991 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.991 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.992 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:12.992 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.390909+0000", 2024-03-19T11:30:12.992 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00071866300000000003 2024-03-19T11:30:12.992 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.992 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.992 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:12.992 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.390980+0000", 2024-03-19T11:30:12.992 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.0407000000000005e-05 2024-03-19T11:30:12.992 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.992 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.992 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:12.992 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.391036+0000", 2024-03-19T11:30:12.992 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6243999999999999e-05 2024-03-19T11:30:12.993 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.993 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.993 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.993 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.993 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.993 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1005 4.4 4:22709789:::smithi03829856-41:head [setxattr _header (56) in=7b,truncate 3193297] snapc 5=[5,2] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-03-19T11:30:12.993 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.386191+0000", 2024-03-19T11:30:12.993 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.51635421999999997, 2024-03-19T11:30:12.993 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050222280000000001, 2024-03-19T11:30:12.993 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:12.993 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:12.993 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:12.993 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:12.993 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:12.994 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1005 2024-03-19T11:30:12.994 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.994 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:12.994 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.994 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:12.994 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.386191+0000", 2024-03-19T11:30:12.994 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.994 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.994 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.994 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:12.994 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.386191+0000", 2024-03-19T11:30:12.994 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:12.994 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.994 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.995 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:12.995 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.386191+0000", 2024-03-19T11:30:12.995 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3399999999999999e-07 2024-03-19T11:30:12.995 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.995 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.995 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:12.995 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.386196+0000", 2024-03-19T11:30:12.995 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2370000000000003e-06 2024-03-19T11:30:12.995 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.995 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.995 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:12.995 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.386207+0000", 2024-03-19T11:30:12.995 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1453e-05 2024-03-19T11:30:12.995 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.995 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.996 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:12.996 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.386215+0000", 2024-03-19T11:30:12.996 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7819999999999994e-06 2024-03-19T11:30:12.996 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.996 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.996 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:12.996 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.389602+0000", 2024-03-19T11:30:12.996 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033873800000000002 2024-03-19T11:30:12.996 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.996 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.996 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:12.996 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.389722+0000", 2024-03-19T11:30:12.996 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000119923 2024-03-19T11:30:12.996 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.997 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.997 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 2", 2024-03-19T11:30:12.997 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.389921+0000", 2024-03-19T11:30:12.997 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000198927 2024-03-19T11:30:12.997 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.997 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.997 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:12.997 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.390492+0000", 2024-03-19T11:30:12.997 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00057140399999999997 2024-03-19T11:30:12.997 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.997 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.997 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:12.997 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.391126+0000", 2024-03-19T11:30:12.997 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00063363799999999995 2024-03-19T11:30:12.998 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.998 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.998 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:12.998 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.391159+0000", 2024-03-19T11:30:12.998 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2539000000000001e-05 2024-03-19T11:30:12.998 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.998 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.998 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:12.998 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.391213+0000", 2024-03-19T11:30:12.998 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4311000000000002e-05 2024-03-19T11:30:12.998 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.998 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:12.998 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:12.998 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.998 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.999 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1006 4.4 4:22709789:::smithi03829856-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-03-19T11:30:12.999 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.386381+0000", 2024-03-19T11:30:12.999 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.51616441800000001, 2024-03-19T11:30:12.999 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.004978957, 2024-03-19T11:30:12.999 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:12.999 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:12.999 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:12.999 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:12.999 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:12.999 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1006 2024-03-19T11:30:12.999 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:12.999 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:12.999 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:12.999 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:13.000 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.386381+0000", 2024-03-19T11:30:13.000 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:13.000 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.000 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.000 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:13.000 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.386381+0000", 2024-03-19T11:30:13.000 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:13.000 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.000 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.000 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:13.000 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.386381+0000", 2024-03-19T11:30:13.000 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.37e-07 2024-03-19T11:30:13.000 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.000 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.001 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:13.001 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.386385+0000", 2024-03-19T11:30:13.001 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3440000000000001e-06 2024-03-19T11:30:13.001 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.001 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.001 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:13.001 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.386394+0000", 2024-03-19T11:30:13.001 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6209999999999999e-06 2024-03-19T11:30:13.001 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.001 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.001 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:13.001 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.386404+0000", 2024-03-19T11:30:13.001 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9089999999999995e-06 2024-03-19T11:30:13.001 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.001 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.002 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:13.002 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.390222+0000", 2024-03-19T11:30:13.002 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038176550000000001 2024-03-19T11:30:13.002 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.002 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.002 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-03-19T11:30:13.002 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.390322+0000", 2024-03-19T11:30:13.002 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000100363 2024-03-19T11:30:13.002 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.002 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.002 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:13.002 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.391248+0000", 2024-03-19T11:30:13.003 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00092588100000000003 2024-03-19T11:30:13.003 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.003 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.003 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:13.003 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.391296+0000", 2024-03-19T11:30:13.003 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7586999999999997e-05 2024-03-19T11:30:13.003 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.003 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.003 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:13.003 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.391360+0000", 2024-03-19T11:30:13.003 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3860000000000002e-05 2024-03-19T11:30:13.003 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:13.003 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:13.003 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:13.003 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:13.004 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:13.004 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:13.005 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-19T11:30:13.006 INFO:tasks.rados.rados.0.smithi038.stdout:691: expect (ObjNum 55 snap 1 seq_num 55) 2024-03-19T11:30:13.006 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:13.012+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:13.046 INFO:tasks.rados.rados.0.smithi038.stdout:694: expect (ObjNum 11 snap 0 seq_num 11) 2024-03-19T11:30:13.106 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:13.119+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:13.113 INFO:tasks.rados.rados.0.smithi038.stdout:695: finishing set_chunk to oid 2 2024-03-19T11:30:13.114 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 2 set_chunk returned error code -95 offset: 601600 length: 207872 2024-03-19T11:30:13.114 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-03-19T11:30:13.114 INFO:tasks.rados.rados.0.smithi038.stdout:693: got (0) Success 2024-03-19T11:30:13.114 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 20 v 12 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-03-19T11:30:13.114 INFO:tasks.rados.rados.0.smithi038.stdout:696: finishing write tid 1 to smithi03829856-34 2024-03-19T11:30:13.114 INFO:tasks.rados.rados.0.smithi038.stdout:696: finishing write tid 2 to smithi03829856-34 2024-03-19T11:30:13.114 INFO:tasks.rados.rados.0.smithi038.stdout:696: finishing write tid 3 to smithi03829856-34 2024-03-19T11:30:13.114 INFO:tasks.rados.rados.0.smithi038.stdout:697: expect (ObjNum 28 snap 0 seq_num 28) 2024-03-19T11:30:13.163 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:13.163 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:30:13.163 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:30:13.163 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:13.177 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-19T11:30:13.193 INFO:tasks.rados.rados.0.smithi038.stdout:696: finishing write tid 4 to smithi03829856-34 2024-03-19T11:30:13.193 INFO:tasks.rados.rados.0.smithi038.stdout:696: finishing write tid 5 to smithi03829856-34 2024-03-19T11:30:13.193 INFO:tasks.rados.rados.0.smithi038.stdout:696: oid 34 updating version 0 to 19 2024-03-19T11:30:13.193 INFO:tasks.rados.rados.0.smithi038.stdout:696: oid 34 version 19 is already newer than 16 2024-03-19T11:30:13.193 INFO:tasks.rados.rados.0.smithi038.stdout:696: oid 34 version 19 is already newer than 18 2024-03-19T11:30:13.193 INFO:tasks.rados.rados.0.smithi038.stdout:696: oid 34 version 19 is already newer than 17 2024-03-19T11:30:13.193 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 34 v 19 (ObjNum 72 snap 5 seq_num 72) dirty exists 2024-03-19T11:30:13.194 INFO:tasks.rados.rados.0.smithi038.stdout:696: left oid 34 (ObjNum 72 snap 5 seq_num 72) 2024-03-19T11:30:13.194 INFO:tasks.rados.rados.0.smithi038.stdout:691: done (7 left) 2024-03-19T11:30:13.194 INFO:tasks.rados.rados.0.smithi038.stdout:692: done (6 left) 2024-03-19T11:30:13.194 INFO:tasks.rados.rados.0.smithi038.stdout:693: done (5 left) 2024-03-19T11:30:13.194 INFO:tasks.rados.rados.0.smithi038.stdout:694: done (4 left) 2024-03-19T11:30:13.194 INFO:tasks.rados.rados.0.smithi038.stdout:695: done (3 left) 2024-03-19T11:30:13.194 INFO:tasks.rados.rados.0.smithi038.stdout:696: done (2 left) 2024-03-19T11:30:13.194 INFO:tasks.rados.rados.0.smithi038.stdout:697: done (1 left) 2024-03-19T11:30:13.194 INFO:tasks.rados.rados.0.smithi038.stdout:698: done (0 left) 2024-03-19T11:30:13.194 INFO:tasks.rados.rados.0.smithi038.stdout:699: read oid 32 snap -1 2024-03-19T11:30:13.194 INFO:tasks.rados.rados.0.smithi038.stdout:699: expect (ObjNum 31 snap 0 seq_num 31) 2024-03-19T11:30:13.194 INFO:tasks.rados.rados.0.smithi038.stdout:700: read oid 35 snap -1 2024-03-19T11:30:13.194 INFO:tasks.rados.rados.0.smithi038.stdout:700: expect (ObjNum 34 snap 0 seq_num 34) 2024-03-19T11:30:13.195 INFO:tasks.rados.rados.0.smithi038.stdout:701: tier_promote oid 42 2024-03-19T11:30:13.195 INFO:tasks.rados.rados.0.smithi038.stdout:702: read oid 13 snap -1 2024-03-19T11:30:13.195 INFO:tasks.rados.rados.0.smithi038.stdout:702: expect (ObjNum 12 snap 0 seq_num 12) 2024-03-19T11:30:13.195 INFO:tasks.rados.rados.0.smithi038.stdout:703: set_chunk oid 33 target oid 2024-03-19T11:30:13.200 INFO:tasks.rados.rados.0.smithi038.stdout:703: set_chunk oid 33 offset: 848384 length: 446976 target oid bdb3b5647d9daba88ca1ec369a6a4ecc66592b64bbd6df6ad063af4137f6b5b0 offset: 0 snap -1 2024-03-19T11:30:13.200 INFO:tasks.rados.rados.0.smithi038.stdout:704: read oid 6 snap -1 2024-03-19T11:30:13.200 INFO:tasks.rados.rados.0.smithi038.stdout:704: expect (ObjNum 53 snap 0 seq_num 53) 2024-03-19T11:30:13.200 INFO:tasks.rados.rados.0.smithi038.stdout:705: write (excl) oid 18 current snap is 5 2024-03-19T11:30:13.200 INFO:tasks.rados.rados.0.smithi038.stdout:705: seq_num 74 ranges {443718=615603,1564138=470341,2633066=610652} 2024-03-19T11:30:13.206 INFO:tasks.rados.rados.0.smithi038.stdout:705: writing smithi03829856-18 from 443718 to 1059321 tid 1 2024-03-19T11:30:13.207 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:13.215+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:13.211 INFO:tasks.rados.rados.0.smithi038.stdout:705: writing smithi03829856-18 from 1564138 to 2034479 tid 2 2024-03-19T11:30:13.219 INFO:tasks.rados.rados.0.smithi038.stdout:705: writing smithi03829856-18 from 2633066 to 3243718 tid 3 2024-03-19T11:30:13.219 INFO:tasks.rados.rados.0.smithi038.stdout:706: delete oid 11 current snap is 5 2024-03-19T11:30:13.219 INFO:tasks.rados.rados.0.smithi038.stdout:699: expect (ObjNum 31 snap 0 seq_num 31) 2024-03-19T11:30:13.308 INFO:tasks.rados.rados.0.smithi038.stdout:701: got (0) Success 2024-03-19T11:30:13.308 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 42 v 28 (ObjNum 52 snap 0 seq_num 52) dirty exists 2024-03-19T11:30:13.308 INFO:tasks.rados.rados.0.smithi038.stdout:703: finishing set_chunk to oid 33 2024-03-19T11:30:13.308 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 33 set_chunk returned error code -95 offset: 848384 length: 446976 2024-03-19T11:30:13.308 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-03-19T11:30:13.308 INFO:tasks.rados.rados.0.smithi038.stdout:700: expect (ObjNum 34 snap 0 seq_num 34) 2024-03-19T11:30:13.309 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:13.309 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:30:13.309 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:30:13.309 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:30:13.309 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:13.309 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:13.316+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:13.321 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-19T11:30:13.387 INFO:tasks.rados.rados.0.smithi038.stdout:704: expect (ObjNum 53 snap 0 seq_num 53) 2024-03-19T11:30:13.410 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:13.419+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:13.488 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:13.488 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:30:13.488 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:30:13.488 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:30:13.488 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.488 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:9 4.d e28/21 4:b4ef97e0:::smithi03829856-3:head v 28'1, mlcod=0'0)", 2024-03-19T11:30:13.488 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.707728+0000", 2024-03-19T11:30:13.488 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.7908167240000008, 2024-03-19T11:30:13.488 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0037487950000000001, 2024-03-19T11:30:13.489 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:13.489 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:13.489 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:13.489 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.489 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:13.489 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.707728+0000", 2024-03-19T11:30:13.489 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:13.489 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.489 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.489 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:13.489 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.707728+0000", 2024-03-19T11:30:13.489 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:13.489 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.489 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.489 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:13.490 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.707729+0000", 2024-03-19T11:30:13.490 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6899999999999998e-07 2024-03-19T11:30:13.490 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.490 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.490 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:13.490 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.708625+0000", 2024-03-19T11:30:13.490 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00089603899999999995 2024-03-19T11:30:13.490 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.490 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.490 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:13.490 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.708645+0000", 2024-03-19T11:30:13.490 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9629000000000001e-05 2024-03-19T11:30:13.490 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.490 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.491 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:13.491 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.708657+0000", 2024-03-19T11:30:13.491 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2537999999999999e-05 2024-03-19T11:30:13.491 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.491 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.491 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:13.491 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.708740+0000", 2024-03-19T11:30:13.491 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2490999999999993e-05 2024-03-19T11:30:13.491 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.491 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.491 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:13.491 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.708774+0000", 2024-03-19T11:30:13.491 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4362000000000001e-05 2024-03-19T11:30:13.492 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.492 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.492 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:13.492 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.711419+0000", 2024-03-19T11:30:13.492 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0026446830000000001 2024-03-19T11:30:13.492 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.492 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.492 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:13.492 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.711477+0000", 2024-03-19T11:30:13.492 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8684000000000003e-05 2024-03-19T11:30:13.492 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:13.492 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:13.492 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:13.492 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.493 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.493 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:10 4.d e28/21 4:b4ef97e0:::smithi03829856-3:head v 28'2, mlcod=0'0)", 2024-03-19T11:30:13.493 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.713599+0000", 2024-03-19T11:30:13.493 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.7849462660000004, 2024-03-19T11:30:13.493 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002599905, 2024-03-19T11:30:13.493 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:13.493 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:13.493 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:13.493 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.493 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:13.493 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.713599+0000", 2024-03-19T11:30:13.493 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:13.493 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.493 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.493 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:13.494 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.713599+0000", 2024-03-19T11:30:13.494 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:13.494 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.494 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.494 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:13.494 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.713619+0000", 2024-03-19T11:30:13.494 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0398999999999999e-05 2024-03-19T11:30:13.494 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.494 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.494 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:13.494 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.713955+0000", 2024-03-19T11:30:13.494 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000335691 2024-03-19T11:30:13.494 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.494 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.495 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:13.495 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.713976+0000", 2024-03-19T11:30:13.495 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0866999999999999e-05 2024-03-19T11:30:13.495 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.495 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.495 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:13.495 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.713988+0000", 2024-03-19T11:30:13.495 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2228e-05 2024-03-19T11:30:13.495 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.495 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.495 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:13.495 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.714058+0000", 2024-03-19T11:30:13.495 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9868000000000003e-05 2024-03-19T11:30:13.495 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.495 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.496 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:13.496 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.714086+0000", 2024-03-19T11:30:13.496 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8124999999999999e-05 2024-03-19T11:30:13.496 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.496 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.496 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:13.496 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.716138+0000", 2024-03-19T11:30:13.496 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0020517759999999999 2024-03-19T11:30:13.496 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.496 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.496 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:13.496 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.716199+0000", 2024-03-19T11:30:13.496 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0951e-05 2024-03-19T11:30:13.496 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:13.496 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:13.497 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:13.497 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.497 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.497 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:13 4.4 e28/21 4:21eac9b2:::smithi03829856-4:head v 28'1, mlcod=0'0)", 2024-03-19T11:30:13.497 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.723258+0000", 2024-03-19T11:30:13.497 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.7752867289999994, 2024-03-19T11:30:13.497 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0048979840000000002, 2024-03-19T11:30:13.497 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:13.497 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:13.497 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:13.497 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.497 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:13.497 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723258+0000", 2024-03-19T11:30:13.497 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:13.497 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.498 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.498 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:13.498 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723258+0000", 2024-03-19T11:30:13.498 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:13.498 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.498 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.498 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:13.498 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723259+0000", 2024-03-19T11:30:13.498 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.7399999999999999e-07 2024-03-19T11:30:13.498 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.498 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.498 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:13.498 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723812+0000", 2024-03-19T11:30:13.498 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00055371000000000001 2024-03-19T11:30:13.499 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.499 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.499 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:13.499 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723833+0000", 2024-03-19T11:30:13.499 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0990000000000001e-05 2024-03-19T11:30:13.499 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.499 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.499 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:13.499 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723847+0000", 2024-03-19T11:30:13.499 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3774000000000001e-05 2024-03-19T11:30:13.499 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.499 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.499 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:13.499 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723912+0000", 2024-03-19T11:30:13.500 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4758000000000006e-05 2024-03-19T11:30:13.500 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.500 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.500 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:13.500 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723933+0000", 2024-03-19T11:30:13.500 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1046000000000001e-05 2024-03-19T11:30:13.500 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.500 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.500 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:13.500 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.728103+0000", 2024-03-19T11:30:13.500 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0041695079999999997 2024-03-19T11:30:13.500 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.500 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.500 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:13.500 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.728156+0000", 2024-03-19T11:30:13.501 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3924000000000001e-05 2024-03-19T11:30:13.501 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:13.501 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:13.501 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:13.501 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.501 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.501 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:14 4.4 e28/21 4:21eac9b2:::smithi03829856-4:head v 28'2, mlcod=0'0)", 2024-03-19T11:30:13.501 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.726103+0000", 2024-03-19T11:30:13.501 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.7724418790000005, 2024-03-19T11:30:13.501 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025805440000000002, 2024-03-19T11:30:13.501 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:13.501 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:13.501 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:13.501 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.501 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:13.502 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.726103+0000", 2024-03-19T11:30:13.502 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:13.502 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.502 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.502 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:13.502 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.726103+0000", 2024-03-19T11:30:13.502 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:13.502 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.502 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.502 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:13.502 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.726104+0000", 2024-03-19T11:30:13.502 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0920000000000001e-06 2024-03-19T11:30:13.502 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.503 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.503 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:13.503 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.726321+0000", 2024-03-19T11:30:13.503 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00021668499999999999 2024-03-19T11:30:13.503 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.503 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.503 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:13.503 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.726338+0000", 2024-03-19T11:30:13.503 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6382999999999999e-05 2024-03-19T11:30:13.503 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.503 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.503 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:13.503 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.726347+0000", 2024-03-19T11:30:13.503 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7399999999999999e-06 2024-03-19T11:30:13.504 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.504 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.504 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:13.504 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.726422+0000", 2024-03-19T11:30:13.504 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4365999999999999e-05 2024-03-19T11:30:13.504 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.504 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.504 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:13.504 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.726451+0000", 2024-03-19T11:30:13.504 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9085e-05 2024-03-19T11:30:13.504 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.504 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.504 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:13.504 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.728643+0000", 2024-03-19T11:30:13.504 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0021926620000000002 2024-03-19T11:30:13.505 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.505 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.505 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:13.505 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.728684+0000", 2024-03-19T11:30:13.505 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0531000000000002e-05 2024-03-19T11:30:13.505 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:13.505 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:13.505 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:13.505 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.505 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.505 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:33 4.d e28/21 4:b65e6658:::smithi03829856-9:head v 28'4, mlcod=28'2)", 2024-03-19T11:30:13.505 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.785780+0000", 2024-03-19T11:30:13.505 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.7127649579999993, 2024-03-19T11:30:13.505 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0032708569999999998, 2024-03-19T11:30:13.505 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:13.506 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:13.506 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:13.506 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.506 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:13.506 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.785780+0000", 2024-03-19T11:30:13.506 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:13.506 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.506 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.506 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:13.506 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.785780+0000", 2024-03-19T11:30:13.506 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:13.506 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.506 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.506 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:13.507 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.785783+0000", 2024-03-19T11:30:13.507 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4150000000000002e-06 2024-03-19T11:30:13.507 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.507 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.507 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:13.507 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.786482+0000", 2024-03-19T11:30:13.507 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00069957299999999999 2024-03-19T11:30:13.507 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.507 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.507 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:13.507 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.786507+0000", 2024-03-19T11:30:13.507 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5160999999999998e-05 2024-03-19T11:30:13.507 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.507 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.507 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:13.508 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.786521+0000", 2024-03-19T11:30:13.508 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3301e-05 2024-03-19T11:30:13.508 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.508 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.508 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:13.508 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.786599+0000", 2024-03-19T11:30:13.508 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7905999999999996e-05 2024-03-19T11:30:13.508 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.508 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.508 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:13.508 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.786630+0000", 2024-03-19T11:30:13.508 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1636e-05 2024-03-19T11:30:13.508 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.508 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.508 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:13.509 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.789001+0000", 2024-03-19T11:30:13.509 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023710530000000001 2024-03-19T11:30:13.509 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.509 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.509 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:13.509 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.789051+0000", 2024-03-19T11:30:13.509 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9811999999999997e-05 2024-03-19T11:30:13.509 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:13.509 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:13.509 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:13.509 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.509 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.509 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:34 4.d e28/21 4:b65e6658:::smithi03829856-9:head v 28'5, mlcod=28'2)", 2024-03-19T11:30:13.509 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.790321+0000", 2024-03-19T11:30:13.509 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.7082237550000006, 2024-03-19T11:30:13.510 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001391556, 2024-03-19T11:30:13.510 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:13.510 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:13.510 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:13.510 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.510 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:13.510 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.790321+0000", 2024-03-19T11:30:13.510 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:13.510 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.510 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.510 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:13.510 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.790321+0000", 2024-03-19T11:30:13.510 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:13.510 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.511 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.511 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:13.511 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.790322+0000", 2024-03-19T11:30:13.511 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0270000000000001e-06 2024-03-19T11:30:13.511 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.511 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.511 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:13.511 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.790505+0000", 2024-03-19T11:30:13.511 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00018226900000000001 2024-03-19T11:30:13.511 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.511 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.511 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:13.511 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.790518+0000", 2024-03-19T11:30:13.512 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3711e-05 2024-03-19T11:30:13.512 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.512 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.512 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:13.512 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.790527+0000", 2024-03-19T11:30:13.512 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9139999999999997e-06 2024-03-19T11:30:13.512 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.512 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.512 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:13.512 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.790582+0000", 2024-03-19T11:30:13.512 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4678000000000001e-05 2024-03-19T11:30:13.512 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.512 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.512 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:13.512 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.790603+0000", 2024-03-19T11:30:13.513 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0562000000000001e-05 2024-03-19T11:30:13.513 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.513 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.513 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:13.513 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.791674+0000", 2024-03-19T11:30:13.513 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0010713090000000001 2024-03-19T11:30:13.513 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.513 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.513 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:13.513 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.791713+0000", 2024-03-19T11:30:13.513 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.9085999999999998e-05 2024-03-19T11:30:13.513 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:13.513 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:13.513 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:13.513 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.514 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.514 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:871 4.d e32/21 4:b4ef97e0:::smithi03829856-3:head v 32'19, mlcod=28'16)", 2024-03-19T11:30:13.514 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.029331+0000", 2024-03-19T11:30:13.514 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.4692140560000002, 2024-03-19T11:30:13.514 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057816339999999999, 2024-03-19T11:30:13.514 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:13.514 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:13.514 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:13.514 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.514 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:13.514 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.029331+0000", 2024-03-19T11:30:13.514 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:13.515 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.515 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.515 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:13.515 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.029331+0000", 2024-03-19T11:30:13.515 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:13.515 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.515 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.515 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:13.515 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.029331+0000", 2024-03-19T11:30:13.515 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0400000000000002e-07 2024-03-19T11:30:13.515 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.515 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.515 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:13.516 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.030605+0000", 2024-03-19T11:30:13.516 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0012733899999999999 2024-03-19T11:30:13.516 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.516 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.516 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:13.516 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.030620+0000", 2024-03-19T11:30:13.516 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5557999999999999e-05 2024-03-19T11:30:13.516 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.516 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.516 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:13.516 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.030630+0000", 2024-03-19T11:30:13.516 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2660000000000007e-06 2024-03-19T11:30:13.516 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.516 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.517 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:13.517 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.030727+0000", 2024-03-19T11:30:13.517 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7452000000000001e-05 2024-03-19T11:30:13.517 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.517 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.517 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:13.517 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.030757+0000", 2024-03-19T11:30:13.517 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0018999999999998e-05 2024-03-19T11:30:13.517 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.517 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.517 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:13.517 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.035063+0000", 2024-03-19T11:30:13.517 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0043055350000000001 2024-03-19T11:30:13.517 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.517 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.518 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:13.518 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.035113+0000", 2024-03-19T11:30:13.518 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.011e-05 2024-03-19T11:30:13.518 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:13.518 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:13.518 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:13.518 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.518 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.518 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:872 4.d e32/21 4:b4ef97e0:::smithi03829856-3:head v 32'20, mlcod=28'16)", 2024-03-19T11:30:13.518 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.036163+0000", 2024-03-19T11:30:13.518 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.46238183, 2024-03-19T11:30:13.518 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029333250000000001, 2024-03-19T11:30:13.518 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:13.518 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:13.519 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:13.519 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.519 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:13.519 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.036163+0000", 2024-03-19T11:30:13.519 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:13.519 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.519 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.519 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:13.519 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.036163+0000", 2024-03-19T11:30:13.519 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:13.519 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.519 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.519 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:13.520 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.036164+0000", 2024-03-19T11:30:13.520 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7000000000000003e-07 2024-03-19T11:30:13.520 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.520 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.520 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:13.520 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.036721+0000", 2024-03-19T11:30:13.520 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00055646700000000005 2024-03-19T11:30:13.520 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.520 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.520 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:13.520 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.036732+0000", 2024-03-19T11:30:13.520 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1649e-05 2024-03-19T11:30:13.520 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.520 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.520 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:13.521 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.036739+0000", 2024-03-19T11:30:13.521 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6760000000000001e-06 2024-03-19T11:30:13.521 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.521 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.521 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:13.521 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.036785+0000", 2024-03-19T11:30:13.521 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6177000000000002e-05 2024-03-19T11:30:13.521 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.521 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.521 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:13.521 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.036808+0000", 2024-03-19T11:30:13.521 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2654999999999999e-05 2024-03-19T11:30:13.521 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.521 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.521 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:13.522 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.039022+0000", 2024-03-19T11:30:13.522 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022145530000000002 2024-03-19T11:30:13.522 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.522 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.522 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:13.522 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.039097+0000", 2024-03-19T11:30:13.522 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4277999999999996e-05 2024-03-19T11:30:13.522 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:13.522 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:13.522 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:13.522 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.522 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.522 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:893 3.0 e32/20 3:227a6c36:::c8c8170dc54fef3b4e25ae28106dacc8accf0657c6d6457bff8aa63e56c9678d:head v 32'52, mlcod=28'50)", 2024-03-19T11:30:13.523 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.291375+0000", 2024-03-19T11:30:13.523 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.2071698099999999, 2024-03-19T11:30:13.523 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0020008869999999998, 2024-03-19T11:30:13.523 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:13.523 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:13.523 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:13.523 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.523 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:13.523 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.291375+0000", 2024-03-19T11:30:13.523 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:13.523 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.523 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.523 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:13.523 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.291375+0000", 2024-03-19T11:30:13.524 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:13.524 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.524 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.524 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:13.524 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.291376+0000", 2024-03-19T11:30:13.524 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0200000000000003e-07 2024-03-19T11:30:13.524 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.524 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.524 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:13.524 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.291682+0000", 2024-03-19T11:30:13.524 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000306657 2024-03-19T11:30:13.524 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.524 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.524 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:13.525 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.291699+0000", 2024-03-19T11:30:13.525 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6291999999999999e-05 2024-03-19T11:30:13.525 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.525 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.525 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:13.525 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.291710+0000", 2024-03-19T11:30:13.525 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1117e-05 2024-03-19T11:30:13.525 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.525 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.525 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:13.525 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.291780+0000", 2024-03-19T11:30:13.525 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9751000000000004e-05 2024-03-19T11:30:13.525 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.526 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.526 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:13.526 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.291803+0000", 2024-03-19T11:30:13.526 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3787000000000001e-05 2024-03-19T11:30:13.526 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.526 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.526 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:13.526 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.293338+0000", 2024-03-19T11:30:13.526 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0015349649999999999 2024-03-19T11:30:13.526 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.526 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.526 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:13.527 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.293376+0000", 2024-03-19T11:30:13.527 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7916000000000001e-05 2024-03-19T11:30:13.527 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:13.527 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:13.527 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:13.527 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.527 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.527 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(osd.6.13:39 3.0 e32/20 3:3dcd2456:::039914993b6f4082c204b24ae0ff925ebffccc22519067b64afa9eff107cf533:head v 32'54, mlcod=32'52)", 2024-03-19T11:30:13.527 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.308854+0000", 2024-03-19T11:30:13.527 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.1896915410000002, 2024-03-19T11:30:13.527 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003468175, 2024-03-19T11:30:13.527 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:13.527 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:13.528 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:13.528 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.528 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:13.528 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308854+0000", 2024-03-19T11:30:13.528 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:13.528 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.528 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.528 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:13.528 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308854+0000", 2024-03-19T11:30:13.528 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:13.528 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.528 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.528 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:13.529 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308855+0000", 2024-03-19T11:30:13.529 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5999999999999999e-06 2024-03-19T11:30:13.529 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.529 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.529 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:13.529 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309019+0000", 2024-03-19T11:30:13.529 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000163311 2024-03-19T11:30:13.529 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.529 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.529 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:13.529 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309040+0000", 2024-03-19T11:30:13.529 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1795999999999999e-05 2024-03-19T11:30:13.529 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.530 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.530 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:13.530 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309060+0000", 2024-03-19T11:30:13.530 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9677000000000002e-05 2024-03-19T11:30:13.530 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.530 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.530 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:13.530 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309151+0000", 2024-03-19T11:30:13.530 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0507000000000006e-05 2024-03-19T11:30:13.530 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.530 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.530 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:13.530 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309187+0000", 2024-03-19T11:30:13.531 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6675999999999999e-05 2024-03-19T11:30:13.531 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.531 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.531 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:13.531 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.312256+0000", 2024-03-19T11:30:13.531 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030692060000000001 2024-03-19T11:30:13.531 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.531 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.531 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:13.531 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.312322+0000", 2024-03-19T11:30:13.531 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5401999999999995e-05 2024-03-19T11:30:13.531 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:13.531 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:13.532 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:13.532 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.532 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.532 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(osd.6.13:40 3.0 e32/20 3:0f3835fc:::79445f5816960fdb5fb7d7699594b1dfc30a0a44639e804aee69d6cb400c02a5:head v 32'55, mlcod=32'52)", 2024-03-19T11:30:13.532 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.310652+0000", 2024-03-19T11:30:13.532 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.1878936879999999, 2024-03-19T11:30:13.532 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022725919999999999, 2024-03-19T11:30:13.532 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:13.532 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:13.532 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:13.532 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.533 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:13.533 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310652+0000", 2024-03-19T11:30:13.533 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:13.533 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.533 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.533 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:13.533 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310652+0000", 2024-03-19T11:30:13.533 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:13.533 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.533 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.533 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:13.533 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310653+0000", 2024-03-19T11:30:13.534 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4339999999999999e-06 2024-03-19T11:30:13.534 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.534 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.534 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:13.534 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310764+0000", 2024-03-19T11:30:13.534 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000111174 2024-03-19T11:30:13.534 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.534 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.534 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:13.534 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310779+0000", 2024-03-19T11:30:13.534 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5024999999999999e-05 2024-03-19T11:30:13.534 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.534 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.535 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:13.535 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310789+0000", 2024-03-19T11:30:13.535 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0098999999999999e-05 2024-03-19T11:30:13.535 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.535 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.535 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:13.535 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310896+0000", 2024-03-19T11:30:13.535 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010627499999999999 2024-03-19T11:30:13.535 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.535 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.535 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:13.535 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310927+0000", 2024-03-19T11:30:13.535 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1078000000000003e-05 2024-03-19T11:30:13.535 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.535 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.536 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:13.536 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.312864+0000", 2024-03-19T11:30:13.536 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001937137 2024-03-19T11:30:13.536 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.536 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.536 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:13.536 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.312924+0000", 2024-03-19T11:30:13.536 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.037e-05 2024-03-19T11:30:13.536 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:13.536 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:13.536 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:13.536 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.536 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.536 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(osd.6.13:48 3.0 e32/20 3:0dfaac1c:::smithi03829856-20 low_tier:head v 32'58, mlcod=32'54)", 2024-03-19T11:30:13.536 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.324282+0000", 2024-03-19T11:30:13.537 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.1742633490000003, 2024-03-19T11:30:13.537 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017139049999999999, 2024-03-19T11:30:13.537 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:13.537 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:13.537 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:13.537 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.537 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:13.537 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.324282+0000", 2024-03-19T11:30:13.537 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:13.537 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.537 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.537 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:13.537 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.324282+0000", 2024-03-19T11:30:13.537 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:13.538 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.538 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.538 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:13.538 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.324283+0000", 2024-03-19T11:30:13.538 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0500000000000002e-07 2024-03-19T11:30:13.538 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.538 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.538 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:13.538 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.324292+0000", 2024-03-19T11:30:13.538 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.5270000000000004e-06 2024-03-19T11:30:13.538 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.538 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.538 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:13.538 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.324308+0000", 2024-03-19T11:30:13.539 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5920999999999999e-05 2024-03-19T11:30:13.539 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.539 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.539 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:13.539 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.324317+0000", 2024-03-19T11:30:13.539 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1239999999999994e-06 2024-03-19T11:30:13.539 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.539 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.539 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:13.539 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.324523+0000", 2024-03-19T11:30:13.539 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020557 2024-03-19T11:30:13.539 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.539 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.539 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:13.539 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.324536+0000", 2024-03-19T11:30:13.540 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3077e-05 2024-03-19T11:30:13.540 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.540 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.540 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:13.540 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.325937+0000", 2024-03-19T11:30:13.540 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014012720000000001 2024-03-19T11:30:13.540 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.540 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.540 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:13.540 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.325996+0000", 2024-03-19T11:30:13.540 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8508999999999998e-05 2024-03-19T11:30:13.540 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:13.540 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:13.540 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:13.540 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.541 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.541 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(osd.6.13:49 3.0 e32/20 3:0dfaac1c:::smithi03829856-20 low_tier:head v 32'59, mlcod=32'54)", 2024-03-19T11:30:13.541 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.324675+0000", 2024-03-19T11:30:13.541 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.1738704279999999, 2024-03-19T11:30:13.541 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001371389, 2024-03-19T11:30:13.541 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:13.541 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:13.541 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:13.541 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.541 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:13.541 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.324675+0000", 2024-03-19T11:30:13.541 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:13.541 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.541 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.542 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:13.542 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.324675+0000", 2024-03-19T11:30:13.542 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:13.542 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.542 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.542 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:13.542 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.324676+0000", 2024-03-19T11:30:13.542 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3300000000000001e-07 2024-03-19T11:30:13.542 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.542 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.542 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:13.542 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.324683+0000", 2024-03-19T11:30:13.542 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6820000000000003e-06 2024-03-19T11:30:13.542 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.542 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.543 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:13.543 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.324699+0000", 2024-03-19T11:30:13.543 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5605000000000001e-05 2024-03-19T11:30:13.543 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.543 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.543 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:13.543 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.324708+0000", 2024-03-19T11:30:13.543 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.064e-06 2024-03-19T11:30:13.543 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.543 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.543 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:13.543 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.324780+0000", 2024-03-19T11:30:13.543 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1574000000000003e-05 2024-03-19T11:30:13.543 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.543 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.544 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:13.544 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.324813+0000", 2024-03-19T11:30:13.544 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.341e-05 2024-03-19T11:30:13.544 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.544 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.544 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:13.544 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.326001+0000", 2024-03-19T11:30:13.544 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00118829 2024-03-19T11:30:13.544 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.544 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.544 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:13.544 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.326046+0000", 2024-03-19T11:30:13.544 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4931000000000001e-05 2024-03-19T11:30:13.544 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:13.545 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:13.545 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:13.545 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.545 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.545 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:901 4.d e32/21 4:ba44105a:::smithi03829856-26:head v 32'23, mlcod=32'20)", 2024-03-19T11:30:13.545 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.465629+0000", 2024-03-19T11:30:13.545 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.0329161779999998, 2024-03-19T11:30:13.545 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002624361, 2024-03-19T11:30:13.545 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:13.545 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:13.545 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:13.545 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.545 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:13.545 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.465629+0000", 2024-03-19T11:30:13.545 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:13.546 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.546 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.546 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:13.546 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.465629+0000", 2024-03-19T11:30:13.546 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:13.546 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.546 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.546 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:13.546 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.465631+0000", 2024-03-19T11:30:13.546 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5740000000000001e-06 2024-03-19T11:30:13.546 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.547 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.547 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:13.547 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.465642+0000", 2024-03-19T11:30:13.547 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1343e-05 2024-03-19T11:30:13.547 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.547 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.547 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:13.547 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.465663+0000", 2024-03-19T11:30:13.547 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1535e-05 2024-03-19T11:30:13.547 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.547 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.547 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:13.547 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.465676+0000", 2024-03-19T11:30:13.547 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2281e-05 2024-03-19T11:30:13.547 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.548 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.548 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:13.548 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.465777+0000", 2024-03-19T11:30:13.548 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010083500000000001 2024-03-19T11:30:13.548 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.548 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.548 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:13.548 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.465806+0000", 2024-03-19T11:30:13.548 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9414999999999999e-05 2024-03-19T11:30:13.548 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.548 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.548 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:13.548 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.468174+0000", 2024-03-19T11:30:13.548 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023684299999999999 2024-03-19T11:30:13.549 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.549 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.549 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:13.549 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.468253+0000", 2024-03-19T11:30:13.549 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8948000000000004e-05 2024-03-19T11:30:13.549 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:13.549 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:13.549 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:13.549 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.549 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.549 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:997 4.4 e34/21 4:27909373:::smithi03829856-46:head v 34'36, mlcod=32'32)", 2024-03-19T11:30:13.549 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.368638+0000", 2024-03-19T11:30:13.549 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 1.1299070010000001, 2024-03-19T11:30:13.549 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001933892, 2024-03-19T11:30:13.549 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:13.550 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:13.550 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:13.550 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.550 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:13.550 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368638+0000", 2024-03-19T11:30:13.550 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:13.550 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.550 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.550 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:13.550 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368638+0000", 2024-03-19T11:30:13.550 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:13.550 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.550 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.550 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:13.551 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368638+0000", 2024-03-19T11:30:13.551 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8900000000000001e-07 2024-03-19T11:30:13.551 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.551 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.551 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:13.551 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368644+0000", 2024-03-19T11:30:13.551 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3229999999999997e-06 2024-03-19T11:30:13.551 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.551 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.551 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:13.551 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368660+0000", 2024-03-19T11:30:13.551 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6344000000000001e-05 2024-03-19T11:30:13.551 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.551 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.551 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:13.552 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368671+0000", 2024-03-19T11:30:13.552 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1094000000000001e-05 2024-03-19T11:30:13.552 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.552 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.552 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:13.552 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368751+0000", 2024-03-19T11:30:13.552 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9357999999999995e-05 2024-03-19T11:30:13.552 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.552 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.552 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:13.552 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368776+0000", 2024-03-19T11:30:13.552 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5389e-05 2024-03-19T11:30:13.552 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.552 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.552 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:13.553 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370511+0000", 2024-03-19T11:30:13.553 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001735098 2024-03-19T11:30:13.553 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.553 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.553 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:13.553 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370572+0000", 2024-03-19T11:30:13.553 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1097000000000001e-05 2024-03-19T11:30:13.553 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:13.553 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:13.553 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:13.553 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.553 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.553 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:998 4.4 e34/21 4:27909373:::smithi03829856-46:head v 34'37, mlcod=32'32)", 2024-03-19T11:30:13.553 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.370628+0000", 2024-03-19T11:30:13.553 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 1.1279173280000001, 2024-03-19T11:30:13.554 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050953099999999996, 2024-03-19T11:30:13.554 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:13.554 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:13.554 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:13.554 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.554 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:13.554 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370628+0000", 2024-03-19T11:30:13.554 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:13.554 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.554 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.554 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:13.554 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370628+0000", 2024-03-19T11:30:13.554 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:13.554 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.555 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.555 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:13.555 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370629+0000", 2024-03-19T11:30:13.555 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.5900000000000005e-07 2024-03-19T11:30:13.555 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.555 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.555 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:13.555 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370638+0000", 2024-03-19T11:30:13.555 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8189999999999998e-06 2024-03-19T11:30:13.555 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.555 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.555 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:13.555 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370650+0000", 2024-03-19T11:30:13.555 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.275e-05 2024-03-19T11:30:13.556 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.556 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.556 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:13.556 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370658+0000", 2024-03-19T11:30:13.556 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5769999999999996e-06 2024-03-19T11:30:13.556 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.556 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.556 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:13.556 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370747+0000", 2024-03-19T11:30:13.556 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8642999999999998e-05 2024-03-19T11:30:13.556 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.556 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.556 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:13.556 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370796+0000", 2024-03-19T11:30:13.557 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9428000000000002e-05 2024-03-19T11:30:13.557 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.557 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.557 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:13.557 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.375658+0000", 2024-03-19T11:30:13.557 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0048621519999999998 2024-03-19T11:30:13.557 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.558 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.558 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:13.558 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.375723+0000", 2024-03-19T11:30:13.558 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4981999999999995e-05 2024-03-19T11:30:13.558 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:13.558 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:13.558 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:13.558 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.558 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.558 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:999 4.4 e34/21 4:27909373:::smithi03829856-46:head v 34'38, mlcod=32'32)", 2024-03-19T11:30:13.558 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.372487+0000", 2024-03-19T11:30:13.558 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 1.126058563, 2024-03-19T11:30:13.558 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0035853069999999998, 2024-03-19T11:30:13.559 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:13.559 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:13.559 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:13.559 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.559 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:13.559 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372487+0000", 2024-03-19T11:30:13.559 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:13.559 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.559 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.559 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:13.559 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372487+0000", 2024-03-19T11:30:13.559 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:13.559 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.559 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.559 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:13.560 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372488+0000", 2024-03-19T11:30:13.560 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1069999999999999e-06 2024-03-19T11:30:13.560 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.560 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.560 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:13.560 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372498+0000", 2024-03-19T11:30:13.560 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7510000000000003e-06 2024-03-19T11:30:13.560 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.560 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.560 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:13.560 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372544+0000", 2024-03-19T11:30:13.560 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6218999999999999e-05 2024-03-19T11:30:13.560 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.560 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.560 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:13.561 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372553+0000", 2024-03-19T11:30:13.561 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6320000000000003e-06 2024-03-19T11:30:13.561 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.561 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.561 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:13.561 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372636+0000", 2024-03-19T11:30:13.561 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3125000000000001e-05 2024-03-19T11:30:13.561 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.561 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.561 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:13.561 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372651+0000", 2024-03-19T11:30:13.561 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4676e-05 2024-03-19T11:30:13.561 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.561 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.561 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:13.562 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.376026+0000", 2024-03-19T11:30:13.562 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033745630000000001 2024-03-19T11:30:13.562 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.562 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.562 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:13.562 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.376072+0000", 2024-03-19T11:30:13.562 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6233999999999998e-05 2024-03-19T11:30:13.562 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:13.562 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:13.562 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:13.562 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.562 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.562 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:1003 4.4 e34/21 4:22709789:::smithi03829856-41:head v 34'40, mlcod=34'37)", 2024-03-19T11:30:13.562 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.378551+0000", 2024-03-19T11:30:13.563 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 1.119994476, 2024-03-19T11:30:13.563 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0041156680000000003, 2024-03-19T11:30:13.563 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:13.563 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:13.563 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:13.563 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.563 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:13.563 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.378551+0000", 2024-03-19T11:30:13.563 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:13.563 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.563 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.563 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:13.563 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.378551+0000", 2024-03-19T11:30:13.563 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:13.563 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.564 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.564 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:13.564 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.378552+0000", 2024-03-19T11:30:13.564 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0400000000000005e-07 2024-03-19T11:30:13.564 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.564 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.564 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:13.564 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.378929+0000", 2024-03-19T11:30:13.564 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037751700000000001 2024-03-19T11:30:13.564 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.564 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.564 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:13.564 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.378944+0000", 2024-03-19T11:30:13.564 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5038000000000001e-05 2024-03-19T11:30:13.564 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.565 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.565 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:13.565 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.378953+0000", 2024-03-19T11:30:13.565 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7649999999999999e-06 2024-03-19T11:30:13.565 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.565 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.565 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:13.565 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.379017+0000", 2024-03-19T11:30:13.565 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3614000000000005e-05 2024-03-19T11:30:13.565 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.565 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.565 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:13.565 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.379042+0000", 2024-03-19T11:30:13.565 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5910000000000001e-05 2024-03-19T11:30:13.565 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.566 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.566 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:13.566 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.382610+0000", 2024-03-19T11:30:13.566 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0035678340000000002 2024-03-19T11:30:13.566 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.566 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.566 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:13.566 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.382666+0000", 2024-03-19T11:30:13.566 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6085999999999998e-05 2024-03-19T11:30:13.566 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:13.566 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:13.566 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:13.566 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.566 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.567 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:1004 4.4 e34/21 4:22709789:::smithi03829856-41:head v 34'41, mlcod=34'38)", 2024-03-19T11:30:13.567 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.386855+0000", 2024-03-19T11:30:13.567 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 1.111690667, 2024-03-19T11:30:13.567 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0039133320000000003, 2024-03-19T11:30:13.567 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:13.567 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:13.567 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:13.567 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.567 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:13.567 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.386855+0000", 2024-03-19T11:30:13.567 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:13.567 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.567 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.567 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:13.567 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.386855+0000", 2024-03-19T11:30:13.568 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:13.568 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.568 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.568 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:13.568 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.386856+0000", 2024-03-19T11:30:13.568 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.528e-06 2024-03-19T11:30:13.568 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.568 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.568 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:13.568 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.387143+0000", 2024-03-19T11:30:13.568 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00028658600000000002 2024-03-19T11:30:13.568 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.568 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.568 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:13.569 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.387159+0000", 2024-03-19T11:30:13.569 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6796e-05 2024-03-19T11:30:13.569 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.569 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.569 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:13.569 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.387167+0000", 2024-03-19T11:30:13.569 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2729999999999997e-06 2024-03-19T11:30:13.569 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.569 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.569 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:13.569 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.387230+0000", 2024-03-19T11:30:13.569 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2862999999999998e-05 2024-03-19T11:30:13.569 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.569 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.570 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:13.570 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.387255+0000", 2024-03-19T11:30:13.570 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4905e-05 2024-03-19T11:30:13.570 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.570 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.570 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:13.570 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.390720+0000", 2024-03-19T11:30:13.570 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034652699999999999 2024-03-19T11:30:13.570 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.570 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.570 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:13.570 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.390768+0000", 2024-03-19T11:30:13.570 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8111000000000001e-05 2024-03-19T11:30:13.570 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:13.570 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:13.571 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:13.571 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.571 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.571 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:1042 3.0 e35/20 3:0e3fb112:::bdb3b5647d9daba88ca1ec369a6a4ecc66592b64bbd6df6ad063af4137f6b5b0:head v 35'62, mlcod=32'60)", 2024-03-19T11:30:13.571 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.211286+0000", 2024-03-19T11:30:13.571 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.28725883299999999, 2024-03-19T11:30:13.571 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0024813159999999999, 2024-03-19T11:30:13.571 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:13.571 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:13.571 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:13.571 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.571 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:13.571 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.211286+0000", 2024-03-19T11:30:13.571 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:13.572 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.572 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.572 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:13.572 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.211286+0000", 2024-03-19T11:30:13.572 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:13.572 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.572 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.572 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:13.572 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.211287+0000", 2024-03-19T11:30:13.572 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.67e-07 2024-03-19T11:30:13.572 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.572 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.572 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:13.572 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.211562+0000", 2024-03-19T11:30:13.572 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00027568999999999998 2024-03-19T11:30:13.572 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.573 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.573 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:13.573 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.211579+0000", 2024-03-19T11:30:13.573 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6489000000000001e-05 2024-03-19T11:30:13.573 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.573 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.573 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:13.573 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.211593+0000", 2024-03-19T11:30:13.573 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4351e-05 2024-03-19T11:30:13.573 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.573 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.573 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:13.573 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.211678+0000", 2024-03-19T11:30:13.573 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4924000000000002e-05 2024-03-19T11:30:13.574 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.574 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.574 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:13.574 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.211713+0000", 2024-03-19T11:30:13.574 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4492000000000001e-05 2024-03-19T11:30:13.574 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.574 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.574 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:13.574 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.213730+0000", 2024-03-19T11:30:13.574 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002017114 2024-03-19T11:30:13.574 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:13.574 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:13.574 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:13.574 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.213768+0000", 2024-03-19T11:30:13.574 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7988999999999998e-05 2024-03-19T11:30:13.575 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:13.575 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:13.575 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:13.575 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:13.575 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:13.575 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:13.576 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-19T11:30:13.577 INFO:tasks.rados.rados.0.smithi038.stdout:702: expect (ObjNum 12 snap 0 seq_num 12) 2024-03-19T11:30:13.577 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:13.587+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:13.582 INFO:tasks.rados.rados.0.smithi038.stdout:705: finishing write tid 1 to smithi03829856-18 2024-03-19T11:30:13.583 INFO:tasks.rados.rados.0.smithi038.stdout:705: finishing write tid 2 to smithi03829856-18 2024-03-19T11:30:13.583 INFO:tasks.rados.rados.0.smithi038.stdout:705: finishing write tid 3 to smithi03829856-18 2024-03-19T11:30:13.583 INFO:tasks.rados.rados.0.smithi038.stdout:705: finishing write tid 4 to smithi03829856-18 2024-03-19T11:30:13.583 INFO:tasks.rados.rados.0.smithi038.stdout:705: finishing write tid 5 to smithi03829856-18 2024-03-19T11:30:13.583 INFO:tasks.rados.rados.0.smithi038.stdout:705: oid 18 updating version 0 to 34 2024-03-19T11:30:13.583 INFO:tasks.rados.rados.0.smithi038.stdout:705: oid 18 updating version 34 to 35 2024-03-19T11:30:13.583 INFO:tasks.rados.rados.0.smithi038.stdout:705: oid 18 version 35 is already newer than 32 2024-03-19T11:30:13.583 INFO:tasks.rados.rados.0.smithi038.stdout:705: oid 18 version 35 is already newer than 33 2024-03-19T11:30:13.583 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 18 v 35 (ObjNum 74 snap 5 seq_num 74) dirty exists 2024-03-19T11:30:13.583 INFO:tasks.rados.rados.0.smithi038.stdout:705: left oid 18 (ObjNum 74 snap 5 seq_num 74) 2024-03-19T11:30:13.583 INFO:tasks.rados.rados.0.smithi038.stdout:699: done (7 left) 2024-03-19T11:30:13.583 INFO:tasks.rados.rados.0.smithi038.stdout:700: done (6 left) 2024-03-19T11:30:13.583 INFO:tasks.rados.rados.0.smithi038.stdout:701: done (5 left) 2024-03-19T11:30:13.584 INFO:tasks.rados.rados.0.smithi038.stdout:702: done (4 left) 2024-03-19T11:30:13.584 INFO:tasks.rados.rados.0.smithi038.stdout:703: done (3 left) 2024-03-19T11:30:13.584 INFO:tasks.rados.rados.0.smithi038.stdout:704: done (2 left) 2024-03-19T11:30:13.584 INFO:tasks.rados.rados.0.smithi038.stdout:705: done (1 left) 2024-03-19T11:30:13.584 INFO:tasks.rados.rados.0.smithi038.stdout:706: done (0 left) 2024-03-19T11:30:13.584 INFO:tasks.rados.rados.0.smithi038.stdout:707: snap_create 2024-03-19T11:30:13.677 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:13.684+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:13.703 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:13.703 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:30:13.703 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:30:13.703 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:13.716 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-19T11:30:13.738 INFO:teuthology.orchestra.run.smithi038.stderr:set 4.c pg_upmap_items mapping to [3->4,0->1] 2024-03-19T11:30:13.739 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:13.747+0000 7fb17dd72640 -1 osd.4 35 handle_osd_map injecting map crc failure 2024-03-19T11:30:13.747 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:13.756+0000 7f6712f92640 -1 osd.2 35 handle_osd_map injecting map crc failure 2024-03-19T11:30:13.751 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-03-19T11:30:13.754 INFO:tasks.rados.rados.0.smithi038.stdout:707: done (0 left) 2024-03-19T11:30:13.754 INFO:tasks.rados.rados.0.smithi038.stdout:708: read oid 16 snap 4 2024-03-19T11:30:13.754 INFO:tasks.rados.rados.0.smithi038.stdout:708: expect (ObjNum 65 snap 3 seq_num 65) 2024-03-19T11:30:13.754 INFO:tasks.rados.rados.0.smithi038.stdout:709: rollback oid 31 current snap is 6 2024-03-19T11:30:13.754 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 31 to 3 2024-03-19T11:30:13.754 INFO:tasks.rados.rados.0.smithi038.stdout:710: set_chunk oid 32 target oid 2024-03-19T11:30:13.754 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:13.764+0000 7f6a7f9da640 -1 osd.1 35 handle_osd_map injecting map crc failure 2024-03-19T11:30:13.770 INFO:tasks.rados.rados.0.smithi038.stdout:710: set_chunk oid 32 offset: 1154048 length: 508928 target oid 727f83c09505a9e6c5af0b055849bb93e385b34c7319dfb26afbcd3c5ae450db offset: 0 snap -1 2024-03-19T11:30:13.771 INFO:tasks.rados.rados.0.smithi038.stdout:711: tier_flush oid 47 2024-03-19T11:30:13.771 INFO:tasks.rados.rados.0.smithi038.stdout:711: tier_flush oid 47 snap -1 2024-03-19T11:30:13.771 INFO:tasks.rados.rados.0.smithi038.stdout:709: finishing rollback tid 0 to smithi03829856-31 2024-03-19T11:30:13.771 INFO:tasks.rados.rados.0.smithi038.stdout:712: read oid 30 snap -1 2024-03-19T11:30:13.771 INFO:tasks.rados.rados.0.smithi038.stdout:712: expect (ObjNum 29 snap 0 seq_num 29) 2024-03-19T11:30:13.771 INFO:tasks.rados.rados.0.smithi038.stdout:713: write oid 19 current snap is 6 2024-03-19T11:30:13.771 INFO:tasks.rados.rados.0.smithi038.stdout:709: finishing rollback tid 1 to smithi03829856-31 2024-03-19T11:30:13.771 INFO:tasks.rados.rados.0.smithi038.stdout:709: finishing rollback tid 2 to smithi03829856-31 2024-03-19T11:30:13.771 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 31 v 36 (ObjNum 56 snap 1 seq_num 56) dirty exists 2024-03-19T11:30:13.771 INFO:tasks.rados.rados.0.smithi038.stdout:708: expect (ObjNum 65 snap 3 seq_num 65) 2024-03-19T11:30:13.779 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:13.787+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:13.871 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:13.871 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:30:13.872 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:30:13.872 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:30:13.872 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:13.879 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:13.888+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:13.888 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-19T11:30:13.900 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-19T11:30:13.927 INFO:tasks.rados.rados.0.smithi038.stdout:710: finishing set_chunk to oid 32 2024-03-19T11:30:13.928 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 32 set_chunk returned error code -95 offset: 1154048 length: 508928 2024-03-19T11:30:13.928 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-03-19T11:30:13.928 INFO:tasks.rados.rados.0.smithi038.stdout:713: seq_num 76 ranges {477332=766101,1681102=474010,2687912=589420} 2024-03-19T11:30:13.935 INFO:tasks.rados.rados.0.smithi038.stdout:713: writing smithi03829856-19 from 477332 to 1243433 tid 1 2024-03-19T11:30:13.941 INFO:tasks.rados.rados.0.smithi038.stdout:713: writing smithi03829856-19 from 1681102 to 2155112 tid 2 2024-03-19T11:30:13.949 INFO:tasks.rados.rados.0.smithi038.stdout:713: writing smithi03829856-19 from 2687912 to 3277332 tid 3 2024-03-19T11:30:13.949 INFO:tasks.rados.rados.0.smithi038.stdout:708: done (5 left) 2024-03-19T11:30:13.949 INFO:tasks.rados.rados.0.smithi038.stdout:709: done (4 left) 2024-03-19T11:30:13.950 INFO:tasks.rados.rados.0.smithi038.stdout:710: done (3 left) 2024-03-19T11:30:13.950 INFO:tasks.rados.rados.0.smithi038.stdout:714: write (excl) oid 43 current snap is 6 2024-03-19T11:30:13.950 INFO:tasks.rados.rados.0.smithi038.stdout:714: seq_num 77 ranges {494139=557703,1455937=734021,2948041=346098} 2024-03-19T11:30:13.956 INFO:tasks.rados.rados.0.smithi038.stdout:714: writing smithi03829856-43 from 494139 to 1051842 tid 1 2024-03-19T11:30:13.964 INFO:tasks.rados.rados.0.smithi038.stdout:714: writing smithi03829856-43 from 1455937 to 2189958 tid 2 2024-03-19T11:30:13.971 INFO:tasks.rados.rados.0.smithi038.stdout:714: writing smithi03829856-43 from 2948041 to 3294139 tid 3 2024-03-19T11:30:13.971 INFO:tasks.rados.rados.0.smithi038.stdout:715: write (excl) oid 14 current snap is 6 2024-03-19T11:30:13.971 INFO:tasks.rados.rados.0.smithi038.stdout:715: seq_num 78 ranges {510946=632952,1798066=477679,2742758=457217} 2024-03-19T11:30:13.978 INFO:tasks.rados.rados.0.smithi038.stdout:715: writing smithi03829856-14 from 510946 to 1143898 tid 1 2024-03-19T11:30:13.980 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:13.991+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:13.985 INFO:tasks.rados.rados.0.smithi038.stdout:715: writing smithi03829856-14 from 1798066 to 2275745 tid 2 2024-03-19T11:30:13.992 INFO:tasks.rados.rados.0.smithi038.stdout:715: writing smithi03829856-14 from 2742758 to 3199975 tid 3 2024-03-19T11:30:13.992 INFO:tasks.rados.rados.0.smithi038.stdout:716: read oid 42 snap -1 2024-03-19T11:30:13.992 INFO:tasks.rados.rados.0.smithi038.stdout:716: expect (ObjNum 52 snap 0 seq_num 52) 2024-03-19T11:30:13.992 INFO:tasks.rados.rados.0.smithi038.stdout:717: read oid 2 snap 0 2024-03-19T11:30:13.992 INFO:tasks.rados.rados.0.smithi038.stdout:717: expect (ObjNum 1 snap 0 seq_num 1) 2024-03-19T11:30:13.993 INFO:tasks.rados.rados.0.smithi038.stdout:718: read oid 38 snap -1 2024-03-19T11:30:13.993 INFO:tasks.rados.rados.0.smithi038.stdout:718: expect (ObjNum 68 snap 3 seq_num 68) 2024-03-19T11:30:13.993 INFO:tasks.rados.rados.0.smithi038.stdout:719: write oid 25 current snap is 6 2024-03-19T11:30:13.993 INFO:tasks.rados.rados.0.smithi038.stdout:719: seq_num 79 ranges {527753=708201,1856548=737690,3286534=41219} 2024-03-19T11:30:14.000 INFO:tasks.rados.rados.0.smithi038.stdout:719: writing smithi03829856-25 from 527753 to 1235954 tid 1 2024-03-19T11:30:14.008 INFO:tasks.rados.rados.0.smithi038.stdout:719: writing smithi03829856-25 from 1856548 to 2594238 tid 2 2024-03-19T11:30:14.013 INFO:tasks.rados.rados.0.smithi038.stdout:719: writing smithi03829856-25 from 3286534 to 3327753 tid 3 2024-03-19T11:30:14.013 INFO:tasks.rados.rados.0.smithi038.stdout:720: set_chunk oid 44 target oid 2024-03-19T11:30:14.019 INFO:tasks.rados.rados.0.smithi038.stdout:720: set_chunk oid 44 offset: 420864 length: 270336 target oid 1dbe3ea172a960421ded4894bb5873096352e4b1c590a896121b72efea9a7be1 offset: 0 snap 5 2024-03-19T11:30:14.019 INFO:tasks.rados.rados.0.smithi038.stdout:721: tier_promote oid 3 2024-03-19T11:30:14.019 INFO:tasks.rados.rados.0.smithi038.stdout:712: expect (ObjNum 29 snap 0 seq_num 29) 2024-03-19T11:30:14.070 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:30:14.071 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:30:14.081 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:14.091+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:14.086 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:30:14.087 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:30:14.090 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:14.090 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:30:14.090 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:30:14.090 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:30:14.090 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.090 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:875 4.c e32/21 4:3c71e808:::smithi03829856-34:head v 32'9, mlcod=28'6)", 2024-03-19T11:30:14.090 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.042270+0000", 2024-03-19T11:30:14.090 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.060220202, 2024-03-19T11:30:14.090 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0045267149999999997, 2024-03-19T11:30:14.090 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:14.090 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:14.090 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:14.091 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.091 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:14.091 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.042270+0000", 2024-03-19T11:30:14.091 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:14.091 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.091 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.091 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:14.091 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.042270+0000", 2024-03-19T11:30:14.091 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:14.091 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.091 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.091 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:14.091 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.042271+0000", 2024-03-19T11:30:14.091 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0900000000000001e-07 2024-03-19T11:30:14.092 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.092 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.092 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:14.092 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.043590+0000", 2024-03-19T11:30:14.092 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0013197440000000001 2024-03-19T11:30:14.092 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.092 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.092 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:14.092 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.043604+0000", 2024-03-19T11:30:14.092 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.326e-05 2024-03-19T11:30:14.092 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.092 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.092 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:14.092 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.043621+0000", 2024-03-19T11:30:14.092 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7353000000000001e-05 2024-03-19T11:30:14.093 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.093 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.093 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:14.093 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.043699+0000", 2024-03-19T11:30:14.093 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7729000000000001e-05 2024-03-19T11:30:14.093 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.093 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.093 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:14.093 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.043722+0000", 2024-03-19T11:30:14.093 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2914000000000001e-05 2024-03-19T11:30:14.093 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.093 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.093 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:14.093 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.046748+0000", 2024-03-19T11:30:14.093 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030260399999999998 2024-03-19T11:30:14.094 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.094 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.094 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:14.094 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.046797+0000", 2024-03-19T11:30:14.094 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9066000000000001e-05 2024-03-19T11:30:14.094 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:14.094 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:14.094 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:14.094 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.094 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.094 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:876 4.c e32/21 4:3c71e808:::smithi03829856-34:head v 32'10, mlcod=28'7)", 2024-03-19T11:30:14.094 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.048779+0000", 2024-03-19T11:30:14.094 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.0537116490000002, 2024-03-19T11:30:14.095 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0032962239999999999, 2024-03-19T11:30:14.095 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:14.095 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:14.095 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:14.095 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.095 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:14.095 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.048779+0000", 2024-03-19T11:30:14.095 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:14.095 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.095 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.095 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:14.095 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.048779+0000", 2024-03-19T11:30:14.095 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:14.095 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.095 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.095 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:14.096 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.048780+0000", 2024-03-19T11:30:14.096 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9199999999999999e-07 2024-03-19T11:30:14.096 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.096 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.096 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:14.096 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.049469+0000", 2024-03-19T11:30:14.096 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00068986600000000003 2024-03-19T11:30:14.096 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.096 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.096 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:14.096 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.049483+0000", 2024-03-19T11:30:14.096 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3808999999999999e-05 2024-03-19T11:30:14.097 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.097 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.097 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:14.097 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.049493+0000", 2024-03-19T11:30:14.097 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3559999999999999e-06 2024-03-19T11:30:14.097 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.097 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.097 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:14.097 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.049563+0000", 2024-03-19T11:30:14.097 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.0741999999999995e-05 2024-03-19T11:30:14.097 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.097 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.097 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:14.097 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.049584+0000", 2024-03-19T11:30:14.097 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0601000000000001e-05 2024-03-19T11:30:14.098 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.098 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.098 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:14.098 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.052026+0000", 2024-03-19T11:30:14.098 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002442371 2024-03-19T11:30:14.098 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.098 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.098 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:14.098 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.052075+0000", 2024-03-19T11:30:14.098 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8587000000000001e-05 2024-03-19T11:30:14.098 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:14.098 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:14.098 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:14.098 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.099 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.099 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:38 3.2 3:5595a68d:::07854d2fef297a06ba81685e660c332de36d5d18d546927d30daad6d7fda1541:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:14.099 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.307388+0000", 2024-03-19T11:30:14.099 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.7951019630000005, 2024-03-19T11:30:14.099 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0080668630000000005, 2024-03-19T11:30:14.099 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:14.099 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:14.099 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:14.099 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.099 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:14.099 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.307388+0000", 2024-03-19T11:30:14.099 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:14.099 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.099 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.100 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:14.100 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.307388+0000", 2024-03-19T11:30:14.100 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:14.100 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.100 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.100 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:14.100 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.307391+0000", 2024-03-19T11:30:14.100 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3039999999999999e-06 2024-03-19T11:30:14.100 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.100 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.100 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:14.100 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308352+0000", 2024-03-19T11:30:14.100 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00096137600000000001 2024-03-19T11:30:14.100 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.100 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.101 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:14.101 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308376+0000", 2024-03-19T11:30:14.101 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4196999999999999e-05 2024-03-19T11:30:14.101 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.101 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.101 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:14.101 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308387+0000", 2024-03-19T11:30:14.101 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1068e-05 2024-03-19T11:30:14.101 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.101 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.101 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:14.101 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308539+0000", 2024-03-19T11:30:14.101 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015186600000000001 2024-03-19T11:30:14.101 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.101 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.102 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:14.102 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308731+0000", 2024-03-19T11:30:14.102 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019146199999999999 2024-03-19T11:30:14.102 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.102 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.102 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:14.102 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309028+0000", 2024-03-19T11:30:14.102 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00029751799999999998 2024-03-19T11:30:14.102 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.102 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.102 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:14.102 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.314461+0000", 2024-03-19T11:30:14.102 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0054329570000000004 2024-03-19T11:30:14.102 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.103 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.103 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:14.103 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.315339+0000", 2024-03-19T11:30:14.103 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00087765200000000003 2024-03-19T11:30:14.103 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.103 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.103 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:14.103 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.315400+0000", 2024-03-19T11:30:14.103 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1650999999999996e-05 2024-03-19T11:30:14.103 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.103 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.103 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:14.103 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.315455+0000", 2024-03-19T11:30:14.103 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4812000000000003e-05 2024-03-19T11:30:14.103 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:14.104 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:14.104 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:14.104 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.104 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.104 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:41 3.2 3:525f009b:::a57e3dbafedf1ec1a7132467f167cfc4a4343cb541b36fc481376f0086aece71:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:14.104 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.308522+0000", 2024-03-19T11:30:14.104 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.7939680639999995, 2024-03-19T11:30:14.104 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0079565969999999993, 2024-03-19T11:30:14.104 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:14.104 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:14.104 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:14.104 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.104 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:14.104 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308522+0000", 2024-03-19T11:30:14.104 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:14.105 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.105 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.105 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:14.105 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308522+0000", 2024-03-19T11:30:14.105 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:14.105 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.105 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.105 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:14.105 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308524+0000", 2024-03-19T11:30:14.105 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.809e-06 2024-03-19T11:30:14.105 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.105 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.105 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:14.105 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308627+0000", 2024-03-19T11:30:14.105 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000103319 2024-03-19T11:30:14.106 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.106 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.106 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:14.106 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308643+0000", 2024-03-19T11:30:14.106 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5279000000000001e-05 2024-03-19T11:30:14.106 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.106 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.106 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:14.106 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308653+0000", 2024-03-19T11:30:14.106 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0557000000000001e-05 2024-03-19T11:30:14.106 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.106 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.106 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:14.106 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310052+0000", 2024-03-19T11:30:14.107 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0013986770000000001 2024-03-19T11:30:14.107 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.107 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.107 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:14.107 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310231+0000", 2024-03-19T11:30:14.107 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017891000000000001 2024-03-19T11:30:14.107 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.107 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.107 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:14.107 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310515+0000", 2024-03-19T11:30:14.107 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00028459299999999998 2024-03-19T11:30:14.107 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.107 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.107 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:14.107 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.315472+0000", 2024-03-19T11:30:14.108 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.004956874 2024-03-19T11:30:14.108 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.108 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.108 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:14.108 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.316378+0000", 2024-03-19T11:30:14.108 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00090556600000000001 2024-03-19T11:30:14.108 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.108 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.108 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:14.108 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.316435+0000", 2024-03-19T11:30:14.108 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7126000000000002e-05 2024-03-19T11:30:14.108 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.108 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.108 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:14.108 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.316479+0000", 2024-03-19T11:30:14.109 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3887000000000002e-05 2024-03-19T11:30:14.109 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:14.109 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:14.109 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:14.109 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.109 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.109 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:42 3.2 3:5aa75d1e:::b3c26d797de4ffcce4e862009e6b8eb8576a53cb86ea102ce51145c0038aeb97:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:14.109 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.308689+0000", 2024-03-19T11:30:14.109 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.7938013079999999, 2024-03-19T11:30:14.109 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0079809319999999996, 2024-03-19T11:30:14.109 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:14.109 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:14.109 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:14.109 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.109 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:14.110 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308689+0000", 2024-03-19T11:30:14.110 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:14.110 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.110 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.110 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:14.110 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308689+0000", 2024-03-19T11:30:14.110 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:14.110 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.110 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.110 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:14.110 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308690+0000", 2024-03-19T11:30:14.110 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.003e-06 2024-03-19T11:30:14.110 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.110 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.111 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:14.111 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309612+0000", 2024-03-19T11:30:14.111 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00092193199999999998 2024-03-19T11:30:14.111 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.111 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.111 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:14.111 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309645+0000", 2024-03-19T11:30:14.111 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2554e-05 2024-03-19T11:30:14.111 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.111 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.111 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:14.111 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309654+0000", 2024-03-19T11:30:14.111 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6209999999999999e-06 2024-03-19T11:30:14.111 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.111 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.112 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:14.112 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.311778+0000", 2024-03-19T11:30:14.112 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002123897 2024-03-19T11:30:14.112 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.112 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.112 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:14.112 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.311913+0000", 2024-03-19T11:30:14.112 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013471199999999999 2024-03-19T11:30:14.112 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.112 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.112 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:14.112 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.312123+0000", 2024-03-19T11:30:14.112 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00021064099999999999 2024-03-19T11:30:14.112 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.112 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.113 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:14.113 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.315479+0000", 2024-03-19T11:30:14.113 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033557230000000001 2024-03-19T11:30:14.113 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.113 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.113 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:14.113 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.316574+0000", 2024-03-19T11:30:14.113 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0010944990000000001 2024-03-19T11:30:14.113 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.113 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.113 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:14.113 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.316629+0000", 2024-03-19T11:30:14.113 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5169e-05 2024-03-19T11:30:14.113 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.113 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.114 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:14.114 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.316670+0000", 2024-03-19T11:30:14.114 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1180999999999998e-05 2024-03-19T11:30:14.114 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:14.114 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:14.114 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:14.114 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.114 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.114 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:43 3.2 3:6d82e5d5:::455c613336f373ae48b33af990f91f317265a0c5a920cc1b46533fdb41d8319c:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:14.114 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.309666+0000", 2024-03-19T11:30:14.114 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.7928240129999997, 2024-03-19T11:30:14.114 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077817820000000001, 2024-03-19T11:30:14.114 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:14.114 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:14.115 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:14.115 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.115 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:14.115 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309666+0000", 2024-03-19T11:30:14.115 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:14.115 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.115 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.115 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:14.115 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309666+0000", 2024-03-19T11:30:14.115 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:14.115 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.115 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.115 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:14.115 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309667+0000", 2024-03-19T11:30:14.115 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.5099999999999998e-07 2024-03-19T11:30:14.116 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.116 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.116 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:14.116 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310025+0000", 2024-03-19T11:30:14.116 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00035726900000000001 2024-03-19T11:30:14.116 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.116 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.116 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:14.116 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310093+0000", 2024-03-19T11:30:14.116 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8604999999999998e-05 2024-03-19T11:30:14.116 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.116 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.116 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:14.116 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310157+0000", 2024-03-19T11:30:14.117 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3688000000000004e-05 2024-03-19T11:30:14.117 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.117 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.117 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:14.117 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.313280+0000", 2024-03-19T11:30:14.117 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031230770000000001 2024-03-19T11:30:14.117 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.117 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.117 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:14.117 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.313429+0000", 2024-03-19T11:30:14.117 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000148763 2024-03-19T11:30:14.117 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.117 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.117 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:14.117 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.313700+0000", 2024-03-19T11:30:14.118 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00027178899999999998 2024-03-19T11:30:14.118 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.118 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.118 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:14.118 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.315489+0000", 2024-03-19T11:30:14.118 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001788848 2024-03-19T11:30:14.118 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.118 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.118 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:14.118 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.317381+0000", 2024-03-19T11:30:14.118 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0018913249999999999 2024-03-19T11:30:14.118 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.118 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.118 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:14.119 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.317411+0000", 2024-03-19T11:30:14.119 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0001e-05 2024-03-19T11:30:14.119 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.119 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.119 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:14.119 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.317448+0000", 2024-03-19T11:30:14.119 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7465999999999997e-05 2024-03-19T11:30:14.119 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:14.119 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:14.119 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:14.119 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.119 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.119 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:44 3.2 3:5f1d333e:::bfcdf551f6f2543327e9c601a4377e9a5ef396f5b97ae8331892d1bcf6b55a9d:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:14.119 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.310172+0000", 2024-03-19T11:30:14.119 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.7923178399999999, 2024-03-19T11:30:14.120 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0086762909999999992, 2024-03-19T11:30:14.120 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:14.120 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:14.120 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:14.120 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.120 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:14.120 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310172+0000", 2024-03-19T11:30:14.120 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:14.120 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.120 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.120 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:14.120 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310172+0000", 2024-03-19T11:30:14.120 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:14.120 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.120 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.120 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:14.121 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310173+0000", 2024-03-19T11:30:14.121 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0500000000000002e-07 2024-03-19T11:30:14.121 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.121 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.121 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:14.121 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310242+0000", 2024-03-19T11:30:14.121 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8239999999999997e-05 2024-03-19T11:30:14.121 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.121 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.121 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:14.121 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310253+0000", 2024-03-19T11:30:14.121 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0879e-05 2024-03-19T11:30:14.121 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.121 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.121 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:14.122 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310263+0000", 2024-03-19T11:30:14.122 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0045e-05 2024-03-19T11:30:14.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.122 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:14.122 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.314507+0000", 2024-03-19T11:30:14.122 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0042442060000000004 2024-03-19T11:30:14.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.122 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:14.122 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.314588+0000", 2024-03-19T11:30:14.122 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0912000000000001e-05 2024-03-19T11:30:14.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.123 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:14.123 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.314736+0000", 2024-03-19T11:30:14.123 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014782099999999999 2024-03-19T11:30:14.123 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.123 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.123 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:14.123 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.316529+0000", 2024-03-19T11:30:14.123 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00179327 2024-03-19T11:30:14.123 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.123 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.123 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:14.123 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.318785+0000", 2024-03-19T11:30:14.123 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022563660000000001 2024-03-19T11:30:14.123 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.123 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.124 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:14.124 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.318813+0000", 2024-03-19T11:30:14.124 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8206000000000001e-05 2024-03-19T11:30:14.124 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.124 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.124 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:14.124 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.318849+0000", 2024-03-19T11:30:14.124 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5441000000000002e-05 2024-03-19T11:30:14.124 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:14.124 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:14.124 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:14.124 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.124 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.124 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:45 3.2 3:78b4b14e:::e1ef0c0c594c36c534f3829406e07ffb60d57b70360f62977e0d545d8ca2bf18:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:14.124 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.310274+0000", 2024-03-19T11:30:14.125 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.7922163849999997, 2024-03-19T11:30:14.125 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094234769999999995, 2024-03-19T11:30:14.125 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:14.125 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:14.125 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:14.125 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.125 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:14.125 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310274+0000", 2024-03-19T11:30:14.125 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:14.125 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.125 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.125 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:14.125 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310274+0000", 2024-03-19T11:30:14.125 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:14.125 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.126 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.126 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:14.126 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310275+0000", 2024-03-19T11:30:14.126 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2099999999999996e-07 2024-03-19T11:30:14.126 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.126 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.126 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:14.126 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310329+0000", 2024-03-19T11:30:14.126 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4440000000000001e-05 2024-03-19T11:30:14.126 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.126 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.126 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:14.126 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310340+0000", 2024-03-19T11:30:14.126 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1321999999999999e-05 2024-03-19T11:30:14.126 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.127 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.127 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:14.127 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310349+0000", 2024-03-19T11:30:14.127 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6789999999999994e-06 2024-03-19T11:30:14.127 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.127 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.127 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:14.127 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.315512+0000", 2024-03-19T11:30:14.127 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0051629689999999999 2024-03-19T11:30:14.127 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.127 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.127 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:14.127 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.315581+0000", 2024-03-19T11:30:14.127 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9400999999999995e-05 2024-03-19T11:30:14.127 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.128 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.128 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:14.128 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.315759+0000", 2024-03-19T11:30:14.128 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000177491 2024-03-19T11:30:14.128 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.128 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.128 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:14.128 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.317063+0000", 2024-03-19T11:30:14.128 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0013040630000000001 2024-03-19T11:30:14.128 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.128 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.128 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:14.128 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.319635+0000", 2024-03-19T11:30:14.128 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025715880000000001 2024-03-19T11:30:14.129 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.129 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.129 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:14.129 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.319663+0000", 2024-03-19T11:30:14.129 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8554000000000001e-05 2024-03-19T11:30:14.129 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.129 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.129 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:14.129 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.319697+0000", 2024-03-19T11:30:14.129 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4348999999999999e-05 2024-03-19T11:30:14.129 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:14.129 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:14.129 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:14.129 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.129 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.130 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1008 4.0 4:0dc3b91e:::smithi03829856-28:head [tier-promote] snapc 6=[6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:14.130 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.760589+0000", 2024-03-19T11:30:14.130 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 1.3419008859999999, 2024-03-19T11:30:14.130 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005233105, 2024-03-19T11:30:14.130 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:14.130 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:14.130 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:14.130 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:14.130 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:14.130 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1008 2024-03-19T11:30:14.130 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.131 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:14.131 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.131 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:14.131 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.760589+0000", 2024-03-19T11:30:14.131 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:14.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.131 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.131 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:14.131 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.760589+0000", 2024-03-19T11:30:14.131 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:14.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.131 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.131 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:14.131 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.760592+0000", 2024-03-19T11:30:14.131 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3910000000000001e-06 2024-03-19T11:30:14.132 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.132 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.132 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:14.132 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.760602+0000", 2024-03-19T11:30:14.132 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0093e-05 2024-03-19T11:30:14.132 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.132 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.132 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:14.132 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.760623+0000", 2024-03-19T11:30:14.132 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0982e-05 2024-03-19T11:30:14.132 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.132 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.132 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:14.132 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.760635+0000", 2024-03-19T11:30:14.132 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1595e-05 2024-03-19T11:30:14.133 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.133 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.133 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:14.133 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.760713+0000", 2024-03-19T11:30:14.133 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8045000000000002e-05 2024-03-19T11:30:14.133 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.133 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.133 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-03-19T11:30:14.133 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.760717+0000", 2024-03-19T11:30:14.133 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8849999999999998e-06 2024-03-19T11:30:14.133 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.133 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.133 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:14.133 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.763491+0000", 2024-03-19T11:30:14.133 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0027735619999999998 2024-03-19T11:30:14.134 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.134 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.134 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:14.134 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.763573+0000", 2024-03-19T11:30:14.134 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2424999999999998e-05 2024-03-19T11:30:14.134 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.134 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.134 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:14.134 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.765823+0000", 2024-03-19T11:30:14.134 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022491270000000001 2024-03-19T11:30:14.134 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:14.134 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:14.134 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:14.134 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.135 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.135 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:52 3.2 3:5595a68d:::07854d2fef297a06ba81685e660c332de36d5d18d546927d30daad6d7fda1541:head [read 0~524288 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:14.135 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.761730+0000", 2024-03-19T11:30:14.135 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 1.340760752, 2024-03-19T11:30:14.135 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003807618, 2024-03-19T11:30:14.135 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:14.135 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:14.135 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:14.135 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.135 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:14.135 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761730+0000", 2024-03-19T11:30:14.135 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:14.135 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.135 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.135 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:14.136 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761730+0000", 2024-03-19T11:30:14.136 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:14.136 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.136 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.136 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:14.136 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761731+0000", 2024-03-19T11:30:14.136 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2109999999999999e-06 2024-03-19T11:30:14.136 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.136 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.136 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:14.136 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761738+0000", 2024-03-19T11:30:14.136 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8349999999999998e-06 2024-03-19T11:30:14.136 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.136 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.137 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:14.137 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761757+0000", 2024-03-19T11:30:14.137 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9015e-05 2024-03-19T11:30:14.137 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.137 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.137 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:14.137 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761766+0000", 2024-03-19T11:30:14.137 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8810000000000002e-06 2024-03-19T11:30:14.137 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.137 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.137 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:14.137 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761828+0000", 2024-03-19T11:30:14.137 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1983999999999995e-05 2024-03-19T11:30:14.137 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.137 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.137 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-03-19T11:30:14.138 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761834+0000", 2024-03-19T11:30:14.138 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7750000000000004e-06 2024-03-19T11:30:14.138 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.138 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.138 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:14.138 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.764961+0000", 2024-03-19T11:30:14.138 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031267439999999999 2024-03-19T11:30:14.138 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.138 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.138 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:14.138 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.765056+0000", 2024-03-19T11:30:14.138 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.5247999999999993e-05 2024-03-19T11:30:14.138 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.139 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.139 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:14.139 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.765537+0000", 2024-03-19T11:30:14.139 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00048092499999999999 2024-03-19T11:30:14.139 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:14.139 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:14.139 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:14.139 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.139 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.139 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:55 3.2 3:525f009b:::a57e3dbafedf1ec1a7132467f167cfc4a4343cb541b36fc481376f0086aece71:head [read 0~121882 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:14.139 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.761904+0000", 2024-03-19T11:30:14.139 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 1.340585889, 2024-03-19T11:30:14.139 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010829357, 2024-03-19T11:30:14.139 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:14.139 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:14.139 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:14.140 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.140 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:14.140 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761904+0000", 2024-03-19T11:30:14.140 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:14.140 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.140 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.140 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:14.140 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761904+0000", 2024-03-19T11:30:14.140 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:14.140 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.140 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.140 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:14.140 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761905+0000", 2024-03-19T11:30:14.140 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4900000000000005e-07 2024-03-19T11:30:14.140 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.141 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.141 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:14.141 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761917+0000", 2024-03-19T11:30:14.141 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1806e-05 2024-03-19T11:30:14.141 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.141 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.141 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:14.141 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761934+0000", 2024-03-19T11:30:14.141 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6509000000000001e-05 2024-03-19T11:30:14.141 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.141 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.141 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:14.141 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761941+0000", 2024-03-19T11:30:14.141 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8110000000000003e-06 2024-03-19T11:30:14.141 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.142 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.142 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:14.142 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762115+0000", 2024-03-19T11:30:14.142 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017358200000000001 2024-03-19T11:30:14.142 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.142 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.142 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:14.142 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762120+0000", 2024-03-19T11:30:14.142 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3630000000000002e-06 2024-03-19T11:30:14.142 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.142 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.142 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:14.142 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.765567+0000", 2024-03-19T11:30:14.142 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034469990000000001 2024-03-19T11:30:14.142 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.143 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.143 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:14.143 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.765665+0000", 2024-03-19T11:30:14.143 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7476e-05 2024-03-19T11:30:14.143 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.143 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.143 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:14.143 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.772734+0000", 2024-03-19T11:30:14.143 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0070690620000000001 2024-03-19T11:30:14.143 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:14.143 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:14.143 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:14.143 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.143 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.143 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:56 3.2 3:5aa75d1e:::b3c26d797de4ffcce4e862009e6b8eb8576a53cb86ea102ce51145c0038aeb97:head [read 0~524288 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:14.144 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.761958+0000", 2024-03-19T11:30:14.144 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 1.340532488, 2024-03-19T11:30:14.144 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012078836000000001, 2024-03-19T11:30:14.144 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:14.144 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:14.144 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:14.144 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.144 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:14.144 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761958+0000", 2024-03-19T11:30:14.144 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:14.144 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.144 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.144 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:14.144 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761958+0000", 2024-03-19T11:30:14.144 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:14.145 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.145 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.145 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:14.145 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761958+0000", 2024-03-19T11:30:14.145 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9900000000000001e-07 2024-03-19T11:30:14.145 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.145 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.145 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:14.145 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761961+0000", 2024-03-19T11:30:14.145 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9280000000000002e-06 2024-03-19T11:30:14.145 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.145 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.145 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:14.145 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761969+0000", 2024-03-19T11:30:14.145 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1440000000000003e-06 2024-03-19T11:30:14.145 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.146 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.146 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:14.146 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761976+0000", 2024-03-19T11:30:14.146 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1020000000000002e-06 2024-03-19T11:30:14.146 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.146 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.146 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:14.146 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762144+0000", 2024-03-19T11:30:14.146 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016818200000000001 2024-03-19T11:30:14.146 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.146 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.146 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:14.146 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762148+0000", 2024-03-19T11:30:14.146 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0969999999999999e-06 2024-03-19T11:30:14.146 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.147 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.147 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:14.147 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.772788+0000", 2024-03-19T11:30:14.147 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010640124000000001 2024-03-19T11:30:14.147 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.147 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.147 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:14.147 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.772898+0000", 2024-03-19T11:30:14.147 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000110512 2024-03-19T11:30:14.147 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.147 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.147 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:14.147 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.774037+0000", 2024-03-19T11:30:14.147 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011382479999999999 2024-03-19T11:30:14.148 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:14.148 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:14.148 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:14.148 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.148 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.148 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:57 3.2 3:6d82e5d5:::455c613336f373ae48b33af990f91f317265a0c5a920cc1b46533fdb41d8319c:head [read 0~380357 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:14.148 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.761987+0000", 2024-03-19T11:30:14.148 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 1.340503526, 2024-03-19T11:30:14.148 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012691991999999999, 2024-03-19T11:30:14.148 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:14.148 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:14.148 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:14.148 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.148 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:14.148 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761987+0000", 2024-03-19T11:30:14.149 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:14.149 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.149 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.149 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:14.149 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761987+0000", 2024-03-19T11:30:14.149 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:14.149 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.149 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.149 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:14.149 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761987+0000", 2024-03-19T11:30:14.149 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4099999999999999e-07 2024-03-19T11:30:14.149 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.149 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.149 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:14.149 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761990+0000", 2024-03-19T11:30:14.150 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3829999999999999e-06 2024-03-19T11:30:14.150 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.150 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.150 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:14.150 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762000+0000", 2024-03-19T11:30:14.150 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0229e-05 2024-03-19T11:30:14.150 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.150 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.150 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:14.150 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762006+0000", 2024-03-19T11:30:14.150 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7069999999999997e-06 2024-03-19T11:30:14.150 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.150 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.150 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:14.150 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762171+0000", 2024-03-19T11:30:14.151 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000165341 2024-03-19T11:30:14.151 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.151 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.151 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:14.151 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762176+0000", 2024-03-19T11:30:14.151 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6879999999999998e-06 2024-03-19T11:30:14.151 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.151 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.151 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:14.151 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.774081+0000", 2024-03-19T11:30:14.151 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011905094999999999 2024-03-19T11:30:14.151 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.151 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.151 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:14.151 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.774169+0000", 2024-03-19T11:30:14.152 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8089000000000003e-05 2024-03-19T11:30:14.152 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.152 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.152 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:14.152 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.774679+0000", 2024-03-19T11:30:14.152 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00051001900000000001 2024-03-19T11:30:14.152 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:14.152 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:14.152 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:14.152 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.152 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.152 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:58 3.2 3:5f1d333e:::bfcdf551f6f2543327e9c601a4377e9a5ef396f5b97ae8331892d1bcf6b55a9d:head [read 0~161292 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:14.152 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.762024+0000", 2024-03-19T11:30:14.152 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 1.3404665769999999, 2024-03-19T11:30:14.152 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013495629, 2024-03-19T11:30:14.153 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:14.153 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:14.153 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:14.153 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.153 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:14.153 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762024+0000", 2024-03-19T11:30:14.153 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:14.153 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.153 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.153 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:14.153 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762024+0000", 2024-03-19T11:30:14.153 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:14.153 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.153 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.153 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:14.154 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762024+0000", 2024-03-19T11:30:14.154 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9999999999999997e-07 2024-03-19T11:30:14.154 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.154 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.154 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:14.154 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762027+0000", 2024-03-19T11:30:14.154 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0410000000000002e-06 2024-03-19T11:30:14.154 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.154 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.154 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:14.154 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762037+0000", 2024-03-19T11:30:14.154 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1020000000000003e-06 2024-03-19T11:30:14.154 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.154 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.155 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:14.155 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762043+0000", 2024-03-19T11:30:14.155 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4409999999999998e-06 2024-03-19T11:30:14.155 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.155 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.155 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:14.155 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762198+0000", 2024-03-19T11:30:14.155 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015540000000000001 2024-03-19T11:30:14.155 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.155 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.155 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:14.155 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762203+0000", 2024-03-19T11:30:14.155 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5109999999999997e-06 2024-03-19T11:30:14.155 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.155 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.156 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:14.156 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.774710+0000", 2024-03-19T11:30:14.156 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012507452 2024-03-19T11:30:14.156 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.156 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.156 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:14.156 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.774754+0000", 2024-03-19T11:30:14.156 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3869999999999998e-05 2024-03-19T11:30:14.156 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.156 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.156 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:14.156 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.775519+0000", 2024-03-19T11:30:14.156 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00076521200000000003 2024-03-19T11:30:14.156 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:14.157 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:14.157 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:14.157 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.157 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.157 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:59 3.2 3:78b4b14e:::e1ef0c0c594c36c534f3829406e07ffb60d57b70360f62977e0d545d8ca2bf18:head [read 0~149977 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:14.157 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.762054+0000", 2024-03-19T11:30:14.157 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 1.3404361460000001, 2024-03-19T11:30:14.157 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015354483, 2024-03-19T11:30:14.157 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:14.157 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:14.157 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:14.157 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.157 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:14.157 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762054+0000", 2024-03-19T11:30:14.157 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:14.158 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.158 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.158 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:14.158 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762054+0000", 2024-03-19T11:30:14.158 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:14.158 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.158 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.158 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:14.158 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762055+0000", 2024-03-19T11:30:14.158 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.32e-07 2024-03-19T11:30:14.158 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.158 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.158 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:14.159 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762058+0000", 2024-03-19T11:30:14.159 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9189999999999999e-06 2024-03-19T11:30:14.159 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.159 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.159 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:14.159 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762065+0000", 2024-03-19T11:30:14.159 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2409999999999998e-06 2024-03-19T11:30:14.159 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.159 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.159 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:14.159 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762070+0000", 2024-03-19T11:30:14.159 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6910000000000002e-06 2024-03-19T11:30:14.159 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.159 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.160 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:14.160 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762227+0000", 2024-03-19T11:30:14.160 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015606999999999999 2024-03-19T11:30:14.160 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.160 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.160 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:14.160 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762232+0000", 2024-03-19T11:30:14.160 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3029999999999999e-06 2024-03-19T11:30:14.160 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.160 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.160 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:14.160 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.776984+0000", 2024-03-19T11:30:14.160 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014752487999999999 2024-03-19T11:30:14.160 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.160 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.161 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:14.161 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.777030+0000", 2024-03-19T11:30:14.161 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5359999999999999e-05 2024-03-19T11:30:14.161 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.161 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.161 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:14.161 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.777409+0000", 2024-03-19T11:30:14.161 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037897899999999999 2024-03-19T11:30:14.161 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:14.161 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:14.161 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:14.161 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.161 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.161 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1020 3.2 3:59279e72:::24df73eb14d5465f466b1889d4952f96785c09a615e42a99c9b768b4b3347e65:head [write 0~207872] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:14.162 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.773460+0000", 2024-03-19T11:30:14.162 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 1.3290302940000001, 2024-03-19T11:30:14.162 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0070966579999999996, 2024-03-19T11:30:14.162 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:14.162 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:14.162 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:14.162 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:14.162 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:14.162 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1020 2024-03-19T11:30:14.162 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.162 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:14.162 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.162 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:14.162 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.773460+0000", 2024-03-19T11:30:14.162 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:14.163 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.163 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.163 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:14.163 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.773460+0000", 2024-03-19T11:30:14.163 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:14.163 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.163 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.163 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:14.163 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.773462+0000", 2024-03-19T11:30:14.163 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.984e-06 2024-03-19T11:30:14.163 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.163 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.163 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:14.163 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.773537+0000", 2024-03-19T11:30:14.164 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4726000000000002e-05 2024-03-19T11:30:14.164 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.164 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.164 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:14.164 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.773550+0000", 2024-03-19T11:30:14.164 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3721e-05 2024-03-19T11:30:14.164 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.164 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.164 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:14.164 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.773561+0000", 2024-03-19T11:30:14.164 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1017e-05 2024-03-19T11:30:14.164 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.164 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.165 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:14.165 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.775560+0000", 2024-03-19T11:30:14.165 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0019981510000000001 2024-03-19T11:30:14.165 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.165 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.165 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:14.165 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.775716+0000", 2024-03-19T11:30:14.165 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000156265 2024-03-19T11:30:14.165 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.165 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.165 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:14.165 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.775878+0000", 2024-03-19T11:30:14.165 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016257899999999999 2024-03-19T11:30:14.165 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.165 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.166 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:14.166 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.777476+0000", 2024-03-19T11:30:14.166 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001597465 2024-03-19T11:30:14.166 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.166 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.166 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:14.166 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.780481+0000", 2024-03-19T11:30:14.166 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030052450000000001 2024-03-19T11:30:14.166 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.166 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.166 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:14.166 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.780532+0000", 2024-03-19T11:30:14.166 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0484e-05 2024-03-19T11:30:14.166 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.167 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.167 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:14.167 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.780557+0000", 2024-03-19T11:30:14.167 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5021e-05 2024-03-19T11:30:14.167 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:14.167 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:14.167 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:14.167 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.167 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.167 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:1022 4.c e35/21 4:3c71e808:::smithi03829856-34:head v 35'16, mlcod=32'12)", 2024-03-19T11:30:14.167 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.787728+0000", 2024-03-19T11:30:14.167 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 1.31476214, 2024-03-19T11:30:14.167 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0032506259999999999, 2024-03-19T11:30:14.167 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:14.168 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:14.169 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:14.169 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.169 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:14.169 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.787728+0000", 2024-03-19T11:30:14.169 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:14.169 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.169 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.169 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:14.170 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.787728+0000", 2024-03-19T11:30:14.170 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:14.170 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.170 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.170 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:14.170 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.787729+0000", 2024-03-19T11:30:14.170 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.27e-07 2024-03-19T11:30:14.170 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.170 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.170 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:14.170 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.788629+0000", 2024-03-19T11:30:14.170 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00090012100000000004 2024-03-19T11:30:14.170 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.170 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.170 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:14.171 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.788644+0000", 2024-03-19T11:30:14.171 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5444e-05 2024-03-19T11:30:14.171 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.171 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.171 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:14.171 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.788656+0000", 2024-03-19T11:30:14.171 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2176e-05 2024-03-19T11:30:14.171 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.171 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.171 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:14.171 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.788719+0000", 2024-03-19T11:30:14.171 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2556000000000002e-05 2024-03-19T11:30:14.171 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.171 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.171 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:14.172 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.788737+0000", 2024-03-19T11:30:14.172 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8598e-05 2024-03-19T11:30:14.172 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.172 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.172 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:14.172 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.790908+0000", 2024-03-19T11:30:14.172 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0021702430000000001 2024-03-19T11:30:14.172 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.172 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.172 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:14.172 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.790979+0000", 2024-03-19T11:30:14.172 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1161000000000005e-05 2024-03-19T11:30:14.172 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:14.172 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:14.172 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:14.173 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.173 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.173 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1030 4.0 4:0dc3b91e:::smithi03829856-28:head [delete] snapc 6=[6,5,2] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:14.173 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.799207+0000", 2024-03-19T11:30:14.173 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 1.3032834040000001, 2024-03-19T11:30:14.173 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0058498509999999997, 2024-03-19T11:30:14.173 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:14.173 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:14.173 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:14.173 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:14.173 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:14.173 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1030 2024-03-19T11:30:14.173 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.173 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:14.173 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.173 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:14.174 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.799207+0000", 2024-03-19T11:30:14.174 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:14.174 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.174 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.174 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:14.174 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.799207+0000", 2024-03-19T11:30:14.174 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:14.174 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.174 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.174 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:14.174 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.799209+0000", 2024-03-19T11:30:14.174 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5500000000000001e-06 2024-03-19T11:30:14.174 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.174 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.174 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:14.175 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.799219+0000", 2024-03-19T11:30:14.175 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.397e-06 2024-03-19T11:30:14.175 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.175 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.175 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:14.175 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.799236+0000", 2024-03-19T11:30:14.175 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7187999999999999e-05 2024-03-19T11:30:14.175 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.175 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.175 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:14.175 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.799247+0000", 2024-03-19T11:30:14.175 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0949e-05 2024-03-19T11:30:14.175 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.175 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.175 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:14.176 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.799314+0000", 2024-03-19T11:30:14.176 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7377000000000002e-05 2024-03-19T11:30:14.176 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.176 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.176 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:14.176 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.799420+0000", 2024-03-19T11:30:14.176 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010555799999999999 2024-03-19T11:30:14.176 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.176 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.176 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:14.176 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.799619+0000", 2024-03-19T11:30:14.176 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000199318 2024-03-19T11:30:14.176 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.176 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.176 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:14.177 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.802335+0000", 2024-03-19T11:30:14.177 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0027158820000000002 2024-03-19T11:30:14.177 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.177 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.177 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:14.177 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.804928+0000", 2024-03-19T11:30:14.177 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025925760000000001 2024-03-19T11:30:14.177 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.177 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.177 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:14.177 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.804995+0000", 2024-03-19T11:30:14.177 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7000000000000002e-05 2024-03-19T11:30:14.177 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.177 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.177 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:14.178 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.805057+0000", 2024-03-19T11:30:14.178 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2056000000000004e-05 2024-03-19T11:30:14.178 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:14.178 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:14.178 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:14.178 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.180 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.180 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.1.14:32 3.2 3:5595a68d:::07854d2fef297a06ba81685e660c332de36d5d18d546927d30daad6d7fda1541:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:14.181 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.794221+0000", 2024-03-19T11:30:14.181 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.30826901000000001, 2024-03-19T11:30:14.181 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038657729999999999, 2024-03-19T11:30:14.181 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:14.181 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:14.181 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:14.181 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.181 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:14.181 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.794221+0000", 2024-03-19T11:30:14.181 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:14.181 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.181 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.181 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:14.181 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.794221+0000", 2024-03-19T11:30:14.182 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:14.182 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.182 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.182 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:14.182 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.794224+0000", 2024-03-19T11:30:14.182 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5220000000000002e-06 2024-03-19T11:30:14.182 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.182 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.182 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:14.182 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.794692+0000", 2024-03-19T11:30:14.182 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00046814600000000001 2024-03-19T11:30:14.182 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.182 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.182 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:14.183 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.794717+0000", 2024-03-19T11:30:14.183 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5148e-05 2024-03-19T11:30:14.183 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.183 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.183 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:14.183 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.794729+0000", 2024-03-19T11:30:14.183 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1572e-05 2024-03-19T11:30:14.183 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.183 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.183 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:14.183 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.794806+0000", 2024-03-19T11:30:14.183 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7661000000000001e-05 2024-03-19T11:30:14.183 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.183 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.184 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:14.184 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.795016+0000", 2024-03-19T11:30:14.184 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020970900000000001 2024-03-19T11:30:14.184 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.184 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.184 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:14.184 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.795256+0000", 2024-03-19T11:30:14.184 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000239485 2024-03-19T11:30:14.184 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.184 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.184 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:14.184 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.796086+0000", 2024-03-19T11:30:14.184 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00083003899999999997 2024-03-19T11:30:14.184 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.184 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.184 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:14.185 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.798003+0000", 2024-03-19T11:30:14.185 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001917752 2024-03-19T11:30:14.185 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.185 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.185 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:14.185 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.798040+0000", 2024-03-19T11:30:14.185 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6857000000000003e-05 2024-03-19T11:30:14.185 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.185 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.185 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:14.185 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.798087+0000", 2024-03-19T11:30:14.185 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6882000000000003e-05 2024-03-19T11:30:14.185 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:14.185 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:14.186 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:14.186 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.186 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.186 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.1.14:38 3.2 3:457739b0:::013a25d9e3151bb52f935d5c2bbdf891de4971b39a9076b2c7a4d91e7060598c:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:14.186 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.795888+0000", 2024-03-19T11:30:14.186 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.30660191399999998, 2024-03-19T11:30:14.186 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053241210000000002, 2024-03-19T11:30:14.186 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:14.186 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:14.186 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:14.186 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.186 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:14.186 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.795888+0000", 2024-03-19T11:30:14.186 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:14.187 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.187 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.187 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:14.187 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.795888+0000", 2024-03-19T11:30:14.187 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:14.187 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.187 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.187 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:14.187 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.795891+0000", 2024-03-19T11:30:14.187 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3120000000000001e-06 2024-03-19T11:30:14.187 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.187 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.187 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:14.187 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.796068+0000", 2024-03-19T11:30:14.188 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017687700000000001 2024-03-19T11:30:14.188 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.188 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.188 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:14.188 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.796082+0000", 2024-03-19T11:30:14.188 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4859e-05 2024-03-19T11:30:14.188 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.188 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.188 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:14.188 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.796092+0000", 2024-03-19T11:30:14.188 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.346e-06 2024-03-19T11:30:14.188 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.188 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.188 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:14.188 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.796557+0000", 2024-03-19T11:30:14.189 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00046493400000000002 2024-03-19T11:30:14.189 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.189 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.189 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:14.189 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.796742+0000", 2024-03-19T11:30:14.189 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000184981 2024-03-19T11:30:14.189 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.189 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.189 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:14.189 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.797215+0000", 2024-03-19T11:30:14.189 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00047335999999999998 2024-03-19T11:30:14.189 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.189 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.189 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:14.189 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.799986+0000", 2024-03-19T11:30:14.190 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0027706810000000001 2024-03-19T11:30:14.190 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.190 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.190 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:14.190 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.801130+0000", 2024-03-19T11:30:14.190 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011437419999999999 2024-03-19T11:30:14.190 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.190 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.190 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:14.190 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.801166+0000", 2024-03-19T11:30:14.190 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6803e-05 2024-03-19T11:30:14.190 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:14.190 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:14.190 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:14.190 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.801213+0000", 2024-03-19T11:30:14.191 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6226000000000001e-05 2024-03-19T11:30:14.191 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:14.191 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:14.191 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:14.191 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:14.191 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:14.191 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:14.193 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-19T11:30:14.193 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:30:14.193 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:30:14.194 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:30:14.194 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:30:14.195 INFO:tasks.rados.rados.0.smithi038.stdout:711: got (0) Success 2024-03-19T11:30:14.195 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-03-19T11:30:14.195 INFO:tasks.rados.rados.0.smithi038.stdout:713: finishing write tid 1 to smithi03829856-19 2024-03-19T11:30:14.195 INFO:tasks.rados.rados.0.smithi038.stdout:713: finishing write tid 2 to smithi03829856-19 2024-03-19T11:30:14.195 INFO:tasks.rados.rados.0.smithi038.stdout:713: finishing write tid 3 to smithi03829856-19 2024-03-19T11:30:14.195 INFO:tasks.rados.rados.0.smithi038.stdout:711: done (10 left) 2024-03-19T11:30:14.195 INFO:tasks.rados.rados.0.smithi038.stdout:712: done (9 left) 2024-03-19T11:30:14.195 INFO:tasks.rados.rados.0.smithi038.stdout:722: read oid 6 snap 4 2024-03-19T11:30:14.196 INFO:tasks.rados.rados.0.smithi038.stdout:722: expect (ObjNum 53 snap 0 seq_num 53) 2024-03-19T11:30:14.196 INFO:tasks.rados.rados.0.smithi038.stdout:723: write oid 41 current snap is 6 2024-03-19T11:30:14.196 INFO:tasks.rados.rados.0.smithi038.stdout:713: finishing write tid 4 to smithi03829856-19 2024-03-19T11:30:14.196 INFO:tasks.rados.rados.0.smithi038.stdout:713: finishing write tid 5 to smithi03829856-19 2024-03-19T11:30:14.196 INFO:tasks.rados.rados.0.smithi038.stdout:713: oid 19 updating version 0 to 18 2024-03-19T11:30:14.196 INFO:tasks.rados.rados.0.smithi038.stdout:713: oid 19 updating version 18 to 19 2024-03-19T11:30:14.196 INFO:tasks.rados.rados.0.smithi038.stdout:713: oid 19 updating version 19 to 21 2024-03-19T11:30:14.196 INFO:tasks.rados.rados.0.smithi038.stdout:713: oid 19 version 21 is already newer than 20 2024-03-19T11:30:14.196 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 19 v 21 (ObjNum 76 snap 6 seq_num 76) dirty exists 2024-03-19T11:30:14.196 INFO:tasks.rados.rados.0.smithi038.stdout:713: left oid 19 (ObjNum 76 snap 6 seq_num 76) 2024-03-19T11:30:14.196 INFO:tasks.rados.rados.0.smithi038.stdout:714: finishing write tid 1 to smithi03829856-43 2024-03-19T11:30:14.196 INFO:tasks.rados.rados.0.smithi038.stdout:714: finishing write tid 2 to smithi03829856-43 2024-03-19T11:30:14.196 INFO:tasks.rados.rados.0.smithi038.stdout:723: seq_num 80 ranges {544560=783450,1915030=481348,2797604=479761} 2024-03-19T11:30:14.196 INFO:tasks.rados.rados.0.smithi038.stdout:723: writing smithi03829856-41 from 544560 to 1328010 tid 1 2024-03-19T11:30:14.196 INFO:tasks.rados.rados.0.smithi038.stdout:723: writing smithi03829856-41 from 1915030 to 2396378 tid 2 2024-03-19T11:30:14.197 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:30:14.197 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:30:14.198 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:14.203+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:14.200 INFO:tasks.rados.rados.0.smithi038.stdout:723: writing smithi03829856-41 from 2797604 to 3277365 tid 3 2024-03-19T11:30:14.200 INFO:tasks.rados.rados.0.smithi038.stdout:713: done (10 left) 2024-03-19T11:30:14.200 INFO:tasks.rados.rados.0.smithi038.stdout:724: read oid 9 snap -1 2024-03-19T11:30:14.200 INFO:tasks.rados.rados.0.smithi038.stdout:724: expect (ObjNum 8 snap 0 seq_num 8) 2024-03-19T11:30:14.200 INFO:tasks.rados.rados.0.smithi038.stdout:725: rollback oid 10 current snap is 6 2024-03-19T11:30:14.200 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 10 to 3 2024-03-19T11:30:14.200 INFO:tasks.rados.rados.0.smithi038.stdout:714: finishing write tid 3 to smithi03829856-43 2024-03-19T11:30:14.200 INFO:tasks.rados.rados.0.smithi038.stdout:714: finishing write tid 4 to smithi03829856-43 2024-03-19T11:30:14.200 INFO:tasks.rados.rados.0.smithi038.stdout:714: finishing write tid 5 to smithi03829856-43 2024-03-19T11:30:14.200 INFO:tasks.rados.rados.0.smithi038.stdout:714: oid 43 updating version 0 to 38 2024-03-19T11:30:14.200 INFO:tasks.rados.rados.0.smithi038.stdout:714: oid 43 updating version 38 to 41 2024-03-19T11:30:14.201 INFO:tasks.rados.rados.0.smithi038.stdout:714: oid 43 version 41 is already newer than 40 2024-03-19T11:30:14.201 INFO:tasks.rados.rados.0.smithi038.stdout:714: oid 43 version 41 is already newer than 39 2024-03-19T11:30:14.201 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 43 v 41 (ObjNum 77 snap 6 seq_num 77) dirty exists 2024-03-19T11:30:14.201 INFO:tasks.rados.rados.0.smithi038.stdout:714: left oid 43 (ObjNum 77 snap 6 seq_num 77) 2024-03-19T11:30:14.201 INFO:tasks.rados.rados.0.smithi038.stdout:715: finishing write tid 1 to smithi03829856-14 2024-03-19T11:30:14.201 INFO:tasks.rados.rados.0.smithi038.stdout:714: done (11 left) 2024-03-19T11:30:14.201 INFO:tasks.rados.rados.0.smithi038.stdout:726: read oid 5 snap -1 2024-03-19T11:30:14.201 INFO:tasks.rados.rados.0.smithi038.stdout:726: expect (ObjNum 4 snap 0 seq_num 4) 2024-03-19T11:30:14.201 INFO:tasks.rados.rados.0.smithi038.stdout:727: read oid 37 snap -1 2024-03-19T11:30:14.201 INFO:tasks.rados.rados.0.smithi038.stdout:727: expect (ObjNum 36 snap 0 seq_num 36) 2024-03-19T11:30:14.201 INFO:tasks.rados.rados.0.smithi038.stdout:728: tier_promote oid 24 2024-03-19T11:30:14.201 INFO:tasks.rados.rados.0.smithi038.stdout:729: tier_promote oid 17 2024-03-19T11:30:14.201 INFO:tasks.rados.rados.0.smithi038.stdout:730: write (excl) oid 18 current snap is 6 2024-03-19T11:30:14.201 INFO:tasks.rados.rados.0.smithi038.stdout:730: seq_num 81 ranges {561367=458699,1573512=741359,2941380=419987} 2024-03-19T11:30:14.201 INFO:tasks.rados.rados.0.smithi038.stdout:730: writing smithi03829856-18 from 561367 to 1020066 tid 1 2024-03-19T11:30:14.202 INFO:tasks.rados.rados.0.smithi038.stdout:730: writing smithi03829856-18 from 1573512 to 2314871 tid 2 2024-03-19T11:30:14.202 INFO:tasks.rados.rados.0.smithi038.stdout:730: writing smithi03829856-18 from 2941380 to 3361367 tid 3 2024-03-19T11:30:14.202 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:14.202 INFO:tasks.rados.rados.0.smithi038.stdout:715: finishing write tid 2 to smithi03829856-14 2024-03-19T11:30:14.202 INFO:tasks.rados.rados.0.smithi038.stdout:715: finishing write tid 3 to smithi03829856-14 2024-03-19T11:30:14.202 INFO:tasks.rados.rados.0.smithi038.stdout:715: finishing write tid 4 to smithi03829856-14 2024-03-19T11:30:14.202 INFO:tasks.rados.rados.0.smithi038.stdout:715: finishing write tid 5 to smithi03829856-14 2024-03-19T11:30:14.202 INFO:tasks.rados.rados.0.smithi038.stdout:715: oid 14 updating version 0 to 17 2024-03-19T11:30:14.202 INFO:tasks.rados.rados.0.smithi038.stdout:715: oid 14 version 17 is already newer than 14 2024-03-19T11:30:14.202 INFO:tasks.rados.rados.0.smithi038.stdout:715: oid 14 version 17 is already newer than 15 2024-03-19T11:30:14.202 INFO:tasks.rados.rados.0.smithi038.stdout:715: oid 14 version 17 is already newer than 16 2024-03-19T11:30:14.202 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 14 v 17 (ObjNum 78 snap 6 seq_num 78) dirty exists 2024-03-19T11:30:14.202 INFO:tasks.rados.rados.0.smithi038.stdout:715: left oid 14 (ObjNum 78 snap 6 seq_num 78) 2024-03-19T11:30:14.202 INFO:tasks.rados.rados.0.smithi038.stdout:716: expect (ObjNum 52 snap 0 seq_num 52) 2024-03-19T11:30:14.203 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:30:14.203 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:30:14.204 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:30:14.204 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:30:14.222 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:30:14.222 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:30:14.294 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:14.303+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:14.312 INFO:tasks.rados.rados.0.smithi038.stdout:717: expect (ObjNum 1 snap 0 seq_num 1) 2024-03-19T11:30:14.330 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:14.330 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:14.345 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-19T11:30:14.376 INFO:tasks.rados.rados.0.smithi038.stdout:718: expect (ObjNum 68 snap 3 seq_num 68) 2024-03-19T11:30:14.394 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:14.404+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:14.495 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:14.504+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:14.509 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:14.510 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:14.516 INFO:tasks.rados.rados.0.smithi038.stdout:719: finishing write tid 1 to smithi03829856-25 2024-03-19T11:30:14.516 INFO:tasks.rados.rados.0.smithi038.stdout:720: finishing set_chunk to oid 44 2024-03-19T11:30:14.517 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 44 set_chunk returned error code -95 offset: 420864 length: 270336 2024-03-19T11:30:14.517 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 44 v 0 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-03-19T11:30:14.517 INFO:tasks.rados.rados.0.smithi038.stdout:719: finishing write tid 2 to smithi03829856-25 2024-03-19T11:30:14.517 INFO:tasks.rados.rados.0.smithi038.stdout:719: finishing write tid 3 to smithi03829856-25 2024-03-19T11:30:14.517 INFO:tasks.rados.rados.0.smithi038.stdout:721: got (0) Success 2024-03-19T11:30:14.517 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 3 v 21 (ObjNum 62 snap 3 seq_num 62) dirty exists 2024-03-19T11:30:14.517 INFO:tasks.rados.rados.0.smithi038.stdout:719: finishing write tid 4 to smithi03829856-25 2024-03-19T11:30:14.517 INFO:tasks.rados.rados.0.smithi038.stdout:719: finishing write tid 5 to smithi03829856-25 2024-03-19T11:30:14.517 INFO:tasks.rados.rados.0.smithi038.stdout:719: oid 25 updating version 0 to 39 2024-03-19T11:30:14.517 INFO:tasks.rados.rados.0.smithi038.stdout:719: oid 25 updating version 39 to 40 2024-03-19T11:30:14.517 INFO:tasks.rados.rados.0.smithi038.stdout:719: oid 25 version 40 is already newer than 37 2024-03-19T11:30:14.517 INFO:tasks.rados.rados.0.smithi038.stdout:719: oid 25 version 40 is already newer than 38 2024-03-19T11:30:14.517 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 25 v 40 (ObjNum 79 snap 6 seq_num 79) dirty exists 2024-03-19T11:30:14.517 INFO:tasks.rados.rados.0.smithi038.stdout:719: left oid 25 (ObjNum 79 snap 6 seq_num 79) 2024-03-19T11:30:14.517 INFO:tasks.rados.rados.0.smithi038.stdout:722: expect (ObjNum 53 snap 0 seq_num 53) 2024-03-19T11:30:14.523 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-19T11:30:14.596 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:14.607+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:14.645 INFO:tasks.rados.rados.0.smithi038.stdout:723: finishing write tid 1 to smithi03829856-41 2024-03-19T11:30:14.645 INFO:tasks.rados.rados.0.smithi038.stdout:723: finishing write tid 2 to smithi03829856-41 2024-03-19T11:30:14.645 INFO:tasks.rados.rados.0.smithi038.stdout:723: finishing write tid 3 to smithi03829856-41 2024-03-19T11:30:14.645 INFO:tasks.rados.rados.0.smithi038.stdout:725: finishing rollback tid 0 to smithi03829856-10 2024-03-19T11:30:14.645 INFO:tasks.rados.rados.0.smithi038.stdout:725: finishing rollback tid 1 to smithi03829856-10 2024-03-19T11:30:14.645 INFO:tasks.rados.rados.0.smithi038.stdout:726: expect (ObjNum 4 snap 0 seq_num 4) 2024-03-19T11:30:14.645 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:14.646 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:14.660 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-19T11:30:14.697 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:14.704+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:14.707 INFO:tasks.rados.rados.0.smithi038.stdout:725: finishing rollback tid 2 to smithi03829856-10 2024-03-19T11:30:14.707 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 10 v 38 (ObjNum 55 snap 1 seq_num 55) dirty exists 2024-03-19T11:30:14.708 INFO:tasks.rados.rados.0.smithi038.stdout:723: finishing write tid 4 to smithi03829856-41 2024-03-19T11:30:14.708 INFO:tasks.rados.rados.0.smithi038.stdout:723: finishing write tid 5 to smithi03829856-41 2024-03-19T11:30:14.708 INFO:tasks.rados.rados.0.smithi038.stdout:723: oid 41 updating version 0 to 47 2024-03-19T11:30:14.708 INFO:tasks.rados.rados.0.smithi038.stdout:723: oid 41 version 47 is already newer than 46 2024-03-19T11:30:14.708 INFO:tasks.rados.rados.0.smithi038.stdout:723: oid 41 updating version 47 to 48 2024-03-19T11:30:14.708 INFO:tasks.rados.rados.0.smithi038.stdout:723: oid 41 version 48 is already newer than 45 2024-03-19T11:30:14.708 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 41 v 48 (ObjNum 80 snap 6 seq_num 80) dirty exists 2024-03-19T11:30:14.708 INFO:tasks.rados.rados.0.smithi038.stdout:723: left oid 41 (ObjNum 80 snap 6 seq_num 80) 2024-03-19T11:30:14.708 INFO:tasks.rados.rados.0.smithi038.stdout:727: expect (ObjNum 36 snap 0 seq_num 36) 2024-03-19T11:30:14.765 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-03-19T11:30:14.789 INFO:tasks.rados.rados.0.smithi038.stdout:728: got (0) Success 2024-03-19T11:30:14.789 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 24 v 9 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-03-19T11:30:14.789 INFO:tasks.rados.rados.0.smithi038.stdout:715: done (15 left) 2024-03-19T11:30:14.789 INFO:tasks.rados.rados.0.smithi038.stdout:716: done (14 left) 2024-03-19T11:30:14.789 INFO:tasks.rados.rados.0.smithi038.stdout:717: done (13 left) 2024-03-19T11:30:14.789 INFO:tasks.rados.rados.0.smithi038.stdout:718: done (12 left) 2024-03-19T11:30:14.789 INFO:tasks.rados.rados.0.smithi038.stdout:719: done (11 left) 2024-03-19T11:30:14.789 INFO:tasks.rados.rados.0.smithi038.stdout:720: done (10 left) 2024-03-19T11:30:14.789 INFO:tasks.rados.rados.0.smithi038.stdout:721: done (9 left) 2024-03-19T11:30:14.789 INFO:tasks.rados.rados.0.smithi038.stdout:722: done (8 left) 2024-03-19T11:30:14.790 INFO:tasks.rados.rados.0.smithi038.stdout:723: done (7 left) 2024-03-19T11:30:14.790 INFO:tasks.rados.rados.0.smithi038.stdout:725: done (6 left) 2024-03-19T11:30:14.790 INFO:tasks.rados.rados.0.smithi038.stdout:726: done (5 left) 2024-03-19T11:30:14.790 INFO:tasks.rados.rados.0.smithi038.stdout:727: done (4 left) 2024-03-19T11:30:14.790 INFO:tasks.rados.rados.0.smithi038.stdout:728: done (3 left) 2024-03-19T11:30:14.790 INFO:tasks.rados.rados.0.smithi038.stdout:731: read oid 28 snap -1 2024-03-19T11:30:14.790 INFO:tasks.rados.rados.0.smithi038.stdout:731: expect deleted 2024-03-19T11:30:14.790 INFO:tasks.rados.rados.0.smithi038.stdout:732: snap_remove snap 5 2024-03-19T11:30:14.799 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:14.808+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:14.828 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:14.829 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:14.843 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-19T11:30:14.899 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:14.907+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:14.966 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:14.966 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:14.980 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-19T11:30:15.000 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:15.008+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:15.101 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:15.111+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:15.148 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:15.148 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:15.162 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-19T11:30:15.202 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:15.212+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:15.292 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:15.300+0000 7f70c4fdb640 -1 osd.0 36 handle_osd_map injecting map crc failure 2024-03-19T11:30:15.303 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:15.311+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:15.328 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:15.328 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:15.342 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-19T11:30:15.404 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:15.416+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:15.505 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:15.512+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:15.509 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:15.509 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:15.523 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-19T11:30:15.606 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:15.620+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:15.610 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:15.619+0000 7f143b8b4640 -1 osd.5 36 handle_osd_map injecting map crc failure 2024-03-19T11:30:15.650 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:15.650 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:15.665 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-19T11:30:15.708 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:15.720+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:15.784 INFO:tasks.rados.rados.0.smithi038.stdout:732: done (4 left) 2024-03-19T11:30:15.784 INFO:tasks.rados.rados.0.smithi038.stdout:733: set_chunk oid 26 target oid 2024-03-19T11:30:15.784 INFO:tasks.rados.rados.0.smithi038.stdout:733: set_chunk oid 26 offset: 0 length: 4000000 target oid smithi03829856- offset: 0 snap -1 2024-03-19T11:30:15.784 INFO:tasks.rados.rados.0.smithi038.stdout:734: read oid 16 snap -1 2024-03-19T11:30:15.784 INFO:tasks.rados.rados.0.smithi038.stdout:734: expect (ObjNum 65 snap 3 seq_num 65) 2024-03-19T11:30:15.784 INFO:tasks.rados.rados.0.smithi038.stdout:735: snap_create 2024-03-19T11:30:15.784 INFO:tasks.rados.rados.0.smithi038.stdout:724: expect (ObjNum 8 snap 0 seq_num 8) 2024-03-19T11:30:15.784 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:15.792+0000 7f6a7f9da640 -1 osd.1 37 handle_osd_map injecting map crc failure 2024-03-19T11:30:15.788 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:15.791+0000 7f923da1c640 -1 osd.7 37 handle_osd_map injecting map crc failure 2024-03-19T11:30:15.789 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:15.789 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:15.803 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-19T11:30:15.814 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:15.824+0000 7fa5de8d7640 -1 osd.3 37 handle_osd_map injecting map crc failure 2024-03-19T11:30:15.852 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:15.863+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:15.856 INFO:tasks.rados.rados.0.smithi038.stdout:729: got (0) Success 2024-03-19T11:30:15.856 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 17 v 9 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-03-19T11:30:15.856 INFO:tasks.rados.rados.0.smithi038.stdout:730: finishing write tid 1 to smithi03829856-18 2024-03-19T11:30:15.856 INFO:tasks.rados.rados.0.smithi038.stdout:730: finishing write tid 2 to smithi03829856-18 2024-03-19T11:30:15.856 INFO:tasks.rados.rados.0.smithi038.stdout:730: finishing write tid 3 to smithi03829856-18 2024-03-19T11:30:15.857 INFO:tasks.rados.rados.0.smithi038.stdout:730: finishing write tid 4 to smithi03829856-18 2024-03-19T11:30:15.857 INFO:tasks.rados.rados.0.smithi038.stdout:730: finishing write tid 5 to smithi03829856-18 2024-03-19T11:30:15.857 INFO:tasks.rados.rados.0.smithi038.stdout:730: oid 18 updating version 0 to 45 2024-03-19T11:30:15.857 INFO:tasks.rados.rados.0.smithi038.stdout:730: oid 18 version 45 is already newer than 42 2024-03-19T11:30:15.857 INFO:tasks.rados.rados.0.smithi038.stdout:730: oid 18 version 45 is already newer than 43 2024-03-19T11:30:15.857 INFO:tasks.rados.rados.0.smithi038.stdout:730: oid 18 version 45 is already newer than 44 2024-03-19T11:30:15.858 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 18 v 45 (ObjNum 81 snap 6 seq_num 81) dirty exists 2024-03-19T11:30:15.858 INFO:tasks.rados.rados.0.smithi038.stdout:730: left oid 18 (ObjNum 81 snap 6 seq_num 81) 2024-03-19T11:30:15.858 INFO:tasks.rados.rados.0.smithi038.stdout:733: finishing set_chunk to oid 26 2024-03-19T11:30:15.858 INFO:tasks.rados.rados.0.smithi038.stdout:733: got expected ERANGE (src dne) 2024-03-19T11:30:15.858 INFO:tasks.rados.rados.0.smithi038.stdout:734: expect (ObjNum 65 snap 3 seq_num 65) 2024-03-19T11:30:15.909 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:15.919+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:15.925 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:15.925 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:15.939 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-19T11:30:16.024 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:16.035+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:16.103 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:16.103 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:16.111 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:16.120+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:16.117 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-19T11:30:16.212 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:16.223+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:16.238 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-19T11:30:16.246 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:16.247 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:16.263 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-19T11:30:16.354 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:16.364+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:16.381 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:30:16.381 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:30:16.398 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:30:16.399 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:30:16.414 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:16.424+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:16.415 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:30:16.415 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:30:16.433 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:30:16.433 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:30:16.439 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:16.439 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:30:16.439 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:30:16.439 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:30:16.439 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:16.454 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-19T11:30:16.454 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:30:16.455 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:30:16.475 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:30:16.475 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:30:16.498 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:30:16.498 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:30:16.516 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:16.523+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:16.520 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:30:16.520 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:30:16.587 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:16.588 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:30:16.588 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:30:16.588 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:30:16.588 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.588 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:34 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'20, mlcod=15'16)", 2024-03-19T11:30:16.588 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.077542+0000", 2024-03-19T11:30:16.588 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 31.522388166999999, 2024-03-19T11:30:16.588 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0064305179999999997, 2024-03-19T11:30:16.588 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:16.588 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:16.588 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:16.588 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.588 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:16.589 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.077542+0000", 2024-03-19T11:30:16.589 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:16.589 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.589 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.589 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:16.589 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.077542+0000", 2024-03-19T11:30:16.589 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:16.589 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.589 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.589 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:16.589 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.077543+0000", 2024-03-19T11:30:16.589 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.387e-06 2024-03-19T11:30:16.589 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.589 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.590 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:16.590 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.077555+0000", 2024-03-19T11:30:16.590 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1759999999999999e-05 2024-03-19T11:30:16.590 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.590 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.590 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:16.590 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.077580+0000", 2024-03-19T11:30:16.590 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.529e-05 2024-03-19T11:30:16.590 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.590 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.590 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:16.590 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.077596+0000", 2024-03-19T11:30:16.590 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5466000000000001e-05 2024-03-19T11:30:16.591 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.591 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.591 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:16.591 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.077753+0000", 2024-03-19T11:30:16.591 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000157026 2024-03-19T11:30:16.591 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.591 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.591 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:16.591 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.077789+0000", 2024-03-19T11:30:16.591 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6259e-05 2024-03-19T11:30:16.591 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.591 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.591 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:16.591 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.083910+0000", 2024-03-19T11:30:16.592 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0061212080000000004 2024-03-19T11:30:16.592 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.592 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.592 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:16.592 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.083972+0000", 2024-03-19T11:30:16.592 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2121999999999999e-05 2024-03-19T11:30:16.592 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:16.592 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:16.592 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:16.592 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.592 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.592 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4201.0:132 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'90, mlcod=16'85)", 2024-03-19T11:30:16.592 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:29:45.806470+0000", 2024-03-19T11:30:16.592 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 30.793460189000001, 2024-03-19T11:30:16.593 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0066812390000000003, 2024-03-19T11:30:16.593 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:16.593 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:16.593 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:16.593 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.593 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:16.593 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806470+0000", 2024-03-19T11:30:16.593 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:16.593 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.593 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.593 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:16.593 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806470+0000", 2024-03-19T11:30:16.593 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:16.593 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.593 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.594 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:16.594 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806471+0000", 2024-03-19T11:30:16.594 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.486e-06 2024-03-19T11:30:16.594 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.594 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.594 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:16.594 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806485+0000", 2024-03-19T11:30:16.594 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3380000000000001e-05 2024-03-19T11:30:16.594 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.594 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.594 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:16.594 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806509+0000", 2024-03-19T11:30:16.594 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4617000000000002e-05 2024-03-19T11:30:16.595 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.595 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.595 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:16.595 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.806524+0000", 2024-03-19T11:30:16.595 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4688999999999999e-05 2024-03-19T11:30:16.595 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.595 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.595 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:16.595 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807070+0000", 2024-03-19T11:30:16.595 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00054601300000000003 2024-03-19T11:30:16.595 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.595 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.595 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:16.595 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.807104+0000", 2024-03-19T11:30:16.595 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3575999999999998e-05 2024-03-19T11:30:16.596 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.596 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.596 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:16.596 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813077+0000", 2024-03-19T11:30:16.596 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0059733590000000001 2024-03-19T11:30:16.596 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.596 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.596 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:16.596 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:29:45.813151+0000", 2024-03-19T11:30:16.596 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4119e-05 2024-03-19T11:30:16.596 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:16.596 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:16.596 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:16.596 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.597 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.597 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:6 4.6 4:626acd4d:::smithi03829856-2:head [write 1542129~474678] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-03-19T11:30:16.597 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.696120+0000", 2024-03-19T11:30:16.597 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 11.903810180000001, 2024-03-19T11:30:16.597 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0082896359999999995, 2024-03-19T11:30:16.597 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:16.597 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:16.597 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:16.597 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:16.597 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:16.597 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 6 2024-03-19T11:30:16.597 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.597 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:16.597 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.597 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:16.598 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696120+0000", 2024-03-19T11:30:16.598 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:16.598 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.598 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.598 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:16.598 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696120+0000", 2024-03-19T11:30:16.598 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:16.598 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.598 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.598 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:16.598 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696123+0000", 2024-03-19T11:30:16.598 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.661e-06 2024-03-19T11:30:16.598 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.598 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.599 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:16.599 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696282+0000", 2024-03-19T11:30:16.599 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015960400000000001 2024-03-19T11:30:16.599 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.599 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.599 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:16.599 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696307+0000", 2024-03-19T11:30:16.599 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5352999999999999e-05 2024-03-19T11:30:16.599 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.599 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.599 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:16.599 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696320+0000", 2024-03-19T11:30:16.599 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2847000000000001e-05 2024-03-19T11:30:16.599 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.600 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.600 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:16.600 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696403+0000", 2024-03-19T11:30:16.600 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2507000000000001e-05 2024-03-19T11:30:16.600 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.600 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.600 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:16.600 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696585+0000", 2024-03-19T11:30:16.600 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00018234999999999999 2024-03-19T11:30:16.600 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.600 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.600 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:16.600 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696770+0000", 2024-03-19T11:30:16.600 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000184377 2024-03-19T11:30:16.600 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.601 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.601 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:16.601 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.699608+0000", 2024-03-19T11:30:16.601 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00283797 2024-03-19T11:30:16.601 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.601 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.601 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:16.601 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.704252+0000", 2024-03-19T11:30:16.601 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046443980000000001 2024-03-19T11:30:16.601 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.601 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.601 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:16.601 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.704349+0000", 2024-03-19T11:30:16.601 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7388999999999999e-05 2024-03-19T11:30:16.602 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.602 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.602 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:16.602 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.704409+0000", 2024-03-19T11:30:16.602 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0180000000000003e-05 2024-03-19T11:30:16.602 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:16.602 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:16.602 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:16.602 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.602 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.602 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:7 4.6 4:626acd4d:::smithi03829856-2:head [setxattr _header (55) in=7b,truncate 2016807] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-03-19T11:30:16.602 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.696352+0000", 2024-03-19T11:30:16.602 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 11.903578039999999, 2024-03-19T11:30:16.603 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0084077100000000005, 2024-03-19T11:30:16.603 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:16.603 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:16.603 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:16.603 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:16.603 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:16.603 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 7 2024-03-19T11:30:16.603 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.603 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:16.603 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.603 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:16.603 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696352+0000", 2024-03-19T11:30:16.603 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:16.604 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.604 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.604 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:16.604 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696352+0000", 2024-03-19T11:30:16.604 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:16.604 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.604 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.604 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:16.604 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696353+0000", 2024-03-19T11:30:16.604 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1399999999999996e-07 2024-03-19T11:30:16.604 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.604 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.604 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:16.604 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696358+0000", 2024-03-19T11:30:16.605 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5300000000000004e-06 2024-03-19T11:30:16.605 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.605 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.605 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:16.605 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696368+0000", 2024-03-19T11:30:16.605 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6209999999999999e-06 2024-03-19T11:30:16.605 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.605 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.605 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:16.605 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696374+0000", 2024-03-19T11:30:16.605 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5599999999999999e-06 2024-03-19T11:30:16.605 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.605 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.605 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:16.606 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.698823+0000", 2024-03-19T11:30:16.606 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0024489899999999998 2024-03-19T11:30:16.606 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.606 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.606 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:16.606 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.698925+0000", 2024-03-19T11:30:16.606 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010166399999999999 2024-03-19T11:30:16.606 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.606 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.606 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:16.606 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.699071+0000", 2024-03-19T11:30:16.606 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000146014 2024-03-19T11:30:16.606 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.606 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.607 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:16.607 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.703885+0000", 2024-03-19T11:30:16.607 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.004813866 2024-03-19T11:30:16.607 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.607 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.607 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:16.607 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.704622+0000", 2024-03-19T11:30:16.607 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00073729600000000002 2024-03-19T11:30:16.607 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.607 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.607 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:16.607 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.704688+0000", 2024-03-19T11:30:16.607 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6192000000000006e-05 2024-03-19T11:30:16.607 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.607 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.607 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:16.608 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.704760+0000", 2024-03-19T11:30:16.608 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1262999999999999e-05 2024-03-19T11:30:16.608 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:16.608 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:16.608 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:16.608 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.608 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.608 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:8 4.6 4:626acd4d:::smithi03829856-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e27)", 2024-03-19T11:30:16.608 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.696385+0000", 2024-03-19T11:30:16.608 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 11.903544740999999, 2024-03-19T11:30:16.608 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0086443730000000003, 2024-03-19T11:30:16.608 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:16.608 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:16.608 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:16.609 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:16.609 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:16.609 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 8 2024-03-19T11:30:16.609 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.609 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:16.609 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.609 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:16.609 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696385+0000", 2024-03-19T11:30:16.609 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:16.609 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.609 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.609 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:16.609 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696385+0000", 2024-03-19T11:30:16.609 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:16.609 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.610 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.610 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:16.610 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696386+0000", 2024-03-19T11:30:16.610 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3e-07 2024-03-19T11:30:16.610 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.610 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.610 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:16.610 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696390+0000", 2024-03-19T11:30:16.610 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0130000000000004e-06 2024-03-19T11:30:16.610 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.610 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.610 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:16.610 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696399+0000", 2024-03-19T11:30:16.611 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9139999999999997e-06 2024-03-19T11:30:16.611 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.611 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.611 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:16.611 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696405+0000", 2024-03-19T11:30:16.611 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4699999999999999e-06 2024-03-19T11:30:16.611 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.611 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.611 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:16.611 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.699495+0000", 2024-03-19T11:30:16.611 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030890969999999998 2024-03-19T11:30:16.612 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.612 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.612 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-03-19T11:30:16.612 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.699562+0000", 2024-03-19T11:30:16.612 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7980999999999998e-05 2024-03-19T11:30:16.612 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.612 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.612 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:16.612 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.704809+0000", 2024-03-19T11:30:16.612 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0052468769999999996 2024-03-19T11:30:16.612 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.612 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.612 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:16.612 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.704909+0000", 2024-03-19T11:30:16.612 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9189000000000002e-05 2024-03-19T11:30:16.613 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.613 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.613 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:16.613 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.705030+0000", 2024-03-19T11:30:16.613 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012110199999999999 2024-03-19T11:30:16.613 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:16.613 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:16.613 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:16.613 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.613 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.613 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:21 4.b 4:db322469:::smithi03829856-6:head [write 484035~776245,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:30:16.613 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.746924+0000", 2024-03-19T11:30:16.613 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 11.853005997, 2024-03-19T11:30:16.613 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074997270000000003, 2024-03-19T11:30:16.613 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:16.613 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:16.614 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:16.614 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:16.614 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:16.614 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 21 2024-03-19T11:30:16.614 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.614 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:16.614 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.614 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:16.614 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.746924+0000", 2024-03-19T11:30:16.614 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:16.614 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.614 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.614 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:16.614 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.746924+0000", 2024-03-19T11:30:16.615 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:16.615 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.615 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.615 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:16.615 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.746927+0000", 2024-03-19T11:30:16.615 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4820000000000001e-06 2024-03-19T11:30:16.615 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.615 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.615 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:16.615 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.747304+0000", 2024-03-19T11:30:16.615 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037742299999999999 2024-03-19T11:30:16.615 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.615 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.615 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:16.616 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.747326+0000", 2024-03-19T11:30:16.616 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1648999999999999e-05 2024-03-19T11:30:16.616 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.616 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.616 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:16.616 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.747336+0000", 2024-03-19T11:30:16.616 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0288e-05 2024-03-19T11:30:16.616 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.616 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.616 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:16.616 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.747414+0000", 2024-03-19T11:30:16.616 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8085999999999997e-05 2024-03-19T11:30:16.616 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.616 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.616 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-03-19T11:30:16.617 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.747421+0000", 2024-03-19T11:30:16.617 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9609999999999999e-06 2024-03-19T11:30:16.617 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.617 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.617 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:16.617 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.749216+0000", 2024-03-19T11:30:16.617 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001795172 2024-03-19T11:30:16.617 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.618 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.618 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:16.618 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.749355+0000", 2024-03-19T11:30:16.618 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013904799999999999 2024-03-19T11:30:16.618 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.618 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.618 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:16.619 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.749513+0000", 2024-03-19T11:30:16.619 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000157453 2024-03-19T11:30:16.619 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.619 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.619 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:16.619 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.754081+0000", 2024-03-19T11:30:16.619 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0045681230000000003 2024-03-19T11:30:16.619 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.619 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.619 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:16.619 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.754323+0000", 2024-03-19T11:30:16.619 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00024241900000000001 2024-03-19T11:30:16.619 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.619 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.620 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:16.620 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.754373+0000", 2024-03-19T11:30:16.620 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9632999999999997e-05 2024-03-19T11:30:16.620 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.620 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.620 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:16.620 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.754424+0000", 2024-03-19T11:30:16.620 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0989999999999998e-05 2024-03-19T11:30:16.620 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:16.620 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:16.620 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:16.620 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.620 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.620 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:889 3.1 3:99a49d1b:::895a63407b6109b3cca9a6f5ef827134d56679bef42daadc5892269682c428ed:head [write 0~781312] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:16.621 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.277573+0000", 2024-03-19T11:30:16.621 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 7.3223571229999997, 2024-03-19T11:30:16.621 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074918249999999997, 2024-03-19T11:30:16.621 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:16.621 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:16.621 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:16.621 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:16.621 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:16.621 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 889 2024-03-19T11:30:16.621 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.621 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:16.621 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.622 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:16.622 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.277573+0000", 2024-03-19T11:30:16.622 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:16.622 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.622 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.622 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:16.622 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.277573+0000", 2024-03-19T11:30:16.622 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:16.622 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.622 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.622 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:16.622 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.277576+0000", 2024-03-19T11:30:16.622 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0869999999999998e-06 2024-03-19T11:30:16.622 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.623 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.623 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:16.623 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.278553+0000", 2024-03-19T11:30:16.623 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0009769990000000001 2024-03-19T11:30:16.623 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.623 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.623 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:16.623 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.278604+0000", 2024-03-19T11:30:16.623 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0679e-05 2024-03-19T11:30:16.623 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.623 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.623 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:16.623 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.278629+0000", 2024-03-19T11:30:16.623 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5673e-05 2024-03-19T11:30:16.624 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.624 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.624 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:16.624 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.278741+0000", 2024-03-19T11:30:16.624 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000112151 2024-03-19T11:30:16.624 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.624 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.624 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:16.624 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.278925+0000", 2024-03-19T11:30:16.624 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000183673 2024-03-19T11:30:16.624 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.624 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.624 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4", 2024-03-19T11:30:16.624 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.279096+0000", 2024-03-19T11:30:16.625 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017097 2024-03-19T11:30:16.625 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.625 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.625 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:16.625 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.281583+0000", 2024-03-19T11:30:16.625 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0024863720000000001 2024-03-19T11:30:16.625 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.625 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.625 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:16.625 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.284892+0000", 2024-03-19T11:30:16.625 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033099359999999999 2024-03-19T11:30:16.625 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.625 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.625 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:16.625 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.284988+0000", 2024-03-19T11:30:16.626 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.5159000000000002e-05 2024-03-19T11:30:16.626 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.626 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.626 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:16.626 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.285065+0000", 2024-03-19T11:30:16.626 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7126000000000007e-05 2024-03-19T11:30:16.626 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:16.626 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:16.626 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:16.626 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.626 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.626 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1009 4.6 4:6868af91:::smithi03829856-10:6 [sparse-read 0~1716640,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:16.626 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.760673+0000", 2024-03-19T11:30:16.626 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.839256813, 2024-03-19T11:30:16.626 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0090082879999999997, 2024-03-19T11:30:16.627 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:16.627 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:16.627 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:16.627 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:16.627 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:16.627 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1009 2024-03-19T11:30:16.627 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.627 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:16.627 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.627 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:16.627 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.760673+0000", 2024-03-19T11:30:16.627 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:16.627 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.627 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.628 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:16.628 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.760673+0000", 2024-03-19T11:30:16.628 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:16.628 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.628 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.628 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:16.628 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.760675+0000", 2024-03-19T11:30:16.628 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9549999999999999e-06 2024-03-19T11:30:16.628 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.628 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.628 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:16.628 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.760685+0000", 2024-03-19T11:30:16.628 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0152999999999999e-05 2024-03-19T11:30:16.628 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.628 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.629 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:16.629 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.760712+0000", 2024-03-19T11:30:16.629 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6268e-05 2024-03-19T11:30:16.629 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.629 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.629 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:16.629 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.760723+0000", 2024-03-19T11:30:16.629 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.115e-05 2024-03-19T11:30:16.629 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.629 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.629 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:16.629 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.760803+0000", 2024-03-19T11:30:16.629 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9875000000000004e-05 2024-03-19T11:30:16.629 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.630 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.630 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-03-19T11:30:16.630 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.760810+0000", 2024-03-19T11:30:16.630 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.345e-06 2024-03-19T11:30:16.630 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.630 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.630 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:16.630 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.763506+0000", 2024-03-19T11:30:16.630 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0026961989999999998 2024-03-19T11:30:16.630 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.630 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.630 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:16.630 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.763596+0000", 2024-03-19T11:30:16.630 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9584000000000002e-05 2024-03-19T11:30:16.631 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.631 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.631 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:16.631 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.769681+0000", 2024-03-19T11:30:16.631 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0060857589999999996 2024-03-19T11:30:16.631 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:16.631 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:16.631 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:16.631 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.631 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.631 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1010 4.6 4:6868af91:::smithi03829856-10:6 [sparse-read 0~1716640] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:16.631 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.760749+0000", 2024-03-19T11:30:16.631 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.8391810639999999, 2024-03-19T11:30:16.631 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0092191900000000004, 2024-03-19T11:30:16.631 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:16.632 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:16.632 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:16.632 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:16.632 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:16.632 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1010 2024-03-19T11:30:16.632 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.632 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:16.632 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.632 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:16.632 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.760749+0000", 2024-03-19T11:30:16.632 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:16.632 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.632 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.632 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:16.632 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.760749+0000", 2024-03-19T11:30:16.633 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:16.633 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.633 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.633 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:16.633 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.760750+0000", 2024-03-19T11:30:16.633 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5400000000000001e-07 2024-03-19T11:30:16.633 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.633 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.633 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:16.633 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.760753+0000", 2024-03-19T11:30:16.633 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3560000000000001e-06 2024-03-19T11:30:16.633 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.633 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.633 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:16.634 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.760762+0000", 2024-03-19T11:30:16.634 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2320000000000007e-06 2024-03-19T11:30:16.634 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.634 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.634 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:16.634 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.760766+0000", 2024-03-19T11:30:16.634 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0470000000000004e-06 2024-03-19T11:30:16.634 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.634 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.634 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:16.634 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.760873+0000", 2024-03-19T11:30:16.634 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000106638 2024-03-19T11:30:16.634 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.634 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.635 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:16.635 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.760878+0000", 2024-03-19T11:30:16.635 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.429e-06 2024-03-19T11:30:16.635 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.635 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.635 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:16.635 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.769723+0000", 2024-03-19T11:30:16.635 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0088447219999999993 2024-03-19T11:30:16.635 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.635 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.635 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:16.635 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.769793+0000", 2024-03-19T11:30:16.635 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9663e-05 2024-03-19T11:30:16.635 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.635 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.636 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:16.636 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.769968+0000", 2024-03-19T11:30:16.636 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000175549 2024-03-19T11:30:16.636 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:16.636 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:16.636 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:16.636 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.636 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.636 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1011 4.6 4:6868af91:::smithi03829856-10:6 [read 0~2924385,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:16.636 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.760795+0000", 2024-03-19T11:30:16.636 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.839135395, 2024-03-19T11:30:16.636 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010026474, 2024-03-19T11:30:16.636 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:16.636 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:16.636 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:16.637 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:16.637 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:16.637 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1011 2024-03-19T11:30:16.637 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.637 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:16.637 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.637 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:16.637 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.760795+0000", 2024-03-19T11:30:16.637 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:16.637 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.637 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.637 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:16.637 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.760795+0000", 2024-03-19T11:30:16.637 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:16.637 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.638 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.638 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:16.638 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.760795+0000", 2024-03-19T11:30:16.638 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0900000000000002e-07 2024-03-19T11:30:16.638 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.638 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.638 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:16.638 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.760799+0000", 2024-03-19T11:30:16.638 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2119999999999997e-06 2024-03-19T11:30:16.638 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.638 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.638 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:16.638 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.760808+0000", 2024-03-19T11:30:16.639 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6219999999999998e-06 2024-03-19T11:30:16.639 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.639 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.639 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:16.639 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.760814+0000", 2024-03-19T11:30:16.639 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2450000000000003e-06 2024-03-19T11:30:16.639 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.639 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.639 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:16.639 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.760921+0000", 2024-03-19T11:30:16.639 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000106838 2024-03-19T11:30:16.639 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.639 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.639 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:16.639 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.760927+0000", 2024-03-19T11:30:16.640 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5419999999999996e-06 2024-03-19T11:30:16.640 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.640 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.640 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:16.640 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.769991+0000", 2024-03-19T11:30:16.640 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0090641920000000004 2024-03-19T11:30:16.640 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.640 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.640 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:16.640 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.770042+0000", 2024-03-19T11:30:16.640 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1190000000000003e-05 2024-03-19T11:30:16.640 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.640 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.640 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:16.640 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.770821+0000", 2024-03-19T11:30:16.641 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00077912399999999996 2024-03-19T11:30:16.641 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:16.641 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:16.641 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:16.641 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.641 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.641 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1019 4.6 4:626acd4d:::smithi03829856-2:head [read 601600~207872] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:16.641 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.760890+0000", 2024-03-19T11:30:16.641 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.8390401980000002, 2024-03-19T11:30:16.641 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011505345, 2024-03-19T11:30:16.641 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:16.641 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:16.641 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:16.641 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:16.641 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:16.642 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1019 2024-03-19T11:30:16.642 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.642 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:16.642 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.642 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:16.642 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.760890+0000", 2024-03-19T11:30:16.642 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:16.642 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.642 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.642 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:16.642 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.760890+0000", 2024-03-19T11:30:16.642 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:16.642 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.643 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.643 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:16.643 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.760890+0000", 2024-03-19T11:30:16.643 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0600000000000001e-07 2024-03-19T11:30:16.643 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.643 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.643 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:16.643 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.760893+0000", 2024-03-19T11:30:16.643 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1669999999999999e-06 2024-03-19T11:30:16.643 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.643 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.643 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:16.643 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.760912+0000", 2024-03-19T11:30:16.643 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8359000000000002e-05 2024-03-19T11:30:16.644 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.644 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.644 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:16.644 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.760918+0000", 2024-03-19T11:30:16.644 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6019999999999996e-06 2024-03-19T11:30:16.644 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.644 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.644 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:16.644 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.760954+0000", 2024-03-19T11:30:16.644 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5262999999999998e-05 2024-03-19T11:30:16.644 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.644 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.644 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:16.644 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.760958+0000", 2024-03-19T11:30:16.644 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7690000000000004e-06 2024-03-19T11:30:16.645 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.645 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.645 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:16.645 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.772082+0000", 2024-03-19T11:30:16.645 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011123503999999999 2024-03-19T11:30:16.645 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.645 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.645 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:16.645 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.772204+0000", 2024-03-19T11:30:16.645 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012215899999999999 2024-03-19T11:30:16.645 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.645 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.645 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:16.645 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.772395+0000", 2024-03-19T11:30:16.645 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019111599999999999 2024-03-19T11:30:16.646 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:16.646 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:16.646 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:16.646 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.646 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.646 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1037 4.b 4:d0dfa670:::smithi03829856-42:head [tier-promote] snapc 6=[6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:16.646 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.207136+0000", 2024-03-19T11:30:16.646 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.3927937309999998, 2024-03-19T11:30:16.646 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0068294870000000004, 2024-03-19T11:30:16.646 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:16.646 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:16.646 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:16.646 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:16.647 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:16.647 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1037 2024-03-19T11:30:16.647 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.647 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:16.647 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.647 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:16.647 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.207136+0000", 2024-03-19T11:30:16.647 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:16.647 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.647 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.647 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:16.647 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.207136+0000", 2024-03-19T11:30:16.647 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:16.647 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.647 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.648 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:16.648 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.207137+0000", 2024-03-19T11:30:16.648 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3599999999999999e-07 2024-03-19T11:30:16.648 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.648 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.648 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:16.648 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.207139+0000", 2024-03-19T11:30:16.648 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5339999999999998e-06 2024-03-19T11:30:16.648 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.648 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.648 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:16.648 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.207146+0000", 2024-03-19T11:30:16.648 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6089999999999997e-06 2024-03-19T11:30:16.648 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.648 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.649 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:16.649 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.207155+0000", 2024-03-19T11:30:16.649 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3680000000000008e-06 2024-03-19T11:30:16.649 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.649 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.649 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:16.649 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.211000+0000", 2024-03-19T11:30:16.649 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038443689999999998 2024-03-19T11:30:16.649 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.649 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.649 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:16.649 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.211073+0000", 2024-03-19T11:30:16.649 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3731000000000003e-05 2024-03-19T11:30:16.649 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.649 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.650 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:16.650 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.213966+0000", 2024-03-19T11:30:16.650 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00289244 2024-03-19T11:30:16.650 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:16.650 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:16.650 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:16.650 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.650 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.650 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1061 3.1 3:97ecf0cc:::727f83c09505a9e6c5af0b055849bb93e385b34c7319dfb26afbcd3c5ae450db:head [write 0~508928] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:16.650 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.777106+0000", 2024-03-19T11:30:16.650 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 2.8228238289999998, 2024-03-19T11:30:16.650 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0076119940000000004, 2024-03-19T11:30:16.650 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:16.650 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:16.651 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:16.651 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:16.651 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:16.651 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1061 2024-03-19T11:30:16.651 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.651 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:16.651 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.651 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:16.651 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.777106+0000", 2024-03-19T11:30:16.651 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:16.651 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.651 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.651 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:16.651 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.777106+0000", 2024-03-19T11:30:16.652 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:16.652 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.652 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.652 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:16.652 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.777109+0000", 2024-03-19T11:30:16.652 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.118e-06 2024-03-19T11:30:16.652 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.652 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.652 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:16.652 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.777288+0000", 2024-03-19T11:30:16.652 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000178247 2024-03-19T11:30:16.652 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.652 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.652 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:16.652 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.777324+0000", 2024-03-19T11:30:16.653 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6010000000000003e-05 2024-03-19T11:30:16.653 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.653 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.653 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:16.653 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.777349+0000", 2024-03-19T11:30:16.653 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5043000000000001e-05 2024-03-19T11:30:16.653 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.653 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.653 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:16.653 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.777439+0000", 2024-03-19T11:30:16.653 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0044999999999996e-05 2024-03-19T11:30:16.653 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.653 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.653 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:16.653 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.777600+0000", 2024-03-19T11:30:16.654 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016176999999999999 2024-03-19T11:30:16.654 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.654 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.654 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4", 2024-03-19T11:30:16.654 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.777761+0000", 2024-03-19T11:30:16.654 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016063000000000001 2024-03-19T11:30:16.654 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.654 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.654 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:16.654 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.779292+0000", 2024-03-19T11:30:16.654 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001530577 2024-03-19T11:30:16.654 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.654 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.655 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:16.655 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784612+0000", 2024-03-19T11:30:16.655 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053201170000000001 2024-03-19T11:30:16.655 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.655 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.655 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:16.655 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784675+0000", 2024-03-19T11:30:16.655 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3674000000000001e-05 2024-03-19T11:30:16.655 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.655 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.655 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:16.655 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784718+0000", 2024-03-19T11:30:16.655 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2762999999999997e-05 2024-03-19T11:30:16.655 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:16.655 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:16.656 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:16.656 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.656 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.656 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.1.14:34 3.1 3:a4be7fb8:::7f04e86900586c72cd4c57b7e0dbaf875a66e7b107baf407c3fc7b5fe6f6321e:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:16.656 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.794309+0000", 2024-03-19T11:30:16.656 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 2.8056207070000001, 2024-03-19T11:30:16.656 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0090512300000000004, 2024-03-19T11:30:16.656 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:16.656 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:16.656 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:16.656 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.656 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:16.656 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.794309+0000", 2024-03-19T11:30:16.656 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:16.657 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.657 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.657 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:16.657 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.794309+0000", 2024-03-19T11:30:16.657 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:16.657 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.657 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.657 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:16.657 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.794312+0000", 2024-03-19T11:30:16.657 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4849999999999999e-06 2024-03-19T11:30:16.657 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.657 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.657 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:16.657 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.794524+0000", 2024-03-19T11:30:16.658 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00021189099999999999 2024-03-19T11:30:16.658 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.658 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.658 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:16.658 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.794548+0000", 2024-03-19T11:30:16.658 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4805000000000001e-05 2024-03-19T11:30:16.658 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.658 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.658 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:16.658 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.794586+0000", 2024-03-19T11:30:16.658 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7954000000000003e-05 2024-03-19T11:30:16.658 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.658 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.658 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:16.659 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.794693+0000", 2024-03-19T11:30:16.659 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010689200000000001 2024-03-19T11:30:16.659 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.659 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.659 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:16.659 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.794892+0000", 2024-03-19T11:30:16.659 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019886400000000001 2024-03-19T11:30:16.659 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.659 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.659 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4", 2024-03-19T11:30:16.659 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.795156+0000", 2024-03-19T11:30:16.659 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00026380300000000001 2024-03-19T11:30:16.659 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.659 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.659 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:16.660 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.799722+0000", 2024-03-19T11:30:16.660 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0045656430000000003 2024-03-19T11:30:16.660 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.660 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.660 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:16.660 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.803296+0000", 2024-03-19T11:30:16.660 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0035743020000000001 2024-03-19T11:30:16.660 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.660 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.660 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:16.660 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.803330+0000", 2024-03-19T11:30:16.660 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4196999999999998e-05 2024-03-19T11:30:16.660 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.660 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.660 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:16.661 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.803361+0000", 2024-03-19T11:30:16.661 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0394000000000001e-05 2024-03-19T11:30:16.661 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:16.661 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:16.661 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:16.661 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.661 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.661 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.1.14:40 3.1 3:91b37eb6:::4698f7ab424670a44a0ed8f4fcb5f6dcadf322a90fdca07dd4e592a352bb4762:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:16.661 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.795578+0000", 2024-03-19T11:30:16.661 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 2.8043521710000001, 2024-03-19T11:30:16.661 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.007904065, 2024-03-19T11:30:16.661 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:16.661 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:16.661 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:16.661 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.662 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:16.662 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.795578+0000", 2024-03-19T11:30:16.662 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:16.662 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.662 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.662 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:16.662 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.795578+0000", 2024-03-19T11:30:16.662 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:16.662 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.662 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.662 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:16.662 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.795580+0000", 2024-03-19T11:30:16.662 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9429999999999999e-06 2024-03-19T11:30:16.662 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.663 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.663 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:16.663 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.795644+0000", 2024-03-19T11:30:16.663 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4398000000000002e-05 2024-03-19T11:30:16.663 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.663 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.663 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:16.663 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.795659+0000", 2024-03-19T11:30:16.663 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5304000000000001e-05 2024-03-19T11:30:16.663 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.663 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.663 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:16.663 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.795669+0000", 2024-03-19T11:30:16.663 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2890000000000003e-06 2024-03-19T11:30:16.663 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.664 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.664 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:16.664 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.796325+0000", 2024-03-19T11:30:16.664 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00065582499999999996 2024-03-19T11:30:16.664 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.664 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.664 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:16.664 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.796452+0000", 2024-03-19T11:30:16.664 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012785699999999999 2024-03-19T11:30:16.664 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.664 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.664 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4", 2024-03-19T11:30:16.664 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.796711+0000", 2024-03-19T11:30:16.664 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000259034 2024-03-19T11:30:16.664 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.665 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.665 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:16.665 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.800313+0000", 2024-03-19T11:30:16.665 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0036010059999999999 2024-03-19T11:30:16.665 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.665 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.665 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:16.665 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.803406+0000", 2024-03-19T11:30:16.665 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030933010000000001 2024-03-19T11:30:16.665 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.665 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.665 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:16.665 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.803439+0000", 2024-03-19T11:30:16.665 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2702e-05 2024-03-19T11:30:16.665 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.666 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.666 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:16.666 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.803482+0000", 2024-03-19T11:30:16.666 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3405999999999997e-05 2024-03-19T11:30:16.666 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:16.666 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:16.666 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:16.666 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.666 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.666 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.1.14:41 3.1 3:87527c9e:::cae462634b7a968897fe5e1960a361d42b8eb9292891e6b1d17dd542420f2e5f:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:16.666 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.795690+0000", 2024-03-19T11:30:16.666 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 2.8042403020000002, 2024-03-19T11:30:16.666 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0082936139999999995, 2024-03-19T11:30:16.666 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:16.667 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:16.667 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:16.667 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.667 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:16.667 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.795690+0000", 2024-03-19T11:30:16.667 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:16.667 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.667 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.667 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:16.667 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.795690+0000", 2024-03-19T11:30:16.667 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:16.667 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.667 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.668 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:16.668 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.795690+0000", 2024-03-19T11:30:16.668 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1500000000000004e-07 2024-03-19T11:30:16.668 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.668 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.668 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:16.668 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.795790+0000", 2024-03-19T11:30:16.668 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9870000000000004e-05 2024-03-19T11:30:16.668 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.668 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.668 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:16.668 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.795800+0000", 2024-03-19T11:30:16.668 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4329999999999993e-06 2024-03-19T11:30:16.668 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.669 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.669 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:16.669 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.795806+0000", 2024-03-19T11:30:16.669 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.849e-06 2024-03-19T11:30:16.669 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.669 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.669 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:16.669 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.798296+0000", 2024-03-19T11:30:16.669 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0024897740000000002 2024-03-19T11:30:16.669 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.669 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.669 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:16.669 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.798442+0000", 2024-03-19T11:30:16.669 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014593700000000001 2024-03-19T11:30:16.669 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.670 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.670 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4", 2024-03-19T11:30:16.670 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.798951+0000", 2024-03-19T11:30:16.670 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00050883000000000005 2024-03-19T11:30:16.670 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.670 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.670 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:16.670 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.800685+0000", 2024-03-19T11:30:16.670 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017336820000000001 2024-03-19T11:30:16.670 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.670 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.670 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:16.670 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.803934+0000", 2024-03-19T11:30:16.671 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0032496449999999998 2024-03-19T11:30:16.671 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.671 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.671 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:16.671 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.803963+0000", 2024-03-19T11:30:16.671 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8481000000000001e-05 2024-03-19T11:30:16.671 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.671 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.671 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:16.671 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.803983+0000", 2024-03-19T11:30:16.671 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0497999999999999e-05 2024-03-19T11:30:16.671 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:16.671 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:16.671 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:16.672 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.672 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.672 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1087 4.6 4:626acd4d:::smithi03829856-2:2 [read 0~2016807,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:16.672 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:14.006599+0000", 2024-03-19T11:30:16.672 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 2.5933314039999997, 2024-03-19T11:30:16.672 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0091590560000000005, 2024-03-19T11:30:16.672 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:16.672 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:16.672 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:16.672 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:16.672 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:16.672 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1087 2024-03-19T11:30:16.672 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.672 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:16.673 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.673 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:16.673 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006599+0000", 2024-03-19T11:30:16.673 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:16.673 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.673 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.673 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:16.673 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006599+0000", 2024-03-19T11:30:16.673 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:16.673 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.673 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.673 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:16.673 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006599+0000", 2024-03-19T11:30:16.673 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2e-07 2024-03-19T11:30:16.674 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.674 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.674 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:16.674 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006602+0000", 2024-03-19T11:30:16.674 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0900000000000001e-06 2024-03-19T11:30:16.674 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.674 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.674 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:16.674 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006610+0000", 2024-03-19T11:30:16.674 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3699999999999995e-06 2024-03-19T11:30:16.674 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.674 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.674 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:16.674 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006616+0000", 2024-03-19T11:30:16.675 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4210000000000003e-06 2024-03-19T11:30:16.675 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.675 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.675 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:16.675 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.008650+0000", 2024-03-19T11:30:16.675 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0020340919999999999 2024-03-19T11:30:16.675 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.675 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.675 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:16.675 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.008746+0000", 2024-03-19T11:30:16.675 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6372000000000005e-05 2024-03-19T11:30:16.675 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.675 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.675 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:16.675 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.015758+0000", 2024-03-19T11:30:16.675 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0070112910000000002 2024-03-19T11:30:16.676 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:16.676 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:16.676 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:16.676 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.676 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.676 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1088 4.6 4:6d9093b1:::smithi03829856-38:head [read 0~3142876,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:16.676 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:14.006622+0000", 2024-03-19T11:30:16.676 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 2.5933076220000002, 2024-03-19T11:30:16.676 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01172611, 2024-03-19T11:30:16.676 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:16.676 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:16.676 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:16.676 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:16.676 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:16.677 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1088 2024-03-19T11:30:16.677 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.677 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:16.677 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.677 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:16.677 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006622+0000", 2024-03-19T11:30:16.677 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:16.677 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.677 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.677 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:16.677 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006622+0000", 2024-03-19T11:30:16.677 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:16.677 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.677 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.678 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:16.678 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006623+0000", 2024-03-19T11:30:16.678 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4000000000000002e-07 2024-03-19T11:30:16.678 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.678 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.678 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:16.678 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006626+0000", 2024-03-19T11:30:16.678 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0340000000000001e-06 2024-03-19T11:30:16.678 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.678 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.678 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:16.678 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006632+0000", 2024-03-19T11:30:16.678 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3620000000000002e-06 2024-03-19T11:30:16.678 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.679 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.679 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:16.679 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006637+0000", 2024-03-19T11:30:16.679 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1390000000000003e-06 2024-03-19T11:30:16.679 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.679 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.679 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:16.679 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.015824+0000", 2024-03-19T11:30:16.679 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0091862899999999997 2024-03-19T11:30:16.679 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.679 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.679 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:16.679 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.015955+0000", 2024-03-19T11:30:16.679 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013139400000000001 2024-03-19T11:30:16.680 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.680 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.680 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:16.680 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.018349+0000", 2024-03-19T11:30:16.680 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002393451 2024-03-19T11:30:16.680 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:16.680 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:16.680 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:16.680 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.680 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.680 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1089 4.6 4:6d9093b1:::smithi03829856-38:head [sparse-read 0~1502675] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:16.680 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:14.006646+0000", 2024-03-19T11:30:16.680 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 2.5932841780000002, 2024-03-19T11:30:16.680 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012196624999999999, 2024-03-19T11:30:16.681 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:16.681 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:16.681 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:16.681 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:16.681 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:16.681 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1089 2024-03-19T11:30:16.681 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.681 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:16.681 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.681 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:16.681 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006646+0000", 2024-03-19T11:30:16.681 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:16.681 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.681 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.681 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:16.682 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006646+0000", 2024-03-19T11:30:16.682 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:16.682 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.682 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.682 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:16.682 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006646+0000", 2024-03-19T11:30:16.682 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.9999999999999998e-07 2024-03-19T11:30:16.682 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.682 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.682 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:16.682 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006649+0000", 2024-03-19T11:30:16.682 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5639999999999999e-06 2024-03-19T11:30:16.682 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.682 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.683 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:16.683 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006656+0000", 2024-03-19T11:30:16.683 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1810000000000004e-06 2024-03-19T11:30:16.683 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.683 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.683 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:16.683 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006661+0000", 2024-03-19T11:30:16.683 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4999999999999999e-06 2024-03-19T11:30:16.683 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.683 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.683 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:16.683 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.018416+0000", 2024-03-19T11:30:16.683 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011754647999999999 2024-03-19T11:30:16.683 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.683 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.684 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:16.684 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.018528+0000", 2024-03-19T11:30:16.684 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000111872 2024-03-19T11:30:16.684 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.684 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.684 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:16.684 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.018842+0000", 2024-03-19T11:30:16.684 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00031446000000000002 2024-03-19T11:30:16.684 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:16.684 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:16.684 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:16.684 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.684 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.684 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1090 4.6 4:6d9093b1:::smithi03829856-38:head [sparse-read 0~1502675] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:16.684 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:14.006669+0000", 2024-03-19T11:30:16.685 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 2.593261257, 2024-03-19T11:30:16.685 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012590499999999999, 2024-03-19T11:30:16.685 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:16.685 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:16.685 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:16.685 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:16.685 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:16.685 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1090 2024-03-19T11:30:16.685 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.685 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:16.685 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.685 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:16.685 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006669+0000", 2024-03-19T11:30:16.685 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:16.686 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.686 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.686 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:16.686 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006669+0000", 2024-03-19T11:30:16.686 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:16.686 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.686 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.686 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:16.686 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006669+0000", 2024-03-19T11:30:16.686 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1800000000000001e-07 2024-03-19T11:30:16.686 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.686 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.686 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:16.686 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006673+0000", 2024-03-19T11:30:16.686 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.9530000000000001e-06 2024-03-19T11:30:16.687 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.687 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.687 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:16.687 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006680+0000", 2024-03-19T11:30:16.687 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9410000000000001e-06 2024-03-19T11:30:16.687 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.687 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.687 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:16.687 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006685+0000", 2024-03-19T11:30:16.687 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.327e-06 2024-03-19T11:30:16.687 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.687 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.687 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:16.687 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.018877+0000", 2024-03-19T11:30:16.688 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012191936 2024-03-19T11:30:16.688 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.688 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.688 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:16.688 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.018972+0000", 2024-03-19T11:30:16.688 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4754999999999997e-05 2024-03-19T11:30:16.688 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:16.688 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:16.688 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:16.688 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.019259+0000", 2024-03-19T11:30:16.688 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00028717 2024-03-19T11:30:16.688 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:16.688 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:16.688 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:16.688 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:16.689 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:16.689 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:16.690 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-19T11:30:16.691 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:16.699+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:16.783 INFO:tasks.rados.rados.0.smithi038.stdout:724: done (6 left) 2024-03-19T11:30:16.783 INFO:tasks.rados.rados.0.smithi038.stdout:729: done (5 left) 2024-03-19T11:30:16.783 INFO:tasks.rados.rados.0.smithi038.stdout:730: done (4 left) 2024-03-19T11:30:16.784 INFO:tasks.rados.rados.0.smithi038.stdout:731: done (3 left) 2024-03-19T11:30:16.784 INFO:tasks.rados.rados.0.smithi038.stdout:733: done (2 left) 2024-03-19T11:30:16.784 INFO:tasks.rados.rados.0.smithi038.stdout:734: done (1 left) 2024-03-19T11:30:16.784 INFO:tasks.rados.rados.0.smithi038.stdout:735: done (0 left) 2024-03-19T11:30:16.784 INFO:tasks.rados.rados.0.smithi038.stdout:736: set_chunk oid 41 target oid 2024-03-19T11:30:16.786 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-03-19T11:30:16.803 INFO:tasks.rados.rados.0.smithi038.stdout:736: set_chunk oid 41 offset: 565248 length: 1159168 target oid 742a8355e2ffe0db2e5526e145a2863c365631538bec5a37652161a9f0233d7c offset: 0 snap -1 2024-03-19T11:30:16.803 INFO:tasks.rados.rados.0.smithi038.stdout:737: delete oid 27 current snap is 7 2024-03-19T11:30:16.804 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:16.812+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:16.807 INFO:tasks.rados.rados.0.smithi038.stdout:736: finishing set_chunk to oid 41 2024-03-19T11:30:16.808 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 41 v 48 (ObjNum 80 snap 6 seq_num 80) dirty exists 2024-03-19T11:30:16.812 INFO:tasks.rados.rados.0.smithi038.stdout:736: done (1 left) 2024-03-19T11:30:16.812 INFO:tasks.rados.rados.0.smithi038.stdout:737: done (0 left) 2024-03-19T11:30:16.812 INFO:tasks.rados.rados.0.smithi038.stdout:738: read oid 46 snap 6 2024-03-19T11:30:16.812 INFO:tasks.rados.rados.0.smithi038.stdout:738: expect (ObjNum 45 snap 0 seq_num 45) 2024-03-19T11:30:16.812 INFO:tasks.rados.rados.0.smithi038.stdout:739: read oid 17 snap -1 2024-03-19T11:30:16.813 INFO:tasks.rados.rados.0.smithi038.stdout:739: expect (ObjNum 16 snap 0 seq_num 16) 2024-03-19T11:30:16.813 INFO:tasks.rados.rados.0.smithi038.stdout:740: read oid 36 snap -1 2024-03-19T11:30:16.813 INFO:tasks.rados.rados.0.smithi038.stdout:740: expect (ObjNum 35 snap 0 seq_num 35) 2024-03-19T11:30:16.813 INFO:tasks.rados.rados.0.smithi038.stdout:741: read oid 49 snap 3 2024-03-19T11:30:16.813 INFO:tasks.rados.rados.0.smithi038.stdout:741: expect (ObjNum 48 snap 0 seq_num 48) 2024-03-19T11:30:16.813 INFO:tasks.rados.rados.0.smithi038.stdout:742: read oid 13 snap 6 2024-03-19T11:30:16.813 INFO:tasks.rados.rados.0.smithi038.stdout:742: expect (ObjNum 12 snap 0 seq_num 12) 2024-03-19T11:30:16.813 INFO:tasks.rados.rados.0.smithi038.stdout:743: read oid 29 snap -1 2024-03-19T11:30:16.813 INFO:tasks.rados.rados.0.smithi038.stdout:743: expect (ObjNum 28 snap 0 seq_num 28) 2024-03-19T11:30:16.813 INFO:tasks.rados.rados.0.smithi038.stdout:744: tier_promote oid 23 2024-03-19T11:30:16.813 INFO:tasks.rados.rados.0.smithi038.stdout:745: read oid 39 snap -1 2024-03-19T11:30:16.813 INFO:tasks.rados.rados.0.smithi038.stdout:745: expect (ObjNum 70 snap 4 seq_num 70) 2024-03-19T11:30:16.813 INFO:tasks.rados.rados.0.smithi038.stdout:746: write oid 43 current snap is 7 2024-03-19T11:30:16.813 INFO:tasks.rados.rados.0.smithi038.stdout:746: seq_num 83 ranges {594981=609197,1974123=745028,3279873=115108} 2024-03-19T11:30:16.823 INFO:tasks.rados.rados.0.smithi038.stdout:746: writing smithi03829856-43 from 594981 to 1204178 tid 1 2024-03-19T11:30:16.834 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:16.835 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:30:16.835 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:30:16.835 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:16.836 INFO:tasks.rados.rados.0.smithi038.stdout:746: writing smithi03829856-43 from 1974123 to 2719151 tid 2 2024-03-19T11:30:16.841 INFO:tasks.rados.rados.0.smithi038.stdout:746: writing smithi03829856-43 from 3279873 to 3394981 tid 3 2024-03-19T11:30:16.841 INFO:tasks.rados.rados.0.smithi038.stdout:747: read oid 35 snap -1 2024-03-19T11:30:16.841 INFO:tasks.rados.rados.0.smithi038.stdout:747: expect (ObjNum 34 snap 0 seq_num 34) 2024-03-19T11:30:16.841 INFO:tasks.rados.rados.0.smithi038.stdout:748: read oid 32 snap -1 2024-03-19T11:30:16.841 INFO:tasks.rados.rados.0.smithi038.stdout:748: expect (ObjNum 31 snap 0 seq_num 31) 2024-03-19T11:30:16.841 INFO:tasks.rados.rados.0.smithi038.stdout:749: write (excl) oid 2 current snap is 7 2024-03-19T11:30:16.841 INFO:tasks.rados.rados.0.smithi038.stdout:749: seq_num 84 ranges {611788=400799,1748958=488686,2907296=504492} 2024-03-19T11:30:16.848 INFO:tasks.rados.rados.0.smithi038.stdout:749: writing smithi03829856-2 from 611788 to 1012587 tid 1 2024-03-19T11:30:16.855 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-19T11:30:16.855 INFO:tasks.rados.rados.0.smithi038.stdout:749: writing smithi03829856-2 from 1748958 to 2237644 tid 2 2024-03-19T11:30:16.864 INFO:tasks.rados.rados.0.smithi038.stdout:749: writing smithi03829856-2 from 2907296 to 3411788 tid 3 2024-03-19T11:30:16.864 INFO:tasks.rados.rados.0.smithi038.stdout:750: read oid 22 snap -1 2024-03-19T11:30:16.865 INFO:tasks.rados.rados.0.smithi038.stdout:750: expect (ObjNum 21 snap 0 seq_num 21) 2024-03-19T11:30:16.865 INFO:tasks.rados.rados.0.smithi038.stdout:751: set_chunk oid 26 target oid 2024-03-19T11:30:16.865 INFO:tasks.rados.rados.0.smithi038.stdout:751: set_chunk oid 26 offset: 0 length: 4000000 target oid smithi03829856- offset: 0 snap -1 2024-03-19T11:30:16.865 INFO:tasks.rados.rados.0.smithi038.stdout:752: read oid 33 snap -1 2024-03-19T11:30:16.865 INFO:tasks.rados.rados.0.smithi038.stdout:752: expect (ObjNum 32 snap 0 seq_num 32) 2024-03-19T11:30:16.865 INFO:tasks.rados.rados.0.smithi038.stdout:753: read oid 8 snap -1 2024-03-19T11:30:16.865 INFO:tasks.rados.rados.0.smithi038.stdout:753: expect (ObjNum 60 snap 3 seq_num 60) 2024-03-19T11:30:16.865 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:16.865 INFO:tasks.rados.rados.0.smithi038.stdout:738: expect (ObjNum 45 snap 0 seq_num 45) 2024-03-19T11:30:16.891 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:16.900+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:16.952 INFO:tasks.rados.rados.0.smithi038.stdout:740: expect (ObjNum 35 snap 0 seq_num 35) 2024-03-19T11:30:16.976 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:16.976 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:30:16.976 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:30:16.976 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:30:16.976 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:16.989 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-19T11:30:17.030 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:17.040+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:17.037 INFO:tasks.rados.rados.0.smithi038.stdout:741: expect (ObjNum 48 snap 0 seq_num 48) 2024-03-19T11:30:17.093 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:17.100+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:17.127 INFO:tasks.rados.rados.0.smithi038.stdout:739: expect (ObjNum 16 snap 0 seq_num 16) 2024-03-19T11:30:17.147 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:17.147 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:30:17.148 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:30:17.148 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:30:17.148 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.148 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:6 4.6 e27/21 4:626acd4d:::smithi03829856-2:head v 27'2, mlcod=0'0)", 2024-03-19T11:30:17.148 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.696903+0000", 2024-03-19T11:30:17.148 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 12.461240318, 2024-03-19T11:30:17.148 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0071029129999999998, 2024-03-19T11:30:17.148 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:17.148 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:17.148 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:17.148 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.148 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:17.148 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696903+0000", 2024-03-19T11:30:17.148 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.149 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.149 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.149 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:17.149 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696903+0000", 2024-03-19T11:30:17.149 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.149 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.149 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.149 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:17.149 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696904+0000", 2024-03-19T11:30:17.149 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4270000000000001e-06 2024-03-19T11:30:17.149 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.149 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.149 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:17.149 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.697064+0000", 2024-03-19T11:30:17.149 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015956100000000001 2024-03-19T11:30:17.150 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.150 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.150 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:17.150 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.697082+0000", 2024-03-19T11:30:17.150 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8868999999999998e-05 2024-03-19T11:30:17.150 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.150 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.150 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:17.150 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.697096+0000", 2024-03-19T11:30:17.150 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3155e-05 2024-03-19T11:30:17.150 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.150 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.150 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:17.150 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.697169+0000", 2024-03-19T11:30:17.151 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3226999999999995e-05 2024-03-19T11:30:17.151 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.151 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.151 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:17.151 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.697193+0000", 2024-03-19T11:30:17.151 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4153000000000001e-05 2024-03-19T11:30:17.151 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.151 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.151 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:17.151 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.703936+0000", 2024-03-19T11:30:17.151 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0067430570000000002 2024-03-19T11:30:17.151 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.151 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.151 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:17.152 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.704006+0000", 2024-03-19T11:30:17.152 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9463999999999998e-05 2024-03-19T11:30:17.152 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.152 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:17.152 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.152 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.152 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.152 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:13 4.4 4:21eac9b2:::smithi03829856-4:head [write 450421~625747,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:30:17.152 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.720170+0000", 2024-03-19T11:30:17.152 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 12.437973004, 2024-03-19T11:30:17.152 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0090522840000000007, 2024-03-19T11:30:17.152 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:17.152 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:17.152 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:17.152 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:17.153 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:17.153 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 13 2024-03-19T11:30:17.153 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.153 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:17.153 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.153 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:17.153 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.720170+0000", 2024-03-19T11:30:17.153 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.153 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.153 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.154 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:17.154 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.720170+0000", 2024-03-19T11:30:17.154 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.154 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.154 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.154 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:17.154 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.720172+0000", 2024-03-19T11:30:17.154 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9709999999999998e-06 2024-03-19T11:30:17.154 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.154 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.155 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:17.155 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.720601+0000", 2024-03-19T11:30:17.155 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00042897999999999998 2024-03-19T11:30:17.155 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.155 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.155 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:17.155 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.720615+0000", 2024-03-19T11:30:17.155 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4617000000000001e-05 2024-03-19T11:30:17.155 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.155 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.155 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:17.156 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.720627+0000", 2024-03-19T11:30:17.156 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1513e-05 2024-03-19T11:30:17.156 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.156 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.156 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:17.156 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.720705+0000", 2024-03-19T11:30:17.156 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7880000000000007e-05 2024-03-19T11:30:17.156 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.156 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.156 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-03-19T11:30:17.156 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.720709+0000", 2024-03-19T11:30:17.157 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5549999999999996e-06 2024-03-19T11:30:17.157 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.157 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.157 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:17.157 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.722748+0000", 2024-03-19T11:30:17.157 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0020388630000000001 2024-03-19T11:30:17.157 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.157 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.157 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:17.157 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.722891+0000", 2024-03-19T11:30:17.157 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014286 2024-03-19T11:30:17.157 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.157 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.157 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 2", 2024-03-19T11:30:17.157 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723057+0000", 2024-03-19T11:30:17.158 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000166286 2024-03-19T11:30:17.158 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.158 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.158 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:17.158 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.725297+0000", 2024-03-19T11:30:17.158 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022397770000000001 2024-03-19T11:30:17.158 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.158 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.158 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:17.158 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.729110+0000", 2024-03-19T11:30:17.158 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038130109999999998 2024-03-19T11:30:17.158 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.158 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.158 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:17.158 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.729169+0000", 2024-03-19T11:30:17.159 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8616000000000003e-05 2024-03-19T11:30:17.159 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.159 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.159 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:17.159 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.729222+0000", 2024-03-19T11:30:17.159 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3355e-05 2024-03-19T11:30:17.159 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.159 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:17.159 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.159 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.159 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.159 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:844 4.7 4:e41d47fb:::smithi03829856-14:head [write 574806~778913] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:17.159 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:08.749074+0000", 2024-03-19T11:30:17.159 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.4090692229999995, 2024-03-19T11:30:17.159 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013996124, 2024-03-19T11:30:17.160 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:17.160 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:17.160 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:17.160 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:17.160 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:17.160 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 844 2024-03-19T11:30:17.160 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.160 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:17.160 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.160 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:17.160 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.749074+0000", 2024-03-19T11:30:17.160 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.160 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.160 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.160 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:17.161 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.749074+0000", 2024-03-19T11:30:17.161 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.161 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.161 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.161 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:17.161 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.749076+0000", 2024-03-19T11:30:17.161 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.588e-06 2024-03-19T11:30:17.161 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.161 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.161 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:17.161 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.749714+0000", 2024-03-19T11:30:17.161 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00063805999999999997 2024-03-19T11:30:17.161 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.161 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.161 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:17.162 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.749750+0000", 2024-03-19T11:30:17.162 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5709e-05 2024-03-19T11:30:17.162 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.162 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.162 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:17.162 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.749767+0000", 2024-03-19T11:30:17.162 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7127000000000001e-05 2024-03-19T11:30:17.162 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.162 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.162 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:17.162 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.749860+0000", 2024-03-19T11:30:17.162 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2875999999999996e-05 2024-03-19T11:30:17.162 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.162 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.163 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:17.163 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.749943+0000", 2024-03-19T11:30:17.163 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2915000000000002e-05 2024-03-19T11:30:17.163 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.163 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.163 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:17.163 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.750083+0000", 2024-03-19T11:30:17.163 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013953300000000001 2024-03-19T11:30:17.164 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.164 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.164 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:17.164 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.753673+0000", 2024-03-19T11:30:17.164 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0035904470000000001 2024-03-19T11:30:17.164 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.164 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.164 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:17.164 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.762965+0000", 2024-03-19T11:30:17.164 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0092923390000000002 2024-03-19T11:30:17.164 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.164 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.164 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:17.164 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.763001+0000", 2024-03-19T11:30:17.164 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6087000000000002e-05 2024-03-19T11:30:17.165 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.165 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.165 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:17.165 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.763070+0000", 2024-03-19T11:30:17.165 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8442999999999995e-05 2024-03-19T11:30:17.165 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.165 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:17.165 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.165 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.165 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.165 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:845 4.7 4:e41d47fb:::smithi03829856-14:head [write 2061132~557342] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:17.165 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:08.756359+0000", 2024-03-19T11:30:17.165 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.4017842649999999, 2024-03-19T11:30:17.165 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074752910000000002, 2024-03-19T11:30:17.166 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:17.166 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:17.166 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:17.166 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:17.166 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:17.166 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 845 2024-03-19T11:30:17.166 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.166 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:17.166 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.166 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:17.166 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756359+0000", 2024-03-19T11:30:17.166 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.166 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.166 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.166 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:17.167 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756359+0000", 2024-03-19T11:30:17.167 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.167 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.167 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.167 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:17.167 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756361+0000", 2024-03-19T11:30:17.167 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5280000000000002e-06 2024-03-19T11:30:17.167 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.167 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.168 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:17.168 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756535+0000", 2024-03-19T11:30:17.168 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000173697 2024-03-19T11:30:17.168 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.168 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.168 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:17.168 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756552+0000", 2024-03-19T11:30:17.168 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7397e-05 2024-03-19T11:30:17.168 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.168 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.168 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:17.168 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756561+0000", 2024-03-19T11:30:17.169 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8769999999999999e-06 2024-03-19T11:30:17.169 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.169 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.169 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:17.169 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756614+0000", 2024-03-19T11:30:17.169 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2425e-05 2024-03-19T11:30:17.169 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.169 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.169 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:17.169 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756693+0000", 2024-03-19T11:30:17.169 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9338999999999994e-05 2024-03-19T11:30:17.169 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.169 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.169 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:17.170 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756801+0000", 2024-03-19T11:30:17.170 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000108175 2024-03-19T11:30:17.170 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.170 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.170 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:17.170 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.758827+0000", 2024-03-19T11:30:17.170 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0020263379999999999 2024-03-19T11:30:17.170 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.170 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.170 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:17.170 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.763778+0000", 2024-03-19T11:30:17.171 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0049505740000000001 2024-03-19T11:30:17.171 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.171 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.171 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:17.171 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.763807+0000", 2024-03-19T11:30:17.171 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8702999999999999e-05 2024-03-19T11:30:17.171 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.171 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.171 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:17.171 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.763834+0000", 2024-03-19T11:30:17.171 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.7237999999999999e-05 2024-03-19T11:30:17.171 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.171 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:17.171 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.171 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.172 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.172 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:846 4.7 4:e41d47fb:::smithi03829856-14:head [setxattr _header (56) in=7b,truncate 2974806] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:17.172 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:08.756597+0000", 2024-03-19T11:30:17.172 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.4015454550000008, 2024-03-19T11:30:17.172 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0092954909999999995, 2024-03-19T11:30:17.172 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:17.172 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:17.172 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:17.172 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:17.172 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:17.172 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 846 2024-03-19T11:30:17.172 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.172 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:17.172 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.172 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:17.173 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756597+0000", 2024-03-19T11:30:17.173 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.173 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.173 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.173 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:17.173 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756597+0000", 2024-03-19T11:30:17.173 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.173 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.173 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.173 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:17.173 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756598+0000", 2024-03-19T11:30:17.173 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4600000000000004e-07 2024-03-19T11:30:17.174 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.174 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.174 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:17.174 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756604+0000", 2024-03-19T11:30:17.174 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5010000000000004e-06 2024-03-19T11:30:17.174 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.174 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.174 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:17.174 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756613+0000", 2024-03-19T11:30:17.174 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3999999999999998e-06 2024-03-19T11:30:17.174 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.175 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.175 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:17.175 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756620+0000", 2024-03-19T11:30:17.175 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.968e-06 2024-03-19T11:30:17.175 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.175 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.175 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:17.175 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.758321+0000", 2024-03-19T11:30:17.175 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001701337 2024-03-19T11:30:17.175 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.175 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.175 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:17.175 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.758414+0000", 2024-03-19T11:30:17.175 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2375999999999998e-05 2024-03-19T11:30:17.175 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.176 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.176 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:17.176 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.758553+0000", 2024-03-19T11:30:17.176 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013910300000000001 2024-03-19T11:30:17.176 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.176 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.176 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:17.176 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.758987+0000", 2024-03-19T11:30:17.176 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00043375100000000002 2024-03-19T11:30:17.176 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.176 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.176 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:17.176 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.765803+0000", 2024-03-19T11:30:17.176 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0068161089999999999 2024-03-19T11:30:17.177 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.177 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.177 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:17.177 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.765846+0000", 2024-03-19T11:30:17.177 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3226000000000003e-05 2024-03-19T11:30:17.177 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.177 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.177 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:17.177 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.765893+0000", 2024-03-19T11:30:17.177 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6974000000000001e-05 2024-03-19T11:30:17.177 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.177 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:17.177 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.177 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.177 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.178 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:847 4.7 4:e41d47fb:::smithi03829856-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:17.178 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:08.756635+0000", 2024-03-19T11:30:17.178 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.4015075929999998, 2024-03-19T11:30:17.178 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093966859999999996, 2024-03-19T11:30:17.178 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:17.178 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:17.178 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:17.178 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:17.178 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:17.178 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 847 2024-03-19T11:30:17.178 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.178 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:17.178 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.178 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:17.178 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756635+0000", 2024-03-19T11:30:17.179 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.179 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.179 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.179 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:17.179 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756635+0000", 2024-03-19T11:30:17.179 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.179 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.179 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.179 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:17.179 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756636+0000", 2024-03-19T11:30:17.179 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9500000000000002e-07 2024-03-19T11:30:17.179 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.179 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.179 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:17.179 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756640+0000", 2024-03-19T11:30:17.180 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7349999999999998e-06 2024-03-19T11:30:17.180 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.180 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.180 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:17.180 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756647+0000", 2024-03-19T11:30:17.180 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7130000000000005e-06 2024-03-19T11:30:17.180 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.180 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.180 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:17.180 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756653+0000", 2024-03-19T11:30:17.180 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3229999999999997e-06 2024-03-19T11:30:17.180 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.180 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.180 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:17.180 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.758756+0000", 2024-03-19T11:30:17.181 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0021037170000000002 2024-03-19T11:30:17.181 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.181 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.181 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-03-19T11:30:17.181 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.758814+0000", 2024-03-19T11:30:17.181 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7253999999999998e-05 2024-03-19T11:30:17.181 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.181 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.181 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:17.181 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.765926+0000", 2024-03-19T11:30:17.181 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0071118350000000004 2024-03-19T11:30:17.181 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.181 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.181 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:17.181 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.765970+0000", 2024-03-19T11:30:17.182 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4761000000000001e-05 2024-03-19T11:30:17.182 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.182 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.182 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:17.182 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.766032+0000", 2024-03-19T11:30:17.182 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1753000000000003e-05 2024-03-19T11:30:17.182 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.182 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:17.182 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.182 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.182 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.182 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:998 4.4 4:27909373:::smithi03829856-46:head [rollback 2] snapc 5=[5,2] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-03-19T11:30:17.182 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.368196+0000", 2024-03-19T11:30:17.182 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.7899467290000004, 2024-03-19T11:30:17.183 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0079626200000000001, 2024-03-19T11:30:17.183 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:17.183 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:17.183 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:17.183 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:17.183 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:17.183 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 998 2024-03-19T11:30:17.183 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.183 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:17.183 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.183 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:17.183 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368196+0000", 2024-03-19T11:30:17.183 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.183 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.183 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.184 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:17.184 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368196+0000", 2024-03-19T11:30:17.184 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.184 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.184 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.184 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:17.184 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368197+0000", 2024-03-19T11:30:17.184 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6899999999999998e-07 2024-03-19T11:30:17.184 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.185 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.185 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:17.185 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368200+0000", 2024-03-19T11:30:17.185 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1389999999999999e-06 2024-03-19T11:30:17.185 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.185 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.185 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:17.185 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368207+0000", 2024-03-19T11:30:17.185 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9870000000000002e-06 2024-03-19T11:30:17.185 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.185 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.185 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:17.185 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368212+0000", 2024-03-19T11:30:17.185 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3820000000000003e-06 2024-03-19T11:30:17.185 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.186 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.186 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:17.186 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.369920+0000", 2024-03-19T11:30:17.186 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017078 2024-03-19T11:30:17.186 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.186 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.186 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:17.186 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370087+0000", 2024-03-19T11:30:17.186 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016742800000000001 2024-03-19T11:30:17.186 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.186 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.186 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 2", 2024-03-19T11:30:17.186 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370522+0000", 2024-03-19T11:30:17.186 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00043494300000000003 2024-03-19T11:30:17.187 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.187 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.187 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:17.187 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.375478+0000", 2024-03-19T11:30:17.187 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0049562240000000004 2024-03-19T11:30:17.187 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.187 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.187 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:17.187 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.375918+0000", 2024-03-19T11:30:17.187 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00043914699999999997 2024-03-19T11:30:17.187 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.187 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.187 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:17.187 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.376083+0000", 2024-03-19T11:30:17.187 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0001649 2024-03-19T11:30:17.188 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.188 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.188 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:17.188 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.376159+0000", 2024-03-19T11:30:17.188 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6301e-05 2024-03-19T11:30:17.188 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.188 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:17.188 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.188 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.188 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.188 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:999 4.4 4:27909373:::smithi03829856-46:head [append 0~0] snapc 5=[5,2] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-03-19T11:30:17.188 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.368233+0000", 2024-03-19T11:30:17.188 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.789909754, 2024-03-19T11:30:17.188 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0081209219999999992, 2024-03-19T11:30:17.188 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:17.188 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:17.189 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:17.189 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:17.189 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:17.189 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 999 2024-03-19T11:30:17.189 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.189 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:17.189 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.189 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:17.189 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368233+0000", 2024-03-19T11:30:17.189 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.189 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.189 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.189 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:17.189 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368233+0000", 2024-03-19T11:30:17.189 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.190 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.190 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.190 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:17.190 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368234+0000", 2024-03-19T11:30:17.190 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5400000000000002e-07 2024-03-19T11:30:17.190 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.190 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.190 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:17.190 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368236+0000", 2024-03-19T11:30:17.190 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.7010000000000001e-06 2024-03-19T11:30:17.190 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.190 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.190 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:17.191 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368244+0000", 2024-03-19T11:30:17.191 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.289e-06 2024-03-19T11:30:17.191 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.191 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.191 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:17.191 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368248+0000", 2024-03-19T11:30:17.191 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9429999999999999e-06 2024-03-19T11:30:17.191 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.191 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.191 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:17.191 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372084+0000", 2024-03-19T11:30:17.191 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038354769999999999 2024-03-19T11:30:17.191 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.191 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.192 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:17.192 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372185+0000", 2024-03-19T11:30:17.192 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010088400000000001 2024-03-19T11:30:17.192 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.192 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.192 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 2", 2024-03-19T11:30:17.192 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372376+0000", 2024-03-19T11:30:17.192 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000191545 2024-03-19T11:30:17.192 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.192 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.192 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:17.192 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.376187+0000", 2024-03-19T11:30:17.193 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038108159999999999 2024-03-19T11:30:17.193 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.193 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.193 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:17.193 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.376243+0000", 2024-03-19T11:30:17.193 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6032000000000002e-05 2024-03-19T11:30:17.193 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.193 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.193 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:17.193 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.376304+0000", 2024-03-19T11:30:17.193 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0834000000000001e-05 2024-03-19T11:30:17.194 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.194 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.194 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:17.194 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.376354+0000", 2024-03-19T11:30:17.194 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0046999999999997e-05 2024-03-19T11:30:17.194 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.194 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:17.194 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.194 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.195 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.195 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1003 4.4 4:22709789:::smithi03829856-41:head [write 793297~789856,stat] snapc 5=[5,2] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-03-19T11:30:17.195 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.377573+0000", 2024-03-19T11:30:17.195 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.7805696280000003, 2024-03-19T11:30:17.195 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0060098649999999997, 2024-03-19T11:30:17.195 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:17.195 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:17.195 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:17.195 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:17.195 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:17.195 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1003 2024-03-19T11:30:17.195 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.195 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:17.195 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.195 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:17.196 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.377573+0000", 2024-03-19T11:30:17.196 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.196 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.196 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.196 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:17.196 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.377573+0000", 2024-03-19T11:30:17.196 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.196 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.196 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.196 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:17.196 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.377574+0000", 2024-03-19T11:30:17.196 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.192e-06 2024-03-19T11:30:17.196 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.196 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.196 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:17.197 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.377951+0000", 2024-03-19T11:30:17.197 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037638 2024-03-19T11:30:17.197 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.197 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.197 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:17.197 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.377965+0000", 2024-03-19T11:30:17.197 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4177e-05 2024-03-19T11:30:17.197 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.197 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.197 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:17.197 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.377972+0000", 2024-03-19T11:30:17.197 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5889999999999999e-06 2024-03-19T11:30:17.197 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.197 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.198 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:17.198 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.378047+0000", 2024-03-19T11:30:17.198 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5810999999999996e-05 2024-03-19T11:30:17.198 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.198 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.198 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:17.198 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.378181+0000", 2024-03-19T11:30:17.198 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000133485 2024-03-19T11:30:17.198 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.198 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.198 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 2", 2024-03-19T11:30:17.198 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.378419+0000", 2024-03-19T11:30:17.198 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023761100000000001 2024-03-19T11:30:17.198 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.198 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.199 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:17.199 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.382891+0000", 2024-03-19T11:30:17.199 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0044720180000000003 2024-03-19T11:30:17.199 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.199 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.199 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:17.199 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.383452+0000", 2024-03-19T11:30:17.199 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00056113599999999995 2024-03-19T11:30:17.199 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.199 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.199 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:17.200 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.383513+0000", 2024-03-19T11:30:17.200 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0850000000000002e-05 2024-03-19T11:30:17.200 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.200 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.200 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:17.200 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.383583+0000", 2024-03-19T11:30:17.200 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.0616000000000003e-05 2024-03-19T11:30:17.200 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.200 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:17.200 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.200 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.200 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.200 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:53 3.0 3:3dcd2456:::039914993b6f4082c204b24ae0ff925ebffccc22519067b64afa9eff107cf533:head [read 0~243792 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:17.200 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.761757+0000", 2024-03-19T11:30:17.201 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.3963856459999997, 2024-03-19T11:30:17.201 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0076641110000000004, 2024-03-19T11:30:17.201 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:17.201 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:17.201 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:17.201 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.201 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:17.201 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761757+0000", 2024-03-19T11:30:17.201 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.201 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.201 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.201 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:17.201 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761757+0000", 2024-03-19T11:30:17.201 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.201 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.201 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.202 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:17.202 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761759+0000", 2024-03-19T11:30:17.202 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0980000000000001e-06 2024-03-19T11:30:17.202 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.202 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.202 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:17.202 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761771+0000", 2024-03-19T11:30:17.202 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1625e-05 2024-03-19T11:30:17.202 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.202 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.202 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:17.203 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761793+0000", 2024-03-19T11:30:17.203 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1940999999999999e-05 2024-03-19T11:30:17.203 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.203 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.203 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:17.203 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761803+0000", 2024-03-19T11:30:17.203 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0234999999999999e-05 2024-03-19T11:30:17.203 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.203 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.203 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:17.203 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761892+0000", 2024-03-19T11:30:17.204 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8974999999999994e-05 2024-03-19T11:30:17.204 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.204 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.204 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-03-19T11:30:17.204 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761899+0000", 2024-03-19T11:30:17.204 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1709999999999996e-06 2024-03-19T11:30:17.204 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.204 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.204 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:17.204 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.768690+0000", 2024-03-19T11:30:17.204 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.006790941 2024-03-19T11:30:17.204 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.204 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.206 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:17.206 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.768790+0000", 2024-03-19T11:30:17.206 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9569999999999997e-05 2024-03-19T11:30:17.206 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.206 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.206 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:17.206 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.769421+0000", 2024-03-19T11:30:17.206 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00063155500000000001 2024-03-19T11:30:17.206 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.206 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:17.206 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.206 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.207 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.207 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:54 3.0 3:0f3835fc:::79445f5816960fdb5fb7d7699594b1dfc30a0a44639e804aee69d6cb400c02a5:head [read 0~213457 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:17.207 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.761831+0000", 2024-03-19T11:30:17.207 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.3963121269999998, 2024-03-19T11:30:17.207 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0086999490000000002, 2024-03-19T11:30:17.207 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:17.207 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:17.207 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:17.207 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.207 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:17.207 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761831+0000", 2024-03-19T11:30:17.207 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.208 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.208 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.208 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:17.208 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761831+0000", 2024-03-19T11:30:17.208 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.208 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.208 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.208 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:17.208 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761831+0000", 2024-03-19T11:30:17.208 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8699999999999995e-07 2024-03-19T11:30:17.208 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.208 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.208 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:17.208 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761835+0000", 2024-03-19T11:30:17.208 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5039999999999998e-06 2024-03-19T11:30:17.209 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.209 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.209 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:17.209 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761847+0000", 2024-03-19T11:30:17.209 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.242e-05 2024-03-19T11:30:17.209 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.209 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.209 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:17.209 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761855+0000", 2024-03-19T11:30:17.209 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2459999999999997e-06 2024-03-19T11:30:17.209 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.209 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.209 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:17.209 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762004+0000", 2024-03-19T11:30:17.210 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014920499999999999 2024-03-19T11:30:17.210 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.210 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.210 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:17.210 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762009+0000", 2024-03-19T11:30:17.210 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7350000000000001e-06 2024-03-19T11:30:17.210 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.210 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.210 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:17.210 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.769460+0000", 2024-03-19T11:30:17.210 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074506529999999998 2024-03-19T11:30:17.211 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.211 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.211 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:17.211 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.769564+0000", 2024-03-19T11:30:17.211 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000103608 2024-03-19T11:30:17.211 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.211 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.211 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:17.211 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.770531+0000", 2024-03-19T11:30:17.211 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00096699100000000005 2024-03-19T11:30:17.211 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.211 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:17.211 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.211 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.212 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.212 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1054 4.4 4:266590a5:::smithi03829856-16:6 [sparse-read 0~1716701,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:17.212 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.768596+0000", 2024-03-19T11:30:17.212 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.38954676, 2024-03-19T11:30:17.212 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0066594410000000003, 2024-03-19T11:30:17.212 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:17.212 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:17.212 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:17.212 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:17.212 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:17.212 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1054 2024-03-19T11:30:17.212 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.212 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:17.212 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.212 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:17.213 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768596+0000", 2024-03-19T11:30:17.213 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.213 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.213 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.213 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:17.213 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768596+0000", 2024-03-19T11:30:17.213 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.213 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.213 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.213 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:17.213 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768599+0000", 2024-03-19T11:30:17.213 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.118e-06 2024-03-19T11:30:17.213 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.213 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.213 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:17.214 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768616+0000", 2024-03-19T11:30:17.214 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6222000000000002e-05 2024-03-19T11:30:17.214 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.214 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.214 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:17.214 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768648+0000", 2024-03-19T11:30:17.214 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2027999999999999e-05 2024-03-19T11:30:17.214 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.214 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.214 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:17.214 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768670+0000", 2024-03-19T11:30:17.214 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2864999999999999e-05 2024-03-19T11:30:17.214 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.214 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.215 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:17.215 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768831+0000", 2024-03-19T11:30:17.215 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016088300000000001 2024-03-19T11:30:17.215 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.215 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.215 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-03-19T11:30:17.215 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768838+0000", 2024-03-19T11:30:17.215 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1019999999999999e-06 2024-03-19T11:30:17.215 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.215 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.215 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:17.216 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.774813+0000", 2024-03-19T11:30:17.216 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0059743909999999999 2024-03-19T11:30:17.216 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.216 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.216 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:17.216 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.774902+0000", 2024-03-19T11:30:17.216 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9479000000000002e-05 2024-03-19T11:30:17.216 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.216 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.216 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:17.216 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.775256+0000", 2024-03-19T11:30:17.216 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00035335299999999999 2024-03-19T11:30:17.216 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.217 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:17.217 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.217 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.217 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.217 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1055 4.4 4:266590a5:::smithi03829856-16:6 [read 0~3092455,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:17.217 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.768707+0000", 2024-03-19T11:30:17.217 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.3894361370000001, 2024-03-19T11:30:17.217 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0088150090000000004, 2024-03-19T11:30:17.217 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:17.217 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:17.217 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:17.217 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:17.218 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:17.218 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1055 2024-03-19T11:30:17.218 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.218 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:17.218 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.218 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:17.218 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768707+0000", 2024-03-19T11:30:17.218 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.218 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.218 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.218 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:17.218 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768707+0000", 2024-03-19T11:30:17.218 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.219 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.219 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.219 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:17.219 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768708+0000", 2024-03-19T11:30:17.219 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2200000000000003e-07 2024-03-19T11:30:17.219 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.219 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.219 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:17.219 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768716+0000", 2024-03-19T11:30:17.219 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4810000000000006e-06 2024-03-19T11:30:17.219 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.220 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.220 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:17.220 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768728+0000", 2024-03-19T11:30:17.220 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2167999999999999e-05 2024-03-19T11:30:17.220 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.220 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.220 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:17.220 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768737+0000", 2024-03-19T11:30:17.220 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8419999999999994e-06 2024-03-19T11:30:17.220 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.220 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.220 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:17.220 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768895+0000", 2024-03-19T11:30:17.220 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000157455 2024-03-19T11:30:17.221 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.221 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.221 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:17.221 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768897+0000", 2024-03-19T11:30:17.221 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.7580000000000001e-06 2024-03-19T11:30:17.221 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.221 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.221 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:17.221 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.775284+0000", 2024-03-19T11:30:17.221 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0063862069999999996 2024-03-19T11:30:17.221 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.221 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.221 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:17.221 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.775362+0000", 2024-03-19T11:30:17.221 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8604999999999997e-05 2024-03-19T11:30:17.222 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.222 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.222 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:17.222 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.777522+0000", 2024-03-19T11:30:17.222 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0021596710000000002 2024-03-19T11:30:17.222 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.222 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:17.222 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.222 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.222 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.222 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1056 4.4 4:266590a5:::smithi03829856-16:6 [read 0~3092455,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:17.222 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.768750+0000", 2024-03-19T11:30:17.222 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.3893933330000001, 2024-03-19T11:30:17.222 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010087227000000001, 2024-03-19T11:30:17.223 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:17.223 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:17.223 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:17.223 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:17.223 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:17.223 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1056 2024-03-19T11:30:17.223 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.223 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:17.223 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.223 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:17.223 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768750+0000", 2024-03-19T11:30:17.224 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.224 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.224 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.224 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:17.224 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768750+0000", 2024-03-19T11:30:17.224 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.224 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.224 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.224 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:17.224 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768750+0000", 2024-03-19T11:30:17.224 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6199999999999997e-07 2024-03-19T11:30:17.224 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.224 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.224 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:17.224 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768754+0000", 2024-03-19T11:30:17.225 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.9920000000000001e-06 2024-03-19T11:30:17.225 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.225 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.225 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:17.225 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768764+0000", 2024-03-19T11:30:17.225 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0052e-05 2024-03-19T11:30:17.225 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.225 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.225 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:17.225 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768773+0000", 2024-03-19T11:30:17.225 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7309999999999999e-06 2024-03-19T11:30:17.225 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.225 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.225 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:17.225 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768982+0000", 2024-03-19T11:30:17.226 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000208653 2024-03-19T11:30:17.226 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.226 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.226 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:17.226 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768990+0000", 2024-03-19T11:30:17.226 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3000000000000002e-06 2024-03-19T11:30:17.226 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.226 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.226 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:17.227 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.777544+0000", 2024-03-19T11:30:17.227 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.008554374 2024-03-19T11:30:17.227 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.227 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.227 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:17.227 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.777599+0000", 2024-03-19T11:30:17.227 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4880999999999999e-05 2024-03-19T11:30:17.227 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.227 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.227 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:17.228 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.778837+0000", 2024-03-19T11:30:17.228 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0012375820000000001 2024-03-19T11:30:17.228 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.228 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:17.228 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.228 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.228 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.228 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1063 4.4 4:214026b3:::smithi03829856-47:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:17.228 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.784904+0000", 2024-03-19T11:30:17.228 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.3732388919999998, 2024-03-19T11:30:17.228 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024425912000000001, 2024-03-19T11:30:17.228 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:17.228 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:17.229 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:17.229 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:17.229 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:17.229 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1063 2024-03-19T11:30:17.229 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.229 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:17.229 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.229 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:17.229 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784904+0000", 2024-03-19T11:30:17.229 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.229 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.229 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.229 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:17.230 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784904+0000", 2024-03-19T11:30:17.230 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.230 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.230 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.230 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:17.230 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784906+0000", 2024-03-19T11:30:17.230 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1409999999999999e-06 2024-03-19T11:30:17.230 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.230 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.230 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:17.230 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784915+0000", 2024-03-19T11:30:17.230 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5459999999999992e-06 2024-03-19T11:30:17.230 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.230 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.231 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:17.231 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784935+0000", 2024-03-19T11:30:17.231 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0310999999999999e-05 2024-03-19T11:30:17.231 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.231 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.231 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:17.231 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784946+0000", 2024-03-19T11:30:17.231 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0675e-05 2024-03-19T11:30:17.231 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.231 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.231 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:17.231 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.785032+0000", 2024-03-19T11:30:17.231 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6049000000000002e-05 2024-03-19T11:30:17.231 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.231 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.232 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:17.232 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.785136+0000", 2024-03-19T11:30:17.232 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000104209 2024-03-19T11:30:17.232 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.232 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.232 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:17.232 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.809330+0000", 2024-03-19T11:30:17.232 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024193981 2024-03-19T11:30:17.232 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.232 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:17.232 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.232 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.232 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.232 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1077 4.7 4:e41d47fb:::smithi03829856-14:head [write 510946~632952,stat] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:17.232 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.993080+0000", 2024-03-19T11:30:17.233 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.1650629710000002, 2024-03-19T11:30:17.233 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0095463609999999997, 2024-03-19T11:30:17.233 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:17.233 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:17.233 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:17.233 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:17.233 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:17.233 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1077 2024-03-19T11:30:17.233 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.233 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:17.233 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.233 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:17.233 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.993080+0000", 2024-03-19T11:30:17.233 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.233 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.234 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.234 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:17.234 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.993080+0000", 2024-03-19T11:30:17.234 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.234 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.234 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.234 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:17.234 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.993082+0000", 2024-03-19T11:30:17.234 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3199999999999998e-06 2024-03-19T11:30:17.234 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.235 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.235 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:17.235 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.993356+0000", 2024-03-19T11:30:17.235 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00027414699999999998 2024-03-19T11:30:17.235 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.235 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.235 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:17.235 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.993377+0000", 2024-03-19T11:30:17.235 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1015000000000001e-05 2024-03-19T11:30:17.235 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.235 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.235 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:17.235 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.993387+0000", 2024-03-19T11:30:17.235 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.5200000000000003e-06 2024-03-19T11:30:17.236 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.236 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.236 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:17.236 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.993468+0000", 2024-03-19T11:30:17.236 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1145999999999999e-05 2024-03-19T11:30:17.236 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.236 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.236 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:17.236 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.993595+0000", 2024-03-19T11:30:17.236 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012649199999999999 2024-03-19T11:30:17.236 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.236 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.236 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:17.236 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.993826+0000", 2024-03-19T11:30:17.237 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000231616 2024-03-19T11:30:17.237 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.237 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.237 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:17.237 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.997639+0000", 2024-03-19T11:30:17.237 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038124830000000002 2024-03-19T11:30:17.237 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.237 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.237 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:17.237 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.002505+0000", 2024-03-19T11:30:17.237 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0048662339999999997 2024-03-19T11:30:17.237 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.237 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.237 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:17.237 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.002569+0000", 2024-03-19T11:30:17.238 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3629999999999999e-05 2024-03-19T11:30:17.238 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.238 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.238 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:17.238 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.002626+0000", 2024-03-19T11:30:17.238 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7757999999999998e-05 2024-03-19T11:30:17.238 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.238 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:17.238 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.238 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.238 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.238 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1078 4.7 4:e41d47fb:::smithi03829856-14:head [write 1798066~477679] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:17.238 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:14.000345+0000", 2024-03-19T11:30:17.238 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.1577981020000001, 2024-03-19T11:30:17.239 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0054387289999999998, 2024-03-19T11:30:17.239 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:17.239 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:17.239 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:17.239 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:17.239 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:17.239 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1078 2024-03-19T11:30:17.239 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.239 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:17.239 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.239 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:17.239 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.000345+0000", 2024-03-19T11:30:17.239 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.239 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.239 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.240 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:17.240 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.000345+0000", 2024-03-19T11:30:17.240 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.240 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.240 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.240 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:17.240 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.000347+0000", 2024-03-19T11:30:17.240 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7010000000000001e-06 2024-03-19T11:30:17.240 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.240 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.240 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:17.240 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.000478+0000", 2024-03-19T11:30:17.240 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013129099999999999 2024-03-19T11:30:17.240 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.240 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.241 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:17.241 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.000495+0000", 2024-03-19T11:30:17.241 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6711999999999999e-05 2024-03-19T11:30:17.241 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.241 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.241 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:17.241 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.000513+0000", 2024-03-19T11:30:17.241 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8960000000000001e-05 2024-03-19T11:30:17.241 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.241 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.241 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:17.241 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.000571+0000", 2024-03-19T11:30:17.241 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7543000000000001e-05 2024-03-19T11:30:17.241 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.242 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.242 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:17.242 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.000643+0000", 2024-03-19T11:30:17.242 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2059999999999998e-05 2024-03-19T11:30:17.242 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.242 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.242 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:17.242 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.000749+0000", 2024-03-19T11:30:17.242 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000106302 2024-03-19T11:30:17.242 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.242 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.242 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:17.242 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.002758+0000", 2024-03-19T11:30:17.242 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0020088240000000002 2024-03-19T11:30:17.243 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.243 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.243 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:17.243 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.005694+0000", 2024-03-19T11:30:17.243 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002936035 2024-03-19T11:30:17.243 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.243 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.243 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:17.243 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.005731+0000", 2024-03-19T11:30:17.243 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6395e-05 2024-03-19T11:30:17.243 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.243 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.243 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:17.243 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.005784+0000", 2024-03-19T11:30:17.243 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2905999999999998e-05 2024-03-19T11:30:17.244 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.244 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:17.244 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.244 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.244 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.244 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1079 4.7 4:e41d47fb:::smithi03829856-14:head [write 2742758~457217] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:17.244 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:14.006264+0000", 2024-03-19T11:30:17.244 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.1518787810000002, 2024-03-19T11:30:17.244 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0068948910000000002, 2024-03-19T11:30:17.244 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:17.244 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:17.244 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:17.244 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:17.244 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:17.244 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1079 2024-03-19T11:30:17.245 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.245 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:17.245 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.245 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:17.245 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006264+0000", 2024-03-19T11:30:17.245 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.245 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.245 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.245 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:17.245 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006264+0000", 2024-03-19T11:30:17.245 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.245 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.245 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.245 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:17.245 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006266+0000", 2024-03-19T11:30:17.245 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3999999999999999e-06 2024-03-19T11:30:17.246 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.246 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.246 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:17.246 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006402+0000", 2024-03-19T11:30:17.246 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000136539 2024-03-19T11:30:17.246 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.246 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.246 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:17.246 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006420+0000", 2024-03-19T11:30:17.246 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7725000000000001e-05 2024-03-19T11:30:17.246 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.246 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.246 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:17.246 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006429+0000", 2024-03-19T11:30:17.247 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0100000000000001e-06 2024-03-19T11:30:17.247 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.247 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.247 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:17.247 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006484+0000", 2024-03-19T11:30:17.247 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4882000000000001e-05 2024-03-19T11:30:17.247 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.247 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.247 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:17.247 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006574+0000", 2024-03-19T11:30:17.247 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0091000000000002e-05 2024-03-19T11:30:17.247 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.247 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.247 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:17.247 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006722+0000", 2024-03-19T11:30:17.248 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014864300000000001 2024-03-19T11:30:17.248 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.248 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.248 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:17.248 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.009743+0000", 2024-03-19T11:30:17.248 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030209109999999998 2024-03-19T11:30:17.248 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.248 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.248 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:17.248 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.013026+0000", 2024-03-19T11:30:17.248 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003282414 2024-03-19T11:30:17.248 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.248 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.248 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:17.249 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.013108+0000", 2024-03-19T11:30:17.249 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2097999999999999e-05 2024-03-19T11:30:17.249 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.249 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.249 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:17.249 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.013159+0000", 2024-03-19T11:30:17.249 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1177999999999997e-05 2024-03-19T11:30:17.249 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.249 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:17.249 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.249 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.249 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.249 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1080 4.7 4:e41d47fb:::smithi03829856-14:head [setxattr _header (56) in=7b,truncate 3310946] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:17.249 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:14.006463+0000", 2024-03-19T11:30:17.250 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.1516797730000001, 2024-03-19T11:30:17.250 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0069389680000000002, 2024-03-19T11:30:17.250 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:17.250 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:17.250 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:17.250 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:17.250 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:17.250 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1080 2024-03-19T11:30:17.250 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.250 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:17.250 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.250 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:17.250 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006463+0000", 2024-03-19T11:30:17.250 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.250 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.251 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.251 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:17.251 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006463+0000", 2024-03-19T11:30:17.251 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.251 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.251 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.251 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:17.251 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006464+0000", 2024-03-19T11:30:17.251 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.54e-07 2024-03-19T11:30:17.251 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.251 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.251 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:17.251 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006469+0000", 2024-03-19T11:30:17.251 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9860000000000002e-06 2024-03-19T11:30:17.252 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.252 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.252 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:17.252 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006480+0000", 2024-03-19T11:30:17.252 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0831e-05 2024-03-19T11:30:17.252 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.252 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.252 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:17.252 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006486+0000", 2024-03-19T11:30:17.252 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0719999999999996e-06 2024-03-19T11:30:17.252 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.252 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.252 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:17.252 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.009073+0000", 2024-03-19T11:30:17.252 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025868169999999999 2024-03-19T11:30:17.252 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.252 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.253 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:17.253 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.009166+0000", 2024-03-19T11:30:17.253 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3066999999999995e-05 2024-03-19T11:30:17.253 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.253 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.253 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:17.253 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.009286+0000", 2024-03-19T11:30:17.253 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012071200000000001 2024-03-19T11:30:17.253 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.253 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.253 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:17.253 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.009869+0000", 2024-03-19T11:30:17.253 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00058305200000000003 2024-03-19T11:30:17.253 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.253 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.254 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:17.254 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.013314+0000", 2024-03-19T11:30:17.254 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034446350000000001 2024-03-19T11:30:17.254 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.254 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.254 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:17.254 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.013352+0000", 2024-03-19T11:30:17.254 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7462999999999997e-05 2024-03-19T11:30:17.254 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.254 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.254 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:17.254 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.013402+0000", 2024-03-19T11:30:17.254 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0578999999999998e-05 2024-03-19T11:30:17.254 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.254 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:17.255 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.255 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.255 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.255 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1081 4.7 4:e41d47fb:::smithi03829856-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:17.255 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:14.006496+0000", 2024-03-19T11:30:17.255 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.1516469960000002, 2024-03-19T11:30:17.255 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0070742670000000004, 2024-03-19T11:30:17.255 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:17.255 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:17.255 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:17.255 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:17.255 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:17.255 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1081 2024-03-19T11:30:17.255 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.255 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:17.256 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.256 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:17.256 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006496+0000", 2024-03-19T11:30:17.256 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.256 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.256 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.256 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:17.256 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006496+0000", 2024-03-19T11:30:17.256 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.256 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.256 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.256 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:17.256 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006496+0000", 2024-03-19T11:30:17.256 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2100000000000002e-07 2024-03-19T11:30:17.256 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.257 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.257 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:17.257 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006499+0000", 2024-03-19T11:30:17.257 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5560000000000002e-06 2024-03-19T11:30:17.257 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.257 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.257 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:17.257 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006506+0000", 2024-03-19T11:30:17.257 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4240000000000004e-06 2024-03-19T11:30:17.257 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.257 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.257 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:17.257 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006515+0000", 2024-03-19T11:30:17.257 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5949999999999999e-06 2024-03-19T11:30:17.257 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.258 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.258 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:17.258 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.009510+0000", 2024-03-19T11:30:17.258 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002995546 2024-03-19T11:30:17.258 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.258 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.258 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-03-19T11:30:17.258 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.009586+0000", 2024-03-19T11:30:17.258 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5859999999999995e-05 2024-03-19T11:30:17.258 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.258 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.258 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:17.258 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.013438+0000", 2024-03-19T11:30:17.258 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038516850000000001 2024-03-19T11:30:17.259 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.259 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.259 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:17.259 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.013499+0000", 2024-03-19T11:30:17.259 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0951e-05 2024-03-19T11:30:17.259 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.259 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.259 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:17.259 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.013570+0000", 2024-03-19T11:30:17.259 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1229000000000005e-05 2024-03-19T11:30:17.259 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.259 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:17.259 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.259 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.259 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:17.259 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:17.261 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-19T11:30:17.262 INFO:tasks.rados.rados.0.smithi038.stdout:745: expect (ObjNum 70 snap 4 seq_num 70) 2024-03-19T11:30:17.262 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:17.268+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:17.345 INFO:tasks.rados.rados.0.smithi038.stdout:742: expect (ObjNum 12 snap 0 seq_num 12) 2024-03-19T11:30:17.362 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:17.375+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:17.411 INFO:tasks.rados.rados.0.smithi038.stdout:743: expect (ObjNum 28 snap 0 seq_num 28) 2024-03-19T11:30:17.417 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:17.417 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:30:17.417 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:30:17.417 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:17.430 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-19T11:30:17.463 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:17.475+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:17.489 INFO:tasks.rados.rados.0.smithi038.stdout:748: expect (ObjNum 31 snap 0 seq_num 31) 2024-03-19T11:30:17.564 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:17.572+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:17.566 INFO:tasks.rados.rados.0.smithi038.stdout:744: got (0) Success 2024-03-19T11:30:17.567 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 23 v 9 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-03-19T11:30:17.567 INFO:tasks.rados.rados.0.smithi038.stdout:746: finishing write tid 1 to smithi03829856-43 2024-03-19T11:30:17.567 INFO:tasks.rados.rados.0.smithi038.stdout:746: finishing write tid 2 to smithi03829856-43 2024-03-19T11:30:17.568 INFO:tasks.rados.rados.0.smithi038.stdout:746: finishing write tid 3 to smithi03829856-43 2024-03-19T11:30:17.568 INFO:tasks.rados.rados.0.smithi038.stdout:746: finishing write tid 4 to smithi03829856-43 2024-03-19T11:30:17.568 INFO:tasks.rados.rados.0.smithi038.stdout:747: expect (ObjNum 34 snap 0 seq_num 34) 2024-03-19T11:30:17.595 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:17.595 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:30:17.595 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:30:17.595 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:30:17.595 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:17.608 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-19T11:30:17.645 INFO:tasks.rados.rados.0.smithi038.stdout:749: finishing write tid 1 to smithi03829856-2 2024-03-19T11:30:17.645 INFO:tasks.rados.rados.0.smithi038.stdout:746: finishing write tid 5 to smithi03829856-43 2024-03-19T11:30:17.645 INFO:tasks.rados.rados.0.smithi038.stdout:746: oid 43 updating version 0 to 47 2024-03-19T11:30:17.645 INFO:tasks.rados.rados.0.smithi038.stdout:746: oid 43 updating version 47 to 48 2024-03-19T11:30:17.645 INFO:tasks.rados.rados.0.smithi038.stdout:746: oid 43 updating version 48 to 49 2024-03-19T11:30:17.646 INFO:tasks.rados.rados.0.smithi038.stdout:746: oid 43 updating version 49 to 50 2024-03-19T11:30:17.646 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 43 v 50 (ObjNum 83 snap 7 seq_num 83) dirty exists 2024-03-19T11:30:17.646 INFO:tasks.rados.rados.0.smithi038.stdout:746: left oid 43 (ObjNum 83 snap 7 seq_num 83) 2024-03-19T11:30:17.646 INFO:tasks.rados.rados.0.smithi038.stdout:749: finishing write tid 2 to smithi03829856-2 2024-03-19T11:30:17.646 INFO:tasks.rados.rados.0.smithi038.stdout:751: finishing set_chunk to oid 26 2024-03-19T11:30:17.646 INFO:tasks.rados.rados.0.smithi038.stdout:751: got expected ERANGE (src dne) 2024-03-19T11:30:17.646 INFO:tasks.rados.rados.0.smithi038.stdout:749: finishing write tid 3 to smithi03829856-2 2024-03-19T11:30:17.646 INFO:tasks.rados.rados.0.smithi038.stdout:750: expect (ObjNum 21 snap 0 seq_num 21) 2024-03-19T11:30:17.690 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:17.704+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:17.720 INFO:tasks.rados.rados.0.smithi038.stdout:749: finishing write tid 4 to smithi03829856-2 2024-03-19T11:30:17.720 INFO:tasks.rados.rados.0.smithi038.stdout:749: finishing write tid 5 to smithi03829856-2 2024-03-19T11:30:17.720 INFO:tasks.rados.rados.0.smithi038.stdout:749: oid 2 updating version 0 to 45 2024-03-19T11:30:17.720 INFO:tasks.rados.rados.0.smithi038.stdout:749: oid 2 version 45 is already newer than 44 2024-03-19T11:30:17.720 INFO:tasks.rados.rados.0.smithi038.stdout:749: oid 2 updating version 45 to 47 2024-03-19T11:30:17.720 INFO:tasks.rados.rados.0.smithi038.stdout:749: oid 2 version 47 is already newer than 46 2024-03-19T11:30:17.720 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 2 v 47 (ObjNum 84 snap 7 seq_num 84) dirty exists 2024-03-19T11:30:17.721 INFO:tasks.rados.rados.0.smithi038.stdout:749: left oid 2 (ObjNum 84 snap 7 seq_num 84) 2024-03-19T11:30:17.721 INFO:tasks.rados.rados.0.smithi038.stdout:752: expect (ObjNum 32 snap 0 seq_num 32) 2024-03-19T11:30:17.766 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:17.780+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:17.773 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:17.773 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:30:17.773 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:30:17.773 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:30:17.773 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.774 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:9 4.d e28/21 4:b4ef97e0:::smithi03829856-3:head v 28'1, mlcod=0'0)", 2024-03-19T11:30:17.774 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.707728+0000", 2024-03-19T11:30:17.774 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 13.078106861, 2024-03-19T11:30:17.774 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0037487950000000001, 2024-03-19T11:30:17.774 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:17.774 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:17.774 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:17.774 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.774 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:17.774 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.707728+0000", 2024-03-19T11:30:17.774 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.774 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.774 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.775 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:17.775 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.707728+0000", 2024-03-19T11:30:17.775 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.775 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.775 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.775 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:17.775 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.707729+0000", 2024-03-19T11:30:17.775 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6899999999999998e-07 2024-03-19T11:30:17.775 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.775 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.775 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:17.775 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.708625+0000", 2024-03-19T11:30:17.775 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00089603899999999995 2024-03-19T11:30:17.775 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.776 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.776 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:17.776 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.708645+0000", 2024-03-19T11:30:17.776 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9629000000000001e-05 2024-03-19T11:30:17.776 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.776 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.776 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:17.776 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.708657+0000", 2024-03-19T11:30:17.776 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2537999999999999e-05 2024-03-19T11:30:17.776 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.776 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.776 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:17.776 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.708740+0000", 2024-03-19T11:30:17.776 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2490999999999993e-05 2024-03-19T11:30:17.776 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.777 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.777 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:17.777 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.708774+0000", 2024-03-19T11:30:17.777 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4362000000000001e-05 2024-03-19T11:30:17.777 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.777 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.777 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:17.777 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.711419+0000", 2024-03-19T11:30:17.777 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0026446830000000001 2024-03-19T11:30:17.777 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.777 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.777 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:17.777 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.711477+0000", 2024-03-19T11:30:17.777 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8684000000000003e-05 2024-03-19T11:30:17.777 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.778 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:17.778 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.778 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.778 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.778 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:10 4.d e28/21 4:b4ef97e0:::smithi03829856-3:head v 28'2, mlcod=0'0)", 2024-03-19T11:30:17.778 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.713599+0000", 2024-03-19T11:30:17.778 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 13.072236403, 2024-03-19T11:30:17.778 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002599905, 2024-03-19T11:30:17.778 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:17.779 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:17.779 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:17.779 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.779 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:17.779 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.713599+0000", 2024-03-19T11:30:17.779 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.779 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.779 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.779 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:17.779 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.713599+0000", 2024-03-19T11:30:17.779 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.779 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.780 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.780 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:17.780 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.713619+0000", 2024-03-19T11:30:17.780 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0398999999999999e-05 2024-03-19T11:30:17.780 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.780 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.780 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:17.780 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.713955+0000", 2024-03-19T11:30:17.780 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000335691 2024-03-19T11:30:17.780 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.781 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.781 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:17.781 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.713976+0000", 2024-03-19T11:30:17.781 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0866999999999999e-05 2024-03-19T11:30:17.781 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.781 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.781 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:17.781 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.713988+0000", 2024-03-19T11:30:17.781 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2228e-05 2024-03-19T11:30:17.781 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.781 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.782 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:17.782 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.714058+0000", 2024-03-19T11:30:17.782 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9868000000000003e-05 2024-03-19T11:30:17.782 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.782 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.782 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:17.782 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.714086+0000", 2024-03-19T11:30:17.782 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8124999999999999e-05 2024-03-19T11:30:17.782 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.782 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.782 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:17.782 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.716138+0000", 2024-03-19T11:30:17.782 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0020517759999999999 2024-03-19T11:30:17.782 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.783 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.783 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:17.783 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.716199+0000", 2024-03-19T11:30:17.783 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0951e-05 2024-03-19T11:30:17.783 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.783 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:17.783 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.783 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.783 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.783 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:13 4.4 e28/21 4:21eac9b2:::smithi03829856-4:head v 28'1, mlcod=0'0)", 2024-03-19T11:30:17.783 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.723258+0000", 2024-03-19T11:30:17.783 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 13.062576866000001, 2024-03-19T11:30:17.783 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0048979840000000002, 2024-03-19T11:30:17.783 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:17.783 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:17.784 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:17.784 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.784 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:17.784 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723258+0000", 2024-03-19T11:30:17.784 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.784 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.784 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.784 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:17.784 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723258+0000", 2024-03-19T11:30:17.784 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.784 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.784 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.784 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:17.784 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723259+0000", 2024-03-19T11:30:17.785 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.7399999999999999e-07 2024-03-19T11:30:17.785 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.785 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.785 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:17.785 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723812+0000", 2024-03-19T11:30:17.785 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00055371000000000001 2024-03-19T11:30:17.785 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.785 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.785 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:17.785 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723833+0000", 2024-03-19T11:30:17.785 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0990000000000001e-05 2024-03-19T11:30:17.785 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.785 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.785 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:17.785 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723847+0000", 2024-03-19T11:30:17.786 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3774000000000001e-05 2024-03-19T11:30:17.786 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.786 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.786 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:17.786 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723912+0000", 2024-03-19T11:30:17.786 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4758000000000006e-05 2024-03-19T11:30:17.786 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.786 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.786 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:17.786 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723933+0000", 2024-03-19T11:30:17.786 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1046000000000001e-05 2024-03-19T11:30:17.786 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.786 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.786 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:17.787 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.728103+0000", 2024-03-19T11:30:17.787 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0041695079999999997 2024-03-19T11:30:17.787 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.787 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.787 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:17.787 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.728156+0000", 2024-03-19T11:30:17.787 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3924000000000001e-05 2024-03-19T11:30:17.787 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.787 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:17.787 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.787 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.787 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.787 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:14 4.4 e28/21 4:21eac9b2:::smithi03829856-4:head v 28'2, mlcod=0'0)", 2024-03-19T11:30:17.787 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.726103+0000", 2024-03-19T11:30:17.787 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 13.059732016, 2024-03-19T11:30:17.788 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025805440000000002, 2024-03-19T11:30:17.788 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:17.788 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:17.788 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:17.788 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.788 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:17.788 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.726103+0000", 2024-03-19T11:30:17.788 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.788 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.788 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.788 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:17.788 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.726103+0000", 2024-03-19T11:30:17.788 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.788 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.789 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.789 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:17.789 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.726104+0000", 2024-03-19T11:30:17.789 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0920000000000001e-06 2024-03-19T11:30:17.789 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.789 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.789 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:17.790 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.726321+0000", 2024-03-19T11:30:17.790 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00021668499999999999 2024-03-19T11:30:17.790 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.790 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.790 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:17.790 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.726338+0000", 2024-03-19T11:30:17.790 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6382999999999999e-05 2024-03-19T11:30:17.790 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.790 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.790 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:17.790 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.726347+0000", 2024-03-19T11:30:17.790 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7399999999999999e-06 2024-03-19T11:30:17.790 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.791 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.791 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:17.791 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.726422+0000", 2024-03-19T11:30:17.791 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4365999999999999e-05 2024-03-19T11:30:17.791 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.791 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.791 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:17.792 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.726451+0000", 2024-03-19T11:30:17.792 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9085e-05 2024-03-19T11:30:17.792 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.792 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.792 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:17.792 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.728643+0000", 2024-03-19T11:30:17.792 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0021926620000000002 2024-03-19T11:30:17.792 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.792 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.792 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:17.792 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.728684+0000", 2024-03-19T11:30:17.792 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0531000000000002e-05 2024-03-19T11:30:17.792 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.792 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:17.792 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.793 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.793 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.793 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:33 4.d e28/21 4:b65e6658:::smithi03829856-9:head v 28'4, mlcod=28'2)", 2024-03-19T11:30:17.793 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.785780+0000", 2024-03-19T11:30:17.793 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 13.000055095, 2024-03-19T11:30:17.793 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0032708569999999998, 2024-03-19T11:30:17.793 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:17.793 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:17.793 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:17.793 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.793 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:17.793 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.785780+0000", 2024-03-19T11:30:17.793 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.793 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.794 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.794 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:17.794 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.785780+0000", 2024-03-19T11:30:17.794 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.794 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.794 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.794 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:17.794 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.785783+0000", 2024-03-19T11:30:17.794 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4150000000000002e-06 2024-03-19T11:30:17.794 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.794 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.794 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:17.794 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.786482+0000", 2024-03-19T11:30:17.794 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00069957299999999999 2024-03-19T11:30:17.795 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.795 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.795 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:17.795 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.786507+0000", 2024-03-19T11:30:17.795 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5160999999999998e-05 2024-03-19T11:30:17.795 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.795 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.795 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:17.795 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.786521+0000", 2024-03-19T11:30:17.795 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3301e-05 2024-03-19T11:30:17.795 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.795 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.795 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:17.795 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.786599+0000", 2024-03-19T11:30:17.796 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7905999999999996e-05 2024-03-19T11:30:17.796 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.796 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.796 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:17.796 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.786630+0000", 2024-03-19T11:30:17.796 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1636e-05 2024-03-19T11:30:17.796 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.796 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.796 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:17.796 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.789001+0000", 2024-03-19T11:30:17.796 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023710530000000001 2024-03-19T11:30:17.796 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.796 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.796 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:17.796 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.789051+0000", 2024-03-19T11:30:17.797 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9811999999999997e-05 2024-03-19T11:30:17.797 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.797 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:17.797 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.797 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.797 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.797 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:871 4.d e32/21 4:b4ef97e0:::smithi03829856-3:head v 32'19, mlcod=28'16)", 2024-03-19T11:30:17.797 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.029331+0000", 2024-03-19T11:30:17.797 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.7565041929999996, 2024-03-19T11:30:17.797 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057816339999999999, 2024-03-19T11:30:17.797 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:17.797 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:17.797 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:17.797 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.797 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:17.798 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.029331+0000", 2024-03-19T11:30:17.798 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.798 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.798 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.798 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:17.798 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.029331+0000", 2024-03-19T11:30:17.798 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.798 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.798 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.798 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:17.798 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.029331+0000", 2024-03-19T11:30:17.798 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0400000000000002e-07 2024-03-19T11:30:17.798 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.798 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.799 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:17.799 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.030605+0000", 2024-03-19T11:30:17.799 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0012733899999999999 2024-03-19T11:30:17.799 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.799 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.799 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:17.799 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.030620+0000", 2024-03-19T11:30:17.799 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5557999999999999e-05 2024-03-19T11:30:17.800 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.800 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.800 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:17.800 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.030630+0000", 2024-03-19T11:30:17.800 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2660000000000007e-06 2024-03-19T11:30:17.800 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.800 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.800 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:17.800 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.030727+0000", 2024-03-19T11:30:17.800 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7452000000000001e-05 2024-03-19T11:30:17.800 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.800 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.800 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:17.800 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.030757+0000", 2024-03-19T11:30:17.801 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0018999999999998e-05 2024-03-19T11:30:17.801 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.801 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.801 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:17.801 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.035063+0000", 2024-03-19T11:30:17.801 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0043055350000000001 2024-03-19T11:30:17.801 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.801 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.801 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:17.801 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.035113+0000", 2024-03-19T11:30:17.801 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.011e-05 2024-03-19T11:30:17.801 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.801 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:17.801 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.802 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.802 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.802 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:872 4.d e32/21 4:b4ef97e0:::smithi03829856-3:head v 32'20, mlcod=28'16)", 2024-03-19T11:30:17.802 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.036163+0000", 2024-03-19T11:30:17.802 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.7496719669999994, 2024-03-19T11:30:17.802 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029333250000000001, 2024-03-19T11:30:17.802 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:17.802 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:17.802 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:17.802 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.802 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:17.802 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.036163+0000", 2024-03-19T11:30:17.802 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.802 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.803 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.803 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:17.803 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.036163+0000", 2024-03-19T11:30:17.803 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.803 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.803 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.803 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:17.803 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.036164+0000", 2024-03-19T11:30:17.803 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7000000000000003e-07 2024-03-19T11:30:17.803 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.803 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.803 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:17.803 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.036721+0000", 2024-03-19T11:30:17.803 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00055646700000000005 2024-03-19T11:30:17.804 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.804 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.804 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:17.804 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.036732+0000", 2024-03-19T11:30:17.804 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1649e-05 2024-03-19T11:30:17.804 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.804 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.804 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:17.804 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.036739+0000", 2024-03-19T11:30:17.804 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6760000000000001e-06 2024-03-19T11:30:17.804 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.804 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.804 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:17.804 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.036785+0000", 2024-03-19T11:30:17.805 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6177000000000002e-05 2024-03-19T11:30:17.805 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.805 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.805 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:17.805 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.036808+0000", 2024-03-19T11:30:17.805 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2654999999999999e-05 2024-03-19T11:30:17.805 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.805 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.805 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:17.805 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.039022+0000", 2024-03-19T11:30:17.805 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022145530000000002 2024-03-19T11:30:17.805 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.805 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.805 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:17.805 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.039097+0000", 2024-03-19T11:30:17.806 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4277999999999996e-05 2024-03-19T11:30:17.806 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.806 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:17.806 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.806 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.806 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.806 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:893 3.0 e32/20 3:227a6c36:::c8c8170dc54fef3b4e25ae28106dacc8accf0657c6d6457bff8aa63e56c9678d:head v 32'52, mlcod=28'50)", 2024-03-19T11:30:17.806 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.291375+0000", 2024-03-19T11:30:17.806 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.4944599469999993, 2024-03-19T11:30:17.806 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0020008869999999998, 2024-03-19T11:30:17.806 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:17.806 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:17.806 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:17.806 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.807 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:17.807 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.291375+0000", 2024-03-19T11:30:17.807 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.807 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.807 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.807 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:17.807 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.291375+0000", 2024-03-19T11:30:17.807 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.807 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.807 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.807 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:17.807 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.291376+0000", 2024-03-19T11:30:17.807 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0200000000000003e-07 2024-03-19T11:30:17.807 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.808 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.808 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:17.808 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.291682+0000", 2024-03-19T11:30:17.808 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000306657 2024-03-19T11:30:17.808 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.808 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.808 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:17.808 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.291699+0000", 2024-03-19T11:30:17.808 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6291999999999999e-05 2024-03-19T11:30:17.808 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.808 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.808 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:17.808 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.291710+0000", 2024-03-19T11:30:17.808 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1117e-05 2024-03-19T11:30:17.808 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.809 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.809 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:17.809 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.291780+0000", 2024-03-19T11:30:17.809 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9751000000000004e-05 2024-03-19T11:30:17.809 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.809 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.809 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:17.809 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.291803+0000", 2024-03-19T11:30:17.809 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3787000000000001e-05 2024-03-19T11:30:17.809 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.809 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.809 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:17.809 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.293338+0000", 2024-03-19T11:30:17.809 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0015349649999999999 2024-03-19T11:30:17.810 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.810 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.810 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:17.810 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.293376+0000", 2024-03-19T11:30:17.810 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7916000000000001e-05 2024-03-19T11:30:17.810 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.810 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:17.810 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.810 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.810 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.810 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(osd.6.13:39 3.0 e32/20 3:3dcd2456:::039914993b6f4082c204b24ae0ff925ebffccc22519067b64afa9eff107cf533:head v 32'54, mlcod=32'52)", 2024-03-19T11:30:17.810 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.308854+0000", 2024-03-19T11:30:17.810 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.4769816779999996, 2024-03-19T11:30:17.810 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003468175, 2024-03-19T11:30:17.811 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:17.811 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:17.811 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:17.811 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.811 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:17.811 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308854+0000", 2024-03-19T11:30:17.811 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.811 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.811 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.811 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:17.811 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308854+0000", 2024-03-19T11:30:17.811 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.811 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.811 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.812 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:17.812 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308855+0000", 2024-03-19T11:30:17.812 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5999999999999999e-06 2024-03-19T11:30:17.812 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.812 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.812 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:17.812 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309019+0000", 2024-03-19T11:30:17.812 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000163311 2024-03-19T11:30:17.812 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.812 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.812 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:17.812 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309040+0000", 2024-03-19T11:30:17.812 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1795999999999999e-05 2024-03-19T11:30:17.812 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.812 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.813 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:17.813 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309060+0000", 2024-03-19T11:30:17.813 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9677000000000002e-05 2024-03-19T11:30:17.813 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.813 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.813 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:17.813 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309151+0000", 2024-03-19T11:30:17.813 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0507000000000006e-05 2024-03-19T11:30:17.813 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.813 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.813 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:17.813 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309187+0000", 2024-03-19T11:30:17.813 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6675999999999999e-05 2024-03-19T11:30:17.813 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.814 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.814 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:17.814 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.312256+0000", 2024-03-19T11:30:17.814 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030692060000000001 2024-03-19T11:30:17.814 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.814 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.814 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:17.814 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.312322+0000", 2024-03-19T11:30:17.814 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5401999999999995e-05 2024-03-19T11:30:17.814 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.814 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:17.814 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.814 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.814 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.814 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(osd.6.13:40 3.0 e32/20 3:0f3835fc:::79445f5816960fdb5fb7d7699594b1dfc30a0a44639e804aee69d6cb400c02a5:head v 32'55, mlcod=32'52)", 2024-03-19T11:30:17.815 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.310652+0000", 2024-03-19T11:30:17.815 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.4751838250000002, 2024-03-19T11:30:17.815 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022725919999999999, 2024-03-19T11:30:17.815 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:17.815 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:17.815 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:17.815 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.815 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:17.815 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310652+0000", 2024-03-19T11:30:17.815 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.815 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.815 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.815 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:17.815 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310652+0000", 2024-03-19T11:30:17.816 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.816 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.816 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.816 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:17.816 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310653+0000", 2024-03-19T11:30:17.816 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4339999999999999e-06 2024-03-19T11:30:17.816 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.816 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.817 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:17.817 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310764+0000", 2024-03-19T11:30:17.817 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000111174 2024-03-19T11:30:17.817 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.817 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.817 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:17.817 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310779+0000", 2024-03-19T11:30:17.817 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5024999999999999e-05 2024-03-19T11:30:17.817 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.817 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.817 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:17.817 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310789+0000", 2024-03-19T11:30:17.817 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0098999999999999e-05 2024-03-19T11:30:17.817 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.818 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.818 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:17.818 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310896+0000", 2024-03-19T11:30:17.818 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010627499999999999 2024-03-19T11:30:17.818 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.818 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.818 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:17.818 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310927+0000", 2024-03-19T11:30:17.818 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1078000000000003e-05 2024-03-19T11:30:17.818 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.818 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.818 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:17.818 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.312864+0000", 2024-03-19T11:30:17.818 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001937137 2024-03-19T11:30:17.819 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.819 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.819 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:17.819 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.312924+0000", 2024-03-19T11:30:17.819 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.037e-05 2024-03-19T11:30:17.819 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.819 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:17.819 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.819 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.819 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.819 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(osd.6.13:48 3.0 e32/20 3:0dfaac1c:::smithi03829856-20 low_tier:head v 32'58, mlcod=32'54)", 2024-03-19T11:30:17.819 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.324282+0000", 2024-03-19T11:30:17.819 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.4615534859999997, 2024-03-19T11:30:17.819 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017139049999999999, 2024-03-19T11:30:17.820 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:17.820 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:17.820 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:17.820 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.820 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:17.820 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.324282+0000", 2024-03-19T11:30:17.820 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.820 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.820 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.820 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:17.820 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.324282+0000", 2024-03-19T11:30:17.820 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.820 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.820 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.821 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:17.821 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.324283+0000", 2024-03-19T11:30:17.821 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0500000000000002e-07 2024-03-19T11:30:17.821 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.821 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.821 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:17.821 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.324292+0000", 2024-03-19T11:30:17.821 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.5270000000000004e-06 2024-03-19T11:30:17.821 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.821 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.821 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:17.821 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.324308+0000", 2024-03-19T11:30:17.822 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5920999999999999e-05 2024-03-19T11:30:17.822 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.822 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.822 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:17.822 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.324317+0000", 2024-03-19T11:30:17.822 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1239999999999994e-06 2024-03-19T11:30:17.822 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.822 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.822 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:17.822 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.324523+0000", 2024-03-19T11:30:17.822 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020557 2024-03-19T11:30:17.822 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.822 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.822 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:17.823 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.324536+0000", 2024-03-19T11:30:17.823 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3077e-05 2024-03-19T11:30:17.823 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.823 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.823 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:17.823 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.325937+0000", 2024-03-19T11:30:17.823 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014012720000000001 2024-03-19T11:30:17.823 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.823 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.823 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:17.823 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.325996+0000", 2024-03-19T11:30:17.823 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8508999999999998e-05 2024-03-19T11:30:17.823 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.823 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:17.823 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.824 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.824 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.824 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:901 4.d e32/21 4:ba44105a:::smithi03829856-26:head v 32'23, mlcod=32'20)", 2024-03-19T11:30:17.824 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.465629+0000", 2024-03-19T11:30:17.824 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.3202063150000001, 2024-03-19T11:30:17.824 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002624361, 2024-03-19T11:30:17.824 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:17.824 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:17.824 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:17.824 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.824 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:17.824 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.465629+0000", 2024-03-19T11:30:17.824 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.824 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.825 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.825 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:17.825 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.465629+0000", 2024-03-19T11:30:17.825 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.825 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.825 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.825 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:17.825 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.465631+0000", 2024-03-19T11:30:17.825 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5740000000000001e-06 2024-03-19T11:30:17.825 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.825 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.825 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:17.825 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.465642+0000", 2024-03-19T11:30:17.825 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1343e-05 2024-03-19T11:30:17.825 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.826 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.826 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:17.826 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.465663+0000", 2024-03-19T11:30:17.826 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1535e-05 2024-03-19T11:30:17.826 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.826 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.826 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:17.826 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.465676+0000", 2024-03-19T11:30:17.826 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2281e-05 2024-03-19T11:30:17.826 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.826 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.826 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:17.826 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.465777+0000", 2024-03-19T11:30:17.826 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010083500000000001 2024-03-19T11:30:17.827 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.827 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.827 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:17.827 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.465806+0000", 2024-03-19T11:30:17.827 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9414999999999999e-05 2024-03-19T11:30:17.827 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.827 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.827 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:17.827 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.468174+0000", 2024-03-19T11:30:17.827 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023684299999999999 2024-03-19T11:30:17.827 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.827 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.827 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:17.827 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.468253+0000", 2024-03-19T11:30:17.827 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8948000000000004e-05 2024-03-19T11:30:17.828 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.828 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:17.828 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.828 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.828 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.828 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:997 4.4 e34/21 4:27909373:::smithi03829856-46:head v 34'36, mlcod=32'32)", 2024-03-19T11:30:17.828 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.368638+0000", 2024-03-19T11:30:17.828 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.4171971379999997, 2024-03-19T11:30:17.828 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001933892, 2024-03-19T11:30:17.828 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:17.828 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:17.828 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:17.828 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.828 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:17.828 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368638+0000", 2024-03-19T11:30:17.829 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.829 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.829 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.829 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:17.829 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368638+0000", 2024-03-19T11:30:17.829 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.829 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.829 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.829 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:17.829 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368638+0000", 2024-03-19T11:30:17.829 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8900000000000001e-07 2024-03-19T11:30:17.829 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.829 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.829 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:17.830 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368644+0000", 2024-03-19T11:30:17.830 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3229999999999997e-06 2024-03-19T11:30:17.830 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.830 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.830 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:17.830 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368660+0000", 2024-03-19T11:30:17.830 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6344000000000001e-05 2024-03-19T11:30:17.830 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.830 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.830 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:17.830 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368671+0000", 2024-03-19T11:30:17.830 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1094000000000001e-05 2024-03-19T11:30:17.830 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.830 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.831 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:17.831 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368751+0000", 2024-03-19T11:30:17.831 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9357999999999995e-05 2024-03-19T11:30:17.831 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.831 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.831 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:17.831 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368776+0000", 2024-03-19T11:30:17.831 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5389e-05 2024-03-19T11:30:17.831 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.831 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.831 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:17.831 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370511+0000", 2024-03-19T11:30:17.831 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001735098 2024-03-19T11:30:17.831 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.832 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.832 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:17.832 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370572+0000", 2024-03-19T11:30:17.832 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1097000000000001e-05 2024-03-19T11:30:17.832 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.832 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:17.832 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.832 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.832 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.832 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:998 4.4 e34/21 4:27909373:::smithi03829856-46:head v 34'37, mlcod=32'32)", 2024-03-19T11:30:17.832 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.370628+0000", 2024-03-19T11:30:17.832 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.4152074649999999, 2024-03-19T11:30:17.832 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050953099999999996, 2024-03-19T11:30:17.832 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:17.832 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:17.833 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:17.833 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.833 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:17.833 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370628+0000", 2024-03-19T11:30:17.833 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.833 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.833 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:17.833 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370628+0000", 2024-03-19T11:30:17.833 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.833 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.833 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:17.833 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370629+0000", 2024-03-19T11:30:17.834 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.5900000000000005e-07 2024-03-19T11:30:17.834 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.834 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.834 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:17.834 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370638+0000", 2024-03-19T11:30:17.834 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8189999999999998e-06 2024-03-19T11:30:17.834 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.834 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.834 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:17.834 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370650+0000", 2024-03-19T11:30:17.834 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.275e-05 2024-03-19T11:30:17.834 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.834 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.834 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:17.835 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370658+0000", 2024-03-19T11:30:17.835 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5769999999999996e-06 2024-03-19T11:30:17.835 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.835 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.835 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:17.835 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370747+0000", 2024-03-19T11:30:17.835 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8642999999999998e-05 2024-03-19T11:30:17.835 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.835 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.835 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:17.835 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370796+0000", 2024-03-19T11:30:17.835 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9428000000000002e-05 2024-03-19T11:30:17.835 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.835 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.835 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:17.836 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.375658+0000", 2024-03-19T11:30:17.836 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0048621519999999998 2024-03-19T11:30:17.836 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.836 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.836 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:17.836 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.375723+0000", 2024-03-19T11:30:17.836 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4981999999999995e-05 2024-03-19T11:30:17.836 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.836 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:17.836 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.836 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.836 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.836 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:999 4.4 e34/21 4:27909373:::smithi03829856-46:head v 34'38, mlcod=32'32)", 2024-03-19T11:30:17.836 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.372487+0000", 2024-03-19T11:30:17.837 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.4133487000000002, 2024-03-19T11:30:17.837 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0035853069999999998, 2024-03-19T11:30:17.837 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:17.837 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:17.837 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:17.837 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.837 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:17.837 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372487+0000", 2024-03-19T11:30:17.837 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.837 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.837 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.837 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:17.837 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372487+0000", 2024-03-19T11:30:17.837 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.837 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.838 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.838 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:17.838 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372488+0000", 2024-03-19T11:30:17.838 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1069999999999999e-06 2024-03-19T11:30:17.838 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.838 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.838 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:17.838 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372498+0000", 2024-03-19T11:30:17.838 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7510000000000003e-06 2024-03-19T11:30:17.838 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.838 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.838 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:17.838 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372544+0000", 2024-03-19T11:30:17.838 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6218999999999999e-05 2024-03-19T11:30:17.841 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.841 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.841 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:17.842 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372553+0000", 2024-03-19T11:30:17.842 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6320000000000003e-06 2024-03-19T11:30:17.842 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.842 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.842 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:17.842 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372636+0000", 2024-03-19T11:30:17.842 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3125000000000001e-05 2024-03-19T11:30:17.842 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.842 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.842 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:17.842 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372651+0000", 2024-03-19T11:30:17.842 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4676e-05 2024-03-19T11:30:17.842 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.843 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.843 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:17.843 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.376026+0000", 2024-03-19T11:30:17.843 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033745630000000001 2024-03-19T11:30:17.843 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.843 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.843 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:17.843 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.376072+0000", 2024-03-19T11:30:17.843 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6233999999999998e-05 2024-03-19T11:30:17.843 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.843 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:17.843 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.843 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.843 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.843 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:1003 4.4 e34/21 4:22709789:::smithi03829856-41:head v 34'40, mlcod=34'37)", 2024-03-19T11:30:17.844 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.378551+0000", 2024-03-19T11:30:17.844 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.4072846129999999, 2024-03-19T11:30:17.844 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0041156680000000003, 2024-03-19T11:30:17.844 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:17.844 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:17.844 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:17.844 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.844 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:17.844 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.378551+0000", 2024-03-19T11:30:17.844 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.844 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.844 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.844 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:17.844 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.378551+0000", 2024-03-19T11:30:17.845 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.845 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.845 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.845 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:17.845 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.378552+0000", 2024-03-19T11:30:17.845 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0400000000000005e-07 2024-03-19T11:30:17.845 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.845 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.845 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:17.845 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.378929+0000", 2024-03-19T11:30:17.845 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037751700000000001 2024-03-19T11:30:17.845 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.845 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.845 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:17.845 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.378944+0000", 2024-03-19T11:30:17.846 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5038000000000001e-05 2024-03-19T11:30:17.846 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.846 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.846 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:17.846 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.378953+0000", 2024-03-19T11:30:17.846 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7649999999999999e-06 2024-03-19T11:30:17.846 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.846 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.846 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:17.846 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.379017+0000", 2024-03-19T11:30:17.846 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3614000000000005e-05 2024-03-19T11:30:17.846 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.846 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.846 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:17.847 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.379042+0000", 2024-03-19T11:30:17.847 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5910000000000001e-05 2024-03-19T11:30:17.847 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.847 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.847 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:17.847 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.382610+0000", 2024-03-19T11:30:17.847 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0035678340000000002 2024-03-19T11:30:17.847 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.847 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.847 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:17.847 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.382666+0000", 2024-03-19T11:30:17.848 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6085999999999998e-05 2024-03-19T11:30:17.848 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.848 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:17.848 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.848 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.848 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.848 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:1004 4.4 e34/21 4:22709789:::smithi03829856-41:head v 34'41, mlcod=34'38)", 2024-03-19T11:30:17.848 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.386855+0000", 2024-03-19T11:30:17.848 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.3989808039999998, 2024-03-19T11:30:17.848 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0039133320000000003, 2024-03-19T11:30:17.848 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:17.848 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:17.848 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:17.848 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.849 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:17.849 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.386855+0000", 2024-03-19T11:30:17.849 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.849 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.849 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.849 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:17.849 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.386855+0000", 2024-03-19T11:30:17.849 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.849 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.849 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.849 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:17.849 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.386856+0000", 2024-03-19T11:30:17.849 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.528e-06 2024-03-19T11:30:17.849 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.850 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.850 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:17.850 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.387143+0000", 2024-03-19T11:30:17.850 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00028658600000000002 2024-03-19T11:30:17.850 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.850 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.850 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:17.850 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.387159+0000", 2024-03-19T11:30:17.850 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6796e-05 2024-03-19T11:30:17.850 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.850 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.850 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:17.850 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.387167+0000", 2024-03-19T11:30:17.850 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2729999999999997e-06 2024-03-19T11:30:17.851 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.851 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.851 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:17.851 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.387230+0000", 2024-03-19T11:30:17.851 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2862999999999998e-05 2024-03-19T11:30:17.851 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.851 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.851 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:17.851 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.387255+0000", 2024-03-19T11:30:17.851 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4905e-05 2024-03-19T11:30:17.851 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.851 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.851 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:17.851 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.390720+0000", 2024-03-19T11:30:17.851 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034652699999999999 2024-03-19T11:30:17.852 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.852 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.852 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:17.852 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.390768+0000", 2024-03-19T11:30:17.852 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8111000000000001e-05 2024-03-19T11:30:17.852 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.852 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:17.852 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.852 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.852 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.852 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:1042 3.0 e35/20 3:0e3fb112:::bdb3b5647d9daba88ca1ec369a6a4ecc66592b64bbd6df6ad063af4137f6b5b0:head v 35'62, mlcod=32'60)", 2024-03-19T11:30:17.853 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.211286+0000", 2024-03-19T11:30:17.853 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.5745489700000004, 2024-03-19T11:30:17.853 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0024813159999999999, 2024-03-19T11:30:17.853 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:17.853 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:17.853 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:17.853 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.853 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:17.853 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.211286+0000", 2024-03-19T11:30:17.853 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.853 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.853 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.853 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:17.853 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.211286+0000", 2024-03-19T11:30:17.854 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.854 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.854 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.854 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:17.854 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.211287+0000", 2024-03-19T11:30:17.854 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.67e-07 2024-03-19T11:30:17.854 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.854 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.854 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:17.854 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.211562+0000", 2024-03-19T11:30:17.854 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00027568999999999998 2024-03-19T11:30:17.854 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.854 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.854 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:17.855 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.211579+0000", 2024-03-19T11:30:17.855 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6489000000000001e-05 2024-03-19T11:30:17.855 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.855 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.855 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:17.855 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.211593+0000", 2024-03-19T11:30:17.855 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4351e-05 2024-03-19T11:30:17.855 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.855 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.855 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:17.855 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.211678+0000", 2024-03-19T11:30:17.855 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4924000000000002e-05 2024-03-19T11:30:17.855 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.855 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.855 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:17.856 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.211713+0000", 2024-03-19T11:30:17.856 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4492000000000001e-05 2024-03-19T11:30:17.856 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.856 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.856 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:17.856 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.213730+0000", 2024-03-19T11:30:17.856 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002017114 2024-03-19T11:30:17.856 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.856 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.856 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:17.856 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.213768+0000", 2024-03-19T11:30:17.856 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7988999999999998e-05 2024-03-19T11:30:17.856 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.856 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:17.857 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.857 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.857 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.857 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(osd.1.14:37 3.0 e36/20 3:112c7abb:::2bf9877f5461003658d9336168a0bc98c40a21a7becad71a3b9fe0391d35a5b0:head v 36'63, mlcod=32'61)", 2024-03-19T11:30:17.857 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.796971+0000", 2024-03-19T11:30:17.857 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.9888641979999999, 2024-03-19T11:30:17.857 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0016589759999999999, 2024-03-19T11:30:17.857 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:17.857 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:17.857 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:17.857 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.857 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:17.857 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.796971+0000", 2024-03-19T11:30:17.857 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.857 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.858 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.858 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:17.858 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.796971+0000", 2024-03-19T11:30:17.858 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.858 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.858 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.858 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:17.858 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.796972+0000", 2024-03-19T11:30:17.858 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7899999999999998e-07 2024-03-19T11:30:17.858 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.858 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.858 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:17.858 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.797243+0000", 2024-03-19T11:30:17.858 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00027107299999999998 2024-03-19T11:30:17.859 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.859 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.859 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:17.859 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.797264+0000", 2024-03-19T11:30:17.859 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1335000000000002e-05 2024-03-19T11:30:17.859 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.859 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.859 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:17.859 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.797274+0000", 2024-03-19T11:30:17.859 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9729999999999992e-06 2024-03-19T11:30:17.859 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.859 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.859 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:17.859 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.797365+0000", 2024-03-19T11:30:17.859 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0919000000000001e-05 2024-03-19T11:30:17.860 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.860 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.860 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:17.860 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.797385+0000", 2024-03-19T11:30:17.860 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.016e-05 2024-03-19T11:30:17.860 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.860 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.860 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:17.860 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.798584+0000", 2024-03-19T11:30:17.860 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00119863 2024-03-19T11:30:17.860 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.860 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.860 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:17.860 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.798630+0000", 2024-03-19T11:30:17.861 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6207e-05 2024-03-19T11:30:17.861 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.861 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:17.861 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.861 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.861 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.861 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(osd.1.14:39 3.0 e36/20 3:2496b85f:::77b213eb5a71944892061f01e7e26ebe9b44d37bd5fe5c50cc57111bcedd8676:head v 36'64, mlcod=32'61)", 2024-03-19T11:30:17.861 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.798662+0000", 2024-03-19T11:30:17.861 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.9871735689999999, 2024-03-19T11:30:17.861 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00182927, 2024-03-19T11:30:17.861 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:17.861 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:17.861 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:17.861 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.861 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:17.862 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.798662+0000", 2024-03-19T11:30:17.862 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.862 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.862 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.862 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:17.862 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.798662+0000", 2024-03-19T11:30:17.862 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:17.862 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.862 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.862 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:17.862 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.798663+0000", 2024-03-19T11:30:17.862 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8899999999999998e-07 2024-03-19T11:30:17.862 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.862 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.863 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:17.863 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.798790+0000", 2024-03-19T11:30:17.863 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012753100000000001 2024-03-19T11:30:17.863 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.863 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.863 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:17.863 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.798802+0000", 2024-03-19T11:30:17.863 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1654999999999999e-05 2024-03-19T11:30:17.863 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.863 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.863 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:17.863 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.798809+0000", 2024-03-19T11:30:17.863 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1289999999999998e-06 2024-03-19T11:30:17.863 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.863 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.864 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:17.864 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.798872+0000", 2024-03-19T11:30:17.864 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3462999999999999e-05 2024-03-19T11:30:17.864 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.864 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.864 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:17.864 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.798892+0000", 2024-03-19T11:30:17.864 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9823999999999999e-05 2024-03-19T11:30:17.864 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.864 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.864 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:17.864 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.800452+0000", 2024-03-19T11:30:17.864 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001559398 2024-03-19T11:30:17.864 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:17.864 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:17.865 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:17.865 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.800491+0000", 2024-03-19T11:30:17.865 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.9381000000000001e-05 2024-03-19T11:30:17.865 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.865 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:17.865 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.865 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:17.865 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:17.865 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:17.866 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:17.807+0000 7f1837407640 -1 osd.6 39 handle_osd_map injecting map crc failure 2024-03-19T11:30:17.868 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-19T11:30:17.868 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-03-19T11:30:17.868 INFO:tasks.rados.rados.0.smithi038.stdout:753: expect (ObjNum 60 snap 3 seq_num 60) 2024-03-19T11:30:17.869 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:17.880+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:17.941 INFO:tasks.rados.rados.0.smithi038.stdout:738: done (15 left) 2024-03-19T11:30:17.941 INFO:tasks.rados.rados.0.smithi038.stdout:739: done (14 left) 2024-03-19T11:30:17.941 INFO:tasks.rados.rados.0.smithi038.stdout:740: done (13 left) 2024-03-19T11:30:17.941 INFO:tasks.rados.rados.0.smithi038.stdout:741: done (12 left) 2024-03-19T11:30:17.941 INFO:tasks.rados.rados.0.smithi038.stdout:742: done (11 left) 2024-03-19T11:30:17.941 INFO:tasks.rados.rados.0.smithi038.stdout:743: done (10 left) 2024-03-19T11:30:17.941 INFO:tasks.rados.rados.0.smithi038.stdout:744: done (9 left) 2024-03-19T11:30:17.942 INFO:tasks.rados.rados.0.smithi038.stdout:745: done (8 left) 2024-03-19T11:30:17.942 INFO:tasks.rados.rados.0.smithi038.stdout:746: done (7 left) 2024-03-19T11:30:17.942 INFO:tasks.rados.rados.0.smithi038.stdout:747: done (6 left) 2024-03-19T11:30:17.942 INFO:tasks.rados.rados.0.smithi038.stdout:748: done (5 left) 2024-03-19T11:30:17.942 INFO:tasks.rados.rados.0.smithi038.stdout:749: done (4 left) 2024-03-19T11:30:17.942 INFO:tasks.rados.rados.0.smithi038.stdout:750: done (3 left) 2024-03-19T11:30:17.942 INFO:tasks.rados.rados.0.smithi038.stdout:751: done (2 left) 2024-03-19T11:30:17.942 INFO:tasks.rados.rados.0.smithi038.stdout:752: done (1 left) 2024-03-19T11:30:17.942 INFO:tasks.rados.rados.0.smithi038.stdout:753: done (0 left) 2024-03-19T11:30:17.942 INFO:tasks.rados.rados.0.smithi038.stdout:754: read oid 47 snap -1 2024-03-19T11:30:17.942 INFO:tasks.rados.rados.0.smithi038.stdout:754: expect (ObjNum 46 snap 0 seq_num 46) 2024-03-19T11:30:17.942 INFO:tasks.rados.rados.0.smithi038.stdout:755: write oid 46 current snap is 7 2024-03-19T11:30:17.942 INFO:tasks.rados.rados.0.smithi038.stdout:755: seq_num 85 ranges {628595=476048,1807440=632344,2934719=452474} 2024-03-19T11:30:17.950 INFO:tasks.rados.rados.0.smithi038.stdout:755: writing smithi03829856-46 from 628595 to 1104643 tid 1 2024-03-19T11:30:17.962 INFO:tasks.rados.rados.0.smithi038.stdout:755: writing smithi03829856-46 from 1807440 to 2439784 tid 2 2024-03-19T11:30:17.968 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:17.980+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:17.970 INFO:tasks.rados.rados.0.smithi038.stdout:755: writing smithi03829856-46 from 2934719 to 3387193 tid 3 2024-03-19T11:30:17.971 INFO:tasks.rados.rados.0.smithi038.stdout:756: snap_create 2024-03-19T11:30:17.971 INFO:tasks.rados.rados.0.smithi038.stdout:754: expect (ObjNum 46 snap 0 seq_num 46) 2024-03-19T11:30:18.032 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:18.032 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:30:18.032 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:30:18.032 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:18.046 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-19T11:30:18.062 INFO:tasks.rados.rados.0.smithi038.stdout:755: finishing write tid 1 to smithi03829856-46 2024-03-19T11:30:18.062 INFO:tasks.rados.rados.0.smithi038.stdout:755: finishing write tid 2 to smithi03829856-46 2024-03-19T11:30:18.062 INFO:tasks.rados.rados.0.smithi038.stdout:755: finishing write tid 3 to smithi03829856-46 2024-03-19T11:30:18.062 INFO:tasks.rados.rados.0.smithi038.stdout:755: finishing write tid 4 to smithi03829856-46 2024-03-19T11:30:18.062 INFO:tasks.rados.rados.0.smithi038.stdout:755: finishing write tid 5 to smithi03829856-46 2024-03-19T11:30:18.062 INFO:tasks.rados.rados.0.smithi038.stdout:755: oid 46 updating version 0 to 56 2024-03-19T11:30:18.062 INFO:tasks.rados.rados.0.smithi038.stdout:755: oid 46 version 56 is already newer than 54 2024-03-19T11:30:18.062 INFO:tasks.rados.rados.0.smithi038.stdout:755: oid 46 version 56 is already newer than 55 2024-03-19T11:30:18.062 INFO:tasks.rados.rados.0.smithi038.stdout:755: oid 46 version 56 is already newer than 53 2024-03-19T11:30:18.062 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 46 v 56 (ObjNum 85 snap 7 seq_num 85) dirty exists 2024-03-19T11:30:18.063 INFO:tasks.rados.rados.0.smithi038.stdout:755: left oid 46 (ObjNum 85 snap 7 seq_num 85) 2024-03-19T11:30:18.063 INFO:tasks.rados.rados.0.smithi038.stdout:754: done (2 left) 2024-03-19T11:30:18.063 INFO:tasks.rados.rados.0.smithi038.stdout:755: done (1 left) 2024-03-19T11:30:18.063 INFO:tasks.rados.rados.0.smithi038.stdout:756: done (0 left) 2024-03-19T11:30:18.063 INFO:tasks.rados.rados.0.smithi038.stdout:757: set_chunk oid 12 target oid 2024-03-19T11:30:18.074 INFO:tasks.rados.rados.0.smithi038.stdout:757: set_chunk oid 12 offset: 121344 length: 140800 target oid bba4463319d17dafd261b99651ecef74f1041be91d6b6cc86bb61a938e0e1818 offset: 0 snap 7 2024-03-19T11:30:18.074 INFO:tasks.rados.rados.0.smithi038.stdout:758: set_chunk oid 24 target oid 2024-03-19T11:30:18.074 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:18.084+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:18.084 INFO:tasks.rados.rados.0.smithi038.stdout:758: set_chunk oid 24 offset: 983040 length: 260096 target oid 7406da473b49beabda826a07c8f4c991034595a03ec98de318e81865678a987a offset: 0 snap 7 2024-03-19T11:30:18.084 INFO:tasks.rados.rados.0.smithi038.stdout:759: set_chunk oid 43 target oid 2024-03-19T11:30:18.097 INFO:tasks.rados.rados.0.smithi038.stdout:759: set_chunk oid 43 offset: 207872 length: 629760 target oid 3f6fba4c84be253b08a2cdff731f52d707dcb0afe6cafc88e72d771e6f671edd offset: 0 snap -1 2024-03-19T11:30:18.097 INFO:tasks.rados.rados.0.smithi038.stdout:760: read oid 31 snap -1 2024-03-19T11:30:18.097 INFO:tasks.rados.rados.0.smithi038.stdout:760: expect (ObjNum 56 snap 1 seq_num 56) 2024-03-19T11:30:18.097 INFO:tasks.rados.rados.0.smithi038.stdout:761: write (excl) oid 18 current snap is 8 2024-03-19T11:30:18.097 INFO:tasks.rados.rados.0.smithi038.stdout:757: finishing set_chunk to oid 12 2024-03-19T11:30:18.097 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 12 set_chunk returned error code -95 offset: 121344 length: 140800 2024-03-19T11:30:18.097 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 12 v 0 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-03-19T11:30:18.097 INFO:tasks.rados.rados.0.smithi038.stdout:761: seq_num 86 ranges {645402=551297,1749569=492355,2845789=599613} 2024-03-19T11:30:18.106 INFO:tasks.rados.rados.0.smithi038.stdout:761: writing smithi03829856-18 from 645402 to 1196699 tid 1 2024-03-19T11:30:18.113 INFO:tasks.rados.rados.0.smithi038.stdout:761: writing smithi03829856-18 from 1749569 to 2241924 tid 2 2024-03-19T11:30:18.122 INFO:tasks.rados.rados.0.smithi038.stdout:761: writing smithi03829856-18 from 2845789 to 3445402 tid 3 2024-03-19T11:30:18.122 INFO:tasks.rados.rados.0.smithi038.stdout:757: done (4 left) 2024-03-19T11:30:18.122 INFO:tasks.rados.rados.0.smithi038.stdout:762: set_chunk oid 44 target oid 2024-03-19T11:30:18.143 INFO:tasks.rados.rados.0.smithi038.stdout:762: set_chunk oid 44 offset: 192512 length: 1520640 target oid 98f4ba27fa68608c540dca9b03a4531ee556ff2734aa426809de93c45c84ab96 offset: 0 snap -1 2024-03-19T11:30:18.143 INFO:tasks.rados.rados.0.smithi038.stdout:763: tier_flush oid 15 2024-03-19T11:30:18.143 INFO:tasks.rados.rados.0.smithi038.stdout:763: tier_flush oid 15 snap -1 2024-03-19T11:30:18.143 INFO:tasks.rados.rados.0.smithi038.stdout:758: finishing set_chunk to oid 24 2024-03-19T11:30:18.144 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 24 set_chunk returned error code -95 offset: 983040 length: 260096 2024-03-19T11:30:18.144 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-03-19T11:30:18.144 INFO:tasks.rados.rados.0.smithi038.stdout:758: done (5 left) 2024-03-19T11:30:18.144 INFO:tasks.rados.rados.0.smithi038.stdout:764: read oid 33 snap -1 2024-03-19T11:30:18.144 INFO:tasks.rados.rados.0.smithi038.stdout:764: expect (ObjNum 32 snap 0 seq_num 32) 2024-03-19T11:30:18.144 INFO:tasks.rados.rados.0.smithi038.stdout:765: read oid 32 snap -1 2024-03-19T11:30:18.144 INFO:tasks.rados.rados.0.smithi038.stdout:765: expect (ObjNum 31 snap 0 seq_num 31) 2024-03-19T11:30:18.144 INFO:tasks.rados.rados.0.smithi038.stdout:766: read oid 47 snap 6 2024-03-19T11:30:18.144 INFO:tasks.rados.rados.0.smithi038.stdout:766: expect (ObjNum 46 snap 0 seq_num 46) 2024-03-19T11:30:18.144 INFO:tasks.rados.rados.0.smithi038.stdout:767: read oid 2 snap -1 2024-03-19T11:30:18.144 INFO:tasks.rados.rados.0.smithi038.stdout:767: expect (ObjNum 84 snap 7 seq_num 84) 2024-03-19T11:30:18.144 INFO:tasks.rados.rados.0.smithi038.stdout:768: write oid 28 current snap is 8 2024-03-19T11:30:18.144 INFO:tasks.rados.rados.0.smithi038.stdout:768: seq_num 87 ranges {662209=626546,1808051=636013,2873212=588997} 2024-03-19T11:30:18.150 INFO:tasks.rados.rados.0.smithi038.stdout:768: writing smithi03829856-28 from 662209 to 1288755 tid 1 2024-03-19T11:30:18.157 INFO:tasks.rados.rados.0.smithi038.stdout:768: writing smithi03829856-28 from 1808051 to 2444064 tid 2 2024-03-19T11:30:18.164 INFO:tasks.rados.rados.0.smithi038.stdout:768: writing smithi03829856-28 from 2873212 to 3462209 tid 3 2024-03-19T11:30:18.164 INFO:tasks.rados.rados.0.smithi038.stdout:769: write (excl) oid 14 current snap is 8 2024-03-19T11:30:18.164 INFO:tasks.rados.rados.0.smithi038.stdout:760: expect (ObjNum 56 snap 1 seq_num 56) 2024-03-19T11:30:18.170 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:18.183+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:18.209 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:18.209 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:30:18.209 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:30:18.210 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:30:18.210 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:18.222 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-19T11:30:18.271 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:18.283+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:18.308 INFO:tasks.rados.rados.0.smithi038.stdout:759: finishing set_chunk to oid 43 2024-03-19T11:30:18.308 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 43 v 50 (ObjNum 83 snap 7 seq_num 83) dirty exists 2024-03-19T11:30:18.309 INFO:tasks.rados.rados.0.smithi038.stdout:761: finishing write tid 1 to smithi03829856-18 2024-03-19T11:30:18.309 INFO:tasks.rados.rados.0.smithi038.stdout:761: finishing write tid 2 to smithi03829856-18 2024-03-19T11:30:18.309 INFO:tasks.rados.rados.0.smithi038.stdout:761: finishing write tid 3 to smithi03829856-18 2024-03-19T11:30:18.309 INFO:tasks.rados.rados.0.smithi038.stdout:761: finishing write tid 4 to smithi03829856-18 2024-03-19T11:30:18.309 INFO:tasks.rados.rados.0.smithi038.stdout:761: finishing write tid 5 to smithi03829856-18 2024-03-19T11:30:18.309 INFO:tasks.rados.rados.0.smithi038.stdout:761: oid 18 updating version 0 to 55 2024-03-19T11:30:18.309 INFO:tasks.rados.rados.0.smithi038.stdout:761: oid 18 version 55 is already newer than 53 2024-03-19T11:30:18.309 INFO:tasks.rados.rados.0.smithi038.stdout:761: oid 18 version 55 is already newer than 54 2024-03-19T11:30:18.309 INFO:tasks.rados.rados.0.smithi038.stdout:761: oid 18 updating version 55 to 56 2024-03-19T11:30:18.309 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 18 v 56 (ObjNum 86 snap 8 seq_num 86) dirty exists 2024-03-19T11:30:18.309 INFO:tasks.rados.rados.0.smithi038.stdout:761: left oid 18 (ObjNum 86 snap 8 seq_num 86) 2024-03-19T11:30:18.309 INFO:tasks.rados.rados.0.smithi038.stdout:769: seq_num 88 ranges {679016=701795,1866533=496024,2900635=402643} 2024-03-19T11:30:18.317 INFO:tasks.rados.rados.0.smithi038.stdout:769: writing smithi03829856-14 from 679016 to 1380811 tid 1 2024-03-19T11:30:18.324 INFO:tasks.rados.rados.0.smithi038.stdout:769: writing smithi03829856-14 from 1866533 to 2362557 tid 2 2024-03-19T11:30:18.330 INFO:tasks.rados.rados.0.smithi038.stdout:769: writing smithi03829856-14 from 2900635 to 3303278 tid 3 2024-03-19T11:30:18.330 INFO:tasks.rados.rados.0.smithi038.stdout:759: done (10 left) 2024-03-19T11:30:18.330 INFO:tasks.rados.rados.0.smithi038.stdout:760: done (9 left) 2024-03-19T11:30:18.330 INFO:tasks.rados.rados.0.smithi038.stdout:761: done (8 left) 2024-03-19T11:30:18.331 INFO:tasks.rados.rados.0.smithi038.stdout:770: read oid 49 snap 4 2024-03-19T11:30:18.331 INFO:tasks.rados.rados.0.smithi038.stdout:770: expect (ObjNum 48 snap 0 seq_num 48) 2024-03-19T11:30:18.331 INFO:tasks.rados.rados.0.smithi038.stdout:771: tier_promote oid 5 2024-03-19T11:30:18.331 INFO:tasks.rados.rados.0.smithi038.stdout:762: finishing set_chunk to oid 44 2024-03-19T11:30:18.331 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 44 set_chunk returned error code -95 offset: 192512 length: 1520640 2024-03-19T11:30:18.331 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 44 v 0 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-03-19T11:30:18.331 INFO:tasks.rados.rados.0.smithi038.stdout:766: expect (ObjNum 46 snap 0 seq_num 46) 2024-03-19T11:30:18.371 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:18.384+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:18.392 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:18.392 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:30:18.393 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:30:18.393 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:30:18.393 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.393 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:38 3.2 3:5595a68d:::07854d2fef297a06ba81685e660c332de36d5d18d546927d30daad6d7fda1541:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:18.393 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.307388+0000", 2024-03-19T11:30:18.393 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 9.0961192539999995, 2024-03-19T11:30:18.393 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0080668630000000005, 2024-03-19T11:30:18.393 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:18.393 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:18.393 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:18.393 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.393 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:18.393 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.307388+0000", 2024-03-19T11:30:18.393 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:18.394 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.394 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.394 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:18.394 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.307388+0000", 2024-03-19T11:30:18.394 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:18.394 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.394 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.394 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:18.394 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.307391+0000", 2024-03-19T11:30:18.394 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3039999999999999e-06 2024-03-19T11:30:18.394 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.394 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.394 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:18.395 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308352+0000", 2024-03-19T11:30:18.395 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00096137600000000001 2024-03-19T11:30:18.395 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.395 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.395 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:18.395 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308376+0000", 2024-03-19T11:30:18.395 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4196999999999999e-05 2024-03-19T11:30:18.395 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.395 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.395 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:18.395 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308387+0000", 2024-03-19T11:30:18.395 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1068e-05 2024-03-19T11:30:18.395 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.395 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.396 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:18.396 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308539+0000", 2024-03-19T11:30:18.396 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015186600000000001 2024-03-19T11:30:18.396 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.396 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.396 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:18.396 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308731+0000", 2024-03-19T11:30:18.396 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019146199999999999 2024-03-19T11:30:18.396 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.396 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.396 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:18.396 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309028+0000", 2024-03-19T11:30:18.396 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00029751799999999998 2024-03-19T11:30:18.396 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.396 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.397 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:18.397 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.314461+0000", 2024-03-19T11:30:18.397 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0054329570000000004 2024-03-19T11:30:18.397 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.397 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.397 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:18.397 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.315339+0000", 2024-03-19T11:30:18.397 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00087765200000000003 2024-03-19T11:30:18.397 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.397 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.397 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:18.397 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.315400+0000", 2024-03-19T11:30:18.397 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1650999999999996e-05 2024-03-19T11:30:18.397 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.397 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.398 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:18.398 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.315455+0000", 2024-03-19T11:30:18.398 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4812000000000003e-05 2024-03-19T11:30:18.398 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:18.398 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:18.398 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:18.398 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.398 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.398 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:41 3.2 3:525f009b:::a57e3dbafedf1ec1a7132467f167cfc4a4343cb541b36fc481376f0086aece71:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:18.398 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.308522+0000", 2024-03-19T11:30:18.398 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 9.0949853550000004, 2024-03-19T11:30:18.398 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0079565969999999993, 2024-03-19T11:30:18.399 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:18.399 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:18.399 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:18.399 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.399 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:18.399 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308522+0000", 2024-03-19T11:30:18.399 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:18.399 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.399 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.399 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:18.399 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308522+0000", 2024-03-19T11:30:18.399 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:18.399 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.399 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.399 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:18.400 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308524+0000", 2024-03-19T11:30:18.400 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.809e-06 2024-03-19T11:30:18.400 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.400 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.400 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:18.400 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308627+0000", 2024-03-19T11:30:18.400 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000103319 2024-03-19T11:30:18.400 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.400 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.400 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:18.400 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308643+0000", 2024-03-19T11:30:18.400 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5279000000000001e-05 2024-03-19T11:30:18.400 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.400 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.400 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:18.401 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308653+0000", 2024-03-19T11:30:18.401 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0557000000000001e-05 2024-03-19T11:30:18.401 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.401 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.401 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:18.401 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310052+0000", 2024-03-19T11:30:18.401 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0013986770000000001 2024-03-19T11:30:18.401 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.401 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.401 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:18.401 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310231+0000", 2024-03-19T11:30:18.401 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017891000000000001 2024-03-19T11:30:18.401 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.401 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.401 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:18.402 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310515+0000", 2024-03-19T11:30:18.402 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00028459299999999998 2024-03-19T11:30:18.402 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.402 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.402 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:18.402 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.315472+0000", 2024-03-19T11:30:18.402 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.004956874 2024-03-19T11:30:18.402 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.402 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.402 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:18.402 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.316378+0000", 2024-03-19T11:30:18.402 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00090556600000000001 2024-03-19T11:30:18.402 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.402 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.403 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:18.403 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.316435+0000", 2024-03-19T11:30:18.403 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7126000000000002e-05 2024-03-19T11:30:18.403 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.403 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.403 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:18.403 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.316479+0000", 2024-03-19T11:30:18.403 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3887000000000002e-05 2024-03-19T11:30:18.403 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:18.403 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:18.403 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:18.403 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.403 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.403 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:42 3.2 3:5aa75d1e:::b3c26d797de4ffcce4e862009e6b8eb8576a53cb86ea102ce51145c0038aeb97:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:18.404 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.308689+0000", 2024-03-19T11:30:18.404 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 9.0948185989999999, 2024-03-19T11:30:18.404 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0079809319999999996, 2024-03-19T11:30:18.404 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:18.404 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:18.404 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:18.404 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.404 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:18.404 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308689+0000", 2024-03-19T11:30:18.404 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:18.404 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.404 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.404 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:18.404 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308689+0000", 2024-03-19T11:30:18.404 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:18.405 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.405 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.405 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:18.405 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308690+0000", 2024-03-19T11:30:18.405 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.003e-06 2024-03-19T11:30:18.405 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.405 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.405 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:18.405 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309612+0000", 2024-03-19T11:30:18.405 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00092193199999999998 2024-03-19T11:30:18.405 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.405 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.405 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:18.405 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309645+0000", 2024-03-19T11:30:18.405 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2554e-05 2024-03-19T11:30:18.406 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.406 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.406 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:18.406 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309654+0000", 2024-03-19T11:30:18.406 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6209999999999999e-06 2024-03-19T11:30:18.406 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.406 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.406 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:18.406 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.311778+0000", 2024-03-19T11:30:18.406 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002123897 2024-03-19T11:30:18.406 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.406 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.406 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:18.406 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.311913+0000", 2024-03-19T11:30:18.407 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013471199999999999 2024-03-19T11:30:18.407 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.407 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.407 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:18.407 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.312123+0000", 2024-03-19T11:30:18.407 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00021064099999999999 2024-03-19T11:30:18.407 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.407 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.407 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:18.407 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.315479+0000", 2024-03-19T11:30:18.407 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033557230000000001 2024-03-19T11:30:18.407 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.407 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.407 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:18.408 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.316574+0000", 2024-03-19T11:30:18.408 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0010944990000000001 2024-03-19T11:30:18.408 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.408 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.408 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:18.408 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.316629+0000", 2024-03-19T11:30:18.408 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5169e-05 2024-03-19T11:30:18.408 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.408 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.408 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:18.408 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.316670+0000", 2024-03-19T11:30:18.408 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1180999999999998e-05 2024-03-19T11:30:18.408 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:18.408 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:18.409 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:18.409 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.409 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.409 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:43 3.2 3:6d82e5d5:::455c613336f373ae48b33af990f91f317265a0c5a920cc1b46533fdb41d8319c:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:18.409 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.309666+0000", 2024-03-19T11:30:18.409 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 9.0938413039999997, 2024-03-19T11:30:18.409 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077817820000000001, 2024-03-19T11:30:18.409 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:18.409 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:18.409 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:18.409 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.409 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:18.409 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309666+0000", 2024-03-19T11:30:18.410 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:18.410 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.410 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.410 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:18.410 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309666+0000", 2024-03-19T11:30:18.410 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:18.410 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.410 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.410 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:18.410 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309667+0000", 2024-03-19T11:30:18.410 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.5099999999999998e-07 2024-03-19T11:30:18.410 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.410 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.410 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:18.411 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310025+0000", 2024-03-19T11:30:18.411 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00035726900000000001 2024-03-19T11:30:18.411 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.411 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.411 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:18.411 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310093+0000", 2024-03-19T11:30:18.411 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8604999999999998e-05 2024-03-19T11:30:18.411 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.411 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.411 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:18.411 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310157+0000", 2024-03-19T11:30:18.411 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3688000000000004e-05 2024-03-19T11:30:18.411 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.411 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.411 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:18.412 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.313280+0000", 2024-03-19T11:30:18.412 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031230770000000001 2024-03-19T11:30:18.412 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.412 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.412 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:18.412 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.313429+0000", 2024-03-19T11:30:18.412 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000148763 2024-03-19T11:30:18.412 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.412 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.412 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:18.412 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.313700+0000", 2024-03-19T11:30:18.412 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00027178899999999998 2024-03-19T11:30:18.412 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.413 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.413 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:18.413 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.315489+0000", 2024-03-19T11:30:18.413 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001788848 2024-03-19T11:30:18.413 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.413 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.413 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:18.413 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.317381+0000", 2024-03-19T11:30:18.413 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0018913249999999999 2024-03-19T11:30:18.413 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.413 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.414 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:18.414 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.317411+0000", 2024-03-19T11:30:18.414 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0001e-05 2024-03-19T11:30:18.414 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.414 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.414 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:18.414 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.317448+0000", 2024-03-19T11:30:18.414 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7465999999999997e-05 2024-03-19T11:30:18.414 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:18.414 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:18.414 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:18.414 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.414 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.414 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:44 3.2 3:5f1d333e:::bfcdf551f6f2543327e9c601a4377e9a5ef396f5b97ae8331892d1bcf6b55a9d:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:18.415 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.310172+0000", 2024-03-19T11:30:18.415 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 9.0933351309999999, 2024-03-19T11:30:18.415 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0086762909999999992, 2024-03-19T11:30:18.415 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:18.415 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:18.415 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:18.415 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.415 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:18.415 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310172+0000", 2024-03-19T11:30:18.415 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:18.415 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.415 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.415 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:18.415 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310172+0000", 2024-03-19T11:30:18.416 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:18.416 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.416 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.416 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:18.416 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310173+0000", 2024-03-19T11:30:18.416 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0500000000000002e-07 2024-03-19T11:30:18.416 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.416 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.416 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:18.416 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310242+0000", 2024-03-19T11:30:18.416 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8239999999999997e-05 2024-03-19T11:30:18.416 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.416 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.416 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:18.416 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310253+0000", 2024-03-19T11:30:18.417 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0879e-05 2024-03-19T11:30:18.417 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.417 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.417 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:18.417 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310263+0000", 2024-03-19T11:30:18.417 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0045e-05 2024-03-19T11:30:18.417 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.417 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.417 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:18.417 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.314507+0000", 2024-03-19T11:30:18.417 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0042442060000000004 2024-03-19T11:30:18.417 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.417 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.417 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:18.418 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.314588+0000", 2024-03-19T11:30:18.418 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0912000000000001e-05 2024-03-19T11:30:18.418 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.418 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.418 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:18.418 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.314736+0000", 2024-03-19T11:30:18.418 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014782099999999999 2024-03-19T11:30:18.418 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.418 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.418 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:18.418 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.316529+0000", 2024-03-19T11:30:18.418 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00179327 2024-03-19T11:30:18.418 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.419 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.419 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:18.419 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.318785+0000", 2024-03-19T11:30:18.419 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022563660000000001 2024-03-19T11:30:18.419 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.419 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.419 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:18.419 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.318813+0000", 2024-03-19T11:30:18.419 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8206000000000001e-05 2024-03-19T11:30:18.419 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.419 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.419 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:18.419 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.318849+0000", 2024-03-19T11:30:18.419 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5441000000000002e-05 2024-03-19T11:30:18.419 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:18.420 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:18.420 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:18.420 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.420 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.420 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:45 3.2 3:78b4b14e:::e1ef0c0c594c36c534f3829406e07ffb60d57b70360f62977e0d545d8ca2bf18:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:18.420 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.310274+0000", 2024-03-19T11:30:18.420 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 9.0932336760000005, 2024-03-19T11:30:18.420 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094234769999999995, 2024-03-19T11:30:18.420 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:18.420 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:18.420 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:18.420 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.420 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:18.420 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310274+0000", 2024-03-19T11:30:18.420 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:18.421 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.421 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.421 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:18.421 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310274+0000", 2024-03-19T11:30:18.421 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:18.421 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.421 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.421 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:18.421 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310275+0000", 2024-03-19T11:30:18.421 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2099999999999996e-07 2024-03-19T11:30:18.421 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.421 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.421 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:18.421 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310329+0000", 2024-03-19T11:30:18.421 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4440000000000001e-05 2024-03-19T11:30:18.422 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.422 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.422 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:18.422 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310340+0000", 2024-03-19T11:30:18.422 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1321999999999999e-05 2024-03-19T11:30:18.422 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.422 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.422 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:18.422 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310349+0000", 2024-03-19T11:30:18.422 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6789999999999994e-06 2024-03-19T11:30:18.422 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.422 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.422 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:18.422 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.315512+0000", 2024-03-19T11:30:18.423 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0051629689999999999 2024-03-19T11:30:18.423 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.423 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.423 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:18.423 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.315581+0000", 2024-03-19T11:30:18.423 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9400999999999995e-05 2024-03-19T11:30:18.423 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.423 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.423 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:18.423 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.315759+0000", 2024-03-19T11:30:18.423 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000177491 2024-03-19T11:30:18.423 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.423 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.423 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:18.424 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.317063+0000", 2024-03-19T11:30:18.424 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0013040630000000001 2024-03-19T11:30:18.424 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.424 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.424 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:18.424 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.319635+0000", 2024-03-19T11:30:18.424 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025715880000000001 2024-03-19T11:30:18.424 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.424 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.424 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:18.424 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.319663+0000", 2024-03-19T11:30:18.424 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8554000000000001e-05 2024-03-19T11:30:18.424 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.424 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.424 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:18.425 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.319697+0000", 2024-03-19T11:30:18.425 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4348999999999999e-05 2024-03-19T11:30:18.425 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:18.425 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:18.425 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:18.425 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.425 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.425 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:55 3.2 3:525f009b:::a57e3dbafedf1ec1a7132467f167cfc4a4343cb541b36fc481376f0086aece71:head [read 0~121882 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:18.425 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.761904+0000", 2024-03-19T11:30:18.425 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.6416031799999997, 2024-03-19T11:30:18.425 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010829357, 2024-03-19T11:30:18.425 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:18.425 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:18.425 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:18.425 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.425 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:18.426 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761904+0000", 2024-03-19T11:30:18.426 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:18.426 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.426 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.426 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:18.426 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761904+0000", 2024-03-19T11:30:18.426 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:18.426 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.426 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.426 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:18.426 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761905+0000", 2024-03-19T11:30:18.426 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4900000000000005e-07 2024-03-19T11:30:18.426 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.426 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.427 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:18.427 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761917+0000", 2024-03-19T11:30:18.427 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1806e-05 2024-03-19T11:30:18.427 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.427 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.427 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:18.427 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761934+0000", 2024-03-19T11:30:18.427 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6509000000000001e-05 2024-03-19T11:30:18.427 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.427 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.427 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:18.427 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761941+0000", 2024-03-19T11:30:18.427 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8110000000000003e-06 2024-03-19T11:30:18.427 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.428 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.428 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:18.428 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762115+0000", 2024-03-19T11:30:18.428 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017358200000000001 2024-03-19T11:30:18.428 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.428 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.428 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:18.428 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762120+0000", 2024-03-19T11:30:18.428 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3630000000000002e-06 2024-03-19T11:30:18.428 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.428 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.428 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:18.428 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.765567+0000", 2024-03-19T11:30:18.428 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034469990000000001 2024-03-19T11:30:18.429 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.429 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.429 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:18.429 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.765665+0000", 2024-03-19T11:30:18.429 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7476e-05 2024-03-19T11:30:18.429 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.429 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.429 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:18.429 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.772734+0000", 2024-03-19T11:30:18.429 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0070690620000000001 2024-03-19T11:30:18.429 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:18.429 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:18.429 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:18.429 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.429 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.430 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:56 3.2 3:5aa75d1e:::b3c26d797de4ffcce4e862009e6b8eb8576a53cb86ea102ce51145c0038aeb97:head [read 0~524288 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:18.430 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.761958+0000", 2024-03-19T11:30:18.430 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.641549779, 2024-03-19T11:30:18.430 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012078836000000001, 2024-03-19T11:30:18.430 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:18.430 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:18.430 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:18.430 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.430 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:18.430 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761958+0000", 2024-03-19T11:30:18.430 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:18.430 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.431 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.431 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:18.431 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761958+0000", 2024-03-19T11:30:18.431 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:18.431 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.431 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.431 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:18.431 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761958+0000", 2024-03-19T11:30:18.431 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9900000000000001e-07 2024-03-19T11:30:18.431 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.431 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.431 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:18.431 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761961+0000", 2024-03-19T11:30:18.431 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9280000000000002e-06 2024-03-19T11:30:18.432 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.432 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.432 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:18.432 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761969+0000", 2024-03-19T11:30:18.432 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1440000000000003e-06 2024-03-19T11:30:18.432 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.432 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.432 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:18.432 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761976+0000", 2024-03-19T11:30:18.432 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1020000000000002e-06 2024-03-19T11:30:18.432 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.432 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.432 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:18.432 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762144+0000", 2024-03-19T11:30:18.432 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016818200000000001 2024-03-19T11:30:18.433 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.433 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.433 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:18.433 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762148+0000", 2024-03-19T11:30:18.433 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0969999999999999e-06 2024-03-19T11:30:18.433 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.433 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.433 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:18.433 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.772788+0000", 2024-03-19T11:30:18.433 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010640124000000001 2024-03-19T11:30:18.433 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.433 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.433 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:18.434 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.772898+0000", 2024-03-19T11:30:18.434 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000110512 2024-03-19T11:30:18.434 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.434 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.434 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:18.434 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.774037+0000", 2024-03-19T11:30:18.434 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011382479999999999 2024-03-19T11:30:18.434 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:18.434 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:18.434 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:18.434 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.434 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.434 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:57 3.2 3:6d82e5d5:::455c613336f373ae48b33af990f91f317265a0c5a920cc1b46533fdb41d8319c:head [read 0~380357 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:18.434 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.761987+0000", 2024-03-19T11:30:18.435 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.641520817, 2024-03-19T11:30:18.435 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012691991999999999, 2024-03-19T11:30:18.435 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:18.435 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:18.435 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:18.435 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.435 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:18.435 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761987+0000", 2024-03-19T11:30:18.435 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:18.435 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.435 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.435 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:18.435 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761987+0000", 2024-03-19T11:30:18.435 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:18.436 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.436 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.436 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:18.436 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761987+0000", 2024-03-19T11:30:18.436 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4099999999999999e-07 2024-03-19T11:30:18.436 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.436 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.436 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:18.436 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761990+0000", 2024-03-19T11:30:18.436 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3829999999999999e-06 2024-03-19T11:30:18.436 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.436 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.436 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:18.436 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762000+0000", 2024-03-19T11:30:18.437 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0229e-05 2024-03-19T11:30:18.437 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.437 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.437 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:18.437 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762006+0000", 2024-03-19T11:30:18.437 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7069999999999997e-06 2024-03-19T11:30:18.437 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.437 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.437 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:18.437 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762171+0000", 2024-03-19T11:30:18.437 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000165341 2024-03-19T11:30:18.437 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.437 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.438 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:18.438 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762176+0000", 2024-03-19T11:30:18.438 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6879999999999998e-06 2024-03-19T11:30:18.438 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.438 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.438 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:18.438 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.774081+0000", 2024-03-19T11:30:18.438 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011905094999999999 2024-03-19T11:30:18.438 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.438 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.438 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:18.438 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.774169+0000", 2024-03-19T11:30:18.438 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8089000000000003e-05 2024-03-19T11:30:18.438 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.439 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.439 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:18.439 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.774679+0000", 2024-03-19T11:30:18.439 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00051001900000000001 2024-03-19T11:30:18.439 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:18.439 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:18.439 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:18.439 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.439 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.439 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:58 3.2 3:5f1d333e:::bfcdf551f6f2543327e9c601a4377e9a5ef396f5b97ae8331892d1bcf6b55a9d:head [read 0~161292 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:18.439 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.762024+0000", 2024-03-19T11:30:18.439 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.6414838679999999, 2024-03-19T11:30:18.440 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013495629, 2024-03-19T11:30:18.440 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:18.440 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:18.440 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:18.440 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.440 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:18.440 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762024+0000", 2024-03-19T11:30:18.440 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:18.440 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.440 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.440 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:18.440 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762024+0000", 2024-03-19T11:30:18.440 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:18.440 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.441 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.441 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:18.441 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762024+0000", 2024-03-19T11:30:18.441 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9999999999999997e-07 2024-03-19T11:30:18.441 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.441 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.441 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:18.441 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762027+0000", 2024-03-19T11:30:18.441 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0410000000000002e-06 2024-03-19T11:30:18.441 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.441 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.441 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:18.441 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762037+0000", 2024-03-19T11:30:18.441 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1020000000000003e-06 2024-03-19T11:30:18.441 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.442 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.442 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:18.442 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762043+0000", 2024-03-19T11:30:18.442 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4409999999999998e-06 2024-03-19T11:30:18.442 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.442 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.442 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:18.442 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762198+0000", 2024-03-19T11:30:18.442 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015540000000000001 2024-03-19T11:30:18.442 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.442 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.442 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:18.442 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762203+0000", 2024-03-19T11:30:18.442 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5109999999999997e-06 2024-03-19T11:30:18.443 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.443 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.443 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:18.443 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.774710+0000", 2024-03-19T11:30:18.443 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012507452 2024-03-19T11:30:18.443 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.443 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.443 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:18.443 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.774754+0000", 2024-03-19T11:30:18.443 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3869999999999998e-05 2024-03-19T11:30:18.443 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.443 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.443 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:18.443 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.775519+0000", 2024-03-19T11:30:18.444 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00076521200000000003 2024-03-19T11:30:18.444 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:18.444 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:18.444 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:18.444 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.444 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.444 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:59 3.2 3:78b4b14e:::e1ef0c0c594c36c534f3829406e07ffb60d57b70360f62977e0d545d8ca2bf18:head [read 0~149977 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:18.444 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.762054+0000", 2024-03-19T11:30:18.444 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.641453437, 2024-03-19T11:30:18.444 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015354483, 2024-03-19T11:30:18.444 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:18.444 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:18.444 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:18.444 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.444 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:18.445 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762054+0000", 2024-03-19T11:30:18.445 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:18.445 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.445 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.445 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:18.445 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762054+0000", 2024-03-19T11:30:18.445 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:18.445 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.445 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.445 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:18.445 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762055+0000", 2024-03-19T11:30:18.445 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.32e-07 2024-03-19T11:30:18.445 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.445 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.446 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:18.446 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762058+0000", 2024-03-19T11:30:18.446 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9189999999999999e-06 2024-03-19T11:30:18.446 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.446 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.446 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:18.446 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762065+0000", 2024-03-19T11:30:18.446 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2409999999999998e-06 2024-03-19T11:30:18.446 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.446 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.446 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:18.446 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762070+0000", 2024-03-19T11:30:18.446 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6910000000000002e-06 2024-03-19T11:30:18.446 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.447 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.447 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:18.447 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762227+0000", 2024-03-19T11:30:18.447 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015606999999999999 2024-03-19T11:30:18.447 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.447 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.447 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:18.447 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762232+0000", 2024-03-19T11:30:18.447 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3029999999999999e-06 2024-03-19T11:30:18.447 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.447 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.447 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:18.447 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.776984+0000", 2024-03-19T11:30:18.447 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014752487999999999 2024-03-19T11:30:18.448 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.448 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.448 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:18.448 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.777030+0000", 2024-03-19T11:30:18.448 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5359999999999999e-05 2024-03-19T11:30:18.448 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.448 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.448 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:18.448 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.777409+0000", 2024-03-19T11:30:18.448 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037897899999999999 2024-03-19T11:30:18.448 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:18.448 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:18.448 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:18.448 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.448 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.448 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1020 3.2 3:59279e72:::24df73eb14d5465f466b1889d4952f96785c09a615e42a99c9b768b4b3347e65:head [write 0~207872] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:18.449 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.773460+0000", 2024-03-19T11:30:18.449 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.6300475849999998, 2024-03-19T11:30:18.449 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0070966579999999996, 2024-03-19T11:30:18.449 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:18.449 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:18.449 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:18.449 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:18.449 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:18.449 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1020 2024-03-19T11:30:18.449 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.449 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:18.449 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.449 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:18.449 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.773460+0000", 2024-03-19T11:30:18.449 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:18.450 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.450 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.450 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:18.450 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.773460+0000", 2024-03-19T11:30:18.450 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:18.450 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.450 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.450 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:18.450 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.773462+0000", 2024-03-19T11:30:18.450 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.984e-06 2024-03-19T11:30:18.450 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.450 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.450 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:18.450 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.773537+0000", 2024-03-19T11:30:18.451 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4726000000000002e-05 2024-03-19T11:30:18.451 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.451 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.451 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:18.451 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.773550+0000", 2024-03-19T11:30:18.451 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3721e-05 2024-03-19T11:30:18.451 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.451 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.451 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:18.451 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.773561+0000", 2024-03-19T11:30:18.451 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1017e-05 2024-03-19T11:30:18.451 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.451 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.451 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:18.451 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.775560+0000", 2024-03-19T11:30:18.452 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0019981510000000001 2024-03-19T11:30:18.452 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.452 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.452 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:18.452 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.775716+0000", 2024-03-19T11:30:18.452 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000156265 2024-03-19T11:30:18.452 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.452 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.452 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:18.452 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.775878+0000", 2024-03-19T11:30:18.452 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016257899999999999 2024-03-19T11:30:18.452 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.452 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.452 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:18.452 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.777476+0000", 2024-03-19T11:30:18.453 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001597465 2024-03-19T11:30:18.453 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.453 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.453 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:18.453 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.780481+0000", 2024-03-19T11:30:18.453 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030052450000000001 2024-03-19T11:30:18.453 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.453 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.453 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:18.453 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.780532+0000", 2024-03-19T11:30:18.453 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0484e-05 2024-03-19T11:30:18.453 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.453 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.453 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:18.453 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.780557+0000", 2024-03-19T11:30:18.454 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5021e-05 2024-03-19T11:30:18.454 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:18.454 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:18.454 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:18.454 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.454 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.454 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1067 4.0 4:01018511:::smithi03829856-19:head [write 477332~766101] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:18.454 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.950506+0000", 2024-03-19T11:30:18.454 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.453001725, 2024-03-19T11:30:18.454 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016279426, 2024-03-19T11:30:18.454 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:18.454 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:18.454 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:18.454 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:18.455 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:18.455 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1067 2024-03-19T11:30:18.455 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.455 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:18.455 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.455 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:18.455 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950506+0000", 2024-03-19T11:30:18.455 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:18.455 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.455 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.455 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:18.455 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950506+0000", 2024-03-19T11:30:18.455 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:18.455 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.455 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.456 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:18.456 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950509+0000", 2024-03-19T11:30:18.456 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.021e-06 2024-03-19T11:30:18.456 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.456 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.456 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:18.456 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950892+0000", 2024-03-19T11:30:18.456 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00038263300000000001 2024-03-19T11:30:18.456 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.456 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.456 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:18.456 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950912+0000", 2024-03-19T11:30:18.456 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0539000000000001e-05 2024-03-19T11:30:18.456 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.456 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.457 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:18.457 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950922+0000", 2024-03-19T11:30:18.457 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.577e-06 2024-03-19T11:30:18.457 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.457 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.457 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:18.457 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950982+0000", 2024-03-19T11:30:18.457 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0760000000000001e-05 2024-03-19T11:30:18.457 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.457 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.457 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:18.457 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.951058+0000", 2024-03-19T11:30:18.457 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5078000000000002e-05 2024-03-19T11:30:18.457 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.457 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.458 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:18.458 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.951173+0000", 2024-03-19T11:30:18.458 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011516200000000001 2024-03-19T11:30:18.458 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.458 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.458 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:18.458 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.955064+0000", 2024-03-19T11:30:18.458 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003890896 2024-03-19T11:30:18.458 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.458 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.458 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:18.458 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966670+0000", 2024-03-19T11:30:18.458 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011606795 2024-03-19T11:30:18.458 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.459 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.459 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:18.459 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966731+0000", 2024-03-19T11:30:18.459 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0528000000000001e-05 2024-03-19T11:30:18.459 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.459 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.459 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:18.459 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966785+0000", 2024-03-19T11:30:18.459 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4437000000000001e-05 2024-03-19T11:30:18.459 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:18.459 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:18.459 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:18.459 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.459 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.459 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1068 4.0 4:01018511:::smithi03829856-19:head [write 1681102~474010] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:18.460 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.956456+0000", 2024-03-19T11:30:18.461 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.44705116, 2024-03-19T11:30:18.461 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010432765, 2024-03-19T11:30:18.461 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:18.461 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:18.461 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:18.461 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:18.461 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:18.461 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1068 2024-03-19T11:30:18.461 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.462 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:18.462 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.462 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:18.462 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956456+0000", 2024-03-19T11:30:18.462 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:18.462 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.462 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.462 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:18.462 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956456+0000", 2024-03-19T11:30:18.462 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:18.462 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.462 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.462 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:18.463 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956458+0000", 2024-03-19T11:30:18.463 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.742e-06 2024-03-19T11:30:18.463 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.463 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.463 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:18.463 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956655+0000", 2024-03-19T11:30:18.463 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019682200000000001 2024-03-19T11:30:18.463 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.463 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.463 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:18.463 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956669+0000", 2024-03-19T11:30:18.463 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3978000000000001e-05 2024-03-19T11:30:18.463 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.463 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.463 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:18.464 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956678+0000", 2024-03-19T11:30:18.464 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5029999999999997e-06 2024-03-19T11:30:18.464 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.464 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.464 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:18.464 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956729+0000", 2024-03-19T11:30:18.464 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1765999999999999e-05 2024-03-19T11:30:18.464 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.464 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.464 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:18.464 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956795+0000", 2024-03-19T11:30:18.464 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5701e-05 2024-03-19T11:30:18.464 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.465 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.465 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:18.465 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956886+0000", 2024-03-19T11:30:18.465 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1226999999999999e-05 2024-03-19T11:30:18.465 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.465 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.465 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:18.465 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.958609+0000", 2024-03-19T11:30:18.465 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017227049999999999 2024-03-19T11:30:18.465 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.465 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.465 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:18.465 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966837+0000", 2024-03-19T11:30:18.465 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0082276439999999992 2024-03-19T11:30:18.465 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.466 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.466 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:18.466 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966866+0000", 2024-03-19T11:30:18.466 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9323999999999999e-05 2024-03-19T11:30:18.466 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.466 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.466 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:18.466 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966889+0000", 2024-03-19T11:30:18.466 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3353000000000002e-05 2024-03-19T11:30:18.466 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:18.466 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:18.466 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:18.466 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.466 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.466 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.0.13:50 3.2 3:5595a68d:::07854d2fef297a06ba81685e660c332de36d5d18d546927d30daad6d7fda1541:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:18.467 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.166341+0000", 2024-03-19T11:30:18.467 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.237166294, 2024-03-19T11:30:18.467 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010067153000000001, 2024-03-19T11:30:18.467 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:18.467 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:18.467 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:18.467 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.467 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:18.467 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166341+0000", 2024-03-19T11:30:18.467 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:18.467 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.467 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.467 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:18.467 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166341+0000", 2024-03-19T11:30:18.467 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:18.468 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.468 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.468 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:18.468 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166344+0000", 2024-03-19T11:30:18.468 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1370000000000002e-06 2024-03-19T11:30:18.468 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.468 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.468 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:18.468 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166828+0000", 2024-03-19T11:30:18.468 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00048343600000000001 2024-03-19T11:30:18.468 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.468 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.468 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:18.469 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166857+0000", 2024-03-19T11:30:18.469 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8880000000000001e-05 2024-03-19T11:30:18.469 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.469 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.469 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:18.469 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166871+0000", 2024-03-19T11:30:18.469 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4384e-05 2024-03-19T11:30:18.469 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.469 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.469 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:18.469 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167222+0000", 2024-03-19T11:30:18.469 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00035059000000000002 2024-03-19T11:30:18.469 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.469 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.469 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:18.470 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167334+0000", 2024-03-19T11:30:18.470 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000112663 2024-03-19T11:30:18.470 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.470 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.470 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:18.470 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167533+0000", 2024-03-19T11:30:18.470 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019901399999999999 2024-03-19T11:30:18.470 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.470 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.470 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:18.470 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176237+0000", 2024-03-19T11:30:18.470 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0087039890000000005 2024-03-19T11:30:18.470 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.470 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.471 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:18.471 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176260+0000", 2024-03-19T11:30:18.471 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2889000000000001e-05 2024-03-19T11:30:18.471 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.471 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.471 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:18.471 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176343+0000", 2024-03-19T11:30:18.471 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2651000000000005e-05 2024-03-19T11:30:18.471 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.471 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.471 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:18.471 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176408+0000", 2024-03-19T11:30:18.471 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5519999999999996e-05 2024-03-19T11:30:18.471 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:18.471 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:18.472 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:18.472 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.472 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.472 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.0.13:51 3.2 3:587bed34:::13074eb9c7a477811d595d489af192576515e7c716c0b501b140fcc4687b768b:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:18.472 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.166912+0000", 2024-03-19T11:30:18.472 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.23659549099999999, 2024-03-19T11:30:18.472 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0097465639999999992, 2024-03-19T11:30:18.472 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:18.472 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:18.472 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:18.472 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.472 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:18.472 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166912+0000", 2024-03-19T11:30:18.472 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:18.472 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.473 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.473 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:18.473 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166912+0000", 2024-03-19T11:30:18.473 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:18.473 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.473 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.473 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:18.473 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166914+0000", 2024-03-19T11:30:18.473 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.468e-06 2024-03-19T11:30:18.473 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.473 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.473 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:18.473 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167000+0000", 2024-03-19T11:30:18.473 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6311999999999996e-05 2024-03-19T11:30:18.474 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.474 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.474 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:18.474 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167011+0000", 2024-03-19T11:30:18.474 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.153e-05 2024-03-19T11:30:18.474 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.474 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.474 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:18.474 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167019+0000", 2024-03-19T11:30:18.474 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7970000000000001e-06 2024-03-19T11:30:18.474 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.474 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.474 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:18.474 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167864+0000", 2024-03-19T11:30:18.474 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00084482600000000002 2024-03-19T11:30:18.475 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.475 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.475 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:18.475 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.169110+0000", 2024-03-19T11:30:18.475 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0012456399999999999 2024-03-19T11:30:18.475 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.475 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.475 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:18.475 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.169467+0000", 2024-03-19T11:30:18.475 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00035692100000000001 2024-03-19T11:30:18.475 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.475 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.475 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:18.475 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176426+0000", 2024-03-19T11:30:18.475 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0069589600000000001 2024-03-19T11:30:18.476 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.476 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.476 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:18.476 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176524+0000", 2024-03-19T11:30:18.476 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8183000000000005e-05 2024-03-19T11:30:18.476 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.476 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.476 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:18.476 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176603+0000", 2024-03-19T11:30:18.476 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9035999999999993e-05 2024-03-19T11:30:18.476 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.476 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.476 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:18.476 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176659+0000", 2024-03-19T11:30:18.477 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5890999999999997e-05 2024-03-19T11:30:18.477 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:18.477 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:18.477 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:18.477 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.477 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.477 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.0.13:52 3.2 3:46a26328:::b2b262672c1f15ae09b098a7a21c74662e4816397daed8ca08fb83d0cd3a0cac:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:18.477 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.167033+0000", 2024-03-19T11:30:18.477 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.23647468199999999, 2024-03-19T11:30:18.477 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010113297, 2024-03-19T11:30:18.477 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:18.477 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:18.477 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:18.477 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.477 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:18.478 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167033+0000", 2024-03-19T11:30:18.478 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:18.478 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.478 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.478 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:18.478 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167033+0000", 2024-03-19T11:30:18.478 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:18.478 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.478 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.478 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:18.478 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167034+0000", 2024-03-19T11:30:18.478 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6599999999999996e-07 2024-03-19T11:30:18.478 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.478 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.479 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:18.479 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167092+0000", 2024-03-19T11:30:18.479 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7989999999999999e-05 2024-03-19T11:30:18.479 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.479 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.479 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:18.479 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167100+0000", 2024-03-19T11:30:18.479 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9670000000000001e-06 2024-03-19T11:30:18.479 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.479 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.479 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:18.479 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167108+0000", 2024-03-19T11:30:18.479 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3559999999999993e-06 2024-03-19T11:30:18.479 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.479 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.480 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:18.480 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.170462+0000", 2024-03-19T11:30:18.480 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033540229999999998 2024-03-19T11:30:18.480 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.480 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.480 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:18.480 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.171193+0000", 2024-03-19T11:30:18.480 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00073086899999999998 2024-03-19T11:30:18.480 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.480 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.480 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:18.480 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.171558+0000", 2024-03-19T11:30:18.480 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00036518899999999998 2024-03-19T11:30:18.480 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.481 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.481 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:18.481 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176440+0000", 2024-03-19T11:30:18.481 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0048817260000000003 2024-03-19T11:30:18.481 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.481 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.481 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:18.481 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177053+0000", 2024-03-19T11:30:18.481 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00061337100000000003 2024-03-19T11:30:18.481 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.481 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.481 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:18.481 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177096+0000", 2024-03-19T11:30:18.481 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2846999999999998e-05 2024-03-19T11:30:18.481 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.481 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.482 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:18.482 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177146+0000", 2024-03-19T11:30:18.482 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0293000000000001e-05 2024-03-19T11:30:18.482 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:18.482 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:18.482 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:18.482 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.482 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.482 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.0.13:54 3.2 3:7b614c59:::a7a06fca755baa7858207cb4b067fffe8c25e8183433b950d6879f6969d41b0b:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:18.482 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.167117+0000", 2024-03-19T11:30:18.482 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.23639017400000001, 2024-03-19T11:30:18.482 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010796765999999999, 2024-03-19T11:30:18.482 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:18.482 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:18.483 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:18.483 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.483 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:18.483 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167117+0000", 2024-03-19T11:30:18.483 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:18.483 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.483 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.483 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:18.483 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167117+0000", 2024-03-19T11:30:18.483 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:18.483 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.483 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.483 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:18.483 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167118+0000", 2024-03-19T11:30:18.484 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2600000000000002e-07 2024-03-19T11:30:18.484 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.484 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.484 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:18.484 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167150+0000", 2024-03-19T11:30:18.484 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1576999999999999e-05 2024-03-19T11:30:18.484 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.484 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.484 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:18.484 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167157+0000", 2024-03-19T11:30:18.484 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2779999999999996e-06 2024-03-19T11:30:18.484 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.484 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.484 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:18.485 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167163+0000", 2024-03-19T11:30:18.485 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0120000000000002e-06 2024-03-19T11:30:18.485 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.485 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.485 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:18.485 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173324+0000", 2024-03-19T11:30:18.485 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0061614149999999999 2024-03-19T11:30:18.485 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.485 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.485 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:18.485 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173489+0000", 2024-03-19T11:30:18.485 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016438999999999999 2024-03-19T11:30:18.485 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.485 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.485 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:18.486 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173774+0000", 2024-03-19T11:30:18.486 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00028548000000000001 2024-03-19T11:30:18.486 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.486 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.486 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:18.486 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177668+0000", 2024-03-19T11:30:18.486 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038942450000000002 2024-03-19T11:30:18.486 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.486 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.486 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:18.486 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177792+0000", 2024-03-19T11:30:18.486 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012398199999999999 2024-03-19T11:30:18.486 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.486 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.487 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:18.487 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177861+0000", 2024-03-19T11:30:18.487 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8633999999999994e-05 2024-03-19T11:30:18.487 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.487 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.487 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:18.487 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177914+0000", 2024-03-19T11:30:18.487 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3226999999999997e-05 2024-03-19T11:30:18.487 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:18.487 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:18.487 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:18.487 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.487 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.487 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1230 4.0 4:0dc3b91e:::smithi03829856-28:head [write 662209~626546] snapc 9=[9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:18.487 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.170758+0000", 2024-03-19T11:30:18.488 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.23274945999999999, 2024-03-19T11:30:18.488 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011180066000000001, 2024-03-19T11:30:18.488 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:18.488 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:18.488 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:18.488 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:18.488 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:18.488 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1230 2024-03-19T11:30:18.488 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.488 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:18.488 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.488 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:18.488 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.170758+0000", 2024-03-19T11:30:18.488 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:18.488 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.489 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.489 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:18.489 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.170758+0000", 2024-03-19T11:30:18.489 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:18.489 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.489 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.489 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:18.489 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.170762+0000", 2024-03-19T11:30:18.489 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4960000000000001e-06 2024-03-19T11:30:18.489 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.489 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.489 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:18.489 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.171449+0000", 2024-03-19T11:30:18.489 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00068700499999999997 2024-03-19T11:30:18.489 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.490 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.490 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:18.490 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.171467+0000", 2024-03-19T11:30:18.490 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8389e-05 2024-03-19T11:30:18.490 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.490 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.490 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:18.490 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.171482+0000", 2024-03-19T11:30:18.490 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4619e-05 2024-03-19T11:30:18.490 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.490 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.490 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:18.490 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.171566+0000", 2024-03-19T11:30:18.490 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4788000000000002e-05 2024-03-19T11:30:18.491 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.491 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.491 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:18.491 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.171685+0000", 2024-03-19T11:30:18.491 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011861700000000001 2024-03-19T11:30:18.491 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.491 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.491 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:18.491 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.171835+0000", 2024-03-19T11:30:18.491 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014956299999999999 2024-03-19T11:30:18.491 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.491 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.491 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:18.492 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.181520+0000", 2024-03-19T11:30:18.492 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0096855750000000001 2024-03-19T11:30:18.492 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.492 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.492 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:18.492 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.181640+0000", 2024-03-19T11:30:18.492 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011966799999999999 2024-03-19T11:30:18.492 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.492 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.492 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:18.492 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.181811+0000", 2024-03-19T11:30:18.492 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017062699999999999 2024-03-19T11:30:18.492 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.492 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.492 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:18.493 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.181938+0000", 2024-03-19T11:30:18.493 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012771899999999999 2024-03-19T11:30:18.493 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:18.493 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:18.493 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:18.493 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.493 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.493 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1231 4.0 4:0dc3b91e:::smithi03829856-28:head [write 1808051~636013] snapc 9=[9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:18.493 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.173016+0000", 2024-03-19T11:30:18.493 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.23049161200000001, 2024-03-19T11:30:18.493 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013973719000000001, 2024-03-19T11:30:18.493 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:18.493 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:18.493 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:18.493 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:18.494 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:18.494 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1231 2024-03-19T11:30:18.494 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.494 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:18.494 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.494 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:18.494 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173016+0000", 2024-03-19T11:30:18.494 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:18.494 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.494 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.494 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:18.494 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173016+0000", 2024-03-19T11:30:18.494 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:18.495 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.495 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.495 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:18.495 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173019+0000", 2024-03-19T11:30:18.495 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.253e-06 2024-03-19T11:30:18.495 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.495 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.495 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:18.495 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176143+0000", 2024-03-19T11:30:18.495 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031240930000000001 2024-03-19T11:30:18.495 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.495 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.495 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:18.495 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176177+0000", 2024-03-19T11:30:18.495 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3237e-05 2024-03-19T11:30:18.496 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.496 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.496 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:18.496 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176196+0000", 2024-03-19T11:30:18.496 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9358e-05 2024-03-19T11:30:18.496 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.496 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.496 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:18.496 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176493+0000", 2024-03-19T11:30:18.496 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00029743899999999998 2024-03-19T11:30:18.496 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.496 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.496 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:18.496 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176786+0000", 2024-03-19T11:30:18.496 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00029259000000000002 2024-03-19T11:30:18.497 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.497 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.497 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:18.497 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177250+0000", 2024-03-19T11:30:18.497 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00046376000000000002 2024-03-19T11:30:18.497 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.497 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.497 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:18.497 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.182511+0000", 2024-03-19T11:30:18.497 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0052609019999999996 2024-03-19T11:30:18.497 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.497 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.497 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:18.497 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.186896+0000", 2024-03-19T11:30:18.497 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0043857009999999997 2024-03-19T11:30:18.498 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.498 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.498 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:18.498 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.186948+0000", 2024-03-19T11:30:18.498 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1653000000000002e-05 2024-03-19T11:30:18.498 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:18.498 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:18.498 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:18.498 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.186990+0000", 2024-03-19T11:30:18.498 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1733000000000002e-05 2024-03-19T11:30:18.498 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:18.498 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:18.498 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:18.498 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:18.498 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:18.499 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:18.500 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-19T11:30:18.501 INFO:tasks.rados.rados.0.smithi038.stdout:765: expect (ObjNum 31 snap 0 seq_num 31) 2024-03-19T11:30:18.501 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:18.512+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:18.502 INFO:tasks.rados.rados.0.smithi038.stdout:767: expect (ObjNum 84 snap 7 seq_num 84) 2024-03-19T11:30:18.539 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-19T11:30:18.602 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:18.615+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:18.631 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:18.631 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:18.631 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:30:18.631 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:18.645 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-19T11:30:18.646 INFO:tasks.rados.rados.0.smithi038.stdout:763: got (0) Success 2024-03-19T11:30:18.646 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-03-19T11:30:18.646 INFO:tasks.rados.rados.0.smithi038.stdout:764: expect (ObjNum 32 snap 0 seq_num 32) 2024-03-19T11:30:18.679 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:30:18.679 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:30:18.695 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:30:18.695 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:30:18.702 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:18.712+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:18.711 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:30:18.712 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:30:18.729 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:30:18.729 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:30:18.730 INFO:tasks.rados.rados.0.smithi038.stdout:768: finishing write tid 1 to smithi03829856-28 2024-03-19T11:30:18.730 INFO:tasks.rados.rados.0.smithi038.stdout:768: finishing write tid 2 to smithi03829856-28 2024-03-19T11:30:18.730 INFO:tasks.rados.rados.0.smithi038.stdout:768: finishing write tid 3 to smithi03829856-28 2024-03-19T11:30:18.730 INFO:tasks.rados.rados.0.smithi038.stdout:768: finishing write tid 4 to smithi03829856-28 2024-03-19T11:30:18.731 INFO:tasks.rados.rados.0.smithi038.stdout:768: finishing write tid 5 to smithi03829856-28 2024-03-19T11:30:18.731 INFO:tasks.rados.rados.0.smithi038.stdout:768: oid 28 updating version 0 to 27 2024-03-19T11:30:18.731 INFO:tasks.rados.rados.0.smithi038.stdout:768: oid 28 version 27 is already newer than 25 2024-03-19T11:30:18.731 INFO:tasks.rados.rados.0.smithi038.stdout:768: oid 28 version 27 is already newer than 26 2024-03-19T11:30:18.731 INFO:tasks.rados.rados.0.smithi038.stdout:768: oid 28 version 27 is already newer than 24 2024-03-19T11:30:18.731 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 28 v 27 (ObjNum 87 snap 8 seq_num 87) dirty exists 2024-03-19T11:30:18.732 INFO:tasks.rados.rados.0.smithi038.stdout:768: left oid 28 (ObjNum 87 snap 8 seq_num 87) 2024-03-19T11:30:18.732 INFO:tasks.rados.rados.0.smithi038.stdout:769: finishing write tid 1 to smithi03829856-14 2024-03-19T11:30:18.732 INFO:tasks.rados.rados.0.smithi038.stdout:770: expect (ObjNum 48 snap 0 seq_num 48) 2024-03-19T11:30:18.747 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:30:18.747 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:30:18.763 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-19T11:30:18.764 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-19T11:30:18.764 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-03-19T11:30:18.764 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T11:30:18.765 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:30:18.765 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:30:18.784 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:30:18.784 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:30:18.803 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:30:18.803 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:30:18.803 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:18.816+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:18.813 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:18.813 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:18.817 INFO:tasks.rados.rados.0.smithi038.stdout:769: finishing write tid 2 to smithi03829856-14 2024-03-19T11:30:18.818 INFO:tasks.rados.rados.0.smithi038.stdout:769: finishing write tid 3 to smithi03829856-14 2024-03-19T11:30:18.818 INFO:tasks.rados.rados.0.smithi038.stdout:769: finishing write tid 4 to smithi03829856-14 2024-03-19T11:30:18.818 INFO:tasks.rados.rados.0.smithi038.stdout:769: finishing write tid 5 to smithi03829856-14 2024-03-19T11:30:18.818 INFO:tasks.rados.rados.0.smithi038.stdout:769: oid 14 updating version 0 to 21 2024-03-19T11:30:18.818 INFO:tasks.rados.rados.0.smithi038.stdout:769: oid 14 updating version 21 to 24 2024-03-19T11:30:18.818 INFO:tasks.rados.rados.0.smithi038.stdout:769: oid 14 version 24 is already newer than 22 2024-03-19T11:30:18.818 INFO:tasks.rados.rados.0.smithi038.stdout:769: oid 14 version 24 is already newer than 23 2024-03-19T11:30:18.818 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 14 v 24 (ObjNum 88 snap 8 seq_num 88) dirty exists 2024-03-19T11:30:18.818 INFO:tasks.rados.rados.0.smithi038.stdout:769: left oid 14 (ObjNum 88 snap 8 seq_num 88) 2024-03-19T11:30:18.818 INFO:tasks.rados.rados.0.smithi038.stdout:771: got (0) Success 2024-03-19T11:30:18.818 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 5 v 3 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-03-19T11:30:18.818 INFO:tasks.rados.rados.0.smithi038.stdout:762: done (9 left) 2024-03-19T11:30:18.818 INFO:tasks.rados.rados.0.smithi038.stdout:763: done (8 left) 2024-03-19T11:30:18.819 INFO:tasks.rados.rados.0.smithi038.stdout:764: done (7 left) 2024-03-19T11:30:18.819 INFO:tasks.rados.rados.0.smithi038.stdout:765: done (6 left) 2024-03-19T11:30:18.819 INFO:tasks.rados.rados.0.smithi038.stdout:766: done (5 left) 2024-03-19T11:30:18.819 INFO:tasks.rados.rados.0.smithi038.stdout:767: done (4 left) 2024-03-19T11:30:18.819 INFO:tasks.rados.rados.0.smithi038.stdout:768: done (3 left) 2024-03-19T11:30:18.819 INFO:tasks.rados.rados.0.smithi038.stdout:769: done (2 left) 2024-03-19T11:30:18.819 INFO:tasks.rados.rados.0.smithi038.stdout:770: done (1 left) 2024-03-19T11:30:18.819 INFO:tasks.rados.rados.0.smithi038.stdout:771: done (0 left) 2024-03-19T11:30:18.819 INFO:tasks.rados.rados.0.smithi038.stdout:772: read oid 42 snap -1 2024-03-19T11:30:18.819 INFO:tasks.rados.rados.0.smithi038.stdout:772: expect (ObjNum 52 snap 0 seq_num 52) 2024-03-19T11:30:18.819 INFO:tasks.rados.rados.0.smithi038.stdout:773: read oid 49 snap -1 2024-03-19T11:30:18.819 INFO:tasks.rados.rados.0.smithi038.stdout:773: expect (ObjNum 48 snap 0 seq_num 48) 2024-03-19T11:30:18.819 INFO:tasks.rados.rados.0.smithi038.stdout:774: snap_create 2024-03-19T11:30:18.822 INFO:tasks.rados.rados.0.smithi038.stdout:772: expect (ObjNum 52 snap 0 seq_num 52) 2024-03-19T11:30:18.828 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-19T11:30:18.904 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:18.916+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:18.949 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:18.950 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:18.957 INFO:tasks.rados.rados.0.smithi038.stdout:773: expect (ObjNum 48 snap 0 seq_num 48) 2024-03-19T11:30:18.963 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-19T11:30:19.013 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:19.027+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:19.068 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:30:19.069 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-03-19T11:30:19.087 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":52,"stamp":"2024-03-19T11:30:18.139023+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":292692612,"num_objects":155,"num_object_clones":27,"num_object_copies":310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":2,"num_read":891,"num_read_kb":509681,"num_write":748,"num_write_kb":231282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":66,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":794,"ondisk_log_size":794,"up":58,"acting":58,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":510644,"kb_used_data":290292,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754464076,"statfs":{"total":773094113280,"available":772571213824,"internally_reserved":0,"allocated":297259008,"data_stored":328175463,"data_compressed":10066355,"data_compressed_allocated":10874880,"data_compressed_original":24154112,"omap_allocated":12751,"internal_metadata":225496625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":277187,"num_objects":5,"num_object_clones":2,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":113,"num_read_kb":116979,"num_write":19,"num_write_kb":3889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.357116"},"pg_stats":[{"pgid":"4.f","version":"0'0","reported_seq":57,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-19T11:30:13.774383+0000","last_change":"2024-03-19T11:30:10.741224+0000","last_active":"2024-03-19T11:30:13.774383+0000","last_peered":"2024-03-19T11:30:13.774383+0000","last_clean":"2024-03-19T11:30:13.774383+0000","last_became_active":"2024-03-19T11:29:58.551505+0000","last_became_peered":"2024-03-19T11:29:58.551505+0000","last_unstale":"2024-03-19T11:30:13.774383+0000","last_undegraded":"2024-03-19T11:30:13.774383+0000","last_fullsized":"2024-03-19T11:30:13.774383+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:26.324440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013402990000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"3","length":"2"}]},{"pgid":"4.e","version":"34'19","reported_seq":87,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T11:30:12.783575+0000","last_change":"2024-03-19T11:30:10.740240+0000","last_active":"2024-03-19T11:30:12.783575+0000","last_peered":"2024-03-19T11:30:12.783575+0000","last_clean":"2024-03-19T11:30:12.783575+0000","last_became_active":"2024-03-19T11:29:58.554727+0000","last_became_peered":"2024-03-19T11:29:58.554727+0000","last_unstale":"2024-03-19T11:30:12.783575+0000","last_undegraded":"2024-03-19T11:30:12.783575+0000","last_fullsized":"2024-03-19T11:30:12.783575+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:25.672319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0053605559999999998,"stat_sum":{"num_bytes":9106743,"num_objects":4,"num_object_clones":1,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":19131,"num_write":16,"num_write_kb":4704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"2"}]},{"pgid":"4.d","version":"33'27","reported_seq":91,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T11:30:12.787073+0000","last_change":"2024-03-19T11:30:10.745705+0000","last_active":"2024-03-19T11:30:12.787073+0000","last_peered":"2024-03-19T11:30:12.787073+0000","last_clean":"2024-03-19T11:30:12.787073+0000","last_became_active":"2024-03-19T11:29:58.554560+0000","last_became_peered":"2024-03-19T11:29:58.554560+0000","last_unstale":"2024-03-19T11:30:12.787073+0000","last_undegraded":"2024-03-19T11:30:12.787073+0000","last_fullsized":"2024-03-19T11:30:12.787073+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:59.616819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010721741999999999,"stat_sum":{"num_bytes":8531481,"num_objects":5,"num_object_clones":2,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":18,"num_read_kb":11861,"num_write":17,"num_write_kb":4249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"2"}]},{"pgid":"4.c","version":"35'19","reported_seq":80,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T11:30:12.809795+0000","last_change":"2024-03-19T11:30:10.740569+0000","last_active":"2024-03-19T11:30:12.809795+0000","last_peered":"2024-03-19T11:30:12.809795+0000","last_clean":"2024-03-19T11:30:12.809795+0000","last_became_active":"2024-03-19T11:29:58.552036+0000","last_became_peered":"2024-03-19T11:29:58.552036+0000","last_unstale":"2024-03-19T11:30:12.809795+0000","last_undegraded":"2024-03-19T11:30:12.809795+0000","last_fullsized":"2024-03-19T11:30:12.809795+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:21.268933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.005789565,"stat_sum":{"num_bytes":5685998,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":13,"num_read_kb":13306,"num_write":14,"num_write_kb":3672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"2"}]},{"pgid":"4.b","version":"35'33","reported_seq":144,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-19T11:30:16.857178+0000","last_change":"2024-03-19T11:30:15.833105+0000","last_active":"2024-03-19T11:30:16.857178+0000","last_peered":"2024-03-19T11:30:16.857178+0000","last_clean":"2024-03-19T11:30:16.857178+0000","last_became_active":"2024-03-19T11:29:58.555569+0000","last_became_peered":"2024-03-19T11:29:58.555569+0000","last_unstale":"2024-03-19T11:30:16.857178+0000","last_undegraded":"2024-03-19T11:30:16.857178+0000","last_fullsized":"2024-03-19T11:30:16.857178+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:58.838732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013977779999999999,"stat_sum":{"num_bytes":10873997,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":102,"num_read_kb":102381,"num_write":26,"num_write_kb":6751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"2"},{"start":"7","length":"1"}]},{"pgid":"4.a","version":"33'32","reported_seq":110,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T11:30:12.884686+0000","last_change":"2024-03-19T11:30:12.885203+0000","last_active":"2024-03-19T11:30:12.884686+0000","last_peered":"2024-03-19T11:30:12.884686+0000","last_clean":"2024-03-19T11:30:12.884686+0000","last_became_active":"2024-03-19T11:29:58.553514+0000","last_became_peered":"2024-03-19T11:29:58.553514+0000","last_unstale":"2024-03-19T11:30:12.884686+0000","last_undegraded":"2024-03-19T11:30:12.884686+0000","last_fullsized":"2024-03-19T11:30:12.884686+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:09.712456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.12917921600000001,"stat_sum":{"num_bytes":13243751,"num_objects":5,"num_object_clones":1,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":37668,"num_write":22,"num_write_kb":6357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"3","length":"2"}]},{"pgid":"4.9","version":"33'35","reported_seq":118,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T11:30:12.783398+0000","last_change":"2024-03-19T11:30:10.740025+0000","last_active":"2024-03-19T11:30:12.783398+0000","last_peered":"2024-03-19T11:30:12.783398+0000","last_clean":"2024-03-19T11:30:12.783398+0000","last_became_active":"2024-03-19T11:29:58.552840+0000","last_became_peered":"2024-03-19T11:29:58.552840+0000","last_unstale":"2024-03-19T11:30:12.783398+0000","last_undegraded":"2024-03-19T11:30:12.783398+0000","last_fullsized":"2024-03-19T11:30:12.783398+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:05.496404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051836210000000002,"stat_sum":{"num_bytes":13947854,"num_objects":6,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":56,"num_read_kb":44958,"num_write":25,"num_write_kb":7210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":6,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"2"}]},{"pgid":"4.8","version":"36'40","reported_seq":107,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-19T11:30:14.037792+0000","last_change":"2024-03-19T11:30:10.746766+0000","last_active":"2024-03-19T11:30:14.037792+0000","last_peered":"2024-03-19T11:30:14.037792+0000","last_clean":"2024-03-19T11:30:14.037792+0000","last_became_active":"2024-03-19T11:29:58.641526+0000","last_became_peered":"2024-03-19T11:29:58.641526+0000","last_unstale":"2024-03-19T11:30:14.037792+0000","last_undegraded":"2024-03-19T11:30:14.037792+0000","last_fullsized":"2024-03-19T11:30:14.037792+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:02.763571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042345000000000002,"stat_sum":{"num_bytes":16951683,"num_objects":7,"num_object_clones":3,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":3543,"num_write":36,"num_write_kb":9714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"2"}]},{"pgid":"4.5","version":"35'56","reported_seq":153,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T11:30:12.892975+0000","last_change":"2024-03-19T11:30:12.893429+0000","last_active":"2024-03-19T11:30:12.892975+0000","last_peered":"2024-03-19T11:30:12.892975+0000","last_clean":"2024-03-19T11:30:12.892975+0000","last_became_active":"2024-03-19T11:29:58.550423+0000","last_became_peered":"2024-03-19T11:29:58.550423+0000","last_unstale":"2024-03-19T11:30:12.892975+0000","last_undegraded":"2024-03-19T11:30:12.892975+0000","last_fullsized":"2024-03-19T11:30:12.892975+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:04.744453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.13737002300000001,"stat_sum":{"num_bytes":18179943,"num_objects":8,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":90,"num_read_kb":88789,"num_write":41,"num_write_kb":10650,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":8,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"3","length":"2"}]},{"pgid":"1.0","version":"17'117","reported_seq":201,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-19T11:30:13.927810+0000","last_change":"2024-03-19T11:29:45.008018+0000","last_active":"2024-03-19T11:30:13.927810+0000","last_peered":"2024-03-19T11:30:13.927810+0000","last_clean":"2024-03-19T11:30:13.927810+0000","last_became_active":"2024-03-19T11:29:45.005482+0000","last_became_peered":"2024-03-19T11:29:45.005482+0000","last_unstale":"2024-03-19T11:30:13.927810+0000","last_undegraded":"2024-03-19T11:30:13.927810+0000","last_fullsized":"2024-03-19T11:30:13.927810+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:43.166618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:43.166618+0000","last_clean_scrub_stamp":"2024-03-19T11:29:43.166618+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:02.385897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-19T11:30:13.774090+0000","last_change":"2024-03-19T11:29:46.322491+0000","last_active":"2024-03-19T11:30:13.774090+0000","last_peered":"2024-03-19T11:30:13.774090+0000","last_clean":"2024-03-19T11:30:13.774090+0000","last_became_active":"2024-03-19T11:29:46.321760+0000","last_became_peered":"2024-03-19T11:29:46.321760+0000","last_unstale":"2024-03-19T11:30:13.774090+0000","last_undegraded":"2024-03-19T11:30:13.774090+0000","last_fullsized":"2024-03-19T11:30:13.774090+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:04.561915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"36'53","reported_seq":100,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-19T11:30:13.801168+0000","last_change":"2024-03-19T11:29:57.549277+0000","last_active":"2024-03-19T11:30:13.801168+0000","last_peered":"2024-03-19T11:30:13.801168+0000","last_clean":"2024-03-19T11:30:13.801168+0000","last_became_active":"2024-03-19T11:29:57.548946+0000","last_became_peered":"2024-03-19T11:29:57.548946+0000","last_unstale":"2024-03-19T11:30:13.801168+0000","last_undegraded":"2024-03-19T11:30:13.801168+0000","last_fullsized":"2024-03-19T11:30:13.801168+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:08.380405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28269046,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":47,"num_read_kb":3072,"num_write":60,"num_write_kb":30258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"39'47","reported_seq":138,"reported_epoch":39,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:30:16.882443+0000","last_change":"2024-03-19T11:30:15.831424+0000","last_active":"2024-03-19T11:30:16.882443+0000","last_peered":"2024-03-19T11:30:16.882443+0000","last_clean":"2024-03-19T11:30:16.882443+0000","last_became_active":"2024-03-19T11:29:58.552259+0000","last_became_peered":"2024-03-19T11:29:58.552259+0000","last_unstale":"2024-03-19T11:30:16.882443+0000","last_undegraded":"2024-03-19T11:30:16.882443+0000","last_fullsized":"2024-03-19T11:30:16.882443+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:08.793674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050003899999999997,"stat_sum":{"num_bytes":14520882,"num_objects":8,"num_object_clones":4,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":59,"num_read_kb":54660,"num_write":38,"num_write_kb":9943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":8,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"2"}]},{"pgid":"2.0","version":"0'0","reported_seq":47,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-19T11:30:13.927517+0000","last_change":"2024-03-19T11:29:46.436253+0000","last_active":"2024-03-19T11:30:13.927517+0000","last_peered":"2024-03-19T11:30:13.927517+0000","last_clean":"2024-03-19T11:30:13.927517+0000","last_became_active":"2024-03-19T11:29:46.435783+0000","last_became_peered":"2024-03-19T11:29:46.435783+0000","last_unstale":"2024-03-19T11:30:13.927517+0000","last_undegraded":"2024-03-19T11:30:13.927517+0000","last_fullsized":"2024-03-19T11:30:13.927517+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:59.997466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"36'60","reported_seq":111,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-19T11:30:16.830332+0000","last_change":"2024-03-19T11:29:57.555187+0000","last_active":"2024-03-19T11:30:16.830332+0000","last_peered":"2024-03-19T11:30:16.830332+0000","last_clean":"2024-03-19T11:30:16.830332+0000","last_became_active":"2024-03-19T11:29:57.554825+0000","last_became_peered":"2024-03-19T11:29:57.554825+0000","last_unstale":"2024-03-19T11:30:16.830332+0000","last_undegraded":"2024-03-19T11:30:16.830332+0000","last_fullsized":"2024-03-19T11:30:16.830332+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:58.878342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33553143,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":47,"num_read_kb":3434,"num_write":63,"num_write_kb":35483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"36'17","reported_seq":76,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-19T11:30:14.013353+0000","last_change":"2024-03-19T11:30:10.745473+0000","last_active":"2024-03-19T11:30:14.013353+0000","last_peered":"2024-03-19T11:30:14.013353+0000","last_clean":"2024-03-19T11:30:14.013353+0000","last_became_active":"2024-03-19T11:29:58.555355+0000","last_became_peered":"2024-03-19T11:29:58.555355+0000","last_unstale":"2024-03-19T11:30:14.013353+0000","last_undegraded":"2024-03-19T11:30:14.013353+0000","last_fullsized":"2024-03-19T11:30:14.013353+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:14.841303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051348610000000001,"stat_sum":{"num_bytes":5589897,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":13,"num_write_kb":3721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"3","length":"2"}]},{"pgid":"2.1","version":"0'0","reported_seq":48,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-19T11:30:13.768614+0000","last_change":"2024-03-19T11:29:46.319811+0000","last_active":"2024-03-19T11:30:13.768614+0000","last_peered":"2024-03-19T11:30:13.768614+0000","last_clean":"2024-03-19T11:30:13.768614+0000","last_became_active":"2024-03-19T11:29:46.319334+0000","last_became_peered":"2024-03-19T11:29:46.319334+0000","last_unstale":"2024-03-19T11:30:13.768614+0000","last_undegraded":"2024-03-19T11:30:13.768614+0000","last_fullsized":"2024-03-19T11:30:13.768614+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:04.320463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"36'64","reported_seq":109,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-19T11:30:14.172561+0000","last_change":"2024-03-19T11:29:57.550320+0000","last_active":"2024-03-19T11:30:14.172561+0000","last_peered":"2024-03-19T11:30:14.172561+0000","last_clean":"2024-03-19T11:30:14.172561+0000","last_became_active":"2024-03-19T11:29:57.549928+0000","last_became_peered":"2024-03-19T11:29:57.549928+0000","last_unstale":"2024-03-19T11:30:14.172561+0000","last_undegraded":"2024-03-19T11:30:14.172561+0000","last_fullsized":"2024-03-19T11:30:14.172561+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:01.987302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30002271,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":55,"num_read_kb":1578,"num_write":68,"num_write_kb":31572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"36'48","reported_seq":126,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-19T11:30:14.168968+0000","last_change":"2024-03-19T11:30:10.750504+0000","last_active":"2024-03-19T11:30:14.168968+0000","last_peered":"2024-03-19T11:30:14.168968+0000","last_clean":"2024-03-19T11:30:14.168968+0000","last_became_active":"2024-03-19T11:29:58.555204+0000","last_became_peered":"2024-03-19T11:29:58.555204+0000","last_unstale":"2024-03-19T11:30:14.168968+0000","last_undegraded":"2024-03-19T11:30:14.168968+0000","last_fullsized":"2024-03-19T11:30:14.168968+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:15.225888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0095649659999999994,"stat_sum":{"num_bytes":17638063,"num_objects":9,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":47,"num_read_kb":32732,"num_write":37,"num_write_kb":11786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":9,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"3","length":"2"}]},{"pgid":"2.2","version":"0'0","reported_seq":46,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-19T11:30:13.774348+0000","last_change":"2024-03-19T11:29:46.321255+0000","last_active":"2024-03-19T11:30:13.774348+0000","last_peered":"2024-03-19T11:30:13.774348+0000","last_clean":"2024-03-19T11:30:13.774348+0000","last_became_active":"2024-03-19T11:29:46.320679+0000","last_became_peered":"2024-03-19T11:29:46.320679+0000","last_unstale":"2024-03-19T11:30:13.774348+0000","last_undegraded":"2024-03-19T11:30:13.774348+0000","last_fullsized":"2024-03-19T11:30:13.774348+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:14.793563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"34'50","reported_seq":91,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T11:30:12.783258+0000","last_change":"2024-03-19T11:29:57.554995+0000","last_active":"2024-03-19T11:30:12.783258+0000","last_peered":"2024-03-19T11:30:12.783258+0000","last_clean":"2024-03-19T11:30:12.783258+0000","last_became_active":"2024-03-19T11:29:57.554481+0000","last_became_peered":"2024-03-19T11:29:57.554481+0000","last_unstale":"2024-03-19T11:30:12.783258+0000","last_undegraded":"2024-03-19T11:30:12.783258+0000","last_fullsized":"2024-03-19T11:30:12.783258+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:12.584961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36797042,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":35,"num_read_kb":691,"num_write":50,"num_write_kb":35942,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"28'6","reported_seq":68,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-19T11:30:13.774030+0000","last_change":"2024-03-19T11:30:10.741349+0000","last_active":"2024-03-19T11:30:13.774030+0000","last_peered":"2024-03-19T11:30:13.774030+0000","last_clean":"2024-03-19T11:30:13.774030+0000","last_became_active":"2024-03-19T11:29:58.555026+0000","last_became_peered":"2024-03-19T11:29:58.555026+0000","last_unstale":"2024-03-19T11:30:13.774030+0000","last_undegraded":"2024-03-19T11:30:13.774030+0000","last_fullsized":"2024-03-19T11:30:13.774030+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:14.132274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011524930000000001,"stat_sum":{"num_bytes":2352947,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":6561,"num_write":4,"num_write_kb":981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"3","length":"2"}]},{"pgid":"2.4","version":"0'0","reported_seq":47,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-19T11:30:13.774770+0000","last_change":"2024-03-19T11:29:46.541233+0000","last_active":"2024-03-19T11:30:13.774770+0000","last_peered":"2024-03-19T11:30:13.774770+0000","last_clean":"2024-03-19T11:30:13.774770+0000","last_became_active":"2024-03-19T11:29:46.540939+0000","last_became_peered":"2024-03-19T11:29:46.540939+0000","last_unstale":"2024-03-19T11:30:13.774770+0000","last_undegraded":"2024-03-19T11:30:13.774770+0000","last_fullsized":"2024-03-19T11:30:13.774770+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:54.761098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"33'18","reported_seq":103,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-19T11:30:16.831528+0000","last_change":"2024-03-19T11:30:15.832348+0000","last_active":"2024-03-19T11:30:16.831528+0000","last_peered":"2024-03-19T11:30:16.831528+0000","last_clean":"2024-03-19T11:30:16.831528+0000","last_became_active":"2024-03-19T11:29:58.555039+0000","last_became_peered":"2024-03-19T11:29:58.555039+0000","last_unstale":"2024-03-19T11:30:16.831528+0000","last_undegraded":"2024-03-19T11:30:16.831528+0000","last_fullsized":"2024-03-19T11:30:16.831528+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:03.269894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072717599999999995,"stat_sum":{"num_bytes":8970335,"num_objects":4,"num_object_clones":1,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":35021,"num_write":16,"num_write_kb":4517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"2"},{"start":"7","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":47,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-19T11:30:13.927344+0000","last_change":"2024-03-19T11:29:46.542897+0000","last_active":"2024-03-19T11:30:13.927344+0000","last_peered":"2024-03-19T11:30:13.927344+0000","last_clean":"2024-03-19T11:30:13.927344+0000","last_became_active":"2024-03-19T11:29:46.542092+0000","last_became_peered":"2024-03-19T11:29:46.542092+0000","last_unstale":"2024-03-19T11:30:13.927344+0000","last_undegraded":"2024-03-19T11:30:13.927344+0000","last_fullsized":"2024-03-19T11:30:13.927344+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:46.318698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"36'21","reported_seq":84,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-19T11:30:13.969763+0000","last_change":"2024-03-19T11:30:10.739101+0000","last_active":"2024-03-19T11:30:13.969763+0000","last_peered":"2024-03-19T11:30:13.969763+0000","last_clean":"2024-03-19T11:30:13.969763+0000","last_became_active":"2024-03-19T11:29:58.642619+0000","last_became_peered":"2024-03-19T11:29:58.642619+0000","last_unstale":"2024-03-19T11:30:13.969763+0000","last_undegraded":"2024-03-19T11:30:13.969763+0000","last_fullsized":"2024-03-19T11:30:13.969763+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:04.908726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092617600000000002,"stat_sum":{"num_bytes":8452942,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":7,"num_read_kb":5,"num_write":19,"num_write_kb":5252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3","length":"2"}]},{"pgid":"2.6","version":"0'0","reported_seq":47,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-19T11:30:13.774229+0000","last_change":"2024-03-19T11:29:46.423438+0000","last_active":"2024-03-19T11:30:13.774229+0000","last_peered":"2024-03-19T11:30:13.774229+0000","last_clean":"2024-03-19T11:30:13.774229+0000","last_became_active":"2024-03-19T11:29:46.423027+0000","last_became_peered":"2024-03-19T11:29:46.423027+0000","last_unstale":"2024-03-19T11:30:13.774229+0000","last_undegraded":"2024-03-19T11:30:13.774229+0000","last_fullsized":"2024-03-19T11:30:13.774229+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:55.305083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"33'32","reported_seq":114,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-03-19T11:30:12.805750+0000","last_change":"2024-03-19T11:30:10.740545+0000","last_active":"2024-03-19T11:30:12.805750+0000","last_peered":"2024-03-19T11:30:12.805750+0000","last_clean":"2024-03-19T11:30:12.805750+0000","last_became_active":"2024-03-19T11:29:58.552749+0000","last_became_peered":"2024-03-19T11:29:58.552749+0000","last_unstale":"2024-03-19T11:30:12.805750+0000","last_undegraded":"2024-03-19T11:30:12.805750+0000","last_fullsized":"2024-03-19T11:30:12.805750+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:00.935177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.005651105,"stat_sum":{"num_bytes":9434226,"num_objects":5,"num_object_clones":2,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":59,"num_read_kb":50187,"num_write":24,"num_write_kb":6104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"2"}]},{"pgid":"2.7","version":"0'0","reported_seq":30,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-03-19T11:30:13.768712+0000","last_change":"2024-03-19T11:30:03.276845+0000","last_active":"2024-03-19T11:30:13.768712+0000","last_peered":"2024-03-19T11:30:13.768712+0000","last_clean":"2024-03-19T11:30:13.768712+0000","last_became_active":"2024-03-19T11:30:03.276220+0000","last_became_peered":"2024-03-19T11:30:03.276220+0000","last_unstale":"2024-03-19T11:30:13.768712+0000","last_undegraded":"2024-03-19T11:30:13.768712+0000","last_fullsized":"2024-03-19T11:30:13.768712+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:01.829767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":163480742,"num_objects":77,"num_object_clones":27,"num_object_copies":154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":2,"num_read":591,"num_read_kb":500806,"num_write":348,"num_write_kb":95611,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":66,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180580352,"data_stored":212334768,"data_compressed":339078,"data_compressed_allocated":368640,"data_compressed_original":1179648,"omap_allocated":0,"internal_metadata":0},"log_size":450,"ondisk_log_size":450,"up":32,"acting":32,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":128621502,"num_objects":76,"num_object_clones":0,"num_object_copies":152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":0,"num_read":184,"num_read_kb":8775,"num_write":241,"num_write_kb":133255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":126664704,"data_stored":138621739,"data_compressed":9812108,"data_compressed_allocated":10559488,"data_compressed_original":22695936,"omap_allocated":0,"internal_metadata":0},"log_size":227,"ondisk_log_size":227,"up":8,"acting":8,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":12128,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542154,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41796,"kb_used_data":14768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330044,"statfs":{"total":96636764160,"available":96593965056,"internally_reserved":0,"allocated":15122432,"data_stored":17763148,"data_compressed":6064,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542154,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60984,"kb_used_data":33956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94310856,"statfs":{"total":96636764160,"available":96574316544,"internally_reserved":0,"allocated":34770944,"data_stored":38488028,"data_compressed":1162692,"data_compressed_allocated":1245184,"data_compressed_original":2547712,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542154,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48832,"kb_used_data":21812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94323008,"statfs":{"total":96636764160,"available":96586760192,"internally_reserved":0,"allocated":22335488,"data_stored":24865120,"data_compressed":1166432,"data_compressed_allocated":1249280,"data_compressed_original":2572288,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542153,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54164,"kb_used_data":27136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94317676,"statfs":{"total":96636764160,"available":96581300224,"internally_reserved":0,"allocated":27787264,"data_stored":31627898,"data_compressed":1140267,"data_compressed_allocated":1224704,"data_compressed_original":2621440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542153,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82448,"kb_used_data":51268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94289392,"statfs":{"total":96636764160,"available":96552337408,"internally_reserved":0,"allocated":52498432,"data_stored":57314171,"data_compressed":1140267,"data_compressed_allocated":1224704,"data_compressed_original":2621440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46952,"kb_used_data":19928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94324888,"statfs":{"total":96636764160,"available":96588685312,"internally_reserved":0,"allocated":20406272,"data_stored":20603999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94864,"kb_used_data":67848,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94276976,"statfs":{"total":96636764160,"available":96539623424,"internally_reserved":0,"allocated":69476352,"data_stored":75536338,"data_compressed":2693246,"data_compressed_allocated":2899968,"data_compressed_original":6291456,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80604,"kb_used_data":53576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94291236,"statfs":{"total":96636764160,"available":96554225664,"internally_reserved":0,"allocated":54861824,"data_stored":61976761,"data_compressed":2757387,"data_compressed_allocated":2998272,"data_compressed_original":6987776,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6064,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6064,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36499456,"data_stored":39802054,"data_compressed":2707578,"data_compressed_allocated":2916352,"data_compressed_original":6279168,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14491648,"data_stored":15839753,"data_compressed":1039210,"data_compressed_allocated":1122304,"data_compressed_original":2490368,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14491648,"data_stored":15839753,"data_compressed":1039210,"data_compressed_allocated":1122304,"data_compressed_original":2490368,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12787712,"data_stored":14093613,"data_compressed":1166432,"data_compressed_allocated":1249280,"data_compressed_original":2572288,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":35606528,"data_stored":38952953,"data_compressed":2693246,"data_compressed_allocated":2899968,"data_compressed_original":6291456,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12787712,"data_stored":14093613,"data_compressed":1166432,"data_compressed_allocated":1249280,"data_compressed_original":2572288,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":17891328,"data_stored":21453889,"data_compressed":43745,"data_compressed_allocated":49152,"data_compressed_original":196608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":41324544,"data_stored":48053867,"data_compressed":161760,"data_compressed_allocated":172032,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":15044608,"data_stored":18380425,"data_compressed":118015,"data_compressed_allocated":122880,"data_compressed_original":196608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":9211904,"data_stored":10606032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":35012608,"data_stored":38446009,"data_compressed":7779,"data_compressed_allocated":12288,"data_compressed_original":196608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":26845184,"data_stored":35064751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20045824,"data_stored":20633646,"data_compressed":7779,"data_compressed_allocated":12288,"data_compressed_original":196608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":15204352,"data_stored":19696149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T11:30:19.088 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2024-03-19T11:30:19.089 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 26 2024-03-19T11:30:19.089 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:19.089 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:19.101 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-19T11:30:19.131 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:19.144+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:19.207 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:19.219+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:19.271 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:19.271 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:19.286 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-19T11:30:19.308 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:19.320+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:19.409 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:19.423+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:19.452 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:19.452 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:19.466 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-19T11:30:19.510 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:19.520+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:19.611 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:19.624+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:19.635 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:19.635 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:19.650 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-19T11:30:19.712 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:19.724+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:19.768 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:19.768 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:19.783 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-19T11:30:19.816 INFO:teuthology.orchestra.run.smithi038.stderr:set pool 4 pg_num to 26 2024-03-19T11:30:19.826 INFO:tasks.rados.rados.0.smithi038.stdout:772: done (2 left) 2024-03-19T11:30:19.826 INFO:tasks.rados.rados.0.smithi038.stdout:773: done (1 left) 2024-03-19T11:30:19.826 INFO:tasks.rados.rados.0.smithi038.stdout:774: done (0 left) 2024-03-19T11:30:19.826 INFO:tasks.rados.rados.0.smithi038.stdout:775: set_chunk oid 4 target oid 2024-03-19T11:30:19.827 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:19.840+0000 7f6a7f9da640 -1 osd.1 41 handle_osd_map injecting map crc failure 2024-03-19T11:30:19.829 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:19.843+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:19.834 INFO:tasks.rados.rados.0.smithi038.stdout:775: set_chunk oid 4 offset: 1368576 length: 59392 target oid 9ed858526fa2cfec6d8643c6cb514106f4135360c53e2a332e9febcaba6df2cb offset: 0 snap -1 2024-03-19T11:30:19.834 INFO:tasks.rados.rados.0.smithi038.stdout:776: read oid 27 snap 6 2024-03-19T11:30:19.834 INFO:tasks.rados.rados.0.smithi038.stdout:776: expect (ObjNum 26 snap 0 seq_num 26) 2024-03-19T11:30:19.834 INFO:tasks.rados.rados.0.smithi038.stdout:777: snap_create 2024-03-19T11:30:19.834 INFO:tasks.rados.rados.0.smithi038.stdout:775: finishing set_chunk to oid 4 2024-03-19T11:30:19.834 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 4 set_chunk returned error code -95 offset: 1368576 length: 59392 2024-03-19T11:30:19.835 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-03-19T11:30:19.845 INFO:tasks.rados.rados.0.smithi038.stdout:776: expect (ObjNum 26 snap 0 seq_num 26) 2024-03-19T11:30:19.870 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-03-19T11:30:19.906 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:19.906 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:19.914 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:19.927+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:19.921 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-19T11:30:20.015 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:20.028+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:20.084 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:20.085 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:20.099 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-19T11:30:20.116 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:20.128+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:20.216 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:20.231+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:20.222 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:20.222 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:20.237 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-19T11:30:20.325 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:20.339+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:20.404 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:20.404 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:20.418 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-19T11:30:20.461 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:20.475+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:20.520 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:20.531+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:20.555 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:20.555 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:20.569 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-19T11:30:20.621 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:20.635+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:20.695 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:20.695 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:20.709 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-19T11:30:20.758 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:20.772+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:20.822 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-19T11:30:20.824 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:20.836+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:20.831 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-03-19T11:30:20.834 INFO:tasks.rados.rados.0.smithi038.stdout:775: done (2 left) 2024-03-19T11:30:20.834 INFO:tasks.rados.rados.0.smithi038.stdout:776: done (1 left) 2024-03-19T11:30:20.834 INFO:tasks.rados.rados.0.smithi038.stdout:777: done (0 left) 2024-03-19T11:30:20.834 INFO:tasks.rados.rados.0.smithi038.stdout:778: tier_promote oid 47 2024-03-19T11:30:20.834 INFO:tasks.rados.rados.0.smithi038.stdout:779: write oid 31 current snap is 10 2024-03-19T11:30:20.834 INFO:tasks.rados.rados.0.smithi038.stdout:779: seq_num 89 ranges {695823=777044,1925015=639682,3328058=167765} 2024-03-19T11:30:20.844 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:20.844 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:20.851 INFO:tasks.rados.rados.0.smithi038.stdout:779: writing smithi03829856-31 from 695823 to 1472867 tid 1 2024-03-19T11:30:20.861 INFO:tasks.rados.rados.0.smithi038.stdout:779: writing smithi03829856-31 from 1925015 to 2564697 tid 2 2024-03-19T11:30:20.863 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-19T11:30:20.869 INFO:tasks.rados.rados.0.smithi038.stdout:779: writing smithi03829856-31 from 3328058 to 3495823 tid 3 2024-03-19T11:30:20.869 INFO:tasks.rados.rados.0.smithi038.stdout:780: delete oid 39 current snap is 10 2024-03-19T11:30:20.878 INFO:tasks.rados.rados.0.smithi038.stdout:780: done (2 left) 2024-03-19T11:30:20.878 INFO:tasks.rados.rados.0.smithi038.stdout:781: write oid 43 current snap is 10 2024-03-19T11:30:20.878 INFO:tasks.rados.rados.0.smithi038.stdout:781: seq_num 91 ranges {729437=527542,1925626=643351,3266551=262886} 2024-03-19T11:30:20.887 INFO:tasks.rados.rados.0.smithi038.stdout:781: writing smithi03829856-43 from 729437 to 1256979 tid 1 2024-03-19T11:30:20.897 INFO:tasks.rados.rados.0.smithi038.stdout:781: writing smithi03829856-43 from 1925626 to 2568977 tid 2 2024-03-19T11:30:20.905 INFO:tasks.rados.rados.0.smithi038.stdout:781: writing smithi03829856-43 from 3266551 to 3529437 tid 3 2024-03-19T11:30:20.905 INFO:tasks.rados.rados.0.smithi038.stdout:782: delete oid 2 current snap is 10 2024-03-19T11:30:20.925 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:20.936+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:20.989 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:30:20.990 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:30:21.001 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:21.001 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:21.008 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:30:21.009 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:30:21.016 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-19T11:30:21.026 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:30:21.027 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:30:21.027 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:21.040+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:21.047 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:30:21.047 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:30:21.067 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:30:21.067 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:30:21.088 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:30:21.088 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:30:21.109 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:30:21.109 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:30:21.127 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:21.140+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:21.130 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:30:21.131 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:30:21.146 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:21.146 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:30:21.147 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:30:21.147 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:30:21.147 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:21.161 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-19T11:30:21.228 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:21.238+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:21.325 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:21.325 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:30:21.325 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:30:21.325 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:30:21.325 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.325 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:6 4.6 e27/21 4:626acd4d:::smithi03829856-2:head v 27'2, mlcod=0'0)", 2024-03-19T11:30:21.325 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.696903+0000", 2024-03-19T11:30:21.325 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 16.637910494, 2024-03-19T11:30:21.325 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0071029129999999998, 2024-03-19T11:30:21.325 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:21.326 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:21.326 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:21.326 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.326 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:21.326 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696903+0000", 2024-03-19T11:30:21.326 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.326 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.326 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.326 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:21.326 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696903+0000", 2024-03-19T11:30:21.326 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.326 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.326 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.327 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:21.327 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696904+0000", 2024-03-19T11:30:21.327 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4270000000000001e-06 2024-03-19T11:30:21.327 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.327 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.327 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:21.327 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.697064+0000", 2024-03-19T11:30:21.327 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015956100000000001 2024-03-19T11:30:21.327 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.327 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.327 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:21.327 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.697082+0000", 2024-03-19T11:30:21.327 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8868999999999998e-05 2024-03-19T11:30:21.327 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.328 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.328 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:21.328 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.697096+0000", 2024-03-19T11:30:21.328 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3155e-05 2024-03-19T11:30:21.328 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.328 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.328 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:21.328 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.697169+0000", 2024-03-19T11:30:21.328 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3226999999999995e-05 2024-03-19T11:30:21.328 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.328 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.328 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:21.328 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.697193+0000", 2024-03-19T11:30:21.329 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4153000000000001e-05 2024-03-19T11:30:21.329 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.329 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.329 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:21.329 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.703936+0000", 2024-03-19T11:30:21.329 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0067430570000000002 2024-03-19T11:30:21.329 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.329 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.329 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:21.329 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.704006+0000", 2024-03-19T11:30:21.329 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9463999999999998e-05 2024-03-19T11:30:21.329 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.329 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:21.329 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.330 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.330 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.330 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:13 4.4 4:21eac9b2:::smithi03829856-4:head [write 450421~625747,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:30:21.330 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.720170+0000", 2024-03-19T11:30:21.330 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 16.614643180000002, 2024-03-19T11:30:21.330 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0090522840000000007, 2024-03-19T11:30:21.330 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:21.330 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:21.330 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:21.330 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:21.330 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:21.330 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 13 2024-03-19T11:30:21.330 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.330 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:21.331 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.331 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:21.331 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.720170+0000", 2024-03-19T11:30:21.331 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.331 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.331 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.331 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:21.331 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.720170+0000", 2024-03-19T11:30:21.331 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.331 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.331 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.331 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:21.331 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.720172+0000", 2024-03-19T11:30:21.331 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9709999999999998e-06 2024-03-19T11:30:21.332 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.332 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.332 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:21.332 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.720601+0000", 2024-03-19T11:30:21.332 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00042897999999999998 2024-03-19T11:30:21.332 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.332 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.332 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:21.332 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.720615+0000", 2024-03-19T11:30:21.332 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4617000000000001e-05 2024-03-19T11:30:21.333 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.333 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.333 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:21.333 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.720627+0000", 2024-03-19T11:30:21.333 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1513e-05 2024-03-19T11:30:21.333 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.333 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.334 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:21.334 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.720705+0000", 2024-03-19T11:30:21.334 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7880000000000007e-05 2024-03-19T11:30:21.334 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.334 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.334 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-03-19T11:30:21.334 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.720709+0000", 2024-03-19T11:30:21.334 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5549999999999996e-06 2024-03-19T11:30:21.334 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.334 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.334 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:21.334 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.722748+0000", 2024-03-19T11:30:21.334 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0020388630000000001 2024-03-19T11:30:21.334 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.335 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.335 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:21.335 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.722891+0000", 2024-03-19T11:30:21.335 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014286 2024-03-19T11:30:21.335 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.335 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.335 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 2", 2024-03-19T11:30:21.335 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723057+0000", 2024-03-19T11:30:21.335 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000166286 2024-03-19T11:30:21.335 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.335 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.335 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:21.335 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.725297+0000", 2024-03-19T11:30:21.335 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022397770000000001 2024-03-19T11:30:21.336 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.336 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.336 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:21.336 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.729110+0000", 2024-03-19T11:30:21.336 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038130109999999998 2024-03-19T11:30:21.336 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.336 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.336 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:21.336 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.729169+0000", 2024-03-19T11:30:21.336 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8616000000000003e-05 2024-03-19T11:30:21.336 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.336 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.336 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:21.336 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.729222+0000", 2024-03-19T11:30:21.336 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3355e-05 2024-03-19T11:30:21.337 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.337 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:21.337 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.337 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.337 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.337 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:844 4.7 4:e41d47fb:::smithi03829856-14:head [write 574806~778913] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:21.337 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:08.749074+0000", 2024-03-19T11:30:21.337 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 12.585739398999999, 2024-03-19T11:30:21.337 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013996124, 2024-03-19T11:30:21.337 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:21.337 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:21.337 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:21.337 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:21.337 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:21.338 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 844 2024-03-19T11:30:21.338 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.338 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:21.338 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.338 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:21.338 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.749074+0000", 2024-03-19T11:30:21.338 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.338 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.338 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.338 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:21.338 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.749074+0000", 2024-03-19T11:30:21.338 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.338 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.338 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.339 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:21.339 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.749076+0000", 2024-03-19T11:30:21.339 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.588e-06 2024-03-19T11:30:21.339 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.339 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.339 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:21.339 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.749714+0000", 2024-03-19T11:30:21.339 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00063805999999999997 2024-03-19T11:30:21.339 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.339 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.339 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:21.339 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.749750+0000", 2024-03-19T11:30:21.339 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5709e-05 2024-03-19T11:30:21.339 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.339 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.340 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:21.340 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.749767+0000", 2024-03-19T11:30:21.340 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7127000000000001e-05 2024-03-19T11:30:21.340 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.340 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.340 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:21.340 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.749860+0000", 2024-03-19T11:30:21.340 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2875999999999996e-05 2024-03-19T11:30:21.340 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.340 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.340 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:21.340 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.749943+0000", 2024-03-19T11:30:21.340 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2915000000000002e-05 2024-03-19T11:30:21.340 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.341 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.341 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:21.341 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.750083+0000", 2024-03-19T11:30:21.341 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013953300000000001 2024-03-19T11:30:21.341 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.341 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.341 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:21.341 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.753673+0000", 2024-03-19T11:30:21.341 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0035904470000000001 2024-03-19T11:30:21.341 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.341 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.341 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:21.341 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.762965+0000", 2024-03-19T11:30:21.341 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0092923390000000002 2024-03-19T11:30:21.341 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.342 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.342 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:21.342 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.763001+0000", 2024-03-19T11:30:21.342 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6087000000000002e-05 2024-03-19T11:30:21.342 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.342 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.342 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:21.342 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.763070+0000", 2024-03-19T11:30:21.342 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8442999999999995e-05 2024-03-19T11:30:21.342 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.342 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:21.342 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.342 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.342 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.343 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:845 4.7 4:e41d47fb:::smithi03829856-14:head [write 2061132~557342] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:21.343 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:08.756359+0000", 2024-03-19T11:30:21.343 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 12.578454441, 2024-03-19T11:30:21.343 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074752910000000002, 2024-03-19T11:30:21.343 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:21.343 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:21.343 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:21.343 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:21.343 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:21.343 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 845 2024-03-19T11:30:21.343 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.343 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:21.343 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.344 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:21.344 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756359+0000", 2024-03-19T11:30:21.344 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.344 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.344 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.344 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:21.344 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756359+0000", 2024-03-19T11:30:21.344 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.344 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.344 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.344 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:21.344 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756361+0000", 2024-03-19T11:30:21.344 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5280000000000002e-06 2024-03-19T11:30:21.344 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.344 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.345 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:21.345 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756535+0000", 2024-03-19T11:30:21.345 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000173697 2024-03-19T11:30:21.345 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.345 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.345 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:21.345 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756552+0000", 2024-03-19T11:30:21.345 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7397e-05 2024-03-19T11:30:21.345 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.345 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.346 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:21.346 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756561+0000", 2024-03-19T11:30:21.346 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8769999999999999e-06 2024-03-19T11:30:21.346 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.346 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.346 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:21.346 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756614+0000", 2024-03-19T11:30:21.346 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2425e-05 2024-03-19T11:30:21.346 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.346 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.346 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:21.346 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756693+0000", 2024-03-19T11:30:21.346 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9338999999999994e-05 2024-03-19T11:30:21.347 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.347 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.347 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:21.347 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756801+0000", 2024-03-19T11:30:21.347 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000108175 2024-03-19T11:30:21.347 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.347 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.347 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:21.347 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.758827+0000", 2024-03-19T11:30:21.347 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0020263379999999999 2024-03-19T11:30:21.347 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.348 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.348 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:21.348 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.763778+0000", 2024-03-19T11:30:21.348 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0049505740000000001 2024-03-19T11:30:21.348 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.349 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.349 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:21.349 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.763807+0000", 2024-03-19T11:30:21.349 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8702999999999999e-05 2024-03-19T11:30:21.349 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.349 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.349 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:21.349 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.763834+0000", 2024-03-19T11:30:21.349 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.7237999999999999e-05 2024-03-19T11:30:21.349 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.349 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:21.349 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.350 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.350 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.350 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:846 4.7 4:e41d47fb:::smithi03829856-14:head [setxattr _header (56) in=7b,truncate 2974806] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:21.350 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:08.756597+0000", 2024-03-19T11:30:21.350 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 12.578215630999999, 2024-03-19T11:30:21.350 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0092954909999999995, 2024-03-19T11:30:21.350 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:21.350 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:21.351 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:21.351 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:21.351 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:21.351 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 846 2024-03-19T11:30:21.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.351 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:21.351 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.351 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:21.351 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756597+0000", 2024-03-19T11:30:21.351 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.351 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.351 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:21.351 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756597+0000", 2024-03-19T11:30:21.352 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.352 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.352 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.352 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:21.352 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756598+0000", 2024-03-19T11:30:21.352 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4600000000000004e-07 2024-03-19T11:30:21.352 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.352 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.352 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:21.352 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756604+0000", 2024-03-19T11:30:21.352 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5010000000000004e-06 2024-03-19T11:30:21.352 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.353 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.353 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:21.353 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756613+0000", 2024-03-19T11:30:21.353 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3999999999999998e-06 2024-03-19T11:30:21.353 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.353 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.353 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:21.353 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756620+0000", 2024-03-19T11:30:21.353 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.968e-06 2024-03-19T11:30:21.354 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.354 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.354 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:21.354 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.758321+0000", 2024-03-19T11:30:21.354 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001701337 2024-03-19T11:30:21.354 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.354 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.354 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:21.354 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.758414+0000", 2024-03-19T11:30:21.354 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2375999999999998e-05 2024-03-19T11:30:21.354 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.354 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.355 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:21.355 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.758553+0000", 2024-03-19T11:30:21.355 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013910300000000001 2024-03-19T11:30:21.355 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.355 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.355 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:21.355 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.758987+0000", 2024-03-19T11:30:21.355 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00043375100000000002 2024-03-19T11:30:21.355 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.356 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.356 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:21.356 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.765803+0000", 2024-03-19T11:30:21.356 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0068161089999999999 2024-03-19T11:30:21.356 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.357 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.357 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:21.357 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.765846+0000", 2024-03-19T11:30:21.357 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3226000000000003e-05 2024-03-19T11:30:21.357 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.358 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.358 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:21.358 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.765893+0000", 2024-03-19T11:30:21.358 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6974000000000001e-05 2024-03-19T11:30:21.358 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.358 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:21.358 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.358 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.358 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.359 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:847 4.7 4:e41d47fb:::smithi03829856-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:21.359 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:08.756635+0000", 2024-03-19T11:30:21.359 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 12.578177769, 2024-03-19T11:30:21.359 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093966859999999996, 2024-03-19T11:30:21.359 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:21.359 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:21.359 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:21.359 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:21.359 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:21.360 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 847 2024-03-19T11:30:21.360 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.360 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:21.360 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.360 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:21.360 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756635+0000", 2024-03-19T11:30:21.360 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.360 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.360 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.360 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:21.360 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756635+0000", 2024-03-19T11:30:21.360 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.360 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.361 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.361 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:21.361 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756636+0000", 2024-03-19T11:30:21.361 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9500000000000002e-07 2024-03-19T11:30:21.361 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.361 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.361 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:21.361 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756640+0000", 2024-03-19T11:30:21.361 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7349999999999998e-06 2024-03-19T11:30:21.361 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.361 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.361 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:21.361 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756647+0000", 2024-03-19T11:30:21.362 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7130000000000005e-06 2024-03-19T11:30:21.362 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.362 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.362 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:21.362 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756653+0000", 2024-03-19T11:30:21.362 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3229999999999997e-06 2024-03-19T11:30:21.362 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.362 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.362 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:21.362 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.758756+0000", 2024-03-19T11:30:21.362 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0021037170000000002 2024-03-19T11:30:21.362 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.362 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.362 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-03-19T11:30:21.363 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.758814+0000", 2024-03-19T11:30:21.363 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7253999999999998e-05 2024-03-19T11:30:21.363 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.363 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.363 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:21.363 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.765926+0000", 2024-03-19T11:30:21.363 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0071118350000000004 2024-03-19T11:30:21.363 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.363 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.363 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:21.363 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.765970+0000", 2024-03-19T11:30:21.363 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4761000000000001e-05 2024-03-19T11:30:21.363 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.364 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.364 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:21.364 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.766032+0000", 2024-03-19T11:30:21.364 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1753000000000003e-05 2024-03-19T11:30:21.364 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.364 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:21.364 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.364 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.364 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.364 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:998 4.4 4:27909373:::smithi03829856-46:head [rollback 2] snapc 5=[5,2] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-03-19T11:30:21.364 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.368196+0000", 2024-03-19T11:30:21.364 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.9666169050000004, 2024-03-19T11:30:21.364 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0079626200000000001, 2024-03-19T11:30:21.364 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:21.364 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:21.365 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:21.365 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:21.365 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:21.365 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 998 2024-03-19T11:30:21.365 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.365 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:21.365 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.366 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:21.366 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368196+0000", 2024-03-19T11:30:21.366 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.366 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.366 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.366 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:21.366 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368196+0000", 2024-03-19T11:30:21.366 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.366 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.366 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.366 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:21.366 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368197+0000", 2024-03-19T11:30:21.366 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6899999999999998e-07 2024-03-19T11:30:21.366 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.367 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.367 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:21.367 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368200+0000", 2024-03-19T11:30:21.367 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1389999999999999e-06 2024-03-19T11:30:21.367 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.367 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.367 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:21.367 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368207+0000", 2024-03-19T11:30:21.367 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9870000000000002e-06 2024-03-19T11:30:21.367 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.367 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.367 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:21.367 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368212+0000", 2024-03-19T11:30:21.367 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3820000000000003e-06 2024-03-19T11:30:21.367 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.368 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.368 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:21.368 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.369920+0000", 2024-03-19T11:30:21.368 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017078 2024-03-19T11:30:21.368 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.368 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.368 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:21.368 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370087+0000", 2024-03-19T11:30:21.368 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016742800000000001 2024-03-19T11:30:21.368 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.368 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.368 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 2", 2024-03-19T11:30:21.368 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370522+0000", 2024-03-19T11:30:21.368 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00043494300000000003 2024-03-19T11:30:21.369 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.369 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.369 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:21.369 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.375478+0000", 2024-03-19T11:30:21.369 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0049562240000000004 2024-03-19T11:30:21.369 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.369 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.369 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:21.369 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.375918+0000", 2024-03-19T11:30:21.369 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00043914699999999997 2024-03-19T11:30:21.369 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.369 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.369 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:21.370 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.376083+0000", 2024-03-19T11:30:21.370 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0001649 2024-03-19T11:30:21.370 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.370 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.370 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:21.370 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.376159+0000", 2024-03-19T11:30:21.370 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6301e-05 2024-03-19T11:30:21.370 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.370 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:21.370 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.370 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.370 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.370 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:999 4.4 4:27909373:::smithi03829856-46:head [append 0~0] snapc 5=[5,2] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-03-19T11:30:21.370 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.368233+0000", 2024-03-19T11:30:21.371 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.96657993, 2024-03-19T11:30:21.371 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0081209219999999992, 2024-03-19T11:30:21.371 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:21.371 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:21.371 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:21.371 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:21.371 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:21.371 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 999 2024-03-19T11:30:21.371 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.371 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:21.371 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.371 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:21.371 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368233+0000", 2024-03-19T11:30:21.372 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.372 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.372 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.372 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:21.372 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368233+0000", 2024-03-19T11:30:21.372 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.372 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.372 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.372 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:21.372 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368234+0000", 2024-03-19T11:30:21.372 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5400000000000002e-07 2024-03-19T11:30:21.373 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.373 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.373 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:21.373 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368236+0000", 2024-03-19T11:30:21.373 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.7010000000000001e-06 2024-03-19T11:30:21.373 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.373 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.373 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:21.373 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368244+0000", 2024-03-19T11:30:21.373 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.289e-06 2024-03-19T11:30:21.373 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.373 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.373 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:21.373 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368248+0000", 2024-03-19T11:30:21.374 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9429999999999999e-06 2024-03-19T11:30:21.374 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.374 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.374 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:21.374 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372084+0000", 2024-03-19T11:30:21.374 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038354769999999999 2024-03-19T11:30:21.374 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.374 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.374 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:21.374 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372185+0000", 2024-03-19T11:30:21.374 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010088400000000001 2024-03-19T11:30:21.375 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.375 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.375 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 2", 2024-03-19T11:30:21.375 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372376+0000", 2024-03-19T11:30:21.375 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000191545 2024-03-19T11:30:21.375 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.375 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.375 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:21.375 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.376187+0000", 2024-03-19T11:30:21.375 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038108159999999999 2024-03-19T11:30:21.375 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.375 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.375 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:21.375 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.376243+0000", 2024-03-19T11:30:21.375 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6032000000000002e-05 2024-03-19T11:30:21.376 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.376 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.376 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:21.376 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.376304+0000", 2024-03-19T11:30:21.376 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0834000000000001e-05 2024-03-19T11:30:21.376 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.376 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.376 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:21.376 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.376354+0000", 2024-03-19T11:30:21.376 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0046999999999997e-05 2024-03-19T11:30:21.376 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.376 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:21.376 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.376 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.377 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.377 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:53 3.0 3:3dcd2456:::039914993b6f4082c204b24ae0ff925ebffccc22519067b64afa9eff107cf533:head [read 0~243792 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:21.377 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.761757+0000", 2024-03-19T11:30:21.377 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.5730558220000006, 2024-03-19T11:30:21.377 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0076641110000000004, 2024-03-19T11:30:21.378 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:21.378 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:21.378 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:21.378 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.378 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:21.380 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761757+0000", 2024-03-19T11:30:21.381 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.381 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.382 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.383 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:21.383 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761757+0000", 2024-03-19T11:30:21.385 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.385 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.385 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.386 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:21.389 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761759+0000", 2024-03-19T11:30:21.389 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0980000000000001e-06 2024-03-19T11:30:21.389 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.390 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.390 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:21.390 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761771+0000", 2024-03-19T11:30:21.390 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1625e-05 2024-03-19T11:30:21.390 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.391 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.391 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:21.391 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761793+0000", 2024-03-19T11:30:21.391 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1940999999999999e-05 2024-03-19T11:30:21.391 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.391 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.391 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:21.391 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761803+0000", 2024-03-19T11:30:21.391 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0234999999999999e-05 2024-03-19T11:30:21.391 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.391 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.392 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:21.392 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761892+0000", 2024-03-19T11:30:21.392 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8974999999999994e-05 2024-03-19T11:30:21.393 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.394 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.394 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-03-19T11:30:21.394 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761899+0000", 2024-03-19T11:30:21.394 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1709999999999996e-06 2024-03-19T11:30:21.397 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.397 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.399 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:21.399 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.768690+0000", 2024-03-19T11:30:21.400 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.006790941 2024-03-19T11:30:21.400 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.400 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.400 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:21.400 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.768790+0000", 2024-03-19T11:30:21.400 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9569999999999997e-05 2024-03-19T11:30:21.400 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.400 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.400 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:21.401 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.769421+0000", 2024-03-19T11:30:21.401 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00063155500000000001 2024-03-19T11:30:21.401 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.401 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:21.401 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.401 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.401 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.402 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:54 3.0 3:0f3835fc:::79445f5816960fdb5fb7d7699594b1dfc30a0a44639e804aee69d6cb400c02a5:head [read 0~213457 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:21.402 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.761831+0000", 2024-03-19T11:30:21.402 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.5729823029999999, 2024-03-19T11:30:21.402 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0086999490000000002, 2024-03-19T11:30:21.402 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:21.402 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:21.402 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:21.403 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.403 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:21.403 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761831+0000", 2024-03-19T11:30:21.403 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.403 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.403 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.403 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:21.403 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761831+0000", 2024-03-19T11:30:21.404 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.404 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.404 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.404 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:21.404 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761831+0000", 2024-03-19T11:30:21.404 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8699999999999995e-07 2024-03-19T11:30:21.404 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.405 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.405 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:21.405 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761835+0000", 2024-03-19T11:30:21.405 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5039999999999998e-06 2024-03-19T11:30:21.405 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.405 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.405 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:21.406 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761847+0000", 2024-03-19T11:30:21.406 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.242e-05 2024-03-19T11:30:21.406 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.406 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.406 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:21.406 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761855+0000", 2024-03-19T11:30:21.406 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2459999999999997e-06 2024-03-19T11:30:21.407 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.408 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.408 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:21.408 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762004+0000", 2024-03-19T11:30:21.408 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014920499999999999 2024-03-19T11:30:21.408 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.409 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.409 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:21.409 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762009+0000", 2024-03-19T11:30:21.409 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7350000000000001e-06 2024-03-19T11:30:21.409 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.409 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.409 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:21.409 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.769460+0000", 2024-03-19T11:30:21.410 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074506529999999998 2024-03-19T11:30:21.410 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.410 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.410 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:21.410 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.769564+0000", 2024-03-19T11:30:21.410 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000103608 2024-03-19T11:30:21.410 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.410 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.410 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:21.411 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.770531+0000", 2024-03-19T11:30:21.411 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00096699100000000005 2024-03-19T11:30:21.411 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.411 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:21.411 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.412 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.412 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.412 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1055 4.4 4:266590a5:::smithi03829856-16:6 [read 0~3092455,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:21.412 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.768707+0000", 2024-03-19T11:30:21.412 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 7.5661063129999997, 2024-03-19T11:30:21.412 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0088150090000000004, 2024-03-19T11:30:21.412 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:21.412 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:21.413 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:21.413 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:21.413 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:21.413 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1055 2024-03-19T11:30:21.413 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.413 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:21.413 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.414 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:21.414 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768707+0000", 2024-03-19T11:30:21.414 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.414 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.414 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.414 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:21.414 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768707+0000", 2024-03-19T11:30:21.414 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.415 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.415 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.415 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:21.415 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768708+0000", 2024-03-19T11:30:21.415 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2200000000000003e-07 2024-03-19T11:30:21.415 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.416 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.416 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:21.416 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768716+0000", 2024-03-19T11:30:21.416 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4810000000000006e-06 2024-03-19T11:30:21.416 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.416 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.416 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:21.417 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768728+0000", 2024-03-19T11:30:21.417 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2167999999999999e-05 2024-03-19T11:30:21.417 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.417 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.417 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:21.417 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768737+0000", 2024-03-19T11:30:21.417 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8419999999999994e-06 2024-03-19T11:30:21.417 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.418 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.419 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:21.419 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768895+0000", 2024-03-19T11:30:21.419 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000157455 2024-03-19T11:30:21.419 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.420 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.420 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:21.420 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768897+0000", 2024-03-19T11:30:21.420 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.7580000000000001e-06 2024-03-19T11:30:21.420 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.420 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.421 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:21.421 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.775284+0000", 2024-03-19T11:30:21.421 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0063862069999999996 2024-03-19T11:30:21.421 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.421 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.421 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:21.421 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.775362+0000", 2024-03-19T11:30:21.421 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8604999999999997e-05 2024-03-19T11:30:21.422 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.422 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.422 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:21.422 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.777522+0000", 2024-03-19T11:30:21.422 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0021596710000000002 2024-03-19T11:30:21.422 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.422 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:21.422 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.422 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.423 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.423 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1056 4.4 4:266590a5:::smithi03829856-16:6 [read 0~3092455,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:21.423 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.768750+0000", 2024-03-19T11:30:21.423 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 7.5660635090000001, 2024-03-19T11:30:21.423 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010087227000000001, 2024-03-19T11:30:21.423 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:21.423 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:21.424 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:21.424 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:21.424 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:21.424 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1056 2024-03-19T11:30:21.424 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.425 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:21.425 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.425 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:21.425 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768750+0000", 2024-03-19T11:30:21.425 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.425 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.425 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.426 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:21.426 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768750+0000", 2024-03-19T11:30:21.426 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.430 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.430 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.430 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:21.430 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768750+0000", 2024-03-19T11:30:21.430 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6199999999999997e-07 2024-03-19T11:30:21.430 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.430 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.430 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:21.430 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768754+0000", 2024-03-19T11:30:21.430 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.9920000000000001e-06 2024-03-19T11:30:21.430 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.430 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.431 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:21.431 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768764+0000", 2024-03-19T11:30:21.431 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0052e-05 2024-03-19T11:30:21.431 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.431 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.431 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:21.431 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768773+0000", 2024-03-19T11:30:21.431 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7309999999999999e-06 2024-03-19T11:30:21.431 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.431 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.431 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:21.431 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768982+0000", 2024-03-19T11:30:21.431 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000208653 2024-03-19T11:30:21.431 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.431 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.432 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:21.432 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768990+0000", 2024-03-19T11:30:21.432 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3000000000000002e-06 2024-03-19T11:30:21.432 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.432 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.432 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:21.432 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.777544+0000", 2024-03-19T11:30:21.432 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.008554374 2024-03-19T11:30:21.432 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.432 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.432 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:21.432 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.777599+0000", 2024-03-19T11:30:21.432 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4880999999999999e-05 2024-03-19T11:30:21.432 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.432 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.432 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:21.433 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.778837+0000", 2024-03-19T11:30:21.433 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0012375820000000001 2024-03-19T11:30:21.433 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.433 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:21.433 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.433 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.433 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.433 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1063 4.4 4:214026b3:::smithi03829856-47:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:21.433 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.784904+0000", 2024-03-19T11:30:21.433 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 7.5499090679999998, 2024-03-19T11:30:21.433 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024425912000000001, 2024-03-19T11:30:21.433 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:21.433 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:21.433 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:21.433 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:21.434 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:21.434 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1063 2024-03-19T11:30:21.434 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.434 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:21.434 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.434 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:21.434 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784904+0000", 2024-03-19T11:30:21.434 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.434 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.434 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.434 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:21.434 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784904+0000", 2024-03-19T11:30:21.434 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.434 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.434 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.435 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:21.435 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784906+0000", 2024-03-19T11:30:21.435 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1409999999999999e-06 2024-03-19T11:30:21.435 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.435 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.435 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:21.435 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784915+0000", 2024-03-19T11:30:21.435 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5459999999999992e-06 2024-03-19T11:30:21.435 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.435 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.435 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:21.435 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784935+0000", 2024-03-19T11:30:21.435 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0310999999999999e-05 2024-03-19T11:30:21.435 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.435 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.436 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:21.436 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784946+0000", 2024-03-19T11:30:21.436 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0675e-05 2024-03-19T11:30:21.436 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.436 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.436 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:21.436 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.785032+0000", 2024-03-19T11:30:21.436 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6049000000000002e-05 2024-03-19T11:30:21.436 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.436 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.436 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:21.436 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.785136+0000", 2024-03-19T11:30:21.436 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000104209 2024-03-19T11:30:21.436 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.436 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.437 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:21.437 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.809330+0000", 2024-03-19T11:30:21.437 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024193981 2024-03-19T11:30:21.437 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.437 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:21.437 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.437 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.437 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.437 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1077 4.7 4:e41d47fb:::smithi03829856-14:head [write 510946~632952,stat] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:21.437 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.993080+0000", 2024-03-19T11:30:21.437 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 7.3417331470000002, 2024-03-19T11:30:21.437 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0095463609999999997, 2024-03-19T11:30:21.437 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:21.437 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:21.438 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:21.438 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:21.438 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:21.438 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1077 2024-03-19T11:30:21.438 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.438 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:21.438 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.438 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:21.438 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.993080+0000", 2024-03-19T11:30:21.438 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.438 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.438 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.438 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:21.438 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.993080+0000", 2024-03-19T11:30:21.438 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.439 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.439 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.439 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:21.439 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.993082+0000", 2024-03-19T11:30:21.439 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3199999999999998e-06 2024-03-19T11:30:21.439 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.439 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.439 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:21.439 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.993356+0000", 2024-03-19T11:30:21.439 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00027414699999999998 2024-03-19T11:30:21.439 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.439 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.439 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:21.439 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.993377+0000", 2024-03-19T11:30:21.439 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1015000000000001e-05 2024-03-19T11:30:21.440 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.440 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.440 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:21.440 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.993387+0000", 2024-03-19T11:30:21.440 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.5200000000000003e-06 2024-03-19T11:30:21.440 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.440 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.440 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:21.440 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.993468+0000", 2024-03-19T11:30:21.440 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1145999999999999e-05 2024-03-19T11:30:21.440 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.440 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.440 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:21.440 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.993595+0000", 2024-03-19T11:30:21.441 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012649199999999999 2024-03-19T11:30:21.441 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.441 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.441 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:21.441 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.993826+0000", 2024-03-19T11:30:21.441 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000231616 2024-03-19T11:30:21.441 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.441 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.441 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:21.441 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.997639+0000", 2024-03-19T11:30:21.441 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038124830000000002 2024-03-19T11:30:21.441 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.441 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.441 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:21.441 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.002505+0000", 2024-03-19T11:30:21.442 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0048662339999999997 2024-03-19T11:30:21.442 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.442 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.442 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:21.442 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.002569+0000", 2024-03-19T11:30:21.442 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3629999999999999e-05 2024-03-19T11:30:21.442 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.442 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.442 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:21.442 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.002626+0000", 2024-03-19T11:30:21.442 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7757999999999998e-05 2024-03-19T11:30:21.442 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.442 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:21.442 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.442 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.443 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.443 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1081 4.7 4:e41d47fb:::smithi03829856-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:21.443 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:14.006496+0000", 2024-03-19T11:30:21.443 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 7.3283171720000002, 2024-03-19T11:30:21.443 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0070742670000000004, 2024-03-19T11:30:21.443 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:21.443 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:21.443 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:21.443 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:21.443 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:21.443 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1081 2024-03-19T11:30:21.443 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.443 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:21.443 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.443 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:21.444 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006496+0000", 2024-03-19T11:30:21.444 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.444 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.444 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.444 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:21.444 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006496+0000", 2024-03-19T11:30:21.444 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.444 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.444 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.444 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:21.444 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006496+0000", 2024-03-19T11:30:21.444 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2100000000000002e-07 2024-03-19T11:30:21.444 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.444 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.444 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:21.444 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006499+0000", 2024-03-19T11:30:21.445 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5560000000000002e-06 2024-03-19T11:30:21.445 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.445 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.445 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:21.445 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006506+0000", 2024-03-19T11:30:21.445 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4240000000000004e-06 2024-03-19T11:30:21.445 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.445 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.445 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:21.445 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006515+0000", 2024-03-19T11:30:21.445 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5949999999999999e-06 2024-03-19T11:30:21.445 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.445 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.445 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:21.445 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.009510+0000", 2024-03-19T11:30:21.446 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002995546 2024-03-19T11:30:21.446 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.446 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.446 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-03-19T11:30:21.446 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.009586+0000", 2024-03-19T11:30:21.446 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5859999999999995e-05 2024-03-19T11:30:21.446 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.446 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.446 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:21.446 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.013438+0000", 2024-03-19T11:30:21.446 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038516850000000001 2024-03-19T11:30:21.446 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.446 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.446 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:21.446 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.013499+0000", 2024-03-19T11:30:21.447 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0951e-05 2024-03-19T11:30:21.447 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.447 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.447 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:21.447 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.013570+0000", 2024-03-19T11:30:21.447 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1229000000000005e-05 2024-03-19T11:30:21.447 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.447 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:21.447 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.447 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.447 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.447 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1215 3.0 3:0c9406a2:::98f4ba27fa68608c540dca9b03a4531ee556ff2734aa426809de93c45c84ab96:head [write 0~1520640] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:21.447 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.147833+0000", 2024-03-19T11:30:21.447 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.1869803079999999, 2024-03-19T11:30:21.447 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0078358579999999994, 2024-03-19T11:30:21.448 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:21.448 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:21.448 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:21.448 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:21.448 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:21.448 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1215 2024-03-19T11:30:21.448 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.448 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:21.448 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.448 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:21.448 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.147833+0000", 2024-03-19T11:30:21.448 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.448 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.448 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.448 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:21.448 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.147833+0000", 2024-03-19T11:30:21.449 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.449 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.449 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.449 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:21.449 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.147835+0000", 2024-03-19T11:30:21.449 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5119999999999998e-06 2024-03-19T11:30:21.449 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.449 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.449 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:21.449 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.148763+0000", 2024-03-19T11:30:21.449 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00092776399999999998 2024-03-19T11:30:21.449 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.449 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.449 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:21.450 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.148792+0000", 2024-03-19T11:30:21.450 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9040999999999999e-05 2024-03-19T11:30:21.450 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.450 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.450 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:21.450 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.148805+0000", 2024-03-19T11:30:21.450 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3079e-05 2024-03-19T11:30:21.450 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.450 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.450 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:21.450 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.148897+0000", 2024-03-19T11:30:21.450 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1942999999999997e-05 2024-03-19T11:30:21.450 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.450 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.450 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-03-19T11:30:21.451 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.148906+0000", 2024-03-19T11:30:21.451 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5050000000000007e-06 2024-03-19T11:30:21.451 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.451 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.451 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:21.451 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.150778+0000", 2024-03-19T11:30:21.451 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0018720880000000001 2024-03-19T11:30:21.451 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.451 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.451 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:21.451 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.150930+0000", 2024-03-19T11:30:21.451 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015192299999999999 2024-03-19T11:30:21.451 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.451 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.451 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 2", 2024-03-19T11:30:21.452 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.151086+0000", 2024-03-19T11:30:21.452 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015680199999999999 2024-03-19T11:30:21.452 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.452 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.452 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:21.452 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155119+0000", 2024-03-19T11:30:21.452 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0040322680000000003 2024-03-19T11:30:21.452 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.452 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.452 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:21.452 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155570+0000", 2024-03-19T11:30:21.452 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00045105999999999998 2024-03-19T11:30:21.452 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.452 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.452 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:21.453 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155623+0000", 2024-03-19T11:30:21.453 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3133000000000001e-05 2024-03-19T11:30:21.453 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.453 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.453 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:21.453 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155669+0000", 2024-03-19T11:30:21.453 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5739999999999999e-05 2024-03-19T11:30:21.453 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.453 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:21.453 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.453 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.453 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.453 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.0.13:55 3.0 3:0d712e74:::bb863e2508028016f9ab1e6296121a5f2f252d02d0ddc1e79616682e5e6f5b3c:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:21.453 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.170762+0000", 2024-03-19T11:30:21.453 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.1640512420000002, 2024-03-19T11:30:21.453 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077288690000000002, 2024-03-19T11:30:21.454 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:21.454 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:21.454 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:21.454 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.454 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:21.454 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.170762+0000", 2024-03-19T11:30:21.454 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.454 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.454 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.454 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:21.454 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.170762+0000", 2024-03-19T11:30:21.454 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.454 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.454 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.455 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:21.455 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.170765+0000", 2024-03-19T11:30:21.455 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1159999999999999e-06 2024-03-19T11:30:21.455 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.455 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.455 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:21.455 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173312+0000", 2024-03-19T11:30:21.455 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025465319999999998 2024-03-19T11:30:21.455 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.455 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.455 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:21.455 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173370+0000", 2024-03-19T11:30:21.455 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8050999999999997e-05 2024-03-19T11:30:21.455 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.455 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.456 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:21.456 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173397+0000", 2024-03-19T11:30:21.456 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.7874000000000001e-05 2024-03-19T11:30:21.456 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.456 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.456 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:21.456 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173562+0000", 2024-03-19T11:30:21.456 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016487300000000001 2024-03-19T11:30:21.456 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.456 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.456 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:21.456 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173742+0000", 2024-03-19T11:30:21.456 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017933200000000001 2024-03-19T11:30:21.456 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.456 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.457 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 2", 2024-03-19T11:30:21.457 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.174093+0000", 2024-03-19T11:30:21.457 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000351257 2024-03-19T11:30:21.457 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.457 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.457 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:21.457 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177271+0000", 2024-03-19T11:30:21.457 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031785210000000001 2024-03-19T11:30:21.457 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.457 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.457 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:21.457 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.178330+0000", 2024-03-19T11:30:21.457 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00105832 2024-03-19T11:30:21.457 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.457 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.457 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:21.458 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.178432+0000", 2024-03-19T11:30:21.458 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000102709 2024-03-19T11:30:21.458 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.458 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.458 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:21.458 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.178491+0000", 2024-03-19T11:30:21.458 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8284e-05 2024-03-19T11:30:21.458 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.458 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:21.458 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.458 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.458 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.458 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1235 4.7 4:e41d47fb:::smithi03829856-14:head [write 679016~701795,stat] snapc 9=[9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:21.458 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.331979+0000", 2024-03-19T11:30:21.459 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.0028344809999998, 2024-03-19T11:30:21.459 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012331527, 2024-03-19T11:30:21.459 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:21.459 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:21.459 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:21.459 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:21.459 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:21.459 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1235 2024-03-19T11:30:21.459 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.459 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:21.459 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.459 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:21.459 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.331979+0000", 2024-03-19T11:30:21.459 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.459 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.460 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.460 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:21.460 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.331979+0000", 2024-03-19T11:30:21.460 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.460 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.460 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.460 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:21.460 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.331981+0000", 2024-03-19T11:30:21.460 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.593e-06 2024-03-19T11:30:21.460 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.460 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.460 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:21.460 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.332253+0000", 2024-03-19T11:30:21.460 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00027209199999999998 2024-03-19T11:30:21.460 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.460 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.461 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:21.461 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.332289+0000", 2024-03-19T11:30:21.461 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5519999999999999e-05 2024-03-19T11:30:21.461 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.461 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.461 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:21.461 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.332314+0000", 2024-03-19T11:30:21.461 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5151e-05 2024-03-19T11:30:21.461 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.461 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.461 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:21.461 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.332446+0000", 2024-03-19T11:30:21.461 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013162100000000001 2024-03-19T11:30:21.461 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.461 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.462 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:21.462 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.332546+0000", 2024-03-19T11:30:21.462 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000100382 2024-03-19T11:30:21.462 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.462 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.462 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:21.462 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.332718+0000", 2024-03-19T11:30:21.462 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017210700000000001 2024-03-19T11:30:21.462 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.462 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.462 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:21.462 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.336840+0000", 2024-03-19T11:30:21.462 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0041217279999999999 2024-03-19T11:30:21.462 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.463 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.463 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:21.463 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.344205+0000", 2024-03-19T11:30:21.463 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0073654389999999997 2024-03-19T11:30:21.463 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.463 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.463 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:21.463 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.344258+0000", 2024-03-19T11:30:21.463 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2598e-05 2024-03-19T11:30:21.463 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.463 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.463 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:21.463 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.344310+0000", 2024-03-19T11:30:21.463 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2296000000000002e-05 2024-03-19T11:30:21.463 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.464 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:21.464 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.464 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.464 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.464 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1236 4.7 4:e41d47fb:::smithi03829856-14:head [write 1866533~496024] snapc 9=[9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:21.464 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.338603+0000", 2024-03-19T11:30:21.464 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 2.9962103249999998, 2024-03-19T11:30:21.464 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011626472000000001, 2024-03-19T11:30:21.464 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:21.464 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:21.464 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:21.464 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:21.464 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:21.464 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1236 2024-03-19T11:30:21.464 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.465 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:21.465 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.465 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:21.465 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.338603+0000", 2024-03-19T11:30:21.465 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.465 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.465 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.465 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:21.465 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.338603+0000", 2024-03-19T11:30:21.465 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.465 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.465 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.465 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:21.465 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.338604+0000", 2024-03-19T11:30:21.466 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5799999999999999e-06 2024-03-19T11:30:21.466 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.466 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.466 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:21.466 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.338918+0000", 2024-03-19T11:30:21.466 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000313663 2024-03-19T11:30:21.466 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.466 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.466 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:21.466 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.338935+0000", 2024-03-19T11:30:21.466 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7418000000000002e-05 2024-03-19T11:30:21.466 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.466 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.466 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:21.467 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.338945+0000", 2024-03-19T11:30:21.467 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4390000000000006e-06 2024-03-19T11:30:21.467 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.467 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.467 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:21.467 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.339011+0000", 2024-03-19T11:30:21.467 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5816999999999997e-05 2024-03-19T11:30:21.467 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.467 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.467 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:21.467 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.339098+0000", 2024-03-19T11:30:21.467 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7658999999999995e-05 2024-03-19T11:30:21.467 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.467 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.467 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:21.468 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.339216+0000", 2024-03-19T11:30:21.468 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011810499999999999 2024-03-19T11:30:21.468 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.468 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.468 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:21.468 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.341808+0000", 2024-03-19T11:30:21.468 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025918059999999999 2024-03-19T11:30:21.468 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.468 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.468 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:21.468 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.350075+0000", 2024-03-19T11:30:21.468 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0082672360000000007 2024-03-19T11:30:21.468 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.468 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.468 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:21.469 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.350144+0000", 2024-03-19T11:30:21.469 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8796999999999999e-05 2024-03-19T11:30:21.469 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.469 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.469 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:21.469 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.350229+0000", 2024-03-19T11:30:21.469 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4951999999999996e-05 2024-03-19T11:30:21.469 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.469 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:21.469 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.469 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.469 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.469 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(osd.4.0:197 4.9 e41/21 4:9ef41243:::smithi03829856-5:head v 41'36, mlcod=32'33)", 2024-03-19T11:30:21.469 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.352401+0000", 2024-03-19T11:30:21.469 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 2.9824120220000001, 2024-03-19T11:30:21.470 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0079977950000000003, 2024-03-19T11:30:21.470 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:21.470 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:21.470 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:21.470 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.470 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:21.470 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.352401+0000", 2024-03-19T11:30:21.470 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.470 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.470 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.470 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:21.470 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.352401+0000", 2024-03-19T11:30:21.470 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.470 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.470 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.471 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:21.471 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.352401+0000", 2024-03-19T11:30:21.471 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.35e-07 2024-03-19T11:30:21.471 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.471 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.471 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:21.471 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.355415+0000", 2024-03-19T11:30:21.471 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030131899999999998 2024-03-19T11:30:21.471 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.471 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.471 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:21.471 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.355443+0000", 2024-03-19T11:30:21.471 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8710000000000001e-05 2024-03-19T11:30:21.471 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.471 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.472 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:21.472 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.355456+0000", 2024-03-19T11:30:21.472 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2489e-05 2024-03-19T11:30:21.472 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.472 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.472 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:21.472 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.355552+0000", 2024-03-19T11:30:21.472 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6523999999999999e-05 2024-03-19T11:30:21.472 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.472 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.472 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:21.472 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.355586+0000", 2024-03-19T11:30:21.472 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3694e-05 2024-03-19T11:30:21.472 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.472 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.473 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:21.473 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.360345+0000", 2024-03-19T11:30:21.473 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0047591309999999998 2024-03-19T11:30:21.473 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.473 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.473 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:21.473 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.360399+0000", 2024-03-19T11:30:21.473 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3822000000000001e-05 2024-03-19T11:30:21.473 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.473 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:21.473 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.473 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.473 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:21.473 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:21.475 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-19T11:30:21.475 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:21.486+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:21.576 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:21.586+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:21.607 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:21.607 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:30:21.607 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:30:21.607 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:21.621 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-19T11:30:21.676 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:21.688+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:21.747 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:21.747 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:30:21.747 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:30:21.747 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:30:21.747 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:21.761 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-19T11:30:21.778 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:21.790+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:21.853 INFO:tasks.rados.rados.0.smithi038.stdout:782: done (3 left) 2024-03-19T11:30:21.853 INFO:tasks.rados.rados.0.smithi038.stdout:783: set_chunk oid 45 target oid 2024-03-19T11:30:21.879 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:21.890+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:21.917 INFO:tasks.rados.rados.0.smithi038.stdout:783: set_chunk oid 45 offset: 795136 length: 1008128 target oid 03fdcc249fa0f5f142a5d2f0c8cc0fb175aa6aa851954545163110fc1226886a offset: 0 snap 0 2024-03-19T11:30:21.917 INFO:tasks.rados.rados.0.smithi038.stdout:784: read oid 26 snap -1 2024-03-19T11:30:21.917 INFO:tasks.rados.rados.0.smithi038.stdout:784: expect deleted 2024-03-19T11:30:21.918 INFO:tasks.rados.rados.0.smithi038.stdout:785: write (excl) oid 19 current snap is 10 2024-03-19T11:30:21.918 INFO:tasks.rados.rados.0.smithi038.stdout:778: got (0) Success 2024-03-19T11:30:21.918 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 47 v 15 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-03-19T11:30:21.918 INFO:tasks.rados.rados.0.smithi038.stdout:779: finishing write tid 1 to smithi03829856-31 2024-03-19T11:30:21.918 INFO:tasks.rados.rados.0.smithi038.stdout:779: finishing write tid 2 to smithi03829856-31 2024-03-19T11:30:21.918 INFO:tasks.rados.rados.0.smithi038.stdout:779: finishing write tid 3 to smithi03829856-31 2024-03-19T11:30:21.918 INFO:tasks.rados.rados.0.smithi038.stdout:779: finishing write tid 4 to smithi03829856-31 2024-03-19T11:30:21.918 INFO:tasks.rados.rados.0.smithi038.stdout:779: finishing write tid 5 to smithi03829856-31 2024-03-19T11:30:21.918 INFO:tasks.rados.rados.0.smithi038.stdout:779: oid 31 updating version 0 to 54 2024-03-19T11:30:21.918 INFO:tasks.rados.rados.0.smithi038.stdout:779: oid 31 updating version 54 to 56 2024-03-19T11:30:21.918 INFO:tasks.rados.rados.0.smithi038.stdout:779: oid 31 version 56 is already newer than 53 2024-03-19T11:30:21.918 INFO:tasks.rados.rados.0.smithi038.stdout:779: oid 31 version 56 is already newer than 55 2024-03-19T11:30:21.918 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 31 v 56 (ObjNum 89 snap 10 seq_num 89) dirty exists 2024-03-19T11:30:21.918 INFO:tasks.rados.rados.0.smithi038.stdout:779: left oid 31 (ObjNum 89 snap 10 seq_num 89) 2024-03-19T11:30:21.919 INFO:tasks.rados.rados.0.smithi038.stdout:781: finishing write tid 1 to smithi03829856-43 2024-03-19T11:30:21.919 INFO:tasks.rados.rados.0.smithi038.stdout:781: finishing write tid 2 to smithi03829856-43 2024-03-19T11:30:21.919 INFO:tasks.rados.rados.0.smithi038.stdout:781: finishing write tid 3 to smithi03829856-43 2024-03-19T11:30:21.919 INFO:tasks.rados.rados.0.smithi038.stdout:781: finishing write tid 4 to smithi03829856-43 2024-03-19T11:30:21.919 INFO:tasks.rados.rados.0.smithi038.stdout:781: finishing write tid 5 to smithi03829856-43 2024-03-19T11:30:21.919 INFO:tasks.rados.rados.0.smithi038.stdout:781: oid 43 updating version 0 to 61 2024-03-19T11:30:21.919 INFO:tasks.rados.rados.0.smithi038.stdout:781: oid 43 version 61 is already newer than 59 2024-03-19T11:30:21.919 INFO:tasks.rados.rados.0.smithi038.stdout:781: oid 43 version 61 is already newer than 58 2024-03-19T11:30:21.919 INFO:tasks.rados.rados.0.smithi038.stdout:781: oid 43 version 61 is already newer than 60 2024-03-19T11:30:21.919 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 43 v 61 (ObjNum 91 snap 10 seq_num 91) dirty exists 2024-03-19T11:30:21.919 INFO:tasks.rados.rados.0.smithi038.stdout:781: left oid 43 (ObjNum 91 snap 10 seq_num 91) 2024-03-19T11:30:21.919 INFO:tasks.rados.rados.0.smithi038.stdout:785: seq_num 93 ranges {763051=794393,2158943=647020,3437750=125301} 2024-03-19T11:30:21.939 INFO:tasks.rados.rados.0.smithi038.stdout:785: writing smithi03829856-19 from 763051 to 1557444 tid 1 2024-03-19T11:30:21.943 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:21.943 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:30:21.943 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:30:21.943 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:30:21.943 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.943 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:9 4.d e28/21 4:b4ef97e0:::smithi03829856-3:head v 28'1, mlcod=0'0)", 2024-03-19T11:30:21.946 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.707728+0000", 2024-03-19T11:30:21.946 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 17.245367560999998, 2024-03-19T11:30:21.946 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0037487950000000001, 2024-03-19T11:30:21.946 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:21.946 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:21.946 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:21.946 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.946 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:21.946 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.707728+0000", 2024-03-19T11:30:21.946 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.946 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.947 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.947 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:21.947 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.707728+0000", 2024-03-19T11:30:21.947 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.947 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.947 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.947 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:21.947 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.707729+0000", 2024-03-19T11:30:21.947 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6899999999999998e-07 2024-03-19T11:30:21.947 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.947 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.947 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:21.947 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.708625+0000", 2024-03-19T11:30:21.947 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00089603899999999995 2024-03-19T11:30:21.948 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.948 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.948 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:21.948 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.708645+0000", 2024-03-19T11:30:21.948 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9629000000000001e-05 2024-03-19T11:30:21.948 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.948 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.948 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:21.948 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.708657+0000", 2024-03-19T11:30:21.948 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2537999999999999e-05 2024-03-19T11:30:21.948 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.948 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.948 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:21.948 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.708740+0000", 2024-03-19T11:30:21.948 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2490999999999993e-05 2024-03-19T11:30:21.949 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.949 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.949 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:21.949 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.708774+0000", 2024-03-19T11:30:21.949 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4362000000000001e-05 2024-03-19T11:30:21.949 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.949 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.949 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:21.949 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.711419+0000", 2024-03-19T11:30:21.949 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0026446830000000001 2024-03-19T11:30:21.949 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.949 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.949 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:21.949 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.711477+0000", 2024-03-19T11:30:21.949 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8684000000000003e-05 2024-03-19T11:30:21.950 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.950 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:21.950 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.950 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.950 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.950 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:10 4.d e28/21 4:b4ef97e0:::smithi03829856-3:head v 28'2, mlcod=0'0)", 2024-03-19T11:30:21.950 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.713599+0000", 2024-03-19T11:30:21.950 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 17.239497103000001, 2024-03-19T11:30:21.950 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002599905, 2024-03-19T11:30:21.950 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:21.950 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:21.950 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:21.950 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.950 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:21.951 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.713599+0000", 2024-03-19T11:30:21.951 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.951 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.951 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.951 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:21.951 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.713599+0000", 2024-03-19T11:30:21.951 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.951 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.951 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.951 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:21.951 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.713619+0000", 2024-03-19T11:30:21.951 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0398999999999999e-05 2024-03-19T11:30:21.951 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.951 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.951 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:21.952 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.713955+0000", 2024-03-19T11:30:21.952 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000335691 2024-03-19T11:30:21.952 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.952 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.952 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:21.952 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.713976+0000", 2024-03-19T11:30:21.952 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0866999999999999e-05 2024-03-19T11:30:21.952 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.952 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.952 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:21.952 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.713988+0000", 2024-03-19T11:30:21.952 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2228e-05 2024-03-19T11:30:21.952 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.952 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.952 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:21.953 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.714058+0000", 2024-03-19T11:30:21.953 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9868000000000003e-05 2024-03-19T11:30:21.953 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.953 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.953 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:21.953 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.714086+0000", 2024-03-19T11:30:21.953 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8124999999999999e-05 2024-03-19T11:30:21.953 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.953 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.953 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:21.953 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.716138+0000", 2024-03-19T11:30:21.953 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0020517759999999999 2024-03-19T11:30:21.953 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.953 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.953 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:21.954 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.716199+0000", 2024-03-19T11:30:21.954 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0951e-05 2024-03-19T11:30:21.954 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.954 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:21.954 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.954 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.954 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.954 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:13 4.4 e28/21 4:21eac9b2:::smithi03829856-4:head v 28'1, mlcod=0'0)", 2024-03-19T11:30:21.954 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.723258+0000", 2024-03-19T11:30:21.954 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 17.229837566, 2024-03-19T11:30:21.954 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0048979840000000002, 2024-03-19T11:30:21.954 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:21.954 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:21.954 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:21.954 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.955 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:21.955 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723258+0000", 2024-03-19T11:30:21.955 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.955 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.955 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.955 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:21.955 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723258+0000", 2024-03-19T11:30:21.955 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.955 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.955 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.955 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:21.955 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723259+0000", 2024-03-19T11:30:21.955 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.7399999999999999e-07 2024-03-19T11:30:21.955 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.955 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.956 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:21.956 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723812+0000", 2024-03-19T11:30:21.956 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00055371000000000001 2024-03-19T11:30:21.956 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.956 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.956 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:21.956 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723833+0000", 2024-03-19T11:30:21.956 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0990000000000001e-05 2024-03-19T11:30:21.956 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.956 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.956 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:21.956 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723847+0000", 2024-03-19T11:30:21.956 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3774000000000001e-05 2024-03-19T11:30:21.956 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.956 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.957 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:21.957 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723912+0000", 2024-03-19T11:30:21.957 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4758000000000006e-05 2024-03-19T11:30:21.957 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.957 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.957 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:21.957 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723933+0000", 2024-03-19T11:30:21.957 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1046000000000001e-05 2024-03-19T11:30:21.957 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.957 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.957 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:21.957 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.728103+0000", 2024-03-19T11:30:21.957 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0041695079999999997 2024-03-19T11:30:21.957 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.958 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.958 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:21.958 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.728156+0000", 2024-03-19T11:30:21.958 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3924000000000001e-05 2024-03-19T11:30:21.958 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.958 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:21.958 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.958 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.958 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.958 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:14 4.4 e28/21 4:21eac9b2:::smithi03829856-4:head v 28'2, mlcod=0'0)", 2024-03-19T11:30:21.958 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.726103+0000", 2024-03-19T11:30:21.958 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 17.226992716000002, 2024-03-19T11:30:21.958 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025805440000000002, 2024-03-19T11:30:21.958 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:21.958 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:21.959 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:21.959 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.959 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:21.959 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.726103+0000", 2024-03-19T11:30:21.959 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.959 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.959 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.959 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:21.959 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.726103+0000", 2024-03-19T11:30:21.959 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.960 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.960 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.960 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:21.960 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.726104+0000", 2024-03-19T11:30:21.960 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0920000000000001e-06 2024-03-19T11:30:21.960 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.960 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.961 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:21.961 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.726321+0000", 2024-03-19T11:30:21.961 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00021668499999999999 2024-03-19T11:30:21.961 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.961 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.961 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:21.961 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.726338+0000", 2024-03-19T11:30:21.961 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6382999999999999e-05 2024-03-19T11:30:21.961 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.961 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.961 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:21.961 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.726347+0000", 2024-03-19T11:30:21.961 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7399999999999999e-06 2024-03-19T11:30:21.961 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.961 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.962 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:21.962 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.726422+0000", 2024-03-19T11:30:21.962 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4365999999999999e-05 2024-03-19T11:30:21.962 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.962 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.962 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:21.962 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.726451+0000", 2024-03-19T11:30:21.962 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9085e-05 2024-03-19T11:30:21.962 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.962 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.962 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:21.962 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.728643+0000", 2024-03-19T11:30:21.962 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0021926620000000002 2024-03-19T11:30:21.962 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.963 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.963 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:21.963 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.728684+0000", 2024-03-19T11:30:21.963 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0531000000000002e-05 2024-03-19T11:30:21.963 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.963 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:21.963 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.963 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.963 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.963 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:33 4.d e28/21 4:b65e6658:::smithi03829856-9:head v 28'4, mlcod=28'2)", 2024-03-19T11:30:21.963 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.785780+0000", 2024-03-19T11:30:21.963 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 17.167315795, 2024-03-19T11:30:21.963 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0032708569999999998, 2024-03-19T11:30:21.963 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:21.963 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:21.964 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:21.964 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.964 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:21.964 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.785780+0000", 2024-03-19T11:30:21.964 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.964 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.964 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.964 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:21.964 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.785780+0000", 2024-03-19T11:30:21.964 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.964 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.964 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.964 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:21.964 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.785783+0000", 2024-03-19T11:30:21.965 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4150000000000002e-06 2024-03-19T11:30:21.965 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.965 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.965 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:21.965 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.786482+0000", 2024-03-19T11:30:21.965 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00069957299999999999 2024-03-19T11:30:21.965 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.965 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.965 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:21.965 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.786507+0000", 2024-03-19T11:30:21.965 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5160999999999998e-05 2024-03-19T11:30:21.966 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.966 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.966 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:21.966 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.786521+0000", 2024-03-19T11:30:21.966 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3301e-05 2024-03-19T11:30:21.966 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.966 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.966 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:21.966 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.786599+0000", 2024-03-19T11:30:21.967 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7905999999999996e-05 2024-03-19T11:30:21.967 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.967 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.967 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:21.967 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.786630+0000", 2024-03-19T11:30:21.967 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1636e-05 2024-03-19T11:30:21.967 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.967 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.967 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:21.967 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.789001+0000", 2024-03-19T11:30:21.967 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023710530000000001 2024-03-19T11:30:21.967 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.967 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.968 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:21.968 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.789051+0000", 2024-03-19T11:30:21.968 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9811999999999997e-05 2024-03-19T11:30:21.968 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.968 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:21.968 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.968 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.968 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.968 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:871 4.d e32/21 4:b4ef97e0:::smithi03829856-3:head v 32'19, mlcod=28'16)", 2024-03-19T11:30:21.968 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.029331+0000", 2024-03-19T11:30:21.968 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 12.923764893, 2024-03-19T11:30:21.969 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057816339999999999, 2024-03-19T11:30:21.969 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:21.969 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:21.969 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:21.969 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.969 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:21.969 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.029331+0000", 2024-03-19T11:30:21.969 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.969 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.969 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.969 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:21.969 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.029331+0000", 2024-03-19T11:30:21.969 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.969 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.970 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.970 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:21.970 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.029331+0000", 2024-03-19T11:30:21.970 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0400000000000002e-07 2024-03-19T11:30:21.970 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.970 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.970 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:21.970 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.030605+0000", 2024-03-19T11:30:21.970 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0012733899999999999 2024-03-19T11:30:21.970 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.970 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.970 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:21.970 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.030620+0000", 2024-03-19T11:30:21.970 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5557999999999999e-05 2024-03-19T11:30:21.970 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.971 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.971 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:21.971 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.030630+0000", 2024-03-19T11:30:21.971 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2660000000000007e-06 2024-03-19T11:30:21.971 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.971 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.971 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:21.971 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.030727+0000", 2024-03-19T11:30:21.971 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7452000000000001e-05 2024-03-19T11:30:21.971 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.971 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.971 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:21.971 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.030757+0000", 2024-03-19T11:30:21.972 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0018999999999998e-05 2024-03-19T11:30:21.972 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.972 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.972 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:21.972 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.035063+0000", 2024-03-19T11:30:21.972 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0043055350000000001 2024-03-19T11:30:21.972 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.972 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.972 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:21.972 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.035113+0000", 2024-03-19T11:30:21.972 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.011e-05 2024-03-19T11:30:21.972 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.972 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:21.972 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.973 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.973 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.973 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:872 4.d e32/21 4:b4ef97e0:::smithi03829856-3:head v 32'20, mlcod=28'16)", 2024-03-19T11:30:21.973 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.036163+0000", 2024-03-19T11:30:21.973 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 12.916932666999999, 2024-03-19T11:30:21.973 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029333250000000001, 2024-03-19T11:30:21.973 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:21.973 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:21.973 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:21.973 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.973 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:21.973 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.036163+0000", 2024-03-19T11:30:21.973 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.973 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.974 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.974 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:21.974 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.036163+0000", 2024-03-19T11:30:21.974 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.974 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.974 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.974 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:21.974 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.036164+0000", 2024-03-19T11:30:21.974 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7000000000000003e-07 2024-03-19T11:30:21.974 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.974 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.974 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:21.974 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.036721+0000", 2024-03-19T11:30:21.974 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00055646700000000005 2024-03-19T11:30:21.975 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.975 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.975 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:21.975 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.036732+0000", 2024-03-19T11:30:21.975 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1649e-05 2024-03-19T11:30:21.975 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.975 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.975 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:21.975 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.036739+0000", 2024-03-19T11:30:21.975 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6760000000000001e-06 2024-03-19T11:30:21.975 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.975 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.975 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:21.975 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.036785+0000", 2024-03-19T11:30:21.976 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6177000000000002e-05 2024-03-19T11:30:21.976 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.976 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.976 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:21.976 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.036808+0000", 2024-03-19T11:30:21.976 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2654999999999999e-05 2024-03-19T11:30:21.976 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.976 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.976 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:21.976 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.039022+0000", 2024-03-19T11:30:21.976 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022145530000000002 2024-03-19T11:30:21.976 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.976 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.976 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:21.976 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.039097+0000", 2024-03-19T11:30:21.977 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4277999999999996e-05 2024-03-19T11:30:21.977 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.977 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:21.977 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.977 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.977 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.977 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(osd.6.13:39 3.0 e32/20 3:3dcd2456:::039914993b6f4082c204b24ae0ff925ebffccc22519067b64afa9eff107cf533:head v 32'54, mlcod=32'52)", 2024-03-19T11:30:21.977 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.308854+0000", 2024-03-19T11:30:21.977 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 12.644242378, 2024-03-19T11:30:21.977 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003468175, 2024-03-19T11:30:21.977 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:21.977 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:21.977 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:21.977 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.978 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:21.978 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308854+0000", 2024-03-19T11:30:21.978 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.978 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.978 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.978 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:21.978 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308854+0000", 2024-03-19T11:30:21.978 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.978 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.978 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.978 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:21.979 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308855+0000", 2024-03-19T11:30:21.979 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5999999999999999e-06 2024-03-19T11:30:21.979 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.979 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.979 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:21.979 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309019+0000", 2024-03-19T11:30:21.979 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000163311 2024-03-19T11:30:21.979 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.979 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.979 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:21.979 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309040+0000", 2024-03-19T11:30:21.979 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1795999999999999e-05 2024-03-19T11:30:21.979 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.979 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.979 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:21.980 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309060+0000", 2024-03-19T11:30:21.980 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9677000000000002e-05 2024-03-19T11:30:21.980 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.980 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.980 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:21.980 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309151+0000", 2024-03-19T11:30:21.980 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0507000000000006e-05 2024-03-19T11:30:21.980 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.980 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.980 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:21.980 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309187+0000", 2024-03-19T11:30:21.980 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6675999999999999e-05 2024-03-19T11:30:21.980 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.980 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.981 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:21.981 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.312256+0000", 2024-03-19T11:30:21.981 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030692060000000001 2024-03-19T11:30:21.981 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.981 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.981 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:21.981 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.312322+0000", 2024-03-19T11:30:21.981 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5401999999999995e-05 2024-03-19T11:30:21.981 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.981 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:21.981 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.982 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.982 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.982 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:901 4.d e32/21 4:ba44105a:::smithi03829856-26:head v 32'23, mlcod=32'20)", 2024-03-19T11:30:21.982 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.465629+0000", 2024-03-19T11:30:21.982 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 12.487467015, 2024-03-19T11:30:21.982 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002624361, 2024-03-19T11:30:21.982 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:21.982 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:21.982 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:21.982 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.983 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:21.983 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.465629+0000", 2024-03-19T11:30:21.983 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.983 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.983 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.984 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:21.984 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.465629+0000", 2024-03-19T11:30:21.984 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.984 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.984 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.984 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:21.985 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.465631+0000", 2024-03-19T11:30:21.985 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5740000000000001e-06 2024-03-19T11:30:21.985 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.985 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.985 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:21.985 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.465642+0000", 2024-03-19T11:30:21.986 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1343e-05 2024-03-19T11:30:21.986 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.986 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.986 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:21.986 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.465663+0000", 2024-03-19T11:30:21.986 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1535e-05 2024-03-19T11:30:21.987 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.987 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.987 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:21.987 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.465676+0000", 2024-03-19T11:30:21.987 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2281e-05 2024-03-19T11:30:21.987 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.988 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.988 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:21.988 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.465777+0000", 2024-03-19T11:30:21.988 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010083500000000001 2024-03-19T11:30:21.988 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.989 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.989 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:21.989 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.465806+0000", 2024-03-19T11:30:21.989 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9414999999999999e-05 2024-03-19T11:30:21.989 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.989 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.990 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:21.990 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.468174+0000", 2024-03-19T11:30:21.990 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023684299999999999 2024-03-19T11:30:21.990 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.990 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.990 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:21.991 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.468253+0000", 2024-03-19T11:30:21.991 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8948000000000004e-05 2024-03-19T11:30:21.991 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.991 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:21.991 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:21.991 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.992 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.992 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:998 4.4 e34/21 4:27909373:::smithi03829856-46:head v 34'37, mlcod=32'32)", 2024-03-19T11:30:21.992 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.370628+0000", 2024-03-19T11:30:21.992 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 9.5824681649999999, 2024-03-19T11:30:21.992 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050953099999999996, 2024-03-19T11:30:21.992 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:21.993 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:21.993 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:21.993 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.993 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:21.993 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370628+0000", 2024-03-19T11:30:21.993 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.994 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.994 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.994 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:21.994 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370628+0000", 2024-03-19T11:30:21.994 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:21.994 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.995 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.995 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:21.995 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370629+0000", 2024-03-19T11:30:21.995 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.5900000000000005e-07 2024-03-19T11:30:21.995 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.995 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.996 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:21.996 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370638+0000", 2024-03-19T11:30:21.996 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8189999999999998e-06 2024-03-19T11:30:21.996 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.996 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.996 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:21.997 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370650+0000", 2024-03-19T11:30:21.997 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.275e-05 2024-03-19T11:30:21.997 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.997 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.997 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:21.997 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370658+0000", 2024-03-19T11:30:21.998 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5769999999999996e-06 2024-03-19T11:30:21.998 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.998 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.998 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:21.998 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370747+0000", 2024-03-19T11:30:21.998 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8642999999999998e-05 2024-03-19T11:30:21.999 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:21.999 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:21.999 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:21.999 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370796+0000", 2024-03-19T11:30:21.999 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9428000000000002e-05 2024-03-19T11:30:21.999 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.000 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.000 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:22.000 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.375658+0000", 2024-03-19T11:30:22.000 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0048621519999999998 2024-03-19T11:30:22.000 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.000 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.001 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:22.001 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.375723+0000", 2024-03-19T11:30:22.001 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4981999999999995e-05 2024-03-19T11:30:22.001 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.001 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:22.001 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.002 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.002 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.002 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:999 4.4 e34/21 4:27909373:::smithi03829856-46:head v 34'38, mlcod=32'32)", 2024-03-19T11:30:22.002 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.372487+0000", 2024-03-19T11:30:22.002 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 9.5806094000000002, 2024-03-19T11:30:22.003 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0035853069999999998, 2024-03-19T11:30:22.003 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:22.003 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:22.003 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:22.003 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.003 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:22.003 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372487+0000", 2024-03-19T11:30:22.004 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.004 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.004 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.004 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:22.004 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372487+0000", 2024-03-19T11:30:22.004 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.005 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.005 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.005 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:22.005 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372488+0000", 2024-03-19T11:30:22.005 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1069999999999999e-06 2024-03-19T11:30:22.005 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.006 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.006 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:22.006 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372498+0000", 2024-03-19T11:30:22.006 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7510000000000003e-06 2024-03-19T11:30:22.006 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.006 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.007 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:22.007 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372544+0000", 2024-03-19T11:30:22.007 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6218999999999999e-05 2024-03-19T11:30:22.007 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.007 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.007 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:22.008 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372553+0000", 2024-03-19T11:30:22.008 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6320000000000003e-06 2024-03-19T11:30:22.008 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.008 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.008 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:22.008 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372636+0000", 2024-03-19T11:30:22.009 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3125000000000001e-05 2024-03-19T11:30:22.009 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.009 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.009 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:22.009 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372651+0000", 2024-03-19T11:30:22.009 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4676e-05 2024-03-19T11:30:22.009 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.010 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.010 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:22.010 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.376026+0000", 2024-03-19T11:30:22.010 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033745630000000001 2024-03-19T11:30:22.010 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.010 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.011 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:22.011 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.376072+0000", 2024-03-19T11:30:22.011 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6233999999999998e-05 2024-03-19T11:30:22.011 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.011 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:22.011 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.011 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.012 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.012 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:1003 4.4 e34/21 4:22709789:::smithi03829856-41:head v 34'40, mlcod=34'37)", 2024-03-19T11:30:22.012 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.378551+0000", 2024-03-19T11:30:22.012 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 9.5745453129999998, 2024-03-19T11:30:22.012 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0041156680000000003, 2024-03-19T11:30:22.012 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:22.012 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:22.013 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:22.013 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.013 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:22.013 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.378551+0000", 2024-03-19T11:30:22.013 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.013 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.013 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.014 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:22.014 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.378551+0000", 2024-03-19T11:30:22.014 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.014 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.014 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.014 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:22.015 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.378552+0000", 2024-03-19T11:30:22.015 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0400000000000005e-07 2024-03-19T11:30:22.015 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.015 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.015 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:22.015 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.378929+0000", 2024-03-19T11:30:22.016 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037751700000000001 2024-03-19T11:30:22.016 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.016 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.016 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:22.016 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.378944+0000", 2024-03-19T11:30:22.016 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5038000000000001e-05 2024-03-19T11:30:22.017 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.017 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.017 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:22.017 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.378953+0000", 2024-03-19T11:30:22.017 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7649999999999999e-06 2024-03-19T11:30:22.017 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.018 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.018 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:22.018 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.379017+0000", 2024-03-19T11:30:22.018 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3614000000000005e-05 2024-03-19T11:30:22.018 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.018 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.019 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:22.019 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.379042+0000", 2024-03-19T11:30:22.019 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5910000000000001e-05 2024-03-19T11:30:22.019 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.019 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.019 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:22.019 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.382610+0000", 2024-03-19T11:30:22.020 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0035678340000000002 2024-03-19T11:30:22.020 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.020 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.020 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:22.020 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.382666+0000", 2024-03-19T11:30:22.020 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6085999999999998e-05 2024-03-19T11:30:22.021 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.021 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:22.021 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.021 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.021 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.021 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:1004 4.4 e34/21 4:22709789:::smithi03829856-41:head v 34'41, mlcod=34'38)", 2024-03-19T11:30:22.022 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.386855+0000", 2024-03-19T11:30:22.022 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 9.5662415040000006, 2024-03-19T11:30:22.022 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0039133320000000003, 2024-03-19T11:30:22.022 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:22.022 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:22.022 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:22.023 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.023 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:22.023 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.386855+0000", 2024-03-19T11:30:22.023 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.023 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.023 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.024 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:22.024 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.386855+0000", 2024-03-19T11:30:22.024 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.024 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.024 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.024 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:22.025 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.386856+0000", 2024-03-19T11:30:22.025 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.528e-06 2024-03-19T11:30:22.025 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.025 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.025 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:22.025 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.387143+0000", 2024-03-19T11:30:22.025 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00028658600000000002 2024-03-19T11:30:22.026 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.026 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.026 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:22.026 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.387159+0000", 2024-03-19T11:30:22.026 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6796e-05 2024-03-19T11:30:22.026 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.027 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.027 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:22.027 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.387167+0000", 2024-03-19T11:30:22.027 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2729999999999997e-06 2024-03-19T11:30:22.027 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.027 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.028 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:22.028 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.387230+0000", 2024-03-19T11:30:22.028 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2862999999999998e-05 2024-03-19T11:30:22.028 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.028 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.028 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:22.029 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.387255+0000", 2024-03-19T11:30:22.029 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4905e-05 2024-03-19T11:30:22.029 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.029 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.029 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:22.029 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.390720+0000", 2024-03-19T11:30:22.030 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034652699999999999 2024-03-19T11:30:22.030 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.030 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.030 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:22.030 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.390768+0000", 2024-03-19T11:30:22.030 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8111000000000001e-05 2024-03-19T11:30:22.031 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.031 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:22.031 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.031 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.031 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.031 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:1042 3.0 e35/20 3:0e3fb112:::bdb3b5647d9daba88ca1ec369a6a4ecc66592b64bbd6df6ad063af4137f6b5b0:head v 35'62, mlcod=32'60)", 2024-03-19T11:30:22.032 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.211286+0000", 2024-03-19T11:30:22.032 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.7418096700000003, 2024-03-19T11:30:22.032 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0024813159999999999, 2024-03-19T11:30:22.032 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:22.032 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:22.032 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:22.033 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.033 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:22.033 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.211286+0000", 2024-03-19T11:30:22.033 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.033 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.033 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.034 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:22.034 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.211286+0000", 2024-03-19T11:30:22.034 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.034 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.034 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.034 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:22.035 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.211287+0000", 2024-03-19T11:30:22.035 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.67e-07 2024-03-19T11:30:22.035 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.035 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.035 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:22.035 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.211562+0000", 2024-03-19T11:30:22.036 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00027568999999999998 2024-03-19T11:30:22.036 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.036 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.036 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:22.036 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.211579+0000", 2024-03-19T11:30:22.036 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6489000000000001e-05 2024-03-19T11:30:22.037 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.037 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.037 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:22.037 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.211593+0000", 2024-03-19T11:30:22.037 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4351e-05 2024-03-19T11:30:22.037 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.038 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.038 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:22.038 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.211678+0000", 2024-03-19T11:30:22.038 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4924000000000002e-05 2024-03-19T11:30:22.038 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.038 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.039 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:22.039 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.211713+0000", 2024-03-19T11:30:22.039 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4492000000000001e-05 2024-03-19T11:30:22.039 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.039 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.039 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:22.040 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.213730+0000", 2024-03-19T11:30:22.040 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002017114 2024-03-19T11:30:22.040 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.040 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.040 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:22.040 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.213768+0000", 2024-03-19T11:30:22.040 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7988999999999998e-05 2024-03-19T11:30:22.041 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.041 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:22.041 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.041 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.041 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.041 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:1191 4.4 e40/21 4:27909373:::smithi03829856-46:head v 40'53, mlcod=38'50)", 2024-03-19T11:30:22.042 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:17.966218+0000", 2024-03-19T11:30:22.042 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.9868776719999999, 2024-03-19T11:30:22.042 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038990449999999999, 2024-03-19T11:30:22.042 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:22.042 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:22.042 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:22.043 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.043 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:22.043 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.966218+0000", 2024-03-19T11:30:22.043 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.043 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.043 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.044 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:22.044 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.966218+0000", 2024-03-19T11:30:22.044 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.044 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.044 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.044 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:22.045 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.966219+0000", 2024-03-19T11:30:22.045 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.96e-07 2024-03-19T11:30:22.045 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.045 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.045 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:22.045 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.966448+0000", 2024-03-19T11:30:22.046 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00022896799999999999 2024-03-19T11:30:22.046 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.046 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.046 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:22.046 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.966459+0000", 2024-03-19T11:30:22.046 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1654999999999999e-05 2024-03-19T11:30:22.046 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.046 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.046 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:22.046 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.966466+0000", 2024-03-19T11:30:22.046 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3770000000000001e-06 2024-03-19T11:30:22.046 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.046 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.047 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:22.047 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.966536+0000", 2024-03-19T11:30:22.047 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9808999999999995e-05 2024-03-19T11:30:22.047 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.047 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.047 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:22.047 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.966564+0000", 2024-03-19T11:30:22.047 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8319000000000001e-05 2024-03-19T11:30:22.047 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.047 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.047 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:22.047 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.970055+0000", 2024-03-19T11:30:22.047 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034915860000000001 2024-03-19T11:30:22.047 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.047 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.048 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:22.048 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.970117+0000", 2024-03-19T11:30:22.048 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1934999999999996e-05 2024-03-19T11:30:22.048 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.048 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:22.048 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.048 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.048 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.048 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:1192 4.4 e40/21 4:27909373:::smithi03829856-46:head v 40'54, mlcod=39'51)", 2024-03-19T11:30:22.048 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:17.977421+0000", 2024-03-19T11:30:22.048 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.9756748310000001, 2024-03-19T11:30:22.048 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0040889289999999998, 2024-03-19T11:30:22.048 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:22.048 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:22.048 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:22.049 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.049 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:22.049 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.977421+0000", 2024-03-19T11:30:22.049 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.049 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.049 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.049 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:22.049 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.977421+0000", 2024-03-19T11:30:22.049 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.049 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.049 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.049 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:22.049 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.977422+0000", 2024-03-19T11:30:22.049 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.271e-06 2024-03-19T11:30:22.049 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.050 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.050 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:22.050 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.977700+0000", 2024-03-19T11:30:22.050 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00027715299999999998 2024-03-19T11:30:22.050 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.050 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.050 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:22.050 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.977718+0000", 2024-03-19T11:30:22.050 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8067000000000002e-05 2024-03-19T11:30:22.050 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.050 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.050 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:22.050 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.977727+0000", 2024-03-19T11:30:22.050 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4919999999999999e-06 2024-03-19T11:30:22.051 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.051 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.051 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:22.051 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.977792+0000", 2024-03-19T11:30:22.051 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4945999999999998e-05 2024-03-19T11:30:22.051 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.051 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.051 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:22.051 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.977818+0000", 2024-03-19T11:30:22.051 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5519000000000001e-05 2024-03-19T11:30:22.051 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.051 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.051 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:22.051 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.981471+0000", 2024-03-19T11:30:22.051 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0036535980000000001 2024-03-19T11:30:22.052 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.052 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.052 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:22.052 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.981510+0000", 2024-03-19T11:30:22.052 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8883e-05 2024-03-19T11:30:22.052 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.052 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:22.052 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.052 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.052 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.052 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:1193 4.4 e40/21 4:27909373:::smithi03829856-46:head v 40'55, mlcod=40'53)", 2024-03-19T11:30:22.052 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:17.985632+0000", 2024-03-19T11:30:22.052 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.9674637380000002, 2024-03-19T11:30:22.052 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025355970000000001, 2024-03-19T11:30:22.053 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:22.053 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:22.053 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:22.053 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.053 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:22.053 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.985632+0000", 2024-03-19T11:30:22.053 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.053 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.053 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.053 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:22.053 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.985632+0000", 2024-03-19T11:30:22.053 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.053 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.053 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.053 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:22.054 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.985633+0000", 2024-03-19T11:30:22.054 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.158e-06 2024-03-19T11:30:22.054 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.054 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.054 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:22.054 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.985871+0000", 2024-03-19T11:30:22.054 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023778699999999999 2024-03-19T11:30:22.054 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.054 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.054 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:22.054 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.985891+0000", 2024-03-19T11:30:22.054 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9389999999999999e-05 2024-03-19T11:30:22.054 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.054 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.054 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:22.055 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.985900+0000", 2024-03-19T11:30:22.055 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4870000000000008e-06 2024-03-19T11:30:22.055 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.055 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.055 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:22.055 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.985985+0000", 2024-03-19T11:30:22.055 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4593999999999997e-05 2024-03-19T11:30:22.055 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.055 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.055 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:22.055 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.986010+0000", 2024-03-19T11:30:22.055 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5429999999999999e-05 2024-03-19T11:30:22.055 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.055 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.056 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:22.056 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.988112+0000", 2024-03-19T11:30:22.056 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0021015529999999999 2024-03-19T11:30:22.056 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.056 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.056 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:22.056 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.988168+0000", 2024-03-19T11:30:22.056 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6198e-05 2024-03-19T11:30:22.056 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.056 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:22.056 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.056 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.056 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.056 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:1215 3.0 e41/20 3:0c9406a2:::98f4ba27fa68608c540dca9b03a4531ee556ff2734aa426809de93c45c84ab96:head v 41'65, mlcod=36'63)", 2024-03-19T11:30:22.056 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.151676+0000", 2024-03-19T11:30:22.057 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.8014201910000001, 2024-03-19T11:30:22.057 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0037067860000000001, 2024-03-19T11:30:22.057 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:22.057 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:22.057 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:22.057 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.057 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:22.057 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.151676+0000", 2024-03-19T11:30:22.057 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.057 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.057 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.057 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:22.057 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.151676+0000", 2024-03-19T11:30:22.057 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.057 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.058 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.058 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:22.058 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.151676+0000", 2024-03-19T11:30:22.058 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2500000000000001e-07 2024-03-19T11:30:22.058 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.058 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.058 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:22.058 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.152439+0000", 2024-03-19T11:30:22.058 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00076323500000000002 2024-03-19T11:30:22.058 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.058 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.058 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:22.058 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.152452+0000", 2024-03-19T11:30:22.058 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2356e-05 2024-03-19T11:30:22.058 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.059 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.059 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:22.059 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.152460+0000", 2024-03-19T11:30:22.059 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2609999999999993e-06 2024-03-19T11:30:22.059 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.059 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.059 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:22.059 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.152543+0000", 2024-03-19T11:30:22.059 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2837000000000007e-05 2024-03-19T11:30:22.059 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.059 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.059 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:22.059 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.152571+0000", 2024-03-19T11:30:22.059 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8228000000000001e-05 2024-03-19T11:30:22.059 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.060 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.060 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:22.060 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155345+0000", 2024-03-19T11:30:22.060 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0027742019999999999 2024-03-19T11:30:22.060 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.060 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.060 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:22.060 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155383+0000", 2024-03-19T11:30:22.060 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7342000000000003e-05 2024-03-19T11:30:22.060 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.060 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:22.060 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.060 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.060 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.060 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(osd.0.13:55 3.0 e41/20 3:0d712e74:::bb863e2508028016f9ab1e6296121a5f2f252d02d0ddc1e79616682e5e6f5b3c:head v 41'66, mlcod=36'64)", 2024-03-19T11:30:22.061 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.175590+0000", 2024-03-19T11:30:22.061 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.7775059710000001, 2024-03-19T11:30:22.061 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002551345, 2024-03-19T11:30:22.061 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:22.061 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:22.061 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:22.061 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.061 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:22.061 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.175590+0000", 2024-03-19T11:30:22.061 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.061 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.061 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.061 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:22.061 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.175590+0000", 2024-03-19T11:30:22.061 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.061 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.062 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.062 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:22.062 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.175592+0000", 2024-03-19T11:30:22.062 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.595e-06 2024-03-19T11:30:22.062 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.062 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.062 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:22.062 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.175772+0000", 2024-03-19T11:30:22.062 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000180619 2024-03-19T11:30:22.062 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.062 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.062 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:22.062 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.175805+0000", 2024-03-19T11:30:22.062 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2271000000000003e-05 2024-03-19T11:30:22.062 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.063 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.063 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:22.063 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.175818+0000", 2024-03-19T11:30:22.063 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3764000000000001e-05 2024-03-19T11:30:22.063 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.063 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.063 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:22.063 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.175909+0000", 2024-03-19T11:30:22.063 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1039999999999996e-05 2024-03-19T11:30:22.063 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.063 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.063 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:22.063 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.175940+0000", 2024-03-19T11:30:22.063 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0501000000000002e-05 2024-03-19T11:30:22.063 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.064 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.064 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:22.064 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.178091+0000", 2024-03-19T11:30:22.064 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0021512940000000002 2024-03-19T11:30:22.064 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.064 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.064 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:22.064 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.178141+0000", 2024-03-19T11:30:22.064 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0260999999999999e-05 2024-03-19T11:30:22.064 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.064 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:22.064 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.064 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.064 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.064 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(osd.1.0:142 4.4 e44/43 4:214026b3:::smithi03829856-47:head v 44'57, mlcod=0'0)", 2024-03-19T11:30:22.065 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:21.867611+0000", 2024-03-19T11:30:22.065 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.085485451000000004, 2024-03-19T11:30:22.065 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0081946840000000007, 2024-03-19T11:30:22.065 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:22.065 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:22.065 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:22.065 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.065 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:22.065 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.867611+0000", 2024-03-19T11:30:22.065 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.065 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.065 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.065 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:22.065 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.867611+0000", 2024-03-19T11:30:22.065 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.066 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.066 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.066 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:22.066 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.867613+0000", 2024-03-19T11:30:22.066 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9250000000000002e-06 2024-03-19T11:30:22.066 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.066 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.066 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:22.066 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.869767+0000", 2024-03-19T11:30:22.066 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002154071 2024-03-19T11:30:22.066 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.066 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.066 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:22.066 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.869792+0000", 2024-03-19T11:30:22.066 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5327e-05 2024-03-19T11:30:22.067 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.067 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.067 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:22.067 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.869803+0000", 2024-03-19T11:30:22.067 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0883999999999999e-05 2024-03-19T11:30:22.067 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.067 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.067 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:22.067 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.869890+0000", 2024-03-19T11:30:22.067 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7049e-05 2024-03-19T11:30:22.067 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.067 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.067 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:22.067 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.869927+0000", 2024-03-19T11:30:22.067 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7465999999999997e-05 2024-03-19T11:30:22.068 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.068 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.068 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:22.068 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.875750+0000", 2024-03-19T11:30:22.068 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0058225969999999997 2024-03-19T11:30:22.068 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.068 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.068 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:22.068 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.875805+0000", 2024-03-19T11:30:22.068 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5365000000000003e-05 2024-03-19T11:30:22.068 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.068 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:22.068 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.068 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.068 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:22.069 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:22.069 INFO:tasks.rados.rados.0.smithi038.stdout:785: writing smithi03829856-19 from 2158943 to 2805963 tid 2 2024-03-19T11:30:22.070 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-19T11:30:22.071 INFO:tasks.rados.rados.0.smithi038.stdout:785: writing smithi03829856-19 from 3437750 to 3563051 tid 3 2024-03-19T11:30:22.071 INFO:tasks.rados.rados.0.smithi038.stdout:778: done (5 left) 2024-03-19T11:30:22.071 INFO:tasks.rados.rados.0.smithi038.stdout:779: done (4 left) 2024-03-19T11:30:22.071 INFO:tasks.rados.rados.0.smithi038.stdout:781: done (3 left) 2024-03-19T11:30:22.071 INFO:tasks.rados.rados.0.smithi038.stdout:786: read oid 27 snap -1 2024-03-19T11:30:22.071 INFO:tasks.rados.rados.0.smithi038.stdout:786: expect deleted 2024-03-19T11:30:22.071 INFO:tasks.rados.rados.0.smithi038.stdout:787: snap_remove snap 7 2024-03-19T11:30:22.072 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:22.082+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:22.171 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:22.184+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:22.202 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:22.202 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:30:22.202 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:30:22.202 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:22.218 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-19T11:30:22.272 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:22.282+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:22.373 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:22.384+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:22.388 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:22.388 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:30:22.388 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:30:22.388 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:30:22.388 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:22.402 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-19T11:30:22.491 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:22.504+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:22.575 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:22.576 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:30:22.576 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:30:22.576 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:30:22.576 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.576 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:38 3.2 3:5595a68d:::07854d2fef297a06ba81685e660c332de36d5d18d546927d30daad6d7fda1541:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:22.576 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.307388+0000", 2024-03-19T11:30:22.576 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 13.278061075, 2024-03-19T11:30:22.576 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0080668630000000005, 2024-03-19T11:30:22.576 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:22.576 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:22.576 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:22.576 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.576 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:22.577 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.307388+0000", 2024-03-19T11:30:22.577 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.577 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.577 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.577 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:22.577 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.307388+0000", 2024-03-19T11:30:22.577 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.577 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.577 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.577 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:22.577 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.307391+0000", 2024-03-19T11:30:22.577 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3039999999999999e-06 2024-03-19T11:30:22.577 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.577 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.577 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:22.578 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308352+0000", 2024-03-19T11:30:22.578 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00096137600000000001 2024-03-19T11:30:22.578 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.578 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.578 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:22.578 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308376+0000", 2024-03-19T11:30:22.578 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4196999999999999e-05 2024-03-19T11:30:22.578 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.578 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.578 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:22.578 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308387+0000", 2024-03-19T11:30:22.578 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1068e-05 2024-03-19T11:30:22.578 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.578 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.579 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:22.579 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308539+0000", 2024-03-19T11:30:22.579 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015186600000000001 2024-03-19T11:30:22.579 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.580 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.580 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:22.580 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308731+0000", 2024-03-19T11:30:22.580 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019146199999999999 2024-03-19T11:30:22.580 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.580 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.581 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:22.581 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309028+0000", 2024-03-19T11:30:22.581 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00029751799999999998 2024-03-19T11:30:22.581 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.581 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.581 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:22.581 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.314461+0000", 2024-03-19T11:30:22.581 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0054329570000000004 2024-03-19T11:30:22.581 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.581 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.582 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:22.582 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.315339+0000", 2024-03-19T11:30:22.582 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00087765200000000003 2024-03-19T11:30:22.582 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.582 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.582 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:22.582 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.315400+0000", 2024-03-19T11:30:22.582 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1650999999999996e-05 2024-03-19T11:30:22.582 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.582 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.582 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:22.582 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.315455+0000", 2024-03-19T11:30:22.582 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4812000000000003e-05 2024-03-19T11:30:22.582 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.583 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:22.583 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.583 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.583 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.583 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:41 3.2 3:525f009b:::a57e3dbafedf1ec1a7132467f167cfc4a4343cb541b36fc481376f0086aece71:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:22.583 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.308522+0000", 2024-03-19T11:30:22.583 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 13.276927175999999, 2024-03-19T11:30:22.583 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0079565969999999993, 2024-03-19T11:30:22.583 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:22.583 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:22.583 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:22.583 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.583 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:22.583 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308522+0000", 2024-03-19T11:30:22.583 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.584 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.584 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.584 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:22.584 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308522+0000", 2024-03-19T11:30:22.584 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.584 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.584 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.584 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:22.584 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308524+0000", 2024-03-19T11:30:22.584 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.809e-06 2024-03-19T11:30:22.584 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.584 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.584 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:22.584 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308627+0000", 2024-03-19T11:30:22.584 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000103319 2024-03-19T11:30:22.585 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.585 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.585 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:22.585 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308643+0000", 2024-03-19T11:30:22.585 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5279000000000001e-05 2024-03-19T11:30:22.585 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.585 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.585 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:22.585 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308653+0000", 2024-03-19T11:30:22.585 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0557000000000001e-05 2024-03-19T11:30:22.585 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.585 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.585 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:22.585 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310052+0000", 2024-03-19T11:30:22.585 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0013986770000000001 2024-03-19T11:30:22.586 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.586 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.586 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:22.586 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310231+0000", 2024-03-19T11:30:22.586 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017891000000000001 2024-03-19T11:30:22.586 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.586 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.586 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:22.586 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310515+0000", 2024-03-19T11:30:22.586 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00028459299999999998 2024-03-19T11:30:22.586 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.586 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.586 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:22.587 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.315472+0000", 2024-03-19T11:30:22.587 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.004956874 2024-03-19T11:30:22.587 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.587 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.587 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:22.587 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.316378+0000", 2024-03-19T11:30:22.587 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00090556600000000001 2024-03-19T11:30:22.587 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.587 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.587 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:22.587 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.316435+0000", 2024-03-19T11:30:22.587 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7126000000000002e-05 2024-03-19T11:30:22.587 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.587 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.587 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:22.588 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.316479+0000", 2024-03-19T11:30:22.588 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3887000000000002e-05 2024-03-19T11:30:22.588 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.588 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:22.588 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.588 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.588 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.588 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:42 3.2 3:5aa75d1e:::b3c26d797de4ffcce4e862009e6b8eb8576a53cb86ea102ce51145c0038aeb97:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:22.588 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.308689+0000", 2024-03-19T11:30:22.588 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 13.27676042, 2024-03-19T11:30:22.588 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0079809319999999996, 2024-03-19T11:30:22.588 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:22.588 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:22.588 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:22.588 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.589 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:22.589 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308689+0000", 2024-03-19T11:30:22.589 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.589 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.589 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.589 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:22.589 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308689+0000", 2024-03-19T11:30:22.589 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.589 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.589 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.589 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:22.589 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308690+0000", 2024-03-19T11:30:22.589 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.003e-06 2024-03-19T11:30:22.590 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.590 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.590 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:22.590 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309612+0000", 2024-03-19T11:30:22.590 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00092193199999999998 2024-03-19T11:30:22.590 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.590 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.590 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:22.590 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309645+0000", 2024-03-19T11:30:22.590 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2554e-05 2024-03-19T11:30:22.590 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.590 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.590 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:22.591 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309654+0000", 2024-03-19T11:30:22.591 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6209999999999999e-06 2024-03-19T11:30:22.591 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.591 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.591 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:22.591 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.311778+0000", 2024-03-19T11:30:22.591 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002123897 2024-03-19T11:30:22.591 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.591 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.591 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:22.591 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.311913+0000", 2024-03-19T11:30:22.591 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013471199999999999 2024-03-19T11:30:22.591 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.592 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.592 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:22.592 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.312123+0000", 2024-03-19T11:30:22.592 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00021064099999999999 2024-03-19T11:30:22.592 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.592 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.592 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:22.592 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.315479+0000", 2024-03-19T11:30:22.592 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033557230000000001 2024-03-19T11:30:22.592 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.592 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.592 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:22.592 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.316574+0000", 2024-03-19T11:30:22.593 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0010944990000000001 2024-03-19T11:30:22.593 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.593 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.593 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:22.593 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.316629+0000", 2024-03-19T11:30:22.593 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5169e-05 2024-03-19T11:30:22.593 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.593 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.593 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:22.593 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.316670+0000", 2024-03-19T11:30:22.593 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1180999999999998e-05 2024-03-19T11:30:22.594 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.594 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:22.594 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.594 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.594 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.594 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:43 3.2 3:6d82e5d5:::455c613336f373ae48b33af990f91f317265a0c5a920cc1b46533fdb41d8319c:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:22.594 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.309666+0000", 2024-03-19T11:30:22.594 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 13.275783125, 2024-03-19T11:30:22.594 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077817820000000001, 2024-03-19T11:30:22.594 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:22.594 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:22.595 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:22.595 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.595 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:22.595 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309666+0000", 2024-03-19T11:30:22.595 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.595 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.595 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.595 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:22.595 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309666+0000", 2024-03-19T11:30:22.595 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.596 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.596 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.596 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:22.596 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309667+0000", 2024-03-19T11:30:22.596 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.5099999999999998e-07 2024-03-19T11:30:22.596 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.596 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.596 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:22.596 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310025+0000", 2024-03-19T11:30:22.596 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00035726900000000001 2024-03-19T11:30:22.596 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.596 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.596 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:22.596 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310093+0000", 2024-03-19T11:30:22.596 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8604999999999998e-05 2024-03-19T11:30:22.597 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.597 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.597 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:22.597 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310157+0000", 2024-03-19T11:30:22.597 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3688000000000004e-05 2024-03-19T11:30:22.597 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.597 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.597 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:22.597 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.313280+0000", 2024-03-19T11:30:22.597 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031230770000000001 2024-03-19T11:30:22.597 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.597 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.597 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:22.597 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.313429+0000", 2024-03-19T11:30:22.598 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000148763 2024-03-19T11:30:22.598 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.598 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.598 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:22.598 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.313700+0000", 2024-03-19T11:30:22.598 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00027178899999999998 2024-03-19T11:30:22.598 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.598 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.598 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:22.598 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.315489+0000", 2024-03-19T11:30:22.599 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001788848 2024-03-19T11:30:22.599 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.599 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.599 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:22.599 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.317381+0000", 2024-03-19T11:30:22.599 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0018913249999999999 2024-03-19T11:30:22.599 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.599 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.599 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:22.599 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.317411+0000", 2024-03-19T11:30:22.599 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0001e-05 2024-03-19T11:30:22.599 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.599 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.599 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:22.599 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.317448+0000", 2024-03-19T11:30:22.600 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7465999999999997e-05 2024-03-19T11:30:22.600 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.600 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:22.600 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.600 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.600 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.600 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:44 3.2 3:5f1d333e:::bfcdf551f6f2543327e9c601a4377e9a5ef396f5b97ae8331892d1bcf6b55a9d:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:22.600 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.310172+0000", 2024-03-19T11:30:22.600 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 13.275276952, 2024-03-19T11:30:22.600 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0086762909999999992, 2024-03-19T11:30:22.600 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:22.600 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:22.600 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:22.600 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.601 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:22.601 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310172+0000", 2024-03-19T11:30:22.601 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.601 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.601 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.601 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:22.601 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310172+0000", 2024-03-19T11:30:22.601 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.601 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.601 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.601 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:22.601 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310173+0000", 2024-03-19T11:30:22.601 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0500000000000002e-07 2024-03-19T11:30:22.601 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.601 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.602 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:22.602 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310242+0000", 2024-03-19T11:30:22.602 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8239999999999997e-05 2024-03-19T11:30:22.602 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.602 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.602 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:22.602 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310253+0000", 2024-03-19T11:30:22.602 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0879e-05 2024-03-19T11:30:22.602 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.602 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.602 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:22.602 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310263+0000", 2024-03-19T11:30:22.602 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0045e-05 2024-03-19T11:30:22.602 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.603 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.603 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:22.603 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.314507+0000", 2024-03-19T11:30:22.603 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0042442060000000004 2024-03-19T11:30:22.603 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.603 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.603 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:22.603 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.314588+0000", 2024-03-19T11:30:22.603 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0912000000000001e-05 2024-03-19T11:30:22.603 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.603 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.603 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:22.603 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.314736+0000", 2024-03-19T11:30:22.603 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014782099999999999 2024-03-19T11:30:22.603 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.603 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.604 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:22.604 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.316529+0000", 2024-03-19T11:30:22.604 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00179327 2024-03-19T11:30:22.604 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.604 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.604 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:22.604 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.318785+0000", 2024-03-19T11:30:22.604 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022563660000000001 2024-03-19T11:30:22.604 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.604 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.604 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:22.604 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.318813+0000", 2024-03-19T11:30:22.604 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8206000000000001e-05 2024-03-19T11:30:22.604 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.605 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.605 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:22.605 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.318849+0000", 2024-03-19T11:30:22.605 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5441000000000002e-05 2024-03-19T11:30:22.605 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.605 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:22.605 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.605 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.605 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.605 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:45 3.2 3:78b4b14e:::e1ef0c0c594c36c534f3829406e07ffb60d57b70360f62977e0d545d8ca2bf18:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:22.605 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.310274+0000", 2024-03-19T11:30:22.605 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 13.275175496999999, 2024-03-19T11:30:22.605 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094234769999999995, 2024-03-19T11:30:22.605 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:22.605 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:22.606 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:22.606 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.606 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:22.606 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310274+0000", 2024-03-19T11:30:22.606 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.606 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.606 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.606 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:22.606 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310274+0000", 2024-03-19T11:30:22.606 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.607 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.607 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.607 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:22.607 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310275+0000", 2024-03-19T11:30:22.607 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2099999999999996e-07 2024-03-19T11:30:22.607 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.607 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.607 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:22.607 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310329+0000", 2024-03-19T11:30:22.607 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4440000000000001e-05 2024-03-19T11:30:22.607 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.607 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.607 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:22.607 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310340+0000", 2024-03-19T11:30:22.607 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1321999999999999e-05 2024-03-19T11:30:22.608 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.608 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.608 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:22.608 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310349+0000", 2024-03-19T11:30:22.608 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6789999999999994e-06 2024-03-19T11:30:22.608 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.608 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.608 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:22.608 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.315512+0000", 2024-03-19T11:30:22.608 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0051629689999999999 2024-03-19T11:30:22.608 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.608 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.608 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:22.608 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.315581+0000", 2024-03-19T11:30:22.608 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9400999999999995e-05 2024-03-19T11:30:22.608 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.609 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.609 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:22.609 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.315759+0000", 2024-03-19T11:30:22.609 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000177491 2024-03-19T11:30:22.609 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.609 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.609 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:22.609 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.317063+0000", 2024-03-19T11:30:22.609 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0013040630000000001 2024-03-19T11:30:22.609 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.609 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.609 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:22.609 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.319635+0000", 2024-03-19T11:30:22.609 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025715880000000001 2024-03-19T11:30:22.609 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.610 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.610 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:22.610 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.319663+0000", 2024-03-19T11:30:22.610 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8554000000000001e-05 2024-03-19T11:30:22.610 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.610 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.610 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:22.610 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.319697+0000", 2024-03-19T11:30:22.610 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4348999999999999e-05 2024-03-19T11:30:22.610 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.610 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:22.610 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.610 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.611 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.611 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:55 3.2 3:525f009b:::a57e3dbafedf1ec1a7132467f167cfc4a4343cb541b36fc481376f0086aece71:head [read 0~121882 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:22.611 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.761904+0000", 2024-03-19T11:30:22.611 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 9.8235450009999994, 2024-03-19T11:30:22.611 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010829357, 2024-03-19T11:30:22.611 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:22.611 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:22.611 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:22.611 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.611 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:22.611 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761904+0000", 2024-03-19T11:30:22.611 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.611 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.611 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.612 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:22.612 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761904+0000", 2024-03-19T11:30:22.612 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.612 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.612 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.612 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:22.612 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761905+0000", 2024-03-19T11:30:22.612 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4900000000000005e-07 2024-03-19T11:30:22.612 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.612 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.612 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:22.612 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761917+0000", 2024-03-19T11:30:22.612 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1806e-05 2024-03-19T11:30:22.613 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.613 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.613 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:22.613 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761934+0000", 2024-03-19T11:30:22.613 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6509000000000001e-05 2024-03-19T11:30:22.613 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.613 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.613 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:22.613 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761941+0000", 2024-03-19T11:30:22.613 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8110000000000003e-06 2024-03-19T11:30:22.613 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.613 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.613 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:22.614 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762115+0000", 2024-03-19T11:30:22.614 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017358200000000001 2024-03-19T11:30:22.614 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.614 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.614 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:22.614 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762120+0000", 2024-03-19T11:30:22.614 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3630000000000002e-06 2024-03-19T11:30:22.614 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.614 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.614 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:22.615 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.765567+0000", 2024-03-19T11:30:22.615 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034469990000000001 2024-03-19T11:30:22.615 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.615 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.615 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:22.615 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.765665+0000", 2024-03-19T11:30:22.615 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7476e-05 2024-03-19T11:30:22.615 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.615 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.615 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:22.615 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.772734+0000", 2024-03-19T11:30:22.615 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0070690620000000001 2024-03-19T11:30:22.615 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.615 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:22.616 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.616 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.616 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.616 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:56 3.2 3:5aa75d1e:::b3c26d797de4ffcce4e862009e6b8eb8576a53cb86ea102ce51145c0038aeb97:head [read 0~524288 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:22.616 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.761958+0000", 2024-03-19T11:30:22.616 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 9.8234916000000005, 2024-03-19T11:30:22.616 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012078836000000001, 2024-03-19T11:30:22.616 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:22.616 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:22.616 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:22.616 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.616 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:22.616 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761958+0000", 2024-03-19T11:30:22.616 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.616 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.617 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.617 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:22.617 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761958+0000", 2024-03-19T11:30:22.617 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.617 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.617 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.617 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:22.617 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761958+0000", 2024-03-19T11:30:22.618 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9900000000000001e-07 2024-03-19T11:30:22.618 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.618 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.618 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:22.618 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761961+0000", 2024-03-19T11:30:22.618 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9280000000000002e-06 2024-03-19T11:30:22.618 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.618 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.619 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:22.619 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761969+0000", 2024-03-19T11:30:22.619 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1440000000000003e-06 2024-03-19T11:30:22.619 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.619 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.619 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:22.619 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761976+0000", 2024-03-19T11:30:22.619 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1020000000000002e-06 2024-03-19T11:30:22.619 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.619 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.619 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:22.619 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762144+0000", 2024-03-19T11:30:22.619 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016818200000000001 2024-03-19T11:30:22.620 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.620 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.620 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:22.620 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762148+0000", 2024-03-19T11:30:22.620 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0969999999999999e-06 2024-03-19T11:30:22.620 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.620 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.620 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:22.620 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.772788+0000", 2024-03-19T11:30:22.620 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010640124000000001 2024-03-19T11:30:22.620 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.620 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.620 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:22.621 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.772898+0000", 2024-03-19T11:30:22.621 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000110512 2024-03-19T11:30:22.621 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.621 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.621 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:22.621 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.774037+0000", 2024-03-19T11:30:22.621 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011382479999999999 2024-03-19T11:30:22.621 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.621 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:22.621 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.621 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.621 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.621 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:57 3.2 3:6d82e5d5:::455c613336f373ae48b33af990f91f317265a0c5a920cc1b46533fdb41d8319c:head [read 0~380357 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:22.621 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.761987+0000", 2024-03-19T11:30:22.621 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 9.8234626380000005, 2024-03-19T11:30:22.622 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012691991999999999, 2024-03-19T11:30:22.622 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:22.622 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:22.622 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:22.622 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.622 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:22.622 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761987+0000", 2024-03-19T11:30:22.622 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.622 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.622 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.622 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:22.622 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761987+0000", 2024-03-19T11:30:22.622 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.622 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.623 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.623 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:22.623 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761987+0000", 2024-03-19T11:30:22.623 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4099999999999999e-07 2024-03-19T11:30:22.623 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.623 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.623 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:22.623 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761990+0000", 2024-03-19T11:30:22.623 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3829999999999999e-06 2024-03-19T11:30:22.623 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.623 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.623 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:22.623 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762000+0000", 2024-03-19T11:30:22.623 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0229e-05 2024-03-19T11:30:22.623 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.624 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.624 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:22.624 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762006+0000", 2024-03-19T11:30:22.624 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7069999999999997e-06 2024-03-19T11:30:22.624 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.624 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.624 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:22.624 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762171+0000", 2024-03-19T11:30:22.624 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000165341 2024-03-19T11:30:22.624 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.624 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.624 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:22.624 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762176+0000", 2024-03-19T11:30:22.624 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6879999999999998e-06 2024-03-19T11:30:22.624 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.625 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.625 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:22.625 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.774081+0000", 2024-03-19T11:30:22.625 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011905094999999999 2024-03-19T11:30:22.625 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.625 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.625 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:22.625 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.774169+0000", 2024-03-19T11:30:22.625 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8089000000000003e-05 2024-03-19T11:30:22.625 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.625 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.625 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:22.625 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.774679+0000", 2024-03-19T11:30:22.625 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00051001900000000001 2024-03-19T11:30:22.625 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.626 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:22.626 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.626 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.626 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.626 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:58 3.2 3:5f1d333e:::bfcdf551f6f2543327e9c601a4377e9a5ef396f5b97ae8331892d1bcf6b55a9d:head [read 0~161292 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:22.626 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.762024+0000", 2024-03-19T11:30:22.626 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 9.8234256890000005, 2024-03-19T11:30:22.626 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013495629, 2024-03-19T11:30:22.626 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:22.626 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:22.626 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:22.626 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.626 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:22.626 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762024+0000", 2024-03-19T11:30:22.626 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.627 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.627 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.627 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:22.627 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762024+0000", 2024-03-19T11:30:22.627 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.627 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.627 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.627 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:22.627 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762024+0000", 2024-03-19T11:30:22.627 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9999999999999997e-07 2024-03-19T11:30:22.627 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.627 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.627 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:22.627 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762027+0000", 2024-03-19T11:30:22.627 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0410000000000002e-06 2024-03-19T11:30:22.628 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.628 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.628 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:22.628 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762037+0000", 2024-03-19T11:30:22.628 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1020000000000003e-06 2024-03-19T11:30:22.628 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.628 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.628 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:22.628 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762043+0000", 2024-03-19T11:30:22.628 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4409999999999998e-06 2024-03-19T11:30:22.628 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.628 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.628 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:22.628 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762198+0000", 2024-03-19T11:30:22.628 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015540000000000001 2024-03-19T11:30:22.629 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.629 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.629 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:22.629 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762203+0000", 2024-03-19T11:30:22.629 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5109999999999997e-06 2024-03-19T11:30:22.629 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.629 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.629 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:22.629 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.774710+0000", 2024-03-19T11:30:22.629 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012507452 2024-03-19T11:30:22.629 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.629 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.629 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:22.630 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.774754+0000", 2024-03-19T11:30:22.630 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3869999999999998e-05 2024-03-19T11:30:22.630 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.630 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.630 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:22.630 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.775519+0000", 2024-03-19T11:30:22.630 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00076521200000000003 2024-03-19T11:30:22.630 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.630 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:22.630 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.630 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.630 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.630 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:59 3.2 3:78b4b14e:::e1ef0c0c594c36c534f3829406e07ffb60d57b70360f62977e0d545d8ca2bf18:head [read 0~149977 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:22.631 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.762054+0000", 2024-03-19T11:30:22.631 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 9.8233952579999997, 2024-03-19T11:30:22.631 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015354483, 2024-03-19T11:30:22.631 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:22.631 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:22.631 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:22.631 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.631 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:22.631 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762054+0000", 2024-03-19T11:30:22.631 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.631 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.631 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.631 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:22.632 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762054+0000", 2024-03-19T11:30:22.632 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.632 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.632 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.632 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:22.632 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762055+0000", 2024-03-19T11:30:22.632 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.32e-07 2024-03-19T11:30:22.632 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.632 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.632 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:22.633 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762058+0000", 2024-03-19T11:30:22.633 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9189999999999999e-06 2024-03-19T11:30:22.633 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.633 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.633 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:22.633 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762065+0000", 2024-03-19T11:30:22.633 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2409999999999998e-06 2024-03-19T11:30:22.633 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.633 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.633 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:22.633 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762070+0000", 2024-03-19T11:30:22.633 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6910000000000002e-06 2024-03-19T11:30:22.633 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.634 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.634 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:22.634 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762227+0000", 2024-03-19T11:30:22.634 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015606999999999999 2024-03-19T11:30:22.634 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.634 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.634 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:22.634 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762232+0000", 2024-03-19T11:30:22.634 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3029999999999999e-06 2024-03-19T11:30:22.634 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.634 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.635 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:22.635 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.776984+0000", 2024-03-19T11:30:22.635 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014752487999999999 2024-03-19T11:30:22.635 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.635 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.635 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:22.635 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.777030+0000", 2024-03-19T11:30:22.635 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5359999999999999e-05 2024-03-19T11:30:22.635 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.635 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.635 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:22.635 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.777409+0000", 2024-03-19T11:30:22.636 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037897899999999999 2024-03-19T11:30:22.636 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.636 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:22.636 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.636 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.636 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.636 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1067 4.0 4:01018511:::smithi03829856-19:head [write 477332~766101] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:22.636 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.950506+0000", 2024-03-19T11:30:22.636 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.6349435460000006, 2024-03-19T11:30:22.636 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016279426, 2024-03-19T11:30:22.636 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:22.636 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:22.637 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:22.637 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:22.637 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:22.637 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1067 2024-03-19T11:30:22.637 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.637 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:22.637 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.637 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:22.637 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950506+0000", 2024-03-19T11:30:22.637 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.637 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.637 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.637 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:22.638 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950506+0000", 2024-03-19T11:30:22.638 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.638 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.638 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.638 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:22.638 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950509+0000", 2024-03-19T11:30:22.638 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.021e-06 2024-03-19T11:30:22.638 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.638 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.638 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:22.638 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950892+0000", 2024-03-19T11:30:22.639 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00038263300000000001 2024-03-19T11:30:22.639 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.639 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.639 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:22.639 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950912+0000", 2024-03-19T11:30:22.639 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0539000000000001e-05 2024-03-19T11:30:22.639 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.639 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.639 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:22.639 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950922+0000", 2024-03-19T11:30:22.639 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.577e-06 2024-03-19T11:30:22.639 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.639 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.639 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:22.640 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950982+0000", 2024-03-19T11:30:22.640 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0760000000000001e-05 2024-03-19T11:30:22.640 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.640 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.640 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:22.640 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.951058+0000", 2024-03-19T11:30:22.640 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5078000000000002e-05 2024-03-19T11:30:22.640 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.640 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.640 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:22.640 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.951173+0000", 2024-03-19T11:30:22.640 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011516200000000001 2024-03-19T11:30:22.640 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.640 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.640 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:22.641 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.955064+0000", 2024-03-19T11:30:22.641 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003890896 2024-03-19T11:30:22.641 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.641 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.641 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:22.641 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966670+0000", 2024-03-19T11:30:22.641 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011606795 2024-03-19T11:30:22.641 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.641 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.641 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:22.641 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966731+0000", 2024-03-19T11:30:22.641 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0528000000000001e-05 2024-03-19T11:30:22.641 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.641 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.641 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:22.642 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966785+0000", 2024-03-19T11:30:22.642 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4437000000000001e-05 2024-03-19T11:30:22.642 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.642 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:22.642 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.642 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.642 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.642 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1068 4.0 4:01018511:::smithi03829856-19:head [write 1681102~474010] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:22.642 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.956456+0000", 2024-03-19T11:30:22.642 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.6289929809999997, 2024-03-19T11:30:22.642 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010432765, 2024-03-19T11:30:22.642 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:22.642 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:22.642 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:22.642 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:22.643 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:22.643 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1068 2024-03-19T11:30:22.643 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.643 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:22.643 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.643 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:22.643 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956456+0000", 2024-03-19T11:30:22.643 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.643 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.643 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.643 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:22.643 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956456+0000", 2024-03-19T11:30:22.643 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.643 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.643 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.644 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:22.644 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956458+0000", 2024-03-19T11:30:22.644 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.742e-06 2024-03-19T11:30:22.644 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.644 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.644 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:22.644 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956655+0000", 2024-03-19T11:30:22.644 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019682200000000001 2024-03-19T11:30:22.644 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.644 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.644 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:22.644 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956669+0000", 2024-03-19T11:30:22.644 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3978000000000001e-05 2024-03-19T11:30:22.644 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.644 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.645 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:22.645 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956678+0000", 2024-03-19T11:30:22.645 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5029999999999997e-06 2024-03-19T11:30:22.645 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.645 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.645 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:22.645 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956729+0000", 2024-03-19T11:30:22.645 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1765999999999999e-05 2024-03-19T11:30:22.645 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.645 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.645 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:22.645 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956795+0000", 2024-03-19T11:30:22.645 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5701e-05 2024-03-19T11:30:22.645 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.645 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.646 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:22.646 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956886+0000", 2024-03-19T11:30:22.646 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1226999999999999e-05 2024-03-19T11:30:22.646 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.646 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.646 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:22.646 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.958609+0000", 2024-03-19T11:30:22.646 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017227049999999999 2024-03-19T11:30:22.646 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.646 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.646 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:22.646 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966837+0000", 2024-03-19T11:30:22.646 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0082276439999999992 2024-03-19T11:30:22.646 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.647 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.647 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:22.647 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966866+0000", 2024-03-19T11:30:22.647 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9323999999999999e-05 2024-03-19T11:30:22.647 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.647 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.647 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:22.647 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966889+0000", 2024-03-19T11:30:22.647 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3353000000000002e-05 2024-03-19T11:30:22.647 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.647 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:22.647 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.647 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.647 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.647 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.0.13:50 3.2 3:5595a68d:::07854d2fef297a06ba81685e660c332de36d5d18d546927d30daad6d7fda1541:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:22.648 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.166341+0000", 2024-03-19T11:30:22.648 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.4191081150000002, 2024-03-19T11:30:22.648 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010067153000000001, 2024-03-19T11:30:22.648 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:22.648 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:22.648 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:22.648 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.648 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:22.648 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166341+0000", 2024-03-19T11:30:22.648 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.648 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.649 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.649 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:22.649 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166341+0000", 2024-03-19T11:30:22.649 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.649 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.649 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.649 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:22.649 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166344+0000", 2024-03-19T11:30:22.649 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1370000000000002e-06 2024-03-19T11:30:22.649 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.649 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.649 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:22.650 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166828+0000", 2024-03-19T11:30:22.650 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00048343600000000001 2024-03-19T11:30:22.650 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.650 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.650 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:22.650 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166857+0000", 2024-03-19T11:30:22.650 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8880000000000001e-05 2024-03-19T11:30:22.650 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.650 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.650 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:22.650 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166871+0000", 2024-03-19T11:30:22.650 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4384e-05 2024-03-19T11:30:22.650 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.651 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.651 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:22.651 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167222+0000", 2024-03-19T11:30:22.651 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00035059000000000002 2024-03-19T11:30:22.651 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.651 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.651 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:22.651 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167334+0000", 2024-03-19T11:30:22.652 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000112663 2024-03-19T11:30:22.652 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.652 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.652 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:22.652 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167533+0000", 2024-03-19T11:30:22.652 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019901399999999999 2024-03-19T11:30:22.652 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.652 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.652 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:22.652 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176237+0000", 2024-03-19T11:30:22.652 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0087039890000000005 2024-03-19T11:30:22.652 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.652 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.652 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:22.653 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176260+0000", 2024-03-19T11:30:22.653 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2889000000000001e-05 2024-03-19T11:30:22.653 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.653 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.653 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:22.653 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176343+0000", 2024-03-19T11:30:22.653 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2651000000000005e-05 2024-03-19T11:30:22.653 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.653 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.653 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:22.653 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176408+0000", 2024-03-19T11:30:22.653 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5519999999999996e-05 2024-03-19T11:30:22.653 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.653 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:22.653 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.654 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.654 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.654 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.0.13:51 3.2 3:587bed34:::13074eb9c7a477811d595d489af192576515e7c716c0b501b140fcc4687b768b:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:22.654 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.166912+0000", 2024-03-19T11:30:22.654 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.4185373119999998, 2024-03-19T11:30:22.654 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0097465639999999992, 2024-03-19T11:30:22.654 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:22.654 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:22.654 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:22.654 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.654 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:22.654 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166912+0000", 2024-03-19T11:30:22.654 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.654 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.654 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.655 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:22.655 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166912+0000", 2024-03-19T11:30:22.655 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.655 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.655 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.655 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:22.655 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166914+0000", 2024-03-19T11:30:22.655 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.468e-06 2024-03-19T11:30:22.655 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.655 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.655 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:22.655 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167000+0000", 2024-03-19T11:30:22.655 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6311999999999996e-05 2024-03-19T11:30:22.655 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.655 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.655 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:22.656 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167011+0000", 2024-03-19T11:30:22.656 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.153e-05 2024-03-19T11:30:22.656 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.656 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.656 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:22.656 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167019+0000", 2024-03-19T11:30:22.656 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7970000000000001e-06 2024-03-19T11:30:22.656 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.656 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.656 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:22.656 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167864+0000", 2024-03-19T11:30:22.656 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00084482600000000002 2024-03-19T11:30:22.656 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.656 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.656 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:22.657 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.169110+0000", 2024-03-19T11:30:22.657 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0012456399999999999 2024-03-19T11:30:22.657 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.657 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.657 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:22.657 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.169467+0000", 2024-03-19T11:30:22.657 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00035692100000000001 2024-03-19T11:30:22.657 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.657 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.657 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:22.657 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176426+0000", 2024-03-19T11:30:22.657 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0069589600000000001 2024-03-19T11:30:22.657 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.657 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.657 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:22.658 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176524+0000", 2024-03-19T11:30:22.658 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8183000000000005e-05 2024-03-19T11:30:22.658 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.658 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.658 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:22.658 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176603+0000", 2024-03-19T11:30:22.658 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9035999999999993e-05 2024-03-19T11:30:22.658 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.658 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.658 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:22.658 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176659+0000", 2024-03-19T11:30:22.658 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5890999999999997e-05 2024-03-19T11:30:22.658 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.658 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:22.658 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.659 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.659 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.659 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.0.13:52 3.2 3:46a26328:::b2b262672c1f15ae09b098a7a21c74662e4816397daed8ca08fb83d0cd3a0cac:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:22.659 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.167033+0000", 2024-03-19T11:30:22.659 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.4184165029999996, 2024-03-19T11:30:22.659 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010113297, 2024-03-19T11:30:22.659 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:22.659 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:22.659 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:22.659 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.659 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:22.659 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167033+0000", 2024-03-19T11:30:22.659 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.659 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.659 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.660 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:22.660 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167033+0000", 2024-03-19T11:30:22.660 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.660 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.660 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.660 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:22.660 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167034+0000", 2024-03-19T11:30:22.660 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6599999999999996e-07 2024-03-19T11:30:22.660 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.660 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.660 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:22.660 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167092+0000", 2024-03-19T11:30:22.660 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7989999999999999e-05 2024-03-19T11:30:22.660 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.660 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.661 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:22.661 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167100+0000", 2024-03-19T11:30:22.661 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9670000000000001e-06 2024-03-19T11:30:22.661 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.661 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.661 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:22.661 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167108+0000", 2024-03-19T11:30:22.661 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3559999999999993e-06 2024-03-19T11:30:22.661 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.661 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.661 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:22.661 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.170462+0000", 2024-03-19T11:30:22.661 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033540229999999998 2024-03-19T11:30:22.661 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.662 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.662 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:22.662 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.171193+0000", 2024-03-19T11:30:22.662 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00073086899999999998 2024-03-19T11:30:22.662 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.662 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.662 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:22.662 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.171558+0000", 2024-03-19T11:30:22.662 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00036518899999999998 2024-03-19T11:30:22.662 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.662 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.662 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:22.662 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176440+0000", 2024-03-19T11:30:22.662 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0048817260000000003 2024-03-19T11:30:22.662 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.663 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.663 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:22.663 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177053+0000", 2024-03-19T11:30:22.663 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00061337100000000003 2024-03-19T11:30:22.663 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.663 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.663 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:22.663 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177096+0000", 2024-03-19T11:30:22.663 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2846999999999998e-05 2024-03-19T11:30:22.663 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.663 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.663 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:22.663 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177146+0000", 2024-03-19T11:30:22.663 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0293000000000001e-05 2024-03-19T11:30:22.663 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.664 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:22.664 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.664 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.664 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.664 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.0.13:54 3.2 3:7b614c59:::a7a06fca755baa7858207cb4b067fffe8c25e8183433b950d6879f6969d41b0b:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:22.664 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.167117+0000", 2024-03-19T11:30:22.664 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.418331995, 2024-03-19T11:30:22.664 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010796765999999999, 2024-03-19T11:30:22.664 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:22.664 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:22.664 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:22.664 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.664 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:22.664 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167117+0000", 2024-03-19T11:30:22.664 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.665 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.665 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.665 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:22.665 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167117+0000", 2024-03-19T11:30:22.665 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.665 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.665 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.665 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:22.665 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167118+0000", 2024-03-19T11:30:22.665 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2600000000000002e-07 2024-03-19T11:30:22.665 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.665 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.665 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:22.665 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167150+0000", 2024-03-19T11:30:22.665 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1576999999999999e-05 2024-03-19T11:30:22.666 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.666 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.666 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:22.666 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167157+0000", 2024-03-19T11:30:22.666 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2779999999999996e-06 2024-03-19T11:30:22.666 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.666 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.666 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:22.666 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167163+0000", 2024-03-19T11:30:22.666 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0120000000000002e-06 2024-03-19T11:30:22.666 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.666 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.666 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:22.666 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173324+0000", 2024-03-19T11:30:22.667 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0061614149999999999 2024-03-19T11:30:22.667 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.667 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.667 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:22.667 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173489+0000", 2024-03-19T11:30:22.667 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016438999999999999 2024-03-19T11:30:22.667 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.667 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.667 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:22.667 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173774+0000", 2024-03-19T11:30:22.667 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00028548000000000001 2024-03-19T11:30:22.667 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.667 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.667 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:22.668 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177668+0000", 2024-03-19T11:30:22.668 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038942450000000002 2024-03-19T11:30:22.668 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.668 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.668 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:22.668 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177792+0000", 2024-03-19T11:30:22.668 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012398199999999999 2024-03-19T11:30:22.668 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.668 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.668 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:22.668 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177861+0000", 2024-03-19T11:30:22.668 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8633999999999994e-05 2024-03-19T11:30:22.668 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.668 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.668 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:22.669 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177914+0000", 2024-03-19T11:30:22.669 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3226999999999997e-05 2024-03-19T11:30:22.669 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.669 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:22.669 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.669 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.669 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.669 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1230 4.0 4:0dc3b91e:::smithi03829856-28:head [write 662209~626546] snapc 9=[9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:22.669 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.170758+0000", 2024-03-19T11:30:22.669 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.4146912809999996, 2024-03-19T11:30:22.669 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011180066000000001, 2024-03-19T11:30:22.669 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:22.669 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:22.669 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:22.669 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:22.670 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:22.670 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1230 2024-03-19T11:30:22.670 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.670 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:22.670 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.670 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:22.670 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.170758+0000", 2024-03-19T11:30:22.670 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.670 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.670 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.670 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:22.670 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.170758+0000", 2024-03-19T11:30:22.670 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.670 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.671 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.671 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:22.671 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.170762+0000", 2024-03-19T11:30:22.671 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4960000000000001e-06 2024-03-19T11:30:22.671 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.671 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.671 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:22.671 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.171449+0000", 2024-03-19T11:30:22.671 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00068700499999999997 2024-03-19T11:30:22.671 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.671 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.671 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:22.671 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.171467+0000", 2024-03-19T11:30:22.671 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8389e-05 2024-03-19T11:30:22.671 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.672 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.672 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:22.672 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.171482+0000", 2024-03-19T11:30:22.672 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4619e-05 2024-03-19T11:30:22.672 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.672 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.672 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:22.672 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.171566+0000", 2024-03-19T11:30:22.672 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4788000000000002e-05 2024-03-19T11:30:22.672 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.672 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.672 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:22.672 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.171685+0000", 2024-03-19T11:30:22.672 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011861700000000001 2024-03-19T11:30:22.672 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.673 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.673 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:22.673 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.171835+0000", 2024-03-19T11:30:22.673 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014956299999999999 2024-03-19T11:30:22.673 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.673 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.673 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:22.673 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.181520+0000", 2024-03-19T11:30:22.673 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0096855750000000001 2024-03-19T11:30:22.673 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.673 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.673 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:22.673 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.181640+0000", 2024-03-19T11:30:22.673 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011966799999999999 2024-03-19T11:30:22.673 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.674 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.674 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:22.674 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.181811+0000", 2024-03-19T11:30:22.674 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017062699999999999 2024-03-19T11:30:22.674 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.674 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.674 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:22.674 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.181938+0000", 2024-03-19T11:30:22.674 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012771899999999999 2024-03-19T11:30:22.674 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.674 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:22.674 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.674 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.674 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.674 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1231 4.0 4:0dc3b91e:::smithi03829856-28:head [write 1808051~636013] snapc 9=[9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:22.675 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.173016+0000", 2024-03-19T11:30:22.675 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.4124334330000003, 2024-03-19T11:30:22.675 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013973719000000001, 2024-03-19T11:30:22.675 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:22.675 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:22.675 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:22.675 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:22.675 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:22.675 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1231 2024-03-19T11:30:22.675 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.675 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:22.675 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.675 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:22.675 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173016+0000", 2024-03-19T11:30:22.676 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.676 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.676 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.676 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:22.676 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173016+0000", 2024-03-19T11:30:22.676 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.676 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.676 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.676 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:22.676 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173019+0000", 2024-03-19T11:30:22.676 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.253e-06 2024-03-19T11:30:22.676 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.676 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.676 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:22.676 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176143+0000", 2024-03-19T11:30:22.677 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031240930000000001 2024-03-19T11:30:22.677 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.677 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.677 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:22.677 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176177+0000", 2024-03-19T11:30:22.677 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3237e-05 2024-03-19T11:30:22.677 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.677 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.677 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:22.677 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176196+0000", 2024-03-19T11:30:22.677 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9358e-05 2024-03-19T11:30:22.677 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.677 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.677 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:22.677 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176493+0000", 2024-03-19T11:30:22.678 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00029743899999999998 2024-03-19T11:30:22.678 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.678 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.678 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:22.678 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176786+0000", 2024-03-19T11:30:22.678 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00029259000000000002 2024-03-19T11:30:22.678 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.678 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.678 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:22.678 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177250+0000", 2024-03-19T11:30:22.678 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00046376000000000002 2024-03-19T11:30:22.678 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.678 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.678 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:22.678 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.182511+0000", 2024-03-19T11:30:22.679 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0052609019999999996 2024-03-19T11:30:22.679 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.679 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.679 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:22.679 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.186896+0000", 2024-03-19T11:30:22.679 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0043857009999999997 2024-03-19T11:30:22.679 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.679 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.679 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:22.679 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.186948+0000", 2024-03-19T11:30:22.679 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1653000000000002e-05 2024-03-19T11:30:22.679 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.679 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.679 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:22.679 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.186990+0000", 2024-03-19T11:30:22.680 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1733000000000002e-05 2024-03-19T11:30:22.680 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.680 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:22.680 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.680 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.680 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.680 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1277 4.0 4:01018511:::smithi03829856-19:head [write 763051~794393,stat] snapc b=[b,a,9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-03-19T11:30:22.680 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:21.953732+0000", 2024-03-19T11:30:22.680 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.63171719599999998, 2024-03-19T11:30:22.680 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011784380000000001, 2024-03-19T11:30:22.680 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:22.680 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:22.680 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:22.680 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:22.681 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:22.681 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1277 2024-03-19T11:30:22.681 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.681 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:22.681 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.681 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:22.681 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.953732+0000", 2024-03-19T11:30:22.681 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.681 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.681 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.681 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:22.681 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.953732+0000", 2024-03-19T11:30:22.681 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:22.681 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.681 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.682 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:22.682 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.953735+0000", 2024-03-19T11:30:22.682 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1999999999999999e-06 2024-03-19T11:30:22.682 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.682 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.682 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:22.682 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.954556+0000", 2024-03-19T11:30:22.682 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00082013599999999996 2024-03-19T11:30:22.682 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.682 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.682 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:22.682 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.954596+0000", 2024-03-19T11:30:22.682 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0741000000000002e-05 2024-03-19T11:30:22.682 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.682 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.683 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:22.683 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.954617+0000", 2024-03-19T11:30:22.683 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0375999999999999e-05 2024-03-19T11:30:22.683 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.683 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.683 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:22.683 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.954747+0000", 2024-03-19T11:30:22.683 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000130648 2024-03-19T11:30:22.683 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.683 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.683 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:22.683 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.954954+0000", 2024-03-19T11:30:22.683 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020694299999999999 2024-03-19T11:30:22.683 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.683 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.684 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:22.684 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.955252+0000", 2024-03-19T11:30:22.684 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00029740700000000002 2024-03-19T11:30:22.684 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.684 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.684 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:22.684 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.961677+0000", 2024-03-19T11:30:22.684 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0064253469999999997 2024-03-19T11:30:22.684 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.684 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.684 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:22.684 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.965373+0000", 2024-03-19T11:30:22.684 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0036962649999999998 2024-03-19T11:30:22.684 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.684 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.685 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:22.685 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.965451+0000", 2024-03-19T11:30:22.685 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7204000000000001e-05 2024-03-19T11:30:22.685 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:22.685 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:22.685 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:22.685 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.965517+0000", 2024-03-19T11:30:22.685 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6112999999999995e-05 2024-03-19T11:30:22.685 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.685 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:22.685 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.685 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:22.685 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:22.685 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:22.686 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:22.588+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:22.687 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-19T11:30:22.688 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:22.698+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:22.789 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:22.800+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:22.820 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:22.820 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:22.820 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:30:22.820 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:22.834 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-19T11:30:22.847 INFO:tasks.rados.rados.0.smithi038.stdout:787: done (4 left) 2024-03-19T11:30:22.847 INFO:tasks.rados.rados.0.smithi038.stdout:788: write oid 3 current snap is 10 2024-03-19T11:30:22.847 INFO:tasks.rados.rados.0.smithi038.stdout:788: seq_num 94 ranges {779858=469642,1817425=507031,2781526=586628} 2024-03-19T11:30:22.851 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:22.862+0000 7f1835403640 -1 osd.6 44 handle_osd_map injecting map crc failure 2024-03-19T11:30:22.865 INFO:tasks.rados.rados.0.smithi038.stdout:788: writing smithi03829856-3 from 779858 to 1249500 tid 1 2024-03-19T11:30:22.873 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-03-19T11:30:22.880 INFO:tasks.rados.rados.0.smithi038.stdout:788: writing smithi03829856-3 from 1817425 to 2324456 tid 2 2024-03-19T11:30:22.889 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:22.902+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:22.896 INFO:tasks.rados.rados.0.smithi038.stdout:788: writing smithi03829856-3 from 2781526 to 3368154 tid 3 2024-03-19T11:30:22.896 INFO:tasks.rados.rados.0.smithi038.stdout:789: write oid 29 current snap is 10 2024-03-19T11:30:22.896 INFO:tasks.rados.rados.0.smithi038.stdout:783: finishing set_chunk to oid 45 2024-03-19T11:30:22.896 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 45 set_chunk returned error code -95 offset: 795136 length: 1008128 2024-03-19T11:30:22.896 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 45 v 0 (ObjNum 59 snap 3 seq_num 59) dirty exists 2024-03-19T11:30:22.896 INFO:tasks.rados.rados.0.smithi038.stdout:785: finishing write tid 1 to smithi03829856-19 2024-03-19T11:30:22.897 INFO:tasks.rados.rados.0.smithi038.stdout:785: finishing write tid 2 to smithi03829856-19 2024-03-19T11:30:22.897 INFO:tasks.rados.rados.0.smithi038.stdout:789: seq_num 95 ranges {796665=544891,1759554=650689,2976243=620422} 2024-03-19T11:30:22.917 INFO:tasks.rados.rados.0.smithi038.stdout:789: writing smithi03829856-29 from 796665 to 1341556 tid 1 2024-03-19T11:30:22.929 INFO:tasks.rados.rados.0.smithi038.stdout:789: writing smithi03829856-29 from 1759554 to 2410243 tid 2 2024-03-19T11:30:22.942 INFO:tasks.rados.rados.0.smithi038.stdout:789: writing smithi03829856-29 from 2976243 to 3596665 tid 3 2024-03-19T11:30:22.942 INFO:tasks.rados.rados.0.smithi038.stdout:783: done (5 left) 2024-03-19T11:30:22.942 INFO:tasks.rados.rados.0.smithi038.stdout:784: done (4 left) 2024-03-19T11:30:22.942 INFO:tasks.rados.rados.0.smithi038.stdout:790: read oid 34 snap -1 2024-03-19T11:30:22.942 INFO:tasks.rados.rados.0.smithi038.stdout:790: expect (ObjNum 72 snap 5 seq_num 72) 2024-03-19T11:30:22.942 INFO:tasks.rados.rados.0.smithi038.stdout:791: read oid 35 snap -1 2024-03-19T11:30:22.942 INFO:tasks.rados.rados.0.smithi038.stdout:791: expect (ObjNum 34 snap 0 seq_num 34) 2024-03-19T11:30:22.943 INFO:tasks.rados.rados.0.smithi038.stdout:792: write (excl) oid 17 current snap is 10 2024-03-19T11:30:22.943 INFO:tasks.rados.rados.0.smithi038.stdout:792: seq_num 96 ranges {413472=620140,1818036=510700,3120019=493453} 2024-03-19T11:30:22.951 INFO:tasks.rados.rados.0.smithi038.stdout:792: writing smithi03829856-17 from 413472 to 1033612 tid 1 2024-03-19T11:30:22.962 INFO:tasks.rados.rados.0.smithi038.stdout:792: writing smithi03829856-17 from 1818036 to 2328736 tid 2 2024-03-19T11:30:22.972 INFO:tasks.rados.rados.0.smithi038.stdout:792: writing smithi03829856-17 from 3120019 to 3613472 tid 3 2024-03-19T11:30:22.972 INFO:tasks.rados.rados.0.smithi038.stdout:793: snap_remove snap 6 2024-03-19T11:30:22.973 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:22.973 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:22.973 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:30:22.973 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:30:22.973 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:22.990 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-19T11:30:23.033 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:23.046+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:23.093 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:23.104+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:23.150 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-19T11:30:23.159 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:23.159 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:30:23.160 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:30:23.160 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:30:23.160 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.160 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:9 4.d 4:b4ef97e0:::smithi03829856-3:head [write 433614~550498,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-03-19T11:30:23.160 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:04.706000+0000", 2024-03-19T11:30:23.160 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.463559272000001, 2024-03-19T11:30:23.160 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013227001, 2024-03-19T11:30:23.160 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:23.160 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:23.160 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:23.160 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:23.160 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:23.160 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 9 2024-03-19T11:30:23.160 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.160 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:23.161 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.161 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:23.161 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.706000+0000", 2024-03-19T11:30:23.161 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:23.161 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.161 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.161 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:23.161 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.706000+0000", 2024-03-19T11:30:23.161 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:23.161 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.161 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.161 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:23.161 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.706002+0000", 2024-03-19T11:30:23.161 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5569999999999998e-06 2024-03-19T11:30:23.162 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.162 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.162 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:23.162 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.707261+0000", 2024-03-19T11:30:23.162 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001258817 2024-03-19T11:30:23.162 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.162 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.162 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:23.162 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.707288+0000", 2024-03-19T11:30:23.162 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7093e-05 2024-03-19T11:30:23.162 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.162 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.162 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:23.163 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.707305+0000", 2024-03-19T11:30:23.163 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6330000000000001e-05 2024-03-19T11:30:23.163 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.163 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.163 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:23.163 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.707610+0000", 2024-03-19T11:30:23.163 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00030568799999999998 2024-03-19T11:30:23.163 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.163 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.163 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:23.163 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.707984+0000", 2024-03-19T11:30:23.163 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00037399299999999998 2024-03-19T11:30:23.163 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.163 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.163 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2", 2024-03-19T11:30:23.164 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.708409+0000", 2024-03-19T11:30:23.164 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00042503099999999998 2024-03-19T11:30:23.164 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.164 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.164 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:23.164 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.712919+0000", 2024-03-19T11:30:23.164 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0045099010000000002 2024-03-19T11:30:23.164 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.164 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.164 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:23.164 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.719037+0000", 2024-03-19T11:30:23.164 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061173770000000002 2024-03-19T11:30:23.164 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.164 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.165 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:23.165 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.719116+0000", 2024-03-19T11:30:23.165 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9935e-05 2024-03-19T11:30:23.165 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.165 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.165 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:23.165 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.719227+0000", 2024-03-19T11:30:23.165 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011027900000000001 2024-03-19T11:30:23.165 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:23.165 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:23.165 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:23.165 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.165 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.166 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:848 4.1 4:866f3dcc:::smithi03829856-31:head [read 0~2941192,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:23.166 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.757468+0000", 2024-03-19T11:30:23.166 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.41209128, 2024-03-19T11:30:23.166 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015020110999999999, 2024-03-19T11:30:23.166 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:23.166 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:23.166 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:23.166 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:23.166 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:23.166 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 848 2024-03-19T11:30:23.166 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.166 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:23.166 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.166 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:23.167 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757468+0000", 2024-03-19T11:30:23.167 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:23.167 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.167 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.167 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:23.167 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757468+0000", 2024-03-19T11:30:23.167 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:23.167 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.167 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.168 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:23.168 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757471+0000", 2024-03-19T11:30:23.168 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.67e-06 2024-03-19T11:30:23.168 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.168 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.168 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:23.168 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757488+0000", 2024-03-19T11:30:23.168 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6223e-05 2024-03-19T11:30:23.168 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.168 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.168 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:23.168 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757521+0000", 2024-03-19T11:30:23.168 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.358e-05 2024-03-19T11:30:23.169 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.169 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.169 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:23.169 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757543+0000", 2024-03-19T11:30:23.169 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1330000000000001e-05 2024-03-19T11:30:23.169 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.169 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.169 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:23.169 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757745+0000", 2024-03-19T11:30:23.169 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020259799999999999 2024-03-19T11:30:23.169 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.169 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.170 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op must wait for map", 2024-03-19T11:30:23.170 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757760+0000", 2024-03-19T11:30:23.170 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4647000000000001e-05 2024-03-19T11:30:23.170 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.170 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.170 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:23.170 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763972+0000", 2024-03-19T11:30:23.170 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062126480000000003 2024-03-19T11:30:23.170 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.170 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.170 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:23.170 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.764547+0000", 2024-03-19T11:30:23.170 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057440200000000001 2024-03-19T11:30:23.170 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.171 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.171 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:23.171 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.772488+0000", 2024-03-19T11:30:23.171 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079410130000000002 2024-03-19T11:30:23.171 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:23.171 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:23.171 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:23.171 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.171 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.171 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:849 4.1 4:866f3dcc:::smithi03829856-31:head [sparse-read 0~1359577] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:23.171 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.757583+0000", 2024-03-19T11:30:23.171 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.411975588000001, 2024-03-19T11:30:23.171 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016241202, 2024-03-19T11:30:23.172 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:23.172 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:23.172 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:23.172 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:23.172 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:23.172 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 849 2024-03-19T11:30:23.172 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.172 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:23.172 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.172 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:23.172 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757583+0000", 2024-03-19T11:30:23.172 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:23.172 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.172 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.172 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:23.173 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757583+0000", 2024-03-19T11:30:23.173 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:23.173 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.173 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.173 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:23.173 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757584+0000", 2024-03-19T11:30:23.173 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.077e-06 2024-03-19T11:30:23.173 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.173 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.173 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:23.173 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757593+0000", 2024-03-19T11:30:23.173 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3059999999999998e-06 2024-03-19T11:30:23.173 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.173 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.173 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:23.174 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757608+0000", 2024-03-19T11:30:23.174 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5526999999999999e-05 2024-03-19T11:30:23.174 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.174 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.174 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:23.174 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757618+0000", 2024-03-19T11:30:23.174 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5999999999999996e-06 2024-03-19T11:30:23.174 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.174 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.174 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:23.174 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757886+0000", 2024-03-19T11:30:23.174 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026839099999999999 2024-03-19T11:30:23.174 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.175 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.175 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:23.175 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757899+0000", 2024-03-19T11:30:23.175 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3004e-05 2024-03-19T11:30:23.175 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.175 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.175 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:23.175 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.772651+0000", 2024-03-19T11:30:23.175 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014751702 2024-03-19T11:30:23.175 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.175 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.175 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:23.175 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.772924+0000", 2024-03-19T11:30:23.175 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027314199999999998 2024-03-19T11:30:23.175 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.176 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.176 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:23.176 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.773825+0000", 2024-03-19T11:30:23.176 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00090045299999999995 2024-03-19T11:30:23.176 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:23.176 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:23.176 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:23.176 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.176 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.176 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:850 4.1 4:866f3dcc:::smithi03829856-31:head [read 0~2941192,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:23.176 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.757636+0000", 2024-03-19T11:30:23.176 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.411922918, 2024-03-19T11:30:23.176 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018834801000000002, 2024-03-19T11:30:23.176 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:23.177 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:23.177 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:23.177 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:23.177 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:23.177 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 850 2024-03-19T11:30:23.177 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.177 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:23.177 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.177 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:23.177 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757636+0000", 2024-03-19T11:30:23.177 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:23.177 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.177 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.177 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:23.177 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757636+0000", 2024-03-19T11:30:23.178 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:23.178 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.178 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.178 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:23.178 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757637+0000", 2024-03-19T11:30:23.178 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5999999999999991e-07 2024-03-19T11:30:23.178 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.178 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.178 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:23.178 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757643+0000", 2024-03-19T11:30:23.178 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3199999999999996e-06 2024-03-19T11:30:23.178 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.178 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.178 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:23.179 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757670+0000", 2024-03-19T11:30:23.179 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6682999999999998e-05 2024-03-19T11:30:23.179 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.179 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.179 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:23.179 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757679+0000", 2024-03-19T11:30:23.179 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.986e-06 2024-03-19T11:30:23.179 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.179 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.179 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:23.179 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757961+0000", 2024-03-19T11:30:23.179 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028230799999999999 2024-03-19T11:30:23.179 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.179 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.179 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:23.180 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757971+0000", 2024-03-19T11:30:23.180 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.38e-06 2024-03-19T11:30:23.180 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.180 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.180 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:23.180 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.774079+0000", 2024-03-19T11:30:23.180 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016108573000000001 2024-03-19T11:30:23.180 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.180 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.180 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:23.180 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.774306+0000", 2024-03-19T11:30:23.180 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022637499999999999 2024-03-19T11:30:23.180 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.180 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.180 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:23.181 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776471+0000", 2024-03-19T11:30:23.181 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021652160000000002 2024-03-19T11:30:23.181 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:23.181 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:23.181 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:23.181 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.181 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.181 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:851 4.9 4:953e720b:::smithi03829856-45:head [write 591613~454162,stat] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:23.181 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.763514+0000", 2024-03-19T11:30:23.181 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.406044726999999, 2024-03-19T11:30:23.181 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027825888999999999, 2024-03-19T11:30:23.181 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:23.181 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:23.181 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:23.181 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:23.182 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:23.182 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 851 2024-03-19T11:30:23.182 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.182 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:23.182 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.182 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:23.182 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763514+0000", 2024-03-19T11:30:23.182 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:23.182 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.182 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.182 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:23.182 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763514+0000", 2024-03-19T11:30:23.182 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:23.182 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.183 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.183 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:23.183 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763517+0000", 2024-03-19T11:30:23.183 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8499999999999998e-06 2024-03-19T11:30:23.183 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.183 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.183 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:23.183 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.764714+0000", 2024-03-19T11:30:23.183 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011971270000000001 2024-03-19T11:30:23.183 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.183 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.183 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:23.183 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.764745+0000", 2024-03-19T11:30:23.183 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0546999999999997e-05 2024-03-19T11:30:23.183 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.184 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.184 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:23.184 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.764767+0000", 2024-03-19T11:30:23.184 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2286999999999999e-05 2024-03-19T11:30:23.184 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.184 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.184 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:23.184 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.773924+0000", 2024-03-19T11:30:23.184 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0091567360000000004 2024-03-19T11:30:23.184 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.184 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.184 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:23.184 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.774217+0000", 2024-03-19T11:30:23.184 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029350099999999999 2024-03-19T11:30:23.184 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.184 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.185 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:23.185 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.774770+0000", 2024-03-19T11:30:23.185 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00055288600000000005 2024-03-19T11:30:23.185 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.185 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.185 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:23.185 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.786413+0000", 2024-03-19T11:30:23.185 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011643229999999999 2024-03-19T11:30:23.185 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.185 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.185 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:23.185 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791104+0000", 2024-03-19T11:30:23.185 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0046900830000000003 2024-03-19T11:30:23.185 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.186 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.186 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:23.186 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791248+0000", 2024-03-19T11:30:23.186 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014410200000000001 2024-03-19T11:30:23.186 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.186 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.186 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:23.186 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791340+0000", 2024-03-19T11:30:23.186 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2540000000000005e-05 2024-03-19T11:30:23.186 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:23.186 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:23.186 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:23.186 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.186 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.187 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:852 4.9 4:953e720b:::smithi03829856-45:head [write 1719614~417353] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:23.187 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.769707+0000", 2024-03-19T11:30:23.187 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.399851700999999, 2024-03-19T11:30:23.187 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022476643000000001, 2024-03-19T11:30:23.187 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:23.187 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:23.187 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:23.187 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:23.187 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:23.187 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 852 2024-03-19T11:30:23.187 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.187 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:23.187 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.187 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:23.188 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.769707+0000", 2024-03-19T11:30:23.188 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:23.188 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.188 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.188 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:23.188 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.769707+0000", 2024-03-19T11:30:23.188 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:23.188 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.188 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.188 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:23.188 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.769710+0000", 2024-03-19T11:30:23.188 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6230000000000001e-06 2024-03-19T11:30:23.188 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.188 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.188 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:23.188 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.770385+0000", 2024-03-19T11:30:23.189 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00067504900000000005 2024-03-19T11:30:23.189 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.189 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.189 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:23.189 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.770419+0000", 2024-03-19T11:30:23.189 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3528999999999997e-05 2024-03-19T11:30:23.189 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.189 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.189 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:23.189 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.770436+0000", 2024-03-19T11:30:23.189 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7615999999999999e-05 2024-03-19T11:30:23.189 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.189 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.189 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:23.190 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.781327+0000", 2024-03-19T11:30:23.190 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010891343 2024-03-19T11:30:23.190 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.190 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.190 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:23.190 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.781616+0000", 2024-03-19T11:30:23.190 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028857399999999999 2024-03-19T11:30:23.190 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.190 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.190 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:23.190 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.782125+0000", 2024-03-19T11:30:23.190 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00050941599999999995 2024-03-19T11:30:23.190 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.190 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.191 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:23.191 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.786606+0000", 2024-03-19T11:30:23.191 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044809560000000003 2024-03-19T11:30:23.191 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.191 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.191 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:23.191 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791995+0000", 2024-03-19T11:30:23.191 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053889150000000002 2024-03-19T11:30:23.191 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.191 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.191 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:23.191 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.792084+0000", 2024-03-19T11:30:23.191 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8633000000000003e-05 2024-03-19T11:30:23.191 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.191 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.192 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:23.192 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.792184+0000", 2024-03-19T11:30:23.192 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9988999999999994e-05 2024-03-19T11:30:23.192 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:23.192 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:23.192 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:23.192 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.192 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.192 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:853 4.9 4:953e720b:::smithi03829856-45:head [write 2570780~420833] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:23.192 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.775636+0000", 2024-03-19T11:30:23.192 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.39392308, 2024-03-19T11:30:23.192 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018795850999999999, 2024-03-19T11:30:23.192 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:23.192 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:23.192 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:23.193 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:23.193 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:23.193 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 853 2024-03-19T11:30:23.193 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.193 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:23.193 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.193 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:23.193 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.775636+0000", 2024-03-19T11:30:23.193 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:23.193 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.193 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.193 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:23.193 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.775636+0000", 2024-03-19T11:30:23.193 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:23.193 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.194 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.194 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:23.194 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.775639+0000", 2024-03-19T11:30:23.194 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4529999999999998e-06 2024-03-19T11:30:23.194 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.194 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.194 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:23.194 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776492+0000", 2024-03-19T11:30:23.194 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00085245000000000004 2024-03-19T11:30:23.194 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.194 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.194 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:23.194 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776526+0000", 2024-03-19T11:30:23.194 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4090000000000001e-05 2024-03-19T11:30:23.195 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.195 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.195 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:23.195 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776550+0000", 2024-03-19T11:30:23.195 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4139999999999999e-05 2024-03-19T11:30:23.195 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.195 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.195 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:23.195 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.786752+0000", 2024-03-19T11:30:23.195 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010201834 2024-03-19T11:30:23.195 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.195 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.195 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:23.195 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.786994+0000", 2024-03-19T11:30:23.195 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000241952 2024-03-19T11:30:23.195 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.196 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.196 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:23.196 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.787301+0000", 2024-03-19T11:30:23.196 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00030690699999999998 2024-03-19T11:30:23.196 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.196 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.196 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:23.196 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791965+0000", 2024-03-19T11:30:23.196 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0046644379999999999 2024-03-19T11:30:23.196 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.196 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.196 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:23.196 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.794165+0000", 2024-03-19T11:30:23.196 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021995389999999999 2024-03-19T11:30:23.196 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.197 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.197 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:23.197 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.794313+0000", 2024-03-19T11:30:23.197 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000148369 2024-03-19T11:30:23.197 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.197 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.197 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:23.197 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.794432+0000", 2024-03-19T11:30:23.197 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011867900000000001 2024-03-19T11:30:23.197 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:23.197 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:23.197 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:23.197 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.197 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.198 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:854 4.9 4:953e720b:::smithi03829856-45:head [setxattr _header (56) in=7b,truncate 2991613] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:23.198 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.776592+0000", 2024-03-19T11:30:23.198 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.392967358, 2024-03-19T11:30:23.198 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022815621000000001, 2024-03-19T11:30:23.198 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:23.198 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:23.198 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:23.198 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:23.198 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:23.198 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 854 2024-03-19T11:30:23.198 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.198 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:23.198 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.198 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:23.199 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776592+0000", 2024-03-19T11:30:23.199 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:23.199 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.199 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:23.199 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776592+0000", 2024-03-19T11:30:23.199 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:23.199 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.199 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:23.199 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776594+0000", 2024-03-19T11:30:23.199 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.95e-06 2024-03-19T11:30:23.199 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.199 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:23.200 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776603+0000", 2024-03-19T11:30:23.200 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2399999999999996e-06 2024-03-19T11:30:23.200 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.200 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.200 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:23.200 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776623+0000", 2024-03-19T11:30:23.200 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.035e-05 2024-03-19T11:30:23.200 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.200 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.200 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:23.200 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776639+0000", 2024-03-19T11:30:23.200 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5642999999999999e-05 2024-03-19T11:30:23.200 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.200 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.201 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:23.201 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.792324+0000", 2024-03-19T11:30:23.201 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015685075999999999 2024-03-19T11:30:23.201 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.201 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.201 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:23.201 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.792599+0000", 2024-03-19T11:30:23.201 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000275264 2024-03-19T11:30:23.201 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.201 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.201 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:23.201 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.793092+0000", 2024-03-19T11:30:23.201 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00049264600000000001 2024-03-19T11:30:23.201 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.201 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.202 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:23.202 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.794668+0000", 2024-03-19T11:30:23.202 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001576072 2024-03-19T11:30:23.202 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.202 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.202 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:23.202 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.799082+0000", 2024-03-19T11:30:23.202 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044145190000000004 2024-03-19T11:30:23.202 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.202 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.202 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:23.202 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.799249+0000", 2024-03-19T11:30:23.202 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016701199999999999 2024-03-19T11:30:23.202 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.203 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:23.203 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.799407+0000", 2024-03-19T11:30:23.203 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000157849 2024-03-19T11:30:23.203 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:23.203 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:23.203 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:23.203 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.203 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:855 4.9 4:953e720b:::smithi03829856-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:23.203 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.776660+0000", 2024-03-19T11:30:23.203 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.392898765, 2024-03-19T11:30:23.203 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023419786000000001, 2024-03-19T11:30:23.203 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:23.203 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:23.204 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:23.204 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:23.204 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:23.204 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 855 2024-03-19T11:30:23.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.204 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:23.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.204 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:23.204 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776660+0000", 2024-03-19T11:30:23.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:23.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.204 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:23.204 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776660+0000", 2024-03-19T11:30:23.205 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:23.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.205 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:23.205 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776662+0000", 2024-03-19T11:30:23.205 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3430000000000001e-06 2024-03-19T11:30:23.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.205 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:23.205 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776668+0000", 2024-03-19T11:30:23.205 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3500000000000002e-06 2024-03-19T11:30:23.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.205 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:23.205 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776687+0000", 2024-03-19T11:30:23.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9273999999999999e-05 2024-03-19T11:30:23.206 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.206 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.206 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:23.206 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776705+0000", 2024-03-19T11:30:23.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7839999999999999e-05 2024-03-19T11:30:23.206 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.206 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.206 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:23.206 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.793874+0000", 2024-03-19T11:30:23.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017168597000000001 2024-03-19T11:30:23.206 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.206 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.206 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-03-19T11:30:23.207 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.794132+0000", 2024-03-19T11:30:23.207 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025790399999999999 2024-03-19T11:30:23.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.207 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:23.207 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.799504+0000", 2024-03-19T11:30:23.207 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053724159999999997 2024-03-19T11:30:23.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.207 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:23.207 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.799782+0000", 2024-03-19T11:30:23.207 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027785499999999997 2024-03-19T11:30:23.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.208 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:23.208 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.800080+0000", 2024-03-19T11:30:23.208 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029820699999999999 2024-03-19T11:30:23.208 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:23.208 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:23.208 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:23.208 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.208 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.208 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:871 4.d 4:b4ef97e0:::smithi03829856-3:head [write 642034~796262,stat] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:23.208 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:09.028192+0000", 2024-03-19T11:30:23.208 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.141367034, 2024-03-19T11:30:23.208 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013367027, 2024-03-19T11:30:23.208 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:23.208 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:23.208 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:23.209 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:23.209 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:23.209 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 871 2024-03-19T11:30:23.209 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.209 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:23.209 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.209 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:23.209 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.028192+0000", 2024-03-19T11:30:23.209 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:23.209 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.209 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.209 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:23.209 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.028192+0000", 2024-03-19T11:30:23.209 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:23.209 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.210 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.210 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:23.210 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.028194+0000", 2024-03-19T11:30:23.210 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3999999999999999e-06 2024-03-19T11:30:23.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.210 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.210 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:23.210 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.029358+0000", 2024-03-19T11:30:23.210 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011635549999999999 2024-03-19T11:30:23.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.210 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.210 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:23.210 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.029390+0000", 2024-03-19T11:30:23.210 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2215999999999998e-05 2024-03-19T11:30:23.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.211 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:23.211 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.029406+0000", 2024-03-19T11:30:23.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5543e-05 2024-03-19T11:30:23.211 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.211 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:23.211 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.029525+0000", 2024-03-19T11:30:23.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000119269 2024-03-19T11:30:23.211 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.211 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:23.211 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.029676+0000", 2024-03-19T11:30:23.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000150729 2024-03-19T11:30:23.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.212 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2", 2024-03-19T11:30:23.212 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.029994+0000", 2024-03-19T11:30:23.212 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00031876099999999998 2024-03-19T11:30:23.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.212 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:23.212 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.040441+0000", 2024-03-19T11:30:23.212 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010446446 2024-03-19T11:30:23.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.212 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:23.212 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.041327+0000", 2024-03-19T11:30:23.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00088595299999999998 2024-03-19T11:30:23.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.213 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.213 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:23.213 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.041409+0000", 2024-03-19T11:30:23.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2332999999999999e-05 2024-03-19T11:30:23.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.213 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.213 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:23.213 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.041559+0000", 2024-03-19T11:30:23.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000149822 2024-03-19T11:30:23.213 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:23.213 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:23.213 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:23.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.214 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.214 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:993 4.e 4:72163e4c:::smithi03829856-39:head [write 776490~714607] snapc 5=[5,2] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-03-19T11:30:23.214 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:12.362932+0000", 2024-03-19T11:30:23.214 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.806627341, 2024-03-19T11:30:23.214 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018349900999999998, 2024-03-19T11:30:23.214 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:23.214 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:23.214 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:23.214 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:23.214 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:23.214 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 993 2024-03-19T11:30:23.214 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.214 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:23.214 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.215 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:23.215 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.362932+0000", 2024-03-19T11:30:23.215 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:23.215 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.215 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.215 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:23.215 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.362932+0000", 2024-03-19T11:30:23.215 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:23.215 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.215 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.215 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:23.215 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.362935+0000", 2024-03-19T11:30:23.215 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.88e-06 2024-03-19T11:30:23.216 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.216 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:23.216 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.366756+0000", 2024-03-19T11:30:23.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003821217 2024-03-19T11:30:23.216 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.216 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:23.216 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.366796+0000", 2024-03-19T11:30:23.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9866000000000001e-05 2024-03-19T11:30:23.216 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.216 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:23.216 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.366813+0000", 2024-03-19T11:30:23.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.755e-05 2024-03-19T11:30:23.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:23.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.366993+0000", 2024-03-19T11:30:23.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000179605 2024-03-19T11:30:23.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:23.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.367167+0000", 2024-03-19T11:30:23.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017421200000000001 2024-03-19T11:30:23.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-19T11:30:23.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.367505+0000", 2024-03-19T11:30:23.218 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033800099999999999 2024-03-19T11:30:23.218 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.218 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.218 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:23.218 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.380361+0000", 2024-03-19T11:30:23.218 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012855961000000001 2024-03-19T11:30:23.218 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.218 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.218 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:23.218 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.380987+0000", 2024-03-19T11:30:23.218 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00062590499999999995 2024-03-19T11:30:23.218 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.218 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.218 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:23.219 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.381149+0000", 2024-03-19T11:30:23.219 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000162239 2024-03-19T11:30:23.219 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.219 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:23.219 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.381282+0000", 2024-03-19T11:30:23.219 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013246500000000001 2024-03-19T11:30:23.219 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:23.219 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:23.219 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:23.219 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.219 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(osd.1.14:42 3.3 3:f0b412b0:::85cff1f6132709ad3f6974c608c4ca3d22c7ae775eb2def51616b490c1e0306e:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:23.219 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:13.796997+0000", 2024-03-19T11:30:23.220 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.3725619410000007, 2024-03-19T11:30:23.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012796138, 2024-03-19T11:30:23.220 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:23.220 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:23.220 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:23.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.220 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:23.220 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.796997+0000", 2024-03-19T11:30:23.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:23.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.220 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:23.220 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.796997+0000", 2024-03-19T11:30:23.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:23.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:23.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.796998+0000", 2024-03-19T11:30:23.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5999999999999991e-07 2024-03-19T11:30:23.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:23.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.797240+0000", 2024-03-19T11:30:23.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00024155800000000001 2024-03-19T11:30:23.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:23.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.797255+0000", 2024-03-19T11:30:23.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4983e-05 2024-03-19T11:30:23.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:23.222 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.797270+0000", 2024-03-19T11:30:23.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4953000000000001e-05 2024-03-19T11:30:23.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:23.222 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.805058+0000", 2024-03-19T11:30:23.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0077889200000000004 2024-03-19T11:30:23.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:23.222 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.805289+0000", 2024-03-19T11:30:23.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000230115 2024-03-19T11:30:23.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.223 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:23.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.805648+0000", 2024-03-19T11:30:23.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00035961400000000002 2024-03-19T11:30:23.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.223 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:23.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.809089+0000", 2024-03-19T11:30:23.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034412499999999999 2024-03-19T11:30:23.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.223 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:23.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.809502+0000", 2024-03-19T11:30:23.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00041262100000000001 2024-03-19T11:30:23.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.224 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:23.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.809645+0000", 2024-03-19T11:30:23.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000142659 2024-03-19T11:30:23.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.224 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:23.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.809793+0000", 2024-03-19T11:30:23.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000148505 2024-03-19T11:30:23.224 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:23.224 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:23.224 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:23.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.224 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(osd.1.14:43 3.3 3:e8dad25b:::1ae0f6dc7c6ed0f10ae413d6bba06d7bb8915eb6bbafbf428184acbe98e3e057:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:23.224 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:13.797286+0000", 2024-03-19T11:30:23.224 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.3722730829999996, 2024-03-19T11:30:23.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012678534999999999, 2024-03-19T11:30:23.225 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:23.225 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:23.225 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:23.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:23.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.797286+0000", 2024-03-19T11:30:23.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:23.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:23.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.797286+0000", 2024-03-19T11:30:23.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:23.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:23.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.797287+0000", 2024-03-19T11:30:23.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.16e-07 2024-03-19T11:30:23.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:23.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.797426+0000", 2024-03-19T11:30:23.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013880899999999999 2024-03-19T11:30:23.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:23.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.797452+0000", 2024-03-19T11:30:23.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.616e-05 2024-03-19T11:30:23.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.227 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:23.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.797468+0000", 2024-03-19T11:30:23.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6249999999999999e-05 2024-03-19T11:30:23.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.227 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:23.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.807167+0000", 2024-03-19T11:30:23.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0096989290000000002 2024-03-19T11:30:23.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.227 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:23.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.807529+0000", 2024-03-19T11:30:23.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00036221000000000002 2024-03-19T11:30:23.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.228 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.228 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:23.228 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.808040+0000", 2024-03-19T11:30:23.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00051056599999999995 2024-03-19T11:30:23.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.228 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.228 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:23.228 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.809455+0000", 2024-03-19T11:30:23.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0014148819999999999 2024-03-19T11:30:23.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.228 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.228 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:23.229 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.809833+0000", 2024-03-19T11:30:23.229 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000378694 2024-03-19T11:30:23.229 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.229 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.229 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:23.229 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.809911+0000", 2024-03-19T11:30:23.229 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7200000000000006e-05 2024-03-19T11:30:23.229 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.229 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.229 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:23.229 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.809964+0000", 2024-03-19T11:30:23.229 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3919000000000003e-05 2024-03-19T11:30:23.229 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:23.229 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:23.230 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:23.230 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.230 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1072 4.1 4:8401c1f0:::smithi03829856-43:head [write 494139~557703,stat] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:23.230 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:13.972007+0000", 2024-03-19T11:30:23.230 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.1975515909999999, 2024-03-19T11:30:23.230 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.01935839, 2024-03-19T11:30:23.230 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:23.230 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:23.230 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:23.230 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:23.230 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:23.230 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1072 2024-03-19T11:30:23.230 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.230 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:23.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.231 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:23.231 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.972007+0000", 2024-03-19T11:30:23.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:23.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.231 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:23.231 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.972007+0000", 2024-03-19T11:30:23.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:23.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.231 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:23.231 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.972011+0000", 2024-03-19T11:30:23.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9360000000000001e-06 2024-03-19T11:30:23.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.232 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:23.232 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.972881+0000", 2024-03-19T11:30:23.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00086922699999999998 2024-03-19T11:30:23.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.232 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:23.232 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.972917+0000", 2024-03-19T11:30:23.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6359999999999997e-05 2024-03-19T11:30:23.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.232 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:23.232 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.972934+0000", 2024-03-19T11:30:23.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7249e-05 2024-03-19T11:30:23.233 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.233 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:23.233 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.973132+0000", 2024-03-19T11:30:23.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019760200000000001 2024-03-19T11:30:23.233 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.233 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:23.233 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.973303+0000", 2024-03-19T11:30:23.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017142899999999999 2024-03-19T11:30:23.233 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.233 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:23.233 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.973619+0000", 2024-03-19T11:30:23.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00031594700000000002 2024-03-19T11:30:23.233 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.234 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.234 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:23.234 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.985856+0000", 2024-03-19T11:30:23.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012236349000000001 2024-03-19T11:30:23.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.234 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.234 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:23.234 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.991132+0000", 2024-03-19T11:30:23.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052764960000000003 2024-03-19T11:30:23.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.234 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.234 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:23.234 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.991241+0000", 2024-03-19T11:30:23.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000108842 2024-03-19T11:30:23.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:23.235 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.991366+0000", 2024-03-19T11:30:23.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012495299999999999 2024-03-19T11:30:23.235 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:23.235 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:23.235 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:23.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.235 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1162 4.1 4:8401c1f0:::smithi03829856-43:head [write 594981~609197] snapc 8=[8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-03-19T11:30:23.235 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.838981+0000", 2024-03-19T11:30:23.235 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.330578236, 2024-03-19T11:30:23.236 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027704349999999999, 2024-03-19T11:30:23.236 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:23.236 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:23.236 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:23.236 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:23.236 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:23.236 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1162 2024-03-19T11:30:23.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.236 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:23.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:23.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.838981+0000", 2024-03-19T11:30:23.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:23.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:23.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.838981+0000", 2024-03-19T11:30:23.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:23.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:23.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.838983+0000", 2024-03-19T11:30:23.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5399999999999998e-06 2024-03-19T11:30:23.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:23.238 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841322+0000", 2024-03-19T11:30:23.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002338592 2024-03-19T11:30:23.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:23.238 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841371+0000", 2024-03-19T11:30:23.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9123999999999999e-05 2024-03-19T11:30:23.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:23.238 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841399+0000", 2024-03-19T11:30:23.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7718000000000001e-05 2024-03-19T11:30:23.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:23.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841577+0000", 2024-03-19T11:30:23.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000178097 2024-03-19T11:30:23.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:23.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841825+0000", 2024-03-19T11:30:23.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00024824299999999999 2024-03-19T11:30:23.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:23.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842227+0000", 2024-03-19T11:30:23.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00040212199999999998 2024-03-19T11:30:23.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:23.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.865182+0000", 2024-03-19T11:30:23.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022954810999999999 2024-03-19T11:30:23.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:23.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.866360+0000", 2024-03-19T11:30:23.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001178449 2024-03-19T11:30:23.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:23.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.866538+0000", 2024-03-19T11:30:23.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017791000000000001 2024-03-19T11:30:23.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:23.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.866685+0000", 2024-03-19T11:30:23.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000146744 2024-03-19T11:30:23.241 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:23.241 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:23.241 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:23.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.241 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1163 4.1 4:8401c1f0:::smithi03829856-43:head [write 1974123~745028] snapc 8=[8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-03-19T11:30:23.241 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.851879+0000", 2024-03-19T11:30:23.241 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.3176801139999998, 2024-03-19T11:30:23.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015531774, 2024-03-19T11:30:23.241 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:23.241 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:23.241 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:23.242 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:23.242 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:23.242 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1163 2024-03-19T11:30:23.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.242 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:23.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.242 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:23.242 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.851879+0000", 2024-03-19T11:30:23.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:23.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.242 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:23.242 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.851879+0000", 2024-03-19T11:30:23.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:23.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.243 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:23.243 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.851882+0000", 2024-03-19T11:30:23.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6960000000000001e-06 2024-03-19T11:30:23.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.243 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:23.243 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.853204+0000", 2024-03-19T11:30:23.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0013226620000000001 2024-03-19T11:30:23.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.243 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:23.243 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.853231+0000", 2024-03-19T11:30:23.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6775e-05 2024-03-19T11:30:23.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:23.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.853246+0000", 2024-03-19T11:30:23.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5403000000000001e-05 2024-03-19T11:30:23.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:23.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.853363+0000", 2024-03-19T11:30:23.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000116296 2024-03-19T11:30:23.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:23.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.853572+0000", 2024-03-19T11:30:23.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020931500000000001 2024-03-19T11:30:23.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:23.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.853797+0000", 2024-03-19T11:30:23.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022544700000000001 2024-03-19T11:30:23.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:23.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.865195+0000", 2024-03-19T11:30:23.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011397981999999999 2024-03-19T11:30:23.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:23.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.866916+0000", 2024-03-19T11:30:23.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017204449999999999 2024-03-19T11:30:23.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:23.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.867129+0000", 2024-03-19T11:30:23.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021317999999999999 2024-03-19T11:30:23.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:23.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.867411+0000", 2024-03-19T11:30:23.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028157300000000002 2024-03-19T11:30:23.246 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:23.246 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:23.246 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:23.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.247 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1205 4.1 4:8401c1f0:::smithi03829856-43:head [assert-version v50,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:23.247 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:18.112028+0000", 2024-03-19T11:30:23.247 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.0575309800000001, 2024-03-19T11:30:23.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013579206, 2024-03-19T11:30:23.247 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:23.247 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:23.247 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:23.247 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:23.247 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:23.247 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1205 2024-03-19T11:30:23.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.247 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:23.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.247 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:23.247 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112028+0000", 2024-03-19T11:30:23.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:23.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.248 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:23.248 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112028+0000", 2024-03-19T11:30:23.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:23.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.248 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:23.248 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112030+0000", 2024-03-19T11:30:23.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4549999999999998e-06 2024-03-19T11:30:23.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:23.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112043+0000", 2024-03-19T11:30:23.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2721999999999999e-05 2024-03-19T11:30:23.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:23.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112075+0000", 2024-03-19T11:30:23.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.205e-05 2024-03-19T11:30:23.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:23.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112091+0000", 2024-03-19T11:30:23.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5364000000000001e-05 2024-03-19T11:30:23.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:23.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112223+0000", 2024-03-19T11:30:23.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013224100000000001 2024-03-19T11:30:23.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:23.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112393+0000", 2024-03-19T11:30:23.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000169681 2024-03-19T11:30:23.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:23.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.120919+0000", 2024-03-19T11:30:23.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085260140000000002 2024-03-19T11:30:23.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:23.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125207+0000", 2024-03-19T11:30:23.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004288722 2024-03-19T11:30:23.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:23.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125343+0000", 2024-03-19T11:30:23.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013558400000000001 2024-03-19T11:30:23.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:23.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125497+0000", 2024-03-19T11:30:23.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000153678 2024-03-19T11:30:23.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:23.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125607+0000", 2024-03-19T11:30:23.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110695 2024-03-19T11:30:23.252 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:23.252 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:23.252 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:23.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.252 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1208 4.1 4:866f3dcc:::smithi03829856-31:head [read 0~2941192,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:23.252 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:18.112252+0000", 2024-03-19T11:30:23.252 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.0573074099999999, 2024-03-19T11:30:23.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.01291702, 2024-03-19T11:30:23.252 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:23.253 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:23.253 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:23.253 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:23.253 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:23.253 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1208 2024-03-19T11:30:23.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.253 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:23.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:23.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112252+0000", 2024-03-19T11:30:23.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:23.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:23.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112252+0000", 2024-03-19T11:30:23.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:23.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.254 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:23.254 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112253+0000", 2024-03-19T11:30:23.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6299999999999993e-07 2024-03-19T11:30:23.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.254 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:23.254 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112257+0000", 2024-03-19T11:30:23.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4360000000000004e-06 2024-03-19T11:30:23.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.254 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:23.254 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112272+0000", 2024-03-19T11:30:23.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.518e-05 2024-03-19T11:30:23.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:23.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112281+0000", 2024-03-19T11:30:23.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.422e-06 2024-03-19T11:30:23.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:23.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.122861+0000", 2024-03-19T11:30:23.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010580523 2024-03-19T11:30:23.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:23.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123148+0000", 2024-03-19T11:30:23.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028714499999999999 2024-03-19T11:30:23.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:23.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125169+0000", 2024-03-19T11:30:23.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0020203510000000001 2024-03-19T11:30:23.256 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:23.256 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:23.256 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:23.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.256 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1243 4.9 4:9ef41243:::smithi03829856-5:head [tier-promote] snapc 9=[9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:23.256 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:18.345759+0000", 2024-03-19T11:30:23.256 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 4.8237997989999997, 2024-03-19T11:30:23.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018400962999999999, 2024-03-19T11:30:23.256 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:23.256 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:23.257 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:23.257 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:23.257 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:23.257 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1243 2024-03-19T11:30:23.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.257 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:23.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:23.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.345759+0000", 2024-03-19T11:30:23.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:23.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:23.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.345759+0000", 2024-03-19T11:30:23.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:23.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:23.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.345763+0000", 2024-03-19T11:30:23.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3790000000000002e-06 2024-03-19T11:30:23.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:23.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.345776+0000", 2024-03-19T11:30:23.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3784000000000001e-05 2024-03-19T11:30:23.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:23.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.345812+0000", 2024-03-19T11:30:23.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5599999999999998e-05 2024-03-19T11:30:23.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:23.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.345829+0000", 2024-03-19T11:30:23.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6722999999999999e-05 2024-03-19T11:30:23.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:23.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.345964+0000", 2024-03-19T11:30:23.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013554800000000001 2024-03-19T11:30:23.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:23.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.346133+0000", 2024-03-19T11:30:23.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000169211 2024-03-19T11:30:23.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:23.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.364160+0000", 2024-03-19T11:30:23.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018026718000000001 2024-03-19T11:30:23.260 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:23.260 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:23.260 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:23.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.260 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1271 4.9 4:953e720b:::smithi03829856-45:2 [read 795136~1008128] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-03-19T11:30:23.260 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:21.868056+0000", 2024-03-19T11:30:23.260 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.301502731, 2024-03-19T11:30:23.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.047270591000000001, 2024-03-19T11:30:23.261 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:23.261 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:23.261 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:23.261 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:23.261 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:23.261 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1271 2024-03-19T11:30:23.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.261 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:23.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.261 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:23.261 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868056+0000", 2024-03-19T11:30:23.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:23.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.262 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:23.262 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868056+0000", 2024-03-19T11:30:23.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:23.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.262 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:23.262 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868059+0000", 2024-03-19T11:30:23.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0469999999999998e-06 2024-03-19T11:30:23.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.262 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:23.262 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868072+0000", 2024-03-19T11:30:23.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2700999999999999e-05 2024-03-19T11:30:23.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.263 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:23.263 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868106+0000", 2024-03-19T11:30:23.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4152000000000001e-05 2024-03-19T11:30:23.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.263 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:23.263 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868126+0000", 2024-03-19T11:30:23.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9386999999999999e-05 2024-03-19T11:30:23.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.263 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:23.263 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.911447+0000", 2024-03-19T11:30:23.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.043321164000000002 2024-03-19T11:30:23.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:23.264 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.911762+0000", 2024-03-19T11:30:23.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00031492400000000001 2024-03-19T11:30:23.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:23.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:23.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:23.264 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.915327+0000", 2024-03-19T11:30:23.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003565216 2024-03-19T11:30:23.264 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:23.264 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:23.264 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:23.264 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:23.264 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:23.264 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:23.266 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-19T11:30:23.266 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:23.275+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:23.307 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:30:23.308 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:30:23.324 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:30:23.324 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:30:23.340 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:30:23.340 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:30:23.358 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:30:23.358 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:30:23.366 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:23.378+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:23.378 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:30:23.378 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:30:23.393 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:23.394 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:23.394 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:30:23.394 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:23.399 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:30:23.400 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:30:23.407 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-19T11:30:23.419 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:30:23.419 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:30:23.438 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:30:23.438 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:30:23.503 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:23.515+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:23.567 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:23.567 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:23.567 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:23.579+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:23.581 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-19T11:30:23.669 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:23.679+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:23.700 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:23.700 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:23.714 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-19T11:30:23.770 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:23.779+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:23.839 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:23.839 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:23.853 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-19T11:30:23.865 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:23.878+0000 7f143b8b4640 -1 osd.5 45 handle_osd_map injecting map crc failure 2024-03-19T11:30:23.866 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-03-19T11:30:23.867 INFO:tasks.rados.rados.0.smithi038.stdout:793: done (7 left) 2024-03-19T11:30:23.867 INFO:tasks.rados.rados.0.smithi038.stdout:794: rollback oid 13 current snap is 10 2024-03-19T11:30:23.867 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 13 to 3 2024-03-19T11:30:23.867 INFO:tasks.rados.rados.0.smithi038.stdout:785: finishing write tid 3 to smithi03829856-19 2024-03-19T11:30:23.867 INFO:tasks.rados.rados.0.smithi038.stdout:785: finishing write tid 4 to smithi03829856-19 2024-03-19T11:30:23.867 INFO:tasks.rados.rados.0.smithi038.stdout:785: finishing write tid 5 to smithi03829856-19 2024-03-19T11:30:23.867 INFO:tasks.rados.rados.0.smithi038.stdout:785: oid 19 updating version 0 to 32 2024-03-19T11:30:23.867 INFO:tasks.rados.rados.0.smithi038.stdout:785: oid 19 version 32 is already newer than 29 2024-03-19T11:30:23.867 INFO:tasks.rados.rados.0.smithi038.stdout:785: oid 19 version 32 is already newer than 31 2024-03-19T11:30:23.868 INFO:tasks.rados.rados.0.smithi038.stdout:785: oid 19 version 32 is already newer than 30 2024-03-19T11:30:23.868 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 19 v 32 (ObjNum 93 snap 10 seq_num 93) dirty exists 2024-03-19T11:30:23.868 INFO:tasks.rados.rados.0.smithi038.stdout:785: left oid 19 (ObjNum 93 snap 10 seq_num 93) 2024-03-19T11:30:23.868 INFO:tasks.rados.rados.0.smithi038.stdout:788: finishing write tid 1 to smithi03829856-3 2024-03-19T11:30:23.868 INFO:tasks.rados.rados.0.smithi038.stdout:785: done (7 left) 2024-03-19T11:30:23.868 INFO:tasks.rados.rados.0.smithi038.stdout:786: done (6 left) 2024-03-19T11:30:23.868 INFO:tasks.rados.rados.0.smithi038.stdout:795: write (excl) oid 36 current snap is 10 2024-03-19T11:30:23.868 INFO:tasks.rados.rados.0.smithi038.stdout:795: seq_num 97 ranges {430279=695389,1876518=654358,3031089=536797} 2024-03-19T11:30:23.871 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:23.882+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:23.874 INFO:tasks.rados.rados.0.smithi038.stdout:795: writing smithi03829856-36 from 430279 to 1125668 tid 1 2024-03-19T11:30:23.883 INFO:tasks.rados.rados.0.smithi038.stdout:795: writing smithi03829856-36 from 1876518 to 2530876 tid 2 2024-03-19T11:30:23.890 INFO:tasks.rados.rados.0.smithi038.stdout:795: writing smithi03829856-36 from 3031089 to 3567886 tid 3 2024-03-19T11:30:23.891 INFO:tasks.rados.rados.0.smithi038.stdout:796: read oid 16 snap -1 2024-03-19T11:30:23.891 INFO:tasks.rados.rados.0.smithi038.stdout:796: expect (ObjNum 65 snap 3 seq_num 65) 2024-03-19T11:30:23.891 INFO:tasks.rados.rados.0.smithi038.stdout:797: rollback oid 12 current snap is 10 2024-03-19T11:30:23.891 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 12 to 8 2024-03-19T11:30:23.891 INFO:tasks.rados.rados.0.smithi038.stdout:798: set_chunk oid 20 target oid 2024-03-19T11:30:23.900 INFO:tasks.rados.rados.0.smithi038.stdout:798: set_chunk oid 20 offset: 1601024 length: 358912 target oid 5e5c52bbd610eab2ed64a0fbc7c2ff361f309405838b08a6a837b471e862b3b1 offset: 0 snap 4 2024-03-19T11:30:23.900 INFO:tasks.rados.rados.0.smithi038.stdout:799: set_chunk oid 6 target oid 2024-03-19T11:30:23.908 INFO:tasks.rados.rados.0.smithi038.stdout:799: set_chunk oid 6 offset: 1552384 length: 828928 target oid edd9bf5132eb8e83145987238fab545bac80e88c8be187123ae178e472cfecf8 offset: 0 snap -1 2024-03-19T11:30:23.908 INFO:tasks.rados.rados.0.smithi038.stdout:800: write (excl) oid 42 current snap is 10 2024-03-19T11:30:23.909 INFO:tasks.rados.rados.0.smithi038.stdout:788: finishing write tid 2 to smithi03829856-3 2024-03-19T11:30:23.909 INFO:tasks.rados.rados.0.smithi038.stdout:800: seq_num 98 ranges {447086=770638,1935000=798016,3458512=188574} 2024-03-19T11:30:23.915 INFO:tasks.rados.rados.0.smithi038.stdout:800: writing smithi03829856-42 from 447086 to 1217724 tid 1 2024-03-19T11:30:23.924 INFO:tasks.rados.rados.0.smithi038.stdout:800: writing smithi03829856-42 from 1935000 to 2733016 tid 2 2024-03-19T11:30:23.930 INFO:tasks.rados.rados.0.smithi038.stdout:800: writing smithi03829856-42 from 3458512 to 3647086 tid 3 2024-03-19T11:30:23.930 INFO:tasks.rados.rados.0.smithi038.stdout:801: tier_evict oid 47 2024-03-19T11:30:23.930 INFO:tasks.rados.rados.0.smithi038.stdout:801: tier_evict oid 47 snap -1 2024-03-19T11:30:23.930 INFO:tasks.rados.rados.0.smithi038.stdout:788: finishing write tid 3 to smithi03829856-3 2024-03-19T11:30:23.930 INFO:tasks.rados.rados.0.smithi038.stdout:788: finishing write tid 4 to smithi03829856-3 2024-03-19T11:30:23.930 INFO:tasks.rados.rados.0.smithi038.stdout:788: finishing write tid 5 to smithi03829856-3 2024-03-19T11:30:23.930 INFO:tasks.rados.rados.0.smithi038.stdout:788: oid 3 updating version 0 to 30 2024-03-19T11:30:23.930 INFO:tasks.rados.rados.0.smithi038.stdout:788: oid 3 updating version 30 to 31 2024-03-19T11:30:23.930 INFO:tasks.rados.rados.0.smithi038.stdout:788: oid 3 updating version 31 to 33 2024-03-19T11:30:23.930 INFO:tasks.rados.rados.0.smithi038.stdout:788: oid 3 version 33 is already newer than 32 2024-03-19T11:30:23.930 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 3 v 33 (ObjNum 94 snap 10 seq_num 94) dirty exists 2024-03-19T11:30:23.931 INFO:tasks.rados.rados.0.smithi038.stdout:788: left oid 3 (ObjNum 94 snap 10 seq_num 94) 2024-03-19T11:30:23.931 INFO:tasks.rados.rados.0.smithi038.stdout:789: finishing write tid 1 to smithi03829856-29 2024-03-19T11:30:23.931 INFO:tasks.rados.rados.0.smithi038.stdout:789: finishing write tid 2 to smithi03829856-29 2024-03-19T11:30:23.931 INFO:tasks.rados.rados.0.smithi038.stdout:788: done (12 left) 2024-03-19T11:30:23.931 INFO:tasks.rados.rados.0.smithi038.stdout:802: set_chunk oid 41 target oid 2024-03-19T11:30:23.934 INFO:tasks.rados.rados.0.smithi038.stdout:802: set_chunk oid 41 offset: 68096 length: 363008 target oid 291b77ba4e406a423c8c38bde1953378dd949f809575886160f448d58c7e8400 offset: 0 snap -1 2024-03-19T11:30:23.934 INFO:tasks.rados.rados.0.smithi038.stdout:803: read oid 40 snap 4 2024-03-19T11:30:23.934 INFO:tasks.rados.rados.0.smithi038.stdout:803: expect (ObjNum 39 snap 0 seq_num 39) 2024-03-19T11:30:23.934 INFO:tasks.rados.rados.0.smithi038.stdout:804: set_chunk oid 43 target oid 2024-03-19T11:30:23.939 INFO:tasks.rados.rados.0.smithi038.stdout:804: set_chunk oid 43 offset: 1411584 length: 215040 target oid 70df764c92895eff303397274ae416541f20320d3954ade3c6d66e4b52683239 offset: 0 snap -1 2024-03-19T11:30:23.939 INFO:tasks.rados.rados.0.smithi038.stdout:805: tier_promote oid 14 2024-03-19T11:30:23.939 INFO:tasks.rados.rados.0.smithi038.stdout:791: expect (ObjNum 34 snap 0 seq_num 34) 2024-03-19T11:30:23.972 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:23.983+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:23.980 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:23.980 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:23.996 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-19T11:30:24.020 INFO:tasks.rados.rados.0.smithi038.stdout:790: expect (ObjNum 72 snap 5 seq_num 72) 2024-03-19T11:30:24.081 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:24.094+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:24.163 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:24.163 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:24.173 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:24.183+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:24.177 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-19T11:30:24.200 INFO:tasks.rados.rados.0.smithi038.stdout:789: finishing write tid 3 to smithi03829856-29 2024-03-19T11:30:24.200 INFO:tasks.rados.rados.0.smithi038.stdout:789: finishing write tid 4 to smithi03829856-29 2024-03-19T11:30:24.201 INFO:tasks.rados.rados.0.smithi038.stdout:789: finishing write tid 5 to smithi03829856-29 2024-03-19T11:30:24.201 INFO:tasks.rados.rados.0.smithi038.stdout:789: oid 29 updating version 0 to 54 2024-03-19T11:30:24.201 INFO:tasks.rados.rados.0.smithi038.stdout:789: oid 29 updating version 54 to 56 2024-03-19T11:30:24.201 INFO:tasks.rados.rados.0.smithi038.stdout:789: oid 29 version 56 is already newer than 53 2024-03-19T11:30:24.201 INFO:tasks.rados.rados.0.smithi038.stdout:789: oid 29 version 56 is already newer than 55 2024-03-19T11:30:24.201 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 29 v 56 (ObjNum 95 snap 10 seq_num 95) dirty exists 2024-03-19T11:30:24.201 INFO:tasks.rados.rados.0.smithi038.stdout:789: left oid 29 (ObjNum 95 snap 10 seq_num 95) 2024-03-19T11:30:24.201 INFO:tasks.rados.rados.0.smithi038.stdout:792: finishing write tid 1 to smithi03829856-17 2024-03-19T11:30:24.201 INFO:tasks.rados.rados.0.smithi038.stdout:792: finishing write tid 2 to smithi03829856-17 2024-03-19T11:30:24.201 INFO:tasks.rados.rados.0.smithi038.stdout:792: finishing write tid 3 to smithi03829856-17 2024-03-19T11:30:24.201 INFO:tasks.rados.rados.0.smithi038.stdout:792: finishing write tid 4 to smithi03829856-17 2024-03-19T11:30:24.201 INFO:tasks.rados.rados.0.smithi038.stdout:792: finishing write tid 5 to smithi03829856-17 2024-03-19T11:30:24.201 INFO:tasks.rados.rados.0.smithi038.stdout:792: oid 17 updating version 0 to 60 2024-03-19T11:30:24.201 INFO:tasks.rados.rados.0.smithi038.stdout:792: oid 17 updating version 60 to 62 2024-03-19T11:30:24.202 INFO:tasks.rados.rados.0.smithi038.stdout:792: oid 17 updating version 62 to 63 2024-03-19T11:30:24.202 INFO:tasks.rados.rados.0.smithi038.stdout:792: oid 17 version 63 is already newer than 61 2024-03-19T11:30:24.202 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 17 v 63 (ObjNum 96 snap 10 seq_num 96) dirty exists 2024-03-19T11:30:24.202 INFO:tasks.rados.rados.0.smithi038.stdout:792: left oid 17 (ObjNum 96 snap 10 seq_num 96) 2024-03-19T11:30:24.202 INFO:tasks.rados.rados.0.smithi038.stdout:794: finishing rollback tid 0 to smithi03829856-13 2024-03-19T11:30:24.202 INFO:tasks.rados.rados.0.smithi038.stdout:794: finishing rollback tid 1 to smithi03829856-13 2024-03-19T11:30:24.202 INFO:tasks.rados.rados.0.smithi038.stdout:794: finishing rollback tid 2 to smithi03829856-13 2024-03-19T11:30:24.202 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 13 v 62 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-03-19T11:30:24.202 INFO:tasks.rados.rados.0.smithi038.stdout:795: finishing write tid 1 to smithi03829856-36 2024-03-19T11:30:24.202 INFO:tasks.rados.rados.0.smithi038.stdout:795: finishing write tid 2 to smithi03829856-36 2024-03-19T11:30:24.202 INFO:tasks.rados.rados.0.smithi038.stdout:797: finishing rollback tid 0 to smithi03829856-12 2024-03-19T11:30:24.202 INFO:tasks.rados.rados.0.smithi038.stdout:789: done (15 left) 2024-03-19T11:30:24.203 INFO:tasks.rados.rados.0.smithi038.stdout:790: done (14 left) 2024-03-19T11:30:24.203 INFO:tasks.rados.rados.0.smithi038.stdout:791: done (13 left) 2024-03-19T11:30:24.203 INFO:tasks.rados.rados.0.smithi038.stdout:792: done (12 left) 2024-03-19T11:30:24.203 INFO:tasks.rados.rados.0.smithi038.stdout:794: done (11 left) 2024-03-19T11:30:24.203 INFO:tasks.rados.rados.0.smithi038.stdout:806: set_chunk oid 26 target oid 2024-03-19T11:30:24.203 INFO:tasks.rados.rados.0.smithi038.stdout:806: set_chunk oid 26 offset: 0 length: 4000000 target oid smithi03829856- offset: 0 snap -1 2024-03-19T11:30:24.203 INFO:tasks.rados.rados.0.smithi038.stdout:807: set_chunk oid 49 target oid 2024-03-19T11:30:24.209 INFO:tasks.rados.rados.0.smithi038.stdout:807: set_chunk oid 49 offset: 481792 length: 760832 target oid 72d1e68e705f825ded69bf3d79082882be5a047a30701fd94c1568ddec35d849 offset: 0 snap 4 2024-03-19T11:30:24.210 INFO:tasks.rados.rados.0.smithi038.stdout:808: write oid 45 current snap is 10 2024-03-19T11:30:24.210 INFO:tasks.rados.rados.0.smithi038.stdout:797: finishing rollback tid 1 to smithi03829856-12 2024-03-19T11:30:24.210 INFO:tasks.rados.rados.0.smithi038.stdout:795: finishing write tid 3 to smithi03829856-36 2024-03-19T11:30:24.210 INFO:tasks.rados.rados.0.smithi038.stdout:797: finishing rollback tid 2 to smithi03829856-12 2024-03-19T11:30:24.210 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 12 v 40 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-03-19T11:30:24.210 INFO:tasks.rados.rados.0.smithi038.stdout:795: finishing write tid 4 to smithi03829856-36 2024-03-19T11:30:24.210 INFO:tasks.rados.rados.0.smithi038.stdout:808: seq_num 99 ranges {463893=562240,1593482=658027,2685935=559341} 2024-03-19T11:30:24.220 INFO:tasks.rados.rados.0.smithi038.stdout:808: writing smithi03829856-45 from 463893 to 1026133 tid 1 2024-03-19T11:30:24.233 INFO:tasks.rados.rados.0.smithi038.stdout:808: writing smithi03829856-45 from 1593482 to 2251509 tid 2 2024-03-19T11:30:24.243 INFO:tasks.rados.rados.0.smithi038.stdout:808: writing smithi03829856-45 from 2685935 to 3245276 tid 3 2024-03-19T11:30:24.243 INFO:tasks.rados.rados.0.smithi038.stdout:797: done (13 left) 2024-03-19T11:30:24.243 INFO:tasks.rados.rados.0.smithi038.stdout:809: write (excl) oid 18 current snap is 10 2024-03-19T11:30:24.243 INFO:tasks.rados.rados.0.smithi038.stdout:809: seq_num 100 ranges {480700=637489,1651964=401685,2713358=770613} 2024-03-19T11:30:24.255 INFO:tasks.rados.rados.0.smithi038.stdout:809: writing smithi03829856-18 from 480700 to 1118189 tid 1 2024-03-19T11:30:24.264 INFO:tasks.rados.rados.0.smithi038.stdout:809: writing smithi03829856-18 from 1651964 to 2053649 tid 2 2024-03-19T11:30:24.275 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:24.287+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:24.277 INFO:tasks.rados.rados.0.smithi038.stdout:809: writing smithi03829856-18 from 2713358 to 3483971 tid 3 2024-03-19T11:30:24.277 INFO:tasks.rados.rados.0.smithi038.stdout:810: snap_remove snap 0 2024-03-19T11:30:24.345 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:24.345 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:24.358 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-19T11:30:24.376 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:24.387+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:24.476 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:24.477 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:24.477 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:24.486+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:24.491 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-19T11:30:24.578 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:24.591+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:24.655 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:24.655 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:24.669 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-19T11:30:24.715 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:24.727+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:24.780 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:24.790+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:24.805 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:24.806 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:24.821 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-19T11:30:24.848 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-19T11:30:24.848 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-19T11:30:24.848 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-03-19T11:30:24.848 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 110 3.0 2024-03-19T11:30:24.879 INFO:tasks.rados.rados.0.smithi038.stdout:810: done (14 left) 2024-03-19T11:30:24.879 INFO:tasks.rados.rados.0.smithi038.stdout:811: read oid 4 snap -1 2024-03-19T11:30:24.879 INFO:tasks.rados.rados.0.smithi038.stdout:811: expect (ObjNum 3 snap 0 seq_num 3) 2024-03-19T11:30:24.879 INFO:tasks.rados.rados.0.smithi038.stdout:812: write (excl) oid 3 current snap is 10 2024-03-19T11:30:24.879 INFO:tasks.rados.rados.0.smithi038.stdout:795: finishing write tid 5 to smithi03829856-36 2024-03-19T11:30:24.879 INFO:tasks.rados.rados.0.smithi038.stdout:795: oid 36 updating version 0 to 36 2024-03-19T11:30:24.879 INFO:tasks.rados.rados.0.smithi038.stdout:795: oid 36 updating version 36 to 38 2024-03-19T11:30:24.879 INFO:tasks.rados.rados.0.smithi038.stdout:795: oid 36 version 38 is already newer than 35 2024-03-19T11:30:24.879 INFO:tasks.rados.rados.0.smithi038.stdout:795: oid 36 version 38 is already newer than 37 2024-03-19T11:30:24.879 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 36 v 38 (ObjNum 97 snap 10 seq_num 97) dirty exists 2024-03-19T11:30:24.880 INFO:tasks.rados.rados.0.smithi038.stdout:795: left oid 36 (ObjNum 97 snap 10 seq_num 97) 2024-03-19T11:30:24.880 INFO:tasks.rados.rados.0.smithi038.stdout:796: expect (ObjNum 65 snap 3 seq_num 65) 2024-03-19T11:30:24.881 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:24.890+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:24.887 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:24.898+0000 7f143b8b4640 -1 osd.5 46 handle_osd_map injecting map crc failure 2024-03-19T11:30:24.982 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:24.994+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:24.994 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:24.994 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:25.014 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-19T11:30:25.067 INFO:tasks.rados.rados.0.smithi038.stdout:798: finishing set_chunk to oid 20 2024-03-19T11:30:25.067 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 20 set_chunk returned error code -95 offset: 1601024 length: 358912 2024-03-19T11:30:25.067 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 20 v 0 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-03-19T11:30:25.067 INFO:tasks.rados.rados.0.smithi038.stdout:799: finishing set_chunk to oid 6 2024-03-19T11:30:25.067 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 6 v 31 (ObjNum 53 snap 0 seq_num 53) dirty exists 2024-03-19T11:30:25.067 INFO:tasks.rados.rados.0.smithi038.stdout:800: finishing write tid 1 to smithi03829856-42 2024-03-19T11:30:25.068 INFO:tasks.rados.rados.0.smithi038.stdout:801: got (22) Invalid argument 2024-03-19T11:30:25.068 INFO:tasks.rados.rados.0.smithi038.stdout:800: finishing write tid 2 to smithi03829856-42 2024-03-19T11:30:25.068 INFO:tasks.rados.rados.0.smithi038.stdout:800: finishing write tid 3 to smithi03829856-42 2024-03-19T11:30:25.068 INFO:tasks.rados.rados.0.smithi038.stdout:800: finishing write tid 4 to smithi03829856-42 2024-03-19T11:30:25.068 INFO:tasks.rados.rados.0.smithi038.stdout:800: finishing write tid 5 to smithi03829856-42 2024-03-19T11:30:25.068 INFO:tasks.rados.rados.0.smithi038.stdout:800: oid 42 updating version 0 to 41 2024-03-19T11:30:25.068 INFO:tasks.rados.rados.0.smithi038.stdout:800: oid 42 updating version 41 to 43 2024-03-19T11:30:25.068 INFO:tasks.rados.rados.0.smithi038.stdout:800: oid 42 updating version 43 to 44 2024-03-19T11:30:25.068 INFO:tasks.rados.rados.0.smithi038.stdout:800: oid 42 version 44 is already newer than 42 2024-03-19T11:30:25.068 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 42 v 44 (ObjNum 98 snap 10 seq_num 98) dirty exists 2024-03-19T11:30:25.068 INFO:tasks.rados.rados.0.smithi038.stdout:800: left oid 42 (ObjNum 98 snap 10 seq_num 98) 2024-03-19T11:30:25.068 INFO:tasks.rados.rados.0.smithi038.stdout:802: finishing set_chunk to oid 41 2024-03-19T11:30:25.068 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 41 set_chunk returned error code -95 offset: 68096 length: 363008 2024-03-19T11:30:25.068 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 41 v 0 (ObjNum 80 snap 6 seq_num 80) dirty exists 2024-03-19T11:30:25.068 INFO:tasks.rados.rados.0.smithi038.stdout:812: seq_num 101 ranges {497507=712738,1710446=661696,3140781=556726} 2024-03-19T11:30:25.073 INFO:tasks.rados.rados.0.smithi038.stdout:812: writing smithi03829856-3 from 497507 to 1210245 tid 1 2024-03-19T11:30:25.080 INFO:tasks.rados.rados.0.smithi038.stdout:812: writing smithi03829856-3 from 1710446 to 2372142 tid 2 2024-03-19T11:30:25.083 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:25.091+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:25.088 INFO:tasks.rados.rados.0.smithi038.stdout:812: writing smithi03829856-3 from 3140781 to 3697507 tid 3 2024-03-19T11:30:25.088 INFO:tasks.rados.rados.0.smithi038.stdout:795: done (15 left) 2024-03-19T11:30:25.088 INFO:tasks.rados.rados.0.smithi038.stdout:796: done (14 left) 2024-03-19T11:30:25.088 INFO:tasks.rados.rados.0.smithi038.stdout:798: done (13 left) 2024-03-19T11:30:25.088 INFO:tasks.rados.rados.0.smithi038.stdout:799: done (12 left) 2024-03-19T11:30:25.088 INFO:tasks.rados.rados.0.smithi038.stdout:800: done (11 left) 2024-03-19T11:30:25.088 INFO:tasks.rados.rados.0.smithi038.stdout:801: done (10 left) 2024-03-19T11:30:25.088 INFO:tasks.rados.rados.0.smithi038.stdout:802: done (9 left) 2024-03-19T11:30:25.089 INFO:tasks.rados.rados.0.smithi038.stdout:813: delete oid 11 current snap is 10 2024-03-19T11:30:25.089 INFO:tasks.rados.rados.0.smithi038.stdout:805: got (0) Success 2024-03-19T11:30:25.089 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 14 v 24 (ObjNum 88 snap 8 seq_num 88) dirty exists 2024-03-19T11:30:25.089 INFO:tasks.rados.rados.0.smithi038.stdout:803: expect (ObjNum 39 snap 0 seq_num 39) 2024-03-19T11:30:25.135 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:25.135 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:25.150 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-19T11:30:25.173 INFO:tasks.rados.rados.0.smithi038.stdout:804: finishing set_chunk to oid 43 2024-03-19T11:30:25.173 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 43 v 61 (ObjNum 91 snap 10 seq_num 91) dirty exists 2024-03-19T11:30:25.173 INFO:tasks.rados.rados.0.smithi038.stdout:806: finishing set_chunk to oid 26 2024-03-19T11:30:25.173 INFO:tasks.rados.rados.0.smithi038.stdout:806: got expected ERANGE (src dne) 2024-03-19T11:30:25.173 INFO:tasks.rados.rados.0.smithi038.stdout:807: finishing set_chunk to oid 49 2024-03-19T11:30:25.174 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 49 set_chunk returned error code -95 offset: 481792 length: 760832 2024-03-19T11:30:25.174 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-03-19T11:30:25.174 INFO:tasks.rados.rados.0.smithi038.stdout:808: finishing write tid 1 to smithi03829856-45 2024-03-19T11:30:25.174 INFO:tasks.rados.rados.0.smithi038.stdout:808: finishing write tid 2 to smithi03829856-45 2024-03-19T11:30:25.174 INFO:tasks.rados.rados.0.smithi038.stdout:808: finishing write tid 3 to smithi03829856-45 2024-03-19T11:30:25.174 INFO:tasks.rados.rados.0.smithi038.stdout:808: finishing write tid 4 to smithi03829856-45 2024-03-19T11:30:25.174 INFO:tasks.rados.rados.0.smithi038.stdout:808: finishing write tid 5 to smithi03829856-45 2024-03-19T11:30:25.174 INFO:tasks.rados.rados.0.smithi038.stdout:808: oid 45 updating version 0 to 42 2024-03-19T11:30:25.174 INFO:tasks.rados.rados.0.smithi038.stdout:808: oid 45 version 42 is already newer than 40 2024-03-19T11:30:25.174 INFO:tasks.rados.rados.0.smithi038.stdout:808: oid 45 version 42 is already newer than 39 2024-03-19T11:30:25.174 INFO:tasks.rados.rados.0.smithi038.stdout:808: oid 45 version 42 is already newer than 41 2024-03-19T11:30:25.174 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 45 v 42 (ObjNum 99 snap 10 seq_num 99) dirty exists 2024-03-19T11:30:25.174 INFO:tasks.rados.rados.0.smithi038.stdout:808: left oid 45 (ObjNum 99 snap 10 seq_num 99) 2024-03-19T11:30:25.175 INFO:tasks.rados.rados.0.smithi038.stdout:809: finishing write tid 1 to smithi03829856-18 2024-03-19T11:30:25.175 INFO:tasks.rados.rados.0.smithi038.stdout:809: finishing write tid 2 to smithi03829856-18 2024-03-19T11:30:25.175 INFO:tasks.rados.rados.0.smithi038.stdout:803: done (9 left) 2024-03-19T11:30:25.175 INFO:tasks.rados.rados.0.smithi038.stdout:804: done (8 left) 2024-03-19T11:30:25.175 INFO:tasks.rados.rados.0.smithi038.stdout:805: done (7 left) 2024-03-19T11:30:25.175 INFO:tasks.rados.rados.0.smithi038.stdout:806: done (6 left) 2024-03-19T11:30:25.175 INFO:tasks.rados.rados.0.smithi038.stdout:807: done (5 left) 2024-03-19T11:30:25.175 INFO:tasks.rados.rados.0.smithi038.stdout:808: done (4 left) 2024-03-19T11:30:25.175 INFO:tasks.rados.rados.0.smithi038.stdout:813: done (3 left) 2024-03-19T11:30:25.175 INFO:tasks.rados.rados.0.smithi038.stdout:814: write oid 21 current snap is 10 2024-03-19T11:30:25.175 INFO:tasks.rados.rados.0.smithi038.stdout:809: finishing write tid 3 to smithi03829856-18 2024-03-19T11:30:25.175 INFO:tasks.rados.rados.0.smithi038.stdout:814: seq_num 103 ranges {531121=463236,1711057=665365,3079274=651847} 2024-03-19T11:30:25.178 INFO:tasks.rados.rados.0.smithi038.stdout:814: writing smithi03829856-21 from 531121 to 994357 tid 1 2024-03-19T11:30:25.184 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:25.195+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:25.186 INFO:tasks.rados.rados.0.smithi038.stdout:814: writing smithi03829856-21 from 1711057 to 2376422 tid 2 2024-03-19T11:30:25.195 INFO:tasks.rados.rados.0.smithi038.stdout:814: writing smithi03829856-21 from 3079274 to 3731121 tid 3 2024-03-19T11:30:25.195 INFO:tasks.rados.rados.0.smithi038.stdout:815: read oid 44 snap -1 2024-03-19T11:30:25.195 INFO:tasks.rados.rados.0.smithi038.stdout:815: expect (ObjNum 43 snap 0 seq_num 43) 2024-03-19T11:30:25.195 INFO:tasks.rados.rados.0.smithi038.stdout:816: read oid 31 snap -1 2024-03-19T11:30:25.195 INFO:tasks.rados.rados.0.smithi038.stdout:816: expect (ObjNum 89 snap 10 seq_num 89) 2024-03-19T11:30:25.195 INFO:tasks.rados.rados.0.smithi038.stdout:817: write (excl) oid 10 current snap is 10 2024-03-19T11:30:25.195 INFO:tasks.rados.rados.0.smithi038.stdout:809: finishing write tid 4 to smithi03829856-18 2024-03-19T11:30:25.195 INFO:tasks.rados.rados.0.smithi038.stdout:809: finishing write tid 5 to smithi03829856-18 2024-03-19T11:30:25.195 INFO:tasks.rados.rados.0.smithi038.stdout:809: oid 18 updating version 0 to 60 2024-03-19T11:30:25.195 INFO:tasks.rados.rados.0.smithi038.stdout:809: oid 18 updating version 60 to 63 2024-03-19T11:30:25.195 INFO:tasks.rados.rados.0.smithi038.stdout:809: oid 18 version 63 is already newer than 61 2024-03-19T11:30:25.195 INFO:tasks.rados.rados.0.smithi038.stdout:809: oid 18 version 63 is already newer than 62 2024-03-19T11:30:25.195 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 18 v 63 (ObjNum 100 snap 10 seq_num 100) dirty exists 2024-03-19T11:30:25.195 INFO:tasks.rados.rados.0.smithi038.stdout:809: left oid 18 (ObjNum 100 snap 10 seq_num 100) 2024-03-19T11:30:25.196 INFO:tasks.rados.rados.0.smithi038.stdout:811: expect (ObjNum 3 snap 0 seq_num 3) 2024-03-19T11:30:25.259 INFO:tasks.rados.rados.0.smithi038.stdout:812: finishing write tid 1 to smithi03829856-3 2024-03-19T11:30:25.259 INFO:tasks.rados.rados.0.smithi038.stdout:812: finishing write tid 2 to smithi03829856-3 2024-03-19T11:30:25.259 INFO:tasks.rados.rados.0.smithi038.stdout:817: seq_num 104 ranges {547928=538485,1769539=409023,2706697=532054} 2024-03-19T11:30:25.265 INFO:tasks.rados.rados.0.smithi038.stdout:817: writing smithi03829856-10 from 547928 to 1086413 tid 1 2024-03-19T11:30:25.272 INFO:tasks.rados.rados.0.smithi038.stdout:817: writing smithi03829856-10 from 1769539 to 2178562 tid 2 2024-03-19T11:30:25.280 INFO:tasks.rados.rados.0.smithi038.stdout:817: writing smithi03829856-10 from 2706697 to 3238751 tid 3 2024-03-19T11:30:25.280 INFO:tasks.rados.rados.0.smithi038.stdout:809: done (6 left) 2024-03-19T11:30:25.280 INFO:tasks.rados.rados.0.smithi038.stdout:811: done (5 left) 2024-03-19T11:30:25.280 INFO:tasks.rados.rados.0.smithi038.stdout:818: rollback oid 45 current snap is 10 2024-03-19T11:30:25.281 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 45 to 3 2024-03-19T11:30:25.281 INFO:tasks.rados.rados.0.smithi038.stdout:812: finishing write tid 3 to smithi03829856-3 2024-03-19T11:30:25.282 INFO:tasks.rados.rados.0.smithi038.stdout:812: finishing write tid 4 to smithi03829856-3 2024-03-19T11:30:25.282 INFO:tasks.rados.rados.0.smithi038.stdout:812: finishing write tid 5 to smithi03829856-3 2024-03-19T11:30:25.282 INFO:tasks.rados.rados.0.smithi038.stdout:812: oid 3 updating version 0 to 40 2024-03-19T11:30:25.282 INFO:tasks.rados.rados.0.smithi038.stdout:812: oid 3 updating version 40 to 41 2024-03-19T11:30:25.282 INFO:tasks.rados.rados.0.smithi038.stdout:812: oid 3 updating version 41 to 43 2024-03-19T11:30:25.282 INFO:tasks.rados.rados.0.smithi038.stdout:812: oid 3 version 43 is already newer than 42 2024-03-19T11:30:25.282 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 3 v 43 (ObjNum 101 snap 10 seq_num 101) dirty exists 2024-03-19T11:30:25.282 INFO:tasks.rados.rados.0.smithi038.stdout:812: left oid 3 (ObjNum 101 snap 10 seq_num 101) 2024-03-19T11:30:25.282 INFO:tasks.rados.rados.0.smithi038.stdout:814: finishing write tid 1 to smithi03829856-21 2024-03-19T11:30:25.282 INFO:tasks.rados.rados.0.smithi038.stdout:814: finishing write tid 2 to smithi03829856-21 2024-03-19T11:30:25.282 INFO:tasks.rados.rados.0.smithi038.stdout:812: done (5 left) 2024-03-19T11:30:25.282 INFO:tasks.rados.rados.0.smithi038.stdout:819: set_chunk oid 5 target oid 2024-03-19T11:30:25.284 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:25.295+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:25.299 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:25.299 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:25.310 INFO:tasks.rados.rados.0.smithi038.stdout:819: set_chunk oid 5 offset: 159744 length: 1709568 target oid 00aac5d9fa5c6aae79f46e03315fe624f95a1c3702d6abad9c9db885f6b5f4de offset: 0 snap -1 2024-03-19T11:30:25.310 INFO:tasks.rados.rados.0.smithi038.stdout:820: tier_evict oid 38 2024-03-19T11:30:25.310 INFO:tasks.rados.rados.0.smithi038.stdout:820: tier_evict oid 38 snap -1 2024-03-19T11:30:25.310 INFO:tasks.rados.rados.0.smithi038.stdout:815: expect (ObjNum 43 snap 0 seq_num 43) 2024-03-19T11:30:25.315 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-19T11:30:25.385 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:25.394+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:25.395 INFO:tasks.rados.rados.0.smithi038.stdout:814: finishing write tid 3 to smithi03829856-21 2024-03-19T11:30:25.395 INFO:tasks.rados.rados.0.smithi038.stdout:814: finishing write tid 4 to smithi03829856-21 2024-03-19T11:30:25.395 INFO:tasks.rados.rados.0.smithi038.stdout:814: finishing write tid 5 to smithi03829856-21 2024-03-19T11:30:25.396 INFO:tasks.rados.rados.0.smithi038.stdout:814: oid 21 updating version 0 to 70 2024-03-19T11:30:25.396 INFO:tasks.rados.rados.0.smithi038.stdout:814: oid 21 version 70 is already newer than 68 2024-03-19T11:30:25.396 INFO:tasks.rados.rados.0.smithi038.stdout:814: oid 21 version 70 is already newer than 67 2024-03-19T11:30:25.396 INFO:tasks.rados.rados.0.smithi038.stdout:814: oid 21 version 70 is already newer than 69 2024-03-19T11:30:25.396 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 21 v 70 (ObjNum 103 snap 10 seq_num 103) dirty exists 2024-03-19T11:30:25.396 INFO:tasks.rados.rados.0.smithi038.stdout:814: left oid 21 (ObjNum 103 snap 10 seq_num 103) 2024-03-19T11:30:25.396 INFO:tasks.rados.rados.0.smithi038.stdout:816: expect (ObjNum 89 snap 10 seq_num 89) 2024-03-19T11:30:25.457 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-19T11:30:25.484 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:25.484 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:25.485 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:25.498+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:25.499 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-19T11:30:25.586 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:25.598+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:25.594 INFO:tasks.rados.rados.0.smithi038.stdout:817: finishing write tid 1 to smithi03829856-10 2024-03-19T11:30:25.594 INFO:tasks.rados.rados.0.smithi038.stdout:817: finishing write tid 2 to smithi03829856-10 2024-03-19T11:30:25.594 INFO:tasks.rados.rados.0.smithi038.stdout:817: finishing write tid 3 to smithi03829856-10 2024-03-19T11:30:25.595 INFO:tasks.rados.rados.0.smithi038.stdout:814: done (6 left) 2024-03-19T11:30:25.595 INFO:tasks.rados.rados.0.smithi038.stdout:815: done (5 left) 2024-03-19T11:30:25.595 INFO:tasks.rados.rados.0.smithi038.stdout:816: done (4 left) 2024-03-19T11:30:25.595 INFO:tasks.rados.rados.0.smithi038.stdout:821: set_chunk oid 36 target oid 2024-03-19T11:30:25.607 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:30:25.608 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:30:25.613 INFO:tasks.rados.rados.0.smithi038.stdout:821: set_chunk oid 36 offset: 534016 length: 1646592 target oid e0bae094c40b8fb2f998060ab50b4f50f566dd5a2496a9ab8bd7a4cb8a13f36c offset: 0 snap 8 2024-03-19T11:30:25.613 INFO:tasks.rados.rados.0.smithi038.stdout:822: read oid 33 snap -1 2024-03-19T11:30:25.613 INFO:tasks.rados.rados.0.smithi038.stdout:822: expect (ObjNum 32 snap 0 seq_num 32) 2024-03-19T11:30:25.613 INFO:tasks.rados.rados.0.smithi038.stdout:823: set_chunk oid 49 target oid 2024-03-19T11:30:25.626 INFO:tasks.rados.rados.0.smithi038.stdout:823: set_chunk oid 49 offset: 177664 length: 467968 target oid 0f3e06be76b807f8fa75b015df0fa98b6275933071d127c16cd8e14b9ab19288 offset: 0 snap -1 2024-03-19T11:30:25.627 INFO:tasks.rados.rados.0.smithi038.stdout:824: read oid 15 snap 3 2024-03-19T11:30:25.627 INFO:tasks.rados.rados.0.smithi038.stdout:824: expect (ObjNum 14 snap 0 seq_num 14) 2024-03-19T11:30:25.627 INFO:tasks.rados.rados.0.smithi038.stdout:825: snap_create 2024-03-19T11:30:25.628 INFO:tasks.rados.rados.0.smithi038.stdout:817: finishing write tid 4 to smithi03829856-10 2024-03-19T11:30:25.628 INFO:tasks.rados.rados.0.smithi038.stdout:817: finishing write tid 5 to smithi03829856-10 2024-03-19T11:30:25.628 INFO:tasks.rados.rados.0.smithi038.stdout:817: oid 10 updating version 0 to 50 2024-03-19T11:30:25.628 INFO:tasks.rados.rados.0.smithi038.stdout:817: oid 10 updating version 50 to 52 2024-03-19T11:30:25.628 INFO:tasks.rados.rados.0.smithi038.stdout:817: oid 10 version 52 is already newer than 51 2024-03-19T11:30:25.628 INFO:tasks.rados.rados.0.smithi038.stdout:817: oid 10 version 52 is already newer than 49 2024-03-19T11:30:25.628 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 10 v 52 (ObjNum 104 snap 10 seq_num 104) dirty exists 2024-03-19T11:30:25.628 INFO:tasks.rados.rados.0.smithi038.stdout:817: left oid 10 (ObjNum 104 snap 10 seq_num 104) 2024-03-19T11:30:25.628 INFO:tasks.rados.rados.0.smithi038.stdout:818: finishing rollback tid 0 to smithi03829856-45 2024-03-19T11:30:25.628 INFO:tasks.rados.rados.0.smithi038.stdout:818: finishing rollback tid 1 to smithi03829856-45 2024-03-19T11:30:25.628 INFO:tasks.rados.rados.0.smithi038.stdout:818: finishing rollback tid 2 to smithi03829856-45 2024-03-19T11:30:25.629 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 45 v 47 (ObjNum 59 snap 3 seq_num 59) dirty exists 2024-03-19T11:30:25.629 INFO:tasks.rados.rados.0.smithi038.stdout:820: got (22) Invalid argument 2024-03-19T11:30:25.629 INFO:tasks.rados.rados.0.smithi038.stdout:819: finishing set_chunk to oid 5 2024-03-19T11:30:25.629 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 5 set_chunk returned error code -95 offset: 159744 length: 1709568 2024-03-19T11:30:25.629 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-03-19T11:30:25.629 INFO:tasks.rados.rados.0.smithi038.stdout:821: finishing set_chunk to oid 36 2024-03-19T11:30:25.629 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 36 set_chunk returned error code -95 offset: 534016 length: 1646592 2024-03-19T11:30:25.629 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 36 v 0 (ObjNum 97 snap 10 seq_num 97) dirty exists 2024-03-19T11:30:25.629 INFO:tasks.rados.rados.0.smithi038.stdout:822: expect (ObjNum 32 snap 0 seq_num 32) 2024-03-19T11:30:25.632 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:30:25.633 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:30:25.661 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:30:25.661 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:30:25.668 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:25.669 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:30:25.669 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:30:25.669 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:30:25.669 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.669 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:6 4.6 e27/21 4:626acd4d:::smithi03829856-2:head v 27'2, mlcod=0'0)", 2024-03-19T11:30:25.669 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.696903+0000", 2024-03-19T11:30:25.669 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 20.983611745000001, 2024-03-19T11:30:25.669 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0071029129999999998, 2024-03-19T11:30:25.669 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:25.669 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:25.669 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:25.669 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.669 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:25.669 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696903+0000", 2024-03-19T11:30:25.670 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:25.670 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.670 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.670 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:25.670 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696903+0000", 2024-03-19T11:30:25.670 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:25.670 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.670 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.671 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:25.671 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.696904+0000", 2024-03-19T11:30:25.671 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4270000000000001e-06 2024-03-19T11:30:25.671 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.671 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.671 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:25.671 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.697064+0000", 2024-03-19T11:30:25.671 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015956100000000001 2024-03-19T11:30:25.671 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.671 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.671 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:25.671 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.697082+0000", 2024-03-19T11:30:25.671 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8868999999999998e-05 2024-03-19T11:30:25.671 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.671 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.672 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:25.672 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.697096+0000", 2024-03-19T11:30:25.672 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3155e-05 2024-03-19T11:30:25.672 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.672 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.672 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:25.672 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.697169+0000", 2024-03-19T11:30:25.672 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3226999999999995e-05 2024-03-19T11:30:25.672 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.672 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.672 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:25.672 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.697193+0000", 2024-03-19T11:30:25.672 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4153000000000001e-05 2024-03-19T11:30:25.672 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.672 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.673 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:25.673 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.703936+0000", 2024-03-19T11:30:25.673 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0067430570000000002 2024-03-19T11:30:25.673 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.673 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.673 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:25.673 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.704006+0000", 2024-03-19T11:30:25.673 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9463999999999998e-05 2024-03-19T11:30:25.673 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:25.673 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:25.673 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:25.673 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.673 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.673 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:13 4.4 4:21eac9b2:::smithi03829856-4:head [write 450421~625747,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:30:25.674 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.720170+0000", 2024-03-19T11:30:25.674 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 20.960344430999999, 2024-03-19T11:30:25.674 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0090522840000000007, 2024-03-19T11:30:25.674 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:25.674 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:25.674 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:25.674 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:25.674 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:25.674 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 13 2024-03-19T11:30:25.674 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.674 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:25.674 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.674 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:25.674 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.720170+0000", 2024-03-19T11:30:25.674 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:25.675 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.675 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.675 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:25.675 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.720170+0000", 2024-03-19T11:30:25.675 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:25.675 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.675 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.675 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:25.675 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.720172+0000", 2024-03-19T11:30:25.675 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9709999999999998e-06 2024-03-19T11:30:25.675 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.675 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.675 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:25.675 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.720601+0000", 2024-03-19T11:30:25.675 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00042897999999999998 2024-03-19T11:30:25.676 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.676 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.676 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:25.676 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.720615+0000", 2024-03-19T11:30:25.676 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4617000000000001e-05 2024-03-19T11:30:25.676 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.676 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.676 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:25.676 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.720627+0000", 2024-03-19T11:30:25.676 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1513e-05 2024-03-19T11:30:25.676 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.676 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.676 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:25.676 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.720705+0000", 2024-03-19T11:30:25.676 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7880000000000007e-05 2024-03-19T11:30:25.677 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.677 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.677 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-03-19T11:30:25.677 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.720709+0000", 2024-03-19T11:30:25.677 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5549999999999996e-06 2024-03-19T11:30:25.677 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.677 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.677 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:25.677 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.722748+0000", 2024-03-19T11:30:25.677 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0020388630000000001 2024-03-19T11:30:25.677 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.677 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.677 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:25.677 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.722891+0000", 2024-03-19T11:30:25.678 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014286 2024-03-19T11:30:25.678 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.678 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.678 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 2", 2024-03-19T11:30:25.678 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723057+0000", 2024-03-19T11:30:25.678 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000166286 2024-03-19T11:30:25.678 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.678 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.678 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:25.678 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.725297+0000", 2024-03-19T11:30:25.678 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022397770000000001 2024-03-19T11:30:25.678 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.678 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.678 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:25.678 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.729110+0000", 2024-03-19T11:30:25.679 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038130109999999998 2024-03-19T11:30:25.679 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.679 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.679 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:25.679 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.729169+0000", 2024-03-19T11:30:25.679 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8616000000000003e-05 2024-03-19T11:30:25.679 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.679 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.679 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:25.679 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.729222+0000", 2024-03-19T11:30:25.679 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3355e-05 2024-03-19T11:30:25.679 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:25.679 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:25.679 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:25.679 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.680 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.680 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:844 4.7 4:e41d47fb:::smithi03829856-14:head [write 574806~778913] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:25.680 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:08.749074+0000", 2024-03-19T11:30:25.680 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 16.931440649999999, 2024-03-19T11:30:25.680 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013996124, 2024-03-19T11:30:25.680 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:25.680 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:25.680 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:25.680 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:25.680 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:25.681 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 844 2024-03-19T11:30:25.681 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.681 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:25.681 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.681 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:25.681 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.749074+0000", 2024-03-19T11:30:25.681 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:25.681 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.681 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.681 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:25.681 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.749074+0000", 2024-03-19T11:30:25.681 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:25.681 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.681 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.682 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:25.682 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.749076+0000", 2024-03-19T11:30:25.682 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.588e-06 2024-03-19T11:30:25.682 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.682 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.682 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:25.682 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.749714+0000", 2024-03-19T11:30:25.682 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00063805999999999997 2024-03-19T11:30:25.682 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.682 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.682 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:25.682 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.749750+0000", 2024-03-19T11:30:25.682 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5709e-05 2024-03-19T11:30:25.682 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.683 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.683 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:25.683 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.749767+0000", 2024-03-19T11:30:25.683 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7127000000000001e-05 2024-03-19T11:30:25.683 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.683 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.683 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:25.683 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.749860+0000", 2024-03-19T11:30:25.683 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2875999999999996e-05 2024-03-19T11:30:25.683 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.683 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.683 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:25.683 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.749943+0000", 2024-03-19T11:30:25.683 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2915000000000002e-05 2024-03-19T11:30:25.683 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.684 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.684 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:25.684 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.750083+0000", 2024-03-19T11:30:25.684 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013953300000000001 2024-03-19T11:30:25.684 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.684 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.684 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:25.684 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.753673+0000", 2024-03-19T11:30:25.684 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0035904470000000001 2024-03-19T11:30:25.684 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.684 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.684 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:25.684 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.762965+0000", 2024-03-19T11:30:25.684 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0092923390000000002 2024-03-19T11:30:25.684 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.684 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.685 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:25.685 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.763001+0000", 2024-03-19T11:30:25.685 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6087000000000002e-05 2024-03-19T11:30:25.685 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.685 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.685 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:25.685 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.763070+0000", 2024-03-19T11:30:25.685 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8442999999999995e-05 2024-03-19T11:30:25.685 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:25.685 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:25.685 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:25.685 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.685 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.685 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:845 4.7 4:e41d47fb:::smithi03829856-14:head [write 2061132~557342] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:25.686 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:08.756359+0000", 2024-03-19T11:30:25.686 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 16.924155691999999, 2024-03-19T11:30:25.686 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074752910000000002, 2024-03-19T11:30:25.686 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:25.686 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:25.686 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:25.686 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:25.686 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:25.686 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 845 2024-03-19T11:30:25.686 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.686 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:25.686 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.686 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:25.686 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756359+0000", 2024-03-19T11:30:25.686 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:25.687 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.687 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.687 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:25.687 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756359+0000", 2024-03-19T11:30:25.687 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:25.687 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.687 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.687 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:25.687 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756361+0000", 2024-03-19T11:30:25.687 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5280000000000002e-06 2024-03-19T11:30:25.687 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.687 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.687 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:25.687 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756535+0000", 2024-03-19T11:30:25.688 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000173697 2024-03-19T11:30:25.688 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.688 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.688 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:25.688 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756552+0000", 2024-03-19T11:30:25.688 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7397e-05 2024-03-19T11:30:25.688 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.688 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.688 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:25.688 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756561+0000", 2024-03-19T11:30:25.688 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8769999999999999e-06 2024-03-19T11:30:25.688 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.688 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.688 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:25.688 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756614+0000", 2024-03-19T11:30:25.689 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2425e-05 2024-03-19T11:30:25.689 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.689 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.689 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:25.689 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756693+0000", 2024-03-19T11:30:25.689 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9338999999999994e-05 2024-03-19T11:30:25.689 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.689 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.689 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:25.689 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756801+0000", 2024-03-19T11:30:25.689 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000108175 2024-03-19T11:30:25.689 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.689 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.689 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:25.689 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.758827+0000", 2024-03-19T11:30:25.690 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0020263379999999999 2024-03-19T11:30:25.690 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.690 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.690 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:25.690 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.763778+0000", 2024-03-19T11:30:25.690 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0049505740000000001 2024-03-19T11:30:25.690 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.690 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.690 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:25.690 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.763807+0000", 2024-03-19T11:30:25.690 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8702999999999999e-05 2024-03-19T11:30:25.690 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.690 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.690 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:25.690 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.763834+0000", 2024-03-19T11:30:25.691 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.7237999999999999e-05 2024-03-19T11:30:25.691 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:25.691 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:25.691 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:25.691 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.691 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.691 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:846 4.7 4:e41d47fb:::smithi03829856-14:head [setxattr _header (56) in=7b,truncate 2974806] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:25.691 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:08.756597+0000", 2024-03-19T11:30:25.691 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 16.923916882, 2024-03-19T11:30:25.692 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0092954909999999995, 2024-03-19T11:30:25.692 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:25.692 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:25.692 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:25.692 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:25.692 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:25.692 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 846 2024-03-19T11:30:25.692 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.692 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:25.692 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.692 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:25.692 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756597+0000", 2024-03-19T11:30:25.692 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:25.692 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.692 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.693 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:25.693 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756597+0000", 2024-03-19T11:30:25.693 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:25.693 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.693 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.693 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:25.693 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756598+0000", 2024-03-19T11:30:25.693 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4600000000000004e-07 2024-03-19T11:30:25.693 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.693 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.693 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:25.693 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756604+0000", 2024-03-19T11:30:25.693 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5010000000000004e-06 2024-03-19T11:30:25.693 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.693 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.694 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:25.694 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756613+0000", 2024-03-19T11:30:25.694 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3999999999999998e-06 2024-03-19T11:30:25.694 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.694 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.694 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:25.694 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756620+0000", 2024-03-19T11:30:25.694 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.968e-06 2024-03-19T11:30:25.694 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.694 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.694 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:25.694 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.758321+0000", 2024-03-19T11:30:25.694 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001701337 2024-03-19T11:30:25.694 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.694 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.695 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:25.695 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.758414+0000", 2024-03-19T11:30:25.695 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2375999999999998e-05 2024-03-19T11:30:25.695 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.695 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.695 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:25.695 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.758553+0000", 2024-03-19T11:30:25.695 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013910300000000001 2024-03-19T11:30:25.695 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.695 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.695 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:25.695 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.758987+0000", 2024-03-19T11:30:25.695 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00043375100000000002 2024-03-19T11:30:25.695 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.695 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.696 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:25.696 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.765803+0000", 2024-03-19T11:30:25.696 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0068161089999999999 2024-03-19T11:30:25.696 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.696 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.696 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:25.696 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.765846+0000", 2024-03-19T11:30:25.696 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3226000000000003e-05 2024-03-19T11:30:25.696 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.696 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.696 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:25.696 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.765893+0000", 2024-03-19T11:30:25.696 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6974000000000001e-05 2024-03-19T11:30:25.696 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:25.696 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:25.697 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:25.697 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.697 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.697 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:847 4.7 4:e41d47fb:::smithi03829856-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:25.697 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:08.756635+0000", 2024-03-19T11:30:25.697 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 16.923879020000001, 2024-03-19T11:30:25.697 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093966859999999996, 2024-03-19T11:30:25.697 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:25.697 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:25.697 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:25.697 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:25.697 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:25.697 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 847 2024-03-19T11:30:25.697 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.698 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:25.698 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.698 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:25.698 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756635+0000", 2024-03-19T11:30:25.698 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:25.698 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.698 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.698 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:25.698 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756635+0000", 2024-03-19T11:30:25.698 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:25.698 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.698 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.698 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:25.698 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756636+0000", 2024-03-19T11:30:25.699 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9500000000000002e-07 2024-03-19T11:30:25.699 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.699 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.699 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:25.699 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756640+0000", 2024-03-19T11:30:25.699 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7349999999999998e-06 2024-03-19T11:30:25.699 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.699 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.699 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:25.699 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756647+0000", 2024-03-19T11:30:25.699 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7130000000000005e-06 2024-03-19T11:30:25.699 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.699 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.699 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:25.699 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.756653+0000", 2024-03-19T11:30:25.700 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3229999999999997e-06 2024-03-19T11:30:25.700 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.700 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.700 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:25.700 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.758756+0000", 2024-03-19T11:30:25.700 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0021037170000000002 2024-03-19T11:30:25.700 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.700 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.700 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-03-19T11:30:25.700 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.758814+0000", 2024-03-19T11:30:25.700 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7253999999999998e-05 2024-03-19T11:30:25.700 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.700 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.700 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:25.701 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.765926+0000", 2024-03-19T11:30:25.701 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0071118350000000004 2024-03-19T11:30:25.701 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.701 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.701 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:25.701 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.765970+0000", 2024-03-19T11:30:25.701 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4761000000000001e-05 2024-03-19T11:30:25.701 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.701 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.702 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:25.702 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:08.766032+0000", 2024-03-19T11:30:25.702 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1753000000000003e-05 2024-03-19T11:30:25.702 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:25.702 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:25.702 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:25.702 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.702 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.702 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:998 4.4 4:27909373:::smithi03829856-46:head [rollback 2] snapc 5=[5,2] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-03-19T11:30:25.702 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.368196+0000", 2024-03-19T11:30:25.702 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 13.312318156, 2024-03-19T11:30:25.702 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0079626200000000001, 2024-03-19T11:30:25.702 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:25.702 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:25.703 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:25.703 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:25.703 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:25.703 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 998 2024-03-19T11:30:25.703 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.703 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:25.703 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.703 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:25.703 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368196+0000", 2024-03-19T11:30:25.703 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:25.703 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.703 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.703 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:25.703 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368196+0000", 2024-03-19T11:30:25.703 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:25.704 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.704 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.704 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:25.704 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368197+0000", 2024-03-19T11:30:25.704 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6899999999999998e-07 2024-03-19T11:30:25.704 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.704 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.704 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:25.704 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368200+0000", 2024-03-19T11:30:25.704 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1389999999999999e-06 2024-03-19T11:30:25.704 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.704 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.704 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:25.704 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368207+0000", 2024-03-19T11:30:25.704 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9870000000000002e-06 2024-03-19T11:30:25.705 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.705 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.705 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:25.705 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368212+0000", 2024-03-19T11:30:25.705 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3820000000000003e-06 2024-03-19T11:30:25.705 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.705 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.705 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:25.705 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.369920+0000", 2024-03-19T11:30:25.705 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017078 2024-03-19T11:30:25.705 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.705 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.705 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:25.705 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370087+0000", 2024-03-19T11:30:25.705 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016742800000000001 2024-03-19T11:30:25.705 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.706 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.706 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 2", 2024-03-19T11:30:25.706 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370522+0000", 2024-03-19T11:30:25.706 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00043494300000000003 2024-03-19T11:30:25.706 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.706 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.706 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:25.706 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.375478+0000", 2024-03-19T11:30:25.706 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0049562240000000004 2024-03-19T11:30:25.706 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.706 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.706 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:25.706 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.375918+0000", 2024-03-19T11:30:25.706 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00043914699999999997 2024-03-19T11:30:25.707 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.707 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.707 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:25.707 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.376083+0000", 2024-03-19T11:30:25.707 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0001649 2024-03-19T11:30:25.707 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.707 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.707 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:25.707 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.376159+0000", 2024-03-19T11:30:25.707 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6301e-05 2024-03-19T11:30:25.707 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:25.707 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:25.707 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:25.707 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.707 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.708 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:999 4.4 4:27909373:::smithi03829856-46:head [append 0~0] snapc 5=[5,2] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-03-19T11:30:25.708 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.368233+0000", 2024-03-19T11:30:25.708 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 13.312281180999999, 2024-03-19T11:30:25.708 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0081209219999999992, 2024-03-19T11:30:25.708 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:25.708 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:25.708 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:25.708 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:25.708 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:25.708 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 999 2024-03-19T11:30:25.708 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.708 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:25.708 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.708 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:25.708 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368233+0000", 2024-03-19T11:30:25.709 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:25.709 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.709 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.709 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:25.709 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368233+0000", 2024-03-19T11:30:25.709 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:25.709 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.709 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.709 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:25.709 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368234+0000", 2024-03-19T11:30:25.709 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5400000000000002e-07 2024-03-19T11:30:25.709 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.709 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.709 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:25.710 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368236+0000", 2024-03-19T11:30:25.710 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.7010000000000001e-06 2024-03-19T11:30:25.710 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.710 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.710 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:25.710 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368244+0000", 2024-03-19T11:30:25.710 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.289e-06 2024-03-19T11:30:25.710 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.710 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.710 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:25.710 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.368248+0000", 2024-03-19T11:30:25.710 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9429999999999999e-06 2024-03-19T11:30:25.710 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.711 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.711 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:25.711 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372084+0000", 2024-03-19T11:30:25.711 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038354769999999999 2024-03-19T11:30:25.711 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.711 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.711 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:25.711 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372185+0000", 2024-03-19T11:30:25.711 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010088400000000001 2024-03-19T11:30:25.711 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.711 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.712 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 2", 2024-03-19T11:30:25.712 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372376+0000", 2024-03-19T11:30:25.712 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000191545 2024-03-19T11:30:25.712 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.712 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.712 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:25.712 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.376187+0000", 2024-03-19T11:30:25.712 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038108159999999999 2024-03-19T11:30:25.712 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.712 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.712 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:25.712 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.376243+0000", 2024-03-19T11:30:25.712 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6032000000000002e-05 2024-03-19T11:30:25.712 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.713 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.713 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:25.713 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.376304+0000", 2024-03-19T11:30:25.713 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0834000000000001e-05 2024-03-19T11:30:25.713 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.713 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.713 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:25.713 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.376354+0000", 2024-03-19T11:30:25.713 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0046999999999997e-05 2024-03-19T11:30:25.713 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:25.713 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:25.713 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:25.713 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.713 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.714 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:53 3.0 3:3dcd2456:::039914993b6f4082c204b24ae0ff925ebffccc22519067b64afa9eff107cf533:head [read 0~243792 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:25.714 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.761757+0000", 2024-03-19T11:30:25.714 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 12.918757073, 2024-03-19T11:30:25.714 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0076641110000000004, 2024-03-19T11:30:25.714 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:25.714 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:25.714 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:25.714 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.714 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:25.714 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761757+0000", 2024-03-19T11:30:25.714 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:25.714 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.714 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.714 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:25.715 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761757+0000", 2024-03-19T11:30:25.715 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:25.715 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.715 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.715 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:25.715 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761759+0000", 2024-03-19T11:30:25.715 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0980000000000001e-06 2024-03-19T11:30:25.715 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.715 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.715 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:25.715 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761771+0000", 2024-03-19T11:30:25.715 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1625e-05 2024-03-19T11:30:25.715 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.715 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.715 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:25.715 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761793+0000", 2024-03-19T11:30:25.716 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1940999999999999e-05 2024-03-19T11:30:25.716 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.716 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.716 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:25.716 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761803+0000", 2024-03-19T11:30:25.716 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0234999999999999e-05 2024-03-19T11:30:25.716 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.716 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.716 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:25.716 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761892+0000", 2024-03-19T11:30:25.716 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8974999999999994e-05 2024-03-19T11:30:25.716 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.716 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.716 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-03-19T11:30:25.716 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761899+0000", 2024-03-19T11:30:25.717 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1709999999999996e-06 2024-03-19T11:30:25.717 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.717 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.717 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:25.717 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.768690+0000", 2024-03-19T11:30:25.717 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.006790941 2024-03-19T11:30:25.717 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.717 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.717 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:25.717 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.768790+0000", 2024-03-19T11:30:25.717 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9569999999999997e-05 2024-03-19T11:30:25.717 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.717 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.717 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:25.717 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.769421+0000", 2024-03-19T11:30:25.718 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00063155500000000001 2024-03-19T11:30:25.718 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:25.718 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:25.718 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:25.718 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.718 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.718 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:54 3.0 3:0f3835fc:::79445f5816960fdb5fb7d7699594b1dfc30a0a44639e804aee69d6cb400c02a5:head [read 0~213457 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:25.718 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.761831+0000", 2024-03-19T11:30:25.718 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 12.918683553999999, 2024-03-19T11:30:25.718 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0086999490000000002, 2024-03-19T11:30:25.718 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:25.718 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:25.718 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:25.718 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.719 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:25.719 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761831+0000", 2024-03-19T11:30:25.719 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:25.719 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.719 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.719 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:25.719 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761831+0000", 2024-03-19T11:30:25.719 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:25.719 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.719 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.719 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:25.719 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761831+0000", 2024-03-19T11:30:25.719 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8699999999999995e-07 2024-03-19T11:30:25.719 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.719 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.720 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:25.720 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761835+0000", 2024-03-19T11:30:25.720 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5039999999999998e-06 2024-03-19T11:30:25.720 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.720 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.720 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:25.720 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761847+0000", 2024-03-19T11:30:25.720 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.242e-05 2024-03-19T11:30:25.720 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.720 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.720 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:25.720 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761855+0000", 2024-03-19T11:30:25.720 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2459999999999997e-06 2024-03-19T11:30:25.720 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.720 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.721 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:25.721 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762004+0000", 2024-03-19T11:30:25.721 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014920499999999999 2024-03-19T11:30:25.721 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.721 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.721 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:25.721 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762009+0000", 2024-03-19T11:30:25.721 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7350000000000001e-06 2024-03-19T11:30:25.721 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.721 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.721 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:25.721 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.769460+0000", 2024-03-19T11:30:25.721 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074506529999999998 2024-03-19T11:30:25.721 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.721 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.722 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:25.722 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.769564+0000", 2024-03-19T11:30:25.722 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000103608 2024-03-19T11:30:25.722 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.722 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.722 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:25.722 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.770531+0000", 2024-03-19T11:30:25.722 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00096699100000000005 2024-03-19T11:30:25.722 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:25.722 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:25.723 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:25.723 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.723 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.723 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1055 4.4 4:266590a5:::smithi03829856-16:6 [read 0~3092455,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:25.723 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.768707+0000", 2024-03-19T11:30:25.723 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 11.911807564, 2024-03-19T11:30:25.723 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0088150090000000004, 2024-03-19T11:30:25.723 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:25.723 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:25.723 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:25.723 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:25.723 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:25.723 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1055 2024-03-19T11:30:25.724 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.724 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:25.724 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.724 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:25.724 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768707+0000", 2024-03-19T11:30:25.724 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:25.724 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.724 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.724 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:25.724 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768707+0000", 2024-03-19T11:30:25.724 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:25.724 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.724 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.724 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:25.724 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768708+0000", 2024-03-19T11:30:25.725 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2200000000000003e-07 2024-03-19T11:30:25.725 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.725 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.725 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:25.725 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768716+0000", 2024-03-19T11:30:25.725 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4810000000000006e-06 2024-03-19T11:30:25.725 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.725 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.725 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:25.725 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768728+0000", 2024-03-19T11:30:25.725 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2167999999999999e-05 2024-03-19T11:30:25.725 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.725 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.725 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:25.726 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768737+0000", 2024-03-19T11:30:25.726 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8419999999999994e-06 2024-03-19T11:30:25.726 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.726 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.726 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:25.726 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768895+0000", 2024-03-19T11:30:25.726 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000157455 2024-03-19T11:30:25.726 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.726 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.726 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:25.726 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768897+0000", 2024-03-19T11:30:25.726 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.7580000000000001e-06 2024-03-19T11:30:25.726 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.726 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.726 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:25.727 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.775284+0000", 2024-03-19T11:30:25.727 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0063862069999999996 2024-03-19T11:30:25.727 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.727 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.727 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:25.727 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.775362+0000", 2024-03-19T11:30:25.727 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8604999999999997e-05 2024-03-19T11:30:25.727 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.727 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.727 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:25.727 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.777522+0000", 2024-03-19T11:30:25.727 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0021596710000000002 2024-03-19T11:30:25.727 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:25.727 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:25.727 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:25.728 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.728 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.728 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1056 4.4 4:266590a5:::smithi03829856-16:6 [read 0~3092455,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:25.728 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.768750+0000", 2024-03-19T11:30:25.728 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 11.911764760000001, 2024-03-19T11:30:25.728 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010087227000000001, 2024-03-19T11:30:25.728 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:25.728 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:25.728 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:25.728 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:25.728 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:25.728 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1056 2024-03-19T11:30:25.728 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.728 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:25.728 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.729 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:25.729 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768750+0000", 2024-03-19T11:30:25.729 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:25.729 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.729 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.729 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:25.729 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768750+0000", 2024-03-19T11:30:25.729 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:25.729 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.729 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.729 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:25.729 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768750+0000", 2024-03-19T11:30:25.729 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6199999999999997e-07 2024-03-19T11:30:25.729 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.729 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.730 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:25.730 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768754+0000", 2024-03-19T11:30:25.730 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.9920000000000001e-06 2024-03-19T11:30:25.730 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.730 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.730 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:25.730 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768764+0000", 2024-03-19T11:30:25.730 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0052e-05 2024-03-19T11:30:25.730 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.730 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.730 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:25.730 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768773+0000", 2024-03-19T11:30:25.730 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7309999999999999e-06 2024-03-19T11:30:25.730 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.730 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.731 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:25.731 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768982+0000", 2024-03-19T11:30:25.731 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000208653 2024-03-19T11:30:25.731 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.731 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.731 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:25.731 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.768990+0000", 2024-03-19T11:30:25.731 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3000000000000002e-06 2024-03-19T11:30:25.731 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.731 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.731 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:25.731 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.777544+0000", 2024-03-19T11:30:25.731 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.008554374 2024-03-19T11:30:25.731 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.732 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.732 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:25.732 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.777599+0000", 2024-03-19T11:30:25.732 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4880999999999999e-05 2024-03-19T11:30:25.732 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.732 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.732 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:25.732 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.778837+0000", 2024-03-19T11:30:25.732 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0012375820000000001 2024-03-19T11:30:25.732 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:25.732 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:25.733 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:25.733 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.733 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.733 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1063 4.4 4:214026b3:::smithi03829856-47:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:25.733 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.784904+0000", 2024-03-19T11:30:25.733 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 11.895610318999999, 2024-03-19T11:30:25.733 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024425912000000001, 2024-03-19T11:30:25.733 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:25.733 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:25.733 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:25.733 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:25.733 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:25.733 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1063 2024-03-19T11:30:25.733 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.733 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:25.734 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.734 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:25.734 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784904+0000", 2024-03-19T11:30:25.734 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:25.734 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.734 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.734 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:25.734 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784904+0000", 2024-03-19T11:30:25.734 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:25.734 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.734 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.734 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:25.734 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784906+0000", 2024-03-19T11:30:25.734 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1409999999999999e-06 2024-03-19T11:30:25.735 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.735 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.735 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:25.735 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784915+0000", 2024-03-19T11:30:25.735 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5459999999999992e-06 2024-03-19T11:30:25.735 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.735 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.735 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:25.735 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784935+0000", 2024-03-19T11:30:25.735 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0310999999999999e-05 2024-03-19T11:30:25.735 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.735 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.735 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:25.735 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784946+0000", 2024-03-19T11:30:25.735 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0675e-05 2024-03-19T11:30:25.736 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.736 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.736 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:25.736 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.785032+0000", 2024-03-19T11:30:25.736 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6049000000000002e-05 2024-03-19T11:30:25.736 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.736 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.736 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:25.736 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.785136+0000", 2024-03-19T11:30:25.736 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000104209 2024-03-19T11:30:25.736 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.736 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.736 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:25.736 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.809330+0000", 2024-03-19T11:30:25.737 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024193981 2024-03-19T11:30:25.737 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:25.737 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:25.737 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:25.737 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.737 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.737 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1077 4.7 4:e41d47fb:::smithi03829856-14:head [write 510946~632952,stat] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:25.737 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.993080+0000", 2024-03-19T11:30:25.737 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 11.687434398000001, 2024-03-19T11:30:25.737 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0095463609999999997, 2024-03-19T11:30:25.737 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:25.737 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:25.737 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:25.737 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:25.738 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:25.738 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1077 2024-03-19T11:30:25.738 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.738 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:25.738 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.738 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:25.738 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.993080+0000", 2024-03-19T11:30:25.738 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:25.738 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.738 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.738 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:25.738 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.993080+0000", 2024-03-19T11:30:25.738 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:25.738 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.739 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.739 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:25.739 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.993082+0000", 2024-03-19T11:30:25.739 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3199999999999998e-06 2024-03-19T11:30:25.739 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.739 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.739 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:25.739 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.993356+0000", 2024-03-19T11:30:25.739 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00027414699999999998 2024-03-19T11:30:25.739 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.739 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.739 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:25.739 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.993377+0000", 2024-03-19T11:30:25.739 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1015000000000001e-05 2024-03-19T11:30:25.739 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.739 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.740 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:25.740 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.993387+0000", 2024-03-19T11:30:25.740 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.5200000000000003e-06 2024-03-19T11:30:25.740 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.740 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.740 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:25.740 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.993468+0000", 2024-03-19T11:30:25.740 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1145999999999999e-05 2024-03-19T11:30:25.740 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.740 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.740 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:25.740 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.993595+0000", 2024-03-19T11:30:25.740 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012649199999999999 2024-03-19T11:30:25.740 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.740 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.740 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:25.741 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.993826+0000", 2024-03-19T11:30:25.741 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000231616 2024-03-19T11:30:25.741 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.741 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.741 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:25.741 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.997639+0000", 2024-03-19T11:30:25.741 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038124830000000002 2024-03-19T11:30:25.741 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.741 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.741 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:25.741 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.002505+0000", 2024-03-19T11:30:25.741 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0048662339999999997 2024-03-19T11:30:25.741 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.741 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.741 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:25.742 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.002569+0000", 2024-03-19T11:30:25.742 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3629999999999999e-05 2024-03-19T11:30:25.742 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.742 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.742 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:25.742 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.002626+0000", 2024-03-19T11:30:25.742 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7757999999999998e-05 2024-03-19T11:30:25.742 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:25.742 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:25.742 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:25.742 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.742 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.742 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1081 4.7 4:e41d47fb:::smithi03829856-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:25.743 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:14.006496+0000", 2024-03-19T11:30:25.743 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 11.674018423, 2024-03-19T11:30:25.743 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0070742670000000004, 2024-03-19T11:30:25.743 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:25.743 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:25.743 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:25.743 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:25.743 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:25.743 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1081 2024-03-19T11:30:25.743 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.743 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:25.743 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.743 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:25.743 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006496+0000", 2024-03-19T11:30:25.743 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:25.744 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.744 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.744 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:25.744 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006496+0000", 2024-03-19T11:30:25.744 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:25.744 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.744 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.744 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:25.744 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006496+0000", 2024-03-19T11:30:25.744 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2100000000000002e-07 2024-03-19T11:30:25.744 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.744 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.744 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:25.744 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006499+0000", 2024-03-19T11:30:25.744 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5560000000000002e-06 2024-03-19T11:30:25.745 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.745 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.745 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:25.745 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006506+0000", 2024-03-19T11:30:25.745 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4240000000000004e-06 2024-03-19T11:30:25.745 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.745 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.745 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:25.745 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006515+0000", 2024-03-19T11:30:25.745 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5949999999999999e-06 2024-03-19T11:30:25.745 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.745 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.745 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:25.745 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.009510+0000", 2024-03-19T11:30:25.745 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002995546 2024-03-19T11:30:25.745 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.746 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.746 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-03-19T11:30:25.746 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.009586+0000", 2024-03-19T11:30:25.746 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5859999999999995e-05 2024-03-19T11:30:25.746 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.746 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.746 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:25.746 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.013438+0000", 2024-03-19T11:30:25.746 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038516850000000001 2024-03-19T11:30:25.746 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.746 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.746 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:25.746 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.013499+0000", 2024-03-19T11:30:25.746 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0951e-05 2024-03-19T11:30:25.747 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.747 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.747 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:25.747 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.013570+0000", 2024-03-19T11:30:25.747 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1229000000000005e-05 2024-03-19T11:30:25.747 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:25.747 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:25.747 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:25.747 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.747 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.747 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1215 3.0 3:0c9406a2:::98f4ba27fa68608c540dca9b03a4531ee556ff2734aa426809de93c45c84ab96:head [write 0~1520640] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:25.747 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.147833+0000", 2024-03-19T11:30:25.747 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 7.5326815590000002, 2024-03-19T11:30:25.747 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0078358579999999994, 2024-03-19T11:30:25.747 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:25.748 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:25.748 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:25.748 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:25.748 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:25.748 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1215 2024-03-19T11:30:25.748 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.748 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:25.748 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.748 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:25.748 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.147833+0000", 2024-03-19T11:30:25.748 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:25.748 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.748 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.748 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:25.748 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.147833+0000", 2024-03-19T11:30:25.749 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:25.749 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.749 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.749 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:25.749 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.147835+0000", 2024-03-19T11:30:25.749 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5119999999999998e-06 2024-03-19T11:30:25.749 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.749 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.749 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:25.749 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.148763+0000", 2024-03-19T11:30:25.749 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00092776399999999998 2024-03-19T11:30:25.749 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.749 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.749 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:25.749 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.148792+0000", 2024-03-19T11:30:25.750 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9040999999999999e-05 2024-03-19T11:30:25.750 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.750 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.750 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:25.750 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.148805+0000", 2024-03-19T11:30:25.750 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3079e-05 2024-03-19T11:30:25.750 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.750 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.750 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:25.750 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.148897+0000", 2024-03-19T11:30:25.750 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1942999999999997e-05 2024-03-19T11:30:25.750 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.750 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.751 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-03-19T11:30:25.751 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.148906+0000", 2024-03-19T11:30:25.751 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5050000000000007e-06 2024-03-19T11:30:25.751 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.751 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.751 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:25.751 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.150778+0000", 2024-03-19T11:30:25.751 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0018720880000000001 2024-03-19T11:30:25.751 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.751 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.751 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:25.751 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.150930+0000", 2024-03-19T11:30:25.751 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015192299999999999 2024-03-19T11:30:25.751 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.751 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.752 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 2", 2024-03-19T11:30:25.752 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.151086+0000", 2024-03-19T11:30:25.752 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015680199999999999 2024-03-19T11:30:25.752 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.752 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.752 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:25.752 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155119+0000", 2024-03-19T11:30:25.752 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0040322680000000003 2024-03-19T11:30:25.752 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.752 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.752 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:25.752 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155570+0000", 2024-03-19T11:30:25.752 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00045105999999999998 2024-03-19T11:30:25.752 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.752 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.753 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:25.753 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155623+0000", 2024-03-19T11:30:25.753 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3133000000000001e-05 2024-03-19T11:30:25.753 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.753 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.753 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:25.753 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155669+0000", 2024-03-19T11:30:25.753 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5739999999999999e-05 2024-03-19T11:30:25.753 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:25.753 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:25.753 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:25.753 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.753 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.753 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.0.13:55 3.0 3:0d712e74:::bb863e2508028016f9ab1e6296121a5f2f252d02d0ddc1e79616682e5e6f5b3c:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:25.753 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.170762+0000", 2024-03-19T11:30:25.754 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 7.5097524929999997, 2024-03-19T11:30:25.754 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077288690000000002, 2024-03-19T11:30:25.754 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:25.754 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:25.754 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:25.754 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.754 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:25.754 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.170762+0000", 2024-03-19T11:30:25.754 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:25.754 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.754 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.754 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:25.754 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.170762+0000", 2024-03-19T11:30:25.754 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:25.755 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.755 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.755 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:25.755 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.170765+0000", 2024-03-19T11:30:25.755 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1159999999999999e-06 2024-03-19T11:30:25.755 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.755 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.755 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:25.755 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173312+0000", 2024-03-19T11:30:25.755 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025465319999999998 2024-03-19T11:30:25.755 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.755 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.755 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:25.755 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173370+0000", 2024-03-19T11:30:25.755 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8050999999999997e-05 2024-03-19T11:30:25.755 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.756 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.756 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:25.756 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173397+0000", 2024-03-19T11:30:25.756 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.7874000000000001e-05 2024-03-19T11:30:25.756 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.756 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.756 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:25.756 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173562+0000", 2024-03-19T11:30:25.756 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016487300000000001 2024-03-19T11:30:25.756 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.756 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.756 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:25.756 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173742+0000", 2024-03-19T11:30:25.756 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017933200000000001 2024-03-19T11:30:25.756 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.757 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.757 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 2", 2024-03-19T11:30:25.757 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.174093+0000", 2024-03-19T11:30:25.757 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000351257 2024-03-19T11:30:25.757 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.757 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.757 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:25.757 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177271+0000", 2024-03-19T11:30:25.757 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031785210000000001 2024-03-19T11:30:25.757 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.757 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.757 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:25.757 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.178330+0000", 2024-03-19T11:30:25.757 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00105832 2024-03-19T11:30:25.757 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.758 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.758 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:25.758 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.178432+0000", 2024-03-19T11:30:25.758 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000102709 2024-03-19T11:30:25.758 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.758 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.758 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:25.758 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.178491+0000", 2024-03-19T11:30:25.758 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8284e-05 2024-03-19T11:30:25.758 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:25.758 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:25.758 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:25.758 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.758 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.758 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1235 4.7 4:e41d47fb:::smithi03829856-14:head [write 679016~701795,stat] snapc 9=[9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:25.759 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.331979+0000", 2024-03-19T11:30:25.759 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 7.3485357320000002, 2024-03-19T11:30:25.759 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012331527, 2024-03-19T11:30:25.759 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:25.759 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:25.759 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:25.759 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:25.759 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:25.759 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1235 2024-03-19T11:30:25.759 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.759 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:25.759 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.759 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:25.759 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.331979+0000", 2024-03-19T11:30:25.759 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:25.760 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.760 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.760 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:25.760 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.331979+0000", 2024-03-19T11:30:25.760 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:25.760 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.760 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.760 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:25.760 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.331981+0000", 2024-03-19T11:30:25.760 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.593e-06 2024-03-19T11:30:25.760 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.760 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.760 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:25.760 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.332253+0000", 2024-03-19T11:30:25.760 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00027209199999999998 2024-03-19T11:30:25.761 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.761 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.761 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:25.761 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.332289+0000", 2024-03-19T11:30:25.761 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5519999999999999e-05 2024-03-19T11:30:25.761 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.761 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.761 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:25.761 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.332314+0000", 2024-03-19T11:30:25.761 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5151e-05 2024-03-19T11:30:25.761 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.761 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.761 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:25.761 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.332446+0000", 2024-03-19T11:30:25.761 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013162100000000001 2024-03-19T11:30:25.762 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.762 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.762 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:25.762 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.332546+0000", 2024-03-19T11:30:25.762 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000100382 2024-03-19T11:30:25.762 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.762 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.762 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:25.762 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.332718+0000", 2024-03-19T11:30:25.762 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017210700000000001 2024-03-19T11:30:25.762 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.762 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.762 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:25.762 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.336840+0000", 2024-03-19T11:30:25.763 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0041217279999999999 2024-03-19T11:30:25.763 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.763 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.763 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:25.763 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.344205+0000", 2024-03-19T11:30:25.763 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0073654389999999997 2024-03-19T11:30:25.763 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.763 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.763 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:25.763 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.344258+0000", 2024-03-19T11:30:25.763 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2598e-05 2024-03-19T11:30:25.763 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.763 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.763 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:25.764 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.344310+0000", 2024-03-19T11:30:25.764 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2296000000000002e-05 2024-03-19T11:30:25.764 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:25.764 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:25.764 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:25.764 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.764 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.764 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1236 4.7 4:e41d47fb:::smithi03829856-14:head [write 1866533~496024] snapc 9=[9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:25.764 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.338603+0000", 2024-03-19T11:30:25.764 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 7.3419115760000002, 2024-03-19T11:30:25.764 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011626472000000001, 2024-03-19T11:30:25.764 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:25.764 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:25.764 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:25.764 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:25.764 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:25.765 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1236 2024-03-19T11:30:25.765 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.765 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:25.765 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.765 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:25.765 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.338603+0000", 2024-03-19T11:30:25.765 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:25.765 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.765 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.765 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:25.765 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.338603+0000", 2024-03-19T11:30:25.765 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:25.765 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.765 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.765 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:25.766 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.338604+0000", 2024-03-19T11:30:25.766 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5799999999999999e-06 2024-03-19T11:30:25.766 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.766 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.766 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:25.766 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.338918+0000", 2024-03-19T11:30:25.766 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000313663 2024-03-19T11:30:25.766 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.766 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.766 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:25.766 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.338935+0000", 2024-03-19T11:30:25.766 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7418000000000002e-05 2024-03-19T11:30:25.766 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.766 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.767 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:25.767 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.338945+0000", 2024-03-19T11:30:25.767 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4390000000000006e-06 2024-03-19T11:30:25.767 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.767 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.767 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:25.767 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.339011+0000", 2024-03-19T11:30:25.767 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5816999999999997e-05 2024-03-19T11:30:25.767 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.767 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.767 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:25.767 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.339098+0000", 2024-03-19T11:30:25.767 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7658999999999995e-05 2024-03-19T11:30:25.767 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.767 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.768 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:25.768 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.339216+0000", 2024-03-19T11:30:25.768 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011810499999999999 2024-03-19T11:30:25.768 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.768 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.768 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:25.768 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.341808+0000", 2024-03-19T11:30:25.768 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025918059999999999 2024-03-19T11:30:25.768 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.768 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.768 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:25.768 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.350075+0000", 2024-03-19T11:30:25.768 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0082672360000000007 2024-03-19T11:30:25.768 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.769 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.769 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:25.769 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.350144+0000", 2024-03-19T11:30:25.769 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8796999999999999e-05 2024-03-19T11:30:25.769 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.769 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.769 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:25.769 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.350229+0000", 2024-03-19T11:30:25.769 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4951999999999996e-05 2024-03-19T11:30:25.769 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:25.769 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:25.769 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:25.769 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.769 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.769 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(osd.4.0:197 4.9 e41/21 4:9ef41243:::smithi03829856-5:head v 41'36, mlcod=32'33)", 2024-03-19T11:30:25.769 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.352401+0000", 2024-03-19T11:30:25.770 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 7.3281132729999996, 2024-03-19T11:30:25.770 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0079977950000000003, 2024-03-19T11:30:25.770 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:25.770 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:25.770 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:25.770 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.770 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:25.770 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.352401+0000", 2024-03-19T11:30:25.770 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:25.770 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.770 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.770 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:25.770 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.352401+0000", 2024-03-19T11:30:25.771 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:25.771 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.771 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.771 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:25.771 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.352401+0000", 2024-03-19T11:30:25.771 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.35e-07 2024-03-19T11:30:25.771 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.771 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.771 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:25.771 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.355415+0000", 2024-03-19T11:30:25.771 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030131899999999998 2024-03-19T11:30:25.771 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.771 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.771 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:25.771 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.355443+0000", 2024-03-19T11:30:25.772 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8710000000000001e-05 2024-03-19T11:30:25.772 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.772 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.772 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:25.772 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.355456+0000", 2024-03-19T11:30:25.772 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2489e-05 2024-03-19T11:30:25.772 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.772 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.772 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:25.772 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.355552+0000", 2024-03-19T11:30:25.772 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6523999999999999e-05 2024-03-19T11:30:25.772 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.772 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.772 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:25.772 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.355586+0000", 2024-03-19T11:30:25.773 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3694e-05 2024-03-19T11:30:25.773 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.773 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.773 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:25.773 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.360345+0000", 2024-03-19T11:30:25.773 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0047591309999999998 2024-03-19T11:30:25.773 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:25.773 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:25.773 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:25.773 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.360399+0000", 2024-03-19T11:30:25.773 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3822000000000001e-05 2024-03-19T11:30:25.773 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:25.773 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:25.773 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:25.773 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:25.774 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:25.774 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:25.775 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-19T11:30:25.776 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:30:25.776 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:30:25.776 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:30:25.776 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:30:25.777 INFO:tasks.rados.rados.0.smithi038.stdout:823: finishing set_chunk to oid 49 2024-03-19T11:30:25.777 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 49 set_chunk returned error code -95 offset: 177664 length: 467968 2024-03-19T11:30:25.777 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-03-19T11:30:25.777 INFO:tasks.rados.rados.0.smithi038.stdout:824: expect (ObjNum 14 snap 0 seq_num 14) 2024-03-19T11:30:25.777 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:30:25.778 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:30:25.778 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:30:25.778 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:30:25.779 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:25.786+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:25.782 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:30:25.782 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:30:25.876 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:25.886+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:25.894 INFO:teuthology.orchestra.run.smithi038.stdout:moved 8 / 78 (10.2564%) 2024-03-19T11:30:25.894 INFO:teuthology.orchestra.run.smithi038.stdout:avg 9.75 2024-03-19T11:30:25.895 INFO:teuthology.orchestra.run.smithi038.stdout:stddev 3.99218 -> 3.11247 (expected baseline 2.92083) 2024-03-19T11:30:25.895 INFO:teuthology.orchestra.run.smithi038.stdout:min osd.6 with 5 -> 5 pgs (0.512821 -> 0.512821 * mean) 2024-03-19T11:30:25.895 INFO:teuthology.orchestra.run.smithi038.stdout:max osd.1 with 18 -> 15 pgs (1.84615 -> 1.53846 * mean) 2024-03-19T11:30:25.895 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:30:25.895 INFO:teuthology.orchestra.run.smithi038.stdout:oload 110 2024-03-19T11:30:25.895 INFO:teuthology.orchestra.run.smithi038.stdout:max_change 3 2024-03-19T11:30:25.895 INFO:teuthology.orchestra.run.smithi038.stdout:max_change_osds 4 2024-03-19T11:30:25.895 INFO:teuthology.orchestra.run.smithi038.stdout:average_utilization 0.0007 2024-03-19T11:30:25.895 INFO:teuthology.orchestra.run.smithi038.stdout:overload_utilization 0.0008 2024-03-19T11:30:25.895 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4 weight 1.0000 -> 0.6495 2024-03-19T11:30:25.895 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1 weight 1.0000 -> 0.7791 2024-03-19T11:30:25.895 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0 weight 1.0000 -> 0.8696 2024-03-19T11:30:25.902 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:25.911+0000 7fa5de8d7640 -1 osd.3 47 handle_osd_map injecting map crc failure 2024-03-19T11:30:25.904 INFO:tasks.rados.rados.0.smithi038.stdout:817: done (8 left) 2024-03-19T11:30:25.904 INFO:tasks.rados.rados.0.smithi038.stdout:818: done (7 left) 2024-03-19T11:30:25.904 INFO:tasks.rados.rados.0.smithi038.stdout:819: done (6 left) 2024-03-19T11:30:25.904 INFO:tasks.rados.rados.0.smithi038.stdout:820: done (5 left) 2024-03-19T11:30:25.905 INFO:tasks.rados.rados.0.smithi038.stdout:821: done (4 left) 2024-03-19T11:30:25.905 INFO:tasks.rados.rados.0.smithi038.stdout:822: done (3 left) 2024-03-19T11:30:25.905 INFO:tasks.rados.rados.0.smithi038.stdout:823: done (2 left) 2024-03-19T11:30:25.905 INFO:tasks.rados.rados.0.smithi038.stdout:824: done (1 left) 2024-03-19T11:30:25.905 INFO:tasks.rados.rados.0.smithi038.stdout:825: done (0 left) 2024-03-19T11:30:25.905 INFO:tasks.rados.rados.0.smithi038.stdout:826: read oid 27 snap -1 2024-03-19T11:30:25.905 INFO:tasks.rados.rados.0.smithi038.stdout:826: expect deleted 2024-03-19T11:30:25.905 INFO:tasks.rados.rados.0.smithi038.stdout:827: read oid 28 snap -1 2024-03-19T11:30:25.905 INFO:tasks.rados.rados.0.smithi038.stdout:827: expect (ObjNum 87 snap 8 seq_num 87) 2024-03-19T11:30:25.905 INFO:tasks.rados.rados.0.smithi038.stdout:828: read oid 8 snap -1 2024-03-19T11:30:25.905 INFO:tasks.rados.rados.0.smithi038.stdout:828: expect (ObjNum 60 snap 3 seq_num 60) 2024-03-19T11:30:25.905 INFO:tasks.rados.rados.0.smithi038.stdout:829: set_chunk oid 44 target oid 2024-03-19T11:30:25.905 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-03-19T11:30:25.911 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:25.911 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:30:25.911 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:30:25.911 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:25.927 INFO:tasks.rados.rados.0.smithi038.stdout:829: set_chunk oid 44 offset: 448000 length: 1731072 target oid 5a675e3a2c8b0683d62c457d46b2f98be18ec92014cc458a4e2237e707e8ab7b offset: 0 snap -1 2024-03-19T11:30:25.927 INFO:tasks.rados.rados.0.smithi038.stdout:830: write (excl) oid 1 current snap is 11 2024-03-19T11:30:25.927 INFO:tasks.rados.rados.0.smithi038.stdout:830: seq_num 105 ranges {564735=613734,1828021=669034,3134120=630615} 2024-03-19T11:30:25.929 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-03-19T11:30:25.929 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 1000 0.05 2024-03-19T11:30:25.935 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-19T11:30:25.935 INFO:tasks.rados.rados.0.smithi038.stdout:830: writing smithi03829856-1 from 564735 to 1178469 tid 1 2024-03-19T11:30:25.937 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:25.950+0000 7f923da1c640 -1 osd.7 47 handle_osd_map injecting map crc failure 2024-03-19T11:30:25.943 INFO:tasks.rados.rados.0.smithi038.stdout:830: writing smithi03829856-1 from 1828021 to 2497055 tid 2 2024-03-19T11:30:25.954 INFO:tasks.rados.rados.0.smithi038.stdout:830: writing smithi03829856-1 from 3134120 to 3764735 tid 3 2024-03-19T11:30:25.954 INFO:tasks.rados.rados.0.smithi038.stdout:831: read oid 41 snap -1 2024-03-19T11:30:25.954 INFO:tasks.rados.rados.0.smithi038.stdout:831: expect (ObjNum 80 snap 6 seq_num 80) 2024-03-19T11:30:25.954 INFO:tasks.rados.rados.0.smithi038.stdout:832: read oid 39 snap -1 2024-03-19T11:30:25.954 INFO:tasks.rados.rados.0.smithi038.stdout:832: expect deleted 2024-03-19T11:30:25.954 INFO:tasks.rados.rados.0.smithi038.stdout:833: read oid 13 snap -1 2024-03-19T11:30:25.954 INFO:tasks.rados.rados.0.smithi038.stdout:833: expect (ObjNum 12 snap 0 seq_num 12) 2024-03-19T11:30:25.954 INFO:tasks.rados.rados.0.smithi038.stdout:834: set_chunk oid 7 target oid 2024-03-19T11:30:25.963 INFO:tasks.rados.rados.0.smithi038.stdout:834: set_chunk oid 7 offset: 2386944 length: 172032 target oid a109c37f2676c2485211b6ebddb2b748105d65040e598a6ac66aa6fa98bb1d13 offset: 0 snap -1 2024-03-19T11:30:25.964 INFO:tasks.rados.rados.0.smithi038.stdout:835: read oid 24 snap 10 2024-03-19T11:30:25.964 INFO:tasks.rados.rados.0.smithi038.stdout:835: expect (ObjNum 23 snap 0 seq_num 23) 2024-03-19T11:30:25.964 INFO:tasks.rados.rados.0.smithi038.stdout:836: read oid 35 snap -1 2024-03-19T11:30:25.964 INFO:tasks.rados.rados.0.smithi038.stdout:836: expect (ObjNum 34 snap 0 seq_num 34) 2024-03-19T11:30:25.964 INFO:tasks.rados.rados.0.smithi038.stdout:837: read oid 47 snap -1 2024-03-19T11:30:25.964 INFO:tasks.rados.rados.0.smithi038.stdout:837: expect (ObjNum 46 snap 0 seq_num 46) 2024-03-19T11:30:25.964 INFO:tasks.rados.rados.0.smithi038.stdout:838: tier_evict oid 2 2024-03-19T11:30:25.964 INFO:tasks.rados.rados.0.smithi038.stdout:838: tier_evict oid 2 snap -1 2024-03-19T11:30:25.964 INFO:tasks.rados.rados.0.smithi038.stdout:839: read oid 22 snap -1 2024-03-19T11:30:25.964 INFO:tasks.rados.rados.0.smithi038.stdout:839: expect (ObjNum 21 snap 0 seq_num 21) 2024-03-19T11:30:25.964 INFO:tasks.rados.rados.0.smithi038.stdout:840: read oid 42 snap -1 2024-03-19T11:30:25.964 INFO:tasks.rados.rados.0.smithi038.stdout:840: expect (ObjNum 98 snap 10 seq_num 98) 2024-03-19T11:30:25.964 INFO:tasks.rados.rados.0.smithi038.stdout:841: write (excl) oid 18 current snap is 11 2024-03-19T11:30:25.964 INFO:tasks.rados.rados.0.smithi038.stdout:841: seq_num 106 ranges {581542=688983,1886503=412692,2761543=670951} 2024-03-19T11:30:25.971 INFO:tasks.rados.rados.0.smithi038.stdout:841: writing smithi03829856-18 from 581542 to 1270525 tid 1 2024-03-19T11:30:25.976 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:25.987+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:25.978 INFO:tasks.rados.rados.0.smithi038.stdout:841: writing smithi03829856-18 from 1886503 to 2299195 tid 2 2024-03-19T11:30:25.986 INFO:tasks.rados.rados.0.smithi038.stdout:841: writing smithi03829856-18 from 2761543 to 3432494 tid 3 2024-03-19T11:30:25.986 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:25.986 INFO:tasks.rados.rados.0.smithi038.stdout:828: expect (ObjNum 60 snap 3 seq_num 60) 2024-03-19T11:30:26.066 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:26.066 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:30:26.066 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:30:26.066 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:30:26.066 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:26.079 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:26.087+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:26.085 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-19T11:30:26.139 INFO:tasks.rados.rados.0.smithi038.stdout:827: expect (ObjNum 87 snap 8 seq_num 87) 2024-03-19T11:30:26.179 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:26.191+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:26.243 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:26.243 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:30:26.243 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:30:26.243 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:30:26.243 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.243 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:9 4.d e28/21 4:b4ef97e0:::smithi03829856-3:head v 28'1, mlcod=0'0)", 2024-03-19T11:30:26.243 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.707728+0000", 2024-03-19T11:30:26.243 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 21.547519896000001, 2024-03-19T11:30:26.243 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0037487950000000001, 2024-03-19T11:30:26.244 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:26.244 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:26.244 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:26.244 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.244 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:26.244 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.707728+0000", 2024-03-19T11:30:26.244 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.244 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.244 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.244 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:26.244 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.707728+0000", 2024-03-19T11:30:26.244 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.244 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.244 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.245 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:26.245 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.707729+0000", 2024-03-19T11:30:26.245 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6899999999999998e-07 2024-03-19T11:30:26.245 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.245 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.245 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:26.245 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.708625+0000", 2024-03-19T11:30:26.245 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00089603899999999995 2024-03-19T11:30:26.245 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.245 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.245 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:26.245 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.708645+0000", 2024-03-19T11:30:26.245 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9629000000000001e-05 2024-03-19T11:30:26.245 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.245 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.246 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:26.246 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.708657+0000", 2024-03-19T11:30:26.246 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2537999999999999e-05 2024-03-19T11:30:26.246 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.246 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.246 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:26.246 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.708740+0000", 2024-03-19T11:30:26.246 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2490999999999993e-05 2024-03-19T11:30:26.246 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.246 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.247 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:26.247 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.708774+0000", 2024-03-19T11:30:26.247 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4362000000000001e-05 2024-03-19T11:30:26.247 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.247 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.247 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:26.247 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.711419+0000", 2024-03-19T11:30:26.247 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0026446830000000001 2024-03-19T11:30:26.247 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.247 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.247 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:26.247 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.711477+0000", 2024-03-19T11:30:26.247 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8684000000000003e-05 2024-03-19T11:30:26.247 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.248 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:26.248 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.248 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.248 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.248 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:13 4.4 e28/21 4:21eac9b2:::smithi03829856-4:head v 28'1, mlcod=0'0)", 2024-03-19T11:30:26.248 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.723258+0000", 2024-03-19T11:30:26.248 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 21.531989900999999, 2024-03-19T11:30:26.248 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0048979840000000002, 2024-03-19T11:30:26.248 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:26.248 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:26.248 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:26.248 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.248 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:26.248 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723258+0000", 2024-03-19T11:30:26.248 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.249 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.249 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.249 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:26.249 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723258+0000", 2024-03-19T11:30:26.249 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.249 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.249 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.249 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:26.249 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723259+0000", 2024-03-19T11:30:26.249 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.7399999999999999e-07 2024-03-19T11:30:26.249 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.249 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.249 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:26.249 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723812+0000", 2024-03-19T11:30:26.249 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00055371000000000001 2024-03-19T11:30:26.250 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.250 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.250 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:26.250 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723833+0000", 2024-03-19T11:30:26.250 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0990000000000001e-05 2024-03-19T11:30:26.250 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.250 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.250 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:26.250 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723847+0000", 2024-03-19T11:30:26.250 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3774000000000001e-05 2024-03-19T11:30:26.251 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.251 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.251 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:26.251 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723912+0000", 2024-03-19T11:30:26.251 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4758000000000006e-05 2024-03-19T11:30:26.251 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.251 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.251 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:26.251 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723933+0000", 2024-03-19T11:30:26.251 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1046000000000001e-05 2024-03-19T11:30:26.251 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.252 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.252 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:26.252 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.728103+0000", 2024-03-19T11:30:26.252 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0041695079999999997 2024-03-19T11:30:26.252 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.252 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.252 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:26.252 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.728156+0000", 2024-03-19T11:30:26.252 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3924000000000001e-05 2024-03-19T11:30:26.252 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.252 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:26.252 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.252 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.252 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.252 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:33 4.d e28/21 4:b65e6658:::smithi03829856-9:head v 28'4, mlcod=28'2)", 2024-03-19T11:30:26.252 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.785780+0000", 2024-03-19T11:30:26.253 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 21.469468129999999, 2024-03-19T11:30:26.253 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0032708569999999998, 2024-03-19T11:30:26.253 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:26.253 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:26.253 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:26.253 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.253 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:26.253 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.785780+0000", 2024-03-19T11:30:26.253 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.253 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.253 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.253 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:26.253 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.785780+0000", 2024-03-19T11:30:26.253 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.253 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.254 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.254 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:26.254 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.785783+0000", 2024-03-19T11:30:26.254 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4150000000000002e-06 2024-03-19T11:30:26.254 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.254 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.254 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:26.254 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.786482+0000", 2024-03-19T11:30:26.254 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00069957299999999999 2024-03-19T11:30:26.254 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.254 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.254 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:26.254 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.786507+0000", 2024-03-19T11:30:26.254 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5160999999999998e-05 2024-03-19T11:30:26.254 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.255 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.255 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:26.255 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.786521+0000", 2024-03-19T11:30:26.255 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3301e-05 2024-03-19T11:30:26.255 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.255 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.255 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:26.255 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.786599+0000", 2024-03-19T11:30:26.255 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7905999999999996e-05 2024-03-19T11:30:26.255 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.255 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.255 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:26.255 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.786630+0000", 2024-03-19T11:30:26.255 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1636e-05 2024-03-19T11:30:26.255 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.256 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.256 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:26.256 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.789001+0000", 2024-03-19T11:30:26.256 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023710530000000001 2024-03-19T11:30:26.256 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.256 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.256 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:26.256 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.789051+0000", 2024-03-19T11:30:26.256 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9811999999999997e-05 2024-03-19T11:30:26.256 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.256 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:26.256 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.256 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.256 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.256 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:871 4.d e32/21 4:b4ef97e0:::smithi03829856-3:head v 32'19, mlcod=28'16)", 2024-03-19T11:30:26.256 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.029331+0000", 2024-03-19T11:30:26.257 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 17.225917228, 2024-03-19T11:30:26.257 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057816339999999999, 2024-03-19T11:30:26.257 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:26.257 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:26.257 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:26.257 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.257 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:26.257 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.029331+0000", 2024-03-19T11:30:26.257 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.257 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.257 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.257 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:26.257 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.029331+0000", 2024-03-19T11:30:26.257 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.258 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.258 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.258 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:26.258 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.029331+0000", 2024-03-19T11:30:26.258 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0400000000000002e-07 2024-03-19T11:30:26.258 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.258 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.258 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:26.258 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.030605+0000", 2024-03-19T11:30:26.258 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0012733899999999999 2024-03-19T11:30:26.258 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.258 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.258 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:26.258 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.030620+0000", 2024-03-19T11:30:26.258 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5557999999999999e-05 2024-03-19T11:30:26.259 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.259 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.259 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:26.259 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.030630+0000", 2024-03-19T11:30:26.259 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2660000000000007e-06 2024-03-19T11:30:26.259 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.259 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.259 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:26.259 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.030727+0000", 2024-03-19T11:30:26.259 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7452000000000001e-05 2024-03-19T11:30:26.259 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.259 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.259 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:26.259 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.030757+0000", 2024-03-19T11:30:26.259 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0018999999999998e-05 2024-03-19T11:30:26.260 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.260 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.260 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:26.260 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.035063+0000", 2024-03-19T11:30:26.260 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0043055350000000001 2024-03-19T11:30:26.260 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.260 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.260 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:26.260 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.035113+0000", 2024-03-19T11:30:26.260 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.011e-05 2024-03-19T11:30:26.260 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.260 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:26.260 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.260 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.260 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.260 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:872 4.d e32/21 4:b4ef97e0:::smithi03829856-3:head v 32'20, mlcod=28'16)", 2024-03-19T11:30:26.261 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.036163+0000", 2024-03-19T11:30:26.261 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 17.219085002, 2024-03-19T11:30:26.261 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029333250000000001, 2024-03-19T11:30:26.261 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:26.261 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:26.261 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:26.261 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.261 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:26.261 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.036163+0000", 2024-03-19T11:30:26.261 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.261 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.261 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.261 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:26.261 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.036163+0000", 2024-03-19T11:30:26.261 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.262 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.262 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.262 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:26.262 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.036164+0000", 2024-03-19T11:30:26.262 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7000000000000003e-07 2024-03-19T11:30:26.262 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.262 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.262 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:26.262 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.036721+0000", 2024-03-19T11:30:26.262 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00055646700000000005 2024-03-19T11:30:26.262 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.262 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.262 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:26.262 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.036732+0000", 2024-03-19T11:30:26.262 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1649e-05 2024-03-19T11:30:26.263 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.263 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.263 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:26.263 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.036739+0000", 2024-03-19T11:30:26.263 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6760000000000001e-06 2024-03-19T11:30:26.263 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.263 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.263 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:26.263 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.036785+0000", 2024-03-19T11:30:26.263 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6177000000000002e-05 2024-03-19T11:30:26.263 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.263 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.263 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:26.263 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.036808+0000", 2024-03-19T11:30:26.263 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2654999999999999e-05 2024-03-19T11:30:26.264 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.264 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.264 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:26.264 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.039022+0000", 2024-03-19T11:30:26.264 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022145530000000002 2024-03-19T11:30:26.264 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.264 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.264 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:26.264 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.039097+0000", 2024-03-19T11:30:26.264 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4277999999999996e-05 2024-03-19T11:30:26.264 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.264 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:26.264 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.264 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.264 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.264 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(osd.6.13:39 3.0 e32/20 3:3dcd2456:::039914993b6f4082c204b24ae0ff925ebffccc22519067b64afa9eff107cf533:head v 32'54, mlcod=32'52)", 2024-03-19T11:30:26.265 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.308854+0000", 2024-03-19T11:30:26.265 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 16.946394713, 2024-03-19T11:30:26.265 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003468175, 2024-03-19T11:30:26.265 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:26.265 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:26.265 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:26.265 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.265 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:26.265 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308854+0000", 2024-03-19T11:30:26.265 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.265 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.265 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.265 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:26.265 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308854+0000", 2024-03-19T11:30:26.265 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.266 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.266 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.266 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:26.266 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308855+0000", 2024-03-19T11:30:26.266 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5999999999999999e-06 2024-03-19T11:30:26.266 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.266 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.266 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:26.266 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309019+0000", 2024-03-19T11:30:26.266 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000163311 2024-03-19T11:30:26.266 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.266 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.266 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:26.266 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309040+0000", 2024-03-19T11:30:26.266 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1795999999999999e-05 2024-03-19T11:30:26.267 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.267 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.267 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:26.267 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309060+0000", 2024-03-19T11:30:26.267 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9677000000000002e-05 2024-03-19T11:30:26.267 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.267 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.267 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:26.267 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309151+0000", 2024-03-19T11:30:26.267 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0507000000000006e-05 2024-03-19T11:30:26.267 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.267 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.267 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:26.267 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309187+0000", 2024-03-19T11:30:26.267 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6675999999999999e-05 2024-03-19T11:30:26.268 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.268 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.268 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:26.268 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.312256+0000", 2024-03-19T11:30:26.268 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030692060000000001 2024-03-19T11:30:26.268 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.268 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.268 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:26.268 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.312322+0000", 2024-03-19T11:30:26.268 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5401999999999995e-05 2024-03-19T11:30:26.268 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.268 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:26.268 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.268 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.268 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.269 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:998 4.4 e34/21 4:27909373:::smithi03829856-46:head v 34'37, mlcod=32'32)", 2024-03-19T11:30:26.269 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.370628+0000", 2024-03-19T11:30:26.269 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 13.8846205, 2024-03-19T11:30:26.269 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050953099999999996, 2024-03-19T11:30:26.269 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:26.269 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:26.269 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:26.269 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.269 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:26.269 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370628+0000", 2024-03-19T11:30:26.269 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.269 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.269 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.269 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:26.269 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370628+0000", 2024-03-19T11:30:26.269 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.270 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.270 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.270 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:26.270 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370629+0000", 2024-03-19T11:30:26.270 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.5900000000000005e-07 2024-03-19T11:30:26.270 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.270 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.270 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:26.270 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370638+0000", 2024-03-19T11:30:26.270 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8189999999999998e-06 2024-03-19T11:30:26.270 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.270 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.270 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:26.270 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370650+0000", 2024-03-19T11:30:26.271 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.275e-05 2024-03-19T11:30:26.271 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.271 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.271 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:26.271 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370658+0000", 2024-03-19T11:30:26.271 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5769999999999996e-06 2024-03-19T11:30:26.271 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.271 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.271 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:26.271 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370747+0000", 2024-03-19T11:30:26.271 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8642999999999998e-05 2024-03-19T11:30:26.271 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.271 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.271 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:26.271 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370796+0000", 2024-03-19T11:30:26.272 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9428000000000002e-05 2024-03-19T11:30:26.272 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.272 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.272 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:26.272 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.375658+0000", 2024-03-19T11:30:26.272 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0048621519999999998 2024-03-19T11:30:26.272 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.272 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.272 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:26.272 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.375723+0000", 2024-03-19T11:30:26.272 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4981999999999995e-05 2024-03-19T11:30:26.272 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.272 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:26.272 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.272 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.273 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.273 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:999 4.4 e34/21 4:27909373:::smithi03829856-46:head v 34'38, mlcod=32'32)", 2024-03-19T11:30:26.273 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.372487+0000", 2024-03-19T11:30:26.273 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 13.882761735000001, 2024-03-19T11:30:26.273 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0035853069999999998, 2024-03-19T11:30:26.273 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:26.273 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:26.273 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:26.273 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.273 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:26.273 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372487+0000", 2024-03-19T11:30:26.273 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.274 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.274 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.274 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:26.274 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372487+0000", 2024-03-19T11:30:26.274 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.274 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.274 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.274 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:26.274 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372488+0000", 2024-03-19T11:30:26.274 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1069999999999999e-06 2024-03-19T11:30:26.274 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.274 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.274 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:26.274 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372498+0000", 2024-03-19T11:30:26.275 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7510000000000003e-06 2024-03-19T11:30:26.275 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.275 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.275 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:26.275 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372544+0000", 2024-03-19T11:30:26.275 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6218999999999999e-05 2024-03-19T11:30:26.275 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.275 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.275 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:26.275 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372553+0000", 2024-03-19T11:30:26.275 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6320000000000003e-06 2024-03-19T11:30:26.275 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.275 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.275 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:26.275 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372636+0000", 2024-03-19T11:30:26.276 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3125000000000001e-05 2024-03-19T11:30:26.276 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.276 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.276 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:26.276 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372651+0000", 2024-03-19T11:30:26.276 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4676e-05 2024-03-19T11:30:26.276 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.276 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.276 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:26.276 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.376026+0000", 2024-03-19T11:30:26.276 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033745630000000001 2024-03-19T11:30:26.276 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.276 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.276 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:26.276 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.376072+0000", 2024-03-19T11:30:26.277 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6233999999999998e-05 2024-03-19T11:30:26.277 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.277 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:26.277 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.277 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.277 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.277 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:1003 4.4 e34/21 4:22709789:::smithi03829856-41:head v 34'40, mlcod=34'37)", 2024-03-19T11:30:26.277 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.378551+0000", 2024-03-19T11:30:26.277 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 13.876697648, 2024-03-19T11:30:26.277 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0041156680000000003, 2024-03-19T11:30:26.277 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:26.277 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:26.278 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:26.278 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.278 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:26.278 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.378551+0000", 2024-03-19T11:30:26.278 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.278 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.278 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.278 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:26.278 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.378551+0000", 2024-03-19T11:30:26.278 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.278 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.278 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.278 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:26.279 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.378552+0000", 2024-03-19T11:30:26.279 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0400000000000005e-07 2024-03-19T11:30:26.279 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.279 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.279 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:26.279 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.378929+0000", 2024-03-19T11:30:26.279 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037751700000000001 2024-03-19T11:30:26.279 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.279 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.279 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:26.279 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.378944+0000", 2024-03-19T11:30:26.279 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5038000000000001e-05 2024-03-19T11:30:26.279 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.279 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.279 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:26.280 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.378953+0000", 2024-03-19T11:30:26.280 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7649999999999999e-06 2024-03-19T11:30:26.280 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.280 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.280 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:26.280 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.379017+0000", 2024-03-19T11:30:26.280 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3614000000000005e-05 2024-03-19T11:30:26.280 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.280 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.280 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:26.280 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.379042+0000", 2024-03-19T11:30:26.280 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5910000000000001e-05 2024-03-19T11:30:26.280 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.280 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.280 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:26.281 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.382610+0000", 2024-03-19T11:30:26.281 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0035678340000000002 2024-03-19T11:30:26.281 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.281 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.281 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:26.281 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.382666+0000", 2024-03-19T11:30:26.281 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6085999999999998e-05 2024-03-19T11:30:26.281 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.281 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:26.281 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.281 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.281 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.281 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:1004 4.4 e34/21 4:22709789:::smithi03829856-41:head v 34'41, mlcod=34'38)", 2024-03-19T11:30:26.281 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.386855+0000", 2024-03-19T11:30:26.282 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 13.868393838999999, 2024-03-19T11:30:26.282 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0039133320000000003, 2024-03-19T11:30:26.282 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:26.282 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:26.282 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:26.282 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.282 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:26.282 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.386855+0000", 2024-03-19T11:30:26.282 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.282 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.282 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.282 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:26.282 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.386855+0000", 2024-03-19T11:30:26.282 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.282 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.283 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.283 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:26.283 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.386856+0000", 2024-03-19T11:30:26.283 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.528e-06 2024-03-19T11:30:26.283 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.283 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.283 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:26.283 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.387143+0000", 2024-03-19T11:30:26.283 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00028658600000000002 2024-03-19T11:30:26.283 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.283 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.284 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:26.284 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.387159+0000", 2024-03-19T11:30:26.284 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6796e-05 2024-03-19T11:30:26.284 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.284 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.284 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:26.284 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.387167+0000", 2024-03-19T11:30:26.284 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2729999999999997e-06 2024-03-19T11:30:26.284 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.284 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.284 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:26.284 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.387230+0000", 2024-03-19T11:30:26.284 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2862999999999998e-05 2024-03-19T11:30:26.284 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.285 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.285 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:26.285 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.387255+0000", 2024-03-19T11:30:26.285 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4905e-05 2024-03-19T11:30:26.285 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.285 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.285 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:26.285 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.390720+0000", 2024-03-19T11:30:26.285 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034652699999999999 2024-03-19T11:30:26.285 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.285 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.285 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:26.285 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.390768+0000", 2024-03-19T11:30:26.285 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8111000000000001e-05 2024-03-19T11:30:26.285 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.286 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:26.286 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.286 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.286 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.286 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:1191 4.4 e40/21 4:27909373:::smithi03829856-46:head v 40'53, mlcod=38'50)", 2024-03-19T11:30:26.286 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:17.966218+0000", 2024-03-19T11:30:26.286 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.2890300069999991, 2024-03-19T11:30:26.286 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038990449999999999, 2024-03-19T11:30:26.286 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:26.286 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:26.286 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:26.286 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.286 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:26.286 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.966218+0000", 2024-03-19T11:30:26.286 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.287 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.287 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.287 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:26.287 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.966218+0000", 2024-03-19T11:30:26.287 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.287 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.287 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.287 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:26.287 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.966219+0000", 2024-03-19T11:30:26.287 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.96e-07 2024-03-19T11:30:26.287 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.287 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.287 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:26.287 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.966448+0000", 2024-03-19T11:30:26.289 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00022896799999999999 2024-03-19T11:30:26.289 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.289 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.289 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:26.289 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.966459+0000", 2024-03-19T11:30:26.289 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1654999999999999e-05 2024-03-19T11:30:26.289 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.289 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.290 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:26.290 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.966466+0000", 2024-03-19T11:30:26.290 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3770000000000001e-06 2024-03-19T11:30:26.290 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.290 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.290 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:26.290 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.966536+0000", 2024-03-19T11:30:26.290 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9808999999999995e-05 2024-03-19T11:30:26.290 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.290 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.290 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:26.290 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.966564+0000", 2024-03-19T11:30:26.290 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8319000000000001e-05 2024-03-19T11:30:26.290 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.290 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.291 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:26.291 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.970055+0000", 2024-03-19T11:30:26.291 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034915860000000001 2024-03-19T11:30:26.291 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.291 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.291 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:26.291 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.970117+0000", 2024-03-19T11:30:26.291 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1934999999999996e-05 2024-03-19T11:30:26.291 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.291 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:26.291 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.291 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.291 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.292 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:1192 4.4 e40/21 4:27909373:::smithi03829856-46:head v 40'54, mlcod=39'51)", 2024-03-19T11:30:26.292 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:17.977421+0000", 2024-03-19T11:30:26.292 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.2778271659999998, 2024-03-19T11:30:26.292 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0040889289999999998, 2024-03-19T11:30:26.292 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:26.292 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:26.292 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:26.292 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.292 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:26.292 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.977421+0000", 2024-03-19T11:30:26.292 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.292 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.292 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.292 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:26.292 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.977421+0000", 2024-03-19T11:30:26.293 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.293 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.293 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.293 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:26.293 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.977422+0000", 2024-03-19T11:30:26.293 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.271e-06 2024-03-19T11:30:26.293 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.293 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.294 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:26.294 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.977700+0000", 2024-03-19T11:30:26.294 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00027715299999999998 2024-03-19T11:30:26.294 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.294 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.294 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:26.294 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.977718+0000", 2024-03-19T11:30:26.294 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8067000000000002e-05 2024-03-19T11:30:26.294 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.294 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.294 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:26.294 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.977727+0000", 2024-03-19T11:30:26.294 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4919999999999999e-06 2024-03-19T11:30:26.294 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.295 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.295 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:26.295 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.977792+0000", 2024-03-19T11:30:26.295 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4945999999999998e-05 2024-03-19T11:30:26.295 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.295 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.295 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:26.295 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.977818+0000", 2024-03-19T11:30:26.295 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5519000000000001e-05 2024-03-19T11:30:26.295 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.295 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.295 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:26.295 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.981471+0000", 2024-03-19T11:30:26.295 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0036535980000000001 2024-03-19T11:30:26.295 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.295 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.296 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:26.296 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.981510+0000", 2024-03-19T11:30:26.296 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8883e-05 2024-03-19T11:30:26.296 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.296 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:26.296 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.296 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.296 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.296 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:1215 3.0 e41/20 3:0c9406a2:::98f4ba27fa68608c540dca9b03a4531ee556ff2734aa426809de93c45c84ab96:head v 41'65, mlcod=36'63)", 2024-03-19T11:30:26.296 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.151676+0000", 2024-03-19T11:30:26.296 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.1035725260000007, 2024-03-19T11:30:26.296 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0037067860000000001, 2024-03-19T11:30:26.296 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:26.296 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:26.296 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:26.297 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.297 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:26.297 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.151676+0000", 2024-03-19T11:30:26.297 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.297 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.297 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.297 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:26.298 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.151676+0000", 2024-03-19T11:30:26.298 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.298 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.298 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.298 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:26.298 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.151676+0000", 2024-03-19T11:30:26.298 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2500000000000001e-07 2024-03-19T11:30:26.298 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.298 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.298 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:26.298 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.152439+0000", 2024-03-19T11:30:26.298 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00076323500000000002 2024-03-19T11:30:26.298 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.298 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.299 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:26.299 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.152452+0000", 2024-03-19T11:30:26.299 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2356e-05 2024-03-19T11:30:26.299 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.299 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.299 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:26.299 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.152460+0000", 2024-03-19T11:30:26.299 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2609999999999993e-06 2024-03-19T11:30:26.299 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.299 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.299 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:26.299 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.152543+0000", 2024-03-19T11:30:26.299 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2837000000000007e-05 2024-03-19T11:30:26.299 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.299 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.299 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:26.300 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.152571+0000", 2024-03-19T11:30:26.300 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8228000000000001e-05 2024-03-19T11:30:26.300 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.300 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.300 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:26.300 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155345+0000", 2024-03-19T11:30:26.300 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0027742019999999999 2024-03-19T11:30:26.300 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.300 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.300 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:26.300 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155383+0000", 2024-03-19T11:30:26.300 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7342000000000003e-05 2024-03-19T11:30:26.300 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.300 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:26.301 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.301 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.301 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.301 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(osd.1.0:142 4.4 e44/43 4:214026b3:::smithi03829856-47:head v 44'57, mlcod=0'0)", 2024-03-19T11:30:26.301 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:21.867611+0000", 2024-03-19T11:30:26.301 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.387637786, 2024-03-19T11:30:26.301 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0081946840000000007, 2024-03-19T11:30:26.301 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:26.301 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:26.301 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:26.301 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.301 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:26.301 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.867611+0000", 2024-03-19T11:30:26.301 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.301 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.302 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.302 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:26.302 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.867611+0000", 2024-03-19T11:30:26.302 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.302 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.302 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.302 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:26.302 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.867613+0000", 2024-03-19T11:30:26.302 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9250000000000002e-06 2024-03-19T11:30:26.303 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.303 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.303 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:26.303 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.869767+0000", 2024-03-19T11:30:26.303 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002154071 2024-03-19T11:30:26.303 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.303 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.303 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:26.304 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.869792+0000", 2024-03-19T11:30:26.304 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5327e-05 2024-03-19T11:30:26.304 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.304 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.304 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:26.304 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.869803+0000", 2024-03-19T11:30:26.304 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0883999999999999e-05 2024-03-19T11:30:26.304 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.304 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.304 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:26.304 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.869890+0000", 2024-03-19T11:30:26.304 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7049e-05 2024-03-19T11:30:26.304 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.305 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.305 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:26.305 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.869927+0000", 2024-03-19T11:30:26.305 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7465999999999997e-05 2024-03-19T11:30:26.305 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.305 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.305 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:26.305 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.875750+0000", 2024-03-19T11:30:26.305 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0058225969999999997 2024-03-19T11:30:26.305 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.305 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.305 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:26.306 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.875805+0000", 2024-03-19T11:30:26.306 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5365000000000003e-05 2024-03-19T11:30:26.306 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.306 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:26.306 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.306 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.306 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.306 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:1286 4.d e45/21 4:b4ef97e0:::smithi03829856-3:head v 45'30, mlcod=33'27)", 2024-03-19T11:30:26.306 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:22.881538+0000", 2024-03-19T11:30:26.306 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.3737105860000001, 2024-03-19T11:30:26.306 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0073916399999999997, 2024-03-19T11:30:26.306 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:26.306 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:26.306 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:22.881538+0000", 2024-03-19T11:30:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:22.881538+0000", 2024-03-19T11:30:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:22.881538+0000", 2024-03-19T11:30:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3399999999999999e-07 2024-03-19T11:30:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:22.882375+0000", 2024-03-19T11:30:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00083699100000000004 2024-03-19T11:30:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:22.882402+0000", 2024-03-19T11:30:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6571e-05 2024-03-19T11:30:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:22.882417+0000", 2024-03-19T11:30:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4822e-05 2024-03-19T11:30:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:22.882549+0000", 2024-03-19T11:30:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013247899999999999 2024-03-19T11:30:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:22.882591+0000", 2024-03-19T11:30:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1230000000000003e-05 2024-03-19T11:30:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:22.888839+0000", 2024-03-19T11:30:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0062484940000000003 2024-03-19T11:30:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:22.888929+0000", 2024-03-19T11:30:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0419000000000003e-05 2024-03-19T11:30:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:1287 4.d e45/21 4:b4ef97e0:::smithi03829856-3:head v 45'31, mlcod=36'28)", 2024-03-19T11:30:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:22.896045+0000", 2024-03-19T11:30:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.3592033909999999, 2024-03-19T11:30:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053379370000000001, 2024-03-19T11:30:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:22.896045+0000", 2024-03-19T11:30:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:22.896045+0000", 2024-03-19T11:30:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:22.896047+0000", 2024-03-19T11:30:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2589999999999999e-06 2024-03-19T11:30:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:22.896577+0000", 2024-03-19T11:30:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00052966299999999999 2024-03-19T11:30:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:22.896614+0000", 2024-03-19T11:30:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6610999999999999e-05 2024-03-19T11:30:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:22.896633+0000", 2024-03-19T11:30:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9927000000000001e-05 2024-03-19T11:30:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:22.896765+0000", 2024-03-19T11:30:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000131278 2024-03-19T11:30:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:22.896806+0000", 2024-03-19T11:30:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1433000000000001e-05 2024-03-19T11:30:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:22.901321+0000", 2024-03-19T11:30:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0045147329999999999 2024-03-19T11:30:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:22.901383+0000", 2024-03-19T11:30:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2032999999999994e-05 2024-03-19T11:30:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:1288 4.d e45/21 4:b4ef97e0:::smithi03829856-3:head v 45'32, mlcod=45'30)", 2024-03-19T11:30:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:22.911394+0000", 2024-03-19T11:30:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.343854731, 2024-03-19T11:30:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033045100000000001, 2024-03-19T11:30:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:22.911394+0000", 2024-03-19T11:30:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.316 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:26.316 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:22.911394+0000", 2024-03-19T11:30:26.316 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.316 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.316 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.316 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:26.316 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:22.911396+0000", 2024-03-19T11:30:26.316 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.413e-06 2024-03-19T11:30:26.316 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.316 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.316 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:26.316 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:22.911993+0000", 2024-03-19T11:30:26.316 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00059690100000000005 2024-03-19T11:30:26.316 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.316 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.317 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:26.317 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:22.912028+0000", 2024-03-19T11:30:26.317 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4950000000000002e-05 2024-03-19T11:30:26.317 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.317 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.317 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:26.317 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:22.912044+0000", 2024-03-19T11:30:26.317 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5699e-05 2024-03-19T11:30:26.317 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.317 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.317 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:26.317 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:22.912133+0000", 2024-03-19T11:30:26.317 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9073999999999995e-05 2024-03-19T11:30:26.317 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.317 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.318 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:26.318 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:22.912166+0000", 2024-03-19T11:30:26.318 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3745000000000003e-05 2024-03-19T11:30:26.318 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.318 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.318 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:26.318 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:22.914628+0000", 2024-03-19T11:30:26.318 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002461614 2024-03-19T11:30:26.318 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.318 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.318 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:26.318 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:22.914698+0000", 2024-03-19T11:30:26.318 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.0114e-05 2024-03-19T11:30:26.318 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.318 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:26.319 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.319 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.319 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.319 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(osd.6.13:63 3.0 e47/20 3:053256fd:::smithi03829856-48 low_tier:head v 47'76, mlcod=47'74)", 2024-03-19T11:30:26.319 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.898311+0000", 2024-03-19T11:30:26.319 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.35693702500000002, 2024-03-19T11:30:26.319 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029412459999999998, 2024-03-19T11:30:26.319 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:26.319 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:26.319 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:26.319 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.319 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:26.319 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.898311+0000", 2024-03-19T11:30:26.319 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.319 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.320 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.320 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:26.320 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.898311+0000", 2024-03-19T11:30:26.320 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.320 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.320 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.320 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:26.320 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.898313+0000", 2024-03-19T11:30:26.320 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.03e-06 2024-03-19T11:30:26.320 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.321 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.321 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:26.321 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.898329+0000", 2024-03-19T11:30:26.321 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5542000000000001e-05 2024-03-19T11:30:26.321 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.321 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.321 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:26.321 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.898361+0000", 2024-03-19T11:30:26.321 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2153999999999997e-05 2024-03-19T11:30:26.321 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.321 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.321 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:26.321 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.898380+0000", 2024-03-19T11:30:26.322 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8972000000000001e-05 2024-03-19T11:30:26.322 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.322 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.322 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:26.322 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.898498+0000", 2024-03-19T11:30:26.322 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000118002 2024-03-19T11:30:26.322 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.322 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.322 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:26.322 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.898522+0000", 2024-03-19T11:30:26.322 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3557000000000002e-05 2024-03-19T11:30:26.322 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.322 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.322 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:26.322 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.901121+0000", 2024-03-19T11:30:26.323 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025989950000000002 2024-03-19T11:30:26.323 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.323 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.323 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:26.323 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.901253+0000", 2024-03-19T11:30:26.323 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000131994 2024-03-19T11:30:26.323 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.323 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:26.323 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.323 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.323 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.323 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(osd.6.13:64 3.0 e47/20 3:053256fd:::smithi03829856-48 low_tier:head v 47'77, mlcod=47'74)", 2024-03-19T11:30:26.323 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.898958+0000", 2024-03-19T11:30:26.323 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.35629058400000002, 2024-03-19T11:30:26.323 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0028234879999999999, 2024-03-19T11:30:26.324 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:26.324 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:26.324 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:26.324 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.324 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:26.324 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.898958+0000", 2024-03-19T11:30:26.324 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.324 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.324 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.325 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:26.325 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.898958+0000", 2024-03-19T11:30:26.325 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.325 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.325 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.325 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:26.325 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.898960+0000", 2024-03-19T11:30:26.325 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.193e-06 2024-03-19T11:30:26.325 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.325 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.325 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:26.325 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.898974+0000", 2024-03-19T11:30:26.325 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3735e-05 2024-03-19T11:30:26.325 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.325 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.326 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:26.326 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.899007+0000", 2024-03-19T11:30:26.326 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3608000000000001e-05 2024-03-19T11:30:26.326 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.326 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.326 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:26.326 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.899034+0000", 2024-03-19T11:30:26.326 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6373999999999999e-05 2024-03-19T11:30:26.326 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.326 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.326 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:26.326 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.899104+0000", 2024-03-19T11:30:26.326 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.0667999999999995e-05 2024-03-19T11:30:26.326 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.326 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.327 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:26.327 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.899125+0000", 2024-03-19T11:30:26.327 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0191999999999999e-05 2024-03-19T11:30:26.327 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.327 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.327 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:26.327 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.901680+0000", 2024-03-19T11:30:26.327 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025552830000000002 2024-03-19T11:30:26.327 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.327 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.327 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:26.327 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.901781+0000", 2024-03-19T11:30:26.327 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010143499999999999 2024-03-19T11:30:26.327 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.328 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:26.328 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.328 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.328 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.328 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:1414 3.0 e48/20 3:308d3303:::5a675e3a2c8b0683d62c457d46b2f98be18ec92014cc458a4e2237e707e8ab7b:head v 48'78, mlcod=47'76)", 2024-03-19T11:30:26.328 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.935090+0000", 2024-03-19T11:30:26.328 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.32015790599999999, 2024-03-19T11:30:26.328 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005407495, 2024-03-19T11:30:26.328 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:26.328 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:26.328 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:26.328 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.328 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:26.328 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.935090+0000", 2024-03-19T11:30:26.328 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.329 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.329 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.329 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:26.329 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.935090+0000", 2024-03-19T11:30:26.329 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.329 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.329 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.329 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:26.329 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.935092+0000", 2024-03-19T11:30:26.329 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.928e-06 2024-03-19T11:30:26.329 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.329 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.329 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:26.329 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.936094+0000", 2024-03-19T11:30:26.329 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001001776 2024-03-19T11:30:26.330 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.330 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.330 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:26.330 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.936119+0000", 2024-03-19T11:30:26.330 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4465999999999999e-05 2024-03-19T11:30:26.330 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.330 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.330 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:26.330 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.936130+0000", 2024-03-19T11:30:26.330 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1691e-05 2024-03-19T11:30:26.330 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.330 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.330 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:26.330 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.936220+0000", 2024-03-19T11:30:26.330 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9980000000000002e-05 2024-03-19T11:30:26.331 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.331 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.331 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:26.331 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.936253+0000", 2024-03-19T11:30:26.331 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2234999999999999e-05 2024-03-19T11:30:26.331 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.331 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.331 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:26.331 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.940434+0000", 2024-03-19T11:30:26.331 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.004181059 2024-03-19T11:30:26.331 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.331 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.331 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:26.331 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.940498+0000", 2024-03-19T11:30:26.331 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.436e-05 2024-03-19T11:30:26.332 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.332 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:26.332 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.332 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.332 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:26.332 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:26.333 INFO:tasks.rados.rados.0.smithi038.stdout:829: finishing set_chunk to oid 44 2024-03-19T11:30:26.333 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 44 set_chunk returned error code -95 offset: 448000 length: 1731072 2024-03-19T11:30:26.333 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 44 v 0 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-03-19T11:30:26.333 INFO:tasks.rados.rados.0.smithi038.stdout:830: finishing write tid 1 to smithi03829856-1 2024-03-19T11:30:26.333 INFO:tasks.rados.rados.0.smithi038.stdout:830: finishing write tid 2 to smithi03829856-1 2024-03-19T11:30:26.333 INFO:tasks.rados.rados.0.smithi038.stdout:831: expect (ObjNum 80 snap 6 seq_num 80) 2024-03-19T11:30:26.333 INFO:teuthology.orchestra.run.smithi038.stdout:moved 0 / 78 (0%) 2024-03-19T11:30:26.334 INFO:teuthology.orchestra.run.smithi038.stdout:avg 9.75 2024-03-19T11:30:26.334 INFO:teuthology.orchestra.run.smithi038.stdout:stddev 3.11247 -> 3.11247 (expected baseline 2.92083) 2024-03-19T11:30:26.334 INFO:teuthology.orchestra.run.smithi038.stdout:min osd.6 with 5 -> 5 pgs (0.512821 -> 0.512821 * mean) 2024-03-19T11:30:26.334 INFO:teuthology.orchestra.run.smithi038.stdout:max osd.1 with 15 -> 15 pgs (1.53846 -> 1.53846 * mean) 2024-03-19T11:30:26.334 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:30:26.334 INFO:teuthology.orchestra.run.smithi038.stdout:oload 1000 2024-03-19T11:30:26.334 INFO:teuthology.orchestra.run.smithi038.stdout:max_change 0.05 2024-03-19T11:30:26.334 INFO:teuthology.orchestra.run.smithi038.stdout:max_change_osds 4 2024-03-19T11:30:26.334 INFO:teuthology.orchestra.run.smithi038.stdout:average_utilization 0.0007 2024-03-19T11:30:26.334 INFO:teuthology.orchestra.run.smithi038.stdout:overload_utilization 0.0075 2024-03-19T11:30:26.334 INFO:teuthology.orchestra.run.smithi038.stderr:no change 2024-03-19T11:30:26.335 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-19T11:30:26.336 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-03-19T11:30:26.336 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 110 0.05 2024-03-19T11:30:26.337 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:26.343+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:26.394 INFO:tasks.rados.rados.0.smithi038.stdout:833: expect (ObjNum 12 snap 0 seq_num 12) 2024-03-19T11:30:26.442 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:26.451+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:26.461 INFO:tasks.rados.rados.0.smithi038.stdout:830: finishing write tid 3 to smithi03829856-1 2024-03-19T11:30:26.462 INFO:tasks.rados.rados.0.smithi038.stdout:830: finishing write tid 4 to smithi03829856-1 2024-03-19T11:30:26.462 INFO:tasks.rados.rados.0.smithi038.stdout:830: finishing write tid 5 to smithi03829856-1 2024-03-19T11:30:26.462 INFO:tasks.rados.rados.0.smithi038.stdout:830: oid 1 updating version 0 to 67 2024-03-19T11:30:26.462 INFO:tasks.rados.rados.0.smithi038.stdout:830: oid 1 updating version 67 to 69 2024-03-19T11:30:26.462 INFO:tasks.rados.rados.0.smithi038.stdout:830: oid 1 updating version 69 to 70 2024-03-19T11:30:26.462 INFO:tasks.rados.rados.0.smithi038.stdout:830: oid 1 version 70 is already newer than 68 2024-03-19T11:30:26.462 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 1 v 70 (ObjNum 105 snap 11 seq_num 105) dirty exists 2024-03-19T11:30:26.462 INFO:tasks.rados.rados.0.smithi038.stdout:830: left oid 1 (ObjNum 105 snap 11 seq_num 105) 2024-03-19T11:30:26.462 INFO:tasks.rados.rados.0.smithi038.stdout:837: expect (ObjNum 46 snap 0 seq_num 46) 2024-03-19T11:30:26.492 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:26.492 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:30:26.493 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:30:26.493 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:26.509 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-19T11:30:26.537 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:26.546+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:26.549 INFO:tasks.rados.rados.0.smithi038.stdout:836: expect (ObjNum 34 snap 0 seq_num 34) 2024-03-19T11:30:26.629 INFO:tasks.rados.rados.0.smithi038.stdout:840: expect (ObjNum 98 snap 10 seq_num 98) 2024-03-19T11:30:26.638 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:26.647+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:26.661 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:26.661 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:30:26.661 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:30:26.661 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:30:26.661 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:26.674 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-19T11:30:26.741 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:26.750+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:26.835 INFO:tasks.rados.rados.0.smithi038.stdout:834: finishing set_chunk to oid 7 2024-03-19T11:30:26.836 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 7 v 18 (ObjNum 67 snap 3 seq_num 67) dirty exists 2024-03-19T11:30:26.836 INFO:tasks.rados.rados.0.smithi038.stdout:841: finishing write tid 1 to smithi03829856-18 2024-03-19T11:30:26.836 INFO:tasks.rados.rados.0.smithi038.stdout:841: finishing write tid 2 to smithi03829856-18 2024-03-19T11:30:26.836 INFO:tasks.rados.rados.0.smithi038.stdout:841: finishing write tid 3 to smithi03829856-18 2024-03-19T11:30:26.836 INFO:tasks.rados.rados.0.smithi038.stdout:841: finishing write tid 4 to smithi03829856-18 2024-03-19T11:30:26.836 INFO:tasks.rados.rados.0.smithi038.stdout:841: finishing write tid 5 to smithi03829856-18 2024-03-19T11:30:26.836 INFO:tasks.rados.rados.0.smithi038.stdout:841: oid 18 updating version 0 to 73 2024-03-19T11:30:26.836 INFO:tasks.rados.rados.0.smithi038.stdout:841: oid 18 version 73 is already newer than 72 2024-03-19T11:30:26.836 INFO:tasks.rados.rados.0.smithi038.stdout:841: oid 18 updating version 73 to 74 2024-03-19T11:30:26.836 INFO:tasks.rados.rados.0.smithi038.stdout:841: oid 18 updating version 74 to 75 2024-03-19T11:30:26.836 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 18 v 75 (ObjNum 106 snap 11 seq_num 106) dirty exists 2024-03-19T11:30:26.836 INFO:tasks.rados.rados.0.smithi038.stdout:841: left oid 18 (ObjNum 106 snap 11 seq_num 106) 2024-03-19T11:30:26.836 INFO:tasks.rados.rados.0.smithi038.stdout:826: done (15 left) 2024-03-19T11:30:26.837 INFO:tasks.rados.rados.0.smithi038.stdout:827: done (14 left) 2024-03-19T11:30:26.837 INFO:tasks.rados.rados.0.smithi038.stdout:828: done (13 left) 2024-03-19T11:30:26.837 INFO:tasks.rados.rados.0.smithi038.stdout:829: done (12 left) 2024-03-19T11:30:26.837 INFO:tasks.rados.rados.0.smithi038.stdout:830: done (11 left) 2024-03-19T11:30:26.837 INFO:tasks.rados.rados.0.smithi038.stdout:831: done (10 left) 2024-03-19T11:30:26.837 INFO:tasks.rados.rados.0.smithi038.stdout:832: done (9 left) 2024-03-19T11:30:26.837 INFO:tasks.rados.rados.0.smithi038.stdout:833: done (8 left) 2024-03-19T11:30:26.837 INFO:tasks.rados.rados.0.smithi038.stdout:834: done (7 left) 2024-03-19T11:30:26.837 INFO:tasks.rados.rados.0.smithi038.stdout:836: done (6 left) 2024-03-19T11:30:26.837 INFO:tasks.rados.rados.0.smithi038.stdout:837: done (5 left) 2024-03-19T11:30:26.837 INFO:tasks.rados.rados.0.smithi038.stdout:840: done (4 left) 2024-03-19T11:30:26.837 INFO:tasks.rados.rados.0.smithi038.stdout:841: done (3 left) 2024-03-19T11:30:26.837 INFO:tasks.rados.rados.0.smithi038.stdout:842: set_chunk oid 48 target oid 2024-03-19T11:30:26.838 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:26.838 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:30:26.838 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:30:26.838 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:30:26.838 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.839 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:38 3.2 3:5595a68d:::07854d2fef297a06ba81685e660c332de36d5d18d546927d30daad6d7fda1541:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:26.839 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.307388+0000", 2024-03-19T11:30:26.839 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 17.535940399000001, 2024-03-19T11:30:26.839 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0080668630000000005, 2024-03-19T11:30:26.839 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:26.839 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:26.839 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:26.839 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.839 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:26.839 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.307388+0000", 2024-03-19T11:30:26.839 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.839 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.839 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.839 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:26.840 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.307388+0000", 2024-03-19T11:30:26.840 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.840 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.840 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.840 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:26.840 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.307391+0000", 2024-03-19T11:30:26.840 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3039999999999999e-06 2024-03-19T11:30:26.840 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.840 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.840 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:26.840 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308352+0000", 2024-03-19T11:30:26.840 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00096137600000000001 2024-03-19T11:30:26.840 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.840 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.840 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:26.841 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308376+0000", 2024-03-19T11:30:26.841 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4196999999999999e-05 2024-03-19T11:30:26.841 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.841 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.841 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:26.841 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308387+0000", 2024-03-19T11:30:26.841 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1068e-05 2024-03-19T11:30:26.841 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.841 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.841 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:26.841 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308539+0000", 2024-03-19T11:30:26.841 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015186600000000001 2024-03-19T11:30:26.841 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.841 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.842 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:26.842 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308731+0000", 2024-03-19T11:30:26.842 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019146199999999999 2024-03-19T11:30:26.842 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.842 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.842 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:26.842 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309028+0000", 2024-03-19T11:30:26.842 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00029751799999999998 2024-03-19T11:30:26.842 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.842 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.842 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:26.842 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.314461+0000", 2024-03-19T11:30:26.842 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0054329570000000004 2024-03-19T11:30:26.842 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.843 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.843 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:26.843 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.315339+0000", 2024-03-19T11:30:26.843 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00087765200000000003 2024-03-19T11:30:26.843 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.843 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.843 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:26.843 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.315400+0000", 2024-03-19T11:30:26.843 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1650999999999996e-05 2024-03-19T11:30:26.843 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.843 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.843 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:26.843 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.315455+0000", 2024-03-19T11:30:26.843 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4812000000000003e-05 2024-03-19T11:30:26.843 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.844 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:26.844 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.844 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.844 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.844 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:42 3.2 3:5aa75d1e:::b3c26d797de4ffcce4e862009e6b8eb8576a53cb86ea102ce51145c0038aeb97:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:26.844 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.308689+0000", 2024-03-19T11:30:26.844 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 17.534639744, 2024-03-19T11:30:26.844 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0079809319999999996, 2024-03-19T11:30:26.844 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:26.844 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:26.844 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:26.844 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.844 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:26.845 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308689+0000", 2024-03-19T11:30:26.845 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.845 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.845 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.845 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:26.845 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308689+0000", 2024-03-19T11:30:26.846 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.846 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.846 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.846 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:26.846 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308690+0000", 2024-03-19T11:30:26.846 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.003e-06 2024-03-19T11:30:26.846 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.846 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.846 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:26.846 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309612+0000", 2024-03-19T11:30:26.846 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00092193199999999998 2024-03-19T11:30:26.846 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.846 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.846 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:26.847 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309645+0000", 2024-03-19T11:30:26.847 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2554e-05 2024-03-19T11:30:26.847 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.847 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.847 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:26.847 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309654+0000", 2024-03-19T11:30:26.847 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6209999999999999e-06 2024-03-19T11:30:26.847 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.847 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.847 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:26.847 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.311778+0000", 2024-03-19T11:30:26.847 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002123897 2024-03-19T11:30:26.847 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.847 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.848 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:26.848 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.311913+0000", 2024-03-19T11:30:26.848 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013471199999999999 2024-03-19T11:30:26.848 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.848 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.848 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:26.848 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.312123+0000", 2024-03-19T11:30:26.848 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00021064099999999999 2024-03-19T11:30:26.848 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.848 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.848 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:26.848 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.315479+0000", 2024-03-19T11:30:26.848 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033557230000000001 2024-03-19T11:30:26.848 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.849 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.849 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:26.849 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.316574+0000", 2024-03-19T11:30:26.849 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0010944990000000001 2024-03-19T11:30:26.849 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.849 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.849 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:26.849 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.316629+0000", 2024-03-19T11:30:26.849 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5169e-05 2024-03-19T11:30:26.849 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.849 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.849 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:26.849 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.316670+0000", 2024-03-19T11:30:26.849 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1180999999999998e-05 2024-03-19T11:30:26.849 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.850 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:26.850 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.850 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.850 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.850 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:44 3.2 3:5f1d333e:::bfcdf551f6f2543327e9c601a4377e9a5ef396f5b97ae8331892d1bcf6b55a9d:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:26.850 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.310172+0000", 2024-03-19T11:30:26.850 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 17.533156276, 2024-03-19T11:30:26.850 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0086762909999999992, 2024-03-19T11:30:26.850 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:26.850 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:26.850 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:26.850 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.850 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:26.850 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310172+0000", 2024-03-19T11:30:26.851 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.851 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.851 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.851 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:26.851 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310172+0000", 2024-03-19T11:30:26.851 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.851 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.851 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.851 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:26.851 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310173+0000", 2024-03-19T11:30:26.851 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0500000000000002e-07 2024-03-19T11:30:26.851 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.851 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.851 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:26.851 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310242+0000", 2024-03-19T11:30:26.852 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8239999999999997e-05 2024-03-19T11:30:26.852 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.852 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.852 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:26.852 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310253+0000", 2024-03-19T11:30:26.852 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0879e-05 2024-03-19T11:30:26.852 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.852 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.852 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:26.852 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310263+0000", 2024-03-19T11:30:26.852 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0045e-05 2024-03-19T11:30:26.852 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.852 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.852 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:26.853 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.314507+0000", 2024-03-19T11:30:26.853 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0042442060000000004 2024-03-19T11:30:26.853 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.853 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.853 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:26.853 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.314588+0000", 2024-03-19T11:30:26.853 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0912000000000001e-05 2024-03-19T11:30:26.853 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.853 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.853 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:26.853 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.314736+0000", 2024-03-19T11:30:26.853 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014782099999999999 2024-03-19T11:30:26.853 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.853 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.854 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:26.854 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.316529+0000", 2024-03-19T11:30:26.854 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00179327 2024-03-19T11:30:26.854 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.854 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.854 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:26.854 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.318785+0000", 2024-03-19T11:30:26.854 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022563660000000001 2024-03-19T11:30:26.854 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.854 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.854 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:26.854 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.318813+0000", 2024-03-19T11:30:26.854 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8206000000000001e-05 2024-03-19T11:30:26.854 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.855 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.855 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:26.855 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.318849+0000", 2024-03-19T11:30:26.855 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5441000000000002e-05 2024-03-19T11:30:26.855 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.855 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:26.855 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.855 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.855 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.855 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:45 3.2 3:78b4b14e:::e1ef0c0c594c36c534f3829406e07ffb60d57b70360f62977e0d545d8ca2bf18:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:26.855 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.310274+0000", 2024-03-19T11:30:26.855 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 17.533054821, 2024-03-19T11:30:26.855 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0094234769999999995, 2024-03-19T11:30:26.855 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:26.855 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:26.856 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:26.856 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.856 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:26.856 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310274+0000", 2024-03-19T11:30:26.856 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.856 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.856 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.856 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:26.856 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310274+0000", 2024-03-19T11:30:26.856 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.856 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.856 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.856 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:26.856 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310275+0000", 2024-03-19T11:30:26.856 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2099999999999996e-07 2024-03-19T11:30:26.857 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.857 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.857 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:26.857 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310329+0000", 2024-03-19T11:30:26.857 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4440000000000001e-05 2024-03-19T11:30:26.857 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.857 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.857 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:26.857 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310340+0000", 2024-03-19T11:30:26.857 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1321999999999999e-05 2024-03-19T11:30:26.857 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.857 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.857 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:26.857 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.310349+0000", 2024-03-19T11:30:26.857 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6789999999999994e-06 2024-03-19T11:30:26.858 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.858 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.858 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:26.858 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.315512+0000", 2024-03-19T11:30:26.858 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0051629689999999999 2024-03-19T11:30:26.858 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.858 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.858 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:26.858 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.315581+0000", 2024-03-19T11:30:26.858 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9400999999999995e-05 2024-03-19T11:30:26.858 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.858 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.858 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:26.858 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.315759+0000", 2024-03-19T11:30:26.859 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000177491 2024-03-19T11:30:26.859 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.859 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.859 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:26.859 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.317063+0000", 2024-03-19T11:30:26.859 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0013040630000000001 2024-03-19T11:30:26.859 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.859 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.859 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:26.859 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.319635+0000", 2024-03-19T11:30:26.859 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025715880000000001 2024-03-19T11:30:26.859 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.859 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.859 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:26.859 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.319663+0000", 2024-03-19T11:30:26.860 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8554000000000001e-05 2024-03-19T11:30:26.860 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.860 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.860 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:26.860 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.319697+0000", 2024-03-19T11:30:26.860 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4348999999999999e-05 2024-03-19T11:30:26.860 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.860 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:26.860 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.860 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.860 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.860 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:55 3.2 3:525f009b:::a57e3dbafedf1ec1a7132467f167cfc4a4343cb541b36fc481376f0086aece71:head [read 0~121882 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:26.860 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.761904+0000", 2024-03-19T11:30:26.860 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 14.081424325, 2024-03-19T11:30:26.860 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010829357, 2024-03-19T11:30:26.861 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:26.861 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:26.861 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:26.861 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.861 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:26.861 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761904+0000", 2024-03-19T11:30:26.861 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.861 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.861 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.861 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:26.861 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761904+0000", 2024-03-19T11:30:26.861 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.861 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.861 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.861 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:26.862 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761905+0000", 2024-03-19T11:30:26.862 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4900000000000005e-07 2024-03-19T11:30:26.862 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.862 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.862 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:26.862 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761917+0000", 2024-03-19T11:30:26.862 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1806e-05 2024-03-19T11:30:26.862 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.862 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.862 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:26.862 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761934+0000", 2024-03-19T11:30:26.862 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6509000000000001e-05 2024-03-19T11:30:26.862 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.862 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.863 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:26.863 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761941+0000", 2024-03-19T11:30:26.863 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8110000000000003e-06 2024-03-19T11:30:26.863 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.863 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.863 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:26.863 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762115+0000", 2024-03-19T11:30:26.863 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017358200000000001 2024-03-19T11:30:26.863 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.863 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.863 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:26.863 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762120+0000", 2024-03-19T11:30:26.863 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3630000000000002e-06 2024-03-19T11:30:26.863 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.864 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.864 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:26.864 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.765567+0000", 2024-03-19T11:30:26.864 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034469990000000001 2024-03-19T11:30:26.864 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.864 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.864 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:26.864 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.765665+0000", 2024-03-19T11:30:26.864 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7476e-05 2024-03-19T11:30:26.864 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.864 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.864 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:26.864 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.772734+0000", 2024-03-19T11:30:26.864 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0070690620000000001 2024-03-19T11:30:26.864 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.865 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:26.865 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.865 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.865 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.865 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:56 3.2 3:5aa75d1e:::b3c26d797de4ffcce4e862009e6b8eb8576a53cb86ea102ce51145c0038aeb97:head [read 0~524288 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:26.865 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.761958+0000", 2024-03-19T11:30:26.865 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 14.081370924, 2024-03-19T11:30:26.865 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012078836000000001, 2024-03-19T11:30:26.865 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:26.865 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:26.865 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:26.865 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.865 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:26.865 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761958+0000", 2024-03-19T11:30:26.866 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.866 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.866 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.866 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:26.866 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761958+0000", 2024-03-19T11:30:26.866 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.866 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.866 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.866 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:26.866 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761958+0000", 2024-03-19T11:30:26.866 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9900000000000001e-07 2024-03-19T11:30:26.866 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.866 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.866 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:26.867 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761961+0000", 2024-03-19T11:30:26.867 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9280000000000002e-06 2024-03-19T11:30:26.867 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.867 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.867 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:26.867 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761969+0000", 2024-03-19T11:30:26.867 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1440000000000003e-06 2024-03-19T11:30:26.867 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.867 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.867 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:26.867 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761976+0000", 2024-03-19T11:30:26.867 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1020000000000002e-06 2024-03-19T11:30:26.867 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.867 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.867 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:26.868 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762144+0000", 2024-03-19T11:30:26.868 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016818200000000001 2024-03-19T11:30:26.868 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.868 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.868 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:26.868 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762148+0000", 2024-03-19T11:30:26.868 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0969999999999999e-06 2024-03-19T11:30:26.868 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.868 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.868 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:26.868 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.772788+0000", 2024-03-19T11:30:26.868 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010640124000000001 2024-03-19T11:30:26.868 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.868 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.868 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:26.869 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.772898+0000", 2024-03-19T11:30:26.869 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000110512 2024-03-19T11:30:26.869 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.869 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.869 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:26.869 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.774037+0000", 2024-03-19T11:30:26.869 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011382479999999999 2024-03-19T11:30:26.869 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.869 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:26.869 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.869 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.869 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.869 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:57 3.2 3:6d82e5d5:::455c613336f373ae48b33af990f91f317265a0c5a920cc1b46533fdb41d8319c:head [read 0~380357 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:26.869 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.761987+0000", 2024-03-19T11:30:26.869 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 14.081341962, 2024-03-19T11:30:26.870 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012691991999999999, 2024-03-19T11:30:26.870 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:26.870 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:26.870 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:26.870 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.870 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:26.870 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761987+0000", 2024-03-19T11:30:26.870 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.870 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.870 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.870 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:26.870 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761987+0000", 2024-03-19T11:30:26.870 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.870 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.871 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.871 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:26.871 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761987+0000", 2024-03-19T11:30:26.871 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4099999999999999e-07 2024-03-19T11:30:26.871 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.871 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.871 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:26.871 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761990+0000", 2024-03-19T11:30:26.871 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3829999999999999e-06 2024-03-19T11:30:26.871 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.871 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.871 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:26.871 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762000+0000", 2024-03-19T11:30:26.871 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0229e-05 2024-03-19T11:30:26.872 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.872 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.872 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:26.872 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762006+0000", 2024-03-19T11:30:26.872 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7069999999999997e-06 2024-03-19T11:30:26.872 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.872 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.872 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:26.872 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762171+0000", 2024-03-19T11:30:26.872 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000165341 2024-03-19T11:30:26.872 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.872 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.872 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:26.872 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762176+0000", 2024-03-19T11:30:26.873 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6879999999999998e-06 2024-03-19T11:30:26.873 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.873 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.873 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:26.873 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.774081+0000", 2024-03-19T11:30:26.873 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011905094999999999 2024-03-19T11:30:26.873 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.873 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.873 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:26.873 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.774169+0000", 2024-03-19T11:30:26.873 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8089000000000003e-05 2024-03-19T11:30:26.873 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.873 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.873 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:26.873 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.774679+0000", 2024-03-19T11:30:26.874 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00051001900000000001 2024-03-19T11:30:26.874 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.874 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:26.874 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.874 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.874 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.874 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:58 3.2 3:5f1d333e:::bfcdf551f6f2543327e9c601a4377e9a5ef396f5b97ae8331892d1bcf6b55a9d:head [read 0~161292 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:26.874 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.762024+0000", 2024-03-19T11:30:26.874 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 14.081305013, 2024-03-19T11:30:26.874 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013495629, 2024-03-19T11:30:26.874 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:26.874 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:26.874 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:26.874 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.875 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:26.875 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762024+0000", 2024-03-19T11:30:26.875 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.875 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.875 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.875 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:26.875 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762024+0000", 2024-03-19T11:30:26.875 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.875 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.875 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.875 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:26.875 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762024+0000", 2024-03-19T11:30:26.875 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9999999999999997e-07 2024-03-19T11:30:26.875 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.876 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.876 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:26.876 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762027+0000", 2024-03-19T11:30:26.876 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0410000000000002e-06 2024-03-19T11:30:26.876 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.876 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.876 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:26.876 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762037+0000", 2024-03-19T11:30:26.876 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1020000000000003e-06 2024-03-19T11:30:26.876 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.876 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.876 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:26.876 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762043+0000", 2024-03-19T11:30:26.876 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4409999999999998e-06 2024-03-19T11:30:26.876 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.877 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.877 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:26.877 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762198+0000", 2024-03-19T11:30:26.877 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015540000000000001 2024-03-19T11:30:26.877 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.877 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.877 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:26.877 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762203+0000", 2024-03-19T11:30:26.877 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5109999999999997e-06 2024-03-19T11:30:26.877 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.877 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.877 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:26.877 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.774710+0000", 2024-03-19T11:30:26.877 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012507452 2024-03-19T11:30:26.878 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.878 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.878 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:26.878 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.774754+0000", 2024-03-19T11:30:26.878 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3869999999999998e-05 2024-03-19T11:30:26.878 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.878 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.878 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:26.878 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.775519+0000", 2024-03-19T11:30:26.878 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00076521200000000003 2024-03-19T11:30:26.878 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.878 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:26.878 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.878 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.879 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.879 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:59 3.2 3:78b4b14e:::e1ef0c0c594c36c534f3829406e07ffb60d57b70360f62977e0d545d8ca2bf18:head [read 0~149977 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:26.879 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.762054+0000", 2024-03-19T11:30:26.879 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 14.081274582000001, 2024-03-19T11:30:26.879 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015354483, 2024-03-19T11:30:26.879 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:26.879 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:26.879 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:26.879 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.879 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:26.879 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762054+0000", 2024-03-19T11:30:26.879 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.879 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.879 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.879 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:26.880 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762054+0000", 2024-03-19T11:30:26.880 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.880 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.880 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.880 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:26.880 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762055+0000", 2024-03-19T11:30:26.880 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.32e-07 2024-03-19T11:30:26.880 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.880 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.880 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:26.880 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762058+0000", 2024-03-19T11:30:26.880 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9189999999999999e-06 2024-03-19T11:30:26.880 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.880 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.881 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:26.881 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762065+0000", 2024-03-19T11:30:26.881 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2409999999999998e-06 2024-03-19T11:30:26.881 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.881 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.881 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:26.881 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762070+0000", 2024-03-19T11:30:26.881 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6910000000000002e-06 2024-03-19T11:30:26.881 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.881 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.881 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:26.881 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762227+0000", 2024-03-19T11:30:26.881 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015606999999999999 2024-03-19T11:30:26.881 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.881 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.882 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:26.882 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762232+0000", 2024-03-19T11:30:26.882 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3029999999999999e-06 2024-03-19T11:30:26.882 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.882 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.882 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:26.882 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.776984+0000", 2024-03-19T11:30:26.882 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014752487999999999 2024-03-19T11:30:26.882 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.882 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.882 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:26.882 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.777030+0000", 2024-03-19T11:30:26.882 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5359999999999999e-05 2024-03-19T11:30:26.882 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.882 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.883 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:26.883 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.777409+0000", 2024-03-19T11:30:26.883 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037897899999999999 2024-03-19T11:30:26.883 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.883 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:26.883 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.883 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.883 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.883 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1067 4.0 4:01018511:::smithi03829856-19:head [write 477332~766101] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:26.883 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.950506+0000", 2024-03-19T11:30:26.883 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 12.89282287, 2024-03-19T11:30:26.883 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016279426, 2024-03-19T11:30:26.883 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:26.883 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:26.883 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:26.884 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:26.884 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:26.884 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1067 2024-03-19T11:30:26.884 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.884 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:26.884 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.884 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:26.884 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950506+0000", 2024-03-19T11:30:26.884 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.884 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.884 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.884 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:26.884 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950506+0000", 2024-03-19T11:30:26.884 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.884 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.885 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.885 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:26.885 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950509+0000", 2024-03-19T11:30:26.885 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.021e-06 2024-03-19T11:30:26.885 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.885 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.885 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:26.885 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950892+0000", 2024-03-19T11:30:26.885 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00038263300000000001 2024-03-19T11:30:26.885 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.885 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.885 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:26.885 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950912+0000", 2024-03-19T11:30:26.885 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0539000000000001e-05 2024-03-19T11:30:26.885 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.886 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.886 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:26.886 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950922+0000", 2024-03-19T11:30:26.886 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.577e-06 2024-03-19T11:30:26.886 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.886 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.886 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:26.886 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950982+0000", 2024-03-19T11:30:26.886 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0760000000000001e-05 2024-03-19T11:30:26.886 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.886 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.886 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:26.886 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.951058+0000", 2024-03-19T11:30:26.886 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5078000000000002e-05 2024-03-19T11:30:26.887 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.887 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.887 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:26.887 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.951173+0000", 2024-03-19T11:30:26.887 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011516200000000001 2024-03-19T11:30:26.887 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.887 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.887 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:26.887 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.955064+0000", 2024-03-19T11:30:26.887 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003890896 2024-03-19T11:30:26.887 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.887 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.887 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:26.887 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966670+0000", 2024-03-19T11:30:26.887 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011606795 2024-03-19T11:30:26.888 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.888 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.888 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:26.888 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966731+0000", 2024-03-19T11:30:26.888 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0528000000000001e-05 2024-03-19T11:30:26.888 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.888 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.888 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:26.888 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966785+0000", 2024-03-19T11:30:26.888 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4437000000000001e-05 2024-03-19T11:30:26.888 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.888 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:26.888 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.888 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.888 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.889 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1068 4.0 4:01018511:::smithi03829856-19:head [write 1681102~474010] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:26.889 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.956456+0000", 2024-03-19T11:30:26.889 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 12.886872305000001, 2024-03-19T11:30:26.889 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010432765, 2024-03-19T11:30:26.889 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:26.889 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:26.889 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:26.889 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:26.889 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:26.889 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1068 2024-03-19T11:30:26.889 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.889 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:26.889 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.889 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:26.889 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956456+0000", 2024-03-19T11:30:26.890 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.890 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.890 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.890 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:26.890 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956456+0000", 2024-03-19T11:30:26.890 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.890 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.890 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.890 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:26.890 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956458+0000", 2024-03-19T11:30:26.890 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.742e-06 2024-03-19T11:30:26.890 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.890 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.890 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:26.891 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956655+0000", 2024-03-19T11:30:26.891 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019682200000000001 2024-03-19T11:30:26.891 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.891 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.891 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:26.891 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956669+0000", 2024-03-19T11:30:26.891 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3978000000000001e-05 2024-03-19T11:30:26.891 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.891 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.891 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:26.891 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956678+0000", 2024-03-19T11:30:26.891 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5029999999999997e-06 2024-03-19T11:30:26.891 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.891 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.891 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:26.892 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956729+0000", 2024-03-19T11:30:26.892 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1765999999999999e-05 2024-03-19T11:30:26.892 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.892 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.892 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:26.892 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956795+0000", 2024-03-19T11:30:26.892 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5701e-05 2024-03-19T11:30:26.892 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.892 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.892 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:26.892 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956886+0000", 2024-03-19T11:30:26.892 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1226999999999999e-05 2024-03-19T11:30:26.892 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.893 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.893 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:26.893 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.958609+0000", 2024-03-19T11:30:26.893 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017227049999999999 2024-03-19T11:30:26.893 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.893 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.893 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:26.893 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966837+0000", 2024-03-19T11:30:26.893 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0082276439999999992 2024-03-19T11:30:26.893 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.894 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.894 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:26.894 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966866+0000", 2024-03-19T11:30:26.894 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9323999999999999e-05 2024-03-19T11:30:26.894 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.894 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.894 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:26.894 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966889+0000", 2024-03-19T11:30:26.894 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3353000000000002e-05 2024-03-19T11:30:26.894 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.894 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:26.894 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.894 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.894 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.895 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.0.13:50 3.2 3:5595a68d:::07854d2fef297a06ba81685e660c332de36d5d18d546927d30daad6d7fda1541:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:26.895 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.166341+0000", 2024-03-19T11:30:26.895 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.6769874389999995, 2024-03-19T11:30:26.895 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010067153000000001, 2024-03-19T11:30:26.895 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:26.895 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:26.895 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:26.895 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.895 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:26.895 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166341+0000", 2024-03-19T11:30:26.895 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.895 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.895 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.895 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:26.895 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166341+0000", 2024-03-19T11:30:26.896 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.896 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.896 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.896 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:26.896 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166344+0000", 2024-03-19T11:30:26.896 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1370000000000002e-06 2024-03-19T11:30:26.896 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.896 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.896 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:26.896 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166828+0000", 2024-03-19T11:30:26.896 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00048343600000000001 2024-03-19T11:30:26.896 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.896 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.896 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:26.896 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166857+0000", 2024-03-19T11:30:26.897 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8880000000000001e-05 2024-03-19T11:30:26.897 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.897 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.897 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:26.897 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166871+0000", 2024-03-19T11:30:26.897 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4384e-05 2024-03-19T11:30:26.897 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.897 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.897 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:26.897 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167222+0000", 2024-03-19T11:30:26.897 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00035059000000000002 2024-03-19T11:30:26.897 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.897 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.897 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:26.897 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167334+0000", 2024-03-19T11:30:26.898 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000112663 2024-03-19T11:30:26.898 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.898 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.898 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:26.898 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167533+0000", 2024-03-19T11:30:26.898 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019901399999999999 2024-03-19T11:30:26.898 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.898 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.898 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:26.898 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176237+0000", 2024-03-19T11:30:26.898 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0087039890000000005 2024-03-19T11:30:26.898 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.898 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.898 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:26.899 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176260+0000", 2024-03-19T11:30:26.899 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2889000000000001e-05 2024-03-19T11:30:26.899 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.899 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.899 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:26.899 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176343+0000", 2024-03-19T11:30:26.899 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2651000000000005e-05 2024-03-19T11:30:26.899 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.899 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.899 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:26.899 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176408+0000", 2024-03-19T11:30:26.899 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5519999999999996e-05 2024-03-19T11:30:26.899 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.899 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:26.899 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.900 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.900 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.900 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.0.13:51 3.2 3:587bed34:::13074eb9c7a477811d595d489af192576515e7c716c0b501b140fcc4687b768b:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:26.900 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.166912+0000", 2024-03-19T11:30:26.900 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.6764166360000008, 2024-03-19T11:30:26.900 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0097465639999999992, 2024-03-19T11:30:26.900 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:26.900 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:26.900 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:26.900 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.900 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:26.900 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166912+0000", 2024-03-19T11:30:26.900 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.900 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.900 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.901 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:26.901 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166912+0000", 2024-03-19T11:30:26.901 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.901 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.901 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.901 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:26.901 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166914+0000", 2024-03-19T11:30:26.901 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.468e-06 2024-03-19T11:30:26.901 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.901 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.901 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:26.901 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167000+0000", 2024-03-19T11:30:26.901 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6311999999999996e-05 2024-03-19T11:30:26.901 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.901 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.902 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:26.902 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167011+0000", 2024-03-19T11:30:26.902 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.153e-05 2024-03-19T11:30:26.902 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.902 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.902 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:26.902 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167019+0000", 2024-03-19T11:30:26.902 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7970000000000001e-06 2024-03-19T11:30:26.902 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.902 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.902 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:26.902 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167864+0000", 2024-03-19T11:30:26.902 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00084482600000000002 2024-03-19T11:30:26.902 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.903 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.903 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:26.903 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.169110+0000", 2024-03-19T11:30:26.903 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0012456399999999999 2024-03-19T11:30:26.903 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.903 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.903 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:26.903 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.169467+0000", 2024-03-19T11:30:26.903 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00035692100000000001 2024-03-19T11:30:26.903 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.903 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.903 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:26.903 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176426+0000", 2024-03-19T11:30:26.903 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0069589600000000001 2024-03-19T11:30:26.903 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.904 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.904 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:26.904 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176524+0000", 2024-03-19T11:30:26.904 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8183000000000005e-05 2024-03-19T11:30:26.904 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.904 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.904 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:26.904 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176603+0000", 2024-03-19T11:30:26.904 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9035999999999993e-05 2024-03-19T11:30:26.904 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.904 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.904 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:26.904 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176659+0000", 2024-03-19T11:30:26.904 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5890999999999997e-05 2024-03-19T11:30:26.905 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.905 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:26.905 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.905 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.905 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.905 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.0.13:52 3.2 3:46a26328:::b2b262672c1f15ae09b098a7a21c74662e4816397daed8ca08fb83d0cd3a0cac:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:26.905 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.167033+0000", 2024-03-19T11:30:26.905 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.6762958270000006, 2024-03-19T11:30:26.905 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010113297, 2024-03-19T11:30:26.905 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:26.905 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:26.905 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:26.905 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.905 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:26.905 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167033+0000", 2024-03-19T11:30:26.906 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.906 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.906 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.906 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:26.906 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167033+0000", 2024-03-19T11:30:26.906 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.906 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.906 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.906 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:26.906 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167034+0000", 2024-03-19T11:30:26.906 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6599999999999996e-07 2024-03-19T11:30:26.906 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.906 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.906 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:26.907 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167092+0000", 2024-03-19T11:30:26.907 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7989999999999999e-05 2024-03-19T11:30:26.907 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.907 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.907 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:26.907 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167100+0000", 2024-03-19T11:30:26.907 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9670000000000001e-06 2024-03-19T11:30:26.907 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.907 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.907 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:26.907 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167108+0000", 2024-03-19T11:30:26.907 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3559999999999993e-06 2024-03-19T11:30:26.907 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.907 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.907 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:26.908 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.170462+0000", 2024-03-19T11:30:26.908 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033540229999999998 2024-03-19T11:30:26.908 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.908 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.908 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:26.908 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.171193+0000", 2024-03-19T11:30:26.908 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00073086899999999998 2024-03-19T11:30:26.908 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.908 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.908 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:26.908 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.171558+0000", 2024-03-19T11:30:26.908 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00036518899999999998 2024-03-19T11:30:26.908 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.908 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.908 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:26.909 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176440+0000", 2024-03-19T11:30:26.909 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0048817260000000003 2024-03-19T11:30:26.909 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.909 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.909 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:26.909 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177053+0000", 2024-03-19T11:30:26.909 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00061337100000000003 2024-03-19T11:30:26.909 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.909 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.909 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:26.909 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177096+0000", 2024-03-19T11:30:26.909 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2846999999999998e-05 2024-03-19T11:30:26.909 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.909 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.909 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:26.910 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177146+0000", 2024-03-19T11:30:26.910 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0293000000000001e-05 2024-03-19T11:30:26.910 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.910 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:26.910 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.910 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.910 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.910 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.0.13:54 3.2 3:7b614c59:::a7a06fca755baa7858207cb4b067fffe8c25e8183433b950d6879f6969d41b0b:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:26.910 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.167117+0000", 2024-03-19T11:30:26.910 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.6762113190000001, 2024-03-19T11:30:26.910 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010796765999999999, 2024-03-19T11:30:26.910 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:26.910 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:26.910 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:26.911 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.911 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:26.911 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167117+0000", 2024-03-19T11:30:26.911 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.911 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.911 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.911 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:26.911 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167117+0000", 2024-03-19T11:30:26.911 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.911 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.911 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.911 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:26.911 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167118+0000", 2024-03-19T11:30:26.911 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2600000000000002e-07 2024-03-19T11:30:26.911 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.912 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.912 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:26.912 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167150+0000", 2024-03-19T11:30:26.912 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1576999999999999e-05 2024-03-19T11:30:26.912 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.912 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.912 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:26.912 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167157+0000", 2024-03-19T11:30:26.912 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2779999999999996e-06 2024-03-19T11:30:26.912 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.912 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.912 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:26.912 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167163+0000", 2024-03-19T11:30:26.912 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0120000000000002e-06 2024-03-19T11:30:26.912 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.913 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.913 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:26.913 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173324+0000", 2024-03-19T11:30:26.913 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0061614149999999999 2024-03-19T11:30:26.913 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.913 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.913 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:26.913 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173489+0000", 2024-03-19T11:30:26.913 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016438999999999999 2024-03-19T11:30:26.913 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.913 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.913 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:26.913 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173774+0000", 2024-03-19T11:30:26.913 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00028548000000000001 2024-03-19T11:30:26.913 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.914 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.914 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:26.914 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177668+0000", 2024-03-19T11:30:26.914 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038942450000000002 2024-03-19T11:30:26.914 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.914 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.914 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:26.914 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177792+0000", 2024-03-19T11:30:26.914 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012398199999999999 2024-03-19T11:30:26.914 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.914 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.914 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:26.914 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177861+0000", 2024-03-19T11:30:26.914 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8633999999999994e-05 2024-03-19T11:30:26.914 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.915 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.915 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:26.915 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177914+0000", 2024-03-19T11:30:26.915 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3226999999999997e-05 2024-03-19T11:30:26.915 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.915 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:26.915 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.915 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.915 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.915 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1230 4.0 4:0dc3b91e:::smithi03829856-28:head [write 662209~626546] snapc 9=[9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:26.915 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.170758+0000", 2024-03-19T11:30:26.915 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.6725706050000007, 2024-03-19T11:30:26.915 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011180066000000001, 2024-03-19T11:30:26.915 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:26.915 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:26.916 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:26.916 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:26.916 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:26.916 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1230 2024-03-19T11:30:26.916 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.916 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:26.916 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.916 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:26.916 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.170758+0000", 2024-03-19T11:30:26.916 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.916 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.916 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.916 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:26.916 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.170758+0000", 2024-03-19T11:30:26.917 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.917 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.917 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.917 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:26.917 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.170762+0000", 2024-03-19T11:30:26.917 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4960000000000001e-06 2024-03-19T11:30:26.917 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.917 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.917 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:26.917 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.171449+0000", 2024-03-19T11:30:26.917 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00068700499999999997 2024-03-19T11:30:26.917 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.917 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.917 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:26.917 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.171467+0000", 2024-03-19T11:30:26.918 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8389e-05 2024-03-19T11:30:26.918 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.918 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.918 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:26.918 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.171482+0000", 2024-03-19T11:30:26.918 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4619e-05 2024-03-19T11:30:26.918 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.918 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.918 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:26.918 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.171566+0000", 2024-03-19T11:30:26.918 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4788000000000002e-05 2024-03-19T11:30:26.918 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.918 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.918 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:26.919 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.171685+0000", 2024-03-19T11:30:26.919 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011861700000000001 2024-03-19T11:30:26.919 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.919 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.919 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:26.919 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.171835+0000", 2024-03-19T11:30:26.919 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014956299999999999 2024-03-19T11:30:26.919 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.919 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.919 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:26.919 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.181520+0000", 2024-03-19T11:30:26.919 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0096855750000000001 2024-03-19T11:30:26.919 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.919 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.919 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:26.920 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.181640+0000", 2024-03-19T11:30:26.920 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011966799999999999 2024-03-19T11:30:26.920 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.920 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.920 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:26.920 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.181811+0000", 2024-03-19T11:30:26.920 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017062699999999999 2024-03-19T11:30:26.920 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.920 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.920 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:26.920 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.181938+0000", 2024-03-19T11:30:26.920 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012771899999999999 2024-03-19T11:30:26.920 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.920 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:26.920 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.921 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.921 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.921 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1231 4.0 4:0dc3b91e:::smithi03829856-28:head [write 1808051~636013] snapc 9=[9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:26.921 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.173016+0000", 2024-03-19T11:30:26.921 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.6703127569999996, 2024-03-19T11:30:26.921 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013973719000000001, 2024-03-19T11:30:26.921 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:26.921 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:26.921 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:26.921 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:26.921 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:26.921 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1231 2024-03-19T11:30:26.921 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.921 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:26.921 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.922 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:26.922 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173016+0000", 2024-03-19T11:30:26.922 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.922 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.922 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.922 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:26.922 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173016+0000", 2024-03-19T11:30:26.922 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.922 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.922 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.922 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:26.922 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173019+0000", 2024-03-19T11:30:26.922 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.253e-06 2024-03-19T11:30:26.922 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.923 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.923 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:26.923 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176143+0000", 2024-03-19T11:30:26.923 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031240930000000001 2024-03-19T11:30:26.923 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.923 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.923 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:26.923 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176177+0000", 2024-03-19T11:30:26.923 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3237e-05 2024-03-19T11:30:26.923 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.923 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.923 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:26.923 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176196+0000", 2024-03-19T11:30:26.923 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9358e-05 2024-03-19T11:30:26.923 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.924 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.924 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:26.924 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176493+0000", 2024-03-19T11:30:26.924 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00029743899999999998 2024-03-19T11:30:26.924 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.924 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.924 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:26.924 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176786+0000", 2024-03-19T11:30:26.924 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00029259000000000002 2024-03-19T11:30:26.924 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.924 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.924 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:26.924 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177250+0000", 2024-03-19T11:30:26.924 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00046376000000000002 2024-03-19T11:30:26.924 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.925 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.925 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:26.925 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.182511+0000", 2024-03-19T11:30:26.925 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0052609019999999996 2024-03-19T11:30:26.925 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.925 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.925 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:26.925 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.186896+0000", 2024-03-19T11:30:26.925 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0043857009999999997 2024-03-19T11:30:26.925 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.925 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.925 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:26.925 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.186948+0000", 2024-03-19T11:30:26.925 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1653000000000002e-05 2024-03-19T11:30:26.925 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.926 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.926 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:26.926 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.186990+0000", 2024-03-19T11:30:26.926 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1733000000000002e-05 2024-03-19T11:30:26.926 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.926 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:26.926 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.926 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.926 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.926 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1277 4.0 4:01018511:::smithi03829856-19:head [write 763051~794393,stat] snapc b=[b,a,9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-03-19T11:30:26.926 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:21.953732+0000", 2024-03-19T11:30:26.926 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.8895965199999996, 2024-03-19T11:30:26.926 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011784380000000001, 2024-03-19T11:30:26.926 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:26.926 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:26.927 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:26.927 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:26.927 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:26.927 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1277 2024-03-19T11:30:26.927 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.927 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:26.927 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.927 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:26.927 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.953732+0000", 2024-03-19T11:30:26.927 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.927 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.927 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.927 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:26.927 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.953732+0000", 2024-03-19T11:30:26.927 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.928 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.928 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.928 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:26.928 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.953735+0000", 2024-03-19T11:30:26.928 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1999999999999999e-06 2024-03-19T11:30:26.928 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.928 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.928 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:26.928 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.954556+0000", 2024-03-19T11:30:26.928 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00082013599999999996 2024-03-19T11:30:26.928 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.928 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.928 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:26.928 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.954596+0000", 2024-03-19T11:30:26.928 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0741000000000002e-05 2024-03-19T11:30:26.929 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.929 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.929 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:26.929 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.954617+0000", 2024-03-19T11:30:26.929 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0375999999999999e-05 2024-03-19T11:30:26.929 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.929 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.929 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:26.929 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.954747+0000", 2024-03-19T11:30:26.929 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000130648 2024-03-19T11:30:26.929 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.929 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.929 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:26.929 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.954954+0000", 2024-03-19T11:30:26.930 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020694299999999999 2024-03-19T11:30:26.930 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.930 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.930 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:26.930 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.955252+0000", 2024-03-19T11:30:26.930 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00029740700000000002 2024-03-19T11:30:26.930 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.930 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.930 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:26.930 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.961677+0000", 2024-03-19T11:30:26.930 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0064253469999999997 2024-03-19T11:30:26.930 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.930 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.930 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:26.930 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.965373+0000", 2024-03-19T11:30:26.931 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0036962649999999998 2024-03-19T11:30:26.931 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.931 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.931 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:26.931 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.965451+0000", 2024-03-19T11:30:26.931 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7204000000000001e-05 2024-03-19T11:30:26.931 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.931 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.931 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:26.931 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.965517+0000", 2024-03-19T11:30:26.931 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6112999999999995e-05 2024-03-19T11:30:26.931 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.931 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:26.931 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.931 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.932 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.932 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1408 4.10 4:0dc3b91e:::smithi03829856-28:head [read 0~3462209,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:26.932 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.918220+0000", 2024-03-19T11:30:26.932 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.92510830200000005, 2024-03-19T11:30:26.932 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013199458000000001, 2024-03-19T11:30:26.932 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:26.932 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:26.932 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:26.932 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:26.932 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:26.932 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1408 2024-03-19T11:30:26.932 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.932 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:26.932 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.933 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:26.933 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918220+0000", 2024-03-19T11:30:26.933 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.933 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.933 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.933 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:26.933 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918220+0000", 2024-03-19T11:30:26.933 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.933 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.933 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.933 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:26.933 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918221+0000", 2024-03-19T11:30:26.933 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0999999999999999e-07 2024-03-19T11:30:26.933 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.933 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.934 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:26.934 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918225+0000", 2024-03-19T11:30:26.934 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2710000000000003e-06 2024-03-19T11:30:26.934 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.934 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.934 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:26.934 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918233+0000", 2024-03-19T11:30:26.934 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9489999999999996e-06 2024-03-19T11:30:26.934 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.934 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.934 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:26.934 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918239+0000", 2024-03-19T11:30:26.934 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5820000000000001e-06 2024-03-19T11:30:26.934 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.935 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.935 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:26.935 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.923634+0000", 2024-03-19T11:30:26.935 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005395703 2024-03-19T11:30:26.935 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.935 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.935 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:26.935 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.923786+0000", 2024-03-19T11:30:26.935 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015117900000000001 2024-03-19T11:30:26.935 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.935 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.935 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:26.935 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.931420+0000", 2024-03-19T11:30:26.935 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.007634264 2024-03-19T11:30:26.935 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.936 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:26.936 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.936 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.936 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.936 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1409 4.10 4:0dc3b91e:::smithi03829856-28:head [sparse-read 0~1851556] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:26.936 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.918248+0000", 2024-03-19T11:30:26.936 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.92508046799999999, 2024-03-19T11:30:26.936 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013726986, 2024-03-19T11:30:26.936 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:26.936 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:26.936 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:26.936 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:26.936 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:26.936 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1409 2024-03-19T11:30:26.936 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.937 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:26.937 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.937 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:26.937 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918248+0000", 2024-03-19T11:30:26.937 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.937 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.937 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.937 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:26.937 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918248+0000", 2024-03-19T11:30:26.937 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:26.937 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.937 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.937 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:26.937 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918249+0000", 2024-03-19T11:30:26.937 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4700000000000002e-07 2024-03-19T11:30:26.937 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.938 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.938 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:26.938 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918251+0000", 2024-03-19T11:30:26.938 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.542e-06 2024-03-19T11:30:26.938 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.938 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.938 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:26.938 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918258+0000", 2024-03-19T11:30:26.938 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4629999999999998e-06 2024-03-19T11:30:26.938 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.938 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.938 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:26.938 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918263+0000", 2024-03-19T11:30:26.938 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8049999999999997e-06 2024-03-19T11:30:26.939 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.939 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.939 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:26.939 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.931469+0000", 2024-03-19T11:30:26.939 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013206458000000001 2024-03-19T11:30:26.939 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.939 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.939 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:26.939 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.931586+0000", 2024-03-19T11:30:26.939 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000116655 2024-03-19T11:30:26.939 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:26.939 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:26.939 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:26.939 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.931975+0000", 2024-03-19T11:30:26.940 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00038961600000000002 2024-03-19T11:30:26.940 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.940 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:26.940 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.940 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:26.940 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:26.940 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:26.941 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:26.910+0000 7f923da1c640 -1 osd.7 48 handle_osd_map injecting map crc failure 2024-03-19T11:30:26.941 INFO:tasks.rados.rados.0.smithi038.stdout:842: set_chunk oid 48 offset: 752640 length: 339456 target oid bc8e9c01cb2bcb6a0700111676db3c3b844509371237983b5c0a544f4603e8ba offset: 0 snap 4 2024-03-19T11:30:26.941 INFO:tasks.rados.rados.0.smithi038.stdout:843: set_chunk oid 16 target oid 2024-03-19T11:30:26.942 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-19T11:30:26.943 INFO:tasks.rados.rados.0.smithi038.stdout:843: set_chunk oid 16 offset: 1180160 length: 859136 target oid d93831687a48e76c865eb26f9c54e5f4aca1a926fe5c356a9bc202c48a9e61ec offset: 0 snap -1 2024-03-19T11:30:26.943 INFO:tasks.rados.rados.0.smithi038.stdout:844: read oid 21 snap -1 2024-03-19T11:30:26.943 INFO:tasks.rados.rados.0.smithi038.stdout:844: expect (ObjNum 103 snap 10 seq_num 103) 2024-03-19T11:30:26.944 INFO:tasks.rados.rados.0.smithi038.stdout:845: set_chunk oid 15 target oid 2024-03-19T11:30:26.944 INFO:tasks.rados.rados.0.smithi038.stdout:845: set_chunk oid 15 offset: 1468928 length: 657920 target oid 8704546b0445ac2b641acf4bff7c91a6378e6600fc905a73fee7cd87d9f145ba offset: 0 snap 8 2024-03-19T11:30:26.944 INFO:tasks.rados.rados.0.smithi038.stdout:846: delete oid 46 current snap is 11 2024-03-19T11:30:26.944 INFO:tasks.rados.rados.0.smithi038.stdout:842: finishing set_chunk to oid 48 2024-03-19T11:30:26.944 INFO:tasks.rados.rados.0.smithi038.stdout:843: finishing set_chunk to oid 16 2024-03-19T11:30:26.944 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 16 v 32 (ObjNum 65 snap 3 seq_num 65) dirty exists 2024-03-19T11:30:26.944 INFO:tasks.rados.rados.0.smithi038.stdout:844: expect (ObjNum 103 snap 10 seq_num 103) 2024-03-19T11:30:26.944 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-03-19T11:30:26.944 INFO:teuthology.orchestra.run.smithi038.stdout:moved 3 / 78 (3.84615%) 2024-03-19T11:30:26.945 INFO:teuthology.orchestra.run.smithi038.stdout:avg 9.75 2024-03-19T11:30:26.945 INFO:teuthology.orchestra.run.smithi038.stdout:stddev 3.11247 -> 2.86138 (expected baseline 2.92083) 2024-03-19T11:30:26.945 INFO:teuthology.orchestra.run.smithi038.stdout:min osd.6 with 5 -> 7 pgs (0.512821 -> 0.717949 * mean) 2024-03-19T11:30:26.945 INFO:teuthology.orchestra.run.smithi038.stdout:max osd.1 with 15 -> 14 pgs (1.53846 -> 1.4359 * mean) 2024-03-19T11:30:26.945 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:30:26.945 INFO:teuthology.orchestra.run.smithi038.stdout:oload 110 2024-03-19T11:30:26.945 INFO:teuthology.orchestra.run.smithi038.stdout:max_change 0.05 2024-03-19T11:30:26.945 INFO:teuthology.orchestra.run.smithi038.stdout:max_change_osds 4 2024-03-19T11:30:26.945 INFO:teuthology.orchestra.run.smithi038.stdout:average_utilization 0.0007 2024-03-19T11:30:26.945 INFO:teuthology.orchestra.run.smithi038.stdout:overload_utilization 0.0008 2024-03-19T11:30:26.945 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4 weight 0.6495 -> 0.5995 2024-03-19T11:30:26.945 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1 weight 0.7791 -> 0.7291 2024-03-19T11:30:26.945 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0 weight 0.8696 -> 0.8196 2024-03-19T11:30:26.946 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:26.955+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:26.953 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-03-19T11:30:26.953 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 110 1.0 2024-03-19T11:30:27.044 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:27.054+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:27.058 INFO:tasks.rados.rados.0.smithi038.stdout:845: finishing set_chunk to oid 15 2024-03-19T11:30:27.058 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 15 set_chunk returned error code -95 offset: 1468928 length: 657920 2024-03-19T11:30:27.058 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-03-19T11:30:27.058 INFO:tasks.rados.rados.0.smithi038.stdout:842: done (7 left) 2024-03-19T11:30:27.058 INFO:tasks.rados.rados.0.smithi038.stdout:843: done (6 left) 2024-03-19T11:30:27.058 INFO:tasks.rados.rados.0.smithi038.stdout:844: done (5 left) 2024-03-19T11:30:27.058 INFO:tasks.rados.rados.0.smithi038.stdout:845: done (4 left) 2024-03-19T11:30:27.058 INFO:tasks.rados.rados.0.smithi038.stdout:846: done (3 left) 2024-03-19T11:30:27.058 INFO:tasks.rados.rados.0.smithi038.stdout:847: write oid 44 current snap is 11 2024-03-19T11:30:27.059 INFO:tasks.rados.rados.0.smithi038.stdout:847: seq_num 108 ranges {615156=555834,1603467=416361,2816389=693495} 2024-03-19T11:30:27.065 INFO:tasks.rados.rados.0.smithi038.stdout:847: writing smithi03829856-44 from 615156 to 1170990 tid 1 2024-03-19T11:30:27.069 INFO:tasks.rados.rados.0.smithi038.stdout:847: writing smithi03829856-44 from 1603467 to 2019828 tid 2 2024-03-19T11:30:27.078 INFO:tasks.rados.rados.0.smithi038.stdout:847: writing smithi03829856-44 from 2816389 to 3509884 tid 3 2024-03-19T11:30:27.078 INFO:tasks.rados.rados.0.smithi038.stdout:848: set_chunk oid 28 target oid 2024-03-19T11:30:27.081 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:27.081 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:27.081 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:30:27.081 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:27.096 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-19T11:30:27.145 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:27.155+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:27.247 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:27.259+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:27.270 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:27.270 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:27.270 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:30:27.270 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:30:27.270 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:27.283 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-19T11:30:27.348 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:27.358+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:27.413 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:27.414 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:30:27.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:30:27.414 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:30:27.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.414 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:9 4.d 4:b4ef97e0:::smithi03829856-3:head [write 433614~550498,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-03-19T11:30:27.414 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:04.706000+0000", 2024-03-19T11:30:27.414 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.717597755, 2024-03-19T11:30:27.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013227001, 2024-03-19T11:30:27.414 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:27.414 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:27.414 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:27.414 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:27.415 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:27.415 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 9 2024-03-19T11:30:27.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.415 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:27.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:27.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.706000+0000", 2024-03-19T11:30:27.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:27.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:27.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.706000+0000", 2024-03-19T11:30:27.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:27.416 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.416 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:27.416 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.706002+0000", 2024-03-19T11:30:27.416 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5569999999999998e-06 2024-03-19T11:30:27.416 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.416 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:27.416 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.707261+0000", 2024-03-19T11:30:27.416 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001258817 2024-03-19T11:30:27.416 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.417 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:27.417 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.707288+0000", 2024-03-19T11:30:27.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7093e-05 2024-03-19T11:30:27.417 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.417 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:27.417 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.707305+0000", 2024-03-19T11:30:27.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6330000000000001e-05 2024-03-19T11:30:27.417 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.417 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:27.417 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.707610+0000", 2024-03-19T11:30:27.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00030568799999999998 2024-03-19T11:30:27.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:27.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.707984+0000", 2024-03-19T11:30:27.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00037399299999999998 2024-03-19T11:30:27.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2", 2024-03-19T11:30:27.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.708409+0000", 2024-03-19T11:30:27.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00042503099999999998 2024-03-19T11:30:27.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:27.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.712919+0000", 2024-03-19T11:30:27.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0045099010000000002 2024-03-19T11:30:27.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:27.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.719037+0000", 2024-03-19T11:30:27.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061173770000000002 2024-03-19T11:30:27.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:27.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.719116+0000", 2024-03-19T11:30:27.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9935e-05 2024-03-19T11:30:27.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:27.421 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.719227+0000", 2024-03-19T11:30:27.421 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011027900000000001 2024-03-19T11:30:27.421 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:27.421 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:27.421 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:27.421 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.421 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.421 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:848 4.1 4:866f3dcc:::smithi03829856-31:head [read 0~2941192,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:27.421 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.757468+0000", 2024-03-19T11:30:27.421 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.666129763000001, 2024-03-19T11:30:27.421 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015020110999999999, 2024-03-19T11:30:27.422 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:27.422 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:27.422 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:27.422 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:27.422 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:27.422 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 848 2024-03-19T11:30:27.422 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.422 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:27.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.422 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:27.422 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757468+0000", 2024-03-19T11:30:27.422 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:27.422 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:27.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757468+0000", 2024-03-19T11:30:27.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:27.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:27.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757471+0000", 2024-03-19T11:30:27.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.67e-06 2024-03-19T11:30:27.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:27.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757488+0000", 2024-03-19T11:30:27.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6223e-05 2024-03-19T11:30:27.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:27.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757521+0000", 2024-03-19T11:30:27.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.358e-05 2024-03-19T11:30:27.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:27.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757543+0000", 2024-03-19T11:30:27.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1330000000000001e-05 2024-03-19T11:30:27.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:27.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757745+0000", 2024-03-19T11:30:27.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020259799999999999 2024-03-19T11:30:27.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op must wait for map", 2024-03-19T11:30:27.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757760+0000", 2024-03-19T11:30:27.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4647000000000001e-05 2024-03-19T11:30:27.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:27.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763972+0000", 2024-03-19T11:30:27.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062126480000000003 2024-03-19T11:30:27.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:27.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.764547+0000", 2024-03-19T11:30:27.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057440200000000001 2024-03-19T11:30:27.426 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.426 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:27.426 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.772488+0000", 2024-03-19T11:30:27.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079410130000000002 2024-03-19T11:30:27.426 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:27.426 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:27.426 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:27.426 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.426 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:849 4.1 4:866f3dcc:::smithi03829856-31:head [sparse-read 0~1359577] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:27.426 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.757583+0000", 2024-03-19T11:30:27.426 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.666014070999999, 2024-03-19T11:30:27.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016241202, 2024-03-19T11:30:27.427 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:27.427 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:27.427 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:27.427 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:27.427 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:27.427 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 849 2024-03-19T11:30:27.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.427 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:27.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.427 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:27.427 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757583+0000", 2024-03-19T11:30:27.427 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:27.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.427 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:27.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757583+0000", 2024-03-19T11:30:27.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:27.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.428 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:27.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757584+0000", 2024-03-19T11:30:27.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.077e-06 2024-03-19T11:30:27.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.428 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:27.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757593+0000", 2024-03-19T11:30:27.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3059999999999998e-06 2024-03-19T11:30:27.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.428 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:27.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757608+0000", 2024-03-19T11:30:27.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5526999999999999e-05 2024-03-19T11:30:27.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:27.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757618+0000", 2024-03-19T11:30:27.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5999999999999996e-06 2024-03-19T11:30:27.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:27.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757886+0000", 2024-03-19T11:30:27.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026839099999999999 2024-03-19T11:30:27.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:27.430 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757899+0000", 2024-03-19T11:30:27.430 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3004e-05 2024-03-19T11:30:27.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.430 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.430 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:27.430 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.772651+0000", 2024-03-19T11:30:27.430 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014751702 2024-03-19T11:30:27.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.430 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.430 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:27.430 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.772924+0000", 2024-03-19T11:30:27.430 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027314199999999998 2024-03-19T11:30:27.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.430 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:27.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.773825+0000", 2024-03-19T11:30:27.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00090045299999999995 2024-03-19T11:30:27.431 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:27.431 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:27.431 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:27.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.431 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:850 4.1 4:866f3dcc:::smithi03829856-31:head [read 0~2941192,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:27.431 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.757636+0000", 2024-03-19T11:30:27.431 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.665961401000001, 2024-03-19T11:30:27.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018834801000000002, 2024-03-19T11:30:27.431 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:27.431 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:27.432 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:27.432 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:27.432 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:27.432 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 850 2024-03-19T11:30:27.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.432 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:27.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.432 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:27.432 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757636+0000", 2024-03-19T11:30:27.432 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:27.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.432 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:27.432 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757636+0000", 2024-03-19T11:30:27.432 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:27.433 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.433 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:27.433 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757637+0000", 2024-03-19T11:30:27.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5999999999999991e-07 2024-03-19T11:30:27.433 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.433 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:27.433 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757643+0000", 2024-03-19T11:30:27.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3199999999999996e-06 2024-03-19T11:30:27.433 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.433 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:27.433 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757670+0000", 2024-03-19T11:30:27.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6682999999999998e-05 2024-03-19T11:30:27.434 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.434 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.434 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:27.434 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757679+0000", 2024-03-19T11:30:27.434 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.986e-06 2024-03-19T11:30:27.434 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.434 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.434 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:27.434 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757961+0000", 2024-03-19T11:30:27.434 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028230799999999999 2024-03-19T11:30:27.434 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.434 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.434 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:27.434 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757971+0000", 2024-03-19T11:30:27.435 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.38e-06 2024-03-19T11:30:27.435 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.435 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:27.435 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.774079+0000", 2024-03-19T11:30:27.435 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016108573000000001 2024-03-19T11:30:27.435 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.435 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:27.435 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.774306+0000", 2024-03-19T11:30:27.435 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022637499999999999 2024-03-19T11:30:27.435 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.435 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:27.435 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776471+0000", 2024-03-19T11:30:27.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021652160000000002 2024-03-19T11:30:27.436 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:27.436 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:27.436 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:27.436 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.436 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.436 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:851 4.9 4:953e720b:::smithi03829856-45:head [write 591613~454162,stat] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:27.436 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.763514+0000", 2024-03-19T11:30:27.436 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.66008321, 2024-03-19T11:30:27.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027825888999999999, 2024-03-19T11:30:27.436 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:27.436 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:27.436 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:27.436 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:27.436 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:27.437 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 851 2024-03-19T11:30:27.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.437 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:27.437 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.437 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:27.437 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763514+0000", 2024-03-19T11:30:27.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:27.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.437 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.437 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:27.437 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763514+0000", 2024-03-19T11:30:27.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:27.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.437 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.437 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:27.437 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763517+0000", 2024-03-19T11:30:27.438 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8499999999999998e-06 2024-03-19T11:30:27.438 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.438 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:27.438 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.764714+0000", 2024-03-19T11:30:27.438 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011971270000000001 2024-03-19T11:30:27.438 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.438 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:27.438 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.764745+0000", 2024-03-19T11:30:27.438 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0546999999999997e-05 2024-03-19T11:30:27.438 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.439 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:27.439 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.764767+0000", 2024-03-19T11:30:27.439 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2286999999999999e-05 2024-03-19T11:30:27.439 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.439 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.439 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:27.439 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.773924+0000", 2024-03-19T11:30:27.439 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0091567360000000004 2024-03-19T11:30:27.439 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.439 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.439 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:27.439 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.774217+0000", 2024-03-19T11:30:27.439 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029350099999999999 2024-03-19T11:30:27.439 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.439 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.440 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:27.440 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.774770+0000", 2024-03-19T11:30:27.440 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00055288600000000005 2024-03-19T11:30:27.440 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.440 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.440 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:27.440 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.786413+0000", 2024-03-19T11:30:27.440 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011643229999999999 2024-03-19T11:30:27.440 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.440 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.440 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:27.440 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791104+0000", 2024-03-19T11:30:27.440 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0046900830000000003 2024-03-19T11:30:27.440 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.440 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.441 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:27.441 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791248+0000", 2024-03-19T11:30:27.441 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014410200000000001 2024-03-19T11:30:27.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.441 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.441 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:27.441 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791340+0000", 2024-03-19T11:30:27.441 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2540000000000005e-05 2024-03-19T11:30:27.441 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:27.441 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:27.441 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:27.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.441 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.441 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:852 4.9 4:953e720b:::smithi03829856-45:head [write 1719614~417353] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:27.441 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.769707+0000", 2024-03-19T11:30:27.442 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.653890184000002, 2024-03-19T11:30:27.442 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022476643000000001, 2024-03-19T11:30:27.442 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:27.442 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:27.442 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:27.442 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:27.442 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:27.442 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 852 2024-03-19T11:30:27.442 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.442 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:27.442 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.442 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:27.442 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.769707+0000", 2024-03-19T11:30:27.442 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:27.443 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.443 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.443 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:27.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.769707+0000", 2024-03-19T11:30:27.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:27.443 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.443 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.443 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:27.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.769710+0000", 2024-03-19T11:30:27.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6230000000000001e-06 2024-03-19T11:30:27.443 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.443 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.443 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:27.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.770385+0000", 2024-03-19T11:30:27.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00067504900000000005 2024-03-19T11:30:27.444 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.444 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.444 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:27.444 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.770419+0000", 2024-03-19T11:30:27.444 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3528999999999997e-05 2024-03-19T11:30:27.444 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.444 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.444 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:27.444 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.770436+0000", 2024-03-19T11:30:27.444 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7615999999999999e-05 2024-03-19T11:30:27.444 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.444 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.444 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:27.444 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.781327+0000", 2024-03-19T11:30:27.444 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010891343 2024-03-19T11:30:27.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:27.445 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.781616+0000", 2024-03-19T11:30:27.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028857399999999999 2024-03-19T11:30:27.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:27.445 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.782125+0000", 2024-03-19T11:30:27.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00050941599999999995 2024-03-19T11:30:27.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.446 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:27.446 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.786606+0000", 2024-03-19T11:30:27.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044809560000000003 2024-03-19T11:30:27.446 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.446 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:27.446 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791995+0000", 2024-03-19T11:30:27.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053889150000000002 2024-03-19T11:30:27.446 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.446 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:27.446 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.792084+0000", 2024-03-19T11:30:27.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8633000000000003e-05 2024-03-19T11:30:27.447 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.447 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.447 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:27.447 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.792184+0000", 2024-03-19T11:30:27.447 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9988999999999994e-05 2024-03-19T11:30:27.447 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:27.447 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:27.447 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:27.447 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.447 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.447 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:853 4.9 4:953e720b:::smithi03829856-45:head [write 2570780~420833] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:27.447 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.775636+0000", 2024-03-19T11:30:27.447 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.647961562999999, 2024-03-19T11:30:27.447 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018795850999999999, 2024-03-19T11:30:27.448 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:27.448 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:27.448 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:27.448 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:27.448 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:27.448 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 853 2024-03-19T11:30:27.448 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.448 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:27.448 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.448 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:27.448 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.775636+0000", 2024-03-19T11:30:27.448 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:27.448 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.448 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.449 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:27.449 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.775636+0000", 2024-03-19T11:30:27.449 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:27.449 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.449 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.449 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:27.449 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.775639+0000", 2024-03-19T11:30:27.449 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4529999999999998e-06 2024-03-19T11:30:27.449 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.449 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.449 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:27.449 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776492+0000", 2024-03-19T11:30:27.449 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00085245000000000004 2024-03-19T11:30:27.449 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.449 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.449 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:27.450 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776526+0000", 2024-03-19T11:30:27.450 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4090000000000001e-05 2024-03-19T11:30:27.450 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.450 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.450 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:27.450 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776550+0000", 2024-03-19T11:30:27.450 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4139999999999999e-05 2024-03-19T11:30:27.450 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.450 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.450 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:27.450 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.786752+0000", 2024-03-19T11:30:27.450 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010201834 2024-03-19T11:30:27.450 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.450 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.451 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:27.451 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.786994+0000", 2024-03-19T11:30:27.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000241952 2024-03-19T11:30:27.451 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.451 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.451 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:27.451 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.787301+0000", 2024-03-19T11:30:27.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00030690699999999998 2024-03-19T11:30:27.451 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.451 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.451 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:27.451 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791965+0000", 2024-03-19T11:30:27.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0046644379999999999 2024-03-19T11:30:27.451 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.451 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.452 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:27.452 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.794165+0000", 2024-03-19T11:30:27.452 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021995389999999999 2024-03-19T11:30:27.452 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.452 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:27.452 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.794313+0000", 2024-03-19T11:30:27.452 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000148369 2024-03-19T11:30:27.452 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.452 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:27.452 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.794432+0000", 2024-03-19T11:30:27.452 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011867900000000001 2024-03-19T11:30:27.452 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:27.452 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:27.453 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:27.453 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.453 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.453 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:854 4.9 4:953e720b:::smithi03829856-45:head [setxattr _header (56) in=7b,truncate 2991613] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:27.453 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.776592+0000", 2024-03-19T11:30:27.453 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.647005840999999, 2024-03-19T11:30:27.453 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022815621000000001, 2024-03-19T11:30:27.453 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:27.453 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:27.453 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:27.453 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:27.453 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:27.453 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 854 2024-03-19T11:30:27.453 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.453 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:27.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:27.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776592+0000", 2024-03-19T11:30:27.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:27.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:27.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776592+0000", 2024-03-19T11:30:27.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:27.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:27.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776594+0000", 2024-03-19T11:30:27.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.95e-06 2024-03-19T11:30:27.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:27.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776603+0000", 2024-03-19T11:30:27.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2399999999999996e-06 2024-03-19T11:30:27.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:27.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776623+0000", 2024-03-19T11:30:27.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.035e-05 2024-03-19T11:30:27.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:27.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776639+0000", 2024-03-19T11:30:27.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5642999999999999e-05 2024-03-19T11:30:27.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.456 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:27.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.792324+0000", 2024-03-19T11:30:27.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015685075999999999 2024-03-19T11:30:27.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.456 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:27.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.792599+0000", 2024-03-19T11:30:27.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000275264 2024-03-19T11:30:27.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.456 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:27.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.793092+0000", 2024-03-19T11:30:27.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00049264600000000001 2024-03-19T11:30:27.457 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.457 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.457 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:27.457 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.794668+0000", 2024-03-19T11:30:27.457 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001576072 2024-03-19T11:30:27.457 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.457 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.457 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:27.457 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.799082+0000", 2024-03-19T11:30:27.457 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044145190000000004 2024-03-19T11:30:27.457 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.457 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.457 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:27.457 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.799249+0000", 2024-03-19T11:30:27.457 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016701199999999999 2024-03-19T11:30:27.458 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.458 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.458 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:27.458 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.799407+0000", 2024-03-19T11:30:27.458 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000157849 2024-03-19T11:30:27.458 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:27.458 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:27.458 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:27.458 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.458 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.458 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:855 4.9 4:953e720b:::smithi03829856-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:27.458 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.776660+0000", 2024-03-19T11:30:27.458 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.646937248, 2024-03-19T11:30:27.458 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023419786000000001, 2024-03-19T11:30:27.459 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:27.459 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:27.459 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:27.459 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:27.459 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:27.459 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 855 2024-03-19T11:30:27.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.459 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:27.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.459 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:27.459 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776660+0000", 2024-03-19T11:30:27.459 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:27.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.459 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:27.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776660+0000", 2024-03-19T11:30:27.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:27.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.460 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:27.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776662+0000", 2024-03-19T11:30:27.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3430000000000001e-06 2024-03-19T11:30:27.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.460 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:27.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776668+0000", 2024-03-19T11:30:27.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3500000000000002e-06 2024-03-19T11:30:27.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.460 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:27.461 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776687+0000", 2024-03-19T11:30:27.461 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9273999999999999e-05 2024-03-19T11:30:27.461 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.461 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:27.461 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776705+0000", 2024-03-19T11:30:27.461 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7839999999999999e-05 2024-03-19T11:30:27.461 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.461 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:27.461 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.793874+0000", 2024-03-19T11:30:27.461 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017168597000000001 2024-03-19T11:30:27.461 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.461 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-03-19T11:30:27.461 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.794132+0000", 2024-03-19T11:30:27.462 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025790399999999999 2024-03-19T11:30:27.462 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.462 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.462 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:27.462 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.799504+0000", 2024-03-19T11:30:27.462 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053724159999999997 2024-03-19T11:30:27.462 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.462 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.462 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:27.462 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.799782+0000", 2024-03-19T11:30:27.462 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027785499999999997 2024-03-19T11:30:27.462 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.462 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.462 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:27.463 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.800080+0000", 2024-03-19T11:30:27.463 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029820699999999999 2024-03-19T11:30:27.463 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:27.463 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:27.463 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:27.463 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.463 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.463 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:871 4.d 4:b4ef97e0:::smithi03829856-3:head [write 642034~796262,stat] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:27.463 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:09.028192+0000", 2024-03-19T11:30:27.463 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.395405517, 2024-03-19T11:30:27.463 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013367027, 2024-03-19T11:30:27.463 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:27.463 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:27.463 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:27.466 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:27.466 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:27.466 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 871 2024-03-19T11:30:27.466 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.466 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:27.466 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.466 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:27.467 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.028192+0000", 2024-03-19T11:30:27.467 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:27.467 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.467 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.467 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:27.467 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.028192+0000", 2024-03-19T11:30:27.467 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:27.467 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.467 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.467 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:27.467 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.028194+0000", 2024-03-19T11:30:27.467 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3999999999999999e-06 2024-03-19T11:30:27.467 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.467 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.468 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:27.468 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.029358+0000", 2024-03-19T11:30:27.468 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011635549999999999 2024-03-19T11:30:27.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.468 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:27.468 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.029390+0000", 2024-03-19T11:30:27.468 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2215999999999998e-05 2024-03-19T11:30:27.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.469 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.469 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:27.469 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.029406+0000", 2024-03-19T11:30:27.469 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5543e-05 2024-03-19T11:30:27.469 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.469 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.469 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:27.470 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.029525+0000", 2024-03-19T11:30:27.470 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000119269 2024-03-19T11:30:27.470 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.470 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.470 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:27.470 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.029676+0000", 2024-03-19T11:30:27.470 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000150729 2024-03-19T11:30:27.470 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.470 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.470 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2", 2024-03-19T11:30:27.470 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.029994+0000", 2024-03-19T11:30:27.470 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00031876099999999998 2024-03-19T11:30:27.470 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.470 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.471 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:27.471 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.040441+0000", 2024-03-19T11:30:27.471 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010446446 2024-03-19T11:30:27.471 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.471 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.471 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:27.471 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.041327+0000", 2024-03-19T11:30:27.471 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00088595299999999998 2024-03-19T11:30:27.471 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.471 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.471 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:27.471 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.041409+0000", 2024-03-19T11:30:27.471 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2332999999999999e-05 2024-03-19T11:30:27.472 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.472 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.472 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:27.472 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.041559+0000", 2024-03-19T11:30:27.472 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000149822 2024-03-19T11:30:27.472 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:27.472 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:27.472 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:27.472 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.472 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.472 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:993 4.e 4:72163e4c:::smithi03829856-39:head [write 776490~714607] snapc 5=[5,2] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-03-19T11:30:27.472 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:12.362932+0000", 2024-03-19T11:30:27.472 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.060665824000001, 2024-03-19T11:30:27.472 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018349900999999998, 2024-03-19T11:30:27.473 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:27.473 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:27.473 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:27.473 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:27.473 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:27.473 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 993 2024-03-19T11:30:27.473 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.473 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:27.473 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.473 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:27.473 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.362932+0000", 2024-03-19T11:30:27.473 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:27.473 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.473 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.473 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:27.474 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.362932+0000", 2024-03-19T11:30:27.474 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:27.474 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.474 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.474 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:27.474 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.362935+0000", 2024-03-19T11:30:27.474 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.88e-06 2024-03-19T11:30:27.474 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.474 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.474 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:27.474 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.366756+0000", 2024-03-19T11:30:27.474 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003821217 2024-03-19T11:30:27.474 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.475 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.475 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:27.475 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.366796+0000", 2024-03-19T11:30:27.475 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9866000000000001e-05 2024-03-19T11:30:27.475 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.475 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.475 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:27.475 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.366813+0000", 2024-03-19T11:30:27.475 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.755e-05 2024-03-19T11:30:27.475 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.475 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.475 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:27.475 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.366993+0000", 2024-03-19T11:30:27.475 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000179605 2024-03-19T11:30:27.475 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.476 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.476 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:27.476 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.367167+0000", 2024-03-19T11:30:27.476 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017421200000000001 2024-03-19T11:30:27.476 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.476 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.476 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-19T11:30:27.476 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.367505+0000", 2024-03-19T11:30:27.476 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033800099999999999 2024-03-19T11:30:27.476 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.476 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.476 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:27.476 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.380361+0000", 2024-03-19T11:30:27.476 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012855961000000001 2024-03-19T11:30:27.476 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.477 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.477 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:27.477 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.380987+0000", 2024-03-19T11:30:27.477 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00062590499999999995 2024-03-19T11:30:27.477 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.477 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.477 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:27.477 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.381149+0000", 2024-03-19T11:30:27.477 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000162239 2024-03-19T11:30:27.477 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.477 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.477 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:27.477 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.381282+0000", 2024-03-19T11:30:27.477 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013246500000000001 2024-03-19T11:30:27.477 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:27.478 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:27.478 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:27.478 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.478 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.478 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(osd.1.14:42 3.3 3:f0b412b0:::85cff1f6132709ad3f6974c608c4ca3d22c7ae775eb2def51616b490c1e0306e:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:27.478 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:13.796997+0000", 2024-03-19T11:30:27.478 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 13.626600423999999, 2024-03-19T11:30:27.478 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012796138, 2024-03-19T11:30:27.478 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:27.478 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:27.478 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:27.478 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.478 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:27.478 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.796997+0000", 2024-03-19T11:30:27.479 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:27.479 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.479 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.479 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:27.479 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.796997+0000", 2024-03-19T11:30:27.479 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:27.479 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.479 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.479 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:27.479 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.796998+0000", 2024-03-19T11:30:27.479 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5999999999999991e-07 2024-03-19T11:30:27.479 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.479 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.479 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:27.479 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.797240+0000", 2024-03-19T11:30:27.480 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00024155800000000001 2024-03-19T11:30:27.480 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.480 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.480 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:27.480 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.797255+0000", 2024-03-19T11:30:27.480 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4983e-05 2024-03-19T11:30:27.480 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.480 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.480 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:27.480 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.797270+0000", 2024-03-19T11:30:27.480 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4953000000000001e-05 2024-03-19T11:30:27.480 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.480 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.480 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:27.480 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.805058+0000", 2024-03-19T11:30:27.481 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0077889200000000004 2024-03-19T11:30:27.481 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.481 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.481 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:27.481 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.805289+0000", 2024-03-19T11:30:27.481 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000230115 2024-03-19T11:30:27.481 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.481 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.481 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:27.481 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.805648+0000", 2024-03-19T11:30:27.481 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00035961400000000002 2024-03-19T11:30:27.481 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.481 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.481 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:27.481 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.809089+0000", 2024-03-19T11:30:27.482 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034412499999999999 2024-03-19T11:30:27.482 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.482 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.482 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:27.482 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.809502+0000", 2024-03-19T11:30:27.482 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00041262100000000001 2024-03-19T11:30:27.482 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.482 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.482 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:27.482 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.809645+0000", 2024-03-19T11:30:27.482 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000142659 2024-03-19T11:30:27.483 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.483 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.483 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:27.483 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.809793+0000", 2024-03-19T11:30:27.483 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000148505 2024-03-19T11:30:27.483 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:27.483 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:27.483 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:27.483 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.483 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.483 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(osd.1.14:43 3.3 3:e8dad25b:::1ae0f6dc7c6ed0f10ae413d6bba06d7bb8915eb6bbafbf428184acbe98e3e057:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:27.483 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:13.797286+0000", 2024-03-19T11:30:27.483 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 13.626311566, 2024-03-19T11:30:27.483 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012678534999999999, 2024-03-19T11:30:27.483 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:27.484 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:27.484 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:27.484 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.484 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:27.484 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.797286+0000", 2024-03-19T11:30:27.484 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:27.484 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.484 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.484 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:27.484 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.797286+0000", 2024-03-19T11:30:27.484 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:27.484 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.484 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.484 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:27.484 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.797287+0000", 2024-03-19T11:30:27.485 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.16e-07 2024-03-19T11:30:27.485 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.485 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.485 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:27.485 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.797426+0000", 2024-03-19T11:30:27.485 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013880899999999999 2024-03-19T11:30:27.485 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.485 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.485 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:27.485 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.797452+0000", 2024-03-19T11:30:27.485 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.616e-05 2024-03-19T11:30:27.485 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.485 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.485 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:27.486 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.797468+0000", 2024-03-19T11:30:27.486 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6249999999999999e-05 2024-03-19T11:30:27.486 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.486 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.486 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:27.486 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.807167+0000", 2024-03-19T11:30:27.486 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0096989290000000002 2024-03-19T11:30:27.486 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.486 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.486 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:27.486 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.807529+0000", 2024-03-19T11:30:27.486 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00036221000000000002 2024-03-19T11:30:27.486 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.486 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.486 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:27.487 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.808040+0000", 2024-03-19T11:30:27.487 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00051056599999999995 2024-03-19T11:30:27.487 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.487 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.487 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:27.487 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.809455+0000", 2024-03-19T11:30:27.487 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0014148819999999999 2024-03-19T11:30:27.487 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.487 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.487 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:27.487 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.809833+0000", 2024-03-19T11:30:27.487 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000378694 2024-03-19T11:30:27.487 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.487 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.487 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:27.488 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.809911+0000", 2024-03-19T11:30:27.488 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7200000000000006e-05 2024-03-19T11:30:27.488 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.488 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.488 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:27.488 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.809964+0000", 2024-03-19T11:30:27.488 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3919000000000003e-05 2024-03-19T11:30:27.488 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:27.488 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:27.488 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:27.488 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.488 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.488 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1072 4.1 4:8401c1f0:::smithi03829856-43:head [write 494139~557703,stat] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:27.488 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:13.972007+0000", 2024-03-19T11:30:27.488 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 13.451590074, 2024-03-19T11:30:27.489 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.01935839, 2024-03-19T11:30:27.489 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:27.489 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:27.489 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:27.489 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:27.489 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:27.489 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1072 2024-03-19T11:30:27.489 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.489 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:27.489 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.489 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:27.489 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.972007+0000", 2024-03-19T11:30:27.489 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:27.489 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.489 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.490 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:27.490 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.972007+0000", 2024-03-19T11:30:27.490 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:27.490 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.490 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.490 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:27.490 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.972011+0000", 2024-03-19T11:30:27.490 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9360000000000001e-06 2024-03-19T11:30:27.490 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.490 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.490 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:27.490 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.972881+0000", 2024-03-19T11:30:27.490 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00086922699999999998 2024-03-19T11:30:27.490 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.491 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.491 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:27.491 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.972917+0000", 2024-03-19T11:30:27.491 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6359999999999997e-05 2024-03-19T11:30:27.491 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.491 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.491 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:27.491 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.972934+0000", 2024-03-19T11:30:27.491 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7249e-05 2024-03-19T11:30:27.491 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.491 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.491 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:27.491 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.973132+0000", 2024-03-19T11:30:27.491 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019760200000000001 2024-03-19T11:30:27.491 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.492 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.492 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:27.492 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.973303+0000", 2024-03-19T11:30:27.492 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017142899999999999 2024-03-19T11:30:27.492 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.492 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.492 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:27.492 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.973619+0000", 2024-03-19T11:30:27.492 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00031594700000000002 2024-03-19T11:30:27.492 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.492 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.492 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:27.492 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.985856+0000", 2024-03-19T11:30:27.492 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012236349000000001 2024-03-19T11:30:27.492 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.493 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.493 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:27.493 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.991132+0000", 2024-03-19T11:30:27.493 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052764960000000003 2024-03-19T11:30:27.493 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.493 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.493 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:27.493 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.991241+0000", 2024-03-19T11:30:27.493 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000108842 2024-03-19T11:30:27.493 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.493 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.493 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:27.493 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.991366+0000", 2024-03-19T11:30:27.493 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012495299999999999 2024-03-19T11:30:27.493 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:27.494 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:27.494 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:27.494 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.494 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.494 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1162 4.1 4:8401c1f0:::smithi03829856-43:head [write 594981~609197] snapc 8=[8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-03-19T11:30:27.494 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.838981+0000", 2024-03-19T11:30:27.494 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.584616719, 2024-03-19T11:30:27.494 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027704349999999999, 2024-03-19T11:30:27.494 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:27.494 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:27.494 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:27.494 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:27.494 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:27.494 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1162 2024-03-19T11:30:27.494 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.495 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:27.495 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.495 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:27.495 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.838981+0000", 2024-03-19T11:30:27.495 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:27.495 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.495 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.495 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:27.495 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.838981+0000", 2024-03-19T11:30:27.495 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:27.495 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.495 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.495 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:27.495 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.838983+0000", 2024-03-19T11:30:27.495 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5399999999999998e-06 2024-03-19T11:30:27.496 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.496 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.496 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:27.496 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841322+0000", 2024-03-19T11:30:27.496 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002338592 2024-03-19T11:30:27.496 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.496 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.496 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:27.496 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841371+0000", 2024-03-19T11:30:27.496 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9123999999999999e-05 2024-03-19T11:30:27.496 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.496 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.496 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:27.496 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841399+0000", 2024-03-19T11:30:27.496 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7718000000000001e-05 2024-03-19T11:30:27.497 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.497 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.497 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:27.497 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841577+0000", 2024-03-19T11:30:27.497 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000178097 2024-03-19T11:30:27.497 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.497 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.497 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:27.497 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841825+0000", 2024-03-19T11:30:27.497 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00024824299999999999 2024-03-19T11:30:27.497 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.497 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.497 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:27.497 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842227+0000", 2024-03-19T11:30:27.497 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00040212199999999998 2024-03-19T11:30:27.498 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.498 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.498 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:27.498 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.865182+0000", 2024-03-19T11:30:27.498 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022954810999999999 2024-03-19T11:30:27.498 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.498 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.498 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:27.498 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.866360+0000", 2024-03-19T11:30:27.498 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001178449 2024-03-19T11:30:27.498 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.498 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.498 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:27.498 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.866538+0000", 2024-03-19T11:30:27.499 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017791000000000001 2024-03-19T11:30:27.499 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.499 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.499 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:27.499 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.866685+0000", 2024-03-19T11:30:27.499 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000146744 2024-03-19T11:30:27.499 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:27.499 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:27.499 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:27.499 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.499 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.499 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1163 4.1 4:8401c1f0:::smithi03829856-43:head [write 1974123~745028] snapc 8=[8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-03-19T11:30:27.499 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.851879+0000", 2024-03-19T11:30:27.499 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.571718597, 2024-03-19T11:30:27.500 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015531774, 2024-03-19T11:30:27.500 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:27.500 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:27.500 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:27.500 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:27.500 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:27.500 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1163 2024-03-19T11:30:27.500 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.500 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:27.500 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.500 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:27.500 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.851879+0000", 2024-03-19T11:30:27.500 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:27.500 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.500 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.501 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:27.501 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.851879+0000", 2024-03-19T11:30:27.501 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:27.501 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.501 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.501 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:27.501 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.851882+0000", 2024-03-19T11:30:27.501 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6960000000000001e-06 2024-03-19T11:30:27.501 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.501 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.501 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:27.501 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.853204+0000", 2024-03-19T11:30:27.501 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0013226620000000001 2024-03-19T11:30:27.501 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.502 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.502 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:27.502 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.853231+0000", 2024-03-19T11:30:27.502 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6775e-05 2024-03-19T11:30:27.502 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.502 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.502 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:27.502 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.853246+0000", 2024-03-19T11:30:27.502 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5403000000000001e-05 2024-03-19T11:30:27.502 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.502 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.502 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:27.502 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.853363+0000", 2024-03-19T11:30:27.502 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000116296 2024-03-19T11:30:27.503 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.503 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.503 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:27.503 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.853572+0000", 2024-03-19T11:30:27.503 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020931500000000001 2024-03-19T11:30:27.503 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.503 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.503 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:27.503 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.853797+0000", 2024-03-19T11:30:27.503 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022544700000000001 2024-03-19T11:30:27.503 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.503 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.503 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:27.503 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.865195+0000", 2024-03-19T11:30:27.503 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011397981999999999 2024-03-19T11:30:27.504 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.504 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.504 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:27.504 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.866916+0000", 2024-03-19T11:30:27.504 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017204449999999999 2024-03-19T11:30:27.504 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.504 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.504 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:27.504 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.867129+0000", 2024-03-19T11:30:27.504 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021317999999999999 2024-03-19T11:30:27.504 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.504 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.504 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:27.504 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.867411+0000", 2024-03-19T11:30:27.505 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028157300000000002 2024-03-19T11:30:27.505 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:27.505 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:27.505 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:27.505 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.505 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.505 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1205 4.1 4:8401c1f0:::smithi03829856-43:head [assert-version v50,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:27.505 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:18.112028+0000", 2024-03-19T11:30:27.505 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.3115694629999997, 2024-03-19T11:30:27.505 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013579206, 2024-03-19T11:30:27.505 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:27.505 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:27.505 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:27.505 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:27.505 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:27.506 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1205 2024-03-19T11:30:27.506 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.506 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:27.506 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.506 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:27.506 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112028+0000", 2024-03-19T11:30:27.506 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:27.506 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.506 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.506 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:27.506 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112028+0000", 2024-03-19T11:30:27.506 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:27.506 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.507 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.507 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:27.507 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112030+0000", 2024-03-19T11:30:27.507 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4549999999999998e-06 2024-03-19T11:30:27.507 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.507 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.507 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:27.507 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112043+0000", 2024-03-19T11:30:27.507 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2721999999999999e-05 2024-03-19T11:30:27.507 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.507 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.507 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:27.507 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112075+0000", 2024-03-19T11:30:27.507 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.205e-05 2024-03-19T11:30:27.507 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.508 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.508 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:27.508 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112091+0000", 2024-03-19T11:30:27.508 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5364000000000001e-05 2024-03-19T11:30:27.508 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.508 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.508 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:27.508 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112223+0000", 2024-03-19T11:30:27.508 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013224100000000001 2024-03-19T11:30:27.508 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.509 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.509 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:27.509 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112393+0000", 2024-03-19T11:30:27.509 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000169681 2024-03-19T11:30:27.509 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.509 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.509 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:27.509 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.120919+0000", 2024-03-19T11:30:27.509 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085260140000000002 2024-03-19T11:30:27.509 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.509 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.509 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:27.509 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125207+0000", 2024-03-19T11:30:27.509 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004288722 2024-03-19T11:30:27.510 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.510 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.510 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:27.510 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125343+0000", 2024-03-19T11:30:27.510 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013558400000000001 2024-03-19T11:30:27.510 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.510 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.510 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:27.510 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125497+0000", 2024-03-19T11:30:27.510 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000153678 2024-03-19T11:30:27.510 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.510 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.510 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:27.510 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125607+0000", 2024-03-19T11:30:27.511 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110695 2024-03-19T11:30:27.511 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:27.511 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:27.511 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:27.511 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.511 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.511 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1208 4.1 4:866f3dcc:::smithi03829856-31:head [read 0~2941192,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:27.511 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:18.112252+0000", 2024-03-19T11:30:27.511 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.3113458930000004, 2024-03-19T11:30:27.511 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.01291702, 2024-03-19T11:30:27.511 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:27.511 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:27.511 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:27.511 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:27.512 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:27.512 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1208 2024-03-19T11:30:27.512 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.512 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:27.512 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.512 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:27.512 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112252+0000", 2024-03-19T11:30:27.512 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:27.512 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.512 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.512 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:27.512 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112252+0000", 2024-03-19T11:30:27.512 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:27.512 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.512 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.513 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:27.513 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112253+0000", 2024-03-19T11:30:27.513 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6299999999999993e-07 2024-03-19T11:30:27.513 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.513 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.513 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:27.513 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112257+0000", 2024-03-19T11:30:27.513 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4360000000000004e-06 2024-03-19T11:30:27.513 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.513 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.513 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:27.513 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112272+0000", 2024-03-19T11:30:27.513 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.518e-05 2024-03-19T11:30:27.513 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.513 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.514 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:27.514 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112281+0000", 2024-03-19T11:30:27.514 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.422e-06 2024-03-19T11:30:27.514 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.514 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.514 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:27.514 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.122861+0000", 2024-03-19T11:30:27.514 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010580523 2024-03-19T11:30:27.514 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.514 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.514 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:27.514 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123148+0000", 2024-03-19T11:30:27.514 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028714499999999999 2024-03-19T11:30:27.514 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.515 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.515 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:27.515 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125169+0000", 2024-03-19T11:30:27.515 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0020203510000000001 2024-03-19T11:30:27.515 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:27.515 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:27.515 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:27.515 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.515 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.515 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1243 4.9 4:9ef41243:::smithi03829856-5:head [tier-promote] snapc 9=[9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:27.515 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:18.345759+0000", 2024-03-19T11:30:27.515 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.0778382820000001, 2024-03-19T11:30:27.515 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018400962999999999, 2024-03-19T11:30:27.515 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:27.515 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:27.515 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:27.516 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:27.516 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:27.516 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1243 2024-03-19T11:30:27.516 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.516 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:27.516 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.516 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:27.516 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.345759+0000", 2024-03-19T11:30:27.516 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:27.516 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.516 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.516 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:27.516 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.345759+0000", 2024-03-19T11:30:27.516 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:27.516 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.517 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.517 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:27.517 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.345763+0000", 2024-03-19T11:30:27.517 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3790000000000002e-06 2024-03-19T11:30:27.517 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.517 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.517 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:27.517 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.345776+0000", 2024-03-19T11:30:27.517 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3784000000000001e-05 2024-03-19T11:30:27.517 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.518 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.518 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:27.518 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.345812+0000", 2024-03-19T11:30:27.518 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5599999999999998e-05 2024-03-19T11:30:27.518 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.518 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.518 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:27.518 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.345829+0000", 2024-03-19T11:30:27.518 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6722999999999999e-05 2024-03-19T11:30:27.518 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.518 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.518 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:27.518 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.345964+0000", 2024-03-19T11:30:27.518 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013554800000000001 2024-03-19T11:30:27.519 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.519 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.519 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:27.519 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.346133+0000", 2024-03-19T11:30:27.519 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000169211 2024-03-19T11:30:27.519 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.519 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.519 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:27.519 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.364160+0000", 2024-03-19T11:30:27.519 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018026718000000001 2024-03-19T11:30:27.519 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:27.519 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:27.519 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:27.519 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.519 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.520 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1271 4.9 4:953e720b:::smithi03829856-45:2 [read 795136~1008128] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-03-19T11:30:27.520 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:21.868056+0000", 2024-03-19T11:30:27.520 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.5555412139999998, 2024-03-19T11:30:27.520 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.047270591000000001, 2024-03-19T11:30:27.520 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:27.520 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:27.520 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:27.520 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:27.520 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:27.520 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1271 2024-03-19T11:30:27.520 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.520 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:27.520 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.520 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:27.520 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868056+0000", 2024-03-19T11:30:27.521 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:27.521 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.521 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.521 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:27.521 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868056+0000", 2024-03-19T11:30:27.521 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:27.521 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.521 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.521 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:27.521 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868059+0000", 2024-03-19T11:30:27.521 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0469999999999998e-06 2024-03-19T11:30:27.522 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.522 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.522 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:27.522 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868072+0000", 2024-03-19T11:30:27.522 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2700999999999999e-05 2024-03-19T11:30:27.522 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.522 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.522 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:27.522 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868106+0000", 2024-03-19T11:30:27.522 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4152000000000001e-05 2024-03-19T11:30:27.522 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.522 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.522 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:27.522 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868126+0000", 2024-03-19T11:30:27.523 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9386999999999999e-05 2024-03-19T11:30:27.523 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.523 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.523 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:27.523 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.911447+0000", 2024-03-19T11:30:27.523 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.043321164000000002 2024-03-19T11:30:27.523 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.523 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.523 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:27.523 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.911762+0000", 2024-03-19T11:30:27.523 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00031492400000000001 2024-03-19T11:30:27.523 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.523 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.523 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:27.523 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.915327+0000", 2024-03-19T11:30:27.523 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003565216 2024-03-19T11:30:27.524 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:27.524 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:27.524 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:27.524 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:27.524 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:27.524 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:27.525 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-19T11:30:27.526 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:27.535+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:27.626 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:27.635+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:27.700 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:27.700 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:27.701 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:30:27.701 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:27.714 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-19T11:30:27.727 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:27.739+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:27.801 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-19T11:30:27.831 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:27.831 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:27.832 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:30:27.832 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:30:27.832 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:27.845 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-19T11:30:27.847 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:27.859+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:27.899 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:27.911+0000 7fa5e08db640 -1 osd.3 49 handle_osd_map injecting map crc failure 2024-03-19T11:30:27.902 INFO:teuthology.orchestra.run.smithi038.stdout:moved 10 / 78 (12.8205%) 2024-03-19T11:30:27.902 INFO:teuthology.orchestra.run.smithi038.stdout:avg 9.75 2024-03-19T11:30:27.902 INFO:teuthology.orchestra.run.smithi038.stdout:stddev 3.15238 -> 2.04634 (expected baseline 2.92083) 2024-03-19T11:30:27.902 INFO:teuthology.orchestra.run.smithi038.stdout:min osd.3 with 7 -> 9 pgs (0.717949 -> 0.923077 * mean) 2024-03-19T11:30:27.903 INFO:teuthology.orchestra.run.smithi038.stdout:max osd.1 with 15 -> 12 pgs (1.53846 -> 1.23077 * mean) 2024-03-19T11:30:27.903 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:30:27.903 INFO:teuthology.orchestra.run.smithi038.stdout:oload 110 2024-03-19T11:30:27.903 INFO:teuthology.orchestra.run.smithi038.stdout:max_change 1 2024-03-19T11:30:27.903 INFO:teuthology.orchestra.run.smithi038.stdout:max_change_osds 4 2024-03-19T11:30:27.903 INFO:teuthology.orchestra.run.smithi038.stdout:average_utilization 0.0007 2024-03-19T11:30:27.903 INFO:teuthology.orchestra.run.smithi038.stdout:overload_utilization 0.0008 2024-03-19T11:30:27.903 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4 weight 0.5995 -> 0.3894 2024-03-19T11:30:27.903 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1 weight 0.7291 -> 0.5681 2024-03-19T11:30:27.903 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0 weight 0.8196 -> 0.7127 2024-03-19T11:30:27.921 INFO:tasks.rados.rados.0.smithi038.stdout:848: set_chunk oid 28 offset: 534528 length: 391680 target oid 34fe995401eca27dd3a4e66fa05e763b0a8a42fb8547dc1dac7363a23d09ccfb offset: 0 snap -1 2024-03-19T11:30:27.922 INFO:tasks.rados.rados.0.smithi038.stdout:849: read oid 23 snap 4 2024-03-19T11:30:27.922 INFO:tasks.rados.rados.0.smithi038.stdout:849: expect (ObjNum 22 snap 0 seq_num 22) 2024-03-19T11:30:27.922 INFO:tasks.rados.rados.0.smithi038.stdout:850: read oid 38 snap 9 2024-03-19T11:30:27.922 INFO:tasks.rados.rados.0.smithi038.stdout:850: expect (ObjNum 68 snap 3 seq_num 68) 2024-03-19T11:30:27.922 INFO:tasks.rados.rados.0.smithi038.stdout:851: read oid 43 snap 3 2024-03-19T11:30:27.922 INFO:tasks.rados.rados.0.smithi038.stdout:851: expect (ObjNum 50 snap 0 seq_num 50) 2024-03-19T11:30:27.923 INFO:tasks.rados.rados.0.smithi038.stdout:852: delete oid 25 current snap is 11 2024-03-19T11:30:27.923 INFO:tasks.rados.rados.0.smithi038.stdout:838: got (22) Invalid argument 2024-03-19T11:30:27.923 INFO:tasks.rados.rados.0.smithi038.stdout:835: expect (ObjNum 23 snap 0 seq_num 23) 2024-03-19T11:30:27.928 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:27.939+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:27.943 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-03-19T11:30:27.943 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 110 3.0 2024-03-19T11:30:27.969 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:30:27.970 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:30:27.986 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:27.986 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:30:27.986 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:30:27.986 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:30:27.986 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.986 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:844 4.7 e32/21 4:e41d47fb:::smithi03829856-14:head v 32'8, mlcod=28'5)", 2024-03-19T11:30:27.986 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.751447+0000", 2024-03-19T11:30:27.986 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.244184927999999, 2024-03-19T11:30:27.987 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012301835000000001, 2024-03-19T11:30:27.987 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:27.987 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:27.987 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:27.987 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.987 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:27.987 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.751447+0000", 2024-03-19T11:30:27.987 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:27.987 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.987 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.987 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:27.987 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.751447+0000", 2024-03-19T11:30:27.987 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:27.988 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.988 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.988 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:27.988 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.751447+0000", 2024-03-19T11:30:27.988 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4300000000000003e-07 2024-03-19T11:30:27.988 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.988 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.988 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:27.988 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.752747+0000", 2024-03-19T11:30:27.988 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012995699999999999 2024-03-19T11:30:27.988 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.988 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.988 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:27.988 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.752771+0000", 2024-03-19T11:30:27.989 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3592999999999999e-05 2024-03-19T11:30:27.989 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.989 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.989 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:27.989 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.752789+0000", 2024-03-19T11:30:27.989 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8703e-05 2024-03-19T11:30:27.989 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.989 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.989 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:27.989 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.752911+0000", 2024-03-19T11:30:27.989 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012139899999999999 2024-03-19T11:30:27.989 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.989 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.989 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:27.989 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.752949+0000", 2024-03-19T11:30:27.990 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8436999999999998e-05 2024-03-19T11:30:27.990 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.990 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.990 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:27.990 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763568+0000", 2024-03-19T11:30:27.990 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010618768000000001 2024-03-19T11:30:27.990 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.990 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.990 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:27.990 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763749+0000", 2024-03-19T11:30:27.990 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000180822 2024-03-19T11:30:27.990 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:27.990 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:27.991 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:27.991 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.991 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.991 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1047 4.8 4:1d58db11:::smithi03829856-18:head [write 443718~615603,stat] snapc 6=[6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:27.991 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:13.221766+0000", 2024-03-19T11:30:27.991 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.773865761, 2024-03-19T11:30:27.991 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020932596000000001, 2024-03-19T11:30:27.991 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:27.991 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:27.991 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:27.991 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:27.991 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:27.991 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1047 2024-03-19T11:30:27.991 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.992 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:27.992 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.992 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:27.992 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.221766+0000", 2024-03-19T11:30:27.992 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:27.992 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.992 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.992 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:27.992 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.221766+0000", 2024-03-19T11:30:27.992 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:27.992 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.992 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.992 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:27.992 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.221770+0000", 2024-03-19T11:30:27.992 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.67e-06 2024-03-19T11:30:27.993 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.993 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.993 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:27.993 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.223938+0000", 2024-03-19T11:30:27.993 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021683399999999999 2024-03-19T11:30:27.993 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.993 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.993 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:27.993 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.223979+0000", 2024-03-19T11:30:27.993 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1433000000000001e-05 2024-03-19T11:30:27.993 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.993 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.993 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:27.993 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.223998+0000", 2024-03-19T11:30:27.993 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8853000000000001e-05 2024-03-19T11:30:27.994 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.994 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.994 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:27.994 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.224150+0000", 2024-03-19T11:30:27.994 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000151679 2024-03-19T11:30:27.994 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.994 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.994 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:27.994 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.224362+0000", 2024-03-19T11:30:27.994 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021154499999999999 2024-03-19T11:30:27.994 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.994 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.994 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:27.995 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.224745+0000", 2024-03-19T11:30:27.995 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00038342399999999999 2024-03-19T11:30:27.995 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.995 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.995 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:27.995 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.241167+0000", 2024-03-19T11:30:27.995 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0164218 2024-03-19T11:30:27.995 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.995 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.995 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:27.995 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.242406+0000", 2024-03-19T11:30:27.995 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012390039999999999 2024-03-19T11:30:27.995 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.995 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.996 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:27.996 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.242575+0000", 2024-03-19T11:30:27.996 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000168925 2024-03-19T11:30:27.996 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.996 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.996 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:27.996 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.242699+0000", 2024-03-19T11:30:27.996 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012392300000000001 2024-03-19T11:30:27.996 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:27.996 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:27.996 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:27.996 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.996 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.996 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1048 4.8 4:1d58db11:::smithi03829856-18:head [write 1564138~470341] snapc 6=[6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:27.996 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:13.227287+0000", 2024-03-19T11:30:27.997 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.768344493000001, 2024-03-19T11:30:27.997 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015802897999999999, 2024-03-19T11:30:27.997 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:27.997 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:27.997 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:27.997 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:27.997 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:27.997 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1048 2024-03-19T11:30:27.997 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.997 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:27.997 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.997 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:27.997 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.227287+0000", 2024-03-19T11:30:27.997 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:27.997 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.998 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.998 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:27.998 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.227287+0000", 2024-03-19T11:30:27.998 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:27.998 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.998 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.998 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:27.998 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.227290+0000", 2024-03-19T11:30:27.998 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.88e-06 2024-03-19T11:30:27.998 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.998 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.998 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:27.998 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.228224+0000", 2024-03-19T11:30:27.998 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00093389600000000003 2024-03-19T11:30:27.999 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.999 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.999 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:27.999 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.228258+0000", 2024-03-19T11:30:27.999 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4282999999999997e-05 2024-03-19T11:30:27.999 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.999 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.999 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:27.999 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.228273+0000", 2024-03-19T11:30:27.999 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4833e-05 2024-03-19T11:30:27.999 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:27.999 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:27.999 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:27.999 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.231996+0000", 2024-03-19T11:30:28.000 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037223410000000001 2024-03-19T11:30:28.000 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.000 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.000 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:28.000 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.232143+0000", 2024-03-19T11:30:28.000 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014714900000000001 2024-03-19T11:30:28.000 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.000 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.000 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:28.000 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.232395+0000", 2024-03-19T11:30:28.000 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025201500000000002 2024-03-19T11:30:28.000 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.000 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.001 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:28.001 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.242848+0000", 2024-03-19T11:30:28.001 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010453393 2024-03-19T11:30:28.001 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.001 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.001 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:28.001 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.242872+0000", 2024-03-19T11:30:28.001 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3816e-05 2024-03-19T11:30:28.001 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.001 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.001 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:28.001 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.242966+0000", 2024-03-19T11:30:28.001 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4560000000000003e-05 2024-03-19T11:30:28.001 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.001 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.002 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:28.002 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.243090+0000", 2024-03-19T11:30:28.002 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012373199999999999 2024-03-19T11:30:28.002 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:28.002 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:28.002 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:28.002 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.002 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.002 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1072 4.1 e36/21 4:8401c1f0:::smithi03829856-43:head v 36'38, mlcod=36'35)", 2024-03-19T11:30:28.002 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:13.973829+0000", 2024-03-19T11:30:28.002 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.02180317, 2024-03-19T11:30:28.002 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013548782000000001, 2024-03-19T11:30:28.002 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:28.002 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:28.003 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:28.003 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.003 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:28.003 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.973829+0000", 2024-03-19T11:30:28.003 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:28.003 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.003 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.003 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:28.003 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.973829+0000", 2024-03-19T11:30:28.003 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:28.003 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.003 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.003 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:28.003 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.973831+0000", 2024-03-19T11:30:28.003 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0899999999999999e-06 2024-03-19T11:30:28.004 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.004 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.004 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:28.004 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.974489+0000", 2024-03-19T11:30:28.004 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00065813199999999997 2024-03-19T11:30:28.004 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.004 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.004 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:28.004 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.974523+0000", 2024-03-19T11:30:28.004 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3841999999999999e-05 2024-03-19T11:30:28.004 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.004 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.004 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:28.004 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.974544+0000", 2024-03-19T11:30:28.004 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0907000000000002e-05 2024-03-19T11:30:28.005 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.005 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.005 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:28.005 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.974697+0000", 2024-03-19T11:30:28.005 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015295599999999999 2024-03-19T11:30:28.005 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.005 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.005 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:28.005 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.974742+0000", 2024-03-19T11:30:28.005 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5735999999999997e-05 2024-03-19T11:30:28.005 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.005 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.005 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:28.005 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.987261+0000", 2024-03-19T11:30:28.005 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012518316999999999 2024-03-19T11:30:28.006 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.006 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.006 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:28.006 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.987377+0000", 2024-03-19T11:30:28.006 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000116802 2024-03-19T11:30:28.006 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:28.006 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:28.006 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:28.006 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.006 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.006 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1091 4.8 4:14611006:::smithi03829856-25:head [write 527753~708201] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:28.006 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:14.015839+0000", 2024-03-19T11:30:28.006 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 13.979793110999999, 2024-03-19T11:30:28.006 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016721983999999999, 2024-03-19T11:30:28.007 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:28.007 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:28.007 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:28.007 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:28.007 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:28.007 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1091 2024-03-19T11:30:28.007 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.007 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:28.007 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.007 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:28.007 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.015839+0000", 2024-03-19T11:30:28.007 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:28.007 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.007 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.008 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:28.008 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.015839+0000", 2024-03-19T11:30:28.008 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:28.008 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.008 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.008 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:28.008 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.015842+0000", 2024-03-19T11:30:28.008 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3569999999999998e-06 2024-03-19T11:30:28.008 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.008 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.008 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:28.008 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.016678+0000", 2024-03-19T11:30:28.008 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00083620299999999999 2024-03-19T11:30:28.008 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.008 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.009 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:28.009 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.016710+0000", 2024-03-19T11:30:28.009 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1337000000000002e-05 2024-03-19T11:30:28.009 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.009 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.009 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:28.009 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.016724+0000", 2024-03-19T11:30:28.009 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4323e-05 2024-03-19T11:30:28.009 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.009 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.009 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:28.009 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.016847+0000", 2024-03-19T11:30:28.009 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012304199999999999 2024-03-19T11:30:28.009 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.010 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.010 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:28.010 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.017023+0000", 2024-03-19T11:30:28.010 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000175945 2024-03-19T11:30:28.010 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.010 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.010 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:28.010 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.017324+0000", 2024-03-19T11:30:28.010 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00030151499999999998 2024-03-19T11:30:28.010 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.010 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.010 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:28.010 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.032104+0000", 2024-03-19T11:30:28.010 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014779232999999999 2024-03-19T11:30:28.010 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.011 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.011 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:28.011 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.032190+0000", 2024-03-19T11:30:28.011 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6304999999999994e-05 2024-03-19T11:30:28.011 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.011 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.011 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:28.011 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.032350+0000", 2024-03-19T11:30:28.011 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000159813 2024-03-19T11:30:28.011 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.011 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.011 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:28.011 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.032561+0000", 2024-03-19T11:30:28.011 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021091099999999999 2024-03-19T11:30:28.012 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:28.012 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:28.012 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:28.012 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.012 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.012 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1158 4.8 4:15708482:::smithi03829856-23:head [tier-promote] snapc 8=[8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-03-19T11:30:28.012 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.827517+0000", 2024-03-19T11:30:28.012 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.168115182999999, 2024-03-19T11:30:28.012 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.034096983999999997, 2024-03-19T11:30:28.012 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:28.012 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:28.012 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:28.012 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:28.012 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:28.012 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1158 2024-03-19T11:30:28.013 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.013 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:28.013 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.013 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:28.013 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827517+0000", 2024-03-19T11:30:28.013 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:28.013 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.013 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.013 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:28.013 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827517+0000", 2024-03-19T11:30:28.013 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:28.013 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.013 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.014 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:28.014 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827520+0000", 2024-03-19T11:30:28.014 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.45e-06 2024-03-19T11:30:28.014 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.014 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.014 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:28.014 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827535+0000", 2024-03-19T11:30:28.014 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.509e-05 2024-03-19T11:30:28.014 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.014 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.014 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:28.014 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827579+0000", 2024-03-19T11:30:28.014 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4211000000000001e-05 2024-03-19T11:30:28.015 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.015 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.015 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:28.015 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827603+0000", 2024-03-19T11:30:28.015 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3586000000000001e-05 2024-03-19T11:30:28.015 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.015 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.015 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:28.015 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827789+0000", 2024-03-19T11:30:28.015 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018580199999999999 2024-03-19T11:30:28.015 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.015 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.015 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:28.015 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.828077+0000", 2024-03-19T11:30:28.016 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028846500000000001 2024-03-19T11:30:28.016 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.016 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.016 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:28.016 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.861614+0000", 2024-03-19T11:30:28.016 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033536379999999998 2024-03-19T11:30:28.016 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:28.016 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:28.016 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:28.016 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.016 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.016 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1162 4.1 e39/21 4:8401c1f0:::smithi03829856-43:head v 39'47, mlcod=38'43)", 2024-03-19T11:30:28.016 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.842598+0000", 2024-03-19T11:30:28.016 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.153033412999999, 2024-03-19T11:30:28.016 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0233756, 2024-03-19T11:30:28.017 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:28.017 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:28.017 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:28.017 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.017 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:28.017 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842598+0000", 2024-03-19T11:30:28.017 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:28.017 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.017 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.017 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:28.017 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842598+0000", 2024-03-19T11:30:28.017 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:28.017 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.017 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.018 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:28.018 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842599+0000", 2024-03-19T11:30:28.018 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6000000000000003e-07 2024-03-19T11:30:28.018 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.018 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.018 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:28.018 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.843566+0000", 2024-03-19T11:30:28.018 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00096688700000000004 2024-03-19T11:30:28.018 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.018 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.018 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:28.018 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.843601+0000", 2024-03-19T11:30:28.018 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.502e-05 2024-03-19T11:30:28.018 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.019 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.019 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:28.019 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.843621+0000", 2024-03-19T11:30:28.019 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0305e-05 2024-03-19T11:30:28.019 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.019 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.019 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:28.019 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.843882+0000", 2024-03-19T11:30:28.019 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026027100000000002 2024-03-19T11:30:28.019 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.019 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.019 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:28.019 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.844022+0000", 2024-03-19T11:30:28.019 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014020100000000001 2024-03-19T11:30:28.019 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.020 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.020 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:28.020 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.865862+0000", 2024-03-19T11:30:28.020 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021839725000000001 2024-03-19T11:30:28.020 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.020 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.020 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:28.020 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.865974+0000", 2024-03-19T11:30:28.020 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011243099999999999 2024-03-19T11:30:28.020 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:28.020 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:28.020 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:28.020 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.020 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.021 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1163 4.1 e39/21 4:8401c1f0:::smithi03829856-43:head v 39'48, mlcod=38'43)", 2024-03-19T11:30:28.021 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.854006+0000", 2024-03-19T11:30:28.021 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.141625726000001, 2024-03-19T11:30:28.021 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012036872000000001, 2024-03-19T11:30:28.021 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:28.021 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:28.021 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:28.021 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.021 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:28.021 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.854006+0000", 2024-03-19T11:30:28.021 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:28.021 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.021 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.021 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:28.022 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.854006+0000", 2024-03-19T11:30:28.022 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:28.022 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.022 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.022 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:28.022 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.854008+0000", 2024-03-19T11:30:28.022 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0190000000000001e-06 2024-03-19T11:30:28.022 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.022 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.022 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:28.022 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.854542+0000", 2024-03-19T11:30:28.022 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00053378200000000001 2024-03-19T11:30:28.022 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.022 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.023 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:28.023 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.854577+0000", 2024-03-19T11:30:28.023 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4740000000000003e-05 2024-03-19T11:30:28.023 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.023 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.023 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:28.023 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.854597+0000", 2024-03-19T11:30:28.023 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0225999999999999e-05 2024-03-19T11:30:28.023 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.023 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.023 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:28.023 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.858165+0000", 2024-03-19T11:30:28.023 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035682790000000002 2024-03-19T11:30:28.023 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.023 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.024 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:28.024 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.858231+0000", 2024-03-19T11:30:28.024 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5916999999999999e-05 2024-03-19T11:30:28.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.024 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:28.024 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.865995+0000", 2024-03-19T11:30:28.024 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0077640720000000003 2024-03-19T11:30:28.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.024 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:28.024 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.866043+0000", 2024-03-19T11:30:28.024 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7837000000000003e-05 2024-03-19T11:30:28.024 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:28.025 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:28.025 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:28.025 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.025 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.025 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1209 4.8 4:1d58db11:::smithi03829856-18:head [write 645402~551297,stat] snapc 9=[9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:28.025 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:18.121657+0000", 2024-03-19T11:30:28.025 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.8739744110000007, 2024-03-19T11:30:28.025 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.025074349999999999, 2024-03-19T11:30:28.025 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:28.025 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:28.025 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:28.025 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:28.025 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:28.025 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1209 2024-03-19T11:30:28.025 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.026 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:28.026 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.026 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:28.026 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.121657+0000", 2024-03-19T11:30:28.026 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:28.026 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.026 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.026 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:28.026 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.121657+0000", 2024-03-19T11:30:28.026 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:28.026 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.026 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.026 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:28.026 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.121661+0000", 2024-03-19T11:30:28.026 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0489999999999997e-06 2024-03-19T11:30:28.027 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.027 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.027 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:28.027 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123747+0000", 2024-03-19T11:30:28.027 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0020857720000000001 2024-03-19T11:30:28.027 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.027 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.027 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:28.027 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123779+0000", 2024-03-19T11:30:28.027 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2258000000000002e-05 2024-03-19T11:30:28.027 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.027 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.027 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:28.027 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123795+0000", 2024-03-19T11:30:28.028 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5177000000000001e-05 2024-03-19T11:30:28.028 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.028 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:28.028 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123949+0000", 2024-03-19T11:30:28.028 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015403699999999999 2024-03-19T11:30:28.028 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.028 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:28.028 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.124134+0000", 2024-03-19T11:30:28.028 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000185484 2024-03-19T11:30:28.028 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.028 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:28.029 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.124554+0000", 2024-03-19T11:30:28.029 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00042019699999999998 2024-03-19T11:30:28.029 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.029 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.029 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:28.029 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146267+0000", 2024-03-19T11:30:28.029 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021712838000000002 2024-03-19T11:30:28.029 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.029 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.029 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:28.029 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146424+0000", 2024-03-19T11:30:28.029 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015647299999999999 2024-03-19T11:30:28.029 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.029 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.029 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:28.030 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146597+0000", 2024-03-19T11:30:28.030 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017303699999999999 2024-03-19T11:30:28.030 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.030 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.030 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:28.030 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146732+0000", 2024-03-19T11:30:28.030 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013502800000000001 2024-03-19T11:30:28.030 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:28.030 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:28.030 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:28.030 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.030 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.030 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1210 4.8 4:1d58db11:::smithi03829856-18:head [write 1749569~492355] snapc 9=[9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:28.030 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:18.129257+0000", 2024-03-19T11:30:28.031 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.8663750540000006, 2024-03-19T11:30:28.031 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018110444999999999, 2024-03-19T11:30:28.031 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:28.031 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:28.031 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:28.031 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:28.031 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:28.031 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1210 2024-03-19T11:30:28.031 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.031 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:28.031 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.031 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:28.031 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.129257+0000", 2024-03-19T11:30:28.031 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:28.032 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.032 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.032 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:28.032 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.129257+0000", 2024-03-19T11:30:28.032 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:28.032 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.032 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.032 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:28.032 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.129261+0000", 2024-03-19T11:30:28.032 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7119999999999999e-06 2024-03-19T11:30:28.032 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.032 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.032 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:28.032 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.129717+0000", 2024-03-19T11:30:28.033 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00045526300000000002 2024-03-19T11:30:28.033 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.033 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.033 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:28.033 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.129749+0000", 2024-03-19T11:30:28.033 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2700999999999997e-05 2024-03-19T11:30:28.033 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.033 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.033 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:28.033 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.129774+0000", 2024-03-19T11:30:28.033 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4592000000000001e-05 2024-03-19T11:30:28.033 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.033 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.033 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:28.033 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.134875+0000", 2024-03-19T11:30:28.034 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.005100477 2024-03-19T11:30:28.034 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.034 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.034 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:28.034 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.135035+0000", 2024-03-19T11:30:28.034 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016054200000000001 2024-03-19T11:30:28.034 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.034 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.034 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:28.034 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.135268+0000", 2024-03-19T11:30:28.034 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000233253 2024-03-19T11:30:28.034 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.034 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.034 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:28.035 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146872+0000", 2024-03-19T11:30:28.035 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011603732 2024-03-19T11:30:28.035 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.035 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.035 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:28.035 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.147103+0000", 2024-03-19T11:30:28.035 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023070299999999999 2024-03-19T11:30:28.035 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.035 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.035 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:28.035 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.147181+0000", 2024-03-19T11:30:28.035 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8567999999999997e-05 2024-03-19T11:30:28.035 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.035 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.035 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:28.036 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.147367+0000", 2024-03-19T11:30:28.036 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000185902 2024-03-19T11:30:28.036 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:28.036 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:28.036 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:28.036 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.036 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1259 4.1 e44/43 4:866f3dcc:::smithi03829856-31:head v 44'53, mlcod=0'0)", 2024-03-19T11:30:28.036 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:21.869109+0000", 2024-03-19T11:30:28.036 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.126523164, 2024-03-19T11:30:28.036 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014777017999999999, 2024-03-19T11:30:28.036 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:28.036 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:28.036 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:28.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.037 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:28.037 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.869109+0000", 2024-03-19T11:30:28.037 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:28.037 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.037 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.037 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:28.037 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.869109+0000", 2024-03-19T11:30:28.037 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:28.037 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.037 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.037 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:28.037 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.869110+0000", 2024-03-19T11:30:28.037 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0559999999999999e-06 2024-03-19T11:30:28.037 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.037 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.038 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:28.038 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.870408+0000", 2024-03-19T11:30:28.038 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012978499999999999 2024-03-19T11:30:28.038 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.038 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.038 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:28.038 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.870437+0000", 2024-03-19T11:30:28.038 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9638999999999999e-05 2024-03-19T11:30:28.038 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.038 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.038 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:28.038 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.870460+0000", 2024-03-19T11:30:28.038 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2517000000000001e-05 2024-03-19T11:30:28.038 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.039 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:28.039 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.870681+0000", 2024-03-19T11:30:28.039 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022102800000000001 2024-03-19T11:30:28.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.039 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:28.039 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.870735+0000", 2024-03-19T11:30:28.039 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4616e-05 2024-03-19T11:30:28.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.039 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:28.039 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.883773+0000", 2024-03-19T11:30:28.039 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013038088 2024-03-19T11:30:28.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.040 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.040 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:28.040 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.883886+0000", 2024-03-19T11:30:28.040 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000112224 2024-03-19T11:30:28.040 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:28.040 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:28.040 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:28.040 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.040 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.040 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1416 4.18 4:1dabc0f1:::smithi03829856-1:head [write 564735~613734,stat] snapc c=[c,b,a,6,5] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:28.040 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.949508+0000", 2024-03-19T11:30:28.040 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 2.0461238339999999, 2024-03-19T11:30:28.040 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016253106, 2024-03-19T11:30:28.040 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:28.041 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:28.041 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:28.041 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:28.041 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:28.041 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1416 2024-03-19T11:30:28.041 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.041 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:28.041 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.041 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:28.041 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.949508+0000", 2024-03-19T11:30:28.041 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:28.041 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.041 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.041 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:28.042 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.949508+0000", 2024-03-19T11:30:28.042 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:28.042 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.042 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.042 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:28.042 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.949511+0000", 2024-03-19T11:30:28.042 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2370000000000001e-06 2024-03-19T11:30:28.042 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.042 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.042 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:28.042 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.950555+0000", 2024-03-19T11:30:28.042 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001043925 2024-03-19T11:30:28.042 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.042 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.043 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:28.043 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.950594+0000", 2024-03-19T11:30:28.043 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9351999999999999e-05 2024-03-19T11:30:28.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.043 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:28.043 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.950612+0000", 2024-03-19T11:30:28.043 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7524e-05 2024-03-19T11:30:28.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.043 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:28.043 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.950745+0000", 2024-03-19T11:30:28.043 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013291 2024-03-19T11:30:28.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.044 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:28.044 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.951003+0000", 2024-03-19T11:30:28.044 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025828400000000001 2024-03-19T11:30:28.044 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.044 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.044 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:28.044 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.951335+0000", 2024-03-19T11:30:28.044 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033130200000000001 2024-03-19T11:30:28.044 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.044 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.044 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:28.044 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.965379+0000", 2024-03-19T11:30:28.044 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014044581 2024-03-19T11:30:28.044 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.045 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.045 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:28.045 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.965502+0000", 2024-03-19T11:30:28.045 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012290099999999999 2024-03-19T11:30:28.045 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.045 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.045 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:28.045 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.965668+0000", 2024-03-19T11:30:28.045 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016619200000000001 2024-03-19T11:30:28.045 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.045 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.045 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:28.045 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.965761+0000", 2024-03-19T11:30:28.045 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2898000000000004e-05 2024-03-19T11:30:28.046 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:28.046 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:28.046 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:28.046 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.046 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.046 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1417 4.18 4:1dabc0f1:::smithi03829856-1:head [write 1828021~669034] snapc c=[c,b,a,6,5] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:28.046 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.958407+0000", 2024-03-19T11:30:28.046 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 2.037224433, 2024-03-19T11:30:28.046 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016467840000000001, 2024-03-19T11:30:28.046 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:28.046 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:28.046 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:28.046 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:28.046 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:28.047 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1417 2024-03-19T11:30:28.047 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.047 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:28.047 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.047 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:28.047 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.958407+0000", 2024-03-19T11:30:28.047 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:28.047 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.047 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.047 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:28.047 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.958407+0000", 2024-03-19T11:30:28.047 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:28.047 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.047 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.048 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:28.048 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.958410+0000", 2024-03-19T11:30:28.048 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9830000000000001e-06 2024-03-19T11:30:28.048 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.048 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.048 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:28.048 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.959442+0000", 2024-03-19T11:30:28.048 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001031957 2024-03-19T11:30:28.048 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.048 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.048 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:28.048 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.959475+0000", 2024-03-19T11:30:28.048 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3145999999999997e-05 2024-03-19T11:30:28.048 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.048 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.049 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:28.049 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.959494+0000", 2024-03-19T11:30:28.049 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8994000000000001e-05 2024-03-19T11:30:28.049 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.049 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.049 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:28.049 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.959717+0000", 2024-03-19T11:30:28.049 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000222272 2024-03-19T11:30:28.049 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.049 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.049 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:28.049 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.960015+0000", 2024-03-19T11:30:28.049 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029870599999999998 2024-03-19T11:30:28.049 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.050 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.050 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:28.050 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.960312+0000", 2024-03-19T11:30:28.050 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029640299999999999 2024-03-19T11:30:28.050 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.050 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.050 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:28.050 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.966435+0000", 2024-03-19T11:30:28.050 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061234999999999996 2024-03-19T11:30:28.050 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.050 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.050 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:28.050 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.974556+0000", 2024-03-19T11:30:28.050 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.008120759 2024-03-19T11:30:28.051 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.051 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.051 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:28.051 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.974729+0000", 2024-03-19T11:30:28.051 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000173191 2024-03-19T11:30:28.051 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.051 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.051 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:28.051 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.974875+0000", 2024-03-19T11:30:28.051 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014592899999999999 2024-03-19T11:30:28.051 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:28.051 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:28.051 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:28.051 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.052 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.052 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1419 4.18 4:1dabc0f1:::smithi03829856-1:head [setxattr _header (56) in=7b,truncate 3764735] snapc c=[c,b,a,6,5] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:28.052 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.968061+0000", 2024-03-19T11:30:28.052 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 2.0275705739999998, 2024-03-19T11:30:28.052 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012422488000000001, 2024-03-19T11:30:28.052 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:28.052 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:28.052 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:28.052 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:28.052 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:28.052 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1419 2024-03-19T11:30:28.052 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.052 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:28.052 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.053 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:28.053 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968061+0000", 2024-03-19T11:30:28.053 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:28.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.053 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.053 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:28.053 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968061+0000", 2024-03-19T11:30:28.053 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:28.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.053 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.053 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:28.053 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968063+0000", 2024-03-19T11:30:28.053 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.767e-06 2024-03-19T11:30:28.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.053 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.054 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:28.054 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968074+0000", 2024-03-19T11:30:28.054 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0525e-05 2024-03-19T11:30:28.054 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.054 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:28.054 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968096+0000", 2024-03-19T11:30:28.054 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2926999999999999e-05 2024-03-19T11:30:28.054 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.054 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:28.054 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968113+0000", 2024-03-19T11:30:28.054 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6665000000000001e-05 2024-03-19T11:30:28.054 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.055 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.055 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:28.055 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.973218+0000", 2024-03-19T11:30:28.055 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0051044920000000004 2024-03-19T11:30:28.055 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.055 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.055 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:28.055 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.973511+0000", 2024-03-19T11:30:28.055 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029303899999999998 2024-03-19T11:30:28.055 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.055 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.055 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:28.055 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.973931+0000", 2024-03-19T11:30:28.055 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00041991400000000002 2024-03-19T11:30:28.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.056 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:28.056 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.977029+0000", 2024-03-19T11:30:28.056 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030983040000000001 2024-03-19T11:30:28.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.056 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:28.056 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.980091+0000", 2024-03-19T11:30:28.056 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003061851 2024-03-19T11:30:28.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.056 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:28.056 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.980285+0000", 2024-03-19T11:30:28.056 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000194279 2024-03-19T11:30:28.057 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.057 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:28.057 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.980484+0000", 2024-03-19T11:30:28.057 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019872499999999999 2024-03-19T11:30:28.057 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:28.057 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:28.057 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:28.057 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.057 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1420 4.18 4:1dabc0f1:::smithi03829856-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:28.057 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.968138+0000", 2024-03-19T11:30:28.057 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 2.0274933640000001, 2024-03-19T11:30:28.057 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013109305999999999, 2024-03-19T11:30:28.058 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:28.058 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:28.058 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:28.058 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:28.058 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:28.058 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1420 2024-03-19T11:30:28.058 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.058 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:28.058 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.058 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:28.058 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968138+0000", 2024-03-19T11:30:28.058 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:28.058 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.058 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.059 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:28.059 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968138+0000", 2024-03-19T11:30:28.059 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:28.059 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.059 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.059 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:28.059 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968140+0000", 2024-03-19T11:30:28.059 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4139999999999999e-06 2024-03-19T11:30:28.059 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.059 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.059 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:28.059 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968147+0000", 2024-03-19T11:30:28.059 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.815e-06 2024-03-19T11:30:28.059 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.059 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.060 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:28.060 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968167+0000", 2024-03-19T11:30:28.060 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0018e-05 2024-03-19T11:30:28.060 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.060 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.060 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:28.060 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968182+0000", 2024-03-19T11:30:28.060 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5475e-05 2024-03-19T11:30:28.060 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.060 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.060 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:28.060 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.974950+0000", 2024-03-19T11:30:28.060 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.006767856 2024-03-19T11:30:28.060 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.061 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-03-19T11:30:28.061 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.975095+0000", 2024-03-19T11:30:28.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014501499999999999 2024-03-19T11:30:28.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.061 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:28.061 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.980614+0000", 2024-03-19T11:30:28.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0055184789999999997 2024-03-19T11:30:28.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.061 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:28.061 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.980912+0000", 2024-03-19T11:30:28.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029858199999999998 2024-03-19T11:30:28.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:28.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.981248+0000", 2024-03-19T11:30:28.062 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033565199999999998 2024-03-19T11:30:28.062 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:28.062 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:28.062 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:28.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.062 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1443 4.2 4:43b32242:::smithi03829856-22:head [read 0~2352947,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:28.062 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.979417+0000", 2024-03-19T11:30:28.062 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 2.016214738, 2024-03-19T11:30:28.062 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.96719988700000004, 2024-03-19T11:30:28.063 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:28.063 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:28.063 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:28.063 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:28.063 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:28.063 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1443 2024-03-19T11:30:28.063 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.063 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:28.063 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.063 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:28.063 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979417+0000", 2024-03-19T11:30:28.063 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:28.063 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.063 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.064 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:28.064 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979417+0000", 2024-03-19T11:30:28.064 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:28.064 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.064 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.064 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:28.064 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979420+0000", 2024-03-19T11:30:28.064 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.366e-06 2024-03-19T11:30:28.064 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.064 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.064 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:28.064 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979437+0000", 2024-03-19T11:30:28.064 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6192e-05 2024-03-19T11:30:28.064 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.064 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.065 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:28.065 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979474+0000", 2024-03-19T11:30:28.065 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7601000000000001e-05 2024-03-19T11:30:28.065 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.065 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.065 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:28.065 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979496+0000", 2024-03-19T11:30:28.065 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1506999999999999e-05 2024-03-19T11:30:28.065 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.065 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.065 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:28.065 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979712+0000", 2024-03-19T11:30:28.065 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021641299999999999 2024-03-19T11:30:28.065 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.066 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.066 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:30:28.066 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979746+0000", 2024-03-19T11:30:28.066 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3386000000000002e-05 2024-03-19T11:30:28.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.066 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.066 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:28.066 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.931552+0000", 2024-03-19T11:30:28.066 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.95180659899999998 2024-03-19T11:30:28.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.066 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.066 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-03-19T11:30:28.066 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.934681+0000", 2024-03-19T11:30:28.066 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0031292289999999999 2024-03-19T11:30:28.067 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.067 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.067 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:28.067 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.942701+0000", 2024-03-19T11:30:28.067 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0080198809999999995 2024-03-19T11:30:28.067 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.067 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.067 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:28.067 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.942839+0000", 2024-03-19T11:30:28.067 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013816999999999999 2024-03-19T11:30:28.067 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.067 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.067 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:28.067 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.946617+0000", 2024-03-19T11:30:28.068 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003777543 2024-03-19T11:30:28.068 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:28.068 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:28.068 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:28.068 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.068 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.068 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1444 4.2 4:43b32242:::smithi03829856-22:head [read 0~2352947,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:28.068 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.979553+0000", 2024-03-19T11:30:28.068 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 2.0160790350000002, 2024-03-19T11:30:28.068 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.97015945400000003, 2024-03-19T11:30:28.068 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:28.068 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:28.068 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:28.068 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:28.069 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:28.069 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1444 2024-03-19T11:30:28.069 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.069 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:28.069 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.069 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:28.069 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979553+0000", 2024-03-19T11:30:28.069 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:28.069 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.069 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.069 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:28.069 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979553+0000", 2024-03-19T11:30:28.069 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:28.069 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.070 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.070 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:28.070 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979554+0000", 2024-03-19T11:30:28.070 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.403e-06 2024-03-19T11:30:28.070 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.070 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.070 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:28.070 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979564+0000", 2024-03-19T11:30:28.070 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0032e-05 2024-03-19T11:30:28.070 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.070 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.070 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:28.070 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979579+0000", 2024-03-19T11:30:28.071 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4875000000000001e-05 2024-03-19T11:30:28.071 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.071 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.071 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:28.071 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979595+0000", 2024-03-19T11:30:28.071 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6300999999999999e-05 2024-03-19T11:30:28.071 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.071 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.071 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:28.071 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979874+0000", 2024-03-19T11:30:28.071 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000279065 2024-03-19T11:30:28.071 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.071 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.071 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:30:28.072 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979904+0000", 2024-03-19T11:30:28.072 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.94e-05 2024-03-19T11:30:28.072 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.072 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.072 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:28.072 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.934781+0000", 2024-03-19T11:30:28.072 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.95487744900000004 2024-03-19T11:30:28.072 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.072 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.072 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-03-19T11:30:28.072 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.934946+0000", 2024-03-19T11:30:28.072 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016485999999999999 2024-03-19T11:30:28.072 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.072 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.072 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:28.073 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.946829+0000", 2024-03-19T11:30:28.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011882482999999999 2024-03-19T11:30:28.073 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:28.073 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.946989+0000", 2024-03-19T11:30:28.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000160421 2024-03-19T11:30:28.073 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:28.073 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.949712+0000", 2024-03-19T11:30:28.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002723165 2024-03-19T11:30:28.073 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:28.073 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:28.074 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:28.074 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.074 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.074 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1445 4.2 4:43b32242:::smithi03829856-22:head [read 0~2352947,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:28.074 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.979653+0000", 2024-03-19T11:30:28.074 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 2.0159789039999998, 2024-03-19T11:30:28.074 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.97249507999999996, 2024-03-19T11:30:28.074 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:28.074 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:28.074 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:28.074 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:28.074 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:28.074 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1445 2024-03-19T11:30:28.074 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.075 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:28.075 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.075 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:28.075 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979653+0000", 2024-03-19T11:30:28.075 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:28.075 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.075 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.075 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:28.075 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979653+0000", 2024-03-19T11:30:28.075 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:28.075 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.075 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.075 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:28.075 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979654+0000", 2024-03-19T11:30:28.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.372e-06 2024-03-19T11:30:28.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.076 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:28.076 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979662+0000", 2024-03-19T11:30:28.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9959999999999993e-06 2024-03-19T11:30:28.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.076 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:28.076 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979698+0000", 2024-03-19T11:30:28.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5763000000000003e-05 2024-03-19T11:30:28.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.076 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:28.076 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979714+0000", 2024-03-19T11:30:28.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6283999999999999e-05 2024-03-19T11:30:28.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.077 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:28.077 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979990+0000", 2024-03-19T11:30:28.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000275829 2024-03-19T11:30:28.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.077 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:30:28.077 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.980018+0000", 2024-03-19T11:30:28.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8082999999999999e-05 2024-03-19T11:30:28.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.077 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:28.078 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.935620+0000", 2024-03-19T11:30:28.078 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.95560133800000002 2024-03-19T11:30:28.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.078 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-03-19T11:30:28.078 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.935795+0000", 2024-03-19T11:30:28.078 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000175656 2024-03-19T11:30:28.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.078 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:28.078 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.949835+0000", 2024-03-19T11:30:28.078 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014040083 2024-03-19T11:30:28.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.079 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:28.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.950002+0000", 2024-03-19T11:30:28.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016673 2024-03-19T11:30:28.079 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.079 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.079 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:28.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.952148+0000", 2024-03-19T11:30:28.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021459460000000001 2024-03-19T11:30:28.079 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:28.079 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:28.079 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:28.079 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.079 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.079 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1449 4.18 4:1d58db11:::smithi03829856-18:head [write 581542~688983,stat] snapc c=[c,b,a,6,5] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:28.080 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.989354+0000", 2024-03-19T11:30:28.080 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 2.0062774509999999, 2024-03-19T11:30:28.080 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019237318, 2024-03-19T11:30:28.080 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:28.080 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:28.080 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:28.080 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:28.080 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:28.080 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1449 2024-03-19T11:30:28.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.080 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:28.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.080 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:28.080 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.989354+0000", 2024-03-19T11:30:28.081 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:28.081 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.081 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.081 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:28.081 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.989354+0000", 2024-03-19T11:30:28.081 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:28.081 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.081 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.081 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:28.081 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.989357+0000", 2024-03-19T11:30:28.081 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2740000000000002e-06 2024-03-19T11:30:28.081 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.081 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.081 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:28.082 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.990109+0000", 2024-03-19T11:30:28.082 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00075226500000000001 2024-03-19T11:30:28.082 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.082 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.082 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:28.082 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.990142+0000", 2024-03-19T11:30:28.082 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3328999999999999e-05 2024-03-19T11:30:28.082 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.082 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.082 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:28.082 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.990168+0000", 2024-03-19T11:30:28.082 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6016999999999998e-05 2024-03-19T11:30:28.082 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.082 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.083 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:28.083 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.990291+0000", 2024-03-19T11:30:28.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000122607 2024-03-19T11:30:28.083 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.083 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:28.083 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.990461+0000", 2024-03-19T11:30:28.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000169743 2024-03-19T11:30:28.083 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.083 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:28.083 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.990792+0000", 2024-03-19T11:30:28.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033135100000000002 2024-03-19T11:30:28.083 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.084 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:28.084 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.008238+0000", 2024-03-19T11:30:28.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017445591999999999 2024-03-19T11:30:28.084 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.084 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.084 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:28.084 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.008345+0000", 2024-03-19T11:30:28.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010762 2024-03-19T11:30:28.084 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.084 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.084 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:28.084 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.008436+0000", 2024-03-19T11:30:28.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1116e-05 2024-03-19T11:30:28.084 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.084 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.085 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:28.085 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.008592+0000", 2024-03-19T11:30:28.085 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015540399999999999 2024-03-19T11:30:28.085 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:28.085 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:28.085 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:28.085 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.085 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1450 4.18 4:1d58db11:::smithi03829856-18:head [write 1886503~412692] snapc c=[c,b,a,6,5] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:28.085 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.992973+0000", 2024-03-19T11:30:28.085 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 2.0026586819999999, 2024-03-19T11:30:28.085 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016912580999999999, 2024-03-19T11:30:28.085 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:28.085 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:28.085 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:28.086 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:28.086 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:28.086 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1450 2024-03-19T11:30:28.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.086 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:28.086 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.086 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:28.086 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.992973+0000", 2024-03-19T11:30:28.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:28.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.086 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.086 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:28.086 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.992973+0000", 2024-03-19T11:30:28.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:28.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:28.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.992976+0000", 2024-03-19T11:30:28.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0060000000000001e-06 2024-03-19T11:30:28.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:28.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.993655+0000", 2024-03-19T11:30:28.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00067901099999999998 2024-03-19T11:30:28.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:28.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.993691+0000", 2024-03-19T11:30:28.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5546000000000002e-05 2024-03-19T11:30:28.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.088 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:28.088 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.993713+0000", 2024-03-19T11:30:28.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2277e-05 2024-03-19T11:30:28.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.088 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:28.088 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.997961+0000", 2024-03-19T11:30:28.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0042479659999999997 2024-03-19T11:30:28.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.088 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:28.089 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.998130+0000", 2024-03-19T11:30:28.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016908900000000001 2024-03-19T11:30:28.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.089 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:28.089 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.998397+0000", 2024-03-19T11:30:28.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026707599999999999 2024-03-19T11:30:28.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.089 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:28.089 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.008811+0000", 2024-03-19T11:30:28.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010413751000000001 2024-03-19T11:30:28.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:28.090 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.009539+0000", 2024-03-19T11:30:28.090 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00072811499999999999 2024-03-19T11:30:28.090 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:28.090 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.009711+0000", 2024-03-19T11:30:28.090 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017196900000000001 2024-03-19T11:30:28.090 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:28.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:28.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:28.090 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.009886+0000", 2024-03-19T11:30:28.090 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017477500000000001 2024-03-19T11:30:28.090 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:28.091 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:28.091 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:28.091 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:28.091 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:28.091 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:28.092 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:30:28.092 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:30:28.093 INFO:tasks.rados.rados.0.smithi038.stdout:839: expect (ObjNum 21 snap 0 seq_num 21) 2024-03-19T11:30:28.093 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:30:28.093 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:30:28.094 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:28.027+0000 7fa5de8d7640 -1 osd.3 50 handle_osd_map injecting map crc failure 2024-03-19T11:30:28.094 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-19T11:30:28.094 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:28.102+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:28.095 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:30:28.095 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:30:28.095 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:30:28.096 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:30:28.096 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:30:28.096 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:30:28.097 INFO:tasks.rados.rados.0.smithi038.stdout:847: finishing write tid 1 to smithi03829856-44 2024-03-19T11:30:28.097 INFO:tasks.rados.rados.0.smithi038.stdout:847: finishing write tid 2 to smithi03829856-44 2024-03-19T11:30:28.097 INFO:tasks.rados.rados.0.smithi038.stdout:835: done (8 left) 2024-03-19T11:30:28.097 INFO:tasks.rados.rados.0.smithi038.stdout:838: done (7 left) 2024-03-19T11:30:28.097 INFO:tasks.rados.rados.0.smithi038.stdout:839: done (6 left) 2024-03-19T11:30:28.097 INFO:tasks.rados.rados.0.smithi038.stdout:852: done (5 left) 2024-03-19T11:30:28.097 INFO:tasks.rados.rados.0.smithi038.stdout:853: read oid 19 snap -1 2024-03-19T11:30:28.097 INFO:tasks.rados.rados.0.smithi038.stdout:853: expect (ObjNum 93 snap 10 seq_num 93) 2024-03-19T11:30:28.097 INFO:tasks.rados.rados.0.smithi038.stdout:854: delete oid 48 current snap is 11 2024-03-19T11:30:28.097 INFO:tasks.rados.rados.0.smithi038.stdout:847: finishing write tid 3 to smithi03829856-44 2024-03-19T11:30:28.098 INFO:tasks.rados.rados.0.smithi038.stdout:847: finishing write tid 4 to smithi03829856-44 2024-03-19T11:30:28.098 INFO:tasks.rados.rados.0.smithi038.stdout:847: finishing write tid 5 to smithi03829856-44 2024-03-19T11:30:28.098 INFO:tasks.rados.rados.0.smithi038.stdout:847: oid 44 updating version 0 to 48 2024-03-19T11:30:28.098 INFO:tasks.rados.rados.0.smithi038.stdout:847: oid 44 updating version 48 to 50 2024-03-19T11:30:28.098 INFO:tasks.rados.rados.0.smithi038.stdout:847: oid 44 version 50 is already newer than 47 2024-03-19T11:30:28.098 INFO:tasks.rados.rados.0.smithi038.stdout:847: oid 44 version 50 is already newer than 49 2024-03-19T11:30:28.098 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 44 v 50 (ObjNum 108 snap 11 seq_num 108) dirty exists 2024-03-19T11:30:28.098 INFO:tasks.rados.rados.0.smithi038.stdout:847: left oid 44 (ObjNum 108 snap 11 seq_num 108) 2024-03-19T11:30:28.098 INFO:tasks.rados.rados.0.smithi038.stdout:850: expect (ObjNum 68 snap 3 seq_num 68) 2024-03-19T11:30:28.110 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:30:28.110 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:30:28.133 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:30:28.133 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:30:28.194 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:28.206+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:28.240 INFO:tasks.rados.rados.0.smithi038.stdout:849: expect (ObjNum 22 snap 0 seq_num 22) 2024-03-19T11:30:28.262 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:28.262 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:28.275 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-19T11:30:28.317 INFO:tasks.rados.rados.0.smithi038.stdout:851: expect (ObjNum 50 snap 0 seq_num 50) 2024-03-19T11:30:28.318 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:28.330+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:28.395 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:28.407+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:28.436 INFO:tasks.rados.rados.0.smithi038.stdout:848: finishing set_chunk to oid 28 2024-03-19T11:30:28.436 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 28 v 27 (ObjNum 87 snap 8 seq_num 87) dirty exists 2024-03-19T11:30:28.436 INFO:tasks.rados.rados.0.smithi038.stdout:853: expect (ObjNum 93 snap 10 seq_num 93) 2024-03-19T11:30:28.441 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:28.441 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:28.454 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-19T11:30:28.498 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:28.510+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:28.576 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:28.576 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:28.590 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-19T11:30:28.597 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:28.607+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:28.626 INFO:tasks.rados.rados.0.smithi038.stdout:847: done (6 left) 2024-03-19T11:30:28.627 INFO:tasks.rados.rados.0.smithi038.stdout:848: done (5 left) 2024-03-19T11:30:28.627 INFO:tasks.rados.rados.0.smithi038.stdout:849: done (4 left) 2024-03-19T11:30:28.627 INFO:tasks.rados.rados.0.smithi038.stdout:850: done (3 left) 2024-03-19T11:30:28.627 INFO:tasks.rados.rados.0.smithi038.stdout:851: done (2 left) 2024-03-19T11:30:28.627 INFO:tasks.rados.rados.0.smithi038.stdout:853: done (1 left) 2024-03-19T11:30:28.627 INFO:tasks.rados.rados.0.smithi038.stdout:854: done (0 left) 2024-03-19T11:30:28.627 INFO:tasks.rados.rados.0.smithi038.stdout:855: rollback oid 40 current snap is 11 2024-03-19T11:30:28.627 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 40 to 10 2024-03-19T11:30:28.627 INFO:tasks.rados.rados.0.smithi038.stdout:856: tier_promote oid 14 2024-03-19T11:30:28.627 INFO:tasks.rados.rados.0.smithi038.stdout:857: set_chunk oid 11 target oid 2024-03-19T11:30:28.627 INFO:tasks.rados.rados.0.smithi038.stdout:857: set_chunk oid 11 offset: 0 length: 4000000 target oid smithi03829856- offset: 0 snap -1 2024-03-19T11:30:28.627 INFO:tasks.rados.rados.0.smithi038.stdout:858: read oid 5 snap -1 2024-03-19T11:30:28.627 INFO:tasks.rados.rados.0.smithi038.stdout:858: expect (ObjNum 4 snap 0 seq_num 4) 2024-03-19T11:30:28.628 INFO:tasks.rados.rados.0.smithi038.stdout:859: set_chunk oid 7 target oid 2024-03-19T11:30:28.670 INFO:tasks.rados.rados.0.smithi038.stdout:859: set_chunk oid 7 offset: 94720 length: 2314752 target oid 3e1d9741b3fad8b2c483e9e1cc06aa66738e3b920a7e3bc3a87a95247d3eaba4 offset: 0 snap -1 2024-03-19T11:30:28.670 INFO:tasks.rados.rados.0.smithi038.stdout:860: tier_promote oid 50 2024-03-19T11:30:28.670 INFO:tasks.rados.rados.0.smithi038.stdout:856: got (0) Success 2024-03-19T11:30:28.670 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 14 v 24 (ObjNum 88 snap 8 seq_num 88) dirty exists 2024-03-19T11:30:28.670 INFO:tasks.rados.rados.0.smithi038.stdout:856: done (5 left) 2024-03-19T11:30:28.670 INFO:tasks.rados.rados.0.smithi038.stdout:861: write (excl) oid 31 current snap is 11 2024-03-19T11:30:28.670 INFO:tasks.rados.rados.0.smithi038.stdout:861: seq_num 111 ranges {665577=781581,2062560=563688,3182305=643664} 2024-03-19T11:30:28.685 INFO:tasks.rados.rados.0.smithi038.stdout:861: writing smithi03829856-31 from 665577 to 1447158 tid 1 2024-03-19T11:30:28.698 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:28.710+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:28.700 INFO:tasks.rados.rados.0.smithi038.stdout:861: writing smithi03829856-31 from 2062560 to 2626248 tid 2 2024-03-19T11:30:28.714 INFO:tasks.rados.rados.0.smithi038.stdout:861: writing smithi03829856-31 from 3182305 to 3825969 tid 3 2024-03-19T11:30:28.715 INFO:tasks.rados.rados.0.smithi038.stdout:862: tier_flush oid 39 2024-03-19T11:30:28.715 INFO:tasks.rados.rados.0.smithi038.stdout:862: tier_flush oid 39 snap -1 2024-03-19T11:30:28.715 INFO:tasks.rados.rados.0.smithi038.stdout:858: expect (ObjNum 4 snap 0 seq_num 4) 2024-03-19T11:30:28.759 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:28.759 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:28.773 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-19T11:30:28.783 INFO:tasks.rados.rados.0.smithi038.stdout:857: finishing set_chunk to oid 11 2024-03-19T11:30:28.783 INFO:tasks.rados.rados.0.smithi038.stdout:857: got expected ERANGE (src dne) 2024-03-19T11:30:28.783 INFO:tasks.rados.rados.0.smithi038.stdout:855: finishing rollback tid 0 to smithi03829856-40 2024-03-19T11:30:28.783 INFO:tasks.rados.rados.0.smithi038.stdout:855: finishing rollback tid 1 to smithi03829856-40 2024-03-19T11:30:28.783 INFO:tasks.rados.rados.0.smithi038.stdout:855: finishing rollback tid 2 to smithi03829856-40 2024-03-19T11:30:28.783 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 40 v 36 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-03-19T11:30:28.784 INFO:tasks.rados.rados.0.smithi038.stdout:859: finishing set_chunk to oid 7 2024-03-19T11:30:28.784 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 7 set_chunk returned error code -95 offset: 94720 length: 2314752 2024-03-19T11:30:28.784 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 7 v 0 (ObjNum 67 snap 3 seq_num 67) dirty exists 2024-03-19T11:30:28.784 INFO:tasks.rados.rados.0.smithi038.stdout:860: got (0) Success 2024-03-19T11:30:28.784 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 50 v 15 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-03-19T11:30:28.784 INFO:tasks.rados.rados.0.smithi038.stdout:861: finishing write tid 1 to smithi03829856-31 2024-03-19T11:30:28.784 INFO:tasks.rados.rados.0.smithi038.stdout:861: finishing write tid 2 to smithi03829856-31 2024-03-19T11:30:28.784 INFO:tasks.rados.rados.0.smithi038.stdout:855: done (6 left) 2024-03-19T11:30:28.784 INFO:tasks.rados.rados.0.smithi038.stdout:857: done (5 left) 2024-03-19T11:30:28.784 INFO:tasks.rados.rados.0.smithi038.stdout:858: done (4 left) 2024-03-19T11:30:28.784 INFO:tasks.rados.rados.0.smithi038.stdout:859: done (3 left) 2024-03-19T11:30:28.784 INFO:tasks.rados.rados.0.smithi038.stdout:860: done (2 left) 2024-03-19T11:30:28.784 INFO:tasks.rados.rados.0.smithi038.stdout:863: rollback oid 5 current snap is 11 2024-03-19T11:30:28.784 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 5 to 8 2024-03-19T11:30:28.784 INFO:tasks.rados.rados.0.smithi038.stdout:862: got (16) Device or resource busy 2024-03-19T11:30:28.785 INFO:tasks.rados.rados.0.smithi038.stdout:check_oldest_snap_flushed oid 39 v 0 (ObjNum 97 snap 0 seq_num 1) unflushed 10 2024-03-19T11:30:28.785 INFO:tasks.rados.rados.0.smithi038.stdout:861: finishing write tid 3 to smithi03829856-31 2024-03-19T11:30:28.785 INFO:tasks.rados.rados.0.smithi038.stdout:861: finishing write tid 4 to smithi03829856-31 2024-03-19T11:30:28.785 INFO:tasks.rados.rados.0.smithi038.stdout:861: finishing write tid 5 to smithi03829856-31 2024-03-19T11:30:28.785 INFO:tasks.rados.rados.0.smithi038.stdout:861: oid 31 updating version 0 to 79 2024-03-19T11:30:28.785 INFO:tasks.rados.rados.0.smithi038.stdout:861: oid 31 version 79 is already newer than 77 2024-03-19T11:30:28.785 INFO:tasks.rados.rados.0.smithi038.stdout:861: oid 31 version 79 is already newer than 78 2024-03-19T11:30:28.785 INFO:tasks.rados.rados.0.smithi038.stdout:861: oid 31 version 79 is already newer than 76 2024-03-19T11:30:28.785 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 31 v 79 (ObjNum 111 snap 11 seq_num 111) dirty exists 2024-03-19T11:30:28.785 INFO:tasks.rados.rados.0.smithi038.stdout:861: left oid 31 (ObjNum 111 snap 11 seq_num 111) 2024-03-19T11:30:28.785 INFO:tasks.rados.rados.0.smithi038.stdout:861: done (2 left) 2024-03-19T11:30:28.785 INFO:tasks.rados.rados.0.smithi038.stdout:862: done (1 left) 2024-03-19T11:30:28.785 INFO:tasks.rados.rados.0.smithi038.stdout:864: write (excl) oid 41 current snap is 11 2024-03-19T11:30:28.785 INFO:tasks.rados.rados.0.smithi038.stdout:864: seq_num 112 ranges {682384=456830,1721042=423699,2809728=454936,3731357=151027} 2024-03-19T11:30:28.795 INFO:tasks.rados.rados.0.smithi038.stdout:864: writing smithi03829856-41 from 682384 to 1139214 tid 1 2024-03-19T11:30:28.799 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:28.811+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:28.806 INFO:tasks.rados.rados.0.smithi038.stdout:864: writing smithi03829856-41 from 1721042 to 2144741 tid 2 2024-03-19T11:30:28.818 INFO:tasks.rados.rados.0.smithi038.stdout:864: writing smithi03829856-41 from 2809728 to 3264664 tid 3 2024-03-19T11:30:28.825 INFO:tasks.rados.rados.0.smithi038.stdout:864: writing smithi03829856-41 from 3731357 to 3882384 tid 4 2024-03-19T11:30:28.825 INFO:tasks.rados.rados.0.smithi038.stdout:865: write (excl) oid 26 current snap is 11 2024-03-19T11:30:28.825 INFO:tasks.rados.rados.0.smithi038.stdout:863: finishing rollback tid 0 to smithi03829856-5 2024-03-19T11:30:28.826 INFO:tasks.rados.rados.0.smithi038.stdout:863: finishing rollback tid 1 to smithi03829856-5 2024-03-19T11:30:28.826 INFO:tasks.rados.rados.0.smithi038.stdout:863: finishing rollback tid 2 to smithi03829856-5 2024-03-19T11:30:28.826 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 5 v 43 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-03-19T11:30:28.826 INFO:tasks.rados.rados.0.smithi038.stdout:864: finishing write tid 1 to smithi03829856-41 2024-03-19T11:30:28.826 INFO:tasks.rados.rados.0.smithi038.stdout:864: finishing write tid 2 to smithi03829856-41 2024-03-19T11:30:28.826 INFO:tasks.rados.rados.0.smithi038.stdout:864: finishing write tid 3 to smithi03829856-41 2024-03-19T11:30:28.826 INFO:tasks.rados.rados.0.smithi038.stdout:865: seq_num 113 ranges {699191=532079,1779524=567357,2837151=666208} 2024-03-19T11:30:28.834 INFO:tasks.rados.rados.0.smithi038.stdout:865: writing smithi03829856-26 from 699191 to 1231270 tid 1 2024-03-19T11:30:28.844 INFO:tasks.rados.rados.0.smithi038.stdout:865: writing smithi03829856-26 from 1779524 to 2346881 tid 2 2024-03-19T11:30:28.856 INFO:tasks.rados.rados.0.smithi038.stdout:865: writing smithi03829856-26 from 2837151 to 3503359 tid 3 2024-03-19T11:30:28.856 INFO:tasks.rados.rados.0.smithi038.stdout:863: done (2 left) 2024-03-19T11:30:28.856 INFO:tasks.rados.rados.0.smithi038.stdout:866: read oid 27 snap -1 2024-03-19T11:30:28.856 INFO:tasks.rados.rados.0.smithi038.stdout:866: expect deleted 2024-03-19T11:30:28.857 INFO:tasks.rados.rados.0.smithi038.stdout:867: set_chunk oid 46 target oid 2024-03-19T11:30:28.863 INFO:tasks.rados.rados.0.smithi038.stdout:867: set_chunk oid 46 offset: 637952 length: 125440 target oid 90a682c9229545d9ce8b615cd6ff5f162816e26fd84f23a916a930c7b74b5e4f offset: 0 snap 9 2024-03-19T11:30:28.864 INFO:tasks.rados.rados.0.smithi038.stdout:868: read oid 45 snap -1 2024-03-19T11:30:28.864 INFO:tasks.rados.rados.0.smithi038.stdout:868: expect (ObjNum 59 snap 3 seq_num 59) 2024-03-19T11:30:28.864 INFO:tasks.rados.rados.0.smithi038.stdout:869: read oid 43 snap -1 2024-03-19T11:30:28.864 INFO:tasks.rados.rados.0.smithi038.stdout:869: expect (ObjNum 91 snap 10 seq_num 91) 2024-03-19T11:30:28.864 INFO:tasks.rados.rados.0.smithi038.stdout:870: rollback oid 34 current snap is 11 2024-03-19T11:30:28.865 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 34 to 9 2024-03-19T11:30:28.865 INFO:tasks.rados.rados.0.smithi038.stdout:871: read oid 19 snap -1 2024-03-19T11:30:28.865 INFO:tasks.rados.rados.0.smithi038.stdout:871: expect (ObjNum 93 snap 10 seq_num 93) 2024-03-19T11:30:28.865 INFO:tasks.rados.rados.0.smithi038.stdout:872: write (excl) oid 50 current snap is 11 2024-03-19T11:30:28.865 INFO:tasks.rados.rados.0.smithi038.stdout:872: seq_num 114 ranges {715998=607328,1838006=427368,2864574=477480,3863835=52163} 2024-03-19T11:30:28.877 INFO:tasks.rados.rados.0.smithi038.stdout:872: writing smithi03829856-50 from 715998 to 1323326 tid 1 2024-03-19T11:30:28.886 INFO:tasks.rados.rados.0.smithi038.stdout:872: writing smithi03829856-50 from 1838006 to 2265374 tid 2 2024-03-19T11:30:28.895 INFO:tasks.rados.rados.0.smithi038.stdout:872: writing smithi03829856-50 from 2864574 to 3342054 tid 3 2024-03-19T11:30:28.899 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:28.909+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:28.900 INFO:tasks.rados.rados.0.smithi038.stdout:872: writing smithi03829856-50 from 3863835 to 3915998 tid 4 2024-03-19T11:30:28.900 INFO:tasks.rados.rados.0.smithi038.stdout:873: read oid 31 snap -1 2024-03-19T11:30:28.900 INFO:tasks.rados.rados.0.smithi038.stdout:873: expect (ObjNum 111 snap 11 seq_num 111) 2024-03-19T11:30:28.901 INFO:tasks.rados.rados.0.smithi038.stdout:874: write oid 48 current snap is 11 2024-03-19T11:30:28.901 INFO:tasks.rados.rados.0.smithi038.stdout:874: seq_num 115 ranges {732805=798930,2296488=571026,3291997=405105} 2024-03-19T11:30:28.910 INFO:tasks.rados.rados.0.smithi038.stdout:874: writing smithi03829856-48 from 732805 to 1531735 tid 1 2024-03-19T11:30:28.919 INFO:tasks.rados.rados.0.smithi038.stdout:874: writing smithi03829856-48 from 2296488 to 2867514 tid 2 2024-03-19T11:30:28.925 INFO:tasks.rados.rados.0.smithi038.stdout:874: writing smithi03829856-48 from 3291997 to 3697102 tid 3 2024-03-19T11:30:28.925 INFO:tasks.rados.rados.0.smithi038.stdout:875: read oid 23 snap 8 2024-03-19T11:30:28.925 INFO:tasks.rados.rados.0.smithi038.stdout:875: expect (ObjNum 22 snap 0 seq_num 22) 2024-03-19T11:30:28.925 INFO:tasks.rados.rados.0.smithi038.stdout:876: snap_create 2024-03-19T11:30:28.926 INFO:tasks.rados.rados.0.smithi038.stdout:864: finishing write tid 4 to smithi03829856-41 2024-03-19T11:30:28.926 INFO:tasks.rados.rados.0.smithi038.stdout:864: finishing write tid 5 to smithi03829856-41 2024-03-19T11:30:28.926 INFO:tasks.rados.rados.0.smithi038.stdout:864: finishing write tid 6 to smithi03829856-41 2024-03-19T11:30:28.926 INFO:tasks.rados.rados.0.smithi038.stdout:864: oid 41 updating version 0 to 73 2024-03-19T11:30:28.926 INFO:tasks.rados.rados.0.smithi038.stdout:864: oid 41 version 73 is already newer than 71 2024-03-19T11:30:28.926 INFO:tasks.rados.rados.0.smithi038.stdout:864: oid 41 version 73 is already newer than 72 2024-03-19T11:30:28.926 INFO:tasks.rados.rados.0.smithi038.stdout:864: oid 41 updating version 73 to 74 2024-03-19T11:30:28.926 INFO:tasks.rados.rados.0.smithi038.stdout:864: oid 41 updating version 74 to 75 2024-03-19T11:30:28.926 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 41 v 75 (ObjNum 112 snap 11 seq_num 112) dirty exists 2024-03-19T11:30:28.926 INFO:tasks.rados.rados.0.smithi038.stdout:864: left oid 41 (ObjNum 112 snap 11 seq_num 112) 2024-03-19T11:30:28.926 INFO:tasks.rados.rados.0.smithi038.stdout:865: finishing write tid 1 to smithi03829856-26 2024-03-19T11:30:28.926 INFO:tasks.rados.rados.0.smithi038.stdout:865: finishing write tid 2 to smithi03829856-26 2024-03-19T11:30:28.926 INFO:tasks.rados.rados.0.smithi038.stdout:865: finishing write tid 3 to smithi03829856-26 2024-03-19T11:30:28.927 INFO:tasks.rados.rados.0.smithi038.stdout:865: finishing write tid 4 to smithi03829856-26 2024-03-19T11:30:28.927 INFO:tasks.rados.rados.0.smithi038.stdout:865: finishing write tid 5 to smithi03829856-26 2024-03-19T11:30:28.927 INFO:tasks.rados.rados.0.smithi038.stdout:865: oid 26 updating version 0 to 44 2024-03-19T11:30:28.927 INFO:tasks.rados.rados.0.smithi038.stdout:865: oid 26 updating version 44 to 45 2024-03-19T11:30:28.927 INFO:tasks.rados.rados.0.smithi038.stdout:865: oid 26 updating version 45 to 46 2024-03-19T11:30:28.927 INFO:tasks.rados.rados.0.smithi038.stdout:865: oid 26 updating version 46 to 47 2024-03-19T11:30:28.927 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 26 v 47 (ObjNum 113 snap 11 seq_num 113) dirty exists 2024-03-19T11:30:28.927 INFO:tasks.rados.rados.0.smithi038.stdout:865: left oid 26 (ObjNum 113 snap 11 seq_num 113) 2024-03-19T11:30:28.927 INFO:tasks.rados.rados.0.smithi038.stdout:871: expect (ObjNum 93 snap 10 seq_num 93) 2024-03-19T11:30:28.945 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:28.946 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:28.947 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-03-19T11:30:28.960 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-19T11:30:29.002 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:29.013+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:29.021 INFO:teuthology.orchestra.run.smithi038.stdout:moved 5 / 78 (6.41026%) 2024-03-19T11:30:29.022 INFO:teuthology.orchestra.run.smithi038.stdout:avg 9.75 2024-03-19T11:30:29.022 INFO:teuthology.orchestra.run.smithi038.stdout:stddev 2.04634 -> 2.58602 (expected baseline 2.92083) 2024-03-19T11:30:29.022 INFO:teuthology.orchestra.run.smithi038.stdout:min osd.4 with 6 -> 4 pgs (0.615385 -> 0.410256 * mean) 2024-03-19T11:30:29.022 INFO:teuthology.orchestra.run.smithi038.stdout:max osd.5 with 13 -> 13 pgs (1.33333 -> 1.33333 * mean) 2024-03-19T11:30:29.022 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:30:29.022 INFO:teuthology.orchestra.run.smithi038.stdout:oload 110 2024-03-19T11:30:29.022 INFO:teuthology.orchestra.run.smithi038.stdout:max_change 3 2024-03-19T11:30:29.022 INFO:teuthology.orchestra.run.smithi038.stdout:max_change_osds 4 2024-03-19T11:30:29.022 INFO:teuthology.orchestra.run.smithi038.stdout:average_utilization 0.0008 2024-03-19T11:30:29.022 INFO:teuthology.orchestra.run.smithi038.stdout:overload_utilization 0.0008 2024-03-19T11:30:29.022 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4 weight 0.3894 -> 0.2543 2024-03-19T11:30:29.022 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1 weight 0.5681 -> 0.4450 2024-03-19T11:30:29.023 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0 weight 0.7127 -> 0.6006 2024-03-19T11:30:29.026 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:29.035+0000 7f70c4fdb640 -1 osd.0 51 handle_osd_map injecting map crc failure 2024-03-19T11:30:29.101 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:29.109+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:29.129 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:29.129 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:29.144 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-19T11:30:29.169 INFO:tasks.rados.rados.0.smithi038.stdout:867: finishing set_chunk to oid 46 2024-03-19T11:30:29.169 INFO:tasks.rados.rados.0.smithi038.stdout:870: finishing rollback tid 0 to smithi03829856-34 2024-03-19T11:30:29.169 INFO:tasks.rados.rados.0.smithi038.stdout:864: done (12 left) 2024-03-19T11:30:29.170 INFO:tasks.rados.rados.0.smithi038.stdout:865: done (11 left) 2024-03-19T11:30:29.170 INFO:tasks.rados.rados.0.smithi038.stdout:866: done (10 left) 2024-03-19T11:30:29.170 INFO:tasks.rados.rados.0.smithi038.stdout:867: done (9 left) 2024-03-19T11:30:29.170 INFO:tasks.rados.rados.0.smithi038.stdout:871: done (8 left) 2024-03-19T11:30:29.170 INFO:tasks.rados.rados.0.smithi038.stdout:876: done (7 left) 2024-03-19T11:30:29.170 INFO:tasks.rados.rados.0.smithi038.stdout:877: read oid 42 snap -1 2024-03-19T11:30:29.170 INFO:tasks.rados.rados.0.smithi038.stdout:877: expect (ObjNum 98 snap 10 seq_num 98) 2024-03-19T11:30:29.170 INFO:tasks.rados.rados.0.smithi038.stdout:878: read oid 27 snap -1 2024-03-19T11:30:29.170 INFO:tasks.rados.rados.0.smithi038.stdout:878: expect deleted 2024-03-19T11:30:29.171 INFO:tasks.rados.rados.0.smithi038.stdout:879: write (excl) oid 40 current snap is 12 2024-03-19T11:30:29.171 INFO:tasks.rados.rados.0.smithi038.stdout:868: expect (ObjNum 59 snap 3 seq_num 59) 2024-03-19T11:30:29.202 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:29.211+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:29.270 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:29.270 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:29.284 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-19T11:30:29.303 INFO:tasks.rados.rados.0.smithi038.stdout:870: finishing rollback tid 1 to smithi03829856-34 2024-03-19T11:30:29.303 INFO:tasks.rados.rados.0.smithi038.stdout:870: finishing rollback tid 2 to smithi03829856-34 2024-03-19T11:30:29.303 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 34 v 25 (ObjNum 72 snap 5 seq_num 72) dirty exists 2024-03-19T11:30:29.303 INFO:tasks.rados.rados.0.smithi038.stdout:869: expect (ObjNum 91 snap 10 seq_num 91) 2024-03-19T11:30:29.306 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:29.315+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:29.406 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:29.415+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:29.420 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:29.421 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:29.435 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-19T11:30:29.507 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:29.517+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:29.550 INFO:tasks.rados.rados.0.smithi038.stdout:872: finishing write tid 1 to smithi03829856-50 2024-03-19T11:30:29.551 INFO:tasks.rados.rados.0.smithi038.stdout:872: finishing write tid 2 to smithi03829856-50 2024-03-19T11:30:29.551 INFO:tasks.rados.rados.0.smithi038.stdout:872: finishing write tid 3 to smithi03829856-50 2024-03-19T11:30:29.551 INFO:tasks.rados.rados.0.smithi038.stdout:872: finishing write tid 4 to smithi03829856-50 2024-03-19T11:30:29.551 INFO:tasks.rados.rados.0.smithi038.stdout:872: finishing write tid 5 to smithi03829856-50 2024-03-19T11:30:29.551 INFO:tasks.rados.rados.0.smithi038.stdout:872: finishing write tid 6 to smithi03829856-50 2024-03-19T11:30:29.551 INFO:tasks.rados.rados.0.smithi038.stdout:872: oid 50 updating version 0 to 47 2024-03-19T11:30:29.551 INFO:tasks.rados.rados.0.smithi038.stdout:872: oid 50 version 47 is already newer than 46 2024-03-19T11:30:29.551 INFO:tasks.rados.rados.0.smithi038.stdout:872: oid 50 updating version 47 to 49 2024-03-19T11:30:29.551 INFO:tasks.rados.rados.0.smithi038.stdout:872: oid 50 version 49 is already newer than 48 2024-03-19T11:30:29.551 INFO:tasks.rados.rados.0.smithi038.stdout:872: oid 50 version 49 is already newer than 45 2024-03-19T11:30:29.551 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 50 v 49 (ObjNum 114 snap 11 seq_num 114) dirty exists 2024-03-19T11:30:29.552 INFO:tasks.rados.rados.0.smithi038.stdout:872: left oid 50 (ObjNum 114 snap 11 seq_num 114) 2024-03-19T11:30:29.552 INFO:tasks.rados.rados.0.smithi038.stdout:873: expect (ObjNum 111 snap 11 seq_num 111) 2024-03-19T11:30:29.598 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:29.598 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:29.608 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:29.617+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:29.612 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-19T11:30:29.712 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:29.725+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:29.732 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:29.733 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:29.746 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-19T11:30:29.814 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:29.823+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:29.819 INFO:tasks.rados.rados.0.smithi038.stdout:874: finishing write tid 1 to smithi03829856-48 2024-03-19T11:30:29.819 INFO:tasks.rados.rados.0.smithi038.stdout:874: finishing write tid 2 to smithi03829856-48 2024-03-19T11:30:29.819 INFO:tasks.rados.rados.0.smithi038.stdout:874: finishing write tid 3 to smithi03829856-48 2024-03-19T11:30:29.819 INFO:tasks.rados.rados.0.smithi038.stdout:874: finishing write tid 4 to smithi03829856-48 2024-03-19T11:30:29.819 INFO:tasks.rados.rados.0.smithi038.stdout:874: finishing write tid 5 to smithi03829856-48 2024-03-19T11:30:29.819 INFO:tasks.rados.rados.0.smithi038.stdout:874: oid 48 updating version 0 to 55 2024-03-19T11:30:29.819 INFO:tasks.rados.rados.0.smithi038.stdout:874: oid 48 updating version 55 to 56 2024-03-19T11:30:29.820 INFO:tasks.rados.rados.0.smithi038.stdout:874: oid 48 updating version 56 to 58 2024-03-19T11:30:29.820 INFO:tasks.rados.rados.0.smithi038.stdout:874: oid 48 version 58 is already newer than 57 2024-03-19T11:30:29.820 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 48 v 58 (ObjNum 115 snap 11 seq_num 115) dirty exists 2024-03-19T11:30:29.820 INFO:tasks.rados.rados.0.smithi038.stdout:874: left oid 48 (ObjNum 115 snap 11 seq_num 115) 2024-03-19T11:30:29.820 INFO:tasks.rados.rados.0.smithi038.stdout:875: expect (ObjNum 22 snap 0 seq_num 22) 2024-03-19T11:30:29.861 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:29.861 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:29.875 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-19T11:30:29.893 INFO:tasks.rados.rados.0.smithi038.stdout:877: expect (ObjNum 98 snap 10 seq_num 98) 2024-03-19T11:30:29.916 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:29.925+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:30.008 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:30.008 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:30.016 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:30.025+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:30.020 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-19T11:30:30.021 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-03-19T11:30:30.102 INFO:tasks.rados.rados.0.smithi038.stdout:879: seq_num 116 ranges {749612=474179,1954970=431037,2919420=616377} 2024-03-19T11:30:30.109 INFO:tasks.rados.rados.0.smithi038.stdout:879: writing smithi03829856-40 from 749612 to 1223791 tid 1 2024-03-19T11:30:30.116 INFO:tasks.rados.rados.0.smithi038.stdout:879: writing smithi03829856-40 from 1954970 to 2386007 tid 2 2024-03-19T11:30:30.117 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:30.125+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:30.125 INFO:tasks.rados.rados.0.smithi038.stdout:879: writing smithi03829856-40 from 2919420 to 3535797 tid 3 2024-03-19T11:30:30.125 INFO:tasks.rados.rados.0.smithi038.stdout:868: done (9 left) 2024-03-19T11:30:30.125 INFO:tasks.rados.rados.0.smithi038.stdout:869: done (8 left) 2024-03-19T11:30:30.125 INFO:tasks.rados.rados.0.smithi038.stdout:870: done (7 left) 2024-03-19T11:30:30.125 INFO:tasks.rados.rados.0.smithi038.stdout:872: done (6 left) 2024-03-19T11:30:30.125 INFO:tasks.rados.rados.0.smithi038.stdout:873: done (5 left) 2024-03-19T11:30:30.125 INFO:tasks.rados.rados.0.smithi038.stdout:874: done (4 left) 2024-03-19T11:30:30.125 INFO:tasks.rados.rados.0.smithi038.stdout:875: done (3 left) 2024-03-19T11:30:30.125 INFO:tasks.rados.rados.0.smithi038.stdout:877: done (2 left) 2024-03-19T11:30:30.126 INFO:tasks.rados.rados.0.smithi038.stdout:878: done (1 left) 2024-03-19T11:30:30.126 INFO:tasks.rados.rados.0.smithi038.stdout:880: read oid 39 snap -1 2024-03-19T11:30:30.126 INFO:tasks.rados.rados.0.smithi038.stdout:880: expect deleted 2024-03-19T11:30:30.126 INFO:tasks.rados.rados.0.smithi038.stdout:881: snap_create 2024-03-19T11:30:30.126 INFO:tasks.rados.rados.0.smithi038.stdout:879: finishing write tid 1 to smithi03829856-40 2024-03-19T11:30:30.126 INFO:tasks.rados.rados.0.smithi038.stdout:879: finishing write tid 2 to smithi03829856-40 2024-03-19T11:30:30.136 INFO:tasks.rados.rados.0.smithi038.stdout:879: finishing write tid 3 to smithi03829856-40 2024-03-19T11:30:30.136 INFO:tasks.rados.rados.0.smithi038.stdout:879: finishing write tid 4 to smithi03829856-40 2024-03-19T11:30:30.137 INFO:tasks.rados.rados.0.smithi038.stdout:879: finishing write tid 5 to smithi03829856-40 2024-03-19T11:30:30.137 INFO:tasks.rados.rados.0.smithi038.stdout:879: oid 40 updating version 0 to 39 2024-03-19T11:30:30.137 INFO:tasks.rados.rados.0.smithi038.stdout:879: oid 40 updating version 39 to 40 2024-03-19T11:30:30.137 INFO:tasks.rados.rados.0.smithi038.stdout:879: oid 40 updating version 40 to 41 2024-03-19T11:30:30.137 INFO:tasks.rados.rados.0.smithi038.stdout:879: oid 40 version 41 is already newer than 38 2024-03-19T11:30:30.137 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 40 v 41 (ObjNum 116 snap 12 seq_num 116) dirty exists 2024-03-19T11:30:30.137 INFO:tasks.rados.rados.0.smithi038.stdout:879: left oid 40 (ObjNum 116 snap 12 seq_num 116) 2024-03-19T11:30:30.149 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-19T11:30:30.154 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:30.154 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:30.169 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-19T11:30:30.259 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:30.271+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:30.286 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:30:30.286 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:30:30.302 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:30:30.303 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:30:30.318 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:30:30.319 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:30:30.319 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:30.329+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:30.334 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:30:30.335 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:30:30.337 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:30.337 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:30:30.338 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:30:30.338 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:30:30.338 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:30.350 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-19T11:30:30.355 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:30:30.356 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:30:30.376 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:30:30.376 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:30:30.397 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:30:30.397 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:30:30.417 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:30:30.418 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:30:30.419 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:30.431+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:30.469 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:30.469 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:30:30.470 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:30:30.470 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:30:30.470 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.470 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:9 4.d e28/21 4:b4ef97e0:::smithi03829856-3:head v 28'1, mlcod=0'0)", 2024-03-19T11:30:30.470 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.707728+0000", 2024-03-19T11:30:30.470 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 25.773205488999999, 2024-03-19T11:30:30.470 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0037487950000000001, 2024-03-19T11:30:30.470 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:30.470 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:30.470 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:30.470 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.470 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:30.470 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.707728+0000", 2024-03-19T11:30:30.471 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:30.471 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.471 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.471 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:30.471 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.707728+0000", 2024-03-19T11:30:30.471 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:30.471 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.471 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.471 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:30.471 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.707729+0000", 2024-03-19T11:30:30.471 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6899999999999998e-07 2024-03-19T11:30:30.471 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.471 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.472 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:30.472 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.708625+0000", 2024-03-19T11:30:30.472 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00089603899999999995 2024-03-19T11:30:30.472 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.472 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.472 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:30.472 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.708645+0000", 2024-03-19T11:30:30.472 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9629000000000001e-05 2024-03-19T11:30:30.472 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.472 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.472 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:30.472 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.708657+0000", 2024-03-19T11:30:30.472 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2537999999999999e-05 2024-03-19T11:30:30.472 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.473 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.473 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:30.473 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.708740+0000", 2024-03-19T11:30:30.473 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2490999999999993e-05 2024-03-19T11:30:30.473 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.473 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.473 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:30.473 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.708774+0000", 2024-03-19T11:30:30.473 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4362000000000001e-05 2024-03-19T11:30:30.473 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.473 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.473 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:30.473 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.711419+0000", 2024-03-19T11:30:30.473 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0026446830000000001 2024-03-19T11:30:30.474 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.474 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.474 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:30.474 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.711477+0000", 2024-03-19T11:30:30.474 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8684000000000003e-05 2024-03-19T11:30:30.474 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:30.474 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:30.474 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:30.474 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.474 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.474 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:13 4.4 e28/21 4:21eac9b2:::smithi03829856-4:head v 28'1, mlcod=0'0)", 2024-03-19T11:30:30.474 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:04.723258+0000", 2024-03-19T11:30:30.474 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 25.757675494000001, 2024-03-19T11:30:30.474 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0048979840000000002, 2024-03-19T11:30:30.475 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:30.475 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:30.475 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:30.475 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.475 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:30.475 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723258+0000", 2024-03-19T11:30:30.475 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:30.475 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.475 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.475 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:30.475 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723258+0000", 2024-03-19T11:30:30.475 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:30.475 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.475 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.475 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:30.476 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723259+0000", 2024-03-19T11:30:30.476 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.7399999999999999e-07 2024-03-19T11:30:30.476 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.476 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.476 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:30.476 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723812+0000", 2024-03-19T11:30:30.476 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00055371000000000001 2024-03-19T11:30:30.476 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.476 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.476 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:30.476 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723833+0000", 2024-03-19T11:30:30.476 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0990000000000001e-05 2024-03-19T11:30:30.476 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.476 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.476 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:30.477 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723847+0000", 2024-03-19T11:30:30.477 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3774000000000001e-05 2024-03-19T11:30:30.477 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.477 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.477 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:30.477 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723912+0000", 2024-03-19T11:30:30.477 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4758000000000006e-05 2024-03-19T11:30:30.477 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.477 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.477 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:30.477 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.723933+0000", 2024-03-19T11:30:30.477 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1046000000000001e-05 2024-03-19T11:30:30.477 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.477 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.477 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:30.478 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.728103+0000", 2024-03-19T11:30:30.478 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0041695079999999997 2024-03-19T11:30:30.478 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.478 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.478 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:30.478 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:04.728156+0000", 2024-03-19T11:30:30.478 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3924000000000001e-05 2024-03-19T11:30:30.478 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:30.478 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:30.478 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:30.478 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.478 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.478 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:871 4.d e32/21 4:b4ef97e0:::smithi03829856-3:head v 32'19, mlcod=28'16)", 2024-03-19T11:30:30.479 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.029331+0000", 2024-03-19T11:30:30.479 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 21.451602821000002, 2024-03-19T11:30:30.479 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057816339999999999, 2024-03-19T11:30:30.479 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:30.479 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:30.479 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:30.479 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.479 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:30.479 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.029331+0000", 2024-03-19T11:30:30.479 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:30.479 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.479 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.479 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:30.479 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.029331+0000", 2024-03-19T11:30:30.479 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:30.480 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.480 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.480 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:30.480 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.029331+0000", 2024-03-19T11:30:30.480 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0400000000000002e-07 2024-03-19T11:30:30.480 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.480 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.480 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:30.480 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.030605+0000", 2024-03-19T11:30:30.480 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0012733899999999999 2024-03-19T11:30:30.480 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.480 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.480 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:30.480 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.030620+0000", 2024-03-19T11:30:30.481 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5557999999999999e-05 2024-03-19T11:30:30.481 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.481 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.481 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:30.481 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.030630+0000", 2024-03-19T11:30:30.481 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2660000000000007e-06 2024-03-19T11:30:30.481 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.481 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.481 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:30.481 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.030727+0000", 2024-03-19T11:30:30.481 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7452000000000001e-05 2024-03-19T11:30:30.481 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.481 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.481 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:30.482 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.030757+0000", 2024-03-19T11:30:30.482 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0018999999999998e-05 2024-03-19T11:30:30.482 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.482 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.482 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:30.482 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.035063+0000", 2024-03-19T11:30:30.482 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0043055350000000001 2024-03-19T11:30:30.482 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.482 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.482 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:30.482 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.035113+0000", 2024-03-19T11:30:30.482 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.011e-05 2024-03-19T11:30:30.482 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:30.482 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:30.483 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:30.483 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.483 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.483 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(osd.6.13:39 3.0 e32/20 3:3dcd2456:::039914993b6f4082c204b24ae0ff925ebffccc22519067b64afa9eff107cf533:head v 32'54, mlcod=32'52)", 2024-03-19T11:30:30.483 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:09.308854+0000", 2024-03-19T11:30:30.483 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 21.172080306000002, 2024-03-19T11:30:30.483 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003468175, 2024-03-19T11:30:30.483 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:30.483 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:30.483 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:30.483 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.483 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:30.483 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308854+0000", 2024-03-19T11:30:30.483 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:30.484 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.484 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.484 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:30.484 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308854+0000", 2024-03-19T11:30:30.484 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:30.484 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.484 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.484 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:30.484 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.308855+0000", 2024-03-19T11:30:30.484 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5999999999999999e-06 2024-03-19T11:30:30.484 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.484 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.484 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:30.484 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309019+0000", 2024-03-19T11:30:30.485 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000163311 2024-03-19T11:30:30.485 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.485 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.485 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:30.485 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309040+0000", 2024-03-19T11:30:30.485 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1795999999999999e-05 2024-03-19T11:30:30.485 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.485 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.485 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:30.485 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309060+0000", 2024-03-19T11:30:30.485 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9677000000000002e-05 2024-03-19T11:30:30.485 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.485 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.486 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:30.486 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309151+0000", 2024-03-19T11:30:30.486 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0507000000000006e-05 2024-03-19T11:30:30.486 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.486 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.486 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:30.486 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.309187+0000", 2024-03-19T11:30:30.486 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6675999999999999e-05 2024-03-19T11:30:30.486 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.486 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.487 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:30.487 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.312256+0000", 2024-03-19T11:30:30.487 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030692060000000001 2024-03-19T11:30:30.487 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.487 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.487 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:30.487 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:09.312322+0000", 2024-03-19T11:30:30.487 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5401999999999995e-05 2024-03-19T11:30:30.487 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:30.487 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:30.487 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:30.488 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.488 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.488 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:998 4.4 e34/21 4:27909373:::smithi03829856-46:head v 34'37, mlcod=32'32)", 2024-03-19T11:30:30.488 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.370628+0000", 2024-03-19T11:30:30.488 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.110306092999998, 2024-03-19T11:30:30.488 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050953099999999996, 2024-03-19T11:30:30.488 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:30.488 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:30.488 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:30.488 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.488 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:30.488 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370628+0000", 2024-03-19T11:30:30.488 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:30.488 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.489 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.489 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:30.489 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370628+0000", 2024-03-19T11:30:30.489 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:30.489 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.489 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.489 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:30.489 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370629+0000", 2024-03-19T11:30:30.489 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.5900000000000005e-07 2024-03-19T11:30:30.489 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.489 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.489 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:30.489 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370638+0000", 2024-03-19T11:30:30.489 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8189999999999998e-06 2024-03-19T11:30:30.489 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.490 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.490 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:30.490 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370650+0000", 2024-03-19T11:30:30.490 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.275e-05 2024-03-19T11:30:30.490 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.490 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.490 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:30.490 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370658+0000", 2024-03-19T11:30:30.490 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5769999999999996e-06 2024-03-19T11:30:30.490 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.490 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.490 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:30.490 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370747+0000", 2024-03-19T11:30:30.490 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8642999999999998e-05 2024-03-19T11:30:30.491 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.491 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.491 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:30.491 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.370796+0000", 2024-03-19T11:30:30.491 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9428000000000002e-05 2024-03-19T11:30:30.491 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.491 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.491 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:30.491 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.375658+0000", 2024-03-19T11:30:30.491 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0048621519999999998 2024-03-19T11:30:30.491 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.491 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.491 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:30.491 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.375723+0000", 2024-03-19T11:30:30.492 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4981999999999995e-05 2024-03-19T11:30:30.492 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:30.492 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:30.492 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:30.492 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.492 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.492 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:999 4.4 e34/21 4:27909373:::smithi03829856-46:head v 34'38, mlcod=32'32)", 2024-03-19T11:30:30.492 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.372487+0000", 2024-03-19T11:30:30.492 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.108447328, 2024-03-19T11:30:30.492 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0035853069999999998, 2024-03-19T11:30:30.492 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:30.492 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:30.492 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:30.492 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.492 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:30.493 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372487+0000", 2024-03-19T11:30:30.493 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:30.493 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.493 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.493 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:30.493 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372487+0000", 2024-03-19T11:30:30.493 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:30.493 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.493 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.493 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:30.493 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372488+0000", 2024-03-19T11:30:30.493 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1069999999999999e-06 2024-03-19T11:30:30.493 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.493 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.494 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:30.494 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372498+0000", 2024-03-19T11:30:30.494 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7510000000000003e-06 2024-03-19T11:30:30.494 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.494 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.494 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:30.494 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372544+0000", 2024-03-19T11:30:30.494 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6218999999999999e-05 2024-03-19T11:30:30.494 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.494 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.494 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:30.494 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372553+0000", 2024-03-19T11:30:30.495 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6320000000000003e-06 2024-03-19T11:30:30.495 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.495 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.495 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:30.495 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372636+0000", 2024-03-19T11:30:30.495 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3125000000000001e-05 2024-03-19T11:30:30.495 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.495 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.495 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:30.495 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.372651+0000", 2024-03-19T11:30:30.495 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4676e-05 2024-03-19T11:30:30.496 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.496 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.496 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:30.496 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.376026+0000", 2024-03-19T11:30:30.496 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033745630000000001 2024-03-19T11:30:30.496 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.496 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.496 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:30.496 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.376072+0000", 2024-03-19T11:30:30.496 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6233999999999998e-05 2024-03-19T11:30:30.496 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:30.496 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:30.497 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:30.497 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.497 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.497 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:1003 4.4 e34/21 4:22709789:::smithi03829856-41:head v 34'40, mlcod=34'37)", 2024-03-19T11:30:30.497 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.378551+0000", 2024-03-19T11:30:30.497 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.102383240999998, 2024-03-19T11:30:30.497 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0041156680000000003, 2024-03-19T11:30:30.497 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:30.497 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:30.497 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:30.497 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.497 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:30.497 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.378551+0000", 2024-03-19T11:30:30.498 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:30.498 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.498 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.498 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:30.498 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.378551+0000", 2024-03-19T11:30:30.498 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:30.498 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.498 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.498 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:30.498 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.378552+0000", 2024-03-19T11:30:30.498 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0400000000000005e-07 2024-03-19T11:30:30.498 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.498 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.499 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:30.499 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.378929+0000", 2024-03-19T11:30:30.499 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037751700000000001 2024-03-19T11:30:30.499 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.499 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.499 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:30.499 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.378944+0000", 2024-03-19T11:30:30.499 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5038000000000001e-05 2024-03-19T11:30:30.499 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.499 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.499 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:30.499 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.378953+0000", 2024-03-19T11:30:30.499 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7649999999999999e-06 2024-03-19T11:30:30.499 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.500 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.500 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:30.500 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.379017+0000", 2024-03-19T11:30:30.500 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3614000000000005e-05 2024-03-19T11:30:30.500 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.500 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.500 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:30.500 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.379042+0000", 2024-03-19T11:30:30.501 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5910000000000001e-05 2024-03-19T11:30:30.501 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.501 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.501 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:30.501 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.382610+0000", 2024-03-19T11:30:30.501 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0035678340000000002 2024-03-19T11:30:30.501 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.501 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.501 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:30.501 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.382666+0000", 2024-03-19T11:30:30.501 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6085999999999998e-05 2024-03-19T11:30:30.501 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:30.501 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:30.501 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:30.501 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.502 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.502 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:1004 4.4 e34/21 4:22709789:::smithi03829856-41:head v 34'41, mlcod=34'38)", 2024-03-19T11:30:30.502 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.386855+0000", 2024-03-19T11:30:30.502 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.094079432000001, 2024-03-19T11:30:30.502 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0039133320000000003, 2024-03-19T11:30:30.502 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:30.502 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:30.502 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:30.502 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.502 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:30.502 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.386855+0000", 2024-03-19T11:30:30.502 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:30.502 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.502 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.503 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:30.503 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.386855+0000", 2024-03-19T11:30:30.503 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:30.503 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.503 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.503 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:30.503 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.386856+0000", 2024-03-19T11:30:30.503 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.528e-06 2024-03-19T11:30:30.503 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.503 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.503 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:30.503 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.387143+0000", 2024-03-19T11:30:30.503 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00028658600000000002 2024-03-19T11:30:30.503 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.504 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.504 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:30.504 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.387159+0000", 2024-03-19T11:30:30.504 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6796e-05 2024-03-19T11:30:30.504 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.504 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.504 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:30.504 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.387167+0000", 2024-03-19T11:30:30.504 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2729999999999997e-06 2024-03-19T11:30:30.504 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.504 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.504 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:30.504 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.387230+0000", 2024-03-19T11:30:30.505 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2862999999999998e-05 2024-03-19T11:30:30.505 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.505 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.505 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:30.505 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.387255+0000", 2024-03-19T11:30:30.505 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4905e-05 2024-03-19T11:30:30.505 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.505 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.505 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:30.505 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.390720+0000", 2024-03-19T11:30:30.505 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034652699999999999 2024-03-19T11:30:30.505 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.505 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.506 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:30.506 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.390768+0000", 2024-03-19T11:30:30.506 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8111000000000001e-05 2024-03-19T11:30:30.506 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:30.506 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:30.506 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:30.506 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.506 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.506 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:1191 4.4 e40/21 4:27909373:::smithi03829856-46:head v 40'53, mlcod=38'50)", 2024-03-19T11:30:30.506 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:17.966218+0000", 2024-03-19T11:30:30.506 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 12.514715600000001, 2024-03-19T11:30:30.506 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038990449999999999, 2024-03-19T11:30:30.507 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:30.507 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:30.507 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:30.507 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.507 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:30.507 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.966218+0000", 2024-03-19T11:30:30.507 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:30.507 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.507 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.507 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:30.507 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.966218+0000", 2024-03-19T11:30:30.507 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:30.507 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.508 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.508 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:30.508 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.966219+0000", 2024-03-19T11:30:30.508 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.96e-07 2024-03-19T11:30:30.508 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.508 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.508 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:30.508 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.966448+0000", 2024-03-19T11:30:30.508 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00022896799999999999 2024-03-19T11:30:30.508 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.508 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.508 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:30.508 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.966459+0000", 2024-03-19T11:30:30.509 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1654999999999999e-05 2024-03-19T11:30:30.509 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.509 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.509 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:30.509 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.966466+0000", 2024-03-19T11:30:30.509 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3770000000000001e-06 2024-03-19T11:30:30.509 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.509 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.509 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:30.509 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.966536+0000", 2024-03-19T11:30:30.509 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9808999999999995e-05 2024-03-19T11:30:30.509 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.509 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.510 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:30.510 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.966564+0000", 2024-03-19T11:30:30.510 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8319000000000001e-05 2024-03-19T11:30:30.510 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.510 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.510 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:30.510 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.970055+0000", 2024-03-19T11:30:30.510 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034915860000000001 2024-03-19T11:30:30.510 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.510 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.510 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:30.510 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.970117+0000", 2024-03-19T11:30:30.510 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1934999999999996e-05 2024-03-19T11:30:30.511 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:30.511 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:30.511 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:30.511 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.511 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.511 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:1192 4.4 e40/21 4:27909373:::smithi03829856-46:head v 40'54, mlcod=39'51)", 2024-03-19T11:30:30.511 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:17.977421+0000", 2024-03-19T11:30:30.511 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 12.503512758999999, 2024-03-19T11:30:30.511 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0040889289999999998, 2024-03-19T11:30:30.511 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:30.511 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:30.511 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:30.511 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.512 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:30.512 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.977421+0000", 2024-03-19T11:30:30.512 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:30.512 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.512 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.512 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:30.512 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.977421+0000", 2024-03-19T11:30:30.512 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:30.512 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.512 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.512 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:30.512 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.977422+0000", 2024-03-19T11:30:30.513 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.271e-06 2024-03-19T11:30:30.513 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.513 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.513 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:30.513 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.977700+0000", 2024-03-19T11:30:30.513 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00027715299999999998 2024-03-19T11:30:30.513 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.513 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.513 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:30.513 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.977718+0000", 2024-03-19T11:30:30.513 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8067000000000002e-05 2024-03-19T11:30:30.514 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.514 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.514 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:30.514 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.977727+0000", 2024-03-19T11:30:30.514 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4919999999999999e-06 2024-03-19T11:30:30.514 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.514 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.514 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:30.514 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.977792+0000", 2024-03-19T11:30:30.514 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4945999999999998e-05 2024-03-19T11:30:30.514 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.514 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.514 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:30.515 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.977818+0000", 2024-03-19T11:30:30.515 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5519000000000001e-05 2024-03-19T11:30:30.515 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.515 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.515 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:30.515 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.981471+0000", 2024-03-19T11:30:30.515 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0036535980000000001 2024-03-19T11:30:30.515 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.515 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.515 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:30.515 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:17.981510+0000", 2024-03-19T11:30:30.515 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8883e-05 2024-03-19T11:30:30.515 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:30.515 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:30.516 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:30.516 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.516 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.516 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:1215 3.0 e41/20 3:0c9406a2:::98f4ba27fa68608c540dca9b03a4531ee556ff2734aa426809de93c45c84ab96:head v 41'65, mlcod=36'63)", 2024-03-19T11:30:30.516 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.151676+0000", 2024-03-19T11:30:30.516 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 12.329258119, 2024-03-19T11:30:30.516 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0037067860000000001, 2024-03-19T11:30:30.516 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:30.516 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:30.516 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:30.516 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.516 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:30.516 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.151676+0000", 2024-03-19T11:30:30.517 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:30.517 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.517 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.517 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:30.517 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.151676+0000", 2024-03-19T11:30:30.517 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:30.517 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.517 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.517 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:30.517 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.151676+0000", 2024-03-19T11:30:30.517 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2500000000000001e-07 2024-03-19T11:30:30.517 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.517 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.517 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:30.518 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.152439+0000", 2024-03-19T11:30:30.518 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00076323500000000002 2024-03-19T11:30:30.518 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.518 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.518 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:30.518 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.152452+0000", 2024-03-19T11:30:30.518 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2356e-05 2024-03-19T11:30:30.518 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.518 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.518 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:30.518 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.152460+0000", 2024-03-19T11:30:30.518 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2609999999999993e-06 2024-03-19T11:30:30.519 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.519 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.519 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:30.519 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.152543+0000", 2024-03-19T11:30:30.519 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2837000000000007e-05 2024-03-19T11:30:30.519 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.519 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.519 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:30.519 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.152571+0000", 2024-03-19T11:30:30.519 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8228000000000001e-05 2024-03-19T11:30:30.519 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.519 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.519 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:30.520 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155345+0000", 2024-03-19T11:30:30.520 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0027742019999999999 2024-03-19T11:30:30.520 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.520 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.520 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:30.520 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155383+0000", 2024-03-19T11:30:30.520 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7342000000000003e-05 2024-03-19T11:30:30.520 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:30.520 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:30.520 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:30.520 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.520 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.521 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(osd.1.0:142 4.4 e44/43 4:214026b3:::smithi03829856-47:head v 44'57, mlcod=0'0)", 2024-03-19T11:30:30.521 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:21.867611+0000", 2024-03-19T11:30:30.521 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.6133233790000006, 2024-03-19T11:30:30.521 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0081946840000000007, 2024-03-19T11:30:30.521 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:30.521 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:30.521 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:30.521 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.521 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:30.521 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.867611+0000", 2024-03-19T11:30:30.521 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:30.521 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.522 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.522 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:30.522 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.867611+0000", 2024-03-19T11:30:30.522 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:30.522 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.522 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.522 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:30.522 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.867613+0000", 2024-03-19T11:30:30.522 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9250000000000002e-06 2024-03-19T11:30:30.522 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.522 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.522 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:30.523 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.869767+0000", 2024-03-19T11:30:30.523 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002154071 2024-03-19T11:30:30.523 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.523 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.523 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:30.523 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.869792+0000", 2024-03-19T11:30:30.523 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5327e-05 2024-03-19T11:30:30.523 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.523 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.523 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:30.523 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.869803+0000", 2024-03-19T11:30:30.523 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0883999999999999e-05 2024-03-19T11:30:30.523 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.524 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.524 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:30.524 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.869890+0000", 2024-03-19T11:30:30.526 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7049e-05 2024-03-19T11:30:30.526 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.526 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.526 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:30.526 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.869927+0000", 2024-03-19T11:30:30.526 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7465999999999997e-05 2024-03-19T11:30:30.526 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.526 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.526 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:30.526 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.875750+0000", 2024-03-19T11:30:30.526 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0058225969999999997 2024-03-19T11:30:30.526 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.527 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.527 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:30.527 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.875805+0000", 2024-03-19T11:30:30.527 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5365000000000003e-05 2024-03-19T11:30:30.527 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:30.527 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:30.527 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:30.527 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.527 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.527 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:1286 4.d e45/21 4:b4ef97e0:::smithi03829856-3:head v 45'30, mlcod=33'27)", 2024-03-19T11:30:30.527 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:22.881538+0000", 2024-03-19T11:30:30.527 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 7.5993961790000002, 2024-03-19T11:30:30.527 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0073916399999999997, 2024-03-19T11:30:30.527 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:30.527 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:30.528 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:30.528 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.528 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:30.528 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:22.881538+0000", 2024-03-19T11:30:30.528 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:30.528 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.528 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.528 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:30.528 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:22.881538+0000", 2024-03-19T11:30:30.528 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:30.528 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.528 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.528 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:30.529 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:22.881538+0000", 2024-03-19T11:30:30.529 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3399999999999999e-07 2024-03-19T11:30:30.529 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.529 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.529 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:30.529 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:22.882375+0000", 2024-03-19T11:30:30.529 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00083699100000000004 2024-03-19T11:30:30.529 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.529 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.529 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:30.529 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:22.882402+0000", 2024-03-19T11:30:30.529 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6571e-05 2024-03-19T11:30:30.529 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.529 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.529 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:30.530 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:22.882417+0000", 2024-03-19T11:30:30.530 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4822e-05 2024-03-19T11:30:30.530 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.530 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.530 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:30.530 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:22.882549+0000", 2024-03-19T11:30:30.530 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013247899999999999 2024-03-19T11:30:30.530 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.530 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.530 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:30.530 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:22.882591+0000", 2024-03-19T11:30:30.530 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1230000000000003e-05 2024-03-19T11:30:30.530 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.530 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.531 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:30.531 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:22.888839+0000", 2024-03-19T11:30:30.531 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0062484940000000003 2024-03-19T11:30:30.531 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.531 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.531 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:30.531 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:22.888929+0000", 2024-03-19T11:30:30.531 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0419000000000003e-05 2024-03-19T11:30:30.531 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:30.531 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:30.531 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:30.531 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.531 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.531 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:1287 4.d e45/21 4:b4ef97e0:::smithi03829856-3:head v 45'31, mlcod=36'28)", 2024-03-19T11:30:30.532 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:22.896045+0000", 2024-03-19T11:30:30.532 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 7.584888984, 2024-03-19T11:30:30.532 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053379370000000001, 2024-03-19T11:30:30.532 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:30.532 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:30.532 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:30.532 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.532 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:30.532 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:22.896045+0000", 2024-03-19T11:30:30.532 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:30.532 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.532 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.532 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:30.532 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:22.896045+0000", 2024-03-19T11:30:30.533 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:30.533 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.533 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.533 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:30.533 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:22.896047+0000", 2024-03-19T11:30:30.533 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2589999999999999e-06 2024-03-19T11:30:30.533 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.533 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.533 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:30.533 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:22.896577+0000", 2024-03-19T11:30:30.533 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00052966299999999999 2024-03-19T11:30:30.533 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.533 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.533 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:30.533 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:22.896614+0000", 2024-03-19T11:30:30.534 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6610999999999999e-05 2024-03-19T11:30:30.534 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.534 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.534 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:30.534 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:22.896633+0000", 2024-03-19T11:30:30.534 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9927000000000001e-05 2024-03-19T11:30:30.534 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.534 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.534 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:30.534 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:22.896765+0000", 2024-03-19T11:30:30.534 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000131278 2024-03-19T11:30:30.534 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.534 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.534 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:30.535 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:22.896806+0000", 2024-03-19T11:30:30.535 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1433000000000001e-05 2024-03-19T11:30:30.535 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.535 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.535 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:30.535 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:22.901321+0000", 2024-03-19T11:30:30.535 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0045147329999999999 2024-03-19T11:30:30.535 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.535 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.535 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:30.535 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:22.901383+0000", 2024-03-19T11:30:30.535 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2032999999999994e-05 2024-03-19T11:30:30.535 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:30.535 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:30.536 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:30.536 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.536 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.536 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:1414 3.0 e48/20 3:308d3303:::5a675e3a2c8b0683d62c457d46b2f98be18ec92014cc458a4e2237e707e8ab7b:head v 48'78, mlcod=47'76)", 2024-03-19T11:30:30.536 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.935090+0000", 2024-03-19T11:30:30.536 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.5458434990000001, 2024-03-19T11:30:30.536 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005407495, 2024-03-19T11:30:30.536 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:30.536 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:30.536 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:30.536 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.536 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:30.536 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.935090+0000", 2024-03-19T11:30:30.537 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:30.537 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.537 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.537 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:30.537 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.935090+0000", 2024-03-19T11:30:30.537 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:30.537 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.537 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.537 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:30.537 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.935092+0000", 2024-03-19T11:30:30.537 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.928e-06 2024-03-19T11:30:30.537 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.537 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.537 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:30.537 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.936094+0000", 2024-03-19T11:30:30.538 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001001776 2024-03-19T11:30:30.538 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.538 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.538 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:30.538 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.936119+0000", 2024-03-19T11:30:30.538 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4465999999999999e-05 2024-03-19T11:30:30.538 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.538 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.538 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:30.538 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.936130+0000", 2024-03-19T11:30:30.538 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1691e-05 2024-03-19T11:30:30.538 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.538 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.538 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:30.539 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.936220+0000", 2024-03-19T11:30:30.539 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9980000000000002e-05 2024-03-19T11:30:30.539 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.539 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.539 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:30.539 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.936253+0000", 2024-03-19T11:30:30.539 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2234999999999999e-05 2024-03-19T11:30:30.539 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.539 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.539 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:30.539 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.940434+0000", 2024-03-19T11:30:30.539 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.004181059 2024-03-19T11:30:30.539 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.539 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.540 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:30.540 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.940498+0000", 2024-03-19T11:30:30.540 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.436e-05 2024-03-19T11:30:30.540 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:30.540 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:30.540 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:30.540 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.540 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.540 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:1458 3.0 e48/20 3:3530bb15:::d93831687a48e76c865eb26f9c54e5f4aca1a926fe5c356a9bc202c48a9e61ec:head v 48'81, mlcod=48'79)", 2024-03-19T11:30:30.540 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:26.864790+0000", 2024-03-19T11:30:30.540 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.6161441500000002, 2024-03-19T11:30:30.540 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0039561809999999996, 2024-03-19T11:30:30.540 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:30.540 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:30.541 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:30.541 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.541 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:30.541 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.864790+0000", 2024-03-19T11:30:30.541 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:30.541 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.541 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.541 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:30.541 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.864790+0000", 2024-03-19T11:30:30.541 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:30.541 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.541 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.541 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:30.542 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.864792+0000", 2024-03-19T11:30:30.542 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1770000000000001e-06 2024-03-19T11:30:30.542 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.542 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.542 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:30.542 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.865611+0000", 2024-03-19T11:30:30.542 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00081893700000000005 2024-03-19T11:30:30.542 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.542 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.542 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:30.542 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.865643+0000", 2024-03-19T11:30:30.542 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1591000000000003e-05 2024-03-19T11:30:30.542 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.542 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.543 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:30.543 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.865657+0000", 2024-03-19T11:30:30.543 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4406000000000001e-05 2024-03-19T11:30:30.543 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.543 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.543 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:30.543 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.865796+0000", 2024-03-19T11:30:30.543 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013873600000000001 2024-03-19T11:30:30.543 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.543 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.543 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:30.543 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.865840+0000", 2024-03-19T11:30:30.543 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3989999999999997e-05 2024-03-19T11:30:30.543 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.544 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.544 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:30.544 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.868681+0000", 2024-03-19T11:30:30.544 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002841723 2024-03-19T11:30:30.544 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.544 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.544 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:30.544 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.868746+0000", 2024-03-19T11:30:30.544 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4621000000000004e-05 2024-03-19T11:30:30.544 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:30.544 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:30.544 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:30.544 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.545 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.545 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.16 49/48 [PushOp(4:6868af91:::smithi03829856-10:2, version: 29'22, data_included: [551263~793594,1893632~254982], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:2@29'22, size: 2151263, copy_subset: [(0, 2151263)], clone_subset: {}, snapset: b=[]:{2: [2], 7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2148614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:6d9093b1:::smithi03829856-38:4, version: 33'29, data_included: [621859~449625,1865716~598951], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6d9093b1:::smithi03829856-38:4@33'29, size: 2621859, copy_subset: [(0, 2621859)], clone_subset: {}, snapset: 4=[]:{4: [2]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2464667, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:6d9093b1:::smithi03829856-38:head, version: 33'33, data_included: [621859~568773,1729599~479803], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:6d9093b1:::smithi03829856-38:head@33'33, size: 3142876, copy_subset: [(0, 3142876)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2209402, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:6be89c06:::smithi03829856-24:head, version: 36'39, data_included: [741376~589824,1753088~458752], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:6be89c06:::smithi03829856-24:head@36'39, size: 2386561, copy_subset: [(0, 2386561)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2211840, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:6868af91:::smithi03829856-10:7, version: 38'41, data_included: [524385~820472,1602039~228104], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:7@38'41, size: 2924385, copy_subset: [(0, 2924385)], clone_subset: {}, snapset: b=[]:{2: [2], 7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1830143, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:6868af91:::smithi03829856-10:b, version: 47'48, data_included: [524385~820472,1602039~228104], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:b@47'48, size: 2924385, copy_subset: [(0, 2924385)], clone_subset: {}, snapset: b=[]:{2: [2], 7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1830143, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:6868af91:::smithi03829856-10:head, version: 47'52, data_included: [524385~820472,1602039~228104], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:head@47'52, size: 3747928, copy_subset: [(0, 3747928)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1830143, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:30:30.545 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:26.954289+0000", 2024-03-19T11:30:30.545 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.5266449309999999, 2024-03-19T11:30:30.545 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029260781999999999, 2024-03-19T11:30:30.545 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:30.545 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:30.545 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:30.545 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.545 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:30.545 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954289+0000", 2024-03-19T11:30:30.545 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:30.545 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.546 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.546 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:30.546 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954289+0000", 2024-03-19T11:30:30.546 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:30.546 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.546 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.546 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:30.546 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954291+0000", 2024-03-19T11:30:30.546 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8050000000000001e-06 2024-03-19T11:30:30.546 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.546 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.546 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:30.546 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.958775+0000", 2024-03-19T11:30:30.547 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0044845859999999996 2024-03-19T11:30:30.547 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.547 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.547 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:30.547 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.958851+0000", 2024-03-19T11:30:30.547 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6050999999999994e-05 2024-03-19T11:30:30.547 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.547 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.547 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:30.547 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.958931+0000", 2024-03-19T11:30:30.547 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9099999999999998e-05 2024-03-19T11:30:30.547 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.547 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.547 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:30.548 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.959206+0000", 2024-03-19T11:30:30.548 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00027495999999999998 2024-03-19T11:30:30.548 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.548 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.548 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:30.548 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.959246+0000", 2024-03-19T11:30:30.548 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0326e-05 2024-03-19T11:30:30.548 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.548 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.548 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:30.548 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.983550+0000", 2024-03-19T11:30:30.548 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024303953999999999 2024-03-19T11:30:30.548 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:30.548 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:30.549 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:30.549 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.549 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.549 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPushReply(4.d 49/48 [PushReplyOp(4:b65e6658:::smithi03829856-9:head),PushReplyOp(4:b4ef97e0:::smithi03829856-3:b),PushReplyOp(4:b4ef97e0:::smithi03829856-3:head)])", 2024-03-19T11:30:30.549 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:26.962881+0000", 2024-03-19T11:30:30.549 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.5180534159999999, 2024-03-19T11:30:30.549 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0043461029999999996, 2024-03-19T11:30:30.549 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:30.549 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-03-19T11:30:30.549 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:30.549 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.549 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:30.549 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.962881+0000", 2024-03-19T11:30:30.549 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:30.550 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.550 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.550 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:30.550 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.962881+0000", 2024-03-19T11:30:30.550 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:30.550 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.550 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.550 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:30.550 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.962882+0000", 2024-03-19T11:30:30.550 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.702e-06 2024-03-19T11:30:30.550 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.550 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.550 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:30.550 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.962893+0000", 2024-03-19T11:30:30.551 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0505e-05 2024-03-19T11:30:30.551 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.551 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.551 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:30.551 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.962913+0000", 2024-03-19T11:30:30.551 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0191e-05 2024-03-19T11:30:30.551 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.551 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.551 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:30.551 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.962961+0000", 2024-03-19T11:30:30.551 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8099999999999997e-05 2024-03-19T11:30:30.551 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.551 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.551 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:30.551 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.963094+0000", 2024-03-19T11:30:30.552 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013322200000000001 2024-03-19T11:30:30.552 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.552 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.552 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:30.552 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.967227+0000", 2024-03-19T11:30:30.552 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0041323829999999999 2024-03-19T11:30:30.552 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:30.552 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:30.552 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:30.552 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.552 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.552 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.6 49/48 [PushOp(4:626acd4d:::smithi03829856-2:8, version: 39'43, data_included: [416807~475249,1542129~474678], data_size: 949927, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:626acd4d:::smithi03829856-2:8@39'43, size: 2016807, copy_subset: [(0, 2016807)], clone_subset: {}, snapset: b=[]:{8: [8, 6, 5, 2], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2016807, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:626acd4d:::smithi03829856-2:b, version: 45'50, data_included: [416807~595780,1542129~452796], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:626acd4d:::smithi03829856-2:b@45'50, size: 3411788, copy_subset: [(0, 3411788)], clone_subset: {}, snapset: b=[]:{8: [8, 6, 5, 2], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1994925, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:30:30.552 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:26.964817+0000", 2024-03-19T11:30:30.552 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.5161165370000003, 2024-03-19T11:30:30.553 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0079895639999999993, 2024-03-19T11:30:30.553 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:30.553 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:30.553 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:30.553 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.553 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:30.553 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.964817+0000", 2024-03-19T11:30:30.553 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:30.553 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.553 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.553 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:30.553 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.964817+0000", 2024-03-19T11:30:30.553 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:30.553 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.554 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.554 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:30.554 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.964819+0000", 2024-03-19T11:30:30.554 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.079e-06 2024-03-19T11:30:30.554 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.554 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.554 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:30.554 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.966029+0000", 2024-03-19T11:30:30.554 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001210271 2024-03-19T11:30:30.554 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.554 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.554 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:30.554 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.966064+0000", 2024-03-19T11:30:30.555 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5670000000000002e-05 2024-03-19T11:30:30.555 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.555 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.555 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:30.555 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.966105+0000", 2024-03-19T11:30:30.555 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0345000000000001e-05 2024-03-19T11:30:30.555 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.555 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.555 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:30.555 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.966324+0000", 2024-03-19T11:30:30.555 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00021928299999999999 2024-03-19T11:30:30.555 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.555 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.556 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:30.556 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.966344+0000", 2024-03-19T11:30:30.556 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0373e-05 2024-03-19T11:30:30.556 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.556 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.556 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:30.556 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.972807+0000", 2024-03-19T11:30:30.556 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0064625430000000003 2024-03-19T11:30:30.556 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:30.556 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:30.556 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:30.556 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.556 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.557 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.16 49/48 [PushOp(4:6868af91:::smithi03829856-10:2, version: 29'22, data_included: [2148614~2649], data_size: 2649, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:2@29'22, size: 2151263, copy_subset: [(0, 2151263)], clone_subset: {}, snapset: b=[]:{2: [2], 7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2151263, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2148614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:6d9093b1:::smithi03829856-38:4, version: 33'29, data_included: [2464667~157192], data_size: 157192, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:6d9093b1:::smithi03829856-38:4@33'29, size: 2621859, copy_subset: [(0, 2621859)], clone_subset: {}, snapset: 4=[]:{4: [2]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2621859, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2464667, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:6d9093b1:::smithi03829856-38:head, version: 33'33, data_included: [2209402~412457,3101234~41642], data_size: 454099, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:6d9093b1:::smithi03829856-38:head@33'33, size: 3142876, copy_subset: [(0, 3142876)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3142876, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2209402, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:6be89c06:::smithi03829856-24:head, version: 36'39, data_included: [2211840~51851], data_size: 51851, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:6be89c06:::smithi03829856-24:head@36'39, size: 2386561, copy_subset: [(0, 2386561)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2386561, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2211840, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:6868af91:::smithi03829856-10:7, version: 38'41, data_included: [1830143~321120,2577441~346944], data_size: 668064, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:7@38'41, size: 2924385, copy_subset: [(0, 2924385)], clone_subset: {}, snapset: b=[]:{2: [2], 7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2924385, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1830143, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:6868af91:::smithi03829856-10:b, version: 47'48, data_included: [1830143~321120,2577441~346944], data_size: 668064, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:b@47'48, size: 2924385, copy_subset: [(0, 2924385)], clone_subset: {}, snapset: b=[]:{2: [2], 7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2924385, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1830143, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:6868af91:::smithi03829856-10:head, version: 47'52, data_included: [1830143~348419,2577441~661310], data_size: 1009729, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:head@47'52, size: 3747928, copy_subset: [(0, 3747928)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3747928, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1830143, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:30:30.557 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:26.998857+0000", 2024-03-19T11:30:30.557 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.4820771710000002, 2024-03-19T11:30:30.557 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010149703, 2024-03-19T11:30:30.557 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:30.557 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:30.557 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:30.557 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.557 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:30.557 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.998857+0000", 2024-03-19T11:30:30.557 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:30.557 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.557 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.557 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:30.558 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.998857+0000", 2024-03-19T11:30:30.558 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:30.558 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.558 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.558 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:30.558 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.998858+0000", 2024-03-19T11:30:30.558 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.065e-06 2024-03-19T11:30:30.558 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.558 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.558 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:30.558 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.999629+0000", 2024-03-19T11:30:30.558 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000771132 2024-03-19T11:30:30.558 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.558 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.558 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:30.559 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.999688+0000", 2024-03-19T11:30:30.559 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9147000000000001e-05 2024-03-19T11:30:30.559 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.559 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.559 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:30.559 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.999755+0000", 2024-03-19T11:30:30.559 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6891999999999996e-05 2024-03-19T11:30:30.559 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.559 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.559 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:30.559 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.000047+0000", 2024-03-19T11:30:30.559 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000292176 2024-03-19T11:30:30.559 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.559 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.559 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:30.560 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.000094+0000", 2024-03-19T11:30:30.560 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6485e-05 2024-03-19T11:30:30.560 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:30.560 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:30.560 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:30.560 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.009007+0000", 2024-03-19T11:30:30.560 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0089128060000000005 2024-03-19T11:30:30.560 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:30.560 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:30.560 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:30.560 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:30.560 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:30.560 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:30.562 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-19T11:30:30.562 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:30.571+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:30.662 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:30.673+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:30.733 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:30.733 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:30:30.733 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:30:30.733 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:30.747 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-19T11:30:30.764 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:30.773+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:30.865 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:30.875+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:30.871 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:30.871 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:30:30.871 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:30:30.871 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:30:30.871 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:30.886 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-19T11:30:30.966 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:30.977+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:31.028 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:31.039+0000 7f6712f92640 -1 osd.2 53 handle_osd_map injecting map crc failure 2024-03-19T11:30:31.032 INFO:tasks.rados.rados.0.smithi038.stdout:879: done (2 left) 2024-03-19T11:30:31.032 INFO:tasks.rados.rados.0.smithi038.stdout:880: done (1 left) 2024-03-19T11:30:31.032 INFO:tasks.rados.rados.0.smithi038.stdout:881: done (0 left) 2024-03-19T11:30:31.032 INFO:tasks.rados.rados.0.smithi038.stdout:882: set_chunk oid 37 target oid 2024-03-19T11:30:31.060 INFO:tasks.rados.rados.0.smithi038.stdout:882: set_chunk oid 37 offset: 496640 length: 690176 target oid 29e6b93d552e814dee748e3bcd87d7c4922ba06bce5c60b6e8c172d8a3e958f0 offset: 0 snap 12 2024-03-19T11:30:31.060 INFO:tasks.rados.rados.0.smithi038.stdout:883: set_chunk oid 26 target oid 2024-03-19T11:30:31.060 INFO:tasks.rados.rados.0.smithi038.stdout:883: set_chunk oid 26 offset: 2048 length: 3951104 target oid smithi03829856- offset: 2048 snap 10 2024-03-19T11:30:31.060 INFO:tasks.rados.rados.0.smithi038.stdout:884: read oid 38 snap -1 2024-03-19T11:30:31.060 INFO:tasks.rados.rados.0.smithi038.stdout:884: expect (ObjNum 68 snap 3 seq_num 68) 2024-03-19T11:30:31.061 INFO:tasks.rados.rados.0.smithi038.stdout:885: read oid 4 snap -1 2024-03-19T11:30:31.061 INFO:tasks.rados.rados.0.smithi038.stdout:885: expect (ObjNum 3 snap 0 seq_num 3) 2024-03-19T11:30:31.061 INFO:tasks.rados.rados.0.smithi038.stdout:886: read oid 24 snap 8 2024-03-19T11:30:31.061 INFO:tasks.rados.rados.0.smithi038.stdout:886: expect (ObjNum 23 snap 0 seq_num 23) 2024-03-19T11:30:31.061 INFO:tasks.rados.rados.0.smithi038.stdout:887: snap_remove snap 3 2024-03-19T11:30:31.063 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:31.063 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:30:31.063 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:30:31.063 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:30:31.063 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.063 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:55 3.2 3:525f009b:::a57e3dbafedf1ec1a7132467f167cfc4a4343cb541b36fc481376f0086aece71:head [read 0~121882 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:31.063 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.761904+0000", 2024-03-19T11:30:31.064 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.312071092, 2024-03-19T11:30:31.064 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010829357, 2024-03-19T11:30:31.064 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:31.064 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:31.064 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:31.064 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.064 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:31.064 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761904+0000", 2024-03-19T11:30:31.064 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:31.064 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.064 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.064 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:31.064 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761904+0000", 2024-03-19T11:30:31.064 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:31.064 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.065 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.065 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:31.065 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761905+0000", 2024-03-19T11:30:31.065 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4900000000000005e-07 2024-03-19T11:30:31.065 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.065 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.065 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:31.065 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761917+0000", 2024-03-19T11:30:31.065 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1806e-05 2024-03-19T11:30:31.065 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.065 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.065 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:31.065 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761934+0000", 2024-03-19T11:30:31.065 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6509000000000001e-05 2024-03-19T11:30:31.065 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.066 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.066 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:31.066 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761941+0000", 2024-03-19T11:30:31.066 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8110000000000003e-06 2024-03-19T11:30:31.066 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.066 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.066 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:31.066 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762115+0000", 2024-03-19T11:30:31.066 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017358200000000001 2024-03-19T11:30:31.066 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.066 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.067 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:31.067 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762120+0000", 2024-03-19T11:30:31.067 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3630000000000002e-06 2024-03-19T11:30:31.067 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.067 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.067 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:31.067 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.765567+0000", 2024-03-19T11:30:31.068 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034469990000000001 2024-03-19T11:30:31.068 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.068 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.068 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:31.068 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.765665+0000", 2024-03-19T11:30:31.068 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7476e-05 2024-03-19T11:30:31.068 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.068 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.068 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:31.068 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.772734+0000", 2024-03-19T11:30:31.068 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0070690620000000001 2024-03-19T11:30:31.068 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:31.068 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:31.068 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:31.068 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.068 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.069 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:56 3.2 3:5aa75d1e:::b3c26d797de4ffcce4e862009e6b8eb8576a53cb86ea102ce51145c0038aeb97:head [read 0~524288 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:31.069 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.761958+0000", 2024-03-19T11:30:31.069 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.312017691000001, 2024-03-19T11:30:31.069 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012078836000000001, 2024-03-19T11:30:31.069 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:31.069 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:31.069 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:31.069 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.069 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:31.069 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761958+0000", 2024-03-19T11:30:31.069 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:31.069 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.069 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.069 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:31.070 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761958+0000", 2024-03-19T11:30:31.070 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:31.070 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.071 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.071 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:31.071 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761958+0000", 2024-03-19T11:30:31.071 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9900000000000001e-07 2024-03-19T11:30:31.071 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.071 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.071 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:31.071 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761961+0000", 2024-03-19T11:30:31.071 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9280000000000002e-06 2024-03-19T11:30:31.072 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.072 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.072 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:31.072 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761969+0000", 2024-03-19T11:30:31.072 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1440000000000003e-06 2024-03-19T11:30:31.072 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.072 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.072 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:31.072 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761976+0000", 2024-03-19T11:30:31.072 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1020000000000002e-06 2024-03-19T11:30:31.072 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.072 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.072 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:31.072 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762144+0000", 2024-03-19T11:30:31.072 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016818200000000001 2024-03-19T11:30:31.073 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.073 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.073 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:31.073 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762148+0000", 2024-03-19T11:30:31.073 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0969999999999999e-06 2024-03-19T11:30:31.073 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.073 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.073 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:31.073 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.772788+0000", 2024-03-19T11:30:31.073 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010640124000000001 2024-03-19T11:30:31.073 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.073 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.073 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:31.073 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.772898+0000", 2024-03-19T11:30:31.073 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000110512 2024-03-19T11:30:31.074 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.074 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.074 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:31.074 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.774037+0000", 2024-03-19T11:30:31.074 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011382479999999999 2024-03-19T11:30:31.074 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:31.074 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:31.074 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:31.074 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.074 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.074 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:57 3.2 3:6d82e5d5:::455c613336f373ae48b33af990f91f317265a0c5a920cc1b46533fdb41d8319c:head [read 0~380357 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:31.074 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.761987+0000", 2024-03-19T11:30:31.074 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.311988728999999, 2024-03-19T11:30:31.074 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012691991999999999, 2024-03-19T11:30:31.074 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:31.075 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:31.075 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:31.075 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.075 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:31.075 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761987+0000", 2024-03-19T11:30:31.075 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:31.075 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.075 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.075 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:31.075 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761987+0000", 2024-03-19T11:30:31.075 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:31.075 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.075 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.075 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:31.075 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761987+0000", 2024-03-19T11:30:31.076 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4099999999999999e-07 2024-03-19T11:30:31.076 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.076 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.076 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:31.076 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761990+0000", 2024-03-19T11:30:31.076 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3829999999999999e-06 2024-03-19T11:30:31.076 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.076 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.076 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:31.077 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762000+0000", 2024-03-19T11:30:31.077 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0229e-05 2024-03-19T11:30:31.077 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.077 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.077 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:31.077 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762006+0000", 2024-03-19T11:30:31.077 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7069999999999997e-06 2024-03-19T11:30:31.077 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.077 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.077 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:31.077 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762171+0000", 2024-03-19T11:30:31.077 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000165341 2024-03-19T11:30:31.077 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.077 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.077 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:31.078 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762176+0000", 2024-03-19T11:30:31.078 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6879999999999998e-06 2024-03-19T11:30:31.078 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.078 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.078 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:31.078 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.774081+0000", 2024-03-19T11:30:31.078 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011905094999999999 2024-03-19T11:30:31.078 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.078 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.078 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:31.078 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.774169+0000", 2024-03-19T11:30:31.078 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8089000000000003e-05 2024-03-19T11:30:31.078 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.078 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.079 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:31.079 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.774679+0000", 2024-03-19T11:30:31.079 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00051001900000000001 2024-03-19T11:30:31.079 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:31.079 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:31.079 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:31.079 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.079 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.079 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:58 3.2 3:5f1d333e:::bfcdf551f6f2543327e9c601a4377e9a5ef396f5b97ae8331892d1bcf6b55a9d:head [read 0~161292 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:31.079 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.762024+0000", 2024-03-19T11:30:31.079 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.311951780000001, 2024-03-19T11:30:31.079 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013495629, 2024-03-19T11:30:31.079 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:31.080 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:31.080 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:31.080 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.080 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:31.080 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762024+0000", 2024-03-19T11:30:31.080 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:31.080 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.081 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.081 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:31.081 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762024+0000", 2024-03-19T11:30:31.081 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:31.081 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.082 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.082 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:31.082 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762024+0000", 2024-03-19T11:30:31.082 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9999999999999997e-07 2024-03-19T11:30:31.082 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.082 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.082 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:31.082 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762027+0000", 2024-03-19T11:30:31.083 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0410000000000002e-06 2024-03-19T11:30:31.083 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.083 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.083 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:31.083 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762037+0000", 2024-03-19T11:30:31.083 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1020000000000003e-06 2024-03-19T11:30:31.083 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.083 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.083 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:31.083 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762043+0000", 2024-03-19T11:30:31.083 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4409999999999998e-06 2024-03-19T11:30:31.083 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.083 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.084 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:31.084 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762198+0000", 2024-03-19T11:30:31.084 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015540000000000001 2024-03-19T11:30:31.084 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.084 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.084 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:31.084 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762203+0000", 2024-03-19T11:30:31.084 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5109999999999997e-06 2024-03-19T11:30:31.084 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.084 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.084 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:31.084 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.774710+0000", 2024-03-19T11:30:31.085 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012507452 2024-03-19T11:30:31.085 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.085 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.085 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:31.085 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.774754+0000", 2024-03-19T11:30:31.085 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3869999999999998e-05 2024-03-19T11:30:31.085 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.085 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.085 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:31.085 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.775519+0000", 2024-03-19T11:30:31.085 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00076521200000000003 2024-03-19T11:30:31.086 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:31.086 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:31.086 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:31.086 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.086 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.086 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:59 3.2 3:78b4b14e:::e1ef0c0c594c36c534f3829406e07ffb60d57b70360f62977e0d545d8ca2bf18:head [read 0~149977 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:31.086 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.762054+0000", 2024-03-19T11:30:31.086 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.311921348999999, 2024-03-19T11:30:31.086 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015354483, 2024-03-19T11:30:31.086 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:31.086 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:31.087 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:31.087 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.087 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:31.087 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762054+0000", 2024-03-19T11:30:31.087 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:31.087 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.087 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.087 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:31.087 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762054+0000", 2024-03-19T11:30:31.087 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:31.088 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.088 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.088 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:31.088 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762055+0000", 2024-03-19T11:30:31.088 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.32e-07 2024-03-19T11:30:31.088 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.088 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.088 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:31.088 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762058+0000", 2024-03-19T11:30:31.089 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9189999999999999e-06 2024-03-19T11:30:31.089 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.089 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.089 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:31.089 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762065+0000", 2024-03-19T11:30:31.089 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2409999999999998e-06 2024-03-19T11:30:31.089 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.089 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.089 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:31.089 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762070+0000", 2024-03-19T11:30:31.089 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6910000000000002e-06 2024-03-19T11:30:31.089 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.090 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.090 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:31.090 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762227+0000", 2024-03-19T11:30:31.090 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015606999999999999 2024-03-19T11:30:31.090 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.090 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.090 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:31.091 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762232+0000", 2024-03-19T11:30:31.091 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3029999999999999e-06 2024-03-19T11:30:31.091 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.091 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.091 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:31.091 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.776984+0000", 2024-03-19T11:30:31.091 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014752487999999999 2024-03-19T11:30:31.091 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.091 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.091 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:31.092 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.777030+0000", 2024-03-19T11:30:31.092 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5359999999999999e-05 2024-03-19T11:30:31.092 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.092 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.092 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:31.092 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.777409+0000", 2024-03-19T11:30:31.092 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037897899999999999 2024-03-19T11:30:31.092 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:31.092 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:31.092 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:31.092 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.092 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.093 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1067 4.0 4:01018511:::smithi03829856-19:head [write 477332~766101] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:31.093 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.950506+0000", 2024-03-19T11:30:31.093 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 17.123469636999999, 2024-03-19T11:30:31.093 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016279426, 2024-03-19T11:30:31.093 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:31.093 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:31.093 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:31.093 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:31.093 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:31.093 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1067 2024-03-19T11:30:31.093 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.093 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:31.093 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.093 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:31.093 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950506+0000", 2024-03-19T11:30:31.093 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:31.094 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.094 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.094 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:31.094 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950506+0000", 2024-03-19T11:30:31.094 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:31.094 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.094 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.094 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:31.094 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950509+0000", 2024-03-19T11:30:31.095 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.021e-06 2024-03-19T11:30:31.095 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.095 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.095 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:31.095 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950892+0000", 2024-03-19T11:30:31.096 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00038263300000000001 2024-03-19T11:30:31.096 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.096 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.096 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:31.096 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950912+0000", 2024-03-19T11:30:31.096 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0539000000000001e-05 2024-03-19T11:30:31.096 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.096 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.096 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:31.096 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950922+0000", 2024-03-19T11:30:31.096 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.577e-06 2024-03-19T11:30:31.096 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.096 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.097 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:31.097 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950982+0000", 2024-03-19T11:30:31.097 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0760000000000001e-05 2024-03-19T11:30:31.097 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.097 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.097 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:31.097 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.951058+0000", 2024-03-19T11:30:31.097 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5078000000000002e-05 2024-03-19T11:30:31.097 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.097 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.097 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:31.097 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.951173+0000", 2024-03-19T11:30:31.097 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011516200000000001 2024-03-19T11:30:31.097 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.097 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.098 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:31.098 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.955064+0000", 2024-03-19T11:30:31.098 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003890896 2024-03-19T11:30:31.098 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.098 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.098 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:31.098 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966670+0000", 2024-03-19T11:30:31.098 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011606795 2024-03-19T11:30:31.098 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.098 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.098 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:31.098 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966731+0000", 2024-03-19T11:30:31.098 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0528000000000001e-05 2024-03-19T11:30:31.098 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.098 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.099 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:31.099 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966785+0000", 2024-03-19T11:30:31.099 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4437000000000001e-05 2024-03-19T11:30:31.099 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:31.099 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:31.099 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:31.099 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.099 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.099 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1068 4.0 4:01018511:::smithi03829856-19:head [write 1681102~474010] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:31.099 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.956456+0000", 2024-03-19T11:30:31.099 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 17.117519072, 2024-03-19T11:30:31.099 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010432765, 2024-03-19T11:30:31.099 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:31.099 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:31.099 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:31.100 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:31.100 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:31.100 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1068 2024-03-19T11:30:31.100 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.100 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:31.100 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.100 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:31.100 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956456+0000", 2024-03-19T11:30:31.100 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:31.100 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.100 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.100 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:31.100 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956456+0000", 2024-03-19T11:30:31.100 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:31.100 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.101 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.101 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:31.101 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956458+0000", 2024-03-19T11:30:31.101 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.742e-06 2024-03-19T11:30:31.101 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.101 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.101 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:31.101 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956655+0000", 2024-03-19T11:30:31.101 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019682200000000001 2024-03-19T11:30:31.101 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.101 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.101 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:31.101 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956669+0000", 2024-03-19T11:30:31.101 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3978000000000001e-05 2024-03-19T11:30:31.101 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.101 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.102 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:31.102 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956678+0000", 2024-03-19T11:30:31.102 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5029999999999997e-06 2024-03-19T11:30:31.102 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.102 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.102 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:31.102 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956729+0000", 2024-03-19T11:30:31.102 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1765999999999999e-05 2024-03-19T11:30:31.102 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.102 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.102 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:31.102 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956795+0000", 2024-03-19T11:30:31.102 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5701e-05 2024-03-19T11:30:31.103 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.103 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.103 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:31.103 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956886+0000", 2024-03-19T11:30:31.103 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1226999999999999e-05 2024-03-19T11:30:31.103 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.103 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.103 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:31.103 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.958609+0000", 2024-03-19T11:30:31.103 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017227049999999999 2024-03-19T11:30:31.103 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.103 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.103 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:31.104 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966837+0000", 2024-03-19T11:30:31.104 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0082276439999999992 2024-03-19T11:30:31.104 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.104 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.104 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:31.104 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966866+0000", 2024-03-19T11:30:31.104 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9323999999999999e-05 2024-03-19T11:30:31.104 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.104 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.104 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:31.104 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966889+0000", 2024-03-19T11:30:31.104 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3353000000000002e-05 2024-03-19T11:30:31.104 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:31.104 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:31.104 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:31.105 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.105 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.105 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.0.13:50 3.2 3:5595a68d:::07854d2fef297a06ba81685e660c332de36d5d18d546927d30daad6d7fda1541:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:31.105 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.166341+0000", 2024-03-19T11:30:31.105 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 12.907634206000001, 2024-03-19T11:30:31.105 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010067153000000001, 2024-03-19T11:30:31.105 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:31.105 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:31.105 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:31.105 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.105 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:31.105 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166341+0000", 2024-03-19T11:30:31.105 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:31.105 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.105 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.105 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:31.106 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166341+0000", 2024-03-19T11:30:31.106 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:31.106 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.106 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.106 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:31.106 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166344+0000", 2024-03-19T11:30:31.106 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1370000000000002e-06 2024-03-19T11:30:31.106 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.106 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.106 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:31.106 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166828+0000", 2024-03-19T11:30:31.106 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00048343600000000001 2024-03-19T11:30:31.106 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.106 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.107 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:31.107 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166857+0000", 2024-03-19T11:30:31.107 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8880000000000001e-05 2024-03-19T11:30:31.107 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.107 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.107 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:31.107 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166871+0000", 2024-03-19T11:30:31.107 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4384e-05 2024-03-19T11:30:31.107 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.107 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.107 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:31.107 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167222+0000", 2024-03-19T11:30:31.107 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00035059000000000002 2024-03-19T11:30:31.107 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.107 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.108 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:31.108 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167334+0000", 2024-03-19T11:30:31.108 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000112663 2024-03-19T11:30:31.108 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.108 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.108 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:31.108 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167533+0000", 2024-03-19T11:30:31.108 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019901399999999999 2024-03-19T11:30:31.108 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.108 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.108 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:31.108 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176237+0000", 2024-03-19T11:30:31.108 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0087039890000000005 2024-03-19T11:30:31.108 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.108 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.109 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:31.109 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176260+0000", 2024-03-19T11:30:31.109 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2889000000000001e-05 2024-03-19T11:30:31.109 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.109 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.109 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:31.109 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176343+0000", 2024-03-19T11:30:31.109 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2651000000000005e-05 2024-03-19T11:30:31.109 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.109 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.109 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:31.109 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176408+0000", 2024-03-19T11:30:31.109 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5519999999999996e-05 2024-03-19T11:30:31.109 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:31.109 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:31.110 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:31.110 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.110 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.110 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.0.13:51 3.2 3:587bed34:::13074eb9c7a477811d595d489af192576515e7c716c0b501b140fcc4687b768b:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:31.110 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.166912+0000", 2024-03-19T11:30:31.110 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 12.907063403, 2024-03-19T11:30:31.110 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0097465639999999992, 2024-03-19T11:30:31.110 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:31.110 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:31.111 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:31.111 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.111 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:31.111 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166912+0000", 2024-03-19T11:30:31.111 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:31.111 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.111 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.111 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:31.111 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166912+0000", 2024-03-19T11:30:31.111 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:31.111 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.111 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.112 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:31.112 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166914+0000", 2024-03-19T11:30:31.112 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.468e-06 2024-03-19T11:30:31.112 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.112 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.112 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:31.112 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167000+0000", 2024-03-19T11:30:31.112 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6311999999999996e-05 2024-03-19T11:30:31.112 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.112 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.112 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:31.112 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167011+0000", 2024-03-19T11:30:31.112 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.153e-05 2024-03-19T11:30:31.112 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.112 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.113 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:31.113 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167019+0000", 2024-03-19T11:30:31.113 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7970000000000001e-06 2024-03-19T11:30:31.113 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.113 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.113 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:31.113 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167864+0000", 2024-03-19T11:30:31.113 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00084482600000000002 2024-03-19T11:30:31.113 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.113 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.113 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:31.113 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.169110+0000", 2024-03-19T11:30:31.113 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0012456399999999999 2024-03-19T11:30:31.113 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.114 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.114 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:31.114 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.169467+0000", 2024-03-19T11:30:31.114 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00035692100000000001 2024-03-19T11:30:31.114 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.114 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.114 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:31.114 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176426+0000", 2024-03-19T11:30:31.114 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0069589600000000001 2024-03-19T11:30:31.114 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.114 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.114 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:31.114 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176524+0000", 2024-03-19T11:30:31.114 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8183000000000005e-05 2024-03-19T11:30:31.114 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.115 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.115 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:31.115 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176603+0000", 2024-03-19T11:30:31.115 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9035999999999993e-05 2024-03-19T11:30:31.115 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.115 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.115 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:31.115 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176659+0000", 2024-03-19T11:30:31.115 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5890999999999997e-05 2024-03-19T11:30:31.115 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:31.115 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:31.115 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:31.115 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.115 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.115 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.0.13:52 3.2 3:46a26328:::b2b262672c1f15ae09b098a7a21c74662e4816397daed8ca08fb83d0cd3a0cac:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:31.116 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.167033+0000", 2024-03-19T11:30:31.116 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 12.906942594, 2024-03-19T11:30:31.116 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010113297, 2024-03-19T11:30:31.116 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:31.116 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:31.116 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:31.116 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.116 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:31.116 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167033+0000", 2024-03-19T11:30:31.116 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:31.116 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.116 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.116 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:31.116 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167033+0000", 2024-03-19T11:30:31.116 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:31.117 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.117 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.117 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:31.117 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167034+0000", 2024-03-19T11:30:31.117 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6599999999999996e-07 2024-03-19T11:30:31.117 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.117 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.117 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:31.117 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167092+0000", 2024-03-19T11:30:31.117 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7989999999999999e-05 2024-03-19T11:30:31.117 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.117 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.117 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:31.117 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167100+0000", 2024-03-19T11:30:31.117 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9670000000000001e-06 2024-03-19T11:30:31.118 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.118 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.118 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:31.118 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167108+0000", 2024-03-19T11:30:31.118 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3559999999999993e-06 2024-03-19T11:30:31.118 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.118 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.119 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:31.119 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.170462+0000", 2024-03-19T11:30:31.119 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033540229999999998 2024-03-19T11:30:31.119 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.119 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.119 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:31.119 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.171193+0000", 2024-03-19T11:30:31.119 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00073086899999999998 2024-03-19T11:30:31.119 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.119 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.119 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:31.119 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.171558+0000", 2024-03-19T11:30:31.119 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00036518899999999998 2024-03-19T11:30:31.119 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.119 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.120 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:31.120 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176440+0000", 2024-03-19T11:30:31.120 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0048817260000000003 2024-03-19T11:30:31.120 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.120 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.120 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:31.120 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177053+0000", 2024-03-19T11:30:31.120 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00061337100000000003 2024-03-19T11:30:31.120 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.120 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.121 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:31.121 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177096+0000", 2024-03-19T11:30:31.121 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2846999999999998e-05 2024-03-19T11:30:31.121 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.121 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.121 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:31.121 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177146+0000", 2024-03-19T11:30:31.121 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0293000000000001e-05 2024-03-19T11:30:31.121 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:31.121 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:31.121 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:31.121 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.121 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.122 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.0.13:54 3.2 3:7b614c59:::a7a06fca755baa7858207cb4b067fffe8c25e8183433b950d6879f6969d41b0b:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:31.122 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.167117+0000", 2024-03-19T11:30:31.122 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 12.906858086, 2024-03-19T11:30:31.122 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010796765999999999, 2024-03-19T11:30:31.122 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:31.122 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:31.122 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:31.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.122 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:31.122 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167117+0000", 2024-03-19T11:30:31.122 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:31.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.123 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:31.123 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167117+0000", 2024-03-19T11:30:31.123 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:31.123 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.123 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.123 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:31.123 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167118+0000", 2024-03-19T11:30:31.124 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2600000000000002e-07 2024-03-19T11:30:31.124 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.124 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.124 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:31.124 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167150+0000", 2024-03-19T11:30:31.124 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1576999999999999e-05 2024-03-19T11:30:31.124 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.124 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.124 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:31.124 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167157+0000", 2024-03-19T11:30:31.124 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2779999999999996e-06 2024-03-19T11:30:31.124 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.124 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.125 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:31.125 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167163+0000", 2024-03-19T11:30:31.125 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0120000000000002e-06 2024-03-19T11:30:31.125 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.125 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.125 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:31.125 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173324+0000", 2024-03-19T11:30:31.125 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0061614149999999999 2024-03-19T11:30:31.125 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.126 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.126 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:31.126 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173489+0000", 2024-03-19T11:30:31.126 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016438999999999999 2024-03-19T11:30:31.126 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.126 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.126 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:31.126 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173774+0000", 2024-03-19T11:30:31.126 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00028548000000000001 2024-03-19T11:30:31.127 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.127 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.127 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:31.127 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177668+0000", 2024-03-19T11:30:31.127 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038942450000000002 2024-03-19T11:30:31.127 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.127 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.127 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:31.127 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177792+0000", 2024-03-19T11:30:31.127 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012398199999999999 2024-03-19T11:30:31.127 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.127 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.127 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:31.128 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177861+0000", 2024-03-19T11:30:31.128 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8633999999999994e-05 2024-03-19T11:30:31.128 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.128 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.128 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:31.128 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177914+0000", 2024-03-19T11:30:31.128 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3226999999999997e-05 2024-03-19T11:30:31.128 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:31.129 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:31.129 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:31.129 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.129 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.129 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1230 4.0 4:0dc3b91e:::smithi03829856-28:head [write 662209~626546] snapc 9=[9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:31.129 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.170758+0000", 2024-03-19T11:30:31.129 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 12.903217372, 2024-03-19T11:30:31.129 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011180066000000001, 2024-03-19T11:30:31.130 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:31.130 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:31.130 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:31.130 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:31.130 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:31.130 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1230 2024-03-19T11:30:31.130 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.130 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:31.130 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.131 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:31.131 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.170758+0000", 2024-03-19T11:30:31.131 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:31.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.131 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.131 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:31.131 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.170758+0000", 2024-03-19T11:30:31.131 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:31.132 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.132 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.132 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:31.132 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.170762+0000", 2024-03-19T11:30:31.132 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4960000000000001e-06 2024-03-19T11:30:31.132 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.132 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.132 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:31.132 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.171449+0000", 2024-03-19T11:30:31.132 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00068700499999999997 2024-03-19T11:30:31.132 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.132 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.132 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:31.133 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.171467+0000", 2024-03-19T11:30:31.133 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8389e-05 2024-03-19T11:30:31.133 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.133 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.133 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:31.133 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.171482+0000", 2024-03-19T11:30:31.133 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4619e-05 2024-03-19T11:30:31.133 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.133 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.133 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:31.133 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.171566+0000", 2024-03-19T11:30:31.133 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4788000000000002e-05 2024-03-19T11:30:31.133 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.134 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.134 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:31.134 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.171685+0000", 2024-03-19T11:30:31.134 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011861700000000001 2024-03-19T11:30:31.134 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.134 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.134 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:31.134 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.171835+0000", 2024-03-19T11:30:31.134 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014956299999999999 2024-03-19T11:30:31.134 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.138 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.138 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:31.138 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.181520+0000", 2024-03-19T11:30:31.139 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0096855750000000001 2024-03-19T11:30:31.139 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.139 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.139 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:31.139 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.181640+0000", 2024-03-19T11:30:31.139 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011966799999999999 2024-03-19T11:30:31.139 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.139 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.139 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:31.139 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.181811+0000", 2024-03-19T11:30:31.139 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017062699999999999 2024-03-19T11:30:31.139 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.139 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.139 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:31.139 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.181938+0000", 2024-03-19T11:30:31.140 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012771899999999999 2024-03-19T11:30:31.140 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:31.140 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:31.140 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:31.140 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.140 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.140 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1231 4.0 4:0dc3b91e:::smithi03829856-28:head [write 1808051~636013] snapc 9=[9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:31.140 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.173016+0000", 2024-03-19T11:30:31.140 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 12.900959523999999, 2024-03-19T11:30:31.140 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013973719000000001, 2024-03-19T11:30:31.140 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:31.140 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:31.140 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:31.140 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:31.140 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:31.141 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1231 2024-03-19T11:30:31.141 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.141 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:31.141 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.141 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:31.141 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173016+0000", 2024-03-19T11:30:31.141 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:31.141 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.141 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.141 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:31.141 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173016+0000", 2024-03-19T11:30:31.141 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:31.141 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.141 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.141 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:31.142 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173019+0000", 2024-03-19T11:30:31.142 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.253e-06 2024-03-19T11:30:31.142 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.142 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.142 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:31.142 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176143+0000", 2024-03-19T11:30:31.142 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031240930000000001 2024-03-19T11:30:31.142 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.142 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.142 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:31.142 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176177+0000", 2024-03-19T11:30:31.142 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3237e-05 2024-03-19T11:30:31.142 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.142 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.143 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:31.143 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176196+0000", 2024-03-19T11:30:31.143 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9358e-05 2024-03-19T11:30:31.143 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.143 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.143 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:31.143 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176493+0000", 2024-03-19T11:30:31.143 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00029743899999999998 2024-03-19T11:30:31.143 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.143 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.143 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:31.143 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176786+0000", 2024-03-19T11:30:31.143 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00029259000000000002 2024-03-19T11:30:31.143 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.143 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.144 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:31.144 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177250+0000", 2024-03-19T11:30:31.144 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00046376000000000002 2024-03-19T11:30:31.144 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.144 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.144 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:31.144 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.182511+0000", 2024-03-19T11:30:31.144 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0052609019999999996 2024-03-19T11:30:31.144 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.144 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.144 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:31.144 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.186896+0000", 2024-03-19T11:30:31.144 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0043857009999999997 2024-03-19T11:30:31.144 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.144 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.145 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:31.145 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.186948+0000", 2024-03-19T11:30:31.145 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1653000000000002e-05 2024-03-19T11:30:31.145 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.145 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.145 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:31.145 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.186990+0000", 2024-03-19T11:30:31.145 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1733000000000002e-05 2024-03-19T11:30:31.145 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:31.145 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:31.145 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:31.145 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.145 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.145 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1277 4.0 4:01018511:::smithi03829856-19:head [write 763051~794393,stat] snapc b=[b,a,9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-03-19T11:30:31.145 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:21.953732+0000", 2024-03-19T11:30:31.146 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 9.1202432869999992, 2024-03-19T11:30:31.146 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011784380000000001, 2024-03-19T11:30:31.146 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:31.146 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:31.146 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:31.146 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:31.146 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:31.146 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1277 2024-03-19T11:30:31.146 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.146 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:31.146 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.146 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:31.146 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.953732+0000", 2024-03-19T11:30:31.146 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:31.146 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.147 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.147 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:31.147 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.953732+0000", 2024-03-19T11:30:31.147 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:31.147 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.147 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.147 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:31.147 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.953735+0000", 2024-03-19T11:30:31.147 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1999999999999999e-06 2024-03-19T11:30:31.147 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.147 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.147 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:31.147 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.954556+0000", 2024-03-19T11:30:31.147 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00082013599999999996 2024-03-19T11:30:31.147 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.148 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.148 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:31.148 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.954596+0000", 2024-03-19T11:30:31.148 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0741000000000002e-05 2024-03-19T11:30:31.148 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.148 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.148 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:31.148 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.954617+0000", 2024-03-19T11:30:31.148 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0375999999999999e-05 2024-03-19T11:30:31.148 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.148 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.148 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:31.148 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.954747+0000", 2024-03-19T11:30:31.148 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000130648 2024-03-19T11:30:31.148 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.149 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.149 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:31.149 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.954954+0000", 2024-03-19T11:30:31.149 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020694299999999999 2024-03-19T11:30:31.149 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.149 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.149 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:31.149 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.955252+0000", 2024-03-19T11:30:31.149 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00029740700000000002 2024-03-19T11:30:31.149 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.149 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.149 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:31.149 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.961677+0000", 2024-03-19T11:30:31.149 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0064253469999999997 2024-03-19T11:30:31.149 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.150 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.150 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:31.150 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.965373+0000", 2024-03-19T11:30:31.150 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0036962649999999998 2024-03-19T11:30:31.150 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.150 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.150 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:31.150 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.965451+0000", 2024-03-19T11:30:31.150 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7204000000000001e-05 2024-03-19T11:30:31.150 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.150 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.150 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:31.150 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.965517+0000", 2024-03-19T11:30:31.150 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6112999999999995e-05 2024-03-19T11:30:31.151 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:31.151 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:31.151 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:31.151 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.151 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.151 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1408 4.10 4:0dc3b91e:::smithi03829856-28:head [read 0~3462209,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:31.151 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.918220+0000", 2024-03-19T11:30:31.151 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.1557550689999996, 2024-03-19T11:30:31.151 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013199458000000001, 2024-03-19T11:30:31.151 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:31.151 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:31.151 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:31.151 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:31.151 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:31.152 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1408 2024-03-19T11:30:31.152 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.152 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:31.152 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.152 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:31.152 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918220+0000", 2024-03-19T11:30:31.152 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:31.152 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.152 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.152 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:31.152 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918220+0000", 2024-03-19T11:30:31.152 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:31.152 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.152 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.152 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:31.153 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918221+0000", 2024-03-19T11:30:31.153 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0999999999999999e-07 2024-03-19T11:30:31.153 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.153 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.153 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:31.153 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918225+0000", 2024-03-19T11:30:31.153 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2710000000000003e-06 2024-03-19T11:30:31.153 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.153 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.153 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:31.153 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918233+0000", 2024-03-19T11:30:31.153 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9489999999999996e-06 2024-03-19T11:30:31.153 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.153 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.154 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:31.154 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918239+0000", 2024-03-19T11:30:31.154 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5820000000000001e-06 2024-03-19T11:30:31.154 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.154 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.154 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:31.154 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.923634+0000", 2024-03-19T11:30:31.154 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005395703 2024-03-19T11:30:31.154 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.154 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.154 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:31.154 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.923786+0000", 2024-03-19T11:30:31.154 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015117900000000001 2024-03-19T11:30:31.154 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.155 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.155 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:31.155 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.931420+0000", 2024-03-19T11:30:31.155 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.007634264 2024-03-19T11:30:31.155 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:31.155 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:31.155 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:31.155 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.155 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.155 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1409 4.10 4:0dc3b91e:::smithi03829856-28:head [sparse-read 0~1851556] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:31.155 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.918248+0000", 2024-03-19T11:30:31.155 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.1557272349999996, 2024-03-19T11:30:31.155 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013726986, 2024-03-19T11:30:31.156 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:31.156 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:31.156 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:31.156 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:31.156 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:31.156 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1409 2024-03-19T11:30:31.156 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.156 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:31.156 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.156 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:31.156 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918248+0000", 2024-03-19T11:30:31.156 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:31.156 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.157 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.157 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:31.157 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918248+0000", 2024-03-19T11:30:31.157 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:31.157 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.157 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.157 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:31.157 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918249+0000", 2024-03-19T11:30:31.157 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4700000000000002e-07 2024-03-19T11:30:31.157 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.157 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.158 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:31.158 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918251+0000", 2024-03-19T11:30:31.158 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.542e-06 2024-03-19T11:30:31.158 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.158 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.158 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:31.158 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918258+0000", 2024-03-19T11:30:31.158 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4629999999999998e-06 2024-03-19T11:30:31.158 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.158 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.158 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:31.158 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918263+0000", 2024-03-19T11:30:31.159 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8049999999999997e-06 2024-03-19T11:30:31.159 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.159 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.159 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:31.159 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.931469+0000", 2024-03-19T11:30:31.159 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013206458000000001 2024-03-19T11:30:31.159 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.159 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.159 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:31.159 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.931586+0000", 2024-03-19T11:30:31.159 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000116655 2024-03-19T11:30:31.159 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.159 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.160 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:31.160 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.931975+0000", 2024-03-19T11:30:31.160 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00038961600000000002 2024-03-19T11:30:31.160 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:31.160 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:31.160 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:31.160 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.160 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.161 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(3.1 49/48 [PushOp(3:9205291d:::smithi03829856-50 low_tier:head, version: 28'36, data_included: [393216~851968,1703936~196608], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:9205291d:::smithi03829856-50 low_tier:head@28'36, size: 2823543, copy_subset: [(0, 2823543)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1900544, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:baab691a:::smithi03829856-2 low_tier:head, version: 28'37, data_included: [393216~524288,1507328~509479], data_size: 1033767, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:baab691a:::smithi03829856-2 low_tier:head@28'37, size: 2016807, copy_subset: [(0, 2016807)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2016807, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:98ce8426:::smithi03829856-4 low_tier:head, version: 28'38, data_included: [393216~720896,1638400~327680], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:98ce8426:::smithi03829856-4 low_tier:head@28'38, size: 2050421, copy_subset: [(0, 2050421)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1966080, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:a86be07b:::smithi03829856-44 low_tier:head, version: 28'41, data_included: [720896~655360,1769472~393216], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:a86be07b:::smithi03829856-44 low_tier:head@28'41, size: 2722701, copy_subset: [(0, 2722701)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2162688, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:bb56919e:::smithi03829856-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'44, data_included: [393216~786432,1703936~262144], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:bb56919e:::smithi03829856-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'44, size: 2806736, copy_subset: [(0, 2806736)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1966080, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:88687f71:::smithi03829856-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'45, data_included: [589824~655360,1638400~393216], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:88687f71:::smithi03829856-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'45, size: 2201684, copy_subset: [(0, 2201684)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2031616, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:b1f822cf:::smithi03829856-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'46, data_included: [720896~786432,1966080~262144], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:b1f822cf:::smithi03829856-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'46, size: 2369754, copy_subset: [(0, 2369754)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2228224, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:a5e8313a:::smithi03829856-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'47, data_included: [524288~720896,1769472~327680], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:a5e8313a:::smithi03829856-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'47, size: 2571438, copy_subset: [(0, 2571438)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:bca5ec64:::e3a26bf9230b1b95b8cf1a4cb01909236bf2bf3fd59b083b2291d278ab98264d:head, version: 28'48, data_included: [0~393216], data_size: 393216, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bca5ec64:::e3a26bf9230b1b95b8cf1a4cb01909236bf2bf3fd59b083b2291d278ab98264d:head@28'48, size: 533504, copy_subset: [(0, 533504)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 533504, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:30:31.161 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:26.948137+0000", 2024-03-19T11:30:31.161 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.1258384220000002, 2024-03-19T11:30:31.161 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.036085578, 2024-03-19T11:30:31.161 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:31.161 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:31.161 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:31.161 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.161 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:31.161 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.948137+0000", 2024-03-19T11:30:31.161 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:31.161 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.162 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.162 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:31.162 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.948137+0000", 2024-03-19T11:30:31.162 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:31.162 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.162 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.163 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:31.163 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.948139+0000", 2024-03-19T11:30:31.163 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1399999999999998e-06 2024-03-19T11:30:31.163 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.163 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.163 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:31.163 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954288+0000", 2024-03-19T11:30:31.163 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0061487429999999999 2024-03-19T11:30:31.163 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.163 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.163 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:31.163 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954388+0000", 2024-03-19T11:30:31.163 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010022 2024-03-19T11:30:31.163 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.164 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.164 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:31.164 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954523+0000", 2024-03-19T11:30:31.164 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000134332 2024-03-19T11:30:31.164 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.164 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.164 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:31.164 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.955120+0000", 2024-03-19T11:30:31.164 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00059695799999999997 2024-03-19T11:30:31.164 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.164 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.165 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:31.165 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.955194+0000", 2024-03-19T11:30:31.165 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4077000000000002e-05 2024-03-19T11:30:31.165 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.165 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.165 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:31.165 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.984223+0000", 2024-03-19T11:30:31.165 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029029108000000001 2024-03-19T11:30:31.165 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:31.166 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:31.166 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:31.166 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.166 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.166 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(3.1 49/48 [PushOp(3:99a49d1b:::895a63407b6109b3cca9a6f5ef827134d56679bef42daadc5892269682c428ed:head, version: 32'49, data_included: [393216~388096], data_size: 388096, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:99a49d1b:::895a63407b6109b3cca9a6f5ef827134d56679bef42daadc5892269682c428ed:head@32'49, size: 781312, copy_subset: [(0, 781312)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 781312, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:30:31.166 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:26.954637+0000", 2024-03-19T11:30:31.166 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.1193388049999999, 2024-03-19T11:30:31.166 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.030729057000000001, 2024-03-19T11:30:31.166 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:31.166 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:31.166 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:31.167 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.167 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:31.167 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954637+0000", 2024-03-19T11:30:31.167 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:31.167 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.167 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.167 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:31.167 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954637+0000", 2024-03-19T11:30:31.167 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:31.167 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.167 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.167 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:31.167 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954637+0000", 2024-03-19T11:30:31.168 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7299999999999995e-07 2024-03-19T11:30:31.168 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.168 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.168 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:31.168 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954873+0000", 2024-03-19T11:30:31.168 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023520299999999999 2024-03-19T11:30:31.168 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.168 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.169 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:31.169 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954897+0000", 2024-03-19T11:30:31.169 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4110000000000001e-05 2024-03-19T11:30:31.169 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.169 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.169 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:31.169 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954917+0000", 2024-03-19T11:30:31.169 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.067e-05 2024-03-19T11:30:31.169 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.169 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.169 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:31.170 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.984289+0000", 2024-03-19T11:30:31.170 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029371191000000001 2024-03-19T11:30:31.170 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.170 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.170 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:31.170 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.984303+0000", 2024-03-19T11:30:31.170 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4888999999999999e-05 2024-03-19T11:30:31.170 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.170 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.171 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:31.171 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.985366+0000", 2024-03-19T11:30:31.171 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001062321 2024-03-19T11:30:31.171 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:31.171 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:31.171 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:31.171 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.171 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.171 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1472 4.10 4:0dc3b91e:::smithi03829856-28:head [read 534528~391680] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-03-19T11:30:31.171 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:27.091748+0000", 2024-03-19T11:30:31.171 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.9822279169999999, 2024-03-19T11:30:31.172 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.83574118500000005, 2024-03-19T11:30:31.172 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:31.172 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:31.172 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:31.172 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:31.172 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:31.172 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1472 2024-03-19T11:30:31.172 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.172 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:31.172 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.172 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:31.172 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091748+0000", 2024-03-19T11:30:31.172 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:31.172 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.173 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.173 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:31.173 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091748+0000", 2024-03-19T11:30:31.173 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:31.173 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.173 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.173 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:31.173 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091750+0000", 2024-03-19T11:30:31.173 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3130000000000001e-06 2024-03-19T11:30:31.173 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.173 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.173 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:31.173 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091760+0000", 2024-03-19T11:30:31.173 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0560999999999999e-05 2024-03-19T11:30:31.173 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.174 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.174 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:31.174 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091784+0000", 2024-03-19T11:30:31.174 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3915000000000001e-05 2024-03-19T11:30:31.174 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.174 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.174 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:31.174 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091797+0000", 2024-03-19T11:30:31.174 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2660000000000001e-05 2024-03-19T11:30:31.174 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.174 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.174 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:31.174 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091874+0000", 2024-03-19T11:30:31.174 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6757999999999999e-05 2024-03-19T11:30:31.174 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.175 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.175 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:30:31.175 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091882+0000", 2024-03-19T11:30:31.175 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5520000000000005e-06 2024-03-19T11:30:31.175 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.175 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.175 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:31.175 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.927033+0000", 2024-03-19T11:30:31.175 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.83515094099999998 2024-03-19T11:30:31.175 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.175 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.175 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:31.175 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.927250+0000", 2024-03-19T11:30:31.175 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00021675199999999999 2024-03-19T11:30:31.175 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.176 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.176 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:31.176 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.927489+0000", 2024-03-19T11:30:31.176 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023873300000000001 2024-03-19T11:30:31.176 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:31.176 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:31.176 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:31.176 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.176 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.176 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1474 4.10 4:0dc3b91e:::smithi03829856-28:head [assert-version v27,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-03-19T11:30:31.176 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:27.934139+0000", 2024-03-19T11:30:31.176 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.139836131, 2024-03-19T11:30:31.176 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024764884000000001, 2024-03-19T11:30:31.176 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:31.176 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:31.177 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:31.177 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:31.177 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:31.177 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1474 2024-03-19T11:30:31.177 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.177 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:31.177 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.177 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:31.177 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934139+0000", 2024-03-19T11:30:31.177 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:31.177 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.177 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.177 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:31.177 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934139+0000", 2024-03-19T11:30:31.177 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:31.178 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.178 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.178 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:31.178 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934142+0000", 2024-03-19T11:30:31.178 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5459999999999998e-06 2024-03-19T11:30:31.178 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.178 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.178 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:31.178 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934152+0000", 2024-03-19T11:30:31.178 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9399999999999997e-06 2024-03-19T11:30:31.178 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.178 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.178 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:31.178 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934173+0000", 2024-03-19T11:30:31.178 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1522000000000002e-05 2024-03-19T11:30:31.179 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.179 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.179 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:31.179 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934185+0000", 2024-03-19T11:30:31.179 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1946999999999999e-05 2024-03-19T11:30:31.179 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.179 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.179 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:31.179 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934282+0000", 2024-03-19T11:30:31.179 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6272000000000003e-05 2024-03-19T11:30:31.179 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.179 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.179 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:30:31.179 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934379+0000", 2024-03-19T11:30:31.179 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6966999999999995e-05 2024-03-19T11:30:31.180 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.180 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.180 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:31.180 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.955023+0000", 2024-03-19T11:30:31.180 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020644302 2024-03-19T11:30:31.180 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.180 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.180 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:31.180 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.955167+0000", 2024-03-19T11:30:31.180 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014404500000000001 2024-03-19T11:30:31.180 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.180 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.180 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:31.180 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.957913+0000", 2024-03-19T11:30:31.180 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0027463779999999998 2024-03-19T11:30:31.180 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.181 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.181 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:31.181 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.958548+0000", 2024-03-19T11:30:31.181 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00063495700000000001 2024-03-19T11:30:31.181 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.181 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.181 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:31.181 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.958785+0000", 2024-03-19T11:30:31.181 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000236872 2024-03-19T11:30:31.181 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.181 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.181 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:31.181 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.958854+0000", 2024-03-19T11:30:31.181 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9099999999999999e-05 2024-03-19T11:30:31.181 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:31.182 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:31.182 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:31.182 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.958904+0000", 2024-03-19T11:30:31.182 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0036e-05 2024-03-19T11:30:31.182 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:31.182 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:31.182 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:31.182 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:31.182 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:31.182 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:31.184 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-19T11:30:31.184 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:31.193+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:31.285 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:31.295+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:31.315 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:31.315 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:31.315 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:30:31.315 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:31.329 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-19T11:30:31.386 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:31.395+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:31.451 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:31.452 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:31.452 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:30:31.452 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:30:31.452 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:31.465 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-19T11:30:31.510 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:31.521+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:31.589 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:31.597+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:31.593 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:31.593 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:30:31.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:30:31.594 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:30:31.594 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.594 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:9 4.d 4:b4ef97e0:::smithi03829856-3:head [write 433614~550498,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-03-19T11:30:31.594 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:04.706000+0000", 2024-03-19T11:30:31.594 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 26.897137239999999, 2024-03-19T11:30:31.594 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013227001, 2024-03-19T11:30:31.594 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:31.594 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:31.594 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:31.594 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:31.594 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:31.594 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 9 2024-03-19T11:30:31.594 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.594 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:31.595 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.595 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:31.595 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.706000+0000", 2024-03-19T11:30:31.595 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:31.595 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.595 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.595 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:31.595 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.706000+0000", 2024-03-19T11:30:31.595 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:31.595 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.595 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.595 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:31.595 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.706002+0000", 2024-03-19T11:30:31.595 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5569999999999998e-06 2024-03-19T11:30:31.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:31.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.707261+0000", 2024-03-19T11:30:31.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001258817 2024-03-19T11:30:31.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:31.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.707288+0000", 2024-03-19T11:30:31.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7093e-05 2024-03-19T11:30:31.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:31.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.707305+0000", 2024-03-19T11:30:31.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6330000000000001e-05 2024-03-19T11:30:31.597 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.597 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:31.597 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.707610+0000", 2024-03-19T11:30:31.597 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00030568799999999998 2024-03-19T11:30:31.597 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.597 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:31.597 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.707984+0000", 2024-03-19T11:30:31.597 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00037399299999999998 2024-03-19T11:30:31.597 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.597 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2", 2024-03-19T11:30:31.597 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.708409+0000", 2024-03-19T11:30:31.597 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00042503099999999998 2024-03-19T11:30:31.598 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.598 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.598 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:31.598 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.712919+0000", 2024-03-19T11:30:31.598 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0045099010000000002 2024-03-19T11:30:31.598 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.598 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.598 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:31.598 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.719037+0000", 2024-03-19T11:30:31.598 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061173770000000002 2024-03-19T11:30:31.598 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.598 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.598 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:31.598 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.719116+0000", 2024-03-19T11:30:31.598 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9935e-05 2024-03-19T11:30:31.599 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.599 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.599 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:31.599 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.719227+0000", 2024-03-19T11:30:31.599 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011027900000000001 2024-03-19T11:30:31.599 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:31.599 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:31.599 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:31.599 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.599 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.599 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:848 4.1 4:866f3dcc:::smithi03829856-31:head [read 0~2941192,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:31.599 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.757468+0000", 2024-03-19T11:30:31.599 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.845669248, 2024-03-19T11:30:31.599 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015020110999999999, 2024-03-19T11:30:31.599 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:31.600 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:31.600 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:31.600 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:31.600 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:31.600 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 848 2024-03-19T11:30:31.600 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.600 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:31.600 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.600 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:31.600 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757468+0000", 2024-03-19T11:30:31.600 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:31.600 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.600 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.600 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:31.600 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757468+0000", 2024-03-19T11:30:31.601 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:31.601 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.601 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.601 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:31.601 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757471+0000", 2024-03-19T11:30:31.601 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.67e-06 2024-03-19T11:30:31.601 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.601 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.601 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:31.601 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757488+0000", 2024-03-19T11:30:31.601 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6223e-05 2024-03-19T11:30:31.601 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.601 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.601 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:31.601 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757521+0000", 2024-03-19T11:30:31.602 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.358e-05 2024-03-19T11:30:31.602 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.602 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.602 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:31.602 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757543+0000", 2024-03-19T11:30:31.602 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1330000000000001e-05 2024-03-19T11:30:31.602 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.602 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.602 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:31.602 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757745+0000", 2024-03-19T11:30:31.602 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020259799999999999 2024-03-19T11:30:31.602 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.602 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.603 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op must wait for map", 2024-03-19T11:30:31.603 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757760+0000", 2024-03-19T11:30:31.603 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4647000000000001e-05 2024-03-19T11:30:31.603 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.603 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.603 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:31.603 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763972+0000", 2024-03-19T11:30:31.603 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062126480000000003 2024-03-19T11:30:31.603 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.603 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.603 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:31.603 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.764547+0000", 2024-03-19T11:30:31.603 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057440200000000001 2024-03-19T11:30:31.603 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.603 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.604 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:31.604 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.772488+0000", 2024-03-19T11:30:31.604 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079410130000000002 2024-03-19T11:30:31.604 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:31.604 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:31.604 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:31.604 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.604 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.604 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:849 4.1 4:866f3dcc:::smithi03829856-31:head [sparse-read 0~1359577] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:31.604 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.757583+0000", 2024-03-19T11:30:31.604 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.845553555999999, 2024-03-19T11:30:31.604 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016241202, 2024-03-19T11:30:31.604 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:31.604 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:31.604 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:31.605 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:31.605 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:31.605 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 849 2024-03-19T11:30:31.605 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.605 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:31.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.605 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:31.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757583+0000", 2024-03-19T11:30:31.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:31.605 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.605 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:31.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757583+0000", 2024-03-19T11:30:31.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:31.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.606 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:31.606 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757584+0000", 2024-03-19T11:30:31.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.077e-06 2024-03-19T11:30:31.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.606 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:31.606 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757593+0000", 2024-03-19T11:30:31.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3059999999999998e-06 2024-03-19T11:30:31.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.606 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:31.606 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757608+0000", 2024-03-19T11:30:31.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5526999999999999e-05 2024-03-19T11:30:31.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:31.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757618+0000", 2024-03-19T11:30:31.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5999999999999996e-06 2024-03-19T11:30:31.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:31.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757886+0000", 2024-03-19T11:30:31.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026839099999999999 2024-03-19T11:30:31.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:31.608 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757899+0000", 2024-03-19T11:30:31.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3004e-05 2024-03-19T11:30:31.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.608 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:31.608 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.772651+0000", 2024-03-19T11:30:31.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014751702 2024-03-19T11:30:31.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.608 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:31.608 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.772924+0000", 2024-03-19T11:30:31.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027314199999999998 2024-03-19T11:30:31.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.608 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:31.609 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.773825+0000", 2024-03-19T11:30:31.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00090045299999999995 2024-03-19T11:30:31.609 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:31.609 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:31.609 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:31.609 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.609 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:850 4.1 4:866f3dcc:::smithi03829856-31:head [read 0~2941192,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:31.609 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.757636+0000", 2024-03-19T11:30:31.609 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.845500886, 2024-03-19T11:30:31.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018834801000000002, 2024-03-19T11:30:31.609 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:31.609 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:31.609 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:31.609 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:31.610 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:31.610 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 850 2024-03-19T11:30:31.610 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.610 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:31.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.610 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:31.610 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757636+0000", 2024-03-19T11:30:31.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:31.610 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.610 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:31.610 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757636+0000", 2024-03-19T11:30:31.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:31.610 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:31.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757637+0000", 2024-03-19T11:30:31.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5999999999999991e-07 2024-03-19T11:30:31.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:31.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757643+0000", 2024-03-19T11:30:31.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3199999999999996e-06 2024-03-19T11:30:31.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:31.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757670+0000", 2024-03-19T11:30:31.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6682999999999998e-05 2024-03-19T11:30:31.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:31.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757679+0000", 2024-03-19T11:30:31.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.986e-06 2024-03-19T11:30:31.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:31.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757961+0000", 2024-03-19T11:30:31.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028230799999999999 2024-03-19T11:30:31.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:31.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757971+0000", 2024-03-19T11:30:31.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.38e-06 2024-03-19T11:30:31.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.613 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:31.613 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.774079+0000", 2024-03-19T11:30:31.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016108573000000001 2024-03-19T11:30:31.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.613 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:31.613 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.774306+0000", 2024-03-19T11:30:31.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022637499999999999 2024-03-19T11:30:31.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.613 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:31.613 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776471+0000", 2024-03-19T11:30:31.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021652160000000002 2024-03-19T11:30:31.613 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:31.614 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:31.614 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:31.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.614 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:851 4.9 4:953e720b:::smithi03829856-45:head [write 591613~454162,stat] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:31.614 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.763514+0000", 2024-03-19T11:30:31.614 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.839622694999999, 2024-03-19T11:30:31.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027825888999999999, 2024-03-19T11:30:31.614 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:31.614 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:31.614 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:31.614 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:31.614 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:31.614 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 851 2024-03-19T11:30:31.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.615 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:31.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:31.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763514+0000", 2024-03-19T11:30:31.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:31.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:31.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763514+0000", 2024-03-19T11:30:31.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:31.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:31.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763517+0000", 2024-03-19T11:30:31.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8499999999999998e-06 2024-03-19T11:30:31.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:31.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.764714+0000", 2024-03-19T11:30:31.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011971270000000001 2024-03-19T11:30:31.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:31.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.764745+0000", 2024-03-19T11:30:31.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0546999999999997e-05 2024-03-19T11:30:31.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:31.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.764767+0000", 2024-03-19T11:30:31.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2286999999999999e-05 2024-03-19T11:30:31.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.617 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:31.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.773924+0000", 2024-03-19T11:30:31.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0091567360000000004 2024-03-19T11:30:31.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.617 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:31.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.774217+0000", 2024-03-19T11:30:31.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029350099999999999 2024-03-19T11:30:31.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.617 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:31.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.774770+0000", 2024-03-19T11:30:31.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00055288600000000005 2024-03-19T11:30:31.618 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.618 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:31.618 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.786413+0000", 2024-03-19T11:30:31.618 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011643229999999999 2024-03-19T11:30:31.618 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.618 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:31.618 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791104+0000", 2024-03-19T11:30:31.618 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0046900830000000003 2024-03-19T11:30:31.618 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.618 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:31.618 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791248+0000", 2024-03-19T11:30:31.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014410200000000001 2024-03-19T11:30:31.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:31.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791340+0000", 2024-03-19T11:30:31.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2540000000000005e-05 2024-03-19T11:30:31.619 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:31.619 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:31.619 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:31.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.619 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:852 4.9 4:953e720b:::smithi03829856-45:head [write 1719614~417353] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:31.619 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.769707+0000", 2024-03-19T11:30:31.619 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.833429669000001, 2024-03-19T11:30:31.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022476643000000001, 2024-03-19T11:30:31.620 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:31.620 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:31.620 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:31.620 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:31.620 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:31.620 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 852 2024-03-19T11:30:31.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.620 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:31.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:31.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.769707+0000", 2024-03-19T11:30:31.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:31.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:31.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.769707+0000", 2024-03-19T11:30:31.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:31.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:31.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.769710+0000", 2024-03-19T11:30:31.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6230000000000001e-06 2024-03-19T11:30:31.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:31.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.770385+0000", 2024-03-19T11:30:31.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00067504900000000005 2024-03-19T11:30:31.622 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.622 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.622 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:31.622 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.770419+0000", 2024-03-19T11:30:31.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3528999999999997e-05 2024-03-19T11:30:31.622 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.622 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.622 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:31.622 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.770436+0000", 2024-03-19T11:30:31.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7615999999999999e-05 2024-03-19T11:30:31.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:31.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.781327+0000", 2024-03-19T11:30:31.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010891343 2024-03-19T11:30:31.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:31.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.781616+0000", 2024-03-19T11:30:31.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028857399999999999 2024-03-19T11:30:31.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:31.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.782125+0000", 2024-03-19T11:30:31.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00050941599999999995 2024-03-19T11:30:31.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:31.624 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.786606+0000", 2024-03-19T11:30:31.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044809560000000003 2024-03-19T11:30:31.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:31.624 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791995+0000", 2024-03-19T11:30:31.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053889150000000002 2024-03-19T11:30:31.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:31.624 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.792084+0000", 2024-03-19T11:30:31.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8633000000000003e-05 2024-03-19T11:30:31.625 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:31.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.792184+0000", 2024-03-19T11:30:31.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9988999999999994e-05 2024-03-19T11:30:31.625 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:31.625 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:31.625 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:31.625 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.625 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:853 4.9 4:953e720b:::smithi03829856-45:head [write 2570780~420833] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:31.625 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.775636+0000", 2024-03-19T11:30:31.625 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.827501047999998, 2024-03-19T11:30:31.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018795850999999999, 2024-03-19T11:30:31.626 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:31.626 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:31.626 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:31.626 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:31.626 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:31.626 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 853 2024-03-19T11:30:31.626 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.626 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:31.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.626 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:31.626 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.775636+0000", 2024-03-19T11:30:31.626 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:31.626 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:31.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.775636+0000", 2024-03-19T11:30:31.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:31.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:31.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.775639+0000", 2024-03-19T11:30:31.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4529999999999998e-06 2024-03-19T11:30:31.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:31.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776492+0000", 2024-03-19T11:30:31.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00085245000000000004 2024-03-19T11:30:31.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:31.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776526+0000", 2024-03-19T11:30:31.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4090000000000001e-05 2024-03-19T11:30:31.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:31.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776550+0000", 2024-03-19T11:30:31.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4139999999999999e-05 2024-03-19T11:30:31.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:31.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.786752+0000", 2024-03-19T11:30:31.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010201834 2024-03-19T11:30:31.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.629 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:31.629 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.786994+0000", 2024-03-19T11:30:31.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000241952 2024-03-19T11:30:31.629 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.629 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.629 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:31.629 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.787301+0000", 2024-03-19T11:30:31.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00030690699999999998 2024-03-19T11:30:31.629 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.629 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.629 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:31.629 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791965+0000", 2024-03-19T11:30:31.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0046644379999999999 2024-03-19T11:30:31.629 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.629 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.630 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:31.630 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.794165+0000", 2024-03-19T11:30:31.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021995389999999999 2024-03-19T11:30:31.630 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.630 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:31.630 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.794313+0000", 2024-03-19T11:30:31.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000148369 2024-03-19T11:30:31.630 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.630 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:31.630 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.794432+0000", 2024-03-19T11:30:31.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011867900000000001 2024-03-19T11:30:31.630 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:31.630 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:31.631 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:31.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.631 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:854 4.9 4:953e720b:::smithi03829856-45:head [setxattr _header (56) in=7b,truncate 2991613] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:31.631 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.776592+0000", 2024-03-19T11:30:31.631 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.826545326000002, 2024-03-19T11:30:31.631 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022815621000000001, 2024-03-19T11:30:31.631 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:31.631 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:31.631 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:31.631 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:31.631 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:31.631 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 854 2024-03-19T11:30:31.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.632 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:31.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:31.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776592+0000", 2024-03-19T11:30:31.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:31.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:31.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776592+0000", 2024-03-19T11:30:31.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:31.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:31.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776594+0000", 2024-03-19T11:30:31.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.95e-06 2024-03-19T11:30:31.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:31.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776603+0000", 2024-03-19T11:30:31.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2399999999999996e-06 2024-03-19T11:30:31.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:31.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776623+0000", 2024-03-19T11:30:31.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.035e-05 2024-03-19T11:30:31.634 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.634 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.634 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:31.634 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776639+0000", 2024-03-19T11:30:31.634 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5642999999999999e-05 2024-03-19T11:30:31.634 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.634 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.634 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:31.634 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.792324+0000", 2024-03-19T11:30:31.634 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015685075999999999 2024-03-19T11:30:31.634 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.634 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.634 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:31.634 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.792599+0000", 2024-03-19T11:30:31.634 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000275264 2024-03-19T11:30:31.635 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.635 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:31.635 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.793092+0000", 2024-03-19T11:30:31.635 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00049264600000000001 2024-03-19T11:30:31.635 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.635 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:31.635 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.794668+0000", 2024-03-19T11:30:31.635 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001576072 2024-03-19T11:30:31.635 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.635 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:31.635 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.799082+0000", 2024-03-19T11:30:31.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044145190000000004 2024-03-19T11:30:31.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:31.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.799249+0000", 2024-03-19T11:30:31.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016701199999999999 2024-03-19T11:30:31.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:31.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.799407+0000", 2024-03-19T11:30:31.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000157849 2024-03-19T11:30:31.636 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:31.636 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:31.636 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:31.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.637 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:855 4.9 4:953e720b:::smithi03829856-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:31.637 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.776660+0000", 2024-03-19T11:30:31.637 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.826476733, 2024-03-19T11:30:31.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023419786000000001, 2024-03-19T11:30:31.637 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:31.637 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:31.637 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:31.637 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:31.637 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:31.637 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 855 2024-03-19T11:30:31.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.637 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:31.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.637 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:31.638 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776660+0000", 2024-03-19T11:30:31.638 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:31.638 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.638 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.638 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:31.638 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776660+0000", 2024-03-19T11:30:31.638 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:31.638 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.638 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.638 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:31.638 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776662+0000", 2024-03-19T11:30:31.638 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3430000000000001e-06 2024-03-19T11:30:31.638 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.638 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.638 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:31.639 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776668+0000", 2024-03-19T11:30:31.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3500000000000002e-06 2024-03-19T11:30:31.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.639 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:31.639 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776687+0000", 2024-03-19T11:30:31.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9273999999999999e-05 2024-03-19T11:30:31.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.639 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:31.639 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776705+0000", 2024-03-19T11:30:31.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7839999999999999e-05 2024-03-19T11:30:31.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.639 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:31.640 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.793874+0000", 2024-03-19T11:30:31.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017168597000000001 2024-03-19T11:30:31.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-03-19T11:30:31.640 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.794132+0000", 2024-03-19T11:30:31.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025790399999999999 2024-03-19T11:30:31.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:31.640 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.799504+0000", 2024-03-19T11:30:31.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053724159999999997 2024-03-19T11:30:31.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:31.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.799782+0000", 2024-03-19T11:30:31.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027785499999999997 2024-03-19T11:30:31.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.641 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:31.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.800080+0000", 2024-03-19T11:30:31.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029820699999999999 2024-03-19T11:30:31.641 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:31.641 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:31.641 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:31.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.641 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:871 4.d 4:b4ef97e0:::smithi03829856-3:head [write 642034~796262,stat] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:31.641 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:09.028192+0000", 2024-03-19T11:30:31.641 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.574945002, 2024-03-19T11:30:31.642 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013367027, 2024-03-19T11:30:31.642 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:31.642 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:31.642 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:31.642 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:31.642 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:31.642 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 871 2024-03-19T11:30:31.642 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.642 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:31.642 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.642 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:31.642 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.028192+0000", 2024-03-19T11:30:31.642 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:31.642 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.643 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:31.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.028192+0000", 2024-03-19T11:30:31.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:31.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.643 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:31.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.028194+0000", 2024-03-19T11:30:31.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3999999999999999e-06 2024-03-19T11:30:31.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.643 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:31.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.029358+0000", 2024-03-19T11:30:31.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011635549999999999 2024-03-19T11:30:31.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.644 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:31.644 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.029390+0000", 2024-03-19T11:30:31.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2215999999999998e-05 2024-03-19T11:30:31.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.644 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:31.644 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.029406+0000", 2024-03-19T11:30:31.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5543e-05 2024-03-19T11:30:31.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.644 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:31.644 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.029525+0000", 2024-03-19T11:30:31.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000119269 2024-03-19T11:30:31.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:31.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.029676+0000", 2024-03-19T11:30:31.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000150729 2024-03-19T11:30:31.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2", 2024-03-19T11:30:31.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.029994+0000", 2024-03-19T11:30:31.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00031876099999999998 2024-03-19T11:30:31.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:31.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.040441+0000", 2024-03-19T11:30:31.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010446446 2024-03-19T11:30:31.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:31.646 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.041327+0000", 2024-03-19T11:30:31.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00088595299999999998 2024-03-19T11:30:31.646 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:31.646 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.041409+0000", 2024-03-19T11:30:31.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2332999999999999e-05 2024-03-19T11:30:31.646 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:31.646 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.041559+0000", 2024-03-19T11:30:31.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000149822 2024-03-19T11:30:31.647 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:31.647 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:31.647 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:31.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.647 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:993 4.e 4:72163e4c:::smithi03829856-39:head [write 776490~714607] snapc 5=[5,2] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-03-19T11:30:31.647 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:12.362932+0000", 2024-03-19T11:30:31.647 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.240205309, 2024-03-19T11:30:31.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018349900999999998, 2024-03-19T11:30:31.647 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:31.647 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:31.647 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:31.647 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:31.647 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:31.648 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 993 2024-03-19T11:30:31.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.648 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:31.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.648 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:31.648 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.362932+0000", 2024-03-19T11:30:31.648 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:31.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.648 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:31.648 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.362932+0000", 2024-03-19T11:30:31.648 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:31.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:31.649 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.362935+0000", 2024-03-19T11:30:31.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.88e-06 2024-03-19T11:30:31.649 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:31.649 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.366756+0000", 2024-03-19T11:30:31.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003821217 2024-03-19T11:30:31.649 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.650 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:31.650 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.366796+0000", 2024-03-19T11:30:31.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9866000000000001e-05 2024-03-19T11:30:31.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.650 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.650 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:31.650 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.366813+0000", 2024-03-19T11:30:31.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.755e-05 2024-03-19T11:30:31.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.650 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.650 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:31.650 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.366993+0000", 2024-03-19T11:30:31.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000179605 2024-03-19T11:30:31.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.651 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:31.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.367167+0000", 2024-03-19T11:30:31.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017421200000000001 2024-03-19T11:30:31.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.651 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-19T11:30:31.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.367505+0000", 2024-03-19T11:30:31.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033800099999999999 2024-03-19T11:30:31.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.651 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:31.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.380361+0000", 2024-03-19T11:30:31.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012855961000000001 2024-03-19T11:30:31.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.652 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.652 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:31.652 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.380987+0000", 2024-03-19T11:30:31.652 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00062590499999999995 2024-03-19T11:30:31.652 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.652 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.652 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:31.652 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.381149+0000", 2024-03-19T11:30:31.652 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000162239 2024-03-19T11:30:31.652 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.652 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.652 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:31.652 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.381282+0000", 2024-03-19T11:30:31.652 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013246500000000001 2024-03-19T11:30:31.652 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:31.653 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:31.653 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:31.653 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.653 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.653 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(osd.1.14:42 3.3 3:f0b412b0:::85cff1f6132709ad3f6974c608c4ca3d22c7ae775eb2def51616b490c1e0306e:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:31.653 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:13.796997+0000", 2024-03-19T11:30:31.653 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 17.806139908999999, 2024-03-19T11:30:31.653 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012796138, 2024-03-19T11:30:31.653 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:31.653 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:31.653 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:31.653 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.653 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:31.653 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.796997+0000", 2024-03-19T11:30:31.653 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:31.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.654 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:31.654 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.796997+0000", 2024-03-19T11:30:31.654 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:31.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.654 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:31.654 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.796998+0000", 2024-03-19T11:30:31.654 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5999999999999991e-07 2024-03-19T11:30:31.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.654 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:31.654 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.797240+0000", 2024-03-19T11:30:31.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00024155800000000001 2024-03-19T11:30:31.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:31.655 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.797255+0000", 2024-03-19T11:30:31.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4983e-05 2024-03-19T11:30:31.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:31.655 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.797270+0000", 2024-03-19T11:30:31.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4953000000000001e-05 2024-03-19T11:30:31.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:31.655 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.805058+0000", 2024-03-19T11:30:31.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0077889200000000004 2024-03-19T11:30:31.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.656 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:31.656 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.805289+0000", 2024-03-19T11:30:31.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000230115 2024-03-19T11:30:31.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.656 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:31.656 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.805648+0000", 2024-03-19T11:30:31.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00035961400000000002 2024-03-19T11:30:31.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.656 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:31.656 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.809089+0000", 2024-03-19T11:30:31.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034412499999999999 2024-03-19T11:30:31.657 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.657 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.657 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:31.657 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.809502+0000", 2024-03-19T11:30:31.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00041262100000000001 2024-03-19T11:30:31.657 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.657 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.657 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:31.657 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.809645+0000", 2024-03-19T11:30:31.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000142659 2024-03-19T11:30:31.657 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.657 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.657 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:31.657 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.809793+0000", 2024-03-19T11:30:31.658 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000148505 2024-03-19T11:30:31.658 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:31.658 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:31.658 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:31.658 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.658 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(osd.1.14:43 3.3 3:e8dad25b:::1ae0f6dc7c6ed0f10ae413d6bba06d7bb8915eb6bbafbf428184acbe98e3e057:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:31.658 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:13.797286+0000", 2024-03-19T11:30:31.658 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 17.805851051000001, 2024-03-19T11:30:31.658 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012678534999999999, 2024-03-19T11:30:31.658 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:31.658 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:31.658 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:31.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:31.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.797286+0000", 2024-03-19T11:30:31.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:31.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:31.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.797286+0000", 2024-03-19T11:30:31.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:31.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:31.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.797287+0000", 2024-03-19T11:30:31.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.16e-07 2024-03-19T11:30:31.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.660 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:31.660 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.797426+0000", 2024-03-19T11:30:31.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013880899999999999 2024-03-19T11:30:31.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.660 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:31.660 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.797452+0000", 2024-03-19T11:30:31.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.616e-05 2024-03-19T11:30:31.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.660 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:31.660 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.797468+0000", 2024-03-19T11:30:31.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6249999999999999e-05 2024-03-19T11:30:31.661 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.661 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:31.661 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.807167+0000", 2024-03-19T11:30:31.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0096989290000000002 2024-03-19T11:30:31.661 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.661 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:31.661 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.807529+0000", 2024-03-19T11:30:31.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00036221000000000002 2024-03-19T11:30:31.661 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.661 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:31.661 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.808040+0000", 2024-03-19T11:30:31.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00051056599999999995 2024-03-19T11:30:31.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.662 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:31.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.809455+0000", 2024-03-19T11:30:31.662 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0014148819999999999 2024-03-19T11:30:31.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.662 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:31.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.809833+0000", 2024-03-19T11:30:31.662 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000378694 2024-03-19T11:30:31.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.662 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:31.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.809911+0000", 2024-03-19T11:30:31.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7200000000000006e-05 2024-03-19T11:30:31.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.663 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:31.663 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.809964+0000", 2024-03-19T11:30:31.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3919000000000003e-05 2024-03-19T11:30:31.663 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:31.663 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:31.663 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:31.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.663 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1072 4.1 4:8401c1f0:::smithi03829856-43:head [write 494139~557703,stat] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:31.663 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:13.972007+0000", 2024-03-19T11:30:31.663 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 17.631129559000001, 2024-03-19T11:30:31.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.01935839, 2024-03-19T11:30:31.664 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:31.664 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:31.664 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:31.664 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:31.664 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:31.664 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1072 2024-03-19T11:30:31.664 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.664 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:31.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.664 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:31.664 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.972007+0000", 2024-03-19T11:30:31.664 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:31.664 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.664 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:31.665 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.972007+0000", 2024-03-19T11:30:31.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:31.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.665 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:31.665 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.972011+0000", 2024-03-19T11:30:31.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9360000000000001e-06 2024-03-19T11:30:31.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.665 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:31.665 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.972881+0000", 2024-03-19T11:30:31.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00086922699999999998 2024-03-19T11:30:31.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.665 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:31.666 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.972917+0000", 2024-03-19T11:30:31.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6359999999999997e-05 2024-03-19T11:30:31.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.666 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:31.666 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.972934+0000", 2024-03-19T11:30:31.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7249e-05 2024-03-19T11:30:31.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.666 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:31.666 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.973132+0000", 2024-03-19T11:30:31.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019760200000000001 2024-03-19T11:30:31.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.666 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:31.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.973303+0000", 2024-03-19T11:30:31.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017142899999999999 2024-03-19T11:30:31.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:31.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.973619+0000", 2024-03-19T11:30:31.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00031594700000000002 2024-03-19T11:30:31.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:31.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.985856+0000", 2024-03-19T11:30:31.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012236349000000001 2024-03-19T11:30:31.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:31.668 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.991132+0000", 2024-03-19T11:30:31.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052764960000000003 2024-03-19T11:30:31.668 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.668 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.668 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:31.668 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.991241+0000", 2024-03-19T11:30:31.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000108842 2024-03-19T11:30:31.668 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.668 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.668 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:31.668 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.991366+0000", 2024-03-19T11:30:31.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012495299999999999 2024-03-19T11:30:31.668 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:31.668 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:31.668 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:31.668 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.669 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.669 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1162 4.1 4:8401c1f0:::smithi03829856-43:head [write 594981~609197] snapc 8=[8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-03-19T11:30:31.669 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.838981+0000", 2024-03-19T11:30:31.669 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.764156204000001, 2024-03-19T11:30:31.669 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027704349999999999, 2024-03-19T11:30:31.669 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:31.669 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:31.669 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:31.669 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:31.669 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:31.669 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1162 2024-03-19T11:30:31.669 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.669 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:31.669 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.670 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:31.670 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.838981+0000", 2024-03-19T11:30:31.670 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:31.670 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.670 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.670 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:31.670 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.838981+0000", 2024-03-19T11:30:31.670 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:31.670 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.670 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.670 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:31.670 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.838983+0000", 2024-03-19T11:30:31.670 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5399999999999998e-06 2024-03-19T11:30:31.670 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.670 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.671 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:31.671 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841322+0000", 2024-03-19T11:30:31.671 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002338592 2024-03-19T11:30:31.671 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.671 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:31.671 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841371+0000", 2024-03-19T11:30:31.671 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9123999999999999e-05 2024-03-19T11:30:31.671 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.671 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:31.671 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841399+0000", 2024-03-19T11:30:31.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7718000000000001e-05 2024-03-19T11:30:31.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:31.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841577+0000", 2024-03-19T11:30:31.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000178097 2024-03-19T11:30:31.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:31.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841825+0000", 2024-03-19T11:30:31.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00024824299999999999 2024-03-19T11:30:31.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:31.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842227+0000", 2024-03-19T11:30:31.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00040212199999999998 2024-03-19T11:30:31.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.673 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:31.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.865182+0000", 2024-03-19T11:30:31.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022954810999999999 2024-03-19T11:30:31.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.673 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:31.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.866360+0000", 2024-03-19T11:30:31.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001178449 2024-03-19T11:30:31.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.673 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:31.674 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.866538+0000", 2024-03-19T11:30:31.674 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017791000000000001 2024-03-19T11:30:31.674 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.674 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.674 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:31.674 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.866685+0000", 2024-03-19T11:30:31.674 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000146744 2024-03-19T11:30:31.674 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:31.674 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:31.674 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:31.674 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.674 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.674 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1163 4.1 4:8401c1f0:::smithi03829856-43:head [write 1974123~745028] snapc 8=[8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-03-19T11:30:31.674 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.851879+0000", 2024-03-19T11:30:31.675 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.751258082, 2024-03-19T11:30:31.675 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015531774, 2024-03-19T11:30:31.675 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:31.675 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:31.675 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:31.675 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:31.675 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:31.675 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1163 2024-03-19T11:30:31.675 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.675 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:31.675 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.675 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:31.675 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.851879+0000", 2024-03-19T11:30:31.675 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:31.675 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.676 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.676 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:31.676 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.851879+0000", 2024-03-19T11:30:31.676 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:31.676 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.676 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.676 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:31.676 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.851882+0000", 2024-03-19T11:30:31.676 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6960000000000001e-06 2024-03-19T11:30:31.676 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.676 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.676 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:31.676 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.853204+0000", 2024-03-19T11:30:31.676 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0013226620000000001 2024-03-19T11:30:31.676 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.677 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:31.677 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.853231+0000", 2024-03-19T11:30:31.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6775e-05 2024-03-19T11:30:31.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.677 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:31.677 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.853246+0000", 2024-03-19T11:30:31.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5403000000000001e-05 2024-03-19T11:30:31.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.677 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:31.677 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.853363+0000", 2024-03-19T11:30:31.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000116296 2024-03-19T11:30:31.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.678 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.678 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:31.678 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.853572+0000", 2024-03-19T11:30:31.678 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020931500000000001 2024-03-19T11:30:31.678 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.678 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.678 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:31.678 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.853797+0000", 2024-03-19T11:30:31.678 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022544700000000001 2024-03-19T11:30:31.678 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.678 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.678 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:31.678 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.865195+0000", 2024-03-19T11:30:31.678 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011397981999999999 2024-03-19T11:30:31.679 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.679 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:31.679 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.866916+0000", 2024-03-19T11:30:31.679 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017204449999999999 2024-03-19T11:30:31.679 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.679 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:31.679 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.867129+0000", 2024-03-19T11:30:31.679 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021317999999999999 2024-03-19T11:30:31.679 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.679 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:31.679 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.867411+0000", 2024-03-19T11:30:31.679 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028157300000000002 2024-03-19T11:30:31.680 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:31.680 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:31.680 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:31.680 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.680 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.680 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1205 4.1 4:8401c1f0:::smithi03829856-43:head [assert-version v50,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:31.680 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:18.112028+0000", 2024-03-19T11:30:31.680 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 13.491108948000001, 2024-03-19T11:30:31.680 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013579206, 2024-03-19T11:30:31.680 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:31.680 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:31.680 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:31.680 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:31.680 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:31.680 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1205 2024-03-19T11:30:31.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.681 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:31.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.681 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:31.681 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112028+0000", 2024-03-19T11:30:31.681 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:31.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.681 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:31.681 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112028+0000", 2024-03-19T11:30:31.681 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:31.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.681 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:31.681 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112030+0000", 2024-03-19T11:30:31.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4549999999999998e-06 2024-03-19T11:30:31.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.682 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.682 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:31.682 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112043+0000", 2024-03-19T11:30:31.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2721999999999999e-05 2024-03-19T11:30:31.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.682 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.682 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:31.682 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112075+0000", 2024-03-19T11:30:31.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.205e-05 2024-03-19T11:30:31.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.682 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.682 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:31.683 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112091+0000", 2024-03-19T11:30:31.683 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5364000000000001e-05 2024-03-19T11:30:31.683 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.683 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.683 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:31.683 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112223+0000", 2024-03-19T11:30:31.683 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013224100000000001 2024-03-19T11:30:31.683 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.683 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.683 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:31.683 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112393+0000", 2024-03-19T11:30:31.683 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000169681 2024-03-19T11:30:31.683 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.683 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.683 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:31.684 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.120919+0000", 2024-03-19T11:30:31.684 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085260140000000002 2024-03-19T11:30:31.684 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.684 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.684 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:31.684 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125207+0000", 2024-03-19T11:30:31.684 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004288722 2024-03-19T11:30:31.684 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.684 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.684 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:31.684 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125343+0000", 2024-03-19T11:30:31.684 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013558400000000001 2024-03-19T11:30:31.684 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.684 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.684 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:31.685 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125497+0000", 2024-03-19T11:30:31.685 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000153678 2024-03-19T11:30:31.685 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.685 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.685 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:31.685 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125607+0000", 2024-03-19T11:30:31.685 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110695 2024-03-19T11:30:31.685 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:31.685 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:31.685 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:31.685 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.685 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.685 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1208 4.1 4:866f3dcc:::smithi03829856-31:head [read 0~2941192,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:31.685 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:18.112252+0000", 2024-03-19T11:30:31.685 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 13.490885378, 2024-03-19T11:30:31.685 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.01291702, 2024-03-19T11:30:31.686 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:31.686 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:31.686 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:31.686 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:31.686 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:31.686 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1208 2024-03-19T11:30:31.686 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.686 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:31.686 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.686 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:31.686 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112252+0000", 2024-03-19T11:30:31.686 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:31.686 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.686 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.687 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:31.687 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112252+0000", 2024-03-19T11:30:31.687 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:31.687 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.687 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.687 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:31.687 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112253+0000", 2024-03-19T11:30:31.687 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6299999999999993e-07 2024-03-19T11:30:31.687 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.687 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.687 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:31.687 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112257+0000", 2024-03-19T11:30:31.687 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4360000000000004e-06 2024-03-19T11:30:31.687 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.688 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.688 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:31.688 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112272+0000", 2024-03-19T11:30:31.688 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.518e-05 2024-03-19T11:30:31.688 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.688 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.688 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:31.688 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112281+0000", 2024-03-19T11:30:31.688 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.422e-06 2024-03-19T11:30:31.688 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.688 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.688 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:31.688 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.122861+0000", 2024-03-19T11:30:31.688 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010580523 2024-03-19T11:30:31.688 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.689 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.689 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:31.689 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123148+0000", 2024-03-19T11:30:31.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028714499999999999 2024-03-19T11:30:31.689 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.689 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.689 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:31.689 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125169+0000", 2024-03-19T11:30:31.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0020203510000000001 2024-03-19T11:30:31.689 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:31.689 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:31.689 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:31.689 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.689 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.689 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1243 4.9 4:9ef41243:::smithi03829856-5:head [tier-promote] snapc 9=[9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:31.690 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:18.345759+0000", 2024-03-19T11:30:31.690 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 13.257377766999999, 2024-03-19T11:30:31.690 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018400962999999999, 2024-03-19T11:30:31.690 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:31.690 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:31.690 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:31.690 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:31.690 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:31.690 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1243 2024-03-19T11:30:31.690 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.690 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:31.690 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.690 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:31.690 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.345759+0000", 2024-03-19T11:30:31.690 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:31.691 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.691 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.691 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:31.691 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.345759+0000", 2024-03-19T11:30:31.691 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:31.691 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.691 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.691 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:31.691 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.345763+0000", 2024-03-19T11:30:31.691 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3790000000000002e-06 2024-03-19T11:30:31.691 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.691 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.691 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:31.691 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.345776+0000", 2024-03-19T11:30:31.691 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3784000000000001e-05 2024-03-19T11:30:31.692 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.692 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.692 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:31.692 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.345812+0000", 2024-03-19T11:30:31.692 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5599999999999998e-05 2024-03-19T11:30:31.692 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.692 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.692 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:31.692 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.345829+0000", 2024-03-19T11:30:31.692 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6722999999999999e-05 2024-03-19T11:30:31.692 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.692 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.692 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:31.692 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.345964+0000", 2024-03-19T11:30:31.692 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013554800000000001 2024-03-19T11:30:31.693 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.693 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.693 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:31.693 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.346133+0000", 2024-03-19T11:30:31.693 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000169211 2024-03-19T11:30:31.693 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.693 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.693 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:31.693 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.364160+0000", 2024-03-19T11:30:31.693 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018026718000000001 2024-03-19T11:30:31.693 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:31.693 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:31.693 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:31.693 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.693 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.694 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1271 4.9 4:953e720b:::smithi03829856-45:2 [read 795136~1008128] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-03-19T11:30:31.694 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:21.868056+0000", 2024-03-19T11:30:31.694 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.7350806989999992, 2024-03-19T11:30:31.694 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.047270591000000001, 2024-03-19T11:30:31.694 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:31.694 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:31.694 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:31.694 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:31.694 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:31.694 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1271 2024-03-19T11:30:31.694 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.694 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:31.694 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.694 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:31.694 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868056+0000", 2024-03-19T11:30:31.695 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:31.695 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.695 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.695 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:31.695 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868056+0000", 2024-03-19T11:30:31.695 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:31.695 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.695 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.695 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:31.695 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868059+0000", 2024-03-19T11:30:31.695 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0469999999999998e-06 2024-03-19T11:30:31.695 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.695 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.695 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:31.695 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868072+0000", 2024-03-19T11:30:31.696 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2700999999999999e-05 2024-03-19T11:30:31.696 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.696 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.696 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:31.696 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868106+0000", 2024-03-19T11:30:31.696 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4152000000000001e-05 2024-03-19T11:30:31.696 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.696 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.696 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:31.696 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868126+0000", 2024-03-19T11:30:31.696 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9386999999999999e-05 2024-03-19T11:30:31.696 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.696 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.696 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:31.696 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.911447+0000", 2024-03-19T11:30:31.697 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.043321164000000002 2024-03-19T11:30:31.697 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.697 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.697 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:31.697 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.911762+0000", 2024-03-19T11:30:31.697 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00031492400000000001 2024-03-19T11:30:31.697 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:31.697 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:31.697 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:31.697 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.915327+0000", 2024-03-19T11:30:31.697 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003565216 2024-03-19T11:30:31.697 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:31.697 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:31.698 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:31.698 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:31.698 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:31.698 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:31.699 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-19T11:30:31.700 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:31.709+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:31.800 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:31.809+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:31.871 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:31.871 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:31.871 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:30:31.871 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:31.885 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-19T11:30:31.900 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:31.911+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:32.001 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:32.011+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:32.047 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-03-19T11:30:32.050 INFO:tasks.rados.rados.0.smithi038.stdout:887: done (5 left) 2024-03-19T11:30:32.051 INFO:tasks.rados.rados.0.smithi038.stdout:888: write oid 2 current snap is 13 2024-03-19T11:30:32.051 INFO:tasks.rados.rados.0.smithi038.stdout:882: finishing set_chunk to oid 37 2024-03-19T11:30:32.051 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 37 set_chunk returned error code -95 offset: 496640 length: 690176 2024-03-19T11:30:32.051 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 37 v 0 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-03-19T11:30:32.051 INFO:tasks.rados.rados.0.smithi038.stdout:883: finishing set_chunk to oid 26 2024-03-19T11:30:32.051 INFO:tasks.rados.rados.0.smithi038.stdout:883: got expected ENOENT (src dne) 2024-03-19T11:30:32.051 INFO:tasks.rados.rados.0.smithi038.stdout:885: expect (ObjNum 3 snap 0 seq_num 3) 2024-03-19T11:30:32.055 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:32.055 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:32.055 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:30:32.055 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:30:32.055 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:32.071 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-19T11:30:32.118 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:32.129+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:32.124 INFO:tasks.rados.rados.0.smithi038.stdout:884: expect (ObjNum 68 snap 3 seq_num 68) 2024-03-19T11:30:32.197 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:32.198 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:30:32.198 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:30:32.198 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:30:32.198 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.198 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:844 4.7 e32/21 4:e41d47fb:::smithi03829856-14:head v 32'8, mlcod=28'5)", 2024-03-19T11:30:32.198 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.751447+0000", 2024-03-19T11:30:32.198 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.455814455999999, 2024-03-19T11:30:32.198 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012301835000000001, 2024-03-19T11:30:32.198 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:32.198 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:32.198 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:32.198 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.198 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:32.199 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.751447+0000", 2024-03-19T11:30:32.199 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:32.199 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.199 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:32.199 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.751447+0000", 2024-03-19T11:30:32.199 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:32.199 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.199 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:32.199 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.751447+0000", 2024-03-19T11:30:32.199 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4300000000000003e-07 2024-03-19T11:30:32.199 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.199 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:32.200 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.752747+0000", 2024-03-19T11:30:32.200 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012995699999999999 2024-03-19T11:30:32.200 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.200 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.200 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:32.200 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.752771+0000", 2024-03-19T11:30:32.200 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3592999999999999e-05 2024-03-19T11:30:32.200 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.200 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.200 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:32.200 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.752789+0000", 2024-03-19T11:30:32.200 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8703e-05 2024-03-19T11:30:32.200 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.200 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.200 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:32.201 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.752911+0000", 2024-03-19T11:30:32.201 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012139899999999999 2024-03-19T11:30:32.201 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.201 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.201 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:32.201 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.752949+0000", 2024-03-19T11:30:32.201 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8436999999999998e-05 2024-03-19T11:30:32.201 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.201 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.201 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:32.201 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763568+0000", 2024-03-19T11:30:32.201 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010618768000000001 2024-03-19T11:30:32.201 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.201 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.201 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:32.202 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763749+0000", 2024-03-19T11:30:32.202 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000180822 2024-03-19T11:30:32.202 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:32.202 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:32.202 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:32.202 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.202 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.202 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1047 4.8 4:1d58db11:::smithi03829856-18:head [write 443718~615603,stat] snapc 6=[6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:32.202 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:13.221766+0000", 2024-03-19T11:30:32.202 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.985495288999999, 2024-03-19T11:30:32.202 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020932596000000001, 2024-03-19T11:30:32.202 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:32.202 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:32.202 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:32.203 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:32.203 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:32.203 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1047 2024-03-19T11:30:32.203 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.203 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:32.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.203 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:32.203 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.221766+0000", 2024-03-19T11:30:32.203 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:32.203 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.203 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:32.203 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.221766+0000", 2024-03-19T11:30:32.203 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:32.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.204 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:32.204 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.221770+0000", 2024-03-19T11:30:32.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.67e-06 2024-03-19T11:30:32.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.204 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:32.204 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.223938+0000", 2024-03-19T11:30:32.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021683399999999999 2024-03-19T11:30:32.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.204 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:32.204 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.223979+0000", 2024-03-19T11:30:32.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1433000000000001e-05 2024-03-19T11:30:32.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.205 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:32.205 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.223998+0000", 2024-03-19T11:30:32.205 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8853000000000001e-05 2024-03-19T11:30:32.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.205 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:32.205 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.224150+0000", 2024-03-19T11:30:32.205 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000151679 2024-03-19T11:30:32.206 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.206 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.206 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:32.206 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.224362+0000", 2024-03-19T11:30:32.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021154499999999999 2024-03-19T11:30:32.206 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.206 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.206 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:32.206 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.224745+0000", 2024-03-19T11:30:32.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00038342399999999999 2024-03-19T11:30:32.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.207 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:32.207 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.241167+0000", 2024-03-19T11:30:32.207 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0164218 2024-03-19T11:30:32.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.207 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:32.207 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.242406+0000", 2024-03-19T11:30:32.207 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012390039999999999 2024-03-19T11:30:32.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.207 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:32.208 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.242575+0000", 2024-03-19T11:30:32.208 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000168925 2024-03-19T11:30:32.208 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.208 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.209 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:32.209 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.242699+0000", 2024-03-19T11:30:32.209 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012392300000000001 2024-03-19T11:30:32.209 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:32.209 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:32.209 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:32.209 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.209 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.209 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1048 4.8 4:1d58db11:::smithi03829856-18:head [write 1564138~470341] snapc 6=[6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:32.209 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:13.227287+0000", 2024-03-19T11:30:32.209 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.979974021, 2024-03-19T11:30:32.209 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015802897999999999, 2024-03-19T11:30:32.209 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:32.210 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:32.210 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:32.210 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:32.210 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:32.210 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1048 2024-03-19T11:30:32.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.210 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:32.210 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.210 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:32.210 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.227287+0000", 2024-03-19T11:30:32.210 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:32.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.210 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.210 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:32.210 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.227287+0000", 2024-03-19T11:30:32.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:32.211 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.211 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:32.211 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.227290+0000", 2024-03-19T11:30:32.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.88e-06 2024-03-19T11:30:32.211 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.211 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:32.211 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.228224+0000", 2024-03-19T11:30:32.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00093389600000000003 2024-03-19T11:30:32.211 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.211 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:32.211 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.228258+0000", 2024-03-19T11:30:32.212 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4282999999999997e-05 2024-03-19T11:30:32.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.212 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:32.212 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.228273+0000", 2024-03-19T11:30:32.212 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4833e-05 2024-03-19T11:30:32.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.212 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:32.212 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.231996+0000", 2024-03-19T11:30:32.212 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037223410000000001 2024-03-19T11:30:32.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.212 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:32.213 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.232143+0000", 2024-03-19T11:30:32.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014714900000000001 2024-03-19T11:30:32.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.213 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.213 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:32.213 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.232395+0000", 2024-03-19T11:30:32.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025201500000000002 2024-03-19T11:30:32.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.213 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.213 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:32.213 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.242848+0000", 2024-03-19T11:30:32.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010453393 2024-03-19T11:30:32.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.213 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.213 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:32.214 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.242872+0000", 2024-03-19T11:30:32.214 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3816e-05 2024-03-19T11:30:32.214 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.214 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.214 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:32.214 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.242966+0000", 2024-03-19T11:30:32.214 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4560000000000003e-05 2024-03-19T11:30:32.214 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.214 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.214 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:32.214 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.243090+0000", 2024-03-19T11:30:32.214 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012373199999999999 2024-03-19T11:30:32.214 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:32.214 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:32.215 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:32.215 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.215 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.215 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1072 4.1 e36/21 4:8401c1f0:::smithi03829856-43:head v 36'38, mlcod=36'35)", 2024-03-19T11:30:32.215 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:13.973829+0000", 2024-03-19T11:30:32.215 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.233432698000001, 2024-03-19T11:30:32.215 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013548782000000001, 2024-03-19T11:30:32.215 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:32.215 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:32.215 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:32.215 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.215 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:32.215 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.973829+0000", 2024-03-19T11:30:32.215 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:32.215 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.216 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:32.216 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.973829+0000", 2024-03-19T11:30:32.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:32.216 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.216 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:32.216 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.973831+0000", 2024-03-19T11:30:32.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0899999999999999e-06 2024-03-19T11:30:32.216 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.216 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:32.216 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.974489+0000", 2024-03-19T11:30:32.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00065813199999999997 2024-03-19T11:30:32.216 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:32.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.974523+0000", 2024-03-19T11:30:32.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3841999999999999e-05 2024-03-19T11:30:32.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:32.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.974544+0000", 2024-03-19T11:30:32.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0907000000000002e-05 2024-03-19T11:30:32.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:32.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.974697+0000", 2024-03-19T11:30:32.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015295599999999999 2024-03-19T11:30:32.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.218 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.218 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:32.218 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.974742+0000", 2024-03-19T11:30:32.218 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5735999999999997e-05 2024-03-19T11:30:32.218 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.218 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.218 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:32.218 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.987261+0000", 2024-03-19T11:30:32.218 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012518316999999999 2024-03-19T11:30:32.218 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.218 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.218 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:32.218 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.987377+0000", 2024-03-19T11:30:32.218 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000116802 2024-03-19T11:30:32.218 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:32.219 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:32.219 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:32.219 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.219 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1091 4.8 4:14611006:::smithi03829856-25:head [write 527753~708201] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:32.219 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:14.015839+0000", 2024-03-19T11:30:32.219 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.191422638999999, 2024-03-19T11:30:32.219 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016721983999999999, 2024-03-19T11:30:32.219 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:32.219 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:32.219 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:32.219 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:32.219 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:32.219 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1091 2024-03-19T11:30:32.219 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.220 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:32.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.220 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:32.220 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.015839+0000", 2024-03-19T11:30:32.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:32.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.220 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:32.220 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.015839+0000", 2024-03-19T11:30:32.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:32.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.220 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:32.220 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.015842+0000", 2024-03-19T11:30:32.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3569999999999998e-06 2024-03-19T11:30:32.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:32.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.016678+0000", 2024-03-19T11:30:32.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00083620299999999999 2024-03-19T11:30:32.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:32.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.016710+0000", 2024-03-19T11:30:32.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1337000000000002e-05 2024-03-19T11:30:32.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:32.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.016724+0000", 2024-03-19T11:30:32.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4323e-05 2024-03-19T11:30:32.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:32.222 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.016847+0000", 2024-03-19T11:30:32.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012304199999999999 2024-03-19T11:30:32.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:32.222 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.017023+0000", 2024-03-19T11:30:32.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000175945 2024-03-19T11:30:32.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:32.222 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.017324+0000", 2024-03-19T11:30:32.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00030151499999999998 2024-03-19T11:30:32.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.223 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:32.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.032104+0000", 2024-03-19T11:30:32.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014779232999999999 2024-03-19T11:30:32.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.223 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:32.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.032190+0000", 2024-03-19T11:30:32.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6304999999999994e-05 2024-03-19T11:30:32.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.223 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:32.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.032350+0000", 2024-03-19T11:30:32.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000159813 2024-03-19T11:30:32.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.224 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:32.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.032561+0000", 2024-03-19T11:30:32.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021091099999999999 2024-03-19T11:30:32.224 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:32.224 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:32.224 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:32.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.224 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1158 4.8 4:15708482:::smithi03829856-23:head [tier-promote] snapc 8=[8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-03-19T11:30:32.224 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.827517+0000", 2024-03-19T11:30:32.224 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.379744711000001, 2024-03-19T11:30:32.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.034096983999999997, 2024-03-19T11:30:32.224 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:32.224 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:32.224 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:32.225 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:32.225 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:32.225 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1158 2024-03-19T11:30:32.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.225 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:32.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:32.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827517+0000", 2024-03-19T11:30:32.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:32.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:32.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827517+0000", 2024-03-19T11:30:32.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:32.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:32.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827520+0000", 2024-03-19T11:30:32.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.45e-06 2024-03-19T11:30:32.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:32.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827535+0000", 2024-03-19T11:30:32.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.509e-05 2024-03-19T11:30:32.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:32.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827579+0000", 2024-03-19T11:30:32.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4211000000000001e-05 2024-03-19T11:30:32.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.227 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:32.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827603+0000", 2024-03-19T11:30:32.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3586000000000001e-05 2024-03-19T11:30:32.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.227 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:32.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827789+0000", 2024-03-19T11:30:32.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018580199999999999 2024-03-19T11:30:32.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.227 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:32.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.828077+0000", 2024-03-19T11:30:32.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028846500000000001 2024-03-19T11:30:32.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.228 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.228 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:32.228 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.861614+0000", 2024-03-19T11:30:32.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033536379999999998 2024-03-19T11:30:32.228 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:32.228 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:32.228 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:32.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.228 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.228 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1162 4.1 e39/21 4:8401c1f0:::smithi03829856-43:head v 39'47, mlcod=38'43)", 2024-03-19T11:30:32.228 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.842598+0000", 2024-03-19T11:30:32.228 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.364662941000001, 2024-03-19T11:30:32.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0233756, 2024-03-19T11:30:32.228 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:32.228 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:32.229 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:32.229 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.229 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:32.229 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842598+0000", 2024-03-19T11:30:32.229 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:32.229 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.229 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.229 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:32.229 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842598+0000", 2024-03-19T11:30:32.229 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:32.229 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.229 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.229 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:32.229 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842599+0000", 2024-03-19T11:30:32.229 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6000000000000003e-07 2024-03-19T11:30:32.230 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.230 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:32.230 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.843566+0000", 2024-03-19T11:30:32.230 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00096688700000000004 2024-03-19T11:30:32.230 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.230 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:32.230 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.843601+0000", 2024-03-19T11:30:32.230 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.502e-05 2024-03-19T11:30:32.230 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.230 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:32.230 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.843621+0000", 2024-03-19T11:30:32.230 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0305e-05 2024-03-19T11:30:32.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.231 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:32.231 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.843882+0000", 2024-03-19T11:30:32.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026027100000000002 2024-03-19T11:30:32.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.231 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:32.231 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.844022+0000", 2024-03-19T11:30:32.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014020100000000001 2024-03-19T11:30:32.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.231 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:32.231 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.865862+0000", 2024-03-19T11:30:32.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021839725000000001 2024-03-19T11:30:32.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.232 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:32.232 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.865974+0000", 2024-03-19T11:30:32.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011243099999999999 2024-03-19T11:30:32.232 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:32.232 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:32.232 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:32.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.232 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1163 4.1 e39/21 4:8401c1f0:::smithi03829856-43:head v 39'48, mlcod=38'43)", 2024-03-19T11:30:32.232 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.854006+0000", 2024-03-19T11:30:32.232 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.353255254, 2024-03-19T11:30:32.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012036872000000001, 2024-03-19T11:30:32.232 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:32.233 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:32.233 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:32.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.233 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:32.233 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.854006+0000", 2024-03-19T11:30:32.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:32.233 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.233 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:32.233 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.854006+0000", 2024-03-19T11:30:32.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:32.233 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.233 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:32.233 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.854008+0000", 2024-03-19T11:30:32.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0190000000000001e-06 2024-03-19T11:30:32.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.234 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.234 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:32.234 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.854542+0000", 2024-03-19T11:30:32.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00053378200000000001 2024-03-19T11:30:32.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.234 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.234 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:32.234 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.854577+0000", 2024-03-19T11:30:32.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4740000000000003e-05 2024-03-19T11:30:32.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.234 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.234 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:32.235 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.854597+0000", 2024-03-19T11:30:32.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0225999999999999e-05 2024-03-19T11:30:32.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:32.235 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.858165+0000", 2024-03-19T11:30:32.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035682790000000002 2024-03-19T11:30:32.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:32.235 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.858231+0000", 2024-03-19T11:30:32.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5916999999999999e-05 2024-03-19T11:30:32.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:32.236 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.865995+0000", 2024-03-19T11:30:32.236 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0077640720000000003 2024-03-19T11:30:32.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.236 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:32.236 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.866043+0000", 2024-03-19T11:30:32.236 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7837000000000003e-05 2024-03-19T11:30:32.236 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:32.236 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:32.236 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:32.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.236 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1209 4.8 4:1d58db11:::smithi03829856-18:head [write 645402~551297,stat] snapc 9=[9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:32.236 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:18.121657+0000", 2024-03-19T11:30:32.237 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.085603939, 2024-03-19T11:30:32.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.025074349999999999, 2024-03-19T11:30:32.237 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:32.237 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:32.237 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:32.237 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:32.237 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:32.237 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1209 2024-03-19T11:30:32.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.237 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:32.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:32.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.121657+0000", 2024-03-19T11:30:32.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:32.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:32.238 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.121657+0000", 2024-03-19T11:30:32.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:32.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:32.238 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.121661+0000", 2024-03-19T11:30:32.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0489999999999997e-06 2024-03-19T11:30:32.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:32.238 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123747+0000", 2024-03-19T11:30:32.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0020857720000000001 2024-03-19T11:30:32.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:32.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123779+0000", 2024-03-19T11:30:32.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2258000000000002e-05 2024-03-19T11:30:32.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:32.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123795+0000", 2024-03-19T11:30:32.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5177000000000001e-05 2024-03-19T11:30:32.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:32.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123949+0000", 2024-03-19T11:30:32.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015403699999999999 2024-03-19T11:30:32.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:32.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.124134+0000", 2024-03-19T11:30:32.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000185484 2024-03-19T11:30:32.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:32.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.124554+0000", 2024-03-19T11:30:32.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00042019699999999998 2024-03-19T11:30:32.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:32.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146267+0000", 2024-03-19T11:30:32.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021712838000000002 2024-03-19T11:30:32.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:32.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146424+0000", 2024-03-19T11:30:32.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015647299999999999 2024-03-19T11:30:32.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:32.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146597+0000", 2024-03-19T11:30:32.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017303699999999999 2024-03-19T11:30:32.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:32.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146732+0000", 2024-03-19T11:30:32.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013502800000000001 2024-03-19T11:30:32.242 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:32.242 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:32.242 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:32.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.242 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1210 4.8 4:1d58db11:::smithi03829856-18:head [write 1749569~492355] snapc 9=[9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:32.242 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:18.129257+0000", 2024-03-19T11:30:32.242 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.078004582, 2024-03-19T11:30:32.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018110444999999999, 2024-03-19T11:30:32.242 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:32.242 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:32.242 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:32.243 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:32.243 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:32.243 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1210 2024-03-19T11:30:32.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.243 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:32.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.243 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:32.243 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.129257+0000", 2024-03-19T11:30:32.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:32.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:32.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.129257+0000", 2024-03-19T11:30:32.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:32.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:32.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.129261+0000", 2024-03-19T11:30:32.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7119999999999999e-06 2024-03-19T11:30:32.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:32.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.129717+0000", 2024-03-19T11:30:32.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00045526300000000002 2024-03-19T11:30:32.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:32.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.129749+0000", 2024-03-19T11:30:32.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2700999999999997e-05 2024-03-19T11:30:32.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:32.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.129774+0000", 2024-03-19T11:30:32.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4592000000000001e-05 2024-03-19T11:30:32.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:32.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.134875+0000", 2024-03-19T11:30:32.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.005100477 2024-03-19T11:30:32.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:32.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.135035+0000", 2024-03-19T11:30:32.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016054200000000001 2024-03-19T11:30:32.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.247 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:32.247 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.135268+0000", 2024-03-19T11:30:32.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000233253 2024-03-19T11:30:32.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.247 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:32.247 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146872+0000", 2024-03-19T11:30:32.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011603732 2024-03-19T11:30:32.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.248 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:32.248 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.147103+0000", 2024-03-19T11:30:32.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023070299999999999 2024-03-19T11:30:32.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.248 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:32.248 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.147181+0000", 2024-03-19T11:30:32.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8567999999999997e-05 2024-03-19T11:30:32.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:32.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.147367+0000", 2024-03-19T11:30:32.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000185902 2024-03-19T11:30:32.249 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:32.249 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:32.249 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:32.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.249 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1259 4.1 e44/43 4:866f3dcc:::smithi03829856-31:head v 44'53, mlcod=0'0)", 2024-03-19T11:30:32.249 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:21.869109+0000", 2024-03-19T11:30:32.249 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.338152692, 2024-03-19T11:30:32.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014777017999999999, 2024-03-19T11:30:32.250 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:32.250 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:32.250 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:32.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:32.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.869109+0000", 2024-03-19T11:30:32.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:32.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:32.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.869109+0000", 2024-03-19T11:30:32.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:32.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:32.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.869110+0000", 2024-03-19T11:30:32.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0559999999999999e-06 2024-03-19T11:30:32.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:32.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.870408+0000", 2024-03-19T11:30:32.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012978499999999999 2024-03-19T11:30:32.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:32.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.870437+0000", 2024-03-19T11:30:32.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9638999999999999e-05 2024-03-19T11:30:32.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:32.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.870460+0000", 2024-03-19T11:30:32.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2517000000000001e-05 2024-03-19T11:30:32.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:32.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.870681+0000", 2024-03-19T11:30:32.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022102800000000001 2024-03-19T11:30:32.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:32.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.870735+0000", 2024-03-19T11:30:32.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4616e-05 2024-03-19T11:30:32.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:32.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.883773+0000", 2024-03-19T11:30:32.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013038088 2024-03-19T11:30:32.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:32.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.883886+0000", 2024-03-19T11:30:32.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000112224 2024-03-19T11:30:32.253 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:32.253 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:32.253 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:32.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.254 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1416 4.18 4:1dabc0f1:::smithi03829856-1:head [write 564735~613734,stat] snapc c=[c,b,a,6,5] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:32.254 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.949508+0000", 2024-03-19T11:30:32.254 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.2577533619999999, 2024-03-19T11:30:32.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016253106, 2024-03-19T11:30:32.254 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:32.254 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:32.254 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:32.254 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:32.254 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:32.254 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1416 2024-03-19T11:30:32.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.255 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:32.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:32.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.949508+0000", 2024-03-19T11:30:32.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:32.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:32.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.949508+0000", 2024-03-19T11:30:32.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:32.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:32.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.949511+0000", 2024-03-19T11:30:32.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2370000000000001e-06 2024-03-19T11:30:32.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:32.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.950555+0000", 2024-03-19T11:30:32.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001043925 2024-03-19T11:30:32.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:32.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.950594+0000", 2024-03-19T11:30:32.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9351999999999999e-05 2024-03-19T11:30:32.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:32.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.950612+0000", 2024-03-19T11:30:32.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7524e-05 2024-03-19T11:30:32.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:32.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.950745+0000", 2024-03-19T11:30:32.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013291 2024-03-19T11:30:32.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:32.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.951003+0000", 2024-03-19T11:30:32.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025828400000000001 2024-03-19T11:30:32.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:32.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.951335+0000", 2024-03-19T11:30:32.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033130200000000001 2024-03-19T11:30:32.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:32.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.965379+0000", 2024-03-19T11:30:32.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014044581 2024-03-19T11:30:32.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:32.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.965502+0000", 2024-03-19T11:30:32.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012290099999999999 2024-03-19T11:30:32.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:32.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.965668+0000", 2024-03-19T11:30:32.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016619200000000001 2024-03-19T11:30:32.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:32.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.965761+0000", 2024-03-19T11:30:32.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2898000000000004e-05 2024-03-19T11:30:32.259 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:32.259 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:32.259 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:32.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.260 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1417 4.18 4:1dabc0f1:::smithi03829856-1:head [write 1828021~669034] snapc c=[c,b,a,6,5] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:32.260 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.958407+0000", 2024-03-19T11:30:32.260 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.248853961, 2024-03-19T11:30:32.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016467840000000001, 2024-03-19T11:30:32.260 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:32.260 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:32.260 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:32.260 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:32.260 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:32.260 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1417 2024-03-19T11:30:32.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.260 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:32.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:32.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.958407+0000", 2024-03-19T11:30:32.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:32.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.261 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:32.261 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.958407+0000", 2024-03-19T11:30:32.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:32.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.261 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:32.261 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.958410+0000", 2024-03-19T11:30:32.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9830000000000001e-06 2024-03-19T11:30:32.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.261 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:32.262 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.959442+0000", 2024-03-19T11:30:32.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001031957 2024-03-19T11:30:32.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.262 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:32.262 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.959475+0000", 2024-03-19T11:30:32.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3145999999999997e-05 2024-03-19T11:30:32.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.262 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:32.262 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.959494+0000", 2024-03-19T11:30:32.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8994000000000001e-05 2024-03-19T11:30:32.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.263 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:32.263 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.959717+0000", 2024-03-19T11:30:32.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000222272 2024-03-19T11:30:32.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.263 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:32.263 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.960015+0000", 2024-03-19T11:30:32.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029870599999999998 2024-03-19T11:30:32.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.263 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:32.263 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.960312+0000", 2024-03-19T11:30:32.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029640299999999999 2024-03-19T11:30:32.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:32.264 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.966435+0000", 2024-03-19T11:30:32.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061234999999999996 2024-03-19T11:30:32.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:32.264 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.974556+0000", 2024-03-19T11:30:32.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.008120759 2024-03-19T11:30:32.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:32.264 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.974729+0000", 2024-03-19T11:30:32.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000173191 2024-03-19T11:30:32.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:32.265 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.974875+0000", 2024-03-19T11:30:32.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014592899999999999 2024-03-19T11:30:32.265 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:32.265 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:32.265 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:32.265 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.265 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.265 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1419 4.18 4:1dabc0f1:::smithi03829856-1:head [setxattr _header (56) in=7b,truncate 3764735] snapc c=[c,b,a,6,5] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:32.265 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.968061+0000", 2024-03-19T11:30:32.265 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.2392001019999999, 2024-03-19T11:30:32.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012422488000000001, 2024-03-19T11:30:32.265 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:32.265 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:32.265 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:32.265 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:32.266 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:32.266 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1419 2024-03-19T11:30:32.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.266 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:32.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.266 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:32.266 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968061+0000", 2024-03-19T11:30:32.266 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:32.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.266 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:32.266 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968061+0000", 2024-03-19T11:30:32.266 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:32.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.267 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:32.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968063+0000", 2024-03-19T11:30:32.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.767e-06 2024-03-19T11:30:32.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.267 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:32.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968074+0000", 2024-03-19T11:30:32.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0525e-05 2024-03-19T11:30:32.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.267 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:32.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968096+0000", 2024-03-19T11:30:32.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2926999999999999e-05 2024-03-19T11:30:32.268 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.268 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.268 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:32.268 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968113+0000", 2024-03-19T11:30:32.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6665000000000001e-05 2024-03-19T11:30:32.268 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.268 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.268 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:32.268 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.973218+0000", 2024-03-19T11:30:32.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0051044920000000004 2024-03-19T11:30:32.268 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.268 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.268 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:32.268 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.973511+0000", 2024-03-19T11:30:32.269 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029303899999999998 2024-03-19T11:30:32.269 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.269 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.269 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:32.269 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.973931+0000", 2024-03-19T11:30:32.269 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00041991400000000002 2024-03-19T11:30:32.269 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.269 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.269 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:32.269 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.977029+0000", 2024-03-19T11:30:32.269 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030983040000000001 2024-03-19T11:30:32.269 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.269 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.269 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:32.269 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.980091+0000", 2024-03-19T11:30:32.270 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003061851 2024-03-19T11:30:32.270 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.270 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.270 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:32.270 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.980285+0000", 2024-03-19T11:30:32.270 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000194279 2024-03-19T11:30:32.270 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.270 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.270 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:32.270 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.980484+0000", 2024-03-19T11:30:32.270 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019872499999999999 2024-03-19T11:30:32.270 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:32.270 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:32.271 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:32.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.271 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1420 4.18 4:1dabc0f1:::smithi03829856-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:32.271 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.968138+0000", 2024-03-19T11:30:32.271 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.2391228920000001, 2024-03-19T11:30:32.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013109305999999999, 2024-03-19T11:30:32.271 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:32.271 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:32.271 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:32.271 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:32.272 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:32.272 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1420 2024-03-19T11:30:32.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.272 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:32.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:32.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968138+0000", 2024-03-19T11:30:32.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:32.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:32.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968138+0000", 2024-03-19T11:30:32.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:32.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:32.273 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968140+0000", 2024-03-19T11:30:32.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4139999999999999e-06 2024-03-19T11:30:32.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:32.273 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968147+0000", 2024-03-19T11:30:32.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.815e-06 2024-03-19T11:30:32.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:32.273 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968167+0000", 2024-03-19T11:30:32.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0018e-05 2024-03-19T11:30:32.274 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.274 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.274 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:32.274 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968182+0000", 2024-03-19T11:30:32.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5475e-05 2024-03-19T11:30:32.274 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.274 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.274 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:32.274 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.974950+0000", 2024-03-19T11:30:32.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.006767856 2024-03-19T11:30:32.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-03-19T11:30:32.275 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.975095+0000", 2024-03-19T11:30:32.275 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014501499999999999 2024-03-19T11:30:32.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:32.275 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.980614+0000", 2024-03-19T11:30:32.275 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0055184789999999997 2024-03-19T11:30:32.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:32.275 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.980912+0000", 2024-03-19T11:30:32.275 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029858199999999998 2024-03-19T11:30:32.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:32.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.981248+0000", 2024-03-19T11:30:32.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033565199999999998 2024-03-19T11:30:32.276 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:32.276 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:32.276 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:32.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.276 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1443 4.2 4:43b32242:::smithi03829856-22:head [read 0~2352947,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:32.277 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.979417+0000", 2024-03-19T11:30:32.277 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.227844266, 2024-03-19T11:30:32.277 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.96719988700000004, 2024-03-19T11:30:32.277 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:32.277 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:32.277 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:32.277 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:32.277 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:32.277 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1443 2024-03-19T11:30:32.277 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.277 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:32.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.277 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:32.277 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979417+0000", 2024-03-19T11:30:32.277 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:32.278 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.278 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.278 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:32.278 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979417+0000", 2024-03-19T11:30:32.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:32.278 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.278 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.278 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:32.278 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979420+0000", 2024-03-19T11:30:32.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.366e-06 2024-03-19T11:30:32.279 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.279 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:32.279 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979437+0000", 2024-03-19T11:30:32.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6192e-05 2024-03-19T11:30:32.279 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.279 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:32.279 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979474+0000", 2024-03-19T11:30:32.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7601000000000001e-05 2024-03-19T11:30:32.279 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.279 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:32.280 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979496+0000", 2024-03-19T11:30:32.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1506999999999999e-05 2024-03-19T11:30:32.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.280 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:32.280 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979712+0000", 2024-03-19T11:30:32.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021641299999999999 2024-03-19T11:30:32.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.280 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:30:32.280 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979746+0000", 2024-03-19T11:30:32.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3386000000000002e-05 2024-03-19T11:30:32.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:32.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.931552+0000", 2024-03-19T11:30:32.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.95180659899999998 2024-03-19T11:30:32.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-03-19T11:30:32.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.934681+0000", 2024-03-19T11:30:32.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0031292289999999999 2024-03-19T11:30:32.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:32.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.942701+0000", 2024-03-19T11:30:32.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0080198809999999995 2024-03-19T11:30:32.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:32.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.942839+0000", 2024-03-19T11:30:32.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013816999999999999 2024-03-19T11:30:32.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:32.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.946617+0000", 2024-03-19T11:30:32.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003777543 2024-03-19T11:30:32.283 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:32.283 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:32.283 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:32.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.283 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1444 4.2 4:43b32242:::smithi03829856-22:head [read 0~2352947,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:32.284 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.979553+0000", 2024-03-19T11:30:32.284 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.2277085630000002, 2024-03-19T11:30:32.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.97015945400000003, 2024-03-19T11:30:32.284 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:32.284 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:32.284 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:32.284 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:32.284 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:32.284 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1444 2024-03-19T11:30:32.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.284 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:32.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.284 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:32.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979553+0000", 2024-03-19T11:30:32.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:32.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:32.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979553+0000", 2024-03-19T11:30:32.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:32.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:32.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979554+0000", 2024-03-19T11:30:32.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.403e-06 2024-03-19T11:30:32.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:32.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979564+0000", 2024-03-19T11:30:32.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0032e-05 2024-03-19T11:30:32.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:32.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979579+0000", 2024-03-19T11:30:32.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4875000000000001e-05 2024-03-19T11:30:32.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:32.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979595+0000", 2024-03-19T11:30:32.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6300999999999999e-05 2024-03-19T11:30:32.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.287 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:32.287 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979874+0000", 2024-03-19T11:30:32.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000279065 2024-03-19T11:30:32.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.287 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:30:32.287 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979904+0000", 2024-03-19T11:30:32.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.94e-05 2024-03-19T11:30:32.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.287 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:32.287 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.934781+0000", 2024-03-19T11:30:32.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.95487744900000004 2024-03-19T11:30:32.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.288 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-03-19T11:30:32.288 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.934946+0000", 2024-03-19T11:30:32.288 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016485999999999999 2024-03-19T11:30:32.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.288 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:32.288 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.946829+0000", 2024-03-19T11:30:32.288 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011882482999999999 2024-03-19T11:30:32.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.288 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:32.288 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.946989+0000", 2024-03-19T11:30:32.288 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000160421 2024-03-19T11:30:32.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:32.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.949712+0000", 2024-03-19T11:30:32.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002723165 2024-03-19T11:30:32.289 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:32.289 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:32.289 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:32.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.289 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1445 4.2 4:43b32242:::smithi03829856-22:head [read 0~2352947,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:32.289 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.979653+0000", 2024-03-19T11:30:32.289 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.2276084320000002, 2024-03-19T11:30:32.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.97249507999999996, 2024-03-19T11:30:32.289 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:32.289 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:32.289 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:32.289 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:32.290 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:32.290 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1445 2024-03-19T11:30:32.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.290 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:32.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:32.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979653+0000", 2024-03-19T11:30:32.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:32.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:32.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979653+0000", 2024-03-19T11:30:32.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:32.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:32.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979654+0000", 2024-03-19T11:30:32.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.372e-06 2024-03-19T11:30:32.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:32.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979662+0000", 2024-03-19T11:30:32.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9959999999999993e-06 2024-03-19T11:30:32.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:32.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979698+0000", 2024-03-19T11:30:32.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5763000000000003e-05 2024-03-19T11:30:32.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.292 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:32.292 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979714+0000", 2024-03-19T11:30:32.292 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6283999999999999e-05 2024-03-19T11:30:32.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.292 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:32.292 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979990+0000", 2024-03-19T11:30:32.292 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000275829 2024-03-19T11:30:32.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.292 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:30:32.292 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.980018+0000", 2024-03-19T11:30:32.292 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8082999999999999e-05 2024-03-19T11:30:32.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.292 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:32.293 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.935620+0000", 2024-03-19T11:30:32.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.95560133800000002 2024-03-19T11:30:32.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.293 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-03-19T11:30:32.293 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.935795+0000", 2024-03-19T11:30:32.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000175656 2024-03-19T11:30:32.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.293 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:32.293 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.949835+0000", 2024-03-19T11:30:32.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014040083 2024-03-19T11:30:32.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.293 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:32.294 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.950002+0000", 2024-03-19T11:30:32.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016673 2024-03-19T11:30:32.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.294 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:32.294 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.952148+0000", 2024-03-19T11:30:32.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021459460000000001 2024-03-19T11:30:32.294 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:32.294 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:32.294 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:32.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.294 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1449 4.18 4:1d58db11:::smithi03829856-18:head [write 581542~688983,stat] snapc c=[c,b,a,6,5] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:32.294 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.989354+0000", 2024-03-19T11:30:32.294 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.2179069790000003, 2024-03-19T11:30:32.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019237318, 2024-03-19T11:30:32.295 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:32.295 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:32.295 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:32.295 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:32.295 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:32.295 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1449 2024-03-19T11:30:32.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.295 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:32.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:32.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.989354+0000", 2024-03-19T11:30:32.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:32.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.296 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:32.296 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.989354+0000", 2024-03-19T11:30:32.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:32.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.296 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:32.296 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.989357+0000", 2024-03-19T11:30:32.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2740000000000002e-06 2024-03-19T11:30:32.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.296 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:32.296 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.990109+0000", 2024-03-19T11:30:32.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00075226500000000001 2024-03-19T11:30:32.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:32.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.990142+0000", 2024-03-19T11:30:32.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3328999999999999e-05 2024-03-19T11:30:32.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:32.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.990168+0000", 2024-03-19T11:30:32.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6016999999999998e-05 2024-03-19T11:30:32.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:32.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.990291+0000", 2024-03-19T11:30:32.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000122607 2024-03-19T11:30:32.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:32.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.990461+0000", 2024-03-19T11:30:32.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000169743 2024-03-19T11:30:32.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:32.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.990792+0000", 2024-03-19T11:30:32.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033135100000000002 2024-03-19T11:30:32.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:32.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.008238+0000", 2024-03-19T11:30:32.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017445591999999999 2024-03-19T11:30:32.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:32.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.008345+0000", 2024-03-19T11:30:32.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010762 2024-03-19T11:30:32.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:32.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.008436+0000", 2024-03-19T11:30:32.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1116e-05 2024-03-19T11:30:32.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:32.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.008592+0000", 2024-03-19T11:30:32.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015540399999999999 2024-03-19T11:30:32.299 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:32.299 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:32.299 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:32.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.300 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1450 4.18 4:1d58db11:::smithi03829856-18:head [write 1886503~412692] snapc c=[c,b,a,6,5] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:32.300 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.992973+0000", 2024-03-19T11:30:32.300 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.2142882100000003, 2024-03-19T11:30:32.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016912580999999999, 2024-03-19T11:30:32.300 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:32.300 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:32.300 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:32.300 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:32.300 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:32.300 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1450 2024-03-19T11:30:32.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.300 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:32.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.301 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:32.301 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.992973+0000", 2024-03-19T11:30:32.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:32.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.301 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:32.301 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.992973+0000", 2024-03-19T11:30:32.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:32.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.301 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:32.301 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.992976+0000", 2024-03-19T11:30:32.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0060000000000001e-06 2024-03-19T11:30:32.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.301 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:32.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.993655+0000", 2024-03-19T11:30:32.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00067901099999999998 2024-03-19T11:30:32.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:32.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.993691+0000", 2024-03-19T11:30:32.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5546000000000002e-05 2024-03-19T11:30:32.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:32.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.993713+0000", 2024-03-19T11:30:32.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2277e-05 2024-03-19T11:30:32.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:32.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.997961+0000", 2024-03-19T11:30:32.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0042479659999999997 2024-03-19T11:30:32.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:32.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.998130+0000", 2024-03-19T11:30:32.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016908900000000001 2024-03-19T11:30:32.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:32.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.998397+0000", 2024-03-19T11:30:32.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026707599999999999 2024-03-19T11:30:32.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:32.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.008811+0000", 2024-03-19T11:30:32.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010413751000000001 2024-03-19T11:30:32.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:32.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.009539+0000", 2024-03-19T11:30:32.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00072811499999999999 2024-03-19T11:30:32.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:32.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.009711+0000", 2024-03-19T11:30:32.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017196900000000001 2024-03-19T11:30:32.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:32.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:32.305 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:32.305 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.009886+0000", 2024-03-19T11:30:32.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017477500000000001 2024-03-19T11:30:32.305 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:32.305 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:32.305 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:32.305 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:32.305 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:32.305 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:32.306 INFO:tasks.rados.rados.0.smithi038.stdout:886: expect (ObjNum 23 snap 0 seq_num 23) 2024-03-19T11:30:32.307 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-19T11:30:32.307 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:32.317+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:32.346 INFO:tasks.rados.rados.0.smithi038.stdout:888: seq_num 117 ranges {766419=549428,2013452=574695,3346843=427649} 2024-03-19T11:30:32.353 INFO:tasks.rados.rados.0.smithi038.stdout:888: writing smithi03829856-2 from 766419 to 1315847 tid 1 2024-03-19T11:30:32.361 INFO:tasks.rados.rados.0.smithi038.stdout:888: writing smithi03829856-2 from 2013452 to 2588147 tid 2 2024-03-19T11:30:32.371 INFO:tasks.rados.rados.0.smithi038.stdout:888: writing smithi03829856-2 from 3346843 to 3774492 tid 3 2024-03-19T11:30:32.371 INFO:tasks.rados.rados.0.smithi038.stdout:882: done (5 left) 2024-03-19T11:30:32.371 INFO:tasks.rados.rados.0.smithi038.stdout:883: done (4 left) 2024-03-19T11:30:32.371 INFO:tasks.rados.rados.0.smithi038.stdout:884: done (3 left) 2024-03-19T11:30:32.371 INFO:tasks.rados.rados.0.smithi038.stdout:885: done (2 left) 2024-03-19T11:30:32.372 INFO:tasks.rados.rados.0.smithi038.stdout:886: done (1 left) 2024-03-19T11:30:32.372 INFO:tasks.rados.rados.0.smithi038.stdout:889: write (excl) oid 16 current snap is 13 2024-03-19T11:30:32.372 INFO:tasks.rados.rados.0.smithi038.stdout:888: finishing write tid 1 to smithi03829856-2 2024-03-19T11:30:32.372 INFO:tasks.rados.rados.0.smithi038.stdout:889: seq_num 118 ranges {783226=624677,2071934=434706,2974266=638921} 2024-03-19T11:30:32.376 INFO:tasks.rados.rados.0.smithi038.stdout:889: writing smithi03829856-16 from 783226 to 1407903 tid 1 2024-03-19T11:30:32.383 INFO:tasks.rados.rados.0.smithi038.stdout:889: writing smithi03829856-16 from 2071934 to 2506640 tid 2 2024-03-19T11:30:32.389 INFO:tasks.rados.rados.0.smithi038.stdout:889: writing smithi03829856-16 from 2974266 to 3613187 tid 3 2024-03-19T11:30:32.389 INFO:tasks.rados.rados.0.smithi038.stdout:890: read oid 12 snap -1 2024-03-19T11:30:32.389 INFO:tasks.rados.rados.0.smithi038.stdout:890: expect (ObjNum 11 snap 0 seq_num 11) 2024-03-19T11:30:32.390 INFO:tasks.rados.rados.0.smithi038.stdout:891: snap_remove snap 8 2024-03-19T11:30:32.407 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:32.417+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:32.435 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-19T11:30:32.478 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:32.479 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:32.479 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:30:32.479 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:32.492 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-19T11:30:32.534 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:32.545+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:32.587 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:30:32.588 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:30:32.605 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:30:32.605 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:30:32.608 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:32.619+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:32.621 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:30:32.621 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:30:32.640 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:30:32.640 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:30:32.659 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:32.659 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:32.659 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:30:32.659 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:30:32.659 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:32.660 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:30:32.660 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:30:32.674 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-19T11:30:32.680 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:30:32.680 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:30:32.700 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:30:32.700 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:30:32.719 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:30:32.719 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:30:32.720 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:32.731+0000 7f70c4fdb640 -1 osd.0 54 handle_osd_map injecting map crc failure 2024-03-19T11:30:32.726 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:32.737+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:32.798 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:32.798 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:32.811 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:32.819+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:32.812 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-19T11:30:32.911 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:32.921+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:32.984 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:32.984 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:32.998 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-19T11:30:33.012 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:33.023+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:33.016 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-03-19T11:30:33.025 INFO:tasks.rados.rados.0.smithi038.stdout:891: done (3 left) 2024-03-19T11:30:33.025 INFO:tasks.rados.rados.0.smithi038.stdout:892: delete oid 48 current snap is 13 2024-03-19T11:30:33.025 INFO:tasks.rados.rados.0.smithi038.stdout:888: finishing write tid 2 to smithi03829856-2 2024-03-19T11:30:33.025 INFO:tasks.rados.rados.0.smithi038.stdout:888: finishing write tid 3 to smithi03829856-2 2024-03-19T11:30:33.025 INFO:tasks.rados.rados.0.smithi038.stdout:888: finishing write tid 4 to smithi03829856-2 2024-03-19T11:30:33.025 INFO:tasks.rados.rados.0.smithi038.stdout:888: finishing write tid 5 to smithi03829856-2 2024-03-19T11:30:33.025 INFO:tasks.rados.rados.0.smithi038.stdout:888: oid 2 updating version 0 to 59 2024-03-19T11:30:33.025 INFO:tasks.rados.rados.0.smithi038.stdout:888: oid 2 version 59 is already newer than 57 2024-03-19T11:30:33.025 INFO:tasks.rados.rados.0.smithi038.stdout:888: oid 2 version 59 is already newer than 58 2024-03-19T11:30:33.026 INFO:tasks.rados.rados.0.smithi038.stdout:888: oid 2 version 59 is already newer than 56 2024-03-19T11:30:33.026 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 2 v 59 (ObjNum 117 snap 13 seq_num 117) dirty exists 2024-03-19T11:30:33.026 INFO:tasks.rados.rados.0.smithi038.stdout:888: left oid 2 (ObjNum 117 snap 13 seq_num 117) 2024-03-19T11:30:33.026 INFO:tasks.rados.rados.0.smithi038.stdout:889: finishing write tid 1 to smithi03829856-16 2024-03-19T11:30:33.026 INFO:tasks.rados.rados.0.smithi038.stdout:889: finishing write tid 2 to smithi03829856-16 2024-03-19T11:30:33.026 INFO:tasks.rados.rados.0.smithi038.stdout:889: finishing write tid 3 to smithi03829856-16 2024-03-19T11:30:33.026 INFO:tasks.rados.rados.0.smithi038.stdout:889: finishing write tid 4 to smithi03829856-16 2024-03-19T11:30:33.026 INFO:tasks.rados.rados.0.smithi038.stdout:889: finishing write tid 5 to smithi03829856-16 2024-03-19T11:30:33.026 INFO:tasks.rados.rados.0.smithi038.stdout:889: oid 16 updating version 0 to 84 2024-03-19T11:30:33.026 INFO:tasks.rados.rados.0.smithi038.stdout:889: oid 16 version 84 is already newer than 82 2024-03-19T11:30:33.026 INFO:tasks.rados.rados.0.smithi038.stdout:889: oid 16 version 84 is already newer than 83 2024-03-19T11:30:33.026 INFO:tasks.rados.rados.0.smithi038.stdout:889: oid 16 updating version 84 to 85 2024-03-19T11:30:33.026 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 16 v 85 (ObjNum 118 snap 13 seq_num 118) dirty exists 2024-03-19T11:30:33.026 INFO:tasks.rados.rados.0.smithi038.stdout:889: left oid 16 (ObjNum 118 snap 13 seq_num 118) 2024-03-19T11:30:33.027 INFO:tasks.rados.rados.0.smithi038.stdout:890: expect (ObjNum 11 snap 0 seq_num 11) 2024-03-19T11:30:33.103 INFO:tasks.rados.rados.0.smithi038.stdout:888: done (3 left) 2024-03-19T11:30:33.104 INFO:tasks.rados.rados.0.smithi038.stdout:889: done (2 left) 2024-03-19T11:30:33.104 INFO:tasks.rados.rados.0.smithi038.stdout:890: done (1 left) 2024-03-19T11:30:33.104 INFO:tasks.rados.rados.0.smithi038.stdout:892: done (0 left) 2024-03-19T11:30:33.104 INFO:tasks.rados.rados.0.smithi038.stdout:893: delete oid 42 current snap is 13 2024-03-19T11:30:33.112 INFO:tasks.rados.rados.0.smithi038.stdout:893: done (0 left) 2024-03-19T11:30:33.112 INFO:tasks.rados.rados.0.smithi038.stdout:894: write (excl) oid 19 current snap is 13 2024-03-19T11:30:33.112 INFO:tasks.rados.rados.0.smithi038.stdout:894: seq_num 121 ranges {433647=450424,1331027=582033} 2024-03-19T11:30:33.112 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:33.123+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:33.119 INFO:tasks.rados.rados.0.smithi038.stdout:894: writing smithi03829856-19 from 433647 to 884071 tid 1 2024-03-19T11:30:33.126 INFO:tasks.rados.rados.0.smithi038.stdout:894: writing smithi03829856-19 from 1331027 to 1913060 tid 2 2024-03-19T11:30:33.127 INFO:tasks.rados.rados.0.smithi038.stdout:895: snap_create 2024-03-19T11:30:33.132 INFO:tasks.rados.rados.0.smithi038.stdout:894: finishing write tid 1 to smithi03829856-19 2024-03-19T11:30:33.132 INFO:tasks.rados.rados.0.smithi038.stdout:894: finishing write tid 2 to smithi03829856-19 2024-03-19T11:30:33.132 INFO:tasks.rados.rados.0.smithi038.stdout:894: finishing write tid 3 to smithi03829856-19 2024-03-19T11:30:33.132 INFO:tasks.rados.rados.0.smithi038.stdout:894: finishing write tid 4 to smithi03829856-19 2024-03-19T11:30:33.132 INFO:tasks.rados.rados.0.smithi038.stdout:894: oid 19 updating version 0 to 44 2024-03-19T11:30:33.132 INFO:tasks.rados.rados.0.smithi038.stdout:894: oid 19 version 44 is already newer than 42 2024-03-19T11:30:33.132 INFO:tasks.rados.rados.0.smithi038.stdout:894: oid 19 version 44 is already newer than 43 2024-03-19T11:30:33.132 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 19 v 44 (ObjNum 121 snap 13 seq_num 121) dirty exists 2024-03-19T11:30:33.132 INFO:tasks.rados.rados.0.smithi038.stdout:894: left oid 19 (ObjNum 121 snap 13 seq_num 121) 2024-03-19T11:30:33.162 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:33.163 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:33.176 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-19T11:30:33.222 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:33.233+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:33.300 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:33.300 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:33.314 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:33.325+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:33.315 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-19T11:30:33.416 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:33.425+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:33.449 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:33.449 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:33.463 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-19T11:30:33.517 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:33.525+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:33.586 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:33.586 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:33.602 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-19T11:30:33.618 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:33.629+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:33.719 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:33.729+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:33.721 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:33.722 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:33.736 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-19T11:30:33.820 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:33.829+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:33.862 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:33.862 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:33.877 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-19T11:30:33.923 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:33.935+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:33.997 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:33.997 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:34.012 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-19T11:30:34.021 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:34.029+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:34.052 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-19T11:30:34.052 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-19T11:30:34.052 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-03-19T11:30:34.052 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T11:30:34.074 INFO:tasks.rados.rados.0.smithi038.stdout:894: done (1 left) 2024-03-19T11:30:34.074 INFO:tasks.rados.rados.0.smithi038.stdout:895: done (0 left) 2024-03-19T11:30:34.074 INFO:tasks.rados.rados.0.smithi038.stdout:896: snap_create 2024-03-19T11:30:34.122 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:34.131+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:34.183 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:34.183 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:34.200 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-19T11:30:34.224 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:34.233+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:34.325 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:34.333+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:34.366 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:34.366 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:34.368 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:30:34.369 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-03-19T11:30:34.383 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-19T11:30:34.390 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2024-03-19T11:30:34.144036+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":316538207,"num_objects":195,"num_object_clones":42,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":3,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":192,"num_whiteouts":7,"num_read":1008,"num_read_kb":619651,"num_write":840,"num_write_kb":254670,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":110806449,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":61,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1599,"ondisk_log_size":1599,"up":78,"acting":78,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":79,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":598336,"kb_used_data":367420,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":754376384,"statfs":{"total":773094113280,"available":772481417216,"internally_reserved":0,"allocated":376238080,"data_stored":431206055,"data_compressed":10847965,"data_compressed_allocated":11583488,"data_compressed_original":26669056,"omap_allocated":12751,"internal_metadata":234016305},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1,0,0,0,0,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7215458,"num_objects":6,"num_object_clones":6,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":-273,"num_read_kb":-282600,"num_write":-64,"num_write_kb":-17143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":22999678,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001041"},"pg_stats":[{"pgid":"4.19","version":"56'57","reported_seq":269,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-19T11:30:33.038526+0000","last_change":"2024-03-19T11:30:33.038910+0000","last_active":"2024-03-19T11:30:33.038526+0000","last_peered":"2024-03-19T11:30:33.038526+0000","last_clean":"2024-03-19T11:30:33.038526+0000","last_became_active":"2024-03-19T11:30:30.051696+0000","last_became_peered":"2024-03-19T11:30:30.051696+0000","last_unstale":"2024-03-19T11:30:33.038526+0000","last_undegraded":"2024-03-19T11:30:33.038526+0000","last_fullsized":"2024-03-19T11:30:33.038526+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":5,"last_scrub":"41'37","last_scrub_stamp":"2024-03-19T11:30:27.705159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:27.705159+0000","objects_scrubbed":4,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:42.260816+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0063715789999999996,"stat_sum":{"num_bytes":12043149,"num_objects":6,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":10,"num_read_kb":9227,"num_write":11,"num_write_kb":2452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":8331740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":6,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"},{"start":"5","length":"1"},{"start":"a","length":"1"}]},{"pgid":"4.18","version":"48'77","reported_seq":207,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-19T11:30:29.045743+0000","last_change":"2024-03-19T11:30:28.881543+0000","last_active":"2024-03-19T11:30:29.045743+0000","last_peered":"2024-03-19T11:30:29.045743+0000","last_clean":"2024-03-19T11:30:29.045743+0000","last_became_active":"2024-03-19T11:30:21.869150+0000","last_became_peered":"2024-03-19T11:30:21.869150+0000","last_unstale":"2024-03-19T11:30:29.045743+0000","last_undegraded":"2024-03-19T11:30:29.045743+0000","last_fullsized":"2024-03-19T11:30:29.045743+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.85038975500000002,"stat_sum":{"num_bytes":11859254,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":1776,"num_write":40,"num_write_kb":12663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"2"}]},{"pgid":"4.17","version":"41'24","reported_seq":118,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-19T11:30:29.046020+0000","last_change":"2024-03-19T11:30:28.040142+0000","last_active":"2024-03-19T11:30:29.046020+0000","last_peered":"2024-03-19T11:30:29.046020+0000","last_clean":"2024-03-19T11:30:29.046020+0000","last_became_active":"2024-03-19T11:30:28.039784+0000","last_became_peered":"2024-03-19T11:30:28.039784+0000","last_unstale":"2024-03-19T11:30:29.046020+0000","last_undegraded":"2024-03-19T11:30:29.046020+0000","last_fullsized":"2024-03-19T11:30:29.046020+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3679215,"num_objects":2,"num_object_clones":2,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":9,"num_write_kb":2643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"2"}]},{"pgid":"4.16","version":"49'56","reported_seq":228,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-19T11:30:29.046242+0000","last_change":"2024-03-19T11:30:27.016107+0000","last_active":"2024-03-19T11:30:29.046242+0000","last_peered":"2024-03-19T11:30:29.046242+0000","last_clean":"2024-03-19T11:30:29.046242+0000","last_became_active":"2024-03-19T11:30:26.921950+0000","last_became_peered":"2024-03-19T11:30:26.921950+0000","last_unstale":"2024-03-19T11:30:29.046242+0000","last_undegraded":"2024-03-19T11:30:29.046242+0000","last_fullsized":"2024-03-19T11:30:29.046242+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.004615264,"stat_sum":{"num_bytes":7262556,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":51,"num_read_kb":46224,"num_write":24,"num_write_kb":6418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":10351680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"2"}]},{"pgid":"4.15","version":"55'78","reported_seq":321,"reported_epoch":56,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:30:33.042664+0000","last_change":"2024-03-19T11:30:32.067428+0000","last_active":"2024-03-19T11:30:33.042664+0000","last_peered":"2024-03-19T11:30:33.042664+0000","last_clean":"2024-03-19T11:30:33.042664+0000","last_became_active":"2024-03-19T11:30:28.043875+0000","last_became_peered":"2024-03-19T11:30:28.043875+0000","last_unstale":"2024-03-19T11:30:33.042664+0000","last_undegraded":"2024-03-19T11:30:33.042664+0000","last_fullsized":"2024-03-19T11:30:33.042664+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":5,"last_scrub":"49'74","last_scrub_stamp":"2024-03-19T11:30:29.457542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:29.457542+0000","objects_scrubbed":7,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:50.094386+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00082877000000000005,"stat_sum":{"num_bytes":17403119,"num_objects":7,"num_object_clones":2,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":6,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"4.14","version":"40'56","reported_seq":176,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-19T11:30:29.041605+0000","last_change":"2024-03-19T11:30:28.331300+0000","last_active":"2024-03-19T11:30:29.041605+0000","last_peered":"2024-03-19T11:30:29.041605+0000","last_clean":"2024-03-19T11:30:29.041605+0000","last_became_active":"2024-03-19T11:30:28.038075+0000","last_became_peered":"2024-03-19T11:30:28.038075+0000","last_unstale":"2024-03-19T11:30:29.041605+0000","last_undegraded":"2024-03-19T11:30:29.041605+0000","last_fullsized":"2024-03-19T11:30:29.041605+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":5,"last_scrub":"40'56","last_scrub_stamp":"2024-03-19T11:30:28.331178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:28.331178+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:44.273814+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"2"}]},{"pgid":"4.13","version":"48'22","reported_seq":184,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-19T11:30:31.066166+0000","last_change":"2024-03-19T11:30:27.183422+0000","last_active":"2024-03-19T11:30:31.066166+0000","last_peered":"2024-03-19T11:30:31.066166+0000","last_clean":"2024-03-19T11:30:31.066166+0000","last_became_active":"2024-03-19T11:30:21.867725+0000","last_became_peered":"2024-03-19T11:30:21.867725+0000","last_unstale":"2024-03-19T11:30:31.066166+0000","last_undegraded":"2024-03-19T11:30:31.066166+0000","last_fullsized":"2024-03-19T11:30:31.066166+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":5,"last_scrub":"48'22","last_scrub_stamp":"2024-03-19T11:30:27.183294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:27.183294+0000","objects_scrubbed":2,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:44.111496+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.99227805199999997,"stat_sum":{"num_bytes":5361367,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":2261,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"2"}]},{"pgid":"4.12","version":"28'6","reported_seq":57,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-19T11:30:29.045836+0000","last_change":"2024-03-19T11:30:28.409310+0000","last_active":"2024-03-19T11:30:29.045836+0000","last_peered":"2024-03-19T11:30:29.045836+0000","last_clean":"2024-03-19T11:30:29.045836+0000","last_became_active":"2024-03-19T11:30:26.929728+0000","last_became_peered":"2024-03-19T11:30:26.929728+0000","last_unstale":"2024-03-19T11:30:29.045836+0000","last_undegraded":"2024-03-19T11:30:29.045836+0000","last_fullsized":"2024-03-19T11:30:29.045836+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":5,"last_scrub":"28'6","last_scrub_stamp":"2024-03-19T11:30:28.409040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:28.409040+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:33.792462+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"2"}]},{"pgid":"4.11","version":"47'60","reported_seq":2,"reported_epoch":52,"state":"peering","last_fresh":"2024-03-19T11:30:28.031803+0000","last_change":"2024-03-19T11:30:29.177305+0000","last_active":"2024-03-19T11:30:29.177305+0000","last_peered":"2024-03-19T11:30:28.031803+0000","last_clean":"2024-03-19T11:30:28.031803+0000","last_became_active":"2024-03-19T11:30:27.928671+0000","last_became_peered":"2024-03-19T11:30:27.928671+0000","last_unstale":"2024-03-19T11:30:28.031803+0000","last_undegraded":"2024-03-19T11:30:28.031803+0000","last_fullsized":"2024-03-19T11:30:28.031803+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0070494470000000004,"stat_sum":{"num_bytes":8646185,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":3,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":42,"num_read_kb":37168,"num_write":23,"num_write_kb":6345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":3404232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"2"}]},{"pgid":"4.10","version":"50'30","reported_seq":180,"reported_epoch":51,"state":"peering","last_fresh":"2024-03-19T11:30:28.787205+0000","last_change":"2024-03-19T11:30:29.042759+0000","last_active":"2024-03-19T11:30:29.042759+0000","last_peered":"2024-03-19T11:30:28.787205+0000","last_clean":"2024-03-19T11:30:28.787205+0000","last_became_active":"2024-03-19T11:30:27.926287+0000","last_became_peered":"2024-03-19T11:30:27.926287+0000","last_unstale":"2024-03-19T11:30:28.787205+0000","last_undegraded":"2024-03-19T11:30:28.787205+0000","last_fullsized":"2024-03-19T11:30:28.787205+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":5,"last_scrub":"50'30","last_scrub_stamp":"2024-03-19T11:30:28.787058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:28.787058+0000","objects_scrubbed":2,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.97597054500000002,"stat_sum":{"num_bytes":6369787,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,6],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"2"}]},{"pgid":"2.7","version":"0'0","reported_seq":62,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-19T11:30:29.041362+0000","last_change":"2024-03-19T11:30:03.276845+0000","last_active":"2024-03-19T11:30:29.041362+0000","last_peered":"2024-03-19T11:30:29.041362+0000","last_clean":"2024-03-19T11:30:29.041362+0000","last_became_active":"2024-03-19T11:30:03.276220+0000","last_became_peered":"2024-03-19T11:30:03.276220+0000","last_unstale":"2024-03-19T11:30:29.041362+0000","last_undegraded":"2024-03-19T11:30:29.041362+0000","last_fullsized":"2024-03-19T11:30:29.041362+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:01.829767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"51'79","reported_seq":256,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-19T11:30:29.046674+0000","last_change":"2024-03-19T11:30:28.034013+0000","last_active":"2024-03-19T11:30:29.046674+0000","last_peered":"2024-03-19T11:30:29.046674+0000","last_clean":"2024-03-19T11:30:29.046674+0000","last_became_active":"2024-03-19T11:30:27.928947+0000","last_became_peered":"2024-03-19T11:30:27.928947+0000","last_unstale":"2024-03-19T11:30:29.046674+0000","last_undegraded":"2024-03-19T11:30:29.046674+0000","last_fullsized":"2024-03-19T11:30:29.046674+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.016107153999999999,"stat_sum":{"num_bytes":9889163,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":70,"num_read_kb":76248,"num_write":33,"num_write_kb":9464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":12952921,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"2"}]},{"pgid":"2.6","version":"0'0","reported_seq":21,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-19T11:30:33.042763+0000","last_change":"2024-03-19T11:30:28.039357+0000","last_active":"2024-03-19T11:30:33.042763+0000","last_peered":"2024-03-19T11:30:33.042763+0000","last_clean":"2024-03-19T11:30:33.042763+0000","last_became_active":"2024-03-19T11:30:28.038987+0000","last_became_peered":"2024-03-19T11:30:28.038987+0000","last_unstale":"2024-03-19T11:30:33.042763+0000","last_undegraded":"2024-03-19T11:30:33.042763+0000","last_fullsized":"2024-03-19T11:30:33.042763+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:47.430206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"49'38","reported_seq":172,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-19T11:30:29.044265+0000","last_change":"2024-03-19T11:30:29.041409+0000","last_active":"2024-03-19T11:30:29.044265+0000","last_peered":"2024-03-19T11:30:29.044265+0000","last_clean":"2024-03-19T11:30:29.044265+0000","last_became_active":"2024-03-19T11:30:21.863222+0000","last_became_peered":"2024-03-19T11:30:21.863222+0000","last_unstale":"2024-03-19T11:30:29.044265+0000","last_undegraded":"2024-03-19T11:30:29.044265+0000","last_fullsized":"2024-03-19T11:30:29.044265+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.007879596,"stat_sum":{"num_bytes":7684707,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":21,"num_read_kb":20652,"num_write":17,"num_write_kb":5062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"2"}]},{"pgid":"2.5","version":"0'0","reported_seq":83,"reported_epoch":51,"state":"peering","last_fresh":"2024-03-19T11:30:28.042825+0000","last_change":"2024-03-19T11:30:29.034881+0000","last_active":"2024-03-19T11:30:29.034881+0000","last_peered":"2024-03-19T11:30:28.042825+0000","last_clean":"2024-03-19T11:30:28.042825+0000","last_became_active":"2024-03-19T11:30:28.040621+0000","last_became_peered":"2024-03-19T11:30:28.040621+0000","last_unstale":"2024-03-19T11:30:28.042825+0000","last_undegraded":"2024-03-19T11:30:28.042825+0000","last_fullsized":"2024-03-19T11:30:28.042825+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"41'19","reported_seq":177,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-19T11:30:31.066652+0000","last_change":"2024-03-19T11:30:28.180912+0000","last_active":"2024-03-19T11:30:31.066652+0000","last_peered":"2024-03-19T11:30:31.066652+0000","last_clean":"2024-03-19T11:30:31.066652+0000","last_became_active":"2024-03-19T11:30:21.867677+0000","last_became_peered":"2024-03-19T11:30:21.867677+0000","last_unstale":"2024-03-19T11:30:31.066652+0000","last_undegraded":"2024-03-19T11:30:31.066652+0000","last_fullsized":"2024-03-19T11:30:31.066652+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'19","last_scrub_stamp":"2024-03-19T11:30:28.180772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:28.180772+0000","objects_scrubbed":1,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:36.273528+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.99167969300000003,"stat_sum":{"num_bytes":2806736,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"2"}]},{"pgid":"2.4","version":"0'0","reported_seq":79,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-19T11:30:29.045861+0000","last_change":"2024-03-19T11:29:46.541233+0000","last_active":"2024-03-19T11:30:29.045861+0000","last_peered":"2024-03-19T11:30:29.045861+0000","last_clean":"2024-03-19T11:30:29.045861+0000","last_became_active":"2024-03-19T11:29:46.540939+0000","last_became_peered":"2024-03-19T11:29:46.540939+0000","last_unstale":"2024-03-19T11:30:29.045861+0000","last_undegraded":"2024-03-19T11:30:29.045861+0000","last_fullsized":"2024-03-19T11:30:29.045861+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:54.761098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"28'6","reported_seq":56,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-19T11:30:29.045885+0000","last_change":"2024-03-19T11:30:26.952750+0000","last_active":"2024-03-19T11:30:29.045885+0000","last_peered":"2024-03-19T11:30:29.045885+0000","last_clean":"2024-03-19T11:30:29.045885+0000","last_became_active":"2024-03-19T11:30:26.929632+0000","last_became_peered":"2024-03-19T11:30:26.929632+0000","last_unstale":"2024-03-19T11:30:29.045885+0000","last_undegraded":"2024-03-19T11:30:29.045885+0000","last_fullsized":"2024-03-19T11:30:29.045885+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1176473,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":13792,"num_write":2,"num_write_kb":490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1003172,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"2"}]},{"pgid":"3.3","version":"54'80","reported_seq":243,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-19T11:30:31.072982+0000","last_change":"2024-03-19T11:30:30.271391+0000","last_active":"2024-03-19T11:30:31.072982+0000","last_peered":"2024-03-19T11:30:31.072982+0000","last_clean":"2024-03-19T11:30:31.072982+0000","last_became_active":"2024-03-19T11:30:30.058963+0000","last_became_peered":"2024-03-19T11:30:30.058963+0000","last_unstale":"2024-03-19T11:30:31.072982+0000","last_undegraded":"2024-03-19T11:30:31.072982+0000","last_fullsized":"2024-03-19T11:30:31.072982+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:11.045052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41094544,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":59,"num_read_kb":1485,"num_write":88,"num_write_kb":46414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":22537669,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":78,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-19T11:30:29.045982+0000","last_change":"2024-03-19T11:29:46.321255+0000","last_active":"2024-03-19T11:30:29.045982+0000","last_peered":"2024-03-19T11:30:29.045982+0000","last_clean":"2024-03-19T11:30:29.045982+0000","last_became_active":"2024-03-19T11:29:46.320679+0000","last_became_peered":"2024-03-19T11:29:46.320679+0000","last_unstale":"2024-03-19T11:30:29.045982+0000","last_undegraded":"2024-03-19T11:30:29.045982+0000","last_fullsized":"2024-03-19T11:30:29.045982+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:14.793563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"51'76","reported_seq":271,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-19T11:30:29.041789+0000","last_change":"2024-03-19T11:30:28.127719+0000","last_active":"2024-03-19T11:30:29.041789+0000","last_peered":"2024-03-19T11:30:29.041789+0000","last_clean":"2024-03-19T11:30:29.041789+0000","last_became_active":"2024-03-19T11:30:28.037565+0000","last_became_peered":"2024-03-19T11:30:28.037565+0000","last_unstale":"2024-03-19T11:30:29.041789+0000","last_undegraded":"2024-03-19T11:30:29.041789+0000","last_fullsized":"2024-03-19T11:30:29.041789+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.012299989000000001,"stat_sum":{"num_bytes":10779048,"num_objects":6,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":78,"num_read_kb":74348,"num_write":40,"num_write_kb":10821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":15063091,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"2"}]},{"pgid":"3.0","version":"49'84","reported_seq":163,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-19T11:30:29.046649+0000","last_change":"2024-03-19T11:29:57.550320+0000","last_active":"2024-03-19T11:30:29.046649+0000","last_peered":"2024-03-19T11:30:29.046649+0000","last_clean":"2024-03-19T11:30:29.046649+0000","last_became_active":"2024-03-19T11:29:57.549928+0000","last_became_peered":"2024-03-19T11:29:57.549928+0000","last_unstale":"2024-03-19T11:30:29.046649+0000","last_undegraded":"2024-03-19T11:30:29.046649+0000","last_fullsized":"2024-03-19T11:30:29.046649+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:01.987302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29288645,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":68,"num_read_kb":2311,"num_write":89,"num_write_kb":38148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":80,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-19T11:30:29.041531+0000","last_change":"2024-03-19T11:29:46.319811+0000","last_active":"2024-03-19T11:30:29.041531+0000","last_peered":"2024-03-19T11:30:29.041531+0000","last_clean":"2024-03-19T11:30:29.041531+0000","last_became_active":"2024-03-19T11:29:46.319334+0000","last_became_peered":"2024-03-19T11:29:46.319334+0000","last_unstale":"2024-03-19T11:30:29.041531+0000","last_undegraded":"2024-03-19T11:30:29.041531+0000","last_fullsized":"2024-03-19T11:30:29.041531+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:04.320463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"51'32","reported_seq":181,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-19T11:30:29.045929+0000","last_change":"2024-03-19T11:30:28.074709+0000","last_active":"2024-03-19T11:30:29.045929+0000","last_peered":"2024-03-19T11:30:29.045929+0000","last_clean":"2024-03-19T11:30:29.045929+0000","last_became_active":"2024-03-19T11:30:28.042372+0000","last_became_peered":"2024-03-19T11:30:28.042372+0000","last_unstale":"2024-03-19T11:30:29.045929+0000","last_undegraded":"2024-03-19T11:30:29.045929+0000","last_fullsized":"2024-03-19T11:30:29.045929+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.013102536999999999,"stat_sum":{"num_bytes":1466205,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":9,"num_write_kb":2643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":7273946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"2"}]},{"pgid":"3.1","version":"47'69","reported_seq":211,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-19T11:30:31.065855+0000","last_change":"2024-03-19T11:30:27.033184+0000","last_active":"2024-03-19T11:30:31.065855+0000","last_peered":"2024-03-19T11:30:31.065855+0000","last_clean":"2024-03-19T11:30:31.065855+0000","last_became_active":"2024-03-19T11:30:26.927069+0000","last_became_peered":"2024-03-19T11:30:26.927069+0000","last_unstale":"2024-03-19T11:30:31.065855+0000","last_undegraded":"2024-03-19T11:30:31.065855+0000","last_fullsized":"2024-03-19T11:30:31.065855+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:00.211540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29002497,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":61,"num_read_kb":4830,"num_write":73,"num_write_kb":35882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":16477750,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":84,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-19T11:30:29.034283+0000","last_change":"2024-03-19T11:30:27.926306+0000","last_active":"2024-03-19T11:30:29.034283+0000","last_peered":"2024-03-19T11:30:29.034283+0000","last_clean":"2024-03-19T11:30:29.034283+0000","last_became_active":"2024-03-19T11:30:27.925990+0000","last_became_peered":"2024-03-19T11:30:27.925990+0000","last_unstale":"2024-03-19T11:30:29.034283+0000","last_undegraded":"2024-03-19T11:30:29.034283+0000","last_fullsized":"2024-03-19T11:30:29.034283+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:00.759938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"49'53","reported_seq":212,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-19T11:30:29.046445+0000","last_change":"2024-03-19T11:30:28.258334+0000","last_active":"2024-03-19T11:30:29.046445+0000","last_peered":"2024-03-19T11:30:29.046445+0000","last_clean":"2024-03-19T11:30:29.046445+0000","last_became_active":"2024-03-19T11:30:26.925686+0000","last_became_peered":"2024-03-19T11:30:26.925686+0000","last_unstale":"2024-03-19T11:30:29.046445+0000","last_undegraded":"2024-03-19T11:30:29.046445+0000","last_fullsized":"2024-03-19T11:30:29.046445+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"49'53","last_scrub_stamp":"2024-03-19T11:30:28.258196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:28.258196+0000","objects_scrubbed":3,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:48.844238+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0065094910000000001,"stat_sum":{"num_bytes":4080436,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"2"}]},{"pgid":"3.2","version":"51'66","reported_seq":148,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-19T11:30:29.041793+0000","last_change":"2024-03-19T11:29:57.549277+0000","last_active":"2024-03-19T11:30:29.041793+0000","last_peered":"2024-03-19T11:30:29.041793+0000","last_clean":"2024-03-19T11:30:29.041793+0000","last_became_active":"2024-03-19T11:29:57.548946+0000","last_became_peered":"2024-03-19T11:29:57.548946+0000","last_unstale":"2024-03-19T11:30:29.041793+0000","last_undegraded":"2024-03-19T11:30:29.041793+0000","last_fullsized":"2024-03-19T11:30:29.041793+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:08.380405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31608212,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":59,"num_read_kb":5516,"num_write":76,"num_write_kb":33521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":78,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-19T11:30:29.046256+0000","last_change":"2024-03-19T11:29:46.322491+0000","last_active":"2024-03-19T11:30:29.046256+0000","last_peered":"2024-03-19T11:30:29.046256+0000","last_clean":"2024-03-19T11:30:29.046256+0000","last_became_active":"2024-03-19T11:29:46.321760+0000","last_became_peered":"2024-03-19T11:29:46.321760+0000","last_unstale":"2024-03-19T11:30:29.046256+0000","last_undegraded":"2024-03-19T11:30:29.046256+0000","last_fullsized":"2024-03-19T11:30:29.046256+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:04.561915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":244,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-19T11:30:29.034314+0000","last_change":"2024-03-19T11:30:28.056134+0000","last_active":"2024-03-19T11:30:29.034314+0000","last_peered":"2024-03-19T11:30:29.034314+0000","last_clean":"2024-03-19T11:30:29.034314+0000","last_became_active":"2024-03-19T11:30:28.043738+0000","last_became_peered":"2024-03-19T11:30:28.043738+0000","last_unstale":"2024-03-19T11:30:29.034314+0000","last_undegraded":"2024-03-19T11:30:29.034314+0000","last_fullsized":"2024-03-19T11:30:29.034314+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:43.166618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:43.166618+0000","last_clean_scrub_stamp":"2024-03-19T11:29:43.166618+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:46.409941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"47'64","reported_seq":285,"reported_epoch":56,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:30:33.043000+0000","last_change":"2024-03-19T11:30:32.066011+0000","last_active":"2024-03-19T11:30:33.043000+0000","last_peered":"2024-03-19T11:30:33.043000+0000","last_clean":"2024-03-19T11:30:33.043000+0000","last_became_active":"2024-03-19T11:30:28.041852+0000","last_became_peered":"2024-03-19T11:30:28.041852+0000","last_unstale":"2024-03-19T11:30:33.043000+0000","last_undegraded":"2024-03-19T11:30:33.043000+0000","last_fullsized":"2024-03-19T11:30:33.043000+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'64","last_scrub_stamp":"2024-03-19T11:30:28.487136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:28.487136+0000","objects_scrubbed":3,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:46.122924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.95742021099999997,"stat_sum":{"num_bytes":4806736,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":674,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"4.8","version":"50'60","reported_seq":203,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-19T11:30:29.045926+0000","last_change":"2024-03-19T11:30:27.422552+0000","last_active":"2024-03-19T11:30:29.045926+0000","last_peered":"2024-03-19T11:30:29.045926+0000","last_clean":"2024-03-19T11:30:29.045926+0000","last_became_active":"2024-03-19T11:30:21.859467+0000","last_became_peered":"2024-03-19T11:30:21.859467+0000","last_unstale":"2024-03-19T11:30:29.045926+0000","last_undegraded":"2024-03-19T11:30:29.045926+0000","last_fullsized":"2024-03-19T11:30:29.045926+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'58","last_scrub_stamp":"2024-03-19T11:30:27.422179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:27.422179+0000","objects_scrubbed":3,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:31.926859+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.99643048700000003,"stat_sum":{"num_bytes":6952528,"num_objects":4,"num_object_clones":2,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":17,"num_read_kb":17063,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"2"}]},{"pgid":"4.9","version":"56'51","reported_seq":257,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-19T11:30:33.037416+0000","last_change":"2024-03-19T11:30:33.037713+0000","last_active":"2024-03-19T11:30:33.037416+0000","last_peered":"2024-03-19T11:30:33.037416+0000","last_clean":"2024-03-19T11:30:33.037416+0000","last_became_active":"2024-03-19T11:30:30.050283+0000","last_became_peered":"2024-03-19T11:30:30.050283+0000","last_unstale":"2024-03-19T11:30:33.037416+0000","last_undegraded":"2024-03-19T11:30:33.037416+0000","last_fullsized":"2024-03-19T11:30:33.037416+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'47","last_scrub_stamp":"2024-03-19T11:30:28.676144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:28.676144+0000","objects_scrubbed":2,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:53.055173+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0051152669999999997,"stat_sum":{"num_bytes":4517558,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4129874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"},{"start":"5","length":"1"},{"start":"a","length":"1"}]},{"pgid":"4.a","version":"56'64","reported_seq":226,"reported_epoch":56,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:30:33.042526+0000","last_change":"2024-03-19T11:30:32.067341+0000","last_active":"2024-03-19T11:30:33.042526+0000","last_peered":"2024-03-19T11:30:33.042526+0000","last_clean":"2024-03-19T11:30:33.042526+0000","last_became_active":"2024-03-19T11:29:58.553514+0000","last_became_peered":"2024-03-19T11:29:58.553514+0000","last_unstale":"2024-03-19T11:30:33.042526+0000","last_undegraded":"2024-03-19T11:30:33.042526+0000","last_fullsized":"2024-03-19T11:30:33.042526+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:09.081432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.86280128499999997,"stat_sum":{"num_bytes":16451277,"num_objects":9,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":2,"num_read":80,"num_read_kb":67307,"num_write":37,"num_write_kb":9720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":6,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"4.b","version":"47'48","reported_seq":224,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-19T11:30:31.066366+0000","last_change":"2024-03-19T11:30:27.035208+0000","last_active":"2024-03-19T11:30:31.066366+0000","last_peered":"2024-03-19T11:30:31.066366+0000","last_clean":"2024-03-19T11:30:31.066366+0000","last_became_active":"2024-03-19T11:29:58.555569+0000","last_became_peered":"2024-03-19T11:29:58.555569+0000","last_unstale":"2024-03-19T11:30:31.066366+0000","last_undegraded":"2024-03-19T11:30:31.066366+0000","last_fullsized":"2024-03-19T11:30:31.066366+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:01.446572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.114579942,"stat_sum":{"num_bytes":15607554,"num_objects":6,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":140,"num_read_kb":142315,"num_write":36,"num_write_kb":10314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"2"}]},{"pgid":"4.c","version":"56'29","reported_seq":186,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-19T11:30:33.043983+0000","last_change":"2024-03-19T11:30:33.044257+0000","last_active":"2024-03-19T11:30:33.043983+0000","last_peered":"2024-03-19T11:30:33.043983+0000","last_clean":"2024-03-19T11:30:33.043983+0000","last_became_active":"2024-03-19T11:30:28.042755+0000","last_became_peered":"2024-03-19T11:30:28.042755+0000","last_unstale":"2024-03-19T11:30:33.043983+0000","last_undegraded":"2024-03-19T11:30:33.043983+0000","last_fullsized":"2024-03-19T11:30:33.043983+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:20.491067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011420695,"stat_sum":{"num_bytes":4506149,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":22,"num_read_kb":27635,"num_write":16,"num_write_kb":3672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":3206545,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"},{"start":"5","length":"1"},{"start":"a","length":"1"}]},{"pgid":"4.d","version":"51'47","reported_seq":191,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-19T11:30:29.041978+0000","last_change":"2024-03-19T11:30:26.986995+0000","last_active":"2024-03-19T11:30:29.041978+0000","last_peered":"2024-03-19T11:30:29.041978+0000","last_clean":"2024-03-19T11:30:29.041978+0000","last_became_active":"2024-03-19T11:30:26.923905+0000","last_became_peered":"2024-03-19T11:30:26.923905+0000","last_unstale":"2024-03-19T11:30:29.041978+0000","last_undegraded":"2024-03-19T11:30:29.041978+0000","last_fullsized":"2024-03-19T11:30:29.041978+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:21.512422+0000","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0.023880948999999999,"stat_sum":{"num_bytes":11405351,"num_objects":4,"num_object_clones":1,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":32,"num_read_kb":24377,"num_write":32,"num_write_kb":9391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":5544901,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"2"}]},{"pgid":"4.e","version":"56'51","reported_seq":200,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-19T11:30:33.042402+0000","last_change":"2024-03-19T11:30:33.042648+0000","last_active":"2024-03-19T11:30:33.042402+0000","last_peered":"2024-03-19T11:30:33.042402+0000","last_clean":"2024-03-19T11:30:33.042402+0000","last_became_active":"2024-03-19T11:29:58.554727+0000","last_became_peered":"2024-03-19T11:29:58.554727+0000","last_unstale":"2024-03-19T11:30:33.042402+0000","last_undegraded":"2024-03-19T11:30:33.042402+0000","last_fullsized":"2024-03-19T11:30:33.042402+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:03.426039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0098393890000000005,"stat_sum":{"num_bytes":10199388,"num_objects":7,"num_object_clones":4,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":2,"num_read":44,"num_read_kb":44338,"num_write":25,"num_write_kb":6191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"},{"start":"5","length":"1"},{"start":"a","length":"1"}]},{"pgid":"4.f","version":"0'0","reported_seq":116,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-19T11:30:29.046499+0000","last_change":"2024-03-19T11:30:24.896962+0000","last_active":"2024-03-19T11:30:29.046499+0000","last_peered":"2024-03-19T11:30:29.046499+0000","last_clean":"2024-03-19T11:30:29.046499+0000","last_became_active":"2024-03-19T11:29:58.551505+0000","last_became_peered":"2024-03-19T11:29:58.551505+0000","last_unstale":"2024-03-19T11:30:29.046499+0000","last_undegraded":"2024-03-19T11:30:29.046499+0000","last_fullsized":"2024-03-19T11:30:29.046499+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:05.579017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00139266,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"2"}]}],"pool_stats":[{"poolid":4,"num_pg":26,"stat_sum":{"num_bytes":184953941,"num_objects":87,"num_object_clones":42,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":3,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84,"num_whiteouts":7,"num_read":645,"num_read_kb":605409,"num_write":355,"num_write_kb":98289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":71262102,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":61,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":251027456,"data_stored":299227522,"data_compressed":1189206,"data_compressed_allocated":1032192,"data_compressed_original":3145728,"omap_allocated":0,"internal_metadata":0},"log_size":1183,"ondisk_log_size":1183,"up":52,"acting":52,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":130993898,"num_objects":106,"num_object_clones":0,"num_object_copies":212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":106,"num_whiteouts":0,"num_read":247,"num_read_kb":14142,"num_write":326,"num_write_kb":153965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":39015419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":150835200,"data_stored":164384205,"data_compressed":11252494,"data_compressed_allocated":12095488,"data_compressed_original":25948160,"omap_allocated":0,"internal_metadata":0},"log_size":299,"ondisk_log_size":299,"up":8,"acting":8,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1586784,"data_compressed":19012,"data_compressed_allocated":98304,"data_compressed_original":1548288,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542157,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65380,"kb_used_data":34192,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94306460,"statfs":{"total":96636764160,"available":96569815040,"internally_reserved":0,"allocated":35012608,"data_stored":41666301,"data_compressed":117244,"data_compressed_allocated":151552,"data_compressed_original":905216,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542157,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76488,"kb_used_data":49456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94295352,"statfs":{"total":96636764160,"available":96558440448,"internally_reserved":0,"allocated":50642944,"data_stored":59830953,"data_compressed":1369376,"data_compressed_allocated":1462272,"data_compressed_original":3031040,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542157,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79152,"kb_used_data":52120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94292688,"statfs":{"total":96636764160,"available":96555712512,"internally_reserved":0,"allocated":53370880,"data_stored":57890208,"data_compressed":2537556,"data_compressed_allocated":2719744,"data_compressed_original":6045696,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542156,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70780,"kb_used_data":41524,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94301060,"statfs":{"total":96636764160,"available":96564285440,"internally_reserved":0,"allocated":42520576,"data_stored":47440616,"data_compressed":1261851,"data_compressed_allocated":1359872,"data_compressed_original":3080192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542156,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84668,"kb_used_data":53484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94287172,"statfs":{"total":96636764160,"available":96550064128,"internally_reserved":0,"allocated":54767616,"data_stored":62945614,"data_compressed":1119420,"data_compressed_allocated":1208320,"data_compressed_original":2818048,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574861,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95692,"kb_used_data":68672,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94276148,"statfs":{"total":96636764160,"available":96538775552,"internally_reserved":0,"allocated":70320128,"data_stored":80408937,"data_compressed":1599621,"data_compressed_allocated":1748992,"data_compressed_original":4182016,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574861,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48992,"kb_used_data":17812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322848,"statfs":{"total":96636764160,"available":96586596352,"internally_reserved":0,"allocated":18239488,"data_stored":26361759,"data_compressed":224080,"data_compressed_allocated":118784,"data_compressed_original":524288,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574861,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77184,"kb_used_data":50160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94294656,"statfs":{"total":96636764160,"available":96557727744,"internally_reserved":0,"allocated":51363840,"data_stored":54661667,"data_compressed":2618817,"data_compressed_allocated":2813952,"data_compressed_original":6082560,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6064,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6884,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6064,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36450304,"data_stored":39640059,"data_compressed":2618817,"data_compressed_allocated":2813952,"data_compressed_original":6082560,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14405632,"data_stored":15728702,"data_compressed":1002508,"data_compressed_allocated":1081344,"data_compressed_original":2424832,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14405632,"data_stored":15728702,"data_compressed":1002508,"data_compressed_allocated":1081344,"data_compressed_original":2424832,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":29884416,"data_stored":32927489,"data_compressed":2321958,"data_compressed_allocated":2490368,"data_compressed_original":5586944,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":19230720,"data_stored":20747205,"data_compressed":1455770,"data_compressed_allocated":1568768,"data_compressed_original":3133440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":14946304,"data_stored":16449739,"data_compressed":1277037,"data_compressed_allocated":1368064,"data_compressed_original":2899968,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":21512192,"data_stored":23162309,"data_compressed":1573896,"data_compressed_allocated":1691648,"data_compressed_original":3395584,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":14319616,"data_stored":14731212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":45002752,"data_stored":53956612,"data_compressed":267825,"data_compressed_allocated":167936,"data_compressed_original":720896,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":30887936,"data_stored":35959980,"data_compressed":259343,"data_compressed_allocated":278528,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24616960,"data_stored":27571298,"data_compressed":215598,"data_compressed_allocated":229376,"data_compressed_original":458752,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16982016,"data_stored":24914062,"data_compressed":224080,"data_compressed_allocated":118784,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":37302272,"data_stored":45959616,"data_compressed":92339,"data_compressed_allocated":94208,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":47456256,"data_stored":55258697,"data_compressed":18841,"data_compressed_allocated":24576,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":34459648,"data_stored":40876045,"data_compressed":111180,"data_compressed_allocated":118784,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T11:30:34.393 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2024-03-19T11:30:34.393 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 36 2024-03-19T11:30:34.421 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:34.431+0000 7f70c4fdb640 -1 osd.0 56 handle_osd_map injecting map crc failure 2024-03-19T11:30:34.425 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:34.433+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:34.501 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:34.502 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:34.517 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-19T11:30:34.567 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:34.579+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:34.628 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:34.637+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:34.637 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:34.637 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:34.651 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-19T11:30:34.729 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:34.737+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:34.740 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-19T11:30:34.830 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:34.837+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:34.831 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:34.831 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:34.846 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-19T11:30:34.880 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:30:34.881 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:30:34.897 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:30:34.897 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:30:34.916 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:30:34.916 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:30:34.931 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:34.941+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:34.933 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:30:34.933 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:30:34.954 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:30:34.954 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:30:34.966 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:34.966 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:30:34.966 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:30:34.967 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:30:34.967 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:34.975 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:30:34.975 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:30:34.980 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-19T11:30:34.996 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:30:34.996 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:30:35.018 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:30:35.018 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:30:35.032 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:35.043+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:35.050 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-03-19T11:30:35.092 INFO:teuthology.orchestra.run.smithi038.stderr:set pool 4 pg_num to 36 2024-03-19T11:30:35.093 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:35.103+0000 7fa5e08db640 -1 osd.3 57 handle_osd_map injecting map crc failure 2024-03-19T11:30:35.098 INFO:tasks.rados.rados.0.smithi038.stdout:896: done (0 left) 2024-03-19T11:30:35.098 INFO:tasks.rados.rados.0.smithi038.stdout:897: read oid 4 snap -1 2024-03-19T11:30:35.098 INFO:tasks.rados.rados.0.smithi038.stdout:897: expect (ObjNum 3 snap 0 seq_num 3) 2024-03-19T11:30:35.098 INFO:tasks.rados.rados.0.smithi038.stdout:898: read oid 24 snap -1 2024-03-19T11:30:35.098 INFO:tasks.rados.rados.0.smithi038.stdout:898: expect (ObjNum 23 snap 0 seq_num 23) 2024-03-19T11:30:35.098 INFO:tasks.rados.rados.0.smithi038.stdout:899: set_chunk oid 10 target oid 2024-03-19T11:30:35.120 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:35.120 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:30:35.120 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:30:35.121 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:30:35.121 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.121 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:55 3.2 3:525f009b:::a57e3dbafedf1ec1a7132467f167cfc4a4343cb541b36fc481376f0086aece71:head [read 0~121882 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:35.121 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.761904+0000", 2024-03-19T11:30:35.121 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 22.368025240000001, 2024-03-19T11:30:35.121 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010829357, 2024-03-19T11:30:35.121 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:35.121 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:35.121 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:35.121 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.121 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:35.122 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761904+0000", 2024-03-19T11:30:35.122 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:35.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.122 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:35.122 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761904+0000", 2024-03-19T11:30:35.122 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:35.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.122 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:35.122 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761905+0000", 2024-03-19T11:30:35.123 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4900000000000005e-07 2024-03-19T11:30:35.123 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.123 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.123 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:35.123 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761917+0000", 2024-03-19T11:30:35.123 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1806e-05 2024-03-19T11:30:35.123 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.123 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.123 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:35.123 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761934+0000", 2024-03-19T11:30:35.123 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6509000000000001e-05 2024-03-19T11:30:35.123 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.124 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.124 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:35.124 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761941+0000", 2024-03-19T11:30:35.124 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8110000000000003e-06 2024-03-19T11:30:35.124 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.124 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.124 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:35.124 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762115+0000", 2024-03-19T11:30:35.124 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017358200000000001 2024-03-19T11:30:35.124 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.124 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.125 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:35.125 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762120+0000", 2024-03-19T11:30:35.125 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3630000000000002e-06 2024-03-19T11:30:35.125 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.125 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.125 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:35.125 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.765567+0000", 2024-03-19T11:30:35.125 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034469990000000001 2024-03-19T11:30:35.125 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.125 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.125 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:35.125 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.765665+0000", 2024-03-19T11:30:35.126 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7476e-05 2024-03-19T11:30:35.126 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.126 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.126 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:35.126 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.772734+0000", 2024-03-19T11:30:35.126 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0070690620000000001 2024-03-19T11:30:35.126 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:35.126 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:35.127 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:35.127 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.127 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.127 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:56 3.2 3:5aa75d1e:::b3c26d797de4ffcce4e862009e6b8eb8576a53cb86ea102ce51145c0038aeb97:head [read 0~524288 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:35.127 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.761958+0000", 2024-03-19T11:30:35.127 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 22.367971838999999, 2024-03-19T11:30:35.127 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012078836000000001, 2024-03-19T11:30:35.127 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:35.127 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:35.127 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:35.127 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.127 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:35.128 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761958+0000", 2024-03-19T11:30:35.128 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:35.128 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.128 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.128 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:35.128 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761958+0000", 2024-03-19T11:30:35.128 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:35.128 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.128 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.128 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:35.128 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761958+0000", 2024-03-19T11:30:35.128 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9900000000000001e-07 2024-03-19T11:30:35.129 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.129 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.129 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:35.129 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761961+0000", 2024-03-19T11:30:35.129 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9280000000000002e-06 2024-03-19T11:30:35.129 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.129 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.129 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:35.129 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761969+0000", 2024-03-19T11:30:35.129 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1440000000000003e-06 2024-03-19T11:30:35.129 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.129 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.129 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:35.130 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761976+0000", 2024-03-19T11:30:35.130 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1020000000000002e-06 2024-03-19T11:30:35.130 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.130 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.130 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:35.130 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762144+0000", 2024-03-19T11:30:35.130 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016818200000000001 2024-03-19T11:30:35.130 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.130 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.130 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:35.130 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762148+0000", 2024-03-19T11:30:35.131 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0969999999999999e-06 2024-03-19T11:30:35.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.131 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.131 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:35.131 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.772788+0000", 2024-03-19T11:30:35.131 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010640124000000001 2024-03-19T11:30:35.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.131 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.131 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:35.131 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.772898+0000", 2024-03-19T11:30:35.131 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000110512 2024-03-19T11:30:35.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.131 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.132 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:35.132 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.774037+0000", 2024-03-19T11:30:35.132 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011382479999999999 2024-03-19T11:30:35.132 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:35.132 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:35.132 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:35.132 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.132 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.132 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:57 3.2 3:6d82e5d5:::455c613336f373ae48b33af990f91f317265a0c5a920cc1b46533fdb41d8319c:head [read 0~380357 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:35.132 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.761987+0000", 2024-03-19T11:30:35.133 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 22.367942877000001, 2024-03-19T11:30:35.133 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012691991999999999, 2024-03-19T11:30:35.133 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:35.133 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:35.133 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:35.133 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.133 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:35.133 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761987+0000", 2024-03-19T11:30:35.133 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:35.133 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.133 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.133 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:35.134 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761987+0000", 2024-03-19T11:30:35.134 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:35.134 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.134 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.134 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:35.134 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761987+0000", 2024-03-19T11:30:35.134 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4099999999999999e-07 2024-03-19T11:30:35.134 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.134 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.134 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:35.134 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761990+0000", 2024-03-19T11:30:35.134 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3829999999999999e-06 2024-03-19T11:30:35.135 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.135 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.135 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:35.135 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762000+0000", 2024-03-19T11:30:35.135 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0229e-05 2024-03-19T11:30:35.135 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.135 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.135 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:35.135 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762006+0000", 2024-03-19T11:30:35.135 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7069999999999997e-06 2024-03-19T11:30:35.135 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.136 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.136 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:35.136 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762171+0000", 2024-03-19T11:30:35.136 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000165341 2024-03-19T11:30:35.136 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.136 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.136 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:35.136 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762176+0000", 2024-03-19T11:30:35.136 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6879999999999998e-06 2024-03-19T11:30:35.136 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.136 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.137 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:35.137 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.774081+0000", 2024-03-19T11:30:35.137 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011905094999999999 2024-03-19T11:30:35.137 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.137 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.137 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:35.137 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.774169+0000", 2024-03-19T11:30:35.137 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8089000000000003e-05 2024-03-19T11:30:35.137 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.137 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.138 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:35.138 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.774679+0000", 2024-03-19T11:30:35.138 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00051001900000000001 2024-03-19T11:30:35.138 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:35.138 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:35.138 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:35.138 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.138 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.138 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:58 3.2 3:5f1d333e:::bfcdf551f6f2543327e9c601a4377e9a5ef396f5b97ae8331892d1bcf6b55a9d:head [read 0~161292 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:35.138 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.762024+0000", 2024-03-19T11:30:35.138 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 22.367905927999999, 2024-03-19T11:30:35.138 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013495629, 2024-03-19T11:30:35.138 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:35.139 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:35.139 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:35.139 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.139 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:35.139 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762024+0000", 2024-03-19T11:30:35.139 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:35.139 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.139 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.139 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:35.139 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762024+0000", 2024-03-19T11:30:35.140 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:35.140 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.140 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.140 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:35.140 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762024+0000", 2024-03-19T11:30:35.140 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9999999999999997e-07 2024-03-19T11:30:35.140 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.140 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.140 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:35.140 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762027+0000", 2024-03-19T11:30:35.140 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0410000000000002e-06 2024-03-19T11:30:35.140 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.140 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.141 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:35.141 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762037+0000", 2024-03-19T11:30:35.141 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1020000000000003e-06 2024-03-19T11:30:35.141 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.142 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.142 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:35.142 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762043+0000", 2024-03-19T11:30:35.142 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4409999999999998e-06 2024-03-19T11:30:35.142 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.142 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.142 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:35.142 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762198+0000", 2024-03-19T11:30:35.142 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015540000000000001 2024-03-19T11:30:35.142 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.142 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.142 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:35.142 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762203+0000", 2024-03-19T11:30:35.142 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5109999999999997e-06 2024-03-19T11:30:35.143 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.143 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.143 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:35.143 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.774710+0000", 2024-03-19T11:30:35.143 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012507452 2024-03-19T11:30:35.143 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.143 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.143 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:35.143 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.774754+0000", 2024-03-19T11:30:35.143 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3869999999999998e-05 2024-03-19T11:30:35.143 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.144 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.144 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:35.144 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.775519+0000", 2024-03-19T11:30:35.144 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00076521200000000003 2024-03-19T11:30:35.144 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:35.144 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:35.144 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:35.144 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.144 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.144 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:59 3.2 3:78b4b14e:::e1ef0c0c594c36c534f3829406e07ffb60d57b70360f62977e0d545d8ca2bf18:head [read 0~149977 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:35.144 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.762054+0000", 2024-03-19T11:30:35.145 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 22.367875497, 2024-03-19T11:30:35.145 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015354483, 2024-03-19T11:30:35.145 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:35.145 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:35.145 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:35.145 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.145 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:35.145 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762054+0000", 2024-03-19T11:30:35.145 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:35.145 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.146 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.146 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:35.146 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762054+0000", 2024-03-19T11:30:35.146 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:35.146 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.146 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.146 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:35.146 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762055+0000", 2024-03-19T11:30:35.146 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.32e-07 2024-03-19T11:30:35.146 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.147 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.148 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:35.148 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762058+0000", 2024-03-19T11:30:35.148 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9189999999999999e-06 2024-03-19T11:30:35.148 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.148 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.148 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:35.148 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762065+0000", 2024-03-19T11:30:35.148 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2409999999999998e-06 2024-03-19T11:30:35.148 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.148 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.148 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:35.148 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762070+0000", 2024-03-19T11:30:35.148 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6910000000000002e-06 2024-03-19T11:30:35.148 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.149 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.149 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:35.149 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762227+0000", 2024-03-19T11:30:35.149 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015606999999999999 2024-03-19T11:30:35.149 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.149 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.149 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:35.149 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762232+0000", 2024-03-19T11:30:35.149 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3029999999999999e-06 2024-03-19T11:30:35.149 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.149 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.149 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:35.149 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.776984+0000", 2024-03-19T11:30:35.149 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014752487999999999 2024-03-19T11:30:35.149 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.150 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.150 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:35.150 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.777030+0000", 2024-03-19T11:30:35.150 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5359999999999999e-05 2024-03-19T11:30:35.150 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.150 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.150 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:35.150 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.777409+0000", 2024-03-19T11:30:35.150 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037897899999999999 2024-03-19T11:30:35.150 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:35.150 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:35.150 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:35.150 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.150 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.151 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1067 4.0 4:01018511:::smithi03829856-19:head [write 477332~766101] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:35.151 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.950506+0000", 2024-03-19T11:30:35.151 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 21.179423785000001, 2024-03-19T11:30:35.151 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016279426, 2024-03-19T11:30:35.151 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:35.151 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:35.151 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:35.151 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:35.151 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:35.151 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1067 2024-03-19T11:30:35.151 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.151 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:35.151 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.151 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:35.151 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950506+0000", 2024-03-19T11:30:35.151 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:35.152 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.152 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.152 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:35.152 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950506+0000", 2024-03-19T11:30:35.152 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:35.152 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.152 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.152 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:35.152 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950509+0000", 2024-03-19T11:30:35.152 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.021e-06 2024-03-19T11:30:35.152 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.152 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.152 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:35.152 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950892+0000", 2024-03-19T11:30:35.152 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00038263300000000001 2024-03-19T11:30:35.153 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.153 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.153 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:35.153 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950912+0000", 2024-03-19T11:30:35.153 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0539000000000001e-05 2024-03-19T11:30:35.153 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.153 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.153 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:35.153 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950922+0000", 2024-03-19T11:30:35.153 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.577e-06 2024-03-19T11:30:35.153 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.153 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.153 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:35.153 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950982+0000", 2024-03-19T11:30:35.153 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0760000000000001e-05 2024-03-19T11:30:35.154 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.154 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.154 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:35.154 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.951058+0000", 2024-03-19T11:30:35.154 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5078000000000002e-05 2024-03-19T11:30:35.154 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.154 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.154 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:35.154 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.951173+0000", 2024-03-19T11:30:35.154 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011516200000000001 2024-03-19T11:30:35.154 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.154 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.154 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:35.154 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.955064+0000", 2024-03-19T11:30:35.155 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003890896 2024-03-19T11:30:35.155 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.155 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.155 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:35.155 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966670+0000", 2024-03-19T11:30:35.155 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011606795 2024-03-19T11:30:35.155 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.155 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.155 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:35.155 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966731+0000", 2024-03-19T11:30:35.155 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0528000000000001e-05 2024-03-19T11:30:35.155 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.155 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.155 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:35.155 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966785+0000", 2024-03-19T11:30:35.156 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4437000000000001e-05 2024-03-19T11:30:35.156 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:35.156 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:35.156 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:35.156 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.156 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.156 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1068 4.0 4:01018511:::smithi03829856-19:head [write 1681102~474010] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:35.156 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.956456+0000", 2024-03-19T11:30:35.156 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 21.173473220000002, 2024-03-19T11:30:35.156 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010432765, 2024-03-19T11:30:35.156 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:35.156 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:35.156 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:35.156 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:35.157 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:35.157 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1068 2024-03-19T11:30:35.157 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.157 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:35.157 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.157 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:35.157 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956456+0000", 2024-03-19T11:30:35.157 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:35.157 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.157 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.157 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:35.157 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956456+0000", 2024-03-19T11:30:35.157 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:35.157 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.157 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.157 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:35.158 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956458+0000", 2024-03-19T11:30:35.158 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.742e-06 2024-03-19T11:30:35.158 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.158 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.158 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:35.158 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956655+0000", 2024-03-19T11:30:35.158 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019682200000000001 2024-03-19T11:30:35.158 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.158 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.158 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:35.158 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956669+0000", 2024-03-19T11:30:35.158 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3978000000000001e-05 2024-03-19T11:30:35.158 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.158 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.159 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:35.159 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956678+0000", 2024-03-19T11:30:35.159 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5029999999999997e-06 2024-03-19T11:30:35.159 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.159 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.159 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:35.159 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956729+0000", 2024-03-19T11:30:35.159 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1765999999999999e-05 2024-03-19T11:30:35.159 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.159 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.159 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:35.159 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956795+0000", 2024-03-19T11:30:35.159 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5701e-05 2024-03-19T11:30:35.159 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.159 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.160 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:35.160 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956886+0000", 2024-03-19T11:30:35.160 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1226999999999999e-05 2024-03-19T11:30:35.160 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.160 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.160 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:35.160 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.958609+0000", 2024-03-19T11:30:35.160 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017227049999999999 2024-03-19T11:30:35.160 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.160 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.160 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:35.160 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966837+0000", 2024-03-19T11:30:35.160 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0082276439999999992 2024-03-19T11:30:35.160 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.160 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.161 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:35.161 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966866+0000", 2024-03-19T11:30:35.161 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9323999999999999e-05 2024-03-19T11:30:35.161 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.161 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.161 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:35.161 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966889+0000", 2024-03-19T11:30:35.161 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3353000000000002e-05 2024-03-19T11:30:35.161 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:35.161 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:35.161 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:35.161 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.161 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.161 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.0.13:50 3.2 3:5595a68d:::07854d2fef297a06ba81685e660c332de36d5d18d546927d30daad6d7fda1541:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:35.161 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.166341+0000", 2024-03-19T11:30:35.162 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 16.963588353999999, 2024-03-19T11:30:35.162 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010067153000000001, 2024-03-19T11:30:35.162 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:35.162 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:35.162 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:35.162 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.162 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:35.162 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166341+0000", 2024-03-19T11:30:35.162 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:35.162 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.162 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.162 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:35.162 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166341+0000", 2024-03-19T11:30:35.162 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:35.163 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.163 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.163 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:35.163 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166344+0000", 2024-03-19T11:30:35.163 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1370000000000002e-06 2024-03-19T11:30:35.163 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.163 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.163 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:35.163 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166828+0000", 2024-03-19T11:30:35.163 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00048343600000000001 2024-03-19T11:30:35.163 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.163 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.163 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:35.163 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166857+0000", 2024-03-19T11:30:35.163 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8880000000000001e-05 2024-03-19T11:30:35.164 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.164 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.164 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:35.164 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166871+0000", 2024-03-19T11:30:35.164 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4384e-05 2024-03-19T11:30:35.164 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.164 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.164 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:35.164 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167222+0000", 2024-03-19T11:30:35.164 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00035059000000000002 2024-03-19T11:30:35.164 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.164 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.164 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:35.164 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167334+0000", 2024-03-19T11:30:35.164 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000112663 2024-03-19T11:30:35.164 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.165 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.165 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:35.165 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167533+0000", 2024-03-19T11:30:35.165 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019901399999999999 2024-03-19T11:30:35.165 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.165 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.165 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:35.165 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176237+0000", 2024-03-19T11:30:35.165 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0087039890000000005 2024-03-19T11:30:35.165 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.165 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.165 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:35.165 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176260+0000", 2024-03-19T11:30:35.166 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2889000000000001e-05 2024-03-19T11:30:35.166 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.166 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.166 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:35.166 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176343+0000", 2024-03-19T11:30:35.166 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2651000000000005e-05 2024-03-19T11:30:35.166 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.166 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.166 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:35.166 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176408+0000", 2024-03-19T11:30:35.166 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5519999999999996e-05 2024-03-19T11:30:35.166 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:35.166 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:35.167 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:35.167 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.167 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.167 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.0.13:51 3.2 3:587bed34:::13074eb9c7a477811d595d489af192576515e7c716c0b501b140fcc4687b768b:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:35.167 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.166912+0000", 2024-03-19T11:30:35.167 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 16.963017551, 2024-03-19T11:30:35.167 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0097465639999999992, 2024-03-19T11:30:35.167 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:35.167 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:35.167 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:35.167 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.167 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:35.167 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166912+0000", 2024-03-19T11:30:35.167 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:35.167 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.168 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.168 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:35.168 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166912+0000", 2024-03-19T11:30:35.168 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:35.168 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.168 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.168 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:35.168 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166914+0000", 2024-03-19T11:30:35.168 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.468e-06 2024-03-19T11:30:35.168 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.168 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.168 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:35.168 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167000+0000", 2024-03-19T11:30:35.168 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6311999999999996e-05 2024-03-19T11:30:35.168 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.169 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.169 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:35.169 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167011+0000", 2024-03-19T11:30:35.169 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.153e-05 2024-03-19T11:30:35.169 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.169 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.169 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:35.169 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167019+0000", 2024-03-19T11:30:35.169 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7970000000000001e-06 2024-03-19T11:30:35.169 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.169 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.169 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:35.169 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167864+0000", 2024-03-19T11:30:35.169 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00084482600000000002 2024-03-19T11:30:35.170 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.170 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.170 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:35.170 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.169110+0000", 2024-03-19T11:30:35.170 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0012456399999999999 2024-03-19T11:30:35.170 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.170 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.170 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:35.170 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.169467+0000", 2024-03-19T11:30:35.170 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00035692100000000001 2024-03-19T11:30:35.170 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.170 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.170 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:35.170 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176426+0000", 2024-03-19T11:30:35.170 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0069589600000000001 2024-03-19T11:30:35.171 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.171 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.171 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:35.171 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176524+0000", 2024-03-19T11:30:35.171 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8183000000000005e-05 2024-03-19T11:30:35.171 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.171 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.171 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:35.171 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176603+0000", 2024-03-19T11:30:35.171 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9035999999999993e-05 2024-03-19T11:30:35.171 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.171 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.171 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:35.171 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176659+0000", 2024-03-19T11:30:35.172 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5890999999999997e-05 2024-03-19T11:30:35.172 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:35.172 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:35.172 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:35.172 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.172 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.172 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.0.13:52 3.2 3:46a26328:::b2b262672c1f15ae09b098a7a21c74662e4816397daed8ca08fb83d0cd3a0cac:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:35.172 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.167033+0000", 2024-03-19T11:30:35.172 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 16.962896742000002, 2024-03-19T11:30:35.172 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010113297, 2024-03-19T11:30:35.172 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:35.172 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:35.172 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:35.172 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.172 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:35.172 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167033+0000", 2024-03-19T11:30:35.173 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:35.173 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.173 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.173 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:35.173 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167033+0000", 2024-03-19T11:30:35.173 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:35.173 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.173 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.173 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:35.173 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167034+0000", 2024-03-19T11:30:35.173 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6599999999999996e-07 2024-03-19T11:30:35.173 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.173 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.173 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:35.173 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167092+0000", 2024-03-19T11:30:35.174 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7989999999999999e-05 2024-03-19T11:30:35.174 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.174 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.174 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:35.174 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167100+0000", 2024-03-19T11:30:35.174 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9670000000000001e-06 2024-03-19T11:30:35.174 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.174 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.174 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:35.174 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167108+0000", 2024-03-19T11:30:35.174 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3559999999999993e-06 2024-03-19T11:30:35.174 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.174 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.174 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:35.175 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.170462+0000", 2024-03-19T11:30:35.175 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033540229999999998 2024-03-19T11:30:35.175 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.175 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.175 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:35.175 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.171193+0000", 2024-03-19T11:30:35.175 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00073086899999999998 2024-03-19T11:30:35.175 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.175 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.175 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:35.175 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.171558+0000", 2024-03-19T11:30:35.175 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00036518899999999998 2024-03-19T11:30:35.175 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.175 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.175 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:35.176 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176440+0000", 2024-03-19T11:30:35.176 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0048817260000000003 2024-03-19T11:30:35.176 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.176 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.176 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:35.176 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177053+0000", 2024-03-19T11:30:35.176 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00061337100000000003 2024-03-19T11:30:35.176 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.176 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.176 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:35.176 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177096+0000", 2024-03-19T11:30:35.176 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2846999999999998e-05 2024-03-19T11:30:35.176 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.176 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.177 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:35.177 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177146+0000", 2024-03-19T11:30:35.177 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0293000000000001e-05 2024-03-19T11:30:35.177 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:35.177 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:35.177 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:35.177 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.177 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.177 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.0.13:54 3.2 3:7b614c59:::a7a06fca755baa7858207cb4b067fffe8c25e8183433b950d6879f6969d41b0b:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:35.177 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.167117+0000", 2024-03-19T11:30:35.177 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 16.962812234000001, 2024-03-19T11:30:35.177 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010796765999999999, 2024-03-19T11:30:35.177 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:35.177 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:35.177 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:35.178 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.178 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:35.178 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167117+0000", 2024-03-19T11:30:35.178 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:35.178 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.178 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.178 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:35.178 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167117+0000", 2024-03-19T11:30:35.178 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:35.178 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.178 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.178 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:35.178 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167118+0000", 2024-03-19T11:30:35.178 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2600000000000002e-07 2024-03-19T11:30:35.179 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.179 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.179 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:35.179 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167150+0000", 2024-03-19T11:30:35.179 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1576999999999999e-05 2024-03-19T11:30:35.179 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.179 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.179 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:35.179 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167157+0000", 2024-03-19T11:30:35.179 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2779999999999996e-06 2024-03-19T11:30:35.179 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.179 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.179 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:35.179 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167163+0000", 2024-03-19T11:30:35.179 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0120000000000002e-06 2024-03-19T11:30:35.180 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.180 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.180 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:35.180 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173324+0000", 2024-03-19T11:30:35.180 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0061614149999999999 2024-03-19T11:30:35.180 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.180 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.180 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:35.180 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173489+0000", 2024-03-19T11:30:35.180 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016438999999999999 2024-03-19T11:30:35.180 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.180 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.180 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:35.180 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173774+0000", 2024-03-19T11:30:35.181 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00028548000000000001 2024-03-19T11:30:35.181 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.181 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.181 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:35.181 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177668+0000", 2024-03-19T11:30:35.181 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038942450000000002 2024-03-19T11:30:35.181 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.181 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.181 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:35.181 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177792+0000", 2024-03-19T11:30:35.181 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012398199999999999 2024-03-19T11:30:35.181 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.181 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.181 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:35.181 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177861+0000", 2024-03-19T11:30:35.181 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8633999999999994e-05 2024-03-19T11:30:35.182 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.182 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.182 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:35.182 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177914+0000", 2024-03-19T11:30:35.182 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3226999999999997e-05 2024-03-19T11:30:35.182 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:35.182 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:35.182 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:35.182 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.182 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.182 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1230 4.0 4:0dc3b91e:::smithi03829856-28:head [write 662209~626546] snapc 9=[9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:35.182 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.170758+0000", 2024-03-19T11:30:35.182 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 16.959171520000002, 2024-03-19T11:30:35.182 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011180066000000001, 2024-03-19T11:30:35.183 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:35.183 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:35.183 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:35.183 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:35.183 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:35.183 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1230 2024-03-19T11:30:35.183 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.183 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:35.183 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.183 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:35.183 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.170758+0000", 2024-03-19T11:30:35.183 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:35.183 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.183 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.183 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:35.184 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.170758+0000", 2024-03-19T11:30:35.184 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:35.184 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.184 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.184 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:35.184 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.170762+0000", 2024-03-19T11:30:35.184 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4960000000000001e-06 2024-03-19T11:30:35.184 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.184 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.184 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:35.184 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.171449+0000", 2024-03-19T11:30:35.184 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00068700499999999997 2024-03-19T11:30:35.184 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.184 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.185 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:35.185 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.171467+0000", 2024-03-19T11:30:35.185 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8389e-05 2024-03-19T11:30:35.185 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.185 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.185 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:35.185 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.171482+0000", 2024-03-19T11:30:35.185 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4619e-05 2024-03-19T11:30:35.185 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.185 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.185 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:35.185 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.171566+0000", 2024-03-19T11:30:35.185 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4788000000000002e-05 2024-03-19T11:30:35.185 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.185 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.185 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:35.186 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.171685+0000", 2024-03-19T11:30:35.186 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011861700000000001 2024-03-19T11:30:35.186 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.186 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.186 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:35.186 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.171835+0000", 2024-03-19T11:30:35.186 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014956299999999999 2024-03-19T11:30:35.186 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.186 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.186 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:35.186 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.181520+0000", 2024-03-19T11:30:35.186 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0096855750000000001 2024-03-19T11:30:35.186 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.187 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.187 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:35.187 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.181640+0000", 2024-03-19T11:30:35.187 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011966799999999999 2024-03-19T11:30:35.187 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.187 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.187 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:35.187 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.181811+0000", 2024-03-19T11:30:35.187 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017062699999999999 2024-03-19T11:30:35.187 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.187 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.187 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:35.187 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.181938+0000", 2024-03-19T11:30:35.187 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012771899999999999 2024-03-19T11:30:35.188 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:35.188 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:35.188 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:35.188 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.188 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.188 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1231 4.0 4:0dc3b91e:::smithi03829856-28:head [write 1808051~636013] snapc 9=[9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:35.188 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.173016+0000", 2024-03-19T11:30:35.188 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 16.956913671999999, 2024-03-19T11:30:35.188 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013973719000000001, 2024-03-19T11:30:35.188 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:35.188 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:35.188 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:35.188 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:35.188 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:35.188 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1231 2024-03-19T11:30:35.189 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.189 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:35.189 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.189 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:35.189 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173016+0000", 2024-03-19T11:30:35.189 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:35.189 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.189 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.189 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:35.189 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173016+0000", 2024-03-19T11:30:35.189 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:35.189 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.189 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.189 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:35.189 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173019+0000", 2024-03-19T11:30:35.190 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.253e-06 2024-03-19T11:30:35.190 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.190 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.190 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:35.190 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176143+0000", 2024-03-19T11:30:35.190 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031240930000000001 2024-03-19T11:30:35.190 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.190 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.190 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:35.190 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176177+0000", 2024-03-19T11:30:35.190 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3237e-05 2024-03-19T11:30:35.190 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.190 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.190 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:35.191 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176196+0000", 2024-03-19T11:30:35.191 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9358e-05 2024-03-19T11:30:35.191 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.191 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.191 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:35.191 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176493+0000", 2024-03-19T11:30:35.191 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00029743899999999998 2024-03-19T11:30:35.191 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.191 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.191 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:35.191 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176786+0000", 2024-03-19T11:30:35.191 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00029259000000000002 2024-03-19T11:30:35.191 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.191 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.192 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:35.192 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177250+0000", 2024-03-19T11:30:35.192 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00046376000000000002 2024-03-19T11:30:35.192 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.192 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.192 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:35.192 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.182511+0000", 2024-03-19T11:30:35.192 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0052609019999999996 2024-03-19T11:30:35.192 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.192 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.192 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:35.192 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.186896+0000", 2024-03-19T11:30:35.192 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0043857009999999997 2024-03-19T11:30:35.192 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.192 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.193 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:35.193 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.186948+0000", 2024-03-19T11:30:35.193 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1653000000000002e-05 2024-03-19T11:30:35.193 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.193 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.193 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:35.193 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.186990+0000", 2024-03-19T11:30:35.193 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1733000000000002e-05 2024-03-19T11:30:35.193 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:35.193 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:35.193 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:35.193 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.193 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.193 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1277 4.0 4:01018511:::smithi03829856-19:head [write 763051~794393,stat] snapc b=[b,a,9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-03-19T11:30:35.193 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:21.953732+0000", 2024-03-19T11:30:35.194 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 13.176197435000001, 2024-03-19T11:30:35.194 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011784380000000001, 2024-03-19T11:30:35.194 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:35.194 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:35.194 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:35.194 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:35.194 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:35.194 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1277 2024-03-19T11:30:35.194 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.194 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:35.194 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.194 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:35.194 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.953732+0000", 2024-03-19T11:30:35.195 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:35.195 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.195 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.195 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:35.195 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.953732+0000", 2024-03-19T11:30:35.195 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:35.195 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.195 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.195 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:35.195 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.953735+0000", 2024-03-19T11:30:35.195 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1999999999999999e-06 2024-03-19T11:30:35.195 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.195 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.195 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:35.195 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.954556+0000", 2024-03-19T11:30:35.196 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00082013599999999996 2024-03-19T11:30:35.196 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.196 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.196 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:35.196 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.954596+0000", 2024-03-19T11:30:35.196 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0741000000000002e-05 2024-03-19T11:30:35.196 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.196 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.196 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:35.196 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.954617+0000", 2024-03-19T11:30:35.196 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0375999999999999e-05 2024-03-19T11:30:35.196 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.196 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.196 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:35.196 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.954747+0000", 2024-03-19T11:30:35.196 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000130648 2024-03-19T11:30:35.197 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.197 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.197 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:35.197 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.954954+0000", 2024-03-19T11:30:35.197 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020694299999999999 2024-03-19T11:30:35.197 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.197 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.197 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:35.197 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.955252+0000", 2024-03-19T11:30:35.197 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00029740700000000002 2024-03-19T11:30:35.197 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.197 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.197 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:35.197 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.961677+0000", 2024-03-19T11:30:35.197 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0064253469999999997 2024-03-19T11:30:35.198 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.198 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.198 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:35.198 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.965373+0000", 2024-03-19T11:30:35.198 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0036962649999999998 2024-03-19T11:30:35.198 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.198 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.198 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:35.198 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.965451+0000", 2024-03-19T11:30:35.198 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7204000000000001e-05 2024-03-19T11:30:35.198 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.198 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.198 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:35.198 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.965517+0000", 2024-03-19T11:30:35.198 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6112999999999995e-05 2024-03-19T11:30:35.199 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:35.199 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:35.199 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:35.199 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.199 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.199 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1408 4.10 4:0dc3b91e:::smithi03829856-28:head [read 0~3462209,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:35.199 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.918220+0000", 2024-03-19T11:30:35.199 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 9.2117092169999992, 2024-03-19T11:30:35.199 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013199458000000001, 2024-03-19T11:30:35.199 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:35.199 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:35.199 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:35.199 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:35.199 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:35.199 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1408 2024-03-19T11:30:35.200 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.200 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:35.200 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.200 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:35.200 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918220+0000", 2024-03-19T11:30:35.200 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:35.200 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.200 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.200 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:35.200 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918220+0000", 2024-03-19T11:30:35.200 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:35.200 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.200 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.200 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:35.200 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918221+0000", 2024-03-19T11:30:35.200 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0999999999999999e-07 2024-03-19T11:30:35.201 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.201 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.201 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:35.201 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918225+0000", 2024-03-19T11:30:35.201 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2710000000000003e-06 2024-03-19T11:30:35.201 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.201 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.201 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:35.201 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918233+0000", 2024-03-19T11:30:35.201 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9489999999999996e-06 2024-03-19T11:30:35.201 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.201 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.201 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:35.201 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918239+0000", 2024-03-19T11:30:35.201 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5820000000000001e-06 2024-03-19T11:30:35.202 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.202 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.202 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:35.202 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.923634+0000", 2024-03-19T11:30:35.202 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005395703 2024-03-19T11:30:35.202 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.202 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.202 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:35.202 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.923786+0000", 2024-03-19T11:30:35.202 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015117900000000001 2024-03-19T11:30:35.202 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.202 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.202 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:35.202 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.931420+0000", 2024-03-19T11:30:35.202 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.007634264 2024-03-19T11:30:35.203 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:35.203 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:35.203 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:35.203 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.203 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.203 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1409 4.10 4:0dc3b91e:::smithi03829856-28:head [sparse-read 0~1851556] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:35.203 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.918248+0000", 2024-03-19T11:30:35.203 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 9.2116813830000002, 2024-03-19T11:30:35.203 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013726986, 2024-03-19T11:30:35.203 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:35.203 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:35.203 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:35.203 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:35.203 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:35.203 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1409 2024-03-19T11:30:35.204 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.204 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:35.204 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.204 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:35.204 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918248+0000", 2024-03-19T11:30:35.204 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:35.204 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.204 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.204 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:35.204 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918248+0000", 2024-03-19T11:30:35.204 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:35.204 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.204 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.204 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:35.204 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918249+0000", 2024-03-19T11:30:35.205 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4700000000000002e-07 2024-03-19T11:30:35.205 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.205 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.205 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:35.205 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918251+0000", 2024-03-19T11:30:35.205 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.542e-06 2024-03-19T11:30:35.205 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.205 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.205 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:35.205 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918258+0000", 2024-03-19T11:30:35.205 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4629999999999998e-06 2024-03-19T11:30:35.205 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.205 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.205 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:35.206 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918263+0000", 2024-03-19T11:30:35.206 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8049999999999997e-06 2024-03-19T11:30:35.206 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.206 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.206 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:35.206 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.931469+0000", 2024-03-19T11:30:35.206 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013206458000000001 2024-03-19T11:30:35.206 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.206 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.206 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:35.206 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.931586+0000", 2024-03-19T11:30:35.206 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000116655 2024-03-19T11:30:35.206 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.206 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.206 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:35.207 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.931975+0000", 2024-03-19T11:30:35.207 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00038961600000000002 2024-03-19T11:30:35.207 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:35.207 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:35.207 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:35.207 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.207 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.207 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(3.1 49/48 [PushOp(3:9205291d:::smithi03829856-50 low_tier:head, version: 28'36, data_included: [393216~851968,1703936~196608], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:9205291d:::smithi03829856-50 low_tier:head@28'36, size: 2823543, copy_subset: [(0, 2823543)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1900544, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:baab691a:::smithi03829856-2 low_tier:head, version: 28'37, data_included: [393216~524288,1507328~509479], data_size: 1033767, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:baab691a:::smithi03829856-2 low_tier:head@28'37, size: 2016807, copy_subset: [(0, 2016807)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2016807, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:98ce8426:::smithi03829856-4 low_tier:head, version: 28'38, data_included: [393216~720896,1638400~327680], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:98ce8426:::smithi03829856-4 low_tier:head@28'38, size: 2050421, copy_subset: [(0, 2050421)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1966080, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:a86be07b:::smithi03829856-44 low_tier:head, version: 28'41, data_included: [720896~655360,1769472~393216], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:a86be07b:::smithi03829856-44 low_tier:head@28'41, size: 2722701, copy_subset: [(0, 2722701)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2162688, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:bb56919e:::smithi03829856-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'44, data_included: [393216~786432,1703936~262144], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:bb56919e:::smithi03829856-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'44, size: 2806736, copy_subset: [(0, 2806736)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1966080, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:88687f71:::smithi03829856-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'45, data_included: [589824~655360,1638400~393216], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:88687f71:::smithi03829856-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'45, size: 2201684, copy_subset: [(0, 2201684)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2031616, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:b1f822cf:::smithi03829856-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'46, data_included: [720896~786432,1966080~262144], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:b1f822cf:::smithi03829856-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'46, size: 2369754, copy_subset: [(0, 2369754)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2228224, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:a5e8313a:::smithi03829856-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'47, data_included: [524288~720896,1769472~327680], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:a5e8313a:::smithi03829856-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'47, size: 2571438, copy_subset: [(0, 2571438)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:bca5ec64:::e3a26bf9230b1b95b8cf1a4cb01909236bf2bf3fd59b083b2291d278ab98264d:head, version: 28'48, data_included: [0~393216], data_size: 393216, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bca5ec64:::e3a26bf9230b1b95b8cf1a4cb01909236bf2bf3fd59b083b2291d278ab98264d:head@28'48, size: 533504, copy_subset: [(0, 533504)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 533504, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:30:35.207 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:26.948137+0000", 2024-03-19T11:30:35.207 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.1817925700000007, 2024-03-19T11:30:35.207 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.036085578, 2024-03-19T11:30:35.207 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:35.207 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:35.207 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:35.208 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.208 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:35.208 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.948137+0000", 2024-03-19T11:30:35.208 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:35.208 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.208 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.208 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:35.208 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.948137+0000", 2024-03-19T11:30:35.208 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:35.208 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.208 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.208 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:35.208 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.948139+0000", 2024-03-19T11:30:35.208 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1399999999999998e-06 2024-03-19T11:30:35.208 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.209 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.209 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:35.209 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954288+0000", 2024-03-19T11:30:35.209 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0061487429999999999 2024-03-19T11:30:35.209 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.209 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.209 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:35.209 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954388+0000", 2024-03-19T11:30:35.209 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010022 2024-03-19T11:30:35.209 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.209 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.209 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:35.209 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954523+0000", 2024-03-19T11:30:35.209 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000134332 2024-03-19T11:30:35.209 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.209 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.210 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:35.210 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.955120+0000", 2024-03-19T11:30:35.210 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00059695799999999997 2024-03-19T11:30:35.210 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.210 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.210 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:35.210 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.955194+0000", 2024-03-19T11:30:35.210 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4077000000000002e-05 2024-03-19T11:30:35.210 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.210 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.210 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:35.210 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.984223+0000", 2024-03-19T11:30:35.210 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029029108000000001 2024-03-19T11:30:35.210 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:35.211 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:35.211 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:35.211 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.211 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.211 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(3.1 49/48 [PushOp(3:99a49d1b:::895a63407b6109b3cca9a6f5ef827134d56679bef42daadc5892269682c428ed:head, version: 32'49, data_included: [393216~388096], data_size: 388096, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:99a49d1b:::895a63407b6109b3cca9a6f5ef827134d56679bef42daadc5892269682c428ed:head@32'49, size: 781312, copy_subset: [(0, 781312)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 781312, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:30:35.211 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:26.954637+0000", 2024-03-19T11:30:35.211 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.1752929529999996, 2024-03-19T11:30:35.211 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.030729057000000001, 2024-03-19T11:30:35.211 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:35.211 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:35.211 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:35.211 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.211 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:35.211 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954637+0000", 2024-03-19T11:30:35.211 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:35.212 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.212 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.212 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:35.212 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954637+0000", 2024-03-19T11:30:35.212 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:35.212 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.212 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.212 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:35.212 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954637+0000", 2024-03-19T11:30:35.212 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7299999999999995e-07 2024-03-19T11:30:35.212 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.212 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.212 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:35.212 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954873+0000", 2024-03-19T11:30:35.212 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023520299999999999 2024-03-19T11:30:35.213 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.213 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.213 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:35.213 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954897+0000", 2024-03-19T11:30:35.213 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4110000000000001e-05 2024-03-19T11:30:35.213 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.213 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.213 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:35.213 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954917+0000", 2024-03-19T11:30:35.213 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.067e-05 2024-03-19T11:30:35.213 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.213 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.213 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:35.213 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.984289+0000", 2024-03-19T11:30:35.213 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029371191000000001 2024-03-19T11:30:35.214 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.214 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.214 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:35.214 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.984303+0000", 2024-03-19T11:30:35.214 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4888999999999999e-05 2024-03-19T11:30:35.214 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.214 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.214 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:35.214 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.985366+0000", 2024-03-19T11:30:35.214 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001062321 2024-03-19T11:30:35.214 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:35.214 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:35.214 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:35.214 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.214 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.215 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1472 4.10 4:0dc3b91e:::smithi03829856-28:head [read 534528~391680] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-03-19T11:30:35.215 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:27.091748+0000", 2024-03-19T11:30:35.215 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.0381820650000009, 2024-03-19T11:30:35.215 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.83574118500000005, 2024-03-19T11:30:35.215 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:35.215 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:35.215 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:35.215 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:35.215 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:35.215 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1472 2024-03-19T11:30:35.215 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.215 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:35.215 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.215 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:35.216 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091748+0000", 2024-03-19T11:30:35.216 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:35.216 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.216 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.216 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:35.216 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091748+0000", 2024-03-19T11:30:35.216 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:35.216 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.216 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.216 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:35.216 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091750+0000", 2024-03-19T11:30:35.216 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3130000000000001e-06 2024-03-19T11:30:35.216 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.216 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.216 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:35.217 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091760+0000", 2024-03-19T11:30:35.217 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0560999999999999e-05 2024-03-19T11:30:35.217 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.217 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.217 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:35.217 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091784+0000", 2024-03-19T11:30:35.217 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3915000000000001e-05 2024-03-19T11:30:35.217 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.217 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.217 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:35.217 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091797+0000", 2024-03-19T11:30:35.217 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2660000000000001e-05 2024-03-19T11:30:35.217 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.217 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.217 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:35.218 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091874+0000", 2024-03-19T11:30:35.218 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6757999999999999e-05 2024-03-19T11:30:35.218 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.218 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.218 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:30:35.218 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091882+0000", 2024-03-19T11:30:35.218 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5520000000000005e-06 2024-03-19T11:30:35.218 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.218 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.218 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:35.218 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.927033+0000", 2024-03-19T11:30:35.218 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.83515094099999998 2024-03-19T11:30:35.218 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.218 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.218 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:35.219 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.927250+0000", 2024-03-19T11:30:35.219 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00021675199999999999 2024-03-19T11:30:35.219 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.219 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.219 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:35.219 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.927489+0000", 2024-03-19T11:30:35.219 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023873300000000001 2024-03-19T11:30:35.219 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:35.219 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:35.219 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:35.219 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.219 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.219 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1474 4.10 4:0dc3b91e:::smithi03829856-28:head [assert-version v27,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-03-19T11:30:35.219 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:27.934139+0000", 2024-03-19T11:30:35.220 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 7.1957902789999997, 2024-03-19T11:30:35.220 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024764884000000001, 2024-03-19T11:30:35.220 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:35.220 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:35.220 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:35.220 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:35.220 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:35.220 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1474 2024-03-19T11:30:35.220 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.220 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:35.220 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.220 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:35.220 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934139+0000", 2024-03-19T11:30:35.220 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:35.220 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.220 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.221 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:35.221 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934139+0000", 2024-03-19T11:30:35.221 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:35.221 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.221 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.221 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:35.221 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934142+0000", 2024-03-19T11:30:35.221 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5459999999999998e-06 2024-03-19T11:30:35.221 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.221 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.221 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:35.221 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934152+0000", 2024-03-19T11:30:35.221 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9399999999999997e-06 2024-03-19T11:30:35.221 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.221 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.222 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:35.222 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934173+0000", 2024-03-19T11:30:35.222 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1522000000000002e-05 2024-03-19T11:30:35.222 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.222 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.222 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:35.222 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934185+0000", 2024-03-19T11:30:35.222 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1946999999999999e-05 2024-03-19T11:30:35.222 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.222 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.222 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:35.222 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934282+0000", 2024-03-19T11:30:35.222 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6272000000000003e-05 2024-03-19T11:30:35.222 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.222 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.223 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:30:35.223 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934379+0000", 2024-03-19T11:30:35.223 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6966999999999995e-05 2024-03-19T11:30:35.223 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.223 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.223 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:35.223 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.955023+0000", 2024-03-19T11:30:35.223 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020644302 2024-03-19T11:30:35.223 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.223 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.223 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:35.223 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.955167+0000", 2024-03-19T11:30:35.223 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014404500000000001 2024-03-19T11:30:35.223 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.224 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.224 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:35.224 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.957913+0000", 2024-03-19T11:30:35.224 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0027463779999999998 2024-03-19T11:30:35.224 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.224 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.224 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:35.224 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.958548+0000", 2024-03-19T11:30:35.224 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00063495700000000001 2024-03-19T11:30:35.224 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.224 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.224 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:35.224 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.958785+0000", 2024-03-19T11:30:35.224 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000236872 2024-03-19T11:30:35.224 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.225 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.225 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:35.225 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.958854+0000", 2024-03-19T11:30:35.225 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9099999999999999e-05 2024-03-19T11:30:35.225 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:35.225 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:35.225 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:35.225 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.958904+0000", 2024-03-19T11:30:35.225 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0036e-05 2024-03-19T11:30:35.225 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:35.225 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:35.225 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:35.225 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:35.225 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:35.225 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:35.227 INFO:tasks.rados.rados.0.smithi038.stdout:899: set_chunk oid 10 offset: 182784 length: 2893824 target oid e7a0126b74590803e762ea9614dff8447a0272ff3000fd5c74e92cd9ee65bec5 offset: 0 snap -1 2024-03-19T11:30:35.227 INFO:tasks.rados.rados.0.smithi038.stdout:900: write (excl) oid 11 current snap is 15 2024-03-19T11:30:35.227 INFO:tasks.rados.rados.0.smithi038.stdout:900: seq_num 122 ranges {450454=642026,1505862=544592} 2024-03-19T11:30:35.228 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-19T11:30:35.229 INFO:tasks.rados.rados.0.smithi038.stdout:900: writing smithi03829856-11 from 450454 to 1092480 tid 1 2024-03-19T11:30:35.230 INFO:tasks.rados.rados.0.smithi038.stdout:900: writing smithi03829856-11 from 1505862 to 2050454 tid 2 2024-03-19T11:30:35.230 INFO:tasks.rados.rados.0.smithi038.stdout:901: set_chunk oid 8 target oid 2024-03-19T11:30:35.230 INFO:tasks.rados.rados.0.smithi038.stdout:901: set_chunk oid 8 offset: 920064 length: 1650176 target oid f1224694c14fcb009af239eade27a8235cc6a98f1542b7554b67795eb195d6e5 offset: 0 snap 10 2024-03-19T11:30:35.230 INFO:tasks.rados.rados.0.smithi038.stdout:902: read oid 49 snap -1 2024-03-19T11:30:35.230 INFO:tasks.rados.rados.0.smithi038.stdout:902: expect (ObjNum 48 snap 0 seq_num 48) 2024-03-19T11:30:35.230 INFO:tasks.rados.rados.0.smithi038.stdout:903: read oid 16 snap -1 2024-03-19T11:30:35.230 INFO:tasks.rados.rados.0.smithi038.stdout:903: expect (ObjNum 118 snap 13 seq_num 118) 2024-03-19T11:30:35.230 INFO:tasks.rados.rados.0.smithi038.stdout:904: write oid 26 current snap is 15 2024-03-19T11:30:35.230 INFO:tasks.rados.rados.0.smithi038.stdout:897: expect (ObjNum 3 snap 0 seq_num 3) 2024-03-19T11:30:35.231 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:35.237+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:35.233 INFO:tasks.rados.rados.0.smithi038.stdout:898: expect (ObjNum 23 snap 0 seq_num 23) 2024-03-19T11:30:35.311 INFO:tasks.rados.rados.0.smithi038.stdout:899: finishing set_chunk to oid 10 2024-03-19T11:30:35.311 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 10 set_chunk returned error code -95 offset: 182784 length: 2893824 2024-03-19T11:30:35.311 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 10 v 0 (ObjNum 104 snap 10 seq_num 104) dirty exists 2024-03-19T11:30:35.311 INFO:tasks.rados.rados.0.smithi038.stdout:900: finishing write tid 1 to smithi03829856-11 2024-03-19T11:30:35.311 INFO:tasks.rados.rados.0.smithi038.stdout:904: seq_num 123 ranges {467261=717275,1847991=219270} 2024-03-19T11:30:35.317 INFO:tasks.rados.rados.0.smithi038.stdout:904: writing smithi03829856-26 from 467261 to 1184536 tid 1 2024-03-19T11:30:35.323 INFO:tasks.rados.rados.0.smithi038.stdout:904: writing smithi03829856-26 from 1847991 to 2067261 tid 2 2024-03-19T11:30:35.323 INFO:tasks.rados.rados.0.smithi038.stdout:897: done (7 left) 2024-03-19T11:30:35.323 INFO:tasks.rados.rados.0.smithi038.stdout:898: done (6 left) 2024-03-19T11:30:35.323 INFO:tasks.rados.rados.0.smithi038.stdout:899: done (5 left) 2024-03-19T11:30:35.323 INFO:tasks.rados.rados.0.smithi038.stdout:905: set_chunk oid 7 target oid 2024-03-19T11:30:35.327 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:35.337+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:35.330 INFO:tasks.rados.rados.0.smithi038.stdout:905: set_chunk oid 7 offset: 2382336 length: 109568 target oid c32764c6f12d9e812333bdd4020d52bce496e061d10a9dceeb78bf2ab92b8504 offset: 0 snap 11 2024-03-19T11:30:35.330 INFO:tasks.rados.rados.0.smithi038.stdout:906: write (excl) oid 14 current snap is 15 2024-03-19T11:30:35.330 INFO:tasks.rados.rados.0.smithi038.stdout:900: finishing write tid 2 to smithi03829856-11 2024-03-19T11:30:35.330 INFO:tasks.rados.rados.0.smithi038.stdout:900: finishing write tid 3 to smithi03829856-11 2024-03-19T11:30:35.330 INFO:tasks.rados.rados.0.smithi038.stdout:900: finishing write tid 4 to smithi03829856-11 2024-03-19T11:30:35.330 INFO:tasks.rados.rados.0.smithi038.stdout:900: oid 11 updating version 0 to 54 2024-03-19T11:30:35.330 INFO:tasks.rados.rados.0.smithi038.stdout:900: oid 11 version 54 is already newer than 52 2024-03-19T11:30:35.330 INFO:tasks.rados.rados.0.smithi038.stdout:900: oid 11 version 54 is already newer than 53 2024-03-19T11:30:35.330 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 11 v 54 (ObjNum 122 snap 15 seq_num 122) dirty exists 2024-03-19T11:30:35.331 INFO:tasks.rados.rados.0.smithi038.stdout:900: left oid 11 (ObjNum 122 snap 15 seq_num 122) 2024-03-19T11:30:35.331 INFO:tasks.rados.rados.0.smithi038.stdout:901: finishing set_chunk to oid 8 2024-03-19T11:30:35.331 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 8 set_chunk returned error code -95 offset: 920064 length: 1650176 2024-03-19T11:30:35.331 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 8 v 0 (ObjNum 60 snap 3 seq_num 60) dirty exists 2024-03-19T11:30:35.331 INFO:tasks.rados.rados.0.smithi038.stdout:902: expect (ObjNum 48 snap 0 seq_num 48) 2024-03-19T11:30:35.361 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:35.361 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:35.361 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:30:35.361 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:35.375 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-19T11:30:35.417 INFO:tasks.rados.rados.0.smithi038.stdout:903: expect (ObjNum 118 snap 13 seq_num 118) 2024-03-19T11:30:35.462 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:35.473+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:35.532 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:35.543+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:35.544 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:35.544 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:35.544 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:30:35.544 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:30:35.544 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:35.558 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-19T11:30:35.635 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:35.647+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:35.638 INFO:tasks.rados.rados.0.smithi038.stdout:905: finishing set_chunk to oid 7 2024-03-19T11:30:35.638 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 7 set_chunk returned error code -95 offset: 2382336 length: 109568 2024-03-19T11:30:35.638 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 7 v 0 (ObjNum 67 snap 3 seq_num 67) dirty exists 2024-03-19T11:30:35.638 INFO:tasks.rados.rados.0.smithi038.stdout:904: finishing write tid 1 to smithi03829856-26 2024-03-19T11:30:35.638 INFO:tasks.rados.rados.0.smithi038.stdout:904: finishing write tid 2 to smithi03829856-26 2024-03-19T11:30:35.638 INFO:tasks.rados.rados.0.smithi038.stdout:904: finishing write tid 3 to smithi03829856-26 2024-03-19T11:30:35.638 INFO:tasks.rados.rados.0.smithi038.stdout:904: finishing write tid 4 to smithi03829856-26 2024-03-19T11:30:35.638 INFO:tasks.rados.rados.0.smithi038.stdout:904: oid 26 updating version 0 to 53 2024-03-19T11:30:35.638 INFO:tasks.rados.rados.0.smithi038.stdout:904: oid 26 updating version 53 to 55 2024-03-19T11:30:35.639 INFO:tasks.rados.rados.0.smithi038.stdout:904: oid 26 version 55 is already newer than 54 2024-03-19T11:30:35.639 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 26 v 55 (ObjNum 123 snap 15 seq_num 123) dirty exists 2024-03-19T11:30:35.639 INFO:tasks.rados.rados.0.smithi038.stdout:904: left oid 26 (ObjNum 123 snap 15 seq_num 123) 2024-03-19T11:30:35.639 INFO:tasks.rados.rados.0.smithi038.stdout:906: seq_num 124 ranges {484068=792524,1906473=177595} 2024-03-19T11:30:35.659 INFO:tasks.rados.rados.0.smithi038.stdout:906: writing smithi03829856-14 from 484068 to 1276592 tid 1 2024-03-19T11:30:35.672 INFO:tasks.rados.rados.0.smithi038.stdout:906: writing smithi03829856-14 from 1906473 to 2084068 tid 2 2024-03-19T11:30:35.672 INFO:tasks.rados.rados.0.smithi038.stdout:900: done (6 left) 2024-03-19T11:30:35.673 INFO:tasks.rados.rados.0.smithi038.stdout:901: done (5 left) 2024-03-19T11:30:35.673 INFO:tasks.rados.rados.0.smithi038.stdout:902: done (4 left) 2024-03-19T11:30:35.673 INFO:tasks.rados.rados.0.smithi038.stdout:903: done (3 left) 2024-03-19T11:30:35.673 INFO:tasks.rados.rados.0.smithi038.stdout:904: done (2 left) 2024-03-19T11:30:35.673 INFO:tasks.rados.rados.0.smithi038.stdout:905: done (1 left) 2024-03-19T11:30:35.673 INFO:tasks.rados.rados.0.smithi038.stdout:907: read oid 20 snap -1 2024-03-19T11:30:35.673 INFO:tasks.rados.rados.0.smithi038.stdout:907: expect (ObjNum 19 snap 0 seq_num 19) 2024-03-19T11:30:35.673 INFO:tasks.rados.rados.0.smithi038.stdout:908: read oid 33 snap -1 2024-03-19T11:30:35.673 INFO:tasks.rados.rados.0.smithi038.stdout:908: expect (ObjNum 32 snap 0 seq_num 32) 2024-03-19T11:30:35.673 INFO:tasks.rados.rados.0.smithi038.stdout:909: snap_remove snap 13 2024-03-19T11:30:35.736 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:35.747+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:35.738 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:35.738 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:30:35.739 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:30:35.739 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:30:35.739 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.739 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:9 4.d 4:b4ef97e0:::smithi03829856-3:head [write 433614~550498,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-03-19T11:30:35.739 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:04.706000+0000", 2024-03-19T11:30:35.739 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 31.042018416000001, 2024-03-19T11:30:35.739 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013227001, 2024-03-19T11:30:35.740 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:35.740 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:35.740 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:35.740 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:35.740 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:35.740 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 9 2024-03-19T11:30:35.740 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.741 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:35.741 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.741 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:35.741 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.706000+0000", 2024-03-19T11:30:35.741 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:35.741 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.741 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.741 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:35.741 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.706000+0000", 2024-03-19T11:30:35.741 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:35.742 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.742 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.742 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:35.742 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.706002+0000", 2024-03-19T11:30:35.742 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5569999999999998e-06 2024-03-19T11:30:35.742 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.742 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.742 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:35.742 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.707261+0000", 2024-03-19T11:30:35.742 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001258817 2024-03-19T11:30:35.743 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.743 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.743 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:35.743 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.707288+0000", 2024-03-19T11:30:35.743 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7093e-05 2024-03-19T11:30:35.743 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.743 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.743 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:35.743 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.707305+0000", 2024-03-19T11:30:35.743 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6330000000000001e-05 2024-03-19T11:30:35.743 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.743 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.743 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:35.743 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.707610+0000", 2024-03-19T11:30:35.744 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00030568799999999998 2024-03-19T11:30:35.744 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.744 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.744 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:35.744 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.707984+0000", 2024-03-19T11:30:35.744 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00037399299999999998 2024-03-19T11:30:35.744 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.744 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.744 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2", 2024-03-19T11:30:35.744 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.708409+0000", 2024-03-19T11:30:35.744 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00042503099999999998 2024-03-19T11:30:35.744 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.744 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.744 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:35.745 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.712919+0000", 2024-03-19T11:30:35.745 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0045099010000000002 2024-03-19T11:30:35.745 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.745 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.745 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:35.745 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.719037+0000", 2024-03-19T11:30:35.745 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061173770000000002 2024-03-19T11:30:35.745 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.745 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.745 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:35.745 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.719116+0000", 2024-03-19T11:30:35.745 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9935e-05 2024-03-19T11:30:35.745 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.745 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.745 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:35.746 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.719227+0000", 2024-03-19T11:30:35.746 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011027900000000001 2024-03-19T11:30:35.746 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:35.746 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:35.746 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:35.746 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.746 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.746 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:848 4.1 4:866f3dcc:::smithi03829856-31:head [read 0~2941192,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:35.746 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.757468+0000", 2024-03-19T11:30:35.746 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 26.990550423999998, 2024-03-19T11:30:35.746 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015020110999999999, 2024-03-19T11:30:35.746 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:35.746 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:35.746 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:35.747 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:35.747 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:35.747 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 848 2024-03-19T11:30:35.747 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.747 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:35.747 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.747 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:35.747 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757468+0000", 2024-03-19T11:30:35.747 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:35.747 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.747 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.747 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:35.747 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757468+0000", 2024-03-19T11:30:35.747 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:35.748 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.748 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.748 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:35.748 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757471+0000", 2024-03-19T11:30:35.748 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.67e-06 2024-03-19T11:30:35.748 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.748 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.748 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:35.748 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757488+0000", 2024-03-19T11:30:35.748 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6223e-05 2024-03-19T11:30:35.748 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.748 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.748 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:35.748 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757521+0000", 2024-03-19T11:30:35.748 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.358e-05 2024-03-19T11:30:35.749 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.749 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.749 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:35.749 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757543+0000", 2024-03-19T11:30:35.749 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1330000000000001e-05 2024-03-19T11:30:35.749 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.749 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.749 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:35.749 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757745+0000", 2024-03-19T11:30:35.749 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020259799999999999 2024-03-19T11:30:35.749 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.749 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.749 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op must wait for map", 2024-03-19T11:30:35.749 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757760+0000", 2024-03-19T11:30:35.749 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4647000000000001e-05 2024-03-19T11:30:35.750 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.750 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.750 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:35.750 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763972+0000", 2024-03-19T11:30:35.750 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062126480000000003 2024-03-19T11:30:35.750 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.750 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.750 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:35.750 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.764547+0000", 2024-03-19T11:30:35.750 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057440200000000001 2024-03-19T11:30:35.750 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.750 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.750 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:35.750 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.772488+0000", 2024-03-19T11:30:35.751 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079410130000000002 2024-03-19T11:30:35.751 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:35.751 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:35.751 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:35.751 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.751 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.751 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:849 4.1 4:866f3dcc:::smithi03829856-31:head [sparse-read 0~1359577] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:35.751 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.757583+0000", 2024-03-19T11:30:35.751 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 26.990434732000001, 2024-03-19T11:30:35.751 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016241202, 2024-03-19T11:30:35.751 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:35.751 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:35.751 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:35.751 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:35.751 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:35.752 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 849 2024-03-19T11:30:35.752 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.752 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:35.752 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.752 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:35.752 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757583+0000", 2024-03-19T11:30:35.752 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:35.752 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.752 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.752 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:35.752 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757583+0000", 2024-03-19T11:30:35.752 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:35.752 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.752 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.752 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:35.753 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757584+0000", 2024-03-19T11:30:35.753 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.077e-06 2024-03-19T11:30:35.753 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.753 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.753 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:35.753 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757593+0000", 2024-03-19T11:30:35.753 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3059999999999998e-06 2024-03-19T11:30:35.753 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.753 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.753 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:35.753 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757608+0000", 2024-03-19T11:30:35.753 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5526999999999999e-05 2024-03-19T11:30:35.753 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.753 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.753 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:35.754 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757618+0000", 2024-03-19T11:30:35.754 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5999999999999996e-06 2024-03-19T11:30:35.754 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.754 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.754 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:35.754 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757886+0000", 2024-03-19T11:30:35.754 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026839099999999999 2024-03-19T11:30:35.754 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.754 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.754 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:35.754 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757899+0000", 2024-03-19T11:30:35.754 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3004e-05 2024-03-19T11:30:35.754 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.754 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.755 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:35.755 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.772651+0000", 2024-03-19T11:30:35.755 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014751702 2024-03-19T11:30:35.755 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.755 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.755 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:35.755 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.772924+0000", 2024-03-19T11:30:35.755 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027314199999999998 2024-03-19T11:30:35.755 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.755 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.755 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:35.755 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.773825+0000", 2024-03-19T11:30:35.755 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00090045299999999995 2024-03-19T11:30:35.756 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:35.756 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:35.756 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:35.756 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.756 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.756 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:850 4.1 4:866f3dcc:::smithi03829856-31:head [read 0~2941192,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:35.756 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.757636+0000", 2024-03-19T11:30:35.756 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 26.990382061999998, 2024-03-19T11:30:35.756 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018834801000000002, 2024-03-19T11:30:35.756 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:35.756 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:35.756 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:35.756 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:35.756 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:35.756 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 850 2024-03-19T11:30:35.757 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.757 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:35.757 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.757 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:35.757 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757636+0000", 2024-03-19T11:30:35.757 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:35.757 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.757 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.757 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:35.757 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757636+0000", 2024-03-19T11:30:35.757 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:35.757 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.757 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.757 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:35.758 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757637+0000", 2024-03-19T11:30:35.758 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5999999999999991e-07 2024-03-19T11:30:35.758 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.758 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.758 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:35.758 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757643+0000", 2024-03-19T11:30:35.758 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3199999999999996e-06 2024-03-19T11:30:35.758 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.758 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.758 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:35.758 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757670+0000", 2024-03-19T11:30:35.758 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6682999999999998e-05 2024-03-19T11:30:35.758 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.759 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:35.759 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757679+0000", 2024-03-19T11:30:35.759 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.986e-06 2024-03-19T11:30:35.759 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.759 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:35.759 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757961+0000", 2024-03-19T11:30:35.759 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028230799999999999 2024-03-19T11:30:35.759 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.759 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:35.759 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757971+0000", 2024-03-19T11:30:35.759 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.38e-06 2024-03-19T11:30:35.759 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.760 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.760 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:35.760 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.774079+0000", 2024-03-19T11:30:35.760 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016108573000000001 2024-03-19T11:30:35.760 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.760 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.760 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:35.760 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.774306+0000", 2024-03-19T11:30:35.760 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022637499999999999 2024-03-19T11:30:35.760 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.760 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.760 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:35.760 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776471+0000", 2024-03-19T11:30:35.760 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021652160000000002 2024-03-19T11:30:35.760 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:35.761 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:35.761 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:35.761 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.761 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.761 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:851 4.9 4:953e720b:::smithi03829856-45:head [write 591613~454162,stat] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:35.761 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.763514+0000", 2024-03-19T11:30:35.761 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 26.984503871000001, 2024-03-19T11:30:35.761 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027825888999999999, 2024-03-19T11:30:35.761 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:35.761 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:35.761 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:35.761 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:35.761 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:35.761 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 851 2024-03-19T11:30:35.761 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.762 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:35.762 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.762 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:35.762 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763514+0000", 2024-03-19T11:30:35.762 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:35.762 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.762 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.762 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:35.762 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763514+0000", 2024-03-19T11:30:35.762 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:35.762 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.762 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.762 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:35.763 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763517+0000", 2024-03-19T11:30:35.763 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8499999999999998e-06 2024-03-19T11:30:35.763 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.763 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:35.763 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.764714+0000", 2024-03-19T11:30:35.763 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011971270000000001 2024-03-19T11:30:35.763 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.763 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:35.763 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.764745+0000", 2024-03-19T11:30:35.763 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0546999999999997e-05 2024-03-19T11:30:35.763 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.764 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:35.764 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.764767+0000", 2024-03-19T11:30:35.764 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2286999999999999e-05 2024-03-19T11:30:35.764 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.764 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:35.764 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.773924+0000", 2024-03-19T11:30:35.764 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0091567360000000004 2024-03-19T11:30:35.764 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.764 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:35.764 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.774217+0000", 2024-03-19T11:30:35.764 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029350099999999999 2024-03-19T11:30:35.764 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.765 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:35.765 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.774770+0000", 2024-03-19T11:30:35.765 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00055288600000000005 2024-03-19T11:30:35.765 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.765 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.765 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:35.765 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.786413+0000", 2024-03-19T11:30:35.765 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011643229999999999 2024-03-19T11:30:35.765 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.765 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.765 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:35.765 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791104+0000", 2024-03-19T11:30:35.765 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0046900830000000003 2024-03-19T11:30:35.765 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.766 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.766 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:35.766 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791248+0000", 2024-03-19T11:30:35.766 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014410200000000001 2024-03-19T11:30:35.766 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.766 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.766 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:35.766 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791340+0000", 2024-03-19T11:30:35.766 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2540000000000005e-05 2024-03-19T11:30:35.766 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:35.766 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:35.766 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:35.766 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.766 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.767 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:852 4.9 4:953e720b:::smithi03829856-45:head [write 1719614~417353] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:35.767 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.769707+0000", 2024-03-19T11:30:35.767 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 26.978310844999999, 2024-03-19T11:30:35.767 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022476643000000001, 2024-03-19T11:30:35.767 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:35.767 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:35.767 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:35.767 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:35.767 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:35.767 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 852 2024-03-19T11:30:35.767 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.767 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:35.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.767 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:35.768 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.769707+0000", 2024-03-19T11:30:35.768 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:35.768 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.768 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.768 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:35.768 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.769707+0000", 2024-03-19T11:30:35.768 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:35.768 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.768 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.768 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:35.768 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.769710+0000", 2024-03-19T11:30:35.768 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6230000000000001e-06 2024-03-19T11:30:35.768 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.768 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.768 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:35.769 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.770385+0000", 2024-03-19T11:30:35.769 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00067504900000000005 2024-03-19T11:30:35.769 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.769 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.769 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:35.769 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.770419+0000", 2024-03-19T11:30:35.769 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3528999999999997e-05 2024-03-19T11:30:35.769 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.769 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.769 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:35.769 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.770436+0000", 2024-03-19T11:30:35.769 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7615999999999999e-05 2024-03-19T11:30:35.769 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.769 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.770 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:35.770 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.781327+0000", 2024-03-19T11:30:35.770 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010891343 2024-03-19T11:30:35.770 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.770 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.770 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:35.770 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.781616+0000", 2024-03-19T11:30:35.770 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028857399999999999 2024-03-19T11:30:35.770 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.770 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.770 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:35.770 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.782125+0000", 2024-03-19T11:30:35.770 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00050941599999999995 2024-03-19T11:30:35.770 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.771 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:35.771 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.786606+0000", 2024-03-19T11:30:35.771 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044809560000000003 2024-03-19T11:30:35.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.771 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:35.771 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791995+0000", 2024-03-19T11:30:35.771 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053889150000000002 2024-03-19T11:30:35.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.771 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:35.771 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.792084+0000", 2024-03-19T11:30:35.771 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8633000000000003e-05 2024-03-19T11:30:35.772 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.772 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.772 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:35.772 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.792184+0000", 2024-03-19T11:30:35.772 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9988999999999994e-05 2024-03-19T11:30:35.772 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:35.772 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:35.772 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:35.772 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.772 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.772 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:853 4.9 4:953e720b:::smithi03829856-45:head [write 2570780~420833] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:35.772 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.775636+0000", 2024-03-19T11:30:35.772 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 26.972382224, 2024-03-19T11:30:35.772 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018795850999999999, 2024-03-19T11:30:35.773 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:35.773 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:35.773 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:35.773 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:35.773 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:35.773 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 853 2024-03-19T11:30:35.773 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.773 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:35.773 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.773 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:35.773 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.775636+0000", 2024-03-19T11:30:35.773 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:35.773 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.773 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.774 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:35.774 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.775636+0000", 2024-03-19T11:30:35.774 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:35.774 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.774 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.774 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:35.774 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.775639+0000", 2024-03-19T11:30:35.774 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4529999999999998e-06 2024-03-19T11:30:35.774 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.774 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.774 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:35.774 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776492+0000", 2024-03-19T11:30:35.774 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00085245000000000004 2024-03-19T11:30:35.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:35.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776526+0000", 2024-03-19T11:30:35.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4090000000000001e-05 2024-03-19T11:30:35.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:35.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776550+0000", 2024-03-19T11:30:35.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4139999999999999e-05 2024-03-19T11:30:35.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:35.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.786752+0000", 2024-03-19T11:30:35.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010201834 2024-03-19T11:30:35.776 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.776 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.776 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:35.776 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.786994+0000", 2024-03-19T11:30:35.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000241952 2024-03-19T11:30:35.776 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.776 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.776 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:35.776 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.787301+0000", 2024-03-19T11:30:35.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00030690699999999998 2024-03-19T11:30:35.776 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.776 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.776 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:35.777 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791965+0000", 2024-03-19T11:30:35.777 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0046644379999999999 2024-03-19T11:30:35.777 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.777 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.777 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:35.777 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.794165+0000", 2024-03-19T11:30:35.777 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021995389999999999 2024-03-19T11:30:35.777 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.777 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.777 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:35.777 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.794313+0000", 2024-03-19T11:30:35.777 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000148369 2024-03-19T11:30:35.777 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.777 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.778 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:35.778 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.794432+0000", 2024-03-19T11:30:35.778 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011867900000000001 2024-03-19T11:30:35.778 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:35.778 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:35.778 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:35.778 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.778 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.778 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:854 4.9 4:953e720b:::smithi03829856-45:head [setxattr _header (56) in=7b,truncate 2991613] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:35.778 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.776592+0000", 2024-03-19T11:30:35.778 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 26.971426502, 2024-03-19T11:30:35.778 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022815621000000001, 2024-03-19T11:30:35.778 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:35.779 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:35.779 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:35.779 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:35.779 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:35.779 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 854 2024-03-19T11:30:35.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.779 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:35.779 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:35.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776592+0000", 2024-03-19T11:30:35.779 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:35.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.779 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:35.780 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776592+0000", 2024-03-19T11:30:35.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:35.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.780 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:35.780 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776594+0000", 2024-03-19T11:30:35.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.95e-06 2024-03-19T11:30:35.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.780 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:35.780 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776603+0000", 2024-03-19T11:30:35.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2399999999999996e-06 2024-03-19T11:30:35.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.780 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:35.781 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776623+0000", 2024-03-19T11:30:35.781 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.035e-05 2024-03-19T11:30:35.781 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.781 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.781 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:35.781 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776639+0000", 2024-03-19T11:30:35.781 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5642999999999999e-05 2024-03-19T11:30:35.781 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.781 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.781 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:35.781 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.792324+0000", 2024-03-19T11:30:35.781 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015685075999999999 2024-03-19T11:30:35.781 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.781 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.782 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:35.782 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.792599+0000", 2024-03-19T11:30:35.782 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000275264 2024-03-19T11:30:35.782 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.782 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.782 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:35.782 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.793092+0000", 2024-03-19T11:30:35.782 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00049264600000000001 2024-03-19T11:30:35.782 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.782 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.782 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:35.782 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.794668+0000", 2024-03-19T11:30:35.782 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001576072 2024-03-19T11:30:35.782 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.783 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:35.783 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.799082+0000", 2024-03-19T11:30:35.783 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044145190000000004 2024-03-19T11:30:35.783 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.783 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:35.783 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.799249+0000", 2024-03-19T11:30:35.783 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016701199999999999 2024-03-19T11:30:35.783 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.783 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:35.783 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.799407+0000", 2024-03-19T11:30:35.783 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000157849 2024-03-19T11:30:35.784 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:35.784 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:35.784 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:35.784 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.784 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.784 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:855 4.9 4:953e720b:::smithi03829856-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:35.784 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.776660+0000", 2024-03-19T11:30:35.784 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 26.971357908999998, 2024-03-19T11:30:35.784 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023419786000000001, 2024-03-19T11:30:35.784 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:35.784 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:35.784 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:35.784 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:35.784 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:35.784 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 855 2024-03-19T11:30:35.785 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.785 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:35.785 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.785 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:35.785 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776660+0000", 2024-03-19T11:30:35.785 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:35.785 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.785 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.785 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:35.785 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776660+0000", 2024-03-19T11:30:35.785 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:35.785 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.785 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.785 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:35.786 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776662+0000", 2024-03-19T11:30:35.786 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3430000000000001e-06 2024-03-19T11:30:35.786 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.786 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.786 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:35.786 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776668+0000", 2024-03-19T11:30:35.786 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3500000000000002e-06 2024-03-19T11:30:35.786 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.786 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.786 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:35.786 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776687+0000", 2024-03-19T11:30:35.786 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9273999999999999e-05 2024-03-19T11:30:35.786 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.786 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.787 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:35.787 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776705+0000", 2024-03-19T11:30:35.787 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7839999999999999e-05 2024-03-19T11:30:35.787 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.787 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.787 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:35.787 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.793874+0000", 2024-03-19T11:30:35.787 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017168597000000001 2024-03-19T11:30:35.787 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.787 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.787 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-03-19T11:30:35.788 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.794132+0000", 2024-03-19T11:30:35.788 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025790399999999999 2024-03-19T11:30:35.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.788 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:35.788 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.799504+0000", 2024-03-19T11:30:35.788 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053724159999999997 2024-03-19T11:30:35.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.788 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:35.788 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.799782+0000", 2024-03-19T11:30:35.788 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027785499999999997 2024-03-19T11:30:35.789 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.789 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.789 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:35.789 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.800080+0000", 2024-03-19T11:30:35.789 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029820699999999999 2024-03-19T11:30:35.789 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:35.789 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:35.789 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:35.789 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.789 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.789 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:871 4.d 4:b4ef97e0:::smithi03829856-3:head [write 642034~796262,stat] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:35.789 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:09.028192+0000", 2024-03-19T11:30:35.789 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 26.719826178000002, 2024-03-19T11:30:35.789 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013367027, 2024-03-19T11:30:35.789 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:35.790 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:35.790 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:35.790 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:35.790 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:35.790 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 871 2024-03-19T11:30:35.790 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.790 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:35.790 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.790 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:35.790 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.028192+0000", 2024-03-19T11:30:35.790 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:35.790 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.790 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.790 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:35.791 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.028192+0000", 2024-03-19T11:30:35.791 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:35.791 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.791 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.791 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:35.791 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.028194+0000", 2024-03-19T11:30:35.791 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3999999999999999e-06 2024-03-19T11:30:35.791 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.791 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.791 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:35.791 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.029358+0000", 2024-03-19T11:30:35.791 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011635549999999999 2024-03-19T11:30:35.791 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.791 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.791 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:35.792 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.029390+0000", 2024-03-19T11:30:35.792 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2215999999999998e-05 2024-03-19T11:30:35.792 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.792 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.792 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:35.792 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.029406+0000", 2024-03-19T11:30:35.792 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5543e-05 2024-03-19T11:30:35.792 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.792 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.792 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:35.792 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.029525+0000", 2024-03-19T11:30:35.792 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000119269 2024-03-19T11:30:35.792 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.792 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.793 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:35.793 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.029676+0000", 2024-03-19T11:30:35.793 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000150729 2024-03-19T11:30:35.793 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.793 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2", 2024-03-19T11:30:35.793 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.029994+0000", 2024-03-19T11:30:35.793 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00031876099999999998 2024-03-19T11:30:35.793 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.793 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:35.793 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.040441+0000", 2024-03-19T11:30:35.793 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010446446 2024-03-19T11:30:35.793 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.794 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.794 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:35.794 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.041327+0000", 2024-03-19T11:30:35.794 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00088595299999999998 2024-03-19T11:30:35.794 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.794 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.794 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:35.794 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.041409+0000", 2024-03-19T11:30:35.794 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2332999999999999e-05 2024-03-19T11:30:35.794 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.794 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.794 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:35.794 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.041559+0000", 2024-03-19T11:30:35.794 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000149822 2024-03-19T11:30:35.795 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:35.795 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:35.795 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:35.795 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.795 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:993 4.e 4:72163e4c:::smithi03829856-39:head [write 776490~714607] snapc 5=[5,2] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-03-19T11:30:35.795 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:12.362932+0000", 2024-03-19T11:30:35.795 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.385086484999999, 2024-03-19T11:30:35.795 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018349900999999998, 2024-03-19T11:30:35.795 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:35.795 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:35.795 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:35.795 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:35.795 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:35.796 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 993 2024-03-19T11:30:35.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.796 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:35.796 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.796 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:35.796 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.362932+0000", 2024-03-19T11:30:35.796 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:35.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.796 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.796 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:35.796 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.362932+0000", 2024-03-19T11:30:35.796 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:35.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.796 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.797 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:35.797 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.362935+0000", 2024-03-19T11:30:35.797 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.88e-06 2024-03-19T11:30:35.797 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.797 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.797 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:35.797 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.366756+0000", 2024-03-19T11:30:35.797 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003821217 2024-03-19T11:30:35.797 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.797 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.797 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:35.797 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.366796+0000", 2024-03-19T11:30:35.797 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9866000000000001e-05 2024-03-19T11:30:35.797 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.798 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.798 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:35.798 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.366813+0000", 2024-03-19T11:30:35.798 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.755e-05 2024-03-19T11:30:35.798 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.798 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.798 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:35.798 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.366993+0000", 2024-03-19T11:30:35.798 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000179605 2024-03-19T11:30:35.798 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.798 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.798 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:35.798 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.367167+0000", 2024-03-19T11:30:35.798 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017421200000000001 2024-03-19T11:30:35.799 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.799 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.799 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-19T11:30:35.799 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.367505+0000", 2024-03-19T11:30:35.799 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033800099999999999 2024-03-19T11:30:35.799 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.799 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.799 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:35.799 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.380361+0000", 2024-03-19T11:30:35.799 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012855961000000001 2024-03-19T11:30:35.799 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.799 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.799 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:35.799 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.380987+0000", 2024-03-19T11:30:35.799 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00062590499999999995 2024-03-19T11:30:35.800 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.800 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:35.800 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.381149+0000", 2024-03-19T11:30:35.800 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000162239 2024-03-19T11:30:35.800 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.800 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:35.800 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.381282+0000", 2024-03-19T11:30:35.800 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013246500000000001 2024-03-19T11:30:35.800 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:35.800 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:35.800 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:35.800 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.801 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.801 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(osd.1.14:42 3.3 3:f0b412b0:::85cff1f6132709ad3f6974c608c4ca3d22c7ae775eb2def51616b490c1e0306e:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:35.801 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:13.796997+0000", 2024-03-19T11:30:35.801 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 21.951021085000001, 2024-03-19T11:30:35.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012796138, 2024-03-19T11:30:35.801 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:35.801 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:35.801 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:35.801 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.801 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:35.801 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.796997+0000", 2024-03-19T11:30:35.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:35.801 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.801 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.802 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:35.802 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.796997+0000", 2024-03-19T11:30:35.802 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:35.802 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.802 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.802 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:35.802 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.796998+0000", 2024-03-19T11:30:35.802 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5999999999999991e-07 2024-03-19T11:30:35.802 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.802 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.802 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:35.802 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.797240+0000", 2024-03-19T11:30:35.802 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00024155800000000001 2024-03-19T11:30:35.802 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.803 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:35.803 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.797255+0000", 2024-03-19T11:30:35.803 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4983e-05 2024-03-19T11:30:35.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.803 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:35.803 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.797270+0000", 2024-03-19T11:30:35.803 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4953000000000001e-05 2024-03-19T11:30:35.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.803 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:35.803 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.805058+0000", 2024-03-19T11:30:35.803 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0077889200000000004 2024-03-19T11:30:35.804 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.804 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.804 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:35.804 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.805289+0000", 2024-03-19T11:30:35.804 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000230115 2024-03-19T11:30:35.804 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.804 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.804 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:35.804 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.805648+0000", 2024-03-19T11:30:35.804 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00035961400000000002 2024-03-19T11:30:35.804 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.804 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.804 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:35.804 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.809089+0000", 2024-03-19T11:30:35.804 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034412499999999999 2024-03-19T11:30:35.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.805 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.805 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:35.805 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.809502+0000", 2024-03-19T11:30:35.805 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00041262100000000001 2024-03-19T11:30:35.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.805 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.805 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:35.805 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.809645+0000", 2024-03-19T11:30:35.805 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000142659 2024-03-19T11:30:35.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.805 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.805 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:35.805 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.809793+0000", 2024-03-19T11:30:35.806 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000148505 2024-03-19T11:30:35.806 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:35.806 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:35.806 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:35.806 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.806 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.806 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1072 4.1 4:8401c1f0:::smithi03829856-43:head [write 494139~557703,stat] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:35.806 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:13.972007+0000", 2024-03-19T11:30:35.806 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 21.776010735, 2024-03-19T11:30:35.806 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.01935839, 2024-03-19T11:30:35.806 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:35.806 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:35.806 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:35.806 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:35.807 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:35.807 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1072 2024-03-19T11:30:35.807 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.807 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:35.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.807 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:35.807 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.972007+0000", 2024-03-19T11:30:35.807 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:35.807 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.807 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:35.807 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.972007+0000", 2024-03-19T11:30:35.807 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:35.807 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.808 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:35.808 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.972011+0000", 2024-03-19T11:30:35.808 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9360000000000001e-06 2024-03-19T11:30:35.808 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.808 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.808 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:35.808 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.972881+0000", 2024-03-19T11:30:35.808 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00086922699999999998 2024-03-19T11:30:35.808 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.808 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.808 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:35.808 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.972917+0000", 2024-03-19T11:30:35.808 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6359999999999997e-05 2024-03-19T11:30:35.809 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.809 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.809 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:35.809 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.972934+0000", 2024-03-19T11:30:35.809 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7249e-05 2024-03-19T11:30:35.809 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.809 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.809 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:35.809 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.973132+0000", 2024-03-19T11:30:35.809 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019760200000000001 2024-03-19T11:30:35.809 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.809 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.809 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:35.809 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.973303+0000", 2024-03-19T11:30:35.809 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017142899999999999 2024-03-19T11:30:35.810 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.810 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.810 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:35.810 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.973619+0000", 2024-03-19T11:30:35.810 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00031594700000000002 2024-03-19T11:30:35.810 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.810 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.810 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:35.810 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.985856+0000", 2024-03-19T11:30:35.810 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012236349000000001 2024-03-19T11:30:35.810 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.810 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.810 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:35.811 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.991132+0000", 2024-03-19T11:30:35.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052764960000000003 2024-03-19T11:30:35.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.811 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:35.811 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.991241+0000", 2024-03-19T11:30:35.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000108842 2024-03-19T11:30:35.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.811 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:35.811 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.991366+0000", 2024-03-19T11:30:35.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012495299999999999 2024-03-19T11:30:35.811 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:35.811 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:35.812 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:35.812 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.812 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.812 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1162 4.1 4:8401c1f0:::smithi03829856-43:head [write 594981~609197] snapc 8=[8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-03-19T11:30:35.812 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.838981+0000", 2024-03-19T11:30:35.812 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.909037380000001, 2024-03-19T11:30:35.812 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027704349999999999, 2024-03-19T11:30:35.812 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:35.812 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:35.812 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:35.812 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:35.812 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:35.812 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1162 2024-03-19T11:30:35.812 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.812 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:35.813 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.813 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:35.813 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.838981+0000", 2024-03-19T11:30:35.813 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:35.813 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.813 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.813 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:35.813 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.838981+0000", 2024-03-19T11:30:35.813 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:35.813 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.813 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.813 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:35.813 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.838983+0000", 2024-03-19T11:30:35.813 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5399999999999998e-06 2024-03-19T11:30:35.813 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.814 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.814 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:35.814 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841322+0000", 2024-03-19T11:30:35.814 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002338592 2024-03-19T11:30:35.814 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.814 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.814 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:35.814 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841371+0000", 2024-03-19T11:30:35.814 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9123999999999999e-05 2024-03-19T11:30:35.814 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.814 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.814 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:35.814 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841399+0000", 2024-03-19T11:30:35.814 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7718000000000001e-05 2024-03-19T11:30:35.815 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.815 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.815 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:35.815 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841577+0000", 2024-03-19T11:30:35.815 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000178097 2024-03-19T11:30:35.815 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.815 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.815 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:35.815 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841825+0000", 2024-03-19T11:30:35.815 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00024824299999999999 2024-03-19T11:30:35.815 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.815 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.815 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:35.815 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842227+0000", 2024-03-19T11:30:35.815 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00040212199999999998 2024-03-19T11:30:35.816 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:35.816 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.865182+0000", 2024-03-19T11:30:35.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022954810999999999 2024-03-19T11:30:35.816 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:35.816 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.866360+0000", 2024-03-19T11:30:35.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001178449 2024-03-19T11:30:35.816 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:35.816 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.866538+0000", 2024-03-19T11:30:35.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017791000000000001 2024-03-19T11:30:35.817 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.817 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.817 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:35.817 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.866685+0000", 2024-03-19T11:30:35.817 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000146744 2024-03-19T11:30:35.817 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:35.817 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:35.817 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:35.817 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.817 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.817 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1163 4.1 4:8401c1f0:::smithi03829856-43:head [write 1974123~745028] snapc 8=[8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-03-19T11:30:35.817 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.851879+0000", 2024-03-19T11:30:35.817 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.896139258000002, 2024-03-19T11:30:35.817 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015531774, 2024-03-19T11:30:35.817 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:35.818 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:35.818 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:35.818 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:35.818 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:35.818 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1163 2024-03-19T11:30:35.818 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.818 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:35.818 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.818 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:35.818 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.851879+0000", 2024-03-19T11:30:35.818 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:35.818 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.818 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:35.819 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.851879+0000", 2024-03-19T11:30:35.819 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:35.819 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.819 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:35.819 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.851882+0000", 2024-03-19T11:30:35.819 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6960000000000001e-06 2024-03-19T11:30:35.819 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.819 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:35.819 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.853204+0000", 2024-03-19T11:30:35.819 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0013226620000000001 2024-03-19T11:30:35.819 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.819 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.820 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:35.820 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.853231+0000", 2024-03-19T11:30:35.820 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6775e-05 2024-03-19T11:30:35.820 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.820 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.820 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:35.820 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.853246+0000", 2024-03-19T11:30:35.820 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5403000000000001e-05 2024-03-19T11:30:35.820 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.820 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.820 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:35.820 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.853363+0000", 2024-03-19T11:30:35.820 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000116296 2024-03-19T11:30:35.820 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.820 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.821 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:35.821 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.853572+0000", 2024-03-19T11:30:35.821 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020931500000000001 2024-03-19T11:30:35.821 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.821 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.821 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:35.821 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.853797+0000", 2024-03-19T11:30:35.821 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022544700000000001 2024-03-19T11:30:35.821 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.821 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.821 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:35.821 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.865195+0000", 2024-03-19T11:30:35.821 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011397981999999999 2024-03-19T11:30:35.821 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.822 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.822 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:35.822 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.866916+0000", 2024-03-19T11:30:35.822 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017204449999999999 2024-03-19T11:30:35.822 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.822 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.822 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:35.822 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.867129+0000", 2024-03-19T11:30:35.822 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021317999999999999 2024-03-19T11:30:35.822 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.822 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.822 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:35.822 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.867411+0000", 2024-03-19T11:30:35.822 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028157300000000002 2024-03-19T11:30:35.823 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:35.823 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:35.823 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:35.823 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.823 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.823 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1205 4.1 4:8401c1f0:::smithi03829856-43:head [assert-version v50,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:35.823 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:18.112028+0000", 2024-03-19T11:30:35.823 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 17.635990123999999, 2024-03-19T11:30:35.823 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013579206, 2024-03-19T11:30:35.823 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:35.823 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:35.823 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:35.823 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:35.824 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:35.824 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1205 2024-03-19T11:30:35.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.824 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:35.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.824 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:35.824 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112028+0000", 2024-03-19T11:30:35.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:35.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.824 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:35.824 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112028+0000", 2024-03-19T11:30:35.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:35.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.825 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:35.825 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112030+0000", 2024-03-19T11:30:35.825 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4549999999999998e-06 2024-03-19T11:30:35.825 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.825 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.825 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:35.825 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112043+0000", 2024-03-19T11:30:35.825 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2721999999999999e-05 2024-03-19T11:30:35.825 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.825 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.825 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:35.825 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112075+0000", 2024-03-19T11:30:35.825 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.205e-05 2024-03-19T11:30:35.825 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.825 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.826 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:35.826 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112091+0000", 2024-03-19T11:30:35.826 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5364000000000001e-05 2024-03-19T11:30:35.826 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.826 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.826 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:35.826 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112223+0000", 2024-03-19T11:30:35.826 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013224100000000001 2024-03-19T11:30:35.826 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.826 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.826 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:35.826 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112393+0000", 2024-03-19T11:30:35.826 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000169681 2024-03-19T11:30:35.826 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.827 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.827 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:35.827 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.120919+0000", 2024-03-19T11:30:35.827 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085260140000000002 2024-03-19T11:30:35.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.827 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.827 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:35.827 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125207+0000", 2024-03-19T11:30:35.827 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004288722 2024-03-19T11:30:35.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.827 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.827 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:35.827 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125343+0000", 2024-03-19T11:30:35.827 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013558400000000001 2024-03-19T11:30:35.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.828 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.828 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:35.828 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125497+0000", 2024-03-19T11:30:35.828 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000153678 2024-03-19T11:30:35.828 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.828 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.828 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:35.828 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125607+0000", 2024-03-19T11:30:35.828 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110695 2024-03-19T11:30:35.828 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:35.828 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:35.828 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:35.828 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.828 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.828 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1208 4.1 4:866f3dcc:::smithi03829856-31:head [read 0~2941192,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:35.829 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:18.112252+0000", 2024-03-19T11:30:35.829 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 17.635766554, 2024-03-19T11:30:35.829 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.01291702, 2024-03-19T11:30:35.829 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:35.829 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:35.829 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:35.829 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:35.829 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:35.829 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1208 2024-03-19T11:30:35.829 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.829 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:35.829 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.829 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:35.829 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112252+0000", 2024-03-19T11:30:35.829 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:35.830 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.830 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.830 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:35.830 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112252+0000", 2024-03-19T11:30:35.830 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:35.830 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.830 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.830 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:35.830 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112253+0000", 2024-03-19T11:30:35.830 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6299999999999993e-07 2024-03-19T11:30:35.830 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.830 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.830 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:35.830 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112257+0000", 2024-03-19T11:30:35.830 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4360000000000004e-06 2024-03-19T11:30:35.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.831 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:35.831 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112272+0000", 2024-03-19T11:30:35.831 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.518e-05 2024-03-19T11:30:35.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.831 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:35.831 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112281+0000", 2024-03-19T11:30:35.831 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.422e-06 2024-03-19T11:30:35.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.831 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:35.831 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.122861+0000", 2024-03-19T11:30:35.831 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010580523 2024-03-19T11:30:35.832 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.832 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:35.832 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123148+0000", 2024-03-19T11:30:35.832 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028714499999999999 2024-03-19T11:30:35.832 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.832 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:35.832 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125169+0000", 2024-03-19T11:30:35.832 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0020203510000000001 2024-03-19T11:30:35.832 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:35.832 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:35.832 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:35.832 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.833 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1243 4.9 4:9ef41243:::smithi03829856-5:head [tier-promote] snapc 9=[9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:35.833 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:18.345759+0000", 2024-03-19T11:30:35.833 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 17.402258943, 2024-03-19T11:30:35.833 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018400962999999999, 2024-03-19T11:30:35.833 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:35.833 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:35.833 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:35.833 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:35.833 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:35.833 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1243 2024-03-19T11:30:35.833 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.833 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:35.833 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.833 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:35.833 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.345759+0000", 2024-03-19T11:30:35.834 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:35.834 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.834 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.834 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:35.834 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.345759+0000", 2024-03-19T11:30:35.834 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:35.834 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.834 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.834 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:35.834 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.345763+0000", 2024-03-19T11:30:35.834 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3790000000000002e-06 2024-03-19T11:30:35.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.837 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:35.838 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.345776+0000", 2024-03-19T11:30:35.838 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3784000000000001e-05 2024-03-19T11:30:35.838 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.838 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.838 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:35.838 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.345812+0000", 2024-03-19T11:30:35.838 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5599999999999998e-05 2024-03-19T11:30:35.838 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.838 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.838 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:35.838 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.345829+0000", 2024-03-19T11:30:35.838 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6722999999999999e-05 2024-03-19T11:30:35.838 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.838 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.839 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:35.839 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.345964+0000", 2024-03-19T11:30:35.839 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013554800000000001 2024-03-19T11:30:35.839 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.839 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:35.839 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.346133+0000", 2024-03-19T11:30:35.839 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000169211 2024-03-19T11:30:35.839 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.839 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:35.839 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.364160+0000", 2024-03-19T11:30:35.839 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018026718000000001 2024-03-19T11:30:35.839 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:35.840 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:35.840 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:35.840 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.840 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.840 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1271 4.9 4:953e720b:::smithi03829856-45:2 [read 795136~1008128] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-03-19T11:30:35.840 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:21.868056+0000", 2024-03-19T11:30:35.840 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 13.879961874999999, 2024-03-19T11:30:35.840 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.047270591000000001, 2024-03-19T11:30:35.840 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:35.840 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:35.840 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:35.840 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:35.840 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:35.840 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1271 2024-03-19T11:30:35.840 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.841 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:35.841 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.841 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:35.841 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868056+0000", 2024-03-19T11:30:35.841 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:35.841 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.841 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.841 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:35.841 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868056+0000", 2024-03-19T11:30:35.841 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:35.841 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.841 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.841 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:35.841 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868059+0000", 2024-03-19T11:30:35.841 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0469999999999998e-06 2024-03-19T11:30:35.842 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.842 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.842 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:35.842 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868072+0000", 2024-03-19T11:30:35.842 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2700999999999999e-05 2024-03-19T11:30:35.842 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.842 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.842 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:35.842 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868106+0000", 2024-03-19T11:30:35.842 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4152000000000001e-05 2024-03-19T11:30:35.842 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.842 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.842 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:35.842 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868126+0000", 2024-03-19T11:30:35.843 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9386999999999999e-05 2024-03-19T11:30:35.843 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.843 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:35.843 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.911447+0000", 2024-03-19T11:30:35.843 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.043321164000000002 2024-03-19T11:30:35.843 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.843 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:35.843 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.911762+0000", 2024-03-19T11:30:35.843 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00031492400000000001 2024-03-19T11:30:35.843 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.843 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:35.843 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.915327+0000", 2024-03-19T11:30:35.844 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003565216 2024-03-19T11:30:35.844 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:35.844 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:35.844 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:35.844 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.844 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.844 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1617 4.e 4:7773c291:::smithi03829856-11:head [write 450454~642026,stat] snapc 10=[10,f,e,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-03-19T11:30:35.844 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:35.152724+0000", 2024-03-19T11:30:35.844 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.59529368100000002, 2024-03-19T11:30:35.844 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016139337, 2024-03-19T11:30:35.844 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:35.844 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:35.844 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:35.844 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:35.844 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:35.845 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1617 2024-03-19T11:30:35.845 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.845 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:35.845 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.845 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:35.845 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.152724+0000", 2024-03-19T11:30:35.845 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:35.845 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.845 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.845 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:35.845 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.152724+0000", 2024-03-19T11:30:35.845 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:35.845 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.845 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.845 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:35.846 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.152727+0000", 2024-03-19T11:30:35.846 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8399999999999999e-06 2024-03-19T11:30:35.846 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.846 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.846 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:35.846 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.153790+0000", 2024-03-19T11:30:35.846 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010623849999999999 2024-03-19T11:30:35.846 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.846 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.846 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:35.846 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.153825+0000", 2024-03-19T11:30:35.846 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5280000000000001e-05 2024-03-19T11:30:35.846 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.846 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.847 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:35.847 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.153842+0000", 2024-03-19T11:30:35.847 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6942000000000001e-05 2024-03-19T11:30:35.847 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.847 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.847 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:35.847 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.154018+0000", 2024-03-19T11:30:35.847 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017568 2024-03-19T11:30:35.847 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.847 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.847 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:35.847 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.154302+0000", 2024-03-19T11:30:35.847 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028463400000000003 2024-03-19T11:30:35.847 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.847 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.848 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-19T11:30:35.848 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.154811+0000", 2024-03-19T11:30:35.848 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00050833300000000003 2024-03-19T11:30:35.848 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.848 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.848 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:35.848 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.168286+0000", 2024-03-19T11:30:35.848 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013475354 2024-03-19T11:30:35.848 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.848 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.848 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:35.848 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.168631+0000", 2024-03-19T11:30:35.848 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00034488800000000001 2024-03-19T11:30:35.848 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.849 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.849 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:35.849 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.168749+0000", 2024-03-19T11:30:35.849 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011799900000000001 2024-03-19T11:30:35.849 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:35.849 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:35.849 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:35.849 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.168864+0000", 2024-03-19T11:30:35.849 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011500199999999999 2024-03-19T11:30:35.849 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:35.849 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:35.849 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:35.849 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:35.849 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:35.849 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:35.851 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-19T11:30:35.852 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:35.863+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:35.952 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:35.961+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:35.976 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:35.977 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:35.977 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:30:35.977 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:35.990 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-19T11:30:36.053 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:36.063+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:36.109 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:36.119+0000 7fa5e08db640 -1 osd.3 58 handle_osd_map injecting map crc failure 2024-03-19T11:30:36.113 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-03-19T11:30:36.115 INFO:tasks.rados.rados.0.smithi038.stdout:909: done (3 left) 2024-03-19T11:30:36.115 INFO:tasks.rados.rados.0.smithi038.stdout:910: read oid 32 snap -1 2024-03-19T11:30:36.115 INFO:tasks.rados.rados.0.smithi038.stdout:910: expect (ObjNum 31 snap 0 seq_num 31) 2024-03-19T11:30:36.115 INFO:tasks.rados.rados.0.smithi038.stdout:906: finishing write tid 1 to smithi03829856-14 2024-03-19T11:30:36.115 INFO:tasks.rados.rados.0.smithi038.stdout:906: finishing write tid 2 to smithi03829856-14 2024-03-19T11:30:36.115 INFO:tasks.rados.rados.0.smithi038.stdout:906: finishing write tid 3 to smithi03829856-14 2024-03-19T11:30:36.115 INFO:tasks.rados.rados.0.smithi038.stdout:906: finishing write tid 4 to smithi03829856-14 2024-03-19T11:30:36.115 INFO:tasks.rados.rados.0.smithi038.stdout:906: oid 14 updating version 0 to 38 2024-03-19T11:30:36.115 INFO:tasks.rados.rados.0.smithi038.stdout:906: oid 14 version 38 is already newer than 37 2024-03-19T11:30:36.115 INFO:tasks.rados.rados.0.smithi038.stdout:906: oid 14 version 38 is already newer than 36 2024-03-19T11:30:36.116 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 14 v 38 (ObjNum 124 snap 15 seq_num 124) dirty exists 2024-03-19T11:30:36.116 INFO:tasks.rados.rados.0.smithi038.stdout:906: left oid 14 (ObjNum 124 snap 15 seq_num 124) 2024-03-19T11:30:36.116 INFO:tasks.rados.rados.0.smithi038.stdout:907: expect (ObjNum 19 snap 0 seq_num 19) 2024-03-19T11:30:36.154 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:36.165+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:36.163 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:36.164 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:36.164 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:30:36.164 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:30:36.164 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:36.177 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-19T11:30:36.179 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:36.189+0000 7f923fa20640 -1 osd.7 58 handle_osd_map injecting map crc failure 2024-03-19T11:30:36.194 INFO:tasks.rados.rados.0.smithi038.stdout:908: expect (ObjNum 32 snap 0 seq_num 32) 2024-03-19T11:30:36.255 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:36.265+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:36.276 INFO:tasks.rados.rados.0.smithi038.stdout:910: expect (ObjNum 31 snap 0 seq_num 31) 2024-03-19T11:30:36.302 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:36.303 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:30:36.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:30:36.303 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:30:36.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.303 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:844 4.7 e32/21 4:e41d47fb:::smithi03829856-14:head v 32'8, mlcod=28'5)", 2024-03-19T11:30:36.303 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.751447+0000", 2024-03-19T11:30:36.303 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.562709299000002, 2024-03-19T11:30:36.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012301835000000001, 2024-03-19T11:30:36.303 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:36.303 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:36.303 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:36.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:36.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.751447+0000", 2024-03-19T11:30:36.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:36.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.751447+0000", 2024-03-19T11:30:36.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:36.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.751447+0000", 2024-03-19T11:30:36.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4300000000000003e-07 2024-03-19T11:30:36.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:36.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.752747+0000", 2024-03-19T11:30:36.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012995699999999999 2024-03-19T11:30:36.305 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.305 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:36.305 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.752771+0000", 2024-03-19T11:30:36.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3592999999999999e-05 2024-03-19T11:30:36.305 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.305 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:36.305 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.752789+0000", 2024-03-19T11:30:36.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8703e-05 2024-03-19T11:30:36.305 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:36.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.752911+0000", 2024-03-19T11:30:36.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012139899999999999 2024-03-19T11:30:36.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:36.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.752949+0000", 2024-03-19T11:30:36.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8436999999999998e-05 2024-03-19T11:30:36.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:36.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763568+0000", 2024-03-19T11:30:36.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010618768000000001 2024-03-19T11:30:36.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:36.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763749+0000", 2024-03-19T11:30:36.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000180822 2024-03-19T11:30:36.307 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.307 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:36.307 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.307 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1047 4.8 4:1d58db11:::smithi03829856-18:head [write 443718~615603,stat] snapc 6=[6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:36.307 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:13.221766+0000", 2024-03-19T11:30:36.307 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.092390131999998, 2024-03-19T11:30:36.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020932596000000001, 2024-03-19T11:30:36.308 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:36.308 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:36.308 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:36.308 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:36.308 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:36.308 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1047 2024-03-19T11:30:36.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.308 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:36.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:36.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.221766+0000", 2024-03-19T11:30:36.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.309 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:36.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.221766+0000", 2024-03-19T11:30:36.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.309 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:36.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.221770+0000", 2024-03-19T11:30:36.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.67e-06 2024-03-19T11:30:36.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.309 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:36.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.223938+0000", 2024-03-19T11:30:36.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021683399999999999 2024-03-19T11:30:36.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:36.310 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.223979+0000", 2024-03-19T11:30:36.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1433000000000001e-05 2024-03-19T11:30:36.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:36.310 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.223998+0000", 2024-03-19T11:30:36.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8853000000000001e-05 2024-03-19T11:30:36.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:36.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.224150+0000", 2024-03-19T11:30:36.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000151679 2024-03-19T11:30:36.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:36.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.224362+0000", 2024-03-19T11:30:36.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021154499999999999 2024-03-19T11:30:36.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:36.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.224745+0000", 2024-03-19T11:30:36.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00038342399999999999 2024-03-19T11:30:36.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:36.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.241167+0000", 2024-03-19T11:30:36.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0164218 2024-03-19T11:30:36.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:36.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.242406+0000", 2024-03-19T11:30:36.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012390039999999999 2024-03-19T11:30:36.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:36.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.242575+0000", 2024-03-19T11:30:36.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000168925 2024-03-19T11:30:36.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:36.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.242699+0000", 2024-03-19T11:30:36.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012392300000000001 2024-03-19T11:30:36.313 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.313 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:36.314 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.314 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1048 4.8 4:1d58db11:::smithi03829856-18:head [write 1564138~470341] snapc 6=[6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:36.314 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:13.227287+0000", 2024-03-19T11:30:36.314 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.086868863999999, 2024-03-19T11:30:36.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015802897999999999, 2024-03-19T11:30:36.314 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:36.314 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:36.314 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:36.314 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:36.314 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:36.314 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1048 2024-03-19T11:30:36.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.314 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:36.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:36.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.227287+0000", 2024-03-19T11:30:36.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:36.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.227287+0000", 2024-03-19T11:30:36.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:36.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.227290+0000", 2024-03-19T11:30:36.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.88e-06 2024-03-19T11:30:36.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:36.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.228224+0000", 2024-03-19T11:30:36.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00093389600000000003 2024-03-19T11:30:36.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:36.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.228258+0000", 2024-03-19T11:30:36.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4282999999999997e-05 2024-03-19T11:30:36.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:36.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.228273+0000", 2024-03-19T11:30:36.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4833e-05 2024-03-19T11:30:36.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:36.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.231996+0000", 2024-03-19T11:30:36.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037223410000000001 2024-03-19T11:30:36.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.318 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:36.318 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.232143+0000", 2024-03-19T11:30:36.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014714900000000001 2024-03-19T11:30:36.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.318 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:36.318 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.232395+0000", 2024-03-19T11:30:36.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025201500000000002 2024-03-19T11:30:36.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.318 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:36.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.242848+0000", 2024-03-19T11:30:36.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010453393 2024-03-19T11:30:36.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:36.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.242872+0000", 2024-03-19T11:30:36.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3816e-05 2024-03-19T11:30:36.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:36.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.242966+0000", 2024-03-19T11:30:36.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4560000000000003e-05 2024-03-19T11:30:36.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:36.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.243090+0000", 2024-03-19T11:30:36.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012373199999999999 2024-03-19T11:30:36.320 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.320 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:36.320 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.321 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1072 4.1 e36/21 4:8401c1f0:::smithi03829856-43:head v 36'38, mlcod=36'35)", 2024-03-19T11:30:36.321 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:13.973829+0000", 2024-03-19T11:30:36.321 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.340327541000001, 2024-03-19T11:30:36.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013548782000000001, 2024-03-19T11:30:36.321 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:36.321 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:36.321 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:36.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:36.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.973829+0000", 2024-03-19T11:30:36.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:36.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.973829+0000", 2024-03-19T11:30:36.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:36.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.973831+0000", 2024-03-19T11:30:36.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0899999999999999e-06 2024-03-19T11:30:36.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:36.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.974489+0000", 2024-03-19T11:30:36.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00065813199999999997 2024-03-19T11:30:36.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.323 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:36.323 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.974523+0000", 2024-03-19T11:30:36.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3841999999999999e-05 2024-03-19T11:30:36.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.323 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:36.323 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.974544+0000", 2024-03-19T11:30:36.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0907000000000002e-05 2024-03-19T11:30:36.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.323 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:36.323 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.974697+0000", 2024-03-19T11:30:36.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015295599999999999 2024-03-19T11:30:36.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:36.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.974742+0000", 2024-03-19T11:30:36.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5735999999999997e-05 2024-03-19T11:30:36.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:36.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.987261+0000", 2024-03-19T11:30:36.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012518316999999999 2024-03-19T11:30:36.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:36.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.987377+0000", 2024-03-19T11:30:36.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000116802 2024-03-19T11:30:36.324 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.325 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:36.325 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.325 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1091 4.8 4:14611006:::smithi03829856-25:head [write 527753~708201] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:36.325 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:14.015839+0000", 2024-03-19T11:30:36.325 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.298317482000002, 2024-03-19T11:30:36.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016721983999999999, 2024-03-19T11:30:36.325 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:36.325 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:36.325 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:36.325 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:36.325 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:36.326 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1091 2024-03-19T11:30:36.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.326 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:36.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:36.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.015839+0000", 2024-03-19T11:30:36.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:36.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.015839+0000", 2024-03-19T11:30:36.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:36.327 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.015842+0000", 2024-03-19T11:30:36.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3569999999999998e-06 2024-03-19T11:30:36.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:36.327 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.016678+0000", 2024-03-19T11:30:36.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00083620299999999999 2024-03-19T11:30:36.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:36.327 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.016710+0000", 2024-03-19T11:30:36.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1337000000000002e-05 2024-03-19T11:30:36.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:36.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.016724+0000", 2024-03-19T11:30:36.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4323e-05 2024-03-19T11:30:36.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:36.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.016847+0000", 2024-03-19T11:30:36.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012304199999999999 2024-03-19T11:30:36.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:36.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.017023+0000", 2024-03-19T11:30:36.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000175945 2024-03-19T11:30:36.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:36.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.017324+0000", 2024-03-19T11:30:36.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00030151499999999998 2024-03-19T11:30:36.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:36.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.032104+0000", 2024-03-19T11:30:36.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014779232999999999 2024-03-19T11:30:36.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:36.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.032190+0000", 2024-03-19T11:30:36.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6304999999999994e-05 2024-03-19T11:30:36.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:36.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.032350+0000", 2024-03-19T11:30:36.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000159813 2024-03-19T11:30:36.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:36.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.032561+0000", 2024-03-19T11:30:36.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021091099999999999 2024-03-19T11:30:36.330 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.331 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:36.331 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.331 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1158 4.8 4:15708482:::smithi03829856-23:head [tier-promote] snapc 8=[8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-03-19T11:30:36.331 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.827517+0000", 2024-03-19T11:30:36.331 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.486639554, 2024-03-19T11:30:36.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.034096983999999997, 2024-03-19T11:30:36.331 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:36.331 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:36.332 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:36.332 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:36.332 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:36.332 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1158 2024-03-19T11:30:36.332 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.332 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:36.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.332 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:36.332 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827517+0000", 2024-03-19T11:30:36.332 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.332 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.332 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:36.332 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827517+0000", 2024-03-19T11:30:36.332 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:36.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827520+0000", 2024-03-19T11:30:36.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.45e-06 2024-03-19T11:30:36.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:36.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827535+0000", 2024-03-19T11:30:36.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.509e-05 2024-03-19T11:30:36.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:36.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827579+0000", 2024-03-19T11:30:36.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4211000000000001e-05 2024-03-19T11:30:36.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:36.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827603+0000", 2024-03-19T11:30:36.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3586000000000001e-05 2024-03-19T11:30:36.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.335 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:36.335 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827789+0000", 2024-03-19T11:30:36.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018580199999999999 2024-03-19T11:30:36.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.335 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:36.335 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.828077+0000", 2024-03-19T11:30:36.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028846500000000001 2024-03-19T11:30:36.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.335 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:36.335 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.861614+0000", 2024-03-19T11:30:36.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033536379999999998 2024-03-19T11:30:36.336 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.336 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:36.336 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.336 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1162 4.1 e39/21 4:8401c1f0:::smithi03829856-43:head v 39'47, mlcod=38'43)", 2024-03-19T11:30:36.336 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.842598+0000", 2024-03-19T11:30:36.336 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.471557784000002, 2024-03-19T11:30:36.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0233756, 2024-03-19T11:30:36.336 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:36.336 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:36.336 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:36.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:36.337 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842598+0000", 2024-03-19T11:30:36.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:36.337 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842598+0000", 2024-03-19T11:30:36.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:36.337 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842599+0000", 2024-03-19T11:30:36.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6000000000000003e-07 2024-03-19T11:30:36.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.338 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:36.338 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.843566+0000", 2024-03-19T11:30:36.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00096688700000000004 2024-03-19T11:30:36.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.338 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.338 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:36.338 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.843601+0000", 2024-03-19T11:30:36.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.502e-05 2024-03-19T11:30:36.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.338 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.338 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:36.338 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.843621+0000", 2024-03-19T11:30:36.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0305e-05 2024-03-19T11:30:36.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:36.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.843882+0000", 2024-03-19T11:30:36.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026027100000000002 2024-03-19T11:30:36.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:36.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.844022+0000", 2024-03-19T11:30:36.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014020100000000001 2024-03-19T11:30:36.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:36.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.865862+0000", 2024-03-19T11:30:36.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021839725000000001 2024-03-19T11:30:36.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.340 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:36.340 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.865974+0000", 2024-03-19T11:30:36.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011243099999999999 2024-03-19T11:30:36.340 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.340 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:36.340 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.340 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1209 4.8 4:1d58db11:::smithi03829856-18:head [write 645402~551297,stat] snapc 9=[9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:36.340 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:18.121657+0000", 2024-03-19T11:30:36.340 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.192498782000001, 2024-03-19T11:30:36.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.025074349999999999, 2024-03-19T11:30:36.340 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:36.340 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:36.340 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:36.341 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:36.341 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:36.341 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1209 2024-03-19T11:30:36.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.341 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:36.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.341 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:36.341 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.121657+0000", 2024-03-19T11:30:36.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.342 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:36.342 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.121657+0000", 2024-03-19T11:30:36.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.342 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:36.342 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.121661+0000", 2024-03-19T11:30:36.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0489999999999997e-06 2024-03-19T11:30:36.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:36.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123747+0000", 2024-03-19T11:30:36.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0020857720000000001 2024-03-19T11:30:36.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:36.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123779+0000", 2024-03-19T11:30:36.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2258000000000002e-05 2024-03-19T11:30:36.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:36.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123795+0000", 2024-03-19T11:30:36.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5177000000000001e-05 2024-03-19T11:30:36.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:36.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123949+0000", 2024-03-19T11:30:36.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015403699999999999 2024-03-19T11:30:36.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:36.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.124134+0000", 2024-03-19T11:30:36.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000185484 2024-03-19T11:30:36.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:36.345 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.124554+0000", 2024-03-19T11:30:36.345 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00042019699999999998 2024-03-19T11:30:36.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.345 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.345 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:36.345 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146267+0000", 2024-03-19T11:30:36.345 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021712838000000002 2024-03-19T11:30:36.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.345 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.345 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:36.345 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146424+0000", 2024-03-19T11:30:36.345 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015647299999999999 2024-03-19T11:30:36.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.345 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.345 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:36.345 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146597+0000", 2024-03-19T11:30:36.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017303699999999999 2024-03-19T11:30:36.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:36.346 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146732+0000", 2024-03-19T11:30:36.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013502800000000001 2024-03-19T11:30:36.346 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.346 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:36.346 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.346 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1210 4.8 4:1d58db11:::smithi03829856-18:head [write 1749569~492355] snapc 9=[9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:36.346 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:18.129257+0000", 2024-03-19T11:30:36.346 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.184899425000001, 2024-03-19T11:30:36.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018110444999999999, 2024-03-19T11:30:36.347 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:36.347 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:36.347 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:36.347 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:36.347 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:36.347 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1210 2024-03-19T11:30:36.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.347 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:36.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:36.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.129257+0000", 2024-03-19T11:30:36.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:36.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.129257+0000", 2024-03-19T11:30:36.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:36.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.129261+0000", 2024-03-19T11:30:36.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7119999999999999e-06 2024-03-19T11:30:36.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.349 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.349 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:36.349 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.129717+0000", 2024-03-19T11:30:36.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00045526300000000002 2024-03-19T11:30:36.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.349 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.349 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:36.349 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.129749+0000", 2024-03-19T11:30:36.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2700999999999997e-05 2024-03-19T11:30:36.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.349 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.349 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:36.349 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.129774+0000", 2024-03-19T11:30:36.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4592000000000001e-05 2024-03-19T11:30:36.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.350 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:36.350 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.134875+0000", 2024-03-19T11:30:36.350 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.005100477 2024-03-19T11:30:36.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.350 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:36.350 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.135035+0000", 2024-03-19T11:30:36.350 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016054200000000001 2024-03-19T11:30:36.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.350 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:36.350 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.135268+0000", 2024-03-19T11:30:36.350 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000233253 2024-03-19T11:30:36.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:36.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146872+0000", 2024-03-19T11:30:36.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011603732 2024-03-19T11:30:36.351 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:36.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.147103+0000", 2024-03-19T11:30:36.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023070299999999999 2024-03-19T11:30:36.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.352 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.352 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:36.352 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.147181+0000", 2024-03-19T11:30:36.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8567999999999997e-05 2024-03-19T11:30:36.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.352 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:36.353 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.147367+0000", 2024-03-19T11:30:36.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000185902 2024-03-19T11:30:36.353 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.353 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:36.353 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.353 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1259 4.1 e44/43 4:866f3dcc:::smithi03829856-31:head v 44'53, mlcod=0'0)", 2024-03-19T11:30:36.354 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:21.869109+0000", 2024-03-19T11:30:36.354 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.445047535, 2024-03-19T11:30:36.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014777017999999999, 2024-03-19T11:30:36.354 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:36.354 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:36.354 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:36.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.354 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:36.354 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.869109+0000", 2024-03-19T11:30:36.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.354 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:36.354 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.869109+0000", 2024-03-19T11:30:36.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.355 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:36.355 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.869110+0000", 2024-03-19T11:30:36.355 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0559999999999999e-06 2024-03-19T11:30:36.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.355 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:36.355 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.870408+0000", 2024-03-19T11:30:36.355 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012978499999999999 2024-03-19T11:30:36.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.355 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:36.355 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.870437+0000", 2024-03-19T11:30:36.355 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9638999999999999e-05 2024-03-19T11:30:36.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.356 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:36.356 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.870460+0000", 2024-03-19T11:30:36.356 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2517000000000001e-05 2024-03-19T11:30:36.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.356 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:36.356 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.870681+0000", 2024-03-19T11:30:36.356 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022102800000000001 2024-03-19T11:30:36.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.356 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:36.356 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.870735+0000", 2024-03-19T11:30:36.356 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4616e-05 2024-03-19T11:30:36.357 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.357 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:36.357 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.883773+0000", 2024-03-19T11:30:36.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013038088 2024-03-19T11:30:36.357 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.357 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:36.357 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.883886+0000", 2024-03-19T11:30:36.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000112224 2024-03-19T11:30:36.357 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.357 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:36.357 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.357 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.358 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1416 4.18 4:1dabc0f1:::smithi03829856-1:head [write 564735~613734,stat] snapc c=[c,b,a,6,5] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:36.358 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.949508+0000", 2024-03-19T11:30:36.358 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.364648205, 2024-03-19T11:30:36.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016253106, 2024-03-19T11:30:36.358 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:36.358 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:36.358 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:36.358 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:36.358 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:36.358 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1416 2024-03-19T11:30:36.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.359 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:36.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:36.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.949508+0000", 2024-03-19T11:30:36.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:36.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.949508+0000", 2024-03-19T11:30:36.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:36.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.949511+0000", 2024-03-19T11:30:36.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2370000000000001e-06 2024-03-19T11:30:36.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:36.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.950555+0000", 2024-03-19T11:30:36.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001043925 2024-03-19T11:30:36.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:36.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.950594+0000", 2024-03-19T11:30:36.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9351999999999999e-05 2024-03-19T11:30:36.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:36.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.950612+0000", 2024-03-19T11:30:36.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7524e-05 2024-03-19T11:30:36.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:36.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.950745+0000", 2024-03-19T11:30:36.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013291 2024-03-19T11:30:36.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:36.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.951003+0000", 2024-03-19T11:30:36.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025828400000000001 2024-03-19T11:30:36.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:36.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.951335+0000", 2024-03-19T11:30:36.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033130200000000001 2024-03-19T11:30:36.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.362 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:36.362 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.965379+0000", 2024-03-19T11:30:36.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014044581 2024-03-19T11:30:36.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.362 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:36.362 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.965502+0000", 2024-03-19T11:30:36.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012290099999999999 2024-03-19T11:30:36.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.363 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:36.363 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.965668+0000", 2024-03-19T11:30:36.363 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016619200000000001 2024-03-19T11:30:36.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.363 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.363 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:36.363 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.965761+0000", 2024-03-19T11:30:36.363 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2898000000000004e-05 2024-03-19T11:30:36.363 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.363 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:36.363 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.363 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.363 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1417 4.18 4:1dabc0f1:::smithi03829856-1:head [write 1828021~669034] snapc c=[c,b,a,6,5] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:36.364 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.958407+0000", 2024-03-19T11:30:36.364 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.355748803999999, 2024-03-19T11:30:36.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016467840000000001, 2024-03-19T11:30:36.364 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:36.364 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:36.364 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:36.364 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:36.364 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:36.364 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1417 2024-03-19T11:30:36.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.364 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:36.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.364 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:36.364 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.958407+0000", 2024-03-19T11:30:36.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:36.365 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.958407+0000", 2024-03-19T11:30:36.365 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:36.365 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.958410+0000", 2024-03-19T11:30:36.365 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9830000000000001e-06 2024-03-19T11:30:36.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:36.365 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.959442+0000", 2024-03-19T11:30:36.365 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001031957 2024-03-19T11:30:36.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.366 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:36.366 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.959475+0000", 2024-03-19T11:30:36.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3145999999999997e-05 2024-03-19T11:30:36.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.366 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:36.366 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.959494+0000", 2024-03-19T11:30:36.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8994000000000001e-05 2024-03-19T11:30:36.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.366 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:36.366 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.959717+0000", 2024-03-19T11:30:36.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000222272 2024-03-19T11:30:36.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.367 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:36.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.960015+0000", 2024-03-19T11:30:36.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029870599999999998 2024-03-19T11:30:36.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.367 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:36.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.960312+0000", 2024-03-19T11:30:36.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029640299999999999 2024-03-19T11:30:36.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.367 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:36.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.966435+0000", 2024-03-19T11:30:36.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061234999999999996 2024-03-19T11:30:36.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.368 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:36.368 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.974556+0000", 2024-03-19T11:30:36.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.008120759 2024-03-19T11:30:36.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.368 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:36.368 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.974729+0000", 2024-03-19T11:30:36.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000173191 2024-03-19T11:30:36.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.368 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:36.368 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.974875+0000", 2024-03-19T11:30:36.369 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014592899999999999 2024-03-19T11:30:36.369 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.369 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:36.369 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.369 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.369 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.369 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1419 4.18 4:1dabc0f1:::smithi03829856-1:head [setxattr _header (56) in=7b,truncate 3764735] snapc c=[c,b,a,6,5] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:36.369 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.968061+0000", 2024-03-19T11:30:36.369 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.346094945000001, 2024-03-19T11:30:36.369 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012422488000000001, 2024-03-19T11:30:36.369 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:36.369 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:36.369 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:36.369 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:36.369 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:36.370 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1419 2024-03-19T11:30:36.370 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.370 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:36.370 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.370 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:36.370 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968061+0000", 2024-03-19T11:30:36.370 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.370 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.370 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.370 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:36.370 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968061+0000", 2024-03-19T11:30:36.370 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.370 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.370 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.371 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:36.371 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968063+0000", 2024-03-19T11:30:36.371 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.767e-06 2024-03-19T11:30:36.371 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.371 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.371 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:36.371 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968074+0000", 2024-03-19T11:30:36.371 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0525e-05 2024-03-19T11:30:36.371 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.371 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.371 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:36.371 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968096+0000", 2024-03-19T11:30:36.371 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2926999999999999e-05 2024-03-19T11:30:36.371 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.371 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.371 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:36.372 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968113+0000", 2024-03-19T11:30:36.372 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6665000000000001e-05 2024-03-19T11:30:36.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.372 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:36.372 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.973218+0000", 2024-03-19T11:30:36.372 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0051044920000000004 2024-03-19T11:30:36.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.372 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:36.372 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.973511+0000", 2024-03-19T11:30:36.372 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029303899999999998 2024-03-19T11:30:36.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.372 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:36.373 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.973931+0000", 2024-03-19T11:30:36.373 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00041991400000000002 2024-03-19T11:30:36.373 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.373 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.373 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:36.373 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.977029+0000", 2024-03-19T11:30:36.373 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030983040000000001 2024-03-19T11:30:36.373 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.373 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.373 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:36.373 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.980091+0000", 2024-03-19T11:30:36.373 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003061851 2024-03-19T11:30:36.373 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.373 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.373 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:36.373 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.980285+0000", 2024-03-19T11:30:36.374 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000194279 2024-03-19T11:30:36.374 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.374 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:36.374 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.980484+0000", 2024-03-19T11:30:36.374 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019872499999999999 2024-03-19T11:30:36.374 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.374 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:36.374 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.374 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.374 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1420 4.18 4:1dabc0f1:::smithi03829856-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:36.374 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.968138+0000", 2024-03-19T11:30:36.374 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.346017735, 2024-03-19T11:30:36.375 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013109305999999999, 2024-03-19T11:30:36.375 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:36.375 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:36.375 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:36.375 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:36.375 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:36.375 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1420 2024-03-19T11:30:36.375 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.375 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:36.375 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.375 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:36.375 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968138+0000", 2024-03-19T11:30:36.375 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.375 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.375 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.376 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:36.376 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968138+0000", 2024-03-19T11:30:36.376 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.376 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.376 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.376 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:36.376 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968140+0000", 2024-03-19T11:30:36.376 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4139999999999999e-06 2024-03-19T11:30:36.376 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.376 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.376 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:36.376 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968147+0000", 2024-03-19T11:30:36.376 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.815e-06 2024-03-19T11:30:36.376 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.376 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.377 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:36.377 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968167+0000", 2024-03-19T11:30:36.377 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0018e-05 2024-03-19T11:30:36.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.377 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.377 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:36.377 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968182+0000", 2024-03-19T11:30:36.377 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5475e-05 2024-03-19T11:30:36.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.377 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.377 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:36.377 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.974950+0000", 2024-03-19T11:30:36.377 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.006767856 2024-03-19T11:30:36.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.377 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.378 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-03-19T11:30:36.378 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.975095+0000", 2024-03-19T11:30:36.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014501499999999999 2024-03-19T11:30:36.378 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.378 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.378 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:36.378 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.980614+0000", 2024-03-19T11:30:36.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0055184789999999997 2024-03-19T11:30:36.378 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.378 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.378 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:36.378 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.980912+0000", 2024-03-19T11:30:36.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029858199999999998 2024-03-19T11:30:36.378 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.378 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.379 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:36.379 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.981248+0000", 2024-03-19T11:30:36.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033565199999999998 2024-03-19T11:30:36.379 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.379 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:36.379 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.379 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.379 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.379 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1443 4.2 4:43b32242:::smithi03829856-22:head [read 0~2352947,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:36.379 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.979417+0000", 2024-03-19T11:30:36.379 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.334739108999999, 2024-03-19T11:30:36.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.96719988700000004, 2024-03-19T11:30:36.379 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:36.379 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:36.379 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:36.380 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:36.380 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:36.380 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1443 2024-03-19T11:30:36.380 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.380 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:36.380 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.380 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:36.380 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979417+0000", 2024-03-19T11:30:36.380 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.380 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.380 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.380 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:36.380 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979417+0000", 2024-03-19T11:30:36.380 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.380 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.381 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.381 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:36.381 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979420+0000", 2024-03-19T11:30:36.381 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.366e-06 2024-03-19T11:30:36.381 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.381 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.381 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:36.381 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979437+0000", 2024-03-19T11:30:36.381 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6192e-05 2024-03-19T11:30:36.381 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.381 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.381 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:36.381 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979474+0000", 2024-03-19T11:30:36.381 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7601000000000001e-05 2024-03-19T11:30:36.381 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.382 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:36.382 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979496+0000", 2024-03-19T11:30:36.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1506999999999999e-05 2024-03-19T11:30:36.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.382 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:36.382 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979712+0000", 2024-03-19T11:30:36.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021641299999999999 2024-03-19T11:30:36.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.382 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:30:36.382 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979746+0000", 2024-03-19T11:30:36.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3386000000000002e-05 2024-03-19T11:30:36.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:36.383 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.931552+0000", 2024-03-19T11:30:36.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.95180659899999998 2024-03-19T11:30:36.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-03-19T11:30:36.383 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.934681+0000", 2024-03-19T11:30:36.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0031292289999999999 2024-03-19T11:30:36.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:36.383 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.942701+0000", 2024-03-19T11:30:36.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0080198809999999995 2024-03-19T11:30:36.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.384 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.384 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:36.384 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.942839+0000", 2024-03-19T11:30:36.384 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013816999999999999 2024-03-19T11:30:36.384 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.384 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.384 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:36.384 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.946617+0000", 2024-03-19T11:30:36.384 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003777543 2024-03-19T11:30:36.384 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.384 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:36.384 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.384 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.384 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.384 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1444 4.2 4:43b32242:::smithi03829856-22:head [read 0~2352947,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:36.385 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.979553+0000", 2024-03-19T11:30:36.385 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.334603405999999, 2024-03-19T11:30:36.385 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.97015945400000003, 2024-03-19T11:30:36.385 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:36.385 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:36.385 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:36.385 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:36.385 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:36.385 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1444 2024-03-19T11:30:36.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.385 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:36.385 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.385 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:36.385 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979553+0000", 2024-03-19T11:30:36.385 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.386 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:36.386 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979553+0000", 2024-03-19T11:30:36.386 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.386 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:36.386 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979554+0000", 2024-03-19T11:30:36.386 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.403e-06 2024-03-19T11:30:36.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.386 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:36.386 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979564+0000", 2024-03-19T11:30:36.386 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0032e-05 2024-03-19T11:30:36.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.387 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:36.387 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979579+0000", 2024-03-19T11:30:36.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4875000000000001e-05 2024-03-19T11:30:36.387 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.387 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:36.387 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979595+0000", 2024-03-19T11:30:36.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6300999999999999e-05 2024-03-19T11:30:36.387 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.387 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:36.387 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979874+0000", 2024-03-19T11:30:36.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000279065 2024-03-19T11:30:36.387 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:30:36.388 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979904+0000", 2024-03-19T11:30:36.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.94e-05 2024-03-19T11:30:36.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:36.388 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.934781+0000", 2024-03-19T11:30:36.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.95487744900000004 2024-03-19T11:30:36.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-03-19T11:30:36.388 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.934946+0000", 2024-03-19T11:30:36.389 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016485999999999999 2024-03-19T11:30:36.389 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.389 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.389 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:36.389 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.946829+0000", 2024-03-19T11:30:36.389 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011882482999999999 2024-03-19T11:30:36.389 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.389 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.389 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:36.389 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.946989+0000", 2024-03-19T11:30:36.389 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000160421 2024-03-19T11:30:36.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.390 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.390 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:36.390 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.949712+0000", 2024-03-19T11:30:36.390 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002723165 2024-03-19T11:30:36.390 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.390 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:36.390 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.390 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.390 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1445 4.2 4:43b32242:::smithi03829856-22:head [read 0~2352947,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:36.390 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.979653+0000", 2024-03-19T11:30:36.390 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.334503274999999, 2024-03-19T11:30:36.390 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.97249507999999996, 2024-03-19T11:30:36.391 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:36.391 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:36.391 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:36.391 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:36.391 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:36.391 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1445 2024-03-19T11:30:36.391 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.391 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:36.391 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.391 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:36.391 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979653+0000", 2024-03-19T11:30:36.392 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.392 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.392 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:36.392 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979653+0000", 2024-03-19T11:30:36.392 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.392 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.392 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:36.392 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979654+0000", 2024-03-19T11:30:36.393 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.372e-06 2024-03-19T11:30:36.393 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.393 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:36.393 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979662+0000", 2024-03-19T11:30:36.393 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9959999999999993e-06 2024-03-19T11:30:36.393 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.393 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:36.393 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979698+0000", 2024-03-19T11:30:36.393 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5763000000000003e-05 2024-03-19T11:30:36.393 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.393 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:36.393 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979714+0000", 2024-03-19T11:30:36.394 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6283999999999999e-05 2024-03-19T11:30:36.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.394 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.394 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:36.394 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979990+0000", 2024-03-19T11:30:36.394 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000275829 2024-03-19T11:30:36.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.394 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.394 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:30:36.394 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.980018+0000", 2024-03-19T11:30:36.394 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8082999999999999e-05 2024-03-19T11:30:36.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.394 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.394 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:36.394 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.935620+0000", 2024-03-19T11:30:36.395 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.95560133800000002 2024-03-19T11:30:36.395 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.395 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.395 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-03-19T11:30:36.395 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.935795+0000", 2024-03-19T11:30:36.395 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000175656 2024-03-19T11:30:36.395 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.395 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.395 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:36.395 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.949835+0000", 2024-03-19T11:30:36.395 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014040083 2024-03-19T11:30:36.395 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.395 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.395 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:36.395 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.950002+0000", 2024-03-19T11:30:36.396 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016673 2024-03-19T11:30:36.396 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.396 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.396 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:36.396 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.952148+0000", 2024-03-19T11:30:36.396 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021459460000000001 2024-03-19T11:30:36.396 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.396 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:36.396 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.396 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.396 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.396 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1449 4.18 4:1d58db11:::smithi03829856-18:head [write 581542~688983,stat] snapc c=[c,b,a,6,5] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:36.396 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.989354+0000", 2024-03-19T11:30:36.396 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.324801822, 2024-03-19T11:30:36.396 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019237318, 2024-03-19T11:30:36.397 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:36.397 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:36.397 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:36.397 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:36.397 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:36.397 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1449 2024-03-19T11:30:36.397 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.397 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:36.397 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.397 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:36.397 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.989354+0000", 2024-03-19T11:30:36.397 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.397 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.397 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.397 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:36.397 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.989354+0000", 2024-03-19T11:30:36.398 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.398 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.398 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:36.398 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.989357+0000", 2024-03-19T11:30:36.398 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2740000000000002e-06 2024-03-19T11:30:36.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.398 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.398 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:36.398 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.990109+0000", 2024-03-19T11:30:36.398 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00075226500000000001 2024-03-19T11:30:36.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.398 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.398 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:36.399 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.990142+0000", 2024-03-19T11:30:36.399 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3328999999999999e-05 2024-03-19T11:30:36.399 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.399 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.399 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:36.399 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.990168+0000", 2024-03-19T11:30:36.399 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6016999999999998e-05 2024-03-19T11:30:36.399 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.399 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.399 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:36.399 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.990291+0000", 2024-03-19T11:30:36.399 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000122607 2024-03-19T11:30:36.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.400 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.400 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:36.400 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.990461+0000", 2024-03-19T11:30:36.400 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000169743 2024-03-19T11:30:36.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.400 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.400 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:36.400 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.990792+0000", 2024-03-19T11:30:36.400 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033135100000000002 2024-03-19T11:30:36.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.400 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.400 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:36.400 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.008238+0000", 2024-03-19T11:30:36.400 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017445591999999999 2024-03-19T11:30:36.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.401 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.401 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:36.401 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.008345+0000", 2024-03-19T11:30:36.401 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010762 2024-03-19T11:30:36.401 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.401 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.401 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:36.401 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.008436+0000", 2024-03-19T11:30:36.401 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1116e-05 2024-03-19T11:30:36.401 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.401 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.401 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:36.401 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.008592+0000", 2024-03-19T11:30:36.401 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015540399999999999 2024-03-19T11:30:36.401 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.402 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:36.402 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.402 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.402 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.402 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1450 4.18 4:1d58db11:::smithi03829856-18:head [write 1886503~412692] snapc c=[c,b,a,6,5] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:36.402 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.992973+0000", 2024-03-19T11:30:36.402 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.321183053, 2024-03-19T11:30:36.402 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016912580999999999, 2024-03-19T11:30:36.402 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:36.402 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:36.402 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:36.402 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:36.402 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:36.402 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1450 2024-03-19T11:30:36.403 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.403 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:36.403 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.403 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:36.403 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.992973+0000", 2024-03-19T11:30:36.403 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.403 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.403 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.403 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:36.403 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.992973+0000", 2024-03-19T11:30:36.403 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.404 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.404 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:36.404 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.992976+0000", 2024-03-19T11:30:36.404 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0060000000000001e-06 2024-03-19T11:30:36.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.404 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.404 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:36.404 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.993655+0000", 2024-03-19T11:30:36.404 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00067901099999999998 2024-03-19T11:30:36.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.404 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.404 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:36.404 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.993691+0000", 2024-03-19T11:30:36.404 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5546000000000002e-05 2024-03-19T11:30:36.405 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.405 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.405 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:36.405 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.993713+0000", 2024-03-19T11:30:36.405 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2277e-05 2024-03-19T11:30:36.405 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.405 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.405 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:36.405 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.997961+0000", 2024-03-19T11:30:36.405 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0042479659999999997 2024-03-19T11:30:36.405 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.405 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.405 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:36.405 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.998130+0000", 2024-03-19T11:30:36.405 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016908900000000001 2024-03-19T11:30:36.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.406 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:36.406 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.998397+0000", 2024-03-19T11:30:36.406 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026707599999999999 2024-03-19T11:30:36.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.406 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:36.406 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.008811+0000", 2024-03-19T11:30:36.406 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010413751000000001 2024-03-19T11:30:36.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.406 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:36.406 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.009539+0000", 2024-03-19T11:30:36.406 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00072811499999999999 2024-03-19T11:30:36.407 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.407 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.407 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:36.407 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.009711+0000", 2024-03-19T11:30:36.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017196900000000001 2024-03-19T11:30:36.407 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.407 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.407 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:36.407 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.009886+0000", 2024-03-19T11:30:36.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017477500000000001 2024-03-19T11:30:36.407 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.407 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:36.407 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.407 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.408 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1637 4.7 4:e41d47fb:::smithi03829856-14:head [write 484068~792524,stat] snapc 10=[10,f,e,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-03-19T11:30:36.408 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:35.672719+0000", 2024-03-19T11:30:36.408 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.64143682499999999, 2024-03-19T11:30:36.408 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012961324, 2024-03-19T11:30:36.408 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:36.408 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:36.408 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:36.408 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:36.408 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:36.408 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1637 2024-03-19T11:30:36.408 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.408 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:36.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.408 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:36.408 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.672719+0000", 2024-03-19T11:30:36.409 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.409 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.409 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:36.409 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.672719+0000", 2024-03-19T11:30:36.409 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.409 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.409 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:36.409 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.672722+0000", 2024-03-19T11:30:36.409 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0690000000000002e-06 2024-03-19T11:30:36.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.409 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.409 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:36.409 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.673743+0000", 2024-03-19T11:30:36.410 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010203899999999999 2024-03-19T11:30:36.410 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.410 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:36.410 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.673774+0000", 2024-03-19T11:30:36.410 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1049e-05 2024-03-19T11:30:36.410 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.410 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:36.410 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.673789+0000", 2024-03-19T11:30:36.410 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4729000000000001e-05 2024-03-19T11:30:36.410 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.410 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:36.410 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.673921+0000", 2024-03-19T11:30:36.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013277100000000001 2024-03-19T11:30:36.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.411 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:36.411 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.674053+0000", 2024-03-19T11:30:36.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000131975 2024-03-19T11:30:36.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.411 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:36.411 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.674430+0000", 2024-03-19T11:30:36.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000377031 2024-03-19T11:30:36.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.411 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:36.411 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.684295+0000", 2024-03-19T11:30:36.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0098643120000000001 2024-03-19T11:30:36.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:36.412 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.685359+0000", 2024-03-19T11:30:36.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001064111 2024-03-19T11:30:36.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:36.412 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.685532+0000", 2024-03-19T11:30:36.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017305700000000001 2024-03-19T11:30:36.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:36.412 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.685681+0000", 2024-03-19T11:30:36.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014883 2024-03-19T11:30:36.413 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.413 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:36.413 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.413 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.413 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:36.413 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:36.415 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-19T11:30:36.416 INFO:tasks.rados.rados.0.smithi038.stdout:906: done (3 left) 2024-03-19T11:30:36.416 INFO:tasks.rados.rados.0.smithi038.stdout:907: done (2 left) 2024-03-19T11:30:36.416 INFO:tasks.rados.rados.0.smithi038.stdout:908: done (1 left) 2024-03-19T11:30:36.416 INFO:tasks.rados.rados.0.smithi038.stdout:910: done (0 left) 2024-03-19T11:30:36.417 INFO:tasks.rados.rados.0.smithi038.stdout:911: tier_evict oid 49 2024-03-19T11:30:36.417 INFO:tasks.rados.rados.0.smithi038.stdout:911: tier_evict oid 49 snap -1 2024-03-19T11:30:36.417 INFO:tasks.rados.rados.0.smithi038.stdout:912: set_chunk oid 48 target oid 2024-03-19T11:30:36.417 INFO:tasks.rados.rados.0.smithi038.stdout:912: set_chunk oid 48 offset: 0 length: 4000000 target oid smithi03829856- offset: 0 snap -1 2024-03-19T11:30:36.417 INFO:tasks.rados.rados.0.smithi038.stdout:913: read oid 24 snap -1 2024-03-19T11:30:36.417 INFO:tasks.rados.rados.0.smithi038.stdout:913: expect (ObjNum 23 snap 0 seq_num 23) 2024-03-19T11:30:36.417 INFO:tasks.rados.rados.0.smithi038.stdout:914: read oid 13 snap -1 2024-03-19T11:30:36.417 INFO:tasks.rados.rados.0.smithi038.stdout:914: expect (ObjNum 12 snap 0 seq_num 12) 2024-03-19T11:30:36.417 INFO:tasks.rados.rados.0.smithi038.stdout:915: read oid 29 snap 11 2024-03-19T11:30:36.417 INFO:tasks.rados.rados.0.smithi038.stdout:915: expect (ObjNum 95 snap 10 seq_num 95) 2024-03-19T11:30:36.417 INFO:tasks.rados.rados.0.smithi038.stdout:916: read oid 23 snap -1 2024-03-19T11:30:36.417 INFO:tasks.rados.rados.0.smithi038.stdout:916: expect (ObjNum 22 snap 0 seq_num 22) 2024-03-19T11:30:36.417 INFO:tasks.rados.rados.0.smithi038.stdout:917: tier_promote oid 31 2024-03-19T11:30:36.417 INFO:tasks.rados.rados.0.smithi038.stdout:918: read oid 35 snap -1 2024-03-19T11:30:36.417 INFO:tasks.rados.rados.0.smithi038.stdout:918: expect (ObjNum 34 snap 0 seq_num 34) 2024-03-19T11:30:36.418 INFO:tasks.rados.rados.0.smithi038.stdout:919: write (excl) oid 38 current snap is 15 2024-03-19T11:30:36.418 INFO:tasks.rados.rados.0.smithi038.stdout:919: seq_num 125 ranges {500875=467773,1564955=535920} 2024-03-19T11:30:36.418 INFO:tasks.rados.rados.0.smithi038.stdout:919: writing smithi03829856-38 from 500875 to 968648 tid 1 2024-03-19T11:30:36.418 INFO:tasks.rados.rados.0.smithi038.stdout:919: writing smithi03829856-38 from 1564955 to 2100875 tid 2 2024-03-19T11:30:36.418 INFO:tasks.rados.rados.0.smithi038.stdout:920: read oid 15 snap -1 2024-03-19T11:30:36.418 INFO:tasks.rados.rados.0.smithi038.stdout:920: expect (ObjNum 14 snap 0 seq_num 14) 2024-03-19T11:30:36.418 INFO:tasks.rados.rados.0.smithi038.stdout:921: read oid 16 snap -1 2024-03-19T11:30:36.418 INFO:tasks.rados.rados.0.smithi038.stdout:921: expect (ObjNum 118 snap 13 seq_num 118) 2024-03-19T11:30:36.418 INFO:tasks.rados.rados.0.smithi038.stdout:922: write (excl) oid 39 current snap is 15 2024-03-19T11:30:36.418 INFO:tasks.rados.rados.0.smithi038.stdout:922: seq_num 126 ranges {517682=543022,1623437=494245} 2024-03-19T11:30:36.418 INFO:tasks.rados.rados.0.smithi038.stdout:922: writing smithi03829856-39 from 517682 to 1060704 tid 1 2024-03-19T11:30:36.418 INFO:tasks.rados.rados.0.smithi038.stdout:922: writing smithi03829856-39 from 1623437 to 2117682 tid 2 2024-03-19T11:30:36.418 INFO:tasks.rados.rados.0.smithi038.stdout:923: read oid 45 snap -1 2024-03-19T11:30:36.418 INFO:tasks.rados.rados.0.smithi038.stdout:923: expect (ObjNum 59 snap 3 seq_num 59) 2024-03-19T11:30:36.418 INFO:tasks.rados.rados.0.smithi038.stdout:924: rollback oid 43 current snap is 15 2024-03-19T11:30:36.418 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 43 to 10 2024-03-19T11:30:36.419 INFO:tasks.rados.rados.0.smithi038.stdout:912: finishing set_chunk to oid 48 2024-03-19T11:30:36.419 INFO:tasks.rados.rados.0.smithi038.stdout:912: got expected ERANGE (src dne) 2024-03-19T11:30:36.419 INFO:tasks.rados.rados.0.smithi038.stdout:912: done (13 left) 2024-03-19T11:30:36.419 INFO:tasks.rados.rados.0.smithi038.stdout:925: write oid 1 current snap is 15 2024-03-19T11:30:36.419 INFO:tasks.rados.rados.0.smithi038.stdout:911: got (22) Invalid argument 2024-03-19T11:30:36.419 INFO:tasks.rados.rados.0.smithi038.stdout:914: expect (ObjNum 12 snap 0 seq_num 12) 2024-03-19T11:30:36.419 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:36.423+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:36.454 INFO:tasks.rados.rados.0.smithi038.stdout:915: expect (ObjNum 95 snap 10 seq_num 95) 2024-03-19T11:30:36.517 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:36.527+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:36.585 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:36.585 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:36.586 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:30:36.586 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:36.599 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-19T11:30:36.600 INFO:tasks.rados.rados.0.smithi038.stdout:916: expect (ObjNum 22 snap 0 seq_num 22) 2024-03-19T11:30:36.650 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:36.661+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:36.675 INFO:tasks.rados.rados.0.smithi038.stdout:917: got (0) Success 2024-03-19T11:30:36.676 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 31 v 79 (ObjNum 111 snap 11 seq_num 111) dirty exists 2024-03-19T11:30:36.676 INFO:tasks.rados.rados.0.smithi038.stdout:913: expect (ObjNum 23 snap 0 seq_num 23) 2024-03-19T11:30:36.718 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:36.727+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:36.723 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:36.723 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:36.723 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:30:36.723 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:30:36.723 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:36.737 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-19T11:30:36.753 INFO:tasks.rados.rados.0.smithi038.stdout:918: expect (ObjNum 34 snap 0 seq_num 34) 2024-03-19T11:30:36.819 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:36.831+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:36.834 INFO:tasks.rados.rados.0.smithi038.stdout:921: expect (ObjNum 118 snap 13 seq_num 118) 2024-03-19T11:30:36.902 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:36.902 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:30:36.903 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:30:36.903 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:30:36.903 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.903 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:29 4.5 4:ac9cd3a3:::smithi03829856-8:head [write 517649~526743,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:30:36.903 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:04.772849+0000", 2024-03-19T11:30:36.903 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.140331977999999, 2024-03-19T11:30:36.903 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014076821, 2024-03-19T11:30:36.903 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:36.903 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:36.903 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:36.903 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:36.903 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:36.903 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 29 2024-03-19T11:30:36.903 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.903 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:36.904 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.904 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:36.904 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.772849+0000", 2024-03-19T11:30:36.904 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.904 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.904 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.904 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:36.904 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.772849+0000", 2024-03-19T11:30:36.904 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.904 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.904 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.904 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:36.904 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.772852+0000", 2024-03-19T11:30:36.904 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.79e-06 2024-03-19T11:30:36.904 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.905 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.905 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:36.905 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.773783+0000", 2024-03-19T11:30:36.905 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00093104199999999998 2024-03-19T11:30:36.905 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.905 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.905 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:36.905 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.773810+0000", 2024-03-19T11:30:36.905 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.669e-05 2024-03-19T11:30:36.905 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.905 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.905 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:36.905 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.773826+0000", 2024-03-19T11:30:36.905 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5739999999999998e-05 2024-03-19T11:30:36.905 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.906 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.906 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:36.906 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.773954+0000", 2024-03-19T11:30:36.906 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012796300000000001 2024-03-19T11:30:36.906 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.906 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.906 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op must wait for map", 2024-03-19T11:30:36.906 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.773965+0000", 2024-03-19T11:30:36.906 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1453e-05 2024-03-19T11:30:36.906 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.906 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.906 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:36.906 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.777723+0000", 2024-03-19T11:30:36.906 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003757944 2024-03-19T11:30:36.906 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.907 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.907 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:36.907 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778138+0000", 2024-03-19T11:30:36.907 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00041454699999999997 2024-03-19T11:30:36.907 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.907 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.907 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:30:36.907 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778409+0000", 2024-03-19T11:30:36.907 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027111700000000003 2024-03-19T11:30:36.907 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.907 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.907 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:36.907 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.786631+0000", 2024-03-19T11:30:36.907 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0082221470000000008 2024-03-19T11:30:36.907 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.907 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.908 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:36.908 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.786668+0000", 2024-03-19T11:30:36.908 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.718e-05 2024-03-19T11:30:36.908 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.908 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.908 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:36.908 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.786845+0000", 2024-03-19T11:30:36.908 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000177219 2024-03-19T11:30:36.909 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.909 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.909 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:36.909 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.786926+0000", 2024-03-19T11:30:36.909 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0989000000000007e-05 2024-03-19T11:30:36.909 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.909 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:36.909 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.909 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.909 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.909 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:30 4.5 4:ac9cd3a3:::smithi03829856-8:head [write 1776668~340981] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:30:36.909 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:04.777922+0000", 2024-03-19T11:30:36.909 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.135259347999998, 2024-03-19T11:30:36.909 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015469894, 2024-03-19T11:30:36.910 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:36.910 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:36.910 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:36.910 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:36.910 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:36.910 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 30 2024-03-19T11:30:36.910 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.910 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:36.910 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.910 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:36.911 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.777922+0000", 2024-03-19T11:30:36.911 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.911 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.911 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.911 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:36.911 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.777922+0000", 2024-03-19T11:30:36.911 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.911 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.911 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.911 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:36.911 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.777924+0000", 2024-03-19T11:30:36.911 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3999999999999999e-06 2024-03-19T11:30:36.911 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.911 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.912 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:36.912 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778461+0000", 2024-03-19T11:30:36.912 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00053698299999999999 2024-03-19T11:30:36.912 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.912 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.912 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:36.912 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778487+0000", 2024-03-19T11:30:36.912 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5947000000000001e-05 2024-03-19T11:30:36.912 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.912 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.912 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:36.912 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778502+0000", 2024-03-19T11:30:36.912 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4256e-05 2024-03-19T11:30:36.912 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.912 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.913 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:36.913 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.781978+0000", 2024-03-19T11:30:36.913 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00347693 2024-03-19T11:30:36.913 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.913 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.913 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:36.913 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.782196+0000", 2024-03-19T11:30:36.913 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021726499999999999 2024-03-19T11:30:36.913 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.913 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.913 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:30:36.913 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.782572+0000", 2024-03-19T11:30:36.913 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00037635000000000001 2024-03-19T11:30:36.913 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.913 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.914 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:36.914 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.786960+0000", 2024-03-19T11:30:36.914 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043882130000000002 2024-03-19T11:30:36.914 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.914 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.914 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:36.914 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793113+0000", 2024-03-19T11:30:36.914 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061530760000000004 2024-03-19T11:30:36.914 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.914 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.914 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:36.914 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793276+0000", 2024-03-19T11:30:36.914 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000162652 2024-03-19T11:30:36.914 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.914 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.915 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:36.915 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793392+0000", 2024-03-19T11:30:36.915 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115822 2024-03-19T11:30:36.915 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.915 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:36.915 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.915 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.915 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.915 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:31 4.5 4:ac9cd3a3:::smithi03829856-8:head [setxattr _header (55) in=7b,truncate 2117649] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:30:36.915 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:04.778533+0000", 2024-03-19T11:30:36.915 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.134648212999998, 2024-03-19T11:30:36.915 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015182533, 2024-03-19T11:30:36.915 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:36.915 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:36.915 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:36.916 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:36.916 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:36.916 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 31 2024-03-19T11:30:36.916 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.916 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:36.916 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.916 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:36.916 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778533+0000", 2024-03-19T11:30:36.916 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.916 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.916 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.917 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:36.917 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778533+0000", 2024-03-19T11:30:36.917 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.917 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.917 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.917 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:36.917 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778534+0000", 2024-03-19T11:30:36.917 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1069999999999999e-06 2024-03-19T11:30:36.917 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.917 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.917 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:36.917 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778541+0000", 2024-03-19T11:30:36.917 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2529999999999998e-06 2024-03-19T11:30:36.917 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.917 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.918 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:36.918 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778558+0000", 2024-03-19T11:30:36.918 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6290000000000002e-05 2024-03-19T11:30:36.918 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.918 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.918 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:36.918 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778568+0000", 2024-03-19T11:30:36.918 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0360000000000001e-05 2024-03-19T11:30:36.918 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.918 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.918 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:36.918 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.785511+0000", 2024-03-19T11:30:36.918 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00694306 2024-03-19T11:30:36.919 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.919 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.919 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:36.919 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.785710+0000", 2024-03-19T11:30:36.919 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019837199999999999 2024-03-19T11:30:36.919 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.919 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.919 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:30:36.919 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.785929+0000", 2024-03-19T11:30:36.919 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021960799999999999 2024-03-19T11:30:36.919 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.919 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.919 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:36.919 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.787281+0000", 2024-03-19T11:30:36.919 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001352313 2024-03-19T11:30:36.920 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.920 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.920 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:36.920 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793492+0000", 2024-03-19T11:30:36.920 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062107760000000003 2024-03-19T11:30:36.920 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.920 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.920 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:36.920 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793578+0000", 2024-03-19T11:30:36.920 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5711999999999995e-05 2024-03-19T11:30:36.920 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.920 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.920 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:36.920 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793716+0000", 2024-03-19T11:30:36.920 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000137682 2024-03-19T11:30:36.921 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.921 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:36.921 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.921 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.921 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.921 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:32 4.5 4:ac9cd3a3:::smithi03829856-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:30:36.921 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:04.778582+0000", 2024-03-19T11:30:36.921 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.134599698999999, 2024-03-19T11:30:36.921 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015761827999999999, 2024-03-19T11:30:36.921 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:36.921 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:36.921 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:36.921 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:36.921 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:36.921 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 32 2024-03-19T11:30:36.921 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.922 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:36.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.922 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:36.922 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778582+0000", 2024-03-19T11:30:36.922 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.922 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:36.922 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778582+0000", 2024-03-19T11:30:36.922 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.922 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:36.922 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778583+0000", 2024-03-19T11:30:36.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.026e-06 2024-03-19T11:30:36.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.923 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:36.923 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778587+0000", 2024-03-19T11:30:36.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7870000000000001e-06 2024-03-19T11:30:36.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.923 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:36.923 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778599+0000", 2024-03-19T11:30:36.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1623e-05 2024-03-19T11:30:36.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.923 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:36.923 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778608+0000", 2024-03-19T11:30:36.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7399999999999993e-06 2024-03-19T11:30:36.924 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.924 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:36.924 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.787009+0000", 2024-03-19T11:30:36.924 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0084010490000000007 2024-03-19T11:30:36.924 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.924 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-03-19T11:30:36.924 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.787202+0000", 2024-03-19T11:30:36.924 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019309500000000001 2024-03-19T11:30:36.924 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.924 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:36.924 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793832+0000", 2024-03-19T11:30:36.924 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.006630052 2024-03-19T11:30:36.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.925 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:36.925 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.794048+0000", 2024-03-19T11:30:36.925 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000215505 2024-03-19T11:30:36.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.925 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:36.925 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.794343+0000", 2024-03-19T11:30:36.925 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029595100000000003 2024-03-19T11:30:36.925 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.926 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:36.926 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.926 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.926 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.926 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:856 4.5 4:ac9cd3a3:::smithi03829856-8:head [write 608420~529411] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:36.926 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.781425+0000", 2024-03-19T11:30:36.926 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.131756512999999, 2024-03-19T11:30:36.926 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014634677, 2024-03-19T11:30:36.926 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:36.926 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:36.927 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:36.927 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:36.927 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:36.927 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 856 2024-03-19T11:30:36.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.927 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:36.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.927 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:36.927 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.781425+0000", 2024-03-19T11:30:36.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.927 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:36.928 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.781425+0000", 2024-03-19T11:30:36.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.928 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:36.928 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.781428+0000", 2024-03-19T11:30:36.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.534e-06 2024-03-19T11:30:36.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.928 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:36.928 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.783096+0000", 2024-03-19T11:30:36.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001667897 2024-03-19T11:30:36.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.928 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:36.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.783138+0000", 2024-03-19T11:30:36.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1946000000000001e-05 2024-03-19T11:30:36.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:36.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.783163+0000", 2024-03-19T11:30:36.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4443e-05 2024-03-19T11:30:36.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:36.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.783333+0000", 2024-03-19T11:30:36.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017027899999999999 2024-03-19T11:30:36.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:36.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.783503+0000", 2024-03-19T11:30:36.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000170242 2024-03-19T11:30:36.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:30:36.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.783917+0000", 2024-03-19T11:30:36.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00041371000000000002 2024-03-19T11:30:36.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:36.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.795709+0000", 2024-03-19T11:30:36.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011791693000000001 2024-03-19T11:30:36.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.931 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:36.931 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.795850+0000", 2024-03-19T11:30:36.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000141695 2024-03-19T11:30:36.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.931 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:36.931 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.795940+0000", 2024-03-19T11:30:36.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9705999999999998e-05 2024-03-19T11:30:36.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.931 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:36.931 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.796059+0000", 2024-03-19T11:30:36.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011953199999999999 2024-03-19T11:30:36.931 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.931 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:36.932 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.932 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.932 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.932 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:891 4.5 4:af9d0d56:::smithi03829856-20:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:36.932 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:09.286410+0000", 2024-03-19T11:30:36.932 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.626771367, 2024-03-19T11:30:36.932 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035191405000000002, 2024-03-19T11:30:36.932 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:36.932 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:36.932 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:36.932 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:36.932 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:36.932 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 891 2024-03-19T11:30:36.932 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.933 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:36.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.933 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:36.933 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286410+0000", 2024-03-19T11:30:36.933 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.933 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.933 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:36.933 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286410+0000", 2024-03-19T11:30:36.933 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.933 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.934 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:36.934 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286413+0000", 2024-03-19T11:30:36.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2660000000000002e-06 2024-03-19T11:30:36.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.934 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:36.934 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286426+0000", 2024-03-19T11:30:36.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.258e-05 2024-03-19T11:30:36.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.934 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:36.934 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286460+0000", 2024-03-19T11:30:36.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3943999999999999e-05 2024-03-19T11:30:36.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:36.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286477+0000", 2024-03-19T11:30:36.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6799999999999998e-05 2024-03-19T11:30:36.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:36.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286605+0000", 2024-03-19T11:30:36.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000128892 2024-03-19T11:30:36.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:36.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286769+0000", 2024-03-19T11:30:36.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016359799999999999 2024-03-19T11:30:36.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.936 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:36.936 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.321601+0000", 2024-03-19T11:30:36.936 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.034832324999999997 2024-03-19T11:30:36.936 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.936 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:36.936 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.936 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.936 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.936 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:993 4.e e34/21 4:72163e4c:::smithi03829856-39:head v 34'17, mlcod=28'14)", 2024-03-19T11:30:36.936 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:12.367897+0000", 2024-03-19T11:30:36.936 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.545284820999999, 2024-03-19T11:30:36.936 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012765165, 2024-03-19T11:30:36.936 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:36.936 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:36.936 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:36.936 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.937 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:36.937 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.367897+0000", 2024-03-19T11:30:36.937 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.937 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.937 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.937 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:36.937 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.367897+0000", 2024-03-19T11:30:36.937 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.937 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.937 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.937 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:36.937 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.367897+0000", 2024-03-19T11:30:36.937 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7000000000000003e-07 2024-03-19T11:30:36.938 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.938 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.938 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:36.938 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.368454+0000", 2024-03-19T11:30:36.938 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00055647599999999998 2024-03-19T11:30:36.938 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.938 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.938 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:36.938 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.368532+0000", 2024-03-19T11:30:36.938 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8479000000000005e-05 2024-03-19T11:30:36.938 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.938 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.938 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:36.938 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.368554+0000", 2024-03-19T11:30:36.938 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.156e-05 2024-03-19T11:30:36.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.939 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:36.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.368742+0000", 2024-03-19T11:30:36.939 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000187846 2024-03-19T11:30:36.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.939 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:36.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.368782+0000", 2024-03-19T11:30:36.939 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9836000000000003e-05 2024-03-19T11:30:36.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.939 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:36.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.380340+0000", 2024-03-19T11:30:36.939 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011558847000000001 2024-03-19T11:30:36.940 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.940 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.940 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:36.940 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.380662+0000", 2024-03-19T11:30:36.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00032125099999999999 2024-03-19T11:30:36.940 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.940 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:36.940 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.940 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.940 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.940 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1015 4.5 4:af9d0d56:::smithi03829856-20:head [tier-promote] snapc 6=[6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:36.940 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:12.761855+0000", 2024-03-19T11:30:36.940 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.151325840999998, 2024-03-19T11:30:36.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026760430000000002, 2024-03-19T11:30:36.940 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:36.941 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:36.941 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:36.941 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:36.941 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:36.941 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1015 2024-03-19T11:30:36.941 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.942 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:36.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.942 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:36.942 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761855+0000", 2024-03-19T11:30:36.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:36.943 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761855+0000", 2024-03-19T11:30:36.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:36.943 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761858+0000", 2024-03-19T11:30:36.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8140000000000001e-06 2024-03-19T11:30:36.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:36.944 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761872+0000", 2024-03-19T11:30:36.944 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3813e-05 2024-03-19T11:30:36.944 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.944 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:36.944 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761905+0000", 2024-03-19T11:30:36.944 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2642999999999999e-05 2024-03-19T11:30:36.944 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.944 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:36.944 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761925+0000", 2024-03-19T11:30:36.944 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0489e-05 2024-03-19T11:30:36.944 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.944 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:36.945 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.762070+0000", 2024-03-19T11:30:36.945 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000145153 2024-03-19T11:30:36.945 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.945 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:36.945 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.762365+0000", 2024-03-19T11:30:36.945 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029482100000000002 2024-03-19T11:30:36.945 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.945 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:36.945 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.788616+0000", 2024-03-19T11:30:36.945 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026250697 2024-03-19T11:30:36.945 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.945 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:36.946 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.946 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1121 4.5 4:aa3303a5:::smithi03829856-17:head [tier-promote] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:36.946 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:14.164281+0000", 2024-03-19T11:30:36.946 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.748900595999999, 2024-03-19T11:30:36.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023896435000000001, 2024-03-19T11:30:36.946 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:36.946 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:36.946 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:36.946 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:36.946 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:36.946 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1121 2024-03-19T11:30:36.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.947 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:36.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:36.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164281+0000", 2024-03-19T11:30:36.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:36.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164281+0000", 2024-03-19T11:30:36.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:36.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164283+0000", 2024-03-19T11:30:36.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8500000000000001e-06 2024-03-19T11:30:36.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.948 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:36.948 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164291+0000", 2024-03-19T11:30:36.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.14e-06 2024-03-19T11:30:36.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.948 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:36.948 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164313+0000", 2024-03-19T11:30:36.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.207e-05 2024-03-19T11:30:36.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.948 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:36.948 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164328+0000", 2024-03-19T11:30:36.949 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5403000000000001e-05 2024-03-19T11:30:36.949 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.949 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:36.949 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.167981+0000", 2024-03-19T11:30:36.949 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036529879999999998 2024-03-19T11:30:36.949 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.949 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:36.949 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.168274+0000", 2024-03-19T11:30:36.949 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029313100000000002 2024-03-19T11:30:36.949 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.949 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:36.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.188177+0000", 2024-03-19T11:30:36.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019902853000000002 2024-03-19T11:30:36.950 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.950 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:36.950 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.950 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1154 4.5 4:a0a50d04:::smithi03829856-13:8 [sparse-read 0~1151947] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-03-19T11:30:36.950 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.827529+0000", 2024-03-19T11:30:36.950 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 20.085652178, 2024-03-19T11:30:36.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013620382, 2024-03-19T11:30:36.950 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:36.950 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:36.950 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:36.951 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:36.951 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:36.951 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1154 2024-03-19T11:30:36.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.951 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:36.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:36.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827529+0000", 2024-03-19T11:30:36.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:36.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827529+0000", 2024-03-19T11:30:36.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.952 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:36.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827530+0000", 2024-03-19T11:30:36.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7700000000000004e-07 2024-03-19T11:30:36.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.952 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:36.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827533+0000", 2024-03-19T11:30:36.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4520000000000002e-06 2024-03-19T11:30:36.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.953 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.953 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:36.953 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827543+0000", 2024-03-19T11:30:36.953 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0035e-05 2024-03-19T11:30:36.953 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.953 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.953 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:36.953 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827551+0000", 2024-03-19T11:30:36.953 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7049999999999999e-06 2024-03-19T11:30:36.953 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.953 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.953 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:36.953 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.839931+0000", 2024-03-19T11:30:36.953 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012379602999999999 2024-03-19T11:30:36.954 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.954 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:36.954 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.840290+0000", 2024-03-19T11:30:36.954 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000359031 2024-03-19T11:30:36.954 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.954 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:36.954 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841150+0000", 2024-03-19T11:30:36.954 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00085977899999999997 2024-03-19T11:30:36.954 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.954 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:36.955 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.955 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1302 4.15 4:aa3303a5:::smithi03829856-17:head [write 413472~620140,stat] snapc b=[b,a,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-03-19T11:30:36.955 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:22.966426+0000", 2024-03-19T11:30:36.955 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 13.946754863000001, 2024-03-19T11:30:36.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018043513000000001, 2024-03-19T11:30:36.955 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:36.955 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:36.955 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:36.955 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:36.956 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:36.956 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1302 2024-03-19T11:30:36.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.956 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:36.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.956 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:36.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.966426+0000", 2024-03-19T11:30:36.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.956 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:36.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.966426+0000", 2024-03-19T11:30:36.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.957 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.957 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:36.957 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.966429+0000", 2024-03-19T11:30:36.957 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3829999999999999e-06 2024-03-19T11:30:36.957 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.957 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.957 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:36.957 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972117+0000", 2024-03-19T11:30:36.957 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0056885750000000004 2024-03-19T11:30:36.958 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.958 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.958 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:36.958 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972156+0000", 2024-03-19T11:30:36.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8878000000000003e-05 2024-03-19T11:30:36.958 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.958 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.958 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:36.958 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972184+0000", 2024-03-19T11:30:36.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7509999999999999e-05 2024-03-19T11:30:36.959 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.959 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.959 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:36.959 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972309+0000", 2024-03-19T11:30:36.959 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012525899999999999 2024-03-19T11:30:36.959 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.959 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.959 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:36.959 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972566+0000", 2024-03-19T11:30:36.959 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025740799999999999 2024-03-19T11:30:36.959 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.959 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.959 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:30:36.959 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972899+0000", 2024-03-19T11:30:36.959 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033238799999999998 2024-03-19T11:30:36.960 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.960 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.960 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:36.960 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984088+0000", 2024-03-19T11:30:36.960 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011189466 2024-03-19T11:30:36.960 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.960 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.960 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:36.960 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984138+0000", 2024-03-19T11:30:36.960 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9666999999999997e-05 2024-03-19T11:30:36.960 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.960 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.960 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:36.961 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984329+0000", 2024-03-19T11:30:36.961 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019086799999999999 2024-03-19T11:30:36.961 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.961 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.961 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:36.961 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984470+0000", 2024-03-19T11:30:36.961 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000141111 2024-03-19T11:30:36.961 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.961 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:36.961 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.961 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.961 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.961 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1303 4.15 4:aa3303a5:::smithi03829856-17:head [write 1818036~510700] snapc b=[b,a,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-03-19T11:30:36.961 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:22.976856+0000", 2024-03-19T11:30:36.961 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 13.936325753, 2024-03-19T11:30:36.962 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015045703000000001, 2024-03-19T11:30:36.962 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:36.962 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:36.962 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:36.962 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:36.962 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:36.962 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1303 2024-03-19T11:30:36.962 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.962 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:36.962 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.962 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:36.962 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.976856+0000", 2024-03-19T11:30:36.962 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.963 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:36.963 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.976856+0000", 2024-03-19T11:30:36.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.963 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:36.963 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.976859+0000", 2024-03-19T11:30:36.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3629999999999998e-06 2024-03-19T11:30:36.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.964 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:36.964 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.977533+0000", 2024-03-19T11:30:36.964 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00067380800000000002 2024-03-19T11:30:36.964 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.964 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.964 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:36.964 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.977573+0000", 2024-03-19T11:30:36.964 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9758000000000001e-05 2024-03-19T11:30:36.964 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.964 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.964 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:36.964 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.977602+0000", 2024-03-19T11:30:36.965 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9856e-05 2024-03-19T11:30:36.965 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.965 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.965 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:36.965 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.979943+0000", 2024-03-19T11:30:36.965 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0023406400000000002 2024-03-19T11:30:36.965 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.965 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.965 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:36.965 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.980238+0000", 2024-03-19T11:30:36.965 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029488000000000002 2024-03-19T11:30:36.965 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.966 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:30:36.966 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.980734+0000", 2024-03-19T11:30:36.966 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00049647500000000002 2024-03-19T11:30:36.966 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.966 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:36.966 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.991761+0000", 2024-03-19T11:30:36.966 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011026515000000001 2024-03-19T11:30:36.966 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.966 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:36.967 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.991777+0000", 2024-03-19T11:30:36.967 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5727999999999999e-05 2024-03-19T11:30:36.967 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.967 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.967 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:36.967 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.991840+0000", 2024-03-19T11:30:36.967 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3091999999999998e-05 2024-03-19T11:30:36.967 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.967 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.967 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:36.967 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.991901+0000", 2024-03-19T11:30:36.967 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1587999999999994e-05 2024-03-19T11:30:36.968 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.968 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:36.968 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.968 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.968 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.968 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1467 4.a 4:567251f3:::smithi03829856-44:head [write 615156~555834] snapc c=[c,b,a,6,5] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-03-19T11:30:36.968 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.079231+0000", 2024-03-19T11:30:36.968 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.8339502210000003, 2024-03-19T11:30:36.968 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013571666, 2024-03-19T11:30:36.968 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:36.968 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:36.968 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:36.968 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:36.969 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:36.969 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1467 2024-03-19T11:30:36.970 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.970 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:36.970 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.970 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:36.970 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.079231+0000", 2024-03-19T11:30:36.971 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.971 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.971 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.971 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:36.971 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.079231+0000", 2024-03-19T11:30:36.971 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.971 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.971 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.971 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:36.971 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.079234+0000", 2024-03-19T11:30:36.971 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3689999999999998e-06 2024-03-19T11:30:36.972 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.972 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.972 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:36.972 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.080110+0000", 2024-03-19T11:30:36.972 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00087507100000000005 2024-03-19T11:30:36.972 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.972 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.972 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:36.972 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.080146+0000", 2024-03-19T11:30:36.972 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6062000000000001e-05 2024-03-19T11:30:36.972 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.972 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.972 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:36.972 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.080164+0000", 2024-03-19T11:30:36.973 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7995e-05 2024-03-19T11:30:36.973 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.973 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.973 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:36.973 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.080339+0000", 2024-03-19T11:30:36.973 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017535900000000001 2024-03-19T11:30:36.973 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.973 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.973 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:36.973 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.080630+0000", 2024-03-19T11:30:36.973 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029068900000000001 2024-03-19T11:30:36.973 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.973 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.973 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:36.974 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.081140+0000", 2024-03-19T11:30:36.974 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00051083699999999999 2024-03-19T11:30:36.975 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.975 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.975 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:36.975 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.092570+0000", 2024-03-19T11:30:36.975 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011429557 2024-03-19T11:30:36.975 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.975 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.975 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:36.975 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.092611+0000", 2024-03-19T11:30:36.975 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1196000000000003e-05 2024-03-19T11:30:36.975 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.975 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.975 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:36.976 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.092717+0000", 2024-03-19T11:30:36.976 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000105777 2024-03-19T11:30:36.976 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.976 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.976 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:36.976 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.092803+0000", 2024-03-19T11:30:36.976 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5754000000000006e-05 2024-03-19T11:30:36.976 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.976 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:36.976 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.976 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.976 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.976 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1468 4.a 4:567251f3:::smithi03829856-44:head [write 1603467~416361] snapc c=[c,b,a,6,5] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-03-19T11:30:36.976 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.083922+0000", 2024-03-19T11:30:36.976 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.8292597300000004, 2024-03-19T11:30:36.977 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014508549000000001, 2024-03-19T11:30:36.977 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:36.977 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:36.977 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:36.977 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:36.977 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:36.977 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1468 2024-03-19T11:30:36.977 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.977 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:36.977 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.977 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:36.977 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.083922+0000", 2024-03-19T11:30:36.977 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.977 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.977 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.978 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:36.978 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.083922+0000", 2024-03-19T11:30:36.978 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.978 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.978 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.978 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:36.978 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.083924+0000", 2024-03-19T11:30:36.978 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2800000000000002e-06 2024-03-19T11:30:36.978 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.978 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.978 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:36.978 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.084498+0000", 2024-03-19T11:30:36.978 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057414900000000004 2024-03-19T11:30:36.978 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.979 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.979 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:36.979 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.084529+0000", 2024-03-19T11:30:36.979 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1080000000000001e-05 2024-03-19T11:30:36.979 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.979 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.979 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:36.979 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.084544+0000", 2024-03-19T11:30:36.979 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4498000000000001e-05 2024-03-19T11:30:36.979 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.979 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.979 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:36.979 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.088993+0000", 2024-03-19T11:30:36.979 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044497260000000002 2024-03-19T11:30:36.979 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.980 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.980 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:36.980 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.089152+0000", 2024-03-19T11:30:36.980 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015833199999999999 2024-03-19T11:30:36.980 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.980 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.980 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:36.980 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.089419+0000", 2024-03-19T11:30:36.980 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026691799999999999 2024-03-19T11:30:36.980 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.980 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.980 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:36.980 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.092883+0000", 2024-03-19T11:30:36.980 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034646590000000001 2024-03-19T11:30:36.980 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.981 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.981 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:36.981 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.098276+0000", 2024-03-19T11:30:36.981 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053924680000000001 2024-03-19T11:30:36.981 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.981 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.981 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:36.981 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.098354+0000", 2024-03-19T11:30:36.981 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8763999999999993e-05 2024-03-19T11:30:36.981 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.981 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.981 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:36.981 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.098430+0000", 2024-03-19T11:30:36.981 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5674999999999996e-05 2024-03-19T11:30:36.982 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.982 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:36.982 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.982 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.982 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.982 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.1 50/49 [PushOp(4:866f3dcc:::smithi03829856-31:7, version: 38'44, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@38'44, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 45'62, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@45'62, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 46'68, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@46'68, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:7, version: 47'71, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@47'71, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 47'72, data_included: [440350~816629,1309629~231947], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@47'72, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:30:36.982 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.959457+0000", 2024-03-19T11:30:36.982 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 8.9537244460000007, 2024-03-19T11:30:36.982 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018416666000000002, 2024-03-19T11:30:36.982 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:36.982 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:36.982 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:36.982 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.982 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:36.983 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.959457+0000", 2024-03-19T11:30:36.983 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.983 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.983 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.983 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:36.983 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.959457+0000", 2024-03-19T11:30:36.983 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.983 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.983 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.983 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:36.983 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.959458+0000", 2024-03-19T11:30:36.983 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4899999999999999e-06 2024-03-19T11:30:36.983 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.983 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.983 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:36.984 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.962822+0000", 2024-03-19T11:30:36.984 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0033640620000000001 2024-03-19T11:30:36.984 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.984 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.984 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:36.984 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.962893+0000", 2024-03-19T11:30:36.984 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1060000000000001e-05 2024-03-19T11:30:36.984 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.984 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.984 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:36.984 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.962977+0000", 2024-03-19T11:30:36.984 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3311000000000002e-05 2024-03-19T11:30:36.984 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.984 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.984 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:36.984 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963208+0000", 2024-03-19T11:30:36.985 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023121200000000001 2024-03-19T11:30:36.985 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.985 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.985 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:36.985 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963245+0000", 2024-03-19T11:30:36.985 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6844999999999997e-05 2024-03-19T11:30:36.985 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.985 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.985 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:36.985 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.977874+0000", 2024-03-19T11:30:36.985 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014628686 2024-03-19T11:30:36.985 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.985 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:36.985 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.986 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.986 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.986 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.1 e50,49 4:8401c1f0:::smithi03829856-43:8,46'66,4:866f3dcc:::smithi03829856-31:2,47'73)", 2024-03-19T11:30:36.986 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.963027+0000", 2024-03-19T11:30:36.986 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 8.9501548290000006, 2024-03-19T11:30:36.986 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015116788000000001, 2024-03-19T11:30:36.986 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:36.986 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:36.986 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:36.986 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.986 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:36.986 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963027+0000", 2024-03-19T11:30:36.986 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.986 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.986 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.987 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:36.987 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963027+0000", 2024-03-19T11:30:36.987 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.987 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.987 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.987 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:36.987 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963028+0000", 2024-03-19T11:30:36.987 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.268e-06 2024-03-19T11:30:36.987 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.987 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.987 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:36.987 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963036+0000", 2024-03-19T11:30:36.987 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.926e-06 2024-03-19T11:30:36.987 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.988 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.988 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:36.988 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963050+0000", 2024-03-19T11:30:36.988 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4082e-05 2024-03-19T11:30:36.988 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.988 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.988 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:36.988 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963062+0000", 2024-03-19T11:30:36.988 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2415000000000001e-05 2024-03-19T11:30:36.988 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.988 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.988 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:36.988 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.977954+0000", 2024-03-19T11:30:36.988 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014891572 2024-03-19T11:30:36.988 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.989 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.989 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:36.989 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.977965+0000", 2024-03-19T11:30:36.989 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1654000000000001e-05 2024-03-19T11:30:36.989 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.989 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.989 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:36.989 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.978143+0000", 2024-03-19T11:30:36.989 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017787099999999999 2024-03-19T11:30:36.989 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.989 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:36.989 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.989 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.989 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.989 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.11 50/49 [PushOp(4:8d3763f9:::smithi03829856-29:head, version: 47'60, data_included: [3364276~232389], data_size: 232389, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8d3763f9:::smithi03829856-29:head@47'60, size: 3596665, copy_subset: [(0, 3596665)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3596665, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3364276, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:30:36.990 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.964363+0000", 2024-03-19T11:30:36.990 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 8.9488181620000002, 2024-03-19T11:30:36.990 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014531759999999999, 2024-03-19T11:30:36.990 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:36.990 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:36.990 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:36.990 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.990 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:36.990 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.964363+0000", 2024-03-19T11:30:36.990 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.990 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.990 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.990 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:36.990 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.964363+0000", 2024-03-19T11:30:36.990 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.991 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.992 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.992 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:36.992 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.964364+0000", 2024-03-19T11:30:36.992 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4200000000000004e-07 2024-03-19T11:30:36.992 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.992 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.992 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:36.993 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.964447+0000", 2024-03-19T11:30:36.993 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2430999999999997e-05 2024-03-19T11:30:36.993 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.993 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.993 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:36.993 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.964468+0000", 2024-03-19T11:30:36.993 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1685999999999999e-05 2024-03-19T11:30:36.993 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.993 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.993 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:36.993 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.964483+0000", 2024-03-19T11:30:36.993 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4969e-05 2024-03-19T11:30:36.993 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.994 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.995 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:36.995 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.977934+0000", 2024-03-19T11:30:36.995 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013450963999999999 2024-03-19T11:30:36.995 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.995 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.995 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:36.995 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.977975+0000", 2024-03-19T11:30:36.995 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1125999999999999e-05 2024-03-19T11:30:36.995 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.995 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.995 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:36.996 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.978895+0000", 2024-03-19T11:30:36.996 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00091964200000000003 2024-03-19T11:30:36.996 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.996 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:36.996 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:36.996 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.996 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.996 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.1 50/49 [PushOp(4:866f3dcc:::smithi03829856-31:7, version: 38'44, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@38'44, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 45'62, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@45'62, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 46'68, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@46'68, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:7, version: 47'71, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@47'71, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 47'72, data_included: [1541576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@47'72, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:30:36.996 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.991858+0000", 2024-03-19T11:30:36.996 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 8.9213232389999995, 2024-03-19T11:30:36.996 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014228774, 2024-03-19T11:30:36.996 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:36.996 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:36.996 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:36.997 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.997 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:36.997 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.991858+0000", 2024-03-19T11:30:36.997 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.997 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.997 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.997 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:36.997 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.991858+0000", 2024-03-19T11:30:36.997 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:36.997 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.997 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.997 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:36.997 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.991859+0000", 2024-03-19T11:30:36.997 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.342e-06 2024-03-19T11:30:36.998 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.998 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.998 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:36.998 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.994675+0000", 2024-03-19T11:30:36.998 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028153150000000001 2024-03-19T11:30:36.998 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.998 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.998 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:36.998 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.994733+0000", 2024-03-19T11:30:36.998 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8514999999999998e-05 2024-03-19T11:30:36.998 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.998 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.998 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:36.998 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.994794+0000", 2024-03-19T11:30:36.998 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0588000000000003e-05 2024-03-19T11:30:36.999 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.999 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.999 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:36.999 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.995070+0000", 2024-03-19T11:30:36.999 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027573700000000002 2024-03-19T11:30:36.999 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.999 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.999 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:36.999 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.995100+0000", 2024-03-19T11:30:36.999 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0679000000000002e-05 2024-03-19T11:30:36.999 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:36.999 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:36.999 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:36.999 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.006087+0000", 2024-03-19T11:30:36.999 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010986598 2024-03-19T11:30:37.000 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:37.000 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:37.000 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:37.000 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:37.000 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:37.000 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.1 50/49 [PushOp(4:866f3dcc:::smithi03829856-31:head, version: 47'74, data_included: [504210~968657,1572695~79919], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:head@47'74, size: 3495823, copy_subset: [(0, 3495823)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:30:37.000 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.995921+0000", 2024-03-19T11:30:37.000 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 8.9172600250000009, 2024-03-19T11:30:37.000 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014183898, 2024-03-19T11:30:37.000 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:37.000 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:37.000 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:37.000 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:37.000 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:37.000 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.995921+0000", 2024-03-19T11:30:37.001 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:37.001 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:37.001 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:37.001 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:37.001 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.995921+0000", 2024-03-19T11:30:37.001 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:37.001 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:37.001 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:37.001 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:37.001 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.995922+0000", 2024-03-19T11:30:37.001 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9800000000000002e-07 2024-03-19T11:30:37.001 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:37.001 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:37.001 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:37.002 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.996355+0000", 2024-03-19T11:30:37.002 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000432211 2024-03-19T11:30:37.002 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:37.002 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:37.002 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:37.002 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.996386+0000", 2024-03-19T11:30:37.002 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1384999999999999e-05 2024-03-19T11:30:37.002 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:37.002 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:37.002 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:37.002 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.996412+0000", 2024-03-19T11:30:37.002 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6200999999999999e-05 2024-03-19T11:30:37.002 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:37.002 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:37.003 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:37.003 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.006166+0000", 2024-03-19T11:30:37.003 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0097542859999999992 2024-03-19T11:30:37.003 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:37.003 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:37.003 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:37.003 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.006182+0000", 2024-03-19T11:30:37.003 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5588000000000001e-05 2024-03-19T11:30:37.003 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:37.003 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:37.003 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:37.003 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.010105+0000", 2024-03-19T11:30:37.003 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039232290000000003 2024-03-19T11:30:37.003 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:37.003 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:37.003 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:37.004 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:37.004 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:37.004 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1679 4.e e59/21 4:72163e4c:::smithi03829856-39:head v 59'55, mlcod=58'53)", 2024-03-19T11:30:37.004 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:36.390395+0000", 2024-03-19T11:30:37.004 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.52278613500000004, 2024-03-19T11:30:37.004 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013872282, 2024-03-19T11:30:37.004 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:37.004 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:37.004 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:37.004 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:37.004 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:37.004 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.390395+0000", 2024-03-19T11:30:37.004 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:37.004 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:37.004 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:37.005 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:37.005 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.390395+0000", 2024-03-19T11:30:37.005 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:37.005 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:37.005 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:37.005 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:37.005 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.390396+0000", 2024-03-19T11:30:37.005 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2300000000000002e-07 2024-03-19T11:30:37.005 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:37.005 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:37.005 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:37.005 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.390742+0000", 2024-03-19T11:30:37.005 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00034635599999999998 2024-03-19T11:30:37.005 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:37.005 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:37.005 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:37.006 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.390754+0000", 2024-03-19T11:30:37.006 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2211999999999999e-05 2024-03-19T11:30:37.006 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:37.006 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:37.006 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:37.006 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.390763+0000", 2024-03-19T11:30:37.006 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5769999999999994e-06 2024-03-19T11:30:37.006 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:37.006 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:37.006 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:37.006 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.390864+0000", 2024-03-19T11:30:37.006 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000101693 2024-03-19T11:30:37.006 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:37.006 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:37.006 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:37.007 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.390907+0000", 2024-03-19T11:30:37.007 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2642999999999998e-05 2024-03-19T11:30:37.007 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:37.007 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:37.007 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:37.007 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.404144+0000", 2024-03-19T11:30:37.007 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013237319000000001 2024-03-19T11:30:37.007 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:37.007 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:37.007 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:37.007 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.404267+0000", 2024-03-19T11:30:37.007 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012315899999999999 2024-03-19T11:30:37.007 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:37.007 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:37.008 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:37.008 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:37.008 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:37.008 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:37.009 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-19T11:30:37.010 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:37.017+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:37.037 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-19T11:30:37.059 INFO:tasks.rados.rados.0.smithi038.stdout:919: finishing write tid 1 to smithi03829856-38 2024-03-19T11:30:37.059 INFO:tasks.rados.rados.0.smithi038.stdout:919: finishing write tid 2 to smithi03829856-38 2024-03-19T11:30:37.059 INFO:tasks.rados.rados.0.smithi038.stdout:919: finishing write tid 3 to smithi03829856-38 2024-03-19T11:30:37.059 INFO:tasks.rados.rados.0.smithi038.stdout:919: finishing write tid 4 to smithi03829856-38 2024-03-19T11:30:37.060 INFO:tasks.rados.rados.0.smithi038.stdout:919: oid 38 updating version 0 to 63 2024-03-19T11:30:37.060 INFO:tasks.rados.rados.0.smithi038.stdout:919: oid 38 updating version 63 to 64 2024-03-19T11:30:37.060 INFO:tasks.rados.rados.0.smithi038.stdout:919: oid 38 version 64 is already newer than 62 2024-03-19T11:30:37.060 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 38 v 64 (ObjNum 125 snap 15 seq_num 125) dirty exists 2024-03-19T11:30:37.060 INFO:tasks.rados.rados.0.smithi038.stdout:919: left oid 38 (ObjNum 125 snap 15 seq_num 125) 2024-03-19T11:30:37.060 INFO:tasks.rados.rados.0.smithi038.stdout:920: expect (ObjNum 14 snap 0 seq_num 14) 2024-03-19T11:30:37.110 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:37.119+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:37.123 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:37.133+0000 7f1835403640 -1 osd.6 59 handle_osd_map injecting map crc failure 2024-03-19T11:30:37.132 INFO:tasks.rados.rados.0.smithi038.stdout:924: finishing rollback tid 0 to smithi03829856-43 2024-03-19T11:30:37.132 INFO:tasks.rados.rados.0.smithi038.stdout:924: finishing rollback tid 1 to smithi03829856-43 2024-03-19T11:30:37.132 INFO:tasks.rados.rados.0.smithi038.stdout:924: finishing rollback tid 2 to smithi03829856-43 2024-03-19T11:30:37.132 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 43 v 92 (ObjNum 91 snap 10 seq_num 91) dirty exists 2024-03-19T11:30:37.132 INFO:tasks.rados.rados.0.smithi038.stdout:923: expect (ObjNum 59 snap 3 seq_num 59) 2024-03-19T11:30:37.188 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:37.188 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:37.188 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:30:37.188 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:37.192 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:30:37.192 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:30:37.201 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-19T11:30:37.208 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:30:37.208 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:30:37.211 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:37.223+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:37.224 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:30:37.224 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:30:37.240 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:30:37.241 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:30:37.258 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:30:37.258 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:30:37.263 INFO:tasks.rados.rados.0.smithi038.stdout:922: finishing write tid 1 to smithi03829856-39 2024-03-19T11:30:37.263 INFO:tasks.rados.rados.0.smithi038.stdout:922: finishing write tid 2 to smithi03829856-39 2024-03-19T11:30:37.263 INFO:tasks.rados.rados.0.smithi038.stdout:922: finishing write tid 3 to smithi03829856-39 2024-03-19T11:30:37.263 INFO:tasks.rados.rados.0.smithi038.stdout:922: finishing write tid 4 to smithi03829856-39 2024-03-19T11:30:37.263 INFO:tasks.rados.rados.0.smithi038.stdout:922: oid 39 updating version 0 to 55 2024-03-19T11:30:37.263 INFO:tasks.rados.rados.0.smithi038.stdout:922: oid 39 updating version 55 to 56 2024-03-19T11:30:37.263 INFO:tasks.rados.rados.0.smithi038.stdout:922: oid 39 updating version 56 to 57 2024-03-19T11:30:37.263 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 39 v 57 (ObjNum 126 snap 15 seq_num 126) dirty exists 2024-03-19T11:30:37.263 INFO:tasks.rados.rados.0.smithi038.stdout:922: left oid 39 (ObjNum 126 snap 15 seq_num 126) 2024-03-19T11:30:37.263 INFO:tasks.rados.rados.0.smithi038.stdout:925: seq_num 127 ranges {534489=618271,1565566=568923} 2024-03-19T11:30:37.269 INFO:tasks.rados.rados.0.smithi038.stdout:925: writing smithi03829856-1 from 534489 to 1152760 tid 1 2024-03-19T11:30:37.276 INFO:tasks.rados.rados.0.smithi038.stdout:925: writing smithi03829856-1 from 1565566 to 2134489 tid 2 2024-03-19T11:30:37.277 INFO:tasks.rados.rados.0.smithi038.stdout:911: done (13 left) 2024-03-19T11:30:37.277 INFO:tasks.rados.rados.0.smithi038.stdout:913: done (12 left) 2024-03-19T11:30:37.277 INFO:tasks.rados.rados.0.smithi038.stdout:914: done (11 left) 2024-03-19T11:30:37.277 INFO:tasks.rados.rados.0.smithi038.stdout:915: done (10 left) 2024-03-19T11:30:37.277 INFO:tasks.rados.rados.0.smithi038.stdout:916: done (9 left) 2024-03-19T11:30:37.277 INFO:tasks.rados.rados.0.smithi038.stdout:917: done (8 left) 2024-03-19T11:30:37.277 INFO:tasks.rados.rados.0.smithi038.stdout:918: done (7 left) 2024-03-19T11:30:37.277 INFO:tasks.rados.rados.0.smithi038.stdout:919: done (6 left) 2024-03-19T11:30:37.277 INFO:tasks.rados.rados.0.smithi038.stdout:920: done (5 left) 2024-03-19T11:30:37.277 INFO:tasks.rados.rados.0.smithi038.stdout:921: done (4 left) 2024-03-19T11:30:37.277 INFO:tasks.rados.rados.0.smithi038.stdout:922: done (3 left) 2024-03-19T11:30:37.277 INFO:tasks.rados.rados.0.smithi038.stdout:923: done (2 left) 2024-03-19T11:30:37.277 INFO:tasks.rados.rados.0.smithi038.stdout:924: done (1 left) 2024-03-19T11:30:37.278 INFO:tasks.rados.rados.0.smithi038.stdout:926: tier_flush oid 44 2024-03-19T11:30:37.278 INFO:tasks.rados.rados.0.smithi038.stdout:926: tier_flush oid 44 snap -1 2024-03-19T11:30:37.278 INFO:tasks.rados.rados.0.smithi038.stdout:927: rollback oid 33 current snap is 15 2024-03-19T11:30:37.278 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 33 to 12 2024-03-19T11:30:37.278 INFO:tasks.rados.rados.0.smithi038.stdout:928: write (excl) oid 19 current snap is 15 2024-03-19T11:30:37.278 INFO:tasks.rados.rados.0.smithi038.stdout:928: seq_num 128 ranges {551296=693520,2024048=127248} 2024-03-19T11:30:37.278 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:30:37.279 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:30:37.285 INFO:tasks.rados.rados.0.smithi038.stdout:928: writing smithi03829856-19 from 551296 to 1244816 tid 1 2024-03-19T11:30:37.294 INFO:tasks.rados.rados.0.smithi038.stdout:928: writing smithi03829856-19 from 2024048 to 2151296 tid 2 2024-03-19T11:30:37.294 INFO:tasks.rados.rados.0.smithi038.stdout:929: tier_evict oid 29 2024-03-19T11:30:37.294 INFO:tasks.rados.rados.0.smithi038.stdout:929: tier_evict oid 29 snap -1 2024-03-19T11:30:37.294 INFO:tasks.rados.rados.0.smithi038.stdout:927: finishing rollback tid 0 to smithi03829856-33 2024-03-19T11:30:37.294 INFO:tasks.rados.rados.0.smithi038.stdout:930: write oid 21 current snap is 15 2024-03-19T11:30:37.294 INFO:tasks.rados.rados.0.smithi038.stdout:927: finishing rollback tid 1 to smithi03829856-33 2024-03-19T11:30:37.295 INFO:tasks.rados.rados.0.smithi038.stdout:927: finishing rollback tid 2 to smithi03829856-33 2024-03-19T11:30:37.295 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 33 v 61 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-03-19T11:30:37.295 INFO:tasks.rados.rados.0.smithi038.stdout:925: finishing write tid 1 to smithi03829856-1 2024-03-19T11:30:37.295 INFO:tasks.rados.rados.0.smithi038.stdout:925: finishing write tid 2 to smithi03829856-1 2024-03-19T11:30:37.295 INFO:tasks.rados.rados.0.smithi038.stdout:930: seq_num 129 ranges {568103=768769,2082530=85573} 2024-03-19T11:30:37.295 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:30:37.295 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:30:37.302 INFO:tasks.rados.rados.0.smithi038.stdout:930: writing smithi03829856-21 from 568103 to 1336872 tid 1 2024-03-19T11:30:37.306 INFO:tasks.rados.rados.0.smithi038.stdout:930: writing smithi03829856-21 from 2082530 to 2168103 tid 2 2024-03-19T11:30:37.306 INFO:tasks.rados.rados.0.smithi038.stdout:927: done (5 left) 2024-03-19T11:30:37.307 INFO:tasks.rados.rados.0.smithi038.stdout:931: read oid 13 snap -1 2024-03-19T11:30:37.307 INFO:tasks.rados.rados.0.smithi038.stdout:931: expect (ObjNum 12 snap 0 seq_num 12) 2024-03-19T11:30:37.307 INFO:tasks.rados.rados.0.smithi038.stdout:932: read oid 2 snap -1 2024-03-19T11:30:37.307 INFO:tasks.rados.rados.0.smithi038.stdout:932: expect (ObjNum 117 snap 13 seq_num 117) 2024-03-19T11:30:37.307 INFO:tasks.rados.rados.0.smithi038.stdout:933: read oid 47 snap 11 2024-03-19T11:30:37.307 INFO:tasks.rados.rados.0.smithi038.stdout:933: expect (ObjNum 46 snap 0 seq_num 46) 2024-03-19T11:30:37.307 INFO:tasks.rados.rados.0.smithi038.stdout:934: read oid 4 snap -1 2024-03-19T11:30:37.307 INFO:tasks.rados.rados.0.smithi038.stdout:934: expect (ObjNum 3 snap 0 seq_num 3) 2024-03-19T11:30:37.307 INFO:tasks.rados.rados.0.smithi038.stdout:935: snap_remove snap 12 2024-03-19T11:30:37.313 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:37.323+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:37.314 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:30:37.314 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:30:37.325 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:37.325 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:37.339 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-19T11:30:37.413 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:37.421+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:37.458 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:37.459 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:37.472 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-19T11:30:37.519 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:37.531+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:37.601 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:37.601 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:37.616 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-19T11:30:37.616 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:37.627+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:37.717 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:37.725+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:37.780 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:37.780 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:37.795 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-19T11:30:37.839 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:37.851+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:37.915 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:37.915 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:37.919 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:37.929+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:37.931 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-19T11:30:38.024 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:38.035+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:38.028 INFO:tasks.rados.rados.0.smithi038.stdout:935: done (9 left) 2024-03-19T11:30:38.028 INFO:tasks.rados.rados.0.smithi038.stdout:936: rollback oid 27 current snap is 15 2024-03-19T11:30:38.028 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 27 to 10 2024-03-19T11:30:38.028 INFO:tasks.rados.rados.0.smithi038.stdout:925: finishing write tid 3 to smithi03829856-1 2024-03-19T11:30:38.028 INFO:tasks.rados.rados.0.smithi038.stdout:925: finishing write tid 4 to smithi03829856-1 2024-03-19T11:30:38.029 INFO:tasks.rados.rados.0.smithi038.stdout:925: oid 1 updating version 0 to 87 2024-03-19T11:30:38.029 INFO:tasks.rados.rados.0.smithi038.stdout:925: oid 1 updating version 87 to 89 2024-03-19T11:30:38.029 INFO:tasks.rados.rados.0.smithi038.stdout:925: oid 1 version 89 is already newer than 88 2024-03-19T11:30:38.029 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 1 v 89 (ObjNum 127 snap 15 seq_num 127) dirty exists 2024-03-19T11:30:38.029 INFO:tasks.rados.rados.0.smithi038.stdout:925: left oid 1 (ObjNum 127 snap 15 seq_num 127) 2024-03-19T11:30:38.029 INFO:tasks.rados.rados.0.smithi038.stdout:929: got (22) Invalid argument 2024-03-19T11:30:38.029 INFO:tasks.rados.rados.0.smithi038.stdout:930: finishing write tid 1 to smithi03829856-21 2024-03-19T11:30:38.029 INFO:tasks.rados.rados.0.smithi038.stdout:933: expect (ObjNum 46 snap 0 seq_num 46) 2024-03-19T11:30:38.076 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:38.087+0000 7fa5d3465640 -1 snap_mapper.get_next_objects_to_trim::New Clone-Objects were added to Snap e after trimming was started 2024-03-19T11:30:38.120 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:38.129+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:38.129 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:38.129 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:38.146 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-03-19T11:30:38.150 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-19T11:30:38.155 INFO:tasks.rados.rados.0.smithi038.stdout:934: expect (ObjNum 3 snap 0 seq_num 3) 2024-03-19T11:30:38.222 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:38.230+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:38.245 INFO:tasks.rados.rados.0.smithi038.stdout:931: expect (ObjNum 12 snap 0 seq_num 12) 2024-03-19T11:30:38.321 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:38.329+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:38.325 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:38.325 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:38.335 INFO:tasks.rados.rados.0.smithi038.stdout:930: finishing write tid 2 to smithi03829856-21 2024-03-19T11:30:38.335 INFO:tasks.rados.rados.0.smithi038.stdout:930: finishing write tid 3 to smithi03829856-21 2024-03-19T11:30:38.335 INFO:tasks.rados.rados.0.smithi038.stdout:930: finishing write tid 4 to smithi03829856-21 2024-03-19T11:30:38.335 INFO:tasks.rados.rados.0.smithi038.stdout:930: oid 21 updating version 0 to 84 2024-03-19T11:30:38.335 INFO:tasks.rados.rados.0.smithi038.stdout:930: oid 21 updating version 84 to 85 2024-03-19T11:30:38.335 INFO:tasks.rados.rados.0.smithi038.stdout:930: oid 21 updating version 85 to 86 2024-03-19T11:30:38.335 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 21 v 86 (ObjNum 129 snap 15 seq_num 129) dirty exists 2024-03-19T11:30:38.335 INFO:tasks.rados.rados.0.smithi038.stdout:930: left oid 21 (ObjNum 129 snap 15 seq_num 129) 2024-03-19T11:30:38.336 INFO:tasks.rados.rados.0.smithi038.stdout:932: expect (ObjNum 117 snap 13 seq_num 117) 2024-03-19T11:30:38.340 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-19T11:30:38.422 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:38.433+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:38.523 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:38.534+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:38.524 INFO:tasks.rados.rados.0.smithi038.stdout:925: done (9 left) 2024-03-19T11:30:38.524 INFO:tasks.rados.rados.0.smithi038.stdout:929: done (8 left) 2024-03-19T11:30:38.524 INFO:tasks.rados.rados.0.smithi038.stdout:930: done (7 left) 2024-03-19T11:30:38.524 INFO:tasks.rados.rados.0.smithi038.stdout:931: done (6 left) 2024-03-19T11:30:38.524 INFO:tasks.rados.rados.0.smithi038.stdout:932: done (5 left) 2024-03-19T11:30:38.524 INFO:tasks.rados.rados.0.smithi038.stdout:933: done (4 left) 2024-03-19T11:30:38.524 INFO:tasks.rados.rados.0.smithi038.stdout:934: done (3 left) 2024-03-19T11:30:38.524 INFO:tasks.rados.rados.0.smithi038.stdout:937: delete oid 2 current snap is 15 2024-03-19T11:30:38.524 INFO:tasks.rados.rados.0.smithi038.stdout:926: got (16) Device or resource busy 2024-03-19T11:30:38.525 INFO:tasks.rados.rados.0.smithi038.stdout:check_oldest_snap_flushed oid 44 v 50 (ObjNum 108 snap 11 seq_num 108) unflushed 11 2024-03-19T11:30:38.525 INFO:tasks.rados.rados.0.smithi038.stdout:936: finishing rollback tid 1 to smithi03829856-27 2024-03-19T11:30:38.525 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 27 v 34 (ObjNum 129 snap 0 seq_num 1055832672) dirty dne 2024-03-19T11:30:38.525 INFO:tasks.rados.rados.0.smithi038.stdout:928: finishing write tid 1 to smithi03829856-19 2024-03-19T11:30:38.525 INFO:tasks.rados.rados.0.smithi038.stdout:928: finishing write tid 2 to smithi03829856-19 2024-03-19T11:30:38.525 INFO:tasks.rados.rados.0.smithi038.stdout:928: finishing write tid 3 to smithi03829856-19 2024-03-19T11:30:38.525 INFO:tasks.rados.rados.0.smithi038.stdout:928: finishing write tid 4 to smithi03829856-19 2024-03-19T11:30:38.525 INFO:tasks.rados.rados.0.smithi038.stdout:928: oid 19 updating version 0 to 49 2024-03-19T11:30:38.525 INFO:tasks.rados.rados.0.smithi038.stdout:928: oid 19 updating version 49 to 50 2024-03-19T11:30:38.525 INFO:tasks.rados.rados.0.smithi038.stdout:928: oid 19 version 50 is already newer than 48 2024-03-19T11:30:38.525 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 19 v 50 (ObjNum 128 snap 15 seq_num 128) dirty exists 2024-03-19T11:30:38.525 INFO:tasks.rados.rados.0.smithi038.stdout:928: left oid 19 (ObjNum 128 snap 15 seq_num 128) 2024-03-19T11:30:38.527 INFO:tasks.rados.rados.0.smithi038.stdout:926: done (3 left) 2024-03-19T11:30:38.527 INFO:tasks.rados.rados.0.smithi038.stdout:928: done (2 left) 2024-03-19T11:30:38.527 INFO:tasks.rados.rados.0.smithi038.stdout:936: done (1 left) 2024-03-19T11:30:38.527 INFO:tasks.rados.rados.0.smithi038.stdout:937: done (0 left) 2024-03-19T11:30:38.527 INFO:tasks.rados.rados.0.smithi038.stdout:938: read oid 21 snap -1 2024-03-19T11:30:38.527 INFO:tasks.rados.rados.0.smithi038.stdout:938: expect (ObjNum 129 snap 15 seq_num 129) 2024-03-19T11:30:38.527 INFO:tasks.rados.rados.0.smithi038.stdout:939: read oid 38 snap -1 2024-03-19T11:30:38.527 INFO:tasks.rados.rados.0.smithi038.stdout:939: expect (ObjNum 125 snap 15 seq_num 125) 2024-03-19T11:30:38.528 INFO:tasks.rados.rados.0.smithi038.stdout:940: rollback oid 33 current snap is 15 2024-03-19T11:30:38.528 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 33 to 11 2024-03-19T11:30:38.528 INFO:tasks.rados.rados.0.smithi038.stdout:941: tier_evict oid 41 2024-03-19T11:30:38.528 INFO:tasks.rados.rados.0.smithi038.stdout:941: tier_evict oid 41 snap -1 2024-03-19T11:30:38.528 INFO:tasks.rados.rados.0.smithi038.stdout:942: set_chunk oid 32 target oid 2024-03-19T11:30:38.539 INFO:tasks.rados.rados.0.smithi038.stdout:942: set_chunk oid 32 offset: 105472 length: 1013248 target oid b0eaaa0ee9c1071b958c275730b1d242af6609cdf78b3d21fba94f31a3aeb31b offset: 0 snap -1 2024-03-19T11:30:38.539 INFO:tasks.rados.rados.0.smithi038.stdout:943: write (excl) oid 48 current snap is 15 2024-03-19T11:30:38.540 INFO:tasks.rados.rados.0.smithi038.stdout:943: seq_num 131 ranges {601717=635620,1799494=402223} 2024-03-19T11:30:38.547 INFO:tasks.rados.rados.0.smithi038.stdout:943: writing smithi03829856-48 from 601717 to 1237337 tid 1 2024-03-19T11:30:38.554 INFO:tasks.rados.rados.0.smithi038.stdout:943: writing smithi03829856-48 from 1799494 to 2201717 tid 2 2024-03-19T11:30:38.554 INFO:tasks.rados.rados.0.smithi038.stdout:944: read oid 43 snap -1 2024-03-19T11:30:38.554 INFO:tasks.rados.rados.0.smithi038.stdout:944: expect (ObjNum 91 snap 10 seq_num 91) 2024-03-19T11:30:38.555 INFO:tasks.rados.rados.0.smithi038.stdout:945: tier_promote oid 34 2024-03-19T11:30:38.555 INFO:tasks.rados.rados.0.smithi038.stdout:946: write (excl) oid 23 current snap is 15 2024-03-19T11:30:38.555 INFO:tasks.rados.rados.0.smithi038.stdout:946: seq_num 132 ranges {618524=710869,1857976=360548} 2024-03-19T11:30:38.566 INFO:tasks.rados.rados.0.smithi038.stdout:946: writing smithi03829856-23 from 618524 to 1329393 tid 1 2024-03-19T11:30:38.567 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:38.567 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:38.570 INFO:tasks.rados.rados.0.smithi038.stdout:946: writing smithi03829856-23 from 1857976 to 2218524 tid 2 2024-03-19T11:30:38.570 INFO:tasks.rados.rados.0.smithi038.stdout:941: got (22) Invalid argument 2024-03-19T11:30:38.570 INFO:tasks.rados.rados.0.smithi038.stdout:940: finishing rollback tid 0 to smithi03829856-33 2024-03-19T11:30:38.570 INFO:tasks.rados.rados.0.smithi038.stdout:940: finishing rollback tid 1 to smithi03829856-33 2024-03-19T11:30:38.570 INFO:tasks.rados.rados.0.smithi038.stdout:939: expect (ObjNum 125 snap 15 seq_num 125) 2024-03-19T11:30:38.585 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-19T11:30:38.628 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:38.638+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:38.676 INFO:tasks.rados.rados.0.smithi038.stdout:940: finishing rollback tid 2 to smithi03829856-33 2024-03-19T11:30:38.677 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 33 v 66 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-03-19T11:30:38.677 INFO:tasks.rados.rados.0.smithi038.stdout:938: expect (ObjNum 129 snap 15 seq_num 129) 2024-03-19T11:30:38.725 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:38.733+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:38.784 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:38.784 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:38.797 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-19T11:30:38.821 INFO:tasks.rados.rados.0.smithi038.stdout:942: finishing set_chunk to oid 32 2024-03-19T11:30:38.821 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 32 set_chunk returned error code -95 offset: 105472 length: 1013248 2024-03-19T11:30:38.821 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-03-19T11:30:38.821 INFO:tasks.rados.rados.0.smithi038.stdout:943: finishing write tid 1 to smithi03829856-48 2024-03-19T11:30:38.821 INFO:tasks.rados.rados.0.smithi038.stdout:943: finishing write tid 2 to smithi03829856-48 2024-03-19T11:30:38.821 INFO:tasks.rados.rados.0.smithi038.stdout:943: finishing write tid 3 to smithi03829856-48 2024-03-19T11:30:38.822 INFO:tasks.rados.rados.0.smithi038.stdout:943: finishing write tid 4 to smithi03829856-48 2024-03-19T11:30:38.822 INFO:tasks.rados.rados.0.smithi038.stdout:943: oid 48 updating version 0 to 76 2024-03-19T11:30:38.822 INFO:tasks.rados.rados.0.smithi038.stdout:943: oid 48 version 76 is already newer than 75 2024-03-19T11:30:38.822 INFO:tasks.rados.rados.0.smithi038.stdout:943: oid 48 version 76 is already newer than 74 2024-03-19T11:30:38.822 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 48 v 76 (ObjNum 131 snap 15 seq_num 131) dirty exists 2024-03-19T11:30:38.822 INFO:tasks.rados.rados.0.smithi038.stdout:943: left oid 48 (ObjNum 131 snap 15 seq_num 131) 2024-03-19T11:30:38.822 INFO:tasks.rados.rados.0.smithi038.stdout:944: expect (ObjNum 91 snap 10 seq_num 91) 2024-03-19T11:30:38.827 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:38.834+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:38.928 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:38.938+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:38.937 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:38.938 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:38.953 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-19T11:30:39.030 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:39.036+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:39.095 INFO:tasks.rados.rados.0.smithi038.stdout:946: finishing write tid 1 to smithi03829856-23 2024-03-19T11:30:39.096 INFO:tasks.rados.rados.0.smithi038.stdout:946: finishing write tid 2 to smithi03829856-23 2024-03-19T11:30:39.096 INFO:tasks.rados.rados.0.smithi038.stdout:938: done (8 left) 2024-03-19T11:30:39.096 INFO:tasks.rados.rados.0.smithi038.stdout:939: done (7 left) 2024-03-19T11:30:39.096 INFO:tasks.rados.rados.0.smithi038.stdout:940: done (6 left) 2024-03-19T11:30:39.096 INFO:tasks.rados.rados.0.smithi038.stdout:941: done (5 left) 2024-03-19T11:30:39.096 INFO:tasks.rados.rados.0.smithi038.stdout:942: done (4 left) 2024-03-19T11:30:39.096 INFO:tasks.rados.rados.0.smithi038.stdout:943: done (3 left) 2024-03-19T11:30:39.096 INFO:tasks.rados.rados.0.smithi038.stdout:944: done (2 left) 2024-03-19T11:30:39.096 INFO:tasks.rados.rados.0.smithi038.stdout:947: write oid 39 current snap is 15 2024-03-19T11:30:39.096 INFO:tasks.rados.rados.0.smithi038.stdout:947: seq_num 133 ranges {635331=786118,1916458=318873} 2024-03-19T11:30:39.103 INFO:tasks.rados.rados.0.smithi038.stdout:947: writing smithi03829856-39 from 635331 to 1421449 tid 1 2024-03-19T11:30:39.108 INFO:tasks.rados.rados.0.smithi038.stdout:947: writing smithi03829856-39 from 1916458 to 2235331 tid 2 2024-03-19T11:30:39.108 INFO:tasks.rados.rados.0.smithi038.stdout:948: delete oid 12 current snap is 15 2024-03-19T11:30:39.108 INFO:tasks.rados.rados.0.smithi038.stdout:946: finishing write tid 3 to smithi03829856-23 2024-03-19T11:30:39.108 INFO:tasks.rados.rados.0.smithi038.stdout:946: finishing write tid 4 to smithi03829856-23 2024-03-19T11:30:39.108 INFO:tasks.rados.rados.0.smithi038.stdout:946: oid 23 updating version 0 to 67 2024-03-19T11:30:39.108 INFO:tasks.rados.rados.0.smithi038.stdout:946: oid 23 version 67 is already newer than 66 2024-03-19T11:30:39.109 INFO:tasks.rados.rados.0.smithi038.stdout:946: oid 23 updating version 67 to 68 2024-03-19T11:30:39.109 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 23 v 68 (ObjNum 132 snap 15 seq_num 132) dirty exists 2024-03-19T11:30:39.109 INFO:tasks.rados.rados.0.smithi038.stdout:946: left oid 23 (ObjNum 132 snap 15 seq_num 132) 2024-03-19T11:30:39.116 INFO:tasks.rados.rados.0.smithi038.stdout:946: done (3 left) 2024-03-19T11:30:39.116 INFO:tasks.rados.rados.0.smithi038.stdout:948: done (2 left) 2024-03-19T11:30:39.116 INFO:tasks.rados.rados.0.smithi038.stdout:949: rollback oid 10 current snap is 15 2024-03-19T11:30:39.116 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 10 to 9 2024-03-19T11:30:39.116 INFO:tasks.rados.rados.0.smithi038.stdout:950: tier_flush oid 1 2024-03-19T11:30:39.116 INFO:tasks.rados.rados.0.smithi038.stdout:950: tier_flush oid 1 snap -1 2024-03-19T11:30:39.116 INFO:tasks.rados.rados.0.smithi038.stdout:951: tier_evict oid 26 2024-03-19T11:30:39.116 INFO:tasks.rados.rados.0.smithi038.stdout:951: tier_evict oid 26 snap -1 2024-03-19T11:30:39.117 INFO:tasks.rados.rados.0.smithi038.stdout:952: read oid 25 snap -1 2024-03-19T11:30:39.117 INFO:tasks.rados.rados.0.smithi038.stdout:952: expect deleted 2024-03-19T11:30:39.117 INFO:tasks.rados.rados.0.smithi038.stdout:953: read oid 16 snap 14 2024-03-19T11:30:39.117 INFO:tasks.rados.rados.0.smithi038.stdout:953: expect (ObjNum 118 snap 13 seq_num 118) 2024-03-19T11:30:39.117 INFO:tasks.rados.rados.0.smithi038.stdout:954: read oid 17 snap -1 2024-03-19T11:30:39.117 INFO:tasks.rados.rados.0.smithi038.stdout:954: expect (ObjNum 96 snap 10 seq_num 96) 2024-03-19T11:30:39.117 INFO:tasks.rados.rados.0.smithi038.stdout:955: read oid 8 snap -1 2024-03-19T11:30:39.117 INFO:tasks.rados.rados.0.smithi038.stdout:955: expect (ObjNum 60 snap 3 seq_num 60) 2024-03-19T11:30:39.117 INFO:tasks.rados.rados.0.smithi038.stdout:956: tier_promote oid 12 2024-03-19T11:30:39.118 INFO:tasks.rados.rados.0.smithi038.stdout:957: read oid 47 snap -1 2024-03-19T11:30:39.118 INFO:tasks.rados.rados.0.smithi038.stdout:957: expect (ObjNum 46 snap 0 seq_num 46) 2024-03-19T11:30:39.118 INFO:tasks.rados.rados.0.smithi038.stdout:958: set_chunk oid 24 target oid 2024-03-19T11:30:39.126 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:39.126 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:39.130 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:39.138+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:39.132 INFO:tasks.rados.rados.0.smithi038.stdout:958: set_chunk oid 24 offset: 508928 length: 956928 target oid ab30d9a1f7c615ba997c0384c04f39751a33c69c044fd060a4c6716a29c4cef0 offset: 0 snap 11 2024-03-19T11:30:39.132 INFO:tasks.rados.rados.0.smithi038.stdout:959: read oid 19 snap -1 2024-03-19T11:30:39.132 INFO:tasks.rados.rados.0.smithi038.stdout:959: expect (ObjNum 128 snap 15 seq_num 128) 2024-03-19T11:30:39.132 INFO:tasks.rados.rados.0.smithi038.stdout:947: finishing write tid 1 to smithi03829856-39 2024-03-19T11:30:39.132 INFO:tasks.rados.rados.0.smithi038.stdout:960: read oid 7 snap -1 2024-03-19T11:30:39.132 INFO:tasks.rados.rados.0.smithi038.stdout:960: expect (ObjNum 67 snap 3 seq_num 67) 2024-03-19T11:30:39.132 INFO:tasks.rados.rados.0.smithi038.stdout:961: write (excl) oid 9 current snap is 15 2024-03-19T11:30:39.133 INFO:tasks.rados.rados.0.smithi038.stdout:961: seq_num 135 ranges {668945=536616,1633422=491363} 2024-03-19T11:30:39.141 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-03-19T11:30:39.142 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:39.148+0000 7f143d8b8640 -1 osd.5 61 handle_osd_map injecting map crc failure 2024-03-19T11:30:39.144 INFO:tasks.rados.rados.0.smithi038.stdout:961: writing smithi03829856-9 from 668945 to 1205561 tid 1 2024-03-19T11:30:39.153 INFO:tasks.rados.rados.0.smithi038.stdout:961: writing smithi03829856-9 from 1633422 to 2124785 tid 2 2024-03-19T11:30:39.154 INFO:tasks.rados.rados.0.smithi038.stdout:950: got (16) Device or resource busy 2024-03-19T11:30:39.154 INFO:tasks.rados.rados.0.smithi038.stdout:check_oldest_snap_flushed oid 1 v 89 (ObjNum 127 snap 15 seq_num 127) unflushed 15 2024-03-19T11:30:39.154 INFO:tasks.rados.rados.0.smithi038.stdout:951: got (22) Invalid argument 2024-03-19T11:30:39.154 INFO:tasks.rados.rados.0.smithi038.stdout:947: finishing write tid 2 to smithi03829856-39 2024-03-19T11:30:39.154 INFO:tasks.rados.rados.0.smithi038.stdout:947: finishing write tid 3 to smithi03829856-39 2024-03-19T11:30:39.154 INFO:tasks.rados.rados.0.smithi038.stdout:947: finishing write tid 4 to smithi03829856-39 2024-03-19T11:30:39.154 INFO:tasks.rados.rados.0.smithi038.stdout:947: oid 39 updating version 0 to 58 2024-03-19T11:30:39.154 INFO:tasks.rados.rados.0.smithi038.stdout:947: oid 39 updating version 58 to 60 2024-03-19T11:30:39.154 INFO:tasks.rados.rados.0.smithi038.stdout:947: oid 39 version 60 is already newer than 59 2024-03-19T11:30:39.154 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 39 v 60 (ObjNum 133 snap 15 seq_num 133) dirty exists 2024-03-19T11:30:39.154 INFO:tasks.rados.rados.0.smithi038.stdout:947: left oid 39 (ObjNum 133 snap 15 seq_num 133) 2024-03-19T11:30:39.154 INFO:tasks.rados.rados.0.smithi038.stdout:949: finishing rollback tid 0 to smithi03829856-10 2024-03-19T11:30:39.154 INFO:tasks.rados.rados.0.smithi038.stdout:949: finishing rollback tid 1 to smithi03829856-10 2024-03-19T11:30:39.154 INFO:tasks.rados.rados.0.smithi038.stdout:949: finishing rollback tid 2 to smithi03829856-10 2024-03-19T11:30:39.155 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 10 v 70 (ObjNum 55 snap 1 seq_num 55) dirty exists 2024-03-19T11:30:39.155 INFO:tasks.rados.rados.0.smithi038.stdout:953: expect (ObjNum 118 snap 13 seq_num 118) 2024-03-19T11:30:39.156 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-19T11:30:39.263 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:39.274+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:39.333 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:39.342+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:39.333 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-19T11:30:39.355 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:39.355 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:39.380 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-19T11:30:39.417 INFO:tasks.rados.rados.0.smithi038.stdout:957: expect (ObjNum 46 snap 0 seq_num 46) 2024-03-19T11:30:39.463 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:39.474+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:39.525 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:30:39.525 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:30:39.534 INFO:tasks.rados.rados.0.smithi038.stdout:956: got (0) Success 2024-03-19T11:30:39.535 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 12 v 72 (ObjNum 1055005168 snap 22082 seq_num 304) clean dne 2024-03-19T11:30:39.535 INFO:tasks.rados.rados.0.smithi038.stdout:958: finishing set_chunk to oid 24 2024-03-19T11:30:39.535 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 24 set_chunk returned error code -95 offset: 508928 length: 956928 2024-03-19T11:30:39.535 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-03-19T11:30:39.535 INFO:tasks.rados.rados.0.smithi038.stdout:960: expect (ObjNum 67 snap 3 seq_num 67) 2024-03-19T11:30:39.536 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:39.542+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:39.556 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:30:39.556 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:30:39.577 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:39.578 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:39.588 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:30:39.588 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:30:39.594 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-19T11:30:39.615 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:30:39.616 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:30:39.630 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:39.638+0000 7fa5de8d7640 -1 osd.3 61 handle_osd_map injecting map crc failure 2024-03-19T11:30:39.635 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:39.646+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:39.642 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:30:39.642 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:30:39.663 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:30:39.664 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:30:39.681 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:30:39.682 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:30:39.699 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:30:39.700 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:30:39.704 INFO:tasks.rados.rados.0.smithi038.stdout:959: expect (ObjNum 128 snap 15 seq_num 128) 2024-03-19T11:30:39.751 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:39.751 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:30:39.751 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:30:39.751 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:30:39.752 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.752 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:55 3.2 3:525f009b:::a57e3dbafedf1ec1a7132467f167cfc4a4343cb541b36fc481376f0086aece71:head [read 0~121882 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:39.752 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.761904+0000", 2024-03-19T11:30:39.752 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 27.000157148, 2024-03-19T11:30:39.752 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010829357, 2024-03-19T11:30:39.752 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:39.752 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:39.752 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:39.752 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.752 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:39.753 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761904+0000", 2024-03-19T11:30:39.753 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:39.753 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.753 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.753 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:39.753 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761904+0000", 2024-03-19T11:30:39.753 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:39.753 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.753 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.753 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:39.753 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761905+0000", 2024-03-19T11:30:39.753 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4900000000000005e-07 2024-03-19T11:30:39.753 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.753 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.753 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:39.754 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761917+0000", 2024-03-19T11:30:39.754 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1806e-05 2024-03-19T11:30:39.754 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.754 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.754 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:39.754 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761934+0000", 2024-03-19T11:30:39.754 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6509000000000001e-05 2024-03-19T11:30:39.754 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.754 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.754 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:39.754 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761941+0000", 2024-03-19T11:30:39.754 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8110000000000003e-06 2024-03-19T11:30:39.754 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.754 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.755 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:39.755 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762115+0000", 2024-03-19T11:30:39.755 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017358200000000001 2024-03-19T11:30:39.755 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.755 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.755 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:39.755 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762120+0000", 2024-03-19T11:30:39.755 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3630000000000002e-06 2024-03-19T11:30:39.755 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.755 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.755 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:39.755 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.765567+0000", 2024-03-19T11:30:39.755 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034469990000000001 2024-03-19T11:30:39.755 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.756 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.756 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:39.756 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.765665+0000", 2024-03-19T11:30:39.756 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7476e-05 2024-03-19T11:30:39.756 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.756 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.756 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:39.756 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.772734+0000", 2024-03-19T11:30:39.756 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0070690620000000001 2024-03-19T11:30:39.756 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:39.756 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:39.756 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:39.756 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.756 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.756 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:56 3.2 3:5aa75d1e:::b3c26d797de4ffcce4e862009e6b8eb8576a53cb86ea102ce51145c0038aeb97:head [read 0~524288 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:39.757 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.761958+0000", 2024-03-19T11:30:39.757 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 27.000103747000001, 2024-03-19T11:30:39.757 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012078836000000001, 2024-03-19T11:30:39.757 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:39.757 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:39.757 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:39.757 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.757 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:39.757 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761958+0000", 2024-03-19T11:30:39.757 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:39.757 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.757 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.757 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:39.757 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761958+0000", 2024-03-19T11:30:39.758 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:39.758 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.758 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.758 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:39.758 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761958+0000", 2024-03-19T11:30:39.758 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9900000000000001e-07 2024-03-19T11:30:39.758 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.758 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.758 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:39.758 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761961+0000", 2024-03-19T11:30:39.758 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9280000000000002e-06 2024-03-19T11:30:39.758 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.758 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.759 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:39.759 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761969+0000", 2024-03-19T11:30:39.759 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1440000000000003e-06 2024-03-19T11:30:39.759 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.759 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.759 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:39.759 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761976+0000", 2024-03-19T11:30:39.759 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1020000000000002e-06 2024-03-19T11:30:39.759 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.759 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.759 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:39.759 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762144+0000", 2024-03-19T11:30:39.759 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016818200000000001 2024-03-19T11:30:39.759 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.759 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.760 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:39.760 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762148+0000", 2024-03-19T11:30:39.760 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0969999999999999e-06 2024-03-19T11:30:39.760 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.760 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.760 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:39.760 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.772788+0000", 2024-03-19T11:30:39.760 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010640124000000001 2024-03-19T11:30:39.760 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.760 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.760 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:39.760 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.772898+0000", 2024-03-19T11:30:39.760 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000110512 2024-03-19T11:30:39.760 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.760 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.760 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:39.761 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.774037+0000", 2024-03-19T11:30:39.761 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011382479999999999 2024-03-19T11:30:39.761 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:39.761 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:39.761 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:39.761 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.761 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.761 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:57 3.2 3:6d82e5d5:::455c613336f373ae48b33af990f91f317265a0c5a920cc1b46533fdb41d8319c:head [read 0~380357 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:39.761 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.761987+0000", 2024-03-19T11:30:39.761 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 27.000074784999999, 2024-03-19T11:30:39.761 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012691991999999999, 2024-03-19T11:30:39.761 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:39.761 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:39.761 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:39.761 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.762 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:39.762 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761987+0000", 2024-03-19T11:30:39.762 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:39.762 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.762 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.762 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:39.762 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761987+0000", 2024-03-19T11:30:39.762 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:39.762 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.762 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.763 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:39.763 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761987+0000", 2024-03-19T11:30:39.763 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4099999999999999e-07 2024-03-19T11:30:39.763 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.763 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.763 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:39.763 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.761990+0000", 2024-03-19T11:30:39.763 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3829999999999999e-06 2024-03-19T11:30:39.763 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.763 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.763 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:39.763 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762000+0000", 2024-03-19T11:30:39.763 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0229e-05 2024-03-19T11:30:39.763 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.763 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.764 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:39.764 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762006+0000", 2024-03-19T11:30:39.764 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7069999999999997e-06 2024-03-19T11:30:39.764 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.764 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.764 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:39.764 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762171+0000", 2024-03-19T11:30:39.764 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000165341 2024-03-19T11:30:39.764 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.764 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.764 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:39.764 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762176+0000", 2024-03-19T11:30:39.764 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6879999999999998e-06 2024-03-19T11:30:39.764 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.764 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.765 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:39.765 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.774081+0000", 2024-03-19T11:30:39.765 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011905094999999999 2024-03-19T11:30:39.765 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.765 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.765 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:39.765 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.774169+0000", 2024-03-19T11:30:39.765 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8089000000000003e-05 2024-03-19T11:30:39.765 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.765 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.765 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:39.765 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.774679+0000", 2024-03-19T11:30:39.765 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00051001900000000001 2024-03-19T11:30:39.765 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:39.766 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:39.766 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:39.766 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.766 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.766 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:58 3.2 3:5f1d333e:::bfcdf551f6f2543327e9c601a4377e9a5ef396f5b97ae8331892d1bcf6b55a9d:head [read 0~161292 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:39.766 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.762024+0000", 2024-03-19T11:30:39.766 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 27.000037836000001, 2024-03-19T11:30:39.766 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013495629, 2024-03-19T11:30:39.766 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:39.766 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:39.766 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:39.766 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.766 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:39.766 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762024+0000", 2024-03-19T11:30:39.767 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:39.767 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.767 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.767 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:39.767 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762024+0000", 2024-03-19T11:30:39.767 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:39.767 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.767 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.767 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:39.767 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762024+0000", 2024-03-19T11:30:39.767 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9999999999999997e-07 2024-03-19T11:30:39.767 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.768 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.768 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:39.768 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762027+0000", 2024-03-19T11:30:39.768 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0410000000000002e-06 2024-03-19T11:30:39.768 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.768 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.768 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:39.768 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762037+0000", 2024-03-19T11:30:39.768 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1020000000000003e-06 2024-03-19T11:30:39.768 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.768 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.768 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:39.768 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762043+0000", 2024-03-19T11:30:39.769 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4409999999999998e-06 2024-03-19T11:30:39.769 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.769 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.769 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:39.769 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762198+0000", 2024-03-19T11:30:39.769 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015540000000000001 2024-03-19T11:30:39.769 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.769 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.769 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:39.769 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762203+0000", 2024-03-19T11:30:39.769 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5109999999999997e-06 2024-03-19T11:30:39.769 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.770 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.770 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:39.770 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.774710+0000", 2024-03-19T11:30:39.770 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012507452 2024-03-19T11:30:39.770 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.770 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.770 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:39.770 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.774754+0000", 2024-03-19T11:30:39.770 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3869999999999998e-05 2024-03-19T11:30:39.770 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.770 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.770 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:39.770 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.775519+0000", 2024-03-19T11:30:39.770 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00076521200000000003 2024-03-19T11:30:39.771 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:39.771 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:39.771 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:39.771 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.771 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.771 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:59 3.2 3:78b4b14e:::e1ef0c0c594c36c534f3829406e07ffb60d57b70360f62977e0d545d8ca2bf18:head [read 0~149977 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:39.771 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.762054+0000", 2024-03-19T11:30:39.771 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 27.000007405000002, 2024-03-19T11:30:39.771 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015354483, 2024-03-19T11:30:39.771 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:39.771 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:39.771 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:39.771 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.771 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:39.772 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762054+0000", 2024-03-19T11:30:39.772 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:39.772 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.772 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.772 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:39.772 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762054+0000", 2024-03-19T11:30:39.772 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:39.772 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.772 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.772 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:39.772 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762055+0000", 2024-03-19T11:30:39.772 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.32e-07 2024-03-19T11:30:39.772 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.772 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.772 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:39.773 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762058+0000", 2024-03-19T11:30:39.773 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9189999999999999e-06 2024-03-19T11:30:39.773 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.773 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.773 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:39.773 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762065+0000", 2024-03-19T11:30:39.773 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2409999999999998e-06 2024-03-19T11:30:39.773 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.773 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.773 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:39.773 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762070+0000", 2024-03-19T11:30:39.773 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6910000000000002e-06 2024-03-19T11:30:39.773 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.773 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.773 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:39.774 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762227+0000", 2024-03-19T11:30:39.774 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015606999999999999 2024-03-19T11:30:39.774 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.774 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.774 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:39.774 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762232+0000", 2024-03-19T11:30:39.774 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3029999999999999e-06 2024-03-19T11:30:39.774 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.774 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.774 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:39.774 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.776984+0000", 2024-03-19T11:30:39.774 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014752487999999999 2024-03-19T11:30:39.774 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.774 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.774 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:39.775 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.777030+0000", 2024-03-19T11:30:39.775 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5359999999999999e-05 2024-03-19T11:30:39.775 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.775 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.775 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:39.775 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.777409+0000", 2024-03-19T11:30:39.775 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037897899999999999 2024-03-19T11:30:39.775 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:39.775 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:39.775 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:39.775 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.775 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.775 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1067 4.0 4:01018511:::smithi03829856-19:head [write 477332~766101] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:39.775 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.950506+0000", 2024-03-19T11:30:39.775 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 25.811555692999999, 2024-03-19T11:30:39.776 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016279426, 2024-03-19T11:30:39.776 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:39.776 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:39.776 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:39.776 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:39.776 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:39.776 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1067 2024-03-19T11:30:39.776 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.776 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:39.776 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.776 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:39.776 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950506+0000", 2024-03-19T11:30:39.776 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:39.776 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.776 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.777 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:39.777 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950506+0000", 2024-03-19T11:30:39.777 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:39.777 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.777 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.777 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:39.777 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950509+0000", 2024-03-19T11:30:39.777 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.021e-06 2024-03-19T11:30:39.777 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.777 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.777 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:39.777 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950892+0000", 2024-03-19T11:30:39.777 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00038263300000000001 2024-03-19T11:30:39.778 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.778 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.778 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:39.778 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950912+0000", 2024-03-19T11:30:39.778 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0539000000000001e-05 2024-03-19T11:30:39.778 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.778 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.778 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:39.778 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950922+0000", 2024-03-19T11:30:39.779 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.577e-06 2024-03-19T11:30:39.779 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.779 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.779 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:39.779 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950982+0000", 2024-03-19T11:30:39.779 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0760000000000001e-05 2024-03-19T11:30:39.779 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.779 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.779 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:39.779 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.951058+0000", 2024-03-19T11:30:39.779 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5078000000000002e-05 2024-03-19T11:30:39.779 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.779 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.779 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:39.779 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.951173+0000", 2024-03-19T11:30:39.780 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011516200000000001 2024-03-19T11:30:39.780 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.780 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.780 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:39.780 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.955064+0000", 2024-03-19T11:30:39.780 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003890896 2024-03-19T11:30:39.780 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.780 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.780 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:39.780 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966670+0000", 2024-03-19T11:30:39.780 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011606795 2024-03-19T11:30:39.780 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.780 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.780 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:39.780 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966731+0000", 2024-03-19T11:30:39.781 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0528000000000001e-05 2024-03-19T11:30:39.781 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.781 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.781 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:39.781 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966785+0000", 2024-03-19T11:30:39.781 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4437000000000001e-05 2024-03-19T11:30:39.781 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:39.782 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:39.782 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:39.782 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.782 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.782 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1068 4.0 4:01018511:::smithi03829856-19:head [write 1681102~474010] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:39.782 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.956456+0000", 2024-03-19T11:30:39.782 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 25.805605128, 2024-03-19T11:30:39.783 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010432765, 2024-03-19T11:30:39.786 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:39.786 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:39.786 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:39.786 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:39.786 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:39.786 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1068 2024-03-19T11:30:39.786 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.787 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:39.787 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.787 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:39.787 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956456+0000", 2024-03-19T11:30:39.787 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:39.787 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.787 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.787 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:39.787 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956456+0000", 2024-03-19T11:30:39.787 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:39.787 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.787 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.788 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:39.788 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956458+0000", 2024-03-19T11:30:39.788 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.742e-06 2024-03-19T11:30:39.788 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.788 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.788 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:39.788 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956655+0000", 2024-03-19T11:30:39.788 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019682200000000001 2024-03-19T11:30:39.788 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.789 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.789 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:39.789 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956669+0000", 2024-03-19T11:30:39.789 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3978000000000001e-05 2024-03-19T11:30:39.789 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.789 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.789 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:39.789 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956678+0000", 2024-03-19T11:30:39.789 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5029999999999997e-06 2024-03-19T11:30:39.789 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.789 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.789 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:39.790 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956729+0000", 2024-03-19T11:30:39.790 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1765999999999999e-05 2024-03-19T11:30:39.790 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.790 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.790 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:39.790 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956795+0000", 2024-03-19T11:30:39.790 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5701e-05 2024-03-19T11:30:39.790 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.790 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.790 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:39.790 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.956886+0000", 2024-03-19T11:30:39.790 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1226999999999999e-05 2024-03-19T11:30:39.790 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.791 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.791 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:39.791 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.958609+0000", 2024-03-19T11:30:39.791 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017227049999999999 2024-03-19T11:30:39.791 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.791 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.791 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:39.791 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966837+0000", 2024-03-19T11:30:39.792 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0082276439999999992 2024-03-19T11:30:39.792 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.792 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.792 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:39.792 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966866+0000", 2024-03-19T11:30:39.792 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9323999999999999e-05 2024-03-19T11:30:39.792 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.792 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.792 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:39.792 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966889+0000", 2024-03-19T11:30:39.792 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3353000000000002e-05 2024-03-19T11:30:39.792 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:39.792 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:39.792 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:39.792 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.793 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.793 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.0.13:50 3.2 3:5595a68d:::07854d2fef297a06ba81685e660c332de36d5d18d546927d30daad6d7fda1541:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:39.793 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.166341+0000", 2024-03-19T11:30:39.793 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 21.595720262, 2024-03-19T11:30:39.793 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010067153000000001, 2024-03-19T11:30:39.793 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:39.793 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:39.793 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:39.793 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.793 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:39.793 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166341+0000", 2024-03-19T11:30:39.793 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:39.793 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.793 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.794 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:39.794 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166341+0000", 2024-03-19T11:30:39.794 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:39.794 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.794 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.794 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:39.794 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166344+0000", 2024-03-19T11:30:39.794 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1370000000000002e-06 2024-03-19T11:30:39.794 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.794 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.794 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:39.794 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166828+0000", 2024-03-19T11:30:39.794 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00048343600000000001 2024-03-19T11:30:39.794 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.795 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.795 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:39.795 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166857+0000", 2024-03-19T11:30:39.795 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8880000000000001e-05 2024-03-19T11:30:39.795 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.795 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.795 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:39.795 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.166871+0000", 2024-03-19T11:30:39.795 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4384e-05 2024-03-19T11:30:39.795 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.795 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.795 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:39.795 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167222+0000", 2024-03-19T11:30:39.795 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00035059000000000002 2024-03-19T11:30:39.795 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.796 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.796 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:39.796 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167334+0000", 2024-03-19T11:30:39.796 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000112663 2024-03-19T11:30:39.796 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.796 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.796 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:39.796 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167533+0000", 2024-03-19T11:30:39.796 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019901399999999999 2024-03-19T11:30:39.796 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.796 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.796 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:39.796 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176237+0000", 2024-03-19T11:30:39.796 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0087039890000000005 2024-03-19T11:30:39.797 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.797 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.797 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:39.797 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176260+0000", 2024-03-19T11:30:39.797 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2889000000000001e-05 2024-03-19T11:30:39.797 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.797 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.797 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:39.797 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176343+0000", 2024-03-19T11:30:39.797 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2651000000000005e-05 2024-03-19T11:30:39.797 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.797 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.797 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:39.797 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176408+0000", 2024-03-19T11:30:39.798 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5519999999999996e-05 2024-03-19T11:30:39.798 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:39.798 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:39.798 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:39.798 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.798 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.798 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.0.13:52 3.2 3:46a26328:::b2b262672c1f15ae09b098a7a21c74662e4816397daed8ca08fb83d0cd3a0cac:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:39.798 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.167033+0000", 2024-03-19T11:30:39.798 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 21.59502865, 2024-03-19T11:30:39.798 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010113297, 2024-03-19T11:30:39.799 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:39.799 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:39.799 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:39.799 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.799 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:39.799 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167033+0000", 2024-03-19T11:30:39.799 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:39.799 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.799 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.800 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:39.800 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167033+0000", 2024-03-19T11:30:39.800 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:39.800 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.800 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.800 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:39.800 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167034+0000", 2024-03-19T11:30:39.800 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6599999999999996e-07 2024-03-19T11:30:39.800 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.800 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.800 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:39.800 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167092+0000", 2024-03-19T11:30:39.800 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7989999999999999e-05 2024-03-19T11:30:39.800 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.801 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.801 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:39.801 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167100+0000", 2024-03-19T11:30:39.801 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9670000000000001e-06 2024-03-19T11:30:39.801 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.801 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.801 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:39.801 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167108+0000", 2024-03-19T11:30:39.801 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3559999999999993e-06 2024-03-19T11:30:39.801 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.801 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.801 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:39.801 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.170462+0000", 2024-03-19T11:30:39.801 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033540229999999998 2024-03-19T11:30:39.802 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.802 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.802 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:39.802 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.171193+0000", 2024-03-19T11:30:39.802 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00073086899999999998 2024-03-19T11:30:39.802 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.802 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.803 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:39.803 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.171558+0000", 2024-03-19T11:30:39.803 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00036518899999999998 2024-03-19T11:30:39.803 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.803 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.803 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:39.803 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176440+0000", 2024-03-19T11:30:39.803 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0048817260000000003 2024-03-19T11:30:39.803 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.803 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.803 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:39.803 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177053+0000", 2024-03-19T11:30:39.804 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00061337100000000003 2024-03-19T11:30:39.804 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.804 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.804 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:39.804 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177096+0000", 2024-03-19T11:30:39.804 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2846999999999998e-05 2024-03-19T11:30:39.804 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.804 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.804 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:39.804 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177146+0000", 2024-03-19T11:30:39.804 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0293000000000001e-05 2024-03-19T11:30:39.804 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:39.804 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:39.804 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:39.804 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.805 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.805 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.0.13:54 3.2 3:7b614c59:::a7a06fca755baa7858207cb4b067fffe8c25e8183433b950d6879f6969d41b0b:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:39.805 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.167117+0000", 2024-03-19T11:30:39.805 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 21.594944141999999, 2024-03-19T11:30:39.805 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010796765999999999, 2024-03-19T11:30:39.805 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:39.805 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:39.805 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:39.805 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.805 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:39.805 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167117+0000", 2024-03-19T11:30:39.805 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:39.805 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.806 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.806 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:39.806 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167117+0000", 2024-03-19T11:30:39.806 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:39.806 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.806 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.806 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:39.806 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167118+0000", 2024-03-19T11:30:39.806 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2600000000000002e-07 2024-03-19T11:30:39.806 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.806 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.806 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:39.806 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167150+0000", 2024-03-19T11:30:39.807 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1576999999999999e-05 2024-03-19T11:30:39.807 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.807 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.807 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:39.807 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167157+0000", 2024-03-19T11:30:39.807 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2779999999999996e-06 2024-03-19T11:30:39.807 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.807 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.807 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:39.807 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.167163+0000", 2024-03-19T11:30:39.807 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0120000000000002e-06 2024-03-19T11:30:39.808 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.808 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.808 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:39.808 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173324+0000", 2024-03-19T11:30:39.808 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0061614149999999999 2024-03-19T11:30:39.808 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.808 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.808 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:39.808 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173489+0000", 2024-03-19T11:30:39.808 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016438999999999999 2024-03-19T11:30:39.808 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.808 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.808 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:39.808 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173774+0000", 2024-03-19T11:30:39.808 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00028548000000000001 2024-03-19T11:30:39.809 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.809 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.809 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:39.809 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177668+0000", 2024-03-19T11:30:39.809 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038942450000000002 2024-03-19T11:30:39.809 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.809 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.809 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:39.809 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177792+0000", 2024-03-19T11:30:39.809 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012398199999999999 2024-03-19T11:30:39.809 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.809 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.810 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:39.810 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177861+0000", 2024-03-19T11:30:39.810 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8633999999999994e-05 2024-03-19T11:30:39.810 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.810 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.810 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:39.810 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177914+0000", 2024-03-19T11:30:39.810 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3226999999999997e-05 2024-03-19T11:30:39.810 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:39.810 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:39.810 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:39.811 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.811 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.811 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1230 4.0 4:0dc3b91e:::smithi03829856-28:head [write 662209~626546] snapc 9=[9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:39.811 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.170758+0000", 2024-03-19T11:30:39.811 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 21.591303428, 2024-03-19T11:30:39.811 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011180066000000001, 2024-03-19T11:30:39.811 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:39.811 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:39.811 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:39.811 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:39.811 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:39.811 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1230 2024-03-19T11:30:39.811 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.811 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:39.812 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.812 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:39.812 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.170758+0000", 2024-03-19T11:30:39.812 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:39.812 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.812 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.812 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:39.812 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.170758+0000", 2024-03-19T11:30:39.813 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:39.813 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.813 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.813 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:39.813 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.170762+0000", 2024-03-19T11:30:39.813 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4960000000000001e-06 2024-03-19T11:30:39.813 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.813 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.813 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:39.814 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.171449+0000", 2024-03-19T11:30:39.814 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00068700499999999997 2024-03-19T11:30:39.814 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.814 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.814 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:39.814 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.171467+0000", 2024-03-19T11:30:39.814 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8389e-05 2024-03-19T11:30:39.814 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.814 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.814 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:39.814 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.171482+0000", 2024-03-19T11:30:39.814 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4619e-05 2024-03-19T11:30:39.814 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.815 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.815 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:39.815 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.171566+0000", 2024-03-19T11:30:39.815 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4788000000000002e-05 2024-03-19T11:30:39.815 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.815 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.815 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:39.815 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.171685+0000", 2024-03-19T11:30:39.815 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011861700000000001 2024-03-19T11:30:39.815 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.815 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.815 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:39.815 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.171835+0000", 2024-03-19T11:30:39.816 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014956299999999999 2024-03-19T11:30:39.816 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.816 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.816 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:39.816 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.181520+0000", 2024-03-19T11:30:39.816 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0096855750000000001 2024-03-19T11:30:39.816 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.816 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.816 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:39.816 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.181640+0000", 2024-03-19T11:30:39.816 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011966799999999999 2024-03-19T11:30:39.817 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.817 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.817 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:39.817 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.181811+0000", 2024-03-19T11:30:39.817 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017062699999999999 2024-03-19T11:30:39.817 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.817 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.817 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:39.817 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.181938+0000", 2024-03-19T11:30:39.817 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012771899999999999 2024-03-19T11:30:39.817 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:39.817 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:39.818 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:39.818 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.818 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.818 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1231 4.0 4:0dc3b91e:::smithi03829856-28:head [write 1808051~636013] snapc 9=[9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:39.818 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.173016+0000", 2024-03-19T11:30:39.818 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 21.589045580000001, 2024-03-19T11:30:39.818 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013973719000000001, 2024-03-19T11:30:39.818 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:39.818 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:39.818 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:39.818 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:39.819 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:39.819 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1231 2024-03-19T11:30:39.819 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.819 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:39.819 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.819 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:39.819 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173016+0000", 2024-03-19T11:30:39.819 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:39.819 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.819 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.819 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:39.819 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173016+0000", 2024-03-19T11:30:39.820 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:39.820 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.820 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.820 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:39.820 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173019+0000", 2024-03-19T11:30:39.820 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.253e-06 2024-03-19T11:30:39.820 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.820 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.821 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:39.821 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176143+0000", 2024-03-19T11:30:39.821 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031240930000000001 2024-03-19T11:30:39.821 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.821 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.821 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:39.821 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176177+0000", 2024-03-19T11:30:39.821 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3237e-05 2024-03-19T11:30:39.821 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.821 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.821 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:39.821 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176196+0000", 2024-03-19T11:30:39.821 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9358e-05 2024-03-19T11:30:39.821 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.822 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.822 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:39.822 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176493+0000", 2024-03-19T11:30:39.822 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00029743899999999998 2024-03-19T11:30:39.822 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.822 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.822 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:39.822 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176786+0000", 2024-03-19T11:30:39.822 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00029259000000000002 2024-03-19T11:30:39.822 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.822 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.822 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:39.823 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177250+0000", 2024-03-19T11:30:39.823 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00046376000000000002 2024-03-19T11:30:39.823 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.823 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.823 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:39.823 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.182511+0000", 2024-03-19T11:30:39.823 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0052609019999999996 2024-03-19T11:30:39.823 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.823 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.823 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:39.823 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.186896+0000", 2024-03-19T11:30:39.824 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0043857009999999997 2024-03-19T11:30:39.824 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.824 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.824 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:39.824 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.186948+0000", 2024-03-19T11:30:39.824 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1653000000000002e-05 2024-03-19T11:30:39.824 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.824 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.824 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:39.824 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.186990+0000", 2024-03-19T11:30:39.824 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1733000000000002e-05 2024-03-19T11:30:39.824 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:39.825 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:39.825 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:39.825 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.825 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.825 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1277 4.0 4:01018511:::smithi03829856-19:head [write 763051~794393,stat] snapc b=[b,a,9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-03-19T11:30:39.825 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:21.953732+0000", 2024-03-19T11:30:39.825 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 17.808329343, 2024-03-19T11:30:39.825 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011784380000000001, 2024-03-19T11:30:39.825 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:39.825 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:39.825 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:39.825 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:39.825 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:39.825 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1277 2024-03-19T11:30:39.826 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.826 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:39.826 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.826 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:39.826 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.953732+0000", 2024-03-19T11:30:39.826 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:39.826 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.826 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.826 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:39.826 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.953732+0000", 2024-03-19T11:30:39.826 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:39.826 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.826 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.826 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:39.827 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.953735+0000", 2024-03-19T11:30:39.827 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1999999999999999e-06 2024-03-19T11:30:39.827 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.827 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.827 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:39.827 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.954556+0000", 2024-03-19T11:30:39.827 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00082013599999999996 2024-03-19T11:30:39.827 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.827 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.827 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:39.828 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.954596+0000", 2024-03-19T11:30:39.828 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0741000000000002e-05 2024-03-19T11:30:39.828 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.828 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.828 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:39.828 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.954617+0000", 2024-03-19T11:30:39.828 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0375999999999999e-05 2024-03-19T11:30:39.828 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.828 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.828 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:39.828 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.954747+0000", 2024-03-19T11:30:39.828 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000130648 2024-03-19T11:30:39.828 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.828 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.828 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:39.829 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.954954+0000", 2024-03-19T11:30:39.829 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020694299999999999 2024-03-19T11:30:39.829 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.829 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.829 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:39.829 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.955252+0000", 2024-03-19T11:30:39.829 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00029740700000000002 2024-03-19T11:30:39.829 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.829 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.829 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:39.829 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.961677+0000", 2024-03-19T11:30:39.829 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0064253469999999997 2024-03-19T11:30:39.829 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.830 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.830 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:39.830 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.965373+0000", 2024-03-19T11:30:39.830 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0036962649999999998 2024-03-19T11:30:39.830 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.830 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.830 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:39.830 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.965451+0000", 2024-03-19T11:30:39.830 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7204000000000001e-05 2024-03-19T11:30:39.831 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.831 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.831 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:39.831 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:21.965517+0000", 2024-03-19T11:30:39.831 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6112999999999995e-05 2024-03-19T11:30:39.831 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:39.831 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:39.831 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:39.832 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.832 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.832 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1408 4.10 4:0dc3b91e:::smithi03829856-28:head [read 0~3462209,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:39.832 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.918220+0000", 2024-03-19T11:30:39.832 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 13.843841125000001, 2024-03-19T11:30:39.832 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013199458000000001, 2024-03-19T11:30:39.832 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:39.832 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:39.832 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:39.832 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:39.833 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:39.833 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1408 2024-03-19T11:30:39.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.833 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:39.833 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.833 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:39.833 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918220+0000", 2024-03-19T11:30:39.833 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:39.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.833 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.833 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:39.833 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918220+0000", 2024-03-19T11:30:39.833 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:39.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.833 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.834 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:39.834 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918221+0000", 2024-03-19T11:30:39.834 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0999999999999999e-07 2024-03-19T11:30:39.834 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.834 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.834 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:39.834 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918225+0000", 2024-03-19T11:30:39.834 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2710000000000003e-06 2024-03-19T11:30:39.835 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.835 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.835 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:39.835 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918233+0000", 2024-03-19T11:30:39.835 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9489999999999996e-06 2024-03-19T11:30:39.835 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.835 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.835 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:39.835 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918239+0000", 2024-03-19T11:30:39.835 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5820000000000001e-06 2024-03-19T11:30:39.835 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.835 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.835 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:39.835 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.923634+0000", 2024-03-19T11:30:39.836 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005395703 2024-03-19T11:30:39.836 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.836 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.836 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:39.836 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.923786+0000", 2024-03-19T11:30:39.836 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015117900000000001 2024-03-19T11:30:39.836 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.836 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.836 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:39.836 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.931420+0000", 2024-03-19T11:30:39.836 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.007634264 2024-03-19T11:30:39.836 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:39.836 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:39.836 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:39.837 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.837 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.837 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1409 4.10 4:0dc3b91e:::smithi03829856-28:head [sparse-read 0~1851556] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:39.837 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.918248+0000", 2024-03-19T11:30:39.837 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 13.843813291, 2024-03-19T11:30:39.837 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013726986, 2024-03-19T11:30:39.837 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:39.838 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:39.838 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:39.838 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:39.838 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:39.838 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1409 2024-03-19T11:30:39.838 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.838 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:39.838 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.838 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:39.838 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918248+0000", 2024-03-19T11:30:39.838 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:39.839 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.839 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.839 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:39.839 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918248+0000", 2024-03-19T11:30:39.839 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:39.839 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.839 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.839 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:39.839 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918249+0000", 2024-03-19T11:30:39.839 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4700000000000002e-07 2024-03-19T11:30:39.839 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.839 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.839 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:39.839 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918251+0000", 2024-03-19T11:30:39.839 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.542e-06 2024-03-19T11:30:39.840 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.840 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.840 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:39.840 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918258+0000", 2024-03-19T11:30:39.840 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4629999999999998e-06 2024-03-19T11:30:39.840 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.840 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.840 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:39.840 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918263+0000", 2024-03-19T11:30:39.840 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8049999999999997e-06 2024-03-19T11:30:39.840 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.841 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.841 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:39.841 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.931469+0000", 2024-03-19T11:30:39.841 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013206458000000001 2024-03-19T11:30:39.841 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.841 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.841 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:39.841 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.931586+0000", 2024-03-19T11:30:39.841 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000116655 2024-03-19T11:30:39.841 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.841 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.841 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:39.842 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.931975+0000", 2024-03-19T11:30:39.842 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00038961600000000002 2024-03-19T11:30:39.842 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:39.842 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:39.842 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:39.842 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.842 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.842 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(3.1 49/48 [PushOp(3:9205291d:::smithi03829856-50 low_tier:head, version: 28'36, data_included: [393216~851968,1703936~196608], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:9205291d:::smithi03829856-50 low_tier:head@28'36, size: 2823543, copy_subset: [(0, 2823543)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1900544, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:baab691a:::smithi03829856-2 low_tier:head, version: 28'37, data_included: [393216~524288,1507328~509479], data_size: 1033767, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:baab691a:::smithi03829856-2 low_tier:head@28'37, size: 2016807, copy_subset: [(0, 2016807)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2016807, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:98ce8426:::smithi03829856-4 low_tier:head, version: 28'38, data_included: [393216~720896,1638400~327680], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:98ce8426:::smithi03829856-4 low_tier:head@28'38, size: 2050421, copy_subset: [(0, 2050421)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1966080, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:a86be07b:::smithi03829856-44 low_tier:head, version: 28'41, data_included: [720896~655360,1769472~393216], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:a86be07b:::smithi03829856-44 low_tier:head@28'41, size: 2722701, copy_subset: [(0, 2722701)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2162688, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:bb56919e:::smithi03829856-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'44, data_included: [393216~786432,1703936~262144], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:bb56919e:::smithi03829856-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'44, size: 2806736, copy_subset: [(0, 2806736)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1966080, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:88687f71:::smithi03829856-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'45, data_included: [589824~655360,1638400~393216], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:88687f71:::smithi03829856-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'45, size: 2201684, copy_subset: [(0, 2201684)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2031616, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:b1f822cf:::smithi03829856-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'46, data_included: [720896~786432,1966080~262144], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:b1f822cf:::smithi03829856-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'46, size: 2369754, copy_subset: [(0, 2369754)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2228224, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:a5e8313a:::smithi03829856-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'47, data_included: [524288~720896,1769472~327680], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:a5e8313a:::smithi03829856-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'47, size: 2571438, copy_subset: [(0, 2571438)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:bca5ec64:::e3a26bf9230b1b95b8cf1a4cb01909236bf2bf3fd59b083b2291d278ab98264d:head, version: 28'48, data_included: [0~393216], data_size: 393216, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bca5ec64:::e3a26bf9230b1b95b8cf1a4cb01909236bf2bf3fd59b083b2291d278ab98264d:head@28'48, size: 533504, copy_subset: [(0, 533504)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 533504, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:30:39.842 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:26.948137+0000", 2024-03-19T11:30:39.842 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 12.813924478000001, 2024-03-19T11:30:39.842 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.036085578, 2024-03-19T11:30:39.842 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:39.843 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:39.843 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:39.843 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.843 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:39.843 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.948137+0000", 2024-03-19T11:30:39.843 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:39.843 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.843 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.843 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:39.843 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.948137+0000", 2024-03-19T11:30:39.843 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:39.843 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.843 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.843 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:39.843 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.948139+0000", 2024-03-19T11:30:39.844 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1399999999999998e-06 2024-03-19T11:30:39.844 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.844 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.844 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:39.844 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954288+0000", 2024-03-19T11:30:39.844 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0061487429999999999 2024-03-19T11:30:39.844 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.844 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.844 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:39.844 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954388+0000", 2024-03-19T11:30:39.844 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010022 2024-03-19T11:30:39.844 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.844 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.844 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:39.845 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954523+0000", 2024-03-19T11:30:39.845 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000134332 2024-03-19T11:30:39.845 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.845 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.845 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:39.845 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.955120+0000", 2024-03-19T11:30:39.846 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00059695799999999997 2024-03-19T11:30:39.846 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.846 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.846 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:39.846 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.955194+0000", 2024-03-19T11:30:39.846 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4077000000000002e-05 2024-03-19T11:30:39.846 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.846 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.846 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:39.846 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.984223+0000", 2024-03-19T11:30:39.846 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029029108000000001 2024-03-19T11:30:39.847 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:39.847 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:39.847 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:39.847 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.847 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.847 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(3.1 49/48 [PushOp(3:99a49d1b:::895a63407b6109b3cca9a6f5ef827134d56679bef42daadc5892269682c428ed:head, version: 32'49, data_included: [393216~388096], data_size: 388096, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:99a49d1b:::895a63407b6109b3cca9a6f5ef827134d56679bef42daadc5892269682c428ed:head@32'49, size: 781312, copy_subset: [(0, 781312)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 781312, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:30:39.847 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:26.954637+0000", 2024-03-19T11:30:39.847 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 12.807424860999999, 2024-03-19T11:30:39.847 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.030729057000000001, 2024-03-19T11:30:39.847 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:39.847 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:39.847 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:39.847 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.847 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:39.847 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954637+0000", 2024-03-19T11:30:39.848 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:39.848 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.848 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.848 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:39.848 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954637+0000", 2024-03-19T11:30:39.848 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:39.848 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.848 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.848 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:39.849 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954637+0000", 2024-03-19T11:30:39.849 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7299999999999995e-07 2024-03-19T11:30:39.849 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.849 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.849 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:39.849 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954873+0000", 2024-03-19T11:30:39.849 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023520299999999999 2024-03-19T11:30:39.849 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.849 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.849 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:39.849 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954897+0000", 2024-03-19T11:30:39.849 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4110000000000001e-05 2024-03-19T11:30:39.850 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.850 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.850 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:39.850 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954917+0000", 2024-03-19T11:30:39.850 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.067e-05 2024-03-19T11:30:39.850 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.850 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.850 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:39.850 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.984289+0000", 2024-03-19T11:30:39.850 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029371191000000001 2024-03-19T11:30:39.850 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.850 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.850 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:39.851 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.984303+0000", 2024-03-19T11:30:39.851 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4888999999999999e-05 2024-03-19T11:30:39.851 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.851 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.851 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:39.851 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.985366+0000", 2024-03-19T11:30:39.851 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001062321 2024-03-19T11:30:39.851 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:39.851 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:39.851 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:39.852 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.852 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.852 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1472 4.10 4:0dc3b91e:::smithi03829856-28:head [read 534528~391680] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-03-19T11:30:39.852 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:27.091748+0000", 2024-03-19T11:30:39.852 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 12.670313973000001, 2024-03-19T11:30:39.852 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.83574118500000005, 2024-03-19T11:30:39.852 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:39.852 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:39.852 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:39.852 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:39.852 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:39.852 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1472 2024-03-19T11:30:39.852 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.852 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:39.852 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.853 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:39.853 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091748+0000", 2024-03-19T11:30:39.853 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:39.853 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.853 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.853 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:39.853 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091748+0000", 2024-03-19T11:30:39.853 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:39.853 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.853 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.853 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:39.853 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091750+0000", 2024-03-19T11:30:39.853 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3130000000000001e-06 2024-03-19T11:30:39.854 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.854 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.854 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:39.854 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091760+0000", 2024-03-19T11:30:39.854 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0560999999999999e-05 2024-03-19T11:30:39.854 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.854 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.854 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:39.854 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091784+0000", 2024-03-19T11:30:39.854 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3915000000000001e-05 2024-03-19T11:30:39.855 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.855 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.855 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:39.855 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091797+0000", 2024-03-19T11:30:39.855 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2660000000000001e-05 2024-03-19T11:30:39.855 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.856 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.856 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:39.856 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091874+0000", 2024-03-19T11:30:39.856 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6757999999999999e-05 2024-03-19T11:30:39.856 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.856 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.856 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:30:39.856 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091882+0000", 2024-03-19T11:30:39.857 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5520000000000005e-06 2024-03-19T11:30:39.857 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.857 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.857 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:39.857 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.927033+0000", 2024-03-19T11:30:39.857 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.83515094099999998 2024-03-19T11:30:39.857 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.857 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.858 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:39.858 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.927250+0000", 2024-03-19T11:30:39.858 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00021675199999999999 2024-03-19T11:30:39.858 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.858 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.858 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:39.858 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.927489+0000", 2024-03-19T11:30:39.859 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023873300000000001 2024-03-19T11:30:39.859 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:39.859 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:39.859 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:39.859 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.859 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.859 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1474 4.10 4:0dc3b91e:::smithi03829856-28:head [assert-version v27,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-03-19T11:30:39.859 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:27.934139+0000", 2024-03-19T11:30:39.860 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 11.827922187, 2024-03-19T11:30:39.860 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024764884000000001, 2024-03-19T11:30:39.860 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:39.860 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:39.860 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:39.860 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:39.860 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:39.860 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1474 2024-03-19T11:30:39.861 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.861 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:39.861 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.861 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:39.861 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934139+0000", 2024-03-19T11:30:39.861 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:39.861 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.861 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.862 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:39.862 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934139+0000", 2024-03-19T11:30:39.862 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:39.862 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.862 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.862 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:39.862 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934142+0000", 2024-03-19T11:30:39.863 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5459999999999998e-06 2024-03-19T11:30:39.863 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.863 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.863 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:39.863 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934152+0000", 2024-03-19T11:30:39.863 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9399999999999997e-06 2024-03-19T11:30:39.863 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.863 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.864 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:39.864 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934173+0000", 2024-03-19T11:30:39.864 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1522000000000002e-05 2024-03-19T11:30:39.864 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.864 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.864 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:39.864 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934185+0000", 2024-03-19T11:30:39.864 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1946999999999999e-05 2024-03-19T11:30:39.865 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.865 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.865 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:39.865 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934282+0000", 2024-03-19T11:30:39.865 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6272000000000003e-05 2024-03-19T11:30:39.865 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.865 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.866 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:30:39.866 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934379+0000", 2024-03-19T11:30:39.866 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6966999999999995e-05 2024-03-19T11:30:39.866 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.866 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.866 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:39.866 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.955023+0000", 2024-03-19T11:30:39.866 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020644302 2024-03-19T11:30:39.867 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.867 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.867 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:39.867 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.955167+0000", 2024-03-19T11:30:39.867 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014404500000000001 2024-03-19T11:30:39.867 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.867 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.867 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:39.868 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.957913+0000", 2024-03-19T11:30:39.868 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0027463779999999998 2024-03-19T11:30:39.868 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.868 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.868 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:39.868 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.958548+0000", 2024-03-19T11:30:39.868 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00063495700000000001 2024-03-19T11:30:39.868 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.869 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.869 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:39.869 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.958785+0000", 2024-03-19T11:30:39.869 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000236872 2024-03-19T11:30:39.869 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.869 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.869 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:39.869 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.958854+0000", 2024-03-19T11:30:39.869 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9099999999999999e-05 2024-03-19T11:30:39.869 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.869 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.869 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:39.869 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.958904+0000", 2024-03-19T11:30:39.870 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0036e-05 2024-03-19T11:30:39.870 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:39.870 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:39.870 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:39.870 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.870 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.870 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1622 3.2 3:43782632:::f1224694c14fcb009af239eade27a8235cc6a98f1542b7554b67795eb195d6e5:head [write 0~1650176] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-03-19T11:30:39.870 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:35.170034+0000", 2024-03-19T11:30:39.870 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.5920278230000005, 2024-03-19T11:30:39.870 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0099681470000000001, 2024-03-19T11:30:39.871 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:39.871 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:39.871 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:39.871 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:39.871 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:39.871 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1622 2024-03-19T11:30:39.871 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.871 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:39.871 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.871 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:39.871 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.170034+0000", 2024-03-19T11:30:39.871 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:39.871 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.871 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.871 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:39.872 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.170034+0000", 2024-03-19T11:30:39.872 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:39.872 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.872 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.872 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:39.872 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.170036+0000", 2024-03-19T11:30:39.872 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3180000000000001e-06 2024-03-19T11:30:39.872 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.872 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.872 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:39.872 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.170483+0000", 2024-03-19T11:30:39.872 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044714100000000002 2024-03-19T11:30:39.872 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.872 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.872 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:39.873 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.170506+0000", 2024-03-19T11:30:39.873 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2645000000000001e-05 2024-03-19T11:30:39.873 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.873 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.873 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:39.873 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.170516+0000", 2024-03-19T11:30:39.873 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6479999999999998e-06 2024-03-19T11:30:39.873 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.873 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.873 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:39.873 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.170586+0000", 2024-03-19T11:30:39.873 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.0906000000000002e-05 2024-03-19T11:30:39.873 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.873 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.873 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:39.874 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.170689+0000", 2024-03-19T11:30:39.874 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000102734 2024-03-19T11:30:39.874 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.874 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.874 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:39.874 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.170787+0000", 2024-03-19T11:30:39.874 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7961000000000006e-05 2024-03-19T11:30:39.874 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.874 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.874 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:39.874 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.173638+0000", 2024-03-19T11:30:39.874 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0028512640000000001 2024-03-19T11:30:39.874 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.874 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.875 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:39.875 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.179936+0000", 2024-03-19T11:30:39.875 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00629723 2024-03-19T11:30:39.875 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.875 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.875 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:39.875 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.179970+0000", 2024-03-19T11:30:39.875 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4678999999999998e-05 2024-03-19T11:30:39.875 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:39.875 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:39.875 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:39.875 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.180002+0000", 2024-03-19T11:30:39.875 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1621000000000001e-05 2024-03-19T11:30:39.875 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:39.875 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:39.876 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:39.876 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:39.876 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:39.876 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:39.876 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:39.758+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:39.877 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-19T11:30:39.877 INFO:tasks.rados.rados.0.smithi038.stdout:954: expect (ObjNum 96 snap 10 seq_num 96) 2024-03-19T11:30:39.919 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:39.930+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:39.980 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:39.988+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:39.997 INFO:tasks.rados.rados.0.smithi038.stdout:961: finishing write tid 1 to smithi03829856-9 2024-03-19T11:30:39.997 INFO:tasks.rados.rados.0.smithi038.stdout:961: finishing write tid 2 to smithi03829856-9 2024-03-19T11:30:39.997 INFO:tasks.rados.rados.0.smithi038.stdout:961: finishing write tid 3 to smithi03829856-9 2024-03-19T11:30:39.997 INFO:tasks.rados.rados.0.smithi038.stdout:961: finishing write tid 4 to smithi03829856-9 2024-03-19T11:30:39.998 INFO:tasks.rados.rados.0.smithi038.stdout:961: oid 9 updating version 0 to 61 2024-03-19T11:30:39.998 INFO:tasks.rados.rados.0.smithi038.stdout:961: oid 9 version 61 is already newer than 60 2024-03-19T11:30:39.998 INFO:tasks.rados.rados.0.smithi038.stdout:961: oid 9 version 61 is already newer than 59 2024-03-19T11:30:39.998 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 9 v 61 (ObjNum 135 snap 15 seq_num 135) dirty exists 2024-03-19T11:30:39.998 INFO:tasks.rados.rados.0.smithi038.stdout:961: left oid 9 (ObjNum 135 snap 15 seq_num 135) 2024-03-19T11:30:39.998 INFO:tasks.rados.rados.0.smithi038.stdout:955: expect (ObjNum 60 snap 3 seq_num 60) 2024-03-19T11:30:40.008 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:40.008 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:40.008 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:30:40.008 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:40.022 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-19T11:30:40.080 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:40.090+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:40.130 INFO:tasks.rados.rados.0.smithi038.stdout:945: got (0) Success 2024-03-19T11:30:40.130 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 34 v 25 (ObjNum 72 snap 5 seq_num 72) dirty exists 2024-03-19T11:30:40.130 INFO:tasks.rados.rados.0.smithi038.stdout:945: done (14 left) 2024-03-19T11:30:40.130 INFO:tasks.rados.rados.0.smithi038.stdout:947: done (13 left) 2024-03-19T11:30:40.131 INFO:tasks.rados.rados.0.smithi038.stdout:949: done (12 left) 2024-03-19T11:30:40.131 INFO:tasks.rados.rados.0.smithi038.stdout:950: done (11 left) 2024-03-19T11:30:40.131 INFO:tasks.rados.rados.0.smithi038.stdout:951: done (10 left) 2024-03-19T11:30:40.131 INFO:tasks.rados.rados.0.smithi038.stdout:952: done (9 left) 2024-03-19T11:30:40.131 INFO:tasks.rados.rados.0.smithi038.stdout:953: done (8 left) 2024-03-19T11:30:40.131 INFO:tasks.rados.rados.0.smithi038.stdout:954: done (7 left) 2024-03-19T11:30:40.131 INFO:tasks.rados.rados.0.smithi038.stdout:955: done (6 left) 2024-03-19T11:30:40.131 INFO:tasks.rados.rados.0.smithi038.stdout:956: done (5 left) 2024-03-19T11:30:40.131 INFO:tasks.rados.rados.0.smithi038.stdout:957: done (4 left) 2024-03-19T11:30:40.131 INFO:tasks.rados.rados.0.smithi038.stdout:958: done (3 left) 2024-03-19T11:30:40.131 INFO:tasks.rados.rados.0.smithi038.stdout:959: done (2 left) 2024-03-19T11:30:40.131 INFO:tasks.rados.rados.0.smithi038.stdout:960: done (1 left) 2024-03-19T11:30:40.131 INFO:tasks.rados.rados.0.smithi038.stdout:961: done (0 left) 2024-03-19T11:30:40.132 INFO:tasks.rados.rados.0.smithi038.stdout:962: read oid 6 snap -1 2024-03-19T11:30:40.132 INFO:tasks.rados.rados.0.smithi038.stdout:962: expect (ObjNum 53 snap 0 seq_num 53) 2024-03-19T11:30:40.132 INFO:tasks.rados.rados.0.smithi038.stdout:963: write (excl) oid 49 current snap is 15 2024-03-19T11:30:40.132 INFO:tasks.rados.rados.0.smithi038.stdout:963: seq_num 136 ranges {685752=611865,1975551=310201} 2024-03-19T11:30:40.151 INFO:tasks.rados.rados.0.smithi038.stdout:963: writing smithi03829856-49 from 685752 to 1297617 tid 1 2024-03-19T11:30:40.168 INFO:tasks.rados.rados.0.smithi038.stdout:963: writing smithi03829856-49 from 1975551 to 2285752 tid 2 2024-03-19T11:30:40.168 INFO:tasks.rados.rados.0.smithi038.stdout:964: write (excl) oid 5 current snap is 15 2024-03-19T11:30:40.168 INFO:tasks.rados.rados.0.smithi038.stdout:964: seq_num 137 ranges {702559=403467,1750386=495032} 2024-03-19T11:30:40.182 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:40.190+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:40.185 INFO:tasks.rados.rados.0.smithi038.stdout:964: writing smithi03829856-5 from 702559 to 1106026 tid 1 2024-03-19T11:30:40.195 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:40.195 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:40.196 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:30:40.196 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:30:40.196 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:40.198 INFO:tasks.rados.rados.0.smithi038.stdout:964: writing smithi03829856-5 from 1750386 to 2245418 tid 2 2024-03-19T11:30:40.198 INFO:tasks.rados.rados.0.smithi038.stdout:965: write (excl) oid 17 current snap is 15 2024-03-19T11:30:40.199 INFO:tasks.rados.rados.0.smithi038.stdout:965: seq_num 138 ranges {719366=478716,1808868=510498} 2024-03-19T11:30:40.210 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-19T11:30:40.211 INFO:tasks.rados.rados.0.smithi038.stdout:965: writing smithi03829856-17 from 719366 to 1198082 tid 1 2024-03-19T11:30:40.220 INFO:tasks.rados.rados.0.smithi038.stdout:965: writing smithi03829856-17 from 1808868 to 2319366 tid 2 2024-03-19T11:30:40.220 INFO:tasks.rados.rados.0.smithi038.stdout:966: read oid 13 snap -1 2024-03-19T11:30:40.220 INFO:tasks.rados.rados.0.smithi038.stdout:966: expect (ObjNum 12 snap 0 seq_num 12) 2024-03-19T11:30:40.221 INFO:tasks.rados.rados.0.smithi038.stdout:967: tier_evict oid 15 2024-03-19T11:30:40.221 INFO:tasks.rados.rados.0.smithi038.stdout:967: tier_evict oid 15 snap -1 2024-03-19T11:30:40.221 INFO:tasks.rados.rados.0.smithi038.stdout:968: read oid 24 snap 14 2024-03-19T11:30:40.221 INFO:tasks.rados.rados.0.smithi038.stdout:968: expect (ObjNum 23 snap 0 seq_num 23) 2024-03-19T11:30:40.221 INFO:tasks.rados.rados.0.smithi038.stdout:969: write oid 23 current snap is 15 2024-03-19T11:30:40.221 INFO:tasks.rados.rados.0.smithi038.stdout:969: seq_num 139 ranges {736173=553965,1867350=468823} 2024-03-19T11:30:40.228 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-19T11:30:40.228 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-19T11:30:40.228 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-03-19T11:30:40.229 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T11:30:40.232 INFO:tasks.rados.rados.0.smithi038.stdout:969: writing smithi03829856-23 from 736173 to 1290138 tid 1 2024-03-19T11:30:40.238 INFO:tasks.rados.rados.0.smithi038.stdout:969: writing smithi03829856-23 from 1867350 to 2336173 tid 2 2024-03-19T11:30:40.239 INFO:tasks.rados.rados.0.smithi038.stdout:970: delete oid 31 current snap is 15 2024-03-19T11:30:40.239 INFO:tasks.rados.rados.0.smithi038.stdout:962: expect (ObjNum 53 snap 0 seq_num 53) 2024-03-19T11:30:40.282 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:40.290+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:40.339 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:40.339 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:30:40.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:30:40.340 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:30:40.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.340 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:9 4.d 4:b4ef97e0:::smithi03829856-3:head [write 433614~550498,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-03-19T11:30:40.340 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:04.706000+0000", 2024-03-19T11:30:40.340 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 35.642377603, 2024-03-19T11:30:40.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013227001, 2024-03-19T11:30:40.340 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:40.340 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:40.340 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:40.340 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:40.341 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:40.341 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 9 2024-03-19T11:30:40.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.341 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:40.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.341 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:40.341 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.706000+0000", 2024-03-19T11:30:40.341 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.341 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:40.341 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.706000+0000", 2024-03-19T11:30:40.341 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.342 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:40.342 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.706002+0000", 2024-03-19T11:30:40.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5569999999999998e-06 2024-03-19T11:30:40.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.342 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:40.342 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.707261+0000", 2024-03-19T11:30:40.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001258817 2024-03-19T11:30:40.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.342 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:40.342 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.707288+0000", 2024-03-19T11:30:40.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7093e-05 2024-03-19T11:30:40.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:40.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.707305+0000", 2024-03-19T11:30:40.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6330000000000001e-05 2024-03-19T11:30:40.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:40.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.707610+0000", 2024-03-19T11:30:40.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00030568799999999998 2024-03-19T11:30:40.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:40.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.707984+0000", 2024-03-19T11:30:40.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00037399299999999998 2024-03-19T11:30:40.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.345 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.345 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2", 2024-03-19T11:30:40.345 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.708409+0000", 2024-03-19T11:30:40.345 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00042503099999999998 2024-03-19T11:30:40.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.345 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.345 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:40.345 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.712919+0000", 2024-03-19T11:30:40.345 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0045099010000000002 2024-03-19T11:30:40.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.345 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.345 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:40.346 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.719037+0000", 2024-03-19T11:30:40.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061173770000000002 2024-03-19T11:30:40.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:40.346 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.719116+0000", 2024-03-19T11:30:40.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9935e-05 2024-03-19T11:30:40.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:40.346 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.719227+0000", 2024-03-19T11:30:40.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011027900000000001 2024-03-19T11:30:40.346 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.346 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:40.347 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.347 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.347 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:848 4.1 4:866f3dcc:::smithi03829856-31:head [read 0~2941192,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:40.347 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.757468+0000", 2024-03-19T11:30:40.347 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 31.590909611000001, 2024-03-19T11:30:40.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015020110999999999, 2024-03-19T11:30:40.347 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:40.347 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:40.347 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:40.347 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:40.347 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:40.347 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 848 2024-03-19T11:30:40.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.347 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:40.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:40.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757468+0000", 2024-03-19T11:30:40.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:40.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757468+0000", 2024-03-19T11:30:40.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:40.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757471+0000", 2024-03-19T11:30:40.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.67e-06 2024-03-19T11:30:40.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.349 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.349 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:40.349 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757488+0000", 2024-03-19T11:30:40.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6223e-05 2024-03-19T11:30:40.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.349 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.349 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:40.349 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757521+0000", 2024-03-19T11:30:40.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.358e-05 2024-03-19T11:30:40.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.349 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.349 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:40.350 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757543+0000", 2024-03-19T11:30:40.350 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1330000000000001e-05 2024-03-19T11:30:40.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.350 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:40.350 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757745+0000", 2024-03-19T11:30:40.350 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020259799999999999 2024-03-19T11:30:40.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.350 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op must wait for map", 2024-03-19T11:30:40.350 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757760+0000", 2024-03-19T11:30:40.350 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4647000000000001e-05 2024-03-19T11:30:40.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:40.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763972+0000", 2024-03-19T11:30:40.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062126480000000003 2024-03-19T11:30:40.351 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:40.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.764547+0000", 2024-03-19T11:30:40.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057440200000000001 2024-03-19T11:30:40.351 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:40.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.772488+0000", 2024-03-19T11:30:40.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079410130000000002 2024-03-19T11:30:40.351 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.352 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:40.352 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.352 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.352 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:849 4.1 4:866f3dcc:::smithi03829856-31:head [sparse-read 0~1359577] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:40.352 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.757583+0000", 2024-03-19T11:30:40.352 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 31.590793918999999, 2024-03-19T11:30:40.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016241202, 2024-03-19T11:30:40.352 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:40.352 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:40.352 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:40.352 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:40.352 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:40.352 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 849 2024-03-19T11:30:40.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.353 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:40.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:40.353 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757583+0000", 2024-03-19T11:30:40.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:40.353 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757583+0000", 2024-03-19T11:30:40.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:40.353 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757584+0000", 2024-03-19T11:30:40.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.077e-06 2024-03-19T11:30:40.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.354 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:40.354 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757593+0000", 2024-03-19T11:30:40.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3059999999999998e-06 2024-03-19T11:30:40.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.354 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:40.354 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757608+0000", 2024-03-19T11:30:40.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5526999999999999e-05 2024-03-19T11:30:40.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.354 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:40.355 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757618+0000", 2024-03-19T11:30:40.355 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5999999999999996e-06 2024-03-19T11:30:40.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.355 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:40.355 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757886+0000", 2024-03-19T11:30:40.355 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026839099999999999 2024-03-19T11:30:40.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.355 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:40.356 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757899+0000", 2024-03-19T11:30:40.356 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3004e-05 2024-03-19T11:30:40.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.356 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:40.356 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.772651+0000", 2024-03-19T11:30:40.356 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014751702 2024-03-19T11:30:40.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.356 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:40.356 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.772924+0000", 2024-03-19T11:30:40.356 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027314199999999998 2024-03-19T11:30:40.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.356 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:40.357 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.773825+0000", 2024-03-19T11:30:40.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00090045299999999995 2024-03-19T11:30:40.357 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.357 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:40.357 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.357 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.357 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:850 4.1 4:866f3dcc:::smithi03829856-31:head [read 0~2941192,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:40.357 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.757636+0000", 2024-03-19T11:30:40.357 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 31.590741249000001, 2024-03-19T11:30:40.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018834801000000002, 2024-03-19T11:30:40.357 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:40.357 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:40.357 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:40.358 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:40.358 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:40.358 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 850 2024-03-19T11:30:40.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.358 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:40.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.358 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:40.358 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757636+0000", 2024-03-19T11:30:40.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.358 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:40.358 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757636+0000", 2024-03-19T11:30:40.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:40.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757637+0000", 2024-03-19T11:30:40.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5999999999999991e-07 2024-03-19T11:30:40.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:40.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757643+0000", 2024-03-19T11:30:40.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3199999999999996e-06 2024-03-19T11:30:40.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:40.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757670+0000", 2024-03-19T11:30:40.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6682999999999998e-05 2024-03-19T11:30:40.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:40.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757679+0000", 2024-03-19T11:30:40.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.986e-06 2024-03-19T11:30:40.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:40.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757961+0000", 2024-03-19T11:30:40.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028230799999999999 2024-03-19T11:30:40.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.362 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:40.362 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757971+0000", 2024-03-19T11:30:40.363 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.38e-06 2024-03-19T11:30:40.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.363 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.363 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:40.364 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.774079+0000", 2024-03-19T11:30:40.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016108573000000001 2024-03-19T11:30:40.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.364 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:40.364 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.774306+0000", 2024-03-19T11:30:40.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022637499999999999 2024-03-19T11:30:40.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:40.365 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776471+0000", 2024-03-19T11:30:40.365 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021652160000000002 2024-03-19T11:30:40.365 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.365 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:40.365 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.366 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:851 4.9 4:953e720b:::smithi03829856-45:head [write 591613~454162,stat] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:40.366 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.763514+0000", 2024-03-19T11:30:40.366 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 31.584863058, 2024-03-19T11:30:40.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027825888999999999, 2024-03-19T11:30:40.366 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:40.366 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:40.366 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:40.367 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:40.367 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:40.367 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 851 2024-03-19T11:30:40.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.367 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:40.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.367 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:40.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763514+0000", 2024-03-19T11:30:40.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.368 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:40.368 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763514+0000", 2024-03-19T11:30:40.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.369 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.369 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:40.369 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763517+0000", 2024-03-19T11:30:40.369 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8499999999999998e-06 2024-03-19T11:30:40.369 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.369 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.369 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:40.369 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.764714+0000", 2024-03-19T11:30:40.370 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011971270000000001 2024-03-19T11:30:40.370 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.370 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.370 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:40.370 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.764745+0000", 2024-03-19T11:30:40.370 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0546999999999997e-05 2024-03-19T11:30:40.370 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.370 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.371 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:40.371 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.764767+0000", 2024-03-19T11:30:40.371 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2286999999999999e-05 2024-03-19T11:30:40.371 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.371 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.371 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:40.371 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.773924+0000", 2024-03-19T11:30:40.371 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0091567360000000004 2024-03-19T11:30:40.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.372 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:40.372 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.774217+0000", 2024-03-19T11:30:40.372 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029350099999999999 2024-03-19T11:30:40.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.372 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:40.373 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.774770+0000", 2024-03-19T11:30:40.373 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00055288600000000005 2024-03-19T11:30:40.373 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.373 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.373 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:40.373 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.786413+0000", 2024-03-19T11:30:40.373 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011643229999999999 2024-03-19T11:30:40.373 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.373 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.373 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:40.373 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791104+0000", 2024-03-19T11:30:40.374 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0046900830000000003 2024-03-19T11:30:40.374 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.374 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:40.374 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791248+0000", 2024-03-19T11:30:40.374 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014410200000000001 2024-03-19T11:30:40.374 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.374 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:40.375 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791340+0000", 2024-03-19T11:30:40.375 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2540000000000005e-05 2024-03-19T11:30:40.375 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.375 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:40.375 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.375 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.375 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.376 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:852 4.9 4:953e720b:::smithi03829856-45:head [write 1719614~417353] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:40.376 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.769707+0000", 2024-03-19T11:30:40.376 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 31.578670032000002, 2024-03-19T11:30:40.376 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022476643000000001, 2024-03-19T11:30:40.376 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:40.376 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:40.376 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:40.377 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:40.377 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:40.377 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 852 2024-03-19T11:30:40.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.377 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:40.377 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.377 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:40.377 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.769707+0000", 2024-03-19T11:30:40.377 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.378 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.378 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.378 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:40.378 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.769707+0000", 2024-03-19T11:30:40.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.378 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.378 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.378 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:40.378 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.769710+0000", 2024-03-19T11:30:40.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6230000000000001e-06 2024-03-19T11:30:40.379 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.379 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.379 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:40.379 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.770385+0000", 2024-03-19T11:30:40.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00067504900000000005 2024-03-19T11:30:40.379 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.379 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.380 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:40.380 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.770419+0000", 2024-03-19T11:30:40.380 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3528999999999997e-05 2024-03-19T11:30:40.380 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.380 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.380 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:40.380 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.770436+0000", 2024-03-19T11:30:40.381 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7615999999999999e-05 2024-03-19T11:30:40.381 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.381 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.381 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:40.381 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.781327+0000", 2024-03-19T11:30:40.381 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010891343 2024-03-19T11:30:40.381 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.381 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.381 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:40.381 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.781616+0000", 2024-03-19T11:30:40.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028857399999999999 2024-03-19T11:30:40.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.382 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:40.382 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.782125+0000", 2024-03-19T11:30:40.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00050941599999999995 2024-03-19T11:30:40.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.382 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:40.383 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.786606+0000", 2024-03-19T11:30:40.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044809560000000003 2024-03-19T11:30:40.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:40.383 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791995+0000", 2024-03-19T11:30:40.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053889150000000002 2024-03-19T11:30:40.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.384 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:40.384 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.792084+0000", 2024-03-19T11:30:40.384 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8633000000000003e-05 2024-03-19T11:30:40.384 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.384 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.384 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:40.384 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.792184+0000", 2024-03-19T11:30:40.384 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9988999999999994e-05 2024-03-19T11:30:40.385 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.385 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:40.385 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.385 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.385 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:853 4.9 4:953e720b:::smithi03829856-45:head [write 2570780~420833] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:40.385 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.775636+0000", 2024-03-19T11:30:40.385 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 31.572741410999999, 2024-03-19T11:30:40.386 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018795850999999999, 2024-03-19T11:30:40.386 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:40.386 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:40.386 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:40.386 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:40.386 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:40.386 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 853 2024-03-19T11:30:40.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.386 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:40.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.386 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:40.387 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.775636+0000", 2024-03-19T11:30:40.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.387 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.387 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:40.387 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.775636+0000", 2024-03-19T11:30:40.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.387 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.387 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:40.387 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.775639+0000", 2024-03-19T11:30:40.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4529999999999998e-06 2024-03-19T11:30:40.387 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:40.388 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776492+0000", 2024-03-19T11:30:40.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00085245000000000004 2024-03-19T11:30:40.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:40.388 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776526+0000", 2024-03-19T11:30:40.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4090000000000001e-05 2024-03-19T11:30:40.389 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.389 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.389 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:40.389 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776550+0000", 2024-03-19T11:30:40.389 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4139999999999999e-05 2024-03-19T11:30:40.389 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.389 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.389 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:40.389 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.786752+0000", 2024-03-19T11:30:40.390 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010201834 2024-03-19T11:30:40.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.390 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.390 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:40.390 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.786994+0000", 2024-03-19T11:30:40.390 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000241952 2024-03-19T11:30:40.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.390 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.391 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:40.391 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.787301+0000", 2024-03-19T11:30:40.391 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00030690699999999998 2024-03-19T11:30:40.391 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.391 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.391 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:40.391 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791965+0000", 2024-03-19T11:30:40.391 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0046644379999999999 2024-03-19T11:30:40.392 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.392 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:40.392 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.794165+0000", 2024-03-19T11:30:40.392 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021995389999999999 2024-03-19T11:30:40.392 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.392 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:40.393 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.794313+0000", 2024-03-19T11:30:40.393 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000148369 2024-03-19T11:30:40.393 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.393 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:40.393 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.794432+0000", 2024-03-19T11:30:40.393 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011867900000000001 2024-03-19T11:30:40.393 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.394 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:40.394 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.394 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.394 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:854 4.9 4:953e720b:::smithi03829856-45:head [setxattr _header (56) in=7b,truncate 2991613] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:40.394 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.776592+0000", 2024-03-19T11:30:40.394 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 31.571785688999999, 2024-03-19T11:30:40.394 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022815621000000001, 2024-03-19T11:30:40.395 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:40.395 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:40.395 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:40.395 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:40.395 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:40.395 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 854 2024-03-19T11:30:40.395 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.395 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:40.396 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.396 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:40.396 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776592+0000", 2024-03-19T11:30:40.396 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.396 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.396 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.396 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:40.396 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776592+0000", 2024-03-19T11:30:40.396 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.397 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.397 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.397 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:40.397 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776594+0000", 2024-03-19T11:30:40.397 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.95e-06 2024-03-19T11:30:40.397 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.397 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.397 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:40.397 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776603+0000", 2024-03-19T11:30:40.397 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2399999999999996e-06 2024-03-19T11:30:40.397 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.397 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.397 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:40.398 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776623+0000", 2024-03-19T11:30:40.398 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.035e-05 2024-03-19T11:30:40.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.398 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.398 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:40.398 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776639+0000", 2024-03-19T11:30:40.398 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5642999999999999e-05 2024-03-19T11:30:40.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.398 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.398 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:40.398 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.792324+0000", 2024-03-19T11:30:40.398 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015685075999999999 2024-03-19T11:30:40.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.398 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.399 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:40.399 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.792599+0000", 2024-03-19T11:30:40.399 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000275264 2024-03-19T11:30:40.399 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.399 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.399 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:40.399 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.793092+0000", 2024-03-19T11:30:40.399 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00049264600000000001 2024-03-19T11:30:40.399 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.399 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.399 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:40.399 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.794668+0000", 2024-03-19T11:30:40.399 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001576072 2024-03-19T11:30:40.399 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.400 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.400 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:40.400 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.799082+0000", 2024-03-19T11:30:40.400 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044145190000000004 2024-03-19T11:30:40.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.400 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.400 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:40.400 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.799249+0000", 2024-03-19T11:30:40.400 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016701199999999999 2024-03-19T11:30:40.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.400 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.400 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:40.400 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.799407+0000", 2024-03-19T11:30:40.400 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000157849 2024-03-19T11:30:40.400 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.401 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:40.401 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.401 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.401 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.401 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:855 4.9 4:953e720b:::smithi03829856-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:40.401 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.776660+0000", 2024-03-19T11:30:40.401 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 31.571717096, 2024-03-19T11:30:40.401 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023419786000000001, 2024-03-19T11:30:40.401 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:40.401 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:40.401 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:40.401 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:40.401 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:40.401 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 855 2024-03-19T11:30:40.401 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.402 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:40.402 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.402 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:40.402 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776660+0000", 2024-03-19T11:30:40.402 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.402 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.402 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.402 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:40.402 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776660+0000", 2024-03-19T11:30:40.402 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.402 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.402 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.402 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:40.402 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776662+0000", 2024-03-19T11:30:40.403 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3430000000000001e-06 2024-03-19T11:30:40.403 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.403 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.403 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:40.403 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776668+0000", 2024-03-19T11:30:40.403 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3500000000000002e-06 2024-03-19T11:30:40.403 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.403 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.403 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:40.403 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776687+0000", 2024-03-19T11:30:40.403 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9273999999999999e-05 2024-03-19T11:30:40.403 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.403 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.403 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:40.403 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776705+0000", 2024-03-19T11:30:40.404 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7839999999999999e-05 2024-03-19T11:30:40.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.404 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.404 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:40.404 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.793874+0000", 2024-03-19T11:30:40.404 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017168597000000001 2024-03-19T11:30:40.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.404 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.404 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-03-19T11:30:40.404 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.794132+0000", 2024-03-19T11:30:40.404 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025790399999999999 2024-03-19T11:30:40.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.404 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.404 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:40.405 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.799504+0000", 2024-03-19T11:30:40.405 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053724159999999997 2024-03-19T11:30:40.405 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.405 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.405 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:40.405 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.799782+0000", 2024-03-19T11:30:40.405 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027785499999999997 2024-03-19T11:30:40.405 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.405 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.405 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:40.405 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.800080+0000", 2024-03-19T11:30:40.405 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029820699999999999 2024-03-19T11:30:40.405 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.405 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:40.405 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.406 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:871 4.d 4:b4ef97e0:::smithi03829856-3:head [write 642034~796262,stat] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:40.406 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:09.028192+0000", 2024-03-19T11:30:40.406 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 31.320185365, 2024-03-19T11:30:40.406 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013367027, 2024-03-19T11:30:40.406 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:40.406 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:40.406 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:40.406 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:40.406 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:40.406 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 871 2024-03-19T11:30:40.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.406 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:40.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.407 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:40.407 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.028192+0000", 2024-03-19T11:30:40.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.407 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.407 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.407 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:40.407 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.028192+0000", 2024-03-19T11:30:40.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.407 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.407 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.407 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:40.407 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.028194+0000", 2024-03-19T11:30:40.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3999999999999999e-06 2024-03-19T11:30:40.407 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.408 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:40.408 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.029358+0000", 2024-03-19T11:30:40.408 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011635549999999999 2024-03-19T11:30:40.408 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.408 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:40.408 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.029390+0000", 2024-03-19T11:30:40.408 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2215999999999998e-05 2024-03-19T11:30:40.408 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.408 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:40.408 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.029406+0000", 2024-03-19T11:30:40.408 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5543e-05 2024-03-19T11:30:40.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.409 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.409 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:40.409 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.029525+0000", 2024-03-19T11:30:40.409 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000119269 2024-03-19T11:30:40.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.409 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.409 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:40.409 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.029676+0000", 2024-03-19T11:30:40.409 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000150729 2024-03-19T11:30:40.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.409 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.409 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2", 2024-03-19T11:30:40.409 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.029994+0000", 2024-03-19T11:30:40.410 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00031876099999999998 2024-03-19T11:30:40.410 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.410 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:40.410 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.040441+0000", 2024-03-19T11:30:40.410 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010446446 2024-03-19T11:30:40.410 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.410 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:40.410 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.041327+0000", 2024-03-19T11:30:40.410 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00088595299999999998 2024-03-19T11:30:40.410 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.410 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:40.411 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.041409+0000", 2024-03-19T11:30:40.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2332999999999999e-05 2024-03-19T11:30:40.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.411 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:40.411 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.041559+0000", 2024-03-19T11:30:40.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000149822 2024-03-19T11:30:40.411 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.411 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:40.411 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.411 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:993 4.e 4:72163e4c:::smithi03829856-39:head [write 776490~714607] snapc 5=[5,2] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-03-19T11:30:40.411 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:12.362932+0000", 2024-03-19T11:30:40.411 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.985445672000001, 2024-03-19T11:30:40.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018349900999999998, 2024-03-19T11:30:40.412 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:40.412 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:40.412 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:40.412 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:40.412 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:40.412 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 993 2024-03-19T11:30:40.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.412 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:40.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:40.412 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.362932+0000", 2024-03-19T11:30:40.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.413 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:40.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.362932+0000", 2024-03-19T11:30:40.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.413 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:40.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.362935+0000", 2024-03-19T11:30:40.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.88e-06 2024-03-19T11:30:40.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.413 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:40.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.366756+0000", 2024-03-19T11:30:40.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003821217 2024-03-19T11:30:40.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.414 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:40.414 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.366796+0000", 2024-03-19T11:30:40.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9866000000000001e-05 2024-03-19T11:30:40.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.414 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:40.414 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.366813+0000", 2024-03-19T11:30:40.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.755e-05 2024-03-19T11:30:40.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.414 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:40.414 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.366993+0000", 2024-03-19T11:30:40.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000179605 2024-03-19T11:30:40.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:40.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.367167+0000", 2024-03-19T11:30:40.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017421200000000001 2024-03-19T11:30:40.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-19T11:30:40.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.367505+0000", 2024-03-19T11:30:40.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033800099999999999 2024-03-19T11:30:40.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:40.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.380361+0000", 2024-03-19T11:30:40.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012855961000000001 2024-03-19T11:30:40.416 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.416 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:40.416 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.380987+0000", 2024-03-19T11:30:40.416 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00062590499999999995 2024-03-19T11:30:40.416 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.416 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:40.416 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.381149+0000", 2024-03-19T11:30:40.416 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000162239 2024-03-19T11:30:40.416 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.416 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:40.416 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.381282+0000", 2024-03-19T11:30:40.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013246500000000001 2024-03-19T11:30:40.417 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.417 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:40.417 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.417 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.417 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1072 4.1 4:8401c1f0:::smithi03829856-43:head [write 494139~557703,stat] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:40.417 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:13.972007+0000", 2024-03-19T11:30:40.417 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 26.376369921999999, 2024-03-19T11:30:40.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.01935839, 2024-03-19T11:30:40.417 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:40.417 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:40.418 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:40.418 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:40.418 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:40.418 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1072 2024-03-19T11:30:40.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.418 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:40.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:40.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.972007+0000", 2024-03-19T11:30:40.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:40.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.972007+0000", 2024-03-19T11:30:40.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:40.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.972011+0000", 2024-03-19T11:30:40.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9360000000000001e-06 2024-03-19T11:30:40.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:40.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.972881+0000", 2024-03-19T11:30:40.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00086922699999999998 2024-03-19T11:30:40.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:40.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.972917+0000", 2024-03-19T11:30:40.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6359999999999997e-05 2024-03-19T11:30:40.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:40.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.972934+0000", 2024-03-19T11:30:40.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7249e-05 2024-03-19T11:30:40.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:40.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.973132+0000", 2024-03-19T11:30:40.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019760200000000001 2024-03-19T11:30:40.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:40.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.973303+0000", 2024-03-19T11:30:40.421 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017142899999999999 2024-03-19T11:30:40.421 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.421 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.421 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:40.421 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.973619+0000", 2024-03-19T11:30:40.421 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00031594700000000002 2024-03-19T11:30:40.421 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.421 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.421 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:40.421 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.985856+0000", 2024-03-19T11:30:40.421 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012236349000000001 2024-03-19T11:30:40.421 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.421 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.421 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:40.421 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.991132+0000", 2024-03-19T11:30:40.422 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052764960000000003 2024-03-19T11:30:40.422 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.422 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:40.422 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.991241+0000", 2024-03-19T11:30:40.422 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000108842 2024-03-19T11:30:40.422 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.422 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:40.422 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.991366+0000", 2024-03-19T11:30:40.422 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012495299999999999 2024-03-19T11:30:40.422 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.422 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:40.422 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.423 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1162 4.1 4:8401c1f0:::smithi03829856-43:head [write 594981~609197] snapc 8=[8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-03-19T11:30:40.423 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.838981+0000", 2024-03-19T11:30:40.423 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.509396567, 2024-03-19T11:30:40.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027704349999999999, 2024-03-19T11:30:40.423 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:40.423 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:40.423 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:40.423 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:40.423 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:40.423 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1162 2024-03-19T11:30:40.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.423 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:40.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:40.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.838981+0000", 2024-03-19T11:30:40.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:40.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.838981+0000", 2024-03-19T11:30:40.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:40.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.838983+0000", 2024-03-19T11:30:40.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5399999999999998e-06 2024-03-19T11:30:40.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:40.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841322+0000", 2024-03-19T11:30:40.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002338592 2024-03-19T11:30:40.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:40.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841371+0000", 2024-03-19T11:30:40.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9123999999999999e-05 2024-03-19T11:30:40.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:40.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841399+0000", 2024-03-19T11:30:40.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7718000000000001e-05 2024-03-19T11:30:40.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.426 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:40.426 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841577+0000", 2024-03-19T11:30:40.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000178097 2024-03-19T11:30:40.426 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.426 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:40.426 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841825+0000", 2024-03-19T11:30:40.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00024824299999999999 2024-03-19T11:30:40.426 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.426 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:40.426 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842227+0000", 2024-03-19T11:30:40.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00040212199999999998 2024-03-19T11:30:40.426 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.427 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:40.427 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.865182+0000", 2024-03-19T11:30:40.427 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022954810999999999 2024-03-19T11:30:40.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.427 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:40.427 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.866360+0000", 2024-03-19T11:30:40.427 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001178449 2024-03-19T11:30:40.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.427 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:40.427 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.866538+0000", 2024-03-19T11:30:40.427 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017791000000000001 2024-03-19T11:30:40.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.428 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:40.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.866685+0000", 2024-03-19T11:30:40.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000146744 2024-03-19T11:30:40.428 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.428 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:40.428 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.428 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1163 4.1 4:8401c1f0:::smithi03829856-43:head [write 1974123~745028] snapc 8=[8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-03-19T11:30:40.428 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.851879+0000", 2024-03-19T11:30:40.428 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.496498445, 2024-03-19T11:30:40.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015531774, 2024-03-19T11:30:40.429 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:40.429 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:40.429 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:40.429 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:40.429 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:40.429 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1163 2024-03-19T11:30:40.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.429 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:40.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:40.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.851879+0000", 2024-03-19T11:30:40.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.430 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:40.430 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.851879+0000", 2024-03-19T11:30:40.430 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.430 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.430 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:40.430 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.851882+0000", 2024-03-19T11:30:40.430 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6960000000000001e-06 2024-03-19T11:30:40.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.430 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.430 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:40.430 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.853204+0000", 2024-03-19T11:30:40.430 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0013226620000000001 2024-03-19T11:30:40.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.430 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:40.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.853231+0000", 2024-03-19T11:30:40.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6775e-05 2024-03-19T11:30:40.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:40.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.853246+0000", 2024-03-19T11:30:40.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5403000000000001e-05 2024-03-19T11:30:40.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:40.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.853363+0000", 2024-03-19T11:30:40.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000116296 2024-03-19T11:30:40.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.432 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:40.432 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.853572+0000", 2024-03-19T11:30:40.432 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020931500000000001 2024-03-19T11:30:40.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.432 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:40.432 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.853797+0000", 2024-03-19T11:30:40.432 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022544700000000001 2024-03-19T11:30:40.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.432 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:40.432 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.865195+0000", 2024-03-19T11:30:40.432 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011397981999999999 2024-03-19T11:30:40.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.433 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:40.433 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.866916+0000", 2024-03-19T11:30:40.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017204449999999999 2024-03-19T11:30:40.433 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.433 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:40.433 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.867129+0000", 2024-03-19T11:30:40.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021317999999999999 2024-03-19T11:30:40.433 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.433 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:40.433 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.867411+0000", 2024-03-19T11:30:40.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028157300000000002 2024-03-19T11:30:40.433 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.433 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:40.434 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.434 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.434 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.434 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1205 4.1 4:8401c1f0:::smithi03829856-43:head [assert-version v50,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:40.434 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:18.112028+0000", 2024-03-19T11:30:40.434 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.236349311000001, 2024-03-19T11:30:40.434 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013579206, 2024-03-19T11:30:40.434 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:40.434 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:40.434 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:40.434 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:40.434 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:40.434 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1205 2024-03-19T11:30:40.434 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.435 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:40.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.435 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:40.435 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112028+0000", 2024-03-19T11:30:40.435 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.435 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.435 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:40.435 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112028+0000", 2024-03-19T11:30:40.435 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.435 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.435 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:40.435 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112030+0000", 2024-03-19T11:30:40.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4549999999999998e-06 2024-03-19T11:30:40.436 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.436 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.436 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:40.436 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112043+0000", 2024-03-19T11:30:40.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2721999999999999e-05 2024-03-19T11:30:40.436 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.436 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.436 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:40.436 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112075+0000", 2024-03-19T11:30:40.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.205e-05 2024-03-19T11:30:40.436 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.436 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.436 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:40.436 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112091+0000", 2024-03-19T11:30:40.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5364000000000001e-05 2024-03-19T11:30:40.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.437 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.437 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:40.437 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112223+0000", 2024-03-19T11:30:40.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013224100000000001 2024-03-19T11:30:40.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.437 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.437 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:40.437 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112393+0000", 2024-03-19T11:30:40.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000169681 2024-03-19T11:30:40.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.437 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.437 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:40.438 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.120919+0000", 2024-03-19T11:30:40.438 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085260140000000002 2024-03-19T11:30:40.438 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.438 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:40.438 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125207+0000", 2024-03-19T11:30:40.438 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004288722 2024-03-19T11:30:40.438 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.438 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:40.438 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125343+0000", 2024-03-19T11:30:40.438 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013558400000000001 2024-03-19T11:30:40.438 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.439 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:40.439 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125497+0000", 2024-03-19T11:30:40.439 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000153678 2024-03-19T11:30:40.439 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.439 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.439 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:40.439 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125607+0000", 2024-03-19T11:30:40.439 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110695 2024-03-19T11:30:40.439 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.439 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:40.439 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.439 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.439 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.439 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1208 4.1 4:866f3dcc:::smithi03829856-31:head [read 0~2941192,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:40.439 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:18.112252+0000", 2024-03-19T11:30:40.440 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.236125740999999, 2024-03-19T11:30:40.440 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.01291702, 2024-03-19T11:30:40.440 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:40.440 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:40.440 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:40.440 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:40.440 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:40.440 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1208 2024-03-19T11:30:40.440 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.440 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:40.440 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.440 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:40.440 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112252+0000", 2024-03-19T11:30:40.440 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.441 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.441 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:40.441 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112252+0000", 2024-03-19T11:30:40.441 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.441 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.441 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:40.441 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112253+0000", 2024-03-19T11:30:40.441 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6299999999999993e-07 2024-03-19T11:30:40.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.441 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.441 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:40.441 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112257+0000", 2024-03-19T11:30:40.442 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4360000000000004e-06 2024-03-19T11:30:40.442 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.442 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.442 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:40.442 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112272+0000", 2024-03-19T11:30:40.442 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.518e-05 2024-03-19T11:30:40.442 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.442 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.442 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:40.442 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112281+0000", 2024-03-19T11:30:40.442 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.422e-06 2024-03-19T11:30:40.442 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.442 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.442 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:40.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.122861+0000", 2024-03-19T11:30:40.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010580523 2024-03-19T11:30:40.443 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.443 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.443 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:40.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123148+0000", 2024-03-19T11:30:40.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028714499999999999 2024-03-19T11:30:40.443 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.443 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.443 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:40.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125169+0000", 2024-03-19T11:30:40.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0020203510000000001 2024-03-19T11:30:40.443 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.443 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:40.444 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.444 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.444 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.444 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1243 4.9 4:9ef41243:::smithi03829856-5:head [tier-promote] snapc 9=[9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:40.444 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:18.345759+0000", 2024-03-19T11:30:40.444 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.002618129999998, 2024-03-19T11:30:40.444 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018400962999999999, 2024-03-19T11:30:40.444 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:40.444 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:40.444 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:40.444 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:40.444 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:40.444 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1243 2024-03-19T11:30:40.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.445 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:40.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:40.445 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.345759+0000", 2024-03-19T11:30:40.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:40.445 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.345759+0000", 2024-03-19T11:30:40.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:40.445 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.345763+0000", 2024-03-19T11:30:40.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3790000000000002e-06 2024-03-19T11:30:40.446 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.446 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:40.446 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.345776+0000", 2024-03-19T11:30:40.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3784000000000001e-05 2024-03-19T11:30:40.446 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.446 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:40.446 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.345812+0000", 2024-03-19T11:30:40.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5599999999999998e-05 2024-03-19T11:30:40.446 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.446 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:40.447 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.345829+0000", 2024-03-19T11:30:40.447 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6722999999999999e-05 2024-03-19T11:30:40.447 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.447 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.447 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:40.447 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.345964+0000", 2024-03-19T11:30:40.447 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013554800000000001 2024-03-19T11:30:40.447 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.447 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.447 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:40.447 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.346133+0000", 2024-03-19T11:30:40.447 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000169211 2024-03-19T11:30:40.447 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.447 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.448 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:40.448 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.364160+0000", 2024-03-19T11:30:40.448 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018026718000000001 2024-03-19T11:30:40.448 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.448 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:40.448 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.448 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.448 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.448 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1271 4.9 4:953e720b:::smithi03829856-45:2 [read 795136~1008128] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-03-19T11:30:40.448 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:21.868056+0000", 2024-03-19T11:30:40.448 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.480321062000002, 2024-03-19T11:30:40.448 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.047270591000000001, 2024-03-19T11:30:40.448 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:40.448 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:40.448 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:40.449 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:40.449 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:40.449 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1271 2024-03-19T11:30:40.449 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.449 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:40.449 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.449 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:40.449 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868056+0000", 2024-03-19T11:30:40.449 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.449 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.449 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.449 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:40.449 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868056+0000", 2024-03-19T11:30:40.449 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.449 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.450 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.450 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:40.450 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868059+0000", 2024-03-19T11:30:40.450 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0469999999999998e-06 2024-03-19T11:30:40.450 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.450 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.450 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:40.450 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868072+0000", 2024-03-19T11:30:40.450 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2700999999999999e-05 2024-03-19T11:30:40.450 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.450 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.450 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:40.450 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868106+0000", 2024-03-19T11:30:40.450 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4152000000000001e-05 2024-03-19T11:30:40.451 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.451 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.451 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:40.451 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868126+0000", 2024-03-19T11:30:40.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9386999999999999e-05 2024-03-19T11:30:40.451 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.451 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.451 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:40.451 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.911447+0000", 2024-03-19T11:30:40.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.043321164000000002 2024-03-19T11:30:40.451 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.451 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.451 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:40.451 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.911762+0000", 2024-03-19T11:30:40.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00031492400000000001 2024-03-19T11:30:40.452 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.452 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:40.452 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.915327+0000", 2024-03-19T11:30:40.452 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003565216 2024-03-19T11:30:40.452 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.452 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:40.452 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.452 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.452 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1617 4.e 4:7773c291:::smithi03829856-11:head [write 450454~642026,stat] snapc 10=[10,f,e,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-03-19T11:30:40.452 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:35.152724+0000", 2024-03-19T11:30:40.452 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.1956528679999998, 2024-03-19T11:30:40.452 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016139337, 2024-03-19T11:30:40.453 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:40.453 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:40.453 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:40.453 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:40.453 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:40.453 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1617 2024-03-19T11:30:40.453 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.453 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:40.453 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.453 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:40.453 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.152724+0000", 2024-03-19T11:30:40.453 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.453 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.453 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.453 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:40.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.152724+0000", 2024-03-19T11:30:40.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:40.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.152727+0000", 2024-03-19T11:30:40.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8399999999999999e-06 2024-03-19T11:30:40.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:40.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.153790+0000", 2024-03-19T11:30:40.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010623849999999999 2024-03-19T11:30:40.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:40.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.153825+0000", 2024-03-19T11:30:40.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5280000000000001e-05 2024-03-19T11:30:40.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:40.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.153842+0000", 2024-03-19T11:30:40.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6942000000000001e-05 2024-03-19T11:30:40.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:40.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.154018+0000", 2024-03-19T11:30:40.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017568 2024-03-19T11:30:40.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.456 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:40.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.154302+0000", 2024-03-19T11:30:40.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028463400000000003 2024-03-19T11:30:40.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.456 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-19T11:30:40.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.154811+0000", 2024-03-19T11:30:40.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00050833300000000003 2024-03-19T11:30:40.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.456 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:40.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.168286+0000", 2024-03-19T11:30:40.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013475354 2024-03-19T11:30:40.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.457 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:40.457 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.168631+0000", 2024-03-19T11:30:40.457 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00034488800000000001 2024-03-19T11:30:40.457 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.457 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.457 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:40.457 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.168749+0000", 2024-03-19T11:30:40.457 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011799900000000001 2024-03-19T11:30:40.457 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.457 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.457 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:40.457 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.168864+0000", 2024-03-19T11:30:40.457 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011500199999999999 2024-03-19T11:30:40.457 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.457 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:40.458 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.458 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.458 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.458 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1679 4.e 4:72163e4c:::smithi03829856-39:head [write 517682~543022,stat] snapc 10=[10,e,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-03-19T11:30:40.458 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:36.388669+0000", 2024-03-19T11:30:40.458 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 3.9597085349999999, 2024-03-19T11:30:40.458 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016197706999999999, 2024-03-19T11:30:40.458 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:40.458 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:40.458 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:40.458 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:40.458 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:40.458 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1679 2024-03-19T11:30:40.458 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.458 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:40.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.459 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:40.459 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.388669+0000", 2024-03-19T11:30:40.459 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.459 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:40.459 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.388669+0000", 2024-03-19T11:30:40.459 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.459 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:40.459 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.388672+0000", 2024-03-19T11:30:40.459 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4130000000000002e-06 2024-03-19T11:30:40.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.460 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:40.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.389483+0000", 2024-03-19T11:30:40.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00081121800000000005 2024-03-19T11:30:40.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.460 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:40.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.389507+0000", 2024-03-19T11:30:40.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3482999999999999e-05 2024-03-19T11:30:40.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.460 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:40.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.389517+0000", 2024-03-19T11:30:40.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0251999999999999e-05 2024-03-19T11:30:40.461 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.461 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:40.461 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.389672+0000", 2024-03-19T11:30:40.461 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015465300000000001 2024-03-19T11:30:40.461 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.461 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:40.461 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.389839+0000", 2024-03-19T11:30:40.461 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016710800000000001 2024-03-19T11:30:40.461 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.461 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-19T11:30:40.461 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.390102+0000", 2024-03-19T11:30:40.461 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026344900000000002 2024-03-19T11:30:40.462 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.462 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.462 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:40.462 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.401899+0000", 2024-03-19T11:30:40.462 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011796836 2024-03-19T11:30:40.462 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.462 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.462 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:40.462 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.404530+0000", 2024-03-19T11:30:40.462 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026311949999999998 2024-03-19T11:30:40.462 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.462 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.462 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:40.462 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.404722+0000", 2024-03-19T11:30:40.463 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019109300000000001 2024-03-19T11:30:40.463 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.463 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.463 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:40.463 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.404867+0000", 2024-03-19T11:30:40.463 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000145007 2024-03-19T11:30:40.463 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.463 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:40.463 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.463 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.463 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:40.463 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:40.465 INFO:tasks.rados.rados.0.smithi038.stdout:963: finishing write tid 1 to smithi03829856-49 2024-03-19T11:30:40.465 INFO:tasks.rados.rados.0.smithi038.stdout:963: finishing write tid 2 to smithi03829856-49 2024-03-19T11:30:40.465 INFO:tasks.rados.rados.0.smithi038.stdout:963: finishing write tid 3 to smithi03829856-49 2024-03-19T11:30:40.465 INFO:tasks.rados.rados.0.smithi038.stdout:963: finishing write tid 4 to smithi03829856-49 2024-03-19T11:30:40.465 INFO:tasks.rados.rados.0.smithi038.stdout:963: oid 49 updating version 0 to 22 2024-03-19T11:30:40.465 INFO:tasks.rados.rados.0.smithi038.stdout:963: oid 49 updating version 22 to 23 2024-03-19T11:30:40.465 INFO:tasks.rados.rados.0.smithi038.stdout:963: oid 49 version 23 is already newer than 21 2024-03-19T11:30:40.465 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 49 v 23 (ObjNum 136 snap 15 seq_num 136) dirty exists 2024-03-19T11:30:40.466 INFO:tasks.rados.rados.0.smithi038.stdout:963: left oid 49 (ObjNum 136 snap 15 seq_num 136) 2024-03-19T11:30:40.466 INFO:tasks.rados.rados.0.smithi038.stdout:964: finishing write tid 1 to smithi03829856-5 2024-03-19T11:30:40.466 INFO:tasks.rados.rados.0.smithi038.stdout:964: finishing write tid 2 to smithi03829856-5 2024-03-19T11:30:40.466 INFO:tasks.rados.rados.0.smithi038.stdout:964: finishing write tid 3 to smithi03829856-5 2024-03-19T11:30:40.466 INFO:tasks.rados.rados.0.smithi038.stdout:964: finishing write tid 4 to smithi03829856-5 2024-03-19T11:30:40.466 INFO:tasks.rados.rados.0.smithi038.stdout:964: oid 5 updating version 0 to 70 2024-03-19T11:30:40.466 INFO:tasks.rados.rados.0.smithi038.stdout:964: oid 5 version 70 is already newer than 68 2024-03-19T11:30:40.466 INFO:tasks.rados.rados.0.smithi038.stdout:964: oid 5 version 70 is already newer than 69 2024-03-19T11:30:40.466 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 5 v 70 (ObjNum 137 snap 15 seq_num 137) dirty exists 2024-03-19T11:30:40.466 INFO:tasks.rados.rados.0.smithi038.stdout:964: left oid 5 (ObjNum 137 snap 15 seq_num 137) 2024-03-19T11:30:40.466 INFO:tasks.rados.rados.0.smithi038.stdout:965: finishing write tid 1 to smithi03829856-17 2024-03-19T11:30:40.466 INFO:tasks.rados.rados.0.smithi038.stdout:962: done (8 left) 2024-03-19T11:30:40.466 INFO:tasks.rados.rados.0.smithi038.stdout:963: done (7 left) 2024-03-19T11:30:40.466 INFO:tasks.rados.rados.0.smithi038.stdout:964: done (6 left) 2024-03-19T11:30:40.467 INFO:tasks.rados.rados.0.smithi038.stdout:970: done (5 left) 2024-03-19T11:30:40.467 INFO:tasks.rados.rados.0.smithi038.stdout:971: read oid 30 snap -1 2024-03-19T11:30:40.467 INFO:tasks.rados.rados.0.smithi038.stdout:971: expect (ObjNum 29 snap 0 seq_num 29) 2024-03-19T11:30:40.467 INFO:tasks.rados.rados.0.smithi038.stdout:972: tier_flush oid 20 2024-03-19T11:30:40.467 INFO:tasks.rados.rados.0.smithi038.stdout:972: tier_flush oid 20 snap -1 2024-03-19T11:30:40.467 INFO:tasks.rados.rados.0.smithi038.stdout:967: got (22) Invalid argument 2024-03-19T11:30:40.467 INFO:tasks.rados.rados.0.smithi038.stdout:968: expect (ObjNum 23 snap 0 seq_num 23) 2024-03-19T11:30:40.467 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-19T11:30:40.468 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:40.476+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:40.471 INFO:tasks.rados.rados.0.smithi038.stdout:966: expect (ObjNum 12 snap 0 seq_num 12) 2024-03-19T11:30:40.521 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:30:40.522 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-03-19T11:30:40.538 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":84,"stamp":"2024-03-19T11:30:40.145434+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":358758112,"num_objects":223,"num_object_clones":60,"num_object_copies":446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":220,"num_whiteouts":6,"num_read":972,"num_read_kb":592051,"num_write":732,"num_write_kb":224423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":76616960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":75,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2201,"ondisk_log_size":2201,"up":98,"acting":98,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":642752,"kb_used_data":414096,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":754331968,"statfs":{"total":773094113280,"available":772435935232,"internally_reserved":0,"allocated":424034304,"data_stored":491622666,"data_compressed":11535360,"data_compressed_allocated":12312576,"data_compressed_original":27979776,"omap_allocated":12751,"internal_metadata":234016305},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21140096,"num_objects":19,"num_object_clones":13,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":227,"num_read_kb":224143,"num_write":56,"num_write_kb":16830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":5034309,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":8,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.025611"},"pg_stats":[{"pgid":"4.23","version":"41'19","reported_seq":208,"reported_epoch":59,"state":"unknown","last_fresh":"2024-03-19T11:30:36.137452+0000","last_change":"2024-03-19T11:30:37.171190+0000","last_active":"2024-03-19T11:30:37.171190+0000","last_peered":"2024-03-19T11:30:36.137452+0000","last_clean":"2024-03-19T11:30:36.137452+0000","last_became_active":"2024-03-19T11:30:21.867677+0000","last_became_peered":"2024-03-19T11:30:21.867677+0000","last_unstale":"2024-03-19T11:30:36.137452+0000","last_undegraded":"2024-03-19T11:30:36.137452+0000","last_fullsized":"2024-03-19T11:30:36.137452+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":6,"last_scrub":"41'19","last_scrub_stamp":"2024-03-19T11:30:28.180772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:28.180772+0000","objects_scrubbed":1,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1403368,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4077,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"},{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"f","length":"1"}]},{"pgid":"4.22","version":"28'6","reported_seq":107,"reported_epoch":62,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:30:39.174232+0000","last_change":"2024-03-19T11:30:38.097950+0000","last_active":"2024-03-19T11:30:39.174232+0000","last_peered":"2024-03-19T11:30:39.174232+0000","last_clean":"2024-03-19T11:30:39.174232+0000","last_became_active":"2024-03-19T11:30:38.057759+0000","last_became_peered":"2024-03-19T11:30:38.057759+0000","last_unstale":"2024-03-19T11:30:39.174232+0000","last_undegraded":"2024-03-19T11:30:39.174232+0000","last_fullsized":"2024-03-19T11:30:39.174232+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":6,"last_scrub":"28'6","last_scrub_stamp":"2024-03-19T11:30:30.357009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:30.357009+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:43.357886+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.037415715000000002,"stat_sum":{"num_bytes":1176474,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"}]},{"pgid":"4.21","version":"59'92","reported_seq":321,"reported_epoch":62,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:30:39.174376+0000","last_change":"2024-03-19T11:30:38.099340+0000","last_active":"2024-03-19T11:30:39.174376+0000","last_peered":"2024-03-19T11:30:39.174376+0000","last_clean":"2024-03-19T11:30:39.174376+0000","last_became_active":"2024-03-19T11:30:38.061758+0000","last_became_peered":"2024-03-19T11:30:38.061758+0000","last_unstale":"2024-03-19T11:30:39.174376+0000","last_undegraded":"2024-03-19T11:30:39.174376+0000","last_fullsized":"2024-03-19T11:30:39.174376+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":6,"last_scrub":"51'79","last_scrub_stamp":"2024-03-19T11:30:31.340443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:31.340443+0000","objects_scrubbed":7,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:43.357914+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.035587547999999997,"stat_sum":{"num_bytes":6871552,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":12620,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"}]},{"pgid":"4.20","version":"56'46","reported_seq":233,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-19T11:30:38.073540+0000","last_change":"2024-03-19T11:30:38.073475+0000","last_active":"2024-03-19T11:30:38.073540+0000","last_peered":"2024-03-19T11:30:38.073540+0000","last_clean":"2024-03-19T11:30:38.073540+0000","last_became_active":"2024-03-19T11:30:38.072125+0000","last_became_peered":"2024-03-19T11:30:38.072125+0000","last_unstale":"2024-03-19T11:30:38.073540+0000","last_undegraded":"2024-03-19T11:30:38.073540+0000","last_fullsized":"2024-03-19T11:30:38.073540+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":6,"last_scrub":"49'38","last_scrub_stamp":"2024-03-19T11:30:29.740635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:29.740635+0000","objects_scrubbed":3,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:43.744809+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00086159500000000005,"stat_sum":{"num_bytes":3638516,"num_objects":2,"num_object_clones":2,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.1f","version":"0'0","reported_seq":163,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-19T11:30:39.169648+0000","last_change":"2024-03-19T11:30:38.062802+0000","last_active":"2024-03-19T11:30:39.169648+0000","last_peered":"2024-03-19T11:30:39.169648+0000","last_clean":"2024-03-19T11:30:39.169648+0000","last_became_active":"2024-03-19T11:30:38.055108+0000","last_became_peered":"2024-03-19T11:30:38.055108+0000","last_unstale":"2024-03-19T11:30:39.169648+0000","last_undegraded":"2024-03-19T11:30:39.169648+0000","last_fullsized":"2024-03-19T11:30:39.169648+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:43.298533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0068887510000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.1e","version":"59'57","reported_seq":235,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-19T11:30:38.074926+0000","last_change":"2024-03-19T11:30:38.074463+0000","last_active":"2024-03-19T11:30:38.074926+0000","last_peered":"2024-03-19T11:30:38.074926+0000","last_clean":"2024-03-19T11:30:38.074926+0000","last_became_active":"2024-03-19T11:30:38.071044+0000","last_became_peered":"2024-03-19T11:30:38.071044+0000","last_unstale":"2024-03-19T11:30:38.074926+0000","last_undegraded":"2024-03-19T11:30:38.074926+0000","last_fullsized":"2024-03-19T11:30:38.074926+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024433739999999999,"stat_sum":{"num_bytes":7183762,"num_objects":4,"num_object_clones":2,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":22170,"num_write":17,"num_write_kb":4182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.1d","version":"61'59","reported_seq":248,"reported_epoch":62,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:30:39.160705+0000","last_change":"2024-03-19T11:30:38.053995+0000","last_active":"2024-03-19T11:30:39.160705+0000","last_peered":"2024-03-19T11:30:39.160705+0000","last_clean":"2024-03-19T11:30:39.160705+0000","last_became_active":"2024-03-19T11:30:38.052224+0000","last_became_peered":"2024-03-19T11:30:38.052224+0000","last_unstale":"2024-03-19T11:30:39.160705+0000","last_undegraded":"2024-03-19T11:30:39.160705+0000","last_fullsized":"2024-03-19T11:30:39.160705+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:43.570529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6170948,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":12189,"num_write":18,"num_write_kb":5154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2772451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"}]},{"pgid":"4.1c","version":"56'29","reported_seq":25,"reported_epoch":61,"state":"active+clean+laggy","last_fresh":"2024-03-19T11:30:38.569270+0000","last_change":"2024-03-19T11:30:38.569270+0000","last_active":"2024-03-19T11:30:38.569270+0000","last_peered":"2024-03-19T11:30:38.569270+0000","last_clean":"2024-03-19T11:30:38.569270+0000","last_became_active":"2024-03-19T11:30:38.070619+0000","last_became_peered":"2024-03-19T11:30:38.070619+0000","last_unstale":"2024-03-19T11:30:38.569270+0000","last_undegraded":"2024-03-19T11:30:38.569270+0000","last_fullsized":"2024-03-19T11:30:38.569270+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:43.744849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062618899999999998,"stat_sum":{"num_bytes":2253075,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":13818,"num_write":8,"num_write_kb":1836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":6637582,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.1b","version":"57'58","reported_seq":262,"reported_epoch":59,"state":"unknown","last_fresh":"2024-03-19T11:30:36.749357+0000","last_change":"2024-03-19T11:30:37.171136+0000","last_active":"2024-03-19T11:30:37.171136+0000","last_peered":"2024-03-19T11:30:36.749357+0000","last_clean":"2024-03-19T11:30:36.749357+0000","last_became_active":"2024-03-19T11:29:58.555569+0000","last_became_peered":"2024-03-19T11:29:58.555569+0000","last_unstale":"2024-03-19T11:30:36.749357+0000","last_undegraded":"2024-03-19T11:30:36.749357+0000","last_fullsized":"2024-03-19T11:30:36.749357+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7803777,"num_objects":4,"num_object_clones":2,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":74,"num_read_kb":73036,"num_write":19,"num_write_kb":5157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"},{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"f","length":"1"}]},{"pgid":"4.1a","version":"61'73","reported_seq":264,"reported_epoch":61,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:30:38.083101+0000","last_change":"2024-03-19T11:30:38.075599+0000","last_active":"2024-03-19T11:30:38.083101+0000","last_peered":"2024-03-19T11:30:38.083101+0000","last_clean":"2024-03-19T11:30:38.083101+0000","last_became_active":"2024-03-19T11:30:38.069375+0000","last_became_peered":"2024-03-19T11:30:38.069375+0000","last_unstale":"2024-03-19T11:30:38.083101+0000","last_undegraded":"2024-03-19T11:30:38.083101+0000","last_fullsized":"2024-03-19T11:30:38.083101+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002442091,"stat_sum":{"num_bytes":8225639,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":40,"num_read_kb":33654,"num_write":19,"num_write_kb":4860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"}]},{"pgid":"4.19","version":"61'63","reported_seq":305,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-19T11:30:38.057240+0000","last_change":"2024-03-19T11:30:38.058213+0000","last_active":"2024-03-19T11:30:38.057240+0000","last_peered":"2024-03-19T11:30:38.057240+0000","last_clean":"2024-03-19T11:30:38.057240+0000","last_became_active":"2024-03-19T11:30:30.051696+0000","last_became_peered":"2024-03-19T11:30:30.051696+0000","last_unstale":"2024-03-19T11:30:38.057240+0000","last_undegraded":"2024-03-19T11:30:38.057240+0000","last_fullsized":"2024-03-19T11:30:38.057240+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":5,"last_scrub":"41'37","last_scrub_stamp":"2024-03-19T11:30:27.705159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:27.705159+0000","objects_scrubbed":4,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:57.274624+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.011454681,"stat_sum":{"num_bytes":12043149,"num_objects":7,"num_object_clones":3,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":28,"num_read_kb":25595,"num_write":13,"num_write_kb":2452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":8331740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":7,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.18","version":"61'91","reported_seq":265,"reported_epoch":62,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:30:39.174109+0000","last_change":"2024-03-19T11:30:38.099498+0000","last_active":"2024-03-19T11:30:39.174109+0000","last_peered":"2024-03-19T11:30:39.174109+0000","last_clean":"2024-03-19T11:30:39.174109+0000","last_became_active":"2024-03-19T11:30:21.869150+0000","last_became_peered":"2024-03-19T11:30:21.869150+0000","last_unstale":"2024-03-19T11:30:39.174109+0000","last_undegraded":"2024-03-19T11:30:39.174109+0000","last_fullsized":"2024-03-19T11:30:39.174109+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":5,"last_scrub":"48'77","last_scrub_stamp":"2024-03-19T11:30:29.399762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:29.399762+0000","objects_scrubbed":6,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:57.039817+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.049511285000000002,"stat_sum":{"num_bytes":15590414,"num_objects":7,"num_object_clones":5,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"}]},{"pgid":"4.17","version":"41'24","reported_seq":159,"reported_epoch":62,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:30:39.174197+0000","last_change":"2024-03-19T11:30:38.095714+0000","last_active":"2024-03-19T11:30:39.174197+0000","last_peered":"2024-03-19T11:30:39.174197+0000","last_clean":"2024-03-19T11:30:39.174197+0000","last_became_active":"2024-03-19T11:30:28.039784+0000","last_became_peered":"2024-03-19T11:30:28.039784+0000","last_unstale":"2024-03-19T11:30:39.174197+0000","last_undegraded":"2024-03-19T11:30:39.174197+0000","last_fullsized":"2024-03-19T11:30:39.174197+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:43.357847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.054447817000000003,"stat_sum":{"num_bytes":3679215,"num_objects":2,"num_object_clones":2,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":9,"num_write_kb":2643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"1"}]},{"pgid":"4.16","version":"61'70","reported_seq":310,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-19T11:30:39.169575+0000","last_change":"2024-03-19T11:30:38.051924+0000","last_active":"2024-03-19T11:30:39.169575+0000","last_peered":"2024-03-19T11:30:39.169575+0000","last_clean":"2024-03-19T11:30:39.169575+0000","last_became_active":"2024-03-19T11:30:26.921950+0000","last_became_peered":"2024-03-19T11:30:26.921950+0000","last_unstale":"2024-03-19T11:30:39.169575+0000","last_undegraded":"2024-03-19T11:30:39.169575+0000","last_fullsized":"2024-03-19T11:30:39.169575+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":5,"last_scrub":"49'56","last_scrub_stamp":"2024-03-19T11:30:29.212766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:29.212766+0000","objects_scrubbed":5,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:39.755433+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0096679469999999997,"stat_sum":{"num_bytes":12611450,"num_objects":7,"num_object_clones":4,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":39792,"num_write":6,"num_write_kb":981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":7,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.15","version":"61'88","reported_seq":360,"reported_epoch":61,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:30:38.088150+0000","last_change":"2024-03-19T11:30:38.088150+0000","last_active":"2024-03-19T11:30:38.088150+0000","last_peered":"2024-03-19T11:30:38.088150+0000","last_clean":"2024-03-19T11:30:38.088150+0000","last_became_active":"2024-03-19T11:30:28.043875+0000","last_became_peered":"2024-03-19T11:30:28.043875+0000","last_unstale":"2024-03-19T11:30:38.088150+0000","last_undegraded":"2024-03-19T11:30:38.088150+0000","last_fullsized":"2024-03-19T11:30:38.088150+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":5,"last_scrub":"49'74","last_scrub_stamp":"2024-03-19T11:30:29.457542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:29.457542+0000","objects_scrubbed":7,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:36.250846+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.047621296,"stat_sum":{"num_bytes":18257461,"num_objects":8,"num_object_clones":3,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":17,"num_read_kb":11796,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":6,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"}]},{"pgid":"4.14","version":"40'56","reported_seq":220,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-19T11:30:39.160341+0000","last_change":"2024-03-19T11:30:38.043252+0000","last_active":"2024-03-19T11:30:39.160341+0000","last_peered":"2024-03-19T11:30:39.160341+0000","last_clean":"2024-03-19T11:30:39.160341+0000","last_became_active":"2024-03-19T11:30:28.038075+0000","last_became_peered":"2024-03-19T11:30:28.038075+0000","last_unstale":"2024-03-19T11:30:39.160341+0000","last_undegraded":"2024-03-19T11:30:39.160341+0000","last_fullsized":"2024-03-19T11:30:39.160341+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":5,"last_scrub":"40'56","last_scrub_stamp":"2024-03-19T11:30:28.331178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:28.331178+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:48.322911+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0020917280000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.13","version":"48'22","reported_seq":218,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-03-19T11:30:37.151700+0000","last_change":"2024-03-19T11:30:36.136917+0000","last_active":"2024-03-19T11:30:37.151700+0000","last_peered":"2024-03-19T11:30:37.151700+0000","last_clean":"2024-03-19T11:30:37.151700+0000","last_became_active":"2024-03-19T11:30:21.867725+0000","last_became_peered":"2024-03-19T11:30:21.867725+0000","last_unstale":"2024-03-19T11:30:37.151700+0000","last_undegraded":"2024-03-19T11:30:37.151700+0000","last_fullsized":"2024-03-19T11:30:37.151700+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":5,"last_scrub":"48'22","last_scrub_stamp":"2024-03-19T11:30:27.183294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:27.183294+0000","objects_scrubbed":2,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:27.741271+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0016421859999999999,"stat_sum":{"num_bytes":5361367,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":12018,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"},{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"f","length":"1"}]},{"pgid":"4.12","version":"28'6","reported_seq":22,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-19T11:30:38.068779+0000","last_change":"2024-03-19T11:30:38.063517+0000","last_active":"2024-03-19T11:30:38.068779+0000","last_peered":"2024-03-19T11:30:38.068779+0000","last_clean":"2024-03-19T11:30:38.068779+0000","last_became_active":"2024-03-19T11:30:38.052481+0000","last_became_peered":"2024-03-19T11:30:38.052481+0000","last_unstale":"2024-03-19T11:30:38.068779+0000","last_undegraded":"2024-03-19T11:30:38.068779+0000","last_fullsized":"2024-03-19T11:30:38.068779+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":5,"last_scrub":"28'6","last_scrub_stamp":"2024-03-19T11:30:28.409040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:28.409040+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0097424030000000002,"stat_sum":{"num_bytes":1176474,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.11","version":"56'62","reported_seq":75,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-19T11:30:38.041827+0000","last_change":"2024-03-19T11:30:38.042327+0000","last_active":"2024-03-19T11:30:38.041827+0000","last_peered":"2024-03-19T11:30:38.041827+0000","last_clean":"2024-03-19T11:30:38.041827+0000","last_became_active":"2024-03-19T11:30:30.052184+0000","last_became_peered":"2024-03-19T11:30:30.052184+0000","last_unstale":"2024-03-19T11:30:38.041827+0000","last_undegraded":"2024-03-19T11:30:38.041827+0000","last_fullsized":"2024-03-19T11:30:38.041827+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":5,"last_scrub":"47'60","last_scrub_stamp":"2024-03-19T11:30:30.780348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:30.780348+0000","objects_scrubbed":2,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:59.091223+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00074934399999999999,"stat_sum":{"num_bytes":4792245,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":6829,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.10","version":"55'32","reported_seq":238,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-19T11:30:38.043048+0000","last_change":"2024-03-19T11:30:38.043516+0000","last_active":"2024-03-19T11:30:38.043048+0000","last_peered":"2024-03-19T11:30:38.043048+0000","last_clean":"2024-03-19T11:30:38.043048+0000","last_became_active":"2024-03-19T11:30:30.061489+0000","last_became_peered":"2024-03-19T11:30:30.061489+0000","last_unstale":"2024-03-19T11:30:38.043048+0000","last_undegraded":"2024-03-19T11:30:38.043048+0000","last_fullsized":"2024-03-19T11:30:38.043048+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":5,"last_scrub":"50'30","last_scrub_stamp":"2024-03-19T11:30:28.787058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:28.787058+0000","objects_scrubbed":2,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:49.137384+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00071659299999999996,"stat_sum":{"num_bytes":6369787,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":3551298,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"2.7","version":"0'0","reported_seq":82,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-19T11:30:39.160687+0000","last_change":"2024-03-19T11:30:03.276845+0000","last_active":"2024-03-19T11:30:39.160687+0000","last_peered":"2024-03-19T11:30:39.160687+0000","last_clean":"2024-03-19T11:30:39.160687+0000","last_became_active":"2024-03-19T11:30:03.276220+0000","last_became_peered":"2024-03-19T11:30:03.276220+0000","last_unstale":"2024-03-19T11:30:39.160687+0000","last_undegraded":"2024-03-19T11:30:39.160687+0000","last_fullsized":"2024-03-19T11:30:39.160687+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:01.829767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"59'92","reported_seq":317,"reported_epoch":62,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:30:39.174257+0000","last_change":"2024-03-19T11:30:38.098541+0000","last_active":"2024-03-19T11:30:39.174257+0000","last_peered":"2024-03-19T11:30:39.174257+0000","last_clean":"2024-03-19T11:30:39.174257+0000","last_became_active":"2024-03-19T11:30:38.059069+0000","last_became_peered":"2024-03-19T11:30:38.059069+0000","last_unstale":"2024-03-19T11:30:39.174257+0000","last_undegraded":"2024-03-19T11:30:39.174257+0000","last_fullsized":"2024-03-19T11:30:39.174257+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'79","last_scrub_stamp":"2024-03-19T11:30:31.340443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:31.340443+0000","objects_scrubbed":7,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:43.357898+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.035592660999999998,"stat_sum":{"num_bytes":6871551,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":31,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-19T11:30:38.040021+0000","last_change":"2024-03-19T11:30:28.039357+0000","last_active":"2024-03-19T11:30:38.040021+0000","last_peered":"2024-03-19T11:30:38.040021+0000","last_clean":"2024-03-19T11:30:38.040021+0000","last_became_active":"2024-03-19T11:30:28.038987+0000","last_became_peered":"2024-03-19T11:30:28.038987+0000","last_unstale":"2024-03-19T11:30:38.040021+0000","last_undegraded":"2024-03-19T11:30:38.040021+0000","last_fullsized":"2024-03-19T11:30:38.040021+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:47.430206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"61'52","reported_seq":242,"reported_epoch":61,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:30:39.151757+0000","last_change":"2024-03-19T11:30:38.070435+0000","last_active":"2024-03-19T11:30:39.151757+0000","last_peered":"2024-03-19T11:30:39.151757+0000","last_clean":"2024-03-19T11:30:39.151757+0000","last_became_active":"2024-03-19T11:30:38.065693+0000","last_became_peered":"2024-03-19T11:30:38.065693+0000","last_unstale":"2024-03-19T11:30:39.151757+0000","last_undegraded":"2024-03-19T11:30:39.151757+0000","last_fullsized":"2024-03-19T11:30:39.151757+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"49'38","last_scrub_stamp":"2024-03-19T11:30:29.740635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:29.740635+0000","objects_scrubbed":3,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:43.744775+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00043892099999999999,"stat_sum":{"num_bytes":4459284,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":12,"num_read_kb":10087,"num_write":6,"num_write_kb":1307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"f","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":107,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-19T11:30:38.054338+0000","last_change":"2024-03-19T11:30:30.057337+0000","last_active":"2024-03-19T11:30:38.054338+0000","last_peered":"2024-03-19T11:30:38.054338+0000","last_clean":"2024-03-19T11:30:38.054338+0000","last_became_active":"2024-03-19T11:30:30.056521+0000","last_became_peered":"2024-03-19T11:30:30.056521+0000","last_unstale":"2024-03-19T11:30:38.054338+0000","last_undegraded":"2024-03-19T11:30:38.054338+0000","last_fullsized":"2024-03-19T11:30:38.054338+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:00.627926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"41'19","reported_seq":208,"reported_epoch":59,"state":"peering","last_fresh":"2024-03-19T11:30:36.137452+0000","last_change":"2024-03-19T11:30:37.152820+0000","last_active":"2024-03-19T11:30:37.152820+0000","last_peered":"2024-03-19T11:30:36.137452+0000","last_clean":"2024-03-19T11:30:36.137452+0000","last_became_active":"2024-03-19T11:30:21.867677+0000","last_became_peered":"2024-03-19T11:30:21.867677+0000","last_unstale":"2024-03-19T11:30:36.137452+0000","last_undegraded":"2024-03-19T11:30:36.137452+0000","last_fullsized":"2024-03-19T11:30:36.137452+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'19","last_scrub_stamp":"2024-03-19T11:30:28.180772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:28.180772+0000","objects_scrubbed":1,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001683646,"stat_sum":{"num_bytes":1403368,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4077,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"},{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"f","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":99,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-19T11:30:39.169548+0000","last_change":"2024-03-19T11:29:46.541233+0000","last_active":"2024-03-19T11:30:39.169548+0000","last_peered":"2024-03-19T11:30:39.169548+0000","last_clean":"2024-03-19T11:30:39.169548+0000","last_became_active":"2024-03-19T11:29:46.540939+0000","last_became_peered":"2024-03-19T11:29:46.540939+0000","last_unstale":"2024-03-19T11:30:39.169548+0000","last_undegraded":"2024-03-19T11:30:39.169548+0000","last_fullsized":"2024-03-19T11:30:39.169548+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:54.761098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"28'6","reported_seq":108,"reported_epoch":62,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:30:39.174311+0000","last_change":"2024-03-19T11:30:38.097008+0000","last_active":"2024-03-19T11:30:39.174311+0000","last_peered":"2024-03-19T11:30:39.174311+0000","last_clean":"2024-03-19T11:30:39.174311+0000","last_became_active":"2024-03-19T11:30:38.052772+0000","last_became_peered":"2024-03-19T11:30:38.052772+0000","last_unstale":"2024-03-19T11:30:39.174311+0000","last_undegraded":"2024-03-19T11:30:39.174311+0000","last_fullsized":"2024-03-19T11:30:39.174311+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'6","last_scrub_stamp":"2024-03-19T11:30:30.357009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:30.357009+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:43.357873+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.043032320999999998,"stat_sum":{"num_bytes":1176473,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"}]},{"pgid":"3.3","version":"58'82","reported_seq":257,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-03-19T11:30:37.151898+0000","last_change":"2024-03-19T11:30:30.271391+0000","last_active":"2024-03-19T11:30:37.151898+0000","last_peered":"2024-03-19T11:30:37.151898+0000","last_clean":"2024-03-19T11:30:37.151898+0000","last_became_active":"2024-03-19T11:30:30.058963+0000","last_became_peered":"2024-03-19T11:30:30.058963+0000","last_unstale":"2024-03-19T11:30:37.151898+0000","last_undegraded":"2024-03-19T11:30:37.151898+0000","last_fullsized":"2024-03-19T11:30:37.151898+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:11.045052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41204112,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":60,"num_read_kb":1486,"num_write":90,"num_write_kb":46521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":22537669,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":98,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-19T11:30:39.174407+0000","last_change":"2024-03-19T11:29:46.321255+0000","last_active":"2024-03-19T11:30:39.174407+0000","last_peered":"2024-03-19T11:30:39.174407+0000","last_clean":"2024-03-19T11:30:39.174407+0000","last_became_active":"2024-03-19T11:29:46.320679+0000","last_became_peered":"2024-03-19T11:29:46.320679+0000","last_unstale":"2024-03-19T11:30:39.174407+0000","last_undegraded":"2024-03-19T11:30:39.174407+0000","last_fullsized":"2024-03-19T11:30:39.174407+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:14.793563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"61'93","reported_seq":356,"reported_epoch":62,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:30:39.160613+0000","last_change":"2024-03-19T11:30:38.090450+0000","last_active":"2024-03-19T11:30:39.160613+0000","last_peered":"2024-03-19T11:30:39.160613+0000","last_clean":"2024-03-19T11:30:39.160613+0000","last_became_active":"2024-03-19T11:30:28.037565+0000","last_became_peered":"2024-03-19T11:30:28.037565+0000","last_unstale":"2024-03-19T11:30:39.160613+0000","last_undegraded":"2024-03-19T11:30:39.160613+0000","last_fullsized":"2024-03-19T11:30:39.160613+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'76","last_scrub_stamp":"2024-03-19T11:30:29.353549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:29.353549+0000","objects_scrubbed":11,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:44.709737+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.048805171000000001,"stat_sum":{"num_bytes":21399668,"num_objects":10,"num_object_clones":5,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":69,"num_read_kb":86092,"num_write":5,"num_write_kb":1660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":9,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"}]},{"pgid":"3.0","version":"49'84","reported_seq":183,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-19T11:30:39.169040+0000","last_change":"2024-03-19T11:29:57.550320+0000","last_active":"2024-03-19T11:30:39.169040+0000","last_peered":"2024-03-19T11:30:39.169040+0000","last_clean":"2024-03-19T11:30:39.169040+0000","last_became_active":"2024-03-19T11:29:57.549928+0000","last_became_peered":"2024-03-19T11:29:57.549928+0000","last_unstale":"2024-03-19T11:30:39.169040+0000","last_undegraded":"2024-03-19T11:30:39.169040+0000","last_fullsized":"2024-03-19T11:30:39.169040+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:01.987302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29288645,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":68,"num_read_kb":2311,"num_write":89,"num_write_kb":38148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":100,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-19T11:30:39.160286+0000","last_change":"2024-03-19T11:29:46.319811+0000","last_active":"2024-03-19T11:30:39.160286+0000","last_peered":"2024-03-19T11:30:39.160286+0000","last_clean":"2024-03-19T11:30:39.160286+0000","last_became_active":"2024-03-19T11:29:46.319334+0000","last_became_peered":"2024-03-19T11:29:46.319334+0000","last_unstale":"2024-03-19T11:30:39.160286+0000","last_undegraded":"2024-03-19T11:30:39.160286+0000","last_fullsized":"2024-03-19T11:30:39.160286+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:04.320463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"61'40","reported_seq":227,"reported_epoch":62,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:30:39.174256+0000","last_change":"2024-03-19T11:30:39.164729+0000","last_active":"2024-03-19T11:30:39.174256+0000","last_peered":"2024-03-19T11:30:39.174256+0000","last_clean":"2024-03-19T11:30:39.174256+0000","last_became_active":"2024-03-19T11:30:28.042372+0000","last_became_peered":"2024-03-19T11:30:28.042372+0000","last_unstale":"2024-03-19T11:30:39.174256+0000","last_undegraded":"2024-03-19T11:30:39.174256+0000","last_fullsized":"2024-03-19T11:30:39.174256+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:43.357861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.052246765000000001,"stat_sum":{"num_bytes":2436324,"num_objects":1,"num_object_clones":1,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":13,"num_write_kb":3591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":7273946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"}]},{"pgid":"3.1","version":"58'70","reported_seq":224,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-03-19T11:30:37.147991+0000","last_change":"2024-03-19T11:30:27.033184+0000","last_active":"2024-03-19T11:30:37.147991+0000","last_peered":"2024-03-19T11:30:37.147991+0000","last_clean":"2024-03-19T11:30:37.147991+0000","last_became_active":"2024-03-19T11:30:26.927069+0000","last_became_peered":"2024-03-19T11:30:26.927069+0000","last_unstale":"2024-03-19T11:30:37.147991+0000","last_undegraded":"2024-03-19T11:30:37.147991+0000","last_fullsized":"2024-03-19T11:30:37.147991+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:00.211540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31896321,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":61,"num_read_kb":4830,"num_write":74,"num_write_kb":38708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":16477750,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":102,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-19T11:30:38.054841+0000","last_change":"2024-03-19T11:30:27.926306+0000","last_active":"2024-03-19T11:30:38.054841+0000","last_peered":"2024-03-19T11:30:38.054841+0000","last_clean":"2024-03-19T11:30:38.054841+0000","last_became_active":"2024-03-19T11:30:27.925990+0000","last_became_peered":"2024-03-19T11:30:27.925990+0000","last_unstale":"2024-03-19T11:30:38.054841+0000","last_undegraded":"2024-03-19T11:30:38.054841+0000","last_fullsized":"2024-03-19T11:30:38.054841+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:00.759938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"61'63","reported_seq":269,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-19T11:30:39.169575+0000","last_change":"2024-03-19T11:30:38.053087+0000","last_active":"2024-03-19T11:30:39.169575+0000","last_peered":"2024-03-19T11:30:39.169575+0000","last_clean":"2024-03-19T11:30:39.169575+0000","last_became_active":"2024-03-19T11:30:26.925686+0000","last_became_peered":"2024-03-19T11:30:26.925686+0000","last_unstale":"2024-03-19T11:30:39.169575+0000","last_undegraded":"2024-03-19T11:30:39.169575+0000","last_fullsized":"2024-03-19T11:30:39.169575+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"49'53","last_scrub_stamp":"2024-03-19T11:30:28.258196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:28.258196+0000","objects_scrubbed":3,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:40.593261+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.010461453000000001,"stat_sum":{"num_bytes":8046855,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":10,"num_read_kb":10785,"num_write":6,"num_write_kb":1517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"3.2","version":"61'70","reported_seq":170,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-19T11:30:38.552642+0000","last_change":"2024-03-19T11:29:57.549277+0000","last_active":"2024-03-19T11:30:38.552642+0000","last_peered":"2024-03-19T11:30:38.552642+0000","last_clean":"2024-03-19T11:30:38.552642+0000","last_became_active":"2024-03-19T11:29:57.548946+0000","last_became_peered":"2024-03-19T11:29:57.548946+0000","last_unstale":"2024-03-19T11:30:38.552642+0000","last_undegraded":"2024-03-19T11:30:38.552642+0000","last_fullsized":"2024-03-19T11:30:38.552642+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:08.380405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35228564,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":60,"num_read_kb":5517,"num_write":80,"num_write_kb":37058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":98,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-19T11:30:39.174578+0000","last_change":"2024-03-19T11:29:46.322491+0000","last_active":"2024-03-19T11:30:39.174578+0000","last_peered":"2024-03-19T11:30:39.174578+0000","last_clean":"2024-03-19T11:30:39.174578+0000","last_became_active":"2024-03-19T11:29:46.321760+0000","last_became_peered":"2024-03-19T11:29:46.321760+0000","last_unstale":"2024-03-19T11:30:39.174578+0000","last_undegraded":"2024-03-19T11:30:39.174578+0000","last_fullsized":"2024-03-19T11:30:39.174578+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:04.561915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":262,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-19T11:30:38.055095+0000","last_change":"2024-03-19T11:30:28.056134+0000","last_active":"2024-03-19T11:30:38.055095+0000","last_peered":"2024-03-19T11:30:38.055095+0000","last_clean":"2024-03-19T11:30:38.055095+0000","last_became_active":"2024-03-19T11:30:28.043738+0000","last_became_peered":"2024-03-19T11:30:28.043738+0000","last_unstale":"2024-03-19T11:30:38.055095+0000","last_undegraded":"2024-03-19T11:30:38.055095+0000","last_fullsized":"2024-03-19T11:30:38.055095+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:43.166618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:43.166618+0000","last_clean_scrub_stamp":"2024-03-19T11:29:43.166618+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:46.409941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"56'66","reported_seq":323,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-19T11:30:38.042462+0000","last_change":"2024-03-19T11:30:38.044558+0000","last_active":"2024-03-19T11:30:38.042462+0000","last_peered":"2024-03-19T11:30:38.042462+0000","last_clean":"2024-03-19T11:30:38.042462+0000","last_became_active":"2024-03-19T11:30:28.041852+0000","last_became_peered":"2024-03-19T11:30:28.041852+0000","last_unstale":"2024-03-19T11:30:38.042462+0000","last_undegraded":"2024-03-19T11:30:38.042462+0000","last_fullsized":"2024-03-19T11:30:38.042462+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'64","last_scrub_stamp":"2024-03-19T11:30:28.487136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:28.487136+0000","objects_scrubbed":3,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:54.352531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029151099999999998,"stat_sum":{"num_bytes":4806736,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":10609,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.8","version":"61'68","reported_seq":251,"reported_epoch":62,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:30:39.174115+0000","last_change":"2024-03-19T11:30:39.164699+0000","last_active":"2024-03-19T11:30:39.174115+0000","last_peered":"2024-03-19T11:30:39.174115+0000","last_clean":"2024-03-19T11:30:39.174115+0000","last_became_active":"2024-03-19T11:30:21.859467+0000","last_became_peered":"2024-03-19T11:30:21.859467+0000","last_unstale":"2024-03-19T11:30:39.174115+0000","last_undegraded":"2024-03-19T11:30:39.174115+0000","last_fullsized":"2024-03-19T11:30:39.174115+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'58","last_scrub_stamp":"2024-03-19T11:30:27.422179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:27.422179+0000","objects_scrubbed":3,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:42.220980+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":1.119981178,"stat_sum":{"num_bytes":8023945,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":27,"num_read_kb":23808,"num_write":5,"num_write_kb":1048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"}]},{"pgid":"4.9","version":"56'51","reported_seq":283,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-19T11:30:38.049031+0000","last_change":"2024-03-19T11:30:38.055222+0000","last_active":"2024-03-19T11:30:38.049031+0000","last_peered":"2024-03-19T11:30:38.049031+0000","last_clean":"2024-03-19T11:30:38.049031+0000","last_became_active":"2024-03-19T11:30:30.050283+0000","last_became_peered":"2024-03-19T11:30:30.050283+0000","last_unstale":"2024-03-19T11:30:38.049031+0000","last_undegraded":"2024-03-19T11:30:38.049031+0000","last_fullsized":"2024-03-19T11:30:38.049031+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'47","last_scrub_stamp":"2024-03-19T11:30:28.676144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:28.676144+0000","objects_scrubbed":2,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:51.379073+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0072186050000000003,"stat_sum":{"num_bytes":4517558,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":9882,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4129874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.a","version":"58'70","reported_seq":267,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-19T11:30:38.066449+0000","last_change":"2024-03-19T11:30:38.066449+0000","last_active":"2024-03-19T11:30:38.066449+0000","last_peered":"2024-03-19T11:30:38.066449+0000","last_clean":"2024-03-19T11:30:38.066449+0000","last_became_active":"2024-03-19T11:30:38.053085+0000","last_became_peered":"2024-03-19T11:30:38.053085+0000","last_unstale":"2024-03-19T11:30:38.066449+0000","last_undegraded":"2024-03-19T11:30:38.066449+0000","last_fullsized":"2024-03-19T11:30:38.066449+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011735272,"stat_sum":{"num_bytes":8225638,"num_objects":4,"num_object_clones":2,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":40,"num_read_kb":33653,"num_write":18,"num_write_kb":4860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.b","version":"57'58","reported_seq":263,"reported_epoch":60,"state":"peering","last_fresh":"2024-03-19T11:30:37.149947+0000","last_change":"2024-03-19T11:30:37.149947+0000","last_active":"2024-03-19T11:30:37.149947+0000","last_peered":"2024-03-19T11:30:36.749357+0000","last_clean":"2024-03-19T11:30:36.749357+0000","last_became_active":"2024-03-19T11:29:58.555569+0000","last_became_peered":"2024-03-19T11:29:58.555569+0000","last_unstale":"2024-03-19T11:30:37.149947+0000","last_undegraded":"2024-03-19T11:30:37.149947+0000","last_fullsized":"2024-03-19T11:30:37.149947+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.61376923999999999,"stat_sum":{"num_bytes":7803777,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":73,"num_read_kb":73036,"num_write":18,"num_write_kb":5157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"},{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"f","length":"1"}]},{"pgid":"4.c","version":"56'29","reported_seq":212,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-19T11:30:38.064362+0000","last_change":"2024-03-19T11:30:38.064362+0000","last_active":"2024-03-19T11:30:38.064362+0000","last_peered":"2024-03-19T11:30:38.064362+0000","last_clean":"2024-03-19T11:30:38.064362+0000","last_became_active":"2024-03-19T11:30:38.060919+0000","last_became_peered":"2024-03-19T11:30:38.060919+0000","last_unstale":"2024-03-19T11:30:38.064362+0000","last_undegraded":"2024-03-19T11:30:38.064362+0000","last_fullsized":"2024-03-19T11:30:38.064362+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002377026,"stat_sum":{"num_bytes":2253074,"num_objects":1,"num_object_clones":1,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":13817,"num_write":8,"num_write_kb":1836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1603272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.d","version":"62'61","reported_seq":256,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-19T11:30:39.174101+0000","last_change":"2024-03-19T11:30:38.053532+0000","last_active":"2024-03-19T11:30:39.174101+0000","last_peered":"2024-03-19T11:30:39.174101+0000","last_clean":"2024-03-19T11:30:39.174101+0000","last_became_active":"2024-03-19T11:30:38.049196+0000","last_became_peered":"2024-03-19T11:30:38.049196+0000","last_unstale":"2024-03-19T11:30:39.174101+0000","last_undegraded":"2024-03-19T11:30:39.174101+0000","last_fullsized":"2024-03-19T11:30:39.174101+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:43.570510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003859543,"stat_sum":{"num_bytes":7333416,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":12190,"num_write":22,"num_write_kb":6158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2772450,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.e","version":"59'57","reported_seq":235,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-19T11:30:38.063344+0000","last_change":"2024-03-19T11:30:38.063344+0000","last_active":"2024-03-19T11:30:38.063344+0000","last_peered":"2024-03-19T11:30:38.063344+0000","last_clean":"2024-03-19T11:30:38.063344+0000","last_became_active":"2024-03-19T11:30:38.060487+0000","last_became_peered":"2024-03-19T11:30:38.060487+0000","last_unstale":"2024-03-19T11:30:38.063344+0000","last_undegraded":"2024-03-19T11:30:38.063344+0000","last_fullsized":"2024-03-19T11:30:38.063344+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022237839999999999,"stat_sum":{"num_bytes":7183762,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":22169,"num_write":16,"num_write_kb":4182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.f","version":"0'0","reported_seq":163,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-03-19T11:30:39.169628+0000","last_change":"2024-03-19T11:30:38.052508+0000","last_active":"2024-03-19T11:30:39.169628+0000","last_peered":"2024-03-19T11:30:39.169628+0000","last_clean":"2024-03-19T11:30:39.169628+0000","last_became_active":"2024-03-19T11:30:38.050919+0000","last_became_peered":"2024-03-19T11:30:38.050919+0000","last_unstale":"2024-03-19T11:30:39.169628+0000","last_undegraded":"2024-03-19T11:30:39.169628+0000","last_fullsized":"2024-03-19T11:30:39.169628+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:43.298500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001232969,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]}],"pool_stats":[{"poolid":4,"num_pg":36,"stat_sum":{"num_bytes":220550102,"num_objects":110,"num_object_clones":60,"num_object_copies":220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":107,"num_whiteouts":6,"num_read":607,"num_read_kb":577807,"num_write":240,"num_write_kb":61572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":37072613,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":75,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":283467776,"data_stored":342006681,"data_compressed":1262850,"data_compressed_allocated":1150976,"data_compressed_original":3276800,"omap_allocated":0,"internal_metadata":0},"log_size":1778,"ondisk_log_size":1778,"up":72,"acting":72,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":137617642,"num_objects":111,"num_object_clones":0,"num_object_copies":222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":111,"num_whiteouts":0,"num_read":249,"num_read_kb":14144,"num_write":333,"num_write_kb":160435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":39015419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":158625792,"data_stored":172679629,"data_compressed":12006032,"data_compressed_allocated":12890112,"data_compressed_original":27258880,"omap_allocated":0,"internal_metadata":0},"log_size":306,"ondisk_log_size":306,"up":8,"acting":8,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1586784,"data_compressed":19012,"data_compressed_allocated":98304,"data_compressed_original":1548288,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542159,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73640,"kb_used_data":42448,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94298200,"statfs":{"total":96636764160,"available":96561356800,"internally_reserved":0,"allocated":43466752,"data_stored":52253749,"data_compressed":117244,"data_compressed_allocated":151552,"data_compressed_original":905216,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542159,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78708,"kb_used_data":51688,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94293132,"statfs":{"total":96636764160,"available":96556167168,"internally_reserved":0,"allocated":52928512,"data_stored":65652894,"data_compressed":1433716,"data_compressed_allocated":1531904,"data_compressed_original":3162112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542159,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":104988,"kb_used_data":77968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94266852,"statfs":{"total":96636764160,"available":96529256448,"internally_reserved":0,"allocated":79839232,"data_stored":89237467,"data_compressed":3051499,"data_compressed_allocated":3264512,"data_compressed_original":7094272,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542158,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75948,"kb_used_data":48928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94295892,"statfs":{"total":96636764160,"available":96558993408,"internally_reserved":0,"allocated":50102272,"data_stored":55431013,"data_compressed":1261851,"data_compressed_allocated":1359872,"data_compressed_original":3080192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542158,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81504,"kb_used_data":50320,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94290336,"statfs":{"total":96636764160,"available":96553304064,"internally_reserved":0,"allocated":51527680,"data_stored":57833723,"data_compressed":1046253,"data_compressed_allocated":1130496,"data_compressed_original":2621440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574862,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":98192,"kb_used_data":71176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94273648,"statfs":{"total":96636764160,"available":96536215552,"internally_reserved":0,"allocated":72884224,"data_stored":85804518,"data_compressed":1599621,"data_compressed_allocated":1748992,"data_compressed_original":4182016,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574862,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50568,"kb_used_data":19384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94321272,"statfs":{"total":96636764160,"available":96584982528,"internally_reserved":0,"allocated":19849216,"data_stored":28563715,"data_compressed":224080,"data_compressed_allocated":118784,"data_compressed_original":524288,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574862,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79204,"kb_used_data":52184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94292636,"statfs":{"total":96636764160,"available":96555659264,"internally_reserved":0,"allocated":53436416,"data_stored":56845587,"data_compressed":2801096,"data_compressed_allocated":3006464,"data_compressed_original":6410240,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6064,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6884,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6064,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":38469632,"data_stored":41791483,"data_compressed":2801096,"data_compressed_allocated":3006464,"data_compressed_original":6410240,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14405632,"data_stored":15728702,"data_compressed":1002508,"data_compressed_allocated":1081344,"data_compressed_original":2424832,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14405632,"data_stored":15728702,"data_compressed":1002508,"data_compressed_allocated":1081344,"data_compressed_original":2424832,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":33669120,"data_stored":36965633,"data_compressed":2698727,"data_compressed_allocated":2887680,"data_compressed_original":6242304,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":19230720,"data_stored":20747205,"data_compressed":1455770,"data_compressed_allocated":1568768,"data_compressed_original":3133440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16822272,"data_stored":18446027,"data_compressed":1471527,"data_compressed_allocated":1572864,"data_compressed_original":3227648,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":21622784,"data_stored":23271877,"data_compressed":1573896,"data_compressed_allocated":1691648,"data_compressed_original":3395584,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":14319616,"data_stored":14731212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":38567936,"data_stored":45746757,"data_compressed":43745,"data_compressed_allocated":49152,"data_compressed_original":196608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36102144,"data_stored":42308125,"data_compressed":259343,"data_compressed_allocated":278528,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":46034944,"data_stored":52467315,"data_compressed":439678,"data_compressed_allocated":466944,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16371712,"data_stored":23203016,"data_compressed":224080,"data_compressed_allocated":118784,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":37634048,"data_stored":49249769,"data_compressed":129161,"data_compressed_allocated":94208,"data_compressed_original":196608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":50561024,"data_stored":61674125,"data_compressed":18841,"data_compressed_allocated":24576,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":43876352,"data_stored":52626362,"data_compressed":148002,"data_compressed_allocated":118784,"data_compressed_original":458752,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T11:30:40.541 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2024-03-19T11:30:40.542 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 46 2024-03-19T11:30:40.543 INFO:tasks.rados.rados.0.smithi038.stdout:965: finishing write tid 2 to smithi03829856-17 2024-03-19T11:30:40.543 INFO:tasks.rados.rados.0.smithi038.stdout:965: finishing write tid 3 to smithi03829856-17 2024-03-19T11:30:40.543 INFO:tasks.rados.rados.0.smithi038.stdout:965: finishing write tid 4 to smithi03829856-17 2024-03-19T11:30:40.543 INFO:tasks.rados.rados.0.smithi038.stdout:965: oid 17 updating version 0 to 91 2024-03-19T11:30:40.543 INFO:tasks.rados.rados.0.smithi038.stdout:965: oid 17 updating version 91 to 92 2024-03-19T11:30:40.543 INFO:tasks.rados.rados.0.smithi038.stdout:965: oid 17 version 92 is already newer than 90 2024-03-19T11:30:40.543 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 17 v 92 (ObjNum 138 snap 15 seq_num 138) dirty exists 2024-03-19T11:30:40.543 INFO:tasks.rados.rados.0.smithi038.stdout:965: left oid 17 (ObjNum 138 snap 15 seq_num 138) 2024-03-19T11:30:40.544 INFO:tasks.rados.rados.0.smithi038.stdout:969: finishing write tid 1 to smithi03829856-23 2024-03-19T11:30:40.544 INFO:tasks.rados.rados.0.smithi038.stdout:965: done (6 left) 2024-03-19T11:30:40.544 INFO:tasks.rados.rados.0.smithi038.stdout:966: done (5 left) 2024-03-19T11:30:40.544 INFO:tasks.rados.rados.0.smithi038.stdout:967: done (4 left) 2024-03-19T11:30:40.544 INFO:tasks.rados.rados.0.smithi038.stdout:968: done (3 left) 2024-03-19T11:30:40.544 INFO:tasks.rados.rados.0.smithi038.stdout:973: read oid 50 snap -1 2024-03-19T11:30:40.544 INFO:tasks.rados.rados.0.smithi038.stdout:973: expect (ObjNum 114 snap 11 seq_num 114) 2024-03-19T11:30:40.544 INFO:tasks.rados.rados.0.smithi038.stdout:974: read oid 1 snap -1 2024-03-19T11:30:40.544 INFO:tasks.rados.rados.0.smithi038.stdout:974: expect (ObjNum 127 snap 15 seq_num 127) 2024-03-19T11:30:40.544 INFO:tasks.rados.rados.0.smithi038.stdout:975: tier_evict oid 38 2024-03-19T11:30:40.544 INFO:tasks.rados.rados.0.smithi038.stdout:975: tier_evict oid 38 snap -1 2024-03-19T11:30:40.544 INFO:tasks.rados.rados.0.smithi038.stdout:969: finishing write tid 2 to smithi03829856-23 2024-03-19T11:30:40.544 INFO:tasks.rados.rados.0.smithi038.stdout:969: finishing write tid 3 to smithi03829856-23 2024-03-19T11:30:40.544 INFO:tasks.rados.rados.0.smithi038.stdout:969: finishing write tid 4 to smithi03829856-23 2024-03-19T11:30:40.545 INFO:tasks.rados.rados.0.smithi038.stdout:969: oid 23 updating version 0 to 71 2024-03-19T11:30:40.545 INFO:tasks.rados.rados.0.smithi038.stdout:969: oid 23 version 71 is already newer than 70 2024-03-19T11:30:40.545 INFO:tasks.rados.rados.0.smithi038.stdout:969: oid 23 version 71 is already newer than 69 2024-03-19T11:30:40.545 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 23 v 71 (ObjNum 139 snap 15 seq_num 139) dirty exists 2024-03-19T11:30:40.545 INFO:tasks.rados.rados.0.smithi038.stdout:969: left oid 23 (ObjNum 139 snap 15 seq_num 139) 2024-03-19T11:30:40.545 INFO:tasks.rados.rados.0.smithi038.stdout:972: got (0) Success 2024-03-19T11:30:40.545 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 20 v 12 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-03-19T11:30:40.545 INFO:tasks.rados.rados.0.smithi038.stdout:971: expect (ObjNum 29 snap 0 seq_num 29) 2024-03-19T11:30:40.579 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:40.590+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:40.617 INFO:tasks.rados.rados.0.smithi038.stdout:975: got (22) Invalid argument 2024-03-19T11:30:40.617 INFO:tasks.rados.rados.0.smithi038.stdout:974: expect (ObjNum 127 snap 15 seq_num 127) 2024-03-19T11:30:40.641 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:40.641 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:40.642 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:30:40.642 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:40.655 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-19T11:30:40.699 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:40.708+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:40.733 INFO:tasks.rados.rados.0.smithi038.stdout:973: expect (ObjNum 114 snap 11 seq_num 114) 2024-03-19T11:30:40.768 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:40.776+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:40.776 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:40.776 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:40.776 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:30:40.776 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:30:40.776 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:40.790 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-19T11:30:40.869 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:40.878+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:40.905 INFO:tasks.rados.rados.0.smithi038.stdout:969: done (5 left) 2024-03-19T11:30:40.906 INFO:tasks.rados.rados.0.smithi038.stdout:971: done (4 left) 2024-03-19T11:30:40.906 INFO:tasks.rados.rados.0.smithi038.stdout:972: done (3 left) 2024-03-19T11:30:40.906 INFO:tasks.rados.rados.0.smithi038.stdout:973: done (2 left) 2024-03-19T11:30:40.906 INFO:tasks.rados.rados.0.smithi038.stdout:974: done (1 left) 2024-03-19T11:30:40.906 INFO:tasks.rados.rados.0.smithi038.stdout:975: done (0 left) 2024-03-19T11:30:40.906 INFO:tasks.rados.rados.0.smithi038.stdout:976: tier_evict oid 16 2024-03-19T11:30:40.906 INFO:tasks.rados.rados.0.smithi038.stdout:976: tier_evict oid 16 snap -1 2024-03-19T11:30:40.906 INFO:tasks.rados.rados.0.smithi038.stdout:977: write (excl) oid 40 current snap is 15 2024-03-19T11:30:40.906 INFO:tasks.rados.rados.0.smithi038.stdout:977: seq_num 141 ranges {769787=704463,2267961=101826} 2024-03-19T11:30:40.929 INFO:tasks.rados.rados.0.smithi038.stdout:977: writing smithi03829856-40 from 769787 to 1474250 tid 1 2024-03-19T11:30:40.945 INFO:tasks.rados.rados.0.smithi038.stdout:977: writing smithi03829856-40 from 2267961 to 2369787 tid 2 2024-03-19T11:30:40.945 INFO:tasks.rados.rados.0.smithi038.stdout:978: read oid 43 snap 4 2024-03-19T11:30:40.945 INFO:tasks.rados.rados.0.smithi038.stdout:978: expect (ObjNum 50 snap 0 seq_num 50) 2024-03-19T11:30:40.945 INFO:tasks.rados.rados.0.smithi038.stdout:976: got (22) Invalid argument 2024-03-19T11:30:40.945 INFO:tasks.rados.rados.0.smithi038.stdout:977: finishing write tid 1 to smithi03829856-40 2024-03-19T11:30:40.945 INFO:tasks.rados.rados.0.smithi038.stdout:976: done (2 left) 2024-03-19T11:30:40.945 INFO:tasks.rados.rados.0.smithi038.stdout:979: rollback oid 8 current snap is 15 2024-03-19T11:30:40.945 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 8 to 11 2024-03-19T11:30:40.945 INFO:tasks.rados.rados.0.smithi038.stdout:980: tier_promote oid 32 2024-03-19T11:30:40.945 INFO:tasks.rados.rados.0.smithi038.stdout:981: write (excl) oid 12 current snap is 15 2024-03-19T11:30:40.946 INFO:tasks.rados.rados.0.smithi038.stdout:981: seq_num 142 ranges {786594=779712,2326443=60151} 2024-03-19T11:30:40.968 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:40.968 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:30:40.968 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:30:40.968 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:30:40.968 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.968 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:844 4.7 e32/21 4:e41d47fb:::smithi03829856-14:head v 32'8, mlcod=28'5)", 2024-03-19T11:30:40.968 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.751447+0000", 2024-03-19T11:30:40.968 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.226721570000002, 2024-03-19T11:30:40.968 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012301835000000001, 2024-03-19T11:30:40.969 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:40.969 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:40.969 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:40.969 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.969 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:40.969 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.751447+0000", 2024-03-19T11:30:40.969 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.969 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.969 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.969 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:40.969 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.751447+0000", 2024-03-19T11:30:40.969 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.969 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.970 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.970 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:40.970 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.751447+0000", 2024-03-19T11:30:40.970 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4300000000000003e-07 2024-03-19T11:30:40.970 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.970 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.970 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:40.970 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.752747+0000", 2024-03-19T11:30:40.970 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012995699999999999 2024-03-19T11:30:40.970 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.970 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.970 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:40.970 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.752771+0000", 2024-03-19T11:30:40.971 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3592999999999999e-05 2024-03-19T11:30:40.971 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.971 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.971 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:40.971 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.752789+0000", 2024-03-19T11:30:40.971 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8703e-05 2024-03-19T11:30:40.971 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.971 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.971 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:40.972 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.752911+0000", 2024-03-19T11:30:40.972 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012139899999999999 2024-03-19T11:30:40.972 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.972 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.972 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:40.972 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.752949+0000", 2024-03-19T11:30:40.972 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8436999999999998e-05 2024-03-19T11:30:40.972 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.972 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.973 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:40.973 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763568+0000", 2024-03-19T11:30:40.973 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010618768000000001 2024-03-19T11:30:40.973 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.973 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.973 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:40.973 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763749+0000", 2024-03-19T11:30:40.973 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000180822 2024-03-19T11:30:40.974 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.974 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:40.974 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.974 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.974 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.974 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1047 4.8 4:1d58db11:::smithi03829856-18:head [write 443718~615603,stat] snapc 6=[6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:40.974 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:13.221766+0000", 2024-03-19T11:30:40.974 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.756402402999999, 2024-03-19T11:30:40.974 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020932596000000001, 2024-03-19T11:30:40.974 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:40.974 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:40.974 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:40.974 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:40.974 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:40.974 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1047 2024-03-19T11:30:40.975 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.975 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:40.975 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.975 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:40.975 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.221766+0000", 2024-03-19T11:30:40.975 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.975 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.975 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.975 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:40.975 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.221766+0000", 2024-03-19T11:30:40.975 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.975 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.975 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.975 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:40.975 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.221770+0000", 2024-03-19T11:30:40.976 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.67e-06 2024-03-19T11:30:40.976 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.976 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.976 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:40.977 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.223938+0000", 2024-03-19T11:30:40.977 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021683399999999999 2024-03-19T11:30:40.977 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.977 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.977 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:40.977 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.223979+0000", 2024-03-19T11:30:40.977 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1433000000000001e-05 2024-03-19T11:30:40.977 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.978 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.978 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:40.978 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.223998+0000", 2024-03-19T11:30:40.978 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8853000000000001e-05 2024-03-19T11:30:40.978 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.978 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.978 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:40.978 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.224150+0000", 2024-03-19T11:30:40.978 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000151679 2024-03-19T11:30:40.978 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.978 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.978 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:40.978 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.224362+0000", 2024-03-19T11:30:40.978 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021154499999999999 2024-03-19T11:30:40.979 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.979 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.979 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:40.979 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.224745+0000", 2024-03-19T11:30:40.979 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00038342399999999999 2024-03-19T11:30:40.979 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.979 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.979 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:40.979 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.241167+0000", 2024-03-19T11:30:40.979 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0164218 2024-03-19T11:30:40.979 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.979 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.979 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:40.979 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.242406+0000", 2024-03-19T11:30:40.979 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012390039999999999 2024-03-19T11:30:40.980 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.980 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.980 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:40.980 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.242575+0000", 2024-03-19T11:30:40.980 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000168925 2024-03-19T11:30:40.980 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.980 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.980 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:40.980 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.242699+0000", 2024-03-19T11:30:40.980 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012392300000000001 2024-03-19T11:30:40.980 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.980 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:40.980 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.980 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.981 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.981 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1048 4.8 4:1d58db11:::smithi03829856-18:head [write 1564138~470341] snapc 6=[6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:40.981 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:13.227287+0000", 2024-03-19T11:30:40.981 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.750881135, 2024-03-19T11:30:40.981 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015802897999999999, 2024-03-19T11:30:40.981 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:40.981 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:40.981 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:40.981 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:40.981 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:40.981 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1048 2024-03-19T11:30:40.981 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.981 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:40.981 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.981 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:40.982 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.227287+0000", 2024-03-19T11:30:40.982 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.982 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.982 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.982 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:40.982 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.227287+0000", 2024-03-19T11:30:40.982 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.982 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.982 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.982 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:40.982 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.227290+0000", 2024-03-19T11:30:40.982 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.88e-06 2024-03-19T11:30:40.983 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.983 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.983 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:40.983 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.228224+0000", 2024-03-19T11:30:40.983 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00093389600000000003 2024-03-19T11:30:40.983 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.983 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.983 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:40.983 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.228258+0000", 2024-03-19T11:30:40.983 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4282999999999997e-05 2024-03-19T11:30:40.983 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.983 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.983 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:40.983 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.228273+0000", 2024-03-19T11:30:40.983 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4833e-05 2024-03-19T11:30:40.983 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.984 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.984 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:40.984 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.231996+0000", 2024-03-19T11:30:40.984 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037223410000000001 2024-03-19T11:30:40.984 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.984 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.984 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:40.984 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.232143+0000", 2024-03-19T11:30:40.984 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014714900000000001 2024-03-19T11:30:40.984 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.984 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.984 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:40.984 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.232395+0000", 2024-03-19T11:30:40.985 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025201500000000002 2024-03-19T11:30:40.985 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.985 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.985 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:40.985 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.242848+0000", 2024-03-19T11:30:40.985 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010453393 2024-03-19T11:30:40.985 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.985 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.985 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:40.985 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.242872+0000", 2024-03-19T11:30:40.985 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3816e-05 2024-03-19T11:30:40.985 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.985 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.985 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:40.986 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.242966+0000", 2024-03-19T11:30:40.986 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4560000000000003e-05 2024-03-19T11:30:40.986 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.986 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.986 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:40.986 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.243090+0000", 2024-03-19T11:30:40.986 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012373199999999999 2024-03-19T11:30:40.986 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.986 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:40.986 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.986 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.986 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.986 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1072 4.1 e36/21 4:8401c1f0:::smithi03829856-43:head v 36'38, mlcod=36'35)", 2024-03-19T11:30:40.986 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:13.973829+0000", 2024-03-19T11:30:40.987 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.004339812000001, 2024-03-19T11:30:40.987 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013548782000000001, 2024-03-19T11:30:40.987 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:40.987 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:40.987 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:40.987 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.987 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:40.987 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.973829+0000", 2024-03-19T11:30:40.987 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.987 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.987 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.987 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:40.987 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.973829+0000", 2024-03-19T11:30:40.987 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.988 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.988 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.988 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:40.988 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.973831+0000", 2024-03-19T11:30:40.988 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0899999999999999e-06 2024-03-19T11:30:40.988 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.988 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.988 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:40.988 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.974489+0000", 2024-03-19T11:30:40.988 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00065813199999999997 2024-03-19T11:30:40.988 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.988 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.988 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:40.988 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.974523+0000", 2024-03-19T11:30:40.989 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3841999999999999e-05 2024-03-19T11:30:40.989 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.989 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.989 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:40.989 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.974544+0000", 2024-03-19T11:30:40.989 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0907000000000002e-05 2024-03-19T11:30:40.989 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.989 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.989 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:40.989 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.974697+0000", 2024-03-19T11:30:40.989 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015295599999999999 2024-03-19T11:30:40.989 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.989 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.989 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:40.989 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.974742+0000", 2024-03-19T11:30:40.990 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5735999999999997e-05 2024-03-19T11:30:40.990 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.990 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.990 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:40.990 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.987261+0000", 2024-03-19T11:30:40.990 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012518316999999999 2024-03-19T11:30:40.990 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.990 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.990 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:40.990 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.987377+0000", 2024-03-19T11:30:40.990 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000116802 2024-03-19T11:30:40.990 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.990 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:40.990 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.991 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.991 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.991 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1091 4.8 4:14611006:::smithi03829856-25:head [write 527753~708201] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:40.991 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:14.015839+0000", 2024-03-19T11:30:40.991 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 26.962329752999999, 2024-03-19T11:30:40.991 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016721983999999999, 2024-03-19T11:30:40.991 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:40.991 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:40.991 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:40.991 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:40.991 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:40.991 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1091 2024-03-19T11:30:40.991 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.991 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:40.991 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.992 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:40.992 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.015839+0000", 2024-03-19T11:30:40.992 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.992 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.992 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.992 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:40.992 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.015839+0000", 2024-03-19T11:30:40.992 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.992 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.992 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.992 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:40.992 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.015842+0000", 2024-03-19T11:30:40.992 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3569999999999998e-06 2024-03-19T11:30:40.992 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.993 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.993 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:40.993 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.016678+0000", 2024-03-19T11:30:40.993 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00083620299999999999 2024-03-19T11:30:40.993 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.993 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.993 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:40.993 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.016710+0000", 2024-03-19T11:30:40.993 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1337000000000002e-05 2024-03-19T11:30:40.993 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.993 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.993 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:40.993 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.016724+0000", 2024-03-19T11:30:40.993 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4323e-05 2024-03-19T11:30:40.993 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.994 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.994 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:40.994 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.016847+0000", 2024-03-19T11:30:40.994 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012304199999999999 2024-03-19T11:30:40.994 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.994 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.994 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:40.994 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.017023+0000", 2024-03-19T11:30:40.994 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000175945 2024-03-19T11:30:40.994 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.994 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.994 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:40.994 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.017324+0000", 2024-03-19T11:30:40.994 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00030151499999999998 2024-03-19T11:30:40.995 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.995 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.995 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:40.995 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.032104+0000", 2024-03-19T11:30:40.995 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014779232999999999 2024-03-19T11:30:40.995 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.995 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.995 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:40.995 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.032190+0000", 2024-03-19T11:30:40.995 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6304999999999994e-05 2024-03-19T11:30:40.995 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.995 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.995 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:40.995 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.032350+0000", 2024-03-19T11:30:40.995 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000159813 2024-03-19T11:30:40.996 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.996 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.996 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:40.996 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.032561+0000", 2024-03-19T11:30:40.996 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021091099999999999 2024-03-19T11:30:40.996 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.996 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:40.996 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:40.996 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.996 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.996 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1158 4.8 4:15708482:::smithi03829856-23:head [tier-promote] snapc 8=[8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-03-19T11:30:40.996 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.827517+0000", 2024-03-19T11:30:40.996 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.150651825000001, 2024-03-19T11:30:40.996 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.034096983999999997, 2024-03-19T11:30:40.996 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:40.997 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:40.997 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:40.997 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:40.997 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:40.997 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1158 2024-03-19T11:30:40.997 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.997 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:40.997 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.997 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:40.997 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827517+0000", 2024-03-19T11:30:40.997 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.997 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.997 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.997 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:40.998 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827517+0000", 2024-03-19T11:30:40.998 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:40.998 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.998 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.998 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:40.998 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827520+0000", 2024-03-19T11:30:40.998 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.45e-06 2024-03-19T11:30:40.998 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.998 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.998 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:40.998 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827535+0000", 2024-03-19T11:30:40.998 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.509e-05 2024-03-19T11:30:40.998 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.998 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.999 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:40.999 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827579+0000", 2024-03-19T11:30:40.999 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4211000000000001e-05 2024-03-19T11:30:40.999 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.999 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.999 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:40.999 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827603+0000", 2024-03-19T11:30:40.999 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3586000000000001e-05 2024-03-19T11:30:40.999 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.999 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:40.999 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:40.999 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827789+0000", 2024-03-19T11:30:40.999 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018580199999999999 2024-03-19T11:30:40.999 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:40.999 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.000 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:41.000 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.828077+0000", 2024-03-19T11:30:41.000 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028846500000000001 2024-03-19T11:30:41.000 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.000 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.000 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:41.000 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.861614+0000", 2024-03-19T11:30:41.000 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033536379999999998 2024-03-19T11:30:41.000 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.000 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:41.000 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.000 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.000 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.000 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1162 4.1 e39/21 4:8401c1f0:::smithi03829856-43:head v 39'47, mlcod=38'43)", 2024-03-19T11:30:41.000 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.842598+0000", 2024-03-19T11:30:41.001 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.135570054999999, 2024-03-19T11:30:41.001 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0233756, 2024-03-19T11:30:41.001 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:41.001 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:41.001 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:41.001 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.001 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:41.001 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842598+0000", 2024-03-19T11:30:41.001 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.001 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.001 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.001 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:41.001 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842598+0000", 2024-03-19T11:30:41.001 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.001 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.002 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.002 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:41.002 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842599+0000", 2024-03-19T11:30:41.002 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6000000000000003e-07 2024-03-19T11:30:41.002 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.002 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.002 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:41.002 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.843566+0000", 2024-03-19T11:30:41.002 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00096688700000000004 2024-03-19T11:30:41.002 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.002 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.002 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:41.002 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.843601+0000", 2024-03-19T11:30:41.002 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.502e-05 2024-03-19T11:30:41.003 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.003 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.003 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:41.003 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.843621+0000", 2024-03-19T11:30:41.003 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0305e-05 2024-03-19T11:30:41.003 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.003 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.003 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:41.003 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.843882+0000", 2024-03-19T11:30:41.003 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026027100000000002 2024-03-19T11:30:41.003 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.003 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.003 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:41.003 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.844022+0000", 2024-03-19T11:30:41.004 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014020100000000001 2024-03-19T11:30:41.004 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.004 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.004 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:41.004 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.865862+0000", 2024-03-19T11:30:41.004 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021839725000000001 2024-03-19T11:30:41.004 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.004 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.004 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:41.004 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.865974+0000", 2024-03-19T11:30:41.004 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011243099999999999 2024-03-19T11:30:41.004 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.004 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:41.004 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.004 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.005 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.005 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1209 4.8 4:1d58db11:::smithi03829856-18:head [write 645402~551297,stat] snapc 9=[9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:41.005 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:18.121657+0000", 2024-03-19T11:30:41.005 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.856511052999998, 2024-03-19T11:30:41.005 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.025074349999999999, 2024-03-19T11:30:41.005 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:41.005 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:41.005 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:41.005 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:41.005 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:41.005 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1209 2024-03-19T11:30:41.005 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.005 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:41.005 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.005 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:41.006 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.121657+0000", 2024-03-19T11:30:41.006 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.006 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.006 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.006 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:41.006 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.121657+0000", 2024-03-19T11:30:41.006 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.006 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.006 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.006 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:41.006 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.121661+0000", 2024-03-19T11:30:41.006 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0489999999999997e-06 2024-03-19T11:30:41.006 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.006 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.007 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:41.007 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123747+0000", 2024-03-19T11:30:41.007 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0020857720000000001 2024-03-19T11:30:41.007 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.007 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.007 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:41.007 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123779+0000", 2024-03-19T11:30:41.007 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2258000000000002e-05 2024-03-19T11:30:41.007 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.007 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.007 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:41.007 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123795+0000", 2024-03-19T11:30:41.007 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5177000000000001e-05 2024-03-19T11:30:41.007 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.007 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.008 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:41.008 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123949+0000", 2024-03-19T11:30:41.008 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015403699999999999 2024-03-19T11:30:41.008 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.008 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.008 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:41.008 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.124134+0000", 2024-03-19T11:30:41.008 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000185484 2024-03-19T11:30:41.008 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.008 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.008 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:41.008 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.124554+0000", 2024-03-19T11:30:41.008 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00042019699999999998 2024-03-19T11:30:41.008 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.008 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.009 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:41.009 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146267+0000", 2024-03-19T11:30:41.009 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021712838000000002 2024-03-19T11:30:41.009 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.009 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.009 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:41.009 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146424+0000", 2024-03-19T11:30:41.009 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015647299999999999 2024-03-19T11:30:41.009 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.009 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.009 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:41.009 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146597+0000", 2024-03-19T11:30:41.009 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017303699999999999 2024-03-19T11:30:41.009 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.009 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.010 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:41.010 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146732+0000", 2024-03-19T11:30:41.010 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013502800000000001 2024-03-19T11:30:41.010 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.010 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:41.010 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.010 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.010 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.010 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1210 4.8 4:1d58db11:::smithi03829856-18:head [write 1749569~492355] snapc 9=[9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:41.010 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:18.129257+0000", 2024-03-19T11:30:41.010 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.848911696000002, 2024-03-19T11:30:41.010 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018110444999999999, 2024-03-19T11:30:41.010 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:41.010 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:41.011 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:41.011 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:41.011 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:41.011 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1210 2024-03-19T11:30:41.011 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.011 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:41.011 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.011 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:41.011 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.129257+0000", 2024-03-19T11:30:41.011 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.011 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.011 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.011 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:41.011 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.129257+0000", 2024-03-19T11:30:41.011 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.012 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.012 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.012 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:41.012 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.129261+0000", 2024-03-19T11:30:41.012 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7119999999999999e-06 2024-03-19T11:30:41.012 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.012 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.012 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:41.012 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.129717+0000", 2024-03-19T11:30:41.012 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00045526300000000002 2024-03-19T11:30:41.012 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.012 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.012 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:41.012 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.129749+0000", 2024-03-19T11:30:41.013 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2700999999999997e-05 2024-03-19T11:30:41.013 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.013 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.013 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:41.013 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.129774+0000", 2024-03-19T11:30:41.013 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4592000000000001e-05 2024-03-19T11:30:41.013 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.013 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.013 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:41.013 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.134875+0000", 2024-03-19T11:30:41.013 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.005100477 2024-03-19T11:30:41.013 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.013 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.013 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:41.013 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.135035+0000", 2024-03-19T11:30:41.014 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016054200000000001 2024-03-19T11:30:41.014 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.014 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.014 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:41.014 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.135268+0000", 2024-03-19T11:30:41.014 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000233253 2024-03-19T11:30:41.014 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.014 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.014 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:41.014 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146872+0000", 2024-03-19T11:30:41.014 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011603732 2024-03-19T11:30:41.014 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.014 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.014 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:41.014 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.147103+0000", 2024-03-19T11:30:41.015 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023070299999999999 2024-03-19T11:30:41.015 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.015 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.015 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:41.015 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.147181+0000", 2024-03-19T11:30:41.015 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8567999999999997e-05 2024-03-19T11:30:41.015 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.015 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.015 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:41.015 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.147367+0000", 2024-03-19T11:30:41.015 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000185902 2024-03-19T11:30:41.015 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.015 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:41.015 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.016 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.016 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.016 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1259 4.1 e44/43 4:866f3dcc:::smithi03829856-31:head v 44'53, mlcod=0'0)", 2024-03-19T11:30:41.016 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:21.869109+0000", 2024-03-19T11:30:41.016 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.109059806000001, 2024-03-19T11:30:41.016 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014777017999999999, 2024-03-19T11:30:41.016 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:41.016 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:41.016 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:41.016 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.016 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:41.016 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.869109+0000", 2024-03-19T11:30:41.016 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.016 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.016 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.016 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:41.017 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.869109+0000", 2024-03-19T11:30:41.017 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.017 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.017 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.017 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:41.017 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.869110+0000", 2024-03-19T11:30:41.017 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0559999999999999e-06 2024-03-19T11:30:41.017 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.017 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.017 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:41.017 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.870408+0000", 2024-03-19T11:30:41.017 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012978499999999999 2024-03-19T11:30:41.017 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.017 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.018 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:41.018 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.870437+0000", 2024-03-19T11:30:41.018 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9638999999999999e-05 2024-03-19T11:30:41.018 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.018 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.018 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:41.018 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.870460+0000", 2024-03-19T11:30:41.018 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2517000000000001e-05 2024-03-19T11:30:41.018 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.018 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.018 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:41.018 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.870681+0000", 2024-03-19T11:30:41.018 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022102800000000001 2024-03-19T11:30:41.018 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.018 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.019 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:41.019 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.870735+0000", 2024-03-19T11:30:41.019 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4616e-05 2024-03-19T11:30:41.019 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.019 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.019 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:41.019 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.883773+0000", 2024-03-19T11:30:41.019 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013038088 2024-03-19T11:30:41.019 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.019 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.019 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:41.019 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.883886+0000", 2024-03-19T11:30:41.019 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000112224 2024-03-19T11:30:41.019 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.019 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:41.020 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.020 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.020 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.020 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1416 4.18 4:1dabc0f1:::smithi03829856-1:head [write 564735~613734,stat] snapc c=[c,b,a,6,5] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:41.020 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.949508+0000", 2024-03-19T11:30:41.020 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.028660476000001, 2024-03-19T11:30:41.020 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016253106, 2024-03-19T11:30:41.020 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:41.020 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:41.020 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:41.020 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:41.020 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:41.020 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1416 2024-03-19T11:30:41.020 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.021 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:41.021 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.021 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:41.021 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.949508+0000", 2024-03-19T11:30:41.021 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.021 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.021 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.021 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:41.021 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.949508+0000", 2024-03-19T11:30:41.021 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.021 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.021 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.021 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:41.021 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.949511+0000", 2024-03-19T11:30:41.021 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2370000000000001e-06 2024-03-19T11:30:41.022 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.022 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.022 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:41.022 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.950555+0000", 2024-03-19T11:30:41.022 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001043925 2024-03-19T11:30:41.022 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.022 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.022 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:41.022 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.950594+0000", 2024-03-19T11:30:41.022 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9351999999999999e-05 2024-03-19T11:30:41.022 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.022 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.022 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:41.022 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.950612+0000", 2024-03-19T11:30:41.022 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7524e-05 2024-03-19T11:30:41.023 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.023 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.023 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:41.023 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.950745+0000", 2024-03-19T11:30:41.023 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013291 2024-03-19T11:30:41.023 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.023 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.023 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:41.023 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.951003+0000", 2024-03-19T11:30:41.023 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025828400000000001 2024-03-19T11:30:41.023 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.023 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.023 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:41.023 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.951335+0000", 2024-03-19T11:30:41.023 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033130200000000001 2024-03-19T11:30:41.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.024 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:41.024 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.965379+0000", 2024-03-19T11:30:41.024 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014044581 2024-03-19T11:30:41.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.024 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:41.024 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.965502+0000", 2024-03-19T11:30:41.024 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012290099999999999 2024-03-19T11:30:41.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.024 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:41.024 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.965668+0000", 2024-03-19T11:30:41.024 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016619200000000001 2024-03-19T11:30:41.025 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.025 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.025 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:41.025 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.965761+0000", 2024-03-19T11:30:41.025 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2898000000000004e-05 2024-03-19T11:30:41.025 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.025 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:41.025 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.025 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.025 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.025 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1417 4.18 4:1dabc0f1:::smithi03829856-1:head [write 1828021~669034] snapc c=[c,b,a,6,5] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:41.025 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.958407+0000", 2024-03-19T11:30:41.025 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.019761075, 2024-03-19T11:30:41.025 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016467840000000001, 2024-03-19T11:30:41.025 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:41.026 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:41.026 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:41.026 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:41.026 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:41.026 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1417 2024-03-19T11:30:41.026 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.026 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:41.026 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.026 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:41.026 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.958407+0000", 2024-03-19T11:30:41.026 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.026 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.026 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.026 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:41.027 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.958407+0000", 2024-03-19T11:30:41.027 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.027 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.027 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.027 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:41.027 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.958410+0000", 2024-03-19T11:30:41.027 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9830000000000001e-06 2024-03-19T11:30:41.027 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.027 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.027 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:41.027 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.959442+0000", 2024-03-19T11:30:41.027 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001031957 2024-03-19T11:30:41.027 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.027 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.028 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:41.028 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.959475+0000", 2024-03-19T11:30:41.028 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3145999999999997e-05 2024-03-19T11:30:41.028 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.028 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:41.028 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.959494+0000", 2024-03-19T11:30:41.028 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8994000000000001e-05 2024-03-19T11:30:41.028 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.028 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:41.028 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.959717+0000", 2024-03-19T11:30:41.028 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000222272 2024-03-19T11:30:41.028 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.029 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:41.029 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.960015+0000", 2024-03-19T11:30:41.029 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029870599999999998 2024-03-19T11:30:41.029 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.029 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.029 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:41.029 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.960312+0000", 2024-03-19T11:30:41.029 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029640299999999999 2024-03-19T11:30:41.029 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.029 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.029 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:41.029 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.966435+0000", 2024-03-19T11:30:41.029 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061234999999999996 2024-03-19T11:30:41.029 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.029 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.030 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:41.030 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.974556+0000", 2024-03-19T11:30:41.030 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.008120759 2024-03-19T11:30:41.030 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.030 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.030 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:41.030 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.974729+0000", 2024-03-19T11:30:41.030 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000173191 2024-03-19T11:30:41.030 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.030 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.030 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:41.030 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.974875+0000", 2024-03-19T11:30:41.030 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014592899999999999 2024-03-19T11:30:41.030 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.031 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:41.031 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.031 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.031 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.031 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1419 4.18 4:1dabc0f1:::smithi03829856-1:head [setxattr _header (56) in=7b,truncate 3764735] snapc c=[c,b,a,6,5] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:41.031 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.968061+0000", 2024-03-19T11:30:41.031 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.010107216, 2024-03-19T11:30:41.031 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012422488000000001, 2024-03-19T11:30:41.031 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:41.031 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:41.031 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:41.031 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:41.031 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:41.031 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1419 2024-03-19T11:30:41.031 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.032 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:41.032 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.032 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:41.032 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968061+0000", 2024-03-19T11:30:41.032 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.032 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.032 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.032 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:41.032 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968061+0000", 2024-03-19T11:30:41.032 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.032 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.032 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.032 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:41.032 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968063+0000", 2024-03-19T11:30:41.033 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.767e-06 2024-03-19T11:30:41.033 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.033 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.033 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:41.033 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968074+0000", 2024-03-19T11:30:41.033 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0525e-05 2024-03-19T11:30:41.033 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.033 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.033 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:41.033 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968096+0000", 2024-03-19T11:30:41.033 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2926999999999999e-05 2024-03-19T11:30:41.033 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.033 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.033 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:41.034 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968113+0000", 2024-03-19T11:30:41.034 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6665000000000001e-05 2024-03-19T11:30:41.034 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.034 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.034 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:41.034 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.973218+0000", 2024-03-19T11:30:41.034 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0051044920000000004 2024-03-19T11:30:41.034 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.034 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.034 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:41.034 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.973511+0000", 2024-03-19T11:30:41.034 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029303899999999998 2024-03-19T11:30:41.034 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.035 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.035 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:41.035 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.973931+0000", 2024-03-19T11:30:41.035 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00041991400000000002 2024-03-19T11:30:41.035 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.035 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.035 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:41.035 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.977029+0000", 2024-03-19T11:30:41.035 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030983040000000001 2024-03-19T11:30:41.035 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.035 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.035 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:41.035 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.980091+0000", 2024-03-19T11:30:41.035 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003061851 2024-03-19T11:30:41.035 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.036 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:41.036 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.980285+0000", 2024-03-19T11:30:41.036 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000194279 2024-03-19T11:30:41.036 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.036 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:41.036 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.980484+0000", 2024-03-19T11:30:41.036 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019872499999999999 2024-03-19T11:30:41.036 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.036 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:41.036 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.036 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.036 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1420 4.18 4:1dabc0f1:::smithi03829856-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:41.037 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.968138+0000", 2024-03-19T11:30:41.037 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.010030005999999, 2024-03-19T11:30:41.037 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013109305999999999, 2024-03-19T11:30:41.037 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:41.037 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:41.037 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:41.037 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:41.037 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:41.037 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1420 2024-03-19T11:30:41.037 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.037 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:41.037 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.037 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:41.037 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968138+0000", 2024-03-19T11:30:41.038 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.038 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.038 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.038 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:41.038 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968138+0000", 2024-03-19T11:30:41.038 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.038 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.038 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.038 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:41.038 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968140+0000", 2024-03-19T11:30:41.038 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4139999999999999e-06 2024-03-19T11:30:41.038 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.038 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.038 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:41.039 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968147+0000", 2024-03-19T11:30:41.039 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.815e-06 2024-03-19T11:30:41.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.039 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:41.039 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968167+0000", 2024-03-19T11:30:41.039 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0018e-05 2024-03-19T11:30:41.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.039 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:41.039 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968182+0000", 2024-03-19T11:30:41.039 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5475e-05 2024-03-19T11:30:41.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.040 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:41.040 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.974950+0000", 2024-03-19T11:30:41.040 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.006767856 2024-03-19T11:30:41.040 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.040 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.040 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-03-19T11:30:41.040 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.975095+0000", 2024-03-19T11:30:41.040 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014501499999999999 2024-03-19T11:30:41.040 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.040 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.040 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:41.040 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.980614+0000", 2024-03-19T11:30:41.040 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0055184789999999997 2024-03-19T11:30:41.040 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.040 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.041 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:41.041 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.980912+0000", 2024-03-19T11:30:41.041 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029858199999999998 2024-03-19T11:30:41.041 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.041 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.041 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:41.041 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.981248+0000", 2024-03-19T11:30:41.041 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033565199999999998 2024-03-19T11:30:41.041 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.041 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:41.041 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.041 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.041 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.041 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1443 4.2 4:43b32242:::smithi03829856-22:head [read 0~2352947,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:41.042 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.979417+0000", 2024-03-19T11:30:41.042 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.99875138, 2024-03-19T11:30:41.042 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.96719988700000004, 2024-03-19T11:30:41.042 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:41.042 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:41.042 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:41.042 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:41.042 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:41.042 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1443 2024-03-19T11:30:41.042 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.042 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:41.042 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.042 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:41.042 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979417+0000", 2024-03-19T11:30:41.042 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.043 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:41.043 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979417+0000", 2024-03-19T11:30:41.043 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.043 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:41.043 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979420+0000", 2024-03-19T11:30:41.043 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.366e-06 2024-03-19T11:30:41.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.043 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:41.043 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979437+0000", 2024-03-19T11:30:41.043 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6192e-05 2024-03-19T11:30:41.044 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.044 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.044 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:41.044 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979474+0000", 2024-03-19T11:30:41.044 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7601000000000001e-05 2024-03-19T11:30:41.044 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.044 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.044 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:41.044 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979496+0000", 2024-03-19T11:30:41.044 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1506999999999999e-05 2024-03-19T11:30:41.044 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.044 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.044 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:41.044 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979712+0000", 2024-03-19T11:30:41.044 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021641299999999999 2024-03-19T11:30:41.045 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.045 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.045 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:30:41.045 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979746+0000", 2024-03-19T11:30:41.045 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3386000000000002e-05 2024-03-19T11:30:41.045 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.045 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.045 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:41.045 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.931552+0000", 2024-03-19T11:30:41.045 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.95180659899999998 2024-03-19T11:30:41.045 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.045 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.045 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-03-19T11:30:41.045 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.934681+0000", 2024-03-19T11:30:41.045 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0031292289999999999 2024-03-19T11:30:41.046 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.046 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.046 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:41.046 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.942701+0000", 2024-03-19T11:30:41.046 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0080198809999999995 2024-03-19T11:30:41.046 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.046 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.046 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:41.046 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.942839+0000", 2024-03-19T11:30:41.046 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013816999999999999 2024-03-19T11:30:41.046 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.046 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.046 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:41.046 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.946617+0000", 2024-03-19T11:30:41.046 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003777543 2024-03-19T11:30:41.047 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.047 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:41.047 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.047 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.047 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.047 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1444 4.2 4:43b32242:::smithi03829856-22:head [read 0~2352947,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:41.047 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.979553+0000", 2024-03-19T11:30:41.047 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.998615677, 2024-03-19T11:30:41.047 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.97015945400000003, 2024-03-19T11:30:41.047 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:41.047 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:41.047 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:41.047 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:41.047 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:41.047 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1444 2024-03-19T11:30:41.048 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.048 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:41.048 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.048 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:41.048 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979553+0000", 2024-03-19T11:30:41.048 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.048 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.048 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.048 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:41.048 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979553+0000", 2024-03-19T11:30:41.048 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.048 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.048 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.048 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:41.048 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979554+0000", 2024-03-19T11:30:41.048 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.403e-06 2024-03-19T11:30:41.049 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.049 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.049 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:41.049 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979564+0000", 2024-03-19T11:30:41.049 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0032e-05 2024-03-19T11:30:41.049 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.049 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.049 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:41.049 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979579+0000", 2024-03-19T11:30:41.049 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4875000000000001e-05 2024-03-19T11:30:41.049 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.049 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.049 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:41.049 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979595+0000", 2024-03-19T11:30:41.050 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6300999999999999e-05 2024-03-19T11:30:41.050 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.050 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.050 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:41.050 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979874+0000", 2024-03-19T11:30:41.050 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000279065 2024-03-19T11:30:41.050 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.050 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.050 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:30:41.050 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979904+0000", 2024-03-19T11:30:41.050 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.94e-05 2024-03-19T11:30:41.050 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.050 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.051 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:41.051 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.934781+0000", 2024-03-19T11:30:41.051 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.95487744900000004 2024-03-19T11:30:41.051 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.051 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.051 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-03-19T11:30:41.051 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.934946+0000", 2024-03-19T11:30:41.051 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016485999999999999 2024-03-19T11:30:41.051 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.051 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.051 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:41.051 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.946829+0000", 2024-03-19T11:30:41.051 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011882482999999999 2024-03-19T11:30:41.051 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.051 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.052 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:41.052 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.946989+0000", 2024-03-19T11:30:41.052 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000160421 2024-03-19T11:30:41.052 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.052 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.052 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:41.052 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.949712+0000", 2024-03-19T11:30:41.052 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002723165 2024-03-19T11:30:41.052 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.052 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:41.052 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.052 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.052 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.052 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1445 4.2 4:43b32242:::smithi03829856-22:head [read 0~2352947,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:41.052 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.979653+0000", 2024-03-19T11:30:41.053 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.998515546, 2024-03-19T11:30:41.053 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.97249507999999996, 2024-03-19T11:30:41.053 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:41.053 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:41.053 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:41.053 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:41.053 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:41.053 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1445 2024-03-19T11:30:41.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.053 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:41.053 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.053 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:41.053 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979653+0000", 2024-03-19T11:30:41.053 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.054 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:41.054 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979653+0000", 2024-03-19T11:30:41.054 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.054 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.054 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:41.054 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979654+0000", 2024-03-19T11:30:41.054 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.372e-06 2024-03-19T11:30:41.054 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.054 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:41.054 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979662+0000", 2024-03-19T11:30:41.054 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9959999999999993e-06 2024-03-19T11:30:41.055 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.055 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.055 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:41.055 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979698+0000", 2024-03-19T11:30:41.055 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5763000000000003e-05 2024-03-19T11:30:41.055 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.055 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.055 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:41.055 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979714+0000", 2024-03-19T11:30:41.055 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6283999999999999e-05 2024-03-19T11:30:41.055 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.055 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.055 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:41.055 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979990+0000", 2024-03-19T11:30:41.055 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000275829 2024-03-19T11:30:41.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.056 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:30:41.056 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.980018+0000", 2024-03-19T11:30:41.056 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8082999999999999e-05 2024-03-19T11:30:41.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.056 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:41.056 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.935620+0000", 2024-03-19T11:30:41.056 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.95560133800000002 2024-03-19T11:30:41.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.056 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-03-19T11:30:41.056 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.935795+0000", 2024-03-19T11:30:41.056 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000175656 2024-03-19T11:30:41.057 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.057 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:41.057 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.949835+0000", 2024-03-19T11:30:41.057 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014040083 2024-03-19T11:30:41.057 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.057 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:41.057 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.950002+0000", 2024-03-19T11:30:41.057 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016673 2024-03-19T11:30:41.057 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.057 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:41.057 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.952148+0000", 2024-03-19T11:30:41.058 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021459460000000001 2024-03-19T11:30:41.058 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.058 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:41.058 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.058 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.058 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.058 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1449 4.18 4:1d58db11:::smithi03829856-18:head [write 581542~688983,stat] snapc c=[c,b,a,6,5] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:41.058 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.989354+0000", 2024-03-19T11:30:41.058 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.988814093, 2024-03-19T11:30:41.058 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019237318, 2024-03-19T11:30:41.058 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:41.058 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:41.058 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:41.058 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:41.058 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:41.059 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1449 2024-03-19T11:30:41.059 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.059 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:41.059 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.059 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:41.059 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.989354+0000", 2024-03-19T11:30:41.059 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.059 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.059 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.059 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:41.059 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.989354+0000", 2024-03-19T11:30:41.059 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.059 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.059 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.059 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:41.060 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.989357+0000", 2024-03-19T11:30:41.060 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2740000000000002e-06 2024-03-19T11:30:41.060 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.060 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.060 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:41.060 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.990109+0000", 2024-03-19T11:30:41.060 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00075226500000000001 2024-03-19T11:30:41.060 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.060 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.060 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:41.060 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.990142+0000", 2024-03-19T11:30:41.060 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3328999999999999e-05 2024-03-19T11:30:41.060 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.060 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.061 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:41.061 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.990168+0000", 2024-03-19T11:30:41.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6016999999999998e-05 2024-03-19T11:30:41.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.061 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:41.061 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.990291+0000", 2024-03-19T11:30:41.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000122607 2024-03-19T11:30:41.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.061 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:41.061 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.990461+0000", 2024-03-19T11:30:41.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000169743 2024-03-19T11:30:41.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:41.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.990792+0000", 2024-03-19T11:30:41.062 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033135100000000002 2024-03-19T11:30:41.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:41.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.008238+0000", 2024-03-19T11:30:41.062 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017445591999999999 2024-03-19T11:30:41.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:41.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.008345+0000", 2024-03-19T11:30:41.062 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010762 2024-03-19T11:30:41.063 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.063 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.063 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:41.063 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.008436+0000", 2024-03-19T11:30:41.063 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1116e-05 2024-03-19T11:30:41.063 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.063 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.063 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:41.063 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.008592+0000", 2024-03-19T11:30:41.063 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015540399999999999 2024-03-19T11:30:41.063 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.063 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:41.063 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.063 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.063 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.064 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1450 4.18 4:1d58db11:::smithi03829856-18:head [write 1886503~412692] snapc c=[c,b,a,6,5] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:41.064 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.992973+0000", 2024-03-19T11:30:41.064 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.985195323999999, 2024-03-19T11:30:41.064 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016912580999999999, 2024-03-19T11:30:41.064 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:41.064 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:41.064 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:41.064 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:41.064 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:41.064 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1450 2024-03-19T11:30:41.064 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.064 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:41.064 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.065 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:41.065 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.992973+0000", 2024-03-19T11:30:41.065 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.065 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.065 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.065 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:41.065 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.992973+0000", 2024-03-19T11:30:41.065 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.065 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.065 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.065 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:41.065 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.992976+0000", 2024-03-19T11:30:41.065 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0060000000000001e-06 2024-03-19T11:30:41.065 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.065 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.066 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:41.066 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.993655+0000", 2024-03-19T11:30:41.066 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00067901099999999998 2024-03-19T11:30:41.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.066 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.066 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:41.066 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.993691+0000", 2024-03-19T11:30:41.066 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5546000000000002e-05 2024-03-19T11:30:41.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.066 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.066 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:41.066 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.993713+0000", 2024-03-19T11:30:41.066 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2277e-05 2024-03-19T11:30:41.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.067 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.067 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:41.067 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.997961+0000", 2024-03-19T11:30:41.067 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0042479659999999997 2024-03-19T11:30:41.067 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.067 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.067 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:41.067 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.998130+0000", 2024-03-19T11:30:41.067 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016908900000000001 2024-03-19T11:30:41.067 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.067 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.067 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:41.067 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.998397+0000", 2024-03-19T11:30:41.067 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026707599999999999 2024-03-19T11:30:41.067 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.068 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.068 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:41.068 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.008811+0000", 2024-03-19T11:30:41.068 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010413751000000001 2024-03-19T11:30:41.068 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.068 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.068 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:41.068 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.009539+0000", 2024-03-19T11:30:41.068 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00072811499999999999 2024-03-19T11:30:41.068 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.068 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.068 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:41.068 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.009711+0000", 2024-03-19T11:30:41.068 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017196900000000001 2024-03-19T11:30:41.069 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.069 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.069 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:41.069 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.009886+0000", 2024-03-19T11:30:41.069 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017477500000000001 2024-03-19T11:30:41.069 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.069 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:41.069 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.069 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.069 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.069 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1637 4.7 4:e41d47fb:::smithi03829856-14:head [write 484068~792524,stat] snapc 10=[10,f,e,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-03-19T11:30:41.069 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:35.672719+0000", 2024-03-19T11:30:41.069 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.3054490960000003, 2024-03-19T11:30:41.069 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012961324, 2024-03-19T11:30:41.069 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:41.070 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:41.070 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:41.070 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:41.070 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:41.070 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1637 2024-03-19T11:30:41.070 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.070 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:41.070 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.070 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:41.070 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.672719+0000", 2024-03-19T11:30:41.070 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.070 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.070 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.070 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:41.071 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.672719+0000", 2024-03-19T11:30:41.071 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.071 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.071 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.071 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:41.071 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.672722+0000", 2024-03-19T11:30:41.071 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0690000000000002e-06 2024-03-19T11:30:41.071 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.071 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.071 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:41.071 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.673743+0000", 2024-03-19T11:30:41.071 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010203899999999999 2024-03-19T11:30:41.071 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.071 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.072 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:41.072 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.673774+0000", 2024-03-19T11:30:41.072 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1049e-05 2024-03-19T11:30:41.072 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.072 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.072 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:41.072 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.673789+0000", 2024-03-19T11:30:41.072 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4729000000000001e-05 2024-03-19T11:30:41.072 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.072 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.072 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:41.072 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.673921+0000", 2024-03-19T11:30:41.072 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013277100000000001 2024-03-19T11:30:41.072 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.072 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:41.073 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.674053+0000", 2024-03-19T11:30:41.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000131975 2024-03-19T11:30:41.073 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:41.073 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.674430+0000", 2024-03-19T11:30:41.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000377031 2024-03-19T11:30:41.073 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:41.073 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.684295+0000", 2024-03-19T11:30:41.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0098643120000000001 2024-03-19T11:30:41.073 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.074 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:41.074 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.685359+0000", 2024-03-19T11:30:41.074 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001064111 2024-03-19T11:30:41.074 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.074 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.074 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:41.074 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.685532+0000", 2024-03-19T11:30:41.074 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017305700000000001 2024-03-19T11:30:41.074 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.074 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.074 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:41.074 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.685681+0000", 2024-03-19T11:30:41.074 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014883 2024-03-19T11:30:41.075 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.075 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:41.075 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.075 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.075 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:41.075 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:41.077 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-19T11:30:41.077 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:41.084+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:41.079 INFO:tasks.rados.rados.0.smithi038.stdout:981: writing smithi03829856-12 from 786594 to 1566306 tid 1 2024-03-19T11:30:41.079 INFO:tasks.rados.rados.0.smithi038.stdout:981: writing smithi03829856-12 from 2326443 to 2386594 tid 2 2024-03-19T11:30:41.079 INFO:tasks.rados.rados.0.smithi038.stdout:982: set_chunk oid 44 target oid 2024-03-19T11:30:41.079 INFO:tasks.rados.rados.0.smithi038.stdout:982: set_chunk oid 44 offset: 121856 length: 1858048 target oid 3dbe117da3ff580b0afc9f48e0d0278c54684de5a70b535da0443aaa5992fc19 offset: 0 snap -1 2024-03-19T11:30:41.079 INFO:tasks.rados.rados.0.smithi038.stdout:983: write (excl) oid 46 current snap is 15 2024-03-19T11:30:41.079 INFO:tasks.rados.rados.0.smithi038.stdout:979: finishing rollback tid 0 to smithi03829856-8 2024-03-19T11:30:41.079 INFO:tasks.rados.rados.0.smithi038.stdout:979: finishing rollback tid 1 to smithi03829856-8 2024-03-19T11:30:41.079 INFO:tasks.rados.rados.0.smithi038.stdout:977: finishing write tid 2 to smithi03829856-40 2024-03-19T11:30:41.079 INFO:tasks.rados.rados.0.smithi038.stdout:977: finishing write tid 3 to smithi03829856-40 2024-03-19T11:30:41.079 INFO:tasks.rados.rados.0.smithi038.stdout:983: seq_num 143 ranges {403401=454961,1584925=506039} 2024-03-19T11:30:41.079 INFO:tasks.rados.rados.0.smithi038.stdout:983: writing smithi03829856-46 from 403401 to 858362 tid 1 2024-03-19T11:30:41.079 INFO:tasks.rados.rados.0.smithi038.stdout:983: writing smithi03829856-46 from 1584925 to 2090964 tid 2 2024-03-19T11:30:41.080 INFO:tasks.rados.rados.0.smithi038.stdout:984: read oid 9 snap 4 2024-03-19T11:30:41.080 INFO:tasks.rados.rados.0.smithi038.stdout:984: expect (ObjNum 8 snap 0 seq_num 8) 2024-03-19T11:30:41.080 INFO:tasks.rados.rados.0.smithi038.stdout:985: read oid 29 snap -1 2024-03-19T11:30:41.080 INFO:tasks.rados.rados.0.smithi038.stdout:985: expect (ObjNum 95 snap 10 seq_num 95) 2024-03-19T11:30:41.080 INFO:tasks.rados.rados.0.smithi038.stdout:986: write (excl) oid 10 current snap is 15 2024-03-19T11:30:41.080 INFO:tasks.rados.rados.0.smithi038.stdout:986: seq_num 144 ranges {420208=530210,1527054=766050} 2024-03-19T11:30:41.080 INFO:tasks.rados.rados.0.smithi038.stdout:986: writing smithi03829856-10 from 420208 to 950418 tid 1 2024-03-19T11:30:41.080 INFO:tasks.rados.rados.0.smithi038.stdout:986: writing smithi03829856-10 from 1527054 to 2293104 tid 2 2024-03-19T11:30:41.080 INFO:tasks.rados.rados.0.smithi038.stdout:987: read oid 28 snap -1 2024-03-19T11:30:41.080 INFO:tasks.rados.rados.0.smithi038.stdout:987: expect (ObjNum 87 snap 8 seq_num 87) 2024-03-19T11:30:41.080 INFO:tasks.rados.rados.0.smithi038.stdout:988: delete oid 6 current snap is 15 2024-03-19T11:30:41.080 INFO:tasks.rados.rados.0.smithi038.stdout:979: finishing rollback tid 2 to smithi03829856-8 2024-03-19T11:30:41.080 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 8 v 96 (ObjNum 60 snap 3 seq_num 60) dirty exists 2024-03-19T11:30:41.080 INFO:tasks.rados.rados.0.smithi038.stdout:977: finishing write tid 4 to smithi03829856-40 2024-03-19T11:30:41.080 INFO:tasks.rados.rados.0.smithi038.stdout:977: oid 40 updating version 0 to 60 2024-03-19T11:30:41.081 INFO:tasks.rados.rados.0.smithi038.stdout:977: oid 40 version 60 is already newer than 59 2024-03-19T11:30:41.081 INFO:tasks.rados.rados.0.smithi038.stdout:977: oid 40 updating version 60 to 61 2024-03-19T11:30:41.081 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 40 v 61 (ObjNum 141 snap 15 seq_num 141) dirty exists 2024-03-19T11:30:41.081 INFO:tasks.rados.rados.0.smithi038.stdout:977: left oid 40 (ObjNum 141 snap 15 seq_num 141) 2024-03-19T11:30:41.081 INFO:tasks.rados.rados.0.smithi038.stdout:980: got (0) Success 2024-03-19T11:30:41.081 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 32 v 6 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-03-19T11:30:41.081 INFO:tasks.rados.rados.0.smithi038.stdout:978: expect (ObjNum 50 snap 0 seq_num 50) 2024-03-19T11:30:41.148 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:41.156+0000 7fb17dd72640 -1 osd.4 62 handle_osd_map injecting map crc failure 2024-03-19T11:30:41.157 INFO:teuthology.orchestra.run.smithi038.stderr:set pool 4 pg_num to 46 2024-03-19T11:30:41.172 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-03-19T11:30:41.177 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:41.184+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:41.223 INFO:tasks.rados.rados.0.smithi038.stdout:981: finishing write tid 1 to smithi03829856-12 2024-03-19T11:30:41.223 INFO:tasks.rados.rados.0.smithi038.stdout:981: finishing write tid 2 to smithi03829856-12 2024-03-19T11:30:41.223 INFO:tasks.rados.rados.0.smithi038.stdout:981: finishing write tid 3 to smithi03829856-12 2024-03-19T11:30:41.223 INFO:tasks.rados.rados.0.smithi038.stdout:981: finishing write tid 4 to smithi03829856-12 2024-03-19T11:30:41.223 INFO:tasks.rados.rados.0.smithi038.stdout:981: oid 12 updating version 0 to 74 2024-03-19T11:30:41.223 INFO:tasks.rados.rados.0.smithi038.stdout:981: oid 12 updating version 74 to 76 2024-03-19T11:30:41.223 INFO:tasks.rados.rados.0.smithi038.stdout:981: oid 12 version 76 is already newer than 75 2024-03-19T11:30:41.223 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 12 v 76 (ObjNum 142 snap 15 seq_num 142) dirty exists 2024-03-19T11:30:41.223 INFO:tasks.rados.rados.0.smithi038.stdout:981: left oid 12 (ObjNum 142 snap 15 seq_num 142) 2024-03-19T11:30:41.223 INFO:tasks.rados.rados.0.smithi038.stdout:982: finishing set_chunk to oid 44 2024-03-19T11:30:41.223 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 44 v 50 (ObjNum 108 snap 11 seq_num 108) dirty exists 2024-03-19T11:30:41.223 INFO:tasks.rados.rados.0.smithi038.stdout:983: finishing write tid 1 to smithi03829856-46 2024-03-19T11:30:41.224 INFO:tasks.rados.rados.0.smithi038.stdout:983: finishing write tid 2 to smithi03829856-46 2024-03-19T11:30:41.224 INFO:tasks.rados.rados.0.smithi038.stdout:983: finishing write tid 3 to smithi03829856-46 2024-03-19T11:30:41.224 INFO:tasks.rados.rados.0.smithi038.stdout:977: done (11 left) 2024-03-19T11:30:41.224 INFO:tasks.rados.rados.0.smithi038.stdout:978: done (10 left) 2024-03-19T11:30:41.224 INFO:tasks.rados.rados.0.smithi038.stdout:979: done (9 left) 2024-03-19T11:30:41.224 INFO:tasks.rados.rados.0.smithi038.stdout:980: done (8 left) 2024-03-19T11:30:41.224 INFO:tasks.rados.rados.0.smithi038.stdout:981: done (7 left) 2024-03-19T11:30:41.224 INFO:tasks.rados.rados.0.smithi038.stdout:982: done (6 left) 2024-03-19T11:30:41.224 INFO:tasks.rados.rados.0.smithi038.stdout:988: done (5 left) 2024-03-19T11:30:41.224 INFO:tasks.rados.rados.0.smithi038.stdout:989: tier_evict oid 15 2024-03-19T11:30:41.224 INFO:tasks.rados.rados.0.smithi038.stdout:989: tier_evict oid 15 snap -1 2024-03-19T11:30:41.224 INFO:tasks.rados.rados.0.smithi038.stdout:983: finishing write tid 4 to smithi03829856-46 2024-03-19T11:30:41.224 INFO:tasks.rados.rados.0.smithi038.stdout:983: oid 46 updating version 0 to 95 2024-03-19T11:30:41.224 INFO:tasks.rados.rados.0.smithi038.stdout:983: oid 46 version 95 is already newer than 94 2024-03-19T11:30:41.225 INFO:tasks.rados.rados.0.smithi038.stdout:983: oid 46 updating version 95 to 96 2024-03-19T11:30:41.225 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 46 v 96 (ObjNum 143 snap 15 seq_num 143) dirty exists 2024-03-19T11:30:41.225 INFO:tasks.rados.rados.0.smithi038.stdout:983: left oid 46 (ObjNum 143 snap 15 seq_num 143) 2024-03-19T11:30:41.225 INFO:tasks.rados.rados.0.smithi038.stdout:984: expect (ObjNum 8 snap 0 seq_num 8) 2024-03-19T11:30:41.250 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:41.250 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:41.250 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:30:41.250 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:41.264 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-19T11:30:41.277 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:41.286+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:41.291 INFO:tasks.rados.rados.0.smithi038.stdout:985: expect (ObjNum 95 snap 10 seq_num 95) 2024-03-19T11:30:41.379 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:41.386+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:41.435 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:41.435 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:41.435 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:30:41.435 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:30:41.435 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:41.442 INFO:tasks.rados.rados.0.smithi038.stdout:986: finishing write tid 1 to smithi03829856-10 2024-03-19T11:30:41.442 INFO:tasks.rados.rados.0.smithi038.stdout:987: expect (ObjNum 87 snap 8 seq_num 87) 2024-03-19T11:30:41.449 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-19T11:30:41.479 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:41.490+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:41.550 INFO:tasks.rados.rados.0.smithi038.stdout:986: finishing write tid 2 to smithi03829856-10 2024-03-19T11:30:41.550 INFO:tasks.rados.rados.0.smithi038.stdout:986: finishing write tid 3 to smithi03829856-10 2024-03-19T11:30:41.550 INFO:tasks.rados.rados.0.smithi038.stdout:986: finishing write tid 4 to smithi03829856-10 2024-03-19T11:30:41.550 INFO:tasks.rados.rados.0.smithi038.stdout:986: oid 10 updating version 0 to 73 2024-03-19T11:30:41.550 INFO:tasks.rados.rados.0.smithi038.stdout:986: oid 10 version 73 is already newer than 71 2024-03-19T11:30:41.550 INFO:tasks.rados.rados.0.smithi038.stdout:986: oid 10 version 73 is already newer than 72 2024-03-19T11:30:41.550 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 10 v 73 (ObjNum 144 snap 15 seq_num 144) dirty exists 2024-03-19T11:30:41.551 INFO:tasks.rados.rados.0.smithi038.stdout:986: left oid 10 (ObjNum 144 snap 15 seq_num 144) 2024-03-19T11:30:41.551 INFO:tasks.rados.rados.0.smithi038.stdout:989: got (22) Invalid argument 2024-03-19T11:30:41.551 INFO:tasks.rados.rados.0.smithi038.stdout:983: done (5 left) 2024-03-19T11:30:41.551 INFO:tasks.rados.rados.0.smithi038.stdout:984: done (4 left) 2024-03-19T11:30:41.551 INFO:tasks.rados.rados.0.smithi038.stdout:985: done (3 left) 2024-03-19T11:30:41.551 INFO:tasks.rados.rados.0.smithi038.stdout:986: done (2 left) 2024-03-19T11:30:41.551 INFO:tasks.rados.rados.0.smithi038.stdout:987: done (1 left) 2024-03-19T11:30:41.551 INFO:tasks.rados.rados.0.smithi038.stdout:989: done (0 left) 2024-03-19T11:30:41.551 INFO:tasks.rados.rados.0.smithi038.stdout:990: write oid 11 current snap is 15 2024-03-19T11:30:41.551 INFO:tasks.rados.rados.0.smithi038.stdout:990: seq_num 146 ranges {453822=797061,1760371=653366} 2024-03-19T11:30:41.570 INFO:tasks.rados.rados.0.smithi038.stdout:990: writing smithi03829856-11 from 453822 to 1250883 tid 1 2024-03-19T11:30:41.575 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:41.575 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:30:41.575 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:30:41.575 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:30:41.575 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.576 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:29 4.5 4:ac9cd3a3:::smithi03829856-8:head [write 517649~526743,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:30:41.576 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:04.772849+0000", 2024-03-19T11:30:41.576 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 36.810031510999998, 2024-03-19T11:30:41.576 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014076821, 2024-03-19T11:30:41.576 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:41.576 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:41.576 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:41.576 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:41.576 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:41.576 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 29 2024-03-19T11:30:41.577 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.577 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:41.577 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.577 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:41.577 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.772849+0000", 2024-03-19T11:30:41.577 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.577 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.577 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.577 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:41.577 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.772849+0000", 2024-03-19T11:30:41.578 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.578 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.578 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.578 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:41.578 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.772852+0000", 2024-03-19T11:30:41.578 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.79e-06 2024-03-19T11:30:41.578 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.578 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.578 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:41.578 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.773783+0000", 2024-03-19T11:30:41.578 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00093104199999999998 2024-03-19T11:30:41.578 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.579 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:41.579 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.773810+0000", 2024-03-19T11:30:41.579 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.669e-05 2024-03-19T11:30:41.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.579 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:41.579 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.773826+0000", 2024-03-19T11:30:41.579 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5739999999999998e-05 2024-03-19T11:30:41.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.579 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:41.579 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.773954+0000", 2024-03-19T11:30:41.580 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012796300000000001 2024-03-19T11:30:41.580 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.580 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.580 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op must wait for map", 2024-03-19T11:30:41.580 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.773965+0000", 2024-03-19T11:30:41.580 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1453e-05 2024-03-19T11:30:41.580 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.580 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.580 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:41.580 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.777723+0000", 2024-03-19T11:30:41.580 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003757944 2024-03-19T11:30:41.580 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.580 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.580 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:41.580 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778138+0000", 2024-03-19T11:30:41.581 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00041454699999999997 2024-03-19T11:30:41.581 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.581 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:30:41.581 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778409+0000", 2024-03-19T11:30:41.581 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027111700000000003 2024-03-19T11:30:41.581 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.581 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:41.581 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.786631+0000", 2024-03-19T11:30:41.581 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0082221470000000008 2024-03-19T11:30:41.581 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.581 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:41.581 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.786668+0000", 2024-03-19T11:30:41.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.718e-05 2024-03-19T11:30:41.582 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.582 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:41.582 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.786845+0000", 2024-03-19T11:30:41.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000177219 2024-03-19T11:30:41.582 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.582 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:41.582 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.786926+0000", 2024-03-19T11:30:41.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0989000000000007e-05 2024-03-19T11:30:41.582 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.582 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:41.582 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.583 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:30 4.5 4:ac9cd3a3:::smithi03829856-8:head [write 1776668~340981] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:30:41.583 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:04.777922+0000", 2024-03-19T11:30:41.583 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 36.804958880999997, 2024-03-19T11:30:41.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015469894, 2024-03-19T11:30:41.583 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:41.583 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:41.583 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:41.583 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:41.584 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:41.584 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 30 2024-03-19T11:30:41.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.584 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:41.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.584 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:41.584 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.777922+0000", 2024-03-19T11:30:41.584 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.584 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:41.584 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.777922+0000", 2024-03-19T11:30:41.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.585 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.585 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:41.585 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.777924+0000", 2024-03-19T11:30:41.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3999999999999999e-06 2024-03-19T11:30:41.585 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.585 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:41.585 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778461+0000", 2024-03-19T11:30:41.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00053698299999999999 2024-03-19T11:30:41.586 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.586 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:41.586 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778487+0000", 2024-03-19T11:30:41.586 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5947000000000001e-05 2024-03-19T11:30:41.586 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.586 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:41.586 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778502+0000", 2024-03-19T11:30:41.586 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4256e-05 2024-03-19T11:30:41.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.587 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:41.587 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.781978+0000", 2024-03-19T11:30:41.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00347693 2024-03-19T11:30:41.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.588 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:41.589 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.782196+0000", 2024-03-19T11:30:41.589 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021726499999999999 2024-03-19T11:30:41.589 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.589 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:30:41.589 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.782572+0000", 2024-03-19T11:30:41.589 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00037635000000000001 2024-03-19T11:30:41.590 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.590 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.590 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:41.590 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.786960+0000", 2024-03-19T11:30:41.591 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043882130000000002 2024-03-19T11:30:41.591 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.592 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:41.592 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793113+0000", 2024-03-19T11:30:41.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061530760000000004 2024-03-19T11:30:41.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.592 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.592 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:41.592 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793276+0000", 2024-03-19T11:30:41.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000162652 2024-03-19T11:30:41.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.592 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.592 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:41.593 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793392+0000", 2024-03-19T11:30:41.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115822 2024-03-19T11:30:41.593 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.593 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:41.593 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.593 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.593 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.593 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:31 4.5 4:ac9cd3a3:::smithi03829856-8:head [setxattr _header (55) in=7b,truncate 2117649] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:30:41.593 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:04.778533+0000", 2024-03-19T11:30:41.593 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 36.804347745999998, 2024-03-19T11:30:41.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015182533, 2024-03-19T11:30:41.593 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:41.593 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:41.593 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:41.594 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:41.594 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:41.594 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 31 2024-03-19T11:30:41.594 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.594 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:41.594 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.594 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:41.594 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778533+0000", 2024-03-19T11:30:41.594 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.594 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.594 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.594 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:41.594 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778533+0000", 2024-03-19T11:30:41.594 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.595 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.595 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.595 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:41.595 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778534+0000", 2024-03-19T11:30:41.595 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1069999999999999e-06 2024-03-19T11:30:41.595 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.595 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.595 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:41.595 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778541+0000", 2024-03-19T11:30:41.595 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2529999999999998e-06 2024-03-19T11:30:41.595 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.595 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.595 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:41.595 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778558+0000", 2024-03-19T11:30:41.595 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6290000000000002e-05 2024-03-19T11:30:41.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:41.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778568+0000", 2024-03-19T11:30:41.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0360000000000001e-05 2024-03-19T11:30:41.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:41.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.785511+0000", 2024-03-19T11:30:41.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00694306 2024-03-19T11:30:41.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:41.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.785710+0000", 2024-03-19T11:30:41.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019837199999999999 2024-03-19T11:30:41.597 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.597 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:30:41.597 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.785929+0000", 2024-03-19T11:30:41.597 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021960799999999999 2024-03-19T11:30:41.597 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.597 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:41.598 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.787281+0000", 2024-03-19T11:30:41.598 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001352313 2024-03-19T11:30:41.598 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.598 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.598 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:41.598 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793492+0000", 2024-03-19T11:30:41.598 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062107760000000003 2024-03-19T11:30:41.598 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.598 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.598 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:41.598 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793578+0000", 2024-03-19T11:30:41.598 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5711999999999995e-05 2024-03-19T11:30:41.598 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.598 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.598 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:41.599 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793716+0000", 2024-03-19T11:30:41.599 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000137682 2024-03-19T11:30:41.599 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.599 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:41.599 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.599 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.599 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.599 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:32 4.5 4:ac9cd3a3:::smithi03829856-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:30:41.599 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:04.778582+0000", 2024-03-19T11:30:41.599 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 36.804299231999998, 2024-03-19T11:30:41.599 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015761827999999999, 2024-03-19T11:30:41.599 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:41.599 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:41.599 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:41.599 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:41.600 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:41.600 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 32 2024-03-19T11:30:41.600 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.600 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:41.600 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.600 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:41.600 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778582+0000", 2024-03-19T11:30:41.600 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.600 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.600 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.600 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:41.600 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778582+0000", 2024-03-19T11:30:41.600 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.600 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.600 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.600 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:41.601 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778583+0000", 2024-03-19T11:30:41.601 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.026e-06 2024-03-19T11:30:41.601 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.601 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.601 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:41.601 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778587+0000", 2024-03-19T11:30:41.601 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7870000000000001e-06 2024-03-19T11:30:41.601 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.601 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.601 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:41.601 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778599+0000", 2024-03-19T11:30:41.601 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1623e-05 2024-03-19T11:30:41.601 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.601 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.602 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:41.602 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778608+0000", 2024-03-19T11:30:41.602 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7399999999999993e-06 2024-03-19T11:30:41.602 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.602 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.602 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:41.602 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.787009+0000", 2024-03-19T11:30:41.602 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0084010490000000007 2024-03-19T11:30:41.602 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.602 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.602 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-03-19T11:30:41.602 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.787202+0000", 2024-03-19T11:30:41.602 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019309500000000001 2024-03-19T11:30:41.602 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.603 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.603 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:41.603 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793832+0000", 2024-03-19T11:30:41.603 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.006630052 2024-03-19T11:30:41.603 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.603 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.603 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:41.603 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.794048+0000", 2024-03-19T11:30:41.603 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000215505 2024-03-19T11:30:41.603 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.603 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.603 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:41.603 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.794343+0000", 2024-03-19T11:30:41.604 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029595100000000003 2024-03-19T11:30:41.604 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.604 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:41.604 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.604 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.604 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.604 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:856 4.5 4:ac9cd3a3:::smithi03829856-8:head [write 608420~529411] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:41.604 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.781425+0000", 2024-03-19T11:30:41.604 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.801456045999998, 2024-03-19T11:30:41.604 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014634677, 2024-03-19T11:30:41.604 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:41.604 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:41.605 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:41.605 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:41.605 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:41.605 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 856 2024-03-19T11:30:41.605 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.605 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:41.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.605 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:41.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.781425+0000", 2024-03-19T11:30:41.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.605 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.605 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:41.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.781425+0000", 2024-03-19T11:30:41.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.606 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:41.606 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.781428+0000", 2024-03-19T11:30:41.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.534e-06 2024-03-19T11:30:41.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.606 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:41.606 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.783096+0000", 2024-03-19T11:30:41.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001667897 2024-03-19T11:30:41.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.606 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:41.606 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.783138+0000", 2024-03-19T11:30:41.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1946000000000001e-05 2024-03-19T11:30:41.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:41.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.783163+0000", 2024-03-19T11:30:41.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4443e-05 2024-03-19T11:30:41.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:41.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.783333+0000", 2024-03-19T11:30:41.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017027899999999999 2024-03-19T11:30:41.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.608 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:41.608 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.783503+0000", 2024-03-19T11:30:41.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000170242 2024-03-19T11:30:41.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.608 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:30:41.608 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.783917+0000", 2024-03-19T11:30:41.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00041371000000000002 2024-03-19T11:30:41.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.608 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:41.609 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.795709+0000", 2024-03-19T11:30:41.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011791693000000001 2024-03-19T11:30:41.609 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.609 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:41.609 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.795850+0000", 2024-03-19T11:30:41.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000141695 2024-03-19T11:30:41.609 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.609 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:41.609 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.795940+0000", 2024-03-19T11:30:41.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9705999999999998e-05 2024-03-19T11:30:41.609 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.609 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:41.610 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.796059+0000", 2024-03-19T11:30:41.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011953199999999999 2024-03-19T11:30:41.610 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.610 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:41.610 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.610 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.610 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:891 4.5 4:af9d0d56:::smithi03829856-20:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:41.610 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:09.286410+0000", 2024-03-19T11:30:41.610 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.296470900000003, 2024-03-19T11:30:41.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035191405000000002, 2024-03-19T11:30:41.610 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:41.610 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:41.610 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:41.611 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:41.611 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:41.611 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 891 2024-03-19T11:30:41.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.611 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:41.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:41.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286410+0000", 2024-03-19T11:30:41.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:41.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286410+0000", 2024-03-19T11:30:41.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:41.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286413+0000", 2024-03-19T11:30:41.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2660000000000002e-06 2024-03-19T11:30:41.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:41.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286426+0000", 2024-03-19T11:30:41.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.258e-05 2024-03-19T11:30:41.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:41.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286460+0000", 2024-03-19T11:30:41.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3943999999999999e-05 2024-03-19T11:30:41.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.613 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:41.613 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286477+0000", 2024-03-19T11:30:41.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6799999999999998e-05 2024-03-19T11:30:41.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.613 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:41.613 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286605+0000", 2024-03-19T11:30:41.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000128892 2024-03-19T11:30:41.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.613 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:41.613 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286769+0000", 2024-03-19T11:30:41.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016359799999999999 2024-03-19T11:30:41.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.614 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:41.614 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.321601+0000", 2024-03-19T11:30:41.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.034832324999999997 2024-03-19T11:30:41.614 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.614 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:41.614 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.614 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:993 4.e e34/21 4:72163e4c:::smithi03829856-39:head v 34'17, mlcod=28'14)", 2024-03-19T11:30:41.615 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:12.367897+0000", 2024-03-19T11:30:41.615 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.214984353999998, 2024-03-19T11:30:41.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012765165, 2024-03-19T11:30:41.615 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:41.615 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:41.615 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:41.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:41.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.367897+0000", 2024-03-19T11:30:41.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:41.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.367897+0000", 2024-03-19T11:30:41.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:41.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.367897+0000", 2024-03-19T11:30:41.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7000000000000003e-07 2024-03-19T11:30:41.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:41.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.368454+0000", 2024-03-19T11:30:41.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00055647599999999998 2024-03-19T11:30:41.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.617 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:41.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.368532+0000", 2024-03-19T11:30:41.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8479000000000005e-05 2024-03-19T11:30:41.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.617 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:41.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.368554+0000", 2024-03-19T11:30:41.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.156e-05 2024-03-19T11:30:41.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.618 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:41.618 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.368742+0000", 2024-03-19T11:30:41.618 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000187846 2024-03-19T11:30:41.618 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.618 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:41.618 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.368782+0000", 2024-03-19T11:30:41.618 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9836000000000003e-05 2024-03-19T11:30:41.618 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.618 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:41.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.380340+0000", 2024-03-19T11:30:41.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011558847000000001 2024-03-19T11:30:41.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:41.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.380662+0000", 2024-03-19T11:30:41.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00032125099999999999 2024-03-19T11:30:41.619 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.619 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:41.619 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.619 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1015 4.5 4:af9d0d56:::smithi03829856-20:head [tier-promote] snapc 6=[6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:41.619 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:12.761855+0000", 2024-03-19T11:30:41.619 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.821025374000001, 2024-03-19T11:30:41.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026760430000000002, 2024-03-19T11:30:41.620 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:41.620 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:41.620 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:41.620 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:41.620 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:41.620 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1015 2024-03-19T11:30:41.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.620 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:41.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:41.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761855+0000", 2024-03-19T11:30:41.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:41.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761855+0000", 2024-03-19T11:30:41.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:41.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761858+0000", 2024-03-19T11:30:41.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8140000000000001e-06 2024-03-19T11:30:41.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:41.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761872+0000", 2024-03-19T11:30:41.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3813e-05 2024-03-19T11:30:41.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.622 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:41.622 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761905+0000", 2024-03-19T11:30:41.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2642999999999999e-05 2024-03-19T11:30:41.622 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.622 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.622 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:41.622 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761925+0000", 2024-03-19T11:30:41.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0489e-05 2024-03-19T11:30:41.622 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.622 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.622 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:41.622 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.762070+0000", 2024-03-19T11:30:41.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000145153 2024-03-19T11:30:41.622 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:41.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.762365+0000", 2024-03-19T11:30:41.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029482100000000002 2024-03-19T11:30:41.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:41.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.788616+0000", 2024-03-19T11:30:41.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026250697 2024-03-19T11:30:41.623 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.623 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:41.623 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.624 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1121 4.5 4:aa3303a5:::smithi03829856-17:head [tier-promote] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:41.624 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:14.164281+0000", 2024-03-19T11:30:41.624 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.418600129000001, 2024-03-19T11:30:41.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023896435000000001, 2024-03-19T11:30:41.624 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:41.624 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:41.624 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:41.624 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:41.624 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:41.624 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1121 2024-03-19T11:30:41.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.624 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:41.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:41.624 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164281+0000", 2024-03-19T11:30:41.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.625 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:41.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164281+0000", 2024-03-19T11:30:41.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.625 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:41.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164283+0000", 2024-03-19T11:30:41.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8500000000000001e-06 2024-03-19T11:30:41.625 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:41.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164291+0000", 2024-03-19T11:30:41.626 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.14e-06 2024-03-19T11:30:41.626 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.626 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:41.626 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164313+0000", 2024-03-19T11:30:41.626 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.207e-05 2024-03-19T11:30:41.626 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.626 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:41.626 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164328+0000", 2024-03-19T11:30:41.626 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5403000000000001e-05 2024-03-19T11:30:41.626 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.626 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:41.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.167981+0000", 2024-03-19T11:30:41.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036529879999999998 2024-03-19T11:30:41.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:41.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.168274+0000", 2024-03-19T11:30:41.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029313100000000002 2024-03-19T11:30:41.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:41.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.188177+0000", 2024-03-19T11:30:41.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019902853000000002 2024-03-19T11:30:41.627 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.627 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:41.627 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.628 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1154 4.5 4:a0a50d04:::smithi03829856-13:8 [sparse-read 0~1151947] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-03-19T11:30:41.628 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.827529+0000", 2024-03-19T11:30:41.628 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.755351710999999, 2024-03-19T11:30:41.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013620382, 2024-03-19T11:30:41.628 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:41.628 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:41.628 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:41.629 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:41.629 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:41.629 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1154 2024-03-19T11:30:41.629 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.629 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:41.629 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.629 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:41.629 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827529+0000", 2024-03-19T11:30:41.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.629 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.629 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.629 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:41.629 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827529+0000", 2024-03-19T11:30:41.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.629 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.630 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:41.630 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827530+0000", 2024-03-19T11:30:41.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7700000000000004e-07 2024-03-19T11:30:41.630 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.630 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:41.630 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827533+0000", 2024-03-19T11:30:41.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4520000000000002e-06 2024-03-19T11:30:41.630 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.630 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:41.630 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827543+0000", 2024-03-19T11:30:41.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0035e-05 2024-03-19T11:30:41.630 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.631 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:41.631 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827551+0000", 2024-03-19T11:30:41.631 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7049999999999999e-06 2024-03-19T11:30:41.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.631 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:41.631 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.839931+0000", 2024-03-19T11:30:41.631 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012379602999999999 2024-03-19T11:30:41.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.631 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:41.631 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.840290+0000", 2024-03-19T11:30:41.631 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000359031 2024-03-19T11:30:41.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:41.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841150+0000", 2024-03-19T11:30:41.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00085977899999999997 2024-03-19T11:30:41.632 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.632 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:41.632 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.632 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1302 4.15 4:aa3303a5:::smithi03829856-17:head [write 413472~620140,stat] snapc b=[b,a,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-03-19T11:30:41.632 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:22.966426+0000", 2024-03-19T11:30:41.632 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.616454396000002, 2024-03-19T11:30:41.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018043513000000001, 2024-03-19T11:30:41.632 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:41.633 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:41.633 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:41.633 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:41.633 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:41.633 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1302 2024-03-19T11:30:41.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.633 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:41.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:41.634 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.966426+0000", 2024-03-19T11:30:41.634 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.634 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.634 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.634 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:41.634 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.966426+0000", 2024-03-19T11:30:41.634 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.634 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.634 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.634 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:41.634 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.966429+0000", 2024-03-19T11:30:41.634 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3829999999999999e-06 2024-03-19T11:30:41.634 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.635 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:41.635 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972117+0000", 2024-03-19T11:30:41.635 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0056885750000000004 2024-03-19T11:30:41.635 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.635 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:41.635 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972156+0000", 2024-03-19T11:30:41.635 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8878000000000003e-05 2024-03-19T11:30:41.635 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.635 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:41.635 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972184+0000", 2024-03-19T11:30:41.635 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7509999999999999e-05 2024-03-19T11:30:41.635 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:41.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972309+0000", 2024-03-19T11:30:41.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012525899999999999 2024-03-19T11:30:41.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:41.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972566+0000", 2024-03-19T11:30:41.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025740799999999999 2024-03-19T11:30:41.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:30:41.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972899+0000", 2024-03-19T11:30:41.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033238799999999998 2024-03-19T11:30:41.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.637 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:41.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984088+0000", 2024-03-19T11:30:41.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011189466 2024-03-19T11:30:41.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.637 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:41.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984138+0000", 2024-03-19T11:30:41.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9666999999999997e-05 2024-03-19T11:30:41.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.637 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:41.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984329+0000", 2024-03-19T11:30:41.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019086799999999999 2024-03-19T11:30:41.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.638 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.638 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:41.638 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984470+0000", 2024-03-19T11:30:41.638 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000141111 2024-03-19T11:30:41.638 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.638 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:41.638 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.638 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.638 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.638 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1303 4.15 4:aa3303a5:::smithi03829856-17:head [write 1818036~510700] snapc b=[b,a,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-03-19T11:30:41.638 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:22.976856+0000", 2024-03-19T11:30:41.638 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.606025286000001, 2024-03-19T11:30:41.638 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015045703000000001, 2024-03-19T11:30:41.639 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:41.639 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:41.639 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:41.639 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:41.639 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:41.639 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1303 2024-03-19T11:30:41.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.639 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:41.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.639 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:41.639 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.976856+0000", 2024-03-19T11:30:41.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.639 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:41.640 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.976856+0000", 2024-03-19T11:30:41.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:41.640 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.976859+0000", 2024-03-19T11:30:41.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3629999999999998e-06 2024-03-19T11:30:41.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:41.640 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.977533+0000", 2024-03-19T11:30:41.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00067380800000000002 2024-03-19T11:30:41.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:41.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.977573+0000", 2024-03-19T11:30:41.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9758000000000001e-05 2024-03-19T11:30:41.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.641 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:41.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.977602+0000", 2024-03-19T11:30:41.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9856e-05 2024-03-19T11:30:41.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.641 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:41.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.979943+0000", 2024-03-19T11:30:41.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0023406400000000002 2024-03-19T11:30:41.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.641 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:41.642 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.980238+0000", 2024-03-19T11:30:41.642 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029488000000000002 2024-03-19T11:30:41.642 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.642 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.642 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:30:41.642 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.980734+0000", 2024-03-19T11:30:41.642 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00049647500000000002 2024-03-19T11:30:41.642 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.642 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.642 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:41.642 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.991761+0000", 2024-03-19T11:30:41.642 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011026515000000001 2024-03-19T11:30:41.642 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.642 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.643 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:41.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.991777+0000", 2024-03-19T11:30:41.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5727999999999999e-05 2024-03-19T11:30:41.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.643 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:41.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.991840+0000", 2024-03-19T11:30:41.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3091999999999998e-05 2024-03-19T11:30:41.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.643 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:41.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.991901+0000", 2024-03-19T11:30:41.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1587999999999994e-05 2024-03-19T11:30:41.643 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.643 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:41.643 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.644 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1467 4.a 4:567251f3:::smithi03829856-44:head [write 615156~555834] snapc c=[c,b,a,6,5] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-03-19T11:30:41.644 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.079231+0000", 2024-03-19T11:30:41.644 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.503649754, 2024-03-19T11:30:41.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013571666, 2024-03-19T11:30:41.644 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:41.644 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:41.644 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:41.644 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:41.644 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:41.644 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1467 2024-03-19T11:30:41.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.644 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:41.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:41.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.079231+0000", 2024-03-19T11:30:41.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:41.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.079231+0000", 2024-03-19T11:30:41.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:41.646 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.079234+0000", 2024-03-19T11:30:41.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3689999999999998e-06 2024-03-19T11:30:41.646 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:41.646 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.080110+0000", 2024-03-19T11:30:41.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00087507100000000005 2024-03-19T11:30:41.646 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:41.646 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.080146+0000", 2024-03-19T11:30:41.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6062000000000001e-05 2024-03-19T11:30:41.646 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:41.647 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.080164+0000", 2024-03-19T11:30:41.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7995e-05 2024-03-19T11:30:41.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.647 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:41.647 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.080339+0000", 2024-03-19T11:30:41.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017535900000000001 2024-03-19T11:30:41.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.647 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:41.647 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.080630+0000", 2024-03-19T11:30:41.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029068900000000001 2024-03-19T11:30:41.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.647 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:41.648 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.081140+0000", 2024-03-19T11:30:41.648 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00051083699999999999 2024-03-19T11:30:41.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.648 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:41.648 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.092570+0000", 2024-03-19T11:30:41.648 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011429557 2024-03-19T11:30:41.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.648 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:41.648 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.092611+0000", 2024-03-19T11:30:41.648 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1196000000000003e-05 2024-03-19T11:30:41.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.648 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:41.649 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.092717+0000", 2024-03-19T11:30:41.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000105777 2024-03-19T11:30:41.649 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:41.649 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.092803+0000", 2024-03-19T11:30:41.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5754000000000006e-05 2024-03-19T11:30:41.649 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.649 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:41.649 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.649 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.649 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1468 4.a 4:567251f3:::smithi03829856-44:head [write 1603467~416361] snapc c=[c,b,a,6,5] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-03-19T11:30:41.649 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.083922+0000", 2024-03-19T11:30:41.649 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.498959263, 2024-03-19T11:30:41.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014508549000000001, 2024-03-19T11:30:41.650 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:41.650 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:41.650 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:41.650 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:41.650 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:41.650 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1468 2024-03-19T11:30:41.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.650 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:41.650 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.650 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:41.650 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.083922+0000", 2024-03-19T11:30:41.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.651 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:41.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.083922+0000", 2024-03-19T11:30:41.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.651 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:41.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.083924+0000", 2024-03-19T11:30:41.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2800000000000002e-06 2024-03-19T11:30:41.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.651 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:41.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.084498+0000", 2024-03-19T11:30:41.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057414900000000004 2024-03-19T11:30:41.652 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.652 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.652 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:41.652 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.084529+0000", 2024-03-19T11:30:41.652 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1080000000000001e-05 2024-03-19T11:30:41.652 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.652 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.652 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:41.652 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.084544+0000", 2024-03-19T11:30:41.652 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4498000000000001e-05 2024-03-19T11:30:41.652 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.652 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.652 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:41.652 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.088993+0000", 2024-03-19T11:30:41.652 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044497260000000002 2024-03-19T11:30:41.653 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.653 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.653 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:41.653 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.089152+0000", 2024-03-19T11:30:41.653 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015833199999999999 2024-03-19T11:30:41.653 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.653 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.653 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:41.653 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.089419+0000", 2024-03-19T11:30:41.653 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026691799999999999 2024-03-19T11:30:41.653 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.653 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.653 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:41.653 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.092883+0000", 2024-03-19T11:30:41.653 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034646590000000001 2024-03-19T11:30:41.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.654 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:41.654 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.098276+0000", 2024-03-19T11:30:41.654 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053924680000000001 2024-03-19T11:30:41.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.654 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:41.654 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.098354+0000", 2024-03-19T11:30:41.654 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8763999999999993e-05 2024-03-19T11:30:41.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.654 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:41.654 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.098430+0000", 2024-03-19T11:30:41.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5674999999999996e-05 2024-03-19T11:30:41.655 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.655 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:41.655 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.655 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.1 50/49 [PushOp(4:866f3dcc:::smithi03829856-31:7, version: 38'44, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@38'44, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 45'62, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@45'62, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 46'68, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@46'68, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:7, version: 47'71, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@47'71, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 47'72, data_included: [440350~816629,1309629~231947], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@47'72, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:30:41.655 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.959457+0000", 2024-03-19T11:30:41.655 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 13.623423979, 2024-03-19T11:30:41.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018416666000000002, 2024-03-19T11:30:41.655 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:41.655 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:41.655 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:41.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:41.656 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.959457+0000", 2024-03-19T11:30:41.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.656 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:41.656 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.959457+0000", 2024-03-19T11:30:41.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.656 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:41.656 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.959458+0000", 2024-03-19T11:30:41.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4899999999999999e-06 2024-03-19T11:30:41.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.656 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:41.657 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.962822+0000", 2024-03-19T11:30:41.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0033640620000000001 2024-03-19T11:30:41.657 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.657 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.657 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:41.657 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.962893+0000", 2024-03-19T11:30:41.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1060000000000001e-05 2024-03-19T11:30:41.657 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.657 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.657 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:41.657 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.962977+0000", 2024-03-19T11:30:41.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3311000000000002e-05 2024-03-19T11:30:41.657 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.657 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.657 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:41.657 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963208+0000", 2024-03-19T11:30:41.658 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023121200000000001 2024-03-19T11:30:41.658 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.658 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:41.658 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963245+0000", 2024-03-19T11:30:41.658 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6844999999999997e-05 2024-03-19T11:30:41.658 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.658 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:41.658 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.977874+0000", 2024-03-19T11:30:41.658 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014628686 2024-03-19T11:30:41.658 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.658 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:41.658 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.659 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.1 e50,49 4:8401c1f0:::smithi03829856-43:8,46'66,4:866f3dcc:::smithi03829856-31:2,47'73)", 2024-03-19T11:30:41.659 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.963027+0000", 2024-03-19T11:30:41.659 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 13.619854362, 2024-03-19T11:30:41.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015116788000000001, 2024-03-19T11:30:41.659 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:41.659 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:41.659 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:41.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:41.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963027+0000", 2024-03-19T11:30:41.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.660 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:41.660 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963027+0000", 2024-03-19T11:30:41.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.660 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:41.660 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963028+0000", 2024-03-19T11:30:41.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.268e-06 2024-03-19T11:30:41.661 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.661 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:41.661 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963036+0000", 2024-03-19T11:30:41.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.926e-06 2024-03-19T11:30:41.661 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.661 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:41.661 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963050+0000", 2024-03-19T11:30:41.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4082e-05 2024-03-19T11:30:41.661 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.661 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:41.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963062+0000", 2024-03-19T11:30:41.662 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2415000000000001e-05 2024-03-19T11:30:41.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.662 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:41.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.977954+0000", 2024-03-19T11:30:41.662 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014891572 2024-03-19T11:30:41.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.662 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:41.663 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.977965+0000", 2024-03-19T11:30:41.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1654000000000001e-05 2024-03-19T11:30:41.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.663 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:41.663 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.978143+0000", 2024-03-19T11:30:41.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017787099999999999 2024-03-19T11:30:41.663 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.663 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:41.663 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.663 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.11 50/49 [PushOp(4:8d3763f9:::smithi03829856-29:head, version: 47'60, data_included: [3364276~232389], data_size: 232389, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8d3763f9:::smithi03829856-29:head@47'60, size: 3596665, copy_subset: [(0, 3596665)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3596665, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3364276, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:30:41.663 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.964363+0000", 2024-03-19T11:30:41.663 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 13.618517695, 2024-03-19T11:30:41.664 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014531759999999999, 2024-03-19T11:30:41.664 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:41.664 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:41.664 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:41.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.664 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:41.664 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.964363+0000", 2024-03-19T11:30:41.664 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.664 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.664 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:41.665 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.964363+0000", 2024-03-19T11:30:41.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.665 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:41.665 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.964364+0000", 2024-03-19T11:30:41.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4200000000000004e-07 2024-03-19T11:30:41.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.665 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:41.665 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.964447+0000", 2024-03-19T11:30:41.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2430999999999997e-05 2024-03-19T11:30:41.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.665 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:41.666 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.964468+0000", 2024-03-19T11:30:41.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1685999999999999e-05 2024-03-19T11:30:41.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.666 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:41.666 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.964483+0000", 2024-03-19T11:30:41.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4969e-05 2024-03-19T11:30:41.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.666 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:41.666 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.977934+0000", 2024-03-19T11:30:41.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013450963999999999 2024-03-19T11:30:41.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.666 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:41.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.977975+0000", 2024-03-19T11:30:41.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1125999999999999e-05 2024-03-19T11:30:41.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:41.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.978895+0000", 2024-03-19T11:30:41.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00091964200000000003 2024-03-19T11:30:41.667 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.667 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:41.667 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.667 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.1 50/49 [PushOp(4:866f3dcc:::smithi03829856-31:7, version: 38'44, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@38'44, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 45'62, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@45'62, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 46'68, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@46'68, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:7, version: 47'71, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@47'71, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 47'72, data_included: [1541576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@47'72, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:30:41.667 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.991858+0000", 2024-03-19T11:30:41.668 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 13.591022772000001, 2024-03-19T11:30:41.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014228774, 2024-03-19T11:30:41.668 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:41.668 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:41.668 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:41.668 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.668 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:41.668 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.991858+0000", 2024-03-19T11:30:41.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.668 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.668 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.668 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:41.668 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.991858+0000", 2024-03-19T11:30:41.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.668 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.669 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.669 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:41.669 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.991859+0000", 2024-03-19T11:30:41.669 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.342e-06 2024-03-19T11:30:41.669 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.669 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.669 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:41.669 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.994675+0000", 2024-03-19T11:30:41.669 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028153150000000001 2024-03-19T11:30:41.669 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.669 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.669 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:41.669 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.994733+0000", 2024-03-19T11:30:41.669 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8514999999999998e-05 2024-03-19T11:30:41.669 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.670 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.670 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:41.670 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.994794+0000", 2024-03-19T11:30:41.670 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0588000000000003e-05 2024-03-19T11:30:41.670 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.670 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.670 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:41.670 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.995070+0000", 2024-03-19T11:30:41.670 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027573700000000002 2024-03-19T11:30:41.670 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.670 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.670 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:41.670 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.995100+0000", 2024-03-19T11:30:41.670 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0679000000000002e-05 2024-03-19T11:30:41.670 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.671 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:41.671 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.006087+0000", 2024-03-19T11:30:41.671 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010986598 2024-03-19T11:30:41.671 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.671 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:41.671 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.671 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.671 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.1 50/49 [PushOp(4:866f3dcc:::smithi03829856-31:head, version: 47'74, data_included: [504210~968657,1572695~79919], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:head@47'74, size: 3495823, copy_subset: [(0, 3495823)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:30:41.671 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.995921+0000", 2024-03-19T11:30:41.671 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 13.586959558, 2024-03-19T11:30:41.671 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014183898, 2024-03-19T11:30:41.671 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:41.672 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:41.672 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:41.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:41.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.995921+0000", 2024-03-19T11:30:41.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:41.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.995921+0000", 2024-03-19T11:30:41.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:41.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.995922+0000", 2024-03-19T11:30:41.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9800000000000002e-07 2024-03-19T11:30:41.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.673 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:41.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.996355+0000", 2024-03-19T11:30:41.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000432211 2024-03-19T11:30:41.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.673 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:41.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.996386+0000", 2024-03-19T11:30:41.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1384999999999999e-05 2024-03-19T11:30:41.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.673 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:41.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.996412+0000", 2024-03-19T11:30:41.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6200999999999999e-05 2024-03-19T11:30:41.674 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.674 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.674 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:41.674 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.006166+0000", 2024-03-19T11:30:41.674 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0097542859999999992 2024-03-19T11:30:41.674 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.674 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.674 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:41.674 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.006182+0000", 2024-03-19T11:30:41.674 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5588000000000001e-05 2024-03-19T11:30:41.674 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.674 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.674 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:41.674 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.010105+0000", 2024-03-19T11:30:41.674 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039232290000000003 2024-03-19T11:30:41.675 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.675 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:41.675 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.675 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.675 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.675 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1679 4.e e59/21 4:72163e4c:::smithi03829856-39:head v 59'55, mlcod=58'53)", 2024-03-19T11:30:41.675 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:36.390395+0000", 2024-03-19T11:30:41.675 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.1924856679999998, 2024-03-19T11:30:41.675 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013872282, 2024-03-19T11:30:41.675 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:41.675 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:41.675 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:41.675 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.675 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:41.675 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.390395+0000", 2024-03-19T11:30:41.676 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.676 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.676 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.676 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:41.676 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.390395+0000", 2024-03-19T11:30:41.676 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:41.676 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.676 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.676 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:41.676 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.390396+0000", 2024-03-19T11:30:41.676 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2300000000000002e-07 2024-03-19T11:30:41.676 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.676 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.676 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:41.676 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.390742+0000", 2024-03-19T11:30:41.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00034635599999999998 2024-03-19T11:30:41.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.677 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:41.677 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.390754+0000", 2024-03-19T11:30:41.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2211999999999999e-05 2024-03-19T11:30:41.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.677 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:41.677 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.390763+0000", 2024-03-19T11:30:41.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5769999999999994e-06 2024-03-19T11:30:41.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.677 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:41.677 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.390864+0000", 2024-03-19T11:30:41.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000101693 2024-03-19T11:30:41.678 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.678 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.678 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:41.678 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.390907+0000", 2024-03-19T11:30:41.678 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2642999999999998e-05 2024-03-19T11:30:41.678 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.678 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.678 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:41.678 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.404144+0000", 2024-03-19T11:30:41.678 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013237319000000001 2024-03-19T11:30:41.678 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:41.678 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:41.678 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:41.678 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.404267+0000", 2024-03-19T11:30:41.679 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012315899999999999 2024-03-19T11:30:41.679 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.679 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:41.679 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.679 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:41.679 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:41.679 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:41.681 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-19T11:30:41.681 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:41.688+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:41.682 INFO:tasks.rados.rados.0.smithi038.stdout:990: writing smithi03829856-11 from 1760371 to 2413737 tid 2 2024-03-19T11:30:41.682 INFO:tasks.rados.rados.0.smithi038.stdout:991: write (excl) oid 33 current snap is 15 2024-03-19T11:30:41.683 INFO:tasks.rados.rados.0.smithi038.stdout:991: seq_num 147 ranges {470629=472310,1418853=513377} 2024-03-19T11:30:41.683 INFO:tasks.rados.rados.0.smithi038.stdout:991: writing smithi03829856-33 from 470629 to 942939 tid 1 2024-03-19T11:30:41.683 INFO:tasks.rados.rados.0.smithi038.stdout:991: writing smithi03829856-33 from 1418853 to 1932230 tid 2 2024-03-19T11:30:41.683 INFO:tasks.rados.rados.0.smithi038.stdout:992: read oid 42 snap -1 2024-03-19T11:30:41.683 INFO:tasks.rados.rados.0.smithi038.stdout:992: expect deleted 2024-03-19T11:30:41.683 INFO:tasks.rados.rados.0.smithi038.stdout:993: set_chunk oid 15 target oid 2024-03-19T11:30:41.683 INFO:tasks.rados.rados.0.smithi038.stdout:993: set_chunk oid 15 offset: 914944 length: 792576 target oid 2dca98192d02806abb03e91583fd58bb221ca7514a0dd87d5cad372e4881c8ee offset: 0 snap -1 2024-03-19T11:30:41.683 INFO:tasks.rados.rados.0.smithi038.stdout:994: set_chunk oid 49 target oid 2024-03-19T11:30:41.683 INFO:tasks.rados.rados.0.smithi038.stdout:994: set_chunk oid 49 offset: 409088 length: 1620992 target oid a21d4d098d0511482fdd0a91aaa4966751434ca5bf578b277b61c15ae564d299 offset: 0 snap 10 2024-03-19T11:30:41.683 INFO:tasks.rados.rados.0.smithi038.stdout:995: read oid 22 snap -1 2024-03-19T11:30:41.683 INFO:tasks.rados.rados.0.smithi038.stdout:995: expect (ObjNum 21 snap 0 seq_num 21) 2024-03-19T11:30:41.683 INFO:tasks.rados.rados.0.smithi038.stdout:996: write oid 45 current snap is 15 2024-03-19T11:30:41.683 INFO:tasks.rados.rados.0.smithi038.stdout:990: finishing write tid 1 to smithi03829856-11 2024-03-19T11:30:41.683 INFO:tasks.rados.rados.0.smithi038.stdout:990: finishing write tid 2 to smithi03829856-11 2024-03-19T11:30:41.683 INFO:tasks.rados.rados.0.smithi038.stdout:990: finishing write tid 3 to smithi03829856-11 2024-03-19T11:30:41.684 INFO:tasks.rados.rados.0.smithi038.stdout:996: seq_num 148 ranges {487436=547559,1760982=657035} 2024-03-19T11:30:41.684 INFO:tasks.rados.rados.0.smithi038.stdout:996: writing smithi03829856-45 from 487436 to 1034995 tid 1 2024-03-19T11:30:41.685 INFO:tasks.rados.rados.0.smithi038.stdout:996: writing smithi03829856-45 from 1760982 to 2418017 tid 2 2024-03-19T11:30:41.685 INFO:tasks.rados.rados.0.smithi038.stdout:997: read oid 37 snap -1 2024-03-19T11:30:41.685 INFO:tasks.rados.rados.0.smithi038.stdout:997: expect (ObjNum 36 snap 0 seq_num 36) 2024-03-19T11:30:41.685 INFO:tasks.rados.rados.0.smithi038.stdout:998: rollback oid 16 current snap is 15 2024-03-19T11:30:41.685 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 16 to 11 2024-03-19T11:30:41.685 INFO:tasks.rados.rados.0.smithi038.stdout:990: finishing write tid 4 to smithi03829856-11 2024-03-19T11:30:41.685 INFO:tasks.rados.rados.0.smithi038.stdout:990: oid 11 updating version 0 to 63 2024-03-19T11:30:41.685 INFO:tasks.rados.rados.0.smithi038.stdout:990: oid 11 version 63 is already newer than 61 2024-03-19T11:30:41.685 INFO:tasks.rados.rados.0.smithi038.stdout:990: oid 11 version 63 is already newer than 62 2024-03-19T11:30:41.685 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 11 v 63 (ObjNum 146 snap 15 seq_num 146) dirty exists 2024-03-19T11:30:41.685 INFO:tasks.rados.rados.0.smithi038.stdout:990: left oid 11 (ObjNum 146 snap 15 seq_num 146) 2024-03-19T11:30:41.685 INFO:tasks.rados.rados.0.smithi038.stdout:991: finishing write tid 1 to smithi03829856-33 2024-03-19T11:30:41.685 INFO:tasks.rados.rados.0.smithi038.stdout:990: done (8 left) 2024-03-19T11:30:41.685 INFO:tasks.rados.rados.0.smithi038.stdout:999: tier_flush oid 21 2024-03-19T11:30:41.685 INFO:tasks.rados.rados.0.smithi038.stdout:999: tier_flush oid 21 snap -1 2024-03-19T11:30:41.686 INFO:tasks.rados.rados.0.smithi038.stdout:1000: set_chunk oid 10 target oid 2024-03-19T11:30:41.708 INFO:tasks.rados.rados.0.smithi038.stdout:1000: set_chunk oid 10 offset: 273920 length: 1083904 target oid d3eecde3596f6f04396f3ca89f0552e9c363ede0468bda4e67ab2af462e65d0f offset: 0 snap -1 2024-03-19T11:30:41.708 INFO:tasks.rados.rados.0.smithi038.stdout:1001: read oid 25 snap -1 2024-03-19T11:30:41.708 INFO:tasks.rados.rados.0.smithi038.stdout:1001: expect deleted 2024-03-19T11:30:41.709 INFO:tasks.rados.rados.0.smithi038.stdout:1002: set_chunk oid 34 target oid 2024-03-19T11:30:41.717 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-19T11:30:41.739 INFO:tasks.rados.rados.0.smithi038.stdout:1002: set_chunk oid 34 offset: 82944 length: 2557440 target oid a32a1412677b7b8fb730a0c3225fa9b672680e6bcb3262b9965d5e0bcc3c046f offset: 0 snap 10 2024-03-19T11:30:41.739 INFO:tasks.rados.rados.0.smithi038.stdout:1003: read oid 46 snap -1 2024-03-19T11:30:41.739 INFO:tasks.rados.rados.0.smithi038.stdout:1003: expect (ObjNum 143 snap 15 seq_num 143) 2024-03-19T11:30:41.739 INFO:tasks.rados.rados.0.smithi038.stdout:1004: read oid 48 snap -1 2024-03-19T11:30:41.739 INFO:tasks.rados.rados.0.smithi038.stdout:1004: expect (ObjNum 131 snap 15 seq_num 131) 2024-03-19T11:30:41.739 INFO:tasks.rados.rados.0.smithi038.stdout:1005: set_chunk oid 35 target oid 2024-03-19T11:30:41.764 INFO:tasks.rados.rados.0.smithi038.stdout:1005: set_chunk oid 35 offset: 163328 length: 2273792 target oid a81eaaab8d56ce5137e167414673004dfeb1401d53dd4b3440d7d61aa13edb85 offset: 0 snap -1 2024-03-19T11:30:41.764 INFO:tasks.rados.rados.0.smithi038.stdout:1006: write (excl) oid 1 current snap is 15 2024-03-19T11:30:41.764 INFO:tasks.rados.rados.0.smithi038.stdout:1006: seq_num 149 ranges {504243=622808,1819464=517046} 2024-03-19T11:30:41.777 INFO:tasks.rados.rados.0.smithi038.stdout:1006: writing smithi03829856-1 from 504243 to 1127051 tid 1 2024-03-19T11:30:41.780 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:41.790+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:41.790 INFO:tasks.rados.rados.0.smithi038.stdout:1006: writing smithi03829856-1 from 1819464 to 2336510 tid 2 2024-03-19T11:30:41.790 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:41.790 INFO:tasks.rados.rados.0.smithi038.stdout:991: finishing write tid 2 to smithi03829856-33 2024-03-19T11:30:41.790 INFO:tasks.rados.rados.0.smithi038.stdout:991: finishing write tid 3 to smithi03829856-33 2024-03-19T11:30:41.790 INFO:tasks.rados.rados.0.smithi038.stdout:992: done (15 left) 2024-03-19T11:30:41.790 INFO:tasks.rados.rados.0.smithi038.stdout:1007: snap_create 2024-03-19T11:30:41.790 INFO:tasks.rados.rados.0.smithi038.stdout:991: finishing write tid 4 to smithi03829856-33 2024-03-19T11:30:41.790 INFO:tasks.rados.rados.0.smithi038.stdout:991: oid 33 updating version 0 to 71 2024-03-19T11:30:41.790 INFO:tasks.rados.rados.0.smithi038.stdout:991: oid 33 updating version 71 to 73 2024-03-19T11:30:41.790 INFO:tasks.rados.rados.0.smithi038.stdout:991: oid 33 version 73 is already newer than 72 2024-03-19T11:30:41.791 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 33 v 73 (ObjNum 147 snap 15 seq_num 147) dirty exists 2024-03-19T11:30:41.791 INFO:tasks.rados.rados.0.smithi038.stdout:991: left oid 33 (ObjNum 147 snap 15 seq_num 147) 2024-03-19T11:30:41.791 INFO:tasks.rados.rados.0.smithi038.stdout:993: finishing set_chunk to oid 15 2024-03-19T11:30:41.791 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 15 set_chunk returned error code -95 offset: 914944 length: 792576 2024-03-19T11:30:41.791 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-03-19T11:30:41.791 INFO:tasks.rados.rados.0.smithi038.stdout:994: finishing set_chunk to oid 49 2024-03-19T11:30:41.791 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 49 set_chunk returned error code -95 offset: 409088 length: 1620992 2024-03-19T11:30:41.791 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 49 v 0 (ObjNum 136 snap 15 seq_num 136) dirty exists 2024-03-19T11:30:41.791 INFO:tasks.rados.rados.0.smithi038.stdout:995: expect (ObjNum 21 snap 0 seq_num 21) 2024-03-19T11:30:41.853 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:41.853 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:41.853 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:30:41.853 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:41.867 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-19T11:30:41.884 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:30:41.884 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:30:41.887 INFO:tasks.rados.rados.0.smithi038.stdout:996: finishing write tid 1 to smithi03829856-45 2024-03-19T11:30:41.887 INFO:tasks.rados.rados.0.smithi038.stdout:999: got (16) Device or resource busy 2024-03-19T11:30:41.888 INFO:tasks.rados.rados.0.smithi038.stdout:check_oldest_snap_flushed oid 21 v 86 (ObjNum 129 snap 15 seq_num 129) unflushed 15 2024-03-19T11:30:41.888 INFO:tasks.rados.rados.0.smithi038.stdout:997: expect (ObjNum 36 snap 0 seq_num 36) 2024-03-19T11:30:41.901 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:30:41.901 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:30:41.910 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:41.920+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:41.918 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:30:41.918 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:30:41.935 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:30:41.935 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:30:41.954 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:30:41.955 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:30:41.968 INFO:tasks.rados.rados.0.smithi038.stdout:998: finishing rollback tid 0 to smithi03829856-16 2024-03-19T11:30:41.969 INFO:tasks.rados.rados.0.smithi038.stdout:996: finishing write tid 2 to smithi03829856-45 2024-03-19T11:30:41.969 INFO:tasks.rados.rados.0.smithi038.stdout:996: finishing write tid 3 to smithi03829856-45 2024-03-19T11:30:41.969 INFO:tasks.rados.rados.0.smithi038.stdout:996: finishing write tid 4 to smithi03829856-45 2024-03-19T11:30:41.969 INFO:tasks.rados.rados.0.smithi038.stdout:996: oid 45 updating version 0 to 54 2024-03-19T11:30:41.969 INFO:tasks.rados.rados.0.smithi038.stdout:996: oid 45 updating version 54 to 55 2024-03-19T11:30:41.969 INFO:tasks.rados.rados.0.smithi038.stdout:996: oid 45 version 55 is already newer than 53 2024-03-19T11:30:41.969 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 45 v 55 (ObjNum 148 snap 15 seq_num 148) dirty exists 2024-03-19T11:30:41.969 INFO:tasks.rados.rados.0.smithi038.stdout:996: left oid 45 (ObjNum 148 snap 15 seq_num 148) 2024-03-19T11:30:41.969 INFO:tasks.rados.rados.0.smithi038.stdout:998: finishing rollback tid 1 to smithi03829856-16 2024-03-19T11:30:41.969 INFO:tasks.rados.rados.0.smithi038.stdout:998: finishing rollback tid 2 to smithi03829856-16 2024-03-19T11:30:41.969 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 16 v 100 (ObjNum 65 snap 3 seq_num 65) dirty exists 2024-03-19T11:30:41.969 INFO:tasks.rados.rados.0.smithi038.stdout:1000: finishing set_chunk to oid 10 2024-03-19T11:30:41.969 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 10 v 73 (ObjNum 144 snap 15 seq_num 144) dirty exists 2024-03-19T11:30:41.969 INFO:tasks.rados.rados.0.smithi038.stdout:1002: finishing set_chunk to oid 34 2024-03-19T11:30:41.970 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 34 set_chunk returned error code -95 offset: 82944 length: 2557440 2024-03-19T11:30:41.970 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 34 v 0 (ObjNum 72 snap 5 seq_num 72) dirty exists 2024-03-19T11:30:41.970 INFO:tasks.rados.rados.0.smithi038.stdout:1003: expect (ObjNum 143 snap 15 seq_num 143) 2024-03-19T11:30:41.974 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:30:41.974 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:30:41.982 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:41.990+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:41.985 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:41.985 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:41.986 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:30:41.986 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:30:41.986 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:41.993 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:30:41.993 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:30:42.001 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-19T11:30:42.014 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:30:42.014 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:30:42.047 INFO:tasks.rados.rados.0.smithi038.stdout:1004: expect (ObjNum 131 snap 15 seq_num 131) 2024-03-19T11:30:42.083 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:42.094+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:42.117 INFO:tasks.rados.rados.0.smithi038.stdout:1005: finishing set_chunk to oid 35 2024-03-19T11:30:42.117 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 35 set_chunk returned error code -95 offset: 163328 length: 2273792 2024-03-19T11:30:42.117 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 35 v 0 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-03-19T11:30:42.117 INFO:tasks.rados.rados.0.smithi038.stdout:1006: finishing write tid 1 to smithi03829856-1 2024-03-19T11:30:42.117 INFO:tasks.rados.rados.0.smithi038.stdout:1006: finishing write tid 2 to smithi03829856-1 2024-03-19T11:30:42.117 INFO:tasks.rados.rados.0.smithi038.stdout:1006: finishing write tid 3 to smithi03829856-1 2024-03-19T11:30:42.117 INFO:tasks.rados.rados.0.smithi038.stdout:1006: finishing write tid 4 to smithi03829856-1 2024-03-19T11:30:42.117 INFO:tasks.rados.rados.0.smithi038.stdout:1006: oid 1 updating version 0 to 92 2024-03-19T11:30:42.118 INFO:tasks.rados.rados.0.smithi038.stdout:1006: oid 1 updating version 92 to 94 2024-03-19T11:30:42.118 INFO:tasks.rados.rados.0.smithi038.stdout:1006: oid 1 version 94 is already newer than 93 2024-03-19T11:30:42.118 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 1 v 94 (ObjNum 149 snap 15 seq_num 149) dirty exists 2024-03-19T11:30:42.118 INFO:tasks.rados.rados.0.smithi038.stdout:1006: left oid 1 (ObjNum 149 snap 15 seq_num 149) 2024-03-19T11:30:42.162 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:42.162 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:42.174 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-03-19T11:30:42.176 INFO:tasks.rados.rados.0.smithi038.stdout:991: done (15 left) 2024-03-19T11:30:42.176 INFO:tasks.rados.rados.0.smithi038.stdout:993: done (14 left) 2024-03-19T11:30:42.176 INFO:tasks.rados.rados.0.smithi038.stdout:994: done (13 left) 2024-03-19T11:30:42.176 INFO:tasks.rados.rados.0.smithi038.stdout:995: done (12 left) 2024-03-19T11:30:42.177 INFO:tasks.rados.rados.0.smithi038.stdout:996: done (11 left) 2024-03-19T11:30:42.177 INFO:tasks.rados.rados.0.smithi038.stdout:997: done (10 left) 2024-03-19T11:30:42.177 INFO:tasks.rados.rados.0.smithi038.stdout:998: done (9 left) 2024-03-19T11:30:42.177 INFO:tasks.rados.rados.0.smithi038.stdout:999: done (8 left) 2024-03-19T11:30:42.177 INFO:tasks.rados.rados.0.smithi038.stdout:1000: done (7 left) 2024-03-19T11:30:42.177 INFO:tasks.rados.rados.0.smithi038.stdout:1001: done (6 left) 2024-03-19T11:30:42.178 INFO:tasks.rados.rados.0.smithi038.stdout:1002: done (5 left) 2024-03-19T11:30:42.178 INFO:tasks.rados.rados.0.smithi038.stdout:1003: done (4 left) 2024-03-19T11:30:42.178 INFO:tasks.rados.rados.0.smithi038.stdout:1004: done (3 left) 2024-03-19T11:30:42.178 INFO:tasks.rados.rados.0.smithi038.stdout:1005: done (2 left) 2024-03-19T11:30:42.178 INFO:tasks.rados.rados.0.smithi038.stdout:1006: done (1 left) 2024-03-19T11:30:42.178 INFO:tasks.rados.rados.0.smithi038.stdout:1007: done (0 left) 2024-03-19T11:30:42.178 INFO:tasks.rados.rados.0.smithi038.stdout:1008: write (excl) oid 35 current snap is 16 2024-03-19T11:30:42.178 INFO:tasks.rados.rados.0.smithi038.stdout:1008: seq_num 150 ranges {521050=698057,1877946=643104} 2024-03-19T11:30:42.179 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-19T11:30:42.189 INFO:tasks.rados.rados.0.smithi038.stdout:1008: writing smithi03829856-35 from 521050 to 1219107 tid 1 2024-03-19T11:30:42.190 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:42.198+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:42.200 INFO:tasks.rados.rados.0.smithi038.stdout:1008: writing smithi03829856-35 from 1877946 to 2521050 tid 2 2024-03-19T11:30:42.200 INFO:tasks.rados.rados.0.smithi038.stdout:1009: snap_create 2024-03-19T11:30:42.200 INFO:tasks.rados.rados.0.smithi038.stdout:1008: finishing write tid 1 to smithi03829856-35 2024-03-19T11:30:42.207 INFO:tasks.rados.rados.0.smithi038.stdout:1008: finishing write tid 2 to smithi03829856-35 2024-03-19T11:30:42.207 INFO:tasks.rados.rados.0.smithi038.stdout:1008: finishing write tid 3 to smithi03829856-35 2024-03-19T11:30:42.207 INFO:tasks.rados.rados.0.smithi038.stdout:1008: finishing write tid 4 to smithi03829856-35 2024-03-19T11:30:42.207 INFO:tasks.rados.rados.0.smithi038.stdout:1008: oid 35 updating version 0 to 100 2024-03-19T11:30:42.207 INFO:tasks.rados.rados.0.smithi038.stdout:1008: oid 35 version 100 is already newer than 99 2024-03-19T11:30:42.207 INFO:tasks.rados.rados.0.smithi038.stdout:1008: oid 35 version 100 is already newer than 98 2024-03-19T11:30:42.208 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 35 v 100 (ObjNum 150 snap 16 seq_num 150) dirty exists 2024-03-19T11:30:42.208 INFO:tasks.rados.rados.0.smithi038.stdout:1008: left oid 35 (ObjNum 150 snap 16 seq_num 150) 2024-03-19T11:30:42.284 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:42.294+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:42.301 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:42.301 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:42.316 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-19T11:30:42.407 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:42.418+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:42.486 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:42.496+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:42.489 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:42.489 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:42.504 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-19T11:30:42.591 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:42.602+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:42.677 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:42.677 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:42.688 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:42.696+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:42.691 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-19T11:30:42.789 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:42.796+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:42.864 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:42.864 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:42.886 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-19T11:30:42.889 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:42.896+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:42.989 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:42.998+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:43.012 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:43.013 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:43.027 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-19T11:30:43.115 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:43.126+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:43.191 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:43.202+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:43.195 INFO:tasks.rados.rados.0.smithi038.stdout:1008: done (1 left) 2024-03-19T11:30:43.195 INFO:tasks.rados.rados.0.smithi038.stdout:1009: done (0 left) 2024-03-19T11:30:43.195 INFO:tasks.rados.rados.0.smithi038.stdout:1010: set_chunk oid 42 target oid 2024-03-19T11:30:43.195 INFO:tasks.rados.rados.0.smithi038.stdout:1010: set_chunk oid 42 offset: 0 length: 4000000 target oid smithi03829856- offset: 0 snap -1 2024-03-19T11:30:43.195 INFO:tasks.rados.rados.0.smithi038.stdout:1011: read oid 43 snap -1 2024-03-19T11:30:43.195 INFO:tasks.rados.rados.0.smithi038.stdout:1011: expect (ObjNum 91 snap 10 seq_num 91) 2024-03-19T11:30:43.195 INFO:tasks.rados.rados.0.smithi038.stdout:1012: write (excl) oid 4 current snap is 17 2024-03-19T11:30:43.195 INFO:tasks.rados.rados.0.smithi038.stdout:1012: seq_num 151 ranges {537857=773306,1936428=520715} 2024-03-19T11:30:43.207 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:43.207 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:43.212 INFO:tasks.rados.rados.0.smithi038.stdout:1012: writing smithi03829856-4 from 537857 to 1311163 tid 1 2024-03-19T11:30:43.225 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-19T11:30:43.227 INFO:tasks.rados.rados.0.smithi038.stdout:1012: writing smithi03829856-4 from 1936428 to 2457143 tid 2 2024-03-19T11:30:43.227 INFO:tasks.rados.rados.0.smithi038.stdout:1013: set_chunk oid 13 target oid 2024-03-19T11:30:43.238 INFO:tasks.rados.rados.0.smithi038.stdout:1013: set_chunk oid 13 offset: 1140736 length: 30720 target oid b3b5f43e74705be0b63e281c77870561f3d8d638b53b97e40fbc2e31c48eaa14 offset: 0 snap -1 2024-03-19T11:30:43.238 INFO:tasks.rados.rados.0.smithi038.stdout:1014: tier_promote oid 19 2024-03-19T11:30:43.238 INFO:tasks.rados.rados.0.smithi038.stdout:1010: finishing set_chunk to oid 42 2024-03-19T11:30:43.238 INFO:tasks.rados.rados.0.smithi038.stdout:1010: got expected ERANGE (src dne) 2024-03-19T11:30:43.238 INFO:tasks.rados.rados.0.smithi038.stdout:1011: expect (ObjNum 91 snap 10 seq_num 91) 2024-03-19T11:30:43.292 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:43.302+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:43.350 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:43.351 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:43.366 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-19T11:30:43.393 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:43.400+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:43.485 INFO:tasks.rados.rados.0.smithi038.stdout:1012: finishing write tid 1 to smithi03829856-4 2024-03-19T11:30:43.485 INFO:tasks.rados.rados.0.smithi038.stdout:1012: finishing write tid 2 to smithi03829856-4 2024-03-19T11:30:43.485 INFO:tasks.rados.rados.0.smithi038.stdout:1012: finishing write tid 3 to smithi03829856-4 2024-03-19T11:30:43.485 INFO:tasks.rados.rados.0.smithi038.stdout:1012: finishing write tid 4 to smithi03829856-4 2024-03-19T11:30:43.485 INFO:tasks.rados.rados.0.smithi038.stdout:1012: oid 4 updating version 0 to 104 2024-03-19T11:30:43.485 INFO:tasks.rados.rados.0.smithi038.stdout:1012: oid 4 version 104 is already newer than 103 2024-03-19T11:30:43.486 INFO:tasks.rados.rados.0.smithi038.stdout:1012: oid 4 version 104 is already newer than 102 2024-03-19T11:30:43.486 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 4 v 104 (ObjNum 151 snap 17 seq_num 151) dirty exists 2024-03-19T11:30:43.486 INFO:tasks.rados.rados.0.smithi038.stdout:1012: left oid 4 (ObjNum 151 snap 17 seq_num 151) 2024-03-19T11:30:43.486 INFO:tasks.rados.rados.0.smithi038.stdout:1013: finishing set_chunk to oid 13 2024-03-19T11:30:43.486 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 13 set_chunk returned error code -95 offset: 1140736 length: 30720 2024-03-19T11:30:43.486 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 13 v 0 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-03-19T11:30:43.486 INFO:tasks.rados.rados.0.smithi038.stdout:1014: got (0) Success 2024-03-19T11:30:43.486 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 19 v 50 (ObjNum 128 snap 15 seq_num 128) dirty exists 2024-03-19T11:30:43.486 INFO:tasks.rados.rados.0.smithi038.stdout:1010: done (4 left) 2024-03-19T11:30:43.486 INFO:tasks.rados.rados.0.smithi038.stdout:1011: done (3 left) 2024-03-19T11:30:43.486 INFO:tasks.rados.rados.0.smithi038.stdout:1012: done (2 left) 2024-03-19T11:30:43.486 INFO:tasks.rados.rados.0.smithi038.stdout:1013: done (1 left) 2024-03-19T11:30:43.487 INFO:tasks.rados.rados.0.smithi038.stdout:1014: done (0 left) 2024-03-19T11:30:43.487 INFO:tasks.rados.rados.0.smithi038.stdout:1015: write oid 9 current snap is 17 2024-03-19T11:30:43.487 INFO:tasks.rados.rados.0.smithi038.stdout:1015: seq_num 152 ranges {554664=448555,1478557=664373} 2024-03-19T11:30:43.494 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:43.502+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:43.501 INFO:tasks.rados.rados.0.smithi038.stdout:1015: writing smithi03829856-9 from 554664 to 1003219 tid 1 2024-03-19T11:30:43.519 INFO:tasks.rados.rados.0.smithi038.stdout:1015: writing smithi03829856-9 from 1478557 to 2142930 tid 2 2024-03-19T11:30:43.519 INFO:tasks.rados.rados.0.smithi038.stdout:1016: write oid 19 current snap is 17 2024-03-19T11:30:43.519 INFO:tasks.rados.rados.0.smithi038.stdout:1016: seq_num 153 ranges {571471=640157,1653392=524384} 2024-03-19T11:30:43.530 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:43.530 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:43.534 INFO:tasks.rados.rados.0.smithi038.stdout:1016: writing smithi03829856-19 from 571471 to 1211628 tid 1 2024-03-19T11:30:43.544 INFO:tasks.rados.rados.0.smithi038.stdout:1016: writing smithi03829856-19 from 1653392 to 2177776 tid 2 2024-03-19T11:30:43.544 INFO:tasks.rados.rados.0.smithi038.stdout:1017: write oid 29 current snap is 17 2024-03-19T11:30:43.544 INFO:tasks.rados.rados.0.smithi038.stdout:1017: seq_num 154 ranges {588278=715406,1711874=668042} 2024-03-19T11:30:43.545 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-19T11:30:43.556 INFO:tasks.rados.rados.0.smithi038.stdout:1017: writing smithi03829856-29 from 588278 to 1303684 tid 1 2024-03-19T11:30:43.566 INFO:tasks.rados.rados.0.smithi038.stdout:1017: writing smithi03829856-29 from 1711874 to 2379916 tid 2 2024-03-19T11:30:43.566 INFO:tasks.rados.rados.0.smithi038.stdout:1018: read oid 27 snap -1 2024-03-19T11:30:43.566 INFO:tasks.rados.rados.0.smithi038.stdout:1018: expect deleted 2024-03-19T11:30:43.567 INFO:tasks.rados.rados.0.smithi038.stdout:1019: snap_remove snap 14 2024-03-19T11:30:43.567 INFO:tasks.rados.rados.0.smithi038.stdout:1015: finishing write tid 1 to smithi03829856-9 2024-03-19T11:30:43.567 INFO:tasks.rados.rados.0.smithi038.stdout:1015: finishing write tid 2 to smithi03829856-9 2024-03-19T11:30:43.595 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:43.606+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:43.672 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:43.673 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:43.688 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-19T11:30:43.732 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:43.742+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:43.797 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:43.806+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:43.859 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:43.860 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:43.874 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-19T11:30:43.898 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:43.904+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:43.998 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:44.006+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:44.032 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-19T11:30:44.044 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:44.044 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:44.059 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-19T11:30:44.099 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:44.108+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:44.185 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:30:44.185 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:30:44.198 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:44.206+0000 7f70c6fdf640 -1 osd.0 65 handle_osd_map injecting map crc failure 2024-03-19T11:30:44.199 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-03-19T11:30:44.200 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:44.208+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:44.212 INFO:tasks.rados.rados.0.smithi038.stdout:1019: done (4 left) 2024-03-19T11:30:44.213 INFO:tasks.rados.rados.0.smithi038.stdout:1020: write (excl) oid 43 current snap is 17 2024-03-19T11:30:44.213 INFO:tasks.rados.rados.0.smithi038.stdout:1020: seq_num 155 ranges {605085=790655,2170356=434729} 2024-03-19T11:30:44.213 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:30:44.214 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:30:44.222 INFO:tasks.rados.rados.0.smithi038.stdout:1020: writing smithi03829856-43 from 605085 to 1395740 tid 1 2024-03-19T11:30:44.231 INFO:tasks.rados.rados.0.smithi038.stdout:1020: writing smithi03829856-43 from 2170356 to 2605085 tid 2 2024-03-19T11:30:44.231 INFO:tasks.rados.rados.0.smithi038.stdout:1015: finishing write tid 3 to smithi03829856-9 2024-03-19T11:30:44.231 INFO:tasks.rados.rados.0.smithi038.stdout:1015: finishing write tid 4 to smithi03829856-9 2024-03-19T11:30:44.231 INFO:tasks.rados.rados.0.smithi038.stdout:1015: oid 9 updating version 0 to 63 2024-03-19T11:30:44.231 INFO:tasks.rados.rados.0.smithi038.stdout:1015: oid 9 updating version 63 to 64 2024-03-19T11:30:44.231 INFO:tasks.rados.rados.0.smithi038.stdout:1015: oid 9 updating version 64 to 65 2024-03-19T11:30:44.231 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 9 v 65 (ObjNum 152 snap 17 seq_num 152) dirty exists 2024-03-19T11:30:44.232 INFO:tasks.rados.rados.0.smithi038.stdout:1015: left oid 9 (ObjNum 152 snap 17 seq_num 152) 2024-03-19T11:30:44.232 INFO:tasks.rados.rados.0.smithi038.stdout:1016: finishing write tid 1 to smithi03829856-19 2024-03-19T11:30:44.232 INFO:tasks.rados.rados.0.smithi038.stdout:1016: finishing write tid 2 to smithi03829856-19 2024-03-19T11:30:44.232 INFO:tasks.rados.rados.0.smithi038.stdout:1016: finishing write tid 3 to smithi03829856-19 2024-03-19T11:30:44.232 INFO:tasks.rados.rados.0.smithi038.stdout:1016: finishing write tid 4 to smithi03829856-19 2024-03-19T11:30:44.232 INFO:tasks.rados.rados.0.smithi038.stdout:1016: oid 19 updating version 0 to 55 2024-03-19T11:30:44.232 INFO:tasks.rados.rados.0.smithi038.stdout:1016: oid 19 updating version 55 to 57 2024-03-19T11:30:44.232 INFO:tasks.rados.rados.0.smithi038.stdout:1016: oid 19 version 57 is already newer than 56 2024-03-19T11:30:44.232 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 19 v 57 (ObjNum 153 snap 17 seq_num 153) dirty exists 2024-03-19T11:30:44.232 INFO:tasks.rados.rados.0.smithi038.stdout:1016: left oid 19 (ObjNum 153 snap 17 seq_num 153) 2024-03-19T11:30:44.232 INFO:tasks.rados.rados.0.smithi038.stdout:1017: finishing write tid 1 to smithi03829856-29 2024-03-19T11:30:44.232 INFO:tasks.rados.rados.0.smithi038.stdout:1017: finishing write tid 2 to smithi03829856-29 2024-03-19T11:30:44.233 INFO:tasks.rados.rados.0.smithi038.stdout:1017: finishing write tid 3 to smithi03829856-29 2024-03-19T11:30:44.233 INFO:tasks.rados.rados.0.smithi038.stdout:1017: finishing write tid 4 to smithi03829856-29 2024-03-19T11:30:44.233 INFO:tasks.rados.rados.0.smithi038.stdout:1017: oid 29 updating version 0 to 64 2024-03-19T11:30:44.233 INFO:tasks.rados.rados.0.smithi038.stdout:1017: oid 29 updating version 64 to 65 2024-03-19T11:30:44.233 INFO:tasks.rados.rados.0.smithi038.stdout:1017: oid 29 updating version 65 to 66 2024-03-19T11:30:44.233 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 29 v 66 (ObjNum 154 snap 17 seq_num 154) dirty exists 2024-03-19T11:30:44.233 INFO:tasks.rados.rados.0.smithi038.stdout:1017: left oid 29 (ObjNum 154 snap 17 seq_num 154) 2024-03-19T11:30:44.233 INFO:tasks.rados.rados.0.smithi038.stdout:1015: done (4 left) 2024-03-19T11:30:44.233 INFO:tasks.rados.rados.0.smithi038.stdout:1016: done (3 left) 2024-03-19T11:30:44.233 INFO:tasks.rados.rados.0.smithi038.stdout:1017: done (2 left) 2024-03-19T11:30:44.233 INFO:tasks.rados.rados.0.smithi038.stdout:1018: done (1 left) 2024-03-19T11:30:44.234 INFO:tasks.rados.rados.0.smithi038.stdout:1021: write oid 1 current snap is 17 2024-03-19T11:30:44.234 INFO:tasks.rados.rados.0.smithi038.stdout:1021: seq_num 156 ranges {621892=465904,1712485=671711} 2024-03-19T11:30:44.237 INFO:tasks.rados.rados.0.smithi038.stdout:1021: writing smithi03829856-1 from 621892 to 1087796 tid 1 2024-03-19T11:30:44.238 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:30:44.239 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:30:44.247 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:44.247 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:44.247 INFO:tasks.rados.rados.0.smithi038.stdout:1021: writing smithi03829856-1 from 1712485 to 2384196 tid 2 2024-03-19T11:30:44.248 INFO:tasks.rados.rados.0.smithi038.stdout:1022: read oid 34 snap 10 2024-03-19T11:30:44.248 INFO:tasks.rados.rados.0.smithi038.stdout:1022: expect (ObjNum 72 snap 5 seq_num 72) 2024-03-19T11:30:44.248 INFO:tasks.rados.rados.0.smithi038.stdout:1023: set_chunk oid 10 target oid 2024-03-19T11:30:44.260 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:30:44.260 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:30:44.263 INFO:tasks.rados.rados.0.smithi038.stdout:1023: set_chunk oid 10 offset: 188928 length: 913920 target oid 9d1e86306df5d6817b66534c7924ff12b2398f4420f84b0c4c39401b444549de offset: 0 snap -1 2024-03-19T11:30:44.263 INFO:tasks.rados.rados.0.smithi038.stdout:1024: read oid 44 snap -1 2024-03-19T11:30:44.263 INFO:tasks.rados.rados.0.smithi038.stdout:1024: expect (ObjNum 108 snap 11 seq_num 108) 2024-03-19T11:30:44.263 INFO:tasks.rados.rados.0.smithi038.stdout:1025: set_chunk oid 7 target oid 2024-03-19T11:30:44.269 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-19T11:30:44.280 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:30:44.281 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:30:44.301 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:30:44.301 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:30:44.301 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:44.310+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:44.318 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:30:44.318 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:30:44.338 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:30:44.338 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:30:44.402 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:44.402 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:44.402 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:30:44.402 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:44.402 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:44.408+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:44.416 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-19T11:30:44.504 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:44.514+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:44.580 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:44.580 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:44.580 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:30:44.580 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:30:44.580 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:44.594 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-19T11:30:44.639 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:44.648+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:44.705 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:44.712+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:44.717 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:44.718 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:30:44.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:30:44.718 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:30:44.718 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.718 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:9 4.d 4:b4ef97e0:::smithi03829856-3:head [write 433614~550498,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-03-19T11:30:44.718 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:04.706000+0000", 2024-03-19T11:30:44.718 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 40.020654225000001, 2024-03-19T11:30:44.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013227001, 2024-03-19T11:30:44.718 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:44.718 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:44.718 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:44.718 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:44.719 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:44.719 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 9 2024-03-19T11:30:44.719 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.719 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:44.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.719 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:44.719 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.706000+0000", 2024-03-19T11:30:44.719 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:44.719 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.719 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:44.719 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.706000+0000", 2024-03-19T11:30:44.719 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:44.719 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.720 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:44.720 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.706002+0000", 2024-03-19T11:30:44.720 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5569999999999998e-06 2024-03-19T11:30:44.720 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.720 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:44.720 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.707261+0000", 2024-03-19T11:30:44.720 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001258817 2024-03-19T11:30:44.720 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.720 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:44.720 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.707288+0000", 2024-03-19T11:30:44.720 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7093e-05 2024-03-19T11:30:44.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.721 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.721 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:44.721 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.707305+0000", 2024-03-19T11:30:44.721 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6330000000000001e-05 2024-03-19T11:30:44.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.721 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.721 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:44.721 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.707610+0000", 2024-03-19T11:30:44.721 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00030568799999999998 2024-03-19T11:30:44.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.721 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.721 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:44.721 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.707984+0000", 2024-03-19T11:30:44.722 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00037399299999999998 2024-03-19T11:30:44.722 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.722 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.722 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2", 2024-03-19T11:30:44.722 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.708409+0000", 2024-03-19T11:30:44.722 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00042503099999999998 2024-03-19T11:30:44.723 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.723 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.723 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:44.723 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.712919+0000", 2024-03-19T11:30:44.723 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0045099010000000002 2024-03-19T11:30:44.723 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.723 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.723 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:44.723 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.719037+0000", 2024-03-19T11:30:44.723 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061173770000000002 2024-03-19T11:30:44.724 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.724 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.724 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:44.724 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.719116+0000", 2024-03-19T11:30:44.724 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9935e-05 2024-03-19T11:30:44.724 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.724 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.724 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:44.724 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.719227+0000", 2024-03-19T11:30:44.724 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011027900000000001 2024-03-19T11:30:44.724 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:44.724 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:44.724 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:44.724 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.725 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.725 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:848 4.1 4:866f3dcc:::smithi03829856-31:head [read 0~2941192,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:44.725 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.757468+0000", 2024-03-19T11:30:44.725 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 35.969186233000002, 2024-03-19T11:30:44.725 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015020110999999999, 2024-03-19T11:30:44.725 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:44.725 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:44.725 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:44.725 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:44.726 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:44.726 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 848 2024-03-19T11:30:44.726 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.726 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:44.726 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.726 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:44.726 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757468+0000", 2024-03-19T11:30:44.726 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:44.726 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.726 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.726 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:44.727 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757468+0000", 2024-03-19T11:30:44.727 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:44.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.727 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:44.727 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757471+0000", 2024-03-19T11:30:44.727 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.67e-06 2024-03-19T11:30:44.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.727 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:44.727 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757488+0000", 2024-03-19T11:30:44.727 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6223e-05 2024-03-19T11:30:44.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.727 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:44.728 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757521+0000", 2024-03-19T11:30:44.728 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.358e-05 2024-03-19T11:30:44.728 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.728 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.728 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:44.728 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757543+0000", 2024-03-19T11:30:44.728 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1330000000000001e-05 2024-03-19T11:30:44.728 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.728 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.728 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:44.728 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757745+0000", 2024-03-19T11:30:44.728 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020259799999999999 2024-03-19T11:30:44.728 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.728 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.728 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op must wait for map", 2024-03-19T11:30:44.729 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757760+0000", 2024-03-19T11:30:44.729 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4647000000000001e-05 2024-03-19T11:30:44.729 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.729 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.729 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:44.729 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763972+0000", 2024-03-19T11:30:44.729 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062126480000000003 2024-03-19T11:30:44.729 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.729 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.729 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:44.729 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.764547+0000", 2024-03-19T11:30:44.729 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057440200000000001 2024-03-19T11:30:44.729 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.729 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.730 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:44.730 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.772488+0000", 2024-03-19T11:30:44.730 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079410130000000002 2024-03-19T11:30:44.730 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:44.730 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:44.730 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:44.730 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.730 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.730 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:849 4.1 4:866f3dcc:::smithi03829856-31:head [sparse-read 0~1359577] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:44.730 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.757583+0000", 2024-03-19T11:30:44.730 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 35.969070541000001, 2024-03-19T11:30:44.730 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016241202, 2024-03-19T11:30:44.730 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:44.730 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:44.730 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:44.731 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:44.731 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:44.731 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 849 2024-03-19T11:30:44.731 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.731 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:44.731 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.731 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:44.731 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757583+0000", 2024-03-19T11:30:44.731 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:44.731 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.731 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.731 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:44.731 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757583+0000", 2024-03-19T11:30:44.732 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:44.732 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.732 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.732 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:44.732 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757584+0000", 2024-03-19T11:30:44.732 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.077e-06 2024-03-19T11:30:44.732 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.732 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.732 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:44.732 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757593+0000", 2024-03-19T11:30:44.732 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3059999999999998e-06 2024-03-19T11:30:44.732 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.732 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.733 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:44.734 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757608+0000", 2024-03-19T11:30:44.734 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5526999999999999e-05 2024-03-19T11:30:44.734 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.734 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.734 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:44.734 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757618+0000", 2024-03-19T11:30:44.734 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5999999999999996e-06 2024-03-19T11:30:44.734 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.734 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.734 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:44.735 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757886+0000", 2024-03-19T11:30:44.735 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026839099999999999 2024-03-19T11:30:44.735 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.735 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.735 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:44.735 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757899+0000", 2024-03-19T11:30:44.735 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3004e-05 2024-03-19T11:30:44.735 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.735 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.735 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:44.735 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.772651+0000", 2024-03-19T11:30:44.735 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014751702 2024-03-19T11:30:44.735 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.736 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.736 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:44.736 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.772924+0000", 2024-03-19T11:30:44.736 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027314199999999998 2024-03-19T11:30:44.736 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.736 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.736 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:44.736 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.773825+0000", 2024-03-19T11:30:44.736 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00090045299999999995 2024-03-19T11:30:44.736 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:44.736 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:44.736 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:44.736 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.736 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.736 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:850 4.1 4:866f3dcc:::smithi03829856-31:head [read 0~2941192,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:44.737 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.757636+0000", 2024-03-19T11:30:44.737 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 35.969017870999998, 2024-03-19T11:30:44.737 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018834801000000002, 2024-03-19T11:30:44.737 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:44.737 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:44.737 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:44.737 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:44.737 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:44.737 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 850 2024-03-19T11:30:44.737 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.737 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:44.737 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.737 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:44.737 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757636+0000", 2024-03-19T11:30:44.738 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:44.738 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.738 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:44.738 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757636+0000", 2024-03-19T11:30:44.738 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:44.738 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.738 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:44.738 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757637+0000", 2024-03-19T11:30:44.738 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5999999999999991e-07 2024-03-19T11:30:44.738 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.739 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:44.739 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757643+0000", 2024-03-19T11:30:44.739 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3199999999999996e-06 2024-03-19T11:30:44.739 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.739 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.739 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:44.739 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757670+0000", 2024-03-19T11:30:44.739 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6682999999999998e-05 2024-03-19T11:30:44.739 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.739 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.739 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:44.739 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757679+0000", 2024-03-19T11:30:44.739 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.986e-06 2024-03-19T11:30:44.739 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.739 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.740 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:44.740 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757961+0000", 2024-03-19T11:30:44.740 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028230799999999999 2024-03-19T11:30:44.740 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.740 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.740 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:44.740 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757971+0000", 2024-03-19T11:30:44.740 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.38e-06 2024-03-19T11:30:44.740 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.740 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.740 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:44.740 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.774079+0000", 2024-03-19T11:30:44.740 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016108573000000001 2024-03-19T11:30:44.740 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.741 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.741 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:44.741 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.774306+0000", 2024-03-19T11:30:44.741 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022637499999999999 2024-03-19T11:30:44.741 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.741 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.741 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:44.741 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776471+0000", 2024-03-19T11:30:44.741 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021652160000000002 2024-03-19T11:30:44.741 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:44.741 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:44.741 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:44.741 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.741 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.741 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:851 4.9 4:953e720b:::smithi03829856-45:head [write 591613~454162,stat] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:44.742 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.763514+0000", 2024-03-19T11:30:44.742 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 35.963139679999998, 2024-03-19T11:30:44.742 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027825888999999999, 2024-03-19T11:30:44.742 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:44.742 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:44.742 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:44.742 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:44.742 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:44.742 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 851 2024-03-19T11:30:44.742 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.742 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:44.742 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.742 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:44.742 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763514+0000", 2024-03-19T11:30:44.742 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:44.743 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.743 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.743 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:44.743 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763514+0000", 2024-03-19T11:30:44.743 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:44.743 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.743 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.743 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:44.743 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763517+0000", 2024-03-19T11:30:44.743 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8499999999999998e-06 2024-03-19T11:30:44.743 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.743 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.743 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:44.743 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.764714+0000", 2024-03-19T11:30:44.743 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011971270000000001 2024-03-19T11:30:44.744 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.744 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.744 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:44.744 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.764745+0000", 2024-03-19T11:30:44.744 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0546999999999997e-05 2024-03-19T11:30:44.744 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.744 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.744 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:44.745 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.764767+0000", 2024-03-19T11:30:44.745 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2286999999999999e-05 2024-03-19T11:30:44.745 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.745 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.745 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:44.745 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.773924+0000", 2024-03-19T11:30:44.745 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0091567360000000004 2024-03-19T11:30:44.745 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.745 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.745 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:44.745 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.774217+0000", 2024-03-19T11:30:44.745 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029350099999999999 2024-03-19T11:30:44.745 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.745 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.746 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:44.746 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.774770+0000", 2024-03-19T11:30:44.746 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00055288600000000005 2024-03-19T11:30:44.746 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.746 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.746 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:44.746 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.786413+0000", 2024-03-19T11:30:44.746 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011643229999999999 2024-03-19T11:30:44.746 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.746 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.746 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:44.746 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791104+0000", 2024-03-19T11:30:44.746 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0046900830000000003 2024-03-19T11:30:44.746 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.746 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.747 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:44.747 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791248+0000", 2024-03-19T11:30:44.747 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014410200000000001 2024-03-19T11:30:44.747 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.747 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.747 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:44.747 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791340+0000", 2024-03-19T11:30:44.747 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2540000000000005e-05 2024-03-19T11:30:44.747 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:44.747 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:44.747 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:44.747 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.747 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.747 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:852 4.9 4:953e720b:::smithi03829856-45:head [write 1719614~417353] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:44.747 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.769707+0000", 2024-03-19T11:30:44.748 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 35.956946653999999, 2024-03-19T11:30:44.748 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022476643000000001, 2024-03-19T11:30:44.748 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:44.748 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:44.748 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:44.748 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:44.748 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:44.748 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 852 2024-03-19T11:30:44.748 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.748 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:44.748 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.748 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:44.748 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.769707+0000", 2024-03-19T11:30:44.748 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:44.748 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.749 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.749 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:44.749 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.769707+0000", 2024-03-19T11:30:44.749 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:44.749 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.749 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.749 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:44.749 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.769710+0000", 2024-03-19T11:30:44.749 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6230000000000001e-06 2024-03-19T11:30:44.749 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.749 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.749 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:44.749 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.770385+0000", 2024-03-19T11:30:44.749 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00067504900000000005 2024-03-19T11:30:44.749 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.750 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.750 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:44.750 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.770419+0000", 2024-03-19T11:30:44.750 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3528999999999997e-05 2024-03-19T11:30:44.750 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.750 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.750 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:44.750 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.770436+0000", 2024-03-19T11:30:44.750 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7615999999999999e-05 2024-03-19T11:30:44.750 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.750 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.750 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:44.750 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.781327+0000", 2024-03-19T11:30:44.750 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010891343 2024-03-19T11:30:44.751 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.751 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.751 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:44.751 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.781616+0000", 2024-03-19T11:30:44.751 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028857399999999999 2024-03-19T11:30:44.751 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.751 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.751 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:44.751 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.782125+0000", 2024-03-19T11:30:44.751 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00050941599999999995 2024-03-19T11:30:44.751 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.751 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.751 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:44.752 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.786606+0000", 2024-03-19T11:30:44.752 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044809560000000003 2024-03-19T11:30:44.752 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.752 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.752 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:44.752 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791995+0000", 2024-03-19T11:30:44.752 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053889150000000002 2024-03-19T11:30:44.752 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.752 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.752 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:44.752 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.792084+0000", 2024-03-19T11:30:44.752 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8633000000000003e-05 2024-03-19T11:30:44.752 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.752 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.753 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:44.753 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.792184+0000", 2024-03-19T11:30:44.753 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9988999999999994e-05 2024-03-19T11:30:44.753 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:44.753 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:44.753 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:44.753 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.753 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.753 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:853 4.9 4:953e720b:::smithi03829856-45:head [write 2570780~420833] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:44.753 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.775636+0000", 2024-03-19T11:30:44.753 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 35.951018032999997, 2024-03-19T11:30:44.753 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018795850999999999, 2024-03-19T11:30:44.753 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:44.753 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:44.753 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:44.754 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:44.754 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:44.754 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 853 2024-03-19T11:30:44.754 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.754 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:44.754 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.754 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:44.754 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.775636+0000", 2024-03-19T11:30:44.754 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:44.754 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.754 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.754 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:44.754 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.775636+0000", 2024-03-19T11:30:44.754 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:44.755 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.755 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.755 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:44.755 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.775639+0000", 2024-03-19T11:30:44.755 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4529999999999998e-06 2024-03-19T11:30:44.755 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.755 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.755 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:44.755 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776492+0000", 2024-03-19T11:30:44.755 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00085245000000000004 2024-03-19T11:30:44.755 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.755 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.755 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:44.755 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776526+0000", 2024-03-19T11:30:44.755 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4090000000000001e-05 2024-03-19T11:30:44.756 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.756 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.756 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:44.756 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776550+0000", 2024-03-19T11:30:44.756 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4139999999999999e-05 2024-03-19T11:30:44.756 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.756 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.756 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:44.756 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.786752+0000", 2024-03-19T11:30:44.756 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010201834 2024-03-19T11:30:44.756 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.756 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.756 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:44.756 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.786994+0000", 2024-03-19T11:30:44.757 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000241952 2024-03-19T11:30:44.757 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.757 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.757 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:44.757 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.787301+0000", 2024-03-19T11:30:44.757 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00030690699999999998 2024-03-19T11:30:44.757 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.757 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.757 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:44.757 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791965+0000", 2024-03-19T11:30:44.757 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0046644379999999999 2024-03-19T11:30:44.757 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.757 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.757 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:44.757 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.794165+0000", 2024-03-19T11:30:44.758 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021995389999999999 2024-03-19T11:30:44.758 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.758 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.758 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:44.758 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.794313+0000", 2024-03-19T11:30:44.758 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000148369 2024-03-19T11:30:44.758 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.758 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.758 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:44.758 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.794432+0000", 2024-03-19T11:30:44.758 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011867900000000001 2024-03-19T11:30:44.758 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:44.758 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:44.758 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:44.759 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.759 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:854 4.9 4:953e720b:::smithi03829856-45:head [setxattr _header (56) in=7b,truncate 2991613] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:44.759 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.776592+0000", 2024-03-19T11:30:44.759 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 35.950062311000003, 2024-03-19T11:30:44.759 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022815621000000001, 2024-03-19T11:30:44.759 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:44.759 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:44.759 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:44.759 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:44.759 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:44.759 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 854 2024-03-19T11:30:44.759 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.759 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:44.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.759 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:44.760 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776592+0000", 2024-03-19T11:30:44.760 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:44.760 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.760 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.760 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:44.760 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776592+0000", 2024-03-19T11:30:44.760 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:44.760 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.760 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.760 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:44.760 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776594+0000", 2024-03-19T11:30:44.760 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.95e-06 2024-03-19T11:30:44.760 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.760 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.760 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:44.761 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776603+0000", 2024-03-19T11:30:44.761 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2399999999999996e-06 2024-03-19T11:30:44.761 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.761 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.761 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:44.761 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776623+0000", 2024-03-19T11:30:44.761 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.035e-05 2024-03-19T11:30:44.761 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.761 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.761 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:44.761 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776639+0000", 2024-03-19T11:30:44.761 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5642999999999999e-05 2024-03-19T11:30:44.761 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.761 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.762 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:44.762 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.792324+0000", 2024-03-19T11:30:44.762 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015685075999999999 2024-03-19T11:30:44.762 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.762 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.762 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:44.762 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.792599+0000", 2024-03-19T11:30:44.762 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000275264 2024-03-19T11:30:44.762 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.762 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.762 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:44.762 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.793092+0000", 2024-03-19T11:30:44.762 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00049264600000000001 2024-03-19T11:30:44.762 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.763 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:44.763 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.794668+0000", 2024-03-19T11:30:44.763 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001576072 2024-03-19T11:30:44.763 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.763 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:44.763 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.799082+0000", 2024-03-19T11:30:44.763 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044145190000000004 2024-03-19T11:30:44.763 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.763 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:44.763 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.799249+0000", 2024-03-19T11:30:44.763 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016701199999999999 2024-03-19T11:30:44.763 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.764 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:44.764 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.799407+0000", 2024-03-19T11:30:44.764 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000157849 2024-03-19T11:30:44.764 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:44.764 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:44.764 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:44.764 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.764 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:855 4.9 4:953e720b:::smithi03829856-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:44.764 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.776660+0000", 2024-03-19T11:30:44.764 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 35.949993718000002, 2024-03-19T11:30:44.764 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023419786000000001, 2024-03-19T11:30:44.764 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:44.765 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:44.765 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:44.765 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:44.765 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:44.765 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 855 2024-03-19T11:30:44.765 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.765 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:44.765 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.765 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:44.765 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776660+0000", 2024-03-19T11:30:44.765 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:44.765 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.765 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.765 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:44.765 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776660+0000", 2024-03-19T11:30:44.766 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:44.766 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.766 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.766 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:44.766 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776662+0000", 2024-03-19T11:30:44.766 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3430000000000001e-06 2024-03-19T11:30:44.766 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.766 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.766 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:44.766 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776668+0000", 2024-03-19T11:30:44.766 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3500000000000002e-06 2024-03-19T11:30:44.766 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.766 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.766 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:44.766 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776687+0000", 2024-03-19T11:30:44.767 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9273999999999999e-05 2024-03-19T11:30:44.767 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.767 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:44.767 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776705+0000", 2024-03-19T11:30:44.767 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7839999999999999e-05 2024-03-19T11:30:44.767 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.767 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:44.767 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.793874+0000", 2024-03-19T11:30:44.767 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017168597000000001 2024-03-19T11:30:44.767 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.767 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-03-19T11:30:44.767 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.794132+0000", 2024-03-19T11:30:44.768 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025790399999999999 2024-03-19T11:30:44.768 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.768 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.768 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:44.768 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.799504+0000", 2024-03-19T11:30:44.768 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053724159999999997 2024-03-19T11:30:44.768 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.768 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.768 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:44.768 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.799782+0000", 2024-03-19T11:30:44.768 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027785499999999997 2024-03-19T11:30:44.768 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.768 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.768 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:44.768 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.800080+0000", 2024-03-19T11:30:44.768 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029820699999999999 2024-03-19T11:30:44.769 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:44.769 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:44.769 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:44.769 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.769 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.769 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:871 4.d 4:b4ef97e0:::smithi03829856-3:head [write 642034~796262,stat] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:44.769 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:09.028192+0000", 2024-03-19T11:30:44.769 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 35.698461987000002, 2024-03-19T11:30:44.769 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013367027, 2024-03-19T11:30:44.769 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:44.769 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:44.769 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:44.769 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:44.769 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:44.769 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 871 2024-03-19T11:30:44.770 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.770 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:44.770 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.770 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:44.770 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.028192+0000", 2024-03-19T11:30:44.770 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:44.770 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.770 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.770 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:44.770 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.028192+0000", 2024-03-19T11:30:44.770 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:44.770 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.770 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.770 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:44.771 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.028194+0000", 2024-03-19T11:30:44.771 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3999999999999999e-06 2024-03-19T11:30:44.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.771 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:44.771 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.029358+0000", 2024-03-19T11:30:44.771 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011635549999999999 2024-03-19T11:30:44.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.771 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:44.771 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.029390+0000", 2024-03-19T11:30:44.771 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2215999999999998e-05 2024-03-19T11:30:44.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.772 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:44.772 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.029406+0000", 2024-03-19T11:30:44.772 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5543e-05 2024-03-19T11:30:44.772 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.772 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.772 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:44.772 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.029525+0000", 2024-03-19T11:30:44.772 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000119269 2024-03-19T11:30:44.772 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.772 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.772 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:44.772 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.029676+0000", 2024-03-19T11:30:44.772 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000150729 2024-03-19T11:30:44.772 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.772 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.772 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2", 2024-03-19T11:30:44.773 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.029994+0000", 2024-03-19T11:30:44.773 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00031876099999999998 2024-03-19T11:30:44.773 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.773 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.773 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:44.773 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.040441+0000", 2024-03-19T11:30:44.773 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010446446 2024-03-19T11:30:44.773 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.773 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.773 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:44.773 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.041327+0000", 2024-03-19T11:30:44.773 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00088595299999999998 2024-03-19T11:30:44.773 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.773 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.773 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:44.774 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.041409+0000", 2024-03-19T11:30:44.774 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2332999999999999e-05 2024-03-19T11:30:44.774 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.774 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.774 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:44.774 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.041559+0000", 2024-03-19T11:30:44.774 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000149822 2024-03-19T11:30:44.774 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:44.774 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:44.774 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:44.774 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.774 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.774 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:993 4.e 4:72163e4c:::smithi03829856-39:head [write 776490~714607] snapc 5=[5,2] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-03-19T11:30:44.774 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:12.362932+0000", 2024-03-19T11:30:44.775 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.363722293999999, 2024-03-19T11:30:44.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018349900999999998, 2024-03-19T11:30:44.775 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:44.775 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:44.775 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:44.775 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:44.775 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:44.775 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 993 2024-03-19T11:30:44.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.775 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:44.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:44.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.362932+0000", 2024-03-19T11:30:44.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:44.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.776 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.776 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:44.776 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.362932+0000", 2024-03-19T11:30:44.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:44.776 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.776 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.776 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:44.776 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.362935+0000", 2024-03-19T11:30:44.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.88e-06 2024-03-19T11:30:44.776 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.776 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.776 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:44.776 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.366756+0000", 2024-03-19T11:30:44.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003821217 2024-03-19T11:30:44.776 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.777 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.777 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:44.777 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.366796+0000", 2024-03-19T11:30:44.777 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9866000000000001e-05 2024-03-19T11:30:44.777 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.777 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.777 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:44.777 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.366813+0000", 2024-03-19T11:30:44.777 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.755e-05 2024-03-19T11:30:44.777 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.777 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.777 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:44.777 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.366993+0000", 2024-03-19T11:30:44.777 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000179605 2024-03-19T11:30:44.778 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.778 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.778 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:44.778 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.367167+0000", 2024-03-19T11:30:44.778 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017421200000000001 2024-03-19T11:30:44.778 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.778 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.778 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-19T11:30:44.778 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.367505+0000", 2024-03-19T11:30:44.778 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033800099999999999 2024-03-19T11:30:44.778 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.778 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:44.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.380361+0000", 2024-03-19T11:30:44.779 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012855961000000001 2024-03-19T11:30:44.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.779 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:44.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.380987+0000", 2024-03-19T11:30:44.779 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00062590499999999995 2024-03-19T11:30:44.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.779 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:44.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.381149+0000", 2024-03-19T11:30:44.779 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000162239 2024-03-19T11:30:44.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.779 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.780 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:44.780 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.381282+0000", 2024-03-19T11:30:44.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013246500000000001 2024-03-19T11:30:44.780 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:44.780 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:44.780 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:44.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.780 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1072 4.1 4:8401c1f0:::smithi03829856-43:head [write 494139~557703,stat] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:44.780 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:13.972007+0000", 2024-03-19T11:30:44.780 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 30.754646544, 2024-03-19T11:30:44.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.01935839, 2024-03-19T11:30:44.780 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:44.781 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:44.781 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:44.781 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:44.781 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:44.781 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1072 2024-03-19T11:30:44.781 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.781 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:44.781 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.781 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:44.781 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.972007+0000", 2024-03-19T11:30:44.781 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:44.781 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.781 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.782 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:44.782 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.972007+0000", 2024-03-19T11:30:44.782 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:44.782 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.782 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.782 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:44.782 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.972011+0000", 2024-03-19T11:30:44.782 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9360000000000001e-06 2024-03-19T11:30:44.782 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.782 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.782 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:44.782 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.972881+0000", 2024-03-19T11:30:44.782 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00086922699999999998 2024-03-19T11:30:44.783 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.783 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:44.783 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.972917+0000", 2024-03-19T11:30:44.783 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6359999999999997e-05 2024-03-19T11:30:44.783 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.783 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:44.783 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.972934+0000", 2024-03-19T11:30:44.783 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7249e-05 2024-03-19T11:30:44.783 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.783 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:44.783 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.973132+0000", 2024-03-19T11:30:44.784 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019760200000000001 2024-03-19T11:30:44.784 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.784 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.784 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:44.784 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.973303+0000", 2024-03-19T11:30:44.784 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017142899999999999 2024-03-19T11:30:44.784 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.784 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.784 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:44.784 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.973619+0000", 2024-03-19T11:30:44.784 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00031594700000000002 2024-03-19T11:30:44.784 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.784 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.784 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:44.784 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.985856+0000", 2024-03-19T11:30:44.784 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012236349000000001 2024-03-19T11:30:44.785 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.785 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.785 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:44.785 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.991132+0000", 2024-03-19T11:30:44.785 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052764960000000003 2024-03-19T11:30:44.785 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.785 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.785 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:44.785 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.991241+0000", 2024-03-19T11:30:44.785 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000108842 2024-03-19T11:30:44.785 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.785 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.785 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:44.785 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.991366+0000", 2024-03-19T11:30:44.785 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012495299999999999 2024-03-19T11:30:44.786 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:44.786 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:44.786 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:44.786 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.786 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.786 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1162 4.1 4:8401c1f0:::smithi03829856-43:head [write 594981~609197] snapc 8=[8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-03-19T11:30:44.786 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.838981+0000", 2024-03-19T11:30:44.786 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.887673189000001, 2024-03-19T11:30:44.786 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027704349999999999, 2024-03-19T11:30:44.786 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:44.786 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:44.786 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:44.786 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:44.786 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:44.786 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1162 2024-03-19T11:30:44.787 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.787 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:44.787 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.787 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:44.787 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.838981+0000", 2024-03-19T11:30:44.787 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:44.787 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.787 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.787 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:44.787 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.838981+0000", 2024-03-19T11:30:44.787 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:44.787 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.787 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.787 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:44.788 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.838983+0000", 2024-03-19T11:30:44.788 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5399999999999998e-06 2024-03-19T11:30:44.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.788 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:44.788 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841322+0000", 2024-03-19T11:30:44.788 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002338592 2024-03-19T11:30:44.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.788 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:44.788 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841371+0000", 2024-03-19T11:30:44.788 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9123999999999999e-05 2024-03-19T11:30:44.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.788 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:44.789 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841399+0000", 2024-03-19T11:30:44.789 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7718000000000001e-05 2024-03-19T11:30:44.789 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.789 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.789 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:44.789 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841577+0000", 2024-03-19T11:30:44.789 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000178097 2024-03-19T11:30:44.789 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.789 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.789 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:44.789 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841825+0000", 2024-03-19T11:30:44.789 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00024824299999999999 2024-03-19T11:30:44.789 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.789 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.789 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:44.790 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842227+0000", 2024-03-19T11:30:44.790 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00040212199999999998 2024-03-19T11:30:44.790 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.790 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.790 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:44.790 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.865182+0000", 2024-03-19T11:30:44.790 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022954810999999999 2024-03-19T11:30:44.790 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.790 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.790 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:44.790 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.866360+0000", 2024-03-19T11:30:44.790 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001178449 2024-03-19T11:30:44.790 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.790 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.790 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:44.791 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.866538+0000", 2024-03-19T11:30:44.791 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017791000000000001 2024-03-19T11:30:44.791 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.791 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.791 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:44.791 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.866685+0000", 2024-03-19T11:30:44.791 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000146744 2024-03-19T11:30:44.791 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:44.791 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:44.791 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:44.791 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.791 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.791 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1163 4.1 4:8401c1f0:::smithi03829856-43:head [write 1974123~745028] snapc 8=[8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-03-19T11:30:44.791 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.851879+0000", 2024-03-19T11:30:44.792 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.874775067000002, 2024-03-19T11:30:44.792 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015531774, 2024-03-19T11:30:44.792 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:44.792 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:44.792 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:44.792 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:44.792 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:44.792 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1163 2024-03-19T11:30:44.792 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.792 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:44.792 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.792 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:44.792 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.851879+0000", 2024-03-19T11:30:44.792 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:44.792 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.793 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:44.793 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.851879+0000", 2024-03-19T11:30:44.793 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:44.793 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.793 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:44.793 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.851882+0000", 2024-03-19T11:30:44.793 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6960000000000001e-06 2024-03-19T11:30:44.793 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.793 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:44.793 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.853204+0000", 2024-03-19T11:30:44.793 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0013226620000000001 2024-03-19T11:30:44.793 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.794 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.794 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:44.794 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.853231+0000", 2024-03-19T11:30:44.794 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6775e-05 2024-03-19T11:30:44.794 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.794 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.794 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:44.794 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.853246+0000", 2024-03-19T11:30:44.794 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5403000000000001e-05 2024-03-19T11:30:44.794 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.794 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.794 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:44.794 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.853363+0000", 2024-03-19T11:30:44.794 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000116296 2024-03-19T11:30:44.794 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.795 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:44.795 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.853572+0000", 2024-03-19T11:30:44.795 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020931500000000001 2024-03-19T11:30:44.795 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.795 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:44.795 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.853797+0000", 2024-03-19T11:30:44.795 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022544700000000001 2024-03-19T11:30:44.795 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.795 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:44.795 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.865195+0000", 2024-03-19T11:30:44.795 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011397981999999999 2024-03-19T11:30:44.795 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.796 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.796 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:44.796 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.866916+0000", 2024-03-19T11:30:44.796 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017204449999999999 2024-03-19T11:30:44.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.796 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.796 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:44.796 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.867129+0000", 2024-03-19T11:30:44.796 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021317999999999999 2024-03-19T11:30:44.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.796 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.796 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:44.796 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.867411+0000", 2024-03-19T11:30:44.796 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028157300000000002 2024-03-19T11:30:44.796 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:44.796 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:44.797 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:44.797 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.797 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.797 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1205 4.1 4:8401c1f0:::smithi03829856-43:head [assert-version v50,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:44.797 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:18.112028+0000", 2024-03-19T11:30:44.797 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 26.614625932999999, 2024-03-19T11:30:44.797 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013579206, 2024-03-19T11:30:44.797 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:44.797 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:44.797 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:44.797 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:44.797 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:44.797 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1205 2024-03-19T11:30:44.797 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.797 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:44.798 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.798 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:44.798 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112028+0000", 2024-03-19T11:30:44.798 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:44.798 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.798 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.798 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:44.798 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112028+0000", 2024-03-19T11:30:44.798 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:44.798 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.798 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.798 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:44.798 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112030+0000", 2024-03-19T11:30:44.798 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4549999999999998e-06 2024-03-19T11:30:44.799 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.799 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.799 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:44.799 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112043+0000", 2024-03-19T11:30:44.799 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2721999999999999e-05 2024-03-19T11:30:44.799 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.799 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.799 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:44.799 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112075+0000", 2024-03-19T11:30:44.799 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.205e-05 2024-03-19T11:30:44.799 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.799 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.799 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:44.799 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112091+0000", 2024-03-19T11:30:44.799 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5364000000000001e-05 2024-03-19T11:30:44.799 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.800 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:44.800 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112223+0000", 2024-03-19T11:30:44.800 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013224100000000001 2024-03-19T11:30:44.800 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.800 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:44.800 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112393+0000", 2024-03-19T11:30:44.800 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000169681 2024-03-19T11:30:44.800 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.800 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:44.800 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.120919+0000", 2024-03-19T11:30:44.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085260140000000002 2024-03-19T11:30:44.801 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.801 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.801 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:44.801 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125207+0000", 2024-03-19T11:30:44.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004288722 2024-03-19T11:30:44.801 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.801 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.801 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:44.801 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125343+0000", 2024-03-19T11:30:44.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013558400000000001 2024-03-19T11:30:44.801 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.801 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.801 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:44.801 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125497+0000", 2024-03-19T11:30:44.802 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000153678 2024-03-19T11:30:44.802 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.802 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.802 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:44.802 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125607+0000", 2024-03-19T11:30:44.802 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110695 2024-03-19T11:30:44.802 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:44.802 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:44.802 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:44.802 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.802 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.802 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1208 4.1 4:866f3dcc:::smithi03829856-31:head [read 0~2941192,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:44.802 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:18.112252+0000", 2024-03-19T11:30:44.802 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 26.614402363, 2024-03-19T11:30:44.802 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.01291702, 2024-03-19T11:30:44.803 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:44.803 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:44.803 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:44.803 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:44.803 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:44.803 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1208 2024-03-19T11:30:44.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.803 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:44.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.803 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:44.803 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112252+0000", 2024-03-19T11:30:44.803 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:44.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.803 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:44.804 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112252+0000", 2024-03-19T11:30:44.804 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:44.804 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.804 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.804 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:44.804 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112253+0000", 2024-03-19T11:30:44.804 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6299999999999993e-07 2024-03-19T11:30:44.804 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.804 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.804 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:44.804 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112257+0000", 2024-03-19T11:30:44.804 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4360000000000004e-06 2024-03-19T11:30:44.804 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.804 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.804 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:44.805 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112272+0000", 2024-03-19T11:30:44.805 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.518e-05 2024-03-19T11:30:44.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.805 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.805 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:44.805 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112281+0000", 2024-03-19T11:30:44.805 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.422e-06 2024-03-19T11:30:44.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.805 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.805 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:44.805 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.122861+0000", 2024-03-19T11:30:44.805 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010580523 2024-03-19T11:30:44.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.805 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.805 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:44.806 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123148+0000", 2024-03-19T11:30:44.806 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028714499999999999 2024-03-19T11:30:44.806 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.806 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.806 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:44.806 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125169+0000", 2024-03-19T11:30:44.806 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0020203510000000001 2024-03-19T11:30:44.806 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:44.806 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:44.806 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:44.806 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.806 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.806 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1243 4.9 4:9ef41243:::smithi03829856-5:head [tier-promote] snapc 9=[9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:44.807 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:18.345759+0000", 2024-03-19T11:30:44.807 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 26.380894752, 2024-03-19T11:30:44.807 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018400962999999999, 2024-03-19T11:30:44.807 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:44.807 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:44.807 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:44.807 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:44.807 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:44.807 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1243 2024-03-19T11:30:44.807 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.807 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:44.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.807 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:44.807 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.345759+0000", 2024-03-19T11:30:44.807 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:44.808 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.808 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.808 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:44.808 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.345759+0000", 2024-03-19T11:30:44.808 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:44.808 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.808 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.808 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:44.808 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.345763+0000", 2024-03-19T11:30:44.808 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3790000000000002e-06 2024-03-19T11:30:44.808 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.808 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.808 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:44.808 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.345776+0000", 2024-03-19T11:30:44.808 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3784000000000001e-05 2024-03-19T11:30:44.808 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.809 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.809 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:44.809 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.345812+0000", 2024-03-19T11:30:44.809 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5599999999999998e-05 2024-03-19T11:30:44.809 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.809 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.809 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:44.809 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.345829+0000", 2024-03-19T11:30:44.809 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6722999999999999e-05 2024-03-19T11:30:44.809 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.809 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.809 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:44.809 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.345964+0000", 2024-03-19T11:30:44.809 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013554800000000001 2024-03-19T11:30:44.809 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.810 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.810 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:44.810 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.346133+0000", 2024-03-19T11:30:44.810 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000169211 2024-03-19T11:30:44.810 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.810 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.810 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:44.810 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.364160+0000", 2024-03-19T11:30:44.810 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018026718000000001 2024-03-19T11:30:44.810 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:44.810 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:44.810 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:44.810 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.810 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.810 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1271 4.9 4:953e720b:::smithi03829856-45:2 [read 795136~1008128] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-03-19T11:30:44.811 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:21.868056+0000", 2024-03-19T11:30:44.811 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.858597683999999, 2024-03-19T11:30:44.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.047270591000000001, 2024-03-19T11:30:44.811 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:44.811 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:44.811 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:44.811 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:44.811 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:44.811 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1271 2024-03-19T11:30:44.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.811 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:44.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.811 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:44.811 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868056+0000", 2024-03-19T11:30:44.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:44.812 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.812 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.812 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:44.812 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868056+0000", 2024-03-19T11:30:44.812 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:44.812 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.812 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.812 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:44.812 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868059+0000", 2024-03-19T11:30:44.812 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0469999999999998e-06 2024-03-19T11:30:44.812 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.812 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.812 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:44.812 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868072+0000", 2024-03-19T11:30:44.812 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2700999999999999e-05 2024-03-19T11:30:44.813 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.813 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.813 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:44.813 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868106+0000", 2024-03-19T11:30:44.813 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4152000000000001e-05 2024-03-19T11:30:44.813 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.813 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.813 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:44.813 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868126+0000", 2024-03-19T11:30:44.813 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9386999999999999e-05 2024-03-19T11:30:44.813 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.813 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.813 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:44.813 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.911447+0000", 2024-03-19T11:30:44.813 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.043321164000000002 2024-03-19T11:30:44.814 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.814 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.814 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:44.814 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.911762+0000", 2024-03-19T11:30:44.814 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00031492400000000001 2024-03-19T11:30:44.814 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.814 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.814 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:44.814 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.915327+0000", 2024-03-19T11:30:44.814 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003565216 2024-03-19T11:30:44.814 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:44.814 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:44.814 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:44.814 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.815 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.815 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1617 4.e 4:7773c291:::smithi03829856-11:head [write 450454~642026,stat] snapc 10=[10,f,e,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-03-19T11:30:44.815 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:35.152724+0000", 2024-03-19T11:30:44.815 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.5739294899999994, 2024-03-19T11:30:44.815 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016139337, 2024-03-19T11:30:44.815 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:44.815 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:44.815 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:44.815 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:44.815 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:44.815 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1617 2024-03-19T11:30:44.815 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.815 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:44.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:44.816 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.152724+0000", 2024-03-19T11:30:44.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:44.816 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:44.816 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.152724+0000", 2024-03-19T11:30:44.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:44.816 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:44.816 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.152727+0000", 2024-03-19T11:30:44.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8399999999999999e-06 2024-03-19T11:30:44.817 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.817 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.817 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:44.817 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.153790+0000", 2024-03-19T11:30:44.817 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010623849999999999 2024-03-19T11:30:44.817 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.817 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.817 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:44.817 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.153825+0000", 2024-03-19T11:30:44.817 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5280000000000001e-05 2024-03-19T11:30:44.817 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.817 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.817 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:44.817 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.153842+0000", 2024-03-19T11:30:44.818 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6942000000000001e-05 2024-03-19T11:30:44.818 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.818 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.818 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:44.818 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.154018+0000", 2024-03-19T11:30:44.818 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017568 2024-03-19T11:30:44.818 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.818 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.818 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:44.818 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.154302+0000", 2024-03-19T11:30:44.818 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028463400000000003 2024-03-19T11:30:44.818 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.818 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-19T11:30:44.819 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.154811+0000", 2024-03-19T11:30:44.819 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00050833300000000003 2024-03-19T11:30:44.819 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.819 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:44.819 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.168286+0000", 2024-03-19T11:30:44.819 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013475354 2024-03-19T11:30:44.819 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.819 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:44.819 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.168631+0000", 2024-03-19T11:30:44.819 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00034488800000000001 2024-03-19T11:30:44.819 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.820 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.820 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:44.820 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.168749+0000", 2024-03-19T11:30:44.820 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011799900000000001 2024-03-19T11:30:44.820 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.820 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.820 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:44.820 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.168864+0000", 2024-03-19T11:30:44.820 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011500199999999999 2024-03-19T11:30:44.820 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:44.820 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:44.821 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:44.821 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.821 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.821 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1679 4.e 4:72163e4c:::smithi03829856-39:head [write 517682~543022,stat] snapc 10=[10,e,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-03-19T11:30:44.821 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:36.388669+0000", 2024-03-19T11:30:44.821 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 8.3379851570000003, 2024-03-19T11:30:44.821 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016197706999999999, 2024-03-19T11:30:44.821 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:44.821 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:44.821 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:44.821 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:44.821 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:44.821 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1679 2024-03-19T11:30:44.821 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.821 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:44.822 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.822 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:44.822 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.388669+0000", 2024-03-19T11:30:44.822 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:44.822 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.822 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.822 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:44.822 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.388669+0000", 2024-03-19T11:30:44.822 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:44.822 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.822 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.822 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:44.822 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.388672+0000", 2024-03-19T11:30:44.822 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4130000000000002e-06 2024-03-19T11:30:44.823 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.823 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.823 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:44.823 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.389483+0000", 2024-03-19T11:30:44.823 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00081121800000000005 2024-03-19T11:30:44.823 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.823 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.823 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:44.823 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.389507+0000", 2024-03-19T11:30:44.823 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3482999999999999e-05 2024-03-19T11:30:44.823 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.823 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.823 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:44.823 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.389517+0000", 2024-03-19T11:30:44.823 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0251999999999999e-05 2024-03-19T11:30:44.823 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.824 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:44.824 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.389672+0000", 2024-03-19T11:30:44.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015465300000000001 2024-03-19T11:30:44.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.824 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:44.824 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.389839+0000", 2024-03-19T11:30:44.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016710800000000001 2024-03-19T11:30:44.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.824 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-19T11:30:44.824 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.390102+0000", 2024-03-19T11:30:44.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026344900000000002 2024-03-19T11:30:44.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.825 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.825 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:44.825 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.401899+0000", 2024-03-19T11:30:44.825 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011796836 2024-03-19T11:30:44.825 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.825 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.825 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:44.825 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.404530+0000", 2024-03-19T11:30:44.825 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026311949999999998 2024-03-19T11:30:44.825 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.825 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.825 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:44.825 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.404722+0000", 2024-03-19T11:30:44.825 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019109300000000001 2024-03-19T11:30:44.825 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:44.826 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:44.826 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:44.826 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.404867+0000", 2024-03-19T11:30:44.826 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000145007 2024-03-19T11:30:44.826 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:44.826 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:44.826 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:44.826 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:44.826 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:44.826 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:44.828 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-19T11:30:44.828 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:44.838+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:44.928 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:44.938+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:44.997 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:44.997 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:44.997 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:30:44.997 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:45.011 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-19T11:30:45.029 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:45.038+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:45.129 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:45.129 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:45.129 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:30:45.129 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:30:45.129 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:45.130 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:45.136+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:45.143 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-19T11:30:45.231 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:45.240+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:45.247 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-03-19T11:30:45.286 INFO:tasks.rados.rados.0.smithi038.stdout:1025: set_chunk oid 7 offset: 1510400 length: 1327616 target oid dcc48245af828a6ebde8b0a3299a4754dd97f74e9d30494cae6d8d1aec40fb87 offset: 0 snap -1 2024-03-19T11:30:45.286 INFO:tasks.rados.rados.0.smithi038.stdout:1026: write oid 31 current snap is 17 2024-03-19T11:30:45.286 INFO:tasks.rados.rados.0.smithi038.stdout:1026: seq_num 157 ranges {638699=541153,1770967=531722} 2024-03-19T11:30:45.293 INFO:tasks.rados.rados.0.smithi038.stdout:1026: writing smithi03829856-31 from 638699 to 1179852 tid 1 2024-03-19T11:30:45.299 INFO:tasks.rados.rados.0.smithi038.stdout:1026: writing smithi03829856-31 from 1770967 to 2302689 tid 2 2024-03-19T11:30:45.299 INFO:tasks.rados.rados.0.smithi038.stdout:1027: tier_promote oid 47 2024-03-19T11:30:45.299 INFO:tasks.rados.rados.0.smithi038.stdout:1020: finishing write tid 1 to smithi03829856-43 2024-03-19T11:30:45.299 INFO:tasks.rados.rados.0.smithi038.stdout:1028: write (excl) oid 35 current snap is 17 2024-03-19T11:30:45.299 INFO:tasks.rados.rados.0.smithi038.stdout:1028: seq_num 158 ranges {655506=616402,1829449=675380} 2024-03-19T11:30:45.306 INFO:tasks.rados.rados.0.smithi038.stdout:1028: writing smithi03829856-35 from 655506 to 1271908 tid 1 2024-03-19T11:30:45.314 INFO:tasks.rados.rados.0.smithi038.stdout:1028: writing smithi03829856-35 from 1829449 to 2504829 tid 2 2024-03-19T11:30:45.314 INFO:tasks.rados.rados.0.smithi038.stdout:1029: delete oid 41 current snap is 17 2024-03-19T11:30:45.314 INFO:tasks.rados.rados.0.smithi038.stdout:1020: finishing write tid 2 to smithi03829856-43 2024-03-19T11:30:45.314 INFO:tasks.rados.rados.0.smithi038.stdout:1020: finishing write tid 3 to smithi03829856-43 2024-03-19T11:30:45.314 INFO:tasks.rados.rados.0.smithi038.stdout:1020: finishing write tid 4 to smithi03829856-43 2024-03-19T11:30:45.314 INFO:tasks.rados.rados.0.smithi038.stdout:1020: oid 43 updating version 0 to 104 2024-03-19T11:30:45.314 INFO:tasks.rados.rados.0.smithi038.stdout:1020: oid 43 version 104 is already newer than 103 2024-03-19T11:30:45.314 INFO:tasks.rados.rados.0.smithi038.stdout:1020: oid 43 version 104 is already newer than 102 2024-03-19T11:30:45.315 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 43 v 104 (ObjNum 155 snap 17 seq_num 155) dirty exists 2024-03-19T11:30:45.315 INFO:tasks.rados.rados.0.smithi038.stdout:1020: left oid 43 (ObjNum 155 snap 17 seq_num 155) 2024-03-19T11:30:45.315 INFO:tasks.rados.rados.0.smithi038.stdout:1022: expect (ObjNum 72 snap 5 seq_num 72) 2024-03-19T11:30:45.332 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:45.340+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:45.333 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:45.333 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:30:45.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:30:45.333 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:30:45.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.333 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:844 4.7 e32/21 4:e41d47fb:::smithi03829856-14:head v 32'8, mlcod=28'5)", 2024-03-19T11:30:45.333 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.751447+0000", 2024-03-19T11:30:45.333 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 36.590946178000003, 2024-03-19T11:30:45.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012301835000000001, 2024-03-19T11:30:45.333 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:45.333 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:45.333 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:45.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:45.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.751447+0000", 2024-03-19T11:30:45.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:45.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:45.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.751447+0000", 2024-03-19T11:30:45.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:45.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:45.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.751447+0000", 2024-03-19T11:30:45.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4300000000000003e-07 2024-03-19T11:30:45.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.335 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:45.335 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.752747+0000", 2024-03-19T11:30:45.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012995699999999999 2024-03-19T11:30:45.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.335 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:45.336 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.752771+0000", 2024-03-19T11:30:45.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3592999999999999e-05 2024-03-19T11:30:45.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.336 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:45.336 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.752789+0000", 2024-03-19T11:30:45.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8703e-05 2024-03-19T11:30:45.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.336 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:45.336 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.752911+0000", 2024-03-19T11:30:45.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012139899999999999 2024-03-19T11:30:45.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:45.337 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.752949+0000", 2024-03-19T11:30:45.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8436999999999998e-05 2024-03-19T11:30:45.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:45.337 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763568+0000", 2024-03-19T11:30:45.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010618768000000001 2024-03-19T11:30:45.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:45.337 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763749+0000", 2024-03-19T11:30:45.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000180822 2024-03-19T11:30:45.337 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:45.338 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:45.338 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:45.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.338 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.338 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1047 4.8 4:1d58db11:::smithi03829856-18:head [write 443718~615603,stat] snapc 6=[6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:45.338 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:13.221766+0000", 2024-03-19T11:30:45.338 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.120627011000003, 2024-03-19T11:30:45.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020932596000000001, 2024-03-19T11:30:45.338 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:45.338 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:45.338 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:45.338 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:45.338 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:45.338 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1047 2024-03-19T11:30:45.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.339 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:45.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:45.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.221766+0000", 2024-03-19T11:30:45.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:45.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:45.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.221766+0000", 2024-03-19T11:30:45.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:45.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.340 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:45.340 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.221770+0000", 2024-03-19T11:30:45.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.67e-06 2024-03-19T11:30:45.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.340 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:45.340 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.223938+0000", 2024-03-19T11:30:45.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021683399999999999 2024-03-19T11:30:45.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.340 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:45.340 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.223979+0000", 2024-03-19T11:30:45.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1433000000000001e-05 2024-03-19T11:30:45.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.341 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:45.341 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.223998+0000", 2024-03-19T11:30:45.341 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8853000000000001e-05 2024-03-19T11:30:45.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.341 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:45.341 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.224150+0000", 2024-03-19T11:30:45.341 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000151679 2024-03-19T11:30:45.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.342 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:45.342 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.224362+0000", 2024-03-19T11:30:45.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021154499999999999 2024-03-19T11:30:45.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:45.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.224745+0000", 2024-03-19T11:30:45.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00038342399999999999 2024-03-19T11:30:45.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:45.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.241167+0000", 2024-03-19T11:30:45.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0164218 2024-03-19T11:30:45.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:45.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.242406+0000", 2024-03-19T11:30:45.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012390039999999999 2024-03-19T11:30:45.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:45.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.242575+0000", 2024-03-19T11:30:45.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000168925 2024-03-19T11:30:45.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:45.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.242699+0000", 2024-03-19T11:30:45.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012392300000000001 2024-03-19T11:30:45.344 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:45.344 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:45.344 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:45.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.344 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1048 4.8 4:1d58db11:::smithi03829856-18:head [write 1564138~470341] snapc 6=[6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:45.345 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:13.227287+0000", 2024-03-19T11:30:45.345 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.115105743000001, 2024-03-19T11:30:45.345 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015802897999999999, 2024-03-19T11:30:45.345 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:45.345 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:45.345 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:45.345 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:45.345 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:45.345 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1048 2024-03-19T11:30:45.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.345 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:45.345 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.345 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:45.345 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.227287+0000", 2024-03-19T11:30:45.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:45.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:45.346 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.227287+0000", 2024-03-19T11:30:45.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:45.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:45.346 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.227290+0000", 2024-03-19T11:30:45.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.88e-06 2024-03-19T11:30:45.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:45.347 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.228224+0000", 2024-03-19T11:30:45.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00093389600000000003 2024-03-19T11:30:45.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.347 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.347 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:45.347 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.228258+0000", 2024-03-19T11:30:45.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4282999999999997e-05 2024-03-19T11:30:45.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.347 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.347 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:45.347 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.228273+0000", 2024-03-19T11:30:45.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4833e-05 2024-03-19T11:30:45.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.347 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:45.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.231996+0000", 2024-03-19T11:30:45.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037223410000000001 2024-03-19T11:30:45.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:45.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.232143+0000", 2024-03-19T11:30:45.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014714900000000001 2024-03-19T11:30:45.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:45.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.232395+0000", 2024-03-19T11:30:45.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025201500000000002 2024-03-19T11:30:45.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.349 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.349 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:45.349 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.242848+0000", 2024-03-19T11:30:45.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010453393 2024-03-19T11:30:45.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.349 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.349 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:45.349 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.242872+0000", 2024-03-19T11:30:45.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3816e-05 2024-03-19T11:30:45.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.349 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.349 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:45.349 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.242966+0000", 2024-03-19T11:30:45.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4560000000000003e-05 2024-03-19T11:30:45.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.350 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:45.350 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.243090+0000", 2024-03-19T11:30:45.350 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012373199999999999 2024-03-19T11:30:45.350 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:45.350 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:45.350 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:45.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.350 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1072 4.1 e36/21 4:8401c1f0:::smithi03829856-43:head v 36'38, mlcod=36'35)", 2024-03-19T11:30:45.350 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:13.973829+0000", 2024-03-19T11:30:45.350 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 31.368564419999998, 2024-03-19T11:30:45.350 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013548782000000001, 2024-03-19T11:30:45.350 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:45.351 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:45.351 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:45.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:45.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.973829+0000", 2024-03-19T11:30:45.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:45.351 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:45.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.973829+0000", 2024-03-19T11:30:45.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:45.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.352 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.352 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:45.352 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.973831+0000", 2024-03-19T11:30:45.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0899999999999999e-06 2024-03-19T11:30:45.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:45.353 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.974489+0000", 2024-03-19T11:30:45.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00065813199999999997 2024-03-19T11:30:45.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:45.353 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.974523+0000", 2024-03-19T11:30:45.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3841999999999999e-05 2024-03-19T11:30:45.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:45.354 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.974544+0000", 2024-03-19T11:30:45.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0907000000000002e-05 2024-03-19T11:30:45.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.354 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:45.354 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.974697+0000", 2024-03-19T11:30:45.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015295599999999999 2024-03-19T11:30:45.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.354 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:45.354 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.974742+0000", 2024-03-19T11:30:45.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5735999999999997e-05 2024-03-19T11:30:45.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.354 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:45.355 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.987261+0000", 2024-03-19T11:30:45.355 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012518316999999999 2024-03-19T11:30:45.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.355 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:45.355 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.987377+0000", 2024-03-19T11:30:45.355 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000116802 2024-03-19T11:30:45.355 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:45.355 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:45.355 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:45.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.355 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1091 4.8 4:14611006:::smithi03829856-25:head [write 527753~708201] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:45.355 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:14.015839+0000", 2024-03-19T11:30:45.355 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 31.326554360999999, 2024-03-19T11:30:45.356 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016721983999999999, 2024-03-19T11:30:45.356 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:45.356 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:45.356 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:45.356 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:45.356 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:45.356 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1091 2024-03-19T11:30:45.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.356 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:45.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.356 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:45.356 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.015839+0000", 2024-03-19T11:30:45.356 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:45.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.357 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:45.357 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.015839+0000", 2024-03-19T11:30:45.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:45.357 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.357 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:45.357 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.015842+0000", 2024-03-19T11:30:45.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3569999999999998e-06 2024-03-19T11:30:45.357 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.357 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:45.357 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.016678+0000", 2024-03-19T11:30:45.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00083620299999999999 2024-03-19T11:30:45.357 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.358 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:45.358 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.016710+0000", 2024-03-19T11:30:45.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1337000000000002e-05 2024-03-19T11:30:45.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.358 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:45.358 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.016724+0000", 2024-03-19T11:30:45.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4323e-05 2024-03-19T11:30:45.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.358 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:45.358 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.016847+0000", 2024-03-19T11:30:45.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012304199999999999 2024-03-19T11:30:45.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:45.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.017023+0000", 2024-03-19T11:30:45.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000175945 2024-03-19T11:30:45.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:45.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.017324+0000", 2024-03-19T11:30:45.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00030151499999999998 2024-03-19T11:30:45.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:45.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.032104+0000", 2024-03-19T11:30:45.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014779232999999999 2024-03-19T11:30:45.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:45.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.032190+0000", 2024-03-19T11:30:45.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6304999999999994e-05 2024-03-19T11:30:45.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:45.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.032350+0000", 2024-03-19T11:30:45.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000159813 2024-03-19T11:30:45.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:45.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.032561+0000", 2024-03-19T11:30:45.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021091099999999999 2024-03-19T11:30:45.361 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:45.361 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:45.361 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:45.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.361 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1158 4.8 4:15708482:::smithi03829856-23:head [tier-promote] snapc 8=[8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-03-19T11:30:45.361 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.827517+0000", 2024-03-19T11:30:45.361 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.514876433000001, 2024-03-19T11:30:45.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.034096983999999997, 2024-03-19T11:30:45.361 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:45.362 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:45.362 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:45.362 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:45.362 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:45.362 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1158 2024-03-19T11:30:45.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.363 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:45.363 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.363 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:45.363 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827517+0000", 2024-03-19T11:30:45.363 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:45.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.364 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:45.364 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827517+0000", 2024-03-19T11:30:45.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:45.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.364 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:45.364 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827520+0000", 2024-03-19T11:30:45.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.45e-06 2024-03-19T11:30:45.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.364 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:45.364 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827535+0000", 2024-03-19T11:30:45.365 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.509e-05 2024-03-19T11:30:45.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:45.365 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827579+0000", 2024-03-19T11:30:45.365 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4211000000000001e-05 2024-03-19T11:30:45.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:45.365 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827603+0000", 2024-03-19T11:30:45.365 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3586000000000001e-05 2024-03-19T11:30:45.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:45.365 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827789+0000", 2024-03-19T11:30:45.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018580199999999999 2024-03-19T11:30:45.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.366 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:45.366 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.828077+0000", 2024-03-19T11:30:45.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028846500000000001 2024-03-19T11:30:45.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.366 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:45.366 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.861614+0000", 2024-03-19T11:30:45.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033536379999999998 2024-03-19T11:30:45.366 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:45.366 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:45.366 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:45.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.367 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1162 4.1 e39/21 4:8401c1f0:::smithi03829856-43:head v 39'47, mlcod=38'43)", 2024-03-19T11:30:45.367 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.842598+0000", 2024-03-19T11:30:45.367 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.499794662999999, 2024-03-19T11:30:45.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0233756, 2024-03-19T11:30:45.367 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:45.367 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:45.367 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:45.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.367 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:45.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842598+0000", 2024-03-19T11:30:45.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:45.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.368 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:45.368 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842598+0000", 2024-03-19T11:30:45.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:45.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.368 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:45.368 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842599+0000", 2024-03-19T11:30:45.369 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6000000000000003e-07 2024-03-19T11:30:45.369 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.369 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.369 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:45.369 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.843566+0000", 2024-03-19T11:30:45.369 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00096688700000000004 2024-03-19T11:30:45.369 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.369 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.370 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:45.370 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.843601+0000", 2024-03-19T11:30:45.370 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.502e-05 2024-03-19T11:30:45.370 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.370 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.370 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:45.370 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.843621+0000", 2024-03-19T11:30:45.370 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0305e-05 2024-03-19T11:30:45.370 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.370 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.370 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:45.371 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.843882+0000", 2024-03-19T11:30:45.371 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026027100000000002 2024-03-19T11:30:45.371 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.371 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.371 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:45.371 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.844022+0000", 2024-03-19T11:30:45.371 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014020100000000001 2024-03-19T11:30:45.371 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.371 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.371 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:45.371 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.865862+0000", 2024-03-19T11:30:45.371 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021839725000000001 2024-03-19T11:30:45.371 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.372 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:45.372 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.865974+0000", 2024-03-19T11:30:45.372 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011243099999999999 2024-03-19T11:30:45.372 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:45.372 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:45.372 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:45.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.373 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1209 4.8 4:1d58db11:::smithi03829856-18:head [write 645402~551297,stat] snapc 9=[9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:45.373 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:18.121657+0000", 2024-03-19T11:30:45.373 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.220735660999999, 2024-03-19T11:30:45.373 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.025074349999999999, 2024-03-19T11:30:45.373 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:45.373 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:45.373 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:45.373 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:45.374 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:45.374 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1209 2024-03-19T11:30:45.374 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.374 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:45.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.374 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:45.374 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.121657+0000", 2024-03-19T11:30:45.374 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:45.374 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.375 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.375 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:45.375 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.121657+0000", 2024-03-19T11:30:45.375 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:45.375 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.375 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.375 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:45.375 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.121661+0000", 2024-03-19T11:30:45.375 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0489999999999997e-06 2024-03-19T11:30:45.375 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.375 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.375 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:45.375 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123747+0000", 2024-03-19T11:30:45.375 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0020857720000000001 2024-03-19T11:30:45.375 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.376 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.376 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:45.376 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123779+0000", 2024-03-19T11:30:45.376 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2258000000000002e-05 2024-03-19T11:30:45.376 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.376 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.376 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:45.376 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123795+0000", 2024-03-19T11:30:45.376 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5177000000000001e-05 2024-03-19T11:30:45.376 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.376 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.376 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:45.376 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123949+0000", 2024-03-19T11:30:45.376 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015403699999999999 2024-03-19T11:30:45.376 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.377 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.377 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:45.377 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.124134+0000", 2024-03-19T11:30:45.377 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000185484 2024-03-19T11:30:45.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.377 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.377 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:45.377 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.124554+0000", 2024-03-19T11:30:45.377 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00042019699999999998 2024-03-19T11:30:45.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.377 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.377 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:45.377 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146267+0000", 2024-03-19T11:30:45.377 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021712838000000002 2024-03-19T11:30:45.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.378 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.378 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:45.378 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146424+0000", 2024-03-19T11:30:45.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015647299999999999 2024-03-19T11:30:45.378 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.378 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.378 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:45.378 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146597+0000", 2024-03-19T11:30:45.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017303699999999999 2024-03-19T11:30:45.378 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.378 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.378 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:45.378 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146732+0000", 2024-03-19T11:30:45.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013502800000000001 2024-03-19T11:30:45.379 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:45.379 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:45.379 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:45.379 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.379 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.379 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1210 4.8 4:1d58db11:::smithi03829856-18:head [write 1749569~492355] snapc 9=[9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:45.379 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:18.129257+0000", 2024-03-19T11:30:45.379 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.213136303999999, 2024-03-19T11:30:45.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018110444999999999, 2024-03-19T11:30:45.379 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:45.379 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:45.379 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:45.379 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:45.379 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:45.380 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1210 2024-03-19T11:30:45.380 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.380 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:45.380 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.380 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:45.380 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.129257+0000", 2024-03-19T11:30:45.380 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:45.380 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.380 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.380 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:45.380 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.129257+0000", 2024-03-19T11:30:45.380 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:45.380 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.380 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.380 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:45.381 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.129261+0000", 2024-03-19T11:30:45.381 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7119999999999999e-06 2024-03-19T11:30:45.381 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.381 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.381 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:45.381 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.129717+0000", 2024-03-19T11:30:45.381 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00045526300000000002 2024-03-19T11:30:45.381 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.381 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.381 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:45.381 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.129749+0000", 2024-03-19T11:30:45.381 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2700999999999997e-05 2024-03-19T11:30:45.381 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.381 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.381 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:45.382 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.129774+0000", 2024-03-19T11:30:45.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4592000000000001e-05 2024-03-19T11:30:45.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.382 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:45.382 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.134875+0000", 2024-03-19T11:30:45.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.005100477 2024-03-19T11:30:45.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.382 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:45.382 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.135035+0000", 2024-03-19T11:30:45.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016054200000000001 2024-03-19T11:30:45.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:45.383 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.135268+0000", 2024-03-19T11:30:45.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000233253 2024-03-19T11:30:45.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:45.383 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146872+0000", 2024-03-19T11:30:45.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011603732 2024-03-19T11:30:45.384 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.384 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.384 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:45.384 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.147103+0000", 2024-03-19T11:30:45.385 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023070299999999999 2024-03-19T11:30:45.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.385 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.385 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:45.385 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.147181+0000", 2024-03-19T11:30:45.385 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8567999999999997e-05 2024-03-19T11:30:45.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.385 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.385 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:45.385 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.147367+0000", 2024-03-19T11:30:45.385 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000185902 2024-03-19T11:30:45.385 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:45.385 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:45.385 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:45.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.386 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1259 4.1 e44/43 4:866f3dcc:::smithi03829856-31:head v 44'53, mlcod=0'0)", 2024-03-19T11:30:45.386 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:21.869109+0000", 2024-03-19T11:30:45.386 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.473284413999998, 2024-03-19T11:30:45.386 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014777017999999999, 2024-03-19T11:30:45.386 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:45.386 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:45.386 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:45.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.386 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:45.386 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.869109+0000", 2024-03-19T11:30:45.386 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:45.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.387 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:45.387 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.869109+0000", 2024-03-19T11:30:45.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:45.387 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.387 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:45.387 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.869110+0000", 2024-03-19T11:30:45.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0559999999999999e-06 2024-03-19T11:30:45.387 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.387 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:45.387 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.870408+0000", 2024-03-19T11:30:45.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012978499999999999 2024-03-19T11:30:45.387 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:45.388 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.870437+0000", 2024-03-19T11:30:45.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9638999999999999e-05 2024-03-19T11:30:45.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:45.388 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.870460+0000", 2024-03-19T11:30:45.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2517000000000001e-05 2024-03-19T11:30:45.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:45.388 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.870681+0000", 2024-03-19T11:30:45.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022102800000000001 2024-03-19T11:30:45.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.389 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.389 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:45.389 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.870735+0000", 2024-03-19T11:30:45.389 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4616e-05 2024-03-19T11:30:45.389 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.389 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.389 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:45.389 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.883773+0000", 2024-03-19T11:30:45.389 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013038088 2024-03-19T11:30:45.389 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.389 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.389 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:45.389 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.883886+0000", 2024-03-19T11:30:45.389 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000112224 2024-03-19T11:30:45.389 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:45.390 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:45.390 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:45.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.390 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.390 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1416 4.18 4:1dabc0f1:::smithi03829856-1:head [write 564735~613734,stat] snapc c=[c,b,a,6,5] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:45.390 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.949508+0000", 2024-03-19T11:30:45.390 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.392885084, 2024-03-19T11:30:45.390 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016253106, 2024-03-19T11:30:45.390 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:45.390 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:45.390 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:45.390 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:45.390 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:45.390 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1416 2024-03-19T11:30:45.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.391 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:45.391 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.391 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:45.391 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.949508+0000", 2024-03-19T11:30:45.391 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:45.391 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.391 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.391 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:45.391 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.949508+0000", 2024-03-19T11:30:45.391 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:45.391 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.391 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.391 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:45.391 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.949511+0000", 2024-03-19T11:30:45.391 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2370000000000001e-06 2024-03-19T11:30:45.392 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.392 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:45.392 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.950555+0000", 2024-03-19T11:30:45.392 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001043925 2024-03-19T11:30:45.392 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.392 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:45.392 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.950594+0000", 2024-03-19T11:30:45.392 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9351999999999999e-05 2024-03-19T11:30:45.392 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.392 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:45.392 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.950612+0000", 2024-03-19T11:30:45.392 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7524e-05 2024-03-19T11:30:45.393 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.393 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:45.393 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.950745+0000", 2024-03-19T11:30:45.393 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013291 2024-03-19T11:30:45.393 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.393 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:45.393 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.951003+0000", 2024-03-19T11:30:45.393 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025828400000000001 2024-03-19T11:30:45.393 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.393 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:45.393 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.951335+0000", 2024-03-19T11:30:45.394 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033130200000000001 2024-03-19T11:30:45.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.394 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.394 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:45.394 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.965379+0000", 2024-03-19T11:30:45.395 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014044581 2024-03-19T11:30:45.395 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.395 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.395 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:45.395 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.965502+0000", 2024-03-19T11:30:45.395 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012290099999999999 2024-03-19T11:30:45.395 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.395 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.395 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:45.395 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.965668+0000", 2024-03-19T11:30:45.395 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016619200000000001 2024-03-19T11:30:45.395 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.395 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.395 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:45.396 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.965761+0000", 2024-03-19T11:30:45.396 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2898000000000004e-05 2024-03-19T11:30:45.396 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:45.396 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:45.396 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:45.396 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.396 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.396 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1417 4.18 4:1dabc0f1:::smithi03829856-1:head [write 1828021~669034] snapc c=[c,b,a,6,5] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:45.396 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.958407+0000", 2024-03-19T11:30:45.396 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.383985682999999, 2024-03-19T11:30:45.396 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016467840000000001, 2024-03-19T11:30:45.396 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:45.396 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:45.396 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:45.397 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:45.397 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:45.397 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1417 2024-03-19T11:30:45.397 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.397 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:45.397 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.397 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:45.397 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.958407+0000", 2024-03-19T11:30:45.397 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:45.397 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.397 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.397 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:45.397 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.958407+0000", 2024-03-19T11:30:45.397 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:45.397 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.398 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.398 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:45.398 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.958410+0000", 2024-03-19T11:30:45.398 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9830000000000001e-06 2024-03-19T11:30:45.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.398 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.398 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:45.398 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.959442+0000", 2024-03-19T11:30:45.398 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001031957 2024-03-19T11:30:45.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.398 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.398 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:45.398 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.959475+0000", 2024-03-19T11:30:45.398 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3145999999999997e-05 2024-03-19T11:30:45.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.399 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.399 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:45.399 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.959494+0000", 2024-03-19T11:30:45.399 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8994000000000001e-05 2024-03-19T11:30:45.399 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.399 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.399 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:45.399 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.959717+0000", 2024-03-19T11:30:45.399 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000222272 2024-03-19T11:30:45.399 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.399 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.399 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:45.399 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.960015+0000", 2024-03-19T11:30:45.399 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029870599999999998 2024-03-19T11:30:45.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.400 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.400 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:45.400 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.960312+0000", 2024-03-19T11:30:45.400 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029640299999999999 2024-03-19T11:30:45.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.400 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.400 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:45.400 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.966435+0000", 2024-03-19T11:30:45.400 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061234999999999996 2024-03-19T11:30:45.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.401 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.401 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:45.401 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.974556+0000", 2024-03-19T11:30:45.401 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.008120759 2024-03-19T11:30:45.401 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.401 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.401 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:45.401 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.974729+0000", 2024-03-19T11:30:45.401 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000173191 2024-03-19T11:30:45.401 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.401 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.401 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:45.402 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.974875+0000", 2024-03-19T11:30:45.402 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014592899999999999 2024-03-19T11:30:45.402 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:45.402 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:45.402 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:45.402 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.402 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.402 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1419 4.18 4:1dabc0f1:::smithi03829856-1:head [setxattr _header (56) in=7b,truncate 3764735] snapc c=[c,b,a,6,5] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:45.402 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.968061+0000", 2024-03-19T11:30:45.402 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.374331823999999, 2024-03-19T11:30:45.402 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012422488000000001, 2024-03-19T11:30:45.402 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:45.402 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:45.403 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:45.403 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:45.403 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:45.403 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1419 2024-03-19T11:30:45.403 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.403 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:45.403 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.403 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:45.403 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968061+0000", 2024-03-19T11:30:45.403 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:45.403 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.403 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.403 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:45.403 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968061+0000", 2024-03-19T11:30:45.403 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:45.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.404 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.404 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:45.404 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968063+0000", 2024-03-19T11:30:45.404 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.767e-06 2024-03-19T11:30:45.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.404 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.404 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:45.404 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968074+0000", 2024-03-19T11:30:45.404 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0525e-05 2024-03-19T11:30:45.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.404 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.404 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:45.404 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968096+0000", 2024-03-19T11:30:45.404 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2926999999999999e-05 2024-03-19T11:30:45.405 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.405 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.405 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:45.405 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968113+0000", 2024-03-19T11:30:45.405 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6665000000000001e-05 2024-03-19T11:30:45.405 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.405 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.405 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:45.405 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.973218+0000", 2024-03-19T11:30:45.406 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0051044920000000004 2024-03-19T11:30:45.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.406 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:45.406 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.973511+0000", 2024-03-19T11:30:45.406 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029303899999999998 2024-03-19T11:30:45.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.406 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:45.406 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.973931+0000", 2024-03-19T11:30:45.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00041991400000000002 2024-03-19T11:30:45.407 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.407 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.407 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:45.407 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.977029+0000", 2024-03-19T11:30:45.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030983040000000001 2024-03-19T11:30:45.407 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.407 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.407 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:45.407 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.980091+0000", 2024-03-19T11:30:45.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003061851 2024-03-19T11:30:45.407 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.407 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.407 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:45.407 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.980285+0000", 2024-03-19T11:30:45.408 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000194279 2024-03-19T11:30:45.408 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.408 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:45.408 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.980484+0000", 2024-03-19T11:30:45.408 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019872499999999999 2024-03-19T11:30:45.408 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:45.408 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:45.408 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:45.408 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.408 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1420 4.18 4:1dabc0f1:::smithi03829856-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:45.408 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.968138+0000", 2024-03-19T11:30:45.408 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.374254614000002, 2024-03-19T11:30:45.408 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013109305999999999, 2024-03-19T11:30:45.409 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:45.409 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:45.409 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:45.409 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:45.409 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:45.409 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1420 2024-03-19T11:30:45.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.409 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:45.409 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.409 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:45.409 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968138+0000", 2024-03-19T11:30:45.409 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:45.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.409 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.410 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:45.410 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968138+0000", 2024-03-19T11:30:45.410 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:45.410 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.410 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:45.410 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968140+0000", 2024-03-19T11:30:45.410 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4139999999999999e-06 2024-03-19T11:30:45.410 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.410 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:45.410 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968147+0000", 2024-03-19T11:30:45.410 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.815e-06 2024-03-19T11:30:45.410 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.411 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:45.411 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968167+0000", 2024-03-19T11:30:45.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0018e-05 2024-03-19T11:30:45.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.411 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:45.411 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968182+0000", 2024-03-19T11:30:45.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5475e-05 2024-03-19T11:30:45.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.411 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:45.411 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.974950+0000", 2024-03-19T11:30:45.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.006767856 2024-03-19T11:30:45.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-03-19T11:30:45.412 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.975095+0000", 2024-03-19T11:30:45.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014501499999999999 2024-03-19T11:30:45.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:45.412 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.980614+0000", 2024-03-19T11:30:45.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0055184789999999997 2024-03-19T11:30:45.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:45.412 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.980912+0000", 2024-03-19T11:30:45.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029858199999999998 2024-03-19T11:30:45.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.413 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:45.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.981248+0000", 2024-03-19T11:30:45.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033565199999999998 2024-03-19T11:30:45.413 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:45.413 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:45.413 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:45.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.413 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1443 4.2 4:43b32242:::smithi03829856-22:head [read 0~2352947,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:45.413 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.979417+0000", 2024-03-19T11:30:45.413 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.362975987999999, 2024-03-19T11:30:45.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.96719988700000004, 2024-03-19T11:30:45.413 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:45.413 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:45.413 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:45.414 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:45.414 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:45.414 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1443 2024-03-19T11:30:45.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.414 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:45.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.414 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:45.414 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979417+0000", 2024-03-19T11:30:45.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:45.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.414 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:45.414 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979417+0000", 2024-03-19T11:30:45.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:45.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:45.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979420+0000", 2024-03-19T11:30:45.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.366e-06 2024-03-19T11:30:45.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:45.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979437+0000", 2024-03-19T11:30:45.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6192e-05 2024-03-19T11:30:45.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:45.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979474+0000", 2024-03-19T11:30:45.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7601000000000001e-05 2024-03-19T11:30:45.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.416 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:45.416 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979496+0000", 2024-03-19T11:30:45.416 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1506999999999999e-05 2024-03-19T11:30:45.416 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.416 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:45.416 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979712+0000", 2024-03-19T11:30:45.416 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021641299999999999 2024-03-19T11:30:45.416 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.416 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:30:45.416 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979746+0000", 2024-03-19T11:30:45.416 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3386000000000002e-05 2024-03-19T11:30:45.416 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.417 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:45.417 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.931552+0000", 2024-03-19T11:30:45.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.95180659899999998 2024-03-19T11:30:45.417 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.417 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-03-19T11:30:45.417 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.934681+0000", 2024-03-19T11:30:45.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0031292289999999999 2024-03-19T11:30:45.417 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.417 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:45.417 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.942701+0000", 2024-03-19T11:30:45.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0080198809999999995 2024-03-19T11:30:45.417 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:45.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.942839+0000", 2024-03-19T11:30:45.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013816999999999999 2024-03-19T11:30:45.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:45.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.946617+0000", 2024-03-19T11:30:45.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003777543 2024-03-19T11:30:45.418 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:45.418 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:45.418 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:45.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.419 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1444 4.2 4:43b32242:::smithi03829856-22:head [read 0~2352947,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:45.419 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.979553+0000", 2024-03-19T11:30:45.419 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.362840285000001, 2024-03-19T11:30:45.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.97015945400000003, 2024-03-19T11:30:45.419 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:45.419 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:45.419 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:45.419 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:45.419 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:45.419 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1444 2024-03-19T11:30:45.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.419 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:45.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:45.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979553+0000", 2024-03-19T11:30:45.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:45.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:45.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979553+0000", 2024-03-19T11:30:45.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:45.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:45.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979554+0000", 2024-03-19T11:30:45.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.403e-06 2024-03-19T11:30:45.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:45.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979564+0000", 2024-03-19T11:30:45.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0032e-05 2024-03-19T11:30:45.421 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.421 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.421 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:45.421 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979579+0000", 2024-03-19T11:30:45.421 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4875000000000001e-05 2024-03-19T11:30:45.421 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.421 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.421 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:45.421 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979595+0000", 2024-03-19T11:30:45.421 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6300999999999999e-05 2024-03-19T11:30:45.421 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.421 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.421 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:45.422 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979874+0000", 2024-03-19T11:30:45.422 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000279065 2024-03-19T11:30:45.422 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.422 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:30:45.422 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979904+0000", 2024-03-19T11:30:45.422 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.94e-05 2024-03-19T11:30:45.422 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.422 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:45.422 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.934781+0000", 2024-03-19T11:30:45.422 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.95487744900000004 2024-03-19T11:30:45.422 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-03-19T11:30:45.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.934946+0000", 2024-03-19T11:30:45.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016485999999999999 2024-03-19T11:30:45.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:45.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.946829+0000", 2024-03-19T11:30:45.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011882482999999999 2024-03-19T11:30:45.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:45.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.946989+0000", 2024-03-19T11:30:45.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000160421 2024-03-19T11:30:45.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:45.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.949712+0000", 2024-03-19T11:30:45.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002723165 2024-03-19T11:30:45.424 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:45.424 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:45.424 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:45.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.424 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1445 4.2 4:43b32242:::smithi03829856-22:head [read 0~2352947,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:45.424 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.979653+0000", 2024-03-19T11:30:45.424 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.362740154000001, 2024-03-19T11:30:45.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.97249507999999996, 2024-03-19T11:30:45.424 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:45.424 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:45.424 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:45.425 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:45.425 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:45.425 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1445 2024-03-19T11:30:45.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.425 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:45.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:45.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979653+0000", 2024-03-19T11:30:45.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:45.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:45.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979653+0000", 2024-03-19T11:30:45.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:45.426 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.426 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:45.426 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979654+0000", 2024-03-19T11:30:45.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.372e-06 2024-03-19T11:30:45.426 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.426 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:45.426 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979662+0000", 2024-03-19T11:30:45.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9959999999999993e-06 2024-03-19T11:30:45.426 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.426 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:45.427 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979698+0000", 2024-03-19T11:30:45.427 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5763000000000003e-05 2024-03-19T11:30:45.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.427 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:45.427 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979714+0000", 2024-03-19T11:30:45.427 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6283999999999999e-05 2024-03-19T11:30:45.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.427 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:45.427 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979990+0000", 2024-03-19T11:30:45.427 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000275829 2024-03-19T11:30:45.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.427 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:30:45.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.980018+0000", 2024-03-19T11:30:45.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8082999999999999e-05 2024-03-19T11:30:45.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.428 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:45.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.935620+0000", 2024-03-19T11:30:45.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.95560133800000002 2024-03-19T11:30:45.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.428 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-03-19T11:30:45.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.935795+0000", 2024-03-19T11:30:45.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000175656 2024-03-19T11:30:45.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.428 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:45.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.949835+0000", 2024-03-19T11:30:45.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014040083 2024-03-19T11:30:45.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:45.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.950002+0000", 2024-03-19T11:30:45.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016673 2024-03-19T11:30:45.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:45.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.952148+0000", 2024-03-19T11:30:45.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021459460000000001 2024-03-19T11:30:45.429 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:45.429 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:45.429 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:45.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.430 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.430 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1449 4.18 4:1d58db11:::smithi03829856-18:head [write 581542~688983,stat] snapc c=[c,b,a,6,5] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:45.430 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.989354+0000", 2024-03-19T11:30:45.430 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.353038700999999, 2024-03-19T11:30:45.430 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019237318, 2024-03-19T11:30:45.430 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:45.430 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:45.430 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:45.430 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:45.430 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:45.430 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1449 2024-03-19T11:30:45.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.430 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:45.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:45.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.989354+0000", 2024-03-19T11:30:45.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:45.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:45.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.989354+0000", 2024-03-19T11:30:45.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:45.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:45.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.989357+0000", 2024-03-19T11:30:45.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2740000000000002e-06 2024-03-19T11:30:45.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.432 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:45.432 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.990109+0000", 2024-03-19T11:30:45.432 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00075226500000000001 2024-03-19T11:30:45.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.432 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:45.432 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.990142+0000", 2024-03-19T11:30:45.432 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3328999999999999e-05 2024-03-19T11:30:45.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.432 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:45.432 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.990168+0000", 2024-03-19T11:30:45.432 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6016999999999998e-05 2024-03-19T11:30:45.433 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.433 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:45.433 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.990291+0000", 2024-03-19T11:30:45.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000122607 2024-03-19T11:30:45.433 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.433 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:45.433 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.990461+0000", 2024-03-19T11:30:45.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000169743 2024-03-19T11:30:45.433 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.433 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:45.433 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.990792+0000", 2024-03-19T11:30:45.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033135100000000002 2024-03-19T11:30:45.434 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.434 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.434 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:45.434 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.008238+0000", 2024-03-19T11:30:45.434 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017445591999999999 2024-03-19T11:30:45.434 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.434 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.434 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:45.434 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.008345+0000", 2024-03-19T11:30:45.434 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010762 2024-03-19T11:30:45.434 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.434 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.434 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:45.434 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.008436+0000", 2024-03-19T11:30:45.435 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1116e-05 2024-03-19T11:30:45.435 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.435 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:45.435 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.008592+0000", 2024-03-19T11:30:45.435 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015540399999999999 2024-03-19T11:30:45.435 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:45.435 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:45.435 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:45.435 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.435 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1450 4.18 4:1d58db11:::smithi03829856-18:head [write 1886503~412692] snapc c=[c,b,a,6,5] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:45.435 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.992973+0000", 2024-03-19T11:30:45.435 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.349419932, 2024-03-19T11:30:45.435 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016912580999999999, 2024-03-19T11:30:45.436 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:45.436 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:45.436 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:45.436 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:45.436 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:45.436 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1450 2024-03-19T11:30:45.436 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.436 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:45.436 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.436 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:45.436 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.992973+0000", 2024-03-19T11:30:45.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:45.436 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.436 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.436 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:45.437 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.992973+0000", 2024-03-19T11:30:45.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:45.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.437 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.437 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:45.437 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.992976+0000", 2024-03-19T11:30:45.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0060000000000001e-06 2024-03-19T11:30:45.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.437 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.437 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:45.437 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.993655+0000", 2024-03-19T11:30:45.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00067901099999999998 2024-03-19T11:30:45.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.437 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.437 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:45.438 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.993691+0000", 2024-03-19T11:30:45.438 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5546000000000002e-05 2024-03-19T11:30:45.438 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.438 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:45.438 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.993713+0000", 2024-03-19T11:30:45.438 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2277e-05 2024-03-19T11:30:45.438 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.438 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:45.438 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.997961+0000", 2024-03-19T11:30:45.438 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0042479659999999997 2024-03-19T11:30:45.438 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.439 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:45.439 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.998130+0000", 2024-03-19T11:30:45.439 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016908900000000001 2024-03-19T11:30:45.439 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.439 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.439 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:45.439 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.998397+0000", 2024-03-19T11:30:45.439 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026707599999999999 2024-03-19T11:30:45.439 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.439 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.439 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:45.439 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.008811+0000", 2024-03-19T11:30:45.439 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010413751000000001 2024-03-19T11:30:45.439 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.439 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.440 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:45.440 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.009539+0000", 2024-03-19T11:30:45.440 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00072811499999999999 2024-03-19T11:30:45.440 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.440 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.440 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:45.440 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.009711+0000", 2024-03-19T11:30:45.440 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017196900000000001 2024-03-19T11:30:45.440 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.440 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.440 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:45.440 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.009886+0000", 2024-03-19T11:30:45.440 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017477500000000001 2024-03-19T11:30:45.440 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:45.440 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:45.441 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:45.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.441 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.441 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1637 4.7 4:e41d47fb:::smithi03829856-14:head [write 484068~792524,stat] snapc 10=[10,f,e,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-03-19T11:30:45.441 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:35.672719+0000", 2024-03-19T11:30:45.441 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.6696737040000009, 2024-03-19T11:30:45.441 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012961324, 2024-03-19T11:30:45.441 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:45.441 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:45.441 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:45.441 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:45.441 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:45.441 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1637 2024-03-19T11:30:45.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.441 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:45.442 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.442 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:45.442 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.672719+0000", 2024-03-19T11:30:45.442 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:45.442 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.442 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.442 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:45.442 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.672719+0000", 2024-03-19T11:30:45.442 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:45.442 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.442 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.442 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:45.442 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.672722+0000", 2024-03-19T11:30:45.442 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0690000000000002e-06 2024-03-19T11:30:45.442 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.443 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.443 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:45.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.673743+0000", 2024-03-19T11:30:45.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010203899999999999 2024-03-19T11:30:45.443 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.443 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.443 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:45.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.673774+0000", 2024-03-19T11:30:45.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1049e-05 2024-03-19T11:30:45.443 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.443 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.443 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:45.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.673789+0000", 2024-03-19T11:30:45.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4729000000000001e-05 2024-03-19T11:30:45.443 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.444 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.444 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:45.444 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.673921+0000", 2024-03-19T11:30:45.444 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013277100000000001 2024-03-19T11:30:45.444 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.444 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.444 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:45.444 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.674053+0000", 2024-03-19T11:30:45.444 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000131975 2024-03-19T11:30:45.444 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.444 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.444 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:45.444 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.674430+0000", 2024-03-19T11:30:45.444 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000377031 2024-03-19T11:30:45.444 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:45.445 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.684295+0000", 2024-03-19T11:30:45.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0098643120000000001 2024-03-19T11:30:45.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:45.445 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.685359+0000", 2024-03-19T11:30:45.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001064111 2024-03-19T11:30:45.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:45.445 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.685532+0000", 2024-03-19T11:30:45.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017305700000000001 2024-03-19T11:30:45.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:45.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:45.446 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:45.446 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.685681+0000", 2024-03-19T11:30:45.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014883 2024-03-19T11:30:45.446 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:45.446 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:45.446 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:45.446 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:45.446 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:45.446 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:45.448 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-19T11:30:45.448 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:45.458+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:45.497 INFO:tasks.rados.rados.0.smithi038.stdout:1021: finishing write tid 1 to smithi03829856-1 2024-03-19T11:30:45.497 INFO:tasks.rados.rados.0.smithi038.stdout:1021: finishing write tid 2 to smithi03829856-1 2024-03-19T11:30:45.497 INFO:tasks.rados.rados.0.smithi038.stdout:1021: finishing write tid 3 to smithi03829856-1 2024-03-19T11:30:45.497 INFO:tasks.rados.rados.0.smithi038.stdout:1021: finishing write tid 4 to smithi03829856-1 2024-03-19T11:30:45.497 INFO:tasks.rados.rados.0.smithi038.stdout:1021: oid 1 updating version 0 to 100 2024-03-19T11:30:45.497 INFO:tasks.rados.rados.0.smithi038.stdout:1021: oid 1 version 100 is already newer than 99 2024-03-19T11:30:45.497 INFO:tasks.rados.rados.0.smithi038.stdout:1021: oid 1 version 100 is already newer than 98 2024-03-19T11:30:45.497 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 1 v 100 (ObjNum 156 snap 17 seq_num 156) dirty exists 2024-03-19T11:30:45.497 INFO:tasks.rados.rados.0.smithi038.stdout:1021: left oid 1 (ObjNum 156 snap 17 seq_num 156) 2024-03-19T11:30:45.498 INFO:tasks.rados.rados.0.smithi038.stdout:1020: done (9 left) 2024-03-19T11:30:45.498 INFO:tasks.rados.rados.0.smithi038.stdout:1021: done (8 left) 2024-03-19T11:30:45.498 INFO:tasks.rados.rados.0.smithi038.stdout:1022: done (7 left) 2024-03-19T11:30:45.498 INFO:tasks.rados.rados.0.smithi038.stdout:1029: done (6 left) 2024-03-19T11:30:45.498 INFO:tasks.rados.rados.0.smithi038.stdout:1030: write (excl) oid 23 current snap is 17 2024-03-19T11:30:45.498 INFO:tasks.rados.rados.0.smithi038.stdout:1023: finishing set_chunk to oid 10 2024-03-19T11:30:45.498 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 10 set_chunk returned error code -95 offset: 188928 length: 913920 2024-03-19T11:30:45.498 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 10 v 0 (ObjNum 144 snap 15 seq_num 144) dirty exists 2024-03-19T11:30:45.498 INFO:tasks.rados.rados.0.smithi038.stdout:1024: expect (ObjNum 108 snap 11 seq_num 108) 2024-03-19T11:30:45.549 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:45.556+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:45.618 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:45.619 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:45.619 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:30:45.619 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:45.632 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-19T11:30:45.670 INFO:tasks.rados.rados.0.smithi038.stdout:1025: finishing set_chunk to oid 7 2024-03-19T11:30:45.670 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 7 set_chunk returned error code -95 offset: 1510400 length: 1327616 2024-03-19T11:30:45.671 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 7 v 0 (ObjNum 67 snap 3 seq_num 67) dirty exists 2024-03-19T11:30:45.671 INFO:tasks.rados.rados.0.smithi038.stdout:1026: finishing write tid 1 to smithi03829856-31 2024-03-19T11:30:45.671 INFO:tasks.rados.rados.0.smithi038.stdout:1026: finishing write tid 2 to smithi03829856-31 2024-03-19T11:30:45.671 INFO:tasks.rados.rados.0.smithi038.stdout:1026: finishing write tid 3 to smithi03829856-31 2024-03-19T11:30:45.671 INFO:tasks.rados.rados.0.smithi038.stdout:1026: finishing write tid 4 to smithi03829856-31 2024-03-19T11:30:45.671 INFO:tasks.rados.rados.0.smithi038.stdout:1026: oid 31 updating version 0 to 105 2024-03-19T11:30:45.671 INFO:tasks.rados.rados.0.smithi038.stdout:1026: oid 31 updating version 105 to 107 2024-03-19T11:30:45.671 INFO:tasks.rados.rados.0.smithi038.stdout:1026: oid 31 version 107 is already newer than 106 2024-03-19T11:30:45.671 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 31 v 107 (ObjNum 157 snap 17 seq_num 157) dirty exists 2024-03-19T11:30:45.671 INFO:tasks.rados.rados.0.smithi038.stdout:1026: left oid 31 (ObjNum 157 snap 17 seq_num 157) 2024-03-19T11:30:45.671 INFO:tasks.rados.rados.0.smithi038.stdout:1027: got (0) Success 2024-03-19T11:30:45.671 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 47 v 15 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-03-19T11:30:45.671 INFO:tasks.rados.rados.0.smithi038.stdout:1028: finishing write tid 1 to smithi03829856-35 2024-03-19T11:30:45.672 INFO:tasks.rados.rados.0.smithi038.stdout:1028: finishing write tid 2 to smithi03829856-35 2024-03-19T11:30:45.672 INFO:tasks.rados.rados.0.smithi038.stdout:1028: finishing write tid 3 to smithi03829856-35 2024-03-19T11:30:45.672 INFO:tasks.rados.rados.0.smithi038.stdout:1028: finishing write tid 4 to smithi03829856-35 2024-03-19T11:30:45.672 INFO:tasks.rados.rados.0.smithi038.stdout:1028: oid 35 updating version 0 to 107 2024-03-19T11:30:45.672 INFO:tasks.rados.rados.0.smithi038.stdout:1028: oid 35 version 107 is already newer than 106 2024-03-19T11:30:45.672 INFO:tasks.rados.rados.0.smithi038.stdout:1028: oid 35 updating version 107 to 108 2024-03-19T11:30:45.672 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 35 v 108 (ObjNum 158 snap 17 seq_num 158) dirty exists 2024-03-19T11:30:45.672 INFO:tasks.rados.rados.0.smithi038.stdout:1028: left oid 35 (ObjNum 158 snap 17 seq_num 158) 2024-03-19T11:30:45.672 INFO:tasks.rados.rados.0.smithi038.stdout:1030: seq_num 160 ranges {689120=483253,1946413=679049} 2024-03-19T11:30:45.679 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:45.688+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:45.684 INFO:tasks.rados.rados.0.smithi038.stdout:1030: writing smithi03829856-23 from 689120 to 1172373 tid 1 2024-03-19T11:30:45.703 INFO:tasks.rados.rados.0.smithi038.stdout:1030: writing smithi03829856-23 from 1946413 to 2625462 tid 2 2024-03-19T11:30:45.703 INFO:tasks.rados.rados.0.smithi038.stdout:1023: done (6 left) 2024-03-19T11:30:45.703 INFO:tasks.rados.rados.0.smithi038.stdout:1024: done (5 left) 2024-03-19T11:30:45.703 INFO:tasks.rados.rados.0.smithi038.stdout:1025: done (4 left) 2024-03-19T11:30:45.703 INFO:tasks.rados.rados.0.smithi038.stdout:1026: done (3 left) 2024-03-19T11:30:45.703 INFO:tasks.rados.rados.0.smithi038.stdout:1027: done (2 left) 2024-03-19T11:30:45.703 INFO:tasks.rados.rados.0.smithi038.stdout:1028: done (1 left) 2024-03-19T11:30:45.703 INFO:tasks.rados.rados.0.smithi038.stdout:1031: write (excl) oid 12 current snap is 17 2024-03-19T11:30:45.703 INFO:tasks.rados.rados.0.smithi038.stdout:1031: seq_num 161 ranges {705927=558502,2004895=422707} 2024-03-19T11:30:45.718 INFO:tasks.rados.rados.0.smithi038.stdout:1031: writing smithi03829856-12 from 705927 to 1264429 tid 1 2024-03-19T11:30:45.732 INFO:tasks.rados.rados.0.smithi038.stdout:1031: writing smithi03829856-12 from 2004895 to 2427602 tid 2 2024-03-19T11:30:45.732 INFO:tasks.rados.rados.0.smithi038.stdout:1032: read oid 50 snap -1 2024-03-19T11:30:45.732 INFO:tasks.rados.rados.0.smithi038.stdout:1032: expect (ObjNum 114 snap 11 seq_num 114) 2024-03-19T11:30:45.732 INFO:tasks.rados.rados.0.smithi038.stdout:1033: write oid 13 current snap is 17 2024-03-19T11:30:45.732 INFO:tasks.rados.rados.0.smithi038.stdout:1030: finishing write tid 1 to smithi03829856-23 2024-03-19T11:30:45.733 INFO:tasks.rados.rados.0.smithi038.stdout:1030: finishing write tid 2 to smithi03829856-23 2024-03-19T11:30:45.733 INFO:tasks.rados.rados.0.smithi038.stdout:1030: finishing write tid 3 to smithi03829856-23 2024-03-19T11:30:45.733 INFO:tasks.rados.rados.0.smithi038.stdout:1033: seq_num 162 ranges {722734=633751,2063377=659357} 2024-03-19T11:30:45.745 INFO:tasks.rados.rados.0.smithi038.stdout:1033: writing smithi03829856-13 from 722734 to 1356485 tid 1 2024-03-19T11:30:45.750 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:45.756+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:45.758 INFO:tasks.rados.rados.0.smithi038.stdout:1033: writing smithi03829856-13 from 2063377 to 2722734 tid 2 2024-03-19T11:30:45.758 INFO:tasks.rados.rados.0.smithi038.stdout:1034: set_chunk oid 9 target oid 2024-03-19T11:30:45.763 INFO:tasks.rados.rados.0.smithi038.stdout:1034: set_chunk oid 9 offset: 308224 length: 46592 target oid 862afd8e1d7d8c855520d0e3e02b21bbe2e614062e3dc3ed4e7b0f93803b8c85 offset: 0 snap -1 2024-03-19T11:30:45.763 INFO:tasks.rados.rados.0.smithi038.stdout:1035: read oid 14 snap -1 2024-03-19T11:30:45.764 INFO:tasks.rados.rados.0.smithi038.stdout:1035: expect (ObjNum 124 snap 15 seq_num 124) 2024-03-19T11:30:45.764 INFO:tasks.rados.rados.0.smithi038.stdout:1036: write (excl) oid 39 current snap is 17 2024-03-19T11:30:45.764 INFO:tasks.rados.rados.0.smithi038.stdout:1030: finishing write tid 4 to smithi03829856-23 2024-03-19T11:30:45.764 INFO:tasks.rados.rados.0.smithi038.stdout:1030: oid 23 updating version 0 to 75 2024-03-19T11:30:45.764 INFO:tasks.rados.rados.0.smithi038.stdout:1030: oid 23 version 75 is already newer than 74 2024-03-19T11:30:45.764 INFO:tasks.rados.rados.0.smithi038.stdout:1030: oid 23 version 75 is already newer than 73 2024-03-19T11:30:45.764 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 23 v 75 (ObjNum 160 snap 17 seq_num 160) dirty exists 2024-03-19T11:30:45.764 INFO:tasks.rados.rados.0.smithi038.stdout:1030: left oid 23 (ObjNum 160 snap 17 seq_num 160) 2024-03-19T11:30:45.764 INFO:tasks.rados.rados.0.smithi038.stdout:1031: finishing write tid 1 to smithi03829856-12 2024-03-19T11:30:45.764 INFO:tasks.rados.rados.0.smithi038.stdout:1031: finishing write tid 2 to smithi03829856-12 2024-03-19T11:30:45.764 INFO:tasks.rados.rados.0.smithi038.stdout:1031: finishing write tid 3 to smithi03829856-12 2024-03-19T11:30:45.764 INFO:tasks.rados.rados.0.smithi038.stdout:1036: seq_num 163 ranges {739541=709000,2121859=426376} 2024-03-19T11:30:45.779 INFO:tasks.rados.rados.0.smithi038.stdout:1036: writing smithi03829856-39 from 739541 to 1448541 tid 1 2024-03-19T11:30:45.791 INFO:tasks.rados.rados.0.smithi038.stdout:1036: writing smithi03829856-39 from 2121859 to 2548235 tid 2 2024-03-19T11:30:45.791 INFO:tasks.rados.rados.0.smithi038.stdout:1030: done (6 left) 2024-03-19T11:30:45.791 INFO:tasks.rados.rados.0.smithi038.stdout:1037: tier_promote oid 10 2024-03-19T11:30:45.791 INFO:tasks.rados.rados.0.smithi038.stdout:1038: read oid 47 snap -1 2024-03-19T11:30:45.791 INFO:tasks.rados.rados.0.smithi038.stdout:1038: expect (ObjNum 46 snap 0 seq_num 46) 2024-03-19T11:30:45.792 INFO:tasks.rados.rados.0.smithi038.stdout:1039: rollback oid 6 current snap is 17 2024-03-19T11:30:45.792 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 6 to 4 2024-03-19T11:30:45.792 INFO:tasks.rados.rados.0.smithi038.stdout:1031: finishing write tid 4 to smithi03829856-12 2024-03-19T11:30:45.792 INFO:tasks.rados.rados.0.smithi038.stdout:1031: oid 12 updating version 0 to 81 2024-03-19T11:30:45.792 INFO:tasks.rados.rados.0.smithi038.stdout:1031: oid 12 updating version 81 to 82 2024-03-19T11:30:45.792 INFO:tasks.rados.rados.0.smithi038.stdout:1031: oid 12 updating version 82 to 83 2024-03-19T11:30:45.792 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 12 v 83 (ObjNum 161 snap 17 seq_num 161) dirty exists 2024-03-19T11:30:45.792 INFO:tasks.rados.rados.0.smithi038.stdout:1031: left oid 12 (ObjNum 161 snap 17 seq_num 161) 2024-03-19T11:30:45.792 INFO:tasks.rados.rados.0.smithi038.stdout:1032: expect (ObjNum 114 snap 11 seq_num 114) 2024-03-19T11:30:45.816 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:45.816 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:45.816 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:30:45.816 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:30:45.816 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:45.830 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-19T11:30:45.851 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:45.858+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:45.952 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:45.960+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:46.005 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:46.005 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:30:46.005 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:30:46.005 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:30:46.005 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.005 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:29 4.5 4:ac9cd3a3:::smithi03829856-8:head [write 517649~526743,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:30:46.005 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:04.772849+0000", 2024-03-19T11:30:46.005 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 41.240538880999999, 2024-03-19T11:30:46.005 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014076821, 2024-03-19T11:30:46.005 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:46.006 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:46.006 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:46.006 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:46.006 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:46.006 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 29 2024-03-19T11:30:46.006 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.006 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:46.006 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.006 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:46.006 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.772849+0000", 2024-03-19T11:30:46.006 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:46.006 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.006 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.006 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:46.007 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.772849+0000", 2024-03-19T11:30:46.007 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:46.007 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.007 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.007 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:46.007 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.772852+0000", 2024-03-19T11:30:46.007 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.79e-06 2024-03-19T11:30:46.007 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.007 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.007 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:46.007 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.773783+0000", 2024-03-19T11:30:46.007 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00093104199999999998 2024-03-19T11:30:46.007 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.007 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.008 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:46.008 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.773810+0000", 2024-03-19T11:30:46.008 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.669e-05 2024-03-19T11:30:46.008 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.008 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.008 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:46.008 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.773826+0000", 2024-03-19T11:30:46.008 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5739999999999998e-05 2024-03-19T11:30:46.008 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.008 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.008 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:46.008 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.773954+0000", 2024-03-19T11:30:46.008 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012796300000000001 2024-03-19T11:30:46.008 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.008 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.009 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op must wait for map", 2024-03-19T11:30:46.009 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.773965+0000", 2024-03-19T11:30:46.009 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1453e-05 2024-03-19T11:30:46.009 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.009 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.009 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:46.009 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.777723+0000", 2024-03-19T11:30:46.009 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003757944 2024-03-19T11:30:46.009 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.009 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.009 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:46.009 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778138+0000", 2024-03-19T11:30:46.009 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00041454699999999997 2024-03-19T11:30:46.009 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.009 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.010 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:30:46.010 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778409+0000", 2024-03-19T11:30:46.010 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027111700000000003 2024-03-19T11:30:46.010 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.010 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.010 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:46.010 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.786631+0000", 2024-03-19T11:30:46.010 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0082221470000000008 2024-03-19T11:30:46.010 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.010 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.010 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:46.010 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.786668+0000", 2024-03-19T11:30:46.010 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.718e-05 2024-03-19T11:30:46.010 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.010 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.011 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:46.011 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.786845+0000", 2024-03-19T11:30:46.011 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000177219 2024-03-19T11:30:46.011 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.011 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.011 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:46.011 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.786926+0000", 2024-03-19T11:30:46.011 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0989000000000007e-05 2024-03-19T11:30:46.011 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:46.011 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:46.011 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:46.011 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.011 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.011 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:30 4.5 4:ac9cd3a3:::smithi03829856-8:head [write 1776668~340981] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:30:46.011 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:04.777922+0000", 2024-03-19T11:30:46.012 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 41.235466250999998, 2024-03-19T11:30:46.012 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015469894, 2024-03-19T11:30:46.012 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:46.012 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:46.012 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:46.012 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:46.012 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:46.012 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 30 2024-03-19T11:30:46.012 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.012 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:46.012 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.012 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:46.012 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.777922+0000", 2024-03-19T11:30:46.012 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:46.012 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.013 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.013 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:46.013 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.777922+0000", 2024-03-19T11:30:46.013 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:46.013 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.013 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.013 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:46.013 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.777924+0000", 2024-03-19T11:30:46.013 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3999999999999999e-06 2024-03-19T11:30:46.013 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.013 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.013 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:46.013 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778461+0000", 2024-03-19T11:30:46.013 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00053698299999999999 2024-03-19T11:30:46.013 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.013 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.014 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:46.014 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778487+0000", 2024-03-19T11:30:46.014 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5947000000000001e-05 2024-03-19T11:30:46.014 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.014 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.014 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:46.014 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778502+0000", 2024-03-19T11:30:46.014 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4256e-05 2024-03-19T11:30:46.014 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.014 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.014 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:46.014 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.781978+0000", 2024-03-19T11:30:46.014 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00347693 2024-03-19T11:30:46.014 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.015 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.015 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:46.015 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.782196+0000", 2024-03-19T11:30:46.015 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021726499999999999 2024-03-19T11:30:46.015 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.015 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.015 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:30:46.015 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.782572+0000", 2024-03-19T11:30:46.015 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00037635000000000001 2024-03-19T11:30:46.015 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.015 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.015 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:46.015 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.786960+0000", 2024-03-19T11:30:46.015 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043882130000000002 2024-03-19T11:30:46.015 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.016 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.016 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:46.016 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793113+0000", 2024-03-19T11:30:46.016 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061530760000000004 2024-03-19T11:30:46.016 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.016 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.016 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:46.016 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793276+0000", 2024-03-19T11:30:46.016 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000162652 2024-03-19T11:30:46.016 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.016 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.016 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:46.016 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793392+0000", 2024-03-19T11:30:46.016 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115822 2024-03-19T11:30:46.016 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:46.016 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:46.017 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:46.017 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.017 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.017 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:31 4.5 4:ac9cd3a3:::smithi03829856-8:head [setxattr _header (55) in=7b,truncate 2117649] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:30:46.017 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:04.778533+0000", 2024-03-19T11:30:46.017 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 41.234855115999999, 2024-03-19T11:30:46.017 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015182533, 2024-03-19T11:30:46.017 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:46.017 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:46.017 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:46.017 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:46.017 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:46.017 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 31 2024-03-19T11:30:46.017 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.017 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:46.018 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.018 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:46.018 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778533+0000", 2024-03-19T11:30:46.018 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:46.018 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.018 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.018 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:46.018 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778533+0000", 2024-03-19T11:30:46.018 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:46.018 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.018 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.018 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:46.018 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778534+0000", 2024-03-19T11:30:46.018 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1069999999999999e-06 2024-03-19T11:30:46.019 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.019 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.019 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:46.019 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778541+0000", 2024-03-19T11:30:46.019 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2529999999999998e-06 2024-03-19T11:30:46.019 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.019 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.019 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:46.019 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778558+0000", 2024-03-19T11:30:46.019 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6290000000000002e-05 2024-03-19T11:30:46.019 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.019 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.019 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:46.019 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778568+0000", 2024-03-19T11:30:46.019 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0360000000000001e-05 2024-03-19T11:30:46.020 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.020 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.020 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:46.020 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.785511+0000", 2024-03-19T11:30:46.020 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00694306 2024-03-19T11:30:46.020 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.020 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.020 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:46.020 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.785710+0000", 2024-03-19T11:30:46.020 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019837199999999999 2024-03-19T11:30:46.020 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.020 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.020 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:30:46.020 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.785929+0000", 2024-03-19T11:30:46.020 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021960799999999999 2024-03-19T11:30:46.021 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.021 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.021 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:46.021 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.787281+0000", 2024-03-19T11:30:46.021 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001352313 2024-03-19T11:30:46.021 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.021 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.021 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:46.021 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793492+0000", 2024-03-19T11:30:46.021 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062107760000000003 2024-03-19T11:30:46.021 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.021 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.021 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:46.021 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793578+0000", 2024-03-19T11:30:46.021 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5711999999999995e-05 2024-03-19T11:30:46.022 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.022 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.022 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:46.022 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793716+0000", 2024-03-19T11:30:46.022 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000137682 2024-03-19T11:30:46.022 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:46.022 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:46.022 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:46.022 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.022 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.022 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:32 4.5 4:ac9cd3a3:::smithi03829856-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:30:46.022 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:04.778582+0000", 2024-03-19T11:30:46.022 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 41.234806601999999, 2024-03-19T11:30:46.022 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015761827999999999, 2024-03-19T11:30:46.022 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:46.023 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:46.023 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:46.023 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:46.023 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:46.023 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 32 2024-03-19T11:30:46.023 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.023 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:46.023 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.023 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:46.023 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778582+0000", 2024-03-19T11:30:46.023 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:46.023 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.023 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.023 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:46.023 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778582+0000", 2024-03-19T11:30:46.024 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:46.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.024 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:46.024 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778583+0000", 2024-03-19T11:30:46.024 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.026e-06 2024-03-19T11:30:46.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.024 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:46.024 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778587+0000", 2024-03-19T11:30:46.024 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7870000000000001e-06 2024-03-19T11:30:46.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.024 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:46.024 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778599+0000", 2024-03-19T11:30:46.025 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1623e-05 2024-03-19T11:30:46.025 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.025 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.025 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:46.025 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778608+0000", 2024-03-19T11:30:46.025 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7399999999999993e-06 2024-03-19T11:30:46.025 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.025 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.025 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:46.025 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.787009+0000", 2024-03-19T11:30:46.025 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0084010490000000007 2024-03-19T11:30:46.025 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.025 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.025 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-03-19T11:30:46.025 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.787202+0000", 2024-03-19T11:30:46.025 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019309500000000001 2024-03-19T11:30:46.026 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.026 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.026 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:46.026 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793832+0000", 2024-03-19T11:30:46.026 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.006630052 2024-03-19T11:30:46.026 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.026 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.026 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:46.026 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.794048+0000", 2024-03-19T11:30:46.026 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000215505 2024-03-19T11:30:46.026 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.026 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.026 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:46.026 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.794343+0000", 2024-03-19T11:30:46.027 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029595100000000003 2024-03-19T11:30:46.027 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:46.027 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:46.027 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:46.027 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.027 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.027 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:856 4.5 4:ac9cd3a3:::smithi03829856-8:head [write 608420~529411] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:46.027 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.781425+0000", 2024-03-19T11:30:46.027 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 37.231963415999999, 2024-03-19T11:30:46.027 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014634677, 2024-03-19T11:30:46.027 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:46.027 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:46.027 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:46.027 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:46.027 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:46.028 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 856 2024-03-19T11:30:46.028 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.028 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:46.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.028 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:46.028 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.781425+0000", 2024-03-19T11:30:46.028 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:46.028 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.028 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:46.028 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.781425+0000", 2024-03-19T11:30:46.028 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:46.028 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.028 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:46.028 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.781428+0000", 2024-03-19T11:30:46.029 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.534e-06 2024-03-19T11:30:46.029 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.029 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.029 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:46.029 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.783096+0000", 2024-03-19T11:30:46.029 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001667897 2024-03-19T11:30:46.029 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.029 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.029 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:46.029 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.783138+0000", 2024-03-19T11:30:46.029 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1946000000000001e-05 2024-03-19T11:30:46.029 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.029 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.029 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:46.029 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.783163+0000", 2024-03-19T11:30:46.030 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4443e-05 2024-03-19T11:30:46.030 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.030 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.030 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:46.030 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.783333+0000", 2024-03-19T11:30:46.030 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017027899999999999 2024-03-19T11:30:46.030 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.030 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.030 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:46.030 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.783503+0000", 2024-03-19T11:30:46.030 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000170242 2024-03-19T11:30:46.030 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.030 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.030 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:30:46.030 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.783917+0000", 2024-03-19T11:30:46.031 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00041371000000000002 2024-03-19T11:30:46.031 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.031 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.031 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:46.031 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.795709+0000", 2024-03-19T11:30:46.031 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011791693000000001 2024-03-19T11:30:46.031 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.031 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.031 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:46.031 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.795850+0000", 2024-03-19T11:30:46.031 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000141695 2024-03-19T11:30:46.031 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.031 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.031 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:46.032 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.795940+0000", 2024-03-19T11:30:46.032 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9705999999999998e-05 2024-03-19T11:30:46.032 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.032 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.032 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:46.032 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.796059+0000", 2024-03-19T11:30:46.032 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011953199999999999 2024-03-19T11:30:46.032 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:46.032 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:46.032 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:46.032 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.032 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.032 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:891 4.5 4:af9d0d56:::smithi03829856-20:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:46.032 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:09.286410+0000", 2024-03-19T11:30:46.032 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 36.726978270000004, 2024-03-19T11:30:46.033 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035191405000000002, 2024-03-19T11:30:46.033 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:46.033 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:46.033 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:46.033 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:46.033 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:46.033 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 891 2024-03-19T11:30:46.033 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.033 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:46.033 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.033 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:46.033 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286410+0000", 2024-03-19T11:30:46.033 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:46.033 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.033 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.034 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:46.034 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286410+0000", 2024-03-19T11:30:46.034 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:46.034 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.034 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.034 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:46.034 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286413+0000", 2024-03-19T11:30:46.034 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2660000000000002e-06 2024-03-19T11:30:46.034 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.034 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.034 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:46.034 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286426+0000", 2024-03-19T11:30:46.034 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.258e-05 2024-03-19T11:30:46.034 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.034 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.035 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:46.035 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286460+0000", 2024-03-19T11:30:46.035 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3943999999999999e-05 2024-03-19T11:30:46.035 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.035 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.035 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:46.035 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286477+0000", 2024-03-19T11:30:46.035 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6799999999999998e-05 2024-03-19T11:30:46.035 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.035 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.035 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:46.035 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286605+0000", 2024-03-19T11:30:46.035 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000128892 2024-03-19T11:30:46.035 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.035 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.036 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:46.036 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286769+0000", 2024-03-19T11:30:46.036 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016359799999999999 2024-03-19T11:30:46.036 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.036 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:46.036 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.321601+0000", 2024-03-19T11:30:46.036 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.034832324999999997 2024-03-19T11:30:46.036 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:46.036 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:46.036 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:46.036 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.036 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:993 4.e e34/21 4:72163e4c:::smithi03829856-39:head v 34'17, mlcod=28'14)", 2024-03-19T11:30:46.036 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:12.367897+0000", 2024-03-19T11:30:46.037 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 33.645491724000003, 2024-03-19T11:30:46.037 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012765165, 2024-03-19T11:30:46.037 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:46.037 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:46.037 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:46.037 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.037 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:46.037 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.367897+0000", 2024-03-19T11:30:46.037 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:46.037 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.037 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.037 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:46.037 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.367897+0000", 2024-03-19T11:30:46.037 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:46.037 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.038 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.038 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:46.038 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.367897+0000", 2024-03-19T11:30:46.038 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7000000000000003e-07 2024-03-19T11:30:46.038 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.038 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.038 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:46.038 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.368454+0000", 2024-03-19T11:30:46.038 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00055647599999999998 2024-03-19T11:30:46.038 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.038 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.038 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:46.038 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.368532+0000", 2024-03-19T11:30:46.038 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8479000000000005e-05 2024-03-19T11:30:46.038 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.039 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:46.039 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.368554+0000", 2024-03-19T11:30:46.039 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.156e-05 2024-03-19T11:30:46.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.039 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:46.039 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.368742+0000", 2024-03-19T11:30:46.039 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000187846 2024-03-19T11:30:46.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.039 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:46.039 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.368782+0000", 2024-03-19T11:30:46.039 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9836000000000003e-05 2024-03-19T11:30:46.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.040 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.040 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:46.040 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.380340+0000", 2024-03-19T11:30:46.040 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011558847000000001 2024-03-19T11:30:46.040 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.040 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.040 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:46.040 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.380662+0000", 2024-03-19T11:30:46.040 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00032125099999999999 2024-03-19T11:30:46.040 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:46.040 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:46.040 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:46.040 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.040 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.040 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1015 4.5 4:af9d0d56:::smithi03829856-20:head [tier-promote] snapc 6=[6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:46.040 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:12.761855+0000", 2024-03-19T11:30:46.041 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 33.251532744000002, 2024-03-19T11:30:46.041 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026760430000000002, 2024-03-19T11:30:46.041 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:46.041 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:46.041 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:46.041 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:46.041 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:46.041 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1015 2024-03-19T11:30:46.041 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.041 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:46.041 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.041 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:46.041 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761855+0000", 2024-03-19T11:30:46.041 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:46.041 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.042 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.042 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:46.042 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761855+0000", 2024-03-19T11:30:46.042 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:46.042 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.042 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.042 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:46.042 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761858+0000", 2024-03-19T11:30:46.042 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8140000000000001e-06 2024-03-19T11:30:46.042 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.042 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.042 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:46.042 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761872+0000", 2024-03-19T11:30:46.042 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3813e-05 2024-03-19T11:30:46.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.043 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:46.043 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761905+0000", 2024-03-19T11:30:46.043 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2642999999999999e-05 2024-03-19T11:30:46.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.043 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:46.043 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761925+0000", 2024-03-19T11:30:46.043 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0489e-05 2024-03-19T11:30:46.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.043 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:46.043 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.762070+0000", 2024-03-19T11:30:46.043 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000145153 2024-03-19T11:30:46.044 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.044 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.044 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:46.044 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.762365+0000", 2024-03-19T11:30:46.044 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029482100000000002 2024-03-19T11:30:46.044 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.044 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.044 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:46.044 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.788616+0000", 2024-03-19T11:30:46.044 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026250697 2024-03-19T11:30:46.044 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:46.044 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:46.044 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:46.044 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.044 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.045 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1121 4.5 4:aa3303a5:::smithi03829856-17:head [tier-promote] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:46.045 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:14.164281+0000", 2024-03-19T11:30:46.045 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 31.849107498999999, 2024-03-19T11:30:46.045 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023896435000000001, 2024-03-19T11:30:46.045 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:46.045 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:46.045 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:46.045 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:46.045 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:46.045 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1121 2024-03-19T11:30:46.045 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.045 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:46.045 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.045 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:46.045 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164281+0000", 2024-03-19T11:30:46.046 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:46.046 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.046 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.046 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:46.046 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164281+0000", 2024-03-19T11:30:46.046 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:46.046 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.046 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.046 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:46.046 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164283+0000", 2024-03-19T11:30:46.046 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8500000000000001e-06 2024-03-19T11:30:46.046 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.046 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.046 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:46.046 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164291+0000", 2024-03-19T11:30:46.047 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.14e-06 2024-03-19T11:30:46.047 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.047 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.047 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:46.047 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164313+0000", 2024-03-19T11:30:46.047 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.207e-05 2024-03-19T11:30:46.047 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.047 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.047 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:46.047 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164328+0000", 2024-03-19T11:30:46.047 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5403000000000001e-05 2024-03-19T11:30:46.047 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.047 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.047 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:46.047 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.167981+0000", 2024-03-19T11:30:46.048 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036529879999999998 2024-03-19T11:30:46.048 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.048 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.048 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:46.048 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.168274+0000", 2024-03-19T11:30:46.048 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029313100000000002 2024-03-19T11:30:46.048 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.048 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.048 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:46.048 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.188177+0000", 2024-03-19T11:30:46.048 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019902853000000002 2024-03-19T11:30:46.048 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:46.048 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:46.048 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:46.048 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.049 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.049 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1154 4.5 4:a0a50d04:::smithi03829856-13:8 [sparse-read 0~1151947] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-03-19T11:30:46.049 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.827529+0000", 2024-03-19T11:30:46.049 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.185859081, 2024-03-19T11:30:46.049 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013620382, 2024-03-19T11:30:46.049 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:46.049 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:46.049 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:46.049 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:46.049 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:46.049 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1154 2024-03-19T11:30:46.049 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.049 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:46.049 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.049 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:46.050 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827529+0000", 2024-03-19T11:30:46.050 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:46.050 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.050 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.050 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:46.050 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827529+0000", 2024-03-19T11:30:46.050 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:46.050 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.050 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.050 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:46.050 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827530+0000", 2024-03-19T11:30:46.050 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7700000000000004e-07 2024-03-19T11:30:46.050 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.050 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.050 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:46.051 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827533+0000", 2024-03-19T11:30:46.051 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4520000000000002e-06 2024-03-19T11:30:46.051 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.051 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.051 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:46.051 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827543+0000", 2024-03-19T11:30:46.051 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0035e-05 2024-03-19T11:30:46.051 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.051 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.051 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:46.051 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827551+0000", 2024-03-19T11:30:46.051 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7049999999999999e-06 2024-03-19T11:30:46.051 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.051 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.051 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:46.051 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.839931+0000", 2024-03-19T11:30:46.052 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012379602999999999 2024-03-19T11:30:46.052 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.052 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.052 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:46.052 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.840290+0000", 2024-03-19T11:30:46.052 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000359031 2024-03-19T11:30:46.052 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.052 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.052 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:46.052 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841150+0000", 2024-03-19T11:30:46.052 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00085977899999999997 2024-03-19T11:30:46.052 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:46.052 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:46.053 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:46.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.053 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.053 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1302 4.15 4:aa3303a5:::smithi03829856-17:head [write 413472~620140,stat] snapc b=[b,a,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-03-19T11:30:46.053 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:22.966426+0000", 2024-03-19T11:30:46.053 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.046961765999999, 2024-03-19T11:30:46.053 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018043513000000001, 2024-03-19T11:30:46.053 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:46.053 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:46.053 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:46.053 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:46.053 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:46.053 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1302 2024-03-19T11:30:46.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.053 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:46.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.054 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:46.054 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.966426+0000", 2024-03-19T11:30:46.054 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:46.054 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.054 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:46.054 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.966426+0000", 2024-03-19T11:30:46.054 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:46.054 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.054 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:46.054 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.966429+0000", 2024-03-19T11:30:46.054 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3829999999999999e-06 2024-03-19T11:30:46.054 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.055 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.055 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:46.055 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972117+0000", 2024-03-19T11:30:46.055 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0056885750000000004 2024-03-19T11:30:46.055 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.055 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.055 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:46.055 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972156+0000", 2024-03-19T11:30:46.055 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8878000000000003e-05 2024-03-19T11:30:46.055 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.055 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.055 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:46.055 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972184+0000", 2024-03-19T11:30:46.055 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7509999999999999e-05 2024-03-19T11:30:46.055 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.055 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.056 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:46.056 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972309+0000", 2024-03-19T11:30:46.056 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012525899999999999 2024-03-19T11:30:46.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.056 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:46.056 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972566+0000", 2024-03-19T11:30:46.056 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025740799999999999 2024-03-19T11:30:46.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.056 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:30:46.056 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972899+0000", 2024-03-19T11:30:46.056 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033238799999999998 2024-03-19T11:30:46.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.057 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:46.057 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984088+0000", 2024-03-19T11:30:46.057 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011189466 2024-03-19T11:30:46.057 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.057 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:46.057 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984138+0000", 2024-03-19T11:30:46.057 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9666999999999997e-05 2024-03-19T11:30:46.057 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.057 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:46.057 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984329+0000", 2024-03-19T11:30:46.057 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019086799999999999 2024-03-19T11:30:46.057 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.058 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:46.058 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984470+0000", 2024-03-19T11:30:46.058 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000141111 2024-03-19T11:30:46.058 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:46.058 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:46.058 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:46.058 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.058 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.058 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1303 4.15 4:aa3303a5:::smithi03829856-17:head [write 1818036~510700] snapc b=[b,a,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-03-19T11:30:46.058 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:22.976856+0000", 2024-03-19T11:30:46.058 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.036532655999999, 2024-03-19T11:30:46.058 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015045703000000001, 2024-03-19T11:30:46.058 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:46.058 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:46.059 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:46.059 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:46.059 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:46.059 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1303 2024-03-19T11:30:46.059 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.059 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:46.059 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.059 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:46.059 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.976856+0000", 2024-03-19T11:30:46.059 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:46.059 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.059 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.059 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:46.059 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.976856+0000", 2024-03-19T11:30:46.059 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:46.060 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.060 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.060 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:46.060 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.976859+0000", 2024-03-19T11:30:46.060 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3629999999999998e-06 2024-03-19T11:30:46.060 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.060 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.060 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:46.060 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.977533+0000", 2024-03-19T11:30:46.060 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00067380800000000002 2024-03-19T11:30:46.060 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.060 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.060 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:46.060 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.977573+0000", 2024-03-19T11:30:46.060 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9758000000000001e-05 2024-03-19T11:30:46.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.061 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:46.061 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.977602+0000", 2024-03-19T11:30:46.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9856e-05 2024-03-19T11:30:46.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.061 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:46.061 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.979943+0000", 2024-03-19T11:30:46.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0023406400000000002 2024-03-19T11:30:46.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.061 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:46.061 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.980238+0000", 2024-03-19T11:30:46.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029488000000000002 2024-03-19T11:30:46.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:30:46.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.980734+0000", 2024-03-19T11:30:46.062 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00049647500000000002 2024-03-19T11:30:46.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:46.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.991761+0000", 2024-03-19T11:30:46.062 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011026515000000001 2024-03-19T11:30:46.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:46.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.991777+0000", 2024-03-19T11:30:46.063 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5727999999999999e-05 2024-03-19T11:30:46.063 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.065 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.066 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:46.066 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.991840+0000", 2024-03-19T11:30:46.066 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3091999999999998e-05 2024-03-19T11:30:46.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.066 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.066 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:46.066 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.991901+0000", 2024-03-19T11:30:46.066 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1587999999999994e-05 2024-03-19T11:30:46.066 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:46.066 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:46.066 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:46.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.066 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.066 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1467 4.a 4:567251f3:::smithi03829856-44:head [write 615156~555834] snapc c=[c,b,a,6,5] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-03-19T11:30:46.067 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.079231+0000", 2024-03-19T11:30:46.067 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.934157123999999, 2024-03-19T11:30:46.067 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013571666, 2024-03-19T11:30:46.067 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:46.067 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:46.067 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:46.067 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:46.067 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:46.067 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1467 2024-03-19T11:30:46.067 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.067 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:46.067 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.067 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:46.067 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.079231+0000", 2024-03-19T11:30:46.067 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:46.068 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.068 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.068 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:46.068 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.079231+0000", 2024-03-19T11:30:46.068 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:46.068 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.068 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.068 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:46.068 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.079234+0000", 2024-03-19T11:30:46.068 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3689999999999998e-06 2024-03-19T11:30:46.068 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.068 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.068 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:46.068 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.080110+0000", 2024-03-19T11:30:46.068 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00087507100000000005 2024-03-19T11:30:46.069 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.069 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.069 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:46.069 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.080146+0000", 2024-03-19T11:30:46.069 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6062000000000001e-05 2024-03-19T11:30:46.069 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.069 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.069 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:46.069 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.080164+0000", 2024-03-19T11:30:46.069 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7995e-05 2024-03-19T11:30:46.069 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.069 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.069 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:46.069 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.080339+0000", 2024-03-19T11:30:46.069 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017535900000000001 2024-03-19T11:30:46.069 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.070 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.070 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:46.070 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.080630+0000", 2024-03-19T11:30:46.070 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029068900000000001 2024-03-19T11:30:46.070 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.070 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.070 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:46.070 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.081140+0000", 2024-03-19T11:30:46.070 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00051083699999999999 2024-03-19T11:30:46.070 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.070 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.070 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:46.070 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.092570+0000", 2024-03-19T11:30:46.070 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011429557 2024-03-19T11:30:46.070 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.071 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.071 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:46.071 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.092611+0000", 2024-03-19T11:30:46.071 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1196000000000003e-05 2024-03-19T11:30:46.071 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.071 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.071 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:46.071 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.092717+0000", 2024-03-19T11:30:46.071 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000105777 2024-03-19T11:30:46.071 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.071 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.071 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:46.071 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.092803+0000", 2024-03-19T11:30:46.071 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5754000000000006e-05 2024-03-19T11:30:46.071 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:46.072 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:46.072 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:46.072 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.072 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.072 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1468 4.a 4:567251f3:::smithi03829856-44:head [write 1603467~416361] snapc c=[c,b,a,6,5] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-03-19T11:30:46.072 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.083922+0000", 2024-03-19T11:30:46.072 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.929466633000001, 2024-03-19T11:30:46.072 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014508549000000001, 2024-03-19T11:30:46.072 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:46.072 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:46.072 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:46.072 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:46.072 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:46.072 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1468 2024-03-19T11:30:46.072 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.073 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:46.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:46.073 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.083922+0000", 2024-03-19T11:30:46.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:46.073 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:46.073 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.083922+0000", 2024-03-19T11:30:46.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:46.073 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:46.073 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.083924+0000", 2024-03-19T11:30:46.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2800000000000002e-06 2024-03-19T11:30:46.074 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.074 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.074 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:46.074 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.084498+0000", 2024-03-19T11:30:46.074 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057414900000000004 2024-03-19T11:30:46.074 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.074 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.074 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:46.074 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.084529+0000", 2024-03-19T11:30:46.074 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1080000000000001e-05 2024-03-19T11:30:46.074 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.074 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.074 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:46.074 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.084544+0000", 2024-03-19T11:30:46.074 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4498000000000001e-05 2024-03-19T11:30:46.075 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.075 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.075 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:46.075 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.088993+0000", 2024-03-19T11:30:46.075 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044497260000000002 2024-03-19T11:30:46.075 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.075 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.075 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:46.075 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.089152+0000", 2024-03-19T11:30:46.075 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015833199999999999 2024-03-19T11:30:46.075 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.075 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.075 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:46.075 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.089419+0000", 2024-03-19T11:30:46.075 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026691799999999999 2024-03-19T11:30:46.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.076 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:46.076 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.092883+0000", 2024-03-19T11:30:46.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034646590000000001 2024-03-19T11:30:46.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.076 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:46.076 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.098276+0000", 2024-03-19T11:30:46.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053924680000000001 2024-03-19T11:30:46.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.076 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:46.076 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.098354+0000", 2024-03-19T11:30:46.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8763999999999993e-05 2024-03-19T11:30:46.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.077 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:46.077 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.098430+0000", 2024-03-19T11:30:46.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5674999999999996e-05 2024-03-19T11:30:46.077 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:46.077 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:46.077 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:46.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.077 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.1 50/49 [PushOp(4:866f3dcc:::smithi03829856-31:7, version: 38'44, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@38'44, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 45'62, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@45'62, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 46'68, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@46'68, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:7, version: 47'71, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@47'71, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 47'72, data_included: [440350~816629,1309629~231947], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@47'72, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:30:46.077 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.959457+0000", 2024-03-19T11:30:46.077 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.053931348999999, 2024-03-19T11:30:46.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018416666000000002, 2024-03-19T11:30:46.077 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:46.077 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:46.078 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:46.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.078 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:46.078 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.959457+0000", 2024-03-19T11:30:46.078 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:46.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.078 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:46.078 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.959457+0000", 2024-03-19T11:30:46.078 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:46.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.078 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:46.078 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.959458+0000", 2024-03-19T11:30:46.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4899999999999999e-06 2024-03-19T11:30:46.079 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.079 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.079 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:46.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.962822+0000", 2024-03-19T11:30:46.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0033640620000000001 2024-03-19T11:30:46.079 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.079 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.079 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:46.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.962893+0000", 2024-03-19T11:30:46.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1060000000000001e-05 2024-03-19T11:30:46.079 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.079 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.079 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:46.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.962977+0000", 2024-03-19T11:30:46.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3311000000000002e-05 2024-03-19T11:30:46.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.080 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:46.080 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963208+0000", 2024-03-19T11:30:46.080 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023121200000000001 2024-03-19T11:30:46.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.080 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:46.080 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963245+0000", 2024-03-19T11:30:46.080 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6844999999999997e-05 2024-03-19T11:30:46.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.080 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:46.080 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.977874+0000", 2024-03-19T11:30:46.080 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014628686 2024-03-19T11:30:46.081 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:46.081 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:46.081 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:46.081 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.081 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.081 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.1 e50,49 4:8401c1f0:::smithi03829856-43:8,46'66,4:866f3dcc:::smithi03829856-31:2,47'73)", 2024-03-19T11:30:46.081 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.963027+0000", 2024-03-19T11:30:46.081 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.050361731999999, 2024-03-19T11:30:46.081 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015116788000000001, 2024-03-19T11:30:46.081 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:46.081 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:46.081 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:46.081 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.081 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:46.081 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963027+0000", 2024-03-19T11:30:46.082 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:46.082 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.082 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.082 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:46.082 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963027+0000", 2024-03-19T11:30:46.082 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:46.082 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.082 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.082 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:46.082 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963028+0000", 2024-03-19T11:30:46.082 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.268e-06 2024-03-19T11:30:46.082 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.082 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.082 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:46.082 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963036+0000", 2024-03-19T11:30:46.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.926e-06 2024-03-19T11:30:46.083 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.083 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:46.083 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963050+0000", 2024-03-19T11:30:46.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4082e-05 2024-03-19T11:30:46.083 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.083 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:46.083 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963062+0000", 2024-03-19T11:30:46.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2415000000000001e-05 2024-03-19T11:30:46.083 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.083 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:46.083 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.977954+0000", 2024-03-19T11:30:46.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014891572 2024-03-19T11:30:46.084 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.084 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.084 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:46.084 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.977965+0000", 2024-03-19T11:30:46.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1654000000000001e-05 2024-03-19T11:30:46.084 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.084 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.084 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:46.084 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.978143+0000", 2024-03-19T11:30:46.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017787099999999999 2024-03-19T11:30:46.084 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:46.084 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:46.084 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:46.084 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.085 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.11 50/49 [PushOp(4:8d3763f9:::smithi03829856-29:head, version: 47'60, data_included: [3364276~232389], data_size: 232389, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8d3763f9:::smithi03829856-29:head@47'60, size: 3596665, copy_subset: [(0, 3596665)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3596665, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3364276, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:30:46.085 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.964363+0000", 2024-03-19T11:30:46.085 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.049025064999999, 2024-03-19T11:30:46.085 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014531759999999999, 2024-03-19T11:30:46.085 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:46.085 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:46.085 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:46.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.085 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:46.085 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.964363+0000", 2024-03-19T11:30:46.085 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:46.085 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.085 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:46.086 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.964363+0000", 2024-03-19T11:30:46.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:46.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.086 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.086 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:46.086 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.964364+0000", 2024-03-19T11:30:46.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4200000000000004e-07 2024-03-19T11:30:46.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.086 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.086 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:46.086 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.964447+0000", 2024-03-19T11:30:46.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2430999999999997e-05 2024-03-19T11:30:46.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.086 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.086 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:46.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.964468+0000", 2024-03-19T11:30:46.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1685999999999999e-05 2024-03-19T11:30:46.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:46.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.964483+0000", 2024-03-19T11:30:46.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4969e-05 2024-03-19T11:30:46.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:46.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.977934+0000", 2024-03-19T11:30:46.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013450963999999999 2024-03-19T11:30:46.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:46.088 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.977975+0000", 2024-03-19T11:30:46.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1125999999999999e-05 2024-03-19T11:30:46.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.088 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:46.088 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.978895+0000", 2024-03-19T11:30:46.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00091964200000000003 2024-03-19T11:30:46.088 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:46.088 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:46.088 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:46.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.088 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.1 50/49 [PushOp(4:866f3dcc:::smithi03829856-31:7, version: 38'44, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@38'44, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 45'62, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@45'62, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 46'68, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@46'68, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:7, version: 47'71, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@47'71, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 47'72, data_included: [1541576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@47'72, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:30:46.088 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.991858+0000", 2024-03-19T11:30:46.088 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.021530142, 2024-03-19T11:30:46.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014228774, 2024-03-19T11:30:46.089 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:46.089 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:46.089 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:46.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.089 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:46.089 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.991858+0000", 2024-03-19T11:30:46.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:46.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.089 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:46.089 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.991858+0000", 2024-03-19T11:30:46.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:46.090 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:46.090 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.991859+0000", 2024-03-19T11:30:46.090 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.342e-06 2024-03-19T11:30:46.090 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:46.090 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.994675+0000", 2024-03-19T11:30:46.090 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028153150000000001 2024-03-19T11:30:46.090 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:46.090 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.994733+0000", 2024-03-19T11:30:46.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8514999999999998e-05 2024-03-19T11:30:46.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.091 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:46.091 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.994794+0000", 2024-03-19T11:30:46.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0588000000000003e-05 2024-03-19T11:30:46.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.091 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:46.091 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.995070+0000", 2024-03-19T11:30:46.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027573700000000002 2024-03-19T11:30:46.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.091 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:46.091 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.995100+0000", 2024-03-19T11:30:46.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0679000000000002e-05 2024-03-19T11:30:46.092 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.092 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:46.092 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.006087+0000", 2024-03-19T11:30:46.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010986598 2024-03-19T11:30:46.092 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:46.092 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:46.092 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:46.092 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.092 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.1 50/49 [PushOp(4:866f3dcc:::smithi03829856-31:head, version: 47'74, data_included: [504210~968657,1572695~79919], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:head@47'74, size: 3495823, copy_subset: [(0, 3495823)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:30:46.092 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.995921+0000", 2024-03-19T11:30:46.092 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.017466928000001, 2024-03-19T11:30:46.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014183898, 2024-03-19T11:30:46.093 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:46.093 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:46.093 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:46.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.093 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:46.093 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.995921+0000", 2024-03-19T11:30:46.093 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:46.093 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.093 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:46.093 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.995921+0000", 2024-03-19T11:30:46.093 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:46.093 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.093 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:46.094 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.995922+0000", 2024-03-19T11:30:46.094 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9800000000000002e-07 2024-03-19T11:30:46.094 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.094 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:46.094 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.996355+0000", 2024-03-19T11:30:46.094 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000432211 2024-03-19T11:30:46.094 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.094 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:46.094 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.996386+0000", 2024-03-19T11:30:46.094 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1384999999999999e-05 2024-03-19T11:30:46.094 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.094 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:46.095 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.996412+0000", 2024-03-19T11:30:46.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6200999999999999e-05 2024-03-19T11:30:46.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.095 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:46.095 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.006166+0000", 2024-03-19T11:30:46.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0097542859999999992 2024-03-19T11:30:46.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.095 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:46.095 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.006182+0000", 2024-03-19T11:30:46.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5588000000000001e-05 2024-03-19T11:30:46.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.095 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:46.096 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.010105+0000", 2024-03-19T11:30:46.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039232290000000003 2024-03-19T11:30:46.096 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:46.096 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:46.096 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:46.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.096 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1679 4.e e59/21 4:72163e4c:::smithi03829856-39:head v 59'55, mlcod=58'53)", 2024-03-19T11:30:46.096 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:36.390395+0000", 2024-03-19T11:30:46.096 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.6229930380000006, 2024-03-19T11:30:46.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013872282, 2024-03-19T11:30:46.096 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:46.096 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:46.096 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:46.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:46.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.390395+0000", 2024-03-19T11:30:46.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:46.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:46.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.390395+0000", 2024-03-19T11:30:46.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:46.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:46.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.390396+0000", 2024-03-19T11:30:46.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2300000000000002e-07 2024-03-19T11:30:46.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:46.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.390742+0000", 2024-03-19T11:30:46.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00034635599999999998 2024-03-19T11:30:46.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:46.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.390754+0000", 2024-03-19T11:30:46.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2211999999999999e-05 2024-03-19T11:30:46.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:46.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.390763+0000", 2024-03-19T11:30:46.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5769999999999994e-06 2024-03-19T11:30:46.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.099 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:46.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.390864+0000", 2024-03-19T11:30:46.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000101693 2024-03-19T11:30:46.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.099 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:46.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.390907+0000", 2024-03-19T11:30:46.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2642999999999998e-05 2024-03-19T11:30:46.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.099 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:46.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.404144+0000", 2024-03-19T11:30:46.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013237319000000001 2024-03-19T11:30:46.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:46.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:46.100 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:46.100 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.404267+0000", 2024-03-19T11:30:46.100 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012315899999999999 2024-03-19T11:30:46.100 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:46.100 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:46.100 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:46.100 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:46.100 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:46.100 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:46.101 INFO:tasks.rados.rados.0.smithi038.stdout:1033: finishing write tid 1 to smithi03829856-13 2024-03-19T11:30:46.101 INFO:tasks.rados.rados.0.smithi038.stdout:1034: finishing set_chunk to oid 9 2024-03-19T11:30:46.101 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 9 set_chunk returned error code -95 offset: 308224 length: 46592 2024-03-19T11:30:46.101 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 9 v 0 (ObjNum 152 snap 17 seq_num 152) dirty exists 2024-03-19T11:30:46.101 INFO:tasks.rados.rados.0.smithi038.stdout:1033: finishing write tid 2 to smithi03829856-13 2024-03-19T11:30:46.101 INFO:tasks.rados.rados.0.smithi038.stdout:1033: finishing write tid 3 to smithi03829856-13 2024-03-19T11:30:46.101 INFO:tasks.rados.rados.0.smithi038.stdout:1033: finishing write tid 4 to smithi03829856-13 2024-03-19T11:30:46.101 INFO:tasks.rados.rados.0.smithi038.stdout:1033: oid 13 updating version 0 to 70 2024-03-19T11:30:46.102 INFO:tasks.rados.rados.0.smithi038.stdout:1033: oid 13 version 70 is already newer than 69 2024-03-19T11:30:46.102 INFO:tasks.rados.rados.0.smithi038.stdout:1033: oid 13 version 70 is already newer than 68 2024-03-19T11:30:46.102 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 13 v 70 (ObjNum 162 snap 17 seq_num 162) dirty exists 2024-03-19T11:30:46.102 INFO:tasks.rados.rados.0.smithi038.stdout:1033: left oid 13 (ObjNum 162 snap 17 seq_num 162) 2024-03-19T11:30:46.102 INFO:tasks.rados.rados.0.smithi038.stdout:1035: expect (ObjNum 124 snap 15 seq_num 124) 2024-03-19T11:30:46.103 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-19T11:30:46.103 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:46.108+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:46.156 INFO:tasks.rados.rados.0.smithi038.stdout:1036: finishing write tid 1 to smithi03829856-39 2024-03-19T11:30:46.156 INFO:tasks.rados.rados.0.smithi038.stdout:1036: finishing write tid 2 to smithi03829856-39 2024-03-19T11:30:46.156 INFO:tasks.rados.rados.0.smithi038.stdout:1039: finishing rollback tid 1 to smithi03829856-6 2024-03-19T11:30:46.156 INFO:tasks.rados.rados.0.smithi038.stdout:1036: finishing write tid 3 to smithi03829856-39 2024-03-19T11:30:46.156 INFO:tasks.rados.rados.0.smithi038.stdout:1036: finishing write tid 4 to smithi03829856-39 2024-03-19T11:30:46.157 INFO:tasks.rados.rados.0.smithi038.stdout:1036: oid 39 updating version 0 to 69 2024-03-19T11:30:46.157 INFO:tasks.rados.rados.0.smithi038.stdout:1036: oid 39 version 69 is already newer than 68 2024-03-19T11:30:46.157 INFO:tasks.rados.rados.0.smithi038.stdout:1036: oid 39 version 69 is already newer than 67 2024-03-19T11:30:46.157 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 39 v 69 (ObjNum 163 snap 17 seq_num 163) dirty exists 2024-03-19T11:30:46.157 INFO:tasks.rados.rados.0.smithi038.stdout:1036: left oid 39 (ObjNum 163 snap 17 seq_num 163) 2024-03-19T11:30:46.157 INFO:tasks.rados.rados.0.smithi038.stdout:1039: finishing rollback tid 2 to smithi03829856-6 2024-03-19T11:30:46.157 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 6 v 66 (ObjNum 53 snap 0 seq_num 53) dirty exists 2024-03-19T11:30:46.157 INFO:tasks.rados.rados.0.smithi038.stdout:1038: expect (ObjNum 46 snap 0 seq_num 46) 2024-03-19T11:30:46.183 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-19T11:30:46.184 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-19T11:30:46.184 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_0 2024-03-19T11:30:46.184 INFO:tasks.ceph.ceph_manager.ceph:contract_pool unique_pool_0 by 10 min 8 2024-03-19T11:30:46.184 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T11:30:46.202 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:46.208+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:46.252 INFO:tasks.rados.rados.0.smithi038.stdout:1037: got (0) Success 2024-03-19T11:30:46.252 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 10 v 73 (ObjNum 144 snap 15 seq_num 144) dirty exists 2024-03-19T11:30:46.252 INFO:tasks.rados.rados.0.smithi038.stdout:1031: done (8 left) 2024-03-19T11:30:46.252 INFO:tasks.rados.rados.0.smithi038.stdout:1032: done (7 left) 2024-03-19T11:30:46.252 INFO:tasks.rados.rados.0.smithi038.stdout:1033: done (6 left) 2024-03-19T11:30:46.252 INFO:tasks.rados.rados.0.smithi038.stdout:1034: done (5 left) 2024-03-19T11:30:46.252 INFO:tasks.rados.rados.0.smithi038.stdout:1035: done (4 left) 2024-03-19T11:30:46.252 INFO:tasks.rados.rados.0.smithi038.stdout:1036: done (3 left) 2024-03-19T11:30:46.253 INFO:tasks.rados.rados.0.smithi038.stdout:1037: done (2 left) 2024-03-19T11:30:46.253 INFO:tasks.rados.rados.0.smithi038.stdout:1038: done (1 left) 2024-03-19T11:30:46.253 INFO:tasks.rados.rados.0.smithi038.stdout:1039: done (0 left) 2024-03-19T11:30:46.253 INFO:tasks.rados.rados.0.smithi038.stdout:1040: tier_evict oid 7 2024-03-19T11:30:46.253 INFO:tasks.rados.rados.0.smithi038.stdout:1040: tier_evict oid 7 snap -1 2024-03-19T11:30:46.253 INFO:tasks.rados.rados.0.smithi038.stdout:1041: read oid 19 snap 10 2024-03-19T11:30:46.253 INFO:tasks.rados.rados.0.smithi038.stdout:1041: expect (ObjNum 93 snap 10 seq_num 93) 2024-03-19T11:30:46.253 INFO:tasks.rados.rados.0.smithi038.stdout:1042: read oid 2 snap 4 2024-03-19T11:30:46.253 INFO:tasks.rados.rados.0.smithi038.stdout:1042: expect (ObjNum 1 snap 0 seq_num 1) 2024-03-19T11:30:46.253 INFO:tasks.rados.rados.0.smithi038.stdout:1043: read oid 42 snap -1 2024-03-19T11:30:46.253 INFO:tasks.rados.rados.0.smithi038.stdout:1043: expect deleted 2024-03-19T11:30:46.253 INFO:tasks.rados.rados.0.smithi038.stdout:1044: read oid 31 snap -1 2024-03-19T11:30:46.253 INFO:tasks.rados.rados.0.smithi038.stdout:1044: expect (ObjNum 157 snap 17 seq_num 157) 2024-03-19T11:30:46.253 INFO:tasks.rados.rados.0.smithi038.stdout:1045: tier_promote oid 12 2024-03-19T11:30:46.253 INFO:tasks.rados.rados.0.smithi038.stdout:1046: rollback oid 50 current snap is 17 2024-03-19T11:30:46.254 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 50 to 11 2024-03-19T11:30:46.254 INFO:tasks.rados.rados.0.smithi038.stdout:1047: read oid 3 snap -1 2024-03-19T11:30:46.254 INFO:tasks.rados.rados.0.smithi038.stdout:1047: expect (ObjNum 101 snap 10 seq_num 101) 2024-03-19T11:30:46.254 INFO:tasks.rados.rados.0.smithi038.stdout:1048: read oid 27 snap -1 2024-03-19T11:30:46.254 INFO:tasks.rados.rados.0.smithi038.stdout:1048: expect deleted 2024-03-19T11:30:46.254 INFO:tasks.rados.rados.0.smithi038.stdout:1049: read oid 32 snap -1 2024-03-19T11:30:46.254 INFO:tasks.rados.rados.0.smithi038.stdout:1049: expect (ObjNum 31 snap 0 seq_num 31) 2024-03-19T11:30:46.254 INFO:tasks.rados.rados.0.smithi038.stdout:1050: read oid 40 snap -1 2024-03-19T11:30:46.254 INFO:tasks.rados.rados.0.smithi038.stdout:1050: expect (ObjNum 141 snap 15 seq_num 141) 2024-03-19T11:30:46.255 INFO:tasks.rados.rados.0.smithi038.stdout:1051: read oid 34 snap -1 2024-03-19T11:30:46.255 INFO:tasks.rados.rados.0.smithi038.stdout:1051: expect (ObjNum 72 snap 5 seq_num 72) 2024-03-19T11:30:46.255 INFO:tasks.rados.rados.0.smithi038.stdout:1052: read oid 22 snap -1 2024-03-19T11:30:46.255 INFO:tasks.rados.rados.0.smithi038.stdout:1052: expect (ObjNum 21 snap 0 seq_num 21) 2024-03-19T11:30:46.255 INFO:tasks.rados.rados.0.smithi038.stdout:1053: read oid 4 snap -1 2024-03-19T11:30:46.255 INFO:tasks.rados.rados.0.smithi038.stdout:1053: expect (ObjNum 151 snap 17 seq_num 151) 2024-03-19T11:30:46.255 INFO:tasks.rados.rados.0.smithi038.stdout:1054: write oid 36 current snap is 17 2024-03-19T11:30:46.255 INFO:tasks.rados.rados.0.smithi038.stdout:1054: seq_num 164 ranges {756348=784249,2063988=686387} 2024-03-19T11:30:46.255 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:46.260+0000 7fb17bd6e640 -1 osd.4 67 handle_osd_map injecting map crc failure 2024-03-19T11:30:46.262 INFO:tasks.rados.rados.0.smithi038.stdout:1054: writing smithi03829856-36 from 756348 to 1540597 tid 1 2024-03-19T11:30:46.273 INFO:tasks.rados.rados.0.smithi038.stdout:1054: writing smithi03829856-36 from 2063988 to 2750375 tid 2 2024-03-19T11:30:46.273 INFO:tasks.rados.rados.0.smithi038.stdout:1040: got (22) Invalid argument 2024-03-19T11:30:46.273 INFO:tasks.rados.rados.0.smithi038.stdout:1042: expect (ObjNum 1 snap 0 seq_num 1) 2024-03-19T11:30:46.286 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:46.286 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:46.286 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:30:46.286 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:46.301 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-19T11:30:46.339 INFO:tasks.rados.rados.0.smithi038.stdout:1045: got (0) Success 2024-03-19T11:30:46.339 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 12 v 83 (ObjNum 161 snap 17 seq_num 161) dirty exists 2024-03-19T11:30:46.339 INFO:tasks.rados.rados.0.smithi038.stdout:1049: expect (ObjNum 31 snap 0 seq_num 31) 2024-03-19T11:30:46.343 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:46.352+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:46.358 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-19T11:30:46.403 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:46.412+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:46.418 INFO:tasks.rados.rados.0.smithi038.stdout:1053: expect (ObjNum 151 snap 17 seq_num 151) 2024-03-19T11:30:46.468 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:46.468 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:46.468 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:30:46.469 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:30:46.469 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:46.482 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-19T11:30:46.499 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:30:46.499 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:30:46.504 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:46.514+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:46.517 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:30:46.517 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:30:46.529 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:30:46.530 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-03-19T11:30:46.534 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:30:46.534 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:30:46.536 INFO:tasks.rados.rados.0.smithi038.stdout:1051: expect (ObjNum 72 snap 5 seq_num 72) 2024-03-19T11:30:46.548 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":93,"stamp":"2024-03-19T11:30:46.242259+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":387941838,"num_objects":242,"num_object_clones":75,"num_object_copies":484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":239,"num_whiteouts":7,"num_read":1019,"num_read_kb":647760,"num_write":713,"num_write_kb":226311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":62831205,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":78,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2308,"ondisk_log_size":2308,"up":98,"acting":98,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":98,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":687412,"kb_used_data":455612,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":754287308,"statfs":{"total":773094113280,"available":772390203392,"internally_reserved":0,"allocated":466546688,"data_stored":545562173,"data_compressed":12683151,"data_compressed_allocated":13164544,"data_compressed_original":30371840,"omap_allocated":12751,"internal_metadata":234016305},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18074908,"num_objects":11,"num_object_clones":13,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":26,"num_read_kb":42834,"num_write":-35,"num_write_kb":-9657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-13,"num_bytes_recovered":-13785755,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.061443"},"pg_stats":[{"pgid":"4.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-19T11:30:44.200875+0000","last_change":"2024-03-19T11:30:44.200875+0000","last_active":"2024-03-19T11:30:44.200875+0000","last_peered":"2024-03-19T11:30:44.200875+0000","last_clean":"2024-03-19T11:30:44.200875+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T11:30:44.200875+0000","last_undegraded":"2024-03-19T11:30:44.200875+0000","last_fullsized":"2024-03-19T11:30:44.200875+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:30:44.200875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:30:44.200875+0000","last_clean_scrub_stamp":"2024-03-19T11:30:44.200875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-19T11:30:44.200875+0000","last_change":"2024-03-19T11:30:44.200875+0000","last_active":"2024-03-19T11:30:44.200875+0000","last_peered":"2024-03-19T11:30:44.200875+0000","last_clean":"2024-03-19T11:30:44.200875+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T11:30:44.200875+0000","last_undegraded":"2024-03-19T11:30:44.200875+0000","last_fullsized":"2024-03-19T11:30:44.200875+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:30:44.200875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:30:44.200875+0000","last_clean_scrub_stamp":"2024-03-19T11:30:44.200875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-19T11:30:44.200875+0000","last_change":"2024-03-19T11:30:44.200875+0000","last_active":"2024-03-19T11:30:44.200875+0000","last_peered":"2024-03-19T11:30:44.200875+0000","last_clean":"2024-03-19T11:30:44.200875+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T11:30:44.200875+0000","last_undegraded":"2024-03-19T11:30:44.200875+0000","last_fullsized":"2024-03-19T11:30:44.200875+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:30:44.200875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:30:44.200875+0000","last_clean_scrub_stamp":"2024-03-19T11:30:44.200875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-19T11:30:44.200875+0000","last_change":"2024-03-19T11:30:44.200875+0000","last_active":"2024-03-19T11:30:44.200875+0000","last_peered":"2024-03-19T11:30:44.200875+0000","last_clean":"2024-03-19T11:30:44.200875+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T11:30:44.200875+0000","last_undegraded":"2024-03-19T11:30:44.200875+0000","last_fullsized":"2024-03-19T11:30:44.200875+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:30:44.200875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:30:44.200875+0000","last_clean_scrub_stamp":"2024-03-19T11:30:44.200875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-19T11:30:44.200875+0000","last_change":"2024-03-19T11:30:44.200875+0000","last_active":"2024-03-19T11:30:44.200875+0000","last_peered":"2024-03-19T11:30:44.200875+0000","last_clean":"2024-03-19T11:30:44.200875+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T11:30:44.200875+0000","last_undegraded":"2024-03-19T11:30:44.200875+0000","last_fullsized":"2024-03-19T11:30:44.200875+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:30:44.200875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:30:44.200875+0000","last_clean_scrub_stamp":"2024-03-19T11:30:44.200875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-19T11:30:44.200875+0000","last_change":"2024-03-19T11:30:44.200875+0000","last_active":"2024-03-19T11:30:44.200875+0000","last_peered":"2024-03-19T11:30:44.200875+0000","last_clean":"2024-03-19T11:30:44.200875+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T11:30:44.200875+0000","last_undegraded":"2024-03-19T11:30:44.200875+0000","last_fullsized":"2024-03-19T11:30:44.200875+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:30:44.200875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:30:44.200875+0000","last_clean_scrub_stamp":"2024-03-19T11:30:44.200875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-19T11:30:44.200875+0000","last_change":"2024-03-19T11:30:44.200875+0000","last_active":"2024-03-19T11:30:44.200875+0000","last_peered":"2024-03-19T11:30:44.200875+0000","last_clean":"2024-03-19T11:30:44.200875+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T11:30:44.200875+0000","last_undegraded":"2024-03-19T11:30:44.200875+0000","last_fullsized":"2024-03-19T11:30:44.200875+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:30:44.200875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:30:44.200875+0000","last_clean_scrub_stamp":"2024-03-19T11:30:44.200875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-19T11:30:44.200875+0000","last_change":"2024-03-19T11:30:44.200875+0000","last_active":"2024-03-19T11:30:44.200875+0000","last_peered":"2024-03-19T11:30:44.200875+0000","last_clean":"2024-03-19T11:30:44.200875+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T11:30:44.200875+0000","last_undegraded":"2024-03-19T11:30:44.200875+0000","last_fullsized":"2024-03-19T11:30:44.200875+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:30:44.200875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:30:44.200875+0000","last_clean_scrub_stamp":"2024-03-19T11:30:44.200875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-19T11:30:44.200875+0000","last_change":"2024-03-19T11:30:44.200875+0000","last_active":"2024-03-19T11:30:44.200875+0000","last_peered":"2024-03-19T11:30:44.200875+0000","last_clean":"2024-03-19T11:30:44.200875+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T11:30:44.200875+0000","last_undegraded":"2024-03-19T11:30:44.200875+0000","last_fullsized":"2024-03-19T11:30:44.200875+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:30:44.200875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:30:44.200875+0000","last_clean_scrub_stamp":"2024-03-19T11:30:44.200875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-19T11:30:44.200875+0000","last_change":"2024-03-19T11:30:44.200875+0000","last_active":"2024-03-19T11:30:44.200875+0000","last_peered":"2024-03-19T11:30:44.200875+0000","last_clean":"2024-03-19T11:30:44.200875+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-19T11:30:44.200875+0000","last_undegraded":"2024-03-19T11:30:44.200875+0000","last_fullsized":"2024-03-19T11:30:44.200875+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:30:44.200875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:30:44.200875+0000","last_clean_scrub_stamp":"2024-03-19T11:30:44.200875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.23","version":"62'23","reported_seq":231,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-03-19T11:30:41.171800+0000","last_change":"2024-03-19T11:30:38.061772+0000","last_active":"2024-03-19T11:30:41.171800+0000","last_peered":"2024-03-19T11:30:41.171800+0000","last_clean":"2024-03-19T11:30:41.171800+0000","last_became_active":"2024-03-19T11:30:38.059263+0000","last_became_peered":"2024-03-19T11:30:38.059263+0000","last_unstale":"2024-03-19T11:30:41.171800+0000","last_undegraded":"2024-03-19T11:30:41.171800+0000","last_fullsized":"2024-03-19T11:30:41.171800+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":6,"last_scrub":"41'19","last_scrub_stamp":"2024-03-19T11:30:28.180772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:28.180772+0000","objects_scrubbed":1,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001840455,"stat_sum":{"num_bytes":2325434,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":5661,"num_write":4,"num_write_kb":901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.22","version":"28'6","reported_seq":131,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-19T11:30:43.204541+0000","last_change":"2024-03-19T11:30:42.357960+0000","last_active":"2024-03-19T11:30:43.204541+0000","last_peered":"2024-03-19T11:30:43.204541+0000","last_clean":"2024-03-19T11:30:43.204541+0000","last_became_active":"2024-03-19T11:30:38.057759+0000","last_became_peered":"2024-03-19T11:30:38.057759+0000","last_unstale":"2024-03-19T11:30:43.204541+0000","last_undegraded":"2024-03-19T11:30:43.204541+0000","last_fullsized":"2024-03-19T11:30:43.204541+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":6,"last_scrub":"28'6","last_scrub_stamp":"2024-03-19T11:30:42.357697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.357697+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:50.724242+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.002595619,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.21","version":"62'96","reported_seq":354,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-19T11:30:43.401052+0000","last_change":"2024-03-19T11:30:43.401143+0000","last_active":"2024-03-19T11:30:43.401052+0000","last_peered":"2024-03-19T11:30:43.401052+0000","last_clean":"2024-03-19T11:30:43.401052+0000","last_became_active":"2024-03-19T11:30:38.061758+0000","last_became_peered":"2024-03-19T11:30:38.061758+0000","last_unstale":"2024-03-19T11:30:43.401052+0000","last_undegraded":"2024-03-19T11:30:43.401052+0000","last_fullsized":"2024-03-19T11:30:43.401052+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":6,"last_scrub":"62'96","last_scrub_stamp":"2024-03-19T11:30:43.400885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:43.400885+0000","objects_scrubbed":9,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:48.652908+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.94141351600000001,"stat_sum":{"num_bytes":13743103,"num_objects":9,"num_object_clones":7,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":7,"num_read_kb":8587,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.20","version":"56'46","reported_seq":252,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-19T11:30:43.230017+0000","last_change":"2024-03-19T11:30:41.767294+0000","last_active":"2024-03-19T11:30:43.230017+0000","last_peered":"2024-03-19T11:30:43.230017+0000","last_clean":"2024-03-19T11:30:43.230017+0000","last_became_active":"2024-03-19T11:30:38.072125+0000","last_became_peered":"2024-03-19T11:30:38.072125+0000","last_unstale":"2024-03-19T11:30:43.230017+0000","last_undegraded":"2024-03-19T11:30:43.230017+0000","last_fullsized":"2024-03-19T11:30:43.230017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":6,"last_scrub":"56'46","last_scrub_stamp":"2024-03-19T11:30:41.767100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:41.767100+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:53.119208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086159500000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.1f","version":"0'0","reported_seq":180,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-19T11:30:43.248349+0000","last_change":"2024-03-19T11:30:42.378199+0000","last_active":"2024-03-19T11:30:43.248349+0000","last_peered":"2024-03-19T11:30:43.248349+0000","last_clean":"2024-03-19T11:30:43.248349+0000","last_became_active":"2024-03-19T11:30:38.055108+0000","last_became_peered":"2024-03-19T11:30:38.055108+0000","last_unstale":"2024-03-19T11:30:43.248349+0000","last_undegraded":"2024-03-19T11:30:43.248349+0000","last_fullsized":"2024-03-19T11:30:43.248349+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:30:42.377963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.377963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:48.233650+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0068887510000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.1e","version":"64'62","reported_seq":256,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-03-19T11:30:42.778358+0000","last_change":"2024-03-19T11:30:42.778446+0000","last_active":"2024-03-19T11:30:42.778358+0000","last_peered":"2024-03-19T11:30:42.778358+0000","last_clean":"2024-03-19T11:30:42.778358+0000","last_became_active":"2024-03-19T11:30:38.071044+0000","last_became_peered":"2024-03-19T11:30:38.071044+0000","last_unstale":"2024-03-19T11:30:42.778358+0000","last_undegraded":"2024-03-19T11:30:42.778358+0000","last_fullsized":"2024-03-19T11:30:42.778358+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":5,"last_scrub":"64'62","last_scrub_stamp":"2024-03-19T11:30:42.778159+0000","last_deep_scrub":"64'62","last_deep_scrub_stamp":"2024-03-19T11:30:42.778159+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.778159+0000","objects_scrubbed":4,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:56.771748+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0.0024433739999999999,"stat_sum":{"num_bytes":5661117,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.1d","version":"61'59","reported_seq":271,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-19T11:30:43.204007+0000","last_change":"2024-03-19T11:30:42.602052+0000","last_active":"2024-03-19T11:30:43.204007+0000","last_peered":"2024-03-19T11:30:43.204007+0000","last_clean":"2024-03-19T11:30:43.204007+0000","last_became_active":"2024-03-19T11:30:38.052224+0000","last_became_peered":"2024-03-19T11:30:38.052224+0000","last_unstale":"2024-03-19T11:30:43.204007+0000","last_undegraded":"2024-03-19T11:30:43.204007+0000","last_fullsized":"2024-03-19T11:30:43.204007+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":5,"last_scrub":"61'59","last_scrub_stamp":"2024-03-19T11:30:42.601863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.601863+0000","objects_scrubbed":2,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:05.140766+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.90529512000000001,"stat_sum":{"num_bytes":4835736,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.1c","version":"62'31","reported_seq":48,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-19T11:30:43.229612+0000","last_change":"2024-03-19T11:30:42.771682+0000","last_active":"2024-03-19T11:30:43.229612+0000","last_peered":"2024-03-19T11:30:43.229612+0000","last_clean":"2024-03-19T11:30:43.229612+0000","last_became_active":"2024-03-19T11:30:38.070619+0000","last_became_peered":"2024-03-19T11:30:38.070619+0000","last_unstale":"2024-03-19T11:30:43.229612+0000","last_undegraded":"2024-03-19T11:30:43.229612+0000","last_fullsized":"2024-03-19T11:30:43.229612+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":5,"last_scrub":"62'31","last_scrub_stamp":"2024-03-19T11:30:42.771396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.771396+0000","objects_scrubbed":3,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:00.301545+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00062618899999999998,"stat_sum":{"num_bytes":4506149,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.1b","version":"62'62","reported_seq":287,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-03-19T11:30:41.171084+0000","last_change":"2024-03-19T11:30:38.061307+0000","last_active":"2024-03-19T11:30:41.171084+0000","last_peered":"2024-03-19T11:30:41.171084+0000","last_clean":"2024-03-19T11:30:41.171084+0000","last_became_active":"2024-03-19T11:30:38.059164+0000","last_became_peered":"2024-03-19T11:30:38.059164+0000","last_unstale":"2024-03-19T11:30:41.171084+0000","last_undegraded":"2024-03-19T11:30:41.171084+0000","last_fullsized":"2024-03-19T11:30:41.171084+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015044570000000001,"stat_sum":{"num_bytes":7803777,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":2,"num_read":82,"num_read_kb":77495,"num_write":23,"num_write_kb":5851,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.1a","version":"61'76","reported_seq":293,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-03-19T11:30:42.207026+0000","last_change":"2024-03-19T11:30:41.433737+0000","last_active":"2024-03-19T11:30:42.207026+0000","last_peered":"2024-03-19T11:30:42.207026+0000","last_clean":"2024-03-19T11:30:42.207026+0000","last_became_active":"2024-03-19T11:30:38.069375+0000","last_became_peered":"2024-03-19T11:30:38.069375+0000","last_unstale":"2024-03-19T11:30:42.207026+0000","last_undegraded":"2024-03-19T11:30:42.207026+0000","last_fullsized":"2024-03-19T11:30:42.207026+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":5,"last_scrub":"61'76","last_scrub_stamp":"2024-03-19T11:30:41.433501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:41.433501+0000","objects_scrubbed":5,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:09.435677+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.44029753700000002,"stat_sum":{"num_bytes":11680766,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":9,"num_read_kb":9622,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.19","version":"63'73","reported_seq":330,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-03-19T11:30:42.183481+0000","last_change":"2024-03-19T11:30:38.058213+0000","last_active":"2024-03-19T11:30:42.183481+0000","last_peered":"2024-03-19T11:30:42.183481+0000","last_clean":"2024-03-19T11:30:42.183481+0000","last_became_active":"2024-03-19T11:30:30.051696+0000","last_became_peered":"2024-03-19T11:30:30.051696+0000","last_unstale":"2024-03-19T11:30:42.183481+0000","last_undegraded":"2024-03-19T11:30:42.183481+0000","last_fullsized":"2024-03-19T11:30:42.183481+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":5,"last_scrub":"41'37","last_scrub_stamp":"2024-03-19T11:30:27.705159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:27.705159+0000","objects_scrubbed":4,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:32.430835+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.011454681,"stat_sum":{"num_bytes":13984476,"num_objects":8,"num_object_clones":4,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":53815,"num_write":23,"num_write_kb":4295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":8331740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":7,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.18","version":"63'94","reported_seq":287,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-19T11:30:43.205098+0000","last_change":"2024-03-19T11:30:41.168589+0000","last_active":"2024-03-19T11:30:43.205098+0000","last_peered":"2024-03-19T11:30:43.205098+0000","last_clean":"2024-03-19T11:30:43.205098+0000","last_became_active":"2024-03-19T11:30:21.869150+0000","last_became_peered":"2024-03-19T11:30:21.869150+0000","last_unstale":"2024-03-19T11:30:43.205098+0000","last_undegraded":"2024-03-19T11:30:43.205098+0000","last_fullsized":"2024-03-19T11:30:43.205098+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":5,"last_scrub":"48'77","last_scrub_stamp":"2024-03-19T11:30:29.399762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:29.399762+0000","objects_scrubbed":6,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:34.113959+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0019695120000000001,"stat_sum":{"num_bytes":15990414,"num_objects":7,"num_object_clones":5,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":13,"num_read_kb":12516,"num_write":8,"num_write_kb":2274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.17","version":"41'24","reported_seq":171,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-19T11:30:43.204678+0000","last_change":"2024-03-19T11:30:40.103917+0000","last_active":"2024-03-19T11:30:43.204678+0000","last_peered":"2024-03-19T11:30:43.204678+0000","last_clean":"2024-03-19T11:30:43.204678+0000","last_became_active":"2024-03-19T11:30:28.039784+0000","last_became_peered":"2024-03-19T11:30:28.039784+0000","last_unstale":"2024-03-19T11:30:43.204678+0000","last_undegraded":"2024-03-19T11:30:43.204678+0000","last_fullsized":"2024-03-19T11:30:43.204678+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.92963434700000003,"stat_sum":{"num_bytes":3679215,"num_objects":2,"num_object_clones":2,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":9,"num_write_kb":2643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.16","version":"63'74","reported_seq":328,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-19T11:30:43.248446+0000","last_change":"2024-03-19T11:30:38.051924+0000","last_active":"2024-03-19T11:30:43.248446+0000","last_peered":"2024-03-19T11:30:43.248446+0000","last_clean":"2024-03-19T11:30:43.248446+0000","last_became_active":"2024-03-19T11:30:26.921950+0000","last_became_peered":"2024-03-19T11:30:26.921950+0000","last_unstale":"2024-03-19T11:30:43.248446+0000","last_undegraded":"2024-03-19T11:30:43.248446+0000","last_fullsized":"2024-03-19T11:30:43.248446+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":5,"last_scrub":"49'56","last_scrub_stamp":"2024-03-19T11:30:29.212766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:29.212766+0000","objects_scrubbed":5,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:50.807054+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0096679469999999997,"stat_sum":{"num_bytes":12878509,"num_objects":7,"num_object_clones":4,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":56,"num_read_kb":51255,"num_write":10,"num_write_kb":2248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":7,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.15","version":"64'100","reported_seq":400,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-03-19T11:30:42.220264+0000","last_change":"2024-03-19T11:30:40.097411+0000","last_active":"2024-03-19T11:30:42.220264+0000","last_peered":"2024-03-19T11:30:42.220264+0000","last_clean":"2024-03-19T11:30:42.220264+0000","last_became_active":"2024-03-19T11:30:28.043875+0000","last_became_peered":"2024-03-19T11:30:28.043875+0000","last_unstale":"2024-03-19T11:30:42.220264+0000","last_undegraded":"2024-03-19T11:30:42.220264+0000","last_fullsized":"2024-03-19T11:30:42.220264+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":5,"last_scrub":"49'74","last_scrub_stamp":"2024-03-19T11:30:29.457542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:29.457542+0000","objects_scrubbed":7,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:40.820090+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.452283092,"stat_sum":{"num_bytes":20587836,"num_objects":11,"num_object_clones":6,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":57,"num_read_kb":62837,"num_write":14,"num_write_kb":3113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":8,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.14","version":"40'56","reported_seq":229,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-19T11:30:43.203779+0000","last_change":"2024-03-19T11:30:38.043252+0000","last_active":"2024-03-19T11:30:43.203779+0000","last_peered":"2024-03-19T11:30:43.203779+0000","last_clean":"2024-03-19T11:30:43.203779+0000","last_became_active":"2024-03-19T11:30:28.038075+0000","last_became_peered":"2024-03-19T11:30:28.038075+0000","last_unstale":"2024-03-19T11:30:43.203779+0000","last_undegraded":"2024-03-19T11:30:43.203779+0000","last_fullsized":"2024-03-19T11:30:43.203779+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":5,"last_scrub":"40'56","last_scrub_stamp":"2024-03-19T11:30:28.331178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:28.331178+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:33.553926+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0020917280000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.13","version":"48'22","reported_seq":234,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-03-19T11:30:41.172237+0000","last_change":"2024-03-19T11:30:38.044451+0000","last_active":"2024-03-19T11:30:41.172237+0000","last_peered":"2024-03-19T11:30:41.172237+0000","last_clean":"2024-03-19T11:30:41.172237+0000","last_became_active":"2024-03-19T11:30:21.867725+0000","last_became_peered":"2024-03-19T11:30:21.867725+0000","last_unstale":"2024-03-19T11:30:41.172237+0000","last_undegraded":"2024-03-19T11:30:41.172237+0000","last_fullsized":"2024-03-19T11:30:41.172237+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":5,"last_scrub":"48'22","last_scrub_stamp":"2024-03-19T11:30:27.183294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:27.183294+0000","objects_scrubbed":2,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:29.317300+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0018063840000000001,"stat_sum":{"num_bytes":5361367,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":19,"num_read_kb":18850,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.12","version":"28'6","reported_seq":38,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-03-19T11:30:42.391190+0000","last_change":"2024-03-19T11:30:42.391317+0000","last_active":"2024-03-19T11:30:42.391190+0000","last_peered":"2024-03-19T11:30:42.391190+0000","last_clean":"2024-03-19T11:30:42.391190+0000","last_became_active":"2024-03-19T11:30:38.052481+0000","last_became_peered":"2024-03-19T11:30:38.052481+0000","last_unstale":"2024-03-19T11:30:42.391190+0000","last_undegraded":"2024-03-19T11:30:42.391190+0000","last_fullsized":"2024-03-19T11:30:42.391190+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":5,"last_scrub":"28'6","last_scrub_stamp":"2024-03-19T11:30:42.390965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.390965+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:52.946772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0097424030000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.11","version":"65'66","reported_seq":93,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-19T11:30:43.587231+0000","last_change":"2024-03-19T11:30:38.042327+0000","last_active":"2024-03-19T11:30:43.587231+0000","last_peered":"2024-03-19T11:30:43.587231+0000","last_clean":"2024-03-19T11:30:43.587231+0000","last_became_active":"2024-03-19T11:30:30.052184+0000","last_became_peered":"2024-03-19T11:30:30.052184+0000","last_unstale":"2024-03-19T11:30:43.587231+0000","last_undegraded":"2024-03-19T11:30:43.587231+0000","last_fullsized":"2024-03-19T11:30:43.587231+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":5,"last_scrub":"47'60","last_scrub_stamp":"2024-03-19T11:30:30.780348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:30.780348+0000","objects_scrubbed":2,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:43.789768+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00074934399999999999,"stat_sum":{"num_bytes":6175693,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":23161,"num_write":4,"num_write_kb":1352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.10","version":"55'32","reported_seq":252,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-19T11:30:43.229391+0000","last_change":"2024-03-19T11:30:38.043516+0000","last_active":"2024-03-19T11:30:43.229391+0000","last_peered":"2024-03-19T11:30:43.229391+0000","last_clean":"2024-03-19T11:30:43.229391+0000","last_became_active":"2024-03-19T11:30:30.061489+0000","last_became_peered":"2024-03-19T11:30:30.061489+0000","last_unstale":"2024-03-19T11:30:43.229391+0000","last_undegraded":"2024-03-19T11:30:43.229391+0000","last_fullsized":"2024-03-19T11:30:43.229391+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":5,"last_scrub":"50'30","last_scrub_stamp":"2024-03-19T11:30:28.787058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:28.787058+0000","objects_scrubbed":2,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:52.137587+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00071659299999999996,"stat_sum":{"num_bytes":6369787,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":5431,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":3551298,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"2.7","version":"0'0","reported_seq":88,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-19T11:30:43.203959+0000","last_change":"2024-03-19T11:30:03.276845+0000","last_active":"2024-03-19T11:30:43.203959+0000","last_peered":"2024-03-19T11:30:43.203959+0000","last_clean":"2024-03-19T11:30:43.203959+0000","last_became_active":"2024-03-19T11:30:03.276220+0000","last_became_peered":"2024-03-19T11:30:03.276220+0000","last_unstale":"2024-03-19T11:30:43.203959+0000","last_undegraded":"2024-03-19T11:30:43.203959+0000","last_fullsized":"2024-03-19T11:30:43.203959+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:01.829767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"59'92","reported_seq":341,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-19T11:30:43.204822+0000","last_change":"2024-03-19T11:30:41.351623+0000","last_active":"2024-03-19T11:30:43.204822+0000","last_peered":"2024-03-19T11:30:43.204822+0000","last_clean":"2024-03-19T11:30:43.204822+0000","last_became_active":"2024-03-19T11:30:38.059069+0000","last_became_peered":"2024-03-19T11:30:38.059069+0000","last_unstale":"2024-03-19T11:30:43.204822+0000","last_undegraded":"2024-03-19T11:30:43.204822+0000","last_fullsized":"2024-03-19T11:30:43.204822+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'92","last_scrub_stamp":"2024-03-19T11:30:41.351520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:41.351520+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:03.452944+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.002969115,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"2.6","version":"0'0","reported_seq":37,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-03-19T11:30:42.207262+0000","last_change":"2024-03-19T11:30:28.039357+0000","last_active":"2024-03-19T11:30:42.207262+0000","last_peered":"2024-03-19T11:30:42.207262+0000","last_clean":"2024-03-19T11:30:42.207262+0000","last_became_active":"2024-03-19T11:30:28.038987+0000","last_became_peered":"2024-03-19T11:30:28.038987+0000","last_unstale":"2024-03-19T11:30:42.207262+0000","last_undegraded":"2024-03-19T11:30:42.207262+0000","last_fullsized":"2024-03-19T11:30:42.207262+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:47.430206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"65'57","reported_seq":268,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-19T11:30:43.792402+0000","last_change":"2024-03-19T11:30:43.792434+0000","last_active":"2024-03-19T11:30:43.792402+0000","last_peered":"2024-03-19T11:30:43.792402+0000","last_clean":"2024-03-19T11:30:43.792402+0000","last_became_active":"2024-03-19T11:30:38.065693+0000","last_became_peered":"2024-03-19T11:30:38.065693+0000","last_unstale":"2024-03-19T11:30:43.792402+0000","last_undegraded":"2024-03-19T11:30:43.792402+0000","last_fullsized":"2024-03-19T11:30:43.792402+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'57","last_scrub_stamp":"2024-03-19T11:30:43.792326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:43.792326+0000","objects_scrubbed":6,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:12.943939+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.44858210799999998,"stat_sum":{"num_bytes":9656036,"num_objects":6,"num_object_clones":5,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"2.5","version":"0'0","reported_seq":115,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-19T11:30:43.254373+0000","last_change":"2024-03-19T11:30:30.057337+0000","last_active":"2024-03-19T11:30:43.254373+0000","last_peered":"2024-03-19T11:30:43.254373+0000","last_clean":"2024-03-19T11:30:43.254373+0000","last_became_active":"2024-03-19T11:30:30.056521+0000","last_became_peered":"2024-03-19T11:30:30.056521+0000","last_unstale":"2024-03-19T11:30:43.254373+0000","last_undegraded":"2024-03-19T11:30:43.254373+0000","last_fullsized":"2024-03-19T11:30:43.254373+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:00.627926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"41'19","reported_seq":225,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-03-19T11:30:41.172489+0000","last_change":"2024-03-19T11:30:38.061249+0000","last_active":"2024-03-19T11:30:41.172489+0000","last_peered":"2024-03-19T11:30:41.172489+0000","last_clean":"2024-03-19T11:30:41.172489+0000","last_became_active":"2024-03-19T11:30:38.058860+0000","last_became_peered":"2024-03-19T11:30:38.058860+0000","last_unstale":"2024-03-19T11:30:41.172489+0000","last_undegraded":"2024-03-19T11:30:41.172489+0000","last_fullsized":"2024-03-19T11:30:41.172489+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'19","last_scrub_stamp":"2024-03-19T11:30:28.180772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:28.180772+0000","objects_scrubbed":1,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001192235,"stat_sum":{"num_bytes":1403368,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4077,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"2.4","version":"0'0","reported_seq":105,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-19T11:30:43.248166+0000","last_change":"2024-03-19T11:29:46.541233+0000","last_active":"2024-03-19T11:30:43.248166+0000","last_peered":"2024-03-19T11:30:43.248166+0000","last_clean":"2024-03-19T11:30:43.248166+0000","last_became_active":"2024-03-19T11:29:46.540939+0000","last_became_peered":"2024-03-19T11:29:46.540939+0000","last_unstale":"2024-03-19T11:30:43.248166+0000","last_undegraded":"2024-03-19T11:30:43.248166+0000","last_fullsized":"2024-03-19T11:30:43.248166+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:54.761098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"28'6","reported_seq":127,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-19T11:30:43.204701+0000","last_change":"2024-03-19T11:30:41.168975+0000","last_active":"2024-03-19T11:30:43.204701+0000","last_peered":"2024-03-19T11:30:43.204701+0000","last_clean":"2024-03-19T11:30:43.204701+0000","last_became_active":"2024-03-19T11:30:38.052772+0000","last_became_peered":"2024-03-19T11:30:38.052772+0000","last_unstale":"2024-03-19T11:30:43.204701+0000","last_undegraded":"2024-03-19T11:30:43.204701+0000","last_fullsized":"2024-03-19T11:30:43.204701+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'6","last_scrub_stamp":"2024-03-19T11:30:30.357009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:30.357009+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0022384689999999999,"stat_sum":{"num_bytes":1176473,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":6560,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"3.3","version":"63'85","reported_seq":266,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-03-19T11:30:41.172075+0000","last_change":"2024-03-19T11:30:30.271391+0000","last_active":"2024-03-19T11:30:41.172075+0000","last_peered":"2024-03-19T11:30:41.172075+0000","last_clean":"2024-03-19T11:30:41.172075+0000","last_became_active":"2024-03-19T11:30:30.058963+0000","last_became_peered":"2024-03-19T11:30:30.058963+0000","last_unstale":"2024-03-19T11:30:41.172075+0000","last_undegraded":"2024-03-19T11:30:41.172075+0000","last_fullsized":"2024-03-19T11:30:41.172075+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:11.045052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44683152,"num_objects":37,"num_object_clones":0,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":61,"num_read_kb":1486,"num_write":93,"num_write_kb":49919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":22537669,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":104,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-19T11:30:43.205039+0000","last_change":"2024-03-19T11:29:46.321255+0000","last_active":"2024-03-19T11:30:43.205039+0000","last_peered":"2024-03-19T11:30:43.205039+0000","last_clean":"2024-03-19T11:30:43.205039+0000","last_became_active":"2024-03-19T11:29:46.320679+0000","last_became_peered":"2024-03-19T11:29:46.320679+0000","last_unstale":"2024-03-19T11:30:43.205039+0000","last_undegraded":"2024-03-19T11:30:43.205039+0000","last_fullsized":"2024-03-19T11:30:43.205039+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:14.793563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"65'104","reported_seq":385,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-19T11:30:43.243940+0000","last_change":"2024-03-19T11:30:40.095983+0000","last_active":"2024-03-19T11:30:43.243940+0000","last_peered":"2024-03-19T11:30:43.243940+0000","last_clean":"2024-03-19T11:30:43.243940+0000","last_became_active":"2024-03-19T11:30:28.037565+0000","last_became_peered":"2024-03-19T11:30:28.037565+0000","last_unstale":"2024-03-19T11:30:43.243940+0000","last_undegraded":"2024-03-19T11:30:43.243940+0000","last_fullsized":"2024-03-19T11:30:43.243940+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'76","last_scrub_stamp":"2024-03-19T11:30:29.353549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:29.353549+0000","objects_scrubbed":11,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:45.232842+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.935303357,"stat_sum":{"num_bytes":26355376,"num_objects":12,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":82,"num_read_kb":96429,"num_write":15,"num_write_kb":3865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":11,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"3.0","version":"65'86","reported_seq":191,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-19T11:30:43.248078+0000","last_change":"2024-03-19T11:29:57.550320+0000","last_active":"2024-03-19T11:30:43.248078+0000","last_peered":"2024-03-19T11:30:43.248078+0000","last_clean":"2024-03-19T11:30:43.248078+0000","last_became_active":"2024-03-19T11:29:57.549928+0000","last_became_peered":"2024-03-19T11:29:57.549928+0000","last_unstale":"2024-03-19T11:30:43.248078+0000","last_undegraded":"2024-03-19T11:30:43.248078+0000","last_fullsized":"2024-03-19T11:30:43.248078+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:01.987302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29319365,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":69,"num_read_kb":2312,"num_write":91,"num_write_kb":38178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":106,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-19T11:30:43.203649+0000","last_change":"2024-03-19T11:29:46.319811+0000","last_active":"2024-03-19T11:30:43.203649+0000","last_peered":"2024-03-19T11:30:43.203649+0000","last_clean":"2024-03-19T11:30:43.203649+0000","last_became_active":"2024-03-19T11:29:46.319334+0000","last_became_peered":"2024-03-19T11:29:46.319334+0000","last_unstale":"2024-03-19T11:30:43.203649+0000","last_undegraded":"2024-03-19T11:30:43.203649+0000","last_fullsized":"2024-03-19T11:30:43.203649+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:04.320463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"61'40","reported_seq":242,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-19T11:30:43.205179+0000","last_change":"2024-03-19T11:30:41.170222+0000","last_active":"2024-03-19T11:30:43.205179+0000","last_peered":"2024-03-19T11:30:43.205179+0000","last_clean":"2024-03-19T11:30:43.205179+0000","last_became_active":"2024-03-19T11:30:28.042372+0000","last_became_peered":"2024-03-19T11:30:28.042372+0000","last_unstale":"2024-03-19T11:30:43.205179+0000","last_undegraded":"2024-03-19T11:30:43.205179+0000","last_fullsized":"2024-03-19T11:30:43.205179+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002884742,"stat_sum":{"num_bytes":2436324,"num_objects":1,"num_object_clones":1,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":13,"num_write_kb":3591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":7273946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"3.1","version":"63'75","reported_seq":235,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-03-19T11:30:41.751854+0000","last_change":"2024-03-19T11:30:27.033184+0000","last_active":"2024-03-19T11:30:41.751854+0000","last_peered":"2024-03-19T11:30:41.751854+0000","last_clean":"2024-03-19T11:30:41.751854+0000","last_became_active":"2024-03-19T11:30:26.927069+0000","last_became_peered":"2024-03-19T11:30:26.927069+0000","last_unstale":"2024-03-19T11:30:41.751854+0000","last_undegraded":"2024-03-19T11:30:41.751854+0000","last_fullsized":"2024-03-19T11:30:41.751854+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:00.211540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38604033,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":62,"num_read_kb":4830,"num_write":79,"num_write_kb":45260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":16477750,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":110,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-19T11:30:43.254468+0000","last_change":"2024-03-19T11:30:27.926306+0000","last_active":"2024-03-19T11:30:43.254468+0000","last_peered":"2024-03-19T11:30:43.254468+0000","last_clean":"2024-03-19T11:30:43.254468+0000","last_became_active":"2024-03-19T11:30:27.925990+0000","last_became_peered":"2024-03-19T11:30:27.925990+0000","last_unstale":"2024-03-19T11:30:43.254468+0000","last_undegraded":"2024-03-19T11:30:43.254468+0000","last_fullsized":"2024-03-19T11:30:43.254468+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:00.759938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"61'63","reported_seq":278,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-19T11:30:43.248626+0000","last_change":"2024-03-19T11:30:38.053087+0000","last_active":"2024-03-19T11:30:43.248626+0000","last_peered":"2024-03-19T11:30:43.248626+0000","last_clean":"2024-03-19T11:30:43.248626+0000","last_became_active":"2024-03-19T11:30:26.925686+0000","last_became_peered":"2024-03-19T11:30:26.925686+0000","last_unstale":"2024-03-19T11:30:43.248626+0000","last_undegraded":"2024-03-19T11:30:43.248626+0000","last_fullsized":"2024-03-19T11:30:43.248626+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"49'53","last_scrub_stamp":"2024-03-19T11:30:28.258196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:28.258196+0000","objects_scrubbed":3,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:42.527502+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.010461453000000001,"stat_sum":{"num_bytes":8046855,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":10,"num_read_kb":10785,"num_write":6,"num_write_kb":1517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"3.2","version":"61'70","reported_seq":182,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-19T11:30:43.229591+0000","last_change":"2024-03-19T11:29:57.549277+0000","last_active":"2024-03-19T11:30:43.229591+0000","last_peered":"2024-03-19T11:30:43.229591+0000","last_clean":"2024-03-19T11:30:43.229591+0000","last_became_active":"2024-03-19T11:29:57.548946+0000","last_became_peered":"2024-03-19T11:29:57.548946+0000","last_unstale":"2024-03-19T11:30:43.229591+0000","last_undegraded":"2024-03-19T11:30:43.229591+0000","last_fullsized":"2024-03-19T11:30:43.229591+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:08.380405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35228564,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":64,"num_read_kb":6282,"num_write":80,"num_write_kb":37058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":104,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-19T11:30:43.205107+0000","last_change":"2024-03-19T11:29:46.322491+0000","last_active":"2024-03-19T11:30:43.205107+0000","last_peered":"2024-03-19T11:30:43.205107+0000","last_clean":"2024-03-19T11:30:43.205107+0000","last_became_active":"2024-03-19T11:29:46.321760+0000","last_became_peered":"2024-03-19T11:29:46.321760+0000","last_unstale":"2024-03-19T11:30:43.205107+0000","last_undegraded":"2024-03-19T11:30:43.205107+0000","last_fullsized":"2024-03-19T11:30:43.205107+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:04.561915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":270,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-19T11:30:43.254477+0000","last_change":"2024-03-19T11:30:28.056134+0000","last_active":"2024-03-19T11:30:43.254477+0000","last_peered":"2024-03-19T11:30:43.254477+0000","last_clean":"2024-03-19T11:30:43.254477+0000","last_became_active":"2024-03-19T11:30:28.043738+0000","last_became_peered":"2024-03-19T11:30:28.043738+0000","last_unstale":"2024-03-19T11:30:43.254477+0000","last_undegraded":"2024-03-19T11:30:43.254477+0000","last_fullsized":"2024-03-19T11:30:43.254477+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:43.166618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:43.166618+0000","last_clean_scrub_stamp":"2024-03-19T11:29:43.166618+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:30:46.409941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"56'66","reported_seq":337,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-03-19T11:30:42.207236+0000","last_change":"2024-03-19T11:30:38.044558+0000","last_active":"2024-03-19T11:30:42.207236+0000","last_peered":"2024-03-19T11:30:42.207236+0000","last_clean":"2024-03-19T11:30:42.207236+0000","last_became_active":"2024-03-19T11:30:28.041852+0000","last_became_peered":"2024-03-19T11:30:28.041852+0000","last_unstale":"2024-03-19T11:30:42.207236+0000","last_undegraded":"2024-03-19T11:30:42.207236+0000","last_fullsized":"2024-03-19T11:30:42.207236+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'64","last_scrub_stamp":"2024-03-19T11:30:28.487136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:28.487136+0000","objects_scrubbed":3,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:37.324348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029151099999999998,"stat_sum":{"num_bytes":4806736,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":31,"num_read_kb":21294,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.8","version":"62'71","reported_seq":271,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-19T11:30:43.205233+0000","last_change":"2024-03-19T11:30:41.170879+0000","last_active":"2024-03-19T11:30:43.205233+0000","last_peered":"2024-03-19T11:30:43.205233+0000","last_clean":"2024-03-19T11:30:43.205233+0000","last_became_active":"2024-03-19T11:30:21.859467+0000","last_became_peered":"2024-03-19T11:30:21.859467+0000","last_unstale":"2024-03-19T11:30:43.205233+0000","last_undegraded":"2024-03-19T11:30:43.205233+0000","last_fullsized":"2024-03-19T11:30:43.205233+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'58","last_scrub_stamp":"2024-03-19T11:30:27.422179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:27.422179+0000","objects_scrubbed":3,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:52.932995+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.004221186,"stat_sum":{"num_bytes":8141594,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":28,"num_read_kb":23809,"num_write":9,"num_write_kb":2047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.9","version":"63'55","reported_seq":295,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-03-19T11:30:42.183408+0000","last_change":"2024-03-19T11:30:38.055222+0000","last_active":"2024-03-19T11:30:42.183408+0000","last_peered":"2024-03-19T11:30:42.183408+0000","last_clean":"2024-03-19T11:30:42.183408+0000","last_became_active":"2024-03-19T11:30:30.050283+0000","last_became_peered":"2024-03-19T11:30:30.050283+0000","last_unstale":"2024-03-19T11:30:42.183408+0000","last_undegraded":"2024-03-19T11:30:42.183408+0000","last_fullsized":"2024-03-19T11:30:42.183408+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'47","last_scrub_stamp":"2024-03-19T11:30:28.676144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:28.676144+0000","objects_scrubbed":2,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:53.516684+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0072186050000000003,"stat_sum":{"num_bytes":5722152,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":9883,"num_write":4,"num_write_kb":1177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4129874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.a","version":"62'77","reported_seq":283,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-03-19T11:30:42.207479+0000","last_change":"2024-03-19T11:30:38.066449+0000","last_active":"2024-03-19T11:30:42.207479+0000","last_peered":"2024-03-19T11:30:42.207479+0000","last_clean":"2024-03-19T11:30:42.207479+0000","last_became_active":"2024-03-19T11:30:38.053085+0000","last_became_peered":"2024-03-19T11:30:38.053085+0000","last_unstale":"2024-03-19T11:30:42.207479+0000","last_undegraded":"2024-03-19T11:30:42.207479+0000","last_fullsized":"2024-03-19T11:30:42.207479+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011735272,"stat_sum":{"num_bytes":10612232,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":43,"num_read_kb":35469,"num_write":23,"num_write_kb":5681,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.b","version":"61'60","reported_seq":282,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-03-19T11:30:41.172404+0000","last_change":"2024-03-19T11:30:40.067487+0000","last_active":"2024-03-19T11:30:41.172404+0000","last_peered":"2024-03-19T11:30:41.172404+0000","last_clean":"2024-03-19T11:30:41.172404+0000","last_became_active":"2024-03-19T11:30:38.055197+0000","last_became_peered":"2024-03-19T11:30:38.055197+0000","last_unstale":"2024-03-19T11:30:41.172404+0000","last_undegraded":"2024-03-19T11:30:41.172404+0000","last_fullsized":"2024-03-19T11:30:41.172404+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.42235675900000003,"stat_sum":{"num_bytes":7803777,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":73,"num_read_kb":73036,"num_write":18,"num_write_kb":5157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.c","version":"56'29","reported_seq":228,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-03-19T11:30:42.183467+0000","last_change":"2024-03-19T11:30:41.792427+0000","last_active":"2024-03-19T11:30:42.183467+0000","last_peered":"2024-03-19T11:30:42.183467+0000","last_clean":"2024-03-19T11:30:42.183467+0000","last_became_active":"2024-03-19T11:30:38.060919+0000","last_became_peered":"2024-03-19T11:30:38.060919+0000","last_unstale":"2024-03-19T11:30:42.183467+0000","last_undegraded":"2024-03-19T11:30:42.183467+0000","last_fullsized":"2024-03-19T11:30:42.183467+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'29","last_scrub_stamp":"2024-03-19T11:30:41.792218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:41.792218+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:56.963011+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.002377026,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.d","version":"65'65","reported_seq":280,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-19T11:30:43.540653+0000","last_change":"2024-03-19T11:30:41.616567+0000","last_active":"2024-03-19T11:30:43.540653+0000","last_peered":"2024-03-19T11:30:43.540653+0000","last_clean":"2024-03-19T11:30:43.540653+0000","last_became_active":"2024-03-19T11:30:38.049196+0000","last_became_peered":"2024-03-19T11:30:38.049196+0000","last_unstale":"2024-03-19T11:30:43.540653+0000","last_undegraded":"2024-03-19T11:30:43.540653+0000","last_fullsized":"2024-03-19T11:30:43.540653+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'61","last_scrub_stamp":"2024-03-19T11:30:41.616358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:41.616358+0000","objects_scrubbed":4,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:52.491131+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.003859543,"stat_sum":{"num_bytes":10067275,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.e","version":"63'63","reported_seq":251,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-03-19T11:30:42.183517+0000","last_change":"2024-03-19T11:30:38.063344+0000","last_active":"2024-03-19T11:30:42.183517+0000","last_peered":"2024-03-19T11:30:42.183517+0000","last_clean":"2024-03-19T11:30:42.183517+0000","last_became_active":"2024-03-19T11:30:38.060487+0000","last_became_peered":"2024-03-19T11:30:38.060487+0000","last_unstale":"2024-03-19T11:30:42.183517+0000","last_undegraded":"2024-03-19T11:30:42.183517+0000","last_fullsized":"2024-03-19T11:30:42.183517+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:29:57.528796+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022237839999999999,"stat_sum":{"num_bytes":7704779,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":22171,"num_write":24,"num_write_kb":6680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]},{"pgid":"4.f","version":"0'0","reported_seq":180,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-19T11:30:43.412116+0000","last_change":"2024-03-19T11:30:43.412186+0000","last_active":"2024-03-19T11:30:43.412116+0000","last_peered":"2024-03-19T11:30:43.412116+0000","last_clean":"2024-03-19T11:30:43.412116+0000","last_became_active":"2024-03-19T11:30:38.050919+0000","last_became_peered":"2024-03-19T11:30:38.050919+0000","last_unstale":"2024-03-19T11:30:43.412116+0000","last_undegraded":"2024-03-19T11:30:43.412116+0000","last_fullsized":"2024-03-19T11:30:43.412116+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:30:43.411893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:43.411893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:59.383144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001232969,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"5","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"}]}],"pool_stats":[{"poolid":4,"num_pg":46,"stat_sum":{"num_bytes":239516356,"num_objects":122,"num_object_clones":75,"num_object_copies":244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":119,"num_whiteouts":7,"num_read":647,"num_read_kb":632750,"num_write":211,"num_write_kb":53480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":23286858,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":78,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":317874176,"data_stored":386675326,"data_compressed":1607220,"data_compressed_allocated":1232896,"data_compressed_original":4194304,"omap_allocated":0,"internal_metadata":0},"log_size":1875,"ondisk_log_size":1875,"up":72,"acting":72,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":147835114,"num_objects":118,"num_object_clones":0,"num_object_copies":236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":118,"num_whiteouts":0,"num_read":256,"num_read_kb":14910,"num_write":343,"num_write_kb":170415,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":39015419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":171216896,"data_stored":186015181,"data_compressed":12840128,"data_compressed_allocated":13774848,"data_compressed_original":28897280,"omap_allocated":0,"internal_metadata":0},"log_size":316,"ondisk_log_size":316,"up":8,"acting":8,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1586784,"data_compressed":19012,"data_compressed_allocated":98304,"data_compressed_original":1548288,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542160,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78252,"kb_used_data":46180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94293588,"statfs":{"total":96636764160,"available":96556634112,"internally_reserved":0,"allocated":47288320,"data_stored":57954079,"data_compressed":154066,"data_compressed_allocated":151552,"data_compressed_original":970752,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542160,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83008,"kb_used_data":55092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94288832,"statfs":{"total":96636764160,"available":96551763968,"internally_reserved":0,"allocated":56414208,"data_stored":69839590,"data_compressed":1600688,"data_compressed_allocated":1667072,"data_compressed_original":3424256,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542160,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":117292,"kb_used_data":90276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94254548,"statfs":{"total":96636764160,"available":96516657152,"internally_reserved":0,"allocated":92442624,"data_stored":105503891,"data_compressed":3530823,"data_compressed_allocated":3629056,"data_compressed_original":8044544,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542159,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81628,"kb_used_data":54612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94290212,"statfs":{"total":96636764160,"available":96553177088,"internally_reserved":0,"allocated":55922688,"data_stored":64187191,"data_compressed":1323292,"data_compressed_allocated":1359872,"data_compressed_original":3211264,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542159,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85720,"kb_used_data":54540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94286120,"statfs":{"total":96636764160,"available":96548986880,"internally_reserved":0,"allocated":55848960,"data_stored":62772261,"data_compressed":1083830,"data_compressed_allocated":1171456,"data_compressed_original":2752512,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574863,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":107168,"kb_used_data":79236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94264672,"statfs":{"total":96636764160,"available":96527024128,"internally_reserved":0,"allocated":81137664,"data_stored":96367246,"data_compressed":1758859,"data_compressed_allocated":1916928,"data_compressed_original":4509696,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574863,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52644,"kb_used_data":21464,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94319196,"statfs":{"total":96636764160,"available":96582856704,"internally_reserved":0,"allocated":21979136,"data_stored":29308517,"data_compressed":297247,"data_compressed_allocated":118784,"data_compressed_original":720896,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574863,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81700,"kb_used_data":54212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94290140,"statfs":{"total":96636764160,"available":96553103360,"internally_reserved":0,"allocated":55513088,"data_stored":59629398,"data_compressed":2934346,"data_compressed_allocated":3149824,"data_compressed_original":6737920,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6064,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6884,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6064,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":44732416,"data_stored":48428539,"data_compressed":3218144,"data_compressed_allocated":3448832,"data_compressed_original":7229440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14438400,"data_stored":15759422,"data_compressed":1002508,"data_compressed_allocated":1081344,"data_compressed_original":2424832,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14438400,"data_stored":15759422,"data_compressed":1002508,"data_compressed_allocated":1081344,"data_compressed_original":2424832,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":37789696,"data_stored":41303297,"data_compressed":2956537,"data_compressed_allocated":3162112,"data_compressed_original":6733824,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":19230720,"data_stored":20747205,"data_compressed":1455770,"data_compressed_allocated":1568768,"data_compressed_original":3133440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16822272,"data_stored":18446027,"data_compressed":1471527,"data_compressed_allocated":1572864,"data_compressed_original":3227648,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":23764992,"data_stored":25571269,"data_compressed":1733134,"data_compressed_allocated":1859584,"data_compressed_original":3723264,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":15249408,"data_stored":16292398,"data_compressed":37577,"data_compressed_allocated":40960,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":41844736,"data_stored":48036968,"data_compressed":81322,"data_compressed_allocated":90112,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40804352,"data_stored":48071101,"data_compressed":320784,"data_compressed_allocated":278528,"data_compressed_original":786432,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53993472,"data_stored":63846221,"data_compressed":574286,"data_compressed_allocated":466944,"data_compressed_original":1310720,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":21307392,"data_stored":28956149,"data_compressed":297247,"data_compressed_allocated":118784,"data_compressed_original":720896,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":38912000,"data_stored":51037054,"data_compressed":129161,"data_compressed_allocated":94208,"data_compressed_original":196608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":56623104,"data_stored":69910830,"data_compressed":18841,"data_compressed_allocated":24576,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49139712,"data_stored":60524605,"data_compressed":148002,"data_compressed_allocated":118784,"data_compressed_original":458752,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T11:30:46.551 INFO:tasks.ceph.ceph_manager.ceph:decrease pool size by 10 2024-03-19T11:30:46.551 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 36 2024-03-19T11:30:46.551 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:30:46.552 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:30:46.569 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:30:46.569 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:30:46.587 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:30:46.587 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:30:46.606 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:46.614+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:46.607 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:30:46.607 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:30:46.627 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:30:46.627 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:30:46.650 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:46.651 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:46.664 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-19T11:30:46.716 INFO:tasks.rados.rados.0.smithi038.stdout:1047: expect (ObjNum 101 snap 10 seq_num 101) 2024-03-19T11:30:46.717 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:46.726+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:46.800 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:46.800 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:46.808 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:46.820+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:46.813 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-19T11:30:46.909 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:46.918+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:46.952 INFO:tasks.rados.rados.0.smithi038.stdout:1041: expect (ObjNum 93 snap 10 seq_num 93) 2024-03-19T11:30:46.966 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:46.966 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:46.980 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-19T11:30:47.023 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:47.034+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:47.113 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:47.124+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:47.136 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:47.136 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:47.139 INFO:tasks.rados.rados.0.smithi038.stdout:1052: expect (ObjNum 21 snap 0 seq_num 21) 2024-03-19T11:30:47.150 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-19T11:30:47.214 INFO:tasks.rados.rados.0.smithi038.stdout:1046: finishing rollback tid 0 to smithi03829856-50 2024-03-19T11:30:47.214 INFO:tasks.rados.rados.0.smithi038.stdout:1044: expect (ObjNum 157 snap 17 seq_num 157) 2024-03-19T11:30:47.214 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:47.222+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:47.244 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:47.254+0000 7fa5e08db640 -1 osd.3 68 handle_osd_map injecting map crc failure 2024-03-19T11:30:47.252 INFO:teuthology.orchestra.run.smithi038.stderr:set pool 4 pg_num to 36 2024-03-19T11:30:47.275 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:47.276 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:47.281 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-03-19T11:30:47.289 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-19T11:30:47.302 INFO:tasks.rados.rados.0.smithi038.stdout:1050: expect (ObjNum 141 snap 15 seq_num 141) 2024-03-19T11:30:47.356 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:47.366+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:47.414 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:47.414 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:47.415 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:47.426+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:47.425 INFO:tasks.rados.rados.0.smithi038.stdout:1046: finishing rollback tid 1 to smithi03829856-50 2024-03-19T11:30:47.425 INFO:tasks.rados.rados.0.smithi038.stdout:1046: finishing rollback tid 2 to smithi03829856-50 2024-03-19T11:30:47.426 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 50 v 81 (ObjNum 114 snap 11 seq_num 114) dirty exists 2024-03-19T11:30:47.426 INFO:tasks.rados.rados.0.smithi038.stdout:1054: finishing write tid 1 to smithi03829856-36 2024-03-19T11:30:47.426 INFO:tasks.rados.rados.0.smithi038.stdout:1040: done (14 left) 2024-03-19T11:30:47.426 INFO:tasks.rados.rados.0.smithi038.stdout:1041: done (13 left) 2024-03-19T11:30:47.426 INFO:tasks.rados.rados.0.smithi038.stdout:1042: done (12 left) 2024-03-19T11:30:47.426 INFO:tasks.rados.rados.0.smithi038.stdout:1043: done (11 left) 2024-03-19T11:30:47.426 INFO:tasks.rados.rados.0.smithi038.stdout:1044: done (10 left) 2024-03-19T11:30:47.426 INFO:tasks.rados.rados.0.smithi038.stdout:1045: done (9 left) 2024-03-19T11:30:47.426 INFO:tasks.rados.rados.0.smithi038.stdout:1046: done (8 left) 2024-03-19T11:30:47.426 INFO:tasks.rados.rados.0.smithi038.stdout:1047: done (7 left) 2024-03-19T11:30:47.426 INFO:tasks.rados.rados.0.smithi038.stdout:1048: done (6 left) 2024-03-19T11:30:47.426 INFO:tasks.rados.rados.0.smithi038.stdout:1049: done (5 left) 2024-03-19T11:30:47.426 INFO:tasks.rados.rados.0.smithi038.stdout:1050: done (4 left) 2024-03-19T11:30:47.426 INFO:tasks.rados.rados.0.smithi038.stdout:1051: done (3 left) 2024-03-19T11:30:47.427 INFO:tasks.rados.rados.0.smithi038.stdout:1052: done (2 left) 2024-03-19T11:30:47.427 INFO:tasks.rados.rados.0.smithi038.stdout:1053: done (1 left) 2024-03-19T11:30:47.427 INFO:tasks.rados.rados.0.smithi038.stdout:1055: read oid 42 snap -1 2024-03-19T11:30:47.427 INFO:tasks.rados.rados.0.smithi038.stdout:1055: expect deleted 2024-03-19T11:30:47.427 INFO:tasks.rados.rados.0.smithi038.stdout:1056: write oid 29 current snap is 17 2024-03-19T11:30:47.427 INFO:tasks.rados.rados.0.smithi038.stdout:1054: finishing write tid 2 to smithi03829856-36 2024-03-19T11:30:47.427 INFO:tasks.rados.rados.0.smithi038.stdout:1054: finishing write tid 3 to smithi03829856-36 2024-03-19T11:30:47.427 INFO:tasks.rados.rados.0.smithi038.stdout:1054: finishing write tid 4 to smithi03829856-36 2024-03-19T11:30:47.427 INFO:tasks.rados.rados.0.smithi038.stdout:1054: oid 36 updating version 0 to 69 2024-03-19T11:30:47.427 INFO:tasks.rados.rados.0.smithi038.stdout:1054: oid 36 version 69 is already newer than 68 2024-03-19T11:30:47.427 INFO:tasks.rados.rados.0.smithi038.stdout:1054: oid 36 updating version 69 to 70 2024-03-19T11:30:47.427 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 36 v 70 (ObjNum 164 snap 17 seq_num 164) dirty exists 2024-03-19T11:30:47.427 INFO:tasks.rados.rados.0.smithi038.stdout:1054: left oid 36 (ObjNum 164 snap 17 seq_num 164) 2024-03-19T11:30:47.427 INFO:tasks.rados.rados.0.smithi038.stdout:1056: seq_num 165 ranges {773155=459498,1722470=430045,2648676=124479} 2024-03-19T11:30:47.430 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-19T11:30:47.431 INFO:tasks.rados.rados.0.smithi038.stdout:1056: writing smithi03829856-29 from 773155 to 1232653 tid 1 2024-03-19T11:30:47.437 INFO:tasks.rados.rados.0.smithi038.stdout:1056: writing smithi03829856-29 from 1722470 to 2152515 tid 2 2024-03-19T11:30:47.441 INFO:tasks.rados.rados.0.smithi038.stdout:1056: writing smithi03829856-29 from 2648676 to 2773155 tid 3 2024-03-19T11:30:47.441 INFO:tasks.rados.rados.0.smithi038.stdout:1054: done (2 left) 2024-03-19T11:30:47.441 INFO:tasks.rados.rados.0.smithi038.stdout:1057: read oid 37 snap 15 2024-03-19T11:30:47.441 INFO:tasks.rados.rados.0.smithi038.stdout:1057: expect (ObjNum 36 snap 0 seq_num 36) 2024-03-19T11:30:47.441 INFO:tasks.rados.rados.0.smithi038.stdout:1058: set_chunk oid 18 target oid 2024-03-19T11:30:47.471 INFO:tasks.rados.rados.0.smithi038.stdout:1058: set_chunk oid 18 offset: 2180608 length: 1040896 target oid 6bf86c0b7e81a69df2aee31dfaa8e7eddd83670b1a601b72b96d182a79ba179d offset: 0 snap -1 2024-03-19T11:30:47.471 INFO:tasks.rados.rados.0.smithi038.stdout:1059: set_chunk oid 39 target oid 2024-03-19T11:30:47.507 INFO:tasks.rados.rados.0.smithi038.stdout:1059: set_chunk oid 39 offset: 90624 length: 2113536 target oid b1c10db0f5ad7971942371f522bcd730a20a1af48652460ccc4e43512011d619 offset: 0 snap -1 2024-03-19T11:30:47.508 INFO:tasks.rados.rados.0.smithi038.stdout:1060: read oid 10 snap -1 2024-03-19T11:30:47.508 INFO:tasks.rados.rados.0.smithi038.stdout:1060: expect (ObjNum 144 snap 15 seq_num 144) 2024-03-19T11:30:47.508 INFO:tasks.rados.rados.0.smithi038.stdout:1061: set_chunk oid 4 target oid 2024-03-19T11:30:47.511 INFO:tasks.rados.rados.0.smithi038.stdout:1061: set_chunk oid 4 offset: 1454592 length: 202240 target oid 2246b52a34a02988d1dcbac38a1942a178b305c494b125096d447d9201943f9c offset: 0 snap -1 2024-03-19T11:30:47.511 INFO:tasks.rados.rados.0.smithi038.stdout:1062: read oid 34 snap 4 2024-03-19T11:30:47.511 INFO:tasks.rados.rados.0.smithi038.stdout:1062: expect (ObjNum 63 snap 3 seq_num 63) 2024-03-19T11:30:47.511 INFO:tasks.rados.rados.0.smithi038.stdout:1063: set_chunk oid 38 target oid 2024-03-19T11:30:47.519 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:47.526+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:47.519 INFO:tasks.rados.rados.0.smithi038.stdout:1063: set_chunk oid 38 offset: 1236480 length: 391168 target oid a8ae829fcaab0d68e85c0cd7988d5bba69fe6fcb71cb62bf24d2ed297d27e8e8 offset: 0 snap -1 2024-03-19T11:30:47.520 INFO:tasks.rados.rados.0.smithi038.stdout:1064: write oid 17 current snap is 17 2024-03-19T11:30:47.520 INFO:tasks.rados.rados.0.smithi038.stdout:1064: seq_num 166 ranges {789962=534747,1780952=690056} 2024-03-19T11:30:47.531 INFO:tasks.rados.rados.0.smithi038.stdout:1064: writing smithi03829856-17 from 789962 to 1324709 tid 1 2024-03-19T11:30:47.540 INFO:tasks.rados.rados.0.smithi038.stdout:1064: writing smithi03829856-17 from 1780952 to 2471008 tid 2 2024-03-19T11:30:47.540 INFO:tasks.rados.rados.0.smithi038.stdout:1065: write oid 31 current snap is 17 2024-03-19T11:30:47.540 INFO:tasks.rados.rados.0.smithi038.stdout:1065: seq_num 167 ranges {406769=609996,1439434=433714,2303522=503247} 2024-03-19T11:30:47.549 INFO:tasks.rados.rados.0.smithi038.stdout:1065: writing smithi03829856-31 from 406769 to 1016765 tid 1 2024-03-19T11:30:47.552 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:47.552 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:47.553 INFO:tasks.rados.rados.0.smithi038.stdout:1065: writing smithi03829856-31 from 1439434 to 1873148 tid 2 2024-03-19T11:30:47.561 INFO:tasks.rados.rados.0.smithi038.stdout:1065: writing smithi03829856-31 from 2303522 to 2806769 tid 3 2024-03-19T11:30:47.561 INFO:tasks.rados.rados.0.smithi038.stdout:1066: read oid 26 snap -1 2024-03-19T11:30:47.561 INFO:tasks.rados.rados.0.smithi038.stdout:1066: expect (ObjNum 123 snap 15 seq_num 123) 2024-03-19T11:30:47.561 INFO:tasks.rados.rados.0.smithi038.stdout:1067: read oid 35 snap -1 2024-03-19T11:30:47.562 INFO:tasks.rados.rados.0.smithi038.stdout:1067: expect (ObjNum 158 snap 17 seq_num 158) 2024-03-19T11:30:47.562 INFO:tasks.rados.rados.0.smithi038.stdout:1068: read oid 21 snap -1 2024-03-19T11:30:47.562 INFO:tasks.rados.rados.0.smithi038.stdout:1068: expect (ObjNum 129 snap 15 seq_num 129) 2024-03-19T11:30:47.562 INFO:tasks.rados.rados.0.smithi038.stdout:1069: snap_remove snap 10 2024-03-19T11:30:47.567 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-19T11:30:47.617 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:47.624+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:47.718 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:47.724+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:47.736 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:47.736 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:47.749 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-19T11:30:47.819 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:47.826+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:47.874 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:47.875 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:47.886 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-19T11:30:47.919 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:47.928+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:48.020 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:48.028+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:48.022 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-03-19T11:30:48.026 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:48.032+0000 7f143d8b8640 -1 osd.5 69 handle_osd_map injecting map crc failure 2024-03-19T11:30:48.032 INFO:tasks.rados.rados.0.smithi038.stdout:1069: done (14 left) 2024-03-19T11:30:48.032 INFO:tasks.rados.rados.0.smithi038.stdout:1070: read oid 33 snap 16 2024-03-19T11:30:48.032 INFO:tasks.rados.rados.0.smithi038.stdout:1070: expect (ObjNum 147 snap 15 seq_num 147) 2024-03-19T11:30:48.032 INFO:tasks.rados.rados.0.smithi038.stdout:1071: write oid 22 current snap is 17 2024-03-19T11:30:48.032 INFO:tasks.rados.rados.0.smithi038.stdout:1071: seq_num 168 ranges {423576=401598,1497916=693725} 2024-03-19T11:30:48.040 INFO:tasks.rados.rados.0.smithi038.stdout:1071: writing smithi03829856-22 from 423576 to 825174 tid 1 2024-03-19T11:30:48.052 INFO:tasks.rados.rados.0.smithi038.stdout:1071: writing smithi03829856-22 from 1497916 to 2191641 tid 2 2024-03-19T11:30:48.052 INFO:tasks.rados.rados.0.smithi038.stdout: waiting on 16 2024-03-19T11:30:48.055 INFO:tasks.rados.rados.0.smithi038.stdout:1056: finishing write tid 1 to smithi03829856-29 2024-03-19T11:30:48.055 INFO:tasks.rados.rados.0.smithi038.stdout:1056: finishing write tid 2 to smithi03829856-29 2024-03-19T11:30:48.055 INFO:tasks.rados.rados.0.smithi038.stdout:1056: finishing write tid 3 to smithi03829856-29 2024-03-19T11:30:48.055 INFO:tasks.rados.rados.0.smithi038.stdout:1056: finishing write tid 4 to smithi03829856-29 2024-03-19T11:30:48.055 INFO:tasks.rados.rados.0.smithi038.stdout:1057: expect (ObjNum 36 snap 0 seq_num 36) 2024-03-19T11:30:48.064 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:48.064 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:48.077 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-19T11:30:48.122 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:48.128+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:48.183 INFO:tasks.rados.rados.0.smithi038.stdout:1056: finishing write tid 5 to smithi03829856-29 2024-03-19T11:30:48.184 INFO:tasks.rados.rados.0.smithi038.stdout:1056: oid 29 updating version 0 to 69 2024-03-19T11:30:48.184 INFO:tasks.rados.rados.0.smithi038.stdout:1056: oid 29 updating version 69 to 70 2024-03-19T11:30:48.184 INFO:tasks.rados.rados.0.smithi038.stdout:1056: oid 29 updating version 70 to 71 2024-03-19T11:30:48.184 INFO:tasks.rados.rados.0.smithi038.stdout:1056: oid 29 updating version 71 to 72 2024-03-19T11:30:48.184 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 29 v 72 (ObjNum 165 snap 17 seq_num 165) dirty exists 2024-03-19T11:30:48.184 INFO:tasks.rados.rados.0.smithi038.stdout:1056: left oid 29 (ObjNum 165 snap 17 seq_num 165) 2024-03-19T11:30:48.184 INFO:tasks.rados.rados.0.smithi038.stdout:1058: finishing set_chunk to oid 18 2024-03-19T11:30:48.184 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 18 v 75 (ObjNum 106 snap 11 seq_num 106) dirty exists 2024-03-19T11:30:48.184 INFO:tasks.rados.rados.0.smithi038.stdout:1060: expect (ObjNum 144 snap 15 seq_num 144) 2024-03-19T11:30:48.223 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:48.230+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:48.234 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:48.234 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:48.248 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-19T11:30:48.313 INFO:tasks.rados.rados.0.smithi038.stdout:1059: finishing set_chunk to oid 39 2024-03-19T11:30:48.313 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 39 v 69 (ObjNum 163 snap 17 seq_num 163) dirty exists 2024-03-19T11:30:48.313 INFO:tasks.rados.rados.0.smithi038.stdout:1063: finishing set_chunk to oid 38 2024-03-19T11:30:48.313 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 38 set_chunk returned error code -95 offset: 1236480 length: 391168 2024-03-19T11:30:48.313 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 38 v 0 (ObjNum 125 snap 15 seq_num 125) dirty exists 2024-03-19T11:30:48.313 INFO:tasks.rados.rados.0.smithi038.stdout:1061: finishing set_chunk to oid 4 2024-03-19T11:30:48.313 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 4 v 104 (ObjNum 151 snap 17 seq_num 151) dirty exists 2024-03-19T11:30:48.313 INFO:tasks.rados.rados.0.smithi038.stdout:1062: expect (ObjNum 63 snap 3 seq_num 63) 2024-03-19T11:30:48.323 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:48.334+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:48.404 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:48.404 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:48.417 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-19T11:30:48.425 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:48.432+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:48.454 INFO:tasks.rados.rados.0.smithi038.stdout:1064: finishing write tid 1 to smithi03829856-17 2024-03-19T11:30:48.454 INFO:tasks.rados.rados.0.smithi038.stdout:1064: finishing write tid 2 to smithi03829856-17 2024-03-19T11:30:48.455 INFO:tasks.rados.rados.0.smithi038.stdout:1064: finishing write tid 3 to smithi03829856-17 2024-03-19T11:30:48.455 INFO:tasks.rados.rados.0.smithi038.stdout:1064: finishing write tid 4 to smithi03829856-17 2024-03-19T11:30:48.455 INFO:tasks.rados.rados.0.smithi038.stdout:1064: oid 17 updating version 0 to 114 2024-03-19T11:30:48.455 INFO:tasks.rados.rados.0.smithi038.stdout:1064: oid 17 updating version 114 to 115 2024-03-19T11:30:48.455 INFO:tasks.rados.rados.0.smithi038.stdout:1064: oid 17 updating version 115 to 116 2024-03-19T11:30:48.455 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 17 v 116 (ObjNum 166 snap 17 seq_num 166) dirty exists 2024-03-19T11:30:48.455 INFO:tasks.rados.rados.0.smithi038.stdout:1064: left oid 17 (ObjNum 166 snap 17 seq_num 166) 2024-03-19T11:30:48.455 INFO:tasks.rados.rados.0.smithi038.stdout:1066: expect (ObjNum 123 snap 15 seq_num 123) 2024-03-19T11:30:48.526 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:48.532+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:48.550 INFO:tasks.rados.rados.0.smithi038.stdout:1065: finishing write tid 1 to smithi03829856-31 2024-03-19T11:30:48.551 INFO:tasks.rados.rados.0.smithi038.stdout:1065: finishing write tid 2 to smithi03829856-31 2024-03-19T11:30:48.551 INFO:tasks.rados.rados.0.smithi038.stdout:1065: finishing write tid 3 to smithi03829856-31 2024-03-19T11:30:48.551 INFO:tasks.rados.rados.0.smithi038.stdout:1065: finishing write tid 4 to smithi03829856-31 2024-03-19T11:30:48.551 INFO:tasks.rados.rados.0.smithi038.stdout:1065: finishing write tid 5 to smithi03829856-31 2024-03-19T11:30:48.551 INFO:tasks.rados.rados.0.smithi038.stdout:1065: oid 31 updating version 0 to 111 2024-03-19T11:30:48.551 INFO:tasks.rados.rados.0.smithi038.stdout:1065: oid 31 version 111 is already newer than 110 2024-03-19T11:30:48.551 INFO:tasks.rados.rados.0.smithi038.stdout:1065: oid 31 version 111 is already newer than 109 2024-03-19T11:30:48.551 INFO:tasks.rados.rados.0.smithi038.stdout:1065: oid 31 version 111 is already newer than 108 2024-03-19T11:30:48.551 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 31 v 111 (ObjNum 167 snap 17 seq_num 167) dirty exists 2024-03-19T11:30:48.551 INFO:tasks.rados.rados.0.smithi038.stdout:1065: left oid 31 (ObjNum 167 snap 17 seq_num 167) 2024-03-19T11:30:48.551 INFO:tasks.rados.rados.0.smithi038.stdout:1067: expect (ObjNum 158 snap 17 seq_num 158) 2024-03-19T11:30:48.580 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:48.580 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:48.594 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-19T11:30:48.626 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:48.634+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:48.645 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-19T11:30:48.706 INFO:tasks.rados.rados.0.smithi038.stdout:1068: expect (ObjNum 129 snap 15 seq_num 129) 2024-03-19T11:30:48.725 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:48.725 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:48.728 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:48.736+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:48.739 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-19T11:30:48.787 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:30:48.788 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:30:48.804 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:30:48.805 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:30:48.808 INFO:tasks.rados.rados.0.smithi038.stdout:1070: expect (ObjNum 147 snap 15 seq_num 147) 2024-03-19T11:30:48.822 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:30:48.822 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:30:48.829 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:48.836+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:48.838 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:30:48.839 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:30:48.856 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:30:48.857 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:30:48.875 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:30:48.876 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:30:48.896 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:30:48.896 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:30:48.906 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:48.906 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:48.906 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:30:48.906 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:30:48.906 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:48.917 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:30:48.917 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:30:48.920 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-19T11:30:48.927 INFO:tasks.rados.rados.0.smithi038.stdout:1071: finishing write tid 1 to smithi03829856-22 2024-03-19T11:30:48.927 INFO:tasks.rados.rados.0.smithi038.stdout:1071: finishing write tid 2 to smithi03829856-22 2024-03-19T11:30:48.928 INFO:tasks.rados.rados.0.smithi038.stdout:1071: finishing write tid 3 to smithi03829856-22 2024-03-19T11:30:48.928 INFO:tasks.rados.rados.0.smithi038.stdout:1071: finishing write tid 4 to smithi03829856-22 2024-03-19T11:30:48.928 INFO:tasks.rados.rados.0.smithi038.stdout:1071: oid 22 updating version 0 to 9 2024-03-19T11:30:48.928 INFO:tasks.rados.rados.0.smithi038.stdout:1071: oid 22 updating version 9 to 10 2024-03-19T11:30:48.928 INFO:tasks.rados.rados.0.smithi038.stdout:1071: oid 22 version 10 is already newer than 8 2024-03-19T11:30:48.928 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 22 v 10 (ObjNum 168 snap 17 seq_num 168) dirty exists 2024-03-19T11:30:48.928 INFO:tasks.rados.rados.0.smithi038.stdout:1071: left oid 22 (ObjNum 168 snap 17 seq_num 168) 2024-03-19T11:30:48.928 INFO:tasks.rados.rados.0.smithi038.stdout:1055: done (15 left) 2024-03-19T11:30:48.928 INFO:tasks.rados.rados.0.smithi038.stdout:1056: done (14 left) 2024-03-19T11:30:48.928 INFO:tasks.rados.rados.0.smithi038.stdout:1057: done (13 left) 2024-03-19T11:30:48.928 INFO:tasks.rados.rados.0.smithi038.stdout:1058: done (12 left) 2024-03-19T11:30:48.928 INFO:tasks.rados.rados.0.smithi038.stdout:1059: done (11 left) 2024-03-19T11:30:48.928 INFO:tasks.rados.rados.0.smithi038.stdout:1060: done (10 left) 2024-03-19T11:30:48.928 INFO:tasks.rados.rados.0.smithi038.stdout:1061: done (9 left) 2024-03-19T11:30:48.928 INFO:tasks.rados.rados.0.smithi038.stdout:1062: done (8 left) 2024-03-19T11:30:48.929 INFO:tasks.rados.rados.0.smithi038.stdout:1063: done (7 left) 2024-03-19T11:30:48.929 INFO:tasks.rados.rados.0.smithi038.stdout:1064: done (6 left) 2024-03-19T11:30:48.929 INFO:tasks.rados.rados.0.smithi038.stdout:1065: done (5 left) 2024-03-19T11:30:48.929 INFO:tasks.rados.rados.0.smithi038.stdout:1066: done (4 left) 2024-03-19T11:30:48.929 INFO:tasks.rados.rados.0.smithi038.stdout:1067: done (3 left) 2024-03-19T11:30:48.929 INFO:tasks.rados.rados.0.smithi038.stdout:1068: done (2 left) 2024-03-19T11:30:48.929 INFO:tasks.rados.rados.0.smithi038.stdout:1070: done (1 left) 2024-03-19T11:30:48.929 INFO:tasks.rados.rados.0.smithi038.stdout:1071: done (0 left) 2024-03-19T11:30:48.929 INFO:tasks.rados.rados.0.smithi038.stdout:1072: tier_evict oid 27 2024-03-19T11:30:48.929 INFO:tasks.rados.rados.0.smithi038.stdout:1072: tier_evict oid 27 snap -1 2024-03-19T11:30:48.929 INFO:tasks.rados.rados.0.smithi038.stdout:1073: delete oid 3 current snap is 17 2024-03-19T11:30:48.929 INFO:tasks.rados.rados.0.smithi038.stdout:1072: got (22) Invalid argument 2024-03-19T11:30:48.929 INFO:tasks.rados.rados.0.smithi038.stdout:1072: done (1 left) 2024-03-19T11:30:48.929 INFO:tasks.rados.rados.0.smithi038.stdout:1073: done (0 left) 2024-03-19T11:30:48.929 INFO:tasks.rados.rados.0.smithi038.stdout:1074: set_chunk oid 15 target oid 2024-03-19T11:30:48.931 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:48.938+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:48.933 INFO:tasks.rados.rados.0.smithi038.stdout:1074: set_chunk oid 15 offset: 1031168 length: 337920 target oid 16093ca7663312b861f18d6d69997a611e9de39c17bba102e32a69982333a558 offset: 0 snap -1 2024-03-19T11:30:48.933 INFO:tasks.rados.rados.0.smithi038.stdout:1075: read oid 20 snap -1 2024-03-19T11:30:48.933 INFO:tasks.rados.rados.0.smithi038.stdout:1075: expect (ObjNum 19 snap 0 seq_num 19) 2024-03-19T11:30:48.933 INFO:tasks.rados.rados.0.smithi038.stdout:1076: write (excl) oid 41 current snap is 17 2024-03-19T11:30:48.933 INFO:tasks.rados.rados.0.smithi038.stdout:1076: seq_num 170 ranges {457190=552096,1614880=581041,2785791=71399} 2024-03-19T11:30:48.938 INFO:tasks.rados.rados.0.smithi038.stdout:1076: writing smithi03829856-41 from 457190 to 1009286 tid 1 2024-03-19T11:30:48.945 INFO:tasks.rados.rados.0.smithi038.stdout:1076: writing smithi03829856-41 from 1614880 to 2195921 tid 2 2024-03-19T11:30:48.950 INFO:tasks.rados.rados.0.smithi038.stdout:1076: writing smithi03829856-41 from 2785791 to 2857190 tid 3 2024-03-19T11:30:48.950 INFO:tasks.rados.rados.0.smithi038.stdout:1077: read oid 28 snap 9 2024-03-19T11:30:48.951 INFO:tasks.rados.rados.0.smithi038.stdout:1077: expect (ObjNum 87 snap 8 seq_num 87) 2024-03-19T11:30:48.951 INFO:tasks.rados.rados.0.smithi038.stdout:1078: tier_flush oid 10 2024-03-19T11:30:48.951 INFO:tasks.rados.rados.0.smithi038.stdout:1078: tier_flush oid 10 snap -1 2024-03-19T11:30:48.951 INFO:tasks.rados.rados.0.smithi038.stdout:1074: finishing set_chunk to oid 15 2024-03-19T11:30:48.951 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 15 set_chunk returned error code -95 offset: 1031168 length: 337920 2024-03-19T11:30:48.952 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-03-19T11:30:48.952 INFO:tasks.rados.rados.0.smithi038.stdout:1074: done (4 left) 2024-03-19T11:30:48.952 INFO:tasks.rados.rados.0.smithi038.stdout:1079: rollback oid 14 current snap is 17 2024-03-19T11:30:48.952 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 14 to 9 2024-03-19T11:30:48.952 INFO:tasks.rados.rados.0.smithi038.stdout:1076: finishing write tid 1 to smithi03829856-41 2024-03-19T11:30:48.953 INFO:tasks.rados.rados.0.smithi038.stdout:1075: expect (ObjNum 19 snap 0 seq_num 19) 2024-03-19T11:30:49.026 INFO:tasks.rados.rados.0.smithi038.stdout:1076: finishing write tid 2 to smithi03829856-41 2024-03-19T11:30:49.026 INFO:tasks.rados.rados.0.smithi038.stdout:1078: got (16) Device or resource busy 2024-03-19T11:30:49.026 INFO:tasks.rados.rados.0.smithi038.stdout:check_oldest_snap_flushed oid 10 v 73 (ObjNum 144 snap 15 seq_num 144) unflushed 15 2024-03-19T11:30:49.026 INFO:tasks.rados.rados.0.smithi038.stdout:1076: finishing write tid 3 to smithi03829856-41 2024-03-19T11:30:49.026 INFO:tasks.rados.rados.0.smithi038.stdout:1076: finishing write tid 4 to smithi03829856-41 2024-03-19T11:30:49.027 INFO:tasks.rados.rados.0.smithi038.stdout:1076: finishing write tid 5 to smithi03829856-41 2024-03-19T11:30:49.027 INFO:tasks.rados.rados.0.smithi038.stdout:1076: oid 41 updating version 0 to 116 2024-03-19T11:30:49.027 INFO:tasks.rados.rados.0.smithi038.stdout:1076: oid 41 updating version 116 to 119 2024-03-19T11:30:49.027 INFO:tasks.rados.rados.0.smithi038.stdout:1076: oid 41 version 119 is already newer than 117 2024-03-19T11:30:49.028 INFO:tasks.rados.rados.0.smithi038.stdout:1076: oid 41 version 119 is already newer than 118 2024-03-19T11:30:49.028 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 41 v 119 (ObjNum 170 snap 17 seq_num 170) dirty exists 2024-03-19T11:30:49.028 INFO:tasks.rados.rados.0.smithi038.stdout:1076: left oid 41 (ObjNum 170 snap 17 seq_num 170) 2024-03-19T11:30:49.028 INFO:tasks.rados.rados.0.smithi038.stdout:1077: expect (ObjNum 87 snap 8 seq_num 87) 2024-03-19T11:30:49.033 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:49.040+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:49.087 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:49.088 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:30:49.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:30:49.088 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:30:49.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.088 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:9 4.d 4:b4ef97e0:::smithi03829856-3:head [write 433614~550498,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-03-19T11:30:49.088 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:04.706000+0000", 2024-03-19T11:30:49.088 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 44.392236779999998, 2024-03-19T11:30:49.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013227001, 2024-03-19T11:30:49.088 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:49.088 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:49.088 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:49.089 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:49.089 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:49.089 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 9 2024-03-19T11:30:49.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.089 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:49.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.089 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:49.089 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.706000+0000", 2024-03-19T11:30:49.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.089 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:49.089 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.706000+0000", 2024-03-19T11:30:49.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.090 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:49.090 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.706002+0000", 2024-03-19T11:30:49.090 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5569999999999998e-06 2024-03-19T11:30:49.090 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:49.090 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.707261+0000", 2024-03-19T11:30:49.090 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001258817 2024-03-19T11:30:49.090 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:49.090 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.707288+0000", 2024-03-19T11:30:49.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7093e-05 2024-03-19T11:30:49.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.091 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:49.091 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.707305+0000", 2024-03-19T11:30:49.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6330000000000001e-05 2024-03-19T11:30:49.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.091 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.091 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.707610+0000", 2024-03-19T11:30:49.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00030568799999999998 2024-03-19T11:30:49.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.091 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:49.092 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.707984+0000", 2024-03-19T11:30:49.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00037399299999999998 2024-03-19T11:30:49.092 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.092 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2", 2024-03-19T11:30:49.092 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.708409+0000", 2024-03-19T11:30:49.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00042503099999999998 2024-03-19T11:30:49.092 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.092 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:49.092 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.712919+0000", 2024-03-19T11:30:49.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0045099010000000002 2024-03-19T11:30:49.092 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.092 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:49.093 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.719037+0000", 2024-03-19T11:30:49.093 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061173770000000002 2024-03-19T11:30:49.093 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.093 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:49.093 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.719116+0000", 2024-03-19T11:30:49.093 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9935e-05 2024-03-19T11:30:49.093 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.093 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:49.093 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.719227+0000", 2024-03-19T11:30:49.093 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011027900000000001 2024-03-19T11:30:49.093 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.093 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:49.094 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.094 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.094 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:848 4.1 4:866f3dcc:::smithi03829856-31:head [read 0~2941192,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:49.094 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.757468+0000", 2024-03-19T11:30:49.094 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 40.340768787999998, 2024-03-19T11:30:49.094 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015020110999999999, 2024-03-19T11:30:49.094 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:49.094 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:49.094 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:49.094 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:49.094 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:49.094 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 848 2024-03-19T11:30:49.094 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.095 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:49.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.095 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:49.095 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757468+0000", 2024-03-19T11:30:49.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.095 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:49.095 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757468+0000", 2024-03-19T11:30:49.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.095 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:49.095 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757471+0000", 2024-03-19T11:30:49.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.67e-06 2024-03-19T11:30:49.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.096 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:49.096 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757488+0000", 2024-03-19T11:30:49.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6223e-05 2024-03-19T11:30:49.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.096 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:49.096 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757521+0000", 2024-03-19T11:30:49.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.358e-05 2024-03-19T11:30:49.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.096 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:49.096 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757543+0000", 2024-03-19T11:30:49.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1330000000000001e-05 2024-03-19T11:30:49.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757745+0000", 2024-03-19T11:30:49.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020259799999999999 2024-03-19T11:30:49.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op must wait for map", 2024-03-19T11:30:49.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757760+0000", 2024-03-19T11:30:49.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4647000000000001e-05 2024-03-19T11:30:49.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763972+0000", 2024-03-19T11:30:49.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062126480000000003 2024-03-19T11:30:49.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:49.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.764547+0000", 2024-03-19T11:30:49.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057440200000000001 2024-03-19T11:30:49.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:49.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.772488+0000", 2024-03-19T11:30:49.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079410130000000002 2024-03-19T11:30:49.098 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.098 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:49.098 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.099 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:849 4.1 4:866f3dcc:::smithi03829856-31:head [sparse-read 0~1359577] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:49.099 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.757583+0000", 2024-03-19T11:30:49.099 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 40.340653095999997, 2024-03-19T11:30:49.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016241202, 2024-03-19T11:30:49.099 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:49.099 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:49.099 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:49.099 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:49.099 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:49.099 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 849 2024-03-19T11:30:49.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.099 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:49.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.100 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:49.100 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757583+0000", 2024-03-19T11:30:49.100 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.100 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:49.100 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757583+0000", 2024-03-19T11:30:49.100 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.100 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:49.100 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757584+0000", 2024-03-19T11:30:49.100 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.077e-06 2024-03-19T11:30:49.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.101 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:49.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757593+0000", 2024-03-19T11:30:49.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3059999999999998e-06 2024-03-19T11:30:49.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.101 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:49.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757608+0000", 2024-03-19T11:30:49.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5526999999999999e-05 2024-03-19T11:30:49.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.101 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:49.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757618+0000", 2024-03-19T11:30:49.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5999999999999996e-06 2024-03-19T11:30:49.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757886+0000", 2024-03-19T11:30:49.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026839099999999999 2024-03-19T11:30:49.102 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:49.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757899+0000", 2024-03-19T11:30:49.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3004e-05 2024-03-19T11:30:49.102 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.772651+0000", 2024-03-19T11:30:49.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014751702 2024-03-19T11:30:49.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.103 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:49.103 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.772924+0000", 2024-03-19T11:30:49.103 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027314199999999998 2024-03-19T11:30:49.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.103 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:49.103 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.773825+0000", 2024-03-19T11:30:49.103 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00090045299999999995 2024-03-19T11:30:49.103 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.103 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:49.103 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.104 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:850 4.1 4:866f3dcc:::smithi03829856-31:head [read 0~2941192,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:49.104 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.757636+0000", 2024-03-19T11:30:49.104 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 40.340600426000002, 2024-03-19T11:30:49.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018834801000000002, 2024-03-19T11:30:49.104 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:49.104 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:49.104 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:49.104 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:49.104 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:49.104 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 850 2024-03-19T11:30:49.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.104 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:49.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.104 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:49.105 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757636+0000", 2024-03-19T11:30:49.105 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.105 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:49.105 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757636+0000", 2024-03-19T11:30:49.105 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.105 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:49.105 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757637+0000", 2024-03-19T11:30:49.105 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5999999999999991e-07 2024-03-19T11:30:49.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.105 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:49.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757643+0000", 2024-03-19T11:30:49.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3199999999999996e-06 2024-03-19T11:30:49.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:49.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757670+0000", 2024-03-19T11:30:49.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6682999999999998e-05 2024-03-19T11:30:49.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:49.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757679+0000", 2024-03-19T11:30:49.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.986e-06 2024-03-19T11:30:49.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.107 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.107 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757961+0000", 2024-03-19T11:30:49.107 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028230799999999999 2024-03-19T11:30:49.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.107 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:49.107 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.757971+0000", 2024-03-19T11:30:49.107 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.38e-06 2024-03-19T11:30:49.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.107 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.107 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.774079+0000", 2024-03-19T11:30:49.107 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016108573000000001 2024-03-19T11:30:49.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.108 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:49.108 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.774306+0000", 2024-03-19T11:30:49.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022637499999999999 2024-03-19T11:30:49.108 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.108 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:49.108 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776471+0000", 2024-03-19T11:30:49.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021652160000000002 2024-03-19T11:30:49.108 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.108 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:49.108 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.108 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.108 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:851 4.9 4:953e720b:::smithi03829856-45:head [write 591613~454162,stat] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:49.109 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.763514+0000", 2024-03-19T11:30:49.109 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 40.334722235000001, 2024-03-19T11:30:49.109 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027825888999999999, 2024-03-19T11:30:49.109 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:49.109 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:49.109 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:49.109 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:49.109 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:49.109 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 851 2024-03-19T11:30:49.109 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.109 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:49.109 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.109 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:49.109 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763514+0000", 2024-03-19T11:30:49.109 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.110 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.110 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:49.110 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763514+0000", 2024-03-19T11:30:49.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.110 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.110 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:49.110 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763517+0000", 2024-03-19T11:30:49.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8499999999999998e-06 2024-03-19T11:30:49.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.110 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.110 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:49.110 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.764714+0000", 2024-03-19T11:30:49.111 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011971270000000001 2024-03-19T11:30:49.111 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.111 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.111 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:49.111 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.764745+0000", 2024-03-19T11:30:49.111 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0546999999999997e-05 2024-03-19T11:30:49.111 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.111 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.111 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:49.111 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.764767+0000", 2024-03-19T11:30:49.111 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2286999999999999e-05 2024-03-19T11:30:49.111 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.111 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.111 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.112 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.773924+0000", 2024-03-19T11:30:49.112 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0091567360000000004 2024-03-19T11:30:49.112 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.112 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.112 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:49.112 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.774217+0000", 2024-03-19T11:30:49.112 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029350099999999999 2024-03-19T11:30:49.112 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.112 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.112 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:49.112 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.774770+0000", 2024-03-19T11:30:49.112 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00055288600000000005 2024-03-19T11:30:49.112 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.112 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.112 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:49.113 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.786413+0000", 2024-03-19T11:30:49.113 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011643229999999999 2024-03-19T11:30:49.113 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.113 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.113 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:49.113 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791104+0000", 2024-03-19T11:30:49.113 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0046900830000000003 2024-03-19T11:30:49.113 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.113 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.113 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:49.113 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791248+0000", 2024-03-19T11:30:49.113 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014410200000000001 2024-03-19T11:30:49.113 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.113 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.113 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:49.114 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791340+0000", 2024-03-19T11:30:49.114 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2540000000000005e-05 2024-03-19T11:30:49.114 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.114 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:49.114 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.114 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.114 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.114 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:852 4.9 4:953e720b:::smithi03829856-45:head [write 1719614~417353] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:49.114 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.769707+0000", 2024-03-19T11:30:49.114 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 40.328529209000003, 2024-03-19T11:30:49.114 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022476643000000001, 2024-03-19T11:30:49.114 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:49.114 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:49.115 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:49.115 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:49.115 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:49.115 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 852 2024-03-19T11:30:49.115 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.115 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:49.115 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.115 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:49.115 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.769707+0000", 2024-03-19T11:30:49.115 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.115 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.115 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.115 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:49.115 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.769707+0000", 2024-03-19T11:30:49.116 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.116 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.116 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.116 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:49.116 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.769710+0000", 2024-03-19T11:30:49.116 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6230000000000001e-06 2024-03-19T11:30:49.116 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.116 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.116 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:49.116 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.770385+0000", 2024-03-19T11:30:49.116 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00067504900000000005 2024-03-19T11:30:49.116 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.116 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.116 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:49.116 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.770419+0000", 2024-03-19T11:30:49.117 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3528999999999997e-05 2024-03-19T11:30:49.117 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.117 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.117 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:49.117 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.770436+0000", 2024-03-19T11:30:49.117 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7615999999999999e-05 2024-03-19T11:30:49.117 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.117 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.117 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.117 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.781327+0000", 2024-03-19T11:30:49.117 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010891343 2024-03-19T11:30:49.117 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.117 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.117 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:49.118 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.781616+0000", 2024-03-19T11:30:49.118 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028857399999999999 2024-03-19T11:30:49.118 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.118 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.118 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:49.118 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.782125+0000", 2024-03-19T11:30:49.118 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00050941599999999995 2024-03-19T11:30:49.118 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.118 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.118 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:49.118 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.786606+0000", 2024-03-19T11:30:49.118 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044809560000000003 2024-03-19T11:30:49.118 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.118 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.118 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:49.119 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791995+0000", 2024-03-19T11:30:49.119 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053889150000000002 2024-03-19T11:30:49.119 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.119 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.119 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:49.119 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.792084+0000", 2024-03-19T11:30:49.119 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8633000000000003e-05 2024-03-19T11:30:49.119 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.119 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.119 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:49.119 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.792184+0000", 2024-03-19T11:30:49.119 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9988999999999994e-05 2024-03-19T11:30:49.119 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.120 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:49.120 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.120 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.120 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.120 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:853 4.9 4:953e720b:::smithi03829856-45:head [write 2570780~420833] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:49.120 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.775636+0000", 2024-03-19T11:30:49.120 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 40.322600588, 2024-03-19T11:30:49.120 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018795850999999999, 2024-03-19T11:30:49.120 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:49.120 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:49.120 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:49.120 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:49.120 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:49.120 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 853 2024-03-19T11:30:49.121 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.121 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:49.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.121 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:49.121 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.775636+0000", 2024-03-19T11:30:49.121 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.121 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.121 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:49.121 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.775636+0000", 2024-03-19T11:30:49.121 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.121 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.121 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:49.121 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.775639+0000", 2024-03-19T11:30:49.122 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4529999999999998e-06 2024-03-19T11:30:49.122 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.122 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.122 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:49.122 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776492+0000", 2024-03-19T11:30:49.122 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00085245000000000004 2024-03-19T11:30:49.122 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.122 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.122 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:49.122 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776526+0000", 2024-03-19T11:30:49.122 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4090000000000001e-05 2024-03-19T11:30:49.122 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.122 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.122 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:49.123 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776550+0000", 2024-03-19T11:30:49.123 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4139999999999999e-05 2024-03-19T11:30:49.123 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.123 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.123 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.123 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.786752+0000", 2024-03-19T11:30:49.123 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010201834 2024-03-19T11:30:49.123 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.123 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.123 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:49.123 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.786994+0000", 2024-03-19T11:30:49.123 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000241952 2024-03-19T11:30:49.123 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.123 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.123 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:49.124 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.787301+0000", 2024-03-19T11:30:49.124 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00030690699999999998 2024-03-19T11:30:49.124 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.124 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.124 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:49.124 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791965+0000", 2024-03-19T11:30:49.124 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0046644379999999999 2024-03-19T11:30:49.124 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.124 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.124 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:49.124 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.794165+0000", 2024-03-19T11:30:49.124 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021995389999999999 2024-03-19T11:30:49.126 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.126 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.126 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:49.126 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.794313+0000", 2024-03-19T11:30:49.126 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000148369 2024-03-19T11:30:49.126 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.126 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.126 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:49.126 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.794432+0000", 2024-03-19T11:30:49.127 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011867900000000001 2024-03-19T11:30:49.127 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.127 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:49.127 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.127 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:854 4.9 4:953e720b:::smithi03829856-45:head [setxattr _header (56) in=7b,truncate 2991613] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:49.127 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.776592+0000", 2024-03-19T11:30:49.127 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 40.321644866, 2024-03-19T11:30:49.127 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022815621000000001, 2024-03-19T11:30:49.127 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:49.127 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:49.127 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:49.127 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:49.127 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:49.128 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 854 2024-03-19T11:30:49.128 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.128 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:49.128 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.128 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:49.128 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776592+0000", 2024-03-19T11:30:49.128 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.128 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.128 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.128 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:49.128 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776592+0000", 2024-03-19T11:30:49.128 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.128 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.128 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.129 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:49.129 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776594+0000", 2024-03-19T11:30:49.129 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.95e-06 2024-03-19T11:30:49.129 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.129 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:49.129 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776603+0000", 2024-03-19T11:30:49.129 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2399999999999996e-06 2024-03-19T11:30:49.129 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.129 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:49.129 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776623+0000", 2024-03-19T11:30:49.129 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.035e-05 2024-03-19T11:30:49.129 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.130 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:49.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776639+0000", 2024-03-19T11:30:49.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5642999999999999e-05 2024-03-19T11:30:49.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.130 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.792324+0000", 2024-03-19T11:30:49.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015685075999999999 2024-03-19T11:30:49.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.130 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:49.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.792599+0000", 2024-03-19T11:30:49.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000275264 2024-03-19T11:30:49.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.131 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:49.131 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.793092+0000", 2024-03-19T11:30:49.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00049264600000000001 2024-03-19T11:30:49.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.131 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:49.131 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.794668+0000", 2024-03-19T11:30:49.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001576072 2024-03-19T11:30:49.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.131 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:49.131 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.799082+0000", 2024-03-19T11:30:49.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044145190000000004 2024-03-19T11:30:49.132 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.132 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:49.132 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.799249+0000", 2024-03-19T11:30:49.132 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016701199999999999 2024-03-19T11:30:49.132 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.132 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:49.132 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.799407+0000", 2024-03-19T11:30:49.132 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000157849 2024-03-19T11:30:49.132 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.132 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:49.132 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.132 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.133 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:855 4.9 4:953e720b:::smithi03829856-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:49.133 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.776660+0000", 2024-03-19T11:30:49.133 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 40.321576272999998, 2024-03-19T11:30:49.133 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023419786000000001, 2024-03-19T11:30:49.133 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:49.133 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:49.133 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:49.133 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:49.133 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:49.133 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 855 2024-03-19T11:30:49.133 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.133 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:49.133 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.133 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:49.133 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776660+0000", 2024-03-19T11:30:49.134 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.134 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.134 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.134 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:49.134 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776660+0000", 2024-03-19T11:30:49.134 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.134 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.134 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.134 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:49.134 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776662+0000", 2024-03-19T11:30:49.134 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3430000000000001e-06 2024-03-19T11:30:49.134 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.134 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.134 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:49.135 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776668+0000", 2024-03-19T11:30:49.135 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3500000000000002e-06 2024-03-19T11:30:49.135 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.135 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.135 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:49.135 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776687+0000", 2024-03-19T11:30:49.135 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9273999999999999e-05 2024-03-19T11:30:49.135 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.135 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.135 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:49.135 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.776705+0000", 2024-03-19T11:30:49.135 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7839999999999999e-05 2024-03-19T11:30:49.135 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.135 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.135 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.136 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.793874+0000", 2024-03-19T11:30:49.136 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017168597000000001 2024-03-19T11:30:49.136 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.136 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.136 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-03-19T11:30:49.136 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.794132+0000", 2024-03-19T11:30:49.136 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025790399999999999 2024-03-19T11:30:49.136 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.136 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.136 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.136 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.799504+0000", 2024-03-19T11:30:49.136 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053724159999999997 2024-03-19T11:30:49.136 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.136 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.136 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:49.137 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.799782+0000", 2024-03-19T11:30:49.137 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027785499999999997 2024-03-19T11:30:49.137 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.137 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.137 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:49.137 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.800080+0000", 2024-03-19T11:30:49.137 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029820699999999999 2024-03-19T11:30:49.137 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.137 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:49.137 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.137 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.137 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.137 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:871 4.d 4:b4ef97e0:::smithi03829856-3:head [write 642034~796262,stat] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:49.137 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:09.028192+0000", 2024-03-19T11:30:49.138 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 40.070044541999998, 2024-03-19T11:30:49.138 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013367027, 2024-03-19T11:30:49.138 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:49.138 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:49.138 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:49.138 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:49.138 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:49.138 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 871 2024-03-19T11:30:49.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.138 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:49.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.138 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:49.138 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.028192+0000", 2024-03-19T11:30:49.138 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.139 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.139 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:49.139 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.028192+0000", 2024-03-19T11:30:49.139 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.139 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.139 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.139 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:49.139 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.028194+0000", 2024-03-19T11:30:49.139 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3999999999999999e-06 2024-03-19T11:30:49.139 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.139 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.139 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:49.139 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.029358+0000", 2024-03-19T11:30:49.139 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011635549999999999 2024-03-19T11:30:49.139 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.140 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:49.140 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.029390+0000", 2024-03-19T11:30:49.140 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2215999999999998e-05 2024-03-19T11:30:49.140 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.140 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:49.140 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.029406+0000", 2024-03-19T11:30:49.140 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5543e-05 2024-03-19T11:30:49.140 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.140 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.140 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.029525+0000", 2024-03-19T11:30:49.140 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000119269 2024-03-19T11:30:49.140 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.141 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.141 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:49.141 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.029676+0000", 2024-03-19T11:30:49.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000150729 2024-03-19T11:30:49.141 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.141 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.141 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 2", 2024-03-19T11:30:49.141 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.029994+0000", 2024-03-19T11:30:49.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00031876099999999998 2024-03-19T11:30:49.141 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.141 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.141 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:49.141 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.040441+0000", 2024-03-19T11:30:49.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010446446 2024-03-19T11:30:49.141 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.142 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:49.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.041327+0000", 2024-03-19T11:30:49.142 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00088595299999999998 2024-03-19T11:30:49.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.142 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:49.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.041409+0000", 2024-03-19T11:30:49.142 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2332999999999999e-05 2024-03-19T11:30:49.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.142 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:49.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.041559+0000", 2024-03-19T11:30:49.143 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000149822 2024-03-19T11:30:49.143 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.143 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:49.143 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.143 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.143 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.143 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:993 4.e 4:72163e4c:::smithi03829856-39:head [write 776490~714607] snapc 5=[5,2] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-03-19T11:30:49.143 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:12.362932+0000", 2024-03-19T11:30:49.143 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 36.735304849000002, 2024-03-19T11:30:49.143 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018349900999999998, 2024-03-19T11:30:49.143 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:49.143 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:49.143 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:49.143 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:49.144 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:49.144 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 993 2024-03-19T11:30:49.144 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.144 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:49.144 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.144 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:49.144 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.362932+0000", 2024-03-19T11:30:49.144 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.144 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.144 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.144 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:49.144 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.362932+0000", 2024-03-19T11:30:49.144 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.144 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.144 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.145 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:49.145 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.362935+0000", 2024-03-19T11:30:49.145 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.88e-06 2024-03-19T11:30:49.145 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.145 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.145 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:49.145 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.366756+0000", 2024-03-19T11:30:49.145 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003821217 2024-03-19T11:30:49.145 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.145 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.145 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:49.145 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.366796+0000", 2024-03-19T11:30:49.145 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9866000000000001e-05 2024-03-19T11:30:49.145 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.145 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.146 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:49.146 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.366813+0000", 2024-03-19T11:30:49.146 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.755e-05 2024-03-19T11:30:49.146 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.146 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.146 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.146 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.366993+0000", 2024-03-19T11:30:49.146 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000179605 2024-03-19T11:30:49.146 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.146 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.146 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:49.146 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.367167+0000", 2024-03-19T11:30:49.146 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017421200000000001 2024-03-19T11:30:49.146 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.147 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-19T11:30:49.147 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.367505+0000", 2024-03-19T11:30:49.147 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033800099999999999 2024-03-19T11:30:49.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.147 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:49.147 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.380361+0000", 2024-03-19T11:30:49.147 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012855961000000001 2024-03-19T11:30:49.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.147 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:49.147 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.380987+0000", 2024-03-19T11:30:49.147 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00062590499999999995 2024-03-19T11:30:49.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.148 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.148 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:49.148 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.381149+0000", 2024-03-19T11:30:49.148 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000162239 2024-03-19T11:30:49.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.148 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.148 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:49.148 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.381282+0000", 2024-03-19T11:30:49.148 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013246500000000001 2024-03-19T11:30:49.148 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.148 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:49.148 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.148 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.148 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1072 4.1 4:8401c1f0:::smithi03829856-43:head [write 494139~557703,stat] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:49.149 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:13.972007+0000", 2024-03-19T11:30:49.149 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 35.126229099, 2024-03-19T11:30:49.149 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.01935839, 2024-03-19T11:30:49.149 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:49.149 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:49.149 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:49.149 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:49.149 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:49.149 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1072 2024-03-19T11:30:49.149 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.149 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:49.149 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.149 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:49.149 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.972007+0000", 2024-03-19T11:30:49.149 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.150 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.150 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.150 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:49.150 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.972007+0000", 2024-03-19T11:30:49.150 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.150 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.150 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.150 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:49.150 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.972011+0000", 2024-03-19T11:30:49.150 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9360000000000001e-06 2024-03-19T11:30:49.150 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.150 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.150 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:49.150 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.972881+0000", 2024-03-19T11:30:49.151 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00086922699999999998 2024-03-19T11:30:49.151 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.151 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.151 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:49.151 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.972917+0000", 2024-03-19T11:30:49.151 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6359999999999997e-05 2024-03-19T11:30:49.151 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.151 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.151 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:49.151 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.972934+0000", 2024-03-19T11:30:49.151 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7249e-05 2024-03-19T11:30:49.151 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.151 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.151 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.151 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.973132+0000", 2024-03-19T11:30:49.152 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019760200000000001 2024-03-19T11:30:49.152 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.152 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:49.152 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.973303+0000", 2024-03-19T11:30:49.152 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017142899999999999 2024-03-19T11:30:49.152 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.152 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:49.152 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.973619+0000", 2024-03-19T11:30:49.152 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00031594700000000002 2024-03-19T11:30:49.152 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.152 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:49.152 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.985856+0000", 2024-03-19T11:30:49.153 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012236349000000001 2024-03-19T11:30:49.153 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.153 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.153 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:49.153 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.991132+0000", 2024-03-19T11:30:49.153 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052764960000000003 2024-03-19T11:30:49.153 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.153 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.153 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:49.153 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.991241+0000", 2024-03-19T11:30:49.153 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000108842 2024-03-19T11:30:49.153 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.153 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.153 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:49.153 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.991366+0000", 2024-03-19T11:30:49.154 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012495299999999999 2024-03-19T11:30:49.154 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.154 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:49.154 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.154 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.154 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.154 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1162 4.1 4:8401c1f0:::smithi03829856-43:head [write 594981~609197] snapc 8=[8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-03-19T11:30:49.154 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.838981+0000", 2024-03-19T11:30:49.154 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.259255744000001, 2024-03-19T11:30:49.154 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027704349999999999, 2024-03-19T11:30:49.154 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:49.154 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:49.154 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:49.154 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:49.155 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:49.155 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1162 2024-03-19T11:30:49.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.155 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:49.155 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.155 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:49.155 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.838981+0000", 2024-03-19T11:30:49.155 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.155 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.155 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:49.155 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.838981+0000", 2024-03-19T11:30:49.155 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.155 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.156 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:49.156 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.838983+0000", 2024-03-19T11:30:49.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5399999999999998e-06 2024-03-19T11:30:49.156 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.156 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:49.156 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841322+0000", 2024-03-19T11:30:49.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002338592 2024-03-19T11:30:49.156 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.156 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:49.156 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841371+0000", 2024-03-19T11:30:49.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9123999999999999e-05 2024-03-19T11:30:49.156 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.157 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.157 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:49.157 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841399+0000", 2024-03-19T11:30:49.157 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7718000000000001e-05 2024-03-19T11:30:49.157 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.157 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.157 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.157 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841577+0000", 2024-03-19T11:30:49.157 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000178097 2024-03-19T11:30:49.157 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.157 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.157 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:49.157 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841825+0000", 2024-03-19T11:30:49.157 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00024824299999999999 2024-03-19T11:30:49.157 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.158 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.158 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:49.158 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842227+0000", 2024-03-19T11:30:49.158 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00040212199999999998 2024-03-19T11:30:49.158 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.158 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.158 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:49.158 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.865182+0000", 2024-03-19T11:30:49.158 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022954810999999999 2024-03-19T11:30:49.158 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.158 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.158 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:49.158 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.866360+0000", 2024-03-19T11:30:49.158 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001178449 2024-03-19T11:30:49.159 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.159 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.159 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:49.159 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.866538+0000", 2024-03-19T11:30:49.159 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017791000000000001 2024-03-19T11:30:49.159 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.159 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.159 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:49.159 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.866685+0000", 2024-03-19T11:30:49.159 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000146744 2024-03-19T11:30:49.159 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.159 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:49.159 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.159 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.159 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.160 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1163 4.1 4:8401c1f0:::smithi03829856-43:head [write 1974123~745028] snapc 8=[8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-03-19T11:30:49.160 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.851879+0000", 2024-03-19T11:30:49.160 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.246357621999998, 2024-03-19T11:30:49.160 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015531774, 2024-03-19T11:30:49.160 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:49.160 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:49.160 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:49.160 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:49.160 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:49.160 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1163 2024-03-19T11:30:49.160 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.160 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:49.160 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.160 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:49.160 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.851879+0000", 2024-03-19T11:30:49.161 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.161 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.161 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.161 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:49.161 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.851879+0000", 2024-03-19T11:30:49.161 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.161 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.161 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.161 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:49.161 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.851882+0000", 2024-03-19T11:30:49.161 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6960000000000001e-06 2024-03-19T11:30:49.161 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.161 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.161 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:49.162 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.853204+0000", 2024-03-19T11:30:49.162 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0013226620000000001 2024-03-19T11:30:49.162 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.162 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.162 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:49.162 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.853231+0000", 2024-03-19T11:30:49.162 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6775e-05 2024-03-19T11:30:49.162 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.162 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.162 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:49.162 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.853246+0000", 2024-03-19T11:30:49.162 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5403000000000001e-05 2024-03-19T11:30:49.162 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.162 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.162 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.163 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.853363+0000", 2024-03-19T11:30:49.163 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000116296 2024-03-19T11:30:49.163 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.163 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.163 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:49.163 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.853572+0000", 2024-03-19T11:30:49.163 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020931500000000001 2024-03-19T11:30:49.163 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.163 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.163 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:49.163 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.853797+0000", 2024-03-19T11:30:49.163 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022544700000000001 2024-03-19T11:30:49.163 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.163 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.164 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:49.164 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.865195+0000", 2024-03-19T11:30:49.164 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011397981999999999 2024-03-19T11:30:49.164 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.164 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.164 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:49.164 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.866916+0000", 2024-03-19T11:30:49.164 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0017204449999999999 2024-03-19T11:30:49.164 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.164 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.164 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:49.164 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.867129+0000", 2024-03-19T11:30:49.164 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021317999999999999 2024-03-19T11:30:49.164 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.164 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.165 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:49.165 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.867411+0000", 2024-03-19T11:30:49.165 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028157300000000002 2024-03-19T11:30:49.165 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.165 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:49.165 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.165 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.165 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.165 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1205 4.1 4:8401c1f0:::smithi03829856-43:head [assert-version v50,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:49.165 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:18.112028+0000", 2024-03-19T11:30:49.165 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 30.986208487999999, 2024-03-19T11:30:49.165 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013579206, 2024-03-19T11:30:49.165 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:49.165 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:49.165 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:49.166 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:49.166 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:49.166 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1205 2024-03-19T11:30:49.166 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.166 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:49.166 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.166 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:49.166 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112028+0000", 2024-03-19T11:30:49.166 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.166 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.166 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.166 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:49.166 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112028+0000", 2024-03-19T11:30:49.166 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.166 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.167 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.167 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:49.167 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112030+0000", 2024-03-19T11:30:49.167 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4549999999999998e-06 2024-03-19T11:30:49.167 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.167 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.167 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:49.167 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112043+0000", 2024-03-19T11:30:49.167 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2721999999999999e-05 2024-03-19T11:30:49.167 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.167 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.167 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:49.167 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112075+0000", 2024-03-19T11:30:49.167 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.205e-05 2024-03-19T11:30:49.167 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.168 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.168 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:49.168 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112091+0000", 2024-03-19T11:30:49.168 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5364000000000001e-05 2024-03-19T11:30:49.168 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.168 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.168 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.168 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112223+0000", 2024-03-19T11:30:49.168 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013224100000000001 2024-03-19T11:30:49.168 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.168 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.168 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:49.168 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112393+0000", 2024-03-19T11:30:49.168 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000169681 2024-03-19T11:30:49.168 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.169 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.169 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:49.169 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.120919+0000", 2024-03-19T11:30:49.169 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085260140000000002 2024-03-19T11:30:49.169 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.169 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.169 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:49.169 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125207+0000", 2024-03-19T11:30:49.169 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004288722 2024-03-19T11:30:49.169 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.169 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.169 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:49.169 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125343+0000", 2024-03-19T11:30:49.169 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013558400000000001 2024-03-19T11:30:49.170 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.170 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.170 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:49.170 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125497+0000", 2024-03-19T11:30:49.170 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000153678 2024-03-19T11:30:49.170 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.170 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.170 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:49.170 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125607+0000", 2024-03-19T11:30:49.170 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000110695 2024-03-19T11:30:49.170 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.170 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:49.170 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.170 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.171 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.171 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1208 4.1 4:866f3dcc:::smithi03829856-31:head [read 0~2941192,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:49.171 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:18.112252+0000", 2024-03-19T11:30:49.171 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 30.985984918, 2024-03-19T11:30:49.171 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.01291702, 2024-03-19T11:30:49.171 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:49.171 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:49.171 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:49.171 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:49.171 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:49.171 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1208 2024-03-19T11:30:49.171 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.171 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:49.171 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.171 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:49.172 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112252+0000", 2024-03-19T11:30:49.172 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.172 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.172 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.172 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:49.172 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112252+0000", 2024-03-19T11:30:49.172 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.172 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.172 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.172 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:49.172 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112253+0000", 2024-03-19T11:30:49.172 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6299999999999993e-07 2024-03-19T11:30:49.172 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.172 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.173 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:49.173 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112257+0000", 2024-03-19T11:30:49.173 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4360000000000004e-06 2024-03-19T11:30:49.173 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.173 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.173 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:49.173 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112272+0000", 2024-03-19T11:30:49.173 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.518e-05 2024-03-19T11:30:49.173 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.173 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.173 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:49.173 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.112281+0000", 2024-03-19T11:30:49.173 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.422e-06 2024-03-19T11:30:49.173 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.173 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.174 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.174 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.122861+0000", 2024-03-19T11:30:49.174 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010580523 2024-03-19T11:30:49.174 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.174 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.174 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:49.174 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123148+0000", 2024-03-19T11:30:49.174 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028714499999999999 2024-03-19T11:30:49.174 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.174 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.174 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:49.174 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125169+0000", 2024-03-19T11:30:49.174 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0020203510000000001 2024-03-19T11:30:49.174 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.175 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:49.175 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.175 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.175 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.175 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1243 4.9 4:9ef41243:::smithi03829856-5:head [tier-promote] snapc 9=[9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:49.175 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:18.345759+0000", 2024-03-19T11:30:49.175 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 30.752477306999999, 2024-03-19T11:30:49.175 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018400962999999999, 2024-03-19T11:30:49.175 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:49.175 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:49.175 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:49.175 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:49.175 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:49.175 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1243 2024-03-19T11:30:49.175 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.176 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:49.176 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.176 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:49.176 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.345759+0000", 2024-03-19T11:30:49.176 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.176 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.176 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.176 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:49.176 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.345759+0000", 2024-03-19T11:30:49.176 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.176 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.176 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.176 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:49.176 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.345763+0000", 2024-03-19T11:30:49.176 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3790000000000002e-06 2024-03-19T11:30:49.177 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.177 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.177 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:49.177 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.345776+0000", 2024-03-19T11:30:49.177 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3784000000000001e-05 2024-03-19T11:30:49.177 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.177 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.177 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:49.177 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.345812+0000", 2024-03-19T11:30:49.177 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5599999999999998e-05 2024-03-19T11:30:49.177 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.177 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.177 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:49.177 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.345829+0000", 2024-03-19T11:30:49.178 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6722999999999999e-05 2024-03-19T11:30:49.178 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.178 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.178 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.178 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.345964+0000", 2024-03-19T11:30:49.178 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013554800000000001 2024-03-19T11:30:49.178 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.178 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.178 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:49.178 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.346133+0000", 2024-03-19T11:30:49.178 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000169211 2024-03-19T11:30:49.178 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.178 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.178 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:49.178 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.364160+0000", 2024-03-19T11:30:49.179 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018026718000000001 2024-03-19T11:30:49.179 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.179 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:49.179 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.179 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.179 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.179 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1271 4.9 4:953e720b:::smithi03829856-45:2 [read 795136~1008128] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-03-19T11:30:49.179 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:21.868056+0000", 2024-03-19T11:30:49.179 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.230180238999999, 2024-03-19T11:30:49.179 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.047270591000000001, 2024-03-19T11:30:49.179 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:49.179 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:49.179 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:49.179 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:49.179 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:49.180 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1271 2024-03-19T11:30:49.180 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.180 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:49.180 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.180 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:49.180 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868056+0000", 2024-03-19T11:30:49.180 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.180 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.180 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.180 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:49.180 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868056+0000", 2024-03-19T11:30:49.180 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.180 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.180 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.180 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:49.181 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868059+0000", 2024-03-19T11:30:49.181 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0469999999999998e-06 2024-03-19T11:30:49.181 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.181 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.181 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:49.181 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868072+0000", 2024-03-19T11:30:49.181 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2700999999999999e-05 2024-03-19T11:30:49.181 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.181 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.181 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:49.181 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868106+0000", 2024-03-19T11:30:49.181 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4152000000000001e-05 2024-03-19T11:30:49.181 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.181 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.181 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:49.182 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868126+0000", 2024-03-19T11:30:49.182 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9386999999999999e-05 2024-03-19T11:30:49.182 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.182 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.182 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.182 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.911447+0000", 2024-03-19T11:30:49.182 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.043321164000000002 2024-03-19T11:30:49.182 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.182 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.182 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:49.182 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.911762+0000", 2024-03-19T11:30:49.182 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00031492400000000001 2024-03-19T11:30:49.182 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.182 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.183 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:49.183 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.915327+0000", 2024-03-19T11:30:49.183 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003565216 2024-03-19T11:30:49.183 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.183 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:49.183 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.183 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.183 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.183 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1617 4.e 4:7773c291:::smithi03829856-11:head [write 450454~642026,stat] snapc 10=[10,f,e,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-03-19T11:30:49.183 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:35.152724+0000", 2024-03-19T11:30:49.183 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 13.945512044999999, 2024-03-19T11:30:49.183 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016139337, 2024-03-19T11:30:49.183 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:49.183 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:49.183 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:49.184 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:49.184 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:49.184 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1617 2024-03-19T11:30:49.184 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.184 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:49.184 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.184 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:49.184 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.152724+0000", 2024-03-19T11:30:49.184 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.184 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.184 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.184 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:49.184 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.152724+0000", 2024-03-19T11:30:49.184 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.185 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.185 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.185 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:49.185 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.152727+0000", 2024-03-19T11:30:49.185 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8399999999999999e-06 2024-03-19T11:30:49.185 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.185 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.185 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:49.185 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.153790+0000", 2024-03-19T11:30:49.185 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010623849999999999 2024-03-19T11:30:49.185 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.185 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.185 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:49.185 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.153825+0000", 2024-03-19T11:30:49.185 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5280000000000001e-05 2024-03-19T11:30:49.186 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.186 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.186 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:49.186 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.153842+0000", 2024-03-19T11:30:49.186 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6942000000000001e-05 2024-03-19T11:30:49.186 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.186 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.186 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.186 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.154018+0000", 2024-03-19T11:30:49.186 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017568 2024-03-19T11:30:49.186 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.186 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.186 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:49.186 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.154302+0000", 2024-03-19T11:30:49.186 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028463400000000003 2024-03-19T11:30:49.187 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.187 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.187 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-19T11:30:49.187 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.154811+0000", 2024-03-19T11:30:49.187 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00050833300000000003 2024-03-19T11:30:49.187 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.187 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.187 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:49.187 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.168286+0000", 2024-03-19T11:30:49.187 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013475354 2024-03-19T11:30:49.187 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.187 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.187 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:49.187 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.168631+0000", 2024-03-19T11:30:49.188 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00034488800000000001 2024-03-19T11:30:49.188 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.188 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.188 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:49.188 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.168749+0000", 2024-03-19T11:30:49.188 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011799900000000001 2024-03-19T11:30:49.188 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.188 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.188 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:49.188 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.168864+0000", 2024-03-19T11:30:49.188 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011500199999999999 2024-03-19T11:30:49.188 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.188 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:49.188 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.188 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.188 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.189 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1679 4.e 4:72163e4c:::smithi03829856-39:head [write 517682~543022,stat] snapc 10=[10,e,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-03-19T11:30:49.189 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:36.388669+0000", 2024-03-19T11:30:49.189 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 12.709567712, 2024-03-19T11:30:49.189 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016197706999999999, 2024-03-19T11:30:49.189 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:49.189 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:49.189 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:49.189 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:49.189 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:49.189 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1679 2024-03-19T11:30:49.189 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.189 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:49.189 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.189 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:49.190 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.388669+0000", 2024-03-19T11:30:49.190 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.190 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.190 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.190 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:49.190 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.388669+0000", 2024-03-19T11:30:49.190 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.190 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.190 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.190 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:49.190 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.388672+0000", 2024-03-19T11:30:49.190 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4130000000000002e-06 2024-03-19T11:30:49.190 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.190 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.190 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:49.191 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.389483+0000", 2024-03-19T11:30:49.191 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00081121800000000005 2024-03-19T11:30:49.191 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.191 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.191 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:49.191 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.389507+0000", 2024-03-19T11:30:49.191 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3482999999999999e-05 2024-03-19T11:30:49.191 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.191 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.191 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:49.191 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.389517+0000", 2024-03-19T11:30:49.191 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0251999999999999e-05 2024-03-19T11:30:49.191 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.191 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.191 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.192 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.389672+0000", 2024-03-19T11:30:49.192 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015465300000000001 2024-03-19T11:30:49.192 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.192 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.192 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:49.192 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.389839+0000", 2024-03-19T11:30:49.192 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016710800000000001 2024-03-19T11:30:49.192 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.192 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.192 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 6", 2024-03-19T11:30:49.192 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.390102+0000", 2024-03-19T11:30:49.192 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026344900000000002 2024-03-19T11:30:49.192 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.192 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.192 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:49.193 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.401899+0000", 2024-03-19T11:30:49.193 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011796836 2024-03-19T11:30:49.193 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.193 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.193 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:49.193 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.404530+0000", 2024-03-19T11:30:49.193 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026311949999999998 2024-03-19T11:30:49.193 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.193 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.193 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:49.193 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.404722+0000", 2024-03-19T11:30:49.193 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019109300000000001 2024-03-19T11:30:49.193 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.193 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.194 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:49.194 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.404867+0000", 2024-03-19T11:30:49.194 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000145007 2024-03-19T11:30:49.194 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.194 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:49.194 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.194 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.194 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:49.194 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:49.196 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-19T11:30:49.197 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:49.204+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:49.197 INFO:tasks.rados.rados.0.smithi038.stdout:1079: finishing rollback tid 0 to smithi03829856-14 2024-03-19T11:30:49.198 INFO:tasks.rados.rados.0.smithi038.stdout:1079: finishing rollback tid 1 to smithi03829856-14 2024-03-19T11:30:49.198 INFO:tasks.rados.rados.0.smithi038.stdout:1079: finishing rollback tid 2 to smithi03829856-14 2024-03-19T11:30:49.198 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 14 v 46 (ObjNum 88 snap 8 seq_num 88) dirty exists 2024-03-19T11:30:49.198 INFO:tasks.rados.rados.0.smithi038.stdout:1075: done (4 left) 2024-03-19T11:30:49.198 INFO:tasks.rados.rados.0.smithi038.stdout:1076: done (3 left) 2024-03-19T11:30:49.198 INFO:tasks.rados.rados.0.smithi038.stdout:1077: done (2 left) 2024-03-19T11:30:49.198 INFO:tasks.rados.rados.0.smithi038.stdout:1078: done (1 left) 2024-03-19T11:30:49.198 INFO:tasks.rados.rados.0.smithi038.stdout:1079: done (0 left) 2024-03-19T11:30:49.198 INFO:tasks.rados.rados.0.smithi038.stdout:1080: read oid 43 snap 4 2024-03-19T11:30:49.198 INFO:tasks.rados.rados.0.smithi038.stdout:1080: expect (ObjNum 50 snap 0 seq_num 50) 2024-03-19T11:30:49.198 INFO:tasks.rados.rados.0.smithi038.stdout:1081: read oid 1 snap 9 2024-03-19T11:30:49.198 INFO:tasks.rados.rados.0.smithi038.stdout:1081: expect (ObjNum 51 snap 0 seq_num 51) 2024-03-19T11:30:49.198 INFO:tasks.rados.rados.0.smithi038.stdout:1082: read oid 38 snap -1 2024-03-19T11:30:49.198 INFO:tasks.rados.rados.0.smithi038.stdout:1082: expect (ObjNum 125 snap 15 seq_num 125) 2024-03-19T11:30:49.199 INFO:tasks.rados.rados.0.smithi038.stdout:1083: write (excl) oid 27 current snap is 17 2024-03-19T11:30:49.199 INFO:tasks.rados.rados.0.smithi038.stdout:1083: seq_num 171 ranges {473997=627345,1673362=441052,2529567=344430} 2024-03-19T11:30:49.199 INFO:tasks.rados.rados.0.smithi038.stdout:1083: writing smithi03829856-27 from 473997 to 1101342 tid 1 2024-03-19T11:30:49.199 INFO:tasks.rados.rados.0.smithi038.stdout:1083: writing smithi03829856-27 from 1673362 to 2114414 tid 2 2024-03-19T11:30:49.199 INFO:tasks.rados.rados.0.smithi038.stdout:1083: writing smithi03829856-27 from 2529567 to 2873997 tid 3 2024-03-19T11:30:49.199 INFO:tasks.rados.rados.0.smithi038.stdout:1084: tier_evict oid 12 2024-03-19T11:30:49.199 INFO:tasks.rados.rados.0.smithi038.stdout:1084: tier_evict oid 12 snap -1 2024-03-19T11:30:49.199 INFO:tasks.rados.rados.0.smithi038.stdout:1082: expect (ObjNum 125 snap 15 seq_num 125) 2024-03-19T11:30:49.264 INFO:tasks.rados.rados.0.smithi038.stdout:1080: expect (ObjNum 50 snap 0 seq_num 50) 2024-03-19T11:30:49.296 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:49.304+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:49.367 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:49.368 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:49.368 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:30:49.368 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:49.379 INFO:tasks.rados.rados.0.smithi038.stdout:1081: expect (ObjNum 51 snap 0 seq_num 51) 2024-03-19T11:30:49.381 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-19T11:30:49.397 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:49.406+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:49.498 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:49.506+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:49.504 INFO:tasks.rados.rados.0.smithi038.stdout:1083: finishing write tid 1 to smithi03829856-27 2024-03-19T11:30:49.505 INFO:tasks.rados.rados.0.smithi038.stdout:1083: finishing write tid 2 to smithi03829856-27 2024-03-19T11:30:49.505 INFO:tasks.rados.rados.0.smithi038.stdout:1084: got (22) Invalid argument 2024-03-19T11:30:49.505 INFO:tasks.rados.rados.0.smithi038.stdout:1083: finishing write tid 3 to smithi03829856-27 2024-03-19T11:30:49.505 INFO:tasks.rados.rados.0.smithi038.stdout:1083: finishing write tid 4 to smithi03829856-27 2024-03-19T11:30:49.505 INFO:tasks.rados.rados.0.smithi038.stdout:1083: finishing write tid 5 to smithi03829856-27 2024-03-19T11:30:49.505 INFO:tasks.rados.rados.0.smithi038.stdout:1083: oid 27 updating version 0 to 82 2024-03-19T11:30:49.505 INFO:tasks.rados.rados.0.smithi038.stdout:1083: oid 27 version 82 is already newer than 81 2024-03-19T11:30:49.505 INFO:tasks.rados.rados.0.smithi038.stdout:1083: oid 27 version 82 is already newer than 79 2024-03-19T11:30:49.505 INFO:tasks.rados.rados.0.smithi038.stdout:1083: oid 27 version 82 is already newer than 80 2024-03-19T11:30:49.505 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 27 v 82 (ObjNum 171 snap 17 seq_num 171) dirty exists 2024-03-19T11:30:49.505 INFO:tasks.rados.rados.0.smithi038.stdout:1083: left oid 27 (ObjNum 171 snap 17 seq_num 171) 2024-03-19T11:30:49.505 INFO:tasks.rados.rados.0.smithi038.stdout:1080: done (4 left) 2024-03-19T11:30:49.505 INFO:tasks.rados.rados.0.smithi038.stdout:1081: done (3 left) 2024-03-19T11:30:49.506 INFO:tasks.rados.rados.0.smithi038.stdout:1082: done (2 left) 2024-03-19T11:30:49.506 INFO:tasks.rados.rados.0.smithi038.stdout:1083: done (1 left) 2024-03-19T11:30:49.506 INFO:tasks.rados.rados.0.smithi038.stdout:1084: done (0 left) 2024-03-19T11:30:49.506 INFO:tasks.rados.rados.0.smithi038.stdout:1085: read oid 39 snap -1 2024-03-19T11:30:49.506 INFO:tasks.rados.rados.0.smithi038.stdout:1085: expect (ObjNum 163 snap 17 seq_num 163) 2024-03-19T11:30:49.506 INFO:tasks.rados.rados.0.smithi038.stdout:1086: read oid 1 snap -1 2024-03-19T11:30:49.506 INFO:tasks.rados.rados.0.smithi038.stdout:1086: expect (ObjNum 156 snap 17 seq_num 156) 2024-03-19T11:30:49.506 INFO:tasks.rados.rados.0.smithi038.stdout:1087: set_chunk oid 20 target oid 2024-03-19T11:30:49.519 INFO:tasks.rados.rados.0.smithi038.stdout:1087: set_chunk oid 20 offset: 246784 length: 502784 target oid 509f29e62b6d8dfa4f7b707436d53870a30e329ada1da172e3b16968da862bd5 offset: 0 snap -1 2024-03-19T11:30:49.519 INFO:tasks.rados.rados.0.smithi038.stdout:1088: read oid 45 snap -1 2024-03-19T11:30:49.520 INFO:tasks.rados.rados.0.smithi038.stdout:1088: expect (ObjNum 148 snap 15 seq_num 148) 2024-03-19T11:30:49.520 INFO:tasks.rados.rados.0.smithi038.stdout:1089: snap_create 2024-03-19T11:30:49.520 INFO:tasks.rados.rados.0.smithi038.stdout:1086: expect (ObjNum 156 snap 17 seq_num 156) 2024-03-19T11:30:49.551 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:49.551 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:49.551 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:30:49.551 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:30:49.551 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:49.565 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-19T11:30:49.599 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:49.610+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:49.690 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:49.690 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:30:49.690 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:30:49.690 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:30:49.690 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.691 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:844 4.7 e32/21 4:e41d47fb:::smithi03829856-14:head v 32'8, mlcod=28'5)", 2024-03-19T11:30:49.691 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.751447+0000", 2024-03-19T11:30:49.691 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 40.947743748000001, 2024-03-19T11:30:49.691 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012301835000000001, 2024-03-19T11:30:49.691 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:49.691 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:49.691 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:49.691 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.691 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:49.691 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.751447+0000", 2024-03-19T11:30:49.691 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.691 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.691 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.691 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:49.691 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.751447+0000", 2024-03-19T11:30:49.692 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.692 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.692 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.692 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:49.692 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.751447+0000", 2024-03-19T11:30:49.692 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4300000000000003e-07 2024-03-19T11:30:49.692 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.692 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.692 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:49.692 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.752747+0000", 2024-03-19T11:30:49.692 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012995699999999999 2024-03-19T11:30:49.692 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.692 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.692 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:49.693 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.752771+0000", 2024-03-19T11:30:49.693 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3592999999999999e-05 2024-03-19T11:30:49.693 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.693 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.693 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:49.693 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.752789+0000", 2024-03-19T11:30:49.693 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8703e-05 2024-03-19T11:30:49.693 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.693 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.693 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.693 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.752911+0000", 2024-03-19T11:30:49.693 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012139899999999999 2024-03-19T11:30:49.693 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.693 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.694 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:49.694 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.752949+0000", 2024-03-19T11:30:49.694 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8436999999999998e-05 2024-03-19T11:30:49.694 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.694 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.694 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:49.694 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763568+0000", 2024-03-19T11:30:49.694 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010618768000000001 2024-03-19T11:30:49.694 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.694 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.694 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:49.694 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763749+0000", 2024-03-19T11:30:49.695 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000180822 2024-03-19T11:30:49.695 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.695 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:49.695 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.695 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.695 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.695 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1047 4.8 4:1d58db11:::smithi03829856-18:head [write 443718~615603,stat] snapc 6=[6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:49.695 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:13.221766+0000", 2024-03-19T11:30:49.695 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 36.477424581000001, 2024-03-19T11:30:49.695 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020932596000000001, 2024-03-19T11:30:49.695 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:49.695 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:49.695 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:49.695 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:49.696 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:49.696 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1047 2024-03-19T11:30:49.696 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.696 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:49.696 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.696 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:49.696 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.221766+0000", 2024-03-19T11:30:49.696 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.696 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.696 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.696 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:49.696 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.221766+0000", 2024-03-19T11:30:49.696 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.696 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.696 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.696 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:49.697 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.221770+0000", 2024-03-19T11:30:49.697 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.67e-06 2024-03-19T11:30:49.697 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.697 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.697 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:49.697 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.223938+0000", 2024-03-19T11:30:49.697 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021683399999999999 2024-03-19T11:30:49.697 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.697 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.697 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:49.697 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.223979+0000", 2024-03-19T11:30:49.697 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1433000000000001e-05 2024-03-19T11:30:49.697 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.698 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.698 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:49.698 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.223998+0000", 2024-03-19T11:30:49.698 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8853000000000001e-05 2024-03-19T11:30:49.698 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.698 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.698 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.698 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.224150+0000", 2024-03-19T11:30:49.698 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000151679 2024-03-19T11:30:49.698 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.698 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.699 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:49.699 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.224362+0000", 2024-03-19T11:30:49.699 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021154499999999999 2024-03-19T11:30:49.699 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.699 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.699 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:49.699 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.224745+0000", 2024-03-19T11:30:49.699 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00038342399999999999 2024-03-19T11:30:49.699 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.699 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.699 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:49.699 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.241167+0000", 2024-03-19T11:30:49.699 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0164218 2024-03-19T11:30:49.699 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.700 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.700 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:49.700 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.242406+0000", 2024-03-19T11:30:49.700 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012390039999999999 2024-03-19T11:30:49.700 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.700 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.700 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:49.700 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.242575+0000", 2024-03-19T11:30:49.700 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000168925 2024-03-19T11:30:49.700 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.700 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.701 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:49.701 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.242699+0000", 2024-03-19T11:30:49.701 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012392300000000001 2024-03-19T11:30:49.701 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.701 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:49.701 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.701 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.701 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.701 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1048 4.8 4:1d58db11:::smithi03829856-18:head [write 1564138~470341] snapc 6=[6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:49.701 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:13.227287+0000", 2024-03-19T11:30:49.701 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 36.471903312999999, 2024-03-19T11:30:49.701 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015802897999999999, 2024-03-19T11:30:49.701 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:49.702 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:49.702 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:49.702 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:49.702 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:49.702 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1048 2024-03-19T11:30:49.702 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.702 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:49.702 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.702 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:49.702 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.227287+0000", 2024-03-19T11:30:49.702 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.702 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.702 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.702 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:49.703 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.227287+0000", 2024-03-19T11:30:49.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.703 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.703 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.703 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:49.703 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.227290+0000", 2024-03-19T11:30:49.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.88e-06 2024-03-19T11:30:49.704 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.704 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.704 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:49.704 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.228224+0000", 2024-03-19T11:30:49.704 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00093389600000000003 2024-03-19T11:30:49.704 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.705 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.705 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:49.705 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.228258+0000", 2024-03-19T11:30:49.705 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4282999999999997e-05 2024-03-19T11:30:49.705 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.705 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.706 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:49.706 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.228273+0000", 2024-03-19T11:30:49.706 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4833e-05 2024-03-19T11:30:49.706 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.706 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.706 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.707 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.231996+0000", 2024-03-19T11:30:49.707 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037223410000000001 2024-03-19T11:30:49.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.707 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:49.707 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.232143+0000", 2024-03-19T11:30:49.707 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014714900000000001 2024-03-19T11:30:49.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.707 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:49.708 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.232395+0000", 2024-03-19T11:30:49.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025201500000000002 2024-03-19T11:30:49.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.708 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.708 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:49.708 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.242848+0000", 2024-03-19T11:30:49.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010453393 2024-03-19T11:30:49.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.708 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.708 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:49.709 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.242872+0000", 2024-03-19T11:30:49.709 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3816e-05 2024-03-19T11:30:49.709 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.709 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.709 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:49.709 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.242966+0000", 2024-03-19T11:30:49.709 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4560000000000003e-05 2024-03-19T11:30:49.709 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.709 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.709 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:49.709 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.243090+0000", 2024-03-19T11:30:49.709 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012373199999999999 2024-03-19T11:30:49.709 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.709 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:49.709 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.710 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.710 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1072 4.1 e36/21 4:8401c1f0:::smithi03829856-43:head v 36'38, mlcod=36'35)", 2024-03-19T11:30:49.710 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:13.973829+0000", 2024-03-19T11:30:49.710 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 35.725361990000003, 2024-03-19T11:30:49.710 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013548782000000001, 2024-03-19T11:30:49.710 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:49.710 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:49.710 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:49.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.710 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:49.710 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.973829+0000", 2024-03-19T11:30:49.710 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.711 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.711 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.711 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:49.711 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.973829+0000", 2024-03-19T11:30:49.711 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.711 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.711 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.711 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:49.711 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.973831+0000", 2024-03-19T11:30:49.711 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0899999999999999e-06 2024-03-19T11:30:49.711 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.711 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.711 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:49.712 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.974489+0000", 2024-03-19T11:30:49.712 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00065813199999999997 2024-03-19T11:30:49.712 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.712 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.712 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:49.712 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.974523+0000", 2024-03-19T11:30:49.712 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3841999999999999e-05 2024-03-19T11:30:49.713 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.713 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.713 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:49.713 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.974544+0000", 2024-03-19T11:30:49.713 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0907000000000002e-05 2024-03-19T11:30:49.713 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.714 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.714 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.714 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.974697+0000", 2024-03-19T11:30:49.714 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015295599999999999 2024-03-19T11:30:49.714 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.714 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.715 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:49.715 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.974742+0000", 2024-03-19T11:30:49.715 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5735999999999997e-05 2024-03-19T11:30:49.715 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.715 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.715 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:49.715 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.987261+0000", 2024-03-19T11:30:49.715 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012518316999999999 2024-03-19T11:30:49.715 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.715 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.716 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:49.716 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.987377+0000", 2024-03-19T11:30:49.716 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000116802 2024-03-19T11:30:49.716 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.716 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:49.716 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.716 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.716 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.716 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1091 4.8 4:14611006:::smithi03829856-25:head [write 527753~708201] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:49.716 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:14.015839+0000", 2024-03-19T11:30:49.716 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 35.683351930999997, 2024-03-19T11:30:49.716 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016721983999999999, 2024-03-19T11:30:49.716 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:49.716 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:49.716 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:49.717 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:49.717 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:49.717 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1091 2024-03-19T11:30:49.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.717 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:49.717 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.717 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:49.717 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.015839+0000", 2024-03-19T11:30:49.717 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.717 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.717 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:49.717 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.015839+0000", 2024-03-19T11:30:49.717 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.718 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.718 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:49.718 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.015842+0000", 2024-03-19T11:30:49.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3569999999999998e-06 2024-03-19T11:30:49.718 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.718 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.718 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:49.718 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.016678+0000", 2024-03-19T11:30:49.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00083620299999999999 2024-03-19T11:30:49.718 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.718 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.718 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:49.718 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.016710+0000", 2024-03-19T11:30:49.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1337000000000002e-05 2024-03-19T11:30:49.719 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.719 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:49.719 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.016724+0000", 2024-03-19T11:30:49.719 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4323e-05 2024-03-19T11:30:49.719 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.719 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.720 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.016847+0000", 2024-03-19T11:30:49.720 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012304199999999999 2024-03-19T11:30:49.720 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.720 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:49.720 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.017023+0000", 2024-03-19T11:30:49.720 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000175945 2024-03-19T11:30:49.720 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.720 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:49.720 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.017324+0000", 2024-03-19T11:30:49.720 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00030151499999999998 2024-03-19T11:30:49.720 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.720 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:49.721 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.032104+0000", 2024-03-19T11:30:49.721 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014779232999999999 2024-03-19T11:30:49.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.721 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.721 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:49.721 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.032190+0000", 2024-03-19T11:30:49.721 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6304999999999994e-05 2024-03-19T11:30:49.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.721 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.721 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:49.721 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.032350+0000", 2024-03-19T11:30:49.721 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000159813 2024-03-19T11:30:49.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.721 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.722 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:49.722 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.032561+0000", 2024-03-19T11:30:49.722 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021091099999999999 2024-03-19T11:30:49.722 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.722 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:49.722 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.722 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.722 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.722 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1158 4.8 4:15708482:::smithi03829856-23:head [tier-promote] snapc 8=[8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-03-19T11:30:49.723 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.827517+0000", 2024-03-19T11:30:49.723 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.871674003000003, 2024-03-19T11:30:49.723 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.034096983999999997, 2024-03-19T11:30:49.723 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:49.723 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:49.723 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:49.724 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:49.724 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:49.724 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1158 2024-03-19T11:30:49.724 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.724 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:49.724 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.724 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:49.725 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827517+0000", 2024-03-19T11:30:49.725 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.725 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.725 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.725 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:49.725 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827517+0000", 2024-03-19T11:30:49.725 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.725 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.726 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.726 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:49.726 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827520+0000", 2024-03-19T11:30:49.726 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.45e-06 2024-03-19T11:30:49.726 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.726 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.726 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:49.726 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827535+0000", 2024-03-19T11:30:49.726 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.509e-05 2024-03-19T11:30:49.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.727 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:49.727 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827579+0000", 2024-03-19T11:30:49.727 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4211000000000001e-05 2024-03-19T11:30:49.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.727 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:49.727 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827603+0000", 2024-03-19T11:30:49.727 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3586000000000001e-05 2024-03-19T11:30:49.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.727 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.727 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827789+0000", 2024-03-19T11:30:49.727 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018580199999999999 2024-03-19T11:30:49.728 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.728 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.728 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:49.728 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.828077+0000", 2024-03-19T11:30:49.728 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028846500000000001 2024-03-19T11:30:49.728 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.728 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.728 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:49.728 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.861614+0000", 2024-03-19T11:30:49.728 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033536379999999998 2024-03-19T11:30:49.728 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.728 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:49.728 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.728 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.728 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.729 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1162 4.1 e39/21 4:8401c1f0:::smithi03829856-43:head v 39'47, mlcod=38'43)", 2024-03-19T11:30:49.729 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.842598+0000", 2024-03-19T11:30:49.729 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.856592233000001, 2024-03-19T11:30:49.729 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0233756, 2024-03-19T11:30:49.729 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:49.729 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:49.729 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:49.730 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.730 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:49.730 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842598+0000", 2024-03-19T11:30:49.730 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.730 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.730 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.730 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:49.730 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842598+0000", 2024-03-19T11:30:49.730 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.730 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.731 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.731 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:49.731 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842599+0000", 2024-03-19T11:30:49.731 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6000000000000003e-07 2024-03-19T11:30:49.731 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.731 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.731 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:49.731 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.843566+0000", 2024-03-19T11:30:49.731 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00096688700000000004 2024-03-19T11:30:49.732 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.732 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.732 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:49.732 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.843601+0000", 2024-03-19T11:30:49.732 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.502e-05 2024-03-19T11:30:49.732 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.732 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.732 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:49.732 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.843621+0000", 2024-03-19T11:30:49.733 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0305e-05 2024-03-19T11:30:49.733 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.733 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.733 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.733 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.843882+0000", 2024-03-19T11:30:49.733 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026027100000000002 2024-03-19T11:30:49.733 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.733 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.733 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:49.733 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.844022+0000", 2024-03-19T11:30:49.733 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014020100000000001 2024-03-19T11:30:49.733 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.733 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.733 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:49.734 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.865862+0000", 2024-03-19T11:30:49.734 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021839725000000001 2024-03-19T11:30:49.734 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.734 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.734 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:49.734 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.865974+0000", 2024-03-19T11:30:49.734 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011243099999999999 2024-03-19T11:30:49.734 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.734 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:49.734 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.734 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.734 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.734 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1209 4.8 4:1d58db11:::smithi03829856-18:head [write 645402~551297,stat] snapc 9=[9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:49.734 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:18.121657+0000", 2024-03-19T11:30:49.735 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 31.577533231, 2024-03-19T11:30:49.735 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.025074349999999999, 2024-03-19T11:30:49.735 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:49.735 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:49.735 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:49.735 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:49.735 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:49.735 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1209 2024-03-19T11:30:49.735 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.735 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:49.735 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.735 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:49.735 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.121657+0000", 2024-03-19T11:30:49.735 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.735 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.736 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.736 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:49.736 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.121657+0000", 2024-03-19T11:30:49.736 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.736 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.736 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.736 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:49.736 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.121661+0000", 2024-03-19T11:30:49.736 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0489999999999997e-06 2024-03-19T11:30:49.736 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.736 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.736 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:49.736 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123747+0000", 2024-03-19T11:30:49.736 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0020857720000000001 2024-03-19T11:30:49.737 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.737 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.737 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:49.737 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123779+0000", 2024-03-19T11:30:49.737 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2258000000000002e-05 2024-03-19T11:30:49.737 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.737 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.737 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:49.737 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123795+0000", 2024-03-19T11:30:49.737 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5177000000000001e-05 2024-03-19T11:30:49.737 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.738 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.738 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123949+0000", 2024-03-19T11:30:49.738 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015403699999999999 2024-03-19T11:30:49.738 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.739 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.739 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:49.739 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.124134+0000", 2024-03-19T11:30:49.739 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000185484 2024-03-19T11:30:49.739 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.739 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.739 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:49.739 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.124554+0000", 2024-03-19T11:30:49.740 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00042019699999999998 2024-03-19T11:30:49.740 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.740 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.740 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:49.740 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146267+0000", 2024-03-19T11:30:49.740 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021712838000000002 2024-03-19T11:30:49.740 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.740 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.740 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:49.740 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146424+0000", 2024-03-19T11:30:49.740 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015647299999999999 2024-03-19T11:30:49.740 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.740 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.740 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:49.741 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146597+0000", 2024-03-19T11:30:49.741 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017303699999999999 2024-03-19T11:30:49.741 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.741 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.741 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:49.741 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146732+0000", 2024-03-19T11:30:49.741 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013502800000000001 2024-03-19T11:30:49.741 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.741 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:49.741 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.742 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.742 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.742 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1210 4.8 4:1d58db11:::smithi03829856-18:head [write 1749569~492355] snapc 9=[9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:49.742 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:18.129257+0000", 2024-03-19T11:30:49.742 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 31.569933874, 2024-03-19T11:30:49.742 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018110444999999999, 2024-03-19T11:30:49.742 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:49.742 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:49.742 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:49.742 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:49.743 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:49.743 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1210 2024-03-19T11:30:49.743 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.743 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:49.746 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.746 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:49.746 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.129257+0000", 2024-03-19T11:30:49.746 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.746 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.746 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.746 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:49.747 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.129257+0000", 2024-03-19T11:30:49.747 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.747 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.747 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.747 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:49.747 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.129261+0000", 2024-03-19T11:30:49.747 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7119999999999999e-06 2024-03-19T11:30:49.747 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.747 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.747 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:49.747 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.129717+0000", 2024-03-19T11:30:49.747 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00045526300000000002 2024-03-19T11:30:49.747 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.747 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.747 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:49.748 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.129749+0000", 2024-03-19T11:30:49.748 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2700999999999997e-05 2024-03-19T11:30:49.748 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.748 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.748 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:49.748 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.129774+0000", 2024-03-19T11:30:49.748 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4592000000000001e-05 2024-03-19T11:30:49.748 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.748 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.748 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.748 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.134875+0000", 2024-03-19T11:30:49.748 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.005100477 2024-03-19T11:30:49.748 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.748 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.748 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:49.749 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.135035+0000", 2024-03-19T11:30:49.749 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016054200000000001 2024-03-19T11:30:49.749 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.749 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.749 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:49.749 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.135268+0000", 2024-03-19T11:30:49.749 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000233253 2024-03-19T11:30:49.749 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.749 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.749 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:49.749 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146872+0000", 2024-03-19T11:30:49.749 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011603732 2024-03-19T11:30:49.749 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.749 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.749 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:49.750 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.147103+0000", 2024-03-19T11:30:49.750 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023070299999999999 2024-03-19T11:30:49.750 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.750 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.750 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:49.750 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.147181+0000", 2024-03-19T11:30:49.750 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8567999999999997e-05 2024-03-19T11:30:49.750 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.750 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.750 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:49.750 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.147367+0000", 2024-03-19T11:30:49.750 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000185902 2024-03-19T11:30:49.750 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.750 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:49.750 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.751 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.751 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.751 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1259 4.1 e44/43 4:866f3dcc:::smithi03829856-31:head v 44'53, mlcod=0'0)", 2024-03-19T11:30:49.751 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:21.869109+0000", 2024-03-19T11:30:49.751 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.830081984, 2024-03-19T11:30:49.751 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014777017999999999, 2024-03-19T11:30:49.751 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:49.751 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:49.751 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:49.751 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.751 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:49.751 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.869109+0000", 2024-03-19T11:30:49.751 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.751 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.751 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.752 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:49.752 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.869109+0000", 2024-03-19T11:30:49.752 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.752 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.752 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.752 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:49.752 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.869110+0000", 2024-03-19T11:30:49.752 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0559999999999999e-06 2024-03-19T11:30:49.752 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.752 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.752 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:49.752 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.870408+0000", 2024-03-19T11:30:49.752 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012978499999999999 2024-03-19T11:30:49.752 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.752 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.753 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:49.753 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.870437+0000", 2024-03-19T11:30:49.753 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9638999999999999e-05 2024-03-19T11:30:49.753 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.753 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.753 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:49.753 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.870460+0000", 2024-03-19T11:30:49.753 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2517000000000001e-05 2024-03-19T11:30:49.753 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.753 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.753 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.753 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.870681+0000", 2024-03-19T11:30:49.753 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022102800000000001 2024-03-19T11:30:49.753 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.753 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.754 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:49.754 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.870735+0000", 2024-03-19T11:30:49.754 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4616e-05 2024-03-19T11:30:49.754 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.754 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.754 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:49.754 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.883773+0000", 2024-03-19T11:30:49.754 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013038088 2024-03-19T11:30:49.754 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.754 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.754 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:49.754 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.883886+0000", 2024-03-19T11:30:49.755 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000112224 2024-03-19T11:30:49.755 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.755 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:49.755 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.755 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.755 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.755 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1416 4.18 4:1dabc0f1:::smithi03829856-1:head [write 564735~613734,stat] snapc c=[c,b,a,6,5] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:49.755 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.949508+0000", 2024-03-19T11:30:49.755 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.749682654000001, 2024-03-19T11:30:49.755 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016253106, 2024-03-19T11:30:49.755 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:49.756 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:49.756 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:49.756 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:49.756 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:49.756 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1416 2024-03-19T11:30:49.756 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.756 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:49.756 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.756 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:49.756 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.949508+0000", 2024-03-19T11:30:49.756 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.756 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.756 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.756 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:49.757 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.949508+0000", 2024-03-19T11:30:49.757 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.757 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.757 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.757 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:49.757 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.949511+0000", 2024-03-19T11:30:49.757 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2370000000000001e-06 2024-03-19T11:30:49.757 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.757 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.757 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:49.757 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.950555+0000", 2024-03-19T11:30:49.757 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001043925 2024-03-19T11:30:49.757 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.757 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.758 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:49.758 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.950594+0000", 2024-03-19T11:30:49.758 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9351999999999999e-05 2024-03-19T11:30:49.758 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.758 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.758 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:49.758 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.950612+0000", 2024-03-19T11:30:49.758 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7524e-05 2024-03-19T11:30:49.758 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.758 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.758 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.759 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.950745+0000", 2024-03-19T11:30:49.759 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013291 2024-03-19T11:30:49.759 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.759 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:49.759 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.951003+0000", 2024-03-19T11:30:49.759 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025828400000000001 2024-03-19T11:30:49.759 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.759 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:49.759 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.951335+0000", 2024-03-19T11:30:49.759 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033130200000000001 2024-03-19T11:30:49.760 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.760 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.760 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:49.760 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.965379+0000", 2024-03-19T11:30:49.760 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014044581 2024-03-19T11:30:49.760 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.760 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.760 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:49.760 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.965502+0000", 2024-03-19T11:30:49.760 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012290099999999999 2024-03-19T11:30:49.760 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.761 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.761 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:49.761 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.965668+0000", 2024-03-19T11:30:49.761 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016619200000000001 2024-03-19T11:30:49.761 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.761 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.761 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:49.761 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.965761+0000", 2024-03-19T11:30:49.761 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2898000000000004e-05 2024-03-19T11:30:49.761 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.761 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:49.762 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.762 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.762 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.762 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1417 4.18 4:1dabc0f1:::smithi03829856-1:head [write 1828021~669034] snapc c=[c,b,a,6,5] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:49.762 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.958407+0000", 2024-03-19T11:30:49.762 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.740783253, 2024-03-19T11:30:49.762 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016467840000000001, 2024-03-19T11:30:49.762 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:49.762 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:49.762 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:49.762 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:49.763 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:49.763 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1417 2024-03-19T11:30:49.763 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.763 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:49.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.763 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:49.763 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.958407+0000", 2024-03-19T11:30:49.763 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.763 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.763 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:49.764 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.958407+0000", 2024-03-19T11:30:49.764 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.764 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.764 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:49.764 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.958410+0000", 2024-03-19T11:30:49.764 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9830000000000001e-06 2024-03-19T11:30:49.764 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.764 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:49.764 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.959442+0000", 2024-03-19T11:30:49.764 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001031957 2024-03-19T11:30:49.764 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.764 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:49.765 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.959475+0000", 2024-03-19T11:30:49.765 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3145999999999997e-05 2024-03-19T11:30:49.765 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.765 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.765 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:49.765 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.959494+0000", 2024-03-19T11:30:49.765 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8994000000000001e-05 2024-03-19T11:30:49.765 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.765 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.765 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.765 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.959717+0000", 2024-03-19T11:30:49.765 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000222272 2024-03-19T11:30:49.765 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.765 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.765 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:49.766 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.960015+0000", 2024-03-19T11:30:49.766 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029870599999999998 2024-03-19T11:30:49.766 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.766 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.766 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:49.766 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.960312+0000", 2024-03-19T11:30:49.766 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029640299999999999 2024-03-19T11:30:49.766 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.766 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.766 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:49.766 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.966435+0000", 2024-03-19T11:30:49.766 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061234999999999996 2024-03-19T11:30:49.766 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.766 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.766 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:49.767 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.974556+0000", 2024-03-19T11:30:49.767 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.008120759 2024-03-19T11:30:49.767 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.767 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:49.767 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.974729+0000", 2024-03-19T11:30:49.767 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000173191 2024-03-19T11:30:49.767 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.767 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:49.767 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.974875+0000", 2024-03-19T11:30:49.767 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014592899999999999 2024-03-19T11:30:49.767 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.767 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:49.767 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.768 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.768 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.768 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1419 4.18 4:1dabc0f1:::smithi03829856-1:head [setxattr _header (56) in=7b,truncate 3764735] snapc c=[c,b,a,6,5] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:49.768 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.968061+0000", 2024-03-19T11:30:49.768 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.731129394, 2024-03-19T11:30:49.768 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012422488000000001, 2024-03-19T11:30:49.768 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:49.768 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:49.768 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:49.768 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:49.768 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:49.768 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1419 2024-03-19T11:30:49.768 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.768 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:49.768 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.769 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:49.769 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968061+0000", 2024-03-19T11:30:49.769 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.769 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.769 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.769 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:49.769 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968061+0000", 2024-03-19T11:30:49.769 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.769 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.769 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.769 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:49.769 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968063+0000", 2024-03-19T11:30:49.769 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.767e-06 2024-03-19T11:30:49.769 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.769 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.769 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:49.770 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968074+0000", 2024-03-19T11:30:49.770 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0525e-05 2024-03-19T11:30:49.770 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.770 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.770 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:49.770 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968096+0000", 2024-03-19T11:30:49.770 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2926999999999999e-05 2024-03-19T11:30:49.770 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.770 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.770 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:49.770 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968113+0000", 2024-03-19T11:30:49.770 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6665000000000001e-05 2024-03-19T11:30:49.770 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.770 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.771 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.771 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.973218+0000", 2024-03-19T11:30:49.771 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0051044920000000004 2024-03-19T11:30:49.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.771 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:49.771 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.973511+0000", 2024-03-19T11:30:49.771 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029303899999999998 2024-03-19T11:30:49.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.771 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:49.771 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.973931+0000", 2024-03-19T11:30:49.771 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00041991400000000002 2024-03-19T11:30:49.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.772 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:49.772 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.977029+0000", 2024-03-19T11:30:49.772 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0030983040000000001 2024-03-19T11:30:49.772 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.772 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.772 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:49.772 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.980091+0000", 2024-03-19T11:30:49.772 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003061851 2024-03-19T11:30:49.772 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.772 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.772 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:49.772 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.980285+0000", 2024-03-19T11:30:49.772 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000194279 2024-03-19T11:30:49.772 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.772 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.773 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:49.773 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.980484+0000", 2024-03-19T11:30:49.773 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019872499999999999 2024-03-19T11:30:49.773 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.773 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:49.773 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.773 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.773 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.773 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1420 4.18 4:1dabc0f1:::smithi03829856-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:49.773 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.968138+0000", 2024-03-19T11:30:49.773 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.731052183999999, 2024-03-19T11:30:49.773 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013109305999999999, 2024-03-19T11:30:49.773 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:49.773 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:49.774 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:49.774 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:49.774 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:49.774 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1420 2024-03-19T11:30:49.774 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.774 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:49.774 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.774 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:49.774 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968138+0000", 2024-03-19T11:30:49.774 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.774 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.774 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.774 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:49.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968138+0000", 2024-03-19T11:30:49.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:49.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968140+0000", 2024-03-19T11:30:49.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4139999999999999e-06 2024-03-19T11:30:49.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:49.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968147+0000", 2024-03-19T11:30:49.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.815e-06 2024-03-19T11:30:49.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.776 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:49.776 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968167+0000", 2024-03-19T11:30:49.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0018e-05 2024-03-19T11:30:49.776 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.776 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.776 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:49.776 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.968182+0000", 2024-03-19T11:30:49.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5475e-05 2024-03-19T11:30:49.776 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.776 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.776 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.776 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.974950+0000", 2024-03-19T11:30:49.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.006767856 2024-03-19T11:30:49.776 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.777 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.777 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-03-19T11:30:49.777 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.975095+0000", 2024-03-19T11:30:49.777 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014501499999999999 2024-03-19T11:30:49.777 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.777 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.777 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.777 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.980614+0000", 2024-03-19T11:30:49.777 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0055184789999999997 2024-03-19T11:30:49.777 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.777 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.777 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:49.777 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.980912+0000", 2024-03-19T11:30:49.778 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029858199999999998 2024-03-19T11:30:49.778 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.778 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.778 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:49.778 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.981248+0000", 2024-03-19T11:30:49.778 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033565199999999998 2024-03-19T11:30:49.778 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.778 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:49.778 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.778 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.778 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.778 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1443 4.2 4:43b32242:::smithi03829856-22:head [read 0~2352947,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:49.779 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.979417+0000", 2024-03-19T11:30:49.779 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.719773558, 2024-03-19T11:30:49.779 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.96719988700000004, 2024-03-19T11:30:49.779 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:49.779 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:49.779 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:49.779 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:49.779 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:49.779 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1443 2024-03-19T11:30:49.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.779 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:49.779 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:49.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979417+0000", 2024-03-19T11:30:49.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.780 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:49.780 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979417+0000", 2024-03-19T11:30:49.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.780 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:49.780 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979420+0000", 2024-03-19T11:30:49.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.366e-06 2024-03-19T11:30:49.781 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.781 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.781 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:49.781 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979437+0000", 2024-03-19T11:30:49.781 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6192e-05 2024-03-19T11:30:49.781 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.781 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.781 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:49.781 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979474+0000", 2024-03-19T11:30:49.781 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7601000000000001e-05 2024-03-19T11:30:49.781 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.781 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.781 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:49.781 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979496+0000", 2024-03-19T11:30:49.782 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1506999999999999e-05 2024-03-19T11:30:49.782 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.782 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.782 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.782 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979712+0000", 2024-03-19T11:30:49.782 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021641299999999999 2024-03-19T11:30:49.782 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.782 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.782 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:30:49.783 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979746+0000", 2024-03-19T11:30:49.783 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3386000000000002e-05 2024-03-19T11:30:49.783 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.783 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.783 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.931552+0000", 2024-03-19T11:30:49.783 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.95180659899999998 2024-03-19T11:30:49.783 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.783 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-03-19T11:30:49.783 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.934681+0000", 2024-03-19T11:30:49.783 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0031292289999999999 2024-03-19T11:30:49.783 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.784 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.784 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.942701+0000", 2024-03-19T11:30:49.784 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0080198809999999995 2024-03-19T11:30:49.784 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.784 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.784 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:49.784 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.942839+0000", 2024-03-19T11:30:49.784 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013816999999999999 2024-03-19T11:30:49.784 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.784 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.784 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:49.784 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.946617+0000", 2024-03-19T11:30:49.784 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003777543 2024-03-19T11:30:49.784 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.784 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:49.785 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.785 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.785 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.785 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1444 4.2 4:43b32242:::smithi03829856-22:head [read 0~2352947,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:49.785 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.979553+0000", 2024-03-19T11:30:49.785 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.719637854999998, 2024-03-19T11:30:49.785 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.97015945400000003, 2024-03-19T11:30:49.785 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:49.785 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:49.785 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:49.785 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:49.785 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:49.785 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1444 2024-03-19T11:30:49.785 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.786 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:49.786 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.786 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:49.786 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979553+0000", 2024-03-19T11:30:49.786 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.786 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.786 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.786 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:49.786 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979553+0000", 2024-03-19T11:30:49.786 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.786 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.786 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.786 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:49.786 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979554+0000", 2024-03-19T11:30:49.787 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.403e-06 2024-03-19T11:30:49.787 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.787 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.787 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:49.787 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979564+0000", 2024-03-19T11:30:49.787 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0032e-05 2024-03-19T11:30:49.787 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.787 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.787 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:49.787 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979579+0000", 2024-03-19T11:30:49.787 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4875000000000001e-05 2024-03-19T11:30:49.787 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.787 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.787 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:49.787 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979595+0000", 2024-03-19T11:30:49.788 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6300999999999999e-05 2024-03-19T11:30:49.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.788 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.788 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979874+0000", 2024-03-19T11:30:49.788 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000279065 2024-03-19T11:30:49.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.788 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:30:49.788 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979904+0000", 2024-03-19T11:30:49.788 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.94e-05 2024-03-19T11:30:49.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.788 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.789 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.934781+0000", 2024-03-19T11:30:49.789 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.95487744900000004 2024-03-19T11:30:49.789 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.789 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.789 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-03-19T11:30:49.789 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.934946+0000", 2024-03-19T11:30:49.789 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016485999999999999 2024-03-19T11:30:49.789 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.789 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.789 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.789 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.946829+0000", 2024-03-19T11:30:49.789 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011882482999999999 2024-03-19T11:30:49.789 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.789 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.789 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:49.790 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.946989+0000", 2024-03-19T11:30:49.790 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000160421 2024-03-19T11:30:49.790 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.790 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.790 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:49.790 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.949712+0000", 2024-03-19T11:30:49.790 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002723165 2024-03-19T11:30:49.790 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.790 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:49.791 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.791 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.791 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.791 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1445 4.2 4:43b32242:::smithi03829856-22:head [read 0~2352947,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:49.791 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.979653+0000", 2024-03-19T11:30:49.791 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.719537723999998, 2024-03-19T11:30:49.791 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.97249507999999996, 2024-03-19T11:30:49.791 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:49.791 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:49.791 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:49.791 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:49.792 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:49.792 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1445 2024-03-19T11:30:49.792 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.792 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:49.792 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.792 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:49.792 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979653+0000", 2024-03-19T11:30:49.793 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.793 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.793 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:49.793 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979653+0000", 2024-03-19T11:30:49.793 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.793 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.793 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:49.793 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979654+0000", 2024-03-19T11:30:49.794 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.372e-06 2024-03-19T11:30:49.794 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.794 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.794 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:49.794 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979662+0000", 2024-03-19T11:30:49.794 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9959999999999993e-06 2024-03-19T11:30:49.794 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.794 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.794 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:49.794 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979698+0000", 2024-03-19T11:30:49.794 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5763000000000003e-05 2024-03-19T11:30:49.794 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.794 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.794 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:49.795 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979714+0000", 2024-03-19T11:30:49.795 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6283999999999999e-05 2024-03-19T11:30:49.795 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.795 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.795 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979990+0000", 2024-03-19T11:30:49.795 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000275829 2024-03-19T11:30:49.795 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.795 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:30:49.795 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.980018+0000", 2024-03-19T11:30:49.795 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8082999999999999e-05 2024-03-19T11:30:49.795 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.795 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.796 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.935620+0000", 2024-03-19T11:30:49.796 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.95560133800000002 2024-03-19T11:30:49.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.796 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.796 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-03-19T11:30:49.796 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.935795+0000", 2024-03-19T11:30:49.796 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000175656 2024-03-19T11:30:49.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.796 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.796 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.796 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.949835+0000", 2024-03-19T11:30:49.796 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014040083 2024-03-19T11:30:49.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.796 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.796 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:49.797 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.950002+0000", 2024-03-19T11:30:49.797 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016673 2024-03-19T11:30:49.797 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.797 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.797 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:49.797 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.952148+0000", 2024-03-19T11:30:49.797 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021459460000000001 2024-03-19T11:30:49.797 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.797 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:49.797 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.797 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.797 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.797 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1449 4.18 4:1d58db11:::smithi03829856-18:head [write 581542~688983,stat] snapc c=[c,b,a,6,5] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:49.797 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.989354+0000", 2024-03-19T11:30:49.798 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.709836271, 2024-03-19T11:30:49.798 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019237318, 2024-03-19T11:30:49.798 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:49.798 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:49.798 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:49.798 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:49.798 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:49.798 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1449 2024-03-19T11:30:49.798 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.798 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:49.798 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.798 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:49.798 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.989354+0000", 2024-03-19T11:30:49.798 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.799 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.799 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.799 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:49.799 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.989354+0000", 2024-03-19T11:30:49.799 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.799 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.799 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.799 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:49.799 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.989357+0000", 2024-03-19T11:30:49.799 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2740000000000002e-06 2024-03-19T11:30:49.799 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.799 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.799 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:49.799 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.990109+0000", 2024-03-19T11:30:49.800 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00075226500000000001 2024-03-19T11:30:49.800 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.800 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:49.800 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.990142+0000", 2024-03-19T11:30:49.800 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3328999999999999e-05 2024-03-19T11:30:49.800 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.800 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:49.800 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.990168+0000", 2024-03-19T11:30:49.800 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6016999999999998e-05 2024-03-19T11:30:49.800 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.800 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.801 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.990291+0000", 2024-03-19T11:30:49.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000122607 2024-03-19T11:30:49.801 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.801 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.801 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:49.801 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.990461+0000", 2024-03-19T11:30:49.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000169743 2024-03-19T11:30:49.801 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.801 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.801 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:49.801 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.990792+0000", 2024-03-19T11:30:49.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033135100000000002 2024-03-19T11:30:49.801 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.802 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.802 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:49.802 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.008238+0000", 2024-03-19T11:30:49.802 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017445591999999999 2024-03-19T11:30:49.802 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.802 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.802 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:49.802 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.008345+0000", 2024-03-19T11:30:49.802 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010762 2024-03-19T11:30:49.802 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.802 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.802 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:49.803 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.008436+0000", 2024-03-19T11:30:49.803 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1116e-05 2024-03-19T11:30:49.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.803 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:49.803 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.008592+0000", 2024-03-19T11:30:49.803 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015540399999999999 2024-03-19T11:30:49.803 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.803 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:49.803 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.804 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1450 4.18 4:1d58db11:::smithi03829856-18:head [write 1886503~412692] snapc c=[c,b,a,6,5] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:49.804 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.992973+0000", 2024-03-19T11:30:49.804 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.706217502000001, 2024-03-19T11:30:49.804 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016912580999999999, 2024-03-19T11:30:49.804 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:49.804 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:49.804 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:49.805 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:49.805 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:49.805 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1450 2024-03-19T11:30:49.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.805 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:49.805 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.805 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:49.805 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.992973+0000", 2024-03-19T11:30:49.806 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.806 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.806 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.806 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:49.806 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.992973+0000", 2024-03-19T11:30:49.806 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.807 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.807 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:49.807 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.992976+0000", 2024-03-19T11:30:49.807 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0060000000000001e-06 2024-03-19T11:30:49.807 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.808 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:49.808 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.993655+0000", 2024-03-19T11:30:49.808 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00067901099999999998 2024-03-19T11:30:49.808 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.808 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.808 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:49.808 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.993691+0000", 2024-03-19T11:30:49.809 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5546000000000002e-05 2024-03-19T11:30:49.809 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.809 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.809 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:49.809 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.993713+0000", 2024-03-19T11:30:49.809 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2277e-05 2024-03-19T11:30:49.809 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.809 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.810 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.810 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.997961+0000", 2024-03-19T11:30:49.810 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0042479659999999997 2024-03-19T11:30:49.810 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.810 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.810 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:49.810 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.998130+0000", 2024-03-19T11:30:49.810 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016908900000000001 2024-03-19T11:30:49.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.811 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:49.811 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.998397+0000", 2024-03-19T11:30:49.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026707599999999999 2024-03-19T11:30:49.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.812 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:49.812 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.008811+0000", 2024-03-19T11:30:49.812 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010413751000000001 2024-03-19T11:30:49.812 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.812 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.812 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:49.812 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.009539+0000", 2024-03-19T11:30:49.813 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00072811499999999999 2024-03-19T11:30:49.813 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.813 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.813 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:49.813 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.009711+0000", 2024-03-19T11:30:49.813 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017196900000000001 2024-03-19T11:30:49.813 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.814 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.814 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:49.814 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.009886+0000", 2024-03-19T11:30:49.814 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017477500000000001 2024-03-19T11:30:49.814 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.814 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:49.814 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.814 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.814 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.815 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1637 4.7 4:e41d47fb:::smithi03829856-14:head [write 484068~792524,stat] snapc 10=[10,f,e,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-03-19T11:30:49.815 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:35.672719+0000", 2024-03-19T11:30:49.815 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.026471274, 2024-03-19T11:30:49.815 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012961324, 2024-03-19T11:30:49.815 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:49.815 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:49.815 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:49.815 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:49.815 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:49.815 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1637 2024-03-19T11:30:49.815 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.815 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:49.815 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:49.816 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.672719+0000", 2024-03-19T11:30:49.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.816 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:49.816 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.672719+0000", 2024-03-19T11:30:49.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:49.816 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:49.816 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.672722+0000", 2024-03-19T11:30:49.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0690000000000002e-06 2024-03-19T11:30:49.816 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:49.817 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.673743+0000", 2024-03-19T11:30:49.817 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0010203899999999999 2024-03-19T11:30:49.817 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.817 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.817 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:49.817 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.673774+0000", 2024-03-19T11:30:49.817 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1049e-05 2024-03-19T11:30:49.817 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.817 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.817 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:49.817 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.673789+0000", 2024-03-19T11:30:49.817 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4729000000000001e-05 2024-03-19T11:30:49.817 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.817 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.818 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:49.818 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.673921+0000", 2024-03-19T11:30:49.818 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013277100000000001 2024-03-19T11:30:49.818 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.818 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.818 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:49.818 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.674053+0000", 2024-03-19T11:30:49.818 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000131975 2024-03-19T11:30:49.818 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.818 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.818 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:49.818 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.674430+0000", 2024-03-19T11:30:49.818 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000377031 2024-03-19T11:30:49.819 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.819 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:49.819 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.684295+0000", 2024-03-19T11:30:49.819 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0098643120000000001 2024-03-19T11:30:49.819 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.819 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:49.820 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.685359+0000", 2024-03-19T11:30:49.820 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001064111 2024-03-19T11:30:49.820 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.820 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.820 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:49.820 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.685532+0000", 2024-03-19T11:30:49.821 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017305700000000001 2024-03-19T11:30:49.821 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:49.821 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:49.821 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:49.821 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:35.685681+0000", 2024-03-19T11:30:49.821 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014883 2024-03-19T11:30:49.821 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.822 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:49.822 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.822 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:49.822 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:49.822 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:49.824 INFO:tasks.rados.rados.0.smithi038.stdout:1087: finishing set_chunk to oid 20 2024-03-19T11:30:49.824 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 20 set_chunk returned error code -95 offset: 246784 length: 502784 2024-03-19T11:30:49.824 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 20 v 0 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-03-19T11:30:49.824 INFO:tasks.rados.rados.0.smithi038.stdout:1085: expect (ObjNum 163 snap 17 seq_num 163) 2024-03-19T11:30:49.825 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:49.834+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:49.826 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-19T11:30:49.893 INFO:tasks.rados.rados.0.smithi038.stdout:1088: expect (ObjNum 148 snap 15 seq_num 148) 2024-03-19T11:30:49.925 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:49.932+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:49.952 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:49.952 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:49.952 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:30:49.953 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:49.966 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-19T11:30:50.025 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:50.034+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:50.051 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-03-19T11:30:50.126 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:50.134+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:50.129 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:50.129 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:50.129 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:30:50.129 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:30:50.129 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:50.144 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-19T11:30:50.227 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:50.236+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:50.269 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:50.269 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:30:50.269 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:30:50.270 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:30:50.270 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.270 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:29 4.5 4:ac9cd3a3:::smithi03829856-8:head [write 517649~526743,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:30:50.270 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:04.772849+0000", 2024-03-19T11:30:50.270 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 45.503644024000003, 2024-03-19T11:30:50.270 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014076821, 2024-03-19T11:30:50.270 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:50.270 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:50.270 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:50.270 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:50.270 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:50.270 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 29 2024-03-19T11:30:50.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.271 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:50.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.271 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:50.271 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.772849+0000", 2024-03-19T11:30:50.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.271 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:50.271 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.772849+0000", 2024-03-19T11:30:50.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.271 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:50.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.772852+0000", 2024-03-19T11:30:50.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.79e-06 2024-03-19T11:30:50.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:50.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.773783+0000", 2024-03-19T11:30:50.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00093104199999999998 2024-03-19T11:30:50.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:50.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.773810+0000", 2024-03-19T11:30:50.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.669e-05 2024-03-19T11:30:50.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:50.273 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.773826+0000", 2024-03-19T11:30:50.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5739999999999998e-05 2024-03-19T11:30:50.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:50.273 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.773954+0000", 2024-03-19T11:30:50.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012796300000000001 2024-03-19T11:30:50.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op must wait for map", 2024-03-19T11:30:50.274 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.773965+0000", 2024-03-19T11:30:50.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1453e-05 2024-03-19T11:30:50.274 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.274 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.274 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:50.274 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.777723+0000", 2024-03-19T11:30:50.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003757944 2024-03-19T11:30:50.274 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.274 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.274 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:50.274 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778138+0000", 2024-03-19T11:30:50.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00041454699999999997 2024-03-19T11:30:50.274 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:30:50.275 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778409+0000", 2024-03-19T11:30:50.275 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027111700000000003 2024-03-19T11:30:50.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:50.275 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.786631+0000", 2024-03-19T11:30:50.275 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0082221470000000008 2024-03-19T11:30:50.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:50.275 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.786668+0000", 2024-03-19T11:30:50.275 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.718e-05 2024-03-19T11:30:50.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:50.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.786845+0000", 2024-03-19T11:30:50.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000177219 2024-03-19T11:30:50.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:50.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.786926+0000", 2024-03-19T11:30:50.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0989000000000007e-05 2024-03-19T11:30:50.276 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.276 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:50.276 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.276 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:30 4.5 4:ac9cd3a3:::smithi03829856-8:head [write 1776668~340981] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:30:50.277 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:04.777922+0000", 2024-03-19T11:30:50.277 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 45.498571394000002, 2024-03-19T11:30:50.277 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015469894, 2024-03-19T11:30:50.277 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:50.277 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:50.277 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:50.277 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:50.277 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:50.277 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 30 2024-03-19T11:30:50.277 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.277 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:50.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.277 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:50.277 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.777922+0000", 2024-03-19T11:30:50.277 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.278 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.278 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.278 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:50.278 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.777922+0000", 2024-03-19T11:30:50.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.278 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.278 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.278 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:50.278 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.777924+0000", 2024-03-19T11:30:50.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3999999999999999e-06 2024-03-19T11:30:50.278 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.278 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.278 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:50.278 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778461+0000", 2024-03-19T11:30:50.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00053698299999999999 2024-03-19T11:30:50.279 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.279 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:50.279 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778487+0000", 2024-03-19T11:30:50.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5947000000000001e-05 2024-03-19T11:30:50.279 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.279 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:50.279 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778502+0000", 2024-03-19T11:30:50.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4256e-05 2024-03-19T11:30:50.279 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.279 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:50.279 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.781978+0000", 2024-03-19T11:30:50.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00347693 2024-03-19T11:30:50.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.280 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:50.280 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.782196+0000", 2024-03-19T11:30:50.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021726499999999999 2024-03-19T11:30:50.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.280 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:30:50.280 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.782572+0000", 2024-03-19T11:30:50.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00037635000000000001 2024-03-19T11:30:50.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.280 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:50.280 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.786960+0000", 2024-03-19T11:30:50.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043882130000000002 2024-03-19T11:30:50.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:50.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793113+0000", 2024-03-19T11:30:50.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061530760000000004 2024-03-19T11:30:50.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:50.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793276+0000", 2024-03-19T11:30:50.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000162652 2024-03-19T11:30:50.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:50.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793392+0000", 2024-03-19T11:30:50.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115822 2024-03-19T11:30:50.282 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.282 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:50.282 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.282 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:31 4.5 4:ac9cd3a3:::smithi03829856-8:head [setxattr _header (55) in=7b,truncate 2117649] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:30:50.282 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:04.778533+0000", 2024-03-19T11:30:50.282 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 45.497960259000003, 2024-03-19T11:30:50.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015182533, 2024-03-19T11:30:50.282 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:50.282 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:50.282 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:50.283 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:50.283 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:50.283 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 31 2024-03-19T11:30:50.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.283 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:50.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:50.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778533+0000", 2024-03-19T11:30:50.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:50.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778533+0000", 2024-03-19T11:30:50.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.284 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:50.284 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778534+0000", 2024-03-19T11:30:50.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1069999999999999e-06 2024-03-19T11:30:50.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.284 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:50.284 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778541+0000", 2024-03-19T11:30:50.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2529999999999998e-06 2024-03-19T11:30:50.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.284 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:50.284 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778558+0000", 2024-03-19T11:30:50.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6290000000000002e-05 2024-03-19T11:30:50.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:50.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778568+0000", 2024-03-19T11:30:50.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0360000000000001e-05 2024-03-19T11:30:50.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:50.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.785511+0000", 2024-03-19T11:30:50.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00694306 2024-03-19T11:30:50.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:50.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.785710+0000", 2024-03-19T11:30:50.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019837199999999999 2024-03-19T11:30:50.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:30:50.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.785929+0000", 2024-03-19T11:30:50.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021960799999999999 2024-03-19T11:30:50.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:50.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.787281+0000", 2024-03-19T11:30:50.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001352313 2024-03-19T11:30:50.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.287 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:50.287 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793492+0000", 2024-03-19T11:30:50.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062107760000000003 2024-03-19T11:30:50.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.287 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:50.287 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793578+0000", 2024-03-19T11:30:50.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5711999999999995e-05 2024-03-19T11:30:50.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.287 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:50.287 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793716+0000", 2024-03-19T11:30:50.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000137682 2024-03-19T11:30:50.287 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.287 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:50.288 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.288 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:32 4.5 4:ac9cd3a3:::smithi03829856-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:30:50.288 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:04.778582+0000", 2024-03-19T11:30:50.288 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 45.497911745000003, 2024-03-19T11:30:50.288 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015761827999999999, 2024-03-19T11:30:50.288 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:50.288 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:50.288 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:50.288 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:50.288 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:50.288 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 32 2024-03-19T11:30:50.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.289 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:50.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:50.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778582+0000", 2024-03-19T11:30:50.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:50.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778582+0000", 2024-03-19T11:30:50.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:50.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778583+0000", 2024-03-19T11:30:50.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.026e-06 2024-03-19T11:30:50.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:50.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778587+0000", 2024-03-19T11:30:50.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7870000000000001e-06 2024-03-19T11:30:50.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:50.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778599+0000", 2024-03-19T11:30:50.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1623e-05 2024-03-19T11:30:50.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:50.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778608+0000", 2024-03-19T11:30:50.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7399999999999993e-06 2024-03-19T11:30:50.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:50.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.787009+0000", 2024-03-19T11:30:50.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0084010490000000007 2024-03-19T11:30:50.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-03-19T11:30:50.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.787202+0000", 2024-03-19T11:30:50.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019309500000000001 2024-03-19T11:30:50.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:50.292 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793832+0000", 2024-03-19T11:30:50.292 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.006630052 2024-03-19T11:30:50.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.292 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:50.292 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.794048+0000", 2024-03-19T11:30:50.292 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000215505 2024-03-19T11:30:50.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.292 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:50.292 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.794343+0000", 2024-03-19T11:30:50.292 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029595100000000003 2024-03-19T11:30:50.292 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.292 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:50.293 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.293 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:856 4.5 4:ac9cd3a3:::smithi03829856-8:head [write 608420~529411] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:50.293 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.781425+0000", 2024-03-19T11:30:50.293 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 41.495068559000003, 2024-03-19T11:30:50.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014634677, 2024-03-19T11:30:50.293 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:50.293 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:50.293 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:50.293 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:50.293 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:50.293 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 856 2024-03-19T11:30:50.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.293 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:50.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.294 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:50.294 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.781425+0000", 2024-03-19T11:30:50.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.294 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:50.294 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.781425+0000", 2024-03-19T11:30:50.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.294 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:50.294 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.781428+0000", 2024-03-19T11:30:50.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.534e-06 2024-03-19T11:30:50.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:50.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.783096+0000", 2024-03-19T11:30:50.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001667897 2024-03-19T11:30:50.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:50.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.783138+0000", 2024-03-19T11:30:50.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1946000000000001e-05 2024-03-19T11:30:50.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:50.296 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.783163+0000", 2024-03-19T11:30:50.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4443e-05 2024-03-19T11:30:50.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.296 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:50.296 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.783333+0000", 2024-03-19T11:30:50.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017027899999999999 2024-03-19T11:30:50.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.296 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:50.296 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.783503+0000", 2024-03-19T11:30:50.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000170242 2024-03-19T11:30:50.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:30:50.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.783917+0000", 2024-03-19T11:30:50.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00041371000000000002 2024-03-19T11:30:50.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:50.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.795709+0000", 2024-03-19T11:30:50.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011791693000000001 2024-03-19T11:30:50.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:50.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.795850+0000", 2024-03-19T11:30:50.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000141695 2024-03-19T11:30:50.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:50.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.795940+0000", 2024-03-19T11:30:50.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9705999999999998e-05 2024-03-19T11:30:50.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:50.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.796059+0000", 2024-03-19T11:30:50.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011953199999999999 2024-03-19T11:30:50.299 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.299 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:50.299 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.299 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:891 4.5 4:af9d0d56:::smithi03829856-20:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:50.299 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:09.286410+0000", 2024-03-19T11:30:50.299 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 40.990083413000001, 2024-03-19T11:30:50.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035191405000000002, 2024-03-19T11:30:50.299 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:50.300 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:50.300 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:50.300 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:50.300 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:50.300 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 891 2024-03-19T11:30:50.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.300 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:50.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.300 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:50.300 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286410+0000", 2024-03-19T11:30:50.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.300 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:50.300 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286410+0000", 2024-03-19T11:30:50.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.301 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:50.301 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286413+0000", 2024-03-19T11:30:50.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2660000000000002e-06 2024-03-19T11:30:50.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.301 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:50.301 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286426+0000", 2024-03-19T11:30:50.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.258e-05 2024-03-19T11:30:50.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.301 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:50.301 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286460+0000", 2024-03-19T11:30:50.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3943999999999999e-05 2024-03-19T11:30:50.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:50.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286477+0000", 2024-03-19T11:30:50.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6799999999999998e-05 2024-03-19T11:30:50.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:50.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286605+0000", 2024-03-19T11:30:50.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000128892 2024-03-19T11:30:50.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:50.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286769+0000", 2024-03-19T11:30:50.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016359799999999999 2024-03-19T11:30:50.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:50.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.321601+0000", 2024-03-19T11:30:50.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.034832324999999997 2024-03-19T11:30:50.303 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.303 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:50.303 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.303 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:993 4.e e34/21 4:72163e4c:::smithi03829856-39:head v 34'17, mlcod=28'14)", 2024-03-19T11:30:50.303 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:12.367897+0000", 2024-03-19T11:30:50.304 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 37.908596867, 2024-03-19T11:30:50.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012765165, 2024-03-19T11:30:50.304 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:50.304 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:50.304 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:50.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:50.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.367897+0000", 2024-03-19T11:30:50.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:50.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.367897+0000", 2024-03-19T11:30:50.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.305 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:50.305 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.367897+0000", 2024-03-19T11:30:50.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7000000000000003e-07 2024-03-19T11:30:50.305 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.305 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:50.305 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.368454+0000", 2024-03-19T11:30:50.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00055647599999999998 2024-03-19T11:30:50.305 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.305 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:50.305 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.368532+0000", 2024-03-19T11:30:50.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8479000000000005e-05 2024-03-19T11:30:50.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:50.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.368554+0000", 2024-03-19T11:30:50.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.156e-05 2024-03-19T11:30:50.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:50.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.368742+0000", 2024-03-19T11:30:50.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000187846 2024-03-19T11:30:50.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:50.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.368782+0000", 2024-03-19T11:30:50.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9836000000000003e-05 2024-03-19T11:30:50.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:50.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.380340+0000", 2024-03-19T11:30:50.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011558847000000001 2024-03-19T11:30:50.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:50.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.380662+0000", 2024-03-19T11:30:50.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00032125099999999999 2024-03-19T11:30:50.308 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.308 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:50.308 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.308 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1015 4.5 4:af9d0d56:::smithi03829856-20:head [tier-promote] snapc 6=[6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:50.308 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:12.761855+0000", 2024-03-19T11:30:50.308 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 37.514637886999999, 2024-03-19T11:30:50.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026760430000000002, 2024-03-19T11:30:50.308 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:50.308 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:50.308 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:50.308 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:50.308 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:50.309 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1015 2024-03-19T11:30:50.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.309 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:50.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.309 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:50.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761855+0000", 2024-03-19T11:30:50.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.309 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:50.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761855+0000", 2024-03-19T11:30:50.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.309 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:50.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761858+0000", 2024-03-19T11:30:50.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8140000000000001e-06 2024-03-19T11:30:50.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:50.310 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761872+0000", 2024-03-19T11:30:50.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3813e-05 2024-03-19T11:30:50.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:50.310 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761905+0000", 2024-03-19T11:30:50.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2642999999999999e-05 2024-03-19T11:30:50.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:50.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761925+0000", 2024-03-19T11:30:50.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0489e-05 2024-03-19T11:30:50.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:50.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.762070+0000", 2024-03-19T11:30:50.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000145153 2024-03-19T11:30:50.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:50.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.762365+0000", 2024-03-19T11:30:50.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029482100000000002 2024-03-19T11:30:50.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:50.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.788616+0000", 2024-03-19T11:30:50.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026250697 2024-03-19T11:30:50.312 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.312 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:50.312 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.312 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1121 4.5 4:aa3303a5:::smithi03829856-17:head [tier-promote] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:50.312 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:14.164281+0000", 2024-03-19T11:30:50.312 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 36.112212642000003, 2024-03-19T11:30:50.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023896435000000001, 2024-03-19T11:30:50.312 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:50.312 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:50.313 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:50.313 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:50.313 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:50.313 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1121 2024-03-19T11:30:50.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.313 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:50.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:50.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164281+0000", 2024-03-19T11:30:50.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:50.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164281+0000", 2024-03-19T11:30:50.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.314 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:50.314 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164283+0000", 2024-03-19T11:30:50.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8500000000000001e-06 2024-03-19T11:30:50.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.314 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:50.314 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164291+0000", 2024-03-19T11:30:50.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.14e-06 2024-03-19T11:30:50.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.314 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:50.314 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164313+0000", 2024-03-19T11:30:50.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.207e-05 2024-03-19T11:30:50.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:50.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164328+0000", 2024-03-19T11:30:50.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5403000000000001e-05 2024-03-19T11:30:50.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:50.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.167981+0000", 2024-03-19T11:30:50.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036529879999999998 2024-03-19T11:30:50.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:50.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.168274+0000", 2024-03-19T11:30:50.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029313100000000002 2024-03-19T11:30:50.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:50.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.188177+0000", 2024-03-19T11:30:50.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019902853000000002 2024-03-19T11:30:50.316 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.316 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:50.316 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.316 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1154 4.5 4:a0a50d04:::smithi03829856-13:8 [sparse-read 0~1151947] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-03-19T11:30:50.316 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.827529+0000", 2024-03-19T11:30:50.316 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 33.448964224000001, 2024-03-19T11:30:50.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013620382, 2024-03-19T11:30:50.317 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:50.317 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:50.317 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:50.317 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:50.317 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:50.317 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1154 2024-03-19T11:30:50.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.317 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:50.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:50.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827529+0000", 2024-03-19T11:30:50.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:50.318 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827529+0000", 2024-03-19T11:30:50.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.318 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:50.318 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827530+0000", 2024-03-19T11:30:50.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7700000000000004e-07 2024-03-19T11:30:50.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.318 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:50.318 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827533+0000", 2024-03-19T11:30:50.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4520000000000002e-06 2024-03-19T11:30:50.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.318 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:50.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827543+0000", 2024-03-19T11:30:50.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0035e-05 2024-03-19T11:30:50.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:50.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827551+0000", 2024-03-19T11:30:50.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7049999999999999e-06 2024-03-19T11:30:50.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:50.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.839931+0000", 2024-03-19T11:30:50.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012379602999999999 2024-03-19T11:30:50.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:50.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.840290+0000", 2024-03-19T11:30:50.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000359031 2024-03-19T11:30:50.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:50.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841150+0000", 2024-03-19T11:30:50.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00085977899999999997 2024-03-19T11:30:50.320 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.320 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:50.320 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.320 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1302 4.15 4:aa3303a5:::smithi03829856-17:head [write 413472~620140,stat] snapc b=[b,a,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-03-19T11:30:50.320 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:22.966426+0000", 2024-03-19T11:30:50.320 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.310066909, 2024-03-19T11:30:50.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018043513000000001, 2024-03-19T11:30:50.321 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:50.321 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:50.321 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:50.321 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:50.321 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:50.321 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1302 2024-03-19T11:30:50.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.321 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:50.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:50.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.966426+0000", 2024-03-19T11:30:50.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:50.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.966426+0000", 2024-03-19T11:30:50.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:50.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.966429+0000", 2024-03-19T11:30:50.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3829999999999999e-06 2024-03-19T11:30:50.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:50.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972117+0000", 2024-03-19T11:30:50.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0056885750000000004 2024-03-19T11:30:50.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.323 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:50.323 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972156+0000", 2024-03-19T11:30:50.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8878000000000003e-05 2024-03-19T11:30:50.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.323 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:50.323 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972184+0000", 2024-03-19T11:30:50.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7509999999999999e-05 2024-03-19T11:30:50.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.323 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:50.323 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972309+0000", 2024-03-19T11:30:50.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012525899999999999 2024-03-19T11:30:50.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:50.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972566+0000", 2024-03-19T11:30:50.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025740799999999999 2024-03-19T11:30:50.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:30:50.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972899+0000", 2024-03-19T11:30:50.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033238799999999998 2024-03-19T11:30:50.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:50.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984088+0000", 2024-03-19T11:30:50.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011189466 2024-03-19T11:30:50.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:50.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984138+0000", 2024-03-19T11:30:50.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9666999999999997e-05 2024-03-19T11:30:50.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:50.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984329+0000", 2024-03-19T11:30:50.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019086799999999999 2024-03-19T11:30:50.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:50.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984470+0000", 2024-03-19T11:30:50.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000141111 2024-03-19T11:30:50.326 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.326 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:50.326 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.326 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1303 4.15 4:aa3303a5:::smithi03829856-17:head [write 1818036~510700] snapc b=[b,a,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-03-19T11:30:50.326 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:22.976856+0000", 2024-03-19T11:30:50.326 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.299637798999999, 2024-03-19T11:30:50.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015045703000000001, 2024-03-19T11:30:50.326 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:50.326 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:50.326 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:50.326 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:50.327 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:50.327 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1303 2024-03-19T11:30:50.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.327 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:50.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:50.327 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.976856+0000", 2024-03-19T11:30:50.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:50.327 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.976856+0000", 2024-03-19T11:30:50.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:50.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.976859+0000", 2024-03-19T11:30:50.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3629999999999998e-06 2024-03-19T11:30:50.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:50.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.977533+0000", 2024-03-19T11:30:50.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00067380800000000002 2024-03-19T11:30:50.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:50.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.977573+0000", 2024-03-19T11:30:50.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9758000000000001e-05 2024-03-19T11:30:50.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:50.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.977602+0000", 2024-03-19T11:30:50.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9856e-05 2024-03-19T11:30:50.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:50.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.979943+0000", 2024-03-19T11:30:50.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0023406400000000002 2024-03-19T11:30:50.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:50.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.980238+0000", 2024-03-19T11:30:50.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029488000000000002 2024-03-19T11:30:50.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:30:50.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.980734+0000", 2024-03-19T11:30:50.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00049647500000000002 2024-03-19T11:30:50.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:50.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.991761+0000", 2024-03-19T11:30:50.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011026515000000001 2024-03-19T11:30:50.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:50.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.991777+0000", 2024-03-19T11:30:50.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5727999999999999e-05 2024-03-19T11:30:50.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.331 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:50.331 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.991840+0000", 2024-03-19T11:30:50.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3091999999999998e-05 2024-03-19T11:30:50.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.331 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:50.331 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.991901+0000", 2024-03-19T11:30:50.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1587999999999994e-05 2024-03-19T11:30:50.331 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.331 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:50.331 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.332 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1467 4.a 4:567251f3:::smithi03829856-44:head [write 615156~555834] snapc c=[c,b,a,6,5] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-03-19T11:30:50.332 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.079231+0000", 2024-03-19T11:30:50.332 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.197262266999999, 2024-03-19T11:30:50.332 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013571666, 2024-03-19T11:30:50.332 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:50.332 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:50.332 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:50.332 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:50.332 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:50.332 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1467 2024-03-19T11:30:50.332 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.332 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:50.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.332 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:50.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.079231+0000", 2024-03-19T11:30:50.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:50.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.079231+0000", 2024-03-19T11:30:50.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:50.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.079234+0000", 2024-03-19T11:30:50.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3689999999999998e-06 2024-03-19T11:30:50.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:50.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.080110+0000", 2024-03-19T11:30:50.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00087507100000000005 2024-03-19T11:30:50.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:50.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.080146+0000", 2024-03-19T11:30:50.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6062000000000001e-05 2024-03-19T11:30:50.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:50.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.080164+0000", 2024-03-19T11:30:50.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7995e-05 2024-03-19T11:30:50.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.335 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:50.335 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.080339+0000", 2024-03-19T11:30:50.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017535900000000001 2024-03-19T11:30:50.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.335 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:50.335 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.080630+0000", 2024-03-19T11:30:50.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029068900000000001 2024-03-19T11:30:50.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.336 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:50.336 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.081140+0000", 2024-03-19T11:30:50.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00051083699999999999 2024-03-19T11:30:50.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.336 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:50.336 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.092570+0000", 2024-03-19T11:30:50.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011429557 2024-03-19T11:30:50.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.336 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:50.336 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.092611+0000", 2024-03-19T11:30:50.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1196000000000003e-05 2024-03-19T11:30:50.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:50.337 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.092717+0000", 2024-03-19T11:30:50.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000105777 2024-03-19T11:30:50.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:50.337 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.092803+0000", 2024-03-19T11:30:50.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5754000000000006e-05 2024-03-19T11:30:50.337 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.337 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:50.337 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.337 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1468 4.a 4:567251f3:::smithi03829856-44:head [write 1603467~416361] snapc c=[c,b,a,6,5] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-03-19T11:30:50.337 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.083922+0000", 2024-03-19T11:30:50.338 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.192571776000001, 2024-03-19T11:30:50.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014508549000000001, 2024-03-19T11:30:50.338 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:50.338 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:50.338 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:50.338 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:50.338 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:50.338 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1468 2024-03-19T11:30:50.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.338 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:50.338 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.338 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:50.338 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.083922+0000", 2024-03-19T11:30:50.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:50.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.083922+0000", 2024-03-19T11:30:50.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:50.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.083924+0000", 2024-03-19T11:30:50.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2800000000000002e-06 2024-03-19T11:30:50.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:50.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.084498+0000", 2024-03-19T11:30:50.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057414900000000004 2024-03-19T11:30:50.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.340 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:50.340 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.084529+0000", 2024-03-19T11:30:50.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1080000000000001e-05 2024-03-19T11:30:50.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.340 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:50.340 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.084544+0000", 2024-03-19T11:30:50.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4498000000000001e-05 2024-03-19T11:30:50.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.340 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:50.340 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.088993+0000", 2024-03-19T11:30:50.341 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044497260000000002 2024-03-19T11:30:50.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.341 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:50.341 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.089152+0000", 2024-03-19T11:30:50.341 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015833199999999999 2024-03-19T11:30:50.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.341 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:50.341 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.089419+0000", 2024-03-19T11:30:50.341 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026691799999999999 2024-03-19T11:30:50.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.341 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:50.341 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.092883+0000", 2024-03-19T11:30:50.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034646590000000001 2024-03-19T11:30:50.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.342 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:50.342 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.098276+0000", 2024-03-19T11:30:50.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053924680000000001 2024-03-19T11:30:50.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.342 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:50.342 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.098354+0000", 2024-03-19T11:30:50.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8763999999999993e-05 2024-03-19T11:30:50.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:50.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.098430+0000", 2024-03-19T11:30:50.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5674999999999996e-05 2024-03-19T11:30:50.343 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.343 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:50.343 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.343 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.1 50/49 [PushOp(4:866f3dcc:::smithi03829856-31:7, version: 38'44, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@38'44, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 45'62, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@45'62, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 46'68, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@46'68, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:7, version: 47'71, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@47'71, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 47'72, data_included: [440350~816629,1309629~231947], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@47'72, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:30:50.343 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.959457+0000", 2024-03-19T11:30:50.343 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.317036492, 2024-03-19T11:30:50.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018416666000000002, 2024-03-19T11:30:50.343 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:50.343 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:50.343 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:50.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:50.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.959457+0000", 2024-03-19T11:30:50.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:50.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.959457+0000", 2024-03-19T11:30:50.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:50.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.959458+0000", 2024-03-19T11:30:50.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4899999999999999e-06 2024-03-19T11:30:50.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.345 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.345 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:50.345 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.962822+0000", 2024-03-19T11:30:50.345 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0033640620000000001 2024-03-19T11:30:50.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.345 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.345 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:50.345 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.962893+0000", 2024-03-19T11:30:50.345 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1060000000000001e-05 2024-03-19T11:30:50.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.345 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.345 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:50.345 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.962977+0000", 2024-03-19T11:30:50.345 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3311000000000002e-05 2024-03-19T11:30:50.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:50.346 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963208+0000", 2024-03-19T11:30:50.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023121200000000001 2024-03-19T11:30:50.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:50.346 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963245+0000", 2024-03-19T11:30:50.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6844999999999997e-05 2024-03-19T11:30:50.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:50.346 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.977874+0000", 2024-03-19T11:30:50.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014628686 2024-03-19T11:30:50.347 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.347 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:50.347 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.347 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.347 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.1 e50,49 4:8401c1f0:::smithi03829856-43:8,46'66,4:866f3dcc:::smithi03829856-31:2,47'73)", 2024-03-19T11:30:50.347 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.963027+0000", 2024-03-19T11:30:50.347 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.313466875, 2024-03-19T11:30:50.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015116788000000001, 2024-03-19T11:30:50.347 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:50.347 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:50.347 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:50.347 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.347 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:50.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963027+0000", 2024-03-19T11:30:50.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:50.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963027+0000", 2024-03-19T11:30:50.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:50.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963028+0000", 2024-03-19T11:30:50.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.268e-06 2024-03-19T11:30:50.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.349 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:50.349 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963036+0000", 2024-03-19T11:30:50.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.926e-06 2024-03-19T11:30:50.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.349 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.349 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:50.349 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963050+0000", 2024-03-19T11:30:50.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4082e-05 2024-03-19T11:30:50.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.349 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.349 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:50.349 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963062+0000", 2024-03-19T11:30:50.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2415000000000001e-05 2024-03-19T11:30:50.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.349 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.350 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:50.350 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.977954+0000", 2024-03-19T11:30:50.350 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014891572 2024-03-19T11:30:50.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.350 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:50.350 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.977965+0000", 2024-03-19T11:30:50.350 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1654000000000001e-05 2024-03-19T11:30:50.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.350 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:50.350 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.978143+0000", 2024-03-19T11:30:50.350 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017787099999999999 2024-03-19T11:30:50.350 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.351 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:50.351 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.351 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.351 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.11 50/49 [PushOp(4:8d3763f9:::smithi03829856-29:head, version: 47'60, data_included: [3364276~232389], data_size: 232389, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8d3763f9:::smithi03829856-29:head@47'60, size: 3596665, copy_subset: [(0, 3596665)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3596665, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3364276, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:30:50.351 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.964363+0000", 2024-03-19T11:30:50.351 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.312130207999999, 2024-03-19T11:30:50.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014531759999999999, 2024-03-19T11:30:50.351 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:50.351 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:50.351 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:50.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:50.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.964363+0000", 2024-03-19T11:30:50.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.352 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.352 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:50.352 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.964363+0000", 2024-03-19T11:30:50.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.352 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.352 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:50.352 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.964364+0000", 2024-03-19T11:30:50.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4200000000000004e-07 2024-03-19T11:30:50.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.352 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.352 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:50.352 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.964447+0000", 2024-03-19T11:30:50.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2430999999999997e-05 2024-03-19T11:30:50.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:50.353 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.964468+0000", 2024-03-19T11:30:50.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1685999999999999e-05 2024-03-19T11:30:50.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:50.353 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.964483+0000", 2024-03-19T11:30:50.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4969e-05 2024-03-19T11:30:50.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:50.353 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.977934+0000", 2024-03-19T11:30:50.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013450963999999999 2024-03-19T11:30:50.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.354 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:50.354 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.977975+0000", 2024-03-19T11:30:50.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1125999999999999e-05 2024-03-19T11:30:50.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.354 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:50.354 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.978895+0000", 2024-03-19T11:30:50.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00091964200000000003 2024-03-19T11:30:50.354 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.354 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:50.354 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.355 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.1 50/49 [PushOp(4:866f3dcc:::smithi03829856-31:7, version: 38'44, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@38'44, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 45'62, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@45'62, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 46'68, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@46'68, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:7, version: 47'71, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@47'71, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 47'72, data_included: [1541576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@47'72, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:30:50.355 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.991858+0000", 2024-03-19T11:30:50.358 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.284635285, 2024-03-19T11:30:50.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014228774, 2024-03-19T11:30:50.358 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:50.358 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:50.358 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:50.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.358 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:50.358 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.991858+0000", 2024-03-19T11:30:50.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:50.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.991858+0000", 2024-03-19T11:30:50.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:50.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.991859+0000", 2024-03-19T11:30:50.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.342e-06 2024-03-19T11:30:50.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:50.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.994675+0000", 2024-03-19T11:30:50.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028153150000000001 2024-03-19T11:30:50.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:50.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.994733+0000", 2024-03-19T11:30:50.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8514999999999998e-05 2024-03-19T11:30:50.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:50.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.994794+0000", 2024-03-19T11:30:50.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0588000000000003e-05 2024-03-19T11:30:50.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:50.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.995070+0000", 2024-03-19T11:30:50.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027573700000000002 2024-03-19T11:30:50.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:50.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.995100+0000", 2024-03-19T11:30:50.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0679000000000002e-05 2024-03-19T11:30:50.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:50.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.006087+0000", 2024-03-19T11:30:50.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010986598 2024-03-19T11:30:50.361 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.361 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:50.362 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.362 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.1 50/49 [PushOp(4:866f3dcc:::smithi03829856-31:head, version: 47'74, data_included: [504210~968657,1572695~79919], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:head@47'74, size: 3495823, copy_subset: [(0, 3495823)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:30:50.362 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.995921+0000", 2024-03-19T11:30:50.362 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.280572071000002, 2024-03-19T11:30:50.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014183898, 2024-03-19T11:30:50.362 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:50.362 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:50.362 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:50.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.362 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:50.362 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.995921+0000", 2024-03-19T11:30:50.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.363 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.363 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:50.363 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.995921+0000", 2024-03-19T11:30:50.363 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.363 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.363 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:50.363 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.995922+0000", 2024-03-19T11:30:50.363 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9800000000000002e-07 2024-03-19T11:30:50.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.363 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.363 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:50.363 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.996355+0000", 2024-03-19T11:30:50.363 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000432211 2024-03-19T11:30:50.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.364 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:50.364 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.996386+0000", 2024-03-19T11:30:50.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1384999999999999e-05 2024-03-19T11:30:50.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.364 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:50.364 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.996412+0000", 2024-03-19T11:30:50.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6200999999999999e-05 2024-03-19T11:30:50.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.364 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:50.364 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.006166+0000", 2024-03-19T11:30:50.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0097542859999999992 2024-03-19T11:30:50.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:50.365 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.006182+0000", 2024-03-19T11:30:50.365 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5588000000000001e-05 2024-03-19T11:30:50.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:50.365 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.010105+0000", 2024-03-19T11:30:50.365 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039232290000000003 2024-03-19T11:30:50.365 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.365 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:50.365 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.366 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1679 4.e e59/21 4:72163e4c:::smithi03829856-39:head v 59'55, mlcod=58'53)", 2024-03-19T11:30:50.366 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:36.390395+0000", 2024-03-19T11:30:50.366 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 13.886098180999999, 2024-03-19T11:30:50.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013872282, 2024-03-19T11:30:50.366 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:50.366 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:50.366 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:50.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.366 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:50.366 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.390395+0000", 2024-03-19T11:30:50.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.367 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:50.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.390395+0000", 2024-03-19T11:30:50.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.367 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:50.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.390396+0000", 2024-03-19T11:30:50.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2300000000000002e-07 2024-03-19T11:30:50.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.367 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:50.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.390742+0000", 2024-03-19T11:30:50.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00034635599999999998 2024-03-19T11:30:50.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.368 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:50.368 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.390754+0000", 2024-03-19T11:30:50.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2211999999999999e-05 2024-03-19T11:30:50.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.368 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:50.368 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.390763+0000", 2024-03-19T11:30:50.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5769999999999994e-06 2024-03-19T11:30:50.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.368 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:50.368 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.390864+0000", 2024-03-19T11:30:50.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000101693 2024-03-19T11:30:50.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.369 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:50.369 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.390907+0000", 2024-03-19T11:30:50.369 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2642999999999998e-05 2024-03-19T11:30:50.369 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.369 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.369 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:50.369 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.404144+0000", 2024-03-19T11:30:50.369 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013237319000000001 2024-03-19T11:30:50.369 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.369 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.369 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:50.369 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:36.404267+0000", 2024-03-19T11:30:50.369 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012315899999999999 2024-03-19T11:30:50.369 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.369 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:50.370 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.370 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.370 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:50.370 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:50.371 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:50.292+0000 7f1837407640 -1 osd.6 70 handle_osd_map injecting map crc failure 2024-03-19T11:30:50.372 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-19T11:30:50.374 INFO:tasks.rados.rados.0.smithi038.stdout:1085: done (4 left) 2024-03-19T11:30:50.374 INFO:tasks.rados.rados.0.smithi038.stdout:1086: done (3 left) 2024-03-19T11:30:50.374 INFO:tasks.rados.rados.0.smithi038.stdout:1087: done (2 left) 2024-03-19T11:30:50.374 INFO:tasks.rados.rados.0.smithi038.stdout:1088: done (1 left) 2024-03-19T11:30:50.374 INFO:tasks.rados.rados.0.smithi038.stdout:1089: done (0 left) 2024-03-19T11:30:50.374 INFO:tasks.rados.rados.0.smithi038.stdout:1090: read oid 1 snap -1 2024-03-19T11:30:50.374 INFO:tasks.rados.rados.0.smithi038.stdout:1090: expect (ObjNum 156 snap 17 seq_num 156) 2024-03-19T11:30:50.374 INFO:tasks.rados.rados.0.smithi038.stdout:1091: set_chunk oid 45 target oid 2024-03-19T11:30:50.374 INFO:tasks.rados.rados.0.smithi038.stdout:1091: set_chunk oid 45 offset: 1105408 length: 11264 target oid b562edced8a9744b2a5af655e42912fceaf70e07f97b1d588870f516168f0065 offset: 0 snap 17 2024-03-19T11:30:50.375 INFO:tasks.rados.rados.0.smithi038.stdout:1092: read oid 44 snap -1 2024-03-19T11:30:50.375 INFO:tasks.rados.rados.0.smithi038.stdout:1092: expect (ObjNum 108 snap 11 seq_num 108) 2024-03-19T11:30:50.375 INFO:tasks.rados.rados.0.smithi038.stdout:1093: read oid 50 snap -1 2024-03-19T11:30:50.375 INFO:tasks.rados.rados.0.smithi038.stdout:1093: expect (ObjNum 114 snap 11 seq_num 114) 2024-03-19T11:30:50.375 INFO:tasks.rados.rados.0.smithi038.stdout:1094: write (excl) oid 37 current snap is 18 2024-03-19T11:30:50.375 INFO:tasks.rados.rados.0.smithi038.stdout:1094: seq_num 172 ranges {490804=702594,1615491=584710,2724284=166520} 2024-03-19T11:30:50.375 INFO:tasks.rados.rados.0.smithi038.stdout:1094: writing smithi03829856-37 from 490804 to 1193398 tid 1 2024-03-19T11:30:50.375 INFO:tasks.rados.rados.0.smithi038.stdout:1094: writing smithi03829856-37 from 1615491 to 2200201 tid 2 2024-03-19T11:30:50.375 INFO:tasks.rados.rados.0.smithi038.stdout:1094: writing smithi03829856-37 from 2724284 to 2890804 tid 3 2024-03-19T11:30:50.375 INFO:tasks.rados.rados.0.smithi038.stdout:1095: write oid 33 current snap is 18 2024-03-19T11:30:50.375 INFO:tasks.rados.rados.0.smithi038.stdout:1095: seq_num 173 ranges {507611=777843,2073973=444721} 2024-03-19T11:30:50.375 INFO:tasks.rados.rados.0.smithi038.stdout:1095: writing smithi03829856-33 from 507611 to 1285454 tid 1 2024-03-19T11:30:50.375 INFO:tasks.rados.rados.0.smithi038.stdout:1095: writing smithi03829856-33 from 2073973 to 2518694 tid 2 2024-03-19T11:30:50.375 INFO:tasks.rados.rados.0.smithi038.stdout:1096: read oid 41 snap -1 2024-03-19T11:30:50.375 INFO:tasks.rados.rados.0.smithi038.stdout:1096: expect (ObjNum 170 snap 17 seq_num 170) 2024-03-19T11:30:50.376 INFO:tasks.rados.rados.0.smithi038.stdout:1097: read oid 22 snap -1 2024-03-19T11:30:50.376 INFO:tasks.rados.rados.0.smithi038.stdout:1097: expect (ObjNum 168 snap 17 seq_num 168) 2024-03-19T11:30:50.376 INFO:tasks.rados.rados.0.smithi038.stdout:1098: delete oid 19 current snap is 18 2024-03-19T11:30:50.376 INFO:tasks.rados.rados.0.smithi038.stdout:1090: expect (ObjNum 156 snap 17 seq_num 156) 2024-03-19T11:30:50.376 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:50.378+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:50.472 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:50.480+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:50.541 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:50.541 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:50.541 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:30:50.541 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:50.550 INFO:tasks.rados.rados.0.smithi038.stdout:1093: expect (ObjNum 114 snap 11 seq_num 114) 2024-03-19T11:30:50.555 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-19T11:30:50.599 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:50.608+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:50.674 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:50.682+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:50.714 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:50.715 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:50.715 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:30:50.715 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:30:50.715 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:50.726 INFO:tasks.rados.rados.0.smithi038.stdout:1091: finishing set_chunk to oid 45 2024-03-19T11:30:50.726 INFO:tasks.rados.rados.0.smithi038.stdout:1092: expect (ObjNum 108 snap 11 seq_num 108) 2024-03-19T11:30:50.729 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-19T11:30:50.775 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:50.784+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:50.849 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:50.849 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:30:50.850 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:30:50.850 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:30:50.850 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.850 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:30:50.850 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:44.200920+0000", 2024-03-19T11:30:50.850 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 66.658867126999994, 2024-03-19T11:30:50.850 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.80881896499999995, 2024-03-19T11:30:50.850 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:50.850 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:50.850 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:50.850 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:50.850 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:50.850 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1 2024-03-19T11:30:50.850 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.851 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:50.851 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.851 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:50.851 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200920+0000", 2024-03-19T11:30:50.851 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.851 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.851 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:50.851 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200920+0000", 2024-03-19T11:30:50.851 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.851 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.851 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:50.851 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200923+0000", 2024-03-19T11:30:50.852 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6469999999999998e-06 2024-03-19T11:30:50.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.852 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.852 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:50.852 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200937+0000", 2024-03-19T11:30:50.852 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.399e-05 2024-03-19T11:30:50.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.852 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.852 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:50.852 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200960+0000", 2024-03-19T11:30:50.852 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2603e-05 2024-03-19T11:30:50.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.852 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.852 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:50.853 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200978+0000", 2024-03-19T11:30:50.853 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.791e-05 2024-03-19T11:30:50.853 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.853 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.853 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:50.853 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.201081+0000", 2024-03-19T11:30:50.853 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010344900000000001 2024-03-19T11:30:50.853 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.853 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.853 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:30:50.853 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.201096+0000", 2024-03-19T11:30:50.853 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4443e-05 2024-03-19T11:30:50.853 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.853 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.853 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:50.854 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.005978+0000", 2024-03-19T11:30:50.854 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.80488281500000003 2024-03-19T11:30:50.854 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.854 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.854 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:50.854 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.006267+0000", 2024-03-19T11:30:50.854 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000288688 2024-03-19T11:30:50.854 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.854 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.854 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:50.854 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.006534+0000", 2024-03-19T11:30:50.854 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026725799999999998 2024-03-19T11:30:50.854 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.855 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.855 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:50.855 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.008457+0000", 2024-03-19T11:30:50.855 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0019227820000000001 2024-03-19T11:30:50.855 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.855 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.855 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:50.855 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.009570+0000", 2024-03-19T11:30:50.855 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011131750000000001 2024-03-19T11:30:50.855 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.855 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.855 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:50.855 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.009670+0000", 2024-03-19T11:30:50.855 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9458999999999998e-05 2024-03-19T11:30:50.855 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.856 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.856 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:50.856 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.009739+0000", 2024-03-19T11:30:50.856 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9746000000000006e-05 2024-03-19T11:30:50.856 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.856 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:50.856 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.856 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.856 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.856 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:30:50.856 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.071728+0000", 2024-03-19T11:30:50.856 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 65.788059689999997, 2024-03-19T11:30:50.856 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013714690999999999, 2024-03-19T11:30:50.856 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:50.856 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:50.857 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:50.857 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:50.857 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:50.857 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 34 2024-03-19T11:30:50.857 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.857 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:50.857 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.857 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:50.857 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071728+0000", 2024-03-19T11:30:50.857 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.857 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.857 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.857 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:50.857 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071728+0000", 2024-03-19T11:30:50.857 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.858 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.858 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.858 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:50.858 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071729+0000", 2024-03-19T11:30:50.858 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.47e-07 2024-03-19T11:30:50.858 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.858 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.858 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:50.858 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071735+0000", 2024-03-19T11:30:50.858 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3799999999999999e-06 2024-03-19T11:30:50.858 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.858 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.858 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:50.858 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071748+0000", 2024-03-19T11:30:50.859 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2806000000000001e-05 2024-03-19T11:30:50.859 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.859 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.859 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:50.859 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071756+0000", 2024-03-19T11:30:50.859 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1240000000000005e-06 2024-03-19T11:30:50.859 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.859 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.859 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:50.859 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.077570+0000", 2024-03-19T11:30:50.859 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.005813342 2024-03-19T11:30:50.859 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.859 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.859 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:50.860 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.077793+0000", 2024-03-19T11:30:50.860 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022391500000000001 2024-03-19T11:30:50.860 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.860 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.860 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:50.860 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.078146+0000", 2024-03-19T11:30:50.860 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00035219 2024-03-19T11:30:50.860 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.860 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.860 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:50.860 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.083390+0000", 2024-03-19T11:30:50.860 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052445590000000002 2024-03-19T11:30:50.860 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.860 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.860 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:50.861 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085288+0000", 2024-03-19T11:30:50.861 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0018977060000000001 2024-03-19T11:30:50.861 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.861 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.861 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:50.861 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085377+0000", 2024-03-19T11:30:50.861 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9068999999999997e-05 2024-03-19T11:30:50.861 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.861 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.861 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:50.861 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085443+0000", 2024-03-19T11:30:50.861 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5653000000000003e-05 2024-03-19T11:30:50.861 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.861 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:50.861 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.862 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.862 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.862 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:30:50.862 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.071777+0000", 2024-03-19T11:30:50.862 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 65.788010580000005, 2024-03-19T11:30:50.862 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013962445, 2024-03-19T11:30:50.862 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:50.862 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:50.862 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:50.862 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:50.862 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:50.862 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 35 2024-03-19T11:30:50.862 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.862 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:50.863 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.863 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:50.863 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071777+0000", 2024-03-19T11:30:50.863 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.863 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.863 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.863 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:50.863 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071777+0000", 2024-03-19T11:30:50.863 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.863 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.863 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.863 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:50.863 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071778+0000", 2024-03-19T11:30:50.863 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5000000000000001e-07 2024-03-19T11:30:50.863 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.864 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.864 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:50.864 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071917+0000", 2024-03-19T11:30:50.864 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013880199999999999 2024-03-19T11:30:50.864 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.864 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.864 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:50.864 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071931+0000", 2024-03-19T11:30:50.864 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4080000000000001e-05 2024-03-19T11:30:50.864 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.864 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.864 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:50.864 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071940+0000", 2024-03-19T11:30:50.864 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.363e-06 2024-03-19T11:30:50.864 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.864 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.865 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:50.865 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.080462+0000", 2024-03-19T11:30:50.865 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085221810000000002 2024-03-19T11:30:50.865 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.865 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.865 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:50.865 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.080582+0000", 2024-03-19T11:30:50.865 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011935299999999999 2024-03-19T11:30:50.865 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.865 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.865 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:50.865 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.080782+0000", 2024-03-19T11:30:50.865 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020077100000000001 2024-03-19T11:30:50.865 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.866 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.866 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:50.866 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.083401+0000", 2024-03-19T11:30:50.866 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002618333 2024-03-19T11:30:50.866 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.866 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.866 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:50.866 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085571+0000", 2024-03-19T11:30:50.866 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021702829999999998 2024-03-19T11:30:50.866 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.866 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.866 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:50.866 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085672+0000", 2024-03-19T11:30:50.866 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100876 2024-03-19T11:30:50.867 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.867 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.867 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:50.867 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085739+0000", 2024-03-19T11:30:50.867 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7552999999999995e-05 2024-03-19T11:30:50.867 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.867 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:50.867 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.867 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.867 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.867 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:30:50.867 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.071957+0000", 2024-03-19T11:30:50.867 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 65.787830392000004, 2024-03-19T11:30:50.867 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013989822000000001, 2024-03-19T11:30:50.867 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:50.868 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:50.868 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:50.868 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:50.868 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:50.868 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 36 2024-03-19T11:30:50.868 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.868 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:50.868 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.868 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:50.868 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071957+0000", 2024-03-19T11:30:50.868 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.868 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.868 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.868 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:50.868 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071957+0000", 2024-03-19T11:30:50.869 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.869 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.869 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.869 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:50.869 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071958+0000", 2024-03-19T11:30:50.869 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.06e-06 2024-03-19T11:30:50.869 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.869 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.869 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:50.869 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071965+0000", 2024-03-19T11:30:50.869 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8929999999999999e-06 2024-03-19T11:30:50.869 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.869 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.869 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:50.869 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071976+0000", 2024-03-19T11:30:50.869 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1046999999999999e-05 2024-03-19T11:30:50.870 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.870 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.870 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:50.870 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071984+0000", 2024-03-19T11:30:50.870 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7170000000000008e-06 2024-03-19T11:30:50.870 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.870 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.870 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:50.870 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.081761+0000", 2024-03-19T11:30:50.870 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0097769310000000009 2024-03-19T11:30:50.870 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.870 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.870 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:50.870 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.081928+0000", 2024-03-19T11:30:50.871 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000167219 2024-03-19T11:30:50.871 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.871 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.871 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:50.871 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.082131+0000", 2024-03-19T11:30:50.871 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020244499999999999 2024-03-19T11:30:50.871 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.871 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.871 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:50.871 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.083455+0000", 2024-03-19T11:30:50.871 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001324073 2024-03-19T11:30:50.871 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.871 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.871 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:50.871 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085816+0000", 2024-03-19T11:30:50.872 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0023613309999999999 2024-03-19T11:30:50.872 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.872 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.872 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:50.872 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085892+0000", 2024-03-19T11:30:50.872 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6154000000000003e-05 2024-03-19T11:30:50.872 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.872 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.872 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:50.872 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085947+0000", 2024-03-19T11:30:50.872 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4951999999999998e-05 2024-03-19T11:30:50.872 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.873 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:50.873 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.873 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.873 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.873 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:30:50.873 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.072000+0000", 2024-03-19T11:30:50.873 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 65.787787641999998, 2024-03-19T11:30:50.873 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014418268999999999, 2024-03-19T11:30:50.873 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:50.873 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:50.873 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:50.873 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:50.873 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:50.873 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 37 2024-03-19T11:30:50.874 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.874 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:50.874 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.874 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:50.874 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072000+0000", 2024-03-19T11:30:50.874 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.874 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.874 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.874 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:50.874 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072000+0000", 2024-03-19T11:30:50.874 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.874 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.874 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.874 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:50.874 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072001+0000", 2024-03-19T11:30:50.875 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0699999999999996e-07 2024-03-19T11:30:50.875 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.875 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.875 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:50.875 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072007+0000", 2024-03-19T11:30:50.875 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2060000000000004e-06 2024-03-19T11:30:50.875 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.875 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.875 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:50.875 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072023+0000", 2024-03-19T11:30:50.875 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6617e-05 2024-03-19T11:30:50.875 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.875 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.875 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:50.876 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072031+0000", 2024-03-19T11:30:50.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8900000000000007e-06 2024-03-19T11:30:50.876 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.876 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.876 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:50.876 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.082682+0000", 2024-03-19T11:30:50.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010650868000000001 2024-03-19T11:30:50.876 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.876 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.876 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:50.876 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.082804+0000", 2024-03-19T11:30:50.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012202899999999999 2024-03-19T11:30:50.876 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.876 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.876 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:50.877 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.083006+0000", 2024-03-19T11:30:50.877 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020127799999999999 2024-03-19T11:30:50.877 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.877 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.877 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:50.877 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.084145+0000", 2024-03-19T11:30:50.877 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001139791 2024-03-19T11:30:50.877 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.877 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.877 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:50.877 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.086292+0000", 2024-03-19T11:30:50.877 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021461969999999999 2024-03-19T11:30:50.877 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.877 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.877 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:50.878 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.086364+0000", 2024-03-19T11:30:50.878 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2439000000000002e-05 2024-03-19T11:30:50.878 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.878 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.878 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:50.878 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.086418+0000", 2024-03-19T11:30:50.878 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4246999999999997e-05 2024-03-19T11:30:50.878 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.878 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:50.878 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.878 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.878 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.878 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:30:50.878 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.339510+0000", 2024-03-19T11:30:50.879 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 65.520277793999995, 2024-03-19T11:30:50.879 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012358388999999999, 2024-03-19T11:30:50.879 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:50.879 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:50.879 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:50.879 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:50.879 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:50.879 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 111 2024-03-19T11:30:50.879 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.879 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:50.879 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.879 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:50.879 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339510+0000", 2024-03-19T11:30:50.879 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.880 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.880 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:50.880 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339510+0000", 2024-03-19T11:30:50.880 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.880 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.880 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:50.880 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339511+0000", 2024-03-19T11:30:50.880 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4399999999999998e-07 2024-03-19T11:30:50.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.880 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.880 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:50.880 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339518+0000", 2024-03-19T11:30:50.880 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9999999999999999e-06 2024-03-19T11:30:50.881 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.881 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.881 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:50.881 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339535+0000", 2024-03-19T11:30:50.881 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7552999999999999e-05 2024-03-19T11:30:50.881 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.881 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.881 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:50.881 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339550+0000", 2024-03-19T11:30:50.881 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5003e-05 2024-03-19T11:30:50.881 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.881 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.881 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:50.881 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.348403+0000", 2024-03-19T11:30:50.882 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0088530810000000005 2024-03-19T11:30:50.882 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.882 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.882 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:50.882 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.348629+0000", 2024-03-19T11:30:50.882 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022534599999999999 2024-03-19T11:30:50.882 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.882 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.882 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:50.882 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.348976+0000", 2024-03-19T11:30:50.882 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00034710699999999998 2024-03-19T11:30:50.882 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.882 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.883 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:50.883 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.351479+0000", 2024-03-19T11:30:50.883 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0025035309999999998 2024-03-19T11:30:50.883 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.883 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.883 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:50.883 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.351602+0000", 2024-03-19T11:30:50.883 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000122912 2024-03-19T11:30:50.883 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.883 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.883 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:50.883 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.351766+0000", 2024-03-19T11:30:50.883 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016410599999999999 2024-03-19T11:30:50.883 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.884 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.884 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:50.884 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.351868+0000", 2024-03-19T11:30:50.884 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010180599999999999 2024-03-19T11:30:50.884 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.884 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:50.884 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.884 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.884 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.884 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:30:50.884 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.339576+0000", 2024-03-19T11:30:50.884 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 65.520211369999998, 2024-03-19T11:30:50.884 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012548515999999999, 2024-03-19T11:30:50.885 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:50.885 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:50.885 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:50.885 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:50.885 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:50.885 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 112 2024-03-19T11:30:50.885 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.885 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:50.885 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.885 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:50.885 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339576+0000", 2024-03-19T11:30:50.885 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.885 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.885 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.886 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:50.886 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339576+0000", 2024-03-19T11:30:50.886 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.886 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.886 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.886 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:50.886 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339577+0000", 2024-03-19T11:30:50.886 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2529999999999999e-06 2024-03-19T11:30:50.886 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.886 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.886 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:50.886 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339813+0000", 2024-03-19T11:30:50.886 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000235811 2024-03-19T11:30:50.886 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.887 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.887 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:50.887 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339848+0000", 2024-03-19T11:30:50.887 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4437000000000002e-05 2024-03-19T11:30:50.887 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.887 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.887 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:50.887 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339867+0000", 2024-03-19T11:30:50.887 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9046e-05 2024-03-19T11:30:50.887 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.887 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.887 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:50.887 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.349815+0000", 2024-03-19T11:30:50.888 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0099484029999999998 2024-03-19T11:30:50.888 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.888 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.888 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:50.888 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.349955+0000", 2024-03-19T11:30:50.888 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000140106 2024-03-19T11:30:50.888 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.888 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.888 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:50.888 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.350163+0000", 2024-03-19T11:30:50.888 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020803799999999999 2024-03-19T11:30:50.888 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.888 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.889 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:50.889 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.351984+0000", 2024-03-19T11:30:50.889 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0018202159999999999 2024-03-19T11:30:50.889 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.889 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.889 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:50.889 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352002+0000", 2024-03-19T11:30:50.889 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8430000000000001e-05 2024-03-19T11:30:50.889 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.889 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.889 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:50.889 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352070+0000", 2024-03-19T11:30:50.890 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7623e-05 2024-03-19T11:30:50.890 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.890 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.890 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:50.890 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352125+0000", 2024-03-19T11:30:50.890 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5152999999999999e-05 2024-03-19T11:30:50.890 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.890 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:50.890 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.890 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.890 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.890 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:30:50.891 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.339898+0000", 2024-03-19T11:30:50.891 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 65.519890020000005, 2024-03-19T11:30:50.891 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019501793, 2024-03-19T11:30:50.891 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:50.891 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:50.891 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:50.891 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:50.891 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:50.891 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113 2024-03-19T11:30:50.891 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.891 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:50.891 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.891 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:50.891 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339898+0000", 2024-03-19T11:30:50.891 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.892 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.892 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.892 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:50.892 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339898+0000", 2024-03-19T11:30:50.892 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.892 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.892 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.892 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:50.892 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339899+0000", 2024-03-19T11:30:50.892 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.353e-06 2024-03-19T11:30:50.892 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.892 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.892 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:50.892 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339907+0000", 2024-03-19T11:30:50.893 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3000000000000002e-06 2024-03-19T11:30:50.893 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.893 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:50.893 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339924+0000", 2024-03-19T11:30:50.893 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6906999999999999e-05 2024-03-19T11:30:50.893 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.893 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:50.893 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339933+0000", 2024-03-19T11:30:50.893 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8969999999999997e-06 2024-03-19T11:30:50.893 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.893 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:50.894 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352239+0000", 2024-03-19T11:30:50.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012305814999999999 2024-03-19T11:30:50.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:50.894 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352466+0000", 2024-03-19T11:30:50.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022684100000000001 2024-03-19T11:30:50.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:50.894 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352816+0000", 2024-03-19T11:30:50.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00035030499999999997 2024-03-19T11:30:50.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:50.895 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.358956+0000", 2024-03-19T11:30:50.895 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061400079999999998 2024-03-19T11:30:50.895 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.895 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.895 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:50.895 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359118+0000", 2024-03-19T11:30:50.895 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016237599999999999 2024-03-19T11:30:50.895 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.895 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.895 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:50.895 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359287+0000", 2024-03-19T11:30:50.895 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000168488 2024-03-19T11:30:50.895 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.895 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.895 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:50.896 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359399+0000", 2024-03-19T11:30:50.896 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000112503 2024-03-19T11:30:50.896 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.896 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:50.896 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.896 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.896 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.896 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:30:50.896 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.339947+0000", 2024-03-19T11:30:50.896 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 65.519840252999998, 2024-03-19T11:30:50.896 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021923069999999999, 2024-03-19T11:30:50.896 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:50.896 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:50.896 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:50.897 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:50.897 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:50.897 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 114 2024-03-19T11:30:50.897 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.897 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:50.897 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.897 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:50.897 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339947+0000", 2024-03-19T11:30:50.897 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.897 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.897 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.897 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:50.897 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339947+0000", 2024-03-19T11:30:50.898 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.898 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.898 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.898 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:50.898 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339948+0000", 2024-03-19T11:30:50.898 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6000000000000003e-07 2024-03-19T11:30:50.898 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.898 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.898 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:50.898 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339956+0000", 2024-03-19T11:30:50.898 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1559999999999995e-06 2024-03-19T11:30:50.898 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.898 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.898 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:50.899 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339967+0000", 2024-03-19T11:30:50.899 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0669999999999999e-05 2024-03-19T11:30:50.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.899 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.899 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:50.899 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339975+0000", 2024-03-19T11:30:50.899 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0099999999999995e-06 2024-03-19T11:30:50.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.899 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.899 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:50.899 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.353672+0000", 2024-03-19T11:30:50.899 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013696583999999999 2024-03-19T11:30:50.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.900 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.900 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:50.900 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.353808+0000", 2024-03-19T11:30:50.900 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013652299999999999 2024-03-19T11:30:50.900 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.900 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.900 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:50.900 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.354036+0000", 2024-03-19T11:30:50.900 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022744799999999999 2024-03-19T11:30:50.900 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.900 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.900 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:50.900 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.360480+0000", 2024-03-19T11:30:50.900 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0064441630000000001 2024-03-19T11:30:50.900 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.901 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.901 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:50.901 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361722+0000", 2024-03-19T11:30:50.901 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001242754 2024-03-19T11:30:50.901 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.901 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.901 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:50.901 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361807+0000", 2024-03-19T11:30:50.901 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4605999999999996e-05 2024-03-19T11:30:50.901 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.901 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.901 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:50.901 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361870+0000", 2024-03-19T11:30:50.901 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3396000000000001e-05 2024-03-19T11:30:50.901 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.902 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:50.902 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.902 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.902 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.902 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:30:50.902 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.340017+0000", 2024-03-19T11:30:50.902 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 65.519770937000004, 2024-03-19T11:30:50.902 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022169355000000002, 2024-03-19T11:30:50.902 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:50.902 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:50.902 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:50.902 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:50.902 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:50.902 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 115 2024-03-19T11:30:50.903 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.903 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:50.903 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.903 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:50.903 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340017+0000", 2024-03-19T11:30:50.903 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.903 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.903 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.903 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:50.903 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340017+0000", 2024-03-19T11:30:50.903 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.903 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.903 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.903 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:50.904 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340018+0000", 2024-03-19T11:30:50.904 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.043e-06 2024-03-19T11:30:50.904 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.904 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.904 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:50.904 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340139+0000", 2024-03-19T11:30:50.904 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121502 2024-03-19T11:30:50.904 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.904 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.904 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:50.904 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340157+0000", 2024-03-19T11:30:50.904 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7382999999999999e-05 2024-03-19T11:30:50.904 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.904 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.904 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:50.905 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340166+0000", 2024-03-19T11:30:50.905 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2469999999999997e-06 2024-03-19T11:30:50.905 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.905 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.905 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:50.905 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.357470+0000", 2024-03-19T11:30:50.905 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017304119999999999 2024-03-19T11:30:50.905 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.905 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.905 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:50.905 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.357671+0000", 2024-03-19T11:30:50.905 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000200798 2024-03-19T11:30:50.905 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.905 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.906 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:50.906 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.358002+0000", 2024-03-19T11:30:50.906 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033090099999999998 2024-03-19T11:30:50.906 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.906 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.906 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:50.906 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361892+0000", 2024-03-19T11:30:50.906 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00389018 2024-03-19T11:30:50.906 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.906 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.906 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:50.906 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361983+0000", 2024-03-19T11:30:50.906 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0976000000000004e-05 2024-03-19T11:30:50.906 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.907 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.907 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:50.907 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362071+0000", 2024-03-19T11:30:50.907 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8022999999999994e-05 2024-03-19T11:30:50.907 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.907 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.907 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:50.907 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362186+0000", 2024-03-19T11:30:50.907 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115182 2024-03-19T11:30:50.907 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.907 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:50.907 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.907 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.908 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.908 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:30:50.908 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.340183+0000", 2024-03-19T11:30:50.908 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 65.519604104999999, 2024-03-19T11:30:50.908 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022279841000000002, 2024-03-19T11:30:50.908 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:50.908 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:50.908 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:50.908 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:50.908 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:50.908 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 116 2024-03-19T11:30:50.908 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.908 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:50.909 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.909 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:50.909 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340183+0000", 2024-03-19T11:30:50.909 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.909 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.909 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.909 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:50.909 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340183+0000", 2024-03-19T11:30:50.909 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.909 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.909 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.909 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:50.909 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340184+0000", 2024-03-19T11:30:50.909 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9000000000000005e-07 2024-03-19T11:30:50.910 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.910 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.910 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:50.910 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340191+0000", 2024-03-19T11:30:50.910 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0360000000000003e-06 2024-03-19T11:30:50.910 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.910 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.910 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:50.910 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340201+0000", 2024-03-19T11:30:50.910 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0844e-05 2024-03-19T11:30:50.910 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.910 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.910 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:50.911 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340209+0000", 2024-03-19T11:30:50.911 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7360000000000001e-06 2024-03-19T11:30:50.911 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.911 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.911 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:50.911 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359535+0000", 2024-03-19T11:30:50.911 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019325546999999998 2024-03-19T11:30:50.911 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.911 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.911 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:50.911 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359672+0000", 2024-03-19T11:30:50.911 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013706600000000001 2024-03-19T11:30:50.911 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.911 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.912 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:50.912 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359905+0000", 2024-03-19T11:30:50.912 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023334500000000001 2024-03-19T11:30:50.912 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.912 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.912 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:50.912 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362306+0000", 2024-03-19T11:30:50.912 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002401041 2024-03-19T11:30:50.912 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.912 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.912 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:50.912 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362329+0000", 2024-03-19T11:30:50.912 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3283999999999999e-05 2024-03-19T11:30:50.912 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.912 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.913 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:50.913 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362402+0000", 2024-03-19T11:30:50.913 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2459000000000005e-05 2024-03-19T11:30:50.913 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.913 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.913 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:50.913 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362463+0000", 2024-03-19T11:30:50.913 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1493000000000002e-05 2024-03-19T11:30:50.913 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.913 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:50.913 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.913 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.913 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.914 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:30:50.914 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.340222+0000", 2024-03-19T11:30:50.914 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 65.519565521999994, 2024-03-19T11:30:50.914 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022500218999999998, 2024-03-19T11:30:50.914 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:50.914 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:50.914 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:50.914 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:50.914 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:50.914 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 117 2024-03-19T11:30:50.914 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.914 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:50.914 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.914 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:50.915 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340222+0000", 2024-03-19T11:30:50.915 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.915 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.915 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.915 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:50.915 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340222+0000", 2024-03-19T11:30:50.915 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.915 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.915 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.915 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:50.915 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340223+0000", 2024-03-19T11:30:50.915 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7999999999999995e-07 2024-03-19T11:30:50.915 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.915 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.916 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:50.916 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340229+0000", 2024-03-19T11:30:50.916 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0399999999999998e-06 2024-03-19T11:30:50.916 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.916 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.916 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:50.916 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340239+0000", 2024-03-19T11:30:50.916 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0190000000000001e-05 2024-03-19T11:30:50.916 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.916 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.916 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:50.916 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340250+0000", 2024-03-19T11:30:50.916 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0720000000000001e-05 2024-03-19T11:30:50.916 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.917 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.917 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:50.917 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.360587+0000", 2024-03-19T11:30:50.917 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020337542 2024-03-19T11:30:50.917 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.917 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.917 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:50.917 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.360818+0000", 2024-03-19T11:30:50.917 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023120500000000001 2024-03-19T11:30:50.917 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.917 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.917 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:50.917 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361218+0000", 2024-03-19T11:30:50.917 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000399637 2024-03-19T11:30:50.917 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.918 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.918 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:50.918 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362483+0000", 2024-03-19T11:30:50.918 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012649200000000001 2024-03-19T11:30:50.918 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.918 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.918 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:50.918 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362538+0000", 2024-03-19T11:30:50.918 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4490000000000002e-05 2024-03-19T11:30:50.918 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.918 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.919 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:50.919 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362617+0000", 2024-03-19T11:30:50.919 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9839000000000006e-05 2024-03-19T11:30:50.919 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.919 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.919 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:50.919 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362722+0000", 2024-03-19T11:30:50.919 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104956 2024-03-19T11:30:50.919 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.919 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:50.919 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.919 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.919 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.919 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1047 4.8 e35/21 4:1d58db11:::smithi03829856-18:head v 35'32, mlcod=33'29)", 2024-03-19T11:30:50.920 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:13.225023+0000", 2024-03-19T11:30:50.920 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 37.634764736999998, 2024-03-19T11:30:50.920 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013059688999999999, 2024-03-19T11:30:50.920 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:50.920 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:50.920 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:50.920 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.920 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:50.920 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.225023+0000", 2024-03-19T11:30:50.920 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.920 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.920 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.920 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:50.920 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.225023+0000", 2024-03-19T11:30:50.921 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.921 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.921 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.921 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:50.921 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.225025+0000", 2024-03-19T11:30:50.921 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.17e-06 2024-03-19T11:30:50.921 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.921 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.921 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:50.921 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.225771+0000", 2024-03-19T11:30:50.921 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000746267 2024-03-19T11:30:50.921 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.921 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.921 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:50.921 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.225807+0000", 2024-03-19T11:30:50.921 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5487000000000001e-05 2024-03-19T11:30:50.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.922 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:50.922 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.225823+0000", 2024-03-19T11:30:50.922 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6215999999999999e-05 2024-03-19T11:30:50.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.922 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:50.922 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.225990+0000", 2024-03-19T11:30:50.922 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016680899999999999 2024-03-19T11:30:50.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.922 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:50.922 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.226058+0000", 2024-03-19T11:30:50.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8672999999999998e-05 2024-03-19T11:30:50.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.923 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:50.923 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.237940+0000", 2024-03-19T11:30:50.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011881311 2024-03-19T11:30:50.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.923 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:50.923 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.238083+0000", 2024-03-19T11:30:50.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014275600000000001 2024-03-19T11:30:50.923 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.923 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:50.923 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.924 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1067 4.0 e36/21 4:01018511:::smithi03829856-19:head v 36'18, mlcod=35'14)", 2024-03-19T11:30:50.924 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:13.952492+0000", 2024-03-19T11:30:50.924 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 36.90729528, 2024-03-19T11:30:50.924 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013554725, 2024-03-19T11:30:50.924 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:50.924 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:50.924 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:50.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.924 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:50.924 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.952492+0000", 2024-03-19T11:30:50.924 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.924 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.924 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:50.925 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.952492+0000", 2024-03-19T11:30:50.925 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.925 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:50.925 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.952493+0000", 2024-03-19T11:30:50.925 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9299999999999997e-07 2024-03-19T11:30:50.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.925 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:50.925 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.954109+0000", 2024-03-19T11:30:50.925 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0016163569999999999 2024-03-19T11:30:50.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.925 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:50.925 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.954135+0000", 2024-03-19T11:30:50.926 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5352999999999999e-05 2024-03-19T11:30:50.926 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.926 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.926 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:50.926 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.954149+0000", 2024-03-19T11:30:50.926 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3879999999999999e-05 2024-03-19T11:30:50.926 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.926 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.926 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:50.926 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.954288+0000", 2024-03-19T11:30:50.926 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000139476 2024-03-19T11:30:50.926 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.926 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.926 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:50.927 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.954356+0000", 2024-03-19T11:30:50.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7533000000000005e-05 2024-03-19T11:30:50.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.927 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:50.927 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.965391+0000", 2024-03-19T11:30:50.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011035764999999999 2024-03-19T11:30:50.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.927 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:50.927 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.966047+0000", 2024-03-19T11:30:50.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00065556800000000004 2024-03-19T11:30:50.927 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.927 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:50.928 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.928 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1122 4.8 e36/21 4:1d58db11:::smithi03829856-18:head v 36'42, mlcod=36'39)", 2024-03-19T11:30:50.928 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:14.172666+0000", 2024-03-19T11:30:50.928 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 36.687121505999997, 2024-03-19T11:30:50.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018015000999999999, 2024-03-19T11:30:50.928 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:50.928 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:50.928 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:50.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.928 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:50.928 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.172666+0000", 2024-03-19T11:30:50.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:50.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.172666+0000", 2024-03-19T11:30:50.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:50.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.172669+0000", 2024-03-19T11:30:50.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1470000000000001e-06 2024-03-19T11:30:50.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:50.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.173140+0000", 2024-03-19T11:30:50.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00047074200000000001 2024-03-19T11:30:50.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:50.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.173188+0000", 2024-03-19T11:30:50.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8287e-05 2024-03-19T11:30:50.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:50.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.173211+0000", 2024-03-19T11:30:50.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3023e-05 2024-03-19T11:30:50.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.931 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:50.931 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.173378+0000", 2024-03-19T11:30:50.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000166742 2024-03-19T11:30:50.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.931 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:50.931 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.173557+0000", 2024-03-19T11:30:50.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000178569 2024-03-19T11:30:50.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.931 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:50.931 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.190416+0000", 2024-03-19T11:30:50.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016859570000000001 2024-03-19T11:30:50.932 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.932 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.932 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:50.932 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.190681+0000", 2024-03-19T11:30:50.932 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026492099999999999 2024-03-19T11:30:50.932 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.932 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:50.932 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.932 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.932 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.932 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.5.0:43 4.8 e39/21 4:15708482:::smithi03829856-23:head v 39'50, mlcod=38'47)", 2024-03-19T11:30:50.932 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.842819+0000", 2024-03-19T11:30:50.933 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.016969003, 2024-03-19T11:30:50.933 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017626513, 2024-03-19T11:30:50.933 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:50.933 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:50.933 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:50.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.933 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:50.933 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842819+0000", 2024-03-19T11:30:50.933 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.933 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.933 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:50.933 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842819+0000", 2024-03-19T11:30:50.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.934 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:50.934 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842820+0000", 2024-03-19T11:30:50.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.39e-06 2024-03-19T11:30:50.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.934 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:50.934 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.846437+0000", 2024-03-19T11:30:50.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003616953 2024-03-19T11:30:50.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.934 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:50.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.846471+0000", 2024-03-19T11:30:50.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4133e-05 2024-03-19T11:30:50.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:50.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.846539+0000", 2024-03-19T11:30:50.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8072999999999997e-05 2024-03-19T11:30:50.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:50.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.847094+0000", 2024-03-19T11:30:50.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00055472099999999999 2024-03-19T11:30:50.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.936 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.936 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:50.936 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.847128+0000", 2024-03-19T11:30:50.936 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3875999999999999e-05 2024-03-19T11:30:50.936 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.936 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.936 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:50.936 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.860308+0000", 2024-03-19T11:30:50.936 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013180712000000001 2024-03-19T11:30:50.936 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.936 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.936 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:50.936 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.860445+0000", 2024-03-19T11:30:50.936 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013665500000000001 2024-03-19T11:30:50.937 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.937 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:50.937 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.937 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.937 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.937 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1209 4.8 e41/21 4:1d58db11:::smithi03829856-18:head v 41'53, mlcod=39'50)", 2024-03-19T11:30:50.937 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:18.124878+0000", 2024-03-19T11:30:50.937 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.734909309999999, 2024-03-19T11:30:50.937 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016809343000000001, 2024-03-19T11:30:50.937 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:50.937 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:50.937 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:50.937 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.937 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:50.937 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.124878+0000", 2024-03-19T11:30:50.938 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.938 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.938 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.938 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:50.938 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.124878+0000", 2024-03-19T11:30:50.938 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.938 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.938 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.938 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:50.938 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.124880+0000", 2024-03-19T11:30:50.938 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4959999999999999e-06 2024-03-19T11:30:50.938 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.938 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.938 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:50.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125845+0000", 2024-03-19T11:30:50.939 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00096535299999999996 2024-03-19T11:30:50.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.939 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:50.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125884+0000", 2024-03-19T11:30:50.939 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8645e-05 2024-03-19T11:30:50.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.939 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:50.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125909+0000", 2024-03-19T11:30:50.939 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4783e-05 2024-03-19T11:30:50.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.940 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.940 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:50.940 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.126085+0000", 2024-03-19T11:30:50.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000176649 2024-03-19T11:30:50.940 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.940 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.940 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:50.940 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.126150+0000", 2024-03-19T11:30:50.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4290999999999998e-05 2024-03-19T11:30:50.940 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.940 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.940 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:50.940 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.141552+0000", 2024-03-19T11:30:50.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015402295 2024-03-19T11:30:50.941 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.941 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.941 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:50.941 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.141688+0000", 2024-03-19T11:30:50.941 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013583099999999999 2024-03-19T11:30:50.941 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.941 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:50.941 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.941 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.941 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.941 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.15 51/50 [PushOp(4:afa9f208:::smithi03829856-35:head, version: 28'46, data_included: [571438~623878,1806623~424698], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:afa9f208:::smithi03829856-35:head@28'46, size: 2571438, copy_subset: [(0, 2571438)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2231321, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:af9d0d56:::smithi03829856-20:head, version: 35'55, data_included: [655360~589824,1900544~418789], data_size: 1008613, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:af9d0d56:::smithi03829856-20:head@35'55, size: 2319333, copy_subset: [(0, 2319333)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2319333, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ad3ad1b3:::smithi03829856-21:b, version: 47'66, data_included: [736140~537686,2020581~315559], data_size: 853245, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:ad3ad1b3:::smithi03829856-21:b@47'66, size: 2336140, copy_subset: [(0, 2336140)], clone_subset: {}, snapset: b=[]:{b: [b, a, 6, 5]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2336140, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ad3ad1b3:::smithi03829856-21:head, version: 47'70, data_included: [531121~742705,1711057~305871], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:ad3ad1b3:::smithi03829856-21:head@47'70, size: 3731121, copy_subset: [(0, 3731121)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2016928, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ac9cd3a3:::smithi03829856-8:head, version: 49'72, data_included: [517649~620182,1776668~428394], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:ac9cd3a3:::smithi03829856-8:head@49'72, size: 3008420, copy_subset: [(0, 3008420)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2205062, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:aa3303a5:::smithi03829856-17:b, version: 49'73, data_included: [614400~692224,1903006~356352], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:aa3303a5:::smithi03829856-17:b@49'73, size: 2268912, copy_subset: [(0, 2268912)], clone_subset: {}, snapset: b=[]:{b: [b, a, 6, 5]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2259358, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:aa3303a5:::smithi03829856-17:head, version: 49'74, data_included: [413472~893152,1818036~155424], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:aa3303a5:::smithi03829856-17:head@49'74, size: 3613472, copy_subset: [(0, 3613472)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1973460, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:30:50.941 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:28.058610+0000", 2024-03-19T11:30:50.941 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.80117735, 2024-03-19T11:30:50.941 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027100032, 2024-03-19T11:30:50.942 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:50.942 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:50.942 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:50.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.942 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:50.942 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.058610+0000", 2024-03-19T11:30:50.942 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.942 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.942 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:50.942 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.058610+0000", 2024-03-19T11:30:50.942 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.942 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.942 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:50.943 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.058611+0000", 2024-03-19T11:30:50.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1599999999999999e-06 2024-03-19T11:30:50.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:50.943 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.063874+0000", 2024-03-19T11:30:50.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052625270000000004 2024-03-19T11:30:50.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:50.943 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.063935+0000", 2024-03-19T11:30:50.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0594000000000003e-05 2024-03-19T11:30:50.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:50.944 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064024+0000", 2024-03-19T11:30:50.944 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9273999999999999e-05 2024-03-19T11:30:50.944 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.944 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:50.944 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064385+0000", 2024-03-19T11:30:50.944 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00036129100000000002 2024-03-19T11:30:50.944 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.944 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:50.944 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064437+0000", 2024-03-19T11:30:50.944 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2346000000000003e-05 2024-03-19T11:30:50.944 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.944 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:50.945 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085710+0000", 2024-03-19T11:30:50.945 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021272840000000001 2024-03-19T11:30:50.945 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.945 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:50.945 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.945 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.945 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.15 e51,50 4:ac9cd3a3:::smithi03829856-8:4,49'71)", 2024-03-19T11:30:50.945 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:28.064112+0000", 2024-03-19T11:30:50.945 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.795675524, 2024-03-19T11:30:50.945 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026480417999999999, 2024-03-19T11:30:50.945 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:50.945 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:50.945 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:50.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.945 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:50.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064112+0000", 2024-03-19T11:30:50.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.946 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:50.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064112+0000", 2024-03-19T11:30:50.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.946 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:50.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064113+0000", 2024-03-19T11:30:50.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3600000000000003e-07 2024-03-19T11:30:50.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:50.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064118+0000", 2024-03-19T11:30:50.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0549999999999999e-06 2024-03-19T11:30:50.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:50.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064128+0000", 2024-03-19T11:30:50.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0349e-05 2024-03-19T11:30:50.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:50.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064136+0000", 2024-03-19T11:30:50.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6249999999999998e-06 2024-03-19T11:30:50.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:50.948 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085773+0000", 2024-03-19T11:30:50.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021636829999999999 2024-03-19T11:30:50.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.948 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:50.948 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085788+0000", 2024-03-19T11:30:50.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5512999999999999e-05 2024-03-19T11:30:50.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.948 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:50.948 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.090592+0000", 2024-03-19T11:30:50.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0048044100000000003 2024-03-19T11:30:50.948 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.948 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:50.948 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.949 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.949 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.5 51/50 [PushOp(4:a38b9453:::smithi03829856-37:head, version: 28'45, data_included: [605052~774376,1923587~274200], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:a38b9453:::smithi03829856-37:head@28'45, size: 2605052, copy_subset: [(0, 2605052)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2197787, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:a0a50d04:::smithi03829856-13:b, version: 47'63, data_included: [601684~619341,1669078~429235], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a0a50d04:::smithi03829856-13:b@47'63, size: 2201684, copy_subset: [(0, 2201684)], clone_subset: {}, snapset: b=[]:{b: [b, a, 6, 5]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2098313, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:30:50.949 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:28.064324+0000", 2024-03-19T11:30:50.949 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.795463680000001, 2024-03-19T11:30:50.949 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027179020000000002, 2024-03-19T11:30:50.949 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:50.949 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:50.949 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:50.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.949 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:50.949 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064324+0000", 2024-03-19T11:30:50.949 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.949 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:50.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064324+0000", 2024-03-19T11:30:50.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:50.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:50.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064324+0000", 2024-03-19T11:30:50.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1199999999999998e-07 2024-03-19T11:30:50.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:50.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.065527+0000", 2024-03-19T11:30:50.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012028939999999999 2024-03-19T11:30:50.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:50.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.065552+0000", 2024-03-19T11:30:50.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4440999999999999e-05 2024-03-19T11:30:50.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:50.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.065568+0000", 2024-03-19T11:30:50.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6498e-05 2024-03-19T11:30:50.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:50.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085812+0000", 2024-03-19T11:30:50.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020243954000000002 2024-03-19T11:30:50.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.952 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:50.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085825+0000", 2024-03-19T11:30:50.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3298999999999999e-05 2024-03-19T11:30:50.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:50.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:50.952 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:50.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.091503+0000", 2024-03-19T11:30:50.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.005677522 2024-03-19T11:30:50.952 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.952 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:50.952 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.952 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:50.952 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:50.952 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:50.953 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-19T11:30:50.954 INFO:tasks.rados.rados.0.smithi038.stdout:1094: finishing write tid 1 to smithi03829856-37 2024-03-19T11:30:50.954 INFO:tasks.rados.rados.0.smithi038.stdout:1094: finishing write tid 2 to smithi03829856-37 2024-03-19T11:30:50.954 INFO:tasks.rados.rados.0.smithi038.stdout:1094: finishing write tid 3 to smithi03829856-37 2024-03-19T11:30:50.954 INFO:tasks.rados.rados.0.smithi038.stdout:1094: finishing write tid 4 to smithi03829856-37 2024-03-19T11:30:50.954 INFO:tasks.rados.rados.0.smithi038.stdout:1094: finishing write tid 5 to smithi03829856-37 2024-03-19T11:30:50.954 INFO:tasks.rados.rados.0.smithi038.stdout:1094: oid 37 updating version 0 to 74 2024-03-19T11:30:50.954 INFO:tasks.rados.rados.0.smithi038.stdout:1094: oid 37 updating version 74 to 76 2024-03-19T11:30:50.954 INFO:tasks.rados.rados.0.smithi038.stdout:1094: oid 37 version 76 is already newer than 75 2024-03-19T11:30:50.954 INFO:tasks.rados.rados.0.smithi038.stdout:1094: oid 37 updating version 76 to 77 2024-03-19T11:30:50.954 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 37 v 77 (ObjNum 172 snap 18 seq_num 172) dirty exists 2024-03-19T11:30:50.955 INFO:tasks.rados.rados.0.smithi038.stdout:1094: left oid 37 (ObjNum 172 snap 18 seq_num 172) 2024-03-19T11:30:50.955 INFO:tasks.rados.rados.0.smithi038.stdout:1095: finishing write tid 1 to smithi03829856-33 2024-03-19T11:30:50.955 INFO:tasks.rados.rados.0.smithi038.stdout:1095: finishing write tid 2 to smithi03829856-33 2024-03-19T11:30:50.955 INFO:tasks.rados.rados.0.smithi038.stdout:1095: finishing write tid 3 to smithi03829856-33 2024-03-19T11:30:50.955 INFO:tasks.rados.rados.0.smithi038.stdout:1095: finishing write tid 4 to smithi03829856-33 2024-03-19T11:30:50.955 INFO:tasks.rados.rados.0.smithi038.stdout:1095: oid 33 updating version 0 to 89 2024-03-19T11:30:50.955 INFO:tasks.rados.rados.0.smithi038.stdout:1095: oid 33 updating version 89 to 91 2024-03-19T11:30:50.955 INFO:tasks.rados.rados.0.smithi038.stdout:1095: oid 33 version 91 is already newer than 90 2024-03-19T11:30:50.955 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 33 v 91 (ObjNum 173 snap 18 seq_num 173) dirty exists 2024-03-19T11:30:50.955 INFO:tasks.rados.rados.0.smithi038.stdout:1095: left oid 33 (ObjNum 173 snap 18 seq_num 173) 2024-03-19T11:30:50.955 INFO:tasks.rados.rados.0.smithi038.stdout:1096: expect (ObjNum 170 snap 17 seq_num 170) 2024-03-19T11:30:50.956 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:50.962+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:50.956 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-19T11:30:50.992 INFO:tasks.rados.rados.0.smithi038.stdout:1097: expect (ObjNum 168 snap 17 seq_num 168) 2024-03-19T11:30:51.055 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:51.062+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:51.089 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:51.090 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:30:51.090 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:30:51.090 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:51.094 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:30:51.094 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:30:51.103 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-19T11:30:51.110 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:30:51.111 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:30:51.116 INFO:tasks.rados.rados.0.smithi038.stdout:1090: done (8 left) 2024-03-19T11:30:51.116 INFO:tasks.rados.rados.0.smithi038.stdout:1091: done (7 left) 2024-03-19T11:30:51.116 INFO:tasks.rados.rados.0.smithi038.stdout:1092: done (6 left) 2024-03-19T11:30:51.116 INFO:tasks.rados.rados.0.smithi038.stdout:1093: done (5 left) 2024-03-19T11:30:51.116 INFO:tasks.rados.rados.0.smithi038.stdout:1094: done (4 left) 2024-03-19T11:30:51.116 INFO:tasks.rados.rados.0.smithi038.stdout:1095: done (3 left) 2024-03-19T11:30:51.116 INFO:tasks.rados.rados.0.smithi038.stdout:1096: done (2 left) 2024-03-19T11:30:51.116 INFO:tasks.rados.rados.0.smithi038.stdout:1097: done (1 left) 2024-03-19T11:30:51.116 INFO:tasks.rados.rados.0.smithi038.stdout:1098: done (0 left) 2024-03-19T11:30:51.116 INFO:tasks.rados.rados.0.smithi038.stdout:1099: tier_promote oid 9 2024-03-19T11:30:51.116 INFO:tasks.rados.rados.0.smithi038.stdout:1100: set_chunk oid 4 target oid 2024-03-19T11:30:51.122 INFO:tasks.rados.rados.0.smithi038.stdout:1100: set_chunk oid 4 offset: 381952 length: 356864 target oid ad40acbd875e72373b1f736fe4b01f977616cd0b2bcc1fe7398de67a8dbdae58 offset: 0 snap 4 2024-03-19T11:30:51.122 INFO:tasks.rados.rados.0.smithi038.stdout:1101: set_chunk oid 45 target oid 2024-03-19T11:30:51.128 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:30:51.128 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:30:51.140 INFO:tasks.rados.rados.0.smithi038.stdout:1101: set_chunk oid 45 offset: 394752 length: 1915392 target oid 6c94bf138d50964f000573b26cead684f768977f07d9428544a9bd6b1d35bd78 offset: 0 snap -1 2024-03-19T11:30:51.141 INFO:tasks.rados.rados.0.smithi038.stdout:1102: tier_flush oid 14 2024-03-19T11:30:51.141 INFO:tasks.rados.rados.0.smithi038.stdout:1102: tier_flush oid 14 snap -1 2024-03-19T11:30:51.141 INFO:tasks.rados.rados.0.smithi038.stdout:1100: finishing set_chunk to oid 4 2024-03-19T11:30:51.141 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 4 set_chunk returned error code -95 offset: 381952 length: 356864 2024-03-19T11:30:51.141 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 4 v 0 (ObjNum 151 snap 17 seq_num 151) dirty exists 2024-03-19T11:30:51.141 INFO:tasks.rados.rados.0.smithi038.stdout:1100: done (3 left) 2024-03-19T11:30:51.141 INFO:tasks.rados.rados.0.smithi038.stdout:1103: read oid 15 snap -1 2024-03-19T11:30:51.141 INFO:tasks.rados.rados.0.smithi038.stdout:1103: expect (ObjNum 14 snap 0 seq_num 14) 2024-03-19T11:30:51.141 INFO:tasks.rados.rados.0.smithi038.stdout:1104: read oid 43 snap -1 2024-03-19T11:30:51.141 INFO:tasks.rados.rados.0.smithi038.stdout:1104: expect (ObjNum 155 snap 17 seq_num 155) 2024-03-19T11:30:51.141 INFO:tasks.rados.rados.0.smithi038.stdout:1105: tier_evict oid 30 2024-03-19T11:30:51.141 INFO:tasks.rados.rados.0.smithi038.stdout:1105: tier_evict oid 30 snap -1 2024-03-19T11:30:51.142 INFO:tasks.rados.rados.0.smithi038.stdout:1106: tier_promote oid 17 2024-03-19T11:30:51.142 INFO:tasks.rados.rados.0.smithi038.stdout:1107: read oid 6 snap 9 2024-03-19T11:30:51.142 INFO:tasks.rados.rados.0.smithi038.stdout:1107: expect (ObjNum 53 snap 0 seq_num 53) 2024-03-19T11:30:51.142 INFO:tasks.rados.rados.0.smithi038.stdout:1108: read oid 2 snap -1 2024-03-19T11:30:51.142 INFO:tasks.rados.rados.0.smithi038.stdout:1108: expect deleted 2024-03-19T11:30:51.142 INFO:tasks.rados.rados.0.smithi038.stdout:1109: read oid 19 snap -1 2024-03-19T11:30:51.142 INFO:tasks.rados.rados.0.smithi038.stdout:1109: expect deleted 2024-03-19T11:30:51.142 INFO:tasks.rados.rados.0.smithi038.stdout:1110: write (excl) oid 23 current snap is 18 2024-03-19T11:30:51.142 INFO:tasks.rados.rados.0.smithi038.stdout:1110: seq_num 175 ranges {541225=644694,1907290=448390} 2024-03-19T11:30:51.147 INFO:tasks.rados.rados.0.smithi038.stdout:1110: writing smithi03829856-23 from 541225 to 1185919 tid 1 2024-03-19T11:30:51.152 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:30:51.152 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:30:51.154 INFO:tasks.rados.rados.0.smithi038.stdout:1110: writing smithi03829856-23 from 1907290 to 2355680 tid 2 2024-03-19T11:30:51.155 INFO:tasks.rados.rados.0.smithi038.stdout:1111: rollback oid 33 current snap is 18 2024-03-19T11:30:51.155 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 33 to 4 2024-03-19T11:30:51.155 INFO:tasks.rados.rados.0.smithi038.stdout:1102: got (16) Device or resource busy 2024-03-19T11:30:51.155 INFO:tasks.rados.rados.0.smithi038.stdout:check_oldest_snap_flushed oid 14 v 46 (ObjNum 88 snap 8 seq_num 88) unflushed 17 2024-03-19T11:30:51.155 INFO:tasks.rados.rados.0.smithi038.stdout:1101: finishing set_chunk to oid 45 2024-03-19T11:30:51.155 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 45 set_chunk returned error code -95 offset: 394752 length: 1915392 2024-03-19T11:30:51.155 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 45 v 0 (ObjNum 148 snap 15 seq_num 148) dirty exists 2024-03-19T11:30:51.155 INFO:tasks.rados.rados.0.smithi038.stdout:1105: got (22) Invalid argument 2024-03-19T11:30:51.155 INFO:tasks.rados.rados.0.smithi038.stdout:1106: got (0) Success 2024-03-19T11:30:51.155 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 17 v 116 (ObjNum 166 snap 17 seq_num 166) dirty exists 2024-03-19T11:30:51.155 INFO:tasks.rados.rados.0.smithi038.stdout:1107: expect (ObjNum 53 snap 0 seq_num 53) 2024-03-19T11:30:51.156 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:51.166+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:51.178 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:30:51.179 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:30:51.198 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:30:51.199 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:30:51.221 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:30:51.221 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:30:51.227 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:51.228 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:51.241 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-19T11:30:51.244 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:30:51.244 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:30:51.258 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:51.264+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:51.266 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-03-19T11:30:51.268 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:51.278+0000 7f70c6fdf640 -1 osd.0 71 handle_osd_map injecting map crc failure 2024-03-19T11:30:51.292 INFO:tasks.rados.rados.0.smithi038.stdout:1103: expect (ObjNum 14 snap 0 seq_num 14) 2024-03-19T11:30:51.359 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:51.368+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:51.365 INFO:tasks.rados.rados.0.smithi038.stdout:1104: expect (ObjNum 155 snap 17 seq_num 155) 2024-03-19T11:30:51.371 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:51.372 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:51.386 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-19T11:30:51.461 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:51.468+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:51.539 INFO:tasks.rados.rados.0.smithi038.stdout:1111: finishing rollback tid 0 to smithi03829856-33 2024-03-19T11:30:51.539 INFO:tasks.rados.rados.0.smithi038.stdout:1111: finishing rollback tid 1 to smithi03829856-33 2024-03-19T11:30:51.539 INFO:tasks.rados.rados.0.smithi038.stdout:1111: finishing rollback tid 2 to smithi03829856-33 2024-03-19T11:30:51.539 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 33 v 94 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-03-19T11:30:51.540 INFO:tasks.rados.rados.0.smithi038.stdout:1110: finishing write tid 1 to smithi03829856-23 2024-03-19T11:30:51.540 INFO:tasks.rados.rados.0.smithi038.stdout:1110: finishing write tid 2 to smithi03829856-23 2024-03-19T11:30:51.540 INFO:tasks.rados.rados.0.smithi038.stdout:1110: finishing write tid 3 to smithi03829856-23 2024-03-19T11:30:51.540 INFO:tasks.rados.rados.0.smithi038.stdout:1110: finishing write tid 4 to smithi03829856-23 2024-03-19T11:30:51.541 INFO:tasks.rados.rados.0.smithi038.stdout:1110: oid 23 updating version 0 to 84 2024-03-19T11:30:51.541 INFO:tasks.rados.rados.0.smithi038.stdout:1110: oid 23 updating version 84 to 85 2024-03-19T11:30:51.541 INFO:tasks.rados.rados.0.smithi038.stdout:1110: oid 23 version 85 is already newer than 83 2024-03-19T11:30:51.541 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 23 v 85 (ObjNum 175 snap 18 seq_num 175) dirty exists 2024-03-19T11:30:51.541 INFO:tasks.rados.rados.0.smithi038.stdout:1110: left oid 23 (ObjNum 175 snap 18 seq_num 175) 2024-03-19T11:30:51.541 INFO:tasks.rados.rados.0.smithi038.stdout:1101: done (11 left) 2024-03-19T11:30:51.542 INFO:tasks.rados.rados.0.smithi038.stdout:1102: done (10 left) 2024-03-19T11:30:51.542 INFO:tasks.rados.rados.0.smithi038.stdout:1103: done (9 left) 2024-03-19T11:30:51.542 INFO:tasks.rados.rados.0.smithi038.stdout:1104: done (8 left) 2024-03-19T11:30:51.542 INFO:tasks.rados.rados.0.smithi038.stdout:1105: done (7 left) 2024-03-19T11:30:51.542 INFO:tasks.rados.rados.0.smithi038.stdout:1106: done (6 left) 2024-03-19T11:30:51.542 INFO:tasks.rados.rados.0.smithi038.stdout:1107: done (5 left) 2024-03-19T11:30:51.543 INFO:tasks.rados.rados.0.smithi038.stdout:1108: done (4 left) 2024-03-19T11:30:51.543 INFO:tasks.rados.rados.0.smithi038.stdout:1109: done (3 left) 2024-03-19T11:30:51.543 INFO:tasks.rados.rados.0.smithi038.stdout:1110: done (2 left) 2024-03-19T11:30:51.543 INFO:tasks.rados.rados.0.smithi038.stdout:1111: done (1 left) 2024-03-19T11:30:51.543 INFO:tasks.rados.rados.0.smithi038.stdout:1112: set_chunk oid 17 target oid 2024-03-19T11:30:51.545 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:51.546 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:51.560 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-19T11:30:51.563 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:51.568+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:51.579 INFO:tasks.rados.rados.0.smithi038.stdout:1112: set_chunk oid 17 offset: 24576 length: 2477568 target oid a4e830cd2d1fb07ae82bbf960b259b23973d06756a1673e9fa5886e3e0742038 offset: 0 snap 11 2024-03-19T11:30:51.579 INFO:tasks.rados.rados.0.smithi038.stdout:1113: tier_flush oid 47 2024-03-19T11:30:51.579 INFO:tasks.rados.rados.0.smithi038.stdout:1113: tier_flush oid 47 snap -1 2024-03-19T11:30:51.579 INFO:tasks.rados.rados.0.smithi038.stdout:1114: read oid 5 snap -1 2024-03-19T11:30:51.580 INFO:tasks.rados.rados.0.smithi038.stdout:1114: expect (ObjNum 137 snap 15 seq_num 137) 2024-03-19T11:30:51.580 INFO:tasks.rados.rados.0.smithi038.stdout:1115: read oid 44 snap 11 2024-03-19T11:30:51.580 INFO:tasks.rados.rados.0.smithi038.stdout:1115: expect (ObjNum 108 snap 11 seq_num 108) 2024-03-19T11:30:51.580 INFO:tasks.rados.rados.0.smithi038.stdout:1116: read oid 34 snap -1 2024-03-19T11:30:51.580 INFO:tasks.rados.rados.0.smithi038.stdout:1116: expect (ObjNum 72 snap 5 seq_num 72) 2024-03-19T11:30:51.580 INFO:tasks.rados.rados.0.smithi038.stdout:1117: snap_create 2024-03-19T11:30:51.580 INFO:tasks.rados.rados.0.smithi038.stdout:1113: got (0) Success 2024-03-19T11:30:51.581 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 47 v 15 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-03-19T11:30:51.581 INFO:tasks.rados.rados.0.smithi038.stdout:1112: finishing set_chunk to oid 17 2024-03-19T11:30:51.581 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 17 set_chunk returned error code -95 offset: 24576 length: 2477568 2024-03-19T11:30:51.582 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 17 v 0 (ObjNum 166 snap 17 seq_num 166) dirty exists 2024-03-19T11:30:51.586 INFO:tasks.rados.rados.0.smithi038.stdout:1116: expect (ObjNum 72 snap 5 seq_num 72) 2024-03-19T11:30:51.663 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:51.670+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:51.676 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:51.677 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:51.690 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-19T11:30:51.763 INFO:tasks.rados.rados.0.smithi038.stdout:1114: expect (ObjNum 137 snap 15 seq_num 137) 2024-03-19T11:30:51.764 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:51.774+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:51.850 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:51.850 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:51.863 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-19T11:30:51.864 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:51.876+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:51.872 INFO:tasks.rados.rados.0.smithi038.stdout:1115: expect (ObjNum 108 snap 11 seq_num 108) 2024-03-19T11:30:51.966 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:51.974+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:51.984 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:51.984 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:51.998 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-19T11:30:52.067 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:52.076+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:52.152 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:52.153 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:52.166 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-19T11:30:52.169 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:52.176+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:52.269 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:52.276+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:52.277 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-19T11:30:52.278 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-19T11:30:52.278 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-03-19T11:30:52.278 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-19T11:30:52.278 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-19T11:30:52.278 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set filestore_inject_stall 3 2024-03-19T11:30:52.287 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:52.294+0000 7fa5de8d7640 -1 osd.3 72 handle_osd_map injecting map crc failure 2024-03-19T11:30:52.287 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:52.287 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:52.292 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:52.298+0000 7fa5de8d7640 -1 osd.3 72 handle_osd_map injecting map crc failure 2024-03-19T11:30:52.297 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:52.302+0000 7fa5de8d7640 -1 osd.3 72 handle_osd_map injecting map crc failure 2024-03-19T11:30:52.297 INFO:tasks.rados.rados.0.smithi038.stdout:1112: done (6 left) 2024-03-19T11:30:52.297 INFO:tasks.rados.rados.0.smithi038.stdout:1113: done (5 left) 2024-03-19T11:30:52.297 INFO:tasks.rados.rados.0.smithi038.stdout:1114: done (4 left) 2024-03-19T11:30:52.297 INFO:tasks.rados.rados.0.smithi038.stdout:1115: done (3 left) 2024-03-19T11:30:52.297 INFO:tasks.rados.rados.0.smithi038.stdout:1116: done (2 left) 2024-03-19T11:30:52.297 INFO:tasks.rados.rados.0.smithi038.stdout:1117: done (1 left) 2024-03-19T11:30:52.297 INFO:tasks.rados.rados.0.smithi038.stdout:1118: write oid 37 current snap is 19 2024-03-19T11:30:52.297 INFO:tasks.rados.rados.0.smithi038.stdout:1118: seq_num 176 ranges {558032=719943,1849419=592048} 2024-03-19T11:30:52.302 INFO:tasks.rados.rados.0.smithi038.stdout:1118: writing smithi03829856-37 from 558032 to 1277975 tid 1 2024-03-19T11:30:52.311 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-19T11:30:52.316 INFO:tasks.rados.rados.0.smithi038.stdout:1118: writing smithi03829856-37 from 1849419 to 2441467 tid 2 2024-03-19T11:30:52.316 INFO:tasks.rados.rados.0.smithi038.stdout:1119: read oid 3 snap -1 2024-03-19T11:30:52.316 INFO:tasks.rados.rados.0.smithi038.stdout:1119: expect deleted 2024-03-19T11:30:52.316 INFO:tasks.rados.rados.0.smithi038.stdout:1120: read oid 32 snap -1 2024-03-19T11:30:52.316 INFO:tasks.rados.rados.0.smithi038.stdout:1120: expect (ObjNum 31 snap 0 seq_num 31) 2024-03-19T11:30:52.316 INFO:tasks.rados.rados.0.smithi038.stdout:1121: snap_remove snap 18 2024-03-19T11:30:52.369 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:52.376+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:52.402 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:52.402 INFO:teuthology.orchestra.run.smithi038.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-19T11:30:52.402 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:52.428 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:52.428 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:52.440 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-19T11:30:52.470 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:52.480+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:52.571 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:52.580+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:52.606 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:52.606 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:52.621 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-19T11:30:52.672 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:52.680+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:52.747 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:52.747 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:52.762 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-19T11:30:52.807 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:52.816+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:52.874 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:52.882+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:52.896 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:52.897 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:52.912 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-19T11:30:52.975 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:52.986+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:53.035 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:53.043+0000 7f1835403640 -1 osd.6 73 handle_osd_map injecting map crc failure 2024-03-19T11:30:53.055 INFO:tasks.rados.rados.0.smithi038.stdout:1121: done (4 left) 2024-03-19T11:30:53.055 INFO:tasks.rados.rados.0.smithi038.stdout:1122: write (excl) oid 48 current snap is 19 2024-03-19T11:30:53.055 INFO:tasks.rados.rados.0.smithi038.stdout:1122: seq_num 177 ranges {574839=795192,1907901=452059} 2024-03-19T11:30:53.070 INFO:tasks.rados.rados.0.smithi038.stdout:1122: writing smithi03829856-48 from 574839 to 1370031 tid 1 2024-03-19T11:30:53.076 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:53.083+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:53.081 INFO:tasks.rados.rados.0.smithi038.stdout:1122: writing smithi03829856-48 from 1907901 to 2359960 tid 2 2024-03-19T11:30:53.083 INFO:tasks.rados.rados.0.smithi038.stdout:1123: tier_promote oid 12 2024-03-19T11:30:53.083 INFO:tasks.rados.rados.0.smithi038.stdout:1118: finishing write tid 1 to smithi03829856-37 2024-03-19T11:30:53.083 INFO:tasks.rados.rados.0.smithi038.stdout:1118: finishing write tid 2 to smithi03829856-37 2024-03-19T11:30:53.083 INFO:tasks.rados.rados.0.smithi038.stdout:1118: finishing write tid 3 to smithi03829856-37 2024-03-19T11:30:53.084 INFO:tasks.rados.rados.0.smithi038.stdout:1118: finishing write tid 4 to smithi03829856-37 2024-03-19T11:30:53.084 INFO:tasks.rados.rados.0.smithi038.stdout:1118: oid 37 updating version 0 to 81 2024-03-19T11:30:53.084 INFO:tasks.rados.rados.0.smithi038.stdout:1118: oid 37 version 81 is already newer than 79 2024-03-19T11:30:53.084 INFO:tasks.rados.rados.0.smithi038.stdout:1118: oid 37 version 81 is already newer than 80 2024-03-19T11:30:53.084 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 37 v 81 (ObjNum 176 snap 19 seq_num 176) dirty exists 2024-03-19T11:30:53.084 INFO:tasks.rados.rados.0.smithi038.stdout:1118: left oid 37 (ObjNum 176 snap 19 seq_num 176) 2024-03-19T11:30:53.084 INFO:tasks.rados.rados.0.smithi038.stdout:1099: got (0) Success 2024-03-19T11:30:53.084 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 9 v 65 (ObjNum 152 snap 17 seq_num 152) dirty exists 2024-03-19T11:30:53.084 INFO:tasks.rados.rados.0.smithi038.stdout:1099: done (5 left) 2024-03-19T11:30:53.084 INFO:tasks.rados.rados.0.smithi038.stdout:1118: done (4 left) 2024-03-19T11:30:53.084 INFO:tasks.rados.rados.0.smithi038.stdout:1119: done (3 left) 2024-03-19T11:30:53.084 INFO:tasks.rados.rados.0.smithi038.stdout:1124: read oid 46 snap -1 2024-03-19T11:30:53.084 INFO:tasks.rados.rados.0.smithi038.stdout:1124: expect (ObjNum 143 snap 15 seq_num 143) 2024-03-19T11:30:53.084 INFO:tasks.rados.rados.0.smithi038.stdout:1125: read oid 3 snap 4 2024-03-19T11:30:53.084 INFO:tasks.rados.rados.0.smithi038.stdout:1125: expect (ObjNum 62 snap 3 seq_num 62) 2024-03-19T11:30:53.085 INFO:tasks.rados.rados.0.smithi038.stdout:1126: read oid 2 snap -1 2024-03-19T11:30:53.085 INFO:tasks.rados.rados.0.smithi038.stdout:1126: expect deleted 2024-03-19T11:30:53.085 INFO:tasks.rados.rados.0.smithi038.stdout:1127: write (excl) oid 5 current snap is 19 2024-03-19T11:30:53.085 INFO:tasks.rados.rados.0.smithi038.stdout:1127: seq_num 178 ranges {591646=470441,1566383=595717,2888822=102824} 2024-03-19T11:30:53.092 INFO:tasks.rados.rados.0.smithi038.stdout:1127: writing smithi03829856-5 from 591646 to 1062087 tid 1 2024-03-19T11:30:53.093 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:53.093 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:53.101 INFO:tasks.rados.rados.0.smithi038.stdout:1127: writing smithi03829856-5 from 1566383 to 2162100 tid 2 2024-03-19T11:30:53.108 INFO:tasks.rados.rados.0.smithi038.stdout:1127: writing smithi03829856-5 from 2888822 to 2991646 tid 3 2024-03-19T11:30:53.108 INFO:tasks.rados.rados.0.smithi038.stdout:1128: write oid 10 current snap is 19 2024-03-19T11:30:53.109 INFO:tasks.rados.rados.0.smithi038.stdout:1123: got (0) Success 2024-03-19T11:30:53.109 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 12 v 83 (ObjNum 161 snap 17 seq_num 161) dirty exists 2024-03-19T11:30:53.109 INFO:tasks.rados.rados.0.smithi038.stdout:1125: expect (ObjNum 62 snap 3 seq_num 62) 2024-03-19T11:30:53.111 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-19T11:30:53.177 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:53.186+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:53.262 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-19T11:30:53.276 INFO:tasks.rados.rados.0.smithi038.stdout:1122: finishing write tid 1 to smithi03829856-48 2024-03-19T11:30:53.276 INFO:tasks.rados.rados.0.smithi038.stdout:1122: finishing write tid 2 to smithi03829856-48 2024-03-19T11:30:53.276 INFO:tasks.rados.rados.0.smithi038.stdout:1128: seq_num 179 ranges {608453=545690,1624865=455728,2632598=375855} 2024-03-19T11:30:53.277 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:53.286+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:53.282 INFO:tasks.rados.rados.0.smithi038.stdout:1128: writing smithi03829856-10 from 608453 to 1154143 tid 1 2024-03-19T11:30:53.288 INFO:tasks.rados.rados.0.smithi038.stdout:1128: writing smithi03829856-10 from 1624865 to 2080593 tid 2 2024-03-19T11:30:53.289 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:53.289 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:53.289 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-03-19T11:30:53.294 INFO:tasks.rados.rados.0.smithi038.stdout:1128: writing smithi03829856-10 from 2632598 to 3008453 tid 3 2024-03-19T11:30:53.294 INFO:tasks.rados.rados.0.smithi038.stdout:1123: done (7 left) 2024-03-19T11:30:53.294 INFO:tasks.rados.rados.0.smithi038.stdout:1125: done (6 left) 2024-03-19T11:30:53.294 INFO:tasks.rados.rados.0.smithi038.stdout:1126: done (5 left) 2024-03-19T11:30:53.294 INFO:tasks.rados.rados.0.smithi038.stdout:1129: read oid 38 snap -1 2024-03-19T11:30:53.294 INFO:tasks.rados.rados.0.smithi038.stdout:1129: expect (ObjNum 125 snap 15 seq_num 125) 2024-03-19T11:30:53.294 INFO:tasks.rados.rados.0.smithi038.stdout:1130: read oid 36 snap 9 2024-03-19T11:30:53.294 INFO:tasks.rados.rados.0.smithi038.stdout:1130: expect (ObjNum 35 snap 0 seq_num 35) 2024-03-19T11:30:53.294 INFO:tasks.rados.rados.0.smithi038.stdout:1131: write (excl) oid 13 current snap is 19 2024-03-19T11:30:53.294 INFO:tasks.rados.rados.0.smithi038.stdout:1131: seq_num 180 ranges {625260=620939,1683347=599386,2943668=81592} 2024-03-19T11:30:53.302 INFO:tasks.rados.rados.0.smithi038.stdout:1131: writing smithi03829856-13 from 625260 to 1246199 tid 1 2024-03-19T11:30:53.308 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-19T11:30:53.309 INFO:tasks.rados.rados.0.smithi038.stdout:1131: writing smithi03829856-13 from 1683347 to 2282733 tid 2 2024-03-19T11:30:53.312 INFO:tasks.rados.rados.0.smithi038.stdout:1131: writing smithi03829856-13 from 2943668 to 3025260 tid 3 2024-03-19T11:30:53.312 INFO:tasks.rados.rados.0.smithi038.stdout:1132: write (excl) oid 17 current snap is 19 2024-03-19T11:30:53.312 INFO:tasks.rados.rados.0.smithi038.stdout:1122: finishing write tid 3 to smithi03829856-48 2024-03-19T11:30:53.312 INFO:tasks.rados.rados.0.smithi038.stdout:1122: finishing write tid 4 to smithi03829856-48 2024-03-19T11:30:53.312 INFO:tasks.rados.rados.0.smithi038.stdout:1122: oid 48 updating version 0 to 85 2024-03-19T11:30:53.312 INFO:tasks.rados.rados.0.smithi038.stdout:1122: oid 48 updating version 85 to 86 2024-03-19T11:30:53.312 INFO:tasks.rados.rados.0.smithi038.stdout:1122: oid 48 version 86 is already newer than 84 2024-03-19T11:30:53.312 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 48 v 86 (ObjNum 177 snap 19 seq_num 177) dirty exists 2024-03-19T11:30:53.312 INFO:tasks.rados.rados.0.smithi038.stdout:1122: left oid 48 (ObjNum 177 snap 19 seq_num 177) 2024-03-19T11:30:53.312 INFO:tasks.rados.rados.0.smithi038.stdout:1120: expect (ObjNum 31 snap 0 seq_num 31) 2024-03-19T11:30:53.378 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:53.383+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:53.392 INFO:tasks.rados.rados.0.smithi038.stdout:1127: finishing write tid 1 to smithi03829856-5 2024-03-19T11:30:53.392 INFO:tasks.rados.rados.0.smithi038.stdout:1127: finishing write tid 2 to smithi03829856-5 2024-03-19T11:30:53.392 INFO:tasks.rados.rados.0.smithi038.stdout:1127: finishing write tid 3 to smithi03829856-5 2024-03-19T11:30:53.392 INFO:tasks.rados.rados.0.smithi038.stdout:1127: finishing write tid 4 to smithi03829856-5 2024-03-19T11:30:53.392 INFO:tasks.rados.rados.0.smithi038.stdout:1132: seq_num 181 ranges {642067=696188,2025476=459397,2971091=70976} 2024-03-19T11:30:53.399 INFO:tasks.rados.rados.0.smithi038.stdout:1132: writing smithi03829856-17 from 642067 to 1338255 tid 1 2024-03-19T11:30:53.407 INFO:tasks.rados.rados.0.smithi038.stdout:1132: writing smithi03829856-17 from 2025476 to 2484873 tid 2 2024-03-19T11:30:53.412 INFO:tasks.rados.rados.0.smithi038.stdout:1132: writing smithi03829856-17 from 2971091 to 3042067 tid 3 2024-03-19T11:30:53.412 INFO:tasks.rados.rados.0.smithi038.stdout:1120: done (8 left) 2024-03-19T11:30:53.412 INFO:tasks.rados.rados.0.smithi038.stdout:1122: done (7 left) 2024-03-19T11:30:53.412 INFO:tasks.rados.rados.0.smithi038.stdout:1133: tier_flush oid 25 2024-03-19T11:30:53.412 INFO:tasks.rados.rados.0.smithi038.stdout:1133: tier_flush oid 25 snap -1 2024-03-19T11:30:53.412 INFO:tasks.rados.rados.0.smithi038.stdout:1127: finishing write tid 5 to smithi03829856-5 2024-03-19T11:30:53.412 INFO:tasks.rados.rados.0.smithi038.stdout:1127: oid 5 updating version 0 to 97 2024-03-19T11:30:53.412 INFO:tasks.rados.rados.0.smithi038.stdout:1127: oid 5 updating version 97 to 98 2024-03-19T11:30:53.413 INFO:tasks.rados.rados.0.smithi038.stdout:1127: oid 5 version 98 is already newer than 96 2024-03-19T11:30:53.413 INFO:tasks.rados.rados.0.smithi038.stdout:1127: oid 5 updating version 98 to 99 2024-03-19T11:30:53.413 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 5 v 99 (ObjNum 178 snap 19 seq_num 178) dirty exists 2024-03-19T11:30:53.413 INFO:tasks.rados.rados.0.smithi038.stdout:1127: left oid 5 (ObjNum 178 snap 19 seq_num 178) 2024-03-19T11:30:53.413 INFO:tasks.rados.rados.0.smithi038.stdout:1128: finishing write tid 1 to smithi03829856-10 2024-03-19T11:30:53.413 INFO:tasks.rados.rados.0.smithi038.stdout:1127: done (7 left) 2024-03-19T11:30:53.413 INFO:tasks.rados.rados.0.smithi038.stdout:1134: read oid 9 snap -1 2024-03-19T11:30:53.413 INFO:tasks.rados.rados.0.smithi038.stdout:1134: expect (ObjNum 152 snap 17 seq_num 152) 2024-03-19T11:30:53.413 INFO:tasks.rados.rados.0.smithi038.stdout:1135: tier_promote oid 18 2024-03-19T11:30:53.413 INFO:tasks.rados.rados.0.smithi038.stdout:1128: finishing write tid 2 to smithi03829856-10 2024-03-19T11:30:53.413 INFO:tasks.rados.rados.0.smithi038.stdout:1130: expect (ObjNum 35 snap 0 seq_num 35) 2024-03-19T11:30:53.423 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:30:53.424 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:30:53.430 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:53.430 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:53.442 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:30:53.442 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:30:53.444 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-19T11:30:53.460 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:30:53.461 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:30:53.477 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:30:53.477 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:30:53.478 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:53.486+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:53.497 INFO:tasks.rados.rados.0.smithi038.stdout:1129: expect (ObjNum 125 snap 15 seq_num 125) 2024-03-19T11:30:53.497 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:30:53.498 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:30:53.517 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:30:53.517 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:30:53.535 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:30:53.535 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:30:53.553 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:30:53.553 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:30:53.580 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:53.587+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:53.584 INFO:tasks.rados.rados.0.smithi038.stdout:1128: finishing write tid 3 to smithi03829856-10 2024-03-19T11:30:53.584 INFO:tasks.rados.rados.0.smithi038.stdout:1128: finishing write tid 4 to smithi03829856-10 2024-03-19T11:30:53.584 INFO:tasks.rados.rados.0.smithi038.stdout:1128: finishing write tid 5 to smithi03829856-10 2024-03-19T11:30:53.584 INFO:tasks.rados.rados.0.smithi038.stdout:1128: oid 10 updating version 0 to 88 2024-03-19T11:30:53.584 INFO:tasks.rados.rados.0.smithi038.stdout:1128: oid 10 updating version 88 to 89 2024-03-19T11:30:53.584 INFO:tasks.rados.rados.0.smithi038.stdout:1128: oid 10 version 89 is already newer than 86 2024-03-19T11:30:53.584 INFO:tasks.rados.rados.0.smithi038.stdout:1128: oid 10 version 89 is already newer than 87 2024-03-19T11:30:53.584 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 10 v 89 (ObjNum 179 snap 19 seq_num 179) dirty exists 2024-03-19T11:30:53.584 INFO:tasks.rados.rados.0.smithi038.stdout:1128: left oid 10 (ObjNum 179 snap 19 seq_num 179) 2024-03-19T11:30:53.584 INFO:tasks.rados.rados.0.smithi038.stdout:1131: finishing write tid 1 to smithi03829856-13 2024-03-19T11:30:53.584 INFO:tasks.rados.rados.0.smithi038.stdout:1131: finishing write tid 2 to smithi03829856-13 2024-03-19T11:30:53.584 INFO:tasks.rados.rados.0.smithi038.stdout:1131: finishing write tid 3 to smithi03829856-13 2024-03-19T11:30:53.584 INFO:tasks.rados.rados.0.smithi038.stdout:1131: finishing write tid 4 to smithi03829856-13 2024-03-19T11:30:53.585 INFO:tasks.rados.rados.0.smithi038.stdout:1128: done (8 left) 2024-03-19T11:30:53.585 INFO:tasks.rados.rados.0.smithi038.stdout:1129: done (7 left) 2024-03-19T11:30:53.585 INFO:tasks.rados.rados.0.smithi038.stdout:1130: done (6 left) 2024-03-19T11:30:53.585 INFO:tasks.rados.rados.0.smithi038.stdout:1136: set_chunk oid 35 target oid 2024-03-19T11:30:53.610 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:53.610 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:53.610 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:30:53.611 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:30:53.611 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:53.625 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-19T11:30:53.680 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:53.690+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:53.781 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:53.790+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:53.793 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:53.793 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:30:53.793 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:30:53.793 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:30:53.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.793 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1047 4.8 4:1d58db11:::smithi03829856-18:head [write 443718~615603,stat] snapc 6=[6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:53.794 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:13.221766+0000", 2024-03-19T11:30:53.794 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 40.580029484000001, 2024-03-19T11:30:53.794 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020932596000000001, 2024-03-19T11:30:53.794 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:53.794 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:53.794 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:53.794 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:53.794 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:53.794 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1047 2024-03-19T11:30:53.794 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.794 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:53.794 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.794 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:53.794 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.221766+0000", 2024-03-19T11:30:53.795 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:53.795 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.795 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:53.795 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.221766+0000", 2024-03-19T11:30:53.795 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:53.795 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.795 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:53.795 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.221770+0000", 2024-03-19T11:30:53.795 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.67e-06 2024-03-19T11:30:53.795 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.795 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:53.796 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.223938+0000", 2024-03-19T11:30:53.796 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021683399999999999 2024-03-19T11:30:53.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.796 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.796 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:53.796 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.223979+0000", 2024-03-19T11:30:53.796 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1433000000000001e-05 2024-03-19T11:30:53.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.796 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.796 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:53.796 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.223998+0000", 2024-03-19T11:30:53.796 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8853000000000001e-05 2024-03-19T11:30:53.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.796 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.796 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:53.797 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.224150+0000", 2024-03-19T11:30:53.797 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000151679 2024-03-19T11:30:53.797 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.797 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.797 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:53.797 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.224362+0000", 2024-03-19T11:30:53.797 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021154499999999999 2024-03-19T11:30:53.797 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.797 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.797 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:53.797 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.224745+0000", 2024-03-19T11:30:53.797 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00038342399999999999 2024-03-19T11:30:53.797 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.797 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.797 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:53.798 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.241167+0000", 2024-03-19T11:30:53.798 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0164218 2024-03-19T11:30:53.798 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.798 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.798 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:53.798 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.242406+0000", 2024-03-19T11:30:53.798 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012390039999999999 2024-03-19T11:30:53.798 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.798 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.798 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:53.798 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.242575+0000", 2024-03-19T11:30:53.798 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000168925 2024-03-19T11:30:53.798 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.799 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.799 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:53.799 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.242699+0000", 2024-03-19T11:30:53.799 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012392300000000001 2024-03-19T11:30:53.799 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:53.799 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:53.799 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:53.799 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.799 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.799 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1048 4.8 4:1d58db11:::smithi03829856-18:head [write 1564138~470341] snapc 6=[6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:53.799 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:13.227287+0000", 2024-03-19T11:30:53.799 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 40.574508215999998, 2024-03-19T11:30:53.799 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015802897999999999, 2024-03-19T11:30:53.799 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:53.799 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:53.800 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:53.800 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:53.800 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:53.800 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1048 2024-03-19T11:30:53.800 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.800 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:53.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.800 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:53.800 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.227287+0000", 2024-03-19T11:30:53.800 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:53.800 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.800 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:53.800 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.227287+0000", 2024-03-19T11:30:53.800 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:53.800 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.801 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.801 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:53.801 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.227290+0000", 2024-03-19T11:30:53.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.88e-06 2024-03-19T11:30:53.801 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.801 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.801 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:53.801 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.228224+0000", 2024-03-19T11:30:53.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00093389600000000003 2024-03-19T11:30:53.801 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.801 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.801 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:53.801 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.228258+0000", 2024-03-19T11:30:53.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4282999999999997e-05 2024-03-19T11:30:53.801 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.802 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.802 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:53.802 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.228273+0000", 2024-03-19T11:30:53.802 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4833e-05 2024-03-19T11:30:53.802 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.802 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.802 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:53.802 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.231996+0000", 2024-03-19T11:30:53.802 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037223410000000001 2024-03-19T11:30:53.802 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.802 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.802 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:53.802 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.232143+0000", 2024-03-19T11:30:53.802 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014714900000000001 2024-03-19T11:30:53.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.803 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:53.803 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.232395+0000", 2024-03-19T11:30:53.803 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025201500000000002 2024-03-19T11:30:53.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.803 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:53.803 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.242848+0000", 2024-03-19T11:30:53.803 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010453393 2024-03-19T11:30:53.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.803 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:53.803 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.242872+0000", 2024-03-19T11:30:53.803 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3816e-05 2024-03-19T11:30:53.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.804 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.804 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:53.804 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.242966+0000", 2024-03-19T11:30:53.804 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4560000000000003e-05 2024-03-19T11:30:53.804 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.804 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.804 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:53.804 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.243090+0000", 2024-03-19T11:30:53.804 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012373199999999999 2024-03-19T11:30:53.804 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:53.804 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:53.804 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:53.804 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.804 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.804 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1072 4.1 e36/21 4:8401c1f0:::smithi03829856-43:head v 36'38, mlcod=36'35)", 2024-03-19T11:30:53.805 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:13.973829+0000", 2024-03-19T11:30:53.805 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 39.827966893000003, 2024-03-19T11:30:53.805 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013548782000000001, 2024-03-19T11:30:53.805 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:53.805 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:53.805 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:53.805 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.805 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:53.805 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.973829+0000", 2024-03-19T11:30:53.805 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:53.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.805 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.805 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:53.805 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.973829+0000", 2024-03-19T11:30:53.805 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:53.806 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.806 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.806 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:53.806 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.973831+0000", 2024-03-19T11:30:53.806 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0899999999999999e-06 2024-03-19T11:30:53.806 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.806 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.806 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:53.806 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.974489+0000", 2024-03-19T11:30:53.806 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00065813199999999997 2024-03-19T11:30:53.807 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.807 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:53.807 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.974523+0000", 2024-03-19T11:30:53.807 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3841999999999999e-05 2024-03-19T11:30:53.807 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.807 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:53.807 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.974544+0000", 2024-03-19T11:30:53.807 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0907000000000002e-05 2024-03-19T11:30:53.807 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.807 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:53.807 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.974697+0000", 2024-03-19T11:30:53.808 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015295599999999999 2024-03-19T11:30:53.808 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.808 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.808 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:53.808 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.974742+0000", 2024-03-19T11:30:53.808 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5735999999999997e-05 2024-03-19T11:30:53.808 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.808 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.808 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:53.808 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.987261+0000", 2024-03-19T11:30:53.808 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012518316999999999 2024-03-19T11:30:53.808 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.808 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.808 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:53.808 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.987377+0000", 2024-03-19T11:30:53.809 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000116802 2024-03-19T11:30:53.809 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:53.809 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:53.809 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:53.809 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.809 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.809 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1091 4.8 4:14611006:::smithi03829856-25:head [write 527753~708201] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:53.809 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:14.015839+0000", 2024-03-19T11:30:53.809 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 39.785956833999997, 2024-03-19T11:30:53.809 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016721983999999999, 2024-03-19T11:30:53.809 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:53.809 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:53.809 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:53.810 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:53.810 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:53.810 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1091 2024-03-19T11:30:53.810 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.810 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:53.810 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.810 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:53.810 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.015839+0000", 2024-03-19T11:30:53.810 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:53.810 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.810 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.810 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:53.810 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.015839+0000", 2024-03-19T11:30:53.810 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:53.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.811 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:53.811 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.015842+0000", 2024-03-19T11:30:53.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3569999999999998e-06 2024-03-19T11:30:53.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.811 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:53.811 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.016678+0000", 2024-03-19T11:30:53.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00083620299999999999 2024-03-19T11:30:53.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.811 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:53.811 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.016710+0000", 2024-03-19T11:30:53.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1337000000000002e-05 2024-03-19T11:30:53.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.812 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.812 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:53.812 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.016724+0000", 2024-03-19T11:30:53.812 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4323e-05 2024-03-19T11:30:53.812 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.812 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.812 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:53.812 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.016847+0000", 2024-03-19T11:30:53.812 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012304199999999999 2024-03-19T11:30:53.812 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.812 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.812 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:53.812 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.017023+0000", 2024-03-19T11:30:53.812 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000175945 2024-03-19T11:30:53.812 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.813 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.813 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:53.813 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.017324+0000", 2024-03-19T11:30:53.813 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00030151499999999998 2024-03-19T11:30:53.813 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.813 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.813 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:53.813 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.032104+0000", 2024-03-19T11:30:53.813 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014779232999999999 2024-03-19T11:30:53.813 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.813 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.814 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:53.814 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.032190+0000", 2024-03-19T11:30:53.814 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6304999999999994e-05 2024-03-19T11:30:53.814 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.814 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.814 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:53.814 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.032350+0000", 2024-03-19T11:30:53.814 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000159813 2024-03-19T11:30:53.814 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.814 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.814 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:53.814 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.032561+0000", 2024-03-19T11:30:53.814 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021091099999999999 2024-03-19T11:30:53.814 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:53.814 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:53.815 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:53.815 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.815 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.815 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1158 4.8 4:15708482:::smithi03829856-23:head [tier-promote] snapc 8=[8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-03-19T11:30:53.815 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.827517+0000", 2024-03-19T11:30:53.815 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 36.974278906000002, 2024-03-19T11:30:53.815 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.034096983999999997, 2024-03-19T11:30:53.815 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:53.815 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:53.815 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:53.815 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:53.815 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:53.815 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1158 2024-03-19T11:30:53.815 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.815 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:53.815 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:53.816 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827517+0000", 2024-03-19T11:30:53.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:53.816 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:53.816 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827517+0000", 2024-03-19T11:30:53.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:53.816 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:53.816 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827520+0000", 2024-03-19T11:30:53.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.45e-06 2024-03-19T11:30:53.816 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.817 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:53.817 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827535+0000", 2024-03-19T11:30:53.817 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.509e-05 2024-03-19T11:30:53.817 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.817 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.817 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:53.817 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827579+0000", 2024-03-19T11:30:53.817 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4211000000000001e-05 2024-03-19T11:30:53.817 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.817 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.817 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:53.817 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827603+0000", 2024-03-19T11:30:53.817 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3586000000000001e-05 2024-03-19T11:30:53.817 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.817 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.818 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:53.818 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827789+0000", 2024-03-19T11:30:53.818 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018580199999999999 2024-03-19T11:30:53.818 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.818 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.818 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:53.818 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.828077+0000", 2024-03-19T11:30:53.818 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028846500000000001 2024-03-19T11:30:53.818 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.818 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.818 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:53.818 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.861614+0000", 2024-03-19T11:30:53.818 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033536379999999998 2024-03-19T11:30:53.818 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:53.818 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:53.819 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:53.819 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.819 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.819 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1162 4.1 e39/21 4:8401c1f0:::smithi03829856-43:head v 39'47, mlcod=38'43)", 2024-03-19T11:30:53.819 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.842598+0000", 2024-03-19T11:30:53.819 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 36.959197136, 2024-03-19T11:30:53.819 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0233756, 2024-03-19T11:30:53.819 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:53.819 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:53.819 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:53.819 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:53.819 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842598+0000", 2024-03-19T11:30:53.819 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:53.819 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.819 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.820 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:53.820 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842598+0000", 2024-03-19T11:30:53.820 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:53.820 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.820 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.820 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:53.820 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842599+0000", 2024-03-19T11:30:53.820 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6000000000000003e-07 2024-03-19T11:30:53.820 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.820 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.820 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:53.820 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.843566+0000", 2024-03-19T11:30:53.820 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00096688700000000004 2024-03-19T11:30:53.820 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.820 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.821 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:53.821 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.843601+0000", 2024-03-19T11:30:53.821 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.502e-05 2024-03-19T11:30:53.821 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.821 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.821 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:53.821 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.843621+0000", 2024-03-19T11:30:53.821 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0305e-05 2024-03-19T11:30:53.821 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.821 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.821 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:53.821 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.843882+0000", 2024-03-19T11:30:53.821 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026027100000000002 2024-03-19T11:30:53.821 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.821 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.822 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:53.822 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.844022+0000", 2024-03-19T11:30:53.822 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014020100000000001 2024-03-19T11:30:53.822 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.822 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.822 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:53.822 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.865862+0000", 2024-03-19T11:30:53.822 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021839725000000001 2024-03-19T11:30:53.822 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.822 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.822 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:53.823 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.865974+0000", 2024-03-19T11:30:53.823 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011243099999999999 2024-03-19T11:30:53.823 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:53.823 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:53.823 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:53.823 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.823 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.823 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1209 4.8 4:1d58db11:::smithi03829856-18:head [write 645402~551297,stat] snapc 9=[9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:53.823 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:18.121657+0000", 2024-03-19T11:30:53.823 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 35.680138134000003, 2024-03-19T11:30:53.823 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.025074349999999999, 2024-03-19T11:30:53.823 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:53.823 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:53.823 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:53.823 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:53.824 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:53.824 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1209 2024-03-19T11:30:53.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.824 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:53.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.824 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:53.824 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.121657+0000", 2024-03-19T11:30:53.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:53.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.824 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:53.824 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.121657+0000", 2024-03-19T11:30:53.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:53.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.825 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:53.825 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.121661+0000", 2024-03-19T11:30:53.825 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0489999999999997e-06 2024-03-19T11:30:53.825 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.825 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.825 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:53.825 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123747+0000", 2024-03-19T11:30:53.825 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0020857720000000001 2024-03-19T11:30:53.825 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.825 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.825 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:53.825 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123779+0000", 2024-03-19T11:30:53.825 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2258000000000002e-05 2024-03-19T11:30:53.826 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.826 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.826 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:53.826 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123795+0000", 2024-03-19T11:30:53.826 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5177000000000001e-05 2024-03-19T11:30:53.826 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.826 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.826 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:53.826 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123949+0000", 2024-03-19T11:30:53.826 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015403699999999999 2024-03-19T11:30:53.826 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.826 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.826 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:53.826 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.124134+0000", 2024-03-19T11:30:53.826 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000185484 2024-03-19T11:30:53.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.827 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.827 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:53.827 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.124554+0000", 2024-03-19T11:30:53.827 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00042019699999999998 2024-03-19T11:30:53.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.827 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.827 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:53.827 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146267+0000", 2024-03-19T11:30:53.827 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021712838000000002 2024-03-19T11:30:53.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.827 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.827 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:53.827 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146424+0000", 2024-03-19T11:30:53.827 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015647299999999999 2024-03-19T11:30:53.828 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.828 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.828 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:53.828 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146597+0000", 2024-03-19T11:30:53.828 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017303699999999999 2024-03-19T11:30:53.828 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.828 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.828 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:53.828 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146732+0000", 2024-03-19T11:30:53.828 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013502800000000001 2024-03-19T11:30:53.828 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:53.828 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:53.828 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:53.828 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.828 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.829 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1210 4.8 4:1d58db11:::smithi03829856-18:head [write 1749569~492355] snapc 9=[9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:53.829 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:18.129257+0000", 2024-03-19T11:30:53.829 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 35.672538777, 2024-03-19T11:30:53.829 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018110444999999999, 2024-03-19T11:30:53.829 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:53.829 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:53.829 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:53.829 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:53.829 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:53.829 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1210 2024-03-19T11:30:53.829 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.829 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:53.829 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.829 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:53.829 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.129257+0000", 2024-03-19T11:30:53.829 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:53.830 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.830 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.830 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:53.830 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.129257+0000", 2024-03-19T11:30:53.830 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:53.830 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.830 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.830 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:53.830 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.129261+0000", 2024-03-19T11:30:53.830 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7119999999999999e-06 2024-03-19T11:30:53.830 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.830 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.830 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:53.830 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.129717+0000", 2024-03-19T11:30:53.831 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00045526300000000002 2024-03-19T11:30:53.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.831 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:53.831 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.129749+0000", 2024-03-19T11:30:53.831 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2700999999999997e-05 2024-03-19T11:30:53.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.831 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:53.831 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.129774+0000", 2024-03-19T11:30:53.831 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4592000000000001e-05 2024-03-19T11:30:53.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.831 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:53.831 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.134875+0000", 2024-03-19T11:30:53.831 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.005100477 2024-03-19T11:30:53.832 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.832 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:53.832 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.135035+0000", 2024-03-19T11:30:53.832 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016054200000000001 2024-03-19T11:30:53.832 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.832 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:53.832 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.135268+0000", 2024-03-19T11:30:53.832 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000233253 2024-03-19T11:30:53.832 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.832 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:53.832 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146872+0000", 2024-03-19T11:30:53.832 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011603732 2024-03-19T11:30:53.833 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.833 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.833 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:53.833 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.147103+0000", 2024-03-19T11:30:53.833 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023070299999999999 2024-03-19T11:30:53.833 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.833 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.833 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:53.833 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.147181+0000", 2024-03-19T11:30:53.833 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8567999999999997e-05 2024-03-19T11:30:53.833 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.833 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.833 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:53.833 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.147367+0000", 2024-03-19T11:30:53.834 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000185902 2024-03-19T11:30:53.834 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:53.834 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:53.834 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:53.834 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.834 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.834 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1259 4.1 e44/43 4:866f3dcc:::smithi03829856-31:head v 44'53, mlcod=0'0)", 2024-03-19T11:30:53.834 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:21.869109+0000", 2024-03-19T11:30:53.834 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 31.932686886999999, 2024-03-19T11:30:53.834 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014777017999999999, 2024-03-19T11:30:53.834 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:53.834 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:53.834 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:53.834 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.834 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:53.835 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.869109+0000", 2024-03-19T11:30:53.835 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:53.835 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.835 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.835 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:53.835 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.869109+0000", 2024-03-19T11:30:53.835 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:53.835 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.835 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.835 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:53.835 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.869110+0000", 2024-03-19T11:30:53.835 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0559999999999999e-06 2024-03-19T11:30:53.835 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.835 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.835 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:53.836 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.870408+0000", 2024-03-19T11:30:53.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012978499999999999 2024-03-19T11:30:53.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.836 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:53.836 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.870437+0000", 2024-03-19T11:30:53.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9638999999999999e-05 2024-03-19T11:30:53.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.836 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:53.836 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.870460+0000", 2024-03-19T11:30:53.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2517000000000001e-05 2024-03-19T11:30:53.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.836 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:53.837 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.870681+0000", 2024-03-19T11:30:53.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022102800000000001 2024-03-19T11:30:53.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.837 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:53.837 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.870735+0000", 2024-03-19T11:30:53.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4616e-05 2024-03-19T11:30:53.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.837 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:53.837 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.883773+0000", 2024-03-19T11:30:53.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013038088 2024-03-19T11:30:53.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.837 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:53.838 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.883886+0000", 2024-03-19T11:30:53.838 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000112224 2024-03-19T11:30:53.838 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:53.838 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:53.838 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:53.838 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.838 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.838 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1416 4.18 4:1dabc0f1:::smithi03829856-1:head [write 564735~613734,stat] snapc c=[c,b,a,6,5] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:53.838 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.949508+0000", 2024-03-19T11:30:53.838 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.852287557, 2024-03-19T11:30:53.838 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016253106, 2024-03-19T11:30:53.838 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:53.838 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:53.839 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:53.839 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:53.839 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:53.839 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1416 2024-03-19T11:30:53.839 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.839 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:53.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.839 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:53.839 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.949508+0000", 2024-03-19T11:30:53.839 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:53.839 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.839 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:53.839 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.949508+0000", 2024-03-19T11:30:53.839 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:53.840 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.840 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.840 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:53.840 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.949511+0000", 2024-03-19T11:30:53.840 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2370000000000001e-06 2024-03-19T11:30:53.840 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.840 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.840 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:53.840 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.950555+0000", 2024-03-19T11:30:53.840 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001043925 2024-03-19T11:30:53.840 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.840 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.840 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:53.840 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.950594+0000", 2024-03-19T11:30:53.840 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9351999999999999e-05 2024-03-19T11:30:53.840 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.841 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.841 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:53.841 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.950612+0000", 2024-03-19T11:30:53.841 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7524e-05 2024-03-19T11:30:53.841 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.841 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.841 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:53.841 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.950745+0000", 2024-03-19T11:30:53.841 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013291 2024-03-19T11:30:53.841 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.841 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.841 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:53.841 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.951003+0000", 2024-03-19T11:30:53.841 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025828400000000001 2024-03-19T11:30:53.841 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.842 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.842 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:53.842 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.951335+0000", 2024-03-19T11:30:53.842 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033130200000000001 2024-03-19T11:30:53.842 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.842 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.842 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:53.842 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.965379+0000", 2024-03-19T11:30:53.842 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014044581 2024-03-19T11:30:53.842 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.842 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.842 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:53.842 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.965502+0000", 2024-03-19T11:30:53.842 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012290099999999999 2024-03-19T11:30:53.842 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.843 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:53.843 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.965668+0000", 2024-03-19T11:30:53.843 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016619200000000001 2024-03-19T11:30:53.843 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.843 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:53.843 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.965761+0000", 2024-03-19T11:30:53.843 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2898000000000004e-05 2024-03-19T11:30:53.843 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:53.843 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:53.843 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:53.843 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.843 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1417 4.18 4:1dabc0f1:::smithi03829856-1:head [write 1828021~669034] snapc c=[c,b,a,6,5] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:53.844 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.958407+0000", 2024-03-19T11:30:53.844 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.843388156, 2024-03-19T11:30:53.844 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016467840000000001, 2024-03-19T11:30:53.844 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:53.844 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:53.844 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:53.844 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:53.844 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:53.844 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1417 2024-03-19T11:30:53.844 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.844 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:53.844 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.844 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:53.844 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.958407+0000", 2024-03-19T11:30:53.844 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:53.844 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.845 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.845 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:53.845 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.958407+0000", 2024-03-19T11:30:53.845 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:53.845 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.845 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.845 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:53.845 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.958410+0000", 2024-03-19T11:30:53.845 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9830000000000001e-06 2024-03-19T11:30:53.845 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.845 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.845 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:53.845 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.959442+0000", 2024-03-19T11:30:53.845 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001031957 2024-03-19T11:30:53.845 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.846 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.846 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:53.846 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.959475+0000", 2024-03-19T11:30:53.846 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3145999999999997e-05 2024-03-19T11:30:53.846 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.846 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.846 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:53.846 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.959494+0000", 2024-03-19T11:30:53.846 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8994000000000001e-05 2024-03-19T11:30:53.846 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.846 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.846 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:53.846 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.959717+0000", 2024-03-19T11:30:53.846 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000222272 2024-03-19T11:30:53.846 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.847 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.847 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:53.847 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.960015+0000", 2024-03-19T11:30:53.847 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029870599999999998 2024-03-19T11:30:53.847 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.847 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.847 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:53.847 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.960312+0000", 2024-03-19T11:30:53.847 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029640299999999999 2024-03-19T11:30:53.847 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.847 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.847 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:53.847 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.966435+0000", 2024-03-19T11:30:53.847 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061234999999999996 2024-03-19T11:30:53.847 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.848 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.848 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:53.848 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.974556+0000", 2024-03-19T11:30:53.848 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.008120759 2024-03-19T11:30:53.848 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.848 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.848 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:53.848 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.974729+0000", 2024-03-19T11:30:53.848 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000173191 2024-03-19T11:30:53.848 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.848 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.848 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:53.848 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.974875+0000", 2024-03-19T11:30:53.848 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014592899999999999 2024-03-19T11:30:53.848 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:53.849 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:53.849 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:53.849 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.849 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.849 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1443 4.2 4:43b32242:::smithi03829856-22:head [read 0~2352947,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:53.849 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.979417+0000", 2024-03-19T11:30:53.849 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.822378461, 2024-03-19T11:30:53.849 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.96719988700000004, 2024-03-19T11:30:53.849 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:53.849 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:53.849 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:53.849 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:53.849 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:53.849 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1443 2024-03-19T11:30:53.849 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.850 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:53.850 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.850 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:53.850 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979417+0000", 2024-03-19T11:30:53.850 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:53.850 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.850 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.850 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:53.850 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979417+0000", 2024-03-19T11:30:53.850 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:53.850 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.850 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.850 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:53.850 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979420+0000", 2024-03-19T11:30:53.850 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.366e-06 2024-03-19T11:30:53.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.851 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.851 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:53.851 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979437+0000", 2024-03-19T11:30:53.851 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6192e-05 2024-03-19T11:30:53.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.851 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.851 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:53.851 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979474+0000", 2024-03-19T11:30:53.851 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7601000000000001e-05 2024-03-19T11:30:53.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.851 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.851 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:53.851 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979496+0000", 2024-03-19T11:30:53.851 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1506999999999999e-05 2024-03-19T11:30:53.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.852 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.852 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:53.852 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979712+0000", 2024-03-19T11:30:53.852 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021641299999999999 2024-03-19T11:30:53.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.852 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.852 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:30:53.852 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979746+0000", 2024-03-19T11:30:53.852 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3386000000000002e-05 2024-03-19T11:30:53.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.852 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.852 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:53.852 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.931552+0000", 2024-03-19T11:30:53.852 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.95180659899999998 2024-03-19T11:30:53.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.853 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.853 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-03-19T11:30:53.853 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.934681+0000", 2024-03-19T11:30:53.853 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0031292289999999999 2024-03-19T11:30:53.853 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.853 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.853 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:53.853 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.942701+0000", 2024-03-19T11:30:53.853 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0080198809999999995 2024-03-19T11:30:53.853 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.853 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.853 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:53.853 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.942839+0000", 2024-03-19T11:30:53.853 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013816999999999999 2024-03-19T11:30:53.853 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.854 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.854 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:53.854 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.946617+0000", 2024-03-19T11:30:53.854 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003777543 2024-03-19T11:30:53.854 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:53.854 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:53.854 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:53.854 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.854 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.854 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1444 4.2 4:43b32242:::smithi03829856-22:head [read 0~2352947,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:53.854 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.979553+0000", 2024-03-19T11:30:53.854 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.822242758000002, 2024-03-19T11:30:53.854 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.97015945400000003, 2024-03-19T11:30:53.854 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:53.855 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:53.855 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:53.855 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:53.855 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:53.855 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1444 2024-03-19T11:30:53.855 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.855 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:53.855 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.855 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:53.855 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979553+0000", 2024-03-19T11:30:53.855 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:53.855 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.855 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.855 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:53.855 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979553+0000", 2024-03-19T11:30:53.855 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:53.856 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.856 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.856 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:53.856 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979554+0000", 2024-03-19T11:30:53.856 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.403e-06 2024-03-19T11:30:53.856 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.856 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.856 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:53.856 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979564+0000", 2024-03-19T11:30:53.856 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0032e-05 2024-03-19T11:30:53.856 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.856 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.856 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:53.856 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979579+0000", 2024-03-19T11:30:53.856 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4875000000000001e-05 2024-03-19T11:30:53.857 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.857 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.857 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:53.857 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979595+0000", 2024-03-19T11:30:53.857 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6300999999999999e-05 2024-03-19T11:30:53.857 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.857 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.857 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:53.857 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979874+0000", 2024-03-19T11:30:53.857 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000279065 2024-03-19T11:30:53.857 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.857 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.857 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:30:53.857 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979904+0000", 2024-03-19T11:30:53.857 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.94e-05 2024-03-19T11:30:53.858 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.858 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.858 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:53.858 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.934781+0000", 2024-03-19T11:30:53.858 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.95487744900000004 2024-03-19T11:30:53.858 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.858 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.858 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-03-19T11:30:53.858 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.934946+0000", 2024-03-19T11:30:53.858 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016485999999999999 2024-03-19T11:30:53.858 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.858 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.858 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:53.859 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.946829+0000", 2024-03-19T11:30:53.859 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011882482999999999 2024-03-19T11:30:53.859 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.859 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.859 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:53.859 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.946989+0000", 2024-03-19T11:30:53.859 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000160421 2024-03-19T11:30:53.859 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.859 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.859 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:53.859 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.949712+0000", 2024-03-19T11:30:53.859 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002723165 2024-03-19T11:30:53.859 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:53.859 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:53.860 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:53.860 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.860 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.860 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1445 4.2 4:43b32242:::smithi03829856-22:head [read 0~2352947,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:53.860 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.979653+0000", 2024-03-19T11:30:53.860 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.822142627000002, 2024-03-19T11:30:53.860 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.97249507999999996, 2024-03-19T11:30:53.860 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:53.860 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:53.860 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:53.860 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:53.860 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:53.860 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1445 2024-03-19T11:30:53.860 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.860 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:53.861 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.861 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:53.861 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979653+0000", 2024-03-19T11:30:53.861 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:53.861 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.861 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.861 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:53.861 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979653+0000", 2024-03-19T11:30:53.861 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:53.861 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.861 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.861 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:53.861 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979654+0000", 2024-03-19T11:30:53.861 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.372e-06 2024-03-19T11:30:53.862 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.862 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.862 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:53.862 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979662+0000", 2024-03-19T11:30:53.862 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9959999999999993e-06 2024-03-19T11:30:53.862 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.862 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.862 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:53.862 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979698+0000", 2024-03-19T11:30:53.862 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5763000000000003e-05 2024-03-19T11:30:53.862 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.862 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.862 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:53.862 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979714+0000", 2024-03-19T11:30:53.862 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6283999999999999e-05 2024-03-19T11:30:53.863 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.863 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.863 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:53.863 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979990+0000", 2024-03-19T11:30:53.863 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000275829 2024-03-19T11:30:53.863 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.863 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.863 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:30:53.863 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.980018+0000", 2024-03-19T11:30:53.863 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8082999999999999e-05 2024-03-19T11:30:53.863 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.863 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.863 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:53.863 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.935620+0000", 2024-03-19T11:30:53.863 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.95560133800000002 2024-03-19T11:30:53.863 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.864 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.864 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-03-19T11:30:53.864 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.935795+0000", 2024-03-19T11:30:53.864 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000175656 2024-03-19T11:30:53.864 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.864 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.864 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:53.864 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.949835+0000", 2024-03-19T11:30:53.864 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014040083 2024-03-19T11:30:53.864 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.864 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.864 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:53.864 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.950002+0000", 2024-03-19T11:30:53.864 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016673 2024-03-19T11:30:53.864 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.865 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.865 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:53.865 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.952148+0000", 2024-03-19T11:30:53.865 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021459460000000001 2024-03-19T11:30:53.865 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:53.865 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:53.865 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:53.865 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.865 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.865 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1449 4.18 4:1d58db11:::smithi03829856-18:head [write 581542~688983,stat] snapc c=[c,b,a,6,5] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:53.865 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.989354+0000", 2024-03-19T11:30:53.865 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.812441174, 2024-03-19T11:30:53.865 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019237318, 2024-03-19T11:30:53.865 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:53.865 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:53.866 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:53.866 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:53.866 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:53.866 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1449 2024-03-19T11:30:53.866 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.866 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:53.866 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.866 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:53.866 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.989354+0000", 2024-03-19T11:30:53.866 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:53.866 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.866 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.866 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:53.866 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.989354+0000", 2024-03-19T11:30:53.866 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:53.867 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.867 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.867 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:53.867 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.989357+0000", 2024-03-19T11:30:53.867 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2740000000000002e-06 2024-03-19T11:30:53.867 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.867 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.867 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:53.867 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.990109+0000", 2024-03-19T11:30:53.867 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00075226500000000001 2024-03-19T11:30:53.867 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.867 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.867 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:53.867 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.990142+0000", 2024-03-19T11:30:53.867 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3328999999999999e-05 2024-03-19T11:30:53.868 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.868 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.868 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:53.868 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.990168+0000", 2024-03-19T11:30:53.868 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6016999999999998e-05 2024-03-19T11:30:53.868 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.868 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.868 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:53.868 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.990291+0000", 2024-03-19T11:30:53.868 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000122607 2024-03-19T11:30:53.868 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.868 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.868 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:53.868 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.990461+0000", 2024-03-19T11:30:53.868 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000169743 2024-03-19T11:30:53.869 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.869 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.869 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:53.869 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.990792+0000", 2024-03-19T11:30:53.869 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033135100000000002 2024-03-19T11:30:53.869 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.869 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.869 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:53.869 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.008238+0000", 2024-03-19T11:30:53.869 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017445591999999999 2024-03-19T11:30:53.870 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.870 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.870 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:53.870 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.008345+0000", 2024-03-19T11:30:53.870 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010762 2024-03-19T11:30:53.870 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.870 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.870 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:53.870 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.008436+0000", 2024-03-19T11:30:53.870 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1116e-05 2024-03-19T11:30:53.870 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.870 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.870 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:53.870 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.008592+0000", 2024-03-19T11:30:53.871 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015540399999999999 2024-03-19T11:30:53.871 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:53.871 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:53.871 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:53.871 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.871 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.871 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1450 4.18 4:1d58db11:::smithi03829856-18:head [write 1886503~412692] snapc c=[c,b,a,6,5] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:53.871 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.992973+0000", 2024-03-19T11:30:53.871 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.808822405000001, 2024-03-19T11:30:53.871 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016912580999999999, 2024-03-19T11:30:53.871 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:53.871 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:53.872 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:53.872 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:53.872 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:53.872 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1450 2024-03-19T11:30:53.872 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.872 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:53.872 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.872 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:53.872 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.992973+0000", 2024-03-19T11:30:53.872 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:53.872 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.872 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.872 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:53.872 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.992973+0000", 2024-03-19T11:30:53.873 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:53.873 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.873 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.873 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:53.873 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.992976+0000", 2024-03-19T11:30:53.873 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0060000000000001e-06 2024-03-19T11:30:53.873 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.873 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.873 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:53.873 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.993655+0000", 2024-03-19T11:30:53.873 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00067901099999999998 2024-03-19T11:30:53.873 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.873 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.873 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:53.873 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.993691+0000", 2024-03-19T11:30:53.874 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5546000000000002e-05 2024-03-19T11:30:53.874 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.874 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.874 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:53.874 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.993713+0000", 2024-03-19T11:30:53.874 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2277e-05 2024-03-19T11:30:53.874 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.874 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.874 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:53.874 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.997961+0000", 2024-03-19T11:30:53.874 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0042479659999999997 2024-03-19T11:30:53.874 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.874 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.874 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:53.874 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.998130+0000", 2024-03-19T11:30:53.875 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016908900000000001 2024-03-19T11:30:53.875 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.875 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.875 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:53.875 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.998397+0000", 2024-03-19T11:30:53.875 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026707599999999999 2024-03-19T11:30:53.875 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.875 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.875 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:53.875 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.008811+0000", 2024-03-19T11:30:53.875 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010413751000000001 2024-03-19T11:30:53.875 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.875 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.875 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:53.875 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.009539+0000", 2024-03-19T11:30:53.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00072811499999999999 2024-03-19T11:30:53.876 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.876 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.876 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:53.876 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.009711+0000", 2024-03-19T11:30:53.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017196900000000001 2024-03-19T11:30:53.876 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.876 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.876 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:53.876 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.009886+0000", 2024-03-19T11:30:53.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017477500000000001 2024-03-19T11:30:53.876 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:53.876 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:53.876 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:53.876 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.876 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.877 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2207 4.28 4:15708482:::smithi03829856-23:head [write 541225~644694,stat] snapc 13=[13,12,11,d,b,6] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-03-19T11:30:53.877 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:51.160660+0000", 2024-03-19T11:30:53.877 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 2.6411358009999999, 2024-03-19T11:30:53.877 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.025400267000000001, 2024-03-19T11:30:53.877 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:53.877 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:53.877 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:53.877 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:53.877 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:53.877 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2207 2024-03-19T11:30:53.877 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.877 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:53.877 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.877 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:53.877 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.160660+0000", 2024-03-19T11:30:53.878 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:53.878 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.878 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.878 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:53.878 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.160660+0000", 2024-03-19T11:30:53.878 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:53.878 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.878 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.878 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:53.878 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.160688+0000", 2024-03-19T11:30:53.878 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8058000000000001e-05 2024-03-19T11:30:53.878 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.878 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.878 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:53.878 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.168614+0000", 2024-03-19T11:30:53.879 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079262810000000003 2024-03-19T11:30:53.879 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.879 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.879 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:53.879 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.168662+0000", 2024-03-19T11:30:53.879 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8327e-05 2024-03-19T11:30:53.879 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.879 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.879 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:53.879 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.168679+0000", 2024-03-19T11:30:53.879 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6305999999999999e-05 2024-03-19T11:30:53.879 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.879 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.879 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:53.879 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.168830+0000", 2024-03-19T11:30:53.880 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000151714 2024-03-19T11:30:53.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.880 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.880 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:53.880 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.169107+0000", 2024-03-19T11:30:53.880 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027608299999999999 2024-03-19T11:30:53.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.880 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.880 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:53.880 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.169605+0000", 2024-03-19T11:30:53.880 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00049874999999999995 2024-03-19T11:30:53.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.880 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.880 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:53.880 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.185769+0000", 2024-03-19T11:30:53.881 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016163397999999999 2024-03-19T11:30:53.881 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.881 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.881 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:53.881 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.185809+0000", 2024-03-19T11:30:53.881 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0598e-05 2024-03-19T11:30:53.881 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.881 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.881 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:53.881 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.185898+0000", 2024-03-19T11:30:53.881 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8270999999999996e-05 2024-03-19T11:30:53.881 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.881 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.881 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:53.881 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.186060+0000", 2024-03-19T11:30:53.881 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016248099999999999 2024-03-19T11:30:53.882 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:53.882 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:53.882 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:53.882 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.882 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.882 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2208 4.28 4:15708482:::smithi03829856-23:head [write 1907290~448390] snapc 13=[13,12,11,d,b,6] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-03-19T11:30:53.882 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:51.168854+0000", 2024-03-19T11:30:53.882 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 2.6329417470000003, 2024-03-19T11:30:53.882 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020462595, 2024-03-19T11:30:53.882 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:53.882 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:53.882 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:53.882 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:53.882 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:53.882 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2208 2024-03-19T11:30:53.883 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.883 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:53.883 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.883 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:53.883 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.168854+0000", 2024-03-19T11:30:53.883 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:53.883 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.883 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.883 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:53.883 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.168854+0000", 2024-03-19T11:30:53.883 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:53.883 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.883 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.883 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:53.883 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.168856+0000", 2024-03-19T11:30:53.884 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.367e-06 2024-03-19T11:30:53.884 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.884 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.884 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:53.884 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.171560+0000", 2024-03-19T11:30:53.884 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027039389999999998 2024-03-19T11:30:53.884 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.884 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.884 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:53.884 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.171590+0000", 2024-03-19T11:30:53.884 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9981e-05 2024-03-19T11:30:53.884 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.884 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.884 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:53.884 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.171610+0000", 2024-03-19T11:30:53.885 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9891999999999999e-05 2024-03-19T11:30:53.885 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.885 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.885 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:53.885 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.179690+0000", 2024-03-19T11:30:53.885 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0080796010000000005 2024-03-19T11:30:53.885 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.885 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.885 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:53.885 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.179984+0000", 2024-03-19T11:30:53.885 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029420899999999997 2024-03-19T11:30:53.885 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.885 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.885 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:53.885 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.180404+0000", 2024-03-19T11:30:53.886 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00042013100000000002 2024-03-19T11:30:53.886 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.886 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.886 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:53.886 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.186130+0000", 2024-03-19T11:30:53.886 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.005726358 2024-03-19T11:30:53.886 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.886 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.886 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:53.886 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.189066+0000", 2024-03-19T11:30:53.886 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029353349999999999 2024-03-19T11:30:53.886 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.886 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.886 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:53.886 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.189190+0000", 2024-03-19T11:30:53.887 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012390599999999999 2024-03-19T11:30:53.887 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.887 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.887 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:53.887 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.189316+0000", 2024-03-19T11:30:53.887 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000126876 2024-03-19T11:30:53.887 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:53.887 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:53.887 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:53.887 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.887 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.887 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2209 4.28 4:15708482:::smithi03829856-23:head [setxattr _header (57) in=7b,truncate 2941225] snapc 13=[13,12,11,d,b,6] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-03-19T11:30:53.887 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:51.171657+0000", 2024-03-19T11:30:53.887 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 2.6301380669999999, 2024-03-19T11:30:53.887 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017949024000000001, 2024-03-19T11:30:53.888 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:53.888 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:53.888 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:53.888 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:53.888 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:53.888 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2209 2024-03-19T11:30:53.888 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.888 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:53.888 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.888 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:53.888 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.171657+0000", 2024-03-19T11:30:53.888 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:53.888 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.888 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.888 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:53.889 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.171657+0000", 2024-03-19T11:30:53.889 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:53.889 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.889 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.889 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:53.889 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.171659+0000", 2024-03-19T11:30:53.889 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9130000000000001e-06 2024-03-19T11:30:53.889 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.889 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.889 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:53.889 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.171667+0000", 2024-03-19T11:30:53.889 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.089e-06 2024-03-19T11:30:53.889 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.889 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.889 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:53.890 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.171704+0000", 2024-03-19T11:30:53.890 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6721000000000003e-05 2024-03-19T11:30:53.890 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.890 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.890 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:53.890 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.171723+0000", 2024-03-19T11:30:53.890 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9196000000000001e-05 2024-03-19T11:30:53.890 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.890 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.890 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:53.890 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.184321+0000", 2024-03-19T11:30:53.890 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012597994 2024-03-19T11:30:53.890 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.890 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.890 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:53.891 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.184622+0000", 2024-03-19T11:30:53.891 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00030102900000000002 2024-03-19T11:30:53.891 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.891 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.891 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:30:53.891 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.185158+0000", 2024-03-19T11:30:53.891 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000535854 2024-03-19T11:30:53.891 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.891 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.891 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:53.891 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.186151+0000", 2024-03-19T11:30:53.891 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00099323100000000002 2024-03-19T11:30:53.891 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.891 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.891 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:53.892 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.189409+0000", 2024-03-19T11:30:53.892 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032579620000000001 2024-03-19T11:30:53.892 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.892 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.892 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:53.892 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.189511+0000", 2024-03-19T11:30:53.892 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000101851 2024-03-19T11:30:53.892 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.892 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.892 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:53.892 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.189606+0000", 2024-03-19T11:30:53.892 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5184000000000002e-05 2024-03-19T11:30:53.892 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:53.892 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:53.892 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:53.892 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.893 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2210 4.28 4:15708482:::smithi03829856-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e71)", 2024-03-19T11:30:53.893 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:51.171743+0000", 2024-03-19T11:30:53.893 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 2.6300522219999998, 2024-03-19T11:30:53.893 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018593261999999999, 2024-03-19T11:30:53.893 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:53.893 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:53.893 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:53.893 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:53.893 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:53.893 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2210 2024-03-19T11:30:53.893 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.893 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:53.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.893 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:53.894 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.171743+0000", 2024-03-19T11:30:53.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:53.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:53.894 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.171743+0000", 2024-03-19T11:30:53.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:53.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:53.894 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.171745+0000", 2024-03-19T11:30:53.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3960000000000001e-06 2024-03-19T11:30:53.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:53.895 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.171751+0000", 2024-03-19T11:30:53.895 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3890000000000002e-06 2024-03-19T11:30:53.895 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.895 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.895 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:53.895 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.171773+0000", 2024-03-19T11:30:53.895 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1719e-05 2024-03-19T11:30:53.895 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.895 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.895 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:53.895 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.171787+0000", 2024-03-19T11:30:53.895 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4559000000000001e-05 2024-03-19T11:30:53.895 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.895 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.895 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:53.896 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.186248+0000", 2024-03-19T11:30:53.896 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014460847000000001 2024-03-19T11:30:53.896 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.896 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.896 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-03-19T11:30:53.896 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.186543+0000", 2024-03-19T11:30:53.896 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029462899999999997 2024-03-19T11:30:53.896 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.896 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.896 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:53.896 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.189741+0000", 2024-03-19T11:30:53.896 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0031981119999999999 2024-03-19T11:30:53.896 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.896 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.896 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:53.897 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.190005+0000", 2024-03-19T11:30:53.897 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026447800000000001 2024-03-19T11:30:53.897 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:53.897 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:53.897 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:53.897 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.190337+0000", 2024-03-19T11:30:53.897 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000331133 2024-03-19T11:30:53.897 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:53.897 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:53.897 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:53.897 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:53.897 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:53.897 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:53.899 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-19T11:30:53.899 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:53.906+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:54.000 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:54.010+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:54.025 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:54.025 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:30:54.025 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.025 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2284 4.15 4.104f95f5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-03-19T11:30:54.025 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:53.596775+0000", 2024-03-19T11:30:54.025 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.44061003199999998, 2024-03-19T11:30:54.026 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.44065250299999997, 2024-03-19T11:30:54.026 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:54.026 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "waiting for peered", 2024-03-19T11:30:54.026 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:54.026 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:54.026 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:54.026 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2284 2024-03-19T11:30:54.026 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.026 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:54.026 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.026 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:54.026 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596775+0000", 2024-03-19T11:30:54.026 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.026 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.026 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.027 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:54.027 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596775+0000", 2024-03-19T11:30:54.027 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.027 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.027 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.027 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:54.027 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596778+0000", 2024-03-19T11:30:54.027 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.174e-06 2024-03-19T11:30:54.027 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.027 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.027 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:54.027 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596794+0000", 2024-03-19T11:30:54.027 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5372000000000001e-05 2024-03-19T11:30:54.027 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.027 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.028 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:54.028 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596828+0000", 2024-03-19T11:30:54.028 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4107999999999999e-05 2024-03-19T11:30:54.028 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.028 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:54.028 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596844+0000", 2024-03-19T11:30:54.028 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6296000000000001e-05 2024-03-19T11:30:54.028 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.028 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:54.028 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596968+0000", 2024-03-19T11:30:54.028 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012383699999999999 2024-03-19T11:30:54.028 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.029 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:30:54.029 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596988+0000", 2024-03-19T11:30:54.029 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0645e-05 2024-03-19T11:30:54.029 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.029 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:54.029 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.029 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.029 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2024-03-19T11:30:54.029 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 1 2024-03-19T11:30:54.029 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:54.039 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-03-19T11:30:54.044 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-19T11:30:54.071 INFO:tasks.rados.rados.0.smithi038.stdout:1136: set_chunk oid 35 offset: 1314816 length: 702976 target oid d249c5a10a455b4319923be58337c00aa401bd2355208e7f4a1eff309bd9edd8 offset: 0 snap -1 2024-03-19T11:30:54.071 INFO:tasks.rados.rados.0.smithi038.stdout:1137: snap_create 2024-03-19T11:30:54.071 INFO:tasks.rados.rados.0.smithi038.stdout:1131: finishing write tid 5 to smithi03829856-13 2024-03-19T11:30:54.071 INFO:tasks.rados.rados.0.smithi038.stdout:1131: oid 13 updating version 0 to 88 2024-03-19T11:30:54.071 INFO:tasks.rados.rados.0.smithi038.stdout:1131: oid 13 version 88 is already newer than 87 2024-03-19T11:30:54.071 INFO:tasks.rados.rados.0.smithi038.stdout:1131: oid 13 version 88 is already newer than 86 2024-03-19T11:30:54.072 INFO:tasks.rados.rados.0.smithi038.stdout:1131: oid 13 version 88 is already newer than 85 2024-03-19T11:30:54.072 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 13 v 88 (ObjNum 180 snap 19 seq_num 180) dirty exists 2024-03-19T11:30:54.072 INFO:tasks.rados.rados.0.smithi038.stdout:1131: left oid 13 (ObjNum 180 snap 19 seq_num 180) 2024-03-19T11:30:54.072 INFO:tasks.rados.rados.0.smithi038.stdout:1133: got (16) Device or resource busy 2024-03-19T11:30:54.072 INFO:tasks.rados.rados.0.smithi038.stdout:check_oldest_snap_flushed oid 25 v 0 (ObjNum 49 snap 0 seq_num 1953066355) unflushed 11 2024-03-19T11:30:54.072 INFO:tasks.rados.rados.0.smithi038.stdout:1134: expect (ObjNum 152 snap 17 seq_num 152) 2024-03-19T11:30:54.101 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:54.107+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:54.190 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:54.190 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:54.190 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:30:54.190 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:30:54.190 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:54.202 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:54.210+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:54.204 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-19T11:30:54.241 INFO:tasks.rados.rados.0.smithi038.stdout:1132: finishing write tid 1 to smithi03829856-17 2024-03-19T11:30:54.241 INFO:tasks.rados.rados.0.smithi038.stdout:1132: finishing write tid 2 to smithi03829856-17 2024-03-19T11:30:54.241 INFO:tasks.rados.rados.0.smithi038.stdout:1132: finishing write tid 3 to smithi03829856-17 2024-03-19T11:30:54.241 INFO:tasks.rados.rados.0.smithi038.stdout:1132: finishing write tid 4 to smithi03829856-17 2024-03-19T11:30:54.241 INFO:tasks.rados.rados.0.smithi038.stdout:1132: finishing write tid 5 to smithi03829856-17 2024-03-19T11:30:54.242 INFO:tasks.rados.rados.0.smithi038.stdout:1132: oid 17 updating version 0 to 129 2024-03-19T11:30:54.242 INFO:tasks.rados.rados.0.smithi038.stdout:1132: oid 17 version 129 is already newer than 127 2024-03-19T11:30:54.242 INFO:tasks.rados.rados.0.smithi038.stdout:1132: oid 17 updating version 129 to 130 2024-03-19T11:30:54.242 INFO:tasks.rados.rados.0.smithi038.stdout:1132: oid 17 version 130 is already newer than 128 2024-03-19T11:30:54.242 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 17 v 130 (ObjNum 181 snap 19 seq_num 181) dirty exists 2024-03-19T11:30:54.242 INFO:tasks.rados.rados.0.smithi038.stdout:1132: left oid 17 (ObjNum 181 snap 19 seq_num 181) 2024-03-19T11:30:54.242 INFO:tasks.rados.rados.0.smithi038.stdout:1124: expect (ObjNum 143 snap 15 seq_num 143) 2024-03-19T11:30:54.302 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:54.310+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:54.332 INFO:tasks.rados.rados.0.smithi038.stdout:1136: finishing set_chunk to oid 35 2024-03-19T11:30:54.332 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 35 v 108 (ObjNum 158 snap 17 seq_num 158) dirty exists 2024-03-19T11:30:54.332 INFO:tasks.rados.rados.0.smithi038.stdout:1135: got (0) Success 2024-03-19T11:30:54.332 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 18 v 75 (ObjNum 106 snap 11 seq_num 106) dirty exists 2024-03-19T11:30:54.378 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:54.378 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:30:54.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:30:54.378 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:30:54.379 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.379 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:30 4.5 4:ac9cd3a3:::smithi03829856-8:head [write 1776668~340981] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:30:54.379 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:04.777922+0000", 2024-03-19T11:30:54.379 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 49.608417764000002, 2024-03-19T11:30:54.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015469894, 2024-03-19T11:30:54.379 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:54.379 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:54.379 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:54.379 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:54.379 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:54.379 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 30 2024-03-19T11:30:54.379 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.379 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:54.379 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.380 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:54.380 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.777922+0000", 2024-03-19T11:30:54.380 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.380 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.380 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.380 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:54.380 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.777922+0000", 2024-03-19T11:30:54.380 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.380 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.380 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.380 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:54.380 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.777924+0000", 2024-03-19T11:30:54.380 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3999999999999999e-06 2024-03-19T11:30:54.380 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.380 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.381 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:54.381 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778461+0000", 2024-03-19T11:30:54.381 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00053698299999999999 2024-03-19T11:30:54.381 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.381 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.381 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:54.381 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778487+0000", 2024-03-19T11:30:54.381 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5947000000000001e-05 2024-03-19T11:30:54.381 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.381 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.381 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:54.381 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778502+0000", 2024-03-19T11:30:54.381 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4256e-05 2024-03-19T11:30:54.381 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.381 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.382 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:54.382 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.781978+0000", 2024-03-19T11:30:54.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00347693 2024-03-19T11:30:54.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.382 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:54.382 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.782196+0000", 2024-03-19T11:30:54.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021726499999999999 2024-03-19T11:30:54.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.382 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:30:54.382 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.782572+0000", 2024-03-19T11:30:54.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00037635000000000001 2024-03-19T11:30:54.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:54.383 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.786960+0000", 2024-03-19T11:30:54.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043882130000000002 2024-03-19T11:30:54.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:54.383 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793113+0000", 2024-03-19T11:30:54.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061530760000000004 2024-03-19T11:30:54.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:54.383 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793276+0000", 2024-03-19T11:30:54.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000162652 2024-03-19T11:30:54.384 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.384 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.384 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:54.384 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793392+0000", 2024-03-19T11:30:54.384 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115822 2024-03-19T11:30:54.384 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.384 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:54.384 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.384 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.384 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.384 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:31 4.5 4:ac9cd3a3:::smithi03829856-8:head [setxattr _header (55) in=7b,truncate 2117649] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:30:54.384 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:04.778533+0000", 2024-03-19T11:30:54.384 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 49.607806629000002, 2024-03-19T11:30:54.384 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015182533, 2024-03-19T11:30:54.384 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:54.385 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:54.385 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:54.385 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:54.385 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:54.385 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 31 2024-03-19T11:30:54.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.385 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:54.385 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.385 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:54.385 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778533+0000", 2024-03-19T11:30:54.385 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.385 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.385 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:54.385 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778533+0000", 2024-03-19T11:30:54.386 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.386 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:54.386 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778534+0000", 2024-03-19T11:30:54.386 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1069999999999999e-06 2024-03-19T11:30:54.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.386 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:54.386 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778541+0000", 2024-03-19T11:30:54.386 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2529999999999998e-06 2024-03-19T11:30:54.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.386 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:54.386 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778558+0000", 2024-03-19T11:30:54.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6290000000000002e-05 2024-03-19T11:30:54.387 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.387 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:54.387 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778568+0000", 2024-03-19T11:30:54.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0360000000000001e-05 2024-03-19T11:30:54.387 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.387 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:54.387 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.785511+0000", 2024-03-19T11:30:54.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00694306 2024-03-19T11:30:54.387 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.387 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:54.388 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.785710+0000", 2024-03-19T11:30:54.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019837199999999999 2024-03-19T11:30:54.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:30:54.388 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.785929+0000", 2024-03-19T11:30:54.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021960799999999999 2024-03-19T11:30:54.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:54.388 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.787281+0000", 2024-03-19T11:30:54.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001352313 2024-03-19T11:30:54.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:54.389 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793492+0000", 2024-03-19T11:30:54.389 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062107760000000003 2024-03-19T11:30:54.389 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.389 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.389 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:54.389 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793578+0000", 2024-03-19T11:30:54.389 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5711999999999995e-05 2024-03-19T11:30:54.389 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.389 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.389 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:54.389 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793716+0000", 2024-03-19T11:30:54.389 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000137682 2024-03-19T11:30:54.389 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.389 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:54.389 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.390 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.390 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:32 4.5 4:ac9cd3a3:::smithi03829856-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:30:54.390 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:04.778582+0000", 2024-03-19T11:30:54.390 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 49.607758115000003, 2024-03-19T11:30:54.390 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015761827999999999, 2024-03-19T11:30:54.390 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:54.390 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:54.390 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:54.390 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:54.390 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:54.390 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 32 2024-03-19T11:30:54.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.390 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:54.390 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.391 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:54.391 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778582+0000", 2024-03-19T11:30:54.391 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.391 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.391 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.391 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:54.391 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778582+0000", 2024-03-19T11:30:54.391 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.391 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.391 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.391 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:54.391 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778583+0000", 2024-03-19T11:30:54.391 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.026e-06 2024-03-19T11:30:54.391 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.392 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:54.392 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778587+0000", 2024-03-19T11:30:54.392 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7870000000000001e-06 2024-03-19T11:30:54.392 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.392 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:54.392 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778599+0000", 2024-03-19T11:30:54.392 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1623e-05 2024-03-19T11:30:54.392 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.392 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:54.392 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778608+0000", 2024-03-19T11:30:54.392 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7399999999999993e-06 2024-03-19T11:30:54.392 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.393 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:54.393 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.787009+0000", 2024-03-19T11:30:54.393 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0084010490000000007 2024-03-19T11:30:54.393 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.393 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-03-19T11:30:54.393 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.787202+0000", 2024-03-19T11:30:54.393 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019309500000000001 2024-03-19T11:30:54.393 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.393 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:54.393 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793832+0000", 2024-03-19T11:30:54.394 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.006630052 2024-03-19T11:30:54.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.394 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.394 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:54.394 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.794048+0000", 2024-03-19T11:30:54.394 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000215505 2024-03-19T11:30:54.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.394 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.394 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:54.394 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.794343+0000", 2024-03-19T11:30:54.394 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029595100000000003 2024-03-19T11:30:54.394 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.394 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:54.394 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.395 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.395 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.395 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:856 4.5 4:ac9cd3a3:::smithi03829856-8:head [write 608420~529411] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:54.395 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.781425+0000", 2024-03-19T11:30:54.395 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 45.604914929000003, 2024-03-19T11:30:54.395 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014634677, 2024-03-19T11:30:54.395 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:54.395 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:54.395 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:54.395 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:54.395 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:54.395 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 856 2024-03-19T11:30:54.396 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.396 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:54.396 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.396 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:54.396 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.781425+0000", 2024-03-19T11:30:54.396 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.396 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.396 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.396 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:54.396 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.781425+0000", 2024-03-19T11:30:54.396 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.396 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.396 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.396 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:54.397 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.781428+0000", 2024-03-19T11:30:54.397 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.534e-06 2024-03-19T11:30:54.397 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.397 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.397 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:54.397 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.783096+0000", 2024-03-19T11:30:54.397 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001667897 2024-03-19T11:30:54.397 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.397 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.397 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:54.397 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.783138+0000", 2024-03-19T11:30:54.397 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1946000000000001e-05 2024-03-19T11:30:54.397 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.397 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.397 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:54.398 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.783163+0000", 2024-03-19T11:30:54.398 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4443e-05 2024-03-19T11:30:54.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.398 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.398 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:54.398 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.783333+0000", 2024-03-19T11:30:54.398 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017027899999999999 2024-03-19T11:30:54.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.398 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.398 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:54.398 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.783503+0000", 2024-03-19T11:30:54.398 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000170242 2024-03-19T11:30:54.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.398 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.399 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:30:54.399 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.783917+0000", 2024-03-19T11:30:54.399 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00041371000000000002 2024-03-19T11:30:54.399 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.399 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.399 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:54.399 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.795709+0000", 2024-03-19T11:30:54.399 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011791693000000001 2024-03-19T11:30:54.399 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.399 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.399 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:54.399 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.795850+0000", 2024-03-19T11:30:54.399 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000141695 2024-03-19T11:30:54.399 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.399 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.400 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:54.400 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.795940+0000", 2024-03-19T11:30:54.400 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9705999999999998e-05 2024-03-19T11:30:54.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.400 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.400 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:54.400 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.796059+0000", 2024-03-19T11:30:54.400 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011953199999999999 2024-03-19T11:30:54.400 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.400 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:54.400 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.400 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.400 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:891 4.5 4:af9d0d56:::smithi03829856-20:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:54.400 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:09.286410+0000", 2024-03-19T11:30:54.401 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 45.099929783, 2024-03-19T11:30:54.401 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035191405000000002, 2024-03-19T11:30:54.401 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:54.401 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:54.401 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:54.401 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:54.401 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:54.401 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 891 2024-03-19T11:30:54.401 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.401 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:54.401 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.401 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:54.401 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286410+0000", 2024-03-19T11:30:54.401 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.401 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.402 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.402 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:54.402 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286410+0000", 2024-03-19T11:30:54.402 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.402 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.402 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.402 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:54.402 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286413+0000", 2024-03-19T11:30:54.402 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2660000000000002e-06 2024-03-19T11:30:54.402 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.402 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.402 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:54.402 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286426+0000", 2024-03-19T11:30:54.402 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.258e-05 2024-03-19T11:30:54.402 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.403 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.403 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:54.403 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286460+0000", 2024-03-19T11:30:54.403 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3943999999999999e-05 2024-03-19T11:30:54.403 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.403 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.403 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:54.403 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286477+0000", 2024-03-19T11:30:54.403 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6799999999999998e-05 2024-03-19T11:30:54.403 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.403 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.403 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:54.403 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286605+0000", 2024-03-19T11:30:54.403 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000128892 2024-03-19T11:30:54.403 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.404 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.404 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:54.404 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286769+0000", 2024-03-19T11:30:54.404 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016359799999999999 2024-03-19T11:30:54.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.404 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.404 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:54.404 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.321601+0000", 2024-03-19T11:30:54.404 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.034832324999999997 2024-03-19T11:30:54.404 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.404 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:54.404 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.404 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.405 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1015 4.5 4:af9d0d56:::smithi03829856-20:head [tier-promote] snapc 6=[6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:54.405 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:12.761855+0000", 2024-03-19T11:30:54.405 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 41.624484256999999, 2024-03-19T11:30:54.405 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026760430000000002, 2024-03-19T11:30:54.405 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:54.405 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:54.405 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:54.405 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:54.405 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:54.405 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1015 2024-03-19T11:30:54.405 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.405 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:54.405 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.405 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:54.406 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761855+0000", 2024-03-19T11:30:54.406 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.406 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:54.406 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761855+0000", 2024-03-19T11:30:54.406 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.406 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:54.406 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761858+0000", 2024-03-19T11:30:54.406 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8140000000000001e-06 2024-03-19T11:30:54.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.407 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:54.407 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761872+0000", 2024-03-19T11:30:54.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3813e-05 2024-03-19T11:30:54.407 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.407 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.407 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:54.407 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761905+0000", 2024-03-19T11:30:54.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2642999999999999e-05 2024-03-19T11:30:54.407 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.407 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.407 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:54.407 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761925+0000", 2024-03-19T11:30:54.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0489e-05 2024-03-19T11:30:54.407 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.408 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:54.408 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.762070+0000", 2024-03-19T11:30:54.408 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000145153 2024-03-19T11:30:54.408 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.408 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:54.408 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.762365+0000", 2024-03-19T11:30:54.408 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029482100000000002 2024-03-19T11:30:54.408 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.408 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:54.408 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.788616+0000", 2024-03-19T11:30:54.408 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026250697 2024-03-19T11:30:54.409 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.409 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:54.409 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.409 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.409 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1121 4.5 4:aa3303a5:::smithi03829856-17:head [tier-promote] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:54.409 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:14.164281+0000", 2024-03-19T11:30:54.409 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 40.222059012000003, 2024-03-19T11:30:54.409 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023896435000000001, 2024-03-19T11:30:54.409 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:54.409 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:54.409 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:54.409 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:54.409 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:54.409 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1121 2024-03-19T11:30:54.410 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.410 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:54.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.410 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:54.410 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164281+0000", 2024-03-19T11:30:54.410 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.410 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.410 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:54.410 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164281+0000", 2024-03-19T11:30:54.410 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.410 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.410 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:54.411 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164283+0000", 2024-03-19T11:30:54.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8500000000000001e-06 2024-03-19T11:30:54.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.411 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:54.411 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164291+0000", 2024-03-19T11:30:54.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.14e-06 2024-03-19T11:30:54.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.411 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:54.411 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164313+0000", 2024-03-19T11:30:54.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.207e-05 2024-03-19T11:30:54.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:54.412 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164328+0000", 2024-03-19T11:30:54.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5403000000000001e-05 2024-03-19T11:30:54.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:54.412 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.167981+0000", 2024-03-19T11:30:54.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036529879999999998 2024-03-19T11:30:54.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:54.412 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.168274+0000", 2024-03-19T11:30:54.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029313100000000002 2024-03-19T11:30:54.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.413 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:54.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.188177+0000", 2024-03-19T11:30:54.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019902853000000002 2024-03-19T11:30:54.413 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.413 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:54.413 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.413 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1302 4.15 4:aa3303a5:::smithi03829856-17:head [write 413472~620140,stat] snapc b=[b,a,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-03-19T11:30:54.413 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:22.966426+0000", 2024-03-19T11:30:54.413 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 31.419913278999999, 2024-03-19T11:30:54.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018043513000000001, 2024-03-19T11:30:54.414 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:54.414 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:54.414 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:54.414 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:54.414 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:54.414 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1302 2024-03-19T11:30:54.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.414 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:54.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.414 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:54.414 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.966426+0000", 2024-03-19T11:30:54.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:54.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.966426+0000", 2024-03-19T11:30:54.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:54.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.966429+0000", 2024-03-19T11:30:54.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3829999999999999e-06 2024-03-19T11:30:54.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.416 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:54.416 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972117+0000", 2024-03-19T11:30:54.416 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0056885750000000004 2024-03-19T11:30:54.416 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.416 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:54.416 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972156+0000", 2024-03-19T11:30:54.416 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8878000000000003e-05 2024-03-19T11:30:54.416 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.417 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:54.417 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972184+0000", 2024-03-19T11:30:54.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7509999999999999e-05 2024-03-19T11:30:54.417 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.417 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:54.417 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972309+0000", 2024-03-19T11:30:54.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012525899999999999 2024-03-19T11:30:54.417 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.417 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:54.417 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972566+0000", 2024-03-19T11:30:54.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025740799999999999 2024-03-19T11:30:54.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:30:54.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972899+0000", 2024-03-19T11:30:54.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033238799999999998 2024-03-19T11:30:54.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:54.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984088+0000", 2024-03-19T11:30:54.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011189466 2024-03-19T11:30:54.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:54.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984138+0000", 2024-03-19T11:30:54.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9666999999999997e-05 2024-03-19T11:30:54.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:54.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984329+0000", 2024-03-19T11:30:54.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019086799999999999 2024-03-19T11:30:54.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:54.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984470+0000", 2024-03-19T11:30:54.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000141111 2024-03-19T11:30:54.420 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.420 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:54.420 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.420 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1303 4.15 4:aa3303a5:::smithi03829856-17:head [write 1818036~510700] snapc b=[b,a,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-03-19T11:30:54.420 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:22.976856+0000", 2024-03-19T11:30:54.420 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 31.409484168999999, 2024-03-19T11:30:54.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015045703000000001, 2024-03-19T11:30:54.420 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:54.420 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:54.420 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:54.420 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:54.420 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:54.421 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1303 2024-03-19T11:30:54.421 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.421 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:54.421 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.421 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:54.421 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.976856+0000", 2024-03-19T11:30:54.421 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.421 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.421 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.421 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:54.421 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.976856+0000", 2024-03-19T11:30:54.421 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.421 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.422 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:54.422 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.976859+0000", 2024-03-19T11:30:54.422 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3629999999999998e-06 2024-03-19T11:30:54.422 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.422 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:54.422 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.977533+0000", 2024-03-19T11:30:54.422 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00067380800000000002 2024-03-19T11:30:54.422 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.422 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:54.422 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.977573+0000", 2024-03-19T11:30:54.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9758000000000001e-05 2024-03-19T11:30:54.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:54.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.977602+0000", 2024-03-19T11:30:54.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9856e-05 2024-03-19T11:30:54.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:54.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.979943+0000", 2024-03-19T11:30:54.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0023406400000000002 2024-03-19T11:30:54.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:54.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.980238+0000", 2024-03-19T11:30:54.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029488000000000002 2024-03-19T11:30:54.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:30:54.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.980734+0000", 2024-03-19T11:30:54.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00049647500000000002 2024-03-19T11:30:54.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:54.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.991761+0000", 2024-03-19T11:30:54.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011026515000000001 2024-03-19T11:30:54.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:54.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.991777+0000", 2024-03-19T11:30:54.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5727999999999999e-05 2024-03-19T11:30:54.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:54.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.991840+0000", 2024-03-19T11:30:54.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3091999999999998e-05 2024-03-19T11:30:54.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:54.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.991901+0000", 2024-03-19T11:30:54.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1587999999999994e-05 2024-03-19T11:30:54.426 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.426 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:54.426 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.426 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.426 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1468 4.a 4:567251f3:::smithi03829856-44:head [write 1603467~416361] snapc c=[c,b,a,6,5] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-03-19T11:30:54.426 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.083922+0000", 2024-03-19T11:30:54.426 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.302418146000001, 2024-03-19T11:30:54.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014508549000000001, 2024-03-19T11:30:54.426 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:54.426 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:54.426 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:54.426 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:54.427 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:54.427 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1468 2024-03-19T11:30:54.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.427 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:54.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.427 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:54.427 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.083922+0000", 2024-03-19T11:30:54.427 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.427 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:54.427 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.083922+0000", 2024-03-19T11:30:54.427 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.428 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:54.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.083924+0000", 2024-03-19T11:30:54.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2800000000000002e-06 2024-03-19T11:30:54.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.428 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:54.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.084498+0000", 2024-03-19T11:30:54.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057414900000000004 2024-03-19T11:30:54.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.428 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:54.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.084529+0000", 2024-03-19T11:30:54.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1080000000000001e-05 2024-03-19T11:30:54.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:54.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.084544+0000", 2024-03-19T11:30:54.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4498000000000001e-05 2024-03-19T11:30:54.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:54.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.088993+0000", 2024-03-19T11:30:54.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044497260000000002 2024-03-19T11:30:54.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.430 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.430 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:54.430 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.089152+0000", 2024-03-19T11:30:54.430 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015833199999999999 2024-03-19T11:30:54.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.430 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.430 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:54.430 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.089419+0000", 2024-03-19T11:30:54.430 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026691799999999999 2024-03-19T11:30:54.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.430 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.430 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:54.430 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.092883+0000", 2024-03-19T11:30:54.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034646590000000001 2024-03-19T11:30:54.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:54.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.098276+0000", 2024-03-19T11:30:54.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053924680000000001 2024-03-19T11:30:54.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:54.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.098354+0000", 2024-03-19T11:30:54.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8763999999999993e-05 2024-03-19T11:30:54.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.432 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:54.432 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.098430+0000", 2024-03-19T11:30:54.432 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5674999999999996e-05 2024-03-19T11:30:54.432 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.432 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:54.432 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.432 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.1 50/49 [PushOp(4:866f3dcc:::smithi03829856-31:7, version: 38'44, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@38'44, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 45'62, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@45'62, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 46'68, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@46'68, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:7, version: 47'71, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@47'71, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 47'72, data_included: [440350~816629,1309629~231947], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@47'72, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:30:54.432 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.959457+0000", 2024-03-19T11:30:54.432 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 26.426882861999999, 2024-03-19T11:30:54.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018416666000000002, 2024-03-19T11:30:54.433 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:54.433 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:54.433 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:54.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.433 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:54.433 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.959457+0000", 2024-03-19T11:30:54.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.433 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.433 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:54.433 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.959457+0000", 2024-03-19T11:30:54.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.434 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.434 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.434 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:54.434 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.959458+0000", 2024-03-19T11:30:54.434 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4899999999999999e-06 2024-03-19T11:30:54.434 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.434 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.434 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:54.434 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.962822+0000", 2024-03-19T11:30:54.434 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0033640620000000001 2024-03-19T11:30:54.434 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.435 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:54.435 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.962893+0000", 2024-03-19T11:30:54.435 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1060000000000001e-05 2024-03-19T11:30:54.435 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.435 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:54.435 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.962977+0000", 2024-03-19T11:30:54.435 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3311000000000002e-05 2024-03-19T11:30:54.435 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.435 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:54.436 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963208+0000", 2024-03-19T11:30:54.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023121200000000001 2024-03-19T11:30:54.436 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.436 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.436 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:54.436 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963245+0000", 2024-03-19T11:30:54.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6844999999999997e-05 2024-03-19T11:30:54.436 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.436 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.436 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:54.436 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.977874+0000", 2024-03-19T11:30:54.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014628686 2024-03-19T11:30:54.436 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.437 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:54.437 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.437 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.437 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.1 e50,49 4:8401c1f0:::smithi03829856-43:8,46'66,4:866f3dcc:::smithi03829856-31:2,47'73)", 2024-03-19T11:30:54.437 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.963027+0000", 2024-03-19T11:30:54.437 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 26.423313244999999, 2024-03-19T11:30:54.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015116788000000001, 2024-03-19T11:30:54.437 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:54.437 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:54.437 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:54.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.438 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:54.438 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963027+0000", 2024-03-19T11:30:54.438 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.438 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.438 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:54.438 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963027+0000", 2024-03-19T11:30:54.438 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.438 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.439 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:54.439 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963028+0000", 2024-03-19T11:30:54.439 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.268e-06 2024-03-19T11:30:54.439 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.439 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.439 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:54.439 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963036+0000", 2024-03-19T11:30:54.439 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.926e-06 2024-03-19T11:30:54.439 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.439 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.439 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:54.440 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963050+0000", 2024-03-19T11:30:54.440 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4082e-05 2024-03-19T11:30:54.440 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.440 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.440 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:54.440 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963062+0000", 2024-03-19T11:30:54.440 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2415000000000001e-05 2024-03-19T11:30:54.440 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.440 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.440 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:54.441 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.977954+0000", 2024-03-19T11:30:54.441 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014891572 2024-03-19T11:30:54.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.441 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.441 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:54.441 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.977965+0000", 2024-03-19T11:30:54.441 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1654000000000001e-05 2024-03-19T11:30:54.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.441 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.441 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:54.441 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.978143+0000", 2024-03-19T11:30:54.442 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017787099999999999 2024-03-19T11:30:54.442 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.442 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:54.442 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.442 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.442 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.442 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.11 50/49 [PushOp(4:8d3763f9:::smithi03829856-29:head, version: 47'60, data_included: [3364276~232389], data_size: 232389, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8d3763f9:::smithi03829856-29:head@47'60, size: 3596665, copy_subset: [(0, 3596665)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3596665, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3364276, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:30:54.442 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.964363+0000", 2024-03-19T11:30:54.442 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 26.421976577999999, 2024-03-19T11:30:54.442 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014531759999999999, 2024-03-19T11:30:54.442 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:54.442 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:54.442 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:54.442 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.443 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:54.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.964363+0000", 2024-03-19T11:30:54.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.443 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.443 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.443 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:54.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.964363+0000", 2024-03-19T11:30:54.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.443 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.443 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.443 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:54.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.964364+0000", 2024-03-19T11:30:54.444 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4200000000000004e-07 2024-03-19T11:30:54.444 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.444 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.444 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:54.444 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.964447+0000", 2024-03-19T11:30:54.444 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2430999999999997e-05 2024-03-19T11:30:54.444 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.444 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.444 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:54.444 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.964468+0000", 2024-03-19T11:30:54.444 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1685999999999999e-05 2024-03-19T11:30:54.444 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.444 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.444 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:54.444 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.964483+0000", 2024-03-19T11:30:54.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4969e-05 2024-03-19T11:30:54.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:54.445 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.977934+0000", 2024-03-19T11:30:54.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013450963999999999 2024-03-19T11:30:54.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:54.445 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.977975+0000", 2024-03-19T11:30:54.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1125999999999999e-05 2024-03-19T11:30:54.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:54.445 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.978895+0000", 2024-03-19T11:30:54.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00091964200000000003 2024-03-19T11:30:54.446 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.446 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:54.446 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.446 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.446 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.1 50/49 [PushOp(4:866f3dcc:::smithi03829856-31:7, version: 38'44, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@38'44, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 45'62, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@45'62, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 46'68, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@46'68, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:7, version: 47'71, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@47'71, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 47'72, data_included: [1541576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@47'72, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:30:54.446 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.991858+0000", 2024-03-19T11:30:54.446 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 26.394481655, 2024-03-19T11:30:54.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014228774, 2024-03-19T11:30:54.446 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:54.446 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:54.446 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:54.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.447 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:54.447 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.991858+0000", 2024-03-19T11:30:54.447 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.447 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.447 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.447 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:54.447 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.991858+0000", 2024-03-19T11:30:54.447 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.447 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.447 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.447 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:54.447 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.991859+0000", 2024-03-19T11:30:54.447 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.342e-06 2024-03-19T11:30:54.447 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.447 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.448 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:54.448 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.994675+0000", 2024-03-19T11:30:54.448 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028153150000000001 2024-03-19T11:30:54.448 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.448 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.448 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:54.448 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.994733+0000", 2024-03-19T11:30:54.448 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8514999999999998e-05 2024-03-19T11:30:54.448 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.448 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.448 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:54.448 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.994794+0000", 2024-03-19T11:30:54.448 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0588000000000003e-05 2024-03-19T11:30:54.448 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.448 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.449 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:54.449 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.995070+0000", 2024-03-19T11:30:54.449 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027573700000000002 2024-03-19T11:30:54.449 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.449 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.449 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:54.449 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.995100+0000", 2024-03-19T11:30:54.449 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0679000000000002e-05 2024-03-19T11:30:54.449 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.449 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.449 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:54.449 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.006087+0000", 2024-03-19T11:30:54.449 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010986598 2024-03-19T11:30:54.449 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.449 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:54.450 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.450 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.450 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.450 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2284 4.15 4:afa9f208:::smithi03829856-35:head [read 1314816~702976] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-03-19T11:30:54.450 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:53.596775+0000", 2024-03-19T11:30:54.450 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.78956477599999997, 2024-03-19T11:30:54.450 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.47260528299999999, 2024-03-19T11:30:54.450 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:54.450 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:54.450 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:54.450 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:54.450 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:54.450 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2284 2024-03-19T11:30:54.450 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.450 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:54.451 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.451 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:54.451 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596775+0000", 2024-03-19T11:30:54.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.451 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.451 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.451 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:54.451 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596775+0000", 2024-03-19T11:30:54.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.451 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.451 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.451 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:54.451 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596778+0000", 2024-03-19T11:30:54.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.174e-06 2024-03-19T11:30:54.452 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.452 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:54.452 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596794+0000", 2024-03-19T11:30:54.452 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5372000000000001e-05 2024-03-19T11:30:54.452 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.453 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:54.453 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596828+0000", 2024-03-19T11:30:54.453 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4107999999999999e-05 2024-03-19T11:30:54.453 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.453 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:54.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596844+0000", 2024-03-19T11:30:54.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6296000000000001e-05 2024-03-19T11:30:54.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:54.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596968+0000", 2024-03-19T11:30:54.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012383699999999999 2024-03-19T11:30:54.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:30:54.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596988+0000", 2024-03-19T11:30:54.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0645e-05 2024-03-19T11:30:54.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:54.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.068916+0000", 2024-03-19T11:30:54.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.47192744800000003 2024-03-19T11:30:54.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:54.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.069015+0000", 2024-03-19T11:30:54.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9500999999999995e-05 2024-03-19T11:30:54.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:54.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.069380+0000", 2024-03-19T11:30:54.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00036490200000000001 2024-03-19T11:30:54.456 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.456 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:54.456 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.456 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2286 4.15 4:afa9f208:::smithi03829856-35:head [assert-version v108,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e75)", 2024-03-19T11:30:54.456 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.084826+0000", 2024-03-19T11:30:54.456 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.30151340900000001, 2024-03-19T11:30:54.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.055799556, 2024-03-19T11:30:54.456 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:54.456 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:54.456 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:54.456 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:54.456 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:54.456 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2286 2024-03-19T11:30:54.457 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.457 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:54.457 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.457 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:54.457 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084826+0000", 2024-03-19T11:30:54.457 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.457 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.457 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.457 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:54.457 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084826+0000", 2024-03-19T11:30:54.457 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.457 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.457 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.457 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:54.458 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084828+0000", 2024-03-19T11:30:54.458 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1409999999999999e-06 2024-03-19T11:30:54.458 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.458 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.462 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:54.462 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084838+0000", 2024-03-19T11:30:54.462 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3740000000000004e-06 2024-03-19T11:30:54.462 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.462 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.462 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:54.462 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084859+0000", 2024-03-19T11:30:54.462 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1644000000000001e-05 2024-03-19T11:30:54.463 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.463 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.463 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:54.463 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084871+0000", 2024-03-19T11:30:54.463 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1127e-05 2024-03-19T11:30:54.463 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.463 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.463 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:54.463 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084961+0000", 2024-03-19T11:30:54.463 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0371999999999995e-05 2024-03-19T11:30:54.463 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.463 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.463 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:54.463 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.085130+0000", 2024-03-19T11:30:54.463 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016884800000000001 2024-03-19T11:30:54.463 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.464 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.464 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5,7", 2024-03-19T11:30:54.464 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.089309+0000", 2024-03-19T11:30:54.464 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041788650000000004 2024-03-19T11:30:54.464 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.464 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.464 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:54.464 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.090362+0000", 2024-03-19T11:30:54.464 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001053362 2024-03-19T11:30:54.464 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.464 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.464 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:54.464 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.090511+0000", 2024-03-19T11:30:54.464 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014887200000000001 2024-03-19T11:30:54.464 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.465 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.465 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:54.465 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.140479+0000", 2024-03-19T11:30:54.465 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.049968006000000002 2024-03-19T11:30:54.465 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.465 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.465 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:54.465 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.140534+0000", 2024-03-19T11:30:54.465 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4917000000000003e-05 2024-03-19T11:30:54.465 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.465 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.465 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:54.465 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.140626+0000", 2024-03-19T11:30:54.465 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2027999999999994e-05 2024-03-19T11:30:54.465 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.466 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:54.466 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.466 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.466 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.466 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1d58db11:::smithi03829856-18:6, version: 55'80, data_included: [759683~639358,1904434~409218], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:6@55'80, size: 3159683, copy_subset: [(0, 3159683)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2313652, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1d58db11:::smithi03829856-18:b, version: 58'84, data_included: [443718~955323,1564138~93253], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:b@58'84, size: 3445402, copy_subset: [(0, 3445402)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1657391, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:10, version: 66'95, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:10@66'95, size: 3764735, copy_subset: [(0, 3764735)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:12, version: 66'97, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:12@66'97, size: 2504243, copy_subset: [(0, 2504243)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:c, version: 70'102, data_included: [416807~576167,1542129~472409], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:c@70'102, size: 2857157, copy_subset: [(0, 2857157)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2014538, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:30:54.466 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.274749+0000", 2024-03-19T11:30:54.466 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.11159047800000001, 2024-03-19T11:30:54.466 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017652601, 2024-03-19T11:30:54.466 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:54.466 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:54.466 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:54.466 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.466 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:54.467 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.274749+0000", 2024-03-19T11:30:54.467 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.467 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.467 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.467 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:54.467 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.274749+0000", 2024-03-19T11:30:54.467 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.467 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.467 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.467 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:54.467 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.274751+0000", 2024-03-19T11:30:54.467 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8500000000000001e-06 2024-03-19T11:30:54.467 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.467 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.467 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:54.468 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.278426+0000", 2024-03-19T11:30:54.468 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036747559999999999 2024-03-19T11:30:54.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.468 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:54.468 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.278470+0000", 2024-03-19T11:30:54.468 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4119000000000002e-05 2024-03-19T11:30:54.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.468 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:54.468 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.278506+0000", 2024-03-19T11:30:54.468 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6066000000000003e-05 2024-03-19T11:30:54.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.468 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:54.469 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.278709+0000", 2024-03-19T11:30:54.469 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000202711 2024-03-19T11:30:54.469 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.469 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.469 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:54.469 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.278731+0000", 2024-03-19T11:30:54.469 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2657e-05 2024-03-19T11:30:54.469 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.469 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.469 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:54.469 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.292402+0000", 2024-03-19T11:30:54.469 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013670442 2024-03-19T11:30:54.469 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.469 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:54.470 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.470 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.470 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.470 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1dabc0f1:::smithi03829856-1:head, version: 70'103, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:head@70'103, size: 2621892, copy_subset: [(0, 2621892)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:30:54.470 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.280012+0000", 2024-03-19T11:30:54.470 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.106327833, 2024-03-19T11:30:54.470 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018569209, 2024-03-19T11:30:54.470 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:54.470 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:54.470 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:54.470 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.470 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:54.470 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280012+0000", 2024-03-19T11:30:54.471 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.471 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.471 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.471 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:54.471 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280012+0000", 2024-03-19T11:30:54.471 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.471 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.471 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.471 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:54.471 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280013+0000", 2024-03-19T11:30:54.471 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9899999999999999e-07 2024-03-19T11:30:54.471 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.471 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.471 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:54.471 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280584+0000", 2024-03-19T11:30:54.472 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057090299999999999 2024-03-19T11:30:54.472 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.472 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.472 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:54.472 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280612+0000", 2024-03-19T11:30:54.472 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8442999999999999e-05 2024-03-19T11:30:54.472 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.472 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.472 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:54.472 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280631+0000", 2024-03-19T11:30:54.472 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8896999999999999e-05 2024-03-19T11:30:54.472 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.472 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.472 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:54.472 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.295041+0000", 2024-03-19T11:30:54.473 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014410127999999999 2024-03-19T11:30:54.473 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.473 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.473 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:54.473 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.295050+0000", 2024-03-19T11:30:54.473 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2350000000000005e-06 2024-03-19T11:30:54.473 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.473 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.473 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:54.473 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.298581+0000", 2024-03-19T11:30:54.473 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035308039999999998 2024-03-19T11:30:54.473 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.473 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:54.473 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.473 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.474 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.474 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.18 e75,74 4:1d58db11:::smithi03829856-18:c,70'104)", 2024-03-19T11:30:54.474 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.280680+0000", 2024-03-19T11:30:54.474 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.10565935799999999, 2024-03-19T11:30:54.474 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018106229000000001, 2024-03-19T11:30:54.474 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:54.474 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:54.474 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:54.474 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.474 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:54.474 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280680+0000", 2024-03-19T11:30:54.474 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.474 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.474 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.474 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:54.475 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280680+0000", 2024-03-19T11:30:54.475 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.475 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.475 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.475 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:54.475 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280681+0000", 2024-03-19T11:30:54.475 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2099999999999995e-07 2024-03-19T11:30:54.475 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.475 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.475 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:54.475 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280686+0000", 2024-03-19T11:30:54.475 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2240000000000003e-06 2024-03-19T11:30:54.475 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.475 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.475 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:54.476 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280698+0000", 2024-03-19T11:30:54.476 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1698999999999999e-05 2024-03-19T11:30:54.476 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.476 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.476 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:54.476 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280705+0000", 2024-03-19T11:30:54.476 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4350000000000002e-06 2024-03-19T11:30:54.476 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.476 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.476 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:54.476 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.298643+0000", 2024-03-19T11:30:54.476 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017938352000000001 2024-03-19T11:30:54.476 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.476 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.476 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:54.476 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.298656+0000", 2024-03-19T11:30:54.477 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3084999999999999e-05 2024-03-19T11:30:54.477 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.477 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.477 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:54.477 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.298787+0000", 2024-03-19T11:30:54.477 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013061299999999999 2024-03-19T11:30:54.477 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.477 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:54.477 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.477 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.477 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.477 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.5.0:143 4.18 e75/74 4:1d58db11:::smithi03829856-18:head v 75'106, mlcod=0'0)", 2024-03-19T11:30:54.477 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.282327+0000", 2024-03-19T11:30:54.477 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.104012987, 2024-03-19T11:30:54.477 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016753244, 2024-03-19T11:30:54.478 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:54.478 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:54.478 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:54.478 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.478 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:54.478 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.282327+0000", 2024-03-19T11:30:54.478 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.478 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.478 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.478 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:54.478 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.282327+0000", 2024-03-19T11:30:54.478 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.478 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.478 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.479 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:54.479 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.282328+0000", 2024-03-19T11:30:54.479 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.488e-06 2024-03-19T11:30:54.479 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.479 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.479 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:54.479 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.283127+0000", 2024-03-19T11:30:54.479 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00079853699999999999 2024-03-19T11:30:54.479 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.479 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.479 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:54.479 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.283152+0000", 2024-03-19T11:30:54.479 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5215000000000002e-05 2024-03-19T11:30:54.479 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.480 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.480 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:54.480 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.283167+0000", 2024-03-19T11:30:54.480 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4635e-05 2024-03-19T11:30:54.480 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.480 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.480 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:54.480 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.292633+0000", 2024-03-19T11:30:54.480 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0094664339999999993 2024-03-19T11:30:54.480 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.480 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.480 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:54.480 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.292655+0000", 2024-03-19T11:30:54.480 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1934999999999999e-05 2024-03-19T11:30:54.481 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.481 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.481 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:54.481 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.298981+0000", 2024-03-19T11:30:54.481 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0063255129999999996 2024-03-19T11:30:54.481 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.481 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.481 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:54.481 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.299080+0000", 2024-03-19T11:30:54.481 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9487000000000005e-05 2024-03-19T11:30:54.481 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.481 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:54.481 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.481 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.481 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:54.482 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:54.483 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-19T11:30:54.483 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:54.491+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:54.584 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:54.594+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:54.654 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:54.654 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:54.654 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:30:54.654 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:54.668 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-19T11:30:54.685 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:54.694+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:54.786 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:54.794+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:54.791 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:54.791 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:54.791 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:30:54.791 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:30:54.791 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:54.804 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-19T11:30:54.887 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:54.894+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:54.927 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:54.927 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:30:54.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:30:54.927 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:30:54.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.927 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:30:54.927 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:44.200920+0000", 2024-03-19T11:30:54.927 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 70.736098838999993, 2024-03-19T11:30:54.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.80881896499999995, 2024-03-19T11:30:54.927 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:54.927 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:54.928 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:54.928 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:54.928 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:54.928 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1 2024-03-19T11:30:54.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.928 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:54.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.928 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:54.928 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200920+0000", 2024-03-19T11:30:54.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.928 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:54.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200920+0000", 2024-03-19T11:30:54.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:54.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200923+0000", 2024-03-19T11:30:54.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6469999999999998e-06 2024-03-19T11:30:54.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:54.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200937+0000", 2024-03-19T11:30:54.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.399e-05 2024-03-19T11:30:54.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:54.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200960+0000", 2024-03-19T11:30:54.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2603e-05 2024-03-19T11:30:54.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:54.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200978+0000", 2024-03-19T11:30:54.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.791e-05 2024-03-19T11:30:54.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:54.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.201081+0000", 2024-03-19T11:30:54.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010344900000000001 2024-03-19T11:30:54.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.931 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:30:54.931 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.201096+0000", 2024-03-19T11:30:54.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4443e-05 2024-03-19T11:30:54.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.931 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:54.931 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.005978+0000", 2024-03-19T11:30:54.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.80488281500000003 2024-03-19T11:30:54.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.931 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:54.931 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.006267+0000", 2024-03-19T11:30:54.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000288688 2024-03-19T11:30:54.932 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.932 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.932 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:54.932 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.006534+0000", 2024-03-19T11:30:54.932 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026725799999999998 2024-03-19T11:30:54.932 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.932 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.932 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:54.932 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.008457+0000", 2024-03-19T11:30:54.932 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0019227820000000001 2024-03-19T11:30:54.932 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.932 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.933 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:54.933 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.009570+0000", 2024-03-19T11:30:54.933 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011131750000000001 2024-03-19T11:30:54.933 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.933 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:54.933 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.009670+0000", 2024-03-19T11:30:54.933 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9458999999999998e-05 2024-03-19T11:30:54.933 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.933 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:54.933 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.009739+0000", 2024-03-19T11:30:54.933 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9746000000000006e-05 2024-03-19T11:30:54.934 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.934 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:54.934 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.934 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:30:54.934 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.071728+0000", 2024-03-19T11:30:54.934 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 69.865291401999997, 2024-03-19T11:30:54.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013714690999999999, 2024-03-19T11:30:54.934 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:54.934 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:54.934 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:54.934 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:54.934 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:54.935 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 34 2024-03-19T11:30:54.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.935 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:54.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:54.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071728+0000", 2024-03-19T11:30:54.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.936 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:54.936 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071728+0000", 2024-03-19T11:30:54.936 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.936 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.936 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.936 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:54.936 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071729+0000", 2024-03-19T11:30:54.936 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.47e-07 2024-03-19T11:30:54.936 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.936 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.936 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:54.936 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071735+0000", 2024-03-19T11:30:54.936 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3799999999999999e-06 2024-03-19T11:30:54.936 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.937 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.937 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:54.937 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071748+0000", 2024-03-19T11:30:54.937 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2806000000000001e-05 2024-03-19T11:30:54.937 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.937 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.937 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:54.937 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071756+0000", 2024-03-19T11:30:54.937 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1240000000000005e-06 2024-03-19T11:30:54.937 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.937 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.938 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:54.938 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.077570+0000", 2024-03-19T11:30:54.938 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.005813342 2024-03-19T11:30:54.938 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.938 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.938 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:54.938 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.077793+0000", 2024-03-19T11:30:54.938 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022391500000000001 2024-03-19T11:30:54.938 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.938 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.938 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:54.938 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.078146+0000", 2024-03-19T11:30:54.938 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00035219 2024-03-19T11:30:54.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.939 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:54.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.083390+0000", 2024-03-19T11:30:54.939 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052445590000000002 2024-03-19T11:30:54.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.939 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:54.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085288+0000", 2024-03-19T11:30:54.939 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0018977060000000001 2024-03-19T11:30:54.940 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.940 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.940 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:54.940 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085377+0000", 2024-03-19T11:30:54.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9068999999999997e-05 2024-03-19T11:30:54.940 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.940 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.940 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:54.940 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085443+0000", 2024-03-19T11:30:54.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5653000000000003e-05 2024-03-19T11:30:54.940 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.940 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:54.941 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.941 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.941 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.941 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:30:54.941 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.071777+0000", 2024-03-19T11:30:54.941 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 69.865242292000005, 2024-03-19T11:30:54.941 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013962445, 2024-03-19T11:30:54.941 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:54.941 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:54.941 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:54.941 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:54.941 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:54.942 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 35 2024-03-19T11:30:54.942 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.942 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:54.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.942 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:54.942 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071777+0000", 2024-03-19T11:30:54.942 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.942 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.942 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:54.943 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071777+0000", 2024-03-19T11:30:54.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:54.943 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071778+0000", 2024-03-19T11:30:54.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5000000000000001e-07 2024-03-19T11:30:54.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:54.944 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071917+0000", 2024-03-19T11:30:54.944 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013880199999999999 2024-03-19T11:30:54.944 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.944 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:54.944 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071931+0000", 2024-03-19T11:30:54.944 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4080000000000001e-05 2024-03-19T11:30:54.944 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.944 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:54.945 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071940+0000", 2024-03-19T11:30:54.945 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.363e-06 2024-03-19T11:30:54.945 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.945 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:54.945 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.080462+0000", 2024-03-19T11:30:54.945 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085221810000000002 2024-03-19T11:30:54.945 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.945 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:54.945 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.080582+0000", 2024-03-19T11:30:54.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011935299999999999 2024-03-19T11:30:54.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.946 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:54.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.080782+0000", 2024-03-19T11:30:54.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020077100000000001 2024-03-19T11:30:54.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.946 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:54.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.083401+0000", 2024-03-19T11:30:54.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002618333 2024-03-19T11:30:54.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.946 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:54.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085571+0000", 2024-03-19T11:30:54.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021702829999999998 2024-03-19T11:30:54.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:54.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085672+0000", 2024-03-19T11:30:54.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100876 2024-03-19T11:30:54.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:54.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085739+0000", 2024-03-19T11:30:54.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7552999999999995e-05 2024-03-19T11:30:54.947 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.947 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:54.947 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.948 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:30:54.948 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.071957+0000", 2024-03-19T11:30:54.948 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 69.865062104000003, 2024-03-19T11:30:54.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013989822000000001, 2024-03-19T11:30:54.948 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:54.948 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:54.948 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:54.948 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:54.948 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:54.948 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 36 2024-03-19T11:30:54.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.948 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:54.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.949 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:54.949 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071957+0000", 2024-03-19T11:30:54.949 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.949 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.949 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:54.949 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071957+0000", 2024-03-19T11:30:54.949 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.949 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.949 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:54.949 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071958+0000", 2024-03-19T11:30:54.949 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.06e-06 2024-03-19T11:30:54.949 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:54.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071965+0000", 2024-03-19T11:30:54.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8929999999999999e-06 2024-03-19T11:30:54.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:54.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071976+0000", 2024-03-19T11:30:54.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1046999999999999e-05 2024-03-19T11:30:54.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:54.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071984+0000", 2024-03-19T11:30:54.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7170000000000008e-06 2024-03-19T11:30:54.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:54.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.081761+0000", 2024-03-19T11:30:54.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0097769310000000009 2024-03-19T11:30:54.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:54.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.081928+0000", 2024-03-19T11:30:54.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000167219 2024-03-19T11:30:54.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.952 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:54.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.082131+0000", 2024-03-19T11:30:54.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020244499999999999 2024-03-19T11:30:54.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.952 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:54.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.083455+0000", 2024-03-19T11:30:54.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001324073 2024-03-19T11:30:54.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.952 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:54.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085816+0000", 2024-03-19T11:30:54.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0023613309999999999 2024-03-19T11:30:54.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.953 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:54.953 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085892+0000", 2024-03-19T11:30:54.953 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6154000000000003e-05 2024-03-19T11:30:54.953 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.953 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.953 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:54.953 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085947+0000", 2024-03-19T11:30:54.953 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4951999999999998e-05 2024-03-19T11:30:54.953 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.953 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:54.953 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.953 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.953 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.953 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:30:54.954 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.072000+0000", 2024-03-19T11:30:54.954 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 69.865019353999998, 2024-03-19T11:30:54.954 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014418268999999999, 2024-03-19T11:30:54.954 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:54.954 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:54.954 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:54.954 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:54.954 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:54.954 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 37 2024-03-19T11:30:54.954 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.954 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:54.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.954 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:54.954 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072000+0000", 2024-03-19T11:30:54.954 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:54.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072000+0000", 2024-03-19T11:30:54.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:54.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072001+0000", 2024-03-19T11:30:54.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0699999999999996e-07 2024-03-19T11:30:54.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:54.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072007+0000", 2024-03-19T11:30:54.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2060000000000004e-06 2024-03-19T11:30:54.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.956 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:54.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072023+0000", 2024-03-19T11:30:54.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6617e-05 2024-03-19T11:30:54.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.956 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:54.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072031+0000", 2024-03-19T11:30:54.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8900000000000007e-06 2024-03-19T11:30:54.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.956 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:54.957 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.082682+0000", 2024-03-19T11:30:54.957 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010650868000000001 2024-03-19T11:30:54.957 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.957 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.957 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:54.957 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.082804+0000", 2024-03-19T11:30:54.957 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012202899999999999 2024-03-19T11:30:54.957 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.957 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.957 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:54.957 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.083006+0000", 2024-03-19T11:30:54.957 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020127799999999999 2024-03-19T11:30:54.957 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.957 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.957 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:54.958 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.084145+0000", 2024-03-19T11:30:54.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001139791 2024-03-19T11:30:54.958 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.958 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.958 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:54.958 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.086292+0000", 2024-03-19T11:30:54.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021461969999999999 2024-03-19T11:30:54.958 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.958 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.958 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:54.958 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.086364+0000", 2024-03-19T11:30:54.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2439000000000002e-05 2024-03-19T11:30:54.958 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.958 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.959 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:54.959 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.086418+0000", 2024-03-19T11:30:54.959 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4246999999999997e-05 2024-03-19T11:30:54.959 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.959 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:54.959 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.959 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.959 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.959 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:30:54.959 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.339898+0000", 2024-03-19T11:30:54.959 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 69.597121732000005, 2024-03-19T11:30:54.959 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019501793, 2024-03-19T11:30:54.960 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:54.960 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:54.960 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:54.960 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:54.960 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:54.960 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113 2024-03-19T11:30:54.960 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.960 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:54.960 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.960 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:54.960 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339898+0000", 2024-03-19T11:30:54.960 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.960 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.960 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.960 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:54.961 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339898+0000", 2024-03-19T11:30:54.961 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.961 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.961 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.961 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:54.961 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339899+0000", 2024-03-19T11:30:54.961 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.353e-06 2024-03-19T11:30:54.961 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.961 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.961 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:54.961 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339907+0000", 2024-03-19T11:30:54.961 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3000000000000002e-06 2024-03-19T11:30:54.961 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.961 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.961 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:54.962 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339924+0000", 2024-03-19T11:30:54.962 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6906999999999999e-05 2024-03-19T11:30:54.962 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.962 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.962 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:54.962 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339933+0000", 2024-03-19T11:30:54.962 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8969999999999997e-06 2024-03-19T11:30:54.962 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.962 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.962 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:54.962 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352239+0000", 2024-03-19T11:30:54.962 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012305814999999999 2024-03-19T11:30:54.962 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.962 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.963 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:54.963 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352466+0000", 2024-03-19T11:30:54.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022684100000000001 2024-03-19T11:30:54.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.963 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:54.963 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352816+0000", 2024-03-19T11:30:54.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00035030499999999997 2024-03-19T11:30:54.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.963 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:54.963 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.358956+0000", 2024-03-19T11:30:54.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061400079999999998 2024-03-19T11:30:54.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.964 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:54.964 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359118+0000", 2024-03-19T11:30:54.964 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016237599999999999 2024-03-19T11:30:54.964 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.964 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.964 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:54.964 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359287+0000", 2024-03-19T11:30:54.964 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000168488 2024-03-19T11:30:54.964 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.964 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.964 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:54.964 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359399+0000", 2024-03-19T11:30:54.964 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000112503 2024-03-19T11:30:54.965 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.965 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:54.965 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.965 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.965 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.965 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:30:54.965 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.339947+0000", 2024-03-19T11:30:54.965 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 69.597071964999998, 2024-03-19T11:30:54.965 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021923069999999999, 2024-03-19T11:30:54.965 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:54.965 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:54.965 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:54.965 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:54.965 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:54.966 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 114 2024-03-19T11:30:54.966 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.966 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:54.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.966 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:54.966 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339947+0000", 2024-03-19T11:30:54.966 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.966 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.966 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:54.966 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339947+0000", 2024-03-19T11:30:54.966 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.966 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.967 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.967 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:54.967 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339948+0000", 2024-03-19T11:30:54.967 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6000000000000003e-07 2024-03-19T11:30:54.967 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.967 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.967 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:54.967 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339956+0000", 2024-03-19T11:30:54.967 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1559999999999995e-06 2024-03-19T11:30:54.967 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.967 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.967 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:54.967 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339967+0000", 2024-03-19T11:30:54.967 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0669999999999999e-05 2024-03-19T11:30:54.967 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.968 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.968 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:54.968 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339975+0000", 2024-03-19T11:30:54.968 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0099999999999995e-06 2024-03-19T11:30:54.968 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.968 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.968 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:54.968 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.353672+0000", 2024-03-19T11:30:54.968 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013696583999999999 2024-03-19T11:30:54.968 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.968 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.968 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:54.968 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.353808+0000", 2024-03-19T11:30:54.968 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013652299999999999 2024-03-19T11:30:54.969 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.969 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.969 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:54.969 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.354036+0000", 2024-03-19T11:30:54.969 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022744799999999999 2024-03-19T11:30:54.969 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.969 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.969 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:54.969 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.360480+0000", 2024-03-19T11:30:54.969 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0064441630000000001 2024-03-19T11:30:54.969 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.969 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.969 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:54.969 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361722+0000", 2024-03-19T11:30:54.969 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001242754 2024-03-19T11:30:54.970 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.970 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.970 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:54.970 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361807+0000", 2024-03-19T11:30:54.970 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4605999999999996e-05 2024-03-19T11:30:54.970 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.970 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.970 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:54.970 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361870+0000", 2024-03-19T11:30:54.970 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3396000000000001e-05 2024-03-19T11:30:54.971 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.971 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:54.971 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.971 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.971 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.971 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:30:54.971 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.340017+0000", 2024-03-19T11:30:54.971 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 69.597002649000004, 2024-03-19T11:30:54.971 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022169355000000002, 2024-03-19T11:30:54.971 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:54.971 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:54.971 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:54.971 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:54.971 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:54.971 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 115 2024-03-19T11:30:54.972 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.972 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:54.972 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.972 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:54.972 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340017+0000", 2024-03-19T11:30:54.972 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.972 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.972 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.972 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:54.972 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340017+0000", 2024-03-19T11:30:54.972 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.973 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.973 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.973 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:54.973 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340018+0000", 2024-03-19T11:30:54.973 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.043e-06 2024-03-19T11:30:54.973 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.973 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.973 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:54.973 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340139+0000", 2024-03-19T11:30:54.973 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121502 2024-03-19T11:30:54.973 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.973 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.973 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:54.973 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340157+0000", 2024-03-19T11:30:54.974 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7382999999999999e-05 2024-03-19T11:30:54.974 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.974 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.974 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:54.974 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340166+0000", 2024-03-19T11:30:54.974 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2469999999999997e-06 2024-03-19T11:30:54.974 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.974 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.974 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:54.974 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.357470+0000", 2024-03-19T11:30:54.974 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017304119999999999 2024-03-19T11:30:54.974 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.974 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.975 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:54.975 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.357671+0000", 2024-03-19T11:30:54.975 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000200798 2024-03-19T11:30:54.975 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.975 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.975 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:54.975 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.358002+0000", 2024-03-19T11:30:54.975 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033090099999999998 2024-03-19T11:30:54.975 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.975 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.975 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:54.976 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361892+0000", 2024-03-19T11:30:54.976 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00389018 2024-03-19T11:30:54.976 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.976 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.976 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:54.976 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361983+0000", 2024-03-19T11:30:54.976 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0976000000000004e-05 2024-03-19T11:30:54.976 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.976 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.976 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:54.976 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362071+0000", 2024-03-19T11:30:54.977 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8022999999999994e-05 2024-03-19T11:30:54.977 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.977 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.977 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:54.977 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362186+0000", 2024-03-19T11:30:54.977 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115182 2024-03-19T11:30:54.977 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.977 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:54.977 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.977 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.977 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.977 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:30:54.977 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.340183+0000", 2024-03-19T11:30:54.978 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 69.596835816999999, 2024-03-19T11:30:54.978 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022279841000000002, 2024-03-19T11:30:54.978 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:54.978 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:54.978 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:54.978 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:54.978 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:54.978 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 116 2024-03-19T11:30:54.978 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.978 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:54.978 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.979 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:54.979 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340183+0000", 2024-03-19T11:30:54.979 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.979 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.979 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.979 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:54.979 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340183+0000", 2024-03-19T11:30:54.979 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.979 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.979 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.979 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:54.980 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340184+0000", 2024-03-19T11:30:54.980 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9000000000000005e-07 2024-03-19T11:30:54.980 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.980 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.980 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:54.980 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340191+0000", 2024-03-19T11:30:54.980 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0360000000000003e-06 2024-03-19T11:30:54.980 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.980 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.980 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:54.981 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340201+0000", 2024-03-19T11:30:54.981 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0844e-05 2024-03-19T11:30:54.981 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.981 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.981 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:54.981 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340209+0000", 2024-03-19T11:30:54.981 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7360000000000001e-06 2024-03-19T11:30:54.981 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.981 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.982 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:54.982 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359535+0000", 2024-03-19T11:30:54.982 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019325546999999998 2024-03-19T11:30:54.982 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.982 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.982 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:54.982 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359672+0000", 2024-03-19T11:30:54.982 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013706600000000001 2024-03-19T11:30:54.982 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.983 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.983 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:54.983 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359905+0000", 2024-03-19T11:30:54.983 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023334500000000001 2024-03-19T11:30:54.983 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.983 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.983 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:54.983 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362306+0000", 2024-03-19T11:30:54.983 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002401041 2024-03-19T11:30:54.983 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.983 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.983 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:54.983 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362329+0000", 2024-03-19T11:30:54.983 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3283999999999999e-05 2024-03-19T11:30:54.983 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.984 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.984 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:54.984 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362402+0000", 2024-03-19T11:30:54.984 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2459000000000005e-05 2024-03-19T11:30:54.984 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.984 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.984 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:54.984 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362463+0000", 2024-03-19T11:30:54.984 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1493000000000002e-05 2024-03-19T11:30:54.984 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.984 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:54.984 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.984 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.984 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.984 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:30:54.984 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.340222+0000", 2024-03-19T11:30:54.985 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 69.596797233999993, 2024-03-19T11:30:54.985 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022500218999999998, 2024-03-19T11:30:54.985 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:54.985 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:54.985 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:54.985 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:54.985 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:54.985 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 117 2024-03-19T11:30:54.985 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.985 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:54.985 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.985 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:54.985 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340222+0000", 2024-03-19T11:30:54.985 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.986 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.986 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.986 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:54.986 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340222+0000", 2024-03-19T11:30:54.986 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.986 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.986 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.986 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:54.986 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340223+0000", 2024-03-19T11:30:54.986 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7999999999999995e-07 2024-03-19T11:30:54.986 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.986 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.986 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:54.987 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340229+0000", 2024-03-19T11:30:54.987 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0399999999999998e-06 2024-03-19T11:30:54.987 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.987 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.987 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:54.987 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340239+0000", 2024-03-19T11:30:54.987 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0190000000000001e-05 2024-03-19T11:30:54.987 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.987 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.987 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:54.987 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340250+0000", 2024-03-19T11:30:54.987 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0720000000000001e-05 2024-03-19T11:30:54.987 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.987 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.987 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:54.988 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.360587+0000", 2024-03-19T11:30:54.988 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020337542 2024-03-19T11:30:54.988 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.988 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.988 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:54.988 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.360818+0000", 2024-03-19T11:30:54.988 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023120500000000001 2024-03-19T11:30:54.988 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.988 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.988 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:54.988 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361218+0000", 2024-03-19T11:30:54.988 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000399637 2024-03-19T11:30:54.988 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.988 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.988 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:54.989 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362483+0000", 2024-03-19T11:30:54.989 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012649200000000001 2024-03-19T11:30:54.989 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.989 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.989 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:54.989 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362538+0000", 2024-03-19T11:30:54.989 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4490000000000002e-05 2024-03-19T11:30:54.989 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.989 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.989 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:54.989 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362617+0000", 2024-03-19T11:30:54.989 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9839000000000006e-05 2024-03-19T11:30:54.989 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.990 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.990 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:54.990 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362722+0000", 2024-03-19T11:30:54.990 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104956 2024-03-19T11:30:54.990 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.990 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:54.990 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.990 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.990 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.990 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1047 4.8 e35/21 4:1d58db11:::smithi03829856-18:head v 35'32, mlcod=33'29)", 2024-03-19T11:30:54.990 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:13.225023+0000", 2024-03-19T11:30:54.990 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 41.711996448999997, 2024-03-19T11:30:54.990 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013059688999999999, 2024-03-19T11:30:54.990 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:54.991 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:54.991 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:54.991 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.991 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:54.991 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.225023+0000", 2024-03-19T11:30:54.991 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.991 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.991 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.991 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:54.991 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.225023+0000", 2024-03-19T11:30:54.991 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.991 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.991 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.991 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:54.992 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.225025+0000", 2024-03-19T11:30:54.992 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.17e-06 2024-03-19T11:30:54.992 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.992 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.992 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:54.992 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.225771+0000", 2024-03-19T11:30:54.992 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000746267 2024-03-19T11:30:54.992 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.992 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.992 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:54.992 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.225807+0000", 2024-03-19T11:30:54.992 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5487000000000001e-05 2024-03-19T11:30:54.992 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.992 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.993 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:54.993 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.225823+0000", 2024-03-19T11:30:54.993 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6215999999999999e-05 2024-03-19T11:30:54.993 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.993 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.993 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:54.993 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.225990+0000", 2024-03-19T11:30:54.993 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016680899999999999 2024-03-19T11:30:54.993 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.993 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.993 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:54.993 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.226058+0000", 2024-03-19T11:30:54.993 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8672999999999998e-05 2024-03-19T11:30:54.993 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.994 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.994 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:54.994 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.237940+0000", 2024-03-19T11:30:54.994 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011881311 2024-03-19T11:30:54.994 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.994 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.994 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:54.994 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.238083+0000", 2024-03-19T11:30:54.994 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014275600000000001 2024-03-19T11:30:54.994 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.994 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:54.995 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.995 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.995 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.995 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1067 4.0 e36/21 4:01018511:::smithi03829856-19:head v 36'18, mlcod=35'14)", 2024-03-19T11:30:54.995 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:13.952492+0000", 2024-03-19T11:30:54.995 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 40.984526991999999, 2024-03-19T11:30:54.995 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013554725, 2024-03-19T11:30:54.995 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:54.995 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:54.995 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:54.995 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.995 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:54.995 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.952492+0000", 2024-03-19T11:30:54.996 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.996 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.996 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.996 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:54.996 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.952492+0000", 2024-03-19T11:30:54.996 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:54.996 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.996 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.996 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:54.996 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.952493+0000", 2024-03-19T11:30:54.997 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9299999999999997e-07 2024-03-19T11:30:54.997 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.997 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.997 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:54.997 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.954109+0000", 2024-03-19T11:30:54.997 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0016163569999999999 2024-03-19T11:30:54.997 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.997 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.997 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:54.997 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.954135+0000", 2024-03-19T11:30:54.997 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5352999999999999e-05 2024-03-19T11:30:54.997 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.997 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.997 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:54.997 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.954149+0000", 2024-03-19T11:30:54.998 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3879999999999999e-05 2024-03-19T11:30:54.998 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.998 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.998 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:54.998 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.954288+0000", 2024-03-19T11:30:54.998 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000139476 2024-03-19T11:30:54.998 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.998 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.998 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:54.998 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.954356+0000", 2024-03-19T11:30:54.998 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7533000000000005e-05 2024-03-19T11:30:54.998 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.998 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.998 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:54.998 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.965391+0000", 2024-03-19T11:30:54.999 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011035764999999999 2024-03-19T11:30:54.999 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.999 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.999 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:54.999 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.966047+0000", 2024-03-19T11:30:54.999 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00065556800000000004 2024-03-19T11:30:54.999 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.999 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:54.999 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:54.999 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:54.999 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:54.999 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1122 4.8 e36/21 4:1d58db11:::smithi03829856-18:head v 36'42, mlcod=36'39)", 2024-03-19T11:30:54.999 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:14.172666+0000", 2024-03-19T11:30:55.000 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 40.764353217999997, 2024-03-19T11:30:55.000 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018015000999999999, 2024-03-19T11:30:55.000 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:55.000 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:55.000 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:55.000 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.000 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:55.000 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.172666+0000", 2024-03-19T11:30:55.000 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:55.000 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.000 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.000 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:55.000 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.172666+0000", 2024-03-19T11:30:55.000 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:55.000 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.001 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.001 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:55.001 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.172669+0000", 2024-03-19T11:30:55.001 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1470000000000001e-06 2024-03-19T11:30:55.001 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.001 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.001 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:55.001 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.173140+0000", 2024-03-19T11:30:55.001 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00047074200000000001 2024-03-19T11:30:55.001 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.001 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.001 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:55.001 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.173188+0000", 2024-03-19T11:30:55.001 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8287e-05 2024-03-19T11:30:55.001 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.002 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.002 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:55.002 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.173211+0000", 2024-03-19T11:30:55.002 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3023e-05 2024-03-19T11:30:55.002 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.002 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.002 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:55.002 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.173378+0000", 2024-03-19T11:30:55.002 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000166742 2024-03-19T11:30:55.002 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.002 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.002 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:55.002 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.173557+0000", 2024-03-19T11:30:55.003 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000178569 2024-03-19T11:30:55.003 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.003 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.003 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:55.003 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.190416+0000", 2024-03-19T11:30:55.003 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016859570000000001 2024-03-19T11:30:55.003 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.003 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.003 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:55.003 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.190681+0000", 2024-03-19T11:30:55.003 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026492099999999999 2024-03-19T11:30:55.003 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:55.003 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:55.004 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:55.004 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.004 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.004 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.5.0:43 4.8 e39/21 4:15708482:::smithi03829856-23:head v 39'50, mlcod=38'47)", 2024-03-19T11:30:55.004 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.842819+0000", 2024-03-19T11:30:55.004 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 38.094200714999999, 2024-03-19T11:30:55.004 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017626513, 2024-03-19T11:30:55.004 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:55.004 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:55.004 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:55.004 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.004 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:55.004 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842819+0000", 2024-03-19T11:30:55.004 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:55.005 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.005 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.005 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:55.005 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842819+0000", 2024-03-19T11:30:55.005 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:55.005 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.005 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.005 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:55.005 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842820+0000", 2024-03-19T11:30:55.005 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.39e-06 2024-03-19T11:30:55.006 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.006 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.006 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:55.006 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.846437+0000", 2024-03-19T11:30:55.006 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003616953 2024-03-19T11:30:55.006 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.006 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.006 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:55.006 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.846471+0000", 2024-03-19T11:30:55.006 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4133e-05 2024-03-19T11:30:55.006 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.006 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.006 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:55.006 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.846539+0000", 2024-03-19T11:30:55.007 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8072999999999997e-05 2024-03-19T11:30:55.007 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.007 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.007 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:55.007 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.847094+0000", 2024-03-19T11:30:55.007 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00055472099999999999 2024-03-19T11:30:55.007 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.007 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.007 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:55.007 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.847128+0000", 2024-03-19T11:30:55.008 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3875999999999999e-05 2024-03-19T11:30:55.008 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.008 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.008 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:55.008 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.860308+0000", 2024-03-19T11:30:55.008 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013180712000000001 2024-03-19T11:30:55.009 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.009 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.009 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:55.009 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.860445+0000", 2024-03-19T11:30:55.009 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013665500000000001 2024-03-19T11:30:55.009 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:55.009 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:55.009 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:55.009 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.009 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.009 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1209 4.8 e41/21 4:1d58db11:::smithi03829856-18:head v 41'53, mlcod=39'50)", 2024-03-19T11:30:55.009 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:18.124878+0000", 2024-03-19T11:30:55.009 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 36.812141021999999, 2024-03-19T11:30:55.010 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016809343000000001, 2024-03-19T11:30:55.010 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:55.010 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:55.010 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:55.010 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.010 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:55.010 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.124878+0000", 2024-03-19T11:30:55.010 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:55.010 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.010 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.010 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:55.010 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.124878+0000", 2024-03-19T11:30:55.010 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:55.010 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.010 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.011 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:55.011 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.124880+0000", 2024-03-19T11:30:55.011 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4959999999999999e-06 2024-03-19T11:30:55.011 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.011 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.011 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:55.011 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125845+0000", 2024-03-19T11:30:55.011 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00096535299999999996 2024-03-19T11:30:55.011 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.011 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.011 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:55.011 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125884+0000", 2024-03-19T11:30:55.011 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8645e-05 2024-03-19T11:30:55.011 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.012 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.012 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:55.012 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125909+0000", 2024-03-19T11:30:55.012 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4783e-05 2024-03-19T11:30:55.012 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.012 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.012 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:55.012 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.126085+0000", 2024-03-19T11:30:55.012 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000176649 2024-03-19T11:30:55.012 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.012 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.012 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:55.012 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.126150+0000", 2024-03-19T11:30:55.012 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4290999999999998e-05 2024-03-19T11:30:55.012 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.013 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.013 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:55.013 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.141552+0000", 2024-03-19T11:30:55.013 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015402295 2024-03-19T11:30:55.013 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.013 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.013 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:55.013 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.141688+0000", 2024-03-19T11:30:55.013 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013583099999999999 2024-03-19T11:30:55.013 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:55.013 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:55.013 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:55.013 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.013 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.013 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.15 51/50 [PushOp(4:afa9f208:::smithi03829856-35:head, version: 28'46, data_included: [571438~623878,1806623~424698], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:afa9f208:::smithi03829856-35:head@28'46, size: 2571438, copy_subset: [(0, 2571438)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2231321, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:af9d0d56:::smithi03829856-20:head, version: 35'55, data_included: [655360~589824,1900544~418789], data_size: 1008613, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:af9d0d56:::smithi03829856-20:head@35'55, size: 2319333, copy_subset: [(0, 2319333)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2319333, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ad3ad1b3:::smithi03829856-21:b, version: 47'66, data_included: [736140~537686,2020581~315559], data_size: 853245, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:ad3ad1b3:::smithi03829856-21:b@47'66, size: 2336140, copy_subset: [(0, 2336140)], clone_subset: {}, snapset: b=[]:{b: [b, a, 6, 5]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2336140, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ad3ad1b3:::smithi03829856-21:head, version: 47'70, data_included: [531121~742705,1711057~305871], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:ad3ad1b3:::smithi03829856-21:head@47'70, size: 3731121, copy_subset: [(0, 3731121)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2016928, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ac9cd3a3:::smithi03829856-8:head, version: 49'72, data_included: [517649~620182,1776668~428394], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:ac9cd3a3:::smithi03829856-8:head@49'72, size: 3008420, copy_subset: [(0, 3008420)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2205062, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:aa3303a5:::smithi03829856-17:b, version: 49'73, data_included: [614400~692224,1903006~356352], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:aa3303a5:::smithi03829856-17:b@49'73, size: 2268912, copy_subset: [(0, 2268912)], clone_subset: {}, snapset: b=[]:{b: [b, a, 6, 5]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2259358, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:aa3303a5:::smithi03829856-17:head, version: 49'74, data_included: [413472~893152,1818036~155424], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:aa3303a5:::smithi03829856-17:head@49'74, size: 3613472, copy_subset: [(0, 3613472)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1973460, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:30:55.014 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:28.058610+0000", 2024-03-19T11:30:55.014 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 26.878409061999999, 2024-03-19T11:30:55.014 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027100032, 2024-03-19T11:30:55.014 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:55.014 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:55.014 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:55.014 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.014 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:55.014 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.058610+0000", 2024-03-19T11:30:55.014 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:55.014 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.014 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.014 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:55.014 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.058610+0000", 2024-03-19T11:30:55.015 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:55.015 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.015 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.015 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:55.015 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.058611+0000", 2024-03-19T11:30:55.015 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1599999999999999e-06 2024-03-19T11:30:55.015 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.015 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.015 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:55.015 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.063874+0000", 2024-03-19T11:30:55.015 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052625270000000004 2024-03-19T11:30:55.015 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.015 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.015 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:55.015 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.063935+0000", 2024-03-19T11:30:55.015 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0594000000000003e-05 2024-03-19T11:30:55.016 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.016 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.016 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:55.016 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064024+0000", 2024-03-19T11:30:55.016 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9273999999999999e-05 2024-03-19T11:30:55.016 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.016 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.016 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:55.016 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064385+0000", 2024-03-19T11:30:55.016 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00036129100000000002 2024-03-19T11:30:55.016 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.016 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.016 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:55.016 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064437+0000", 2024-03-19T11:30:55.016 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2346000000000003e-05 2024-03-19T11:30:55.017 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.017 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.017 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:55.017 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085710+0000", 2024-03-19T11:30:55.017 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021272840000000001 2024-03-19T11:30:55.017 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:55.017 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:55.017 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:55.017 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.017 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.017 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.15 e51,50 4:ac9cd3a3:::smithi03829856-8:4,49'71)", 2024-03-19T11:30:55.017 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:28.064112+0000", 2024-03-19T11:30:55.017 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 26.872907236, 2024-03-19T11:30:55.017 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026480417999999999, 2024-03-19T11:30:55.017 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:55.018 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:55.018 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:55.018 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.018 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:55.018 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064112+0000", 2024-03-19T11:30:55.018 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:55.018 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.018 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.018 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:55.018 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064112+0000", 2024-03-19T11:30:55.018 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:55.018 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.018 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.018 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:55.019 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064113+0000", 2024-03-19T11:30:55.019 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3600000000000003e-07 2024-03-19T11:30:55.019 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.019 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.019 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:55.019 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064118+0000", 2024-03-19T11:30:55.019 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0549999999999999e-06 2024-03-19T11:30:55.019 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.019 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.019 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:55.019 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064128+0000", 2024-03-19T11:30:55.019 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0349e-05 2024-03-19T11:30:55.019 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.019 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.020 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:55.020 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064136+0000", 2024-03-19T11:30:55.020 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6249999999999998e-06 2024-03-19T11:30:55.020 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.020 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.020 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:55.020 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085773+0000", 2024-03-19T11:30:55.020 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021636829999999999 2024-03-19T11:30:55.020 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.020 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.020 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:55.020 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085788+0000", 2024-03-19T11:30:55.020 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5512999999999999e-05 2024-03-19T11:30:55.020 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.020 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.021 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:55.021 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.090592+0000", 2024-03-19T11:30:55.021 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0048044100000000003 2024-03-19T11:30:55.021 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:55.021 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:55.021 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:55.021 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.021 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.021 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.5 51/50 [PushOp(4:a38b9453:::smithi03829856-37:head, version: 28'45, data_included: [605052~774376,1923587~274200], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:a38b9453:::smithi03829856-37:head@28'45, size: 2605052, copy_subset: [(0, 2605052)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2197787, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:a0a50d04:::smithi03829856-13:b, version: 47'63, data_included: [601684~619341,1669078~429235], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a0a50d04:::smithi03829856-13:b@47'63, size: 2201684, copy_subset: [(0, 2201684)], clone_subset: {}, snapset: b=[]:{b: [b, a, 6, 5]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2098313, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:30:55.021 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:28.064324+0000", 2024-03-19T11:30:55.021 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 26.872695392000001, 2024-03-19T11:30:55.021 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027179020000000002, 2024-03-19T11:30:55.021 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:55.021 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:55.021 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:55.022 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.022 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:55.022 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064324+0000", 2024-03-19T11:30:55.022 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:55.022 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.022 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.022 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:55.022 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064324+0000", 2024-03-19T11:30:55.022 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:55.022 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.022 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.022 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:55.022 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064324+0000", 2024-03-19T11:30:55.022 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1199999999999998e-07 2024-03-19T11:30:55.023 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.023 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.023 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:55.023 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.065527+0000", 2024-03-19T11:30:55.023 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012028939999999999 2024-03-19T11:30:55.023 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.023 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.023 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:55.023 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.065552+0000", 2024-03-19T11:30:55.023 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4440999999999999e-05 2024-03-19T11:30:55.023 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.023 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.023 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:55.023 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.065568+0000", 2024-03-19T11:30:55.023 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6498e-05 2024-03-19T11:30:55.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.024 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:55.024 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085812+0000", 2024-03-19T11:30:55.024 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020243954000000002 2024-03-19T11:30:55.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.024 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:55.024 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085825+0000", 2024-03-19T11:30:55.024 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3298999999999999e-05 2024-03-19T11:30:55.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.024 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:55.024 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.091503+0000", 2024-03-19T11:30:55.024 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.005677522 2024-03-19T11:30:55.024 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:55.025 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:55.025 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:55.025 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.025 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.025 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.24 75/74 [PushOp(4:27909373:::smithi03829856-46:8, version: 46'59, data_included: [756315~767970,1933572~280606], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:8@46'59, size: 2756315, copy_subset: [(0, 2756315)], clone_subset: {}, snapset: 10=[]:{8: [6], c: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2214178, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:266590a5:::smithi03829856-16:e, version: 70'107, data_included: [652105~662359,1728171~386217], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:e@70'107, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 10=[]:{e: [d, b, 6]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:266590a5:::smithi03829856-16:head, version: 70'108, data_included: [652105~662359,1728171~386217], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:head@70'108, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:27909373:::smithi03829856-46:c, version: 70'109, data_included: [628595~895690,1807440~152886], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:c@70'109, size: 3428595, copy_subset: [(0, 3428595)], clone_subset: {}, snapset: 10=[]:{8: [6], c: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1960326, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:27909373:::smithi03829856-46:head, version: 70'110, data_included: [403401~454961,1584925~506039], data_size: 961000, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:head@70'110, size: 2403401, copy_subset: [(0, 2403401)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2403401, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:30:55.025 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.099172+0000", 2024-03-19T11:30:55.025 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.83784695799999998, 2024-03-19T11:30:55.025 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.024528580000000001, 2024-03-19T11:30:55.025 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:55.025 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:55.025 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:55.025 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.025 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:55.025 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.099172+0000", 2024-03-19T11:30:55.026 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:55.026 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.026 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.026 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:55.026 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.099172+0000", 2024-03-19T11:30:55.026 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:55.026 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.026 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.026 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:55.026 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.099175+0000", 2024-03-19T11:30:55.026 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7750000000000001e-06 2024-03-19T11:30:55.026 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.026 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.026 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:55.027 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105219+0000", 2024-03-19T11:30:55.027 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0060440789999999999 2024-03-19T11:30:55.027 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.027 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.027 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:55.027 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105285+0000", 2024-03-19T11:30:55.027 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6260000000000006e-05 2024-03-19T11:30:55.027 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.027 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.027 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:55.027 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105366+0000", 2024-03-19T11:30:55.027 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.038e-05 2024-03-19T11:30:55.027 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.027 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.027 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:55.028 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105637+0000", 2024-03-19T11:30:55.028 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027085199999999997 2024-03-19T11:30:55.028 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.028 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:55.028 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105679+0000", 2024-03-19T11:30:55.028 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2088000000000001e-05 2024-03-19T11:30:55.028 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.028 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:55.028 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.123701+0000", 2024-03-19T11:30:55.028 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018022145999999999 2024-03-19T11:30:55.028 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:55.028 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:55.028 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:55.028 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.029 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.029 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.24 e75,74 4:266590a5:::smithi03829856-16:4,47'63,4:27909373:::smithi03829856-46:5,55'79,4:266590a5:::smithi03829856-16:10,67'105)", 2024-03-19T11:30:55.029 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.105437+0000", 2024-03-19T11:30:55.029 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 0.83158195499999998, 2024-03-19T11:30:55.029 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.01860726, 2024-03-19T11:30:55.029 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:55.029 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:55.029 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:55.029 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.029 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:55.029 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105437+0000", 2024-03-19T11:30:55.029 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:55.029 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.029 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.029 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:55.030 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105437+0000", 2024-03-19T11:30:55.030 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:55.030 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.030 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.030 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:55.030 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105438+0000", 2024-03-19T11:30:55.030 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6799999999999999e-07 2024-03-19T11:30:55.030 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.030 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.030 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:55.030 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105445+0000", 2024-03-19T11:30:55.030 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3749999999999999e-06 2024-03-19T11:30:55.030 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.030 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.030 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:55.031 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105454+0000", 2024-03-19T11:30:55.031 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7909999999999999e-06 2024-03-19T11:30:55.031 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.031 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.031 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:55.031 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105465+0000", 2024-03-19T11:30:55.031 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.045e-05 2024-03-19T11:30:55.031 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.031 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.031 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:55.031 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.123753+0000", 2024-03-19T11:30:55.031 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0182884 2024-03-19T11:30:55.031 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.031 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.031 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:55.032 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.123765+0000", 2024-03-19T11:30:55.032 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2143e-05 2024-03-19T11:30:55.032 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:55.032 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:55.032 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:55.032 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.124045+0000", 2024-03-19T11:30:55.032 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027923299999999999 2024-03-19T11:30:55.032 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:55.032 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:55.032 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:55.032 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:55.032 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:55.032 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:55.034 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:55.043+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:55.034 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-19T11:30:55.042 INFO:tasks.rados.rados.0.smithi038.stdout:1124: done (7 left) 2024-03-19T11:30:55.042 INFO:tasks.rados.rados.0.smithi038.stdout:1131: done (6 left) 2024-03-19T11:30:55.042 INFO:tasks.rados.rados.0.smithi038.stdout:1132: done (5 left) 2024-03-19T11:30:55.042 INFO:tasks.rados.rados.0.smithi038.stdout:1133: done (4 left) 2024-03-19T11:30:55.042 INFO:tasks.rados.rados.0.smithi038.stdout:1134: done (3 left) 2024-03-19T11:30:55.042 INFO:tasks.rados.rados.0.smithi038.stdout:1135: done (2 left) 2024-03-19T11:30:55.042 INFO:tasks.rados.rados.0.smithi038.stdout:1136: done (1 left) 2024-03-19T11:30:55.042 INFO:tasks.rados.rados.0.smithi038.stdout:1137: done (0 left) 2024-03-19T11:30:55.043 INFO:tasks.rados.rados.0.smithi038.stdout:1138: read oid 46 snap -1 2024-03-19T11:30:55.043 INFO:tasks.rados.rados.0.smithi038.stdout:1138: expect (ObjNum 143 snap 15 seq_num 143) 2024-03-19T11:30:55.043 INFO:tasks.rados.rados.0.smithi038.stdout:1139: set_chunk oid 5 target oid 2024-03-19T11:30:55.061 INFO:tasks.rados.rados.0.smithi038.stdout:1139: set_chunk oid 5 offset: 2799104 length: 42496 target oid ffcb82381cebfa5927b6bf675c96fa99e4818c04e04e1da588f02886f8375a8c offset: 0 snap -1 2024-03-19T11:30:55.061 INFO:tasks.rados.rados.0.smithi038.stdout:1140: read oid 37 snap 9 2024-03-19T11:30:55.061 INFO:tasks.rados.rados.0.smithi038.stdout:1140: expect (ObjNum 36 snap 0 seq_num 36) 2024-03-19T11:30:55.061 INFO:tasks.rados.rados.0.smithi038.stdout:1141: read oid 19 snap -1 2024-03-19T11:30:55.061 INFO:tasks.rados.rados.0.smithi038.stdout:1141: expect deleted 2024-03-19T11:30:55.061 INFO:tasks.rados.rados.0.smithi038.stdout:1142: read oid 24 snap -1 2024-03-19T11:30:55.061 INFO:tasks.rados.rados.0.smithi038.stdout:1142: expect (ObjNum 23 snap 0 seq_num 23) 2024-03-19T11:30:55.061 INFO:tasks.rados.rados.0.smithi038.stdout:1143: read oid 11 snap -1 2024-03-19T11:30:55.061 INFO:tasks.rados.rados.0.smithi038.stdout:1143: expect (ObjNum 146 snap 15 seq_num 146) 2024-03-19T11:30:55.061 INFO:tasks.rados.rados.0.smithi038.stdout:1144: delete oid 18 current snap is 20 2024-03-19T11:30:55.068 INFO:tasks.rados.rados.0.smithi038.stdout:1142: expect (ObjNum 23 snap 0 seq_num 23) 2024-03-19T11:30:55.134 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:55.142+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:55.148 INFO:tasks.rados.rados.0.smithi038.stdout:1139: finishing set_chunk to oid 5 2024-03-19T11:30:55.148 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 5 v 99 (ObjNum 178 snap 19 seq_num 178) dirty exists 2024-03-19T11:30:55.148 INFO:tasks.rados.rados.0.smithi038.stdout:1140: expect (ObjNum 36 snap 0 seq_num 36) 2024-03-19T11:30:55.168 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:55.168 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:30:55.168 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:30:55.168 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:55.182 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-19T11:30:55.228 INFO:tasks.rados.rados.0.smithi038.stdout:1143: expect (ObjNum 146 snap 15 seq_num 146) 2024-03-19T11:30:55.236 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:55.243+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:55.337 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:55.346+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:55.348 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:55.348 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:30:55.348 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:30:55.348 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:30:55.348 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:55.363 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-19T11:30:55.452 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:55.462+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:55.538 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:55.538 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:30:55.539 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:30:55.539 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:30:55.539 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.539 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1019 4.6 4:626acd4d:::smithi03829856-2:head [read 601600~207872] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:55.539 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.760890+0000", 2024-03-19T11:30:55.539 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 42.785231056999997, 2024-03-19T11:30:55.539 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011505345, 2024-03-19T11:30:55.539 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:55.539 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:55.539 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:55.539 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:55.539 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:55.539 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1019 2024-03-19T11:30:55.539 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.540 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:55.540 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.540 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:55.540 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.760890+0000", 2024-03-19T11:30:55.540 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:55.540 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.540 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.540 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:55.540 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.760890+0000", 2024-03-19T11:30:55.540 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:55.540 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.540 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.540 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:55.540 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.760890+0000", 2024-03-19T11:30:55.540 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0600000000000001e-07 2024-03-19T11:30:55.541 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.541 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.541 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:55.541 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.760893+0000", 2024-03-19T11:30:55.541 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1669999999999999e-06 2024-03-19T11:30:55.541 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.541 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.541 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:55.541 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.760912+0000", 2024-03-19T11:30:55.541 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8359000000000002e-05 2024-03-19T11:30:55.541 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.541 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.541 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:55.541 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.760918+0000", 2024-03-19T11:30:55.542 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6019999999999996e-06 2024-03-19T11:30:55.542 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.542 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.542 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:55.542 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.760954+0000", 2024-03-19T11:30:55.542 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5262999999999998e-05 2024-03-19T11:30:55.542 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.542 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.542 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:30:55.542 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.760958+0000", 2024-03-19T11:30:55.542 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7690000000000004e-06 2024-03-19T11:30:55.542 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.543 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.543 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:55.543 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.772082+0000", 2024-03-19T11:30:55.543 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011123503999999999 2024-03-19T11:30:55.543 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.543 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.543 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:55.543 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.772204+0000", 2024-03-19T11:30:55.543 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012215899999999999 2024-03-19T11:30:55.543 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.543 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.543 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:55.543 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.772395+0000", 2024-03-19T11:30:55.543 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019111599999999999 2024-03-19T11:30:55.544 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:55.544 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:55.544 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:55.544 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.544 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.544 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1088 4.6 4:6d9093b1:::smithi03829856-38:head [read 0~3142876,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:55.544 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:14.006622+0000", 2024-03-19T11:30:55.544 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 41.539498481000003, 2024-03-19T11:30:55.544 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01172611, 2024-03-19T11:30:55.544 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:55.544 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:55.544 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:55.544 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:55.544 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:55.544 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1088 2024-03-19T11:30:55.545 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.545 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:55.545 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.545 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:55.545 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006622+0000", 2024-03-19T11:30:55.545 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:55.545 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.545 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.545 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:55.545 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006622+0000", 2024-03-19T11:30:55.545 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:55.545 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.545 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.545 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:55.546 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006623+0000", 2024-03-19T11:30:55.546 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4000000000000002e-07 2024-03-19T11:30:55.546 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.546 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.546 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:55.546 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006626+0000", 2024-03-19T11:30:55.546 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0340000000000001e-06 2024-03-19T11:30:55.546 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.546 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.546 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:55.546 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006632+0000", 2024-03-19T11:30:55.546 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3620000000000002e-06 2024-03-19T11:30:55.546 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.546 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.547 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:55.547 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006637+0000", 2024-03-19T11:30:55.547 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1390000000000003e-06 2024-03-19T11:30:55.547 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.547 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.547 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:55.547 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.015824+0000", 2024-03-19T11:30:55.547 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0091862899999999997 2024-03-19T11:30:55.547 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.547 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.547 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:55.547 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.015955+0000", 2024-03-19T11:30:55.547 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013139400000000001 2024-03-19T11:30:55.547 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.547 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.548 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:55.548 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.018349+0000", 2024-03-19T11:30:55.548 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002393451 2024-03-19T11:30:55.548 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:55.548 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:55.548 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:55.548 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.548 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.548 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1089 4.6 4:6d9093b1:::smithi03829856-38:head [sparse-read 0~1502675] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:55.548 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:14.006646+0000", 2024-03-19T11:30:55.548 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 41.539475037000003, 2024-03-19T11:30:55.548 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012196624999999999, 2024-03-19T11:30:55.548 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:55.548 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:55.549 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:55.549 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:55.549 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:55.549 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1089 2024-03-19T11:30:55.549 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.549 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:55.549 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.549 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:55.549 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006646+0000", 2024-03-19T11:30:55.549 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:55.549 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.549 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.549 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:55.549 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006646+0000", 2024-03-19T11:30:55.549 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:55.549 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.550 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.550 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:55.550 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006646+0000", 2024-03-19T11:30:55.550 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.9999999999999998e-07 2024-03-19T11:30:55.550 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.550 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.550 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:55.550 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006649+0000", 2024-03-19T11:30:55.550 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5639999999999999e-06 2024-03-19T11:30:55.550 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.550 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.550 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:55.550 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006656+0000", 2024-03-19T11:30:55.550 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1810000000000004e-06 2024-03-19T11:30:55.550 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.551 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.551 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:55.551 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006661+0000", 2024-03-19T11:30:55.551 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4999999999999999e-06 2024-03-19T11:30:55.551 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.551 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.551 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:55.551 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.018416+0000", 2024-03-19T11:30:55.551 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011754647999999999 2024-03-19T11:30:55.551 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.551 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.551 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:55.551 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.018528+0000", 2024-03-19T11:30:55.551 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000111872 2024-03-19T11:30:55.551 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.552 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.552 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:55.552 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.018842+0000", 2024-03-19T11:30:55.552 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00031446000000000002 2024-03-19T11:30:55.552 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:55.552 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:55.552 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:55.552 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.552 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.552 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1090 4.6 4:6d9093b1:::smithi03829856-38:head [sparse-read 0~1502675] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:55.552 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:14.006669+0000", 2024-03-19T11:30:55.552 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 41.539452116, 2024-03-19T11:30:55.552 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012590499999999999, 2024-03-19T11:30:55.552 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:55.553 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:55.553 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:55.553 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:55.553 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:55.553 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1090 2024-03-19T11:30:55.553 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.553 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:55.553 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.554 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:55.554 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006669+0000", 2024-03-19T11:30:55.554 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:55.554 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.554 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.554 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:55.554 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006669+0000", 2024-03-19T11:30:55.554 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:55.554 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.554 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.554 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:55.554 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006669+0000", 2024-03-19T11:30:55.554 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1800000000000001e-07 2024-03-19T11:30:55.554 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.555 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.555 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:55.555 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006673+0000", 2024-03-19T11:30:55.555 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.9530000000000001e-06 2024-03-19T11:30:55.555 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.555 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.555 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:55.555 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006680+0000", 2024-03-19T11:30:55.555 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9410000000000001e-06 2024-03-19T11:30:55.555 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.555 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.555 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:55.555 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006685+0000", 2024-03-19T11:30:55.555 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.327e-06 2024-03-19T11:30:55.555 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.556 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.556 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:55.556 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.018877+0000", 2024-03-19T11:30:55.556 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012191936 2024-03-19T11:30:55.556 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.556 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.557 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:55.557 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.018972+0000", 2024-03-19T11:30:55.557 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4754999999999997e-05 2024-03-19T11:30:55.557 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.557 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.557 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:55.557 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.019259+0000", 2024-03-19T11:30:55.557 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00028717 2024-03-19T11:30:55.557 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:55.557 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:55.557 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:55.557 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.557 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.558 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1217 4.3 4:c8b09e87:::smithi03829856-15:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:55.558 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.155966+0000", 2024-03-19T11:30:55.558 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 37.390154965000001, 2024-03-19T11:30:55.558 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.023984647000000001, 2024-03-19T11:30:55.558 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:55.558 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:55.558 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:55.558 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:55.558 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:55.558 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1217 2024-03-19T11:30:55.558 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.558 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:55.558 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.558 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:55.559 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155966+0000", 2024-03-19T11:30:55.559 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:55.559 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.559 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.559 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:55.559 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155966+0000", 2024-03-19T11:30:55.559 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:55.559 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.559 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.559 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:55.559 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155968+0000", 2024-03-19T11:30:55.559 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.795e-06 2024-03-19T11:30:55.559 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.559 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.559 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:55.560 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155978+0000", 2024-03-19T11:30:55.560 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0574000000000001e-05 2024-03-19T11:30:55.560 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.560 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.560 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:55.560 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.156000+0000", 2024-03-19T11:30:55.560 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1250000000000002e-05 2024-03-19T11:30:55.560 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.560 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.560 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:55.560 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.156010+0000", 2024-03-19T11:30:55.560 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0168e-05 2024-03-19T11:30:55.560 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.560 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.560 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:55.561 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.156119+0000", 2024-03-19T11:30:55.561 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010950699999999999 2024-03-19T11:30:55.561 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.561 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.561 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:55.561 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.156231+0000", 2024-03-19T11:30:55.561 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000111445 2024-03-19T11:30:55.561 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.561 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.561 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:55.561 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.179951+0000", 2024-03-19T11:30:55.561 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.023719908000000001 2024-03-19T11:30:55.561 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:55.561 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:55.561 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:55.562 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.562 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.562 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1432 4.13 4:cef44a0d:::smithi03829856-7:head [assert-version v18,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:55.562 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.977382+0000", 2024-03-19T11:30:55.562 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 29.568738693, 2024-03-19T11:30:55.562 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012855814, 2024-03-19T11:30:55.562 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:55.562 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:55.562 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:55.562 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:55.562 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:55.562 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1432 2024-03-19T11:30:55.562 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.563 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:55.563 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.563 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:55.563 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.977382+0000", 2024-03-19T11:30:55.563 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:55.563 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.563 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.563 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:55.563 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.977382+0000", 2024-03-19T11:30:55.563 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:55.563 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.563 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.563 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:55.563 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.977385+0000", 2024-03-19T11:30:55.563 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9380000000000001e-06 2024-03-19T11:30:55.564 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.564 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.564 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:55.564 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.977397+0000", 2024-03-19T11:30:55.564 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2136e-05 2024-03-19T11:30:55.564 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.564 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.564 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:55.564 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.977424+0000", 2024-03-19T11:30:55.564 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6917999999999999e-05 2024-03-19T11:30:55.564 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.564 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.564 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:55.564 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.977437+0000", 2024-03-19T11:30:55.564 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.242e-05 2024-03-19T11:30:55.565 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.565 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.565 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:55.565 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.977534+0000", 2024-03-19T11:30:55.565 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6974999999999999e-05 2024-03-19T11:30:55.565 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.565 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.565 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:55.565 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.977639+0000", 2024-03-19T11:30:55.565 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000105713 2024-03-19T11:30:55.565 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.565 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.565 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:55.565 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.987924+0000", 2024-03-19T11:30:55.566 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010284847999999999 2024-03-19T11:30:55.566 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.566 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.566 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:55.566 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.988536+0000", 2024-03-19T11:30:55.566 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00061207099999999995 2024-03-19T11:30:55.566 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.566 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.566 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:55.566 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.990098+0000", 2024-03-19T11:30:55.566 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001561523 2024-03-19T11:30:55.566 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.566 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.566 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:55.567 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.990176+0000", 2024-03-19T11:30:55.567 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8316e-05 2024-03-19T11:30:55.567 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.567 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.567 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:55.567 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.990238+0000", 2024-03-19T11:30:55.567 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1956000000000001e-05 2024-03-19T11:30:55.567 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:55.567 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:55.567 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:55.567 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.567 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.567 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1615 3.1 3:a1add8ca:::e7a0126b74590803e762ea9614dff8447a0272ff3000fd5c74e92cd9ee65bec5:head [write 0~2893824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-03-19T11:30:55.567 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:35.133510+0000", 2024-03-19T11:30:55.568 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 20.412610916999999, 2024-03-19T11:30:55.568 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012574642, 2024-03-19T11:30:55.568 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:55.568 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:55.568 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:55.568 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:55.568 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:55.568 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1615 2024-03-19T11:30:55.568 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.568 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:55.568 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.568 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:55.568 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.133510+0000", 2024-03-19T11:30:55.568 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:55.568 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.569 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.569 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:55.569 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.133510+0000", 2024-03-19T11:30:55.569 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:55.569 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.569 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.569 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:55.569 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.133514+0000", 2024-03-19T11:30:55.569 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6990000000000001e-06 2024-03-19T11:30:55.569 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.569 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.569 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:55.569 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.135072+0000", 2024-03-19T11:30:55.569 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0015580590000000001 2024-03-19T11:30:55.570 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.570 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.570 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:55.570 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.135107+0000", 2024-03-19T11:30:55.570 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5663000000000001e-05 2024-03-19T11:30:55.570 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.570 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.570 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:55.570 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.135122+0000", 2024-03-19T11:30:55.570 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4657e-05 2024-03-19T11:30:55.570 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.570 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.570 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:55.570 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.135211+0000", 2024-03-19T11:30:55.571 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9101e-05 2024-03-19T11:30:55.571 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.571 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.571 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-03-19T11:30:55.571 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.135222+0000", 2024-03-19T11:30:55.571 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0944999999999999e-05 2024-03-19T11:30:55.571 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.571 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.571 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:55.571 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.137108+0000", 2024-03-19T11:30:55.571 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0018854900000000001 2024-03-19T11:30:55.571 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.571 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.571 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:55.571 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.137204+0000", 2024-03-19T11:30:55.571 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6836999999999995e-05 2024-03-19T11:30:55.572 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.572 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.572 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3", 2024-03-19T11:30:55.572 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.137311+0000", 2024-03-19T11:30:55.572 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010635499999999999 2024-03-19T11:30:55.572 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.572 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.572 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:55.572 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.142840+0000", 2024-03-19T11:30:55.572 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055292010000000001 2024-03-19T11:30:55.572 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.572 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.572 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:55.572 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.145965+0000", 2024-03-19T11:30:55.572 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003125128 2024-03-19T11:30:55.573 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.573 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.573 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:55.573 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.146031+0000", 2024-03-19T11:30:55.573 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5529000000000002e-05 2024-03-19T11:30:55.573 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.573 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.573 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:55.573 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.146085+0000", 2024-03-19T11:30:55.573 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3977999999999997e-05 2024-03-19T11:30:55.573 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:55.573 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:55.573 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:55.573 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.573 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.574 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1675 4.13 4:c8b09e87:::smithi03829856-15:head [read 0~2235298,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e59)", 2024-03-19T11:30:55.574 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:36.381502+0000", 2024-03-19T11:30:55.574 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 19.164618490999999, 2024-03-19T11:30:55.574 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011105683, 2024-03-19T11:30:55.574 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:55.574 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:55.574 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:55.574 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:55.574 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:55.574 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1675 2024-03-19T11:30:55.574 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.574 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:55.574 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.574 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:55.574 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:36.381502+0000", 2024-03-19T11:30:55.575 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:55.575 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.575 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.575 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:55.575 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:36.381502+0000", 2024-03-19T11:30:55.575 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:55.575 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.575 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.575 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:55.575 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:36.381503+0000", 2024-03-19T11:30:55.575 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2100000000000002e-07 2024-03-19T11:30:55.575 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.575 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.575 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:55.575 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:36.381508+0000", 2024-03-19T11:30:55.576 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7759999999999997e-06 2024-03-19T11:30:55.576 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.576 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.576 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:55.576 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:36.381516+0000", 2024-03-19T11:30:55.576 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1310000000000006e-06 2024-03-19T11:30:55.576 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.576 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.576 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:55.576 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:36.381521+0000", 2024-03-19T11:30:55.576 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5620000000000003e-06 2024-03-19T11:30:55.576 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.576 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.576 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:55.577 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:36.388637+0000", 2024-03-19T11:30:55.577 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0071156270000000002 2024-03-19T11:30:55.577 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.577 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.577 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:55.577 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:36.389253+0000", 2024-03-19T11:30:55.577 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00061614500000000002 2024-03-19T11:30:55.577 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.577 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.577 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:55.577 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:36.392608+0000", 2024-03-19T11:30:55.577 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033550210000000001 2024-03-19T11:30:55.577 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:55.577 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:55.577 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:55.578 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.578 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.578 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1788 4.23 4:c4406a0a:::smithi03829856-49:head [write 685752~611865,stat] snapc 10=[10,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-03-19T11:30:55.578 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:40.165160+0000", 2024-03-19T11:30:55.578 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 15.380961037000001, 2024-03-19T11:30:55.578 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014138061, 2024-03-19T11:30:55.578 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:55.578 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:55.578 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:55.578 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:55.578 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:55.578 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1788 2024-03-19T11:30:55.578 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.579 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:55.579 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.579 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:55.579 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.165160+0000", 2024-03-19T11:30:55.579 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:55.579 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.579 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.579 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:55.579 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.165160+0000", 2024-03-19T11:30:55.579 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:55.579 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.579 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.579 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:55.579 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.165163+0000", 2024-03-19T11:30:55.579 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.729e-06 2024-03-19T11:30:55.580 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.580 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.580 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:55.580 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.165455+0000", 2024-03-19T11:30:55.580 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00029193900000000001 2024-03-19T11:30:55.580 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.580 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.580 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:55.580 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.165474+0000", 2024-03-19T11:30:55.580 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9910000000000001e-05 2024-03-19T11:30:55.580 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.580 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.580 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:55.580 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.165483+0000", 2024-03-19T11:30:55.580 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9609999999999994e-06 2024-03-19T11:30:55.581 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.581 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.581 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:55.581 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.165638+0000", 2024-03-19T11:30:55.581 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015434300000000001 2024-03-19T11:30:55.581 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.581 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.581 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:55.581 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.166078+0000", 2024-03-19T11:30:55.581 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044032099999999997 2024-03-19T11:30:55.581 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.581 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.581 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:55.581 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.166669+0000", 2024-03-19T11:30:55.582 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00059135100000000005 2024-03-19T11:30:55.582 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.582 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.582 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:55.582 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.174542+0000", 2024-03-19T11:30:55.582 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0078724880000000004 2024-03-19T11:30:55.582 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.582 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.582 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:55.582 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.178949+0000", 2024-03-19T11:30:55.582 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0044075720000000002 2024-03-19T11:30:55.582 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.582 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.582 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:55.583 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.179110+0000", 2024-03-19T11:30:55.583 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016055900000000001 2024-03-19T11:30:55.583 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.583 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.583 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:55.583 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.179298+0000", 2024-03-19T11:30:55.583 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00018788799999999999 2024-03-19T11:30:55.583 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:55.583 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:55.583 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:55.583 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.583 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.583 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1834 4.1b 4:db3790a7:::smithi03829856-32:head [tier-promote] snapc 10=[10,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-03-19T11:30:55.583 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:40.957892+0000", 2024-03-19T11:30:55.583 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 14.588229160999999, 2024-03-19T11:30:55.584 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013588032, 2024-03-19T11:30:55.584 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:55.584 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:55.584 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:55.584 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:55.584 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:55.584 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1834 2024-03-19T11:30:55.584 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.584 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:55.584 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.584 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:55.584 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.957892+0000", 2024-03-19T11:30:55.584 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:55.584 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.584 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.585 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:55.585 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.957892+0000", 2024-03-19T11:30:55.585 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:55.585 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.585 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.585 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:55.585 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.957894+0000", 2024-03-19T11:30:55.585 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2699999999999999e-06 2024-03-19T11:30:55.585 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.585 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.585 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:55.585 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.957903+0000", 2024-03-19T11:30:55.585 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5520000000000005e-06 2024-03-19T11:30:55.585 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.586 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.586 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:55.586 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.957924+0000", 2024-03-19T11:30:55.586 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1531000000000001e-05 2024-03-19T11:30:55.586 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.586 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.586 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:55.586 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.957934+0000", 2024-03-19T11:30:55.586 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.009e-05 2024-03-19T11:30:55.586 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.586 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.586 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:55.586 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.957987+0000", 2024-03-19T11:30:55.586 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2772000000000002e-05 2024-03-19T11:30:55.586 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.587 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.587 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:55.587 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.958044+0000", 2024-03-19T11:30:55.587 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6715000000000002e-05 2024-03-19T11:30:55.587 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.587 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.587 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:55.587 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.971480+0000", 2024-03-19T11:30:55.587 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013436102 2024-03-19T11:30:55.587 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:55.587 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:55.587 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:55.587 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.587 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.587 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1899 3.1 3:8a4cf997:::a32a1412677b7b8fb730a0c3225fa9b672680e6bcb3262b9965d5e0bcc3c046f:head [write 0~2557440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-03-19T11:30:55.588 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:41.740847+0000", 2024-03-19T11:30:55.588 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 13.805274035, 2024-03-19T11:30:55.588 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011051768999999999, 2024-03-19T11:30:55.588 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:55.588 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:55.588 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:55.588 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:55.588 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:55.588 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1899 2024-03-19T11:30:55.588 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.588 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:55.588 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.588 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:55.588 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.740847+0000", 2024-03-19T11:30:55.588 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:55.589 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.589 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.589 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:55.589 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.740847+0000", 2024-03-19T11:30:55.589 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:55.589 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.589 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.589 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:55.589 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.740851+0000", 2024-03-19T11:30:55.589 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7120000000000002e-06 2024-03-19T11:30:55.589 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.589 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.589 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:55.589 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.742923+0000", 2024-03-19T11:30:55.589 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00207223 2024-03-19T11:30:55.590 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.590 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.590 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:55.590 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.742958+0000", 2024-03-19T11:30:55.590 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4759999999999999e-05 2024-03-19T11:30:55.590 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.590 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.590 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:55.590 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.742976+0000", 2024-03-19T11:30:55.590 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8386e-05 2024-03-19T11:30:55.590 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.590 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.590 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:55.590 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.743175+0000", 2024-03-19T11:30:55.590 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019891 2024-03-19T11:30:55.591 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.591 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.591 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:55.591 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.743633+0000", 2024-03-19T11:30:55.591 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000458481 2024-03-19T11:30:55.591 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.591 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.591 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3", 2024-03-19T11:30:55.591 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.744083+0000", 2024-03-19T11:30:55.591 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044985100000000003 2024-03-19T11:30:55.591 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.591 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.591 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:55.591 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.751212+0000", 2024-03-19T11:30:55.591 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0071290590000000001 2024-03-19T11:30:55.592 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.592 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.592 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:55.592 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.751797+0000", 2024-03-19T11:30:55.592 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00058487999999999997 2024-03-19T11:30:55.592 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.592 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.592 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:55.592 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.751852+0000", 2024-03-19T11:30:55.592 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4580999999999998e-05 2024-03-19T11:30:55.592 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.592 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.592 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:55.592 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.751899+0000", 2024-03-19T11:30:55.592 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6919000000000003e-05 2024-03-19T11:30:55.593 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:55.593 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:55.593 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:55.593 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.593 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.593 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2047 4.1b 4:dccaff13:::smithi03829856-36:head [write 756348~784249] snapc 12=[12,11,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-03-19T11:30:55.593 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:46.278700+0000", 2024-03-19T11:30:55.593 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 9.2674213569999999, 2024-03-19T11:30:55.593 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014428785, 2024-03-19T11:30:55.593 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:55.593 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:55.593 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:55.593 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:55.593 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:55.593 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2047 2024-03-19T11:30:55.594 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.594 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:55.594 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.594 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:55.594 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.278700+0000", 2024-03-19T11:30:55.594 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:55.594 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.594 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.594 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:55.594 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.278700+0000", 2024-03-19T11:30:55.594 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:55.594 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.594 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.594 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:55.595 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.278703+0000", 2024-03-19T11:30:55.595 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1690000000000001e-06 2024-03-19T11:30:55.595 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.595 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.595 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:55.595 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.278955+0000", 2024-03-19T11:30:55.595 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025264100000000001 2024-03-19T11:30:55.595 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.595 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.595 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:55.595 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.278982+0000", 2024-03-19T11:30:55.595 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6933000000000001e-05 2024-03-19T11:30:55.595 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.595 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.595 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:55.595 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.278997+0000", 2024-03-19T11:30:55.596 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4440999999999999e-05 2024-03-19T11:30:55.596 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.596 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.596 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:55.596 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.279383+0000", 2024-03-19T11:30:55.596 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00038659700000000003 2024-03-19T11:30:55.596 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.596 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.596 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-03-19T11:30:55.596 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.279397+0000", 2024-03-19T11:30:55.596 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3574000000000001e-05 2024-03-19T11:30:55.596 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.596 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.596 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:55.597 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.282908+0000", 2024-03-19T11:30:55.597 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0035110110000000001 2024-03-19T11:30:55.597 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.597 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.597 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:55.597 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.283059+0000", 2024-03-19T11:30:55.597 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015160799999999999 2024-03-19T11:30:55.597 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.597 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.597 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:55.597 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.283194+0000", 2024-03-19T11:30:55.597 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013435399999999999 2024-03-19T11:30:55.597 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.597 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.597 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:55.597 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.292934+0000", 2024-03-19T11:30:55.598 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0097397539999999998 2024-03-19T11:30:55.598 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.598 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.598 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:55.598 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.293026+0000", 2024-03-19T11:30:55.598 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2195999999999996e-05 2024-03-19T11:30:55.598 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.598 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.598 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:55.598 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.293075+0000", 2024-03-19T11:30:55.598 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9484000000000003e-05 2024-03-19T11:30:55.598 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.598 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.598 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:55.599 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.293128+0000", 2024-03-19T11:30:55.599 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3022999999999997e-05 2024-03-19T11:30:55.599 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:55.599 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:55.599 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:55.599 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.599 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.599 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.24 75/74 [PushOp(4:27909373:::smithi03829856-46:8, version: 46'59, data_included: [756315~767970,1933572~280606], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:8@46'59, size: 2756315, copy_subset: [(0, 2756315)], clone_subset: {}, snapset: 10=[]:{8: [6], c: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2214178, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:266590a5:::smithi03829856-16:e, version: 70'107, data_included: [652105~662359,1728171~386217], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:e@70'107, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 10=[]:{e: [d, b, 6]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:266590a5:::smithi03829856-16:head, version: 70'108, data_included: [652105~662359,1728171~386217], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:head@70'108, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:27909373:::smithi03829856-46:c, version: 70'109, data_included: [628595~895690,1807440~152886], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:c@70'109, size: 3428595, copy_subset: [(0, 3428595)], clone_subset: {}, snapset: 10=[]:{8: [6], c: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1960326, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:27909373:::smithi03829856-46:head, version: 70'110, data_included: [403401~454961,1584925~506039], data_size: 961000, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:head@70'110, size: 2403401, copy_subset: [(0, 2403401)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2403401, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:30:55.599 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.099395+0000", 2024-03-19T11:30:55.599 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 1.4467258039999999, 2024-03-19T11:30:55.599 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.045391157000000001, 2024-03-19T11:30:55.599 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:55.599 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:55.599 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:55.599 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.599 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:55.600 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.099395+0000", 2024-03-19T11:30:55.600 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:55.600 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.600 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.600 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:55.600 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.099395+0000", 2024-03-19T11:30:55.600 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:55.600 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.600 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.600 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:55.600 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.099397+0000", 2024-03-19T11:30:55.600 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.691e-06 2024-03-19T11:30:55.600 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.600 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.601 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:55.601 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105009+0000", 2024-03-19T11:30:55.601 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0056121349999999999 2024-03-19T11:30:55.601 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.601 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.601 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:55.601 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105077+0000", 2024-03-19T11:30:55.601 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8561999999999998e-05 2024-03-19T11:30:55.601 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.601 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.601 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:55.601 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105133+0000", 2024-03-19T11:30:55.601 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5590000000000001e-05 2024-03-19T11:30:55.601 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.601 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.602 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:55.602 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105531+0000", 2024-03-19T11:30:55.602 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00039756200000000001 2024-03-19T11:30:55.602 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.602 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.602 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:55.602 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105579+0000", 2024-03-19T11:30:55.602 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8028999999999997e-05 2024-03-19T11:30:55.602 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.602 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.602 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:55.602 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.144786+0000", 2024-03-19T11:30:55.602 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.039207588000000002 2024-03-19T11:30:55.602 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:55.603 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:55.603 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:55.603 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.603 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.603 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.24 e75,74 4:266590a5:::smithi03829856-16:4,47'63,4:27909373:::smithi03829856-46:5,55'79,4:266590a5:::smithi03829856-16:10,67'105)", 2024-03-19T11:30:55.603 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.105191+0000", 2024-03-19T11:30:55.603 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 1.440929704, 2024-03-19T11:30:55.603 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.039580830999999997, 2024-03-19T11:30:55.603 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:55.603 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:55.603 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:55.603 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.603 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:55.603 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105191+0000", 2024-03-19T11:30:55.603 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:55.604 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.604 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.604 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:55.604 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105191+0000", 2024-03-19T11:30:55.604 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:55.604 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.604 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.604 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:55.604 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105192+0000", 2024-03-19T11:30:55.604 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.099e-06 2024-03-19T11:30:55.604 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.604 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.604 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:55.604 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105200+0000", 2024-03-19T11:30:55.605 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5279999999999998e-06 2024-03-19T11:30:55.605 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.605 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.605 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:55.605 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105211+0000", 2024-03-19T11:30:55.605 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1471000000000001e-05 2024-03-19T11:30:55.605 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.605 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.605 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:55.605 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105220+0000", 2024-03-19T11:30:55.605 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3469999999999999e-06 2024-03-19T11:30:55.605 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.605 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.605 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:55.605 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.144219+0000", 2024-03-19T11:30:55.606 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.038999024 2024-03-19T11:30:55.606 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.606 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.606 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:55.606 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.144257+0000", 2024-03-19T11:30:55.606 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8501999999999998e-05 2024-03-19T11:30:55.606 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.606 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.606 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:55.606 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.144772+0000", 2024-03-19T11:30:55.606 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00051486000000000001 2024-03-19T11:30:55.606 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:55.606 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:55.606 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:55.606 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.607 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.607 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 75/74 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:7, version: 55'82, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@55'82, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 58'86, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@58'86, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [504210~968657,1572695~79919], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:12, version: 66'101, data_included: [440350~816629,1309629~231947], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:12@66'101, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:10, version: 70'112, data_included: [440350~816629,1309629~231947], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:10@70'112, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 70'113, data_included: [440350~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@70'113, size: 2605085, copy_subset: [(0, 2605085)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1488926, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:30:55.607 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.107266+0000", 2024-03-19T11:30:55.607 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 1.4388551869999999, 2024-03-19T11:30:55.607 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.048592669999999998, 2024-03-19T11:30:55.607 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:55.607 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:55.607 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:55.607 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.607 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:55.607 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.107266+0000", 2024-03-19T11:30:55.607 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:55.607 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.607 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.608 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:55.608 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.107266+0000", 2024-03-19T11:30:55.608 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:55.608 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.608 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.608 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:55.608 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.107267+0000", 2024-03-19T11:30:55.608 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1459999999999999e-06 2024-03-19T11:30:55.608 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.608 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.608 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:55.608 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.122861+0000", 2024-03-19T11:30:55.608 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01559372 2024-03-19T11:30:55.608 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.608 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.609 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:55.609 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.122932+0000", 2024-03-19T11:30:55.609 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1477999999999996e-05 2024-03-19T11:30:55.609 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.609 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.609 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:55.609 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.122999+0000", 2024-03-19T11:30:55.609 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6832e-05 2024-03-19T11:30:55.609 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.609 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.609 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:55.609 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123332+0000", 2024-03-19T11:30:55.609 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000333271 2024-03-19T11:30:55.609 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.609 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.609 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:55.610 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123371+0000", 2024-03-19T11:30:55.610 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8525999999999997e-05 2024-03-19T11:30:55.610 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.610 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.610 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:55.610 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.155858+0000", 2024-03-19T11:30:55.610 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.032487697000000003 2024-03-19T11:30:55.610 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:55.610 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:55.610 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:55.610 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.610 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.610 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.21 e75,74 4:8401c1f0:::smithi03829856-43:8,46'66,4:866f3dcc:::smithi03829856-31:2,47'73)", 2024-03-19T11:30:55.610 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.123075+0000", 2024-03-19T11:30:55.611 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 1.423045968, 2024-03-19T11:30:55.611 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.033023045000000001, 2024-03-19T11:30:55.611 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:55.611 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:55.611 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:55.611 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.611 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:55.611 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123075+0000", 2024-03-19T11:30:55.611 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:55.611 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.611 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.611 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:55.611 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123075+0000", 2024-03-19T11:30:55.611 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:55.611 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.611 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.612 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:55.612 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123076+0000", 2024-03-19T11:30:55.612 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5e-06 2024-03-19T11:30:55.612 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.612 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.612 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:55.612 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123086+0000", 2024-03-19T11:30:55.612 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8060000000000006e-06 2024-03-19T11:30:55.612 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.612 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.612 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:55.612 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123099+0000", 2024-03-19T11:30:55.612 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3251000000000001e-05 2024-03-19T11:30:55.612 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.612 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.613 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:55.613 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123112+0000", 2024-03-19T11:30:55.613 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2774e-05 2024-03-19T11:30:55.613 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.613 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.613 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:55.613 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.155902+0000", 2024-03-19T11:30:55.613 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.032790225999999999 2024-03-19T11:30:55.613 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.613 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.613 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:55.613 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.155914+0000", 2024-03-19T11:30:55.613 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1476e-05 2024-03-19T11:30:55.613 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.613 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.614 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:55.614 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.156098+0000", 2024-03-19T11:30:55.614 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000184012 2024-03-19T11:30:55.614 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:55.614 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:55.614 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:55.614 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.614 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.614 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.24 75/74 [PushOp(4:27909373:::smithi03829856-46:8, version: 46'59, data_included: [2214178~511064], data_size: 511064, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:8@46'59, size: 2756315, copy_subset: [(0, 2756315)], clone_subset: {}, snapset: 10=[]:{8: [6], c: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2756315, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2214178, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:266590a5:::smithi03829856-16:e, version: 70'107, data_included: [2114388~668125], data_size: 668125, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:e@70'107, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 10=[]:{e: [d, b, 6]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3092455, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:266590a5:::smithi03829856-16:head, version: 70'108, data_included: [2114388~668125], data_size: 668125, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:head@70'108, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3092455, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:27909373:::smithi03829856-46:c, version: 70'109, data_included: [1960326~764916,2934719~283660], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:c@70'109, size: 3428595, copy_subset: [(0, 3428595)], clone_subset: {}, snapset: 10=[]:{8: [6], c: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3218379, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1960326, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:30:55.614 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.153500+0000", 2024-03-19T11:30:55.614 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 1.3926206569999999, 2024-03-19T11:30:55.614 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012221095, 2024-03-19T11:30:55.614 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:55.614 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:55.615 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:55.615 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.615 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:55.615 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.153500+0000", 2024-03-19T11:30:55.615 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:55.615 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.615 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.615 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:55.615 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.153500+0000", 2024-03-19T11:30:55.615 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:55.615 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.615 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.615 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:55.615 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.153503+0000", 2024-03-19T11:30:55.615 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6000000000000001e-06 2024-03-19T11:30:55.616 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.616 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.616 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:55.616 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.155793+0000", 2024-03-19T11:30:55.616 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022901359999999999 2024-03-19T11:30:55.616 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.616 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.616 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:55.616 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.155868+0000", 2024-03-19T11:30:55.616 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5005000000000004e-05 2024-03-19T11:30:55.616 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.616 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.616 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:55.616 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.155927+0000", 2024-03-19T11:30:55.616 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9413000000000002e-05 2024-03-19T11:30:55.617 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.617 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.617 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:55.617 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.156131+0000", 2024-03-19T11:30:55.617 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020398599999999999 2024-03-19T11:30:55.617 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.617 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.617 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:55.617 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.156160+0000", 2024-03-19T11:30:55.617 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8713000000000001e-05 2024-03-19T11:30:55.617 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.617 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.617 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:55.617 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.165721+0000", 2024-03-19T11:30:55.617 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0095612419999999993 2024-03-19T11:30:55.618 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:55.618 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:55.618 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:55.618 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.618 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.618 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 75/74 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:7, version: 55'82, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@55'82, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 58'86, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@58'86, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [1652614~973634,2888511~52681,3182305~22261], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3204566, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:12, version: 66'101, data_included: [1541576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:12@66'101, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:10, version: 70'112, data_included: [1541576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:10@70'112, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 70'113, data_included: [1488926~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@70'113, size: 2605085, copy_subset: [(0, 2605085)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2537502, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1488926, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:30:55.618 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.174030+0000", 2024-03-19T11:30:55.618 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 1.372090764, 2024-03-19T11:30:55.618 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.040799275000000003, 2024-03-19T11:30:55.618 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:55.618 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:55.618 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:55.618 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.618 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:55.619 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.174030+0000", 2024-03-19T11:30:55.619 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:55.619 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.619 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.619 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:55.619 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.174030+0000", 2024-03-19T11:30:55.619 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:55.619 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.619 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.619 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:55.619 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.174031+0000", 2024-03-19T11:30:55.619 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.322e-06 2024-03-19T11:30:55.619 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.619 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.619 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:55.620 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.184771+0000", 2024-03-19T11:30:55.620 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010739227 2024-03-19T11:30:55.620 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.620 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.620 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:55.620 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.184872+0000", 2024-03-19T11:30:55.620 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000101234 2024-03-19T11:30:55.620 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.620 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.620 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:55.620 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.184985+0000", 2024-03-19T11:30:55.620 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000113545 2024-03-19T11:30:55.620 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.620 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.620 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:55.621 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.185463+0000", 2024-03-19T11:30:55.621 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00047749899999999999 2024-03-19T11:30:55.621 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.621 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.621 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:55.621 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.185534+0000", 2024-03-19T11:30:55.621 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1007999999999995e-05 2024-03-19T11:30:55.621 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.621 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.621 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:55.621 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.214829+0000", 2024-03-19T11:30:55.621 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029295439999999999 2024-03-19T11:30:55.621 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:55.621 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:55.621 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:55.622 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.622 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.622 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.21 e75,74 4:866f3dcc:::smithi03829856-31:c,70'114)", 2024-03-19T11:30:55.622 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.197457+0000", 2024-03-19T11:30:55.622 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 1.3486637180000001, 2024-03-19T11:30:55.622 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018493590000000001, 2024-03-19T11:30:55.622 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:55.622 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:55.622 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:55.622 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.622 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:55.622 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.197457+0000", 2024-03-19T11:30:55.622 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:55.622 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.622 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.623 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:55.623 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.197457+0000", 2024-03-19T11:30:55.623 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:55.623 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.623 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.623 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:55.623 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.197459+0000", 2024-03-19T11:30:55.623 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3199999999999998e-06 2024-03-19T11:30:55.623 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.623 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.623 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:55.623 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.197480+0000", 2024-03-19T11:30:55.623 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0366000000000001e-05 2024-03-19T11:30:55.623 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.623 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.624 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:55.624 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.197521+0000", 2024-03-19T11:30:55.624 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0738000000000002e-05 2024-03-19T11:30:55.624 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.624 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.624 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:55.624 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.197547+0000", 2024-03-19T11:30:55.624 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6548999999999999e-05 2024-03-19T11:30:55.624 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.624 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.624 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:55.624 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.214954+0000", 2024-03-19T11:30:55.624 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01740713 2024-03-19T11:30:55.624 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.624 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.625 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:55.625 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.214972+0000", 2024-03-19T11:30:55.625 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7520000000000002e-05 2024-03-19T11:30:55.625 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.625 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.625 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:55.625 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.215951+0000", 2024-03-19T11:30:55.625 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00097896700000000007 2024-03-19T11:30:55.625 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:55.625 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:55.625 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:55.625 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.625 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.625 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 75/74 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [2599014~241336], data_size: 241336, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2840350, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [2599014~241336,2948041~446940], data_size: 688276, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3394981, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [3204566~621403], data_size: 621403, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3865577, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3204566, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:12, version: 66'101, data_included: [2590152~250198,2948041~581396], data_size: 831594, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:12@66'101, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3529437, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:10, version: 70'112, data_included: [2590152~250198,2948041~581396], data_size: 831594, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:10@70'112, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3529437, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 70'113, data_included: [2537502~67583], data_size: 67583, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@70'113, size: 2605085, copy_subset: [(0, 2605085)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2605085, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2537502, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:30:55.626 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.222872+0000", 2024-03-19T11:30:55.626 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 1.3232492709999999, 2024-03-19T11:30:55.626 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016277658, 2024-03-19T11:30:55.626 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:55.626 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:55.626 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:55.626 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.626 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:55.626 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.222872+0000", 2024-03-19T11:30:55.626 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:55.626 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.626 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.626 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:55.626 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.222872+0000", 2024-03-19T11:30:55.627 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:55.627 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.627 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.627 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:55.627 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.222874+0000", 2024-03-19T11:30:55.627 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9999999999999999e-06 2024-03-19T11:30:55.627 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.627 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.627 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:55.627 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.227563+0000", 2024-03-19T11:30:55.627 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046897570000000001 2024-03-19T11:30:55.627 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.627 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.627 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:55.628 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.227639+0000", 2024-03-19T11:30:55.628 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5607999999999998e-05 2024-03-19T11:30:55.628 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.628 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.628 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:55.628 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.227736+0000", 2024-03-19T11:30:55.628 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6854000000000005e-05 2024-03-19T11:30:55.628 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.628 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.628 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:55.628 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.228149+0000", 2024-03-19T11:30:55.628 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00041330000000000002 2024-03-19T11:30:55.628 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.628 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.629 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:55.629 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.228219+0000", 2024-03-19T11:30:55.629 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.0140000000000003e-05 2024-03-19T11:30:55.629 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:55.629 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:55.629 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:55.629 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.239149+0000", 2024-03-19T11:30:55.629 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010929998999999999 2024-03-19T11:30:55.629 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:55.629 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:55.629 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:55.629 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:55.629 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:55.629 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:55.630 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-19T11:30:55.631 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:55.543+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:55.632 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-19T11:30:55.678 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:55.686+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:55.739 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:55.751+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:55.799 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:55.799 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:30:55.799 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:30:55.799 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:55.815 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-19T11:30:55.835 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:30:55.835 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:30:55.840 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:55.850+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:55.852 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:30:55.852 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:30:55.870 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:30:55.871 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:30:55.887 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:30:55.888 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:30:55.909 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:30:55.909 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:30:55.930 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:30:55.930 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:30:55.932 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:55.932 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:55.942 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:55.947+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:55.947 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-19T11:30:55.951 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:30:55.952 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:30:55.973 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:30:55.973 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:30:56.043 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:56.050+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:56.065 INFO:tasks.rados.rados.0.smithi038.stdout:1139: done (6 left) 2024-03-19T11:30:56.065 INFO:tasks.rados.rados.0.smithi038.stdout:1140: done (5 left) 2024-03-19T11:30:56.065 INFO:tasks.rados.rados.0.smithi038.stdout:1141: done (4 left) 2024-03-19T11:30:56.066 INFO:tasks.rados.rados.0.smithi038.stdout:1142: done (3 left) 2024-03-19T11:30:56.066 INFO:tasks.rados.rados.0.smithi038.stdout:1143: done (2 left) 2024-03-19T11:30:56.066 INFO:tasks.rados.rados.0.smithi038.stdout:1144: done (1 left) 2024-03-19T11:30:56.066 INFO:tasks.rados.rados.0.smithi038.stdout:1145: write oid 17 current snap is 20 2024-03-19T11:30:56.066 INFO:tasks.rados.rados.0.smithi038.stdout:1145: seq_num 183 ranges {675681=563039,1858793=746713,3025937=49744} 2024-03-19T11:30:56.068 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-03-19T11:30:56.084 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:56.084 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:56.086 INFO:tasks.rados.rados.0.smithi038.stdout:1145: writing smithi03829856-17 from 675681 to 1238720 tid 1 2024-03-19T11:30:56.100 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-19T11:30:56.107 INFO:tasks.rados.rados.0.smithi038.stdout:1145: writing smithi03829856-17 from 1858793 to 2605506 tid 2 2024-03-19T11:30:56.113 INFO:tasks.rados.rados.0.smithi038.stdout:1145: writing smithi03829856-17 from 3025937 to 3075681 tid 3 2024-03-19T11:30:56.113 INFO:tasks.rados.rados.0.smithi038.stdout:1146: set_chunk oid 45 target oid 2024-03-19T11:30:56.123 INFO:tasks.rados.rados.0.smithi038.stdout:1146: set_chunk oid 45 offset: 2207744 length: 386048 target oid 564a5ca51e8eabe8c1157c837c26572f9e6a808e83c0f3fd6a1f49fc2b66cac0 offset: 0 snap 9 2024-03-19T11:30:56.123 INFO:tasks.rados.rados.0.smithi038.stdout:1147: rollback oid 1 current snap is 20 2024-03-19T11:30:56.123 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 1 to 9 2024-03-19T11:30:56.123 INFO:tasks.rados.rados.0.smithi038.stdout:1138: expect (ObjNum 143 snap 15 seq_num 143) 2024-03-19T11:30:56.144 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:56.151+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:56.221 INFO:tasks.rados.rados.0.smithi038.stdout:1145: finishing write tid 1 to smithi03829856-17 2024-03-19T11:30:56.221 INFO:tasks.rados.rados.0.smithi038.stdout:1145: finishing write tid 2 to smithi03829856-17 2024-03-19T11:30:56.221 INFO:tasks.rados.rados.0.smithi038.stdout:1145: finishing write tid 3 to smithi03829856-17 2024-03-19T11:30:56.221 INFO:tasks.rados.rados.0.smithi038.stdout:1145: finishing write tid 4 to smithi03829856-17 2024-03-19T11:30:56.221 INFO:tasks.rados.rados.0.smithi038.stdout:1145: finishing write tid 5 to smithi03829856-17 2024-03-19T11:30:56.221 INFO:tasks.rados.rados.0.smithi038.stdout:1145: oid 17 updating version 0 to 133 2024-03-19T11:30:56.221 INFO:tasks.rados.rados.0.smithi038.stdout:1145: oid 17 updating version 133 to 134 2024-03-19T11:30:56.221 INFO:tasks.rados.rados.0.smithi038.stdout:1145: oid 17 updating version 134 to 135 2024-03-19T11:30:56.221 INFO:tasks.rados.rados.0.smithi038.stdout:1145: oid 17 updating version 135 to 136 2024-03-19T11:30:56.221 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 17 v 136 (ObjNum 183 snap 20 seq_num 183) dirty exists 2024-03-19T11:30:56.221 INFO:tasks.rados.rados.0.smithi038.stdout:1145: left oid 17 (ObjNum 183 snap 20 seq_num 183) 2024-03-19T11:30:56.221 INFO:tasks.rados.rados.0.smithi038.stdout:1146: finishing set_chunk to oid 45 2024-03-19T11:30:56.222 INFO:tasks.rados.rados.0.smithi038.stdout:1147: finishing rollback tid 0 to smithi03829856-1 2024-03-19T11:30:56.222 INFO:tasks.rados.rados.0.smithi038.stdout:1138: done (3 left) 2024-03-19T11:30:56.222 INFO:tasks.rados.rados.0.smithi038.stdout:1145: done (2 left) 2024-03-19T11:30:56.222 INFO:tasks.rados.rados.0.smithi038.stdout:1146: done (1 left) 2024-03-19T11:30:56.222 INFO:tasks.rados.rados.0.smithi038.stdout:1148: tier_promote oid 38 2024-03-19T11:30:56.222 INFO:tasks.rados.rados.0.smithi038.stdout:1147: finishing rollback tid 1 to smithi03829856-1 2024-03-19T11:30:56.222 INFO:tasks.rados.rados.0.smithi038.stdout:1147: finishing rollback tid 2 to smithi03829856-1 2024-03-19T11:30:56.222 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 1 v 113 (ObjNum 51 snap 0 seq_num 51) dirty exists 2024-03-19T11:30:56.222 INFO:tasks.rados.rados.0.smithi038.stdout:1147: done (1 left) 2024-03-19T11:30:56.222 INFO:tasks.rados.rados.0.smithi038.stdout:1149: tier_evict oid 41 2024-03-19T11:30:56.222 INFO:tasks.rados.rados.0.smithi038.stdout:1149: tier_evict oid 41 snap -1 2024-03-19T11:30:56.222 INFO:tasks.rados.rados.0.smithi038.stdout:1150: set_chunk oid 50 target oid 2024-03-19T11:30:56.230 INFO:tasks.rados.rados.0.smithi038.stdout:1150: set_chunk oid 50 offset: 852992 length: 927232 target oid 7c49bfec6ae52ee5f0fd28d6881f3c3a3fe13dd0ad88761165ae1cc8ef34d02e offset: 0 snap -1 2024-03-19T11:30:56.230 INFO:tasks.rados.rados.0.smithi038.stdout:1151: write (excl) oid 27 current snap is 20 2024-03-19T11:30:56.231 INFO:tasks.rados.rados.0.smithi038.stdout:1151: seq_num 184 ranges {692488=638288,1917275=606724} 2024-03-19T11:30:56.238 INFO:tasks.rados.rados.0.smithi038.stdout:1151: writing smithi03829856-27 from 692488 to 1330776 tid 1 2024-03-19T11:30:56.245 INFO:tasks.rados.rados.0.smithi038.stdout:1151: writing smithi03829856-27 from 1917275 to 2523999 tid 2 2024-03-19T11:30:56.245 INFO:tasks.rados.rados.0.smithi038.stdout:1152: write oid 2 current snap is 20 2024-03-19T11:30:56.245 INFO:tasks.rados.rados.0.smithi038.stdout:1149: got (22) Invalid argument 2024-03-19T11:30:56.245 INFO:tasks.rados.rados.0.smithi038.stdout:1152: seq_num 185 ranges {709295=713537,1859404=750382} 2024-03-19T11:30:56.245 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:56.254+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:56.246 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:56.246 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:56.253 INFO:tasks.rados.rados.0.smithi038.stdout:1152: writing smithi03829856-2 from 709295 to 1422832 tid 1 2024-03-19T11:30:56.260 INFO:tasks.rados.rados.0.smithi038.stdout:1152: writing smithi03829856-2 from 1859404 to 2609786 tid 2 2024-03-19T11:30:56.260 INFO:tasks.rados.rados.0.smithi038.stdout:1149: done (4 left) 2024-03-19T11:30:56.260 INFO:tasks.rados.rados.0.smithi038.stdout:1153: set_chunk oid 46 target oid 2024-03-19T11:30:56.264 INFO:tasks.rados.rados.0.smithi038.stdout:1153: set_chunk oid 46 offset: 2172928 length: 62976 target oid bc9ed4858a12f29b83d80015f097d9340288adef71be2ca35d9dd02779c9812f offset: 0 snap -1 2024-03-19T11:30:56.264 INFO:tasks.rados.rados.0.smithi038.stdout:1154: tier_promote oid 24 2024-03-19T11:30:56.264 INFO:tasks.rados.rados.0.smithi038.stdout:1148: got (0) Success 2024-03-19T11:30:56.264 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 38 v 64 (ObjNum 125 snap 15 seq_num 125) dirty exists 2024-03-19T11:30:56.264 INFO:tasks.rados.rados.0.smithi038.stdout:1150: finishing set_chunk to oid 50 2024-03-19T11:30:56.264 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 50 v 81 (ObjNum 114 snap 11 seq_num 114) dirty exists 2024-03-19T11:30:56.264 INFO:tasks.rados.rados.0.smithi038.stdout:1151: finishing write tid 1 to smithi03829856-27 2024-03-19T11:30:56.264 INFO:tasks.rados.rados.0.smithi038.stdout:1148: done (5 left) 2024-03-19T11:30:56.264 INFO:tasks.rados.rados.0.smithi038.stdout:1150: done (4 left) 2024-03-19T11:30:56.265 INFO:tasks.rados.rados.0.smithi038.stdout:1155: set_chunk oid 40 target oid 2024-03-19T11:30:56.265 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-19T11:30:56.274 INFO:tasks.rados.rados.0.smithi038.stdout:1155: set_chunk oid 40 offset: 1056256 length: 533504 target oid c069fc6dd4fd81505b6b1ff60aadb3319c03a91648a9618b8a38633244414540 offset: 0 snap -1 2024-03-19T11:30:56.274 INFO:tasks.rados.rados.0.smithi038.stdout:1156: read oid 14 snap -1 2024-03-19T11:30:56.274 INFO:tasks.rados.rados.0.smithi038.stdout:1156: expect (ObjNum 88 snap 8 seq_num 88) 2024-03-19T11:30:56.274 INFO:tasks.rados.rados.0.smithi038.stdout:1157: read oid 19 snap -1 2024-03-19T11:30:56.274 INFO:tasks.rados.rados.0.smithi038.stdout:1157: expect deleted 2024-03-19T11:30:56.274 INFO:tasks.rados.rados.0.smithi038.stdout:1158: delete oid 43 current snap is 20 2024-03-19T11:30:56.274 INFO:tasks.rados.rados.0.smithi038.stdout:1151: finishing write tid 2 to smithi03829856-27 2024-03-19T11:30:56.275 INFO:tasks.rados.rados.0.smithi038.stdout:1151: finishing write tid 3 to smithi03829856-27 2024-03-19T11:30:56.275 INFO:tasks.rados.rados.0.smithi038.stdout:1151: finishing write tid 4 to smithi03829856-27 2024-03-19T11:30:56.275 INFO:tasks.rados.rados.0.smithi038.stdout:1151: oid 27 updating version 0 to 90 2024-03-19T11:30:56.275 INFO:tasks.rados.rados.0.smithi038.stdout:1151: oid 27 version 90 is already newer than 88 2024-03-19T11:30:56.275 INFO:tasks.rados.rados.0.smithi038.stdout:1151: oid 27 version 90 is already newer than 89 2024-03-19T11:30:56.275 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 27 v 90 (ObjNum 184 snap 20 seq_num 184) dirty exists 2024-03-19T11:30:56.275 INFO:tasks.rados.rados.0.smithi038.stdout:1151: left oid 27 (ObjNum 184 snap 20 seq_num 184) 2024-03-19T11:30:56.275 INFO:tasks.rados.rados.0.smithi038.stdout:1152: finishing write tid 1 to smithi03829856-2 2024-03-19T11:30:56.275 INFO:tasks.rados.rados.0.smithi038.stdout:1152: finishing write tid 2 to smithi03829856-2 2024-03-19T11:30:56.275 INFO:tasks.rados.rados.0.smithi038.stdout:1152: finishing write tid 3 to smithi03829856-2 2024-03-19T11:30:56.275 INFO:tasks.rados.rados.0.smithi038.stdout:1152: finishing write tid 4 to smithi03829856-2 2024-03-19T11:30:56.275 INFO:tasks.rados.rados.0.smithi038.stdout:1152: oid 2 updating version 0 to 66 2024-03-19T11:30:56.275 INFO:tasks.rados.rados.0.smithi038.stdout:1152: oid 2 updating version 66 to 68 2024-03-19T11:30:56.275 INFO:tasks.rados.rados.0.smithi038.stdout:1152: oid 2 version 68 is already newer than 67 2024-03-19T11:30:56.275 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 2 v 68 (ObjNum 185 snap 20 seq_num 185) dirty exists 2024-03-19T11:30:56.275 INFO:tasks.rados.rados.0.smithi038.stdout:1152: left oid 2 (ObjNum 185 snap 20 seq_num 185) 2024-03-19T11:30:56.276 INFO:tasks.rados.rados.0.smithi038.stdout:1153: finishing set_chunk to oid 46 2024-03-19T11:30:56.276 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 46 v 96 (ObjNum 143 snap 15 seq_num 143) dirty exists 2024-03-19T11:30:56.277 INFO:tasks.rados.rados.0.smithi038.stdout:1151: done (7 left) 2024-03-19T11:30:56.277 INFO:tasks.rados.rados.0.smithi038.stdout:1152: done (6 left) 2024-03-19T11:30:56.277 INFO:tasks.rados.rados.0.smithi038.stdout:1153: done (5 left) 2024-03-19T11:30:56.277 INFO:tasks.rados.rados.0.smithi038.stdout:1157: done (4 left) 2024-03-19T11:30:56.277 INFO:tasks.rados.rados.0.smithi038.stdout:1158: done (3 left) 2024-03-19T11:30:56.277 INFO:tasks.rados.rados.0.smithi038.stdout:1159: snap_create 2024-03-19T11:30:56.278 INFO:tasks.rados.rados.0.smithi038.stdout:1154: got (0) Success 2024-03-19T11:30:56.278 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 24 v 9 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-03-19T11:30:56.279 INFO:tasks.rados.rados.0.smithi038.stdout:1155: finishing set_chunk to oid 40 2024-03-19T11:30:56.279 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 40 v 61 (ObjNum 141 snap 15 seq_num 141) dirty exists 2024-03-19T11:30:56.290 INFO:tasks.rados.rados.0.smithi038.stdout:1156: expect (ObjNum 88 snap 8 seq_num 88) 2024-03-19T11:30:56.346 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:56.351+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:56.446 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:56.446 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:56.447 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:56.454+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:56.464 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-19T11:30:56.548 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:56.555+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:56.631 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:56.631 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:56.646 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-19T11:30:56.648 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:56.659+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:56.750 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:56.758+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:56.774 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:56.775 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:56.789 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-19T11:30:56.876 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:56.886+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:56.951 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:56.962+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:56.960 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:56.960 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:56.974 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-19T11:30:57.044 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-03-19T11:30:57.045 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:57.054+0000 7f6712f92640 -1 osd.2 77 handle_osd_map injecting map crc failure 2024-03-19T11:30:57.048 INFO:tasks.rados.rados.0.smithi038.stdout:1154: done (3 left) 2024-03-19T11:30:57.048 INFO:tasks.rados.rados.0.smithi038.stdout:1155: done (2 left) 2024-03-19T11:30:57.049 INFO:tasks.rados.rados.0.smithi038.stdout:1156: done (1 left) 2024-03-19T11:30:57.049 INFO:tasks.rados.rados.0.smithi038.stdout:1159: done (0 left) 2024-03-19T11:30:57.049 INFO:tasks.rados.rados.0.smithi038.stdout:1160: set_chunk oid 24 target oid 2024-03-19T11:30:57.052 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:57.059+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:57.065 INFO:tasks.rados.rados.0.smithi038.stdout:1160: set_chunk oid 24 offset: 293376 length: 1404416 target oid 841fa9538d3c04e5c7e6327aec67f9a5b8242d1787b4450a51f4874cc68cbf1d offset: 0 snap -1 2024-03-19T11:30:57.065 INFO:tasks.rados.rados.0.smithi038.stdout:1161: rollback oid 18 current snap is 21 2024-03-19T11:30:57.065 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 18 to 11 2024-03-19T11:30:57.066 INFO:tasks.rados.rados.0.smithi038.stdout:1162: write oid 11 current snap is 21 2024-03-19T11:30:57.066 INFO:tasks.rados.rados.0.smithi038.stdout:1162: seq_num 187 ranges {742909=464035,1976368=754051} 2024-03-19T11:30:57.071 INFO:tasks.rados.rados.0.smithi038.stdout:1162: writing smithi03829856-11 from 742909 to 1206944 tid 1 2024-03-19T11:30:57.081 INFO:tasks.rados.rados.0.smithi038.stdout:1162: writing smithi03829856-11 from 1976368 to 2730419 tid 2 2024-03-19T11:30:57.081 INFO:tasks.rados.rados.0.smithi038.stdout:1163: write (excl) oid 31 current snap is 21 2024-03-19T11:30:57.081 INFO:tasks.rados.rados.0.smithi038.stdout:1160: finishing set_chunk to oid 24 2024-03-19T11:30:57.081 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 24 set_chunk returned error code -95 offset: 293376 length: 1404416 2024-03-19T11:30:57.081 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-03-19T11:30:57.081 INFO:tasks.rados.rados.0.smithi038.stdout:1163: seq_num 188 ranges {759716=539284,2034850=614062} 2024-03-19T11:30:57.088 INFO:tasks.rados.rados.0.smithi038.stdout:1163: writing smithi03829856-31 from 759716 to 1299000 tid 1 2024-03-19T11:30:57.096 INFO:tasks.rados.rados.0.smithi038.stdout:1163: writing smithi03829856-31 from 2034850 to 2648912 tid 2 2024-03-19T11:30:57.097 INFO:tasks.rados.rados.0.smithi038.stdout:1160: done (3 left) 2024-03-19T11:30:57.097 INFO:tasks.rados.rados.0.smithi038.stdout:1164: read oid 17 snap 20 2024-03-19T11:30:57.097 INFO:tasks.rados.rados.0.smithi038.stdout:1164: expect (ObjNum 183 snap 20 seq_num 183) 2024-03-19T11:30:57.097 INFO:tasks.rados.rados.0.smithi038.stdout:1161: finishing rollback tid 1 to smithi03829856-18 2024-03-19T11:30:57.097 INFO:tasks.rados.rados.0.smithi038.stdout:1161: finishing rollback tid 2 to smithi03829856-18 2024-03-19T11:30:57.097 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 18 v 115 (ObjNum 106 snap 11 seq_num 106) dirty exists 2024-03-19T11:30:57.097 INFO:tasks.rados.rados.0.smithi038.stdout:1161: done (3 left) 2024-03-19T11:30:57.097 INFO:tasks.rados.rados.0.smithi038.stdout:1165: set_chunk oid 37 target oid 2024-03-19T11:30:57.111 INFO:tasks.rados.rados.0.smithi038.stdout:1165: set_chunk oid 37 offset: 1510912 length: 810496 target oid ea294dbc18e91a936d2b878d26aa488c6ce70dcd16ed17bfa1eb1a816cd96cd7 offset: 0 snap -1 2024-03-19T11:30:57.111 INFO:tasks.rados.rados.0.smithi038.stdout:1166: snap_remove snap 15 2024-03-19T11:30:57.151 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:57.151 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:57.152 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:57.159+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:57.165 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-19T11:30:57.253 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:57.259+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:57.297 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:57.297 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:57.311 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-19T11:30:57.355 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:57.363+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:57.416 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-19T11:30:57.416 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-19T11:30:57.455 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:57.463+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:57.474 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:57.475 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:57.489 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-19T11:30:57.583 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:57.591+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:57.657 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:57.666+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:57.660 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:57.660 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:57.674 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-19T11:30:57.757 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:57.766+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:57.838 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:57.838 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:57.853 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-19T11:30:57.858 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:57.866+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:57.959 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:57.967+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:57.975 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:57.975 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:57.990 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-19T11:30:57.991 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-19T11:30:58.029 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.0 to 1 (10000) 2024-03-19T11:30:58.040 INFO:tasks.rados.rados.0.smithi038.stdout:1166: done (4 left) 2024-03-19T11:30:58.041 INFO:tasks.rados.rados.0.smithi038.stdout:1167: read oid 8 snap 17 2024-03-19T11:30:58.041 INFO:tasks.rados.rados.0.smithi038.stdout:1167: expect (ObjNum 60 snap 3 seq_num 60) 2024-03-19T11:30:58.041 INFO:tasks.rados.rados.0.smithi038.stdout:1162: finishing write tid 1 to smithi03829856-11 2024-03-19T11:30:58.041 INFO:tasks.rados.rados.0.smithi038.stdout:1162: finishing write tid 2 to smithi03829856-11 2024-03-19T11:30:58.041 INFO:tasks.rados.rados.0.smithi038.stdout:1162: finishing write tid 3 to smithi03829856-11 2024-03-19T11:30:58.041 INFO:tasks.rados.rados.0.smithi038.stdout:1162: finishing write tid 4 to smithi03829856-11 2024-03-19T11:30:58.041 INFO:tasks.rados.rados.0.smithi038.stdout:1162: oid 11 updating version 0 to 74 2024-03-19T11:30:58.041 INFO:tasks.rados.rados.0.smithi038.stdout:1162: oid 11 version 74 is already newer than 72 2024-03-19T11:30:58.041 INFO:tasks.rados.rados.0.smithi038.stdout:1162: oid 11 version 74 is already newer than 73 2024-03-19T11:30:58.041 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 11 v 74 (ObjNum 187 snap 21 seq_num 187) dirty exists 2024-03-19T11:30:58.041 INFO:tasks.rados.rados.0.smithi038.stdout:1162: left oid 11 (ObjNum 187 snap 21 seq_num 187) 2024-03-19T11:30:58.041 INFO:tasks.rados.rados.0.smithi038.stdout:1163: finishing write tid 1 to smithi03829856-31 2024-03-19T11:30:58.041 INFO:tasks.rados.rados.0.smithi038.stdout:1163: finishing write tid 2 to smithi03829856-31 2024-03-19T11:30:58.042 INFO:tasks.rados.rados.0.smithi038.stdout:1163: finishing write tid 3 to smithi03829856-31 2024-03-19T11:30:58.042 INFO:tasks.rados.rados.0.smithi038.stdout:1163: finishing write tid 4 to smithi03829856-31 2024-03-19T11:30:58.042 INFO:tasks.rados.rados.0.smithi038.stdout:1163: oid 31 updating version 0 to 121 2024-03-19T11:30:58.042 INFO:tasks.rados.rados.0.smithi038.stdout:1163: oid 31 version 121 is already newer than 119 2024-03-19T11:30:58.042 INFO:tasks.rados.rados.0.smithi038.stdout:1163: oid 31 version 121 is already newer than 120 2024-03-19T11:30:58.042 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 31 v 121 (ObjNum 188 snap 21 seq_num 188) dirty exists 2024-03-19T11:30:58.042 INFO:tasks.rados.rados.0.smithi038.stdout:1163: left oid 31 (ObjNum 188 snap 21 seq_num 188) 2024-03-19T11:30:58.042 INFO:tasks.rados.rados.0.smithi038.stdout:1165: finishing set_chunk to oid 37 2024-03-19T11:30:58.042 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 37 v 81 (ObjNum 176 snap 19 seq_num 176) dirty exists 2024-03-19T11:30:58.042 INFO:tasks.rados.rados.0.smithi038.stdout:1164: expect (ObjNum 183 snap 20 seq_num 183) 2024-03-19T11:30:58.072 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-19T11:30:58.075 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:30:58.083+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:30:58.156 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:30:58.156 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:30:58.161 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:30:58.161 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:30:58.161 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:58.170+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:58.171 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-19T11:30:58.180 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:30:58.180 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:30:58.196 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:30:58.197 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:30:58.214 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:30:58.215 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:30:58.232 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:30:58.232 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:30:58.251 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:30:58.251 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:30:58.263 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:58.267+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:58.268 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:30:58.268 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:30:58.288 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:30:58.289 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:30:58.306 INFO:tasks.rados.rados.0.smithi038.stdout:1167: expect (ObjNum 60 snap 3 seq_num 60) 2024-03-19T11:30:58.342 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:58.342 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:58.342 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:30:58.342 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:58.356 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-19T11:30:58.363 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:58.374+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:58.443 INFO:tasks.rados.rados.0.smithi038.stdout:1168: read oid 16 snap -1 2024-03-19T11:30:58.443 INFO:tasks.rados.rados.0.smithi038.stdout:1168: expect (ObjNum 65 snap 3 seq_num 65) 2024-03-19T11:30:58.443 INFO:tasks.rados.rados.0.smithi038.stdout:1162: done (5 left) 2024-03-19T11:30:58.443 INFO:tasks.rados.rados.0.smithi038.stdout:1163: done (4 left) 2024-03-19T11:30:58.443 INFO:tasks.rados.rados.0.smithi038.stdout:1164: done (3 left) 2024-03-19T11:30:58.443 INFO:tasks.rados.rados.0.smithi038.stdout:1165: done (2 left) 2024-03-19T11:30:58.443 INFO:tasks.rados.rados.0.smithi038.stdout:1167: done (1 left) 2024-03-19T11:30:58.443 INFO:tasks.rados.rados.0.smithi038.stdout:1169: set_chunk oid 10 target oid 2024-03-19T11:30:58.465 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:58.474+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:58.526 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:58.526 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:58.526 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:30:58.526 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:30:58.526 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:58.541 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-19T11:30:58.566 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:58.574+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:58.664 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:58.665 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:30:58.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:30:58.665 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:30:58.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.665 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:30 4.5 4:ac9cd3a3:::smithi03829856-8:head [write 1776668~340981] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:30:58.665 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:04.777922+0000", 2024-03-19T11:30:58.665 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 53.896821496999998, 2024-03-19T11:30:58.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015469894, 2024-03-19T11:30:58.665 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:58.665 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:58.665 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:58.665 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:58.665 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:58.666 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 30 2024-03-19T11:30:58.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.666 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:58.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.666 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:58.666 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.777922+0000", 2024-03-19T11:30:58.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:58.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.666 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:58.666 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.777922+0000", 2024-03-19T11:30:58.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:58.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:58.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.777924+0000", 2024-03-19T11:30:58.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3999999999999999e-06 2024-03-19T11:30:58.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:58.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778461+0000", 2024-03-19T11:30:58.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00053698299999999999 2024-03-19T11:30:58.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:58.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778487+0000", 2024-03-19T11:30:58.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5947000000000001e-05 2024-03-19T11:30:58.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.668 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.668 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:58.668 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778502+0000", 2024-03-19T11:30:58.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4256e-05 2024-03-19T11:30:58.668 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.668 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.668 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:58.668 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.781978+0000", 2024-03-19T11:30:58.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00347693 2024-03-19T11:30:58.668 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.668 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.668 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:58.668 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.782196+0000", 2024-03-19T11:30:58.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021726499999999999 2024-03-19T11:30:58.669 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.669 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.669 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:30:58.669 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.782572+0000", 2024-03-19T11:30:58.669 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00037635000000000001 2024-03-19T11:30:58.669 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.669 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.669 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:58.669 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.786960+0000", 2024-03-19T11:30:58.669 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043882130000000002 2024-03-19T11:30:58.669 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.669 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.669 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:58.669 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793113+0000", 2024-03-19T11:30:58.669 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061530760000000004 2024-03-19T11:30:58.670 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.670 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.670 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:58.670 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793276+0000", 2024-03-19T11:30:58.670 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000162652 2024-03-19T11:30:58.670 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.670 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.670 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:58.670 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793392+0000", 2024-03-19T11:30:58.670 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115822 2024-03-19T11:30:58.670 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:58.670 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:58.670 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:58.670 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.671 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:31 4.5 4:ac9cd3a3:::smithi03829856-8:head [setxattr _header (55) in=7b,truncate 2117649] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:30:58.671 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:04.778533+0000", 2024-03-19T11:30:58.671 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 53.896210361999998, 2024-03-19T11:30:58.671 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015182533, 2024-03-19T11:30:58.671 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:58.671 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:58.671 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:58.671 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:58.671 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:58.672 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 31 2024-03-19T11:30:58.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.672 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:58.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:58.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778533+0000", 2024-03-19T11:30:58.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:58.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:58.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778533+0000", 2024-03-19T11:30:58.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:58.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.673 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:58.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778534+0000", 2024-03-19T11:30:58.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1069999999999999e-06 2024-03-19T11:30:58.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.673 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:58.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778541+0000", 2024-03-19T11:30:58.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2529999999999998e-06 2024-03-19T11:30:58.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.674 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:58.674 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778558+0000", 2024-03-19T11:30:58.675 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6290000000000002e-05 2024-03-19T11:30:58.675 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.676 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.676 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:58.676 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778568+0000", 2024-03-19T11:30:58.676 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0360000000000001e-05 2024-03-19T11:30:58.676 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.676 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.676 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:58.677 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.785511+0000", 2024-03-19T11:30:58.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00694306 2024-03-19T11:30:58.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.677 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:58.677 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.785710+0000", 2024-03-19T11:30:58.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019837199999999999 2024-03-19T11:30:58.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.677 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:30:58.677 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.785929+0000", 2024-03-19T11:30:58.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021960799999999999 2024-03-19T11:30:58.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.677 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:58.678 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.787281+0000", 2024-03-19T11:30:58.678 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001352313 2024-03-19T11:30:58.678 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.678 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.678 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:58.678 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793492+0000", 2024-03-19T11:30:58.678 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062107760000000003 2024-03-19T11:30:58.678 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.678 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.678 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:58.678 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793578+0000", 2024-03-19T11:30:58.678 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5711999999999995e-05 2024-03-19T11:30:58.678 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.678 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.679 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:58.679 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793716+0000", 2024-03-19T11:30:58.679 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000137682 2024-03-19T11:30:58.679 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:58.679 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:58.679 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:58.679 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.679 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:32 4.5 4:ac9cd3a3:::smithi03829856-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:30:58.679 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:04.778582+0000", 2024-03-19T11:30:58.679 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 53.896161847999998, 2024-03-19T11:30:58.679 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015761827999999999, 2024-03-19T11:30:58.679 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:58.679 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:58.679 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:58.680 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:58.680 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:58.680 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 32 2024-03-19T11:30:58.680 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.680 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:58.680 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.680 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:58.680 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778582+0000", 2024-03-19T11:30:58.680 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:58.680 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.680 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.680 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:58.680 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778582+0000", 2024-03-19T11:30:58.680 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:58.680 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.681 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:58.681 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778583+0000", 2024-03-19T11:30:58.681 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.026e-06 2024-03-19T11:30:58.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.681 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:58.681 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778587+0000", 2024-03-19T11:30:58.681 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7870000000000001e-06 2024-03-19T11:30:58.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.681 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:58.681 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778599+0000", 2024-03-19T11:30:58.681 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1623e-05 2024-03-19T11:30:58.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.682 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.682 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:58.682 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778608+0000", 2024-03-19T11:30:58.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7399999999999993e-06 2024-03-19T11:30:58.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.682 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.682 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:58.682 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.787009+0000", 2024-03-19T11:30:58.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0084010490000000007 2024-03-19T11:30:58.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.682 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.683 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-03-19T11:30:58.683 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.787202+0000", 2024-03-19T11:30:58.683 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019309500000000001 2024-03-19T11:30:58.683 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.683 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.683 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:58.683 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793832+0000", 2024-03-19T11:30:58.683 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.006630052 2024-03-19T11:30:58.683 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.683 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.683 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:58.683 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.794048+0000", 2024-03-19T11:30:58.684 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000215505 2024-03-19T11:30:58.684 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.684 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.684 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:58.684 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.794343+0000", 2024-03-19T11:30:58.684 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029595100000000003 2024-03-19T11:30:58.684 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:58.684 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:58.684 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:58.684 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.684 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.684 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:856 4.5 4:ac9cd3a3:::smithi03829856-8:head [write 608420~529411] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:58.685 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.781425+0000", 2024-03-19T11:30:58.685 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 49.893318661999999, 2024-03-19T11:30:58.685 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014634677, 2024-03-19T11:30:58.685 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:58.685 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:58.685 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:58.685 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:58.685 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:58.685 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 856 2024-03-19T11:30:58.685 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.685 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:58.686 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.686 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:58.686 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.781425+0000", 2024-03-19T11:30:58.686 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:58.686 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.686 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.686 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:58.686 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.781425+0000", 2024-03-19T11:30:58.686 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:58.686 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.686 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.687 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:58.687 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.781428+0000", 2024-03-19T11:30:58.687 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.534e-06 2024-03-19T11:30:58.687 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.687 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.687 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:58.687 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.783096+0000", 2024-03-19T11:30:58.687 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001667897 2024-03-19T11:30:58.687 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.687 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.688 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:58.688 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.783138+0000", 2024-03-19T11:30:58.688 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1946000000000001e-05 2024-03-19T11:30:58.688 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.688 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.688 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:58.688 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.783163+0000", 2024-03-19T11:30:58.688 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4443e-05 2024-03-19T11:30:58.688 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.688 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.688 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:58.688 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.783333+0000", 2024-03-19T11:30:58.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017027899999999999 2024-03-19T11:30:58.689 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.689 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.689 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:58.689 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.783503+0000", 2024-03-19T11:30:58.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000170242 2024-03-19T11:30:58.689 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.689 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.689 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:30:58.689 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.783917+0000", 2024-03-19T11:30:58.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00041371000000000002 2024-03-19T11:30:58.689 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.689 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.689 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:58.689 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.795709+0000", 2024-03-19T11:30:58.690 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011791693000000001 2024-03-19T11:30:58.690 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.690 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.690 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:58.690 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.795850+0000", 2024-03-19T11:30:58.690 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000141695 2024-03-19T11:30:58.690 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.690 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.690 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:58.690 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.795940+0000", 2024-03-19T11:30:58.690 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9705999999999998e-05 2024-03-19T11:30:58.690 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.691 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.691 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:58.691 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.796059+0000", 2024-03-19T11:30:58.691 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011953199999999999 2024-03-19T11:30:58.691 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:58.691 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:58.691 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:58.691 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.691 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.691 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:891 4.5 4:af9d0d56:::smithi03829856-20:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:30:58.691 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:09.286410+0000", 2024-03-19T11:30:58.691 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 49.388333516000003, 2024-03-19T11:30:58.691 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035191405000000002, 2024-03-19T11:30:58.691 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:58.691 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:58.692 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:58.692 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:58.692 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:58.692 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 891 2024-03-19T11:30:58.692 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.692 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:58.692 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.692 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:58.692 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286410+0000", 2024-03-19T11:30:58.692 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:58.692 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.692 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.692 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:58.692 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286410+0000", 2024-03-19T11:30:58.692 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:58.692 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.693 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.693 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:58.693 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286413+0000", 2024-03-19T11:30:58.693 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2660000000000002e-06 2024-03-19T11:30:58.693 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.693 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.693 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:58.693 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286426+0000", 2024-03-19T11:30:58.693 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.258e-05 2024-03-19T11:30:58.693 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.693 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.693 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:58.693 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286460+0000", 2024-03-19T11:30:58.693 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3943999999999999e-05 2024-03-19T11:30:58.693 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.694 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.694 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:58.694 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286477+0000", 2024-03-19T11:30:58.694 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6799999999999998e-05 2024-03-19T11:30:58.694 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.694 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.694 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:58.694 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286605+0000", 2024-03-19T11:30:58.694 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000128892 2024-03-19T11:30:58.694 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.694 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.694 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:58.694 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286769+0000", 2024-03-19T11:30:58.695 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016359799999999999 2024-03-19T11:30:58.695 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.695 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.695 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:58.695 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.321601+0000", 2024-03-19T11:30:58.695 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.034832324999999997 2024-03-19T11:30:58.695 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:58.695 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:58.695 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:58.695 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.695 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.695 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1015 4.5 4:af9d0d56:::smithi03829856-20:head [tier-promote] snapc 6=[6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:30:58.695 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:12.761855+0000", 2024-03-19T11:30:58.695 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 45.912887990000002, 2024-03-19T11:30:58.695 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026760430000000002, 2024-03-19T11:30:58.695 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:58.696 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:58.696 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:58.696 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:58.696 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:58.696 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1015 2024-03-19T11:30:58.696 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.696 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:58.696 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.696 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:58.696 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761855+0000", 2024-03-19T11:30:58.696 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:58.696 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.696 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.696 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:58.697 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761855+0000", 2024-03-19T11:30:58.697 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:58.697 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.697 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.697 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:58.697 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761858+0000", 2024-03-19T11:30:58.697 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8140000000000001e-06 2024-03-19T11:30:58.697 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.697 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.697 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:58.697 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761872+0000", 2024-03-19T11:30:58.697 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3813e-05 2024-03-19T11:30:58.697 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.697 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.697 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:58.698 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761905+0000", 2024-03-19T11:30:58.698 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2642999999999999e-05 2024-03-19T11:30:58.698 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.698 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.698 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:58.698 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761925+0000", 2024-03-19T11:30:58.698 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0489e-05 2024-03-19T11:30:58.698 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.698 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.698 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:58.698 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.762070+0000", 2024-03-19T11:30:58.698 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000145153 2024-03-19T11:30:58.698 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.698 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.699 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:58.699 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.762365+0000", 2024-03-19T11:30:58.699 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029482100000000002 2024-03-19T11:30:58.699 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.699 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.699 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:58.699 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.788616+0000", 2024-03-19T11:30:58.699 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026250697 2024-03-19T11:30:58.699 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:58.699 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:58.699 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:58.699 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.699 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.699 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1121 4.5 4:aa3303a5:::smithi03829856-17:head [tier-promote] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:58.699 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:14.164281+0000", 2024-03-19T11:30:58.700 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 44.510462744999998, 2024-03-19T11:30:58.700 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023896435000000001, 2024-03-19T11:30:58.700 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:58.700 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:58.700 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:58.700 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:58.700 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:58.700 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1121 2024-03-19T11:30:58.700 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.700 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:58.701 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.701 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:58.701 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164281+0000", 2024-03-19T11:30:58.701 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:58.701 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.701 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.701 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:58.701 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164281+0000", 2024-03-19T11:30:58.701 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:58.701 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.701 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.701 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:58.701 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164283+0000", 2024-03-19T11:30:58.701 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8500000000000001e-06 2024-03-19T11:30:58.701 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.702 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.702 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:58.702 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164291+0000", 2024-03-19T11:30:58.702 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.14e-06 2024-03-19T11:30:58.702 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.702 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.702 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:58.702 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164313+0000", 2024-03-19T11:30:58.702 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.207e-05 2024-03-19T11:30:58.702 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.702 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.702 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:58.702 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164328+0000", 2024-03-19T11:30:58.702 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5403000000000001e-05 2024-03-19T11:30:58.703 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.703 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.703 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:58.703 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.167981+0000", 2024-03-19T11:30:58.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036529879999999998 2024-03-19T11:30:58.703 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.703 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.703 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:58.703 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.168274+0000", 2024-03-19T11:30:58.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029313100000000002 2024-03-19T11:30:58.703 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.703 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.703 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:58.703 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.188177+0000", 2024-03-19T11:30:58.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019902853000000002 2024-03-19T11:30:58.703 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:58.704 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:58.704 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:58.704 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.704 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.704 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1302 4.15 4:aa3303a5:::smithi03829856-17:head [write 413472~620140,stat] snapc b=[b,a,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-03-19T11:30:58.704 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:22.966426+0000", 2024-03-19T11:30:58.704 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 35.708317012000002, 2024-03-19T11:30:58.704 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018043513000000001, 2024-03-19T11:30:58.704 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:58.704 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:58.704 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:58.704 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:58.704 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:58.704 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1302 2024-03-19T11:30:58.704 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.705 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:58.705 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.705 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:58.705 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.966426+0000", 2024-03-19T11:30:58.705 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:58.705 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.705 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.705 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:58.705 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.966426+0000", 2024-03-19T11:30:58.705 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:58.705 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.705 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.705 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:58.705 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.966429+0000", 2024-03-19T11:30:58.705 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3829999999999999e-06 2024-03-19T11:30:58.706 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.706 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.706 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:58.706 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972117+0000", 2024-03-19T11:30:58.706 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0056885750000000004 2024-03-19T11:30:58.706 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.706 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.706 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:58.706 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972156+0000", 2024-03-19T11:30:58.706 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8878000000000003e-05 2024-03-19T11:30:58.706 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.706 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.707 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:58.707 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972184+0000", 2024-03-19T11:30:58.707 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7509999999999999e-05 2024-03-19T11:30:58.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.707 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:58.707 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972309+0000", 2024-03-19T11:30:58.707 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012525899999999999 2024-03-19T11:30:58.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.707 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:58.708 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972566+0000", 2024-03-19T11:30:58.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025740799999999999 2024-03-19T11:30:58.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.708 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.708 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:30:58.708 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972899+0000", 2024-03-19T11:30:58.709 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033238799999999998 2024-03-19T11:30:58.709 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.709 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.709 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:58.709 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984088+0000", 2024-03-19T11:30:58.709 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011189466 2024-03-19T11:30:58.709 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.710 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:58.710 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984138+0000", 2024-03-19T11:30:58.710 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9666999999999997e-05 2024-03-19T11:30:58.710 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.711 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:58.711 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984329+0000", 2024-03-19T11:30:58.711 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019086799999999999 2024-03-19T11:30:58.711 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.711 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.711 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:58.711 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984470+0000", 2024-03-19T11:30:58.712 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000141111 2024-03-19T11:30:58.712 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:58.712 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:58.712 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:58.713 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.713 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.713 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1303 4.15 4:aa3303a5:::smithi03829856-17:head [write 1818036~510700] snapc b=[b,a,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-03-19T11:30:58.713 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:22.976856+0000", 2024-03-19T11:30:58.713 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 35.697887901999998, 2024-03-19T11:30:58.713 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015045703000000001, 2024-03-19T11:30:58.714 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:58.714 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:58.714 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:58.714 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:58.714 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:58.714 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1303 2024-03-19T11:30:58.715 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.715 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:58.715 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.715 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:58.715 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.976856+0000", 2024-03-19T11:30:58.715 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:58.715 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.716 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.716 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:58.716 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.976856+0000", 2024-03-19T11:30:58.716 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:58.716 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.716 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.717 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:58.717 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.976859+0000", 2024-03-19T11:30:58.717 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3629999999999998e-06 2024-03-19T11:30:58.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.717 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.717 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:58.717 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.977533+0000", 2024-03-19T11:30:58.717 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00067380800000000002 2024-03-19T11:30:58.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.718 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.718 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:58.718 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.977573+0000", 2024-03-19T11:30:58.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9758000000000001e-05 2024-03-19T11:30:58.718 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.719 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:58.719 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.977602+0000", 2024-03-19T11:30:58.719 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9856e-05 2024-03-19T11:30:58.719 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.720 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:58.720 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.979943+0000", 2024-03-19T11:30:58.720 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0023406400000000002 2024-03-19T11:30:58.720 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.720 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:58.720 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.980238+0000", 2024-03-19T11:30:58.720 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029488000000000002 2024-03-19T11:30:58.720 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.720 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:30:58.720 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.980734+0000", 2024-03-19T11:30:58.720 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00049647500000000002 2024-03-19T11:30:58.720 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.721 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.721 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:58.721 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.991761+0000", 2024-03-19T11:30:58.721 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011026515000000001 2024-03-19T11:30:58.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.721 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.721 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:58.721 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.991777+0000", 2024-03-19T11:30:58.721 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5727999999999999e-05 2024-03-19T11:30:58.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.721 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.721 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:58.721 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.991840+0000", 2024-03-19T11:30:58.721 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3091999999999998e-05 2024-03-19T11:30:58.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.722 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.722 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:58.722 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.991901+0000", 2024-03-19T11:30:58.722 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1587999999999994e-05 2024-03-19T11:30:58.722 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:58.722 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:58.722 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:58.722 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.722 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.722 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1468 4.a 4:567251f3:::smithi03829856-44:head [write 1603467~416361] snapc c=[c,b,a,6,5] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-03-19T11:30:58.722 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.083922+0000", 2024-03-19T11:30:58.722 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 31.590821879, 2024-03-19T11:30:58.722 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014508549000000001, 2024-03-19T11:30:58.723 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:58.723 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:58.723 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:58.723 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:58.723 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:58.723 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1468 2024-03-19T11:30:58.723 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.723 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:58.723 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.723 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:58.723 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.083922+0000", 2024-03-19T11:30:58.723 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:58.723 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.723 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.723 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:58.724 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.083922+0000", 2024-03-19T11:30:58.724 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:58.724 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.724 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.724 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:58.724 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.083924+0000", 2024-03-19T11:30:58.724 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2800000000000002e-06 2024-03-19T11:30:58.724 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.724 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.724 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:58.724 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.084498+0000", 2024-03-19T11:30:58.724 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057414900000000004 2024-03-19T11:30:58.724 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.724 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.724 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:58.724 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.084529+0000", 2024-03-19T11:30:58.725 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1080000000000001e-05 2024-03-19T11:30:58.725 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.725 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.725 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:58.725 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.084544+0000", 2024-03-19T11:30:58.725 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4498000000000001e-05 2024-03-19T11:30:58.725 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.725 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.725 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:58.725 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.088993+0000", 2024-03-19T11:30:58.725 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044497260000000002 2024-03-19T11:30:58.725 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.725 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.725 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:58.725 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.089152+0000", 2024-03-19T11:30:58.726 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015833199999999999 2024-03-19T11:30:58.726 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.726 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.726 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:58.726 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.089419+0000", 2024-03-19T11:30:58.726 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026691799999999999 2024-03-19T11:30:58.726 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.726 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.726 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:58.726 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.092883+0000", 2024-03-19T11:30:58.726 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034646590000000001 2024-03-19T11:30:58.726 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.726 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.726 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:58.726 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.098276+0000", 2024-03-19T11:30:58.727 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053924680000000001 2024-03-19T11:30:58.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.727 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:58.727 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.098354+0000", 2024-03-19T11:30:58.727 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8763999999999993e-05 2024-03-19T11:30:58.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.727 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:58.727 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.098430+0000", 2024-03-19T11:30:58.727 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5674999999999996e-05 2024-03-19T11:30:58.727 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:58.727 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:58.727 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:58.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.728 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.728 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.1 50/49 [PushOp(4:866f3dcc:::smithi03829856-31:7, version: 38'44, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@38'44, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 45'62, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@45'62, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 46'68, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@46'68, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:7, version: 47'71, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@47'71, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 47'72, data_included: [440350~816629,1309629~231947], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@47'72, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:30:58.728 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.959457+0000", 2024-03-19T11:30:58.728 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 30.715286594999998, 2024-03-19T11:30:58.728 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018416666000000002, 2024-03-19T11:30:58.728 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:58.728 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:58.728 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:58.728 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.728 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:58.728 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.959457+0000", 2024-03-19T11:30:58.728 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:58.728 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.729 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.729 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:58.729 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.959457+0000", 2024-03-19T11:30:58.729 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:58.729 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.729 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.729 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:58.729 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.959458+0000", 2024-03-19T11:30:58.729 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4899999999999999e-06 2024-03-19T11:30:58.729 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.730 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.730 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:58.730 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.962822+0000", 2024-03-19T11:30:58.730 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0033640620000000001 2024-03-19T11:30:58.730 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.730 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.730 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:58.730 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.962893+0000", 2024-03-19T11:30:58.730 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1060000000000001e-05 2024-03-19T11:30:58.730 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.730 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.730 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:58.730 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.962977+0000", 2024-03-19T11:30:58.730 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3311000000000002e-05 2024-03-19T11:30:58.731 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.731 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.731 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:58.731 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963208+0000", 2024-03-19T11:30:58.731 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023121200000000001 2024-03-19T11:30:58.731 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.731 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.731 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:58.731 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963245+0000", 2024-03-19T11:30:58.731 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6844999999999997e-05 2024-03-19T11:30:58.731 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.731 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.731 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:58.731 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.977874+0000", 2024-03-19T11:30:58.731 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014628686 2024-03-19T11:30:58.732 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:58.732 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:58.732 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:58.732 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.732 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.732 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.1 e50,49 4:8401c1f0:::smithi03829856-43:8,46'66,4:866f3dcc:::smithi03829856-31:2,47'73)", 2024-03-19T11:30:58.732 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.963027+0000", 2024-03-19T11:30:58.732 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 30.711716977999998, 2024-03-19T11:30:58.732 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015116788000000001, 2024-03-19T11:30:58.732 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:58.732 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:58.732 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:58.732 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.732 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:58.732 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963027+0000", 2024-03-19T11:30:58.732 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:58.733 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.733 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.733 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:58.733 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963027+0000", 2024-03-19T11:30:58.733 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:58.733 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.733 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.733 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:58.733 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963028+0000", 2024-03-19T11:30:58.733 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.268e-06 2024-03-19T11:30:58.733 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.733 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.733 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:58.733 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963036+0000", 2024-03-19T11:30:58.733 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.926e-06 2024-03-19T11:30:58.734 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.734 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.734 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:58.734 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963050+0000", 2024-03-19T11:30:58.734 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4082e-05 2024-03-19T11:30:58.734 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.734 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.734 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:58.734 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963062+0000", 2024-03-19T11:30:58.734 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2415000000000001e-05 2024-03-19T11:30:58.734 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.734 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.734 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:58.734 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.977954+0000", 2024-03-19T11:30:58.734 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014891572 2024-03-19T11:30:58.735 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.735 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.735 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:58.735 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.977965+0000", 2024-03-19T11:30:58.735 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1654000000000001e-05 2024-03-19T11:30:58.735 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.735 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.735 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:58.735 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.978143+0000", 2024-03-19T11:30:58.735 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017787099999999999 2024-03-19T11:30:58.735 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:58.735 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:58.735 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:58.735 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.735 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.736 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.11 50/49 [PushOp(4:8d3763f9:::smithi03829856-29:head, version: 47'60, data_included: [3364276~232389], data_size: 232389, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8d3763f9:::smithi03829856-29:head@47'60, size: 3596665, copy_subset: [(0, 3596665)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3596665, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3364276, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:30:58.736 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.964363+0000", 2024-03-19T11:30:58.736 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 30.710380311000002, 2024-03-19T11:30:58.736 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014531759999999999, 2024-03-19T11:30:58.736 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:58.736 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:58.736 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:58.736 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.736 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:58.736 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.964363+0000", 2024-03-19T11:30:58.736 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:58.736 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.736 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.736 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:58.736 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.964363+0000", 2024-03-19T11:30:58.737 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:58.737 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.737 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.737 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:58.737 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.964364+0000", 2024-03-19T11:30:58.737 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4200000000000004e-07 2024-03-19T11:30:58.737 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.737 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.737 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:58.737 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.964447+0000", 2024-03-19T11:30:58.737 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2430999999999997e-05 2024-03-19T11:30:58.737 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.737 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.738 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:58.738 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.964468+0000", 2024-03-19T11:30:58.738 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1685999999999999e-05 2024-03-19T11:30:58.738 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.738 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:58.738 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.964483+0000", 2024-03-19T11:30:58.738 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4969e-05 2024-03-19T11:30:58.738 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.738 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:58.738 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.977934+0000", 2024-03-19T11:30:58.738 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013450963999999999 2024-03-19T11:30:58.739 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.739 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.739 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:58.739 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.977975+0000", 2024-03-19T11:30:58.739 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1125999999999999e-05 2024-03-19T11:30:58.739 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.739 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.739 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:58.739 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.978895+0000", 2024-03-19T11:30:58.739 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00091964200000000003 2024-03-19T11:30:58.739 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:58.739 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:58.739 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:58.739 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.739 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.739 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.1 50/49 [PushOp(4:866f3dcc:::smithi03829856-31:7, version: 38'44, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@38'44, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 45'62, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@45'62, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 46'68, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@46'68, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:7, version: 47'71, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@47'71, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 47'72, data_included: [1541576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@47'72, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:30:58.740 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.991858+0000", 2024-03-19T11:30:58.740 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 30.682885387999999, 2024-03-19T11:30:58.740 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014228774, 2024-03-19T11:30:58.740 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:58.740 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:58.740 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:58.740 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.740 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:58.740 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.991858+0000", 2024-03-19T11:30:58.740 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:58.740 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.740 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.740 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:58.740 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.991858+0000", 2024-03-19T11:30:58.740 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:58.741 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.741 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.741 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:58.741 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.991859+0000", 2024-03-19T11:30:58.741 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.342e-06 2024-03-19T11:30:58.741 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.741 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.741 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:58.741 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.994675+0000", 2024-03-19T11:30:58.741 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028153150000000001 2024-03-19T11:30:58.741 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.741 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.741 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:58.741 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.994733+0000", 2024-03-19T11:30:58.741 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8514999999999998e-05 2024-03-19T11:30:58.741 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.742 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.742 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:58.742 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.994794+0000", 2024-03-19T11:30:58.742 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0588000000000003e-05 2024-03-19T11:30:58.742 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.742 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.742 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:58.742 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.995070+0000", 2024-03-19T11:30:58.742 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027573700000000002 2024-03-19T11:30:58.742 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.742 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.742 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:58.742 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.995100+0000", 2024-03-19T11:30:58.742 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0679000000000002e-05 2024-03-19T11:30:58.743 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.743 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.743 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:58.743 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.006087+0000", 2024-03-19T11:30:58.743 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010986598 2024-03-19T11:30:58.743 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:58.743 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:58.743 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:58.743 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.743 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.743 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2284 4.15 4:afa9f208:::smithi03829856-35:head [read 1314816~702976] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-03-19T11:30:58.743 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:53.596775+0000", 2024-03-19T11:30:58.743 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.0779685089999997, 2024-03-19T11:30:58.743 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.47260528299999999, 2024-03-19T11:30:58.743 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:58.744 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:58.744 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:58.744 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:58.744 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:58.744 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2284 2024-03-19T11:30:58.744 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.744 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:58.744 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.744 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:58.744 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596775+0000", 2024-03-19T11:30:58.744 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:58.744 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.744 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.744 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:58.744 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596775+0000", 2024-03-19T11:30:58.744 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:58.745 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.745 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.745 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:58.745 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596778+0000", 2024-03-19T11:30:58.745 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.174e-06 2024-03-19T11:30:58.745 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.745 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.745 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:58.745 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596794+0000", 2024-03-19T11:30:58.745 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5372000000000001e-05 2024-03-19T11:30:58.745 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.745 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.745 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:58.745 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596828+0000", 2024-03-19T11:30:58.745 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4107999999999999e-05 2024-03-19T11:30:58.746 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.746 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.746 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:58.746 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596844+0000", 2024-03-19T11:30:58.746 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6296000000000001e-05 2024-03-19T11:30:58.746 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.746 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.746 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:58.746 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596968+0000", 2024-03-19T11:30:58.746 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012383699999999999 2024-03-19T11:30:58.746 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.746 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.746 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:30:58.746 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596988+0000", 2024-03-19T11:30:58.747 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0645e-05 2024-03-19T11:30:58.747 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.747 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.747 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:58.747 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.068916+0000", 2024-03-19T11:30:58.747 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.47192744800000003 2024-03-19T11:30:58.747 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.747 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.747 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:58.747 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.069015+0000", 2024-03-19T11:30:58.747 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9500999999999995e-05 2024-03-19T11:30:58.747 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.747 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.748 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:58.748 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.069380+0000", 2024-03-19T11:30:58.748 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00036490200000000001 2024-03-19T11:30:58.748 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:58.748 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:58.748 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:58.748 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.748 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.748 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2286 4.15 4:afa9f208:::smithi03829856-35:head [assert-version v108,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e75)", 2024-03-19T11:30:58.748 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.084826+0000", 2024-03-19T11:30:58.748 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 4.589917142, 2024-03-19T11:30:58.748 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.055799556, 2024-03-19T11:30:58.748 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:58.748 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:58.748 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:58.749 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:30:58.749 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:58.749 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2286 2024-03-19T11:30:58.749 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.749 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:58.749 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.749 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:58.749 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084826+0000", 2024-03-19T11:30:58.749 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:58.749 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.749 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.749 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:58.749 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084826+0000", 2024-03-19T11:30:58.749 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:58.750 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.750 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.750 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:58.750 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084828+0000", 2024-03-19T11:30:58.750 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1409999999999999e-06 2024-03-19T11:30:58.750 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.750 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.750 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:58.750 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084838+0000", 2024-03-19T11:30:58.750 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3740000000000004e-06 2024-03-19T11:30:58.750 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.750 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.750 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:58.750 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084859+0000", 2024-03-19T11:30:58.751 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1644000000000001e-05 2024-03-19T11:30:58.751 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.751 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.751 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:58.751 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084871+0000", 2024-03-19T11:30:58.751 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1127e-05 2024-03-19T11:30:58.751 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.751 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.751 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:58.751 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084961+0000", 2024-03-19T11:30:58.751 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0371999999999995e-05 2024-03-19T11:30:58.751 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.751 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.751 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:58.751 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.085130+0000", 2024-03-19T11:30:58.752 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016884800000000001 2024-03-19T11:30:58.752 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.752 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.752 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5,7", 2024-03-19T11:30:58.752 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.089309+0000", 2024-03-19T11:30:58.752 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041788650000000004 2024-03-19T11:30:58.752 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.752 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.752 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:58.752 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.090362+0000", 2024-03-19T11:30:58.752 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001053362 2024-03-19T11:30:58.752 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.752 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.752 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:58.752 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.090511+0000", 2024-03-19T11:30:58.752 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014887200000000001 2024-03-19T11:30:58.753 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.753 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.753 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:58.753 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.140479+0000", 2024-03-19T11:30:58.753 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.049968006000000002 2024-03-19T11:30:58.753 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.753 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.753 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:58.753 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.140534+0000", 2024-03-19T11:30:58.753 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4917000000000003e-05 2024-03-19T11:30:58.753 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.753 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.753 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:58.753 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.140626+0000", 2024-03-19T11:30:58.753 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2027999999999994e-05 2024-03-19T11:30:58.753 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:58.754 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:58.754 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:58.754 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.754 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.754 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1d58db11:::smithi03829856-18:6, version: 55'80, data_included: [759683~639358,1904434~409218], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:6@55'80, size: 3159683, copy_subset: [(0, 3159683)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2313652, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1d58db11:::smithi03829856-18:b, version: 58'84, data_included: [443718~955323,1564138~93253], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:b@58'84, size: 3445402, copy_subset: [(0, 3445402)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1657391, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:10, version: 66'95, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:10@66'95, size: 3764735, copy_subset: [(0, 3764735)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:12, version: 66'97, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:12@66'97, size: 2504243, copy_subset: [(0, 2504243)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:c, version: 70'102, data_included: [416807~576167,1542129~472409], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:c@70'102, size: 2857157, copy_subset: [(0, 2857157)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2014538, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:30:58.754 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.274749+0000", 2024-03-19T11:30:58.754 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 4.3999942110000001, 2024-03-19T11:30:58.754 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017652601, 2024-03-19T11:30:58.754 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:58.754 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:58.754 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:58.754 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.754 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:58.754 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.274749+0000", 2024-03-19T11:30:58.755 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:58.755 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.755 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.755 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:58.755 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.274749+0000", 2024-03-19T11:30:58.755 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:58.755 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.755 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.755 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:58.755 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.274751+0000", 2024-03-19T11:30:58.755 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8500000000000001e-06 2024-03-19T11:30:58.755 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.755 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.755 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:58.755 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.278426+0000", 2024-03-19T11:30:58.756 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036747559999999999 2024-03-19T11:30:58.756 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.756 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.756 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:58.756 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.278470+0000", 2024-03-19T11:30:58.756 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4119000000000002e-05 2024-03-19T11:30:58.756 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.756 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.756 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:58.756 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.278506+0000", 2024-03-19T11:30:58.756 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6066000000000003e-05 2024-03-19T11:30:58.756 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.756 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.756 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:58.756 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.278709+0000", 2024-03-19T11:30:58.757 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000202711 2024-03-19T11:30:58.757 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.757 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.757 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:58.757 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.278731+0000", 2024-03-19T11:30:58.757 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2657e-05 2024-03-19T11:30:58.757 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.757 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.757 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:58.757 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.292402+0000", 2024-03-19T11:30:58.757 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013670442 2024-03-19T11:30:58.757 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:58.757 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:58.757 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:58.758 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.758 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.758 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1dabc0f1:::smithi03829856-1:head, version: 70'103, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:head@70'103, size: 2621892, copy_subset: [(0, 2621892)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:30:58.758 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.280012+0000", 2024-03-19T11:30:58.758 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 4.3947315659999999, 2024-03-19T11:30:58.758 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018569209, 2024-03-19T11:30:58.758 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:58.758 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:58.758 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:58.758 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.758 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:58.758 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280012+0000", 2024-03-19T11:30:58.758 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:58.759 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.759 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:58.759 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280012+0000", 2024-03-19T11:30:58.759 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:58.759 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.759 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:58.759 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280013+0000", 2024-03-19T11:30:58.759 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9899999999999999e-07 2024-03-19T11:30:58.759 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.759 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:58.759 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280584+0000", 2024-03-19T11:30:58.759 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057090299999999999 2024-03-19T11:30:58.760 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.760 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.760 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:58.760 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280612+0000", 2024-03-19T11:30:58.760 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8442999999999999e-05 2024-03-19T11:30:58.760 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.760 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.760 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:58.760 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280631+0000", 2024-03-19T11:30:58.760 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8896999999999999e-05 2024-03-19T11:30:58.760 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.760 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.760 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:58.760 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.295041+0000", 2024-03-19T11:30:58.761 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014410127999999999 2024-03-19T11:30:58.761 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.761 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.761 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:58.761 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.295050+0000", 2024-03-19T11:30:58.761 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2350000000000005e-06 2024-03-19T11:30:58.761 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.761 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.761 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:58.761 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.298581+0000", 2024-03-19T11:30:58.761 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035308039999999998 2024-03-19T11:30:58.761 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:58.761 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:58.761 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:58.761 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.762 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.762 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.18 e75,74 4:1d58db11:::smithi03829856-18:c,70'104)", 2024-03-19T11:30:58.762 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.280680+0000", 2024-03-19T11:30:58.762 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 4.3940630909999996, 2024-03-19T11:30:58.762 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018106229000000001, 2024-03-19T11:30:58.762 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:58.762 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:58.762 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:58.762 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.762 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:58.762 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280680+0000", 2024-03-19T11:30:58.762 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:58.762 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.762 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.763 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:58.763 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280680+0000", 2024-03-19T11:30:58.763 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:58.763 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.763 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:58.763 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280681+0000", 2024-03-19T11:30:58.763 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2099999999999995e-07 2024-03-19T11:30:58.763 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.763 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:58.763 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280686+0000", 2024-03-19T11:30:58.763 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2240000000000003e-06 2024-03-19T11:30:58.763 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.764 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:58.764 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280698+0000", 2024-03-19T11:30:58.764 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1698999999999999e-05 2024-03-19T11:30:58.764 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.764 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:58.764 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280705+0000", 2024-03-19T11:30:58.764 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4350000000000002e-06 2024-03-19T11:30:58.764 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.764 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:58.764 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.298643+0000", 2024-03-19T11:30:58.764 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017938352000000001 2024-03-19T11:30:58.764 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.765 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:58.765 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.298656+0000", 2024-03-19T11:30:58.765 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3084999999999999e-05 2024-03-19T11:30:58.765 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.765 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.765 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:58.765 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.298787+0000", 2024-03-19T11:30:58.765 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013061299999999999 2024-03-19T11:30:58.765 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:58.765 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:58.765 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:58.765 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.765 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.765 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.5.0:143 4.18 e75/74 4:1d58db11:::smithi03829856-18:head v 75'106, mlcod=0'0)", 2024-03-19T11:30:58.766 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.282327+0000", 2024-03-19T11:30:58.766 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 4.3924167199999999, 2024-03-19T11:30:58.766 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016753244, 2024-03-19T11:30:58.766 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:58.766 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:58.766 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:58.766 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.766 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:58.766 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.282327+0000", 2024-03-19T11:30:58.766 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:58.766 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.766 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.766 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:58.766 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.282327+0000", 2024-03-19T11:30:58.766 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:58.767 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.767 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:58.767 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.282328+0000", 2024-03-19T11:30:58.767 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.488e-06 2024-03-19T11:30:58.767 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.767 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:58.767 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.283127+0000", 2024-03-19T11:30:58.767 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00079853699999999999 2024-03-19T11:30:58.767 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.767 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:58.767 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.283152+0000", 2024-03-19T11:30:58.768 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5215000000000002e-05 2024-03-19T11:30:58.768 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.768 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.768 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:58.768 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.283167+0000", 2024-03-19T11:30:58.768 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4635e-05 2024-03-19T11:30:58.768 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.768 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.768 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:58.768 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.292633+0000", 2024-03-19T11:30:58.768 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0094664339999999993 2024-03-19T11:30:58.768 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.768 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.769 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:58.769 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.292655+0000", 2024-03-19T11:30:58.769 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1934999999999999e-05 2024-03-19T11:30:58.769 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.769 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.769 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:58.769 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.298981+0000", 2024-03-19T11:30:58.769 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0063255129999999996 2024-03-19T11:30:58.769 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:58.769 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:58.769 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:58.769 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.299080+0000", 2024-03-19T11:30:58.769 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9487000000000005e-05 2024-03-19T11:30:58.769 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:58.769 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:58.769 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:58.770 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:58.770 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:58.770 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:58.771 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-19T11:30:58.772 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:58.783+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:58.872 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:58.882+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:58.943 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:58.944 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:58.944 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:30:58.944 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:58.958 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-19T11:30:58.972 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:58.982+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:59.051 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.1 to 1 (10000) 2024-03-19T11:30:59.072 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-03-19T11:30:59.073 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:30:59.082+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:30:59.089 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:59.089 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:30:59.089 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:30:59.089 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:30:59.090 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:59.091 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-19T11:30:59.105 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-19T11:30:59.142 INFO:tasks.rados.rados.0.smithi038.stdout:1169: set_chunk oid 10 offset: 832512 length: 818688 target oid 50ef59070277995d5b83c8b65a9f287e91a186c613251218ca510d4e390b76fb offset: 0 snap 9 2024-03-19T11:30:59.142 INFO:tasks.rados.rados.0.smithi038.stdout:1170: set_chunk oid 9 target oid 2024-03-19T11:30:59.151 INFO:tasks.rados.rados.0.smithi038.stdout:1170: set_chunk oid 9 offset: 1457664 length: 612352 target oid 456bdd4683d667fc2d02bfdccf979ad64cdeddb396627b96703eec07352f535a offset: 0 snap 17 2024-03-19T11:30:59.151 INFO:tasks.rados.rados.0.smithi038.stdout:1171: rollback oid 5 current snap is 21 2024-03-19T11:30:59.151 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 5 to 11 2024-03-19T11:30:59.152 INFO:tasks.rados.rados.0.smithi038.stdout:1168: expect (ObjNum 65 snap 3 seq_num 65) 2024-03-19T11:30:59.174 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:59.183+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:59.234 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:30:59.234 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:30:59.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:30:59.234 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:30:59.234 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.234 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:30:59.234 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:44.200920+0000", 2024-03-19T11:30:59.235 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 75.041127008999993, 2024-03-19T11:30:59.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.80881896499999995, 2024-03-19T11:30:59.235 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:59.235 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:59.235 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:59.235 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:59.235 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:59.235 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1 2024-03-19T11:30:59.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.235 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:59.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:59.235 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200920+0000", 2024-03-19T11:30:59.236 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:59.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.236 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:59.236 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200920+0000", 2024-03-19T11:30:59.236 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:59.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.236 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:59.236 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200923+0000", 2024-03-19T11:30:59.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6469999999999998e-06 2024-03-19T11:30:59.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:59.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200937+0000", 2024-03-19T11:30:59.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.399e-05 2024-03-19T11:30:59.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:59.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200960+0000", 2024-03-19T11:30:59.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2603e-05 2024-03-19T11:30:59.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:59.238 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200978+0000", 2024-03-19T11:30:59.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.791e-05 2024-03-19T11:30:59.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:59.238 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.201081+0000", 2024-03-19T11:30:59.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010344900000000001 2024-03-19T11:30:59.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:30:59.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.201096+0000", 2024-03-19T11:30:59.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4443e-05 2024-03-19T11:30:59.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:59.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.005978+0000", 2024-03-19T11:30:59.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.80488281500000003 2024-03-19T11:30:59.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:59.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.006267+0000", 2024-03-19T11:30:59.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000288688 2024-03-19T11:30:59.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:59.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.006534+0000", 2024-03-19T11:30:59.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026725799999999998 2024-03-19T11:30:59.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:59.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.008457+0000", 2024-03-19T11:30:59.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0019227820000000001 2024-03-19T11:30:59.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:59.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.009570+0000", 2024-03-19T11:30:59.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011131750000000001 2024-03-19T11:30:59.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:59.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.009670+0000", 2024-03-19T11:30:59.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9458999999999998e-05 2024-03-19T11:30:59.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:59.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.009739+0000", 2024-03-19T11:30:59.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9746000000000006e-05 2024-03-19T11:30:59.242 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:59.242 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:59.242 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:59.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.242 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:30:59.242 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.071728+0000", 2024-03-19T11:30:59.242 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 74.170319571999997, 2024-03-19T11:30:59.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013714690999999999, 2024-03-19T11:30:59.243 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:59.243 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:59.243 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:59.243 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:59.243 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:59.243 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 34 2024-03-19T11:30:59.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.243 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:59.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:59.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071728+0000", 2024-03-19T11:30:59.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:59.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:59.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071728+0000", 2024-03-19T11:30:59.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:59.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:59.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071729+0000", 2024-03-19T11:30:59.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.47e-07 2024-03-19T11:30:59.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:59.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071735+0000", 2024-03-19T11:30:59.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3799999999999999e-06 2024-03-19T11:30:59.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:59.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071748+0000", 2024-03-19T11:30:59.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2806000000000001e-05 2024-03-19T11:30:59.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:59.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071756+0000", 2024-03-19T11:30:59.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1240000000000005e-06 2024-03-19T11:30:59.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.247 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:59.247 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.077570+0000", 2024-03-19T11:30:59.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.005813342 2024-03-19T11:30:59.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.248 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:59.248 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.077793+0000", 2024-03-19T11:30:59.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022391500000000001 2024-03-19T11:30:59.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.248 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:59.248 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.078146+0000", 2024-03-19T11:30:59.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00035219 2024-03-19T11:30:59.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.248 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:59.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.083390+0000", 2024-03-19T11:30:59.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052445590000000002 2024-03-19T11:30:59.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:59.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085288+0000", 2024-03-19T11:30:59.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0018977060000000001 2024-03-19T11:30:59.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:59.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085377+0000", 2024-03-19T11:30:59.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9068999999999997e-05 2024-03-19T11:30:59.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:59.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085443+0000", 2024-03-19T11:30:59.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5653000000000003e-05 2024-03-19T11:30:59.250 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:59.250 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:59.250 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:59.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.250 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:30:59.250 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.071777+0000", 2024-03-19T11:30:59.251 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 74.170270462000005, 2024-03-19T11:30:59.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013962445, 2024-03-19T11:30:59.251 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:59.251 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:59.251 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:59.251 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:59.251 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:59.251 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 35 2024-03-19T11:30:59.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.251 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:59.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:59.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071777+0000", 2024-03-19T11:30:59.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:59.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:59.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071777+0000", 2024-03-19T11:30:59.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:59.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:59.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071778+0000", 2024-03-19T11:30:59.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5000000000000001e-07 2024-03-19T11:30:59.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:59.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071917+0000", 2024-03-19T11:30:59.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013880199999999999 2024-03-19T11:30:59.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:59.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071931+0000", 2024-03-19T11:30:59.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4080000000000001e-05 2024-03-19T11:30:59.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.254 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:59.254 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071940+0000", 2024-03-19T11:30:59.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.363e-06 2024-03-19T11:30:59.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.254 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:59.254 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.080462+0000", 2024-03-19T11:30:59.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085221810000000002 2024-03-19T11:30:59.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:59.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.080582+0000", 2024-03-19T11:30:59.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011935299999999999 2024-03-19T11:30:59.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:59.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.080782+0000", 2024-03-19T11:30:59.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020077100000000001 2024-03-19T11:30:59.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:59.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.083401+0000", 2024-03-19T11:30:59.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002618333 2024-03-19T11:30:59.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:59.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085571+0000", 2024-03-19T11:30:59.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021702829999999998 2024-03-19T11:30:59.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:59.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085672+0000", 2024-03-19T11:30:59.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100876 2024-03-19T11:30:59.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:59.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085739+0000", 2024-03-19T11:30:59.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7552999999999995e-05 2024-03-19T11:30:59.257 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:59.257 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:59.257 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:59.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.257 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:30:59.257 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.071957+0000", 2024-03-19T11:30:59.257 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 74.170090274000003, 2024-03-19T11:30:59.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013989822000000001, 2024-03-19T11:30:59.258 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:59.258 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:59.258 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:59.258 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:59.258 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:59.258 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 36 2024-03-19T11:30:59.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.258 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:59.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:59.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071957+0000", 2024-03-19T11:30:59.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:59.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:59.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071957+0000", 2024-03-19T11:30:59.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:59.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:59.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071958+0000", 2024-03-19T11:30:59.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.06e-06 2024-03-19T11:30:59.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:59.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071965+0000", 2024-03-19T11:30:59.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8929999999999999e-06 2024-03-19T11:30:59.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:59.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071976+0000", 2024-03-19T11:30:59.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1046999999999999e-05 2024-03-19T11:30:59.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.261 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:59.261 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071984+0000", 2024-03-19T11:30:59.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7170000000000008e-06 2024-03-19T11:30:59.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.261 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:59.261 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.081761+0000", 2024-03-19T11:30:59.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0097769310000000009 2024-03-19T11:30:59.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.261 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:59.261 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.081928+0000", 2024-03-19T11:30:59.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000167219 2024-03-19T11:30:59.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.262 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:59.262 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.082131+0000", 2024-03-19T11:30:59.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020244499999999999 2024-03-19T11:30:59.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.262 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:59.262 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.083455+0000", 2024-03-19T11:30:59.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001324073 2024-03-19T11:30:59.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.262 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:59.262 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085816+0000", 2024-03-19T11:30:59.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0023613309999999999 2024-03-19T11:30:59.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.263 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:59.263 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085892+0000", 2024-03-19T11:30:59.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6154000000000003e-05 2024-03-19T11:30:59.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.263 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:59.263 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085947+0000", 2024-03-19T11:30:59.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4951999999999998e-05 2024-03-19T11:30:59.263 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:59.263 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:59.263 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:59.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.264 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:30:59.264 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.072000+0000", 2024-03-19T11:30:59.264 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 74.170047523999997, 2024-03-19T11:30:59.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014418268999999999, 2024-03-19T11:30:59.264 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:59.264 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:59.264 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:59.264 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:59.264 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:59.264 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 37 2024-03-19T11:30:59.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.264 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:59.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:59.265 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072000+0000", 2024-03-19T11:30:59.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:59.265 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.265 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.265 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:59.265 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072000+0000", 2024-03-19T11:30:59.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:59.265 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.265 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.265 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:59.265 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072001+0000", 2024-03-19T11:30:59.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0699999999999996e-07 2024-03-19T11:30:59.265 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.265 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.265 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:59.266 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072007+0000", 2024-03-19T11:30:59.266 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2060000000000004e-06 2024-03-19T11:30:59.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.266 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:59.266 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072023+0000", 2024-03-19T11:30:59.266 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6617e-05 2024-03-19T11:30:59.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.266 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:59.266 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072031+0000", 2024-03-19T11:30:59.266 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8900000000000007e-06 2024-03-19T11:30:59.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.267 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:59.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.082682+0000", 2024-03-19T11:30:59.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010650868000000001 2024-03-19T11:30:59.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.267 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:59.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.082804+0000", 2024-03-19T11:30:59.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012202899999999999 2024-03-19T11:30:59.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.267 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:59.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.083006+0000", 2024-03-19T11:30:59.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020127799999999999 2024-03-19T11:30:59.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.268 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.268 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:59.268 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.084145+0000", 2024-03-19T11:30:59.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001139791 2024-03-19T11:30:59.268 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.268 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.268 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:59.268 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.086292+0000", 2024-03-19T11:30:59.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021461969999999999 2024-03-19T11:30:59.268 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.268 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.268 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:59.268 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.086364+0000", 2024-03-19T11:30:59.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2439000000000002e-05 2024-03-19T11:30:59.269 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.269 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.269 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:59.269 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.086418+0000", 2024-03-19T11:30:59.269 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4246999999999997e-05 2024-03-19T11:30:59.269 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:59.269 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:59.269 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:59.269 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.269 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.269 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:30:59.269 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.339898+0000", 2024-03-19T11:30:59.269 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 73.902149902000005, 2024-03-19T11:30:59.269 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019501793, 2024-03-19T11:30:59.269 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:59.270 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:59.270 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:59.270 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:59.270 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:59.271 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113 2024-03-19T11:30:59.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.271 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:59.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:59.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339898+0000", 2024-03-19T11:30:59.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:59.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:59.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339898+0000", 2024-03-19T11:30:59.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:59.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:59.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339899+0000", 2024-03-19T11:30:59.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.353e-06 2024-03-19T11:30:59.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:59.273 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339907+0000", 2024-03-19T11:30:59.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3000000000000002e-06 2024-03-19T11:30:59.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:59.273 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339924+0000", 2024-03-19T11:30:59.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6906999999999999e-05 2024-03-19T11:30:59.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.274 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:59.274 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339933+0000", 2024-03-19T11:30:59.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8969999999999997e-06 2024-03-19T11:30:59.274 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.274 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.274 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:59.274 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352239+0000", 2024-03-19T11:30:59.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012305814999999999 2024-03-19T11:30:59.274 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.274 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.274 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:59.274 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352466+0000", 2024-03-19T11:30:59.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022684100000000001 2024-03-19T11:30:59.274 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:59.275 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352816+0000", 2024-03-19T11:30:59.275 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00035030499999999997 2024-03-19T11:30:59.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:59.275 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.358956+0000", 2024-03-19T11:30:59.275 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061400079999999998 2024-03-19T11:30:59.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:59.275 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359118+0000", 2024-03-19T11:30:59.275 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016237599999999999 2024-03-19T11:30:59.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:59.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359287+0000", 2024-03-19T11:30:59.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000168488 2024-03-19T11:30:59.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:59.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359399+0000", 2024-03-19T11:30:59.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000112503 2024-03-19T11:30:59.276 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:59.276 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:59.276 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:59.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.277 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:30:59.277 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.339947+0000", 2024-03-19T11:30:59.277 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 73.902100134999998, 2024-03-19T11:30:59.277 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021923069999999999, 2024-03-19T11:30:59.277 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:59.277 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:59.277 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:59.277 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:59.277 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:59.277 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 114 2024-03-19T11:30:59.277 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.277 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:59.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.277 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:59.277 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339947+0000", 2024-03-19T11:30:59.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:59.278 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.278 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.278 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:59.278 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339947+0000", 2024-03-19T11:30:59.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:59.278 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.278 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.278 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:59.278 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339948+0000", 2024-03-19T11:30:59.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6000000000000003e-07 2024-03-19T11:30:59.278 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.278 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.278 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:59.278 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339956+0000", 2024-03-19T11:30:59.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1559999999999995e-06 2024-03-19T11:30:59.279 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.279 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:59.279 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339967+0000", 2024-03-19T11:30:59.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0669999999999999e-05 2024-03-19T11:30:59.279 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.279 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:59.279 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339975+0000", 2024-03-19T11:30:59.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0099999999999995e-06 2024-03-19T11:30:59.279 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.279 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:59.279 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.353672+0000", 2024-03-19T11:30:59.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013696583999999999 2024-03-19T11:30:59.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.280 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:59.280 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.353808+0000", 2024-03-19T11:30:59.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013652299999999999 2024-03-19T11:30:59.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.280 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:59.280 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.354036+0000", 2024-03-19T11:30:59.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022744799999999999 2024-03-19T11:30:59.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.280 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:59.280 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.360480+0000", 2024-03-19T11:30:59.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0064441630000000001 2024-03-19T11:30:59.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:59.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361722+0000", 2024-03-19T11:30:59.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001242754 2024-03-19T11:30:59.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:59.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361807+0000", 2024-03-19T11:30:59.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4605999999999996e-05 2024-03-19T11:30:59.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:59.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361870+0000", 2024-03-19T11:30:59.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3396000000000001e-05 2024-03-19T11:30:59.282 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:59.282 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:59.282 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:59.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.282 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:30:59.282 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.340017+0000", 2024-03-19T11:30:59.282 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 73.902030819000004, 2024-03-19T11:30:59.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022169355000000002, 2024-03-19T11:30:59.282 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:59.282 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:59.282 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:59.283 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:59.283 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:59.283 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 115 2024-03-19T11:30:59.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.283 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:59.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:59.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340017+0000", 2024-03-19T11:30:59.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:59.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:59.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340017+0000", 2024-03-19T11:30:59.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:59.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.284 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:59.284 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340018+0000", 2024-03-19T11:30:59.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.043e-06 2024-03-19T11:30:59.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.284 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:59.284 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340139+0000", 2024-03-19T11:30:59.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121502 2024-03-19T11:30:59.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.284 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:59.284 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340157+0000", 2024-03-19T11:30:59.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7382999999999999e-05 2024-03-19T11:30:59.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:59.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340166+0000", 2024-03-19T11:30:59.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2469999999999997e-06 2024-03-19T11:30:59.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:59.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.357470+0000", 2024-03-19T11:30:59.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017304119999999999 2024-03-19T11:30:59.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:59.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.357671+0000", 2024-03-19T11:30:59.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000200798 2024-03-19T11:30:59.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:59.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.358002+0000", 2024-03-19T11:30:59.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033090099999999998 2024-03-19T11:30:59.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:59.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361892+0000", 2024-03-19T11:30:59.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00389018 2024-03-19T11:30:59.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:59.287 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361983+0000", 2024-03-19T11:30:59.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0976000000000004e-05 2024-03-19T11:30:59.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.287 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:59.287 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362071+0000", 2024-03-19T11:30:59.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8022999999999994e-05 2024-03-19T11:30:59.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.287 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:59.287 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362186+0000", 2024-03-19T11:30:59.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115182 2024-03-19T11:30:59.287 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:59.287 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:59.287 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:59.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.288 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:30:59.288 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.340183+0000", 2024-03-19T11:30:59.288 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 73.901863986999999, 2024-03-19T11:30:59.288 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022279841000000002, 2024-03-19T11:30:59.288 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:59.288 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:59.288 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:59.288 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:59.288 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:59.288 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 116 2024-03-19T11:30:59.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.288 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:59.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:59.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340183+0000", 2024-03-19T11:30:59.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:59.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:59.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340183+0000", 2024-03-19T11:30:59.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:59.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:59.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340184+0000", 2024-03-19T11:30:59.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9000000000000005e-07 2024-03-19T11:30:59.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:59.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340191+0000", 2024-03-19T11:30:59.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0360000000000003e-06 2024-03-19T11:30:59.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:59.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340201+0000", 2024-03-19T11:30:59.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0844e-05 2024-03-19T11:30:59.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:59.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340209+0000", 2024-03-19T11:30:59.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7360000000000001e-06 2024-03-19T11:30:59.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:59.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359535+0000", 2024-03-19T11:30:59.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019325546999999998 2024-03-19T11:30:59.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.292 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:59.292 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359672+0000", 2024-03-19T11:30:59.292 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013706600000000001 2024-03-19T11:30:59.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.292 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:59.292 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359905+0000", 2024-03-19T11:30:59.292 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023334500000000001 2024-03-19T11:30:59.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.292 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:59.292 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362306+0000", 2024-03-19T11:30:59.292 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002401041 2024-03-19T11:30:59.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.293 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:59.293 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362329+0000", 2024-03-19T11:30:59.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3283999999999999e-05 2024-03-19T11:30:59.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.293 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:59.293 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362402+0000", 2024-03-19T11:30:59.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2459000000000005e-05 2024-03-19T11:30:59.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.293 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:59.293 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362463+0000", 2024-03-19T11:30:59.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1493000000000002e-05 2024-03-19T11:30:59.293 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:59.293 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:59.294 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:59.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.294 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:30:59.294 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.340222+0000", 2024-03-19T11:30:59.294 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 73.901825403999993, 2024-03-19T11:30:59.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022500218999999998, 2024-03-19T11:30:59.294 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:59.294 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:59.294 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:30:59.294 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:30:59.294 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:30:59.294 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 117 2024-03-19T11:30:59.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.295 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:59.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:59.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340222+0000", 2024-03-19T11:30:59.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:59.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:59.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340222+0000", 2024-03-19T11:30:59.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:59.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:59.296 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340223+0000", 2024-03-19T11:30:59.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7999999999999995e-07 2024-03-19T11:30:59.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.296 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:59.296 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340229+0000", 2024-03-19T11:30:59.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0399999999999998e-06 2024-03-19T11:30:59.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.296 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:59.296 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340239+0000", 2024-03-19T11:30:59.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0190000000000001e-05 2024-03-19T11:30:59.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.296 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:59.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340250+0000", 2024-03-19T11:30:59.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0720000000000001e-05 2024-03-19T11:30:59.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:59.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.360587+0000", 2024-03-19T11:30:59.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020337542 2024-03-19T11:30:59.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:59.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.360818+0000", 2024-03-19T11:30:59.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023120500000000001 2024-03-19T11:30:59.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:30:59.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361218+0000", 2024-03-19T11:30:59.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000399637 2024-03-19T11:30:59.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:30:59.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362483+0000", 2024-03-19T11:30:59.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012649200000000001 2024-03-19T11:30:59.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:59.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362538+0000", 2024-03-19T11:30:59.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4490000000000002e-05 2024-03-19T11:30:59.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:59.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362617+0000", 2024-03-19T11:30:59.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9839000000000006e-05 2024-03-19T11:30:59.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:59.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362722+0000", 2024-03-19T11:30:59.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104956 2024-03-19T11:30:59.299 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:59.299 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:59.299 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:59.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.299 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1047 4.8 e35/21 4:1d58db11:::smithi03829856-18:head v 35'32, mlcod=33'29)", 2024-03-19T11:30:59.300 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:13.225023+0000", 2024-03-19T11:30:59.300 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 46.017024618999997, 2024-03-19T11:30:59.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013059688999999999, 2024-03-19T11:30:59.300 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:59.300 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:59.300 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:59.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.300 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:59.300 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.225023+0000", 2024-03-19T11:30:59.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:59.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.300 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:59.300 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.225023+0000", 2024-03-19T11:30:59.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:59.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.301 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:59.301 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.225025+0000", 2024-03-19T11:30:59.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.17e-06 2024-03-19T11:30:59.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.301 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:59.301 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.225771+0000", 2024-03-19T11:30:59.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000746267 2024-03-19T11:30:59.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.301 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:59.301 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.225807+0000", 2024-03-19T11:30:59.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5487000000000001e-05 2024-03-19T11:30:59.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:59.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.225823+0000", 2024-03-19T11:30:59.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6215999999999999e-05 2024-03-19T11:30:59.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:59.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.225990+0000", 2024-03-19T11:30:59.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016680899999999999 2024-03-19T11:30:59.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:59.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.226058+0000", 2024-03-19T11:30:59.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8672999999999998e-05 2024-03-19T11:30:59.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:59.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.237940+0000", 2024-03-19T11:30:59.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011881311 2024-03-19T11:30:59.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:59.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.238083+0000", 2024-03-19T11:30:59.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014275600000000001 2024-03-19T11:30:59.303 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:59.303 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:59.303 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:59.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.304 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1067 4.0 e36/21 4:01018511:::smithi03829856-19:head v 36'18, mlcod=35'14)", 2024-03-19T11:30:59.304 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:13.952492+0000", 2024-03-19T11:30:59.304 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 45.289555161999999, 2024-03-19T11:30:59.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013554725, 2024-03-19T11:30:59.304 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:59.304 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:59.304 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:59.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:59.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.952492+0000", 2024-03-19T11:30:59.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:59.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.305 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:59.305 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.952492+0000", 2024-03-19T11:30:59.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:59.305 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.305 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:59.305 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.952493+0000", 2024-03-19T11:30:59.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9299999999999997e-07 2024-03-19T11:30:59.305 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.305 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:59.305 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.954109+0000", 2024-03-19T11:30:59.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0016163569999999999 2024-03-19T11:30:59.305 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:59.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.954135+0000", 2024-03-19T11:30:59.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5352999999999999e-05 2024-03-19T11:30:59.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:59.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.954149+0000", 2024-03-19T11:30:59.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3879999999999999e-05 2024-03-19T11:30:59.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:59.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.954288+0000", 2024-03-19T11:30:59.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000139476 2024-03-19T11:30:59.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:59.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.954356+0000", 2024-03-19T11:30:59.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7533000000000005e-05 2024-03-19T11:30:59.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:59.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.965391+0000", 2024-03-19T11:30:59.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011035764999999999 2024-03-19T11:30:59.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:59.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.966047+0000", 2024-03-19T11:30:59.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00065556800000000004 2024-03-19T11:30:59.307 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:59.308 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:59.308 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:59.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.308 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1122 4.8 e36/21 4:1d58db11:::smithi03829856-18:head v 36'42, mlcod=36'39)", 2024-03-19T11:30:59.308 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:14.172666+0000", 2024-03-19T11:30:59.308 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 45.069381387999996, 2024-03-19T11:30:59.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018015000999999999, 2024-03-19T11:30:59.308 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:59.308 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:59.308 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:59.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:59.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.172666+0000", 2024-03-19T11:30:59.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:59.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.309 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:59.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.172666+0000", 2024-03-19T11:30:59.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:59.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.309 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:59.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.172669+0000", 2024-03-19T11:30:59.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1470000000000001e-06 2024-03-19T11:30:59.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:59.310 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.173140+0000", 2024-03-19T11:30:59.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00047074200000000001 2024-03-19T11:30:59.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:59.310 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.173188+0000", 2024-03-19T11:30:59.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8287e-05 2024-03-19T11:30:59.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:59.310 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.173211+0000", 2024-03-19T11:30:59.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3023e-05 2024-03-19T11:30:59.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:59.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.173378+0000", 2024-03-19T11:30:59.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000166742 2024-03-19T11:30:59.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:59.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.173557+0000", 2024-03-19T11:30:59.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000178569 2024-03-19T11:30:59.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:59.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.190416+0000", 2024-03-19T11:30:59.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016859570000000001 2024-03-19T11:30:59.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:59.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.190681+0000", 2024-03-19T11:30:59.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026492099999999999 2024-03-19T11:30:59.312 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:59.312 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:59.312 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:59.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.312 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.5.0:43 4.8 e39/21 4:15708482:::smithi03829856-23:head v 39'50, mlcod=38'47)", 2024-03-19T11:30:59.312 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.842819+0000", 2024-03-19T11:30:59.312 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 42.399228884999999, 2024-03-19T11:30:59.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017626513, 2024-03-19T11:30:59.312 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:59.313 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:59.313 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:59.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:59.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842819+0000", 2024-03-19T11:30:59.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:59.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:59.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842819+0000", 2024-03-19T11:30:59.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:59.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:59.314 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842820+0000", 2024-03-19T11:30:59.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.39e-06 2024-03-19T11:30:59.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.314 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:59.314 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.846437+0000", 2024-03-19T11:30:59.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003616953 2024-03-19T11:30:59.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.314 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:59.314 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.846471+0000", 2024-03-19T11:30:59.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4133e-05 2024-03-19T11:30:59.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:59.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.846539+0000", 2024-03-19T11:30:59.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8072999999999997e-05 2024-03-19T11:30:59.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:59.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.847094+0000", 2024-03-19T11:30:59.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00055472099999999999 2024-03-19T11:30:59.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:59.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.847128+0000", 2024-03-19T11:30:59.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3875999999999999e-05 2024-03-19T11:30:59.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:59.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.860308+0000", 2024-03-19T11:30:59.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013180712000000001 2024-03-19T11:30:59.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:59.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.860445+0000", 2024-03-19T11:30:59.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013665500000000001 2024-03-19T11:30:59.316 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:59.316 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:59.316 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:59.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.316 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1209 4.8 e41/21 4:1d58db11:::smithi03829856-18:head v 41'53, mlcod=39'50)", 2024-03-19T11:30:59.316 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:18.124878+0000", 2024-03-19T11:30:59.317 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 41.117169191999999, 2024-03-19T11:30:59.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016809343000000001, 2024-03-19T11:30:59.317 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:59.317 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:59.317 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:59.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:59.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.124878+0000", 2024-03-19T11:30:59.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:59.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:59.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.124878+0000", 2024-03-19T11:30:59.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:59.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.318 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:59.318 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.124880+0000", 2024-03-19T11:30:59.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4959999999999999e-06 2024-03-19T11:30:59.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.318 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:59.318 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125845+0000", 2024-03-19T11:30:59.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00096535299999999996 2024-03-19T11:30:59.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.318 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:59.318 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125884+0000", 2024-03-19T11:30:59.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8645e-05 2024-03-19T11:30:59.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:59.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125909+0000", 2024-03-19T11:30:59.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4783e-05 2024-03-19T11:30:59.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:59.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.126085+0000", 2024-03-19T11:30:59.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000176649 2024-03-19T11:30:59.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:59.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.126150+0000", 2024-03-19T11:30:59.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4290999999999998e-05 2024-03-19T11:30:59.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:30:59.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.141552+0000", 2024-03-19T11:30:59.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015402295 2024-03-19T11:30:59.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:59.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.141688+0000", 2024-03-19T11:30:59.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013583099999999999 2024-03-19T11:30:59.320 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:59.320 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:59.320 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:59.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.321 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.15 51/50 [PushOp(4:afa9f208:::smithi03829856-35:head, version: 28'46, data_included: [571438~623878,1806623~424698], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:afa9f208:::smithi03829856-35:head@28'46, size: 2571438, copy_subset: [(0, 2571438)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2231321, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:af9d0d56:::smithi03829856-20:head, version: 35'55, data_included: [655360~589824,1900544~418789], data_size: 1008613, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:af9d0d56:::smithi03829856-20:head@35'55, size: 2319333, copy_subset: [(0, 2319333)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2319333, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ad3ad1b3:::smithi03829856-21:b, version: 47'66, data_included: [736140~537686,2020581~315559], data_size: 853245, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:ad3ad1b3:::smithi03829856-21:b@47'66, size: 2336140, copy_subset: [(0, 2336140)], clone_subset: {}, snapset: b=[]:{b: [b, a, 6, 5]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2336140, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ad3ad1b3:::smithi03829856-21:head, version: 47'70, data_included: [531121~742705,1711057~305871], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:ad3ad1b3:::smithi03829856-21:head@47'70, size: 3731121, copy_subset: [(0, 3731121)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2016928, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ac9cd3a3:::smithi03829856-8:head, version: 49'72, data_included: [517649~620182,1776668~428394], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:ac9cd3a3:::smithi03829856-8:head@49'72, size: 3008420, copy_subset: [(0, 3008420)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2205062, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:aa3303a5:::smithi03829856-17:b, version: 49'73, data_included: [614400~692224,1903006~356352], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:aa3303a5:::smithi03829856-17:b@49'73, size: 2268912, copy_subset: [(0, 2268912)], clone_subset: {}, snapset: b=[]:{b: [b, a, 6, 5]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2259358, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:aa3303a5:::smithi03829856-17:head, version: 49'74, data_included: [413472~893152,1818036~155424], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:aa3303a5:::smithi03829856-17:head@49'74, size: 3613472, copy_subset: [(0, 3613472)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1973460, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:30:59.321 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:28.058610+0000", 2024-03-19T11:30:59.321 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 31.183437231999999, 2024-03-19T11:30:59.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027100032, 2024-03-19T11:30:59.321 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:59.321 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:59.321 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:59.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:59.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.058610+0000", 2024-03-19T11:30:59.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:59.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:59.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.058610+0000", 2024-03-19T11:30:59.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:59.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:59.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.058611+0000", 2024-03-19T11:30:59.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1599999999999999e-06 2024-03-19T11:30:59.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:59.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.063874+0000", 2024-03-19T11:30:59.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052625270000000004 2024-03-19T11:30:59.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.323 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:59.323 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.063935+0000", 2024-03-19T11:30:59.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0594000000000003e-05 2024-03-19T11:30:59.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.323 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:59.323 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064024+0000", 2024-03-19T11:30:59.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9273999999999999e-05 2024-03-19T11:30:59.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.323 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:59.323 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064385+0000", 2024-03-19T11:30:59.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00036129100000000002 2024-03-19T11:30:59.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:59.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064437+0000", 2024-03-19T11:30:59.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2346000000000003e-05 2024-03-19T11:30:59.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:59.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085710+0000", 2024-03-19T11:30:59.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021272840000000001 2024-03-19T11:30:59.324 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:59.324 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:59.324 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:59.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.324 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.15 e51,50 4:ac9cd3a3:::smithi03829856-8:4,49'71)", 2024-03-19T11:30:59.324 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:28.064112+0000", 2024-03-19T11:30:59.325 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 31.177935406, 2024-03-19T11:30:59.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026480417999999999, 2024-03-19T11:30:59.325 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:59.325 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:59.325 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:59.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:59.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064112+0000", 2024-03-19T11:30:59.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:59.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:59.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064112+0000", 2024-03-19T11:30:59.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:59.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:59.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064113+0000", 2024-03-19T11:30:59.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3600000000000003e-07 2024-03-19T11:30:59.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:59.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064118+0000", 2024-03-19T11:30:59.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0549999999999999e-06 2024-03-19T11:30:59.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:59.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064128+0000", 2024-03-19T11:30:59.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0349e-05 2024-03-19T11:30:59.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:59.327 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064136+0000", 2024-03-19T11:30:59.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6249999999999998e-06 2024-03-19T11:30:59.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:59.327 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085773+0000", 2024-03-19T11:30:59.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021636829999999999 2024-03-19T11:30:59.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:59.327 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085788+0000", 2024-03-19T11:30:59.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5512999999999999e-05 2024-03-19T11:30:59.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:59.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.090592+0000", 2024-03-19T11:30:59.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0048044100000000003 2024-03-19T11:30:59.328 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:59.328 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:59.328 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:59.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.328 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.5 51/50 [PushOp(4:a38b9453:::smithi03829856-37:head, version: 28'45, data_included: [605052~774376,1923587~274200], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:a38b9453:::smithi03829856-37:head@28'45, size: 2605052, copy_subset: [(0, 2605052)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2197787, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:a0a50d04:::smithi03829856-13:b, version: 47'63, data_included: [601684~619341,1669078~429235], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a0a50d04:::smithi03829856-13:b@47'63, size: 2201684, copy_subset: [(0, 2201684)], clone_subset: {}, snapset: b=[]:{b: [b, a, 6, 5]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2098313, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:30:59.328 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:28.064324+0000", 2024-03-19T11:30:59.328 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 31.177723562000001, 2024-03-19T11:30:59.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027179020000000002, 2024-03-19T11:30:59.328 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:59.329 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:59.329 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:59.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:59.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064324+0000", 2024-03-19T11:30:59.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:59.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:59.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064324+0000", 2024-03-19T11:30:59.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:59.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:59.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064324+0000", 2024-03-19T11:30:59.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1199999999999998e-07 2024-03-19T11:30:59.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:59.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.065527+0000", 2024-03-19T11:30:59.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012028939999999999 2024-03-19T11:30:59.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:59.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.065552+0000", 2024-03-19T11:30:59.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4440999999999999e-05 2024-03-19T11:30:59.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:59.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.065568+0000", 2024-03-19T11:30:59.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6498e-05 2024-03-19T11:30:59.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.331 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:59.331 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085812+0000", 2024-03-19T11:30:59.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020243954000000002 2024-03-19T11:30:59.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.331 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:59.331 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085825+0000", 2024-03-19T11:30:59.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3298999999999999e-05 2024-03-19T11:30:59.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.331 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:59.331 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.091503+0000", 2024-03-19T11:30:59.332 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.005677522 2024-03-19T11:30:59.332 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:59.332 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:59.332 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:59.332 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.332 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.24 75/74 [PushOp(4:27909373:::smithi03829856-46:8, version: 46'59, data_included: [756315~767970,1933572~280606], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:8@46'59, size: 2756315, copy_subset: [(0, 2756315)], clone_subset: {}, snapset: 10=[]:{8: [6], c: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2214178, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:266590a5:::smithi03829856-16:e, version: 70'107, data_included: [652105~662359,1728171~386217], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:e@70'107, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 10=[]:{e: [d, b, 6]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:266590a5:::smithi03829856-16:head, version: 70'108, data_included: [652105~662359,1728171~386217], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:head@70'108, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:27909373:::smithi03829856-46:c, version: 70'109, data_included: [628595~895690,1807440~152886], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:c@70'109, size: 3428595, copy_subset: [(0, 3428595)], clone_subset: {}, snapset: 10=[]:{8: [6], c: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1960326, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:27909373:::smithi03829856-46:head, version: 70'110, data_included: [403401~454961,1584925~506039], data_size: 961000, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:head@70'110, size: 2403401, copy_subset: [(0, 2403401)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2403401, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:30:59.332 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.099172+0000", 2024-03-19T11:30:59.332 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.142875128, 2024-03-19T11:30:59.332 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.024528580000000001, 2024-03-19T11:30:59.332 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:59.332 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:59.332 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:59.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.332 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:59.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.099172+0000", 2024-03-19T11:30:59.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:59.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:59.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.099172+0000", 2024-03-19T11:30:59.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:59.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:59.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.099175+0000", 2024-03-19T11:30:59.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7750000000000001e-06 2024-03-19T11:30:59.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:59.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105219+0000", 2024-03-19T11:30:59.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0060440789999999999 2024-03-19T11:30:59.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:59.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105285+0000", 2024-03-19T11:30:59.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6260000000000006e-05 2024-03-19T11:30:59.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:59.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105366+0000", 2024-03-19T11:30:59.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.038e-05 2024-03-19T11:30:59.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.335 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:59.335 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105637+0000", 2024-03-19T11:30:59.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027085199999999997 2024-03-19T11:30:59.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.335 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:59.335 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105679+0000", 2024-03-19T11:30:59.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2088000000000001e-05 2024-03-19T11:30:59.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.335 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:59.335 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.123701+0000", 2024-03-19T11:30:59.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018022145999999999 2024-03-19T11:30:59.335 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:59.335 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:59.336 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:59.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.336 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.24 e75,74 4:266590a5:::smithi03829856-16:4,47'63,4:27909373:::smithi03829856-46:5,55'79,4:266590a5:::smithi03829856-16:10,67'105)", 2024-03-19T11:30:59.336 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.105437+0000", 2024-03-19T11:30:59.336 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.1366101249999998, 2024-03-19T11:30:59.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.01860726, 2024-03-19T11:30:59.336 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:30:59.336 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:30:59.336 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:30:59.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.336 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:30:59.336 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105437+0000", 2024-03-19T11:30:59.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:59.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:30:59.337 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105437+0000", 2024-03-19T11:30:59.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:30:59.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:30:59.337 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105438+0000", 2024-03-19T11:30:59.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6799999999999999e-07 2024-03-19T11:30:59.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:30:59.337 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105445+0000", 2024-03-19T11:30:59.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3749999999999999e-06 2024-03-19T11:30:59.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.338 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.338 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:30:59.338 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105454+0000", 2024-03-19T11:30:59.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7909999999999999e-06 2024-03-19T11:30:59.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.338 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.338 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:30:59.338 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105465+0000", 2024-03-19T11:30:59.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.045e-05 2024-03-19T11:30:59.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.338 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.338 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:30:59.338 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.123753+0000", 2024-03-19T11:30:59.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0182884 2024-03-19T11:30:59.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:30:59.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.123765+0000", 2024-03-19T11:30:59.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2143e-05 2024-03-19T11:30:59.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:30:59.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:30:59.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:30:59.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.124045+0000", 2024-03-19T11:30:59.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027923299999999999 2024-03-19T11:30:59.339 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:59.339 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:59.339 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:59.339 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:30:59.339 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:30:59.339 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:30:59.342 INFO:tasks.rados.rados.0.smithi038.stdout:1169: finishing set_chunk to oid 10 2024-03-19T11:30:59.342 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 10 set_chunk returned error code -95 offset: 832512 length: 818688 2024-03-19T11:30:59.342 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 10 v 0 (ObjNum 179 snap 19 seq_num 179) dirty exists 2024-03-19T11:30:59.342 INFO:tasks.rados.rados.0.smithi038.stdout:1171: finishing rollback tid 0 to smithi03829856-5 2024-03-19T11:30:59.342 INFO:tasks.rados.rados.0.smithi038.stdout:1170: finishing set_chunk to oid 9 2024-03-19T11:30:59.342 INFO:tasks.rados.rados.0.smithi038.stdout:1171: finishing rollback tid 1 to smithi03829856-5 2024-03-19T11:30:59.342 INFO:tasks.rados.rados.0.smithi038.stdout:1171: finishing rollback tid 2 to smithi03829856-5 2024-03-19T11:30:59.342 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 5 v 105 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-03-19T11:30:59.342 INFO:tasks.rados.rados.0.smithi038.stdout:1168: done (3 left) 2024-03-19T11:30:59.342 INFO:tasks.rados.rados.0.smithi038.stdout:1169: done (2 left) 2024-03-19T11:30:59.342 INFO:tasks.rados.rados.0.smithi038.stdout:1170: done (1 left) 2024-03-19T11:30:59.342 INFO:tasks.rados.rados.0.smithi038.stdout:1171: done (0 left) 2024-03-19T11:30:59.342 INFO:tasks.rados.rados.0.smithi038.stdout:1172: set_chunk oid 26 target oid 2024-03-19T11:30:59.343 INFO:tasks.rados.rados.0.smithi038.stdout:1172: set_chunk oid 26 offset: 787456 length: 61440 target oid d3150c9ac21f7ce79ca7f92d9eed1f0d1c24ab39f7b68b837f784db4f891522f offset: 0 snap -1 2024-03-19T11:30:59.343 INFO:tasks.rados.rados.0.smithi038.stdout:1173: read oid 9 snap -1 2024-03-19T11:30:59.343 INFO:tasks.rados.rados.0.smithi038.stdout:1173: expect (ObjNum 152 snap 17 seq_num 152) 2024-03-19T11:30:59.343 INFO:tasks.rados.rados.0.smithi038.stdout:1174: set_chunk oid 22 target oid 2024-03-19T11:30:59.343 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-19T11:30:59.343 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:30:59.350+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:30:59.442 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:30:59.451+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:30:59.503 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:59.503 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:30:59.503 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:30:59.504 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:59.517 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-19T11:30:59.543 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:30:59.555+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:30:59.644 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:59.644 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:30:59.644 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:30:59.644 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:30:59.644 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:59.644 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:59.655+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:30:59.659 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-19T11:30:59.745 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:30:59.754+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:30:59.784 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:30:59.784 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:30:59.785 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:30:59.785 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:30:59.785 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.785 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1090 4.6 4:6d9093b1:::smithi03829856-38:head [sparse-read 0~1502675] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:30:59.785 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:14.006669+0000", 2024-03-19T11:30:59.785 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 45.785630288999997, 2024-03-19T11:30:59.785 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012590499999999999, 2024-03-19T11:30:59.785 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:59.785 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:59.785 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:59.785 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:59.785 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:59.785 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1090 2024-03-19T11:30:59.785 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.786 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:59.786 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.786 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:59.786 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006669+0000", 2024-03-19T11:30:59.786 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:59.786 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.786 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.786 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:59.786 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006669+0000", 2024-03-19T11:30:59.786 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:59.786 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.786 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.786 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:59.786 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006669+0000", 2024-03-19T11:30:59.787 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1800000000000001e-07 2024-03-19T11:30:59.787 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.787 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.787 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:59.787 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006673+0000", 2024-03-19T11:30:59.787 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.9530000000000001e-06 2024-03-19T11:30:59.787 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.787 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.787 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:59.787 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006680+0000", 2024-03-19T11:30:59.787 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9410000000000001e-06 2024-03-19T11:30:59.787 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.787 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.787 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:59.788 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006685+0000", 2024-03-19T11:30:59.788 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.327e-06 2024-03-19T11:30:59.788 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.788 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.788 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:59.788 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.018877+0000", 2024-03-19T11:30:59.788 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012191936 2024-03-19T11:30:59.788 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.788 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.788 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:59.788 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.018972+0000", 2024-03-19T11:30:59.788 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4754999999999997e-05 2024-03-19T11:30:59.788 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.788 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.789 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:59.789 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.019259+0000", 2024-03-19T11:30:59.789 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00028717 2024-03-19T11:30:59.789 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:59.789 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:59.789 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:59.789 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.789 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.789 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1217 4.3 4:c8b09e87:::smithi03829856-15:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:30:59.789 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.155966+0000", 2024-03-19T11:30:59.789 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 41.636333137999998, 2024-03-19T11:30:59.789 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.023984647000000001, 2024-03-19T11:30:59.789 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:59.789 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:59.789 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:59.790 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:59.790 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:59.790 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1217 2024-03-19T11:30:59.790 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.790 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:59.790 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.790 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:59.790 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155966+0000", 2024-03-19T11:30:59.790 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:59.790 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.790 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.790 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:59.790 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155966+0000", 2024-03-19T11:30:59.791 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:59.791 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.791 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.791 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:59.791 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155968+0000", 2024-03-19T11:30:59.791 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.795e-06 2024-03-19T11:30:59.791 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.791 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.791 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:59.791 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155978+0000", 2024-03-19T11:30:59.791 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0574000000000001e-05 2024-03-19T11:30:59.791 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.791 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.791 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:59.792 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.156000+0000", 2024-03-19T11:30:59.792 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1250000000000002e-05 2024-03-19T11:30:59.792 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.792 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.792 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:59.792 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.156010+0000", 2024-03-19T11:30:59.792 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0168e-05 2024-03-19T11:30:59.792 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.792 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.792 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:59.792 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.156119+0000", 2024-03-19T11:30:59.792 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010950699999999999 2024-03-19T11:30:59.792 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.792 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.793 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:59.793 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.156231+0000", 2024-03-19T11:30:59.793 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000111445 2024-03-19T11:30:59.793 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.793 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.793 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:59.793 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.179951+0000", 2024-03-19T11:30:59.793 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.023719908000000001 2024-03-19T11:30:59.794 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:59.794 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:59.794 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:59.794 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.794 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.794 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1432 4.13 4:cef44a0d:::smithi03829856-7:head [assert-version v18,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:30:59.794 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.977382+0000", 2024-03-19T11:30:59.794 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 33.814916865999997, 2024-03-19T11:30:59.794 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012855814, 2024-03-19T11:30:59.794 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:59.794 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:59.794 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:59.794 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:59.795 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:59.795 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1432 2024-03-19T11:30:59.795 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.795 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:59.795 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.795 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:59.795 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.977382+0000", 2024-03-19T11:30:59.795 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:59.795 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.795 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.795 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:59.795 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.977382+0000", 2024-03-19T11:30:59.795 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:59.795 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.795 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.796 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:59.796 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.977385+0000", 2024-03-19T11:30:59.796 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9380000000000001e-06 2024-03-19T11:30:59.796 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.796 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.796 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:59.796 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.977397+0000", 2024-03-19T11:30:59.796 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2136e-05 2024-03-19T11:30:59.796 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.796 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.796 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:59.796 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.977424+0000", 2024-03-19T11:30:59.796 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6917999999999999e-05 2024-03-19T11:30:59.796 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.797 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.797 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:59.797 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.977437+0000", 2024-03-19T11:30:59.797 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.242e-05 2024-03-19T11:30:59.797 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.797 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.797 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:59.797 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.977534+0000", 2024-03-19T11:30:59.797 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6974999999999999e-05 2024-03-19T11:30:59.797 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.797 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.797 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:59.797 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.977639+0000", 2024-03-19T11:30:59.797 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000105713 2024-03-19T11:30:59.797 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.798 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.798 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:59.798 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.987924+0000", 2024-03-19T11:30:59.798 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010284847999999999 2024-03-19T11:30:59.798 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.798 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.798 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:59.798 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.988536+0000", 2024-03-19T11:30:59.798 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00061207099999999995 2024-03-19T11:30:59.798 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.798 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.798 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:59.798 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.990098+0000", 2024-03-19T11:30:59.798 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001561523 2024-03-19T11:30:59.798 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.799 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.799 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:59.799 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.990176+0000", 2024-03-19T11:30:59.799 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8316e-05 2024-03-19T11:30:59.799 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.799 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.799 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:59.799 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.990238+0000", 2024-03-19T11:30:59.799 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1956000000000001e-05 2024-03-19T11:30:59.799 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:59.799 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:59.799 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:59.799 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.799 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.800 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1615 3.1 3:a1add8ca:::e7a0126b74590803e762ea9614dff8447a0272ff3000fd5c74e92cd9ee65bec5:head [write 0~2893824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-03-19T11:30:59.800 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:35.133510+0000", 2024-03-19T11:30:59.800 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 24.658789089999999, 2024-03-19T11:30:59.800 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012574642, 2024-03-19T11:30:59.800 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:59.800 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:59.800 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:59.800 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:59.800 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:59.800 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1615 2024-03-19T11:30:59.800 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.800 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:59.800 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.800 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:59.800 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.133510+0000", 2024-03-19T11:30:59.801 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:59.801 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.801 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.801 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:59.801 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.133510+0000", 2024-03-19T11:30:59.801 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:59.801 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.801 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.801 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:59.801 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.133514+0000", 2024-03-19T11:30:59.801 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6990000000000001e-06 2024-03-19T11:30:59.801 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.801 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.801 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:59.801 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.135072+0000", 2024-03-19T11:30:59.802 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0015580590000000001 2024-03-19T11:30:59.802 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.802 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.802 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:59.802 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.135107+0000", 2024-03-19T11:30:59.802 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5663000000000001e-05 2024-03-19T11:30:59.802 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.802 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.802 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:59.802 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.135122+0000", 2024-03-19T11:30:59.802 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4657e-05 2024-03-19T11:30:59.802 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.802 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.802 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:59.803 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.135211+0000", 2024-03-19T11:30:59.803 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9101e-05 2024-03-19T11:30:59.803 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.803 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.803 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-03-19T11:30:59.803 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.135222+0000", 2024-03-19T11:30:59.803 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0944999999999999e-05 2024-03-19T11:30:59.803 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.803 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.803 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:59.803 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.137108+0000", 2024-03-19T11:30:59.803 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0018854900000000001 2024-03-19T11:30:59.803 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.804 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.804 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:59.804 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.137204+0000", 2024-03-19T11:30:59.804 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6836999999999995e-05 2024-03-19T11:30:59.804 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.804 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.804 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3", 2024-03-19T11:30:59.804 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.137311+0000", 2024-03-19T11:30:59.804 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010635499999999999 2024-03-19T11:30:59.804 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.804 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.804 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:59.804 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.142840+0000", 2024-03-19T11:30:59.805 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055292010000000001 2024-03-19T11:30:59.805 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.805 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.805 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:59.805 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.145965+0000", 2024-03-19T11:30:59.805 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003125128 2024-03-19T11:30:59.805 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.805 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.805 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:59.805 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.146031+0000", 2024-03-19T11:30:59.805 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5529000000000002e-05 2024-03-19T11:30:59.805 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.805 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.805 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:59.805 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.146085+0000", 2024-03-19T11:30:59.806 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3977999999999997e-05 2024-03-19T11:30:59.806 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:59.806 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:59.806 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:59.806 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.806 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.806 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1788 4.23 4:c4406a0a:::smithi03829856-49:head [write 685752~611865,stat] snapc 10=[10,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-03-19T11:30:59.806 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:40.165160+0000", 2024-03-19T11:30:59.806 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 19.627139209999999, 2024-03-19T11:30:59.806 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014138061, 2024-03-19T11:30:59.806 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:59.806 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:59.806 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:59.806 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:59.807 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:59.807 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1788 2024-03-19T11:30:59.807 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.807 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:59.807 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.807 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:59.807 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.165160+0000", 2024-03-19T11:30:59.807 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:59.807 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.807 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.807 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:59.807 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.165160+0000", 2024-03-19T11:30:59.807 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:59.807 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.807 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.807 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:59.808 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.165163+0000", 2024-03-19T11:30:59.808 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.729e-06 2024-03-19T11:30:59.808 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.808 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.808 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:59.808 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.165455+0000", 2024-03-19T11:30:59.808 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00029193900000000001 2024-03-19T11:30:59.808 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.808 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.808 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:59.809 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.165474+0000", 2024-03-19T11:30:59.809 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9910000000000001e-05 2024-03-19T11:30:59.809 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.809 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.809 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:59.809 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.165483+0000", 2024-03-19T11:30:59.809 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9609999999999994e-06 2024-03-19T11:30:59.809 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.809 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.809 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:59.809 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.165638+0000", 2024-03-19T11:30:59.809 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015434300000000001 2024-03-19T11:30:59.809 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.809 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.809 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:59.810 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.166078+0000", 2024-03-19T11:30:59.810 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044032099999999997 2024-03-19T11:30:59.810 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.810 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.810 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:30:59.810 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.166669+0000", 2024-03-19T11:30:59.810 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00059135100000000005 2024-03-19T11:30:59.810 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.810 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.810 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:59.810 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.174542+0000", 2024-03-19T11:30:59.810 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0078724880000000004 2024-03-19T11:30:59.810 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.810 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.810 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:59.811 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.178949+0000", 2024-03-19T11:30:59.811 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0044075720000000002 2024-03-19T11:30:59.811 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.811 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.811 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:59.811 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.179110+0000", 2024-03-19T11:30:59.811 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016055900000000001 2024-03-19T11:30:59.811 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.811 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.811 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:59.811 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.179298+0000", 2024-03-19T11:30:59.811 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00018788799999999999 2024-03-19T11:30:59.811 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:59.811 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:59.812 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:59.812 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.812 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.812 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1834 4.1b 4:db3790a7:::smithi03829856-32:head [tier-promote] snapc 10=[10,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-03-19T11:30:59.812 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:40.957892+0000", 2024-03-19T11:30:59.812 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.834407334000002, 2024-03-19T11:30:59.812 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013588032, 2024-03-19T11:30:59.812 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:59.812 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:59.812 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:59.812 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:59.812 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:59.812 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1834 2024-03-19T11:30:59.812 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.957892+0000", 2024-03-19T11:30:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.957892+0000", 2024-03-19T11:30:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.957894+0000", 2024-03-19T11:30:59.814 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2699999999999999e-06 2024-03-19T11:30:59.814 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.814 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.814 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:59.814 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.957903+0000", 2024-03-19T11:30:59.814 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5520000000000005e-06 2024-03-19T11:30:59.814 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.814 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.814 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:59.814 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.957924+0000", 2024-03-19T11:30:59.814 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1531000000000001e-05 2024-03-19T11:30:59.814 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.814 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.814 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.957934+0000", 2024-03-19T11:30:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.009e-05 2024-03-19T11:30:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.957987+0000", 2024-03-19T11:30:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2772000000000002e-05 2024-03-19T11:30:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.958044+0000", 2024-03-19T11:30:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6715000000000002e-05 2024-03-19T11:30:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:59.816 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.971480+0000", 2024-03-19T11:30:59.816 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013436102 2024-03-19T11:30:59.816 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:59.816 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:59.816 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:59.816 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.816 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.816 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2047 4.1b 4:dccaff13:::smithi03829856-36:head [write 756348~784249] snapc 12=[12,11,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-03-19T11:30:59.816 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:46.278700+0000", 2024-03-19T11:30:59.816 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 13.51359953, 2024-03-19T11:30:59.816 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014428785, 2024-03-19T11:30:59.816 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:59.816 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:30:59.816 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:59.817 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:59.817 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:59.817 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2047 2024-03-19T11:30:59.817 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.817 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:59.817 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.817 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:59.817 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.278700+0000", 2024-03-19T11:30:59.817 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:59.817 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.817 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.817 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:59.817 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.278700+0000", 2024-03-19T11:30:59.817 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:59.817 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.818 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.818 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:59.818 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.278703+0000", 2024-03-19T11:30:59.818 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1690000000000001e-06 2024-03-19T11:30:59.818 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.818 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.818 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:59.818 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.278955+0000", 2024-03-19T11:30:59.818 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025264100000000001 2024-03-19T11:30:59.818 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.818 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.818 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:59.818 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.278982+0000", 2024-03-19T11:30:59.818 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6933000000000001e-05 2024-03-19T11:30:59.819 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.819 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.819 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:59.819 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.278997+0000", 2024-03-19T11:30:59.819 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4440999999999999e-05 2024-03-19T11:30:59.819 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.819 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.819 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:59.819 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.279383+0000", 2024-03-19T11:30:59.819 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00038659700000000003 2024-03-19T11:30:59.819 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.819 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.819 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-03-19T11:30:59.819 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.279397+0000", 2024-03-19T11:30:59.819 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3574000000000001e-05 2024-03-19T11:30:59.820 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.820 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.820 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:59.820 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.282908+0000", 2024-03-19T11:30:59.820 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0035110110000000001 2024-03-19T11:30:59.820 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.820 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.820 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:59.820 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.283059+0000", 2024-03-19T11:30:59.820 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015160799999999999 2024-03-19T11:30:59.820 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.820 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.820 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1", 2024-03-19T11:30:59.820 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.283194+0000", 2024-03-19T11:30:59.821 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013435399999999999 2024-03-19T11:30:59.821 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.821 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.821 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:30:59.821 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.292934+0000", 2024-03-19T11:30:59.821 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0097397539999999998 2024-03-19T11:30:59.821 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.821 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.821 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:30:59.821 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.293026+0000", 2024-03-19T11:30:59.821 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2195999999999996e-05 2024-03-19T11:30:59.821 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.821 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.821 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:30:59.821 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.293075+0000", 2024-03-19T11:30:59.822 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9484000000000003e-05 2024-03-19T11:30:59.822 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.822 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.822 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:59.822 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.293128+0000", 2024-03-19T11:30:59.822 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3022999999999997e-05 2024-03-19T11:30:59.822 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:59.822 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:59.822 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:59.822 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.822 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.822 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.24 75/74 [PushOp(4:27909373:::smithi03829856-46:8, version: 46'59, data_included: [756315~767970,1933572~280606], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:8@46'59, size: 2756315, copy_subset: [(0, 2756315)], clone_subset: {}, snapset: 10=[]:{8: [6], c: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2214178, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:266590a5:::smithi03829856-16:e, version: 70'107, data_included: [652105~662359,1728171~386217], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:e@70'107, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 10=[]:{e: [d, b, 6]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:266590a5:::smithi03829856-16:head, version: 70'108, data_included: [652105~662359,1728171~386217], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:head@70'108, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:27909373:::smithi03829856-46:c, version: 70'109, data_included: [628595~895690,1807440~152886], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:c@70'109, size: 3428595, copy_subset: [(0, 3428595)], clone_subset: {}, snapset: 10=[]:{8: [6], c: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1960326, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:27909373:::smithi03829856-46:head, version: 70'110, data_included: [403401~454961,1584925~506039], data_size: 961000, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:head@70'110, size: 2403401, copy_subset: [(0, 2403401)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2403401, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:30:59.822 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.099395+0000", 2024-03-19T11:30:59.822 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.6929039770000003, 2024-03-19T11:30:59.823 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.045391157000000001, 2024-03-19T11:30:59.823 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:59.823 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:59.823 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:59.823 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.823 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:59.823 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.099395+0000", 2024-03-19T11:30:59.823 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:59.823 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.823 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.823 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:59.823 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.099395+0000", 2024-03-19T11:30:59.823 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:59.823 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.824 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.824 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:59.824 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.099397+0000", 2024-03-19T11:30:59.824 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.691e-06 2024-03-19T11:30:59.824 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.824 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.824 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:59.824 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105009+0000", 2024-03-19T11:30:59.824 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0056121349999999999 2024-03-19T11:30:59.824 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.824 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.824 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:59.824 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105077+0000", 2024-03-19T11:30:59.824 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8561999999999998e-05 2024-03-19T11:30:59.824 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.825 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.825 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:59.825 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105133+0000", 2024-03-19T11:30:59.825 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5590000000000001e-05 2024-03-19T11:30:59.825 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.825 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.825 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:59.825 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105531+0000", 2024-03-19T11:30:59.825 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00039756200000000001 2024-03-19T11:30:59.825 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.825 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.825 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:59.825 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105579+0000", 2024-03-19T11:30:59.825 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8028999999999997e-05 2024-03-19T11:30:59.825 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.826 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.826 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:59.826 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.144786+0000", 2024-03-19T11:30:59.826 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.039207588000000002 2024-03-19T11:30:59.826 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:59.826 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:59.826 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:59.826 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.826 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.826 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.24 e75,74 4:266590a5:::smithi03829856-16:4,47'63,4:27909373:::smithi03829856-46:5,55'79,4:266590a5:::smithi03829856-16:10,67'105)", 2024-03-19T11:30:59.826 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.105191+0000", 2024-03-19T11:30:59.826 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.6871078769999999, 2024-03-19T11:30:59.826 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.039580830999999997, 2024-03-19T11:30:59.827 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:59.827 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:59.827 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:59.827 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.827 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:59.827 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105191+0000", 2024-03-19T11:30:59.827 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:59.827 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.827 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.827 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:59.827 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105191+0000", 2024-03-19T11:30:59.827 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:59.827 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.827 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.827 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:59.828 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105192+0000", 2024-03-19T11:30:59.828 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.099e-06 2024-03-19T11:30:59.828 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.828 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.828 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:59.828 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105200+0000", 2024-03-19T11:30:59.828 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5279999999999998e-06 2024-03-19T11:30:59.828 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.828 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.828 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:59.828 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105211+0000", 2024-03-19T11:30:59.828 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1471000000000001e-05 2024-03-19T11:30:59.828 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.828 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.828 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:59.829 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105220+0000", 2024-03-19T11:30:59.829 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3469999999999999e-06 2024-03-19T11:30:59.829 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.829 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.829 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:59.829 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.144219+0000", 2024-03-19T11:30:59.829 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.038999024 2024-03-19T11:30:59.829 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.829 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.829 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:59.829 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.144257+0000", 2024-03-19T11:30:59.829 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8501999999999998e-05 2024-03-19T11:30:59.829 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.829 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.829 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:59.829 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.144772+0000", 2024-03-19T11:30:59.830 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00051486000000000001 2024-03-19T11:30:59.830 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:59.830 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:59.830 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:59.830 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.830 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.830 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 75/74 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:7, version: 55'82, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@55'82, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 58'86, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@58'86, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [504210~968657,1572695~79919], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:12, version: 66'101, data_included: [440350~816629,1309629~231947], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:12@66'101, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:10, version: 70'112, data_included: [440350~816629,1309629~231947], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:10@70'112, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 70'113, data_included: [440350~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@70'113, size: 2605085, copy_subset: [(0, 2605085)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1488926, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:30:59.830 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.107266+0000", 2024-03-19T11:30:59.830 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.6850333600000003, 2024-03-19T11:30:59.830 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.048592669999999998, 2024-03-19T11:30:59.830 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:59.830 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:59.830 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:59.831 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.831 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:59.831 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.107266+0000", 2024-03-19T11:30:59.831 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:59.831 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.831 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.831 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:59.831 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.107266+0000", 2024-03-19T11:30:59.831 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:59.831 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.831 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.831 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:59.831 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.107267+0000", 2024-03-19T11:30:59.831 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1459999999999999e-06 2024-03-19T11:30:59.831 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.832 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.832 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:59.832 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.122861+0000", 2024-03-19T11:30:59.832 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01559372 2024-03-19T11:30:59.832 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.832 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.832 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:59.832 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.122932+0000", 2024-03-19T11:30:59.832 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1477999999999996e-05 2024-03-19T11:30:59.832 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.832 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.832 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:59.832 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.122999+0000", 2024-03-19T11:30:59.832 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6832e-05 2024-03-19T11:30:59.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.833 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.833 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:59.833 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123332+0000", 2024-03-19T11:30:59.833 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000333271 2024-03-19T11:30:59.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.833 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.833 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:59.833 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123371+0000", 2024-03-19T11:30:59.833 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8525999999999997e-05 2024-03-19T11:30:59.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.833 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.833 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:59.833 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.155858+0000", 2024-03-19T11:30:59.834 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.032487697000000003 2024-03-19T11:30:59.834 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:59.834 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:59.834 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:59.834 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.834 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.834 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.21 e75,74 4:8401c1f0:::smithi03829856-43:8,46'66,4:866f3dcc:::smithi03829856-31:2,47'73)", 2024-03-19T11:30:59.834 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.123075+0000", 2024-03-19T11:30:59.834 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.6692241409999999, 2024-03-19T11:30:59.834 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.033023045000000001, 2024-03-19T11:30:59.834 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:59.834 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:59.834 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:59.834 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.835 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:59.835 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123075+0000", 2024-03-19T11:30:59.835 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:59.835 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.835 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.835 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:59.835 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123075+0000", 2024-03-19T11:30:59.835 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:59.835 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.835 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.835 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:59.835 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123076+0000", 2024-03-19T11:30:59.835 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5e-06 2024-03-19T11:30:59.835 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.835 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.836 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:59.836 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123086+0000", 2024-03-19T11:30:59.836 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8060000000000006e-06 2024-03-19T11:30:59.836 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.836 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.836 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:59.836 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123099+0000", 2024-03-19T11:30:59.836 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3251000000000001e-05 2024-03-19T11:30:59.836 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.836 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.836 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:59.836 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123112+0000", 2024-03-19T11:30:59.836 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2774e-05 2024-03-19T11:30:59.836 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.837 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.837 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:59.837 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.155902+0000", 2024-03-19T11:30:59.837 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.032790225999999999 2024-03-19T11:30:59.837 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.837 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.837 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:59.837 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.155914+0000", 2024-03-19T11:30:59.837 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1476e-05 2024-03-19T11:30:59.837 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.837 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.837 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:59.837 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.156098+0000", 2024-03-19T11:30:59.837 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000184012 2024-03-19T11:30:59.838 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:59.838 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:59.838 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:59.838 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.838 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.838 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 75/74 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:7, version: 55'82, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@55'82, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 58'86, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@58'86, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [1652614~973634,2888511~52681,3182305~22261], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3204566, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:12, version: 66'101, data_included: [1541576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:12@66'101, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:10, version: 70'112, data_included: [1541576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:10@70'112, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 70'113, data_included: [1488926~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@70'113, size: 2605085, copy_subset: [(0, 2605085)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2537502, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1488926, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:30:59.838 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.174030+0000", 2024-03-19T11:30:59.838 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.6182689369999999, 2024-03-19T11:30:59.838 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.040799275000000003, 2024-03-19T11:30:59.838 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:59.838 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:59.838 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:59.839 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.839 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:59.839 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.174030+0000", 2024-03-19T11:30:59.839 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:59.839 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.839 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.839 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:59.839 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.174030+0000", 2024-03-19T11:30:59.839 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:59.839 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.839 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.839 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:59.839 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.174031+0000", 2024-03-19T11:30:59.839 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.322e-06 2024-03-19T11:30:59.839 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.839 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.840 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:59.840 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.184771+0000", 2024-03-19T11:30:59.840 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010739227 2024-03-19T11:30:59.840 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.840 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.840 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:59.840 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.184872+0000", 2024-03-19T11:30:59.840 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000101234 2024-03-19T11:30:59.840 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.840 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.840 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:59.840 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.184985+0000", 2024-03-19T11:30:59.840 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000113545 2024-03-19T11:30:59.840 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.841 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.841 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:59.841 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.185463+0000", 2024-03-19T11:30:59.841 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00047749899999999999 2024-03-19T11:30:59.841 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.841 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.841 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:59.841 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.185534+0000", 2024-03-19T11:30:59.841 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1007999999999995e-05 2024-03-19T11:30:59.841 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.841 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.841 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:59.841 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.214829+0000", 2024-03-19T11:30:59.841 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029295439999999999 2024-03-19T11:30:59.841 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:59.842 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:59.842 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:59.842 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.842 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.842 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.21 e75,74 4:866f3dcc:::smithi03829856-31:c,70'114)", 2024-03-19T11:30:59.842 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.197457+0000", 2024-03-19T11:30:59.842 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.5948418909999997, 2024-03-19T11:30:59.842 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018493590000000001, 2024-03-19T11:30:59.842 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:59.842 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:59.842 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:59.842 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.842 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:59.842 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.197457+0000", 2024-03-19T11:30:59.843 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:59.843 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.843 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.843 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:59.843 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.197457+0000", 2024-03-19T11:30:59.843 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:59.843 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.843 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.843 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:59.843 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.197459+0000", 2024-03-19T11:30:59.843 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3199999999999998e-06 2024-03-19T11:30:59.843 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.843 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.843 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:59.844 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.197480+0000", 2024-03-19T11:30:59.844 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0366000000000001e-05 2024-03-19T11:30:59.844 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.844 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.844 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:59.844 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.197521+0000", 2024-03-19T11:30:59.844 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0738000000000002e-05 2024-03-19T11:30:59.844 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.844 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.844 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:59.844 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.197547+0000", 2024-03-19T11:30:59.844 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6548999999999999e-05 2024-03-19T11:30:59.844 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.844 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.844 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:59.845 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.214954+0000", 2024-03-19T11:30:59.845 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01740713 2024-03-19T11:30:59.845 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.845 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.845 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:59.845 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.214972+0000", 2024-03-19T11:30:59.845 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7520000000000002e-05 2024-03-19T11:30:59.845 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.845 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.845 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:59.845 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.215951+0000", 2024-03-19T11:30:59.845 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00097896700000000007 2024-03-19T11:30:59.845 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:59.845 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:59.845 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:59.846 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.846 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.846 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 75/74 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [2599014~241336], data_size: 241336, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2840350, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [2599014~241336,2948041~446940], data_size: 688276, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3394981, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [3204566~621403], data_size: 621403, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3865577, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3204566, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:12, version: 66'101, data_included: [2590152~250198,2948041~581396], data_size: 831594, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:12@66'101, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3529437, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:10, version: 70'112, data_included: [2590152~250198,2948041~581396], data_size: 831594, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:10@70'112, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3529437, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 70'113, data_included: [2537502~67583], data_size: 67583, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@70'113, size: 2605085, copy_subset: [(0, 2605085)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2605085, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2537502, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:30:59.846 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.222872+0000", 2024-03-19T11:30:59.846 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.5694274440000004, 2024-03-19T11:30:59.846 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016277658, 2024-03-19T11:30:59.846 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:59.846 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:59.846 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:59.846 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.846 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:59.846 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.222872+0000", 2024-03-19T11:30:59.846 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:59.846 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.847 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.847 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:59.847 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.222872+0000", 2024-03-19T11:30:59.847 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:59.847 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.847 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.847 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:59.847 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.222874+0000", 2024-03-19T11:30:59.847 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9999999999999999e-06 2024-03-19T11:30:59.847 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.847 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.847 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:59.847 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.227563+0000", 2024-03-19T11:30:59.847 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046897570000000001 2024-03-19T11:30:59.847 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.848 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.848 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:59.848 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.227639+0000", 2024-03-19T11:30:59.848 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5607999999999998e-05 2024-03-19T11:30:59.848 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.848 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.848 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:59.848 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.227736+0000", 2024-03-19T11:30:59.848 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6854000000000005e-05 2024-03-19T11:30:59.848 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.848 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.848 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:59.848 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.228149+0000", 2024-03-19T11:30:59.848 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00041330000000000002 2024-03-19T11:30:59.848 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.848 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.849 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:59.849 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.228219+0000", 2024-03-19T11:30:59.849 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.0140000000000003e-05 2024-03-19T11:30:59.849 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.849 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.849 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:59.849 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.239149+0000", 2024-03-19T11:30:59.849 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010929998999999999 2024-03-19T11:30:59.849 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:59.849 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:59.849 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:59.849 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.849 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.849 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2372 4.16 4:6868af91:::smithi03829856-10:b [read 832512~818688] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-03-19T11:30:59.850 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:58.455732+0000", 2024-03-19T11:30:59.850 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 1.336567222, 2024-03-19T11:30:59.850 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.69150541200000004, 2024-03-19T11:30:59.850 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:59.850 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:59.850 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:30:59.850 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:30:59.850 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:30:59.850 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2372 2024-03-19T11:30:59.850 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.850 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:59.850 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.850 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:59.850 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455732+0000", 2024-03-19T11:30:59.851 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:59.851 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.851 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.851 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:59.851 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455732+0000", 2024-03-19T11:30:59.851 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:59.851 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.851 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.851 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:59.851 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455735+0000", 2024-03-19T11:30:59.851 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.756e-06 2024-03-19T11:30:59.851 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.851 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.851 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:59.851 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455744+0000", 2024-03-19T11:30:59.852 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1880000000000008e-06 2024-03-19T11:30:59.852 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.852 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.852 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:59.852 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455764+0000", 2024-03-19T11:30:59.852 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0044999999999999e-05 2024-03-19T11:30:59.852 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.852 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.852 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:59.852 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455774+0000", 2024-03-19T11:30:59.852 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8770000000000005e-06 2024-03-19T11:30:59.852 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.852 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.852 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:59.852 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455837+0000", 2024-03-19T11:30:59.853 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3588000000000002e-05 2024-03-19T11:30:59.853 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.853 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.853 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:30:59.853 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455846+0000", 2024-03-19T11:30:59.853 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6030000000000005e-06 2024-03-19T11:30:59.853 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.853 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.853 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:59.853 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.076525+0000", 2024-03-19T11:30:59.853 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.62067875900000002 2024-03-19T11:30:59.853 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.853 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.853 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:30:59.853 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.076981+0000", 2024-03-19T11:30:59.854 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00045633300000000001 2024-03-19T11:30:59.854 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.854 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.854 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:59.854 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.104422+0000", 2024-03-19T11:30:59.854 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027440665999999999 2024-03-19T11:30:59.854 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.854 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.854 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:30:59.854 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.104786+0000", 2024-03-19T11:30:59.854 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00036471499999999999 2024-03-19T11:30:59.854 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.854 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.854 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:59.855 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.140725+0000", 2024-03-19T11:30:59.855 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.035939093999999998 2024-03-19T11:30:59.855 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.855 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.855 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:30:59.855 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.140841+0000", 2024-03-19T11:30:59.855 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000115207 2024-03-19T11:30:59.855 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.855 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.855 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:59.855 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.145559+0000", 2024-03-19T11:30:59.855 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.004718486 2024-03-19T11:30:59.855 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.855 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.855 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:59.856 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.145760+0000", 2024-03-19T11:30:59.856 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000201058 2024-03-19T11:30:59.856 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.856 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.856 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:59.856 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.147237+0000", 2024-03-19T11:30:59.856 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014770370000000001 2024-03-19T11:30:59.856 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:59.856 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:59.856 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:59.856 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.856 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.856 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.16 80/79 [PushOp(4:6d9093b1:::smithi03829856-38:head, version: 77'90, data_included: [500875~689757,1564955~358819], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:6d9093b1:::smithi03829856-38:head@77'90, size: 2100875, copy_subset: [(0, 2100875)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1923774, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:6be89c06:::smithi03829856-24:head, version: 77'92, data_included: [741376~589824,1753088~458752], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:6be89c06:::smithi03829856-24:head@77'92, size: 2386561, copy_subset: [(0, 2386561)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2211840, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:30:59.856 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:59.081830+0000", 2024-03-19T11:30:59.856 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.71046919600000003, 2024-03-19T11:30:59.857 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015229652999999999, 2024-03-19T11:30:59.857 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:59.857 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:59.857 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:59.857 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.857 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:59.857 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.081830+0000", 2024-03-19T11:30:59.857 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:59.857 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.857 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.857 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:59.857 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.081830+0000", 2024-03-19T11:30:59.857 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:59.857 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.857 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.858 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:59.858 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.081832+0000", 2024-03-19T11:30:59.858 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.731e-06 2024-03-19T11:30:59.858 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.858 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.858 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:59.858 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.083140+0000", 2024-03-19T11:30:59.858 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0013082829999999999 2024-03-19T11:30:59.858 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.858 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.858 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:59.858 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.083170+0000", 2024-03-19T11:30:59.858 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0270999999999999e-05 2024-03-19T11:30:59.858 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.859 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.859 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:59.859 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.083199+0000", 2024-03-19T11:30:59.859 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9298e-05 2024-03-19T11:30:59.859 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.859 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.859 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:59.859 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.089969+0000", 2024-03-19T11:30:59.859 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0067691289999999996 2024-03-19T11:30:59.859 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.859 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.859 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:59.859 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.089995+0000", 2024-03-19T11:30:59.859 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5995000000000001e-05 2024-03-19T11:30:59.859 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.860 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.860 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:59.860 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.097060+0000", 2024-03-19T11:30:59.860 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0070649459999999999 2024-03-19T11:30:59.860 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:59.860 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:59.860 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:59.860 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.860 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.860 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.6 80/79 [PushOp(4:626acd4d:::smithi03829856-2:8, version: 55'54, data_included: [416807~475249,1542129~474678], data_size: 949927, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:626acd4d:::smithi03829856-2:8@55'54, size: 2016807, copy_subset: [(0, 2016807)], clone_subset: {}, snapset: 15=[]:{8: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2016807, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:30:59.860 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:59.110928+0000", 2024-03-19T11:30:59.860 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.68137088199999996, 2024-03-19T11:30:59.860 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01605765, 2024-03-19T11:30:59.860 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:59.861 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:59.861 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:59.861 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.861 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:59.861 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.110928+0000", 2024-03-19T11:30:59.861 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:59.861 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.861 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.861 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:59.861 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.110928+0000", 2024-03-19T11:30:59.861 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:59.861 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.861 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.861 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:59.861 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.110929+0000", 2024-03-19T11:30:59.862 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6500000000000008e-07 2024-03-19T11:30:59.862 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.862 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.862 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:59.862 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.111446+0000", 2024-03-19T11:30:59.862 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00051713900000000001 2024-03-19T11:30:59.862 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.862 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.862 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:59.862 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.111464+0000", 2024-03-19T11:30:59.862 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7966e-05 2024-03-19T11:30:59.862 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.862 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.862 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:59.863 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.111483+0000", 2024-03-19T11:30:59.863 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8561000000000001e-05 2024-03-19T11:30:59.863 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.863 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.863 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:59.863 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.118569+0000", 2024-03-19T11:30:59.863 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0070860259999999996 2024-03-19T11:30:59.863 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.863 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.863 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:59.863 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.118586+0000", 2024-03-19T11:30:59.863 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6857000000000001e-05 2024-03-19T11:30:59.863 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.863 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.864 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:59.864 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.126986+0000", 2024-03-19T11:30:59.864 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0084001360000000008 2024-03-19T11:30:59.864 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:59.864 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:59.864 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:59.864 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.864 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.864 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.16 80/79 [PushOp(4:6868af91:::smithi03829856-10:10, version: 70'81, data_included: [524385~820472,1602039~228104], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:10@70'81, size: 3747928, copy_subset: [(0, 3747928)], clone_subset: {}, snapset: 14=[]:{7: [6], b: [b], 10: [d], 14: [13, 12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1830143, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:30:59.864 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:59.111343+0000", 2024-03-19T11:30:59.864 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.68095629199999996, 2024-03-19T11:30:59.864 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019054888999999998, 2024-03-19T11:30:59.864 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:59.864 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:59.864 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:59.865 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.865 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:59.865 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.111343+0000", 2024-03-19T11:30:59.865 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:59.865 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.865 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.865 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:59.865 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.111343+0000", 2024-03-19T11:30:59.865 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:59.865 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.865 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.865 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:59.865 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.111349+0000", 2024-03-19T11:30:59.865 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9619999999999998e-06 2024-03-19T11:30:59.865 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.866 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.866 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:59.866 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.113881+0000", 2024-03-19T11:30:59.866 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025322930000000001 2024-03-19T11:30:59.866 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.866 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.866 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:59.866 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.113917+0000", 2024-03-19T11:30:59.866 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5593999999999998e-05 2024-03-19T11:30:59.866 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.866 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.866 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:59.866 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.113962+0000", 2024-03-19T11:30:59.866 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5362999999999999e-05 2024-03-19T11:30:59.866 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.867 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.867 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:59.867 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.126871+0000", 2024-03-19T11:30:59.867 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012909273000000001 2024-03-19T11:30:59.867 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.867 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.867 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:59.867 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.126892+0000", 2024-03-19T11:30:59.867 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0678000000000001e-05 2024-03-19T11:30:59.867 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.867 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.867 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:59.867 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.130398+0000", 2024-03-19T11:30:59.867 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0035057259999999998 2024-03-19T11:30:59.868 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:59.868 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:59.868 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:59.868 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.868 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.868 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.16 80/79 [PushOp(4:6868af91:::smithi03829856-10:7, version: 55'57, data_included: [524385~820472,1602039~228104], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:7@55'57, size: 2924385, copy_subset: [(0, 2924385)], clone_subset: {}, snapset: 14=[]:{7: [6], b: [b], 10: [d], 14: [13, 12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1830143, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:6868af91:::smithi03829856-10:14, version: 74'85, data_included: [405504~950272,1527054~98304], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:14@74'85, size: 2420208, copy_subset: [(0, 2420208)], clone_subset: {}, snapset: 14=[]:{7: [6], b: [b], 10: [d], 14: [13, 12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1625358, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:30:59.868 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:59.111755+0000", 2024-03-19T11:30:59.868 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.68054384099999998, 2024-03-19T11:30:59.868 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014604127, 2024-03-19T11:30:59.868 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:59.868 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:59.868 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:59.868 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.868 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:59.868 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.111755+0000", 2024-03-19T11:30:59.869 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:59.869 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.869 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.869 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:59.869 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.111755+0000", 2024-03-19T11:30:59.869 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:59.869 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.869 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.869 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:59.869 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.111756+0000", 2024-03-19T11:30:59.869 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4000000000000001e-07 2024-03-19T11:30:59.869 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.869 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.869 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:59.869 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.112850+0000", 2024-03-19T11:30:59.870 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0010945079999999999 2024-03-19T11:30:59.870 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.870 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.870 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:59.870 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.112887+0000", 2024-03-19T11:30:59.870 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6971999999999998e-05 2024-03-19T11:30:59.870 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.870 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.870 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:59.870 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.112916+0000", 2024-03-19T11:30:59.870 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8580000000000001e-05 2024-03-19T11:30:59.870 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.870 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.871 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:59.871 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.118590+0000", 2024-03-19T11:30:59.871 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0056736599999999996 2024-03-19T11:30:59.871 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.871 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.871 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:59.871 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.118617+0000", 2024-03-19T11:30:59.871 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.7572e-05 2024-03-19T11:30:59.871 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.871 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.871 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:59.871 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.126359+0000", 2024-03-19T11:30:59.871 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077420950000000001 2024-03-19T11:30:59.871 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:59.871 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:59.872 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:59.872 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.872 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.872 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.16 80/79 [PushOp(4:6d9093b1:::smithi03829856-38:10, version: 70'83, data_included: [621859~568773,1729599~479803], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6d9093b1:::smithi03829856-38:10@70'83, size: 3142876, copy_subset: [(0, 3142876)], clone_subset: {}, snapset: 10=[]:{10: [d, b, 6]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2209402, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:30:59.872 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:59.114057+0000", 2024-03-19T11:30:59.872 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.67824226300000001, 2024-03-19T11:30:59.872 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019935583, 2024-03-19T11:30:59.872 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:30:59.872 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:30:59.872 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:30:59.872 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.872 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:30:59.872 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.114057+0000", 2024-03-19T11:30:59.872 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:59.873 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.873 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.873 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:30:59.873 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.114057+0000", 2024-03-19T11:30:59.873 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:30:59.873 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.873 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.873 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:30:59.873 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.114058+0000", 2024-03-19T11:30:59.873 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4300000000000001e-07 2024-03-19T11:30:59.873 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.873 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.873 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:30:59.873 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.117243+0000", 2024-03-19T11:30:59.874 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031852550000000001 2024-03-19T11:30:59.874 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.874 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.874 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:30:59.874 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.117263+0000", 2024-03-19T11:30:59.874 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0469999999999999e-05 2024-03-19T11:30:59.874 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.874 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.874 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:30:59.874 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.117276+0000", 2024-03-19T11:30:59.874 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2775000000000001e-05 2024-03-19T11:30:59.874 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.874 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.874 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:30:59.875 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.130681+0000", 2024-03-19T11:30:59.875 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013404879999999999 2024-03-19T11:30:59.875 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.875 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.875 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:30:59.875 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.130699+0000", 2024-03-19T11:30:59.875 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8343e-05 2024-03-19T11:30:59.875 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:30:59.875 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:30:59.875 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:30:59.875 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.133992+0000", 2024-03-19T11:30:59.875 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0032929169999999998 2024-03-19T11:30:59.875 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:59.875 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:59.875 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:59.875 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:30:59.876 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:30:59.876 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:30:59.877 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-19T11:30:59.878 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:30:59.886+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:30:59.978 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:30:59.987+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:00.036 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-03-19T11:31:00.037 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.2 to 1 (10000) 2024-03-19T11:31:00.067 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:00.067 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:31:00.067 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.067 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2387 4.2 4:43b32242:::smithi03829856-22:head [read 596992~684544] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-03-19T11:31:00.067 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:59.319806+0000", 2024-03-19T11:31:00.067 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.75909415099999999, 2024-03-19T11:31:00.067 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.75912522000000004, 2024-03-19T11:31:00.067 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:00.067 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "waiting for missing object", 2024-03-19T11:31:00.067 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:00.067 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:00.068 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:00.068 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2387 2024-03-19T11:31:00.068 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.068 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:00.068 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.068 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:00.068 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319806+0000", 2024-03-19T11:31:00.068 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:00.068 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.068 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.068 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:00.068 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319806+0000", 2024-03-19T11:31:00.068 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:00.068 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.068 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.069 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:00.069 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319807+0000", 2024-03-19T11:31:00.069 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3519999999999999e-06 2024-03-19T11:31:00.069 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.069 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.069 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:00.069 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319812+0000", 2024-03-19T11:31:00.069 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3959999999999997e-06 2024-03-19T11:31:00.069 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.069 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.069 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:00.069 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319827+0000", 2024-03-19T11:31:00.069 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4566000000000001e-05 2024-03-19T11:31:00.069 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.069 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.070 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:00.070 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319835+0000", 2024-03-19T11:31:00.070 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.994e-06 2024-03-19T11:31:00.070 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.070 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.070 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:00.070 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.320119+0000", 2024-03-19T11:31:00.070 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00028445199999999998 2024-03-19T11:31:00.070 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.070 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.070 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:00.070 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.320129+0000", 2024-03-19T11:31:00.070 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9580000000000002e-06 2024-03-19T11:31:00.070 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.070 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.071 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:00.071 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.069842+0000", 2024-03-19T11:31:00.071 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.74971218399999995 2024-03-19T11:31:00.071 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.071 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.071 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:31:00.071 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.070010+0000", 2024-03-19T11:31:00.071 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016846600000000001 2024-03-19T11:31:00.071 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:00.071 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:00.071 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:00.071 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.071 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.071 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.10 81/80 [PushOp(4:0dc3b91e:::smithi03829856-28:6, version: 55'31, data_included: [453789~780782,1543557~267794], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:0dc3b91e:::smithi03829856-28:6@55'31, size: 2907578, copy_subset: [(0, 2907578)], clone_subset: {}, snapset: 9=[]:{6: [6]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1811351, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:0dc3b91e:::smithi03829856-28:head, version: 55'32, data_included: [662209~626546,1808051~422030], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:0dc3b91e:::smithi03829856-28:head@55'32, size: 3462209, copy_subset: [(0, 3462209)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2230081, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:00.071 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:00.071249+0000", 2024-03-19T11:31:00.072 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.0076505829999999999, 2024-03-19T11:31:00.072 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0078217640000000001, 2024-03-19T11:31:00.072 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:00.072 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:00.072 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:00.072 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.072 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:00.072 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.071249+0000", 2024-03-19T11:31:00.072 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:00.072 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.072 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.072 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:00.072 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.071249+0000", 2024-03-19T11:31:00.072 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:00.072 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.073 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.073 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:00.073 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.071249+0000", 2024-03-19T11:31:00.073 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.16e-07 2024-03-19T11:31:00.073 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.073 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.073 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:00.073 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.072569+0000", 2024-03-19T11:31:00.073 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0013198210000000001 2024-03-19T11:31:00.073 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.073 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.073 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:00.073 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.072650+0000", 2024-03-19T11:31:00.073 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1217000000000005e-05 2024-03-19T11:31:00.073 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.074 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.074 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:00.074 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.072693+0000", 2024-03-19T11:31:00.074 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2361999999999999e-05 2024-03-19T11:31:00.074 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.074 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.074 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:00.074 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.073448+0000", 2024-03-19T11:31:00.074 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000755191 2024-03-19T11:31:00.074 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.074 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.074 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:00.074 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.073466+0000", 2024-03-19T11:31:00.074 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8383e-05 2024-03-19T11:31:00.075 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:00.075 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:00.075 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:00.075 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.075 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.075 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.2 81/80 [PushOp(4:43b32242:::smithi03829856-22:head, version: 70'10, data_included: [423576~942306,1497916~106270], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:43b32242:::smithi03829856-22:head@70'10, size: 2823576, copy_subset: [(0, 2823576)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1604186, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:00.075 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:00.073028+0000", 2024-03-19T11:31:00.075 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.0058717659999999996, 2024-03-19T11:31:00.075 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0061224740000000001, 2024-03-19T11:31:00.075 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:00.075 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:00.075 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:00.075 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.075 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:00.076 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.073028+0000", 2024-03-19T11:31:00.076 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:00.076 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.076 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.076 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:00.076 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.073028+0000", 2024-03-19T11:31:00.076 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:00.076 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.076 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.076 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:00.076 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.073029+0000", 2024-03-19T11:31:00.076 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2200000000000003e-07 2024-03-19T11:31:00.076 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.076 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.076 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:00.077 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.073591+0000", 2024-03-19T11:31:00.077 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00056181799999999995 2024-03-19T11:31:00.077 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.077 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.077 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:00.077 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.073610+0000", 2024-03-19T11:31:00.077 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8936e-05 2024-03-19T11:31:00.077 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.077 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.077 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:00.077 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.073632+0000", 2024-03-19T11:31:00.077 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2504e-05 2024-03-19T11:31:00.077 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.078 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.078 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:00.078 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.073747+0000", 2024-03-19T11:31:00.078 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000115058 2024-03-19T11:31:00.078 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.078 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.078 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:00.078 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.073762+0000", 2024-03-19T11:31:00.078 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5325e-05 2024-03-19T11:31:00.078 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:00.078 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:00.078 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:00.078 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:00.078 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-19T11:31:00.079 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 3 2024-03-19T11:31:00.079 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:00.080 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:00.091+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:00.094 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-19T11:31:00.098 INFO:tasks.rados.rados.0.smithi038.stdout:1174: set_chunk oid 22 offset: 596992 length: 684544 target oid 96dd3def858c79aab5d2286a6f2a4c4b7a6843c8992c819c3f05f48c97f51831 offset: 0 snap -1 2024-03-19T11:31:00.098 INFO:tasks.rados.rados.0.smithi038.stdout:1175: read oid 33 snap -1 2024-03-19T11:31:00.098 INFO:tasks.rados.rados.0.smithi038.stdout:1175: expect (ObjNum 32 snap 0 seq_num 32) 2024-03-19T11:31:00.098 INFO:tasks.rados.rados.0.smithi038.stdout:1176: snap_create 2024-03-19T11:31:00.098 INFO:tasks.rados.rados.0.smithi038.stdout:1172: finishing set_chunk to oid 26 2024-03-19T11:31:00.098 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 26 v 55 (ObjNum 123 snap 15 seq_num 123) dirty exists 2024-03-19T11:31:00.099 INFO:tasks.rados.rados.0.smithi038.stdout:1173: expect (ObjNum 152 snap 17 seq_num 152) 2024-03-19T11:31:00.110 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-19T11:31:00.181 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:00.191+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:00.273 INFO:tasks.rados.rados.0.smithi038.stdout:1175: expect (ObjNum 32 snap 0 seq_num 32) 2024-03-19T11:31:00.279 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:00.279 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:31:00.279 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:31:00.279 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:31:00.279 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:00.282 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:00.290+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:00.293 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-19T11:31:00.306 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-19T11:31:00.356 INFO:tasks.rados.rados.0.smithi038.stdout:1174: finishing set_chunk to oid 22 2024-03-19T11:31:00.357 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 22 v 10 (ObjNum 168 snap 17 seq_num 168) dirty exists 2024-03-19T11:31:00.383 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:00.395+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:00.446 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:00.446 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:31:00.446 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:31:00.446 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:31:00.446 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.446 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1063 4.4 4:214026b3:::smithi03829856-47:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:31:00.447 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.784904+0000", 2024-03-19T11:31:00.447 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 46.671518831, 2024-03-19T11:31:00.447 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024425912000000001, 2024-03-19T11:31:00.447 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:00.447 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:00.447 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:00.447 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:00.447 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:00.447 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1063 2024-03-19T11:31:00.447 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.447 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:00.447 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.447 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:00.448 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784904+0000", 2024-03-19T11:31:00.448 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:00.448 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.448 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.448 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:00.448 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784904+0000", 2024-03-19T11:31:00.448 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:00.448 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.448 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.448 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:00.448 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784906+0000", 2024-03-19T11:31:00.448 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1409999999999999e-06 2024-03-19T11:31:00.449 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.449 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.449 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:00.449 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784915+0000", 2024-03-19T11:31:00.449 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5459999999999992e-06 2024-03-19T11:31:00.449 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.449 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.449 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:00.449 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784935+0000", 2024-03-19T11:31:00.449 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0310999999999999e-05 2024-03-19T11:31:00.449 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.449 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.450 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:00.450 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784946+0000", 2024-03-19T11:31:00.450 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0675e-05 2024-03-19T11:31:00.450 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.450 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.450 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:00.450 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.785032+0000", 2024-03-19T11:31:00.450 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6049000000000002e-05 2024-03-19T11:31:00.450 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.450 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.450 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:00.450 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.785136+0000", 2024-03-19T11:31:00.451 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000104209 2024-03-19T11:31:00.451 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.451 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.451 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:00.451 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.809330+0000", 2024-03-19T11:31:00.451 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024193981 2024-03-19T11:31:00.451 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:00.451 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:00.451 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:00.451 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.451 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.451 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1433 4.16 4:6be89c06:::smithi03829856-24:c [sparse-read 0~1100427,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:31:00.451 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.976652+0000", 2024-03-19T11:31:00.452 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 34.479771130000003, 2024-03-19T11:31:00.452 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95684753199999995, 2024-03-19T11:31:00.452 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:00.452 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:00.452 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:00.452 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:00.452 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:00.452 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1433 2024-03-19T11:31:00.452 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.452 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:00.452 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.452 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:00.452 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976652+0000", 2024-03-19T11:31:00.453 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:00.453 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.453 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.453 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:00.453 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976652+0000", 2024-03-19T11:31:00.453 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:00.453 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.453 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.453 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:00.453 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976654+0000", 2024-03-19T11:31:00.453 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6199999999999999e-06 2024-03-19T11:31:00.453 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.453 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.454 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:00.454 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976668+0000", 2024-03-19T11:31:00.454 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3213999999999999e-05 2024-03-19T11:31:00.454 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.454 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.454 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:00.454 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976694+0000", 2024-03-19T11:31:00.454 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6488000000000001e-05 2024-03-19T11:31:00.454 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.454 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.454 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:00.454 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976705+0000", 2024-03-19T11:31:00.455 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1044e-05 2024-03-19T11:31:00.455 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.455 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.455 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:00.455 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976795+0000", 2024-03-19T11:31:00.455 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0283000000000003e-05 2024-03-19T11:31:00.455 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.455 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.455 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:00.455 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976811+0000", 2024-03-19T11:31:00.455 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5262000000000001e-05 2024-03-19T11:31:00.455 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.455 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.456 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:00.456 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.927337+0000", 2024-03-19T11:31:00.456 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95052597100000003 2024-03-19T11:31:00.456 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.456 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.456 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:00.456 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.927495+0000", 2024-03-19T11:31:00.456 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015788600000000001 2024-03-19T11:31:00.456 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.456 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.456 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:00.456 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.933499+0000", 2024-03-19T11:31:00.456 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0060047640000000001 2024-03-19T11:31:00.457 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:00.457 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:00.457 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:00.457 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.457 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.457 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1434 4.16 4:6be89c06:::smithi03829856-24:c [read 0~2386561,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:31:00.457 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.976739+0000", 2024-03-19T11:31:00.457 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 34.479683969, 2024-03-19T11:31:00.457 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95781317399999999, 2024-03-19T11:31:00.457 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:00.457 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:00.457 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:00.457 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:00.458 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:00.458 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1434 2024-03-19T11:31:00.458 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.458 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:00.458 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.458 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:00.458 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976739+0000", 2024-03-19T11:31:00.458 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:00.458 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.458 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.458 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:00.458 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976739+0000", 2024-03-19T11:31:00.459 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:00.459 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.459 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.459 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:00.459 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976740+0000", 2024-03-19T11:31:00.459 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8800000000000001e-07 2024-03-19T11:31:00.459 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.459 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.459 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:00.459 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976748+0000", 2024-03-19T11:31:00.459 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8979999999999996e-06 2024-03-19T11:31:00.459 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.459 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.460 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:00.460 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976757+0000", 2024-03-19T11:31:00.460 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9439999999999994e-06 2024-03-19T11:31:00.460 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.460 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.460 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:00.460 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976763+0000", 2024-03-19T11:31:00.460 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5030000000000002e-06 2024-03-19T11:31:00.460 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.460 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.460 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:00.460 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976849+0000", 2024-03-19T11:31:00.460 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5803999999999994e-05 2024-03-19T11:31:00.461 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.461 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.461 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:00.461 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976860+0000", 2024-03-19T11:31:00.461 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0869e-05 2024-03-19T11:31:00.461 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.461 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.461 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:00.461 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.933562+0000", 2024-03-19T11:31:00.461 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95670232300000002 2024-03-19T11:31:00.461 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.462 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.462 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:00.462 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.933693+0000", 2024-03-19T11:31:00.462 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013110699999999999 2024-03-19T11:31:00.462 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.462 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.462 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:00.462 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.934552+0000", 2024-03-19T11:31:00.462 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00085883800000000001 2024-03-19T11:31:00.462 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:00.462 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:00.462 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:00.463 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.463 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.463 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1435 4.16 4:6be89c06:::smithi03829856-24:c [sparse-read 0~1100427] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:31:00.463 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.976775+0000", 2024-03-19T11:31:00.463 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 34.479648298000001, 2024-03-19T11:31:00.463 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95962779499999995, 2024-03-19T11:31:00.463 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:00.463 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:00.463 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:00.463 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:00.463 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:00.463 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1435 2024-03-19T11:31:00.464 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.464 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:00.464 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.464 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:00.464 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976775+0000", 2024-03-19T11:31:00.464 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:00.464 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.464 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.464 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:00.464 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976775+0000", 2024-03-19T11:31:00.464 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:00.464 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.465 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.465 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:00.465 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976775+0000", 2024-03-19T11:31:00.465 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9800000000000003e-07 2024-03-19T11:31:00.465 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.465 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.465 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:00.465 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976779+0000", 2024-03-19T11:31:00.465 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3830000000000001e-06 2024-03-19T11:31:00.465 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.465 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.465 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:00.465 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976785+0000", 2024-03-19T11:31:00.466 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5649999999999998e-06 2024-03-19T11:31:00.466 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.466 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.466 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:00.466 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976790+0000", 2024-03-19T11:31:00.466 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1379999999999998e-06 2024-03-19T11:31:00.466 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.466 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.466 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:00.466 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976889+0000", 2024-03-19T11:31:00.466 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8331000000000004e-05 2024-03-19T11:31:00.466 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.466 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.467 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:00.467 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976894+0000", 2024-03-19T11:31:00.467 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8270000000000003e-06 2024-03-19T11:31:00.467 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.467 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.467 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:00.467 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.935982+0000", 2024-03-19T11:31:00.467 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95908714100000003 2024-03-19T11:31:00.467 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.467 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.467 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:00.467 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.936093+0000", 2024-03-19T11:31:00.468 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000111557 2024-03-19T11:31:00.468 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.468 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.468 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:00.468 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.936402+0000", 2024-03-19T11:31:00.468 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030925499999999998 2024-03-19T11:31:00.468 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:00.468 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:00.468 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:00.468 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.468 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.468 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1442 4.6 4:626acd4d:::smithi03829856-2:head [cache-evict] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:31:00.468 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.978864+0000", 2024-03-19T11:31:00.469 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 34.477558633999998, 2024-03-19T11:31:00.469 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95708845099999995, 2024-03-19T11:31:00.469 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:00.469 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:00.469 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:00.469 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:00.469 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:00.469 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1442 2024-03-19T11:31:00.469 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.469 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:00.469 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.469 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:00.469 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978864+0000", 2024-03-19T11:31:00.470 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:00.470 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.470 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.470 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:00.470 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978864+0000", 2024-03-19T11:31:00.470 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:00.470 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.470 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.470 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:00.470 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978866+0000", 2024-03-19T11:31:00.470 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0729999999999999e-06 2024-03-19T11:31:00.470 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.471 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.471 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:00.471 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978877+0000", 2024-03-19T11:31:00.471 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0390999999999999e-05 2024-03-19T11:31:00.471 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.471 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.471 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:00.471 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978902+0000", 2024-03-19T11:31:00.471 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5635000000000001e-05 2024-03-19T11:31:00.471 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.471 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.471 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:00.471 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978915+0000", 2024-03-19T11:31:00.472 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2617e-05 2024-03-19T11:31:00.472 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.472 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.472 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:00.472 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.979005+0000", 2024-03-19T11:31:00.472 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0552999999999999e-05 2024-03-19T11:31:00.472 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.472 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.472 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:00.472 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.979021+0000", 2024-03-19T11:31:00.472 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5102999999999999e-05 2024-03-19T11:31:00.472 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.472 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.473 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:00.473 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.926879+0000", 2024-03-19T11:31:00.473 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.94785817900000002 2024-03-19T11:31:00.473 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.473 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.473 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:00.473 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.927277+0000", 2024-03-19T11:31:00.473 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00039819600000000002 2024-03-19T11:31:00.473 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.473 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.473 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:00.473 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.935752+0000", 2024-03-19T11:31:00.474 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0084753420000000003 2024-03-19T11:31:00.474 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.474 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.474 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:00.474 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.935874+0000", 2024-03-19T11:31:00.474 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000121311 2024-03-19T11:31:00.474 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.474 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.474 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:00.474 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.935953+0000", 2024-03-19T11:31:00.474 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9050999999999999e-05 2024-03-19T11:31:00.474 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:00.474 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:00.475 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:00.475 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.475 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.475 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.d 49/48 [PushOp(4:b65e6658:::smithi03829856-9:head, version: 28'17, data_included: [534456~718345,1951503~182953], data_size: 901298, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:b65e6658:::smithi03829856-9:head@28'17, size: 2134456, copy_subset: [(0, 2134456)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2134456, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:b4ef97e0:::smithi03829856-3:b, version: 46'34, data_included: [433614~1004682,1600611~43894], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:b4ef97e0:::smithi03829856-3:b@46'34, size: 3042034, copy_subset: [(0, 3042034)], clone_subset: {}, snapset: b=[]:{b: [b, a, 6, 5]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1644505, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:b4ef97e0:::smithi03829856-3:head, version: 47'43, data_included: [433614~1004682,1600611~43894], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:b4ef97e0:::smithi03829856-3:head@47'43, size: 3697507, copy_subset: [(0, 3697507)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1644505, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:00.475 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:26.943544+0000", 2024-03-19T11:31:00.475 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 33.512879144000003, 2024-03-19T11:31:00.475 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015601487000000001, 2024-03-19T11:31:00.475 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:00.475 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:00.475 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:00.475 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.475 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:00.476 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.943544+0000", 2024-03-19T11:31:00.476 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:00.476 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.476 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.476 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:00.476 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.943544+0000", 2024-03-19T11:31:00.476 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:00.476 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.476 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.476 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:00.476 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.943546+0000", 2024-03-19T11:31:00.476 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.013e-06 2024-03-19T11:31:00.476 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.477 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.477 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:00.477 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.946208+0000", 2024-03-19T11:31:00.477 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0026622820000000002 2024-03-19T11:31:00.477 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.477 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.477 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:00.477 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.946240+0000", 2024-03-19T11:31:00.477 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1568e-05 2024-03-19T11:31:00.477 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.477 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.477 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:00.477 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.946282+0000", 2024-03-19T11:31:00.478 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2599999999999999e-05 2024-03-19T11:31:00.478 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.478 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.478 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:00.478 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.946442+0000", 2024-03-19T11:31:00.478 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015960400000000001 2024-03-19T11:31:00.478 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.478 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.478 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:00.478 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.946458+0000", 2024-03-19T11:31:00.478 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6059999999999999e-05 2024-03-19T11:31:00.478 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.479 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.479 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:00.479 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.959145+0000", 2024-03-19T11:31:00.479 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01268736 2024-03-19T11:31:00.479 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:00.479 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:00.479 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:00.479 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.479 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.479 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2003 4.16 4:6868af91:::smithi03829856-10:head [tier-promote] snapc 12=[12,11,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:31:00.479 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:45.804046+0000", 2024-03-19T11:31:00.479 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 14.652376721, 2024-03-19T11:31:00.479 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018976396, 2024-03-19T11:31:00.479 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:00.480 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:00.480 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:00.480 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:00.480 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:00.480 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2003 2024-03-19T11:31:00.480 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.480 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:00.480 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.480 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:00.480 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804046+0000", 2024-03-19T11:31:00.480 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:00.480 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.480 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.481 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:00.481 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804046+0000", 2024-03-19T11:31:00.481 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:00.481 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.481 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.481 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:00.481 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804052+0000", 2024-03-19T11:31:00.481 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7599999999999999e-06 2024-03-19T11:31:00.481 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.481 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.481 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:00.481 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804090+0000", 2024-03-19T11:31:00.481 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8018000000000001e-05 2024-03-19T11:31:00.482 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.482 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.482 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:00.482 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804256+0000", 2024-03-19T11:31:00.482 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016593599999999999 2024-03-19T11:31:00.482 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.482 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.482 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:00.482 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804300+0000", 2024-03-19T11:31:00.482 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3648e-05 2024-03-19T11:31:00.482 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.482 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.482 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:00.483 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804538+0000", 2024-03-19T11:31:00.483 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000238347 2024-03-19T11:31:00.483 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.483 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.483 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:00.483 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804716+0000", 2024-03-19T11:31:00.483 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000177968 2024-03-19T11:31:00.483 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.483 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.483 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:00.483 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.823023+0000", 2024-03-19T11:31:00.483 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018306718999999999 2024-03-19T11:31:00.483 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:00.484 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:00.484 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:00.484 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.484 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.484 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1d58db11:::smithi03829856-18:6, version: 55'80, data_included: [759683~639358,1904434~409218], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:6@55'80, size: 3159683, copy_subset: [(0, 3159683)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2313652, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1d58db11:::smithi03829856-18:b, version: 58'84, data_included: [443718~955323,1564138~93253], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:b@58'84, size: 3445402, copy_subset: [(0, 3445402)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1657391, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:10, version: 66'95, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:10@66'95, size: 3764735, copy_subset: [(0, 3764735)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:12, version: 66'97, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:12@66'97, size: 2504243, copy_subset: [(0, 2504243)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:c, version: 70'102, data_included: [416807~576167,1542129~472409], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:c@70'102, size: 2857157, copy_subset: [(0, 2857157)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2014538, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:00.484 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.275179+0000", 2024-03-19T11:31:00.484 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 6.1812437310000004, 2024-03-19T11:31:00.484 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026559353000000001, 2024-03-19T11:31:00.484 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:00.484 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:00.484 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:00.485 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.485 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:00.485 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.275179+0000", 2024-03-19T11:31:00.485 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:00.485 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.485 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.485 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:00.485 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.275179+0000", 2024-03-19T11:31:00.485 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:00.485 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.485 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.485 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:00.485 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.275196+0000", 2024-03-19T11:31:00.486 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6946999999999999e-05 2024-03-19T11:31:00.486 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.486 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.486 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:00.486 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281456+0000", 2024-03-19T11:31:00.486 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0062600529999999998 2024-03-19T11:31:00.486 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.486 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.486 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:00.486 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281524+0000", 2024-03-19T11:31:00.486 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7971000000000004e-05 2024-03-19T11:31:00.486 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.486 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.487 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:00.487 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281591+0000", 2024-03-19T11:31:00.487 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6704000000000004e-05 2024-03-19T11:31:00.487 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.487 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.487 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:00.487 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281865+0000", 2024-03-19T11:31:00.487 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000274097 2024-03-19T11:31:00.487 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.487 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.487 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:00.487 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281901+0000", 2024-03-19T11:31:00.487 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5777e-05 2024-03-19T11:31:00.488 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.488 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.488 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:00.488 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.301738+0000", 2024-03-19T11:31:00.488 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019837804000000001 2024-03-19T11:31:00.488 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:00.488 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:00.488 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:00.488 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.488 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.488 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.18 e75,74 4:1d58db11:::smithi03829856-18:2,48'76)", 2024-03-19T11:31:00.488 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.281661+0000", 2024-03-19T11:31:00.488 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 6.1747622, 2024-03-19T11:31:00.489 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020384337999999998, 2024-03-19T11:31:00.489 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:00.489 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:00.489 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:00.489 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.489 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:00.489 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281661+0000", 2024-03-19T11:31:00.489 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:00.489 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.489 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.489 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:00.489 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281661+0000", 2024-03-19T11:31:00.489 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:00.490 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.490 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.490 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:00.490 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281661+0000", 2024-03-19T11:31:00.490 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9100000000000003e-07 2024-03-19T11:31:00.490 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.490 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.490 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:00.490 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281668+0000", 2024-03-19T11:31:00.490 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8199999999999999e-06 2024-03-19T11:31:00.490 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.490 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.491 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:00.491 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281679+0000", 2024-03-19T11:31:00.491 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1107999999999999e-05 2024-03-19T11:31:00.491 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.491 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.491 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:00.491 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281691+0000", 2024-03-19T11:31:00.491 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1399999999999999e-05 2024-03-19T11:31:00.491 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.491 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.491 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:00.491 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.301804+0000", 2024-03-19T11:31:00.492 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020113669000000001 2024-03-19T11:31:00.492 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.492 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.492 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:00.492 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.301818+0000", 2024-03-19T11:31:00.492 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3718e-05 2024-03-19T11:31:00.492 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.492 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.492 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:00.492 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302045+0000", 2024-03-19T11:31:00.492 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000226832 2024-03-19T11:31:00.492 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:00.492 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:00.493 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:00.493 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.493 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.493 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1dabc0f1:::smithi03829856-1:head, version: 70'103, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:head@70'103, size: 2621892, copy_subset: [(0, 2621892)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:00.493 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.281783+0000", 2024-03-19T11:31:00.493 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 6.1746399240000001, 2024-03-19T11:31:00.493 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026187221, 2024-03-19T11:31:00.493 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:00.493 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:00.493 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:00.493 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.493 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:00.493 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281783+0000", 2024-03-19T11:31:00.494 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:00.494 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.494 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.494 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:00.494 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281783+0000", 2024-03-19T11:31:00.494 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:00.494 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.494 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.494 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:00.494 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281783+0000", 2024-03-19T11:31:00.494 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1699999999999999e-07 2024-03-19T11:31:00.494 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.495 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.495 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:00.495 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.282962+0000", 2024-03-19T11:31:00.495 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001178601 2024-03-19T11:31:00.495 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.495 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.495 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:00.495 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.282987+0000", 2024-03-19T11:31:00.495 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4899e-05 2024-03-19T11:31:00.495 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.495 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.495 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:00.496 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283006+0000", 2024-03-19T11:31:00.496 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9494999999999999e-05 2024-03-19T11:31:00.496 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.496 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.496 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:00.496 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.305757+0000", 2024-03-19T11:31:00.496 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.022750570000000001 2024-03-19T11:31:00.496 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.496 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.496 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:00.496 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.305770+0000", 2024-03-19T11:31:00.496 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2791e-05 2024-03-19T11:31:00.496 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.497 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.497 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:00.497 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.307970+0000", 2024-03-19T11:31:00.497 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022005480000000001 2024-03-19T11:31:00.497 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:00.497 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:00.497 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:00.497 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.497 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.497 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.18 e75,74 4:1d58db11:::smithi03829856-18:c,70'104)", 2024-03-19T11:31:00.497 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.283036+0000", 2024-03-19T11:31:00.497 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 6.1733868239999996, 2024-03-19T11:31:00.498 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019146032, 2024-03-19T11:31:00.498 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:00.498 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:00.498 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:00.498 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.498 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:00.498 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283036+0000", 2024-03-19T11:31:00.498 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:00.498 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.498 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.498 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:00.498 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283036+0000", 2024-03-19T11:31:00.498 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:00.499 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.499 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.499 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:00.499 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283037+0000", 2024-03-19T11:31:00.499 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4899999999999995e-07 2024-03-19T11:31:00.499 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.499 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.499 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:00.499 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283042+0000", 2024-03-19T11:31:00.499 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0930000000000002e-06 2024-03-19T11:31:00.499 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.499 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.499 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:00.500 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283053+0000", 2024-03-19T11:31:00.500 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1083999999999999e-05 2024-03-19T11:31:00.500 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.500 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.500 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:00.500 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283059+0000", 2024-03-19T11:31:00.500 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6329999999999999e-06 2024-03-19T11:31:00.500 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.500 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.500 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:00.500 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302090+0000", 2024-03-19T11:31:00.500 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01903038 2024-03-19T11:31:00.500 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.501 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.501 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:00.501 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302097+0000", 2024-03-19T11:31:00.501 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.221e-06 2024-03-19T11:31:00.501 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.501 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.501 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:00.501 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302182+0000", 2024-03-19T11:31:00.501 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5072000000000001e-05 2024-03-19T11:31:00.501 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:00.501 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:00.501 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:00.501 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.502 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.502 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(osd.5.0:143 4.18 e75/74 4:1d58db11:::smithi03829856-18:head v 75'106, mlcod=0'0)", 2024-03-19T11:31:00.502 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.283083+0000", 2024-03-19T11:31:00.502 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 6.1733394969999997, 2024-03-19T11:31:00.502 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025309439, 2024-03-19T11:31:00.502 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:00.502 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:00.502 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:00.502 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.502 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:00.502 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283083+0000", 2024-03-19T11:31:00.502 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:00.502 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.503 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.503 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:00.503 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283083+0000", 2024-03-19T11:31:00.503 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:00.503 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.503 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.503 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:00.503 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283084+0000", 2024-03-19T11:31:00.503 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7800000000000002e-07 2024-03-19T11:31:00.503 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.503 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.503 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:00.504 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.284082+0000", 2024-03-19T11:31:00.504 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0009977880000000001 2024-03-19T11:31:00.504 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.504 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.504 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:00.504 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.284095+0000", 2024-03-19T11:31:00.504 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3256e-05 2024-03-19T11:31:00.504 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.504 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.504 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:00.504 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.284102+0000", 2024-03-19T11:31:00.504 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.605e-06 2024-03-19T11:31:00.504 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.504 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.505 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:00.505 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302285+0000", 2024-03-19T11:31:00.505 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018182423 2024-03-19T11:31:00.505 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.505 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.505 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:00.505 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302341+0000", 2024-03-19T11:31:00.505 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.626e-05 2024-03-19T11:31:00.505 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.505 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.505 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:00.505 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.308338+0000", 2024-03-19T11:31:00.505 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0059968030000000002 2024-03-19T11:31:00.506 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.506 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.506 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:00.506 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.308393+0000", 2024-03-19T11:31:00.506 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4826e-05 2024-03-19T11:31:00.506 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:00.506 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:00.506 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:00.506 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.506 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.506 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "pg_update_log_missing(4.18 epoch 75/74 rep_tid 144 entries 75'107 (0'0) error 4:1d58db11:::smithi03829856-18:head by client.4415.0:2283 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-19T11:31:00.506 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.288252+0000", 2024-03-19T11:31:00.507 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 6.1681709470000001, 2024-03-19T11:31:00.507 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020167377, 2024-03-19T11:31:00.507 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:00.507 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-03-19T11:31:00.507 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:00.507 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.507 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:00.507 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288252+0000", 2024-03-19T11:31:00.507 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:00.507 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.507 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.507 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:00.507 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288252+0000", 2024-03-19T11:31:00.508 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:00.508 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.508 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.508 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:00.508 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288253+0000", 2024-03-19T11:31:00.508 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2109999999999999e-06 2024-03-19T11:31:00.508 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.508 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.508 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:00.508 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288268+0000", 2024-03-19T11:31:00.508 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5003e-05 2024-03-19T11:31:00.508 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.508 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.509 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:00.509 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288301+0000", 2024-03-19T11:31:00.509 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2539999999999997e-05 2024-03-19T11:31:00.509 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.509 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.509 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:00.509 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288319+0000", 2024-03-19T11:31:00.509 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8232000000000001e-05 2024-03-19T11:31:00.509 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.509 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.509 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:00.509 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.305490+0000", 2024-03-19T11:31:00.509 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017170857000000001 2024-03-19T11:31:00.509 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.510 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.510 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:00.510 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.308419+0000", 2024-03-19T11:31:00.510 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029295340000000001 2024-03-19T11:31:00.510 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:00.510 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:00.510 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:00.510 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.510 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.510 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1d58db11:::smithi03829856-18:6, version: 55'80, data_included: [2313652~310127,3128657~31026], data_size: 341153, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:6@55'80, size: 3159683, copy_subset: [(0, 3159683)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3159683, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2313652, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:1d58db11:::smithi03829856-18:b, version: 58'84, data_included: [1657391~966388,2633066~82188], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:b@58'84, size: 3445402, copy_subset: [(0, 3445402)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2715254, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1657391, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:10, version: 66'95, data_included: [1829043~668012,3134120~380564], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:10@66'95, size: 3764735, copy_subset: [(0, 3764735)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3514684, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:12, version: 66'97, data_included: [1829043~507467], data_size: 507467, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:12@66'97, size: 2504243, copy_subset: [(0, 2504243)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2504243, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:c, version: 70'102, data_included: [2014538~156250], data_size: 156250, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:c@70'102, size: 2857157, copy_subset: [(0, 2857157)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2857157, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2014538, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:00.510 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.312362+0000", 2024-03-19T11:31:00.511 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 6.1440608980000002, 2024-03-19T11:31:00.511 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017968951, 2024-03-19T11:31:00.511 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:00.511 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:00.511 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:00.511 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.511 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:00.511 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.312362+0000", 2024-03-19T11:31:00.511 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:00.511 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.511 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.511 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:00.511 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.312362+0000", 2024-03-19T11:31:00.512 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:00.512 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.512 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.512 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:00.512 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.312363+0000", 2024-03-19T11:31:00.512 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.528e-06 2024-03-19T11:31:00.512 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.512 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.512 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:00.512 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315748+0000", 2024-03-19T11:31:00.512 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033842830000000001 2024-03-19T11:31:00.512 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.512 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.513 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:00.513 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315795+0000", 2024-03-19T11:31:00.513 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7382999999999997e-05 2024-03-19T11:31:00.513 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.513 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.513 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:00.513 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315845+0000", 2024-03-19T11:31:00.513 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0319999999999999e-05 2024-03-19T11:31:00.513 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.513 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.513 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:00.513 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.316123+0000", 2024-03-19T11:31:00.513 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00027747200000000002 2024-03-19T11:31:00.514 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.514 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.514 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:00.514 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.316170+0000", 2024-03-19T11:31:00.514 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7466999999999998e-05 2024-03-19T11:31:00.514 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.514 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.514 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:00.514 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.330331+0000", 2024-03-19T11:31:00.514 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014160498000000001 2024-03-19T11:31:00.514 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:00.514 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:00.515 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:00.515 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.515 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.515 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1dabc0f1:::smithi03829856-1:head, version: 70'103, data_included: [1829043~555153], data_size: 555153, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:head@70'103, size: 2621892, copy_subset: [(0, 2621892)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2621892, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:00.515 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.315898+0000", 2024-03-19T11:31:00.515 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 6.1405249120000001, 2024-03-19T11:31:00.515 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016880424000000002, 2024-03-19T11:31:00.515 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:00.515 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:00.515 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:00.515 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.515 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:00.515 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315898+0000", 2024-03-19T11:31:00.516 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:00.516 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.516 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.516 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:00.516 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315898+0000", 2024-03-19T11:31:00.516 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:00.516 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.516 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.516 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:00.516 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315899+0000", 2024-03-19T11:31:00.516 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.003e-06 2024-03-19T11:31:00.516 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.516 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.517 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:00.517 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.316486+0000", 2024-03-19T11:31:00.517 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00058673099999999995 2024-03-19T11:31:00.517 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.517 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.517 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:00.517 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.316519+0000", 2024-03-19T11:31:00.517 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3124000000000003e-05 2024-03-19T11:31:00.517 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.517 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.517 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:00.517 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.316534+0000", 2024-03-19T11:31:00.517 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.485e-05 2024-03-19T11:31:00.518 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.518 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.518 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:00.518 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.330415+0000", 2024-03-19T11:31:00.518 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013880905000000001 2024-03-19T11:31:00.518 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.518 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.518 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:00.518 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.330439+0000", 2024-03-19T11:31:00.518 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4630999999999998e-05 2024-03-19T11:31:00.518 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.518 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.518 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:00.519 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.332778+0000", 2024-03-19T11:31:00.519 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023391800000000002 2024-03-19T11:31:00.519 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:00.519 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:00.519 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:00.519 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.519 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.519 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2387 4.2 4:43b32242:::smithi03829856-22:head [read 596992~684544] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-03-19T11:31:00.519 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:59.319806+0000", 2024-03-19T11:31:00.519 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 1.1366172450000001, 2024-03-19T11:31:00.519 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.78144344399999999, 2024-03-19T11:31:00.519 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:00.519 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:00.520 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:00.520 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:00.520 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:00.520 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2387 2024-03-19T11:31:00.520 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.520 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:00.520 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.520 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:00.520 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319806+0000", 2024-03-19T11:31:00.520 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:00.520 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.520 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.521 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:00.521 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319806+0000", 2024-03-19T11:31:00.521 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:00.521 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.521 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.521 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:00.521 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319807+0000", 2024-03-19T11:31:00.521 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3519999999999999e-06 2024-03-19T11:31:00.521 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.521 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.521 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:00.521 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319812+0000", 2024-03-19T11:31:00.521 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3959999999999997e-06 2024-03-19T11:31:00.522 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.522 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.522 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:00.522 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319827+0000", 2024-03-19T11:31:00.522 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4566000000000001e-05 2024-03-19T11:31:00.522 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.522 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.522 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:00.522 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319835+0000", 2024-03-19T11:31:00.522 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.994e-06 2024-03-19T11:31:00.522 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.522 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.522 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:00.523 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.320119+0000", 2024-03-19T11:31:00.523 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00028445199999999998 2024-03-19T11:31:00.523 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.523 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.523 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:00.523 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.320129+0000", 2024-03-19T11:31:00.523 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9580000000000002e-06 2024-03-19T11:31:00.523 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.523 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.523 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:00.523 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.069842+0000", 2024-03-19T11:31:00.523 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.74971218399999995 2024-03-19T11:31:00.524 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.524 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.524 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:31:00.524 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.070010+0000", 2024-03-19T11:31:00.524 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016846600000000001 2024-03-19T11:31:00.524 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.524 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.524 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:00.524 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.090911+0000", 2024-03-19T11:31:00.524 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020901494999999999 2024-03-19T11:31:00.524 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.524 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.524 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:31:00.525 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.091249+0000", 2024-03-19T11:31:00.525 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000337602 2024-03-19T11:31:00.525 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.525 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.525 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:00.525 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.098478+0000", 2024-03-19T11:31:00.525 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0072287549999999999 2024-03-19T11:31:00.525 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.525 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.525 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:00.525 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.098617+0000", 2024-03-19T11:31:00.525 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013871599999999999 2024-03-19T11:31:00.525 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.526 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.526 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:00.526 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.101249+0000", 2024-03-19T11:31:00.526 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002632508 2024-03-19T11:31:00.526 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:00.526 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:00.526 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:00.526 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.526 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.526 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 81/80 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 15=[]:{7: [6], b: [b], 10: [d], 12: [12, 11], 15: [15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:7, version: 55'82, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@55'82, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 16=[]:{7: [6], b: [b], 10: [d], 16: [16, 15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 15=[]:{7: [6], b: [b], 10: [d], 12: [12, 11], 15: [15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 58'86, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@58'86, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 16=[]:{7: [6], b: [b], 10: [d], 16: [16, 15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [504210~968657,1572695~79919], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 16=[]:{7: [6], b: [b], 10: [d], 16: [16, 15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:00.527 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:00.089924+0000", 2024-03-19T11:31:00.527 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.36649901600000001, 2024-03-19T11:31:00.527 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024686583000000002, 2024-03-19T11:31:00.527 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:00.527 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:00.527 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:00.527 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.527 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:00.527 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.089924+0000", 2024-03-19T11:31:00.527 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:00.527 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.527 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.527 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:00.528 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.089924+0000", 2024-03-19T11:31:00.528 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:00.528 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.528 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.528 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:00.528 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.089926+0000", 2024-03-19T11:31:00.528 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9589999999999998e-06 2024-03-19T11:31:00.528 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.528 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.528 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:00.528 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093311+0000", 2024-03-19T11:31:00.528 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033854409999999999 2024-03-19T11:31:00.529 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.529 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.529 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:00.529 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093365+0000", 2024-03-19T11:31:00.529 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3795000000000002e-05 2024-03-19T11:31:00.529 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.529 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.529 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:00.529 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093438+0000", 2024-03-19T11:31:00.529 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2591999999999999e-05 2024-03-19T11:31:00.529 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.529 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.530 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:00.530 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093690+0000", 2024-03-19T11:31:00.530 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025264999999999999 2024-03-19T11:31:00.530 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.530 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.530 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:00.530 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093731+0000", 2024-03-19T11:31:00.530 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0881999999999999e-05 2024-03-19T11:31:00.530 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.530 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.530 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:00.531 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.114610+0000", 2024-03-19T11:31:00.531 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020879264000000002 2024-03-19T11:31:00.531 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:00.531 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:00.531 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:00.531 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.531 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.531 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.21 e81,80 4:8401c1f0:::smithi03829856-43:8,46'66,4:866f3dcc:::smithi03829856-31:2,47'73)", 2024-03-19T11:31:00.531 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:00.093514+0000", 2024-03-19T11:31:00.531 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.36290841600000001, 2024-03-19T11:31:00.531 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027375713999999999, 2024-03-19T11:31:00.531 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:00.532 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:00.532 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:00.532 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.532 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:00.532 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093514+0000", 2024-03-19T11:31:00.532 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:00.532 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.532 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.532 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:00.532 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093514+0000", 2024-03-19T11:31:00.532 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:00.532 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.532 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.533 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:00.533 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093516+0000", 2024-03-19T11:31:00.533 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1179999999999999e-06 2024-03-19T11:31:00.533 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.533 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.533 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:00.533 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093523+0000", 2024-03-19T11:31:00.533 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7500000000000003e-06 2024-03-19T11:31:00.533 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.533 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.533 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:00.533 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093534+0000", 2024-03-19T11:31:00.533 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0831e-05 2024-03-19T11:31:00.534 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.534 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.534 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:00.534 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093542+0000", 2024-03-19T11:31:00.534 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0139999999999998e-06 2024-03-19T11:31:00.534 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.534 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.534 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:00.534 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.120682+0000", 2024-03-19T11:31:00.534 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027139838999999999 2024-03-19T11:31:00.534 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.534 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.535 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:00.535 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.120693+0000", 2024-03-19T11:31:00.535 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.136e-05 2024-03-19T11:31:00.535 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.535 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.535 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:00.535 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.120890+0000", 2024-03-19T11:31:00.535 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019680199999999999 2024-03-19T11:31:00.535 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:00.535 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:00.535 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:00.535 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.536 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.536 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 81/80 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 15=[]:{7: [6], b: [b], 10: [d], 12: [12, 11], 15: [15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:7, version: 55'82, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@55'82, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 16=[]:{7: [6], b: [b], 10: [d], 16: [16, 15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 15=[]:{7: [6], b: [b], 10: [d], 12: [12, 11], 15: [15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 58'86, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@58'86, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 16=[]:{7: [6], b: [b], 10: [d], 16: [16, 15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [1652614~973634,2888511~52681,3182305~22261], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 16=[]:{7: [6], b: [b], 10: [d], 16: [16, 15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3204566, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:00.536 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:00.126855+0000", 2024-03-19T11:31:00.536 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.329567679, 2024-03-19T11:31:00.536 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016438076999999999, 2024-03-19T11:31:00.536 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:00.536 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:00.536 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:00.536 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.536 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:00.536 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.126855+0000", 2024-03-19T11:31:00.537 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:00.537 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.537 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.537 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:00.537 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.126855+0000", 2024-03-19T11:31:00.537 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:00.537 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.537 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.537 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:00.537 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.126857+0000", 2024-03-19T11:31:00.537 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.739e-06 2024-03-19T11:31:00.537 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.538 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.538 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:00.538 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.129236+0000", 2024-03-19T11:31:00.538 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023792449999999999 2024-03-19T11:31:00.538 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.538 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.538 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:00.538 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.129298+0000", 2024-03-19T11:31:00.538 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2329000000000006e-05 2024-03-19T11:31:00.538 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.538 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.538 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:00.539 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.129365+0000", 2024-03-19T11:31:00.539 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6299999999999999e-05 2024-03-19T11:31:00.539 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.539 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.539 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:00.539 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.132493+0000", 2024-03-19T11:31:00.539 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031279530000000002 2024-03-19T11:31:00.539 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.539 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.539 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:00.539 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.132578+0000", 2024-03-19T11:31:00.539 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5648000000000004e-05 2024-03-19T11:31:00.540 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.540 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.540 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:00.540 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.143293+0000", 2024-03-19T11:31:00.540 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010714863 2024-03-19T11:31:00.540 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:00.540 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:00.540 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:00.540 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.540 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.540 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.29 81/80 [PushOp(4:953e720b:::smithi03829856-45:10, version: 70'58, data_included: [1927574~595528,2570780~420833], data_size: 1016361, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:953e720b:::smithi03829856-45:10@70'58, size: 2991613, copy_subset: [(0, 2991613)], clone_subset: {}, snapset: 10=[]:{b: [b, 6], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2991613, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1927574, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:953e720b:::smithi03829856-45:head, version: 71'60, data_included: [1823397~664039], data_size: 664039, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:953e720b:::smithi03829856-45:head@71'60, size: 2487436, copy_subset: [(0, 2487436)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2487436, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1823397, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:953e720b:::smithi03829856-45:b, version: 77'61, data_included: [1927574~595528,2570780~420833], data_size: 1016361, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:953e720b:::smithi03829856-45:b@77'61, size: 2991613, copy_subset: [(0, 2991613)], clone_subset: {}, snapset: 10=[]:{b: [b, 6], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2991613, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1927574, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:00.540 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:00.151568+0000", 2024-03-19T11:31:00.540 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.30485456500000002, 2024-03-19T11:31:00.541 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015354197999999999, 2024-03-19T11:31:00.541 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:00.541 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:00.541 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:00.541 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.541 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:00.541 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.151568+0000", 2024-03-19T11:31:00.541 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:00.541 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.541 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.541 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:00.541 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.151568+0000", 2024-03-19T11:31:00.541 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:00.542 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.542 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.542 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:00.542 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.151569+0000", 2024-03-19T11:31:00.542 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.17e-06 2024-03-19T11:31:00.542 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.542 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.542 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:00.542 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.156296+0000", 2024-03-19T11:31:00.542 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0047264619999999999 2024-03-19T11:31:00.542 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.543 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.543 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:00.543 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.156334+0000", 2024-03-19T11:31:00.543 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8152000000000003e-05 2024-03-19T11:31:00.543 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.543 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.543 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:00.543 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.156370+0000", 2024-03-19T11:31:00.543 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6304999999999999e-05 2024-03-19T11:31:00.543 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.543 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.543 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:00.543 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.157994+0000", 2024-03-19T11:31:00.544 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0016235399999999999 2024-03-19T11:31:00.544 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.544 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.544 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:00.544 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.158017+0000", 2024-03-19T11:31:00.544 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3197000000000002e-05 2024-03-19T11:31:00.544 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:00.544 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:00.544 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:00.544 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.166922+0000", 2024-03-19T11:31:00.544 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0089053719999999999 2024-03-19T11:31:00.544 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:00.544 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:00.545 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:00.545 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:00.545 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:00.545 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:00.546 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-19T11:31:00.547 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:31:00.547 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:31:00.547 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:31:00.548 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:31:00.548 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:31:00.548 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:31:00.549 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:31:00.549 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:31:00.549 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:31:00.550 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:31:00.550 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:00.554+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:00.557 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:31:00.558 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:31:00.578 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:31:00.579 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:31:00.599 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:31:00.599 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:31:00.659 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:00.670+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:00.679 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:00.679 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:00.691 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-19T11:31:00.748 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:00.759+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:00.849 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:00.859+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:00.857 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:00.857 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:00.871 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-19T11:31:00.950 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:00.958+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:00.990 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:00.990 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:01.005 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-19T11:31:01.040 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.3 to 1 (10000) 2024-03-19T11:31:01.044 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:01.054+0000 7f6712f92640 -1 osd.2 81 handle_osd_map injecting map crc failure 2024-03-19T11:31:01.047 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:01.058+0000 7f6712f92640 -1 osd.2 81 handle_osd_map injecting map crc failure 2024-03-19T11:31:01.048 INFO:tasks.rados.rados.0.smithi038.stdout:1172: done (4 left) 2024-03-19T11:31:01.048 INFO:tasks.rados.rados.0.smithi038.stdout:1173: done (3 left) 2024-03-19T11:31:01.048 INFO:tasks.rados.rados.0.smithi038.stdout:1174: done (2 left) 2024-03-19T11:31:01.048 INFO:tasks.rados.rados.0.smithi038.stdout:1175: done (1 left) 2024-03-19T11:31:01.048 INFO:tasks.rados.rados.0.smithi038.stdout:1176: done (0 left) 2024-03-19T11:31:01.048 INFO:tasks.rados.rados.0.smithi038.stdout:1177: write (excl) oid 26 current snap is 22 2024-03-19T11:31:01.048 INFO:tasks.rados.rados.0.smithi038.stdout:1177: seq_num 189 ranges {776523=614533,1976979=757720} 2024-03-19T11:31:01.050 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:01.058+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:01.056 INFO:tasks.rados.rados.0.smithi038.stdout:1177: writing smithi03829856-26 from 776523 to 1391056 tid 1 2024-03-19T11:31:01.066 INFO:tasks.rados.rados.0.smithi038.stdout:1177: writing smithi03829856-26 from 1976979 to 2734699 tid 2 2024-03-19T11:31:01.066 INFO:tasks.rados.rados.0.smithi038.stdout:1178: read oid 9 snap -1 2024-03-19T11:31:01.066 INFO:tasks.rados.rados.0.smithi038.stdout:1178: expect (ObjNum 152 snap 17 seq_num 152) 2024-03-19T11:31:01.066 INFO:tasks.rados.rados.0.smithi038.stdout:1179: read oid 31 snap 4 2024-03-19T11:31:01.066 INFO:tasks.rados.rados.0.smithi038.stdout:1179: expect (ObjNum 56 snap 1 seq_num 56) 2024-03-19T11:31:01.066 INFO:tasks.rados.rados.0.smithi038.stdout:1180: write (excl) oid 23 current snap is 22 2024-03-19T11:31:01.066 INFO:tasks.rados.rados.0.smithi038.stdout:1180: seq_num 190 ranges {793330=689782,2035461=617731,3101545=91785} 2024-03-19T11:31:01.072 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-03-19T11:31:01.074 INFO:tasks.rados.rados.0.smithi038.stdout:1180: writing smithi03829856-23 from 793330 to 1483112 tid 1 2024-03-19T11:31:01.082 INFO:tasks.rados.rados.0.smithi038.stdout:1180: writing smithi03829856-23 from 2035461 to 2653192 tid 2 2024-03-19T11:31:01.085 INFO:tasks.rados.rados.0.smithi038.stdout:1180: writing smithi03829856-23 from 3101545 to 3193330 tid 3 2024-03-19T11:31:01.086 INFO:tasks.rados.rados.0.smithi038.stdout:1181: rollback oid 13 current snap is 22 2024-03-19T11:31:01.086 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 13 to 9 2024-03-19T11:31:01.086 INFO:tasks.rados.rados.0.smithi038.stdout:1177: finishing write tid 1 to smithi03829856-26 2024-03-19T11:31:01.086 INFO:tasks.rados.rados.0.smithi038.stdout:1178: expect (ObjNum 152 snap 17 seq_num 152) 2024-03-19T11:31:01.151 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:01.163+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:01.188 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:01.188 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:01.201 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-19T11:31:01.252 INFO:tasks.rados.rados.0.smithi038.stdout:1177: finishing write tid 2 to smithi03829856-26 2024-03-19T11:31:01.252 INFO:tasks.rados.rados.0.smithi038.stdout:1179: expect (ObjNum 56 snap 1 seq_num 56) 2024-03-19T11:31:01.252 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:01.262+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:01.352 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:01.362+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:01.355 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:01.355 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:01.368 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-19T11:31:01.388 INFO:tasks.rados.rados.0.smithi038.stdout:1177: finishing write tid 3 to smithi03829856-26 2024-03-19T11:31:01.388 INFO:tasks.rados.rados.0.smithi038.stdout:1177: finishing write tid 4 to smithi03829856-26 2024-03-19T11:31:01.388 INFO:tasks.rados.rados.0.smithi038.stdout:1177: oid 26 updating version 0 to 65 2024-03-19T11:31:01.388 INFO:tasks.rados.rados.0.smithi038.stdout:1177: oid 26 version 65 is already newer than 64 2024-03-19T11:31:01.388 INFO:tasks.rados.rados.0.smithi038.stdout:1177: oid 26 updating version 65 to 66 2024-03-19T11:31:01.389 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 26 v 66 (ObjNum 189 snap 22 seq_num 189) dirty exists 2024-03-19T11:31:01.389 INFO:tasks.rados.rados.0.smithi038.stdout:1177: left oid 26 (ObjNum 189 snap 22 seq_num 189) 2024-03-19T11:31:01.389 INFO:tasks.rados.rados.0.smithi038.stdout:1177: done (4 left) 2024-03-19T11:31:01.389 INFO:tasks.rados.rados.0.smithi038.stdout:1178: done (3 left) 2024-03-19T11:31:01.389 INFO:tasks.rados.rados.0.smithi038.stdout:1179: done (2 left) 2024-03-19T11:31:01.389 INFO:tasks.rados.rados.0.smithi038.stdout:1182: set_chunk oid 12 target oid 2024-03-19T11:31:01.398 INFO:tasks.rados.rados.0.smithi038.stdout:1182: set_chunk oid 12 offset: 1093120 length: 799232 target oid 1ae57675621194c1e425829fba4b2c9aedd4521f1daadc45d2cce06141db8fe4 offset: 0 snap -1 2024-03-19T11:31:01.398 INFO:tasks.rados.rados.0.smithi038.stdout:1183: read oid 18 snap -1 2024-03-19T11:31:01.398 INFO:tasks.rados.rados.0.smithi038.stdout:1183: expect (ObjNum 106 snap 11 seq_num 106) 2024-03-19T11:31:01.398 INFO:tasks.rados.rados.0.smithi038.stdout:1184: snap_remove snap 11 2024-03-19T11:31:01.454 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:01.462+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:01.502 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:01.502 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:01.515 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-19T11:31:01.560 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:01.570+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:01.646 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:01.647 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:01.656 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:01.666+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:01.661 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-19T11:31:01.758 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:01.767+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:01.784 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:01.784 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:01.799 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-19T11:31:01.887 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:01.899+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:01.960 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:01.970+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:01.966 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:01.967 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:01.981 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-19T11:31:02.054 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.4 to 1 (10000) 2024-03-19T11:31:02.060 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:02.070+0000 7fa5e08db640 -1 osd.3 82 handle_osd_map injecting map crc failure 2024-03-19T11:31:02.066 INFO:tasks.rados.rados.0.smithi038.stdout:1184: done (4 left) 2024-03-19T11:31:02.066 INFO:tasks.rados.rados.0.smithi038.stdout:1185: write (excl) oid 29 current snap is 22 2024-03-19T11:31:02.066 INFO:tasks.rados.rados.0.smithi038.stdout:1185: seq_num 191 ranges {410137=481384,1410296=761389,2728968=481169} 2024-03-19T11:31:02.072 INFO:tasks.rados.rados.0.smithi038.stdout:1185: writing smithi03829856-29 from 410137 to 891521 tid 1 2024-03-19T11:31:02.074 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:02.079+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:02.081 INFO:tasks.rados.rados.0.smithi038.stdout:1185: writing smithi03829856-29 from 1410296 to 2171685 tid 2 2024-03-19T11:31:02.087 INFO:tasks.rados.rados.0.smithi038.stdout:1185: writing smithi03829856-29 from 2728968 to 3210137 tid 3 2024-03-19T11:31:02.088 INFO:tasks.rados.rados.0.smithi038.stdout:1186: write oid 27 current snap is 22 2024-03-19T11:31:02.088 INFO:tasks.rados.rados.0.smithi038.stdout:1186: seq_num 192 ranges {426944=556633,1468778=621400,2872744=354200} 2024-03-19T11:31:02.089 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-03-19T11:31:02.093 INFO:tasks.rados.rados.0.smithi038.stdout:1186: writing smithi03829856-27 from 426944 to 983577 tid 1 2024-03-19T11:31:02.095 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-03-19T11:31:02.099 INFO:tasks.rados.rados.0.smithi038.stdout:1186: writing smithi03829856-27 from 1468778 to 2090178 tid 2 2024-03-19T11:31:02.109 INFO:tasks.rados.rados.0.smithi038.stdout:1186: writing smithi03829856-27 from 2872744 to 3226944 tid 3 2024-03-19T11:31:02.109 INFO:tasks.rados.rados.0.smithi038.stdout:1187: set_chunk oid 15 target oid 2024-03-19T11:31:02.121 INFO:tasks.rados.rados.0.smithi038.stdout:1187: set_chunk oid 15 offset: 492032 length: 616960 target oid f16d8e416feb0e0d8e6dae8325f097c25332eafecea36f0f4aaeb6de8ea3ec42 offset: 0 snap 9 2024-03-19T11:31:02.121 INFO:tasks.rados.rados.0.smithi038.stdout:1188: set_chunk oid 4 target oid 2024-03-19T11:31:02.159 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:02.159 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:02.162 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:02.171+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:02.174 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-19T11:31:02.263 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:02.270+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:02.337 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:02.338 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:02.353 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-19T11:31:02.399 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:02.411+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:02.465 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:02.474+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:02.470 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:02.470 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:02.484 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-19T11:31:02.565 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:02.574+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:02.618 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-19T11:31:02.649 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:02.649 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:02.663 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-19T11:31:02.666 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:02.675+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:02.767 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:02.774+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:02.769 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:31:02.770 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:31:02.785 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:02.785 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:02.786 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:31:02.786 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:31:02.799 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-19T11:31:02.802 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:31:02.802 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:31:02.819 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:31:02.819 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:31:02.836 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:31:02.837 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:31:02.857 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:31:02.858 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:31:02.870 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:02.879+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:02.877 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:31:02.877 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:31:02.892 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:31:02.892 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:31:02.925 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:02.925 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:31:02.925 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:31:02.925 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:31:02.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.925 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:30 4.5 4:ac9cd3a3:::smithi03829856-8:head [write 1776668~340981] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:31:02.925 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:04.777922+0000", 2024-03-19T11:31:02.926 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 58.154809634000003, 2024-03-19T11:31:02.926 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015469894, 2024-03-19T11:31:02.926 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:02.926 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:02.926 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:02.926 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:02.926 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:02.926 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 30 2024-03-19T11:31:02.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.927 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:02.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.927 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:02.927 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.777922+0000", 2024-03-19T11:31:02.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:02.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.927 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:02.928 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.777922+0000", 2024-03-19T11:31:02.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:02.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.928 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:02.928 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.777924+0000", 2024-03-19T11:31:02.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3999999999999999e-06 2024-03-19T11:31:02.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:02.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778461+0000", 2024-03-19T11:31:02.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00053698299999999999 2024-03-19T11:31:02.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:02.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778487+0000", 2024-03-19T11:31:02.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5947000000000001e-05 2024-03-19T11:31:02.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:02.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778502+0000", 2024-03-19T11:31:02.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4256e-05 2024-03-19T11:31:02.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.931 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:02.931 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.781978+0000", 2024-03-19T11:31:02.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00347693 2024-03-19T11:31:02.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.932 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:02.932 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.782196+0000", 2024-03-19T11:31:02.932 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021726499999999999 2024-03-19T11:31:02.932 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.932 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.932 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:02.932 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.782572+0000", 2024-03-19T11:31:02.932 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00037635000000000001 2024-03-19T11:31:02.933 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.933 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:02.933 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.786960+0000", 2024-03-19T11:31:02.933 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043882130000000002 2024-03-19T11:31:02.933 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.934 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:02.934 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793113+0000", 2024-03-19T11:31:02.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061530760000000004 2024-03-19T11:31:02.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.934 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:02.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793276+0000", 2024-03-19T11:31:02.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000162652 2024-03-19T11:31:02.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:02.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793392+0000", 2024-03-19T11:31:02.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115822 2024-03-19T11:31:02.935 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:02.936 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:02.936 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:02.936 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.936 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.936 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:31 4.5 4:ac9cd3a3:::smithi03829856-8:head [setxattr _header (55) in=7b,truncate 2117649] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:31:02.936 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:04.778533+0000", 2024-03-19T11:31:02.936 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 58.154198499000003, 2024-03-19T11:31:02.936 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015182533, 2024-03-19T11:31:02.937 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:02.937 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:02.937 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:02.937 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:02.937 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:02.937 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 31 2024-03-19T11:31:02.937 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.937 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:02.938 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.938 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:02.938 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778533+0000", 2024-03-19T11:31:02.938 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:02.938 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.938 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.938 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:02.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778533+0000", 2024-03-19T11:31:02.939 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:02.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.940 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.940 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:02.940 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778534+0000", 2024-03-19T11:31:02.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1069999999999999e-06 2024-03-19T11:31:02.940 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.940 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.941 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:02.941 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778541+0000", 2024-03-19T11:31:02.941 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2529999999999998e-06 2024-03-19T11:31:02.941 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.942 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:02.942 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778558+0000", 2024-03-19T11:31:02.942 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6290000000000002e-05 2024-03-19T11:31:02.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:02.943 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778568+0000", 2024-03-19T11:31:02.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0360000000000001e-05 2024-03-19T11:31:02.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.944 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:02.944 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.785511+0000", 2024-03-19T11:31:02.944 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00694306 2024-03-19T11:31:02.944 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.945 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:02.945 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.785710+0000", 2024-03-19T11:31:02.945 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019837199999999999 2024-03-19T11:31:02.945 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.946 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:02.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.785929+0000", 2024-03-19T11:31:02.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021960799999999999 2024-03-19T11:31:02.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:02.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.787281+0000", 2024-03-19T11:31:02.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001352313 2024-03-19T11:31:02.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.948 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:02.948 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793492+0000", 2024-03-19T11:31:02.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062107760000000003 2024-03-19T11:31:02.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.948 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:02.949 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793578+0000", 2024-03-19T11:31:02.949 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5711999999999995e-05 2024-03-19T11:31:02.949 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.949 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:02.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793716+0000", 2024-03-19T11:31:02.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000137682 2024-03-19T11:31:02.950 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:02.950 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:02.951 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:02.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.951 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:32 4.5 4:ac9cd3a3:::smithi03829856-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:31:02.951 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:04.778582+0000", 2024-03-19T11:31:02.951 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 58.154149984999997, 2024-03-19T11:31:02.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015761827999999999, 2024-03-19T11:31:02.952 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:02.952 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:02.952 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:02.952 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:02.952 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:02.953 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 32 2024-03-19T11:31:02.953 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.953 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:02.953 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.953 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:02.953 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778582+0000", 2024-03-19T11:31:02.954 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:02.954 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.954 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:02.954 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778582+0000", 2024-03-19T11:31:02.954 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:02.954 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:02.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778583+0000", 2024-03-19T11:31:02.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.026e-06 2024-03-19T11:31:02.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:02.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778587+0000", 2024-03-19T11:31:02.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7870000000000001e-06 2024-03-19T11:31:02.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:02.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778599+0000", 2024-03-19T11:31:02.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1623e-05 2024-03-19T11:31:02.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.956 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:02.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778608+0000", 2024-03-19T11:31:02.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7399999999999993e-06 2024-03-19T11:31:02.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.957 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:02.957 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.787009+0000", 2024-03-19T11:31:02.957 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0084010490000000007 2024-03-19T11:31:02.957 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.957 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.957 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-03-19T11:31:02.957 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.787202+0000", 2024-03-19T11:31:02.957 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019309500000000001 2024-03-19T11:31:02.957 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.957 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.958 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:02.958 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793832+0000", 2024-03-19T11:31:02.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.006630052 2024-03-19T11:31:02.958 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.958 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.958 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:02.958 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.794048+0000", 2024-03-19T11:31:02.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000215505 2024-03-19T11:31:02.959 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.959 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.959 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:02.959 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.794343+0000", 2024-03-19T11:31:02.959 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029595100000000003 2024-03-19T11:31:02.959 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:02.959 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:02.959 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:02.959 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.960 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.960 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:856 4.5 4:ac9cd3a3:::smithi03829856-8:head [write 608420~529411] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:31:02.960 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.781425+0000", 2024-03-19T11:31:02.960 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 54.151306798999997, 2024-03-19T11:31:02.960 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014634677, 2024-03-19T11:31:02.960 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:02.960 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:02.961 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:02.961 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:02.961 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:02.961 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 856 2024-03-19T11:31:02.961 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.961 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:02.961 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.961 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:02.962 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.781425+0000", 2024-03-19T11:31:02.962 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:02.962 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.962 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.962 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:02.962 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.781425+0000", 2024-03-19T11:31:02.962 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:02.962 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.963 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:02.963 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.781428+0000", 2024-03-19T11:31:02.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.534e-06 2024-03-19T11:31:02.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.963 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:02.963 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.783096+0000", 2024-03-19T11:31:02.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001667897 2024-03-19T11:31:02.964 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.964 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.964 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:02.964 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.783138+0000", 2024-03-19T11:31:02.964 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1946000000000001e-05 2024-03-19T11:31:02.964 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.964 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.965 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:02.965 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.783163+0000", 2024-03-19T11:31:02.965 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4443e-05 2024-03-19T11:31:02.965 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.965 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.965 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:02.965 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.783333+0000", 2024-03-19T11:31:02.965 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017027899999999999 2024-03-19T11:31:02.965 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.966 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:02.966 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.783503+0000", 2024-03-19T11:31:02.966 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000170242 2024-03-19T11:31:02.966 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.966 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:02.966 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.783917+0000", 2024-03-19T11:31:02.966 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00041371000000000002 2024-03-19T11:31:02.967 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.967 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.967 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:02.967 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.795709+0000", 2024-03-19T11:31:02.967 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011791693000000001 2024-03-19T11:31:02.967 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.967 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.967 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:02.967 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.795850+0000", 2024-03-19T11:31:02.968 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000141695 2024-03-19T11:31:02.968 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.968 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.968 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:02.968 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.795940+0000", 2024-03-19T11:31:02.968 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9705999999999998e-05 2024-03-19T11:31:02.968 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.968 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.969 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:02.969 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.796059+0000", 2024-03-19T11:31:02.969 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011953199999999999 2024-03-19T11:31:02.969 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:02.969 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:02.969 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:02.969 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.969 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.969 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:891 4.5 4:af9d0d56:::smithi03829856-20:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:31:02.970 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:09.286410+0000", 2024-03-19T11:31:02.970 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 53.646321653000001, 2024-03-19T11:31:02.970 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035191405000000002, 2024-03-19T11:31:02.970 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:02.970 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:02.970 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:02.970 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:02.970 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:02.971 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 891 2024-03-19T11:31:02.971 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.971 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:02.971 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.971 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:02.971 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286410+0000", 2024-03-19T11:31:02.971 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:02.972 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.972 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.972 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:02.972 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286410+0000", 2024-03-19T11:31:02.972 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:02.972 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.972 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.972 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:02.972 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286413+0000", 2024-03-19T11:31:02.973 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2660000000000002e-06 2024-03-19T11:31:02.973 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.973 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.973 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:02.973 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286426+0000", 2024-03-19T11:31:02.973 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.258e-05 2024-03-19T11:31:02.973 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.973 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.973 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:02.974 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286460+0000", 2024-03-19T11:31:02.974 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3943999999999999e-05 2024-03-19T11:31:02.974 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.974 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.974 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:02.974 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286477+0000", 2024-03-19T11:31:02.974 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6799999999999998e-05 2024-03-19T11:31:02.974 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.975 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.975 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:02.975 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286605+0000", 2024-03-19T11:31:02.975 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000128892 2024-03-19T11:31:02.975 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.975 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.975 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:02.976 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286769+0000", 2024-03-19T11:31:02.976 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016359799999999999 2024-03-19T11:31:02.976 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.976 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.976 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:02.977 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.321601+0000", 2024-03-19T11:31:02.977 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.034832324999999997 2024-03-19T11:31:02.977 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:02.977 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:02.977 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:02.977 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.977 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.977 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1015 4.5 4:af9d0d56:::smithi03829856-20:head [tier-promote] snapc 6=[6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:31:02.978 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:12.761855+0000", 2024-03-19T11:31:02.978 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 50.170876127, 2024-03-19T11:31:02.978 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026760430000000002, 2024-03-19T11:31:02.978 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:02.978 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:02.978 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:02.978 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:02.978 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:02.979 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1015 2024-03-19T11:31:02.979 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.979 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:02.979 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.979 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:02.979 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761855+0000", 2024-03-19T11:31:02.979 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:02.979 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.979 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.979 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:02.980 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761855+0000", 2024-03-19T11:31:02.980 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:02.980 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.980 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.980 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:02.980 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761858+0000", 2024-03-19T11:31:02.980 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8140000000000001e-06 2024-03-19T11:31:02.980 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.980 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.981 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:02.981 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761872+0000", 2024-03-19T11:31:02.981 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3813e-05 2024-03-19T11:31:02.981 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.981 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.981 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:02.982 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761905+0000", 2024-03-19T11:31:02.982 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2642999999999999e-05 2024-03-19T11:31:02.982 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.982 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.982 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:02.982 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761925+0000", 2024-03-19T11:31:02.982 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0489e-05 2024-03-19T11:31:02.983 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.983 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.983 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:02.983 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.762070+0000", 2024-03-19T11:31:02.983 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000145153 2024-03-19T11:31:02.983 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.983 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.983 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:02.983 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.762365+0000", 2024-03-19T11:31:02.983 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029482100000000002 2024-03-19T11:31:02.983 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.983 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.984 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:02.984 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.788616+0000", 2024-03-19T11:31:02.984 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026250697 2024-03-19T11:31:02.984 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:02.984 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:02.984 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:02.984 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.984 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.984 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1121 4.5 4:aa3303a5:::smithi03829856-17:head [tier-promote] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:31:02.984 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:14.164281+0000", 2024-03-19T11:31:02.984 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.768450882000003, 2024-03-19T11:31:02.984 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023896435000000001, 2024-03-19T11:31:02.984 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:02.985 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:02.985 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:02.985 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:02.985 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:02.985 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1121 2024-03-19T11:31:02.985 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.985 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:02.985 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.985 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:02.986 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164281+0000", 2024-03-19T11:31:02.986 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:02.986 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.986 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.986 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:02.986 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164281+0000", 2024-03-19T11:31:02.986 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:02.986 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.986 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.986 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:02.986 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164283+0000", 2024-03-19T11:31:02.986 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8500000000000001e-06 2024-03-19T11:31:02.987 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.987 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.987 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:02.987 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164291+0000", 2024-03-19T11:31:02.987 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.14e-06 2024-03-19T11:31:02.987 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.987 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.987 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:02.987 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164313+0000", 2024-03-19T11:31:02.987 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.207e-05 2024-03-19T11:31:02.987 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.988 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.988 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:02.988 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164328+0000", 2024-03-19T11:31:02.988 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5403000000000001e-05 2024-03-19T11:31:02.988 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.988 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.988 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:02.988 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.167981+0000", 2024-03-19T11:31:02.988 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036529879999999998 2024-03-19T11:31:02.988 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.988 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.988 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:02.988 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.168274+0000", 2024-03-19T11:31:02.989 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029313100000000002 2024-03-19T11:31:02.989 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.989 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.989 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:02.989 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.188177+0000", 2024-03-19T11:31:02.989 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019902853000000002 2024-03-19T11:31:02.989 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:02.989 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:02.989 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:02.990 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.990 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.990 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1302 4.15 4:aa3303a5:::smithi03829856-17:head [write 413472~620140,stat] snapc b=[b,a,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-03-19T11:31:02.990 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:22.966426+0000", 2024-03-19T11:31:02.990 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 39.966305149, 2024-03-19T11:31:02.990 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018043513000000001, 2024-03-19T11:31:02.990 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:02.990 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:02.991 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:02.991 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:02.991 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:02.991 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1302 2024-03-19T11:31:02.991 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.991 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:02.991 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.991 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:02.991 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.966426+0000", 2024-03-19T11:31:02.992 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:02.992 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.992 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.992 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:02.992 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.966426+0000", 2024-03-19T11:31:02.992 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:02.992 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.993 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.993 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:02.993 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.966429+0000", 2024-03-19T11:31:02.993 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3829999999999999e-06 2024-03-19T11:31:02.993 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.993 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.993 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:02.993 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972117+0000", 2024-03-19T11:31:02.993 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0056885750000000004 2024-03-19T11:31:02.994 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.994 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.994 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:02.994 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972156+0000", 2024-03-19T11:31:02.994 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8878000000000003e-05 2024-03-19T11:31:02.994 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.994 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.995 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:02.995 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972184+0000", 2024-03-19T11:31:02.995 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7509999999999999e-05 2024-03-19T11:31:02.995 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.995 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.995 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:02.995 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972309+0000", 2024-03-19T11:31:02.995 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012525899999999999 2024-03-19T11:31:02.995 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.995 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.995 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:02.995 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972566+0000", 2024-03-19T11:31:02.995 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025740799999999999 2024-03-19T11:31:02.996 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.996 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.996 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:02.996 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972899+0000", 2024-03-19T11:31:02.996 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033238799999999998 2024-03-19T11:31:02.996 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.996 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.997 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:02.997 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984088+0000", 2024-03-19T11:31:02.997 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011189466 2024-03-19T11:31:02.997 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.997 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.997 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:02.997 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984138+0000", 2024-03-19T11:31:02.997 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9666999999999997e-05 2024-03-19T11:31:02.997 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.997 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.998 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:02.998 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984329+0000", 2024-03-19T11:31:02.998 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019086799999999999 2024-03-19T11:31:02.998 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.998 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.998 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:02.998 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984470+0000", 2024-03-19T11:31:02.998 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000141111 2024-03-19T11:31:02.998 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:02.998 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:02.999 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:02.999 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:02.999 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:02.999 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1303 4.15 4:aa3303a5:::smithi03829856-17:head [write 1818036~510700] snapc b=[b,a,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-03-19T11:31:02.999 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:22.976856+0000", 2024-03-19T11:31:02.999 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 39.955876039000003, 2024-03-19T11:31:02.999 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015045703000000001, 2024-03-19T11:31:02.999 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:02.999 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:03.000 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:03.000 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:03.000 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:03.000 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1303 2024-03-19T11:31:03.000 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.000 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:03.000 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.000 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:03.000 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.976856+0000", 2024-03-19T11:31:03.001 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.001 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.001 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.001 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:03.001 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.976856+0000", 2024-03-19T11:31:03.001 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.001 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.001 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.001 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:03.002 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.976859+0000", 2024-03-19T11:31:03.002 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3629999999999998e-06 2024-03-19T11:31:03.002 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.002 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.002 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:03.002 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.977533+0000", 2024-03-19T11:31:03.002 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00067380800000000002 2024-03-19T11:31:03.002 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.003 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.003 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:03.003 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.977573+0000", 2024-03-19T11:31:03.003 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9758000000000001e-05 2024-03-19T11:31:03.003 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.003 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.003 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:03.003 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.977602+0000", 2024-03-19T11:31:03.004 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9856e-05 2024-03-19T11:31:03.004 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.004 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.004 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:03.004 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.979943+0000", 2024-03-19T11:31:03.004 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0023406400000000002 2024-03-19T11:31:03.004 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.004 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.004 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:03.005 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.980238+0000", 2024-03-19T11:31:03.005 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029488000000000002 2024-03-19T11:31:03.005 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.005 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.005 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:03.005 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.980734+0000", 2024-03-19T11:31:03.005 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00049647500000000002 2024-03-19T11:31:03.005 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.005 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.006 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:03.006 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.991761+0000", 2024-03-19T11:31:03.006 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011026515000000001 2024-03-19T11:31:03.006 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.006 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.006 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:03.007 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.991777+0000", 2024-03-19T11:31:03.007 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5727999999999999e-05 2024-03-19T11:31:03.007 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.007 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.007 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:03.007 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.991840+0000", 2024-03-19T11:31:03.007 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3091999999999998e-05 2024-03-19T11:31:03.008 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.008 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.008 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:03.008 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.991901+0000", 2024-03-19T11:31:03.008 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1587999999999994e-05 2024-03-19T11:31:03.008 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.008 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:03.008 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.008 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.008 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.008 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1468 4.a 4:567251f3:::smithi03829856-44:head [write 1603467~416361] snapc c=[c,b,a,6,5] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-03-19T11:31:03.008 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.083922+0000", 2024-03-19T11:31:03.009 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 35.848810016000002, 2024-03-19T11:31:03.009 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014508549000000001, 2024-03-19T11:31:03.009 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:03.009 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:03.009 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:03.009 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:03.009 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:03.009 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1468 2024-03-19T11:31:03.009 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.010 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:03.010 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.010 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:03.010 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.083922+0000", 2024-03-19T11:31:03.010 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.010 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.010 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.010 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:03.010 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.083922+0000", 2024-03-19T11:31:03.010 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.010 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.011 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.011 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:03.011 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.083924+0000", 2024-03-19T11:31:03.011 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2800000000000002e-06 2024-03-19T11:31:03.011 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.011 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.011 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:03.011 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.084498+0000", 2024-03-19T11:31:03.012 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057414900000000004 2024-03-19T11:31:03.012 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.012 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.012 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:03.012 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.084529+0000", 2024-03-19T11:31:03.012 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1080000000000001e-05 2024-03-19T11:31:03.012 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.012 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.012 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:03.013 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.084544+0000", 2024-03-19T11:31:03.013 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4498000000000001e-05 2024-03-19T11:31:03.013 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.013 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.013 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:03.013 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.088993+0000", 2024-03-19T11:31:03.013 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044497260000000002 2024-03-19T11:31:03.013 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.013 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.014 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:03.014 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.089152+0000", 2024-03-19T11:31:03.014 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015833199999999999 2024-03-19T11:31:03.014 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.014 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.014 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-19T11:31:03.014 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.089419+0000", 2024-03-19T11:31:03.015 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026691799999999999 2024-03-19T11:31:03.015 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.015 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.015 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:03.015 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.092883+0000", 2024-03-19T11:31:03.015 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034646590000000001 2024-03-19T11:31:03.015 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.015 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.015 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:03.016 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.098276+0000", 2024-03-19T11:31:03.016 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053924680000000001 2024-03-19T11:31:03.016 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.016 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.016 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:03.016 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.098354+0000", 2024-03-19T11:31:03.016 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8763999999999993e-05 2024-03-19T11:31:03.016 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.016 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.017 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:03.017 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.098430+0000", 2024-03-19T11:31:03.017 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.5674999999999996e-05 2024-03-19T11:31:03.017 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.017 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:03.017 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.017 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.018 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.020 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.1 50/49 [PushOp(4:866f3dcc:::smithi03829856-31:7, version: 38'44, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@38'44, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 45'62, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@45'62, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 46'68, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@46'68, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:7, version: 47'71, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@47'71, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 47'72, data_included: [440350~816629,1309629~231947], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@47'72, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:03.020 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.959457+0000", 2024-03-19T11:31:03.020 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.973274732, 2024-03-19T11:31:03.020 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018416666000000002, 2024-03-19T11:31:03.021 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:03.021 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:03.021 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:03.021 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.021 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:03.021 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.959457+0000", 2024-03-19T11:31:03.021 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.022 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.022 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.022 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:03.022 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.959457+0000", 2024-03-19T11:31:03.022 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.022 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.022 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.023 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:03.023 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.959458+0000", 2024-03-19T11:31:03.023 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4899999999999999e-06 2024-03-19T11:31:03.023 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.023 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.023 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:03.023 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.962822+0000", 2024-03-19T11:31:03.023 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0033640620000000001 2024-03-19T11:31:03.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.024 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:03.024 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.962893+0000", 2024-03-19T11:31:03.024 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1060000000000001e-05 2024-03-19T11:31:03.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.024 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:03.025 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.962977+0000", 2024-03-19T11:31:03.025 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3311000000000002e-05 2024-03-19T11:31:03.025 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.025 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.025 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:03.025 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963208+0000", 2024-03-19T11:31:03.025 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023121200000000001 2024-03-19T11:31:03.025 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.026 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.026 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:03.026 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963245+0000", 2024-03-19T11:31:03.026 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6844999999999997e-05 2024-03-19T11:31:03.026 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.026 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.026 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:03.026 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.977874+0000", 2024-03-19T11:31:03.026 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014628686 2024-03-19T11:31:03.026 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.027 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:03.027 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.027 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.027 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.027 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.1 e50,49 4:8401c1f0:::smithi03829856-43:8,46'66,4:866f3dcc:::smithi03829856-31:2,47'73)", 2024-03-19T11:31:03.027 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.963027+0000", 2024-03-19T11:31:03.027 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.969705114999996, 2024-03-19T11:31:03.027 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015116788000000001, 2024-03-19T11:31:03.027 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:03.028 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:03.028 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:03.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.028 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:03.028 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963027+0000", 2024-03-19T11:31:03.028 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.028 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.029 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.029 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:03.029 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963027+0000", 2024-03-19T11:31:03.029 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.029 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.029 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.029 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:03.029 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963028+0000", 2024-03-19T11:31:03.030 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.268e-06 2024-03-19T11:31:03.030 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.030 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.030 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:03.030 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963036+0000", 2024-03-19T11:31:03.030 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.926e-06 2024-03-19T11:31:03.030 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.030 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.031 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:03.031 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963050+0000", 2024-03-19T11:31:03.031 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4082e-05 2024-03-19T11:31:03.031 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.031 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.031 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:03.031 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963062+0000", 2024-03-19T11:31:03.031 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2415000000000001e-05 2024-03-19T11:31:03.031 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.032 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.032 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:03.032 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.977954+0000", 2024-03-19T11:31:03.032 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014891572 2024-03-19T11:31:03.032 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.032 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.032 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:03.032 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.977965+0000", 2024-03-19T11:31:03.032 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1654000000000001e-05 2024-03-19T11:31:03.032 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.033 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.033 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:03.033 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.978143+0000", 2024-03-19T11:31:03.033 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017787099999999999 2024-03-19T11:31:03.033 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.034 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:03.034 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.034 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.034 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.034 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.11 50/49 [PushOp(4:8d3763f9:::smithi03829856-29:head, version: 47'60, data_included: [3364276~232389], data_size: 232389, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8d3763f9:::smithi03829856-29:head@47'60, size: 3596665, copy_subset: [(0, 3596665)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3596665, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3364276, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:03.034 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.964363+0000", 2024-03-19T11:31:03.034 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.968368448, 2024-03-19T11:31:03.034 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014531759999999999, 2024-03-19T11:31:03.034 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:03.034 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:03.035 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:03.035 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.035 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:03.035 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.964363+0000", 2024-03-19T11:31:03.035 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.035 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.035 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.035 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:03.035 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.964363+0000", 2024-03-19T11:31:03.035 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.035 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.035 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.035 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:03.035 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.964364+0000", 2024-03-19T11:31:03.035 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4200000000000004e-07 2024-03-19T11:31:03.036 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.036 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:03.036 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.964447+0000", 2024-03-19T11:31:03.036 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2430999999999997e-05 2024-03-19T11:31:03.036 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.036 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:03.036 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.964468+0000", 2024-03-19T11:31:03.036 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1685999999999999e-05 2024-03-19T11:31:03.036 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.036 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:03.036 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.964483+0000", 2024-03-19T11:31:03.036 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4969e-05 2024-03-19T11:31:03.037 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.037 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.037 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:03.037 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.977934+0000", 2024-03-19T11:31:03.037 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013450963999999999 2024-03-19T11:31:03.037 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.037 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.037 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:03.037 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.977975+0000", 2024-03-19T11:31:03.037 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1125999999999999e-05 2024-03-19T11:31:03.037 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.037 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.037 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:03.037 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.978895+0000", 2024-03-19T11:31:03.037 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00091964200000000003 2024-03-19T11:31:03.038 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.038 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:03.038 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.038 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.038 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.038 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.1 50/49 [PushOp(4:866f3dcc:::smithi03829856-31:7, version: 38'44, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@38'44, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 45'62, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@45'62, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 46'68, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@46'68, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:7, version: 47'71, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@47'71, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 47'72, data_included: [1541576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@47'72, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:03.038 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.991858+0000", 2024-03-19T11:31:03.038 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.940873525000001, 2024-03-19T11:31:03.038 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014228774, 2024-03-19T11:31:03.038 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:03.038 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:03.038 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:03.038 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.038 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:03.039 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.991858+0000", 2024-03-19T11:31:03.039 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.039 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:03.039 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.991858+0000", 2024-03-19T11:31:03.039 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.039 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:03.039 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.991859+0000", 2024-03-19T11:31:03.039 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.342e-06 2024-03-19T11:31:03.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.039 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:03.040 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.994675+0000", 2024-03-19T11:31:03.040 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0028153150000000001 2024-03-19T11:31:03.040 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.040 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.040 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:03.040 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.994733+0000", 2024-03-19T11:31:03.040 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8514999999999998e-05 2024-03-19T11:31:03.040 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.040 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.040 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:03.040 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.994794+0000", 2024-03-19T11:31:03.040 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0588000000000003e-05 2024-03-19T11:31:03.040 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.040 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.040 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:03.040 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.995070+0000", 2024-03-19T11:31:03.041 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027573700000000002 2024-03-19T11:31:03.041 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.041 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.041 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:03.041 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.995100+0000", 2024-03-19T11:31:03.041 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0679000000000002e-05 2024-03-19T11:31:03.041 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.041 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.041 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:03.041 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.006087+0000", 2024-03-19T11:31:03.041 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010986598 2024-03-19T11:31:03.041 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.041 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:03.041 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.041 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.042 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.042 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2284 4.15 4:afa9f208:::smithi03829856-35:head [read 1314816~702976] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-03-19T11:31:03.042 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:53.596775+0000", 2024-03-19T11:31:03.042 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.3359566459999996, 2024-03-19T11:31:03.042 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.47260528299999999, 2024-03-19T11:31:03.042 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:03.042 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:03.042 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:03.042 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:03.042 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:03.042 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2284 2024-03-19T11:31:03.042 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.043 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:03.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.043 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:03.043 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596775+0000", 2024-03-19T11:31:03.043 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.043 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:03.043 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596775+0000", 2024-03-19T11:31:03.043 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.044 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:03.044 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596778+0000", 2024-03-19T11:31:03.044 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.174e-06 2024-03-19T11:31:03.044 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.044 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.044 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:03.044 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596794+0000", 2024-03-19T11:31:03.044 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5372000000000001e-05 2024-03-19T11:31:03.044 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.044 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.045 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:03.045 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596828+0000", 2024-03-19T11:31:03.045 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4107999999999999e-05 2024-03-19T11:31:03.045 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.045 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.045 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:03.045 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596844+0000", 2024-03-19T11:31:03.045 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6296000000000001e-05 2024-03-19T11:31:03.045 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.046 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.046 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:03.046 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596968+0000", 2024-03-19T11:31:03.046 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012383699999999999 2024-03-19T11:31:03.046 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.046 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.046 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:31:03.046 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596988+0000", 2024-03-19T11:31:03.046 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0645e-05 2024-03-19T11:31:03.046 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.046 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.046 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:03.046 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.068916+0000", 2024-03-19T11:31:03.046 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.47192744800000003 2024-03-19T11:31:03.047 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.047 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.047 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:03.047 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.069015+0000", 2024-03-19T11:31:03.047 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9500999999999995e-05 2024-03-19T11:31:03.047 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.047 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.047 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:03.047 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.069380+0000", 2024-03-19T11:31:03.047 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00036490200000000001 2024-03-19T11:31:03.047 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.047 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:03.047 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.047 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.047 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.048 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2286 4.15 4:afa9f208:::smithi03829856-35:head [assert-version v108,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e75)", 2024-03-19T11:31:03.048 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.084826+0000", 2024-03-19T11:31:03.048 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 8.8479052790000008, 2024-03-19T11:31:03.048 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.055799556, 2024-03-19T11:31:03.048 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:03.048 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:03.048 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:03.048 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:03.048 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:03.048 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2286 2024-03-19T11:31:03.048 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.048 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:03.048 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.048 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:03.048 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084826+0000", 2024-03-19T11:31:03.049 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.049 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.049 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.049 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:03.049 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084826+0000", 2024-03-19T11:31:03.049 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.049 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.049 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.049 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:03.049 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084828+0000", 2024-03-19T11:31:03.049 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1409999999999999e-06 2024-03-19T11:31:03.049 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.049 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.049 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:03.049 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084838+0000", 2024-03-19T11:31:03.050 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3740000000000004e-06 2024-03-19T11:31:03.050 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.050 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.050 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:03.050 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084859+0000", 2024-03-19T11:31:03.050 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1644000000000001e-05 2024-03-19T11:31:03.050 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.050 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.050 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:03.050 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084871+0000", 2024-03-19T11:31:03.050 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1127e-05 2024-03-19T11:31:03.050 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.050 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.050 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:03.050 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084961+0000", 2024-03-19T11:31:03.051 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0371999999999995e-05 2024-03-19T11:31:03.051 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.051 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.051 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:03.051 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.085130+0000", 2024-03-19T11:31:03.051 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016884800000000001 2024-03-19T11:31:03.051 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.051 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.051 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5,7", 2024-03-19T11:31:03.051 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.089309+0000", 2024-03-19T11:31:03.051 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041788650000000004 2024-03-19T11:31:03.051 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.051 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.051 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:03.051 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.090362+0000", 2024-03-19T11:31:03.052 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001053362 2024-03-19T11:31:03.052 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.052 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.052 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:03.052 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.090511+0000", 2024-03-19T11:31:03.052 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014887200000000001 2024-03-19T11:31:03.052 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.052 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.052 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:03.052 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.140479+0000", 2024-03-19T11:31:03.052 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.049968006000000002 2024-03-19T11:31:03.052 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.052 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.052 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:03.052 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.140534+0000", 2024-03-19T11:31:03.052 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4917000000000003e-05 2024-03-19T11:31:03.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.053 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.053 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:03.053 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.140626+0000", 2024-03-19T11:31:03.053 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2027999999999994e-05 2024-03-19T11:31:03.053 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.053 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:03.053 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.053 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.053 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1d58db11:::smithi03829856-18:6, version: 55'80, data_included: [759683~639358,1904434~409218], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:6@55'80, size: 3159683, copy_subset: [(0, 3159683)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2313652, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1d58db11:::smithi03829856-18:b, version: 58'84, data_included: [443718~955323,1564138~93253], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:b@58'84, size: 3445402, copy_subset: [(0, 3445402)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1657391, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:10, version: 66'95, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:10@66'95, size: 3764735, copy_subset: [(0, 3764735)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:12, version: 66'97, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:12@66'97, size: 2504243, copy_subset: [(0, 2504243)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:c, version: 70'102, data_included: [416807~576167,1542129~472409], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:c@70'102, size: 2857157, copy_subset: [(0, 2857157)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2014538, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:03.053 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.274749+0000", 2024-03-19T11:31:03.053 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 8.6579823480000009, 2024-03-19T11:31:03.053 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017652601, 2024-03-19T11:31:03.054 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:03.054 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:03.054 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:03.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.054 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:03.054 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.274749+0000", 2024-03-19T11:31:03.054 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.054 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.054 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:03.054 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.274749+0000", 2024-03-19T11:31:03.054 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.054 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.054 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:03.055 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.274751+0000", 2024-03-19T11:31:03.055 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8500000000000001e-06 2024-03-19T11:31:03.055 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.055 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.055 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:03.055 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.278426+0000", 2024-03-19T11:31:03.055 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036747559999999999 2024-03-19T11:31:03.055 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.055 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.055 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:03.055 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.278470+0000", 2024-03-19T11:31:03.055 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4119000000000002e-05 2024-03-19T11:31:03.055 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.055 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.055 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:03.056 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.278506+0000", 2024-03-19T11:31:03.056 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6066000000000003e-05 2024-03-19T11:31:03.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.056 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:03.056 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.278709+0000", 2024-03-19T11:31:03.056 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000202711 2024-03-19T11:31:03.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.056 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:03.056 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.278731+0000", 2024-03-19T11:31:03.056 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2657e-05 2024-03-19T11:31:03.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.056 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:03.057 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.292402+0000", 2024-03-19T11:31:03.057 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013670442 2024-03-19T11:31:03.057 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.057 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:03.057 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.057 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.057 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1dabc0f1:::smithi03829856-1:head, version: 70'103, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:head@70'103, size: 2621892, copy_subset: [(0, 2621892)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:03.057 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.280012+0000", 2024-03-19T11:31:03.057 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 8.6527197030000007, 2024-03-19T11:31:03.057 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018569209, 2024-03-19T11:31:03.057 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:03.057 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:03.057 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:03.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.058 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:03.058 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280012+0000", 2024-03-19T11:31:03.058 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.058 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.058 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.058 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:03.058 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280012+0000", 2024-03-19T11:31:03.058 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.058 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.058 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.058 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:03.058 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280013+0000", 2024-03-19T11:31:03.058 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9899999999999999e-07 2024-03-19T11:31:03.058 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.058 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.059 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:03.059 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280584+0000", 2024-03-19T11:31:03.059 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057090299999999999 2024-03-19T11:31:03.059 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.059 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.059 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:03.059 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280612+0000", 2024-03-19T11:31:03.059 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8442999999999999e-05 2024-03-19T11:31:03.059 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.059 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.059 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:03.059 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280631+0000", 2024-03-19T11:31:03.059 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8896999999999999e-05 2024-03-19T11:31:03.059 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.059 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.060 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:03.060 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.295041+0000", 2024-03-19T11:31:03.060 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014410127999999999 2024-03-19T11:31:03.060 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.060 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.060 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:03.060 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.295050+0000", 2024-03-19T11:31:03.060 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2350000000000005e-06 2024-03-19T11:31:03.060 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.060 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.060 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:03.060 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.298581+0000", 2024-03-19T11:31:03.060 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035308039999999998 2024-03-19T11:31:03.060 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.060 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:03.061 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.061 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.18 e75,74 4:1d58db11:::smithi03829856-18:c,70'104)", 2024-03-19T11:31:03.061 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.280680+0000", 2024-03-19T11:31:03.061 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 8.6520512279999995, 2024-03-19T11:31:03.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018106229000000001, 2024-03-19T11:31:03.061 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:03.061 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:03.061 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:03.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.061 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:03.061 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280680+0000", 2024-03-19T11:31:03.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:03.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280680+0000", 2024-03-19T11:31:03.062 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:03.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280681+0000", 2024-03-19T11:31:03.062 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2099999999999995e-07 2024-03-19T11:31:03.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:03.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280686+0000", 2024-03-19T11:31:03.062 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2240000000000003e-06 2024-03-19T11:31:03.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.063 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:03.063 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280698+0000", 2024-03-19T11:31:03.063 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1698999999999999e-05 2024-03-19T11:31:03.063 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.063 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.063 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:03.063 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280705+0000", 2024-03-19T11:31:03.063 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4350000000000002e-06 2024-03-19T11:31:03.063 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.063 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.063 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:03.063 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.298643+0000", 2024-03-19T11:31:03.063 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017938352000000001 2024-03-19T11:31:03.063 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.063 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.064 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:03.064 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.298656+0000", 2024-03-19T11:31:03.064 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3084999999999999e-05 2024-03-19T11:31:03.064 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.064 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.064 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:03.064 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.298787+0000", 2024-03-19T11:31:03.064 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013061299999999999 2024-03-19T11:31:03.064 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.064 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:03.064 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.064 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.064 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.064 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.5.0:143 4.18 e75/74 4:1d58db11:::smithi03829856-18:head v 75'106, mlcod=0'0)", 2024-03-19T11:31:03.064 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.282327+0000", 2024-03-19T11:31:03.065 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 8.6504048569999998, 2024-03-19T11:31:03.065 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016753244, 2024-03-19T11:31:03.065 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:03.065 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:03.065 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:03.065 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.065 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:03.065 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.282327+0000", 2024-03-19T11:31:03.065 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.065 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.065 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.065 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:03.065 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.282327+0000", 2024-03-19T11:31:03.065 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.065 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.066 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.066 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:03.066 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.282328+0000", 2024-03-19T11:31:03.066 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.488e-06 2024-03-19T11:31:03.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.066 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.066 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:03.066 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.283127+0000", 2024-03-19T11:31:03.066 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00079853699999999999 2024-03-19T11:31:03.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.066 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.066 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:03.066 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.283152+0000", 2024-03-19T11:31:03.066 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5215000000000002e-05 2024-03-19T11:31:03.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.067 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.067 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:03.067 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.283167+0000", 2024-03-19T11:31:03.067 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4635e-05 2024-03-19T11:31:03.067 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.067 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.067 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:03.067 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.292633+0000", 2024-03-19T11:31:03.067 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0094664339999999993 2024-03-19T11:31:03.067 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.067 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.067 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:03.067 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.292655+0000", 2024-03-19T11:31:03.067 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1934999999999999e-05 2024-03-19T11:31:03.067 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.068 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.068 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:03.068 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.298981+0000", 2024-03-19T11:31:03.068 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0063255129999999996 2024-03-19T11:31:03.068 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.068 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.068 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:03.068 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.299080+0000", 2024-03-19T11:31:03.068 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9487000000000005e-05 2024-03-19T11:31:03.068 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.068 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:03.068 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.068 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.068 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:03.068 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:03.069 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-19T11:31:03.070 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:03.029+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:03.070 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:03.051+0000 7f923da1c640 -1 osd.7 83 handle_osd_map injecting map crc failure 2024-03-19T11:31:03.070 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-03-19T11:31:03.070 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.5 to 1 (10000) 2024-03-19T11:31:03.071 INFO:tasks.rados.rados.0.smithi038.stdout:1188: set_chunk oid 4 offset: 881664 length: 71680 target oid 7583482bc43510e3b098ddda6bd0d25c8822791f0ac578b2257e9627801efed9 offset: 0 snap -1 2024-03-19T11:31:03.072 INFO:tasks.rados.rados.0.smithi038.stdout:1189: read oid 17 snap 19 2024-03-19T11:31:03.072 INFO:tasks.rados.rados.0.smithi038.stdout:1189: expect (ObjNum 181 snap 19 seq_num 181) 2024-03-19T11:31:03.072 INFO:tasks.rados.rados.0.smithi038.stdout:1190: read oid 19 snap -1 2024-03-19T11:31:03.072 INFO:tasks.rados.rados.0.smithi038.stdout:1190: expect deleted 2024-03-19T11:31:03.072 INFO:tasks.rados.rados.0.smithi038.stdout:1191: delete oid 37 current snap is 22 2024-03-19T11:31:03.072 INFO:tasks.rados.rados.0.smithi038.stdout:1181: finishing rollback tid 0 to smithi03829856-13 2024-03-19T11:31:03.072 INFO:tasks.rados.rados.0.smithi038.stdout:1180: finishing write tid 1 to smithi03829856-23 2024-03-19T11:31:03.072 INFO:tasks.rados.rados.0.smithi038.stdout:1180: finishing write tid 2 to smithi03829856-23 2024-03-19T11:31:03.072 INFO:tasks.rados.rados.0.smithi038.stdout:1180: finishing write tid 3 to smithi03829856-23 2024-03-19T11:31:03.072 INFO:tasks.rados.rados.0.smithi038.stdout:1180: finishing write tid 4 to smithi03829856-23 2024-03-19T11:31:03.072 INFO:tasks.rados.rados.0.smithi038.stdout:1180: finishing write tid 5 to smithi03829856-23 2024-03-19T11:31:03.072 INFO:tasks.rados.rados.0.smithi038.stdout:1180: oid 23 updating version 0 to 90 2024-03-19T11:31:03.072 INFO:tasks.rados.rados.0.smithi038.stdout:1180: oid 23 updating version 90 to 92 2024-03-19T11:31:03.072 INFO:tasks.rados.rados.0.smithi038.stdout:1180: oid 23 version 92 is already newer than 89 2024-03-19T11:31:03.073 INFO:tasks.rados.rados.0.smithi038.stdout:1180: oid 23 version 92 is already newer than 91 2024-03-19T11:31:03.073 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 23 v 92 (ObjNum 190 snap 22 seq_num 190) dirty exists 2024-03-19T11:31:03.073 INFO:tasks.rados.rados.0.smithi038.stdout:1180: left oid 23 (ObjNum 190 snap 22 seq_num 190) 2024-03-19T11:31:03.073 INFO:tasks.rados.rados.0.smithi038.stdout:1181: finishing rollback tid 1 to smithi03829856-13 2024-03-19T11:31:03.073 INFO:tasks.rados.rados.0.smithi038.stdout:1181: finishing rollback tid 2 to smithi03829856-13 2024-03-19T11:31:03.073 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 13 v 97 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-03-19T11:31:03.073 INFO:tasks.rados.rados.0.smithi038.stdout:1182: finishing set_chunk to oid 12 2024-03-19T11:31:03.073 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 12 v 83 (ObjNum 161 snap 17 seq_num 161) dirty exists 2024-03-19T11:31:03.073 INFO:tasks.rados.rados.0.smithi038.stdout:1183: expect (ObjNum 106 snap 11 seq_num 106) 2024-03-19T11:31:03.074 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-03-19T11:31:03.120 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:03.129+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:03.222 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:03.229+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:03.233 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:03.233 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:31:03.233 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:31:03.233 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:03.250 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-19T11:31:03.322 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:03.331+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:03.366 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:03.366 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:31:03.366 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:31:03.366 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:31:03.367 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:03.380 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-19T11:31:03.411 INFO:tasks.rados.rados.0.smithi038.stdout:1185: finishing write tid 1 to smithi03829856-29 2024-03-19T11:31:03.411 INFO:tasks.rados.rados.0.smithi038.stdout:1180: done (10 left) 2024-03-19T11:31:03.411 INFO:tasks.rados.rados.0.smithi038.stdout:1181: done (9 left) 2024-03-19T11:31:03.411 INFO:tasks.rados.rados.0.smithi038.stdout:1182: done (8 left) 2024-03-19T11:31:03.411 INFO:tasks.rados.rados.0.smithi038.stdout:1183: done (7 left) 2024-03-19T11:31:03.412 INFO:tasks.rados.rados.0.smithi038.stdout:1191: done (6 left) 2024-03-19T11:31:03.412 INFO:tasks.rados.rados.0.smithi038.stdout:1192: write oid 38 current snap is 22 2024-03-19T11:31:03.412 INFO:tasks.rados.rados.0.smithi038.stdout:1185: finishing write tid 2 to smithi03829856-29 2024-03-19T11:31:03.412 INFO:tasks.rados.rados.0.smithi038.stdout:1192: seq_num 194 ranges {460558=707131,1869389=508716,3094884=165674} 2024-03-19T11:31:03.419 INFO:tasks.rados.rados.0.smithi038.stdout:1192: writing smithi03829856-38 from 460558 to 1167689 tid 1 2024-03-19T11:31:03.423 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:03.429+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:03.427 INFO:tasks.rados.rados.0.smithi038.stdout:1192: writing smithi03829856-38 from 1869389 to 2378105 tid 2 2024-03-19T11:31:03.433 INFO:tasks.rados.rados.0.smithi038.stdout:1192: writing smithi03829856-38 from 3094884 to 3260558 tid 3 2024-03-19T11:31:03.433 INFO:tasks.rados.rados.0.smithi038.stdout:1193: read oid 2 snap -1 2024-03-19T11:31:03.433 INFO:tasks.rados.rados.0.smithi038.stdout:1193: expect (ObjNum 185 snap 20 seq_num 185) 2024-03-19T11:31:03.433 INFO:tasks.rados.rados.0.smithi038.stdout:1194: snap_remove snap 4 2024-03-19T11:31:03.524 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:03.535+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:03.550 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:03.550 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:31:03.551 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:31:03.551 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:31:03.551 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.551 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:31:03.551 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:44.200920+0000", 2024-03-19T11:31:03.552 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 79.357161012000006, 2024-03-19T11:31:03.552 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.80881896499999995, 2024-03-19T11:31:03.552 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:03.552 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:03.552 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:03.552 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:31:03.552 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:31:03.552 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1 2024-03-19T11:31:03.552 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.552 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:03.552 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.552 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:03.552 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200920+0000", 2024-03-19T11:31:03.553 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.553 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.553 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.553 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:03.553 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200920+0000", 2024-03-19T11:31:03.553 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.553 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.553 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.553 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:03.553 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200923+0000", 2024-03-19T11:31:03.553 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6469999999999998e-06 2024-03-19T11:31:03.553 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.553 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.554 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:03.554 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200937+0000", 2024-03-19T11:31:03.554 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.399e-05 2024-03-19T11:31:03.554 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.554 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.554 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:03.554 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200960+0000", 2024-03-19T11:31:03.554 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2603e-05 2024-03-19T11:31:03.554 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.554 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.555 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:03.555 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200978+0000", 2024-03-19T11:31:03.555 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.791e-05 2024-03-19T11:31:03.555 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.555 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.555 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:03.555 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.201081+0000", 2024-03-19T11:31:03.555 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010344900000000001 2024-03-19T11:31:03.555 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.555 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.555 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:31:03.555 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.201096+0000", 2024-03-19T11:31:03.555 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4443e-05 2024-03-19T11:31:03.556 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.556 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.556 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:03.556 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.005978+0000", 2024-03-19T11:31:03.556 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.80488281500000003 2024-03-19T11:31:03.556 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.556 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.556 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:03.556 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.006267+0000", 2024-03-19T11:31:03.556 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000288688 2024-03-19T11:31:03.556 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.556 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.556 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:03.556 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.006534+0000", 2024-03-19T11:31:03.557 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026725799999999998 2024-03-19T11:31:03.557 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.557 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.557 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:03.557 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.008457+0000", 2024-03-19T11:31:03.557 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0019227820000000001 2024-03-19T11:31:03.557 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.557 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.557 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:03.557 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.009570+0000", 2024-03-19T11:31:03.558 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011131750000000001 2024-03-19T11:31:03.558 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.558 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.558 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:03.558 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.009670+0000", 2024-03-19T11:31:03.558 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9458999999999998e-05 2024-03-19T11:31:03.558 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.558 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.558 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:03.558 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.009739+0000", 2024-03-19T11:31:03.558 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9746000000000006e-05 2024-03-19T11:31:03.558 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.558 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:03.559 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.559 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.559 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.559 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:31:03.559 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.071728+0000", 2024-03-19T11:31:03.559 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 78.486353574999995, 2024-03-19T11:31:03.559 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013714690999999999, 2024-03-19T11:31:03.559 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:03.559 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:03.559 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:03.559 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:31:03.559 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:31:03.559 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 34 2024-03-19T11:31:03.559 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.560 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:03.560 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.560 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:03.560 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071728+0000", 2024-03-19T11:31:03.560 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.560 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.560 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.560 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:03.560 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071728+0000", 2024-03-19T11:31:03.560 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.560 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.560 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.561 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:03.561 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071729+0000", 2024-03-19T11:31:03.561 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.47e-07 2024-03-19T11:31:03.561 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.561 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.561 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:03.561 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071735+0000", 2024-03-19T11:31:03.561 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3799999999999999e-06 2024-03-19T11:31:03.561 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.561 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.561 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:03.561 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071748+0000", 2024-03-19T11:31:03.561 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2806000000000001e-05 2024-03-19T11:31:03.561 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.562 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.562 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:03.562 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071756+0000", 2024-03-19T11:31:03.562 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1240000000000005e-06 2024-03-19T11:31:03.562 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.562 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.562 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:03.562 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.077570+0000", 2024-03-19T11:31:03.562 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.005813342 2024-03-19T11:31:03.562 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.562 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.562 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:03.563 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.077793+0000", 2024-03-19T11:31:03.563 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022391500000000001 2024-03-19T11:31:03.563 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.563 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.563 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:03.563 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.078146+0000", 2024-03-19T11:31:03.563 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00035219 2024-03-19T11:31:03.563 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.563 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.563 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:03.563 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.083390+0000", 2024-03-19T11:31:03.564 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052445590000000002 2024-03-19T11:31:03.564 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.564 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.564 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:03.564 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085288+0000", 2024-03-19T11:31:03.564 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0018977060000000001 2024-03-19T11:31:03.564 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.564 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.564 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:03.564 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085377+0000", 2024-03-19T11:31:03.564 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9068999999999997e-05 2024-03-19T11:31:03.564 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.565 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.565 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:03.565 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085443+0000", 2024-03-19T11:31:03.565 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5653000000000003e-05 2024-03-19T11:31:03.565 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.565 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:03.565 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.565 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.565 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.565 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:31:03.565 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.071777+0000", 2024-03-19T11:31:03.565 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 78.486304465000003, 2024-03-19T11:31:03.565 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013962445, 2024-03-19T11:31:03.565 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:03.566 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:03.566 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:03.566 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:31:03.566 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:31:03.566 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 35 2024-03-19T11:31:03.566 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.566 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:03.566 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.566 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:03.566 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071777+0000", 2024-03-19T11:31:03.566 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.566 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.566 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.567 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:03.567 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071777+0000", 2024-03-19T11:31:03.567 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.567 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.567 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.567 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:03.567 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071778+0000", 2024-03-19T11:31:03.567 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5000000000000001e-07 2024-03-19T11:31:03.567 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.567 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.568 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:03.568 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071917+0000", 2024-03-19T11:31:03.568 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013880199999999999 2024-03-19T11:31:03.568 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.568 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.568 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:03.568 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071931+0000", 2024-03-19T11:31:03.568 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4080000000000001e-05 2024-03-19T11:31:03.568 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.568 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.568 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:03.568 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071940+0000", 2024-03-19T11:31:03.568 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.363e-06 2024-03-19T11:31:03.569 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.569 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.569 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:03.569 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.080462+0000", 2024-03-19T11:31:03.569 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085221810000000002 2024-03-19T11:31:03.569 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.569 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.569 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:03.569 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.080582+0000", 2024-03-19T11:31:03.569 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011935299999999999 2024-03-19T11:31:03.569 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.569 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.569 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:03.569 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.080782+0000", 2024-03-19T11:31:03.569 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020077100000000001 2024-03-19T11:31:03.570 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.570 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.570 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:03.570 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.083401+0000", 2024-03-19T11:31:03.570 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002618333 2024-03-19T11:31:03.570 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.570 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.570 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:03.570 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085571+0000", 2024-03-19T11:31:03.570 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021702829999999998 2024-03-19T11:31:03.571 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.571 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.571 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:03.571 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085672+0000", 2024-03-19T11:31:03.571 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100876 2024-03-19T11:31:03.571 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.571 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.571 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:03.571 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085739+0000", 2024-03-19T11:31:03.571 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7552999999999995e-05 2024-03-19T11:31:03.571 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.571 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:03.571 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.571 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.572 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.572 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:31:03.572 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.071957+0000", 2024-03-19T11:31:03.572 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 78.486124277000002, 2024-03-19T11:31:03.572 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013989822000000001, 2024-03-19T11:31:03.572 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:03.572 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:03.572 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:03.572 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:31:03.572 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:31:03.572 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 36 2024-03-19T11:31:03.572 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.572 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:03.572 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.572 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:03.573 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071957+0000", 2024-03-19T11:31:03.573 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.573 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.573 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.573 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:03.573 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071957+0000", 2024-03-19T11:31:03.573 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.573 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.573 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.573 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:03.573 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071958+0000", 2024-03-19T11:31:03.573 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.06e-06 2024-03-19T11:31:03.573 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.573 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.573 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:03.573 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071965+0000", 2024-03-19T11:31:03.574 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8929999999999999e-06 2024-03-19T11:31:03.574 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.574 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.574 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:03.574 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071976+0000", 2024-03-19T11:31:03.574 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1046999999999999e-05 2024-03-19T11:31:03.574 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.574 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.574 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:03.574 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071984+0000", 2024-03-19T11:31:03.574 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7170000000000008e-06 2024-03-19T11:31:03.574 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.574 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.574 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:03.575 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.081761+0000", 2024-03-19T11:31:03.575 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0097769310000000009 2024-03-19T11:31:03.575 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.575 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.575 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:03.575 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.081928+0000", 2024-03-19T11:31:03.575 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000167219 2024-03-19T11:31:03.575 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.575 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.575 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:03.575 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.082131+0000", 2024-03-19T11:31:03.575 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020244499999999999 2024-03-19T11:31:03.575 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.575 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.575 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:03.576 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.083455+0000", 2024-03-19T11:31:03.576 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001324073 2024-03-19T11:31:03.576 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.576 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.576 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:03.576 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085816+0000", 2024-03-19T11:31:03.576 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0023613309999999999 2024-03-19T11:31:03.576 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.576 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.576 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:03.576 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085892+0000", 2024-03-19T11:31:03.576 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6154000000000003e-05 2024-03-19T11:31:03.576 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.576 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.576 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:03.577 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085947+0000", 2024-03-19T11:31:03.577 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4951999999999998e-05 2024-03-19T11:31:03.577 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.577 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:03.577 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.577 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.577 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.577 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:31:03.577 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.072000+0000", 2024-03-19T11:31:03.577 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 78.486081526999996, 2024-03-19T11:31:03.577 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014418268999999999, 2024-03-19T11:31:03.577 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:03.577 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:03.577 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:03.577 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:31:03.578 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:31:03.578 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 37 2024-03-19T11:31:03.578 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.578 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:03.578 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.578 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:03.578 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072000+0000", 2024-03-19T11:31:03.578 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.578 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.578 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.578 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:03.578 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072000+0000", 2024-03-19T11:31:03.578 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.579 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:03.579 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072001+0000", 2024-03-19T11:31:03.579 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0699999999999996e-07 2024-03-19T11:31:03.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.580 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:03.580 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072007+0000", 2024-03-19T11:31:03.580 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2060000000000004e-06 2024-03-19T11:31:03.580 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.580 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.580 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:03.580 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072023+0000", 2024-03-19T11:31:03.580 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6617e-05 2024-03-19T11:31:03.581 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.581 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:03.581 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072031+0000", 2024-03-19T11:31:03.581 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8900000000000007e-06 2024-03-19T11:31:03.581 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.582 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:03.582 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.082682+0000", 2024-03-19T11:31:03.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010650868000000001 2024-03-19T11:31:03.582 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.582 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:03.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.082804+0000", 2024-03-19T11:31:03.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012202899999999999 2024-03-19T11:31:03.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.583 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:03.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.083006+0000", 2024-03-19T11:31:03.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020127799999999999 2024-03-19T11:31:03.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.584 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:03.584 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.084145+0000", 2024-03-19T11:31:03.584 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001139791 2024-03-19T11:31:03.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.585 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:03.585 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.086292+0000", 2024-03-19T11:31:03.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021461969999999999 2024-03-19T11:31:03.585 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.585 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:03.585 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.086364+0000", 2024-03-19T11:31:03.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2439000000000002e-05 2024-03-19T11:31:03.586 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.586 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:03.586 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.086418+0000", 2024-03-19T11:31:03.586 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4246999999999997e-05 2024-03-19T11:31:03.586 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.586 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:03.587 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.587 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:31:03.587 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.339898+0000", 2024-03-19T11:31:03.587 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 78.218183905000004, 2024-03-19T11:31:03.588 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019501793, 2024-03-19T11:31:03.588 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:03.588 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:03.588 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:03.588 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:31:03.588 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:31:03.588 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113 2024-03-19T11:31:03.589 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.589 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:03.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.589 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:03.589 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339898+0000", 2024-03-19T11:31:03.589 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.589 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.590 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:03.590 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339898+0000", 2024-03-19T11:31:03.590 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.590 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.590 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.590 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:03.590 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339899+0000", 2024-03-19T11:31:03.590 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.353e-06 2024-03-19T11:31:03.590 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.590 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.590 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:03.591 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339907+0000", 2024-03-19T11:31:03.591 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3000000000000002e-06 2024-03-19T11:31:03.591 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.591 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:03.591 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339924+0000", 2024-03-19T11:31:03.591 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6906999999999999e-05 2024-03-19T11:31:03.591 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.592 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:03.592 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339933+0000", 2024-03-19T11:31:03.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8969999999999997e-06 2024-03-19T11:31:03.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.592 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.592 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:03.592 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352239+0000", 2024-03-19T11:31:03.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012305814999999999 2024-03-19T11:31:03.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.592 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.592 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:03.592 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352466+0000", 2024-03-19T11:31:03.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022684100000000001 2024-03-19T11:31:03.593 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.593 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.593 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:03.593 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352816+0000", 2024-03-19T11:31:03.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00035030499999999997 2024-03-19T11:31:03.593 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.593 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.593 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:03.593 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.358956+0000", 2024-03-19T11:31:03.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061400079999999998 2024-03-19T11:31:03.593 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.594 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.594 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:03.594 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359118+0000", 2024-03-19T11:31:03.594 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016237599999999999 2024-03-19T11:31:03.594 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.594 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.594 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:03.594 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359287+0000", 2024-03-19T11:31:03.594 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000168488 2024-03-19T11:31:03.594 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.595 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.595 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:03.595 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359399+0000", 2024-03-19T11:31:03.595 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000112503 2024-03-19T11:31:03.595 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.595 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:03.595 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.595 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.595 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.595 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:31:03.595 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.339947+0000", 2024-03-19T11:31:03.595 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 78.218134137999996, 2024-03-19T11:31:03.595 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021923069999999999, 2024-03-19T11:31:03.595 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:03.596 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:03.596 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:03.596 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:31:03.596 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:31:03.596 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 114 2024-03-19T11:31:03.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.596 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:03.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:03.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339947+0000", 2024-03-19T11:31:03.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.597 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:03.597 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339947+0000", 2024-03-19T11:31:03.597 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.597 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.597 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:03.597 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339948+0000", 2024-03-19T11:31:03.597 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6000000000000003e-07 2024-03-19T11:31:03.597 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.598 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:03.598 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339956+0000", 2024-03-19T11:31:03.598 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1559999999999995e-06 2024-03-19T11:31:03.598 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.598 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.598 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:03.598 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339967+0000", 2024-03-19T11:31:03.598 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0669999999999999e-05 2024-03-19T11:31:03.598 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.599 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.599 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:03.599 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339975+0000", 2024-03-19T11:31:03.599 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0099999999999995e-06 2024-03-19T11:31:03.599 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.599 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.599 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:03.599 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.353672+0000", 2024-03-19T11:31:03.599 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013696583999999999 2024-03-19T11:31:03.599 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.599 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.599 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:03.599 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.353808+0000", 2024-03-19T11:31:03.599 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013652299999999999 2024-03-19T11:31:03.600 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.600 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.600 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:03.600 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.354036+0000", 2024-03-19T11:31:03.600 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022744799999999999 2024-03-19T11:31:03.600 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.600 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.600 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:03.600 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.360480+0000", 2024-03-19T11:31:03.600 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0064441630000000001 2024-03-19T11:31:03.600 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.600 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.600 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:03.601 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361722+0000", 2024-03-19T11:31:03.601 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001242754 2024-03-19T11:31:03.601 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.601 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.601 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:03.601 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361807+0000", 2024-03-19T11:31:03.601 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4605999999999996e-05 2024-03-19T11:31:03.601 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.601 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.601 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:03.601 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361870+0000", 2024-03-19T11:31:03.602 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3396000000000001e-05 2024-03-19T11:31:03.602 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.602 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:03.602 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.602 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.602 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.602 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:31:03.602 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.340017+0000", 2024-03-19T11:31:03.602 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 78.218064822000002, 2024-03-19T11:31:03.602 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022169355000000002, 2024-03-19T11:31:03.602 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:03.602 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:03.603 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:03.603 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:31:03.603 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:31:03.603 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 115 2024-03-19T11:31:03.603 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.603 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:03.603 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.603 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:03.604 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340017+0000", 2024-03-19T11:31:03.604 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.604 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.604 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.604 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:03.604 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340017+0000", 2024-03-19T11:31:03.604 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.604 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.604 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.604 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:03.604 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340018+0000", 2024-03-19T11:31:03.604 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.043e-06 2024-03-19T11:31:03.604 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.604 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.604 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:03.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340139+0000", 2024-03-19T11:31:03.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121502 2024-03-19T11:31:03.605 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.605 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:03.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340157+0000", 2024-03-19T11:31:03.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7382999999999999e-05 2024-03-19T11:31:03.605 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.606 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:03.606 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340166+0000", 2024-03-19T11:31:03.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2469999999999997e-06 2024-03-19T11:31:03.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.606 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:03.606 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.357470+0000", 2024-03-19T11:31:03.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017304119999999999 2024-03-19T11:31:03.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.606 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:03.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.357671+0000", 2024-03-19T11:31:03.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000200798 2024-03-19T11:31:03.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:03.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.358002+0000", 2024-03-19T11:31:03.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033090099999999998 2024-03-19T11:31:03.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:03.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361892+0000", 2024-03-19T11:31:03.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00389018 2024-03-19T11:31:03.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:03.608 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361983+0000", 2024-03-19T11:31:03.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0976000000000004e-05 2024-03-19T11:31:03.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.608 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:03.608 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362071+0000", 2024-03-19T11:31:03.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8022999999999994e-05 2024-03-19T11:31:03.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.608 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:03.608 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362186+0000", 2024-03-19T11:31:03.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115182 2024-03-19T11:31:03.608 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.608 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:03.608 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.609 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.609 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:31:03.609 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.340183+0000", 2024-03-19T11:31:03.609 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 78.217897989999997, 2024-03-19T11:31:03.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022279841000000002, 2024-03-19T11:31:03.609 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:03.609 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:03.609 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:03.609 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:31:03.609 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:31:03.609 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 116 2024-03-19T11:31:03.609 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.609 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:03.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.610 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:03.610 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340183+0000", 2024-03-19T11:31:03.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.610 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.610 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:03.610 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340183+0000", 2024-03-19T11:31:03.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.610 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.610 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:03.610 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340184+0000", 2024-03-19T11:31:03.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9000000000000005e-07 2024-03-19T11:31:03.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:03.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340191+0000", 2024-03-19T11:31:03.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0360000000000003e-06 2024-03-19T11:31:03.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:03.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340201+0000", 2024-03-19T11:31:03.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0844e-05 2024-03-19T11:31:03.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:03.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340209+0000", 2024-03-19T11:31:03.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7360000000000001e-06 2024-03-19T11:31:03.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:03.613 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359535+0000", 2024-03-19T11:31:03.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019325546999999998 2024-03-19T11:31:03.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.613 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:03.613 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359672+0000", 2024-03-19T11:31:03.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013706600000000001 2024-03-19T11:31:03.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.613 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:03.613 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359905+0000", 2024-03-19T11:31:03.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023334500000000001 2024-03-19T11:31:03.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.614 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:03.614 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362306+0000", 2024-03-19T11:31:03.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002401041 2024-03-19T11:31:03.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.614 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:03.614 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362329+0000", 2024-03-19T11:31:03.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3283999999999999e-05 2024-03-19T11:31:03.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:03.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362402+0000", 2024-03-19T11:31:03.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2459000000000005e-05 2024-03-19T11:31:03.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:03.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362463+0000", 2024-03-19T11:31:03.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1493000000000002e-05 2024-03-19T11:31:03.615 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.615 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:03.615 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.616 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:31:03.616 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.340222+0000", 2024-03-19T11:31:03.616 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 78.217859407000006, 2024-03-19T11:31:03.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022500218999999998, 2024-03-19T11:31:03.616 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:03.616 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:03.616 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:03.616 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:31:03.616 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:31:03.616 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 117 2024-03-19T11:31:03.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.616 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:03.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.617 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:03.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340222+0000", 2024-03-19T11:31:03.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.617 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:03.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340222+0000", 2024-03-19T11:31:03.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.617 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:03.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340223+0000", 2024-03-19T11:31:03.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7999999999999995e-07 2024-03-19T11:31:03.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.618 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:03.618 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340229+0000", 2024-03-19T11:31:03.618 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0399999999999998e-06 2024-03-19T11:31:03.618 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.618 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:03.618 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340239+0000", 2024-03-19T11:31:03.618 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0190000000000001e-05 2024-03-19T11:31:03.618 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.618 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:03.618 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340250+0000", 2024-03-19T11:31:03.618 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0720000000000001e-05 2024-03-19T11:31:03.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:03.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.360587+0000", 2024-03-19T11:31:03.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020337542 2024-03-19T11:31:03.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:03.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.360818+0000", 2024-03-19T11:31:03.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023120500000000001 2024-03-19T11:31:03.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:03.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361218+0000", 2024-03-19T11:31:03.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000399637 2024-03-19T11:31:03.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:03.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362483+0000", 2024-03-19T11:31:03.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012649200000000001 2024-03-19T11:31:03.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:03.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362538+0000", 2024-03-19T11:31:03.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4490000000000002e-05 2024-03-19T11:31:03.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:03.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362617+0000", 2024-03-19T11:31:03.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9839000000000006e-05 2024-03-19T11:31:03.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:03.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362722+0000", 2024-03-19T11:31:03.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104956 2024-03-19T11:31:03.621 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.621 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:03.621 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.621 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1047 4.8 e35/21 4:1d58db11:::smithi03829856-18:head v 35'32, mlcod=33'29)", 2024-03-19T11:31:03.621 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:13.225023+0000", 2024-03-19T11:31:03.622 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 50.333058622000003, 2024-03-19T11:31:03.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013059688999999999, 2024-03-19T11:31:03.622 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:03.622 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:03.622 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:03.622 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.622 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:03.622 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.225023+0000", 2024-03-19T11:31:03.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.622 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:03.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.225023+0000", 2024-03-19T11:31:03.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:03.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.225025+0000", 2024-03-19T11:31:03.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.17e-06 2024-03-19T11:31:03.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:03.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.225771+0000", 2024-03-19T11:31:03.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000746267 2024-03-19T11:31:03.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:03.624 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.225807+0000", 2024-03-19T11:31:03.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5487000000000001e-05 2024-03-19T11:31:03.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:03.624 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.225823+0000", 2024-03-19T11:31:03.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6215999999999999e-05 2024-03-19T11:31:03.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:03.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.225990+0000", 2024-03-19T11:31:03.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016680899999999999 2024-03-19T11:31:03.625 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:03.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.226058+0000", 2024-03-19T11:31:03.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8672999999999998e-05 2024-03-19T11:31:03.625 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.626 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:03.626 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.237940+0000", 2024-03-19T11:31:03.626 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011881311 2024-03-19T11:31:03.626 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.626 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:03.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.238083+0000", 2024-03-19T11:31:03.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014275600000000001 2024-03-19T11:31:03.627 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.627 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:03.627 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.628 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1067 4.0 e36/21 4:01018511:::smithi03829856-19:head v 36'18, mlcod=35'14)", 2024-03-19T11:31:03.628 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:13.952492+0000", 2024-03-19T11:31:03.628 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 49.605589164999998, 2024-03-19T11:31:03.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013554725, 2024-03-19T11:31:03.628 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:03.628 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:03.628 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:03.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:03.629 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.952492+0000", 2024-03-19T11:31:03.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.629 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.629 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.629 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:03.629 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.952492+0000", 2024-03-19T11:31:03.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.630 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.630 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:03.630 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.952493+0000", 2024-03-19T11:31:03.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9299999999999997e-07 2024-03-19T11:31:03.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.631 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:03.631 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.954109+0000", 2024-03-19T11:31:03.631 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0016163569999999999 2024-03-19T11:31:03.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:03.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.954135+0000", 2024-03-19T11:31:03.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5352999999999999e-05 2024-03-19T11:31:03.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:03.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.954149+0000", 2024-03-19T11:31:03.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3879999999999999e-05 2024-03-19T11:31:03.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:03.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.954288+0000", 2024-03-19T11:31:03.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000139476 2024-03-19T11:31:03.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:03.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.954356+0000", 2024-03-19T11:31:03.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7533000000000005e-05 2024-03-19T11:31:03.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:03.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.965391+0000", 2024-03-19T11:31:03.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011035764999999999 2024-03-19T11:31:03.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:03.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.966047+0000", 2024-03-19T11:31:03.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00065556800000000004 2024-03-19T11:31:03.633 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.634 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:03.634 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.634 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.634 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.634 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1122 4.8 e36/21 4:1d58db11:::smithi03829856-18:head v 36'42, mlcod=36'39)", 2024-03-19T11:31:03.634 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:14.172666+0000", 2024-03-19T11:31:03.634 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 49.385415391000002, 2024-03-19T11:31:03.634 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018015000999999999, 2024-03-19T11:31:03.634 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:03.634 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:03.634 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:03.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.635 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:03.635 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.172666+0000", 2024-03-19T11:31:03.635 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.635 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.635 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:03.635 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.172666+0000", 2024-03-19T11:31:03.635 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.635 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.635 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:03.635 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.172669+0000", 2024-03-19T11:31:03.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1470000000000001e-06 2024-03-19T11:31:03.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:03.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.173140+0000", 2024-03-19T11:31:03.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00047074200000000001 2024-03-19T11:31:03.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:03.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.173188+0000", 2024-03-19T11:31:03.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8287e-05 2024-03-19T11:31:03.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:03.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.173211+0000", 2024-03-19T11:31:03.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3023e-05 2024-03-19T11:31:03.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.637 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:03.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.173378+0000", 2024-03-19T11:31:03.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000166742 2024-03-19T11:31:03.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.638 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:03.638 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.173557+0000", 2024-03-19T11:31:03.638 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000178569 2024-03-19T11:31:03.638 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.638 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.638 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:03.638 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.190416+0000", 2024-03-19T11:31:03.638 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016859570000000001 2024-03-19T11:31:03.638 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.638 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.638 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:03.639 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.190681+0000", 2024-03-19T11:31:03.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026492099999999999 2024-03-19T11:31:03.639 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.639 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:03.639 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.639 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.5.0:43 4.8 e39/21 4:15708482:::smithi03829856-23:head v 39'50, mlcod=38'47)", 2024-03-19T11:31:03.639 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.842819+0000", 2024-03-19T11:31:03.639 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 46.715262887999998, 2024-03-19T11:31:03.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017626513, 2024-03-19T11:31:03.639 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:03.639 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:03.639 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:03.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:03.640 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842819+0000", 2024-03-19T11:31:03.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:03.640 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842819+0000", 2024-03-19T11:31:03.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:03.640 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842820+0000", 2024-03-19T11:31:03.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.39e-06 2024-03-19T11:31:03.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.641 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:03.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.846437+0000", 2024-03-19T11:31:03.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003616953 2024-03-19T11:31:03.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.641 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:03.642 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.846471+0000", 2024-03-19T11:31:03.642 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4133e-05 2024-03-19T11:31:03.642 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.642 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.642 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:03.642 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.846539+0000", 2024-03-19T11:31:03.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8072999999999997e-05 2024-03-19T11:31:03.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:03.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.847094+0000", 2024-03-19T11:31:03.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00055472099999999999 2024-03-19T11:31:03.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:03.646 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.847128+0000", 2024-03-19T11:31:03.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3875999999999999e-05 2024-03-19T11:31:03.646 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:03.647 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.860308+0000", 2024-03-19T11:31:03.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013180712000000001 2024-03-19T11:31:03.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.647 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:03.647 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.860445+0000", 2024-03-19T11:31:03.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013665500000000001 2024-03-19T11:31:03.647 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.647 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:03.648 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.648 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1209 4.8 e41/21 4:1d58db11:::smithi03829856-18:head v 41'53, mlcod=39'50)", 2024-03-19T11:31:03.648 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:18.124878+0000", 2024-03-19T11:31:03.648 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 45.433203194999997, 2024-03-19T11:31:03.648 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016809343000000001, 2024-03-19T11:31:03.648 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:03.648 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:03.648 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:03.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.648 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:03.648 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.124878+0000", 2024-03-19T11:31:03.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.649 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:03.649 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.124878+0000", 2024-03-19T11:31:03.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.649 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:03.649 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.124880+0000", 2024-03-19T11:31:03.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4959999999999999e-06 2024-03-19T11:31:03.649 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:03.649 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125845+0000", 2024-03-19T11:31:03.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00096535299999999996 2024-03-19T11:31:03.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.650 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.650 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:03.650 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125884+0000", 2024-03-19T11:31:03.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8645e-05 2024-03-19T11:31:03.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.650 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.650 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:03.650 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125909+0000", 2024-03-19T11:31:03.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4783e-05 2024-03-19T11:31:03.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.651 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:03.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.126085+0000", 2024-03-19T11:31:03.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000176649 2024-03-19T11:31:03.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.651 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:03.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.126150+0000", 2024-03-19T11:31:03.652 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4290999999999998e-05 2024-03-19T11:31:03.652 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.652 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.652 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:03.652 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.141552+0000", 2024-03-19T11:31:03.652 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015402295 2024-03-19T11:31:03.652 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.652 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.652 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:03.652 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.141688+0000", 2024-03-19T11:31:03.652 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013583099999999999 2024-03-19T11:31:03.652 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.652 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:03.652 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.652 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.653 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.653 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.15 51/50 [PushOp(4:afa9f208:::smithi03829856-35:head, version: 28'46, data_included: [571438~623878,1806623~424698], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:afa9f208:::smithi03829856-35:head@28'46, size: 2571438, copy_subset: [(0, 2571438)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2231321, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:af9d0d56:::smithi03829856-20:head, version: 35'55, data_included: [655360~589824,1900544~418789], data_size: 1008613, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:af9d0d56:::smithi03829856-20:head@35'55, size: 2319333, copy_subset: [(0, 2319333)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2319333, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ad3ad1b3:::smithi03829856-21:b, version: 47'66, data_included: [736140~537686,2020581~315559], data_size: 853245, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:ad3ad1b3:::smithi03829856-21:b@47'66, size: 2336140, copy_subset: [(0, 2336140)], clone_subset: {}, snapset: b=[]:{b: [b, a, 6, 5]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2336140, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ad3ad1b3:::smithi03829856-21:head, version: 47'70, data_included: [531121~742705,1711057~305871], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:ad3ad1b3:::smithi03829856-21:head@47'70, size: 3731121, copy_subset: [(0, 3731121)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2016928, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ac9cd3a3:::smithi03829856-8:head, version: 49'72, data_included: [517649~620182,1776668~428394], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:ac9cd3a3:::smithi03829856-8:head@49'72, size: 3008420, copy_subset: [(0, 3008420)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2205062, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:aa3303a5:::smithi03829856-17:b, version: 49'73, data_included: [614400~692224,1903006~356352], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:aa3303a5:::smithi03829856-17:b@49'73, size: 2268912, copy_subset: [(0, 2268912)], clone_subset: {}, snapset: b=[]:{b: [b, a, 6, 5]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2259358, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:aa3303a5:::smithi03829856-17:head, version: 49'74, data_included: [413472~893152,1818036~155424], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:aa3303a5:::smithi03829856-17:head@49'74, size: 3613472, copy_subset: [(0, 3613472)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1973460, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:03.653 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:28.058610+0000", 2024-03-19T11:31:03.653 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 35.499471235000001, 2024-03-19T11:31:03.653 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027100032, 2024-03-19T11:31:03.653 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:03.653 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:03.653 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:03.653 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.653 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:03.653 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.058610+0000", 2024-03-19T11:31:03.653 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.653 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.653 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.654 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:03.654 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.058610+0000", 2024-03-19T11:31:03.654 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.654 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:03.654 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.058611+0000", 2024-03-19T11:31:03.654 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1599999999999999e-06 2024-03-19T11:31:03.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.654 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:03.654 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.063874+0000", 2024-03-19T11:31:03.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052625270000000004 2024-03-19T11:31:03.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:03.655 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.063935+0000", 2024-03-19T11:31:03.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0594000000000003e-05 2024-03-19T11:31:03.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:03.655 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064024+0000", 2024-03-19T11:31:03.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9273999999999999e-05 2024-03-19T11:31:03.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.656 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:03.656 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064385+0000", 2024-03-19T11:31:03.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00036129100000000002 2024-03-19T11:31:03.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.656 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:03.656 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064437+0000", 2024-03-19T11:31:03.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2346000000000003e-05 2024-03-19T11:31:03.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.656 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:03.657 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085710+0000", 2024-03-19T11:31:03.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021272840000000001 2024-03-19T11:31:03.657 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.657 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:03.657 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.657 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.657 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.657 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.15 e51,50 4:ac9cd3a3:::smithi03829856-8:4,49'71)", 2024-03-19T11:31:03.657 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:28.064112+0000", 2024-03-19T11:31:03.657 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 35.493969409000002, 2024-03-19T11:31:03.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026480417999999999, 2024-03-19T11:31:03.657 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:03.657 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:03.657 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:03.657 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.658 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:03.658 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064112+0000", 2024-03-19T11:31:03.658 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.658 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.658 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:03.658 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064112+0000", 2024-03-19T11:31:03.658 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.658 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:03.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064113+0000", 2024-03-19T11:31:03.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3600000000000003e-07 2024-03-19T11:31:03.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:03.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064118+0000", 2024-03-19T11:31:03.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0549999999999999e-06 2024-03-19T11:31:03.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:03.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064128+0000", 2024-03-19T11:31:03.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0349e-05 2024-03-19T11:31:03.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.660 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:03.660 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064136+0000", 2024-03-19T11:31:03.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6249999999999998e-06 2024-03-19T11:31:03.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.660 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:03.660 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085773+0000", 2024-03-19T11:31:03.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021636829999999999 2024-03-19T11:31:03.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.660 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:03.660 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085788+0000", 2024-03-19T11:31:03.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5512999999999999e-05 2024-03-19T11:31:03.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.661 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:03.661 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.090592+0000", 2024-03-19T11:31:03.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0048044100000000003 2024-03-19T11:31:03.661 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.661 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:03.661 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.661 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.661 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.5 51/50 [PushOp(4:a38b9453:::smithi03829856-37:head, version: 28'45, data_included: [605052~774376,1923587~274200], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:a38b9453:::smithi03829856-37:head@28'45, size: 2605052, copy_subset: [(0, 2605052)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2197787, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:a0a50d04:::smithi03829856-13:b, version: 47'63, data_included: [601684~619341,1669078~429235], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a0a50d04:::smithi03829856-13:b@47'63, size: 2201684, copy_subset: [(0, 2201684)], clone_subset: {}, snapset: b=[]:{b: [b, a, 6, 5]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2098313, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:03.661 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:28.064324+0000", 2024-03-19T11:31:03.661 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 35.493757565000003, 2024-03-19T11:31:03.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027179020000000002, 2024-03-19T11:31:03.661 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:03.661 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:03.662 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:03.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.662 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:03.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064324+0000", 2024-03-19T11:31:03.662 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.662 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:03.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064324+0000", 2024-03-19T11:31:03.662 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.662 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:03.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064324+0000", 2024-03-19T11:31:03.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1199999999999998e-07 2024-03-19T11:31:03.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.663 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:03.663 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.065527+0000", 2024-03-19T11:31:03.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012028939999999999 2024-03-19T11:31:03.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.663 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:03.663 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.065552+0000", 2024-03-19T11:31:03.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4440999999999999e-05 2024-03-19T11:31:03.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.663 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:03.663 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.065568+0000", 2024-03-19T11:31:03.664 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6498e-05 2024-03-19T11:31:03.664 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.664 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:03.664 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085812+0000", 2024-03-19T11:31:03.664 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020243954000000002 2024-03-19T11:31:03.664 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.664 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:03.664 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085825+0000", 2024-03-19T11:31:03.664 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3298999999999999e-05 2024-03-19T11:31:03.664 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.665 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:03.665 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.091503+0000", 2024-03-19T11:31:03.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.005677522 2024-03-19T11:31:03.665 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.665 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:03.665 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.665 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.24 75/74 [PushOp(4:27909373:::smithi03829856-46:8, version: 46'59, data_included: [756315~767970,1933572~280606], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:8@46'59, size: 2756315, copy_subset: [(0, 2756315)], clone_subset: {}, snapset: 10=[]:{8: [6], c: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2214178, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:266590a5:::smithi03829856-16:e, version: 70'107, data_included: [652105~662359,1728171~386217], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:e@70'107, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 10=[]:{e: [d, b, 6]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:266590a5:::smithi03829856-16:head, version: 70'108, data_included: [652105~662359,1728171~386217], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:head@70'108, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:27909373:::smithi03829856-46:c, version: 70'109, data_included: [628595~895690,1807440~152886], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:c@70'109, size: 3428595, copy_subset: [(0, 3428595)], clone_subset: {}, snapset: 10=[]:{8: [6], c: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1960326, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:27909373:::smithi03829856-46:head, version: 70'110, data_included: [403401~454961,1584925~506039], data_size: 961000, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:head@70'110, size: 2403401, copy_subset: [(0, 2403401)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2403401, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:03.665 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.099172+0000", 2024-03-19T11:31:03.665 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.4589091310000004, 2024-03-19T11:31:03.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.024528580000000001, 2024-03-19T11:31:03.665 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:03.665 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:03.666 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:03.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.666 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:03.666 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.099172+0000", 2024-03-19T11:31:03.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:03.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.099172+0000", 2024-03-19T11:31:03.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:03.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.099175+0000", 2024-03-19T11:31:03.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7750000000000001e-06 2024-03-19T11:31:03.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:03.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105219+0000", 2024-03-19T11:31:03.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0060440789999999999 2024-03-19T11:31:03.668 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.668 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.668 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:03.668 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105285+0000", 2024-03-19T11:31:03.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6260000000000006e-05 2024-03-19T11:31:03.668 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.668 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.668 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:03.668 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105366+0000", 2024-03-19T11:31:03.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.038e-05 2024-03-19T11:31:03.668 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.669 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.669 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:03.669 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105637+0000", 2024-03-19T11:31:03.669 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027085199999999997 2024-03-19T11:31:03.669 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.669 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.669 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:03.669 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105679+0000", 2024-03-19T11:31:03.669 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2088000000000001e-05 2024-03-19T11:31:03.669 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.669 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.669 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:03.669 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.123701+0000", 2024-03-19T11:31:03.669 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018022145999999999 2024-03-19T11:31:03.670 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.670 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:03.670 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.670 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.670 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.670 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.24 e75,74 4:266590a5:::smithi03829856-16:4,47'63,4:27909373:::smithi03829856-46:5,55'79,4:266590a5:::smithi03829856-16:10,67'105)", 2024-03-19T11:31:03.670 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.105437+0000", 2024-03-19T11:31:03.670 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.4526441279999993, 2024-03-19T11:31:03.670 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.01860726, 2024-03-19T11:31:03.670 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:03.670 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:03.670 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:03.670 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.670 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:03.670 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105437+0000", 2024-03-19T11:31:03.671 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.671 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.671 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:03.671 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105437+0000", 2024-03-19T11:31:03.671 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:03.671 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.671 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:03.671 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105438+0000", 2024-03-19T11:31:03.671 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6799999999999999e-07 2024-03-19T11:31:03.671 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:03.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105445+0000", 2024-03-19T11:31:03.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3749999999999999e-06 2024-03-19T11:31:03.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:03.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105454+0000", 2024-03-19T11:31:03.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7909999999999999e-06 2024-03-19T11:31:03.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:03.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105465+0000", 2024-03-19T11:31:03.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.045e-05 2024-03-19T11:31:03.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.673 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:03.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.123753+0000", 2024-03-19T11:31:03.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0182884 2024-03-19T11:31:03.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.673 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:03.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.123765+0000", 2024-03-19T11:31:03.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2143e-05 2024-03-19T11:31:03.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:03.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:03.673 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:03.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.124045+0000", 2024-03-19T11:31:03.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027923299999999999 2024-03-19T11:31:03.673 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.673 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:03.674 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.674 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:03.674 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:03.674 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:03.675 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-19T11:31:03.676 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:03.681+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:03.777 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:03.787+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:03.804 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:03.804 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:31:03.804 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:31:03.804 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:03.817 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-19T11:31:03.877 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:03.885+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:03.928 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:03.928 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:31:03.929 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:31:03.929 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:31:03.929 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:03.941 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-19T11:31:03.978 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:03.987+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:04.035 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:04.041+0000 7f6712f92640 -1 osd.2 84 handle_osd_map injecting map crc failure 2024-03-19T11:31:04.039 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.6 to 1 (10000) 2024-03-19T11:31:04.040 INFO:tasks.rados.rados.0.smithi038.stdout:1194: done (8 left) 2024-03-19T11:31:04.040 INFO:tasks.rados.rados.0.smithi038.stdout:1195: read oid 32 snap -1 2024-03-19T11:31:04.040 INFO:tasks.rados.rados.0.smithi038.stdout:1195: expect (ObjNum 31 snap 0 seq_num 31) 2024-03-19T11:31:04.040 INFO:tasks.rados.rados.0.smithi038.stdout:1196: set_chunk oid 6 target oid 2024-03-19T11:31:04.072 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-03-19T11:31:04.079 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:04.091+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:04.089 INFO:tasks.rados.rados.0.smithi038.stdout:1196: set_chunk oid 6 offset: 1022976 length: 1661440 target oid f25cf4cdffbae9b54315ae93e2cdc514e3bf58d758c8445671e9463006f381a6 offset: 0 snap -1 2024-03-19T11:31:04.089 INFO:tasks.rados.rados.0.smithi038.stdout:1197: read oid 35 snap -1 2024-03-19T11:31:04.089 INFO:tasks.rados.rados.0.smithi038.stdout:1197: expect (ObjNum 158 snap 17 seq_num 158) 2024-03-19T11:31:04.089 INFO:tasks.rados.rados.0.smithi038.stdout:1198: tier_evict oid 14 2024-03-19T11:31:04.089 INFO:tasks.rados.rados.0.smithi038.stdout:1198: tier_evict oid 14 snap -1 2024-03-19T11:31:04.095 INFO:tasks.rados.rados.0.smithi038.stdout:1185: finishing write tid 3 to smithi03829856-29 2024-03-19T11:31:04.096 INFO:tasks.rados.rados.0.smithi038.stdout:1185: finishing write tid 4 to smithi03829856-29 2024-03-19T11:31:04.096 INFO:tasks.rados.rados.0.smithi038.stdout:1185: finishing write tid 5 to smithi03829856-29 2024-03-19T11:31:04.096 INFO:tasks.rados.rados.0.smithi038.stdout:1185: oid 29 updating version 0 to 79 2024-03-19T11:31:04.096 INFO:tasks.rados.rados.0.smithi038.stdout:1185: oid 29 updating version 79 to 81 2024-03-19T11:31:04.096 INFO:tasks.rados.rados.0.smithi038.stdout:1185: oid 29 version 81 is already newer than 78 2024-03-19T11:31:04.096 INFO:tasks.rados.rados.0.smithi038.stdout:1185: oid 29 version 81 is already newer than 80 2024-03-19T11:31:04.096 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 29 v 81 (ObjNum 191 snap 22 seq_num 191) dirty exists 2024-03-19T11:31:04.096 INFO:tasks.rados.rados.0.smithi038.stdout:1185: left oid 29 (ObjNum 191 snap 22 seq_num 191) 2024-03-19T11:31:04.096 INFO:tasks.rados.rados.0.smithi038.stdout:1187: finishing set_chunk to oid 15 2024-03-19T11:31:04.096 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 15 set_chunk returned error code -95 offset: 492032 length: 616960 2024-03-19T11:31:04.096 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-03-19T11:31:04.096 INFO:tasks.rados.rados.0.smithi038.stdout:1188: finishing set_chunk to oid 4 2024-03-19T11:31:04.096 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 4 v 104 (ObjNum 151 snap 17 seq_num 151) dirty exists 2024-03-19T11:31:04.096 INFO:tasks.rados.rados.0.smithi038.stdout:1189: expect (ObjNum 181 snap 19 seq_num 181) 2024-03-19T11:31:04.137 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:04.137 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:31:04.137 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:31:04.137 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:31:04.137 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.138 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1090 4.6 4:6d9093b1:::smithi03829856-38:head [sparse-read 0~1502675] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:31:04.138 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:14.006669+0000", 2024-03-19T11:31:04.138 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 50.140203088, 2024-03-19T11:31:04.138 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012590499999999999, 2024-03-19T11:31:04.138 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:04.138 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:04.138 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:04.138 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:04.138 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:04.138 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1090 2024-03-19T11:31:04.138 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.138 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:04.138 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.138 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:04.139 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006669+0000", 2024-03-19T11:31:04.139 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.139 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.139 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.139 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:04.139 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006669+0000", 2024-03-19T11:31:04.139 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.139 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.140 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.140 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:04.140 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006669+0000", 2024-03-19T11:31:04.140 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1800000000000001e-07 2024-03-19T11:31:04.140 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.140 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.141 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:04.141 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006673+0000", 2024-03-19T11:31:04.141 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.9530000000000001e-06 2024-03-19T11:31:04.141 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.141 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.141 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:04.142 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006680+0000", 2024-03-19T11:31:04.142 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9410000000000001e-06 2024-03-19T11:31:04.142 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.142 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.143 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:04.143 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.006685+0000", 2024-03-19T11:31:04.143 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.327e-06 2024-03-19T11:31:04.143 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.143 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.143 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.143 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.018877+0000", 2024-03-19T11:31:04.143 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012191936 2024-03-19T11:31:04.143 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.143 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.143 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:04.143 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.018972+0000", 2024-03-19T11:31:04.144 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4754999999999997e-05 2024-03-19T11:31:04.144 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.144 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.144 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:04.144 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:14.019259+0000", 2024-03-19T11:31:04.144 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00028717 2024-03-19T11:31:04.144 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.144 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:04.144 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.144 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.144 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.144 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1217 4.3 4:c8b09e87:::smithi03829856-15:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:31:04.144 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.155966+0000", 2024-03-19T11:31:04.144 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 45.990905937000001, 2024-03-19T11:31:04.145 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.023984647000000001, 2024-03-19T11:31:04.145 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:04.145 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:04.145 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:04.145 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:04.145 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:04.145 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1217 2024-03-19T11:31:04.145 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.145 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:04.145 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.145 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:04.145 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155966+0000", 2024-03-19T11:31:04.145 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.145 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.145 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.146 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:04.146 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155966+0000", 2024-03-19T11:31:04.146 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.146 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.146 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.146 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:04.146 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155968+0000", 2024-03-19T11:31:04.146 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.795e-06 2024-03-19T11:31:04.146 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.146 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.146 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:04.146 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155978+0000", 2024-03-19T11:31:04.147 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0574000000000001e-05 2024-03-19T11:31:04.147 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.147 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.147 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:04.147 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.156000+0000", 2024-03-19T11:31:04.147 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1250000000000002e-05 2024-03-19T11:31:04.147 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.147 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.147 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:04.147 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.156010+0000", 2024-03-19T11:31:04.147 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0168e-05 2024-03-19T11:31:04.147 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.148 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.148 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.148 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.156119+0000", 2024-03-19T11:31:04.148 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010950699999999999 2024-03-19T11:31:04.148 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.148 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.148 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:04.148 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.156231+0000", 2024-03-19T11:31:04.148 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000111445 2024-03-19T11:31:04.148 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.148 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.148 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:04.149 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.179951+0000", 2024-03-19T11:31:04.149 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.023719908000000001 2024-03-19T11:31:04.149 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.149 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:04.149 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.149 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.149 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.149 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1432 4.13 4:cef44a0d:::smithi03829856-7:head [assert-version v18,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:31:04.149 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.977382+0000", 2024-03-19T11:31:04.149 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 38.169489665, 2024-03-19T11:31:04.149 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012855814, 2024-03-19T11:31:04.149 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:04.149 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:04.149 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:04.150 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:04.150 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:04.150 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1432 2024-03-19T11:31:04.150 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.150 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:04.150 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.150 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:04.150 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.977382+0000", 2024-03-19T11:31:04.150 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.150 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.150 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.151 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:04.151 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.977382+0000", 2024-03-19T11:31:04.151 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.151 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.151 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.151 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:04.152 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.977385+0000", 2024-03-19T11:31:04.152 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9380000000000001e-06 2024-03-19T11:31:04.152 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.152 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.152 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:04.152 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.977397+0000", 2024-03-19T11:31:04.153 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2136e-05 2024-03-19T11:31:04.153 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.153 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.153 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:04.153 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.977424+0000", 2024-03-19T11:31:04.153 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6917999999999999e-05 2024-03-19T11:31:04.154 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.154 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.154 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:04.154 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.977437+0000", 2024-03-19T11:31:04.154 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.242e-05 2024-03-19T11:31:04.155 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.155 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.155 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.155 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.977534+0000", 2024-03-19T11:31:04.155 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6974999999999999e-05 2024-03-19T11:31:04.155 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.156 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.156 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:04.156 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.977639+0000", 2024-03-19T11:31:04.156 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000105713 2024-03-19T11:31:04.156 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.156 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.157 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:31:04.157 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.987924+0000", 2024-03-19T11:31:04.157 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010284847999999999 2024-03-19T11:31:04.157 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.157 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.157 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:04.158 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.988536+0000", 2024-03-19T11:31:04.158 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00061207099999999995 2024-03-19T11:31:04.158 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.158 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.158 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:04.158 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.990098+0000", 2024-03-19T11:31:04.159 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001561523 2024-03-19T11:31:04.159 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.159 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.159 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:04.159 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.990176+0000", 2024-03-19T11:31:04.159 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8316e-05 2024-03-19T11:31:04.160 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.160 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.160 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:04.160 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.990238+0000", 2024-03-19T11:31:04.160 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1956000000000001e-05 2024-03-19T11:31:04.161 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.161 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:04.161 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.161 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.161 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.161 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1788 4.23 4:c4406a0a:::smithi03829856-49:head [write 685752~611865,stat] snapc 10=[10,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-03-19T11:31:04.162 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:40.165160+0000", 2024-03-19T11:31:04.162 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 23.981712008999999, 2024-03-19T11:31:04.162 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014138061, 2024-03-19T11:31:04.162 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:04.162 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:04.162 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:04.163 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:04.163 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:04.163 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1788 2024-03-19T11:31:04.163 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.163 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:04.163 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.164 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:04.164 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.165160+0000", 2024-03-19T11:31:04.164 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.164 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.164 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.165 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:04.165 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.165160+0000", 2024-03-19T11:31:04.165 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.165 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.165 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.165 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:04.166 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.165163+0000", 2024-03-19T11:31:04.166 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.729e-06 2024-03-19T11:31:04.166 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.166 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.166 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:04.166 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.165455+0000", 2024-03-19T11:31:04.167 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00029193900000000001 2024-03-19T11:31:04.167 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.167 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.167 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:04.167 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.165474+0000", 2024-03-19T11:31:04.168 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9910000000000001e-05 2024-03-19T11:31:04.168 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.168 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.168 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:04.168 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.165483+0000", 2024-03-19T11:31:04.168 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9609999999999994e-06 2024-03-19T11:31:04.169 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.169 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.169 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.169 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.165638+0000", 2024-03-19T11:31:04.169 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015434300000000001 2024-03-19T11:31:04.169 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.170 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.170 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:04.170 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.166078+0000", 2024-03-19T11:31:04.170 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00044032099999999997 2024-03-19T11:31:04.170 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.171 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.171 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:31:04.171 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.166669+0000", 2024-03-19T11:31:04.171 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00059135100000000005 2024-03-19T11:31:04.171 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.171 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.172 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:04.172 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.174542+0000", 2024-03-19T11:31:04.172 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0078724880000000004 2024-03-19T11:31:04.172 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.172 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.172 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:04.173 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.178949+0000", 2024-03-19T11:31:04.173 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0044075720000000002 2024-03-19T11:31:04.173 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.173 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.173 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:04.174 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.179110+0000", 2024-03-19T11:31:04.174 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016055900000000001 2024-03-19T11:31:04.174 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.174 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.174 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:04.174 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.179298+0000", 2024-03-19T11:31:04.175 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00018788799999999999 2024-03-19T11:31:04.175 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.175 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:04.175 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.175 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.175 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.176 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1834 4.1b 4:db3790a7:::smithi03829856-32:head [tier-promote] snapc 10=[10,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-03-19T11:31:04.176 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:40.957892+0000", 2024-03-19T11:31:04.176 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 23.188980133000001, 2024-03-19T11:31:04.176 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013588032, 2024-03-19T11:31:04.176 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:04.177 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:04.177 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:04.177 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:04.177 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:04.177 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1834 2024-03-19T11:31:04.177 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.178 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:04.178 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.178 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:04.178 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.957892+0000", 2024-03-19T11:31:04.178 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.178 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.179 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.179 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:04.179 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.957892+0000", 2024-03-19T11:31:04.179 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.179 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.179 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.180 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:04.180 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.957894+0000", 2024-03-19T11:31:04.180 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2699999999999999e-06 2024-03-19T11:31:04.180 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.180 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.180 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:04.180 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.957903+0000", 2024-03-19T11:31:04.180 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5520000000000005e-06 2024-03-19T11:31:04.180 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.180 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.180 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:04.180 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.957924+0000", 2024-03-19T11:31:04.180 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1531000000000001e-05 2024-03-19T11:31:04.180 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.181 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.181 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:04.181 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.957934+0000", 2024-03-19T11:31:04.181 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.009e-05 2024-03-19T11:31:04.181 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.181 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.181 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.181 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.957987+0000", 2024-03-19T11:31:04.181 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2772000000000002e-05 2024-03-19T11:31:04.181 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.181 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.181 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:04.181 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.958044+0000", 2024-03-19T11:31:04.181 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6715000000000002e-05 2024-03-19T11:31:04.181 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.182 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.182 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:04.182 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:40.971480+0000", 2024-03-19T11:31:04.182 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013436102 2024-03-19T11:31:04.182 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.182 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:04.182 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.182 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.182 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.182 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2047 4.1b 4:dccaff13:::smithi03829856-36:head [write 756348~784249] snapc 12=[12,11,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-03-19T11:31:04.182 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:46.278700+0000", 2024-03-19T11:31:04.182 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 17.868172329, 2024-03-19T11:31:04.182 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014428785, 2024-03-19T11:31:04.183 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:04.183 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:04.183 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:04.183 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:04.183 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:04.183 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2047 2024-03-19T11:31:04.183 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.183 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:04.183 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.183 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:04.183 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.278700+0000", 2024-03-19T11:31:04.183 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.183 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.184 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.184 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:04.184 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.278700+0000", 2024-03-19T11:31:04.184 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.184 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.184 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.184 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:04.184 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.278703+0000", 2024-03-19T11:31:04.184 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1690000000000001e-06 2024-03-19T11:31:04.184 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.184 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.184 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:04.184 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.278955+0000", 2024-03-19T11:31:04.185 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025264100000000001 2024-03-19T11:31:04.185 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.185 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.185 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:04.185 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.278982+0000", 2024-03-19T11:31:04.185 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6933000000000001e-05 2024-03-19T11:31:04.186 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.186 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.186 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:04.186 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.278997+0000", 2024-03-19T11:31:04.186 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4440999999999999e-05 2024-03-19T11:31:04.186 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.186 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.186 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.186 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.279383+0000", 2024-03-19T11:31:04.186 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00038659700000000003 2024-03-19T11:31:04.187 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.187 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.187 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-03-19T11:31:04.187 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.279397+0000", 2024-03-19T11:31:04.187 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3574000000000001e-05 2024-03-19T11:31:04.187 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.187 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.187 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.187 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.282908+0000", 2024-03-19T11:31:04.187 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0035110110000000001 2024-03-19T11:31:04.187 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.187 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.187 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:04.187 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.283059+0000", 2024-03-19T11:31:04.187 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015160799999999999 2024-03-19T11:31:04.188 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.188 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.188 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1", 2024-03-19T11:31:04.188 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.283194+0000", 2024-03-19T11:31:04.188 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013435399999999999 2024-03-19T11:31:04.188 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.188 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.188 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:04.188 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.292934+0000", 2024-03-19T11:31:04.188 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0097397539999999998 2024-03-19T11:31:04.188 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.188 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.188 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:04.188 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.293026+0000", 2024-03-19T11:31:04.188 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2195999999999996e-05 2024-03-19T11:31:04.189 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.189 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.189 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:04.189 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.293075+0000", 2024-03-19T11:31:04.189 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9484000000000003e-05 2024-03-19T11:31:04.189 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.189 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.189 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:04.189 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.293128+0000", 2024-03-19T11:31:04.189 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3022999999999997e-05 2024-03-19T11:31:04.189 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.189 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:04.190 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.190 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.190 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.190 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.24 75/74 [PushOp(4:27909373:::smithi03829856-46:8, version: 46'59, data_included: [756315~767970,1933572~280606], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:8@46'59, size: 2756315, copy_subset: [(0, 2756315)], clone_subset: {}, snapset: 10=[]:{8: [6], c: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2214178, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:266590a5:::smithi03829856-16:e, version: 70'107, data_included: [652105~662359,1728171~386217], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:e@70'107, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 10=[]:{e: [d, b, 6]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:266590a5:::smithi03829856-16:head, version: 70'108, data_included: [652105~662359,1728171~386217], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:head@70'108, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:27909373:::smithi03829856-46:c, version: 70'109, data_included: [628595~895690,1807440~152886], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:c@70'109, size: 3428595, copy_subset: [(0, 3428595)], clone_subset: {}, snapset: 10=[]:{8: [6], c: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1960326, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:27909373:::smithi03829856-46:head, version: 70'110, data_included: [403401~454961,1584925~506039], data_size: 961000, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:head@70'110, size: 2403401, copy_subset: [(0, 2403401)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2403401, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:04.190 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.099395+0000", 2024-03-19T11:31:04.190 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 10.047476776, 2024-03-19T11:31:04.190 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.045391157000000001, 2024-03-19T11:31:04.190 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:04.190 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:04.190 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:04.190 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.190 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:04.190 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.099395+0000", 2024-03-19T11:31:04.190 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.191 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.191 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.191 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:04.191 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.099395+0000", 2024-03-19T11:31:04.191 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.191 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.191 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.191 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:04.191 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.099397+0000", 2024-03-19T11:31:04.191 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.691e-06 2024-03-19T11:31:04.191 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.191 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.191 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:04.191 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105009+0000", 2024-03-19T11:31:04.191 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0056121349999999999 2024-03-19T11:31:04.192 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.192 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.192 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:04.192 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105077+0000", 2024-03-19T11:31:04.192 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8561999999999998e-05 2024-03-19T11:31:04.192 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.192 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.192 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:04.192 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105133+0000", 2024-03-19T11:31:04.192 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5590000000000001e-05 2024-03-19T11:31:04.192 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.192 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.192 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.192 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105531+0000", 2024-03-19T11:31:04.193 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00039756200000000001 2024-03-19T11:31:04.193 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.193 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.193 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:04.193 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105579+0000", 2024-03-19T11:31:04.193 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8028999999999997e-05 2024-03-19T11:31:04.193 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.193 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.193 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:04.193 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.144786+0000", 2024-03-19T11:31:04.193 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.039207588000000002 2024-03-19T11:31:04.194 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.194 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:04.194 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.194 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.194 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.194 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.24 e75,74 4:266590a5:::smithi03829856-16:4,47'63,4:27909373:::smithi03829856-46:5,55'79,4:266590a5:::smithi03829856-16:10,67'105)", 2024-03-19T11:31:04.194 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.105191+0000", 2024-03-19T11:31:04.194 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 10.041680676, 2024-03-19T11:31:04.194 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.039580830999999997, 2024-03-19T11:31:04.194 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:04.194 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:04.194 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:04.194 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.194 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:04.195 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105191+0000", 2024-03-19T11:31:04.195 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.195 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.195 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.195 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:04.195 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105191+0000", 2024-03-19T11:31:04.195 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.195 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.195 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.195 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:04.195 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105192+0000", 2024-03-19T11:31:04.195 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.099e-06 2024-03-19T11:31:04.195 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.195 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.195 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:04.196 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105200+0000", 2024-03-19T11:31:04.196 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5279999999999998e-06 2024-03-19T11:31:04.196 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.196 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.196 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:04.196 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105211+0000", 2024-03-19T11:31:04.196 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1471000000000001e-05 2024-03-19T11:31:04.196 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.196 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.196 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:04.196 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105220+0000", 2024-03-19T11:31:04.196 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3469999999999999e-06 2024-03-19T11:31:04.196 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.196 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.196 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.197 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.144219+0000", 2024-03-19T11:31:04.197 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.038999024 2024-03-19T11:31:04.197 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.197 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.197 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:04.197 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.144257+0000", 2024-03-19T11:31:04.197 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8501999999999998e-05 2024-03-19T11:31:04.197 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.197 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.197 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:04.197 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.144772+0000", 2024-03-19T11:31:04.197 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00051486000000000001 2024-03-19T11:31:04.197 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.197 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:04.197 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.198 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.198 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.198 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 75/74 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:7, version: 55'82, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@55'82, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 58'86, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@58'86, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [504210~968657,1572695~79919], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:12, version: 66'101, data_included: [440350~816629,1309629~231947], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:12@66'101, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:10, version: 70'112, data_included: [440350~816629,1309629~231947], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:10@70'112, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 70'113, data_included: [440350~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@70'113, size: 2605085, copy_subset: [(0, 2605085)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1488926, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:04.198 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.107266+0000", 2024-03-19T11:31:04.198 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 10.039606159, 2024-03-19T11:31:04.198 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.048592669999999998, 2024-03-19T11:31:04.198 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:04.198 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:04.198 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:04.198 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.199 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:04.199 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.107266+0000", 2024-03-19T11:31:04.199 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.199 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.199 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.199 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:04.199 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.107266+0000", 2024-03-19T11:31:04.199 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.199 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.199 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.199 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:04.199 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.107267+0000", 2024-03-19T11:31:04.199 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1459999999999999e-06 2024-03-19T11:31:04.199 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.200 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.200 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:04.200 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.122861+0000", 2024-03-19T11:31:04.200 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01559372 2024-03-19T11:31:04.200 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.200 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.200 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:04.200 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.122932+0000", 2024-03-19T11:31:04.200 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1477999999999996e-05 2024-03-19T11:31:04.200 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.200 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.200 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:04.200 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.122999+0000", 2024-03-19T11:31:04.200 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6832e-05 2024-03-19T11:31:04.200 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.201 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.201 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.201 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123332+0000", 2024-03-19T11:31:04.201 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000333271 2024-03-19T11:31:04.201 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.201 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.201 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:04.201 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123371+0000", 2024-03-19T11:31:04.201 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8525999999999997e-05 2024-03-19T11:31:04.201 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.201 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.201 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:04.201 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.155858+0000", 2024-03-19T11:31:04.201 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.032487697000000003 2024-03-19T11:31:04.202 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.202 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:04.202 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.202 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.202 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.202 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.21 e75,74 4:8401c1f0:::smithi03829856-43:8,46'66,4:866f3dcc:::smithi03829856-31:2,47'73)", 2024-03-19T11:31:04.202 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.123075+0000", 2024-03-19T11:31:04.202 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 10.02379694, 2024-03-19T11:31:04.202 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.033023045000000001, 2024-03-19T11:31:04.202 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:04.202 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:04.202 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:04.202 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.202 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:04.203 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123075+0000", 2024-03-19T11:31:04.203 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.203 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.203 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.203 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:04.203 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123075+0000", 2024-03-19T11:31:04.203 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.203 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.203 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.203 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:04.203 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123076+0000", 2024-03-19T11:31:04.203 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5e-06 2024-03-19T11:31:04.203 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.203 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.203 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:04.204 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123086+0000", 2024-03-19T11:31:04.204 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8060000000000006e-06 2024-03-19T11:31:04.204 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.204 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.204 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:04.204 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123099+0000", 2024-03-19T11:31:04.204 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3251000000000001e-05 2024-03-19T11:31:04.204 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.204 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.204 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:04.204 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123112+0000", 2024-03-19T11:31:04.204 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2774e-05 2024-03-19T11:31:04.204 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.204 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.204 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.205 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.155902+0000", 2024-03-19T11:31:04.205 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.032790225999999999 2024-03-19T11:31:04.205 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.205 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.205 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:04.205 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.155914+0000", 2024-03-19T11:31:04.205 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1476e-05 2024-03-19T11:31:04.205 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.205 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.205 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:04.205 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.156098+0000", 2024-03-19T11:31:04.205 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000184012 2024-03-19T11:31:04.205 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.205 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:04.206 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.206 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.206 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.206 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 75/74 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:7, version: 55'82, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@55'82, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 58'86, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@58'86, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [1652614~973634,2888511~52681,3182305~22261], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3204566, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:12, version: 66'101, data_included: [1541576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:12@66'101, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:10, version: 70'112, data_included: [1541576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:10@70'112, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 70'113, data_included: [1488926~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@70'113, size: 2605085, copy_subset: [(0, 2605085)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2537502, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1488926, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:04.206 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.174030+0000", 2024-03-19T11:31:04.206 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 9.9728417359999995, 2024-03-19T11:31:04.206 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.040799275000000003, 2024-03-19T11:31:04.206 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:04.206 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:04.206 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:04.206 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.206 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:04.206 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.174030+0000", 2024-03-19T11:31:04.207 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.207 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.207 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.207 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:04.207 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.174030+0000", 2024-03-19T11:31:04.207 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.207 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.207 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.207 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:04.207 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.174031+0000", 2024-03-19T11:31:04.207 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.322e-06 2024-03-19T11:31:04.207 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.208 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.208 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:04.208 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.184771+0000", 2024-03-19T11:31:04.208 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010739227 2024-03-19T11:31:04.208 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.208 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.208 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:04.208 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.184872+0000", 2024-03-19T11:31:04.208 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000101234 2024-03-19T11:31:04.208 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.208 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.208 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:04.208 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.184985+0000", 2024-03-19T11:31:04.208 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000113545 2024-03-19T11:31:04.209 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.209 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.209 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.209 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.185463+0000", 2024-03-19T11:31:04.209 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00047749899999999999 2024-03-19T11:31:04.209 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.209 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.209 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:04.209 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.185534+0000", 2024-03-19T11:31:04.210 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1007999999999995e-05 2024-03-19T11:31:04.210 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.210 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.210 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:04.210 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.214829+0000", 2024-03-19T11:31:04.210 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029295439999999999 2024-03-19T11:31:04.210 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.210 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:04.210 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.210 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.210 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.210 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.21 e75,74 4:866f3dcc:::smithi03829856-31:c,70'114)", 2024-03-19T11:31:04.210 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.197457+0000", 2024-03-19T11:31:04.210 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 9.9494146899999993, 2024-03-19T11:31:04.211 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018493590000000001, 2024-03-19T11:31:04.211 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:04.211 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:04.211 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:04.211 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.211 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:04.211 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.197457+0000", 2024-03-19T11:31:04.211 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.211 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.211 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.211 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:04.211 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.197457+0000", 2024-03-19T11:31:04.211 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.211 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.212 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.212 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:04.212 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.197459+0000", 2024-03-19T11:31:04.212 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3199999999999998e-06 2024-03-19T11:31:04.212 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.212 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.212 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:04.212 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.197480+0000", 2024-03-19T11:31:04.212 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0366000000000001e-05 2024-03-19T11:31:04.212 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.212 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.212 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:04.212 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.197521+0000", 2024-03-19T11:31:04.212 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0738000000000002e-05 2024-03-19T11:31:04.212 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.212 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.213 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:04.213 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.197547+0000", 2024-03-19T11:31:04.213 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6548999999999999e-05 2024-03-19T11:31:04.213 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.213 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.213 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.213 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.214954+0000", 2024-03-19T11:31:04.213 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01740713 2024-03-19T11:31:04.213 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.213 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.213 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:04.213 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.214972+0000", 2024-03-19T11:31:04.213 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7520000000000002e-05 2024-03-19T11:31:04.213 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.214 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.214 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:04.214 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.215951+0000", 2024-03-19T11:31:04.214 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00097896700000000007 2024-03-19T11:31:04.214 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.214 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:04.214 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.214 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.214 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.214 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 75/74 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [2599014~241336], data_size: 241336, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2840350, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [2599014~241336,2948041~446940], data_size: 688276, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3394981, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [3204566~621403], data_size: 621403, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3865577, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3204566, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:12, version: 66'101, data_included: [2590152~250198,2948041~581396], data_size: 831594, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:12@66'101, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3529437, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:10, version: 70'112, data_included: [2590152~250198,2948041~581396], data_size: 831594, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:10@70'112, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3529437, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 70'113, data_included: [2537502~67583], data_size: 67583, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@70'113, size: 2605085, copy_subset: [(0, 2605085)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2605085, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2537502, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:04.214 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.222872+0000", 2024-03-19T11:31:04.214 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 9.9240002430000001, 2024-03-19T11:31:04.214 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016277658, 2024-03-19T11:31:04.215 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:04.215 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:04.215 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:04.215 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.215 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:04.215 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.222872+0000", 2024-03-19T11:31:04.215 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.215 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.215 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.215 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:04.215 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.222872+0000", 2024-03-19T11:31:04.215 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.215 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.215 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.216 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:04.216 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.222874+0000", 2024-03-19T11:31:04.216 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9999999999999999e-06 2024-03-19T11:31:04.216 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.216 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.216 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:04.216 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.227563+0000", 2024-03-19T11:31:04.216 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046897570000000001 2024-03-19T11:31:04.216 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.216 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.216 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:04.216 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.227639+0000", 2024-03-19T11:31:04.216 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5607999999999998e-05 2024-03-19T11:31:04.216 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.217 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.217 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:04.217 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.227736+0000", 2024-03-19T11:31:04.217 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6854000000000005e-05 2024-03-19T11:31:04.217 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.217 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.217 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.217 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.228149+0000", 2024-03-19T11:31:04.217 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00041330000000000002 2024-03-19T11:31:04.217 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.217 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.217 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:04.217 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.228219+0000", 2024-03-19T11:31:04.217 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.0140000000000003e-05 2024-03-19T11:31:04.218 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.218 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.218 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:04.218 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.239149+0000", 2024-03-19T11:31:04.218 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010929998999999999 2024-03-19T11:31:04.218 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.218 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:04.218 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.218 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.218 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.218 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2372 4.16 4:6868af91:::smithi03829856-10:b [read 832512~818688] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-03-19T11:31:04.218 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:58.455732+0000", 2024-03-19T11:31:04.218 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.6911400209999998, 2024-03-19T11:31:04.218 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.69150541200000004, 2024-03-19T11:31:04.219 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:04.219 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:04.219 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:04.219 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:04.219 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:04.219 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2372 2024-03-19T11:31:04.219 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.219 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:04.219 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.219 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:04.219 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455732+0000", 2024-03-19T11:31:04.219 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.220 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.220 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.220 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:04.220 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455732+0000", 2024-03-19T11:31:04.220 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.220 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.220 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.220 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:04.220 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455735+0000", 2024-03-19T11:31:04.220 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.756e-06 2024-03-19T11:31:04.220 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.220 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.220 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:04.221 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455744+0000", 2024-03-19T11:31:04.221 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1880000000000008e-06 2024-03-19T11:31:04.221 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.221 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.221 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:04.221 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455764+0000", 2024-03-19T11:31:04.221 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0044999999999999e-05 2024-03-19T11:31:04.221 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.221 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.221 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:04.221 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455774+0000", 2024-03-19T11:31:04.221 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8770000000000005e-06 2024-03-19T11:31:04.221 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.221 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.221 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.222 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455837+0000", 2024-03-19T11:31:04.222 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3588000000000002e-05 2024-03-19T11:31:04.222 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.222 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.222 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:04.222 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455846+0000", 2024-03-19T11:31:04.222 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6030000000000005e-06 2024-03-19T11:31:04.222 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.222 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.222 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.222 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.076525+0000", 2024-03-19T11:31:04.222 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.62067875900000002 2024-03-19T11:31:04.222 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.223 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.223 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:31:04.223 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.076981+0000", 2024-03-19T11:31:04.223 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00045633300000000001 2024-03-19T11:31:04.223 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.223 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.223 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.223 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.104422+0000", 2024-03-19T11:31:04.223 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027440665999999999 2024-03-19T11:31:04.223 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.223 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.223 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:31:04.223 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.104786+0000", 2024-03-19T11:31:04.223 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00036471499999999999 2024-03-19T11:31:04.223 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.224 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.224 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.224 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.140725+0000", 2024-03-19T11:31:04.224 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.035939093999999998 2024-03-19T11:31:04.224 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.224 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.224 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:31:04.224 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.140841+0000", 2024-03-19T11:31:04.224 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000115207 2024-03-19T11:31:04.224 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.224 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.224 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.224 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.145559+0000", 2024-03-19T11:31:04.224 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.004718486 2024-03-19T11:31:04.224 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.224 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.225 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:04.225 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.145760+0000", 2024-03-19T11:31:04.225 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000201058 2024-03-19T11:31:04.225 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.225 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.225 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:04.225 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.147237+0000", 2024-03-19T11:31:04.225 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014770370000000001 2024-03-19T11:31:04.225 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.225 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:04.225 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.225 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.225 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.225 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.16 80/79 [PushOp(4:6d9093b1:::smithi03829856-38:head, version: 77'90, data_included: [500875~689757,1564955~358819], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:6d9093b1:::smithi03829856-38:head@77'90, size: 2100875, copy_subset: [(0, 2100875)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1923774, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:6be89c06:::smithi03829856-24:head, version: 77'92, data_included: [741376~589824,1753088~458752], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:6be89c06:::smithi03829856-24:head@77'92, size: 2386561, copy_subset: [(0, 2386561)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2211840, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:04.226 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:59.081830+0000", 2024-03-19T11:31:04.226 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.0650419949999996, 2024-03-19T11:31:04.226 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015229652999999999, 2024-03-19T11:31:04.226 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:04.226 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:04.226 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:04.226 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.226 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:04.226 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.081830+0000", 2024-03-19T11:31:04.226 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.226 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.226 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.226 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:04.226 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.081830+0000", 2024-03-19T11:31:04.227 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.227 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.227 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.227 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:04.227 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.081832+0000", 2024-03-19T11:31:04.227 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.731e-06 2024-03-19T11:31:04.227 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.227 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.227 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:04.227 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.083140+0000", 2024-03-19T11:31:04.227 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0013082829999999999 2024-03-19T11:31:04.227 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.227 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.227 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:04.228 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.083170+0000", 2024-03-19T11:31:04.228 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0270999999999999e-05 2024-03-19T11:31:04.228 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.228 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.228 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:04.228 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.083199+0000", 2024-03-19T11:31:04.228 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9298e-05 2024-03-19T11:31:04.228 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.228 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.228 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.228 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.089969+0000", 2024-03-19T11:31:04.228 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0067691289999999996 2024-03-19T11:31:04.228 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.228 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.228 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:04.229 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.089995+0000", 2024-03-19T11:31:04.229 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5995000000000001e-05 2024-03-19T11:31:04.229 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.229 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.229 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:04.229 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.097060+0000", 2024-03-19T11:31:04.229 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0070649459999999999 2024-03-19T11:31:04.229 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.229 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:04.229 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.229 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.229 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.229 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.6 80/79 [PushOp(4:626acd4d:::smithi03829856-2:8, version: 55'54, data_included: [416807~475249,1542129~474678], data_size: 949927, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:626acd4d:::smithi03829856-2:8@55'54, size: 2016807, copy_subset: [(0, 2016807)], clone_subset: {}, snapset: 15=[]:{8: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2016807, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:04.229 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:59.110928+0000", 2024-03-19T11:31:04.229 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.035943681, 2024-03-19T11:31:04.230 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01605765, 2024-03-19T11:31:04.230 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:04.230 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:04.230 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:04.230 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.230 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:04.230 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.110928+0000", 2024-03-19T11:31:04.230 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.230 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.230 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.231 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:04.231 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.110928+0000", 2024-03-19T11:31:04.231 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.231 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.231 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.231 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:04.231 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.110929+0000", 2024-03-19T11:31:04.231 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6500000000000008e-07 2024-03-19T11:31:04.231 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.231 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.232 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:04.232 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.111446+0000", 2024-03-19T11:31:04.232 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00051713900000000001 2024-03-19T11:31:04.232 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.232 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.232 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:04.232 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.111464+0000", 2024-03-19T11:31:04.232 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7966e-05 2024-03-19T11:31:04.232 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.232 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.232 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:04.232 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.111483+0000", 2024-03-19T11:31:04.232 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8561000000000001e-05 2024-03-19T11:31:04.232 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.233 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.233 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.233 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.118569+0000", 2024-03-19T11:31:04.233 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0070860259999999996 2024-03-19T11:31:04.233 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.233 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.233 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:04.233 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.118586+0000", 2024-03-19T11:31:04.233 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6857000000000001e-05 2024-03-19T11:31:04.233 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.233 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.233 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:04.233 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.126986+0000", 2024-03-19T11:31:04.234 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0084001360000000008 2024-03-19T11:31:04.234 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.234 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:04.234 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.234 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.234 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.234 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.16 80/79 [PushOp(4:6868af91:::smithi03829856-10:10, version: 70'81, data_included: [524385~820472,1602039~228104], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:10@70'81, size: 3747928, copy_subset: [(0, 3747928)], clone_subset: {}, snapset: 14=[]:{7: [6], b: [b], 10: [d], 14: [13, 12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1830143, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:04.234 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:59.111343+0000", 2024-03-19T11:31:04.234 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.0355290909999999, 2024-03-19T11:31:04.234 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019054888999999998, 2024-03-19T11:31:04.234 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:04.234 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:04.234 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:04.235 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.235 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:04.235 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.111343+0000", 2024-03-19T11:31:04.235 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.235 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.235 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.235 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:04.235 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.111343+0000", 2024-03-19T11:31:04.235 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.235 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.235 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.235 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:04.235 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.111349+0000", 2024-03-19T11:31:04.235 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9619999999999998e-06 2024-03-19T11:31:04.235 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.236 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.236 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:04.236 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.113881+0000", 2024-03-19T11:31:04.236 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025322930000000001 2024-03-19T11:31:04.236 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.236 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.236 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:04.236 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.113917+0000", 2024-03-19T11:31:04.236 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5593999999999998e-05 2024-03-19T11:31:04.236 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.236 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.236 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:04.236 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.113962+0000", 2024-03-19T11:31:04.236 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5362999999999999e-05 2024-03-19T11:31:04.237 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.237 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.237 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.237 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.126871+0000", 2024-03-19T11:31:04.237 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012909273000000001 2024-03-19T11:31:04.237 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.237 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.237 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:04.237 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.126892+0000", 2024-03-19T11:31:04.237 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0678000000000001e-05 2024-03-19T11:31:04.237 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.237 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.237 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:04.237 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.130398+0000", 2024-03-19T11:31:04.237 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0035057259999999998 2024-03-19T11:31:04.238 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.238 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:04.238 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.238 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.238 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.238 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.16 80/79 [PushOp(4:6868af91:::smithi03829856-10:7, version: 55'57, data_included: [524385~820472,1602039~228104], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:7@55'57, size: 2924385, copy_subset: [(0, 2924385)], clone_subset: {}, snapset: 14=[]:{7: [6], b: [b], 10: [d], 14: [13, 12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1830143, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:6868af91:::smithi03829856-10:14, version: 74'85, data_included: [405504~950272,1527054~98304], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:14@74'85, size: 2420208, copy_subset: [(0, 2420208)], clone_subset: {}, snapset: 14=[]:{7: [6], b: [b], 10: [d], 14: [13, 12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1625358, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:04.238 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:59.111755+0000", 2024-03-19T11:31:04.238 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.03511664, 2024-03-19T11:31:04.238 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014604127, 2024-03-19T11:31:04.238 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:04.238 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:04.238 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:04.238 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.239 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:04.239 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.111755+0000", 2024-03-19T11:31:04.239 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.239 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.239 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.239 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:04.239 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.111755+0000", 2024-03-19T11:31:04.239 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.239 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.239 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.239 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:04.239 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.111756+0000", 2024-03-19T11:31:04.239 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4000000000000001e-07 2024-03-19T11:31:04.239 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.239 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.240 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:04.240 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.112850+0000", 2024-03-19T11:31:04.240 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0010945079999999999 2024-03-19T11:31:04.240 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.240 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.240 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:04.240 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.112887+0000", 2024-03-19T11:31:04.241 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6971999999999998e-05 2024-03-19T11:31:04.241 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.241 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.241 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:04.241 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.112916+0000", 2024-03-19T11:31:04.241 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8580000000000001e-05 2024-03-19T11:31:04.241 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.241 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.241 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.242 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.118590+0000", 2024-03-19T11:31:04.242 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0056736599999999996 2024-03-19T11:31:04.242 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.242 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.242 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:04.242 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.118617+0000", 2024-03-19T11:31:04.242 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.7572e-05 2024-03-19T11:31:04.242 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.242 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.242 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:04.242 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.126359+0000", 2024-03-19T11:31:04.243 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077420950000000001 2024-03-19T11:31:04.243 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.243 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:04.243 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.243 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.243 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.243 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.16 80/79 [PushOp(4:6d9093b1:::smithi03829856-38:10, version: 70'83, data_included: [621859~568773,1729599~479803], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6d9093b1:::smithi03829856-38:10@70'83, size: 3142876, copy_subset: [(0, 3142876)], clone_subset: {}, snapset: 10=[]:{10: [d, b, 6]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2209402, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:04.243 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:59.114057+0000", 2024-03-19T11:31:04.243 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.0328150620000001, 2024-03-19T11:31:04.243 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019935583, 2024-03-19T11:31:04.243 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:04.243 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:04.244 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:04.244 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.244 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:04.244 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.114057+0000", 2024-03-19T11:31:04.244 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.244 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.244 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.244 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:04.244 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.114057+0000", 2024-03-19T11:31:04.244 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.244 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.244 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.244 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:04.244 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.114058+0000", 2024-03-19T11:31:04.245 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4300000000000001e-07 2024-03-19T11:31:04.245 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.245 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.245 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:04.245 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.117243+0000", 2024-03-19T11:31:04.245 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031852550000000001 2024-03-19T11:31:04.245 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.245 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.245 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:04.245 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.117263+0000", 2024-03-19T11:31:04.245 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0469999999999999e-05 2024-03-19T11:31:04.245 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.245 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.245 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:04.246 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.117276+0000", 2024-03-19T11:31:04.246 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2775000000000001e-05 2024-03-19T11:31:04.246 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.246 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.246 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.246 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.130681+0000", 2024-03-19T11:31:04.246 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013404879999999999 2024-03-19T11:31:04.246 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.246 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.246 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:04.246 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.130699+0000", 2024-03-19T11:31:04.246 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8343e-05 2024-03-19T11:31:04.246 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.247 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.247 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:04.247 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.133992+0000", 2024-03-19T11:31:04.247 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0032929169999999998 2024-03-19T11:31:04.247 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.247 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:04.247 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.247 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.247 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.247 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2442 4.16 4:6d9093b1:::smithi03829856-38:head [write 460558~707131] snapc 17=[17,16,15,13,12,b,6] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-03-19T11:31:04.247 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:03.432382+0000", 2024-03-19T11:31:04.247 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.71448951100000002, 2024-03-19T11:31:04.247 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019918379, 2024-03-19T11:31:04.248 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:04.248 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:04.248 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:04.248 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:04.248 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:04.248 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2442 2024-03-19T11:31:04.248 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.248 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:04.248 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.248 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:04.248 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.432382+0000", 2024-03-19T11:31:04.248 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.248 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.248 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.249 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:04.249 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.432382+0000", 2024-03-19T11:31:04.249 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.249 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.249 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.249 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:04.249 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.432385+0000", 2024-03-19T11:31:04.249 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1e-06 2024-03-19T11:31:04.249 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.249 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.250 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:04.250 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.433115+0000", 2024-03-19T11:31:04.250 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00072939900000000004 2024-03-19T11:31:04.250 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.250 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.250 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:04.250 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.433151+0000", 2024-03-19T11:31:04.250 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6094999999999999e-05 2024-03-19T11:31:04.250 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.250 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.250 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:04.250 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.433167+0000", 2024-03-19T11:31:04.250 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6152999999999999e-05 2024-03-19T11:31:04.250 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.251 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.251 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.251 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.433339+0000", 2024-03-19T11:31:04.251 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000171478 2024-03-19T11:31:04.251 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.251 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.251 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:04.252 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.433621+0000", 2024-03-19T11:31:04.252 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00028283900000000001 2024-03-19T11:31:04.252 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.252 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.252 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3", 2024-03-19T11:31:04.252 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.434169+0000", 2024-03-19T11:31:04.253 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00054786699999999995 2024-03-19T11:31:04.253 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.253 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.253 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:04.253 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.443136+0000", 2024-03-19T11:31:04.253 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0089667419999999998 2024-03-19T11:31:04.254 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.254 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.254 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:04.254 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.452230+0000", 2024-03-19T11:31:04.254 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.009093996 2024-03-19T11:31:04.254 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.255 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.255 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:04.255 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.452264+0000", 2024-03-19T11:31:04.255 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4381000000000002e-05 2024-03-19T11:31:04.255 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.255 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.255 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:04.255 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.452301+0000", 2024-03-19T11:31:04.255 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6328999999999997e-05 2024-03-19T11:31:04.255 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.255 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:04.255 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.255 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.256 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:04.256 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:04.257 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-19T11:31:04.257 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:04.265+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:04.302 INFO:tasks.rados.rados.0.smithi038.stdout:1186: finishing write tid 1 to smithi03829856-27 2024-03-19T11:31:04.302 INFO:tasks.rados.rados.0.smithi038.stdout:1186: finishing write tid 2 to smithi03829856-27 2024-03-19T11:31:04.302 INFO:tasks.rados.rados.0.smithi038.stdout:1186: finishing write tid 3 to smithi03829856-27 2024-03-19T11:31:04.302 INFO:tasks.rados.rados.0.smithi038.stdout:1186: finishing write tid 4 to smithi03829856-27 2024-03-19T11:31:04.302 INFO:tasks.rados.rados.0.smithi038.stdout:1186: finishing write tid 5 to smithi03829856-27 2024-03-19T11:31:04.302 INFO:tasks.rados.rados.0.smithi038.stdout:1186: oid 27 updating version 0 to 94 2024-03-19T11:31:04.302 INFO:tasks.rados.rados.0.smithi038.stdout:1186: oid 27 updating version 94 to 97 2024-03-19T11:31:04.302 INFO:tasks.rados.rados.0.smithi038.stdout:1186: oid 27 version 97 is already newer than 95 2024-03-19T11:31:04.302 INFO:tasks.rados.rados.0.smithi038.stdout:1186: oid 27 version 97 is already newer than 96 2024-03-19T11:31:04.302 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 27 v 97 (ObjNum 192 snap 22 seq_num 192) dirty exists 2024-03-19T11:31:04.302 INFO:tasks.rados.rados.0.smithi038.stdout:1186: left oid 27 (ObjNum 192 snap 22 seq_num 192) 2024-03-19T11:31:04.303 INFO:tasks.rados.rados.0.smithi038.stdout:1192: finishing write tid 1 to smithi03829856-38 2024-03-19T11:31:04.303 INFO:tasks.rados.rados.0.smithi038.stdout:1192: finishing write tid 2 to smithi03829856-38 2024-03-19T11:31:04.303 INFO:tasks.rados.rados.0.smithi038.stdout:1192: finishing write tid 3 to smithi03829856-38 2024-03-19T11:31:04.303 INFO:tasks.rados.rados.0.smithi038.stdout:1192: finishing write tid 4 to smithi03829856-38 2024-03-19T11:31:04.303 INFO:tasks.rados.rados.0.smithi038.stdout:1192: finishing write tid 5 to smithi03829856-38 2024-03-19T11:31:04.303 INFO:tasks.rados.rados.0.smithi038.stdout:1192: oid 38 updating version 0 to 103 2024-03-19T11:31:04.303 INFO:tasks.rados.rados.0.smithi038.stdout:1192: oid 38 version 103 is already newer than 101 2024-03-19T11:31:04.303 INFO:tasks.rados.rados.0.smithi038.stdout:1192: oid 38 updating version 103 to 104 2024-03-19T11:31:04.303 INFO:tasks.rados.rados.0.smithi038.stdout:1192: oid 38 version 104 is already newer than 102 2024-03-19T11:31:04.303 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 38 v 104 (ObjNum 194 snap 22 seq_num 194) dirty exists 2024-03-19T11:31:04.303 INFO:tasks.rados.rados.0.smithi038.stdout:1192: left oid 38 (ObjNum 194 snap 22 seq_num 194) 2024-03-19T11:31:04.303 INFO:tasks.rados.rados.0.smithi038.stdout:1193: expect (ObjNum 185 snap 20 seq_num 185) 2024-03-19T11:31:04.358 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:04.367+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:04.398 INFO:tasks.rados.rados.0.smithi038.stdout:1195: expect (ObjNum 31 snap 0 seq_num 31) 2024-03-19T11:31:04.418 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:04.418 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:31:04.418 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:31:04.419 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:04.433 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-19T11:31:04.459 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:04.471+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:04.474 INFO:tasks.rados.rados.0.smithi038.stdout:1198: got (22) Invalid argument 2024-03-19T11:31:04.475 INFO:tasks.rados.rados.0.smithi038.stdout:1196: finishing set_chunk to oid 6 2024-03-19T11:31:04.475 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 6 set_chunk returned error code -95 offset: 1022976 length: 1661440 2024-03-19T11:31:04.475 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 6 v 0 (ObjNum 53 snap 0 seq_num 53) dirty exists 2024-03-19T11:31:04.475 INFO:tasks.rados.rados.0.smithi038.stdout:1197: expect (ObjNum 158 snap 17 seq_num 158) 2024-03-19T11:31:04.543 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:04.543 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:31:04.543 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:31:04.543 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:31:04.543 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:04.555 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-19T11:31:04.600 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:04.609+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:04.626 INFO:tasks.rados.rados.0.smithi038.stdout:1185: done (11 left) 2024-03-19T11:31:04.626 INFO:tasks.rados.rados.0.smithi038.stdout:1186: done (10 left) 2024-03-19T11:31:04.626 INFO:tasks.rados.rados.0.smithi038.stdout:1187: done (9 left) 2024-03-19T11:31:04.626 INFO:tasks.rados.rados.0.smithi038.stdout:1188: done (8 left) 2024-03-19T11:31:04.626 INFO:tasks.rados.rados.0.smithi038.stdout:1189: done (7 left) 2024-03-19T11:31:04.626 INFO:tasks.rados.rados.0.smithi038.stdout:1190: done (6 left) 2024-03-19T11:31:04.626 INFO:tasks.rados.rados.0.smithi038.stdout:1192: done (5 left) 2024-03-19T11:31:04.627 INFO:tasks.rados.rados.0.smithi038.stdout:1193: done (4 left) 2024-03-19T11:31:04.627 INFO:tasks.rados.rados.0.smithi038.stdout:1195: done (3 left) 2024-03-19T11:31:04.627 INFO:tasks.rados.rados.0.smithi038.stdout:1196: done (2 left) 2024-03-19T11:31:04.627 INFO:tasks.rados.rados.0.smithi038.stdout:1197: done (1 left) 2024-03-19T11:31:04.627 INFO:tasks.rados.rados.0.smithi038.stdout:1198: done (0 left) 2024-03-19T11:31:04.627 INFO:tasks.rados.rados.0.smithi038.stdout:1199: read oid 47 snap -1 2024-03-19T11:31:04.627 INFO:tasks.rados.rados.0.smithi038.stdout:1199: expect (ObjNum 46 snap 0 seq_num 46) 2024-03-19T11:31:04.627 INFO:tasks.rados.rados.0.smithi038.stdout:1200: tier_evict oid 15 2024-03-19T11:31:04.627 INFO:tasks.rados.rados.0.smithi038.stdout:1200: tier_evict oid 15 snap -1 2024-03-19T11:31:04.627 INFO:tasks.rados.rados.0.smithi038.stdout:1201: write (excl) oid 6 current snap is 22 2024-03-19T11:31:04.627 INFO:tasks.rados.rados.0.smithi038.stdout:1201: seq_num 195 ranges {477365=782380,1927871=768727,3122307=155058} 2024-03-19T11:31:04.634 INFO:tasks.rados.rados.0.smithi038.stdout:1201: writing smithi03829856-6 from 477365 to 1259745 tid 1 2024-03-19T11:31:04.641 INFO:tasks.rados.rados.0.smithi038.stdout:1201: writing smithi03829856-6 from 1927871 to 2696598 tid 2 2024-03-19T11:31:04.645 INFO:tasks.rados.rados.0.smithi038.stdout:1201: writing smithi03829856-6 from 3122307 to 3277365 tid 3 2024-03-19T11:31:04.645 INFO:tasks.rados.rados.0.smithi038.stdout:1202: set_chunk oid 35 target oid 2024-03-19T11:31:04.653 INFO:tasks.rados.rados.0.smithi038.stdout:1202: set_chunk oid 35 offset: 1184256 length: 844800 target oid 3009d07cfa21d15bebc8bdd7645a21ffbdcc4a6d3830a1117fc4c2dbf95cee4f offset: 0 snap -1 2024-03-19T11:31:04.654 INFO:tasks.rados.rados.0.smithi038.stdout:1203: read oid 10 snap -1 2024-03-19T11:31:04.654 INFO:tasks.rados.rados.0.smithi038.stdout:1203: expect (ObjNum 179 snap 19 seq_num 179) 2024-03-19T11:31:04.654 INFO:tasks.rados.rados.0.smithi038.stdout:1204: set_chunk oid 50 target oid 2024-03-19T11:31:04.661 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:04.669+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:04.666 INFO:tasks.rados.rados.0.smithi038.stdout:1204: set_chunk oid 50 offset: 386560 length: 1251328 target oid cd4d50e534543db6dae6bc488323fa28594b9cb26f8c7a13861574839994caef offset: 0 snap -1 2024-03-19T11:31:04.666 INFO:tasks.rados.rados.0.smithi038.stdout:1205: read oid 22 snap -1 2024-03-19T11:31:04.666 INFO:tasks.rados.rados.0.smithi038.stdout:1205: expect (ObjNum 168 snap 17 seq_num 168) 2024-03-19T11:31:04.666 INFO:tasks.rados.rados.0.smithi038.stdout:1206: write oid 49 current snap is 22 2024-03-19T11:31:04.666 INFO:tasks.rados.rados.0.smithi038.stdout:1200: got (22) Invalid argument 2024-03-19T11:31:04.666 INFO:tasks.rados.rados.0.smithi038.stdout:1206: seq_num 196 ranges {494172=457629,1586353=512385,2749730=544442} 2024-03-19T11:31:04.670 INFO:tasks.rados.rados.0.smithi038.stdout:1206: writing smithi03829856-49 from 494172 to 951801 tid 1 2024-03-19T11:31:04.677 INFO:tasks.rados.rados.0.smithi038.stdout:1206: writing smithi03829856-49 from 1586353 to 2098738 tid 2 2024-03-19T11:31:04.685 INFO:tasks.rados.rados.0.smithi038.stdout:1206: writing smithi03829856-49 from 2749730 to 3294172 tid 3 2024-03-19T11:31:04.685 INFO:tasks.rados.rados.0.smithi038.stdout:1200: done (7 left) 2024-03-19T11:31:04.685 INFO:tasks.rados.rados.0.smithi038.stdout:1207: tier_promote oid 8 2024-03-19T11:31:04.686 INFO:tasks.rados.rados.0.smithi038.stdout:1199: expect (ObjNum 46 snap 0 seq_num 46) 2024-03-19T11:31:04.724 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:04.724 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:31:04.724 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:31:04.724 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:31:04.724 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.724 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1063 4.4 4:214026b3:::smithi03829856-47:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:31:04.724 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.784904+0000", 2024-03-19T11:31:04.724 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 50.949116545000003, 2024-03-19T11:31:04.724 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024425912000000001, 2024-03-19T11:31:04.724 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:04.724 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:04.725 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:04.725 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:04.725 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:04.725 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1063 2024-03-19T11:31:04.725 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.725 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:04.725 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.725 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:04.725 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784904+0000", 2024-03-19T11:31:04.725 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.725 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.726 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.726 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:04.726 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784904+0000", 2024-03-19T11:31:04.726 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.726 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.726 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.726 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:04.726 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784906+0000", 2024-03-19T11:31:04.726 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1409999999999999e-06 2024-03-19T11:31:04.726 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.726 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.726 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:04.726 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784915+0000", 2024-03-19T11:31:04.726 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5459999999999992e-06 2024-03-19T11:31:04.727 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.727 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.727 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:04.727 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784935+0000", 2024-03-19T11:31:04.727 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0310999999999999e-05 2024-03-19T11:31:04.727 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.727 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.727 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:04.727 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784946+0000", 2024-03-19T11:31:04.727 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0675e-05 2024-03-19T11:31:04.728 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.728 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.728 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.728 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.785032+0000", 2024-03-19T11:31:04.728 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6049000000000002e-05 2024-03-19T11:31:04.728 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.728 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.728 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:04.728 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.785136+0000", 2024-03-19T11:31:04.728 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000104209 2024-03-19T11:31:04.728 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.728 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.728 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:04.729 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.809330+0000", 2024-03-19T11:31:04.729 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024193981 2024-03-19T11:31:04.729 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.729 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:04.729 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.729 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.729 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.729 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1433 4.16 4:6be89c06:::smithi03829856-24:c [sparse-read 0~1100427,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:31:04.729 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.976652+0000", 2024-03-19T11:31:04.729 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 38.757368843999998, 2024-03-19T11:31:04.729 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95684753199999995, 2024-03-19T11:31:04.729 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:04.730 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:04.730 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:04.730 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:04.730 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:04.730 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1433 2024-03-19T11:31:04.730 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.730 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:04.730 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.730 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:04.730 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976652+0000", 2024-03-19T11:31:04.730 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.730 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.731 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.731 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:04.731 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976652+0000", 2024-03-19T11:31:04.731 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.731 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.731 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.731 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:04.731 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976654+0000", 2024-03-19T11:31:04.731 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6199999999999999e-06 2024-03-19T11:31:04.731 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.731 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.732 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:04.732 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976668+0000", 2024-03-19T11:31:04.732 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3213999999999999e-05 2024-03-19T11:31:04.732 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.732 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.732 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:04.732 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976694+0000", 2024-03-19T11:31:04.732 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6488000000000001e-05 2024-03-19T11:31:04.732 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.732 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.732 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:04.733 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976705+0000", 2024-03-19T11:31:04.733 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1044e-05 2024-03-19T11:31:04.733 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.733 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.733 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.733 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976795+0000", 2024-03-19T11:31:04.733 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0283000000000003e-05 2024-03-19T11:31:04.733 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.733 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.733 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:04.733 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976811+0000", 2024-03-19T11:31:04.734 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5262000000000001e-05 2024-03-19T11:31:04.734 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.734 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.734 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.734 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.927337+0000", 2024-03-19T11:31:04.734 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95052597100000003 2024-03-19T11:31:04.734 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.734 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.734 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:04.734 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.927495+0000", 2024-03-19T11:31:04.735 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015788600000000001 2024-03-19T11:31:04.735 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.735 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.735 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:04.735 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.933499+0000", 2024-03-19T11:31:04.735 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0060047640000000001 2024-03-19T11:31:04.735 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.735 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:04.735 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.735 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.735 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.736 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1434 4.16 4:6be89c06:::smithi03829856-24:c [read 0~2386561,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:31:04.736 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.976739+0000", 2024-03-19T11:31:04.736 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 38.757281683000002, 2024-03-19T11:31:04.736 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95781317399999999, 2024-03-19T11:31:04.736 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:04.736 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:04.736 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:04.736 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:04.736 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:04.736 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1434 2024-03-19T11:31:04.736 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.736 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:04.736 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.736 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:04.736 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976739+0000", 2024-03-19T11:31:04.737 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.737 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.737 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.737 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:04.737 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976739+0000", 2024-03-19T11:31:04.737 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.737 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.737 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.737 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:04.737 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976740+0000", 2024-03-19T11:31:04.737 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8800000000000001e-07 2024-03-19T11:31:04.737 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.737 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.737 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:04.737 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976748+0000", 2024-03-19T11:31:04.738 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8979999999999996e-06 2024-03-19T11:31:04.738 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.738 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.738 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:04.738 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976757+0000", 2024-03-19T11:31:04.738 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9439999999999994e-06 2024-03-19T11:31:04.738 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.738 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.738 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:04.738 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976763+0000", 2024-03-19T11:31:04.738 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5030000000000002e-06 2024-03-19T11:31:04.738 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.738 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.738 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.739 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976849+0000", 2024-03-19T11:31:04.739 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5803999999999994e-05 2024-03-19T11:31:04.739 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.739 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.739 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:04.739 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976860+0000", 2024-03-19T11:31:04.739 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0869e-05 2024-03-19T11:31:04.739 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.739 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.739 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.739 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.933562+0000", 2024-03-19T11:31:04.739 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95670232300000002 2024-03-19T11:31:04.739 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.739 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.740 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:04.740 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.933693+0000", 2024-03-19T11:31:04.740 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013110699999999999 2024-03-19T11:31:04.740 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.740 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.740 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:04.740 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.934552+0000", 2024-03-19T11:31:04.740 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00085883800000000001 2024-03-19T11:31:04.740 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.740 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:04.740 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.740 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.740 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.740 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1435 4.16 4:6be89c06:::smithi03829856-24:c [sparse-read 0~1100427] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:31:04.740 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.976775+0000", 2024-03-19T11:31:04.741 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 38.757246012000003, 2024-03-19T11:31:04.741 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95962779499999995, 2024-03-19T11:31:04.741 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:04.741 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:04.741 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:04.741 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:04.741 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:04.741 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1435 2024-03-19T11:31:04.741 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.741 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:04.741 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.741 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:04.741 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976775+0000", 2024-03-19T11:31:04.741 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.741 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.742 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.742 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:04.742 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976775+0000", 2024-03-19T11:31:04.742 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.742 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.742 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.742 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:04.742 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976775+0000", 2024-03-19T11:31:04.742 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9800000000000003e-07 2024-03-19T11:31:04.742 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.742 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.742 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:04.742 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976779+0000", 2024-03-19T11:31:04.742 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3830000000000001e-06 2024-03-19T11:31:04.742 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.743 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.743 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:04.743 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976785+0000", 2024-03-19T11:31:04.743 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5649999999999998e-06 2024-03-19T11:31:04.743 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.743 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.743 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:04.743 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976790+0000", 2024-03-19T11:31:04.743 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1379999999999998e-06 2024-03-19T11:31:04.743 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.743 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.743 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.743 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976889+0000", 2024-03-19T11:31:04.743 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8331000000000004e-05 2024-03-19T11:31:04.743 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.744 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.744 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:04.744 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976894+0000", 2024-03-19T11:31:04.744 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8270000000000003e-06 2024-03-19T11:31:04.744 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.744 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.744 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.744 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.935982+0000", 2024-03-19T11:31:04.744 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95908714100000003 2024-03-19T11:31:04.744 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.744 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.744 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:04.744 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.936093+0000", 2024-03-19T11:31:04.744 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000111557 2024-03-19T11:31:04.744 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.744 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.745 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:04.745 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.936402+0000", 2024-03-19T11:31:04.745 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030925499999999998 2024-03-19T11:31:04.745 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.745 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:04.745 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.745 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.745 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.745 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1442 4.6 4:626acd4d:::smithi03829856-2:head [cache-evict] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:31:04.745 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.978864+0000", 2024-03-19T11:31:04.745 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 38.755156348, 2024-03-19T11:31:04.745 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95708845099999995, 2024-03-19T11:31:04.745 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:04.745 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:04.745 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:04.746 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:04.746 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:04.746 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1442 2024-03-19T11:31:04.746 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.746 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:04.746 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.746 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:04.746 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978864+0000", 2024-03-19T11:31:04.746 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.746 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.746 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.746 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:04.746 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978864+0000", 2024-03-19T11:31:04.746 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.746 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.747 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.747 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:04.747 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978866+0000", 2024-03-19T11:31:04.747 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0729999999999999e-06 2024-03-19T11:31:04.747 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.747 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.747 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:04.747 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978877+0000", 2024-03-19T11:31:04.747 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0390999999999999e-05 2024-03-19T11:31:04.747 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.747 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.747 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:04.747 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978902+0000", 2024-03-19T11:31:04.747 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5635000000000001e-05 2024-03-19T11:31:04.747 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.747 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.748 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:04.748 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978915+0000", 2024-03-19T11:31:04.748 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2617e-05 2024-03-19T11:31:04.748 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.748 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.748 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.748 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.979005+0000", 2024-03-19T11:31:04.748 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0552999999999999e-05 2024-03-19T11:31:04.748 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.748 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.748 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:04.748 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.979021+0000", 2024-03-19T11:31:04.748 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5102999999999999e-05 2024-03-19T11:31:04.748 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.748 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.749 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.749 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.926879+0000", 2024-03-19T11:31:04.749 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.94785817900000002 2024-03-19T11:31:04.749 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.749 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.749 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:04.749 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.927277+0000", 2024-03-19T11:31:04.749 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00039819600000000002 2024-03-19T11:31:04.749 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.749 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.749 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.749 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.935752+0000", 2024-03-19T11:31:04.749 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0084753420000000003 2024-03-19T11:31:04.749 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.749 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.749 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:04.750 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.935874+0000", 2024-03-19T11:31:04.750 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000121311 2024-03-19T11:31:04.750 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.750 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.750 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:04.750 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.935953+0000", 2024-03-19T11:31:04.750 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9050999999999999e-05 2024-03-19T11:31:04.750 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.750 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:04.750 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.750 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.750 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.750 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.d 49/48 [PushOp(4:b65e6658:::smithi03829856-9:head, version: 28'17, data_included: [534456~718345,1951503~182953], data_size: 901298, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:b65e6658:::smithi03829856-9:head@28'17, size: 2134456, copy_subset: [(0, 2134456)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2134456, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:b4ef97e0:::smithi03829856-3:b, version: 46'34, data_included: [433614~1004682,1600611~43894], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:b4ef97e0:::smithi03829856-3:b@46'34, size: 3042034, copy_subset: [(0, 3042034)], clone_subset: {}, snapset: b=[]:{b: [b, a, 6, 5]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1644505, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:b4ef97e0:::smithi03829856-3:head, version: 47'43, data_included: [433614~1004682,1600611~43894], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:b4ef97e0:::smithi03829856-3:head@47'43, size: 3697507, copy_subset: [(0, 3697507)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1644505, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:04.750 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:26.943544+0000", 2024-03-19T11:31:04.751 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 37.790476857999998, 2024-03-19T11:31:04.751 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015601487000000001, 2024-03-19T11:31:04.751 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:04.751 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:04.751 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:04.751 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.751 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:04.751 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.943544+0000", 2024-03-19T11:31:04.751 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.751 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.751 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.751 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:04.751 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.943544+0000", 2024-03-19T11:31:04.751 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.751 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.752 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.752 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:04.752 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.943546+0000", 2024-03-19T11:31:04.752 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.013e-06 2024-03-19T11:31:04.752 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.752 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.752 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:04.752 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.946208+0000", 2024-03-19T11:31:04.752 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0026622820000000002 2024-03-19T11:31:04.752 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.752 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.752 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:04.752 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.946240+0000", 2024-03-19T11:31:04.752 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1568e-05 2024-03-19T11:31:04.752 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.753 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.753 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:04.753 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.946282+0000", 2024-03-19T11:31:04.753 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2599999999999999e-05 2024-03-19T11:31:04.753 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.753 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.753 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.753 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.946442+0000", 2024-03-19T11:31:04.753 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015960400000000001 2024-03-19T11:31:04.753 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.753 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.753 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:04.753 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.946458+0000", 2024-03-19T11:31:04.753 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6059999999999999e-05 2024-03-19T11:31:04.753 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.753 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.754 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:04.754 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.959145+0000", 2024-03-19T11:31:04.754 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01268736 2024-03-19T11:31:04.754 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.754 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:04.754 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.754 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.754 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.754 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2003 4.16 4:6868af91:::smithi03829856-10:head [tier-promote] snapc 12=[12,11,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:31:04.754 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:45.804046+0000", 2024-03-19T11:31:04.754 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.929974434999998, 2024-03-19T11:31:04.754 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018976396, 2024-03-19T11:31:04.754 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:04.754 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:04.755 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:04.755 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:04.755 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:04.755 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2003 2024-03-19T11:31:04.755 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.755 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:04.755 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.755 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:04.755 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804046+0000", 2024-03-19T11:31:04.755 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.755 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.755 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.755 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:04.755 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804046+0000", 2024-03-19T11:31:04.755 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.756 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.756 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.756 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:04.756 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804052+0000", 2024-03-19T11:31:04.756 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7599999999999999e-06 2024-03-19T11:31:04.756 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.756 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.756 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:04.756 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804090+0000", 2024-03-19T11:31:04.756 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8018000000000001e-05 2024-03-19T11:31:04.756 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.756 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.756 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:04.756 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804256+0000", 2024-03-19T11:31:04.756 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016593599999999999 2024-03-19T11:31:04.757 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.757 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.757 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:04.757 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804300+0000", 2024-03-19T11:31:04.757 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3648e-05 2024-03-19T11:31:04.757 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.757 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.757 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.757 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804538+0000", 2024-03-19T11:31:04.757 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000238347 2024-03-19T11:31:04.757 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.757 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.758 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:04.758 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804716+0000", 2024-03-19T11:31:04.758 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000177968 2024-03-19T11:31:04.758 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.758 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.758 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:04.758 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.823023+0000", 2024-03-19T11:31:04.758 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018306718999999999 2024-03-19T11:31:04.758 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.758 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:04.758 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.758 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.759 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.759 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1d58db11:::smithi03829856-18:6, version: 55'80, data_included: [759683~639358,1904434~409218], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:6@55'80, size: 3159683, copy_subset: [(0, 3159683)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2313652, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1d58db11:::smithi03829856-18:b, version: 58'84, data_included: [443718~955323,1564138~93253], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:b@58'84, size: 3445402, copy_subset: [(0, 3445402)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1657391, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:10, version: 66'95, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:10@66'95, size: 3764735, copy_subset: [(0, 3764735)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:12, version: 66'97, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:12@66'97, size: 2504243, copy_subset: [(0, 2504243)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:c, version: 70'102, data_included: [416807~576167,1542129~472409], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:c@70'102, size: 2857157, copy_subset: [(0, 2857157)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2014538, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:04.759 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.275179+0000", 2024-03-19T11:31:04.759 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 10.458841445000001, 2024-03-19T11:31:04.759 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026559353000000001, 2024-03-19T11:31:04.759 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:04.759 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:04.759 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:04.759 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.759 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:04.759 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.275179+0000", 2024-03-19T11:31:04.759 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.759 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.759 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.760 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:04.760 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.275179+0000", 2024-03-19T11:31:04.760 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.760 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.760 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.760 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:04.760 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.275196+0000", 2024-03-19T11:31:04.760 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6946999999999999e-05 2024-03-19T11:31:04.760 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.760 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.761 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:04.761 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281456+0000", 2024-03-19T11:31:04.761 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0062600529999999998 2024-03-19T11:31:04.761 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.761 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.761 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:04.761 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281524+0000", 2024-03-19T11:31:04.761 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7971000000000004e-05 2024-03-19T11:31:04.761 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.761 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.761 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:04.761 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281591+0000", 2024-03-19T11:31:04.761 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6704000000000004e-05 2024-03-19T11:31:04.761 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.761 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.762 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.762 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281865+0000", 2024-03-19T11:31:04.762 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000274097 2024-03-19T11:31:04.762 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.762 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.762 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:04.762 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281901+0000", 2024-03-19T11:31:04.762 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5777e-05 2024-03-19T11:31:04.762 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.762 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.763 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:04.763 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.301738+0000", 2024-03-19T11:31:04.763 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019837804000000001 2024-03-19T11:31:04.763 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.763 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:04.763 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.763 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.763 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.763 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.18 e75,74 4:1d58db11:::smithi03829856-18:2,48'76)", 2024-03-19T11:31:04.763 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.281661+0000", 2024-03-19T11:31:04.763 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 10.452359914000001, 2024-03-19T11:31:04.763 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020384337999999998, 2024-03-19T11:31:04.763 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:04.763 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:04.763 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:04.764 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.764 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:04.764 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281661+0000", 2024-03-19T11:31:04.764 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.764 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.764 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.764 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:04.764 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281661+0000", 2024-03-19T11:31:04.764 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.764 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.765 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.765 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:04.765 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281661+0000", 2024-03-19T11:31:04.765 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9100000000000003e-07 2024-03-19T11:31:04.765 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.765 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.765 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:04.765 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281668+0000", 2024-03-19T11:31:04.765 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8199999999999999e-06 2024-03-19T11:31:04.765 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.765 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.765 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:04.765 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281679+0000", 2024-03-19T11:31:04.765 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1107999999999999e-05 2024-03-19T11:31:04.765 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.766 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.766 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:04.766 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281691+0000", 2024-03-19T11:31:04.766 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1399999999999999e-05 2024-03-19T11:31:04.766 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.766 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.766 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.766 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.301804+0000", 2024-03-19T11:31:04.766 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020113669000000001 2024-03-19T11:31:04.766 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.766 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.766 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:04.766 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.301818+0000", 2024-03-19T11:31:04.766 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3718e-05 2024-03-19T11:31:04.767 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.767 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.767 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:04.767 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302045+0000", 2024-03-19T11:31:04.767 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000226832 2024-03-19T11:31:04.767 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.767 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:04.767 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.767 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.767 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.767 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1dabc0f1:::smithi03829856-1:head, version: 70'103, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:head@70'103, size: 2621892, copy_subset: [(0, 2621892)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:04.767 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.281783+0000", 2024-03-19T11:31:04.767 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 10.452237638, 2024-03-19T11:31:04.767 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026187221, 2024-03-19T11:31:04.767 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:04.768 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:04.768 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:04.768 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.768 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:04.768 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281783+0000", 2024-03-19T11:31:04.768 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.768 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.768 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.768 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:04.768 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281783+0000", 2024-03-19T11:31:04.769 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.769 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.769 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.769 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:04.769 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281783+0000", 2024-03-19T11:31:04.769 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1699999999999999e-07 2024-03-19T11:31:04.769 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.769 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.769 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:04.769 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.282962+0000", 2024-03-19T11:31:04.769 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001178601 2024-03-19T11:31:04.769 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.769 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.769 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:04.769 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.282987+0000", 2024-03-19T11:31:04.770 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4899e-05 2024-03-19T11:31:04.770 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.770 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.770 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:04.770 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283006+0000", 2024-03-19T11:31:04.770 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9494999999999999e-05 2024-03-19T11:31:04.770 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.770 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.770 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.770 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.305757+0000", 2024-03-19T11:31:04.770 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.022750570000000001 2024-03-19T11:31:04.770 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.770 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.770 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:04.771 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.305770+0000", 2024-03-19T11:31:04.771 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2791e-05 2024-03-19T11:31:04.771 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.771 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.771 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:04.771 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.307970+0000", 2024-03-19T11:31:04.771 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022005480000000001 2024-03-19T11:31:04.771 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.771 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:04.771 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.771 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.771 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.771 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.18 e75,74 4:1d58db11:::smithi03829856-18:c,70'104)", 2024-03-19T11:31:04.771 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.283036+0000", 2024-03-19T11:31:04.771 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 10.450984538, 2024-03-19T11:31:04.772 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019146032, 2024-03-19T11:31:04.772 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:04.772 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:04.772 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:04.772 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.772 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:04.772 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283036+0000", 2024-03-19T11:31:04.772 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.772 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.772 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.772 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:04.772 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283036+0000", 2024-03-19T11:31:04.772 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.772 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.772 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.773 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:04.773 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283037+0000", 2024-03-19T11:31:04.773 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4899999999999995e-07 2024-03-19T11:31:04.773 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.773 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.773 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:04.773 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283042+0000", 2024-03-19T11:31:04.773 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0930000000000002e-06 2024-03-19T11:31:04.773 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.773 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.773 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:04.773 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283053+0000", 2024-03-19T11:31:04.773 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1083999999999999e-05 2024-03-19T11:31:04.773 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.773 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.774 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:04.774 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283059+0000", 2024-03-19T11:31:04.774 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6329999999999999e-06 2024-03-19T11:31:04.774 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.774 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.774 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.774 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302090+0000", 2024-03-19T11:31:04.774 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01903038 2024-03-19T11:31:04.774 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.774 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.774 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:04.774 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302097+0000", 2024-03-19T11:31:04.774 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.221e-06 2024-03-19T11:31:04.774 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.775 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.775 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:04.775 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302182+0000", 2024-03-19T11:31:04.775 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5072000000000001e-05 2024-03-19T11:31:04.775 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.775 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:04.775 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.775 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.775 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.775 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(osd.5.0:143 4.18 e75/74 4:1d58db11:::smithi03829856-18:head v 75'106, mlcod=0'0)", 2024-03-19T11:31:04.775 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.283083+0000", 2024-03-19T11:31:04.775 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 10.450937210999999, 2024-03-19T11:31:04.775 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025309439, 2024-03-19T11:31:04.775 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:04.775 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:04.776 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:04.776 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.776 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:04.776 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283083+0000", 2024-03-19T11:31:04.776 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.776 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.776 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.776 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:04.776 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283083+0000", 2024-03-19T11:31:04.776 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.776 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.776 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.776 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:04.776 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283084+0000", 2024-03-19T11:31:04.776 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7800000000000002e-07 2024-03-19T11:31:04.777 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.777 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.777 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:04.777 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.284082+0000", 2024-03-19T11:31:04.777 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0009977880000000001 2024-03-19T11:31:04.777 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.777 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.777 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:04.777 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.284095+0000", 2024-03-19T11:31:04.777 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3256e-05 2024-03-19T11:31:04.777 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.777 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.777 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:04.777 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.284102+0000", 2024-03-19T11:31:04.777 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.605e-06 2024-03-19T11:31:04.777 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.778 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.778 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.778 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302285+0000", 2024-03-19T11:31:04.778 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018182423 2024-03-19T11:31:04.778 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.778 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.778 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:04.778 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302341+0000", 2024-03-19T11:31:04.778 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.626e-05 2024-03-19T11:31:04.778 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.778 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.778 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:04.778 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.308338+0000", 2024-03-19T11:31:04.778 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0059968030000000002 2024-03-19T11:31:04.779 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.779 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.779 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:04.779 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.308393+0000", 2024-03-19T11:31:04.779 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4826e-05 2024-03-19T11:31:04.779 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.779 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:04.779 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.779 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.779 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.779 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "pg_update_log_missing(4.18 epoch 75/74 rep_tid 144 entries 75'107 (0'0) error 4:1d58db11:::smithi03829856-18:head by client.4415.0:2283 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-19T11:31:04.779 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.288252+0000", 2024-03-19T11:31:04.779 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 10.445768661000001, 2024-03-19T11:31:04.779 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020167377, 2024-03-19T11:31:04.780 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:04.780 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-03-19T11:31:04.780 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:04.780 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.780 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:04.780 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288252+0000", 2024-03-19T11:31:04.780 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.780 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.780 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.780 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:04.780 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288252+0000", 2024-03-19T11:31:04.780 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.780 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.780 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.780 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:04.780 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288253+0000", 2024-03-19T11:31:04.781 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2109999999999999e-06 2024-03-19T11:31:04.781 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.781 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.781 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:04.781 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288268+0000", 2024-03-19T11:31:04.781 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5003e-05 2024-03-19T11:31:04.781 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.781 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.781 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:04.781 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288301+0000", 2024-03-19T11:31:04.781 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2539999999999997e-05 2024-03-19T11:31:04.781 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.781 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.781 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:04.781 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288319+0000", 2024-03-19T11:31:04.782 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8232000000000001e-05 2024-03-19T11:31:04.782 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.782 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.782 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.782 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.305490+0000", 2024-03-19T11:31:04.782 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017170857000000001 2024-03-19T11:31:04.782 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.782 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.782 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:04.782 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.308419+0000", 2024-03-19T11:31:04.782 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029295340000000001 2024-03-19T11:31:04.782 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.782 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:04.782 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.782 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.783 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.783 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1d58db11:::smithi03829856-18:6, version: 55'80, data_included: [2313652~310127,3128657~31026], data_size: 341153, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:6@55'80, size: 3159683, copy_subset: [(0, 3159683)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3159683, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2313652, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:1d58db11:::smithi03829856-18:b, version: 58'84, data_included: [1657391~966388,2633066~82188], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:b@58'84, size: 3445402, copy_subset: [(0, 3445402)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2715254, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1657391, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:10, version: 66'95, data_included: [1829043~668012,3134120~380564], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:10@66'95, size: 3764735, copy_subset: [(0, 3764735)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3514684, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:12, version: 66'97, data_included: [1829043~507467], data_size: 507467, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:12@66'97, size: 2504243, copy_subset: [(0, 2504243)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2504243, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:c, version: 70'102, data_included: [2014538~156250], data_size: 156250, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:c@70'102, size: 2857157, copy_subset: [(0, 2857157)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2857157, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2014538, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:04.783 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.312362+0000", 2024-03-19T11:31:04.783 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 10.421658612, 2024-03-19T11:31:04.783 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017968951, 2024-03-19T11:31:04.783 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:04.783 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:04.783 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:04.783 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.783 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:04.783 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.312362+0000", 2024-03-19T11:31:04.783 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.783 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.783 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.783 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:04.784 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.312362+0000", 2024-03-19T11:31:04.784 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.784 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.784 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.784 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:04.784 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.312363+0000", 2024-03-19T11:31:04.784 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.528e-06 2024-03-19T11:31:04.784 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.784 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.784 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:04.784 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315748+0000", 2024-03-19T11:31:04.784 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033842830000000001 2024-03-19T11:31:04.784 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.784 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.785 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:04.785 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315795+0000", 2024-03-19T11:31:04.785 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7382999999999997e-05 2024-03-19T11:31:04.785 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.785 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.785 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:04.785 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315845+0000", 2024-03-19T11:31:04.785 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0319999999999999e-05 2024-03-19T11:31:04.785 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.785 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.785 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.785 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.316123+0000", 2024-03-19T11:31:04.785 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00027747200000000002 2024-03-19T11:31:04.785 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.785 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.785 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:04.786 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.316170+0000", 2024-03-19T11:31:04.786 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7466999999999998e-05 2024-03-19T11:31:04.786 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.786 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.786 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:04.786 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.330331+0000", 2024-03-19T11:31:04.786 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014160498000000001 2024-03-19T11:31:04.786 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.786 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:04.786 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.786 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.786 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.786 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1dabc0f1:::smithi03829856-1:head, version: 70'103, data_included: [1829043~555153], data_size: 555153, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:head@70'103, size: 2621892, copy_subset: [(0, 2621892)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2621892, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:04.786 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.315898+0000", 2024-03-19T11:31:04.787 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 10.418122626000001, 2024-03-19T11:31:04.787 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016880424000000002, 2024-03-19T11:31:04.787 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:04.787 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:04.787 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:04.787 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.787 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:04.787 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315898+0000", 2024-03-19T11:31:04.787 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.787 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.787 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.787 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:04.787 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315898+0000", 2024-03-19T11:31:04.787 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.788 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.788 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.788 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:04.788 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315899+0000", 2024-03-19T11:31:04.788 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.003e-06 2024-03-19T11:31:04.788 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.788 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.788 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:04.788 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.316486+0000", 2024-03-19T11:31:04.788 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00058673099999999995 2024-03-19T11:31:04.788 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.788 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.788 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:04.788 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.316519+0000", 2024-03-19T11:31:04.788 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3124000000000003e-05 2024-03-19T11:31:04.788 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.789 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.789 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:04.789 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.316534+0000", 2024-03-19T11:31:04.789 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.485e-05 2024-03-19T11:31:04.789 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.789 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.789 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.789 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.330415+0000", 2024-03-19T11:31:04.789 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013880905000000001 2024-03-19T11:31:04.789 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.789 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.789 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:04.789 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.330439+0000", 2024-03-19T11:31:04.789 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4630999999999998e-05 2024-03-19T11:31:04.789 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.789 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.790 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:04.790 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.332778+0000", 2024-03-19T11:31:04.790 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023391800000000002 2024-03-19T11:31:04.790 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.790 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:04.790 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.790 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.790 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.790 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2387 4.2 4:43b32242:::smithi03829856-22:head [read 596992~684544] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-03-19T11:31:04.790 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:59.319806+0000", 2024-03-19T11:31:04.790 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.4142149589999997, 2024-03-19T11:31:04.790 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.78144344399999999, 2024-03-19T11:31:04.790 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:04.790 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:04.790 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:04.791 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:04.791 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:04.791 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2387 2024-03-19T11:31:04.791 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.791 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:04.791 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.791 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:04.791 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319806+0000", 2024-03-19T11:31:04.791 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.791 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.791 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.791 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:04.791 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319806+0000", 2024-03-19T11:31:04.791 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.792 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.792 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.792 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:04.792 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319807+0000", 2024-03-19T11:31:04.792 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3519999999999999e-06 2024-03-19T11:31:04.792 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.792 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.792 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:04.792 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319812+0000", 2024-03-19T11:31:04.792 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3959999999999997e-06 2024-03-19T11:31:04.792 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.792 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.792 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:04.792 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319827+0000", 2024-03-19T11:31:04.792 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4566000000000001e-05 2024-03-19T11:31:04.792 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.793 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.793 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:04.793 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319835+0000", 2024-03-19T11:31:04.793 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.994e-06 2024-03-19T11:31:04.793 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.793 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.793 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.793 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.320119+0000", 2024-03-19T11:31:04.793 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00028445199999999998 2024-03-19T11:31:04.793 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.793 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.793 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:04.793 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.320129+0000", 2024-03-19T11:31:04.793 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9580000000000002e-06 2024-03-19T11:31:04.793 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.793 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.793 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.794 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.069842+0000", 2024-03-19T11:31:04.794 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.74971218399999995 2024-03-19T11:31:04.794 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.794 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.794 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:31:04.794 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.070010+0000", 2024-03-19T11:31:04.794 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016846600000000001 2024-03-19T11:31:04.794 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.794 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.794 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.794 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.090911+0000", 2024-03-19T11:31:04.794 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020901494999999999 2024-03-19T11:31:04.794 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.794 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.795 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:31:04.795 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.091249+0000", 2024-03-19T11:31:04.795 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000337602 2024-03-19T11:31:04.795 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.795 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.795 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.795 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.098478+0000", 2024-03-19T11:31:04.795 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0072287549999999999 2024-03-19T11:31:04.795 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.795 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.795 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:04.795 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.098617+0000", 2024-03-19T11:31:04.795 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013871599999999999 2024-03-19T11:31:04.795 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.796 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.796 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:04.796 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.101249+0000", 2024-03-19T11:31:04.796 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002632508 2024-03-19T11:31:04.796 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.796 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:04.796 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.796 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.796 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.796 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 81/80 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 15=[]:{7: [6], b: [b], 10: [d], 12: [12, 11], 15: [15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:7, version: 55'82, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@55'82, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 16=[]:{7: [6], b: [b], 10: [d], 16: [16, 15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 15=[]:{7: [6], b: [b], 10: [d], 12: [12, 11], 15: [15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 58'86, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@58'86, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 16=[]:{7: [6], b: [b], 10: [d], 16: [16, 15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [504210~968657,1572695~79919], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 16=[]:{7: [6], b: [b], 10: [d], 16: [16, 15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:04.796 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:00.089924+0000", 2024-03-19T11:31:04.796 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.6440967300000002, 2024-03-19T11:31:04.796 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024686583000000002, 2024-03-19T11:31:04.796 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:04.796 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:04.797 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:04.797 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.797 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:04.797 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.089924+0000", 2024-03-19T11:31:04.797 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.797 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.797 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.797 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:04.797 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.089924+0000", 2024-03-19T11:31:04.797 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.797 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.797 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.797 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:04.797 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.089926+0000", 2024-03-19T11:31:04.797 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9589999999999998e-06 2024-03-19T11:31:04.798 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.798 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.798 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:04.798 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093311+0000", 2024-03-19T11:31:04.798 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033854409999999999 2024-03-19T11:31:04.798 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.798 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.798 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:04.798 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093365+0000", 2024-03-19T11:31:04.798 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3795000000000002e-05 2024-03-19T11:31:04.798 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.798 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.798 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:04.798 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093438+0000", 2024-03-19T11:31:04.799 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2591999999999999e-05 2024-03-19T11:31:04.799 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.799 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.799 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.799 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093690+0000", 2024-03-19T11:31:04.799 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025264999999999999 2024-03-19T11:31:04.799 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.799 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.799 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:04.799 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093731+0000", 2024-03-19T11:31:04.799 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0881999999999999e-05 2024-03-19T11:31:04.799 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.799 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.800 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:04.800 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.114610+0000", 2024-03-19T11:31:04.800 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020879264000000002 2024-03-19T11:31:04.800 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.800 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:04.800 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.800 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.800 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.800 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.21 e81,80 4:8401c1f0:::smithi03829856-43:8,46'66,4:866f3dcc:::smithi03829856-31:2,47'73)", 2024-03-19T11:31:04.800 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:00.093514+0000", 2024-03-19T11:31:04.800 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.6405061300000003, 2024-03-19T11:31:04.800 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027375713999999999, 2024-03-19T11:31:04.800 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:04.800 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:04.801 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:04.801 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.801 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:04.801 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093514+0000", 2024-03-19T11:31:04.801 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.801 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.801 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.801 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:04.801 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093514+0000", 2024-03-19T11:31:04.801 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.801 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.801 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.801 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:04.801 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093516+0000", 2024-03-19T11:31:04.801 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1179999999999999e-06 2024-03-19T11:31:04.802 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.802 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.802 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:04.802 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093523+0000", 2024-03-19T11:31:04.802 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7500000000000003e-06 2024-03-19T11:31:04.802 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.802 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.802 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:04.802 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093534+0000", 2024-03-19T11:31:04.802 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0831e-05 2024-03-19T11:31:04.802 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.802 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.802 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:04.802 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093542+0000", 2024-03-19T11:31:04.802 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0139999999999998e-06 2024-03-19T11:31:04.803 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.803 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.803 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.803 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.120682+0000", 2024-03-19T11:31:04.803 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027139838999999999 2024-03-19T11:31:04.803 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.803 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.803 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:04.803 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.120693+0000", 2024-03-19T11:31:04.803 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.136e-05 2024-03-19T11:31:04.803 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.803 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.803 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:04.803 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.120890+0000", 2024-03-19T11:31:04.803 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019680199999999999 2024-03-19T11:31:04.804 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.804 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:04.804 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.804 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.804 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.804 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 81/80 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 15=[]:{7: [6], b: [b], 10: [d], 12: [12, 11], 15: [15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:7, version: 55'82, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@55'82, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 16=[]:{7: [6], b: [b], 10: [d], 16: [16, 15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 15=[]:{7: [6], b: [b], 10: [d], 12: [12, 11], 15: [15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 58'86, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@58'86, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 16=[]:{7: [6], b: [b], 10: [d], 16: [16, 15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [1652614~973634,2888511~52681,3182305~22261], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 16=[]:{7: [6], b: [b], 10: [d], 16: [16, 15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3204566, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:04.804 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:00.126855+0000", 2024-03-19T11:31:04.804 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.6071653929999998, 2024-03-19T11:31:04.804 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016438076999999999, 2024-03-19T11:31:04.804 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:04.804 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:04.804 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:04.804 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.804 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:04.805 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.126855+0000", 2024-03-19T11:31:04.805 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.805 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.805 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.805 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:04.805 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.126855+0000", 2024-03-19T11:31:04.805 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.805 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.805 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.805 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:04.805 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.126857+0000", 2024-03-19T11:31:04.805 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.739e-06 2024-03-19T11:31:04.805 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.805 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.805 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:04.805 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.129236+0000", 2024-03-19T11:31:04.806 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023792449999999999 2024-03-19T11:31:04.806 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.806 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.806 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:04.806 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.129298+0000", 2024-03-19T11:31:04.806 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2329000000000006e-05 2024-03-19T11:31:04.806 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.806 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.806 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:04.806 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.129365+0000", 2024-03-19T11:31:04.806 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6299999999999999e-05 2024-03-19T11:31:04.806 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.806 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.806 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.807 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.132493+0000", 2024-03-19T11:31:04.807 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031279530000000002 2024-03-19T11:31:04.807 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.807 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.807 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:04.807 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.132578+0000", 2024-03-19T11:31:04.807 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5648000000000004e-05 2024-03-19T11:31:04.807 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.807 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.807 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:04.807 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.143293+0000", 2024-03-19T11:31:04.807 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010714863 2024-03-19T11:31:04.807 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.807 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:04.807 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.808 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.808 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.808 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.29 81/80 [PushOp(4:953e720b:::smithi03829856-45:10, version: 70'58, data_included: [1927574~595528,2570780~420833], data_size: 1016361, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:953e720b:::smithi03829856-45:10@70'58, size: 2991613, copy_subset: [(0, 2991613)], clone_subset: {}, snapset: 10=[]:{b: [b, 6], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2991613, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1927574, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:953e720b:::smithi03829856-45:head, version: 71'60, data_included: [1823397~664039], data_size: 664039, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:953e720b:::smithi03829856-45:head@71'60, size: 2487436, copy_subset: [(0, 2487436)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2487436, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1823397, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:953e720b:::smithi03829856-45:b, version: 77'61, data_included: [1927574~595528,2570780~420833], data_size: 1016361, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:953e720b:::smithi03829856-45:b@77'61, size: 2991613, copy_subset: [(0, 2991613)], clone_subset: {}, snapset: 10=[]:{b: [b, 6], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2991613, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1927574, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:04.808 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:00.151568+0000", 2024-03-19T11:31:04.808 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.582452279, 2024-03-19T11:31:04.808 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015354197999999999, 2024-03-19T11:31:04.808 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:04.808 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:04.808 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:04.808 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.808 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:04.808 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.151568+0000", 2024-03-19T11:31:04.808 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.808 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.808 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.809 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:04.809 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.151568+0000", 2024-03-19T11:31:04.809 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:04.809 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.809 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.809 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:04.809 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.151569+0000", 2024-03-19T11:31:04.809 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.17e-06 2024-03-19T11:31:04.809 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.809 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.809 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:04.809 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.156296+0000", 2024-03-19T11:31:04.809 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0047264619999999999 2024-03-19T11:31:04.809 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.809 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.809 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:04.810 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.156334+0000", 2024-03-19T11:31:04.810 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8152000000000003e-05 2024-03-19T11:31:04.810 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.810 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.810 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:04.810 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.156370+0000", 2024-03-19T11:31:04.810 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6304999999999999e-05 2024-03-19T11:31:04.810 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.810 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.810 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:04.810 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.157994+0000", 2024-03-19T11:31:04.810 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0016235399999999999 2024-03-19T11:31:04.810 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.810 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.811 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:04.811 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.158017+0000", 2024-03-19T11:31:04.811 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3197000000000002e-05 2024-03-19T11:31:04.811 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:04.811 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:04.811 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:04.811 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.166922+0000", 2024-03-19T11:31:04.811 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0089053719999999999 2024-03-19T11:31:04.811 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.811 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:04.811 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.811 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:04.811 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:04.811 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:04.813 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-19T11:31:04.813 INFO:tasks.rados.rados.0.smithi038.stdout:1201: finishing write tid 1 to smithi03829856-6 2024-03-19T11:31:04.813 INFO:tasks.rados.rados.0.smithi038.stdout:1201: finishing write tid 2 to smithi03829856-6 2024-03-19T11:31:04.813 INFO:tasks.rados.rados.0.smithi038.stdout:1201: finishing write tid 3 to smithi03829856-6 2024-03-19T11:31:04.814 INFO:tasks.rados.rados.0.smithi038.stdout:1201: finishing write tid 4 to smithi03829856-6 2024-03-19T11:31:04.814 INFO:tasks.rados.rados.0.smithi038.stdout:1202: finishing set_chunk to oid 35 2024-03-19T11:31:04.814 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 35 set_chunk returned error code -95 offset: 1184256 length: 844800 2024-03-19T11:31:04.814 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 35 v 0 (ObjNum 158 snap 17 seq_num 158) dirty exists 2024-03-19T11:31:04.814 INFO:tasks.rados.rados.0.smithi038.stdout:1201: finishing write tid 5 to smithi03829856-6 2024-03-19T11:31:04.814 INFO:tasks.rados.rados.0.smithi038.stdout:1201: oid 6 updating version 0 to 85 2024-03-19T11:31:04.814 INFO:tasks.rados.rados.0.smithi038.stdout:1201: oid 6 updating version 85 to 87 2024-03-19T11:31:04.814 INFO:tasks.rados.rados.0.smithi038.stdout:1201: oid 6 version 87 is already newer than 86 2024-03-19T11:31:04.814 INFO:tasks.rados.rados.0.smithi038.stdout:1201: oid 6 version 87 is already newer than 84 2024-03-19T11:31:04.814 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 6 v 87 (ObjNum 195 snap 22 seq_num 195) dirty exists 2024-03-19T11:31:04.814 INFO:tasks.rados.rados.0.smithi038.stdout:1201: left oid 6 (ObjNum 195 snap 22 seq_num 195) 2024-03-19T11:31:04.814 INFO:tasks.rados.rados.0.smithi038.stdout:1203: expect (ObjNum 179 snap 19 seq_num 179) 2024-03-19T11:31:04.815 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:04.823+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:04.910 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-19T11:31:04.913 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:04.923+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:04.944 INFO:tasks.rados.rados.0.smithi038.stdout:1204: finishing set_chunk to oid 50 2024-03-19T11:31:04.944 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 50 set_chunk returned error code -95 offset: 386560 length: 1251328 2024-03-19T11:31:04.944 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 50 v 0 (ObjNum 114 snap 11 seq_num 114) dirty exists 2024-03-19T11:31:04.944 INFO:tasks.rados.rados.0.smithi038.stdout:1205: expect (ObjNum 168 snap 17 seq_num 168) 2024-03-19T11:31:04.988 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:04.988 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:31:04.988 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:31:04.988 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:05.001 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-19T11:31:05.015 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:05.023+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:05.043 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:31:05.043 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:31:05.050 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:05.057+0000 7f6712f92640 -1 osd.2 85 handle_osd_map injecting map crc failure 2024-03-19T11:31:05.051 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:05.057+0000 7fa5de8d7640 -1 osd.3 85 handle_osd_map injecting map crc failure 2024-03-19T11:31:05.053 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.7 to 1 (10000) 2024-03-19T11:31:05.068 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:31:05.068 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:31:05.075 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-03-19T11:31:05.076 INFO:tasks.rados.rados.0.smithi038.stdout:1206: finishing write tid 1 to smithi03829856-49 2024-03-19T11:31:05.076 INFO:tasks.rados.rados.0.smithi038.stdout:1206: finishing write tid 2 to smithi03829856-49 2024-03-19T11:31:05.077 INFO:tasks.rados.rados.0.smithi038.stdout:1207: got (0) Success 2024-03-19T11:31:05.077 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 8 v 96 (ObjNum 60 snap 3 seq_num 60) dirty exists 2024-03-19T11:31:05.077 INFO:tasks.rados.rados.0.smithi038.stdout:1206: finishing write tid 3 to smithi03829856-49 2024-03-19T11:31:05.077 INFO:tasks.rados.rados.0.smithi038.stdout:1199: done (7 left) 2024-03-19T11:31:05.077 INFO:tasks.rados.rados.0.smithi038.stdout:1201: done (6 left) 2024-03-19T11:31:05.077 INFO:tasks.rados.rados.0.smithi038.stdout:1202: done (5 left) 2024-03-19T11:31:05.077 INFO:tasks.rados.rados.0.smithi038.stdout:1203: done (4 left) 2024-03-19T11:31:05.078 INFO:tasks.rados.rados.0.smithi038.stdout:1204: done (3 left) 2024-03-19T11:31:05.078 INFO:tasks.rados.rados.0.smithi038.stdout:1205: done (2 left) 2024-03-19T11:31:05.078 INFO:tasks.rados.rados.0.smithi038.stdout:1207: done (1 left) 2024-03-19T11:31:05.078 INFO:tasks.rados.rados.0.smithi038.stdout:1208: write oid 21 current snap is 22 2024-03-19T11:31:05.078 INFO:tasks.rados.rados.0.smithi038.stdout:1208: seq_num 197 ranges {510979=532878,1528482=772396,3060800=250179} 2024-03-19T11:31:05.082 INFO:tasks.rados.rados.0.smithi038.stdout:1208: writing smithi03829856-21 from 510979 to 1043857 tid 1 2024-03-19T11:31:05.083 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-19T11:31:05.083 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T11:31:05.090 INFO:tasks.rados.rados.0.smithi038.stdout:1208: writing smithi03829856-21 from 1528482 to 2300878 tid 2 2024-03-19T11:31:05.092 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:31:05.093 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:31:05.096 INFO:tasks.rados.rados.0.smithi038.stdout:1208: writing smithi03829856-21 from 3060800 to 3310979 tid 3 2024-03-19T11:31:05.096 INFO:tasks.rados.rados.0.smithi038.stdout:1209: read oid 11 snap -1 2024-03-19T11:31:05.096 INFO:tasks.rados.rados.0.smithi038.stdout:1209: expect (ObjNum 187 snap 21 seq_num 187) 2024-03-19T11:31:05.096 INFO:tasks.rados.rados.0.smithi038.stdout:1210: set_chunk oid 36 target oid 2024-03-19T11:31:05.116 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:05.125+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:05.117 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:31:05.117 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:31:05.122 INFO:tasks.rados.rados.0.smithi038.stdout:1210: set_chunk oid 36 offset: 167936 length: 1373184 target oid 1b8ee3a2a12c8c49e2f7518921c8a817223b77fea680b94e0449e3c29ffc6a36 offset: 0 snap -1 2024-03-19T11:31:05.122 INFO:tasks.rados.rados.0.smithi038.stdout:1211: read oid 16 snap -1 2024-03-19T11:31:05.122 INFO:tasks.rados.rados.0.smithi038.stdout:1211: expect (ObjNum 65 snap 3 seq_num 65) 2024-03-19T11:31:05.122 INFO:tasks.rados.rados.0.smithi038.stdout:1212: read oid 32 snap -1 2024-03-19T11:31:05.122 INFO:tasks.rados.rados.0.smithi038.stdout:1212: expect (ObjNum 31 snap 0 seq_num 31) 2024-03-19T11:31:05.122 INFO:tasks.rados.rados.0.smithi038.stdout:1213: read oid 37 snap -1 2024-03-19T11:31:05.122 INFO:tasks.rados.rados.0.smithi038.stdout:1213: expect deleted 2024-03-19T11:31:05.122 INFO:tasks.rados.rados.0.smithi038.stdout:1214: rollback oid 24 current snap is 22 2024-03-19T11:31:05.122 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 24 to 21 2024-03-19T11:31:05.122 INFO:tasks.rados.rados.0.smithi038.stdout:1215: write oid 43 current snap is 22 2024-03-19T11:31:05.122 INFO:tasks.rados.rados.0.smithi038.stdout:1215: seq_num 198 ranges {527786=724480,1703317=516054,2804576=435035} 2024-03-19T11:31:05.130 INFO:tasks.rados.rados.0.smithi038.stdout:1215: writing smithi03829856-43 from 527786 to 1252266 tid 1 2024-03-19T11:31:05.139 INFO:tasks.rados.rados.0.smithi038.stdout:1215: writing smithi03829856-43 from 1703317 to 2219371 tid 2 2024-03-19T11:31:05.146 INFO:tasks.rados.rados.0.smithi038.stdout:1215: writing smithi03829856-43 from 2804576 to 3239611 tid 3 2024-03-19T11:31:05.146 INFO:tasks.rados.rados.0.smithi038.stdout:1206: finishing write tid 4 to smithi03829856-49 2024-03-19T11:31:05.146 INFO:tasks.rados.rados.0.smithi038.stdout:1206: finishing write tid 5 to smithi03829856-49 2024-03-19T11:31:05.146 INFO:tasks.rados.rados.0.smithi038.stdout:1206: oid 49 updating version 0 to 32 2024-03-19T11:31:05.146 INFO:tasks.rados.rados.0.smithi038.stdout:1206: oid 49 version 32 is already newer than 31 2024-03-19T11:31:05.146 INFO:tasks.rados.rados.0.smithi038.stdout:1206: oid 49 updating version 32 to 33 2024-03-19T11:31:05.146 INFO:tasks.rados.rados.0.smithi038.stdout:1206: oid 49 updating version 33 to 34 2024-03-19T11:31:05.146 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 49 v 34 (ObjNum 196 snap 22 seq_num 196) dirty exists 2024-03-19T11:31:05.147 INFO:tasks.rados.rados.0.smithi038.stdout:1206: left oid 49 (ObjNum 196 snap 22 seq_num 196) 2024-03-19T11:31:05.147 INFO:tasks.rados.rados.0.smithi038.stdout:1208: finishing write tid 1 to smithi03829856-21 2024-03-19T11:31:05.147 INFO:tasks.rados.rados.0.smithi038.stdout:1208: finishing write tid 2 to smithi03829856-21 2024-03-19T11:31:05.147 INFO:tasks.rados.rados.0.smithi038.stdout:1208: finishing write tid 3 to smithi03829856-21 2024-03-19T11:31:05.147 INFO:tasks.rados.rados.0.smithi038.stdout:1208: finishing write tid 4 to smithi03829856-21 2024-03-19T11:31:05.147 INFO:tasks.rados.rados.0.smithi038.stdout:1208: finishing write tid 5 to smithi03829856-21 2024-03-19T11:31:05.147 INFO:tasks.rados.rados.0.smithi038.stdout:1208: oid 21 updating version 0 to 155 2024-03-19T11:31:05.147 INFO:tasks.rados.rados.0.smithi038.stdout:1208: oid 21 version 155 is already newer than 153 2024-03-19T11:31:05.147 INFO:tasks.rados.rados.0.smithi038.stdout:1208: oid 21 version 155 is already newer than 154 2024-03-19T11:31:05.147 INFO:tasks.rados.rados.0.smithi038.stdout:1208: oid 21 version 155 is already newer than 152 2024-03-19T11:31:05.147 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 21 v 155 (ObjNum 197 snap 22 seq_num 197) dirty exists 2024-03-19T11:31:05.147 INFO:tasks.rados.rados.0.smithi038.stdout:1208: left oid 21 (ObjNum 197 snap 22 seq_num 197) 2024-03-19T11:31:05.147 INFO:tasks.rados.rados.0.smithi038.stdout:1209: expect (ObjNum 187 snap 21 seq_num 187) 2024-03-19T11:31:05.153 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:31:05.153 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:31:05.178 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:05.178 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:05.185 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:31:05.185 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:31:05.193 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-19T11:31:05.207 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:31:05.207 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:31:05.217 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:05.227+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:05.238 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:31:05.238 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:31:05.317 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:05.325+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:05.326 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:05.326 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:05.333 INFO:tasks.rados.rados.0.smithi038.stdout:1211: expect (ObjNum 65 snap 3 seq_num 65) 2024-03-19T11:31:05.339 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-19T11:31:05.418 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:05.427+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:05.472 INFO:tasks.rados.rados.0.smithi038.stdout:1214: finishing rollback tid 0 to smithi03829856-24 2024-03-19T11:31:05.472 INFO:tasks.rados.rados.0.smithi038.stdout:1214: finishing rollback tid 1 to smithi03829856-24 2024-03-19T11:31:05.472 INFO:tasks.rados.rados.0.smithi038.stdout:1214: finishing rollback tid 2 to smithi03829856-24 2024-03-19T11:31:05.472 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 24 v 112 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-03-19T11:31:05.472 INFO:tasks.rados.rados.0.smithi038.stdout:1212: expect (ObjNum 31 snap 0 seq_num 31) 2024-03-19T11:31:05.484 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:31:05.485 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-03-19T11:31:05.492 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:05.492 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:05.501 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":120,"stamp":"2024-03-19T11:31:05.054717+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":439566058,"num_objects":287,"num_object_clones":91,"num_object_copies":574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":284,"num_whiteouts":6,"num_read":1086,"num_read_kb":736938,"num_write":752,"num_write_kb":241538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":220,"num_bytes_recovered":240180385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":93,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3272,"ondisk_log_size":3272,"up":114,"acting":114,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":124,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":797424,"kb_used_data":550732,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":754177296,"statfs":{"total":773094113280,"available":772277551104,"internally_reserved":0,"allocated":563949568,"data_stored":630418149,"data_compressed":14387442,"data_compressed_allocated":14413824,"data_compressed_original":32731136,"omap_allocated":12751,"internal_metadata":246795825},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13842679,"num_objects":7,"num_object_clones":-2,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":-8,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":69,"num_read_kb":66097,"num_write":44,"num_write_kb":11101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":108743946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.998594"},"pg_stats":[{"pgid":"4.2b","version":"61'60","reported_seq":375,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-19T11:31:02.077571+0000","last_change":"2024-03-19T11:31:02.082219+0000","last_active":"2024-03-19T11:31:02.077571+0000","last_peered":"2024-03-19T11:31:02.077571+0000","last_clean":"2024-03-19T11:31:02.077571+0000","last_became_active":"2024-03-19T11:30:45.263076+0000","last_became_peered":"2024-03-19T11:30:45.263076+0000","last_unstale":"2024-03-19T11:31:02.077571+0000","last_undegraded":"2024-03-19T11:31:02.077571+0000","last_fullsized":"2024-03-19T11:31:02.077571+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":6,"last_scrub":"61'60","last_scrub_stamp":"2024-03-19T11:30:55.143981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:55.143981+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:59.980040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00639216,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.2a","version":"83'94","reported_seq":400,"reported_epoch":84,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:03.052847+0000","last_change":"2024-03-19T11:31:02.085627+0000","last_active":"2024-03-19T11:31:03.052847+0000","last_peered":"2024-03-19T11:31:03.052847+0000","last_clean":"2024-03-19T11:31:03.052847+0000","last_became_active":"2024-03-19T11:30:45.260578+0000","last_became_peered":"2024-03-19T11:30:45.260578+0000","last_unstale":"2024-03-19T11:31:03.052847+0000","last_undegraded":"2024-03-19T11:31:03.052847+0000","last_fullsized":"2024-03-19T11:31:03.052847+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":6,"last_scrub":"74'89","last_scrub_stamp":"2024-03-19T11:30:53.312593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:53.312593+0000","objects_scrubbed":6,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:07.058558+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.015736934000000001,"stat_sum":{"num_bytes":10618356,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":781,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.29","version":"83'63","reported_seq":409,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-19T11:31:03.046373+0000","last_change":"2024-03-19T11:31:02.085998+0000","last_active":"2024-03-19T11:31:03.046373+0000","last_peered":"2024-03-19T11:31:03.046373+0000","last_clean":"2024-03-19T11:31:03.046373+0000","last_became_active":"2024-03-19T11:31:00.070287+0000","last_became_peered":"2024-03-19T11:31:00.070287+0000","last_unstale":"2024-03-19T11:31:03.046373+0000","last_undegraded":"2024-03-19T11:31:03.046373+0000","last_fullsized":"2024-03-19T11:31:03.046373+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":6,"last_scrub":"71'60","last_scrub_stamp":"2024-03-19T11:30:52.748479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:52.748479+0000","objects_scrubbed":3,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:56.409127+0000","scrub_duration":7,"objects_trimmed":1,"snaptrim_duration":0.016516072,"stat_sum":{"num_bytes":4387250,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":377,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":5842489,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.28","version":"85'98","reported_seq":383,"reported_epoch":85,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:04.086552+0000","last_change":"2024-03-19T11:31:04.078290+0000","last_active":"2024-03-19T11:31:04.086552+0000","last_peered":"2024-03-19T11:31:04.086552+0000","last_clean":"2024-03-19T11:31:04.086552+0000","last_became_active":"2024-03-19T11:30:45.257600+0000","last_became_peered":"2024-03-19T11:30:45.257600+0000","last_unstale":"2024-03-19T11:31:04.086552+0000","last_undegraded":"2024-03-19T11:31:04.086552+0000","last_fullsized":"2024-03-19T11:31:04.086552+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":6,"last_scrub":"71'85","last_scrub_stamp":"2024-03-19T11:30:52.343454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:52.343454+0000","objects_scrubbed":7,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:16.280349+0000","scrub_duration":7,"objects_trimmed":1,"snaptrim_duration":0.0090263340000000004,"stat_sum":{"num_bytes":11017340,"num_objects":7,"num_object_clones":5,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":6,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.27","version":"85'54","reported_seq":53,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.074043+0000","last_change":"2024-03-19T11:31:04.074317+0000","last_active":"2024-03-19T11:31:04.074043+0000","last_peered":"2024-03-19T11:31:04.074043+0000","last_clean":"2024-03-19T11:31:04.074043+0000","last_became_active":"2024-03-19T11:31:00.070337+0000","last_became_peered":"2024-03-19T11:31:00.070337+0000","last_unstale":"2024-03-19T11:31:04.074043+0000","last_undegraded":"2024-03-19T11:31:04.074043+0000","last_fullsized":"2024-03-19T11:31:04.074043+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":6,"last_scrub":"71'48","last_scrub_stamp":"2024-03-19T11:30:55.467716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:55.467716+0000","objects_scrubbed":4,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:00.305683+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.018507597000000001,"stat_sum":{"num_bytes":6814202,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":11198,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":7070511,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"},{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.26","version":"61'63","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-19T11:31:02.087385+0000","last_change":"2024-03-19T11:31:02.087385+0000","last_active":"2024-03-19T11:31:02.087385+0000","last_peered":"2024-03-19T11:31:02.087385+0000","last_clean":"2024-03-19T11:31:02.087385+0000","last_became_active":"2024-03-19T11:30:59.074337+0000","last_became_peered":"2024-03-19T11:30:59.074337+0000","last_unstale":"2024-03-19T11:31:02.087385+0000","last_undegraded":"2024-03-19T11:31:02.087385+0000","last_fullsized":"2024-03-19T11:31:02.087385+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":6,"last_scrub":"61'63","last_scrub_stamp":"2024-03-19T11:30:52.406723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:52.406723+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:03.655668+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.010852772,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.25","version":"56'66","reported_seq":51,"reported_epoch":85,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:04.077517+0000","last_change":"2024-03-19T11:31:04.077570+0000","last_active":"2024-03-19T11:31:04.077517+0000","last_peered":"2024-03-19T11:31:04.077517+0000","last_clean":"2024-03-19T11:31:04.077517+0000","last_became_active":"2024-03-19T11:30:53.084473+0000","last_became_peered":"2024-03-19T11:30:53.084473+0000","last_unstale":"2024-03-19T11:31:04.077517+0000","last_undegraded":"2024-03-19T11:31:04.077517+0000","last_fullsized":"2024-03-19T11:31:04.077517+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":6,"last_scrub":"47'64","last_scrub_stamp":"2024-03-19T11:30:28.487136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:28.487136+0000","objects_scrubbed":3,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0087584710000000003,"stat_sum":{"num_bytes":2403368,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":10662,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6","length":"1"},{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.24","version":"85'117","reported_seq":477,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.064686+0000","last_change":"2024-03-19T11:31:04.064961+0000","last_active":"2024-03-19T11:31:04.064686+0000","last_peered":"2024-03-19T11:31:04.064686+0000","last_clean":"2024-03-19T11:31:04.064686+0000","last_became_active":"2024-03-19T11:31:03.063705+0000","last_became_peered":"2024-03-19T11:31:03.063705+0000","last_unstale":"2024-03-19T11:31:04.064686+0000","last_undegraded":"2024-03-19T11:31:04.064686+0000","last_fullsized":"2024-03-19T11:31:04.064686+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":6,"last_scrub":"84'113","last_scrub_stamp":"2024-03-19T11:31:03.304881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:31:03.304881+0000","objects_scrubbed":5,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:26.481361+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.0099064849999999996,"stat_sum":{"num_bytes":10102023,"num_objects":4,"num_object_clones":2,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"},{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.23","version":"85'29","reported_seq":59,"reported_epoch":85,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:04.098447+0000","last_change":"2024-03-19T11:31:04.098447+0000","last_active":"2024-03-19T11:31:04.098447+0000","last_peered":"2024-03-19T11:31:04.098447+0000","last_clean":"2024-03-19T11:31:04.098447+0000","last_became_active":"2024-03-19T11:30:53.076092+0000","last_became_peered":"2024-03-19T11:30:53.076092+0000","last_unstale":"2024-03-19T11:31:04.098447+0000","last_undegraded":"2024-03-19T11:31:04.098447+0000","last_fullsized":"2024-03-19T11:31:04.098447+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":6,"last_scrub":"62'23","last_scrub_stamp":"2024-03-19T11:30:43.163509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:43.163509+0000","objects_scrubbed":2,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:46.525658+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.023454183999999999,"stat_sum":{"num_bytes":3728802,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2833035,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.22","version":"28'6","reported_seq":55,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-19T11:31:03.045458+0000","last_change":"2024-03-19T11:31:02.071949+0000","last_active":"2024-03-19T11:31:03.045458+0000","last_peered":"2024-03-19T11:31:03.045458+0000","last_clean":"2024-03-19T11:31:03.045458+0000","last_became_active":"2024-03-19T11:30:53.068010+0000","last_became_peered":"2024-03-19T11:30:53.068010+0000","last_unstale":"2024-03-19T11:31:03.045458+0000","last_undegraded":"2024-03-19T11:31:03.045458+0000","last_fullsized":"2024-03-19T11:31:03.045458+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":6,"last_scrub":"28'6","last_scrub_stamp":"2024-03-19T11:30:55.724671+0000","last_deep_scrub":"28'6","last_deep_scrub_stamp":"2024-03-19T11:30:55.724671+0000","last_clean_scrub_stamp":"2024-03-19T11:30:55.724671+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:00.824445+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0024412639999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.21","version":"83'125","reported_seq":500,"reported_epoch":85,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:04.100621+0000","last_change":"2024-03-19T11:31:04.100668+0000","last_active":"2024-03-19T11:31:04.100621+0000","last_peered":"2024-03-19T11:31:04.100621+0000","last_clean":"2024-03-19T11:31:04.100621+0000","last_became_active":"2024-03-19T11:31:00.069759+0000","last_became_peered":"2024-03-19T11:31:00.069759+0000","last_unstale":"2024-03-19T11:31:04.100621+0000","last_undegraded":"2024-03-19T11:31:04.100621+0000","last_fullsized":"2024-03-19T11:31:04.100621+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":6,"last_scrub":"62'96","last_scrub_stamp":"2024-03-19T11:30:43.400885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:43.400885+0000","objects_scrubbed":9,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:55.649626+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.050771269000000001,"stat_sum":{"num_bytes":15281651,"num_objects":9,"num_object_clones":7,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":48,"num_read_kb":51399,"num_write":18,"num_write_kb":4886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":22983003,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"2"},{"start":"14","length":"1"}]},{"pgid":"4.20","version":"56'46","reported_seq":47,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-19T11:31:03.052424+0000","last_change":"2024-03-19T11:31:02.086750+0000","last_active":"2024-03-19T11:31:03.052424+0000","last_peered":"2024-03-19T11:31:03.052424+0000","last_clean":"2024-03-19T11:31:03.052424+0000","last_became_active":"2024-03-19T11:30:53.067251+0000","last_became_peered":"2024-03-19T11:30:53.067251+0000","last_unstale":"2024-03-19T11:31:03.052424+0000","last_undegraded":"2024-03-19T11:31:03.052424+0000","last_fullsized":"2024-03-19T11:31:03.052424+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":6,"last_scrub":"56'46","last_scrub_stamp":"2024-03-19T11:30:41.767100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:41.767100+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:52.904341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0046137030000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.1f","version":"0'0","reported_seq":55,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-19T11:31:03.053712+0000","last_change":"2024-03-19T11:31:02.082397+0000","last_active":"2024-03-19T11:31:03.053712+0000","last_peered":"2024-03-19T11:31:03.053712+0000","last_clean":"2024-03-19T11:31:03.053712+0000","last_became_active":"2024-03-19T11:30:53.087211+0000","last_became_peered":"2024-03-19T11:30:53.087211+0000","last_unstale":"2024-03-19T11:31:03.053712+0000","last_undegraded":"2024-03-19T11:31:03.053712+0000","last_fullsized":"2024-03-19T11:31:03.053712+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:30:42.377963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.377963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:01.745397+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0021919130000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.1e","version":"83'69","reported_seq":62,"reported_epoch":83,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:02.087501+0000","last_change":"2024-03-19T11:31:02.087501+0000","last_active":"2024-03-19T11:31:02.087501+0000","last_peered":"2024-03-19T11:31:02.087501+0000","last_clean":"2024-03-19T11:31:02.087501+0000","last_became_active":"2024-03-19T11:30:53.084452+0000","last_became_peered":"2024-03-19T11:30:53.084452+0000","last_unstale":"2024-03-19T11:31:02.087501+0000","last_undegraded":"2024-03-19T11:31:02.087501+0000","last_fullsized":"2024-03-19T11:31:02.087501+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"64'62","last_scrub_stamp":"2024-03-19T11:30:42.778159+0000","last_deep_scrub":"64'62","last_deep_scrub_stamp":"2024-03-19T11:30:42.778159+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.778159+0000","objects_scrubbed":4,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:43.449616+0000","scrub_duration":31,"objects_trimmed":1,"snaptrim_duration":0.00063602400000000001,"stat_sum":{"num_bytes":3794928,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":7565,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3779426,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.1d","version":"82'66","reported_seq":61,"reported_epoch":85,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:04.050025+0000","last_change":"2024-03-19T11:31:02.078403+0000","last_active":"2024-03-19T11:31:04.050025+0000","last_peered":"2024-03-19T11:31:04.050025+0000","last_clean":"2024-03-19T11:31:04.050025+0000","last_became_active":"2024-03-19T11:30:53.067748+0000","last_became_peered":"2024-03-19T11:30:53.067748+0000","last_unstale":"2024-03-19T11:31:04.050025+0000","last_undegraded":"2024-03-19T11:31:04.050025+0000","last_fullsized":"2024-03-19T11:31:04.050025+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"61'59","last_scrub_stamp":"2024-03-19T11:30:42.601863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.601863+0000","objects_scrubbed":2,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:52.206103+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.022205709000000001,"stat_sum":{"num_bytes":6649813,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":61,"num_write":4,"num_write_kb":1341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2817390,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.1c","version":"85'35","reported_seq":69,"reported_epoch":85,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:04.055902+0000","last_change":"2024-03-19T11:31:04.055902+0000","last_active":"2024-03-19T11:31:04.055902+0000","last_peered":"2024-03-19T11:31:04.055902+0000","last_clean":"2024-03-19T11:31:04.055902+0000","last_became_active":"2024-03-19T11:30:53.082655+0000","last_became_peered":"2024-03-19T11:30:53.082655+0000","last_unstale":"2024-03-19T11:31:04.055902+0000","last_undegraded":"2024-03-19T11:31:04.055902+0000","last_fullsized":"2024-03-19T11:31:04.055902+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"62'31","last_scrub_stamp":"2024-03-19T11:30:42.771396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.771396+0000","objects_scrubbed":3,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:55.484122+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.00073920500000000005,"stat_sum":{"num_bytes":3210104,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":47464,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":5034309,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.1b","version":"84'78","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-19T11:31:03.165322+0000","last_change":"2024-03-19T11:31:03.165322+0000","last_active":"2024-03-19T11:31:03.165322+0000","last_peered":"2024-03-19T11:31:03.165322+0000","last_clean":"2024-03-19T11:31:03.165322+0000","last_became_active":"2024-03-19T11:31:03.056769+0000","last_became_peered":"2024-03-19T11:31:03.056769+0000","last_unstale":"2024-03-19T11:31:03.165322+0000","last_undegraded":"2024-03-19T11:31:03.165322+0000","last_fullsized":"2024-03-19T11:31:03.165322+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":5,"last_scrub":"62'62","last_scrub_stamp":"2024-03-19T11:30:42.181424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.181424+0000","objects_scrubbed":5,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:54.474706+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0047244899999999996,"stat_sum":{"num_bytes":14753221,"num_objects":6,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":47032,"num_write":5,"num_write_kb":1437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":17966556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.1a","version":"84'99","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-19T11:31:03.175029+0000","last_change":"2024-03-19T11:31:03.175029+0000","last_active":"2024-03-19T11:31:03.175029+0000","last_peered":"2024-03-19T11:31:03.175029+0000","last_clean":"2024-03-19T11:31:03.175029+0000","last_became_active":"2024-03-19T11:31:03.056520+0000","last_became_peered":"2024-03-19T11:31:03.056520+0000","last_unstale":"2024-03-19T11:31:03.175029+0000","last_undegraded":"2024-03-19T11:31:03.175029+0000","last_fullsized":"2024-03-19T11:31:03.175029+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":5,"last_scrub":"61'76","last_scrub_stamp":"2024-03-19T11:30:41.433501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:41.433501+0000","objects_scrubbed":5,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:59.309150+0000","scrub_duration":11,"objects_trimmed":1,"snaptrim_duration":0.0078747160000000004,"stat_sum":{"num_bytes":15479824,"num_objects":7,"num_object_clones":5,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":15,"num_read_kb":9625,"num_write":18,"num_write_kb":5314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":16291222,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.19","version":"83'111","reported_seq":471,"reported_epoch":85,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:04.098365+0000","last_change":"2024-03-19T11:31:04.098409+0000","last_active":"2024-03-19T11:31:04.098365+0000","last_peered":"2024-03-19T11:31:04.098365+0000","last_clean":"2024-03-19T11:31:04.098365+0000","last_became_active":"2024-03-19T11:30:53.076188+0000","last_became_peered":"2024-03-19T11:30:53.076188+0000","last_unstale":"2024-03-19T11:31:04.098365+0000","last_undegraded":"2024-03-19T11:31:04.098365+0000","last_fullsized":"2024-03-19T11:31:04.098365+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"41'37","last_scrub_stamp":"2024-03-19T11:30:27.705159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:27.705159+0000","objects_scrubbed":4,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:29.203476+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.047949758000000002,"stat_sum":{"num_bytes":18546999,"num_objects":12,"num_object_clones":8,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":88,"num_read_kb":101447,"num_write":38,"num_write_kb":6633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":21870393,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":11,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.18","version":"79'119","reported_seq":413,"reported_epoch":85,"state":"active+recovering+remapped","last_fresh":"2024-03-19T11:31:04.184848+0000","last_change":"2024-03-19T11:31:04.070548+0000","last_active":"2024-03-19T11:31:04.184848+0000","last_peered":"2024-03-19T11:31:04.184848+0000","last_clean":"2024-03-19T11:30:58.054172+0000","last_became_active":"2024-03-19T11:31:04.066044+0000","last_became_peered":"2024-03-19T11:31:04.066044+0000","last_unstale":"2024-03-19T11:31:04.184848+0000","last_undegraded":"2024-03-19T11:31:04.184848+0000","last_fullsized":"2024-03-19T11:31:04.184848+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":5,"last_scrub":"48'77","last_scrub_stamp":"2024-03-19T11:30:29.399762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:29.399762+0000","objects_scrubbed":6,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.011956031000000001,"stat_sum":{"num_bytes":21618293,"num_objects":9,"num_object_clones":7,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":40,"num_read_kb":37994,"num_write":17,"num_write_kb":4402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":14888652,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":7,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":1,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.17","version":"41'24","reported_seq":55,"reported_epoch":85,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:04.050331+0000","last_change":"2024-03-19T11:31:02.079786+0000","last_active":"2024-03-19T11:31:04.050331+0000","last_peered":"2024-03-19T11:31:04.050331+0000","last_clean":"2024-03-19T11:31:04.050331+0000","last_became_active":"2024-03-19T11:30:53.074270+0000","last_became_peered":"2024-03-19T11:30:53.074270+0000","last_unstale":"2024-03-19T11:31:04.050331+0000","last_undegraded":"2024-03-19T11:31:04.050331+0000","last_fullsized":"2024-03-19T11:31:04.050331+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"41'24","last_scrub_stamp":"2024-03-19T11:30:53.993428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:53.993428+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:54.020328+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.022407469999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.16","version":"83'99","reported_seq":52,"reported_epoch":83,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:02.090231+0000","last_change":"2024-03-19T11:31:02.077638+0000","last_active":"2024-03-19T11:31:02.090231+0000","last_peered":"2024-03-19T11:31:02.090231+0000","last_clean":"2024-03-19T11:31:02.090231+0000","last_became_active":"2024-03-19T11:30:59.074063+0000","last_became_peered":"2024-03-19T11:30:59.074063+0000","last_unstale":"2024-03-19T11:31:02.090231+0000","last_undegraded":"2024-03-19T11:31:02.090231+0000","last_fullsized":"2024-03-19T11:31:02.090231+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":5,"last_scrub":"49'56","last_scrub_stamp":"2024-03-19T11:30:29.212766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:29.212766+0000","objects_scrubbed":5,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:47.363425+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.096278618999999996,"stat_sum":{"num_bytes":12479433,"num_objects":7,"num_object_clones":4,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":95,"num_read_kb":91318,"num_write":20,"num_write_kb":6321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":24165549,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":7,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.15","version":"85'148","reported_seq":547,"reported_epoch":85,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:04.108361+0000","last_change":"2024-03-19T11:31:02.074320+0000","last_active":"2024-03-19T11:31:04.108361+0000","last_peered":"2024-03-19T11:31:04.108361+0000","last_clean":"2024-03-19T11:31:04.108361+0000","last_became_active":"2024-03-19T11:30:56.063100+0000","last_became_peered":"2024-03-19T11:30:56.063100+0000","last_unstale":"2024-03-19T11:31:04.108361+0000","last_undegraded":"2024-03-19T11:31:04.108361+0000","last_fullsized":"2024-03-19T11:31:04.108361+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":5,"last_scrub":"49'74","last_scrub_stamp":"2024-03-19T11:30:29.457542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:29.457542+0000","objects_scrubbed":7,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:56.479079+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.019267638,"stat_sum":{"num_bytes":22140490,"num_objects":13,"num_object_clones":8,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":128,"num_read_kb":146180,"num_write":32,"num_write_kb":8100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":18744415,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":8,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.14","version":"40'56","reported_seq":220,"reported_epoch":85,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:04.051467+0000","last_change":"2024-03-19T11:31:02.093175+0000","last_active":"2024-03-19T11:31:04.051467+0000","last_peered":"2024-03-19T11:31:04.051467+0000","last_clean":"2024-03-19T11:31:04.051467+0000","last_became_active":"2024-03-19T11:31:00.065648+0000","last_became_peered":"2024-03-19T11:31:00.065648+0000","last_unstale":"2024-03-19T11:31:04.051467+0000","last_undegraded":"2024-03-19T11:31:04.051467+0000","last_fullsized":"2024-03-19T11:31:04.051467+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":5,"last_scrub":"40'56","last_scrub_stamp":"2024-03-19T11:30:54.998282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:54.998282+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:09.561569+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.014000106999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"2"},{"start":"14","length":"1"}]},{"pgid":"4.13","version":"48'22","reported_seq":92,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-19T11:31:03.047937+0000","last_change":"2024-03-19T11:31:02.077145+0000","last_active":"2024-03-19T11:31:03.047937+0000","last_peered":"2024-03-19T11:31:03.047937+0000","last_clean":"2024-03-19T11:31:03.047937+0000","last_became_active":"2024-03-19T11:30:45.245367+0000","last_became_peered":"2024-03-19T11:30:45.245367+0000","last_unstale":"2024-03-19T11:31:03.047937+0000","last_undegraded":"2024-03-19T11:31:03.047937+0000","last_fullsized":"2024-03-19T11:31:03.047937+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":5,"last_scrub":"48'22","last_scrub_stamp":"2024-03-19T11:30:27.183294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:27.183294+0000","objects_scrubbed":2,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:48.571632+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.007358018,"stat_sum":{"num_bytes":5361367,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":31,"num_read_kb":30730,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":3001789,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.12","version":"28'6","reported_seq":33,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.055008+0000","last_change":"2024-03-19T11:31:04.055613+0000","last_active":"2024-03-19T11:31:04.055008+0000","last_peered":"2024-03-19T11:31:04.055008+0000","last_clean":"2024-03-19T11:31:04.055008+0000","last_became_active":"2024-03-19T11:31:00.064380+0000","last_became_peered":"2024-03-19T11:31:00.064380+0000","last_unstale":"2024-03-19T11:31:04.055008+0000","last_undegraded":"2024-03-19T11:31:04.055008+0000","last_fullsized":"2024-03-19T11:31:04.055008+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":5,"last_scrub":"28'6","last_scrub_stamp":"2024-03-19T11:30:42.390965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.390965+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077181699999999995,"stat_sum":{"num_bytes":1176474,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"},{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.11","version":"83'81","reported_seq":178,"reported_epoch":85,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:04.050088+0000","last_change":"2024-03-19T11:31:02.092082+0000","last_active":"2024-03-19T11:31:04.050088+0000","last_peered":"2024-03-19T11:31:04.050088+0000","last_clean":"2024-03-19T11:31:04.050088+0000","last_became_active":"2024-03-19T11:30:30.052184+0000","last_became_peered":"2024-03-19T11:30:30.052184+0000","last_unstale":"2024-03-19T11:31:04.050088+0000","last_undegraded":"2024-03-19T11:31:04.050088+0000","last_fullsized":"2024-03-19T11:31:04.050088+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":5,"last_scrub":"47'60","last_scrub_stamp":"2024-03-19T11:30:30.780348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:30.780348+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:31.151678+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.014285984999999999,"stat_sum":{"num_bytes":8084512,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":20,"num_read_kb":23163,"num_write":14,"num_write_kb":4028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.10","version":"85'34","reported_seq":340,"reported_epoch":85,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:04.100696+0000","last_change":"2024-03-19T11:31:04.100696+0000","last_active":"2024-03-19T11:31:04.100696+0000","last_peered":"2024-03-19T11:31:04.100696+0000","last_clean":"2024-03-19T11:31:04.100696+0000","last_became_active":"2024-03-19T11:31:00.065617+0000","last_became_peered":"2024-03-19T11:31:00.065617+0000","last_unstale":"2024-03-19T11:31:04.100696+0000","last_undegraded":"2024-03-19T11:31:04.100696+0000","last_fullsized":"2024-03-19T11:31:04.100696+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":5,"last_scrub":"50'30","last_scrub_stamp":"2024-03-19T11:30:28.787058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:28.787058+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:51.738618+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.0079239749999999998,"stat_sum":{"num_bytes":3462209,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":20772,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":7102596,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.054605+0000","last_change":"2024-03-19T11:30:03.276845+0000","last_active":"2024-03-19T11:31:04.054605+0000","last_peered":"2024-03-19T11:31:04.054605+0000","last_clean":"2024-03-19T11:31:04.054605+0000","last_became_active":"2024-03-19T11:30:03.276220+0000","last_became_peered":"2024-03-19T11:30:03.276220+0000","last_unstale":"2024-03-19T11:31:04.054605+0000","last_undegraded":"2024-03-19T11:31:04.054605+0000","last_fullsized":"2024-03-19T11:31:04.054605+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:07.472401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"59'92","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-19T11:31:03.065335+0000","last_change":"2024-03-19T11:31:03.059104+0000","last_active":"2024-03-19T11:31:03.065335+0000","last_peered":"2024-03-19T11:31:03.065335+0000","last_clean":"2024-03-19T11:31:03.065335+0000","last_became_active":"2024-03-19T11:31:03.052657+0000","last_became_peered":"2024-03-19T11:31:03.052657+0000","last_unstale":"2024-03-19T11:31:03.065335+0000","last_undegraded":"2024-03-19T11:31:03.065335+0000","last_fullsized":"2024-03-19T11:31:03.065335+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'92","last_scrub_stamp":"2024-03-19T11:30:41.351520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:41.351520+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:07.204778+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0052965570000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.055239+0000","last_change":"2024-03-19T11:31:00.067375+0000","last_active":"2024-03-19T11:31:04.055239+0000","last_peered":"2024-03-19T11:31:04.055239+0000","last_clean":"2024-03-19T11:31:04.055239+0000","last_became_active":"2024-03-19T11:31:00.067099+0000","last_became_peered":"2024-03-19T11:31:00.067099+0000","last_unstale":"2024-03-19T11:31:04.055239+0000","last_undegraded":"2024-03-19T11:31:04.055239+0000","last_fullsized":"2024-03-19T11:31:04.055239+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:06.356796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"79'65","reported_seq":347,"reported_epoch":85,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:04.050073+0000","last_change":"2024-03-19T11:31:02.078237+0000","last_active":"2024-03-19T11:31:04.050073+0000","last_peered":"2024-03-19T11:31:04.050073+0000","last_clean":"2024-03-19T11:31:04.050073+0000","last_became_active":"2024-03-19T11:30:38.065693+0000","last_became_peered":"2024-03-19T11:30:38.065693+0000","last_unstale":"2024-03-19T11:31:04.050073+0000","last_undegraded":"2024-03-19T11:31:04.050073+0000","last_fullsized":"2024-03-19T11:31:04.050073+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'57","last_scrub_stamp":"2024-03-19T11:30:43.792326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:43.792326+0000","objects_scrubbed":6,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:55.566850+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.022132443000000002,"stat_sum":{"num_bytes":9323261,"num_objects":6,"num_object_clones":5,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":11,"num_read_kb":16661,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":160,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.052066+0000","last_change":"2024-03-19T11:30:59.072585+0000","last_active":"2024-03-19T11:31:04.052066+0000","last_peered":"2024-03-19T11:31:04.052066+0000","last_clean":"2024-03-19T11:31:04.052066+0000","last_became_active":"2024-03-19T11:30:59.071925+0000","last_became_peered":"2024-03-19T11:30:59.071925+0000","last_unstale":"2024-03-19T11:31:04.052066+0000","last_undegraded":"2024-03-19T11:31:04.052066+0000","last_fullsized":"2024-03-19T11:31:04.052066+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:08.925537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"41'19","reported_seq":295,"reported_epoch":83,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:02.077549+0000","last_change":"2024-03-19T11:31:02.077529+0000","last_active":"2024-03-19T11:31:02.077549+0000","last_peered":"2024-03-19T11:31:02.077549+0000","last_clean":"2024-03-19T11:31:02.077549+0000","last_became_active":"2024-03-19T11:30:38.058860+0000","last_became_peered":"2024-03-19T11:30:38.058860+0000","last_unstale":"2024-03-19T11:31:02.077549+0000","last_undegraded":"2024-03-19T11:31:02.077549+0000","last_fullsized":"2024-03-19T11:31:02.077549+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'19","last_scrub_stamp":"2024-03-19T11:30:28.180772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:28.180772+0000","objects_scrubbed":1,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00380665,"stat_sum":{"num_bytes":1403368,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4077,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":153,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.055174+0000","last_change":"2024-03-19T11:30:55.366872+0000","last_active":"2024-03-19T11:31:04.055174+0000","last_peered":"2024-03-19T11:31:04.055174+0000","last_clean":"2024-03-19T11:31:04.055174+0000","last_became_active":"2024-03-19T11:29:46.540939+0000","last_became_peered":"2024-03-19T11:29:46.540939+0000","last_unstale":"2024-03-19T11:31:04.055174+0000","last_undegraded":"2024-03-19T11:31:04.055174+0000","last_fullsized":"2024-03-19T11:31:04.055174+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:30:55.366631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:30:55.366631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:10.640223+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"85'17","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.067823+0000","last_change":"2024-03-19T11:31:04.068099+0000","last_active":"2024-03-19T11:31:04.067823+0000","last_peered":"2024-03-19T11:31:04.067823+0000","last_clean":"2024-03-19T11:31:04.067823+0000","last_became_active":"2024-03-19T11:31:00.069200+0000","last_became_peered":"2024-03-19T11:31:00.069200+0000","last_unstale":"2024-03-19T11:31:04.067823+0000","last_undegraded":"2024-03-19T11:31:04.067823+0000","last_fullsized":"2024-03-19T11:31:04.067823+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'10","last_scrub_stamp":"2024-03-19T11:30:54.429030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:54.429030+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.013022977999999999,"stat_sum":{"num_bytes":2742425,"num_objects":1,"num_object_clones":1,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":10173,"num_write":4,"num_write_kb":1071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2800509,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"},{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"3.3","version":"77'99","reported_seq":319,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-19T11:30:57.062257+0000","last_change":"2024-03-19T11:30:30.271391+0000","last_active":"2024-03-19T11:30:57.062257+0000","last_peered":"2024-03-19T11:30:57.062257+0000","last_clean":"2024-03-19T11:30:57.062257+0000","last_became_active":"2024-03-19T11:30:30.058963+0000","last_became_peered":"2024-03-19T11:30:30.058963+0000","last_unstale":"2024-03-19T11:30:57.062257+0000","last_undegraded":"2024-03-19T11:30:57.062257+0000","last_fullsized":"2024-03-19T11:30:57.062257+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:11.045052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53399952,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":74,"num_read_kb":3737,"num_write":107,"num_write_kb":58435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":22537669,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":144,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.068842+0000","last_change":"2024-03-19T11:29:46.321255+0000","last_active":"2024-03-19T11:31:04.068842+0000","last_peered":"2024-03-19T11:31:04.068842+0000","last_clean":"2024-03-19T11:31:04.068842+0000","last_became_active":"2024-03-19T11:29:46.320679+0000","last_became_peered":"2024-03-19T11:29:46.320679+0000","last_unstale":"2024-03-19T11:31:04.068842+0000","last_undegraded":"2024-03-19T11:31:04.068842+0000","last_fullsized":"2024-03-19T11:31:04.068842+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:14.793563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"85'132","reported_seq":508,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.061158+0000","last_change":"2024-03-19T11:31:04.061533+0000","last_active":"2024-03-19T11:31:04.061158+0000","last_peered":"2024-03-19T11:31:04.061158+0000","last_clean":"2024-03-19T11:31:04.061158+0000","last_became_active":"2024-03-19T11:31:03.054624+0000","last_became_peered":"2024-03-19T11:31:03.054624+0000","last_unstale":"2024-03-19T11:31:04.061158+0000","last_undegraded":"2024-03-19T11:31:04.061158+0000","last_fullsized":"2024-03-19T11:31:04.061158+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'119","last_scrub_stamp":"2024-03-19T11:30:52.512203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:52.512203+0000","objects_scrubbed":7,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:02.677965+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.0068329189999999998,"stat_sum":{"num_bytes":14273317,"num_objects":6,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":70,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"},{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"3.0","version":"85'100","reported_seq":247,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.068253+0000","last_change":"2024-03-19T11:29:57.550320+0000","last_active":"2024-03-19T11:31:04.068253+0000","last_peered":"2024-03-19T11:31:04.068253+0000","last_clean":"2024-03-19T11:31:04.068253+0000","last_became_active":"2024-03-19T11:29:57.549928+0000","last_became_peered":"2024-03-19T11:29:57.549928+0000","last_unstale":"2024-03-19T11:31:04.068253+0000","last_undegraded":"2024-03-19T11:31:04.068253+0000","last_fullsized":"2024-03-19T11:31:04.068253+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28242922,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":80,"num_read_kb":3042,"num_write":105,"num_write_kb":39820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":146,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.053846+0000","last_change":"2024-03-19T11:29:46.319811+0000","last_active":"2024-03-19T11:31:04.053846+0000","last_peered":"2024-03-19T11:31:04.053846+0000","last_clean":"2024-03-19T11:31:04.053846+0000","last_became_active":"2024-03-19T11:29:46.319334+0000","last_became_peered":"2024-03-19T11:29:46.319334+0000","last_unstale":"2024-03-19T11:31:04.053846+0000","last_undegraded":"2024-03-19T11:31:04.053846+0000","last_fullsized":"2024-03-19T11:31:04.053846+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:04.320463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"61'40","reported_seq":33,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.065031+0000","last_change":"2024-03-19T11:31:04.065328+0000","last_active":"2024-03-19T11:31:04.065031+0000","last_peered":"2024-03-19T11:31:04.065031+0000","last_clean":"2024-03-19T11:31:04.065031+0000","last_became_active":"2024-03-19T11:31:00.069809+0000","last_became_peered":"2024-03-19T11:31:00.069809+0000","last_unstale":"2024-03-19T11:31:04.065031+0000","last_undegraded":"2024-03-19T11:31:04.065031+0000","last_fullsized":"2024-03-19T11:31:04.065031+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'40","last_scrub_stamp":"2024-03-19T11:30:56.505586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:56.505586+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:09.446385+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0098007430000000006,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"},{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"3.1","version":"82'85","reported_seq":289,"reported_epoch":82,"state":"peering","last_fresh":"2024-03-19T11:31:01.114104+0000","last_change":"2024-03-19T11:31:02.076669+0000","last_active":"2024-03-19T11:31:02.076669+0000","last_peered":"2024-03-19T11:31:01.114104+0000","last_clean":"2024-03-19T11:31:01.114104+0000","last_became_active":"2024-03-19T11:30:26.927069+0000","last_became_peered":"2024-03-19T11:30:26.927069+0000","last_unstale":"2024-03-19T11:31:01.114104+0000","last_undegraded":"2024-03-19T11:31:01.114104+0000","last_fullsized":"2024-03-19T11:31:01.114104+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42832129,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":71,"num_read_kb":6652,"num_write":89,"num_write_kb":49391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":16477750,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,4],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":155,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.052299+0000","last_change":"2024-03-19T11:31:00.070027+0000","last_active":"2024-03-19T11:31:04.052299+0000","last_peered":"2024-03-19T11:31:04.052299+0000","last_clean":"2024-03-19T11:31:04.052299+0000","last_became_active":"2024-03-19T11:31:00.068989+0000","last_became_peered":"2024-03-19T11:31:00.068989+0000","last_unstale":"2024-03-19T11:31:04.052299+0000","last_undegraded":"2024-03-19T11:31:04.052299+0000","last_fullsized":"2024-03-19T11:31:04.052299+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:11.599600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"77'68","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-19T11:31:02.085910+0000","last_change":"2024-03-19T11:31:02.085910+0000","last_active":"2024-03-19T11:31:02.085910+0000","last_peered":"2024-03-19T11:31:02.085910+0000","last_clean":"2024-03-19T11:31:02.085910+0000","last_became_active":"2024-03-19T11:30:59.075057+0000","last_became_peered":"2024-03-19T11:30:59.075057+0000","last_unstale":"2024-03-19T11:31:02.085910+0000","last_undegraded":"2024-03-19T11:31:02.085910+0000","last_fullsized":"2024-03-19T11:31:02.085910+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'65","last_scrub_stamp":"2024-03-19T11:30:53.385414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:53.385414+0000","objects_scrubbed":3,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:05.375039+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.009635546,"stat_sum":{"num_bytes":7189731,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4209633,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"3.2","version":"85'84","reported_seq":239,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.049868+0000","last_change":"2024-03-19T11:29:57.549277+0000","last_active":"2024-03-19T11:31:04.049868+0000","last_peered":"2024-03-19T11:31:04.049868+0000","last_clean":"2024-03-19T11:31:04.049868+0000","last_became_active":"2024-03-19T11:29:57.548946+0000","last_became_peered":"2024-03-19T11:29:57.548946+0000","last_unstale":"2024-03-19T11:31:04.049868+0000","last_undegraded":"2024-03-19T11:31:04.049868+0000","last_fullsized":"2024-03-19T11:31:04.049868+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:08.380405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39345556,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":76,"num_read_kb":8248,"num_write":94,"num_write_kb":41080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":144,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.067990+0000","last_change":"2024-03-19T11:29:46.322491+0000","last_active":"2024-03-19T11:31:04.067990+0000","last_peered":"2024-03-19T11:31:04.067990+0000","last_clean":"2024-03-19T11:31:04.067990+0000","last_became_active":"2024-03-19T11:29:46.321760+0000","last_became_peered":"2024-03-19T11:29:46.321760+0000","last_unstale":"2024-03-19T11:31:04.067990+0000","last_undegraded":"2024-03-19T11:31:04.067990+0000","last_fullsized":"2024-03-19T11:31:04.067990+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:04.561915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":329,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.052318+0000","last_change":"2024-03-19T11:31:03.367805+0000","last_active":"2024-03-19T11:31:04.052318+0000","last_peered":"2024-03-19T11:31:04.052318+0000","last_clean":"2024-03-19T11:31:04.052318+0000","last_became_active":"2024-03-19T11:30:59.073709+0000","last_became_peered":"2024-03-19T11:30:59.073709+0000","last_unstale":"2024-03-19T11:31:04.052318+0000","last_undegraded":"2024-03-19T11:31:04.052318+0000","last_fullsized":"2024-03-19T11:31:04.052318+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2024-03-19T11:31:03.367669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:43.166618+0000","last_clean_scrub_stamp":"2024-03-19T11:31:03.367669+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:29.977573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"84'103","reported_seq":480,"reported_epoch":84,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:03.169808+0000","last_change":"2024-03-19T11:31:03.168375+0000","last_active":"2024-03-19T11:31:03.169808+0000","last_peered":"2024-03-19T11:31:03.169808+0000","last_clean":"2024-03-19T11:31:03.169808+0000","last_became_active":"2024-03-19T11:31:03.065344+0000","last_became_peered":"2024-03-19T11:31:03.065344+0000","last_unstale":"2024-03-19T11:31:03.169808+0000","last_undegraded":"2024-03-19T11:31:03.169808+0000","last_fullsized":"2024-03-19T11:31:03.169808+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'81","last_scrub_stamp":"2024-03-19T11:30:52.360396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:52.360396+0000","objects_scrubbed":6,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:11.500650+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.032057466999999999,"stat_sum":{"num_bytes":10838384,"num_objects":8,"num_object_clones":6,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":13,"num_read_kb":12349,"num_write":8,"num_write_kb":1273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":10558635,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"c","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.8","version":"62'71","reported_seq":359,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.077614+0000","last_change":"2024-03-19T11:31:04.078135+0000","last_active":"2024-03-19T11:31:04.077614+0000","last_peered":"2024-03-19T11:31:04.077614+0000","last_clean":"2024-03-19T11:31:04.077614+0000","last_became_active":"2024-03-19T11:30:45.251283+0000","last_became_peered":"2024-03-19T11:30:45.251283+0000","last_unstale":"2024-03-19T11:31:04.077614+0000","last_undegraded":"2024-03-19T11:31:04.077614+0000","last_fullsized":"2024-03-19T11:31:04.077614+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'71","last_scrub_stamp":"2024-03-19T11:30:53.381386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:53.381386+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:18.275213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010232903999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6","length":"1"},{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.9","version":"63'55","reported_seq":384,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-19T11:31:03.046677+0000","last_change":"2024-03-19T11:31:02.070296+0000","last_active":"2024-03-19T11:31:03.046677+0000","last_peered":"2024-03-19T11:31:03.046677+0000","last_clean":"2024-03-19T11:31:03.046677+0000","last_became_active":"2024-03-19T11:31:00.069789+0000","last_became_peered":"2024-03-19T11:31:00.069789+0000","last_unstale":"2024-03-19T11:31:03.046677+0000","last_undegraded":"2024-03-19T11:31:03.046677+0000","last_fullsized":"2024-03-19T11:31:03.046677+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'55","last_scrub_stamp":"2024-03-19T11:30:54.706235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:54.706235+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:01.278994+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00082106599999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.a","version":"62'77","reported_seq":375,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-19T11:31:03.053296+0000","last_change":"2024-03-19T11:31:02.084460+0000","last_active":"2024-03-19T11:31:03.053296+0000","last_peered":"2024-03-19T11:31:03.053296+0000","last_clean":"2024-03-19T11:31:03.053296+0000","last_became_active":"2024-03-19T11:30:45.256124+0000","last_became_peered":"2024-03-19T11:30:45.256124+0000","last_unstale":"2024-03-19T11:31:03.053296+0000","last_undegraded":"2024-03-19T11:31:03.053296+0000","last_fullsized":"2024-03-19T11:31:03.053296+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'77","last_scrub_stamp":"2024-03-19T11:30:55.291823+0000","last_deep_scrub":"62'77","last_deep_scrub_stamp":"2024-03-19T11:30:55.291823+0000","last_clean_scrub_stamp":"2024-03-19T11:30:55.291823+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:11.588174+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00325363,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.b","version":"70'62","reported_seq":375,"reported_epoch":83,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:02.077320+0000","last_change":"2024-03-19T11:31:02.077300+0000","last_active":"2024-03-19T11:31:02.077320+0000","last_peered":"2024-03-19T11:31:02.077320+0000","last_clean":"2024-03-19T11:31:02.077320+0000","last_became_active":"2024-03-19T11:30:45.261756+0000","last_became_peered":"2024-03-19T11:30:45.261756+0000","last_unstale":"2024-03-19T11:31:02.077320+0000","last_undegraded":"2024-03-19T11:31:02.077320+0000","last_fullsized":"2024-03-19T11:31:02.077320+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'62","last_scrub_stamp":"2024-03-19T11:30:56.095039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:56.095039+0000","objects_scrubbed":3,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:18.760680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063617100000000005,"stat_sum":{"num_bytes":5215740,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.c","version":"56'29","reported_seq":10,"reported_epoch":84,"state":"clean+premerge+peered","last_fresh":"2024-03-19T11:31:03.063714+0000","last_change":"2024-03-19T11:31:03.058030+0000","last_active":"2024-03-19T11:31:01.070481+0000","last_peered":"2024-03-19T11:31:03.063714+0000","last_clean":"2024-03-19T11:31:03.063714+0000","last_became_active":"2024-03-19T11:30:56.061560+0000","last_became_peered":"2024-03-19T11:31:03.056794+0000","last_unstale":"2024-03-19T11:31:03.063714+0000","last_undegraded":"2024-03-19T11:31:03.063714+0000","last_fullsized":"2024-03-19T11:31:03.063714+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'29","last_scrub_stamp":"2024-03-19T11:30:53.743385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:53.743385+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:02.142968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.021151900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.d","version":"85'82","reported_seq":393,"reported_epoch":85,"state":"active+clean+remapped+snaptrim","last_fresh":"2024-03-19T11:31:04.184330+0000","last_change":"2024-03-19T11:31:04.182796+0000","last_active":"2024-03-19T11:31:04.184330+0000","last_peered":"2024-03-19T11:31:04.184330+0000","last_clean":"2024-03-19T11:31:04.184330+0000","last_became_active":"2024-03-19T11:31:04.064814+0000","last_became_peered":"2024-03-19T11:31:04.064814+0000","last_unstale":"2024-03-19T11:31:04.184330+0000","last_undegraded":"2024-03-19T11:31:04.184330+0000","last_fullsized":"2024-03-19T11:31:04.184330+0000","mapping_epoch":84,"log_start":"70'71","ondisk_log_start":"70'71","created":21,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'73","last_scrub_stamp":"2024-03-19T11:30:53.519178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:53.519178+0000","objects_scrubbed":6,"log_size":11,"log_dups_size":50,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:15.690636+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0030257930000000001,"stat_sum":{"num_bytes":10067275,"num_objects":6,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":9,"num_read_kb":11335,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":8146998,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[1,2],"avail_no_missing":["1","2","4"],"object_location_counts":[{"shards":"1,2","objects":6}],"blocked_by":[],"up_primary":4,"acting_primary":1,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.e","version":"79'78","reported_seq":352,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-19T11:31:03.046968+0000","last_change":"2024-03-19T11:31:02.074204+0000","last_active":"2024-03-19T11:31:03.046968+0000","last_peered":"2024-03-19T11:31:03.046968+0000","last_clean":"2024-03-19T11:31:03.046968+0000","last_became_active":"2024-03-19T11:30:38.060487+0000","last_became_peered":"2024-03-19T11:30:38.060487+0000","last_unstale":"2024-03-19T11:31:03.046968+0000","last_undegraded":"2024-03-19T11:31:03.046968+0000","last_fullsized":"2024-03-19T11:31:03.046968+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'65","last_scrub_stamp":"2024-03-19T11:30:43.824620+0000","last_deep_scrub":"65'65","last_deep_scrub_stamp":"2024-03-19T11:30:43.824620+0000","last_clean_scrub_stamp":"2024-03-19T11:30:43.824620+0000","objects_scrubbed":4,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:48.641497+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0.0046356100000000001,"stat_sum":{"num_bytes":12990971,"num_objects":6,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":21,"num_read_kb":22723,"num_write":8,"num_write_kb":2301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.f","version":"0'0","reported_seq":256,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.065434+0000","last_change":"2024-03-19T11:31:04.065959+0000","last_active":"2024-03-19T11:31:04.065434+0000","last_peered":"2024-03-19T11:31:04.065434+0000","last_clean":"2024-03-19T11:31:04.065434+0000","last_became_active":"2024-03-19T11:30:38.050919+0000","last_became_peered":"2024-03-19T11:30:38.050919+0000","last_unstale":"2024-03-19T11:31:04.065434+0000","last_undegraded":"2024-03-19T11:31:04.065434+0000","last_fullsized":"2024-03-19T11:31:04.065434+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:30:43.411893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:43.411893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:54.719455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010140278000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"},{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]}],"pool_stats":[{"poolid":4,"num_pg":44,"stat_sum":{"num_bytes":275155131,"num_objects":142,"num_object_clones":91,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":139,"num_whiteouts":6,"num_read":669,"num_read_kb":715159,"num_write":198,"num_write_kb":50396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":200107110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":93,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":402448384,"data_stored":457616802,"data_compressed":2587576,"data_compressed_allocated":1728512,"data_compressed_original":5898240,"omap_allocated":0,"internal_metadata":0},"log_size":2787,"ondisk_log_size":2787,"up":88,"acting":88,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":163820559,"num_objects":143,"num_object_clones":0,"num_object_copies":286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":143,"num_whiteouts":0,"num_read":301,"num_read_kb":21679,"num_write":395,"num_write_kb":188726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":39015419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":165703680,"data_stored":179494984,"data_compressed":12248719,"data_compressed_allocated":13119488,"data_compressed_original":27238400,"omap_allocated":0,"internal_metadata":0},"log_size":368,"ondisk_log_size":368,"up":8,"acting":8,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":1586784,"data_compressed":19832,"data_compressed_allocated":98304,"data_compressed_original":1560576,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542164,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83364,"kb_used_data":52184,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94288476,"statfs":{"total":96636764160,"available":96551399424,"internally_reserved":0,"allocated":53436416,"data_stored":62427174,"data_compressed":227710,"data_compressed_allocated":151552,"data_compressed_original":1101824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542164,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84092,"kb_used_data":52908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94287748,"statfs":{"total":96636764160,"available":96550653952,"internally_reserved":0,"allocated":54177792,"data_stored":62533221,"data_compressed":1917086,"data_compressed_allocated":1929216,"data_compressed_original":4210688,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542164,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":129452,"kb_used_data":102432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94242388,"statfs":{"total":96636764160,"available":96504205312,"internally_reserved":0,"allocated":104890368,"data_stored":120523535,"data_compressed":2532072,"data_compressed_allocated":2379776,"data_compressed_original":5521408,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542163,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79208,"kb_used_data":52184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94292632,"statfs":{"total":96636764160,"available":96555655168,"internally_reserved":0,"allocated":53436416,"data_stored":60429347,"data_compressed":1749130,"data_compressed_allocated":1527808,"data_compressed_original":4194304,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542163,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":153924,"kb_used_data":113028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94217916,"statfs":{"total":96636764160,"available":96479145984,"internally_reserved":0,"allocated":115740672,"data_stored":123349286,"data_compressed":1407043,"data_compressed_allocated":1449984,"data_compressed_original":3211264,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95020,"kb_used_data":68000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94276820,"statfs":{"total":96636764160,"available":96539463680,"internally_reserved":0,"allocated":69632000,"data_stored":81512192,"data_compressed":2377281,"data_compressed_allocated":2482176,"data_compressed_original":4935680,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574867,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46512,"kb_used_data":15328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325328,"statfs":{"total":96636764160,"available":96589135872,"internally_reserved":0,"allocated":15695872,"data_stored":17201968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574867,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":125852,"kb_used_data":94668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94245988,"statfs":{"total":96636764160,"available":96507891712,"internally_reserved":0,"allocated":96940032,"data_stored":102441426,"data_compressed":4177120,"data_compressed_allocated":4493312,"data_compressed_original":9555968,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6884,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6884,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6064,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53927936,"data_stored":58198011,"data_compressed":3980813,"data_compressed_allocated":4255744,"data_compressed_original":8638464,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14168064,"data_stored":15397438,"data_compressed":898614,"data_compressed_allocated":970752,"data_compressed_original":2228224,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14168064,"data_stored":15397438,"data_compressed":898614,"data_compressed_allocated":970752,"data_compressed_original":2228224,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":22126592,"data_stored":23997131,"data_compressed":1849977,"data_compressed_allocated":1974272,"data_compressed_original":3883008,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11079680,"data_stored":12191862,"data_compressed":660988,"data_compressed_allocated":712704,"data_compressed_original":1835008,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":19947520,"data_stored":21724875,"data_compressed":1695440,"data_compressed_allocated":1810432,"data_compressed_original":3620864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":30285824,"data_stored":32588229,"data_compressed":2264273,"data_compressed_allocated":2424832,"data_compressed_original":4804608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":38711296,"data_stored":39908792,"data_compressed":189423,"data_compressed_allocated":204800,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90685440,"data_stored":97008266,"data_compressed":390047,"data_compressed_allocated":352256,"data_compressed_original":720896,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":34271232,"data_stored":39685709,"data_compressed":732134,"data_compressed_allocated":430080,"data_compressed_original":1703936,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":78860288,"data_stored":93060511,"data_compressed":682095,"data_compressed_allocated":405504,"data_compressed_original":1638400,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":43638784,"data_stored":47398962,"data_compressed":37577,"data_compressed_allocated":40960,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":29360128,"data_stored":35596983,"data_compressed":221646,"data_compressed_allocated":118784,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":38412288,"data_stored":48442122,"data_compressed":113008,"data_compressed_allocated":57344,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":48508928,"data_stored":56515457,"data_compressed":221646,"data_compressed_allocated":118784,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T11:31:05.503 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T11:31:05.506 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-19T11:31:05.548 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:05.557+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:05.550 INFO:tasks.rados.rados.0.smithi038.stdout:1210: finishing set_chunk to oid 36 2024-03-19T11:31:05.550 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 36 v 70 (ObjNum 164 snap 17 seq_num 164) dirty exists 2024-03-19T11:31:05.550 INFO:tasks.rados.rados.0.smithi038.stdout:1215: finishing write tid 1 to smithi03829856-43 2024-03-19T11:31:05.550 INFO:tasks.rados.rados.0.smithi038.stdout:1206: done (8 left) 2024-03-19T11:31:05.550 INFO:tasks.rados.rados.0.smithi038.stdout:1208: done (7 left) 2024-03-19T11:31:05.550 INFO:tasks.rados.rados.0.smithi038.stdout:1209: done (6 left) 2024-03-19T11:31:05.550 INFO:tasks.rados.rados.0.smithi038.stdout:1210: done (5 left) 2024-03-19T11:31:05.550 INFO:tasks.rados.rados.0.smithi038.stdout:1211: done (4 left) 2024-03-19T11:31:05.551 INFO:tasks.rados.rados.0.smithi038.stdout:1212: done (3 left) 2024-03-19T11:31:05.551 INFO:tasks.rados.rados.0.smithi038.stdout:1213: done (2 left) 2024-03-19T11:31:05.551 INFO:tasks.rados.rados.0.smithi038.stdout:1214: done (1 left) 2024-03-19T11:31:05.551 INFO:tasks.rados.rados.0.smithi038.stdout:1216: write oid 18 current snap is 22 2024-03-19T11:31:05.551 INFO:tasks.rados.rados.0.smithi038.stdout:1215: finishing write tid 2 to smithi03829856-43 2024-03-19T11:31:05.551 INFO:tasks.rados.rados.0.smithi038.stdout:1216: seq_num 199 ranges {544593=799729,1761799=776065,2948352=396241} 2024-03-19T11:31:05.558 INFO:tasks.rados.rados.0.smithi038.stdout:1216: writing smithi03829856-18 from 544593 to 1344322 tid 1 2024-03-19T11:31:05.567 INFO:tasks.rados.rados.0.smithi038.stdout:1216: writing smithi03829856-18 from 1761799 to 2537864 tid 2 2024-03-19T11:31:05.572 INFO:tasks.rados.rados.0.smithi038.stdout:1216: writing smithi03829856-18 from 2948352 to 3344593 tid 3 2024-03-19T11:31:05.572 INFO:tasks.rados.rados.0.smithi038.stdout:1217: read oid 2 snap -1 2024-03-19T11:31:05.572 INFO:tasks.rados.rados.0.smithi038.stdout:1217: expect (ObjNum 185 snap 20 seq_num 185) 2024-03-19T11:31:05.572 INFO:tasks.rados.rados.0.smithi038.stdout:1218: set_chunk oid 17 target oid 2024-03-19T11:31:05.598 INFO:tasks.rados.rados.0.smithi038.stdout:1218: set_chunk oid 17 offset: 282112 length: 1740288 target oid 14344e4389367f38af6188cd767d246d69ce2c8ae2aa751b43f7cabd7b27b4cf offset: 0 snap -1 2024-03-19T11:31:05.598 INFO:tasks.rados.rados.0.smithi038.stdout:1219: snap_remove snap 16 2024-03-19T11:31:05.620 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:05.629+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:05.659 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:05.659 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:05.673 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-19T11:31:05.760 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:05.769+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:05.799 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:31:05.799 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-03-19T11:31:05.815 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":120,"stamp":"2024-03-19T11:31:05.054717+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":439566058,"num_objects":287,"num_object_clones":91,"num_object_copies":574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":284,"num_whiteouts":6,"num_read":1086,"num_read_kb":736938,"num_write":752,"num_write_kb":241538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":220,"num_bytes_recovered":240180385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":93,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3272,"ondisk_log_size":3272,"up":114,"acting":114,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":124,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":797424,"kb_used_data":550732,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":754177296,"statfs":{"total":773094113280,"available":772277551104,"internally_reserved":0,"allocated":563949568,"data_stored":630418149,"data_compressed":14387442,"data_compressed_allocated":14413824,"data_compressed_original":32731136,"omap_allocated":12751,"internal_metadata":246795825},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13842679,"num_objects":7,"num_object_clones":-2,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":-8,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":69,"num_read_kb":66097,"num_write":44,"num_write_kb":11101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":108743946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.998594"},"pg_stats":[{"pgid":"4.2b","version":"61'60","reported_seq":375,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-19T11:31:02.077571+0000","last_change":"2024-03-19T11:31:02.082219+0000","last_active":"2024-03-19T11:31:02.077571+0000","last_peered":"2024-03-19T11:31:02.077571+0000","last_clean":"2024-03-19T11:31:02.077571+0000","last_became_active":"2024-03-19T11:30:45.263076+0000","last_became_peered":"2024-03-19T11:30:45.263076+0000","last_unstale":"2024-03-19T11:31:02.077571+0000","last_undegraded":"2024-03-19T11:31:02.077571+0000","last_fullsized":"2024-03-19T11:31:02.077571+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":6,"last_scrub":"61'60","last_scrub_stamp":"2024-03-19T11:30:55.143981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:55.143981+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:59.980040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00639216,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.2a","version":"83'94","reported_seq":400,"reported_epoch":84,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:03.052847+0000","last_change":"2024-03-19T11:31:02.085627+0000","last_active":"2024-03-19T11:31:03.052847+0000","last_peered":"2024-03-19T11:31:03.052847+0000","last_clean":"2024-03-19T11:31:03.052847+0000","last_became_active":"2024-03-19T11:30:45.260578+0000","last_became_peered":"2024-03-19T11:30:45.260578+0000","last_unstale":"2024-03-19T11:31:03.052847+0000","last_undegraded":"2024-03-19T11:31:03.052847+0000","last_fullsized":"2024-03-19T11:31:03.052847+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":6,"last_scrub":"74'89","last_scrub_stamp":"2024-03-19T11:30:53.312593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:53.312593+0000","objects_scrubbed":6,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:07.058558+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.015736934000000001,"stat_sum":{"num_bytes":10618356,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":781,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.29","version":"83'63","reported_seq":409,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-19T11:31:03.046373+0000","last_change":"2024-03-19T11:31:02.085998+0000","last_active":"2024-03-19T11:31:03.046373+0000","last_peered":"2024-03-19T11:31:03.046373+0000","last_clean":"2024-03-19T11:31:03.046373+0000","last_became_active":"2024-03-19T11:31:00.070287+0000","last_became_peered":"2024-03-19T11:31:00.070287+0000","last_unstale":"2024-03-19T11:31:03.046373+0000","last_undegraded":"2024-03-19T11:31:03.046373+0000","last_fullsized":"2024-03-19T11:31:03.046373+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":6,"last_scrub":"71'60","last_scrub_stamp":"2024-03-19T11:30:52.748479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:52.748479+0000","objects_scrubbed":3,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:56.409127+0000","scrub_duration":7,"objects_trimmed":1,"snaptrim_duration":0.016516072,"stat_sum":{"num_bytes":4387250,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":377,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":5842489,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.28","version":"85'98","reported_seq":383,"reported_epoch":85,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:04.086552+0000","last_change":"2024-03-19T11:31:04.078290+0000","last_active":"2024-03-19T11:31:04.086552+0000","last_peered":"2024-03-19T11:31:04.086552+0000","last_clean":"2024-03-19T11:31:04.086552+0000","last_became_active":"2024-03-19T11:30:45.257600+0000","last_became_peered":"2024-03-19T11:30:45.257600+0000","last_unstale":"2024-03-19T11:31:04.086552+0000","last_undegraded":"2024-03-19T11:31:04.086552+0000","last_fullsized":"2024-03-19T11:31:04.086552+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":6,"last_scrub":"71'85","last_scrub_stamp":"2024-03-19T11:30:52.343454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:52.343454+0000","objects_scrubbed":7,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:16.280349+0000","scrub_duration":7,"objects_trimmed":1,"snaptrim_duration":0.0090263340000000004,"stat_sum":{"num_bytes":11017340,"num_objects":7,"num_object_clones":5,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":6,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.27","version":"85'54","reported_seq":53,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.074043+0000","last_change":"2024-03-19T11:31:04.074317+0000","last_active":"2024-03-19T11:31:04.074043+0000","last_peered":"2024-03-19T11:31:04.074043+0000","last_clean":"2024-03-19T11:31:04.074043+0000","last_became_active":"2024-03-19T11:31:00.070337+0000","last_became_peered":"2024-03-19T11:31:00.070337+0000","last_unstale":"2024-03-19T11:31:04.074043+0000","last_undegraded":"2024-03-19T11:31:04.074043+0000","last_fullsized":"2024-03-19T11:31:04.074043+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":6,"last_scrub":"71'48","last_scrub_stamp":"2024-03-19T11:30:55.467716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:55.467716+0000","objects_scrubbed":4,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:00.305683+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.018507597000000001,"stat_sum":{"num_bytes":6814202,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":11198,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":7070511,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"},{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.26","version":"61'63","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-19T11:31:02.087385+0000","last_change":"2024-03-19T11:31:02.087385+0000","last_active":"2024-03-19T11:31:02.087385+0000","last_peered":"2024-03-19T11:31:02.087385+0000","last_clean":"2024-03-19T11:31:02.087385+0000","last_became_active":"2024-03-19T11:30:59.074337+0000","last_became_peered":"2024-03-19T11:30:59.074337+0000","last_unstale":"2024-03-19T11:31:02.087385+0000","last_undegraded":"2024-03-19T11:31:02.087385+0000","last_fullsized":"2024-03-19T11:31:02.087385+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":6,"last_scrub":"61'63","last_scrub_stamp":"2024-03-19T11:30:52.406723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:52.406723+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:03.655668+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.010852772,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.25","version":"56'66","reported_seq":51,"reported_epoch":85,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:04.077517+0000","last_change":"2024-03-19T11:31:04.077570+0000","last_active":"2024-03-19T11:31:04.077517+0000","last_peered":"2024-03-19T11:31:04.077517+0000","last_clean":"2024-03-19T11:31:04.077517+0000","last_became_active":"2024-03-19T11:30:53.084473+0000","last_became_peered":"2024-03-19T11:30:53.084473+0000","last_unstale":"2024-03-19T11:31:04.077517+0000","last_undegraded":"2024-03-19T11:31:04.077517+0000","last_fullsized":"2024-03-19T11:31:04.077517+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":6,"last_scrub":"47'64","last_scrub_stamp":"2024-03-19T11:30:28.487136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:28.487136+0000","objects_scrubbed":3,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0087584710000000003,"stat_sum":{"num_bytes":2403368,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":10662,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6","length":"1"},{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.24","version":"85'117","reported_seq":477,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.064686+0000","last_change":"2024-03-19T11:31:04.064961+0000","last_active":"2024-03-19T11:31:04.064686+0000","last_peered":"2024-03-19T11:31:04.064686+0000","last_clean":"2024-03-19T11:31:04.064686+0000","last_became_active":"2024-03-19T11:31:03.063705+0000","last_became_peered":"2024-03-19T11:31:03.063705+0000","last_unstale":"2024-03-19T11:31:04.064686+0000","last_undegraded":"2024-03-19T11:31:04.064686+0000","last_fullsized":"2024-03-19T11:31:04.064686+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":6,"last_scrub":"84'113","last_scrub_stamp":"2024-03-19T11:31:03.304881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:31:03.304881+0000","objects_scrubbed":5,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:26.481361+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.0099064849999999996,"stat_sum":{"num_bytes":10102023,"num_objects":4,"num_object_clones":2,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"},{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.23","version":"85'29","reported_seq":59,"reported_epoch":85,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:04.098447+0000","last_change":"2024-03-19T11:31:04.098447+0000","last_active":"2024-03-19T11:31:04.098447+0000","last_peered":"2024-03-19T11:31:04.098447+0000","last_clean":"2024-03-19T11:31:04.098447+0000","last_became_active":"2024-03-19T11:30:53.076092+0000","last_became_peered":"2024-03-19T11:30:53.076092+0000","last_unstale":"2024-03-19T11:31:04.098447+0000","last_undegraded":"2024-03-19T11:31:04.098447+0000","last_fullsized":"2024-03-19T11:31:04.098447+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":6,"last_scrub":"62'23","last_scrub_stamp":"2024-03-19T11:30:43.163509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:43.163509+0000","objects_scrubbed":2,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:46.525658+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.023454183999999999,"stat_sum":{"num_bytes":3728802,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2833035,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.22","version":"28'6","reported_seq":55,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-19T11:31:03.045458+0000","last_change":"2024-03-19T11:31:02.071949+0000","last_active":"2024-03-19T11:31:03.045458+0000","last_peered":"2024-03-19T11:31:03.045458+0000","last_clean":"2024-03-19T11:31:03.045458+0000","last_became_active":"2024-03-19T11:30:53.068010+0000","last_became_peered":"2024-03-19T11:30:53.068010+0000","last_unstale":"2024-03-19T11:31:03.045458+0000","last_undegraded":"2024-03-19T11:31:03.045458+0000","last_fullsized":"2024-03-19T11:31:03.045458+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":6,"last_scrub":"28'6","last_scrub_stamp":"2024-03-19T11:30:55.724671+0000","last_deep_scrub":"28'6","last_deep_scrub_stamp":"2024-03-19T11:30:55.724671+0000","last_clean_scrub_stamp":"2024-03-19T11:30:55.724671+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:00.824445+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0024412639999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.21","version":"83'125","reported_seq":500,"reported_epoch":85,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:04.100621+0000","last_change":"2024-03-19T11:31:04.100668+0000","last_active":"2024-03-19T11:31:04.100621+0000","last_peered":"2024-03-19T11:31:04.100621+0000","last_clean":"2024-03-19T11:31:04.100621+0000","last_became_active":"2024-03-19T11:31:00.069759+0000","last_became_peered":"2024-03-19T11:31:00.069759+0000","last_unstale":"2024-03-19T11:31:04.100621+0000","last_undegraded":"2024-03-19T11:31:04.100621+0000","last_fullsized":"2024-03-19T11:31:04.100621+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":6,"last_scrub":"62'96","last_scrub_stamp":"2024-03-19T11:30:43.400885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:43.400885+0000","objects_scrubbed":9,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:55.649626+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.050771269000000001,"stat_sum":{"num_bytes":15281651,"num_objects":9,"num_object_clones":7,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":48,"num_read_kb":51399,"num_write":18,"num_write_kb":4886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":22983003,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"2"},{"start":"14","length":"1"}]},{"pgid":"4.20","version":"56'46","reported_seq":47,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-19T11:31:03.052424+0000","last_change":"2024-03-19T11:31:02.086750+0000","last_active":"2024-03-19T11:31:03.052424+0000","last_peered":"2024-03-19T11:31:03.052424+0000","last_clean":"2024-03-19T11:31:03.052424+0000","last_became_active":"2024-03-19T11:30:53.067251+0000","last_became_peered":"2024-03-19T11:30:53.067251+0000","last_unstale":"2024-03-19T11:31:03.052424+0000","last_undegraded":"2024-03-19T11:31:03.052424+0000","last_fullsized":"2024-03-19T11:31:03.052424+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":6,"last_scrub":"56'46","last_scrub_stamp":"2024-03-19T11:30:41.767100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:41.767100+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:52.904341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0046137030000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.1f","version":"0'0","reported_seq":55,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-19T11:31:03.053712+0000","last_change":"2024-03-19T11:31:02.082397+0000","last_active":"2024-03-19T11:31:03.053712+0000","last_peered":"2024-03-19T11:31:03.053712+0000","last_clean":"2024-03-19T11:31:03.053712+0000","last_became_active":"2024-03-19T11:30:53.087211+0000","last_became_peered":"2024-03-19T11:30:53.087211+0000","last_unstale":"2024-03-19T11:31:03.053712+0000","last_undegraded":"2024-03-19T11:31:03.053712+0000","last_fullsized":"2024-03-19T11:31:03.053712+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:30:42.377963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.377963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:01.745397+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0021919130000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.1e","version":"83'69","reported_seq":62,"reported_epoch":83,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:02.087501+0000","last_change":"2024-03-19T11:31:02.087501+0000","last_active":"2024-03-19T11:31:02.087501+0000","last_peered":"2024-03-19T11:31:02.087501+0000","last_clean":"2024-03-19T11:31:02.087501+0000","last_became_active":"2024-03-19T11:30:53.084452+0000","last_became_peered":"2024-03-19T11:30:53.084452+0000","last_unstale":"2024-03-19T11:31:02.087501+0000","last_undegraded":"2024-03-19T11:31:02.087501+0000","last_fullsized":"2024-03-19T11:31:02.087501+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"64'62","last_scrub_stamp":"2024-03-19T11:30:42.778159+0000","last_deep_scrub":"64'62","last_deep_scrub_stamp":"2024-03-19T11:30:42.778159+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.778159+0000","objects_scrubbed":4,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:43.449616+0000","scrub_duration":31,"objects_trimmed":1,"snaptrim_duration":0.00063602400000000001,"stat_sum":{"num_bytes":3794928,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":7565,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3779426,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.1d","version":"82'66","reported_seq":61,"reported_epoch":85,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:04.050025+0000","last_change":"2024-03-19T11:31:02.078403+0000","last_active":"2024-03-19T11:31:04.050025+0000","last_peered":"2024-03-19T11:31:04.050025+0000","last_clean":"2024-03-19T11:31:04.050025+0000","last_became_active":"2024-03-19T11:30:53.067748+0000","last_became_peered":"2024-03-19T11:30:53.067748+0000","last_unstale":"2024-03-19T11:31:04.050025+0000","last_undegraded":"2024-03-19T11:31:04.050025+0000","last_fullsized":"2024-03-19T11:31:04.050025+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"61'59","last_scrub_stamp":"2024-03-19T11:30:42.601863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.601863+0000","objects_scrubbed":2,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:52.206103+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.022205709000000001,"stat_sum":{"num_bytes":6649813,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":61,"num_write":4,"num_write_kb":1341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2817390,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.1c","version":"85'35","reported_seq":69,"reported_epoch":85,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:04.055902+0000","last_change":"2024-03-19T11:31:04.055902+0000","last_active":"2024-03-19T11:31:04.055902+0000","last_peered":"2024-03-19T11:31:04.055902+0000","last_clean":"2024-03-19T11:31:04.055902+0000","last_became_active":"2024-03-19T11:30:53.082655+0000","last_became_peered":"2024-03-19T11:30:53.082655+0000","last_unstale":"2024-03-19T11:31:04.055902+0000","last_undegraded":"2024-03-19T11:31:04.055902+0000","last_fullsized":"2024-03-19T11:31:04.055902+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"62'31","last_scrub_stamp":"2024-03-19T11:30:42.771396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.771396+0000","objects_scrubbed":3,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:55.484122+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.00073920500000000005,"stat_sum":{"num_bytes":3210104,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":47464,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":5034309,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.1b","version":"84'78","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-19T11:31:03.165322+0000","last_change":"2024-03-19T11:31:03.165322+0000","last_active":"2024-03-19T11:31:03.165322+0000","last_peered":"2024-03-19T11:31:03.165322+0000","last_clean":"2024-03-19T11:31:03.165322+0000","last_became_active":"2024-03-19T11:31:03.056769+0000","last_became_peered":"2024-03-19T11:31:03.056769+0000","last_unstale":"2024-03-19T11:31:03.165322+0000","last_undegraded":"2024-03-19T11:31:03.165322+0000","last_fullsized":"2024-03-19T11:31:03.165322+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":5,"last_scrub":"62'62","last_scrub_stamp":"2024-03-19T11:30:42.181424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.181424+0000","objects_scrubbed":5,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:54.474706+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0047244899999999996,"stat_sum":{"num_bytes":14753221,"num_objects":6,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":47032,"num_write":5,"num_write_kb":1437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":17966556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.1a","version":"84'99","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-19T11:31:03.175029+0000","last_change":"2024-03-19T11:31:03.175029+0000","last_active":"2024-03-19T11:31:03.175029+0000","last_peered":"2024-03-19T11:31:03.175029+0000","last_clean":"2024-03-19T11:31:03.175029+0000","last_became_active":"2024-03-19T11:31:03.056520+0000","last_became_peered":"2024-03-19T11:31:03.056520+0000","last_unstale":"2024-03-19T11:31:03.175029+0000","last_undegraded":"2024-03-19T11:31:03.175029+0000","last_fullsized":"2024-03-19T11:31:03.175029+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":5,"last_scrub":"61'76","last_scrub_stamp":"2024-03-19T11:30:41.433501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:41.433501+0000","objects_scrubbed":5,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:59.309150+0000","scrub_duration":11,"objects_trimmed":1,"snaptrim_duration":0.0078747160000000004,"stat_sum":{"num_bytes":15479824,"num_objects":7,"num_object_clones":5,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":15,"num_read_kb":9625,"num_write":18,"num_write_kb":5314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":16291222,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.19","version":"83'111","reported_seq":471,"reported_epoch":85,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:04.098365+0000","last_change":"2024-03-19T11:31:04.098409+0000","last_active":"2024-03-19T11:31:04.098365+0000","last_peered":"2024-03-19T11:31:04.098365+0000","last_clean":"2024-03-19T11:31:04.098365+0000","last_became_active":"2024-03-19T11:30:53.076188+0000","last_became_peered":"2024-03-19T11:30:53.076188+0000","last_unstale":"2024-03-19T11:31:04.098365+0000","last_undegraded":"2024-03-19T11:31:04.098365+0000","last_fullsized":"2024-03-19T11:31:04.098365+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"41'37","last_scrub_stamp":"2024-03-19T11:30:27.705159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:27.705159+0000","objects_scrubbed":4,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:29.203476+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.047949758000000002,"stat_sum":{"num_bytes":18546999,"num_objects":12,"num_object_clones":8,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":88,"num_read_kb":101447,"num_write":38,"num_write_kb":6633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":21870393,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":11,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.18","version":"79'119","reported_seq":413,"reported_epoch":85,"state":"active+recovering+remapped","last_fresh":"2024-03-19T11:31:04.184848+0000","last_change":"2024-03-19T11:31:04.070548+0000","last_active":"2024-03-19T11:31:04.184848+0000","last_peered":"2024-03-19T11:31:04.184848+0000","last_clean":"2024-03-19T11:30:58.054172+0000","last_became_active":"2024-03-19T11:31:04.066044+0000","last_became_peered":"2024-03-19T11:31:04.066044+0000","last_unstale":"2024-03-19T11:31:04.184848+0000","last_undegraded":"2024-03-19T11:31:04.184848+0000","last_fullsized":"2024-03-19T11:31:04.184848+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":5,"last_scrub":"48'77","last_scrub_stamp":"2024-03-19T11:30:29.399762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:29.399762+0000","objects_scrubbed":6,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.011956031000000001,"stat_sum":{"num_bytes":21618293,"num_objects":9,"num_object_clones":7,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":40,"num_read_kb":37994,"num_write":17,"num_write_kb":4402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":14888652,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":7,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":1,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.17","version":"41'24","reported_seq":55,"reported_epoch":85,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:04.050331+0000","last_change":"2024-03-19T11:31:02.079786+0000","last_active":"2024-03-19T11:31:04.050331+0000","last_peered":"2024-03-19T11:31:04.050331+0000","last_clean":"2024-03-19T11:31:04.050331+0000","last_became_active":"2024-03-19T11:30:53.074270+0000","last_became_peered":"2024-03-19T11:30:53.074270+0000","last_unstale":"2024-03-19T11:31:04.050331+0000","last_undegraded":"2024-03-19T11:31:04.050331+0000","last_fullsized":"2024-03-19T11:31:04.050331+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"41'24","last_scrub_stamp":"2024-03-19T11:30:53.993428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:53.993428+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:54.020328+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.022407469999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.16","version":"83'99","reported_seq":52,"reported_epoch":83,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:02.090231+0000","last_change":"2024-03-19T11:31:02.077638+0000","last_active":"2024-03-19T11:31:02.090231+0000","last_peered":"2024-03-19T11:31:02.090231+0000","last_clean":"2024-03-19T11:31:02.090231+0000","last_became_active":"2024-03-19T11:30:59.074063+0000","last_became_peered":"2024-03-19T11:30:59.074063+0000","last_unstale":"2024-03-19T11:31:02.090231+0000","last_undegraded":"2024-03-19T11:31:02.090231+0000","last_fullsized":"2024-03-19T11:31:02.090231+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":5,"last_scrub":"49'56","last_scrub_stamp":"2024-03-19T11:30:29.212766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:29.212766+0000","objects_scrubbed":5,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:47.363425+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.096278618999999996,"stat_sum":{"num_bytes":12479433,"num_objects":7,"num_object_clones":4,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":95,"num_read_kb":91318,"num_write":20,"num_write_kb":6321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":24165549,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":7,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.15","version":"85'148","reported_seq":547,"reported_epoch":85,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:04.108361+0000","last_change":"2024-03-19T11:31:02.074320+0000","last_active":"2024-03-19T11:31:04.108361+0000","last_peered":"2024-03-19T11:31:04.108361+0000","last_clean":"2024-03-19T11:31:04.108361+0000","last_became_active":"2024-03-19T11:30:56.063100+0000","last_became_peered":"2024-03-19T11:30:56.063100+0000","last_unstale":"2024-03-19T11:31:04.108361+0000","last_undegraded":"2024-03-19T11:31:04.108361+0000","last_fullsized":"2024-03-19T11:31:04.108361+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":5,"last_scrub":"49'74","last_scrub_stamp":"2024-03-19T11:30:29.457542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:29.457542+0000","objects_scrubbed":7,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:56.479079+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.019267638,"stat_sum":{"num_bytes":22140490,"num_objects":13,"num_object_clones":8,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":128,"num_read_kb":146180,"num_write":32,"num_write_kb":8100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":18744415,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":8,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.14","version":"40'56","reported_seq":220,"reported_epoch":85,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:04.051467+0000","last_change":"2024-03-19T11:31:02.093175+0000","last_active":"2024-03-19T11:31:04.051467+0000","last_peered":"2024-03-19T11:31:04.051467+0000","last_clean":"2024-03-19T11:31:04.051467+0000","last_became_active":"2024-03-19T11:31:00.065648+0000","last_became_peered":"2024-03-19T11:31:00.065648+0000","last_unstale":"2024-03-19T11:31:04.051467+0000","last_undegraded":"2024-03-19T11:31:04.051467+0000","last_fullsized":"2024-03-19T11:31:04.051467+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":5,"last_scrub":"40'56","last_scrub_stamp":"2024-03-19T11:30:54.998282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:54.998282+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:09.561569+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.014000106999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"2"},{"start":"14","length":"1"}]},{"pgid":"4.13","version":"48'22","reported_seq":92,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-19T11:31:03.047937+0000","last_change":"2024-03-19T11:31:02.077145+0000","last_active":"2024-03-19T11:31:03.047937+0000","last_peered":"2024-03-19T11:31:03.047937+0000","last_clean":"2024-03-19T11:31:03.047937+0000","last_became_active":"2024-03-19T11:30:45.245367+0000","last_became_peered":"2024-03-19T11:30:45.245367+0000","last_unstale":"2024-03-19T11:31:03.047937+0000","last_undegraded":"2024-03-19T11:31:03.047937+0000","last_fullsized":"2024-03-19T11:31:03.047937+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":5,"last_scrub":"48'22","last_scrub_stamp":"2024-03-19T11:30:27.183294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:27.183294+0000","objects_scrubbed":2,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:48.571632+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.007358018,"stat_sum":{"num_bytes":5361367,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":31,"num_read_kb":30730,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":3001789,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.12","version":"28'6","reported_seq":33,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.055008+0000","last_change":"2024-03-19T11:31:04.055613+0000","last_active":"2024-03-19T11:31:04.055008+0000","last_peered":"2024-03-19T11:31:04.055008+0000","last_clean":"2024-03-19T11:31:04.055008+0000","last_became_active":"2024-03-19T11:31:00.064380+0000","last_became_peered":"2024-03-19T11:31:00.064380+0000","last_unstale":"2024-03-19T11:31:04.055008+0000","last_undegraded":"2024-03-19T11:31:04.055008+0000","last_fullsized":"2024-03-19T11:31:04.055008+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":5,"last_scrub":"28'6","last_scrub_stamp":"2024-03-19T11:30:42.390965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.390965+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077181699999999995,"stat_sum":{"num_bytes":1176474,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"},{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.11","version":"83'81","reported_seq":178,"reported_epoch":85,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:04.050088+0000","last_change":"2024-03-19T11:31:02.092082+0000","last_active":"2024-03-19T11:31:04.050088+0000","last_peered":"2024-03-19T11:31:04.050088+0000","last_clean":"2024-03-19T11:31:04.050088+0000","last_became_active":"2024-03-19T11:30:30.052184+0000","last_became_peered":"2024-03-19T11:30:30.052184+0000","last_unstale":"2024-03-19T11:31:04.050088+0000","last_undegraded":"2024-03-19T11:31:04.050088+0000","last_fullsized":"2024-03-19T11:31:04.050088+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":5,"last_scrub":"47'60","last_scrub_stamp":"2024-03-19T11:30:30.780348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:30.780348+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:31.151678+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.014285984999999999,"stat_sum":{"num_bytes":8084512,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":20,"num_read_kb":23163,"num_write":14,"num_write_kb":4028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.10","version":"85'34","reported_seq":340,"reported_epoch":85,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:04.100696+0000","last_change":"2024-03-19T11:31:04.100696+0000","last_active":"2024-03-19T11:31:04.100696+0000","last_peered":"2024-03-19T11:31:04.100696+0000","last_clean":"2024-03-19T11:31:04.100696+0000","last_became_active":"2024-03-19T11:31:00.065617+0000","last_became_peered":"2024-03-19T11:31:00.065617+0000","last_unstale":"2024-03-19T11:31:04.100696+0000","last_undegraded":"2024-03-19T11:31:04.100696+0000","last_fullsized":"2024-03-19T11:31:04.100696+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":5,"last_scrub":"50'30","last_scrub_stamp":"2024-03-19T11:30:28.787058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:28.787058+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:51.738618+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.0079239749999999998,"stat_sum":{"num_bytes":3462209,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":20772,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":7102596,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.054605+0000","last_change":"2024-03-19T11:30:03.276845+0000","last_active":"2024-03-19T11:31:04.054605+0000","last_peered":"2024-03-19T11:31:04.054605+0000","last_clean":"2024-03-19T11:31:04.054605+0000","last_became_active":"2024-03-19T11:30:03.276220+0000","last_became_peered":"2024-03-19T11:30:03.276220+0000","last_unstale":"2024-03-19T11:31:04.054605+0000","last_undegraded":"2024-03-19T11:31:04.054605+0000","last_fullsized":"2024-03-19T11:31:04.054605+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:07.472401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"59'92","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-19T11:31:03.065335+0000","last_change":"2024-03-19T11:31:03.059104+0000","last_active":"2024-03-19T11:31:03.065335+0000","last_peered":"2024-03-19T11:31:03.065335+0000","last_clean":"2024-03-19T11:31:03.065335+0000","last_became_active":"2024-03-19T11:31:03.052657+0000","last_became_peered":"2024-03-19T11:31:03.052657+0000","last_unstale":"2024-03-19T11:31:03.065335+0000","last_undegraded":"2024-03-19T11:31:03.065335+0000","last_fullsized":"2024-03-19T11:31:03.065335+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'92","last_scrub_stamp":"2024-03-19T11:30:41.351520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:41.351520+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:07.204778+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0052965570000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.055239+0000","last_change":"2024-03-19T11:31:00.067375+0000","last_active":"2024-03-19T11:31:04.055239+0000","last_peered":"2024-03-19T11:31:04.055239+0000","last_clean":"2024-03-19T11:31:04.055239+0000","last_became_active":"2024-03-19T11:31:00.067099+0000","last_became_peered":"2024-03-19T11:31:00.067099+0000","last_unstale":"2024-03-19T11:31:04.055239+0000","last_undegraded":"2024-03-19T11:31:04.055239+0000","last_fullsized":"2024-03-19T11:31:04.055239+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:06.356796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"79'65","reported_seq":347,"reported_epoch":85,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:04.050073+0000","last_change":"2024-03-19T11:31:02.078237+0000","last_active":"2024-03-19T11:31:04.050073+0000","last_peered":"2024-03-19T11:31:04.050073+0000","last_clean":"2024-03-19T11:31:04.050073+0000","last_became_active":"2024-03-19T11:30:38.065693+0000","last_became_peered":"2024-03-19T11:30:38.065693+0000","last_unstale":"2024-03-19T11:31:04.050073+0000","last_undegraded":"2024-03-19T11:31:04.050073+0000","last_fullsized":"2024-03-19T11:31:04.050073+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'57","last_scrub_stamp":"2024-03-19T11:30:43.792326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:43.792326+0000","objects_scrubbed":6,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:55.566850+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.022132443000000002,"stat_sum":{"num_bytes":9323261,"num_objects":6,"num_object_clones":5,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":11,"num_read_kb":16661,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":160,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.052066+0000","last_change":"2024-03-19T11:30:59.072585+0000","last_active":"2024-03-19T11:31:04.052066+0000","last_peered":"2024-03-19T11:31:04.052066+0000","last_clean":"2024-03-19T11:31:04.052066+0000","last_became_active":"2024-03-19T11:30:59.071925+0000","last_became_peered":"2024-03-19T11:30:59.071925+0000","last_unstale":"2024-03-19T11:31:04.052066+0000","last_undegraded":"2024-03-19T11:31:04.052066+0000","last_fullsized":"2024-03-19T11:31:04.052066+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:08.925537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"41'19","reported_seq":295,"reported_epoch":83,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:02.077549+0000","last_change":"2024-03-19T11:31:02.077529+0000","last_active":"2024-03-19T11:31:02.077549+0000","last_peered":"2024-03-19T11:31:02.077549+0000","last_clean":"2024-03-19T11:31:02.077549+0000","last_became_active":"2024-03-19T11:30:38.058860+0000","last_became_peered":"2024-03-19T11:30:38.058860+0000","last_unstale":"2024-03-19T11:31:02.077549+0000","last_undegraded":"2024-03-19T11:31:02.077549+0000","last_fullsized":"2024-03-19T11:31:02.077549+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'19","last_scrub_stamp":"2024-03-19T11:30:28.180772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:28.180772+0000","objects_scrubbed":1,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00380665,"stat_sum":{"num_bytes":1403368,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4077,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":153,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.055174+0000","last_change":"2024-03-19T11:30:55.366872+0000","last_active":"2024-03-19T11:31:04.055174+0000","last_peered":"2024-03-19T11:31:04.055174+0000","last_clean":"2024-03-19T11:31:04.055174+0000","last_became_active":"2024-03-19T11:29:46.540939+0000","last_became_peered":"2024-03-19T11:29:46.540939+0000","last_unstale":"2024-03-19T11:31:04.055174+0000","last_undegraded":"2024-03-19T11:31:04.055174+0000","last_fullsized":"2024-03-19T11:31:04.055174+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:30:55.366631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:30:55.366631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:10.640223+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"85'17","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.067823+0000","last_change":"2024-03-19T11:31:04.068099+0000","last_active":"2024-03-19T11:31:04.067823+0000","last_peered":"2024-03-19T11:31:04.067823+0000","last_clean":"2024-03-19T11:31:04.067823+0000","last_became_active":"2024-03-19T11:31:00.069200+0000","last_became_peered":"2024-03-19T11:31:00.069200+0000","last_unstale":"2024-03-19T11:31:04.067823+0000","last_undegraded":"2024-03-19T11:31:04.067823+0000","last_fullsized":"2024-03-19T11:31:04.067823+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'10","last_scrub_stamp":"2024-03-19T11:30:54.429030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:54.429030+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.013022977999999999,"stat_sum":{"num_bytes":2742425,"num_objects":1,"num_object_clones":1,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":10173,"num_write":4,"num_write_kb":1071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2800509,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"},{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"3.3","version":"77'99","reported_seq":319,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-19T11:30:57.062257+0000","last_change":"2024-03-19T11:30:30.271391+0000","last_active":"2024-03-19T11:30:57.062257+0000","last_peered":"2024-03-19T11:30:57.062257+0000","last_clean":"2024-03-19T11:30:57.062257+0000","last_became_active":"2024-03-19T11:30:30.058963+0000","last_became_peered":"2024-03-19T11:30:30.058963+0000","last_unstale":"2024-03-19T11:30:57.062257+0000","last_undegraded":"2024-03-19T11:30:57.062257+0000","last_fullsized":"2024-03-19T11:30:57.062257+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:11.045052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53399952,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":74,"num_read_kb":3737,"num_write":107,"num_write_kb":58435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":22537669,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":144,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.068842+0000","last_change":"2024-03-19T11:29:46.321255+0000","last_active":"2024-03-19T11:31:04.068842+0000","last_peered":"2024-03-19T11:31:04.068842+0000","last_clean":"2024-03-19T11:31:04.068842+0000","last_became_active":"2024-03-19T11:29:46.320679+0000","last_became_peered":"2024-03-19T11:29:46.320679+0000","last_unstale":"2024-03-19T11:31:04.068842+0000","last_undegraded":"2024-03-19T11:31:04.068842+0000","last_fullsized":"2024-03-19T11:31:04.068842+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:14.793563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"85'132","reported_seq":508,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.061158+0000","last_change":"2024-03-19T11:31:04.061533+0000","last_active":"2024-03-19T11:31:04.061158+0000","last_peered":"2024-03-19T11:31:04.061158+0000","last_clean":"2024-03-19T11:31:04.061158+0000","last_became_active":"2024-03-19T11:31:03.054624+0000","last_became_peered":"2024-03-19T11:31:03.054624+0000","last_unstale":"2024-03-19T11:31:04.061158+0000","last_undegraded":"2024-03-19T11:31:04.061158+0000","last_fullsized":"2024-03-19T11:31:04.061158+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'119","last_scrub_stamp":"2024-03-19T11:30:52.512203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:52.512203+0000","objects_scrubbed":7,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:02.677965+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.0068329189999999998,"stat_sum":{"num_bytes":14273317,"num_objects":6,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":70,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"},{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"3.0","version":"85'100","reported_seq":247,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.068253+0000","last_change":"2024-03-19T11:29:57.550320+0000","last_active":"2024-03-19T11:31:04.068253+0000","last_peered":"2024-03-19T11:31:04.068253+0000","last_clean":"2024-03-19T11:31:04.068253+0000","last_became_active":"2024-03-19T11:29:57.549928+0000","last_became_peered":"2024-03-19T11:29:57.549928+0000","last_unstale":"2024-03-19T11:31:04.068253+0000","last_undegraded":"2024-03-19T11:31:04.068253+0000","last_fullsized":"2024-03-19T11:31:04.068253+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28242922,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":80,"num_read_kb":3042,"num_write":105,"num_write_kb":39820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":146,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.053846+0000","last_change":"2024-03-19T11:29:46.319811+0000","last_active":"2024-03-19T11:31:04.053846+0000","last_peered":"2024-03-19T11:31:04.053846+0000","last_clean":"2024-03-19T11:31:04.053846+0000","last_became_active":"2024-03-19T11:29:46.319334+0000","last_became_peered":"2024-03-19T11:29:46.319334+0000","last_unstale":"2024-03-19T11:31:04.053846+0000","last_undegraded":"2024-03-19T11:31:04.053846+0000","last_fullsized":"2024-03-19T11:31:04.053846+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:04.320463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"61'40","reported_seq":33,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.065031+0000","last_change":"2024-03-19T11:31:04.065328+0000","last_active":"2024-03-19T11:31:04.065031+0000","last_peered":"2024-03-19T11:31:04.065031+0000","last_clean":"2024-03-19T11:31:04.065031+0000","last_became_active":"2024-03-19T11:31:00.069809+0000","last_became_peered":"2024-03-19T11:31:00.069809+0000","last_unstale":"2024-03-19T11:31:04.065031+0000","last_undegraded":"2024-03-19T11:31:04.065031+0000","last_fullsized":"2024-03-19T11:31:04.065031+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'40","last_scrub_stamp":"2024-03-19T11:30:56.505586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:56.505586+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:09.446385+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0098007430000000006,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"},{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"3.1","version":"82'85","reported_seq":289,"reported_epoch":82,"state":"peering","last_fresh":"2024-03-19T11:31:01.114104+0000","last_change":"2024-03-19T11:31:02.076669+0000","last_active":"2024-03-19T11:31:02.076669+0000","last_peered":"2024-03-19T11:31:01.114104+0000","last_clean":"2024-03-19T11:31:01.114104+0000","last_became_active":"2024-03-19T11:30:26.927069+0000","last_became_peered":"2024-03-19T11:30:26.927069+0000","last_unstale":"2024-03-19T11:31:01.114104+0000","last_undegraded":"2024-03-19T11:31:01.114104+0000","last_fullsized":"2024-03-19T11:31:01.114104+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42832129,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":71,"num_read_kb":6652,"num_write":89,"num_write_kb":49391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":16477750,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,4],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":155,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.052299+0000","last_change":"2024-03-19T11:31:00.070027+0000","last_active":"2024-03-19T11:31:04.052299+0000","last_peered":"2024-03-19T11:31:04.052299+0000","last_clean":"2024-03-19T11:31:04.052299+0000","last_became_active":"2024-03-19T11:31:00.068989+0000","last_became_peered":"2024-03-19T11:31:00.068989+0000","last_unstale":"2024-03-19T11:31:04.052299+0000","last_undegraded":"2024-03-19T11:31:04.052299+0000","last_fullsized":"2024-03-19T11:31:04.052299+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:11.599600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"77'68","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-19T11:31:02.085910+0000","last_change":"2024-03-19T11:31:02.085910+0000","last_active":"2024-03-19T11:31:02.085910+0000","last_peered":"2024-03-19T11:31:02.085910+0000","last_clean":"2024-03-19T11:31:02.085910+0000","last_became_active":"2024-03-19T11:30:59.075057+0000","last_became_peered":"2024-03-19T11:30:59.075057+0000","last_unstale":"2024-03-19T11:31:02.085910+0000","last_undegraded":"2024-03-19T11:31:02.085910+0000","last_fullsized":"2024-03-19T11:31:02.085910+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'65","last_scrub_stamp":"2024-03-19T11:30:53.385414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:53.385414+0000","objects_scrubbed":3,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:05.375039+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.009635546,"stat_sum":{"num_bytes":7189731,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4209633,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"3.2","version":"85'84","reported_seq":239,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.049868+0000","last_change":"2024-03-19T11:29:57.549277+0000","last_active":"2024-03-19T11:31:04.049868+0000","last_peered":"2024-03-19T11:31:04.049868+0000","last_clean":"2024-03-19T11:31:04.049868+0000","last_became_active":"2024-03-19T11:29:57.548946+0000","last_became_peered":"2024-03-19T11:29:57.548946+0000","last_unstale":"2024-03-19T11:31:04.049868+0000","last_undegraded":"2024-03-19T11:31:04.049868+0000","last_fullsized":"2024-03-19T11:31:04.049868+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:08.380405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39345556,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":76,"num_read_kb":8248,"num_write":94,"num_write_kb":41080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":144,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.067990+0000","last_change":"2024-03-19T11:29:46.322491+0000","last_active":"2024-03-19T11:31:04.067990+0000","last_peered":"2024-03-19T11:31:04.067990+0000","last_clean":"2024-03-19T11:31:04.067990+0000","last_became_active":"2024-03-19T11:29:46.321760+0000","last_became_peered":"2024-03-19T11:29:46.321760+0000","last_unstale":"2024-03-19T11:31:04.067990+0000","last_undegraded":"2024-03-19T11:31:04.067990+0000","last_fullsized":"2024-03-19T11:31:04.067990+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:04.561915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":329,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.052318+0000","last_change":"2024-03-19T11:31:03.367805+0000","last_active":"2024-03-19T11:31:04.052318+0000","last_peered":"2024-03-19T11:31:04.052318+0000","last_clean":"2024-03-19T11:31:04.052318+0000","last_became_active":"2024-03-19T11:30:59.073709+0000","last_became_peered":"2024-03-19T11:30:59.073709+0000","last_unstale":"2024-03-19T11:31:04.052318+0000","last_undegraded":"2024-03-19T11:31:04.052318+0000","last_fullsized":"2024-03-19T11:31:04.052318+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2024-03-19T11:31:03.367669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:43.166618+0000","last_clean_scrub_stamp":"2024-03-19T11:31:03.367669+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:29.977573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"84'103","reported_seq":480,"reported_epoch":84,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:03.169808+0000","last_change":"2024-03-19T11:31:03.168375+0000","last_active":"2024-03-19T11:31:03.169808+0000","last_peered":"2024-03-19T11:31:03.169808+0000","last_clean":"2024-03-19T11:31:03.169808+0000","last_became_active":"2024-03-19T11:31:03.065344+0000","last_became_peered":"2024-03-19T11:31:03.065344+0000","last_unstale":"2024-03-19T11:31:03.169808+0000","last_undegraded":"2024-03-19T11:31:03.169808+0000","last_fullsized":"2024-03-19T11:31:03.169808+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'81","last_scrub_stamp":"2024-03-19T11:30:52.360396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:52.360396+0000","objects_scrubbed":6,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:11.500650+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.032057466999999999,"stat_sum":{"num_bytes":10838384,"num_objects":8,"num_object_clones":6,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":13,"num_read_kb":12349,"num_write":8,"num_write_kb":1273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":10558635,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"c","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.8","version":"62'71","reported_seq":359,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.077614+0000","last_change":"2024-03-19T11:31:04.078135+0000","last_active":"2024-03-19T11:31:04.077614+0000","last_peered":"2024-03-19T11:31:04.077614+0000","last_clean":"2024-03-19T11:31:04.077614+0000","last_became_active":"2024-03-19T11:30:45.251283+0000","last_became_peered":"2024-03-19T11:30:45.251283+0000","last_unstale":"2024-03-19T11:31:04.077614+0000","last_undegraded":"2024-03-19T11:31:04.077614+0000","last_fullsized":"2024-03-19T11:31:04.077614+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'71","last_scrub_stamp":"2024-03-19T11:30:53.381386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:53.381386+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:18.275213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010232903999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6","length":"1"},{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.9","version":"63'55","reported_seq":384,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-19T11:31:03.046677+0000","last_change":"2024-03-19T11:31:02.070296+0000","last_active":"2024-03-19T11:31:03.046677+0000","last_peered":"2024-03-19T11:31:03.046677+0000","last_clean":"2024-03-19T11:31:03.046677+0000","last_became_active":"2024-03-19T11:31:00.069789+0000","last_became_peered":"2024-03-19T11:31:00.069789+0000","last_unstale":"2024-03-19T11:31:03.046677+0000","last_undegraded":"2024-03-19T11:31:03.046677+0000","last_fullsized":"2024-03-19T11:31:03.046677+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'55","last_scrub_stamp":"2024-03-19T11:30:54.706235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:54.706235+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:01.278994+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00082106599999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.a","version":"62'77","reported_seq":375,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-19T11:31:03.053296+0000","last_change":"2024-03-19T11:31:02.084460+0000","last_active":"2024-03-19T11:31:03.053296+0000","last_peered":"2024-03-19T11:31:03.053296+0000","last_clean":"2024-03-19T11:31:03.053296+0000","last_became_active":"2024-03-19T11:30:45.256124+0000","last_became_peered":"2024-03-19T11:30:45.256124+0000","last_unstale":"2024-03-19T11:31:03.053296+0000","last_undegraded":"2024-03-19T11:31:03.053296+0000","last_fullsized":"2024-03-19T11:31:03.053296+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'77","last_scrub_stamp":"2024-03-19T11:30:55.291823+0000","last_deep_scrub":"62'77","last_deep_scrub_stamp":"2024-03-19T11:30:55.291823+0000","last_clean_scrub_stamp":"2024-03-19T11:30:55.291823+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:11.588174+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00325363,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.b","version":"70'62","reported_seq":375,"reported_epoch":83,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:02.077320+0000","last_change":"2024-03-19T11:31:02.077300+0000","last_active":"2024-03-19T11:31:02.077320+0000","last_peered":"2024-03-19T11:31:02.077320+0000","last_clean":"2024-03-19T11:31:02.077320+0000","last_became_active":"2024-03-19T11:30:45.261756+0000","last_became_peered":"2024-03-19T11:30:45.261756+0000","last_unstale":"2024-03-19T11:31:02.077320+0000","last_undegraded":"2024-03-19T11:31:02.077320+0000","last_fullsized":"2024-03-19T11:31:02.077320+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'62","last_scrub_stamp":"2024-03-19T11:30:56.095039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:56.095039+0000","objects_scrubbed":3,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:18.760680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063617100000000005,"stat_sum":{"num_bytes":5215740,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.c","version":"56'29","reported_seq":10,"reported_epoch":84,"state":"clean+premerge+peered","last_fresh":"2024-03-19T11:31:03.063714+0000","last_change":"2024-03-19T11:31:03.058030+0000","last_active":"2024-03-19T11:31:01.070481+0000","last_peered":"2024-03-19T11:31:03.063714+0000","last_clean":"2024-03-19T11:31:03.063714+0000","last_became_active":"2024-03-19T11:30:56.061560+0000","last_became_peered":"2024-03-19T11:31:03.056794+0000","last_unstale":"2024-03-19T11:31:03.063714+0000","last_undegraded":"2024-03-19T11:31:03.063714+0000","last_fullsized":"2024-03-19T11:31:03.063714+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'29","last_scrub_stamp":"2024-03-19T11:30:53.743385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:53.743385+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:02.142968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.021151900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.d","version":"85'82","reported_seq":393,"reported_epoch":85,"state":"active+clean+remapped+snaptrim","last_fresh":"2024-03-19T11:31:04.184330+0000","last_change":"2024-03-19T11:31:04.182796+0000","last_active":"2024-03-19T11:31:04.184330+0000","last_peered":"2024-03-19T11:31:04.184330+0000","last_clean":"2024-03-19T11:31:04.184330+0000","last_became_active":"2024-03-19T11:31:04.064814+0000","last_became_peered":"2024-03-19T11:31:04.064814+0000","last_unstale":"2024-03-19T11:31:04.184330+0000","last_undegraded":"2024-03-19T11:31:04.184330+0000","last_fullsized":"2024-03-19T11:31:04.184330+0000","mapping_epoch":84,"log_start":"70'71","ondisk_log_start":"70'71","created":21,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'73","last_scrub_stamp":"2024-03-19T11:30:53.519178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:53.519178+0000","objects_scrubbed":6,"log_size":11,"log_dups_size":50,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:15.690636+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0030257930000000001,"stat_sum":{"num_bytes":10067275,"num_objects":6,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":9,"num_read_kb":11335,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":8146998,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[1,2],"avail_no_missing":["1","2","4"],"object_location_counts":[{"shards":"1,2","objects":6}],"blocked_by":[],"up_primary":4,"acting_primary":1,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.e","version":"79'78","reported_seq":352,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-19T11:31:03.046968+0000","last_change":"2024-03-19T11:31:02.074204+0000","last_active":"2024-03-19T11:31:03.046968+0000","last_peered":"2024-03-19T11:31:03.046968+0000","last_clean":"2024-03-19T11:31:03.046968+0000","last_became_active":"2024-03-19T11:30:38.060487+0000","last_became_peered":"2024-03-19T11:30:38.060487+0000","last_unstale":"2024-03-19T11:31:03.046968+0000","last_undegraded":"2024-03-19T11:31:03.046968+0000","last_fullsized":"2024-03-19T11:31:03.046968+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'65","last_scrub_stamp":"2024-03-19T11:30:43.824620+0000","last_deep_scrub":"65'65","last_deep_scrub_stamp":"2024-03-19T11:30:43.824620+0000","last_clean_scrub_stamp":"2024-03-19T11:30:43.824620+0000","objects_scrubbed":4,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:48.641497+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0.0046356100000000001,"stat_sum":{"num_bytes":12990971,"num_objects":6,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":21,"num_read_kb":22723,"num_write":8,"num_write_kb":2301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.f","version":"0'0","reported_seq":256,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.065434+0000","last_change":"2024-03-19T11:31:04.065959+0000","last_active":"2024-03-19T11:31:04.065434+0000","last_peered":"2024-03-19T11:31:04.065434+0000","last_clean":"2024-03-19T11:31:04.065434+0000","last_became_active":"2024-03-19T11:30:38.050919+0000","last_became_peered":"2024-03-19T11:30:38.050919+0000","last_unstale":"2024-03-19T11:31:04.065434+0000","last_undegraded":"2024-03-19T11:31:04.065434+0000","last_fullsized":"2024-03-19T11:31:04.065434+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:30:43.411893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:43.411893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:54.719455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010140278000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"},{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]}],"pool_stats":[{"poolid":4,"num_pg":44,"stat_sum":{"num_bytes":275155131,"num_objects":142,"num_object_clones":91,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":139,"num_whiteouts":6,"num_read":669,"num_read_kb":715159,"num_write":198,"num_write_kb":50396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":200107110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":93,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":402448384,"data_stored":457616802,"data_compressed":2587576,"data_compressed_allocated":1728512,"data_compressed_original":5898240,"omap_allocated":0,"internal_metadata":0},"log_size":2787,"ondisk_log_size":2787,"up":88,"acting":88,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":163820559,"num_objects":143,"num_object_clones":0,"num_object_copies":286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":143,"num_whiteouts":0,"num_read":301,"num_read_kb":21679,"num_write":395,"num_write_kb":188726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":39015419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":165703680,"data_stored":179494984,"data_compressed":12248719,"data_compressed_allocated":13119488,"data_compressed_original":27238400,"omap_allocated":0,"internal_metadata":0},"log_size":368,"ondisk_log_size":368,"up":8,"acting":8,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":1586784,"data_compressed":19832,"data_compressed_allocated":98304,"data_compressed_original":1560576,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542164,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83364,"kb_used_data":52184,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94288476,"statfs":{"total":96636764160,"available":96551399424,"internally_reserved":0,"allocated":53436416,"data_stored":62427174,"data_compressed":227710,"data_compressed_allocated":151552,"data_compressed_original":1101824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542164,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84092,"kb_used_data":52908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94287748,"statfs":{"total":96636764160,"available":96550653952,"internally_reserved":0,"allocated":54177792,"data_stored":62533221,"data_compressed":1917086,"data_compressed_allocated":1929216,"data_compressed_original":4210688,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542164,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":129452,"kb_used_data":102432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94242388,"statfs":{"total":96636764160,"available":96504205312,"internally_reserved":0,"allocated":104890368,"data_stored":120523535,"data_compressed":2532072,"data_compressed_allocated":2379776,"data_compressed_original":5521408,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542163,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79208,"kb_used_data":52184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94292632,"statfs":{"total":96636764160,"available":96555655168,"internally_reserved":0,"allocated":53436416,"data_stored":60429347,"data_compressed":1749130,"data_compressed_allocated":1527808,"data_compressed_original":4194304,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542163,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":153924,"kb_used_data":113028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94217916,"statfs":{"total":96636764160,"available":96479145984,"internally_reserved":0,"allocated":115740672,"data_stored":123349286,"data_compressed":1407043,"data_compressed_allocated":1449984,"data_compressed_original":3211264,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95020,"kb_used_data":68000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94276820,"statfs":{"total":96636764160,"available":96539463680,"internally_reserved":0,"allocated":69632000,"data_stored":81512192,"data_compressed":2377281,"data_compressed_allocated":2482176,"data_compressed_original":4935680,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574867,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46512,"kb_used_data":15328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325328,"statfs":{"total":96636764160,"available":96589135872,"internally_reserved":0,"allocated":15695872,"data_stored":17201968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574867,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":125852,"kb_used_data":94668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94245988,"statfs":{"total":96636764160,"available":96507891712,"internally_reserved":0,"allocated":96940032,"data_stored":102441426,"data_compressed":4177120,"data_compressed_allocated":4493312,"data_compressed_original":9555968,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6884,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6884,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6064,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53927936,"data_stored":58198011,"data_compressed":3980813,"data_compressed_allocated":4255744,"data_compressed_original":8638464,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14168064,"data_stored":15397438,"data_compressed":898614,"data_compressed_allocated":970752,"data_compressed_original":2228224,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14168064,"data_stored":15397438,"data_compressed":898614,"data_compressed_allocated":970752,"data_compressed_original":2228224,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":22126592,"data_stored":23997131,"data_compressed":1849977,"data_compressed_allocated":1974272,"data_compressed_original":3883008,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11079680,"data_stored":12191862,"data_compressed":660988,"data_compressed_allocated":712704,"data_compressed_original":1835008,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":19947520,"data_stored":21724875,"data_compressed":1695440,"data_compressed_allocated":1810432,"data_compressed_original":3620864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":30285824,"data_stored":32588229,"data_compressed":2264273,"data_compressed_allocated":2424832,"data_compressed_original":4804608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":38711296,"data_stored":39908792,"data_compressed":189423,"data_compressed_allocated":204800,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90685440,"data_stored":97008266,"data_compressed":390047,"data_compressed_allocated":352256,"data_compressed_original":720896,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":34271232,"data_stored":39685709,"data_compressed":732134,"data_compressed_allocated":430080,"data_compressed_original":1703936,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":78860288,"data_stored":93060511,"data_compressed":682095,"data_compressed_allocated":405504,"data_compressed_original":1638400,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":43638784,"data_stored":47398962,"data_compressed":37577,"data_compressed_allocated":40960,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":29360128,"data_stored":35596983,"data_compressed":221646,"data_compressed_allocated":118784,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":38412288,"data_stored":48442122,"data_compressed":113008,"data_compressed_allocated":57344,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":48508928,"data_stored":56515457,"data_compressed":221646,"data_compressed_allocated":118784,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T11:31:05.818 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-19T11:31:05.821 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:05.831+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:05.829 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:05.829 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:05.843 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-19T11:31:05.922 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:05.931+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:05.973 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:05.974 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:05.987 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-19T11:31:06.023 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:06.029+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:06.043 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-03-19T11:31:06.044 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:06.053+0000 7fa5e08db640 -1 osd.3 86 handle_osd_map injecting map crc failure 2024-03-19T11:31:06.054 INFO:tasks.rados.rados.0.smithi038.stdout:1219: done (4 left) 2024-03-19T11:31:06.054 INFO:tasks.rados.rados.0.smithi038.stdout:1220: read oid 41 snap -1 2024-03-19T11:31:06.054 INFO:tasks.rados.rados.0.smithi038.stdout:1220: expect (ObjNum 170 snap 17 seq_num 170) 2024-03-19T11:31:06.054 INFO:tasks.rados.rados.0.smithi038.stdout:1221: read oid 27 snap -1 2024-03-19T11:31:06.054 INFO:tasks.rados.rados.0.smithi038.stdout:1221: expect (ObjNum 192 snap 22 seq_num 192) 2024-03-19T11:31:06.054 INFO:tasks.rados.rados.0.smithi038.stdout:1222: read oid 45 snap -1 2024-03-19T11:31:06.054 INFO:tasks.rados.rados.0.smithi038.stdout:1222: expect (ObjNum 148 snap 15 seq_num 148) 2024-03-19T11:31:06.055 INFO:tasks.rados.rados.0.smithi038.stdout:1223: write oid 35 current snap is 22 2024-03-19T11:31:06.055 INFO:tasks.rados.rados.0.smithi038.stdout:1223: seq_num 200 ranges {561400=474978,1820281=519723,2859422=457579} 2024-03-19T11:31:06.061 INFO:tasks.rados.rados.0.smithi038.stdout:1223: writing smithi03829856-35 from 561400 to 1036378 tid 1 2024-03-19T11:31:06.069 INFO:tasks.rados.rados.0.smithi038.stdout:1223: writing smithi03829856-35 from 1820281 to 2340004 tid 2 2024-03-19T11:31:06.075 INFO:tasks.rados.rados.0.smithi038.stdout:1223: writing smithi03829856-35 from 2859422 to 3317001 tid 3 2024-03-19T11:31:06.076 INFO:tasks.rados.rados.0.smithi038.stdout:1215: finishing write tid 3 to smithi03829856-43 2024-03-19T11:31:06.076 INFO:tasks.rados.rados.0.smithi038.stdout:1215: finishing write tid 4 to smithi03829856-43 2024-03-19T11:31:06.076 INFO:tasks.rados.rados.0.smithi038.stdout:1215: finishing write tid 5 to smithi03829856-43 2024-03-19T11:31:06.076 INFO:tasks.rados.rados.0.smithi038.stdout:1215: oid 43 updating version 0 to 126 2024-03-19T11:31:06.076 INFO:tasks.rados.rados.0.smithi038.stdout:1215: oid 43 updating version 126 to 128 2024-03-19T11:31:06.076 INFO:tasks.rados.rados.0.smithi038.stdout:1215: oid 43 updating version 128 to 129 2024-03-19T11:31:06.076 INFO:tasks.rados.rados.0.smithi038.stdout:1215: oid 43 version 129 is already newer than 127 2024-03-19T11:31:06.076 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 43 v 129 (ObjNum 198 snap 22 seq_num 198) dirty exists 2024-03-19T11:31:06.076 INFO:tasks.rados.rados.0.smithi038.stdout:1215: left oid 43 (ObjNum 198 snap 22 seq_num 198) 2024-03-19T11:31:06.076 INFO:tasks.rados.rados.0.smithi038.stdout:1217: expect (ObjNum 185 snap 20 seq_num 185) 2024-03-19T11:31:06.079 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:06.079+0000 7f923da1c640 -1 osd.7 86 handle_osd_map injecting map crc failure 2024-03-19T11:31:06.124 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:06.133+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:06.175 INFO:tasks.rados.rados.0.smithi038.stdout:1218: finishing set_chunk to oid 17 2024-03-19T11:31:06.175 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 17 v 136 (ObjNum 183 snap 20 seq_num 183) dirty exists 2024-03-19T11:31:06.175 INFO:tasks.rados.rados.0.smithi038.stdout:1220: expect (ObjNum 170 snap 17 seq_num 170) 2024-03-19T11:31:06.176 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:06.176 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:06.180 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:31:06.181 INFO:teuthology.orchestra.run.smithi038.stdout:{"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","health":{"status":"HEALTH_WARN","checks":{"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 1 pg inactive, 1 pg peering","count":2},"muted":false},"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"1 pool(s) do not have an application enabled","count":1},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":92,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":87,"num_osds":8,"num_up_osds":8,"osd_up_since":1710847783,"num_in_osds":8,"osd_in_since":1710847775,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":26},{"state_name":"peering","count":10},{"state_name":"active+clean+snaptrim_wait","count":9},{"state_name":"active+clean+snaptrim","count":9},{"state_name":"unknown","count":2},{"state_name":"clean+premerge+peered","count":1}],"num_pgs":57,"num_pools":4,"num_objects":278,"data_bytes":424358023,"bytes_used":820072448,"bytes_avail":772274040832,"bytes_total":773094113280,"unknown_pgs_ratio":0.035087719559669495,"inactive_pgs_ratio":0.19298245012760162,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":13037894,"recovering_keys_per_sec":0,"num_objects_recovered":40,"num_bytes_recovered":52168844,"num_keys_recovered":0,"read_bytes_sec":4140197,"write_bytes_sec":1157760,"read_op_per_sec":4,"write_op_per_sec":4},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-03-19T11:30:46.147482+0000","services":{}},"progress_events":{}} 2024-03-19T11:31:06.191 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-19T11:31:06.198 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout 2024-03-19T11:31:06.199 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T11:31:06.239 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:06.251+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:06.264 INFO:tasks.rados.rados.0.smithi038.stdout:1222: expect (ObjNum 148 snap 15 seq_num 148) 2024-03-19T11:31:06.313 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:06.313 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:06.327 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-19T11:31:06.327 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:06.333+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:06.407 INFO:tasks.rados.rados.0.smithi038.stdout:1221: expect (ObjNum 192 snap 22 seq_num 192) 2024-03-19T11:31:06.427 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:06.439+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:06.494 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:06.494 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:06.508 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-19T11:31:06.528 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:06.537+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:06.555 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:31:06.555 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-03-19T11:31:06.571 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":122,"stamp":"2024-03-19T11:31:06.154426+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":439566058,"num_objects":287,"num_object_clones":91,"num_object_copies":574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":284,"num_whiteouts":6,"num_read":1086,"num_read_kb":736938,"num_write":752,"num_write_kb":241538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":220,"num_bytes_recovered":240180385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":93,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3272,"ondisk_log_size":3272,"up":114,"acting":114,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":124,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":797424,"kb_used_data":550732,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":754177296,"statfs":{"total":773094113280,"available":772277551104,"internally_reserved":0,"allocated":563949568,"data_stored":630418149,"data_compressed":14387442,"data_compressed_allocated":14413824,"data_compressed_original":32731136,"omap_allocated":12751,"internal_metadata":246795825},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13842679,"num_objects":7,"num_object_clones":-2,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":-8,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":69,"num_read_kb":66097,"num_write":44,"num_write_kb":11101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":108743946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001232"},"pg_stats":[{"pgid":"4.2b","version":"61'60","reported_seq":375,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-19T11:31:02.077571+0000","last_change":"2024-03-19T11:31:02.082219+0000","last_active":"2024-03-19T11:31:02.077571+0000","last_peered":"2024-03-19T11:31:02.077571+0000","last_clean":"2024-03-19T11:31:02.077571+0000","last_became_active":"2024-03-19T11:30:45.263076+0000","last_became_peered":"2024-03-19T11:30:45.263076+0000","last_unstale":"2024-03-19T11:31:02.077571+0000","last_undegraded":"2024-03-19T11:31:02.077571+0000","last_fullsized":"2024-03-19T11:31:02.077571+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":6,"last_scrub":"61'60","last_scrub_stamp":"2024-03-19T11:30:55.143981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:55.143981+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:59.980040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00639216,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.2a","version":"83'94","reported_seq":400,"reported_epoch":84,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:03.052847+0000","last_change":"2024-03-19T11:31:02.085627+0000","last_active":"2024-03-19T11:31:03.052847+0000","last_peered":"2024-03-19T11:31:03.052847+0000","last_clean":"2024-03-19T11:31:03.052847+0000","last_became_active":"2024-03-19T11:30:45.260578+0000","last_became_peered":"2024-03-19T11:30:45.260578+0000","last_unstale":"2024-03-19T11:31:03.052847+0000","last_undegraded":"2024-03-19T11:31:03.052847+0000","last_fullsized":"2024-03-19T11:31:03.052847+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":6,"last_scrub":"74'89","last_scrub_stamp":"2024-03-19T11:30:53.312593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:53.312593+0000","objects_scrubbed":6,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:07.058558+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.015736934000000001,"stat_sum":{"num_bytes":10618356,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":781,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.29","version":"83'63","reported_seq":409,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-19T11:31:03.046373+0000","last_change":"2024-03-19T11:31:02.085998+0000","last_active":"2024-03-19T11:31:03.046373+0000","last_peered":"2024-03-19T11:31:03.046373+0000","last_clean":"2024-03-19T11:31:03.046373+0000","last_became_active":"2024-03-19T11:31:00.070287+0000","last_became_peered":"2024-03-19T11:31:00.070287+0000","last_unstale":"2024-03-19T11:31:03.046373+0000","last_undegraded":"2024-03-19T11:31:03.046373+0000","last_fullsized":"2024-03-19T11:31:03.046373+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":6,"last_scrub":"71'60","last_scrub_stamp":"2024-03-19T11:30:52.748479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:52.748479+0000","objects_scrubbed":3,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:56.409127+0000","scrub_duration":7,"objects_trimmed":1,"snaptrim_duration":0.016516072,"stat_sum":{"num_bytes":4387250,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":377,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":5842489,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.28","version":"85'98","reported_seq":383,"reported_epoch":85,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:04.086552+0000","last_change":"2024-03-19T11:31:04.078290+0000","last_active":"2024-03-19T11:31:04.086552+0000","last_peered":"2024-03-19T11:31:04.086552+0000","last_clean":"2024-03-19T11:31:04.086552+0000","last_became_active":"2024-03-19T11:30:45.257600+0000","last_became_peered":"2024-03-19T11:30:45.257600+0000","last_unstale":"2024-03-19T11:31:04.086552+0000","last_undegraded":"2024-03-19T11:31:04.086552+0000","last_fullsized":"2024-03-19T11:31:04.086552+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":6,"last_scrub":"71'85","last_scrub_stamp":"2024-03-19T11:30:52.343454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:52.343454+0000","objects_scrubbed":7,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:16.280349+0000","scrub_duration":7,"objects_trimmed":1,"snaptrim_duration":0.0090263340000000004,"stat_sum":{"num_bytes":11017340,"num_objects":7,"num_object_clones":5,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":6,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.27","version":"85'54","reported_seq":53,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.074043+0000","last_change":"2024-03-19T11:31:04.074317+0000","last_active":"2024-03-19T11:31:04.074043+0000","last_peered":"2024-03-19T11:31:04.074043+0000","last_clean":"2024-03-19T11:31:04.074043+0000","last_became_active":"2024-03-19T11:31:00.070337+0000","last_became_peered":"2024-03-19T11:31:00.070337+0000","last_unstale":"2024-03-19T11:31:04.074043+0000","last_undegraded":"2024-03-19T11:31:04.074043+0000","last_fullsized":"2024-03-19T11:31:04.074043+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":6,"last_scrub":"71'48","last_scrub_stamp":"2024-03-19T11:30:55.467716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:55.467716+0000","objects_scrubbed":4,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:00.305683+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.018507597000000001,"stat_sum":{"num_bytes":6814202,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":11198,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":7070511,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"},{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.26","version":"61'63","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-19T11:31:02.087385+0000","last_change":"2024-03-19T11:31:02.087385+0000","last_active":"2024-03-19T11:31:02.087385+0000","last_peered":"2024-03-19T11:31:02.087385+0000","last_clean":"2024-03-19T11:31:02.087385+0000","last_became_active":"2024-03-19T11:30:59.074337+0000","last_became_peered":"2024-03-19T11:30:59.074337+0000","last_unstale":"2024-03-19T11:31:02.087385+0000","last_undegraded":"2024-03-19T11:31:02.087385+0000","last_fullsized":"2024-03-19T11:31:02.087385+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":6,"last_scrub":"61'63","last_scrub_stamp":"2024-03-19T11:30:52.406723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:52.406723+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:03.655668+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.010852772,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.25","version":"56'66","reported_seq":51,"reported_epoch":85,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:04.077517+0000","last_change":"2024-03-19T11:31:04.077570+0000","last_active":"2024-03-19T11:31:04.077517+0000","last_peered":"2024-03-19T11:31:04.077517+0000","last_clean":"2024-03-19T11:31:04.077517+0000","last_became_active":"2024-03-19T11:30:53.084473+0000","last_became_peered":"2024-03-19T11:30:53.084473+0000","last_unstale":"2024-03-19T11:31:04.077517+0000","last_undegraded":"2024-03-19T11:31:04.077517+0000","last_fullsized":"2024-03-19T11:31:04.077517+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":6,"last_scrub":"47'64","last_scrub_stamp":"2024-03-19T11:30:28.487136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:28.487136+0000","objects_scrubbed":3,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0087584710000000003,"stat_sum":{"num_bytes":2403368,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":10662,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6","length":"1"},{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.24","version":"85'117","reported_seq":477,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.064686+0000","last_change":"2024-03-19T11:31:04.064961+0000","last_active":"2024-03-19T11:31:04.064686+0000","last_peered":"2024-03-19T11:31:04.064686+0000","last_clean":"2024-03-19T11:31:04.064686+0000","last_became_active":"2024-03-19T11:31:03.063705+0000","last_became_peered":"2024-03-19T11:31:03.063705+0000","last_unstale":"2024-03-19T11:31:04.064686+0000","last_undegraded":"2024-03-19T11:31:04.064686+0000","last_fullsized":"2024-03-19T11:31:04.064686+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":6,"last_scrub":"84'113","last_scrub_stamp":"2024-03-19T11:31:03.304881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:31:03.304881+0000","objects_scrubbed":5,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:26.481361+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.0099064849999999996,"stat_sum":{"num_bytes":10102023,"num_objects":4,"num_object_clones":2,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"},{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.23","version":"85'29","reported_seq":59,"reported_epoch":85,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:04.098447+0000","last_change":"2024-03-19T11:31:04.098447+0000","last_active":"2024-03-19T11:31:04.098447+0000","last_peered":"2024-03-19T11:31:04.098447+0000","last_clean":"2024-03-19T11:31:04.098447+0000","last_became_active":"2024-03-19T11:30:53.076092+0000","last_became_peered":"2024-03-19T11:30:53.076092+0000","last_unstale":"2024-03-19T11:31:04.098447+0000","last_undegraded":"2024-03-19T11:31:04.098447+0000","last_fullsized":"2024-03-19T11:31:04.098447+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":6,"last_scrub":"62'23","last_scrub_stamp":"2024-03-19T11:30:43.163509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:43.163509+0000","objects_scrubbed":2,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:46.525658+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.023454183999999999,"stat_sum":{"num_bytes":3728802,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2833035,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.22","version":"28'6","reported_seq":55,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-19T11:31:03.045458+0000","last_change":"2024-03-19T11:31:02.071949+0000","last_active":"2024-03-19T11:31:03.045458+0000","last_peered":"2024-03-19T11:31:03.045458+0000","last_clean":"2024-03-19T11:31:03.045458+0000","last_became_active":"2024-03-19T11:30:53.068010+0000","last_became_peered":"2024-03-19T11:30:53.068010+0000","last_unstale":"2024-03-19T11:31:03.045458+0000","last_undegraded":"2024-03-19T11:31:03.045458+0000","last_fullsized":"2024-03-19T11:31:03.045458+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":6,"last_scrub":"28'6","last_scrub_stamp":"2024-03-19T11:30:55.724671+0000","last_deep_scrub":"28'6","last_deep_scrub_stamp":"2024-03-19T11:30:55.724671+0000","last_clean_scrub_stamp":"2024-03-19T11:30:55.724671+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:00.824445+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0024412639999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.21","version":"83'125","reported_seq":500,"reported_epoch":85,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:04.100621+0000","last_change":"2024-03-19T11:31:04.100668+0000","last_active":"2024-03-19T11:31:04.100621+0000","last_peered":"2024-03-19T11:31:04.100621+0000","last_clean":"2024-03-19T11:31:04.100621+0000","last_became_active":"2024-03-19T11:31:00.069759+0000","last_became_peered":"2024-03-19T11:31:00.069759+0000","last_unstale":"2024-03-19T11:31:04.100621+0000","last_undegraded":"2024-03-19T11:31:04.100621+0000","last_fullsized":"2024-03-19T11:31:04.100621+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":6,"last_scrub":"62'96","last_scrub_stamp":"2024-03-19T11:30:43.400885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:43.400885+0000","objects_scrubbed":9,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:55.649626+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.050771269000000001,"stat_sum":{"num_bytes":15281651,"num_objects":9,"num_object_clones":7,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":48,"num_read_kb":51399,"num_write":18,"num_write_kb":4886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":22983003,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"2"},{"start":"14","length":"1"}]},{"pgid":"4.20","version":"56'46","reported_seq":47,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-19T11:31:03.052424+0000","last_change":"2024-03-19T11:31:02.086750+0000","last_active":"2024-03-19T11:31:03.052424+0000","last_peered":"2024-03-19T11:31:03.052424+0000","last_clean":"2024-03-19T11:31:03.052424+0000","last_became_active":"2024-03-19T11:30:53.067251+0000","last_became_peered":"2024-03-19T11:30:53.067251+0000","last_unstale":"2024-03-19T11:31:03.052424+0000","last_undegraded":"2024-03-19T11:31:03.052424+0000","last_fullsized":"2024-03-19T11:31:03.052424+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":6,"last_scrub":"56'46","last_scrub_stamp":"2024-03-19T11:30:41.767100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:41.767100+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:52.904341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0046137030000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.1f","version":"0'0","reported_seq":55,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-19T11:31:03.053712+0000","last_change":"2024-03-19T11:31:02.082397+0000","last_active":"2024-03-19T11:31:03.053712+0000","last_peered":"2024-03-19T11:31:03.053712+0000","last_clean":"2024-03-19T11:31:03.053712+0000","last_became_active":"2024-03-19T11:30:53.087211+0000","last_became_peered":"2024-03-19T11:30:53.087211+0000","last_unstale":"2024-03-19T11:31:03.053712+0000","last_undegraded":"2024-03-19T11:31:03.053712+0000","last_fullsized":"2024-03-19T11:31:03.053712+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:30:42.377963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.377963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:01.745397+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0021919130000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.1e","version":"83'69","reported_seq":62,"reported_epoch":83,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:02.087501+0000","last_change":"2024-03-19T11:31:02.087501+0000","last_active":"2024-03-19T11:31:02.087501+0000","last_peered":"2024-03-19T11:31:02.087501+0000","last_clean":"2024-03-19T11:31:02.087501+0000","last_became_active":"2024-03-19T11:30:53.084452+0000","last_became_peered":"2024-03-19T11:30:53.084452+0000","last_unstale":"2024-03-19T11:31:02.087501+0000","last_undegraded":"2024-03-19T11:31:02.087501+0000","last_fullsized":"2024-03-19T11:31:02.087501+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"64'62","last_scrub_stamp":"2024-03-19T11:30:42.778159+0000","last_deep_scrub":"64'62","last_deep_scrub_stamp":"2024-03-19T11:30:42.778159+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.778159+0000","objects_scrubbed":4,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:43.449616+0000","scrub_duration":31,"objects_trimmed":1,"snaptrim_duration":0.00063602400000000001,"stat_sum":{"num_bytes":3794928,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":7565,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3779426,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.1d","version":"82'66","reported_seq":61,"reported_epoch":85,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:04.050025+0000","last_change":"2024-03-19T11:31:02.078403+0000","last_active":"2024-03-19T11:31:04.050025+0000","last_peered":"2024-03-19T11:31:04.050025+0000","last_clean":"2024-03-19T11:31:04.050025+0000","last_became_active":"2024-03-19T11:30:53.067748+0000","last_became_peered":"2024-03-19T11:30:53.067748+0000","last_unstale":"2024-03-19T11:31:04.050025+0000","last_undegraded":"2024-03-19T11:31:04.050025+0000","last_fullsized":"2024-03-19T11:31:04.050025+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"61'59","last_scrub_stamp":"2024-03-19T11:30:42.601863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.601863+0000","objects_scrubbed":2,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:52.206103+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.022205709000000001,"stat_sum":{"num_bytes":6649813,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":61,"num_write":4,"num_write_kb":1341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2817390,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.1c","version":"85'35","reported_seq":69,"reported_epoch":85,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:04.055902+0000","last_change":"2024-03-19T11:31:04.055902+0000","last_active":"2024-03-19T11:31:04.055902+0000","last_peered":"2024-03-19T11:31:04.055902+0000","last_clean":"2024-03-19T11:31:04.055902+0000","last_became_active":"2024-03-19T11:30:53.082655+0000","last_became_peered":"2024-03-19T11:30:53.082655+0000","last_unstale":"2024-03-19T11:31:04.055902+0000","last_undegraded":"2024-03-19T11:31:04.055902+0000","last_fullsized":"2024-03-19T11:31:04.055902+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"62'31","last_scrub_stamp":"2024-03-19T11:30:42.771396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.771396+0000","objects_scrubbed":3,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:55.484122+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.00073920500000000005,"stat_sum":{"num_bytes":3210104,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":47464,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":5034309,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.1b","version":"84'78","reported_seq":35,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-19T11:31:03.165322+0000","last_change":"2024-03-19T11:31:03.165322+0000","last_active":"2024-03-19T11:31:03.165322+0000","last_peered":"2024-03-19T11:31:03.165322+0000","last_clean":"2024-03-19T11:31:03.165322+0000","last_became_active":"2024-03-19T11:31:03.056769+0000","last_became_peered":"2024-03-19T11:31:03.056769+0000","last_unstale":"2024-03-19T11:31:03.165322+0000","last_undegraded":"2024-03-19T11:31:03.165322+0000","last_fullsized":"2024-03-19T11:31:03.165322+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":5,"last_scrub":"62'62","last_scrub_stamp":"2024-03-19T11:30:42.181424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.181424+0000","objects_scrubbed":5,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:54.474706+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0047244899999999996,"stat_sum":{"num_bytes":14753221,"num_objects":6,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":47032,"num_write":5,"num_write_kb":1437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":17966556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.1a","version":"84'99","reported_seq":39,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-19T11:31:03.175029+0000","last_change":"2024-03-19T11:31:03.175029+0000","last_active":"2024-03-19T11:31:03.175029+0000","last_peered":"2024-03-19T11:31:03.175029+0000","last_clean":"2024-03-19T11:31:03.175029+0000","last_became_active":"2024-03-19T11:31:03.056520+0000","last_became_peered":"2024-03-19T11:31:03.056520+0000","last_unstale":"2024-03-19T11:31:03.175029+0000","last_undegraded":"2024-03-19T11:31:03.175029+0000","last_fullsized":"2024-03-19T11:31:03.175029+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":5,"last_scrub":"61'76","last_scrub_stamp":"2024-03-19T11:30:41.433501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:41.433501+0000","objects_scrubbed":5,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:59.309150+0000","scrub_duration":11,"objects_trimmed":1,"snaptrim_duration":0.0078747160000000004,"stat_sum":{"num_bytes":15479824,"num_objects":7,"num_object_clones":5,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":15,"num_read_kb":9625,"num_write":18,"num_write_kb":5314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":16291222,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.19","version":"83'111","reported_seq":471,"reported_epoch":85,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:04.098365+0000","last_change":"2024-03-19T11:31:04.098409+0000","last_active":"2024-03-19T11:31:04.098365+0000","last_peered":"2024-03-19T11:31:04.098365+0000","last_clean":"2024-03-19T11:31:04.098365+0000","last_became_active":"2024-03-19T11:30:53.076188+0000","last_became_peered":"2024-03-19T11:30:53.076188+0000","last_unstale":"2024-03-19T11:31:04.098365+0000","last_undegraded":"2024-03-19T11:31:04.098365+0000","last_fullsized":"2024-03-19T11:31:04.098365+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"41'37","last_scrub_stamp":"2024-03-19T11:30:27.705159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:27.705159+0000","objects_scrubbed":4,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:29.203476+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.047949758000000002,"stat_sum":{"num_bytes":18546999,"num_objects":12,"num_object_clones":8,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":88,"num_read_kb":101447,"num_write":38,"num_write_kb":6633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":21870393,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":11,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.18","version":"79'119","reported_seq":413,"reported_epoch":85,"state":"active+recovering+remapped","last_fresh":"2024-03-19T11:31:04.184848+0000","last_change":"2024-03-19T11:31:04.070548+0000","last_active":"2024-03-19T11:31:04.184848+0000","last_peered":"2024-03-19T11:31:04.184848+0000","last_clean":"2024-03-19T11:30:58.054172+0000","last_became_active":"2024-03-19T11:31:04.066044+0000","last_became_peered":"2024-03-19T11:31:04.066044+0000","last_unstale":"2024-03-19T11:31:04.184848+0000","last_undegraded":"2024-03-19T11:31:04.184848+0000","last_fullsized":"2024-03-19T11:31:04.184848+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":5,"last_scrub":"48'77","last_scrub_stamp":"2024-03-19T11:30:29.399762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:29.399762+0000","objects_scrubbed":6,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.011956031000000001,"stat_sum":{"num_bytes":21618293,"num_objects":9,"num_object_clones":7,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":40,"num_read_kb":37994,"num_write":17,"num_write_kb":4402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":14888652,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":7,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":1,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.17","version":"41'24","reported_seq":55,"reported_epoch":85,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:04.050331+0000","last_change":"2024-03-19T11:31:02.079786+0000","last_active":"2024-03-19T11:31:04.050331+0000","last_peered":"2024-03-19T11:31:04.050331+0000","last_clean":"2024-03-19T11:31:04.050331+0000","last_became_active":"2024-03-19T11:30:53.074270+0000","last_became_peered":"2024-03-19T11:30:53.074270+0000","last_unstale":"2024-03-19T11:31:04.050331+0000","last_undegraded":"2024-03-19T11:31:04.050331+0000","last_fullsized":"2024-03-19T11:31:04.050331+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"41'24","last_scrub_stamp":"2024-03-19T11:30:53.993428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:53.993428+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:54.020328+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.022407469999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.16","version":"83'99","reported_seq":52,"reported_epoch":83,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:02.090231+0000","last_change":"2024-03-19T11:31:02.077638+0000","last_active":"2024-03-19T11:31:02.090231+0000","last_peered":"2024-03-19T11:31:02.090231+0000","last_clean":"2024-03-19T11:31:02.090231+0000","last_became_active":"2024-03-19T11:30:59.074063+0000","last_became_peered":"2024-03-19T11:30:59.074063+0000","last_unstale":"2024-03-19T11:31:02.090231+0000","last_undegraded":"2024-03-19T11:31:02.090231+0000","last_fullsized":"2024-03-19T11:31:02.090231+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":5,"last_scrub":"49'56","last_scrub_stamp":"2024-03-19T11:30:29.212766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:29.212766+0000","objects_scrubbed":5,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:47.363425+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.096278618999999996,"stat_sum":{"num_bytes":12479433,"num_objects":7,"num_object_clones":4,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":95,"num_read_kb":91318,"num_write":20,"num_write_kb":6321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":24165549,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":7,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.15","version":"85'148","reported_seq":547,"reported_epoch":85,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:04.108361+0000","last_change":"2024-03-19T11:31:02.074320+0000","last_active":"2024-03-19T11:31:04.108361+0000","last_peered":"2024-03-19T11:31:04.108361+0000","last_clean":"2024-03-19T11:31:04.108361+0000","last_became_active":"2024-03-19T11:30:56.063100+0000","last_became_peered":"2024-03-19T11:30:56.063100+0000","last_unstale":"2024-03-19T11:31:04.108361+0000","last_undegraded":"2024-03-19T11:31:04.108361+0000","last_fullsized":"2024-03-19T11:31:04.108361+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":5,"last_scrub":"49'74","last_scrub_stamp":"2024-03-19T11:30:29.457542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:29.457542+0000","objects_scrubbed":7,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:56.479079+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.019267638,"stat_sum":{"num_bytes":22140490,"num_objects":13,"num_object_clones":8,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":128,"num_read_kb":146180,"num_write":32,"num_write_kb":8100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":18744415,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":8,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.14","version":"40'56","reported_seq":220,"reported_epoch":85,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:04.051467+0000","last_change":"2024-03-19T11:31:02.093175+0000","last_active":"2024-03-19T11:31:04.051467+0000","last_peered":"2024-03-19T11:31:04.051467+0000","last_clean":"2024-03-19T11:31:04.051467+0000","last_became_active":"2024-03-19T11:31:00.065648+0000","last_became_peered":"2024-03-19T11:31:00.065648+0000","last_unstale":"2024-03-19T11:31:04.051467+0000","last_undegraded":"2024-03-19T11:31:04.051467+0000","last_fullsized":"2024-03-19T11:31:04.051467+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":5,"last_scrub":"40'56","last_scrub_stamp":"2024-03-19T11:30:54.998282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:54.998282+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:09.561569+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.014000106999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"2"},{"start":"14","length":"1"}]},{"pgid":"4.13","version":"48'22","reported_seq":92,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-19T11:31:03.047937+0000","last_change":"2024-03-19T11:31:02.077145+0000","last_active":"2024-03-19T11:31:03.047937+0000","last_peered":"2024-03-19T11:31:03.047937+0000","last_clean":"2024-03-19T11:31:03.047937+0000","last_became_active":"2024-03-19T11:30:45.245367+0000","last_became_peered":"2024-03-19T11:30:45.245367+0000","last_unstale":"2024-03-19T11:31:03.047937+0000","last_undegraded":"2024-03-19T11:31:03.047937+0000","last_fullsized":"2024-03-19T11:31:03.047937+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":5,"last_scrub":"48'22","last_scrub_stamp":"2024-03-19T11:30:27.183294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:27.183294+0000","objects_scrubbed":2,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:48.571632+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.007358018,"stat_sum":{"num_bytes":5361367,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":31,"num_read_kb":30730,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":3001789,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.12","version":"28'6","reported_seq":33,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.055008+0000","last_change":"2024-03-19T11:31:04.055613+0000","last_active":"2024-03-19T11:31:04.055008+0000","last_peered":"2024-03-19T11:31:04.055008+0000","last_clean":"2024-03-19T11:31:04.055008+0000","last_became_active":"2024-03-19T11:31:00.064380+0000","last_became_peered":"2024-03-19T11:31:00.064380+0000","last_unstale":"2024-03-19T11:31:04.055008+0000","last_undegraded":"2024-03-19T11:31:04.055008+0000","last_fullsized":"2024-03-19T11:31:04.055008+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":5,"last_scrub":"28'6","last_scrub_stamp":"2024-03-19T11:30:42.390965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.390965+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077181699999999995,"stat_sum":{"num_bytes":1176474,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"},{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.11","version":"83'81","reported_seq":178,"reported_epoch":85,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:04.050088+0000","last_change":"2024-03-19T11:31:02.092082+0000","last_active":"2024-03-19T11:31:04.050088+0000","last_peered":"2024-03-19T11:31:04.050088+0000","last_clean":"2024-03-19T11:31:04.050088+0000","last_became_active":"2024-03-19T11:30:30.052184+0000","last_became_peered":"2024-03-19T11:30:30.052184+0000","last_unstale":"2024-03-19T11:31:04.050088+0000","last_undegraded":"2024-03-19T11:31:04.050088+0000","last_fullsized":"2024-03-19T11:31:04.050088+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":5,"last_scrub":"47'60","last_scrub_stamp":"2024-03-19T11:30:30.780348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:30.780348+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:31.151678+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.014285984999999999,"stat_sum":{"num_bytes":8084512,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":20,"num_read_kb":23163,"num_write":14,"num_write_kb":4028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.10","version":"85'34","reported_seq":340,"reported_epoch":85,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:04.100696+0000","last_change":"2024-03-19T11:31:04.100696+0000","last_active":"2024-03-19T11:31:04.100696+0000","last_peered":"2024-03-19T11:31:04.100696+0000","last_clean":"2024-03-19T11:31:04.100696+0000","last_became_active":"2024-03-19T11:31:00.065617+0000","last_became_peered":"2024-03-19T11:31:00.065617+0000","last_unstale":"2024-03-19T11:31:04.100696+0000","last_undegraded":"2024-03-19T11:31:04.100696+0000","last_fullsized":"2024-03-19T11:31:04.100696+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":5,"last_scrub":"50'30","last_scrub_stamp":"2024-03-19T11:30:28.787058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:28.787058+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:51.738618+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.0079239749999999998,"stat_sum":{"num_bytes":3462209,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":20772,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":7102596,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.054605+0000","last_change":"2024-03-19T11:30:03.276845+0000","last_active":"2024-03-19T11:31:04.054605+0000","last_peered":"2024-03-19T11:31:04.054605+0000","last_clean":"2024-03-19T11:31:04.054605+0000","last_became_active":"2024-03-19T11:30:03.276220+0000","last_became_peered":"2024-03-19T11:30:03.276220+0000","last_unstale":"2024-03-19T11:31:04.054605+0000","last_undegraded":"2024-03-19T11:31:04.054605+0000","last_fullsized":"2024-03-19T11:31:04.054605+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:07.472401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"59'92","reported_seq":14,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-19T11:31:03.065335+0000","last_change":"2024-03-19T11:31:03.059104+0000","last_active":"2024-03-19T11:31:03.065335+0000","last_peered":"2024-03-19T11:31:03.065335+0000","last_clean":"2024-03-19T11:31:03.065335+0000","last_became_active":"2024-03-19T11:31:03.052657+0000","last_became_peered":"2024-03-19T11:31:03.052657+0000","last_unstale":"2024-03-19T11:31:03.065335+0000","last_undegraded":"2024-03-19T11:31:03.065335+0000","last_fullsized":"2024-03-19T11:31:03.065335+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'92","last_scrub_stamp":"2024-03-19T11:30:41.351520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:41.351520+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:07.204778+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0052965570000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.055239+0000","last_change":"2024-03-19T11:31:00.067375+0000","last_active":"2024-03-19T11:31:04.055239+0000","last_peered":"2024-03-19T11:31:04.055239+0000","last_clean":"2024-03-19T11:31:04.055239+0000","last_became_active":"2024-03-19T11:31:00.067099+0000","last_became_peered":"2024-03-19T11:31:00.067099+0000","last_unstale":"2024-03-19T11:31:04.055239+0000","last_undegraded":"2024-03-19T11:31:04.055239+0000","last_fullsized":"2024-03-19T11:31:04.055239+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:06.356796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"79'65","reported_seq":347,"reported_epoch":85,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:04.050073+0000","last_change":"2024-03-19T11:31:02.078237+0000","last_active":"2024-03-19T11:31:04.050073+0000","last_peered":"2024-03-19T11:31:04.050073+0000","last_clean":"2024-03-19T11:31:04.050073+0000","last_became_active":"2024-03-19T11:30:38.065693+0000","last_became_peered":"2024-03-19T11:30:38.065693+0000","last_unstale":"2024-03-19T11:31:04.050073+0000","last_undegraded":"2024-03-19T11:31:04.050073+0000","last_fullsized":"2024-03-19T11:31:04.050073+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'57","last_scrub_stamp":"2024-03-19T11:30:43.792326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:43.792326+0000","objects_scrubbed":6,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:55.566850+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.022132443000000002,"stat_sum":{"num_bytes":9323261,"num_objects":6,"num_object_clones":5,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":11,"num_read_kb":16661,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":160,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.052066+0000","last_change":"2024-03-19T11:30:59.072585+0000","last_active":"2024-03-19T11:31:04.052066+0000","last_peered":"2024-03-19T11:31:04.052066+0000","last_clean":"2024-03-19T11:31:04.052066+0000","last_became_active":"2024-03-19T11:30:59.071925+0000","last_became_peered":"2024-03-19T11:30:59.071925+0000","last_unstale":"2024-03-19T11:31:04.052066+0000","last_undegraded":"2024-03-19T11:31:04.052066+0000","last_fullsized":"2024-03-19T11:31:04.052066+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:08.925537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"41'19","reported_seq":295,"reported_epoch":83,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:02.077549+0000","last_change":"2024-03-19T11:31:02.077529+0000","last_active":"2024-03-19T11:31:02.077549+0000","last_peered":"2024-03-19T11:31:02.077549+0000","last_clean":"2024-03-19T11:31:02.077549+0000","last_became_active":"2024-03-19T11:30:38.058860+0000","last_became_peered":"2024-03-19T11:30:38.058860+0000","last_unstale":"2024-03-19T11:31:02.077549+0000","last_undegraded":"2024-03-19T11:31:02.077549+0000","last_fullsized":"2024-03-19T11:31:02.077549+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'19","last_scrub_stamp":"2024-03-19T11:30:28.180772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:28.180772+0000","objects_scrubbed":1,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00380665,"stat_sum":{"num_bytes":1403368,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4077,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":153,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.055174+0000","last_change":"2024-03-19T11:30:55.366872+0000","last_active":"2024-03-19T11:31:04.055174+0000","last_peered":"2024-03-19T11:31:04.055174+0000","last_clean":"2024-03-19T11:31:04.055174+0000","last_became_active":"2024-03-19T11:29:46.540939+0000","last_became_peered":"2024-03-19T11:29:46.540939+0000","last_unstale":"2024-03-19T11:31:04.055174+0000","last_undegraded":"2024-03-19T11:31:04.055174+0000","last_fullsized":"2024-03-19T11:31:04.055174+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:30:55.366631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:30:55.366631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:10.640223+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"85'17","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.067823+0000","last_change":"2024-03-19T11:31:04.068099+0000","last_active":"2024-03-19T11:31:04.067823+0000","last_peered":"2024-03-19T11:31:04.067823+0000","last_clean":"2024-03-19T11:31:04.067823+0000","last_became_active":"2024-03-19T11:31:00.069200+0000","last_became_peered":"2024-03-19T11:31:00.069200+0000","last_unstale":"2024-03-19T11:31:04.067823+0000","last_undegraded":"2024-03-19T11:31:04.067823+0000","last_fullsized":"2024-03-19T11:31:04.067823+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'10","last_scrub_stamp":"2024-03-19T11:30:54.429030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:54.429030+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.013022977999999999,"stat_sum":{"num_bytes":2742425,"num_objects":1,"num_object_clones":1,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":10173,"num_write":4,"num_write_kb":1071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2800509,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"},{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"3.3","version":"77'99","reported_seq":319,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-03-19T11:30:57.062257+0000","last_change":"2024-03-19T11:30:30.271391+0000","last_active":"2024-03-19T11:30:57.062257+0000","last_peered":"2024-03-19T11:30:57.062257+0000","last_clean":"2024-03-19T11:30:57.062257+0000","last_became_active":"2024-03-19T11:30:30.058963+0000","last_became_peered":"2024-03-19T11:30:30.058963+0000","last_unstale":"2024-03-19T11:30:57.062257+0000","last_undegraded":"2024-03-19T11:30:57.062257+0000","last_fullsized":"2024-03-19T11:30:57.062257+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:11.045052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53399952,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":74,"num_read_kb":3737,"num_write":107,"num_write_kb":58435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":22537669,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":144,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.068842+0000","last_change":"2024-03-19T11:29:46.321255+0000","last_active":"2024-03-19T11:31:04.068842+0000","last_peered":"2024-03-19T11:31:04.068842+0000","last_clean":"2024-03-19T11:31:04.068842+0000","last_became_active":"2024-03-19T11:29:46.320679+0000","last_became_peered":"2024-03-19T11:29:46.320679+0000","last_unstale":"2024-03-19T11:31:04.068842+0000","last_undegraded":"2024-03-19T11:31:04.068842+0000","last_fullsized":"2024-03-19T11:31:04.068842+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:14.793563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"85'132","reported_seq":508,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.061158+0000","last_change":"2024-03-19T11:31:04.061533+0000","last_active":"2024-03-19T11:31:04.061158+0000","last_peered":"2024-03-19T11:31:04.061158+0000","last_clean":"2024-03-19T11:31:04.061158+0000","last_became_active":"2024-03-19T11:31:03.054624+0000","last_became_peered":"2024-03-19T11:31:03.054624+0000","last_unstale":"2024-03-19T11:31:04.061158+0000","last_undegraded":"2024-03-19T11:31:04.061158+0000","last_fullsized":"2024-03-19T11:31:04.061158+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'119","last_scrub_stamp":"2024-03-19T11:30:52.512203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:52.512203+0000","objects_scrubbed":7,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:02.677965+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.0068329189999999998,"stat_sum":{"num_bytes":14273317,"num_objects":6,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":70,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"},{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"3.0","version":"85'100","reported_seq":247,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.068253+0000","last_change":"2024-03-19T11:29:57.550320+0000","last_active":"2024-03-19T11:31:04.068253+0000","last_peered":"2024-03-19T11:31:04.068253+0000","last_clean":"2024-03-19T11:31:04.068253+0000","last_became_active":"2024-03-19T11:29:57.549928+0000","last_became_peered":"2024-03-19T11:29:57.549928+0000","last_unstale":"2024-03-19T11:31:04.068253+0000","last_undegraded":"2024-03-19T11:31:04.068253+0000","last_fullsized":"2024-03-19T11:31:04.068253+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28242922,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":80,"num_read_kb":3042,"num_write":105,"num_write_kb":39820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":146,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.053846+0000","last_change":"2024-03-19T11:29:46.319811+0000","last_active":"2024-03-19T11:31:04.053846+0000","last_peered":"2024-03-19T11:31:04.053846+0000","last_clean":"2024-03-19T11:31:04.053846+0000","last_became_active":"2024-03-19T11:29:46.319334+0000","last_became_peered":"2024-03-19T11:29:46.319334+0000","last_unstale":"2024-03-19T11:31:04.053846+0000","last_undegraded":"2024-03-19T11:31:04.053846+0000","last_fullsized":"2024-03-19T11:31:04.053846+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:04.320463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"61'40","reported_seq":33,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.065031+0000","last_change":"2024-03-19T11:31:04.065328+0000","last_active":"2024-03-19T11:31:04.065031+0000","last_peered":"2024-03-19T11:31:04.065031+0000","last_clean":"2024-03-19T11:31:04.065031+0000","last_became_active":"2024-03-19T11:31:00.069809+0000","last_became_peered":"2024-03-19T11:31:00.069809+0000","last_unstale":"2024-03-19T11:31:04.065031+0000","last_undegraded":"2024-03-19T11:31:04.065031+0000","last_fullsized":"2024-03-19T11:31:04.065031+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'40","last_scrub_stamp":"2024-03-19T11:30:56.505586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:56.505586+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:09.446385+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0098007430000000006,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"},{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"3.1","version":"82'85","reported_seq":289,"reported_epoch":82,"state":"peering","last_fresh":"2024-03-19T11:31:01.114104+0000","last_change":"2024-03-19T11:31:02.076669+0000","last_active":"2024-03-19T11:31:02.076669+0000","last_peered":"2024-03-19T11:31:01.114104+0000","last_clean":"2024-03-19T11:31:01.114104+0000","last_became_active":"2024-03-19T11:30:26.927069+0000","last_became_peered":"2024-03-19T11:30:26.927069+0000","last_unstale":"2024-03-19T11:31:01.114104+0000","last_undegraded":"2024-03-19T11:31:01.114104+0000","last_fullsized":"2024-03-19T11:31:01.114104+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42832129,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":71,"num_read_kb":6652,"num_write":89,"num_write_kb":49391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":16477750,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,4],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":155,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.052299+0000","last_change":"2024-03-19T11:31:00.070027+0000","last_active":"2024-03-19T11:31:04.052299+0000","last_peered":"2024-03-19T11:31:04.052299+0000","last_clean":"2024-03-19T11:31:04.052299+0000","last_became_active":"2024-03-19T11:31:00.068989+0000","last_became_peered":"2024-03-19T11:31:00.068989+0000","last_unstale":"2024-03-19T11:31:04.052299+0000","last_undegraded":"2024-03-19T11:31:04.052299+0000","last_fullsized":"2024-03-19T11:31:04.052299+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:11.599600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"77'68","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-19T11:31:02.085910+0000","last_change":"2024-03-19T11:31:02.085910+0000","last_active":"2024-03-19T11:31:02.085910+0000","last_peered":"2024-03-19T11:31:02.085910+0000","last_clean":"2024-03-19T11:31:02.085910+0000","last_became_active":"2024-03-19T11:30:59.075057+0000","last_became_peered":"2024-03-19T11:30:59.075057+0000","last_unstale":"2024-03-19T11:31:02.085910+0000","last_undegraded":"2024-03-19T11:31:02.085910+0000","last_fullsized":"2024-03-19T11:31:02.085910+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'65","last_scrub_stamp":"2024-03-19T11:30:53.385414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:53.385414+0000","objects_scrubbed":3,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:05.375039+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.009635546,"stat_sum":{"num_bytes":7189731,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4209633,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"3.2","version":"85'84","reported_seq":239,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.049868+0000","last_change":"2024-03-19T11:29:57.549277+0000","last_active":"2024-03-19T11:31:04.049868+0000","last_peered":"2024-03-19T11:31:04.049868+0000","last_clean":"2024-03-19T11:31:04.049868+0000","last_became_active":"2024-03-19T11:29:57.548946+0000","last_became_peered":"2024-03-19T11:29:57.548946+0000","last_unstale":"2024-03-19T11:31:04.049868+0000","last_undegraded":"2024-03-19T11:31:04.049868+0000","last_fullsized":"2024-03-19T11:31:04.049868+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:08.380405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39345556,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":76,"num_read_kb":8248,"num_write":94,"num_write_kb":41080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":144,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.067990+0000","last_change":"2024-03-19T11:29:46.322491+0000","last_active":"2024-03-19T11:31:04.067990+0000","last_peered":"2024-03-19T11:31:04.067990+0000","last_clean":"2024-03-19T11:31:04.067990+0000","last_became_active":"2024-03-19T11:29:46.321760+0000","last_became_peered":"2024-03-19T11:29:46.321760+0000","last_unstale":"2024-03-19T11:31:04.067990+0000","last_undegraded":"2024-03-19T11:31:04.067990+0000","last_fullsized":"2024-03-19T11:31:04.067990+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:04.561915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":329,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.052318+0000","last_change":"2024-03-19T11:31:03.367805+0000","last_active":"2024-03-19T11:31:04.052318+0000","last_peered":"2024-03-19T11:31:04.052318+0000","last_clean":"2024-03-19T11:31:04.052318+0000","last_became_active":"2024-03-19T11:30:59.073709+0000","last_became_peered":"2024-03-19T11:30:59.073709+0000","last_unstale":"2024-03-19T11:31:04.052318+0000","last_undegraded":"2024-03-19T11:31:04.052318+0000","last_fullsized":"2024-03-19T11:31:04.052318+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2024-03-19T11:31:03.367669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:43.166618+0000","last_clean_scrub_stamp":"2024-03-19T11:31:03.367669+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:29.977573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"84'103","reported_seq":480,"reported_epoch":84,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:03.169808+0000","last_change":"2024-03-19T11:31:03.168375+0000","last_active":"2024-03-19T11:31:03.169808+0000","last_peered":"2024-03-19T11:31:03.169808+0000","last_clean":"2024-03-19T11:31:03.169808+0000","last_became_active":"2024-03-19T11:31:03.065344+0000","last_became_peered":"2024-03-19T11:31:03.065344+0000","last_unstale":"2024-03-19T11:31:03.169808+0000","last_undegraded":"2024-03-19T11:31:03.169808+0000","last_fullsized":"2024-03-19T11:31:03.169808+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'81","last_scrub_stamp":"2024-03-19T11:30:52.360396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:52.360396+0000","objects_scrubbed":6,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:11.500650+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.032057466999999999,"stat_sum":{"num_bytes":10838384,"num_objects":8,"num_object_clones":6,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":13,"num_read_kb":12349,"num_write":8,"num_write_kb":1273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":10558635,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"c","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.8","version":"62'71","reported_seq":359,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.077614+0000","last_change":"2024-03-19T11:31:04.078135+0000","last_active":"2024-03-19T11:31:04.077614+0000","last_peered":"2024-03-19T11:31:04.077614+0000","last_clean":"2024-03-19T11:31:04.077614+0000","last_became_active":"2024-03-19T11:30:45.251283+0000","last_became_peered":"2024-03-19T11:30:45.251283+0000","last_unstale":"2024-03-19T11:31:04.077614+0000","last_undegraded":"2024-03-19T11:31:04.077614+0000","last_fullsized":"2024-03-19T11:31:04.077614+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'71","last_scrub_stamp":"2024-03-19T11:30:53.381386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:53.381386+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:18.275213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010232903999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6","length":"1"},{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.9","version":"63'55","reported_seq":384,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-19T11:31:03.046677+0000","last_change":"2024-03-19T11:31:02.070296+0000","last_active":"2024-03-19T11:31:03.046677+0000","last_peered":"2024-03-19T11:31:03.046677+0000","last_clean":"2024-03-19T11:31:03.046677+0000","last_became_active":"2024-03-19T11:31:00.069789+0000","last_became_peered":"2024-03-19T11:31:00.069789+0000","last_unstale":"2024-03-19T11:31:03.046677+0000","last_undegraded":"2024-03-19T11:31:03.046677+0000","last_fullsized":"2024-03-19T11:31:03.046677+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'55","last_scrub_stamp":"2024-03-19T11:30:54.706235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:54.706235+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:01.278994+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00082106599999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.a","version":"62'77","reported_seq":375,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-19T11:31:03.053296+0000","last_change":"2024-03-19T11:31:02.084460+0000","last_active":"2024-03-19T11:31:03.053296+0000","last_peered":"2024-03-19T11:31:03.053296+0000","last_clean":"2024-03-19T11:31:03.053296+0000","last_became_active":"2024-03-19T11:30:45.256124+0000","last_became_peered":"2024-03-19T11:30:45.256124+0000","last_unstale":"2024-03-19T11:31:03.053296+0000","last_undegraded":"2024-03-19T11:31:03.053296+0000","last_fullsized":"2024-03-19T11:31:03.053296+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'77","last_scrub_stamp":"2024-03-19T11:30:55.291823+0000","last_deep_scrub":"62'77","last_deep_scrub_stamp":"2024-03-19T11:30:55.291823+0000","last_clean_scrub_stamp":"2024-03-19T11:30:55.291823+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:11.588174+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00325363,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.b","version":"70'62","reported_seq":375,"reported_epoch":83,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:02.077320+0000","last_change":"2024-03-19T11:31:02.077300+0000","last_active":"2024-03-19T11:31:02.077320+0000","last_peered":"2024-03-19T11:31:02.077320+0000","last_clean":"2024-03-19T11:31:02.077320+0000","last_became_active":"2024-03-19T11:30:45.261756+0000","last_became_peered":"2024-03-19T11:30:45.261756+0000","last_unstale":"2024-03-19T11:31:02.077320+0000","last_undegraded":"2024-03-19T11:31:02.077320+0000","last_fullsized":"2024-03-19T11:31:02.077320+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'62","last_scrub_stamp":"2024-03-19T11:30:56.095039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:56.095039+0000","objects_scrubbed":3,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:18.760680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063617100000000005,"stat_sum":{"num_bytes":5215740,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.c","version":"56'29","reported_seq":10,"reported_epoch":84,"state":"clean+premerge+peered","last_fresh":"2024-03-19T11:31:03.063714+0000","last_change":"2024-03-19T11:31:03.058030+0000","last_active":"2024-03-19T11:31:01.070481+0000","last_peered":"2024-03-19T11:31:03.063714+0000","last_clean":"2024-03-19T11:31:03.063714+0000","last_became_active":"2024-03-19T11:30:56.061560+0000","last_became_peered":"2024-03-19T11:31:03.056794+0000","last_unstale":"2024-03-19T11:31:03.063714+0000","last_undegraded":"2024-03-19T11:31:03.063714+0000","last_fullsized":"2024-03-19T11:31:03.063714+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'29","last_scrub_stamp":"2024-03-19T11:30:53.743385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:53.743385+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:02.142968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.021151900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.d","version":"85'82","reported_seq":393,"reported_epoch":85,"state":"active+clean+remapped+snaptrim","last_fresh":"2024-03-19T11:31:04.184330+0000","last_change":"2024-03-19T11:31:04.182796+0000","last_active":"2024-03-19T11:31:04.184330+0000","last_peered":"2024-03-19T11:31:04.184330+0000","last_clean":"2024-03-19T11:31:04.184330+0000","last_became_active":"2024-03-19T11:31:04.064814+0000","last_became_peered":"2024-03-19T11:31:04.064814+0000","last_unstale":"2024-03-19T11:31:04.184330+0000","last_undegraded":"2024-03-19T11:31:04.184330+0000","last_fullsized":"2024-03-19T11:31:04.184330+0000","mapping_epoch":84,"log_start":"70'71","ondisk_log_start":"70'71","created":21,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'73","last_scrub_stamp":"2024-03-19T11:30:53.519178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:53.519178+0000","objects_scrubbed":6,"log_size":11,"log_dups_size":50,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:15.690636+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0030257930000000001,"stat_sum":{"num_bytes":10067275,"num_objects":6,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":9,"num_read_kb":11335,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":8146998,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[1,2],"avail_no_missing":["1","2","4"],"object_location_counts":[{"shards":"1,2","objects":6}],"blocked_by":[],"up_primary":4,"acting_primary":1,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.e","version":"79'78","reported_seq":352,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-19T11:31:03.046968+0000","last_change":"2024-03-19T11:31:02.074204+0000","last_active":"2024-03-19T11:31:03.046968+0000","last_peered":"2024-03-19T11:31:03.046968+0000","last_clean":"2024-03-19T11:31:03.046968+0000","last_became_active":"2024-03-19T11:30:38.060487+0000","last_became_peered":"2024-03-19T11:30:38.060487+0000","last_unstale":"2024-03-19T11:31:03.046968+0000","last_undegraded":"2024-03-19T11:31:03.046968+0000","last_fullsized":"2024-03-19T11:31:03.046968+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'65","last_scrub_stamp":"2024-03-19T11:30:43.824620+0000","last_deep_scrub":"65'65","last_deep_scrub_stamp":"2024-03-19T11:30:43.824620+0000","last_clean_scrub_stamp":"2024-03-19T11:30:43.824620+0000","objects_scrubbed":4,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:48.641497+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0.0046356100000000001,"stat_sum":{"num_bytes":12990971,"num_objects":6,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":21,"num_read_kb":22723,"num_write":8,"num_write_kb":2301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.f","version":"0'0","reported_seq":256,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.065434+0000","last_change":"2024-03-19T11:31:04.065959+0000","last_active":"2024-03-19T11:31:04.065434+0000","last_peered":"2024-03-19T11:31:04.065434+0000","last_clean":"2024-03-19T11:31:04.065434+0000","last_became_active":"2024-03-19T11:30:38.050919+0000","last_became_peered":"2024-03-19T11:30:38.050919+0000","last_unstale":"2024-03-19T11:31:04.065434+0000","last_undegraded":"2024-03-19T11:31:04.065434+0000","last_fullsized":"2024-03-19T11:31:04.065434+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:30:43.411893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:43.411893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:54.719455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010140278000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"},{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]}],"pool_stats":[{"poolid":4,"num_pg":44,"stat_sum":{"num_bytes":275155131,"num_objects":142,"num_object_clones":91,"num_object_copies":284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":139,"num_whiteouts":6,"num_read":669,"num_read_kb":715159,"num_write":198,"num_write_kb":50396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":200107110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":93,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":402448384,"data_stored":457616802,"data_compressed":2587576,"data_compressed_allocated":1728512,"data_compressed_original":5898240,"omap_allocated":0,"internal_metadata":0},"log_size":2787,"ondisk_log_size":2787,"up":88,"acting":88,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":163820559,"num_objects":143,"num_object_clones":0,"num_object_copies":286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":143,"num_whiteouts":0,"num_read":301,"num_read_kb":21679,"num_write":395,"num_write_kb":188726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":39015419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":165703680,"data_stored":179494984,"data_compressed":12248719,"data_compressed_allocated":13119488,"data_compressed_original":27238400,"omap_allocated":0,"internal_metadata":0},"log_size":368,"ondisk_log_size":368,"up":8,"acting":8,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":1586784,"data_compressed":19832,"data_compressed_allocated":98304,"data_compressed_original":1560576,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542164,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83364,"kb_used_data":52184,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94288476,"statfs":{"total":96636764160,"available":96551399424,"internally_reserved":0,"allocated":53436416,"data_stored":62427174,"data_compressed":227710,"data_compressed_allocated":151552,"data_compressed_original":1101824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542164,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84092,"kb_used_data":52908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94287748,"statfs":{"total":96636764160,"available":96550653952,"internally_reserved":0,"allocated":54177792,"data_stored":62533221,"data_compressed":1917086,"data_compressed_allocated":1929216,"data_compressed_original":4210688,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542164,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":129452,"kb_used_data":102432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94242388,"statfs":{"total":96636764160,"available":96504205312,"internally_reserved":0,"allocated":104890368,"data_stored":120523535,"data_compressed":2532072,"data_compressed_allocated":2379776,"data_compressed_original":5521408,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542163,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79208,"kb_used_data":52184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94292632,"statfs":{"total":96636764160,"available":96555655168,"internally_reserved":0,"allocated":53436416,"data_stored":60429347,"data_compressed":1749130,"data_compressed_allocated":1527808,"data_compressed_original":4194304,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542163,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":153924,"kb_used_data":113028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94217916,"statfs":{"total":96636764160,"available":96479145984,"internally_reserved":0,"allocated":115740672,"data_stored":123349286,"data_compressed":1407043,"data_compressed_allocated":1449984,"data_compressed_original":3211264,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95020,"kb_used_data":68000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94276820,"statfs":{"total":96636764160,"available":96539463680,"internally_reserved":0,"allocated":69632000,"data_stored":81512192,"data_compressed":2377281,"data_compressed_allocated":2482176,"data_compressed_original":4935680,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574867,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46512,"kb_used_data":15328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325328,"statfs":{"total":96636764160,"available":96589135872,"internally_reserved":0,"allocated":15695872,"data_stored":17201968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574867,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":125852,"kb_used_data":94668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94245988,"statfs":{"total":96636764160,"available":96507891712,"internally_reserved":0,"allocated":96940032,"data_stored":102441426,"data_compressed":4177120,"data_compressed_allocated":4493312,"data_compressed_original":9555968,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6884,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6884,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6064,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53927936,"data_stored":58198011,"data_compressed":3980813,"data_compressed_allocated":4255744,"data_compressed_original":8638464,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14168064,"data_stored":15397438,"data_compressed":898614,"data_compressed_allocated":970752,"data_compressed_original":2228224,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14168064,"data_stored":15397438,"data_compressed":898614,"data_compressed_allocated":970752,"data_compressed_original":2228224,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":22126592,"data_stored":23997131,"data_compressed":1849977,"data_compressed_allocated":1974272,"data_compressed_original":3883008,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11079680,"data_stored":12191862,"data_compressed":660988,"data_compressed_allocated":712704,"data_compressed_original":1835008,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":19947520,"data_stored":21724875,"data_compressed":1695440,"data_compressed_allocated":1810432,"data_compressed_original":3620864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":30285824,"data_stored":32588229,"data_compressed":2264273,"data_compressed_allocated":2424832,"data_compressed_original":4804608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":38711296,"data_stored":39908792,"data_compressed":189423,"data_compressed_allocated":204800,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90685440,"data_stored":97008266,"data_compressed":390047,"data_compressed_allocated":352256,"data_compressed_original":720896,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":34271232,"data_stored":39685709,"data_compressed":732134,"data_compressed_allocated":430080,"data_compressed_original":1703936,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":78860288,"data_stored":93060511,"data_compressed":682095,"data_compressed_allocated":405504,"data_compressed_original":1638400,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":43638784,"data_stored":47398962,"data_compressed":37577,"data_compressed_allocated":40960,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":29360128,"data_stored":35596983,"data_compressed":221646,"data_compressed_allocated":118784,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":38412288,"data_stored":48442122,"data_compressed":113008,"data_compressed_allocated":57344,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":48508928,"data_stored":56515457,"data_compressed":221646,"data_compressed_allocated":118784,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T11:31:06.584 INFO:tasks.rados.rados.0.smithi038.stdout:1216: finishing write tid 1 to smithi03829856-18 2024-03-19T11:31:06.584 INFO:tasks.rados.rados.0.smithi038.stdout:1216: finishing write tid 2 to smithi03829856-18 2024-03-19T11:31:06.584 INFO:tasks.rados.rados.0.smithi038.stdout:1215: done (7 left) 2024-03-19T11:31:06.584 INFO:tasks.rados.rados.0.smithi038.stdout:1217: done (6 left) 2024-03-19T11:31:06.584 INFO:tasks.rados.rados.0.smithi038.stdout:1218: done (5 left) 2024-03-19T11:31:06.584 INFO:tasks.rados.rados.0.smithi038.stdout:1220: done (4 left) 2024-03-19T11:31:06.584 INFO:tasks.rados.rados.0.smithi038.stdout:1221: done (3 left) 2024-03-19T11:31:06.584 INFO:tasks.rados.rados.0.smithi038.stdout:1222: done (2 left) 2024-03-19T11:31:06.584 INFO:tasks.rados.rados.0.smithi038.stdout:1224: tier_evict oid 16 2024-03-19T11:31:06.584 INFO:tasks.rados.rados.0.smithi038.stdout:1224: tier_evict oid 16 snap -1 2024-03-19T11:31:06.585 INFO:tasks.rados.rados.0.smithi038.stdout:1216: finishing write tid 3 to smithi03829856-18 2024-03-19T11:31:06.585 INFO:tasks.rados.rados.0.smithi038.stdout:1216: finishing write tid 4 to smithi03829856-18 2024-03-19T11:31:06.585 INFO:tasks.rados.rados.0.smithi038.stdout:1216: finishing write tid 5 to smithi03829856-18 2024-03-19T11:31:06.585 INFO:tasks.rados.rados.0.smithi038.stdout:1216: oid 18 updating version 0 to 122 2024-03-19T11:31:06.585 INFO:tasks.rados.rados.0.smithi038.stdout:1216: oid 18 updating version 122 to 123 2024-03-19T11:31:06.585 INFO:tasks.rados.rados.0.smithi038.stdout:1216: oid 18 version 123 is already newer than 121 2024-03-19T11:31:06.585 INFO:tasks.rados.rados.0.smithi038.stdout:1216: oid 18 updating version 123 to 124 2024-03-19T11:31:06.585 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 18 v 124 (ObjNum 199 snap 22 seq_num 199) dirty exists 2024-03-19T11:31:06.585 INFO:tasks.rados.rados.0.smithi038.stdout:1216: left oid 18 (ObjNum 199 snap 22 seq_num 199) 2024-03-19T11:31:06.585 INFO:tasks.rados.rados.0.smithi038.stdout:1223: finishing write tid 1 to smithi03829856-35 2024-03-19T11:31:06.585 INFO:tasks.rados.rados.0.smithi038.stdout:1223: finishing write tid 2 to smithi03829856-35 2024-03-19T11:31:06.585 INFO:tasks.rados.rados.0.smithi038.stdout:1223: finishing write tid 3 to smithi03829856-35 2024-03-19T11:31:06.585 INFO:tasks.rados.rados.0.smithi038.stdout:1223: finishing write tid 4 to smithi03829856-35 2024-03-19T11:31:06.585 INFO:tasks.rados.rados.0.smithi038.stdout:1223: finishing write tid 5 to smithi03829856-35 2024-03-19T11:31:06.586 INFO:tasks.rados.rados.0.smithi038.stdout:1223: oid 35 updating version 0 to 160 2024-03-19T11:31:06.586 INFO:tasks.rados.rados.0.smithi038.stdout:1223: oid 35 updating version 160 to 161 2024-03-19T11:31:06.586 INFO:tasks.rados.rados.0.smithi038.stdout:1223: oid 35 version 161 is already newer than 159 2024-03-19T11:31:06.586 INFO:tasks.rados.rados.0.smithi038.stdout:1223: oid 35 version 161 is already newer than 158 2024-03-19T11:31:06.586 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 35 v 161 (ObjNum 200 snap 22 seq_num 200) dirty exists 2024-03-19T11:31:06.586 INFO:tasks.rados.rados.0.smithi038.stdout:1223: left oid 35 (ObjNum 200 snap 22 seq_num 200) 2024-03-19T11:31:06.586 INFO:tasks.rados.rados.0.smithi038.stdout:1216: done (2 left) 2024-03-19T11:31:06.586 INFO:tasks.rados.rados.0.smithi038.stdout:1223: done (1 left) 2024-03-19T11:31:06.586 INFO:tasks.rados.rados.0.smithi038.stdout:1225: write oid 49 current snap is 22 2024-03-19T11:31:06.586 INFO:tasks.rados.rados.0.smithi038.stdout:1225: seq_num 201 ranges {578207=550227,1762410=779734,3286845=91362} 2024-03-19T11:31:06.589 INFO:tasks.rados.rados.0.smithi038.stdout:1225: writing smithi03829856-49 from 578207 to 1128434 tid 1 2024-03-19T11:31:06.597 INFO:tasks.rados.rados.0.smithi038.stdout:1225: writing smithi03829856-49 from 1762410 to 2542144 tid 2 2024-03-19T11:31:06.602 INFO:tasks.rados.rados.0.smithi038.stdout:1225: writing smithi03829856-49 from 3286845 to 3378207 tid 3 2024-03-19T11:31:06.602 INFO:tasks.rados.rados.0.smithi038.stdout:1226: delete oid 26 current snap is 22 2024-03-19T11:31:06.602 INFO:tasks.rados.rados.0.smithi038.stdout:1224: got (22) Invalid argument 2024-03-19T11:31:06.602 INFO:tasks.rados.rados.0.smithi038.stdout:1225: finishing write tid 1 to smithi03829856-49 2024-03-19T11:31:06.604 INFO:tasks.rados.rados.0.smithi038.stdout:1224: done (2 left) 2024-03-19T11:31:06.604 INFO:tasks.rados.rados.0.smithi038.stdout:1226: done (1 left) 2024-03-19T11:31:06.604 INFO:tasks.rados.rados.0.smithi038.stdout:1227: rollback oid 25 current snap is 22 2024-03-19T11:31:06.604 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 25 to 21 2024-03-19T11:31:06.604 INFO:tasks.rados.rados.0.smithi038.stdout:1228: read oid 44 snap -1 2024-03-19T11:31:06.604 INFO:tasks.rados.rados.0.smithi038.stdout:1228: expect (ObjNum 108 snap 11 seq_num 108) 2024-03-19T11:31:06.604 INFO:tasks.rados.rados.0.smithi038.stdout:1229: rollback oid 11 current snap is 22 2024-03-19T11:31:06.604 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 11 to 21 2024-03-19T11:31:06.604 INFO:tasks.rados.rados.0.smithi038.stdout:1230: write (excl) oid 19 current snap is 22 2024-03-19T11:31:06.604 INFO:tasks.rados.rados.0.smithi038.stdout:1230: seq_num 203 ranges {611821=700725,1879374=783403,3341691=70130} 2024-03-19T11:31:06.614 INFO:tasks.rados.rados.0.smithi038.stdout:1230: writing smithi03829856-19 from 611821 to 1312546 tid 1 2024-03-19T11:31:06.624 INFO:tasks.rados.rados.0.smithi038.stdout:1230: writing smithi03829856-19 from 1879374 to 2662777 tid 2 2024-03-19T11:31:06.629 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:06.637+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:06.630 INFO:tasks.rados.rados.0.smithi038.stdout:1230: writing smithi03829856-19 from 3341691 to 3411821 tid 3 2024-03-19T11:31:06.630 INFO:tasks.rados.rados.0.smithi038.stdout:1231: delete oid 36 current snap is 22 2024-03-19T11:31:06.630 INFO:tasks.rados.rados.0.smithi038.stdout:1225: finishing write tid 2 to smithi03829856-49 2024-03-19T11:31:06.630 INFO:tasks.rados.rados.0.smithi038.stdout:1225: finishing write tid 3 to smithi03829856-49 2024-03-19T11:31:06.630 INFO:tasks.rados.rados.0.smithi038.stdout:1225: finishing write tid 4 to smithi03829856-49 2024-03-19T11:31:06.630 INFO:tasks.rados.rados.0.smithi038.stdout:1227: finishing rollback tid 1 to smithi03829856-25 2024-03-19T11:31:06.631 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 25 v 60 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-03-19T11:31:06.631 INFO:tasks.rados.rados.0.smithi038.stdout:1225: finishing write tid 5 to smithi03829856-49 2024-03-19T11:31:06.631 INFO:tasks.rados.rados.0.smithi038.stdout:1225: oid 49 updating version 0 to 36 2024-03-19T11:31:06.631 INFO:tasks.rados.rados.0.smithi038.stdout:1225: oid 49 updating version 36 to 38 2024-03-19T11:31:06.631 INFO:tasks.rados.rados.0.smithi038.stdout:1225: oid 49 version 38 is already newer than 37 2024-03-19T11:31:06.631 INFO:tasks.rados.rados.0.smithi038.stdout:1225: oid 49 version 38 is already newer than 35 2024-03-19T11:31:06.631 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 49 v 38 (ObjNum 201 snap 22 seq_num 201) dirty exists 2024-03-19T11:31:06.631 INFO:tasks.rados.rados.0.smithi038.stdout:1225: left oid 49 (ObjNum 201 snap 22 seq_num 201) 2024-03-19T11:31:06.631 INFO:tasks.rados.rados.0.smithi038.stdout:1229: finishing rollback tid 0 to smithi03829856-11 2024-03-19T11:31:06.631 INFO:tasks.rados.rados.0.smithi038.stdout:1228: expect (ObjNum 108 snap 11 seq_num 108) 2024-03-19T11:31:06.680 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:06.680 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:06.694 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-19T11:31:06.739 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:06.751+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:06.801 INFO:tasks.rados.rados.0.smithi038.stdout:1229: finishing rollback tid 1 to smithi03829856-11 2024-03-19T11:31:06.801 INFO:tasks.rados.rados.0.smithi038.stdout:1229: finishing rollback tid 2 to smithi03829856-11 2024-03-19T11:31:06.801 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 11 v 90 (ObjNum 187 snap 21 seq_num 187) dirty exists 2024-03-19T11:31:06.801 INFO:tasks.rados.rados.0.smithi038.stdout:1230: finishing write tid 1 to smithi03829856-19 2024-03-19T11:31:06.802 INFO:tasks.rados.rados.0.smithi038.stdout:1230: finishing write tid 2 to smithi03829856-19 2024-03-19T11:31:06.802 INFO:tasks.rados.rados.0.smithi038.stdout:1230: finishing write tid 3 to smithi03829856-19 2024-03-19T11:31:06.802 INFO:tasks.rados.rados.0.smithi038.stdout:1230: finishing write tid 4 to smithi03829856-19 2024-03-19T11:31:06.802 INFO:tasks.rados.rados.0.smithi038.stdout:1230: finishing write tid 5 to smithi03829856-19 2024-03-19T11:31:06.802 INFO:tasks.rados.rados.0.smithi038.stdout:1230: oid 19 updating version 0 to 68 2024-03-19T11:31:06.802 INFO:tasks.rados.rados.0.smithi038.stdout:1230: oid 19 updating version 68 to 69 2024-03-19T11:31:06.802 INFO:tasks.rados.rados.0.smithi038.stdout:1230: oid 19 updating version 69 to 70 2024-03-19T11:31:06.802 INFO:tasks.rados.rados.0.smithi038.stdout:1230: oid 19 updating version 70 to 71 2024-03-19T11:31:06.802 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 19 v 71 (ObjNum 203 snap 22 seq_num 203) dirty exists 2024-03-19T11:31:06.802 INFO:tasks.rados.rados.0.smithi038.stdout:1230: left oid 19 (ObjNum 203 snap 22 seq_num 203) 2024-03-19T11:31:06.802 INFO:tasks.rados.rados.0.smithi038.stdout:1225: done (5 left) 2024-03-19T11:31:06.802 INFO:tasks.rados.rados.0.smithi038.stdout:1227: done (4 left) 2024-03-19T11:31:06.802 INFO:tasks.rados.rados.0.smithi038.stdout:1228: done (3 left) 2024-03-19T11:31:06.802 INFO:tasks.rados.rados.0.smithi038.stdout:1229: done (2 left) 2024-03-19T11:31:06.803 INFO:tasks.rados.rados.0.smithi038.stdout:1230: done (1 left) 2024-03-19T11:31:06.803 INFO:tasks.rados.rados.0.smithi038.stdout:1231: done (0 left) 2024-03-19T11:31:06.803 INFO:tasks.rados.rados.0.smithi038.stdout:1232: snap_remove snap 17 2024-03-19T11:31:06.813 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:06.813 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:06.827 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-19T11:31:06.872 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:06.883+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:06.932 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:06.941+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:06.997 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:06.997 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:07.011 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-19T11:31:07.032 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:07.041+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:07.081 INFO:tasks.rados.rados.0.smithi038.stdout:1232: done (0 left) 2024-03-19T11:31:07.081 INFO:tasks.rados.rados.0.smithi038.stdout:1233: set_chunk oid 20 target oid 2024-03-19T11:31:07.103 INFO:tasks.rados.rados.0.smithi038.stdout:1233: set_chunk oid 20 offset: 1358848 length: 685568 target oid 24bbd5cd85f5d8f03c8dbc2922b42cbb75f818d10afb7dd11f86b69aab2afb17 offset: 0 snap -1 2024-03-19T11:31:07.103 INFO:tasks.rados.rados.0.smithi038.stdout:1234: read oid 16 snap 21 2024-03-19T11:31:07.103 INFO:tasks.rados.rados.0.smithi038.stdout:1234: expect (ObjNum 65 snap 3 seq_num 65) 2024-03-19T11:31:07.103 INFO:tasks.rados.rados.0.smithi038.stdout:1235: read oid 2 snap -1 2024-03-19T11:31:07.103 INFO:tasks.rados.rados.0.smithi038.stdout:1235: expect (ObjNum 185 snap 20 seq_num 185) 2024-03-19T11:31:07.103 INFO:tasks.rados.rados.0.smithi038.stdout:1236: read oid 35 snap 9 2024-03-19T11:31:07.103 INFO:tasks.rados.rados.0.smithi038.stdout:1236: expect (ObjNum 34 snap 0 seq_num 34) 2024-03-19T11:31:07.103 INFO:tasks.rados.rados.0.smithi038.stdout:1237: read oid 41 snap -1 2024-03-19T11:31:07.103 INFO:tasks.rados.rados.0.smithi038.stdout:1237: expect (ObjNum 170 snap 17 seq_num 170) 2024-03-19T11:31:07.104 INFO:tasks.rados.rados.0.smithi038.stdout:1238: read oid 45 snap -1 2024-03-19T11:31:07.104 INFO:tasks.rados.rados.0.smithi038.stdout:1238: expect (ObjNum 148 snap 15 seq_num 148) 2024-03-19T11:31:07.104 INFO:tasks.rados.rados.0.smithi038.stdout:1239: set_chunk oid 29 target oid 2024-03-19T11:31:07.134 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:07.141+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:07.141 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:07.142 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:07.148 INFO:tasks.rados.rados.0.smithi038.stdout:1239: set_chunk oid 29 offset: 732160 length: 2334208 target oid 5a94eec1c857dfa61dadccb9fb9dc1c446f82f4d3b95213b8234945bef32867c offset: 0 snap -1 2024-03-19T11:31:07.148 INFO:tasks.rados.rados.0.smithi038.stdout:1240: tier_evict oid 3 2024-03-19T11:31:07.148 INFO:tasks.rados.rados.0.smithi038.stdout:1240: tier_evict oid 3 snap -1 2024-03-19T11:31:07.148 INFO:tasks.rados.rados.0.smithi038.stdout:1233: finishing set_chunk to oid 20 2024-03-19T11:31:07.148 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 20 set_chunk returned error code -95 offset: 1358848 length: 685568 2024-03-19T11:31:07.148 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 20 v 0 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-03-19T11:31:07.148 INFO:tasks.rados.rados.0.smithi038.stdout:1235: expect (ObjNum 185 snap 20 seq_num 185) 2024-03-19T11:31:07.157 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-19T11:31:07.235 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:07.241+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:07.247 INFO:tasks.rados.rados.0.smithi038.stdout:1238: expect (ObjNum 148 snap 15 seq_num 148) 2024-03-19T11:31:07.261 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-19T11:31:07.282 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:07.282 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:07.297 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-19T11:31:07.343 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:07.355+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:07.393 INFO:tasks.rados.rados.0.smithi038.stdout:1234: expect (ObjNum 65 snap 3 seq_num 65) 2024-03-19T11:31:07.410 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:31:07.411 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:31:07.422 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:07.422 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:07.426 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:31:07.426 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:31:07.435 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-19T11:31:07.437 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:07.445+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:07.443 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:31:07.443 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:31:07.460 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:31:07.481 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:31:07.481 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:31:07.498 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:31:07.498 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:31:07.518 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:31:07.519 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:31:07.533 INFO:tasks.rados.rados.0.smithi038.stdout:1237: expect (ObjNum 170 snap 17 seq_num 170) 2024-03-19T11:31:07.539 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:31:07.539 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:31:07.540 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:07.551+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:07.602 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:07.602 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:31:07.602 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:31:07.602 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:31:07.602 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:07.618 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-19T11:31:07.622 INFO:tasks.rados.rados.0.smithi038.stdout:1236: expect (ObjNum 34 snap 0 seq_num 34) 2024-03-19T11:31:07.660 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:07.671+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:07.702 INFO:tasks.rados.rados.0.smithi038.stdout:1240: got (22) Invalid argument 2024-03-19T11:31:07.702 INFO:tasks.rados.rados.0.smithi038.stdout:1239: finishing set_chunk to oid 29 2024-03-19T11:31:07.702 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 29 v 81 (ObjNum 191 snap 22 seq_num 191) dirty exists 2024-03-19T11:31:07.702 INFO:tasks.rados.rados.0.smithi038.stdout:1233: done (7 left) 2024-03-19T11:31:07.702 INFO:tasks.rados.rados.0.smithi038.stdout:1234: done (6 left) 2024-03-19T11:31:07.702 INFO:tasks.rados.rados.0.smithi038.stdout:1235: done (5 left) 2024-03-19T11:31:07.702 INFO:tasks.rados.rados.0.smithi038.stdout:1236: done (4 left) 2024-03-19T11:31:07.702 INFO:tasks.rados.rados.0.smithi038.stdout:1237: done (3 left) 2024-03-19T11:31:07.703 INFO:tasks.rados.rados.0.smithi038.stdout:1238: done (2 left) 2024-03-19T11:31:07.703 INFO:tasks.rados.rados.0.smithi038.stdout:1239: done (1 left) 2024-03-19T11:31:07.703 INFO:tasks.rados.rados.0.smithi038.stdout:1240: done (0 left) 2024-03-19T11:31:07.703 INFO:tasks.rados.rados.0.smithi038.stdout:1241: rollback oid 21 current snap is 22 2024-03-19T11:31:07.703 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 21 to 20 2024-03-19T11:31:07.703 INFO:tasks.rados.rados.0.smithi038.stdout:1242: write (excl) oid 15 current snap is 22 2024-03-19T11:31:07.703 INFO:tasks.rados.rados.0.smithi038.stdout:1242: seq_num 205 ranges {645435=451223,1879985=787072,3280184=165251} 2024-03-19T11:31:07.720 INFO:tasks.rados.rados.0.smithi038.stdout:1242: writing smithi03829856-15 from 645435 to 1096658 tid 1 2024-03-19T11:31:07.741 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:07.751+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:07.746 INFO:tasks.rados.rados.0.smithi038.stdout:1242: writing smithi03829856-15 from 1879985 to 2667057 tid 2 2024-03-19T11:31:07.756 INFO:tasks.rados.rados.0.smithi038.stdout:1242: writing smithi03829856-15 from 3280184 to 3445435 tid 3 2024-03-19T11:31:07.757 INFO:tasks.rados.rados.0.smithi038.stdout:1243: write oid 2 current snap is 22 2024-03-19T11:31:07.757 INFO:tasks.rados.rados.0.smithi038.stdout:1243: seq_num 206 ranges {662242=642825,2054820=530730,3307607=154635} 2024-03-19T11:31:07.772 INFO:tasks.rados.rados.0.smithi038.stdout:1243: writing smithi03829856-2 from 662242 to 1305067 tid 1 2024-03-19T11:31:07.784 INFO:tasks.rados.rados.0.smithi038.stdout:1243: writing smithi03829856-2 from 2054820 to 2585550 tid 2 2024-03-19T11:31:07.790 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:07.791 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:31:07.791 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:31:07.791 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:31:07.791 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.791 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:31:07.791 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:44.200920+0000", 2024-03-19T11:31:07.791 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 83.598167824000001, 2024-03-19T11:31:07.791 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.80881896499999995, 2024-03-19T11:31:07.791 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:07.791 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:07.791 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:07.791 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:31:07.792 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:31:07.792 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1 2024-03-19T11:31:07.792 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.793 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:07.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.793 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:07.793 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200920+0000", 2024-03-19T11:31:07.793 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:07.793 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.793 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:07.793 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200920+0000", 2024-03-19T11:31:07.793 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:07.793 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.793 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:07.793 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200923+0000", 2024-03-19T11:31:07.794 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6469999999999998e-06 2024-03-19T11:31:07.794 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.794 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.794 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:07.794 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200937+0000", 2024-03-19T11:31:07.794 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.399e-05 2024-03-19T11:31:07.794 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.794 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.795 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:07.795 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200960+0000", 2024-03-19T11:31:07.795 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2603e-05 2024-03-19T11:31:07.795 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.795 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:07.795 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200978+0000", 2024-03-19T11:31:07.795 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.791e-05 2024-03-19T11:31:07.795 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.796 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:07.796 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.201081+0000", 2024-03-19T11:31:07.796 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010344900000000001 2024-03-19T11:31:07.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.796 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.796 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:31:07.796 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.201096+0000", 2024-03-19T11:31:07.796 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4443e-05 2024-03-19T11:31:07.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.796 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.797 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:07.797 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.005978+0000", 2024-03-19T11:31:07.797 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.80488281500000003 2024-03-19T11:31:07.797 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.797 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.797 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:07.798 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.006267+0000", 2024-03-19T11:31:07.798 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000288688 2024-03-19T11:31:07.798 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.798 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.798 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:07.799 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.006534+0000", 2024-03-19T11:31:07.799 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026725799999999998 2024-03-19T11:31:07.799 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.799 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.799 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:07.799 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.008457+0000", 2024-03-19T11:31:07.799 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0019227820000000001 2024-03-19T11:31:07.799 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.800 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:07.800 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.009570+0000", 2024-03-19T11:31:07.800 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011131750000000001 2024-03-19T11:31:07.800 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.800 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:07.800 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.009670+0000", 2024-03-19T11:31:07.800 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9458999999999998e-05 2024-03-19T11:31:07.800 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.801 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:07.801 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.009739+0000", 2024-03-19T11:31:07.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9746000000000006e-05 2024-03-19T11:31:07.801 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:07.801 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:07.801 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:07.801 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.801 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.801 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:31:07.801 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.071728+0000", 2024-03-19T11:31:07.801 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 82.727360387000004, 2024-03-19T11:31:07.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013714690999999999, 2024-03-19T11:31:07.801 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:07.802 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:07.802 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:07.802 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:31:07.802 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:31:07.802 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 34 2024-03-19T11:31:07.802 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.802 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:07.802 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.802 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:07.802 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071728+0000", 2024-03-19T11:31:07.802 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:07.802 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.802 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.803 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:07.803 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071728+0000", 2024-03-19T11:31:07.803 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:07.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.803 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:07.803 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071729+0000", 2024-03-19T11:31:07.803 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.47e-07 2024-03-19T11:31:07.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.803 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:07.803 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071735+0000", 2024-03-19T11:31:07.803 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3799999999999999e-06 2024-03-19T11:31:07.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.804 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.804 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:07.804 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071748+0000", 2024-03-19T11:31:07.804 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2806000000000001e-05 2024-03-19T11:31:07.804 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.804 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.804 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:07.804 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071756+0000", 2024-03-19T11:31:07.804 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1240000000000005e-06 2024-03-19T11:31:07.804 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.804 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.804 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:07.804 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.077570+0000", 2024-03-19T11:31:07.805 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.005813342 2024-03-19T11:31:07.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.805 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.805 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:07.805 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.077793+0000", 2024-03-19T11:31:07.805 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022391500000000001 2024-03-19T11:31:07.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.805 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.805 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:07.805 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.078146+0000", 2024-03-19T11:31:07.805 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00035219 2024-03-19T11:31:07.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.805 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.805 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:07.806 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.083390+0000", 2024-03-19T11:31:07.806 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052445590000000002 2024-03-19T11:31:07.806 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.806 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.806 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:07.806 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085288+0000", 2024-03-19T11:31:07.806 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0018977060000000001 2024-03-19T11:31:07.806 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.807 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:07.807 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085377+0000", 2024-03-19T11:31:07.807 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9068999999999997e-05 2024-03-19T11:31:07.807 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.807 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:07.807 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085443+0000", 2024-03-19T11:31:07.807 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5653000000000003e-05 2024-03-19T11:31:07.807 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:07.807 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:07.807 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:07.807 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.808 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.808 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:31:07.808 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.071777+0000", 2024-03-19T11:31:07.808 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 82.727311276999998, 2024-03-19T11:31:07.808 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013962445, 2024-03-19T11:31:07.808 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:07.808 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:07.808 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:07.808 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:31:07.808 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:31:07.808 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 35 2024-03-19T11:31:07.808 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.809 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:07.809 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.809 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:07.809 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071777+0000", 2024-03-19T11:31:07.809 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:07.809 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.809 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.809 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:07.809 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071777+0000", 2024-03-19T11:31:07.809 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:07.810 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.810 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.810 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:07.810 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071778+0000", 2024-03-19T11:31:07.810 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5000000000000001e-07 2024-03-19T11:31:07.810 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.810 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.810 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:07.810 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071917+0000", 2024-03-19T11:31:07.810 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013880199999999999 2024-03-19T11:31:07.810 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.810 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.811 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:07.811 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071931+0000", 2024-03-19T11:31:07.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4080000000000001e-05 2024-03-19T11:31:07.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.811 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:07.811 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071940+0000", 2024-03-19T11:31:07.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.363e-06 2024-03-19T11:31:07.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.812 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:07.812 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.080462+0000", 2024-03-19T11:31:07.812 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085221810000000002 2024-03-19T11:31:07.812 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.812 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.812 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:07.812 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.080582+0000", 2024-03-19T11:31:07.812 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011935299999999999 2024-03-19T11:31:07.812 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.812 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.812 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:07.813 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.080782+0000", 2024-03-19T11:31:07.813 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020077100000000001 2024-03-19T11:31:07.813 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.813 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.813 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:07.813 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.083401+0000", 2024-03-19T11:31:07.813 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002618333 2024-03-19T11:31:07.813 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.813 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.813 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:07.813 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085571+0000", 2024-03-19T11:31:07.813 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021702829999999998 2024-03-19T11:31:07.813 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.813 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.814 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:07.814 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085672+0000", 2024-03-19T11:31:07.814 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100876 2024-03-19T11:31:07.814 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.814 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.814 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:07.814 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085739+0000", 2024-03-19T11:31:07.814 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7552999999999995e-05 2024-03-19T11:31:07.814 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:07.814 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:07.814 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:07.815 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.815 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.815 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:31:07.815 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.071957+0000", 2024-03-19T11:31:07.815 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 82.727131088999997, 2024-03-19T11:31:07.815 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013989822000000001, 2024-03-19T11:31:07.815 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:07.815 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:07.815 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:07.815 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:31:07.815 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:31:07.815 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 36 2024-03-19T11:31:07.815 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.816 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:07.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:07.816 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071957+0000", 2024-03-19T11:31:07.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:07.816 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:07.816 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071957+0000", 2024-03-19T11:31:07.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:07.816 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.817 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:07.817 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071958+0000", 2024-03-19T11:31:07.817 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.06e-06 2024-03-19T11:31:07.817 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.817 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.817 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:07.817 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071965+0000", 2024-03-19T11:31:07.817 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8929999999999999e-06 2024-03-19T11:31:07.817 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.817 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.817 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:07.817 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071976+0000", 2024-03-19T11:31:07.818 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1046999999999999e-05 2024-03-19T11:31:07.818 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.818 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.818 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:07.818 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071984+0000", 2024-03-19T11:31:07.818 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7170000000000008e-06 2024-03-19T11:31:07.818 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.818 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.818 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:07.818 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.081761+0000", 2024-03-19T11:31:07.818 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0097769310000000009 2024-03-19T11:31:07.818 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.818 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:07.819 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.081928+0000", 2024-03-19T11:31:07.819 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000167219 2024-03-19T11:31:07.819 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.819 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:07.819 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.082131+0000", 2024-03-19T11:31:07.819 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020244499999999999 2024-03-19T11:31:07.819 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.819 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:07.819 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.083455+0000", 2024-03-19T11:31:07.819 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001324073 2024-03-19T11:31:07.819 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.820 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.820 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:07.820 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085816+0000", 2024-03-19T11:31:07.820 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0023613309999999999 2024-03-19T11:31:07.820 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.820 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.820 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:07.820 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085892+0000", 2024-03-19T11:31:07.820 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6154000000000003e-05 2024-03-19T11:31:07.820 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.820 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.820 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:07.820 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085947+0000", 2024-03-19T11:31:07.820 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4951999999999998e-05 2024-03-19T11:31:07.820 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:07.820 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:07.821 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:07.821 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.821 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.821 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:31:07.821 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.072000+0000", 2024-03-19T11:31:07.821 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 82.727088339000005, 2024-03-19T11:31:07.821 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014418268999999999, 2024-03-19T11:31:07.821 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:07.821 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:07.821 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:07.821 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:31:07.821 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:31:07.821 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 37 2024-03-19T11:31:07.821 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.821 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:07.822 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.822 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:07.822 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072000+0000", 2024-03-19T11:31:07.822 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:07.822 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.822 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.822 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:07.822 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072000+0000", 2024-03-19T11:31:07.822 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:07.822 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.822 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.822 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:07.822 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072001+0000", 2024-03-19T11:31:07.823 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0699999999999996e-07 2024-03-19T11:31:07.823 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.823 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.823 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:07.823 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072007+0000", 2024-03-19T11:31:07.823 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2060000000000004e-06 2024-03-19T11:31:07.823 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.823 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.823 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:07.823 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072023+0000", 2024-03-19T11:31:07.823 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6617e-05 2024-03-19T11:31:07.823 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.823 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.823 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:07.824 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072031+0000", 2024-03-19T11:31:07.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8900000000000007e-06 2024-03-19T11:31:07.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.824 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:07.824 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.082682+0000", 2024-03-19T11:31:07.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010650868000000001 2024-03-19T11:31:07.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.824 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:07.824 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.082804+0000", 2024-03-19T11:31:07.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012202899999999999 2024-03-19T11:31:07.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.825 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:07.825 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.083006+0000", 2024-03-19T11:31:07.825 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020127799999999999 2024-03-19T11:31:07.825 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.825 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.825 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:07.825 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.084145+0000", 2024-03-19T11:31:07.825 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001139791 2024-03-19T11:31:07.825 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.825 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.825 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:07.825 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.086292+0000", 2024-03-19T11:31:07.825 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021461969999999999 2024-03-19T11:31:07.825 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.826 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.826 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:07.826 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.086364+0000", 2024-03-19T11:31:07.826 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2439000000000002e-05 2024-03-19T11:31:07.826 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.826 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.826 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:07.826 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.086418+0000", 2024-03-19T11:31:07.826 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4246999999999997e-05 2024-03-19T11:31:07.826 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:07.826 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:07.826 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:07.826 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.826 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.827 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:31:07.827 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.339898+0000", 2024-03-19T11:31:07.827 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 82.459190716999998, 2024-03-19T11:31:07.827 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019501793, 2024-03-19T11:31:07.827 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:07.827 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:07.827 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:07.827 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:31:07.827 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:31:07.827 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113 2024-03-19T11:31:07.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.828 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:07.828 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.828 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:07.828 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339898+0000", 2024-03-19T11:31:07.828 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:07.828 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.828 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.828 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:07.828 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339898+0000", 2024-03-19T11:31:07.828 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:07.828 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.829 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.829 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:07.829 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339899+0000", 2024-03-19T11:31:07.829 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.353e-06 2024-03-19T11:31:07.829 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.829 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.829 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:07.829 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339907+0000", 2024-03-19T11:31:07.829 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3000000000000002e-06 2024-03-19T11:31:07.829 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.829 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.829 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:07.829 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339924+0000", 2024-03-19T11:31:07.830 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6906999999999999e-05 2024-03-19T11:31:07.830 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.830 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.830 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:07.830 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339933+0000", 2024-03-19T11:31:07.830 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8969999999999997e-06 2024-03-19T11:31:07.830 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.831 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:07.831 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352239+0000", 2024-03-19T11:31:07.831 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012305814999999999 2024-03-19T11:31:07.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.831 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:07.831 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352466+0000", 2024-03-19T11:31:07.831 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022684100000000001 2024-03-19T11:31:07.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.831 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:07.831 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352816+0000", 2024-03-19T11:31:07.832 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00035030499999999997 2024-03-19T11:31:07.832 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.832 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:07.832 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.358956+0000", 2024-03-19T11:31:07.832 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061400079999999998 2024-03-19T11:31:07.833 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.833 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.833 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:07.833 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359118+0000", 2024-03-19T11:31:07.833 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016237599999999999 2024-03-19T11:31:07.833 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.833 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.833 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:07.833 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359287+0000", 2024-03-19T11:31:07.833 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000168488 2024-03-19T11:31:07.833 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.833 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.833 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:07.833 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359399+0000", 2024-03-19T11:31:07.834 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000112503 2024-03-19T11:31:07.834 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:07.834 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:07.834 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:07.834 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.834 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.834 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:31:07.834 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.339947+0000", 2024-03-19T11:31:07.835 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 82.459140950000005, 2024-03-19T11:31:07.835 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021923069999999999, 2024-03-19T11:31:07.835 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:07.835 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:07.835 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:07.835 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:31:07.835 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:31:07.835 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 114 2024-03-19T11:31:07.835 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.835 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:07.835 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.835 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:07.836 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339947+0000", 2024-03-19T11:31:07.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:07.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.836 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:07.836 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339947+0000", 2024-03-19T11:31:07.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:07.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.836 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:07.836 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339948+0000", 2024-03-19T11:31:07.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6000000000000003e-07 2024-03-19T11:31:07.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.837 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:07.837 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339956+0000", 2024-03-19T11:31:07.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1559999999999995e-06 2024-03-19T11:31:07.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.837 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:07.837 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339967+0000", 2024-03-19T11:31:07.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0669999999999999e-05 2024-03-19T11:31:07.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.837 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:07.837 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339975+0000", 2024-03-19T11:31:07.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0099999999999995e-06 2024-03-19T11:31:07.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.837 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:07.838 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.353672+0000", 2024-03-19T11:31:07.838 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013696583999999999 2024-03-19T11:31:07.838 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.838 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.838 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:07.838 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.353808+0000", 2024-03-19T11:31:07.838 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013652299999999999 2024-03-19T11:31:07.838 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.838 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.838 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:07.838 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.354036+0000", 2024-03-19T11:31:07.838 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022744799999999999 2024-03-19T11:31:07.838 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.838 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.838 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:07.839 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.360480+0000", 2024-03-19T11:31:07.839 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0064441630000000001 2024-03-19T11:31:07.839 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.839 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:07.839 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361722+0000", 2024-03-19T11:31:07.839 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001242754 2024-03-19T11:31:07.839 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.839 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:07.839 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361807+0000", 2024-03-19T11:31:07.839 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4605999999999996e-05 2024-03-19T11:31:07.839 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.840 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:07.840 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361870+0000", 2024-03-19T11:31:07.840 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3396000000000001e-05 2024-03-19T11:31:07.840 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:07.840 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:07.840 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:07.840 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.840 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.840 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:31:07.840 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.340017+0000", 2024-03-19T11:31:07.840 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 82.459071633999997, 2024-03-19T11:31:07.840 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022169355000000002, 2024-03-19T11:31:07.840 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:07.840 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:07.840 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:07.841 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:31:07.841 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:31:07.841 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 115 2024-03-19T11:31:07.841 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.841 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:07.841 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.841 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:07.841 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340017+0000", 2024-03-19T11:31:07.841 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:07.841 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.841 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.841 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:07.841 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340017+0000", 2024-03-19T11:31:07.842 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:07.842 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.842 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.842 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:07.842 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340018+0000", 2024-03-19T11:31:07.842 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.043e-06 2024-03-19T11:31:07.842 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.842 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.842 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:07.842 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340139+0000", 2024-03-19T11:31:07.842 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121502 2024-03-19T11:31:07.843 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.843 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:07.843 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340157+0000", 2024-03-19T11:31:07.843 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7382999999999999e-05 2024-03-19T11:31:07.843 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.843 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:07.843 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340166+0000", 2024-03-19T11:31:07.843 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2469999999999997e-06 2024-03-19T11:31:07.843 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.844 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:07.844 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.357470+0000", 2024-03-19T11:31:07.844 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017304119999999999 2024-03-19T11:31:07.844 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.844 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.844 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:07.844 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.357671+0000", 2024-03-19T11:31:07.844 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000200798 2024-03-19T11:31:07.844 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.844 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.844 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:07.844 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.358002+0000", 2024-03-19T11:31:07.844 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033090099999999998 2024-03-19T11:31:07.845 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.845 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.845 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:07.845 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361892+0000", 2024-03-19T11:31:07.845 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00389018 2024-03-19T11:31:07.845 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.845 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.845 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:07.845 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361983+0000", 2024-03-19T11:31:07.845 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0976000000000004e-05 2024-03-19T11:31:07.845 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.846 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.846 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:07.846 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362071+0000", 2024-03-19T11:31:07.846 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8022999999999994e-05 2024-03-19T11:31:07.846 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.846 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.846 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:07.846 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362186+0000", 2024-03-19T11:31:07.846 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115182 2024-03-19T11:31:07.846 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:07.846 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:07.846 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:07.847 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.847 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.847 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:31:07.847 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.340183+0000", 2024-03-19T11:31:07.847 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 82.458904802000006, 2024-03-19T11:31:07.847 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022279841000000002, 2024-03-19T11:31:07.847 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:07.847 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:07.847 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:07.847 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:31:07.848 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:31:07.848 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 116 2024-03-19T11:31:07.848 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.848 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:07.848 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.848 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:07.848 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340183+0000", 2024-03-19T11:31:07.848 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:07.848 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.848 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.848 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:07.848 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340183+0000", 2024-03-19T11:31:07.848 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:07.849 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.849 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.849 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:07.849 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340184+0000", 2024-03-19T11:31:07.849 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9000000000000005e-07 2024-03-19T11:31:07.849 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.849 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.849 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:07.849 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340191+0000", 2024-03-19T11:31:07.849 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0360000000000003e-06 2024-03-19T11:31:07.849 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.850 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.850 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:07.850 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340201+0000", 2024-03-19T11:31:07.850 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0844e-05 2024-03-19T11:31:07.850 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.850 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.850 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:07.850 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340209+0000", 2024-03-19T11:31:07.850 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7360000000000001e-06 2024-03-19T11:31:07.850 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.850 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.850 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:07.850 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359535+0000", 2024-03-19T11:31:07.850 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019325546999999998 2024-03-19T11:31:07.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.851 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.851 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:07.851 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359672+0000", 2024-03-19T11:31:07.851 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013706600000000001 2024-03-19T11:31:07.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.851 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.851 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:07.851 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359905+0000", 2024-03-19T11:31:07.851 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023334500000000001 2024-03-19T11:31:07.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.852 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.852 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:07.852 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362306+0000", 2024-03-19T11:31:07.852 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002401041 2024-03-19T11:31:07.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.852 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.852 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:07.852 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362329+0000", 2024-03-19T11:31:07.852 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3283999999999999e-05 2024-03-19T11:31:07.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.852 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.852 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:07.852 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362402+0000", 2024-03-19T11:31:07.853 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2459000000000005e-05 2024-03-19T11:31:07.853 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.853 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.853 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:07.853 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362463+0000", 2024-03-19T11:31:07.853 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1493000000000002e-05 2024-03-19T11:31:07.853 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:07.853 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:07.853 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:07.853 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.853 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.853 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:31:07.854 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.340222+0000", 2024-03-19T11:31:07.854 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 82.458866219000001, 2024-03-19T11:31:07.854 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022500218999999998, 2024-03-19T11:31:07.854 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:07.854 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:07.854 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:07.854 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:31:07.854 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:31:07.854 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 117 2024-03-19T11:31:07.854 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.854 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:07.854 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.855 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:07.855 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340222+0000", 2024-03-19T11:31:07.855 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:07.855 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.855 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.855 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:07.855 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340222+0000", 2024-03-19T11:31:07.855 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:07.855 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.855 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.855 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:07.855 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340223+0000", 2024-03-19T11:31:07.855 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7999999999999995e-07 2024-03-19T11:31:07.856 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.856 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.856 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:07.856 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340229+0000", 2024-03-19T11:31:07.856 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0399999999999998e-06 2024-03-19T11:31:07.856 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.856 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.856 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:07.856 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340239+0000", 2024-03-19T11:31:07.856 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0190000000000001e-05 2024-03-19T11:31:07.856 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.856 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.856 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:07.856 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340250+0000", 2024-03-19T11:31:07.857 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0720000000000001e-05 2024-03-19T11:31:07.857 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.857 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.857 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:07.857 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.360587+0000", 2024-03-19T11:31:07.857 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020337542 2024-03-19T11:31:07.857 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.857 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.857 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:07.857 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.360818+0000", 2024-03-19T11:31:07.857 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023120500000000001 2024-03-19T11:31:07.857 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.857 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.857 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:07.858 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361218+0000", 2024-03-19T11:31:07.858 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000399637 2024-03-19T11:31:07.858 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.858 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.858 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:07.858 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362483+0000", 2024-03-19T11:31:07.858 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012649200000000001 2024-03-19T11:31:07.858 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.858 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.858 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:07.858 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362538+0000", 2024-03-19T11:31:07.858 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4490000000000002e-05 2024-03-19T11:31:07.858 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.859 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.859 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:07.859 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362617+0000", 2024-03-19T11:31:07.859 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9839000000000006e-05 2024-03-19T11:31:07.859 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.859 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.859 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:07.859 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362722+0000", 2024-03-19T11:31:07.859 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104956 2024-03-19T11:31:07.859 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:07.859 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:07.859 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:07.859 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.859 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.859 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1047 4.8 e35/21 4:1d58db11:::smithi03829856-18:head v 35'32, mlcod=33'29)", 2024-03-19T11:31:07.860 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:13.225023+0000", 2024-03-19T11:31:07.860 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 54.574065433999998, 2024-03-19T11:31:07.860 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013059688999999999, 2024-03-19T11:31:07.860 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:07.860 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:07.860 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:07.860 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.860 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:07.860 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.225023+0000", 2024-03-19T11:31:07.860 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:07.860 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.860 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.860 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:07.860 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.225023+0000", 2024-03-19T11:31:07.860 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:07.861 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.861 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.861 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:07.861 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.225025+0000", 2024-03-19T11:31:07.861 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.17e-06 2024-03-19T11:31:07.861 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.861 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.861 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:07.861 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.225771+0000", 2024-03-19T11:31:07.861 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000746267 2024-03-19T11:31:07.861 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.861 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.861 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:07.861 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.225807+0000", 2024-03-19T11:31:07.862 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5487000000000001e-05 2024-03-19T11:31:07.862 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.862 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.862 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:07.862 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.225823+0000", 2024-03-19T11:31:07.862 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6215999999999999e-05 2024-03-19T11:31:07.862 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.862 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.862 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:07.862 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.225990+0000", 2024-03-19T11:31:07.862 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016680899999999999 2024-03-19T11:31:07.862 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.863 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.863 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:07.863 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.226058+0000", 2024-03-19T11:31:07.863 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8672999999999998e-05 2024-03-19T11:31:07.863 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.863 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.863 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:07.864 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.237940+0000", 2024-03-19T11:31:07.864 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011881311 2024-03-19T11:31:07.864 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.864 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.864 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:07.864 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.238083+0000", 2024-03-19T11:31:07.864 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014275600000000001 2024-03-19T11:31:07.864 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:07.864 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:07.864 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:07.864 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.864 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.864 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1067 4.0 e36/21 4:01018511:::smithi03829856-19:head v 36'18, mlcod=35'14)", 2024-03-19T11:31:07.864 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:13.952492+0000", 2024-03-19T11:31:07.864 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 53.846595977, 2024-03-19T11:31:07.865 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013554725, 2024-03-19T11:31:07.865 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:07.865 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:07.865 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:07.865 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.865 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:07.865 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.952492+0000", 2024-03-19T11:31:07.865 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:07.865 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.865 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.865 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:07.865 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.952492+0000", 2024-03-19T11:31:07.865 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:07.866 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.866 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.866 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:07.866 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.952493+0000", 2024-03-19T11:31:07.866 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9299999999999997e-07 2024-03-19T11:31:07.866 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.866 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.866 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:07.866 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.954109+0000", 2024-03-19T11:31:07.866 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0016163569999999999 2024-03-19T11:31:07.866 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.866 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.866 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:07.866 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.954135+0000", 2024-03-19T11:31:07.867 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5352999999999999e-05 2024-03-19T11:31:07.867 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.867 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.867 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:07.867 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.954149+0000", 2024-03-19T11:31:07.867 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3879999999999999e-05 2024-03-19T11:31:07.867 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.867 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.867 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:07.868 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.954288+0000", 2024-03-19T11:31:07.868 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000139476 2024-03-19T11:31:07.868 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.868 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.868 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:07.868 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.954356+0000", 2024-03-19T11:31:07.868 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7533000000000005e-05 2024-03-19T11:31:07.868 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.868 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.868 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:07.868 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.965391+0000", 2024-03-19T11:31:07.868 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011035764999999999 2024-03-19T11:31:07.869 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.869 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.869 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:07.869 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.966047+0000", 2024-03-19T11:31:07.869 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00065556800000000004 2024-03-19T11:31:07.869 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:07.869 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:07.869 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:07.869 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.869 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.869 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1122 4.8 e36/21 4:1d58db11:::smithi03829856-18:head v 36'42, mlcod=36'39)", 2024-03-19T11:31:07.869 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:14.172666+0000", 2024-03-19T11:31:07.870 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 53.626422202999997, 2024-03-19T11:31:07.870 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018015000999999999, 2024-03-19T11:31:07.870 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:07.870 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:07.870 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:07.870 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.870 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:07.870 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.172666+0000", 2024-03-19T11:31:07.871 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:07.871 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.871 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.871 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:07.871 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.172666+0000", 2024-03-19T11:31:07.871 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:07.871 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.871 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.871 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:07.871 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.172669+0000", 2024-03-19T11:31:07.871 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1470000000000001e-06 2024-03-19T11:31:07.871 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.872 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.872 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:07.872 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.173140+0000", 2024-03-19T11:31:07.872 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00047074200000000001 2024-03-19T11:31:07.872 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.872 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.872 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:07.872 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.173188+0000", 2024-03-19T11:31:07.872 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8287e-05 2024-03-19T11:31:07.873 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.873 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.873 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:07.873 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.173211+0000", 2024-03-19T11:31:07.873 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3023e-05 2024-03-19T11:31:07.873 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.873 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.873 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:07.873 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.173378+0000", 2024-03-19T11:31:07.873 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000166742 2024-03-19T11:31:07.873 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.873 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.874 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:07.874 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.173557+0000", 2024-03-19T11:31:07.874 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000178569 2024-03-19T11:31:07.874 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.874 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.874 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:07.874 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.190416+0000", 2024-03-19T11:31:07.874 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016859570000000001 2024-03-19T11:31:07.874 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.874 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.874 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:07.874 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.190681+0000", 2024-03-19T11:31:07.874 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026492099999999999 2024-03-19T11:31:07.875 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:07.875 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:07.875 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:07.875 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.875 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.875 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.5.0:43 4.8 e39/21 4:15708482:::smithi03829856-23:head v 39'50, mlcod=38'47)", 2024-03-19T11:31:07.875 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.842819+0000", 2024-03-19T11:31:07.875 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 50.9562697, 2024-03-19T11:31:07.875 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017626513, 2024-03-19T11:31:07.875 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:07.875 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:07.875 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:07.875 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.875 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:07.876 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842819+0000", 2024-03-19T11:31:07.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:07.876 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.876 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.876 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:07.876 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842819+0000", 2024-03-19T11:31:07.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:07.876 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.876 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.876 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:07.876 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842820+0000", 2024-03-19T11:31:07.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.39e-06 2024-03-19T11:31:07.876 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.876 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.877 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:07.877 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.846437+0000", 2024-03-19T11:31:07.877 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003616953 2024-03-19T11:31:07.877 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.877 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.877 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:07.877 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.846471+0000", 2024-03-19T11:31:07.877 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4133e-05 2024-03-19T11:31:07.877 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.877 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.877 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:07.877 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.846539+0000", 2024-03-19T11:31:07.877 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8072999999999997e-05 2024-03-19T11:31:07.877 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.877 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.878 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:07.878 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.847094+0000", 2024-03-19T11:31:07.878 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00055472099999999999 2024-03-19T11:31:07.878 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.878 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.878 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:07.878 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.847128+0000", 2024-03-19T11:31:07.878 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3875999999999999e-05 2024-03-19T11:31:07.878 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.878 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.878 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:07.878 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.860308+0000", 2024-03-19T11:31:07.878 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013180712000000001 2024-03-19T11:31:07.878 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.879 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.879 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:07.879 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.860445+0000", 2024-03-19T11:31:07.879 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013665500000000001 2024-03-19T11:31:07.879 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:07.879 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:07.879 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:07.879 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.880 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.880 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1209 4.8 e41/21 4:1d58db11:::smithi03829856-18:head v 41'53, mlcod=39'50)", 2024-03-19T11:31:07.880 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:18.124878+0000", 2024-03-19T11:31:07.880 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 49.674210006999999, 2024-03-19T11:31:07.880 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016809343000000001, 2024-03-19T11:31:07.880 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:07.880 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:07.880 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:07.880 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.880 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:07.880 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.124878+0000", 2024-03-19T11:31:07.880 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:07.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.880 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.880 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:07.881 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.124878+0000", 2024-03-19T11:31:07.881 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:07.881 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.881 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.881 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:07.881 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.124880+0000", 2024-03-19T11:31:07.881 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4959999999999999e-06 2024-03-19T11:31:07.881 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.881 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.881 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:07.881 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125845+0000", 2024-03-19T11:31:07.882 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00096535299999999996 2024-03-19T11:31:07.882 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.882 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.882 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:07.882 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125884+0000", 2024-03-19T11:31:07.882 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8645e-05 2024-03-19T11:31:07.882 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.882 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.882 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:07.882 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125909+0000", 2024-03-19T11:31:07.882 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4783e-05 2024-03-19T11:31:07.882 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.882 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.883 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:07.883 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.126085+0000", 2024-03-19T11:31:07.883 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000176649 2024-03-19T11:31:07.883 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.883 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.883 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:07.883 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.126150+0000", 2024-03-19T11:31:07.883 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4290999999999998e-05 2024-03-19T11:31:07.883 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.883 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.883 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:07.883 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.141552+0000", 2024-03-19T11:31:07.883 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015402295 2024-03-19T11:31:07.883 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.884 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.884 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:07.884 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.141688+0000", 2024-03-19T11:31:07.884 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013583099999999999 2024-03-19T11:31:07.884 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:07.884 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:07.884 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:07.884 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.884 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.884 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.15 51/50 [PushOp(4:afa9f208:::smithi03829856-35:head, version: 28'46, data_included: [571438~623878,1806623~424698], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:afa9f208:::smithi03829856-35:head@28'46, size: 2571438, copy_subset: [(0, 2571438)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2231321, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:af9d0d56:::smithi03829856-20:head, version: 35'55, data_included: [655360~589824,1900544~418789], data_size: 1008613, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:af9d0d56:::smithi03829856-20:head@35'55, size: 2319333, copy_subset: [(0, 2319333)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2319333, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ad3ad1b3:::smithi03829856-21:b, version: 47'66, data_included: [736140~537686,2020581~315559], data_size: 853245, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:ad3ad1b3:::smithi03829856-21:b@47'66, size: 2336140, copy_subset: [(0, 2336140)], clone_subset: {}, snapset: b=[]:{b: [b, a, 6, 5]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2336140, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ad3ad1b3:::smithi03829856-21:head, version: 47'70, data_included: [531121~742705,1711057~305871], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:ad3ad1b3:::smithi03829856-21:head@47'70, size: 3731121, copy_subset: [(0, 3731121)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2016928, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ac9cd3a3:::smithi03829856-8:head, version: 49'72, data_included: [517649~620182,1776668~428394], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:ac9cd3a3:::smithi03829856-8:head@49'72, size: 3008420, copy_subset: [(0, 3008420)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2205062, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:aa3303a5:::smithi03829856-17:b, version: 49'73, data_included: [614400~692224,1903006~356352], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:aa3303a5:::smithi03829856-17:b@49'73, size: 2268912, copy_subset: [(0, 2268912)], clone_subset: {}, snapset: b=[]:{b: [b, a, 6, 5]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2259358, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:aa3303a5:::smithi03829856-17:head, version: 49'74, data_included: [413472~893152,1818036~155424], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:aa3303a5:::smithi03829856-17:head@49'74, size: 3613472, copy_subset: [(0, 3613472)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1973460, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:07.884 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:28.058610+0000", 2024-03-19T11:31:07.884 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 39.740478047000003, 2024-03-19T11:31:07.885 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027100032, 2024-03-19T11:31:07.885 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:07.885 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:07.885 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:07.885 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.885 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:07.885 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.058610+0000", 2024-03-19T11:31:07.885 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:07.885 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.885 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.885 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:07.885 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.058610+0000", 2024-03-19T11:31:07.885 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:07.885 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.886 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.886 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:07.886 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.058611+0000", 2024-03-19T11:31:07.886 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1599999999999999e-06 2024-03-19T11:31:07.886 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.886 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.886 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:07.886 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.063874+0000", 2024-03-19T11:31:07.886 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052625270000000004 2024-03-19T11:31:07.886 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.886 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.886 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:07.887 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.063935+0000", 2024-03-19T11:31:07.887 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0594000000000003e-05 2024-03-19T11:31:07.887 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.887 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.887 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:07.887 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064024+0000", 2024-03-19T11:31:07.887 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9273999999999999e-05 2024-03-19T11:31:07.887 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.887 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.887 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:07.887 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064385+0000", 2024-03-19T11:31:07.887 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00036129100000000002 2024-03-19T11:31:07.887 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.887 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.887 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:07.888 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064437+0000", 2024-03-19T11:31:07.888 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2346000000000003e-05 2024-03-19T11:31:07.888 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.888 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.888 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:07.888 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085710+0000", 2024-03-19T11:31:07.888 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021272840000000001 2024-03-19T11:31:07.888 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:07.888 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:07.888 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:07.888 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.888 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.888 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.15 e51,50 4:ac9cd3a3:::smithi03829856-8:4,49'71)", 2024-03-19T11:31:07.889 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:28.064112+0000", 2024-03-19T11:31:07.889 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 39.734976220999997, 2024-03-19T11:31:07.889 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026480417999999999, 2024-03-19T11:31:07.889 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:07.889 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:07.889 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:07.889 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.889 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:07.889 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064112+0000", 2024-03-19T11:31:07.889 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:07.889 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.890 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.890 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:07.890 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064112+0000", 2024-03-19T11:31:07.890 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:07.890 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.890 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.890 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:07.890 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064113+0000", 2024-03-19T11:31:07.890 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3600000000000003e-07 2024-03-19T11:31:07.890 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.890 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.890 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:07.890 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064118+0000", 2024-03-19T11:31:07.890 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0549999999999999e-06 2024-03-19T11:31:07.891 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.891 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.891 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:07.891 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064128+0000", 2024-03-19T11:31:07.891 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0349e-05 2024-03-19T11:31:07.891 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.891 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.891 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:07.891 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064136+0000", 2024-03-19T11:31:07.891 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6249999999999998e-06 2024-03-19T11:31:07.891 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.891 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.891 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:07.891 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085773+0000", 2024-03-19T11:31:07.892 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021636829999999999 2024-03-19T11:31:07.892 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.892 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.892 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:07.892 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085788+0000", 2024-03-19T11:31:07.892 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5512999999999999e-05 2024-03-19T11:31:07.892 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.892 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.892 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:07.892 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.090592+0000", 2024-03-19T11:31:07.892 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0048044100000000003 2024-03-19T11:31:07.892 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:07.892 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:07.893 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:07.893 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.893 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.5 51/50 [PushOp(4:a38b9453:::smithi03829856-37:head, version: 28'45, data_included: [605052~774376,1923587~274200], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:a38b9453:::smithi03829856-37:head@28'45, size: 2605052, copy_subset: [(0, 2605052)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2197787, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:a0a50d04:::smithi03829856-13:b, version: 47'63, data_included: [601684~619341,1669078~429235], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a0a50d04:::smithi03829856-13:b@47'63, size: 2201684, copy_subset: [(0, 2201684)], clone_subset: {}, snapset: b=[]:{b: [b, a, 6, 5]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2098313, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:07.893 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:28.064324+0000", 2024-03-19T11:31:07.893 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 39.734764376999998, 2024-03-19T11:31:07.893 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027179020000000002, 2024-03-19T11:31:07.893 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:07.893 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:07.893 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:07.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.893 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:07.893 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064324+0000", 2024-03-19T11:31:07.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:07.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:07.894 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064324+0000", 2024-03-19T11:31:07.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:07.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:07.894 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064324+0000", 2024-03-19T11:31:07.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1199999999999998e-07 2024-03-19T11:31:07.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.895 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:07.895 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.065527+0000", 2024-03-19T11:31:07.895 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012028939999999999 2024-03-19T11:31:07.895 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.895 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.895 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:07.895 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.065552+0000", 2024-03-19T11:31:07.895 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4440999999999999e-05 2024-03-19T11:31:07.895 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.895 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.895 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:07.895 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.065568+0000", 2024-03-19T11:31:07.896 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6498e-05 2024-03-19T11:31:07.896 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.896 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.896 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:07.896 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085812+0000", 2024-03-19T11:31:07.896 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020243954000000002 2024-03-19T11:31:07.896 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.896 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.896 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:07.896 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085825+0000", 2024-03-19T11:31:07.896 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3298999999999999e-05 2024-03-19T11:31:07.897 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.897 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.897 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:07.897 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.091503+0000", 2024-03-19T11:31:07.897 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.005677522 2024-03-19T11:31:07.897 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:07.897 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:07.897 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:07.897 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.897 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.897 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.24 75/74 [PushOp(4:27909373:::smithi03829856-46:8, version: 46'59, data_included: [756315~767970,1933572~280606], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:8@46'59, size: 2756315, copy_subset: [(0, 2756315)], clone_subset: {}, snapset: 10=[]:{8: [6], c: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2214178, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:266590a5:::smithi03829856-16:e, version: 70'107, data_included: [652105~662359,1728171~386217], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:e@70'107, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 10=[]:{e: [d, b, 6]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:266590a5:::smithi03829856-16:head, version: 70'108, data_included: [652105~662359,1728171~386217], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:head@70'108, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:27909373:::smithi03829856-46:c, version: 70'109, data_included: [628595~895690,1807440~152886], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:c@70'109, size: 3428595, copy_subset: [(0, 3428595)], clone_subset: {}, snapset: 10=[]:{8: [6], c: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1960326, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:27909373:::smithi03829856-46:head, version: 70'110, data_included: [403401~454961,1584925~506039], data_size: 961000, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:head@70'110, size: 2403401, copy_subset: [(0, 2403401)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2403401, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:07.897 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.099172+0000", 2024-03-19T11:31:07.897 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 13.699915943000001, 2024-03-19T11:31:07.897 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.024528580000000001, 2024-03-19T11:31:07.898 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:07.898 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:07.898 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:07.898 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.898 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:07.898 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.099172+0000", 2024-03-19T11:31:07.898 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:07.898 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.898 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.898 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:07.898 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.099172+0000", 2024-03-19T11:31:07.898 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:07.898 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.898 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.899 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:07.899 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.099175+0000", 2024-03-19T11:31:07.899 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7750000000000001e-06 2024-03-19T11:31:07.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.899 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.899 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:07.899 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105219+0000", 2024-03-19T11:31:07.899 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0060440789999999999 2024-03-19T11:31:07.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.899 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.899 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:07.899 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105285+0000", 2024-03-19T11:31:07.899 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6260000000000006e-05 2024-03-19T11:31:07.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.900 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.900 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:07.900 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105366+0000", 2024-03-19T11:31:07.900 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.038e-05 2024-03-19T11:31:07.900 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.900 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.900 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:07.900 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105637+0000", 2024-03-19T11:31:07.900 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027085199999999997 2024-03-19T11:31:07.900 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.900 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.900 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:07.901 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105679+0000", 2024-03-19T11:31:07.901 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2088000000000001e-05 2024-03-19T11:31:07.901 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.901 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.901 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:07.901 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.123701+0000", 2024-03-19T11:31:07.901 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018022145999999999 2024-03-19T11:31:07.901 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:07.901 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:07.901 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:07.901 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.901 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.902 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.24 e75,74 4:266590a5:::smithi03829856-16:4,47'63,4:27909373:::smithi03829856-46:5,55'79,4:266590a5:::smithi03829856-16:10,67'105)", 2024-03-19T11:31:07.902 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.105437+0000", 2024-03-19T11:31:07.902 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 13.693650939999999, 2024-03-19T11:31:07.902 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.01860726, 2024-03-19T11:31:07.902 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:07.902 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:07.902 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:07.902 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.902 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:07.902 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105437+0000", 2024-03-19T11:31:07.902 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:07.902 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.902 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.903 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:07.903 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105437+0000", 2024-03-19T11:31:07.903 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:07.903 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.903 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.903 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:07.903 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105438+0000", 2024-03-19T11:31:07.903 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6799999999999999e-07 2024-03-19T11:31:07.903 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.904 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.904 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:07.904 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105445+0000", 2024-03-19T11:31:07.904 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3749999999999999e-06 2024-03-19T11:31:07.904 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.904 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.904 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:07.904 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105454+0000", 2024-03-19T11:31:07.904 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7909999999999999e-06 2024-03-19T11:31:07.904 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.904 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.904 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:07.904 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105465+0000", 2024-03-19T11:31:07.904 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.045e-05 2024-03-19T11:31:07.904 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.905 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.905 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:07.905 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.123753+0000", 2024-03-19T11:31:07.905 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0182884 2024-03-19T11:31:07.905 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.905 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.905 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:07.905 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.123765+0000", 2024-03-19T11:31:07.905 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2143e-05 2024-03-19T11:31:07.905 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:07.905 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:07.906 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:07.906 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.124045+0000", 2024-03-19T11:31:07.906 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027923299999999999 2024-03-19T11:31:07.906 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:07.906 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:07.906 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:07.906 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:07.906 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:07.906 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:07.909 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:07.919+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:07.909 INFO:tasks.rados.rados.0.smithi038.stdout:1243: writing smithi03829856-2 from 3307607 to 3462242 tid 3 2024-03-19T11:31:07.909 INFO:tasks.rados.rados.0.smithi038.stdout:1244: write oid 8 current snap is 22 2024-03-19T11:31:07.909 INFO:tasks.rados.rados.0.smithi038.stdout:1241: finishing rollback tid 0 to smithi03829856-21 2024-03-19T11:31:07.909 INFO:tasks.rados.rados.0.smithi038.stdout:1241: finishing rollback tid 1 to smithi03829856-21 2024-03-19T11:31:07.909 INFO:tasks.rados.rados.0.smithi038.stdout:1241: finishing rollback tid 2 to smithi03829856-21 2024-03-19T11:31:07.909 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 21 v 168 (ObjNum 129 snap 15 seq_num 129) dirty exists 2024-03-19T11:31:07.909 INFO:tasks.rados.rados.0.smithi038.stdout:1242: finishing write tid 1 to smithi03829856-15 2024-03-19T11:31:07.909 INFO:tasks.rados.rados.0.smithi038.stdout:1242: finishing write tid 2 to smithi03829856-15 2024-03-19T11:31:07.910 INFO:tasks.rados.rados.0.smithi038.stdout:1242: finishing write tid 3 to smithi03829856-15 2024-03-19T11:31:07.910 INFO:tasks.rados.rados.0.smithi038.stdout:1242: finishing write tid 4 to smithi03829856-15 2024-03-19T11:31:07.910 INFO:tasks.rados.rados.0.smithi038.stdout:1242: finishing write tid 5 to smithi03829856-15 2024-03-19T11:31:07.910 INFO:tasks.rados.rados.0.smithi038.stdout:1242: oid 15 updating version 0 to 26 2024-03-19T11:31:07.910 INFO:tasks.rados.rados.0.smithi038.stdout:1242: oid 15 updating version 26 to 27 2024-03-19T11:31:07.910 INFO:tasks.rados.rados.0.smithi038.stdout:1242: oid 15 version 27 is already newer than 25 2024-03-19T11:31:07.910 INFO:tasks.rados.rados.0.smithi038.stdout:1242: oid 15 version 27 is already newer than 24 2024-03-19T11:31:07.910 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 15 v 27 (ObjNum 205 snap 22 seq_num 205) dirty exists 2024-03-19T11:31:07.910 INFO:tasks.rados.rados.0.smithi038.stdout:1242: left oid 15 (ObjNum 205 snap 22 seq_num 205) 2024-03-19T11:31:07.910 INFO:tasks.rados.rados.0.smithi038.stdout:1243: finishing write tid 1 to smithi03829856-2 2024-03-19T11:31:07.910 INFO:tasks.rados.rados.0.smithi038.stdout:1243: finishing write tid 2 to smithi03829856-2 2024-03-19T11:31:07.910 INFO:tasks.rados.rados.0.smithi038.stdout:1244: seq_num 207 ranges {679049=718074,2113302=674388,3335030=144019} 2024-03-19T11:31:07.911 INFO:tasks.rados.rados.0.smithi038.stdout:1244: writing smithi03829856-8 from 679049 to 1397123 tid 1 2024-03-19T11:31:07.911 INFO:tasks.rados.rados.0.smithi038.stdout:1244: writing smithi03829856-8 from 2113302 to 2787690 tid 2 2024-03-19T11:31:07.911 INFO:tasks.rados.rados.0.smithi038.stdout:1244: writing smithi03829856-8 from 3335030 to 3479049 tid 3 2024-03-19T11:31:07.911 INFO:tasks.rados.rados.0.smithi038.stdout:1241: done (3 left) 2024-03-19T11:31:07.911 INFO:tasks.rados.rados.0.smithi038.stdout:1242: done (2 left) 2024-03-19T11:31:07.911 INFO:tasks.rados.rados.0.smithi038.stdout:1245: snap_remove snap 20 2024-03-19T11:31:07.911 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-19T11:31:08.009 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:08.019+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:08.719 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-03-19T11:31:08.721 INFO:tasks.rados.rados.0.smithi038.stdout:1245: done (2 left) 2024-03-19T11:31:08.721 INFO:tasks.rados.rados.0.smithi038.stdout:1246: read oid 9 snap 19 2024-03-19T11:31:08.721 INFO:tasks.rados.rados.0.smithi038.stdout:1246: expect (ObjNum 152 snap 17 seq_num 152) 2024-03-19T11:31:08.722 INFO:tasks.rados.rados.0.smithi038.stdout:1247: read oid 30 snap -1 2024-03-19T11:31:08.722 INFO:tasks.rados.rados.0.smithi038.stdout:1247: expect (ObjNum 29 snap 0 seq_num 29) 2024-03-19T11:31:08.722 INFO:tasks.rados.rados.0.smithi038.stdout:1248: read oid 39 snap -1 2024-03-19T11:31:08.722 INFO:tasks.rados.rados.0.smithi038.stdout:1248: expect (ObjNum 163 snap 17 seq_num 163) 2024-03-19T11:31:08.722 INFO:tasks.rados.rados.0.smithi038.stdout:1249: read oid 18 snap -1 2024-03-19T11:31:08.722 INFO:tasks.rados.rados.0.smithi038.stdout:1249: expect (ObjNum 199 snap 22 seq_num 199) 2024-03-19T11:31:08.722 INFO:tasks.rados.rados.0.smithi038.stdout:1250: read oid 41 snap 9 2024-03-19T11:31:08.722 INFO:tasks.rados.rados.0.smithi038.stdout:1250: expect (ObjNum 80 snap 6 seq_num 80) 2024-03-19T11:31:08.722 INFO:tasks.rados.rados.0.smithi038.stdout:1251: read oid 40 snap -1 2024-03-19T11:31:08.722 INFO:tasks.rados.rados.0.smithi038.stdout:1251: expect (ObjNum 141 snap 15 seq_num 141) 2024-03-19T11:31:08.722 INFO:tasks.rados.rados.0.smithi038.stdout:1252: delete oid 20 current snap is 22 2024-03-19T11:31:08.722 INFO:tasks.rados.rados.0.smithi038.stdout:1243: finishing write tid 3 to smithi03829856-2 2024-03-19T11:31:08.722 INFO:tasks.rados.rados.0.smithi038.stdout:1243: finishing write tid 4 to smithi03829856-2 2024-03-19T11:31:08.722 INFO:tasks.rados.rados.0.smithi038.stdout:1243: finishing write tid 5 to smithi03829856-2 2024-03-19T11:31:08.723 INFO:tasks.rados.rados.0.smithi038.stdout:1243: oid 2 updating version 0 to 72 2024-03-19T11:31:08.723 INFO:tasks.rados.rados.0.smithi038.stdout:1243: oid 2 updating version 72 to 74 2024-03-19T11:31:08.723 INFO:tasks.rados.rados.0.smithi038.stdout:1243: oid 2 version 74 is already newer than 73 2024-03-19T11:31:08.723 INFO:tasks.rados.rados.0.smithi038.stdout:1243: oid 2 updating version 74 to 75 2024-03-19T11:31:08.723 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 2 v 75 (ObjNum 206 snap 22 seq_num 206) dirty exists 2024-03-19T11:31:08.723 INFO:tasks.rados.rados.0.smithi038.stdout:1243: left oid 2 (ObjNum 206 snap 22 seq_num 206) 2024-03-19T11:31:08.723 INFO:tasks.rados.rados.0.smithi038.stdout:1244: finishing write tid 1 to smithi03829856-8 2024-03-19T11:31:08.723 INFO:tasks.rados.rados.0.smithi038.stdout:1244: finishing write tid 2 to smithi03829856-8 2024-03-19T11:31:08.723 INFO:tasks.rados.rados.0.smithi038.stdout:1244: finishing write tid 3 to smithi03829856-8 2024-03-19T11:31:08.723 INFO:tasks.rados.rados.0.smithi038.stdout:1244: finishing write tid 4 to smithi03829856-8 2024-03-19T11:31:08.723 INFO:tasks.rados.rados.0.smithi038.stdout:1244: finishing write tid 5 to smithi03829856-8 2024-03-19T11:31:08.723 INFO:tasks.rados.rados.0.smithi038.stdout:1244: oid 8 updating version 0 to 173 2024-03-19T11:31:08.723 INFO:tasks.rados.rados.0.smithi038.stdout:1244: oid 8 version 173 is already newer than 170 2024-03-19T11:31:08.723 INFO:tasks.rados.rados.0.smithi038.stdout:1244: oid 8 version 173 is already newer than 172 2024-03-19T11:31:08.723 INFO:tasks.rados.rados.0.smithi038.stdout:1244: oid 8 version 173 is already newer than 171 2024-03-19T11:31:08.724 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 8 v 173 (ObjNum 207 snap 22 seq_num 207) dirty exists 2024-03-19T11:31:08.724 INFO:tasks.rados.rados.0.smithi038.stdout:1244: left oid 8 (ObjNum 207 snap 22 seq_num 207) 2024-03-19T11:31:08.724 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:08.724 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:31:08.724 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:08.724 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2621 4.1e 4.7929455e (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e89)", 2024-03-19T11:31:08.724 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:08.055513+0000", 2024-03-19T11:31:08.724 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.0083263450000000006, 2024-03-19T11:31:08.724 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0083537160000000006, 2024-03-19T11:31:08.724 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:08.724 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "queued for pg", 2024-03-19T11:31:08.725 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:08.725 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:08.725 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:08.725 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2621 2024-03-19T11:31:08.725 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:08.725 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:08.725 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:08.725 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:08.725 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:08.055513+0000", 2024-03-19T11:31:08.725 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:08.725 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:08.725 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:08.725 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:08.725 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:08.055513+0000", 2024-03-19T11:31:08.725 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:08.725 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:08.726 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:08.726 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:08.726 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:08.055514+0000", 2024-03-19T11:31:08.726 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0499999999999999e-06 2024-03-19T11:31:08.726 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:08.726 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:08.726 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:08.726 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:08.055520+0000", 2024-03-19T11:31:08.726 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1359999999999996e-06 2024-03-19T11:31:08.726 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:08.726 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:08.726 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:08.726 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:08.055535+0000", 2024-03-19T11:31:08.726 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5979000000000001e-05 2024-03-19T11:31:08.727 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:08.727 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:08.727 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:08.727 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:08.055544+0000", 2024-03-19T11:31:08.727 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7669999999999992e-06 2024-03-19T11:31:08.727 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:08.727 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:08.727 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:08.727 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:08.728 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-19T11:31:08.728 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 1 2024-03-19T11:31:08.728 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:08.729 INFO:tasks.rados.rados.0.smithi038.stdout:1243: done (8 left) 2024-03-19T11:31:08.729 INFO:tasks.rados.rados.0.smithi038.stdout:1244: done (7 left) 2024-03-19T11:31:08.729 INFO:tasks.rados.rados.0.smithi038.stdout:1252: done (6 left) 2024-03-19T11:31:08.729 INFO:tasks.rados.rados.0.smithi038.stdout:1253: read oid 3 snap -1 2024-03-19T11:31:08.729 INFO:tasks.rados.rados.0.smithi038.stdout:1253: expect deleted 2024-03-19T11:31:08.729 INFO:tasks.rados.rados.0.smithi038.stdout:1254: write oid 15 current snap is 22 2024-03-19T11:31:08.729 INFO:tasks.rados.rados.0.smithi038.stdout:1254: seq_num 209 ranges {712663=468572,1713913=678057,2873523=591733} 2024-03-19T11:31:08.729 INFO:tasks.rados.rados.0.smithi038.stdout:1254: writing smithi03829856-15 from 712663 to 1181235 tid 1 2024-03-19T11:31:08.729 INFO:tasks.rados.rados.0.smithi038.stdout:1254: writing smithi03829856-15 from 1713913 to 2391970 tid 2 2024-03-19T11:31:08.731 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-19T11:31:08.733 INFO:tasks.rados.rados.0.smithi038.stdout:1254: writing smithi03829856-15 from 2873523 to 3465256 tid 3 2024-03-19T11:31:08.733 INFO:tasks.rados.rados.0.smithi038.stdout:1255: set_chunk oid 47 target oid 2024-03-19T11:31:08.733 INFO:tasks.rados.rados.0.smithi038.stdout:1255: set_chunk oid 47 offset: 261632 length: 1121792 target oid 513879a5dfd0acff39f0ae791621829b99e51a527d77cc41dfa6d546d77d3939 offset: 0 snap -1 2024-03-19T11:31:08.733 INFO:tasks.rados.rados.0.smithi038.stdout:1256: read oid 46 snap -1 2024-03-19T11:31:08.734 INFO:tasks.rados.rados.0.smithi038.stdout:1256: expect (ObjNum 143 snap 15 seq_num 143) 2024-03-19T11:31:08.734 INFO:tasks.rados.rados.0.smithi038.stdout:1257: set_chunk oid 5 target oid 2024-03-19T11:31:08.734 INFO:tasks.rados.rados.0.smithi038.stdout:1257: set_chunk oid 5 offset: 799744 length: 340480 target oid 0f47e8c1a9db80bbfaf17c21e0564835d2d06ca4745cea6f0934f6d84160c6d7 offset: 0 snap -1 2024-03-19T11:31:08.734 INFO:tasks.rados.rados.0.smithi038.stdout:1258: write oid 7 current snap is 22 2024-03-19T11:31:08.734 INFO:tasks.rados.rados.0.smithi038.stdout:1258: seq_num 210 ranges {729470=543821,1772395=538068,2900946=403005} 2024-03-19T11:31:08.734 INFO:tasks.rados.rados.0.smithi038.stdout:1258: writing smithi03829856-7 from 729470 to 1273291 tid 1 2024-03-19T11:31:08.734 INFO:tasks.rados.rados.0.smithi038.stdout:1258: writing smithi03829856-7 from 1772395 to 2310463 tid 2 2024-03-19T11:31:08.734 INFO:tasks.rados.rados.0.smithi038.stdout:1258: writing smithi03829856-7 from 2900946 to 3303951 tid 3 2024-03-19T11:31:08.734 INFO:tasks.rados.rados.0.smithi038.stdout:1259: set_chunk oid 24 target oid 2024-03-19T11:31:08.734 INFO:tasks.rados.rados.0.smithi038.stdout:1259: set_chunk oid 24 offset: 368640 length: 1660928 target oid 88500b1d171745ecce2b7c148cf6bf8fd1a433ddcaffc8d64403bb9540fadc15 offset: 0 snap -1 2024-03-19T11:31:08.734 INFO:tasks.rados.rados.0.smithi038.stdout:1260: tier_evict oid 4 2024-03-19T11:31:08.734 INFO:tasks.rados.rados.0.smithi038.stdout:1260: tier_evict oid 4 snap -1 2024-03-19T11:31:08.734 INFO:tasks.rados.rados.0.smithi038.stdout:1261: read oid 22 snap -1 2024-03-19T11:31:08.735 INFO:tasks.rados.rados.0.smithi038.stdout:1261: expect (ObjNum 168 snap 17 seq_num 168) 2024-03-19T11:31:08.735 INFO:tasks.rados.rados.0.smithi038.stdout:1262: delete oid 50 current snap is 22 2024-03-19T11:31:08.735 INFO:tasks.rados.rados.0.smithi038.stdout:1247: expect (ObjNum 29 snap 0 seq_num 29) 2024-03-19T11:31:08.735 INFO:tasks.rados.rados.0.smithi038.stdout:1250: expect (ObjNum 80 snap 6 seq_num 80) 2024-03-19T11:31:08.735 INFO:tasks.rados.rados.0.smithi038.stdout:1251: expect (ObjNum 141 snap 15 seq_num 141) 2024-03-19T11:31:08.735 INFO:tasks.rados.rados.0.smithi038.stdout:1248: expect (ObjNum 163 snap 17 seq_num 163) 2024-03-19T11:31:08.735 INFO:tasks.rados.rados.0.smithi038.stdout:1249: expect (ObjNum 199 snap 22 seq_num 199) 2024-03-19T11:31:08.735 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:08.733+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:08.824 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:08.829+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:08.893 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:08.893 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:31:08.893 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:31:08.893 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:31:08.893 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:08.915 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-19T11:31:08.961 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:08.969+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:09.007 INFO:tasks.rados.rados.0.smithi038.stdout:1246: expect (ObjNum 152 snap 17 seq_num 152) 2024-03-19T11:31:09.024 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:09.033+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:09.085 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:09.086 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:31:09.086 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:31:09.086 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:31:09.086 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.086 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1217 4.3 4:c8b09e87:::smithi03829856-15:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:31:09.086 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.155966+0000", 2024-03-19T11:31:09.086 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 50.936660996000001, 2024-03-19T11:31:09.087 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.023984647000000001, 2024-03-19T11:31:09.087 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:09.087 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:09.087 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:09.087 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:09.087 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:09.087 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1217 2024-03-19T11:31:09.087 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.087 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:09.087 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.088 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:09.088 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155966+0000", 2024-03-19T11:31:09.088 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:09.088 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.088 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.088 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:09.088 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155966+0000", 2024-03-19T11:31:09.088 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:09.088 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.088 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.089 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:09.089 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155968+0000", 2024-03-19T11:31:09.089 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.795e-06 2024-03-19T11:31:09.089 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.089 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.089 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:09.089 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155978+0000", 2024-03-19T11:31:09.089 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0574000000000001e-05 2024-03-19T11:31:09.089 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.089 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.090 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:09.090 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.156000+0000", 2024-03-19T11:31:09.090 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1250000000000002e-05 2024-03-19T11:31:09.090 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.090 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.090 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:09.091 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.156010+0000", 2024-03-19T11:31:09.091 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0168e-05 2024-03-19T11:31:09.091 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.091 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.091 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:09.091 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.156119+0000", 2024-03-19T11:31:09.091 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010950699999999999 2024-03-19T11:31:09.091 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.091 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.091 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:09.091 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.156231+0000", 2024-03-19T11:31:09.092 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000111445 2024-03-19T11:31:09.092 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.092 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.092 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:09.092 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.179951+0000", 2024-03-19T11:31:09.092 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.023719908000000001 2024-03-19T11:31:09.092 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:09.092 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:09.092 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:09.093 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.093 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.093 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.24 75/74 [PushOp(4:27909373:::smithi03829856-46:8, version: 46'59, data_included: [756315~767970,1933572~280606], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:8@46'59, size: 2756315, copy_subset: [(0, 2756315)], clone_subset: {}, snapset: 10=[]:{8: [6], c: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2214178, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:266590a5:::smithi03829856-16:e, version: 70'107, data_included: [652105~662359,1728171~386217], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:e@70'107, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 10=[]:{e: [d, b, 6]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:266590a5:::smithi03829856-16:head, version: 70'108, data_included: [652105~662359,1728171~386217], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:head@70'108, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:27909373:::smithi03829856-46:c, version: 70'109, data_included: [628595~895690,1807440~152886], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:c@70'109, size: 3428595, copy_subset: [(0, 3428595)], clone_subset: {}, snapset: 10=[]:{8: [6], c: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1960326, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:27909373:::smithi03829856-46:head, version: 70'110, data_included: [403401~454961,1584925~506039], data_size: 961000, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:head@70'110, size: 2403401, copy_subset: [(0, 2403401)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2403401, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:09.093 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.099395+0000", 2024-03-19T11:31:09.093 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 14.993231835, 2024-03-19T11:31:09.093 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.045391157000000001, 2024-03-19T11:31:09.093 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:09.093 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:09.093 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:09.094 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.094 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:09.094 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.099395+0000", 2024-03-19T11:31:09.094 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:09.094 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.094 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.094 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:09.094 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.099395+0000", 2024-03-19T11:31:09.094 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:09.094 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.095 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.095 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:09.095 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.099397+0000", 2024-03-19T11:31:09.095 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.691e-06 2024-03-19T11:31:09.095 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.095 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.095 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:09.095 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105009+0000", 2024-03-19T11:31:09.096 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0056121349999999999 2024-03-19T11:31:09.096 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.096 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.096 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:09.096 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105077+0000", 2024-03-19T11:31:09.096 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8561999999999998e-05 2024-03-19T11:31:09.096 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.097 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.097 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:09.097 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105133+0000", 2024-03-19T11:31:09.097 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5590000000000001e-05 2024-03-19T11:31:09.097 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.097 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.097 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:09.097 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105531+0000", 2024-03-19T11:31:09.097 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00039756200000000001 2024-03-19T11:31:09.097 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.098 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.098 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:09.098 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105579+0000", 2024-03-19T11:31:09.098 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8028999999999997e-05 2024-03-19T11:31:09.098 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.098 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.098 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:09.099 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.144786+0000", 2024-03-19T11:31:09.099 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.039207588000000002 2024-03-19T11:31:09.099 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:09.099 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:09.099 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:09.099 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.099 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.099 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.24 e75,74 4:266590a5:::smithi03829856-16:4,47'63,4:27909373:::smithi03829856-46:5,55'79,4:266590a5:::smithi03829856-16:10,67'105)", 2024-03-19T11:31:09.100 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.105191+0000", 2024-03-19T11:31:09.100 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 14.987435735, 2024-03-19T11:31:09.100 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.039580830999999997, 2024-03-19T11:31:09.100 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:09.100 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:09.100 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:09.100 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.100 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:09.101 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105191+0000", 2024-03-19T11:31:09.101 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:09.101 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.101 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.101 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:09.101 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105191+0000", 2024-03-19T11:31:09.101 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:09.101 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.101 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.102 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:09.102 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105192+0000", 2024-03-19T11:31:09.102 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.099e-06 2024-03-19T11:31:09.102 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.102 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.102 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:09.102 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105200+0000", 2024-03-19T11:31:09.102 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5279999999999998e-06 2024-03-19T11:31:09.103 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.103 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.103 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:09.103 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105211+0000", 2024-03-19T11:31:09.103 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1471000000000001e-05 2024-03-19T11:31:09.103 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.103 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.104 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:09.104 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105220+0000", 2024-03-19T11:31:09.104 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3469999999999999e-06 2024-03-19T11:31:09.104 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.104 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.104 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:09.104 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.144219+0000", 2024-03-19T11:31:09.104 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.038999024 2024-03-19T11:31:09.105 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.105 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.105 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:09.105 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.144257+0000", 2024-03-19T11:31:09.105 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8501999999999998e-05 2024-03-19T11:31:09.105 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.105 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.105 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:09.106 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.144772+0000", 2024-03-19T11:31:09.106 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00051486000000000001 2024-03-19T11:31:09.106 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:09.106 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:09.106 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:09.106 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.106 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.106 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 75/74 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:7, version: 55'82, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@55'82, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 58'86, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@58'86, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [504210~968657,1572695~79919], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:12, version: 66'101, data_included: [440350~816629,1309629~231947], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:12@66'101, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:10, version: 70'112, data_included: [440350~816629,1309629~231947], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:10@70'112, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 70'113, data_included: [440350~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@70'113, size: 2605085, copy_subset: [(0, 2605085)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1488926, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:09.107 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.107266+0000", 2024-03-19T11:31:09.107 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 14.985361218, 2024-03-19T11:31:09.107 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.048592669999999998, 2024-03-19T11:31:09.107 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:09.107 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:09.107 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:09.107 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.107 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:09.107 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.107266+0000", 2024-03-19T11:31:09.108 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:09.108 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.108 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.108 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:09.108 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.107266+0000", 2024-03-19T11:31:09.108 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:09.108 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.108 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.108 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:09.109 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.107267+0000", 2024-03-19T11:31:09.109 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1459999999999999e-06 2024-03-19T11:31:09.109 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.109 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.109 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:09.109 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.122861+0000", 2024-03-19T11:31:09.109 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01559372 2024-03-19T11:31:09.109 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.109 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.110 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:09.110 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.122932+0000", 2024-03-19T11:31:09.110 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1477999999999996e-05 2024-03-19T11:31:09.110 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.110 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.110 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:09.110 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.122999+0000", 2024-03-19T11:31:09.110 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6832e-05 2024-03-19T11:31:09.110 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.111 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.111 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:09.111 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123332+0000", 2024-03-19T11:31:09.111 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000333271 2024-03-19T11:31:09.111 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.111 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.111 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:09.111 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123371+0000", 2024-03-19T11:31:09.111 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8525999999999997e-05 2024-03-19T11:31:09.112 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.112 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.112 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:09.112 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.155858+0000", 2024-03-19T11:31:09.112 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.032487697000000003 2024-03-19T11:31:09.112 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:09.112 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:09.112 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:09.112 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.112 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.112 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.21 e75,74 4:8401c1f0:::smithi03829856-43:8,46'66,4:866f3dcc:::smithi03829856-31:2,47'73)", 2024-03-19T11:31:09.113 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.123075+0000", 2024-03-19T11:31:09.113 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 14.969551999, 2024-03-19T11:31:09.113 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.033023045000000001, 2024-03-19T11:31:09.113 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:09.113 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:09.113 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:09.113 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.113 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:09.114 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123075+0000", 2024-03-19T11:31:09.114 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:09.114 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.114 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.114 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:09.114 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123075+0000", 2024-03-19T11:31:09.114 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:09.114 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.114 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.115 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:09.115 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123076+0000", 2024-03-19T11:31:09.115 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5e-06 2024-03-19T11:31:09.115 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.115 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.115 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:09.115 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123086+0000", 2024-03-19T11:31:09.115 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8060000000000006e-06 2024-03-19T11:31:09.116 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.116 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.116 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:09.116 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123099+0000", 2024-03-19T11:31:09.116 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3251000000000001e-05 2024-03-19T11:31:09.116 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.116 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.116 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:09.116 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123112+0000", 2024-03-19T11:31:09.117 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2774e-05 2024-03-19T11:31:09.117 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.117 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.117 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:09.117 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.155902+0000", 2024-03-19T11:31:09.117 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.032790225999999999 2024-03-19T11:31:09.117 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.117 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.117 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:09.118 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.155914+0000", 2024-03-19T11:31:09.118 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1476e-05 2024-03-19T11:31:09.118 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.118 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.118 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:09.118 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.156098+0000", 2024-03-19T11:31:09.118 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000184012 2024-03-19T11:31:09.118 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:09.118 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:09.119 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:09.119 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.119 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.119 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 75/74 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:7, version: 55'82, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@55'82, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 58'86, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@58'86, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [1652614~973634,2888511~52681,3182305~22261], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3204566, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:12, version: 66'101, data_included: [1541576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:12@66'101, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:10, version: 70'112, data_included: [1541576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:10@70'112, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 70'113, data_included: [1488926~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@70'113, size: 2605085, copy_subset: [(0, 2605085)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2537502, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1488926, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:09.119 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.174030+0000", 2024-03-19T11:31:09.119 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 14.918596794999999, 2024-03-19T11:31:09.119 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.040799275000000003, 2024-03-19T11:31:09.120 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:09.120 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:09.120 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:09.120 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.120 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:09.120 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.174030+0000", 2024-03-19T11:31:09.120 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:09.120 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.120 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.120 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:09.121 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.174030+0000", 2024-03-19T11:31:09.121 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:09.121 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.121 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.121 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:09.121 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.174031+0000", 2024-03-19T11:31:09.121 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.322e-06 2024-03-19T11:31:09.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.122 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:09.122 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.184771+0000", 2024-03-19T11:31:09.122 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010739227 2024-03-19T11:31:09.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.122 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:09.122 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.184872+0000", 2024-03-19T11:31:09.122 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000101234 2024-03-19T11:31:09.123 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.123 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.123 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:09.123 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.184985+0000", 2024-03-19T11:31:09.123 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000113545 2024-03-19T11:31:09.123 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.123 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.123 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:09.123 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.185463+0000", 2024-03-19T11:31:09.123 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00047749899999999999 2024-03-19T11:31:09.124 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.124 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.124 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:09.124 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.185534+0000", 2024-03-19T11:31:09.124 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1007999999999995e-05 2024-03-19T11:31:09.124 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.124 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.125 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:09.125 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.214829+0000", 2024-03-19T11:31:09.125 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029295439999999999 2024-03-19T11:31:09.125 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:09.125 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:09.125 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:09.125 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.125 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.126 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.21 e75,74 4:866f3dcc:::smithi03829856-31:c,70'114)", 2024-03-19T11:31:09.126 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.197457+0000", 2024-03-19T11:31:09.126 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 14.895169749000001, 2024-03-19T11:31:09.126 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018493590000000001, 2024-03-19T11:31:09.126 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:09.126 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:09.126 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:09.126 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.127 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:09.127 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.197457+0000", 2024-03-19T11:31:09.127 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:09.127 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.127 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.127 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:09.127 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.197457+0000", 2024-03-19T11:31:09.127 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:09.127 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.127 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.128 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:09.128 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.197459+0000", 2024-03-19T11:31:09.128 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3199999999999998e-06 2024-03-19T11:31:09.128 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.128 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.128 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:09.128 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.197480+0000", 2024-03-19T11:31:09.128 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0366000000000001e-05 2024-03-19T11:31:09.128 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.128 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.129 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:09.129 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.197521+0000", 2024-03-19T11:31:09.129 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0738000000000002e-05 2024-03-19T11:31:09.129 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.129 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.129 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:09.129 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.197547+0000", 2024-03-19T11:31:09.129 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6548999999999999e-05 2024-03-19T11:31:09.129 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.130 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.130 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:09.130 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.214954+0000", 2024-03-19T11:31:09.130 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01740713 2024-03-19T11:31:09.130 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.130 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.130 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:09.130 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.214972+0000", 2024-03-19T11:31:09.130 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7520000000000002e-05 2024-03-19T11:31:09.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.131 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.131 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:09.131 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.215951+0000", 2024-03-19T11:31:09.131 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00097896700000000007 2024-03-19T11:31:09.131 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:09.131 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:09.131 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:09.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.131 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.132 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 75/74 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [2599014~241336], data_size: 241336, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2840350, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [2599014~241336,2948041~446940], data_size: 688276, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3394981, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [3204566~621403], data_size: 621403, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3865577, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3204566, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:12, version: 66'101, data_included: [2590152~250198,2948041~581396], data_size: 831594, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:12@66'101, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3529437, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:10, version: 70'112, data_included: [2590152~250198,2948041~581396], data_size: 831594, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:10@70'112, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3529437, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 70'113, data_included: [2537502~67583], data_size: 67583, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@70'113, size: 2605085, copy_subset: [(0, 2605085)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2605085, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2537502, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:09.132 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.222872+0000", 2024-03-19T11:31:09.132 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 14.869755302, 2024-03-19T11:31:09.132 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016277658, 2024-03-19T11:31:09.132 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:09.132 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:09.132 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:09.132 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.132 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:09.132 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.222872+0000", 2024-03-19T11:31:09.133 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:09.133 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.133 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.133 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:09.133 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.222872+0000", 2024-03-19T11:31:09.133 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:09.133 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.133 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.133 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:09.133 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.222874+0000", 2024-03-19T11:31:09.134 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9999999999999999e-06 2024-03-19T11:31:09.134 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.134 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.134 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:09.134 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.227563+0000", 2024-03-19T11:31:09.134 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046897570000000001 2024-03-19T11:31:09.134 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.134 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.134 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:09.134 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.227639+0000", 2024-03-19T11:31:09.135 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5607999999999998e-05 2024-03-19T11:31:09.135 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.135 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.135 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:09.135 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.227736+0000", 2024-03-19T11:31:09.135 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6854000000000005e-05 2024-03-19T11:31:09.135 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.136 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.136 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:09.136 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.228149+0000", 2024-03-19T11:31:09.136 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00041330000000000002 2024-03-19T11:31:09.136 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.136 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.136 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:09.136 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.228219+0000", 2024-03-19T11:31:09.136 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.0140000000000003e-05 2024-03-19T11:31:09.136 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.137 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.137 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:09.137 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.239149+0000", 2024-03-19T11:31:09.137 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010929998999999999 2024-03-19T11:31:09.137 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:09.137 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:09.137 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:09.137 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.137 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.138 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2372 4.16 4:6868af91:::smithi03829856-10:b [read 832512~818688] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-03-19T11:31:09.138 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:58.455732+0000", 2024-03-19T11:31:09.138 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 10.63689508, 2024-03-19T11:31:09.138 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.69150541200000004, 2024-03-19T11:31:09.138 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:09.138 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:09.138 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:09.139 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:09.139 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:09.139 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2372 2024-03-19T11:31:09.139 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.139 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:09.139 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.139 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:09.139 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455732+0000", 2024-03-19T11:31:09.140 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:09.140 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.140 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.140 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:09.140 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455732+0000", 2024-03-19T11:31:09.140 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:09.140 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.140 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.141 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:09.141 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455735+0000", 2024-03-19T11:31:09.141 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.756e-06 2024-03-19T11:31:09.141 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.141 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.141 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:09.141 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455744+0000", 2024-03-19T11:31:09.141 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1880000000000008e-06 2024-03-19T11:31:09.141 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.142 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.142 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:09.142 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455764+0000", 2024-03-19T11:31:09.142 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0044999999999999e-05 2024-03-19T11:31:09.142 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.142 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.142 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:09.143 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455774+0000", 2024-03-19T11:31:09.143 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8770000000000005e-06 2024-03-19T11:31:09.143 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.143 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.143 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:09.143 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455837+0000", 2024-03-19T11:31:09.143 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3588000000000002e-05 2024-03-19T11:31:09.143 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.144 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.144 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:09.144 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455846+0000", 2024-03-19T11:31:09.144 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6030000000000005e-06 2024-03-19T11:31:09.144 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.144 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.144 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:09.144 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.076525+0000", 2024-03-19T11:31:09.145 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.62067875900000002 2024-03-19T11:31:09.145 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.145 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.145 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:31:09.145 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.076981+0000", 2024-03-19T11:31:09.145 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00045633300000000001 2024-03-19T11:31:09.145 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.145 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.145 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:09.145 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.104422+0000", 2024-03-19T11:31:09.146 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027440665999999999 2024-03-19T11:31:09.146 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.146 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.146 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:31:09.146 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.104786+0000", 2024-03-19T11:31:09.146 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00036471499999999999 2024-03-19T11:31:09.146 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.146 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.147 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:09.147 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.140725+0000", 2024-03-19T11:31:09.147 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.035939093999999998 2024-03-19T11:31:09.147 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.147 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.147 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:31:09.147 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.140841+0000", 2024-03-19T11:31:09.148 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000115207 2024-03-19T11:31:09.148 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.148 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.148 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:09.148 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.145559+0000", 2024-03-19T11:31:09.148 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.004718486 2024-03-19T11:31:09.148 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.148 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.148 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:09.148 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.145760+0000", 2024-03-19T11:31:09.148 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000201058 2024-03-19T11:31:09.149 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.149 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.149 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:09.149 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.147237+0000", 2024-03-19T11:31:09.149 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014770370000000001 2024-03-19T11:31:09.149 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:09.149 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:09.150 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:09.150 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.150 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.150 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.16 80/79 [PushOp(4:6d9093b1:::smithi03829856-38:head, version: 77'90, data_included: [500875~689757,1564955~358819], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:6d9093b1:::smithi03829856-38:head@77'90, size: 2100875, copy_subset: [(0, 2100875)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1923774, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:6be89c06:::smithi03829856-24:head, version: 77'92, data_included: [741376~589824,1753088~458752], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:6be89c06:::smithi03829856-24:head@77'92, size: 2386561, copy_subset: [(0, 2386561)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2211840, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:09.150 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:59.081830+0000", 2024-03-19T11:31:09.150 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 10.010797053999999, 2024-03-19T11:31:09.150 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015229652999999999, 2024-03-19T11:31:09.150 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:09.151 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:09.151 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:09.151 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.151 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:09.151 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.081830+0000", 2024-03-19T11:31:09.151 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:09.151 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.151 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.151 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:09.152 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.081830+0000", 2024-03-19T11:31:09.152 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:09.152 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.152 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.152 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:09.152 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.081832+0000", 2024-03-19T11:31:09.152 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.731e-06 2024-03-19T11:31:09.152 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.152 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.152 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:09.152 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.083140+0000", 2024-03-19T11:31:09.153 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0013082829999999999 2024-03-19T11:31:09.153 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.153 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.153 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:09.153 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.083170+0000", 2024-03-19T11:31:09.153 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0270999999999999e-05 2024-03-19T11:31:09.153 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.153 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.153 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:09.153 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.083199+0000", 2024-03-19T11:31:09.153 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9298e-05 2024-03-19T11:31:09.154 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.154 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.154 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:09.154 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.089969+0000", 2024-03-19T11:31:09.154 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0067691289999999996 2024-03-19T11:31:09.154 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.154 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.154 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:09.155 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.089995+0000", 2024-03-19T11:31:09.155 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5995000000000001e-05 2024-03-19T11:31:09.155 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.155 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.155 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:09.155 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.097060+0000", 2024-03-19T11:31:09.155 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0070649459999999999 2024-03-19T11:31:09.156 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:09.156 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:09.156 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:09.156 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.156 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.156 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.6 80/79 [PushOp(4:626acd4d:::smithi03829856-2:8, version: 55'54, data_included: [416807~475249,1542129~474678], data_size: 949927, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:626acd4d:::smithi03829856-2:8@55'54, size: 2016807, copy_subset: [(0, 2016807)], clone_subset: {}, snapset: 15=[]:{8: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2016807, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:09.156 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:59.110928+0000", 2024-03-19T11:31:09.156 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 9.9816987400000006, 2024-03-19T11:31:09.157 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01605765, 2024-03-19T11:31:09.157 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:09.157 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:09.157 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:09.157 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.157 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:09.157 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.110928+0000", 2024-03-19T11:31:09.157 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:09.157 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.157 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.157 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:09.158 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.110928+0000", 2024-03-19T11:31:09.158 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:09.158 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.158 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.158 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:09.158 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.110929+0000", 2024-03-19T11:31:09.158 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6500000000000008e-07 2024-03-19T11:31:09.158 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.158 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.158 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:09.159 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.111446+0000", 2024-03-19T11:31:09.159 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00051713900000000001 2024-03-19T11:31:09.159 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.159 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.159 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:09.159 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.111464+0000", 2024-03-19T11:31:09.159 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7966e-05 2024-03-19T11:31:09.159 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.159 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.160 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:09.160 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.111483+0000", 2024-03-19T11:31:09.160 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8561000000000001e-05 2024-03-19T11:31:09.160 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.160 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.160 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:09.160 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.118569+0000", 2024-03-19T11:31:09.160 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0070860259999999996 2024-03-19T11:31:09.160 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.160 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.161 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:09.161 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.118586+0000", 2024-03-19T11:31:09.161 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6857000000000001e-05 2024-03-19T11:31:09.161 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.161 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.161 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:09.161 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.126986+0000", 2024-03-19T11:31:09.161 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0084001360000000008 2024-03-19T11:31:09.161 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:09.162 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:09.162 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:09.162 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.162 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.162 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.16 80/79 [PushOp(4:6868af91:::smithi03829856-10:10, version: 70'81, data_included: [524385~820472,1602039~228104], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:10@70'81, size: 3747928, copy_subset: [(0, 3747928)], clone_subset: {}, snapset: 14=[]:{7: [6], b: [b], 10: [d], 14: [13, 12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1830143, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:09.162 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:59.111343+0000", 2024-03-19T11:31:09.162 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 9.9812841500000005, 2024-03-19T11:31:09.163 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019054888999999998, 2024-03-19T11:31:09.163 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:09.163 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:09.163 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:09.163 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.163 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:09.163 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.111343+0000", 2024-03-19T11:31:09.163 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:09.163 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.164 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.164 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:09.164 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.111343+0000", 2024-03-19T11:31:09.164 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:09.164 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.164 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.164 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:09.164 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.111349+0000", 2024-03-19T11:31:09.165 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9619999999999998e-06 2024-03-19T11:31:09.165 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.165 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.165 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:09.165 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.113881+0000", 2024-03-19T11:31:09.165 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025322930000000001 2024-03-19T11:31:09.165 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.165 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.166 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:09.166 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.113917+0000", 2024-03-19T11:31:09.166 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5593999999999998e-05 2024-03-19T11:31:09.166 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.166 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.166 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:09.166 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.113962+0000", 2024-03-19T11:31:09.166 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5362999999999999e-05 2024-03-19T11:31:09.166 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.166 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.167 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:09.167 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.126871+0000", 2024-03-19T11:31:09.167 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012909273000000001 2024-03-19T11:31:09.167 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.167 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.167 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:09.167 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.126892+0000", 2024-03-19T11:31:09.168 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0678000000000001e-05 2024-03-19T11:31:09.168 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.168 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.168 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:09.168 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.130398+0000", 2024-03-19T11:31:09.168 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0035057259999999998 2024-03-19T11:31:09.168 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:09.168 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:09.169 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:09.169 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.169 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.169 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.16 80/79 [PushOp(4:6d9093b1:::smithi03829856-38:10, version: 70'83, data_included: [621859~568773,1729599~479803], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6d9093b1:::smithi03829856-38:10@70'83, size: 3142876, copy_subset: [(0, 3142876)], clone_subset: {}, snapset: 10=[]:{10: [d, b, 6]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2209402, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:09.169 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:59.114057+0000", 2024-03-19T11:31:09.169 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 9.9785701210000006, 2024-03-19T11:31:09.169 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019935583, 2024-03-19T11:31:09.169 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:09.169 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:09.170 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:09.170 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.170 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:09.170 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.114057+0000", 2024-03-19T11:31:09.170 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:09.170 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.170 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.170 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:09.171 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.114057+0000", 2024-03-19T11:31:09.171 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:09.171 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.171 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.171 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:09.171 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.114058+0000", 2024-03-19T11:31:09.171 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4300000000000001e-07 2024-03-19T11:31:09.171 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.171 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.172 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:09.172 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.117243+0000", 2024-03-19T11:31:09.172 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031852550000000001 2024-03-19T11:31:09.172 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.172 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.172 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:09.172 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.117263+0000", 2024-03-19T11:31:09.172 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0469999999999999e-05 2024-03-19T11:31:09.172 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.173 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.173 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:09.173 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.117276+0000", 2024-03-19T11:31:09.173 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2775000000000001e-05 2024-03-19T11:31:09.173 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.173 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.173 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:09.173 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.130681+0000", 2024-03-19T11:31:09.173 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013404879999999999 2024-03-19T11:31:09.174 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.174 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.174 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:09.174 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.130699+0000", 2024-03-19T11:31:09.174 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8343e-05 2024-03-19T11:31:09.174 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.174 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.174 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:09.174 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.133992+0000", 2024-03-19T11:31:09.174 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0032929169999999998 2024-03-19T11:31:09.174 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:09.175 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:09.175 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:09.175 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.175 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.175 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2442 4.16 4:6d9093b1:::smithi03829856-38:head [write 460558~707131] snapc 17=[17,16,15,13,12,b,6] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-03-19T11:31:09.175 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:03.432382+0000", 2024-03-19T11:31:09.175 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.6602445699999997, 2024-03-19T11:31:09.175 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019918379, 2024-03-19T11:31:09.175 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:09.176 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:09.176 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:09.176 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:09.176 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:09.176 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2442 2024-03-19T11:31:09.176 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.176 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:09.177 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.177 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:09.177 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.432382+0000", 2024-03-19T11:31:09.177 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:09.177 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.177 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.178 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:09.178 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.432382+0000", 2024-03-19T11:31:09.178 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:09.178 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.178 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.178 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:09.178 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.432385+0000", 2024-03-19T11:31:09.179 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1e-06 2024-03-19T11:31:09.179 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.179 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.179 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:09.179 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.433115+0000", 2024-03-19T11:31:09.179 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00072939900000000004 2024-03-19T11:31:09.179 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.179 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.179 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:09.180 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.433151+0000", 2024-03-19T11:31:09.180 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6094999999999999e-05 2024-03-19T11:31:09.180 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.180 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.180 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:09.180 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.433167+0000", 2024-03-19T11:31:09.180 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6152999999999999e-05 2024-03-19T11:31:09.180 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.180 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.181 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:09.181 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.433339+0000", 2024-03-19T11:31:09.181 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000171478 2024-03-19T11:31:09.181 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.181 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.181 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:09.181 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.433621+0000", 2024-03-19T11:31:09.181 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00028283900000000001 2024-03-19T11:31:09.181 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.182 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.182 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3", 2024-03-19T11:31:09.182 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.434169+0000", 2024-03-19T11:31:09.182 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00054786699999999995 2024-03-19T11:31:09.182 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.182 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.182 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:09.182 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.443136+0000", 2024-03-19T11:31:09.183 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0089667419999999998 2024-03-19T11:31:09.183 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.183 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.183 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:09.183 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.452230+0000", 2024-03-19T11:31:09.183 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.009093996 2024-03-19T11:31:09.183 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.183 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.183 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:09.183 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.452264+0000", 2024-03-19T11:31:09.184 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4381000000000002e-05 2024-03-19T11:31:09.184 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.184 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.184 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:09.184 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.452301+0000", 2024-03-19T11:31:09.184 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6328999999999997e-05 2024-03-19T11:31:09.184 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:09.184 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:09.184 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:09.185 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.185 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.185 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.4.13:82 3.3 3:f6b6d21a:::smithi03829856-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-03-19T11:31:09.185 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:04.069448+0000", 2024-03-19T11:31:09.185 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.0231792950000003, 2024-03-19T11:31:09.185 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.086427773999999999, 2024-03-19T11:31:09.185 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:09.185 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:09.185 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:09.186 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.186 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:09.186 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069448+0000", 2024-03-19T11:31:09.186 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:09.186 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.186 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.186 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:09.186 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069448+0000", 2024-03-19T11:31:09.187 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:09.187 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.187 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.187 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:09.187 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069449+0000", 2024-03-19T11:31:09.187 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8160000000000001e-06 2024-03-19T11:31:09.187 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.187 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.187 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:09.187 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069458+0000", 2024-03-19T11:31:09.188 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2900000000000002e-06 2024-03-19T11:31:09.188 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.188 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.188 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:09.188 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069491+0000", 2024-03-19T11:31:09.188 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3229000000000003e-05 2024-03-19T11:31:09.188 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.188 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.188 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:09.188 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069501+0000", 2024-03-19T11:31:09.189 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7629999999999995e-06 2024-03-19T11:31:09.189 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.189 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.189 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:09.189 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069573+0000", 2024-03-19T11:31:09.189 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2349999999999997e-05 2024-03-19T11:31:09.189 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.189 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.189 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:09.190 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069581+0000", 2024-03-19T11:31:09.190 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.388e-06 2024-03-19T11:31:09.190 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.190 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.190 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:09.190 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.075451+0000", 2024-03-19T11:31:09.190 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005869956 2024-03-19T11:31:09.191 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.191 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.191 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:09.191 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.077511+0000", 2024-03-19T11:31:09.191 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0020592610000000002 2024-03-19T11:31:09.191 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.191 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.191 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:09.191 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.108781+0000", 2024-03-19T11:31:09.192 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.031270344999999998 2024-03-19T11:31:09.192 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.192 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.192 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:09.192 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.108895+0000", 2024-03-19T11:31:09.192 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000114059 2024-03-19T11:31:09.192 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.192 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.193 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:09.193 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.109065+0000", 2024-03-19T11:31:09.193 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016979899999999999 2024-03-19T11:31:09.193 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.193 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.193 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:09.193 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.111730+0000", 2024-03-19T11:31:09.193 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0026651410000000002 2024-03-19T11:31:09.193 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.194 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.194 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:09.194 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.155616+0000", 2024-03-19T11:31:09.194 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.043885986000000002 2024-03-19T11:31:09.194 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.194 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.194 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:09.195 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.155745+0000", 2024-03-19T11:31:09.195 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012948899999999999 2024-03-19T11:31:09.195 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.195 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.195 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:09.195 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.155875+0000", 2024-03-19T11:31:09.195 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012990199999999999 2024-03-19T11:31:09.195 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:09.195 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:09.195 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:09.196 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.196 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.196 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.4.13:83 3.3 3:f6b6d21a:::smithi03829856-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-03-19T11:31:09.196 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:04.069526+0000", 2024-03-19T11:31:09.196 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.0231005150000003, 2024-03-19T11:31:09.196 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.087140880000000004, 2024-03-19T11:31:09.196 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:09.196 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:09.196 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:09.196 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.197 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:09.197 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069526+0000", 2024-03-19T11:31:09.197 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:09.197 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.197 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.197 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:09.197 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069526+0000", 2024-03-19T11:31:09.197 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:09.197 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.198 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.198 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:09.198 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069527+0000", 2024-03-19T11:31:09.198 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2799999999999996e-07 2024-03-19T11:31:09.198 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.198 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.198 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:09.199 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069531+0000", 2024-03-19T11:31:09.199 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1489999999999996e-06 2024-03-19T11:31:09.199 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.199 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.199 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:09.199 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069539+0000", 2024-03-19T11:31:09.199 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3189999999999995e-06 2024-03-19T11:31:09.199 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.199 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.200 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:09.200 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069545+0000", 2024-03-19T11:31:09.200 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.592e-06 2024-03-19T11:31:09.200 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.200 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.200 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:09.200 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069616+0000", 2024-03-19T11:31:09.200 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1130000000000005e-05 2024-03-19T11:31:09.200 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.201 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.201 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:09.201 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069628+0000", 2024-03-19T11:31:09.201 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1697e-05 2024-03-19T11:31:09.201 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.201 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.201 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:09.202 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.077594+0000", 2024-03-19T11:31:09.202 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0079656989999999997 2024-03-19T11:31:09.202 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.202 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.202 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:09.202 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.077674+0000", 2024-03-19T11:31:09.202 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0338000000000002e-05 2024-03-19T11:31:09.202 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.202 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.203 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:09.203 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.112771+0000", 2024-03-19T11:31:09.203 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.035097325999999998 2024-03-19T11:31:09.203 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.203 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.203 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:09.203 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.112882+0000", 2024-03-19T11:31:09.203 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000110949 2024-03-19T11:31:09.203 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.203 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.204 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:09.204 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.113031+0000", 2024-03-19T11:31:09.204 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000148355 2024-03-19T11:31:09.204 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.204 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.204 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:09.204 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.114076+0000", 2024-03-19T11:31:09.204 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001045009 2024-03-19T11:31:09.204 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.205 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.205 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:09.205 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156482+0000", 2024-03-19T11:31:09.205 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.042406354 2024-03-19T11:31:09.205 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.205 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.205 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:09.205 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156549+0000", 2024-03-19T11:31:09.206 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7229000000000003e-05 2024-03-19T11:31:09.206 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.206 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.206 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:09.206 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156667+0000", 2024-03-19T11:31:09.206 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000118106 2024-03-19T11:31:09.206 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:09.206 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:09.207 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:09.207 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.207 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.207 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.4.13:84 3.3 3:ce6951d3:::smithi03829856-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-03-19T11:31:09.207 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:04.071995+0000", 2024-03-19T11:31:09.207 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.0206317399999998, 2024-03-19T11:31:09.207 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.084172497999999998, 2024-03-19T11:31:09.207 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:09.208 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:09.208 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:09.208 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.208 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:09.208 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.071995+0000", 2024-03-19T11:31:09.208 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:09.208 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.208 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.208 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:09.209 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.071995+0000", 2024-03-19T11:31:09.209 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:09.209 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.209 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.209 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:09.209 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.071997+0000", 2024-03-19T11:31:09.209 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6029999999999999e-06 2024-03-19T11:31:09.209 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.209 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.209 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:09.209 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072004+0000", 2024-03-19T11:31:09.210 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8600000000000004e-06 2024-03-19T11:31:09.210 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.210 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.210 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:09.210 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072033+0000", 2024-03-19T11:31:09.210 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9836999999999999e-05 2024-03-19T11:31:09.210 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.210 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.211 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:09.211 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072042+0000", 2024-03-19T11:31:09.211 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9279999999999999e-06 2024-03-19T11:31:09.211 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.211 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.211 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:09.211 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072123+0000", 2024-03-19T11:31:09.211 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0630000000000006e-05 2024-03-19T11:31:09.211 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.212 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.212 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:09.212 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072132+0000", 2024-03-19T11:31:09.212 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5560000000000008e-06 2024-03-19T11:31:09.212 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.212 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.212 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:09.212 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.077712+0000", 2024-03-19T11:31:09.213 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055804970000000002 2024-03-19T11:31:09.213 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.213 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.213 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:09.213 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.079495+0000", 2024-03-19T11:31:09.213 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017831850000000001 2024-03-19T11:31:09.213 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.213 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.213 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:09.214 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.109825+0000", 2024-03-19T11:31:09.214 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.030329819000000001 2024-03-19T11:31:09.214 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.214 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.214 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:09.214 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.109904+0000", 2024-03-19T11:31:09.214 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9269000000000003e-05 2024-03-19T11:31:09.214 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.214 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.214 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:09.215 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.110085+0000", 2024-03-19T11:31:09.215 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00018018600000000001 2024-03-19T11:31:09.215 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.215 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.215 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:09.215 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.113581+0000", 2024-03-19T11:31:09.215 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034962919999999998 2024-03-19T11:31:09.215 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.216 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.216 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:09.216 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.155925+0000", 2024-03-19T11:31:09.216 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.042344404000000002 2024-03-19T11:31:09.216 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.216 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.216 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:09.216 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156026+0000", 2024-03-19T11:31:09.217 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000100599 2024-03-19T11:31:09.217 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.217 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.217 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:09.217 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156168+0000", 2024-03-19T11:31:09.217 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014183300000000001 2024-03-19T11:31:09.217 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:09.217 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:09.218 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:09.218 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.218 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.218 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.4.13:85 3.3 3:ce6951d3:::smithi03829856-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-03-19T11:31:09.218 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:04.072068+0000", 2024-03-19T11:31:09.218 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.0205591040000002, 2024-03-19T11:31:09.218 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.084368466000000003, 2024-03-19T11:31:09.218 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:09.218 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:09.218 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:09.219 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.219 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:09.219 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072068+0000", 2024-03-19T11:31:09.219 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:09.219 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.219 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.219 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:09.219 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072068+0000", 2024-03-19T11:31:09.219 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:09.220 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.220 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.220 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:09.220 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072068+0000", 2024-03-19T11:31:09.220 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9699999999999996e-07 2024-03-19T11:31:09.220 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.220 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.220 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:09.220 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072074+0000", 2024-03-19T11:31:09.220 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2279999999999998e-06 2024-03-19T11:31:09.220 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.221 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.221 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:09.221 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072140+0000", 2024-03-19T11:31:09.221 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6638999999999997e-05 2024-03-19T11:31:09.221 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.221 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.221 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:09.221 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072147+0000", 2024-03-19T11:31:09.221 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8750000000000002e-06 2024-03-19T11:31:09.222 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.222 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.222 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:09.222 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072201+0000", 2024-03-19T11:31:09.222 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3634000000000002e-05 2024-03-19T11:31:09.222 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.222 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.222 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:09.222 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072211+0000", 2024-03-19T11:31:09.223 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0589e-05 2024-03-19T11:31:09.223 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.223 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.223 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:09.223 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.079687+0000", 2024-03-19T11:31:09.223 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074757219999999997 2024-03-19T11:31:09.223 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.223 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.223 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:09.223 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.079769+0000", 2024-03-19T11:31:09.224 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2251999999999998e-05 2024-03-19T11:31:09.224 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.224 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.224 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:09.224 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.111798+0000", 2024-03-19T11:31:09.224 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.032028889999999997 2024-03-19T11:31:09.224 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.224 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.225 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:09.225 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.111902+0000", 2024-03-19T11:31:09.225 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000103773 2024-03-19T11:31:09.225 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.225 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.225 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:09.225 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.112073+0000", 2024-03-19T11:31:09.225 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017102799999999999 2024-03-19T11:31:09.226 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.226 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.226 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:09.226 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.113591+0000", 2024-03-19T11:31:09.226 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0015179569999999999 2024-03-19T11:31:09.226 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.226 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.226 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:09.226 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156223+0000", 2024-03-19T11:31:09.227 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.042632153999999998 2024-03-19T11:31:09.227 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.227 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.227 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:09.227 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156321+0000", 2024-03-19T11:31:09.227 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8246999999999996e-05 2024-03-19T11:31:09.227 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.227 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.227 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:09.227 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156436+0000", 2024-03-19T11:31:09.227 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000114881 2024-03-19T11:31:09.228 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:09.228 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:09.228 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:09.228 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.228 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.228 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2455 3.1 3:8e10ce27:::f25cf4cdffbae9b54315ae93e2cdc514e3bf58d758c8445671e9463006f381a6:head [write 0~1661440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-03-19T11:31:09.228 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:04.086189+0000", 2024-03-19T11:31:09.228 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.0064379580000002, 2024-03-19T11:31:09.228 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015194925999999999, 2024-03-19T11:31:09.228 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:09.229 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:09.229 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:09.229 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:09.229 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:09.229 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2455 2024-03-19T11:31:09.229 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.229 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:09.229 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.230 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:09.230 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.086189+0000", 2024-03-19T11:31:09.230 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:09.230 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.230 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.230 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:09.230 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.086189+0000", 2024-03-19T11:31:09.230 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:09.230 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.231 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.231 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:09.231 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.086191+0000", 2024-03-19T11:31:09.231 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8619999999999999e-06 2024-03-19T11:31:09.231 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.231 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.231 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:09.231 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.086944+0000", 2024-03-19T11:31:09.231 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00075297299999999999 2024-03-19T11:31:09.231 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.231 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.232 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:09.232 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.086966+0000", 2024-03-19T11:31:09.232 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2404000000000001e-05 2024-03-19T11:31:09.232 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.232 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.232 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:09.232 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.086976+0000", 2024-03-19T11:31:09.232 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8069999999999994e-06 2024-03-19T11:31:09.232 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.232 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.233 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:09.233 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.087038+0000", 2024-03-19T11:31:09.233 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2314e-05 2024-03-19T11:31:09.233 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.233 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.233 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:09.233 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.087158+0000", 2024-03-19T11:31:09.233 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000119385 2024-03-19T11:31:09.233 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.233 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.234 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:09.234 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.087286+0000", 2024-03-19T11:31:09.234 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012851599999999999 2024-03-19T11:31:09.234 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.234 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.234 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:09.234 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.089954+0000", 2024-03-19T11:31:09.234 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0026674239999999998 2024-03-19T11:31:09.234 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.234 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.234 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:09.234 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.101235+0000", 2024-03-19T11:31:09.235 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011281807 2024-03-19T11:31:09.235 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.235 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.235 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:09.235 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.101321+0000", 2024-03-19T11:31:09.235 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5359999999999996e-05 2024-03-19T11:31:09.235 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.235 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.235 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:09.235 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.101384+0000", 2024-03-19T11:31:09.235 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3074e-05 2024-03-19T11:31:09.236 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:09.236 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:09.236 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:09.236 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.236 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.236 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2593 4.6 4:626acd4d:::smithi03829856-2:head [write 662242~642825] snapc 17=[17,16,15,b] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-03-19T11:31:09.236 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:07.785391+0000", 2024-03-19T11:31:09.236 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 1.30723569, 2024-03-19T11:31:09.236 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018221268999999998, 2024-03-19T11:31:09.236 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:09.236 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:09.236 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:09.236 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:09.237 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:09.237 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2593 2024-03-19T11:31:09.237 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.237 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:09.237 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.237 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:09.237 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.785391+0000", 2024-03-19T11:31:09.237 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:09.237 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.237 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.237 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:09.237 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.785391+0000", 2024-03-19T11:31:09.237 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:09.237 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.238 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.238 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:09.238 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.785394+0000", 2024-03-19T11:31:09.238 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.903e-06 2024-03-19T11:31:09.238 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.238 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.238 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:09.238 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.786080+0000", 2024-03-19T11:31:09.238 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000686201 2024-03-19T11:31:09.238 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.238 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.238 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:09.238 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.786116+0000", 2024-03-19T11:31:09.239 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5441999999999997e-05 2024-03-19T11:31:09.239 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.239 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.239 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:09.239 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.786132+0000", 2024-03-19T11:31:09.239 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6353000000000001e-05 2024-03-19T11:31:09.239 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.239 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.239 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:09.239 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.786283+0000", 2024-03-19T11:31:09.239 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015133599999999999 2024-03-19T11:31:09.239 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.239 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.240 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:09.240 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.786457+0000", 2024-03-19T11:31:09.240 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017352200000000001 2024-03-19T11:31:09.240 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.240 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.240 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1", 2024-03-19T11:31:09.240 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.786765+0000", 2024-03-19T11:31:09.240 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030768199999999999 2024-03-19T11:31:09.240 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.240 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.240 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:09.240 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.798155+0000", 2024-03-19T11:31:09.240 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011390045999999999 2024-03-19T11:31:09.240 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.241 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.241 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:09.241 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.803291+0000", 2024-03-19T11:31:09.241 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0051363060000000002 2024-03-19T11:31:09.241 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.241 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.241 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:09.241 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.803466+0000", 2024-03-19T11:31:09.241 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017505900000000001 2024-03-19T11:31:09.241 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:09.241 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:09.241 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:09.241 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.803612+0000", 2024-03-19T11:31:09.242 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014641900000000001 2024-03-19T11:31:09.242 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:09.242 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:09.242 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:09.242 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:09.242 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:09.242 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:09.244 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-19T11:31:09.244 INFO:tasks.rados.rados.0.smithi038.stdout:1254: finishing write tid 1 to smithi03829856-15 2024-03-19T11:31:09.245 INFO:tasks.rados.rados.0.smithi038.stdout:1255: finishing set_chunk to oid 47 2024-03-19T11:31:09.245 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 47 set_chunk returned error code -95 offset: 261632 length: 1121792 2024-03-19T11:31:09.245 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-03-19T11:31:09.245 INFO:tasks.rados.rados.0.smithi038.stdout:1254: finishing write tid 2 to smithi03829856-15 2024-03-19T11:31:09.245 INFO:tasks.rados.rados.0.smithi038.stdout:1254: finishing write tid 3 to smithi03829856-15 2024-03-19T11:31:09.245 INFO:tasks.rados.rados.0.smithi038.stdout:1256: expect (ObjNum 143 snap 15 seq_num 143) 2024-03-19T11:31:09.245 INFO:tasks.rados.rados.0.smithi038.stdout:1254: finishing write tid 4 to smithi03829856-15 2024-03-19T11:31:09.245 INFO:tasks.rados.rados.0.smithi038.stdout:1254: finishing write tid 5 to smithi03829856-15 2024-03-19T11:31:09.245 INFO:tasks.rados.rados.0.smithi038.stdout:1254: oid 15 updating version 0 to 32 2024-03-19T11:31:09.245 INFO:tasks.rados.rados.0.smithi038.stdout:1254: oid 15 updating version 32 to 33 2024-03-19T11:31:09.245 INFO:tasks.rados.rados.0.smithi038.stdout:1254: oid 15 version 33 is already newer than 28 2024-03-19T11:31:09.245 INFO:tasks.rados.rados.0.smithi038.stdout:1254: oid 15 version 33 is already newer than 29 2024-03-19T11:31:09.245 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 15 v 33 (ObjNum 209 snap 22 seq_num 209) dirty exists 2024-03-19T11:31:09.245 INFO:tasks.rados.rados.0.smithi038.stdout:1254: left oid 15 (ObjNum 209 snap 22 seq_num 209) 2024-03-19T11:31:09.246 INFO:tasks.rados.rados.0.smithi038.stdout:1257: finishing set_chunk to oid 5 2024-03-19T11:31:09.246 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 5 set_chunk returned error code -95 offset: 799744 length: 340480 2024-03-19T11:31:09.246 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-03-19T11:31:09.246 INFO:tasks.rados.rados.0.smithi038.stdout:1258: finishing write tid 1 to smithi03829856-7 2024-03-19T11:31:09.246 INFO:tasks.rados.rados.0.smithi038.stdout:1258: finishing write tid 2 to smithi03829856-7 2024-03-19T11:31:09.246 INFO:tasks.rados.rados.0.smithi038.stdout:1258: finishing write tid 3 to smithi03829856-7 2024-03-19T11:31:09.246 INFO:tasks.rados.rados.0.smithi038.stdout:1258: finishing write tid 4 to smithi03829856-7 2024-03-19T11:31:09.246 INFO:tasks.rados.rados.0.smithi038.stdout:1258: finishing write tid 5 to smithi03829856-7 2024-03-19T11:31:09.246 INFO:tasks.rados.rados.0.smithi038.stdout:1258: oid 7 updating version 0 to 37 2024-03-19T11:31:09.246 INFO:tasks.rados.rados.0.smithi038.stdout:1258: oid 7 updating version 37 to 38 2024-03-19T11:31:09.246 INFO:tasks.rados.rados.0.smithi038.stdout:1258: oid 7 version 38 is already newer than 36 2024-03-19T11:31:09.246 INFO:tasks.rados.rados.0.smithi038.stdout:1258: oid 7 version 38 is already newer than 35 2024-03-19T11:31:09.247 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 7 v 38 (ObjNum 210 snap 22 seq_num 210) dirty exists 2024-03-19T11:31:09.247 INFO:tasks.rados.rados.0.smithi038.stdout:1258: left oid 7 (ObjNum 210 snap 22 seq_num 210) 2024-03-19T11:31:09.247 INFO:tasks.rados.rados.0.smithi038.stdout:1259: finishing set_chunk to oid 24 2024-03-19T11:31:09.247 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 24 set_chunk returned error code -95 offset: 368640 length: 1660928 2024-03-19T11:31:09.247 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-03-19T11:31:09.247 INFO:tasks.rados.rados.0.smithi038.stdout:1260: got (22) Invalid argument 2024-03-19T11:31:09.247 INFO:tasks.rados.rados.0.smithi038.stdout:1261: expect (ObjNum 168 snap 17 seq_num 168) 2024-03-19T11:31:09.247 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:09.255+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:09.345 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:09.355+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:09.363 INFO:tasks.rados.rados.0.smithi038.stdout:1246: done (15 left) 2024-03-19T11:31:09.363 INFO:tasks.rados.rados.0.smithi038.stdout:1247: done (14 left) 2024-03-19T11:31:09.363 INFO:tasks.rados.rados.0.smithi038.stdout:1248: done (13 left) 2024-03-19T11:31:09.363 INFO:tasks.rados.rados.0.smithi038.stdout:1249: done (12 left) 2024-03-19T11:31:09.363 INFO:tasks.rados.rados.0.smithi038.stdout:1250: done (11 left) 2024-03-19T11:31:09.363 INFO:tasks.rados.rados.0.smithi038.stdout:1251: done (10 left) 2024-03-19T11:31:09.363 INFO:tasks.rados.rados.0.smithi038.stdout:1253: done (9 left) 2024-03-19T11:31:09.363 INFO:tasks.rados.rados.0.smithi038.stdout:1254: done (8 left) 2024-03-19T11:31:09.363 INFO:tasks.rados.rados.0.smithi038.stdout:1255: done (7 left) 2024-03-19T11:31:09.364 INFO:tasks.rados.rados.0.smithi038.stdout:1256: done (6 left) 2024-03-19T11:31:09.364 INFO:tasks.rados.rados.0.smithi038.stdout:1257: done (5 left) 2024-03-19T11:31:09.364 INFO:tasks.rados.rados.0.smithi038.stdout:1258: done (4 left) 2024-03-19T11:31:09.364 INFO:tasks.rados.rados.0.smithi038.stdout:1259: done (3 left) 2024-03-19T11:31:09.364 INFO:tasks.rados.rados.0.smithi038.stdout:1260: done (2 left) 2024-03-19T11:31:09.364 INFO:tasks.rados.rados.0.smithi038.stdout:1261: done (1 left) 2024-03-19T11:31:09.364 INFO:tasks.rados.rados.0.smithi038.stdout:1262: done (0 left) 2024-03-19T11:31:09.364 INFO:tasks.rados.rados.0.smithi038.stdout:1263: tier_evict oid 27 2024-03-19T11:31:09.364 INFO:tasks.rados.rados.0.smithi038.stdout:1263: tier_evict oid 27 snap -1 2024-03-19T11:31:09.364 INFO:tasks.rados.rados.0.smithi038.stdout:1264: write (excl) oid 45 current snap is 22 2024-03-19T11:31:09.364 INFO:tasks.rados.rados.0.smithi038.stdout:1264: seq_num 212 ranges {763084=694319,1889359=541737,2955792=541902} 2024-03-19T11:31:09.375 INFO:tasks.rados.rados.0.smithi038.stdout:1264: writing smithi03829856-45 from 763084 to 1457403 tid 1 2024-03-19T11:31:09.385 INFO:tasks.rados.rados.0.smithi038.stdout:1264: writing smithi03829856-45 from 1889359 to 2431096 tid 2 2024-03-19T11:31:09.397 INFO:tasks.rados.rados.0.smithi038.stdout:1264: writing smithi03829856-45 from 2955792 to 3497694 tid 3 2024-03-19T11:31:09.397 INFO:tasks.rados.rados.0.smithi038.stdout:1265: read oid 47 snap -1 2024-03-19T11:31:09.398 INFO:tasks.rados.rados.0.smithi038.stdout:1265: expect (ObjNum 46 snap 0 seq_num 46) 2024-03-19T11:31:09.398 INFO:tasks.rados.rados.0.smithi038.stdout:1266: write (excl) oid 17 current snap is 22 2024-03-19T11:31:09.398 INFO:tasks.rados.rados.0.smithi038.stdout:1263: got (22) Invalid argument 2024-03-19T11:31:09.398 INFO:tasks.rados.rados.0.smithi038.stdout:1264: finishing write tid 1 to smithi03829856-45 2024-03-19T11:31:09.398 INFO:tasks.rados.rados.0.smithi038.stdout:1264: finishing write tid 2 to smithi03829856-45 2024-03-19T11:31:09.398 INFO:tasks.rados.rados.0.smithi038.stdout:1266: seq_num 213 ranges {779891=485921,1947841=685395,3383215=196676} 2024-03-19T11:31:09.407 INFO:tasks.rados.rados.0.smithi038.stdout:1266: writing smithi03829856-17 from 779891 to 1265812 tid 1 2024-03-19T11:31:09.416 INFO:tasks.rados.rados.0.smithi038.stdout:1266: writing smithi03829856-17 from 1947841 to 2633236 tid 2 2024-03-19T11:31:09.422 INFO:tasks.rados.rados.0.smithi038.stdout:1266: writing smithi03829856-17 from 3383215 to 3579891 tid 3 2024-03-19T11:31:09.422 INFO:tasks.rados.rados.0.smithi038.stdout:1263: done (3 left) 2024-03-19T11:31:09.422 INFO:tasks.rados.rados.0.smithi038.stdout:1267: set_chunk oid 19 target oid 2024-03-19T11:31:09.424 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:09.424 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:31:09.424 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:31:09.424 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:09.431 INFO:tasks.rados.rados.0.smithi038.stdout:1267: set_chunk oid 19 offset: 2086400 length: 566784 target oid f8d4ae469506228d7b3938ec37beaa36c7110062a3010a9c78e07b89dddf6c6f offset: 0 snap -1 2024-03-19T11:31:09.431 INFO:tasks.rados.rados.0.smithi038.stdout:1268: read oid 38 snap -1 2024-03-19T11:31:09.431 INFO:tasks.rados.rados.0.smithi038.stdout:1268: expect (ObjNum 194 snap 22 seq_num 194) 2024-03-19T11:31:09.431 INFO:tasks.rados.rados.0.smithi038.stdout:1269: read oid 28 snap -1 2024-03-19T11:31:09.431 INFO:tasks.rados.rados.0.smithi038.stdout:1269: expect (ObjNum 87 snap 8 seq_num 87) 2024-03-19T11:31:09.431 INFO:tasks.rados.rados.0.smithi038.stdout:1270: read oid 35 snap 9 2024-03-19T11:31:09.431 INFO:tasks.rados.rados.0.smithi038.stdout:1270: expect (ObjNum 34 snap 0 seq_num 34) 2024-03-19T11:31:09.431 INFO:tasks.rados.rados.0.smithi038.stdout:1271: read oid 13 snap -1 2024-03-19T11:31:09.431 INFO:tasks.rados.rados.0.smithi038.stdout:1271: expect (ObjNum 12 snap 0 seq_num 12) 2024-03-19T11:31:09.432 INFO:tasks.rados.rados.0.smithi038.stdout:1272: tier_evict oid 11 2024-03-19T11:31:09.432 INFO:tasks.rados.rados.0.smithi038.stdout:1272: tier_evict oid 11 snap -1 2024-03-19T11:31:09.432 INFO:tasks.rados.rados.0.smithi038.stdout:1273: read oid 42 snap -1 2024-03-19T11:31:09.432 INFO:tasks.rados.rados.0.smithi038.stdout:1273: expect deleted 2024-03-19T11:31:09.432 INFO:tasks.rados.rados.0.smithi038.stdout:1274: write (excl) oid 33 current snap is 22 2024-03-19T11:31:09.432 INFO:tasks.rados.rados.0.smithi038.stdout:1274: seq_num 214 ranges {796698=561170,2006323=545406,3126991=469707} 2024-03-19T11:31:09.440 INFO:tasks.rados.rados.0.smithi038.stdout:1274: writing smithi03829856-33 from 796698 to 1357868 tid 1 2024-03-19T11:31:09.446 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:09.455+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:09.448 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-19T11:31:09.448 INFO:tasks.rados.rados.0.smithi038.stdout:1274: writing smithi03829856-33 from 2006323 to 2551729 tid 2 2024-03-19T11:31:09.455 INFO:tasks.rados.rados.0.smithi038.stdout:1274: writing smithi03829856-33 from 3126991 to 3596698 tid 3 2024-03-19T11:31:09.455 INFO:tasks.rados.rados.0.smithi038.stdout:1275: read oid 25 snap 9 2024-03-19T11:31:09.455 INFO:tasks.rados.rados.0.smithi038.stdout:1275: expect (ObjNum 79 snap 6 seq_num 79) 2024-03-19T11:31:09.455 INFO:tasks.rados.rados.0.smithi038.stdout:1276: read oid 40 snap 21 2024-03-19T11:31:09.455 INFO:tasks.rados.rados.0.smithi038.stdout:1276: expect (ObjNum 141 snap 15 seq_num 141) 2024-03-19T11:31:09.455 INFO:tasks.rados.rados.0.smithi038.stdout:1277: read oid 49 snap -1 2024-03-19T11:31:09.455 INFO:tasks.rados.rados.0.smithi038.stdout:1277: expect (ObjNum 201 snap 22 seq_num 201) 2024-03-19T11:31:09.455 INFO:tasks.rados.rados.0.smithi038.stdout:1278: set_chunk oid 23 target oid 2024-03-19T11:31:09.479 INFO:tasks.rados.rados.0.smithi038.stdout:1278: set_chunk oid 23 offset: 1680896 length: 762368 target oid db645df9f42533fab705068351d684a4db9924d95c777862d010b86c040b2f57 offset: 0 snap -1 2024-03-19T11:31:09.479 INFO:tasks.rados.rados.0.smithi038.stdout:1279: snap_create 2024-03-19T11:31:09.479 INFO:tasks.rados.rados.0.smithi038.stdout:1264: finishing write tid 3 to smithi03829856-45 2024-03-19T11:31:09.479 INFO:tasks.rados.rados.0.smithi038.stdout:1264: finishing write tid 4 to smithi03829856-45 2024-03-19T11:31:09.480 INFO:tasks.rados.rados.0.smithi038.stdout:1264: finishing write tid 5 to smithi03829856-45 2024-03-19T11:31:09.480 INFO:tasks.rados.rados.0.smithi038.stdout:1264: oid 45 updating version 0 to 70 2024-03-19T11:31:09.480 INFO:tasks.rados.rados.0.smithi038.stdout:1264: oid 45 version 70 is already newer than 68 2024-03-19T11:31:09.480 INFO:tasks.rados.rados.0.smithi038.stdout:1264: oid 45 version 70 is already newer than 69 2024-03-19T11:31:09.480 INFO:tasks.rados.rados.0.smithi038.stdout:1264: oid 45 version 70 is already newer than 67 2024-03-19T11:31:09.480 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 45 v 70 (ObjNum 212 snap 22 seq_num 212) dirty exists 2024-03-19T11:31:09.480 INFO:tasks.rados.rados.0.smithi038.stdout:1264: left oid 45 (ObjNum 212 snap 22 seq_num 212) 2024-03-19T11:31:09.480 INFO:tasks.rados.rados.0.smithi038.stdout:1265: expect (ObjNum 46 snap 0 seq_num 46) 2024-03-19T11:31:09.546 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:09.553+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:09.556 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-19T11:31:09.565 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:09.565 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:31:09.565 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:31:09.565 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:31:09.565 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:09.575 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T11:31:09.579 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-19T11:31:09.579 INFO:tasks.rados.rados.0.smithi038.stdout:1266: finishing write tid 1 to smithi03829856-17 2024-03-19T11:31:09.579 INFO:tasks.rados.rados.0.smithi038.stdout:1266: finishing write tid 2 to smithi03829856-17 2024-03-19T11:31:09.579 INFO:tasks.rados.rados.0.smithi038.stdout:1266: finishing write tid 3 to smithi03829856-17 2024-03-19T11:31:09.579 INFO:tasks.rados.rados.0.smithi038.stdout:1269: expect (ObjNum 87 snap 8 seq_num 87) 2024-03-19T11:31:09.648 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:09.659+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:09.691 INFO:tasks.rados.rados.0.smithi038.stdout:1272: got (22) Invalid argument 2024-03-19T11:31:09.692 INFO:tasks.rados.rados.0.smithi038.stdout:1267: finishing set_chunk to oid 19 2024-03-19T11:31:09.692 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 19 v 71 (ObjNum 203 snap 22 seq_num 203) dirty exists 2024-03-19T11:31:09.692 INFO:tasks.rados.rados.0.smithi038.stdout:1268: expect (ObjNum 194 snap 22 seq_num 194) 2024-03-19T11:31:09.701 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:31:09.701 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:31:09.728 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:31:09.729 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:31:09.745 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:31:09.745 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:31:09.749 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:09.759+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:09.762 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:31:09.763 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:31:09.767 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:09.767 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:09.783 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:31:09.784 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:31:09.785 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-19T11:31:09.805 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:31:09.805 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:31:09.826 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:31:09.826 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:31:09.847 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:31:09.848 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:31:09.849 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:09.859+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:09.861 INFO:tasks.rados.rados.0.smithi038.stdout:1271: expect (ObjNum 12 snap 0 seq_num 12) 2024-03-19T11:31:09.897 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:09.898 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:09.909 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:31:09.910 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-03-19T11:31:09.911 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-19T11:31:09.925 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":126,"stamp":"2024-03-19T11:31:09.051239+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":427425662,"num_objects":281,"num_object_clones":85,"num_object_copies":562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":278,"num_whiteouts":7,"num_read":1222,"num_read_kb":889043,"num_write":800,"num_write_kb":256306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":259,"num_bytes_recovered":273446428,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":91,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3262,"ondisk_log_size":3262,"up":112,"acting":112,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":123,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":832056,"kb_used_data":581204,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754142664,"statfs":{"total":773094113280,"available":772242087936,"internally_reserved":0,"allocated":595152896,"data_stored":668969470,"data_compressed":16108946,"data_compressed_allocated":16060416,"data_compressed_original":36302848,"omap_allocated":12751,"internal_metadata":251055665},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-12140396,"num_objects":-6,"num_object_clones":-6,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2,"num_objects_unfound":0,"num_objects_dirty":-6,"num_whiteouts":1,"num_read":136,"num_read_kb":152105,"num_write":48,"num_write_kb":14768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":33266043,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.996522"},"pg_stats":[{"pgid":"4.2a","version":"87'100","reported_seq":441,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.119724+0000","last_change":"2024-03-19T11:31:08.120519+0000","last_active":"2024-03-19T11:31:08.119724+0000","last_peered":"2024-03-19T11:31:08.119724+0000","last_clean":"2024-03-19T11:31:08.119724+0000","last_became_active":"2024-03-19T11:30:45.260578+0000","last_became_peered":"2024-03-19T11:30:45.260578+0000","last_unstale":"2024-03-19T11:31:08.119724+0000","last_undegraded":"2024-03-19T11:31:08.119724+0000","last_fullsized":"2024-03-19T11:31:08.119724+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":6,"last_scrub":"74'89","last_scrub_stamp":"2024-03-19T11:30:53.312593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:53.312593+0000","objects_scrubbed":6,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:03.108309+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.034299200000000002,"stat_sum":{"num_bytes":9678155,"num_objects":4,"num_object_clones":2,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":9,"num_read_kb":12963,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.29","version":"85'65","reported_seq":448,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.074347+0000","last_change":"2024-03-19T11:31:08.075407+0000","last_active":"2024-03-19T11:31:08.074347+0000","last_peered":"2024-03-19T11:31:08.074347+0000","last_clean":"2024-03-19T11:31:08.074347+0000","last_became_active":"2024-03-19T11:31:00.070287+0000","last_became_peered":"2024-03-19T11:31:00.070287+0000","last_unstale":"2024-03-19T11:31:08.074347+0000","last_undegraded":"2024-03-19T11:31:08.074347+0000","last_fullsized":"2024-03-19T11:31:08.074347+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":6,"last_scrub":"71'60","last_scrub_stamp":"2024-03-19T11:30:52.748479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:52.748479+0000","objects_scrubbed":3,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:14.667721+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.015091184000000001,"stat_sum":{"num_bytes":4387250,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":19,"num_read_kb":23171,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":5842489,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.28","version":"85'98","reported_seq":383,"reported_epoch":85,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:04.086552+0000","last_change":"2024-03-19T11:31:04.078290+0000","last_active":"2024-03-19T11:31:04.086552+0000","last_peered":"2024-03-19T11:31:04.086552+0000","last_clean":"2024-03-19T11:31:04.086552+0000","last_became_active":"2024-03-19T11:30:45.257600+0000","last_became_peered":"2024-03-19T11:30:45.257600+0000","last_unstale":"2024-03-19T11:31:04.086552+0000","last_undegraded":"2024-03-19T11:31:04.086552+0000","last_fullsized":"2024-03-19T11:31:04.086552+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":6,"last_scrub":"71'85","last_scrub_stamp":"2024-03-19T11:30:52.343454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:52.343454+0000","objects_scrubbed":7,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:16.280349+0000","scrub_duration":7,"objects_trimmed":1,"snaptrim_duration":0.0090263340000000004,"stat_sum":{"num_bytes":11017340,"num_objects":7,"num_object_clones":5,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":6,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.27","version":"85'54","reported_seq":53,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.074043+0000","last_change":"2024-03-19T11:31:04.074317+0000","last_active":"2024-03-19T11:31:04.074043+0000","last_peered":"2024-03-19T11:31:04.074043+0000","last_clean":"2024-03-19T11:31:04.074043+0000","last_became_active":"2024-03-19T11:31:00.070337+0000","last_became_peered":"2024-03-19T11:31:00.070337+0000","last_unstale":"2024-03-19T11:31:04.074043+0000","last_undegraded":"2024-03-19T11:31:04.074043+0000","last_fullsized":"2024-03-19T11:31:04.074043+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":6,"last_scrub":"71'48","last_scrub_stamp":"2024-03-19T11:30:55.467716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:55.467716+0000","objects_scrubbed":4,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:00.305683+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.018507597000000001,"stat_sum":{"num_bytes":6814202,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":11198,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":7070511,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"},{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.26","version":"61'63","reported_seq":49,"reported_epoch":88,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:07.098364+0000","last_change":"2024-03-19T11:31:07.098333+0000","last_active":"2024-03-19T11:31:07.098364+0000","last_peered":"2024-03-19T11:31:07.098364+0000","last_clean":"2024-03-19T11:31:07.098364+0000","last_became_active":"2024-03-19T11:30:59.074337+0000","last_became_peered":"2024-03-19T11:30:59.074337+0000","last_unstale":"2024-03-19T11:31:07.098364+0000","last_undegraded":"2024-03-19T11:31:07.098364+0000","last_fullsized":"2024-03-19T11:31:07.098364+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":6,"last_scrub":"61'63","last_scrub_stamp":"2024-03-19T11:30:52.406723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:52.406723+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:22.060620+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":1.020104218,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"6","length":"1"},{"start":"c","length":"2"},{"start":"11","length":"2"},{"start":"14","length":"1"}]},{"pgid":"4.25","version":"56'66","reported_seq":51,"reported_epoch":85,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:04.077517+0000","last_change":"2024-03-19T11:31:04.077570+0000","last_active":"2024-03-19T11:31:04.077517+0000","last_peered":"2024-03-19T11:31:04.077517+0000","last_clean":"2024-03-19T11:31:04.077517+0000","last_became_active":"2024-03-19T11:30:53.084473+0000","last_became_peered":"2024-03-19T11:30:53.084473+0000","last_unstale":"2024-03-19T11:31:04.077517+0000","last_undegraded":"2024-03-19T11:31:04.077517+0000","last_fullsized":"2024-03-19T11:31:04.077517+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":6,"last_scrub":"47'64","last_scrub_stamp":"2024-03-19T11:30:28.487136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:28.487136+0000","objects_scrubbed":3,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0087584710000000003,"stat_sum":{"num_bytes":2403368,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":10662,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6","length":"1"},{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.24","version":"85'117","reported_seq":477,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.064686+0000","last_change":"2024-03-19T11:31:04.064961+0000","last_active":"2024-03-19T11:31:04.064686+0000","last_peered":"2024-03-19T11:31:04.064686+0000","last_clean":"2024-03-19T11:31:04.064686+0000","last_became_active":"2024-03-19T11:31:03.063705+0000","last_became_peered":"2024-03-19T11:31:03.063705+0000","last_unstale":"2024-03-19T11:31:04.064686+0000","last_undegraded":"2024-03-19T11:31:04.064686+0000","last_fullsized":"2024-03-19T11:31:04.064686+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":6,"last_scrub":"84'113","last_scrub_stamp":"2024-03-19T11:31:03.304881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:31:03.304881+0000","objects_scrubbed":5,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:26.481361+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.0099064849999999996,"stat_sum":{"num_bytes":10102023,"num_objects":4,"num_object_clones":2,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"},{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.23","version":"85'29","reported_seq":59,"reported_epoch":85,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:04.098447+0000","last_change":"2024-03-19T11:31:04.098447+0000","last_active":"2024-03-19T11:31:04.098447+0000","last_peered":"2024-03-19T11:31:04.098447+0000","last_clean":"2024-03-19T11:31:04.098447+0000","last_became_active":"2024-03-19T11:30:53.076092+0000","last_became_peered":"2024-03-19T11:30:53.076092+0000","last_unstale":"2024-03-19T11:31:04.098447+0000","last_undegraded":"2024-03-19T11:31:04.098447+0000","last_fullsized":"2024-03-19T11:31:04.098447+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":6,"last_scrub":"62'23","last_scrub_stamp":"2024-03-19T11:30:43.163509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:43.163509+0000","objects_scrubbed":2,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:46.525658+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.023454183999999999,"stat_sum":{"num_bytes":3728802,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2833035,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.22","version":"28'6","reported_seq":86,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.075051+0000","last_change":"2024-03-19T11:31:08.076088+0000","last_active":"2024-03-19T11:31:08.075051+0000","last_peered":"2024-03-19T11:31:08.075051+0000","last_clean":"2024-03-19T11:31:08.075051+0000","last_became_active":"2024-03-19T11:30:53.068010+0000","last_became_peered":"2024-03-19T11:30:53.068010+0000","last_unstale":"2024-03-19T11:31:08.075051+0000","last_undegraded":"2024-03-19T11:31:08.075051+0000","last_fullsized":"2024-03-19T11:31:08.075051+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":6,"last_scrub":"28'6","last_scrub_stamp":"2024-03-19T11:30:55.724671+0000","last_deep_scrub":"28'6","last_deep_scrub_stamp":"2024-03-19T11:30:55.724671+0000","last_clean_scrub_stamp":"2024-03-19T11:30:55.724671+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:13.244591+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.015426808,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.21","version":"83'125","reported_seq":500,"reported_epoch":85,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:04.100621+0000","last_change":"2024-03-19T11:31:04.100668+0000","last_active":"2024-03-19T11:31:04.100621+0000","last_peered":"2024-03-19T11:31:04.100621+0000","last_clean":"2024-03-19T11:31:04.100621+0000","last_became_active":"2024-03-19T11:31:00.069759+0000","last_became_peered":"2024-03-19T11:31:00.069759+0000","last_unstale":"2024-03-19T11:31:04.100621+0000","last_undegraded":"2024-03-19T11:31:04.100621+0000","last_fullsized":"2024-03-19T11:31:04.100621+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":6,"last_scrub":"62'96","last_scrub_stamp":"2024-03-19T11:30:43.400885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:43.400885+0000","objects_scrubbed":9,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:55.649626+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.050771269000000001,"stat_sum":{"num_bytes":15281651,"num_objects":9,"num_object_clones":7,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":48,"num_read_kb":51399,"num_write":18,"num_write_kb":4886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":22983003,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"2"},{"start":"14","length":"1"}]},{"pgid":"4.20","version":"56'46","reported_seq":78,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.116754+0000","last_change":"2024-03-19T11:31:08.117528+0000","last_active":"2024-03-19T11:31:08.116754+0000","last_peered":"2024-03-19T11:31:08.116754+0000","last_clean":"2024-03-19T11:31:08.116754+0000","last_became_active":"2024-03-19T11:30:53.067251+0000","last_became_peered":"2024-03-19T11:30:53.067251+0000","last_unstale":"2024-03-19T11:31:08.116754+0000","last_undegraded":"2024-03-19T11:31:08.116754+0000","last_fullsized":"2024-03-19T11:31:08.116754+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":6,"last_scrub":"56'46","last_scrub_stamp":"2024-03-19T11:30:41.767100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:41.767100+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:04.278716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.032365520000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.1f","version":"0'0","reported_seq":86,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.115029+0000","last_change":"2024-03-19T11:31:08.116595+0000","last_active":"2024-03-19T11:31:08.115029+0000","last_peered":"2024-03-19T11:31:08.115029+0000","last_clean":"2024-03-19T11:31:08.115029+0000","last_became_active":"2024-03-19T11:30:53.087211+0000","last_became_peered":"2024-03-19T11:30:53.087211+0000","last_unstale":"2024-03-19T11:31:08.115029+0000","last_undegraded":"2024-03-19T11:31:08.115029+0000","last_fullsized":"2024-03-19T11:31:08.115029+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:30:42.377963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.377963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:05.818292+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.032679030999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.1e","version":"87'71","reported_seq":82,"reported_epoch":88,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:07.098810+0000","last_change":"2024-03-19T11:31:06.103482+0000","last_active":"2024-03-19T11:31:07.098810+0000","last_peered":"2024-03-19T11:31:07.098810+0000","last_clean":"2024-03-19T11:31:07.098810+0000","last_became_active":"2024-03-19T11:30:53.084452+0000","last_became_peered":"2024-03-19T11:30:53.084452+0000","last_unstale":"2024-03-19T11:31:07.098810+0000","last_undegraded":"2024-03-19T11:31:07.098810+0000","last_fullsized":"2024-03-19T11:31:07.098810+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"64'62","last_scrub_stamp":"2024-03-19T11:30:42.778159+0000","last_deep_scrub":"64'62","last_deep_scrub_stamp":"2024-03-19T11:30:42.778159+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.778159+0000","objects_scrubbed":4,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:03.485738+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0.042323323000000003,"stat_sum":{"num_bytes":3794928,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":7565,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3779426,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.1d","version":"82'66","reported_seq":61,"reported_epoch":85,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:04.050025+0000","last_change":"2024-03-19T11:31:02.078403+0000","last_active":"2024-03-19T11:31:04.050025+0000","last_peered":"2024-03-19T11:31:04.050025+0000","last_clean":"2024-03-19T11:31:04.050025+0000","last_became_active":"2024-03-19T11:30:53.067748+0000","last_became_peered":"2024-03-19T11:30:53.067748+0000","last_unstale":"2024-03-19T11:31:04.050025+0000","last_undegraded":"2024-03-19T11:31:04.050025+0000","last_fullsized":"2024-03-19T11:31:04.050025+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"61'59","last_scrub_stamp":"2024-03-19T11:30:42.601863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.601863+0000","objects_scrubbed":2,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:52.206103+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.022205709000000001,"stat_sum":{"num_bytes":6649813,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":61,"num_write":4,"num_write_kb":1341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2817390,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.1c","version":"85'35","reported_seq":69,"reported_epoch":85,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:04.055902+0000","last_change":"2024-03-19T11:31:04.055902+0000","last_active":"2024-03-19T11:31:04.055902+0000","last_peered":"2024-03-19T11:31:04.055902+0000","last_clean":"2024-03-19T11:31:04.055902+0000","last_became_active":"2024-03-19T11:30:53.082655+0000","last_became_peered":"2024-03-19T11:30:53.082655+0000","last_unstale":"2024-03-19T11:31:04.055902+0000","last_undegraded":"2024-03-19T11:31:04.055902+0000","last_fullsized":"2024-03-19T11:31:04.055902+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"62'31","last_scrub_stamp":"2024-03-19T11:30:42.771396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.771396+0000","objects_scrubbed":3,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:55.484122+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.00073920500000000005,"stat_sum":{"num_bytes":3210104,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":47464,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":5034309,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.1b","version":"89'100","reported_seq":92,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.105337+0000","last_change":"2024-03-19T11:31:08.108313+0000","last_active":"2024-03-19T11:31:08.105337+0000","last_peered":"2024-03-19T11:31:08.105337+0000","last_clean":"2024-03-19T11:31:08.105337+0000","last_became_active":"2024-03-19T11:31:03.056769+0000","last_became_peered":"2024-03-19T11:31:03.056769+0000","last_unstale":"2024-03-19T11:31:08.105337+0000","last_undegraded":"2024-03-19T11:31:08.105337+0000","last_fullsized":"2024-03-19T11:31:08.105337+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":5,"last_scrub":"62'62","last_scrub_stamp":"2024-03-19T11:30:42.181424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.181424+0000","objects_scrubbed":5,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:53.941736+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.040344087000000001,"stat_sum":{"num_bytes":14818653,"num_objects":7,"num_object_clones":4,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":58,"num_read_kb":68803,"num_write":11,"num_write_kb":3105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":17966556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":6,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.1a","version":"89'111","reported_seq":83,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.082577+0000","last_change":"2024-03-19T11:31:08.083422+0000","last_active":"2024-03-19T11:31:08.082577+0000","last_peered":"2024-03-19T11:31:08.082577+0000","last_clean":"2024-03-19T11:31:08.082577+0000","last_became_active":"2024-03-19T11:31:03.056520+0000","last_became_peered":"2024-03-19T11:31:03.056520+0000","last_unstale":"2024-03-19T11:31:08.082577+0000","last_undegraded":"2024-03-19T11:31:08.082577+0000","last_fullsized":"2024-03-19T11:31:08.082577+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":5,"last_scrub":"61'76","last_scrub_stamp":"2024-03-19T11:30:41.433501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:41.433501+0000","objects_scrubbed":5,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:46.519318+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.021991492000000001,"stat_sum":{"num_bytes":11953834,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":22,"num_read_kb":17414,"num_write":18,"num_write_kb":5314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":16291222,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.19","version":"83'111","reported_seq":471,"reported_epoch":85,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:04.098365+0000","last_change":"2024-03-19T11:31:04.098409+0000","last_active":"2024-03-19T11:31:04.098365+0000","last_peered":"2024-03-19T11:31:04.098365+0000","last_clean":"2024-03-19T11:31:04.098365+0000","last_became_active":"2024-03-19T11:30:53.076188+0000","last_became_peered":"2024-03-19T11:30:53.076188+0000","last_unstale":"2024-03-19T11:31:04.098365+0000","last_undegraded":"2024-03-19T11:31:04.098365+0000","last_fullsized":"2024-03-19T11:31:04.098365+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"41'37","last_scrub_stamp":"2024-03-19T11:30:27.705159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:27.705159+0000","objects_scrubbed":4,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:29.203476+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.047949758000000002,"stat_sum":{"num_bytes":18546999,"num_objects":12,"num_object_clones":8,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":88,"num_read_kb":101447,"num_write":38,"num_write_kb":6633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":21870393,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":11,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.18","version":"88'138","reported_seq":440,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.074905+0000","last_change":"2024-03-19T11:31:08.074905+0000","last_active":"2024-03-19T11:31:08.074905+0000","last_peered":"2024-03-19T11:31:08.074905+0000","last_clean":"2024-03-19T11:31:08.074905+0000","last_became_active":"2024-03-19T11:31:06.066324+0000","last_became_peered":"2024-03-19T11:31:06.066324+0000","last_unstale":"2024-03-19T11:31:08.074905+0000","last_undegraded":"2024-03-19T11:31:08.074905+0000","last_fullsized":"2024-03-19T11:31:08.074905+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":5,"last_scrub":"48'77","last_scrub_stamp":"2024-03-19T11:30:29.399762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:29.399762+0000","objects_scrubbed":6,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:55.801600+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.014795552,"stat_sum":{"num_bytes":18257745,"num_objects":7,"num_object_clones":5,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":49,"num_read_kb":49877,"num_write":22,"num_write_kb":6328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":19971477,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.17","version":"41'24","reported_seq":55,"reported_epoch":85,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:04.050331+0000","last_change":"2024-03-19T11:31:02.079786+0000","last_active":"2024-03-19T11:31:04.050331+0000","last_peered":"2024-03-19T11:31:04.050331+0000","last_clean":"2024-03-19T11:31:04.050331+0000","last_became_active":"2024-03-19T11:30:53.074270+0000","last_became_peered":"2024-03-19T11:30:53.074270+0000","last_unstale":"2024-03-19T11:31:04.050331+0000","last_undegraded":"2024-03-19T11:31:04.050331+0000","last_fullsized":"2024-03-19T11:31:04.050331+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"41'24","last_scrub_stamp":"2024-03-19T11:30:53.993428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:53.993428+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:54.020328+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.022407469999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.16","version":"87'116","reported_seq":91,"reported_epoch":88,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:07.097994+0000","last_change":"2024-03-19T11:31:07.083681+0000","last_active":"2024-03-19T11:31:07.097994+0000","last_peered":"2024-03-19T11:31:07.097994+0000","last_clean":"2024-03-19T11:31:07.097994+0000","last_became_active":"2024-03-19T11:30:59.074063+0000","last_became_peered":"2024-03-19T11:30:59.074063+0000","last_unstale":"2024-03-19T11:31:07.097994+0000","last_undegraded":"2024-03-19T11:31:07.097994+0000","last_fullsized":"2024-03-19T11:31:07.097994+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":5,"last_scrub":"49'56","last_scrub_stamp":"2024-03-19T11:30:29.212766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:29.212766+0000","objects_scrubbed":5,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:38.049725+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":1.0187149390000001,"stat_sum":{"num_bytes":14577733,"num_objects":8,"num_object_clones":5,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":105,"num_read_kb":105119,"num_write":27,"num_write_kb":7671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":24165549,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":8,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"6","length":"1"},{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.15","version":"85'148","reported_seq":547,"reported_epoch":85,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:04.108361+0000","last_change":"2024-03-19T11:31:02.074320+0000","last_active":"2024-03-19T11:31:04.108361+0000","last_peered":"2024-03-19T11:31:04.108361+0000","last_clean":"2024-03-19T11:31:04.108361+0000","last_became_active":"2024-03-19T11:30:56.063100+0000","last_became_peered":"2024-03-19T11:30:56.063100+0000","last_unstale":"2024-03-19T11:31:04.108361+0000","last_undegraded":"2024-03-19T11:31:04.108361+0000","last_fullsized":"2024-03-19T11:31:04.108361+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":5,"last_scrub":"49'74","last_scrub_stamp":"2024-03-19T11:30:29.457542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:29.457542+0000","objects_scrubbed":7,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:56.479079+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.019267638,"stat_sum":{"num_bytes":22140490,"num_objects":13,"num_object_clones":8,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":128,"num_read_kb":146180,"num_write":32,"num_write_kb":8100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":18744415,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":8,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.14","version":"40'56","reported_seq":220,"reported_epoch":85,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:04.051467+0000","last_change":"2024-03-19T11:31:02.093175+0000","last_active":"2024-03-19T11:31:04.051467+0000","last_peered":"2024-03-19T11:31:04.051467+0000","last_clean":"2024-03-19T11:31:04.051467+0000","last_became_active":"2024-03-19T11:31:00.065648+0000","last_became_peered":"2024-03-19T11:31:00.065648+0000","last_unstale":"2024-03-19T11:31:04.051467+0000","last_undegraded":"2024-03-19T11:31:04.051467+0000","last_fullsized":"2024-03-19T11:31:04.051467+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":5,"last_scrub":"40'56","last_scrub_stamp":"2024-03-19T11:30:54.998282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:54.998282+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:09.561569+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.014000106999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"2"},{"start":"14","length":"1"}]},{"pgid":"4.13","version":"89'38","reported_seq":140,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.143741+0000","last_change":"2024-03-19T11:31:08.107232+0000","last_active":"2024-03-19T11:31:08.143741+0000","last_peered":"2024-03-19T11:31:08.143741+0000","last_clean":"2024-03-19T11:31:08.143741+0000","last_became_active":"2024-03-19T11:30:45.245367+0000","last_became_peered":"2024-03-19T11:30:45.245367+0000","last_unstale":"2024-03-19T11:31:08.143741+0000","last_undegraded":"2024-03-19T11:31:08.143741+0000","last_fullsized":"2024-03-19T11:31:08.143741+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":5,"last_scrub":"48'22","last_scrub_stamp":"2024-03-19T11:30:27.183294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:27.183294+0000","objects_scrubbed":2,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:41.607013+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.039325774000000001,"stat_sum":{"num_bytes":9406330,"num_objects":4,"num_object_clones":2,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":35,"num_read_kb":30733,"num_write":15,"num_write_kb":4523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":3001789,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.12","version":"28'6","reported_seq":33,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.055008+0000","last_change":"2024-03-19T11:31:04.055613+0000","last_active":"2024-03-19T11:31:04.055008+0000","last_peered":"2024-03-19T11:31:04.055008+0000","last_clean":"2024-03-19T11:31:04.055008+0000","last_became_active":"2024-03-19T11:31:00.064380+0000","last_became_peered":"2024-03-19T11:31:00.064380+0000","last_unstale":"2024-03-19T11:31:04.055008+0000","last_undegraded":"2024-03-19T11:31:04.055008+0000","last_fullsized":"2024-03-19T11:31:04.055008+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":5,"last_scrub":"28'6","last_scrub_stamp":"2024-03-19T11:30:42.390965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.390965+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077181699999999995,"stat_sum":{"num_bytes":1176474,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"},{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.11","version":"83'81","reported_seq":178,"reported_epoch":85,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:04.050088+0000","last_change":"2024-03-19T11:31:02.092082+0000","last_active":"2024-03-19T11:31:04.050088+0000","last_peered":"2024-03-19T11:31:04.050088+0000","last_clean":"2024-03-19T11:31:04.050088+0000","last_became_active":"2024-03-19T11:30:30.052184+0000","last_became_peered":"2024-03-19T11:30:30.052184+0000","last_unstale":"2024-03-19T11:31:04.050088+0000","last_undegraded":"2024-03-19T11:31:04.050088+0000","last_fullsized":"2024-03-19T11:31:04.050088+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":5,"last_scrub":"47'60","last_scrub_stamp":"2024-03-19T11:30:30.780348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:30.780348+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:31.151678+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.014285984999999999,"stat_sum":{"num_bytes":8084512,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":20,"num_read_kb":23163,"num_write":14,"num_write_kb":4028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.10","version":"85'34","reported_seq":340,"reported_epoch":85,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:04.100696+0000","last_change":"2024-03-19T11:31:04.100696+0000","last_active":"2024-03-19T11:31:04.100696+0000","last_peered":"2024-03-19T11:31:04.100696+0000","last_clean":"2024-03-19T11:31:04.100696+0000","last_became_active":"2024-03-19T11:31:00.065617+0000","last_became_peered":"2024-03-19T11:31:00.065617+0000","last_unstale":"2024-03-19T11:31:04.100696+0000","last_undegraded":"2024-03-19T11:31:04.100696+0000","last_fullsized":"2024-03-19T11:31:04.100696+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":5,"last_scrub":"50'30","last_scrub_stamp":"2024-03-19T11:30:28.787058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:28.787058+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:51.738618+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.0079239749999999998,"stat_sum":{"num_bytes":3462209,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":20772,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":7102596,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":128,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.054605+0000","last_change":"2024-03-19T11:30:03.276845+0000","last_active":"2024-03-19T11:31:04.054605+0000","last_peered":"2024-03-19T11:31:04.054605+0000","last_clean":"2024-03-19T11:31:04.054605+0000","last_became_active":"2024-03-19T11:30:03.276220+0000","last_became_peered":"2024-03-19T11:30:03.276220+0000","last_unstale":"2024-03-19T11:31:04.054605+0000","last_undegraded":"2024-03-19T11:31:04.054605+0000","last_fullsized":"2024-03-19T11:31:04.054605+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:07.472401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"59'92","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.071050+0000","last_change":"2024-03-19T11:31:08.072335+0000","last_active":"2024-03-19T11:31:08.071050+0000","last_peered":"2024-03-19T11:31:08.071050+0000","last_clean":"2024-03-19T11:31:08.071050+0000","last_became_active":"2024-03-19T11:31:03.052657+0000","last_became_peered":"2024-03-19T11:31:03.052657+0000","last_unstale":"2024-03-19T11:31:08.071050+0000","last_undegraded":"2024-03-19T11:31:08.071050+0000","last_fullsized":"2024-03-19T11:31:08.071050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'92","last_scrub_stamp":"2024-03-19T11:30:41.351520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:41.351520+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:51.023342+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.012111200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.055239+0000","last_change":"2024-03-19T11:31:00.067375+0000","last_active":"2024-03-19T11:31:04.055239+0000","last_peered":"2024-03-19T11:31:04.055239+0000","last_clean":"2024-03-19T11:31:04.055239+0000","last_became_active":"2024-03-19T11:31:00.067099+0000","last_became_peered":"2024-03-19T11:31:00.067099+0000","last_unstale":"2024-03-19T11:31:04.055239+0000","last_undegraded":"2024-03-19T11:31:04.055239+0000","last_fullsized":"2024-03-19T11:31:04.055239+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:06.356796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"79'65","reported_seq":347,"reported_epoch":85,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:04.050073+0000","last_change":"2024-03-19T11:31:02.078237+0000","last_active":"2024-03-19T11:31:04.050073+0000","last_peered":"2024-03-19T11:31:04.050073+0000","last_clean":"2024-03-19T11:31:04.050073+0000","last_became_active":"2024-03-19T11:30:38.065693+0000","last_became_peered":"2024-03-19T11:30:38.065693+0000","last_unstale":"2024-03-19T11:31:04.050073+0000","last_undegraded":"2024-03-19T11:31:04.050073+0000","last_fullsized":"2024-03-19T11:31:04.050073+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'57","last_scrub_stamp":"2024-03-19T11:30:43.792326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:43.792326+0000","objects_scrubbed":6,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:55.566850+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.022132443000000002,"stat_sum":{"num_bytes":9323261,"num_objects":6,"num_object_clones":5,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":11,"num_read_kb":16661,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":160,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.052066+0000","last_change":"2024-03-19T11:30:59.072585+0000","last_active":"2024-03-19T11:31:04.052066+0000","last_peered":"2024-03-19T11:31:04.052066+0000","last_clean":"2024-03-19T11:31:04.052066+0000","last_became_active":"2024-03-19T11:30:59.071925+0000","last_became_peered":"2024-03-19T11:30:59.071925+0000","last_unstale":"2024-03-19T11:31:04.052066+0000","last_undegraded":"2024-03-19T11:31:04.052066+0000","last_fullsized":"2024-03-19T11:31:04.052066+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:08.925537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"41'19","reported_seq":317,"reported_epoch":88,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:07.098065+0000","last_change":"2024-03-19T11:31:07.081723+0000","last_active":"2024-03-19T11:31:07.098065+0000","last_peered":"2024-03-19T11:31:07.098065+0000","last_clean":"2024-03-19T11:31:07.098065+0000","last_became_active":"2024-03-19T11:30:38.058860+0000","last_became_peered":"2024-03-19T11:30:38.058860+0000","last_unstale":"2024-03-19T11:31:07.098065+0000","last_undegraded":"2024-03-19T11:31:07.098065+0000","last_fullsized":"2024-03-19T11:31:07.098065+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'19","last_scrub_stamp":"2024-03-19T11:30:28.180772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:28.180772+0000","objects_scrubbed":1,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":1.0172436659999999,"stat_sum":{"num_bytes":1403368,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4077,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"6","length":"1"},{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":153,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.055174+0000","last_change":"2024-03-19T11:30:55.366872+0000","last_active":"2024-03-19T11:31:04.055174+0000","last_peered":"2024-03-19T11:31:04.055174+0000","last_clean":"2024-03-19T11:31:04.055174+0000","last_became_active":"2024-03-19T11:29:46.540939+0000","last_became_peered":"2024-03-19T11:29:46.540939+0000","last_unstale":"2024-03-19T11:31:04.055174+0000","last_undegraded":"2024-03-19T11:31:04.055174+0000","last_fullsized":"2024-03-19T11:31:04.055174+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:30:55.366631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:30:55.366631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:10.640223+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"85'17","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.067823+0000","last_change":"2024-03-19T11:31:04.068099+0000","last_active":"2024-03-19T11:31:04.067823+0000","last_peered":"2024-03-19T11:31:04.067823+0000","last_clean":"2024-03-19T11:31:04.067823+0000","last_became_active":"2024-03-19T11:31:00.069200+0000","last_became_peered":"2024-03-19T11:31:00.069200+0000","last_unstale":"2024-03-19T11:31:04.067823+0000","last_undegraded":"2024-03-19T11:31:04.067823+0000","last_fullsized":"2024-03-19T11:31:04.067823+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'10","last_scrub_stamp":"2024-03-19T11:30:54.429030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:54.429030+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.013022977999999999,"stat_sum":{"num_bytes":2742425,"num_objects":1,"num_object_clones":1,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":10173,"num_write":4,"num_write_kb":1071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2800509,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"},{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"3.3","version":"87'107","reported_seq":357,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.067397+0000","last_change":"2024-03-19T11:31:06.067504+0000","last_active":"2024-03-19T11:31:08.067397+0000","last_peered":"2024-03-19T11:31:08.067397+0000","last_clean":"2024-03-19T11:31:08.067397+0000","last_became_active":"2024-03-19T11:31:06.066161+0000","last_became_peered":"2024-03-19T11:31:06.066161+0000","last_unstale":"2024-03-19T11:31:08.067397+0000","last_undegraded":"2024-03-19T11:31:08.067397+0000","last_fullsized":"2024-03-19T11:31:08.067397+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:10.489969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47923414,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":80,"num_read_kb":3742,"num_write":115,"num_write_kb":59833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":25111105,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":144,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.068842+0000","last_change":"2024-03-19T11:29:46.321255+0000","last_active":"2024-03-19T11:31:04.068842+0000","last_peered":"2024-03-19T11:31:04.068842+0000","last_clean":"2024-03-19T11:31:04.068842+0000","last_became_active":"2024-03-19T11:29:46.320679+0000","last_became_peered":"2024-03-19T11:29:46.320679+0000","last_unstale":"2024-03-19T11:31:04.068842+0000","last_undegraded":"2024-03-19T11:31:04.068842+0000","last_fullsized":"2024-03-19T11:31:04.068842+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:14.793563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"85'132","reported_seq":508,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.061158+0000","last_change":"2024-03-19T11:31:04.061533+0000","last_active":"2024-03-19T11:31:04.061158+0000","last_peered":"2024-03-19T11:31:04.061158+0000","last_clean":"2024-03-19T11:31:04.061158+0000","last_became_active":"2024-03-19T11:31:03.054624+0000","last_became_peered":"2024-03-19T11:31:03.054624+0000","last_unstale":"2024-03-19T11:31:04.061158+0000","last_undegraded":"2024-03-19T11:31:04.061158+0000","last_fullsized":"2024-03-19T11:31:04.061158+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'119","last_scrub_stamp":"2024-03-19T11:30:52.512203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:52.512203+0000","objects_scrubbed":7,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:02.677965+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.0068329189999999998,"stat_sum":{"num_bytes":14273317,"num_objects":6,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":70,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"},{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"3.0","version":"85'100","reported_seq":247,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.068253+0000","last_change":"2024-03-19T11:29:57.550320+0000","last_active":"2024-03-19T11:31:04.068253+0000","last_peered":"2024-03-19T11:31:04.068253+0000","last_clean":"2024-03-19T11:31:04.068253+0000","last_became_active":"2024-03-19T11:29:57.549928+0000","last_became_peered":"2024-03-19T11:29:57.549928+0000","last_unstale":"2024-03-19T11:31:04.068253+0000","last_undegraded":"2024-03-19T11:31:04.068253+0000","last_fullsized":"2024-03-19T11:31:04.068253+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28242922,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":80,"num_read_kb":3042,"num_write":105,"num_write_kb":39820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":146,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.053846+0000","last_change":"2024-03-19T11:29:46.319811+0000","last_active":"2024-03-19T11:31:04.053846+0000","last_peered":"2024-03-19T11:31:04.053846+0000","last_clean":"2024-03-19T11:31:04.053846+0000","last_became_active":"2024-03-19T11:29:46.319334+0000","last_became_peered":"2024-03-19T11:29:46.319334+0000","last_unstale":"2024-03-19T11:31:04.053846+0000","last_undegraded":"2024-03-19T11:31:04.053846+0000","last_fullsized":"2024-03-19T11:31:04.053846+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:04.320463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"61'40","reported_seq":33,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.065031+0000","last_change":"2024-03-19T11:31:04.065328+0000","last_active":"2024-03-19T11:31:04.065031+0000","last_peered":"2024-03-19T11:31:04.065031+0000","last_clean":"2024-03-19T11:31:04.065031+0000","last_became_active":"2024-03-19T11:31:00.069809+0000","last_became_peered":"2024-03-19T11:31:00.069809+0000","last_unstale":"2024-03-19T11:31:04.065031+0000","last_undegraded":"2024-03-19T11:31:04.065031+0000","last_fullsized":"2024-03-19T11:31:04.065031+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'40","last_scrub_stamp":"2024-03-19T11:30:56.505586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:56.505586+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:09.446385+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0098007430000000006,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"},{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"3.1","version":"88'90","reported_seq":382,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-19T11:31:07.160859+0000","last_change":"2024-03-19T11:31:03.255027+0000","last_active":"2024-03-19T11:31:07.160859+0000","last_peered":"2024-03-19T11:31:07.160859+0000","last_clean":"2024-03-19T11:31:07.160859+0000","last_became_active":"2024-03-19T11:31:03.077272+0000","last_became_peered":"2024-03-19T11:31:03.077272+0000","last_unstale":"2024-03-19T11:31:07.160859+0000","last_undegraded":"2024-03-19T11:31:07.160859+0000","last_fullsized":"2024-03-19T11:31:07.160859+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:23.954117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44609286,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":74,"num_read_kb":6654,"num_write":94,"num_write_kb":53294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":42087532,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":155,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.052299+0000","last_change":"2024-03-19T11:31:00.070027+0000","last_active":"2024-03-19T11:31:04.052299+0000","last_peered":"2024-03-19T11:31:04.052299+0000","last_clean":"2024-03-19T11:31:04.052299+0000","last_became_active":"2024-03-19T11:31:00.068989+0000","last_became_peered":"2024-03-19T11:31:00.068989+0000","last_unstale":"2024-03-19T11:31:04.052299+0000","last_undegraded":"2024-03-19T11:31:04.052299+0000","last_fullsized":"2024-03-19T11:31:04.052299+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:11.599600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"85'70","reported_seq":70,"reported_epoch":88,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:07.116943+0000","last_change":"2024-03-19T11:31:07.098353+0000","last_active":"2024-03-19T11:31:07.116943+0000","last_peered":"2024-03-19T11:31:07.116943+0000","last_clean":"2024-03-19T11:31:07.116943+0000","last_became_active":"2024-03-19T11:30:59.075057+0000","last_became_peered":"2024-03-19T11:30:59.075057+0000","last_unstale":"2024-03-19T11:31:07.116943+0000","last_undegraded":"2024-03-19T11:31:07.116943+0000","last_fullsized":"2024-03-19T11:31:07.116943+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'65","last_scrub_stamp":"2024-03-19T11:30:53.385414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:53.385414+0000","objects_scrubbed":3,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:20.971419+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":1.0213928809999999,"stat_sum":{"num_bytes":6521083,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":31459,"num_write":4,"num_write_kb":1430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4209633,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"6","length":"1"},{"start":"c","length":"2"},{"start":"11","length":"2"},{"start":"14","length":"1"}]},{"pgid":"3.2","version":"85'84","reported_seq":239,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.049868+0000","last_change":"2024-03-19T11:29:57.549277+0000","last_active":"2024-03-19T11:31:04.049868+0000","last_peered":"2024-03-19T11:31:04.049868+0000","last_clean":"2024-03-19T11:31:04.049868+0000","last_became_active":"2024-03-19T11:29:57.548946+0000","last_became_peered":"2024-03-19T11:29:57.548946+0000","last_unstale":"2024-03-19T11:31:04.049868+0000","last_undegraded":"2024-03-19T11:31:04.049868+0000","last_fullsized":"2024-03-19T11:31:04.049868+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:08.380405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39345556,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":76,"num_read_kb":8248,"num_write":94,"num_write_kb":41080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":144,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.067990+0000","last_change":"2024-03-19T11:29:46.322491+0000","last_active":"2024-03-19T11:31:04.067990+0000","last_peered":"2024-03-19T11:31:04.067990+0000","last_clean":"2024-03-19T11:31:04.067990+0000","last_became_active":"2024-03-19T11:29:46.321760+0000","last_became_peered":"2024-03-19T11:29:46.321760+0000","last_unstale":"2024-03-19T11:31:04.067990+0000","last_undegraded":"2024-03-19T11:31:04.067990+0000","last_fullsized":"2024-03-19T11:31:04.067990+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:04.561915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":329,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.052318+0000","last_change":"2024-03-19T11:31:03.367805+0000","last_active":"2024-03-19T11:31:04.052318+0000","last_peered":"2024-03-19T11:31:04.052318+0000","last_clean":"2024-03-19T11:31:04.052318+0000","last_became_active":"2024-03-19T11:30:59.073709+0000","last_became_peered":"2024-03-19T11:30:59.073709+0000","last_unstale":"2024-03-19T11:31:04.052318+0000","last_undegraded":"2024-03-19T11:31:04.052318+0000","last_fullsized":"2024-03-19T11:31:04.052318+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2024-03-19T11:31:03.367669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:43.166618+0000","last_clean_scrub_stamp":"2024-03-19T11:31:03.367669+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:29.977573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"88'111","reported_seq":515,"reported_epoch":89,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:08.083914+0000","last_change":"2024-03-19T11:31:07.189608+0000","last_active":"2024-03-19T11:31:08.083914+0000","last_peered":"2024-03-19T11:31:08.083914+0000","last_clean":"2024-03-19T11:31:08.083914+0000","last_became_active":"2024-03-19T11:31:03.065344+0000","last_became_peered":"2024-03-19T11:31:03.065344+0000","last_unstale":"2024-03-19T11:31:08.083914+0000","last_undegraded":"2024-03-19T11:31:08.083914+0000","last_fullsized":"2024-03-19T11:31:08.083914+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'81","last_scrub_stamp":"2024-03-19T11:30:52.360396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:52.360396+0000","objects_scrubbed":6,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:14.659997+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.103481396,"stat_sum":{"num_bytes":10838384,"num_objects":7,"num_object_clones":5,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":13,"num_read_kb":12349,"num_write":8,"num_write_kb":1273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":10558635,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"1"}]},{"pgid":"4.8","version":"62'71","reported_seq":359,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.077614+0000","last_change":"2024-03-19T11:31:04.078135+0000","last_active":"2024-03-19T11:31:04.077614+0000","last_peered":"2024-03-19T11:31:04.077614+0000","last_clean":"2024-03-19T11:31:04.077614+0000","last_became_active":"2024-03-19T11:30:45.251283+0000","last_became_peered":"2024-03-19T11:30:45.251283+0000","last_unstale":"2024-03-19T11:31:04.077614+0000","last_undegraded":"2024-03-19T11:31:04.077614+0000","last_fullsized":"2024-03-19T11:31:04.077614+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'71","last_scrub_stamp":"2024-03-19T11:30:53.381386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:53.381386+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:18.275213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010232903999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6","length":"1"},{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.9","version":"63'55","reported_seq":415,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.075645+0000","last_change":"2024-03-19T11:31:08.076672+0000","last_active":"2024-03-19T11:31:08.075645+0000","last_peered":"2024-03-19T11:31:08.075645+0000","last_clean":"2024-03-19T11:31:08.075645+0000","last_became_active":"2024-03-19T11:31:00.069789+0000","last_became_peered":"2024-03-19T11:31:00.069789+0000","last_unstale":"2024-03-19T11:31:08.075645+0000","last_undegraded":"2024-03-19T11:31:08.075645+0000","last_fullsized":"2024-03-19T11:31:08.075645+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'55","last_scrub_stamp":"2024-03-19T11:30:54.706235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:54.706235+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:10.998788+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.015421028,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.a","version":"62'77","reported_seq":406,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.117764+0000","last_change":"2024-03-19T11:31:08.118531+0000","last_active":"2024-03-19T11:31:08.117764+0000","last_peered":"2024-03-19T11:31:08.117764+0000","last_clean":"2024-03-19T11:31:08.117764+0000","last_became_active":"2024-03-19T11:30:45.256124+0000","last_became_peered":"2024-03-19T11:30:45.256124+0000","last_unstale":"2024-03-19T11:31:08.117764+0000","last_undegraded":"2024-03-19T11:31:08.117764+0000","last_fullsized":"2024-03-19T11:31:08.117764+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'77","last_scrub_stamp":"2024-03-19T11:30:55.291823+0000","last_deep_scrub":"62'77","last_deep_scrub_stamp":"2024-03-19T11:30:55.291823+0000","last_clean_scrub_stamp":"2024-03-19T11:30:55.291823+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:58.611283+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.033194802000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.b","version":"85'64","reported_seq":394,"reported_epoch":87,"state":"peering","last_fresh":"2024-03-19T11:31:06.074443+0000","last_change":"2024-03-19T11:31:07.098974+0000","last_active":"2024-03-19T11:31:05.071268+0000","last_peered":"2024-03-19T11:31:06.074443+0000","last_clean":"2024-03-19T11:31:06.074443+0000","last_became_active":"2024-03-19T11:30:45.261756+0000","last_became_peered":"2024-03-19T11:31:06.072241+0000","last_unstale":"2024-03-19T11:31:06.074443+0000","last_undegraded":"2024-03-19T11:31:06.074443+0000","last_fullsized":"2024-03-19T11:31:06.074443+0000","mapping_epoch":21,"log_start":"85'64","ondisk_log_start":"85'64","created":21,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'62","last_scrub_stamp":"2024-03-19T11:30:56.095039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:56.095039+0000","objects_scrubbed":3,"log_size":0,"log_dups_size":30,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063617100000000005,"stat_sum":{"num_bytes":5215740,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.c","version":"56'29","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.072859+0000","last_change":"2024-03-19T11:31:08.074168+0000","last_active":"2024-03-19T11:31:08.072859+0000","last_peered":"2024-03-19T11:31:08.072859+0000","last_clean":"2024-03-19T11:31:08.072859+0000","last_became_active":"2024-03-19T11:31:05.073655+0000","last_became_peered":"2024-03-19T11:31:05.073655+0000","last_unstale":"2024-03-19T11:31:08.072859+0000","last_undegraded":"2024-03-19T11:31:08.072859+0000","last_fullsized":"2024-03-19T11:31:08.072859+0000","mapping_epoch":83,"log_start":"56'29","ondisk_log_start":"56'29","created":21,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'29","last_scrub_stamp":"2024-03-19T11:30:53.743385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:53.743385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:56.702924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013861157000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.d","version":"87'88","reported_seq":434,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.086044+0000","last_change":"2024-03-19T11:31:08.086044+0000","last_active":"2024-03-19T11:31:08.086044+0000","last_peered":"2024-03-19T11:31:08.086044+0000","last_clean":"2024-03-19T11:31:08.086044+0000","last_became_active":"2024-03-19T11:31:06.068526+0000","last_became_peered":"2024-03-19T11:31:06.068526+0000","last_unstale":"2024-03-19T11:31:08.086044+0000","last_undegraded":"2024-03-19T11:31:08.086044+0000","last_fullsized":"2024-03-19T11:31:08.086044+0000","mapping_epoch":86,"log_start":"70'71","ondisk_log_start":"70'71","created":21,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'73","last_scrub_stamp":"2024-03-19T11:30:53.519178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:53.519178+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":50,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:02.045989+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.021539165999999998,"stat_sum":{"num_bytes":6903494,"num_objects":4,"num_object_clones":2,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":18,"num_read_kb":22670,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":8146998,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.e","version":"89'94","reported_seq":402,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.081152+0000","last_change":"2024-03-19T11:31:08.081889+0000","last_active":"2024-03-19T11:31:08.081152+0000","last_peered":"2024-03-19T11:31:08.081152+0000","last_clean":"2024-03-19T11:31:08.081152+0000","last_became_active":"2024-03-19T11:30:38.060487+0000","last_became_peered":"2024-03-19T11:30:38.060487+0000","last_unstale":"2024-03-19T11:31:08.081152+0000","last_undegraded":"2024-03-19T11:31:08.081152+0000","last_fullsized":"2024-03-19T11:31:08.081152+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'65","last_scrub_stamp":"2024-03-19T11:30:43.824620+0000","last_deep_scrub":"65'65","last_deep_scrub_stamp":"2024-03-19T11:30:43.824620+0000","last_clean_scrub_stamp":"2024-03-19T11:30:43.824620+0000","objects_scrubbed":4,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:51.683705+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0.020120889999999999,"stat_sum":{"num_bytes":10000429,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":37,"num_read_kb":41805,"num_write":10,"num_write_kb":2301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.f","version":"0'0","reported_seq":256,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T11:31:04.065434+0000","last_change":"2024-03-19T11:31:04.065959+0000","last_active":"2024-03-19T11:31:04.065434+0000","last_peered":"2024-03-19T11:31:04.065434+0000","last_clean":"2024-03-19T11:31:04.065434+0000","last_became_active":"2024-03-19T11:30:38.050919+0000","last_became_peered":"2024-03-19T11:30:38.050919+0000","last_unstale":"2024-03-19T11:31:04.065434+0000","last_undegraded":"2024-03-19T11:31:04.065434+0000","last_fullsized":"2024-03-19T11:31:04.065434+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:30:43.411893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:43.411893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:54.719455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010140278000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"},{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]}],"pool_stats":[{"poolid":4,"num_pg":43,"stat_sum":{"num_bytes":266714116,"num_objects":136,"num_object_clones":85,"num_object_copies":272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":133,"num_whiteouts":7,"num_read":796,"num_read_kb":867257,"num_write":233,"num_write_kb":59863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":158,"num_bytes_recovered":205189935,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":91,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":397172736,"data_stored":458832902,"data_compressed":2800833,"data_compressed_allocated":1785856,"data_compressed_original":6291456,"omap_allocated":0,"internal_metadata":0},"log_size":2764,"ondisk_log_size":2764,"up":86,"acting":86,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":160121178,"num_objects":143,"num_object_clones":0,"num_object_copies":286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":143,"num_whiteouts":0,"num_read":310,"num_read_kb":21686,"num_write":408,"num_write_kb":194027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":67198637,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":204435456,"data_stored":221028451,"data_compressed":15505796,"data_compressed_allocated":16572416,"data_compressed_original":33583104,"omap_allocated":0,"internal_metadata":0},"log_size":381,"ondisk_log_size":381,"up":8,"acting":8,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":1586784,"data_compressed":19832,"data_compressed_allocated":98304,"data_compressed_original":1560576,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542164,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83364,"kb_used_data":52184,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94288476,"statfs":{"total":96636764160,"available":96551399424,"internally_reserved":0,"allocated":53436416,"data_stored":62427174,"data_compressed":227710,"data_compressed_allocated":151552,"data_compressed_original":1101824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542164,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84092,"kb_used_data":52908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94287748,"statfs":{"total":96636764160,"available":96550653952,"internally_reserved":0,"allocated":54177792,"data_stored":62533221,"data_compressed":1917086,"data_compressed_allocated":1929216,"data_compressed_original":4210688,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542164,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":129452,"kb_used_data":102432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94242388,"statfs":{"total":96636764160,"available":96504205312,"internally_reserved":0,"allocated":104890368,"data_stored":120523535,"data_compressed":2532072,"data_compressed_allocated":2379776,"data_compressed_original":5521408,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542163,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79208,"kb_used_data":52184,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94292632,"statfs":{"total":96636764160,"available":96555655168,"internally_reserved":0,"allocated":53436416,"data_stored":60429347,"data_compressed":1749130,"data_compressed_allocated":1527808,"data_compressed_original":4194304,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542163,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":153924,"kb_used_data":113028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94217916,"statfs":{"total":96636764160,"available":96479145984,"internally_reserved":0,"allocated":115740672,"data_stored":123349286,"data_compressed":1407043,"data_compressed_allocated":1449984,"data_compressed_original":3211264,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574868,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64032,"kb_used_data":32844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94307808,"statfs":{"total":96636764160,"available":96571195392,"internally_reserved":0,"allocated":33632256,"data_stored":43833336,"data_compressed":169512,"data_compressed_allocated":57344,"data_compressed_original":196608,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574868,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":137620,"kb_used_data":106440,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94234220,"statfs":{"total":96636764160,"available":96495841280,"internally_reserved":0,"allocated":108994560,"data_stored":119204649,"data_compressed":4084412,"data_compressed_allocated":4292608,"data_compressed_original":8638464,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574868,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":100364,"kb_used_data":69184,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94271476,"statfs":{"total":96636764160,"available":96533991424,"internally_reserved":0,"allocated":70844416,"data_stored":76668922,"data_compressed":4021981,"data_compressed_allocated":4272128,"data_compressed_original":9228288,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6884,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6884,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6064,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":52113408,"data_stored":56190278,"data_compressed":3933827,"data_compressed_allocated":4194304,"data_compressed_original":8376320,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14168064,"data_stored":15397438,"data_compressed":898614,"data_compressed_allocated":970752,"data_compressed_original":2228224,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14168064,"data_stored":15397438,"data_compressed":898614,"data_compressed_allocated":970752,"data_compressed_original":2228224,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":22126592,"data_stored":23997131,"data_compressed":1849977,"data_compressed_allocated":1974272,"data_compressed_original":3883008,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":51625984,"data_stored":55733062,"data_compressed":3965051,"data_compressed_allocated":4227072,"data_compressed_original":8441856,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":19947520,"data_stored":21724875,"data_compressed":1695440,"data_compressed_allocated":1810432,"data_compressed_original":3620864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":30285824,"data_stored":32588229,"data_compressed":2264273,"data_compressed_allocated":2424832,"data_compressed_original":4804608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":19746816,"data_stored":21578762,"data_compressed":233168,"data_compressed_allocated":204800,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90685440,"data_stored":97008266,"data_compressed":390047,"data_compressed_allocated":352256,"data_compressed_original":720896,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":34271232,"data_stored":39685709,"data_compressed":732134,"data_compressed_allocated":430080,"data_compressed_original":1703936,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":78860288,"data_stored":93060511,"data_compressed":682095,"data_compressed_allocated":405504,"data_compressed_original":1638400,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":59510784,"data_stored":67285511,"data_compressed":150585,"data_compressed_allocated":98304,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":29360128,"data_stored":35596983,"data_compressed":221646,"data_compressed_allocated":118784,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36229120,"data_stored":48101703,"data_compressed":169512,"data_compressed_allocated":57344,"data_compressed_original":196608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":48508928,"data_stored":56515457,"data_compressed":221646,"data_compressed_allocated":118784,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T11:31:09.929 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-19T11:31:09.929 INFO:tasks.rados.rados.0.smithi038.stdout:1266: finishing write tid 4 to smithi03829856-17 2024-03-19T11:31:09.929 INFO:tasks.rados.rados.0.smithi038.stdout:1266: finishing write tid 5 to smithi03829856-17 2024-03-19T11:31:09.930 INFO:tasks.rados.rados.0.smithi038.stdout:1266: oid 17 updating version 0 to 182 2024-03-19T11:31:09.930 INFO:tasks.rados.rados.0.smithi038.stdout:1266: oid 17 version 182 is already newer than 181 2024-03-19T11:31:09.930 INFO:tasks.rados.rados.0.smithi038.stdout:1266: oid 17 version 182 is already newer than 180 2024-03-19T11:31:09.930 INFO:tasks.rados.rados.0.smithi038.stdout:1266: oid 17 version 182 is already newer than 179 2024-03-19T11:31:09.930 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 17 v 182 (ObjNum 213 snap 22 seq_num 213) dirty exists 2024-03-19T11:31:09.930 INFO:tasks.rados.rados.0.smithi038.stdout:1266: left oid 17 (ObjNum 213 snap 22 seq_num 213) 2024-03-19T11:31:09.930 INFO:tasks.rados.rados.0.smithi038.stdout:1270: expect (ObjNum 34 snap 0 seq_num 34) 2024-03-19T11:31:09.950 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:09.959+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:10.008 INFO:tasks.rados.rados.0.smithi038.stdout:1274: finishing write tid 1 to smithi03829856-33 2024-03-19T11:31:10.008 INFO:tasks.rados.rados.0.smithi038.stdout:1274: finishing write tid 2 to smithi03829856-33 2024-03-19T11:31:10.008 INFO:tasks.rados.rados.0.smithi038.stdout:1276: expect (ObjNum 141 snap 15 seq_num 141) 2024-03-19T11:31:10.040 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-03-19T11:31:10.047 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:10.047 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:10.051 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:10.057+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:10.067 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-19T11:31:10.132 INFO:tasks.rados.rados.0.smithi038.stdout:1274: finishing write tid 3 to smithi03829856-33 2024-03-19T11:31:10.133 INFO:tasks.rados.rados.0.smithi038.stdout:1274: finishing write tid 4 to smithi03829856-33 2024-03-19T11:31:10.133 INFO:tasks.rados.rados.0.smithi038.stdout:1274: finishing write tid 5 to smithi03829856-33 2024-03-19T11:31:10.133 INFO:tasks.rados.rados.0.smithi038.stdout:1274: oid 33 updating version 0 to 120 2024-03-19T11:31:10.133 INFO:tasks.rados.rados.0.smithi038.stdout:1274: oid 33 updating version 120 to 122 2024-03-19T11:31:10.133 INFO:tasks.rados.rados.0.smithi038.stdout:1274: oid 33 version 122 is already newer than 119 2024-03-19T11:31:10.133 INFO:tasks.rados.rados.0.smithi038.stdout:1274: oid 33 version 122 is already newer than 121 2024-03-19T11:31:10.133 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 33 v 122 (ObjNum 214 snap 22 seq_num 214) dirty exists 2024-03-19T11:31:10.133 INFO:tasks.rados.rados.0.smithi038.stdout:1274: left oid 33 (ObjNum 214 snap 22 seq_num 214) 2024-03-19T11:31:10.133 INFO:tasks.rados.rados.0.smithi038.stdout:1277: expect (ObjNum 201 snap 22 seq_num 201) 2024-03-19T11:31:10.151 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:10.161+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:10.222 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:10.222 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:10.235 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-19T11:31:10.253 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:10.263+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:10.278 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:31:10.279 INFO:teuthology.orchestra.run.smithi038.stdout:{"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false},"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"1 pool(s) do not have an application enabled","count":1},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":96,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":91,"num_osds":8,"num_up_osds":8,"osd_up_since":1710847783,"num_in_osds":8,"osd_in_since":1710847775,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":32},{"state_name":"active+clean+snaptrim_wait","count":11},{"state_name":"active+clean+snaptrim","count":9},{"state_name":"peering","count":1},{"state_name":"clean+premerge+peered","count":1},{"state_name":"active+clean+remapped+snaptrim","count":1},{"state_name":"active+recovering+remapped","count":1}],"num_pgs":56,"num_pools":4,"num_objects":288,"data_bytes":442772867,"bytes_used":790462464,"bytes_avail":772303650816,"bytes_total":773094113280,"inactive_pgs_ratio":0.035714287310838699,"misplaced_objects":2,"misplaced_total":576,"misplaced_ratio":0.003472222222222222,"recovering_objects_per_sec":19,"recovering_bytes_per_sec":20539768,"recovering_keys_per_sec":0,"num_objects_recovered":77,"num_bytes_recovered":82184884,"num_keys_recovered":0,"read_bytes_sec":24358428,"write_bytes_sec":3027528,"read_op_per_sec":22,"write_op_per_sec":9},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-03-19T11:30:46.147482+0000","services":{}},"progress_events":{}} 2024-03-19T11:31:10.297 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout 2024-03-19T11:31:10.297 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T11:31:10.332 INFO:tasks.rados.rados.0.smithi038.stdout:1275: expect (ObjNum 79 snap 6 seq_num 79) 2024-03-19T11:31:10.376 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:10.385+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:10.389 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:10.389 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:10.403 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-19T11:31:10.454 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:10.463+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:10.475 INFO:tasks.rados.rados.0.smithi038.stdout:1278: finishing set_chunk to oid 23 2024-03-19T11:31:10.475 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 23 v 92 (ObjNum 190 snap 22 seq_num 190) dirty exists 2024-03-19T11:31:10.475 INFO:tasks.rados.rados.0.smithi038.stdout:1264: done (15 left) 2024-03-19T11:31:10.475 INFO:tasks.rados.rados.0.smithi038.stdout:1265: done (14 left) 2024-03-19T11:31:10.476 INFO:tasks.rados.rados.0.smithi038.stdout:1266: done (13 left) 2024-03-19T11:31:10.476 INFO:tasks.rados.rados.0.smithi038.stdout:1267: done (12 left) 2024-03-19T11:31:10.476 INFO:tasks.rados.rados.0.smithi038.stdout:1268: done (11 left) 2024-03-19T11:31:10.476 INFO:tasks.rados.rados.0.smithi038.stdout:1269: done (10 left) 2024-03-19T11:31:10.476 INFO:tasks.rados.rados.0.smithi038.stdout:1270: done (9 left) 2024-03-19T11:31:10.476 INFO:tasks.rados.rados.0.smithi038.stdout:1271: done (8 left) 2024-03-19T11:31:10.476 INFO:tasks.rados.rados.0.smithi038.stdout:1272: done (7 left) 2024-03-19T11:31:10.476 INFO:tasks.rados.rados.0.smithi038.stdout:1273: done (6 left) 2024-03-19T11:31:10.476 INFO:tasks.rados.rados.0.smithi038.stdout:1274: done (5 left) 2024-03-19T11:31:10.476 INFO:tasks.rados.rados.0.smithi038.stdout:1275: done (4 left) 2024-03-19T11:31:10.476 INFO:tasks.rados.rados.0.smithi038.stdout:1276: done (3 left) 2024-03-19T11:31:10.476 INFO:tasks.rados.rados.0.smithi038.stdout:1277: done (2 left) 2024-03-19T11:31:10.476 INFO:tasks.rados.rados.0.smithi038.stdout:1278: done (1 left) 2024-03-19T11:31:10.476 INFO:tasks.rados.rados.0.smithi038.stdout:1279: done (0 left) 2024-03-19T11:31:10.477 INFO:tasks.rados.rados.0.smithi038.stdout:1280: read oid 2 snap -1 2024-03-19T11:31:10.477 INFO:tasks.rados.rados.0.smithi038.stdout:1280: expect (ObjNum 206 snap 22 seq_num 206) 2024-03-19T11:31:10.477 INFO:tasks.rados.rados.0.smithi038.stdout:1281: read oid 42 snap -1 2024-03-19T11:31:10.477 INFO:tasks.rados.rados.0.smithi038.stdout:1281: expect deleted 2024-03-19T11:31:10.477 INFO:tasks.rados.rados.0.smithi038.stdout:1282: read oid 50 snap 22 2024-03-19T11:31:10.477 INFO:tasks.rados.rados.0.smithi038.stdout:1282: expect deleted 2024-03-19T11:31:10.477 INFO:tasks.rados.rados.0.smithi038.stdout:1283: set_chunk oid 37 target oid 2024-03-19T11:31:10.477 INFO:tasks.rados.rados.0.smithi038.stdout:1283: set_chunk oid 37 offset: 0 length: 4000000 target oid smithi03829856- offset: 0 snap -1 2024-03-19T11:31:10.477 INFO:tasks.rados.rados.0.smithi038.stdout:1284: write (excl) oid 8 current snap is 23 2024-03-19T11:31:10.477 INFO:tasks.rados.rados.0.smithi038.stdout:1284: seq_num 215 ranges {413505=636419,1664805=689064,3038061=492071} 2024-03-19T11:31:10.481 INFO:tasks.rados.rados.0.smithi038.stdout:1284: writing smithi03829856-8 from 413505 to 1049924 tid 1 2024-03-19T11:31:10.490 INFO:tasks.rados.rados.0.smithi038.stdout:1284: writing smithi03829856-8 from 1664805 to 2353869 tid 2 2024-03-19T11:31:10.498 INFO:tasks.rados.rados.0.smithi038.stdout:1284: writing smithi03829856-8 from 3038061 to 3530132 tid 3 2024-03-19T11:31:10.498 INFO:tasks.rados.rados.0.smithi038.stdout:1285: write (excl) oid 19 current snap is 23 2024-03-19T11:31:10.498 INFO:tasks.rados.rados.0.smithi038.stdout:1285: seq_num 216 ranges {430312=711668,1723287=549075,2781837=703343} 2024-03-19T11:31:10.506 INFO:tasks.rados.rados.0.smithi038.stdout:1285: writing smithi03829856-19 from 430312 to 1141980 tid 1 2024-03-19T11:31:10.516 INFO:tasks.rados.rados.0.smithi038.stdout:1285: writing smithi03829856-19 from 1723287 to 2272362 tid 2 2024-03-19T11:31:10.524 INFO:tasks.rados.rados.0.smithi038.stdout:1285: writing smithi03829856-19 from 2781837 to 3485180 tid 3 2024-03-19T11:31:10.524 INFO:tasks.rados.rados.0.smithi038.stdout:1286: tier_promote oid 15 2024-03-19T11:31:10.524 INFO:tasks.rados.rados.0.smithi038.stdout:1283: finishing set_chunk to oid 37 2024-03-19T11:31:10.524 INFO:tasks.rados.rados.0.smithi038.stdout:1283: got expected ERANGE (src dne) 2024-03-19T11:31:10.524 INFO:tasks.rados.rados.0.smithi038.stdout:1280: expect (ObjNum 206 snap 22 seq_num 206) 2024-03-19T11:31:10.555 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:10.561+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:10.589 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:10.589 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:10.606 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-19T11:31:10.649 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:31:10.651 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-03-19T11:31:10.667 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":128,"stamp":"2024-03-19T11:31:10.155735+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":436181515,"num_objects":284,"num_object_clones":83,"num_object_copies":568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":281,"num_whiteouts":8,"num_read":1295,"num_read_kb":986721,"num_write":844,"num_write_kb":272277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":257,"num_bytes_recovered":270645919,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":88,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3345,"ondisk_log_size":3345,"up":112,"acting":112,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":119,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":782656,"kb_used_data":531788,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754192064,"statfs":{"total":773094113280,"available":772292673536,"internally_reserved":0,"allocated":544550912,"data_stored":611534700,"data_compressed":15915368,"data_compressed_allocated":16035840,"data_compressed_original":36499456,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3384543,"num_objects":-3,"num_object_clones":-8,"num_object_copies":-6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":2,"num_read":209,"num_read_kb":249783,"num_write":92,"num_write_kb":30739,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":30465534,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001309"},"pg_stats":[{"pgid":"4.2a","version":"87'100","reported_seq":441,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.119724+0000","last_change":"2024-03-19T11:31:08.120519+0000","last_active":"2024-03-19T11:31:08.119724+0000","last_peered":"2024-03-19T11:31:08.119724+0000","last_clean":"2024-03-19T11:31:08.119724+0000","last_became_active":"2024-03-19T11:30:45.260578+0000","last_became_peered":"2024-03-19T11:30:45.260578+0000","last_unstale":"2024-03-19T11:31:08.119724+0000","last_undegraded":"2024-03-19T11:31:08.119724+0000","last_fullsized":"2024-03-19T11:31:08.119724+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":6,"last_scrub":"74'89","last_scrub_stamp":"2024-03-19T11:30:53.312593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:53.312593+0000","objects_scrubbed":6,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:03.108309+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.034299200000000002,"stat_sum":{"num_bytes":9678155,"num_objects":4,"num_object_clones":2,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":9,"num_read_kb":12963,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.29","version":"85'65","reported_seq":448,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.074347+0000","last_change":"2024-03-19T11:31:08.075407+0000","last_active":"2024-03-19T11:31:08.074347+0000","last_peered":"2024-03-19T11:31:08.074347+0000","last_clean":"2024-03-19T11:31:08.074347+0000","last_became_active":"2024-03-19T11:31:00.070287+0000","last_became_peered":"2024-03-19T11:31:00.070287+0000","last_unstale":"2024-03-19T11:31:08.074347+0000","last_undegraded":"2024-03-19T11:31:08.074347+0000","last_fullsized":"2024-03-19T11:31:08.074347+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":6,"last_scrub":"71'60","last_scrub_stamp":"2024-03-19T11:30:52.748479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:52.748479+0000","objects_scrubbed":3,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:14.667721+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.015091184000000001,"stat_sum":{"num_bytes":4387250,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":19,"num_read_kb":23171,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":5842489,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.28","version":"89'103","reported_seq":408,"reported_epoch":89,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:08.114830+0000","last_change":"2024-03-19T11:31:08.114830+0000","last_active":"2024-03-19T11:31:08.114830+0000","last_peered":"2024-03-19T11:31:08.114830+0000","last_clean":"2024-03-19T11:31:08.114830+0000","last_became_active":"2024-03-19T11:30:45.257600+0000","last_became_peered":"2024-03-19T11:30:45.257600+0000","last_unstale":"2024-03-19T11:31:08.114830+0000","last_undegraded":"2024-03-19T11:31:08.114830+0000","last_fullsized":"2024-03-19T11:31:08.114830+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":6,"last_scrub":"71'85","last_scrub_stamp":"2024-03-19T11:30:52.343454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:52.343454+0000","objects_scrubbed":7,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:57.454276+0000","scrub_duration":7,"objects_trimmed":1,"snaptrim_duration":0.065578227000000003,"stat_sum":{"num_bytes":9128542,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"2"}]},{"pgid":"4.27","version":"87'56","reported_seq":79,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.059325+0000","last_change":"2024-03-19T11:31:08.060066+0000","last_active":"2024-03-19T11:31:08.059325+0000","last_peered":"2024-03-19T11:31:08.059325+0000","last_clean":"2024-03-19T11:31:08.059325+0000","last_became_active":"2024-03-19T11:31:00.070337+0000","last_became_peered":"2024-03-19T11:31:00.070337+0000","last_unstale":"2024-03-19T11:31:08.059325+0000","last_undegraded":"2024-03-19T11:31:08.059325+0000","last_fullsized":"2024-03-19T11:31:08.059325+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":6,"last_scrub":"71'48","last_scrub_stamp":"2024-03-19T11:30:55.467716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:55.467716+0000","objects_scrubbed":4,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:09.107718+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0027578379999999999,"stat_sum":{"num_bytes":5844083,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":11198,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":7070511,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.26","version":"61'63","reported_seq":49,"reported_epoch":88,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:07.098364+0000","last_change":"2024-03-19T11:31:07.098333+0000","last_active":"2024-03-19T11:31:07.098364+0000","last_peered":"2024-03-19T11:31:07.098364+0000","last_clean":"2024-03-19T11:31:07.098364+0000","last_became_active":"2024-03-19T11:30:59.074337+0000","last_became_peered":"2024-03-19T11:30:59.074337+0000","last_unstale":"2024-03-19T11:31:07.098364+0000","last_undegraded":"2024-03-19T11:31:07.098364+0000","last_fullsized":"2024-03-19T11:31:07.098364+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":6,"last_scrub":"61'63","last_scrub_stamp":"2024-03-19T11:30:52.406723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:52.406723+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:22.060620+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":1.020104218,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"6","length":"1"},{"start":"c","length":"2"},{"start":"11","length":"2"},{"start":"14","length":"1"}]},{"pgid":"4.25","version":"56'66","reported_seq":83,"reported_epoch":89,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:08.114708+0000","last_change":"2024-03-19T11:31:08.114800+0000","last_active":"2024-03-19T11:31:08.114708+0000","last_peered":"2024-03-19T11:31:08.114708+0000","last_clean":"2024-03-19T11:31:08.114708+0000","last_became_active":"2024-03-19T11:30:53.084473+0000","last_became_peered":"2024-03-19T11:30:53.084473+0000","last_unstale":"2024-03-19T11:31:08.114708+0000","last_undegraded":"2024-03-19T11:31:08.114708+0000","last_fullsized":"2024-03-19T11:31:08.114708+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":6,"last_scrub":"56'66","last_scrub_stamp":"2024-03-19T11:31:06.587201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:31:06.587201+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:10.867475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.065786145000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"}]},{"pgid":"4.24","version":"85'117","reported_seq":510,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.107628+0000","last_change":"2024-03-19T11:31:08.069824+0000","last_active":"2024-03-19T11:31:08.107628+0000","last_peered":"2024-03-19T11:31:08.107628+0000","last_clean":"2024-03-19T11:31:08.107628+0000","last_became_active":"2024-03-19T11:31:03.063705+0000","last_became_peered":"2024-03-19T11:31:03.063705+0000","last_unstale":"2024-03-19T11:31:08.107628+0000","last_undegraded":"2024-03-19T11:31:08.107628+0000","last_fullsized":"2024-03-19T11:31:08.107628+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":6,"last_scrub":"84'113","last_scrub_stamp":"2024-03-19T11:31:03.304881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:31:03.304881+0000","objects_scrubbed":5,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:06.612630+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.011640286999999999,"stat_sum":{"num_bytes":10102023,"num_objects":4,"num_object_clones":2,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":34100,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.23","version":"87'38","reported_seq":87,"reported_epoch":90,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:09.056901+0000","last_change":"2024-03-19T11:31:06.101937+0000","last_active":"2024-03-19T11:31:09.056901+0000","last_peered":"2024-03-19T11:31:09.056901+0000","last_clean":"2024-03-19T11:31:09.056901+0000","last_became_active":"2024-03-19T11:30:53.076092+0000","last_became_peered":"2024-03-19T11:30:53.076092+0000","last_unstale":"2024-03-19T11:31:09.056901+0000","last_undegraded":"2024-03-19T11:31:09.056901+0000","last_fullsized":"2024-03-19T11:31:09.056901+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":6,"last_scrub":"62'23","last_scrub_stamp":"2024-03-19T11:30:43.163509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:43.163509+0000","objects_scrubbed":2,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:10.968084+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.035799641,"stat_sum":{"num_bytes":6154918,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":10,"num_write_kb":2870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2833035,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"6","length":"1"},{"start":"11","length":"1"}]},{"pgid":"4.22","version":"28'6","reported_seq":86,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.075051+0000","last_change":"2024-03-19T11:31:08.076088+0000","last_active":"2024-03-19T11:31:08.075051+0000","last_peered":"2024-03-19T11:31:08.075051+0000","last_clean":"2024-03-19T11:31:08.075051+0000","last_became_active":"2024-03-19T11:30:53.068010+0000","last_became_peered":"2024-03-19T11:30:53.068010+0000","last_unstale":"2024-03-19T11:31:08.075051+0000","last_undegraded":"2024-03-19T11:31:08.075051+0000","last_fullsized":"2024-03-19T11:31:08.075051+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":6,"last_scrub":"28'6","last_scrub_stamp":"2024-03-19T11:30:55.724671+0000","last_deep_scrub":"28'6","last_deep_scrub_stamp":"2024-03-19T11:30:55.724671+0000","last_clean_scrub_stamp":"2024-03-19T11:30:55.724671+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:13.244591+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.015426808,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.21","version":"89'133","reported_seq":524,"reported_epoch":90,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:09.056335+0000","last_change":"2024-03-19T11:31:08.112958+0000","last_active":"2024-03-19T11:31:09.056335+0000","last_peered":"2024-03-19T11:31:09.056335+0000","last_clean":"2024-03-19T11:31:09.056335+0000","last_became_active":"2024-03-19T11:31:00.069759+0000","last_became_peered":"2024-03-19T11:31:00.069759+0000","last_unstale":"2024-03-19T11:31:09.056335+0000","last_undegraded":"2024-03-19T11:31:09.056335+0000","last_fullsized":"2024-03-19T11:31:09.056335+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":6,"last_scrub":"62'96","last_scrub_stamp":"2024-03-19T11:30:43.400885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:43.400885+0000","objects_scrubbed":9,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:12.660708+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.050771269000000001,"stat_sum":{"num_bytes":16636184,"num_objects":7,"num_object_clones":5,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":49,"num_read_kb":51400,"num_write":23,"num_write_kb":6523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":22983003,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d","length":"1"}]},{"pgid":"4.20","version":"56'46","reported_seq":78,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.116754+0000","last_change":"2024-03-19T11:31:08.117528+0000","last_active":"2024-03-19T11:31:08.116754+0000","last_peered":"2024-03-19T11:31:08.116754+0000","last_clean":"2024-03-19T11:31:08.116754+0000","last_became_active":"2024-03-19T11:30:53.067251+0000","last_became_peered":"2024-03-19T11:30:53.067251+0000","last_unstale":"2024-03-19T11:31:08.116754+0000","last_undegraded":"2024-03-19T11:31:08.116754+0000","last_fullsized":"2024-03-19T11:31:08.116754+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":6,"last_scrub":"56'46","last_scrub_stamp":"2024-03-19T11:30:41.767100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:41.767100+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:04.278716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.032365520000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.1f","version":"0'0","reported_seq":86,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.115029+0000","last_change":"2024-03-19T11:31:08.116595+0000","last_active":"2024-03-19T11:31:08.115029+0000","last_peered":"2024-03-19T11:31:08.115029+0000","last_clean":"2024-03-19T11:31:08.115029+0000","last_became_active":"2024-03-19T11:30:53.087211+0000","last_became_peered":"2024-03-19T11:30:53.087211+0000","last_unstale":"2024-03-19T11:31:08.115029+0000","last_undegraded":"2024-03-19T11:31:08.115029+0000","last_fullsized":"2024-03-19T11:31:08.115029+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:30:42.377963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.377963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:05.818292+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.032679030999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.1e","version":"87'71","reported_seq":82,"reported_epoch":88,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:07.098810+0000","last_change":"2024-03-19T11:31:06.103482+0000","last_active":"2024-03-19T11:31:07.098810+0000","last_peered":"2024-03-19T11:31:07.098810+0000","last_clean":"2024-03-19T11:31:07.098810+0000","last_became_active":"2024-03-19T11:30:53.084452+0000","last_became_peered":"2024-03-19T11:30:53.084452+0000","last_unstale":"2024-03-19T11:31:07.098810+0000","last_undegraded":"2024-03-19T11:31:07.098810+0000","last_fullsized":"2024-03-19T11:31:07.098810+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"64'62","last_scrub_stamp":"2024-03-19T11:30:42.778159+0000","last_deep_scrub":"64'62","last_deep_scrub_stamp":"2024-03-19T11:30:42.778159+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.778159+0000","objects_scrubbed":4,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:03.485738+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0.042323323000000003,"stat_sum":{"num_bytes":3794928,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":7565,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3779426,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.1d","version":"87'67","reported_seq":81,"reported_epoch":90,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:09.056383+0000","last_change":"2024-03-19T11:31:06.105861+0000","last_active":"2024-03-19T11:31:09.056383+0000","last_peered":"2024-03-19T11:31:09.056383+0000","last_clean":"2024-03-19T11:31:09.056383+0000","last_became_active":"2024-03-19T11:30:53.067748+0000","last_became_peered":"2024-03-19T11:30:53.067748+0000","last_unstale":"2024-03-19T11:31:09.056383+0000","last_undegraded":"2024-03-19T11:31:09.056383+0000","last_fullsized":"2024-03-19T11:31:09.056383+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"61'59","last_scrub_stamp":"2024-03-19T11:30:42.601863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.601863+0000","objects_scrubbed":2,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:45.377695+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.039756313000000001,"stat_sum":{"num_bytes":4835736,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":3,"num_read_kb":61,"num_write":5,"num_write_kb":1341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2817390,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"6","length":"1"},{"start":"11","length":"1"}]},{"pgid":"4.1c","version":"85'35","reported_seq":98,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T11:31:09.057786+0000","last_change":"2024-03-19T11:31:08.052486+0000","last_active":"2024-03-19T11:31:09.057786+0000","last_peered":"2024-03-19T11:31:09.057786+0000","last_clean":"2024-03-19T11:31:09.057786+0000","last_became_active":"2024-03-19T11:30:53.082655+0000","last_became_peered":"2024-03-19T11:30:53.082655+0000","last_unstale":"2024-03-19T11:31:09.057786+0000","last_undegraded":"2024-03-19T11:31:09.057786+0000","last_fullsized":"2024-03-19T11:31:09.057786+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"62'31","last_scrub_stamp":"2024-03-19T11:30:42.771396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.771396+0000","objects_scrubbed":3,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:53.311410+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0013764719999999999,"stat_sum":{"num_bytes":3210104,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":47464,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":5034309,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.1b","version":"89'100","reported_seq":92,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.105337+0000","last_change":"2024-03-19T11:31:08.108313+0000","last_active":"2024-03-19T11:31:08.105337+0000","last_peered":"2024-03-19T11:31:08.105337+0000","last_clean":"2024-03-19T11:31:08.105337+0000","last_became_active":"2024-03-19T11:31:03.056769+0000","last_became_peered":"2024-03-19T11:31:03.056769+0000","last_unstale":"2024-03-19T11:31:08.105337+0000","last_undegraded":"2024-03-19T11:31:08.105337+0000","last_fullsized":"2024-03-19T11:31:08.105337+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":5,"last_scrub":"62'62","last_scrub_stamp":"2024-03-19T11:30:42.181424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.181424+0000","objects_scrubbed":5,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:53.941736+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.040344087000000001,"stat_sum":{"num_bytes":14818653,"num_objects":7,"num_object_clones":4,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":58,"num_read_kb":68803,"num_write":11,"num_write_kb":3105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":17966556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":6,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.1a","version":"89'111","reported_seq":83,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.082577+0000","last_change":"2024-03-19T11:31:08.083422+0000","last_active":"2024-03-19T11:31:08.082577+0000","last_peered":"2024-03-19T11:31:08.082577+0000","last_clean":"2024-03-19T11:31:08.082577+0000","last_became_active":"2024-03-19T11:31:03.056520+0000","last_became_peered":"2024-03-19T11:31:03.056520+0000","last_unstale":"2024-03-19T11:31:08.082577+0000","last_undegraded":"2024-03-19T11:31:08.082577+0000","last_fullsized":"2024-03-19T11:31:08.082577+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":5,"last_scrub":"61'76","last_scrub_stamp":"2024-03-19T11:30:41.433501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:41.433501+0000","objects_scrubbed":5,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:46.519318+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.021991492000000001,"stat_sum":{"num_bytes":11953834,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":22,"num_read_kb":17414,"num_write":18,"num_write_kb":5314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":16291222,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.19","version":"89'117","reported_seq":496,"reported_epoch":90,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:09.056559+0000","last_change":"2024-03-19T11:31:08.110132+0000","last_active":"2024-03-19T11:31:09.056559+0000","last_peered":"2024-03-19T11:31:09.056559+0000","last_clean":"2024-03-19T11:31:09.056559+0000","last_became_active":"2024-03-19T11:30:53.076188+0000","last_became_peered":"2024-03-19T11:30:53.076188+0000","last_unstale":"2024-03-19T11:31:09.056559+0000","last_undegraded":"2024-03-19T11:31:09.056559+0000","last_fullsized":"2024-03-19T11:31:09.056559+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"41'37","last_scrub_stamp":"2024-03-19T11:30:27.705159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:27.705159+0000","objects_scrubbed":4,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:28.833088+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.047949758000000002,"stat_sum":{"num_bytes":18546999,"num_objects":13,"num_object_clones":9,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":99,"num_read_kb":113713,"num_write":39,"num_write_kb":6633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":21870393,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":11,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"4.18","version":"88'138","reported_seq":440,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.074905+0000","last_change":"2024-03-19T11:31:08.074905+0000","last_active":"2024-03-19T11:31:08.074905+0000","last_peered":"2024-03-19T11:31:08.074905+0000","last_clean":"2024-03-19T11:31:08.074905+0000","last_became_active":"2024-03-19T11:31:06.066324+0000","last_became_peered":"2024-03-19T11:31:06.066324+0000","last_unstale":"2024-03-19T11:31:08.074905+0000","last_undegraded":"2024-03-19T11:31:08.074905+0000","last_fullsized":"2024-03-19T11:31:08.074905+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":5,"last_scrub":"48'77","last_scrub_stamp":"2024-03-19T11:30:29.399762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:29.399762+0000","objects_scrubbed":6,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:55.801600+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.014795552,"stat_sum":{"num_bytes":18257745,"num_objects":7,"num_object_clones":5,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":49,"num_read_kb":49877,"num_write":22,"num_write_kb":6328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":19971477,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.17","version":"41'24","reported_seq":74,"reported_epoch":90,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:09.057390+0000","last_change":"2024-03-19T11:31:06.106323+0000","last_active":"2024-03-19T11:31:09.057390+0000","last_peered":"2024-03-19T11:31:09.057390+0000","last_clean":"2024-03-19T11:31:09.057390+0000","last_became_active":"2024-03-19T11:30:53.074270+0000","last_became_peered":"2024-03-19T11:30:53.074270+0000","last_unstale":"2024-03-19T11:31:09.057390+0000","last_undegraded":"2024-03-19T11:31:09.057390+0000","last_fullsized":"2024-03-19T11:31:09.057390+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"41'24","last_scrub_stamp":"2024-03-19T11:30:53.993428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:53.993428+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:22.671214+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.039101238000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"6","length":"1"},{"start":"11","length":"1"}]},{"pgid":"4.16","version":"87'116","reported_seq":91,"reported_epoch":88,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:07.097994+0000","last_change":"2024-03-19T11:31:07.083681+0000","last_active":"2024-03-19T11:31:07.097994+0000","last_peered":"2024-03-19T11:31:07.097994+0000","last_clean":"2024-03-19T11:31:07.097994+0000","last_became_active":"2024-03-19T11:30:59.074063+0000","last_became_peered":"2024-03-19T11:30:59.074063+0000","last_unstale":"2024-03-19T11:31:07.097994+0000","last_undegraded":"2024-03-19T11:31:07.097994+0000","last_fullsized":"2024-03-19T11:31:07.097994+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":5,"last_scrub":"49'56","last_scrub_stamp":"2024-03-19T11:30:29.212766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:29.212766+0000","objects_scrubbed":5,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:38.049725+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":1.0187149390000001,"stat_sum":{"num_bytes":14577733,"num_objects":8,"num_object_clones":5,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":105,"num_read_kb":105119,"num_write":27,"num_write_kb":7671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":24165549,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":8,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"6","length":"1"},{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.15","version":"89'177","reported_seq":596,"reported_epoch":89,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:08.117471+0000","last_change":"2024-03-19T11:31:06.107463+0000","last_active":"2024-03-19T11:31:08.117471+0000","last_peered":"2024-03-19T11:31:08.117471+0000","last_clean":"2024-03-19T11:31:08.117471+0000","last_became_active":"2024-03-19T11:30:56.063100+0000","last_became_peered":"2024-03-19T11:30:56.063100+0000","last_unstale":"2024-03-19T11:31:08.117471+0000","last_undegraded":"2024-03-19T11:31:08.117471+0000","last_fullsized":"2024-03-19T11:31:08.117471+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":5,"last_scrub":"49'74","last_scrub_stamp":"2024-03-19T11:30:29.457542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:29.457542+0000","objects_scrubbed":7,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:42.862571+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.028232337999999999,"stat_sum":{"num_bytes":26876675,"num_objects":17,"num_object_clones":12,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":1,"num_read":142,"num_read_kb":156856,"num_write":52,"num_write_kb":12752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":18744415,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":11,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"d","length":"1"},{"start":"11","length":"1"}]},{"pgid":"4.14","version":"40'56","reported_seq":239,"reported_epoch":90,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:09.057353+0000","last_change":"2024-03-19T11:31:08.110146+0000","last_active":"2024-03-19T11:31:09.057353+0000","last_peered":"2024-03-19T11:31:09.057353+0000","last_clean":"2024-03-19T11:31:09.057353+0000","last_became_active":"2024-03-19T11:31:00.065648+0000","last_became_peered":"2024-03-19T11:31:00.065648+0000","last_unstale":"2024-03-19T11:31:09.057353+0000","last_undegraded":"2024-03-19T11:31:09.057353+0000","last_fullsized":"2024-03-19T11:31:09.057353+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":5,"last_scrub":"40'56","last_scrub_stamp":"2024-03-19T11:30:54.998282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:54.998282+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:12.541661+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.046980873999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"}]},{"pgid":"4.13","version":"89'38","reported_seq":140,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.143741+0000","last_change":"2024-03-19T11:31:08.107232+0000","last_active":"2024-03-19T11:31:08.143741+0000","last_peered":"2024-03-19T11:31:08.143741+0000","last_clean":"2024-03-19T11:31:08.143741+0000","last_became_active":"2024-03-19T11:30:45.245367+0000","last_became_peered":"2024-03-19T11:30:45.245367+0000","last_unstale":"2024-03-19T11:31:08.143741+0000","last_undegraded":"2024-03-19T11:31:08.143741+0000","last_fullsized":"2024-03-19T11:31:08.143741+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":5,"last_scrub":"48'22","last_scrub_stamp":"2024-03-19T11:30:27.183294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:27.183294+0000","objects_scrubbed":2,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:41.607013+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.039325774000000001,"stat_sum":{"num_bytes":9406330,"num_objects":4,"num_object_clones":2,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":35,"num_read_kb":30733,"num_write":15,"num_write_kb":4523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":3001789,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.12","version":"28'6","reported_seq":65,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.057864+0000","last_change":"2024-03-19T11:31:08.058401+0000","last_active":"2024-03-19T11:31:08.057864+0000","last_peered":"2024-03-19T11:31:08.057864+0000","last_clean":"2024-03-19T11:31:08.057864+0000","last_became_active":"2024-03-19T11:31:00.064380+0000","last_became_peered":"2024-03-19T11:31:00.064380+0000","last_unstale":"2024-03-19T11:31:08.057864+0000","last_undegraded":"2024-03-19T11:31:08.057864+0000","last_fullsized":"2024-03-19T11:31:08.057864+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":5,"last_scrub":"28'6","last_scrub_stamp":"2024-03-19T11:31:04.263428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:31:04.263428+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:20.192174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020212310000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.11","version":"88'84","reported_seq":201,"reported_epoch":90,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:09.056453+0000","last_change":"2024-03-19T11:31:08.112910+0000","last_active":"2024-03-19T11:31:09.056453+0000","last_peered":"2024-03-19T11:31:09.056453+0000","last_clean":"2024-03-19T11:31:09.056453+0000","last_became_active":"2024-03-19T11:30:30.052184+0000","last_became_peered":"2024-03-19T11:30:30.052184+0000","last_unstale":"2024-03-19T11:31:09.056453+0000","last_undegraded":"2024-03-19T11:31:09.056453+0000","last_fullsized":"2024-03-19T11:31:09.056453+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":5,"last_scrub":"47'60","last_scrub_stamp":"2024-03-19T11:30:30.780348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:30.780348+0000","objects_scrubbed":2,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:43.351932+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.051827466000000003,"stat_sum":{"num_bytes":8084512,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":25443,"num_write":14,"num_write_kb":4028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"6","length":"1"},{"start":"11","length":"1"}]},{"pgid":"4.10","version":"85'34","reported_seq":358,"reported_epoch":90,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:09.056610+0000","last_change":"2024-03-19T11:31:06.105203+0000","last_active":"2024-03-19T11:31:09.056610+0000","last_peered":"2024-03-19T11:31:09.056610+0000","last_clean":"2024-03-19T11:31:09.056610+0000","last_became_active":"2024-03-19T11:31:00.065617+0000","last_became_peered":"2024-03-19T11:31:00.065617+0000","last_unstale":"2024-03-19T11:31:09.056610+0000","last_undegraded":"2024-03-19T11:31:09.056610+0000","last_fullsized":"2024-03-19T11:31:09.056610+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":5,"last_scrub":"50'30","last_scrub_stamp":"2024-03-19T11:30:28.787058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:28.787058+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:43.836722+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.038831954000000002,"stat_sum":{"num_bytes":3462209,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":20772,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":7102596,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"6","length":"1"},{"start":"11","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":146,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T11:31:09.057749+0000","last_change":"2024-03-19T11:31:08.381703+0000","last_active":"2024-03-19T11:31:09.057749+0000","last_peered":"2024-03-19T11:31:09.057749+0000","last_clean":"2024-03-19T11:31:09.057749+0000","last_became_active":"2024-03-19T11:30:03.276220+0000","last_became_peered":"2024-03-19T11:30:03.276220+0000","last_unstale":"2024-03-19T11:31:09.057749+0000","last_undegraded":"2024-03-19T11:31:09.057749+0000","last_fullsized":"2024-03-19T11:31:09.057749+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:31:08.381559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:31:08.381559+0000","last_clean_scrub_stamp":"2024-03-19T11:31:08.381559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:30.325139+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"59'92","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.071050+0000","last_change":"2024-03-19T11:31:08.072335+0000","last_active":"2024-03-19T11:31:08.071050+0000","last_peered":"2024-03-19T11:31:08.071050+0000","last_clean":"2024-03-19T11:31:08.071050+0000","last_became_active":"2024-03-19T11:31:03.052657+0000","last_became_peered":"2024-03-19T11:31:03.052657+0000","last_unstale":"2024-03-19T11:31:08.071050+0000","last_undegraded":"2024-03-19T11:31:08.071050+0000","last_fullsized":"2024-03-19T11:31:08.071050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'92","last_scrub_stamp":"2024-03-19T11:30:41.351520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:41.351520+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:51.023342+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.012111200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.057268+0000","last_change":"2024-03-19T11:31:00.067375+0000","last_active":"2024-03-19T11:31:08.057268+0000","last_peered":"2024-03-19T11:31:08.057268+0000","last_clean":"2024-03-19T11:31:08.057268+0000","last_became_active":"2024-03-19T11:31:00.067099+0000","last_became_peered":"2024-03-19T11:31:00.067099+0000","last_unstale":"2024-03-19T11:31:08.057268+0000","last_undegraded":"2024-03-19T11:31:08.057268+0000","last_fullsized":"2024-03-19T11:31:08.057268+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"87'71","reported_seq":373,"reported_epoch":90,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:09.057009+0000","last_change":"2024-03-19T11:31:08.109367+0000","last_active":"2024-03-19T11:31:09.057009+0000","last_peered":"2024-03-19T11:31:09.057009+0000","last_clean":"2024-03-19T11:31:09.057009+0000","last_became_active":"2024-03-19T11:30:38.065693+0000","last_became_peered":"2024-03-19T11:30:38.065693+0000","last_unstale":"2024-03-19T11:31:09.057009+0000","last_undegraded":"2024-03-19T11:31:09.057009+0000","last_fullsized":"2024-03-19T11:31:09.057009+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'57","last_scrub_stamp":"2024-03-19T11:30:43.792326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:43.792326+0000","objects_scrubbed":6,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:59.305622+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.048139135999999999,"stat_sum":{"num_bytes":11494971,"num_objects":5,"num_object_clones":4,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":13,"num_read_kb":16662,"num_write":6,"num_write_kb":1520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"6","length":"1"},{"start":"11","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":170,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T11:31:09.137317+0000","last_change":"2024-03-19T11:30:59.072585+0000","last_active":"2024-03-19T11:31:09.137317+0000","last_peered":"2024-03-19T11:31:09.137317+0000","last_clean":"2024-03-19T11:31:09.137317+0000","last_became_active":"2024-03-19T11:30:59.071925+0000","last_became_peered":"2024-03-19T11:30:59.071925+0000","last_unstale":"2024-03-19T11:31:09.137317+0000","last_undegraded":"2024-03-19T11:31:09.137317+0000","last_fullsized":"2024-03-19T11:31:09.137317+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"41'19","reported_seq":317,"reported_epoch":88,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:07.098065+0000","last_change":"2024-03-19T11:31:07.081723+0000","last_active":"2024-03-19T11:31:07.098065+0000","last_peered":"2024-03-19T11:31:07.098065+0000","last_clean":"2024-03-19T11:31:07.098065+0000","last_became_active":"2024-03-19T11:30:38.058860+0000","last_became_peered":"2024-03-19T11:30:38.058860+0000","last_unstale":"2024-03-19T11:31:07.098065+0000","last_undegraded":"2024-03-19T11:31:07.098065+0000","last_fullsized":"2024-03-19T11:31:07.098065+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'19","last_scrub_stamp":"2024-03-19T11:30:28.180772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:28.180772+0000","objects_scrubbed":1,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":1.0172436659999999,"stat_sum":{"num_bytes":1403368,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4077,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"6","length":"1"},{"start":"c","length":"2"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":161,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.058905+0000","last_change":"2024-03-19T11:30:55.366872+0000","last_active":"2024-03-19T11:31:08.058905+0000","last_peered":"2024-03-19T11:31:08.058905+0000","last_clean":"2024-03-19T11:31:08.058905+0000","last_became_active":"2024-03-19T11:29:46.540939+0000","last_became_peered":"2024-03-19T11:29:46.540939+0000","last_unstale":"2024-03-19T11:31:08.058905+0000","last_undegraded":"2024-03-19T11:31:08.058905+0000","last_fullsized":"2024-03-19T11:31:08.058905+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:30:55.366631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:30:55.366631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:10.640223+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"87'19","reported_seq":87,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.154738+0000","last_change":"2024-03-19T11:31:08.063352+0000","last_active":"2024-03-19T11:31:08.154738+0000","last_peered":"2024-03-19T11:31:08.154738+0000","last_clean":"2024-03-19T11:31:08.154738+0000","last_became_active":"2024-03-19T11:31:00.069200+0000","last_became_peered":"2024-03-19T11:31:00.069200+0000","last_unstale":"2024-03-19T11:31:08.154738+0000","last_undegraded":"2024-03-19T11:31:08.154738+0000","last_fullsized":"2024-03-19T11:31:08.154738+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-03-19T11:31:05.279955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:31:05.279955+0000","objects_scrubbed":2,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:06.253899+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0066655569999999999,"stat_sum":{"num_bytes":3918899,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":12792,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"3.3","version":"87'107","reported_seq":357,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.067397+0000","last_change":"2024-03-19T11:31:06.067504+0000","last_active":"2024-03-19T11:31:08.067397+0000","last_peered":"2024-03-19T11:31:08.067397+0000","last_clean":"2024-03-19T11:31:08.067397+0000","last_became_active":"2024-03-19T11:31:06.066161+0000","last_became_peered":"2024-03-19T11:31:06.066161+0000","last_unstale":"2024-03-19T11:31:08.067397+0000","last_undegraded":"2024-03-19T11:31:08.067397+0000","last_fullsized":"2024-03-19T11:31:08.067397+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:10.489969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47923414,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":80,"num_read_kb":3742,"num_write":115,"num_write_kb":59833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":25111105,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":152,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.047165+0000","last_change":"2024-03-19T11:29:46.321255+0000","last_active":"2024-03-19T11:31:08.047165+0000","last_peered":"2024-03-19T11:31:08.047165+0000","last_clean":"2024-03-19T11:31:08.047165+0000","last_became_active":"2024-03-19T11:29:46.320679+0000","last_became_peered":"2024-03-19T11:29:46.320679+0000","last_unstale":"2024-03-19T11:31:08.047165+0000","last_undegraded":"2024-03-19T11:31:08.047165+0000","last_fullsized":"2024-03-19T11:31:08.047165+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:14.793563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"87'136","reported_seq":549,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.071940+0000","last_change":"2024-03-19T11:31:08.071940+0000","last_active":"2024-03-19T11:31:08.071940+0000","last_peered":"2024-03-19T11:31:08.071940+0000","last_clean":"2024-03-19T11:31:08.071940+0000","last_became_active":"2024-03-19T11:31:03.054624+0000","last_became_peered":"2024-03-19T11:31:03.054624+0000","last_unstale":"2024-03-19T11:31:08.071940+0000","last_undegraded":"2024-03-19T11:31:08.071940+0000","last_fullsized":"2024-03-19T11:31:08.071940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'119","last_scrub_stamp":"2024-03-19T11:30:52.512203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:52.512203+0000","objects_scrubbed":7,"log_size":136,"log_dups_size":0,"ondisk_log_size":136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:17.035697+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.013370745999999999,"stat_sum":{"num_bytes":12400028,"num_objects":5,"num_object_clones":2,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":37,"num_read_kb":46256,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"3.0","version":"89'104","reported_seq":261,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.058221+0000","last_change":"2024-03-19T11:29:57.550320+0000","last_active":"2024-03-19T11:31:08.058221+0000","last_peered":"2024-03-19T11:31:08.058221+0000","last_clean":"2024-03-19T11:31:08.058221+0000","last_became_active":"2024-03-19T11:29:57.549928+0000","last_became_peered":"2024-03-19T11:29:57.549928+0000","last_unstale":"2024-03-19T11:31:08.058221+0000","last_undegraded":"2024-03-19T11:31:08.058221+0000","last_fullsized":"2024-03-19T11:31:08.058221+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32146410,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":82,"num_read_kb":3252,"num_write":109,"num_write_kb":43633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":156,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T11:31:09.057613+0000","last_change":"2024-03-19T11:29:46.319811+0000","last_active":"2024-03-19T11:31:09.057613+0000","last_peered":"2024-03-19T11:31:09.057613+0000","last_clean":"2024-03-19T11:31:09.057613+0000","last_became_active":"2024-03-19T11:29:46.319334+0000","last_became_peered":"2024-03-19T11:29:46.319334+0000","last_unstale":"2024-03-19T11:31:09.057613+0000","last_undegraded":"2024-03-19T11:31:09.057613+0000","last_fullsized":"2024-03-19T11:31:09.057613+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"61'40","reported_seq":57,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.058444+0000","last_change":"2024-03-19T11:31:08.059007+0000","last_active":"2024-03-19T11:31:08.058444+0000","last_peered":"2024-03-19T11:31:08.058444+0000","last_clean":"2024-03-19T11:31:08.058444+0000","last_became_active":"2024-03-19T11:31:00.069809+0000","last_became_peered":"2024-03-19T11:31:00.069809+0000","last_unstale":"2024-03-19T11:31:08.058444+0000","last_undegraded":"2024-03-19T11:31:08.058444+0000","last_fullsized":"2024-03-19T11:31:08.058444+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'40","last_scrub_stamp":"2024-03-19T11:30:56.505586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:56.505586+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:56.842849+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.002526866,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"3.1","version":"88'90","reported_seq":382,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-19T11:31:07.160859+0000","last_change":"2024-03-19T11:31:03.255027+0000","last_active":"2024-03-19T11:31:07.160859+0000","last_peered":"2024-03-19T11:31:07.160859+0000","last_clean":"2024-03-19T11:31:07.160859+0000","last_became_active":"2024-03-19T11:31:03.077272+0000","last_became_peered":"2024-03-19T11:31:03.077272+0000","last_unstale":"2024-03-19T11:31:07.160859+0000","last_undegraded":"2024-03-19T11:31:07.160859+0000","last_fullsized":"2024-03-19T11:31:07.160859+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:23.954117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44609286,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":74,"num_read_kb":6654,"num_write":94,"num_write_kb":53294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":42087532,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":165,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T11:31:09.137114+0000","last_change":"2024-03-19T11:31:00.070027+0000","last_active":"2024-03-19T11:31:09.137114+0000","last_peered":"2024-03-19T11:31:09.137114+0000","last_clean":"2024-03-19T11:31:09.137114+0000","last_became_active":"2024-03-19T11:31:00.068989+0000","last_became_peered":"2024-03-19T11:31:00.068989+0000","last_unstale":"2024-03-19T11:31:09.137114+0000","last_undegraded":"2024-03-19T11:31:09.137114+0000","last_fullsized":"2024-03-19T11:31:09.137114+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:11.599600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"85'70","reported_seq":70,"reported_epoch":88,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:07.116943+0000","last_change":"2024-03-19T11:31:07.098353+0000","last_active":"2024-03-19T11:31:07.116943+0000","last_peered":"2024-03-19T11:31:07.116943+0000","last_clean":"2024-03-19T11:31:07.116943+0000","last_became_active":"2024-03-19T11:30:59.075057+0000","last_became_peered":"2024-03-19T11:30:59.075057+0000","last_unstale":"2024-03-19T11:31:07.116943+0000","last_undegraded":"2024-03-19T11:31:07.116943+0000","last_fullsized":"2024-03-19T11:31:07.116943+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'65","last_scrub_stamp":"2024-03-19T11:30:53.385414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:53.385414+0000","objects_scrubbed":3,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:20.971419+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":1.0213928809999999,"stat_sum":{"num_bytes":6521083,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":31459,"num_write":4,"num_write_kb":1430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4209633,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"6","length":"1"},{"start":"c","length":"2"},{"start":"11","length":"2"},{"start":"14","length":"1"}]},{"pgid":"3.2","version":"86'88","reported_seq":253,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T11:31:09.056695+0000","last_change":"2024-03-19T11:29:57.549277+0000","last_active":"2024-03-19T11:31:09.056695+0000","last_peered":"2024-03-19T11:31:09.056695+0000","last_clean":"2024-03-19T11:31:09.056695+0000","last_became_active":"2024-03-19T11:29:57.548946+0000","last_became_peered":"2024-03-19T11:29:57.548946+0000","last_unstale":"2024-03-19T11:31:09.056695+0000","last_undegraded":"2024-03-19T11:31:09.056695+0000","last_fullsized":"2024-03-19T11:31:09.056695+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42459028,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":78,"num_read_kb":8248,"num_write":98,"num_write_kb":44121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":160,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.047494+0000","last_change":"2024-03-19T11:31:04.588296+0000","last_active":"2024-03-19T11:31:08.047494+0000","last_peered":"2024-03-19T11:31:08.047494+0000","last_clean":"2024-03-19T11:31:08.047494+0000","last_became_active":"2024-03-19T11:29:46.321760+0000","last_became_peered":"2024-03-19T11:29:46.321760+0000","last_unstale":"2024-03-19T11:31:08.047494+0000","last_undegraded":"2024-03-19T11:31:08.047494+0000","last_fullsized":"2024-03-19T11:31:08.047494+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:31:04.588071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:31:04.588071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:11.976969+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":339,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T11:31:09.137294+0000","last_change":"2024-03-19T11:31:03.367805+0000","last_active":"2024-03-19T11:31:09.137294+0000","last_peered":"2024-03-19T11:31:09.137294+0000","last_clean":"2024-03-19T11:31:09.137294+0000","last_became_active":"2024-03-19T11:30:59.073709+0000","last_became_peered":"2024-03-19T11:30:59.073709+0000","last_unstale":"2024-03-19T11:31:09.137294+0000","last_undegraded":"2024-03-19T11:31:09.137294+0000","last_fullsized":"2024-03-19T11:31:09.137294+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2024-03-19T11:31:03.367669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:43.166618+0000","last_clean_scrub_stamp":"2024-03-19T11:31:03.367669+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:29.977573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"88'111","reported_seq":515,"reported_epoch":89,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:08.083914+0000","last_change":"2024-03-19T11:31:07.189608+0000","last_active":"2024-03-19T11:31:08.083914+0000","last_peered":"2024-03-19T11:31:08.083914+0000","last_clean":"2024-03-19T11:31:08.083914+0000","last_became_active":"2024-03-19T11:31:03.065344+0000","last_became_peered":"2024-03-19T11:31:03.065344+0000","last_unstale":"2024-03-19T11:31:08.083914+0000","last_undegraded":"2024-03-19T11:31:08.083914+0000","last_fullsized":"2024-03-19T11:31:08.083914+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'81","last_scrub_stamp":"2024-03-19T11:30:52.360396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:52.360396+0000","objects_scrubbed":6,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:14.659997+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.103481396,"stat_sum":{"num_bytes":10838384,"num_objects":7,"num_object_clones":5,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":13,"num_read_kb":12349,"num_write":8,"num_write_kb":1273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":10558635,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"1"}]},{"pgid":"4.8","version":"62'71","reported_seq":380,"reported_epoch":89,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:08.113863+0000","last_change":"2024-03-19T11:31:08.113959+0000","last_active":"2024-03-19T11:31:08.113863+0000","last_peered":"2024-03-19T11:31:08.113863+0000","last_clean":"2024-03-19T11:31:08.113863+0000","last_became_active":"2024-03-19T11:30:45.251283+0000","last_became_peered":"2024-03-19T11:30:45.251283+0000","last_unstale":"2024-03-19T11:31:08.113863+0000","last_undegraded":"2024-03-19T11:31:08.113863+0000","last_fullsized":"2024-03-19T11:31:08.113863+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'71","last_scrub_stamp":"2024-03-19T11:30:53.381386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:53.381386+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:14.007435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.066645368999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"}]},{"pgid":"4.9","version":"63'55","reported_seq":415,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.075645+0000","last_change":"2024-03-19T11:31:08.076672+0000","last_active":"2024-03-19T11:31:08.075645+0000","last_peered":"2024-03-19T11:31:08.075645+0000","last_clean":"2024-03-19T11:31:08.075645+0000","last_became_active":"2024-03-19T11:31:00.069789+0000","last_became_peered":"2024-03-19T11:31:00.069789+0000","last_unstale":"2024-03-19T11:31:08.075645+0000","last_undegraded":"2024-03-19T11:31:08.075645+0000","last_fullsized":"2024-03-19T11:31:08.075645+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'55","last_scrub_stamp":"2024-03-19T11:30:54.706235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:54.706235+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:10.998788+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.015421028,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.a","version":"62'77","reported_seq":406,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.117764+0000","last_change":"2024-03-19T11:31:08.118531+0000","last_active":"2024-03-19T11:31:08.117764+0000","last_peered":"2024-03-19T11:31:08.117764+0000","last_clean":"2024-03-19T11:31:08.117764+0000","last_became_active":"2024-03-19T11:30:45.256124+0000","last_became_peered":"2024-03-19T11:30:45.256124+0000","last_unstale":"2024-03-19T11:31:08.117764+0000","last_undegraded":"2024-03-19T11:31:08.117764+0000","last_fullsized":"2024-03-19T11:31:08.117764+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'77","last_scrub_stamp":"2024-03-19T11:30:55.291823+0000","last_deep_scrub":"62'77","last_deep_scrub_stamp":"2024-03-19T11:30:55.291823+0000","last_clean_scrub_stamp":"2024-03-19T11:30:55.291823+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:58.611283+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.033194802000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.b","version":"85'64","reported_seq":394,"reported_epoch":87,"state":"peering","last_fresh":"2024-03-19T11:31:06.074443+0000","last_change":"2024-03-19T11:31:07.098974+0000","last_active":"2024-03-19T11:31:05.071268+0000","last_peered":"2024-03-19T11:31:06.074443+0000","last_clean":"2024-03-19T11:31:06.074443+0000","last_became_active":"2024-03-19T11:30:45.261756+0000","last_became_peered":"2024-03-19T11:31:06.072241+0000","last_unstale":"2024-03-19T11:31:06.074443+0000","last_undegraded":"2024-03-19T11:31:06.074443+0000","last_fullsized":"2024-03-19T11:31:06.074443+0000","mapping_epoch":21,"log_start":"85'64","ondisk_log_start":"85'64","created":21,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'62","last_scrub_stamp":"2024-03-19T11:30:56.095039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:56.095039+0000","objects_scrubbed":3,"log_size":0,"log_dups_size":30,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063617100000000005,"stat_sum":{"num_bytes":5215740,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"c","length":"1"},{"start":"11","length":"1"},{"start":"14","length":"1"}]},{"pgid":"4.c","version":"56'29","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.072859+0000","last_change":"2024-03-19T11:31:08.074168+0000","last_active":"2024-03-19T11:31:08.072859+0000","last_peered":"2024-03-19T11:31:08.072859+0000","last_clean":"2024-03-19T11:31:08.072859+0000","last_became_active":"2024-03-19T11:31:05.073655+0000","last_became_peered":"2024-03-19T11:31:05.073655+0000","last_unstale":"2024-03-19T11:31:08.072859+0000","last_undegraded":"2024-03-19T11:31:08.072859+0000","last_fullsized":"2024-03-19T11:31:08.072859+0000","mapping_epoch":83,"log_start":"56'29","ondisk_log_start":"56'29","created":21,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'29","last_scrub_stamp":"2024-03-19T11:30:53.743385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:53.743385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:56.702924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013861157000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.d","version":"87'88","reported_seq":434,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.086044+0000","last_change":"2024-03-19T11:31:08.086044+0000","last_active":"2024-03-19T11:31:08.086044+0000","last_peered":"2024-03-19T11:31:08.086044+0000","last_clean":"2024-03-19T11:31:08.086044+0000","last_became_active":"2024-03-19T11:31:06.068526+0000","last_became_peered":"2024-03-19T11:31:06.068526+0000","last_unstale":"2024-03-19T11:31:08.086044+0000","last_undegraded":"2024-03-19T11:31:08.086044+0000","last_fullsized":"2024-03-19T11:31:08.086044+0000","mapping_epoch":86,"log_start":"70'71","ondisk_log_start":"70'71","created":21,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'73","last_scrub_stamp":"2024-03-19T11:30:53.519178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:53.519178+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":50,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:02.045989+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.021539165999999998,"stat_sum":{"num_bytes":6903494,"num_objects":4,"num_object_clones":2,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":18,"num_read_kb":22670,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":8146998,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.e","version":"89'94","reported_seq":402,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.081152+0000","last_change":"2024-03-19T11:31:08.081889+0000","last_active":"2024-03-19T11:31:08.081152+0000","last_peered":"2024-03-19T11:31:08.081152+0000","last_clean":"2024-03-19T11:31:08.081152+0000","last_became_active":"2024-03-19T11:30:38.060487+0000","last_became_peered":"2024-03-19T11:30:38.060487+0000","last_unstale":"2024-03-19T11:31:08.081152+0000","last_undegraded":"2024-03-19T11:31:08.081152+0000","last_fullsized":"2024-03-19T11:31:08.081152+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'65","last_scrub_stamp":"2024-03-19T11:30:43.824620+0000","last_deep_scrub":"65'65","last_deep_scrub_stamp":"2024-03-19T11:30:43.824620+0000","last_clean_scrub_stamp":"2024-03-19T11:30:43.824620+0000","objects_scrubbed":4,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:51.683705+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0.020120889999999999,"stat_sum":{"num_bytes":10000429,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":37,"num_read_kb":41805,"num_write":10,"num_write_kb":2301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.f","version":"0'0","reported_seq":280,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.063623+0000","last_change":"2024-03-19T11:31:08.065011+0000","last_active":"2024-03-19T11:31:08.063623+0000","last_peered":"2024-03-19T11:31:08.063623+0000","last_clean":"2024-03-19T11:31:08.063623+0000","last_became_active":"2024-03-19T11:30:38.050919+0000","last_became_peered":"2024-03-19T11:30:38.050919+0000","last_unstale":"2024-03-19T11:31:08.063623+0000","last_undegraded":"2024-03-19T11:31:08.063623+0000","last_fullsized":"2024-03-19T11:31:08.063623+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:30:43.411893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:43.411893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:04.626662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0068115019999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]}],"pool_stats":[{"poolid":4,"num_pg":43,"stat_sum":{"num_bytes":268453009,"num_objects":133,"num_object_clones":83,"num_object_copies":266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":8,"num_read":865,"num_read_kb":964725,"num_write":269,"num_write_kb":68980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":202389426,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":88,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":365174784,"data_stored":422419430,"data_compressed":2584282,"data_compressed_allocated":1728512,"data_compressed_original":6094848,"omap_allocated":0,"internal_metadata":0},"log_size":2839,"ondisk_log_size":2839,"up":86,"acting":86,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":167138138,"num_objects":149,"num_object_clones":0,"num_object_copies":298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":149,"num_whiteouts":0,"num_read":314,"num_read_kb":21896,"num_write":416,"num_write_kb":200881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":67198637,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210571264,"data_stored":227781731,"data_compressed":15765533,"data_compressed_allocated":16859136,"data_compressed_original":34500608,"omap_allocated":0,"internal_metadata":0},"log_size":389,"ondisk_log_size":389,"up":8,"acting":8,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":1586784,"data_compressed":19832,"data_compressed_allocated":98304,"data_compressed_original":1560576,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542165,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84272,"kb_used_data":52384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94287568,"statfs":{"total":96636764160,"available":96550469632,"internally_reserved":0,"allocated":53641216,"data_stored":62766152,"data_compressed":154066,"data_compressed_allocated":151552,"data_compressed_original":970752,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542165,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82652,"kb_used_data":51472,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94289188,"statfs":{"total":96636764160,"available":96552128512,"internally_reserved":0,"allocated":52707328,"data_stored":61412079,"data_compressed":1859610,"data_compressed_allocated":1949696,"data_compressed_original":4210688,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542165,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":128184,"kb_used_data":96296,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94243656,"statfs":{"total":96636764160,"available":96505503744,"internally_reserved":0,"allocated":98607104,"data_stored":114667063,"data_compressed":2437448,"data_compressed_allocated":2236416,"data_compressed_original":5586944,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542164,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72524,"kb_used_data":45504,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94299316,"statfs":{"total":96636764160,"available":96562499584,"internally_reserved":0,"allocated":46596096,"data_stored":50973803,"data_compressed":1821717,"data_compressed_allocated":1605632,"data_compressed_original":4390912,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542164,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":113008,"kb_used_data":77664,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94258832,"statfs":{"total":96636764160,"available":96521043968,"internally_reserved":0,"allocated":79527936,"data_stored":82008696,"data_compressed":1366622,"data_compressed_allocated":1470464,"data_compressed_original":3276800,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574868,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64032,"kb_used_data":32844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94307808,"statfs":{"total":96636764160,"available":96571195392,"internally_reserved":0,"allocated":33632256,"data_stored":43833336,"data_compressed":169512,"data_compressed_allocated":57344,"data_compressed_original":196608,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574868,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":137620,"kb_used_data":106440,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94234220,"statfs":{"total":96636764160,"available":96495841280,"internally_reserved":0,"allocated":108994560,"data_stored":119204649,"data_compressed":4084412,"data_compressed_allocated":4292608,"data_compressed_original":8638464,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574868,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":100364,"kb_used_data":69184,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94271476,"statfs":{"total":96636764160,"available":96533991424,"internally_reserved":0,"allocated":70844416,"data_stored":76668922,"data_compressed":4021981,"data_compressed_allocated":4272128,"data_compressed_original":9228288,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6884,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6884,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6064,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":52113408,"data_stored":56190278,"data_compressed":3933827,"data_compressed_allocated":4194304,"data_compressed_original":8376320,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16039936,"data_stored":17517630,"data_compressed":1089583,"data_compressed_allocated":1175552,"data_compressed_original":2686976,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16039936,"data_stored":17517630,"data_compressed":1089583,"data_compressed_allocated":1175552,"data_compressed_original":2686976,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":22233088,"data_stored":24117451,"data_compressed":1711608,"data_compressed_allocated":1830912,"data_compressed_original":3751936,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":51625984,"data_stored":55733062,"data_compressed":3965051,"data_compressed_allocated":4227072,"data_compressed_original":8441856,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":22233088,"data_stored":24117451,"data_compressed":1711608,"data_compressed_allocated":1830912,"data_compressed_original":3751936,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":30285824,"data_stored":32588229,"data_compressed":2264273,"data_compressed_allocated":2424832,"data_compressed_original":4804608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":19746816,"data_stored":21578762,"data_compressed":233168,"data_compressed_allocated":204800,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":62513152,"data_stored":63972890,"data_compressed":277039,"data_compressed_allocated":294912,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":29593600,"data_stored":32940595,"data_compressed":732134,"data_compressed_allocated":430080,"data_compressed_original":1703936,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":75440128,"data_stored":90038134,"data_compressed":725840,"data_compressed_allocated":405504,"data_compressed_original":1835008,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":59510784,"data_stored":67285511,"data_compressed":150585,"data_compressed_allocated":98304,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":29507584,"data_stored":36778653,"data_compressed":148002,"data_compressed_allocated":118784,"data_compressed_original":458752,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36229120,"data_stored":48101703,"data_compressed":169512,"data_compressed_allocated":57344,"data_compressed_original":196608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":52633600,"data_stored":61723182,"data_compressed":148002,"data_compressed_allocated":118784,"data_compressed_original":458752,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T11:31:10.669 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:10.673+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:10.710 INFO:tasks.rados.rados.0.smithi038.stdout:1284: finishing write tid 1 to smithi03829856-8 2024-03-19T11:31:10.710 INFO:tasks.rados.rados.0.smithi038.stdout:1284: finishing write tid 2 to smithi03829856-8 2024-03-19T11:31:10.710 INFO:tasks.rados.rados.0.smithi038.stdout:1284: finishing write tid 3 to smithi03829856-8 2024-03-19T11:31:10.710 INFO:tasks.rados.rados.0.smithi038.stdout:1284: finishing write tid 4 to smithi03829856-8 2024-03-19T11:31:10.710 INFO:tasks.rados.rados.0.smithi038.stdout:1284: finishing write tid 5 to smithi03829856-8 2024-03-19T11:31:10.711 INFO:tasks.rados.rados.0.smithi038.stdout:1284: oid 8 updating version 0 to 191 2024-03-19T11:31:10.711 INFO:tasks.rados.rados.0.smithi038.stdout:1284: oid 8 version 191 is already newer than 189 2024-03-19T11:31:10.711 INFO:tasks.rados.rados.0.smithi038.stdout:1284: oid 8 version 191 is already newer than 190 2024-03-19T11:31:10.711 INFO:tasks.rados.rados.0.smithi038.stdout:1284: oid 8 version 191 is already newer than 188 2024-03-19T11:31:10.711 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 8 v 191 (ObjNum 215 snap 23 seq_num 215) dirty exists 2024-03-19T11:31:10.711 INFO:tasks.rados.rados.0.smithi038.stdout:1284: left oid 8 (ObjNum 215 snap 23 seq_num 215) 2024-03-19T11:31:10.711 INFO:tasks.rados.rados.0.smithi038.stdout:1285: finishing write tid 1 to smithi03829856-19 2024-03-19T11:31:10.711 INFO:tasks.rados.rados.0.smithi038.stdout:1285: finishing write tid 2 to smithi03829856-19 2024-03-19T11:31:10.711 INFO:tasks.rados.rados.0.smithi038.stdout:1280: done (6 left) 2024-03-19T11:31:10.711 INFO:tasks.rados.rados.0.smithi038.stdout:1281: done (5 left) 2024-03-19T11:31:10.711 INFO:tasks.rados.rados.0.smithi038.stdout:1282: done (4 left) 2024-03-19T11:31:10.711 INFO:tasks.rados.rados.0.smithi038.stdout:1283: done (3 left) 2024-03-19T11:31:10.711 INFO:tasks.rados.rados.0.smithi038.stdout:1284: done (2 left) 2024-03-19T11:31:10.711 INFO:tasks.rados.rados.0.smithi038.stdout:1287: read oid 8 snap -1 2024-03-19T11:31:10.711 INFO:tasks.rados.rados.0.smithi038.stdout:1287: expect (ObjNum 215 snap 23 seq_num 215) 2024-03-19T11:31:10.712 INFO:tasks.rados.rados.0.smithi038.stdout:1288: read oid 26 snap 19 2024-03-19T11:31:10.712 INFO:tasks.rados.rados.0.smithi038.stdout:1288: expect (ObjNum 123 snap 15 seq_num 123) 2024-03-19T11:31:10.712 INFO:tasks.rados.rados.0.smithi038.stdout:1289: read oid 10 snap -1 2024-03-19T11:31:10.712 INFO:tasks.rados.rados.0.smithi038.stdout:1289: expect (ObjNum 179 snap 19 seq_num 179) 2024-03-19T11:31:10.712 INFO:tasks.rados.rados.0.smithi038.stdout:1290: read oid 4 snap -1 2024-03-19T11:31:10.712 INFO:tasks.rados.rados.0.smithi038.stdout:1290: expect (ObjNum 151 snap 17 seq_num 151) 2024-03-19T11:31:10.712 INFO:tasks.rados.rados.0.smithi038.stdout:1291: write (excl) oid 35 current snap is 23 2024-03-19T11:31:10.712 INFO:tasks.rados.rados.0.smithi038.stdout:1285: finishing write tid 3 to smithi03829856-19 2024-03-19T11:31:10.712 INFO:tasks.rados.rados.0.smithi038.stdout:1285: finishing write tid 4 to smithi03829856-19 2024-03-19T11:31:10.712 INFO:tasks.rados.rados.0.smithi038.stdout:1285: finishing write tid 5 to smithi03829856-19 2024-03-19T11:31:10.712 INFO:tasks.rados.rados.0.smithi038.stdout:1285: oid 19 updating version 0 to 77 2024-03-19T11:31:10.712 INFO:tasks.rados.rados.0.smithi038.stdout:1285: oid 19 version 77 is already newer than 74 2024-03-19T11:31:10.712 INFO:tasks.rados.rados.0.smithi038.stdout:1285: oid 19 version 77 is already newer than 76 2024-03-19T11:31:10.712 INFO:tasks.rados.rados.0.smithi038.stdout:1285: oid 19 version 77 is already newer than 75 2024-03-19T11:31:10.713 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 19 v 77 (ObjNum 216 snap 23 seq_num 216) dirty exists 2024-03-19T11:31:10.713 INFO:tasks.rados.rados.0.smithi038.stdout:1285: left oid 19 (ObjNum 216 snap 23 seq_num 216) 2024-03-19T11:31:10.713 INFO:tasks.rados.rados.0.smithi038.stdout:1286: got (0) Success 2024-03-19T11:31:10.713 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 15 v 33 (ObjNum 209 snap 22 seq_num 209) dirty exists 2024-03-19T11:31:10.713 INFO:tasks.rados.rados.0.smithi038.stdout:1291: seq_num 217 ranges {447119=786917,1665416=692733,2976554=514615} 2024-03-19T11:31:10.720 INFO:tasks.rados.rados.0.smithi038.stdout:1291: writing smithi03829856-35 from 447119 to 1234036 tid 1 2024-03-19T11:31:10.728 INFO:tasks.rados.rados.0.smithi038.stdout:1291: writing smithi03829856-35 from 1665416 to 2358149 tid 2 2024-03-19T11:31:10.736 INFO:tasks.rados.rados.0.smithi038.stdout:1291: writing smithi03829856-35 from 2976554 to 3491169 tid 3 2024-03-19T11:31:10.737 INFO:tasks.rados.rados.0.smithi038.stdout:1285: done (6 left) 2024-03-19T11:31:10.737 INFO:tasks.rados.rados.0.smithi038.stdout:1286: done (5 left) 2024-03-19T11:31:10.737 INFO:tasks.rados.rados.0.smithi038.stdout:1292: read oid 23 snap -1 2024-03-19T11:31:10.737 INFO:tasks.rados.rados.0.smithi038.stdout:1292: expect (ObjNum 190 snap 22 seq_num 190) 2024-03-19T11:31:10.737 INFO:tasks.rados.rados.0.smithi038.stdout:1293: rollback oid 39 current snap is 23 2024-03-19T11:31:10.737 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 39 to 21 2024-03-19T11:31:10.737 INFO:tasks.rados.rados.0.smithi038.stdout:1289: expect (ObjNum 179 snap 19 seq_num 179) 2024-03-19T11:31:10.757 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:10.765+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:10.766 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:10.766 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:10.779 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-19T11:31:10.859 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:10.867+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:10.908 INFO:tasks.rados.rados.0.smithi038.stdout:1288: expect (ObjNum 123 snap 15 seq_num 123) 2024-03-19T11:31:10.911 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:10.911 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:10.925 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-19T11:31:10.972 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:10.983+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:11.001 INFO:tasks.rados.rados.0.smithi038.stdout:1290: expect (ObjNum 151 snap 17 seq_num 151) 2024-03-19T11:31:11.051 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:11.063+0000 7fb17dd72640 -1 osd.4 91 handle_osd_map injecting map crc failure 2024-03-19T11:31:11.061 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:11.069+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:11.102 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:11.102 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:11.116 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-19T11:31:11.118 INFO:tasks.rados.rados.0.smithi038.stdout:1287: expect (ObjNum 215 snap 23 seq_num 215) 2024-03-19T11:31:11.162 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:11.169+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:11.238 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:11.238 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:11.253 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-19T11:31:11.299 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:11.311+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:11.363 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:11.371+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:11.372 INFO:tasks.rados.rados.0.smithi038.stdout:1291: finishing write tid 1 to smithi03829856-35 2024-03-19T11:31:11.372 INFO:tasks.rados.rados.0.smithi038.stdout:1287: done (6 left) 2024-03-19T11:31:11.372 INFO:tasks.rados.rados.0.smithi038.stdout:1288: done (5 left) 2024-03-19T11:31:11.372 INFO:tasks.rados.rados.0.smithi038.stdout:1289: done (4 left) 2024-03-19T11:31:11.372 INFO:tasks.rados.rados.0.smithi038.stdout:1290: done (3 left) 2024-03-19T11:31:11.373 INFO:tasks.rados.rados.0.smithi038.stdout:1294: write (excl) oid 33 current snap is 23 2024-03-19T11:31:11.373 INFO:tasks.rados.rados.0.smithi038.stdout:1294: seq_num 218 ranges {463926=462166,1723898=436391,2603977=725887} 2024-03-19T11:31:11.376 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:11.376 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:11.387 INFO:tasks.rados.rados.0.smithi038.stdout:1294: writing smithi03829856-33 from 463926 to 926092 tid 1 2024-03-19T11:31:11.390 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-19T11:31:11.409 INFO:tasks.rados.rados.0.smithi038.stdout:1294: writing smithi03829856-33 from 1723898 to 2160289 tid 2 2024-03-19T11:31:11.428 INFO:tasks.rados.rados.0.smithi038.stdout:1294: writing smithi03829856-33 from 2603977 to 3329864 tid 3 2024-03-19T11:31:11.428 INFO:tasks.rados.rados.0.smithi038.stdout:1295: set_chunk oid 37 target oid 2024-03-19T11:31:11.428 INFO:tasks.rados.rados.0.smithi038.stdout:1295: set_chunk oid 37 offset: 0 length: 4000000 target oid smithi03829856- offset: 0 snap -1 2024-03-19T11:31:11.428 INFO:tasks.rados.rados.0.smithi038.stdout:1296: tier_evict oid 14 2024-03-19T11:31:11.428 INFO:tasks.rados.rados.0.smithi038.stdout:1296: tier_evict oid 14 snap -1 2024-03-19T11:31:11.428 INFO:tasks.rados.rados.0.smithi038.stdout:1291: finishing write tid 2 to smithi03829856-35 2024-03-19T11:31:11.428 INFO:tasks.rados.rados.0.smithi038.stdout:1291: finishing write tid 3 to smithi03829856-35 2024-03-19T11:31:11.428 INFO:tasks.rados.rados.0.smithi038.stdout:1293: finishing rollback tid 0 to smithi03829856-39 2024-03-19T11:31:11.428 INFO:tasks.rados.rados.0.smithi038.stdout:1291: finishing write tid 4 to smithi03829856-35 2024-03-19T11:31:11.429 INFO:tasks.rados.rados.0.smithi038.stdout:1297: read oid 31 snap -1 2024-03-19T11:31:11.429 INFO:tasks.rados.rados.0.smithi038.stdout:1297: expect (ObjNum 188 snap 21 seq_num 188) 2024-03-19T11:31:11.429 INFO:tasks.rados.rados.0.smithi038.stdout:1298: read oid 44 snap -1 2024-03-19T11:31:11.429 INFO:tasks.rados.rados.0.smithi038.stdout:1298: expect (ObjNum 108 snap 11 seq_num 108) 2024-03-19T11:31:11.429 INFO:tasks.rados.rados.0.smithi038.stdout:1299: read oid 41 snap -1 2024-03-19T11:31:11.429 INFO:tasks.rados.rados.0.smithi038.stdout:1299: expect (ObjNum 170 snap 17 seq_num 170) 2024-03-19T11:31:11.429 INFO:tasks.rados.rados.0.smithi038.stdout:1300: read oid 22 snap -1 2024-03-19T11:31:11.429 INFO:tasks.rados.rados.0.smithi038.stdout:1300: expect (ObjNum 168 snap 17 seq_num 168) 2024-03-19T11:31:11.429 INFO:tasks.rados.rados.0.smithi038.stdout:1301: set_chunk oid 13 target oid 2024-03-19T11:31:11.439 INFO:tasks.rados.rados.0.smithi038.stdout:1301: set_chunk oid 13 offset: 1056768 length: 82432 target oid 9b695228a66cd398b6c83eee911b801b53c4784b13d5e3ed6efb3f7f6352e247 offset: 0 snap -1 2024-03-19T11:31:11.440 INFO:tasks.rados.rados.0.smithi038.stdout:1302: set_chunk oid 20 target oid 2024-03-19T11:31:11.440 INFO:tasks.rados.rados.0.smithi038.stdout:1302: set_chunk oid 20 offset: 0 length: 4000000 target oid smithi03829856- offset: 0 snap -1 2024-03-19T11:31:11.440 INFO:tasks.rados.rados.0.smithi038.stdout:1303: write (excl) oid 29 current snap is 23 2024-03-19T11:31:11.440 INFO:tasks.rados.rados.0.smithi038.stdout:1293: finishing rollback tid 1 to smithi03829856-39 2024-03-19T11:31:11.440 INFO:tasks.rados.rados.0.smithi038.stdout:1293: finishing rollback tid 2 to smithi03829856-39 2024-03-19T11:31:11.440 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 39 v 98 (ObjNum 163 snap 17 seq_num 163) dirty exists 2024-03-19T11:31:11.440 INFO:tasks.rados.rados.0.smithi038.stdout:1292: expect (ObjNum 190 snap 22 seq_num 190) 2024-03-19T11:31:11.476 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:11.487+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:11.553 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:11.553 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:11.565 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:11.575+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:11.567 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-19T11:31:11.666 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:11.673+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:11.708 INFO:tasks.rados.rados.0.smithi038.stdout:1291: finishing write tid 5 to smithi03829856-35 2024-03-19T11:31:11.708 INFO:tasks.rados.rados.0.smithi038.stdout:1291: oid 35 updating version 0 to 193 2024-03-19T11:31:11.708 INFO:tasks.rados.rados.0.smithi038.stdout:1291: oid 35 updating version 193 to 194 2024-03-19T11:31:11.708 INFO:tasks.rados.rados.0.smithi038.stdout:1291: oid 35 updating version 194 to 195 2024-03-19T11:31:11.708 INFO:tasks.rados.rados.0.smithi038.stdout:1291: oid 35 updating version 195 to 196 2024-03-19T11:31:11.708 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 35 v 196 (ObjNum 217 snap 23 seq_num 217) dirty exists 2024-03-19T11:31:11.708 INFO:tasks.rados.rados.0.smithi038.stdout:1291: left oid 35 (ObjNum 217 snap 23 seq_num 217) 2024-03-19T11:31:11.708 INFO:tasks.rados.rados.0.smithi038.stdout:1294: finishing write tid 1 to smithi03829856-33 2024-03-19T11:31:11.708 INFO:tasks.rados.rados.0.smithi038.stdout:1294: finishing write tid 2 to smithi03829856-33 2024-03-19T11:31:11.708 INFO:tasks.rados.rados.0.smithi038.stdout:1295: finishing set_chunk to oid 37 2024-03-19T11:31:11.708 INFO:tasks.rados.rados.0.smithi038.stdout:1295: got expected ERANGE (src dne) 2024-03-19T11:31:11.709 INFO:tasks.rados.rados.0.smithi038.stdout:1296: got (22) Invalid argument 2024-03-19T11:31:11.709 INFO:tasks.rados.rados.0.smithi038.stdout:1300: expect (ObjNum 168 snap 17 seq_num 168) 2024-03-19T11:31:11.729 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:11.729 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:11.743 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-19T11:31:11.788 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:11.799+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:11.830 INFO:tasks.rados.rados.0.smithi038.stdout:1301: finishing set_chunk to oid 13 2024-03-19T11:31:11.830 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 13 set_chunk returned error code -95 offset: 1056768 length: 82432 2024-03-19T11:31:11.830 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 13 v 0 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-03-19T11:31:11.830 INFO:tasks.rados.rados.0.smithi038.stdout:1294: finishing write tid 3 to smithi03829856-33 2024-03-19T11:31:11.830 INFO:tasks.rados.rados.0.smithi038.stdout:1294: finishing write tid 4 to smithi03829856-33 2024-03-19T11:31:11.830 INFO:tasks.rados.rados.0.smithi038.stdout:1294: finishing write tid 5 to smithi03829856-33 2024-03-19T11:31:11.830 INFO:tasks.rados.rados.0.smithi038.stdout:1294: oid 33 updating version 0 to 128 2024-03-19T11:31:11.830 INFO:tasks.rados.rados.0.smithi038.stdout:1294: oid 33 version 128 is already newer than 126 2024-03-19T11:31:11.830 INFO:tasks.rados.rados.0.smithi038.stdout:1294: oid 33 updating version 128 to 129 2024-03-19T11:31:11.830 INFO:tasks.rados.rados.0.smithi038.stdout:1294: oid 33 version 129 is already newer than 127 2024-03-19T11:31:11.831 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 33 v 129 (ObjNum 218 snap 23 seq_num 218) dirty exists 2024-03-19T11:31:11.831 INFO:tasks.rados.rados.0.smithi038.stdout:1294: left oid 33 (ObjNum 218 snap 23 seq_num 218) 2024-03-19T11:31:11.831 INFO:tasks.rados.rados.0.smithi038.stdout:1297: expect (ObjNum 188 snap 21 seq_num 188) 2024-03-19T11:31:11.863 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:11.863 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:11.864 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-19T11:31:11.868 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:11.877+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:11.877 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-19T11:31:11.968 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:11.979+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:11.998 INFO:tasks.rados.rados.0.smithi038.stdout:1299: expect (ObjNum 170 snap 17 seq_num 170) 2024-03-19T11:31:11.998 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:31:11.999 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:31:12.015 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:31:12.015 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:31:12.031 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:31:12.031 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:31:12.045 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:12.046 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:12.046 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:31:12.046 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:31:12.058 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:12.065+0000 7f6712f92640 -1 osd.2 92 handle_osd_map injecting map crc failure 2024-03-19T11:31:12.061 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-19T11:31:12.069 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:31:12.069 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:31:12.069 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:12.077+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:12.070 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-03-19T11:31:12.089 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:31:12.089 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:31:12.089 INFO:tasks.rados.rados.0.smithi038.stdout:1302: finishing set_chunk to oid 20 2024-03-19T11:31:12.090 INFO:tasks.rados.rados.0.smithi038.stdout:1302: got expected ERANGE (src dne) 2024-03-19T11:31:12.090 INFO:tasks.rados.rados.0.smithi038.stdout:1303: seq_num 219 ranges {480733=537415,1782380=696402,3031400=537159} 2024-03-19T11:31:12.095 INFO:tasks.rados.rados.0.smithi038.stdout:1303: writing smithi03829856-29 from 480733 to 1018148 tid 1 2024-03-19T11:31:12.103 INFO:tasks.rados.rados.0.smithi038.stdout:1303: writing smithi03829856-29 from 1782380 to 2478782 tid 2 2024-03-19T11:31:12.110 INFO:tasks.rados.rados.0.smithi038.stdout:1303: writing smithi03829856-29 from 3031400 to 3568559 tid 3 2024-03-19T11:31:12.111 INFO:tasks.rados.rados.0.smithi038.stdout:1291: done (12 left) 2024-03-19T11:31:12.111 INFO:tasks.rados.rados.0.smithi038.stdout:1292: done (11 left) 2024-03-19T11:31:12.111 INFO:tasks.rados.rados.0.smithi038.stdout:1293: done (10 left) 2024-03-19T11:31:12.111 INFO:tasks.rados.rados.0.smithi038.stdout:1294: done (9 left) 2024-03-19T11:31:12.111 INFO:tasks.rados.rados.0.smithi038.stdout:1295: done (8 left) 2024-03-19T11:31:12.111 INFO:tasks.rados.rados.0.smithi038.stdout:1296: done (7 left) 2024-03-19T11:31:12.111 INFO:tasks.rados.rados.0.smithi038.stdout:1297: done (6 left) 2024-03-19T11:31:12.111 INFO:tasks.rados.rados.0.smithi038.stdout:1299: done (5 left) 2024-03-19T11:31:12.111 INFO:tasks.rados.rados.0.smithi038.stdout:1300: done (4 left) 2024-03-19T11:31:12.111 INFO:tasks.rados.rados.0.smithi038.stdout:1301: done (3 left) 2024-03-19T11:31:12.111 INFO:tasks.rados.rados.0.smithi038.stdout:1302: done (2 left) 2024-03-19T11:31:12.111 INFO:tasks.rados.rados.0.smithi038.stdout:1304: read oid 22 snap -1 2024-03-19T11:31:12.111 INFO:tasks.rados.rados.0.smithi038.stdout:1304: expect (ObjNum 168 snap 17 seq_num 168) 2024-03-19T11:31:12.112 INFO:tasks.rados.rados.0.smithi038.stdout:1298: expect (ObjNum 108 snap 11 seq_num 108) 2024-03-19T11:31:12.115 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:31:12.116 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:31:12.136 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:31:12.136 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:31:12.200 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:12.209+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:12.234 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:12.235 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:31:12.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:31:12.235 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:31:12.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.235 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:30 4.5 4:ac9cd3a3:::smithi03829856-8:head [write 1776668~340981] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:31:12.235 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:04.777922+0000", 2024-03-19T11:31:12.235 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 67.464655504000007, 2024-03-19T11:31:12.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015469894, 2024-03-19T11:31:12.235 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:12.235 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:12.236 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:12.236 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:12.236 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:12.236 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 30 2024-03-19T11:31:12.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.236 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:12.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.236 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:12.236 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.777922+0000", 2024-03-19T11:31:12.236 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.236 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:12.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.777922+0000", 2024-03-19T11:31:12.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:12.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.777924+0000", 2024-03-19T11:31:12.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3999999999999999e-06 2024-03-19T11:31:12.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:12.238 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778461+0000", 2024-03-19T11:31:12.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00053698299999999999 2024-03-19T11:31:12.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:12.238 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778487+0000", 2024-03-19T11:31:12.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5947000000000001e-05 2024-03-19T11:31:12.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:12.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778502+0000", 2024-03-19T11:31:12.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4256e-05 2024-03-19T11:31:12.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:12.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.781978+0000", 2024-03-19T11:31:12.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00347693 2024-03-19T11:31:12.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:12.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.782196+0000", 2024-03-19T11:31:12.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021726499999999999 2024-03-19T11:31:12.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:12.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.782572+0000", 2024-03-19T11:31:12.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00037635000000000001 2024-03-19T11:31:12.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:12.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.786960+0000", 2024-03-19T11:31:12.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043882130000000002 2024-03-19T11:31:12.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:12.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793113+0000", 2024-03-19T11:31:12.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061530760000000004 2024-03-19T11:31:12.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:12.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793276+0000", 2024-03-19T11:31:12.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000162652 2024-03-19T11:31:12.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.242 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:12.242 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793392+0000", 2024-03-19T11:31:12.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115822 2024-03-19T11:31:12.242 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.242 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:12.242 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.242 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:31 4.5 4:ac9cd3a3:::smithi03829856-8:head [setxattr _header (55) in=7b,truncate 2117649] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:31:12.243 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:04.778533+0000", 2024-03-19T11:31:12.243 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 67.464044369000007, 2024-03-19T11:31:12.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015182533, 2024-03-19T11:31:12.243 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:12.243 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:12.243 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:12.243 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:12.243 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:12.243 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 31 2024-03-19T11:31:12.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.243 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:12.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:12.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778533+0000", 2024-03-19T11:31:12.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:12.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778533+0000", 2024-03-19T11:31:12.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:12.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778534+0000", 2024-03-19T11:31:12.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1069999999999999e-06 2024-03-19T11:31:12.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:12.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778541+0000", 2024-03-19T11:31:12.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2529999999999998e-06 2024-03-19T11:31:12.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:12.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778558+0000", 2024-03-19T11:31:12.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6290000000000002e-05 2024-03-19T11:31:12.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:12.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778568+0000", 2024-03-19T11:31:12.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0360000000000001e-05 2024-03-19T11:31:12.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:12.247 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.785511+0000", 2024-03-19T11:31:12.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00694306 2024-03-19T11:31:12.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.247 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:12.247 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.785710+0000", 2024-03-19T11:31:12.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019837199999999999 2024-03-19T11:31:12.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.247 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:12.247 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.785929+0000", 2024-03-19T11:31:12.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021960799999999999 2024-03-19T11:31:12.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.248 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:12.248 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.787281+0000", 2024-03-19T11:31:12.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001352313 2024-03-19T11:31:12.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.248 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:12.248 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793492+0000", 2024-03-19T11:31:12.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062107760000000003 2024-03-19T11:31:12.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:12.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793578+0000", 2024-03-19T11:31:12.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5711999999999995e-05 2024-03-19T11:31:12.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:12.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793716+0000", 2024-03-19T11:31:12.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000137682 2024-03-19T11:31:12.249 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.249 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:12.249 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.250 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:32 4.5 4:ac9cd3a3:::smithi03829856-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:31:12.250 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:04.778582+0000", 2024-03-19T11:31:12.250 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 67.463995854999993, 2024-03-19T11:31:12.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015761827999999999, 2024-03-19T11:31:12.250 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:12.250 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:12.250 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:12.250 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:12.250 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:12.250 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 32 2024-03-19T11:31:12.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.251 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:12.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:12.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778582+0000", 2024-03-19T11:31:12.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:12.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778582+0000", 2024-03-19T11:31:12.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:12.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778583+0000", 2024-03-19T11:31:12.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.026e-06 2024-03-19T11:31:12.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:12.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778587+0000", 2024-03-19T11:31:12.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7870000000000001e-06 2024-03-19T11:31:12.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:12.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778599+0000", 2024-03-19T11:31:12.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1623e-05 2024-03-19T11:31:12.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:12.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778608+0000", 2024-03-19T11:31:12.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7399999999999993e-06 2024-03-19T11:31:12.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:12.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.787009+0000", 2024-03-19T11:31:12.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0084010490000000007 2024-03-19T11:31:12.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-03-19T11:31:12.254 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.787202+0000", 2024-03-19T11:31:12.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019309500000000001 2024-03-19T11:31:12.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.254 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:12.254 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793832+0000", 2024-03-19T11:31:12.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.006630052 2024-03-19T11:31:12.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.254 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:12.254 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.794048+0000", 2024-03-19T11:31:12.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000215505 2024-03-19T11:31:12.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:12.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.794343+0000", 2024-03-19T11:31:12.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029595100000000003 2024-03-19T11:31:12.255 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.255 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:12.255 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.255 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:856 4.5 4:ac9cd3a3:::smithi03829856-8:head [write 608420~529411] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:31:12.255 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.781425+0000", 2024-03-19T11:31:12.255 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 63.461152669000001, 2024-03-19T11:31:12.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014634677, 2024-03-19T11:31:12.256 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:12.256 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:12.256 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:12.256 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:12.256 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:12.256 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 856 2024-03-19T11:31:12.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.256 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:12.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:12.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.781425+0000", 2024-03-19T11:31:12.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:12.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.781425+0000", 2024-03-19T11:31:12.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:12.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.781428+0000", 2024-03-19T11:31:12.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.534e-06 2024-03-19T11:31:12.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:12.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.783096+0000", 2024-03-19T11:31:12.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001667897 2024-03-19T11:31:12.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:12.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.783138+0000", 2024-03-19T11:31:12.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1946000000000001e-05 2024-03-19T11:31:12.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:12.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.783163+0000", 2024-03-19T11:31:12.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4443e-05 2024-03-19T11:31:12.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:12.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.783333+0000", 2024-03-19T11:31:12.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017027899999999999 2024-03-19T11:31:12.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:12.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.783503+0000", 2024-03-19T11:31:12.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000170242 2024-03-19T11:31:12.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:12.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.783917+0000", 2024-03-19T11:31:12.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00041371000000000002 2024-03-19T11:31:12.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.261 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:12.261 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.795709+0000", 2024-03-19T11:31:12.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011791693000000001 2024-03-19T11:31:12.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.261 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:12.261 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.795850+0000", 2024-03-19T11:31:12.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000141695 2024-03-19T11:31:12.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.262 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:12.262 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.795940+0000", 2024-03-19T11:31:12.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9705999999999998e-05 2024-03-19T11:31:12.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.262 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:12.262 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.796059+0000", 2024-03-19T11:31:12.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011953199999999999 2024-03-19T11:31:12.262 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.262 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:12.262 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.263 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:891 4.5 4:af9d0d56:::smithi03829856-20:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:31:12.263 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:09.286410+0000", 2024-03-19T11:31:12.263 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 62.956167522999998, 2024-03-19T11:31:12.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035191405000000002, 2024-03-19T11:31:12.264 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:12.264 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:12.264 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:12.264 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:12.264 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:12.264 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 891 2024-03-19T11:31:12.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.264 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:12.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:12.264 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286410+0000", 2024-03-19T11:31:12.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.265 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.265 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.265 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:12.265 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286410+0000", 2024-03-19T11:31:12.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.265 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.265 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.265 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:12.265 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286413+0000", 2024-03-19T11:31:12.266 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2660000000000002e-06 2024-03-19T11:31:12.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.266 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:12.266 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286426+0000", 2024-03-19T11:31:12.266 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.258e-05 2024-03-19T11:31:12.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.266 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:12.266 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286460+0000", 2024-03-19T11:31:12.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3943999999999999e-05 2024-03-19T11:31:12.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.267 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:12.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286477+0000", 2024-03-19T11:31:12.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6799999999999998e-05 2024-03-19T11:31:12.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.267 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:12.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286605+0000", 2024-03-19T11:31:12.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000128892 2024-03-19T11:31:12.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.268 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:12.268 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286769+0000", 2024-03-19T11:31:12.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016359799999999999 2024-03-19T11:31:12.268 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.268 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.268 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:12.268 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.321601+0000", 2024-03-19T11:31:12.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.034832324999999997 2024-03-19T11:31:12.268 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.268 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:12.268 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.269 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.269 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.269 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1015 4.5 4:af9d0d56:::smithi03829856-20:head [tier-promote] snapc 6=[6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:31:12.269 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:12.761855+0000", 2024-03-19T11:31:12.269 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 59.480721997000003, 2024-03-19T11:31:12.269 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026760430000000002, 2024-03-19T11:31:12.269 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:12.269 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:12.269 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:12.269 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:12.269 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:12.270 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1015 2024-03-19T11:31:12.270 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.270 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:12.270 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.270 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:12.270 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761855+0000", 2024-03-19T11:31:12.270 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.270 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.270 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.270 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:12.270 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761855+0000", 2024-03-19T11:31:12.270 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.271 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:12.271 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761858+0000", 2024-03-19T11:31:12.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8140000000000001e-06 2024-03-19T11:31:12.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.271 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:12.271 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761872+0000", 2024-03-19T11:31:12.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3813e-05 2024-03-19T11:31:12.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:12.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761905+0000", 2024-03-19T11:31:12.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2642999999999999e-05 2024-03-19T11:31:12.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:12.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761925+0000", 2024-03-19T11:31:12.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0489e-05 2024-03-19T11:31:12.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:12.273 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.762070+0000", 2024-03-19T11:31:12.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000145153 2024-03-19T11:31:12.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:12.273 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.762365+0000", 2024-03-19T11:31:12.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029482100000000002 2024-03-19T11:31:12.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:12.273 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.788616+0000", 2024-03-19T11:31:12.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026250697 2024-03-19T11:31:12.273 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.274 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:12.274 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.274 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.274 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.274 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1121 4.5 4:aa3303a5:::smithi03829856-17:head [tier-promote] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:31:12.274 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:14.164281+0000", 2024-03-19T11:31:12.274 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 58.078296752, 2024-03-19T11:31:12.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023896435000000001, 2024-03-19T11:31:12.274 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:12.275 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:12.275 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:12.275 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:12.275 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:12.275 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1121 2024-03-19T11:31:12.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.275 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:12.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:12.275 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164281+0000", 2024-03-19T11:31:12.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:12.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164281+0000", 2024-03-19T11:31:12.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:12.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164283+0000", 2024-03-19T11:31:12.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8500000000000001e-06 2024-03-19T11:31:12.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.277 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:12.277 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164291+0000", 2024-03-19T11:31:12.277 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.14e-06 2024-03-19T11:31:12.277 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.277 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:12.277 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164313+0000", 2024-03-19T11:31:12.277 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.207e-05 2024-03-19T11:31:12.277 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.278 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:12.278 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164328+0000", 2024-03-19T11:31:12.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5403000000000001e-05 2024-03-19T11:31:12.278 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.278 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.278 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:12.278 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.167981+0000", 2024-03-19T11:31:12.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036529879999999998 2024-03-19T11:31:12.278 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.278 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.278 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:12.278 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.168274+0000", 2024-03-19T11:31:12.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029313100000000002 2024-03-19T11:31:12.279 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.279 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:12.279 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.188177+0000", 2024-03-19T11:31:12.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019902853000000002 2024-03-19T11:31:12.279 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.279 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:12.279 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.279 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.280 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1302 4.15 4:aa3303a5:::smithi03829856-17:head [write 413472~620140,stat] snapc b=[b,a,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-03-19T11:31:12.280 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:22.966426+0000", 2024-03-19T11:31:12.280 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 49.276151018999997, 2024-03-19T11:31:12.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018043513000000001, 2024-03-19T11:31:12.280 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:12.280 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:12.280 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:12.280 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:12.280 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:12.280 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1302 2024-03-19T11:31:12.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.281 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:12.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:12.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.966426+0000", 2024-03-19T11:31:12.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:12.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.966426+0000", 2024-03-19T11:31:12.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:12.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.966429+0000", 2024-03-19T11:31:12.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3829999999999999e-06 2024-03-19T11:31:12.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:12.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972117+0000", 2024-03-19T11:31:12.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0056885750000000004 2024-03-19T11:31:12.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:12.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972156+0000", 2024-03-19T11:31:12.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8878000000000003e-05 2024-03-19T11:31:12.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:12.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972184+0000", 2024-03-19T11:31:12.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7509999999999999e-05 2024-03-19T11:31:12.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:12.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972309+0000", 2024-03-19T11:31:12.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012525899999999999 2024-03-19T11:31:12.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.284 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:12.284 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972566+0000", 2024-03-19T11:31:12.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025740799999999999 2024-03-19T11:31:12.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.284 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:12.284 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972899+0000", 2024-03-19T11:31:12.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033238799999999998 2024-03-19T11:31:12.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:12.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984088+0000", 2024-03-19T11:31:12.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011189466 2024-03-19T11:31:12.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:12.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984138+0000", 2024-03-19T11:31:12.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9666999999999997e-05 2024-03-19T11:31:12.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:12.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984329+0000", 2024-03-19T11:31:12.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019086799999999999 2024-03-19T11:31:12.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:12.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984470+0000", 2024-03-19T11:31:12.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000141111 2024-03-19T11:31:12.286 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.286 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:12.286 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.286 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1303 4.15 4:aa3303a5:::smithi03829856-17:head [write 1818036~510700] snapc b=[b,a,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-03-19T11:31:12.286 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:22.976856+0000", 2024-03-19T11:31:12.287 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 49.265721909, 2024-03-19T11:31:12.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015045703000000001, 2024-03-19T11:31:12.287 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:12.287 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:12.287 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:12.287 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:12.287 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:12.287 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1303 2024-03-19T11:31:12.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.287 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:12.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.287 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:12.287 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.976856+0000", 2024-03-19T11:31:12.288 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.288 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:12.288 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.976856+0000", 2024-03-19T11:31:12.288 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.288 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:12.288 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.976859+0000", 2024-03-19T11:31:12.288 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3629999999999998e-06 2024-03-19T11:31:12.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:12.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.977533+0000", 2024-03-19T11:31:12.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00067380800000000002 2024-03-19T11:31:12.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:12.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.977573+0000", 2024-03-19T11:31:12.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9758000000000001e-05 2024-03-19T11:31:12.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:12.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.977602+0000", 2024-03-19T11:31:12.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9856e-05 2024-03-19T11:31:12.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:12.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.979943+0000", 2024-03-19T11:31:12.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0023406400000000002 2024-03-19T11:31:12.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:12.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.980238+0000", 2024-03-19T11:31:12.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029488000000000002 2024-03-19T11:31:12.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:12.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.980734+0000", 2024-03-19T11:31:12.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00049647500000000002 2024-03-19T11:31:12.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:12.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.991761+0000", 2024-03-19T11:31:12.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011026515000000001 2024-03-19T11:31:12.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:12.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.991777+0000", 2024-03-19T11:31:12.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5727999999999999e-05 2024-03-19T11:31:12.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.292 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:12.292 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.991840+0000", 2024-03-19T11:31:12.292 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3091999999999998e-05 2024-03-19T11:31:12.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.292 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:12.292 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.991901+0000", 2024-03-19T11:31:12.292 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1587999999999994e-05 2024-03-19T11:31:12.292 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.292 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:12.293 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.293 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.1 50/49 [PushOp(4:866f3dcc:::smithi03829856-31:7, version: 38'44, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@38'44, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 45'62, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@45'62, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 46'68, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@46'68, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:7, version: 47'71, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@47'71, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 47'72, data_included: [440350~816629,1309629~231947], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@47'72, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:12.293 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.959457+0000", 2024-03-19T11:31:12.293 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 44.283120601999997, 2024-03-19T11:31:12.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018416666000000002, 2024-03-19T11:31:12.293 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:12.293 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:12.293 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:12.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.294 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:12.294 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.959457+0000", 2024-03-19T11:31:12.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.294 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:12.294 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.959457+0000", 2024-03-19T11:31:12.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:12.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.959458+0000", 2024-03-19T11:31:12.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4899999999999999e-06 2024-03-19T11:31:12.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:12.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.962822+0000", 2024-03-19T11:31:12.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0033640620000000001 2024-03-19T11:31:12.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.296 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:12.296 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.962893+0000", 2024-03-19T11:31:12.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1060000000000001e-05 2024-03-19T11:31:12.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.296 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:12.296 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.962977+0000", 2024-03-19T11:31:12.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3311000000000002e-05 2024-03-19T11:31:12.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:12.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963208+0000", 2024-03-19T11:31:12.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023121200000000001 2024-03-19T11:31:12.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:12.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963245+0000", 2024-03-19T11:31:12.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6844999999999997e-05 2024-03-19T11:31:12.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:12.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.977874+0000", 2024-03-19T11:31:12.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014628686 2024-03-19T11:31:12.298 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.298 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:12.298 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.299 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.1 e50,49 4:8401c1f0:::smithi03829856-43:8,46'66,4:866f3dcc:::smithi03829856-31:2,47'73)", 2024-03-19T11:31:12.299 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.963027+0000", 2024-03-19T11:31:12.299 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 44.279550985, 2024-03-19T11:31:12.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015116788000000001, 2024-03-19T11:31:12.299 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:12.299 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:12.299 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:12.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:12.300 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963027+0000", 2024-03-19T11:31:12.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.300 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:12.300 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963027+0000", 2024-03-19T11:31:12.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.300 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:12.301 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963028+0000", 2024-03-19T11:31:12.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.268e-06 2024-03-19T11:31:12.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.301 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:12.301 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963036+0000", 2024-03-19T11:31:12.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.926e-06 2024-03-19T11:31:12.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.301 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:12.301 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963050+0000", 2024-03-19T11:31:12.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4082e-05 2024-03-19T11:31:12.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:12.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963062+0000", 2024-03-19T11:31:12.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2415000000000001e-05 2024-03-19T11:31:12.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:12.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.977954+0000", 2024-03-19T11:31:12.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014891572 2024-03-19T11:31:12.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:12.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.977965+0000", 2024-03-19T11:31:12.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1654000000000001e-05 2024-03-19T11:31:12.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:12.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.978143+0000", 2024-03-19T11:31:12.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017787099999999999 2024-03-19T11:31:12.303 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.303 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:12.303 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.304 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.11 50/49 [PushOp(4:8d3763f9:::smithi03829856-29:head, version: 47'60, data_included: [3364276~232389], data_size: 232389, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8d3763f9:::smithi03829856-29:head@47'60, size: 3596665, copy_subset: [(0, 3596665)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3596665, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3364276, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:12.304 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.964363+0000", 2024-03-19T11:31:12.304 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 44.278214318000003, 2024-03-19T11:31:12.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014531759999999999, 2024-03-19T11:31:12.304 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:12.304 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:12.304 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:12.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:12.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.964363+0000", 2024-03-19T11:31:12.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.305 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.305 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:12.305 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.964363+0000", 2024-03-19T11:31:12.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.305 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.305 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:12.305 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.964364+0000", 2024-03-19T11:31:12.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.4200000000000004e-07 2024-03-19T11:31:12.305 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:12.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.964447+0000", 2024-03-19T11:31:12.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2430999999999997e-05 2024-03-19T11:31:12.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:12.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.964468+0000", 2024-03-19T11:31:12.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1685999999999999e-05 2024-03-19T11:31:12.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:12.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.964483+0000", 2024-03-19T11:31:12.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4969e-05 2024-03-19T11:31:12.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:12.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.977934+0000", 2024-03-19T11:31:12.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013450963999999999 2024-03-19T11:31:12.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:12.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.977975+0000", 2024-03-19T11:31:12.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1125999999999999e-05 2024-03-19T11:31:12.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:12.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.978895+0000", 2024-03-19T11:31:12.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00091964200000000003 2024-03-19T11:31:12.308 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.308 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:12.308 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.308 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2284 4.15 4:afa9f208:::smithi03829856-35:head [read 1314816~702976] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-03-19T11:31:12.308 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:53.596775+0000", 2024-03-19T11:31:12.308 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.645802516, 2024-03-19T11:31:12.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.47260528299999999, 2024-03-19T11:31:12.309 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:12.309 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:12.309 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:12.309 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:12.309 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:12.309 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2284 2024-03-19T11:31:12.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.309 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:12.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.309 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:12.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596775+0000", 2024-03-19T11:31:12.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:12.310 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596775+0000", 2024-03-19T11:31:12.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:12.310 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596778+0000", 2024-03-19T11:31:12.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.174e-06 2024-03-19T11:31:12.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:12.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596794+0000", 2024-03-19T11:31:12.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5372000000000001e-05 2024-03-19T11:31:12.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:12.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596828+0000", 2024-03-19T11:31:12.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4107999999999999e-05 2024-03-19T11:31:12.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:12.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596844+0000", 2024-03-19T11:31:12.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6296000000000001e-05 2024-03-19T11:31:12.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:12.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596968+0000", 2024-03-19T11:31:12.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012383699999999999 2024-03-19T11:31:12.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:31:12.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596988+0000", 2024-03-19T11:31:12.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0645e-05 2024-03-19T11:31:12.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:12.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.068916+0000", 2024-03-19T11:31:12.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.47192744800000003 2024-03-19T11:31:12.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.314 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:12.314 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.069015+0000", 2024-03-19T11:31:12.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9500999999999995e-05 2024-03-19T11:31:12.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.314 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:12.314 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.069380+0000", 2024-03-19T11:31:12.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00036490200000000001 2024-03-19T11:31:12.314 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.314 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:12.315 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.315 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2286 4.15 4:afa9f208:::smithi03829856-35:head [assert-version v108,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e75)", 2024-03-19T11:31:12.315 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.084826+0000", 2024-03-19T11:31:12.315 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.157751148999999, 2024-03-19T11:31:12.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.055799556, 2024-03-19T11:31:12.315 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:12.315 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:12.315 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:12.315 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:12.315 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:12.316 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2286 2024-03-19T11:31:12.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.316 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:12.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:12.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084826+0000", 2024-03-19T11:31:12.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:12.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084826+0000", 2024-03-19T11:31:12.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:12.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084828+0000", 2024-03-19T11:31:12.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1409999999999999e-06 2024-03-19T11:31:12.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:12.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084838+0000", 2024-03-19T11:31:12.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3740000000000004e-06 2024-03-19T11:31:12.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.318 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:12.318 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084859+0000", 2024-03-19T11:31:12.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1644000000000001e-05 2024-03-19T11:31:12.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.318 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:12.318 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084871+0000", 2024-03-19T11:31:12.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1127e-05 2024-03-19T11:31:12.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.318 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:12.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084961+0000", 2024-03-19T11:31:12.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0371999999999995e-05 2024-03-19T11:31:12.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:12.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.085130+0000", 2024-03-19T11:31:12.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016884800000000001 2024-03-19T11:31:12.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5,7", 2024-03-19T11:31:12.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.089309+0000", 2024-03-19T11:31:12.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041788650000000004 2024-03-19T11:31:12.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:12.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.090362+0000", 2024-03-19T11:31:12.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001053362 2024-03-19T11:31:12.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:12.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.090511+0000", 2024-03-19T11:31:12.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014887200000000001 2024-03-19T11:31:12.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:12.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.140479+0000", 2024-03-19T11:31:12.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.049968006000000002 2024-03-19T11:31:12.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:12.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.140534+0000", 2024-03-19T11:31:12.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4917000000000003e-05 2024-03-19T11:31:12.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:12.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.140626+0000", 2024-03-19T11:31:12.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2027999999999994e-05 2024-03-19T11:31:12.322 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.322 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:12.322 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.322 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1d58db11:::smithi03829856-18:6, version: 55'80, data_included: [759683~639358,1904434~409218], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:6@55'80, size: 3159683, copy_subset: [(0, 3159683)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2313652, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1d58db11:::smithi03829856-18:b, version: 58'84, data_included: [443718~955323,1564138~93253], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:b@58'84, size: 3445402, copy_subset: [(0, 3445402)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1657391, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:10, version: 66'95, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:10@66'95, size: 3764735, copy_subset: [(0, 3764735)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:12, version: 66'97, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:12@66'97, size: 2504243, copy_subset: [(0, 2504243)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:c, version: 70'102, data_included: [416807~576167,1542129~472409], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:c@70'102, size: 2857157, copy_subset: [(0, 2857157)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2014538, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:12.323 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.274749+0000", 2024-03-19T11:31:12.323 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 17.967828218000001, 2024-03-19T11:31:12.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017652601, 2024-03-19T11:31:12.323 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:12.323 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:12.323 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:12.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.323 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:12.323 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.274749+0000", 2024-03-19T11:31:12.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:12.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.274749+0000", 2024-03-19T11:31:12.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:12.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.274751+0000", 2024-03-19T11:31:12.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8500000000000001e-06 2024-03-19T11:31:12.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:12.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.278426+0000", 2024-03-19T11:31:12.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036747559999999999 2024-03-19T11:31:12.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:12.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.278470+0000", 2024-03-19T11:31:12.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4119000000000002e-05 2024-03-19T11:31:12.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:12.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.278506+0000", 2024-03-19T11:31:12.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6066000000000003e-05 2024-03-19T11:31:12.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:12.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.278709+0000", 2024-03-19T11:31:12.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000202711 2024-03-19T11:31:12.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:12.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.278731+0000", 2024-03-19T11:31:12.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2657e-05 2024-03-19T11:31:12.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:12.327 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.292402+0000", 2024-03-19T11:31:12.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013670442 2024-03-19T11:31:12.327 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.327 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:12.327 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.327 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1dabc0f1:::smithi03829856-1:head, version: 70'103, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:head@70'103, size: 2621892, copy_subset: [(0, 2621892)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:12.327 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.280012+0000", 2024-03-19T11:31:12.328 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 17.962565572999999, 2024-03-19T11:31:12.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018569209, 2024-03-19T11:31:12.328 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:12.328 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:12.328 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:12.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:12.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280012+0000", 2024-03-19T11:31:12.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:12.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280012+0000", 2024-03-19T11:31:12.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:12.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280013+0000", 2024-03-19T11:31:12.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9899999999999999e-07 2024-03-19T11:31:12.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:12.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280584+0000", 2024-03-19T11:31:12.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057090299999999999 2024-03-19T11:31:12.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:12.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280612+0000", 2024-03-19T11:31:12.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8442999999999999e-05 2024-03-19T11:31:12.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:12.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280631+0000", 2024-03-19T11:31:12.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8896999999999999e-05 2024-03-19T11:31:12.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:12.331 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.295041+0000", 2024-03-19T11:31:12.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014410127999999999 2024-03-19T11:31:12.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.331 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:12.331 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.295050+0000", 2024-03-19T11:31:12.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2350000000000005e-06 2024-03-19T11:31:12.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.331 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:12.331 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.298581+0000", 2024-03-19T11:31:12.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035308039999999998 2024-03-19T11:31:12.332 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.332 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:12.332 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.332 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.332 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.18 e75,74 4:1d58db11:::smithi03829856-18:c,70'104)", 2024-03-19T11:31:12.332 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.280680+0000", 2024-03-19T11:31:12.332 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 17.961897098000001, 2024-03-19T11:31:12.332 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018106229000000001, 2024-03-19T11:31:12.332 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:12.333 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:12.333 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:12.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:12.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280680+0000", 2024-03-19T11:31:12.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:12.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280680+0000", 2024-03-19T11:31:12.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:12.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280681+0000", 2024-03-19T11:31:12.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2099999999999995e-07 2024-03-19T11:31:12.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:12.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280686+0000", 2024-03-19T11:31:12.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2240000000000003e-06 2024-03-19T11:31:12.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.335 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:12.335 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280698+0000", 2024-03-19T11:31:12.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1698999999999999e-05 2024-03-19T11:31:12.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.336 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:12.336 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280705+0000", 2024-03-19T11:31:12.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4350000000000002e-06 2024-03-19T11:31:12.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.336 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:12.336 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.298643+0000", 2024-03-19T11:31:12.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017938352000000001 2024-03-19T11:31:12.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:12.337 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.298656+0000", 2024-03-19T11:31:12.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3084999999999999e-05 2024-03-19T11:31:12.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:12.338 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.298787+0000", 2024-03-19T11:31:12.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013061299999999999 2024-03-19T11:31:12.338 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.338 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:12.338 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.338 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.338 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.5.0:143 4.18 e75/74 4:1d58db11:::smithi03829856-18:head v 75'106, mlcod=0'0)", 2024-03-19T11:31:12.338 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.282327+0000", 2024-03-19T11:31:12.338 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 17.960250727000002, 2024-03-19T11:31:12.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016753244, 2024-03-19T11:31:12.339 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:12.339 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:12.339 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:12.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:12.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.282327+0000", 2024-03-19T11:31:12.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.340 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:12.340 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.282327+0000", 2024-03-19T11:31:12.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.340 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:12.340 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.282328+0000", 2024-03-19T11:31:12.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.488e-06 2024-03-19T11:31:12.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.341 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:12.341 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.283127+0000", 2024-03-19T11:31:12.341 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00079853699999999999 2024-03-19T11:31:12.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.341 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:12.341 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.283152+0000", 2024-03-19T11:31:12.341 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5215000000000002e-05 2024-03-19T11:31:12.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.341 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:12.342 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.283167+0000", 2024-03-19T11:31:12.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4635e-05 2024-03-19T11:31:12.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.342 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:12.342 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.292633+0000", 2024-03-19T11:31:12.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0094664339999999993 2024-03-19T11:31:12.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.342 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:12.342 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.292655+0000", 2024-03-19T11:31:12.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1934999999999999e-05 2024-03-19T11:31:12.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:12.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.298981+0000", 2024-03-19T11:31:12.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0063255129999999996 2024-03-19T11:31:12.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.343 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:12.343 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.299080+0000", 2024-03-19T11:31:12.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9487000000000005e-05 2024-03-19T11:31:12.343 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.344 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:12.344 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.344 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2441 4.5 4:a38b9453:::smithi03829856-37:head [delete] snapc 17=[17,16,15,13,12,b,6] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e84)", 2024-03-19T11:31:12.344 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:03.060663+0000", 2024-03-19T11:31:12.344 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.1819148829999993, 2024-03-19T11:31:12.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10917726, 2024-03-19T11:31:12.344 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:12.344 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:12.345 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:12.345 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:12.345 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:12.345 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2441 2024-03-19T11:31:12.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.345 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:12.345 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.345 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:12.345 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060663+0000", 2024-03-19T11:31:12.345 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.345 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:12.346 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060663+0000", 2024-03-19T11:31:12.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:12.346 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060666+0000", 2024-03-19T11:31:12.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.05e-06 2024-03-19T11:31:12.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.347 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.347 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:12.347 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060680+0000", 2024-03-19T11:31:12.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4276e-05 2024-03-19T11:31:12.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.347 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.347 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:12.347 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060714+0000", 2024-03-19T11:31:12.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4325000000000001e-05 2024-03-19T11:31:12.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.347 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:12.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060737+0000", 2024-03-19T11:31:12.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2959000000000002e-05 2024-03-19T11:31:12.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:12.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060895+0000", 2024-03-19T11:31:12.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000157787 2024-03-19T11:31:12.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.349 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:31:12.349 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060923+0000", 2024-03-19T11:31:12.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8019000000000001e-05 2024-03-19T11:31:12.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.349 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.349 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:12.349 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.066222+0000", 2024-03-19T11:31:12.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052989270000000002 2024-03-19T11:31:12.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.350 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:12.350 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.070153+0000", 2024-03-19T11:31:12.350 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039315119999999999 2024-03-19T11:31:12.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.350 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:12.350 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.147930+0000", 2024-03-19T11:31:12.350 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.077776459000000006 2024-03-19T11:31:12.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.350 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:12.350 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.148079+0000", 2024-03-19T11:31:12.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014888 2024-03-19T11:31:12.351 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:12.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.148370+0000", 2024-03-19T11:31:12.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029130999999999997 2024-03-19T11:31:12.351 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:12.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.152531+0000", 2024-03-19T11:31:12.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041605399999999999 2024-03-19T11:31:12.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.352 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.352 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:12.352 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.169762+0000", 2024-03-19T11:31:12.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017231483999999998 2024-03-19T11:31:12.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.352 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.352 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:12.352 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.169807+0000", 2024-03-19T11:31:12.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5161999999999999e-05 2024-03-19T11:31:12.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:12.353 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.169840+0000", 2024-03-19T11:31:12.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2570000000000002e-05 2024-03-19T11:31:12.353 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.353 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:12.353 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.353 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPushReply(4.5 84/83 [PushReplyOp(4:a38b9453:::smithi03829856-37:head)])", 2024-03-19T11:31:12.353 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:03.084896+0000", 2024-03-19T11:31:12.354 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 9.157681749, 2024-03-19T11:31:12.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.02071104, 2024-03-19T11:31:12.354 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:12.354 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-19T11:31:12.354 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:12.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.354 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:12.354 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.084896+0000", 2024-03-19T11:31:12.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.355 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:12.355 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.084896+0000", 2024-03-19T11:31:12.355 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.355 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:12.355 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.084897+0000", 2024-03-19T11:31:12.355 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.686e-06 2024-03-19T11:31:12.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.355 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:12.355 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.084910+0000", 2024-03-19T11:31:12.356 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2948e-05 2024-03-19T11:31:12.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.356 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:12.356 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.084936+0000", 2024-03-19T11:31:12.356 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5982e-05 2024-03-19T11:31:12.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.356 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:12.356 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.084955+0000", 2024-03-19T11:31:12.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8416000000000001e-05 2024-03-19T11:31:12.357 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.357 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:12.357 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.102822+0000", 2024-03-19T11:31:12.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017866756000000001 2024-03-19T11:31:12.357 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.357 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:12.357 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.105607+0000", 2024-03-19T11:31:12.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027852520000000002 2024-03-19T11:31:12.357 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.358 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:12.358 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.358 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.358 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:12.358 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:12.360 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-19T11:31:12.361 INFO:tasks.rados.rados.0.smithi038.stdout:1303: finishing write tid 1 to smithi03829856-29 2024-03-19T11:31:12.361 INFO:tasks.rados.rados.0.smithi038.stdout:1303: finishing write tid 2 to smithi03829856-29 2024-03-19T11:31:12.361 INFO:tasks.rados.rados.0.smithi038.stdout:1298: done (2 left) 2024-03-19T11:31:12.361 INFO:tasks.rados.rados.0.smithi038.stdout:1305: set_chunk oid 16 target oid 2024-03-19T11:31:12.361 INFO:tasks.rados.rados.0.smithi038.stdout:1305: set_chunk oid 16 offset: 287744 length: 2533888 target oid 0a67e437b20d4ad126b67a2338de84fe4ef684c5c77cee818318d652193ba67c offset: 0 snap -1 2024-03-19T11:31:12.361 INFO:tasks.rados.rados.0.smithi038.stdout:1306: read oid 41 snap 19 2024-03-19T11:31:12.361 INFO:tasks.rados.rados.0.smithi038.stdout:1306: expect (ObjNum 170 snap 17 seq_num 170) 2024-03-19T11:31:12.362 INFO:tasks.rados.rados.0.smithi038.stdout:1307: read oid 18 snap -1 2024-03-19T11:31:12.362 INFO:tasks.rados.rados.0.smithi038.stdout:1307: expect (ObjNum 199 snap 22 seq_num 199) 2024-03-19T11:31:12.362 INFO:tasks.rados.rados.0.smithi038.stdout:1308: set_chunk oid 33 target oid 2024-03-19T11:31:12.362 INFO:tasks.rados.rados.0.smithi038.stdout:1308: set_chunk oid 33 offset: 260608 length: 3280896 target oid ca87bd16019841aaa831bb522ee079e5f4e6a1ea67fbcb51e5f89263b6c5d8f6 offset: 0 snap 22 2024-03-19T11:31:12.362 INFO:tasks.rados.rados.0.smithi038.stdout:1309: read oid 27 snap 22 2024-03-19T11:31:12.362 INFO:tasks.rados.rados.0.smithi038.stdout:1309: expect (ObjNum 192 snap 22 seq_num 192) 2024-03-19T11:31:12.362 INFO:tasks.rados.rados.0.smithi038.stdout:1310: snap_create 2024-03-19T11:31:12.362 INFO:tasks.rados.rados.0.smithi038.stdout:1303: finishing write tid 3 to smithi03829856-29 2024-03-19T11:31:12.362 INFO:tasks.rados.rados.0.smithi038.stdout:1303: finishing write tid 4 to smithi03829856-29 2024-03-19T11:31:12.362 INFO:tasks.rados.rados.0.smithi038.stdout:1303: finishing write tid 5 to smithi03829856-29 2024-03-19T11:31:12.362 INFO:tasks.rados.rados.0.smithi038.stdout:1303: oid 29 updating version 0 to 91 2024-03-19T11:31:12.363 INFO:tasks.rados.rados.0.smithi038.stdout:1303: oid 29 version 91 is already newer than 90 2024-03-19T11:31:12.363 INFO:tasks.rados.rados.0.smithi038.stdout:1303: oid 29 version 91 is already newer than 86 2024-03-19T11:31:12.363 INFO:tasks.rados.rados.0.smithi038.stdout:1303: oid 29 version 91 is already newer than 87 2024-03-19T11:31:12.363 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 29 v 91 (ObjNum 219 snap 23 seq_num 219) dirty exists 2024-03-19T11:31:12.363 INFO:tasks.rados.rados.0.smithi038.stdout:1303: left oid 29 (ObjNum 219 snap 23 seq_num 219) 2024-03-19T11:31:12.363 INFO:tasks.rados.rados.0.smithi038.stdout:1304: expect (ObjNum 168 snap 17 seq_num 168) 2024-03-19T11:31:12.363 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:12.369+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:12.461 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:12.469+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:12.470 INFO:tasks.rados.rados.0.smithi038.stdout:1305: finishing set_chunk to oid 16 2024-03-19T11:31:12.470 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 16 set_chunk returned error code -95 offset: 287744 length: 2533888 2024-03-19T11:31:12.471 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 16 v 0 (ObjNum 65 snap 3 seq_num 65) dirty exists 2024-03-19T11:31:12.471 INFO:tasks.rados.rados.0.smithi038.stdout:1306: expect (ObjNum 170 snap 17 seq_num 170) 2024-03-19T11:31:12.490 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:12.490 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:31:12.491 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:31:12.491 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:12.504 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-19T11:31:12.560 INFO:tasks.rados.rados.0.smithi038.stdout:1307: expect (ObjNum 199 snap 22 seq_num 199) 2024-03-19T11:31:12.562 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:12.569+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:12.626 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:12.626 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:31:12.626 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:31:12.627 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:31:12.627 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:12.643 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-19T11:31:12.663 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:12.669+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:12.764 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:12.775+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:12.769 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:12.769 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:31:12.769 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:31:12.769 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:31:12.769 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.769 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:31:12.770 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:44.200920+0000", 2024-03-19T11:31:12.770 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 88.576523768000001, 2024-03-19T11:31:12.770 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.80881896499999995, 2024-03-19T11:31:12.770 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:12.770 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:12.770 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:12.770 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:31:12.770 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:31:12.770 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1 2024-03-19T11:31:12.770 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.771 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:12.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.771 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:12.771 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200920+0000", 2024-03-19T11:31:12.771 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.771 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:12.771 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200920+0000", 2024-03-19T11:31:12.771 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.771 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:12.771 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200923+0000", 2024-03-19T11:31:12.771 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6469999999999998e-06 2024-03-19T11:31:12.772 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.772 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.772 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:12.772 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200937+0000", 2024-03-19T11:31:12.772 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.399e-05 2024-03-19T11:31:12.772 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.772 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.772 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:12.772 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200960+0000", 2024-03-19T11:31:12.772 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2603e-05 2024-03-19T11:31:12.772 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.772 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.772 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:12.772 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200978+0000", 2024-03-19T11:31:12.773 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.791e-05 2024-03-19T11:31:12.773 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.773 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.773 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:12.773 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.201081+0000", 2024-03-19T11:31:12.773 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010344900000000001 2024-03-19T11:31:12.773 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.773 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.773 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:31:12.773 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.201096+0000", 2024-03-19T11:31:12.773 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4443e-05 2024-03-19T11:31:12.773 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.773 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.773 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:12.773 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.005978+0000", 2024-03-19T11:31:12.774 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.80488281500000003 2024-03-19T11:31:12.774 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.774 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.774 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:12.774 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.006267+0000", 2024-03-19T11:31:12.774 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000288688 2024-03-19T11:31:12.774 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.774 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.774 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:12.774 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.006534+0000", 2024-03-19T11:31:12.774 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026725799999999998 2024-03-19T11:31:12.774 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.774 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.774 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:12.774 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.008457+0000", 2024-03-19T11:31:12.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0019227820000000001 2024-03-19T11:31:12.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:12.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.009570+0000", 2024-03-19T11:31:12.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011131750000000001 2024-03-19T11:31:12.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:12.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.009670+0000", 2024-03-19T11:31:12.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9458999999999998e-05 2024-03-19T11:31:12.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:12.776 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.009739+0000", 2024-03-19T11:31:12.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9746000000000006e-05 2024-03-19T11:31:12.776 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.776 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:12.776 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.776 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.776 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.776 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:34 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:31:12.776 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.071728+0000", 2024-03-19T11:31:12.776 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 87.705716331000005, 2024-03-19T11:31:12.777 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013714690999999999, 2024-03-19T11:31:12.777 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:12.777 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:12.777 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:12.777 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:31:12.777 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:31:12.777 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 34 2024-03-19T11:31:12.777 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.777 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:12.777 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.777 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:12.777 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071728+0000", 2024-03-19T11:31:12.777 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.777 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.777 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.778 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:12.778 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071728+0000", 2024-03-19T11:31:12.778 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.778 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.778 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.778 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:12.778 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071729+0000", 2024-03-19T11:31:12.778 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.47e-07 2024-03-19T11:31:12.778 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.778 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:12.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071735+0000", 2024-03-19T11:31:12.779 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3799999999999999e-06 2024-03-19T11:31:12.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.779 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:12.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071748+0000", 2024-03-19T11:31:12.779 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2806000000000001e-05 2024-03-19T11:31:12.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.780 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:12.780 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071756+0000", 2024-03-19T11:31:12.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1240000000000005e-06 2024-03-19T11:31:12.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.780 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:12.780 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.077570+0000", 2024-03-19T11:31:12.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.005813342 2024-03-19T11:31:12.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.781 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:12.781 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.077793+0000", 2024-03-19T11:31:12.781 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022391500000000001 2024-03-19T11:31:12.781 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.781 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.781 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:12.781 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.078146+0000", 2024-03-19T11:31:12.781 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00035219 2024-03-19T11:31:12.781 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.781 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.781 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:12.781 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.083390+0000", 2024-03-19T11:31:12.782 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052445590000000002 2024-03-19T11:31:12.782 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.782 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.782 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:12.782 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085288+0000", 2024-03-19T11:31:12.782 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0018977060000000001 2024-03-19T11:31:12.782 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.782 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.782 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:12.782 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085377+0000", 2024-03-19T11:31:12.782 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9068999999999997e-05 2024-03-19T11:31:12.783 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.783 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:12.783 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085443+0000", 2024-03-19T11:31:12.783 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5653000000000003e-05 2024-03-19T11:31:12.783 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.783 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:12.783 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.783 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.783 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:31:12.783 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.071777+0000", 2024-03-19T11:31:12.783 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 87.705667220999999, 2024-03-19T11:31:12.784 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013962445, 2024-03-19T11:31:12.784 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:12.784 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:12.784 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:12.784 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:31:12.784 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:31:12.784 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 35 2024-03-19T11:31:12.784 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.785 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:12.785 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.785 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:12.785 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071777+0000", 2024-03-19T11:31:12.785 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.785 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.785 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.785 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:12.785 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071777+0000", 2024-03-19T11:31:12.785 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.785 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.786 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.786 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:12.786 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071778+0000", 2024-03-19T11:31:12.786 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5000000000000001e-07 2024-03-19T11:31:12.786 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.786 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.786 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:12.786 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071917+0000", 2024-03-19T11:31:12.786 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013880199999999999 2024-03-19T11:31:12.786 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.786 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.786 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:12.786 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071931+0000", 2024-03-19T11:31:12.787 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4080000000000001e-05 2024-03-19T11:31:12.787 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.787 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.787 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:12.787 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071940+0000", 2024-03-19T11:31:12.787 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.363e-06 2024-03-19T11:31:12.787 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.787 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.787 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:12.787 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.080462+0000", 2024-03-19T11:31:12.787 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085221810000000002 2024-03-19T11:31:12.787 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.787 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.787 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:12.787 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.080582+0000", 2024-03-19T11:31:12.788 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011935299999999999 2024-03-19T11:31:12.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.788 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:12.788 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.080782+0000", 2024-03-19T11:31:12.788 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020077100000000001 2024-03-19T11:31:12.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.788 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:12.788 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.083401+0000", 2024-03-19T11:31:12.788 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002618333 2024-03-19T11:31:12.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.788 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:12.789 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085571+0000", 2024-03-19T11:31:12.789 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021702829999999998 2024-03-19T11:31:12.789 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.789 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.789 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:12.789 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085672+0000", 2024-03-19T11:31:12.789 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100876 2024-03-19T11:31:12.789 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.789 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.789 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:12.789 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085739+0000", 2024-03-19T11:31:12.789 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7552999999999995e-05 2024-03-19T11:31:12.789 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.789 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:12.790 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.790 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.790 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.790 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:31:12.790 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.071957+0000", 2024-03-19T11:31:12.790 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 87.705487032999997, 2024-03-19T11:31:12.790 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013989822000000001, 2024-03-19T11:31:12.790 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:12.790 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:12.790 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:12.790 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:31:12.790 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:31:12.790 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 36 2024-03-19T11:31:12.790 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.791 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:12.791 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.791 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:12.791 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071957+0000", 2024-03-19T11:31:12.791 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.791 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.791 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.791 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:12.791 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071957+0000", 2024-03-19T11:31:12.791 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.791 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.791 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.791 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:12.791 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071958+0000", 2024-03-19T11:31:12.791 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.06e-06 2024-03-19T11:31:12.792 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.792 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.792 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:12.792 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071965+0000", 2024-03-19T11:31:12.792 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8929999999999999e-06 2024-03-19T11:31:12.792 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.792 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.792 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:12.792 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071976+0000", 2024-03-19T11:31:12.793 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1046999999999999e-05 2024-03-19T11:31:12.793 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.793 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:12.793 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.071984+0000", 2024-03-19T11:31:12.793 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7170000000000008e-06 2024-03-19T11:31:12.793 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.793 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:12.793 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.081761+0000", 2024-03-19T11:31:12.794 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0097769310000000009 2024-03-19T11:31:12.794 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.794 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.794 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:12.794 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.081928+0000", 2024-03-19T11:31:12.794 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000167219 2024-03-19T11:31:12.794 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.794 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.794 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:12.794 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.082131+0000", 2024-03-19T11:31:12.794 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020244499999999999 2024-03-19T11:31:12.794 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.795 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:12.795 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.083455+0000", 2024-03-19T11:31:12.795 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001324073 2024-03-19T11:31:12.795 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.795 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:12.795 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085816+0000", 2024-03-19T11:31:12.795 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0023613309999999999 2024-03-19T11:31:12.795 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.796 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:12.796 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085892+0000", 2024-03-19T11:31:12.796 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6154000000000003e-05 2024-03-19T11:31:12.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.796 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.796 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:12.796 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.085947+0000", 2024-03-19T11:31:12.796 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4951999999999998e-05 2024-03-19T11:31:12.796 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.796 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:12.796 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.797 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.797 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.797 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:31:12.797 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.072000+0000", 2024-03-19T11:31:12.797 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 87.705444283000006, 2024-03-19T11:31:12.797 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014418268999999999, 2024-03-19T11:31:12.797 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:12.797 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:12.797 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:12.797 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:31:12.797 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:31:12.797 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 37 2024-03-19T11:31:12.798 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.798 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:12.798 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.798 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:12.798 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072000+0000", 2024-03-19T11:31:12.798 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.798 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.798 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.798 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:12.798 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072000+0000", 2024-03-19T11:31:12.798 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.799 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.799 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.799 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:12.799 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072001+0000", 2024-03-19T11:31:12.799 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0699999999999996e-07 2024-03-19T11:31:12.799 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.799 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.799 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:12.799 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072007+0000", 2024-03-19T11:31:12.799 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2060000000000004e-06 2024-03-19T11:31:12.799 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.800 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:12.800 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072023+0000", 2024-03-19T11:31:12.800 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6617e-05 2024-03-19T11:31:12.800 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.800 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:12.800 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.072031+0000", 2024-03-19T11:31:12.800 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8900000000000007e-06 2024-03-19T11:31:12.800 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.801 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:12.801 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.082682+0000", 2024-03-19T11:31:12.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010650868000000001 2024-03-19T11:31:12.801 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.801 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.801 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:12.801 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.082804+0000", 2024-03-19T11:31:12.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012202899999999999 2024-03-19T11:31:12.801 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.801 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.801 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:12.802 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.083006+0000", 2024-03-19T11:31:12.802 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020127799999999999 2024-03-19T11:31:12.802 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.802 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.802 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:12.802 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.084145+0000", 2024-03-19T11:31:12.802 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001139791 2024-03-19T11:31:12.802 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.802 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.802 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:12.802 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.086292+0000", 2024-03-19T11:31:12.803 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021461969999999999 2024-03-19T11:31:12.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.803 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:12.803 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.086364+0000", 2024-03-19T11:31:12.803 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2439000000000002e-05 2024-03-19T11:31:12.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.803 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:12.803 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.086418+0000", 2024-03-19T11:31:12.803 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4246999999999997e-05 2024-03-19T11:31:12.803 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.804 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:12.804 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.804 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.804 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.804 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:31:12.804 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.339898+0000", 2024-03-19T11:31:12.804 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 87.437546660999999, 2024-03-19T11:31:12.804 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019501793, 2024-03-19T11:31:12.804 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:12.804 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:12.804 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:12.804 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:31:12.804 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:31:12.805 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113 2024-03-19T11:31:12.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.805 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:12.805 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.805 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:12.805 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339898+0000", 2024-03-19T11:31:12.805 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.805 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.805 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:12.805 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339898+0000", 2024-03-19T11:31:12.805 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.805 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.805 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:12.805 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339899+0000", 2024-03-19T11:31:12.806 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.353e-06 2024-03-19T11:31:12.806 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.806 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.806 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:12.806 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339907+0000", 2024-03-19T11:31:12.806 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3000000000000002e-06 2024-03-19T11:31:12.806 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.806 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.806 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:12.806 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339924+0000", 2024-03-19T11:31:12.806 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6906999999999999e-05 2024-03-19T11:31:12.806 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.806 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.806 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:12.807 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339933+0000", 2024-03-19T11:31:12.807 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8969999999999997e-06 2024-03-19T11:31:12.807 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.807 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:12.807 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352239+0000", 2024-03-19T11:31:12.807 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012305814999999999 2024-03-19T11:31:12.807 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.807 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:12.807 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352466+0000", 2024-03-19T11:31:12.807 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022684100000000001 2024-03-19T11:31:12.807 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.808 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:12.808 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352816+0000", 2024-03-19T11:31:12.808 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00035030499999999997 2024-03-19T11:31:12.808 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.808 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.808 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:12.808 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.358956+0000", 2024-03-19T11:31:12.808 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061400079999999998 2024-03-19T11:31:12.808 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.808 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.808 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:12.808 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359118+0000", 2024-03-19T11:31:12.808 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016237599999999999 2024-03-19T11:31:12.808 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.808 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.809 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:12.809 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359287+0000", 2024-03-19T11:31:12.809 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000168488 2024-03-19T11:31:12.809 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.809 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.809 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:12.809 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359399+0000", 2024-03-19T11:31:12.809 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000112503 2024-03-19T11:31:12.809 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.809 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:12.809 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.809 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.809 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.809 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:31:12.809 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.339947+0000", 2024-03-19T11:31:12.810 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 87.437496894000006, 2024-03-19T11:31:12.810 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021923069999999999, 2024-03-19T11:31:12.810 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:12.810 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:12.810 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:12.810 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:31:12.810 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:31:12.810 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 114 2024-03-19T11:31:12.810 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.810 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:12.810 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.810 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:12.810 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339947+0000", 2024-03-19T11:31:12.810 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.811 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:12.811 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339947+0000", 2024-03-19T11:31:12.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.811 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:12.811 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339948+0000", 2024-03-19T11:31:12.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6000000000000003e-07 2024-03-19T11:31:12.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.811 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:12.811 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339956+0000", 2024-03-19T11:31:12.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1559999999999995e-06 2024-03-19T11:31:12.812 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.812 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.812 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:12.812 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339967+0000", 2024-03-19T11:31:12.812 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0669999999999999e-05 2024-03-19T11:31:12.812 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.812 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.812 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:12.812 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339975+0000", 2024-03-19T11:31:12.812 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0099999999999995e-06 2024-03-19T11:31:12.812 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.812 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.812 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:12.812 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.353672+0000", 2024-03-19T11:31:12.812 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013696583999999999 2024-03-19T11:31:12.813 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.813 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.813 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:12.813 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.353808+0000", 2024-03-19T11:31:12.813 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013652299999999999 2024-03-19T11:31:12.813 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.813 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.813 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:12.813 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.354036+0000", 2024-03-19T11:31:12.813 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022744799999999999 2024-03-19T11:31:12.813 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.813 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.813 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:12.813 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.360480+0000", 2024-03-19T11:31:12.814 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0064441630000000001 2024-03-19T11:31:12.814 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.814 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.814 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:12.814 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361722+0000", 2024-03-19T11:31:12.814 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001242754 2024-03-19T11:31:12.814 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.814 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.814 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:12.814 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361807+0000", 2024-03-19T11:31:12.815 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4605999999999996e-05 2024-03-19T11:31:12.815 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.815 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.815 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:12.815 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361870+0000", 2024-03-19T11:31:12.815 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3396000000000001e-05 2024-03-19T11:31:12.815 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.815 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:12.815 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.815 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.815 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.815 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:31:12.815 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.340017+0000", 2024-03-19T11:31:12.815 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 87.437427577999998, 2024-03-19T11:31:12.815 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022169355000000002, 2024-03-19T11:31:12.816 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:12.816 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:12.816 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:12.816 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:31:12.816 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:31:12.816 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 115 2024-03-19T11:31:12.816 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.816 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:12.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.817 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:12.817 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340017+0000", 2024-03-19T11:31:12.817 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.817 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.817 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.817 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:12.817 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340017+0000", 2024-03-19T11:31:12.817 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.817 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.817 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.817 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:12.817 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340018+0000", 2024-03-19T11:31:12.817 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.043e-06 2024-03-19T11:31:12.817 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.817 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.818 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:12.818 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340139+0000", 2024-03-19T11:31:12.818 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121502 2024-03-19T11:31:12.818 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.818 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.818 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:12.818 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340157+0000", 2024-03-19T11:31:12.818 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7382999999999999e-05 2024-03-19T11:31:12.818 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.819 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:12.819 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340166+0000", 2024-03-19T11:31:12.819 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2469999999999997e-06 2024-03-19T11:31:12.819 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.819 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:12.819 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.357470+0000", 2024-03-19T11:31:12.819 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017304119999999999 2024-03-19T11:31:12.819 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.819 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:12.819 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.357671+0000", 2024-03-19T11:31:12.819 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000200798 2024-03-19T11:31:12.820 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.820 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.820 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:12.820 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.358002+0000", 2024-03-19T11:31:12.820 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033090099999999998 2024-03-19T11:31:12.820 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.820 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.820 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:12.820 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361892+0000", 2024-03-19T11:31:12.820 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00389018 2024-03-19T11:31:12.820 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.821 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.821 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:12.821 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361983+0000", 2024-03-19T11:31:12.821 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0976000000000004e-05 2024-03-19T11:31:12.821 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.821 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.821 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:12.821 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362071+0000", 2024-03-19T11:31:12.821 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8022999999999994e-05 2024-03-19T11:31:12.821 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.821 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.821 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:12.821 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362186+0000", 2024-03-19T11:31:12.822 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115182 2024-03-19T11:31:12.822 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.822 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:12.822 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.822 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.822 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.822 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:31:12.822 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.340183+0000", 2024-03-19T11:31:12.822 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 87.437260746000007, 2024-03-19T11:31:12.822 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022279841000000002, 2024-03-19T11:31:12.822 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:12.822 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:12.822 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:12.822 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:31:12.822 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:31:12.823 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 116 2024-03-19T11:31:12.823 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.823 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:12.823 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.823 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:12.823 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340183+0000", 2024-03-19T11:31:12.823 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.823 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.823 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.823 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:12.823 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340183+0000", 2024-03-19T11:31:12.823 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.823 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.823 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.823 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:12.824 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340184+0000", 2024-03-19T11:31:12.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9000000000000005e-07 2024-03-19T11:31:12.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.824 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:12.824 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340191+0000", 2024-03-19T11:31:12.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0360000000000003e-06 2024-03-19T11:31:12.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.824 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:12.824 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340201+0000", 2024-03-19T11:31:12.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0844e-05 2024-03-19T11:31:12.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.824 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:12.825 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340209+0000", 2024-03-19T11:31:12.825 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7360000000000001e-06 2024-03-19T11:31:12.825 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.825 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.825 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:12.825 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359535+0000", 2024-03-19T11:31:12.825 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019325546999999998 2024-03-19T11:31:12.825 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.825 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.825 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:12.825 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359672+0000", 2024-03-19T11:31:12.825 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013706600000000001 2024-03-19T11:31:12.825 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.825 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.826 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:12.826 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359905+0000", 2024-03-19T11:31:12.826 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023334500000000001 2024-03-19T11:31:12.826 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.826 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.826 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:12.826 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362306+0000", 2024-03-19T11:31:12.826 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002401041 2024-03-19T11:31:12.826 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.826 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.826 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:12.826 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362329+0000", 2024-03-19T11:31:12.826 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3283999999999999e-05 2024-03-19T11:31:12.826 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.826 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.827 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:12.827 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362402+0000", 2024-03-19T11:31:12.827 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2459000000000005e-05 2024-03-19T11:31:12.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.827 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.827 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:12.827 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362463+0000", 2024-03-19T11:31:12.827 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1493000000000002e-05 2024-03-19T11:31:12.827 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.827 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:12.827 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.827 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.828 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:31:12.828 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.340222+0000", 2024-03-19T11:31:12.828 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 87.437222163000001, 2024-03-19T11:31:12.828 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022500218999999998, 2024-03-19T11:31:12.828 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:12.828 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:12.828 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:12.828 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:31:12.828 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:31:12.828 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 117 2024-03-19T11:31:12.828 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.828 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:12.828 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.829 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:12.829 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340222+0000", 2024-03-19T11:31:12.829 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.829 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.829 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.829 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:12.829 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340222+0000", 2024-03-19T11:31:12.829 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.829 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.829 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.829 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:12.829 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340223+0000", 2024-03-19T11:31:12.829 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7999999999999995e-07 2024-03-19T11:31:12.829 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.830 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.830 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:12.830 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340229+0000", 2024-03-19T11:31:12.830 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0399999999999998e-06 2024-03-19T11:31:12.830 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.830 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.830 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:12.830 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340239+0000", 2024-03-19T11:31:12.830 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0190000000000001e-05 2024-03-19T11:31:12.830 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.830 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.830 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:12.830 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340250+0000", 2024-03-19T11:31:12.831 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0720000000000001e-05 2024-03-19T11:31:12.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.831 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:12.831 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.360587+0000", 2024-03-19T11:31:12.831 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020337542 2024-03-19T11:31:12.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.831 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:12.831 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.360818+0000", 2024-03-19T11:31:12.831 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023120500000000001 2024-03-19T11:31:12.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.831 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:12.832 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361218+0000", 2024-03-19T11:31:12.832 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000399637 2024-03-19T11:31:12.832 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.832 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:12.832 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362483+0000", 2024-03-19T11:31:12.832 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012649200000000001 2024-03-19T11:31:12.832 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.832 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:12.832 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362538+0000", 2024-03-19T11:31:12.832 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4490000000000002e-05 2024-03-19T11:31:12.833 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.833 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.833 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:12.833 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362617+0000", 2024-03-19T11:31:12.833 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9839000000000006e-05 2024-03-19T11:31:12.833 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.833 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.833 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:12.833 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362722+0000", 2024-03-19T11:31:12.833 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104956 2024-03-19T11:31:12.833 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.833 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:12.833 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.834 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.834 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.834 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1047 4.8 e35/21 4:1d58db11:::smithi03829856-18:head v 35'32, mlcod=33'29)", 2024-03-19T11:31:12.834 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:13.225023+0000", 2024-03-19T11:31:12.834 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 59.552421377999998, 2024-03-19T11:31:12.834 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013059688999999999, 2024-03-19T11:31:12.834 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:12.834 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:12.834 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:12.834 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.834 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:12.834 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.225023+0000", 2024-03-19T11:31:12.834 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.834 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.835 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.835 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:12.835 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.225023+0000", 2024-03-19T11:31:12.835 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.835 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.835 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.835 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:12.835 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.225025+0000", 2024-03-19T11:31:12.835 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.17e-06 2024-03-19T11:31:12.835 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.835 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.835 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:12.836 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.225771+0000", 2024-03-19T11:31:12.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000746267 2024-03-19T11:31:12.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.836 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:12.836 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.225807+0000", 2024-03-19T11:31:12.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5487000000000001e-05 2024-03-19T11:31:12.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.836 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:12.836 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.225823+0000", 2024-03-19T11:31:12.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6215999999999999e-05 2024-03-19T11:31:12.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.837 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:12.837 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.225990+0000", 2024-03-19T11:31:12.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016680899999999999 2024-03-19T11:31:12.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.837 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:12.837 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.226058+0000", 2024-03-19T11:31:12.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8672999999999998e-05 2024-03-19T11:31:12.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.837 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:12.837 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.237940+0000", 2024-03-19T11:31:12.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011881311 2024-03-19T11:31:12.838 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.838 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.838 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:12.838 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.238083+0000", 2024-03-19T11:31:12.838 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014275600000000001 2024-03-19T11:31:12.838 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.838 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:12.838 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.838 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.838 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.838 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1067 4.0 e36/21 4:01018511:::smithi03829856-19:head v 36'18, mlcod=35'14)", 2024-03-19T11:31:12.838 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:13.952492+0000", 2024-03-19T11:31:12.838 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 58.824951921, 2024-03-19T11:31:12.838 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013554725, 2024-03-19T11:31:12.839 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:12.839 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:12.839 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:12.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.839 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:12.839 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.952492+0000", 2024-03-19T11:31:12.839 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.839 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.839 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:12.839 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.952492+0000", 2024-03-19T11:31:12.839 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.839 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.840 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:12.840 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.952493+0000", 2024-03-19T11:31:12.840 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9299999999999997e-07 2024-03-19T11:31:12.840 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.840 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.840 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:12.840 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.954109+0000", 2024-03-19T11:31:12.840 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0016163569999999999 2024-03-19T11:31:12.840 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.840 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.840 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:12.840 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.954135+0000", 2024-03-19T11:31:12.840 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5352999999999999e-05 2024-03-19T11:31:12.840 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.840 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.841 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:12.841 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.954149+0000", 2024-03-19T11:31:12.841 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3879999999999999e-05 2024-03-19T11:31:12.841 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.841 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.841 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:12.841 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.954288+0000", 2024-03-19T11:31:12.841 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000139476 2024-03-19T11:31:12.841 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.841 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.841 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:12.841 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.954356+0000", 2024-03-19T11:31:12.841 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7533000000000005e-05 2024-03-19T11:31:12.841 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.841 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.842 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:12.842 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.965391+0000", 2024-03-19T11:31:12.842 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011035764999999999 2024-03-19T11:31:12.842 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.842 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.842 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:12.842 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:13.966047+0000", 2024-03-19T11:31:12.842 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00065556800000000004 2024-03-19T11:31:12.842 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.842 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:12.842 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.842 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.842 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.842 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1122 4.8 e36/21 4:1d58db11:::smithi03829856-18:head v 36'42, mlcod=36'39)", 2024-03-19T11:31:12.843 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:14.172666+0000", 2024-03-19T11:31:12.843 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 58.604778146999998, 2024-03-19T11:31:12.843 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018015000999999999, 2024-03-19T11:31:12.843 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:12.843 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:12.843 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:12.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.843 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:12.843 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.172666+0000", 2024-03-19T11:31:12.843 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.843 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.843 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:12.843 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.172666+0000", 2024-03-19T11:31:12.844 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.844 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.844 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.844 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:12.844 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.172669+0000", 2024-03-19T11:31:12.844 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1470000000000001e-06 2024-03-19T11:31:12.844 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.844 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.844 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:12.844 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.173140+0000", 2024-03-19T11:31:12.844 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00047074200000000001 2024-03-19T11:31:12.844 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.844 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.844 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:12.844 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.173188+0000", 2024-03-19T11:31:12.845 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8287e-05 2024-03-19T11:31:12.845 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.845 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.845 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:12.845 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.173211+0000", 2024-03-19T11:31:12.845 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3023e-05 2024-03-19T11:31:12.845 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.845 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.845 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:12.845 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.173378+0000", 2024-03-19T11:31:12.845 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000166742 2024-03-19T11:31:12.845 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.845 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.845 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:12.846 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.173557+0000", 2024-03-19T11:31:12.846 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000178569 2024-03-19T11:31:12.846 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.846 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.846 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:12.846 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.190416+0000", 2024-03-19T11:31:12.846 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016859570000000001 2024-03-19T11:31:12.846 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.846 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.846 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:12.846 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.190681+0000", 2024-03-19T11:31:12.846 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026492099999999999 2024-03-19T11:31:12.846 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.846 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:12.847 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.847 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.847 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.847 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.5.0:43 4.8 e39/21 4:15708482:::smithi03829856-23:head v 39'50, mlcod=38'47)", 2024-03-19T11:31:12.847 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.842819+0000", 2024-03-19T11:31:12.847 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 55.934625644, 2024-03-19T11:31:12.847 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017626513, 2024-03-19T11:31:12.847 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:12.847 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:12.847 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:12.847 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.847 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:12.847 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842819+0000", 2024-03-19T11:31:12.847 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.847 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.848 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.848 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:12.848 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842819+0000", 2024-03-19T11:31:12.848 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.848 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.848 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.848 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:12.848 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842820+0000", 2024-03-19T11:31:12.848 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.39e-06 2024-03-19T11:31:12.848 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.849 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.849 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:12.849 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.846437+0000", 2024-03-19T11:31:12.849 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003616953 2024-03-19T11:31:12.849 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.849 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.850 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:12.850 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.846471+0000", 2024-03-19T11:31:12.850 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4133e-05 2024-03-19T11:31:12.850 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.850 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.850 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:12.850 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.846539+0000", 2024-03-19T11:31:12.850 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8072999999999997e-05 2024-03-19T11:31:12.850 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.850 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.850 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:12.850 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.847094+0000", 2024-03-19T11:31:12.850 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00055472099999999999 2024-03-19T11:31:12.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.851 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.851 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:12.851 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.847128+0000", 2024-03-19T11:31:12.851 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3875999999999999e-05 2024-03-19T11:31:12.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.851 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.851 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:12.851 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.860308+0000", 2024-03-19T11:31:12.851 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013180712000000001 2024-03-19T11:31:12.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.852 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.852 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:12.852 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.860445+0000", 2024-03-19T11:31:12.852 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013665500000000001 2024-03-19T11:31:12.852 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.852 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:12.852 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.852 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.852 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1209 4.8 e41/21 4:1d58db11:::smithi03829856-18:head v 41'53, mlcod=39'50)", 2024-03-19T11:31:12.852 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:18.124878+0000", 2024-03-19T11:31:12.852 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 54.652565951, 2024-03-19T11:31:12.852 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016809343000000001, 2024-03-19T11:31:12.853 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:12.853 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:12.853 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:12.853 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.853 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:12.853 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.124878+0000", 2024-03-19T11:31:12.853 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.853 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.853 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.853 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:12.853 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.124878+0000", 2024-03-19T11:31:12.854 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.854 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.854 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.854 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:12.854 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.124880+0000", 2024-03-19T11:31:12.854 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4959999999999999e-06 2024-03-19T11:31:12.854 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.854 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.854 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:12.854 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125845+0000", 2024-03-19T11:31:12.854 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00096535299999999996 2024-03-19T11:31:12.854 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.854 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.855 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:12.855 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125884+0000", 2024-03-19T11:31:12.855 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8645e-05 2024-03-19T11:31:12.855 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.855 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.855 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:12.855 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.125909+0000", 2024-03-19T11:31:12.855 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4783e-05 2024-03-19T11:31:12.855 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.855 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.855 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:12.855 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.126085+0000", 2024-03-19T11:31:12.855 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000176649 2024-03-19T11:31:12.856 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.856 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.856 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:12.857 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.126150+0000", 2024-03-19T11:31:12.857 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4290999999999998e-05 2024-03-19T11:31:12.857 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.857 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.857 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:12.857 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.141552+0000", 2024-03-19T11:31:12.858 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015402295 2024-03-19T11:31:12.858 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.858 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.858 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:12.858 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.141688+0000", 2024-03-19T11:31:12.858 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013583099999999999 2024-03-19T11:31:12.858 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.858 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:12.858 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.858 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.858 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.858 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.15 51/50 [PushOp(4:afa9f208:::smithi03829856-35:head, version: 28'46, data_included: [571438~623878,1806623~424698], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:afa9f208:::smithi03829856-35:head@28'46, size: 2571438, copy_subset: [(0, 2571438)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2231321, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:af9d0d56:::smithi03829856-20:head, version: 35'55, data_included: [655360~589824,1900544~418789], data_size: 1008613, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:af9d0d56:::smithi03829856-20:head@35'55, size: 2319333, copy_subset: [(0, 2319333)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2319333, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ad3ad1b3:::smithi03829856-21:b, version: 47'66, data_included: [736140~537686,2020581~315559], data_size: 853245, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:ad3ad1b3:::smithi03829856-21:b@47'66, size: 2336140, copy_subset: [(0, 2336140)], clone_subset: {}, snapset: b=[]:{b: [b, a, 6, 5]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2336140, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ad3ad1b3:::smithi03829856-21:head, version: 47'70, data_included: [531121~742705,1711057~305871], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:ad3ad1b3:::smithi03829856-21:head@47'70, size: 3731121, copy_subset: [(0, 3731121)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2016928, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ac9cd3a3:::smithi03829856-8:head, version: 49'72, data_included: [517649~620182,1776668~428394], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:ac9cd3a3:::smithi03829856-8:head@49'72, size: 3008420, copy_subset: [(0, 3008420)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2205062, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:aa3303a5:::smithi03829856-17:b, version: 49'73, data_included: [614400~692224,1903006~356352], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:aa3303a5:::smithi03829856-17:b@49'73, size: 2268912, copy_subset: [(0, 2268912)], clone_subset: {}, snapset: b=[]:{b: [b, a, 6, 5]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2259358, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:aa3303a5:::smithi03829856-17:head, version: 49'74, data_included: [413472~893152,1818036~155424], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:aa3303a5:::smithi03829856-17:head@49'74, size: 3613472, copy_subset: [(0, 3613472)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1973460, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:12.858 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:28.058610+0000", 2024-03-19T11:31:12.858 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 44.718833990999997, 2024-03-19T11:31:12.859 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027100032, 2024-03-19T11:31:12.859 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:12.859 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:12.859 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:12.859 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.859 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:12.859 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.058610+0000", 2024-03-19T11:31:12.859 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.859 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.859 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.859 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:12.859 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.058610+0000", 2024-03-19T11:31:12.859 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.859 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.859 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.860 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:12.860 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.058611+0000", 2024-03-19T11:31:12.860 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1599999999999999e-06 2024-03-19T11:31:12.860 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.860 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.860 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:12.860 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.063874+0000", 2024-03-19T11:31:12.860 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052625270000000004 2024-03-19T11:31:12.860 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.860 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.860 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:12.860 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.063935+0000", 2024-03-19T11:31:12.860 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0594000000000003e-05 2024-03-19T11:31:12.860 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.860 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.861 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:12.861 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064024+0000", 2024-03-19T11:31:12.861 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9273999999999999e-05 2024-03-19T11:31:12.861 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.861 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.861 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:12.861 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064385+0000", 2024-03-19T11:31:12.861 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00036129100000000002 2024-03-19T11:31:12.861 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.861 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.861 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:12.861 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064437+0000", 2024-03-19T11:31:12.861 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2346000000000003e-05 2024-03-19T11:31:12.861 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.862 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.862 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:12.862 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085710+0000", 2024-03-19T11:31:12.862 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021272840000000001 2024-03-19T11:31:12.862 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.862 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:12.862 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.862 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.862 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.862 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.15 e51,50 4:ac9cd3a3:::smithi03829856-8:4,49'71)", 2024-03-19T11:31:12.862 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:28.064112+0000", 2024-03-19T11:31:12.862 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 44.713332164999997, 2024-03-19T11:31:12.862 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026480417999999999, 2024-03-19T11:31:12.862 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:12.862 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:12.863 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:12.863 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.863 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:12.863 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064112+0000", 2024-03-19T11:31:12.863 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.863 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.863 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.863 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:12.863 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064112+0000", 2024-03-19T11:31:12.863 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.863 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.863 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.863 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:12.863 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064113+0000", 2024-03-19T11:31:12.863 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3600000000000003e-07 2024-03-19T11:31:12.864 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.864 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.864 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:12.864 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064118+0000", 2024-03-19T11:31:12.864 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0549999999999999e-06 2024-03-19T11:31:12.864 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.864 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.864 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:12.864 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064128+0000", 2024-03-19T11:31:12.864 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0349e-05 2024-03-19T11:31:12.864 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.864 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.864 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:12.864 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064136+0000", 2024-03-19T11:31:12.864 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6249999999999998e-06 2024-03-19T11:31:12.865 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.865 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.865 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:12.865 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085773+0000", 2024-03-19T11:31:12.865 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021636829999999999 2024-03-19T11:31:12.865 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.865 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.865 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:12.865 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085788+0000", 2024-03-19T11:31:12.865 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5512999999999999e-05 2024-03-19T11:31:12.865 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.865 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.865 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:12.865 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.090592+0000", 2024-03-19T11:31:12.865 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0048044100000000003 2024-03-19T11:31:12.866 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.866 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:12.866 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.866 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.866 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.866 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.5 51/50 [PushOp(4:a38b9453:::smithi03829856-37:head, version: 28'45, data_included: [605052~774376,1923587~274200], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:a38b9453:::smithi03829856-37:head@28'45, size: 2605052, copy_subset: [(0, 2605052)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2197787, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:a0a50d04:::smithi03829856-13:b, version: 47'63, data_included: [601684~619341,1669078~429235], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a0a50d04:::smithi03829856-13:b@47'63, size: 2201684, copy_subset: [(0, 2201684)], clone_subset: {}, snapset: b=[]:{b: [b, a, 6, 5]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2098313, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:12.866 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:28.064324+0000", 2024-03-19T11:31:12.866 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 44.713120320999998, 2024-03-19T11:31:12.866 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027179020000000002, 2024-03-19T11:31:12.866 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:12.866 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:12.866 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:12.867 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.867 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:12.867 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064324+0000", 2024-03-19T11:31:12.867 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.867 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.867 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.867 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:12.867 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064324+0000", 2024-03-19T11:31:12.867 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.867 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.867 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.867 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:12.867 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064324+0000", 2024-03-19T11:31:12.868 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1199999999999998e-07 2024-03-19T11:31:12.868 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.868 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.868 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:12.868 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.065527+0000", 2024-03-19T11:31:12.868 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012028939999999999 2024-03-19T11:31:12.868 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.868 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.868 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:12.868 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.065552+0000", 2024-03-19T11:31:12.868 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4440999999999999e-05 2024-03-19T11:31:12.868 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.868 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.868 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:12.869 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.065568+0000", 2024-03-19T11:31:12.869 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6498e-05 2024-03-19T11:31:12.869 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.869 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.869 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:12.869 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085812+0000", 2024-03-19T11:31:12.869 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020243954000000002 2024-03-19T11:31:12.869 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.869 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.869 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:12.869 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085825+0000", 2024-03-19T11:31:12.869 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3298999999999999e-05 2024-03-19T11:31:12.869 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.870 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.870 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:12.870 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.091503+0000", 2024-03-19T11:31:12.870 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.005677522 2024-03-19T11:31:12.870 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.870 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:12.870 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.870 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.870 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.870 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.24 75/74 [PushOp(4:27909373:::smithi03829856-46:8, version: 46'59, data_included: [756315~767970,1933572~280606], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:8@46'59, size: 2756315, copy_subset: [(0, 2756315)], clone_subset: {}, snapset: 10=[]:{8: [6], c: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2214178, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:266590a5:::smithi03829856-16:e, version: 70'107, data_included: [652105~662359,1728171~386217], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:e@70'107, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 10=[]:{e: [d, b, 6]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:266590a5:::smithi03829856-16:head, version: 70'108, data_included: [652105~662359,1728171~386217], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:head@70'108, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:27909373:::smithi03829856-46:c, version: 70'109, data_included: [628595~895690,1807440~152886], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:c@70'109, size: 3428595, copy_subset: [(0, 3428595)], clone_subset: {}, snapset: 10=[]:{8: [6], c: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1960326, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:27909373:::smithi03829856-46:head, version: 70'110, data_included: [403401~454961,1584925~506039], data_size: 961000, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:head@70'110, size: 2403401, copy_subset: [(0, 2403401)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2403401, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:12.870 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.099172+0000", 2024-03-19T11:31:12.870 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.678271887000001, 2024-03-19T11:31:12.870 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.024528580000000001, 2024-03-19T11:31:12.870 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:12.871 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:12.871 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:12.871 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.871 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:12.871 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.099172+0000", 2024-03-19T11:31:12.871 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.871 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.871 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.871 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:12.871 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.099172+0000", 2024-03-19T11:31:12.871 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.871 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.871 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.872 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:12.872 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.099175+0000", 2024-03-19T11:31:12.872 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7750000000000001e-06 2024-03-19T11:31:12.872 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.872 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.872 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:12.872 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105219+0000", 2024-03-19T11:31:12.872 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0060440789999999999 2024-03-19T11:31:12.872 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.872 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.872 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:12.872 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105285+0000", 2024-03-19T11:31:12.872 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6260000000000006e-05 2024-03-19T11:31:12.872 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.872 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.873 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:12.873 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105366+0000", 2024-03-19T11:31:12.873 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.038e-05 2024-03-19T11:31:12.873 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.873 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.873 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:12.873 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105637+0000", 2024-03-19T11:31:12.873 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027085199999999997 2024-03-19T11:31:12.873 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.873 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.873 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:12.873 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105679+0000", 2024-03-19T11:31:12.873 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2088000000000001e-05 2024-03-19T11:31:12.874 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.874 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.874 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:12.874 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.123701+0000", 2024-03-19T11:31:12.874 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018022145999999999 2024-03-19T11:31:12.874 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.874 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:12.874 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.874 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.874 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.874 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.24 e75,74 4:266590a5:::smithi03829856-16:4,47'63,4:27909373:::smithi03829856-46:5,55'79,4:266590a5:::smithi03829856-16:10,67'105)", 2024-03-19T11:31:12.874 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.105437+0000", 2024-03-19T11:31:12.874 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.672006883999998, 2024-03-19T11:31:12.874 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.01860726, 2024-03-19T11:31:12.875 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:12.875 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:12.875 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:12.875 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.875 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:12.875 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105437+0000", 2024-03-19T11:31:12.875 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.875 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.875 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.875 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:12.875 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105437+0000", 2024-03-19T11:31:12.875 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:12.876 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.876 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.876 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:12.876 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105438+0000", 2024-03-19T11:31:12.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6799999999999999e-07 2024-03-19T11:31:12.876 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.876 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.876 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:12.876 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105445+0000", 2024-03-19T11:31:12.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3749999999999999e-06 2024-03-19T11:31:12.876 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.876 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.876 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:12.876 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105454+0000", 2024-03-19T11:31:12.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7909999999999999e-06 2024-03-19T11:31:12.877 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.877 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.877 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:12.877 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105465+0000", 2024-03-19T11:31:12.877 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.045e-05 2024-03-19T11:31:12.877 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.877 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.877 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:12.877 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.123753+0000", 2024-03-19T11:31:12.877 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0182884 2024-03-19T11:31:12.877 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.877 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.877 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:12.878 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.123765+0000", 2024-03-19T11:31:12.878 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2143e-05 2024-03-19T11:31:12.878 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:12.878 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:12.878 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:12.878 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.124045+0000", 2024-03-19T11:31:12.878 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027923299999999999 2024-03-19T11:31:12.878 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.878 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:12.878 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.878 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:12.878 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:12.878 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:12.879 INFO:tasks.rados.rados.0.smithi038.stdout:1308: finishing set_chunk to oid 33 2024-03-19T11:31:12.879 INFO:tasks.rados.rados.0.smithi038.stdout:1309: expect (ObjNum 192 snap 22 seq_num 192) 2024-03-19T11:31:12.880 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-19T11:31:12.881 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:12.889+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:12.980 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:12.989+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:12.994 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:12.995 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:31:12.995 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:31:12.995 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:13.007 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-19T11:31:13.044 INFO:tasks.rados.rados.0.smithi038.stdout:1303: done (7 left) 2024-03-19T11:31:13.045 INFO:tasks.rados.rados.0.smithi038.stdout:1304: done (6 left) 2024-03-19T11:31:13.045 INFO:tasks.rados.rados.0.smithi038.stdout:1305: done (5 left) 2024-03-19T11:31:13.045 INFO:tasks.rados.rados.0.smithi038.stdout:1306: done (4 left) 2024-03-19T11:31:13.045 INFO:tasks.rados.rados.0.smithi038.stdout:1307: done (3 left) 2024-03-19T11:31:13.045 INFO:tasks.rados.rados.0.smithi038.stdout:1308: done (2 left) 2024-03-19T11:31:13.045 INFO:tasks.rados.rados.0.smithi038.stdout:1309: done (1 left) 2024-03-19T11:31:13.045 INFO:tasks.rados.rados.0.smithi038.stdout:1310: done (0 left) 2024-03-19T11:31:13.045 INFO:tasks.rados.rados.0.smithi038.stdout:1311: snap_remove snap 23 2024-03-19T11:31:13.046 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-03-19T11:31:13.112 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:13.121+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:13.156 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:13.156 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:31:13.156 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:31:13.156 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:31:13.156 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:13.169 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-19T11:31:13.182 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:13.191+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:13.283 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:13.295+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:13.339 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:13.339 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:31:13.339 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:31:13.339 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:31:13.339 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.340 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1217 4.3 4:c8b09e87:::smithi03829856-15:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:31:13.340 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.155966+0000", 2024-03-19T11:31:13.340 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 55.190881754999999, 2024-03-19T11:31:13.340 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.023984647000000001, 2024-03-19T11:31:13.340 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:13.340 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:13.340 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:13.340 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:13.340 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:13.340 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1217 2024-03-19T11:31:13.340 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.340 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:13.340 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.340 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:13.340 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155966+0000", 2024-03-19T11:31:13.341 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.341 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.341 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.341 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:13.341 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155966+0000", 2024-03-19T11:31:13.341 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.341 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.341 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.341 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:13.341 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155968+0000", 2024-03-19T11:31:13.341 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.795e-06 2024-03-19T11:31:13.341 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.341 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.341 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:13.341 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155978+0000", 2024-03-19T11:31:13.342 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0574000000000001e-05 2024-03-19T11:31:13.342 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.342 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.342 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:13.342 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.156000+0000", 2024-03-19T11:31:13.342 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1250000000000002e-05 2024-03-19T11:31:13.342 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.342 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.342 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:13.342 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.156010+0000", 2024-03-19T11:31:13.342 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0168e-05 2024-03-19T11:31:13.342 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.342 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.342 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.343 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.156119+0000", 2024-03-19T11:31:13.343 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010950699999999999 2024-03-19T11:31:13.343 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.343 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.343 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:13.343 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.156231+0000", 2024-03-19T11:31:13.343 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000111445 2024-03-19T11:31:13.343 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.343 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.343 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:13.343 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.179951+0000", 2024-03-19T11:31:13.343 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.023719908000000001 2024-03-19T11:31:13.343 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.344 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:13.344 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.344 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.344 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.344 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.24 75/74 [PushOp(4:27909373:::smithi03829856-46:8, version: 46'59, data_included: [756315~767970,1933572~280606], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:8@46'59, size: 2756315, copy_subset: [(0, 2756315)], clone_subset: {}, snapset: 10=[]:{8: [6], c: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2214178, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:266590a5:::smithi03829856-16:e, version: 70'107, data_included: [652105~662359,1728171~386217], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:e@70'107, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 10=[]:{e: [d, b, 6]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:266590a5:::smithi03829856-16:head, version: 70'108, data_included: [652105~662359,1728171~386217], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:head@70'108, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:27909373:::smithi03829856-46:c, version: 70'109, data_included: [628595~895690,1807440~152886], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:c@70'109, size: 3428595, copy_subset: [(0, 3428595)], clone_subset: {}, snapset: 10=[]:{8: [6], c: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1960326, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:27909373:::smithi03829856-46:head, version: 70'110, data_included: [403401~454961,1584925~506039], data_size: 961000, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:head@70'110, size: 2403401, copy_subset: [(0, 2403401)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2403401, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:13.344 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.099395+0000", 2024-03-19T11:31:13.344 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 19.247452593999999, 2024-03-19T11:31:13.344 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.045391157000000001, 2024-03-19T11:31:13.344 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:13.344 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:13.344 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:13.344 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.344 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:13.344 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.099395+0000", 2024-03-19T11:31:13.344 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.345 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.345 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.345 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:13.345 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.099395+0000", 2024-03-19T11:31:13.345 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.345 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.345 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.345 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:13.345 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.099397+0000", 2024-03-19T11:31:13.345 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.691e-06 2024-03-19T11:31:13.345 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.345 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.345 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:13.345 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105009+0000", 2024-03-19T11:31:13.345 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0056121349999999999 2024-03-19T11:31:13.346 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.346 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.346 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:13.346 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105077+0000", 2024-03-19T11:31:13.346 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8561999999999998e-05 2024-03-19T11:31:13.346 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.346 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.346 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:13.346 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105133+0000", 2024-03-19T11:31:13.346 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5590000000000001e-05 2024-03-19T11:31:13.346 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.346 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.346 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.346 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105531+0000", 2024-03-19T11:31:13.346 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00039756200000000001 2024-03-19T11:31:13.347 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.347 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.347 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:13.347 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105579+0000", 2024-03-19T11:31:13.347 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8028999999999997e-05 2024-03-19T11:31:13.347 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.347 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.347 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:13.347 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.144786+0000", 2024-03-19T11:31:13.347 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.039207588000000002 2024-03-19T11:31:13.347 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.347 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:13.347 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.347 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.347 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.348 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.24 e75,74 4:266590a5:::smithi03829856-16:4,47'63,4:27909373:::smithi03829856-46:5,55'79,4:266590a5:::smithi03829856-16:10,67'105)", 2024-03-19T11:31:13.348 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.105191+0000", 2024-03-19T11:31:13.348 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 19.241656494000001, 2024-03-19T11:31:13.348 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.039580830999999997, 2024-03-19T11:31:13.348 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:13.348 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:13.348 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:13.348 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.348 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:13.348 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105191+0000", 2024-03-19T11:31:13.348 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.348 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.348 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.348 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:13.349 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105191+0000", 2024-03-19T11:31:13.349 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.349 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.349 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.349 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:13.349 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105192+0000", 2024-03-19T11:31:13.349 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.099e-06 2024-03-19T11:31:13.349 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.349 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.349 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:13.349 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105200+0000", 2024-03-19T11:31:13.349 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5279999999999998e-06 2024-03-19T11:31:13.349 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.349 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.349 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:13.350 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105211+0000", 2024-03-19T11:31:13.350 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1471000000000001e-05 2024-03-19T11:31:13.350 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.350 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.350 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:13.350 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105220+0000", 2024-03-19T11:31:13.350 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3469999999999999e-06 2024-03-19T11:31:13.350 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.350 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.350 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.350 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.144219+0000", 2024-03-19T11:31:13.350 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.038999024 2024-03-19T11:31:13.350 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.350 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.351 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:13.351 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.144257+0000", 2024-03-19T11:31:13.351 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8501999999999998e-05 2024-03-19T11:31:13.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.351 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.351 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:13.351 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.144772+0000", 2024-03-19T11:31:13.351 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00051486000000000001 2024-03-19T11:31:13.351 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.351 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:13.351 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.351 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.351 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 75/74 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:7, version: 55'82, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@55'82, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 58'86, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@58'86, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [504210~968657,1572695~79919], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:12, version: 66'101, data_included: [440350~816629,1309629~231947], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:12@66'101, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:10, version: 70'112, data_included: [440350~816629,1309629~231947], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:10@70'112, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 70'113, data_included: [440350~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@70'113, size: 2605085, copy_subset: [(0, 2605085)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1488926, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:13.352 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.107266+0000", 2024-03-19T11:31:13.352 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 19.239581977, 2024-03-19T11:31:13.352 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.048592669999999998, 2024-03-19T11:31:13.352 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:13.352 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:13.352 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:13.352 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.352 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:13.352 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.107266+0000", 2024-03-19T11:31:13.352 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.352 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.352 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.352 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:13.352 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.107266+0000", 2024-03-19T11:31:13.353 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.353 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.353 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.353 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:13.353 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.107267+0000", 2024-03-19T11:31:13.353 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1459999999999999e-06 2024-03-19T11:31:13.353 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.353 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.353 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:13.353 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.122861+0000", 2024-03-19T11:31:13.353 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01559372 2024-03-19T11:31:13.353 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.353 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.353 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:13.353 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.122932+0000", 2024-03-19T11:31:13.353 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1477999999999996e-05 2024-03-19T11:31:13.354 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.354 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.354 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:13.354 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.122999+0000", 2024-03-19T11:31:13.354 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6832e-05 2024-03-19T11:31:13.354 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.354 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.354 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.354 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123332+0000", 2024-03-19T11:31:13.354 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000333271 2024-03-19T11:31:13.354 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.354 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.354 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:13.354 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123371+0000", 2024-03-19T11:31:13.355 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8525999999999997e-05 2024-03-19T11:31:13.355 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.355 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.355 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:13.355 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.155858+0000", 2024-03-19T11:31:13.355 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.032487697000000003 2024-03-19T11:31:13.355 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.355 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:13.355 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.355 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.355 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.355 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.21 e75,74 4:8401c1f0:::smithi03829856-43:8,46'66,4:866f3dcc:::smithi03829856-31:2,47'73)", 2024-03-19T11:31:13.355 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.123075+0000", 2024-03-19T11:31:13.355 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 19.223772757999999, 2024-03-19T11:31:13.355 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.033023045000000001, 2024-03-19T11:31:13.356 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:13.356 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:13.356 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:13.356 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.356 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:13.356 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123075+0000", 2024-03-19T11:31:13.356 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.356 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.356 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.356 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:13.356 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123075+0000", 2024-03-19T11:31:13.356 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.356 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.356 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.356 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:13.357 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123076+0000", 2024-03-19T11:31:13.357 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5e-06 2024-03-19T11:31:13.357 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.357 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.357 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:13.357 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123086+0000", 2024-03-19T11:31:13.357 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8060000000000006e-06 2024-03-19T11:31:13.357 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.357 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.357 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:13.357 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123099+0000", 2024-03-19T11:31:13.357 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3251000000000001e-05 2024-03-19T11:31:13.357 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.357 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.358 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:13.358 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123112+0000", 2024-03-19T11:31:13.358 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2774e-05 2024-03-19T11:31:13.358 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.358 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.358 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.358 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.155902+0000", 2024-03-19T11:31:13.358 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.032790225999999999 2024-03-19T11:31:13.358 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.358 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.358 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:13.358 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.155914+0000", 2024-03-19T11:31:13.358 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1476e-05 2024-03-19T11:31:13.358 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.359 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.359 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:13.359 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.156098+0000", 2024-03-19T11:31:13.359 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000184012 2024-03-19T11:31:13.359 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.359 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:13.359 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.359 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.359 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.359 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 75/74 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:7, version: 55'82, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@55'82, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 58'86, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@58'86, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [1652614~973634,2888511~52681,3182305~22261], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3204566, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:12, version: 66'101, data_included: [1541576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:12@66'101, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:10, version: 70'112, data_included: [1541576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:10@70'112, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 70'113, data_included: [1488926~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@70'113, size: 2605085, copy_subset: [(0, 2605085)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2537502, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1488926, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:13.359 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.174030+0000", 2024-03-19T11:31:13.359 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 19.172817554000002, 2024-03-19T11:31:13.359 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.040799275000000003, 2024-03-19T11:31:13.359 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:13.360 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:13.360 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:13.360 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.360 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:13.360 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.174030+0000", 2024-03-19T11:31:13.360 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.360 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.360 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.360 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:13.360 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.174030+0000", 2024-03-19T11:31:13.360 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.360 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.360 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.360 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:13.361 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.174031+0000", 2024-03-19T11:31:13.361 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.322e-06 2024-03-19T11:31:13.361 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.361 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.361 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:13.361 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.184771+0000", 2024-03-19T11:31:13.361 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010739227 2024-03-19T11:31:13.361 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.361 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.361 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:13.361 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.184872+0000", 2024-03-19T11:31:13.361 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000101234 2024-03-19T11:31:13.361 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.361 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.361 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:13.362 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.184985+0000", 2024-03-19T11:31:13.362 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000113545 2024-03-19T11:31:13.362 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.362 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.362 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.362 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.185463+0000", 2024-03-19T11:31:13.362 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00047749899999999999 2024-03-19T11:31:13.362 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.362 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.362 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:13.362 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.185534+0000", 2024-03-19T11:31:13.362 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1007999999999995e-05 2024-03-19T11:31:13.362 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.362 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.363 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:13.363 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.214829+0000", 2024-03-19T11:31:13.363 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029295439999999999 2024-03-19T11:31:13.363 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.363 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:13.363 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.363 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.363 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.363 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.21 e75,74 4:866f3dcc:::smithi03829856-31:c,70'114)", 2024-03-19T11:31:13.363 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.197457+0000", 2024-03-19T11:31:13.363 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 19.149390508, 2024-03-19T11:31:13.363 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018493590000000001, 2024-03-19T11:31:13.363 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:13.363 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:13.363 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:13.364 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.364 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:13.364 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.197457+0000", 2024-03-19T11:31:13.364 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.364 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.364 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.364 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:13.364 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.197457+0000", 2024-03-19T11:31:13.364 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.364 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.364 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.364 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:13.364 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.197459+0000", 2024-03-19T11:31:13.365 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3199999999999998e-06 2024-03-19T11:31:13.365 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.365 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.365 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:13.365 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.197480+0000", 2024-03-19T11:31:13.365 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0366000000000001e-05 2024-03-19T11:31:13.365 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.365 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.365 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:13.365 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.197521+0000", 2024-03-19T11:31:13.365 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0738000000000002e-05 2024-03-19T11:31:13.365 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.365 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.365 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:13.365 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.197547+0000", 2024-03-19T11:31:13.366 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6548999999999999e-05 2024-03-19T11:31:13.366 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.366 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.366 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.366 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.214954+0000", 2024-03-19T11:31:13.366 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01740713 2024-03-19T11:31:13.366 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.366 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.366 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:13.366 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.214972+0000", 2024-03-19T11:31:13.366 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7520000000000002e-05 2024-03-19T11:31:13.366 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.366 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.366 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:13.367 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.215951+0000", 2024-03-19T11:31:13.367 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00097896700000000007 2024-03-19T11:31:13.367 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.367 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:13.367 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.367 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.367 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.367 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 75/74 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [2599014~241336], data_size: 241336, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2840350, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [2599014~241336,2948041~446940], data_size: 688276, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3394981, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [3204566~621403], data_size: 621403, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3865577, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3204566, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:12, version: 66'101, data_included: [2590152~250198,2948041~581396], data_size: 831594, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:12@66'101, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3529437, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:10, version: 70'112, data_included: [2590152~250198,2948041~581396], data_size: 831594, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:10@70'112, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3529437, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 70'113, data_included: [2537502~67583], data_size: 67583, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@70'113, size: 2605085, copy_subset: [(0, 2605085)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2605085, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2537502, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:13.367 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.222872+0000", 2024-03-19T11:31:13.367 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 19.123976061, 2024-03-19T11:31:13.367 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016277658, 2024-03-19T11:31:13.367 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:13.367 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:13.367 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:13.368 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.368 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:13.368 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.222872+0000", 2024-03-19T11:31:13.368 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.368 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.368 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.368 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:13.368 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.222872+0000", 2024-03-19T11:31:13.368 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.368 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.368 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.368 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:13.368 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.222874+0000", 2024-03-19T11:31:13.368 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9999999999999999e-06 2024-03-19T11:31:13.368 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.369 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.369 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:13.369 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.227563+0000", 2024-03-19T11:31:13.369 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046897570000000001 2024-03-19T11:31:13.369 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.369 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.369 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:13.369 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.227639+0000", 2024-03-19T11:31:13.369 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5607999999999998e-05 2024-03-19T11:31:13.369 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.369 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.369 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:13.369 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.227736+0000", 2024-03-19T11:31:13.369 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6854000000000005e-05 2024-03-19T11:31:13.369 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.370 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.370 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.370 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.228149+0000", 2024-03-19T11:31:13.370 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00041330000000000002 2024-03-19T11:31:13.370 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.370 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.370 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:13.370 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.228219+0000", 2024-03-19T11:31:13.370 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.0140000000000003e-05 2024-03-19T11:31:13.370 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.370 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.370 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:13.371 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.239149+0000", 2024-03-19T11:31:13.371 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010929998999999999 2024-03-19T11:31:13.371 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.371 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:13.371 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.371 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.371 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.371 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2372 4.16 4:6868af91:::smithi03829856-10:b [read 832512~818688] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-03-19T11:31:13.371 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:58.455732+0000", 2024-03-19T11:31:13.371 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 14.891115838999999, 2024-03-19T11:31:13.371 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.69150541200000004, 2024-03-19T11:31:13.371 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:13.371 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:13.371 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:13.371 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:13.372 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:13.372 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2372 2024-03-19T11:31:13.372 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.372 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:13.372 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.372 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:13.372 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455732+0000", 2024-03-19T11:31:13.372 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.372 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.372 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.372 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:13.372 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455732+0000", 2024-03-19T11:31:13.372 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.372 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.372 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.373 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:13.373 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455735+0000", 2024-03-19T11:31:13.373 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.756e-06 2024-03-19T11:31:13.373 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.373 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.373 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:13.373 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455744+0000", 2024-03-19T11:31:13.373 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1880000000000008e-06 2024-03-19T11:31:13.373 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.373 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.373 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:13.373 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455764+0000", 2024-03-19T11:31:13.373 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0044999999999999e-05 2024-03-19T11:31:13.373 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.373 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.374 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:13.374 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455774+0000", 2024-03-19T11:31:13.374 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8770000000000005e-06 2024-03-19T11:31:13.374 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.374 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.374 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.374 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455837+0000", 2024-03-19T11:31:13.374 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3588000000000002e-05 2024-03-19T11:31:13.374 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.374 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.374 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:13.374 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455846+0000", 2024-03-19T11:31:13.374 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6030000000000005e-06 2024-03-19T11:31:13.374 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.375 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.375 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.375 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.076525+0000", 2024-03-19T11:31:13.375 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.62067875900000002 2024-03-19T11:31:13.375 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.375 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.375 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:31:13.375 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.076981+0000", 2024-03-19T11:31:13.375 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00045633300000000001 2024-03-19T11:31:13.375 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.375 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.375 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.375 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.104422+0000", 2024-03-19T11:31:13.375 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027440665999999999 2024-03-19T11:31:13.375 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.376 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.376 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:31:13.376 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.104786+0000", 2024-03-19T11:31:13.376 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00036471499999999999 2024-03-19T11:31:13.376 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.376 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.376 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.376 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.140725+0000", 2024-03-19T11:31:13.376 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.035939093999999998 2024-03-19T11:31:13.376 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.376 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.376 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:31:13.376 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.140841+0000", 2024-03-19T11:31:13.376 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000115207 2024-03-19T11:31:13.377 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.377 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.377 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.377 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.145559+0000", 2024-03-19T11:31:13.377 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.004718486 2024-03-19T11:31:13.377 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.377 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.377 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:13.377 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.145760+0000", 2024-03-19T11:31:13.377 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000201058 2024-03-19T11:31:13.377 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.377 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.377 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:13.377 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.147237+0000", 2024-03-19T11:31:13.377 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014770370000000001 2024-03-19T11:31:13.378 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.378 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:13.378 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.378 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.378 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.378 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.16 80/79 [PushOp(4:6d9093b1:::smithi03829856-38:head, version: 77'90, data_included: [500875~689757,1564955~358819], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:6d9093b1:::smithi03829856-38:head@77'90, size: 2100875, copy_subset: [(0, 2100875)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1923774, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:6be89c06:::smithi03829856-24:head, version: 77'92, data_included: [741376~589824,1753088~458752], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:6be89c06:::smithi03829856-24:head@77'92, size: 2386561, copy_subset: [(0, 2386561)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2211840, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:13.378 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:59.081830+0000", 2024-03-19T11:31:13.378 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 14.265017813, 2024-03-19T11:31:13.378 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015229652999999999, 2024-03-19T11:31:13.378 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:13.378 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:13.378 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:13.378 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.378 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:13.379 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.081830+0000", 2024-03-19T11:31:13.379 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.379 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.379 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.379 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:13.379 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.081830+0000", 2024-03-19T11:31:13.379 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.379 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.379 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.379 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:13.379 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.081832+0000", 2024-03-19T11:31:13.379 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.731e-06 2024-03-19T11:31:13.379 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.379 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.379 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:13.379 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.083140+0000", 2024-03-19T11:31:13.380 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0013082829999999999 2024-03-19T11:31:13.380 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.380 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.380 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:13.380 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.083170+0000", 2024-03-19T11:31:13.380 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0270999999999999e-05 2024-03-19T11:31:13.380 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.380 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.380 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:13.380 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.083199+0000", 2024-03-19T11:31:13.380 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9298e-05 2024-03-19T11:31:13.380 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.380 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.380 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.380 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.089969+0000", 2024-03-19T11:31:13.380 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0067691289999999996 2024-03-19T11:31:13.381 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.381 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.381 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:13.381 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.089995+0000", 2024-03-19T11:31:13.381 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5995000000000001e-05 2024-03-19T11:31:13.381 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.381 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.381 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:13.381 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.097060+0000", 2024-03-19T11:31:13.381 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0070649459999999999 2024-03-19T11:31:13.381 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.381 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:13.381 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.381 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.381 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.382 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.6 80/79 [PushOp(4:626acd4d:::smithi03829856-2:8, version: 55'54, data_included: [416807~475249,1542129~474678], data_size: 949927, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:626acd4d:::smithi03829856-2:8@55'54, size: 2016807, copy_subset: [(0, 2016807)], clone_subset: {}, snapset: 15=[]:{8: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2016807, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:13.382 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:59.110928+0000", 2024-03-19T11:31:13.382 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 14.235919499, 2024-03-19T11:31:13.382 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01605765, 2024-03-19T11:31:13.382 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:13.382 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:13.382 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:13.382 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.382 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:13.382 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.110928+0000", 2024-03-19T11:31:13.382 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.382 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.382 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.382 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:13.382 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.110928+0000", 2024-03-19T11:31:13.383 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.383 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.383 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.383 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:13.383 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.110929+0000", 2024-03-19T11:31:13.383 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6500000000000008e-07 2024-03-19T11:31:13.383 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.383 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.383 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:13.383 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.111446+0000", 2024-03-19T11:31:13.383 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00051713900000000001 2024-03-19T11:31:13.383 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.383 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.383 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:13.384 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.111464+0000", 2024-03-19T11:31:13.384 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7966e-05 2024-03-19T11:31:13.384 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.384 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.384 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:13.384 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.111483+0000", 2024-03-19T11:31:13.384 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8561000000000001e-05 2024-03-19T11:31:13.384 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.384 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.384 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.384 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.118569+0000", 2024-03-19T11:31:13.384 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0070860259999999996 2024-03-19T11:31:13.384 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.384 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.384 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:13.385 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.118586+0000", 2024-03-19T11:31:13.385 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6857000000000001e-05 2024-03-19T11:31:13.385 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.385 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.385 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:13.385 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.126986+0000", 2024-03-19T11:31:13.385 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0084001360000000008 2024-03-19T11:31:13.385 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.385 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:13.385 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.385 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.385 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.385 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.16 80/79 [PushOp(4:6868af91:::smithi03829856-10:10, version: 70'81, data_included: [524385~820472,1602039~228104], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:10@70'81, size: 3747928, copy_subset: [(0, 3747928)], clone_subset: {}, snapset: 14=[]:{7: [6], b: [b], 10: [d], 14: [13, 12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1830143, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:13.385 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:59.111343+0000", 2024-03-19T11:31:13.385 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 14.235504908999999, 2024-03-19T11:31:13.385 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019054888999999998, 2024-03-19T11:31:13.386 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:13.386 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:13.386 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:13.386 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.386 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:13.386 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.111343+0000", 2024-03-19T11:31:13.386 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.386 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.386 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.386 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:13.386 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.111343+0000", 2024-03-19T11:31:13.386 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.386 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.387 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.387 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:13.387 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.111349+0000", 2024-03-19T11:31:13.387 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9619999999999998e-06 2024-03-19T11:31:13.387 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.387 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.387 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:13.387 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.113881+0000", 2024-03-19T11:31:13.387 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025322930000000001 2024-03-19T11:31:13.387 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.387 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.387 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:13.387 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.113917+0000", 2024-03-19T11:31:13.387 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5593999999999998e-05 2024-03-19T11:31:13.387 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.388 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.388 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:13.388 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.113962+0000", 2024-03-19T11:31:13.388 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5362999999999999e-05 2024-03-19T11:31:13.388 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.388 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.388 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.388 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.126871+0000", 2024-03-19T11:31:13.388 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012909273000000001 2024-03-19T11:31:13.388 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.388 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.388 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:13.388 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.126892+0000", 2024-03-19T11:31:13.388 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0678000000000001e-05 2024-03-19T11:31:13.389 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.389 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.389 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:13.389 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.130398+0000", 2024-03-19T11:31:13.389 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0035057259999999998 2024-03-19T11:31:13.389 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.389 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:13.389 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.389 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.389 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.389 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.16 80/79 [PushOp(4:6d9093b1:::smithi03829856-38:10, version: 70'83, data_included: [621859~568773,1729599~479803], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6d9093b1:::smithi03829856-38:10@70'83, size: 3142876, copy_subset: [(0, 3142876)], clone_subset: {}, snapset: 10=[]:{10: [d, b, 6]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2209402, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:13.389 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:59.114057+0000", 2024-03-19T11:31:13.389 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 14.23279088, 2024-03-19T11:31:13.389 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019935583, 2024-03-19T11:31:13.389 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:13.390 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:13.390 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:13.390 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.390 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:13.390 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.114057+0000", 2024-03-19T11:31:13.390 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.390 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.390 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.390 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:13.390 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.114057+0000", 2024-03-19T11:31:13.390 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.390 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.390 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.390 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:13.391 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.114058+0000", 2024-03-19T11:31:13.391 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4300000000000001e-07 2024-03-19T11:31:13.391 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.391 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.391 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:13.391 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.117243+0000", 2024-03-19T11:31:13.391 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031852550000000001 2024-03-19T11:31:13.391 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.391 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.391 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:13.391 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.117263+0000", 2024-03-19T11:31:13.391 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0469999999999999e-05 2024-03-19T11:31:13.391 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.391 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.391 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:13.392 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.117276+0000", 2024-03-19T11:31:13.392 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2775000000000001e-05 2024-03-19T11:31:13.392 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.392 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.392 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.392 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.130681+0000", 2024-03-19T11:31:13.392 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013404879999999999 2024-03-19T11:31:13.392 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.392 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.392 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:13.392 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.130699+0000", 2024-03-19T11:31:13.392 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8343e-05 2024-03-19T11:31:13.392 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.392 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.392 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:13.393 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.133992+0000", 2024-03-19T11:31:13.393 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0032929169999999998 2024-03-19T11:31:13.393 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.393 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:13.393 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.393 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.393 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.393 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2442 4.16 4:6d9093b1:::smithi03829856-38:head [write 460558~707131] snapc 17=[17,16,15,13,12,b,6] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-03-19T11:31:13.393 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:03.432382+0000", 2024-03-19T11:31:13.393 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 9.9144653290000004, 2024-03-19T11:31:13.393 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019918379, 2024-03-19T11:31:13.393 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:13.393 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:13.393 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:13.393 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:13.394 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:13.394 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2442 2024-03-19T11:31:13.394 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.394 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:13.394 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.394 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:13.394 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.432382+0000", 2024-03-19T11:31:13.394 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.394 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.394 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.394 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:13.394 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.432382+0000", 2024-03-19T11:31:13.394 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.395 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.395 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.395 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:13.395 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.432385+0000", 2024-03-19T11:31:13.395 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1e-06 2024-03-19T11:31:13.395 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.395 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.395 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:13.395 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.433115+0000", 2024-03-19T11:31:13.395 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00072939900000000004 2024-03-19T11:31:13.395 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.395 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.395 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:13.395 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.433151+0000", 2024-03-19T11:31:13.395 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6094999999999999e-05 2024-03-19T11:31:13.395 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.396 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.396 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:13.396 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.433167+0000", 2024-03-19T11:31:13.396 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6152999999999999e-05 2024-03-19T11:31:13.396 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.396 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.396 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.396 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.433339+0000", 2024-03-19T11:31:13.396 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000171478 2024-03-19T11:31:13.396 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.396 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.396 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:13.396 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.433621+0000", 2024-03-19T11:31:13.396 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00028283900000000001 2024-03-19T11:31:13.396 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.397 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.397 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3", 2024-03-19T11:31:13.397 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.434169+0000", 2024-03-19T11:31:13.397 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00054786699999999995 2024-03-19T11:31:13.397 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.397 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.397 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:13.397 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.443136+0000", 2024-03-19T11:31:13.397 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0089667419999999998 2024-03-19T11:31:13.397 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.397 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.397 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:13.397 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.452230+0000", 2024-03-19T11:31:13.397 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.009093996 2024-03-19T11:31:13.397 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.398 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.398 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:13.398 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.452264+0000", 2024-03-19T11:31:13.398 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4381000000000002e-05 2024-03-19T11:31:13.398 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.398 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.398 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:13.398 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.452301+0000", 2024-03-19T11:31:13.398 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6328999999999997e-05 2024-03-19T11:31:13.398 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.398 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:13.398 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.398 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.398 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.399 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.4.13:82 3.3 3:f6b6d21a:::smithi03829856-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-03-19T11:31:13.399 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:04.069448+0000", 2024-03-19T11:31:13.399 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 9.2774000539999992, 2024-03-19T11:31:13.399 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.086427773999999999, 2024-03-19T11:31:13.399 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:13.399 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:13.399 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:13.399 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.399 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:13.399 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069448+0000", 2024-03-19T11:31:13.399 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.399 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.399 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.400 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:13.400 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069448+0000", 2024-03-19T11:31:13.400 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.400 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.400 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.400 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:13.400 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069449+0000", 2024-03-19T11:31:13.400 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8160000000000001e-06 2024-03-19T11:31:13.400 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.400 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.400 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:13.400 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069458+0000", 2024-03-19T11:31:13.400 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2900000000000002e-06 2024-03-19T11:31:13.400 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.401 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.401 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:13.401 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069491+0000", 2024-03-19T11:31:13.401 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3229000000000003e-05 2024-03-19T11:31:13.401 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.401 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.401 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:13.401 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069501+0000", 2024-03-19T11:31:13.401 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7629999999999995e-06 2024-03-19T11:31:13.401 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.401 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.401 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.401 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069573+0000", 2024-03-19T11:31:13.401 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2349999999999997e-05 2024-03-19T11:31:13.401 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.401 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.402 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:13.402 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069581+0000", 2024-03-19T11:31:13.402 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.388e-06 2024-03-19T11:31:13.402 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.402 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.402 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.402 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.075451+0000", 2024-03-19T11:31:13.402 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005869956 2024-03-19T11:31:13.402 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.402 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.402 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:13.402 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.077511+0000", 2024-03-19T11:31:13.402 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0020592610000000002 2024-03-19T11:31:13.402 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.403 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.403 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.403 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.108781+0000", 2024-03-19T11:31:13.403 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.031270344999999998 2024-03-19T11:31:13.403 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.403 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.403 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:13.403 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.108895+0000", 2024-03-19T11:31:13.403 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000114059 2024-03-19T11:31:13.403 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.403 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.403 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:13.403 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.109065+0000", 2024-03-19T11:31:13.403 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016979899999999999 2024-03-19T11:31:13.403 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.404 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.404 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:13.404 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.111730+0000", 2024-03-19T11:31:13.404 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0026651410000000002 2024-03-19T11:31:13.404 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.404 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.404 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:13.404 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.155616+0000", 2024-03-19T11:31:13.404 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.043885986000000002 2024-03-19T11:31:13.404 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.404 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.404 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:13.404 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.155745+0000", 2024-03-19T11:31:13.404 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012948899999999999 2024-03-19T11:31:13.404 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.405 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.405 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:13.405 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.155875+0000", 2024-03-19T11:31:13.405 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012990199999999999 2024-03-19T11:31:13.405 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.405 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:13.405 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.405 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.405 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.405 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.4.13:83 3.3 3:f6b6d21a:::smithi03829856-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-03-19T11:31:13.405 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:04.069526+0000", 2024-03-19T11:31:13.405 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 9.2773212740000002, 2024-03-19T11:31:13.405 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.087140880000000004, 2024-03-19T11:31:13.405 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:13.405 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:13.406 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:13.406 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.406 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:13.406 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069526+0000", 2024-03-19T11:31:13.406 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.406 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.406 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.406 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:13.406 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069526+0000", 2024-03-19T11:31:13.406 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.406 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.406 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.406 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:13.406 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069527+0000", 2024-03-19T11:31:13.406 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2799999999999996e-07 2024-03-19T11:31:13.407 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.407 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.407 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:13.407 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069531+0000", 2024-03-19T11:31:13.407 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1489999999999996e-06 2024-03-19T11:31:13.407 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.407 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.407 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:13.407 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069539+0000", 2024-03-19T11:31:13.407 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3189999999999995e-06 2024-03-19T11:31:13.407 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.407 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.407 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:13.407 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069545+0000", 2024-03-19T11:31:13.408 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.592e-06 2024-03-19T11:31:13.408 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.408 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.408 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.408 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069616+0000", 2024-03-19T11:31:13.408 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1130000000000005e-05 2024-03-19T11:31:13.408 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.408 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.408 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:13.408 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069628+0000", 2024-03-19T11:31:13.408 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1697e-05 2024-03-19T11:31:13.408 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.408 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.408 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.408 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.077594+0000", 2024-03-19T11:31:13.409 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0079656989999999997 2024-03-19T11:31:13.409 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.409 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.409 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:13.409 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.077674+0000", 2024-03-19T11:31:13.409 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0338000000000002e-05 2024-03-19T11:31:13.409 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.409 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.409 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.409 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.112771+0000", 2024-03-19T11:31:13.409 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.035097325999999998 2024-03-19T11:31:13.409 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.409 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.409 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:13.409 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.112882+0000", 2024-03-19T11:31:13.410 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000110949 2024-03-19T11:31:13.410 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.410 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.410 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:13.410 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.113031+0000", 2024-03-19T11:31:13.410 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000148355 2024-03-19T11:31:13.410 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.410 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.410 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:13.410 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.114076+0000", 2024-03-19T11:31:13.410 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001045009 2024-03-19T11:31:13.410 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.410 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.410 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:13.411 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156482+0000", 2024-03-19T11:31:13.411 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.042406354 2024-03-19T11:31:13.411 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.411 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.411 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:13.411 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156549+0000", 2024-03-19T11:31:13.411 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7229000000000003e-05 2024-03-19T11:31:13.411 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.411 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.411 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:13.411 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156667+0000", 2024-03-19T11:31:13.411 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000118106 2024-03-19T11:31:13.411 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.411 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:13.411 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.412 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.412 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.412 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.4.13:84 3.3 3:ce6951d3:::smithi03829856-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-03-19T11:31:13.412 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:04.071995+0000", 2024-03-19T11:31:13.412 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 9.2748524989999996, 2024-03-19T11:31:13.412 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.084172497999999998, 2024-03-19T11:31:13.412 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:13.412 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:13.412 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:13.412 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.412 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:13.412 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.071995+0000", 2024-03-19T11:31:13.412 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.412 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.413 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.413 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:13.413 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.071995+0000", 2024-03-19T11:31:13.413 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.413 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.413 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.413 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:13.413 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.071997+0000", 2024-03-19T11:31:13.413 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6029999999999999e-06 2024-03-19T11:31:13.413 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.413 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.413 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:13.413 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072004+0000", 2024-03-19T11:31:13.413 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8600000000000004e-06 2024-03-19T11:31:13.413 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.413 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.414 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:13.414 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072033+0000", 2024-03-19T11:31:13.414 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9836999999999999e-05 2024-03-19T11:31:13.414 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.414 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.414 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:13.414 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072042+0000", 2024-03-19T11:31:13.414 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9279999999999999e-06 2024-03-19T11:31:13.414 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.414 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.414 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.414 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072123+0000", 2024-03-19T11:31:13.414 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0630000000000006e-05 2024-03-19T11:31:13.414 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.415 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.415 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:13.415 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072132+0000", 2024-03-19T11:31:13.415 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5560000000000008e-06 2024-03-19T11:31:13.415 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.415 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.415 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.415 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.077712+0000", 2024-03-19T11:31:13.415 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055804970000000002 2024-03-19T11:31:13.415 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.415 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.415 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:13.415 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.079495+0000", 2024-03-19T11:31:13.415 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017831850000000001 2024-03-19T11:31:13.415 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.416 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.416 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.416 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.109825+0000", 2024-03-19T11:31:13.416 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.030329819000000001 2024-03-19T11:31:13.416 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.416 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.416 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:13.416 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.109904+0000", 2024-03-19T11:31:13.416 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9269000000000003e-05 2024-03-19T11:31:13.416 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.416 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.416 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:13.416 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.110085+0000", 2024-03-19T11:31:13.416 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00018018600000000001 2024-03-19T11:31:13.416 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.417 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.417 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:13.417 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.113581+0000", 2024-03-19T11:31:13.417 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034962919999999998 2024-03-19T11:31:13.417 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.417 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.417 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:13.417 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.155925+0000", 2024-03-19T11:31:13.417 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.042344404000000002 2024-03-19T11:31:13.417 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.417 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.417 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:13.417 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156026+0000", 2024-03-19T11:31:13.417 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000100599 2024-03-19T11:31:13.417 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.418 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.418 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:13.418 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156168+0000", 2024-03-19T11:31:13.418 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014183300000000001 2024-03-19T11:31:13.418 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.418 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:13.418 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.418 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.418 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.418 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.4.13:85 3.3 3:ce6951d3:::smithi03829856-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-03-19T11:31:13.418 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:04.072068+0000", 2024-03-19T11:31:13.418 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 9.2747798629999991, 2024-03-19T11:31:13.418 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.084368466000000003, 2024-03-19T11:31:13.418 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:13.418 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:13.419 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:13.419 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.419 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:13.419 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072068+0000", 2024-03-19T11:31:13.419 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.419 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.419 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.419 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:13.419 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072068+0000", 2024-03-19T11:31:13.419 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.419 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.419 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.419 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:13.419 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072068+0000", 2024-03-19T11:31:13.419 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9699999999999996e-07 2024-03-19T11:31:13.420 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.420 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.420 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:13.420 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072074+0000", 2024-03-19T11:31:13.420 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2279999999999998e-06 2024-03-19T11:31:13.420 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.420 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.420 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:13.420 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072140+0000", 2024-03-19T11:31:13.420 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6638999999999997e-05 2024-03-19T11:31:13.420 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.420 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.420 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:13.420 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072147+0000", 2024-03-19T11:31:13.420 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8750000000000002e-06 2024-03-19T11:31:13.420 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.421 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.421 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.421 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072201+0000", 2024-03-19T11:31:13.421 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3634000000000002e-05 2024-03-19T11:31:13.421 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.421 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.421 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:13.421 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072211+0000", 2024-03-19T11:31:13.421 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0589e-05 2024-03-19T11:31:13.421 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.421 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.421 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.421 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.079687+0000", 2024-03-19T11:31:13.421 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074757219999999997 2024-03-19T11:31:13.421 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.422 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.422 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:13.422 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.079769+0000", 2024-03-19T11:31:13.422 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2251999999999998e-05 2024-03-19T11:31:13.422 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.422 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.422 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.422 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.111798+0000", 2024-03-19T11:31:13.422 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.032028889999999997 2024-03-19T11:31:13.422 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.422 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.422 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:13.422 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.111902+0000", 2024-03-19T11:31:13.422 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000103773 2024-03-19T11:31:13.423 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.423 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.423 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:13.423 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.112073+0000", 2024-03-19T11:31:13.423 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017102799999999999 2024-03-19T11:31:13.423 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.423 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.423 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:13.423 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.113591+0000", 2024-03-19T11:31:13.423 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0015179569999999999 2024-03-19T11:31:13.423 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.423 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.423 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:13.423 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156223+0000", 2024-03-19T11:31:13.423 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.042632153999999998 2024-03-19T11:31:13.424 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.424 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.424 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:13.424 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156321+0000", 2024-03-19T11:31:13.424 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8246999999999996e-05 2024-03-19T11:31:13.424 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.424 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.424 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:13.424 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156436+0000", 2024-03-19T11:31:13.424 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000114881 2024-03-19T11:31:13.424 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.424 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:13.424 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.424 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.424 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.425 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2455 3.1 3:8e10ce27:::f25cf4cdffbae9b54315ae93e2cdc514e3bf58d758c8445671e9463006f381a6:head [write 0~1661440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-03-19T11:31:13.425 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:04.086189+0000", 2024-03-19T11:31:13.425 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 9.2606587170000001, 2024-03-19T11:31:13.425 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015194925999999999, 2024-03-19T11:31:13.425 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:13.425 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:13.425 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:13.425 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:13.425 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:13.425 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2455 2024-03-19T11:31:13.425 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.425 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:13.425 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.425 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:13.426 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.086189+0000", 2024-03-19T11:31:13.426 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.426 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.426 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.426 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:13.426 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.086189+0000", 2024-03-19T11:31:13.426 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.426 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.426 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.426 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:13.426 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.086191+0000", 2024-03-19T11:31:13.426 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8619999999999999e-06 2024-03-19T11:31:13.426 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.426 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.426 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:13.427 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.086944+0000", 2024-03-19T11:31:13.427 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00075297299999999999 2024-03-19T11:31:13.427 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.427 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.427 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:13.427 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.086966+0000", 2024-03-19T11:31:13.427 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2404000000000001e-05 2024-03-19T11:31:13.427 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.427 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.427 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:13.427 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.086976+0000", 2024-03-19T11:31:13.427 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8069999999999994e-06 2024-03-19T11:31:13.427 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.427 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.428 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.428 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.087038+0000", 2024-03-19T11:31:13.428 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2314e-05 2024-03-19T11:31:13.428 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.428 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.428 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:13.428 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.087158+0000", 2024-03-19T11:31:13.428 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000119385 2024-03-19T11:31:13.428 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.428 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.428 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:13.428 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.087286+0000", 2024-03-19T11:31:13.428 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012851599999999999 2024-03-19T11:31:13.428 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.429 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.429 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:13.429 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.089954+0000", 2024-03-19T11:31:13.429 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0026674239999999998 2024-03-19T11:31:13.429 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.429 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.429 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:13.429 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.101235+0000", 2024-03-19T11:31:13.429 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011281807 2024-03-19T11:31:13.429 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.429 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.429 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:13.429 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.101321+0000", 2024-03-19T11:31:13.429 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5359999999999996e-05 2024-03-19T11:31:13.429 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.430 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.430 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:13.430 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.101384+0000", 2024-03-19T11:31:13.430 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3074e-05 2024-03-19T11:31:13.430 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.430 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:13.430 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.430 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.430 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.430 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2593 4.6 4:626acd4d:::smithi03829856-2:head [write 662242~642825] snapc 17=[17,16,15,b] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-03-19T11:31:13.430 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:07.785391+0000", 2024-03-19T11:31:13.430 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.5614564489999996, 2024-03-19T11:31:13.430 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018221268999999998, 2024-03-19T11:31:13.430 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:13.431 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:13.431 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:13.431 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:13.431 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:13.431 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2593 2024-03-19T11:31:13.431 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.431 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:13.431 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.431 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:13.431 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.785391+0000", 2024-03-19T11:31:13.431 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.431 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.431 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.431 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:13.431 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.785391+0000", 2024-03-19T11:31:13.432 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.432 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.432 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.432 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:13.432 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.785394+0000", 2024-03-19T11:31:13.432 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.903e-06 2024-03-19T11:31:13.432 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.432 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.432 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:13.432 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.786080+0000", 2024-03-19T11:31:13.432 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000686201 2024-03-19T11:31:13.432 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.432 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.432 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:13.432 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.786116+0000", 2024-03-19T11:31:13.432 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5441999999999997e-05 2024-03-19T11:31:13.433 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.433 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.433 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:13.433 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.786132+0000", 2024-03-19T11:31:13.433 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6353000000000001e-05 2024-03-19T11:31:13.433 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.433 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.433 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.433 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.786283+0000", 2024-03-19T11:31:13.433 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015133599999999999 2024-03-19T11:31:13.433 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.433 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.433 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:13.433 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.786457+0000", 2024-03-19T11:31:13.433 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017352200000000001 2024-03-19T11:31:13.434 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.434 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.434 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1", 2024-03-19T11:31:13.434 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.786765+0000", 2024-03-19T11:31:13.434 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030768199999999999 2024-03-19T11:31:13.434 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.434 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.434 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:13.434 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.798155+0000", 2024-03-19T11:31:13.434 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011390045999999999 2024-03-19T11:31:13.434 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.434 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.434 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:13.434 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.803291+0000", 2024-03-19T11:31:13.434 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0051363060000000002 2024-03-19T11:31:13.435 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.435 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.435 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:13.435 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.803466+0000", 2024-03-19T11:31:13.435 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017505900000000001 2024-03-19T11:31:13.435 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.435 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.435 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:13.435 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.803612+0000", 2024-03-19T11:31:13.435 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014641900000000001 2024-03-19T11:31:13.435 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.435 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:13.435 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.435 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.435 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:13.436 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:13.437 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-19T11:31:13.437 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:13.447+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:13.538 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:13.547+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:13.560 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:13.560 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:31:13.560 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:31:13.560 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:13.574 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-19T11:31:13.639 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:13.645+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:13.671 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T11:31:13.694 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:13.694 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:31:13.694 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:31:13.694 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:31:13.694 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:13.710 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-19T11:31:13.756 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:13.765+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:13.838 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:13.845+0000 7fa5de8d7640 -1 osd.3 93 handle_osd_map injecting map crc failure 2024-03-19T11:31:13.840 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:13.849+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:13.877 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:13.877 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:31:13.877 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:31:13.878 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:31:13.878 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.878 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1063 4.4 4:214026b3:::smithi03829856-47:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:31:13.878 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.784904+0000", 2024-03-19T11:31:13.878 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 60.101836198999997, 2024-03-19T11:31:13.878 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024425912000000001, 2024-03-19T11:31:13.878 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:13.878 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:13.878 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:13.878 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:13.878 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:13.878 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1063 2024-03-19T11:31:13.879 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.879 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:13.879 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.879 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:13.879 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784904+0000", 2024-03-19T11:31:13.879 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.879 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.879 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.879 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:13.879 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784904+0000", 2024-03-19T11:31:13.879 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.880 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.880 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.880 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:13.880 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784906+0000", 2024-03-19T11:31:13.880 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1409999999999999e-06 2024-03-19T11:31:13.880 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.881 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.881 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:13.881 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784915+0000", 2024-03-19T11:31:13.881 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5459999999999992e-06 2024-03-19T11:31:13.881 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.881 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.881 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:13.881 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784935+0000", 2024-03-19T11:31:13.881 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0310999999999999e-05 2024-03-19T11:31:13.881 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.882 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.882 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:13.882 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784946+0000", 2024-03-19T11:31:13.882 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0675e-05 2024-03-19T11:31:13.882 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.882 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.882 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.882 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.785032+0000", 2024-03-19T11:31:13.882 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6049000000000002e-05 2024-03-19T11:31:13.882 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.882 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.882 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:13.882 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.785136+0000", 2024-03-19T11:31:13.883 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000104209 2024-03-19T11:31:13.883 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.883 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.883 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:13.883 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.809330+0000", 2024-03-19T11:31:13.883 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024193981 2024-03-19T11:31:13.883 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.883 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:13.883 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.883 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.883 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.883 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1433 4.16 4:6be89c06:::smithi03829856-24:c [sparse-read 0~1100427,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:31:13.883 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.976652+0000", 2024-03-19T11:31:13.883 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 47.910088498, 2024-03-19T11:31:13.883 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95684753199999995, 2024-03-19T11:31:13.884 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:13.884 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:13.884 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:13.884 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:13.884 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:13.884 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1433 2024-03-19T11:31:13.885 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.886 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:13.886 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.886 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:13.886 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976652+0000", 2024-03-19T11:31:13.886 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.886 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.886 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.886 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:13.886 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976652+0000", 2024-03-19T11:31:13.886 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.886 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.886 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.886 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:13.886 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976654+0000", 2024-03-19T11:31:13.887 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6199999999999999e-06 2024-03-19T11:31:13.887 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.887 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.887 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:13.887 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976668+0000", 2024-03-19T11:31:13.887 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3213999999999999e-05 2024-03-19T11:31:13.887 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.887 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.887 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:13.887 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976694+0000", 2024-03-19T11:31:13.887 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6488000000000001e-05 2024-03-19T11:31:13.887 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.887 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.887 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:13.888 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976705+0000", 2024-03-19T11:31:13.888 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1044e-05 2024-03-19T11:31:13.888 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.888 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.888 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.888 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976795+0000", 2024-03-19T11:31:13.888 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0283000000000003e-05 2024-03-19T11:31:13.888 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.888 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.888 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:13.888 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976811+0000", 2024-03-19T11:31:13.888 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5262000000000001e-05 2024-03-19T11:31:13.888 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.888 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.889 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.889 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.927337+0000", 2024-03-19T11:31:13.889 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95052597100000003 2024-03-19T11:31:13.889 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.889 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.889 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:13.889 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.927495+0000", 2024-03-19T11:31:13.889 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015788600000000001 2024-03-19T11:31:13.889 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.889 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.889 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:13.889 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.933499+0000", 2024-03-19T11:31:13.889 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0060047640000000001 2024-03-19T11:31:13.889 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.889 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:13.890 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.890 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.890 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.890 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1434 4.16 4:6be89c06:::smithi03829856-24:c [read 0~2386561,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:31:13.890 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.976739+0000", 2024-03-19T11:31:13.890 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 47.910001336999997, 2024-03-19T11:31:13.890 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95781317399999999, 2024-03-19T11:31:13.890 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:13.890 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:13.891 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:13.891 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:13.891 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:13.891 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1434 2024-03-19T11:31:13.891 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.891 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:13.891 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.891 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:13.891 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976739+0000", 2024-03-19T11:31:13.891 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.891 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.891 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.891 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:13.891 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976739+0000", 2024-03-19T11:31:13.892 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.892 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.892 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.892 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:13.892 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976740+0000", 2024-03-19T11:31:13.892 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8800000000000001e-07 2024-03-19T11:31:13.892 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.892 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.892 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:13.892 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976748+0000", 2024-03-19T11:31:13.892 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8979999999999996e-06 2024-03-19T11:31:13.892 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.892 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.892 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:13.892 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976757+0000", 2024-03-19T11:31:13.893 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9439999999999994e-06 2024-03-19T11:31:13.893 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.893 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.893 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:13.893 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976763+0000", 2024-03-19T11:31:13.893 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5030000000000002e-06 2024-03-19T11:31:13.893 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.893 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.893 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.893 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976849+0000", 2024-03-19T11:31:13.893 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5803999999999994e-05 2024-03-19T11:31:13.893 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.893 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.893 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:13.893 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976860+0000", 2024-03-19T11:31:13.894 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0869e-05 2024-03-19T11:31:13.894 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.894 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.894 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.894 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.933562+0000", 2024-03-19T11:31:13.894 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95670232300000002 2024-03-19T11:31:13.894 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.894 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.894 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:13.894 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.933693+0000", 2024-03-19T11:31:13.894 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013110699999999999 2024-03-19T11:31:13.894 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.894 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.895 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:13.895 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.934552+0000", 2024-03-19T11:31:13.895 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00085883800000000001 2024-03-19T11:31:13.895 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.895 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:13.895 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.895 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.895 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.895 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1435 4.16 4:6be89c06:::smithi03829856-24:c [sparse-read 0~1100427] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:31:13.895 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.976775+0000", 2024-03-19T11:31:13.895 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 47.909965665999998, 2024-03-19T11:31:13.895 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95962779499999995, 2024-03-19T11:31:13.895 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:13.895 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:13.896 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:13.896 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:13.896 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:13.896 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1435 2024-03-19T11:31:13.896 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.896 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:13.896 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.896 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:13.896 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976775+0000", 2024-03-19T11:31:13.896 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.896 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.896 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.896 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:13.896 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976775+0000", 2024-03-19T11:31:13.896 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.897 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.897 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.897 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:13.897 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976775+0000", 2024-03-19T11:31:13.897 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9800000000000003e-07 2024-03-19T11:31:13.897 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.897 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.897 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:13.897 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976779+0000", 2024-03-19T11:31:13.897 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3830000000000001e-06 2024-03-19T11:31:13.897 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.897 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.897 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:13.897 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976785+0000", 2024-03-19T11:31:13.897 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5649999999999998e-06 2024-03-19T11:31:13.898 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.898 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.898 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:13.898 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976790+0000", 2024-03-19T11:31:13.898 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1379999999999998e-06 2024-03-19T11:31:13.898 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.898 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.898 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.898 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976889+0000", 2024-03-19T11:31:13.898 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8331000000000004e-05 2024-03-19T11:31:13.898 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.898 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.898 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:13.899 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976894+0000", 2024-03-19T11:31:13.899 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8270000000000003e-06 2024-03-19T11:31:13.899 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.899 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.899 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.899 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.935982+0000", 2024-03-19T11:31:13.899 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95908714100000003 2024-03-19T11:31:13.899 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.899 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.899 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:13.899 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.936093+0000", 2024-03-19T11:31:13.899 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000111557 2024-03-19T11:31:13.899 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.899 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.899 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:13.900 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.936402+0000", 2024-03-19T11:31:13.900 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030925499999999998 2024-03-19T11:31:13.900 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.900 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:13.900 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.900 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.900 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.900 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1442 4.6 4:626acd4d:::smithi03829856-2:head [cache-evict] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:31:13.900 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.978864+0000", 2024-03-19T11:31:13.900 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 47.907876002000002, 2024-03-19T11:31:13.900 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95708845099999995, 2024-03-19T11:31:13.900 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:13.900 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:13.900 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:13.900 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:13.901 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:13.901 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1442 2024-03-19T11:31:13.901 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.901 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:13.901 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.901 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:13.901 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978864+0000", 2024-03-19T11:31:13.901 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.901 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.901 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.901 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:13.901 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978864+0000", 2024-03-19T11:31:13.901 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.901 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.902 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.902 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:13.902 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978866+0000", 2024-03-19T11:31:13.902 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0729999999999999e-06 2024-03-19T11:31:13.902 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.902 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.902 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:13.902 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978877+0000", 2024-03-19T11:31:13.902 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0390999999999999e-05 2024-03-19T11:31:13.902 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.902 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.902 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:13.902 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978902+0000", 2024-03-19T11:31:13.902 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5635000000000001e-05 2024-03-19T11:31:13.903 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.903 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.903 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:13.903 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978915+0000", 2024-03-19T11:31:13.903 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2617e-05 2024-03-19T11:31:13.903 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.903 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.903 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.903 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.979005+0000", 2024-03-19T11:31:13.903 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0552999999999999e-05 2024-03-19T11:31:13.903 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.903 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.903 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:13.903 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.979021+0000", 2024-03-19T11:31:13.903 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5102999999999999e-05 2024-03-19T11:31:13.903 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.904 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.904 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.904 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.926879+0000", 2024-03-19T11:31:13.904 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.94785817900000002 2024-03-19T11:31:13.904 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.904 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.904 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:13.904 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.927277+0000", 2024-03-19T11:31:13.904 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00039819600000000002 2024-03-19T11:31:13.904 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.904 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.904 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.904 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.935752+0000", 2024-03-19T11:31:13.904 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0084753420000000003 2024-03-19T11:31:13.904 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.905 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.905 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:13.905 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.935874+0000", 2024-03-19T11:31:13.905 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000121311 2024-03-19T11:31:13.905 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.905 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.905 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:13.905 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.935953+0000", 2024-03-19T11:31:13.905 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9050999999999999e-05 2024-03-19T11:31:13.905 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.905 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:13.905 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.905 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.905 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.906 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.d 49/48 [PushOp(4:b65e6658:::smithi03829856-9:head, version: 28'17, data_included: [534456~718345,1951503~182953], data_size: 901298, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:b65e6658:::smithi03829856-9:head@28'17, size: 2134456, copy_subset: [(0, 2134456)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2134456, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:b4ef97e0:::smithi03829856-3:b, version: 46'34, data_included: [433614~1004682,1600611~43894], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:b4ef97e0:::smithi03829856-3:b@46'34, size: 3042034, copy_subset: [(0, 3042034)], clone_subset: {}, snapset: b=[]:{b: [b, a, 6, 5]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1644505, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:b4ef97e0:::smithi03829856-3:head, version: 47'43, data_included: [433614~1004682,1600611~43894], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:b4ef97e0:::smithi03829856-3:head@47'43, size: 3697507, copy_subset: [(0, 3697507)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1644505, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:13.906 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:26.943544+0000", 2024-03-19T11:31:13.906 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 46.943196512, 2024-03-19T11:31:13.906 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015601487000000001, 2024-03-19T11:31:13.906 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:13.906 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:13.906 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:13.906 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.906 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:13.906 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.943544+0000", 2024-03-19T11:31:13.906 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.906 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.906 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.906 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:13.907 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.943544+0000", 2024-03-19T11:31:13.907 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.907 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.907 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.907 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:13.907 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.943546+0000", 2024-03-19T11:31:13.907 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.013e-06 2024-03-19T11:31:13.907 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.907 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.907 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:13.907 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.946208+0000", 2024-03-19T11:31:13.907 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0026622820000000002 2024-03-19T11:31:13.907 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.907 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.907 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:13.908 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.946240+0000", 2024-03-19T11:31:13.908 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1568e-05 2024-03-19T11:31:13.908 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.908 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.908 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:13.908 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.946282+0000", 2024-03-19T11:31:13.908 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2599999999999999e-05 2024-03-19T11:31:13.908 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.908 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.908 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.908 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.946442+0000", 2024-03-19T11:31:13.908 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015960400000000001 2024-03-19T11:31:13.908 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.908 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.908 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:13.909 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.946458+0000", 2024-03-19T11:31:13.909 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6059999999999999e-05 2024-03-19T11:31:13.909 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.909 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.909 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:13.909 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.959145+0000", 2024-03-19T11:31:13.909 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01268736 2024-03-19T11:31:13.909 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.909 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:13.909 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.909 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.909 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.909 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2003 4.16 4:6868af91:::smithi03829856-10:head [tier-promote] snapc 12=[12,11,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:31:13.909 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:45.804046+0000", 2024-03-19T11:31:13.909 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 28.082694089, 2024-03-19T11:31:13.910 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018976396, 2024-03-19T11:31:13.910 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:13.910 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:13.910 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:13.910 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:13.910 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:13.910 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2003 2024-03-19T11:31:13.910 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.910 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:13.910 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.910 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:13.910 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804046+0000", 2024-03-19T11:31:13.910 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.911 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.911 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.911 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:13.911 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804046+0000", 2024-03-19T11:31:13.911 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.911 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.911 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.911 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:13.911 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804052+0000", 2024-03-19T11:31:13.911 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7599999999999999e-06 2024-03-19T11:31:13.911 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.911 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.911 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:13.911 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804090+0000", 2024-03-19T11:31:13.911 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8018000000000001e-05 2024-03-19T11:31:13.912 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.912 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.912 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:13.912 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804256+0000", 2024-03-19T11:31:13.912 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016593599999999999 2024-03-19T11:31:13.912 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.912 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.912 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:13.912 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804300+0000", 2024-03-19T11:31:13.912 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3648e-05 2024-03-19T11:31:13.912 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.912 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.912 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.912 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804538+0000", 2024-03-19T11:31:13.912 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000238347 2024-03-19T11:31:13.912 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.913 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.913 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:13.913 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804716+0000", 2024-03-19T11:31:13.913 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000177968 2024-03-19T11:31:13.913 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.913 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.913 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:13.913 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.823023+0000", 2024-03-19T11:31:13.913 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018306718999999999 2024-03-19T11:31:13.913 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.913 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:13.913 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.913 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.913 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.914 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1d58db11:::smithi03829856-18:6, version: 55'80, data_included: [759683~639358,1904434~409218], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:6@55'80, size: 3159683, copy_subset: [(0, 3159683)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2313652, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1d58db11:::smithi03829856-18:b, version: 58'84, data_included: [443718~955323,1564138~93253], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:b@58'84, size: 3445402, copy_subset: [(0, 3445402)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1657391, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:10, version: 66'95, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:10@66'95, size: 3764735, copy_subset: [(0, 3764735)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:12, version: 66'97, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:12@66'97, size: 2504243, copy_subset: [(0, 2504243)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:c, version: 70'102, data_included: [416807~576167,1542129~472409], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:c@70'102, size: 2857157, copy_subset: [(0, 2857157)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2014538, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:13.914 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.275179+0000", 2024-03-19T11:31:13.914 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 19.611561098999999, 2024-03-19T11:31:13.914 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026559353000000001, 2024-03-19T11:31:13.914 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:13.914 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:13.914 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:13.914 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.914 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:13.914 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.275179+0000", 2024-03-19T11:31:13.914 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.914 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.914 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.914 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:13.915 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.275179+0000", 2024-03-19T11:31:13.915 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.915 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.915 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.915 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:13.915 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.275196+0000", 2024-03-19T11:31:13.915 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6946999999999999e-05 2024-03-19T11:31:13.915 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.915 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.915 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:13.915 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281456+0000", 2024-03-19T11:31:13.915 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0062600529999999998 2024-03-19T11:31:13.915 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.916 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.916 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:13.916 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281524+0000", 2024-03-19T11:31:13.916 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7971000000000004e-05 2024-03-19T11:31:13.916 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.916 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.916 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:13.916 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281591+0000", 2024-03-19T11:31:13.916 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6704000000000004e-05 2024-03-19T11:31:13.916 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.916 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.916 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.916 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281865+0000", 2024-03-19T11:31:13.916 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000274097 2024-03-19T11:31:13.917 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.917 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.917 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:13.917 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281901+0000", 2024-03-19T11:31:13.917 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5777e-05 2024-03-19T11:31:13.917 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.917 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.917 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:13.917 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.301738+0000", 2024-03-19T11:31:13.917 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019837804000000001 2024-03-19T11:31:13.917 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.917 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:13.917 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.917 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.917 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.918 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.18 e75,74 4:1d58db11:::smithi03829856-18:2,48'76)", 2024-03-19T11:31:13.918 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.281661+0000", 2024-03-19T11:31:13.918 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 19.605079568000001, 2024-03-19T11:31:13.918 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020384337999999998, 2024-03-19T11:31:13.918 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:13.918 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:13.918 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:13.918 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.918 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:13.918 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281661+0000", 2024-03-19T11:31:13.918 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.918 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.918 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.918 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:13.919 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281661+0000", 2024-03-19T11:31:13.919 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.919 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.919 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.919 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:13.919 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281661+0000", 2024-03-19T11:31:13.919 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9100000000000003e-07 2024-03-19T11:31:13.919 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.919 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.919 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:13.919 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281668+0000", 2024-03-19T11:31:13.919 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8199999999999999e-06 2024-03-19T11:31:13.919 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.919 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.919 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:13.919 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281679+0000", 2024-03-19T11:31:13.920 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1107999999999999e-05 2024-03-19T11:31:13.920 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.920 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.920 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:13.920 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281691+0000", 2024-03-19T11:31:13.920 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1399999999999999e-05 2024-03-19T11:31:13.920 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.920 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.920 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.920 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.301804+0000", 2024-03-19T11:31:13.920 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020113669000000001 2024-03-19T11:31:13.920 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.920 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.920 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:13.920 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.301818+0000", 2024-03-19T11:31:13.921 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3718e-05 2024-03-19T11:31:13.921 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.921 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.921 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:13.921 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302045+0000", 2024-03-19T11:31:13.921 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000226832 2024-03-19T11:31:13.921 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.921 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:13.921 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.921 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.921 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.921 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1dabc0f1:::smithi03829856-1:head, version: 70'103, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:head@70'103, size: 2621892, copy_subset: [(0, 2621892)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:13.922 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.281783+0000", 2024-03-19T11:31:13.922 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 19.604957292000002, 2024-03-19T11:31:13.922 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026187221, 2024-03-19T11:31:13.922 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:13.922 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:13.922 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:13.922 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.922 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:13.922 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281783+0000", 2024-03-19T11:31:13.922 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.922 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.922 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.922 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:13.922 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281783+0000", 2024-03-19T11:31:13.922 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.923 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.923 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.923 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:13.923 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281783+0000", 2024-03-19T11:31:13.923 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1699999999999999e-07 2024-03-19T11:31:13.923 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.923 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.923 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:13.923 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.282962+0000", 2024-03-19T11:31:13.923 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001178601 2024-03-19T11:31:13.923 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.923 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.923 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:13.923 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.282987+0000", 2024-03-19T11:31:13.923 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4899e-05 2024-03-19T11:31:13.924 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.924 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.924 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:13.924 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283006+0000", 2024-03-19T11:31:13.924 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9494999999999999e-05 2024-03-19T11:31:13.924 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.924 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.924 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.924 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.305757+0000", 2024-03-19T11:31:13.924 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.022750570000000001 2024-03-19T11:31:13.924 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.924 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.925 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:13.925 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.305770+0000", 2024-03-19T11:31:13.925 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2791e-05 2024-03-19T11:31:13.925 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.925 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.925 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:13.925 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.307970+0000", 2024-03-19T11:31:13.925 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022005480000000001 2024-03-19T11:31:13.925 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.925 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:13.925 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.925 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.925 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.925 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.18 e75,74 4:1d58db11:::smithi03829856-18:c,70'104)", 2024-03-19T11:31:13.925 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.283036+0000", 2024-03-19T11:31:13.926 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 19.603704191999999, 2024-03-19T11:31:13.926 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019146032, 2024-03-19T11:31:13.926 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:13.926 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:13.926 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:13.926 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.926 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:13.926 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283036+0000", 2024-03-19T11:31:13.926 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.926 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.926 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.926 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:13.927 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283036+0000", 2024-03-19T11:31:13.927 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.927 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.927 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.927 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:13.927 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283037+0000", 2024-03-19T11:31:13.927 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4899999999999995e-07 2024-03-19T11:31:13.927 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.927 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.927 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:13.927 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283042+0000", 2024-03-19T11:31:13.927 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0930000000000002e-06 2024-03-19T11:31:13.927 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.928 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.928 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:13.928 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283053+0000", 2024-03-19T11:31:13.928 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1083999999999999e-05 2024-03-19T11:31:13.928 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.928 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.928 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:13.928 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283059+0000", 2024-03-19T11:31:13.928 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6329999999999999e-06 2024-03-19T11:31:13.928 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.928 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.928 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.928 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302090+0000", 2024-03-19T11:31:13.929 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01903038 2024-03-19T11:31:13.929 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.929 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.929 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:13.929 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302097+0000", 2024-03-19T11:31:13.929 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.221e-06 2024-03-19T11:31:13.929 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.929 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.929 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:13.929 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302182+0000", 2024-03-19T11:31:13.929 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5072000000000001e-05 2024-03-19T11:31:13.929 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.929 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:13.930 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.930 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.930 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.930 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(osd.5.0:143 4.18 e75/74 4:1d58db11:::smithi03829856-18:head v 75'106, mlcod=0'0)", 2024-03-19T11:31:13.930 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.283083+0000", 2024-03-19T11:31:13.930 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 19.603656865000001, 2024-03-19T11:31:13.930 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025309439, 2024-03-19T11:31:13.930 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:13.930 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:13.930 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:13.930 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.930 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:13.931 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283083+0000", 2024-03-19T11:31:13.931 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.931 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.931 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.931 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:13.931 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283083+0000", 2024-03-19T11:31:13.931 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.931 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.931 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.931 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:13.931 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283084+0000", 2024-03-19T11:31:13.931 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7800000000000002e-07 2024-03-19T11:31:13.931 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.932 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.932 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:13.932 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.284082+0000", 2024-03-19T11:31:13.932 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0009977880000000001 2024-03-19T11:31:13.932 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.932 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.932 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:13.932 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.284095+0000", 2024-03-19T11:31:13.932 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3256e-05 2024-03-19T11:31:13.932 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.932 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.932 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:13.932 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.284102+0000", 2024-03-19T11:31:13.932 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.605e-06 2024-03-19T11:31:13.933 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.933 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.933 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.933 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302285+0000", 2024-03-19T11:31:13.933 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018182423 2024-03-19T11:31:13.933 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.933 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.933 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:13.933 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302341+0000", 2024-03-19T11:31:13.933 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.626e-05 2024-03-19T11:31:13.933 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.933 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.933 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:13.933 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.308338+0000", 2024-03-19T11:31:13.934 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0059968030000000002 2024-03-19T11:31:13.934 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.934 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.934 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:13.934 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.308393+0000", 2024-03-19T11:31:13.934 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4826e-05 2024-03-19T11:31:13.934 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.934 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:13.934 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.934 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.934 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.934 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "pg_update_log_missing(4.18 epoch 75/74 rep_tid 144 entries 75'107 (0'0) error 4:1d58db11:::smithi03829856-18:head by client.4415.0:2283 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-19T11:31:13.934 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.288252+0000", 2024-03-19T11:31:13.935 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 19.598488315000001, 2024-03-19T11:31:13.935 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020167377, 2024-03-19T11:31:13.935 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:13.935 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-03-19T11:31:13.935 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:13.935 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.935 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:13.935 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288252+0000", 2024-03-19T11:31:13.935 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.935 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.935 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.935 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:13.935 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288252+0000", 2024-03-19T11:31:13.935 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.935 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.936 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.936 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:13.936 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288253+0000", 2024-03-19T11:31:13.936 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2109999999999999e-06 2024-03-19T11:31:13.936 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.936 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.936 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:13.936 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288268+0000", 2024-03-19T11:31:13.936 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5003e-05 2024-03-19T11:31:13.936 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.936 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.936 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:13.936 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288301+0000", 2024-03-19T11:31:13.936 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2539999999999997e-05 2024-03-19T11:31:13.936 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.937 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.937 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:13.937 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288319+0000", 2024-03-19T11:31:13.937 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8232000000000001e-05 2024-03-19T11:31:13.937 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.937 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.937 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.937 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.305490+0000", 2024-03-19T11:31:13.937 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017170857000000001 2024-03-19T11:31:13.937 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.937 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.937 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:13.937 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.308419+0000", 2024-03-19T11:31:13.937 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029295340000000001 2024-03-19T11:31:13.937 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.938 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:13.938 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.938 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.938 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.938 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1d58db11:::smithi03829856-18:6, version: 55'80, data_included: [2313652~310127,3128657~31026], data_size: 341153, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:6@55'80, size: 3159683, copy_subset: [(0, 3159683)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3159683, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2313652, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:1d58db11:::smithi03829856-18:b, version: 58'84, data_included: [1657391~966388,2633066~82188], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:b@58'84, size: 3445402, copy_subset: [(0, 3445402)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2715254, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1657391, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:10, version: 66'95, data_included: [1829043~668012,3134120~380564], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:10@66'95, size: 3764735, copy_subset: [(0, 3764735)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3514684, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:12, version: 66'97, data_included: [1829043~507467], data_size: 507467, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:12@66'97, size: 2504243, copy_subset: [(0, 2504243)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2504243, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:c, version: 70'102, data_included: [2014538~156250], data_size: 156250, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:c@70'102, size: 2857157, copy_subset: [(0, 2857157)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2857157, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2014538, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:13.938 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.312362+0000", 2024-03-19T11:31:13.938 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 19.574378266, 2024-03-19T11:31:13.938 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017968951, 2024-03-19T11:31:13.938 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:13.938 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:13.938 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:13.938 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.938 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:13.938 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.312362+0000", 2024-03-19T11:31:13.939 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.939 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.939 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.939 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:13.939 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.312362+0000", 2024-03-19T11:31:13.939 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.939 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.939 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.939 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:13.939 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.312363+0000", 2024-03-19T11:31:13.939 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.528e-06 2024-03-19T11:31:13.939 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.939 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.939 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:13.939 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315748+0000", 2024-03-19T11:31:13.940 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033842830000000001 2024-03-19T11:31:13.940 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.940 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.940 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:13.940 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315795+0000", 2024-03-19T11:31:13.940 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7382999999999997e-05 2024-03-19T11:31:13.940 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.940 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.940 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:13.940 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315845+0000", 2024-03-19T11:31:13.940 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0319999999999999e-05 2024-03-19T11:31:13.940 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.940 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.940 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.940 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.316123+0000", 2024-03-19T11:31:13.941 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00027747200000000002 2024-03-19T11:31:13.941 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.941 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.941 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:13.941 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.316170+0000", 2024-03-19T11:31:13.941 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7466999999999998e-05 2024-03-19T11:31:13.941 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.941 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.941 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:13.941 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.330331+0000", 2024-03-19T11:31:13.941 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014160498000000001 2024-03-19T11:31:13.941 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.941 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:13.941 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.941 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.942 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.942 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1dabc0f1:::smithi03829856-1:head, version: 70'103, data_included: [1829043~555153], data_size: 555153, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:head@70'103, size: 2621892, copy_subset: [(0, 2621892)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2621892, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:13.942 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.315898+0000", 2024-03-19T11:31:13.942 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 19.570842280000001, 2024-03-19T11:31:13.942 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016880424000000002, 2024-03-19T11:31:13.942 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:13.942 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:13.942 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:13.942 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.942 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:13.942 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315898+0000", 2024-03-19T11:31:13.942 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.942 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.942 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.943 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:13.943 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315898+0000", 2024-03-19T11:31:13.943 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.943 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.943 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.943 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:13.943 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315899+0000", 2024-03-19T11:31:13.943 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.003e-06 2024-03-19T11:31:13.943 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.943 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.943 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:13.943 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.316486+0000", 2024-03-19T11:31:13.943 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00058673099999999995 2024-03-19T11:31:13.943 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.943 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.944 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:13.944 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.316519+0000", 2024-03-19T11:31:13.944 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3124000000000003e-05 2024-03-19T11:31:13.944 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.944 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.944 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:13.944 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.316534+0000", 2024-03-19T11:31:13.944 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.485e-05 2024-03-19T11:31:13.944 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.944 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.944 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.944 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.330415+0000", 2024-03-19T11:31:13.944 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013880905000000001 2024-03-19T11:31:13.944 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.945 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.945 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:13.945 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.330439+0000", 2024-03-19T11:31:13.945 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4630999999999998e-05 2024-03-19T11:31:13.945 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.945 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.945 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:13.945 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.332778+0000", 2024-03-19T11:31:13.945 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023391800000000002 2024-03-19T11:31:13.945 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.945 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:13.945 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.945 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.945 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.945 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2387 4.2 4:43b32242:::smithi03829856-22:head [read 596992~684544] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-03-19T11:31:13.946 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:59.319806+0000", 2024-03-19T11:31:13.946 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 14.566934613000001, 2024-03-19T11:31:13.946 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.78144344399999999, 2024-03-19T11:31:13.946 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:13.946 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:13.946 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:13.946 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:13.946 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:13.946 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2387 2024-03-19T11:31:13.946 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.946 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:13.946 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.946 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:13.946 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319806+0000", 2024-03-19T11:31:13.947 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.947 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.947 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.947 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:13.947 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319806+0000", 2024-03-19T11:31:13.947 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.947 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.947 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.947 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:13.947 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319807+0000", 2024-03-19T11:31:13.947 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3519999999999999e-06 2024-03-19T11:31:13.947 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.947 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.948 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:13.948 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319812+0000", 2024-03-19T11:31:13.948 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3959999999999997e-06 2024-03-19T11:31:13.948 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.948 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.948 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:13.948 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319827+0000", 2024-03-19T11:31:13.948 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4566000000000001e-05 2024-03-19T11:31:13.948 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.948 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.948 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:13.948 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319835+0000", 2024-03-19T11:31:13.948 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.994e-06 2024-03-19T11:31:13.948 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.949 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.949 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.949 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.320119+0000", 2024-03-19T11:31:13.949 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00028445199999999998 2024-03-19T11:31:13.949 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.949 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.949 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:13.949 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.320129+0000", 2024-03-19T11:31:13.949 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9580000000000002e-06 2024-03-19T11:31:13.949 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.949 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.949 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.949 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.069842+0000", 2024-03-19T11:31:13.949 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.74971218399999995 2024-03-19T11:31:13.949 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.950 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.950 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:31:13.950 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.070010+0000", 2024-03-19T11:31:13.950 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016846600000000001 2024-03-19T11:31:13.950 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.950 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.950 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.950 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.090911+0000", 2024-03-19T11:31:13.950 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020901494999999999 2024-03-19T11:31:13.950 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.950 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.950 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:31:13.950 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.091249+0000", 2024-03-19T11:31:13.950 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000337602 2024-03-19T11:31:13.951 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.951 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.951 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.951 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.098478+0000", 2024-03-19T11:31:13.951 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0072287549999999999 2024-03-19T11:31:13.951 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.951 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.951 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:13.951 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.098617+0000", 2024-03-19T11:31:13.951 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013871599999999999 2024-03-19T11:31:13.951 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.951 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.951 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:13.951 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.101249+0000", 2024-03-19T11:31:13.952 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002632508 2024-03-19T11:31:13.952 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.952 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:13.952 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.952 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.952 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.952 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 81/80 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 15=[]:{7: [6], b: [b], 10: [d], 12: [12, 11], 15: [15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:7, version: 55'82, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@55'82, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 16=[]:{7: [6], b: [b], 10: [d], 16: [16, 15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 15=[]:{7: [6], b: [b], 10: [d], 12: [12, 11], 15: [15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 58'86, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@58'86, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 16=[]:{7: [6], b: [b], 10: [d], 16: [16, 15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [504210~968657,1572695~79919], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 16=[]:{7: [6], b: [b], 10: [d], 16: [16, 15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:13.952 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:00.089924+0000", 2024-03-19T11:31:13.952 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 13.796816384, 2024-03-19T11:31:13.952 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024686583000000002, 2024-03-19T11:31:13.952 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:13.952 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:13.952 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:13.952 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.952 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:13.953 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.089924+0000", 2024-03-19T11:31:13.953 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.953 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.953 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.953 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:13.953 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.089924+0000", 2024-03-19T11:31:13.953 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.953 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.953 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.953 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:13.953 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.089926+0000", 2024-03-19T11:31:13.953 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9589999999999998e-06 2024-03-19T11:31:13.953 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.953 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.954 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:13.954 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093311+0000", 2024-03-19T11:31:13.954 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033854409999999999 2024-03-19T11:31:13.954 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.954 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.954 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:13.954 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093365+0000", 2024-03-19T11:31:13.954 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3795000000000002e-05 2024-03-19T11:31:13.954 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.954 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.954 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:13.954 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093438+0000", 2024-03-19T11:31:13.954 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2591999999999999e-05 2024-03-19T11:31:13.954 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.955 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.955 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.955 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093690+0000", 2024-03-19T11:31:13.955 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025264999999999999 2024-03-19T11:31:13.955 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.955 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.955 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:13.955 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093731+0000", 2024-03-19T11:31:13.955 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0881999999999999e-05 2024-03-19T11:31:13.955 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.955 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.955 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:13.955 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.114610+0000", 2024-03-19T11:31:13.955 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020879264000000002 2024-03-19T11:31:13.955 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.956 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:13.956 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.956 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.956 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.956 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.21 e81,80 4:8401c1f0:::smithi03829856-43:8,46'66,4:866f3dcc:::smithi03829856-31:2,47'73)", 2024-03-19T11:31:13.956 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:00.093514+0000", 2024-03-19T11:31:13.956 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 13.793225784000001, 2024-03-19T11:31:13.956 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027375713999999999, 2024-03-19T11:31:13.956 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:13.956 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:13.956 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:13.956 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.956 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:13.956 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093514+0000", 2024-03-19T11:31:13.956 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.957 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.957 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.957 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:13.957 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093514+0000", 2024-03-19T11:31:13.957 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.957 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.957 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.957 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:13.957 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093516+0000", 2024-03-19T11:31:13.957 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1179999999999999e-06 2024-03-19T11:31:13.957 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.957 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.957 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:13.957 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093523+0000", 2024-03-19T11:31:13.958 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7500000000000003e-06 2024-03-19T11:31:13.958 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.958 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.958 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:13.958 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093534+0000", 2024-03-19T11:31:13.958 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0831e-05 2024-03-19T11:31:13.958 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.958 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.958 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:13.958 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093542+0000", 2024-03-19T11:31:13.958 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0139999999999998e-06 2024-03-19T11:31:13.958 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.958 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.959 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.959 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.120682+0000", 2024-03-19T11:31:13.959 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027139838999999999 2024-03-19T11:31:13.959 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.959 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.959 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:13.959 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.120693+0000", 2024-03-19T11:31:13.959 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.136e-05 2024-03-19T11:31:13.959 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.959 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.959 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:13.959 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.120890+0000", 2024-03-19T11:31:13.959 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019680199999999999 2024-03-19T11:31:13.959 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.959 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:13.960 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.960 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.960 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.960 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 81/80 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 15=[]:{7: [6], b: [b], 10: [d], 12: [12, 11], 15: [15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:7, version: 55'82, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@55'82, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 16=[]:{7: [6], b: [b], 10: [d], 16: [16, 15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 15=[]:{7: [6], b: [b], 10: [d], 12: [12, 11], 15: [15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 58'86, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@58'86, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 16=[]:{7: [6], b: [b], 10: [d], 16: [16, 15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [1652614~973634,2888511~52681,3182305~22261], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 16=[]:{7: [6], b: [b], 10: [d], 16: [16, 15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3204566, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:13.960 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:00.126855+0000", 2024-03-19T11:31:13.960 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 13.759885047000001, 2024-03-19T11:31:13.960 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016438076999999999, 2024-03-19T11:31:13.960 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:13.960 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:13.960 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:13.960 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.960 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:13.960 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.126855+0000", 2024-03-19T11:31:13.960 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.961 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.961 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.961 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:13.961 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.126855+0000", 2024-03-19T11:31:13.961 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.961 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.961 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.961 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:13.961 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.126857+0000", 2024-03-19T11:31:13.961 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.739e-06 2024-03-19T11:31:13.961 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.961 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.961 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:13.961 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.129236+0000", 2024-03-19T11:31:13.961 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023792449999999999 2024-03-19T11:31:13.962 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.962 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.962 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:13.962 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.129298+0000", 2024-03-19T11:31:13.962 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2329000000000006e-05 2024-03-19T11:31:13.962 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.962 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.962 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:13.962 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.129365+0000", 2024-03-19T11:31:13.962 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6299999999999999e-05 2024-03-19T11:31:13.962 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.962 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.962 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.963 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.132493+0000", 2024-03-19T11:31:13.963 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031279530000000002 2024-03-19T11:31:13.963 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.963 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.963 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:13.963 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.132578+0000", 2024-03-19T11:31:13.963 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5648000000000004e-05 2024-03-19T11:31:13.963 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.963 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.963 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:13.963 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.143293+0000", 2024-03-19T11:31:13.963 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010714863 2024-03-19T11:31:13.963 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.963 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:13.964 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.964 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.964 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.964 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.29 81/80 [PushOp(4:953e720b:::smithi03829856-45:10, version: 70'58, data_included: [1927574~595528,2570780~420833], data_size: 1016361, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:953e720b:::smithi03829856-45:10@70'58, size: 2991613, copy_subset: [(0, 2991613)], clone_subset: {}, snapset: 10=[]:{b: [b, 6], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2991613, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1927574, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:953e720b:::smithi03829856-45:head, version: 71'60, data_included: [1823397~664039], data_size: 664039, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:953e720b:::smithi03829856-45:head@71'60, size: 2487436, copy_subset: [(0, 2487436)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2487436, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1823397, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:953e720b:::smithi03829856-45:b, version: 77'61, data_included: [1927574~595528,2570780~420833], data_size: 1016361, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:953e720b:::smithi03829856-45:b@77'61, size: 2991613, copy_subset: [(0, 2991613)], clone_subset: {}, snapset: 10=[]:{b: [b, 6], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2991613, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1927574, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:13.964 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:00.151568+0000", 2024-03-19T11:31:13.964 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 13.735171933, 2024-03-19T11:31:13.964 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015354197999999999, 2024-03-19T11:31:13.964 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:13.964 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:13.964 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:13.964 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.964 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:13.964 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.151568+0000", 2024-03-19T11:31:13.964 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.964 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.965 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.965 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:13.965 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.151568+0000", 2024-03-19T11:31:13.965 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:13.965 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.965 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.965 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:13.965 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.151569+0000", 2024-03-19T11:31:13.965 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.17e-06 2024-03-19T11:31:13.965 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.965 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.965 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:13.965 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.156296+0000", 2024-03-19T11:31:13.965 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0047264619999999999 2024-03-19T11:31:13.965 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.966 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.966 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:13.966 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.156334+0000", 2024-03-19T11:31:13.966 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8152000000000003e-05 2024-03-19T11:31:13.966 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.966 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.966 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:13.966 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.156370+0000", 2024-03-19T11:31:13.966 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6304999999999999e-05 2024-03-19T11:31:13.966 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.966 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.966 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:13.966 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.157994+0000", 2024-03-19T11:31:13.966 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0016235399999999999 2024-03-19T11:31:13.967 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.967 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.967 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:13.967 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.158017+0000", 2024-03-19T11:31:13.967 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3197000000000002e-05 2024-03-19T11:31:13.967 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:13.967 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:13.967 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:13.967 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.166922+0000", 2024-03-19T11:31:13.967 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0089053719999999999 2024-03-19T11:31:13.967 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.967 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:13.967 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.967 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:13.967 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:13.968 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:13.969 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-19T11:31:13.969 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:13.977+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:13.986 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:31:13.987 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-03-19T11:31:14.005 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":132,"stamp":"2024-03-19T11:31:13.060224+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":424065246,"num_objects":279,"num_object_clones":80,"num_object_copies":558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":276,"num_whiteouts":8,"num_read":1336,"num_read_kb":1043961,"num_write":853,"num_write_kb":275531,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":257,"num_bytes_recovered":270645919,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":83,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3264,"ondisk_log_size":3264,"up":110,"acting":110,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":116,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":781252,"kb_used_data":530376,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754193468,"statfs":{"total":773094113280,"available":772294111232,"internally_reserved":0,"allocated":543105024,"data_stored":610160259,"data_compressed":15952801,"data_compressed_allocated":16072704,"data_compressed_original":36630528,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3360416,"num_objects":-2,"num_object_clones":-5,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":1,"num_read":114,"num_read_kb":154918,"num_write":53,"num_write_kb":19225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2,"num_bytes_recovered":-2800509,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-8,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.008985"},"pg_stats":[{"pgid":"4.29","version":"85'65","reported_seq":448,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.074347+0000","last_change":"2024-03-19T11:31:08.075407+0000","last_active":"2024-03-19T11:31:08.074347+0000","last_peered":"2024-03-19T11:31:08.074347+0000","last_clean":"2024-03-19T11:31:08.074347+0000","last_became_active":"2024-03-19T11:31:00.070287+0000","last_became_peered":"2024-03-19T11:31:00.070287+0000","last_unstale":"2024-03-19T11:31:08.074347+0000","last_undegraded":"2024-03-19T11:31:08.074347+0000","last_fullsized":"2024-03-19T11:31:08.074347+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":6,"last_scrub":"71'60","last_scrub_stamp":"2024-03-19T11:30:52.748479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:52.748479+0000","objects_scrubbed":3,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:14.667721+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.015091184000000001,"stat_sum":{"num_bytes":4387250,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":19,"num_read_kb":23171,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":5842489,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.28","version":"89'103","reported_seq":408,"reported_epoch":89,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:08.114830+0000","last_change":"2024-03-19T11:31:08.114830+0000","last_active":"2024-03-19T11:31:08.114830+0000","last_peered":"2024-03-19T11:31:08.114830+0000","last_clean":"2024-03-19T11:31:08.114830+0000","last_became_active":"2024-03-19T11:30:45.257600+0000","last_became_peered":"2024-03-19T11:30:45.257600+0000","last_unstale":"2024-03-19T11:31:08.114830+0000","last_undegraded":"2024-03-19T11:31:08.114830+0000","last_fullsized":"2024-03-19T11:31:08.114830+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":6,"last_scrub":"71'85","last_scrub_stamp":"2024-03-19T11:30:52.343454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:52.343454+0000","objects_scrubbed":7,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:57.454276+0000","scrub_duration":7,"objects_trimmed":1,"snaptrim_duration":0.065578227000000003,"stat_sum":{"num_bytes":9128542,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"2"}]},{"pgid":"4.27","version":"87'56","reported_seq":79,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.059325+0000","last_change":"2024-03-19T11:31:08.060066+0000","last_active":"2024-03-19T11:31:08.059325+0000","last_peered":"2024-03-19T11:31:08.059325+0000","last_clean":"2024-03-19T11:31:08.059325+0000","last_became_active":"2024-03-19T11:31:00.070337+0000","last_became_peered":"2024-03-19T11:31:00.070337+0000","last_unstale":"2024-03-19T11:31:08.059325+0000","last_undegraded":"2024-03-19T11:31:08.059325+0000","last_fullsized":"2024-03-19T11:31:08.059325+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":6,"last_scrub":"71'48","last_scrub_stamp":"2024-03-19T11:30:55.467716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:55.467716+0000","objects_scrubbed":4,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:09.107718+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0027578379999999999,"stat_sum":{"num_bytes":5844083,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":11198,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":7070511,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.26","version":"61'63","reported_seq":70,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T11:31:12.135363+0000","last_change":"2024-03-19T11:31:08.115454+0000","last_active":"2024-03-19T11:31:12.135363+0000","last_peered":"2024-03-19T11:31:12.135363+0000","last_clean":"2024-03-19T11:31:12.135363+0000","last_became_active":"2024-03-19T11:30:59.074337+0000","last_became_peered":"2024-03-19T11:30:59.074337+0000","last_unstale":"2024-03-19T11:31:12.135363+0000","last_undegraded":"2024-03-19T11:31:12.135363+0000","last_fullsized":"2024-03-19T11:31:12.135363+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":6,"last_scrub":"61'63","last_scrub_stamp":"2024-03-19T11:30:52.406723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:52.406723+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:57.163804+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.064661005999999993,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.25","version":"56'66","reported_seq":83,"reported_epoch":89,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:08.114708+0000","last_change":"2024-03-19T11:31:08.114800+0000","last_active":"2024-03-19T11:31:08.114708+0000","last_peered":"2024-03-19T11:31:08.114708+0000","last_clean":"2024-03-19T11:31:08.114708+0000","last_became_active":"2024-03-19T11:30:53.084473+0000","last_became_peered":"2024-03-19T11:30:53.084473+0000","last_unstale":"2024-03-19T11:31:08.114708+0000","last_undegraded":"2024-03-19T11:31:08.114708+0000","last_fullsized":"2024-03-19T11:31:08.114708+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":6,"last_scrub":"56'66","last_scrub_stamp":"2024-03-19T11:31:06.587201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:31:06.587201+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:10.867475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.065786145000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"}]},{"pgid":"4.24","version":"85'117","reported_seq":510,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.107628+0000","last_change":"2024-03-19T11:31:08.069824+0000","last_active":"2024-03-19T11:31:08.107628+0000","last_peered":"2024-03-19T11:31:08.107628+0000","last_clean":"2024-03-19T11:31:08.107628+0000","last_became_active":"2024-03-19T11:31:03.063705+0000","last_became_peered":"2024-03-19T11:31:03.063705+0000","last_unstale":"2024-03-19T11:31:08.107628+0000","last_undegraded":"2024-03-19T11:31:08.107628+0000","last_fullsized":"2024-03-19T11:31:08.107628+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":6,"last_scrub":"84'113","last_scrub_stamp":"2024-03-19T11:31:03.304881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:31:03.304881+0000","objects_scrubbed":5,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:06.612630+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.011640286999999999,"stat_sum":{"num_bytes":10102023,"num_objects":4,"num_object_clones":2,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":34100,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.23","version":"87'38","reported_seq":87,"reported_epoch":90,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:09.056901+0000","last_change":"2024-03-19T11:31:06.101937+0000","last_active":"2024-03-19T11:31:09.056901+0000","last_peered":"2024-03-19T11:31:09.056901+0000","last_clean":"2024-03-19T11:31:09.056901+0000","last_became_active":"2024-03-19T11:30:53.076092+0000","last_became_peered":"2024-03-19T11:30:53.076092+0000","last_unstale":"2024-03-19T11:31:09.056901+0000","last_undegraded":"2024-03-19T11:31:09.056901+0000","last_fullsized":"2024-03-19T11:31:09.056901+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":6,"last_scrub":"62'23","last_scrub_stamp":"2024-03-19T11:30:43.163509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:43.163509+0000","objects_scrubbed":2,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:10.968084+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.035799641,"stat_sum":{"num_bytes":6154918,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":10,"num_write_kb":2870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2833035,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"6","length":"1"},{"start":"11","length":"1"}]},{"pgid":"4.22","version":"28'6","reported_seq":86,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.075051+0000","last_change":"2024-03-19T11:31:08.076088+0000","last_active":"2024-03-19T11:31:08.075051+0000","last_peered":"2024-03-19T11:31:08.075051+0000","last_clean":"2024-03-19T11:31:08.075051+0000","last_became_active":"2024-03-19T11:30:53.068010+0000","last_became_peered":"2024-03-19T11:30:53.068010+0000","last_unstale":"2024-03-19T11:31:08.075051+0000","last_undegraded":"2024-03-19T11:31:08.075051+0000","last_fullsized":"2024-03-19T11:31:08.075051+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":6,"last_scrub":"28'6","last_scrub_stamp":"2024-03-19T11:30:55.724671+0000","last_deep_scrub":"28'6","last_deep_scrub_stamp":"2024-03-19T11:30:55.724671+0000","last_clean_scrub_stamp":"2024-03-19T11:30:55.724671+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:13.244591+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.015426808,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.21","version":"89'133","reported_seq":524,"reported_epoch":90,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:09.056335+0000","last_change":"2024-03-19T11:31:08.112958+0000","last_active":"2024-03-19T11:31:09.056335+0000","last_peered":"2024-03-19T11:31:09.056335+0000","last_clean":"2024-03-19T11:31:09.056335+0000","last_became_active":"2024-03-19T11:31:00.069759+0000","last_became_peered":"2024-03-19T11:31:00.069759+0000","last_unstale":"2024-03-19T11:31:09.056335+0000","last_undegraded":"2024-03-19T11:31:09.056335+0000","last_fullsized":"2024-03-19T11:31:09.056335+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":6,"last_scrub":"62'96","last_scrub_stamp":"2024-03-19T11:30:43.400885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:43.400885+0000","objects_scrubbed":9,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:12.660708+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.050771269000000001,"stat_sum":{"num_bytes":16636184,"num_objects":7,"num_object_clones":5,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":49,"num_read_kb":51400,"num_write":23,"num_write_kb":6523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":22983003,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d","length":"1"}]},{"pgid":"4.20","version":"56'46","reported_seq":78,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.116754+0000","last_change":"2024-03-19T11:31:08.117528+0000","last_active":"2024-03-19T11:31:08.116754+0000","last_peered":"2024-03-19T11:31:08.116754+0000","last_clean":"2024-03-19T11:31:08.116754+0000","last_became_active":"2024-03-19T11:30:53.067251+0000","last_became_peered":"2024-03-19T11:30:53.067251+0000","last_unstale":"2024-03-19T11:31:08.116754+0000","last_undegraded":"2024-03-19T11:31:08.116754+0000","last_fullsized":"2024-03-19T11:31:08.116754+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":6,"last_scrub":"56'46","last_scrub_stamp":"2024-03-19T11:30:41.767100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:41.767100+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:04.278716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.032365520000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.1f","version":"0'0","reported_seq":86,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.115029+0000","last_change":"2024-03-19T11:31:08.116595+0000","last_active":"2024-03-19T11:31:08.115029+0000","last_peered":"2024-03-19T11:31:08.115029+0000","last_clean":"2024-03-19T11:31:08.115029+0000","last_became_active":"2024-03-19T11:30:53.087211+0000","last_became_peered":"2024-03-19T11:30:53.087211+0000","last_unstale":"2024-03-19T11:31:08.115029+0000","last_undegraded":"2024-03-19T11:31:08.115029+0000","last_fullsized":"2024-03-19T11:31:08.115029+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:30:42.377963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.377963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:05.818292+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.032679030999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.1e","version":"87'71","reported_seq":116,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T11:31:12.135393+0000","last_change":"2024-03-19T11:31:12.123738+0000","last_active":"2024-03-19T11:31:12.135393+0000","last_peered":"2024-03-19T11:31:12.135393+0000","last_clean":"2024-03-19T11:31:12.135393+0000","last_became_active":"2024-03-19T11:30:53.084452+0000","last_became_peered":"2024-03-19T11:30:53.084452+0000","last_unstale":"2024-03-19T11:31:12.135393+0000","last_undegraded":"2024-03-19T11:31:12.135393+0000","last_fullsized":"2024-03-19T11:31:12.135393+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"64'62","last_scrub_stamp":"2024-03-19T11:30:42.778159+0000","last_deep_scrub":"64'62","last_deep_scrub_stamp":"2024-03-19T11:30:42.778159+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.778159+0000","objects_scrubbed":4,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:11.879105+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":1.046110673,"stat_sum":{"num_bytes":3794928,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":27,"num_read_kb":28503,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3779426,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.1d","version":"87'67","reported_seq":81,"reported_epoch":90,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:09.056383+0000","last_change":"2024-03-19T11:31:06.105861+0000","last_active":"2024-03-19T11:31:09.056383+0000","last_peered":"2024-03-19T11:31:09.056383+0000","last_clean":"2024-03-19T11:31:09.056383+0000","last_became_active":"2024-03-19T11:30:53.067748+0000","last_became_peered":"2024-03-19T11:30:53.067748+0000","last_unstale":"2024-03-19T11:31:09.056383+0000","last_undegraded":"2024-03-19T11:31:09.056383+0000","last_fullsized":"2024-03-19T11:31:09.056383+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"61'59","last_scrub_stamp":"2024-03-19T11:30:42.601863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.601863+0000","objects_scrubbed":2,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:45.377695+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.039756313000000001,"stat_sum":{"num_bytes":4835736,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":3,"num_read_kb":61,"num_write":5,"num_write_kb":1341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2817390,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"6","length":"1"},{"start":"11","length":"1"}]},{"pgid":"4.1c","version":"85'35","reported_seq":98,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T11:31:09.057786+0000","last_change":"2024-03-19T11:31:08.052486+0000","last_active":"2024-03-19T11:31:09.057786+0000","last_peered":"2024-03-19T11:31:09.057786+0000","last_clean":"2024-03-19T11:31:09.057786+0000","last_became_active":"2024-03-19T11:30:53.082655+0000","last_became_peered":"2024-03-19T11:30:53.082655+0000","last_unstale":"2024-03-19T11:31:09.057786+0000","last_undegraded":"2024-03-19T11:31:09.057786+0000","last_fullsized":"2024-03-19T11:31:09.057786+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"62'31","last_scrub_stamp":"2024-03-19T11:30:42.771396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.771396+0000","objects_scrubbed":3,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:53.311410+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0013764719999999999,"stat_sum":{"num_bytes":3210104,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":47464,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":5034309,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.1b","version":"89'100","reported_seq":92,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.105337+0000","last_change":"2024-03-19T11:31:08.108313+0000","last_active":"2024-03-19T11:31:08.105337+0000","last_peered":"2024-03-19T11:31:08.105337+0000","last_clean":"2024-03-19T11:31:08.105337+0000","last_became_active":"2024-03-19T11:31:03.056769+0000","last_became_peered":"2024-03-19T11:31:03.056769+0000","last_unstale":"2024-03-19T11:31:08.105337+0000","last_undegraded":"2024-03-19T11:31:08.105337+0000","last_fullsized":"2024-03-19T11:31:08.105337+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":5,"last_scrub":"62'62","last_scrub_stamp":"2024-03-19T11:30:42.181424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.181424+0000","objects_scrubbed":5,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:53.941736+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.040344087000000001,"stat_sum":{"num_bytes":14818653,"num_objects":7,"num_object_clones":4,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":58,"num_read_kb":68803,"num_write":11,"num_write_kb":3105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":17966556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":6,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.1a","version":"89'111","reported_seq":83,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.082577+0000","last_change":"2024-03-19T11:31:08.083422+0000","last_active":"2024-03-19T11:31:08.082577+0000","last_peered":"2024-03-19T11:31:08.082577+0000","last_clean":"2024-03-19T11:31:08.082577+0000","last_became_active":"2024-03-19T11:31:03.056520+0000","last_became_peered":"2024-03-19T11:31:03.056520+0000","last_unstale":"2024-03-19T11:31:08.082577+0000","last_undegraded":"2024-03-19T11:31:08.082577+0000","last_fullsized":"2024-03-19T11:31:08.082577+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":5,"last_scrub":"61'76","last_scrub_stamp":"2024-03-19T11:30:41.433501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:41.433501+0000","objects_scrubbed":5,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:46.519318+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.021991492000000001,"stat_sum":{"num_bytes":11953834,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":22,"num_read_kb":17414,"num_write":18,"num_write_kb":5314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":16291222,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.19","version":"89'117","reported_seq":496,"reported_epoch":90,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:09.056559+0000","last_change":"2024-03-19T11:31:08.110132+0000","last_active":"2024-03-19T11:31:09.056559+0000","last_peered":"2024-03-19T11:31:09.056559+0000","last_clean":"2024-03-19T11:31:09.056559+0000","last_became_active":"2024-03-19T11:30:53.076188+0000","last_became_peered":"2024-03-19T11:30:53.076188+0000","last_unstale":"2024-03-19T11:31:09.056559+0000","last_undegraded":"2024-03-19T11:31:09.056559+0000","last_fullsized":"2024-03-19T11:31:09.056559+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"41'37","last_scrub_stamp":"2024-03-19T11:30:27.705159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:27.705159+0000","objects_scrubbed":4,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:28.833088+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.047949758000000002,"stat_sum":{"num_bytes":18546999,"num_objects":13,"num_object_clones":9,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":99,"num_read_kb":113713,"num_write":39,"num_write_kb":6633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":21870393,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":11,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"4.18","version":"88'138","reported_seq":440,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.074905+0000","last_change":"2024-03-19T11:31:08.074905+0000","last_active":"2024-03-19T11:31:08.074905+0000","last_peered":"2024-03-19T11:31:08.074905+0000","last_clean":"2024-03-19T11:31:08.074905+0000","last_became_active":"2024-03-19T11:31:06.066324+0000","last_became_peered":"2024-03-19T11:31:06.066324+0000","last_unstale":"2024-03-19T11:31:08.074905+0000","last_undegraded":"2024-03-19T11:31:08.074905+0000","last_fullsized":"2024-03-19T11:31:08.074905+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":5,"last_scrub":"48'77","last_scrub_stamp":"2024-03-19T11:30:29.399762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:29.399762+0000","objects_scrubbed":6,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:55.801600+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.014795552,"stat_sum":{"num_bytes":18257745,"num_objects":7,"num_object_clones":5,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":49,"num_read_kb":49877,"num_write":22,"num_write_kb":6328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":19971477,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.17","version":"41'24","reported_seq":74,"reported_epoch":90,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:09.057390+0000","last_change":"2024-03-19T11:31:06.106323+0000","last_active":"2024-03-19T11:31:09.057390+0000","last_peered":"2024-03-19T11:31:09.057390+0000","last_clean":"2024-03-19T11:31:09.057390+0000","last_became_active":"2024-03-19T11:30:53.074270+0000","last_became_peered":"2024-03-19T11:30:53.074270+0000","last_unstale":"2024-03-19T11:31:09.057390+0000","last_undegraded":"2024-03-19T11:31:09.057390+0000","last_fullsized":"2024-03-19T11:31:09.057390+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"41'24","last_scrub_stamp":"2024-03-19T11:30:53.993428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:53.993428+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:22.671214+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.039101238000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"6","length":"1"},{"start":"11","length":"1"}]},{"pgid":"4.16","version":"92'122","reported_seq":121,"reported_epoch":93,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:12.135253+0000","last_change":"2024-03-19T11:31:11.101970+0000","last_active":"2024-03-19T11:31:12.135253+0000","last_peered":"2024-03-19T11:31:12.135253+0000","last_clean":"2024-03-19T11:31:12.135253+0000","last_became_active":"2024-03-19T11:30:59.074063+0000","last_became_peered":"2024-03-19T11:30:59.074063+0000","last_unstale":"2024-03-19T11:31:12.135253+0000","last_undegraded":"2024-03-19T11:31:12.135253+0000","last_fullsized":"2024-03-19T11:31:12.135253+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":5,"last_scrub":"49'56","last_scrub_stamp":"2024-03-19T11:30:29.212766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:29.212766+0000","objects_scrubbed":5,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:40.544381+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.023712351999999999,"stat_sum":{"num_bytes":13644045,"num_objects":7,"num_object_clones":4,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":124,"num_read_kb":134085,"num_write":27,"num_write_kb":7671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":24165549,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":7,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"2"}]},{"pgid":"4.15","version":"89'177","reported_seq":596,"reported_epoch":89,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:08.117471+0000","last_change":"2024-03-19T11:31:06.107463+0000","last_active":"2024-03-19T11:31:08.117471+0000","last_peered":"2024-03-19T11:31:08.117471+0000","last_clean":"2024-03-19T11:31:08.117471+0000","last_became_active":"2024-03-19T11:30:56.063100+0000","last_became_peered":"2024-03-19T11:30:56.063100+0000","last_unstale":"2024-03-19T11:31:08.117471+0000","last_undegraded":"2024-03-19T11:31:08.117471+0000","last_fullsized":"2024-03-19T11:31:08.117471+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":5,"last_scrub":"49'74","last_scrub_stamp":"2024-03-19T11:30:29.457542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:29.457542+0000","objects_scrubbed":7,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:42.862571+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.028232337999999999,"stat_sum":{"num_bytes":26876675,"num_objects":17,"num_object_clones":12,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":1,"num_read":142,"num_read_kb":156856,"num_write":52,"num_write_kb":12752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":18744415,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":11,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"d","length":"1"},{"start":"11","length":"1"}]},{"pgid":"4.14","version":"40'56","reported_seq":239,"reported_epoch":90,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:09.057353+0000","last_change":"2024-03-19T11:31:08.110146+0000","last_active":"2024-03-19T11:31:09.057353+0000","last_peered":"2024-03-19T11:31:09.057353+0000","last_clean":"2024-03-19T11:31:09.057353+0000","last_became_active":"2024-03-19T11:31:00.065648+0000","last_became_peered":"2024-03-19T11:31:00.065648+0000","last_unstale":"2024-03-19T11:31:09.057353+0000","last_undegraded":"2024-03-19T11:31:09.057353+0000","last_fullsized":"2024-03-19T11:31:09.057353+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":5,"last_scrub":"40'56","last_scrub_stamp":"2024-03-19T11:30:54.998282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:54.998282+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:12.541661+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.046980873999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"}]},{"pgid":"4.13","version":"89'38","reported_seq":140,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.143741+0000","last_change":"2024-03-19T11:31:08.107232+0000","last_active":"2024-03-19T11:31:08.143741+0000","last_peered":"2024-03-19T11:31:08.143741+0000","last_clean":"2024-03-19T11:31:08.143741+0000","last_became_active":"2024-03-19T11:30:45.245367+0000","last_became_peered":"2024-03-19T11:30:45.245367+0000","last_unstale":"2024-03-19T11:31:08.143741+0000","last_undegraded":"2024-03-19T11:31:08.143741+0000","last_fullsized":"2024-03-19T11:31:08.143741+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":5,"last_scrub":"48'22","last_scrub_stamp":"2024-03-19T11:30:27.183294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:27.183294+0000","objects_scrubbed":2,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:41.607013+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.039325774000000001,"stat_sum":{"num_bytes":9406330,"num_objects":4,"num_object_clones":2,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":35,"num_read_kb":30733,"num_write":15,"num_write_kb":4523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":3001789,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.12","version":"28'6","reported_seq":65,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.057864+0000","last_change":"2024-03-19T11:31:08.058401+0000","last_active":"2024-03-19T11:31:08.057864+0000","last_peered":"2024-03-19T11:31:08.057864+0000","last_clean":"2024-03-19T11:31:08.057864+0000","last_became_active":"2024-03-19T11:31:00.064380+0000","last_became_peered":"2024-03-19T11:31:00.064380+0000","last_unstale":"2024-03-19T11:31:08.057864+0000","last_undegraded":"2024-03-19T11:31:08.057864+0000","last_fullsized":"2024-03-19T11:31:08.057864+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":5,"last_scrub":"28'6","last_scrub_stamp":"2024-03-19T11:31:04.263428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:31:04.263428+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:20.192174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020212310000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.11","version":"88'84","reported_seq":201,"reported_epoch":90,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:09.056453+0000","last_change":"2024-03-19T11:31:08.112910+0000","last_active":"2024-03-19T11:31:09.056453+0000","last_peered":"2024-03-19T11:31:09.056453+0000","last_clean":"2024-03-19T11:31:09.056453+0000","last_became_active":"2024-03-19T11:30:30.052184+0000","last_became_peered":"2024-03-19T11:30:30.052184+0000","last_unstale":"2024-03-19T11:31:09.056453+0000","last_undegraded":"2024-03-19T11:31:09.056453+0000","last_fullsized":"2024-03-19T11:31:09.056453+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":5,"last_scrub":"47'60","last_scrub_stamp":"2024-03-19T11:30:30.780348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:30.780348+0000","objects_scrubbed":2,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:43.351932+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.051827466000000003,"stat_sum":{"num_bytes":8084512,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":25443,"num_write":14,"num_write_kb":4028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"6","length":"1"},{"start":"11","length":"1"}]},{"pgid":"4.10","version":"85'34","reported_seq":358,"reported_epoch":90,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:09.056610+0000","last_change":"2024-03-19T11:31:06.105203+0000","last_active":"2024-03-19T11:31:09.056610+0000","last_peered":"2024-03-19T11:31:09.056610+0000","last_clean":"2024-03-19T11:31:09.056610+0000","last_became_active":"2024-03-19T11:31:00.065617+0000","last_became_peered":"2024-03-19T11:31:00.065617+0000","last_unstale":"2024-03-19T11:31:09.056610+0000","last_undegraded":"2024-03-19T11:31:09.056610+0000","last_fullsized":"2024-03-19T11:31:09.056610+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":5,"last_scrub":"50'30","last_scrub_stamp":"2024-03-19T11:30:28.787058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:28.787058+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:43.836722+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.038831954000000002,"stat_sum":{"num_bytes":3462209,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":20772,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":7102596,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"6","length":"1"},{"start":"11","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":146,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T11:31:09.057749+0000","last_change":"2024-03-19T11:31:08.381703+0000","last_active":"2024-03-19T11:31:09.057749+0000","last_peered":"2024-03-19T11:31:09.057749+0000","last_clean":"2024-03-19T11:31:09.057749+0000","last_became_active":"2024-03-19T11:30:03.276220+0000","last_became_peered":"2024-03-19T11:30:03.276220+0000","last_unstale":"2024-03-19T11:31:09.057749+0000","last_undegraded":"2024-03-19T11:31:09.057749+0000","last_fullsized":"2024-03-19T11:31:09.057749+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:31:08.381559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:31:08.381559+0000","last_clean_scrub_stamp":"2024-03-19T11:31:08.381559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:30.325139+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"59'92","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.071050+0000","last_change":"2024-03-19T11:31:08.072335+0000","last_active":"2024-03-19T11:31:08.071050+0000","last_peered":"2024-03-19T11:31:08.071050+0000","last_clean":"2024-03-19T11:31:08.071050+0000","last_became_active":"2024-03-19T11:31:03.052657+0000","last_became_peered":"2024-03-19T11:31:03.052657+0000","last_unstale":"2024-03-19T11:31:08.071050+0000","last_undegraded":"2024-03-19T11:31:08.071050+0000","last_fullsized":"2024-03-19T11:31:08.071050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'92","last_scrub_stamp":"2024-03-19T11:30:41.351520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:41.351520+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:51.023342+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.012111200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.057268+0000","last_change":"2024-03-19T11:31:00.067375+0000","last_active":"2024-03-19T11:31:08.057268+0000","last_peered":"2024-03-19T11:31:08.057268+0000","last_clean":"2024-03-19T11:31:08.057268+0000","last_became_active":"2024-03-19T11:31:00.067099+0000","last_became_peered":"2024-03-19T11:31:00.067099+0000","last_unstale":"2024-03-19T11:31:08.057268+0000","last_undegraded":"2024-03-19T11:31:08.057268+0000","last_fullsized":"2024-03-19T11:31:08.057268+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"87'71","reported_seq":373,"reported_epoch":90,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:09.057009+0000","last_change":"2024-03-19T11:31:08.109367+0000","last_active":"2024-03-19T11:31:09.057009+0000","last_peered":"2024-03-19T11:31:09.057009+0000","last_clean":"2024-03-19T11:31:09.057009+0000","last_became_active":"2024-03-19T11:30:38.065693+0000","last_became_peered":"2024-03-19T11:30:38.065693+0000","last_unstale":"2024-03-19T11:31:09.057009+0000","last_undegraded":"2024-03-19T11:31:09.057009+0000","last_fullsized":"2024-03-19T11:31:09.057009+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'57","last_scrub_stamp":"2024-03-19T11:30:43.792326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:43.792326+0000","objects_scrubbed":6,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:59.305622+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.048139135999999999,"stat_sum":{"num_bytes":11494971,"num_objects":5,"num_object_clones":4,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":13,"num_read_kb":16662,"num_write":6,"num_write_kb":1520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"6","length":"1"},{"start":"11","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":170,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T11:31:09.137317+0000","last_change":"2024-03-19T11:30:59.072585+0000","last_active":"2024-03-19T11:31:09.137317+0000","last_peered":"2024-03-19T11:31:09.137317+0000","last_clean":"2024-03-19T11:31:09.137317+0000","last_became_active":"2024-03-19T11:30:59.071925+0000","last_became_peered":"2024-03-19T11:30:59.071925+0000","last_unstale":"2024-03-19T11:31:09.137317+0000","last_undegraded":"2024-03-19T11:31:09.137317+0000","last_fullsized":"2024-03-19T11:31:09.137317+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"41'19","reported_seq":342,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T11:31:12.135520+0000","last_change":"2024-03-19T11:31:11.101156+0000","last_active":"2024-03-19T11:31:12.135520+0000","last_peered":"2024-03-19T11:31:12.135520+0000","last_clean":"2024-03-19T11:31:12.135520+0000","last_became_active":"2024-03-19T11:30:38.058860+0000","last_became_peered":"2024-03-19T11:30:38.058860+0000","last_unstale":"2024-03-19T11:31:12.135520+0000","last_undegraded":"2024-03-19T11:31:12.135520+0000","last_fullsized":"2024-03-19T11:31:12.135520+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'19","last_scrub_stamp":"2024-03-19T11:30:28.180772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:28.180772+0000","objects_scrubbed":1,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:17.047139+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.024522575000000001,"stat_sum":{"num_bytes":1403368,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4077,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":161,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.058905+0000","last_change":"2024-03-19T11:30:55.366872+0000","last_active":"2024-03-19T11:31:08.058905+0000","last_peered":"2024-03-19T11:31:08.058905+0000","last_clean":"2024-03-19T11:31:08.058905+0000","last_became_active":"2024-03-19T11:29:46.540939+0000","last_became_peered":"2024-03-19T11:29:46.540939+0000","last_unstale":"2024-03-19T11:31:08.058905+0000","last_undegraded":"2024-03-19T11:31:08.058905+0000","last_fullsized":"2024-03-19T11:31:08.058905+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:30:55.366631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:30:55.366631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:10.640223+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"87'19","reported_seq":87,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.154738+0000","last_change":"2024-03-19T11:31:08.063352+0000","last_active":"2024-03-19T11:31:08.154738+0000","last_peered":"2024-03-19T11:31:08.154738+0000","last_clean":"2024-03-19T11:31:08.154738+0000","last_became_active":"2024-03-19T11:31:00.069200+0000","last_became_peered":"2024-03-19T11:31:00.069200+0000","last_unstale":"2024-03-19T11:31:08.154738+0000","last_undegraded":"2024-03-19T11:31:08.154738+0000","last_fullsized":"2024-03-19T11:31:08.154738+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-03-19T11:31:05.279955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:31:05.279955+0000","objects_scrubbed":2,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:06.253899+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0066655569999999999,"stat_sum":{"num_bytes":3918899,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":12792,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"3.3","version":"87'107","reported_seq":357,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.067397+0000","last_change":"2024-03-19T11:31:06.067504+0000","last_active":"2024-03-19T11:31:08.067397+0000","last_peered":"2024-03-19T11:31:08.067397+0000","last_clean":"2024-03-19T11:31:08.067397+0000","last_became_active":"2024-03-19T11:31:06.066161+0000","last_became_peered":"2024-03-19T11:31:06.066161+0000","last_unstale":"2024-03-19T11:31:08.067397+0000","last_undegraded":"2024-03-19T11:31:08.067397+0000","last_fullsized":"2024-03-19T11:31:08.067397+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:10.489969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47923414,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":80,"num_read_kb":3742,"num_write":115,"num_write_kb":59833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":25111105,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":152,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.047165+0000","last_change":"2024-03-19T11:29:46.321255+0000","last_active":"2024-03-19T11:31:08.047165+0000","last_peered":"2024-03-19T11:31:08.047165+0000","last_clean":"2024-03-19T11:31:08.047165+0000","last_became_active":"2024-03-19T11:29:46.320679+0000","last_became_peered":"2024-03-19T11:29:46.320679+0000","last_unstale":"2024-03-19T11:31:08.047165+0000","last_undegraded":"2024-03-19T11:31:08.047165+0000","last_fullsized":"2024-03-19T11:31:08.047165+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:14.793563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"87'136","reported_seq":549,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.071940+0000","last_change":"2024-03-19T11:31:08.071940+0000","last_active":"2024-03-19T11:31:08.071940+0000","last_peered":"2024-03-19T11:31:08.071940+0000","last_clean":"2024-03-19T11:31:08.071940+0000","last_became_active":"2024-03-19T11:31:03.054624+0000","last_became_peered":"2024-03-19T11:31:03.054624+0000","last_unstale":"2024-03-19T11:31:08.071940+0000","last_undegraded":"2024-03-19T11:31:08.071940+0000","last_fullsized":"2024-03-19T11:31:08.071940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'119","last_scrub_stamp":"2024-03-19T11:30:52.512203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:52.512203+0000","objects_scrubbed":7,"log_size":136,"log_dups_size":0,"ondisk_log_size":136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:17.035697+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.013370745999999999,"stat_sum":{"num_bytes":12400028,"num_objects":5,"num_object_clones":2,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":37,"num_read_kb":46256,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"3.0","version":"89'104","reported_seq":261,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.058221+0000","last_change":"2024-03-19T11:29:57.550320+0000","last_active":"2024-03-19T11:31:08.058221+0000","last_peered":"2024-03-19T11:31:08.058221+0000","last_clean":"2024-03-19T11:31:08.058221+0000","last_became_active":"2024-03-19T11:29:57.549928+0000","last_became_peered":"2024-03-19T11:29:57.549928+0000","last_unstale":"2024-03-19T11:31:08.058221+0000","last_undegraded":"2024-03-19T11:31:08.058221+0000","last_fullsized":"2024-03-19T11:31:08.058221+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32146410,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":82,"num_read_kb":3252,"num_write":109,"num_write_kb":43633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":156,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T11:31:09.057613+0000","last_change":"2024-03-19T11:29:46.319811+0000","last_active":"2024-03-19T11:31:09.057613+0000","last_peered":"2024-03-19T11:31:09.057613+0000","last_clean":"2024-03-19T11:31:09.057613+0000","last_became_active":"2024-03-19T11:29:46.319334+0000","last_became_peered":"2024-03-19T11:29:46.319334+0000","last_unstale":"2024-03-19T11:31:09.057613+0000","last_undegraded":"2024-03-19T11:31:09.057613+0000","last_fullsized":"2024-03-19T11:31:09.057613+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"61'40","reported_seq":57,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.058444+0000","last_change":"2024-03-19T11:31:08.059007+0000","last_active":"2024-03-19T11:31:08.058444+0000","last_peered":"2024-03-19T11:31:08.058444+0000","last_clean":"2024-03-19T11:31:08.058444+0000","last_became_active":"2024-03-19T11:31:00.069809+0000","last_became_peered":"2024-03-19T11:31:00.069809+0000","last_unstale":"2024-03-19T11:31:08.058444+0000","last_undegraded":"2024-03-19T11:31:08.058444+0000","last_fullsized":"2024-03-19T11:31:08.058444+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'40","last_scrub_stamp":"2024-03-19T11:30:56.505586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:56.505586+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:56.842849+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.002526866,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"3.1","version":"92'94","reported_seq":396,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T11:31:12.135168+0000","last_change":"2024-03-19T11:31:03.255027+0000","last_active":"2024-03-19T11:31:12.135168+0000","last_peered":"2024-03-19T11:31:12.135168+0000","last_clean":"2024-03-19T11:31:12.135168+0000","last_became_active":"2024-03-19T11:31:03.077272+0000","last_became_peered":"2024-03-19T11:31:03.077272+0000","last_unstale":"2024-03-19T11:31:12.135168+0000","last_undegraded":"2024-03-19T11:31:12.135168+0000","last_fullsized":"2024-03-19T11:31:12.135168+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:23.954117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43920134,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":76,"num_read_kb":6656,"num_write":98,"num_write_kb":55249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":42087532,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":165,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T11:31:09.137114+0000","last_change":"2024-03-19T11:31:00.070027+0000","last_active":"2024-03-19T11:31:09.137114+0000","last_peered":"2024-03-19T11:31:09.137114+0000","last_clean":"2024-03-19T11:31:09.137114+0000","last_became_active":"2024-03-19T11:31:00.068989+0000","last_became_peered":"2024-03-19T11:31:00.068989+0000","last_unstale":"2024-03-19T11:31:09.137114+0000","last_undegraded":"2024-03-19T11:31:09.137114+0000","last_fullsized":"2024-03-19T11:31:09.137114+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:11.599600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"89'77","reported_seq":101,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T11:31:12.135296+0000","last_change":"2024-03-19T11:31:10.118389+0000","last_active":"2024-03-19T11:31:12.135296+0000","last_peered":"2024-03-19T11:31:12.135296+0000","last_clean":"2024-03-19T11:31:12.135296+0000","last_became_active":"2024-03-19T11:30:59.075057+0000","last_became_peered":"2024-03-19T11:30:59.075057+0000","last_unstale":"2024-03-19T11:31:12.135296+0000","last_undegraded":"2024-03-19T11:31:12.135296+0000","last_fullsized":"2024-03-19T11:31:12.135296+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'65","last_scrub_stamp":"2024-03-19T11:30:53.385414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:53.385414+0000","objects_scrubbed":3,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:00.732802+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":1.017056878,"stat_sum":{"num_bytes":8047585,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":37,"num_read_kb":51756,"num_write":9,"num_write_kb":2729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4209633,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"3.2","version":"86'88","reported_seq":253,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T11:31:09.056695+0000","last_change":"2024-03-19T11:29:57.549277+0000","last_active":"2024-03-19T11:31:09.056695+0000","last_peered":"2024-03-19T11:31:09.056695+0000","last_clean":"2024-03-19T11:31:09.056695+0000","last_became_active":"2024-03-19T11:29:57.548946+0000","last_became_peered":"2024-03-19T11:29:57.548946+0000","last_unstale":"2024-03-19T11:31:09.056695+0000","last_undegraded":"2024-03-19T11:31:09.056695+0000","last_fullsized":"2024-03-19T11:31:09.056695+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42459028,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":78,"num_read_kb":8248,"num_write":98,"num_write_kb":44121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":160,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.047494+0000","last_change":"2024-03-19T11:31:04.588296+0000","last_active":"2024-03-19T11:31:08.047494+0000","last_peered":"2024-03-19T11:31:08.047494+0000","last_clean":"2024-03-19T11:31:08.047494+0000","last_became_active":"2024-03-19T11:29:46.321760+0000","last_became_peered":"2024-03-19T11:29:46.321760+0000","last_unstale":"2024-03-19T11:31:08.047494+0000","last_undegraded":"2024-03-19T11:31:08.047494+0000","last_fullsized":"2024-03-19T11:31:08.047494+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:31:04.588071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:31:04.588071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:11.976969+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":339,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T11:31:09.137294+0000","last_change":"2024-03-19T11:31:03.367805+0000","last_active":"2024-03-19T11:31:09.137294+0000","last_peered":"2024-03-19T11:31:09.137294+0000","last_clean":"2024-03-19T11:31:09.137294+0000","last_became_active":"2024-03-19T11:30:59.073709+0000","last_became_peered":"2024-03-19T11:30:59.073709+0000","last_unstale":"2024-03-19T11:31:09.137294+0000","last_undegraded":"2024-03-19T11:31:09.137294+0000","last_fullsized":"2024-03-19T11:31:09.137294+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2024-03-19T11:31:03.367669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:43.166618+0000","last_clean_scrub_stamp":"2024-03-19T11:31:03.367669+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:29.977573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"88'111","reported_seq":515,"reported_epoch":89,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:08.083914+0000","last_change":"2024-03-19T11:31:07.189608+0000","last_active":"2024-03-19T11:31:08.083914+0000","last_peered":"2024-03-19T11:31:08.083914+0000","last_clean":"2024-03-19T11:31:08.083914+0000","last_became_active":"2024-03-19T11:31:03.065344+0000","last_became_peered":"2024-03-19T11:31:03.065344+0000","last_unstale":"2024-03-19T11:31:08.083914+0000","last_undegraded":"2024-03-19T11:31:08.083914+0000","last_fullsized":"2024-03-19T11:31:08.083914+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'81","last_scrub_stamp":"2024-03-19T11:30:52.360396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:52.360396+0000","objects_scrubbed":6,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:14.659997+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.103481396,"stat_sum":{"num_bytes":10838384,"num_objects":7,"num_object_clones":5,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":13,"num_read_kb":12349,"num_write":8,"num_write_kb":1273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":10558635,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"1"}]},{"pgid":"4.8","version":"62'71","reported_seq":380,"reported_epoch":89,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:08.113863+0000","last_change":"2024-03-19T11:31:08.113959+0000","last_active":"2024-03-19T11:31:08.113863+0000","last_peered":"2024-03-19T11:31:08.113863+0000","last_clean":"2024-03-19T11:31:08.113863+0000","last_became_active":"2024-03-19T11:30:45.251283+0000","last_became_peered":"2024-03-19T11:30:45.251283+0000","last_unstale":"2024-03-19T11:31:08.113863+0000","last_undegraded":"2024-03-19T11:31:08.113863+0000","last_fullsized":"2024-03-19T11:31:08.113863+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'71","last_scrub_stamp":"2024-03-19T11:30:53.381386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:53.381386+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:14.007435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.066645368999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"}]},{"pgid":"4.9","version":"63'55","reported_seq":415,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.075645+0000","last_change":"2024-03-19T11:31:08.076672+0000","last_active":"2024-03-19T11:31:08.075645+0000","last_peered":"2024-03-19T11:31:08.075645+0000","last_clean":"2024-03-19T11:31:08.075645+0000","last_became_active":"2024-03-19T11:31:00.069789+0000","last_became_peered":"2024-03-19T11:31:00.069789+0000","last_unstale":"2024-03-19T11:31:08.075645+0000","last_undegraded":"2024-03-19T11:31:08.075645+0000","last_fullsized":"2024-03-19T11:31:08.075645+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'55","last_scrub_stamp":"2024-03-19T11:30:54.706235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:54.706235+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:10.998788+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.015421028,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.a","version":"62'77","reported_seq":406,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.117764+0000","last_change":"2024-03-19T11:31:08.118531+0000","last_active":"2024-03-19T11:31:08.117764+0000","last_peered":"2024-03-19T11:31:08.117764+0000","last_clean":"2024-03-19T11:31:08.117764+0000","last_became_active":"2024-03-19T11:30:45.256124+0000","last_became_peered":"2024-03-19T11:30:45.256124+0000","last_unstale":"2024-03-19T11:31:08.117764+0000","last_undegraded":"2024-03-19T11:31:08.117764+0000","last_fullsized":"2024-03-19T11:31:08.117764+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'77","last_scrub_stamp":"2024-03-19T11:30:55.291823+0000","last_deep_scrub":"62'77","last_deep_scrub_stamp":"2024-03-19T11:30:55.291823+0000","last_clean_scrub_stamp":"2024-03-19T11:30:55.291823+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:58.611283+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.033194802000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.b","version":"90'66","reported_seq":435,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T11:31:12.135465+0000","last_change":"2024-03-19T11:31:12.126422+0000","last_active":"2024-03-19T11:31:12.135465+0000","last_peered":"2024-03-19T11:31:12.135465+0000","last_clean":"2024-03-19T11:31:12.135465+0000","last_became_active":"2024-03-19T11:31:08.066470+0000","last_became_peered":"2024-03-19T11:31:08.066470+0000","last_unstale":"2024-03-19T11:31:12.135465+0000","last_undegraded":"2024-03-19T11:31:12.135465+0000","last_fullsized":"2024-03-19T11:31:12.135465+0000","mapping_epoch":21,"log_start":"85'64","ondisk_log_start":"85'64","created":21,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'62","last_scrub_stamp":"2024-03-19T11:30:56.095039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:56.095039+0000","objects_scrubbed":3,"log_size":2,"log_dups_size":30,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:57.225290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.0499694129999999,"stat_sum":{"num_bytes":2873964,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.c","version":"56'29","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.072859+0000","last_change":"2024-03-19T11:31:08.074168+0000","last_active":"2024-03-19T11:31:08.072859+0000","last_peered":"2024-03-19T11:31:08.072859+0000","last_clean":"2024-03-19T11:31:08.072859+0000","last_became_active":"2024-03-19T11:31:05.073655+0000","last_became_peered":"2024-03-19T11:31:05.073655+0000","last_unstale":"2024-03-19T11:31:08.072859+0000","last_undegraded":"2024-03-19T11:31:08.072859+0000","last_fullsized":"2024-03-19T11:31:08.072859+0000","mapping_epoch":83,"log_start":"56'29","ondisk_log_start":"56'29","created":21,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'29","last_scrub_stamp":"2024-03-19T11:30:53.743385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:53.743385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:56.702924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013861157000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.d","version":"87'88","reported_seq":434,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.086044+0000","last_change":"2024-03-19T11:31:08.086044+0000","last_active":"2024-03-19T11:31:08.086044+0000","last_peered":"2024-03-19T11:31:08.086044+0000","last_clean":"2024-03-19T11:31:08.086044+0000","last_became_active":"2024-03-19T11:31:06.068526+0000","last_became_peered":"2024-03-19T11:31:06.068526+0000","last_unstale":"2024-03-19T11:31:08.086044+0000","last_undegraded":"2024-03-19T11:31:08.086044+0000","last_fullsized":"2024-03-19T11:31:08.086044+0000","mapping_epoch":86,"log_start":"70'71","ondisk_log_start":"70'71","created":21,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'73","last_scrub_stamp":"2024-03-19T11:30:53.519178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:53.519178+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":50,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:02.045989+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.021539165999999998,"stat_sum":{"num_bytes":6903494,"num_objects":4,"num_object_clones":2,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":18,"num_read_kb":22670,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":8146998,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.e","version":"89'94","reported_seq":402,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.081152+0000","last_change":"2024-03-19T11:31:08.081889+0000","last_active":"2024-03-19T11:31:08.081152+0000","last_peered":"2024-03-19T11:31:08.081152+0000","last_clean":"2024-03-19T11:31:08.081152+0000","last_became_active":"2024-03-19T11:30:38.060487+0000","last_became_peered":"2024-03-19T11:30:38.060487+0000","last_unstale":"2024-03-19T11:31:08.081152+0000","last_undegraded":"2024-03-19T11:31:08.081152+0000","last_fullsized":"2024-03-19T11:31:08.081152+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'65","last_scrub_stamp":"2024-03-19T11:30:43.824620+0000","last_deep_scrub":"65'65","last_deep_scrub_stamp":"2024-03-19T11:30:43.824620+0000","last_clean_scrub_stamp":"2024-03-19T11:30:43.824620+0000","objects_scrubbed":4,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:51.683705+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0.020120889999999999,"stat_sum":{"num_bytes":10000429,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":37,"num_read_kb":41805,"num_write":10,"num_write_kb":2301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.f","version":"0'0","reported_seq":280,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.063623+0000","last_change":"2024-03-19T11:31:08.065011+0000","last_active":"2024-03-19T11:31:08.063623+0000","last_peered":"2024-03-19T11:31:08.063623+0000","last_clean":"2024-03-19T11:31:08.063623+0000","last_became_active":"2024-03-19T11:30:38.050919+0000","last_became_peered":"2024-03-19T11:30:38.050919+0000","last_unstale":"2024-03-19T11:31:08.063623+0000","last_undegraded":"2024-03-19T11:31:08.063623+0000","last_fullsized":"2024-03-19T11:31:08.063623+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:30:43.411893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:43.411893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:04.626662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0068115019999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]}],"pool_stats":[{"poolid":4,"num_pg":42,"stat_sum":{"num_bytes":257025892,"num_objects":128,"num_object_clones":80,"num_object_copies":256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":125,"num_whiteouts":8,"num_read":904,"num_read_kb":1021963,"num_write":274,"num_write_kb":70279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":202389426,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":83,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":363048960,"data_stored":420280986,"data_compressed":2511443,"data_compressed_allocated":1650688,"data_compressed_original":5996544,"omap_allocated":0,"internal_metadata":0},"log_size":2754,"ondisk_log_size":2754,"up":84,"acting":84,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":166448986,"num_objects":149,"num_object_clones":0,"num_object_copies":298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":149,"num_whiteouts":0,"num_read":316,"num_read_kb":21898,"num_write":420,"num_write_kb":202836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":67198637,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":209154048,"data_stored":226374243,"data_compressed":15723907,"data_compressed_allocated":16814080,"data_compressed_original":34467840,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":8,"acting":8,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":1586784,"data_compressed":19832,"data_compressed_allocated":98304,"data_compressed_original":1560576,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542165,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84272,"kb_used_data":52384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94287568,"statfs":{"total":96636764160,"available":96550469632,"internally_reserved":0,"allocated":53641216,"data_stored":62766152,"data_compressed":154066,"data_compressed_allocated":151552,"data_compressed_original":970752,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542165,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82652,"kb_used_data":51472,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94289188,"statfs":{"total":96636764160,"available":96552128512,"internally_reserved":0,"allocated":52707328,"data_stored":61412079,"data_compressed":1859610,"data_compressed_allocated":1949696,"data_compressed_original":4210688,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542165,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":128184,"kb_used_data":96296,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94243656,"statfs":{"total":96636764160,"available":96505503744,"internally_reserved":0,"allocated":98607104,"data_stored":114667063,"data_compressed":2437448,"data_compressed_allocated":2236416,"data_compressed_original":5586944,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542164,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72524,"kb_used_data":45504,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94299316,"statfs":{"total":96636764160,"available":96562499584,"internally_reserved":0,"allocated":46596096,"data_stored":50973803,"data_compressed":1821717,"data_compressed_allocated":1605632,"data_compressed_original":4390912,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542164,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":113008,"kb_used_data":77664,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94258832,"statfs":{"total":96636764160,"available":96521043968,"internally_reserved":0,"allocated":79527936,"data_stored":82008696,"data_compressed":1366622,"data_compressed_allocated":1470464,"data_compressed_original":3276800,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574868,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64032,"kb_used_data":32844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94307808,"statfs":{"total":96636764160,"available":96571195392,"internally_reserved":0,"allocated":33632256,"data_stored":43833336,"data_compressed":169512,"data_compressed_allocated":57344,"data_compressed_original":196608,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574868,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":137620,"kb_used_data":106440,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94234220,"statfs":{"total":96636764160,"available":96495841280,"internally_reserved":0,"allocated":108994560,"data_stored":119204649,"data_compressed":4084412,"data_compressed_allocated":4292608,"data_compressed_original":8638464,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":98960,"kb_used_data":67772,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94272880,"statfs":{"total":96636764160,"available":96535429120,"internally_reserved":0,"allocated":69398528,"data_stored":75294481,"data_compressed":4059414,"data_compressed_allocated":4308992,"data_compressed_original":9359360,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6884,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6884,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6064,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":50696192,"data_stored":54782790,"data_compressed":3892201,"data_compressed_allocated":4149248,"data_compressed_original":8343552,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16039936,"data_stored":17517630,"data_compressed":1089583,"data_compressed_allocated":1175552,"data_compressed_original":2686976,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16039936,"data_stored":17517630,"data_compressed":1089583,"data_compressed_allocated":1175552,"data_compressed_original":2686976,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":22233088,"data_stored":24117451,"data_compressed":1711608,"data_compressed_allocated":1830912,"data_compressed_original":3751936,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":51625984,"data_stored":55733062,"data_compressed":3965051,"data_compressed_allocated":4227072,"data_compressed_original":8441856,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":22233088,"data_stored":24117451,"data_compressed":1711608,"data_compressed_allocated":1830912,"data_compressed_original":3751936,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":30285824,"data_stored":32588229,"data_compressed":2264273,"data_compressed_allocated":2424832,"data_compressed_original":4804608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":17620992,"data_stored":19440318,"data_compressed":160329,"data_compressed_allocated":126976,"data_compressed_original":491520,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":62513152,"data_stored":63972890,"data_compressed":277039,"data_compressed_allocated":294912,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":29593600,"data_stored":32940595,"data_compressed":732134,"data_compressed_allocated":430080,"data_compressed_original":1703936,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":75440128,"data_stored":90038134,"data_compressed":725840,"data_compressed_allocated":405504,"data_compressed_original":1835008,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":59510784,"data_stored":67285511,"data_compressed":150585,"data_compressed_allocated":98304,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":29507584,"data_stored":36778653,"data_compressed":148002,"data_compressed_allocated":118784,"data_compressed_original":458752,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36229120,"data_stored":48101703,"data_compressed":169512,"data_compressed_allocated":57344,"data_compressed_original":196608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":52633600,"data_stored":61723182,"data_compressed":148002,"data_compressed_allocated":118784,"data_compressed_original":458752,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T11:31:14.008 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-19T11:31:14.008 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-19T11:31:14.008 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_0 2024-03-19T11:31:14.009 INFO:tasks.ceph.ceph_manager.ceph:contract_pool unique_pool_0 by 10 min 8 2024-03-19T11:31:14.009 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T11:31:14.038 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:14.047+0000 7fb17dd72640 -1 osd.4 94 handle_osd_map injecting map crc failure 2024-03-19T11:31:14.040 INFO:tasks.rados.rados.0.smithi038.stdout:1311: done (0 left) 2024-03-19T11:31:14.040 INFO:tasks.rados.rados.0.smithi038.stdout:1312: read oid 29 snap -1 2024-03-19T11:31:14.040 INFO:tasks.rados.rados.0.smithi038.stdout:1312: expect (ObjNum 219 snap 23 seq_num 219) 2024-03-19T11:31:14.040 INFO:tasks.rados.rados.0.smithi038.stdout:1313: read oid 45 snap -1 2024-03-19T11:31:14.040 INFO:tasks.rados.rados.0.smithi038.stdout:1313: expect (ObjNum 212 snap 22 seq_num 212) 2024-03-19T11:31:14.040 INFO:tasks.rados.rados.0.smithi038.stdout:1314: write oid 23 current snap is 24 2024-03-19T11:31:14.040 INFO:tasks.rados.rados.0.smithi038.stdout:1314: seq_num 220 ranges {497540=612664,1840862=440060,3058823=464784} 2024-03-19T11:31:14.046 INFO:tasks.rados.rados.0.smithi038.stdout:1314: writing smithi03829856-23 from 497540 to 1110204 tid 1 2024-03-19T11:31:14.053 INFO:tasks.rados.rados.0.smithi038.stdout:1314: writing smithi03829856-23 from 1840862 to 2280922 tid 2 2024-03-19T11:31:14.061 INFO:tasks.rados.rados.0.smithi038.stdout:1314: writing smithi03829856-23 from 3058823 to 3523607 tid 3 2024-03-19T11:31:14.061 INFO:tasks.rados.rados.0.smithi038.stdout:1315: tier_evict oid 11 2024-03-19T11:31:14.061 INFO:tasks.rados.rados.0.smithi038.stdout:1315: tier_evict oid 11 snap -1 2024-03-19T11:31:14.061 INFO:tasks.rados.rados.0.smithi038.stdout:1312: expect (ObjNum 219 snap 23 seq_num 219) 2024-03-19T11:31:14.070 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:14.079+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:14.114 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:14.115 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:31:14.115 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:31:14.115 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:14.129 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-19T11:31:14.154 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-19T11:31:14.171 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:14.179+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:14.273 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:14.283+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:14.284 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:14.284 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:31:14.284 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:31:14.284 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:31:14.284 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:14.298 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-19T11:31:14.301 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:31:14.302 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:31:14.319 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:31:14.319 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:31:14.336 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:31:14.337 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-03-19T11:31:14.337 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:31:14.338 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:31:14.353 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:31:14.354 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:31:14.358 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":134,"stamp":"2024-03-19T11:31:14.156731+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":435312284,"num_objects":284,"num_object_clones":83,"num_object_copies":568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":281,"num_whiteouts":8,"num_read":1380,"num_read_kb":1099124,"num_write":862,"num_write_kb":277881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":257,"num_bytes_recovered":270645919,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":89,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3204,"ondisk_log_size":3204,"up":110,"acting":110,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":114,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":786452,"kb_used_data":535572,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754188268,"statfs":{"total":773094113280,"available":772288786432,"internally_reserved":0,"allocated":548425728,"data_stored":620761017,"data_compressed":15911175,"data_compressed_allocated":16027648,"data_compressed_original":36597760,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-869231,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":85,"num_read_kb":112403,"num_write":18,"num_write_kb":5604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000996"},"pg_stats":[{"pgid":"4.29","version":"90'70","reported_seq":465,"reported_epoch":93,"state":"peering","last_fresh":"2024-03-19T11:31:12.070070+0000","last_change":"2024-03-19T11:31:13.052818+0000","last_active":"2024-03-19T11:31:13.052818+0000","last_peered":"2024-03-19T11:31:12.070070+0000","last_clean":"2024-03-19T11:31:12.070070+0000","last_became_active":"2024-03-19T11:31:00.070287+0000","last_became_peered":"2024-03-19T11:31:00.070287+0000","last_unstale":"2024-03-19T11:31:12.070070+0000","last_undegraded":"2024-03-19T11:31:12.070070+0000","last_fullsized":"2024-03-19T11:31:12.070070+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":6,"last_scrub":"71'60","last_scrub_stamp":"2024-03-19T11:30:52.748479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:52.748479+0000","objects_scrubbed":3,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.015091184000000001,"stat_sum":{"num_bytes":6698954,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":21,"num_read_kb":23172,"num_write":5,"num_write_kb":1739,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":5842489,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.28","version":"89'103","reported_seq":408,"reported_epoch":89,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:08.114830+0000","last_change":"2024-03-19T11:31:08.114830+0000","last_active":"2024-03-19T11:31:08.114830+0000","last_peered":"2024-03-19T11:31:08.114830+0000","last_clean":"2024-03-19T11:31:08.114830+0000","last_became_active":"2024-03-19T11:30:45.257600+0000","last_became_peered":"2024-03-19T11:30:45.257600+0000","last_unstale":"2024-03-19T11:31:08.114830+0000","last_undegraded":"2024-03-19T11:31:08.114830+0000","last_fullsized":"2024-03-19T11:31:08.114830+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":6,"last_scrub":"71'85","last_scrub_stamp":"2024-03-19T11:30:52.343454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:52.343454+0000","objects_scrubbed":7,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:57.454276+0000","scrub_duration":7,"objects_trimmed":1,"snaptrim_duration":0.065578227000000003,"stat_sum":{"num_bytes":9128542,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"2"}]},{"pgid":"4.27","version":"87'56","reported_seq":79,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.059325+0000","last_change":"2024-03-19T11:31:08.060066+0000","last_active":"2024-03-19T11:31:08.059325+0000","last_peered":"2024-03-19T11:31:08.059325+0000","last_clean":"2024-03-19T11:31:08.059325+0000","last_became_active":"2024-03-19T11:31:00.070337+0000","last_became_peered":"2024-03-19T11:31:00.070337+0000","last_unstale":"2024-03-19T11:31:08.059325+0000","last_undegraded":"2024-03-19T11:31:08.059325+0000","last_fullsized":"2024-03-19T11:31:08.059325+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":6,"last_scrub":"71'48","last_scrub_stamp":"2024-03-19T11:30:55.467716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:55.467716+0000","objects_scrubbed":4,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:09.107718+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0027578379999999999,"stat_sum":{"num_bytes":5844083,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":11198,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":7070511,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.26","version":"61'63","reported_seq":70,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T11:31:12.135363+0000","last_change":"2024-03-19T11:31:08.115454+0000","last_active":"2024-03-19T11:31:12.135363+0000","last_peered":"2024-03-19T11:31:12.135363+0000","last_clean":"2024-03-19T11:31:12.135363+0000","last_became_active":"2024-03-19T11:30:59.074337+0000","last_became_peered":"2024-03-19T11:30:59.074337+0000","last_unstale":"2024-03-19T11:31:12.135363+0000","last_undegraded":"2024-03-19T11:31:12.135363+0000","last_fullsized":"2024-03-19T11:31:12.135363+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":6,"last_scrub":"61'63","last_scrub_stamp":"2024-03-19T11:30:52.406723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:52.406723+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:57.163804+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.064661005999999993,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.25","version":"56'66","reported_seq":83,"reported_epoch":89,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:08.114708+0000","last_change":"2024-03-19T11:31:08.114800+0000","last_active":"2024-03-19T11:31:08.114708+0000","last_peered":"2024-03-19T11:31:08.114708+0000","last_clean":"2024-03-19T11:31:08.114708+0000","last_became_active":"2024-03-19T11:30:53.084473+0000","last_became_peered":"2024-03-19T11:30:53.084473+0000","last_unstale":"2024-03-19T11:31:08.114708+0000","last_undegraded":"2024-03-19T11:31:08.114708+0000","last_fullsized":"2024-03-19T11:31:08.114708+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":6,"last_scrub":"56'66","last_scrub_stamp":"2024-03-19T11:31:06.587201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:31:06.587201+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:10.867475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.065786145000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"}]},{"pgid":"4.24","version":"85'117","reported_seq":510,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.107628+0000","last_change":"2024-03-19T11:31:08.069824+0000","last_active":"2024-03-19T11:31:08.107628+0000","last_peered":"2024-03-19T11:31:08.107628+0000","last_clean":"2024-03-19T11:31:08.107628+0000","last_became_active":"2024-03-19T11:31:03.063705+0000","last_became_peered":"2024-03-19T11:31:03.063705+0000","last_unstale":"2024-03-19T11:31:08.107628+0000","last_undegraded":"2024-03-19T11:31:08.107628+0000","last_fullsized":"2024-03-19T11:31:08.107628+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":6,"last_scrub":"84'113","last_scrub_stamp":"2024-03-19T11:31:03.304881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:31:03.304881+0000","objects_scrubbed":5,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:06.612630+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.011640286999999999,"stat_sum":{"num_bytes":10102023,"num_objects":4,"num_object_clones":2,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":34100,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.23","version":"87'38","reported_seq":87,"reported_epoch":90,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:09.056901+0000","last_change":"2024-03-19T11:31:06.101937+0000","last_active":"2024-03-19T11:31:09.056901+0000","last_peered":"2024-03-19T11:31:09.056901+0000","last_clean":"2024-03-19T11:31:09.056901+0000","last_became_active":"2024-03-19T11:30:53.076092+0000","last_became_peered":"2024-03-19T11:30:53.076092+0000","last_unstale":"2024-03-19T11:31:09.056901+0000","last_undegraded":"2024-03-19T11:31:09.056901+0000","last_fullsized":"2024-03-19T11:31:09.056901+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":6,"last_scrub":"62'23","last_scrub_stamp":"2024-03-19T11:30:43.163509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:43.163509+0000","objects_scrubbed":2,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:10.968084+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.035799641,"stat_sum":{"num_bytes":6154918,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":10,"num_write_kb":2870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2833035,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"6","length":"1"},{"start":"11","length":"1"}]},{"pgid":"4.22","version":"28'6","reported_seq":100,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T11:31:13.051225+0000","last_change":"2024-03-19T11:31:08.076088+0000","last_active":"2024-03-19T11:31:13.051225+0000","last_peered":"2024-03-19T11:31:13.051225+0000","last_clean":"2024-03-19T11:31:13.051225+0000","last_became_active":"2024-03-19T11:30:53.068010+0000","last_became_peered":"2024-03-19T11:30:53.068010+0000","last_unstale":"2024-03-19T11:31:13.051225+0000","last_undegraded":"2024-03-19T11:31:13.051225+0000","last_fullsized":"2024-03-19T11:31:13.051225+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":6,"last_scrub":"28'6","last_scrub_stamp":"2024-03-19T11:30:55.724671+0000","last_deep_scrub":"28'6","last_deep_scrub_stamp":"2024-03-19T11:30:55.724671+0000","last_clean_scrub_stamp":"2024-03-19T11:30:55.724671+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:22.211935+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.015426808,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.21","version":"89'133","reported_seq":524,"reported_epoch":90,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:09.056335+0000","last_change":"2024-03-19T11:31:08.112958+0000","last_active":"2024-03-19T11:31:09.056335+0000","last_peered":"2024-03-19T11:31:09.056335+0000","last_clean":"2024-03-19T11:31:09.056335+0000","last_became_active":"2024-03-19T11:31:00.069759+0000","last_became_peered":"2024-03-19T11:31:00.069759+0000","last_unstale":"2024-03-19T11:31:09.056335+0000","last_undegraded":"2024-03-19T11:31:09.056335+0000","last_fullsized":"2024-03-19T11:31:09.056335+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":6,"last_scrub":"62'96","last_scrub_stamp":"2024-03-19T11:30:43.400885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:43.400885+0000","objects_scrubbed":9,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:12.660708+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.050771269000000001,"stat_sum":{"num_bytes":16636184,"num_objects":7,"num_object_clones":5,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":49,"num_read_kb":51400,"num_write":23,"num_write_kb":6523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":22983003,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d","length":"1"}]},{"pgid":"4.20","version":"56'46","reported_seq":89,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T11:31:12.073288+0000","last_change":"2024-03-19T11:31:08.117528+0000","last_active":"2024-03-19T11:31:12.073288+0000","last_peered":"2024-03-19T11:31:12.073288+0000","last_clean":"2024-03-19T11:31:12.073288+0000","last_became_active":"2024-03-19T11:30:53.067251+0000","last_became_peered":"2024-03-19T11:30:53.067251+0000","last_unstale":"2024-03-19T11:31:12.073288+0000","last_undegraded":"2024-03-19T11:31:12.073288+0000","last_fullsized":"2024-03-19T11:31:12.073288+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":6,"last_scrub":"56'46","last_scrub_stamp":"2024-03-19T11:30:41.767100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:41.767100+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:51.169007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.032365520000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.1f","version":"0'0","reported_seq":97,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T11:31:12.073489+0000","last_change":"2024-03-19T11:31:08.116595+0000","last_active":"2024-03-19T11:31:12.073489+0000","last_peered":"2024-03-19T11:31:12.073489+0000","last_clean":"2024-03-19T11:31:12.073489+0000","last_became_active":"2024-03-19T11:30:53.087211+0000","last_became_peered":"2024-03-19T11:30:53.087211+0000","last_unstale":"2024-03-19T11:31:12.073489+0000","last_undegraded":"2024-03-19T11:31:12.073489+0000","last_fullsized":"2024-03-19T11:31:12.073489+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:30:42.377963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.377963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:09.545606+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.032679030999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.1e","version":"87'71","reported_seq":116,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T11:31:12.135393+0000","last_change":"2024-03-19T11:31:12.123738+0000","last_active":"2024-03-19T11:31:12.135393+0000","last_peered":"2024-03-19T11:31:12.135393+0000","last_clean":"2024-03-19T11:31:12.135393+0000","last_became_active":"2024-03-19T11:30:53.084452+0000","last_became_peered":"2024-03-19T11:30:53.084452+0000","last_unstale":"2024-03-19T11:31:12.135393+0000","last_undegraded":"2024-03-19T11:31:12.135393+0000","last_fullsized":"2024-03-19T11:31:12.135393+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"64'62","last_scrub_stamp":"2024-03-19T11:30:42.778159+0000","last_deep_scrub":"64'62","last_deep_scrub_stamp":"2024-03-19T11:30:42.778159+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.778159+0000","objects_scrubbed":4,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:11.879105+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":1.046110673,"stat_sum":{"num_bytes":3794928,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":27,"num_read_kb":28503,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3779426,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.1d","version":"87'67","reported_seq":81,"reported_epoch":90,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:09.056383+0000","last_change":"2024-03-19T11:31:06.105861+0000","last_active":"2024-03-19T11:31:09.056383+0000","last_peered":"2024-03-19T11:31:09.056383+0000","last_clean":"2024-03-19T11:31:09.056383+0000","last_became_active":"2024-03-19T11:30:53.067748+0000","last_became_peered":"2024-03-19T11:30:53.067748+0000","last_unstale":"2024-03-19T11:31:09.056383+0000","last_undegraded":"2024-03-19T11:31:09.056383+0000","last_fullsized":"2024-03-19T11:31:09.056383+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"61'59","last_scrub_stamp":"2024-03-19T11:30:42.601863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.601863+0000","objects_scrubbed":2,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:45.377695+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.039756313000000001,"stat_sum":{"num_bytes":4835736,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":3,"num_read_kb":61,"num_write":5,"num_write_kb":1341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2817390,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"6","length":"1"},{"start":"11","length":"1"}]},{"pgid":"4.1c","version":"85'35","reported_seq":98,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T11:31:09.057786+0000","last_change":"2024-03-19T11:31:08.052486+0000","last_active":"2024-03-19T11:31:09.057786+0000","last_peered":"2024-03-19T11:31:09.057786+0000","last_clean":"2024-03-19T11:31:09.057786+0000","last_became_active":"2024-03-19T11:30:53.082655+0000","last_became_peered":"2024-03-19T11:30:53.082655+0000","last_unstale":"2024-03-19T11:31:09.057786+0000","last_undegraded":"2024-03-19T11:31:09.057786+0000","last_fullsized":"2024-03-19T11:31:09.057786+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"62'31","last_scrub_stamp":"2024-03-19T11:30:42.771396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.771396+0000","objects_scrubbed":3,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:53.311410+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0013764719999999999,"stat_sum":{"num_bytes":3210104,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":47464,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":5034309,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.1b","version":"89'100","reported_seq":106,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T11:31:13.051591+0000","last_change":"2024-03-19T11:31:08.108313+0000","last_active":"2024-03-19T11:31:13.051591+0000","last_peered":"2024-03-19T11:31:13.051591+0000","last_clean":"2024-03-19T11:31:13.051591+0000","last_became_active":"2024-03-19T11:31:03.056769+0000","last_became_peered":"2024-03-19T11:31:03.056769+0000","last_unstale":"2024-03-19T11:31:13.051591+0000","last_undegraded":"2024-03-19T11:31:13.051591+0000","last_fullsized":"2024-03-19T11:31:13.051591+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":5,"last_scrub":"62'62","last_scrub_stamp":"2024-03-19T11:30:42.181424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.181424+0000","objects_scrubbed":5,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:48.154772+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.040344087000000001,"stat_sum":{"num_bytes":14818653,"num_objects":7,"num_object_clones":4,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":58,"num_read_kb":68803,"num_write":11,"num_write_kb":3105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":17966556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":6,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.1a","version":"89'111","reported_seq":100,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T11:31:13.050901+0000","last_change":"2024-03-19T11:31:08.083422+0000","last_active":"2024-03-19T11:31:13.050901+0000","last_peered":"2024-03-19T11:31:13.050901+0000","last_clean":"2024-03-19T11:31:13.050901+0000","last_became_active":"2024-03-19T11:31:03.056520+0000","last_became_peered":"2024-03-19T11:31:03.056520+0000","last_unstale":"2024-03-19T11:31:13.050901+0000","last_undegraded":"2024-03-19T11:31:13.050901+0000","last_fullsized":"2024-03-19T11:31:13.050901+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":5,"last_scrub":"61'76","last_scrub_stamp":"2024-03-19T11:30:41.433501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:41.433501+0000","objects_scrubbed":5,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:10.610041+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.021991492000000001,"stat_sum":{"num_bytes":11953834,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":28912,"num_write":18,"num_write_kb":5314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":16291222,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.19","version":"89'117","reported_seq":496,"reported_epoch":90,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:09.056559+0000","last_change":"2024-03-19T11:31:08.110132+0000","last_active":"2024-03-19T11:31:09.056559+0000","last_peered":"2024-03-19T11:31:09.056559+0000","last_clean":"2024-03-19T11:31:09.056559+0000","last_became_active":"2024-03-19T11:30:53.076188+0000","last_became_peered":"2024-03-19T11:30:53.076188+0000","last_unstale":"2024-03-19T11:31:09.056559+0000","last_undegraded":"2024-03-19T11:31:09.056559+0000","last_fullsized":"2024-03-19T11:31:09.056559+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"41'37","last_scrub_stamp":"2024-03-19T11:30:27.705159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:27.705159+0000","objects_scrubbed":4,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:28.833088+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.047949758000000002,"stat_sum":{"num_bytes":18546999,"num_objects":13,"num_object_clones":9,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":99,"num_read_kb":113713,"num_write":39,"num_write_kb":6633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":21870393,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":11,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"11","length":"1"}]},{"pgid":"4.18","version":"88'138","reported_seq":457,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T11:31:13.050770+0000","last_change":"2024-03-19T11:31:08.074905+0000","last_active":"2024-03-19T11:31:13.050770+0000","last_peered":"2024-03-19T11:31:13.050770+0000","last_clean":"2024-03-19T11:31:13.050770+0000","last_became_active":"2024-03-19T11:31:06.066324+0000","last_became_peered":"2024-03-19T11:31:06.066324+0000","last_unstale":"2024-03-19T11:31:13.050770+0000","last_undegraded":"2024-03-19T11:31:13.050770+0000","last_fullsized":"2024-03-19T11:31:13.050770+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":5,"last_scrub":"48'77","last_scrub_stamp":"2024-03-19T11:30:29.399762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:29.399762+0000","objects_scrubbed":6,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:44.362849+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.014795552,"stat_sum":{"num_bytes":18257745,"num_objects":7,"num_object_clones":5,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":57,"num_read_kb":61759,"num_write":22,"num_write_kb":6328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":19971477,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.17","version":"41'24","reported_seq":74,"reported_epoch":90,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:09.057390+0000","last_change":"2024-03-19T11:31:06.106323+0000","last_active":"2024-03-19T11:31:09.057390+0000","last_peered":"2024-03-19T11:31:09.057390+0000","last_clean":"2024-03-19T11:31:09.057390+0000","last_became_active":"2024-03-19T11:30:53.074270+0000","last_became_peered":"2024-03-19T11:30:53.074270+0000","last_unstale":"2024-03-19T11:31:09.057390+0000","last_undegraded":"2024-03-19T11:31:09.057390+0000","last_fullsized":"2024-03-19T11:31:09.057390+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"41'24","last_scrub_stamp":"2024-03-19T11:30:53.993428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:53.993428+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:22.671214+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.039101238000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"6","length":"1"},{"start":"11","length":"1"}]},{"pgid":"4.16","version":"92'122","reported_seq":121,"reported_epoch":93,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:12.135253+0000","last_change":"2024-03-19T11:31:11.101970+0000","last_active":"2024-03-19T11:31:12.135253+0000","last_peered":"2024-03-19T11:31:12.135253+0000","last_clean":"2024-03-19T11:31:12.135253+0000","last_became_active":"2024-03-19T11:30:59.074063+0000","last_became_peered":"2024-03-19T11:30:59.074063+0000","last_unstale":"2024-03-19T11:31:12.135253+0000","last_undegraded":"2024-03-19T11:31:12.135253+0000","last_fullsized":"2024-03-19T11:31:12.135253+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":5,"last_scrub":"49'56","last_scrub_stamp":"2024-03-19T11:30:29.212766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:29.212766+0000","objects_scrubbed":5,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:40.544381+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.023712351999999999,"stat_sum":{"num_bytes":13644045,"num_objects":7,"num_object_clones":4,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":124,"num_read_kb":134085,"num_write":27,"num_write_kb":7671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":24165549,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":7,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"2"}]},{"pgid":"4.15","version":"89'177","reported_seq":596,"reported_epoch":89,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:08.117471+0000","last_change":"2024-03-19T11:31:06.107463+0000","last_active":"2024-03-19T11:31:08.117471+0000","last_peered":"2024-03-19T11:31:08.117471+0000","last_clean":"2024-03-19T11:31:08.117471+0000","last_became_active":"2024-03-19T11:30:56.063100+0000","last_became_peered":"2024-03-19T11:30:56.063100+0000","last_unstale":"2024-03-19T11:31:08.117471+0000","last_undegraded":"2024-03-19T11:31:08.117471+0000","last_fullsized":"2024-03-19T11:31:08.117471+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":5,"last_scrub":"49'74","last_scrub_stamp":"2024-03-19T11:30:29.457542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:29.457542+0000","objects_scrubbed":7,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:42.862571+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.028232337999999999,"stat_sum":{"num_bytes":26876675,"num_objects":17,"num_object_clones":12,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":1,"num_read":142,"num_read_kb":156856,"num_write":52,"num_write_kb":12752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":18744415,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":11,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"d","length":"1"},{"start":"11","length":"1"}]},{"pgid":"4.14","version":"40'56","reported_seq":239,"reported_epoch":90,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:09.057353+0000","last_change":"2024-03-19T11:31:08.110146+0000","last_active":"2024-03-19T11:31:09.057353+0000","last_peered":"2024-03-19T11:31:09.057353+0000","last_clean":"2024-03-19T11:31:09.057353+0000","last_became_active":"2024-03-19T11:31:00.065648+0000","last_became_peered":"2024-03-19T11:31:00.065648+0000","last_unstale":"2024-03-19T11:31:09.057353+0000","last_undegraded":"2024-03-19T11:31:09.057353+0000","last_fullsized":"2024-03-19T11:31:09.057353+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":5,"last_scrub":"40'56","last_scrub_stamp":"2024-03-19T11:30:54.998282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:54.998282+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:12.541661+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.046980873999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"}]},{"pgid":"4.13","version":"91'40","reported_seq":156,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T11:31:13.051722+0000","last_change":"2024-03-19T11:31:08.107232+0000","last_active":"2024-03-19T11:31:13.051722+0000","last_peered":"2024-03-19T11:31:13.051722+0000","last_clean":"2024-03-19T11:31:13.051722+0000","last_became_active":"2024-03-19T11:30:45.245367+0000","last_became_peered":"2024-03-19T11:30:45.245367+0000","last_unstale":"2024-03-19T11:31:13.051722+0000","last_undegraded":"2024-03-19T11:31:13.051722+0000","last_fullsized":"2024-03-19T11:31:13.051722+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":5,"last_scrub":"48'22","last_scrub_stamp":"2024-03-19T11:30:27.183294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:27.183294+0000","objects_scrubbed":2,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:46.079474+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.039325774000000001,"stat_sum":{"num_bytes":9406330,"num_objects":4,"num_object_clones":2,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":35,"num_read_kb":30733,"num_write":17,"num_write_kb":5134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":3001789,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.12","version":"28'6","reported_seq":65,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.057864+0000","last_change":"2024-03-19T11:31:08.058401+0000","last_active":"2024-03-19T11:31:08.057864+0000","last_peered":"2024-03-19T11:31:08.057864+0000","last_clean":"2024-03-19T11:31:08.057864+0000","last_became_active":"2024-03-19T11:31:00.064380+0000","last_became_peered":"2024-03-19T11:31:00.064380+0000","last_unstale":"2024-03-19T11:31:08.057864+0000","last_undegraded":"2024-03-19T11:31:08.057864+0000","last_fullsized":"2024-03-19T11:31:08.057864+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":5,"last_scrub":"28'6","last_scrub_stamp":"2024-03-19T11:31:04.263428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:31:04.263428+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:20.192174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020212310000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.11","version":"88'84","reported_seq":201,"reported_epoch":90,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:09.056453+0000","last_change":"2024-03-19T11:31:08.112910+0000","last_active":"2024-03-19T11:31:09.056453+0000","last_peered":"2024-03-19T11:31:09.056453+0000","last_clean":"2024-03-19T11:31:09.056453+0000","last_became_active":"2024-03-19T11:30:30.052184+0000","last_became_peered":"2024-03-19T11:30:30.052184+0000","last_unstale":"2024-03-19T11:31:09.056453+0000","last_undegraded":"2024-03-19T11:31:09.056453+0000","last_fullsized":"2024-03-19T11:31:09.056453+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":5,"last_scrub":"47'60","last_scrub_stamp":"2024-03-19T11:30:30.780348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:30.780348+0000","objects_scrubbed":2,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:43.351932+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.051827466000000003,"stat_sum":{"num_bytes":8084512,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":25443,"num_write":14,"num_write_kb":4028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"6","length":"1"},{"start":"11","length":"1"}]},{"pgid":"4.10","version":"85'34","reported_seq":358,"reported_epoch":90,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:09.056610+0000","last_change":"2024-03-19T11:31:06.105203+0000","last_active":"2024-03-19T11:31:09.056610+0000","last_peered":"2024-03-19T11:31:09.056610+0000","last_clean":"2024-03-19T11:31:09.056610+0000","last_became_active":"2024-03-19T11:31:00.065617+0000","last_became_peered":"2024-03-19T11:31:00.065617+0000","last_unstale":"2024-03-19T11:31:09.056610+0000","last_undegraded":"2024-03-19T11:31:09.056610+0000","last_fullsized":"2024-03-19T11:31:09.056610+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":5,"last_scrub":"50'30","last_scrub_stamp":"2024-03-19T11:30:28.787058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:28.787058+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:43.836722+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.038831954000000002,"stat_sum":{"num_bytes":3462209,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":20772,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":7102596,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"6","length":"1"},{"start":"11","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":146,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T11:31:09.057749+0000","last_change":"2024-03-19T11:31:08.381703+0000","last_active":"2024-03-19T11:31:09.057749+0000","last_peered":"2024-03-19T11:31:09.057749+0000","last_clean":"2024-03-19T11:31:09.057749+0000","last_became_active":"2024-03-19T11:30:03.276220+0000","last_became_peered":"2024-03-19T11:30:03.276220+0000","last_unstale":"2024-03-19T11:31:09.057749+0000","last_undegraded":"2024-03-19T11:31:09.057749+0000","last_fullsized":"2024-03-19T11:31:09.057749+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:31:08.381559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:31:08.381559+0000","last_clean_scrub_stamp":"2024-03-19T11:31:08.381559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:30.325139+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"59'92","reported_seq":59,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T11:31:13.050830+0000","last_change":"2024-03-19T11:31:08.072335+0000","last_active":"2024-03-19T11:31:13.050830+0000","last_peered":"2024-03-19T11:31:13.050830+0000","last_clean":"2024-03-19T11:31:13.050830+0000","last_became_active":"2024-03-19T11:31:03.052657+0000","last_became_peered":"2024-03-19T11:31:03.052657+0000","last_unstale":"2024-03-19T11:31:13.050830+0000","last_undegraded":"2024-03-19T11:31:13.050830+0000","last_fullsized":"2024-03-19T11:31:13.050830+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'92","last_scrub_stamp":"2024-03-19T11:30:41.351520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:41.351520+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:53.185342+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.012111200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.057268+0000","last_change":"2024-03-19T11:31:00.067375+0000","last_active":"2024-03-19T11:31:08.057268+0000","last_peered":"2024-03-19T11:31:08.057268+0000","last_clean":"2024-03-19T11:31:08.057268+0000","last_became_active":"2024-03-19T11:31:00.067099+0000","last_became_peered":"2024-03-19T11:31:00.067099+0000","last_unstale":"2024-03-19T11:31:08.057268+0000","last_undegraded":"2024-03-19T11:31:08.057268+0000","last_fullsized":"2024-03-19T11:31:08.057268+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"87'71","reported_seq":373,"reported_epoch":90,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:09.057009+0000","last_change":"2024-03-19T11:31:08.109367+0000","last_active":"2024-03-19T11:31:09.057009+0000","last_peered":"2024-03-19T11:31:09.057009+0000","last_clean":"2024-03-19T11:31:09.057009+0000","last_became_active":"2024-03-19T11:30:38.065693+0000","last_became_peered":"2024-03-19T11:30:38.065693+0000","last_unstale":"2024-03-19T11:31:09.057009+0000","last_undegraded":"2024-03-19T11:31:09.057009+0000","last_fullsized":"2024-03-19T11:31:09.057009+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'57","last_scrub_stamp":"2024-03-19T11:30:43.792326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:43.792326+0000","objects_scrubbed":6,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:59.305622+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.048139135999999999,"stat_sum":{"num_bytes":11494971,"num_objects":5,"num_object_clones":4,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":13,"num_read_kb":16662,"num_write":6,"num_write_kb":1520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"6","length":"1"},{"start":"11","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":170,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T11:31:09.137317+0000","last_change":"2024-03-19T11:30:59.072585+0000","last_active":"2024-03-19T11:31:09.137317+0000","last_peered":"2024-03-19T11:31:09.137317+0000","last_clean":"2024-03-19T11:31:09.137317+0000","last_became_active":"2024-03-19T11:30:59.071925+0000","last_became_peered":"2024-03-19T11:30:59.071925+0000","last_unstale":"2024-03-19T11:31:09.137317+0000","last_undegraded":"2024-03-19T11:31:09.137317+0000","last_fullsized":"2024-03-19T11:31:09.137317+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"41'19","reported_seq":342,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T11:31:12.135520+0000","last_change":"2024-03-19T11:31:11.101156+0000","last_active":"2024-03-19T11:31:12.135520+0000","last_peered":"2024-03-19T11:31:12.135520+0000","last_clean":"2024-03-19T11:31:12.135520+0000","last_became_active":"2024-03-19T11:30:38.058860+0000","last_became_peered":"2024-03-19T11:30:38.058860+0000","last_unstale":"2024-03-19T11:31:12.135520+0000","last_undegraded":"2024-03-19T11:31:12.135520+0000","last_fullsized":"2024-03-19T11:31:12.135520+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'19","last_scrub_stamp":"2024-03-19T11:30:28.180772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:28.180772+0000","objects_scrubbed":1,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:17.047139+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.024522575000000001,"stat_sum":{"num_bytes":1403368,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4077,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":161,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.058905+0000","last_change":"2024-03-19T11:30:55.366872+0000","last_active":"2024-03-19T11:31:08.058905+0000","last_peered":"2024-03-19T11:31:08.058905+0000","last_clean":"2024-03-19T11:31:08.058905+0000","last_became_active":"2024-03-19T11:29:46.540939+0000","last_became_peered":"2024-03-19T11:29:46.540939+0000","last_unstale":"2024-03-19T11:31:08.058905+0000","last_undegraded":"2024-03-19T11:31:08.058905+0000","last_fullsized":"2024-03-19T11:31:08.058905+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:30:55.366631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:30:55.366631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:10.640223+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"87'19","reported_seq":87,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.154738+0000","last_change":"2024-03-19T11:31:08.063352+0000","last_active":"2024-03-19T11:31:08.154738+0000","last_peered":"2024-03-19T11:31:08.154738+0000","last_clean":"2024-03-19T11:31:08.154738+0000","last_became_active":"2024-03-19T11:31:00.069200+0000","last_became_peered":"2024-03-19T11:31:00.069200+0000","last_unstale":"2024-03-19T11:31:08.154738+0000","last_undegraded":"2024-03-19T11:31:08.154738+0000","last_fullsized":"2024-03-19T11:31:08.154738+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-03-19T11:31:05.279955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:31:05.279955+0000","objects_scrubbed":2,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:06.253899+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0066655569999999999,"stat_sum":{"num_bytes":3918899,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":12792,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"3.3","version":"87'107","reported_seq":367,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T11:31:13.050735+0000","last_change":"2024-03-19T11:31:06.067504+0000","last_active":"2024-03-19T11:31:13.050735+0000","last_peered":"2024-03-19T11:31:13.050735+0000","last_clean":"2024-03-19T11:31:13.050735+0000","last_became_active":"2024-03-19T11:31:06.066161+0000","last_became_peered":"2024-03-19T11:31:06.066161+0000","last_unstale":"2024-03-19T11:31:13.050735+0000","last_undegraded":"2024-03-19T11:31:13.050735+0000","last_fullsized":"2024-03-19T11:31:13.050735+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:15.606407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47923414,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":80,"num_read_kb":3742,"num_write":115,"num_write_kb":59833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":25111105,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":152,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.047165+0000","last_change":"2024-03-19T11:29:46.321255+0000","last_active":"2024-03-19T11:31:08.047165+0000","last_peered":"2024-03-19T11:31:08.047165+0000","last_clean":"2024-03-19T11:31:08.047165+0000","last_became_active":"2024-03-19T11:29:46.320679+0000","last_became_peered":"2024-03-19T11:29:46.320679+0000","last_unstale":"2024-03-19T11:31:08.047165+0000","last_undegraded":"2024-03-19T11:31:08.047165+0000","last_fullsized":"2024-03-19T11:31:08.047165+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:14.793563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"87'136","reported_seq":549,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.071940+0000","last_change":"2024-03-19T11:31:08.071940+0000","last_active":"2024-03-19T11:31:08.071940+0000","last_peered":"2024-03-19T11:31:08.071940+0000","last_clean":"2024-03-19T11:31:08.071940+0000","last_became_active":"2024-03-19T11:31:03.054624+0000","last_became_peered":"2024-03-19T11:31:03.054624+0000","last_unstale":"2024-03-19T11:31:08.071940+0000","last_undegraded":"2024-03-19T11:31:08.071940+0000","last_fullsized":"2024-03-19T11:31:08.071940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'119","last_scrub_stamp":"2024-03-19T11:30:52.512203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:52.512203+0000","objects_scrubbed":7,"log_size":136,"log_dups_size":0,"ondisk_log_size":136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:17.035697+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.013370745999999999,"stat_sum":{"num_bytes":12400028,"num_objects":5,"num_object_clones":2,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":37,"num_read_kb":46256,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"3.0","version":"89'104","reported_seq":261,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.058221+0000","last_change":"2024-03-19T11:29:57.550320+0000","last_active":"2024-03-19T11:31:08.058221+0000","last_peered":"2024-03-19T11:31:08.058221+0000","last_clean":"2024-03-19T11:31:08.058221+0000","last_became_active":"2024-03-19T11:29:57.549928+0000","last_became_peered":"2024-03-19T11:29:57.549928+0000","last_unstale":"2024-03-19T11:31:08.058221+0000","last_undegraded":"2024-03-19T11:31:08.058221+0000","last_fullsized":"2024-03-19T11:31:08.058221+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32146410,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":82,"num_read_kb":3252,"num_write":109,"num_write_kb":43633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":156,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T11:31:09.057613+0000","last_change":"2024-03-19T11:29:46.319811+0000","last_active":"2024-03-19T11:31:09.057613+0000","last_peered":"2024-03-19T11:31:09.057613+0000","last_clean":"2024-03-19T11:31:09.057613+0000","last_became_active":"2024-03-19T11:29:46.319334+0000","last_became_peered":"2024-03-19T11:29:46.319334+0000","last_unstale":"2024-03-19T11:31:09.057613+0000","last_undegraded":"2024-03-19T11:31:09.057613+0000","last_fullsized":"2024-03-19T11:31:09.057613+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"61'40","reported_seq":57,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.058444+0000","last_change":"2024-03-19T11:31:08.059007+0000","last_active":"2024-03-19T11:31:08.058444+0000","last_peered":"2024-03-19T11:31:08.058444+0000","last_clean":"2024-03-19T11:31:08.058444+0000","last_became_active":"2024-03-19T11:31:00.069809+0000","last_became_peered":"2024-03-19T11:31:00.069809+0000","last_unstale":"2024-03-19T11:31:08.058444+0000","last_undegraded":"2024-03-19T11:31:08.058444+0000","last_fullsized":"2024-03-19T11:31:08.058444+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'40","last_scrub_stamp":"2024-03-19T11:30:56.505586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:56.505586+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:56.842849+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.002526866,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"3.1","version":"92'94","reported_seq":396,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T11:31:12.135168+0000","last_change":"2024-03-19T11:31:03.255027+0000","last_active":"2024-03-19T11:31:12.135168+0000","last_peered":"2024-03-19T11:31:12.135168+0000","last_clean":"2024-03-19T11:31:12.135168+0000","last_became_active":"2024-03-19T11:31:03.077272+0000","last_became_peered":"2024-03-19T11:31:03.077272+0000","last_unstale":"2024-03-19T11:31:12.135168+0000","last_undegraded":"2024-03-19T11:31:12.135168+0000","last_fullsized":"2024-03-19T11:31:12.135168+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:23.954117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43920134,"num_objects":33,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":76,"num_read_kb":6656,"num_write":98,"num_write_kb":55249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":42087532,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":165,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T11:31:09.137114+0000","last_change":"2024-03-19T11:31:00.070027+0000","last_active":"2024-03-19T11:31:09.137114+0000","last_peered":"2024-03-19T11:31:09.137114+0000","last_clean":"2024-03-19T11:31:09.137114+0000","last_became_active":"2024-03-19T11:31:00.068989+0000","last_became_peered":"2024-03-19T11:31:00.068989+0000","last_unstale":"2024-03-19T11:31:09.137114+0000","last_undegraded":"2024-03-19T11:31:09.137114+0000","last_fullsized":"2024-03-19T11:31:09.137114+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:11.599600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"89'77","reported_seq":101,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T11:31:12.135296+0000","last_change":"2024-03-19T11:31:10.118389+0000","last_active":"2024-03-19T11:31:12.135296+0000","last_peered":"2024-03-19T11:31:12.135296+0000","last_clean":"2024-03-19T11:31:12.135296+0000","last_became_active":"2024-03-19T11:30:59.075057+0000","last_became_peered":"2024-03-19T11:30:59.075057+0000","last_unstale":"2024-03-19T11:31:12.135296+0000","last_undegraded":"2024-03-19T11:31:12.135296+0000","last_fullsized":"2024-03-19T11:31:12.135296+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'65","last_scrub_stamp":"2024-03-19T11:30:53.385414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:53.385414+0000","objects_scrubbed":3,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:00.732802+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":1.017056878,"stat_sum":{"num_bytes":8047585,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":37,"num_read_kb":51756,"num_write":9,"num_write_kb":2729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4209633,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"3.2","version":"86'88","reported_seq":253,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T11:31:09.056695+0000","last_change":"2024-03-19T11:29:57.549277+0000","last_active":"2024-03-19T11:31:09.056695+0000","last_peered":"2024-03-19T11:31:09.056695+0000","last_clean":"2024-03-19T11:31:09.056695+0000","last_became_active":"2024-03-19T11:29:57.548946+0000","last_became_peered":"2024-03-19T11:29:57.548946+0000","last_unstale":"2024-03-19T11:31:09.056695+0000","last_undegraded":"2024-03-19T11:31:09.056695+0000","last_fullsized":"2024-03-19T11:31:09.056695+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42459028,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":78,"num_read_kb":8248,"num_write":98,"num_write_kb":44121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":160,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.047494+0000","last_change":"2024-03-19T11:31:04.588296+0000","last_active":"2024-03-19T11:31:08.047494+0000","last_peered":"2024-03-19T11:31:08.047494+0000","last_clean":"2024-03-19T11:31:08.047494+0000","last_became_active":"2024-03-19T11:29:46.321760+0000","last_became_peered":"2024-03-19T11:29:46.321760+0000","last_unstale":"2024-03-19T11:31:08.047494+0000","last_undegraded":"2024-03-19T11:31:08.047494+0000","last_fullsized":"2024-03-19T11:31:08.047494+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:31:04.588071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:31:04.588071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:11.976969+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":339,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-19T11:31:09.137294+0000","last_change":"2024-03-19T11:31:03.367805+0000","last_active":"2024-03-19T11:31:09.137294+0000","last_peered":"2024-03-19T11:31:09.137294+0000","last_clean":"2024-03-19T11:31:09.137294+0000","last_became_active":"2024-03-19T11:30:59.073709+0000","last_became_peered":"2024-03-19T11:30:59.073709+0000","last_unstale":"2024-03-19T11:31:09.137294+0000","last_undegraded":"2024-03-19T11:31:09.137294+0000","last_fullsized":"2024-03-19T11:31:09.137294+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2024-03-19T11:31:03.367669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:43.166618+0000","last_clean_scrub_stamp":"2024-03-19T11:31:03.367669+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:29.977573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"92'117","reported_seq":543,"reported_epoch":93,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:12.073550+0000","last_change":"2024-03-19T11:31:11.204498+0000","last_active":"2024-03-19T11:31:12.073550+0000","last_peered":"2024-03-19T11:31:12.073550+0000","last_clean":"2024-03-19T11:31:12.073550+0000","last_became_active":"2024-03-19T11:31:03.065344+0000","last_became_peered":"2024-03-19T11:31:03.065344+0000","last_unstale":"2024-03-19T11:31:12.073550+0000","last_undegraded":"2024-03-19T11:31:12.073550+0000","last_fullsized":"2024-03-19T11:31:12.073550+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'81","last_scrub_stamp":"2024-03-19T11:30:52.360396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:52.360396+0000","objects_scrubbed":6,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:08.817668+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.132459102,"stat_sum":{"num_bytes":10095563,"num_objects":6,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":22,"num_read_kb":18986,"num_write":8,"num_write_kb":1273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":10558635,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"}]},{"pgid":"4.8","version":"62'71","reported_seq":380,"reported_epoch":89,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:08.113863+0000","last_change":"2024-03-19T11:31:08.113959+0000","last_active":"2024-03-19T11:31:08.113863+0000","last_peered":"2024-03-19T11:31:08.113863+0000","last_clean":"2024-03-19T11:31:08.113863+0000","last_became_active":"2024-03-19T11:30:45.251283+0000","last_became_peered":"2024-03-19T11:30:45.251283+0000","last_unstale":"2024-03-19T11:31:08.113863+0000","last_undegraded":"2024-03-19T11:31:08.113863+0000","last_fullsized":"2024-03-19T11:31:08.113863+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'71","last_scrub_stamp":"2024-03-19T11:30:53.381386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:53.381386+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:14.007435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.066645368999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"}]},{"pgid":"4.9","version":"63'55","reported_seq":427,"reported_epoch":93,"state":"peering","last_fresh":"2024-03-19T11:31:12.070290+0000","last_change":"2024-03-19T11:31:13.052712+0000","last_active":"2024-03-19T11:31:13.052712+0000","last_peered":"2024-03-19T11:31:12.070290+0000","last_clean":"2024-03-19T11:31:12.070290+0000","last_became_active":"2024-03-19T11:31:00.069789+0000","last_became_peered":"2024-03-19T11:31:00.069789+0000","last_unstale":"2024-03-19T11:31:12.070290+0000","last_undegraded":"2024-03-19T11:31:12.070290+0000","last_fullsized":"2024-03-19T11:31:12.070290+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'55","last_scrub_stamp":"2024-03-19T11:30:54.706235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:54.706235+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.015421028,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.a","version":"87'100","reported_seq":427,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T11:31:12.081736+0000","last_change":"2024-03-19T11:31:12.076913+0000","last_active":"2024-03-19T11:31:12.081736+0000","last_peered":"2024-03-19T11:31:12.081736+0000","last_clean":"2024-03-19T11:31:12.081736+0000","last_became_active":"2024-03-19T11:31:12.075247+0000","last_became_peered":"2024-03-19T11:31:12.075247+0000","last_unstale":"2024-03-19T11:31:12.081736+0000","last_undegraded":"2024-03-19T11:31:12.081736+0000","last_fullsized":"2024-03-19T11:31:12.081736+0000","mapping_epoch":21,"log_start":"87'100","ondisk_log_start":"87'100","created":21,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'77","last_scrub_stamp":"2024-03-19T11:30:55.291823+0000","last_deep_scrub":"62'77","last_deep_scrub_stamp":"2024-03-19T11:30:55.291823+0000","last_clean_scrub_stamp":"2024-03-19T11:30:55.291823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:01.547204+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.033194802000000002,"stat_sum":{"num_bytes":9678155,"num_objects":4,"num_object_clones":2,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":25145,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.b","version":"90'66","reported_seq":435,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-19T11:31:12.135465+0000","last_change":"2024-03-19T11:31:12.126422+0000","last_active":"2024-03-19T11:31:12.135465+0000","last_peered":"2024-03-19T11:31:12.135465+0000","last_clean":"2024-03-19T11:31:12.135465+0000","last_became_active":"2024-03-19T11:31:08.066470+0000","last_became_peered":"2024-03-19T11:31:08.066470+0000","last_unstale":"2024-03-19T11:31:12.135465+0000","last_undegraded":"2024-03-19T11:31:12.135465+0000","last_fullsized":"2024-03-19T11:31:12.135465+0000","mapping_epoch":21,"log_start":"85'64","ondisk_log_start":"85'64","created":21,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'62","last_scrub_stamp":"2024-03-19T11:30:56.095039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:56.095039+0000","objects_scrubbed":3,"log_size":2,"log_dups_size":30,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:57.225290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.0499694129999999,"stat_sum":{"num_bytes":2873964,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.c","version":"56'29","reported_seq":62,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T11:31:13.051105+0000","last_change":"2024-03-19T11:31:08.074168+0000","last_active":"2024-03-19T11:31:13.051105+0000","last_peered":"2024-03-19T11:31:13.051105+0000","last_clean":"2024-03-19T11:31:13.051105+0000","last_became_active":"2024-03-19T11:31:05.073655+0000","last_became_peered":"2024-03-19T11:31:05.073655+0000","last_unstale":"2024-03-19T11:31:13.051105+0000","last_undegraded":"2024-03-19T11:31:13.051105+0000","last_fullsized":"2024-03-19T11:31:13.051105+0000","mapping_epoch":83,"log_start":"56'29","ondisk_log_start":"56'29","created":21,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'29","last_scrub_stamp":"2024-03-19T11:30:53.743385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:53.743385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:55.575555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013861157000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.d","version":"87'88","reported_seq":448,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T11:31:13.052023+0000","last_change":"2024-03-19T11:31:08.086044+0000","last_active":"2024-03-19T11:31:13.052023+0000","last_peered":"2024-03-19T11:31:13.052023+0000","last_clean":"2024-03-19T11:31:13.052023+0000","last_became_active":"2024-03-19T11:31:06.068526+0000","last_became_peered":"2024-03-19T11:31:06.068526+0000","last_unstale":"2024-03-19T11:31:13.052023+0000","last_undegraded":"2024-03-19T11:31:13.052023+0000","last_fullsized":"2024-03-19T11:31:13.052023+0000","mapping_epoch":86,"log_start":"70'71","ondisk_log_start":"70'71","created":21,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'73","last_scrub_stamp":"2024-03-19T11:30:53.519178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:53.519178+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":50,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:56.128920+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.021539165999999998,"stat_sum":{"num_bytes":6903494,"num_objects":4,"num_object_clones":2,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":18,"num_read_kb":22670,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":8146998,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.e","version":"91'98","reported_seq":419,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-19T11:31:13.051656+0000","last_change":"2024-03-19T11:31:08.081889+0000","last_active":"2024-03-19T11:31:13.051656+0000","last_peered":"2024-03-19T11:31:13.051656+0000","last_clean":"2024-03-19T11:31:13.051656+0000","last_became_active":"2024-03-19T11:30:38.060487+0000","last_became_peered":"2024-03-19T11:30:38.060487+0000","last_unstale":"2024-03-19T11:31:13.051656+0000","last_undegraded":"2024-03-19T11:31:13.051656+0000","last_fullsized":"2024-03-19T11:31:13.051656+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'65","last_scrub_stamp":"2024-03-19T11:30:43.824620+0000","last_deep_scrub":"65'65","last_deep_scrub_stamp":"2024-03-19T11:30:43.824620+0000","last_clean_scrub_stamp":"2024-03-19T11:30:43.824620+0000","objects_scrubbed":4,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:56.769196+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0.020120889999999999,"stat_sum":{"num_bytes":10000429,"num_objects":6,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":41805,"num_write":12,"num_write_kb":2301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.f","version":"0'0","reported_seq":280,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-19T11:31:08.063623+0000","last_change":"2024-03-19T11:31:08.065011+0000","last_active":"2024-03-19T11:31:08.063623+0000","last_peered":"2024-03-19T11:31:08.063623+0000","last_clean":"2024-03-19T11:31:08.063623+0000","last_became_active":"2024-03-19T11:30:38.050919+0000","last_became_peered":"2024-03-19T11:30:38.050919+0000","last_unstale":"2024-03-19T11:31:08.063623+0000","last_undegraded":"2024-03-19T11:31:08.063623+0000","last_fullsized":"2024-03-19T11:31:08.063623+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:30:43.411893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:43.411893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:04.626662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0068115019999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]}],"pool_stats":[{"poolid":4,"num_pg":42,"stat_sum":{"num_bytes":268272930,"num_objects":133,"num_object_clones":83,"num_object_copies":266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":8,"num_read":948,"num_read_kb":1077126,"num_write":283,"num_write_kb":72629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":202389426,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":89,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":362483712,"data_stored":422655077,"data_compressed":2511443,"data_compressed_allocated":1650688,"data_compressed_original":5996544,"omap_allocated":0,"internal_metadata":0},"log_size":2694,"ondisk_log_size":2694,"up":84,"acting":84,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":166448986,"num_objects":149,"num_object_clones":0,"num_object_copies":298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":149,"num_whiteouts":0,"num_read":316,"num_read_kb":21898,"num_write":420,"num_write_kb":202836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":67198637,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":208224256,"data_stored":225423971,"data_compressed":15651057,"data_compressed_allocated":16736256,"data_compressed_original":34369536,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":8,"acting":8,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":1586784,"data_compressed":19832,"data_compressed_allocated":98304,"data_compressed_original":1560576,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542165,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84272,"kb_used_data":52384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94287568,"statfs":{"total":96636764160,"available":96550469632,"internally_reserved":0,"allocated":53641216,"data_stored":62766152,"data_compressed":154066,"data_compressed_allocated":151552,"data_compressed_original":970752,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542165,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82652,"kb_used_data":51472,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94289188,"statfs":{"total":96636764160,"available":96552128512,"internally_reserved":0,"allocated":52707328,"data_stored":61412079,"data_compressed":1859610,"data_compressed_allocated":1949696,"data_compressed_original":4210688,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542165,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":128184,"kb_used_data":96296,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94243656,"statfs":{"total":96636764160,"available":96505503744,"internally_reserved":0,"allocated":98607104,"data_stored":114667063,"data_compressed":2437448,"data_compressed_allocated":2236416,"data_compressed_original":5586944,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542164,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72524,"kb_used_data":45504,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94299316,"statfs":{"total":96636764160,"available":96562499584,"internally_reserved":0,"allocated":46596096,"data_stored":50973803,"data_compressed":1821717,"data_compressed_allocated":1605632,"data_compressed_original":4390912,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542164,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":113008,"kb_used_data":77664,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94258832,"statfs":{"total":96636764160,"available":96521043968,"internally_reserved":0,"allocated":79527936,"data_stored":82008696,"data_compressed":1366622,"data_compressed_allocated":1470464,"data_compressed_original":3276800,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574869,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66972,"kb_used_data":35788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94304868,"statfs":{"total":96636764160,"available":96568184832,"internally_reserved":0,"allocated":36646912,"data_stored":48697940,"data_compressed":169512,"data_compressed_allocated":57344,"data_compressed_original":196608,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":139880,"kb_used_data":108692,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94231960,"statfs":{"total":96636764160,"available":96493527040,"internally_reserved":0,"allocated":111300608,"data_stored":124940803,"data_compressed":4042786,"data_compressed_allocated":4247552,"data_compressed_original":8605696,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":98960,"kb_used_data":67772,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94272880,"statfs":{"total":96636764160,"available":96535429120,"internally_reserved":0,"allocated":69398528,"data_stored":75294481,"data_compressed":4059414,"data_compressed_allocated":4308992,"data_compressed_original":9359360,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6884,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6884,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6064,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":50696192,"data_stored":54782790,"data_compressed":3892201,"data_compressed_allocated":4149248,"data_compressed_original":8343552,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16039936,"data_stored":17517630,"data_compressed":1089583,"data_compressed_allocated":1175552,"data_compressed_original":2686976,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16039936,"data_stored":17517630,"data_compressed":1089583,"data_compressed_allocated":1175552,"data_compressed_original":2686976,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":22233088,"data_stored":24117451,"data_compressed":1711608,"data_compressed_allocated":1830912,"data_compressed_original":3751936,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":50696192,"data_stored":54782790,"data_compressed":3892201,"data_compressed_allocated":4149248,"data_compressed_original":8343552,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":22233088,"data_stored":24117451,"data_compressed":1711608,"data_compressed_allocated":1830912,"data_compressed_original":3751936,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":30285824,"data_stored":32588229,"data_compressed":2264273,"data_compressed_allocated":2424832,"data_compressed_original":4804608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":17620992,"data_stored":19440318,"data_compressed":160329,"data_compressed_allocated":126976,"data_compressed_original":491520,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":62513152,"data_stored":63972890,"data_compressed":277039,"data_compressed_allocated":294912,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":29593600,"data_stored":32940595,"data_compressed":732134,"data_compressed_allocated":430080,"data_compressed_original":1703936,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":75440128,"data_stored":90038134,"data_compressed":725840,"data_compressed_allocated":405504,"data_compressed_original":1835008,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":59584512,"data_stored":69612402,"data_compressed":150585,"data_compressed_allocated":98304,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":29507584,"data_stored":36778653,"data_compressed":148002,"data_compressed_allocated":118784,"data_compressed_original":458752,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":35590144,"data_stored":48148903,"data_compressed":169512,"data_compressed_allocated":57344,"data_compressed_original":196608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":52633600,"data_stored":61723182,"data_compressed":148002,"data_compressed_allocated":118784,"data_compressed_original":458752,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T11:31:14.361 INFO:tasks.ceph.ceph_manager.ceph:decrease pool size by 10 2024-03-19T11:31:14.361 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 26 2024-03-19T11:31:14.376 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:31:14.376 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:31:14.377 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:14.385+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:14.379 INFO:tasks.rados.rados.0.smithi038.stdout:1315: got (22) Invalid argument 2024-03-19T11:31:14.379 INFO:tasks.rados.rados.0.smithi038.stdout:1314: finishing write tid 1 to smithi03829856-23 2024-03-19T11:31:14.379 INFO:tasks.rados.rados.0.smithi038.stdout:1314: finishing write tid 2 to smithi03829856-23 2024-03-19T11:31:14.379 INFO:tasks.rados.rados.0.smithi038.stdout:1314: finishing write tid 3 to smithi03829856-23 2024-03-19T11:31:14.379 INFO:tasks.rados.rados.0.smithi038.stdout:1314: finishing write tid 4 to smithi03829856-23 2024-03-19T11:31:14.379 INFO:tasks.rados.rados.0.smithi038.stdout:1314: finishing write tid 5 to smithi03829856-23 2024-03-19T11:31:14.379 INFO:tasks.rados.rados.0.smithi038.stdout:1314: oid 23 updating version 0 to 109 2024-03-19T11:31:14.379 INFO:tasks.rados.rados.0.smithi038.stdout:1314: oid 23 updating version 109 to 110 2024-03-19T11:31:14.379 INFO:tasks.rados.rados.0.smithi038.stdout:1314: oid 23 version 110 is already newer than 108 2024-03-19T11:31:14.379 INFO:tasks.rados.rados.0.smithi038.stdout:1314: oid 23 updating version 110 to 111 2024-03-19T11:31:14.379 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 23 v 111 (ObjNum 220 snap 24 seq_num 220) dirty exists 2024-03-19T11:31:14.379 INFO:tasks.rados.rados.0.smithi038.stdout:1314: left oid 23 (ObjNum 220 snap 24 seq_num 220) 2024-03-19T11:31:14.379 INFO:tasks.rados.rados.0.smithi038.stdout:1312: done (3 left) 2024-03-19T11:31:14.380 INFO:tasks.rados.rados.0.smithi038.stdout:1314: done (2 left) 2024-03-19T11:31:14.380 INFO:tasks.rados.rados.0.smithi038.stdout:1315: done (1 left) 2024-03-19T11:31:14.380 INFO:tasks.rados.rados.0.smithi038.stdout:1316: write oid 9 current snap is 24 2024-03-19T11:31:14.380 INFO:tasks.rados.rados.0.smithi038.stdout:1316: seq_num 221 ranges {514347=404266,1499344=700071,2686246=676056} 2024-03-19T11:31:14.384 INFO:tasks.rados.rados.0.smithi038.stdout:1316: writing smithi03829856-9 from 514347 to 918613 tid 1 2024-03-19T11:31:14.393 INFO:tasks.rados.rados.0.smithi038.stdout:1316: writing smithi03829856-9 from 1499344 to 2199415 tid 2 2024-03-19T11:31:14.402 INFO:tasks.rados.rados.0.smithi038.stdout:1316: writing smithi03829856-9 from 2686246 to 3362302 tid 3 2024-03-19T11:31:14.403 INFO:tasks.rados.rados.0.smithi038.stdout:1317: read oid 44 snap -1 2024-03-19T11:31:14.403 INFO:tasks.rados.rados.0.smithi038.stdout:1317: expect (ObjNum 108 snap 11 seq_num 108) 2024-03-19T11:31:14.403 INFO:tasks.rados.rados.0.smithi038.stdout:1318: read oid 34 snap 19 2024-03-19T11:31:14.403 INFO:tasks.rados.rados.0.smithi038.stdout:1318: expect (ObjNum 72 snap 5 seq_num 72) 2024-03-19T11:31:14.403 INFO:tasks.rados.rados.0.smithi038.stdout:1319: write (excl) oid 5 current snap is 24 2024-03-19T11:31:14.403 INFO:tasks.rados.rados.0.smithi038.stdout:1316: finishing write tid 1 to smithi03829856-9 2024-03-19T11:31:14.403 INFO:tasks.rados.rados.0.smithi038.stdout:1316: finishing write tid 2 to smithi03829856-9 2024-03-19T11:31:14.403 INFO:tasks.rados.rados.0.smithi038.stdout:1319: seq_num 222 ranges {531154=479515,1557826=443729,2713669=487328,3679295=51859} 2024-03-19T11:31:14.408 INFO:tasks.rados.rados.0.smithi038.stdout:1319: writing smithi03829856-5 from 531154 to 1010669 tid 1 2024-03-19T11:31:14.413 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:31:14.414 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:31:14.414 INFO:tasks.rados.rados.0.smithi038.stdout:1319: writing smithi03829856-5 from 1557826 to 2001555 tid 2 2024-03-19T11:31:14.421 INFO:tasks.rados.rados.0.smithi038.stdout:1319: writing smithi03829856-5 from 2713669 to 3200997 tid 3 2024-03-19T11:31:14.424 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:14.425 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:14.425 INFO:tasks.rados.rados.0.smithi038.stdout:1319: writing smithi03829856-5 from 3679295 to 3731154 tid 4 2024-03-19T11:31:14.425 INFO:tasks.rados.rados.0.smithi038.stdout:1320: set_chunk oid 10 target oid 2024-03-19T11:31:14.439 INFO:tasks.rados.rados.0.smithi038.stdout:1320: set_chunk oid 10 offset: 1187328 length: 1612288 target oid e01d5166c5b5a963e2d04d666180650b7ad9fd13c3e99522099014a31e75130b offset: 0 snap -1 2024-03-19T11:31:14.439 INFO:tasks.rados.rados.0.smithi038.stdout:1321: set_chunk oid 18 target oid 2024-03-19T11:31:14.449 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-19T11:31:14.450 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:31:14.450 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:31:14.470 INFO:tasks.rados.rados.0.smithi038.stdout:1321: set_chunk oid 18 offset: 648192 length: 2076160 target oid af122e68a596db82af82d38b1b94aad4d1bf388f06da7e2475a205c53a94acf6 offset: 0 snap -1 2024-03-19T11:31:14.470 INFO:tasks.rados.rados.0.smithi038.stdout:1322: read oid 30 snap -1 2024-03-19T11:31:14.470 INFO:tasks.rados.rados.0.smithi038.stdout:1322: expect (ObjNum 29 snap 0 seq_num 29) 2024-03-19T11:31:14.470 INFO:tasks.rados.rados.0.smithi038.stdout:1323: read oid 37 snap -1 2024-03-19T11:31:14.470 INFO:tasks.rados.rados.0.smithi038.stdout:1323: expect deleted 2024-03-19T11:31:14.471 INFO:tasks.rados.rados.0.smithi038.stdout:1324: read oid 33 snap 22 2024-03-19T11:31:14.471 INFO:tasks.rados.rados.0.smithi038.stdout:1324: expect (ObjNum 214 snap 22 seq_num 214) 2024-03-19T11:31:14.471 INFO:tasks.rados.rados.0.smithi038.stdout:1325: read oid 27 snap -1 2024-03-19T11:31:14.471 INFO:tasks.rados.rados.0.smithi038.stdout:1325: expect (ObjNum 192 snap 22 seq_num 192) 2024-03-19T11:31:14.471 INFO:tasks.rados.rados.0.smithi038.stdout:1318: expect (ObjNum 72 snap 5 seq_num 72) 2024-03-19T11:31:14.474 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:14.483+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:14.477 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:31:14.477 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:31:14.565 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:14.566 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:14.575 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:14.581+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:14.578 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-19T11:31:14.651 INFO:tasks.rados.rados.0.smithi038.stdout:1316: finishing write tid 3 to smithi03829856-9 2024-03-19T11:31:14.652 INFO:tasks.rados.rados.0.smithi038.stdout:1316: finishing write tid 4 to smithi03829856-9 2024-03-19T11:31:14.652 INFO:tasks.rados.rados.0.smithi038.stdout:1316: finishing write tid 5 to smithi03829856-9 2024-03-19T11:31:14.652 INFO:tasks.rados.rados.0.smithi038.stdout:1316: oid 9 updating version 0 to 93 2024-03-19T11:31:14.652 INFO:tasks.rados.rados.0.smithi038.stdout:1316: oid 9 version 93 is already newer than 90 2024-03-19T11:31:14.652 INFO:tasks.rados.rados.0.smithi038.stdout:1316: oid 9 version 93 is already newer than 91 2024-03-19T11:31:14.652 INFO:tasks.rados.rados.0.smithi038.stdout:1316: oid 9 version 93 is already newer than 92 2024-03-19T11:31:14.652 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 9 v 93 (ObjNum 221 snap 24 seq_num 221) dirty exists 2024-03-19T11:31:14.652 INFO:tasks.rados.rados.0.smithi038.stdout:1316: left oid 9 (ObjNum 221 snap 24 seq_num 221) 2024-03-19T11:31:14.652 INFO:tasks.rados.rados.0.smithi038.stdout:1319: finishing write tid 1 to smithi03829856-5 2024-03-19T11:31:14.652 INFO:tasks.rados.rados.0.smithi038.stdout:1316: done (10 left) 2024-03-19T11:31:14.652 INFO:tasks.rados.rados.0.smithi038.stdout:1318: done (9 left) 2024-03-19T11:31:14.652 INFO:tasks.rados.rados.0.smithi038.stdout:1326: read oid 17 snap -1 2024-03-19T11:31:14.653 INFO:tasks.rados.rados.0.smithi038.stdout:1326: expect (ObjNum 213 snap 22 seq_num 213) 2024-03-19T11:31:14.653 INFO:tasks.rados.rados.0.smithi038.stdout:1327: read oid 39 snap -1 2024-03-19T11:31:14.653 INFO:tasks.rados.rados.0.smithi038.stdout:1327: expect (ObjNum 163 snap 17 seq_num 163) 2024-03-19T11:31:14.653 INFO:tasks.rados.rados.0.smithi038.stdout:1328: rollback oid 50 current snap is 24 2024-03-19T11:31:14.653 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 50 to 9 2024-03-19T11:31:14.653 INFO:tasks.rados.rados.0.smithi038.stdout:1317: expect (ObjNum 108 snap 11 seq_num 108) 2024-03-19T11:31:14.677 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:14.687+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:14.754 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:14.754 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:14.768 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-19T11:31:14.777 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:14.787+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:14.844 INFO:tasks.rados.rados.0.smithi038.stdout:1319: finishing write tid 2 to smithi03829856-5 2024-03-19T11:31:14.844 INFO:tasks.rados.rados.0.smithi038.stdout:1319: finishing write tid 3 to smithi03829856-5 2024-03-19T11:31:14.844 INFO:tasks.rados.rados.0.smithi038.stdout:1319: finishing write tid 4 to smithi03829856-5 2024-03-19T11:31:14.844 INFO:tasks.rados.rados.0.smithi038.stdout:1319: finishing write tid 5 to smithi03829856-5 2024-03-19T11:31:14.844 INFO:tasks.rados.rados.0.smithi038.stdout:1319: finishing write tid 6 to smithi03829856-5 2024-03-19T11:31:14.844 INFO:tasks.rados.rados.0.smithi038.stdout:1319: oid 5 updating version 0 to 132 2024-03-19T11:31:14.844 INFO:tasks.rados.rados.0.smithi038.stdout:1319: oid 5 updating version 132 to 135 2024-03-19T11:31:14.844 INFO:tasks.rados.rados.0.smithi038.stdout:1319: oid 5 updating version 135 to 136 2024-03-19T11:31:14.844 INFO:tasks.rados.rados.0.smithi038.stdout:1319: oid 5 version 136 is already newer than 133 2024-03-19T11:31:14.844 INFO:tasks.rados.rados.0.smithi038.stdout:1319: oid 5 version 136 is already newer than 134 2024-03-19T11:31:14.844 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 5 v 136 (ObjNum 222 snap 24 seq_num 222) dirty exists 2024-03-19T11:31:14.845 INFO:tasks.rados.rados.0.smithi038.stdout:1319: left oid 5 (ObjNum 222 snap 24 seq_num 222) 2024-03-19T11:31:14.845 INFO:tasks.rados.rados.0.smithi038.stdout:1320: finishing set_chunk to oid 10 2024-03-19T11:31:14.845 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 10 set_chunk returned error code -95 offset: 1187328 length: 1612288 2024-03-19T11:31:14.845 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 10 v 0 (ObjNum 179 snap 19 seq_num 179) dirty exists 2024-03-19T11:31:14.845 INFO:tasks.rados.rados.0.smithi038.stdout:1322: expect (ObjNum 29 snap 0 seq_num 29) 2024-03-19T11:31:14.878 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:14.885+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:14.922 INFO:tasks.rados.rados.0.smithi038.stdout:1321: finishing set_chunk to oid 18 2024-03-19T11:31:14.922 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 18 v 124 (ObjNum 199 snap 22 seq_num 199) dirty exists 2024-03-19T11:31:14.922 INFO:tasks.rados.rados.0.smithi038.stdout:1324: expect (ObjNum 214 snap 22 seq_num 214) 2024-03-19T11:31:14.941 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:14.941 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:14.954 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-19T11:31:14.996 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:15.005+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:15.065 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-03-19T11:31:15.065 INFO:teuthology.orchestra.run.smithi038.stderr:set pool 4 pg_num to 26 2024-03-19T11:31:15.075 INFO:tasks.rados.rados.0.smithi038.stdout:1328: finishing rollback tid 1 to smithi03829856-50 2024-03-19T11:31:15.075 INFO:tasks.rados.rados.0.smithi038.stdout:1328: finishing rollback tid 2 to smithi03829856-50 2024-03-19T11:31:15.075 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 50 v 138 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-03-19T11:31:15.075 INFO:tasks.rados.rados.0.smithi038.stdout:1326: expect (ObjNum 213 snap 22 seq_num 213) 2024-03-19T11:31:15.080 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:15.091+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:15.089 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:15.089 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:15.104 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-19T11:31:15.195 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:15.207+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:15.270 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:15.270 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:15.282 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:15.291+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:15.284 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-19T11:31:15.382 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:15.391+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:15.386 INFO:tasks.rados.rados.0.smithi038.stdout:1325: expect (ObjNum 192 snap 22 seq_num 192) 2024-03-19T11:31:15.445 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:15.445 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:15.458 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-19T11:31:15.483 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:15.489+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:15.562 INFO:tasks.rados.rados.0.smithi038.stdout:1327: expect (ObjNum 163 snap 17 seq_num 163) 2024-03-19T11:31:15.584 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:15.595+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:15.625 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:15.625 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:15.640 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-19T11:31:15.685 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:15.694+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:15.715 INFO:tasks.rados.rados.0.smithi038.stdout:1317: done (11 left) 2024-03-19T11:31:15.716 INFO:tasks.rados.rados.0.smithi038.stdout:1319: done (10 left) 2024-03-19T11:31:15.716 INFO:tasks.rados.rados.0.smithi038.stdout:1320: done (9 left) 2024-03-19T11:31:15.716 INFO:tasks.rados.rados.0.smithi038.stdout:1321: done (8 left) 2024-03-19T11:31:15.716 INFO:tasks.rados.rados.0.smithi038.stdout:1322: done (7 left) 2024-03-19T11:31:15.716 INFO:tasks.rados.rados.0.smithi038.stdout:1323: done (6 left) 2024-03-19T11:31:15.716 INFO:tasks.rados.rados.0.smithi038.stdout:1324: done (5 left) 2024-03-19T11:31:15.716 INFO:tasks.rados.rados.0.smithi038.stdout:1325: done (4 left) 2024-03-19T11:31:15.716 INFO:tasks.rados.rados.0.smithi038.stdout:1326: done (3 left) 2024-03-19T11:31:15.716 INFO:tasks.rados.rados.0.smithi038.stdout:1327: done (2 left) 2024-03-19T11:31:15.716 INFO:tasks.rados.rados.0.smithi038.stdout:1328: done (1 left) 2024-03-19T11:31:15.716 INFO:tasks.rados.rados.0.smithi038.stdout:1329: read oid 48 snap -1 2024-03-19T11:31:15.716 INFO:tasks.rados.rados.0.smithi038.stdout:1329: expect (ObjNum 177 snap 19 seq_num 177) 2024-03-19T11:31:15.716 INFO:tasks.rados.rados.0.smithi038.stdout:1330: rollback oid 22 current snap is 24 2024-03-19T11:31:15.717 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 22 to 9 2024-03-19T11:31:15.717 INFO:tasks.rados.rados.0.smithi038.stdout:1331: snap_create 2024-03-19T11:31:15.723 INFO:tasks.rados.rados.0.smithi038.stdout:1330: finishing rollback tid 0 to smithi03829856-22 2024-03-19T11:31:15.725 INFO:tasks.rados.rados.0.smithi038.stdout:1329: expect (ObjNum 177 snap 19 seq_num 177) 2024-03-19T11:31:15.786 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:15.793+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:15.805 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:15.805 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:15.819 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-19T11:31:15.860 INFO:tasks.rados.rados.0.smithi038.stdout:1330: finishing rollback tid 1 to smithi03829856-22 2024-03-19T11:31:15.860 INFO:tasks.rados.rados.0.smithi038.stdout:1330: finishing rollback tid 2 to smithi03829856-22 2024-03-19T11:31:15.860 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 22 v 23 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-03-19T11:31:15.887 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:15.898+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:15.935 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:15.936 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:15.949 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-19T11:31:15.992 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:16.002+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:16.062 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-03-19T11:31:16.067 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:16.074+0000 7f143d8b8640 -1 osd.5 96 handle_osd_map injecting map crc failure 2024-03-19T11:31:16.077 INFO:tasks.rados.rados.0.smithi038.stdout:1329: done (3 left) 2024-03-19T11:31:16.077 INFO:tasks.rados.rados.0.smithi038.stdout:1330: done (2 left) 2024-03-19T11:31:16.077 INFO:tasks.rados.rados.0.smithi038.stdout:1331: done (1 left) 2024-03-19T11:31:16.077 INFO:tasks.rados.rados.0.smithi038.stdout:1332: set_chunk oid 35 target oid 2024-03-19T11:31:16.116 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:16.117 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:16.127 INFO:tasks.rados.rados.0.smithi038.stdout:1332: set_chunk oid 35 offset: 565760 length: 3012096 target oid 136fb3407ab1f488caf5efdd46309d82c9d84045a5047f6aa3ae8f7ecd772a23 offset: 0 snap -1 2024-03-19T11:31:16.127 INFO:tasks.rados.rados.0.smithi038.stdout:1333: tier_evict oid 2 2024-03-19T11:31:16.127 INFO:tasks.rados.rados.0.smithi038.stdout:1333: tier_evict oid 2 snap -1 2024-03-19T11:31:16.127 INFO:tasks.rados.rados.0.smithi038.stdout:1313: expect (ObjNum 212 snap 22 seq_num 212) 2024-03-19T11:31:16.127 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:16.133+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:16.134 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-19T11:31:16.191 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:16.198+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:16.258 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:16.259 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:16.275 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-19T11:31:16.316 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:16.326+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:16.370 INFO:tasks.rados.rados.0.smithi038.stdout:1333: got (22) Invalid argument 2024-03-19T11:31:16.371 INFO:tasks.rados.rados.0.smithi038.stdout:1332: finishing set_chunk to oid 35 2024-03-19T11:31:16.371 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 35 v 196 (ObjNum 217 snap 23 seq_num 217) dirty exists 2024-03-19T11:31:16.371 INFO:tasks.rados.rados.0.smithi038.stdout:1313: done (2 left) 2024-03-19T11:31:16.371 INFO:tasks.rados.rados.0.smithi038.stdout:1332: done (1 left) 2024-03-19T11:31:16.371 INFO:tasks.rados.rados.0.smithi038.stdout:1333: done (0 left) 2024-03-19T11:31:16.371 INFO:tasks.rados.rados.0.smithi038.stdout:1334: write oid 19 current snap is 25 2024-03-19T11:31:16.371 INFO:tasks.rados.rados.0.smithi038.stdout:1334: seq_num 223 ranges {547961=554764,1616308=703740,2741092=698600} 2024-03-19T11:31:16.388 INFO:tasks.rados.rados.0.smithi038.stdout:1334: writing smithi03829856-19 from 547961 to 1102725 tid 1 2024-03-19T11:31:16.392 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:16.401+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:16.409 INFO:tasks.rados.rados.0.smithi038.stdout:1334: writing smithi03829856-19 from 1616308 to 2320048 tid 2 2024-03-19T11:31:16.426 INFO:tasks.rados.rados.0.smithi038.stdout:1334: writing smithi03829856-19 from 2741092 to 3439692 tid 3 2024-03-19T11:31:16.426 INFO:tasks.rados.rados.0.smithi038.stdout:1335: read oid 13 snap -1 2024-03-19T11:31:16.426 INFO:tasks.rados.rados.0.smithi038.stdout:1335: expect (ObjNum 12 snap 0 seq_num 12) 2024-03-19T11:31:16.427 INFO:tasks.rados.rados.0.smithi038.stdout:1336: rollback oid 36 current snap is 25 2024-03-19T11:31:16.427 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 36 to 22 2024-03-19T11:31:16.427 INFO:tasks.rados.rados.0.smithi038.stdout:1334: finishing write tid 1 to smithi03829856-19 2024-03-19T11:31:16.427 INFO:tasks.rados.rados.0.smithi038.stdout:1334: finishing write tid 2 to smithi03829856-19 2024-03-19T11:31:16.427 INFO:tasks.rados.rados.0.smithi038.stdout:1337: read oid 26 snap -1 2024-03-19T11:31:16.427 INFO:tasks.rados.rados.0.smithi038.stdout:1337: expect deleted 2024-03-19T11:31:16.427 INFO:tasks.rados.rados.0.smithi038.stdout:1338: write oid 23 current snap is 25 2024-03-19T11:31:16.427 INFO:tasks.rados.rados.0.smithi038.stdout:1338: seq_num 224 ranges {564768=630013,1674790=447398,2768515=626225} 2024-03-19T11:31:16.442 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:16.443 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:16.443 INFO:tasks.rados.rados.0.smithi038.stdout:1338: writing smithi03829856-23 from 564768 to 1194781 tid 1 2024-03-19T11:31:16.455 INFO:tasks.rados.rados.0.smithi038.stdout:1338: writing smithi03829856-23 from 1674790 to 2122188 tid 2 2024-03-19T11:31:16.461 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-19T11:31:16.471 INFO:tasks.rados.rados.0.smithi038.stdout:1338: writing smithi03829856-23 from 2768515 to 3394740 tid 3 2024-03-19T11:31:16.471 INFO:tasks.rados.rados.0.smithi038.stdout:1339: read oid 28 snap 22 2024-03-19T11:31:16.471 INFO:tasks.rados.rados.0.smithi038.stdout:1339: expect (ObjNum 87 snap 8 seq_num 87) 2024-03-19T11:31:16.471 INFO:tasks.rados.rados.0.smithi038.stdout:1340: write oid 14 current snap is 25 2024-03-19T11:31:16.472 INFO:tasks.rados.rados.0.smithi038.stdout:1340: seq_num 225 ranges {581575=705262,1733272=707409,3195938=437497} 2024-03-19T11:31:16.482 INFO:tasks.rados.rados.0.smithi038.stdout:1340: writing smithi03829856-14 from 581575 to 1286837 tid 1 2024-03-19T11:31:16.493 INFO:tasks.rados.rados.0.smithi038.stdout:1340: writing smithi03829856-14 from 1733272 to 2440681 tid 2 2024-03-19T11:31:16.495 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-19T11:31:16.502 INFO:tasks.rados.rados.0.smithi038.stdout:1340: writing smithi03829856-14 from 3195938 to 3633435 tid 3 2024-03-19T11:31:16.502 INFO:tasks.rados.rados.0.smithi038.stdout:1341: read oid 43 snap -1 2024-03-19T11:31:16.502 INFO:tasks.rados.rados.0.smithi038.stdout:1341: expect (ObjNum 198 snap 22 seq_num 198) 2024-03-19T11:31:16.503 INFO:tasks.rados.rados.0.smithi038.stdout:1342: read oid 30 snap 22 2024-03-19T11:31:16.503 INFO:tasks.rados.rados.0.smithi038.stdout:1342: expect (ObjNum 29 snap 0 seq_num 29) 2024-03-19T11:31:16.503 INFO:tasks.rados.rados.0.smithi038.stdout:1343: write (excl) oid 1 current snap is 25 2024-03-19T11:31:16.503 INFO:tasks.rados.rados.0.smithi038.stdout:1343: seq_num 226 ranges {598382=780511,2075401=451067,3107008=648769} 2024-03-19T11:31:16.511 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:16.522+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:16.512 INFO:tasks.rados.rados.0.smithi038.stdout:1343: writing smithi03829856-1 from 598382 to 1378893 tid 1 2024-03-19T11:31:16.519 INFO:tasks.rados.rados.0.smithi038.stdout:1343: writing smithi03829856-1 from 2075401 to 2526468 tid 2 2024-03-19T11:31:16.527 INFO:tasks.rados.rados.0.smithi038.stdout:1343: writing smithi03829856-1 from 3107008 to 3755777 tid 3 2024-03-19T11:31:16.527 INFO:tasks.rados.rados.0.smithi038.stdout:1344: snap_create 2024-03-19T11:31:16.527 INFO:tasks.rados.rados.0.smithi038.stdout:1336: finishing rollback tid 1 to smithi03829856-36 2024-03-19T11:31:16.527 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 36 v 92 (ObjNum 513 snap 0 seq_num 0) dirty dne 2024-03-19T11:31:16.527 INFO:tasks.rados.rados.0.smithi038.stdout:1334: finishing write tid 3 to smithi03829856-19 2024-03-19T11:31:16.528 INFO:tasks.rados.rados.0.smithi038.stdout:1334: finishing write tid 4 to smithi03829856-19 2024-03-19T11:31:16.528 INFO:tasks.rados.rados.0.smithi038.stdout:1334: finishing write tid 5 to smithi03829856-19 2024-03-19T11:31:16.528 INFO:tasks.rados.rados.0.smithi038.stdout:1334: oid 19 updating version 0 to 82 2024-03-19T11:31:16.528 INFO:tasks.rados.rados.0.smithi038.stdout:1334: oid 19 updating version 82 to 84 2024-03-19T11:31:16.528 INFO:tasks.rados.rados.0.smithi038.stdout:1334: oid 19 version 84 is already newer than 81 2024-03-19T11:31:16.528 INFO:tasks.rados.rados.0.smithi038.stdout:1334: oid 19 version 84 is already newer than 83 2024-03-19T11:31:16.528 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 19 v 84 (ObjNum 223 snap 25 seq_num 223) dirty exists 2024-03-19T11:31:16.528 INFO:tasks.rados.rados.0.smithi038.stdout:1334: left oid 19 (ObjNum 223 snap 25 seq_num 223) 2024-03-19T11:31:16.528 INFO:tasks.rados.rados.0.smithi038.stdout:1335: expect (ObjNum 12 snap 0 seq_num 12) 2024-03-19T11:31:16.589 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:16.589 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:16.594 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:16.602+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:16.596 INFO:tasks.rados.rados.0.smithi038.stdout:1338: finishing write tid 1 to smithi03829856-23 2024-03-19T11:31:16.596 INFO:tasks.rados.rados.0.smithi038.stdout:1338: finishing write tid 2 to smithi03829856-23 2024-03-19T11:31:16.596 INFO:tasks.rados.rados.0.smithi038.stdout:1339: expect (ObjNum 87 snap 8 seq_num 87) 2024-03-19T11:31:16.603 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-19T11:31:16.639 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:31:16.639 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:31:16.655 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:31:16.656 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:31:16.672 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:31:16.672 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:31:16.688 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:31:16.689 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:31:16.695 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:16.701+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:16.704 INFO:tasks.rados.rados.0.smithi038.stdout:1338: finishing write tid 3 to smithi03829856-23 2024-03-19T11:31:16.705 INFO:tasks.rados.rados.0.smithi038.stdout:1338: finishing write tid 4 to smithi03829856-23 2024-03-19T11:31:16.705 INFO:tasks.rados.rados.0.smithi038.stdout:1338: finishing write tid 5 to smithi03829856-23 2024-03-19T11:31:16.705 INFO:tasks.rados.rados.0.smithi038.stdout:1338: oid 23 updating version 0 to 114 2024-03-19T11:31:16.705 INFO:tasks.rados.rados.0.smithi038.stdout:1338: oid 23 version 114 is already newer than 113 2024-03-19T11:31:16.705 INFO:tasks.rados.rados.0.smithi038.stdout:1338: oid 23 updating version 114 to 116 2024-03-19T11:31:16.705 INFO:tasks.rados.rados.0.smithi038.stdout:1338: oid 23 version 116 is already newer than 115 2024-03-19T11:31:16.705 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 23 v 116 (ObjNum 224 snap 25 seq_num 224) dirty exists 2024-03-19T11:31:16.705 INFO:tasks.rados.rados.0.smithi038.stdout:1338: left oid 23 (ObjNum 224 snap 25 seq_num 224) 2024-03-19T11:31:16.705 INFO:tasks.rados.rados.0.smithi038.stdout:1340: finishing write tid 1 to smithi03829856-14 2024-03-19T11:31:16.705 INFO:tasks.rados.rados.0.smithi038.stdout:1340: finishing write tid 2 to smithi03829856-14 2024-03-19T11:31:16.705 INFO:tasks.rados.rados.0.smithi038.stdout:1340: finishing write tid 3 to smithi03829856-14 2024-03-19T11:31:16.705 INFO:tasks.rados.rados.0.smithi038.stdout:1340: finishing write tid 4 to smithi03829856-14 2024-03-19T11:31:16.705 INFO:tasks.rados.rados.0.smithi038.stdout:1340: finishing write tid 5 to smithi03829856-14 2024-03-19T11:31:16.705 INFO:tasks.rados.rados.0.smithi038.stdout:1340: oid 14 updating version 0 to 61 2024-03-19T11:31:16.706 INFO:tasks.rados.rados.0.smithi038.stdout:1340: oid 14 version 61 is already newer than 60 2024-03-19T11:31:16.706 INFO:tasks.rados.rados.0.smithi038.stdout:1340: oid 14 version 61 is already newer than 58 2024-03-19T11:31:16.706 INFO:tasks.rados.rados.0.smithi038.stdout:1340: oid 14 version 61 is already newer than 59 2024-03-19T11:31:16.706 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 14 v 61 (ObjNum 225 snap 25 seq_num 225) dirty exists 2024-03-19T11:31:16.706 INFO:tasks.rados.rados.0.smithi038.stdout:1340: left oid 14 (ObjNum 225 snap 25 seq_num 225) 2024-03-19T11:31:16.706 INFO:tasks.rados.rados.0.smithi038.stdout:1342: expect (ObjNum 29 snap 0 seq_num 29) 2024-03-19T11:31:16.767 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:16.767 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:16.781 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-19T11:31:16.784 INFO:tasks.rados.rados.0.smithi038.stdout:1341: expect (ObjNum 198 snap 22 seq_num 198) 2024-03-19T11:31:16.797 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:16.805+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:16.889 INFO:tasks.rados.rados.0.smithi038.stdout:1343: finishing write tid 1 to smithi03829856-1 2024-03-19T11:31:16.889 INFO:tasks.rados.rados.0.smithi038.stdout:1343: finishing write tid 2 to smithi03829856-1 2024-03-19T11:31:16.889 INFO:tasks.rados.rados.0.smithi038.stdout:1343: finishing write tid 3 to smithi03829856-1 2024-03-19T11:31:16.889 INFO:tasks.rados.rados.0.smithi038.stdout:1343: finishing write tid 4 to smithi03829856-1 2024-03-19T11:31:16.889 INFO:tasks.rados.rados.0.smithi038.stdout:1343: finishing write tid 5 to smithi03829856-1 2024-03-19T11:31:16.890 INFO:tasks.rados.rados.0.smithi038.stdout:1343: oid 1 updating version 0 to 141 2024-03-19T11:31:16.890 INFO:tasks.rados.rados.0.smithi038.stdout:1343: oid 1 updating version 141 to 144 2024-03-19T11:31:16.890 INFO:tasks.rados.rados.0.smithi038.stdout:1343: oid 1 version 144 is already newer than 143 2024-03-19T11:31:16.890 INFO:tasks.rados.rados.0.smithi038.stdout:1343: oid 1 version 144 is already newer than 142 2024-03-19T11:31:16.890 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 1 v 144 (ObjNum 226 snap 25 seq_num 226) dirty exists 2024-03-19T11:31:16.890 INFO:tasks.rados.rados.0.smithi038.stdout:1343: left oid 1 (ObjNum 226 snap 25 seq_num 226) 2024-03-19T11:31:16.897 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:16.905+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:16.899 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:16.899 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:16.913 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:31:16.913 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:31:16.914 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-19T11:31:16.943 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:31:16.943 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:31:16.972 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:31:16.972 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:31:16.998 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:17.005+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:17.003 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:31:17.003 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:31:17.044 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:17.044 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:31:17.044 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:31:17.045 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:17.059 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-19T11:31:17.099 INFO:tasks.rados.rados.0.smithi038.stdout:1334: done (10 left) 2024-03-19T11:31:17.099 INFO:tasks.rados.rados.0.smithi038.stdout:1335: done (9 left) 2024-03-19T11:31:17.099 INFO:tasks.rados.rados.0.smithi038.stdout:1336: done (8 left) 2024-03-19T11:31:17.099 INFO:tasks.rados.rados.0.smithi038.stdout:1337: done (7 left) 2024-03-19T11:31:17.099 INFO:tasks.rados.rados.0.smithi038.stdout:1338: done (6 left) 2024-03-19T11:31:17.099 INFO:tasks.rados.rados.0.smithi038.stdout:1339: done (5 left) 2024-03-19T11:31:17.099 INFO:tasks.rados.rados.0.smithi038.stdout:1340: done (4 left) 2024-03-19T11:31:17.099 INFO:tasks.rados.rados.0.smithi038.stdout:1341: done (3 left) 2024-03-19T11:31:17.099 INFO:tasks.rados.rados.0.smithi038.stdout:1342: done (2 left) 2024-03-19T11:31:17.100 INFO:tasks.rados.rados.0.smithi038.stdout:1343: done (1 left) 2024-03-19T11:31:17.100 INFO:tasks.rados.rados.0.smithi038.stdout:1344: done (0 left) 2024-03-19T11:31:17.100 INFO:tasks.rados.rados.0.smithi038.stdout:1345: write (excl) oid 40 current snap is 26 2024-03-19T11:31:17.100 INFO:tasks.rados.rados.0.smithi038.stdout:1345: seq_num 227 ranges {615189=455760,1733883=711078,3134431=460041} 2024-03-19T11:31:17.100 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:17.106+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:17.105 INFO:tasks.rados.rados.0.smithi038.stdout:1345: writing smithi03829856-40 from 615189 to 1070949 tid 1 2024-03-19T11:31:17.113 INFO:tasks.rados.rados.0.smithi038.stdout:1345: writing smithi03829856-40 from 1733883 to 2444961 tid 2 2024-03-19T11:31:17.120 INFO:tasks.rados.rados.0.smithi038.stdout:1345: writing smithi03829856-40 from 3134431 to 3594472 tid 3 2024-03-19T11:31:17.120 INFO:tasks.rados.rados.0.smithi038.stdout:1346: read oid 42 snap 21 2024-03-19T11:31:17.120 INFO:tasks.rados.rados.0.smithi038.stdout:1346: expect deleted 2024-03-19T11:31:17.120 INFO:tasks.rados.rados.0.smithi038.stdout:1347: read oid 34 snap -1 2024-03-19T11:31:17.120 INFO:tasks.rados.rados.0.smithi038.stdout:1347: expect (ObjNum 72 snap 5 seq_num 72) 2024-03-19T11:31:17.120 INFO:tasks.rados.rados.0.smithi038.stdout:1348: read oid 43 snap -1 2024-03-19T11:31:17.120 INFO:tasks.rados.rados.0.smithi038.stdout:1348: expect (ObjNum 198 snap 22 seq_num 198) 2024-03-19T11:31:17.120 INFO:tasks.rados.rados.0.smithi038.stdout:1349: set_chunk oid 6 target oid 2024-03-19T11:31:17.130 INFO:tasks.rados.rados.0.smithi038.stdout:1349: set_chunk oid 6 offset: 3005440 length: 95232 target oid 2dad33fbd877bbfffbd98dd5e580c6d35235defb00d8784c599c832811eca8f6 offset: 0 snap -1 2024-03-19T11:31:17.131 INFO:tasks.rados.rados.0.smithi038.stdout:1350: tier_promote oid 38 2024-03-19T11:31:17.131 INFO:tasks.rados.rados.0.smithi038.stdout:1345: finishing write tid 1 to smithi03829856-40 2024-03-19T11:31:17.131 INFO:tasks.rados.rados.0.smithi038.stdout:1345: finishing write tid 2 to smithi03829856-40 2024-03-19T11:31:17.131 INFO:tasks.rados.rados.0.smithi038.stdout:1346: done (5 left) 2024-03-19T11:31:17.131 INFO:tasks.rados.rados.0.smithi038.stdout:1351: read oid 49 snap -1 2024-03-19T11:31:17.131 INFO:tasks.rados.rados.0.smithi038.stdout:1351: expect (ObjNum 201 snap 22 seq_num 201) 2024-03-19T11:31:17.131 INFO:tasks.rados.rados.0.smithi038.stdout:1352: read oid 50 snap -1 2024-03-19T11:31:17.131 INFO:tasks.rados.rados.0.smithi038.stdout:1352: expect (ObjNum 49 snap 0 seq_num 49) 2024-03-19T11:31:17.131 INFO:tasks.rados.rados.0.smithi038.stdout:1353: read oid 2 snap 22 2024-03-19T11:31:17.131 INFO:tasks.rados.rados.0.smithi038.stdout:1353: expect (ObjNum 206 snap 22 seq_num 206) 2024-03-19T11:31:17.131 INFO:tasks.rados.rados.0.smithi038.stdout:1354: snap_remove snap 19 2024-03-19T11:31:17.200 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:17.209+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:17.233 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:17.233 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:31:17.233 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:31:17.233 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:31:17.233 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:17.248 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-19T11:31:17.301 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:17.310+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:17.402 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:17.409+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:17.426 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:17.426 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:31:17.426 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:31:17.427 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:31:17.427 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.427 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1217 4.3 4:c8b09e87:::smithi03829856-15:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:31:17.427 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.155966+0000", 2024-03-19T11:31:17.427 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 59.278008722999999, 2024-03-19T11:31:17.427 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.023984647000000001, 2024-03-19T11:31:17.427 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:17.427 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:17.427 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:17.427 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:17.427 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:17.427 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1217 2024-03-19T11:31:17.427 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.428 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:17.428 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.428 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:17.428 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155966+0000", 2024-03-19T11:31:17.428 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:17.428 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.428 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.428 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:17.428 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155966+0000", 2024-03-19T11:31:17.428 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:17.428 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.428 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.428 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:17.429 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155968+0000", 2024-03-19T11:31:17.429 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.795e-06 2024-03-19T11:31:17.429 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.429 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.429 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:17.429 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155978+0000", 2024-03-19T11:31:17.429 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0574000000000001e-05 2024-03-19T11:31:17.429 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.429 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.429 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:17.429 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.156000+0000", 2024-03-19T11:31:17.429 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1250000000000002e-05 2024-03-19T11:31:17.429 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.429 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.430 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:17.430 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.156010+0000", 2024-03-19T11:31:17.430 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0168e-05 2024-03-19T11:31:17.430 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.430 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.430 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:17.430 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.156119+0000", 2024-03-19T11:31:17.430 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010950699999999999 2024-03-19T11:31:17.430 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.430 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.430 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:17.430 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.156231+0000", 2024-03-19T11:31:17.430 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000111445 2024-03-19T11:31:17.431 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.431 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.431 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:17.431 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.179951+0000", 2024-03-19T11:31:17.431 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.023719908000000001 2024-03-19T11:31:17.431 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:17.431 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:17.431 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:17.431 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.431 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.431 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.24 75/74 [PushOp(4:27909373:::smithi03829856-46:8, version: 46'59, data_included: [756315~767970,1933572~280606], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:8@46'59, size: 2756315, copy_subset: [(0, 2756315)], clone_subset: {}, snapset: 10=[]:{8: [6], c: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2214178, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:266590a5:::smithi03829856-16:e, version: 70'107, data_included: [652105~662359,1728171~386217], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:e@70'107, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 10=[]:{e: [d, b, 6]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:266590a5:::smithi03829856-16:head, version: 70'108, data_included: [652105~662359,1728171~386217], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:head@70'108, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:27909373:::smithi03829856-46:c, version: 70'109, data_included: [628595~895690,1807440~152886], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:c@70'109, size: 3428595, copy_subset: [(0, 3428595)], clone_subset: {}, snapset: 10=[]:{8: [6], c: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1960326, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:27909373:::smithi03829856-46:head, version: 70'110, data_included: [403401~454961,1584925~506039], data_size: 961000, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:head@70'110, size: 2403401, copy_subset: [(0, 2403401)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2403401, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:17.431 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.099395+0000", 2024-03-19T11:31:17.432 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 23.334579561999998, 2024-03-19T11:31:17.432 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.045391157000000001, 2024-03-19T11:31:17.432 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:17.432 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:17.432 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:17.432 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.432 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:17.432 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.099395+0000", 2024-03-19T11:31:17.432 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:17.432 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.432 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.432 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:17.432 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.099395+0000", 2024-03-19T11:31:17.433 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:17.433 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.433 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.433 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:17.433 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.099397+0000", 2024-03-19T11:31:17.433 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.691e-06 2024-03-19T11:31:17.433 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.433 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.433 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:17.433 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105009+0000", 2024-03-19T11:31:17.433 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0056121349999999999 2024-03-19T11:31:17.433 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.434 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.434 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:17.434 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105077+0000", 2024-03-19T11:31:17.434 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8561999999999998e-05 2024-03-19T11:31:17.434 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.434 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.434 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:17.434 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105133+0000", 2024-03-19T11:31:17.434 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5590000000000001e-05 2024-03-19T11:31:17.434 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.434 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.434 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:17.434 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105531+0000", 2024-03-19T11:31:17.435 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00039756200000000001 2024-03-19T11:31:17.435 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.435 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.435 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:17.435 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105579+0000", 2024-03-19T11:31:17.436 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8028999999999997e-05 2024-03-19T11:31:17.436 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.436 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.436 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:17.436 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.144786+0000", 2024-03-19T11:31:17.436 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.039207588000000002 2024-03-19T11:31:17.436 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:17.436 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:17.436 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:17.436 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.437 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.437 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.24 e75,74 4:266590a5:::smithi03829856-16:4,47'63,4:27909373:::smithi03829856-46:5,55'79,4:266590a5:::smithi03829856-16:10,67'105)", 2024-03-19T11:31:17.437 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.105191+0000", 2024-03-19T11:31:17.437 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 23.328783462000001, 2024-03-19T11:31:17.437 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.039580830999999997, 2024-03-19T11:31:17.437 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:17.437 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:17.437 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:17.437 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.437 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:17.437 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105191+0000", 2024-03-19T11:31:17.437 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:17.437 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.437 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.437 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:17.438 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105191+0000", 2024-03-19T11:31:17.438 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:17.438 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.438 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.438 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:17.438 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105192+0000", 2024-03-19T11:31:17.438 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.099e-06 2024-03-19T11:31:17.438 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.438 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.438 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:17.438 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105200+0000", 2024-03-19T11:31:17.438 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5279999999999998e-06 2024-03-19T11:31:17.438 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.438 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.439 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:17.439 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105211+0000", 2024-03-19T11:31:17.439 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1471000000000001e-05 2024-03-19T11:31:17.439 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.439 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.439 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:17.439 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105220+0000", 2024-03-19T11:31:17.439 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3469999999999999e-06 2024-03-19T11:31:17.439 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.439 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.439 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:17.440 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.144219+0000", 2024-03-19T11:31:17.440 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.038999024 2024-03-19T11:31:17.440 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.440 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.440 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:17.440 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.144257+0000", 2024-03-19T11:31:17.440 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8501999999999998e-05 2024-03-19T11:31:17.440 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.440 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.440 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:17.440 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.144772+0000", 2024-03-19T11:31:17.440 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00051486000000000001 2024-03-19T11:31:17.440 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:17.440 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:17.441 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:17.441 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.441 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.441 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 75/74 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:7, version: 55'82, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@55'82, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 58'86, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@58'86, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [504210~968657,1572695~79919], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:12, version: 66'101, data_included: [440350~816629,1309629~231947], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:12@66'101, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:10, version: 70'112, data_included: [440350~816629,1309629~231947], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:10@70'112, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 70'113, data_included: [440350~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@70'113, size: 2605085, copy_subset: [(0, 2605085)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1488926, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:17.441 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.107266+0000", 2024-03-19T11:31:17.441 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 23.326708945, 2024-03-19T11:31:17.441 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.048592669999999998, 2024-03-19T11:31:17.441 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:17.441 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:17.441 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:17.441 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.441 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:17.441 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.107266+0000", 2024-03-19T11:31:17.441 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:17.442 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.442 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.442 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:17.442 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.107266+0000", 2024-03-19T11:31:17.442 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:17.442 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.442 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.442 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:17.442 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.107267+0000", 2024-03-19T11:31:17.442 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1459999999999999e-06 2024-03-19T11:31:17.442 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.442 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.442 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:17.443 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.122861+0000", 2024-03-19T11:31:17.443 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01559372 2024-03-19T11:31:17.443 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.443 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.443 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:17.443 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.122932+0000", 2024-03-19T11:31:17.443 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1477999999999996e-05 2024-03-19T11:31:17.443 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.443 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.443 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:17.443 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.122999+0000", 2024-03-19T11:31:17.443 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6832e-05 2024-03-19T11:31:17.443 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.443 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.443 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:17.444 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123332+0000", 2024-03-19T11:31:17.444 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000333271 2024-03-19T11:31:17.444 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.444 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.444 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:17.444 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123371+0000", 2024-03-19T11:31:17.444 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8525999999999997e-05 2024-03-19T11:31:17.444 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.444 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.444 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:17.444 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.155858+0000", 2024-03-19T11:31:17.444 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.032487697000000003 2024-03-19T11:31:17.444 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:17.444 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:17.445 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:17.445 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.445 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.445 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.21 e75,74 4:8401c1f0:::smithi03829856-43:8,46'66,4:866f3dcc:::smithi03829856-31:2,47'73)", 2024-03-19T11:31:17.445 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.123075+0000", 2024-03-19T11:31:17.445 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 23.310899725999999, 2024-03-19T11:31:17.445 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.033023045000000001, 2024-03-19T11:31:17.445 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:17.446 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:17.446 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:17.446 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.447 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:17.447 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123075+0000", 2024-03-19T11:31:17.447 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:17.447 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.447 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.447 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:17.447 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123075+0000", 2024-03-19T11:31:17.447 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:17.447 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.447 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.447 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:17.447 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123076+0000", 2024-03-19T11:31:17.447 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5e-06 2024-03-19T11:31:17.447 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.447 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.448 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:17.448 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123086+0000", 2024-03-19T11:31:17.448 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8060000000000006e-06 2024-03-19T11:31:17.448 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.448 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.448 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:17.448 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123099+0000", 2024-03-19T11:31:17.448 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3251000000000001e-05 2024-03-19T11:31:17.448 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.448 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.448 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:17.448 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123112+0000", 2024-03-19T11:31:17.448 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2774e-05 2024-03-19T11:31:17.448 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.449 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.449 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:17.449 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.155902+0000", 2024-03-19T11:31:17.449 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.032790225999999999 2024-03-19T11:31:17.449 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.449 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.449 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:17.449 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.155914+0000", 2024-03-19T11:31:17.450 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1476e-05 2024-03-19T11:31:17.450 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.450 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.450 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:17.450 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.156098+0000", 2024-03-19T11:31:17.450 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000184012 2024-03-19T11:31:17.450 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:17.450 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:17.450 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:17.450 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.450 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.450 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 75/74 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:7, version: 55'82, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@55'82, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 58'86, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@58'86, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [1652614~973634,2888511~52681,3182305~22261], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3204566, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:12, version: 66'101, data_included: [1541576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:12@66'101, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:10, version: 70'112, data_included: [1541576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:10@70'112, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 70'113, data_included: [1488926~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@70'113, size: 2605085, copy_subset: [(0, 2605085)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2537502, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1488926, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:17.450 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.174030+0000", 2024-03-19T11:31:17.451 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 23.259944522000001, 2024-03-19T11:31:17.451 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.040799275000000003, 2024-03-19T11:31:17.451 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:17.451 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:17.451 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:17.451 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.451 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:17.451 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.174030+0000", 2024-03-19T11:31:17.451 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:17.451 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.451 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.451 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:17.451 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.174030+0000", 2024-03-19T11:31:17.452 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:17.452 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.452 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.452 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:17.452 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.174031+0000", 2024-03-19T11:31:17.452 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.322e-06 2024-03-19T11:31:17.452 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.452 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.452 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:17.452 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.184771+0000", 2024-03-19T11:31:17.453 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010739227 2024-03-19T11:31:17.453 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.453 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.453 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:17.453 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.184872+0000", 2024-03-19T11:31:17.453 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000101234 2024-03-19T11:31:17.453 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.453 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.453 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:17.453 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.184985+0000", 2024-03-19T11:31:17.453 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000113545 2024-03-19T11:31:17.453 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.453 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.453 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:17.453 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.185463+0000", 2024-03-19T11:31:17.454 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00047749899999999999 2024-03-19T11:31:17.454 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.454 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.454 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:17.454 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.185534+0000", 2024-03-19T11:31:17.454 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1007999999999995e-05 2024-03-19T11:31:17.454 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.454 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.454 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:17.454 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.214829+0000", 2024-03-19T11:31:17.454 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029295439999999999 2024-03-19T11:31:17.454 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:17.454 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:17.455 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:17.455 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.455 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.455 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.21 e75,74 4:866f3dcc:::smithi03829856-31:c,70'114)", 2024-03-19T11:31:17.455 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.197457+0000", 2024-03-19T11:31:17.455 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 23.236517476, 2024-03-19T11:31:17.455 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018493590000000001, 2024-03-19T11:31:17.455 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:17.455 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:17.455 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:17.455 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.455 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:17.455 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.197457+0000", 2024-03-19T11:31:17.455 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:17.455 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.456 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.456 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:17.456 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.197457+0000", 2024-03-19T11:31:17.456 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:17.456 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.456 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.456 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:17.456 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.197459+0000", 2024-03-19T11:31:17.456 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3199999999999998e-06 2024-03-19T11:31:17.457 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.457 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.457 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:17.457 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.197480+0000", 2024-03-19T11:31:17.457 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0366000000000001e-05 2024-03-19T11:31:17.457 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.457 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.457 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:17.457 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.197521+0000", 2024-03-19T11:31:17.457 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0738000000000002e-05 2024-03-19T11:31:17.457 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.457 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.457 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:17.457 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.197547+0000", 2024-03-19T11:31:17.458 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6548999999999999e-05 2024-03-19T11:31:17.458 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.458 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.458 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:17.458 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.214954+0000", 2024-03-19T11:31:17.458 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01740713 2024-03-19T11:31:17.458 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.458 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.458 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:17.458 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.214972+0000", 2024-03-19T11:31:17.458 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7520000000000002e-05 2024-03-19T11:31:17.458 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.458 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.459 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:17.459 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.215951+0000", 2024-03-19T11:31:17.459 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00097896700000000007 2024-03-19T11:31:17.459 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:17.459 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:17.459 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:17.459 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.459 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.459 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 75/74 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [2599014~241336], data_size: 241336, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2840350, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [2599014~241336,2948041~446940], data_size: 688276, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3394981, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [3204566~621403], data_size: 621403, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3865577, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3204566, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:12, version: 66'101, data_included: [2590152~250198,2948041~581396], data_size: 831594, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:12@66'101, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3529437, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:10, version: 70'112, data_included: [2590152~250198,2948041~581396], data_size: 831594, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:10@70'112, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3529437, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 70'113, data_included: [2537502~67583], data_size: 67583, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@70'113, size: 2605085, copy_subset: [(0, 2605085)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2605085, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2537502, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:17.459 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.222872+0000", 2024-03-19T11:31:17.459 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 23.211103029, 2024-03-19T11:31:17.459 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016277658, 2024-03-19T11:31:17.459 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:17.460 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:17.460 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:17.460 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.460 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:17.460 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.222872+0000", 2024-03-19T11:31:17.460 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:17.460 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.460 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.460 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:17.460 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.222872+0000", 2024-03-19T11:31:17.460 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:17.460 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.460 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.460 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:17.460 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.222874+0000", 2024-03-19T11:31:17.461 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9999999999999999e-06 2024-03-19T11:31:17.461 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.461 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.461 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:17.461 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.227563+0000", 2024-03-19T11:31:17.461 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046897570000000001 2024-03-19T11:31:17.461 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.461 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.461 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:17.461 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.227639+0000", 2024-03-19T11:31:17.461 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5607999999999998e-05 2024-03-19T11:31:17.461 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.461 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.462 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:17.462 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.227736+0000", 2024-03-19T11:31:17.462 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6854000000000005e-05 2024-03-19T11:31:17.462 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.462 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.462 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:17.462 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.228149+0000", 2024-03-19T11:31:17.462 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00041330000000000002 2024-03-19T11:31:17.463 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.463 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.463 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:17.463 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.228219+0000", 2024-03-19T11:31:17.463 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.0140000000000003e-05 2024-03-19T11:31:17.463 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.463 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.463 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:17.464 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.239149+0000", 2024-03-19T11:31:17.464 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010929998999999999 2024-03-19T11:31:17.464 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:17.464 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:17.464 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:17.464 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.464 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.464 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2372 4.16 4:6868af91:::smithi03829856-10:b [read 832512~818688] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-03-19T11:31:17.464 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:58.455732+0000", 2024-03-19T11:31:17.465 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.978242807000001, 2024-03-19T11:31:17.465 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.69150541200000004, 2024-03-19T11:31:17.465 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:17.465 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:17.465 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:17.465 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:17.465 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:17.465 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2372 2024-03-19T11:31:17.465 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.466 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:17.466 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.466 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:17.466 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455732+0000", 2024-03-19T11:31:17.467 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:17.467 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.467 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.467 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:17.467 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455732+0000", 2024-03-19T11:31:17.467 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:17.467 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.467 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.468 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:17.468 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455735+0000", 2024-03-19T11:31:17.468 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.756e-06 2024-03-19T11:31:17.468 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.468 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.468 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:17.468 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455744+0000", 2024-03-19T11:31:17.468 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1880000000000008e-06 2024-03-19T11:31:17.468 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.469 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.469 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:17.469 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455764+0000", 2024-03-19T11:31:17.469 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0044999999999999e-05 2024-03-19T11:31:17.469 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.469 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.469 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:17.469 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455774+0000", 2024-03-19T11:31:17.470 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8770000000000005e-06 2024-03-19T11:31:17.470 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.470 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.470 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:17.470 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455837+0000", 2024-03-19T11:31:17.470 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3588000000000002e-05 2024-03-19T11:31:17.470 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.470 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.471 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:17.471 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455846+0000", 2024-03-19T11:31:17.471 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6030000000000005e-06 2024-03-19T11:31:17.471 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.471 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.471 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:17.471 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.076525+0000", 2024-03-19T11:31:17.471 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.62067875900000002 2024-03-19T11:31:17.471 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.472 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.472 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:31:17.472 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.076981+0000", 2024-03-19T11:31:17.472 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00045633300000000001 2024-03-19T11:31:17.472 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.472 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.472 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:17.472 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.104422+0000", 2024-03-19T11:31:17.473 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027440665999999999 2024-03-19T11:31:17.473 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.473 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.473 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:31:17.473 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.104786+0000", 2024-03-19T11:31:17.473 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00036471499999999999 2024-03-19T11:31:17.473 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.473 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.473 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:17.474 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.140725+0000", 2024-03-19T11:31:17.474 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.035939093999999998 2024-03-19T11:31:17.474 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.474 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.474 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:31:17.474 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.140841+0000", 2024-03-19T11:31:17.474 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000115207 2024-03-19T11:31:17.474 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.475 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.475 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:17.475 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.145559+0000", 2024-03-19T11:31:17.475 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.004718486 2024-03-19T11:31:17.475 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.475 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.475 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:17.475 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.145760+0000", 2024-03-19T11:31:17.475 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000201058 2024-03-19T11:31:17.476 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.476 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.476 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:17.476 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.147237+0000", 2024-03-19T11:31:17.476 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014770370000000001 2024-03-19T11:31:17.476 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:17.476 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:17.477 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:17.477 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.477 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.477 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.16 80/79 [PushOp(4:6d9093b1:::smithi03829856-38:head, version: 77'90, data_included: [500875~689757,1564955~358819], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:6d9093b1:::smithi03829856-38:head@77'90, size: 2100875, copy_subset: [(0, 2100875)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1923774, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:6be89c06:::smithi03829856-24:head, version: 77'92, data_included: [741376~589824,1753088~458752], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:6be89c06:::smithi03829856-24:head@77'92, size: 2386561, copy_subset: [(0, 2386561)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2211840, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:17.477 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:59.081830+0000", 2024-03-19T11:31:17.477 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.352144781, 2024-03-19T11:31:17.478 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015229652999999999, 2024-03-19T11:31:17.478 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:17.478 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:17.478 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:17.478 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.478 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:17.478 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.081830+0000", 2024-03-19T11:31:17.478 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:17.478 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.478 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.478 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:17.478 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.081830+0000", 2024-03-19T11:31:17.478 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:17.479 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.479 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.479 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:17.479 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.081832+0000", 2024-03-19T11:31:17.479 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.731e-06 2024-03-19T11:31:17.479 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.479 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.479 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:17.479 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.083140+0000", 2024-03-19T11:31:17.479 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0013082829999999999 2024-03-19T11:31:17.479 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.479 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.479 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:17.479 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.083170+0000", 2024-03-19T11:31:17.479 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0270999999999999e-05 2024-03-19T11:31:17.480 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.480 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.480 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:17.480 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.083199+0000", 2024-03-19T11:31:17.480 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9298e-05 2024-03-19T11:31:17.480 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.480 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.480 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:17.480 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.089969+0000", 2024-03-19T11:31:17.480 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0067691289999999996 2024-03-19T11:31:17.480 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.480 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.480 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:17.480 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.089995+0000", 2024-03-19T11:31:17.480 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5995000000000001e-05 2024-03-19T11:31:17.481 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.481 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.481 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:17.481 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.097060+0000", 2024-03-19T11:31:17.481 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0070649459999999999 2024-03-19T11:31:17.481 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:17.481 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:17.481 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:17.481 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.481 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.481 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.6 80/79 [PushOp(4:626acd4d:::smithi03829856-2:8, version: 55'54, data_included: [416807~475249,1542129~474678], data_size: 949927, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:626acd4d:::smithi03829856-2:8@55'54, size: 2016807, copy_subset: [(0, 2016807)], clone_subset: {}, snapset: 15=[]:{8: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2016807, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:17.481 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:59.110928+0000", 2024-03-19T11:31:17.481 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.323046467000001, 2024-03-19T11:31:17.481 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01605765, 2024-03-19T11:31:17.482 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:17.482 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:17.482 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:17.482 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.482 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:17.482 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.110928+0000", 2024-03-19T11:31:17.482 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:17.482 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.482 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.482 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:17.482 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.110928+0000", 2024-03-19T11:31:17.482 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:17.482 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.483 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.483 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:17.483 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.110929+0000", 2024-03-19T11:31:17.483 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6500000000000008e-07 2024-03-19T11:31:17.483 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.483 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.483 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:17.483 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.111446+0000", 2024-03-19T11:31:17.483 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00051713900000000001 2024-03-19T11:31:17.483 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.483 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.483 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:17.483 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.111464+0000", 2024-03-19T11:31:17.483 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7966e-05 2024-03-19T11:31:17.484 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.484 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.484 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:17.484 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.111483+0000", 2024-03-19T11:31:17.484 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8561000000000001e-05 2024-03-19T11:31:17.484 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.484 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.484 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:17.484 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.118569+0000", 2024-03-19T11:31:17.484 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0070860259999999996 2024-03-19T11:31:17.484 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.484 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.484 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:17.484 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.118586+0000", 2024-03-19T11:31:17.484 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6857000000000001e-05 2024-03-19T11:31:17.485 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.485 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.485 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:17.485 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.126986+0000", 2024-03-19T11:31:17.485 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0084001360000000008 2024-03-19T11:31:17.485 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:17.485 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:17.485 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:17.485 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.485 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.485 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.16 80/79 [PushOp(4:6868af91:::smithi03829856-10:10, version: 70'81, data_included: [524385~820472,1602039~228104], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:10@70'81, size: 3747928, copy_subset: [(0, 3747928)], clone_subset: {}, snapset: 14=[]:{7: [6], b: [b], 10: [d], 14: [13, 12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1830143, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:17.485 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:59.111343+0000", 2024-03-19T11:31:17.486 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.322631876999999, 2024-03-19T11:31:17.486 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019054888999999998, 2024-03-19T11:31:17.486 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:17.486 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:17.486 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:17.486 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.486 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:17.486 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.111343+0000", 2024-03-19T11:31:17.486 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:17.486 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.486 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.486 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:17.486 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.111343+0000", 2024-03-19T11:31:17.487 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:17.487 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.487 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.487 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:17.487 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.111349+0000", 2024-03-19T11:31:17.487 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9619999999999998e-06 2024-03-19T11:31:17.487 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.487 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.487 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:17.487 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.113881+0000", 2024-03-19T11:31:17.487 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025322930000000001 2024-03-19T11:31:17.487 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.487 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.487 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:17.488 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.113917+0000", 2024-03-19T11:31:17.488 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5593999999999998e-05 2024-03-19T11:31:17.488 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.488 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.488 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:17.488 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.113962+0000", 2024-03-19T11:31:17.488 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5362999999999999e-05 2024-03-19T11:31:17.488 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.488 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.488 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:17.488 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.126871+0000", 2024-03-19T11:31:17.488 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012909273000000001 2024-03-19T11:31:17.488 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.488 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.488 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:17.489 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.126892+0000", 2024-03-19T11:31:17.489 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0678000000000001e-05 2024-03-19T11:31:17.489 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.489 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.489 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:17.489 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.130398+0000", 2024-03-19T11:31:17.489 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0035057259999999998 2024-03-19T11:31:17.489 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:17.489 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:17.489 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:17.489 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.490 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.490 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.16 80/79 [PushOp(4:6d9093b1:::smithi03829856-38:10, version: 70'83, data_included: [621859~568773,1729599~479803], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6d9093b1:::smithi03829856-38:10@70'83, size: 3142876, copy_subset: [(0, 3142876)], clone_subset: {}, snapset: 10=[]:{10: [d, b, 6]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2209402, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:17.490 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:59.114057+0000", 2024-03-19T11:31:17.490 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.319917847999999, 2024-03-19T11:31:17.490 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019935583, 2024-03-19T11:31:17.490 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:17.490 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:17.490 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:17.490 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.491 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:17.491 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.114057+0000", 2024-03-19T11:31:17.491 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:17.491 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.491 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.491 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:17.491 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.114057+0000", 2024-03-19T11:31:17.491 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:17.491 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.491 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.491 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:17.491 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.114058+0000", 2024-03-19T11:31:17.491 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4300000000000001e-07 2024-03-19T11:31:17.491 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.492 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.492 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:17.492 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.117243+0000", 2024-03-19T11:31:17.492 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031852550000000001 2024-03-19T11:31:17.492 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.492 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.492 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:17.492 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.117263+0000", 2024-03-19T11:31:17.493 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0469999999999999e-05 2024-03-19T11:31:17.493 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.493 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.493 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:17.493 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.117276+0000", 2024-03-19T11:31:17.493 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2775000000000001e-05 2024-03-19T11:31:17.493 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.494 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.494 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:17.494 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.130681+0000", 2024-03-19T11:31:17.494 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013404879999999999 2024-03-19T11:31:17.494 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.494 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.494 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:17.494 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.130699+0000", 2024-03-19T11:31:17.495 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8343e-05 2024-03-19T11:31:17.495 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.495 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.495 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:17.495 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.133992+0000", 2024-03-19T11:31:17.495 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0032929169999999998 2024-03-19T11:31:17.495 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:17.496 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:17.496 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:17.496 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.496 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.496 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2442 4.16 4:6d9093b1:::smithi03829856-38:head [write 460558~707131] snapc 17=[17,16,15,13,12,b,6] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-03-19T11:31:17.496 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:03.432382+0000", 2024-03-19T11:31:17.497 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 14.001592297, 2024-03-19T11:31:17.497 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019918379, 2024-03-19T11:31:17.497 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:17.497 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:17.497 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:17.497 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:17.497 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:17.498 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2442 2024-03-19T11:31:17.498 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.498 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:17.498 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.498 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:17.498 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.432382+0000", 2024-03-19T11:31:17.498 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:17.499 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.499 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.499 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:17.499 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.432382+0000", 2024-03-19T11:31:17.499 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:17.499 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.499 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.499 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:17.500 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.432385+0000", 2024-03-19T11:31:17.500 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1e-06 2024-03-19T11:31:17.500 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.500 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.501 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:17.501 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.433115+0000", 2024-03-19T11:31:17.501 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00072939900000000004 2024-03-19T11:31:17.501 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.502 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.502 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:17.502 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.433151+0000", 2024-03-19T11:31:17.502 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6094999999999999e-05 2024-03-19T11:31:17.502 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.502 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.505 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:17.505 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.433167+0000", 2024-03-19T11:31:17.505 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6152999999999999e-05 2024-03-19T11:31:17.505 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.506 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.506 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:17.506 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.433339+0000", 2024-03-19T11:31:17.506 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000171478 2024-03-19T11:31:17.506 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.506 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.507 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:17.507 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.433621+0000", 2024-03-19T11:31:17.507 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00028283900000000001 2024-03-19T11:31:17.507 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.507 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.507 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3", 2024-03-19T11:31:17.507 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.434169+0000", 2024-03-19T11:31:17.507 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00054786699999999995 2024-03-19T11:31:17.507 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.508 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.508 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:17.508 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.443136+0000", 2024-03-19T11:31:17.508 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0089667419999999998 2024-03-19T11:31:17.508 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.508 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.508 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:17.508 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.452230+0000", 2024-03-19T11:31:17.508 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.009093996 2024-03-19T11:31:17.509 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.509 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.509 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:17.509 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.452264+0000", 2024-03-19T11:31:17.509 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4381000000000002e-05 2024-03-19T11:31:17.509 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.509 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.509 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:17.509 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.452301+0000", 2024-03-19T11:31:17.509 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6328999999999997e-05 2024-03-19T11:31:17.509 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:17.509 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:17.509 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:17.510 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.510 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.510 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.4.13:82 3.3 3:f6b6d21a:::smithi03829856-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-03-19T11:31:17.510 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:04.069448+0000", 2024-03-19T11:31:17.510 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 13.364527022000001, 2024-03-19T11:31:17.510 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.086427773999999999, 2024-03-19T11:31:17.511 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:17.511 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:17.511 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:17.511 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.511 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:17.511 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069448+0000", 2024-03-19T11:31:17.511 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:17.511 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.511 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.511 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:17.511 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069448+0000", 2024-03-19T11:31:17.512 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:17.512 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.512 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.512 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:17.512 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069449+0000", 2024-03-19T11:31:17.512 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8160000000000001e-06 2024-03-19T11:31:17.512 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.512 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.512 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:17.512 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069458+0000", 2024-03-19T11:31:17.512 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2900000000000002e-06 2024-03-19T11:31:17.512 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.512 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.512 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:17.513 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069491+0000", 2024-03-19T11:31:17.513 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3229000000000003e-05 2024-03-19T11:31:17.513 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.513 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.513 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:17.513 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069501+0000", 2024-03-19T11:31:17.513 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7629999999999995e-06 2024-03-19T11:31:17.513 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.513 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.513 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:17.513 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069573+0000", 2024-03-19T11:31:17.513 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2349999999999997e-05 2024-03-19T11:31:17.513 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.513 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.513 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:17.514 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069581+0000", 2024-03-19T11:31:17.514 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.388e-06 2024-03-19T11:31:17.514 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.514 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.514 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:17.514 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.075451+0000", 2024-03-19T11:31:17.514 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005869956 2024-03-19T11:31:17.514 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.514 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.514 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:17.514 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.077511+0000", 2024-03-19T11:31:17.514 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0020592610000000002 2024-03-19T11:31:17.514 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.515 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.515 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:17.515 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.108781+0000", 2024-03-19T11:31:17.515 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.031270344999999998 2024-03-19T11:31:17.515 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.515 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.515 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:17.515 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.108895+0000", 2024-03-19T11:31:17.515 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000114059 2024-03-19T11:31:17.515 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.515 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.515 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:17.515 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.109065+0000", 2024-03-19T11:31:17.515 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016979899999999999 2024-03-19T11:31:17.515 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.516 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.516 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:17.516 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.111730+0000", 2024-03-19T11:31:17.516 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0026651410000000002 2024-03-19T11:31:17.516 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.516 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.516 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:17.516 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.155616+0000", 2024-03-19T11:31:17.516 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.043885986000000002 2024-03-19T11:31:17.516 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.516 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.516 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:17.516 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.155745+0000", 2024-03-19T11:31:17.516 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012948899999999999 2024-03-19T11:31:17.517 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.517 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.517 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:17.517 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.155875+0000", 2024-03-19T11:31:17.517 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012990199999999999 2024-03-19T11:31:17.517 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:17.517 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:17.517 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:17.517 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.517 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.517 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.4.13:83 3.3 3:f6b6d21a:::smithi03829856-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-03-19T11:31:17.517 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:04.069526+0000", 2024-03-19T11:31:17.517 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 13.364448242, 2024-03-19T11:31:17.518 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.087140880000000004, 2024-03-19T11:31:17.518 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:17.518 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:17.518 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:17.518 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.518 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:17.518 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069526+0000", 2024-03-19T11:31:17.518 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:17.518 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.518 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.518 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:17.518 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069526+0000", 2024-03-19T11:31:17.518 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:17.519 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.519 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.519 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:17.519 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069527+0000", 2024-03-19T11:31:17.519 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2799999999999996e-07 2024-03-19T11:31:17.519 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.519 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.519 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:17.519 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069531+0000", 2024-03-19T11:31:17.519 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1489999999999996e-06 2024-03-19T11:31:17.519 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.519 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.519 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:17.519 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069539+0000", 2024-03-19T11:31:17.520 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3189999999999995e-06 2024-03-19T11:31:17.520 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.520 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.520 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:17.520 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069545+0000", 2024-03-19T11:31:17.520 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.592e-06 2024-03-19T11:31:17.520 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.520 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.520 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:17.520 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069616+0000", 2024-03-19T11:31:17.520 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1130000000000005e-05 2024-03-19T11:31:17.520 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.520 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.520 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:17.520 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069628+0000", 2024-03-19T11:31:17.521 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1697e-05 2024-03-19T11:31:17.521 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.521 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.521 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:17.521 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.077594+0000", 2024-03-19T11:31:17.521 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0079656989999999997 2024-03-19T11:31:17.521 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.521 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.521 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:17.522 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.077674+0000", 2024-03-19T11:31:17.522 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0338000000000002e-05 2024-03-19T11:31:17.522 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.522 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.522 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:17.522 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.112771+0000", 2024-03-19T11:31:17.522 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.035097325999999998 2024-03-19T11:31:17.522 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.523 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.523 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:17.523 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.112882+0000", 2024-03-19T11:31:17.523 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000110949 2024-03-19T11:31:17.523 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.523 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.523 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:17.523 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.113031+0000", 2024-03-19T11:31:17.523 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000148355 2024-03-19T11:31:17.523 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.523 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.523 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:17.523 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.114076+0000", 2024-03-19T11:31:17.524 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001045009 2024-03-19T11:31:17.524 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.524 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.524 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:17.524 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156482+0000", 2024-03-19T11:31:17.524 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.042406354 2024-03-19T11:31:17.524 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.524 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.524 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:17.524 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156549+0000", 2024-03-19T11:31:17.524 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7229000000000003e-05 2024-03-19T11:31:17.524 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.524 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.525 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:17.525 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156667+0000", 2024-03-19T11:31:17.525 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000118106 2024-03-19T11:31:17.525 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:17.525 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:17.525 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:17.525 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.525 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.525 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.4.13:84 3.3 3:ce6951d3:::smithi03829856-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-03-19T11:31:17.525 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:04.071995+0000", 2024-03-19T11:31:17.525 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 13.361979466999999, 2024-03-19T11:31:17.525 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.084172497999999998, 2024-03-19T11:31:17.525 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:17.525 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:17.525 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:17.526 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.526 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:17.526 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.071995+0000", 2024-03-19T11:31:17.526 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:17.526 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.526 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.526 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:17.526 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.071995+0000", 2024-03-19T11:31:17.526 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:17.526 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.526 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.526 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:17.526 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.071997+0000", 2024-03-19T11:31:17.527 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6029999999999999e-06 2024-03-19T11:31:17.527 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.527 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.527 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:17.527 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072004+0000", 2024-03-19T11:31:17.527 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8600000000000004e-06 2024-03-19T11:31:17.527 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.527 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.527 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:17.527 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072033+0000", 2024-03-19T11:31:17.527 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9836999999999999e-05 2024-03-19T11:31:17.527 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.527 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.527 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:17.527 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072042+0000", 2024-03-19T11:31:17.528 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9279999999999999e-06 2024-03-19T11:31:17.528 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.528 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.528 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:17.528 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072123+0000", 2024-03-19T11:31:17.528 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0630000000000006e-05 2024-03-19T11:31:17.528 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.528 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.528 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:17.528 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072132+0000", 2024-03-19T11:31:17.528 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5560000000000008e-06 2024-03-19T11:31:17.528 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.528 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.528 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:17.529 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.077712+0000", 2024-03-19T11:31:17.529 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055804970000000002 2024-03-19T11:31:17.529 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.529 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.529 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:17.529 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.079495+0000", 2024-03-19T11:31:17.529 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017831850000000001 2024-03-19T11:31:17.529 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.529 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.529 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:17.529 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.109825+0000", 2024-03-19T11:31:17.529 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.030329819000000001 2024-03-19T11:31:17.529 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.529 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.530 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:17.530 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.109904+0000", 2024-03-19T11:31:17.530 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9269000000000003e-05 2024-03-19T11:31:17.530 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.530 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.530 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:17.530 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.110085+0000", 2024-03-19T11:31:17.530 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00018018600000000001 2024-03-19T11:31:17.530 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.530 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.530 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:17.530 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.113581+0000", 2024-03-19T11:31:17.530 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034962919999999998 2024-03-19T11:31:17.531 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.531 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.531 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:17.531 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.155925+0000", 2024-03-19T11:31:17.531 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.042344404000000002 2024-03-19T11:31:17.531 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.531 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.531 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:17.531 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156026+0000", 2024-03-19T11:31:17.531 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000100599 2024-03-19T11:31:17.531 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.531 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.531 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:17.531 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156168+0000", 2024-03-19T11:31:17.532 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014183300000000001 2024-03-19T11:31:17.532 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:17.532 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:17.532 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:17.532 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.532 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.532 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.4.13:85 3.3 3:ce6951d3:::smithi03829856-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-03-19T11:31:17.532 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:04.072068+0000", 2024-03-19T11:31:17.532 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 13.361906831000001, 2024-03-19T11:31:17.532 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.084368466000000003, 2024-03-19T11:31:17.532 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:17.533 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:17.533 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:17.533 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.533 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:17.533 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072068+0000", 2024-03-19T11:31:17.533 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:17.533 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.533 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.533 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:17.533 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072068+0000", 2024-03-19T11:31:17.533 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:17.533 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.533 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.533 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:17.533 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072068+0000", 2024-03-19T11:31:17.534 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9699999999999996e-07 2024-03-19T11:31:17.534 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.534 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.534 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:17.534 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072074+0000", 2024-03-19T11:31:17.534 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2279999999999998e-06 2024-03-19T11:31:17.534 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.534 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.534 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:17.534 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072140+0000", 2024-03-19T11:31:17.534 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6638999999999997e-05 2024-03-19T11:31:17.535 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.535 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.535 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:17.535 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072147+0000", 2024-03-19T11:31:17.535 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8750000000000002e-06 2024-03-19T11:31:17.535 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.535 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.535 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:17.535 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072201+0000", 2024-03-19T11:31:17.535 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3634000000000002e-05 2024-03-19T11:31:17.535 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.535 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.535 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:17.535 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072211+0000", 2024-03-19T11:31:17.536 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0589e-05 2024-03-19T11:31:17.536 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.536 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.536 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:17.536 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.079687+0000", 2024-03-19T11:31:17.536 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074757219999999997 2024-03-19T11:31:17.536 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.536 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.536 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:17.536 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.079769+0000", 2024-03-19T11:31:17.536 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2251999999999998e-05 2024-03-19T11:31:17.536 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.536 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.537 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:17.537 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.111798+0000", 2024-03-19T11:31:17.537 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.032028889999999997 2024-03-19T11:31:17.537 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.537 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.537 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:17.537 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.111902+0000", 2024-03-19T11:31:17.537 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000103773 2024-03-19T11:31:17.537 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.537 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.537 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:17.537 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.112073+0000", 2024-03-19T11:31:17.537 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017102799999999999 2024-03-19T11:31:17.537 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.538 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.538 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:17.538 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.113591+0000", 2024-03-19T11:31:17.538 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0015179569999999999 2024-03-19T11:31:17.538 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.538 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.538 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:17.538 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156223+0000", 2024-03-19T11:31:17.538 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.042632153999999998 2024-03-19T11:31:17.538 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.538 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.538 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:17.539 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156321+0000", 2024-03-19T11:31:17.539 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8246999999999996e-05 2024-03-19T11:31:17.539 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.539 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.539 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:17.539 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156436+0000", 2024-03-19T11:31:17.539 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000114881 2024-03-19T11:31:17.539 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:17.539 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:17.539 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:17.539 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.539 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.539 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2455 3.1 3:8e10ce27:::f25cf4cdffbae9b54315ae93e2cdc514e3bf58d758c8445671e9463006f381a6:head [write 0~1661440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-03-19T11:31:17.539 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:04.086189+0000", 2024-03-19T11:31:17.539 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 13.347785685, 2024-03-19T11:31:17.540 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015194925999999999, 2024-03-19T11:31:17.540 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:17.540 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:17.540 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:17.540 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:17.540 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:17.540 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2455 2024-03-19T11:31:17.540 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.540 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:17.540 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.540 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:17.541 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.086189+0000", 2024-03-19T11:31:17.541 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:17.541 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.541 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.541 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:17.541 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.086189+0000", 2024-03-19T11:31:17.541 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:17.541 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.541 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.541 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:17.541 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.086191+0000", 2024-03-19T11:31:17.541 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8619999999999999e-06 2024-03-19T11:31:17.541 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.541 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.542 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:17.542 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.086944+0000", 2024-03-19T11:31:17.542 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00075297299999999999 2024-03-19T11:31:17.542 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.542 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.542 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:17.542 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.086966+0000", 2024-03-19T11:31:17.542 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2404000000000001e-05 2024-03-19T11:31:17.542 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.542 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.542 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:17.542 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.086976+0000", 2024-03-19T11:31:17.542 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8069999999999994e-06 2024-03-19T11:31:17.543 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.543 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.543 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:17.543 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.087038+0000", 2024-03-19T11:31:17.543 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2314e-05 2024-03-19T11:31:17.543 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.543 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.543 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:17.543 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.087158+0000", 2024-03-19T11:31:17.543 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000119385 2024-03-19T11:31:17.543 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.543 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.543 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:17.543 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.087286+0000", 2024-03-19T11:31:17.544 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012851599999999999 2024-03-19T11:31:17.544 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.544 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.544 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:17.544 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.089954+0000", 2024-03-19T11:31:17.544 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0026674239999999998 2024-03-19T11:31:17.544 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.544 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.544 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:17.544 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.101235+0000", 2024-03-19T11:31:17.544 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011281807 2024-03-19T11:31:17.544 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.544 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.544 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:17.544 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.101321+0000", 2024-03-19T11:31:17.545 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5359999999999996e-05 2024-03-19T11:31:17.545 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.545 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.545 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:17.545 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.101384+0000", 2024-03-19T11:31:17.545 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3074e-05 2024-03-19T11:31:17.545 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:17.545 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:17.545 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:17.545 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.545 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.545 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2593 4.6 4:626acd4d:::smithi03829856-2:head [write 662242~642825] snapc 17=[17,16,15,b] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-03-19T11:31:17.545 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:07.785391+0000", 2024-03-19T11:31:17.545 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 9.6485834169999993, 2024-03-19T11:31:17.546 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018221268999999998, 2024-03-19T11:31:17.546 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:17.546 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:17.546 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:17.546 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:17.546 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:17.546 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2593 2024-03-19T11:31:17.546 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.546 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:17.546 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.546 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:17.546 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.785391+0000", 2024-03-19T11:31:17.546 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:17.547 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.547 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.547 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:17.547 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.785391+0000", 2024-03-19T11:31:17.547 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:17.547 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.547 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.547 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:17.547 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.785394+0000", 2024-03-19T11:31:17.547 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.903e-06 2024-03-19T11:31:17.547 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.547 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.547 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:17.547 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.786080+0000", 2024-03-19T11:31:17.547 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000686201 2024-03-19T11:31:17.548 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.548 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.548 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:17.548 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.786116+0000", 2024-03-19T11:31:17.548 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5441999999999997e-05 2024-03-19T11:31:17.548 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.548 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.548 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:17.548 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.786132+0000", 2024-03-19T11:31:17.548 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6353000000000001e-05 2024-03-19T11:31:17.548 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.548 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.548 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:17.548 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.786283+0000", 2024-03-19T11:31:17.549 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015133599999999999 2024-03-19T11:31:17.549 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.549 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.549 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:17.549 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.786457+0000", 2024-03-19T11:31:17.549 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017352200000000001 2024-03-19T11:31:17.549 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.550 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.550 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1", 2024-03-19T11:31:17.550 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.786765+0000", 2024-03-19T11:31:17.550 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030768199999999999 2024-03-19T11:31:17.550 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.550 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.551 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:17.551 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.798155+0000", 2024-03-19T11:31:17.551 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011390045999999999 2024-03-19T11:31:17.551 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.551 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.551 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:17.552 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.803291+0000", 2024-03-19T11:31:17.552 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0051363060000000002 2024-03-19T11:31:17.552 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.552 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.552 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:17.552 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.803466+0000", 2024-03-19T11:31:17.552 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017505900000000001 2024-03-19T11:31:17.553 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:17.553 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:17.553 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:17.553 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.803612+0000", 2024-03-19T11:31:17.554 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014641900000000001 2024-03-19T11:31:17.554 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:17.554 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:17.555 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:17.555 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:17.555 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:17.555 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:17.556 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-19T11:31:17.557 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:17.565+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:17.657 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:17.665+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:17.730 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:17.731 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:31:17.731 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:31:17.731 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:17.745 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-19T11:31:17.758 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:17.766+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:17.859 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:17.870+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:17.907 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:17.907 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:31:17.907 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:31:17.907 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:31:17.907 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:17.921 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-19T11:31:17.968 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:17.977+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:18.037 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:18.046+0000 7f143b8b4640 -1 osd.5 98 handle_osd_map injecting map crc failure 2024-03-19T11:31:18.041 INFO:tasks.rados.rados.0.smithi038.stdout:1354: done (8 left) 2024-03-19T11:31:18.041 INFO:tasks.rados.rados.0.smithi038.stdout:1355: rollback oid 16 current snap is 26 2024-03-19T11:31:18.041 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 16 to 9 2024-03-19T11:31:18.041 INFO:tasks.rados.rados.0.smithi038.stdout:1345: finishing write tid 3 to smithi03829856-40 2024-03-19T11:31:18.041 INFO:tasks.rados.rados.0.smithi038.stdout:1345: finishing write tid 4 to smithi03829856-40 2024-03-19T11:31:18.041 INFO:tasks.rados.rados.0.smithi038.stdout:1347: expect (ObjNum 72 snap 5 seq_num 72) 2024-03-19T11:31:18.043 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:18.049+0000 7f6a7f9da640 -1 osd.1 98 handle_osd_map injecting map crc failure 2024-03-19T11:31:18.061 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:18.070+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:18.063 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:18.063 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:31:18.063 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:31:18.063 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:31:18.063 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.063 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1063 4.4 4:214026b3:::smithi03829856-47:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:31:18.063 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.784904+0000", 2024-03-19T11:31:18.063 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 64.288671734999994, 2024-03-19T11:31:18.064 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024425912000000001, 2024-03-19T11:31:18.064 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:18.064 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:18.064 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:18.064 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:18.064 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:18.064 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1063 2024-03-19T11:31:18.064 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.064 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:18.064 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.064 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:18.064 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784904+0000", 2024-03-19T11:31:18.064 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.064 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.064 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.065 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:18.065 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784904+0000", 2024-03-19T11:31:18.065 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.065 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.065 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.065 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:18.065 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784906+0000", 2024-03-19T11:31:18.065 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1409999999999999e-06 2024-03-19T11:31:18.065 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.065 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.065 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:18.065 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784915+0000", 2024-03-19T11:31:18.065 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5459999999999992e-06 2024-03-19T11:31:18.066 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.066 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.066 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:18.066 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784935+0000", 2024-03-19T11:31:18.066 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0310999999999999e-05 2024-03-19T11:31:18.066 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.066 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.066 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:18.066 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784946+0000", 2024-03-19T11:31:18.066 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0675e-05 2024-03-19T11:31:18.066 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.066 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.066 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.067 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.785032+0000", 2024-03-19T11:31:18.067 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6049000000000002e-05 2024-03-19T11:31:18.067 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.067 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.067 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:18.067 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.785136+0000", 2024-03-19T11:31:18.067 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000104209 2024-03-19T11:31:18.067 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.067 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.067 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:18.067 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.809330+0000", 2024-03-19T11:31:18.067 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024193981 2024-03-19T11:31:18.067 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.067 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:18.067 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.068 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.068 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.068 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1433 4.16 4:6be89c06:::smithi03829856-24:c [sparse-read 0~1100427,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:31:18.068 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.976652+0000", 2024-03-19T11:31:18.068 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 52.096924033999997, 2024-03-19T11:31:18.068 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95684753199999995, 2024-03-19T11:31:18.068 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:18.068 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:18.068 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:18.068 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:18.068 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:18.068 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1433 2024-03-19T11:31:18.068 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.068 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:18.068 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.069 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:18.069 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976652+0000", 2024-03-19T11:31:18.069 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.069 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.069 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.069 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:18.069 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976652+0000", 2024-03-19T11:31:18.069 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.069 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.069 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.069 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:18.069 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976654+0000", 2024-03-19T11:31:18.069 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6199999999999999e-06 2024-03-19T11:31:18.069 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.070 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.070 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:18.070 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976668+0000", 2024-03-19T11:31:18.070 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3213999999999999e-05 2024-03-19T11:31:18.070 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.070 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.070 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:18.070 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976694+0000", 2024-03-19T11:31:18.070 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6488000000000001e-05 2024-03-19T11:31:18.070 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.071 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.071 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:18.071 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976705+0000", 2024-03-19T11:31:18.071 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1044e-05 2024-03-19T11:31:18.071 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.071 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.071 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.071 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976795+0000", 2024-03-19T11:31:18.071 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0283000000000003e-05 2024-03-19T11:31:18.071 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.071 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.071 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:18.071 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976811+0000", 2024-03-19T11:31:18.071 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5262000000000001e-05 2024-03-19T11:31:18.072 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.072 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.072 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.072 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.927337+0000", 2024-03-19T11:31:18.072 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95052597100000003 2024-03-19T11:31:18.072 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.072 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.073 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:18.073 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.927495+0000", 2024-03-19T11:31:18.073 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015788600000000001 2024-03-19T11:31:18.073 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.073 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.073 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:18.073 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.933499+0000", 2024-03-19T11:31:18.073 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0060047640000000001 2024-03-19T11:31:18.073 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.073 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:18.073 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.073 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.073 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.073 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1434 4.16 4:6be89c06:::smithi03829856-24:c [read 0~2386561,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:31:18.074 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.976739+0000", 2024-03-19T11:31:18.074 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 52.096836873000001, 2024-03-19T11:31:18.074 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95781317399999999, 2024-03-19T11:31:18.074 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:18.074 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:18.074 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:18.074 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:18.074 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:18.074 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1434 2024-03-19T11:31:18.074 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.074 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:18.074 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.074 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:18.074 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976739+0000", 2024-03-19T11:31:18.075 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.075 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.075 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.075 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:18.075 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976739+0000", 2024-03-19T11:31:18.075 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.075 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.075 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.075 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:18.075 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976740+0000", 2024-03-19T11:31:18.075 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8800000000000001e-07 2024-03-19T11:31:18.075 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.075 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.075 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:18.076 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976748+0000", 2024-03-19T11:31:18.076 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8979999999999996e-06 2024-03-19T11:31:18.076 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.076 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.076 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:18.076 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976757+0000", 2024-03-19T11:31:18.076 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9439999999999994e-06 2024-03-19T11:31:18.076 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.076 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.076 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:18.076 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976763+0000", 2024-03-19T11:31:18.076 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5030000000000002e-06 2024-03-19T11:31:18.076 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.076 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.076 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.077 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976849+0000", 2024-03-19T11:31:18.077 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5803999999999994e-05 2024-03-19T11:31:18.077 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.077 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.077 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:18.077 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976860+0000", 2024-03-19T11:31:18.077 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0869e-05 2024-03-19T11:31:18.077 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.077 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.077 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.077 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.933562+0000", 2024-03-19T11:31:18.077 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95670232300000002 2024-03-19T11:31:18.077 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.077 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.078 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:18.078 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.933693+0000", 2024-03-19T11:31:18.078 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013110699999999999 2024-03-19T11:31:18.078 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.078 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.078 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:18.078 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.934552+0000", 2024-03-19T11:31:18.078 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00085883800000000001 2024-03-19T11:31:18.078 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.078 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:18.078 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.078 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.078 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.079 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1435 4.16 4:6be89c06:::smithi03829856-24:c [sparse-read 0~1100427] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:31:18.079 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.976775+0000", 2024-03-19T11:31:18.079 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 52.096801202000002, 2024-03-19T11:31:18.079 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95962779499999995, 2024-03-19T11:31:18.079 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:18.079 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:18.079 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:18.079 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:18.079 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:18.079 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1435 2024-03-19T11:31:18.079 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.079 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:18.079 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.079 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:18.080 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976775+0000", 2024-03-19T11:31:18.080 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.080 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.080 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.080 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:18.080 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976775+0000", 2024-03-19T11:31:18.080 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.080 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.080 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.080 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:18.080 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976775+0000", 2024-03-19T11:31:18.080 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9800000000000003e-07 2024-03-19T11:31:18.080 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.080 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.080 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:18.081 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976779+0000", 2024-03-19T11:31:18.081 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3830000000000001e-06 2024-03-19T11:31:18.081 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.081 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.081 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:18.081 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976785+0000", 2024-03-19T11:31:18.081 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5649999999999998e-06 2024-03-19T11:31:18.081 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.081 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.081 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:18.081 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976790+0000", 2024-03-19T11:31:18.081 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1379999999999998e-06 2024-03-19T11:31:18.081 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.081 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.082 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.082 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976889+0000", 2024-03-19T11:31:18.082 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8331000000000004e-05 2024-03-19T11:31:18.082 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.082 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.082 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:18.082 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976894+0000", 2024-03-19T11:31:18.082 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8270000000000003e-06 2024-03-19T11:31:18.082 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.082 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.082 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.082 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.935982+0000", 2024-03-19T11:31:18.082 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95908714100000003 2024-03-19T11:31:18.082 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.083 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.083 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:18.083 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.936093+0000", 2024-03-19T11:31:18.083 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000111557 2024-03-19T11:31:18.083 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.083 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.083 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:18.083 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.936402+0000", 2024-03-19T11:31:18.083 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030925499999999998 2024-03-19T11:31:18.083 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.083 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:18.083 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.083 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.083 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.083 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1442 4.6 4:626acd4d:::smithi03829856-2:head [cache-evict] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:31:18.083 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.978864+0000", 2024-03-19T11:31:18.084 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 52.094711537999999, 2024-03-19T11:31:18.084 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95708845099999995, 2024-03-19T11:31:18.084 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:18.084 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:18.084 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:18.084 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:18.084 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:18.084 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1442 2024-03-19T11:31:18.084 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.084 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:18.084 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.084 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:18.084 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978864+0000", 2024-03-19T11:31:18.084 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.085 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.085 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.085 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:18.085 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978864+0000", 2024-03-19T11:31:18.085 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.085 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.085 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.085 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:18.085 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978866+0000", 2024-03-19T11:31:18.085 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0729999999999999e-06 2024-03-19T11:31:18.085 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.085 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.085 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:18.085 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978877+0000", 2024-03-19T11:31:18.086 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0390999999999999e-05 2024-03-19T11:31:18.086 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.086 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.086 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:18.086 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978902+0000", 2024-03-19T11:31:18.086 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5635000000000001e-05 2024-03-19T11:31:18.086 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.086 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.086 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:18.086 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978915+0000", 2024-03-19T11:31:18.086 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2617e-05 2024-03-19T11:31:18.086 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.086 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.086 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.087 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.979005+0000", 2024-03-19T11:31:18.087 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0552999999999999e-05 2024-03-19T11:31:18.087 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.087 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.087 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:18.087 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.979021+0000", 2024-03-19T11:31:18.087 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5102999999999999e-05 2024-03-19T11:31:18.087 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.087 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.087 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.087 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.926879+0000", 2024-03-19T11:31:18.087 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.94785817900000002 2024-03-19T11:31:18.087 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.087 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.088 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:18.088 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.927277+0000", 2024-03-19T11:31:18.088 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00039819600000000002 2024-03-19T11:31:18.088 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.088 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.088 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.088 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.935752+0000", 2024-03-19T11:31:18.088 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0084753420000000003 2024-03-19T11:31:18.088 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.088 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.088 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:18.088 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.935874+0000", 2024-03-19T11:31:18.089 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000121311 2024-03-19T11:31:18.089 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.089 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.089 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:18.089 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.935953+0000", 2024-03-19T11:31:18.089 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9050999999999999e-05 2024-03-19T11:31:18.089 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.089 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:18.089 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.089 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.089 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.089 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.d 49/48 [PushOp(4:b65e6658:::smithi03829856-9:head, version: 28'17, data_included: [534456~718345,1951503~182953], data_size: 901298, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:b65e6658:::smithi03829856-9:head@28'17, size: 2134456, copy_subset: [(0, 2134456)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2134456, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:b4ef97e0:::smithi03829856-3:b, version: 46'34, data_included: [433614~1004682,1600611~43894], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:b4ef97e0:::smithi03829856-3:b@46'34, size: 3042034, copy_subset: [(0, 3042034)], clone_subset: {}, snapset: b=[]:{b: [b, a, 6, 5]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1644505, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:b4ef97e0:::smithi03829856-3:head, version: 47'43, data_included: [433614~1004682,1600611~43894], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:b4ef97e0:::smithi03829856-3:head@47'43, size: 3697507, copy_subset: [(0, 3697507)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1644505, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:18.089 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:26.943544+0000", 2024-03-19T11:31:18.090 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 51.130032047999997, 2024-03-19T11:31:18.090 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015601487000000001, 2024-03-19T11:31:18.090 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:18.090 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:18.090 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:18.090 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.090 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:18.090 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.943544+0000", 2024-03-19T11:31:18.090 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.090 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.090 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.090 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:18.090 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.943544+0000", 2024-03-19T11:31:18.090 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.091 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.091 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.091 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:18.091 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.943546+0000", 2024-03-19T11:31:18.091 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.013e-06 2024-03-19T11:31:18.091 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.091 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.091 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:18.091 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.946208+0000", 2024-03-19T11:31:18.091 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0026622820000000002 2024-03-19T11:31:18.091 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.091 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.091 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:18.091 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.946240+0000", 2024-03-19T11:31:18.091 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1568e-05 2024-03-19T11:31:18.092 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.092 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.092 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:18.092 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.946282+0000", 2024-03-19T11:31:18.092 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2599999999999999e-05 2024-03-19T11:31:18.092 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.092 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.092 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.092 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.946442+0000", 2024-03-19T11:31:18.092 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015960400000000001 2024-03-19T11:31:18.092 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.092 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.092 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:18.092 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.946458+0000", 2024-03-19T11:31:18.092 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6059999999999999e-05 2024-03-19T11:31:18.093 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.093 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.093 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:18.093 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.959145+0000", 2024-03-19T11:31:18.093 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01268736 2024-03-19T11:31:18.093 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.093 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:18.093 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.093 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.093 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.093 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2003 4.16 4:6868af91:::smithi03829856-10:head [tier-promote] snapc 12=[12,11,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:31:18.093 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:45.804046+0000", 2024-03-19T11:31:18.093 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 32.269529624999997, 2024-03-19T11:31:18.093 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018976396, 2024-03-19T11:31:18.094 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:18.094 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:18.094 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:18.094 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:18.094 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:18.094 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2003 2024-03-19T11:31:18.094 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.094 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:18.094 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.094 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:18.094 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804046+0000", 2024-03-19T11:31:18.094 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.094 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.094 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.094 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:18.095 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804046+0000", 2024-03-19T11:31:18.095 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.095 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.095 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.095 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:18.095 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804052+0000", 2024-03-19T11:31:18.095 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7599999999999999e-06 2024-03-19T11:31:18.095 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.095 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.095 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:18.095 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804090+0000", 2024-03-19T11:31:18.095 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8018000000000001e-05 2024-03-19T11:31:18.095 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.095 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.095 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:18.096 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804256+0000", 2024-03-19T11:31:18.096 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016593599999999999 2024-03-19T11:31:18.096 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.096 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.096 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:18.096 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804300+0000", 2024-03-19T11:31:18.096 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3648e-05 2024-03-19T11:31:18.096 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.096 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.096 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.096 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804538+0000", 2024-03-19T11:31:18.096 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000238347 2024-03-19T11:31:18.096 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.096 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.096 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:18.096 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804716+0000", 2024-03-19T11:31:18.097 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000177968 2024-03-19T11:31:18.097 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.097 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.097 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:18.097 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.823023+0000", 2024-03-19T11:31:18.097 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018306718999999999 2024-03-19T11:31:18.097 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.097 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:18.097 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.097 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.097 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.097 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1d58db11:::smithi03829856-18:6, version: 55'80, data_included: [759683~639358,1904434~409218], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:6@55'80, size: 3159683, copy_subset: [(0, 3159683)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2313652, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1d58db11:::smithi03829856-18:b, version: 58'84, data_included: [443718~955323,1564138~93253], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:b@58'84, size: 3445402, copy_subset: [(0, 3445402)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1657391, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:10, version: 66'95, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:10@66'95, size: 3764735, copy_subset: [(0, 3764735)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:12, version: 66'97, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:12@66'97, size: 2504243, copy_subset: [(0, 2504243)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:c, version: 70'102, data_included: [416807~576167,1542129~472409], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:c@70'102, size: 2857157, copy_subset: [(0, 2857157)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2014538, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:18.097 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.275179+0000", 2024-03-19T11:31:18.098 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 23.798396635, 2024-03-19T11:31:18.098 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026559353000000001, 2024-03-19T11:31:18.098 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:18.098 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:18.098 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:18.098 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.098 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:18.098 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.275179+0000", 2024-03-19T11:31:18.098 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.098 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.098 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.098 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:18.098 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.275179+0000", 2024-03-19T11:31:18.099 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.099 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.099 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.099 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:18.099 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.275196+0000", 2024-03-19T11:31:18.099 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6946999999999999e-05 2024-03-19T11:31:18.099 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.099 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.099 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:18.099 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281456+0000", 2024-03-19T11:31:18.099 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0062600529999999998 2024-03-19T11:31:18.099 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.099 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.100 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:18.100 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281524+0000", 2024-03-19T11:31:18.100 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7971000000000004e-05 2024-03-19T11:31:18.100 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.100 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.100 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:18.100 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281591+0000", 2024-03-19T11:31:18.100 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6704000000000004e-05 2024-03-19T11:31:18.100 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.100 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.100 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.100 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281865+0000", 2024-03-19T11:31:18.100 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000274097 2024-03-19T11:31:18.100 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.100 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.101 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:18.101 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281901+0000", 2024-03-19T11:31:18.101 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5777e-05 2024-03-19T11:31:18.101 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.101 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.101 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:18.101 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.301738+0000", 2024-03-19T11:31:18.101 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019837804000000001 2024-03-19T11:31:18.101 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.101 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:18.101 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.101 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.101 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.101 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.18 e75,74 4:1d58db11:::smithi03829856-18:2,48'76)", 2024-03-19T11:31:18.101 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.281661+0000", 2024-03-19T11:31:18.102 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 23.791915104000001, 2024-03-19T11:31:18.102 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020384337999999998, 2024-03-19T11:31:18.102 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:18.102 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:18.102 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:18.102 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.102 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:18.102 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281661+0000", 2024-03-19T11:31:18.102 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.102 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.102 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.102 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:18.102 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281661+0000", 2024-03-19T11:31:18.102 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.102 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.103 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.103 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:18.103 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281661+0000", 2024-03-19T11:31:18.103 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9100000000000003e-07 2024-03-19T11:31:18.103 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.103 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.103 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:18.103 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281668+0000", 2024-03-19T11:31:18.103 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8199999999999999e-06 2024-03-19T11:31:18.103 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.103 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.103 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:18.103 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281679+0000", 2024-03-19T11:31:18.103 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1107999999999999e-05 2024-03-19T11:31:18.103 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.104 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.104 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:18.104 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281691+0000", 2024-03-19T11:31:18.104 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1399999999999999e-05 2024-03-19T11:31:18.104 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.104 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.104 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.104 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.301804+0000", 2024-03-19T11:31:18.104 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020113669000000001 2024-03-19T11:31:18.104 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.104 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.104 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:18.104 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.301818+0000", 2024-03-19T11:31:18.104 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3718e-05 2024-03-19T11:31:18.104 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.105 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.105 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:18.105 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302045+0000", 2024-03-19T11:31:18.105 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000226832 2024-03-19T11:31:18.105 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.105 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:18.105 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.105 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.105 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.105 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1dabc0f1:::smithi03829856-1:head, version: 70'103, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:head@70'103, size: 2621892, copy_subset: [(0, 2621892)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:18.105 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.281783+0000", 2024-03-19T11:31:18.105 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 23.791792827999998, 2024-03-19T11:31:18.105 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026187221, 2024-03-19T11:31:18.106 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:18.106 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:18.106 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:18.106 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.106 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:18.106 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281783+0000", 2024-03-19T11:31:18.106 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.106 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.106 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.106 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:18.106 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281783+0000", 2024-03-19T11:31:18.106 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.106 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.106 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.107 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:18.107 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281783+0000", 2024-03-19T11:31:18.107 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1699999999999999e-07 2024-03-19T11:31:18.107 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.107 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.107 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:18.107 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.282962+0000", 2024-03-19T11:31:18.107 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001178601 2024-03-19T11:31:18.107 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.107 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.107 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:18.107 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.282987+0000", 2024-03-19T11:31:18.107 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4899e-05 2024-03-19T11:31:18.107 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.107 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.108 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:18.108 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283006+0000", 2024-03-19T11:31:18.108 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9494999999999999e-05 2024-03-19T11:31:18.108 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.108 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.108 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.108 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.305757+0000", 2024-03-19T11:31:18.108 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.022750570000000001 2024-03-19T11:31:18.108 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.108 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.108 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:18.108 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.305770+0000", 2024-03-19T11:31:18.108 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2791e-05 2024-03-19T11:31:18.108 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.108 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.109 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:18.109 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.307970+0000", 2024-03-19T11:31:18.109 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022005480000000001 2024-03-19T11:31:18.109 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.109 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:18.109 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.109 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.109 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.109 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.18 e75,74 4:1d58db11:::smithi03829856-18:c,70'104)", 2024-03-19T11:31:18.109 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.283036+0000", 2024-03-19T11:31:18.109 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 23.790539727999999, 2024-03-19T11:31:18.109 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019146032, 2024-03-19T11:31:18.109 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:18.109 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:18.109 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:18.110 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.110 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:18.110 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283036+0000", 2024-03-19T11:31:18.110 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.110 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.110 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.110 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:18.110 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283036+0000", 2024-03-19T11:31:18.110 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.110 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.110 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.110 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:18.110 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283037+0000", 2024-03-19T11:31:18.110 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4899999999999995e-07 2024-03-19T11:31:18.110 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.111 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.111 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:18.111 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283042+0000", 2024-03-19T11:31:18.111 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0930000000000002e-06 2024-03-19T11:31:18.111 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.111 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.111 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:18.111 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283053+0000", 2024-03-19T11:31:18.111 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1083999999999999e-05 2024-03-19T11:31:18.111 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.111 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.111 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:18.111 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283059+0000", 2024-03-19T11:31:18.111 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6329999999999999e-06 2024-03-19T11:31:18.111 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.112 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.112 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.112 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302090+0000", 2024-03-19T11:31:18.112 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01903038 2024-03-19T11:31:18.112 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.112 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.112 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:18.112 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302097+0000", 2024-03-19T11:31:18.112 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.221e-06 2024-03-19T11:31:18.112 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.112 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.112 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:18.112 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302182+0000", 2024-03-19T11:31:18.112 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5072000000000001e-05 2024-03-19T11:31:18.113 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.113 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:18.113 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.113 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.113 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.113 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(osd.5.0:143 4.18 e75/74 4:1d58db11:::smithi03829856-18:head v 75'106, mlcod=0'0)", 2024-03-19T11:31:18.113 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.283083+0000", 2024-03-19T11:31:18.113 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 23.790492401000002, 2024-03-19T11:31:18.113 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025309439, 2024-03-19T11:31:18.113 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:18.113 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:18.113 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:18.113 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.113 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:18.113 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283083+0000", 2024-03-19T11:31:18.114 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.114 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.114 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.114 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:18.114 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283083+0000", 2024-03-19T11:31:18.114 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.114 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.114 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.114 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:18.114 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283084+0000", 2024-03-19T11:31:18.114 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7800000000000002e-07 2024-03-19T11:31:18.114 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.114 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.114 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:18.115 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.284082+0000", 2024-03-19T11:31:18.115 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0009977880000000001 2024-03-19T11:31:18.115 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.115 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.115 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:18.115 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.284095+0000", 2024-03-19T11:31:18.115 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3256e-05 2024-03-19T11:31:18.115 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.115 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.115 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:18.115 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.284102+0000", 2024-03-19T11:31:18.115 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.605e-06 2024-03-19T11:31:18.115 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.115 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.115 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.116 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302285+0000", 2024-03-19T11:31:18.116 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018182423 2024-03-19T11:31:18.116 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.116 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.116 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:18.116 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302341+0000", 2024-03-19T11:31:18.116 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.626e-05 2024-03-19T11:31:18.116 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.116 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.116 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:18.116 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.308338+0000", 2024-03-19T11:31:18.116 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0059968030000000002 2024-03-19T11:31:18.116 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.116 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.116 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:18.117 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.308393+0000", 2024-03-19T11:31:18.117 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4826e-05 2024-03-19T11:31:18.117 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.117 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:18.117 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.117 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.117 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.117 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "pg_update_log_missing(4.18 epoch 75/74 rep_tid 144 entries 75'107 (0'0) error 4:1d58db11:::smithi03829856-18:head by client.4415.0:2283 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-19T11:31:18.117 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.288252+0000", 2024-03-19T11:31:18.117 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 23.785323851000001, 2024-03-19T11:31:18.117 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020167377, 2024-03-19T11:31:18.117 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:18.117 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-03-19T11:31:18.118 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:18.118 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.118 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:18.118 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288252+0000", 2024-03-19T11:31:18.118 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.118 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.118 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.118 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:18.118 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288252+0000", 2024-03-19T11:31:18.118 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.118 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.118 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.118 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:18.118 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288253+0000", 2024-03-19T11:31:18.119 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2109999999999999e-06 2024-03-19T11:31:18.119 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.119 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.119 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:18.119 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288268+0000", 2024-03-19T11:31:18.119 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5003e-05 2024-03-19T11:31:18.119 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.119 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.119 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:18.119 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288301+0000", 2024-03-19T11:31:18.119 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2539999999999997e-05 2024-03-19T11:31:18.119 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.119 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.120 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:18.120 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288319+0000", 2024-03-19T11:31:18.120 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8232000000000001e-05 2024-03-19T11:31:18.120 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.120 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.120 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.120 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.305490+0000", 2024-03-19T11:31:18.120 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017170857000000001 2024-03-19T11:31:18.120 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.120 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.120 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:18.120 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.308419+0000", 2024-03-19T11:31:18.120 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029295340000000001 2024-03-19T11:31:18.121 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.121 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:18.121 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.121 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.121 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.121 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1d58db11:::smithi03829856-18:6, version: 55'80, data_included: [2313652~310127,3128657~31026], data_size: 341153, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:6@55'80, size: 3159683, copy_subset: [(0, 3159683)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3159683, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2313652, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:1d58db11:::smithi03829856-18:b, version: 58'84, data_included: [1657391~966388,2633066~82188], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:b@58'84, size: 3445402, copy_subset: [(0, 3445402)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2715254, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1657391, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:10, version: 66'95, data_included: [1829043~668012,3134120~380564], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:10@66'95, size: 3764735, copy_subset: [(0, 3764735)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3514684, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:12, version: 66'97, data_included: [1829043~507467], data_size: 507467, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:12@66'97, size: 2504243, copy_subset: [(0, 2504243)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2504243, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:c, version: 70'102, data_included: [2014538~156250], data_size: 156250, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:c@70'102, size: 2857157, copy_subset: [(0, 2857157)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2857157, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2014538, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:18.121 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.312362+0000", 2024-03-19T11:31:18.121 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 23.761213802, 2024-03-19T11:31:18.121 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017968951, 2024-03-19T11:31:18.121 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:18.121 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:18.121 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:18.121 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.122 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:18.122 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.312362+0000", 2024-03-19T11:31:18.122 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.122 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:18.122 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.312362+0000", 2024-03-19T11:31:18.122 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.122 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:18.122 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.312363+0000", 2024-03-19T11:31:18.122 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.528e-06 2024-03-19T11:31:18.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.123 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:18.123 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315748+0000", 2024-03-19T11:31:18.123 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033842830000000001 2024-03-19T11:31:18.123 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.123 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.123 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:18.123 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315795+0000", 2024-03-19T11:31:18.123 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7382999999999997e-05 2024-03-19T11:31:18.123 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.123 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.123 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:18.123 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315845+0000", 2024-03-19T11:31:18.123 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0319999999999999e-05 2024-03-19T11:31:18.123 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.124 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.124 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.124 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.316123+0000", 2024-03-19T11:31:18.124 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00027747200000000002 2024-03-19T11:31:18.124 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.124 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.124 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:18.124 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.316170+0000", 2024-03-19T11:31:18.124 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7466999999999998e-05 2024-03-19T11:31:18.124 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.124 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.124 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:18.124 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.330331+0000", 2024-03-19T11:31:18.124 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014160498000000001 2024-03-19T11:31:18.125 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.125 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:18.125 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.125 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.125 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.125 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1dabc0f1:::smithi03829856-1:head, version: 70'103, data_included: [1829043~555153], data_size: 555153, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:head@70'103, size: 2621892, copy_subset: [(0, 2621892)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2621892, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:18.125 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.315898+0000", 2024-03-19T11:31:18.125 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 23.757677816000001, 2024-03-19T11:31:18.125 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016880424000000002, 2024-03-19T11:31:18.125 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:18.125 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:18.125 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:18.125 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.126 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:18.126 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315898+0000", 2024-03-19T11:31:18.126 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.126 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.126 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.126 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:18.126 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315898+0000", 2024-03-19T11:31:18.126 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.126 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.126 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.126 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:18.126 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315899+0000", 2024-03-19T11:31:18.126 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.003e-06 2024-03-19T11:31:18.126 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.127 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.127 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:18.127 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.316486+0000", 2024-03-19T11:31:18.127 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00058673099999999995 2024-03-19T11:31:18.127 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.127 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.127 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:18.127 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.316519+0000", 2024-03-19T11:31:18.127 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3124000000000003e-05 2024-03-19T11:31:18.127 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.127 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.127 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:18.127 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.316534+0000", 2024-03-19T11:31:18.127 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.485e-05 2024-03-19T11:31:18.127 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.128 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.128 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.128 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.330415+0000", 2024-03-19T11:31:18.128 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013880905000000001 2024-03-19T11:31:18.128 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.128 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.128 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:18.128 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.330439+0000", 2024-03-19T11:31:18.128 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4630999999999998e-05 2024-03-19T11:31:18.128 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.128 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.128 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:18.128 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.332778+0000", 2024-03-19T11:31:18.128 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023391800000000002 2024-03-19T11:31:18.129 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.129 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:18.129 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.129 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.129 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.129 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2387 4.2 4:43b32242:::smithi03829856-22:head [read 596992~684544] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-03-19T11:31:18.129 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:59.319806+0000", 2024-03-19T11:31:18.129 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.753770149000001, 2024-03-19T11:31:18.129 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.78144344399999999, 2024-03-19T11:31:18.129 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:18.129 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:18.129 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:18.129 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:18.129 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:18.129 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2387 2024-03-19T11:31:18.130 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.130 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:18.130 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.130 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:18.130 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319806+0000", 2024-03-19T11:31:18.130 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.130 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.130 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.130 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:18.130 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319806+0000", 2024-03-19T11:31:18.130 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.131 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.131 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:18.131 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319807+0000", 2024-03-19T11:31:18.131 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3519999999999999e-06 2024-03-19T11:31:18.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.131 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.131 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:18.131 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319812+0000", 2024-03-19T11:31:18.131 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3959999999999997e-06 2024-03-19T11:31:18.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.131 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.131 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:18.131 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319827+0000", 2024-03-19T11:31:18.131 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4566000000000001e-05 2024-03-19T11:31:18.132 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.132 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.132 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:18.132 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319835+0000", 2024-03-19T11:31:18.132 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.994e-06 2024-03-19T11:31:18.132 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.132 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.132 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.132 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.320119+0000", 2024-03-19T11:31:18.132 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00028445199999999998 2024-03-19T11:31:18.132 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.132 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.132 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:18.132 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.320129+0000", 2024-03-19T11:31:18.132 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9580000000000002e-06 2024-03-19T11:31:18.133 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.133 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.133 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.133 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.069842+0000", 2024-03-19T11:31:18.133 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.74971218399999995 2024-03-19T11:31:18.133 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.133 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.133 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:31:18.133 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.070010+0000", 2024-03-19T11:31:18.133 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016846600000000001 2024-03-19T11:31:18.133 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.133 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.133 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.133 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.090911+0000", 2024-03-19T11:31:18.134 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020901494999999999 2024-03-19T11:31:18.134 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.134 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.134 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:31:18.134 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.091249+0000", 2024-03-19T11:31:18.134 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000337602 2024-03-19T11:31:18.134 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.134 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.134 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.134 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.098478+0000", 2024-03-19T11:31:18.134 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0072287549999999999 2024-03-19T11:31:18.134 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.134 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.134 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:18.135 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.098617+0000", 2024-03-19T11:31:18.135 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013871599999999999 2024-03-19T11:31:18.135 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.135 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.135 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:18.135 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.101249+0000", 2024-03-19T11:31:18.135 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002632508 2024-03-19T11:31:18.135 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.135 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:18.135 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.135 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.135 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.135 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 81/80 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 15=[]:{7: [6], b: [b], 10: [d], 12: [12, 11], 15: [15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:7, version: 55'82, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@55'82, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 16=[]:{7: [6], b: [b], 10: [d], 16: [16, 15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 15=[]:{7: [6], b: [b], 10: [d], 12: [12, 11], 15: [15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 58'86, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@58'86, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 16=[]:{7: [6], b: [b], 10: [d], 16: [16, 15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [504210~968657,1572695~79919], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 16=[]:{7: [6], b: [b], 10: [d], 16: [16, 15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:18.136 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:00.089924+0000", 2024-03-19T11:31:18.136 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 17.98365192, 2024-03-19T11:31:18.136 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024686583000000002, 2024-03-19T11:31:18.136 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:18.136 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:18.136 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:18.136 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.136 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:18.136 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.089924+0000", 2024-03-19T11:31:18.136 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.136 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.136 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.136 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:18.136 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.089924+0000", 2024-03-19T11:31:18.136 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.137 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.137 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.137 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:18.137 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.089926+0000", 2024-03-19T11:31:18.137 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9589999999999998e-06 2024-03-19T11:31:18.137 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.137 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.137 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:18.137 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093311+0000", 2024-03-19T11:31:18.137 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033854409999999999 2024-03-19T11:31:18.137 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.137 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.137 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:18.137 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093365+0000", 2024-03-19T11:31:18.137 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3795000000000002e-05 2024-03-19T11:31:18.137 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.138 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.138 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:18.138 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093438+0000", 2024-03-19T11:31:18.138 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2591999999999999e-05 2024-03-19T11:31:18.138 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.138 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.138 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.138 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093690+0000", 2024-03-19T11:31:18.138 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025264999999999999 2024-03-19T11:31:18.138 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.138 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.138 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:18.138 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093731+0000", 2024-03-19T11:31:18.138 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0881999999999999e-05 2024-03-19T11:31:18.139 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.139 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.139 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:18.139 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.114610+0000", 2024-03-19T11:31:18.139 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020879264000000002 2024-03-19T11:31:18.139 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.139 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:18.139 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.139 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.139 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.139 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.21 e81,80 4:8401c1f0:::smithi03829856-43:8,46'66,4:866f3dcc:::smithi03829856-31:2,47'73)", 2024-03-19T11:31:18.139 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:00.093514+0000", 2024-03-19T11:31:18.139 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 17.980061320000001, 2024-03-19T11:31:18.139 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027375713999999999, 2024-03-19T11:31:18.139 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:18.140 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:18.140 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:18.140 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.140 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:18.140 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093514+0000", 2024-03-19T11:31:18.140 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.140 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.140 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.140 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:18.140 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093514+0000", 2024-03-19T11:31:18.140 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.140 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.140 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.140 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:18.140 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093516+0000", 2024-03-19T11:31:18.141 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1179999999999999e-06 2024-03-19T11:31:18.141 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.141 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.141 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:18.141 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093523+0000", 2024-03-19T11:31:18.141 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7500000000000003e-06 2024-03-19T11:31:18.141 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.141 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.141 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:18.141 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093534+0000", 2024-03-19T11:31:18.141 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0831e-05 2024-03-19T11:31:18.141 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.141 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.141 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:18.141 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093542+0000", 2024-03-19T11:31:18.142 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0139999999999998e-06 2024-03-19T11:31:18.142 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.142 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.142 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.142 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.120682+0000", 2024-03-19T11:31:18.142 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027139838999999999 2024-03-19T11:31:18.142 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.142 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.142 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:18.142 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.120693+0000", 2024-03-19T11:31:18.142 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.136e-05 2024-03-19T11:31:18.142 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.142 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.143 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:18.143 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.120890+0000", 2024-03-19T11:31:18.143 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019680199999999999 2024-03-19T11:31:18.143 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.143 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:18.143 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.144 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.144 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.144 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 81/80 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 15=[]:{7: [6], b: [b], 10: [d], 12: [12, 11], 15: [15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:7, version: 55'82, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@55'82, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 16=[]:{7: [6], b: [b], 10: [d], 16: [16, 15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 15=[]:{7: [6], b: [b], 10: [d], 12: [12, 11], 15: [15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 58'86, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@58'86, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 16=[]:{7: [6], b: [b], 10: [d], 16: [16, 15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [1652614~973634,2888511~52681,3182305~22261], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 16=[]:{7: [6], b: [b], 10: [d], 16: [16, 15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3204566, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:18.144 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:00.126855+0000", 2024-03-19T11:31:18.144 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 17.946720583000001, 2024-03-19T11:31:18.144 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016438076999999999, 2024-03-19T11:31:18.144 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:18.144 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:18.144 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:18.144 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.144 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:18.144 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.126855+0000", 2024-03-19T11:31:18.144 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.144 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.144 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.145 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:18.145 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.126855+0000", 2024-03-19T11:31:18.145 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.145 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.145 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.145 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:18.145 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.126857+0000", 2024-03-19T11:31:18.145 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.739e-06 2024-03-19T11:31:18.145 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.145 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.145 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:18.145 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.129236+0000", 2024-03-19T11:31:18.145 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023792449999999999 2024-03-19T11:31:18.145 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.145 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.146 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:18.146 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.129298+0000", 2024-03-19T11:31:18.146 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2329000000000006e-05 2024-03-19T11:31:18.146 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.146 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.146 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:18.146 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.129365+0000", 2024-03-19T11:31:18.146 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6299999999999999e-05 2024-03-19T11:31:18.146 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-03-19T11:31:18.148 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.148 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.148 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.148 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.132493+0000", 2024-03-19T11:31:18.148 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031279530000000002 2024-03-19T11:31:18.148 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.148 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.148 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:18.148 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.132578+0000", 2024-03-19T11:31:18.148 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5648000000000004e-05 2024-03-19T11:31:18.148 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.148 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.149 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:18.149 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.143293+0000", 2024-03-19T11:31:18.149 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010714863 2024-03-19T11:31:18.149 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.149 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:18.149 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.149 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.149 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.149 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.29 81/80 [PushOp(4:953e720b:::smithi03829856-45:10, version: 70'58, data_included: [1927574~595528,2570780~420833], data_size: 1016361, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:953e720b:::smithi03829856-45:10@70'58, size: 2991613, copy_subset: [(0, 2991613)], clone_subset: {}, snapset: 10=[]:{b: [b, 6], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2991613, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1927574, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:953e720b:::smithi03829856-45:head, version: 71'60, data_included: [1823397~664039], data_size: 664039, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:953e720b:::smithi03829856-45:head@71'60, size: 2487436, copy_subset: [(0, 2487436)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2487436, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1823397, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:953e720b:::smithi03829856-45:b, version: 77'61, data_included: [1927574~595528,2570780~420833], data_size: 1016361, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:953e720b:::smithi03829856-45:b@77'61, size: 2991613, copy_subset: [(0, 2991613)], clone_subset: {}, snapset: 10=[]:{b: [b, 6], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2991613, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1927574, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:18.149 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:00.151568+0000", 2024-03-19T11:31:18.149 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 17.922007469, 2024-03-19T11:31:18.149 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015354197999999999, 2024-03-19T11:31:18.149 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:18.149 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:18.149 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:18.150 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.150 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:18.150 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.151568+0000", 2024-03-19T11:31:18.150 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.150 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.150 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.150 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:18.150 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.151568+0000", 2024-03-19T11:31:18.150 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.150 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.150 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.150 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:18.150 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.151569+0000", 2024-03-19T11:31:18.150 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.17e-06 2024-03-19T11:31:18.151 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.151 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.151 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:18.151 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.156296+0000", 2024-03-19T11:31:18.151 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0047264619999999999 2024-03-19T11:31:18.151 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.151 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.151 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:18.151 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.156334+0000", 2024-03-19T11:31:18.151 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8152000000000003e-05 2024-03-19T11:31:18.151 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.151 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.151 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:18.151 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.156370+0000", 2024-03-19T11:31:18.151 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6304999999999999e-05 2024-03-19T11:31:18.151 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.152 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.152 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.152 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.157994+0000", 2024-03-19T11:31:18.152 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0016235399999999999 2024-03-19T11:31:18.152 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.152 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.152 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:18.152 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.158017+0000", 2024-03-19T11:31:18.152 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3197000000000002e-05 2024-03-19T11:31:18.152 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.152 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.152 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:18.152 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.166922+0000", 2024-03-19T11:31:18.152 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0089053719999999999 2024-03-19T11:31:18.152 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.153 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:18.153 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.153 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.153 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:18.153 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:18.153 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-19T11:31:18.162 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:18.170+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:18.255 INFO:tasks.rados.rados.0.smithi038.stdout:1345: finishing write tid 5 to smithi03829856-40 2024-03-19T11:31:18.255 INFO:tasks.rados.rados.0.smithi038.stdout:1345: oid 40 updating version 0 to 75 2024-03-19T11:31:18.255 INFO:tasks.rados.rados.0.smithi038.stdout:1345: oid 40 updating version 75 to 76 2024-03-19T11:31:18.255 INFO:tasks.rados.rados.0.smithi038.stdout:1345: oid 40 version 76 is already newer than 74 2024-03-19T11:31:18.255 INFO:tasks.rados.rados.0.smithi038.stdout:1345: oid 40 version 76 is already newer than 73 2024-03-19T11:31:18.255 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 40 v 76 (ObjNum 227 snap 26 seq_num 227) dirty exists 2024-03-19T11:31:18.255 INFO:tasks.rados.rados.0.smithi038.stdout:1345: left oid 40 (ObjNum 227 snap 26 seq_num 227) 2024-03-19T11:31:18.255 INFO:tasks.rados.rados.0.smithi038.stdout:1348: expect (ObjNum 198 snap 22 seq_num 198) 2024-03-19T11:31:18.263 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:18.270+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:18.312 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:18.312 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:31:18.312 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:31:18.313 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:18.328 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-19T11:31:18.361 INFO:tasks.rados.rados.0.smithi038.stdout:1351: expect (ObjNum 201 snap 22 seq_num 201) 2024-03-19T11:31:18.363 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:18.374+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:18.465 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:18.473+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:18.476 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:18.476 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:31:18.477 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:31:18.477 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:31:18.477 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:18.491 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-19T11:31:18.565 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:18.574+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:18.566 INFO:tasks.rados.rados.0.smithi038.stdout:1352: expect (ObjNum 49 snap 0 seq_num 49) 2024-03-19T11:31:18.653 INFO:tasks.rados.rados.0.smithi038.stdout:1353: expect (ObjNum 206 snap 22 seq_num 206) 2024-03-19T11:31:18.656 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:18.656 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:31:18.657 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:31:18.657 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:31:18.657 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.657 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.16 49/48 [PushOp(4:6868af91:::smithi03829856-10:2, version: 29'22, data_included: [551263~793594,1893632~254982], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:2@29'22, size: 2151263, copy_subset: [(0, 2151263)], clone_subset: {}, snapset: b=[]:{2: [2], 7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2148614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:6d9093b1:::smithi03829856-38:4, version: 33'29, data_included: [621859~449625,1865716~598951], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6d9093b1:::smithi03829856-38:4@33'29, size: 2621859, copy_subset: [(0, 2621859)], clone_subset: {}, snapset: 4=[]:{4: [2]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2464667, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:6d9093b1:::smithi03829856-38:head, version: 33'33, data_included: [621859~568773,1729599~479803], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:6d9093b1:::smithi03829856-38:head@33'33, size: 3142876, copy_subset: [(0, 3142876)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2209402, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:6be89c06:::smithi03829856-24:head, version: 36'39, data_included: [741376~589824,1753088~458752], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:6be89c06:::smithi03829856-24:head@36'39, size: 2386561, copy_subset: [(0, 2386561)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2211840, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:6868af91:::smithi03829856-10:7, version: 38'41, data_included: [524385~820472,1602039~228104], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:7@38'41, size: 2924385, copy_subset: [(0, 2924385)], clone_subset: {}, snapset: b=[]:{2: [2], 7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1830143, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:6868af91:::smithi03829856-10:b, version: 47'48, data_included: [524385~820472,1602039~228104], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:b@47'48, size: 2924385, copy_subset: [(0, 2924385)], clone_subset: {}, snapset: b=[]:{2: [2], 7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1830143, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:6868af91:::smithi03829856-10:head, version: 47'52, data_included: [524385~820472,1602039~228104], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:head@47'52, size: 3747928, copy_subset: [(0, 3747928)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1830143, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:18.657 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:26.954289+0000", 2024-03-19T11:31:18.657 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 51.710540149000003, 2024-03-19T11:31:18.657 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029260781999999999, 2024-03-19T11:31:18.657 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:18.657 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:18.657 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:18.657 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.658 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:18.658 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954289+0000", 2024-03-19T11:31:18.658 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.658 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.658 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.658 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:18.658 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954289+0000", 2024-03-19T11:31:18.658 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.658 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.658 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.658 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:18.658 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954291+0000", 2024-03-19T11:31:18.659 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8050000000000001e-06 2024-03-19T11:31:18.659 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.659 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.659 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:18.659 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.958775+0000", 2024-03-19T11:31:18.659 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0044845859999999996 2024-03-19T11:31:18.659 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.659 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.659 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:18.659 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.958851+0000", 2024-03-19T11:31:18.659 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6050999999999994e-05 2024-03-19T11:31:18.659 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.660 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.660 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:18.660 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.958931+0000", 2024-03-19T11:31:18.660 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9099999999999998e-05 2024-03-19T11:31:18.660 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.660 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.660 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.660 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.959206+0000", 2024-03-19T11:31:18.661 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00027495999999999998 2024-03-19T11:31:18.661 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.661 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.661 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:18.661 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.959246+0000", 2024-03-19T11:31:18.661 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0326e-05 2024-03-19T11:31:18.661 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.661 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.661 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:18.661 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.983550+0000", 2024-03-19T11:31:18.661 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024303953999999999 2024-03-19T11:31:18.661 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.661 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:18.662 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.662 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.662 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.662 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.16 49/48 [PushOp(4:6868af91:::smithi03829856-10:2, version: 29'22, data_included: [2148614~2649], data_size: 2649, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:2@29'22, size: 2151263, copy_subset: [(0, 2151263)], clone_subset: {}, snapset: b=[]:{2: [2], 7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2151263, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2148614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:6d9093b1:::smithi03829856-38:4, version: 33'29, data_included: [2464667~157192], data_size: 157192, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:6d9093b1:::smithi03829856-38:4@33'29, size: 2621859, copy_subset: [(0, 2621859)], clone_subset: {}, snapset: 4=[]:{4: [2]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2621859, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2464667, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:6d9093b1:::smithi03829856-38:head, version: 33'33, data_included: [2209402~412457,3101234~41642], data_size: 454099, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:6d9093b1:::smithi03829856-38:head@33'33, size: 3142876, copy_subset: [(0, 3142876)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3142876, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2209402, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:6be89c06:::smithi03829856-24:head, version: 36'39, data_included: [2211840~51851], data_size: 51851, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:6be89c06:::smithi03829856-24:head@36'39, size: 2386561, copy_subset: [(0, 2386561)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2386561, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2211840, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:6868af91:::smithi03829856-10:7, version: 38'41, data_included: [1830143~321120,2577441~346944], data_size: 668064, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:7@38'41, size: 2924385, copy_subset: [(0, 2924385)], clone_subset: {}, snapset: b=[]:{2: [2], 7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2924385, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1830143, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:6868af91:::smithi03829856-10:b, version: 47'48, data_included: [1830143~321120,2577441~346944], data_size: 668064, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:b@47'48, size: 2924385, copy_subset: [(0, 2924385)], clone_subset: {}, snapset: b=[]:{2: [2], 7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2924385, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1830143, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:6868af91:::smithi03829856-10:head, version: 47'52, data_included: [1830143~348419,2577441~661310], data_size: 1009729, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:head@47'52, size: 3747928, copy_subset: [(0, 3747928)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3747928, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1830143, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:18.662 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:26.998857+0000", 2024-03-19T11:31:18.662 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 51.665972388999997, 2024-03-19T11:31:18.662 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010149703, 2024-03-19T11:31:18.662 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:18.663 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:18.663 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:18.663 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.663 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:18.663 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.998857+0000", 2024-03-19T11:31:18.663 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.663 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.663 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.663 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:18.663 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.998857+0000", 2024-03-19T11:31:18.663 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.663 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.663 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.664 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:18.664 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.998858+0000", 2024-03-19T11:31:18.664 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.065e-06 2024-03-19T11:31:18.664 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.664 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.664 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:18.664 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.999629+0000", 2024-03-19T11:31:18.664 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000771132 2024-03-19T11:31:18.664 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.665 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.665 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:18.665 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.999688+0000", 2024-03-19T11:31:18.665 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9147000000000001e-05 2024-03-19T11:31:18.665 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.665 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.665 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:18.665 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.999755+0000", 2024-03-19T11:31:18.665 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6891999999999996e-05 2024-03-19T11:31:18.665 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.665 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.665 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.666 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.000047+0000", 2024-03-19T11:31:18.666 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000292176 2024-03-19T11:31:18.666 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.666 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.666 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:18.666 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.000094+0000", 2024-03-19T11:31:18.666 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6485e-05 2024-03-19T11:31:18.666 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.666 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.667 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:18.667 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.009007+0000", 2024-03-19T11:31:18.667 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0089128060000000005 2024-03-19T11:31:18.667 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.667 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:18.667 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.667 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.667 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.667 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1591 4.4 4:266590a5:::smithi03829856-16:head [write 783226~624677,stat] snapc e=[e,d,c,b,a,6] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-03-19T11:31:18.667 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:32.390032+0000", 2024-03-19T11:31:18.667 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 46.274797604, 2024-03-19T11:31:18.667 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0090431220000000007, 2024-03-19T11:31:18.668 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:18.668 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:18.668 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:18.668 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:18.668 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:18.668 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1591 2024-03-19T11:31:18.668 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.668 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:18.668 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.668 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:18.668 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.390032+0000", 2024-03-19T11:31:18.669 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.669 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.669 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.669 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:18.669 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.390032+0000", 2024-03-19T11:31:18.669 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.669 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.669 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.670 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:18.670 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.390033+0000", 2024-03-19T11:31:18.670 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2839999999999999e-06 2024-03-19T11:31:18.670 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.670 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.670 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:18.670 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.390273+0000", 2024-03-19T11:31:18.670 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00024011099999999999 2024-03-19T11:31:18.671 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.671 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.671 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:18.671 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.390286+0000", 2024-03-19T11:31:18.671 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3314e-05 2024-03-19T11:31:18.671 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.671 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.671 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:18.671 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.390295+0000", 2024-03-19T11:31:18.672 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6330000000000002e-06 2024-03-19T11:31:18.672 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.672 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.672 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.672 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.390368+0000", 2024-03-19T11:31:18.672 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3065999999999995e-05 2024-03-19T11:31:18.672 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.672 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.672 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:18.672 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.390461+0000", 2024-03-19T11:31:18.672 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2582000000000002e-05 2024-03-19T11:31:18.672 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.673 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.673 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3", 2024-03-19T11:31:18.673 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.390648+0000", 2024-03-19T11:31:18.673 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00018695300000000001 2024-03-19T11:31:18.673 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.673 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.673 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:18.673 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.398909+0000", 2024-03-19T11:31:18.673 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0082613559999999992 2024-03-19T11:31:18.673 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.673 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.674 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:18.674 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.398954+0000", 2024-03-19T11:31:18.674 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5553999999999998e-05 2024-03-19T11:31:18.674 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.674 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.674 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:18.674 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.399017+0000", 2024-03-19T11:31:18.674 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2849999999999996e-05 2024-03-19T11:31:18.674 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.674 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.674 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:18.674 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.399075+0000", 2024-03-19T11:31:18.675 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7419e-05 2024-03-19T11:31:18.675 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.675 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:18.675 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.675 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.675 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.675 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1630 4.d 4:ba44105a:::smithi03829856-26:head [write 467261~717275] snapc 10=[10,f,e,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-03-19T11:31:18.675 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:35.331434+0000", 2024-03-19T11:31:18.675 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 43.333395344000003, 2024-03-19T11:31:18.676 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012902375000000001, 2024-03-19T11:31:18.676 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:18.676 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:18.676 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:18.676 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:18.676 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:18.676 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1630 2024-03-19T11:31:18.676 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.676 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:18.676 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.676 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:18.676 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.331434+0000", 2024-03-19T11:31:18.677 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.677 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.677 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.677 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:18.677 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.331434+0000", 2024-03-19T11:31:18.677 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.677 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.677 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.677 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:18.677 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.331437+0000", 2024-03-19T11:31:18.677 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5599999999999998e-06 2024-03-19T11:31:18.677 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.677 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.678 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:18.678 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.331938+0000", 2024-03-19T11:31:18.678 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00050090099999999999 2024-03-19T11:31:18.678 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.678 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.678 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:18.678 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.331976+0000", 2024-03-19T11:31:18.678 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8090999999999998e-05 2024-03-19T11:31:18.678 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.678 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.678 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:18.678 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.331994+0000", 2024-03-19T11:31:18.679 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7566000000000001e-05 2024-03-19T11:31:18.679 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.679 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.679 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.679 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.332086+0000", 2024-03-19T11:31:18.679 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2452000000000002e-05 2024-03-19T11:31:18.679 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.679 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.679 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:18.679 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.332205+0000", 2024-03-19T11:31:18.679 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000118966 2024-03-19T11:31:18.679 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.679 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.680 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1", 2024-03-19T11:31:18.680 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.332410+0000", 2024-03-19T11:31:18.680 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020430099999999999 2024-03-19T11:31:18.680 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.680 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.680 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:18.680 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.339226+0000", 2024-03-19T11:31:18.680 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0068165980000000001 2024-03-19T11:31:18.680 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.680 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.680 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:18.680 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.344191+0000", 2024-03-19T11:31:18.681 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.004965143 2024-03-19T11:31:18.681 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.681 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.681 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:18.681 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.344272+0000", 2024-03-19T11:31:18.681 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0265000000000005e-05 2024-03-19T11:31:18.681 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.681 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.681 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:18.681 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.344336+0000", 2024-03-19T11:31:18.681 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4531999999999998e-05 2024-03-19T11:31:18.681 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.681 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:18.682 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.682 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.682 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.682 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1631 4.d 4:ba44105a:::smithi03829856-26:head [write 1847991~219270] snapc 10=[10,f,e,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-03-19T11:31:18.682 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:35.336760+0000", 2024-03-19T11:31:18.682 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 43.328068741999999, 2024-03-19T11:31:18.682 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0087315580000000004, 2024-03-19T11:31:18.682 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:18.682 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:18.682 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:18.683 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:18.683 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:18.683 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1631 2024-03-19T11:31:18.683 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.683 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:18.683 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.684 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:18.684 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.336760+0000", 2024-03-19T11:31:18.684 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.684 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.685 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.685 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:18.685 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.336760+0000", 2024-03-19T11:31:18.685 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.685 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.685 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.685 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:18.685 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.336762+0000", 2024-03-19T11:31:18.686 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7099999999999999e-06 2024-03-19T11:31:18.686 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.686 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.686 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:18.686 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.336909+0000", 2024-03-19T11:31:18.686 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000147045 2024-03-19T11:31:18.686 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.686 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.686 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:18.686 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.336923+0000", 2024-03-19T11:31:18.686 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3526999999999999e-05 2024-03-19T11:31:18.687 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.687 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.687 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:18.687 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.336943+0000", 2024-03-19T11:31:18.687 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0358999999999999e-05 2024-03-19T11:31:18.687 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.687 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.687 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.687 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.337039+0000", 2024-03-19T11:31:18.687 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.5644999999999997e-05 2024-03-19T11:31:18.687 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.688 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.688 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:18.688 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.337155+0000", 2024-03-19T11:31:18.688 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000115913 2024-03-19T11:31:18.688 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.688 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.688 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1", 2024-03-19T11:31:18.688 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.337341+0000", 2024-03-19T11:31:18.688 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000186472 2024-03-19T11:31:18.688 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.688 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.688 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:18.689 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.339232+0000", 2024-03-19T11:31:18.689 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001890954 2024-03-19T11:31:18.689 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.689 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.689 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:18.689 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.345393+0000", 2024-03-19T11:31:18.689 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.006161232 2024-03-19T11:31:18.689 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.689 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.689 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:18.689 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.345449+0000", 2024-03-19T11:31:18.689 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5340000000000002e-05 2024-03-19T11:31:18.689 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.690 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.690 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:18.690 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.345492+0000", 2024-03-19T11:31:18.690 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3361e-05 2024-03-19T11:31:18.690 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.690 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:18.690 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.690 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.690 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.690 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1632 4.d 4:ba44105a:::smithi03829856-26:head [setxattr _header (57) in=7b,truncate 2067261] snapc 10=[10,f,e,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-03-19T11:31:18.690 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:35.336987+0000", 2024-03-19T11:31:18.690 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 43.327842683999997, 2024-03-19T11:31:18.691 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0091098789999999995, 2024-03-19T11:31:18.691 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:18.691 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:18.691 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:18.691 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:18.691 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:18.691 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1632 2024-03-19T11:31:18.691 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.691 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:18.691 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.691 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:18.691 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.336987+0000", 2024-03-19T11:31:18.692 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.692 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.692 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.692 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:18.692 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.336987+0000", 2024-03-19T11:31:18.692 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.692 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.692 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.692 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:18.692 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.336987+0000", 2024-03-19T11:31:18.692 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4600000000000004e-07 2024-03-19T11:31:18.692 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.692 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.693 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:18.693 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.336992+0000", 2024-03-19T11:31:18.693 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6090000000000003e-06 2024-03-19T11:31:18.693 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.693 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.693 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:18.693 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.337002+0000", 2024-03-19T11:31:18.693 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.006e-05 2024-03-19T11:31:18.693 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.693 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.693 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:18.693 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.337009+0000", 2024-03-19T11:31:18.694 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7229999999999998e-06 2024-03-19T11:31:18.694 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.694 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.694 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.694 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.338513+0000", 2024-03-19T11:31:18.695 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001503868 2024-03-19T11:31:18.695 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.695 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.695 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:18.695 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.338566+0000", 2024-03-19T11:31:18.695 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3260999999999997e-05 2024-03-19T11:31:18.695 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.695 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.695 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1", 2024-03-19T11:31:18.695 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.338652+0000", 2024-03-19T11:31:18.695 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5889000000000004e-05 2024-03-19T11:31:18.695 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.695 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.696 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:18.696 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.340450+0000", 2024-03-19T11:31:18.696 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017987490000000001 2024-03-19T11:31:18.696 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.696 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.696 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:18.696 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.345982+0000", 2024-03-19T11:31:18.696 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055312620000000003 2024-03-19T11:31:18.696 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.696 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.696 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:18.696 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.346039+0000", 2024-03-19T11:31:18.696 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6879999999999998e-05 2024-03-19T11:31:18.696 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.696 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.697 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:18.697 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.346096+0000", 2024-03-19T11:31:18.697 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7831999999999998e-05 2024-03-19T11:31:18.697 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.697 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:18.697 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.697 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.697 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.697 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1633 4.d 4:ba44105a:::smithi03829856-26:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-03-19T11:31:18.697 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:35.337032+0000", 2024-03-19T11:31:18.697 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 43.327796964999997, 2024-03-19T11:31:18.697 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093229679999999992, 2024-03-19T11:31:18.698 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:18.698 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:18.698 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:18.698 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:18.698 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:18.698 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1633 2024-03-19T11:31:18.698 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.698 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:18.698 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.698 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:18.699 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.337032+0000", 2024-03-19T11:31:18.699 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.699 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.699 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.699 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:18.699 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.337032+0000", 2024-03-19T11:31:18.699 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.699 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.699 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.699 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:18.699 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.337033+0000", 2024-03-19T11:31:18.699 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4899999999999995e-07 2024-03-19T11:31:18.699 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.699 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.700 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:18.700 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.337037+0000", 2024-03-19T11:31:18.700 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7249999999999999e-06 2024-03-19T11:31:18.700 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.700 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.700 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:18.700 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.337045+0000", 2024-03-19T11:31:18.700 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5380000000000002e-06 2024-03-19T11:31:18.700 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.701 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.701 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:18.701 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.337051+0000", 2024-03-19T11:31:18.701 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1689999999999997e-06 2024-03-19T11:31:18.701 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.701 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.701 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.701 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.339128+0000", 2024-03-19T11:31:18.701 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002076919 2024-03-19T11:31:18.701 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.701 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.701 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-03-19T11:31:18.701 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.339211+0000", 2024-03-19T11:31:18.701 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2896000000000001e-05 2024-03-19T11:31:18.701 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.702 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.702 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.702 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.346152+0000", 2024-03-19T11:31:18.702 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0069404779999999999 2024-03-19T11:31:18.702 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.702 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.702 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:18.702 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.346242+0000", 2024-03-19T11:31:18.702 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0031000000000006e-05 2024-03-19T11:31:18.702 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.702 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.703 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:18.703 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.346355+0000", 2024-03-19T11:31:18.703 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000113663 2024-03-19T11:31:18.703 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.703 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:18.703 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.703 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.703 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.703 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1844 4.4 4:27909373:::smithi03829856-46:head [setxattr _header (57) in=7b,truncate 2403401] snapc 10=[10,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e62)", 2024-03-19T11:31:18.703 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:41.053712+0000", 2024-03-19T11:31:18.703 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 37.611117446999998, 2024-03-19T11:31:18.703 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0086612310000000001, 2024-03-19T11:31:18.703 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:18.703 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:18.704 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:18.704 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:18.704 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:18.704 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1844 2024-03-19T11:31:18.704 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.704 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:18.704 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.704 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:18.704 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053712+0000", 2024-03-19T11:31:18.704 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.704 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.705 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.705 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:18.705 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053712+0000", 2024-03-19T11:31:18.705 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.705 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.705 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.705 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:18.705 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053714+0000", 2024-03-19T11:31:18.705 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3460000000000001e-06 2024-03-19T11:31:18.705 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.705 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.705 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:18.705 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053725+0000", 2024-03-19T11:31:18.706 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0763e-05 2024-03-19T11:31:18.706 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.706 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.706 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:18.706 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053748+0000", 2024-03-19T11:31:18.706 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3408999999999999e-05 2024-03-19T11:31:18.706 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.706 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.706 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:18.706 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053765+0000", 2024-03-19T11:31:18.706 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6429000000000001e-05 2024-03-19T11:31:18.707 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.707 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.707 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.707 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.058093+0000", 2024-03-19T11:31:18.707 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0043278580000000004 2024-03-19T11:31:18.707 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.707 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.707 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:18.707 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.058263+0000", 2024-03-19T11:31:18.707 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017061100000000001 2024-03-19T11:31:18.707 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.707 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.707 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3", 2024-03-19T11:31:18.707 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.058539+0000", 2024-03-19T11:31:18.707 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00027568500000000001 2024-03-19T11:31:18.708 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.708 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.708 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:18.708 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.061471+0000", 2024-03-19T11:31:18.708 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002932268 2024-03-19T11:31:18.708 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.708 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.708 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:18.708 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.061933+0000", 2024-03-19T11:31:18.708 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00046187700000000002 2024-03-19T11:31:18.708 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.708 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.708 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:18.708 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.062102+0000", 2024-03-19T11:31:18.708 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000169295 2024-03-19T11:31:18.709 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.709 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.709 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:18.709 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.062373+0000", 2024-03-19T11:31:18.709 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00027069000000000003 2024-03-19T11:31:18.709 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.709 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:18.709 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.709 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.709 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.709 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1845 4.4 4:27909373:::smithi03829856-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-03-19T11:31:18.709 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:41.053790+0000", 2024-03-19T11:31:18.709 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 37.611038884000003, 2024-03-19T11:31:18.709 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0091637609999999994, 2024-03-19T11:31:18.709 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:18.710 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:18.710 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:18.710 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:18.710 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:18.710 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1845 2024-03-19T11:31:18.710 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.710 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:18.710 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.710 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:18.710 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053790+0000", 2024-03-19T11:31:18.710 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.710 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.710 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.710 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:18.711 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053790+0000", 2024-03-19T11:31:18.711 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.711 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.711 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.711 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:18.711 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053792+0000", 2024-03-19T11:31:18.711 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2860000000000001e-06 2024-03-19T11:31:18.711 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.711 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.711 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:18.711 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053802+0000", 2024-03-19T11:31:18.711 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.064e-05 2024-03-19T11:31:18.711 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.711 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.711 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:18.712 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053824+0000", 2024-03-19T11:31:18.712 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1611999999999999e-05 2024-03-19T11:31:18.712 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.712 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.712 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:18.712 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053839+0000", 2024-03-19T11:31:18.712 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4922e-05 2024-03-19T11:31:18.712 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.712 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.712 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.712 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.059104+0000", 2024-03-19T11:31:18.712 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0052656179999999997 2024-03-19T11:31:18.712 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.712 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.712 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-03-19T11:31:18.713 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.059267+0000", 2024-03-19T11:31:18.713 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000162776 2024-03-19T11:31:18.713 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.713 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.713 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.713 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.062551+0000", 2024-03-19T11:31:18.713 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003283467 2024-03-19T11:31:18.713 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.713 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.713 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:18.713 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.062738+0000", 2024-03-19T11:31:18.713 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00018781900000000001 2024-03-19T11:31:18.713 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.713 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.713 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:18.714 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.062954+0000", 2024-03-19T11:31:18.714 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00021562099999999999 2024-03-19T11:31:18.714 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.714 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:18.714 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.714 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.714 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.714 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1848 4.d 4:b65e6658:::smithi03829856-9:6 [sparse-read 0~901298] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-03-19T11:31:18.714 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:41.053966+0000", 2024-03-19T11:31:18.714 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 37.610863430000002, 2024-03-19T11:31:18.714 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0090164259999999993, 2024-03-19T11:31:18.714 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:18.714 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:18.714 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:18.714 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:18.715 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:18.715 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1848 2024-03-19T11:31:18.715 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.715 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:18.715 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.715 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:18.715 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053966+0000", 2024-03-19T11:31:18.715 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.715 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.715 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.715 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:18.715 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053966+0000", 2024-03-19T11:31:18.715 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.715 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.715 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.716 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:18.716 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053967+0000", 2024-03-19T11:31:18.716 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.387e-06 2024-03-19T11:31:18.716 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.716 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.716 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:18.716 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053975+0000", 2024-03-19T11:31:18.716 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5299999999999999e-06 2024-03-19T11:31:18.716 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.716 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.716 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:18.716 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053993+0000", 2024-03-19T11:31:18.716 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7929000000000001e-05 2024-03-19T11:31:18.716 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.716 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.717 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:18.717 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.054010+0000", 2024-03-19T11:31:18.717 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7876e-05 2024-03-19T11:31:18.717 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.717 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.717 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.717 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.062407+0000", 2024-03-19T11:31:18.717 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0083966200000000005 2024-03-19T11:31:18.717 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.717 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.717 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:18.717 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.062664+0000", 2024-03-19T11:31:18.717 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025712100000000002 2024-03-19T11:31:18.717 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.717 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.717 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:18.718 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.062982+0000", 2024-03-19T11:31:18.718 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00031796299999999999 2024-03-19T11:31:18.718 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.718 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:18.718 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.718 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.718 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.718 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1889 4.4 4:266590a5:::smithi03829856-16:head [rollback d] snapc 10=[10,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-03-19T11:31:18.718 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:41.697617+0000", 2024-03-19T11:31:18.718 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 36.967212349999997, 2024-03-19T11:31:18.718 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018164059999999999, 2024-03-19T11:31:18.718 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:18.718 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:18.718 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:18.719 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:18.719 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:18.719 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1889 2024-03-19T11:31:18.719 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.719 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:18.719 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.719 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:18.719 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697617+0000", 2024-03-19T11:31:18.719 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.719 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.719 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.719 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:18.719 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697617+0000", 2024-03-19T11:31:18.719 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.719 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.720 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.720 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:18.720 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697619+0000", 2024-03-19T11:31:18.720 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0059999999999999e-06 2024-03-19T11:31:18.720 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.720 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.720 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:18.720 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697631+0000", 2024-03-19T11:31:18.720 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2033e-05 2024-03-19T11:31:18.720 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.720 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.720 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:18.720 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697659+0000", 2024-03-19T11:31:18.720 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8291999999999999e-05 2024-03-19T11:31:18.720 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.721 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.721 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:18.721 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697682+0000", 2024-03-19T11:31:18.721 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3281999999999998e-05 2024-03-19T11:31:18.721 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.721 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.721 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.721 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.701122+0000", 2024-03-19T11:31:18.721 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034396679999999999 2024-03-19T11:31:18.721 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.721 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.721 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:18.721 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.701383+0000", 2024-03-19T11:31:18.721 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00026086399999999997 2024-03-19T11:31:18.721 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.722 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.722 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3", 2024-03-19T11:31:18.722 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.709693+0000", 2024-03-19T11:31:18.722 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0083096139999999999 2024-03-19T11:31:18.722 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.722 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.722 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:18.722 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.715437+0000", 2024-03-19T11:31:18.722 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057442379999999996 2024-03-19T11:31:18.722 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.722 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.722 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:18.722 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.715563+0000", 2024-03-19T11:31:18.722 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000125778 2024-03-19T11:31:18.723 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.723 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.723 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:18.723 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.715677+0000", 2024-03-19T11:31:18.723 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000114728 2024-03-19T11:31:18.723 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.723 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.723 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:18.723 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.715781+0000", 2024-03-19T11:31:18.723 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000103557 2024-03-19T11:31:18.723 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.723 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:18.723 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.723 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.723 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.724 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1890 4.4 4:266590a5:::smithi03829856-16:head [append 0~0] snapc 10=[10,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-03-19T11:31:18.724 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:41.697709+0000", 2024-03-19T11:31:18.724 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 36.967119787999998, 2024-03-19T11:31:18.724 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019825171999999999, 2024-03-19T11:31:18.724 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:18.724 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:18.724 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:18.724 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:18.724 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:18.724 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1890 2024-03-19T11:31:18.724 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.724 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:18.724 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.724 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:18.724 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697709+0000", 2024-03-19T11:31:18.725 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.725 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.725 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.725 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:18.725 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697709+0000", 2024-03-19T11:31:18.725 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.725 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.725 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.725 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:18.725 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697711+0000", 2024-03-19T11:31:18.725 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.296e-06 2024-03-19T11:31:18.725 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.725 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.725 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:18.725 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697718+0000", 2024-03-19T11:31:18.726 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.464e-06 2024-03-19T11:31:18.726 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.726 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.726 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:18.726 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697739+0000", 2024-03-19T11:31:18.726 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1005e-05 2024-03-19T11:31:18.726 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.726 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.726 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:18.726 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697754+0000", 2024-03-19T11:31:18.726 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4976e-05 2024-03-19T11:31:18.726 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.726 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.726 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.726 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.702341+0000", 2024-03-19T11:31:18.727 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0045870980000000004 2024-03-19T11:31:18.727 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.727 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.727 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for blocked object", 2024-03-19T11:31:18.727 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.702482+0000", 2024-03-19T11:31:18.727 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014048000000000001 2024-03-19T11:31:18.727 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.727 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.727 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.727 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.714359+0000", 2024-03-19T11:31:18.727 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011877522999999999 2024-03-19T11:31:18.727 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.727 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.727 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:18.727 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.714550+0000", 2024-03-19T11:31:18.728 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019117 2024-03-19T11:31:18.728 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.728 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.728 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3", 2024-03-19T11:31:18.728 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.714911+0000", 2024-03-19T11:31:18.728 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00036070900000000002 2024-03-19T11:31:18.728 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.728 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.728 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:18.728 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.716211+0000", 2024-03-19T11:31:18.728 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001300329 2024-03-19T11:31:18.728 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.728 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.728 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:18.728 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.717326+0000", 2024-03-19T11:31:18.729 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001114056 2024-03-19T11:31:18.729 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.729 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.729 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:18.729 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.717423+0000", 2024-03-19T11:31:18.729 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7048999999999999e-05 2024-03-19T11:31:18.729 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.729 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.729 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:18.729 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.717535+0000", 2024-03-19T11:31:18.729 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011201699999999999 2024-03-19T11:31:18.729 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.729 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:18.729 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.729 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.730 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.730 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1972 4.4 4:214026b3:::smithi03829856-47:head [tier-promote] snapc 12=[12,11,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:31:18.730 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:45.312422+0000", 2024-03-19T11:31:18.730 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 33.352407114000002, 2024-03-19T11:31:18.730 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013955089, 2024-03-19T11:31:18.730 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:18.730 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:18.730 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:18.730 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:18.730 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:18.730 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1972 2024-03-19T11:31:18.730 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.730 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:18.730 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.731 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:18.731 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.312422+0000", 2024-03-19T11:31:18.731 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.731 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.731 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.731 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:18.731 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.312422+0000", 2024-03-19T11:31:18.731 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.731 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.731 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.731 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:18.731 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.312424+0000", 2024-03-19T11:31:18.731 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9810000000000002e-06 2024-03-19T11:31:18.731 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.731 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.732 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:18.732 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.312433+0000", 2024-03-19T11:31:18.732 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4549999999999995e-06 2024-03-19T11:31:18.732 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.732 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.732 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:18.732 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.312452+0000", 2024-03-19T11:31:18.732 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9380000000000001e-05 2024-03-19T11:31:18.732 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.732 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.732 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:18.732 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.312460+0000", 2024-03-19T11:31:18.732 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2560000000000002e-06 2024-03-19T11:31:18.732 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.733 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.733 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.733 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.312555+0000", 2024-03-19T11:31:18.733 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.5208000000000001e-05 2024-03-19T11:31:18.733 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.733 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.733 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:18.733 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.312704+0000", 2024-03-19T11:31:18.733 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014843899999999999 2024-03-19T11:31:18.733 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.733 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.733 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:18.733 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.326377+0000", 2024-03-19T11:31:18.733 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013673370000000001 2024-03-19T11:31:18.733 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.734 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:18.734 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.734 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.734 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.734 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2005 4.4 4:214026b3:::smithi03829856-47:head [read 0~2773122,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:31:18.734 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:45.804325+0000", 2024-03-19T11:31:18.734 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 32.860504654000003, 2024-03-19T11:31:18.734 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010899282, 2024-03-19T11:31:18.734 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:18.734 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:18.734 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:18.734 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:18.734 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:18.734 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2005 2024-03-19T11:31:18.734 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.735 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:18.735 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.735 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:18.735 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804325+0000", 2024-03-19T11:31:18.735 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.735 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.735 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.735 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:18.735 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804325+0000", 2024-03-19T11:31:18.735 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.735 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.735 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.735 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:18.735 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804326+0000", 2024-03-19T11:31:18.736 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.666e-06 2024-03-19T11:31:18.736 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.736 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.736 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:18.736 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804338+0000", 2024-03-19T11:31:18.736 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2096000000000001e-05 2024-03-19T11:31:18.736 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.736 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.736 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:18.736 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804363+0000", 2024-03-19T11:31:18.736 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4762000000000001e-05 2024-03-19T11:31:18.736 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.736 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.736 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:18.736 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804380+0000", 2024-03-19T11:31:18.737 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.713e-05 2024-03-19T11:31:18.737 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.737 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.737 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.737 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.812236+0000", 2024-03-19T11:31:18.737 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0078554690000000003 2024-03-19T11:31:18.737 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.737 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.737 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:18.737 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.812541+0000", 2024-03-19T11:31:18.737 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030564299999999997 2024-03-19T11:31:18.737 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.737 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.737 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:18.737 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.815224+0000", 2024-03-19T11:31:18.738 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0026825159999999998 2024-03-19T11:31:18.738 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.738 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:18.738 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.738 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.738 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.738 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2006 4.4 4:214026b3:::smithi03829856-47:head [read 0~2773122,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:31:18.738 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:45.804421+0000", 2024-03-19T11:31:18.738 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 32.860407915000003, 2024-03-19T11:31:18.738 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013366463, 2024-03-19T11:31:18.738 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:18.738 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:18.738 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:18.739 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:18.739 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:18.739 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2006 2024-03-19T11:31:18.739 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.739 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:18.739 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.739 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:18.739 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804421+0000", 2024-03-19T11:31:18.739 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.739 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.739 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.739 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:18.739 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804421+0000", 2024-03-19T11:31:18.739 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.739 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.740 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.740 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:18.740 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804423+0000", 2024-03-19T11:31:18.740 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5600000000000001e-06 2024-03-19T11:31:18.740 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.740 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.740 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:18.740 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804433+0000", 2024-03-19T11:31:18.740 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8719999999999997e-06 2024-03-19T11:31:18.740 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.740 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.740 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:18.740 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804454+0000", 2024-03-19T11:31:18.740 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1759e-05 2024-03-19T11:31:18.740 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.741 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.741 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:18.741 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804470+0000", 2024-03-19T11:31:18.741 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5556000000000001e-05 2024-03-19T11:31:18.741 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.741 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.741 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.741 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.815325+0000", 2024-03-19T11:31:18.741 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010855069 2024-03-19T11:31:18.741 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.741 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.741 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:18.741 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.815684+0000", 2024-03-19T11:31:18.741 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00035912699999999999 2024-03-19T11:31:18.741 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.742 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.742 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:18.742 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.817788+0000", 2024-03-19T11:31:18.742 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0021035200000000002 2024-03-19T11:31:18.742 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.742 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:18.742 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.742 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.742 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.742 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(osd.1.0:198 4.16 e67/48 4:6868af91:::smithi03829856-10:head v 67'79, mlcod=66'76)", 2024-03-19T11:31:18.742 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:45.811039+0000", 2024-03-19T11:31:18.742 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 32.853790554, 2024-03-19T11:31:18.742 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01087364, 2024-03-19T11:31:18.743 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:18.743 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:18.743 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:18.743 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.743 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:18.743 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.811039+0000", 2024-03-19T11:31:18.743 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.743 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.743 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.743 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:18.743 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.811039+0000", 2024-03-19T11:31:18.743 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.743 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.743 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.743 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:18.744 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.811041+0000", 2024-03-19T11:31:18.744 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6400000000000001e-06 2024-03-19T11:31:18.744 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.744 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.744 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:18.744 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.812874+0000", 2024-03-19T11:31:18.744 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0018329959999999999 2024-03-19T11:31:18.744 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.744 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.744 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:18.744 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.812916+0000", 2024-03-19T11:31:18.744 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1875000000000001e-05 2024-03-19T11:31:18.744 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.744 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.745 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:18.745 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.812937+0000", 2024-03-19T11:31:18.745 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0985e-05 2024-03-19T11:31:18.745 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.745 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.745 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.745 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.813112+0000", 2024-03-19T11:31:18.745 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000174573 2024-03-19T11:31:18.745 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.745 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.745 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:18.745 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.813170+0000", 2024-03-19T11:31:18.745 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8100999999999998e-05 2024-03-19T11:31:18.745 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.746 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.746 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:18.746 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.821792+0000", 2024-03-19T11:31:18.746 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0086218549999999994 2024-03-19T11:31:18.746 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.746 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.746 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:18.746 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.821912+0000", 2024-03-19T11:31:18.746 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000120615 2024-03-19T11:31:18.746 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.746 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:18.746 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.746 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.746 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.747 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2027 4.2d 4:b4ef97e0:::smithi03829856-3:head [read 0~3697507,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:31:18.747 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:46.264908+0000", 2024-03-19T11:31:18.747 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 32.399921687999999, 2024-03-19T11:31:18.747 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010951414, 2024-03-19T11:31:18.747 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:18.747 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:18.747 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:18.747 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:18.747 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:18.747 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2027 2024-03-19T11:31:18.747 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.747 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:18.747 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.747 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:18.748 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264908+0000", 2024-03-19T11:31:18.748 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.748 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.748 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.748 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:18.748 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264908+0000", 2024-03-19T11:31:18.748 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.748 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.748 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.748 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:18.748 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264908+0000", 2024-03-19T11:31:18.748 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5799999999999999e-07 2024-03-19T11:31:18.748 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.748 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.748 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:18.749 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264912+0000", 2024-03-19T11:31:18.749 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2649999999999998e-06 2024-03-19T11:31:18.749 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.749 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.749 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:18.749 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264920+0000", 2024-03-19T11:31:18.749 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.79e-06 2024-03-19T11:31:18.749 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.749 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.749 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:18.749 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264926+0000", 2024-03-19T11:31:18.749 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9989999999999997e-06 2024-03-19T11:31:18.749 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.749 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.750 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.750 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.273461+0000", 2024-03-19T11:31:18.750 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0085352680000000004 2024-03-19T11:31:18.750 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.750 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.750 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:18.750 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.273598+0000", 2024-03-19T11:31:18.750 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013622299999999999 2024-03-19T11:31:18.750 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.750 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.750 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:18.750 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.275859+0000", 2024-03-19T11:31:18.750 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022613109999999998 2024-03-19T11:31:18.750 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.751 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:18.751 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.751 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.751 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.751 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2028 4.2d 4:b4ef97e0:::smithi03829856-3:head [read 0~3697507,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:31:18.751 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:46.264934+0000", 2024-03-19T11:31:18.751 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 32.399895057000002, 2024-03-19T11:31:18.751 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013708481, 2024-03-19T11:31:18.751 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:18.751 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:18.751 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:18.751 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:18.751 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:18.751 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2028 2024-03-19T11:31:18.751 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.752 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:18.752 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.752 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:18.752 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264934+0000", 2024-03-19T11:31:18.752 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.752 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.752 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.752 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:18.752 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264934+0000", 2024-03-19T11:31:18.752 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.752 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.752 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.752 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:18.752 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264935+0000", 2024-03-19T11:31:18.752 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2599999999999998e-07 2024-03-19T11:31:18.753 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.753 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.753 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:18.753 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264938+0000", 2024-03-19T11:31:18.753 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.163e-06 2024-03-19T11:31:18.753 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.753 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.753 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:18.753 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264945+0000", 2024-03-19T11:31:18.753 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.0369999999999997e-06 2024-03-19T11:31:18.753 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.753 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.753 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:18.753 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264950+0000", 2024-03-19T11:31:18.754 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1170000000000003e-06 2024-03-19T11:31:18.754 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.754 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.754 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.754 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.275914+0000", 2024-03-19T11:31:18.754 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010964150000000001 2024-03-19T11:31:18.754 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.754 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.754 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:18.754 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.276037+0000", 2024-03-19T11:31:18.754 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012250100000000001 2024-03-19T11:31:18.754 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.754 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.754 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:18.754 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.278643+0000", 2024-03-19T11:31:18.755 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002606087 2024-03-19T11:31:18.755 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.755 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:18.755 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.755 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.755 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.755 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2045 4.4 4:21eac9b2:::smithi03829856-4:head [read 0~2537857,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:31:18.755 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:46.265120+0000", 2024-03-19T11:31:18.755 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 32.399708748999998, 2024-03-19T11:31:18.755 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0096591980000000008, 2024-03-19T11:31:18.755 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:18.755 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:18.755 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:18.755 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:18.755 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:18.756 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2045 2024-03-19T11:31:18.756 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.756 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:18.756 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.756 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:18.756 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265120+0000", 2024-03-19T11:31:18.756 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.756 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.756 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.756 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:18.756 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265120+0000", 2024-03-19T11:31:18.756 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.756 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.756 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.756 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:18.757 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265121+0000", 2024-03-19T11:31:18.757 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2399999999999998e-07 2024-03-19T11:31:18.757 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.757 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.757 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:18.757 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265126+0000", 2024-03-19T11:31:18.757 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4870000000000002e-06 2024-03-19T11:31:18.757 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.757 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.757 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:18.757 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265134+0000", 2024-03-19T11:31:18.757 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9740000000000002e-06 2024-03-19T11:31:18.757 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.757 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.757 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:18.758 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265143+0000", 2024-03-19T11:31:18.758 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7530000000000007e-06 2024-03-19T11:31:18.758 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.758 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.758 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.758 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.273765+0000", 2024-03-19T11:31:18.758 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0086213880000000007 2024-03-19T11:31:18.758 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.758 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.758 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:18.758 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.273872+0000", 2024-03-19T11:31:18.758 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000107452 2024-03-19T11:31:18.758 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.758 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.758 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:18.759 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.274780+0000", 2024-03-19T11:31:18.759 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00090762000000000004 2024-03-19T11:31:18.759 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.759 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:18.759 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.759 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.759 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.759 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2046 4.4 4:21eac9b2:::smithi03829856-4:head [read 0~2537857,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:31:18.759 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:46.265152+0000", 2024-03-19T11:31:18.759 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 32.399677390000001, 2024-03-19T11:31:18.759 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013074417, 2024-03-19T11:31:18.759 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:18.759 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:18.759 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:18.759 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:18.760 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:18.760 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2046 2024-03-19T11:31:18.760 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.760 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:18.760 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.760 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:18.760 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265152+0000", 2024-03-19T11:31:18.760 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.760 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.760 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.760 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:18.760 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265152+0000", 2024-03-19T11:31:18.760 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.760 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.760 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.761 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:18.761 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265152+0000", 2024-03-19T11:31:18.761 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8999999999999997e-07 2024-03-19T11:31:18.761 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.761 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.761 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:18.761 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265155+0000", 2024-03-19T11:31:18.761 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0920000000000002e-06 2024-03-19T11:31:18.761 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.761 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.761 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:18.761 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265162+0000", 2024-03-19T11:31:18.761 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6730000000000003e-06 2024-03-19T11:31:18.761 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.761 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.762 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:18.762 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265167+0000", 2024-03-19T11:31:18.762 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1839999999999998e-06 2024-03-19T11:31:18.762 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.762 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.762 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.762 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.276218+0000", 2024-03-19T11:31:18.762 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011050559999999999 2024-03-19T11:31:18.762 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.762 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.762 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:18.762 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.276345+0000", 2024-03-19T11:31:18.762 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000127243 2024-03-19T11:31:18.763 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.763 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.763 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:18.763 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.278226+0000", 2024-03-19T11:31:18.763 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0018811749999999999 2024-03-19T11:31:18.763 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.763 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:18.763 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.763 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.763 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:18.763 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:18.765 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-19T11:31:18.765 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:18.773+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:18.812 INFO:tasks.rados.rados.0.smithi038.stdout:1350: got (0) Success 2024-03-19T11:31:18.812 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 38 v 104 (ObjNum 194 snap 22 seq_num 194) dirty exists 2024-03-19T11:31:18.812 INFO:tasks.rados.rados.0.smithi038.stdout:1349: finishing set_chunk to oid 6 2024-03-19T11:31:18.812 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 6 v 87 (ObjNum 195 snap 22 seq_num 195) dirty exists 2024-03-19T11:31:18.812 INFO:tasks.rados.rados.0.smithi038.stdout:1355: finishing rollback tid 0 to smithi03829856-16 2024-03-19T11:31:18.812 INFO:tasks.rados.rados.0.smithi038.stdout:1355: finishing rollback tid 1 to smithi03829856-16 2024-03-19T11:31:18.812 INFO:tasks.rados.rados.0.smithi038.stdout:1355: finishing rollback tid 2 to smithi03829856-16 2024-03-19T11:31:18.812 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 16 v 121 (ObjNum 65 snap 3 seq_num 65) dirty exists 2024-03-19T11:31:18.813 INFO:tasks.rados.rados.0.smithi038.stdout:1345: done (8 left) 2024-03-19T11:31:18.813 INFO:tasks.rados.rados.0.smithi038.stdout:1347: done (7 left) 2024-03-19T11:31:18.813 INFO:tasks.rados.rados.0.smithi038.stdout:1348: done (6 left) 2024-03-19T11:31:18.813 INFO:tasks.rados.rados.0.smithi038.stdout:1349: done (5 left) 2024-03-19T11:31:18.813 INFO:tasks.rados.rados.0.smithi038.stdout:1350: done (4 left) 2024-03-19T11:31:18.813 INFO:tasks.rados.rados.0.smithi038.stdout:1351: done (3 left) 2024-03-19T11:31:18.813 INFO:tasks.rados.rados.0.smithi038.stdout:1352: done (2 left) 2024-03-19T11:31:18.813 INFO:tasks.rados.rados.0.smithi038.stdout:1353: done (1 left) 2024-03-19T11:31:18.813 INFO:tasks.rados.rados.0.smithi038.stdout:1355: done (0 left) 2024-03-19T11:31:18.813 INFO:tasks.rados.rados.0.smithi038.stdout:1356: write (excl) oid 12 current snap is 26 2024-03-19T11:31:18.813 INFO:tasks.rados.rados.0.smithi038.stdout:1356: seq_num 228 ranges {631996=531009,1792365=454736,2761854=671313} 2024-03-19T11:31:18.831 INFO:tasks.rados.rados.0.smithi038.stdout:1356: writing smithi03829856-12 from 631996 to 1163005 tid 1 2024-03-19T11:31:18.849 INFO:tasks.rados.rados.0.smithi038.stdout:1356: writing smithi03829856-12 from 1792365 to 2247101 tid 2 2024-03-19T11:31:18.865 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:18.874+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:18.867 INFO:tasks.rados.rados.0.smithi038.stdout:1356: writing smithi03829856-12 from 2761854 to 3433167 tid 3 2024-03-19T11:31:18.867 INFO:tasks.rados.rados.0.smithi038.stdout:1357: write (excl) oid 11 current snap is 26 2024-03-19T11:31:18.867 INFO:tasks.rados.rados.0.smithi038.stdout:1357: seq_num 229 ranges {648803=722611,1967200=714747,3421983=426820} 2024-03-19T11:31:18.884 INFO:tasks.rados.rados.0.smithi038.stdout:1357: writing smithi03829856-11 from 648803 to 1371414 tid 1 2024-03-19T11:31:18.898 INFO:tasks.rados.rados.0.smithi038.stdout:1357: writing smithi03829856-11 from 1967200 to 2681947 tid 2 2024-03-19T11:31:18.907 INFO:tasks.rados.rados.0.smithi038.stdout:1357: writing smithi03829856-11 from 3421983 to 3848803 tid 3 2024-03-19T11:31:18.908 INFO:tasks.rados.rados.0.smithi038.stdout:1358: write oid 27 current snap is 26 2024-03-19T11:31:18.908 INFO:tasks.rados.rados.0.smithi038.stdout:1358: seq_num 230 ranges {665610=797860,1909329=458405,2816700=410210} 2024-03-19T11:31:18.918 INFO:tasks.rados.rados.0.smithi038.stdout:1358: writing smithi03829856-27 from 665610 to 1463470 tid 1 2024-03-19T11:31:18.924 INFO:tasks.rados.rados.0.smithi038.stdout:1358: writing smithi03829856-27 from 1909329 to 2367734 tid 2 2024-03-19T11:31:18.927 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:18.927 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:31:18.927 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.927 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:2940 4.1a e99/83 4:5c0af55a:::smithi03829856-27:head v 99'115, mlcod=89'111)", 2024-03-19T11:31:18.927 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:18.933600+0000", 2024-03-19T11:31:18.927 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.0059173239999999998, 2024-03-19T11:31:18.927 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0059598960000000001, 2024-03-19T11:31:18.927 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:18.927 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:18.927 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:18.927 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.928 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:18.928 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.933600+0000", 2024-03-19T11:31:18.928 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.928 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.928 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.928 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:18.928 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.933600+0000", 2024-03-19T11:31:18.928 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:18.928 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.929 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.929 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:18.929 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.933602+0000", 2024-03-19T11:31:18.929 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0530000000000001e-06 2024-03-19T11:31:18.929 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.929 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.929 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:18.929 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.935157+0000", 2024-03-19T11:31:18.929 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0015555949999999999 2024-03-19T11:31:18.929 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.929 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.929 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:18.929 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.935189+0000", 2024-03-19T11:31:18.929 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1535999999999998e-05 2024-03-19T11:31:18.929 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.930 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.930 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:18.930 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.935204+0000", 2024-03-19T11:31:18.930 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5187e-05 2024-03-19T11:31:18.930 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.930 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.930 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:18.930 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.935390+0000", 2024-03-19T11:31:18.930 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000185513 2024-03-19T11:31:18.930 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:18.931 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:18.931 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:18.931 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.935434+0000", 2024-03-19T11:31:18.931 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3992000000000002e-05 2024-03-19T11:31:18.931 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.931 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:18.931 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.931 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:18.931 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-19T11:31:18.931 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 1 2024-03-19T11:31:18.931 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:18.933 INFO:tasks.rados.rados.0.smithi038.stdout:1358: writing smithi03829856-27 from 2816700 to 3226910 tid 3 2024-03-19T11:31:18.933 INFO:tasks.rados.rados.0.smithi038.stdout:1359: read oid 38 snap 22 2024-03-19T11:31:18.933 INFO:tasks.rados.rados.0.smithi038.stdout:1359: expect (ObjNum 194 snap 22 seq_num 194) 2024-03-19T11:31:18.933 INFO:tasks.rados.rados.0.smithi038.stdout:1360: rollback oid 42 current snap is 26 2024-03-19T11:31:18.933 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 42 to 25 2024-03-19T11:31:18.933 INFO:tasks.rados.rados.0.smithi038.stdout:1356: finishing write tid 1 to smithi03829856-12 2024-03-19T11:31:18.933 INFO:tasks.rados.rados.0.smithi038.stdout:1361: rollback oid 25 current snap is 26 2024-03-19T11:31:18.934 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 25 to 25 2024-03-19T11:31:18.934 INFO:tasks.rados.rados.0.smithi038.stdout:1356: finishing write tid 2 to smithi03829856-12 2024-03-19T11:31:18.934 INFO:tasks.rados.rados.0.smithi038.stdout:1356: finishing write tid 3 to smithi03829856-12 2024-03-19T11:31:18.934 INFO:tasks.rados.rados.0.smithi038.stdout:1356: finishing write tid 4 to smithi03829856-12 2024-03-19T11:31:18.934 INFO:tasks.rados.rados.0.smithi038.stdout:1356: finishing write tid 5 to smithi03829856-12 2024-03-19T11:31:18.934 INFO:tasks.rados.rados.0.smithi038.stdout:1356: oid 12 updating version 0 to 104 2024-03-19T11:31:18.934 INFO:tasks.rados.rados.0.smithi038.stdout:1356: oid 12 updating version 104 to 105 2024-03-19T11:31:18.934 INFO:tasks.rados.rados.0.smithi038.stdout:1356: oid 12 version 105 is already newer than 102 2024-03-19T11:31:18.934 INFO:tasks.rados.rados.0.smithi038.stdout:1356: oid 12 version 105 is already newer than 103 2024-03-19T11:31:18.934 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 12 v 105 (ObjNum 228 snap 26 seq_num 228) dirty exists 2024-03-19T11:31:18.934 INFO:tasks.rados.rados.0.smithi038.stdout:1356: left oid 12 (ObjNum 228 snap 26 seq_num 228) 2024-03-19T11:31:18.935 INFO:tasks.rados.rados.0.smithi038.stdout:1357: finishing write tid 1 to smithi03829856-11 2024-03-19T11:31:18.935 INFO:tasks.rados.rados.0.smithi038.stdout:1357: finishing write tid 2 to smithi03829856-11 2024-03-19T11:31:18.935 INFO:tasks.rados.rados.0.smithi038.stdout:1357: finishing write tid 3 to smithi03829856-11 2024-03-19T11:31:18.935 INFO:tasks.rados.rados.0.smithi038.stdout:1357: finishing write tid 4 to smithi03829856-11 2024-03-19T11:31:18.935 INFO:tasks.rados.rados.0.smithi038.stdout:1357: finishing write tid 5 to smithi03829856-11 2024-03-19T11:31:18.935 INFO:tasks.rados.rados.0.smithi038.stdout:1357: oid 11 updating version 0 to 107 2024-03-19T11:31:18.935 INFO:tasks.rados.rados.0.smithi038.stdout:1357: oid 11 version 107 is already newer than 106 2024-03-19T11:31:18.935 INFO:tasks.rados.rados.0.smithi038.stdout:1357: oid 11 version 107 is already newer than 105 2024-03-19T11:31:18.935 INFO:tasks.rados.rados.0.smithi038.stdout:1357: oid 11 version 107 is already newer than 104 2024-03-19T11:31:18.935 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 11 v 107 (ObjNum 229 snap 26 seq_num 229) dirty exists 2024-03-19T11:31:18.935 INFO:tasks.rados.rados.0.smithi038.stdout:1357: left oid 11 (ObjNum 229 snap 26 seq_num 229) 2024-03-19T11:31:18.935 INFO:tasks.rados.rados.0.smithi038.stdout:1356: done (5 left) 2024-03-19T11:31:18.935 INFO:tasks.rados.rados.0.smithi038.stdout:1357: done (4 left) 2024-03-19T11:31:18.935 INFO:tasks.rados.rados.0.smithi038.stdout:1362: snap_create 2024-03-19T11:31:18.936 INFO:tasks.rados.rados.0.smithi038.stdout:1358: finishing write tid 1 to smithi03829856-27 2024-03-19T11:31:18.936 INFO:tasks.rados.rados.0.smithi038.stdout:1360: finishing rollback tid 1 to smithi03829856-42 2024-03-19T11:31:18.937 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 42 v 54 (ObjNum 97 snap 0 seq_num 1) dirty dne 2024-03-19T11:31:18.939 INFO:tasks.rados.rados.0.smithi038.stdout:1359: expect (ObjNum 194 snap 22 seq_num 194) 2024-03-19T11:31:18.945 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-19T11:31:18.966 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:18.974+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:19.027 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-19T11:31:19.053 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-03-19T11:31:19.060 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:19.070+0000 7f923da1c640 -1 osd.7 99 handle_osd_map injecting map crc failure 2024-03-19T11:31:19.067 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:19.074+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:19.100 INFO:tasks.rados.rados.0.smithi038.stdout:1358: finishing write tid 2 to smithi03829856-27 2024-03-19T11:31:19.101 INFO:tasks.rados.rados.0.smithi038.stdout:1358: finishing write tid 3 to smithi03829856-27 2024-03-19T11:31:19.101 INFO:tasks.rados.rados.0.smithi038.stdout:1358: finishing write tid 4 to smithi03829856-27 2024-03-19T11:31:19.101 INFO:tasks.rados.rados.0.smithi038.stdout:1358: finishing write tid 5 to smithi03829856-27 2024-03-19T11:31:19.101 INFO:tasks.rados.rados.0.smithi038.stdout:1358: oid 27 updating version 0 to 118 2024-03-19T11:31:19.101 INFO:tasks.rados.rados.0.smithi038.stdout:1358: oid 27 version 118 is already newer than 115 2024-03-19T11:31:19.101 INFO:tasks.rados.rados.0.smithi038.stdout:1358: oid 27 version 118 is already newer than 116 2024-03-19T11:31:19.101 INFO:tasks.rados.rados.0.smithi038.stdout:1358: oid 27 version 118 is already newer than 117 2024-03-19T11:31:19.101 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 27 v 118 (ObjNum 230 snap 26 seq_num 230) dirty exists 2024-03-19T11:31:19.102 INFO:tasks.rados.rados.0.smithi038.stdout:1358: left oid 27 (ObjNum 230 snap 26 seq_num 230) 2024-03-19T11:31:19.102 INFO:tasks.rados.rados.0.smithi038.stdout:1358: done (4 left) 2024-03-19T11:31:19.102 INFO:tasks.rados.rados.0.smithi038.stdout:1359: done (3 left) 2024-03-19T11:31:19.102 INFO:tasks.rados.rados.0.smithi038.stdout:1360: done (2 left) 2024-03-19T11:31:19.102 INFO:tasks.rados.rados.0.smithi038.stdout:1362: done (1 left) 2024-03-19T11:31:19.102 INFO:tasks.rados.rados.0.smithi038.stdout:1363: delete oid 8 current snap is 27 2024-03-19T11:31:19.109 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:19.109 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:31:19.109 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:31:19.110 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:31:19.110 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:19.110 INFO:tasks.rados.rados.0.smithi038.stdout:1363: done (1 left) 2024-03-19T11:31:19.110 INFO:tasks.rados.rados.0.smithi038.stdout:1364: write oid 31 current snap is 27 2024-03-19T11:31:19.111 INFO:tasks.rados.rados.0.smithi038.stdout:1364: seq_num 232 ranges {699224=548358,2026293=462074,3271546=432754} 2024-03-19T11:31:19.117 INFO:tasks.rados.rados.0.smithi038.stdout:1364: writing smithi03829856-31 from 699224 to 1247582 tid 1 2024-03-19T11:31:19.125 INFO:tasks.rados.rados.0.smithi038.stdout:1364: writing smithi03829856-31 from 2026293 to 2488367 tid 2 2024-03-19T11:31:19.128 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-19T11:31:19.134 INFO:tasks.rados.rados.0.smithi038.stdout:1364: writing smithi03829856-31 from 3271546 to 3704300 tid 3 2024-03-19T11:31:19.134 INFO:tasks.rados.rados.0.smithi038.stdout:1365: set_chunk oid 36 target oid 2024-03-19T11:31:19.134 INFO:tasks.rados.rados.0.smithi038.stdout:1365: set_chunk oid 36 offset: 0 length: 4000000 target oid smithi03829856- offset: 0 snap -1 2024-03-19T11:31:19.135 INFO:tasks.rados.rados.0.smithi038.stdout:1366: read oid 7 snap -1 2024-03-19T11:31:19.135 INFO:tasks.rados.rados.0.smithi038.stdout:1366: expect (ObjNum 210 snap 22 seq_num 210) 2024-03-19T11:31:19.135 INFO:tasks.rados.rados.0.smithi038.stdout:1367: rollback oid 16 current snap is 27 2024-03-19T11:31:19.135 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 16 to 24 2024-03-19T11:31:19.135 INFO:tasks.rados.rados.0.smithi038.stdout:1364: finishing write tid 1 to smithi03829856-31 2024-03-19T11:31:19.135 INFO:tasks.rados.rados.0.smithi038.stdout:1364: finishing write tid 2 to smithi03829856-31 2024-03-19T11:31:19.135 INFO:tasks.rados.rados.0.smithi038.stdout:1368: read oid 44 snap -1 2024-03-19T11:31:19.135 INFO:tasks.rados.rados.0.smithi038.stdout:1368: expect (ObjNum 108 snap 11 seq_num 108) 2024-03-19T11:31:19.135 INFO:tasks.rados.rados.0.smithi038.stdout:1369: set_chunk oid 19 target oid 2024-03-19T11:31:19.143 INFO:tasks.rados.rados.0.smithi038.stdout:1369: set_chunk oid 19 offset: 192000 length: 455168 target oid 182a56298b72a82631fe87d3b0100120f4453d301022e028ba2f79dfae714e69 offset: 0 snap 26 2024-03-19T11:31:19.143 INFO:tasks.rados.rados.0.smithi038.stdout:1370: read oid 40 snap 25 2024-03-19T11:31:19.143 INFO:tasks.rados.rados.0.smithi038.stdout:1370: expect (ObjNum 141 snap 15 seq_num 141) 2024-03-19T11:31:19.143 INFO:tasks.rados.rados.0.smithi038.stdout:1371: read oid 47 snap -1 2024-03-19T11:31:19.143 INFO:tasks.rados.rados.0.smithi038.stdout:1371: expect (ObjNum 46 snap 0 seq_num 46) 2024-03-19T11:31:19.143 INFO:tasks.rados.rados.0.smithi038.stdout:1372: read oid 45 snap -1 2024-03-19T11:31:19.144 INFO:tasks.rados.rados.0.smithi038.stdout:1372: expect (ObjNum 212 snap 22 seq_num 212) 2024-03-19T11:31:19.144 INFO:tasks.rados.rados.0.smithi038.stdout:1373: write oid 38 current snap is 27 2024-03-19T11:31:19.144 INFO:tasks.rados.rados.0.smithi038.stdout:1373: seq_num 233 ranges {716031=623607,2084775=605732,3298969=617062} 2024-03-19T11:31:19.151 INFO:tasks.rados.rados.0.smithi038.stdout:1373: writing smithi03829856-38 from 716031 to 1339638 tid 1 2024-03-19T11:31:19.159 INFO:tasks.rados.rados.0.smithi038.stdout:1373: writing smithi03829856-38 from 2084775 to 2690507 tid 2 2024-03-19T11:31:19.167 INFO:tasks.rados.rados.0.smithi038.stdout:1373: writing smithi03829856-38 from 3298969 to 3916031 tid 3 2024-03-19T11:31:19.168 INFO:tasks.rados.rados.0.smithi038.stdout:1374: set_chunk oid 39 target oid 2024-03-19T11:31:19.168 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:19.178+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:19.176 INFO:tasks.rados.rados.0.smithi038.stdout:1374: set_chunk oid 39 offset: 1139712 length: 269824 target oid 1c9c0e4425d0aa8cd88f2b38ad0e1f22c6d28c7797a12f88069d8a0a4d5a870f offset: 0 snap 21 2024-03-19T11:31:19.176 INFO:tasks.rados.rados.0.smithi038.stdout:1375: read oid 14 snap -1 2024-03-19T11:31:19.176 INFO:tasks.rados.rados.0.smithi038.stdout:1375: expect (ObjNum 225 snap 25 seq_num 225) 2024-03-19T11:31:19.176 INFO:tasks.rados.rados.0.smithi038.stdout:1376: read oid 8 snap -1 2024-03-19T11:31:19.176 INFO:tasks.rados.rados.0.smithi038.stdout:1376: expect deleted 2024-03-19T11:31:19.176 INFO:tasks.rados.rados.0.smithi038.stdout:1377: snap_create 2024-03-19T11:31:19.176 INFO:tasks.rados.rados.0.smithi038.stdout:1365: finishing set_chunk to oid 36 2024-03-19T11:31:19.176 INFO:tasks.rados.rados.0.smithi038.stdout:1365: got expected ERANGE (src dne) 2024-03-19T11:31:19.176 INFO:tasks.rados.rados.0.smithi038.stdout:1364: finishing write tid 3 to smithi03829856-31 2024-03-19T11:31:19.176 INFO:tasks.rados.rados.0.smithi038.stdout:1364: finishing write tid 4 to smithi03829856-31 2024-03-19T11:31:19.176 INFO:tasks.rados.rados.0.smithi038.stdout:1364: finishing write tid 5 to smithi03829856-31 2024-03-19T11:31:19.176 INFO:tasks.rados.rados.0.smithi038.stdout:1364: oid 31 updating version 0 to 139 2024-03-19T11:31:19.176 INFO:tasks.rados.rados.0.smithi038.stdout:1364: oid 31 version 139 is already newer than 138 2024-03-19T11:31:19.177 INFO:tasks.rados.rados.0.smithi038.stdout:1364: oid 31 version 139 is already newer than 137 2024-03-19T11:31:19.177 INFO:tasks.rados.rados.0.smithi038.stdout:1364: oid 31 updating version 139 to 140 2024-03-19T11:31:19.177 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 31 v 140 (ObjNum 232 snap 27 seq_num 232) dirty exists 2024-03-19T11:31:19.177 INFO:tasks.rados.rados.0.smithi038.stdout:1364: left oid 31 (ObjNum 232 snap 27 seq_num 232) 2024-03-19T11:31:19.177 INFO:tasks.rados.rados.0.smithi038.stdout:1367: finishing rollback tid 0 to smithi03829856-16 2024-03-19T11:31:19.177 INFO:tasks.rados.rados.0.smithi038.stdout:1367: finishing rollback tid 1 to smithi03829856-16 2024-03-19T11:31:19.177 INFO:tasks.rados.rados.0.smithi038.stdout:1367: finishing rollback tid 2 to smithi03829856-16 2024-03-19T11:31:19.177 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 16 v 125 (ObjNum 65 snap 3 seq_num 65) dirty exists 2024-03-19T11:31:19.177 INFO:tasks.rados.rados.0.smithi038.stdout:1370: expect (ObjNum 141 snap 15 seq_num 141) 2024-03-19T11:31:19.211 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:31:19.211 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:31:19.229 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:31:19.229 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:31:19.245 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:31:19.245 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:31:19.261 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:31:19.262 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:31:19.268 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:19.277+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:19.271 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:19.271 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:19.284 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-19T11:31:19.293 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:31:19.293 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:31:19.312 INFO:tasks.rados.rados.0.smithi038.stdout:1369: finishing set_chunk to oid 19 2024-03-19T11:31:19.312 INFO:tasks.rados.rados.0.smithi038.stdout:1371: expect (ObjNum 46 snap 0 seq_num 46) 2024-03-19T11:31:19.328 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:31:19.329 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:31:19.346 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:31:19.346 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:31:19.363 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:31:19.364 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:31:19.369 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:19.378+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:19.403 INFO:tasks.rados.rados.0.smithi038.stdout:1368: expect (ObjNum 108 snap 11 seq_num 108) 2024-03-19T11:31:19.417 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:19.418 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:19.431 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-19T11:31:19.471 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:19.477+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:19.571 INFO:tasks.rados.rados.0.smithi038.stdout:1366: expect (ObjNum 210 snap 22 seq_num 210) 2024-03-19T11:31:19.572 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:19.582+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:19.592 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:19.593 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:19.607 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-19T11:31:19.672 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:19.681+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:19.768 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:19.768 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:19.770 INFO:tasks.rados.rados.0.smithi038.stdout:1373: finishing write tid 1 to smithi03829856-38 2024-03-19T11:31:19.770 INFO:tasks.rados.rados.0.smithi038.stdout:1373: finishing write tid 2 to smithi03829856-38 2024-03-19T11:31:19.771 INFO:tasks.rados.rados.0.smithi038.stdout:1372: expect (ObjNum 212 snap 22 seq_num 212) 2024-03-19T11:31:19.773 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:19.781+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:19.782 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-19T11:31:19.874 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:19.882+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:19.901 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:19.901 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:19.915 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-19T11:31:20.003 INFO:tasks.rados.rados.0.smithi038.stdout:1373: finishing write tid 3 to smithi03829856-38 2024-03-19T11:31:20.003 INFO:tasks.rados.rados.0.smithi038.stdout:1373: finishing write tid 4 to smithi03829856-38 2024-03-19T11:31:20.003 INFO:tasks.rados.rados.0.smithi038.stdout:1373: finishing write tid 5 to smithi03829856-38 2024-03-19T11:31:20.004 INFO:tasks.rados.rados.0.smithi038.stdout:1373: oid 38 updating version 0 to 133 2024-03-19T11:31:20.004 INFO:tasks.rados.rados.0.smithi038.stdout:1373: oid 38 version 133 is already newer than 130 2024-03-19T11:31:20.004 INFO:tasks.rados.rados.0.smithi038.stdout:1373: oid 38 version 133 is already newer than 132 2024-03-19T11:31:20.005 INFO:tasks.rados.rados.0.smithi038.stdout:1373: oid 38 version 133 is already newer than 131 2024-03-19T11:31:20.005 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 38 v 133 (ObjNum 233 snap 27 seq_num 233) dirty exists 2024-03-19T11:31:20.005 INFO:tasks.rados.rados.0.smithi038.stdout:1373: left oid 38 (ObjNum 233 snap 27 seq_num 233) 2024-03-19T11:31:20.005 INFO:tasks.rados.rados.0.smithi038.stdout:1374: finishing set_chunk to oid 39 2024-03-19T11:31:20.005 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 39 set_chunk returned error code -95 offset: 1139712 length: 269824 2024-03-19T11:31:20.006 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 39 v 0 (ObjNum 163 snap 17 seq_num 163) dirty exists 2024-03-19T11:31:20.006 INFO:tasks.rados.rados.0.smithi038.stdout:1375: expect (ObjNum 225 snap 25 seq_num 225) 2024-03-19T11:31:20.008 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:20.018+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:20.054 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:20.061+0000 7fa5e08db640 -1 osd.3 100 handle_osd_map injecting map crc failure 2024-03-19T11:31:20.061 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:20.070+0000 7f923da1c640 -1 osd.7 100 handle_osd_map injecting map crc failure 2024-03-19T11:31:20.076 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:20.086+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:20.085 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:20.086 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:20.088 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-19T11:31:20.088 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-19T11:31:20.088 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_0 2024-03-19T11:31:20.088 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T11:31:20.100 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-19T11:31:20.187 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:20.198+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:20.267 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:20.267 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:20.278 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:20.286+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:20.281 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-19T11:31:20.303 INFO:tasks.rados.rados.0.smithi038.stdout:1361: finishing rollback tid 1 to smithi03829856-25 2024-03-19T11:31:20.303 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 25 v 60 (ObjNum 97 snap 0 seq_num 282765376) dirty dne 2024-03-19T11:31:20.304 INFO:tasks.rados.rados.0.smithi038.stdout:1361: done (14 left) 2024-03-19T11:31:20.304 INFO:tasks.rados.rados.0.smithi038.stdout:1364: done (13 left) 2024-03-19T11:31:20.304 INFO:tasks.rados.rados.0.smithi038.stdout:1365: done (12 left) 2024-03-19T11:31:20.304 INFO:tasks.rados.rados.0.smithi038.stdout:1366: done (11 left) 2024-03-19T11:31:20.304 INFO:tasks.rados.rados.0.smithi038.stdout:1367: done (10 left) 2024-03-19T11:31:20.304 INFO:tasks.rados.rados.0.smithi038.stdout:1368: done (9 left) 2024-03-19T11:31:20.304 INFO:tasks.rados.rados.0.smithi038.stdout:1369: done (8 left) 2024-03-19T11:31:20.304 INFO:tasks.rados.rados.0.smithi038.stdout:1370: done (7 left) 2024-03-19T11:31:20.305 INFO:tasks.rados.rados.0.smithi038.stdout:1371: done (6 left) 2024-03-19T11:31:20.305 INFO:tasks.rados.rados.0.smithi038.stdout:1372: done (5 left) 2024-03-19T11:31:20.305 INFO:tasks.rados.rados.0.smithi038.stdout:1373: done (4 left) 2024-03-19T11:31:20.305 INFO:tasks.rados.rados.0.smithi038.stdout:1374: done (3 left) 2024-03-19T11:31:20.305 INFO:tasks.rados.rados.0.smithi038.stdout:1375: done (2 left) 2024-03-19T11:31:20.305 INFO:tasks.rados.rados.0.smithi038.stdout:1376: done (1 left) 2024-03-19T11:31:20.305 INFO:tasks.rados.rados.0.smithi038.stdout:1377: done (0 left) 2024-03-19T11:31:20.305 INFO:tasks.rados.rados.0.smithi038.stdout:1378: write (excl) oid 49 current snap is 28 2024-03-19T11:31:20.305 INFO:tasks.rados.rados.0.smithi038.stdout:1378: seq_num 234 ranges {732838=698856,2026904=465743,3210039=571651} 2024-03-19T11:31:20.313 INFO:tasks.rados.rados.0.smithi038.stdout:1378: writing smithi03829856-49 from 732838 to 1431694 tid 1 2024-03-19T11:31:20.321 INFO:tasks.rados.rados.0.smithi038.stdout:1378: writing smithi03829856-49 from 2026904 to 2492647 tid 2 2024-03-19T11:31:20.331 INFO:tasks.rados.rados.0.smithi038.stdout:1378: writing smithi03829856-49 from 3210039 to 3781690 tid 3 2024-03-19T11:31:20.331 INFO:tasks.rados.rados.0.smithi038.stdout:1379: set_chunk oid 21 target oid 2024-03-19T11:31:20.340 INFO:tasks.rados.rados.0.smithi038.stdout:1379: set_chunk oid 21 offset: 390144 length: 535040 target oid 2e2aeeb32103d546f4116672f718eda59131c1ed12939207b0092e8d6ba7afb3 offset: 0 snap -1 2024-03-19T11:31:20.340 INFO:tasks.rados.rados.0.smithi038.stdout:1380: read oid 25 snap 27 2024-03-19T11:31:20.340 INFO:tasks.rados.rados.0.smithi038.stdout:1380: expect deleted 2024-03-19T11:31:20.340 INFO:tasks.rados.rados.0.smithi038.stdout:1381: read oid 41 snap -1 2024-03-19T11:31:20.340 INFO:tasks.rados.rados.0.smithi038.stdout:1381: expect (ObjNum 170 snap 17 seq_num 170) 2024-03-19T11:31:20.341 INFO:tasks.rados.rados.0.smithi038.stdout:1382: delete oid 39 current snap is 28 2024-03-19T11:31:20.341 INFO:tasks.rados.rados.0.smithi038.stdout:1378: finishing write tid 1 to smithi03829856-49 2024-03-19T11:31:20.341 INFO:tasks.rados.rados.0.smithi038.stdout:1378: finishing write tid 2 to smithi03829856-49 2024-03-19T11:31:20.341 INFO:tasks.rados.rados.0.smithi038.stdout:1378: finishing write tid 3 to smithi03829856-49 2024-03-19T11:31:20.341 INFO:tasks.rados.rados.0.smithi038.stdout:1378: finishing write tid 4 to smithi03829856-49 2024-03-19T11:31:20.341 INFO:tasks.rados.rados.0.smithi038.stdout:1378: finishing write tid 5 to smithi03829856-49 2024-03-19T11:31:20.341 INFO:tasks.rados.rados.0.smithi038.stdout:1378: oid 49 updating version 0 to 45 2024-03-19T11:31:20.341 INFO:tasks.rados.rados.0.smithi038.stdout:1378: oid 49 version 45 is already newer than 42 2024-03-19T11:31:20.341 INFO:tasks.rados.rados.0.smithi038.stdout:1378: oid 49 version 45 is already newer than 44 2024-03-19T11:31:20.342 INFO:tasks.rados.rados.0.smithi038.stdout:1378: oid 49 version 45 is already newer than 43 2024-03-19T11:31:20.342 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 49 v 45 (ObjNum 234 snap 28 seq_num 234) dirty exists 2024-03-19T11:31:20.342 INFO:tasks.rados.rados.0.smithi038.stdout:1378: left oid 49 (ObjNum 234 snap 28 seq_num 234) 2024-03-19T11:31:20.345 INFO:tasks.rados.rados.0.smithi038.stdout:1379: finishing set_chunk to oid 21 2024-03-19T11:31:20.345 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 21 v 168 (ObjNum 129 snap 15 seq_num 129) dirty exists 2024-03-19T11:31:20.346 INFO:tasks.rados.rados.0.smithi038.stdout:1378: done (4 left) 2024-03-19T11:31:20.347 INFO:tasks.rados.rados.0.smithi038.stdout:1379: done (3 left) 2024-03-19T11:31:20.347 INFO:tasks.rados.rados.0.smithi038.stdout:1380: done (2 left) 2024-03-19T11:31:20.347 INFO:tasks.rados.rados.0.smithi038.stdout:1382: done (1 left) 2024-03-19T11:31:20.347 INFO:tasks.rados.rados.0.smithi038.stdout:1383: read oid 2 snap -1 2024-03-19T11:31:20.347 INFO:tasks.rados.rados.0.smithi038.stdout:1383: expect (ObjNum 206 snap 22 seq_num 206) 2024-03-19T11:31:20.347 INFO:tasks.rados.rados.0.smithi038.stdout:1384: write oid 37 current snap is 28 2024-03-19T11:31:20.347 INFO:tasks.rados.rados.0.smithi038.stdout:1381: expect (ObjNum 170 snap 17 seq_num 170) 2024-03-19T11:31:20.379 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:20.385+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:20.387 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:31:20.388 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-03-19T11:31:20.408 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":143,"stamp":"2024-03-19T11:31:20.158975+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":465096157,"num_objects":297,"num_object_clones":87,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":294,"num_whiteouts":8,"num_read":1720,"num_read_kb":1517007,"num_write":975,"num_write_kb":318373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":257,"num_bytes_recovered":270645919,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":96,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3088,"ondisk_log_size":3088,"up":106,"acting":106,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":825448,"kb_used_data":575996,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754149272,"statfs":{"total":773094113280,"available":772248854528,"internally_reserved":0,"allocated":589819904,"data_stored":682054017,"data_compressed":16860464,"data_compressed_allocated":17264640,"data_compressed_original":38547456,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24988119,"num_objects":10,"num_object_clones":5,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":208,"num_read_kb":257837,"num_write":70,"num_write_kb":21847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":5842489,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":7,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001602"},"pg_stats":[{"pgid":"4.27","version":"99'63","reported_seq":126,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-19T11:31:19.138212+0000","last_change":"2024-03-19T11:31:18.082778+0000","last_active":"2024-03-19T11:31:19.138212+0000","last_peered":"2024-03-19T11:31:19.138212+0000","last_clean":"2024-03-19T11:31:19.138212+0000","last_became_active":"2024-03-19T11:31:00.070337+0000","last_became_peered":"2024-03-19T11:31:00.070337+0000","last_unstale":"2024-03-19T11:31:19.138212+0000","last_undegraded":"2024-03-19T11:31:19.138212+0000","last_fullsized":"2024-03-19T11:31:19.138212+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":6,"last_scrub":"71'48","last_scrub_stamp":"2024-03-19T11:30:55.467716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:55.467716+0000","objects_scrubbed":4,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:18.176662+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.016482045000000001,"stat_sum":{"num_bytes":7842391,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":11199,"num_write":5,"num_write_kb":1808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":7070511,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"15","length":"2"},{"start":"19","length":"1"}]},{"pgid":"4.26","version":"61'63","reported_seq":89,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-19T11:31:17.113982+0000","last_change":"2024-03-19T11:31:14.073037+0000","last_active":"2024-03-19T11:31:17.113982+0000","last_peered":"2024-03-19T11:31:17.113982+0000","last_clean":"2024-03-19T11:31:17.113982+0000","last_became_active":"2024-03-19T11:30:59.074337+0000","last_became_peered":"2024-03-19T11:30:59.074337+0000","last_unstale":"2024-03-19T11:31:17.113982+0000","last_undegraded":"2024-03-19T11:31:17.113982+0000","last_fullsized":"2024-03-19T11:31:17.113982+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":6,"last_scrub":"61'63","last_scrub_stamp":"2024-03-19T11:30:52.406723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:52.406723+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:09.398738+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0015434979999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"},{"start":"19","length":"1"}]},{"pgid":"4.25","version":"56'66","reported_seq":126,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-19T11:31:19.070070+0000","last_change":"2024-03-19T11:31:18.056903+0000","last_active":"2024-03-19T11:31:19.070070+0000","last_peered":"2024-03-19T11:31:19.070070+0000","last_clean":"2024-03-19T11:31:19.070070+0000","last_became_active":"2024-03-19T11:30:53.084473+0000","last_became_peered":"2024-03-19T11:30:53.084473+0000","last_unstale":"2024-03-19T11:31:19.070070+0000","last_undegraded":"2024-03-19T11:31:19.070070+0000","last_fullsized":"2024-03-19T11:31:19.070070+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":6,"last_scrub":"56'66","last_scrub_stamp":"2024-03-19T11:31:06.587201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:31:06.587201+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:28.749498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001486933,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"15","length":"2"},{"start":"19","length":"1"}]},{"pgid":"4.24","version":"100'125","reported_seq":557,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-19T11:31:19.154121+0000","last_change":"2024-03-19T11:31:18.079593+0000","last_active":"2024-03-19T11:31:19.154121+0000","last_peered":"2024-03-19T11:31:19.154121+0000","last_clean":"2024-03-19T11:31:19.154121+0000","last_became_active":"2024-03-19T11:31:03.063705+0000","last_became_peered":"2024-03-19T11:31:03.063705+0000","last_unstale":"2024-03-19T11:31:19.154121+0000","last_undegraded":"2024-03-19T11:31:19.154121+0000","last_fullsized":"2024-03-19T11:31:19.154121+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":6,"last_scrub":"84'113","last_scrub_stamp":"2024-03-19T11:31:03.304881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:31:03.304881+0000","objects_scrubbed":5,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:04.322668+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.018434019999999999,"stat_sum":{"num_bytes":12457167,"num_objects":6,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":36575,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":6,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"15","length":"2"},{"start":"19","length":"1"}]},{"pgid":"4.23","version":"95'40","reported_seq":136,"reported_epoch":100,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:19.133123+0000","last_change":"2024-03-19T11:31:18.136973+0000","last_active":"2024-03-19T11:31:19.133123+0000","last_peered":"2024-03-19T11:31:19.133123+0000","last_clean":"2024-03-19T11:31:19.133123+0000","last_became_active":"2024-03-19T11:30:53.076092+0000","last_became_peered":"2024-03-19T11:30:53.076092+0000","last_unstale":"2024-03-19T11:31:19.133123+0000","last_undegraded":"2024-03-19T11:31:19.133123+0000","last_fullsized":"2024-03-19T11:31:19.133123+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":6,"last_scrub":"62'23","last_scrub_stamp":"2024-03-19T11:30:43.163509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:43.163509+0000","objects_scrubbed":2,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:54.656322+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.086548156000000001,"stat_sum":{"num_bytes":6154918,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":19,"num_read_kb":27064,"num_write":10,"num_write_kb":2870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2833035,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d","length":"1"},{"start":"11","length":"3"}]},{"pgid":"4.22","version":"28'6","reported_seq":123,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-19T11:31:18.153168+0000","last_change":"2024-03-19T11:31:18.155053+0000","last_active":"2024-03-19T11:31:18.153168+0000","last_peered":"2024-03-19T11:31:18.153168+0000","last_clean":"2024-03-19T11:31:18.153168+0000","last_became_active":"2024-03-19T11:30:53.068010+0000","last_became_peered":"2024-03-19T11:30:53.068010+0000","last_unstale":"2024-03-19T11:31:18.153168+0000","last_undegraded":"2024-03-19T11:31:18.153168+0000","last_fullsized":"2024-03-19T11:31:18.153168+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":6,"last_scrub":"28'6","last_scrub_stamp":"2024-03-19T11:30:55.724671+0000","last_deep_scrub":"28'6","last_deep_scrub_stamp":"2024-03-19T11:30:55.724671+0000","last_clean_scrub_stamp":"2024-03-19T11:30:55.724671+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:57.399261+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0025233019999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"15","length":"2"},{"start":"19","length":"1"}]},{"pgid":"4.21","version":"100'140","reported_seq":580,"reported_epoch":100,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:19.150106+0000","last_change":"2024-03-19T11:31:18.135992+0000","last_active":"2024-03-19T11:31:19.150106+0000","last_peered":"2024-03-19T11:31:19.150106+0000","last_clean":"2024-03-19T11:31:19.150106+0000","last_became_active":"2024-03-19T11:31:00.069759+0000","last_became_peered":"2024-03-19T11:31:00.069759+0000","last_unstale":"2024-03-19T11:31:19.150106+0000","last_undegraded":"2024-03-19T11:31:19.150106+0000","last_fullsized":"2024-03-19T11:31:19.150106+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":6,"last_scrub":"62'96","last_scrub_stamp":"2024-03-19T11:30:43.400885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:43.400885+0000","objects_scrubbed":9,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:03.708671+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.086175081000000001,"stat_sum":{"num_bytes":17197075,"num_objects":7,"num_object_clones":5,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":76,"num_read_kb":92171,"num_write":28,"num_write_kb":7934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":22983003,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d","length":"1"},{"start":"11","length":"2"}]},{"pgid":"4.20","version":"56'46","reported_seq":115,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-19T11:31:18.165971+0000","last_change":"2024-03-19T11:31:18.167620+0000","last_active":"2024-03-19T11:31:18.165971+0000","last_peered":"2024-03-19T11:31:18.165971+0000","last_clean":"2024-03-19T11:31:18.165971+0000","last_became_active":"2024-03-19T11:30:53.067251+0000","last_became_peered":"2024-03-19T11:30:53.067251+0000","last_unstale":"2024-03-19T11:31:18.165971+0000","last_undegraded":"2024-03-19T11:31:18.165971+0000","last_fullsized":"2024-03-19T11:31:18.165971+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":6,"last_scrub":"56'46","last_scrub_stamp":"2024-03-19T11:30:41.767100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:41.767100+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:11.103961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002371859,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"15","length":"2"},{"start":"19","length":"1"}]},{"pgid":"4.1f","version":"0'0","reported_seq":123,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-19T11:31:18.165639+0000","last_change":"2024-03-19T11:31:18.167836+0000","last_active":"2024-03-19T11:31:18.165639+0000","last_peered":"2024-03-19T11:31:18.165639+0000","last_clean":"2024-03-19T11:31:18.165639+0000","last_became_active":"2024-03-19T11:30:53.087211+0000","last_became_peered":"2024-03-19T11:30:53.087211+0000","last_unstale":"2024-03-19T11:31:18.165639+0000","last_undegraded":"2024-03-19T11:31:18.165639+0000","last_fullsized":"2024-03-19T11:31:18.165639+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:30:42.377963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.377963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:55.978327+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0029810819999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"15","length":"2"},{"start":"19","length":"1"}]},{"pgid":"4.1e","version":"98'76","reported_seq":140,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-19T11:31:17.139895+0000","last_change":"2024-03-19T11:31:14.073144+0000","last_active":"2024-03-19T11:31:17.139895+0000","last_peered":"2024-03-19T11:31:17.139895+0000","last_clean":"2024-03-19T11:31:17.139895+0000","last_became_active":"2024-03-19T11:30:53.084452+0000","last_became_peered":"2024-03-19T11:30:53.084452+0000","last_unstale":"2024-03-19T11:31:17.139895+0000","last_undegraded":"2024-03-19T11:31:17.139895+0000","last_fullsized":"2024-03-19T11:31:17.139895+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"64'62","last_scrub_stamp":"2024-03-19T11:30:42.778159+0000","last_deep_scrub":"64'62","last_deep_scrub_stamp":"2024-03-19T11:30:42.778159+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.778159+0000","objects_scrubbed":4,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:56.359117+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0.0015013209999999999,"stat_sum":{"num_bytes":6331994,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":29,"num_read_kb":28504,"num_write":5,"num_write_kb":1591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3779426,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"},{"start":"19","length":"1"}]},{"pgid":"4.1d","version":"97'71","reported_seq":125,"reported_epoch":100,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:19.133355+0000","last_change":"2024-03-19T11:31:18.134594+0000","last_active":"2024-03-19T11:31:19.133355+0000","last_peered":"2024-03-19T11:31:19.133355+0000","last_clean":"2024-03-19T11:31:19.133355+0000","last_became_active":"2024-03-19T11:30:53.067748+0000","last_became_peered":"2024-03-19T11:30:53.067748+0000","last_unstale":"2024-03-19T11:31:19.133355+0000","last_undegraded":"2024-03-19T11:31:19.133355+0000","last_fullsized":"2024-03-19T11:31:19.133355+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"61'59","last_scrub_stamp":"2024-03-19T11:30:42.601863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.601863+0000","objects_scrubbed":2,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:11.268969+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.084285733000000002,"stat_sum":{"num_bytes":2067261,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":12,"num_read_kb":9169,"num_write":5,"num_write_kb":1341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2817390,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d","length":"1"},{"start":"11","length":"2"}]},{"pgid":"4.1c","version":"85'35","reported_seq":141,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-19T11:31:19.069722+0000","last_change":"2024-03-19T11:31:18.057122+0000","last_active":"2024-03-19T11:31:19.069722+0000","last_peered":"2024-03-19T11:31:19.069722+0000","last_clean":"2024-03-19T11:31:19.069722+0000","last_became_active":"2024-03-19T11:30:53.082655+0000","last_became_peered":"2024-03-19T11:30:53.082655+0000","last_unstale":"2024-03-19T11:31:19.069722+0000","last_undegraded":"2024-03-19T11:31:19.069722+0000","last_fullsized":"2024-03-19T11:31:19.069722+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"62'31","last_scrub_stamp":"2024-03-19T11:30:42.771396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.771396+0000","objects_scrubbed":3,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:46.848482+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001676585,"stat_sum":{"num_bytes":3210104,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":51,"num_read_kb":71637,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":5034309,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"15","length":"2"},{"start":"19","length":"1"}]},{"pgid":"4.1b","version":"99'106","reported_seq":135,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-19T11:31:18.176356+0000","last_change":"2024-03-19T11:31:18.177003+0000","last_active":"2024-03-19T11:31:18.176356+0000","last_peered":"2024-03-19T11:31:18.176356+0000","last_clean":"2024-03-19T11:31:18.176356+0000","last_became_active":"2024-03-19T11:31:03.056769+0000","last_became_peered":"2024-03-19T11:31:03.056769+0000","last_unstale":"2024-03-19T11:31:18.176356+0000","last_undegraded":"2024-03-19T11:31:18.176356+0000","last_fullsized":"2024-03-19T11:31:18.176356+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":5,"last_scrub":"62'62","last_scrub_stamp":"2024-03-19T11:30:42.181424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.181424+0000","objects_scrubbed":5,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:56.692024+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.024441616999999999,"stat_sum":{"num_bytes":14818653,"num_objects":7,"num_object_clones":4,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":59,"num_read_kb":68896,"num_write":11,"num_write_kb":3105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":17966556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":6,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"15","length":"2"},{"start":"19","length":"1"}]},{"pgid":"4.1a","version":"99'113","reported_seq":130,"reported_epoch":99,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:18.183749+0000","last_change":"2024-03-19T11:31:18.179527+0000","last_active":"2024-03-19T11:31:18.183749+0000","last_peered":"2024-03-19T11:31:18.183749+0000","last_clean":"2024-03-19T11:31:18.183749+0000","last_became_active":"2024-03-19T11:31:03.056520+0000","last_became_peered":"2024-03-19T11:31:03.056520+0000","last_unstale":"2024-03-19T11:31:18.183749+0000","last_undegraded":"2024-03-19T11:31:18.183749+0000","last_fullsized":"2024-03-19T11:31:18.183749+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":5,"last_scrub":"61'76","last_scrub_stamp":"2024-03-19T11:30:41.433501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:41.433501+0000","objects_scrubbed":5,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:08.944936+0000","scrub_duration":11,"objects_trimmed":1,"snaptrim_duration":0.0029219609999999998,"stat_sum":{"num_bytes":10708822,"num_objects":4,"num_object_clones":2,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":47,"num_read_kb":52583,"num_write":18,"num_write_kb":5314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":16291222,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"},{"start":"19","length":"1"}]},{"pgid":"4.19","version":"99'144","reported_seq":572,"reported_epoch":100,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:19.133059+0000","last_change":"2024-03-19T11:31:18.137594+0000","last_active":"2024-03-19T11:31:19.133059+0000","last_peered":"2024-03-19T11:31:19.133059+0000","last_clean":"2024-03-19T11:31:19.133059+0000","last_became_active":"2024-03-19T11:30:53.076188+0000","last_became_peered":"2024-03-19T11:30:53.076188+0000","last_unstale":"2024-03-19T11:31:19.133059+0000","last_undegraded":"2024-03-19T11:31:19.133059+0000","last_fullsized":"2024-03-19T11:31:19.133059+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"41'37","last_scrub_stamp":"2024-03-19T11:30:27.705159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:27.705159+0000","objects_scrubbed":4,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:28.959732+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.078120693000000005,"stat_sum":{"num_bytes":26044000,"num_objects":13,"num_object_clones":9,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":139,"num_read_kb":155380,"num_write":56,"num_write_kb":11192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":21870393,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":11,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d","length":"1"},{"start":"11","length":"1"}]},{"pgid":"4.18","version":"99'148","reported_seq":490,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-19T11:31:18.169575+0000","last_change":"2024-03-19T11:31:18.171114+0000","last_active":"2024-03-19T11:31:18.169575+0000","last_peered":"2024-03-19T11:31:18.169575+0000","last_clean":"2024-03-19T11:31:18.169575+0000","last_became_active":"2024-03-19T11:31:06.066324+0000","last_became_peered":"2024-03-19T11:31:06.066324+0000","last_unstale":"2024-03-19T11:31:18.169575+0000","last_undegraded":"2024-03-19T11:31:18.169575+0000","last_fullsized":"2024-03-19T11:31:18.169575+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":5,"last_scrub":"48'77","last_scrub_stamp":"2024-03-19T11:30:29.399762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:29.399762+0000","objects_scrubbed":6,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:49.789577+0000","scrub_duration":3,"objects_trimmed":2,"snaptrim_duration":0.018851340000000001,"stat_sum":{"num_bytes":16194756,"num_objects":6,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":60,"num_read_kb":63788,"num_write":27,"num_write_kb":8166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":19971477,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"15","length":"2"},{"start":"19","length":"1"}]},{"pgid":"4.17","version":"41'24","reported_seq":115,"reported_epoch":100,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:19.132946+0000","last_change":"2024-03-19T11:31:18.137567+0000","last_active":"2024-03-19T11:31:19.132946+0000","last_peered":"2024-03-19T11:31:19.132946+0000","last_clean":"2024-03-19T11:31:19.132946+0000","last_became_active":"2024-03-19T11:30:53.074270+0000","last_became_peered":"2024-03-19T11:30:53.074270+0000","last_unstale":"2024-03-19T11:31:19.132946+0000","last_undegraded":"2024-03-19T11:31:19.132946+0000","last_fullsized":"2024-03-19T11:31:19.132946+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"41'24","last_scrub_stamp":"2024-03-19T11:30:53.993428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:53.993428+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:55.688780+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.087487583999999993,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d","length":"1"},{"start":"11","length":"3"}]},{"pgid":"4.16","version":"98'124","reported_seq":149,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-19T11:31:17.152054+0000","last_change":"2024-03-19T11:31:14.071827+0000","last_active":"2024-03-19T11:31:17.152054+0000","last_peered":"2024-03-19T11:31:17.152054+0000","last_clean":"2024-03-19T11:31:17.152054+0000","last_became_active":"2024-03-19T11:30:59.074063+0000","last_became_peered":"2024-03-19T11:30:59.074063+0000","last_unstale":"2024-03-19T11:31:17.152054+0000","last_undegraded":"2024-03-19T11:31:17.152054+0000","last_fullsized":"2024-03-19T11:31:17.152054+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":5,"last_scrub":"49'56","last_scrub_stamp":"2024-03-19T11:30:29.212766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:29.212766+0000","objects_scrubbed":5,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:32.080038+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00061735200000000005,"stat_sum":{"num_bytes":13644045,"num_objects":7,"num_object_clones":4,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":125,"num_read_kb":135660,"num_write":28,"num_write_kb":7745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":24165549,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":7,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"},{"start":"19","length":"1"}]},{"pgid":"4.15","version":"100'211","reported_seq":684,"reported_epoch":100,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:19.069970+0000","last_change":"2024-03-19T11:31:18.153360+0000","last_active":"2024-03-19T11:31:19.069970+0000","last_peered":"2024-03-19T11:31:19.069970+0000","last_clean":"2024-03-19T11:31:19.069970+0000","last_became_active":"2024-03-19T11:30:56.063100+0000","last_became_peered":"2024-03-19T11:30:56.063100+0000","last_unstale":"2024-03-19T11:31:19.069970+0000","last_undegraded":"2024-03-19T11:31:19.069970+0000","last_fullsized":"2024-03-19T11:31:19.069970+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":5,"last_scrub":"49'74","last_scrub_stamp":"2024-03-19T11:30:29.457542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:29.457542+0000","objects_scrubbed":7,"log_size":211,"log_dups_size":0,"ondisk_log_size":211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:55.281237+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.097702673000000004,"stat_sum":{"num_bytes":28026320,"num_objects":17,"num_object_clones":12,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":2,"num_read":176,"num_read_kb":197753,"num_write":68,"num_write_kb":17815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":18744415,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":12,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"}]},{"pgid":"4.14","version":"40'56","reported_seq":276,"reported_epoch":100,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:19.133057+0000","last_change":"2024-03-19T11:31:18.129047+0000","last_active":"2024-03-19T11:31:19.133057+0000","last_peered":"2024-03-19T11:31:19.133057+0000","last_clean":"2024-03-19T11:31:19.133057+0000","last_became_active":"2024-03-19T11:31:00.065648+0000","last_became_peered":"2024-03-19T11:31:00.065648+0000","last_unstale":"2024-03-19T11:31:19.133057+0000","last_undegraded":"2024-03-19T11:31:19.133057+0000","last_fullsized":"2024-03-19T11:31:19.133057+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":5,"last_scrub":"40'56","last_scrub_stamp":"2024-03-19T11:30:54.998282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:54.998282+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:21.562925+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.079068584999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d","length":"1"},{"start":"11","length":"2"}]},{"pgid":"4.13","version":"91'40","reported_seq":178,"reported_epoch":99,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:18.180612+0000","last_change":"2024-03-19T11:31:18.180728+0000","last_active":"2024-03-19T11:31:18.180612+0000","last_peered":"2024-03-19T11:31:18.180612+0000","last_clean":"2024-03-19T11:31:18.180612+0000","last_became_active":"2024-03-19T11:30:45.245367+0000","last_became_peered":"2024-03-19T11:30:45.245367+0000","last_unstale":"2024-03-19T11:31:18.180612+0000","last_undegraded":"2024-03-19T11:31:18.180612+0000","last_fullsized":"2024-03-19T11:31:18.180612+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":5,"last_scrub":"48'22","last_scrub_stamp":"2024-03-19T11:30:27.183294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:27.183294+0000","objects_scrubbed":2,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:52.208440+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0031083719999999999,"stat_sum":{"num_bytes":9406330,"num_objects":4,"num_object_clones":2,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":35,"num_read_kb":30733,"num_write":17,"num_write_kb":5134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":3001789,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"},{"start":"19","length":"1"}]},{"pgid":"4.12","version":"28'6","reported_seq":105,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-19T11:31:19.137684+0000","last_change":"2024-03-19T11:31:18.063686+0000","last_active":"2024-03-19T11:31:19.137684+0000","last_peered":"2024-03-19T11:31:19.137684+0000","last_clean":"2024-03-19T11:31:19.137684+0000","last_became_active":"2024-03-19T11:31:00.064380+0000","last_became_peered":"2024-03-19T11:31:00.064380+0000","last_unstale":"2024-03-19T11:31:19.137684+0000","last_undegraded":"2024-03-19T11:31:19.137684+0000","last_fullsized":"2024-03-19T11:31:19.137684+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":5,"last_scrub":"28'6","last_scrub_stamp":"2024-03-19T11:31:04.263428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:31:04.263428+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:31.052995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032519509999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"15","length":"2"},{"start":"19","length":"1"}]},{"pgid":"4.11","version":"93'91","reported_seq":248,"reported_epoch":100,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:19.132796+0000","last_change":"2024-03-19T11:31:18.134963+0000","last_active":"2024-03-19T11:31:19.132796+0000","last_peered":"2024-03-19T11:31:19.132796+0000","last_clean":"2024-03-19T11:31:19.132796+0000","last_became_active":"2024-03-19T11:30:30.052184+0000","last_became_peered":"2024-03-19T11:30:30.052184+0000","last_unstale":"2024-03-19T11:31:19.132796+0000","last_undegraded":"2024-03-19T11:31:19.132796+0000","last_fullsized":"2024-03-19T11:31:19.132796+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":5,"last_scrub":"47'60","last_scrub_stamp":"2024-03-19T11:30:30.780348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:30.780348+0000","objects_scrubbed":2,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:35.570420+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.084890249000000001,"stat_sum":{"num_bytes":7831894,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":30,"num_read_kb":34004,"num_write":19,"num_write_kb":5759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d","length":"1"},{"start":"11","length":"2"}]},{"pgid":"4.10","version":"85'34","reported_seq":405,"reported_epoch":100,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:19.132719+0000","last_change":"2024-03-19T11:31:18.136609+0000","last_active":"2024-03-19T11:31:19.132719+0000","last_peered":"2024-03-19T11:31:19.132719+0000","last_clean":"2024-03-19T11:31:19.132719+0000","last_became_active":"2024-03-19T11:31:00.065617+0000","last_became_peered":"2024-03-19T11:31:00.065617+0000","last_unstale":"2024-03-19T11:31:19.132719+0000","last_undegraded":"2024-03-19T11:31:19.132719+0000","last_fullsized":"2024-03-19T11:31:19.132719+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":5,"last_scrub":"50'30","last_scrub_stamp":"2024-03-19T11:30:28.787058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:28.787058+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:56.727717+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.085991317999999997,"stat_sum":{"num_bytes":3462209,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":30,"num_read_kb":31634,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":7102596,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d","length":"1"},{"start":"11","length":"3"}]},{"pgid":"2.7","version":"0'0","reported_seq":166,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-19T11:31:19.070194+0000","last_change":"2024-03-19T11:31:08.381703+0000","last_active":"2024-03-19T11:31:19.070194+0000","last_peered":"2024-03-19T11:31:19.070194+0000","last_clean":"2024-03-19T11:31:19.070194+0000","last_became_active":"2024-03-19T11:30:03.276220+0000","last_became_peered":"2024-03-19T11:30:03.276220+0000","last_unstale":"2024-03-19T11:31:19.070194+0000","last_undegraded":"2024-03-19T11:31:19.070194+0000","last_fullsized":"2024-03-19T11:31:19.070194+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:31:08.381559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:31:08.381559+0000","last_clean_scrub_stamp":"2024-03-19T11:31:08.381559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:30.325139+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"59'92","reported_seq":82,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-19T11:31:18.178669+0000","last_change":"2024-03-19T11:31:18.179369+0000","last_active":"2024-03-19T11:31:18.178669+0000","last_peered":"2024-03-19T11:31:18.178669+0000","last_clean":"2024-03-19T11:31:18.178669+0000","last_became_active":"2024-03-19T11:31:03.052657+0000","last_became_peered":"2024-03-19T11:31:03.052657+0000","last_unstale":"2024-03-19T11:31:18.178669+0000","last_undegraded":"2024-03-19T11:31:18.178669+0000","last_fullsized":"2024-03-19T11:31:18.178669+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'92","last_scrub_stamp":"2024-03-19T11:30:41.351520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:41.351520+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:01.500703+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.025844138999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"15","length":"2"},{"start":"19","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":56,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-19T11:31:19.162570+0000","last_change":"2024-03-19T11:31:19.162627+0000","last_active":"2024-03-19T11:31:19.162570+0000","last_peered":"2024-03-19T11:31:19.162570+0000","last_clean":"2024-03-19T11:31:19.162570+0000","last_became_active":"2024-03-19T11:31:00.067099+0000","last_became_peered":"2024-03-19T11:31:00.067099+0000","last_unstale":"2024-03-19T11:31:19.162570+0000","last_undegraded":"2024-03-19T11:31:19.162570+0000","last_fullsized":"2024-03-19T11:31:19.162570+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:31:19.162377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:31:19.162377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:22.623613+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"100'87","reported_seq":425,"reported_epoch":100,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:19.147311+0000","last_change":"2024-03-19T11:31:18.129075+0000","last_active":"2024-03-19T11:31:19.147311+0000","last_peered":"2024-03-19T11:31:19.147311+0000","last_clean":"2024-03-19T11:31:19.147311+0000","last_became_active":"2024-03-19T11:30:38.065693+0000","last_became_peered":"2024-03-19T11:30:38.065693+0000","last_unstale":"2024-03-19T11:31:19.147311+0000","last_undegraded":"2024-03-19T11:31:19.147311+0000","last_fullsized":"2024-03-19T11:31:19.147311+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'57","last_scrub_stamp":"2024-03-19T11:30:43.792326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:43.792326+0000","objects_scrubbed":6,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:57.135991+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.069188787000000002,"stat_sum":{"num_bytes":13010948,"num_objects":5,"num_object_clones":4,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":18,"num_read_kb":17663,"num_write":16,"num_write_kb":5352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d","length":"1"},{"start":"11","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":198,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-19T11:31:19.079606+0000","last_change":"2024-03-19T11:31:18.554690+0000","last_active":"2024-03-19T11:31:19.079606+0000","last_peered":"2024-03-19T11:31:19.079606+0000","last_clean":"2024-03-19T11:31:19.079606+0000","last_became_active":"2024-03-19T11:30:59.071925+0000","last_became_peered":"2024-03-19T11:30:59.071925+0000","last_unstale":"2024-03-19T11:31:19.079606+0000","last_undegraded":"2024-03-19T11:31:19.079606+0000","last_fullsized":"2024-03-19T11:31:19.079606+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:31:18.554377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:31:18.554377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:39.778063+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"41'19","reported_seq":369,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-19T11:31:17.113006+0000","last_change":"2024-03-19T11:31:17.109034+0000","last_active":"2024-03-19T11:31:17.113006+0000","last_peered":"2024-03-19T11:31:17.113006+0000","last_clean":"2024-03-19T11:31:17.113006+0000","last_became_active":"2024-03-19T11:30:38.058860+0000","last_became_peered":"2024-03-19T11:30:38.058860+0000","last_unstale":"2024-03-19T11:31:17.113006+0000","last_undegraded":"2024-03-19T11:31:17.113006+0000","last_fullsized":"2024-03-19T11:31:17.113006+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'19","last_scrub_stamp":"2024-03-19T11:31:17.108816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:31:17.108816+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:39.641472+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00085534000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"},{"start":"19","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":183,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-19T11:31:19.138872+0000","last_change":"2024-03-19T11:30:55.366872+0000","last_active":"2024-03-19T11:31:19.138872+0000","last_peered":"2024-03-19T11:31:19.138872+0000","last_clean":"2024-03-19T11:31:19.138872+0000","last_became_active":"2024-03-19T11:29:46.540939+0000","last_became_peered":"2024-03-19T11:29:46.540939+0000","last_unstale":"2024-03-19T11:31:19.138872+0000","last_undegraded":"2024-03-19T11:31:19.138872+0000","last_fullsized":"2024-03-19T11:31:19.138872+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:30:55.366631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:30:55.366631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:10.640223+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"99'25","reported_seq":138,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-19T11:31:19.137807+0000","last_change":"2024-03-19T11:31:18.070553+0000","last_active":"2024-03-19T11:31:19.137807+0000","last_peered":"2024-03-19T11:31:19.137807+0000","last_clean":"2024-03-19T11:31:19.137807+0000","last_became_active":"2024-03-19T11:31:00.069200+0000","last_became_peered":"2024-03-19T11:31:00.069200+0000","last_unstale":"2024-03-19T11:31:19.137807+0000","last_undegraded":"2024-03-19T11:31:19.137807+0000","last_fullsized":"2024-03-19T11:31:19.137807+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-03-19T11:31:05.279955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:31:05.279955+0000","objects_scrubbed":2,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:12.810659+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0096179430000000003,"stat_sum":{"num_bytes":5014222,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":28,"num_read_kb":42136,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"15","length":"2"},{"start":"19","length":"1"}]},{"pgid":"3.3","version":"95'109","reported_seq":388,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-19T11:31:18.152108+0000","last_change":"2024-03-19T11:31:16.639529+0000","last_active":"2024-03-19T11:31:18.152108+0000","last_peered":"2024-03-19T11:31:18.152108+0000","last_clean":"2024-03-19T11:31:18.152108+0000","last_became_active":"2024-03-19T11:31:06.066161+0000","last_became_peered":"2024-03-19T11:31:06.066161+0000","last_unstale":"2024-03-19T11:31:18.152108+0000","last_undegraded":"2024-03-19T11:31:18.152108+0000","last_fullsized":"2024-03-19T11:31:18.152108+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'109","last_scrub_stamp":"2024-03-19T11:31:16.639357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:31:16.639357+0000","objects_scrubbed":47,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:31.688365+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49999574,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":82,"num_read_kb":3816,"num_write":117,"num_write_kb":61861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":25111105,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":182,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-19T11:31:19.069590+0000","last_change":"2024-03-19T11:31:17.684706+0000","last_active":"2024-03-19T11:31:19.069590+0000","last_peered":"2024-03-19T11:31:19.069590+0000","last_clean":"2024-03-19T11:31:19.069590+0000","last_became_active":"2024-03-19T11:29:46.320679+0000","last_became_peered":"2024-03-19T11:29:46.320679+0000","last_unstale":"2024-03-19T11:31:19.069590+0000","last_undegraded":"2024-03-19T11:31:19.069590+0000","last_fullsized":"2024-03-19T11:31:19.069590+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:31:17.684492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:31:17.684492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:21.475454+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"87'136","reported_seq":604,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-19T11:31:19.160045+0000","last_change":"2024-03-19T11:31:18.079119+0000","last_active":"2024-03-19T11:31:19.160045+0000","last_peered":"2024-03-19T11:31:19.160045+0000","last_clean":"2024-03-19T11:31:19.160045+0000","last_became_active":"2024-03-19T11:31:03.054624+0000","last_became_peered":"2024-03-19T11:31:03.054624+0000","last_unstale":"2024-03-19T11:31:19.160045+0000","last_undegraded":"2024-03-19T11:31:19.160045+0000","last_fullsized":"2024-03-19T11:31:19.160045+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'119","last_scrub_stamp":"2024-03-19T11:30:52.512203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:52.512203+0000","objects_scrubbed":7,"log_size":136,"log_dups_size":0,"ondisk_log_size":136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:12.078945+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.018421501,"stat_sum":{"num_bytes":12400028,"num_objects":5,"num_object_clones":2,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":82,"num_read_kb":107253,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"15","length":"2"},{"start":"19","length":"1"}]},{"pgid":"3.0","version":"95'110","reported_seq":289,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-19T11:31:19.137596+0000","last_change":"2024-03-19T11:29:57.550320+0000","last_active":"2024-03-19T11:31:19.137596+0000","last_peered":"2024-03-19T11:31:19.137596+0000","last_clean":"2024-03-19T11:31:19.137596+0000","last_became_active":"2024-03-19T11:29:57.549928+0000","last_became_peered":"2024-03-19T11:29:57.549928+0000","last_unstale":"2024-03-19T11:31:19.137596+0000","last_undegraded":"2024-03-19T11:31:19.137596+0000","last_fullsized":"2024-03-19T11:31:19.137596+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35170282,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":84,"num_read_kb":3252,"num_write":115,"num_write_kb":46588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":176,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-19T11:31:19.069985+0000","last_change":"2024-03-19T11:29:46.319811+0000","last_active":"2024-03-19T11:31:19.069985+0000","last_peered":"2024-03-19T11:31:19.069985+0000","last_clean":"2024-03-19T11:31:19.069985+0000","last_became_active":"2024-03-19T11:29:46.319334+0000","last_became_peered":"2024-03-19T11:29:46.319334+0000","last_unstale":"2024-03-19T11:31:19.069985+0000","last_undegraded":"2024-03-19T11:31:19.069985+0000","last_fullsized":"2024-03-19T11:31:19.069985+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:20.548407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"61'40","reported_seq":97,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-19T11:31:19.138086+0000","last_change":"2024-03-19T11:31:18.080274+0000","last_active":"2024-03-19T11:31:19.138086+0000","last_peered":"2024-03-19T11:31:19.138086+0000","last_clean":"2024-03-19T11:31:19.138086+0000","last_became_active":"2024-03-19T11:31:00.069809+0000","last_became_peered":"2024-03-19T11:31:00.069809+0000","last_unstale":"2024-03-19T11:31:19.138086+0000","last_undegraded":"2024-03-19T11:31:19.138086+0000","last_fullsized":"2024-03-19T11:31:19.138086+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'40","last_scrub_stamp":"2024-03-19T11:30:56.505586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:56.505586+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:58.797091+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.014819449,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"15","length":"2"},{"start":"19","length":"1"}]},{"pgid":"3.1","version":"98'99","reported_seq":411,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-19T11:31:17.142942+0000","last_change":"2024-03-19T11:31:03.255027+0000","last_active":"2024-03-19T11:31:17.142942+0000","last_peered":"2024-03-19T11:31:17.142942+0000","last_clean":"2024-03-19T11:31:17.142942+0000","last_became_active":"2024-03-19T11:31:03.077272+0000","last_became_peered":"2024-03-19T11:31:03.077272+0000","last_unstale":"2024-03-19T11:31:17.142942+0000","last_undegraded":"2024-03-19T11:31:17.142942+0000","last_fullsized":"2024-03-19T11:31:17.142942+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:29:56.503774+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:23.954117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44015366,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":80,"num_read_kb":6659,"num_write":103,"num_write_kb":55342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":42087532,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":185,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-19T11:31:19.079837+0000","last_change":"2024-03-19T11:31:00.070027+0000","last_active":"2024-03-19T11:31:19.079837+0000","last_peered":"2024-03-19T11:31:19.079837+0000","last_clean":"2024-03-19T11:31:19.079837+0000","last_became_active":"2024-03-19T11:31:00.068989+0000","last_became_peered":"2024-03-19T11:31:00.068989+0000","last_unstale":"2024-03-19T11:31:19.079837+0000","last_undegraded":"2024-03-19T11:31:19.079837+0000","last_fullsized":"2024-03-19T11:31:19.079837+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:29:45.187160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"89'77","reported_seq":123,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-19T11:31:17.149889+0000","last_change":"2024-03-19T11:31:14.072134+0000","last_active":"2024-03-19T11:31:17.149889+0000","last_peered":"2024-03-19T11:31:17.149889+0000","last_clean":"2024-03-19T11:31:17.149889+0000","last_became_active":"2024-03-19T11:30:59.075057+0000","last_became_peered":"2024-03-19T11:30:59.075057+0000","last_unstale":"2024-03-19T11:31:17.149889+0000","last_undegraded":"2024-03-19T11:31:17.149889+0000","last_fullsized":"2024-03-19T11:31:17.149889+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'65","last_scrub_stamp":"2024-03-19T11:30:53.385414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:53.385414+0000","objects_scrubbed":3,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:14.119190+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00076187499999999997,"stat_sum":{"num_bytes":8047585,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":47,"num_read_kb":72052,"num_write":9,"num_write_kb":2729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4209633,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"},{"start":"19","length":"1"}]},{"pgid":"3.2","version":"99'98","reported_seq":293,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-19T11:31:19.132881+0000","last_change":"2024-03-19T11:31:18.907643+0000","last_active":"2024-03-19T11:31:19.132881+0000","last_peered":"2024-03-19T11:31:19.132881+0000","last_clean":"2024-03-19T11:31:19.132881+0000","last_became_active":"2024-03-19T11:29:57.548946+0000","last_became_peered":"2024-03-19T11:29:57.548946+0000","last_unstale":"2024-03-19T11:31:19.132881+0000","last_undegraded":"2024-03-19T11:31:19.132881+0000","last_fullsized":"2024-03-19T11:31:19.132881+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'98","last_scrub_stamp":"2024-03-19T11:31:18.907500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:31:18.907500+0000","objects_scrubbed":39,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:28.679893+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51285908,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":87,"num_read_kb":8864,"num_write":108,"num_write_kb":52742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":182,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-19T11:31:19.069547+0000","last_change":"2024-03-19T11:31:04.588296+0000","last_active":"2024-03-19T11:31:19.069547+0000","last_peered":"2024-03-19T11:31:19.069547+0000","last_clean":"2024-03-19T11:31:19.069547+0000","last_became_active":"2024-03-19T11:29:46.321760+0000","last_became_peered":"2024-03-19T11:29:46.321760+0000","last_unstale":"2024-03-19T11:31:19.069547+0000","last_undegraded":"2024-03-19T11:31:19.069547+0000","last_fullsized":"2024-03-19T11:31:19.069547+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:31:04.588071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:31:04.588071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:11.976969+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":359,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-19T11:31:19.080093+0000","last_change":"2024-03-19T11:31:03.367805+0000","last_active":"2024-03-19T11:31:19.080093+0000","last_peered":"2024-03-19T11:31:19.080093+0000","last_clean":"2024-03-19T11:31:19.080093+0000","last_became_active":"2024-03-19T11:30:59.073709+0000","last_became_peered":"2024-03-19T11:30:59.073709+0000","last_unstale":"2024-03-19T11:31:19.080093+0000","last_undegraded":"2024-03-19T11:31:19.080093+0000","last_fullsized":"2024-03-19T11:31:19.080093+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2024-03-19T11:31:03.367669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:43.166618+0000","last_clean_scrub_stamp":"2024-03-19T11:31:03.367669+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:29.977573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"99'121","reported_seq":575,"reported_epoch":99,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:18.172832+0000","last_change":"2024-03-19T11:31:18.167828+0000","last_active":"2024-03-19T11:31:18.172832+0000","last_peered":"2024-03-19T11:31:18.172832+0000","last_clean":"2024-03-19T11:31:18.172832+0000","last_became_active":"2024-03-19T11:31:03.065344+0000","last_became_peered":"2024-03-19T11:31:03.065344+0000","last_unstale":"2024-03-19T11:31:18.172832+0000","last_undegraded":"2024-03-19T11:31:18.172832+0000","last_fullsized":"2024-03-19T11:31:18.172832+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'81","last_scrub_stamp":"2024-03-19T11:30:52.360396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:52.360396+0000","objects_scrubbed":6,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:02.676447+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.001232216,"stat_sum":{"num_bytes":10095563,"num_objects":6,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":31,"num_read_kb":28719,"num_write":8,"num_write_kb":1273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":10558635,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"},{"start":"19","length":"1"}]},{"pgid":"4.8","version":"97'116","reported_seq":420,"reported_epoch":99,"state":"peering","last_fresh":"2024-03-19T11:31:18.062727+0000","last_change":"2024-03-19T11:31:19.071700+0000","last_active":"2024-03-19T11:31:17.110057+0000","last_peered":"2024-03-19T11:31:18.062727+0000","last_clean":"2024-03-19T11:31:18.062727+0000","last_became_active":"2024-03-19T11:30:45.251283+0000","last_became_peered":"2024-03-19T11:31:18.058919+0000","last_unstale":"2024-03-19T11:31:18.062727+0000","last_undegraded":"2024-03-19T11:31:18.062727+0000","last_fullsized":"2024-03-19T11:31:18.062727+0000","mapping_epoch":21,"log_start":"97'116","ondisk_log_start":"97'116","created":21,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'71","last_scrub_stamp":"2024-03-19T11:30:53.381386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:53.381386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":80,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.023293851000000001,"stat_sum":{"num_bytes":12590847,"num_objects":7,"num_object_clones":5,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":20,"num_read_kb":21007,"num_write":15,"num_write_kb":4516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":6,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"},{"start":"19","length":"1"}]},{"pgid":"4.9","version":"99'72","reported_seq":463,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-19T11:31:18.174906+0000","last_change":"2024-03-19T11:31:18.175969+0000","last_active":"2024-03-19T11:31:18.174906+0000","last_peered":"2024-03-19T11:31:18.174906+0000","last_clean":"2024-03-19T11:31:18.174906+0000","last_became_active":"2024-03-19T11:31:16.087517+0000","last_became_peered":"2024-03-19T11:31:16.087517+0000","last_unstale":"2024-03-19T11:31:18.174906+0000","last_undegraded":"2024-03-19T11:31:18.174906+0000","last_fullsized":"2024-03-19T11:31:18.174906+0000","mapping_epoch":80,"log_start":"90'70","ondisk_log_start":"90'70","created":21,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'55","last_scrub_stamp":"2024-03-19T11:30:54.706235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:54.706235+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":46,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:23.440168+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.023298282,"stat_sum":{"num_bytes":6698954,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":30,"num_read_kb":39323,"num_write":5,"num_write_kb":1739,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":5842489,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"15","length":"2"},{"start":"19","length":"1"}]},{"pgid":"4.a","version":"87'100","reported_seq":456,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-19T11:31:18.168071+0000","last_change":"2024-03-19T11:31:18.169776+0000","last_active":"2024-03-19T11:31:18.168071+0000","last_peered":"2024-03-19T11:31:18.168071+0000","last_clean":"2024-03-19T11:31:18.168071+0000","last_became_active":"2024-03-19T11:31:12.075247+0000","last_became_peered":"2024-03-19T11:31:12.075247+0000","last_unstale":"2024-03-19T11:31:18.168071+0000","last_undegraded":"2024-03-19T11:31:18.168071+0000","last_fullsized":"2024-03-19T11:31:18.168071+0000","mapping_epoch":21,"log_start":"87'100","ondisk_log_start":"87'100","created":21,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'77","last_scrub_stamp":"2024-03-19T11:30:55.291823+0000","last_deep_scrub":"62'77","last_deep_scrub_stamp":"2024-03-19T11:30:55.291823+0000","last_clean_scrub_stamp":"2024-03-19T11:30:55.291823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:17.544626+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0037957469999999999,"stat_sum":{"num_bytes":9678155,"num_objects":4,"num_object_clones":2,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":42416,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"15","length":"2"},{"start":"19","length":"1"}]},{"pgid":"4.b","version":"90'66","reported_seq":454,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-19T11:31:17.112684+0000","last_change":"2024-03-19T11:31:14.071724+0000","last_active":"2024-03-19T11:31:17.112684+0000","last_peered":"2024-03-19T11:31:17.112684+0000","last_clean":"2024-03-19T11:31:17.112684+0000","last_became_active":"2024-03-19T11:31:08.066470+0000","last_became_peered":"2024-03-19T11:31:08.066470+0000","last_unstale":"2024-03-19T11:31:17.112684+0000","last_undegraded":"2024-03-19T11:31:17.112684+0000","last_fullsized":"2024-03-19T11:31:17.112684+0000","mapping_epoch":21,"log_start":"85'64","ondisk_log_start":"85'64","created":21,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'62","last_scrub_stamp":"2024-03-19T11:30:56.095039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:56.095039+0000","objects_scrubbed":3,"log_size":2,"log_dups_size":30,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:12.262810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051262599999999997,"stat_sum":{"num_bytes":2873964,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"6","length":"1"},{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"16","length":"1"},{"start":"19","length":"1"}]},{"pgid":"4.c","version":"56'29","reported_seq":85,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-19T11:31:18.177181+0000","last_change":"2024-03-19T11:31:18.178557+0000","last_active":"2024-03-19T11:31:18.177181+0000","last_peered":"2024-03-19T11:31:18.177181+0000","last_clean":"2024-03-19T11:31:18.177181+0000","last_became_active":"2024-03-19T11:31:05.073655+0000","last_became_peered":"2024-03-19T11:31:05.073655+0000","last_unstale":"2024-03-19T11:31:18.177181+0000","last_undegraded":"2024-03-19T11:31:18.177181+0000","last_fullsized":"2024-03-19T11:31:18.177181+0000","mapping_epoch":83,"log_start":"56'29","ondisk_log_start":"56'29","created":21,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'29","last_scrub_stamp":"2024-03-19T11:30:53.743385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:53.743385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:53.976729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.025313690999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"15","length":"2"},{"start":"19","length":"1"}]},{"pgid":"4.d","version":"99'95","reported_seq":478,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-19T11:31:18.179352+0000","last_change":"2024-03-19T11:31:18.180336+0000","last_active":"2024-03-19T11:31:18.179352+0000","last_peered":"2024-03-19T11:31:18.179352+0000","last_clean":"2024-03-19T11:31:18.179352+0000","last_became_active":"2024-03-19T11:31:06.068526+0000","last_became_peered":"2024-03-19T11:31:06.068526+0000","last_unstale":"2024-03-19T11:31:18.179352+0000","last_undegraded":"2024-03-19T11:31:18.179352+0000","last_fullsized":"2024-03-19T11:31:18.179352+0000","mapping_epoch":86,"log_start":"70'71","ondisk_log_start":"70'71","created":21,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'73","last_scrub_stamp":"2024-03-19T11:30:53.519178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:53.519178+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":50,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:02.197166+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.027530145999999998,"stat_sum":{"num_bytes":9167514,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":19,"num_read_kb":22671,"num_write":5,"num_write_kb":1740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":8146998,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"15","length":"2"},{"start":"19","length":"1"}]},{"pgid":"4.e","version":"99'102","reported_seq":448,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-19T11:31:18.169410+0000","last_change":"2024-03-19T11:31:18.170905+0000","last_active":"2024-03-19T11:31:18.169410+0000","last_peered":"2024-03-19T11:31:18.169410+0000","last_clean":"2024-03-19T11:31:18.169410+0000","last_became_active":"2024-03-19T11:30:38.060487+0000","last_became_peered":"2024-03-19T11:30:38.060487+0000","last_unstale":"2024-03-19T11:31:18.169410+0000","last_undegraded":"2024-03-19T11:31:18.169410+0000","last_fullsized":"2024-03-19T11:31:18.169410+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'65","last_scrub_stamp":"2024-03-19T11:30:43.824620+0000","last_deep_scrub":"65'65","last_deep_scrub_stamp":"2024-03-19T11:30:43.824620+0000","last_clean_scrub_stamp":"2024-03-19T11:30:43.824620+0000","objects_scrubbed":4,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:10.632766+0000","scrub_duration":35,"objects_trimmed":1,"snaptrim_duration":0.018371851000000002,"stat_sum":{"num_bytes":9058940,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":46,"num_read_kb":54326,"num_write":12,"num_write_kb":2301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"15","length":"2"},{"start":"19","length":"1"}]},{"pgid":"4.f","version":"0'0","reported_seq":320,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-19T11:31:19.138336+0000","last_change":"2024-03-19T11:31:18.062305+0000","last_active":"2024-03-19T11:31:19.138336+0000","last_peered":"2024-03-19T11:31:19.138336+0000","last_clean":"2024-03-19T11:31:19.138336+0000","last_became_active":"2024-03-19T11:30:38.050919+0000","last_became_peered":"2024-03-19T11:30:38.050919+0000","last_unstale":"2024-03-19T11:31:19.138336+0000","last_undegraded":"2024-03-19T11:31:19.138336+0000","last_fullsized":"2024-03-19T11:31:19.138336+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:30:43.411893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:43.411893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:54.608796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034006459999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d","length":"1"},{"start":"11","length":"3"},{"start":"15","length":"2"},{"start":"19","length":"1"}]}],"pool_stats":[{"poolid":4,"num_pg":40,"stat_sum":{"num_bytes":284034659,"num_objects":137,"num_object_clones":87,"num_object_copies":274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":134,"num_whiteouts":8,"num_read":1271,"num_read_kb":1494316,"num_write":373,"num_write_kb":99424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":202389426,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":96,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":381624320,"data_stored":460190542,"data_compressed":2258794,"data_compressed_allocated":1650688,"data_compressed_original":5439488,"omap_allocated":0,"internal_metadata":0},"log_size":2555,"ondisk_log_size":2555,"up":80,"acting":80,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":180471130,"num_objects":158,"num_object_clones":0,"num_object_copies":316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":0,"num_read":333,"num_read_kb":22591,"num_write":443,"num_write_kb":216533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":67198637,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":228605952,"data_stored":246986339,"data_compressed":16796491,"data_compressed_allocated":17973248,"data_compressed_original":36810752,"omap_allocated":0,"internal_metadata":0},"log_size":416,"ondisk_log_size":416,"up":8,"acting":8,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":1586784,"data_compressed":19832,"data_compressed_allocated":98304,"data_compressed_original":1560576,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542167,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":90260,"kb_used_data":59080,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94281580,"statfs":{"total":96636764160,"available":96544337920,"internally_reserved":0,"allocated":60497920,"data_stored":73836589,"data_compressed":154066,"data_compressed_allocated":151552,"data_compressed_original":970752,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542167,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":93960,"kb_used_data":62776,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94277880,"statfs":{"total":96636764160,"available":96540549120,"internally_reserved":0,"allocated":64282624,"data_stored":77451589,"data_compressed":2244718,"data_compressed_allocated":2367488,"data_compressed_original":5038080,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542167,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":131256,"kb_used_data":100076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94240584,"statfs":{"total":96636764160,"available":96502358016,"internally_reserved":0,"allocated":102477824,"data_stored":118634217,"data_compressed":2688064,"data_compressed_allocated":2654208,"data_compressed_original":6119424,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542166,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79016,"kb_used_data":51992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94292824,"statfs":{"total":96636764160,"available":96555851776,"internally_reserved":0,"allocated":53239808,"data_stored":59728988,"data_compressed":1859671,"data_compressed_allocated":1712128,"data_compressed_original":4390912,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542166,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":115832,"kb_used_data":80492,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94256008,"statfs":{"total":96636764160,"available":96518152192,"internally_reserved":0,"allocated":82423808,"data_stored":92028789,"data_compressed":1466229,"data_compressed_allocated":1576960,"data_compressed_original":3473408,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574870,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68856,"kb_used_data":37680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94302984,"statfs":{"total":96636764160,"available":96566255616,"internally_reserved":0,"allocated":38584320,"data_stored":51320082,"data_compressed":169512,"data_compressed_allocated":57344,"data_compressed_original":196608,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574870,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":145392,"kb_used_data":114208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94226448,"statfs":{"total":96636764160,"available":96487882752,"internally_reserved":0,"allocated":116948992,"data_stored":131715658,"data_compressed":4130788,"data_compressed_allocated":4341760,"data_compressed_original":8802304,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574870,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":100876,"kb_used_data":69692,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94270964,"statfs":{"total":96636764160,"available":96533467136,"internally_reserved":0,"allocated":71364608,"data_stored":77338105,"data_compressed":4147416,"data_compressed_allocated":4403200,"data_compressed_original":9555968,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6884,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6884,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6064,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":52604928,"data_stored":56793414,"data_compressed":3980203,"data_compressed_allocated":4243456,"data_compressed_original":8540160,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":18395136,"data_stored":19951678,"data_compressed":1189190,"data_compressed_allocated":1282048,"data_compressed_original":2883584,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":18395136,"data_stored":19951678,"data_compressed":1189190,"data_compressed_allocated":1282048,"data_compressed_original":2883584,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28160000,"data_stored":30453963,"data_compressed":2096716,"data_compressed_allocated":2248704,"data_compressed_original":4579328,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":52604928,"data_stored":56793414,"data_compressed":3980203,"data_compressed_allocated":4243456,"data_compressed_original":8540160,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28160000,"data_stored":30453963,"data_compressed":2096716,"data_compressed_allocated":2248704,"data_compressed_original":4579328,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":30285824,"data_stored":32588229,"data_compressed":2264273,"data_compressed_allocated":2424832,"data_compressed_original":4804608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":19255296,"data_stored":21470498,"data_compressed":160329,"data_compressed_allocated":126976,"data_compressed_original":491520,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":62922752,"data_stored":71485960,"data_compressed":277039,"data_compressed_allocated":294912,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":33763328,"data_stored":39195005,"data_compressed":670481,"data_compressed_allocated":430080,"data_compressed_original":1507328,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":78266368,"data_stored":95871262,"data_compressed":591348,"data_compressed_allocated":405504,"data_compressed_original":1540096,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":58044416,"data_stored":66752235,"data_compressed":150585,"data_compressed_allocated":98304,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":35037184,"data_stored":46415333,"data_compressed":148002,"data_compressed_allocated":118784,"data_compressed_original":458752,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":34967552,"data_stored":44556908,"data_compressed":113008,"data_compressed_allocated":57344,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":59367424,"data_stored":74443341,"data_compressed":148002,"data_compressed_allocated":118784,"data_compressed_original":458752,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T11:31:20.411 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pgp_num 26 2024-03-19T11:31:20.448 INFO:tasks.rados.rados.0.smithi038.stdout:1383: expect (ObjNum 206 snap 22 seq_num 206) 2024-03-19T11:31:20.468 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:20.468 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:20.483 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-19T11:31:20.492 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:20.501+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:20.581 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:20.590+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:20.607 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:20.608 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:20.608 INFO:tasks.rados.rados.0.smithi038.stdout:1384: seq_num 236 ranges {766452=565707,1860221=469412,2981238=594195} 2024-03-19T11:31:20.618 INFO:tasks.rados.rados.0.smithi038.stdout:1384: writing smithi03829856-37 from 766452 to 1332159 tid 1 2024-03-19T11:31:20.621 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-19T11:31:20.627 INFO:tasks.rados.rados.0.smithi038.stdout:1384: writing smithi03829856-37 from 1860221 to 2329633 tid 2 2024-03-19T11:31:20.646 INFO:tasks.rados.rados.0.smithi038.stdout:1384: writing smithi03829856-37 from 2981238 to 3575433 tid 3 2024-03-19T11:31:20.646 INFO:tasks.rados.rados.0.smithi038.stdout:1381: done (2 left) 2024-03-19T11:31:20.646 INFO:tasks.rados.rados.0.smithi038.stdout:1383: done (1 left) 2024-03-19T11:31:20.646 INFO:tasks.rados.rados.0.smithi038.stdout:1385: read oid 3 snap -1 2024-03-19T11:31:20.646 INFO:tasks.rados.rados.0.smithi038.stdout:1385: expect deleted 2024-03-19T11:31:20.647 INFO:tasks.rados.rados.0.smithi038.stdout:1386: snap_create 2024-03-19T11:31:20.647 INFO:tasks.rados.rados.0.smithi038.stdout:1384: finishing write tid 1 to smithi03829856-37 2024-03-19T11:31:20.647 INFO:tasks.rados.rados.0.smithi038.stdout:1384: finishing write tid 2 to smithi03829856-37 2024-03-19T11:31:20.651 INFO:tasks.rados.rados.0.smithi038.stdout:1384: finishing write tid 3 to smithi03829856-37 2024-03-19T11:31:20.652 INFO:tasks.rados.rados.0.smithi038.stdout:1384: finishing write tid 4 to smithi03829856-37 2024-03-19T11:31:20.652 INFO:tasks.rados.rados.0.smithi038.stdout:1384: finishing write tid 5 to smithi03829856-37 2024-03-19T11:31:20.652 INFO:tasks.rados.rados.0.smithi038.stdout:1384: oid 37 updating version 0 to 122 2024-03-19T11:31:20.652 INFO:tasks.rados.rados.0.smithi038.stdout:1384: oid 37 updating version 122 to 124 2024-03-19T11:31:20.652 INFO:tasks.rados.rados.0.smithi038.stdout:1384: oid 37 updating version 124 to 125 2024-03-19T11:31:20.652 INFO:tasks.rados.rados.0.smithi038.stdout:1384: oid 37 version 125 is already newer than 123 2024-03-19T11:31:20.652 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 37 v 125 (ObjNum 236 snap 28 seq_num 236) dirty exists 2024-03-19T11:31:20.653 INFO:tasks.rados.rados.0.smithi038.stdout:1384: left oid 37 (ObjNum 236 snap 28 seq_num 236) 2024-03-19T11:31:20.708 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:20.718+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:20.784 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:20.793+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:20.786 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:20.786 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:20.800 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-19T11:31:20.887 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:20.898+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:20.961 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:20.961 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:20.975 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-19T11:31:21.016 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:21.026+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:21.082 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-03-19T11:31:21.086 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:21.094+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:21.115 INFO:teuthology.orchestra.run.smithi038.stderr:set pool 4 pgp_num to 26 2024-03-19T11:31:21.116 INFO:tasks.rados.rados.0.smithi038.stdout:1384: done (2 left) 2024-03-19T11:31:21.116 INFO:tasks.rados.rados.0.smithi038.stdout:1385: done (1 left) 2024-03-19T11:31:21.116 INFO:tasks.rados.rados.0.smithi038.stdout:1386: done (0 left) 2024-03-19T11:31:21.116 INFO:tasks.rados.rados.0.smithi038.stdout:1387: write oid 43 current snap is 29 2024-03-19T11:31:21.116 INFO:tasks.rados.rados.0.smithi038.stdout:1387: seq_num 237 ranges {783259=640956,1918703=613070,3008661=405467,3821939=161320} 2024-03-19T11:31:21.129 INFO:tasks.rados.rados.0.smithi038.stdout:1387: writing smithi03829856-43 from 783259 to 1424215 tid 1 2024-03-19T11:31:21.136 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:21.136 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:21.140 INFO:tasks.rados.rados.0.smithi038.stdout:1387: writing smithi03829856-43 from 1918703 to 2531773 tid 2 2024-03-19T11:31:21.148 INFO:tasks.rados.rados.0.smithi038.stdout:1387: writing smithi03829856-43 from 3008661 to 3414128 tid 3 2024-03-19T11:31:21.151 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-19T11:31:21.152 INFO:tasks.rados.rados.0.smithi038.stdout:1387: writing smithi03829856-43 from 3821939 to 3983259 tid 4 2024-03-19T11:31:21.152 INFO:tasks.rados.rados.0.smithi038.stdout:1388: read oid 32 snap -1 2024-03-19T11:31:21.152 INFO:tasks.rados.rados.0.smithi038.stdout:1388: expect (ObjNum 31 snap 0 seq_num 31) 2024-03-19T11:31:21.152 INFO:tasks.rados.rados.0.smithi038.stdout:1389: read oid 39 snap 28 2024-03-19T11:31:21.153 INFO:tasks.rados.rados.0.smithi038.stdout:1389: expect deleted 2024-03-19T11:31:21.153 INFO:tasks.rados.rados.0.smithi038.stdout:1390: read oid 29 snap -1 2024-03-19T11:31:21.153 INFO:tasks.rados.rados.0.smithi038.stdout:1390: expect (ObjNum 219 snap 23 seq_num 219) 2024-03-19T11:31:21.153 INFO:tasks.rados.rados.0.smithi038.stdout:1391: set_chunk oid 44 target oid 2024-03-19T11:31:21.172 INFO:tasks.rados.rados.0.smithi038.stdout:1391: set_chunk oid 44 offset: 758272 length: 1283072 target oid c7e2612f64976200df8e83f1ef6433e911b82404e742e97a89363ce8f8c5a87d offset: 0 snap -1 2024-03-19T11:31:21.172 INFO:tasks.rados.rados.0.smithi038.stdout:1392: read oid 5 snap -1 2024-03-19T11:31:21.172 INFO:tasks.rados.rados.0.smithi038.stdout:1392: expect (ObjNum 222 snap 24 seq_num 222) 2024-03-19T11:31:21.172 INFO:tasks.rados.rados.0.smithi038.stdout:1393: rollback oid 41 current snap is 29 2024-03-19T11:31:21.172 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 41 to 24 2024-03-19T11:31:21.173 INFO:tasks.rados.rados.0.smithi038.stdout:1387: finishing write tid 1 to smithi03829856-43 2024-03-19T11:31:21.173 INFO:tasks.rados.rados.0.smithi038.stdout:1387: finishing write tid 2 to smithi03829856-43 2024-03-19T11:31:21.173 INFO:tasks.rados.rados.0.smithi038.stdout:1387: finishing write tid 3 to smithi03829856-43 2024-03-19T11:31:21.174 INFO:tasks.rados.rados.0.smithi038.stdout:1387: finishing write tid 4 to smithi03829856-43 2024-03-19T11:31:21.174 INFO:tasks.rados.rados.0.smithi038.stdout:1387: finishing write tid 5 to smithi03829856-43 2024-03-19T11:31:21.174 INFO:tasks.rados.rados.0.smithi038.stdout:1387: finishing write tid 6 to smithi03829856-43 2024-03-19T11:31:21.174 INFO:tasks.rados.rados.0.smithi038.stdout:1387: oid 43 updating version 0 to 146 2024-03-19T11:31:21.174 INFO:tasks.rados.rados.0.smithi038.stdout:1387: oid 43 version 146 is already newer than 144 2024-03-19T11:31:21.174 INFO:tasks.rados.rados.0.smithi038.stdout:1387: oid 43 version 146 is already newer than 145 2024-03-19T11:31:21.174 INFO:tasks.rados.rados.0.smithi038.stdout:1387: oid 43 version 146 is already newer than 143 2024-03-19T11:31:21.174 INFO:tasks.rados.rados.0.smithi038.stdout:1387: oid 43 version 146 is already newer than 142 2024-03-19T11:31:21.174 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 43 v 146 (ObjNum 237 snap 29 seq_num 237) dirty exists 2024-03-19T11:31:21.174 INFO:tasks.rados.rados.0.smithi038.stdout:1387: left oid 43 (ObjNum 237 snap 29 seq_num 237) 2024-03-19T11:31:21.174 INFO:tasks.rados.rados.0.smithi038.stdout:1388: expect (ObjNum 31 snap 0 seq_num 31) 2024-03-19T11:31:21.187 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:21.193+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:21.272 INFO:tasks.rados.rados.0.smithi038.stdout:1390: expect (ObjNum 219 snap 23 seq_num 219) 2024-03-19T11:31:21.288 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:21.297+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:21.314 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:21.314 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:21.328 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-19T11:31:21.380 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-19T11:31:21.389 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:21.398+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:21.454 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:21.454 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:21.466 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-19T11:31:21.490 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:21.498+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:21.512 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:31:21.514 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:31:21.528 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:31:21.528 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:31:21.545 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:31:21.546 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:31:21.562 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:31:21.562 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:31:21.581 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:21.582 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:31:21.582 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:31:21.582 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:31:21.582 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:21.582 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:31:21.583 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:31:21.590 INFO:tasks.rados.rados.0.smithi038.stdout:1391: finishing set_chunk to oid 44 2024-03-19T11:31:21.591 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 44 set_chunk returned error code -95 offset: 758272 length: 1283072 2024-03-19T11:31:21.591 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 44 v 0 (ObjNum 108 snap 11 seq_num 108) dirty exists 2024-03-19T11:31:21.591 INFO:tasks.rados.rados.0.smithi038.stdout:1392: expect (ObjNum 222 snap 24 seq_num 222) 2024-03-19T11:31:21.591 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:21.597+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:21.594 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-19T11:31:21.604 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:31:21.604 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:31:21.626 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:31:21.627 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:31:21.648 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:31:21.648 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:31:21.708 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:21.717+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:21.720 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:21.720 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:31:21.720 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:31:21.720 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:31:21.720 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.721 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1217 4.3 4:c8b09e87:::smithi03829856-15:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:31:21.721 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.155966+0000", 2024-03-19T11:31:21.721 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 63.571759383, 2024-03-19T11:31:21.721 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.023984647000000001, 2024-03-19T11:31:21.721 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:21.721 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:21.721 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:21.721 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:21.721 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:21.721 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1217 2024-03-19T11:31:21.721 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.721 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:21.721 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.722 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:21.722 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155966+0000", 2024-03-19T11:31:21.722 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:21.722 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.722 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.722 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:21.722 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155966+0000", 2024-03-19T11:31:21.722 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:21.722 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.722 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.722 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:21.722 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155968+0000", 2024-03-19T11:31:21.722 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.795e-06 2024-03-19T11:31:21.723 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.723 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.723 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:21.723 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155978+0000", 2024-03-19T11:31:21.723 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0574000000000001e-05 2024-03-19T11:31:21.723 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.723 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.723 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:21.723 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.156000+0000", 2024-03-19T11:31:21.723 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1250000000000002e-05 2024-03-19T11:31:21.723 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.724 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.724 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:21.724 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.156010+0000", 2024-03-19T11:31:21.724 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0168e-05 2024-03-19T11:31:21.724 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.724 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.724 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:21.724 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.156119+0000", 2024-03-19T11:31:21.725 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010950699999999999 2024-03-19T11:31:21.725 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.725 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.725 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:21.725 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.156231+0000", 2024-03-19T11:31:21.725 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000111445 2024-03-19T11:31:21.725 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.725 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.725 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:21.725 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.179951+0000", 2024-03-19T11:31:21.725 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.023719908000000001 2024-03-19T11:31:21.725 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:21.725 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:21.726 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:21.726 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.726 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.726 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.24 75/74 [PushOp(4:27909373:::smithi03829856-46:8, version: 46'59, data_included: [756315~767970,1933572~280606], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:8@46'59, size: 2756315, copy_subset: [(0, 2756315)], clone_subset: {}, snapset: 10=[]:{8: [6], c: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2214178, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:266590a5:::smithi03829856-16:e, version: 70'107, data_included: [652105~662359,1728171~386217], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:e@70'107, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 10=[]:{e: [d, b, 6]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:266590a5:::smithi03829856-16:head, version: 70'108, data_included: [652105~662359,1728171~386217], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:head@70'108, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:27909373:::smithi03829856-46:c, version: 70'109, data_included: [628595~895690,1807440~152886], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:c@70'109, size: 3428595, copy_subset: [(0, 3428595)], clone_subset: {}, snapset: 10=[]:{8: [6], c: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1960326, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:27909373:::smithi03829856-46:head, version: 70'110, data_included: [403401~454961,1584925~506039], data_size: 961000, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:head@70'110, size: 2403401, copy_subset: [(0, 2403401)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2403401, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:21.726 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.099395+0000", 2024-03-19T11:31:21.726 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 27.628330221999999, 2024-03-19T11:31:21.726 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.045391157000000001, 2024-03-19T11:31:21.726 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:21.726 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:21.726 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:21.726 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.726 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:21.727 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.099395+0000", 2024-03-19T11:31:21.727 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:21.727 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.727 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.727 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:21.727 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.099395+0000", 2024-03-19T11:31:21.727 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:21.727 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.727 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.727 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:21.727 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.099397+0000", 2024-03-19T11:31:21.727 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.691e-06 2024-03-19T11:31:21.727 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.727 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.727 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:21.728 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105009+0000", 2024-03-19T11:31:21.728 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0056121349999999999 2024-03-19T11:31:21.728 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.728 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.728 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:21.728 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105077+0000", 2024-03-19T11:31:21.728 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8561999999999998e-05 2024-03-19T11:31:21.728 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.728 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.728 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:21.728 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105133+0000", 2024-03-19T11:31:21.728 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5590000000000001e-05 2024-03-19T11:31:21.728 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.729 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.729 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:21.729 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105531+0000", 2024-03-19T11:31:21.729 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00039756200000000001 2024-03-19T11:31:21.729 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.729 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.729 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:21.729 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105579+0000", 2024-03-19T11:31:21.729 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8028999999999997e-05 2024-03-19T11:31:21.729 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.729 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.729 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:21.729 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.144786+0000", 2024-03-19T11:31:21.729 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.039207588000000002 2024-03-19T11:31:21.729 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:21.730 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:21.730 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:21.730 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.730 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.730 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.24 e75,74 4:266590a5:::smithi03829856-16:4,47'63,4:27909373:::smithi03829856-46:5,55'79,4:266590a5:::smithi03829856-16:10,67'105)", 2024-03-19T11:31:21.730 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.105191+0000", 2024-03-19T11:31:21.730 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 27.622534122000001, 2024-03-19T11:31:21.730 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.039580830999999997, 2024-03-19T11:31:21.730 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:21.730 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:21.730 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:21.730 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.731 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:21.731 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105191+0000", 2024-03-19T11:31:21.731 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:21.731 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.731 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.731 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:21.731 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105191+0000", 2024-03-19T11:31:21.731 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:21.731 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.731 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.731 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:21.731 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105192+0000", 2024-03-19T11:31:21.732 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.099e-06 2024-03-19T11:31:21.732 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.732 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.732 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:21.732 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105200+0000", 2024-03-19T11:31:21.732 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5279999999999998e-06 2024-03-19T11:31:21.732 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.733 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.733 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:21.733 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105211+0000", 2024-03-19T11:31:21.733 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1471000000000001e-05 2024-03-19T11:31:21.733 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.733 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.733 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:21.733 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105220+0000", 2024-03-19T11:31:21.733 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3469999999999999e-06 2024-03-19T11:31:21.734 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.734 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.734 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:21.734 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.144219+0000", 2024-03-19T11:31:21.734 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.038999024 2024-03-19T11:31:21.734 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.735 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.735 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:21.735 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.144257+0000", 2024-03-19T11:31:21.735 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8501999999999998e-05 2024-03-19T11:31:21.735 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.735 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.735 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:21.736 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.144772+0000", 2024-03-19T11:31:21.736 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00051486000000000001 2024-03-19T11:31:21.736 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:21.736 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:21.736 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:21.736 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.736 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.736 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 75/74 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:7, version: 55'82, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@55'82, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 58'86, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@58'86, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [504210~968657,1572695~79919], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:12, version: 66'101, data_included: [440350~816629,1309629~231947], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:12@66'101, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:10, version: 70'112, data_included: [440350~816629,1309629~231947], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:10@70'112, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 70'113, data_included: [440350~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@70'113, size: 2605085, copy_subset: [(0, 2605085)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1488926, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:21.736 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.107266+0000", 2024-03-19T11:31:21.736 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 27.620459605000001, 2024-03-19T11:31:21.736 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.048592669999999998, 2024-03-19T11:31:21.737 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:21.737 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:21.737 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:21.737 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.737 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:21.737 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.107266+0000", 2024-03-19T11:31:21.737 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:21.737 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.737 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.737 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:21.737 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.107266+0000", 2024-03-19T11:31:21.738 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:21.738 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.738 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.738 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:21.738 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.107267+0000", 2024-03-19T11:31:21.738 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1459999999999999e-06 2024-03-19T11:31:21.738 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.738 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.738 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:21.738 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.122861+0000", 2024-03-19T11:31:21.739 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01559372 2024-03-19T11:31:21.739 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.739 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.739 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:21.739 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.122932+0000", 2024-03-19T11:31:21.739 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1477999999999996e-05 2024-03-19T11:31:21.739 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.739 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.739 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:21.740 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.122999+0000", 2024-03-19T11:31:21.740 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6832e-05 2024-03-19T11:31:21.740 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.740 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.740 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:21.740 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123332+0000", 2024-03-19T11:31:21.740 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000333271 2024-03-19T11:31:21.740 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.740 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.740 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:21.740 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123371+0000", 2024-03-19T11:31:21.740 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8525999999999997e-05 2024-03-19T11:31:21.740 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.741 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.741 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:21.741 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.155858+0000", 2024-03-19T11:31:21.741 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.032487697000000003 2024-03-19T11:31:21.741 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:21.741 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:21.741 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:21.741 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.741 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.741 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.21 e75,74 4:8401c1f0:::smithi03829856-43:8,46'66,4:866f3dcc:::smithi03829856-31:2,47'73)", 2024-03-19T11:31:21.742 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.123075+0000", 2024-03-19T11:31:21.742 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 27.604650385999999, 2024-03-19T11:31:21.742 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.033023045000000001, 2024-03-19T11:31:21.742 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:21.742 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:21.742 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:21.742 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.742 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:21.742 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123075+0000", 2024-03-19T11:31:21.742 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:21.743 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.743 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.743 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:21.743 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123075+0000", 2024-03-19T11:31:21.743 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:21.743 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.743 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.743 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:21.743 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123076+0000", 2024-03-19T11:31:21.743 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5e-06 2024-03-19T11:31:21.744 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.744 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.744 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:21.744 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123086+0000", 2024-03-19T11:31:21.744 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8060000000000006e-06 2024-03-19T11:31:21.744 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.744 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.744 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:21.744 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123099+0000", 2024-03-19T11:31:21.744 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3251000000000001e-05 2024-03-19T11:31:21.745 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.745 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.745 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:21.745 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123112+0000", 2024-03-19T11:31:21.745 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2774e-05 2024-03-19T11:31:21.745 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.745 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.745 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:21.745 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.155902+0000", 2024-03-19T11:31:21.745 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.032790225999999999 2024-03-19T11:31:21.746 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.746 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.746 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:21.746 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.155914+0000", 2024-03-19T11:31:21.746 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1476e-05 2024-03-19T11:31:21.746 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.746 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.746 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:21.746 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.156098+0000", 2024-03-19T11:31:21.746 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000184012 2024-03-19T11:31:21.746 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:21.746 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:21.747 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:21.747 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.747 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.747 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 75/74 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:7, version: 55'82, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@55'82, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 58'86, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@58'86, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [1652614~973634,2888511~52681,3182305~22261], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3204566, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:12, version: 66'101, data_included: [1541576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:12@66'101, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:10, version: 70'112, data_included: [1541576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:10@70'112, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 70'113, data_included: [1488926~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@70'113, size: 2605085, copy_subset: [(0, 2605085)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2537502, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1488926, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:21.747 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.174030+0000", 2024-03-19T11:31:21.747 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 27.553695181999998, 2024-03-19T11:31:21.747 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.040799275000000003, 2024-03-19T11:31:21.747 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:21.747 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:21.747 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:21.748 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.748 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:21.748 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.174030+0000", 2024-03-19T11:31:21.748 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:21.748 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.748 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.748 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:21.748 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.174030+0000", 2024-03-19T11:31:21.748 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:21.748 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.748 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.749 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:21.749 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.174031+0000", 2024-03-19T11:31:21.749 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.322e-06 2024-03-19T11:31:21.749 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.749 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.749 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:21.749 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.184771+0000", 2024-03-19T11:31:21.749 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010739227 2024-03-19T11:31:21.749 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.749 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.749 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:21.750 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.184872+0000", 2024-03-19T11:31:21.750 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000101234 2024-03-19T11:31:21.750 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.750 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.750 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:21.750 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.184985+0000", 2024-03-19T11:31:21.750 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000113545 2024-03-19T11:31:21.750 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.750 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.750 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:21.750 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.185463+0000", 2024-03-19T11:31:21.751 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00047749899999999999 2024-03-19T11:31:21.751 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.751 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.751 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:21.751 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.185534+0000", 2024-03-19T11:31:21.751 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1007999999999995e-05 2024-03-19T11:31:21.751 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.751 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.751 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:21.751 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.214829+0000", 2024-03-19T11:31:21.752 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029295439999999999 2024-03-19T11:31:21.752 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:21.752 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:21.752 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:21.752 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.752 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.752 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.21 e75,74 4:866f3dcc:::smithi03829856-31:c,70'114)", 2024-03-19T11:31:21.752 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.197457+0000", 2024-03-19T11:31:21.752 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 27.530268136, 2024-03-19T11:31:21.752 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018493590000000001, 2024-03-19T11:31:21.752 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:21.752 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:21.752 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:21.753 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.753 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:21.753 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.197457+0000", 2024-03-19T11:31:21.753 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:21.753 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.753 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.753 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:21.753 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.197457+0000", 2024-03-19T11:31:21.753 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:21.753 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.753 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.753 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:21.753 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.197459+0000", 2024-03-19T11:31:21.753 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3199999999999998e-06 2024-03-19T11:31:21.754 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.754 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.754 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:21.754 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.197480+0000", 2024-03-19T11:31:21.754 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0366000000000001e-05 2024-03-19T11:31:21.754 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.754 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.754 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:21.754 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.197521+0000", 2024-03-19T11:31:21.754 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0738000000000002e-05 2024-03-19T11:31:21.754 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.754 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.754 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:21.754 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.197547+0000", 2024-03-19T11:31:21.755 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6548999999999999e-05 2024-03-19T11:31:21.755 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.755 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.755 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:21.755 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.214954+0000", 2024-03-19T11:31:21.755 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01740713 2024-03-19T11:31:21.755 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.755 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.755 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:21.755 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.214972+0000", 2024-03-19T11:31:21.755 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7520000000000002e-05 2024-03-19T11:31:21.755 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.755 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.756 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:21.756 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.215951+0000", 2024-03-19T11:31:21.756 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00097896700000000007 2024-03-19T11:31:21.756 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:21.756 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:21.756 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:21.756 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.756 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.756 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 75/74 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [2599014~241336], data_size: 241336, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2840350, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [2599014~241336,2948041~446940], data_size: 688276, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3394981, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [3204566~621403], data_size: 621403, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3865577, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3204566, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:12, version: 66'101, data_included: [2590152~250198,2948041~581396], data_size: 831594, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:12@66'101, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3529437, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:10, version: 70'112, data_included: [2590152~250198,2948041~581396], data_size: 831594, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:10@70'112, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3529437, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 70'113, data_included: [2537502~67583], data_size: 67583, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@70'113, size: 2605085, copy_subset: [(0, 2605085)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2605085, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2537502, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:21.756 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.222872+0000", 2024-03-19T11:31:21.756 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 27.504853689000001, 2024-03-19T11:31:21.756 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016277658, 2024-03-19T11:31:21.756 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:21.756 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:21.757 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:21.757 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.757 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:21.757 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.222872+0000", 2024-03-19T11:31:21.757 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:21.757 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.757 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.757 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:21.757 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.222872+0000", 2024-03-19T11:31:21.757 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:21.757 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.757 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.758 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:21.758 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.222874+0000", 2024-03-19T11:31:21.758 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9999999999999999e-06 2024-03-19T11:31:21.758 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.758 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.758 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:21.758 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.227563+0000", 2024-03-19T11:31:21.758 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046897570000000001 2024-03-19T11:31:21.758 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.758 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.758 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:21.758 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.227639+0000", 2024-03-19T11:31:21.758 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5607999999999998e-05 2024-03-19T11:31:21.758 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.759 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.759 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:21.759 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.227736+0000", 2024-03-19T11:31:21.759 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6854000000000005e-05 2024-03-19T11:31:21.759 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.759 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.759 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:21.759 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.228149+0000", 2024-03-19T11:31:21.759 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00041330000000000002 2024-03-19T11:31:21.759 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.759 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.759 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:21.759 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.228219+0000", 2024-03-19T11:31:21.760 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.0140000000000003e-05 2024-03-19T11:31:21.760 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.760 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.760 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:21.760 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.239149+0000", 2024-03-19T11:31:21.760 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010929998999999999 2024-03-19T11:31:21.760 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:21.760 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:21.760 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:21.760 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.760 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.760 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2372 4.16 4:6868af91:::smithi03829856-10:b [read 832512~818688] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-03-19T11:31:21.760 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:58.455732+0000", 2024-03-19T11:31:21.760 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 23.271993467000001, 2024-03-19T11:31:21.760 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.69150541200000004, 2024-03-19T11:31:21.761 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:21.761 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:21.761 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:21.761 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:21.761 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:21.761 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2372 2024-03-19T11:31:21.761 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.761 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:21.761 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.761 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:21.761 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455732+0000", 2024-03-19T11:31:21.761 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:21.762 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.762 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.762 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:21.762 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455732+0000", 2024-03-19T11:31:21.762 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:21.762 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.762 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.762 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:21.762 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455735+0000", 2024-03-19T11:31:21.762 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.756e-06 2024-03-19T11:31:21.762 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.762 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.762 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:21.762 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455744+0000", 2024-03-19T11:31:21.762 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1880000000000008e-06 2024-03-19T11:31:21.763 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.763 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.763 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:21.763 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455764+0000", 2024-03-19T11:31:21.763 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0044999999999999e-05 2024-03-19T11:31:21.763 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.763 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.763 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:21.763 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455774+0000", 2024-03-19T11:31:21.763 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8770000000000005e-06 2024-03-19T11:31:21.763 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.763 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.763 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:21.763 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455837+0000", 2024-03-19T11:31:21.763 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3588000000000002e-05 2024-03-19T11:31:21.764 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.764 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.764 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:21.764 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455846+0000", 2024-03-19T11:31:21.764 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6030000000000005e-06 2024-03-19T11:31:21.764 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.764 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.764 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:21.764 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.076525+0000", 2024-03-19T11:31:21.764 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.62067875900000002 2024-03-19T11:31:21.764 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.764 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.765 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:31:21.765 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.076981+0000", 2024-03-19T11:31:21.765 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00045633300000000001 2024-03-19T11:31:21.765 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.765 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.765 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:21.765 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.104422+0000", 2024-03-19T11:31:21.765 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027440665999999999 2024-03-19T11:31:21.765 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.765 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.765 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:31:21.765 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.104786+0000", 2024-03-19T11:31:21.765 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00036471499999999999 2024-03-19T11:31:21.765 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.765 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.766 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:21.766 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.140725+0000", 2024-03-19T11:31:21.766 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.035939093999999998 2024-03-19T11:31:21.766 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.766 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.766 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:31:21.766 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.140841+0000", 2024-03-19T11:31:21.766 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000115207 2024-03-19T11:31:21.766 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.766 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.767 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:21.767 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.145559+0000", 2024-03-19T11:31:21.767 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.004718486 2024-03-19T11:31:21.767 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.767 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.767 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:21.767 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.145760+0000", 2024-03-19T11:31:21.767 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000201058 2024-03-19T11:31:21.767 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.767 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.767 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:21.767 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.147237+0000", 2024-03-19T11:31:21.768 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014770370000000001 2024-03-19T11:31:21.768 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:21.768 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:21.768 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:21.768 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.768 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.768 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.16 80/79 [PushOp(4:6d9093b1:::smithi03829856-38:head, version: 77'90, data_included: [500875~689757,1564955~358819], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:6d9093b1:::smithi03829856-38:head@77'90, size: 2100875, copy_subset: [(0, 2100875)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1923774, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:6be89c06:::smithi03829856-24:head, version: 77'92, data_included: [741376~589824,1753088~458752], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:6be89c06:::smithi03829856-24:head@77'92, size: 2386561, copy_subset: [(0, 2386561)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2211840, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:21.768 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:59.081830+0000", 2024-03-19T11:31:21.768 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 22.645895441, 2024-03-19T11:31:21.768 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015229652999999999, 2024-03-19T11:31:21.768 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:21.769 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:21.769 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:21.769 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.769 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:21.769 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.081830+0000", 2024-03-19T11:31:21.769 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:21.769 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.769 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.769 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:21.770 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.081830+0000", 2024-03-19T11:31:21.770 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:21.770 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.770 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.770 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:21.770 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.081832+0000", 2024-03-19T11:31:21.770 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.731e-06 2024-03-19T11:31:21.770 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.770 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.770 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:21.770 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.083140+0000", 2024-03-19T11:31:21.771 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0013082829999999999 2024-03-19T11:31:21.771 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.771 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.771 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:21.771 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.083170+0000", 2024-03-19T11:31:21.771 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0270999999999999e-05 2024-03-19T11:31:21.771 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.771 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.771 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:21.771 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.083199+0000", 2024-03-19T11:31:21.771 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9298e-05 2024-03-19T11:31:21.772 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.772 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.772 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:21.772 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.089969+0000", 2024-03-19T11:31:21.772 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0067691289999999996 2024-03-19T11:31:21.772 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.772 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.772 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:21.772 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.089995+0000", 2024-03-19T11:31:21.772 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5995000000000001e-05 2024-03-19T11:31:21.772 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.772 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.773 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:21.773 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.097060+0000", 2024-03-19T11:31:21.773 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0070649459999999999 2024-03-19T11:31:21.773 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:21.773 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:21.773 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:21.773 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.773 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.773 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.6 80/79 [PushOp(4:626acd4d:::smithi03829856-2:8, version: 55'54, data_included: [416807~475249,1542129~474678], data_size: 949927, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:626acd4d:::smithi03829856-2:8@55'54, size: 2016807, copy_subset: [(0, 2016807)], clone_subset: {}, snapset: 15=[]:{8: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2016807, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:21.773 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:59.110928+0000", 2024-03-19T11:31:21.773 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 22.616797127000002, 2024-03-19T11:31:21.774 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01605765, 2024-03-19T11:31:21.774 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:21.774 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:21.774 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:21.774 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.774 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:21.774 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.110928+0000", 2024-03-19T11:31:21.774 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:21.774 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.774 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.774 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:21.774 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.110928+0000", 2024-03-19T11:31:21.775 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:21.775 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.775 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.775 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:21.775 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.110929+0000", 2024-03-19T11:31:21.775 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6500000000000008e-07 2024-03-19T11:31:21.775 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.775 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.775 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:21.775 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.111446+0000", 2024-03-19T11:31:21.775 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00051713900000000001 2024-03-19T11:31:21.775 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.776 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.776 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:21.776 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.111464+0000", 2024-03-19T11:31:21.776 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7966e-05 2024-03-19T11:31:21.776 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.776 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.776 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:21.776 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.111483+0000", 2024-03-19T11:31:21.776 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8561000000000001e-05 2024-03-19T11:31:21.776 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.776 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.776 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:21.777 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.118569+0000", 2024-03-19T11:31:21.777 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0070860259999999996 2024-03-19T11:31:21.777 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.777 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.777 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:21.777 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.118586+0000", 2024-03-19T11:31:21.777 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6857000000000001e-05 2024-03-19T11:31:21.777 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.777 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.778 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:21.778 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.126986+0000", 2024-03-19T11:31:21.778 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0084001360000000008 2024-03-19T11:31:21.778 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:21.778 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:21.778 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:21.778 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.778 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.778 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.16 80/79 [PushOp(4:6868af91:::smithi03829856-10:10, version: 70'81, data_included: [524385~820472,1602039~228104], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:10@70'81, size: 3747928, copy_subset: [(0, 3747928)], clone_subset: {}, snapset: 14=[]:{7: [6], b: [b], 10: [d], 14: [13, 12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1830143, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:21.778 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:59.111343+0000", 2024-03-19T11:31:21.779 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 22.616382537, 2024-03-19T11:31:21.779 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019054888999999998, 2024-03-19T11:31:21.779 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:21.779 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:21.779 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:21.779 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.779 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:21.779 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.111343+0000", 2024-03-19T11:31:21.779 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:21.779 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.779 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.780 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:21.780 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.111343+0000", 2024-03-19T11:31:21.780 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:21.780 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.780 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.780 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:21.780 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.111349+0000", 2024-03-19T11:31:21.780 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9619999999999998e-06 2024-03-19T11:31:21.780 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.780 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.781 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:21.781 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.113881+0000", 2024-03-19T11:31:21.781 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025322930000000001 2024-03-19T11:31:21.781 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.781 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.781 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:21.781 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.113917+0000", 2024-03-19T11:31:21.781 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5593999999999998e-05 2024-03-19T11:31:21.781 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.781 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.781 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:21.781 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.113962+0000", 2024-03-19T11:31:21.781 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5362999999999999e-05 2024-03-19T11:31:21.781 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.782 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.782 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:21.782 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.126871+0000", 2024-03-19T11:31:21.782 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012909273000000001 2024-03-19T11:31:21.782 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.782 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.782 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:21.782 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.126892+0000", 2024-03-19T11:31:21.782 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0678000000000001e-05 2024-03-19T11:31:21.782 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.783 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.783 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:21.783 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.130398+0000", 2024-03-19T11:31:21.783 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0035057259999999998 2024-03-19T11:31:21.783 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:21.783 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:21.783 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:21.783 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.783 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.783 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.16 80/79 [PushOp(4:6d9093b1:::smithi03829856-38:10, version: 70'83, data_included: [621859~568773,1729599~479803], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6d9093b1:::smithi03829856-38:10@70'83, size: 3142876, copy_subset: [(0, 3142876)], clone_subset: {}, snapset: 10=[]:{10: [d, b, 6]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2209402, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:21.783 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:59.114057+0000", 2024-03-19T11:31:21.783 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 22.613668508, 2024-03-19T11:31:21.783 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019935583, 2024-03-19T11:31:21.784 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:21.784 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:21.784 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:21.784 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.784 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:21.784 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.114057+0000", 2024-03-19T11:31:21.784 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:21.784 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.784 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.784 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:21.784 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.114057+0000", 2024-03-19T11:31:21.784 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:21.784 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.785 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.785 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:21.785 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.114058+0000", 2024-03-19T11:31:21.785 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4300000000000001e-07 2024-03-19T11:31:21.785 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.785 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.785 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:21.785 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.117243+0000", 2024-03-19T11:31:21.785 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031852550000000001 2024-03-19T11:31:21.785 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.785 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.785 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:21.785 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.117263+0000", 2024-03-19T11:31:21.785 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0469999999999999e-05 2024-03-19T11:31:21.786 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.786 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.786 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:21.786 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.117276+0000", 2024-03-19T11:31:21.786 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2775000000000001e-05 2024-03-19T11:31:21.786 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.786 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.786 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:21.786 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.130681+0000", 2024-03-19T11:31:21.786 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013404879999999999 2024-03-19T11:31:21.786 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.786 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.786 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:21.787 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.130699+0000", 2024-03-19T11:31:21.787 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8343e-05 2024-03-19T11:31:21.787 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.787 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.787 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:21.787 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.133992+0000", 2024-03-19T11:31:21.787 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0032929169999999998 2024-03-19T11:31:21.787 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:21.787 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:21.787 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:21.787 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.787 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.787 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2442 4.16 4:6d9093b1:::smithi03829856-38:head [write 460558~707131] snapc 17=[17,16,15,13,12,b,6] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-03-19T11:31:21.787 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:03.432382+0000", 2024-03-19T11:31:21.788 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.295342956999999, 2024-03-19T11:31:21.788 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019918379, 2024-03-19T11:31:21.788 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:21.788 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:21.788 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:21.788 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:21.788 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:21.788 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2442 2024-03-19T11:31:21.788 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.788 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:21.788 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.788 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:21.789 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.432382+0000", 2024-03-19T11:31:21.789 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:21.789 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.789 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.789 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:21.789 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.432382+0000", 2024-03-19T11:31:21.789 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:21.789 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.789 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.789 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:21.789 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.432385+0000", 2024-03-19T11:31:21.789 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1e-06 2024-03-19T11:31:21.789 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.789 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.789 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:21.790 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.433115+0000", 2024-03-19T11:31:21.790 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00072939900000000004 2024-03-19T11:31:21.790 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.790 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.790 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:21.790 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.433151+0000", 2024-03-19T11:31:21.790 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6094999999999999e-05 2024-03-19T11:31:21.790 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.790 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.790 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:21.790 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.433167+0000", 2024-03-19T11:31:21.790 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6152999999999999e-05 2024-03-19T11:31:21.790 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.790 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.791 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:21.791 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.433339+0000", 2024-03-19T11:31:21.791 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000171478 2024-03-19T11:31:21.791 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.791 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.791 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:21.791 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.433621+0000", 2024-03-19T11:31:21.791 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00028283900000000001 2024-03-19T11:31:21.791 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.791 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.791 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3", 2024-03-19T11:31:21.791 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.434169+0000", 2024-03-19T11:31:21.791 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00054786699999999995 2024-03-19T11:31:21.791 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.791 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.792 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:21.792 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.443136+0000", 2024-03-19T11:31:21.792 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0089667419999999998 2024-03-19T11:31:21.792 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.792 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.792 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:21.792 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.452230+0000", 2024-03-19T11:31:21.792 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.009093996 2024-03-19T11:31:21.792 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.792 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.792 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:21.792 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.452264+0000", 2024-03-19T11:31:21.792 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4381000000000002e-05 2024-03-19T11:31:21.792 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.792 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.793 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:21.793 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:03.452301+0000", 2024-03-19T11:31:21.793 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6328999999999997e-05 2024-03-19T11:31:21.793 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:21.793 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:21.793 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:21.793 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.793 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.793 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.4.13:82 3.3 3:f6b6d21a:::smithi03829856-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-03-19T11:31:21.793 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:04.069448+0000", 2024-03-19T11:31:21.793 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 17.658277682000001, 2024-03-19T11:31:21.793 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.086427773999999999, 2024-03-19T11:31:21.793 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:21.793 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:21.794 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:21.794 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.794 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:21.794 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069448+0000", 2024-03-19T11:31:21.794 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:21.794 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.794 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.794 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:21.794 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069448+0000", 2024-03-19T11:31:21.794 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:21.794 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.794 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.794 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:21.795 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069449+0000", 2024-03-19T11:31:21.795 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8160000000000001e-06 2024-03-19T11:31:21.795 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.795 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.795 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:21.795 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069458+0000", 2024-03-19T11:31:21.795 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2900000000000002e-06 2024-03-19T11:31:21.795 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.795 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.795 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:21.795 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069491+0000", 2024-03-19T11:31:21.795 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3229000000000003e-05 2024-03-19T11:31:21.795 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.795 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.795 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:21.796 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069501+0000", 2024-03-19T11:31:21.796 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7629999999999995e-06 2024-03-19T11:31:21.796 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.796 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.796 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:21.796 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069573+0000", 2024-03-19T11:31:21.796 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2349999999999997e-05 2024-03-19T11:31:21.796 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.796 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.796 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:21.796 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069581+0000", 2024-03-19T11:31:21.796 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.388e-06 2024-03-19T11:31:21.796 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.797 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.797 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:21.797 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.075451+0000", 2024-03-19T11:31:21.797 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005869956 2024-03-19T11:31:21.797 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.797 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.797 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:21.797 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.077511+0000", 2024-03-19T11:31:21.797 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0020592610000000002 2024-03-19T11:31:21.797 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.797 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.797 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:21.797 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.108781+0000", 2024-03-19T11:31:21.797 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.031270344999999998 2024-03-19T11:31:21.797 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.798 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.798 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:21.798 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.108895+0000", 2024-03-19T11:31:21.798 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000114059 2024-03-19T11:31:21.798 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.798 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.798 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:21.798 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.109065+0000", 2024-03-19T11:31:21.798 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016979899999999999 2024-03-19T11:31:21.798 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.798 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.798 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:21.799 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.111730+0000", 2024-03-19T11:31:21.799 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0026651410000000002 2024-03-19T11:31:21.799 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.799 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.799 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:21.799 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.155616+0000", 2024-03-19T11:31:21.799 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.043885986000000002 2024-03-19T11:31:21.799 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.799 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.799 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:21.799 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.155745+0000", 2024-03-19T11:31:21.799 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012948899999999999 2024-03-19T11:31:21.799 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.799 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.799 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:21.800 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.155875+0000", 2024-03-19T11:31:21.800 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012990199999999999 2024-03-19T11:31:21.800 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:21.800 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:21.800 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:21.800 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.800 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.800 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.4.13:83 3.3 3:f6b6d21a:::smithi03829856-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-03-19T11:31:21.800 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:04.069526+0000", 2024-03-19T11:31:21.800 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 17.658198901999999, 2024-03-19T11:31:21.800 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.087140880000000004, 2024-03-19T11:31:21.800 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:21.800 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:21.801 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:21.802 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.802 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:21.802 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069526+0000", 2024-03-19T11:31:21.803 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:21.803 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.803 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.803 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:21.803 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069526+0000", 2024-03-19T11:31:21.803 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:21.803 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.803 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.803 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:21.803 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069527+0000", 2024-03-19T11:31:21.803 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2799999999999996e-07 2024-03-19T11:31:21.803 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.803 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.804 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:21.804 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069531+0000", 2024-03-19T11:31:21.804 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1489999999999996e-06 2024-03-19T11:31:21.804 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.804 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.804 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:21.804 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069539+0000", 2024-03-19T11:31:21.804 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3189999999999995e-06 2024-03-19T11:31:21.804 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.804 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.804 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:21.805 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069545+0000", 2024-03-19T11:31:21.805 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.592e-06 2024-03-19T11:31:21.805 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.805 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.805 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:21.805 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069616+0000", 2024-03-19T11:31:21.805 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1130000000000005e-05 2024-03-19T11:31:21.805 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.805 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.805 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:21.805 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069628+0000", 2024-03-19T11:31:21.805 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1697e-05 2024-03-19T11:31:21.805 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.805 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.806 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:21.806 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.077594+0000", 2024-03-19T11:31:21.806 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0079656989999999997 2024-03-19T11:31:21.806 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.806 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.806 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:21.806 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.077674+0000", 2024-03-19T11:31:21.806 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0338000000000002e-05 2024-03-19T11:31:21.806 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.806 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.807 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:21.807 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.112771+0000", 2024-03-19T11:31:21.807 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.035097325999999998 2024-03-19T11:31:21.807 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.807 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.807 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:21.807 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.112882+0000", 2024-03-19T11:31:21.807 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000110949 2024-03-19T11:31:21.807 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.807 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.807 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:21.808 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.113031+0000", 2024-03-19T11:31:21.808 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000148355 2024-03-19T11:31:21.808 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.808 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.808 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:21.808 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.114076+0000", 2024-03-19T11:31:21.808 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001045009 2024-03-19T11:31:21.808 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.808 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.808 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:21.809 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156482+0000", 2024-03-19T11:31:21.809 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.042406354 2024-03-19T11:31:21.809 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.809 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.809 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:21.809 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156549+0000", 2024-03-19T11:31:21.809 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7229000000000003e-05 2024-03-19T11:31:21.809 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.809 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.809 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:21.809 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156667+0000", 2024-03-19T11:31:21.809 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000118106 2024-03-19T11:31:21.809 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:21.810 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:21.810 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:21.810 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.810 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.810 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.4.13:84 3.3 3:ce6951d3:::smithi03829856-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-03-19T11:31:21.810 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:04.071995+0000", 2024-03-19T11:31:21.810 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 17.655730126999998, 2024-03-19T11:31:21.810 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.084172497999999998, 2024-03-19T11:31:21.810 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:21.810 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:21.810 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:21.811 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.811 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:21.811 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.071995+0000", 2024-03-19T11:31:21.811 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:21.811 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.811 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.811 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:21.811 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.071995+0000", 2024-03-19T11:31:21.811 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:21.811 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.812 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.812 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:21.812 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.071997+0000", 2024-03-19T11:31:21.812 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6029999999999999e-06 2024-03-19T11:31:21.812 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.812 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.812 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:21.812 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072004+0000", 2024-03-19T11:31:21.812 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8600000000000004e-06 2024-03-19T11:31:21.813 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.813 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.813 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:21.813 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072033+0000", 2024-03-19T11:31:21.813 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9836999999999999e-05 2024-03-19T11:31:21.813 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.813 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.813 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:21.813 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072042+0000", 2024-03-19T11:31:21.814 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9279999999999999e-06 2024-03-19T11:31:21.814 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.814 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.814 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:21.814 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072123+0000", 2024-03-19T11:31:21.814 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0630000000000006e-05 2024-03-19T11:31:21.814 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.814 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.814 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:21.815 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072132+0000", 2024-03-19T11:31:21.815 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5560000000000008e-06 2024-03-19T11:31:21.815 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.815 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.815 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:21.815 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.077712+0000", 2024-03-19T11:31:21.815 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055804970000000002 2024-03-19T11:31:21.815 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.815 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.815 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:21.815 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.079495+0000", 2024-03-19T11:31:21.815 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017831850000000001 2024-03-19T11:31:21.815 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.816 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.816 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:21.816 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.109825+0000", 2024-03-19T11:31:21.816 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.030329819000000001 2024-03-19T11:31:21.816 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.816 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.816 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:21.816 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.109904+0000", 2024-03-19T11:31:21.816 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9269000000000003e-05 2024-03-19T11:31:21.816 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.816 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.816 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:21.817 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.110085+0000", 2024-03-19T11:31:21.817 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00018018600000000001 2024-03-19T11:31:21.817 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.817 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.817 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:21.817 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.113581+0000", 2024-03-19T11:31:21.817 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034962919999999998 2024-03-19T11:31:21.817 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.817 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.817 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:21.817 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.155925+0000", 2024-03-19T11:31:21.817 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.042344404000000002 2024-03-19T11:31:21.818 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.818 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.818 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:21.818 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156026+0000", 2024-03-19T11:31:21.818 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000100599 2024-03-19T11:31:21.818 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.818 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.818 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:21.819 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156168+0000", 2024-03-19T11:31:21.819 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014183300000000001 2024-03-19T11:31:21.819 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:21.819 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:21.819 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:21.820 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.820 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.820 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.4.13:85 3.3 3:ce6951d3:::smithi03829856-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-03-19T11:31:21.820 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:04.072068+0000", 2024-03-19T11:31:21.820 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 17.655657490999999, 2024-03-19T11:31:21.820 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.084368466000000003, 2024-03-19T11:31:21.820 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:21.820 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:21.820 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:21.820 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.820 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:21.821 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072068+0000", 2024-03-19T11:31:21.821 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:21.821 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.821 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.821 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:21.821 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072068+0000", 2024-03-19T11:31:21.821 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:21.821 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.822 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.822 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:21.822 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072068+0000", 2024-03-19T11:31:21.822 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9699999999999996e-07 2024-03-19T11:31:21.822 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.822 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.822 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:21.822 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072074+0000", 2024-03-19T11:31:21.822 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2279999999999998e-06 2024-03-19T11:31:21.823 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.823 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.823 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:21.823 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072140+0000", 2024-03-19T11:31:21.823 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6638999999999997e-05 2024-03-19T11:31:21.823 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.823 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.823 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:21.823 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072147+0000", 2024-03-19T11:31:21.823 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8750000000000002e-06 2024-03-19T11:31:21.824 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.824 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.824 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:21.824 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072201+0000", 2024-03-19T11:31:21.824 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3634000000000002e-05 2024-03-19T11:31:21.824 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.824 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.824 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:21.824 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072211+0000", 2024-03-19T11:31:21.824 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0589e-05 2024-03-19T11:31:21.825 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.825 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.825 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:21.825 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.079687+0000", 2024-03-19T11:31:21.825 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074757219999999997 2024-03-19T11:31:21.825 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.825 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.825 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:21.825 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.079769+0000", 2024-03-19T11:31:21.825 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2251999999999998e-05 2024-03-19T11:31:21.826 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.826 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.826 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:21.826 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.111798+0000", 2024-03-19T11:31:21.826 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.032028889999999997 2024-03-19T11:31:21.826 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.826 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.826 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:21.826 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.111902+0000", 2024-03-19T11:31:21.826 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000103773 2024-03-19T11:31:21.826 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.827 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.827 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:21.827 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.112073+0000", 2024-03-19T11:31:21.827 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017102799999999999 2024-03-19T11:31:21.827 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.827 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.827 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:21.827 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.113591+0000", 2024-03-19T11:31:21.827 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0015179569999999999 2024-03-19T11:31:21.827 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.827 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.827 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:21.827 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156223+0000", 2024-03-19T11:31:21.827 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.042632153999999998 2024-03-19T11:31:21.828 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.828 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.828 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:21.828 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156321+0000", 2024-03-19T11:31:21.828 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8246999999999996e-05 2024-03-19T11:31:21.828 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.828 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.828 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:21.828 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156436+0000", 2024-03-19T11:31:21.828 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000114881 2024-03-19T11:31:21.828 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:21.828 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:21.828 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:21.828 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.828 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.829 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2455 3.1 3:8e10ce27:::f25cf4cdffbae9b54315ae93e2cdc514e3bf58d758c8445671e9463006f381a6:head [write 0~1661440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-03-19T11:31:21.829 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:04.086189+0000", 2024-03-19T11:31:21.829 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 17.641536344999999, 2024-03-19T11:31:21.829 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015194925999999999, 2024-03-19T11:31:21.829 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:21.829 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:21.829 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:21.829 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:21.829 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:21.829 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2455 2024-03-19T11:31:21.829 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.829 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:21.829 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.829 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:21.829 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.086189+0000", 2024-03-19T11:31:21.830 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:21.830 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.830 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.830 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:21.830 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.086189+0000", 2024-03-19T11:31:21.830 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:21.830 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.830 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.830 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:21.830 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.086191+0000", 2024-03-19T11:31:21.830 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8619999999999999e-06 2024-03-19T11:31:21.830 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.830 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.830 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:21.831 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.086944+0000", 2024-03-19T11:31:21.831 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00075297299999999999 2024-03-19T11:31:21.831 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.831 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.831 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:21.831 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.086966+0000", 2024-03-19T11:31:21.831 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2404000000000001e-05 2024-03-19T11:31:21.831 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.831 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.831 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:21.831 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.086976+0000", 2024-03-19T11:31:21.831 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8069999999999994e-06 2024-03-19T11:31:21.831 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.831 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.832 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:21.832 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.087038+0000", 2024-03-19T11:31:21.832 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2314e-05 2024-03-19T11:31:21.832 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.832 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.832 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:21.832 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.087158+0000", 2024-03-19T11:31:21.832 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000119385 2024-03-19T11:31:21.832 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.832 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.832 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:21.832 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.087286+0000", 2024-03-19T11:31:21.832 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012851599999999999 2024-03-19T11:31:21.832 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.833 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.833 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:21.833 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.089954+0000", 2024-03-19T11:31:21.833 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0026674239999999998 2024-03-19T11:31:21.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.833 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.833 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:21.833 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.101235+0000", 2024-03-19T11:31:21.833 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011281807 2024-03-19T11:31:21.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.833 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.833 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:21.833 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.101321+0000", 2024-03-19T11:31:21.833 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5359999999999996e-05 2024-03-19T11:31:21.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.834 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.834 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:21.834 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.101384+0000", 2024-03-19T11:31:21.834 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3074e-05 2024-03-19T11:31:21.834 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:21.834 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:21.834 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:21.834 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.834 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.834 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2593 4.6 4:626acd4d:::smithi03829856-2:head [write 662242~642825] snapc 17=[17,16,15,b] ondisk+write+known_if_redirected+supports_pool_eio e88)", 2024-03-19T11:31:21.834 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:07.785391+0000", 2024-03-19T11:31:21.834 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 13.942334077, 2024-03-19T11:31:21.835 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018221268999999998, 2024-03-19T11:31:21.835 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:21.835 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:21.835 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:21.835 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:21.835 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:21.835 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2593 2024-03-19T11:31:21.835 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.835 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:21.835 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.835 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:21.835 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.785391+0000", 2024-03-19T11:31:21.835 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:21.835 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.836 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.836 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:21.836 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.785391+0000", 2024-03-19T11:31:21.836 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:21.836 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.836 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.836 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:21.836 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.785394+0000", 2024-03-19T11:31:21.836 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.903e-06 2024-03-19T11:31:21.836 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.836 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.836 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:21.836 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.786080+0000", 2024-03-19T11:31:21.837 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000686201 2024-03-19T11:31:21.837 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.837 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.837 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:21.837 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.786116+0000", 2024-03-19T11:31:21.837 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5441999999999997e-05 2024-03-19T11:31:21.837 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.837 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.837 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:21.837 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.786132+0000", 2024-03-19T11:31:21.837 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6353000000000001e-05 2024-03-19T11:31:21.837 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.837 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.838 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:21.838 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.786283+0000", 2024-03-19T11:31:21.838 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015133599999999999 2024-03-19T11:31:21.838 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.838 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.838 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:21.838 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.786457+0000", 2024-03-19T11:31:21.838 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017352200000000001 2024-03-19T11:31:21.838 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.839 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.839 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1", 2024-03-19T11:31:21.839 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.786765+0000", 2024-03-19T11:31:21.839 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030768199999999999 2024-03-19T11:31:21.839 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.839 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.839 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:21.839 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.798155+0000", 2024-03-19T11:31:21.839 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011390045999999999 2024-03-19T11:31:21.839 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.839 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.839 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:21.839 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.803291+0000", 2024-03-19T11:31:21.839 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0051363060000000002 2024-03-19T11:31:21.840 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.840 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.840 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:21.840 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.803466+0000", 2024-03-19T11:31:21.840 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017505900000000001 2024-03-19T11:31:21.840 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:21.840 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:21.840 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:21.840 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:07.803612+0000", 2024-03-19T11:31:21.840 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014641900000000001 2024-03-19T11:31:21.840 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:21.841 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:21.841 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:21.841 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:21.841 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:21.841 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:21.843 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-19T11:31:21.843 INFO:tasks.rados.rados.0.smithi038.stdout:1393: finishing rollback tid 0 to smithi03829856-41 2024-03-19T11:31:21.843 INFO:tasks.rados.rados.0.smithi038.stdout:1393: finishing rollback tid 1 to smithi03829856-41 2024-03-19T11:31:21.843 INFO:tasks.rados.rados.0.smithi038.stdout:1393: finishing rollback tid 2 to smithi03829856-41 2024-03-19T11:31:21.843 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 41 v 140 (ObjNum 170 snap 17 seq_num 170) dirty exists 2024-03-19T11:31:21.844 INFO:tasks.rados.rados.0.smithi038.stdout:1387: done (6 left) 2024-03-19T11:31:21.844 INFO:tasks.rados.rados.0.smithi038.stdout:1388: done (5 left) 2024-03-19T11:31:21.844 INFO:tasks.rados.rados.0.smithi038.stdout:1389: done (4 left) 2024-03-19T11:31:21.844 INFO:tasks.rados.rados.0.smithi038.stdout:1390: done (3 left) 2024-03-19T11:31:21.844 INFO:tasks.rados.rados.0.smithi038.stdout:1391: done (2 left) 2024-03-19T11:31:21.844 INFO:tasks.rados.rados.0.smithi038.stdout:1392: done (1 left) 2024-03-19T11:31:21.844 INFO:tasks.rados.rados.0.smithi038.stdout:1393: done (0 left) 2024-03-19T11:31:21.844 INFO:tasks.rados.rados.0.smithi038.stdout:1394: snap_create 2024-03-19T11:31:21.844 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:21.850+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:21.943 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:21.949+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:22.014 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:22.014 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:31:22.014 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:31:22.014 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:22.029 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-19T11:31:22.072 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:22.081+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:22.115 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-03-19T11:31:22.127 INFO:tasks.rados.rados.0.smithi038.stdout:1394: done (0 left) 2024-03-19T11:31:22.127 INFO:tasks.rados.rados.0.smithi038.stdout:1395: write (excl) oid 33 current snap is 30 2024-03-19T11:31:22.127 INFO:tasks.rados.rados.0.smithi038.stdout:1395: seq_num 238 ranges {400066=716205,1860832=139234} 2024-03-19T11:31:22.148 INFO:tasks.rados.rados.0.smithi038.stdout:1395: writing smithi03829856-33 from 400066 to 1116271 tid 1 2024-03-19T11:31:22.148 INFO:tasks.rados.rados.0.smithi038.stdout:1395: writing smithi03829856-33 from 1860832 to 2000066 tid 2 2024-03-19T11:31:22.148 INFO:tasks.rados.rados.0.smithi038.stdout:1396: read oid 49 snap -1 2024-03-19T11:31:22.148 INFO:tasks.rados.rados.0.smithi038.stdout:1396: expect (ObjNum 234 snap 28 seq_num 234) 2024-03-19T11:31:22.148 INFO:tasks.rados.rados.0.smithi038.stdout:1397: write (excl) oid 23 current snap is 30 2024-03-19T11:31:22.148 INFO:tasks.rados.rados.0.smithi038.stdout:1397: seq_num 239 ranges {416873=791454,1919314=97559} 2024-03-19T11:31:22.149 INFO:tasks.rados.rados.0.smithi038.stdout:1397: writing smithi03829856-23 from 416873 to 1208327 tid 1 2024-03-19T11:31:22.149 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:22.157+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:22.153 INFO:tasks.rados.rados.0.smithi038.stdout:1397: writing smithi03829856-23 from 1919314 to 2016873 tid 2 2024-03-19T11:31:22.153 INFO:tasks.rados.rados.0.smithi038.stdout:1398: read oid 7 snap -1 2024-03-19T11:31:22.153 INFO:tasks.rados.rados.0.smithi038.stdout:1398: expect (ObjNum 210 snap 22 seq_num 210) 2024-03-19T11:31:22.153 INFO:tasks.rados.rados.0.smithi038.stdout:1399: read oid 15 snap -1 2024-03-19T11:31:22.153 INFO:tasks.rados.rados.0.smithi038.stdout:1399: expect (ObjNum 209 snap 22 seq_num 209) 2024-03-19T11:31:22.154 INFO:tasks.rados.rados.0.smithi038.stdout:1400: set_chunk oid 13 target oid 2024-03-19T11:31:22.154 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:22.162+0000 7f1837407640 -1 osd.6 102 handle_osd_map injecting map crc failure 2024-03-19T11:31:22.175 INFO:tasks.rados.rados.0.smithi038.stdout:1400: set_chunk oid 13 offset: 1289216 length: 345600 target oid 1ce01a290bb5d79f279d24914c9537cc41c2bb0eef7036371d1ef54cef02d8fa offset: 0 snap -1 2024-03-19T11:31:22.175 INFO:tasks.rados.rados.0.smithi038.stdout:1401: read oid 32 snap 22 2024-03-19T11:31:22.175 INFO:tasks.rados.rados.0.smithi038.stdout:1401: expect (ObjNum 31 snap 0 seq_num 31) 2024-03-19T11:31:22.175 INFO:tasks.rados.rados.0.smithi038.stdout:1402: set_chunk oid 47 target oid 2024-03-19T11:31:22.190 INFO:tasks.rados.rados.0.smithi038.stdout:1402: set_chunk oid 47 offset: 1090048 length: 1353728 target oid 935fbb77ab9b2f6d67968c5b7a9f3462133a6b0f5b914fd609f0538d5e8c429f offset: 0 snap 25 2024-03-19T11:31:22.191 INFO:tasks.rados.rados.0.smithi038.stdout:1403: tier_evict oid 16 2024-03-19T11:31:22.191 INFO:tasks.rados.rados.0.smithi038.stdout:1403: tier_evict oid 16 snap -1 2024-03-19T11:31:22.191 INFO:tasks.rados.rados.0.smithi038.stdout:1396: expect (ObjNum 234 snap 28 seq_num 234) 2024-03-19T11:31:22.221 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:22.221 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:31:22.222 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:31:22.222 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:31:22.222 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:22.235 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-19T11:31:22.280 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:22.289+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:22.346 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:22.353+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:22.404 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:22.405 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:31:22.405 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:31:22.405 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:31:22.405 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.405 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1063 4.4 4:214026b3:::smithi03829856-47:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:31:22.405 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.784904+0000", 2024-03-19T11:31:22.405 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 68.630089161000001, 2024-03-19T11:31:22.405 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024425912000000001, 2024-03-19T11:31:22.405 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:22.405 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:22.405 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:22.405 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:22.405 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:22.406 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1063 2024-03-19T11:31:22.406 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.406 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:22.406 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.406 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:22.406 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784904+0000", 2024-03-19T11:31:22.406 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:22.406 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.406 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.406 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:22.406 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784904+0000", 2024-03-19T11:31:22.406 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:22.407 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.407 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.407 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:22.407 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784906+0000", 2024-03-19T11:31:22.407 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1409999999999999e-06 2024-03-19T11:31:22.407 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.407 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.407 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:22.407 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784915+0000", 2024-03-19T11:31:22.407 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5459999999999992e-06 2024-03-19T11:31:22.407 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.407 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.407 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:22.408 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784935+0000", 2024-03-19T11:31:22.408 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0310999999999999e-05 2024-03-19T11:31:22.408 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.408 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.408 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:22.408 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784946+0000", 2024-03-19T11:31:22.408 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0675e-05 2024-03-19T11:31:22.408 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.408 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.408 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:22.408 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.785032+0000", 2024-03-19T11:31:22.408 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6049000000000002e-05 2024-03-19T11:31:22.408 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.408 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.408 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:22.409 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.785136+0000", 2024-03-19T11:31:22.409 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000104209 2024-03-19T11:31:22.409 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.409 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.409 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:22.409 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.809330+0000", 2024-03-19T11:31:22.409 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024193981 2024-03-19T11:31:22.409 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:22.409 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:22.409 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:22.409 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.409 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.410 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1433 4.16 4:6be89c06:::smithi03829856-24:c [sparse-read 0~1100427,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:31:22.410 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.976652+0000", 2024-03-19T11:31:22.410 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 56.438341459999997, 2024-03-19T11:31:22.410 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95684753199999995, 2024-03-19T11:31:22.410 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:22.410 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:22.410 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:22.410 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:22.410 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:22.411 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1433 2024-03-19T11:31:22.411 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.411 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:22.411 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.411 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:22.411 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976652+0000", 2024-03-19T11:31:22.411 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:22.411 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.411 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.411 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:22.411 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976652+0000", 2024-03-19T11:31:22.411 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:22.411 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.412 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.412 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:22.412 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976654+0000", 2024-03-19T11:31:22.412 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6199999999999999e-06 2024-03-19T11:31:22.412 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.412 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.412 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:22.412 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976668+0000", 2024-03-19T11:31:22.412 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3213999999999999e-05 2024-03-19T11:31:22.412 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.412 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.412 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:22.413 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976694+0000", 2024-03-19T11:31:22.413 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6488000000000001e-05 2024-03-19T11:31:22.413 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.413 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.413 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:22.413 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976705+0000", 2024-03-19T11:31:22.413 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1044e-05 2024-03-19T11:31:22.413 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.413 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.413 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:22.413 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976795+0000", 2024-03-19T11:31:22.413 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0283000000000003e-05 2024-03-19T11:31:22.413 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.414 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.414 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:22.414 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976811+0000", 2024-03-19T11:31:22.414 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5262000000000001e-05 2024-03-19T11:31:22.414 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.414 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.414 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:22.414 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.927337+0000", 2024-03-19T11:31:22.414 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95052597100000003 2024-03-19T11:31:22.414 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.414 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.414 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:22.414 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.927495+0000", 2024-03-19T11:31:22.415 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015788600000000001 2024-03-19T11:31:22.415 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.415 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.415 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:22.415 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.933499+0000", 2024-03-19T11:31:22.415 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0060047640000000001 2024-03-19T11:31:22.415 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:22.415 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:22.415 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:22.415 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.415 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.415 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1434 4.16 4:6be89c06:::smithi03829856-24:c [read 0~2386561,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:31:22.415 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.976739+0000", 2024-03-19T11:31:22.415 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 56.438254299, 2024-03-19T11:31:22.416 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95781317399999999, 2024-03-19T11:31:22.416 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:22.416 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:22.416 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:22.416 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:22.416 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:22.416 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1434 2024-03-19T11:31:22.416 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.416 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:22.416 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.416 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:22.416 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976739+0000", 2024-03-19T11:31:22.416 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:22.416 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.417 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.417 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:22.417 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976739+0000", 2024-03-19T11:31:22.417 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:22.417 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.417 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.417 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:22.417 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976740+0000", 2024-03-19T11:31:22.417 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8800000000000001e-07 2024-03-19T11:31:22.417 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.417 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.417 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:22.417 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976748+0000", 2024-03-19T11:31:22.417 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8979999999999996e-06 2024-03-19T11:31:22.418 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.418 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.418 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:22.418 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976757+0000", 2024-03-19T11:31:22.418 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9439999999999994e-06 2024-03-19T11:31:22.418 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.418 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.418 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:22.418 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976763+0000", 2024-03-19T11:31:22.418 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5030000000000002e-06 2024-03-19T11:31:22.418 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.418 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.418 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:22.419 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976849+0000", 2024-03-19T11:31:22.419 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5803999999999994e-05 2024-03-19T11:31:22.419 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.419 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.419 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:22.419 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976860+0000", 2024-03-19T11:31:22.419 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0869e-05 2024-03-19T11:31:22.419 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.419 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.419 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:22.419 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.933562+0000", 2024-03-19T11:31:22.419 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95670232300000002 2024-03-19T11:31:22.419 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.420 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.420 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:22.420 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.933693+0000", 2024-03-19T11:31:22.420 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013110699999999999 2024-03-19T11:31:22.420 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.420 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.420 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:22.420 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.934552+0000", 2024-03-19T11:31:22.421 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00085883800000000001 2024-03-19T11:31:22.421 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:22.421 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:22.421 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:22.421 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.421 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.421 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1435 4.16 4:6be89c06:::smithi03829856-24:c [sparse-read 0~1100427] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:31:22.421 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.976775+0000", 2024-03-19T11:31:22.421 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 56.438218628000001, 2024-03-19T11:31:22.421 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95962779499999995, 2024-03-19T11:31:22.421 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:22.422 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:22.422 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:22.422 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:22.422 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:22.422 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1435 2024-03-19T11:31:22.422 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.422 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:22.422 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.422 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:22.422 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976775+0000", 2024-03-19T11:31:22.422 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:22.422 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.423 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.423 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:22.423 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976775+0000", 2024-03-19T11:31:22.423 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:22.423 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.423 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.423 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:22.423 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976775+0000", 2024-03-19T11:31:22.423 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9800000000000003e-07 2024-03-19T11:31:22.423 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.423 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.423 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:22.423 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976779+0000", 2024-03-19T11:31:22.423 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3830000000000001e-06 2024-03-19T11:31:22.423 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.424 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.424 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:22.424 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976785+0000", 2024-03-19T11:31:22.424 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5649999999999998e-06 2024-03-19T11:31:22.424 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.424 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.424 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:22.424 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976790+0000", 2024-03-19T11:31:22.424 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1379999999999998e-06 2024-03-19T11:31:22.424 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.424 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.424 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:22.424 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976889+0000", 2024-03-19T11:31:22.424 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8331000000000004e-05 2024-03-19T11:31:22.424 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.425 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.425 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:22.425 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976894+0000", 2024-03-19T11:31:22.425 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8270000000000003e-06 2024-03-19T11:31:22.425 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.425 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.425 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:22.425 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.935982+0000", 2024-03-19T11:31:22.425 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95908714100000003 2024-03-19T11:31:22.425 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.425 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.425 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:22.425 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.936093+0000", 2024-03-19T11:31:22.425 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000111557 2024-03-19T11:31:22.425 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.426 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.426 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:22.426 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.936402+0000", 2024-03-19T11:31:22.426 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030925499999999998 2024-03-19T11:31:22.426 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:22.426 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:22.426 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:22.426 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.426 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.426 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1442 4.6 4:626acd4d:::smithi03829856-2:head [cache-evict] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:31:22.426 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.978864+0000", 2024-03-19T11:31:22.426 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 56.436128963999998, 2024-03-19T11:31:22.426 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95708845099999995, 2024-03-19T11:31:22.426 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:22.426 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:22.427 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:22.427 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:22.427 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:22.427 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1442 2024-03-19T11:31:22.427 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.427 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:22.427 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.427 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:22.427 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978864+0000", 2024-03-19T11:31:22.427 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:22.427 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.427 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.427 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:22.427 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978864+0000", 2024-03-19T11:31:22.427 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:22.428 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.428 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.428 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:22.428 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978866+0000", 2024-03-19T11:31:22.428 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0729999999999999e-06 2024-03-19T11:31:22.428 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.428 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.428 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:22.428 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978877+0000", 2024-03-19T11:31:22.428 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0390999999999999e-05 2024-03-19T11:31:22.428 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.428 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.428 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:22.428 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978902+0000", 2024-03-19T11:31:22.428 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5635000000000001e-05 2024-03-19T11:31:22.429 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.429 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.429 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:22.429 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978915+0000", 2024-03-19T11:31:22.429 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2617e-05 2024-03-19T11:31:22.429 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.429 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.429 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:22.429 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.979005+0000", 2024-03-19T11:31:22.429 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0552999999999999e-05 2024-03-19T11:31:22.429 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.429 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.429 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:22.429 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.979021+0000", 2024-03-19T11:31:22.429 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5102999999999999e-05 2024-03-19T11:31:22.430 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.430 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.430 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:22.430 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.926879+0000", 2024-03-19T11:31:22.430 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.94785817900000002 2024-03-19T11:31:22.430 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.431 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.432 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:22.432 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.927277+0000", 2024-03-19T11:31:22.432 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00039819600000000002 2024-03-19T11:31:22.432 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.432 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.432 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:22.432 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.935752+0000", 2024-03-19T11:31:22.432 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0084753420000000003 2024-03-19T11:31:22.432 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.432 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.432 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:22.432 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.935874+0000", 2024-03-19T11:31:22.432 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000121311 2024-03-19T11:31:22.432 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.433 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.433 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:22.433 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.935953+0000", 2024-03-19T11:31:22.433 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9050999999999999e-05 2024-03-19T11:31:22.433 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:22.433 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:22.433 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:22.433 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.433 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.433 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2003 4.16 4:6868af91:::smithi03829856-10:head [tier-promote] snapc 12=[12,11,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:31:22.434 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:45.804046+0000", 2024-03-19T11:31:22.434 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 36.610947050999997, 2024-03-19T11:31:22.434 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018976396, 2024-03-19T11:31:22.434 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:22.434 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:22.434 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:22.434 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:22.434 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:22.434 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2003 2024-03-19T11:31:22.434 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.435 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:22.435 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.435 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:22.435 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804046+0000", 2024-03-19T11:31:22.435 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:22.435 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.435 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.435 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:22.435 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804046+0000", 2024-03-19T11:31:22.435 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:22.435 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.436 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.436 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:22.436 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804052+0000", 2024-03-19T11:31:22.436 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7599999999999999e-06 2024-03-19T11:31:22.436 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.436 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.436 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:22.436 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804090+0000", 2024-03-19T11:31:22.436 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8018000000000001e-05 2024-03-19T11:31:22.436 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.436 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.436 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:22.436 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804256+0000", 2024-03-19T11:31:22.437 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016593599999999999 2024-03-19T11:31:22.437 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.437 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.437 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:22.437 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804300+0000", 2024-03-19T11:31:22.437 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3648e-05 2024-03-19T11:31:22.437 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.437 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.437 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:22.437 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804538+0000", 2024-03-19T11:31:22.437 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000238347 2024-03-19T11:31:22.437 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.437 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.438 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:22.438 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804716+0000", 2024-03-19T11:31:22.438 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000177968 2024-03-19T11:31:22.438 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.438 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.438 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:22.438 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.823023+0000", 2024-03-19T11:31:22.438 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018306718999999999 2024-03-19T11:31:22.438 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:22.438 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:22.438 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:22.438 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.438 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.439 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1d58db11:::smithi03829856-18:6, version: 55'80, data_included: [759683~639358,1904434~409218], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:6@55'80, size: 3159683, copy_subset: [(0, 3159683)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2313652, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1d58db11:::smithi03829856-18:b, version: 58'84, data_included: [443718~955323,1564138~93253], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:b@58'84, size: 3445402, copy_subset: [(0, 3445402)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1657391, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:10, version: 66'95, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:10@66'95, size: 3764735, copy_subset: [(0, 3764735)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:12, version: 66'97, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:12@66'97, size: 2504243, copy_subset: [(0, 2504243)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:c, version: 70'102, data_included: [416807~576167,1542129~472409], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:c@70'102, size: 2857157, copy_subset: [(0, 2857157)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2014538, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:22.439 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.275179+0000", 2024-03-19T11:31:22.439 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 28.139814060999999, 2024-03-19T11:31:22.439 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026559353000000001, 2024-03-19T11:31:22.439 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:22.439 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:22.439 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:22.439 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.439 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:22.439 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.275179+0000", 2024-03-19T11:31:22.439 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:22.439 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.439 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.440 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:22.440 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.275179+0000", 2024-03-19T11:31:22.440 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:22.440 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.440 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.440 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:22.441 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.275196+0000", 2024-03-19T11:31:22.441 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6946999999999999e-05 2024-03-19T11:31:22.441 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.441 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.441 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:22.441 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281456+0000", 2024-03-19T11:31:22.441 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0062600529999999998 2024-03-19T11:31:22.441 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.441 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.441 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:22.441 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281524+0000", 2024-03-19T11:31:22.442 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7971000000000004e-05 2024-03-19T11:31:22.442 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.442 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.442 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:22.442 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281591+0000", 2024-03-19T11:31:22.442 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6704000000000004e-05 2024-03-19T11:31:22.442 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.442 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.442 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:22.442 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281865+0000", 2024-03-19T11:31:22.442 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000274097 2024-03-19T11:31:22.442 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.442 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.443 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:22.443 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281901+0000", 2024-03-19T11:31:22.443 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5777e-05 2024-03-19T11:31:22.443 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.443 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.443 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:22.443 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.301738+0000", 2024-03-19T11:31:22.443 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019837804000000001 2024-03-19T11:31:22.443 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:22.443 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:22.443 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:22.443 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.443 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.443 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.18 e75,74 4:1d58db11:::smithi03829856-18:2,48'76)", 2024-03-19T11:31:22.444 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.281661+0000", 2024-03-19T11:31:22.444 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 28.133332530000001, 2024-03-19T11:31:22.444 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020384337999999998, 2024-03-19T11:31:22.444 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:22.444 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:22.444 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:22.444 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.444 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:22.444 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281661+0000", 2024-03-19T11:31:22.444 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:22.444 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.444 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.444 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:22.444 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281661+0000", 2024-03-19T11:31:22.445 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:22.445 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.445 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.445 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:22.445 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281661+0000", 2024-03-19T11:31:22.445 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9100000000000003e-07 2024-03-19T11:31:22.445 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.445 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.445 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:22.445 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281668+0000", 2024-03-19T11:31:22.445 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8199999999999999e-06 2024-03-19T11:31:22.445 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.445 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.445 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:22.446 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281679+0000", 2024-03-19T11:31:22.446 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1107999999999999e-05 2024-03-19T11:31:22.446 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.446 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.446 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:22.446 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281691+0000", 2024-03-19T11:31:22.446 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1399999999999999e-05 2024-03-19T11:31:22.446 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.446 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.446 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:22.446 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.301804+0000", 2024-03-19T11:31:22.446 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020113669000000001 2024-03-19T11:31:22.446 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.447 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.447 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:22.447 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.301818+0000", 2024-03-19T11:31:22.447 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3718e-05 2024-03-19T11:31:22.447 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.447 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.447 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:22.447 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302045+0000", 2024-03-19T11:31:22.448 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000226832 2024-03-19T11:31:22.448 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:22.448 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:22.448 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:22.448 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.448 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.448 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1dabc0f1:::smithi03829856-1:head, version: 70'103, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:head@70'103, size: 2621892, copy_subset: [(0, 2621892)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:22.448 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.281783+0000", 2024-03-19T11:31:22.448 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 28.133210254000002, 2024-03-19T11:31:22.448 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026187221, 2024-03-19T11:31:22.448 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:22.448 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:22.448 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:22.448 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.449 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:22.449 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281783+0000", 2024-03-19T11:31:22.449 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:22.449 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.449 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.449 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:22.449 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281783+0000", 2024-03-19T11:31:22.449 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:22.449 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.449 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.449 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:22.449 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281783+0000", 2024-03-19T11:31:22.449 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1699999999999999e-07 2024-03-19T11:31:22.449 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.449 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.450 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:22.450 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.282962+0000", 2024-03-19T11:31:22.450 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001178601 2024-03-19T11:31:22.450 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.450 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.450 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:22.450 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.282987+0000", 2024-03-19T11:31:22.450 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4899e-05 2024-03-19T11:31:22.451 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.451 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.451 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:22.451 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283006+0000", 2024-03-19T11:31:22.451 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9494999999999999e-05 2024-03-19T11:31:22.451 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.451 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.451 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:22.451 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.305757+0000", 2024-03-19T11:31:22.451 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.022750570000000001 2024-03-19T11:31:22.451 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.451 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.451 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:22.451 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.305770+0000", 2024-03-19T11:31:22.451 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2791e-05 2024-03-19T11:31:22.452 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.452 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.452 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:22.452 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.307970+0000", 2024-03-19T11:31:22.452 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022005480000000001 2024-03-19T11:31:22.452 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:22.452 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:22.452 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:22.452 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.452 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.452 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.18 e75,74 4:1d58db11:::smithi03829856-18:c,70'104)", 2024-03-19T11:31:22.452 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.283036+0000", 2024-03-19T11:31:22.452 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 28.131957153999998, 2024-03-19T11:31:22.453 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019146032, 2024-03-19T11:31:22.453 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:22.453 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:22.453 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:22.453 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.453 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:22.453 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283036+0000", 2024-03-19T11:31:22.453 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:22.453 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.453 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.453 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:22.453 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283036+0000", 2024-03-19T11:31:22.453 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:22.453 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.453 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.454 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:22.454 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283037+0000", 2024-03-19T11:31:22.454 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4899999999999995e-07 2024-03-19T11:31:22.454 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.454 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.454 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:22.454 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283042+0000", 2024-03-19T11:31:22.454 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0930000000000002e-06 2024-03-19T11:31:22.454 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.454 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.454 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:22.454 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283053+0000", 2024-03-19T11:31:22.454 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1083999999999999e-05 2024-03-19T11:31:22.454 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.454 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.455 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:22.455 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283059+0000", 2024-03-19T11:31:22.455 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6329999999999999e-06 2024-03-19T11:31:22.455 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.455 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.455 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:22.455 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302090+0000", 2024-03-19T11:31:22.455 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01903038 2024-03-19T11:31:22.455 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.455 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.455 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:22.455 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302097+0000", 2024-03-19T11:31:22.455 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.221e-06 2024-03-19T11:31:22.455 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.455 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.456 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:22.456 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302182+0000", 2024-03-19T11:31:22.456 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5072000000000001e-05 2024-03-19T11:31:22.456 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:22.456 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:22.456 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:22.456 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.456 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.456 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(osd.5.0:143 4.18 e75/74 4:1d58db11:::smithi03829856-18:head v 75'106, mlcod=0'0)", 2024-03-19T11:31:22.456 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.283083+0000", 2024-03-19T11:31:22.456 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 28.131909827000001, 2024-03-19T11:31:22.456 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025309439, 2024-03-19T11:31:22.456 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:22.456 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:22.457 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:22.457 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.457 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:22.457 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283083+0000", 2024-03-19T11:31:22.457 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:22.457 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.457 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.457 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:22.457 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283083+0000", 2024-03-19T11:31:22.457 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:22.457 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.457 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.457 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:22.457 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283084+0000", 2024-03-19T11:31:22.457 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7800000000000002e-07 2024-03-19T11:31:22.458 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.458 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.458 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:22.458 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.284082+0000", 2024-03-19T11:31:22.458 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0009977880000000001 2024-03-19T11:31:22.458 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.458 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.458 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:22.458 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.284095+0000", 2024-03-19T11:31:22.458 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3256e-05 2024-03-19T11:31:22.458 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.458 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.458 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:22.459 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.284102+0000", 2024-03-19T11:31:22.459 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.605e-06 2024-03-19T11:31:22.459 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.459 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.459 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:22.459 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302285+0000", 2024-03-19T11:31:22.459 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018182423 2024-03-19T11:31:22.459 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.459 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.459 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:22.460 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302341+0000", 2024-03-19T11:31:22.460 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.626e-05 2024-03-19T11:31:22.460 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.460 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.460 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:22.460 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.308338+0000", 2024-03-19T11:31:22.460 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0059968030000000002 2024-03-19T11:31:22.460 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.460 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.461 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:22.461 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.308393+0000", 2024-03-19T11:31:22.461 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4826e-05 2024-03-19T11:31:22.461 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:22.461 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:22.461 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:22.461 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.461 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.462 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "pg_update_log_missing(4.18 epoch 75/74 rep_tid 144 entries 75'107 (0'0) error 4:1d58db11:::smithi03829856-18:head by client.4415.0:2283 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-19T11:31:22.462 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.288252+0000", 2024-03-19T11:31:22.462 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 28.126741277000001, 2024-03-19T11:31:22.462 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020167377, 2024-03-19T11:31:22.462 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:22.462 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-03-19T11:31:22.462 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:22.462 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.462 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:22.462 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288252+0000", 2024-03-19T11:31:22.462 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:22.462 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.462 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.463 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:22.463 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288252+0000", 2024-03-19T11:31:22.463 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:22.463 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.463 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.463 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:22.463 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288253+0000", 2024-03-19T11:31:22.463 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2109999999999999e-06 2024-03-19T11:31:22.463 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.463 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.463 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:22.463 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288268+0000", 2024-03-19T11:31:22.463 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5003e-05 2024-03-19T11:31:22.463 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.464 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.464 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:22.464 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288301+0000", 2024-03-19T11:31:22.464 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2539999999999997e-05 2024-03-19T11:31:22.464 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.464 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.464 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:22.464 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288319+0000", 2024-03-19T11:31:22.464 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8232000000000001e-05 2024-03-19T11:31:22.464 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.464 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.464 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:22.464 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.305490+0000", 2024-03-19T11:31:22.464 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017170857000000001 2024-03-19T11:31:22.465 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.465 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.465 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:22.465 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.308419+0000", 2024-03-19T11:31:22.465 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029295340000000001 2024-03-19T11:31:22.465 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:22.465 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:22.465 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:22.465 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.465 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.465 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1d58db11:::smithi03829856-18:6, version: 55'80, data_included: [2313652~310127,3128657~31026], data_size: 341153, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:6@55'80, size: 3159683, copy_subset: [(0, 3159683)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3159683, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2313652, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:1d58db11:::smithi03829856-18:b, version: 58'84, data_included: [1657391~966388,2633066~82188], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:b@58'84, size: 3445402, copy_subset: [(0, 3445402)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2715254, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1657391, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:10, version: 66'95, data_included: [1829043~668012,3134120~380564], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:10@66'95, size: 3764735, copy_subset: [(0, 3764735)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3514684, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:12, version: 66'97, data_included: [1829043~507467], data_size: 507467, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:12@66'97, size: 2504243, copy_subset: [(0, 2504243)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2504243, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:c, version: 70'102, data_included: [2014538~156250], data_size: 156250, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:c@70'102, size: 2857157, copy_subset: [(0, 2857157)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2857157, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2014538, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:22.465 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.312362+0000", 2024-03-19T11:31:22.465 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 28.102631228, 2024-03-19T11:31:22.465 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017968951, 2024-03-19T11:31:22.465 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:22.466 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:22.466 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:22.466 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.466 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:22.466 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.312362+0000", 2024-03-19T11:31:22.466 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:22.466 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.466 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.466 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:22.466 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.312362+0000", 2024-03-19T11:31:22.466 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:22.466 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.466 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.467 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:22.467 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.312363+0000", 2024-03-19T11:31:22.467 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.528e-06 2024-03-19T11:31:22.467 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.467 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.467 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:22.467 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315748+0000", 2024-03-19T11:31:22.467 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033842830000000001 2024-03-19T11:31:22.467 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.467 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.467 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:22.467 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315795+0000", 2024-03-19T11:31:22.467 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7382999999999997e-05 2024-03-19T11:31:22.467 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.468 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.468 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:22.468 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315845+0000", 2024-03-19T11:31:22.468 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0319999999999999e-05 2024-03-19T11:31:22.468 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.468 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.468 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:22.468 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.316123+0000", 2024-03-19T11:31:22.468 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00027747200000000002 2024-03-19T11:31:22.468 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.468 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.469 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:22.469 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.316170+0000", 2024-03-19T11:31:22.469 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7466999999999998e-05 2024-03-19T11:31:22.469 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.469 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.469 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:22.469 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.330331+0000", 2024-03-19T11:31:22.469 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014160498000000001 2024-03-19T11:31:22.469 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:22.469 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:22.469 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:22.469 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.469 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.469 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1dabc0f1:::smithi03829856-1:head, version: 70'103, data_included: [1829043~555153], data_size: 555153, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:head@70'103, size: 2621892, copy_subset: [(0, 2621892)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2621892, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:22.470 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.315898+0000", 2024-03-19T11:31:22.470 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 28.099095242000001, 2024-03-19T11:31:22.470 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016880424000000002, 2024-03-19T11:31:22.470 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:22.470 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:22.470 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:22.470 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.470 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:22.470 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315898+0000", 2024-03-19T11:31:22.470 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:22.470 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.470 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.470 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:22.470 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315898+0000", 2024-03-19T11:31:22.471 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:22.471 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.471 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.471 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:22.471 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315899+0000", 2024-03-19T11:31:22.471 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.003e-06 2024-03-19T11:31:22.471 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.471 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.471 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:22.471 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.316486+0000", 2024-03-19T11:31:22.472 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00058673099999999995 2024-03-19T11:31:22.472 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.472 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.472 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:22.472 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.316519+0000", 2024-03-19T11:31:22.472 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3124000000000003e-05 2024-03-19T11:31:22.472 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.472 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.472 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:22.473 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.316534+0000", 2024-03-19T11:31:22.473 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.485e-05 2024-03-19T11:31:22.473 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.473 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.473 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:22.473 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.330415+0000", 2024-03-19T11:31:22.473 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013880905000000001 2024-03-19T11:31:22.473 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.473 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.473 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:22.474 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.330439+0000", 2024-03-19T11:31:22.474 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4630999999999998e-05 2024-03-19T11:31:22.474 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.474 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.474 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:22.474 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.332778+0000", 2024-03-19T11:31:22.474 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023391800000000002 2024-03-19T11:31:22.474 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:22.474 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:22.474 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:22.474 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.475 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.475 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2387 4.2 4:43b32242:::smithi03829856-22:head [read 596992~684544] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-03-19T11:31:22.475 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:59.319806+0000", 2024-03-19T11:31:22.475 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 23.095187575000001, 2024-03-19T11:31:22.475 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.78144344399999999, 2024-03-19T11:31:22.475 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:22.475 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:22.475 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:22.475 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:22.475 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:22.475 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2387 2024-03-19T11:31:22.475 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.475 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:22.475 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.476 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:22.476 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319806+0000", 2024-03-19T11:31:22.476 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:22.476 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.476 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.476 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:22.476 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319806+0000", 2024-03-19T11:31:22.476 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:22.476 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.476 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.476 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:22.476 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319807+0000", 2024-03-19T11:31:22.476 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3519999999999999e-06 2024-03-19T11:31:22.476 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.477 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.477 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:22.477 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319812+0000", 2024-03-19T11:31:22.477 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3959999999999997e-06 2024-03-19T11:31:22.477 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.477 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.477 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:22.477 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319827+0000", 2024-03-19T11:31:22.477 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4566000000000001e-05 2024-03-19T11:31:22.477 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.477 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.477 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:22.478 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319835+0000", 2024-03-19T11:31:22.478 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.994e-06 2024-03-19T11:31:22.478 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.478 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.478 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:22.478 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.320119+0000", 2024-03-19T11:31:22.478 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00028445199999999998 2024-03-19T11:31:22.478 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.478 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.478 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:22.478 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.320129+0000", 2024-03-19T11:31:22.478 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9580000000000002e-06 2024-03-19T11:31:22.478 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.479 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.479 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:22.479 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.069842+0000", 2024-03-19T11:31:22.479 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.74971218399999995 2024-03-19T11:31:22.479 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.479 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.479 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:31:22.479 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.070010+0000", 2024-03-19T11:31:22.479 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016846600000000001 2024-03-19T11:31:22.479 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.479 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.479 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:22.479 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.090911+0000", 2024-03-19T11:31:22.479 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020901494999999999 2024-03-19T11:31:22.480 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.480 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.480 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:31:22.480 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.091249+0000", 2024-03-19T11:31:22.480 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000337602 2024-03-19T11:31:22.480 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.480 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.480 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:22.480 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.098478+0000", 2024-03-19T11:31:22.480 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0072287549999999999 2024-03-19T11:31:22.480 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.480 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.480 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:22.480 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.098617+0000", 2024-03-19T11:31:22.481 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013871599999999999 2024-03-19T11:31:22.481 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.481 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.481 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:22.481 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.101249+0000", 2024-03-19T11:31:22.481 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002632508 2024-03-19T11:31:22.481 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:22.481 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:22.481 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:22.481 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.481 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.481 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 81/80 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 15=[]:{7: [6], b: [b], 10: [d], 12: [12, 11], 15: [15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:7, version: 55'82, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@55'82, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 16=[]:{7: [6], b: [b], 10: [d], 16: [16, 15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 15=[]:{7: [6], b: [b], 10: [d], 12: [12, 11], 15: [15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 58'86, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@58'86, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 16=[]:{7: [6], b: [b], 10: [d], 16: [16, 15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [504210~968657,1572695~79919], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 16=[]:{7: [6], b: [b], 10: [d], 16: [16, 15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:22.481 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:00.089924+0000", 2024-03-19T11:31:22.481 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 22.325069345999999, 2024-03-19T11:31:22.482 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024686583000000002, 2024-03-19T11:31:22.482 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:22.482 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:22.482 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:22.482 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.482 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:22.482 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.089924+0000", 2024-03-19T11:31:22.482 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:22.482 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.482 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.482 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:22.482 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.089924+0000", 2024-03-19T11:31:22.482 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:22.483 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.483 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.483 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:22.483 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.089926+0000", 2024-03-19T11:31:22.483 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9589999999999998e-06 2024-03-19T11:31:22.483 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.483 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.483 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:22.483 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093311+0000", 2024-03-19T11:31:22.483 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033854409999999999 2024-03-19T11:31:22.483 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.483 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.483 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:22.484 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093365+0000", 2024-03-19T11:31:22.484 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3795000000000002e-05 2024-03-19T11:31:22.484 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.484 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.484 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:22.484 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093438+0000", 2024-03-19T11:31:22.484 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2591999999999999e-05 2024-03-19T11:31:22.484 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.484 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.484 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:22.484 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093690+0000", 2024-03-19T11:31:22.484 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025264999999999999 2024-03-19T11:31:22.484 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.484 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.485 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:22.485 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093731+0000", 2024-03-19T11:31:22.485 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0881999999999999e-05 2024-03-19T11:31:22.485 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.485 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.485 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:22.485 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.114610+0000", 2024-03-19T11:31:22.485 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020879264000000002 2024-03-19T11:31:22.485 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:22.485 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:22.485 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:22.485 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.485 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.485 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.21 e81,80 4:8401c1f0:::smithi03829856-43:8,46'66,4:866f3dcc:::smithi03829856-31:2,47'73)", 2024-03-19T11:31:22.486 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:00.093514+0000", 2024-03-19T11:31:22.486 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 22.321478746, 2024-03-19T11:31:22.486 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027375713999999999, 2024-03-19T11:31:22.486 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:22.486 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:22.486 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:22.486 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.486 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:22.486 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093514+0000", 2024-03-19T11:31:22.487 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:22.487 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.487 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.487 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:22.487 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093514+0000", 2024-03-19T11:31:22.487 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:22.487 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.487 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.487 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:22.487 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093516+0000", 2024-03-19T11:31:22.487 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1179999999999999e-06 2024-03-19T11:31:22.487 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.487 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.487 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:22.487 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093523+0000", 2024-03-19T11:31:22.488 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7500000000000003e-06 2024-03-19T11:31:22.488 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.488 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.489 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:22.489 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093534+0000", 2024-03-19T11:31:22.489 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0831e-05 2024-03-19T11:31:22.489 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.489 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.489 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:22.489 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093542+0000", 2024-03-19T11:31:22.489 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0139999999999998e-06 2024-03-19T11:31:22.489 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.489 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.489 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:22.489 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.120682+0000", 2024-03-19T11:31:22.489 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027139838999999999 2024-03-19T11:31:22.489 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.490 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.490 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:22.490 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.120693+0000", 2024-03-19T11:31:22.490 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.136e-05 2024-03-19T11:31:22.490 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.490 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.490 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:22.490 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.120890+0000", 2024-03-19T11:31:22.490 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019680199999999999 2024-03-19T11:31:22.490 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:22.490 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:22.490 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:22.490 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.491 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.491 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2927 4.24 4:266590a5:::smithi03829856-16:head [append 0~0] snapc 1b=[1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-19T11:31:22.491 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:18.054474+0000", 2024-03-19T11:31:22.491 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.3605194990000005, 2024-03-19T11:31:22.491 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024856390999999999, 2024-03-19T11:31:22.491 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:22.491 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:22.491 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:22.491 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:22.491 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:22.491 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2927 2024-03-19T11:31:22.491 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.491 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:22.491 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.492 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:22.492 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054474+0000", 2024-03-19T11:31:22.492 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:22.492 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.492 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.492 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:22.492 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054474+0000", 2024-03-19T11:31:22.492 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:22.492 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.492 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.492 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:22.492 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054476+0000", 2024-03-19T11:31:22.492 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6469999999999998e-06 2024-03-19T11:31:22.493 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.493 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.493 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:22.493 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054489+0000", 2024-03-19T11:31:22.493 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2612999999999999e-05 2024-03-19T11:31:22.493 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.493 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.493 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:22.493 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054526+0000", 2024-03-19T11:31:22.493 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7481999999999998e-05 2024-03-19T11:31:22.493 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.494 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.494 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:22.494 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054547+0000", 2024-03-19T11:31:22.494 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0820999999999999e-05 2024-03-19T11:31:22.494 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.494 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.494 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:22.494 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054633+0000", 2024-03-19T11:31:22.494 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5556000000000006e-05 2024-03-19T11:31:22.494 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.494 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.494 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-03-19T11:31:22.494 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054639+0000", 2024-03-19T11:31:22.495 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7300000000000002e-06 2024-03-19T11:31:22.495 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.495 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.495 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:22.495 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.061592+0000", 2024-03-19T11:31:22.495 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0069539279999999998 2024-03-19T11:31:22.495 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.495 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.495 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:22.495 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.061746+0000", 2024-03-19T11:31:22.495 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015351699999999999 2024-03-19T11:31:22.495 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.495 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.495 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:22.495 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.062031+0000", 2024-03-19T11:31:22.496 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00028539700000000001 2024-03-19T11:31:22.496 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.496 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.496 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:22.496 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.079164+0000", 2024-03-19T11:31:22.496 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017132865000000001 2024-03-19T11:31:22.496 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.496 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.496 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:22.496 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.079217+0000", 2024-03-19T11:31:22.496 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2373999999999997e-05 2024-03-19T11:31:22.496 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.496 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.496 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:22.496 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.079284+0000", 2024-03-19T11:31:22.497 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7433000000000003e-05 2024-03-19T11:31:22.497 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.497 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.497 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:22.497 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.079330+0000", 2024-03-19T11:31:22.497 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6028e-05 2024-03-19T11:31:22.497 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:22.497 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:22.497 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:22.497 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.497 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.497 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2928 4.24 4:266590a5:::smithi03829856-16:head [rollback b] snapc 1b=[1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-19T11:31:22.497 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:18.054594+0000", 2024-03-19T11:31:22.497 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.3603995119999999, 2024-03-19T11:31:22.498 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025781921999999999, 2024-03-19T11:31:22.498 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:22.498 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:22.498 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:22.498 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:22.498 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:22.498 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2928 2024-03-19T11:31:22.498 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.498 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:22.498 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.498 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:22.498 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054594+0000", 2024-03-19T11:31:22.498 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:22.498 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.499 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.499 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:22.499 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054594+0000", 2024-03-19T11:31:22.499 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:22.499 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.499 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.499 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:22.499 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054595+0000", 2024-03-19T11:31:22.499 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.018e-06 2024-03-19T11:31:22.499 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.499 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.499 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:22.499 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054601+0000", 2024-03-19T11:31:22.499 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2620000000000004e-06 2024-03-19T11:31:22.499 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.500 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.500 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:22.500 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054618+0000", 2024-03-19T11:31:22.500 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6877000000000001e-05 2024-03-19T11:31:22.500 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.500 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.500 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:22.500 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054632+0000", 2024-03-19T11:31:22.500 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4175000000000001e-05 2024-03-19T11:31:22.500 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.500 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.500 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:22.500 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054676+0000", 2024-03-19T11:31:22.500 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3659000000000003e-05 2024-03-19T11:31:22.500 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.501 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.501 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:31:22.501 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054679+0000", 2024-03-19T11:31:22.501 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0649999999999999e-06 2024-03-19T11:31:22.501 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.501 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.501 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:22.501 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.064021+0000", 2024-03-19T11:31:22.501 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093424210000000001 2024-03-19T11:31:22.501 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.501 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.501 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:22.501 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.064158+0000", 2024-03-19T11:31:22.501 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013652099999999999 2024-03-19T11:31:22.501 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.502 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.502 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:22.502 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.072763+0000", 2024-03-19T11:31:22.502 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.008605784 2024-03-19T11:31:22.502 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.502 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.502 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:22.502 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.079171+0000", 2024-03-19T11:31:22.502 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0064071320000000003 2024-03-19T11:31:22.502 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.502 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.502 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:22.502 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.080257+0000", 2024-03-19T11:31:22.502 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0010866280000000001 2024-03-19T11:31:22.503 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.503 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.503 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:22.503 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.080324+0000", 2024-03-19T11:31:22.503 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6976000000000004e-05 2024-03-19T11:31:22.503 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.503 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.503 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:22.503 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.080376+0000", 2024-03-19T11:31:22.503 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1403999999999998e-05 2024-03-19T11:31:22.503 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:22.503 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:22.503 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:22.503 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.503 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.504 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2929 4.24 4:266590a5:::smithi03829856-16:head [append 0~0] snapc 1b=[1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-19T11:31:22.504 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:18.054649+0000", 2024-03-19T11:31:22.504 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.3603440449999997, 2024-03-19T11:31:22.504 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026690519999999999, 2024-03-19T11:31:22.504 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:22.504 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:22.504 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:22.504 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:22.504 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:22.504 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2929 2024-03-19T11:31:22.504 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.504 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:22.504 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.504 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:22.504 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054649+0000", 2024-03-19T11:31:22.505 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:22.505 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.505 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.505 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:22.505 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054649+0000", 2024-03-19T11:31:22.505 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:22.505 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.505 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.505 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:22.505 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054650+0000", 2024-03-19T11:31:22.505 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5199999999999995e-07 2024-03-19T11:31:22.505 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.505 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.505 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:22.505 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054655+0000", 2024-03-19T11:31:22.506 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1529999999999996e-06 2024-03-19T11:31:22.506 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.506 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.506 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:22.506 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054670+0000", 2024-03-19T11:31:22.506 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5029e-05 2024-03-19T11:31:22.506 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.506 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.506 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:22.506 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054683+0000", 2024-03-19T11:31:22.506 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2670000000000001e-05 2024-03-19T11:31:22.506 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.506 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.507 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:22.507 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054756+0000", 2024-03-19T11:31:22.507 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3559000000000005e-05 2024-03-19T11:31:22.507 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.507 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.507 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:31:22.507 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054760+0000", 2024-03-19T11:31:22.507 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1659999999999998e-06 2024-03-19T11:31:22.507 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.507 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.507 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:22.507 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.075329+0000", 2024-03-19T11:31:22.507 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020569918999999999 2024-03-19T11:31:22.507 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.507 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.508 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:22.508 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.075469+0000", 2024-03-19T11:31:22.508 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013990000000000001 2024-03-19T11:31:22.508 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.508 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.508 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:22.508 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.075730+0000", 2024-03-19T11:31:22.508 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00026011300000000002 2024-03-19T11:31:22.508 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.508 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.508 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:22.508 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.079918+0000", 2024-03-19T11:31:22.508 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0041886279999999998 2024-03-19T11:31:22.508 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.508 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.509 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:22.509 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.081271+0000", 2024-03-19T11:31:22.509 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0013529449999999999 2024-03-19T11:31:22.509 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.509 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.509 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:22.509 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.081308+0000", 2024-03-19T11:31:22.509 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6437999999999999e-05 2024-03-19T11:31:22.509 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.509 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.509 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:22.509 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.081340+0000", 2024-03-19T11:31:22.509 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2147999999999998e-05 2024-03-19T11:31:22.509 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:22.509 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:22.510 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:22.510 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:22.510 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:22.510 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:22.511 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-19T11:31:22.512 INFO:tasks.rados.rados.0.smithi038.stdout:1395: finishing write tid 1 to smithi03829856-33 2024-03-19T11:31:22.512 INFO:tasks.rados.rados.0.smithi038.stdout:1395: finishing write tid 2 to smithi03829856-33 2024-03-19T11:31:22.512 INFO:tasks.rados.rados.0.smithi038.stdout:1395: finishing write tid 3 to smithi03829856-33 2024-03-19T11:31:22.512 INFO:tasks.rados.rados.0.smithi038.stdout:1395: finishing write tid 4 to smithi03829856-33 2024-03-19T11:31:22.512 INFO:tasks.rados.rados.0.smithi038.stdout:1395: oid 33 updating version 0 to 150 2024-03-19T11:31:22.512 INFO:tasks.rados.rados.0.smithi038.stdout:1395: oid 33 version 150 is already newer than 149 2024-03-19T11:31:22.512 INFO:tasks.rados.rados.0.smithi038.stdout:1395: oid 33 version 150 is already newer than 148 2024-03-19T11:31:22.513 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 33 v 150 (ObjNum 238 snap 30 seq_num 238) dirty exists 2024-03-19T11:31:22.513 INFO:tasks.rados.rados.0.smithi038.stdout:1395: left oid 33 (ObjNum 238 snap 30 seq_num 238) 2024-03-19T11:31:22.513 INFO:tasks.rados.rados.0.smithi038.stdout:1397: finishing write tid 1 to smithi03829856-23 2024-03-19T11:31:22.513 INFO:tasks.rados.rados.0.smithi038.stdout:1397: finishing write tid 2 to smithi03829856-23 2024-03-19T11:31:22.513 INFO:tasks.rados.rados.0.smithi038.stdout:1398: expect (ObjNum 210 snap 22 seq_num 210) 2024-03-19T11:31:22.513 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:22.517+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:22.613 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:22.622+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:22.678 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:22.678 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:31:22.678 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:31:22.678 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:22.692 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-19T11:31:22.696 INFO:tasks.rados.rados.0.smithi038.stdout:1397: finishing write tid 3 to smithi03829856-23 2024-03-19T11:31:22.696 INFO:tasks.rados.rados.0.smithi038.stdout:1397: finishing write tid 4 to smithi03829856-23 2024-03-19T11:31:22.696 INFO:tasks.rados.rados.0.smithi038.stdout:1397: oid 23 updating version 0 to 121 2024-03-19T11:31:22.696 INFO:tasks.rados.rados.0.smithi038.stdout:1397: oid 23 updating version 121 to 122 2024-03-19T11:31:22.696 INFO:tasks.rados.rados.0.smithi038.stdout:1397: oid 23 updating version 122 to 123 2024-03-19T11:31:22.696 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 23 v 123 (ObjNum 239 snap 30 seq_num 239) dirty exists 2024-03-19T11:31:22.696 INFO:tasks.rados.rados.0.smithi038.stdout:1397: left oid 23 (ObjNum 239 snap 30 seq_num 239) 2024-03-19T11:31:22.696 INFO:tasks.rados.rados.0.smithi038.stdout:1400: finishing set_chunk to oid 13 2024-03-19T11:31:22.696 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 13 set_chunk returned error code -95 offset: 1289216 length: 345600 2024-03-19T11:31:22.696 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 13 v 0 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-03-19T11:31:22.696 INFO:tasks.rados.rados.0.smithi038.stdout:1399: expect (ObjNum 209 snap 22 seq_num 209) 2024-03-19T11:31:22.713 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:22.722+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:22.814 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:22.822+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:22.816 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:22.816 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:31:22.816 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:31:22.816 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:31:22.816 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:22.830 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-19T11:31:22.901 INFO:tasks.rados.rados.0.smithi038.stdout:1401: expect (ObjNum 31 snap 0 seq_num 31) 2024-03-19T11:31:22.914 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:22.921+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:22.978 INFO:tasks.rados.rados.0.smithi038.stdout:1402: finishing set_chunk to oid 47 2024-03-19T11:31:22.978 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 47 set_chunk returned error code -95 offset: 1090048 length: 1353728 2024-03-19T11:31:22.978 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-03-19T11:31:22.978 INFO:tasks.rados.rados.0.smithi038.stdout:1403: got (22) Invalid argument 2024-03-19T11:31:22.978 INFO:tasks.rados.rados.0.smithi038.stdout:1395: done (8 left) 2024-03-19T11:31:22.978 INFO:tasks.rados.rados.0.smithi038.stdout:1396: done (7 left) 2024-03-19T11:31:22.978 INFO:tasks.rados.rados.0.smithi038.stdout:1397: done (6 left) 2024-03-19T11:31:22.978 INFO:tasks.rados.rados.0.smithi038.stdout:1398: done (5 left) 2024-03-19T11:31:22.979 INFO:tasks.rados.rados.0.smithi038.stdout:1399: done (4 left) 2024-03-19T11:31:22.979 INFO:tasks.rados.rados.0.smithi038.stdout:1400: done (3 left) 2024-03-19T11:31:22.979 INFO:tasks.rados.rados.0.smithi038.stdout:1401: done (2 left) 2024-03-19T11:31:22.979 INFO:tasks.rados.rados.0.smithi038.stdout:1402: done (1 left) 2024-03-19T11:31:22.980 INFO:tasks.rados.rados.0.smithi038.stdout:1403: done (0 left) 2024-03-19T11:31:22.980 INFO:tasks.rados.rados.0.smithi038.stdout:1404: set_chunk oid 39 target oid 2024-03-19T11:31:22.980 INFO:tasks.rados.rados.0.smithi038.stdout:1404: set_chunk oid 39 offset: 0 length: 4000000 target oid smithi03829856- offset: 0 snap -1 2024-03-19T11:31:22.980 INFO:tasks.rados.rados.0.smithi038.stdout:1405: tier_evict oid 48 2024-03-19T11:31:22.980 INFO:tasks.rados.rados.0.smithi038.stdout:1405: tier_evict oid 48 snap -1 2024-03-19T11:31:22.980 INFO:tasks.rados.rados.0.smithi038.stdout:1406: write oid 27 current snap is 30 2024-03-19T11:31:22.980 INFO:tasks.rados.rados.0.smithi038.stdout:1406: seq_num 240 ranges {433680=466703,1577796=455884} 2024-03-19T11:31:22.993 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:22.993 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:31:22.993 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:31:22.993 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:31:22.993 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.993 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.16 49/48 [PushOp(4:6868af91:::smithi03829856-10:2, version: 29'22, data_included: [551263~793594,1893632~254982], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:2@29'22, size: 2151263, copy_subset: [(0, 2151263)], clone_subset: {}, snapset: b=[]:{2: [2], 7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2148614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:6d9093b1:::smithi03829856-38:4, version: 33'29, data_included: [621859~449625,1865716~598951], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6d9093b1:::smithi03829856-38:4@33'29, size: 2621859, copy_subset: [(0, 2621859)], clone_subset: {}, snapset: 4=[]:{4: [2]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2464667, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:6d9093b1:::smithi03829856-38:head, version: 33'33, data_included: [621859~568773,1729599~479803], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:6d9093b1:::smithi03829856-38:head@33'33, size: 3142876, copy_subset: [(0, 3142876)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2209402, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:6be89c06:::smithi03829856-24:head, version: 36'39, data_included: [741376~589824,1753088~458752], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:6be89c06:::smithi03829856-24:head@36'39, size: 2386561, copy_subset: [(0, 2386561)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2211840, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:6868af91:::smithi03829856-10:7, version: 38'41, data_included: [524385~820472,1602039~228104], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:7@38'41, size: 2924385, copy_subset: [(0, 2924385)], clone_subset: {}, snapset: b=[]:{2: [2], 7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1830143, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:6868af91:::smithi03829856-10:b, version: 47'48, data_included: [524385~820472,1602039~228104], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:b@47'48, size: 2924385, copy_subset: [(0, 2924385)], clone_subset: {}, snapset: b=[]:{2: [2], 7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1830143, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:6868af91:::smithi03829856-10:head, version: 47'52, data_included: [524385~820472,1602039~228104], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:head@47'52, size: 3747928, copy_subset: [(0, 3747928)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1830143, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:22.993 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:26.954289+0000", 2024-03-19T11:31:22.993 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 56.049532505999998, 2024-03-19T11:31:22.994 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029260781999999999, 2024-03-19T11:31:22.994 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:22.994 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:22.994 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:22.994 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.994 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:22.994 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954289+0000", 2024-03-19T11:31:22.994 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:22.994 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.994 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.994 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:22.994 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954289+0000", 2024-03-19T11:31:22.994 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:22.994 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.995 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.995 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:22.995 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954291+0000", 2024-03-19T11:31:22.995 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8050000000000001e-06 2024-03-19T11:31:22.995 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.995 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.995 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:22.995 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.958775+0000", 2024-03-19T11:31:22.995 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0044845859999999996 2024-03-19T11:31:22.995 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.995 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.995 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:22.995 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.958851+0000", 2024-03-19T11:31:22.995 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6050999999999994e-05 2024-03-19T11:31:22.995 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.996 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.996 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:22.996 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.958931+0000", 2024-03-19T11:31:22.996 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9099999999999998e-05 2024-03-19T11:31:22.996 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.996 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.996 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:22.996 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.959206+0000", 2024-03-19T11:31:22.996 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00027495999999999998 2024-03-19T11:31:22.996 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.996 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.996 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:22.996 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.959246+0000", 2024-03-19T11:31:22.996 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0326e-05 2024-03-19T11:31:22.997 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.997 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.997 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:22.997 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.983550+0000", 2024-03-19T11:31:22.997 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024303953999999999 2024-03-19T11:31:22.997 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:22.997 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:22.997 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:22.997 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.997 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.997 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.16 49/48 [PushOp(4:6868af91:::smithi03829856-10:2, version: 29'22, data_included: [2148614~2649], data_size: 2649, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:2@29'22, size: 2151263, copy_subset: [(0, 2151263)], clone_subset: {}, snapset: b=[]:{2: [2], 7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2151263, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2148614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:6d9093b1:::smithi03829856-38:4, version: 33'29, data_included: [2464667~157192], data_size: 157192, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:6d9093b1:::smithi03829856-38:4@33'29, size: 2621859, copy_subset: [(0, 2621859)], clone_subset: {}, snapset: 4=[]:{4: [2]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2621859, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2464667, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:6d9093b1:::smithi03829856-38:head, version: 33'33, data_included: [2209402~412457,3101234~41642], data_size: 454099, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:6d9093b1:::smithi03829856-38:head@33'33, size: 3142876, copy_subset: [(0, 3142876)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3142876, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2209402, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:6be89c06:::smithi03829856-24:head, version: 36'39, data_included: [2211840~51851], data_size: 51851, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:6be89c06:::smithi03829856-24:head@36'39, size: 2386561, copy_subset: [(0, 2386561)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2386561, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2211840, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:6868af91:::smithi03829856-10:7, version: 38'41, data_included: [1830143~321120,2577441~346944], data_size: 668064, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:7@38'41, size: 2924385, copy_subset: [(0, 2924385)], clone_subset: {}, snapset: b=[]:{2: [2], 7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2924385, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1830143, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:6868af91:::smithi03829856-10:b, version: 47'48, data_included: [1830143~321120,2577441~346944], data_size: 668064, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:b@47'48, size: 2924385, copy_subset: [(0, 2924385)], clone_subset: {}, snapset: b=[]:{2: [2], 7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2924385, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1830143, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:6868af91:::smithi03829856-10:head, version: 47'52, data_included: [1830143~348419,2577441~661310], data_size: 1009729, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:head@47'52, size: 3747928, copy_subset: [(0, 3747928)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3747928, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1830143, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:22.997 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:26.998857+0000", 2024-03-19T11:31:22.997 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 56.004964745999999, 2024-03-19T11:31:22.997 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010149703, 2024-03-19T11:31:22.998 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:22.998 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:22.998 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:22.998 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.998 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:22.998 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.998857+0000", 2024-03-19T11:31:22.998 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:22.998 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.998 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.998 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:22.998 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.998857+0000", 2024-03-19T11:31:22.999 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:22.999 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.999 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.999 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:22.999 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.998858+0000", 2024-03-19T11:31:22.999 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.065e-06 2024-03-19T11:31:22.999 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.999 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.999 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:22.999 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.999629+0000", 2024-03-19T11:31:22.999 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000771132 2024-03-19T11:31:22.999 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:22.999 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:22.999 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:22.999 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.999688+0000", 2024-03-19T11:31:23.000 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9147000000000001e-05 2024-03-19T11:31:23.000 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.000 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.000 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:23.000 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.999755+0000", 2024-03-19T11:31:23.000 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6891999999999996e-05 2024-03-19T11:31:23.000 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.000 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.000 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:23.000 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.000047+0000", 2024-03-19T11:31:23.000 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000292176 2024-03-19T11:31:23.000 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.000 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.001 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:23.001 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.000094+0000", 2024-03-19T11:31:23.001 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6485e-05 2024-03-19T11:31:23.001 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.001 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.001 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:23.001 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.009007+0000", 2024-03-19T11:31:23.001 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0089128060000000005 2024-03-19T11:31:23.001 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.001 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:23.001 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.001 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.001 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.001 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1591 4.4 4:266590a5:::smithi03829856-16:head [write 783226~624677,stat] snapc e=[e,d,c,b,a,6] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-03-19T11:31:23.002 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:32.390032+0000", 2024-03-19T11:31:23.002 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 50.613789961000002, 2024-03-19T11:31:23.002 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0090431220000000007, 2024-03-19T11:31:23.002 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:23.002 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:23.002 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:23.002 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:23.002 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:23.002 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1591 2024-03-19T11:31:23.002 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.002 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:23.002 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.002 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:23.002 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.390032+0000", 2024-03-19T11:31:23.003 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.003 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.003 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.003 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:23.003 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.390032+0000", 2024-03-19T11:31:23.003 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.003 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.003 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.003 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:23.003 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.390033+0000", 2024-03-19T11:31:23.003 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2839999999999999e-06 2024-03-19T11:31:23.003 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.003 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.003 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:23.004 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.390273+0000", 2024-03-19T11:31:23.004 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00024011099999999999 2024-03-19T11:31:23.004 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.004 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.004 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:23.004 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.390286+0000", 2024-03-19T11:31:23.004 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3314e-05 2024-03-19T11:31:23.004 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.004 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.004 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:23.004 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.390295+0000", 2024-03-19T11:31:23.004 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6330000000000002e-06 2024-03-19T11:31:23.004 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.004 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.004 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:23.005 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.390368+0000", 2024-03-19T11:31:23.005 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3065999999999995e-05 2024-03-19T11:31:23.005 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.005 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.005 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:23.005 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.390461+0000", 2024-03-19T11:31:23.005 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2582000000000002e-05 2024-03-19T11:31:23.005 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.005 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.005 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3", 2024-03-19T11:31:23.005 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.390648+0000", 2024-03-19T11:31:23.005 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00018695300000000001 2024-03-19T11:31:23.005 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.006 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.006 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:23.006 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.398909+0000", 2024-03-19T11:31:23.006 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0082613559999999992 2024-03-19T11:31:23.006 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.006 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.006 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:23.006 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.398954+0000", 2024-03-19T11:31:23.006 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5553999999999998e-05 2024-03-19T11:31:23.006 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.006 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.006 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:23.006 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.399017+0000", 2024-03-19T11:31:23.006 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2849999999999996e-05 2024-03-19T11:31:23.007 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.007 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.007 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:23.007 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.399075+0000", 2024-03-19T11:31:23.007 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7419e-05 2024-03-19T11:31:23.007 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.007 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:23.007 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.007 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.007 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.007 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1630 4.d 4:ba44105a:::smithi03829856-26:head [write 467261~717275] snapc 10=[10,f,e,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-03-19T11:31:23.007 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:35.331434+0000", 2024-03-19T11:31:23.007 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 47.672387700999998, 2024-03-19T11:31:23.008 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012902375000000001, 2024-03-19T11:31:23.008 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:23.008 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:23.008 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:23.008 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:23.008 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:23.008 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1630 2024-03-19T11:31:23.008 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.008 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:23.008 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.008 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:23.008 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.331434+0000", 2024-03-19T11:31:23.008 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.008 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.008 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.009 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:23.009 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.331434+0000", 2024-03-19T11:31:23.009 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.009 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.009 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.009 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:23.009 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.331437+0000", 2024-03-19T11:31:23.009 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5599999999999998e-06 2024-03-19T11:31:23.009 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.009 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.010 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:23.010 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.331938+0000", 2024-03-19T11:31:23.010 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00050090099999999999 2024-03-19T11:31:23.010 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.010 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.010 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:23.010 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.331976+0000", 2024-03-19T11:31:23.010 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8090999999999998e-05 2024-03-19T11:31:23.010 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.010 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.010 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:23.010 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.331994+0000", 2024-03-19T11:31:23.010 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7566000000000001e-05 2024-03-19T11:31:23.011 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.011 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.011 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:23.011 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.332086+0000", 2024-03-19T11:31:23.011 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2452000000000002e-05 2024-03-19T11:31:23.011 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.011 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.011 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:23.011 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.332205+0000", 2024-03-19T11:31:23.011 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000118966 2024-03-19T11:31:23.011 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.011 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.011 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1", 2024-03-19T11:31:23.011 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.332410+0000", 2024-03-19T11:31:23.012 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020430099999999999 2024-03-19T11:31:23.012 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.012 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.012 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:23.012 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.339226+0000", 2024-03-19T11:31:23.012 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0068165980000000001 2024-03-19T11:31:23.012 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.012 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.012 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:23.012 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.344191+0000", 2024-03-19T11:31:23.012 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.004965143 2024-03-19T11:31:23.012 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.012 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.012 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:23.013 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.344272+0000", 2024-03-19T11:31:23.013 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0265000000000005e-05 2024-03-19T11:31:23.013 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.013 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.013 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:23.013 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.344336+0000", 2024-03-19T11:31:23.013 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4531999999999998e-05 2024-03-19T11:31:23.013 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.013 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:23.013 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.013 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.014 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.014 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1631 4.d 4:ba44105a:::smithi03829856-26:head [write 1847991~219270] snapc 10=[10,f,e,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-03-19T11:31:23.014 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:35.336760+0000", 2024-03-19T11:31:23.014 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 47.667061099000001, 2024-03-19T11:31:23.014 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0087315580000000004, 2024-03-19T11:31:23.014 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:23.014 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:23.014 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:23.014 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:23.014 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:23.014 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1631 2024-03-19T11:31:23.014 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.014 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:23.014 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.015 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:23.015 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.336760+0000", 2024-03-19T11:31:23.015 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.015 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.015 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.015 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:23.015 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.336760+0000", 2024-03-19T11:31:23.015 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.015 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.015 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.015 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:23.015 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.336762+0000", 2024-03-19T11:31:23.015 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7099999999999999e-06 2024-03-19T11:31:23.015 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.016 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.016 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:23.016 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.336909+0000", 2024-03-19T11:31:23.016 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000147045 2024-03-19T11:31:23.016 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.016 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.016 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:23.016 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.336923+0000", 2024-03-19T11:31:23.016 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3526999999999999e-05 2024-03-19T11:31:23.016 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.016 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.016 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:23.016 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.336943+0000", 2024-03-19T11:31:23.016 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0358999999999999e-05 2024-03-19T11:31:23.016 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.017 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.017 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:23.017 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.337039+0000", 2024-03-19T11:31:23.017 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.5644999999999997e-05 2024-03-19T11:31:23.017 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.017 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.017 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:23.017 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.337155+0000", 2024-03-19T11:31:23.017 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000115913 2024-03-19T11:31:23.017 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.017 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.017 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1", 2024-03-19T11:31:23.017 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.337341+0000", 2024-03-19T11:31:23.017 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000186472 2024-03-19T11:31:23.018 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.018 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.018 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:23.018 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.339232+0000", 2024-03-19T11:31:23.018 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001890954 2024-03-19T11:31:23.018 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.018 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.018 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:23.018 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.345393+0000", 2024-03-19T11:31:23.018 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.006161232 2024-03-19T11:31:23.018 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.018 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.018 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:23.018 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.345449+0000", 2024-03-19T11:31:23.019 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5340000000000002e-05 2024-03-19T11:31:23.019 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.019 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.019 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:23.019 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.345492+0000", 2024-03-19T11:31:23.019 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3361e-05 2024-03-19T11:31:23.019 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.019 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:23.019 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.019 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.019 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.019 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1632 4.d 4:ba44105a:::smithi03829856-26:head [setxattr _header (57) in=7b,truncate 2067261] snapc 10=[10,f,e,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-03-19T11:31:23.019 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:35.336987+0000", 2024-03-19T11:31:23.019 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 47.666835040999999, 2024-03-19T11:31:23.020 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0091098789999999995, 2024-03-19T11:31:23.020 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:23.020 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:23.020 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:23.020 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:23.020 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:23.020 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1632 2024-03-19T11:31:23.020 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.020 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:23.020 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.020 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:23.020 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.336987+0000", 2024-03-19T11:31:23.020 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.020 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.020 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.021 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:23.021 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.336987+0000", 2024-03-19T11:31:23.021 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.021 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.021 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.021 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:23.021 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.336987+0000", 2024-03-19T11:31:23.021 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4600000000000004e-07 2024-03-19T11:31:23.021 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.021 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.021 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:23.021 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.336992+0000", 2024-03-19T11:31:23.021 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6090000000000003e-06 2024-03-19T11:31:23.022 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.022 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.022 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:23.022 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.337002+0000", 2024-03-19T11:31:23.022 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.006e-05 2024-03-19T11:31:23.022 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.022 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.022 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:23.022 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.337009+0000", 2024-03-19T11:31:23.022 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7229999999999998e-06 2024-03-19T11:31:23.022 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.022 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.022 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:23.022 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.338513+0000", 2024-03-19T11:31:23.023 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001503868 2024-03-19T11:31:23.023 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.023 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.023 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:23.023 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.338566+0000", 2024-03-19T11:31:23.023 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3260999999999997e-05 2024-03-19T11:31:23.023 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.023 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.023 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1", 2024-03-19T11:31:23.023 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.338652+0000", 2024-03-19T11:31:23.023 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5889000000000004e-05 2024-03-19T11:31:23.023 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.023 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.023 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:23.024 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.340450+0000", 2024-03-19T11:31:23.024 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017987490000000001 2024-03-19T11:31:23.024 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.024 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.024 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:23.024 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.345982+0000", 2024-03-19T11:31:23.024 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055312620000000003 2024-03-19T11:31:23.024 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.024 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.024 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:23.024 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.346039+0000", 2024-03-19T11:31:23.024 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6879999999999998e-05 2024-03-19T11:31:23.024 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.024 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.024 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:23.025 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.346096+0000", 2024-03-19T11:31:23.025 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7831999999999998e-05 2024-03-19T11:31:23.025 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.025 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:23.025 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.025 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.025 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.025 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1633 4.d 4:ba44105a:::smithi03829856-26:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-03-19T11:31:23.025 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:35.337032+0000", 2024-03-19T11:31:23.025 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 47.666789322, 2024-03-19T11:31:23.025 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093229679999999992, 2024-03-19T11:31:23.025 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:23.025 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:23.025 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:23.026 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:23.026 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:23.026 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1633 2024-03-19T11:31:23.026 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.026 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:23.026 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.026 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:23.026 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.337032+0000", 2024-03-19T11:31:23.026 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.026 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.026 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.026 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:23.026 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.337032+0000", 2024-03-19T11:31:23.026 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.027 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.027 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.027 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:23.027 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.337033+0000", 2024-03-19T11:31:23.027 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4899999999999995e-07 2024-03-19T11:31:23.027 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.027 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.027 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:23.027 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.337037+0000", 2024-03-19T11:31:23.027 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7249999999999999e-06 2024-03-19T11:31:23.027 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.027 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.027 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:23.027 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.337045+0000", 2024-03-19T11:31:23.027 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5380000000000002e-06 2024-03-19T11:31:23.028 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.028 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.028 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:23.028 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.337051+0000", 2024-03-19T11:31:23.028 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1689999999999997e-06 2024-03-19T11:31:23.028 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.028 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.028 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:23.028 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.339128+0000", 2024-03-19T11:31:23.028 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002076919 2024-03-19T11:31:23.028 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.028 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.028 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-03-19T11:31:23.029 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.339211+0000", 2024-03-19T11:31:23.029 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2896000000000001e-05 2024-03-19T11:31:23.029 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.029 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.029 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:23.029 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.346152+0000", 2024-03-19T11:31:23.029 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0069404779999999999 2024-03-19T11:31:23.029 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.029 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.029 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:23.029 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.346242+0000", 2024-03-19T11:31:23.029 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0031000000000006e-05 2024-03-19T11:31:23.029 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.029 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.029 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:23.030 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.346355+0000", 2024-03-19T11:31:23.030 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000113663 2024-03-19T11:31:23.030 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.030 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:23.030 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.030 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.030 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.030 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1845 4.4 4:27909373:::smithi03829856-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-03-19T11:31:23.030 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:41.053790+0000", 2024-03-19T11:31:23.030 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 41.950031240999998, 2024-03-19T11:31:23.030 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0091637609999999994, 2024-03-19T11:31:23.030 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:23.030 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:23.030 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:23.031 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:23.031 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:23.031 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1845 2024-03-19T11:31:23.031 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.031 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:23.031 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.031 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:23.031 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053790+0000", 2024-03-19T11:31:23.031 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.031 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.031 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.031 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:23.031 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053790+0000", 2024-03-19T11:31:23.031 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.031 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.032 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.032 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:23.032 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053792+0000", 2024-03-19T11:31:23.032 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2860000000000001e-06 2024-03-19T11:31:23.032 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.032 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.032 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:23.032 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053802+0000", 2024-03-19T11:31:23.032 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.064e-05 2024-03-19T11:31:23.032 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.032 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.032 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:23.032 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053824+0000", 2024-03-19T11:31:23.032 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1611999999999999e-05 2024-03-19T11:31:23.033 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.033 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.033 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:23.033 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053839+0000", 2024-03-19T11:31:23.034 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4922e-05 2024-03-19T11:31:23.034 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.034 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.034 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:23.034 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.059104+0000", 2024-03-19T11:31:23.034 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0052656179999999997 2024-03-19T11:31:23.034 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.034 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.035 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-03-19T11:31:23.035 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.059267+0000", 2024-03-19T11:31:23.035 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000162776 2024-03-19T11:31:23.035 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.035 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.035 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:23.035 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.062551+0000", 2024-03-19T11:31:23.035 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003283467 2024-03-19T11:31:23.035 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.035 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.035 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:23.035 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.062738+0000", 2024-03-19T11:31:23.035 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00018781900000000001 2024-03-19T11:31:23.035 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.036 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.036 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:23.036 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.062954+0000", 2024-03-19T11:31:23.036 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00021562099999999999 2024-03-19T11:31:23.036 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.036 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:23.036 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.036 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.036 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.036 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1848 4.d 4:b65e6658:::smithi03829856-9:6 [sparse-read 0~901298] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-03-19T11:31:23.036 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:41.053966+0000", 2024-03-19T11:31:23.036 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 41.949855786999997, 2024-03-19T11:31:23.036 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0090164259999999993, 2024-03-19T11:31:23.036 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:23.037 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:23.037 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:23.037 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:23.037 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:23.037 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1848 2024-03-19T11:31:23.037 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.037 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:23.037 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.037 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:23.037 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053966+0000", 2024-03-19T11:31:23.037 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.037 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.037 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.037 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:23.037 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053966+0000", 2024-03-19T11:31:23.038 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.038 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.038 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.038 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:23.038 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053967+0000", 2024-03-19T11:31:23.038 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.387e-06 2024-03-19T11:31:23.038 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.038 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.038 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:23.038 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053975+0000", 2024-03-19T11:31:23.038 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5299999999999999e-06 2024-03-19T11:31:23.038 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.038 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.038 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:23.039 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053993+0000", 2024-03-19T11:31:23.039 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7929000000000001e-05 2024-03-19T11:31:23.039 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.039 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.039 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:23.039 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.054010+0000", 2024-03-19T11:31:23.039 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7876e-05 2024-03-19T11:31:23.039 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.039 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.039 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:23.039 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.062407+0000", 2024-03-19T11:31:23.039 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0083966200000000005 2024-03-19T11:31:23.039 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.039 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.039 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:23.040 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.062664+0000", 2024-03-19T11:31:23.040 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025712100000000002 2024-03-19T11:31:23.040 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.040 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.040 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:23.040 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.062982+0000", 2024-03-19T11:31:23.040 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00031796299999999999 2024-03-19T11:31:23.040 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.040 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:23.040 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.040 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.040 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.040 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1889 4.4 4:266590a5:::smithi03829856-16:head [rollback d] snapc 10=[10,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-03-19T11:31:23.040 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:41.697617+0000", 2024-03-19T11:31:23.041 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 41.306204706999999, 2024-03-19T11:31:23.041 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018164059999999999, 2024-03-19T11:31:23.041 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:23.041 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:23.041 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:23.041 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:23.041 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:23.041 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1889 2024-03-19T11:31:23.041 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.041 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:23.041 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.041 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:23.041 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697617+0000", 2024-03-19T11:31:23.041 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.042 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.042 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.042 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:23.042 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697617+0000", 2024-03-19T11:31:23.042 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.042 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.042 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.042 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:23.042 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697619+0000", 2024-03-19T11:31:23.042 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0059999999999999e-06 2024-03-19T11:31:23.042 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.042 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.042 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:23.042 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697631+0000", 2024-03-19T11:31:23.043 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2033e-05 2024-03-19T11:31:23.043 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.043 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.043 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:23.043 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697659+0000", 2024-03-19T11:31:23.043 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8291999999999999e-05 2024-03-19T11:31:23.043 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.043 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.043 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:23.043 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697682+0000", 2024-03-19T11:31:23.043 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3281999999999998e-05 2024-03-19T11:31:23.043 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.043 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.043 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:23.043 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.701122+0000", 2024-03-19T11:31:23.044 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034396679999999999 2024-03-19T11:31:23.044 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.044 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.044 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:23.044 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.701383+0000", 2024-03-19T11:31:23.044 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00026086399999999997 2024-03-19T11:31:23.044 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.044 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.044 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3", 2024-03-19T11:31:23.044 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.709693+0000", 2024-03-19T11:31:23.044 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0083096139999999999 2024-03-19T11:31:23.044 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.044 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.044 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:23.044 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.715437+0000", 2024-03-19T11:31:23.045 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057442379999999996 2024-03-19T11:31:23.045 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.045 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.045 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:23.045 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.715563+0000", 2024-03-19T11:31:23.045 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000125778 2024-03-19T11:31:23.045 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.045 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.045 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:23.045 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.715677+0000", 2024-03-19T11:31:23.045 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000114728 2024-03-19T11:31:23.045 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.045 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.045 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:23.046 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.715781+0000", 2024-03-19T11:31:23.046 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000103557 2024-03-19T11:31:23.046 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.046 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:23.046 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.046 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.046 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.046 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1890 4.4 4:266590a5:::smithi03829856-16:head [append 0~0] snapc 10=[10,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-03-19T11:31:23.046 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:41.697709+0000", 2024-03-19T11:31:23.046 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 41.306112145, 2024-03-19T11:31:23.046 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019825171999999999, 2024-03-19T11:31:23.046 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:23.046 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:23.046 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:23.047 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:23.047 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:23.047 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1890 2024-03-19T11:31:23.047 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.047 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:23.047 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.047 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:23.047 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697709+0000", 2024-03-19T11:31:23.047 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.047 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.047 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.047 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:23.047 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697709+0000", 2024-03-19T11:31:23.047 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.048 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.048 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.048 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:23.048 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697711+0000", 2024-03-19T11:31:23.048 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.296e-06 2024-03-19T11:31:23.048 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.048 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.048 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:23.048 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697718+0000", 2024-03-19T11:31:23.048 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.464e-06 2024-03-19T11:31:23.048 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.048 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.048 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:23.048 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697739+0000", 2024-03-19T11:31:23.048 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1005e-05 2024-03-19T11:31:23.049 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.049 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.049 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:23.049 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697754+0000", 2024-03-19T11:31:23.049 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4976e-05 2024-03-19T11:31:23.049 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.049 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.049 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:23.049 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.702341+0000", 2024-03-19T11:31:23.049 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0045870980000000004 2024-03-19T11:31:23.049 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.049 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.049 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for blocked object", 2024-03-19T11:31:23.049 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.702482+0000", 2024-03-19T11:31:23.050 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014048000000000001 2024-03-19T11:31:23.050 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.050 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.050 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:23.050 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.714359+0000", 2024-03-19T11:31:23.050 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011877522999999999 2024-03-19T11:31:23.050 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.050 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.050 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:23.050 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.714550+0000", 2024-03-19T11:31:23.050 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019117 2024-03-19T11:31:23.050 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.050 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.050 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3", 2024-03-19T11:31:23.050 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.714911+0000", 2024-03-19T11:31:23.051 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00036070900000000002 2024-03-19T11:31:23.051 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.051 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.051 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:23.051 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.716211+0000", 2024-03-19T11:31:23.051 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001300329 2024-03-19T11:31:23.051 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.051 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.051 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:23.051 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.717326+0000", 2024-03-19T11:31:23.051 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001114056 2024-03-19T11:31:23.051 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.051 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.051 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:23.052 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.717423+0000", 2024-03-19T11:31:23.052 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7048999999999999e-05 2024-03-19T11:31:23.052 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.052 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.052 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:23.052 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.717535+0000", 2024-03-19T11:31:23.052 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011201699999999999 2024-03-19T11:31:23.052 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.052 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:23.052 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.052 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.052 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.052 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1972 4.4 4:214026b3:::smithi03829856-47:head [tier-promote] snapc 12=[12,11,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:31:23.052 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:45.312422+0000", 2024-03-19T11:31:23.053 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 37.691399470999997, 2024-03-19T11:31:23.053 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013955089, 2024-03-19T11:31:23.053 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:23.053 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:23.053 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:23.053 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:23.053 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:23.053 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1972 2024-03-19T11:31:23.053 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.053 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:23.053 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.053 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:23.053 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.312422+0000", 2024-03-19T11:31:23.053 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.053 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.054 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.054 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:23.054 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.312422+0000", 2024-03-19T11:31:23.054 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.054 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.054 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.054 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:23.054 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.312424+0000", 2024-03-19T11:31:23.054 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9810000000000002e-06 2024-03-19T11:31:23.054 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.054 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.054 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:23.054 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.312433+0000", 2024-03-19T11:31:23.054 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4549999999999995e-06 2024-03-19T11:31:23.055 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.055 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.055 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:23.055 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.312452+0000", 2024-03-19T11:31:23.055 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9380000000000001e-05 2024-03-19T11:31:23.055 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.055 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.055 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:23.055 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.312460+0000", 2024-03-19T11:31:23.055 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2560000000000002e-06 2024-03-19T11:31:23.055 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.055 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.055 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:23.055 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.312555+0000", 2024-03-19T11:31:23.055 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.5208000000000001e-05 2024-03-19T11:31:23.056 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.056 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.056 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:23.056 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.312704+0000", 2024-03-19T11:31:23.056 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014843899999999999 2024-03-19T11:31:23.056 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.056 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.056 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:23.056 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.326377+0000", 2024-03-19T11:31:23.056 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013673370000000001 2024-03-19T11:31:23.056 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.056 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:23.056 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.056 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.057 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.057 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2005 4.4 4:214026b3:::smithi03829856-47:head [read 0~2773122,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:31:23.057 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:45.804325+0000", 2024-03-19T11:31:23.057 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 37.199497010999998, 2024-03-19T11:31:23.057 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010899282, 2024-03-19T11:31:23.057 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:23.057 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:23.057 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:23.057 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:23.057 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:23.057 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2005 2024-03-19T11:31:23.057 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.057 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:23.057 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.058 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:23.058 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804325+0000", 2024-03-19T11:31:23.058 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.058 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.058 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.058 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:23.058 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804325+0000", 2024-03-19T11:31:23.058 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.058 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.058 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.058 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:23.058 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804326+0000", 2024-03-19T11:31:23.058 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.666e-06 2024-03-19T11:31:23.058 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.059 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.059 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:23.059 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804338+0000", 2024-03-19T11:31:23.059 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2096000000000001e-05 2024-03-19T11:31:23.059 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.059 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.059 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:23.059 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804363+0000", 2024-03-19T11:31:23.059 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4762000000000001e-05 2024-03-19T11:31:23.059 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.059 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.059 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:23.059 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804380+0000", 2024-03-19T11:31:23.059 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.713e-05 2024-03-19T11:31:23.059 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.060 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.060 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:23.060 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.812236+0000", 2024-03-19T11:31:23.060 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0078554690000000003 2024-03-19T11:31:23.060 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.060 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.060 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:23.060 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.812541+0000", 2024-03-19T11:31:23.060 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030564299999999997 2024-03-19T11:31:23.060 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.060 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.060 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:23.060 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.815224+0000", 2024-03-19T11:31:23.060 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0026825159999999998 2024-03-19T11:31:23.061 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.061 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:23.061 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.061 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.061 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.061 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2006 4.4 4:214026b3:::smithi03829856-47:head [read 0~2773122,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:31:23.061 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:45.804421+0000", 2024-03-19T11:31:23.061 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 37.199400271999998, 2024-03-19T11:31:23.061 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013366463, 2024-03-19T11:31:23.061 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:23.061 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:23.061 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:23.061 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:23.061 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:23.062 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2006 2024-03-19T11:31:23.062 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.062 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:23.062 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.062 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:23.062 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804421+0000", 2024-03-19T11:31:23.062 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.062 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.062 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.062 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:23.062 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804421+0000", 2024-03-19T11:31:23.062 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.062 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.062 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.062 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:23.063 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804423+0000", 2024-03-19T11:31:23.063 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5600000000000001e-06 2024-03-19T11:31:23.063 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.063 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.063 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:23.063 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804433+0000", 2024-03-19T11:31:23.063 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8719999999999997e-06 2024-03-19T11:31:23.063 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.063 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.063 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:23.063 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804454+0000", 2024-03-19T11:31:23.063 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1759e-05 2024-03-19T11:31:23.063 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.063 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.064 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:23.064 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804470+0000", 2024-03-19T11:31:23.064 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5556000000000001e-05 2024-03-19T11:31:23.064 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.064 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.064 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:23.064 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.815325+0000", 2024-03-19T11:31:23.064 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010855069 2024-03-19T11:31:23.064 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.064 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.064 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:23.064 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.815684+0000", 2024-03-19T11:31:23.064 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00035912699999999999 2024-03-19T11:31:23.064 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.064 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.065 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:23.065 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.817788+0000", 2024-03-19T11:31:23.065 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0021035200000000002 2024-03-19T11:31:23.065 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.065 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:23.065 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.065 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.065 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.065 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(osd.1.0:198 4.16 e67/48 4:6868af91:::smithi03829856-10:head v 67'79, mlcod=66'76)", 2024-03-19T11:31:23.065 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:45.811039+0000", 2024-03-19T11:31:23.065 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 37.192782911000002, 2024-03-19T11:31:23.065 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01087364, 2024-03-19T11:31:23.065 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:23.065 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:23.065 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:23.066 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.066 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:23.066 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.811039+0000", 2024-03-19T11:31:23.066 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.066 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.066 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.066 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:23.066 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.811039+0000", 2024-03-19T11:31:23.066 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.066 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.066 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.066 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:23.066 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.811041+0000", 2024-03-19T11:31:23.066 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6400000000000001e-06 2024-03-19T11:31:23.066 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.067 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.067 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:23.067 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.812874+0000", 2024-03-19T11:31:23.067 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0018329959999999999 2024-03-19T11:31:23.067 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.067 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.067 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:23.067 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.812916+0000", 2024-03-19T11:31:23.067 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1875000000000001e-05 2024-03-19T11:31:23.067 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.067 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.067 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:23.067 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.812937+0000", 2024-03-19T11:31:23.067 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0985e-05 2024-03-19T11:31:23.067 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.068 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.068 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:23.068 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.813112+0000", 2024-03-19T11:31:23.068 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000174573 2024-03-19T11:31:23.068 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.068 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.068 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:23.068 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.813170+0000", 2024-03-19T11:31:23.068 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8100999999999998e-05 2024-03-19T11:31:23.068 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.068 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.068 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:23.068 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.821792+0000", 2024-03-19T11:31:23.068 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0086218549999999994 2024-03-19T11:31:23.068 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.069 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.069 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:23.069 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.821912+0000", 2024-03-19T11:31:23.069 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000120615 2024-03-19T11:31:23.069 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.069 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:23.069 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.069 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.069 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.069 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2027 4.2d 4:b4ef97e0:::smithi03829856-3:head [read 0~3697507,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:31:23.069 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:46.264908+0000", 2024-03-19T11:31:23.069 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 36.738914045000001, 2024-03-19T11:31:23.069 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010951414, 2024-03-19T11:31:23.069 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:23.070 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:23.070 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:23.070 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:23.070 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:23.070 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2027 2024-03-19T11:31:23.070 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.070 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:23.070 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.070 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:23.070 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264908+0000", 2024-03-19T11:31:23.070 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.070 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.070 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.070 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:23.070 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264908+0000", 2024-03-19T11:31:23.071 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.071 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.071 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.071 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:23.071 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264908+0000", 2024-03-19T11:31:23.071 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5799999999999999e-07 2024-03-19T11:31:23.071 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.071 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.071 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:23.071 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264912+0000", 2024-03-19T11:31:23.071 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2649999999999998e-06 2024-03-19T11:31:23.071 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.071 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.071 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:23.071 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264920+0000", 2024-03-19T11:31:23.072 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.79e-06 2024-03-19T11:31:23.072 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.072 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.072 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:23.072 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264926+0000", 2024-03-19T11:31:23.072 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9989999999999997e-06 2024-03-19T11:31:23.072 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.072 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.072 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:23.072 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.273461+0000", 2024-03-19T11:31:23.072 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0085352680000000004 2024-03-19T11:31:23.072 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.072 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.072 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:23.072 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.273598+0000", 2024-03-19T11:31:23.073 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013622299999999999 2024-03-19T11:31:23.073 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.073 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.073 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:23.073 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.275859+0000", 2024-03-19T11:31:23.073 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022613109999999998 2024-03-19T11:31:23.073 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.073 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:23.073 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.073 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.073 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.073 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2028 4.2d 4:b4ef97e0:::smithi03829856-3:head [read 0~3697507,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:31:23.073 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:46.264934+0000", 2024-03-19T11:31:23.073 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 36.738887413999997, 2024-03-19T11:31:23.073 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013708481, 2024-03-19T11:31:23.074 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:23.074 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:23.074 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:23.074 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:23.074 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:23.074 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2028 2024-03-19T11:31:23.074 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.074 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:23.074 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.074 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:23.074 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264934+0000", 2024-03-19T11:31:23.074 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.074 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.074 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.075 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:23.075 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264934+0000", 2024-03-19T11:31:23.075 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.075 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.075 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.075 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:23.075 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264935+0000", 2024-03-19T11:31:23.075 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2599999999999998e-07 2024-03-19T11:31:23.075 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.075 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.075 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:23.075 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264938+0000", 2024-03-19T11:31:23.075 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.163e-06 2024-03-19T11:31:23.075 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.076 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.076 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:23.076 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264945+0000", 2024-03-19T11:31:23.076 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.0369999999999997e-06 2024-03-19T11:31:23.076 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.076 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.076 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:23.076 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264950+0000", 2024-03-19T11:31:23.076 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1170000000000003e-06 2024-03-19T11:31:23.076 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.076 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.076 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:23.076 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.275914+0000", 2024-03-19T11:31:23.076 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010964150000000001 2024-03-19T11:31:23.076 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.077 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.077 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:23.077 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.276037+0000", 2024-03-19T11:31:23.077 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012250100000000001 2024-03-19T11:31:23.077 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.077 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.077 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:23.077 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.278643+0000", 2024-03-19T11:31:23.077 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002606087 2024-03-19T11:31:23.077 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.077 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:23.077 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.077 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.077 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.077 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2045 4.4 4:21eac9b2:::smithi03829856-4:head [read 0~2537857,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:31:23.078 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:46.265120+0000", 2024-03-19T11:31:23.078 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 36.738701106000001, 2024-03-19T11:31:23.078 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0096591980000000008, 2024-03-19T11:31:23.078 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:23.078 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:23.078 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:23.078 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:23.078 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:23.078 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2045 2024-03-19T11:31:23.078 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.078 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:23.078 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.078 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:23.078 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265120+0000", 2024-03-19T11:31:23.079 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.079 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.079 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.079 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:23.079 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265120+0000", 2024-03-19T11:31:23.079 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.079 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.079 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.079 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:23.079 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265121+0000", 2024-03-19T11:31:23.079 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2399999999999998e-07 2024-03-19T11:31:23.079 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.079 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.079 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:23.080 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265126+0000", 2024-03-19T11:31:23.080 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4870000000000002e-06 2024-03-19T11:31:23.080 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.080 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.080 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:23.080 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265134+0000", 2024-03-19T11:31:23.080 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9740000000000002e-06 2024-03-19T11:31:23.080 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.080 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.080 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:23.080 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265143+0000", 2024-03-19T11:31:23.080 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7530000000000007e-06 2024-03-19T11:31:23.080 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.080 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.080 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:23.081 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.273765+0000", 2024-03-19T11:31:23.081 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0086213880000000007 2024-03-19T11:31:23.081 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.081 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.081 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:23.081 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.273872+0000", 2024-03-19T11:31:23.081 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000107452 2024-03-19T11:31:23.081 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.081 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.081 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:23.081 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.274780+0000", 2024-03-19T11:31:23.081 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00090762000000000004 2024-03-19T11:31:23.081 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.081 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:23.081 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.082 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.082 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.082 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2046 4.4 4:21eac9b2:::smithi03829856-4:head [read 0~2537857,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:31:23.082 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:46.265152+0000", 2024-03-19T11:31:23.082 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 36.738669747000003, 2024-03-19T11:31:23.082 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013074417, 2024-03-19T11:31:23.082 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:23.082 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:23.082 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:23.082 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:23.082 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:23.082 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2046 2024-03-19T11:31:23.082 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.083 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:23.083 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.083 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:23.083 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265152+0000", 2024-03-19T11:31:23.083 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.083 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.083 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.083 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:23.083 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265152+0000", 2024-03-19T11:31:23.083 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.083 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.083 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.083 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:23.083 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265152+0000", 2024-03-19T11:31:23.083 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8999999999999997e-07 2024-03-19T11:31:23.084 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.084 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.084 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:23.084 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265155+0000", 2024-03-19T11:31:23.084 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0920000000000002e-06 2024-03-19T11:31:23.084 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.084 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.084 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:23.084 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265162+0000", 2024-03-19T11:31:23.084 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6730000000000003e-06 2024-03-19T11:31:23.084 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.084 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.084 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:23.084 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265167+0000", 2024-03-19T11:31:23.084 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1839999999999998e-06 2024-03-19T11:31:23.085 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.085 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.085 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:23.085 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.276218+0000", 2024-03-19T11:31:23.085 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011050559999999999 2024-03-19T11:31:23.085 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.085 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.085 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:23.085 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.276345+0000", 2024-03-19T11:31:23.085 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000127243 2024-03-19T11:31:23.085 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.085 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.085 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:23.085 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.278226+0000", 2024-03-19T11:31:23.085 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0018811749999999999 2024-03-19T11:31:23.086 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.086 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:23.086 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.086 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.086 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.086 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:3046 4.19 e103/73 4:9c74b127:::smithi03829856-33:head v 103'148, mlcod=99'144)", 2024-03-19T11:31:23.086 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:22.149246+0000", 2024-03-19T11:31:23.086 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.85457567899999998, 2024-03-19T11:31:23.086 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014101367, 2024-03-19T11:31:23.086 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:23.086 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:23.086 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:23.086 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.086 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:23.087 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.149246+0000", 2024-03-19T11:31:23.087 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.087 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.087 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.087 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:23.087 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.149246+0000", 2024-03-19T11:31:23.087 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.087 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.087 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.087 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:23.087 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.149246+0000", 2024-03-19T11:31:23.087 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2300000000000002e-07 2024-03-19T11:31:23.087 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.087 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.087 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:23.088 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.149663+0000", 2024-03-19T11:31:23.088 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00041629599999999997 2024-03-19T11:31:23.088 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.088 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.088 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:23.088 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.149682+0000", 2024-03-19T11:31:23.088 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9500999999999998e-05 2024-03-19T11:31:23.088 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.088 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.088 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:23.088 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.149692+0000", 2024-03-19T11:31:23.088 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.825e-06 2024-03-19T11:31:23.088 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.088 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.089 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:23.089 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.149797+0000", 2024-03-19T11:31:23.089 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000105222 2024-03-19T11:31:23.089 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.089 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.089 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:23.089 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.149835+0000", 2024-03-19T11:31:23.089 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7934e-05 2024-03-19T11:31:23.089 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.089 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.089 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:23.089 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.163304+0000", 2024-03-19T11:31:23.089 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013469122 2024-03-19T11:31:23.089 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.089 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.090 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:23.090 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.163347+0000", 2024-03-19T11:31:23.090 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3143999999999999e-05 2024-03-19T11:31:23.090 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.090 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:23.090 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.090 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.090 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:23.090 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:23.090 INFO:tasks.rados.rados.0.smithi038.stdout:1406: writing smithi03829856-27 from 433680 to 900383 tid 1 2024-03-19T11:31:23.091 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-19T11:31:23.093 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:23.042+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:23.095 INFO:tasks.rados.rados.0.smithi038.stdout:1406: writing smithi03829856-27 from 1577796 to 2033680 tid 2 2024-03-19T11:31:23.095 INFO:tasks.rados.rados.0.smithi038.stdout:1407: set_chunk oid 9 target oid 2024-03-19T11:31:23.095 INFO:tasks.rados.rados.0.smithi038.stdout:1407: set_chunk oid 9 offset: 466432 length: 1228800 target oid 8165fb845491c7d96696be9d99d1baf015916704e2e8f9b4ef9a3c1b90cd08f2 offset: 0 snap 9 2024-03-19T11:31:23.095 INFO:tasks.rados.rados.0.smithi038.stdout:1408: set_chunk oid 47 target oid 2024-03-19T11:31:23.095 INFO:tasks.rados.rados.0.smithi038.stdout:1408: set_chunk oid 47 offset: 2088448 length: 120832 target oid 97eb33a73f6c66656d935250d706805ae645fbb84e812c0c8879b6e590b9adac offset: 0 snap -1 2024-03-19T11:31:23.095 INFO:tasks.rados.rados.0.smithi038.stdout:1409: read oid 25 snap 27 2024-03-19T11:31:23.095 INFO:tasks.rados.rados.0.smithi038.stdout:1409: expect deleted 2024-03-19T11:31:23.095 INFO:tasks.rados.rados.0.smithi038.stdout:1410: write oid 14 current snap is 30 2024-03-19T11:31:23.095 INFO:tasks.rados.rados.0.smithi038.stdout:1405: got (22) Invalid argument 2024-03-19T11:31:23.095 INFO:tasks.rados.rados.0.smithi038.stdout:1404: finishing set_chunk to oid 39 2024-03-19T11:31:23.095 INFO:tasks.rados.rados.0.smithi038.stdout:1404: got expected ERANGE (src dne) 2024-03-19T11:31:23.095 INFO:tasks.rados.rados.0.smithi038.stdout:1406: finishing write tid 1 to smithi03829856-27 2024-03-19T11:31:23.096 INFO:tasks.rados.rados.0.smithi038.stdout:1406: finishing write tid 2 to smithi03829856-27 2024-03-19T11:31:23.096 INFO:tasks.rados.rados.0.smithi038.stdout:1410: seq_num 241 ranges {450487=541952,1636278=414209} 2024-03-19T11:31:23.096 INFO:tasks.rados.rados.0.smithi038.stdout:1410: writing smithi03829856-14 from 450487 to 992439 tid 1 2024-03-19T11:31:23.096 INFO:tasks.rados.rados.0.smithi038.stdout:1410: writing smithi03829856-14 from 1636278 to 2050487 tid 2 2024-03-19T11:31:23.096 INFO:tasks.rados.rados.0.smithi038.stdout:1404: done (6 left) 2024-03-19T11:31:23.096 INFO:tasks.rados.rados.0.smithi038.stdout:1405: done (5 left) 2024-03-19T11:31:23.096 INFO:tasks.rados.rados.0.smithi038.stdout:1411: read oid 40 snap -1 2024-03-19T11:31:23.096 INFO:tasks.rados.rados.0.smithi038.stdout:1411: expect (ObjNum 227 snap 26 seq_num 227) 2024-03-19T11:31:23.096 INFO:tasks.rados.rados.0.smithi038.stdout:1412: write oid 33 current snap is 30 2024-03-19T11:31:23.096 INFO:tasks.rados.rados.0.smithi038.stdout:1406: finishing write tid 3 to smithi03829856-27 2024-03-19T11:31:23.096 INFO:tasks.rados.rados.0.smithi038.stdout:1406: finishing write tid 4 to smithi03829856-27 2024-03-19T11:31:23.096 INFO:tasks.rados.rados.0.smithi038.stdout:1406: oid 27 updating version 0 to 122 2024-03-19T11:31:23.096 INFO:tasks.rados.rados.0.smithi038.stdout:1406: oid 27 version 122 is already newer than 121 2024-03-19T11:31:23.096 INFO:tasks.rados.rados.0.smithi038.stdout:1406: oid 27 version 122 is already newer than 120 2024-03-19T11:31:23.096 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 27 v 122 (ObjNum 240 snap 30 seq_num 240) dirty exists 2024-03-19T11:31:23.097 INFO:tasks.rados.rados.0.smithi038.stdout:1406: left oid 27 (ObjNum 240 snap 30 seq_num 240) 2024-03-19T11:31:23.097 INFO:tasks.rados.rados.0.smithi038.stdout:1407: finishing set_chunk to oid 9 2024-03-19T11:31:23.097 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 9 set_chunk returned error code -95 offset: 466432 length: 1228800 2024-03-19T11:31:23.097 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 9 v 0 (ObjNum 221 snap 24 seq_num 221) dirty exists 2024-03-19T11:31:23.097 INFO:tasks.rados.rados.0.smithi038.stdout:1412: seq_num 242 ranges {467294=617201,1578407=488887} 2024-03-19T11:31:23.097 INFO:tasks.rados.rados.0.smithi038.stdout:1412: writing smithi03829856-33 from 467294 to 1084495 tid 1 2024-03-19T11:31:23.097 INFO:tasks.rados.rados.0.smithi038.stdout:1412: writing smithi03829856-33 from 1578407 to 2067294 tid 2 2024-03-19T11:31:23.097 INFO:tasks.rados.rados.0.smithi038.stdout:1406: done (6 left) 2024-03-19T11:31:23.097 INFO:tasks.rados.rados.0.smithi038.stdout:1407: done (5 left) 2024-03-19T11:31:23.097 INFO:tasks.rados.rados.0.smithi038.stdout:1413: read oid 31 snap -1 2024-03-19T11:31:23.097 INFO:tasks.rados.rados.0.smithi038.stdout:1413: expect (ObjNum 232 snap 27 seq_num 232) 2024-03-19T11:31:23.097 INFO:tasks.rados.rados.0.smithi038.stdout:1414: write (excl) oid 35 current snap is 30 2024-03-19T11:31:23.097 INFO:tasks.rados.rados.0.smithi038.stdout:1408: finishing set_chunk to oid 47 2024-03-19T11:31:23.097 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 47 set_chunk returned error code -95 offset: 2088448 length: 120832 2024-03-19T11:31:23.097 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-03-19T11:31:23.098 INFO:tasks.rados.rados.0.smithi038.stdout:1411: expect (ObjNum 227 snap 26 seq_num 227) 2024-03-19T11:31:23.136 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:23.145+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:23.236 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:23.246+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:23.262 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:23.263 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:31:23.263 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:31:23.263 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:23.263 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:23.270+0000 7fb17bd6e640 -1 osd.4 103 handle_osd_map injecting map crc failure 2024-03-19T11:31:23.275 INFO:tasks.rados.rados.0.smithi038.stdout:1412: finishing write tid 1 to smithi03829856-33 2024-03-19T11:31:23.275 INFO:tasks.rados.rados.0.smithi038.stdout:1412: finishing write tid 2 to smithi03829856-33 2024-03-19T11:31:23.275 INFO:tasks.rados.rados.0.smithi038.stdout:1412: finishing write tid 3 to smithi03829856-33 2024-03-19T11:31:23.275 INFO:tasks.rados.rados.0.smithi038.stdout:1412: finishing write tid 4 to smithi03829856-33 2024-03-19T11:31:23.276 INFO:tasks.rados.rados.0.smithi038.stdout:1412: oid 33 updating version 0 to 151 2024-03-19T11:31:23.276 INFO:tasks.rados.rados.0.smithi038.stdout:1412: oid 33 updating version 151 to 152 2024-03-19T11:31:23.276 INFO:tasks.rados.rados.0.smithi038.stdout:1412: oid 33 updating version 152 to 153 2024-03-19T11:31:23.276 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 33 v 153 (ObjNum 242 snap 30 seq_num 242) dirty exists 2024-03-19T11:31:23.276 INFO:tasks.rados.rados.0.smithi038.stdout:1412: left oid 33 (ObjNum 242 snap 30 seq_num 242) 2024-03-19T11:31:23.276 INFO:tasks.rados.rados.0.smithi038.stdout:1413: expect (ObjNum 232 snap 27 seq_num 232) 2024-03-19T11:31:23.277 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-19T11:31:23.337 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:23.346+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:23.397 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:23.397 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:31:23.397 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:31:23.397 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:31:23.397 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:23.413 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-19T11:31:23.438 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:23.446+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:23.533 INFO:tasks.rados.rados.0.smithi038.stdout:1414: seq_num 243 ranges {484101=692450,1636889=447212} 2024-03-19T11:31:23.539 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:23.539 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:31:23.539 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:31:23.539 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:31:23.539 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.539 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:58 3.2 3:5f1d333e:::bfcdf551f6f2543327e9c601a4377e9a5ef396f5b97ae8331892d1bcf6b55a9d:head [read 0~161292 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:31:23.539 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.762024+0000", 2024-03-19T11:31:23.539 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 70.786406299000006, 2024-03-19T11:31:23.540 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013495629, 2024-03-19T11:31:23.540 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:23.540 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:23.540 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:23.540 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.540 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:23.540 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762024+0000", 2024-03-19T11:31:23.540 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.540 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.540 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.540 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:23.541 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762024+0000", 2024-03-19T11:31:23.541 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.541 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.541 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.541 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:23.541 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762024+0000", 2024-03-19T11:31:23.541 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9999999999999997e-07 2024-03-19T11:31:23.541 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.541 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.541 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:23.541 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762027+0000", 2024-03-19T11:31:23.541 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0410000000000002e-06 2024-03-19T11:31:23.541 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.542 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.542 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:23.542 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762037+0000", 2024-03-19T11:31:23.542 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1020000000000003e-06 2024-03-19T11:31:23.542 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.542 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.542 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:23.542 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762043+0000", 2024-03-19T11:31:23.542 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4409999999999998e-06 2024-03-19T11:31:23.542 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.542 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.542 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:23.543 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762198+0000", 2024-03-19T11:31:23.543 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015540000000000001 2024-03-19T11:31:23.543 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.543 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.543 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:31:23.543 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762203+0000", 2024-03-19T11:31:23.543 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5109999999999997e-06 2024-03-19T11:31:23.543 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.543 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.543 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:23.543 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.774710+0000", 2024-03-19T11:31:23.543 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012507452 2024-03-19T11:31:23.544 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.544 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.544 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:23.544 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.774754+0000", 2024-03-19T11:31:23.544 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3869999999999998e-05 2024-03-19T11:31:23.544 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.544 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.544 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:23.544 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.775519+0000", 2024-03-19T11:31:23.544 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00076521200000000003 2024-03-19T11:31:23.544 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.544 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:23.544 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.545 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.545 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.545 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:59 3.2 3:78b4b14e:::e1ef0c0c594c36c534f3829406e07ffb60d57b70360f62977e0d545d8ca2bf18:head [read 0~149977 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:31:23.545 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.762054+0000", 2024-03-19T11:31:23.545 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 70.786375867999993, 2024-03-19T11:31:23.545 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015354483, 2024-03-19T11:31:23.545 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:23.545 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:23.545 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:23.545 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.545 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:23.545 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762054+0000", 2024-03-19T11:31:23.546 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.546 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.546 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.546 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:23.546 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762054+0000", 2024-03-19T11:31:23.546 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.546 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.546 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.546 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:23.546 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762055+0000", 2024-03-19T11:31:23.546 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.32e-07 2024-03-19T11:31:23.546 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.547 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.547 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:23.547 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762058+0000", 2024-03-19T11:31:23.547 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9189999999999999e-06 2024-03-19T11:31:23.547 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.547 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.547 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:23.547 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762065+0000", 2024-03-19T11:31:23.547 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2409999999999998e-06 2024-03-19T11:31:23.547 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.547 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.547 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:23.547 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762070+0000", 2024-03-19T11:31:23.548 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6910000000000002e-06 2024-03-19T11:31:23.548 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.548 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.548 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:23.548 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762227+0000", 2024-03-19T11:31:23.548 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015606999999999999 2024-03-19T11:31:23.548 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.548 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.548 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:31:23.548 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762232+0000", 2024-03-19T11:31:23.548 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3029999999999999e-06 2024-03-19T11:31:23.548 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.549 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.549 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:23.549 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.776984+0000", 2024-03-19T11:31:23.549 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014752487999999999 2024-03-19T11:31:23.549 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.549 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.549 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:23.549 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.777030+0000", 2024-03-19T11:31:23.549 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5359999999999999e-05 2024-03-19T11:31:23.549 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.549 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.549 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:23.549 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.777409+0000", 2024-03-19T11:31:23.550 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037897899999999999 2024-03-19T11:31:23.550 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.550 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:23.550 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.550 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.550 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.550 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1067 4.0 4:01018511:::smithi03829856-19:head [write 477332~766101] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:31:23.550 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.950506+0000", 2024-03-19T11:31:23.550 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 69.597924156000005, 2024-03-19T11:31:23.550 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016279426, 2024-03-19T11:31:23.550 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:23.550 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:23.551 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:23.551 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:23.551 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:23.551 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1067 2024-03-19T11:31:23.551 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.551 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:23.551 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.551 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:23.551 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950506+0000", 2024-03-19T11:31:23.551 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.551 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.551 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.552 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:23.552 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950506+0000", 2024-03-19T11:31:23.552 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.552 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.552 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.552 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:23.552 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950509+0000", 2024-03-19T11:31:23.552 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.021e-06 2024-03-19T11:31:23.552 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.552 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.552 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:23.552 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950892+0000", 2024-03-19T11:31:23.553 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00038263300000000001 2024-03-19T11:31:23.553 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.553 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.553 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:23.553 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950912+0000", 2024-03-19T11:31:23.553 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0539000000000001e-05 2024-03-19T11:31:23.553 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.553 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.553 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:23.553 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950922+0000", 2024-03-19T11:31:23.554 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.577e-06 2024-03-19T11:31:23.554 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.554 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.554 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:23.554 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950982+0000", 2024-03-19T11:31:23.554 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0760000000000001e-05 2024-03-19T11:31:23.554 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.554 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.554 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:23.554 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.951058+0000", 2024-03-19T11:31:23.554 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5078000000000002e-05 2024-03-19T11:31:23.555 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.555 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.555 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:23.555 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.951173+0000", 2024-03-19T11:31:23.555 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011516200000000001 2024-03-19T11:31:23.555 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.555 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.555 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:23.555 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.955064+0000", 2024-03-19T11:31:23.555 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003890896 2024-03-19T11:31:23.555 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.555 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.556 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:23.556 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966670+0000", 2024-03-19T11:31:23.556 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011606795 2024-03-19T11:31:23.556 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.556 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.556 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:23.556 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966731+0000", 2024-03-19T11:31:23.556 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0528000000000001e-05 2024-03-19T11:31:23.556 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.556 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.556 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:23.556 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966785+0000", 2024-03-19T11:31:23.557 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4437000000000001e-05 2024-03-19T11:31:23.557 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.557 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:23.557 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.557 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.557 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.557 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1231 4.0 4:0dc3b91e:::smithi03829856-28:head [write 1808051~636013] snapc 9=[9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:31:23.557 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.173016+0000", 2024-03-19T11:31:23.557 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 65.375414043000006, 2024-03-19T11:31:23.557 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013973719000000001, 2024-03-19T11:31:23.557 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:23.557 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:23.558 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:23.558 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:23.558 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:23.558 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1231 2024-03-19T11:31:23.558 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.558 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:23.558 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.558 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:23.558 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173016+0000", 2024-03-19T11:31:23.558 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.558 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.558 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.559 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:23.559 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173016+0000", 2024-03-19T11:31:23.559 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.559 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.559 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.559 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:23.559 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173019+0000", 2024-03-19T11:31:23.559 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.253e-06 2024-03-19T11:31:23.559 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.559 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.559 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:23.559 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176143+0000", 2024-03-19T11:31:23.559 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031240930000000001 2024-03-19T11:31:23.560 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.560 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.560 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:23.560 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176177+0000", 2024-03-19T11:31:23.560 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3237e-05 2024-03-19T11:31:23.560 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.560 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.560 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:23.560 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176196+0000", 2024-03-19T11:31:23.560 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9358e-05 2024-03-19T11:31:23.560 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.560 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.561 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:23.561 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176493+0000", 2024-03-19T11:31:23.561 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00029743899999999998 2024-03-19T11:31:23.561 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.561 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.561 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:23.561 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176786+0000", 2024-03-19T11:31:23.561 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00029259000000000002 2024-03-19T11:31:23.561 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.561 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.561 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:23.562 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177250+0000", 2024-03-19T11:31:23.562 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00046376000000000002 2024-03-19T11:31:23.562 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.562 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.562 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:23.562 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.182511+0000", 2024-03-19T11:31:23.562 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0052609019999999996 2024-03-19T11:31:23.562 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.562 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.562 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:23.562 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.186896+0000", 2024-03-19T11:31:23.563 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0043857009999999997 2024-03-19T11:31:23.563 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.563 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.563 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:23.563 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.186948+0000", 2024-03-19T11:31:23.563 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1653000000000002e-05 2024-03-19T11:31:23.563 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.563 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.563 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:23.563 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.186990+0000", 2024-03-19T11:31:23.563 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1733000000000002e-05 2024-03-19T11:31:23.564 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.564 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:23.564 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.564 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.564 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.564 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1409 4.10 4:0dc3b91e:::smithi03829856-28:head [sparse-read 0~1851556] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:31:23.564 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.918248+0000", 2024-03-19T11:31:23.564 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 57.630181753999999, 2024-03-19T11:31:23.564 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013726986, 2024-03-19T11:31:23.564 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:23.564 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:23.564 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:23.564 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:23.565 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:23.565 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1409 2024-03-19T11:31:23.565 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.565 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:23.565 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.565 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:23.565 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918248+0000", 2024-03-19T11:31:23.565 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.565 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.565 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.565 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:23.565 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918248+0000", 2024-03-19T11:31:23.566 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.566 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.566 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.566 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:23.566 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918249+0000", 2024-03-19T11:31:23.566 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4700000000000002e-07 2024-03-19T11:31:23.566 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.566 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.566 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:23.566 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918251+0000", 2024-03-19T11:31:23.566 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.542e-06 2024-03-19T11:31:23.567 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.567 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.567 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:23.567 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918258+0000", 2024-03-19T11:31:23.567 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4629999999999998e-06 2024-03-19T11:31:23.567 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.567 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.567 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:23.567 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918263+0000", 2024-03-19T11:31:23.567 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8049999999999997e-06 2024-03-19T11:31:23.567 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.567 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.568 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:23.568 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.931469+0000", 2024-03-19T11:31:23.568 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013206458000000001 2024-03-19T11:31:23.568 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.568 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.568 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:23.568 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.931586+0000", 2024-03-19T11:31:23.568 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000116655 2024-03-19T11:31:23.568 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.568 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.569 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:23.569 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.931975+0000", 2024-03-19T11:31:23.569 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00038961600000000002 2024-03-19T11:31:23.569 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.569 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:23.569 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.569 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.569 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.570 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(3.1 49/48 [PushOp(3:9205291d:::smithi03829856-50 low_tier:head, version: 28'36, data_included: [393216~851968,1703936~196608], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:9205291d:::smithi03829856-50 low_tier:head@28'36, size: 2823543, copy_subset: [(0, 2823543)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1900544, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:baab691a:::smithi03829856-2 low_tier:head, version: 28'37, data_included: [393216~524288,1507328~509479], data_size: 1033767, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:baab691a:::smithi03829856-2 low_tier:head@28'37, size: 2016807, copy_subset: [(0, 2016807)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2016807, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:98ce8426:::smithi03829856-4 low_tier:head, version: 28'38, data_included: [393216~720896,1638400~327680], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:98ce8426:::smithi03829856-4 low_tier:head@28'38, size: 2050421, copy_subset: [(0, 2050421)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1966080, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:a86be07b:::smithi03829856-44 low_tier:head, version: 28'41, data_included: [720896~655360,1769472~393216], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:a86be07b:::smithi03829856-44 low_tier:head@28'41, size: 2722701, copy_subset: [(0, 2722701)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2162688, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:bb56919e:::smithi03829856-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'44, data_included: [393216~786432,1703936~262144], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:bb56919e:::smithi03829856-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'44, size: 2806736, copy_subset: [(0, 2806736)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1966080, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:88687f71:::smithi03829856-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'45, data_included: [589824~655360,1638400~393216], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:88687f71:::smithi03829856-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'45, size: 2201684, copy_subset: [(0, 2201684)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2031616, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:b1f822cf:::smithi03829856-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'46, data_included: [720896~786432,1966080~262144], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:b1f822cf:::smithi03829856-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'46, size: 2369754, copy_subset: [(0, 2369754)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2228224, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:a5e8313a:::smithi03829856-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'47, data_included: [524288~720896,1769472~327680], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:a5e8313a:::smithi03829856-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'47, size: 2571438, copy_subset: [(0, 2571438)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:bca5ec64:::e3a26bf9230b1b95b8cf1a4cb01909236bf2bf3fd59b083b2291d278ab98264d:head, version: 28'48, data_included: [0~393216], data_size: 393216, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bca5ec64:::e3a26bf9230b1b95b8cf1a4cb01909236bf2bf3fd59b083b2291d278ab98264d:head@28'48, size: 533504, copy_subset: [(0, 533504)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 533504, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:23.570 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:26.948137+0000", 2024-03-19T11:31:23.570 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 56.600292940999999, 2024-03-19T11:31:23.570 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.036085578, 2024-03-19T11:31:23.570 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:23.570 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:23.570 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:23.571 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.571 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:23.571 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.948137+0000", 2024-03-19T11:31:23.571 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.571 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.571 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.571 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:23.571 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.948137+0000", 2024-03-19T11:31:23.572 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.572 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.572 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.572 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:23.572 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.948139+0000", 2024-03-19T11:31:23.572 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1399999999999998e-06 2024-03-19T11:31:23.572 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.572 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.573 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:23.573 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954288+0000", 2024-03-19T11:31:23.573 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0061487429999999999 2024-03-19T11:31:23.573 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.573 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.573 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:23.573 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954388+0000", 2024-03-19T11:31:23.573 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010022 2024-03-19T11:31:23.573 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.573 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.573 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:23.574 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954523+0000", 2024-03-19T11:31:23.574 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000134332 2024-03-19T11:31:23.574 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.574 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.574 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:23.574 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.955120+0000", 2024-03-19T11:31:23.574 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00059695799999999997 2024-03-19T11:31:23.574 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.574 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.575 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:23.575 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.955194+0000", 2024-03-19T11:31:23.575 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4077000000000002e-05 2024-03-19T11:31:23.575 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.575 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.575 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:23.576 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.984223+0000", 2024-03-19T11:31:23.576 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029029108000000001 2024-03-19T11:31:23.576 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.576 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:23.576 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.576 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.576 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.577 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(3.1 49/48 [PushOp(3:99a49d1b:::895a63407b6109b3cca9a6f5ef827134d56679bef42daadc5892269682c428ed:head, version: 32'49, data_included: [393216~388096], data_size: 388096, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:99a49d1b:::895a63407b6109b3cca9a6f5ef827134d56679bef42daadc5892269682c428ed:head@32'49, size: 781312, copy_subset: [(0, 781312)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 781312, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:23.577 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:26.954637+0000", 2024-03-19T11:31:23.577 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 56.593793324000004, 2024-03-19T11:31:23.577 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.030729057000000001, 2024-03-19T11:31:23.577 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:23.577 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:23.578 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:23.578 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.578 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:23.578 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954637+0000", 2024-03-19T11:31:23.578 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.578 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.578 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.579 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:23.579 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954637+0000", 2024-03-19T11:31:23.579 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.579 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.579 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.579 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:23.579 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954637+0000", 2024-03-19T11:31:23.579 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7299999999999995e-07 2024-03-19T11:31:23.580 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.580 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.580 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:23.580 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954873+0000", 2024-03-19T11:31:23.580 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023520299999999999 2024-03-19T11:31:23.580 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.580 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.580 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:23.581 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954897+0000", 2024-03-19T11:31:23.581 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4110000000000001e-05 2024-03-19T11:31:23.581 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.581 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.581 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:23.581 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954917+0000", 2024-03-19T11:31:23.581 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.067e-05 2024-03-19T11:31:23.581 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.581 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.581 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:23.582 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.984289+0000", 2024-03-19T11:31:23.582 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029371191000000001 2024-03-19T11:31:23.582 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.582 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.582 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:23.582 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.984303+0000", 2024-03-19T11:31:23.582 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4888999999999999e-05 2024-03-19T11:31:23.582 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.583 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.583 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:23.583 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.985366+0000", 2024-03-19T11:31:23.583 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001062321 2024-03-19T11:31:23.583 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.583 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:23.583 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.583 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.583 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.584 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1472 4.10 4:0dc3b91e:::smithi03829856-28:head [read 534528~391680] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-03-19T11:31:23.584 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:27.091748+0000", 2024-03-19T11:31:23.584 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 56.456682436000001, 2024-03-19T11:31:23.584 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.83574118500000005, 2024-03-19T11:31:23.584 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:23.584 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:23.585 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:23.585 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:23.585 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:23.585 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1472 2024-03-19T11:31:23.585 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.585 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:23.585 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.585 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:23.586 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091748+0000", 2024-03-19T11:31:23.586 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.586 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.586 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.586 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:23.586 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091748+0000", 2024-03-19T11:31:23.586 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.587 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.587 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.587 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:23.587 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091750+0000", 2024-03-19T11:31:23.587 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3130000000000001e-06 2024-03-19T11:31:23.587 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.587 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.587 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:23.587 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091760+0000", 2024-03-19T11:31:23.587 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0560999999999999e-05 2024-03-19T11:31:23.588 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.588 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.588 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:23.588 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091784+0000", 2024-03-19T11:31:23.589 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3915000000000001e-05 2024-03-19T11:31:23.589 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.589 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.589 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:23.590 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091797+0000", 2024-03-19T11:31:23.590 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2660000000000001e-05 2024-03-19T11:31:23.590 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.590 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.590 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:23.590 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091874+0000", 2024-03-19T11:31:23.590 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6757999999999999e-05 2024-03-19T11:31:23.591 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.591 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.591 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:23.591 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091882+0000", 2024-03-19T11:31:23.591 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5520000000000005e-06 2024-03-19T11:31:23.591 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.592 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.592 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:23.592 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.927033+0000", 2024-03-19T11:31:23.592 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.83515094099999998 2024-03-19T11:31:23.593 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.593 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.593 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:23.593 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.927250+0000", 2024-03-19T11:31:23.593 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00021675199999999999 2024-03-19T11:31:23.593 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.593 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.594 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:23.594 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.927489+0000", 2024-03-19T11:31:23.594 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023873300000000001 2024-03-19T11:31:23.594 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.594 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:23.594 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.594 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.595 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.595 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1474 4.10 4:0dc3b91e:::smithi03829856-28:head [assert-version v27,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-03-19T11:31:23.595 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:27.934139+0000", 2024-03-19T11:31:23.595 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 55.614290650000001, 2024-03-19T11:31:23.595 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024764884000000001, 2024-03-19T11:31:23.595 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:23.595 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:23.596 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:23.596 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:23.596 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:23.596 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1474 2024-03-19T11:31:23.596 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.596 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:23.596 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.596 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:23.596 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934139+0000", 2024-03-19T11:31:23.596 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.597 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.597 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.597 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:23.597 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934139+0000", 2024-03-19T11:31:23.597 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.597 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.597 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.597 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:23.597 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934142+0000", 2024-03-19T11:31:23.598 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5459999999999998e-06 2024-03-19T11:31:23.598 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.598 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.598 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:23.598 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934152+0000", 2024-03-19T11:31:23.598 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9399999999999997e-06 2024-03-19T11:31:23.598 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.598 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.598 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:23.598 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934173+0000", 2024-03-19T11:31:23.599 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1522000000000002e-05 2024-03-19T11:31:23.599 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.599 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.599 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:23.599 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934185+0000", 2024-03-19T11:31:23.599 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1946999999999999e-05 2024-03-19T11:31:23.599 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.600 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.600 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:23.600 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934282+0000", 2024-03-19T11:31:23.600 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6272000000000003e-05 2024-03-19T11:31:23.600 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.600 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.600 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:23.601 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934379+0000", 2024-03-19T11:31:23.601 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6966999999999995e-05 2024-03-19T11:31:23.601 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.601 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.601 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:23.601 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.955023+0000", 2024-03-19T11:31:23.602 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020644302 2024-03-19T11:31:23.602 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.602 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.602 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:23.602 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.955167+0000", 2024-03-19T11:31:23.602 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014404500000000001 2024-03-19T11:31:23.602 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.602 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.602 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1", 2024-03-19T11:31:23.602 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.957913+0000", 2024-03-19T11:31:23.603 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0027463779999999998 2024-03-19T11:31:23.603 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.603 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.603 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:23.603 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.958548+0000", 2024-03-19T11:31:23.604 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00063495700000000001 2024-03-19T11:31:23.604 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.604 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.604 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:23.604 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.958785+0000", 2024-03-19T11:31:23.604 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000236872 2024-03-19T11:31:23.605 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.605 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.605 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:23.605 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.958854+0000", 2024-03-19T11:31:23.605 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9099999999999999e-05 2024-03-19T11:31:23.605 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.605 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.605 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:23.606 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.958904+0000", 2024-03-19T11:31:23.606 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0036e-05 2024-03-19T11:31:23.606 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.606 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:23.606 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.606 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.607 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.607 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2011 4.0 4:01018511:::smithi03829856-19:c [read 0~3563051,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:31:23.607 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:46.264515+0000", 2024-03-19T11:31:23.607 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 37.283914983000003, 2024-03-19T11:31:23.607 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013774359999999999, 2024-03-19T11:31:23.607 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:23.607 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:23.608 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:23.608 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:23.608 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:23.608 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2011 2024-03-19T11:31:23.608 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.608 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:23.608 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.608 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:23.608 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264515+0000", 2024-03-19T11:31:23.609 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.609 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.609 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.609 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:23.609 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264515+0000", 2024-03-19T11:31:23.609 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.609 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.609 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.610 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:23.610 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264516+0000", 2024-03-19T11:31:23.610 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5899999999999995e-07 2024-03-19T11:31:23.610 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.610 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.610 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:23.610 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264522+0000", 2024-03-19T11:31:23.610 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6019999999999999e-06 2024-03-19T11:31:23.610 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.611 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.611 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:23.611 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264535+0000", 2024-03-19T11:31:23.611 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3207000000000001e-05 2024-03-19T11:31:23.611 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.611 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.611 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:23.612 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264543+0000", 2024-03-19T11:31:23.612 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6270000000000006e-06 2024-03-19T11:31:23.612 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.612 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.612 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:23.612 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.271665+0000", 2024-03-19T11:31:23.612 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.007121881 2024-03-19T11:31:23.613 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.613 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.613 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:23.613 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.271835+0000", 2024-03-19T11:31:23.613 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000169958 2024-03-19T11:31:23.613 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.613 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.613 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:23.613 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.278289+0000", 2024-03-19T11:31:23.613 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0064542260000000004 2024-03-19T11:31:23.614 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.614 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:23.614 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.614 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.614 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.614 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 75/74 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:7, version: 55'82, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@55'82, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 58'86, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@58'86, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [504210~968657,1572695~79919], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:12, version: 66'101, data_included: [440350~816629,1309629~231947], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:12@66'101, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:10, version: 70'112, data_included: [440350~816629,1309629~231947], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:10@70'112, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 70'113, data_included: [440350~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@70'113, size: 2605085, copy_subset: [(0, 2605085)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1488926, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:23.615 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.108941+0000", 2024-03-19T11:31:23.615 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 29.439488913000002, 2024-03-19T11:31:23.615 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.067446694000000001, 2024-03-19T11:31:23.615 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:23.615 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:23.615 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:23.615 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.616 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:23.616 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.108941+0000", 2024-03-19T11:31:23.616 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.616 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.616 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.616 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:23.616 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.108941+0000", 2024-03-19T11:31:23.616 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.616 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.616 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.616 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:23.617 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.108943+0000", 2024-03-19T11:31:23.617 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2259999999999999e-06 2024-03-19T11:31:23.617 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.617 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.617 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:23.617 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123892+0000", 2024-03-19T11:31:23.618 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014948655999999999 2024-03-19T11:31:23.618 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.618 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.618 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:23.618 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123993+0000", 2024-03-19T11:31:23.618 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000101236 2024-03-19T11:31:23.618 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.618 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.618 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:23.618 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124126+0000", 2024-03-19T11:31:23.619 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000133184 2024-03-19T11:31:23.619 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.619 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.619 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:23.619 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124655+0000", 2024-03-19T11:31:23.619 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00052849600000000004 2024-03-19T11:31:23.619 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.619 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.620 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:23.620 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124723+0000", 2024-03-19T11:31:23.620 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7797000000000002e-05 2024-03-19T11:31:23.620 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.620 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.620 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:23.620 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.176388+0000", 2024-03-19T11:31:23.620 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.051665098999999999 2024-03-19T11:31:23.620 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.620 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:23.621 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.621 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.621 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.621 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.21 e75,74 4:8401c1f0:::smithi03829856-43:8,46'66,4:866f3dcc:::smithi03829856-31:2,47'73)", 2024-03-19T11:31:23.621 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.124226+0000", 2024-03-19T11:31:23.621 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 29.424204257, 2024-03-19T11:31:23.621 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.052217913999999997, 2024-03-19T11:31:23.621 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:23.621 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:23.621 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:23.622 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.622 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:23.622 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124226+0000", 2024-03-19T11:31:23.622 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.623 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.623 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.623 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:23.623 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124226+0000", 2024-03-19T11:31:23.623 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.623 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.624 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.624 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:23.624 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124227+0000", 2024-03-19T11:31:23.624 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.477e-06 2024-03-19T11:31:23.624 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.625 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.625 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:23.625 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124240+0000", 2024-03-19T11:31:23.625 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2379e-05 2024-03-19T11:31:23.625 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.626 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.626 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:23.626 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124264+0000", 2024-03-19T11:31:23.626 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4224999999999999e-05 2024-03-19T11:31:23.626 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.626 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.626 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:23.626 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124287+0000", 2024-03-19T11:31:23.627 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3557000000000002e-05 2024-03-19T11:31:23.627 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.627 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.627 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:23.627 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.176274+0000", 2024-03-19T11:31:23.627 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.051986248999999998 2024-03-19T11:31:23.627 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.627 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.627 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:23.627 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.176285+0000", 2024-03-19T11:31:23.627 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0818999999999999e-05 2024-03-19T11:31:23.627 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.628 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.628 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:23.628 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.176444+0000", 2024-03-19T11:31:23.628 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015920800000000001 2024-03-19T11:31:23.628 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.628 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:23.629 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.629 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.629 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.629 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 75/74 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:7, version: 55'82, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@55'82, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 58'86, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@58'86, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [1652614~973634,2888511~52681,3182305~22261], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3204566, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:12, version: 66'101, data_included: [1541576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:12@66'101, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:10, version: 70'112, data_included: [1541576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:10@70'112, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 70'113, data_included: [1488926~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@70'113, size: 2605085, copy_subset: [(0, 2605085)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2537502, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1488926, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:23.629 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.195615+0000", 2024-03-19T11:31:23.629 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 29.352815360000001, 2024-03-19T11:31:23.629 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.036930510999999999, 2024-03-19T11:31:23.629 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:23.630 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:23.630 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:23.630 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.630 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:23.630 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.195615+0000", 2024-03-19T11:31:23.630 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.630 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.631 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.631 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:23.631 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.195615+0000", 2024-03-19T11:31:23.631 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.631 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.631 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.631 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:23.631 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.195617+0000", 2024-03-19T11:31:23.632 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0490000000000002e-06 2024-03-19T11:31:23.632 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.632 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.632 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:23.632 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204654+0000", 2024-03-19T11:31:23.632 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0090370339999999993 2024-03-19T11:31:23.632 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.632 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.632 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:23.632 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204730+0000", 2024-03-19T11:31:23.633 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5939000000000006e-05 2024-03-19T11:31:23.633 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.633 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.633 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:23.633 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204844+0000", 2024-03-19T11:31:23.633 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000114066 2024-03-19T11:31:23.633 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.634 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.634 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:23.634 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.205232+0000", 2024-03-19T11:31:23.634 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000388045 2024-03-19T11:31:23.634 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.634 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.634 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:23.634 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.205277+0000", 2024-03-19T11:31:23.634 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5071999999999998e-05 2024-03-19T11:31:23.634 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.635 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.635 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:23.635 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.232545+0000", 2024-03-19T11:31:23.635 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027268305999999999 2024-03-19T11:31:23.635 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.635 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:23.635 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.635 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.635 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.635 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.21 e75,74 4:866f3dcc:::smithi03829856-31:c,70'114)", 2024-03-19T11:31:23.635 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.204941+0000", 2024-03-19T11:31:23.635 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 29.343488735000001, 2024-03-19T11:31:23.636 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.028797390999999999, 2024-03-19T11:31:23.636 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:23.636 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:23.636 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:23.636 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.636 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:23.636 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204941+0000", 2024-03-19T11:31:23.636 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.636 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.636 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.637 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:23.637 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204941+0000", 2024-03-19T11:31:23.637 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.637 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.637 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.637 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:23.637 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204943+0000", 2024-03-19T11:31:23.637 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.35e-06 2024-03-19T11:31:23.637 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.637 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.637 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:23.638 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204950+0000", 2024-03-19T11:31:23.638 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.374e-06 2024-03-19T11:31:23.638 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.638 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.638 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:23.638 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204965+0000", 2024-03-19T11:31:23.638 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5169e-05 2024-03-19T11:31:23.638 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.639 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.639 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:23.639 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204979+0000", 2024-03-19T11:31:23.639 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4260000000000001e-05 2024-03-19T11:31:23.639 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.639 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.639 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:23.639 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.232607+0000", 2024-03-19T11:31:23.639 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027627518 2024-03-19T11:31:23.639 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.640 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.640 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:23.640 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.232626+0000", 2024-03-19T11:31:23.640 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9471999999999999e-05 2024-03-19T11:31:23.640 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.640 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.640 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:23.640 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.233739+0000", 2024-03-19T11:31:23.640 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011122479999999999 2024-03-19T11:31:23.641 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.641 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:23.641 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.641 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.641 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.641 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 75/74 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [2599014~241336], data_size: 241336, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2840350, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [2599014~241336,2948041~446940], data_size: 688276, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3394981, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [3204566~621403], data_size: 621403, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3865577, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3204566, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:12, version: 66'101, data_included: [2590152~250198,2948041~581396], data_size: 831594, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:12@66'101, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3529437, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:10, version: 70'112, data_included: [2590152~250198,2948041~581396], data_size: 831594, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:10@70'112, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3529437, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 70'113, data_included: [2537502~67583], data_size: 67583, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@70'113, size: 2605085, copy_subset: [(0, 2605085)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2605085, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2537502, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:23.641 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.241268+0000", 2024-03-19T11:31:23.641 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 29.307161614999998, 2024-03-19T11:31:23.641 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015726741999999998, 2024-03-19T11:31:23.641 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:23.641 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:23.642 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:23.642 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.642 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:23.642 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.241268+0000", 2024-03-19T11:31:23.642 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.642 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.642 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.642 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:23.642 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.241268+0000", 2024-03-19T11:31:23.643 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.643 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.643 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.643 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:23.643 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.241270+0000", 2024-03-19T11:31:23.643 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5659999999999999e-06 2024-03-19T11:31:23.643 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.643 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.643 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:23.644 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.245791+0000", 2024-03-19T11:31:23.644 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0045205810000000001 2024-03-19T11:31:23.644 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.644 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.644 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:23.644 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.245853+0000", 2024-03-19T11:31:23.644 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2671999999999999e-05 2024-03-19T11:31:23.644 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.644 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.644 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:23.644 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.245916+0000", 2024-03-19T11:31:23.645 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2626000000000007e-05 2024-03-19T11:31:23.645 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.645 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.645 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:23.645 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.246176+0000", 2024-03-19T11:31:23.645 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000260136 2024-03-19T11:31:23.645 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.645 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.645 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:23.645 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.246210+0000", 2024-03-19T11:31:23.646 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4183000000000001e-05 2024-03-19T11:31:23.646 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.646 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.646 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:23.646 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.256995+0000", 2024-03-19T11:31:23.646 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010784978000000001 2024-03-19T11:31:23.646 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.646 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:23.646 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.646 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.646 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.646 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 75/74 [PushOp(4:866f3dcc:::smithi03829856-31:head, version: 70'115, data_included: [406769~773083,1439434~275493], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:head@70'115, size: 2806769, copy_subset: [(0, 2806769)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1714927, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:23.647 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.245999+0000", 2024-03-19T11:31:23.647 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 29.302431029000001, 2024-03-19T11:31:23.647 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014650178999999999, 2024-03-19T11:31:23.647 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:23.647 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:23.647 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:23.647 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.647 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:23.648 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.245999+0000", 2024-03-19T11:31:23.648 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.648 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.648 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.648 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:23.648 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.245999+0000", 2024-03-19T11:31:23.648 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.648 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.648 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.648 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:23.648 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.246000+0000", 2024-03-19T11:31:23.649 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9800000000000003e-07 2024-03-19T11:31:23.649 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.649 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.649 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:23.649 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.246544+0000", 2024-03-19T11:31:23.649 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00054456700000000003 2024-03-19T11:31:23.649 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.649 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.649 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:23.649 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.246578+0000", 2024-03-19T11:31:23.650 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3995999999999998e-05 2024-03-19T11:31:23.650 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.650 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.650 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:23.650 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.246595+0000", 2024-03-19T11:31:23.650 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6558e-05 2024-03-19T11:31:23.650 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.651 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.651 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:23.651 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.257102+0000", 2024-03-19T11:31:23.651 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010506925 2024-03-19T11:31:23.651 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.651 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.651 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:23.651 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.257126+0000", 2024-03-19T11:31:23.651 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4443e-05 2024-03-19T11:31:23.652 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.652 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.652 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:23.652 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.260649+0000", 2024-03-19T11:31:23.652 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003522992 2024-03-19T11:31:23.652 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.652 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:23.652 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.652 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.652 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.653 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2609 4.19 4:9bd2ffb1:::smithi03829856-30:head [read 0~2487403,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e89)", 2024-03-19T11:31:23.653 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:08.055259+0000", 2024-03-19T11:31:23.653 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 15.493171438999999, 2024-03-19T11:31:23.653 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014845676, 2024-03-19T11:31:23.653 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:23.653 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:23.653 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:23.653 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:23.653 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:23.654 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2609 2024-03-19T11:31:23.654 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.654 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:23.654 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.654 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:23.654 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:08.055259+0000", 2024-03-19T11:31:23.654 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.654 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.654 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.654 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:23.654 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:08.055259+0000", 2024-03-19T11:31:23.655 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.655 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.655 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.655 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:23.655 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:08.055259+0000", 2024-03-19T11:31:23.655 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8500000000000001e-07 2024-03-19T11:31:23.655 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.655 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.655 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:23.655 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:08.055265+0000", 2024-03-19T11:31:23.655 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.558e-06 2024-03-19T11:31:23.656 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.656 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.656 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:23.656 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:08.055274+0000", 2024-03-19T11:31:23.656 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0000000000000002e-06 2024-03-19T11:31:23.656 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.656 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.656 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:23.656 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:08.055282+0000", 2024-03-19T11:31:23.656 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.011e-06 2024-03-19T11:31:23.657 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.657 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.657 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:23.657 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:08.055667+0000", 2024-03-19T11:31:23.657 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00038538100000000002 2024-03-19T11:31:23.657 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.657 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.657 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:31:23.658 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:08.055688+0000", 2024-03-19T11:31:23.658 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0729000000000001e-05 2024-03-19T11:31:23.658 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.658 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.658 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:23.658 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:08.066671+0000", 2024-03-19T11:31:23.658 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010983504 2024-03-19T11:31:23.658 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.658 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.659 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:23.659 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:08.066996+0000", 2024-03-19T11:31:23.659 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032403899999999998 2024-03-19T11:31:23.659 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.659 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.659 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:23.660 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:08.070104+0000", 2024-03-19T11:31:23.660 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031087689999999999 2024-03-19T11:31:23.660 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.660 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:23.660 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.660 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.660 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.660 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2780 3.2 3:47f177ee:::0a67e437b20d4ad126b67a2338de84fe4ef684c5c77cee818318d652193ba67c:head [write 0~2533888] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-03-19T11:31:23.660 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:12.308707+0000", 2024-03-19T11:31:23.660 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 11.239722684, 2024-03-19T11:31:23.660 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014502703, 2024-03-19T11:31:23.660 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:23.661 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:23.661 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:23.661 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:23.661 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:23.661 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2780 2024-03-19T11:31:23.661 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.661 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:23.661 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.661 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:23.661 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.308707+0000", 2024-03-19T11:31:23.661 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.661 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.662 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.662 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:23.662 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.308707+0000", 2024-03-19T11:31:23.662 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.662 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.663 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.663 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:23.663 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.308709+0000", 2024-03-19T11:31:23.663 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6050000000000001e-06 2024-03-19T11:31:23.663 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.663 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.664 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:23.664 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.309858+0000", 2024-03-19T11:31:23.664 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011490339999999999 2024-03-19T11:31:23.664 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.664 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.664 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:23.665 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.309879+0000", 2024-03-19T11:31:23.665 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1254e-05 2024-03-19T11:31:23.665 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.665 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.665 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:23.665 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.309888+0000", 2024-03-19T11:31:23.665 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.67e-06 2024-03-19T11:31:23.665 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.665 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.666 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:23.666 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.310032+0000", 2024-03-19T11:31:23.666 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000144499 2024-03-19T11:31:23.666 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.666 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.666 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:23.666 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.310438+0000", 2024-03-19T11:31:23.666 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00040561999999999998 2024-03-19T11:31:23.666 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.666 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.666 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:31:23.667 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.310835+0000", 2024-03-19T11:31:23.667 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00039648400000000001 2024-03-19T11:31:23.667 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.667 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.667 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:23.667 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.322685+0000", 2024-03-19T11:31:23.667 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011850673000000001 2024-03-19T11:31:23.668 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.668 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.668 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:23.668 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.323012+0000", 2024-03-19T11:31:23.668 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000327032 2024-03-19T11:31:23.668 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.668 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.668 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:23.668 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.323108+0000", 2024-03-19T11:31:23.668 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.5570999999999997e-05 2024-03-19T11:31:23.669 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.669 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.669 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:23.669 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.323210+0000", 2024-03-19T11:31:23.669 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000102261 2024-03-19T11:31:23.669 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.669 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:23.669 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.669 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.670 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.670 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2789 3.2 3:48367551:::ca87bd16019841aaa831bb522ee079e5f4e6a1ea67fbcb51e5f89263b6c5d8f6:head [write 0~3280896] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-03-19T11:31:23.670 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:12.340289+0000", 2024-03-19T11:31:23.670 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 11.208141018999999, 2024-03-19T11:31:23.670 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016291298999999999, 2024-03-19T11:31:23.670 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:23.670 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:23.670 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:23.670 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:23.670 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:23.670 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2789 2024-03-19T11:31:23.670 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.671 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:23.671 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.671 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:23.671 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.340289+0000", 2024-03-19T11:31:23.671 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.671 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.671 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.671 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:23.672 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.340289+0000", 2024-03-19T11:31:23.672 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.672 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.672 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.672 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:23.672 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.340291+0000", 2024-03-19T11:31:23.672 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1270000000000001e-06 2024-03-19T11:31:23.672 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.673 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.673 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:23.673 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.341426+0000", 2024-03-19T11:31:23.673 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011348459999999999 2024-03-19T11:31:23.673 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.673 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.673 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:23.673 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.341454+0000", 2024-03-19T11:31:23.673 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8204e-05 2024-03-19T11:31:23.673 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.673 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.674 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:23.674 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.341467+0000", 2024-03-19T11:31:23.674 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2700999999999999e-05 2024-03-19T11:31:23.674 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.674 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.674 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:23.674 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.341623+0000", 2024-03-19T11:31:23.674 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015600100000000001 2024-03-19T11:31:23.674 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.675 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.675 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:23.675 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.342034+0000", 2024-03-19T11:31:23.675 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000411247 2024-03-19T11:31:23.675 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.675 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.675 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:31:23.675 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.342443+0000", 2024-03-19T11:31:23.675 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00040901400000000002 2024-03-19T11:31:23.675 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.676 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.676 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:23.676 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.356159+0000", 2024-03-19T11:31:23.676 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013716058999999999 2024-03-19T11:31:23.676 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.676 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.676 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:23.677 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.356462+0000", 2024-03-19T11:31:23.677 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000302594 2024-03-19T11:31:23.677 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.677 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.677 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:23.677 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.356530+0000", 2024-03-19T11:31:23.677 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7852e-05 2024-03-19T11:31:23.677 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.677 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.677 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:23.677 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.356580+0000", 2024-03-19T11:31:23.678 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0654e-05 2024-03-19T11:31:23.678 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.678 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:23.678 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.678 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.678 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.678 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3046 4.19 4:9c74b127:::smithi03829856-33:head [write 400066~716205,stat] snapc 1f=[1f,1e,1d,1c,1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-03-19T11:31:23.678 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:22.147457+0000", 2024-03-19T11:31:23.678 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 1.4009727350000001, 2024-03-19T11:31:23.679 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016153588, 2024-03-19T11:31:23.679 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:23.679 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:23.679 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:23.679 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:23.679 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:23.679 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3046 2024-03-19T11:31:23.680 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.680 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:23.680 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.680 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:23.680 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.147457+0000", 2024-03-19T11:31:23.680 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.680 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.680 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.680 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:23.681 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.147457+0000", 2024-03-19T11:31:23.681 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:23.681 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.681 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.681 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:23.681 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.147459+0000", 2024-03-19T11:31:23.681 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1160000000000002e-06 2024-03-19T11:31:23.681 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.681 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.681 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:23.681 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.148083+0000", 2024-03-19T11:31:23.682 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00062361999999999997 2024-03-19T11:31:23.682 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.682 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.682 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:23.682 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.148100+0000", 2024-03-19T11:31:23.682 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6524999999999998e-05 2024-03-19T11:31:23.682 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.682 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.683 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:23.683 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.148110+0000", 2024-03-19T11:31:23.683 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0093e-05 2024-03-19T11:31:23.683 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.683 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.683 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:23.683 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.148197+0000", 2024-03-19T11:31:23.683 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7639000000000006e-05 2024-03-19T11:31:23.683 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.684 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.684 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:23.684 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.148359+0000", 2024-03-19T11:31:23.684 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000161953 2024-03-19T11:31:23.684 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.684 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.685 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 2", 2024-03-19T11:31:23.685 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.149074+0000", 2024-03-19T11:31:23.685 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00071444100000000001 2024-03-19T11:31:23.685 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.685 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.685 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:23.686 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.158856+0000", 2024-03-19T11:31:23.686 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0097818169999999999 2024-03-19T11:31:23.686 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.686 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.686 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:23.686 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.163525+0000", 2024-03-19T11:31:23.686 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046690270000000001 2024-03-19T11:31:23.686 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.686 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.687 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:23.687 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.163569+0000", 2024-03-19T11:31:23.687 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4279000000000001e-05 2024-03-19T11:31:23.687 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:23.687 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:23.687 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:23.687 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.163611+0000", 2024-03-19T11:31:23.687 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2077999999999999e-05 2024-03-19T11:31:23.687 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.687 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:23.687 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.688 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:23.688 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:23.688 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:23.688 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:23.545+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:23.690 INFO:tasks.rados.rados.0.smithi038.stdout:1414: writing smithi03829856-35 from 484101 to 1176551 tid 1 2024-03-19T11:31:23.690 INFO:tasks.rados.rados.0.smithi038.stdout:1414: writing smithi03829856-35 from 1636889 to 2084101 tid 2 2024-03-19T11:31:23.690 INFO:tasks.rados.rados.0.smithi038.stdout:1408: done (6 left) 2024-03-19T11:31:23.690 INFO:tasks.rados.rados.0.smithi038.stdout:1409: done (5 left) 2024-03-19T11:31:23.690 INFO:tasks.rados.rados.0.smithi038.stdout:1411: done (4 left) 2024-03-19T11:31:23.690 INFO:tasks.rados.rados.0.smithi038.stdout:1412: done (3 left) 2024-03-19T11:31:23.690 INFO:tasks.rados.rados.0.smithi038.stdout:1413: done (2 left) 2024-03-19T11:31:23.691 INFO:tasks.rados.rados.0.smithi038.stdout:1415: write oid 21 current snap is 30 2024-03-19T11:31:23.691 INFO:tasks.rados.rados.0.smithi038.stdout:1415: seq_num 244 ranges {500908=484052,1411724=689184} 2024-03-19T11:31:23.691 INFO:tasks.rados.rados.0.smithi038.stdout:1415: writing smithi03829856-21 from 500908 to 984960 tid 1 2024-03-19T11:31:23.693 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-19T11:31:23.696 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-19T11:31:23.697 INFO:tasks.rados.rados.0.smithi038.stdout:1415: writing smithi03829856-21 from 1411724 to 2100908 tid 2 2024-03-19T11:31:23.697 INFO:tasks.rados.rados.0.smithi038.stdout:1416: read oid 30 snap -1 2024-03-19T11:31:23.697 INFO:tasks.rados.rados.0.smithi038.stdout:1416: expect (ObjNum 29 snap 0 seq_num 29) 2024-03-19T11:31:23.697 INFO:tasks.rados.rados.0.smithi038.stdout:1417: read oid 13 snap -1 2024-03-19T11:31:23.697 INFO:tasks.rados.rados.0.smithi038.stdout:1417: expect (ObjNum 12 snap 0 seq_num 12) 2024-03-19T11:31:23.697 INFO:tasks.rados.rados.0.smithi038.stdout:1418: write (excl) oid 32 current snap is 30 2024-03-19T11:31:23.697 INFO:tasks.rados.rados.0.smithi038.stdout:1418: seq_num 245 ranges {517715=559301,1870206=247509} 2024-03-19T11:31:23.697 INFO:tasks.rados.rados.0.smithi038.stdout:1418: writing smithi03829856-32 from 517715 to 1077016 tid 1 2024-03-19T11:31:23.698 INFO:tasks.rados.rados.0.smithi038.stdout:1418: writing smithi03829856-32 from 1870206 to 2117715 tid 2 2024-03-19T11:31:23.698 INFO:tasks.rados.rados.0.smithi038.stdout:1419: read oid 26 snap -1 2024-03-19T11:31:23.698 INFO:tasks.rados.rados.0.smithi038.stdout:1419: expect deleted 2024-03-19T11:31:23.698 INFO:tasks.rados.rados.0.smithi038.stdout:1420: read oid 19 snap -1 2024-03-19T11:31:23.698 INFO:tasks.rados.rados.0.smithi038.stdout:1420: expect (ObjNum 223 snap 25 seq_num 223) 2024-03-19T11:31:23.698 INFO:tasks.rados.rados.0.smithi038.stdout:1421: read oid 41 snap -1 2024-03-19T11:31:23.698 INFO:tasks.rados.rados.0.smithi038.stdout:1421: expect (ObjNum 170 snap 17 seq_num 170) 2024-03-19T11:31:23.698 INFO:tasks.rados.rados.0.smithi038.stdout:1422: tier_promote oid 3 2024-03-19T11:31:23.699 INFO:tasks.rados.rados.0.smithi038.stdout:1416: expect (ObjNum 29 snap 0 seq_num 29) 2024-03-19T11:31:23.699 INFO:tasks.rados.rados.0.smithi038.stdout:1414: finishing write tid 1 to smithi03829856-35 2024-03-19T11:31:23.699 INFO:tasks.rados.rados.0.smithi038.stdout:1417: expect (ObjNum 12 snap 0 seq_num 12) 2024-03-19T11:31:23.700 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:23.706+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:23.718 INFO:tasks.rados.rados.0.smithi038.stdout:1422: got (0) Success 2024-03-19T11:31:23.719 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 3 v 69 (ObjNum 97 snap 0 seq_num 1) clean dne 2024-03-19T11:31:23.719 INFO:tasks.rados.rados.0.smithi038.stdout:1414: finishing write tid 2 to smithi03829856-35 2024-03-19T11:31:23.719 INFO:tasks.rados.rados.0.smithi038.stdout:1414: finishing write tid 3 to smithi03829856-35 2024-03-19T11:31:23.719 INFO:tasks.rados.rados.0.smithi038.stdout:1421: expect (ObjNum 170 snap 17 seq_num 170) 2024-03-19T11:31:23.803 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:23.809+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:23.808 INFO:tasks.rados.rados.0.smithi038.stdout:1414: finishing write tid 4 to smithi03829856-35 2024-03-19T11:31:23.808 INFO:tasks.rados.rados.0.smithi038.stdout:1414: oid 35 updating version 0 to 222 2024-03-19T11:31:23.808 INFO:tasks.rados.rados.0.smithi038.stdout:1414: oid 35 version 222 is already newer than 220 2024-03-19T11:31:23.808 INFO:tasks.rados.rados.0.smithi038.stdout:1414: oid 35 version 222 is already newer than 221 2024-03-19T11:31:23.808 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 35 v 222 (ObjNum 243 snap 30 seq_num 243) dirty exists 2024-03-19T11:31:23.808 INFO:tasks.rados.rados.0.smithi038.stdout:1414: left oid 35 (ObjNum 243 snap 30 seq_num 243) 2024-03-19T11:31:23.808 INFO:tasks.rados.rados.0.smithi038.stdout:1415: finishing write tid 1 to smithi03829856-21 2024-03-19T11:31:23.809 INFO:tasks.rados.rados.0.smithi038.stdout:1415: finishing write tid 2 to smithi03829856-21 2024-03-19T11:31:23.809 INFO:tasks.rados.rados.0.smithi038.stdout:1420: expect (ObjNum 223 snap 25 seq_num 223) 2024-03-19T11:31:23.833 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:23.833 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:31:23.833 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:31:23.833 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:23.848 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-19T11:31:23.865 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:31:23.865 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:31:23.881 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:31:23.882 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:31:23.897 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:31:23.897 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:31:23.901 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:23.909+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:23.914 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:31:23.915 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:31:23.931 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:31:23.932 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:31:23.951 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:31:23.951 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:31:23.969 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:31:23.970 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:31:23.975 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:23.975 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:23.988 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:31:23.988 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:31:23.989 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-19T11:31:24.005 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:24.014+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:24.034 INFO:tasks.rados.rados.0.smithi038.stdout:1415: finishing write tid 3 to smithi03829856-21 2024-03-19T11:31:24.034 INFO:tasks.rados.rados.0.smithi038.stdout:1415: finishing write tid 4 to smithi03829856-21 2024-03-19T11:31:24.034 INFO:tasks.rados.rados.0.smithi038.stdout:1415: oid 21 updating version 0 to 224 2024-03-19T11:31:24.035 INFO:tasks.rados.rados.0.smithi038.stdout:1415: oid 21 updating version 224 to 225 2024-03-19T11:31:24.035 INFO:tasks.rados.rados.0.smithi038.stdout:1415: oid 21 updating version 225 to 226 2024-03-19T11:31:24.035 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 21 v 226 (ObjNum 244 snap 30 seq_num 244) dirty exists 2024-03-19T11:31:24.035 INFO:tasks.rados.rados.0.smithi038.stdout:1415: left oid 21 (ObjNum 244 snap 30 seq_num 244) 2024-03-19T11:31:24.035 INFO:tasks.rados.rados.0.smithi038.stdout:1418: finishing write tid 1 to smithi03829856-32 2024-03-19T11:31:24.035 INFO:tasks.rados.rados.0.smithi038.stdout:1418: finishing write tid 2 to smithi03829856-32 2024-03-19T11:31:24.035 INFO:tasks.rados.rados.0.smithi038.stdout:1418: finishing write tid 3 to smithi03829856-32 2024-03-19T11:31:24.035 INFO:tasks.rados.rados.0.smithi038.stdout:1418: finishing write tid 4 to smithi03829856-32 2024-03-19T11:31:24.035 INFO:tasks.rados.rados.0.smithi038.stdout:1418: oid 32 updating version 0 to 109 2024-03-19T11:31:24.035 INFO:tasks.rados.rados.0.smithi038.stdout:1418: oid 32 version 109 is already newer than 108 2024-03-19T11:31:24.035 INFO:tasks.rados.rados.0.smithi038.stdout:1418: oid 32 updating version 109 to 110 2024-03-19T11:31:24.035 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 32 v 110 (ObjNum 245 snap 30 seq_num 245) dirty exists 2024-03-19T11:31:24.035 INFO:tasks.rados.rados.0.smithi038.stdout:1418: left oid 32 (ObjNum 245 snap 30 seq_num 245) 2024-03-19T11:31:24.035 INFO:tasks.rados.rados.0.smithi038.stdout:1414: done (9 left) 2024-03-19T11:31:24.036 INFO:tasks.rados.rados.0.smithi038.stdout:1415: done (8 left) 2024-03-19T11:31:24.036 INFO:tasks.rados.rados.0.smithi038.stdout:1416: done (7 left) 2024-03-19T11:31:24.036 INFO:tasks.rados.rados.0.smithi038.stdout:1417: done (6 left) 2024-03-19T11:31:24.036 INFO:tasks.rados.rados.0.smithi038.stdout:1418: done (5 left) 2024-03-19T11:31:24.036 INFO:tasks.rados.rados.0.smithi038.stdout:1419: done (4 left) 2024-03-19T11:31:24.036 INFO:tasks.rados.rados.0.smithi038.stdout:1420: done (3 left) 2024-03-19T11:31:24.036 INFO:tasks.rados.rados.0.smithi038.stdout:1421: done (2 left) 2024-03-19T11:31:24.036 INFO:tasks.rados.rados.0.smithi038.stdout:1422: done (1 left) 2024-03-19T11:31:24.036 INFO:tasks.rados.rados.0.smithi038.stdout:1423: read oid 43 snap -1 2024-03-19T11:31:24.036 INFO:tasks.rados.rados.0.smithi038.stdout:1423: expect (ObjNum 237 snap 29 seq_num 237) 2024-03-19T11:31:24.036 INFO:tasks.rados.rados.0.smithi038.stdout:1424: rollback oid 35 current snap is 30 2024-03-19T11:31:24.036 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 35 to 22 2024-03-19T11:31:24.036 INFO:tasks.rados.rados.0.smithi038.stdout:1425: read oid 37 snap -1 2024-03-19T11:31:24.036 INFO:tasks.rados.rados.0.smithi038.stdout:1425: expect (ObjNum 236 snap 28 seq_num 236) 2024-03-19T11:31:24.037 INFO:tasks.rados.rados.0.smithi038.stdout:1426: read oid 10 snap -1 2024-03-19T11:31:24.037 INFO:tasks.rados.rados.0.smithi038.stdout:1426: expect (ObjNum 179 snap 19 seq_num 179) 2024-03-19T11:31:24.037 INFO:tasks.rados.rados.0.smithi038.stdout:1427: write oid 48 current snap is 30 2024-03-19T11:31:24.037 INFO:tasks.rados.rados.0.smithi038.stdout:1427: seq_num 246 ranges {534522=634550,1812335=322187} 2024-03-19T11:31:24.041 INFO:tasks.rados.rados.0.smithi038.stdout:1427: writing smithi03829856-48 from 534522 to 1169072 tid 1 2024-03-19T11:31:24.043 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:24.049+0000 7f6712f92640 -1 osd.2 104 handle_osd_map injecting map crc failure 2024-03-19T11:31:24.047 INFO:tasks.rados.rados.0.smithi038.stdout:1427: writing smithi03829856-48 from 1812335 to 2134522 tid 2 2024-03-19T11:31:24.048 INFO:tasks.rados.rados.0.smithi038.stdout:1428: read oid 38 snap -1 2024-03-19T11:31:24.048 INFO:tasks.rados.rados.0.smithi038.stdout:1428: expect (ObjNum 233 snap 27 seq_num 233) 2024-03-19T11:31:24.048 INFO:tasks.rados.rados.0.smithi038.stdout:1426: expect (ObjNum 179 snap 19 seq_num 179) 2024-03-19T11:31:24.101 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:24.110+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:24.129 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:24.129 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:24.134 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:24.142+0000 7f923da1c640 -1 osd.7 104 handle_osd_map injecting map crc failure 2024-03-19T11:31:24.145 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-19T11:31:24.150 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-03-19T11:31:24.220 INFO:tasks.rados.rados.0.smithi038.stdout:1424: finishing rollback tid 0 to smithi03829856-35 2024-03-19T11:31:24.220 INFO:tasks.rados.rados.0.smithi038.stdout:1424: finishing rollback tid 1 to smithi03829856-35 2024-03-19T11:31:24.220 INFO:tasks.rados.rados.0.smithi038.stdout:1426: done (6 left) 2024-03-19T11:31:24.220 INFO:tasks.rados.rados.0.smithi038.stdout:1429: read oid 18 snap -1 2024-03-19T11:31:24.220 INFO:tasks.rados.rados.0.smithi038.stdout:1429: expect (ObjNum 199 snap 22 seq_num 199) 2024-03-19T11:31:24.220 INFO:tasks.rados.rados.0.smithi038.stdout:1430: delete oid 39 current snap is 30 2024-03-19T11:31:24.220 INFO:tasks.rados.rados.0.smithi038.stdout:1424: finishing rollback tid 2 to smithi03829856-35 2024-03-19T11:31:24.220 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 35 v 229 (ObjNum 200 snap 22 seq_num 200) dirty exists 2024-03-19T11:31:24.220 INFO:tasks.rados.rados.0.smithi038.stdout:1423: expect (ObjNum 237 snap 29 seq_num 237) 2024-03-19T11:31:24.240 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:24.250+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:24.304 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:24.313+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:24.318 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:24.319 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:24.333 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-19T11:31:24.400 INFO:tasks.rados.rados.0.smithi038.stdout:1425: expect (ObjNum 236 snap 28 seq_num 236) 2024-03-19T11:31:24.405 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:24.413+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:24.506 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:24.513+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:24.506 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:24.506 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:24.520 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-19T11:31:24.530 INFO:tasks.rados.rados.0.smithi038.stdout:1427: finishing write tid 1 to smithi03829856-48 2024-03-19T11:31:24.530 INFO:tasks.rados.rados.0.smithi038.stdout:1427: finishing write tid 2 to smithi03829856-48 2024-03-19T11:31:24.530 INFO:tasks.rados.rados.0.smithi038.stdout:1427: finishing write tid 3 to smithi03829856-48 2024-03-19T11:31:24.530 INFO:tasks.rados.rados.0.smithi038.stdout:1427: finishing write tid 4 to smithi03829856-48 2024-03-19T11:31:24.530 INFO:tasks.rados.rados.0.smithi038.stdout:1427: oid 48 updating version 0 to 126 2024-03-19T11:31:24.530 INFO:tasks.rados.rados.0.smithi038.stdout:1427: oid 48 version 126 is already newer than 125 2024-03-19T11:31:24.531 INFO:tasks.rados.rados.0.smithi038.stdout:1427: oid 48 version 126 is already newer than 124 2024-03-19T11:31:24.531 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 48 v 126 (ObjNum 246 snap 30 seq_num 246) dirty exists 2024-03-19T11:31:24.531 INFO:tasks.rados.rados.0.smithi038.stdout:1427: left oid 48 (ObjNum 246 snap 30 seq_num 246) 2024-03-19T11:31:24.531 INFO:tasks.rados.rados.0.smithi038.stdout:1410: finishing write tid 1 to smithi03829856-14 2024-03-19T11:31:24.531 INFO:tasks.rados.rados.0.smithi038.stdout:1410: finishing write tid 2 to smithi03829856-14 2024-03-19T11:31:24.531 INFO:tasks.rados.rados.0.smithi038.stdout:1410: finishing write tid 3 to smithi03829856-14 2024-03-19T11:31:24.531 INFO:tasks.rados.rados.0.smithi038.stdout:1423: done (7 left) 2024-03-19T11:31:24.531 INFO:tasks.rados.rados.0.smithi038.stdout:1424: done (6 left) 2024-03-19T11:31:24.531 INFO:tasks.rados.rados.0.smithi038.stdout:1425: done (5 left) 2024-03-19T11:31:24.531 INFO:tasks.rados.rados.0.smithi038.stdout:1427: done (4 left) 2024-03-19T11:31:24.531 INFO:tasks.rados.rados.0.smithi038.stdout:1430: done (3 left) 2024-03-19T11:31:24.531 INFO:tasks.rados.rados.0.smithi038.stdout:1431: write (excl) oid 33 current snap is 30 2024-03-19T11:31:24.531 INFO:tasks.rados.rados.0.smithi038.stdout:1431: seq_num 248 ranges {568136=785048,1929299=238837} 2024-03-19T11:31:24.533 INFO:tasks.rados.rados.0.smithi038.stdout:1431: writing smithi03829856-33 from 568136 to 1353184 tid 1 2024-03-19T11:31:24.537 INFO:tasks.rados.rados.0.smithi038.stdout:1431: writing smithi03829856-33 from 1929299 to 2168136 tid 2 2024-03-19T11:31:24.537 INFO:tasks.rados.rados.0.smithi038.stdout:1432: set_chunk oid 42 target oid 2024-03-19T11:31:24.538 INFO:tasks.rados.rados.0.smithi038.stdout:1432: set_chunk oid 42 offset: 1615360 length: 295424 target oid smithi03829856- offset: 1615360 snap 28 2024-03-19T11:31:24.538 INFO:tasks.rados.rados.0.smithi038.stdout:1433: read oid 47 snap 24 2024-03-19T11:31:24.538 INFO:tasks.rados.rados.0.smithi038.stdout:1433: expect (ObjNum 46 snap 0 seq_num 46) 2024-03-19T11:31:24.538 INFO:tasks.rados.rados.0.smithi038.stdout:1434: set_chunk oid 23 target oid 2024-03-19T11:31:24.550 INFO:tasks.rados.rados.0.smithi038.stdout:1434: set_chunk oid 23 offset: 1436672 length: 938496 target oid 4e1819d8f82f3ad57d28a25c19013d2fd639e5cf3df32eb7e04e403769bef3db offset: 0 snap 22 2024-03-19T11:31:24.551 INFO:tasks.rados.rados.0.smithi038.stdout:1435: read oid 40 snap -1 2024-03-19T11:31:24.551 INFO:tasks.rados.rados.0.smithi038.stdout:1435: expect (ObjNum 227 snap 26 seq_num 227) 2024-03-19T11:31:24.551 INFO:tasks.rados.rados.0.smithi038.stdout:1436: tier_flush oid 5 2024-03-19T11:31:24.551 INFO:tasks.rados.rados.0.smithi038.stdout:1436: tier_flush oid 5 snap -1 2024-03-19T11:31:24.551 INFO:tasks.rados.rados.0.smithi038.stdout:1410: finishing write tid 4 to smithi03829856-14 2024-03-19T11:31:24.551 INFO:tasks.rados.rados.0.smithi038.stdout:1410: oid 14 updating version 0 to 65 2024-03-19T11:31:24.551 INFO:tasks.rados.rados.0.smithi038.stdout:1410: oid 14 updating version 65 to 66 2024-03-19T11:31:24.551 INFO:tasks.rados.rados.0.smithi038.stdout:1410: oid 14 updating version 66 to 67 2024-03-19T11:31:24.551 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 14 v 67 (ObjNum 241 snap 30 seq_num 241) dirty exists 2024-03-19T11:31:24.551 INFO:tasks.rados.rados.0.smithi038.stdout:1410: left oid 14 (ObjNum 241 snap 30 seq_num 241) 2024-03-19T11:31:24.551 INFO:tasks.rados.rados.0.smithi038.stdout:1428: expect (ObjNum 233 snap 27 seq_num 233) 2024-03-19T11:31:24.605 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:24.614+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:24.639 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:24.640 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:24.655 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-19T11:31:24.706 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:24.713+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:24.764 INFO:tasks.rados.rados.0.smithi038.stdout:1429: expect (ObjNum 199 snap 22 seq_num 199) 2024-03-19T11:31:24.807 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:24.814+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:24.820 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:24.820 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:24.834 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-19T11:31:24.908 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:24.917+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:25.001 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:25.001 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:25.009 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:25.018+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:25.015 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-19T11:31:25.062 INFO:tasks.rados.rados.0.smithi038.stdout:1432: finishing set_chunk to oid 42 2024-03-19T11:31:25.062 INFO:tasks.rados.rados.0.smithi038.stdout:1432: got expected ERANGE (src dne) 2024-03-19T11:31:25.062 INFO:tasks.rados.rados.0.smithi038.stdout:1431: finishing write tid 1 to smithi03829856-33 2024-03-19T11:31:25.062 INFO:tasks.rados.rados.0.smithi038.stdout:1431: finishing write tid 2 to smithi03829856-33 2024-03-19T11:31:25.063 INFO:tasks.rados.rados.0.smithi038.stdout:1431: finishing write tid 3 to smithi03829856-33 2024-03-19T11:31:25.063 INFO:tasks.rados.rados.0.smithi038.stdout:1433: expect (ObjNum 46 snap 0 seq_num 46) 2024-03-19T11:31:25.110 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:25.117+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:25.150 INFO:tasks.rados.rados.0.smithi038.stdout:1431: finishing write tid 4 to smithi03829856-33 2024-03-19T11:31:25.150 INFO:tasks.rados.rados.0.smithi038.stdout:1431: oid 33 updating version 0 to 155 2024-03-19T11:31:25.150 INFO:tasks.rados.rados.0.smithi038.stdout:1431: oid 33 version 155 is already newer than 154 2024-03-19T11:31:25.150 INFO:tasks.rados.rados.0.smithi038.stdout:1431: oid 33 updating version 155 to 156 2024-03-19T11:31:25.150 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 33 v 156 (ObjNum 248 snap 30 seq_num 248) dirty exists 2024-03-19T11:31:25.150 INFO:tasks.rados.rados.0.smithi038.stdout:1431: left oid 33 (ObjNum 248 snap 30 seq_num 248) 2024-03-19T11:31:25.150 INFO:tasks.rados.rados.0.smithi038.stdout:1434: finishing set_chunk to oid 23 2024-03-19T11:31:25.150 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 23 set_chunk returned error code -95 offset: 1436672 length: 938496 2024-03-19T11:31:25.150 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 23 v 0 (ObjNum 239 snap 30 seq_num 239) dirty exists 2024-03-19T11:31:25.150 INFO:tasks.rados.rados.0.smithi038.stdout:1436: got (16) Device or resource busy 2024-03-19T11:31:25.150 INFO:tasks.rados.rados.0.smithi038.stdout:check_oldest_snap_flushed oid 5 v 136 (ObjNum 222 snap 24 seq_num 222) unflushed 24 2024-03-19T11:31:25.150 INFO:tasks.rados.rados.0.smithi038.stdout:1435: expect (ObjNum 227 snap 26 seq_num 227) 2024-03-19T11:31:25.162 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-03-19T11:31:25.190 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:25.190 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:25.205 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-19T11:31:25.224 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:25.233+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:25.312 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:25.321+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:25.326 INFO:tasks.rados.rados.0.smithi038.stdout:1410: done (8 left) 2024-03-19T11:31:25.327 INFO:tasks.rados.rados.0.smithi038.stdout:1428: done (7 left) 2024-03-19T11:31:25.327 INFO:tasks.rados.rados.0.smithi038.stdout:1429: done (6 left) 2024-03-19T11:31:25.327 INFO:tasks.rados.rados.0.smithi038.stdout:1431: done (5 left) 2024-03-19T11:31:25.327 INFO:tasks.rados.rados.0.smithi038.stdout:1432: done (4 left) 2024-03-19T11:31:25.327 INFO:tasks.rados.rados.0.smithi038.stdout:1433: done (3 left) 2024-03-19T11:31:25.327 INFO:tasks.rados.rados.0.smithi038.stdout:1434: done (2 left) 2024-03-19T11:31:25.327 INFO:tasks.rados.rados.0.smithi038.stdout:1435: done (1 left) 2024-03-19T11:31:25.327 INFO:tasks.rados.rados.0.smithi038.stdout:1436: done (0 left) 2024-03-19T11:31:25.327 INFO:tasks.rados.rados.0.smithi038.stdout:1437: set_chunk oid 48 target oid 2024-03-19T11:31:25.328 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:25.328 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:25.336 INFO:tasks.rados.rados.0.smithi038.stdout:1437: set_chunk oid 48 offset: 1169920 length: 444928 target oid c7e95536e82810194a342db3877849a7a24baf5c721242874e0f5e7c4db464b9 offset: 0 snap -1 2024-03-19T11:31:25.336 INFO:tasks.rados.rados.0.smithi038.stdout:1438: rollback oid 29 current snap is 30 2024-03-19T11:31:25.336 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 29 to 28 2024-03-19T11:31:25.336 INFO:tasks.rados.rados.0.smithi038.stdout:1439: delete oid 39 current snap is 30 2024-03-19T11:31:25.344 INFO:tasks.rados.rados.0.smithi038.stdout:1439: done (2 left) 2024-03-19T11:31:25.344 INFO:tasks.rados.rados.0.smithi038.stdout:1440: read oid 33 snap 21 2024-03-19T11:31:25.344 INFO:tasks.rados.rados.0.smithi038.stdout:1440: expect (ObjNum 32 snap 0 seq_num 32) 2024-03-19T11:31:25.344 INFO:tasks.rados.rados.0.smithi038.stdout:1441: read oid 45 snap -1 2024-03-19T11:31:25.344 INFO:tasks.rados.rados.0.smithi038.stdout:1441: expect (ObjNum 212 snap 22 seq_num 212) 2024-03-19T11:31:25.344 INFO:tasks.rados.rados.0.smithi038.stdout:1442: read oid 34 snap 26 2024-03-19T11:31:25.344 INFO:tasks.rados.rados.0.smithi038.stdout:1442: expect (ObjNum 72 snap 5 seq_num 72) 2024-03-19T11:31:25.344 INFO:tasks.rados.rados.0.smithi038.stdout:1443: read oid 42 snap 29 2024-03-19T11:31:25.344 INFO:tasks.rados.rados.0.smithi038.stdout:1443: expect deleted 2024-03-19T11:31:25.344 INFO:tasks.rados.rados.0.smithi038.stdout:1444: snap_create 2024-03-19T11:31:25.346 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-19T11:31:25.350 INFO:tasks.rados.rados.0.smithi038.stdout:1437: finishing set_chunk to oid 48 2024-03-19T11:31:25.350 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 48 v 126 (ObjNum 246 snap 30 seq_num 246) dirty exists 2024-03-19T11:31:25.351 INFO:tasks.rados.rados.0.smithi038.stdout:1438: finishing rollback tid 0 to smithi03829856-29 2024-03-19T11:31:25.352 INFO:tasks.rados.rados.0.smithi038.stdout:1440: expect (ObjNum 32 snap 0 seq_num 32) 2024-03-19T11:31:25.413 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:25.422+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:25.435 INFO:tasks.rados.rados.0.smithi038.stdout:1438: finishing rollback tid 1 to smithi03829856-29 2024-03-19T11:31:25.435 INFO:tasks.rados.rados.0.smithi038.stdout:1438: finishing rollback tid 2 to smithi03829856-29 2024-03-19T11:31:25.435 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 29 v 97 (ObjNum 219 snap 23 seq_num 219) dirty exists 2024-03-19T11:31:25.435 INFO:tasks.rados.rados.0.smithi038.stdout:1441: expect (ObjNum 212 snap 22 seq_num 212) 2024-03-19T11:31:25.470 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:25.471 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:25.484 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-19T11:31:25.514 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:25.521+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:25.615 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:25.621+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:25.649 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:25.650 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:25.665 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-19T11:31:25.668 INFO:tasks.rados.rados.0.smithi038.stdout:1442: expect (ObjNum 72 snap 5 seq_num 72) 2024-03-19T11:31:25.715 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:25.721+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:25.779 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:25.780 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:25.792 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-19T11:31:25.836 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:25.845+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:25.917 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:25.925+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:25.958 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:25.958 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:25.971 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-19T11:31:26.004 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-19T11:31:26.018 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:26.026+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:26.119 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:26.125+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:26.129 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:26.129 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:26.142 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-19T11:31:26.149 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-19T11:31:26.149 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-19T11:31:26.149 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-19T11:31:26.149 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-03-19T11:31:26.169 INFO:tasks.rados.rados.0.smithi038.stdout:1437: done (6 left) 2024-03-19T11:31:26.170 INFO:tasks.rados.rados.0.smithi038.stdout:1438: done (5 left) 2024-03-19T11:31:26.170 INFO:tasks.rados.rados.0.smithi038.stdout:1440: done (4 left) 2024-03-19T11:31:26.170 INFO:tasks.rados.rados.0.smithi038.stdout:1441: done (3 left) 2024-03-19T11:31:26.170 INFO:tasks.rados.rados.0.smithi038.stdout:1442: done (2 left) 2024-03-19T11:31:26.170 INFO:tasks.rados.rados.0.smithi038.stdout:1443: done (1 left) 2024-03-19T11:31:26.170 INFO:tasks.rados.rados.0.smithi038.stdout:1444: done (0 left) 2024-03-19T11:31:26.170 INFO:tasks.rados.rados.0.smithi038.stdout:1445: snap_remove snap 9 2024-03-19T11:31:26.171 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:31:26.171 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:31:26.190 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:31:26.190 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:31:26.206 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:31:26.207 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:31:26.220 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:26.229+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:26.222 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:31:26.222 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:31:26.242 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:31:26.242 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:31:26.263 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:31:26.263 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:31:26.284 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:26.284 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:31:26.284 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:31:26.284 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:26.284 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:31:26.285 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:31:26.300 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-19T11:31:26.305 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:31:26.306 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:31:26.321 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:26.330+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:26.413 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:26.413 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:31:26.413 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:31:26.413 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:31:26.413 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:26.422 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:26.429+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:26.426 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-19T11:31:26.458 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:31:26.458 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:26.459 INFO:teuthology.orchestra.run.smithi038.stdout: "epoch": 107, 2024-03-19T11:31:26.459 INFO:teuthology.orchestra.run.smithi038.stdout: "fsid": "246bcf68-c431-4aa7-b787-5e9b2d9519d6", 2024-03-19T11:31:26.459 INFO:teuthology.orchestra.run.smithi038.stdout: "created": "2024-03-19T11:29:33.327073+0000", 2024-03-19T11:31:26.459 INFO:teuthology.orchestra.run.smithi038.stdout: "modified": "2024-03-19T11:31:26.167148+0000", 2024-03-19T11:31:26.459 INFO:teuthology.orchestra.run.smithi038.stdout: "last_up_change": "2024-03-19T11:29:43.166618+0000", 2024-03-19T11:31:26.459 INFO:teuthology.orchestra.run.smithi038.stdout: "last_in_change": "2024-03-19T11:29:35.632327+0000", 2024-03-19T11:31:26.459 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-19T11:31:26.459 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_num": 5806080, 2024-03-19T11:31:26.459 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_set": [ 2024-03-19T11:31:26.459 INFO:teuthology.orchestra.run.smithi038.stdout: "nodeep-scrub", 2024-03-19T11:31:26.459 INFO:teuthology.orchestra.run.smithi038.stdout: "noscrub", 2024-03-19T11:31:26.459 INFO:teuthology.orchestra.run.smithi038.stdout: "pglog_hardlimit", 2024-03-19T11:31:26.459 INFO:teuthology.orchestra.run.smithi038.stdout: "purged_snapdirs", 2024-03-19T11:31:26.460 INFO:teuthology.orchestra.run.smithi038.stdout: "recovery_deletes", 2024-03-19T11:31:26.460 INFO:teuthology.orchestra.run.smithi038.stdout: "sortbitwise" 2024-03-19T11:31:26.460 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-19T11:31:26.460 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_version": 5, 2024-03-19T11:31:26.460 INFO:teuthology.orchestra.run.smithi038.stdout: "full_ratio": 0.94999998807907104, 2024-03-19T11:31:26.460 INFO:teuthology.orchestra.run.smithi038.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-19T11:31:26.460 INFO:teuthology.orchestra.run.smithi038.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-19T11:31:26.460 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_snapshot": "", 2024-03-19T11:31:26.460 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_max": 4, 2024-03-19T11:31:26.460 INFO:teuthology.orchestra.run.smithi038.stdout: "max_osd": 8, 2024-03-19T11:31:26.460 INFO:teuthology.orchestra.run.smithi038.stdout: "require_min_compat_client": "reef", 2024-03-19T11:31:26.460 INFO:teuthology.orchestra.run.smithi038.stdout: "min_compat_client": "luminous", 2024-03-19T11:31:26.460 INFO:teuthology.orchestra.run.smithi038.stdout: "require_osd_release": "squid", 2024-03-19T11:31:26.460 INFO:teuthology.orchestra.run.smithi038.stdout: "allow_crimson": false, 2024-03-19T11:31:26.460 INFO:teuthology.orchestra.run.smithi038.stdout: "pools": [ 2024-03-19T11:31:26.461 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.461 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 1, 2024-03-19T11:31:26.461 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": ".mgr", 2024-03-19T11:31:26.461 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-19T11:29:42.219654+0000", 2024-03-19T11:31:26.461 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-19T11:31:26.461 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-03-19T11:31:26.461 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-19T11:31:26.461 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 2, 2024-03-19T11:31:26.461 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-19T11:31:26.461 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-19T11:31:26.461 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-19T11:31:26.461 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-19T11:31:26.461 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-19T11:31:26.461 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-19T11:31:26.462 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-19T11:31:26.462 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-19T11:31:26.462 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 1, 2024-03-19T11:31:26.462 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 1, 2024-03-19T11:31:26.462 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 1, 2024-03-19T11:31:26.462 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 1, 2024-03-19T11:31:26.462 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 1, 2024-03-19T11:31:26.462 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-19T11:31:26.462 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-19T11:31:26.462 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-19T11:31:26.462 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-19T11:31:26.462 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-19T11:31:26.462 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-19T11:31:26.462 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-19T11:31:26.463 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.463 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "16", 2024-03-19T11:31:26.463 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-19T11:31:26.463 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-19T11:31:26.463 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-19T11:31:26.463 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-19T11:31:26.463 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-19T11:31:26.463 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-03-19T11:31:26.463 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-03-19T11:31:26.463 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-19T11:31:26.463 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-19T11:31:26.463 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-19T11:31:26.463 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-19T11:31:26.463 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-19T11:31:26.464 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-19T11:31:26.464 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-19T11:31:26.464 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-19T11:31:26.464 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-19T11:31:26.464 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-19T11:31:26.464 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-19T11:31:26.464 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-19T11:31:26.464 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-19T11:31:26.464 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-19T11:31:26.464 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-19T11:31:26.464 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-19T11:31:26.464 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-19T11:31:26.464 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-19T11:31:26.464 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-19T11:31:26.464 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.465 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-19T11:31:26.465 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-19T11:31:26.465 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-19T11:31:26.465 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-19T11:31:26.465 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-19T11:31:26.465 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-19T11:31:26.465 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-19T11:31:26.465 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-19T11:31:26.465 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-19T11:31:26.465 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-19T11:31:26.465 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-19T11:31:26.465 INFO:teuthology.orchestra.run.smithi038.stdout: "options": { 2024-03-19T11:31:26.465 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_max": 32, 2024-03-19T11:31:26.465 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_min": 1 2024-03-19T11:31:26.465 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.466 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-19T11:31:26.466 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr": {} 2024-03-19T11:31:26.466 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.466 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-19T11:31:26.466 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-19T11:31:26.466 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 8, 2024-03-19T11:31:26.466 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 8, 2024-03-19T11:31:26.466 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.25, 2024-03-19T11:31:26.466 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 2, 2024-03-19T11:31:26.466 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 2, 2024-03-19T11:31:26.466 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 1, 2024-03-19T11:31:26.466 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 1, 2024-03-19T11:31:26.466 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 1 2024-03-19T11:31:26.466 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.467 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.467 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.467 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 2, 2024-03-19T11:31:26.467 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": "rbd", 2024-03-19T11:31:26.467 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-19T11:29:44.428406+0000", 2024-03-19T11:31:26.467 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-19T11:31:26.467 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-03-19T11:31:26.467 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-19T11:31:26.467 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 2, 2024-03-19T11:31:26.467 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-19T11:31:26.467 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-19T11:31:26.467 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-19T11:31:26.467 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-19T11:31:26.467 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-19T11:31:26.467 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-19T11:31:26.468 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-19T11:31:26.468 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-19T11:31:26.468 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 8, 2024-03-19T11:31:26.468 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 8, 2024-03-19T11:31:26.468 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 8, 2024-03-19T11:31:26.468 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 8, 2024-03-19T11:31:26.468 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 8, 2024-03-19T11:31:26.468 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-19T11:31:26.468 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-19T11:31:26.468 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-19T11:31:26.468 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-19T11:31:26.468 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-19T11:31:26.468 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-19T11:31:26.468 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-19T11:31:26.468 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.469 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "18", 2024-03-19T11:31:26.469 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-19T11:31:26.469 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-19T11:31:26.469 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-19T11:31:26.469 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-19T11:31:26.469 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-19T11:31:26.469 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-03-19T11:31:26.469 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-03-19T11:31:26.469 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-19T11:31:26.469 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-19T11:31:26.469 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-19T11:31:26.469 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-19T11:31:26.469 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-19T11:31:26.469 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-19T11:31:26.469 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-19T11:31:26.470 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-19T11:31:26.470 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-19T11:31:26.470 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-19T11:31:26.470 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-19T11:31:26.470 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-19T11:31:26.470 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-19T11:31:26.470 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-19T11:31:26.470 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-19T11:31:26.470 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-19T11:31:26.470 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-19T11:31:26.470 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-19T11:31:26.470 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-19T11:31:26.470 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.470 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-19T11:31:26.471 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-19T11:31:26.471 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-19T11:31:26.471 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-19T11:31:26.471 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-19T11:31:26.471 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-19T11:31:26.471 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-19T11:31:26.471 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-19T11:31:26.471 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-19T11:31:26.471 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-19T11:31:26.471 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-19T11:31:26.471 INFO:teuthology.orchestra.run.smithi038.stdout: "options": {}, 2024-03-19T11:31:26.471 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-19T11:31:26.471 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd": {} 2024-03-19T11:31:26.471 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.472 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-19T11:31:26.472 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-19T11:31:26.472 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 1.9900000095367432, 2024-03-19T11:31:26.472 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 1.9900000095367432, 2024-03-19T11:31:26.472 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.87999999523162842, 2024-03-19T11:31:26.472 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 1.75, 2024-03-19T11:31:26.472 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 1.75, 2024-03-19T11:31:26.472 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 1, 2024-03-19T11:31:26.472 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 1, 2024-03-19T11:31:26.472 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 1 2024-03-19T11:31:26.472 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.472 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.472 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.472 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 3, 2024-03-19T11:31:26.472 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": "low_tier", 2024-03-19T11:31:26.472 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-19T11:29:55.608699+0000", 2024-03-19T11:31:26.473 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-19T11:31:26.473 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-03-19T11:31:26.473 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-19T11:31:26.473 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 2, 2024-03-19T11:31:26.473 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-19T11:31:26.473 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-19T11:31:26.473 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-19T11:31:26.473 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-19T11:31:26.473 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-19T11:31:26.473 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-19T11:31:26.473 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-19T11:31:26.473 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-19T11:31:26.473 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 4, 2024-03-19T11:31:26.473 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 4, 2024-03-19T11:31:26.474 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 4, 2024-03-19T11:31:26.474 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 4, 2024-03-19T11:31:26.474 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 4, 2024-03-19T11:31:26.474 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-19T11:31:26.474 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-19T11:31:26.474 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-19T11:31:26.474 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-19T11:31:26.474 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-19T11:31:26.474 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-19T11:31:26.474 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-19T11:31:26.474 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.474 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "22", 2024-03-19T11:31:26.474 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-19T11:31:26.475 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-19T11:31:26.475 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-19T11:31:26.475 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-19T11:31:26.475 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-19T11:31:26.475 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-03-19T11:31:26.475 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-03-19T11:31:26.475 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-19T11:31:26.475 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-19T11:31:26.475 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-19T11:31:26.475 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-19T11:31:26.475 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-19T11:31:26.475 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-19T11:31:26.475 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-19T11:31:26.475 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-19T11:31:26.476 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-19T11:31:26.476 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-19T11:31:26.476 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-19T11:31:26.476 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-19T11:31:26.476 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-19T11:31:26.476 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-19T11:31:26.476 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-19T11:31:26.476 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-19T11:31:26.476 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-19T11:31:26.476 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-19T11:31:26.476 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-19T11:31:26.476 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.476 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-19T11:31:26.476 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-19T11:31:26.476 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-19T11:31:26.477 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-19T11:31:26.477 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-19T11:31:26.477 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-19T11:31:26.477 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-19T11:31:26.477 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-19T11:31:26.477 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-19T11:31:26.477 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-19T11:31:26.477 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-19T11:31:26.477 INFO:teuthology.orchestra.run.smithi038.stdout: "options": {}, 2024-03-19T11:31:26.477 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": {}, 2024-03-19T11:31:26.477 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-19T11:31:26.477 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-19T11:31:26.477 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 2, 2024-03-19T11:31:26.477 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 2, 2024-03-19T11:31:26.478 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.75, 2024-03-19T11:31:26.478 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 1.5, 2024-03-19T11:31:26.478 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 1.5, 2024-03-19T11:31:26.478 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 1, 2024-03-19T11:31:26.478 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 1, 2024-03-19T11:31:26.478 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 1 2024-03-19T11:31:26.478 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.478 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.478 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.478 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 4, 2024-03-19T11:31:26.478 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": "unique_pool_0", 2024-03-19T11:31:26.478 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-19T11:29:56.945423+0000", 2024-03-19T11:31:26.478 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 8193, 2024-03-19T11:31:26.479 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-19T11:31:26.479 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-19T11:31:26.479 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 2, 2024-03-19T11:31:26.479 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-19T11:31:26.479 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-19T11:31:26.479 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-19T11:31:26.479 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-19T11:31:26.479 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-19T11:31:26.479 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-19T11:31:26.479 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-19T11:31:26.479 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-19T11:31:26.479 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 39, 2024-03-19T11:31:26.479 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 37, 2024-03-19T11:31:26.479 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 26, 2024-03-19T11:31:26.479 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 26, 2024-03-19T11:31:26.480 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 38, 2024-03-19T11:31:26.480 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-19T11:31:26.480 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "4.27", 2024-03-19T11:31:26.480 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 104, 2024-03-19T11:31:26.480 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 103, 2024-03-19T11:31:26.480 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 103, 2024-03-19T11:31:26.480 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "99'63", 2024-03-19T11:31:26.480 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "61'40" 2024-03-19T11:31:26.480 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.480 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "107", 2024-03-19T11:31:26.480 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-19T11:31:26.480 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "106", 2024-03-19T11:31:26.480 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "106", 2024-03-19T11:31:26.481 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-19T11:31:26.481 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-19T11:31:26.481 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 32, 2024-03-19T11:31:26.481 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 107, 2024-03-19T11:31:26.481 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-19T11:31:26.481 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-19T11:31:26.481 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-19T11:31:26.481 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-19T11:31:26.481 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-19T11:31:26.481 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-19T11:31:26.481 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-19T11:31:26.481 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-19T11:31:26.481 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-19T11:31:26.481 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-19T11:31:26.481 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-19T11:31:26.482 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-19T11:31:26.482 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-19T11:31:26.482 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-19T11:31:26.482 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-19T11:31:26.482 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-19T11:31:26.482 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-19T11:31:26.482 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-19T11:31:26.482 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-19T11:31:26.482 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.482 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-19T11:31:26.483 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-19T11:31:26.483 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-19T11:31:26.483 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-19T11:31:26.483 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-19T11:31:26.483 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-19T11:31:26.483 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-19T11:31:26.483 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-19T11:31:26.483 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-19T11:31:26.483 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-19T11:31:26.483 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-19T11:31:26.483 INFO:teuthology.orchestra.run.smithi038.stdout: "options": { 2024-03-19T11:31:26.483 INFO:teuthology.orchestra.run.smithi038.stdout: "dedup_cdc_chunk_size": 131072, 2024-03-19T11:31:26.483 INFO:teuthology.orchestra.run.smithi038.stdout: "dedup_chunk_algorithm": "fastcdc", 2024-03-19T11:31:26.483 INFO:teuthology.orchestra.run.smithi038.stdout: "dedup_tier": 3, 2024-03-19T11:31:26.484 INFO:teuthology.orchestra.run.smithi038.stdout: "fingerprint_algorithm": "sha256" 2024-03-19T11:31:26.484 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.484 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-19T11:31:26.484 INFO:teuthology.orchestra.run.smithi038.stdout: "rados": {} 2024-03-19T11:31:26.484 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.484 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-19T11:31:26.484 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-19T11:31:26.484 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 2.0499999523162842, 2024-03-19T11:31:26.484 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 2.0499999523162842, 2024-03-19T11:31:26.484 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 1, 2024-03-19T11:31:26.484 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 2.0499999523162842, 2024-03-19T11:31:26.484 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 2.0499999523162842, 2024-03-19T11:31:26.484 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 1, 2024-03-19T11:31:26.484 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 1, 2024-03-19T11:31:26.484 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 1 2024-03-19T11:31:26.485 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.485 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.485 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-19T11:31:26.485 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-03-19T11:31:26.485 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.485 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 0, 2024-03-19T11:31:26.485 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "3bb2d2be-5253-4da0-acf3-85a114dc3801", 2024-03-19T11:31:26.485 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-19T11:31:26.485 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-19T11:31:26.485 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-19T11:31:26.485 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-19T11:31:26.486 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-19T11:31:26.487 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-19T11:31:26.487 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 13, 2024-03-19T11:31:26.487 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 106, 2024-03-19T11:31:26.487 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-19T11:31:26.487 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-19T11:31:26.487 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-19T11:31:26.487 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:26.487 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.487 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:26.487 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6808", 2024-03-19T11:31:26.487 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3148574891 2024-03-19T11:31:26.487 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.487 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.488 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:26.488 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6811", 2024-03-19T11:31:26.488 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3148574891 2024-03-19T11:31:26.488 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.488 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.488 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.488 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-19T11:31:26.488 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:26.488 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.488 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:26.488 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6813", 2024-03-19T11:31:26.488 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3148574891 2024-03-19T11:31:26.488 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.488 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.488 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:26.489 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6815", 2024-03-19T11:31:26.489 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3148574891 2024-03-19T11:31:26.489 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.489 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.489 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.489 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-19T11:31:26.489 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:26.489 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.489 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:26.489 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6822", 2024-03-19T11:31:26.489 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3148574891 2024-03-19T11:31:26.489 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.489 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.489 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:26.489 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6823", 2024-03-19T11:31:26.490 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3148574891 2024-03-19T11:31:26.490 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.490 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.490 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.490 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-19T11:31:26.490 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:26.490 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.490 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:26.490 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6818", 2024-03-19T11:31:26.490 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3148574891 2024-03-19T11:31:26.490 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.490 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.490 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:26.490 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6821", 2024-03-19T11:31:26.490 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3148574891 2024-03-19T11:31:26.491 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.491 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.491 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.491 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6811/3148574891", 2024-03-19T11:31:26.491 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6815/3148574891", 2024-03-19T11:31:26.491 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6823/3148574891", 2024-03-19T11:31:26.491 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6821/3148574891", 2024-03-19T11:31:26.491 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-19T11:31:26.491 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-19T11:31:26.491 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-19T11:31:26.491 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.492 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.492 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.492 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 1, 2024-03-19T11:31:26.492 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "e1c926b2-e0be-4ccc-b8a5-d80d0b584ead", 2024-03-19T11:31:26.492 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-19T11:31:26.492 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-19T11:31:26.492 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-19T11:31:26.492 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-19T11:31:26.492 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-19T11:31:26.492 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-19T11:31:26.492 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 14, 2024-03-19T11:31:26.492 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 104, 2024-03-19T11:31:26.492 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-19T11:31:26.492 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-19T11:31:26.493 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-19T11:31:26.493 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:26.493 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.493 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:26.493 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6824", 2024-03-19T11:31:26.493 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 675027622 2024-03-19T11:31:26.493 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.493 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.493 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:26.493 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6825", 2024-03-19T11:31:26.493 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 675027622 2024-03-19T11:31:26.493 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.493 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.493 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.493 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-19T11:31:26.494 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:26.494 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.494 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:26.494 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6826", 2024-03-19T11:31:26.494 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 675027622 2024-03-19T11:31:26.494 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.494 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.494 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:26.494 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6827", 2024-03-19T11:31:26.494 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 675027622 2024-03-19T11:31:26.494 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.494 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.494 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.495 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-19T11:31:26.495 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:26.495 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.495 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:26.495 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6830", 2024-03-19T11:31:26.495 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 675027622 2024-03-19T11:31:26.495 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.495 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.495 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:26.495 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6831", 2024-03-19T11:31:26.495 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 675027622 2024-03-19T11:31:26.495 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.495 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.495 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.495 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-19T11:31:26.496 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:26.496 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.496 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:26.496 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6828", 2024-03-19T11:31:26.496 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 675027622 2024-03-19T11:31:26.496 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.496 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.496 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:26.496 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6829", 2024-03-19T11:31:26.496 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 675027622 2024-03-19T11:31:26.496 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.496 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.496 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.496 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6825/675027622", 2024-03-19T11:31:26.496 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6827/675027622", 2024-03-19T11:31:26.497 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6831/675027622", 2024-03-19T11:31:26.497 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6829/675027622", 2024-03-19T11:31:26.497 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-19T11:31:26.497 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-19T11:31:26.497 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-19T11:31:26.497 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.497 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.497 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.497 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 2, 2024-03-19T11:31:26.497 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "61262219-4f8a-4733-af9d-b5208a17c0ea", 2024-03-19T11:31:26.497 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-19T11:31:26.498 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-19T11:31:26.498 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-19T11:31:26.498 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-19T11:31:26.498 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-19T11:31:26.498 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-19T11:31:26.498 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 14, 2024-03-19T11:31:26.498 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 82, 2024-03-19T11:31:26.498 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-19T11:31:26.498 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-19T11:31:26.498 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-19T11:31:26.498 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:26.498 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.498 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:26.498 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6800", 2024-03-19T11:31:26.499 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 58279616 2024-03-19T11:31:26.499 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.499 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.499 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:26.499 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6801", 2024-03-19T11:31:26.499 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 58279616 2024-03-19T11:31:26.499 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.499 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.499 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.499 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-19T11:31:26.499 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:26.499 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.499 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:26.499 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6802", 2024-03-19T11:31:26.500 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 58279616 2024-03-19T11:31:26.500 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.500 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.500 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:26.500 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6804", 2024-03-19T11:31:26.500 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 58279616 2024-03-19T11:31:26.500 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.500 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.500 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.500 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-19T11:31:26.500 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:26.500 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.500 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:26.500 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6814", 2024-03-19T11:31:26.500 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 58279616 2024-03-19T11:31:26.501 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.501 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.501 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:26.501 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6817", 2024-03-19T11:31:26.501 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 58279616 2024-03-19T11:31:26.501 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.501 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.501 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.501 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-19T11:31:26.501 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:26.501 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.501 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:26.501 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6807", 2024-03-19T11:31:26.501 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 58279616 2024-03-19T11:31:26.502 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.502 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.502 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:26.502 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6810", 2024-03-19T11:31:26.502 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 58279616 2024-03-19T11:31:26.502 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.502 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.502 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.502 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6801/58279616", 2024-03-19T11:31:26.502 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6804/58279616", 2024-03-19T11:31:26.502 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6817/58279616", 2024-03-19T11:31:26.502 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6810/58279616", 2024-03-19T11:31:26.502 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-19T11:31:26.502 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-19T11:31:26.502 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-19T11:31:26.503 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.503 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.503 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.503 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 3, 2024-03-19T11:31:26.503 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "a31f8d1b-4e38-4ca3-b251-0877002d7235", 2024-03-19T11:31:26.503 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-19T11:31:26.503 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-19T11:31:26.503 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-19T11:31:26.503 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-19T11:31:26.503 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-19T11:31:26.503 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-19T11:31:26.503 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 14, 2024-03-19T11:31:26.503 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 82, 2024-03-19T11:31:26.503 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-19T11:31:26.504 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-19T11:31:26.504 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-19T11:31:26.504 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:26.504 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.504 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:26.504 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6803", 2024-03-19T11:31:26.504 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2942163952 2024-03-19T11:31:26.504 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.504 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.504 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:26.504 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6805", 2024-03-19T11:31:26.505 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2942163952 2024-03-19T11:31:26.505 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.505 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.505 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.505 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-19T11:31:26.505 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:26.505 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.505 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:26.505 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6806", 2024-03-19T11:31:26.505 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2942163952 2024-03-19T11:31:26.505 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.505 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.505 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:26.505 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6809", 2024-03-19T11:31:26.505 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2942163952 2024-03-19T11:31:26.506 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.506 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.506 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.506 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-19T11:31:26.506 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:26.506 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.506 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:26.506 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6819", 2024-03-19T11:31:26.506 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2942163952 2024-03-19T11:31:26.506 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.506 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.506 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:26.506 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6820", 2024-03-19T11:31:26.506 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2942163952 2024-03-19T11:31:26.507 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.507 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.507 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.507 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-19T11:31:26.507 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:26.507 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.507 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:26.507 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6812", 2024-03-19T11:31:26.507 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2942163952 2024-03-19T11:31:26.507 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.507 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.507 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:26.507 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6816", 2024-03-19T11:31:26.507 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2942163952 2024-03-19T11:31:26.507 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.508 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.508 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.508 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6805/2942163952", 2024-03-19T11:31:26.508 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6809/2942163952", 2024-03-19T11:31:26.508 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6820/2942163952", 2024-03-19T11:31:26.508 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6816/2942163952", 2024-03-19T11:31:26.508 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-19T11:31:26.508 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-19T11:31:26.508 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-19T11:31:26.508 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.508 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.508 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.508 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 4, 2024-03-19T11:31:26.508 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "6746723d-f59c-40ba-857e-d8c954d2675f", 2024-03-19T11:31:26.508 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-19T11:31:26.509 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-19T11:31:26.509 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-19T11:31:26.509 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-19T11:31:26.509 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-19T11:31:26.509 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-19T11:31:26.509 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 13, 2024-03-19T11:31:26.509 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 96, 2024-03-19T11:31:26.509 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-19T11:31:26.509 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-19T11:31:26.509 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-19T11:31:26.509 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:26.509 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.509 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:26.509 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6808", 2024-03-19T11:31:26.509 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1353919713 2024-03-19T11:31:26.510 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.510 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.510 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:26.510 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6810", 2024-03-19T11:31:26.510 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1353919713 2024-03-19T11:31:26.510 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.510 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.510 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.510 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-19T11:31:26.510 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:26.510 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.510 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:26.510 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6812", 2024-03-19T11:31:26.510 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1353919713 2024-03-19T11:31:26.511 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.511 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.511 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:26.511 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6814", 2024-03-19T11:31:26.511 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1353919713 2024-03-19T11:31:26.511 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.511 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.511 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.511 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-19T11:31:26.511 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:26.511 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.511 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:26.511 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6819", 2024-03-19T11:31:26.511 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1353919713 2024-03-19T11:31:26.512 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.512 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.512 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:26.512 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6821", 2024-03-19T11:31:26.512 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1353919713 2024-03-19T11:31:26.512 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.512 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.512 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.512 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-19T11:31:26.512 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:26.512 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.512 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:26.512 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6816", 2024-03-19T11:31:26.512 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1353919713 2024-03-19T11:31:26.513 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.513 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.513 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:26.513 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6818", 2024-03-19T11:31:26.513 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1353919713 2024-03-19T11:31:26.513 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.513 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.513 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.513 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.172:6810/1353919713", 2024-03-19T11:31:26.513 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.172:6814/1353919713", 2024-03-19T11:31:26.513 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.172:6821/1353919713", 2024-03-19T11:31:26.513 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.172:6818/1353919713", 2024-03-19T11:31:26.513 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-19T11:31:26.513 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-19T11:31:26.514 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-19T11:31:26.514 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.514 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.514 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.514 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 5, 2024-03-19T11:31:26.514 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "f59afdf8-2277-4583-af68-78136f1c5d62", 2024-03-19T11:31:26.514 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-19T11:31:26.514 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-19T11:31:26.514 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-19T11:31:26.514 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-19T11:31:26.514 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-19T11:31:26.514 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-19T11:31:26.514 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 14, 2024-03-19T11:31:26.514 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 100, 2024-03-19T11:31:26.515 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-19T11:31:26.515 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-19T11:31:26.515 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-19T11:31:26.515 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:26.515 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.515 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:26.515 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6809", 2024-03-19T11:31:26.515 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2733060387 2024-03-19T11:31:26.515 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.515 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.515 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:26.515 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6811", 2024-03-19T11:31:26.515 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2733060387 2024-03-19T11:31:26.515 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.515 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.516 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.516 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-19T11:31:26.516 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:26.516 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.516 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:26.516 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6813", 2024-03-19T11:31:26.516 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2733060387 2024-03-19T11:31:26.516 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.516 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.516 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:26.516 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6815", 2024-03-19T11:31:26.516 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2733060387 2024-03-19T11:31:26.516 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.517 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.517 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.517 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-19T11:31:26.517 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:26.517 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.517 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:26.517 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6822", 2024-03-19T11:31:26.517 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2733060387 2024-03-19T11:31:26.517 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.517 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.517 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:26.517 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6823", 2024-03-19T11:31:26.517 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2733060387 2024-03-19T11:31:26.517 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.517 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.518 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.518 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-19T11:31:26.518 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:26.518 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.518 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:26.518 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6817", 2024-03-19T11:31:26.518 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2733060387 2024-03-19T11:31:26.518 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.518 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.518 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:26.518 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6820", 2024-03-19T11:31:26.518 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2733060387 2024-03-19T11:31:26.518 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.518 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.519 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.519 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.172:6811/2733060387", 2024-03-19T11:31:26.519 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.172:6815/2733060387", 2024-03-19T11:31:26.519 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.172:6823/2733060387", 2024-03-19T11:31:26.520 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.172:6820/2733060387", 2024-03-19T11:31:26.520 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-19T11:31:26.520 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-19T11:31:26.520 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-19T11:31:26.520 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.520 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.520 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.520 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 6, 2024-03-19T11:31:26.520 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "f91d8c47-4e10-40c6-a02d-66d6b72a657b", 2024-03-19T11:31:26.521 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-19T11:31:26.521 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-19T11:31:26.521 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-19T11:31:26.521 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-19T11:31:26.521 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-19T11:31:26.521 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-19T11:31:26.521 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 13, 2024-03-19T11:31:26.521 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 92, 2024-03-19T11:31:26.521 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-19T11:31:26.521 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-19T11:31:26.521 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-19T11:31:26.521 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:26.521 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.521 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:26.521 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6800", 2024-03-19T11:31:26.522 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 337743699 2024-03-19T11:31:26.522 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.522 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.522 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:26.522 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6801", 2024-03-19T11:31:26.522 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 337743699 2024-03-19T11:31:26.522 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.522 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.522 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.522 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-19T11:31:26.522 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:26.522 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.522 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:26.522 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6802", 2024-03-19T11:31:26.523 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 337743699 2024-03-19T11:31:26.523 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.523 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.523 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:26.523 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6803", 2024-03-19T11:31:26.523 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 337743699 2024-03-19T11:31:26.523 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.523 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.523 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.523 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-19T11:31:26.523 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:26.523 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.523 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:26.523 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6806", 2024-03-19T11:31:26.524 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 337743699 2024-03-19T11:31:26.524 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.524 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.524 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:26.524 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6807", 2024-03-19T11:31:26.524 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 337743699 2024-03-19T11:31:26.524 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.524 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.524 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.524 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-19T11:31:26.524 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:26.524 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.524 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:26.524 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6804", 2024-03-19T11:31:26.524 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 337743699 2024-03-19T11:31:26.525 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.525 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.525 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:26.525 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6805", 2024-03-19T11:31:26.525 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 337743699 2024-03-19T11:31:26.525 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.525 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.525 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.525 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.172:6801/337743699", 2024-03-19T11:31:26.525 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.172:6803/337743699", 2024-03-19T11:31:26.525 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.172:6807/337743699", 2024-03-19T11:31:26.525 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.172:6805/337743699", 2024-03-19T11:31:26.525 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-19T11:31:26.525 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-19T11:31:26.525 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-19T11:31:26.526 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.526 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.526 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.526 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 7, 2024-03-19T11:31:26.526 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "46fc052a-5708-42ff-9d5a-bed7a0fc457a", 2024-03-19T11:31:26.526 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-19T11:31:26.526 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-19T11:31:26.526 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-19T11:31:26.526 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-19T11:31:26.526 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-19T11:31:26.526 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-19T11:31:26.526 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 14, 2024-03-19T11:31:26.526 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 80, 2024-03-19T11:31:26.526 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-19T11:31:26.526 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-19T11:31:26.527 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-19T11:31:26.527 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:26.527 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.527 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:26.527 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6824", 2024-03-19T11:31:26.527 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 4145361100 2024-03-19T11:31:26.527 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.527 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.527 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:26.527 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6825", 2024-03-19T11:31:26.527 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 4145361100 2024-03-19T11:31:26.527 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.527 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.527 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.527 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-19T11:31:26.528 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:26.528 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.528 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:26.528 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6826", 2024-03-19T11:31:26.528 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 4145361100 2024-03-19T11:31:26.528 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.528 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.528 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:26.528 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6827", 2024-03-19T11:31:26.528 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 4145361100 2024-03-19T11:31:26.528 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.528 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.528 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.528 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-19T11:31:26.529 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:26.529 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.529 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:26.529 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6830", 2024-03-19T11:31:26.529 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 4145361100 2024-03-19T11:31:26.529 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.529 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.529 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:26.529 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6831", 2024-03-19T11:31:26.529 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 4145361100 2024-03-19T11:31:26.529 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.529 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.529 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.529 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-19T11:31:26.530 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:26.530 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.530 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:26.530 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6828", 2024-03-19T11:31:26.530 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 4145361100 2024-03-19T11:31:26.530 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.530 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.530 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:26.530 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6829", 2024-03-19T11:31:26.530 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 4145361100 2024-03-19T11:31:26.530 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.530 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.530 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.530 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.172:6825/4145361100", 2024-03-19T11:31:26.530 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.172:6827/4145361100", 2024-03-19T11:31:26.531 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.172:6831/4145361100", 2024-03-19T11:31:26.531 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.172:6829/4145361100", 2024-03-19T11:31:26.531 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-19T11:31:26.531 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-19T11:31:26.531 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-19T11:31:26.531 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.531 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.531 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-19T11:31:26.531 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_xinfo": [ 2024-03-19T11:31:26.531 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.531 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 0, 2024-03-19T11:31:26.531 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-19T11:31:26.531 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-19T11:31:26.531 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-19T11:31:26.532 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-19T11:31:26.532 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-19T11:31:26.532 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-19T11:31:01.531687+0000", 2024-03-19T11:31:26.532 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-19T11:31:26.532 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.532 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.532 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 1, 2024-03-19T11:31:26.532 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-19T11:31:26.532 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-19T11:31:26.532 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-19T11:31:26.532 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-19T11:31:26.532 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-19T11:31:26.532 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-19T11:30:52.791546+0000", 2024-03-19T11:31:26.532 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-19T11:31:26.532 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.533 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.533 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 2, 2024-03-19T11:31:26.533 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-19T11:31:26.533 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-19T11:31:26.533 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-19T11:31:26.533 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-19T11:31:26.533 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-19T11:31:26.533 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-19T11:30:51.216737+0000", 2024-03-19T11:31:26.533 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-19T11:31:26.533 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.533 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.533 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 3, 2024-03-19T11:31:26.533 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-19T11:31:26.533 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-19T11:31:26.533 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-19T11:31:26.534 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-19T11:31:26.534 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-19T11:31:26.534 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-19T11:31:06.780394+0000", 2024-03-19T11:31:26.534 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-19T11:31:26.534 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.534 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.534 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 4, 2024-03-19T11:31:26.534 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-19T11:31:26.534 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-19T11:31:26.534 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-19T11:31:26.534 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-19T11:31:26.534 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-19T11:31:26.534 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-19T11:31:06.311323+0000", 2024-03-19T11:31:26.534 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-19T11:31:26.535 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.535 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.535 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 5, 2024-03-19T11:31:26.535 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-19T11:31:26.535 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-19T11:31:26.535 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-19T11:31:26.535 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-19T11:31:26.535 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-19T11:31:26.535 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-19T11:30:59.034487+0000", 2024-03-19T11:31:26.535 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-19T11:31:26.535 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.535 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.535 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 6, 2024-03-19T11:31:26.535 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-19T11:31:26.536 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-19T11:31:26.536 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-19T11:31:26.536 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-19T11:31:26.536 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-19T11:31:26.536 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-19T11:31:04.628602+0000", 2024-03-19T11:31:26.536 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-19T11:31:26.536 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.536 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.536 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 7, 2024-03-19T11:31:26.536 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-19T11:31:26.536 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-19T11:31:26.536 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-19T11:31:26.536 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-19T11:31:26.536 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-19T11:31:26.536 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-19T11:31:09.716969+0000", 2024-03-19T11:31:26.537 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-19T11:31:26.537 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.537 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-19T11:31:26.537 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_upmap": [], 2024-03-19T11:31:26.537 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_upmap_items": [ 2024-03-19T11:31:26.537 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.537 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "2.7", 2024-03-19T11:31:26.537 INFO:teuthology.orchestra.run.smithi038.stdout: "mappings": [ 2024-03-19T11:31:26.537 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.537 INFO:teuthology.orchestra.run.smithi038.stdout: "from": 6, 2024-03-19T11:31:26.537 INFO:teuthology.orchestra.run.smithi038.stdout: "to": 2 2024-03-19T11:31:26.537 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.537 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.537 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.537 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-19T11:31:26.538 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_upmap_primaries": [], 2024-03-19T11:31:26.538 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_temp": [], 2024-03-19T11:31:26.538 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_temp": [], 2024-03-19T11:31:26.538 INFO:teuthology.orchestra.run.smithi038.stdout: "blocklist": {}, 2024-03-19T11:31:26.538 INFO:teuthology.orchestra.run.smithi038.stdout: "range_blocklist": {}, 2024-03-19T11:31:26.538 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profiles": { 2024-03-19T11:31:26.538 INFO:teuthology.orchestra.run.smithi038.stdout: "default": { 2024-03-19T11:31:26.538 INFO:teuthology.orchestra.run.smithi038.stdout: "crush-failure-domain": "osd", 2024-03-19T11:31:26.538 INFO:teuthology.orchestra.run.smithi038.stdout: "k": "2", 2024-03-19T11:31:26.538 INFO:teuthology.orchestra.run.smithi038.stdout: "m": "1", 2024-03-19T11:31:26.538 INFO:teuthology.orchestra.run.smithi038.stdout: "plugin": "jerasure", 2024-03-19T11:31:26.538 INFO:teuthology.orchestra.run.smithi038.stdout: "technique": "reed_sol_van" 2024-03-19T11:31:26.538 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.538 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.539 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps_queue": [ 2024-03-19T11:31:26.539 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.539 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 4, 2024-03-19T11:31:26.539 INFO:teuthology.orchestra.run.smithi038.stdout: "snaps": [ 2024-03-19T11:31:26.539 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.539 INFO:teuthology.orchestra.run.smithi038.stdout: "begin": 18, 2024-03-19T11:31:26.539 INFO:teuthology.orchestra.run.smithi038.stdout: "length": 2 2024-03-19T11:31:26.539 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.539 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.539 INFO:teuthology.orchestra.run.smithi038.stdout: "begin": 21, 2024-03-19T11:31:26.539 INFO:teuthology.orchestra.run.smithi038.stdout: "length": 2 2024-03-19T11:31:26.539 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.539 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.539 INFO:teuthology.orchestra.run.smithi038.stdout: "begin": 25, 2024-03-19T11:31:26.539 INFO:teuthology.orchestra.run.smithi038.stdout: "length": 1 2024-03-19T11:31:26.540 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.540 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.540 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.540 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-19T11:31:26.540 INFO:teuthology.orchestra.run.smithi038.stdout: "new_removed_snaps": [], 2024-03-19T11:31:26.540 INFO:teuthology.orchestra.run.smithi038.stdout: "new_purged_snaps": [], 2024-03-19T11:31:26.540 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_node_flags": {}, 2024-03-19T11:31:26.540 INFO:teuthology.orchestra.run.smithi038.stdout: "device_class_flags": {}, 2024-03-19T11:31:26.540 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode": { 2024-03-19T11:31:26.540 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode_enabled": false, 2024-03-19T11:31:26.540 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_bucket_count": 0, 2024-03-19T11:31:26.540 INFO:teuthology.orchestra.run.smithi038.stdout: "degraded_stretch_mode": 0, 2024-03-19T11:31:26.540 INFO:teuthology.orchestra.run.smithi038.stdout: "recovering_stretch_mode": 0, 2024-03-19T11:31:26.540 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode_bucket": 0 2024-03-19T11:31:26.540 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.541 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:26.543 INFO:tasks.thrashosds.thrasher:j is {'epoch': 107, 'fsid': '246bcf68-c431-4aa7-b787-5e9b2d9519d6', 'created': '2024-03-19T11:29:33.327073+0000', 'modified': '2024-03-19T11:31:26.167148+0000', 'last_up_change': '2024-03-19T11:29:43.166618+0000', 'last_in_change': '2024-03-19T11:29:35.632327+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 8, 'require_min_compat_client': 'reef', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T11:29:42.219654+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-03-19T11:29:44.428406+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'low_tier', 'create_time': '2024-03-19T11:29:55.608699+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 0.75, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'unique_pool_0', 'create_time': '2024-03-19T11:29:56.945423+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 39, 'pg_placement_num': 37, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 38, 'last_pg_merge_meta': {'source_pgid': '4.27', 'ready_epoch': 104, 'last_epoch_started': 103, 'last_epoch_clean': 103, 'source_version': "99'63", 'target_version': "61'40"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '106', 'last_force_op_resend_preluminous': '106', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 32, 'snap_epoch': 107, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'dedup_cdc_chunk_size': 131072, 'dedup_chunk_algorithm': 'fastcdc', 'dedup_tier': 3, 'fingerprint_algorithm': 'sha256'}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.049999952316284, 'score_stable': 2.049999952316284, 'optimal_score': 1, 'raw_score_acting': 2.049999952316284, 'raw_score_stable': 2.049999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '3bb2d2be-5253-4da0-acf3-85a114dc3801', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 106, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6808', 'nonce': 3148574891}, {'type': 'v1', 'addr': '172.21.15.38:6811', 'nonce': 3148574891}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6813', 'nonce': 3148574891}, {'type': 'v1', 'addr': '172.21.15.38:6815', 'nonce': 3148574891}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6822', 'nonce': 3148574891}, {'type': 'v1', 'addr': '172.21.15.38:6823', 'nonce': 3148574891}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6818', 'nonce': 3148574891}, {'type': 'v1', 'addr': '172.21.15.38:6821', 'nonce': 3148574891}]}, 'public_addr': '172.21.15.38:6811/3148574891', 'cluster_addr': '172.21.15.38:6815/3148574891', 'heartbeat_back_addr': '172.21.15.38:6823/3148574891', 'heartbeat_front_addr': '172.21.15.38:6821/3148574891', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'e1c926b2-e0be-4ccc-b8a5-d80d0b584ead', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 104, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6824', 'nonce': 675027622}, {'type': 'v1', 'addr': '172.21.15.38:6825', 'nonce': 675027622}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6826', 'nonce': 675027622}, {'type': 'v1', 'addr': '172.21.15.38:6827', 'nonce': 675027622}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6830', 'nonce': 675027622}, {'type': 'v1', 'addr': '172.21.15.38:6831', 'nonce': 675027622}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6828', 'nonce': 675027622}, {'type': 'v1', 'addr': '172.21.15.38:6829', 'nonce': 675027622}]}, 'public_addr': '172.21.15.38:6825/675027622', 'cluster_addr': '172.21.15.38:6827/675027622', 'heartbeat_back_addr': '172.21.15.38:6831/675027622', 'heartbeat_front_addr': '172.21.15.38:6829/675027622', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '61262219-4f8a-4733-af9d-b5208a17c0ea', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 82, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6800', 'nonce': 58279616}, {'type': 'v1', 'addr': '172.21.15.38:6801', 'nonce': 58279616}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6802', 'nonce': 58279616}, {'type': 'v1', 'addr': '172.21.15.38:6804', 'nonce': 58279616}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6814', 'nonce': 58279616}, {'type': 'v1', 'addr': '172.21.15.38:6817', 'nonce': 58279616}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6807', 'nonce': 58279616}, {'type': 'v1', 'addr': '172.21.15.38:6810', 'nonce': 58279616}]}, 'public_addr': '172.21.15.38:6801/58279616', 'cluster_addr': '172.21.15.38:6804/58279616', 'heartbeat_back_addr': '172.21.15.38:6817/58279616', 'heartbeat_front_addr': '172.21.15.38:6810/58279616', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'a31f8d1b-4e38-4ca3-b251-0877002d7235', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 82, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6803', 'nonce': 2942163952}, {'type': 'v1', 'addr': '172.21.15.38:6805', 'nonce': 2942163952}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6806', 'nonce': 2942163952}, {'type': 'v1', 'addr': '172.21.15.38:6809', 'nonce': 2942163952}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6819', 'nonce': 2942163952}, {'type': 'v1', 'addr': '172.21.15.38:6820', 'nonce': 2942163952}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6812', 'nonce': 2942163952}, {'type': 'v1', 'addr': '172.21.15.38:6816', 'nonce': 2942163952}]}, 'public_addr': '172.21.15.38:6805/2942163952', 'cluster_addr': '172.21.15.38:6809/2942163952', 'heartbeat_back_addr': '172.21.15.38:6820/2942163952', 'heartbeat_front_addr': '172.21.15.38:6816/2942163952', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6746723d-f59c-40ba-857e-d8c954d2675f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 96, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6808', 'nonce': 1353919713}, {'type': 'v1', 'addr': '172.21.15.172:6810', 'nonce': 1353919713}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6812', 'nonce': 1353919713}, {'type': 'v1', 'addr': '172.21.15.172:6814', 'nonce': 1353919713}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6819', 'nonce': 1353919713}, {'type': 'v1', 'addr': '172.21.15.172:6821', 'nonce': 1353919713}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6816', 'nonce': 1353919713}, {'type': 'v1', 'addr': '172.21.15.172:6818', 'nonce': 1353919713}]}, 'public_addr': '172.21.15.172:6810/1353919713', 'cluster_addr': '172.21.15.172:6814/1353919713', 'heartbeat_back_addr': '172.21.15.172:6821/1353919713', 'heartbeat_front_addr': '172.21.15.172:6818/1353919713', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'f59afdf8-2277-4583-af68-78136f1c5d62', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 100, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6809', 'nonce': 2733060387}, {'type': 'v1', 'addr': '172.21.15.172:6811', 'nonce': 2733060387}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6813', 'nonce': 2733060387}, {'type': 'v1', 'addr': '172.21.15.172:6815', 'nonce': 2733060387}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6822', 'nonce': 2733060387}, {'type': 'v1', 'addr': '172.21.15.172:6823', 'nonce': 2733060387}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6817', 'nonce': 2733060387}, {'type': 'v1', 'addr': '172.21.15.172:6820', 'nonce': 2733060387}]}, 'public_addr': '172.21.15.172:6811/2733060387', 'cluster_addr': '172.21.15.172:6815/2733060387', 'heartbeat_back_addr': '172.21.15.172:6823/2733060387', 'heartbeat_front_addr': '172.21.15.172:6820/2733060387', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'f91d8c47-4e10-40c6-a02d-66d6b72a657b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 92, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6800', 'nonce': 337743699}, {'type': 'v1', 'addr': '172.21.15.172:6801', 'nonce': 337743699}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6802', 'nonce': 337743699}, {'type': 'v1', 'addr': '172.21.15.172:6803', 'nonce': 337743699}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6806', 'nonce': 337743699}, {'type': 'v1', 'addr': '172.21.15.172:6807', 'nonce': 337743699}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6804', 'nonce': 337743699}, {'type': 'v1', 'addr': '172.21.15.172:6805', 'nonce': 337743699}]}, 'public_addr': '172.21.15.172:6801/337743699', 'cluster_addr': '172.21.15.172:6803/337743699', 'heartbeat_back_addr': '172.21.15.172:6807/337743699', 'heartbeat_front_addr': '172.21.15.172:6805/337743699', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '46fc052a-5708-42ff-9d5a-bed7a0fc457a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 80, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6824', 'nonce': 4145361100}, {'type': 'v1', 'addr': '172.21.15.172:6825', 'nonce': 4145361100}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6826', 'nonce': 4145361100}, {'type': 'v1', 'addr': '172.21.15.172:6827', 'nonce': 4145361100}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6830', 'nonce': 4145361100}, {'type': 'v1', 'addr': '172.21.15.172:6831', 'nonce': 4145361100}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6828', 'nonce': 4145361100}, {'type': 'v1', 'addr': '172.21.15.172:6829', 'nonce': 4145361100}]}, 'public_addr': '172.21.15.172:6825/4145361100', 'cluster_addr': '172.21.15.172:6827/4145361100', 'heartbeat_back_addr': '172.21.15.172:6831/4145361100', 'heartbeat_front_addr': '172.21.15.172:6829/4145361100', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-19T11:31:01.531687+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-19T11:30:52.791546+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-19T11:30:51.216737+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-19T11:31:06.780394+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-19T11:31:06.311323+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-19T11:30:59.034487+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-19T11:31:04.628602+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-19T11:31:09.716969+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.7', 'mappings': [{'from': 6, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 4, 'snaps': [{'begin': 18, 'length': 2}, {'begin': 21, 'length': 2}, {'begin': 25, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-19T11:31:26.543 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 2.7 2024-03-19T11:31:26.543 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd rm-pg-upmap-items 2.7 2024-03-19T11:31:26.544 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:26.550+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:26.642 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:26.649+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:26.671 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:26.671 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:31:26.671 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:31:26.671 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:31:26.672 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.672 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1063 4.4 4:214026b3:::smithi03829856-47:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:31:26.672 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.784904+0000", 2024-03-19T11:31:26.672 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 72.896041331999996, 2024-03-19T11:31:26.672 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024425912000000001, 2024-03-19T11:31:26.672 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:26.672 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:26.672 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:26.672 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:26.672 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:26.672 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1063 2024-03-19T11:31:26.672 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.672 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:26.673 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.673 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:26.673 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784904+0000", 2024-03-19T11:31:26.673 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:26.673 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.673 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.673 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:26.673 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784904+0000", 2024-03-19T11:31:26.673 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:26.673 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.673 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.673 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:26.673 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784906+0000", 2024-03-19T11:31:26.674 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1409999999999999e-06 2024-03-19T11:31:26.674 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.674 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.674 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:26.674 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784915+0000", 2024-03-19T11:31:26.674 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5459999999999992e-06 2024-03-19T11:31:26.674 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.674 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.674 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:26.674 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784935+0000", 2024-03-19T11:31:26.674 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0310999999999999e-05 2024-03-19T11:31:26.674 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.674 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.674 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:26.675 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784946+0000", 2024-03-19T11:31:26.675 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0675e-05 2024-03-19T11:31:26.675 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.675 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.675 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:26.675 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.785032+0000", 2024-03-19T11:31:26.675 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6049000000000002e-05 2024-03-19T11:31:26.675 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.675 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.675 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:26.675 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.785136+0000", 2024-03-19T11:31:26.675 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000104209 2024-03-19T11:31:26.675 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.675 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.676 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:26.676 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.809330+0000", 2024-03-19T11:31:26.676 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024193981 2024-03-19T11:31:26.676 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.676 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.676 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.676 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.676 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.676 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1433 4.16 4:6be89c06:::smithi03829856-24:c [sparse-read 0~1100427,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:31:26.676 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.976652+0000", 2024-03-19T11:31:26.676 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 60.704293630999999, 2024-03-19T11:31:26.676 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95684753199999995, 2024-03-19T11:31:26.676 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:26.676 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:26.676 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:26.677 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:26.677 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:26.677 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1433 2024-03-19T11:31:26.677 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.677 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:26.677 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.677 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:26.677 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976652+0000", 2024-03-19T11:31:26.677 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:26.677 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.677 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.677 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:26.677 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976652+0000", 2024-03-19T11:31:26.678 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:26.678 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.678 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.678 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:26.678 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976654+0000", 2024-03-19T11:31:26.678 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6199999999999999e-06 2024-03-19T11:31:26.678 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.678 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.678 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:26.678 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976668+0000", 2024-03-19T11:31:26.678 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3213999999999999e-05 2024-03-19T11:31:26.678 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.678 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.678 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:26.679 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976694+0000", 2024-03-19T11:31:26.679 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6488000000000001e-05 2024-03-19T11:31:26.679 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.679 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.679 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:26.679 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976705+0000", 2024-03-19T11:31:26.679 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1044e-05 2024-03-19T11:31:26.679 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.679 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.679 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:26.679 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976795+0000", 2024-03-19T11:31:26.679 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0283000000000003e-05 2024-03-19T11:31:26.679 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.679 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.679 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:26.680 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976811+0000", 2024-03-19T11:31:26.680 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5262000000000001e-05 2024-03-19T11:31:26.680 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.680 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.680 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:26.680 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.927337+0000", 2024-03-19T11:31:26.680 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95052597100000003 2024-03-19T11:31:26.680 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.680 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.680 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:26.680 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.927495+0000", 2024-03-19T11:31:26.680 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015788600000000001 2024-03-19T11:31:26.680 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.680 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.681 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:26.681 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.933499+0000", 2024-03-19T11:31:26.681 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0060047640000000001 2024-03-19T11:31:26.681 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.681 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.681 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.681 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.681 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.681 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1434 4.16 4:6be89c06:::smithi03829856-24:c [read 0~2386561,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:31:26.681 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.976739+0000", 2024-03-19T11:31:26.681 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 60.704206470000003, 2024-03-19T11:31:26.681 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95781317399999999, 2024-03-19T11:31:26.681 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:26.681 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:26.681 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:26.682 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:26.682 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:26.682 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1434 2024-03-19T11:31:26.682 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.682 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:26.682 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.682 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:26.682 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976739+0000", 2024-03-19T11:31:26.682 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:26.682 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.682 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.682 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:26.682 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976739+0000", 2024-03-19T11:31:26.682 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:26.683 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.683 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.683 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:26.683 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976740+0000", 2024-03-19T11:31:26.683 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8800000000000001e-07 2024-03-19T11:31:26.683 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.683 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.683 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:26.683 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976748+0000", 2024-03-19T11:31:26.683 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8979999999999996e-06 2024-03-19T11:31:26.683 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.683 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.683 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:26.683 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976757+0000", 2024-03-19T11:31:26.683 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9439999999999994e-06 2024-03-19T11:31:26.684 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.684 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.684 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:26.684 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976763+0000", 2024-03-19T11:31:26.684 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5030000000000002e-06 2024-03-19T11:31:26.684 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.684 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.684 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:26.684 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976849+0000", 2024-03-19T11:31:26.684 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5803999999999994e-05 2024-03-19T11:31:26.684 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.684 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.684 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:26.684 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976860+0000", 2024-03-19T11:31:26.684 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0869e-05 2024-03-19T11:31:26.685 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.685 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.685 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:26.685 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.933562+0000", 2024-03-19T11:31:26.685 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95670232300000002 2024-03-19T11:31:26.685 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.685 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.685 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:26.685 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.933693+0000", 2024-03-19T11:31:26.685 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013110699999999999 2024-03-19T11:31:26.685 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.685 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.685 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:26.686 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.934552+0000", 2024-03-19T11:31:26.686 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00085883800000000001 2024-03-19T11:31:26.686 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.686 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.686 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.686 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.686 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.686 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1435 4.16 4:6be89c06:::smithi03829856-24:c [sparse-read 0~1100427] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:31:26.686 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.976775+0000", 2024-03-19T11:31:26.686 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 60.704170799000003, 2024-03-19T11:31:26.686 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95962779499999995, 2024-03-19T11:31:26.686 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:26.686 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:26.686 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:26.687 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:26.687 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:26.687 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1435 2024-03-19T11:31:26.687 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.687 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:26.687 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.687 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:26.687 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976775+0000", 2024-03-19T11:31:26.687 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:26.687 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.687 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.687 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:26.687 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976775+0000", 2024-03-19T11:31:26.687 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:26.687 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.688 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.688 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:26.688 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976775+0000", 2024-03-19T11:31:26.688 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9800000000000003e-07 2024-03-19T11:31:26.688 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.688 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.688 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:26.688 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976779+0000", 2024-03-19T11:31:26.688 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3830000000000001e-06 2024-03-19T11:31:26.688 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.688 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.688 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:26.688 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976785+0000", 2024-03-19T11:31:26.688 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5649999999999998e-06 2024-03-19T11:31:26.689 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.689 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.689 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:26.689 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976790+0000", 2024-03-19T11:31:26.689 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1379999999999998e-06 2024-03-19T11:31:26.689 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.689 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.689 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:26.689 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976889+0000", 2024-03-19T11:31:26.689 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8331000000000004e-05 2024-03-19T11:31:26.689 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.689 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.689 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:26.689 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976894+0000", 2024-03-19T11:31:26.689 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8270000000000003e-06 2024-03-19T11:31:26.690 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.690 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.690 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:26.690 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.935982+0000", 2024-03-19T11:31:26.690 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95908714100000003 2024-03-19T11:31:26.690 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.690 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.690 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:26.690 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.936093+0000", 2024-03-19T11:31:26.690 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000111557 2024-03-19T11:31:26.690 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.690 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.690 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:26.690 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.936402+0000", 2024-03-19T11:31:26.691 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030925499999999998 2024-03-19T11:31:26.691 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.691 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.691 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.691 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.691 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.691 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1442 4.6 4:626acd4d:::smithi03829856-2:head [cache-evict] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:31:26.691 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.978864+0000", 2024-03-19T11:31:26.691 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 60.702081135, 2024-03-19T11:31:26.691 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95708845099999995, 2024-03-19T11:31:26.691 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:26.691 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:26.691 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:26.691 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:26.691 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:26.692 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1442 2024-03-19T11:31:26.692 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.692 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:26.692 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.692 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:26.692 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978864+0000", 2024-03-19T11:31:26.692 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:26.692 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.692 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.692 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:26.692 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978864+0000", 2024-03-19T11:31:26.692 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:26.692 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.692 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.692 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:26.693 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978866+0000", 2024-03-19T11:31:26.693 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0729999999999999e-06 2024-03-19T11:31:26.693 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.693 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.693 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:26.693 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978877+0000", 2024-03-19T11:31:26.693 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0390999999999999e-05 2024-03-19T11:31:26.693 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.693 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.693 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:26.693 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978902+0000", 2024-03-19T11:31:26.693 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5635000000000001e-05 2024-03-19T11:31:26.693 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.693 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.693 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:26.694 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978915+0000", 2024-03-19T11:31:26.694 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2617e-05 2024-03-19T11:31:26.694 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.694 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.694 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:26.694 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.979005+0000", 2024-03-19T11:31:26.694 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0552999999999999e-05 2024-03-19T11:31:26.694 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.694 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.694 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:26.694 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.979021+0000", 2024-03-19T11:31:26.694 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5102999999999999e-05 2024-03-19T11:31:26.694 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.694 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.695 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:26.695 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.926879+0000", 2024-03-19T11:31:26.695 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.94785817900000002 2024-03-19T11:31:26.695 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.695 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.695 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:26.695 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.927277+0000", 2024-03-19T11:31:26.695 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00039819600000000002 2024-03-19T11:31:26.695 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.695 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.695 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:26.695 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.935752+0000", 2024-03-19T11:31:26.695 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0084753420000000003 2024-03-19T11:31:26.695 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.695 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.696 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:26.696 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.935874+0000", 2024-03-19T11:31:26.696 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000121311 2024-03-19T11:31:26.696 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.696 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.696 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:26.696 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.935953+0000", 2024-03-19T11:31:26.696 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9050999999999999e-05 2024-03-19T11:31:26.696 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.696 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.696 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.696 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.696 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.696 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2003 4.16 4:6868af91:::smithi03829856-10:head [tier-promote] snapc 12=[12,11,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:31:26.696 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:45.804046+0000", 2024-03-19T11:31:26.697 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 40.876899221999999, 2024-03-19T11:31:26.697 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018976396, 2024-03-19T11:31:26.697 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:26.697 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:26.697 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:26.697 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:26.697 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:26.697 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2003 2024-03-19T11:31:26.697 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.697 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:26.697 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.697 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:26.697 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804046+0000", 2024-03-19T11:31:26.697 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:26.698 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.698 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.698 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:26.698 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804046+0000", 2024-03-19T11:31:26.698 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:26.698 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.698 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.698 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:26.698 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804052+0000", 2024-03-19T11:31:26.698 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7599999999999999e-06 2024-03-19T11:31:26.698 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.699 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.699 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:26.699 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804090+0000", 2024-03-19T11:31:26.699 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8018000000000001e-05 2024-03-19T11:31:26.699 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.699 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.699 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:26.699 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804256+0000", 2024-03-19T11:31:26.699 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016593599999999999 2024-03-19T11:31:26.699 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.699 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.699 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:26.699 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804300+0000", 2024-03-19T11:31:26.699 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3648e-05 2024-03-19T11:31:26.700 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.700 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.700 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:26.700 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804538+0000", 2024-03-19T11:31:26.700 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000238347 2024-03-19T11:31:26.700 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.700 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.700 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:26.700 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804716+0000", 2024-03-19T11:31:26.700 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000177968 2024-03-19T11:31:26.700 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.700 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.700 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:26.700 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.823023+0000", 2024-03-19T11:31:26.700 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018306718999999999 2024-03-19T11:31:26.701 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.701 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.701 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.701 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.701 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.701 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1d58db11:::smithi03829856-18:6, version: 55'80, data_included: [759683~639358,1904434~409218], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:6@55'80, size: 3159683, copy_subset: [(0, 3159683)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2313652, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1d58db11:::smithi03829856-18:b, version: 58'84, data_included: [443718~955323,1564138~93253], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:b@58'84, size: 3445402, copy_subset: [(0, 3445402)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1657391, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:10, version: 66'95, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:10@66'95, size: 3764735, copy_subset: [(0, 3764735)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:12, version: 66'97, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:12@66'97, size: 2504243, copy_subset: [(0, 2504243)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:c, version: 70'102, data_included: [416807~576167,1542129~472409], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:c@70'102, size: 2857157, copy_subset: [(0, 2857157)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2014538, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:26.701 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.275179+0000", 2024-03-19T11:31:26.702 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 32.405766231999998, 2024-03-19T11:31:26.702 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026559353000000001, 2024-03-19T11:31:26.702 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:26.702 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:26.702 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:26.702 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.702 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:26.702 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.275179+0000", 2024-03-19T11:31:26.702 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:26.702 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.702 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.702 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:26.702 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.275179+0000", 2024-03-19T11:31:26.703 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:26.703 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.703 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.703 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:26.703 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.275196+0000", 2024-03-19T11:31:26.703 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6946999999999999e-05 2024-03-19T11:31:26.703 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.703 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.703 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:26.703 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281456+0000", 2024-03-19T11:31:26.703 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0062600529999999998 2024-03-19T11:31:26.703 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.703 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.703 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:26.703 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281524+0000", 2024-03-19T11:31:26.704 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7971000000000004e-05 2024-03-19T11:31:26.704 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.704 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.704 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:26.704 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281591+0000", 2024-03-19T11:31:26.704 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6704000000000004e-05 2024-03-19T11:31:26.704 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.704 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.704 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:26.704 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281865+0000", 2024-03-19T11:31:26.704 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000274097 2024-03-19T11:31:26.704 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.704 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.704 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:26.705 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281901+0000", 2024-03-19T11:31:26.705 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5777e-05 2024-03-19T11:31:26.705 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.705 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.705 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:26.705 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.301738+0000", 2024-03-19T11:31:26.705 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019837804000000001 2024-03-19T11:31:26.705 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.705 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.705 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.705 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.705 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.705 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.18 e75,74 4:1d58db11:::smithi03829856-18:2,48'76)", 2024-03-19T11:31:26.705 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.281661+0000", 2024-03-19T11:31:26.705 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 32.399284700999999, 2024-03-19T11:31:26.706 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020384337999999998, 2024-03-19T11:31:26.706 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:26.706 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:26.706 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:26.706 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.706 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:26.706 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281661+0000", 2024-03-19T11:31:26.706 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:26.706 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.706 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.706 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:26.706 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281661+0000", 2024-03-19T11:31:26.706 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:26.706 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.707 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.707 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:26.707 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281661+0000", 2024-03-19T11:31:26.707 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9100000000000003e-07 2024-03-19T11:31:26.707 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.707 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.707 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:26.707 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281668+0000", 2024-03-19T11:31:26.707 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8199999999999999e-06 2024-03-19T11:31:26.707 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.707 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.707 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:26.707 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281679+0000", 2024-03-19T11:31:26.707 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1107999999999999e-05 2024-03-19T11:31:26.708 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.708 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.708 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:26.708 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281691+0000", 2024-03-19T11:31:26.708 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1399999999999999e-05 2024-03-19T11:31:26.708 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.708 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.708 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:26.708 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.301804+0000", 2024-03-19T11:31:26.708 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020113669000000001 2024-03-19T11:31:26.708 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.708 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.708 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:26.708 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.301818+0000", 2024-03-19T11:31:26.709 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3718e-05 2024-03-19T11:31:26.709 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.709 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.709 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:26.709 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302045+0000", 2024-03-19T11:31:26.709 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000226832 2024-03-19T11:31:26.709 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.709 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.709 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.709 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.709 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.709 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1dabc0f1:::smithi03829856-1:head, version: 70'103, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:head@70'103, size: 2621892, copy_subset: [(0, 2621892)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:26.709 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.281783+0000", 2024-03-19T11:31:26.710 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 32.399162425, 2024-03-19T11:31:26.710 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026187221, 2024-03-19T11:31:26.710 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:26.710 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:26.710 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:26.710 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.710 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:26.710 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281783+0000", 2024-03-19T11:31:26.710 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:26.710 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.710 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.710 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:26.710 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281783+0000", 2024-03-19T11:31:26.710 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:26.711 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.711 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.711 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:26.711 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281783+0000", 2024-03-19T11:31:26.711 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1699999999999999e-07 2024-03-19T11:31:26.711 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.711 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.711 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:26.711 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.282962+0000", 2024-03-19T11:31:26.711 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001178601 2024-03-19T11:31:26.711 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.711 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.711 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:26.711 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.282987+0000", 2024-03-19T11:31:26.711 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4899e-05 2024-03-19T11:31:26.712 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.712 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.712 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:26.712 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283006+0000", 2024-03-19T11:31:26.712 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9494999999999999e-05 2024-03-19T11:31:26.712 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.712 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.712 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:26.712 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.305757+0000", 2024-03-19T11:31:26.712 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.022750570000000001 2024-03-19T11:31:26.712 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.712 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.712 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:26.712 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.305770+0000", 2024-03-19T11:31:26.713 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2791e-05 2024-03-19T11:31:26.713 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.713 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.713 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:26.713 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.307970+0000", 2024-03-19T11:31:26.713 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022005480000000001 2024-03-19T11:31:26.713 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.713 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.713 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.713 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.713 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.713 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.18 e75,74 4:1d58db11:::smithi03829856-18:c,70'104)", 2024-03-19T11:31:26.713 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.283036+0000", 2024-03-19T11:31:26.713 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 32.397909325000001, 2024-03-19T11:31:26.714 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019146032, 2024-03-19T11:31:26.714 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:26.714 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:26.714 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:26.714 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.714 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:26.714 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283036+0000", 2024-03-19T11:31:26.714 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:26.714 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.714 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.714 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:26.714 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283036+0000", 2024-03-19T11:31:26.714 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:26.714 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.714 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.715 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:26.715 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283037+0000", 2024-03-19T11:31:26.715 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4899999999999995e-07 2024-03-19T11:31:26.715 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.715 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.715 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:26.715 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283042+0000", 2024-03-19T11:31:26.715 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0930000000000002e-06 2024-03-19T11:31:26.715 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.715 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.715 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:26.715 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283053+0000", 2024-03-19T11:31:26.715 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1083999999999999e-05 2024-03-19T11:31:26.715 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.715 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.716 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:26.716 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283059+0000", 2024-03-19T11:31:26.716 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6329999999999999e-06 2024-03-19T11:31:26.716 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.716 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.716 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:26.716 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302090+0000", 2024-03-19T11:31:26.716 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01903038 2024-03-19T11:31:26.716 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.716 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.716 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:26.716 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302097+0000", 2024-03-19T11:31:26.716 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.221e-06 2024-03-19T11:31:26.716 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.716 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.717 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:26.717 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302182+0000", 2024-03-19T11:31:26.717 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5072000000000001e-05 2024-03-19T11:31:26.717 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.717 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.717 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.717 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.717 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.717 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(osd.5.0:143 4.18 e75/74 4:1d58db11:::smithi03829856-18:head v 75'106, mlcod=0'0)", 2024-03-19T11:31:26.717 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.283083+0000", 2024-03-19T11:31:26.717 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 32.397861998000003, 2024-03-19T11:31:26.717 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025309439, 2024-03-19T11:31:26.717 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:26.717 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:26.718 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:26.718 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.718 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:26.718 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283083+0000", 2024-03-19T11:31:26.718 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:26.718 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.718 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.718 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:26.718 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283083+0000", 2024-03-19T11:31:26.718 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:26.718 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.718 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.718 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:26.718 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283084+0000", 2024-03-19T11:31:26.718 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7800000000000002e-07 2024-03-19T11:31:26.719 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.719 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.719 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:26.719 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.284082+0000", 2024-03-19T11:31:26.719 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0009977880000000001 2024-03-19T11:31:26.719 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.719 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.719 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:26.719 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.284095+0000", 2024-03-19T11:31:26.719 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3256e-05 2024-03-19T11:31:26.719 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.719 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.719 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:26.719 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.284102+0000", 2024-03-19T11:31:26.720 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.605e-06 2024-03-19T11:31:26.720 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.720 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.720 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:26.720 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302285+0000", 2024-03-19T11:31:26.720 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018182423 2024-03-19T11:31:26.720 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.720 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.720 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:26.720 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302341+0000", 2024-03-19T11:31:26.720 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.626e-05 2024-03-19T11:31:26.720 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.720 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.720 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:26.721 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.308338+0000", 2024-03-19T11:31:26.721 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0059968030000000002 2024-03-19T11:31:26.721 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.721 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.721 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:26.721 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.308393+0000", 2024-03-19T11:31:26.721 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4826e-05 2024-03-19T11:31:26.721 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.721 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.721 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.721 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.721 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.721 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "pg_update_log_missing(4.18 epoch 75/74 rep_tid 144 entries 75'107 (0'0) error 4:1d58db11:::smithi03829856-18:head by client.4415.0:2283 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-19T11:31:26.721 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.288252+0000", 2024-03-19T11:31:26.722 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 32.392693448000003, 2024-03-19T11:31:26.722 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020167377, 2024-03-19T11:31:26.722 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:26.722 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-03-19T11:31:26.722 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:26.722 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.722 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:26.722 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288252+0000", 2024-03-19T11:31:26.722 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:26.722 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.722 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.722 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:26.722 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288252+0000", 2024-03-19T11:31:26.722 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:26.722 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.723 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.723 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:26.723 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288253+0000", 2024-03-19T11:31:26.723 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2109999999999999e-06 2024-03-19T11:31:26.723 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.723 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.723 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:26.723 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288268+0000", 2024-03-19T11:31:26.723 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5003e-05 2024-03-19T11:31:26.723 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.723 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.723 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:26.723 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288301+0000", 2024-03-19T11:31:26.723 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2539999999999997e-05 2024-03-19T11:31:26.724 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.724 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.724 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:26.724 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288319+0000", 2024-03-19T11:31:26.724 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8232000000000001e-05 2024-03-19T11:31:26.724 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.724 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.724 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:26.724 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.305490+0000", 2024-03-19T11:31:26.724 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017170857000000001 2024-03-19T11:31:26.724 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.724 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.724 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:26.724 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.308419+0000", 2024-03-19T11:31:26.725 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029295340000000001 2024-03-19T11:31:26.725 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.725 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.725 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.725 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.725 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.725 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1d58db11:::smithi03829856-18:6, version: 55'80, data_included: [2313652~310127,3128657~31026], data_size: 341153, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:6@55'80, size: 3159683, copy_subset: [(0, 3159683)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3159683, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2313652, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:1d58db11:::smithi03829856-18:b, version: 58'84, data_included: [1657391~966388,2633066~82188], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:b@58'84, size: 3445402, copy_subset: [(0, 3445402)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2715254, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1657391, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:10, version: 66'95, data_included: [1829043~668012,3134120~380564], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:10@66'95, size: 3764735, copy_subset: [(0, 3764735)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3514684, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:12, version: 66'97, data_included: [1829043~507467], data_size: 507467, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:12@66'97, size: 2504243, copy_subset: [(0, 2504243)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2504243, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:c, version: 70'102, data_included: [2014538~156250], data_size: 156250, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:c@70'102, size: 2857157, copy_subset: [(0, 2857157)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2857157, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2014538, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:26.725 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.312362+0000", 2024-03-19T11:31:26.725 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 32.368583399000002, 2024-03-19T11:31:26.725 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017968951, 2024-03-19T11:31:26.725 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:26.725 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:26.725 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:26.725 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.725 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:26.726 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.312362+0000", 2024-03-19T11:31:26.726 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:26.726 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.726 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.726 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:26.726 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.312362+0000", 2024-03-19T11:31:26.726 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:26.726 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.726 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.726 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:26.726 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.312363+0000", 2024-03-19T11:31:26.726 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.528e-06 2024-03-19T11:31:26.726 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.726 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.727 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:26.727 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315748+0000", 2024-03-19T11:31:26.727 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033842830000000001 2024-03-19T11:31:26.727 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.727 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.727 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:26.727 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315795+0000", 2024-03-19T11:31:26.727 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7382999999999997e-05 2024-03-19T11:31:26.727 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.727 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.727 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:26.727 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315845+0000", 2024-03-19T11:31:26.727 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0319999999999999e-05 2024-03-19T11:31:26.727 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.727 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.728 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:26.728 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.316123+0000", 2024-03-19T11:31:26.728 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00027747200000000002 2024-03-19T11:31:26.728 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.728 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.728 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:26.728 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.316170+0000", 2024-03-19T11:31:26.728 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7466999999999998e-05 2024-03-19T11:31:26.728 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.728 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.728 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:26.728 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.330331+0000", 2024-03-19T11:31:26.728 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014160498000000001 2024-03-19T11:31:26.728 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.729 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.729 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.729 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.729 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.729 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1dabc0f1:::smithi03829856-1:head, version: 70'103, data_included: [1829043~555153], data_size: 555153, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:head@70'103, size: 2621892, copy_subset: [(0, 2621892)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2621892, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:26.729 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.315898+0000", 2024-03-19T11:31:26.729 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 32.365047412999999, 2024-03-19T11:31:26.729 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016880424000000002, 2024-03-19T11:31:26.729 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:26.729 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:26.729 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:26.729 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.729 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:26.729 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315898+0000", 2024-03-19T11:31:26.729 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:26.730 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.730 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.730 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:26.730 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315898+0000", 2024-03-19T11:31:26.730 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:26.730 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.730 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.730 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:26.730 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315899+0000", 2024-03-19T11:31:26.730 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.003e-06 2024-03-19T11:31:26.730 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.730 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.730 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:26.731 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.316486+0000", 2024-03-19T11:31:26.731 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00058673099999999995 2024-03-19T11:31:26.731 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.731 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.731 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:26.731 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.316519+0000", 2024-03-19T11:31:26.731 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3124000000000003e-05 2024-03-19T11:31:26.731 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.731 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.731 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:26.731 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.316534+0000", 2024-03-19T11:31:26.731 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.485e-05 2024-03-19T11:31:26.731 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.731 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.731 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:26.732 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.330415+0000", 2024-03-19T11:31:26.732 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013880905000000001 2024-03-19T11:31:26.732 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.732 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.732 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:26.732 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.330439+0000", 2024-03-19T11:31:26.732 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4630999999999998e-05 2024-03-19T11:31:26.732 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.732 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.732 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:26.732 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.332778+0000", 2024-03-19T11:31:26.732 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023391800000000002 2024-03-19T11:31:26.732 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.732 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.732 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.733 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.733 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.733 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2387 4.2 4:43b32242:::smithi03829856-22:head [read 596992~684544] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-03-19T11:31:26.733 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:59.319806+0000", 2024-03-19T11:31:26.733 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 27.361139745999999, 2024-03-19T11:31:26.733 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.78144344399999999, 2024-03-19T11:31:26.733 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:26.733 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:26.733 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:26.733 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:26.733 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:26.733 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2387 2024-03-19T11:31:26.733 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.733 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:26.733 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.734 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:26.734 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319806+0000", 2024-03-19T11:31:26.734 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:26.734 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.734 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.734 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:26.734 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319806+0000", 2024-03-19T11:31:26.734 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:26.734 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.734 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.734 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:26.734 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319807+0000", 2024-03-19T11:31:26.734 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3519999999999999e-06 2024-03-19T11:31:26.735 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.735 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.735 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:26.735 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319812+0000", 2024-03-19T11:31:26.735 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3959999999999997e-06 2024-03-19T11:31:26.735 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.735 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.735 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:26.735 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319827+0000", 2024-03-19T11:31:26.735 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4566000000000001e-05 2024-03-19T11:31:26.735 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.735 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.735 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:26.735 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319835+0000", 2024-03-19T11:31:26.735 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.994e-06 2024-03-19T11:31:26.736 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.736 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.736 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:26.736 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.320119+0000", 2024-03-19T11:31:26.736 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00028445199999999998 2024-03-19T11:31:26.736 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.736 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.736 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:26.736 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.320129+0000", 2024-03-19T11:31:26.736 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9580000000000002e-06 2024-03-19T11:31:26.736 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.736 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.736 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:26.736 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.069842+0000", 2024-03-19T11:31:26.736 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.74971218399999995 2024-03-19T11:31:26.737 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.737 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.737 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:31:26.737 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.070010+0000", 2024-03-19T11:31:26.737 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016846600000000001 2024-03-19T11:31:26.737 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.737 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.737 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:26.737 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.090911+0000", 2024-03-19T11:31:26.737 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020901494999999999 2024-03-19T11:31:26.737 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.737 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.737 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:31:26.737 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.091249+0000", 2024-03-19T11:31:26.737 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000337602 2024-03-19T11:31:26.738 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.738 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.738 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:26.738 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.098478+0000", 2024-03-19T11:31:26.738 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0072287549999999999 2024-03-19T11:31:26.738 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.738 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.738 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:26.738 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.098617+0000", 2024-03-19T11:31:26.738 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013871599999999999 2024-03-19T11:31:26.738 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.738 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.738 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:26.738 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.101249+0000", 2024-03-19T11:31:26.739 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002632508 2024-03-19T11:31:26.739 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.739 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.739 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.739 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.739 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.739 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 81/80 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 15=[]:{7: [6], b: [b], 10: [d], 12: [12, 11], 15: [15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:7, version: 55'82, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@55'82, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 16=[]:{7: [6], b: [b], 10: [d], 16: [16, 15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 15=[]:{7: [6], b: [b], 10: [d], 12: [12, 11], 15: [15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 58'86, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@58'86, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 16=[]:{7: [6], b: [b], 10: [d], 16: [16, 15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [504210~968657,1572695~79919], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 16=[]:{7: [6], b: [b], 10: [d], 16: [16, 15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:26.739 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:00.089924+0000", 2024-03-19T11:31:26.739 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 26.591021517000001, 2024-03-19T11:31:26.739 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024686583000000002, 2024-03-19T11:31:26.739 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:26.739 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:26.739 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:26.739 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.740 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:26.740 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.089924+0000", 2024-03-19T11:31:26.740 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:26.740 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.740 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.740 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:26.740 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.089924+0000", 2024-03-19T11:31:26.740 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:26.740 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.740 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.740 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:26.740 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.089926+0000", 2024-03-19T11:31:26.740 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9589999999999998e-06 2024-03-19T11:31:26.740 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.740 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.741 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:26.741 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093311+0000", 2024-03-19T11:31:26.741 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033854409999999999 2024-03-19T11:31:26.741 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.741 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.741 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:26.741 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093365+0000", 2024-03-19T11:31:26.741 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3795000000000002e-05 2024-03-19T11:31:26.741 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.741 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.741 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:26.741 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093438+0000", 2024-03-19T11:31:26.741 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2591999999999999e-05 2024-03-19T11:31:26.741 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.741 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.742 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:26.742 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093690+0000", 2024-03-19T11:31:26.742 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025264999999999999 2024-03-19T11:31:26.742 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.742 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.742 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:26.742 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093731+0000", 2024-03-19T11:31:26.742 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0881999999999999e-05 2024-03-19T11:31:26.742 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.742 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.742 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:26.742 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.114610+0000", 2024-03-19T11:31:26.742 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020879264000000002 2024-03-19T11:31:26.743 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.743 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.743 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.743 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.743 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.743 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.21 e81,80 4:8401c1f0:::smithi03829856-43:8,46'66,4:866f3dcc:::smithi03829856-31:2,47'73)", 2024-03-19T11:31:26.743 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:00.093514+0000", 2024-03-19T11:31:26.743 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 26.587430916999999, 2024-03-19T11:31:26.743 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027375713999999999, 2024-03-19T11:31:26.743 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:26.743 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:26.743 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:26.743 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.743 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:26.744 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093514+0000", 2024-03-19T11:31:26.744 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:26.744 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.744 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.744 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:26.744 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093514+0000", 2024-03-19T11:31:26.744 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:26.744 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.744 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.744 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:26.744 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093516+0000", 2024-03-19T11:31:26.744 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1179999999999999e-06 2024-03-19T11:31:26.744 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.744 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.744 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:26.745 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093523+0000", 2024-03-19T11:31:26.745 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7500000000000003e-06 2024-03-19T11:31:26.745 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.745 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.745 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:26.745 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093534+0000", 2024-03-19T11:31:26.745 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0831e-05 2024-03-19T11:31:26.745 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.745 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.745 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:26.745 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093542+0000", 2024-03-19T11:31:26.745 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0139999999999998e-06 2024-03-19T11:31:26.745 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.745 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.746 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:26.746 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.120682+0000", 2024-03-19T11:31:26.746 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027139838999999999 2024-03-19T11:31:26.746 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.746 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.746 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:26.746 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.120693+0000", 2024-03-19T11:31:26.746 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.136e-05 2024-03-19T11:31:26.746 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.746 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.746 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:26.746 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.120890+0000", 2024-03-19T11:31:26.746 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019680199999999999 2024-03-19T11:31:26.746 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.747 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.747 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.747 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.747 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.747 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2927 4.24 4:266590a5:::smithi03829856-16:head [append 0~0] snapc 1b=[1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-19T11:31:26.747 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:18.054474+0000", 2024-03-19T11:31:26.747 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.6264716700000008, 2024-03-19T11:31:26.747 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024856390999999999, 2024-03-19T11:31:26.747 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:26.747 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:26.747 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:26.747 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:26.747 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:26.747 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2927 2024-03-19T11:31:26.747 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.748 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:26.748 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.748 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:26.748 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054474+0000", 2024-03-19T11:31:26.748 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:26.748 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.748 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.748 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:26.748 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054474+0000", 2024-03-19T11:31:26.748 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:26.748 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.748 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.748 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:26.748 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054476+0000", 2024-03-19T11:31:26.749 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6469999999999998e-06 2024-03-19T11:31:26.749 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.749 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.749 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:26.749 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054489+0000", 2024-03-19T11:31:26.749 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2612999999999999e-05 2024-03-19T11:31:26.749 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.749 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.749 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:26.749 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054526+0000", 2024-03-19T11:31:26.749 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7481999999999998e-05 2024-03-19T11:31:26.749 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.749 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.749 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:26.750 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054547+0000", 2024-03-19T11:31:26.750 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0820999999999999e-05 2024-03-19T11:31:26.750 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.750 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.750 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:26.750 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054633+0000", 2024-03-19T11:31:26.750 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5556000000000006e-05 2024-03-19T11:31:26.750 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.750 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.750 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-03-19T11:31:26.750 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054639+0000", 2024-03-19T11:31:26.750 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7300000000000002e-06 2024-03-19T11:31:26.750 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.750 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.751 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:26.751 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.061592+0000", 2024-03-19T11:31:26.751 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0069539279999999998 2024-03-19T11:31:26.751 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.751 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.751 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:26.751 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.061746+0000", 2024-03-19T11:31:26.751 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015351699999999999 2024-03-19T11:31:26.751 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.751 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.751 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:26.751 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.062031+0000", 2024-03-19T11:31:26.751 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00028539700000000001 2024-03-19T11:31:26.751 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.751 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.752 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:26.752 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.079164+0000", 2024-03-19T11:31:26.752 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017132865000000001 2024-03-19T11:31:26.752 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.752 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.752 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:26.752 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.079217+0000", 2024-03-19T11:31:26.752 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2373999999999997e-05 2024-03-19T11:31:26.752 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.752 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.752 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:26.752 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.079284+0000", 2024-03-19T11:31:26.752 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7433000000000003e-05 2024-03-19T11:31:26.752 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.752 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.753 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:26.753 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.079330+0000", 2024-03-19T11:31:26.753 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6028e-05 2024-03-19T11:31:26.753 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.753 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.753 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.753 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.753 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.753 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2928 4.24 4:266590a5:::smithi03829856-16:head [rollback b] snapc 1b=[1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-19T11:31:26.753 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:18.054594+0000", 2024-03-19T11:31:26.753 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.6263516829999993, 2024-03-19T11:31:26.753 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025781921999999999, 2024-03-19T11:31:26.753 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:26.753 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:26.753 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:26.754 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:26.754 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:26.754 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2928 2024-03-19T11:31:26.754 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.754 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:26.754 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.754 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:26.754 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054594+0000", 2024-03-19T11:31:26.754 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:26.754 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.754 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.754 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:26.754 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054594+0000", 2024-03-19T11:31:26.754 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:26.755 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.755 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.755 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:26.755 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054595+0000", 2024-03-19T11:31:26.755 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.018e-06 2024-03-19T11:31:26.755 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.755 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.755 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:26.755 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054601+0000", 2024-03-19T11:31:26.755 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2620000000000004e-06 2024-03-19T11:31:26.755 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.755 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.755 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:26.755 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054618+0000", 2024-03-19T11:31:26.755 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6877000000000001e-05 2024-03-19T11:31:26.756 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.756 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.756 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:26.756 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054632+0000", 2024-03-19T11:31:26.756 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4175000000000001e-05 2024-03-19T11:31:26.756 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.756 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.756 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:26.756 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054676+0000", 2024-03-19T11:31:26.756 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3659000000000003e-05 2024-03-19T11:31:26.756 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.756 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.756 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:31:26.756 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054679+0000", 2024-03-19T11:31:26.756 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0649999999999999e-06 2024-03-19T11:31:26.757 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.757 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.757 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:26.757 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.064021+0000", 2024-03-19T11:31:26.757 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093424210000000001 2024-03-19T11:31:26.757 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.757 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.757 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:26.757 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.064158+0000", 2024-03-19T11:31:26.757 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013652099999999999 2024-03-19T11:31:26.757 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.757 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.757 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:26.757 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.072763+0000", 2024-03-19T11:31:26.758 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.008605784 2024-03-19T11:31:26.758 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.758 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.758 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:26.758 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.079171+0000", 2024-03-19T11:31:26.758 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0064071320000000003 2024-03-19T11:31:26.758 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.758 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.758 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:26.758 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.080257+0000", 2024-03-19T11:31:26.758 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0010866280000000001 2024-03-19T11:31:26.758 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.758 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.758 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:26.758 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.080324+0000", 2024-03-19T11:31:26.759 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6976000000000004e-05 2024-03-19T11:31:26.759 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.759 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.759 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:26.759 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.080376+0000", 2024-03-19T11:31:26.759 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1403999999999998e-05 2024-03-19T11:31:26.759 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.759 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.759 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.759 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.759 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.759 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2929 4.24 4:266590a5:::smithi03829856-16:head [append 0~0] snapc 1b=[1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-19T11:31:26.759 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:18.054649+0000", 2024-03-19T11:31:26.759 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.6262962160000001, 2024-03-19T11:31:26.760 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026690519999999999, 2024-03-19T11:31:26.760 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:26.760 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:26.760 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:26.760 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:26.760 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:26.760 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2929 2024-03-19T11:31:26.760 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.760 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:26.760 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.760 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:26.760 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054649+0000", 2024-03-19T11:31:26.760 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:26.760 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.760 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.761 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:26.761 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054649+0000", 2024-03-19T11:31:26.761 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:26.761 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.761 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.761 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:26.761 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054650+0000", 2024-03-19T11:31:26.761 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5199999999999995e-07 2024-03-19T11:31:26.761 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.761 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.761 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:26.761 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054655+0000", 2024-03-19T11:31:26.761 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1529999999999996e-06 2024-03-19T11:31:26.761 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.762 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.762 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:26.762 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054670+0000", 2024-03-19T11:31:26.762 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5029e-05 2024-03-19T11:31:26.762 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.762 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.762 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:26.762 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054683+0000", 2024-03-19T11:31:26.762 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2670000000000001e-05 2024-03-19T11:31:26.762 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.762 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.762 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:26.762 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054756+0000", 2024-03-19T11:31:26.762 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3559000000000005e-05 2024-03-19T11:31:26.763 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.763 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.763 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:31:26.763 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054760+0000", 2024-03-19T11:31:26.763 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1659999999999998e-06 2024-03-19T11:31:26.763 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.763 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.763 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:26.763 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.075329+0000", 2024-03-19T11:31:26.763 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020569918999999999 2024-03-19T11:31:26.763 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.763 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.763 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:26.763 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.075469+0000", 2024-03-19T11:31:26.763 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013990000000000001 2024-03-19T11:31:26.764 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.764 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.764 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:26.764 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.075730+0000", 2024-03-19T11:31:26.764 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00026011300000000002 2024-03-19T11:31:26.764 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.764 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.764 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:26.764 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.079918+0000", 2024-03-19T11:31:26.764 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0041886279999999998 2024-03-19T11:31:26.764 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.764 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.764 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:26.764 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.081271+0000", 2024-03-19T11:31:26.764 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0013529449999999999 2024-03-19T11:31:26.765 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.765 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.765 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:26.765 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.081308+0000", 2024-03-19T11:31:26.765 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6437999999999999e-05 2024-03-19T11:31:26.765 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:26.765 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:26.765 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:26.765 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.081340+0000", 2024-03-19T11:31:26.765 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2147999999999998e-05 2024-03-19T11:31:26.765 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.765 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.765 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.765 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:26.765 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:26.766 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:26.767 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-19T11:31:26.767 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:26.773+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:26.869 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:26.878+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:26.927 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:26.928 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:31:26.928 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:31:26.928 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:26.940 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-19T11:31:26.969 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:26.978+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:27.056 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:27.056 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:31:27.056 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:31:27.056 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:31:27.056 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:27.070 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-19T11:31:27.071 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:27.077+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:27.171 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:27.177+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:27.183 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-03-19T11:31:27.189 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:27.189+0000 7fa5de8d7640 -1 osd.3 107 handle_osd_map injecting map crc failure 2024-03-19T11:31:27.190 INFO:teuthology.orchestra.run.smithi038.stderr:clear 2.7 pg_upmap_items mapping 2024-03-19T11:31:27.190 INFO:tasks.rados.rados.0.smithi038.stdout:1445: done (0 left) 2024-03-19T11:31:27.190 INFO:tasks.rados.rados.0.smithi038.stdout:1446: tier_evict oid 1 2024-03-19T11:31:27.190 INFO:tasks.rados.rados.0.smithi038.stdout:1446: tier_evict oid 1 snap -1 2024-03-19T11:31:27.190 INFO:tasks.rados.rados.0.smithi038.stdout:1447: read oid 48 snap -1 2024-03-19T11:31:27.190 INFO:tasks.rados.rados.0.smithi038.stdout:1447: expect (ObjNum 246 snap 30 seq_num 246) 2024-03-19T11:31:27.191 INFO:tasks.rados.rados.0.smithi038.stdout:1448: tier_flush oid 25 2024-03-19T11:31:27.191 INFO:tasks.rados.rados.0.smithi038.stdout:1448: tier_flush oid 25 snap -1 2024-03-19T11:31:27.191 INFO:tasks.rados.rados.0.smithi038.stdout:1449: read oid 9 snap -1 2024-03-19T11:31:27.191 INFO:tasks.rados.rados.0.smithi038.stdout:1449: expect (ObjNum 221 snap 24 seq_num 221) 2024-03-19T11:31:27.191 INFO:tasks.rados.rados.0.smithi038.stdout:1450: set_chunk oid 45 target oid 2024-03-19T11:31:27.194 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:27.198+0000 7f923da1c640 -1 osd.7 107 handle_osd_map injecting map crc failure 2024-03-19T11:31:27.242 INFO:tasks.rados.rados.0.smithi038.stdout:1450: set_chunk oid 45 offset: 461312 length: 927232 target oid 3a52856062567f05d8c7df91dad2c51911138f6216e1ec709153bdff536e4abe offset: 0 snap -1 2024-03-19T11:31:27.242 INFO:tasks.rados.rados.0.smithi038.stdout:1451: read oid 20 snap 24 2024-03-19T11:31:27.242 INFO:tasks.rados.rados.0.smithi038.stdout:1451: expect deleted 2024-03-19T11:31:27.242 INFO:tasks.rados.rados.0.smithi038.stdout:1452: read oid 2 snap 29 2024-03-19T11:31:27.242 INFO:tasks.rados.rados.0.smithi038.stdout:1452: expect (ObjNum 206 snap 22 seq_num 206) 2024-03-19T11:31:27.242 INFO:tasks.rados.rados.0.smithi038.stdout:1453: rollback oid 17 current snap is 31 2024-03-19T11:31:27.242 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 17 to 24 2024-03-19T11:31:27.242 INFO:tasks.rados.rados.0.smithi038.stdout:1446: got (22) Invalid argument 2024-03-19T11:31:27.243 INFO:tasks.rados.rados.0.smithi038.stdout:1448: got (16) Device or resource busy 2024-03-19T11:31:27.243 INFO:tasks.rados.rados.0.smithi038.stdout:check_oldest_snap_flushed oid 25 v 60 (ObjNum 97 snap 0 seq_num 0) unflushed 26 2024-03-19T11:31:27.243 INFO:tasks.rados.rados.0.smithi038.stdout:1447: expect (ObjNum 246 snap 30 seq_num 246) 2024-03-19T11:31:27.272 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:27.272 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:31:27.272 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:31:27.272 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:31:27.272 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.272 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.16 49/48 [PushOp(4:6868af91:::smithi03829856-10:2, version: 29'22, data_included: [551263~793594,1893632~254982], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:2@29'22, size: 2151263, copy_subset: [(0, 2151263)], clone_subset: {}, snapset: b=[]:{2: [2], 7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2148614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:6d9093b1:::smithi03829856-38:4, version: 33'29, data_included: [621859~449625,1865716~598951], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6d9093b1:::smithi03829856-38:4@33'29, size: 2621859, copy_subset: [(0, 2621859)], clone_subset: {}, snapset: 4=[]:{4: [2]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2464667, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:6d9093b1:::smithi03829856-38:head, version: 33'33, data_included: [621859~568773,1729599~479803], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:6d9093b1:::smithi03829856-38:head@33'33, size: 3142876, copy_subset: [(0, 3142876)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2209402, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:6be89c06:::smithi03829856-24:head, version: 36'39, data_included: [741376~589824,1753088~458752], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:6be89c06:::smithi03829856-24:head@36'39, size: 2386561, copy_subset: [(0, 2386561)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2211840, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:6868af91:::smithi03829856-10:7, version: 38'41, data_included: [524385~820472,1602039~228104], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:7@38'41, size: 2924385, copy_subset: [(0, 2924385)], clone_subset: {}, snapset: b=[]:{2: [2], 7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1830143, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:6868af91:::smithi03829856-10:b, version: 47'48, data_included: [524385~820472,1602039~228104], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:b@47'48, size: 2924385, copy_subset: [(0, 2924385)], clone_subset: {}, snapset: b=[]:{2: [2], 7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1830143, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:6868af91:::smithi03829856-10:head, version: 47'52, data_included: [524385~820472,1602039~228104], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:head@47'52, size: 3747928, copy_subset: [(0, 3747928)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1830143, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:27.272 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:26.954289+0000", 2024-03-19T11:31:27.273 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 60.326808806000003, 2024-03-19T11:31:27.273 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029260781999999999, 2024-03-19T11:31:27.273 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:27.273 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:27.273 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:27.273 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.273 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:27.273 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954289+0000", 2024-03-19T11:31:27.273 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.273 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.274 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.274 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:27.274 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954289+0000", 2024-03-19T11:31:27.274 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.274 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.274 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.274 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:27.274 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954291+0000", 2024-03-19T11:31:27.274 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8050000000000001e-06 2024-03-19T11:31:27.274 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.275 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.275 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:27.275 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.958775+0000", 2024-03-19T11:31:27.275 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0044845859999999996 2024-03-19T11:31:27.275 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.275 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.275 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:27.275 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.958851+0000", 2024-03-19T11:31:27.275 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6050999999999994e-05 2024-03-19T11:31:27.275 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.276 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.276 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:27.276 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.958931+0000", 2024-03-19T11:31:27.276 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9099999999999998e-05 2024-03-19T11:31:27.276 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.276 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.276 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:27.276 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.959206+0000", 2024-03-19T11:31:27.276 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00027495999999999998 2024-03-19T11:31:27.276 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.276 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.277 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:27.277 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.959246+0000", 2024-03-19T11:31:27.277 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0326e-05 2024-03-19T11:31:27.277 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.277 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.277 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:27.277 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.983550+0000", 2024-03-19T11:31:27.277 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024303953999999999 2024-03-19T11:31:27.277 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.277 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:27.278 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.278 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.278 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.278 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.16 49/48 [PushOp(4:6868af91:::smithi03829856-10:2, version: 29'22, data_included: [2148614~2649], data_size: 2649, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:2@29'22, size: 2151263, copy_subset: [(0, 2151263)], clone_subset: {}, snapset: b=[]:{2: [2], 7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2151263, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2148614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:6d9093b1:::smithi03829856-38:4, version: 33'29, data_included: [2464667~157192], data_size: 157192, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:6d9093b1:::smithi03829856-38:4@33'29, size: 2621859, copy_subset: [(0, 2621859)], clone_subset: {}, snapset: 4=[]:{4: [2]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2621859, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2464667, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:6d9093b1:::smithi03829856-38:head, version: 33'33, data_included: [2209402~412457,3101234~41642], data_size: 454099, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:6d9093b1:::smithi03829856-38:head@33'33, size: 3142876, copy_subset: [(0, 3142876)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3142876, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2209402, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:6be89c06:::smithi03829856-24:head, version: 36'39, data_included: [2211840~51851], data_size: 51851, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:6be89c06:::smithi03829856-24:head@36'39, size: 2386561, copy_subset: [(0, 2386561)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2386561, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2211840, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:6868af91:::smithi03829856-10:7, version: 38'41, data_included: [1830143~321120,2577441~346944], data_size: 668064, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:7@38'41, size: 2924385, copy_subset: [(0, 2924385)], clone_subset: {}, snapset: b=[]:{2: [2], 7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2924385, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1830143, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:6868af91:::smithi03829856-10:b, version: 47'48, data_included: [1830143~321120,2577441~346944], data_size: 668064, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:b@47'48, size: 2924385, copy_subset: [(0, 2924385)], clone_subset: {}, snapset: b=[]:{2: [2], 7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2924385, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1830143, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:6868af91:::smithi03829856-10:head, version: 47'52, data_included: [1830143~348419,2577441~661310], data_size: 1009729, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:head@47'52, size: 3747928, copy_subset: [(0, 3747928)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3747928, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1830143, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:27.278 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:26.998857+0000", 2024-03-19T11:31:27.278 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 60.282241046000003, 2024-03-19T11:31:27.278 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010149703, 2024-03-19T11:31:27.278 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:27.278 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:27.278 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:27.279 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.279 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:27.279 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.998857+0000", 2024-03-19T11:31:27.279 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.279 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.279 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.279 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:27.279 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.998857+0000", 2024-03-19T11:31:27.279 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.280 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.280 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.280 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:27.280 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.998858+0000", 2024-03-19T11:31:27.280 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.065e-06 2024-03-19T11:31:27.280 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.280 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.280 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:27.280 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.999629+0000", 2024-03-19T11:31:27.280 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000771132 2024-03-19T11:31:27.280 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.281 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.281 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:27.281 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.999688+0000", 2024-03-19T11:31:27.281 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9147000000000001e-05 2024-03-19T11:31:27.281 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.281 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.281 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:27.281 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.999755+0000", 2024-03-19T11:31:27.281 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6891999999999996e-05 2024-03-19T11:31:27.281 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.281 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.281 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:27.281 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.000047+0000", 2024-03-19T11:31:27.281 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000292176 2024-03-19T11:31:27.281 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.282 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.282 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:27.282 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.000094+0000", 2024-03-19T11:31:27.282 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6485e-05 2024-03-19T11:31:27.282 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.282 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.282 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:27.282 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.009007+0000", 2024-03-19T11:31:27.282 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0089128060000000005 2024-03-19T11:31:27.282 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.282 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:27.282 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.282 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.282 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.283 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1591 4.4 4:266590a5:::smithi03829856-16:head [write 783226~624677,stat] snapc e=[e,d,c,b,a,6] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-03-19T11:31:27.283 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:32.390032+0000", 2024-03-19T11:31:27.283 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 54.891066260999999, 2024-03-19T11:31:27.283 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0090431220000000007, 2024-03-19T11:31:27.283 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:27.283 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:27.283 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:27.283 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:27.283 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:27.283 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1591 2024-03-19T11:31:27.283 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.283 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:27.283 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.283 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:27.284 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.390032+0000", 2024-03-19T11:31:27.284 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.284 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.284 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.284 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:27.284 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.390032+0000", 2024-03-19T11:31:27.284 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.284 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.284 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.284 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:27.285 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.390033+0000", 2024-03-19T11:31:27.285 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2839999999999999e-06 2024-03-19T11:31:27.285 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.285 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.285 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:27.285 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.390273+0000", 2024-03-19T11:31:27.285 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00024011099999999999 2024-03-19T11:31:27.285 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.285 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.285 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:27.285 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.390286+0000", 2024-03-19T11:31:27.285 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3314e-05 2024-03-19T11:31:27.286 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.286 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.286 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:27.286 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.390295+0000", 2024-03-19T11:31:27.286 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6330000000000002e-06 2024-03-19T11:31:27.286 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.286 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.286 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:27.286 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.390368+0000", 2024-03-19T11:31:27.286 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3065999999999995e-05 2024-03-19T11:31:27.286 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.286 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.287 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:27.287 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.390461+0000", 2024-03-19T11:31:27.287 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2582000000000002e-05 2024-03-19T11:31:27.287 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.287 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.287 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3", 2024-03-19T11:31:27.287 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.390648+0000", 2024-03-19T11:31:27.287 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00018695300000000001 2024-03-19T11:31:27.287 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.287 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.287 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:27.287 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.398909+0000", 2024-03-19T11:31:27.288 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0082613559999999992 2024-03-19T11:31:27.288 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.288 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.288 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:27.288 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.398954+0000", 2024-03-19T11:31:27.288 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5553999999999998e-05 2024-03-19T11:31:27.288 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.288 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.288 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:27.288 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.399017+0000", 2024-03-19T11:31:27.288 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2849999999999996e-05 2024-03-19T11:31:27.288 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.288 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.288 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:27.288 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.399075+0000", 2024-03-19T11:31:27.289 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7419e-05 2024-03-19T11:31:27.289 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.289 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:27.289 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.289 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.289 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.289 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1630 4.d 4:ba44105a:::smithi03829856-26:head [write 467261~717275] snapc 10=[10,f,e,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-03-19T11:31:27.289 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:35.331434+0000", 2024-03-19T11:31:27.289 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 51.949664001000002, 2024-03-19T11:31:27.289 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012902375000000001, 2024-03-19T11:31:27.289 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:27.289 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:27.289 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:27.290 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:27.290 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:27.290 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1630 2024-03-19T11:31:27.290 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.290 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:27.290 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.290 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:27.290 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.331434+0000", 2024-03-19T11:31:27.290 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.290 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.290 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.290 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:27.290 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.331434+0000", 2024-03-19T11:31:27.290 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.291 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.291 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.291 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:27.291 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.331437+0000", 2024-03-19T11:31:27.291 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5599999999999998e-06 2024-03-19T11:31:27.291 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.291 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.291 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:27.291 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.331938+0000", 2024-03-19T11:31:27.291 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00050090099999999999 2024-03-19T11:31:27.291 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.291 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.291 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:27.291 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.331976+0000", 2024-03-19T11:31:27.292 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8090999999999998e-05 2024-03-19T11:31:27.292 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.292 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.292 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:27.292 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.331994+0000", 2024-03-19T11:31:27.292 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7566000000000001e-05 2024-03-19T11:31:27.292 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.292 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.292 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:27.292 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.332086+0000", 2024-03-19T11:31:27.292 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2452000000000002e-05 2024-03-19T11:31:27.292 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.292 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.292 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:27.293 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.332205+0000", 2024-03-19T11:31:27.293 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000118966 2024-03-19T11:31:27.293 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.293 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.293 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1", 2024-03-19T11:31:27.293 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.332410+0000", 2024-03-19T11:31:27.293 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020430099999999999 2024-03-19T11:31:27.293 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.293 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.293 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:27.293 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.339226+0000", 2024-03-19T11:31:27.293 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0068165980000000001 2024-03-19T11:31:27.293 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.293 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.294 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:27.294 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.344191+0000", 2024-03-19T11:31:27.294 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.004965143 2024-03-19T11:31:27.294 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.294 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.294 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:27.294 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.344272+0000", 2024-03-19T11:31:27.294 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0265000000000005e-05 2024-03-19T11:31:27.294 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.295 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.295 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:27.295 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.344336+0000", 2024-03-19T11:31:27.295 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4531999999999998e-05 2024-03-19T11:31:27.295 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.295 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:27.295 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.295 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.296 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.296 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1631 4.d 4:ba44105a:::smithi03829856-26:head [write 1847991~219270] snapc 10=[10,f,e,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-03-19T11:31:27.296 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:35.336760+0000", 2024-03-19T11:31:27.296 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 51.944337398999998, 2024-03-19T11:31:27.296 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0087315580000000004, 2024-03-19T11:31:27.296 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:27.296 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:27.296 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:27.296 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:27.296 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:27.296 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1631 2024-03-19T11:31:27.296 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.297 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:27.297 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.297 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:27.297 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.336760+0000", 2024-03-19T11:31:27.297 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.297 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.297 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.297 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:27.297 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.336760+0000", 2024-03-19T11:31:27.297 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.298 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.298 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.298 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:27.298 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.336762+0000", 2024-03-19T11:31:27.298 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7099999999999999e-06 2024-03-19T11:31:27.298 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.298 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.298 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:27.298 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.336909+0000", 2024-03-19T11:31:27.298 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000147045 2024-03-19T11:31:27.299 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.299 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.299 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:27.299 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.336923+0000", 2024-03-19T11:31:27.299 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3526999999999999e-05 2024-03-19T11:31:27.299 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.299 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.299 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:27.299 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.336943+0000", 2024-03-19T11:31:27.299 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0358999999999999e-05 2024-03-19T11:31:27.299 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.300 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.300 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:27.300 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.337039+0000", 2024-03-19T11:31:27.300 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.5644999999999997e-05 2024-03-19T11:31:27.300 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.300 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.300 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:27.300 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.337155+0000", 2024-03-19T11:31:27.300 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000115913 2024-03-19T11:31:27.300 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.300 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.301 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1", 2024-03-19T11:31:27.301 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.337341+0000", 2024-03-19T11:31:27.301 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000186472 2024-03-19T11:31:27.301 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.301 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.301 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:27.301 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.339232+0000", 2024-03-19T11:31:27.301 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001890954 2024-03-19T11:31:27.301 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.301 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.301 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:27.302 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.345393+0000", 2024-03-19T11:31:27.302 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.006161232 2024-03-19T11:31:27.302 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.302 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.302 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:27.302 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.345449+0000", 2024-03-19T11:31:27.302 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5340000000000002e-05 2024-03-19T11:31:27.302 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.302 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.302 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:27.303 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.345492+0000", 2024-03-19T11:31:27.303 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3361e-05 2024-03-19T11:31:27.303 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.303 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:27.303 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.303 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.303 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.303 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1632 4.d 4:ba44105a:::smithi03829856-26:head [setxattr _header (57) in=7b,truncate 2067261] snapc 10=[10,f,e,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-03-19T11:31:27.303 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:35.336987+0000", 2024-03-19T11:31:27.303 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 51.944111341000003, 2024-03-19T11:31:27.303 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0091098789999999995, 2024-03-19T11:31:27.304 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:27.304 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:27.304 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:27.304 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:27.304 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:27.304 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1632 2024-03-19T11:31:27.304 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.304 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:27.304 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.304 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:27.304 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.336987+0000", 2024-03-19T11:31:27.305 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.305 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.305 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.305 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:27.305 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.336987+0000", 2024-03-19T11:31:27.305 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.305 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.305 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.305 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:27.305 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.336987+0000", 2024-03-19T11:31:27.306 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4600000000000004e-07 2024-03-19T11:31:27.306 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.306 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.306 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:27.306 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.336992+0000", 2024-03-19T11:31:27.306 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6090000000000003e-06 2024-03-19T11:31:27.306 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.306 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.306 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:27.306 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.337002+0000", 2024-03-19T11:31:27.307 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.006e-05 2024-03-19T11:31:27.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.307 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.307 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:27.307 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.337009+0000", 2024-03-19T11:31:27.307 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7229999999999998e-06 2024-03-19T11:31:27.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.307 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.307 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:27.307 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.338513+0000", 2024-03-19T11:31:27.307 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001503868 2024-03-19T11:31:27.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.307 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.307 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:27.308 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.338566+0000", 2024-03-19T11:31:27.308 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3260999999999997e-05 2024-03-19T11:31:27.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.308 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.308 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1", 2024-03-19T11:31:27.308 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.338652+0000", 2024-03-19T11:31:27.308 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5889000000000004e-05 2024-03-19T11:31:27.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.308 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.308 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:27.308 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.340450+0000", 2024-03-19T11:31:27.308 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017987490000000001 2024-03-19T11:31:27.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.308 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.309 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:27.309 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.345982+0000", 2024-03-19T11:31:27.309 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055312620000000003 2024-03-19T11:31:27.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.309 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.309 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:27.309 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.346039+0000", 2024-03-19T11:31:27.309 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6879999999999998e-05 2024-03-19T11:31:27.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.309 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.309 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:27.309 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.346096+0000", 2024-03-19T11:31:27.309 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7831999999999998e-05 2024-03-19T11:31:27.309 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.309 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:27.310 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.310 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.310 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1633 4.d 4:ba44105a:::smithi03829856-26:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-03-19T11:31:27.310 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:35.337032+0000", 2024-03-19T11:31:27.310 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 51.944065621999997, 2024-03-19T11:31:27.310 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093229679999999992, 2024-03-19T11:31:27.310 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:27.310 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:27.310 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:27.310 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:27.310 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:27.310 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1633 2024-03-19T11:31:27.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.311 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:27.311 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.311 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:27.311 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.337032+0000", 2024-03-19T11:31:27.311 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.311 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.311 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:27.311 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.337032+0000", 2024-03-19T11:31:27.311 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.311 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.311 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:27.311 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.337033+0000", 2024-03-19T11:31:27.312 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4899999999999995e-07 2024-03-19T11:31:27.312 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.312 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.312 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:27.312 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.337037+0000", 2024-03-19T11:31:27.312 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7249999999999999e-06 2024-03-19T11:31:27.312 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.312 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.312 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:27.312 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.337045+0000", 2024-03-19T11:31:27.312 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5380000000000002e-06 2024-03-19T11:31:27.312 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.312 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.312 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:27.313 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.337051+0000", 2024-03-19T11:31:27.313 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1689999999999997e-06 2024-03-19T11:31:27.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.313 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.313 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:27.313 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.339128+0000", 2024-03-19T11:31:27.313 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002076919 2024-03-19T11:31:27.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.313 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.313 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-03-19T11:31:27.313 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.339211+0000", 2024-03-19T11:31:27.313 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2896000000000001e-05 2024-03-19T11:31:27.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.313 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.313 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:27.314 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.346152+0000", 2024-03-19T11:31:27.314 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0069404779999999999 2024-03-19T11:31:27.314 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.314 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.314 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:27.314 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.346242+0000", 2024-03-19T11:31:27.314 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0031000000000006e-05 2024-03-19T11:31:27.314 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.314 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.314 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:27.314 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.346355+0000", 2024-03-19T11:31:27.314 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000113663 2024-03-19T11:31:27.314 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.314 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:27.315 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.315 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.315 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.315 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1845 4.4 4:27909373:::smithi03829856-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-03-19T11:31:27.315 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:41.053790+0000", 2024-03-19T11:31:27.315 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 46.227307541000002, 2024-03-19T11:31:27.315 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0091637609999999994, 2024-03-19T11:31:27.315 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:27.315 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:27.315 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:27.315 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:27.315 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:27.315 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1845 2024-03-19T11:31:27.315 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.316 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:27.316 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.316 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:27.316 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053790+0000", 2024-03-19T11:31:27.316 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.316 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.316 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.316 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:27.316 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053790+0000", 2024-03-19T11:31:27.316 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.316 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.316 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.316 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:27.316 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053792+0000", 2024-03-19T11:31:27.316 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2860000000000001e-06 2024-03-19T11:31:27.317 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.317 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.317 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:27.317 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053802+0000", 2024-03-19T11:31:27.317 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.064e-05 2024-03-19T11:31:27.317 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.317 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.317 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:27.317 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053824+0000", 2024-03-19T11:31:27.317 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1611999999999999e-05 2024-03-19T11:31:27.317 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.317 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.317 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:27.317 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053839+0000", 2024-03-19T11:31:27.318 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4922e-05 2024-03-19T11:31:27.318 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.318 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.318 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:27.318 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.059104+0000", 2024-03-19T11:31:27.318 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0052656179999999997 2024-03-19T11:31:27.318 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.318 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.318 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-03-19T11:31:27.318 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.059267+0000", 2024-03-19T11:31:27.318 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000162776 2024-03-19T11:31:27.318 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.318 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.319 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:27.319 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.062551+0000", 2024-03-19T11:31:27.319 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003283467 2024-03-19T11:31:27.319 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.319 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.319 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:27.319 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.062738+0000", 2024-03-19T11:31:27.319 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00018781900000000001 2024-03-19T11:31:27.319 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.319 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.319 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:27.319 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.062954+0000", 2024-03-19T11:31:27.319 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00021562099999999999 2024-03-19T11:31:27.319 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.320 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:27.320 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.320 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.320 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.320 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1848 4.d 4:b65e6658:::smithi03829856-9:6 [sparse-read 0~901298] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-03-19T11:31:27.320 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:41.053966+0000", 2024-03-19T11:31:27.320 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 46.227132087000001, 2024-03-19T11:31:27.320 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0090164259999999993, 2024-03-19T11:31:27.320 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:27.320 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:27.320 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:27.320 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:27.320 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:27.320 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1848 2024-03-19T11:31:27.321 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.321 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:27.321 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.321 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:27.321 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053966+0000", 2024-03-19T11:31:27.321 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.321 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.321 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.321 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:27.321 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053966+0000", 2024-03-19T11:31:27.321 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.321 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.321 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.321 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:27.322 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053967+0000", 2024-03-19T11:31:27.322 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.387e-06 2024-03-19T11:31:27.322 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.322 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.322 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:27.322 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053975+0000", 2024-03-19T11:31:27.322 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5299999999999999e-06 2024-03-19T11:31:27.322 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.322 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.322 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:27.322 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053993+0000", 2024-03-19T11:31:27.322 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7929000000000001e-05 2024-03-19T11:31:27.322 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.322 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.323 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:27.323 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.054010+0000", 2024-03-19T11:31:27.323 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7876e-05 2024-03-19T11:31:27.323 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.323 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.323 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:27.323 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.062407+0000", 2024-03-19T11:31:27.323 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0083966200000000005 2024-03-19T11:31:27.323 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.323 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.323 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:27.323 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.062664+0000", 2024-03-19T11:31:27.323 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025712100000000002 2024-03-19T11:31:27.323 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.324 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.324 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:27.324 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.062982+0000", 2024-03-19T11:31:27.324 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00031796299999999999 2024-03-19T11:31:27.324 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.324 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:27.324 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.324 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.324 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.324 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1889 4.4 4:266590a5:::smithi03829856-16:head [rollback d] snapc 10=[10,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-03-19T11:31:27.324 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:41.697617+0000", 2024-03-19T11:31:27.324 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 45.583481007000003, 2024-03-19T11:31:27.324 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018164059999999999, 2024-03-19T11:31:27.324 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:27.325 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:27.325 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:27.325 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:27.325 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:27.325 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1889 2024-03-19T11:31:27.325 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.325 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:27.325 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.325 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:27.325 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697617+0000", 2024-03-19T11:31:27.325 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.325 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.325 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.325 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:27.326 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697617+0000", 2024-03-19T11:31:27.326 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.326 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.326 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.326 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:27.326 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697619+0000", 2024-03-19T11:31:27.326 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0059999999999999e-06 2024-03-19T11:31:27.326 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.326 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.326 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:27.326 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697631+0000", 2024-03-19T11:31:27.326 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2033e-05 2024-03-19T11:31:27.326 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.326 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.327 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:27.327 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697659+0000", 2024-03-19T11:31:27.327 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8291999999999999e-05 2024-03-19T11:31:27.327 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.327 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.327 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:27.327 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697682+0000", 2024-03-19T11:31:27.327 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3281999999999998e-05 2024-03-19T11:31:27.327 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.327 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.327 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:27.327 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.701122+0000", 2024-03-19T11:31:27.327 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034396679999999999 2024-03-19T11:31:27.327 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.328 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.328 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:27.328 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.701383+0000", 2024-03-19T11:31:27.328 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00026086399999999997 2024-03-19T11:31:27.328 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.328 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.328 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3", 2024-03-19T11:31:27.328 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.709693+0000", 2024-03-19T11:31:27.328 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0083096139999999999 2024-03-19T11:31:27.328 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.328 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.328 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:27.328 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.715437+0000", 2024-03-19T11:31:27.328 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057442379999999996 2024-03-19T11:31:27.328 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.329 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.329 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:27.329 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.715563+0000", 2024-03-19T11:31:27.329 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000125778 2024-03-19T11:31:27.329 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.329 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.329 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:27.329 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.715677+0000", 2024-03-19T11:31:27.329 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000114728 2024-03-19T11:31:27.329 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.329 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.329 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:27.329 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.715781+0000", 2024-03-19T11:31:27.329 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000103557 2024-03-19T11:31:27.330 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.330 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:27.330 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.330 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.330 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.330 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1890 4.4 4:266590a5:::smithi03829856-16:head [append 0~0] snapc 10=[10,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-03-19T11:31:27.330 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:41.697709+0000", 2024-03-19T11:31:27.330 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 45.583388444999997, 2024-03-19T11:31:27.330 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019825171999999999, 2024-03-19T11:31:27.330 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:27.330 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:27.330 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:27.330 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:27.330 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:27.331 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1890 2024-03-19T11:31:27.331 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.331 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:27.331 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.331 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:27.331 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697709+0000", 2024-03-19T11:31:27.331 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.331 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.331 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.331 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:27.331 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697709+0000", 2024-03-19T11:31:27.331 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.331 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.331 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.332 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:27.332 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697711+0000", 2024-03-19T11:31:27.332 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.296e-06 2024-03-19T11:31:27.332 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.332 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.332 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:27.332 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697718+0000", 2024-03-19T11:31:27.332 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.464e-06 2024-03-19T11:31:27.332 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.332 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.332 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:27.332 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697739+0000", 2024-03-19T11:31:27.333 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1005e-05 2024-03-19T11:31:27.333 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.333 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.333 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:27.333 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697754+0000", 2024-03-19T11:31:27.333 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4976e-05 2024-03-19T11:31:27.333 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.333 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.333 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:27.333 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.702341+0000", 2024-03-19T11:31:27.333 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0045870980000000004 2024-03-19T11:31:27.333 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.333 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.334 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for blocked object", 2024-03-19T11:31:27.334 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.702482+0000", 2024-03-19T11:31:27.334 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014048000000000001 2024-03-19T11:31:27.334 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.334 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.334 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:27.334 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.714359+0000", 2024-03-19T11:31:27.334 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011877522999999999 2024-03-19T11:31:27.334 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.334 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.334 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:27.334 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.714550+0000", 2024-03-19T11:31:27.334 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019117 2024-03-19T11:31:27.334 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.335 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.335 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3", 2024-03-19T11:31:27.335 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.714911+0000", 2024-03-19T11:31:27.335 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00036070900000000002 2024-03-19T11:31:27.335 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.335 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.335 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:27.335 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.716211+0000", 2024-03-19T11:31:27.335 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001300329 2024-03-19T11:31:27.335 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.335 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.335 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:27.335 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.717326+0000", 2024-03-19T11:31:27.335 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001114056 2024-03-19T11:31:27.336 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.336 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.336 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:27.336 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.717423+0000", 2024-03-19T11:31:27.336 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7048999999999999e-05 2024-03-19T11:31:27.336 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.336 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.336 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:27.336 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.717535+0000", 2024-03-19T11:31:27.336 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011201699999999999 2024-03-19T11:31:27.336 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.336 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:27.336 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.336 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.336 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.337 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1972 4.4 4:214026b3:::smithi03829856-47:head [tier-promote] snapc 12=[12,11,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:31:27.337 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:45.312422+0000", 2024-03-19T11:31:27.337 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 41.968675771000001, 2024-03-19T11:31:27.337 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013955089, 2024-03-19T11:31:27.337 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:27.337 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:27.337 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:27.337 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:27.337 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:27.337 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1972 2024-03-19T11:31:27.337 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.337 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:27.337 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.337 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:27.338 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.312422+0000", 2024-03-19T11:31:27.338 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.338 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.338 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.338 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:27.338 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.312422+0000", 2024-03-19T11:31:27.338 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.338 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.338 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.338 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:27.338 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.312424+0000", 2024-03-19T11:31:27.338 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9810000000000002e-06 2024-03-19T11:31:27.338 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.338 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.338 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:27.339 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.312433+0000", 2024-03-19T11:31:27.339 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4549999999999995e-06 2024-03-19T11:31:27.339 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.339 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.339 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:27.339 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.312452+0000", 2024-03-19T11:31:27.339 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9380000000000001e-05 2024-03-19T11:31:27.339 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.339 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.339 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:27.339 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.312460+0000", 2024-03-19T11:31:27.339 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2560000000000002e-06 2024-03-19T11:31:27.339 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.339 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.340 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:27.340 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.312555+0000", 2024-03-19T11:31:27.340 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.5208000000000001e-05 2024-03-19T11:31:27.340 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.340 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.340 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:27.340 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.312704+0000", 2024-03-19T11:31:27.340 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014843899999999999 2024-03-19T11:31:27.340 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.340 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.340 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:27.340 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.326377+0000", 2024-03-19T11:31:27.340 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013673370000000001 2024-03-19T11:31:27.340 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.340 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:27.341 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.341 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.341 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.341 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2005 4.4 4:214026b3:::smithi03829856-47:head [read 0~2773122,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:31:27.341 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:45.804325+0000", 2024-03-19T11:31:27.341 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 41.476773311000002, 2024-03-19T11:31:27.341 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010899282, 2024-03-19T11:31:27.341 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:27.341 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:27.341 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:27.341 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:27.341 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:27.341 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2005 2024-03-19T11:31:27.341 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.342 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:27.342 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.342 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:27.342 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804325+0000", 2024-03-19T11:31:27.342 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.342 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.342 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.342 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:27.342 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804325+0000", 2024-03-19T11:31:27.342 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.342 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.342 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.342 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:27.342 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804326+0000", 2024-03-19T11:31:27.343 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.666e-06 2024-03-19T11:31:27.343 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.343 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.343 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:27.343 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804338+0000", 2024-03-19T11:31:27.343 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2096000000000001e-05 2024-03-19T11:31:27.343 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.343 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.343 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:27.343 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804363+0000", 2024-03-19T11:31:27.343 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4762000000000001e-05 2024-03-19T11:31:27.343 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.343 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.343 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:27.343 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804380+0000", 2024-03-19T11:31:27.344 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.713e-05 2024-03-19T11:31:27.344 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.344 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.344 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:27.344 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.812236+0000", 2024-03-19T11:31:27.344 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0078554690000000003 2024-03-19T11:31:27.344 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.344 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.344 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:27.344 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.812541+0000", 2024-03-19T11:31:27.344 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030564299999999997 2024-03-19T11:31:27.344 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.344 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.344 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:27.345 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.815224+0000", 2024-03-19T11:31:27.345 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0026825159999999998 2024-03-19T11:31:27.345 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.345 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:27.345 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.345 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.345 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.345 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2006 4.4 4:214026b3:::smithi03829856-47:head [read 0~2773122,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:31:27.345 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:45.804421+0000", 2024-03-19T11:31:27.345 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 41.476676572000002, 2024-03-19T11:31:27.345 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013366463, 2024-03-19T11:31:27.345 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:27.345 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:27.345 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:27.346 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:27.346 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:27.346 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2006 2024-03-19T11:31:27.346 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.346 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:27.346 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.346 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:27.346 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804421+0000", 2024-03-19T11:31:27.346 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.346 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.346 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.346 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:27.346 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804421+0000", 2024-03-19T11:31:27.346 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.347 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.347 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.347 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:27.347 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804423+0000", 2024-03-19T11:31:27.347 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5600000000000001e-06 2024-03-19T11:31:27.347 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.347 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.347 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:27.347 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804433+0000", 2024-03-19T11:31:27.347 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8719999999999997e-06 2024-03-19T11:31:27.347 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.347 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.347 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:27.347 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804454+0000", 2024-03-19T11:31:27.348 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1759e-05 2024-03-19T11:31:27.348 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.348 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.348 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:27.348 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804470+0000", 2024-03-19T11:31:27.348 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5556000000000001e-05 2024-03-19T11:31:27.348 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.348 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.348 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:27.349 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.815325+0000", 2024-03-19T11:31:27.349 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010855069 2024-03-19T11:31:27.349 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.349 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.349 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:27.349 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.815684+0000", 2024-03-19T11:31:27.349 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00035912699999999999 2024-03-19T11:31:27.349 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.349 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.349 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:27.349 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.817788+0000", 2024-03-19T11:31:27.349 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0021035200000000002 2024-03-19T11:31:27.349 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.349 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:27.350 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.350 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.350 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.350 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(osd.1.0:198 4.16 e67/48 4:6868af91:::smithi03829856-10:head v 67'79, mlcod=66'76)", 2024-03-19T11:31:27.350 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:45.811039+0000", 2024-03-19T11:31:27.350 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 41.470059210999999, 2024-03-19T11:31:27.350 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01087364, 2024-03-19T11:31:27.350 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:27.350 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:27.350 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:27.350 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.350 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:27.350 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.811039+0000", 2024-03-19T11:31:27.350 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.351 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.351 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:27.351 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.811039+0000", 2024-03-19T11:31:27.351 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.351 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.351 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:27.351 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.811041+0000", 2024-03-19T11:31:27.351 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6400000000000001e-06 2024-03-19T11:31:27.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.351 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.351 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:27.351 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.812874+0000", 2024-03-19T11:31:27.352 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0018329959999999999 2024-03-19T11:31:27.352 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.352 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.352 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:27.352 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.812916+0000", 2024-03-19T11:31:27.352 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1875000000000001e-05 2024-03-19T11:31:27.352 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.352 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.352 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:27.352 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.812937+0000", 2024-03-19T11:31:27.352 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0985e-05 2024-03-19T11:31:27.352 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.352 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.352 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:27.353 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.813112+0000", 2024-03-19T11:31:27.353 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000174573 2024-03-19T11:31:27.353 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.353 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.353 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:27.353 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.813170+0000", 2024-03-19T11:31:27.353 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8100999999999998e-05 2024-03-19T11:31:27.353 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.353 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.353 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:27.353 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.821792+0000", 2024-03-19T11:31:27.353 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0086218549999999994 2024-03-19T11:31:27.353 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.353 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.354 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:27.354 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.821912+0000", 2024-03-19T11:31:27.354 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000120615 2024-03-19T11:31:27.354 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.354 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:27.354 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.354 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.354 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.354 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2027 4.2d 4:b4ef97e0:::smithi03829856-3:head [read 0~3697507,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:31:27.354 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:46.264908+0000", 2024-03-19T11:31:27.354 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 41.016190344999998, 2024-03-19T11:31:27.354 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010951414, 2024-03-19T11:31:27.354 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:27.355 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:27.355 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:27.355 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:27.355 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:27.355 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2027 2024-03-19T11:31:27.355 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.355 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:27.355 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.355 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:27.355 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264908+0000", 2024-03-19T11:31:27.355 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.355 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.355 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.355 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:27.356 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264908+0000", 2024-03-19T11:31:27.356 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.356 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.356 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.356 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:27.356 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264908+0000", 2024-03-19T11:31:27.356 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5799999999999999e-07 2024-03-19T11:31:27.356 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.356 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.356 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:27.356 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264912+0000", 2024-03-19T11:31:27.356 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2649999999999998e-06 2024-03-19T11:31:27.356 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.356 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.356 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:27.357 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264920+0000", 2024-03-19T11:31:27.357 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.79e-06 2024-03-19T11:31:27.357 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.357 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.357 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:27.357 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264926+0000", 2024-03-19T11:31:27.357 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9989999999999997e-06 2024-03-19T11:31:27.357 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.357 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.357 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:27.357 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.273461+0000", 2024-03-19T11:31:27.357 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0085352680000000004 2024-03-19T11:31:27.357 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.357 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.358 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:27.358 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.273598+0000", 2024-03-19T11:31:27.358 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013622299999999999 2024-03-19T11:31:27.358 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.358 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.358 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:27.358 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.275859+0000", 2024-03-19T11:31:27.358 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022613109999999998 2024-03-19T11:31:27.358 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.358 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:27.358 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.358 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.358 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.358 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2028 4.2d 4:b4ef97e0:::smithi03829856-3:head [read 0~3697507,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:31:27.359 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:46.264934+0000", 2024-03-19T11:31:27.359 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 41.016163714000001, 2024-03-19T11:31:27.359 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013708481, 2024-03-19T11:31:27.359 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:27.359 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:27.359 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:27.359 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:27.359 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:27.359 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2028 2024-03-19T11:31:27.359 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.359 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:27.359 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.359 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:27.359 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264934+0000", 2024-03-19T11:31:27.360 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.360 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.360 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.360 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:27.360 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264934+0000", 2024-03-19T11:31:27.360 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.360 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.360 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.360 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:27.360 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264935+0000", 2024-03-19T11:31:27.360 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2599999999999998e-07 2024-03-19T11:31:27.360 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.360 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.361 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:27.361 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264938+0000", 2024-03-19T11:31:27.361 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.163e-06 2024-03-19T11:31:27.361 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.361 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.361 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:27.361 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264945+0000", 2024-03-19T11:31:27.361 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.0369999999999997e-06 2024-03-19T11:31:27.361 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.361 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.361 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:27.361 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264950+0000", 2024-03-19T11:31:27.361 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1170000000000003e-06 2024-03-19T11:31:27.361 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.362 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.362 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:27.362 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.275914+0000", 2024-03-19T11:31:27.362 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010964150000000001 2024-03-19T11:31:27.362 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.362 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.362 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:27.362 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.276037+0000", 2024-03-19T11:31:27.362 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012250100000000001 2024-03-19T11:31:27.362 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.363 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.363 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:27.363 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.278643+0000", 2024-03-19T11:31:27.363 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002606087 2024-03-19T11:31:27.363 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.363 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:27.363 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.363 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.363 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.363 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2045 4.4 4:21eac9b2:::smithi03829856-4:head [read 0~2537857,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:31:27.363 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:46.265120+0000", 2024-03-19T11:31:27.363 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 41.015977405999998, 2024-03-19T11:31:27.364 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0096591980000000008, 2024-03-19T11:31:27.364 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:27.364 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:27.364 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:27.364 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:27.364 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:27.364 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2045 2024-03-19T11:31:27.364 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.364 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:27.364 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.364 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:27.364 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265120+0000", 2024-03-19T11:31:27.365 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.365 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.365 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.365 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:27.365 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265120+0000", 2024-03-19T11:31:27.365 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.365 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.365 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.365 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:27.365 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265121+0000", 2024-03-19T11:31:27.365 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2399999999999998e-07 2024-03-19T11:31:27.365 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.365 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.365 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:27.365 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265126+0000", 2024-03-19T11:31:27.366 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4870000000000002e-06 2024-03-19T11:31:27.366 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.366 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.366 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:27.366 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265134+0000", 2024-03-19T11:31:27.366 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9740000000000002e-06 2024-03-19T11:31:27.366 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.366 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.366 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:27.366 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265143+0000", 2024-03-19T11:31:27.366 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7530000000000007e-06 2024-03-19T11:31:27.366 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.366 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.367 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:27.367 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.273765+0000", 2024-03-19T11:31:27.367 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0086213880000000007 2024-03-19T11:31:27.367 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.367 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.367 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:27.367 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.273872+0000", 2024-03-19T11:31:27.367 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000107452 2024-03-19T11:31:27.367 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.367 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.367 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:27.367 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.274780+0000", 2024-03-19T11:31:27.367 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00090762000000000004 2024-03-19T11:31:27.367 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.368 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:27.368 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.368 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.368 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.368 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2046 4.4 4:21eac9b2:::smithi03829856-4:head [read 0~2537857,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:31:27.368 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:46.265152+0000", 2024-03-19T11:31:27.368 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 41.015946047, 2024-03-19T11:31:27.368 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013074417, 2024-03-19T11:31:27.368 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:27.368 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:27.368 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:27.368 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:27.368 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:27.368 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2046 2024-03-19T11:31:27.369 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.369 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:27.369 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.369 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:27.369 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265152+0000", 2024-03-19T11:31:27.369 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.369 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.369 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.369 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:27.369 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265152+0000", 2024-03-19T11:31:27.369 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.369 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.369 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.369 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:27.370 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265152+0000", 2024-03-19T11:31:27.370 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8999999999999997e-07 2024-03-19T11:31:27.370 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.370 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.370 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:27.370 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265155+0000", 2024-03-19T11:31:27.370 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0920000000000002e-06 2024-03-19T11:31:27.370 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.370 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.370 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:27.370 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265162+0000", 2024-03-19T11:31:27.370 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6730000000000003e-06 2024-03-19T11:31:27.370 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.370 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.371 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:27.371 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265167+0000", 2024-03-19T11:31:27.371 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1839999999999998e-06 2024-03-19T11:31:27.371 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.371 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.371 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:27.371 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.276218+0000", 2024-03-19T11:31:27.371 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011050559999999999 2024-03-19T11:31:27.371 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.371 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.371 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:27.371 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.276345+0000", 2024-03-19T11:31:27.371 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000127243 2024-03-19T11:31:27.371 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.372 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.372 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:27.372 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.278226+0000", 2024-03-19T11:31:27.372 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0018811749999999999 2024-03-19T11:31:27.372 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.372 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:27.372 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.372 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.372 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.372 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:3046 4.19 e103/73 4:9c74b127:::smithi03829856-33:head v 103'148, mlcod=99'144)", 2024-03-19T11:31:27.372 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:22.149246+0000", 2024-03-19T11:31:27.372 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.1318519790000003, 2024-03-19T11:31:27.372 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014101367, 2024-03-19T11:31:27.372 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:27.373 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:27.373 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:27.373 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.373 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:27.373 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.149246+0000", 2024-03-19T11:31:27.373 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.373 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.373 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.373 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:27.373 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.149246+0000", 2024-03-19T11:31:27.373 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.373 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.373 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.373 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:27.373 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.149246+0000", 2024-03-19T11:31:27.374 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2300000000000002e-07 2024-03-19T11:31:27.374 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.374 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.374 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:27.374 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.149663+0000", 2024-03-19T11:31:27.374 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00041629599999999997 2024-03-19T11:31:27.374 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.374 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.374 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:27.374 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.149682+0000", 2024-03-19T11:31:27.374 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9500999999999998e-05 2024-03-19T11:31:27.374 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.374 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.375 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:27.375 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.149692+0000", 2024-03-19T11:31:27.375 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.825e-06 2024-03-19T11:31:27.375 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.375 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.375 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:27.375 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.149797+0000", 2024-03-19T11:31:27.375 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000105222 2024-03-19T11:31:27.375 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.375 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.375 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:27.375 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.149835+0000", 2024-03-19T11:31:27.375 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7934e-05 2024-03-19T11:31:27.375 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.375 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.376 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:27.376 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.163304+0000", 2024-03-19T11:31:27.376 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013469122 2024-03-19T11:31:27.376 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.376 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.376 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:27.376 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.163347+0000", 2024-03-19T11:31:27.376 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3143999999999999e-05 2024-03-19T11:31:27.376 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.376 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:27.376 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.376 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.376 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:27.376 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:27.377 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:27.277+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:27.378 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-19T11:31:27.378 INFO:tasks.rados.rados.0.smithi038.stdout:1449: expect (ObjNum 221 snap 24 seq_num 221) 2024-03-19T11:31:27.380 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:27.390+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:27.481 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:27.490+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:27.544 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:27.544 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:31:27.544 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:31:27.544 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:27.558 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-19T11:31:27.582 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:27.590+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:27.611 INFO:tasks.rados.rados.0.smithi038.stdout:1453: finishing rollback tid 0 to smithi03829856-17 2024-03-19T11:31:27.611 INFO:tasks.rados.rados.0.smithi038.stdout:1453: finishing rollback tid 1 to smithi03829856-17 2024-03-19T11:31:27.611 INFO:tasks.rados.rados.0.smithi038.stdout:1453: finishing rollback tid 2 to smithi03829856-17 2024-03-19T11:31:27.611 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 17 v 237 (ObjNum 213 snap 22 seq_num 213) dirty exists 2024-03-19T11:31:27.612 INFO:tasks.rados.rados.0.smithi038.stdout:1450: finishing set_chunk to oid 45 2024-03-19T11:31:27.612 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 45 v 70 (ObjNum 212 snap 22 seq_num 212) dirty exists 2024-03-19T11:31:27.612 INFO:tasks.rados.rados.0.smithi038.stdout:1446: done (7 left) 2024-03-19T11:31:27.612 INFO:tasks.rados.rados.0.smithi038.stdout:1447: done (6 left) 2024-03-19T11:31:27.612 INFO:tasks.rados.rados.0.smithi038.stdout:1448: done (5 left) 2024-03-19T11:31:27.612 INFO:tasks.rados.rados.0.smithi038.stdout:1449: done (4 left) 2024-03-19T11:31:27.612 INFO:tasks.rados.rados.0.smithi038.stdout:1450: done (3 left) 2024-03-19T11:31:27.612 INFO:tasks.rados.rados.0.smithi038.stdout:1451: done (2 left) 2024-03-19T11:31:27.612 INFO:tasks.rados.rados.0.smithi038.stdout:1453: done (1 left) 2024-03-19T11:31:27.612 INFO:tasks.rados.rados.0.smithi038.stdout:1454: delete oid 35 current snap is 31 2024-03-19T11:31:27.620 INFO:tasks.rados.rados.0.smithi038.stdout:1454: done (1 left) 2024-03-19T11:31:27.620 INFO:tasks.rados.rados.0.smithi038.stdout:1455: read oid 41 snap 26 2024-03-19T11:31:27.620 INFO:tasks.rados.rados.0.smithi038.stdout:1455: expect (ObjNum 170 snap 17 seq_num 170) 2024-03-19T11:31:27.620 INFO:tasks.rados.rados.0.smithi038.stdout:1456: set_chunk oid 29 target oid 2024-03-19T11:31:27.633 INFO:tasks.rados.rados.0.smithi038.stdout:1456: set_chunk oid 29 offset: 511488 length: 733696 target oid 0dbb3b65e0815d2ccfda2b3d25ac92d0baa8887cebbb957efa825c52d1a71472 offset: 0 snap 26 2024-03-19T11:31:27.633 INFO:tasks.rados.rados.0.smithi038.stdout:1457: read oid 20 snap -1 2024-03-19T11:31:27.633 INFO:tasks.rados.rados.0.smithi038.stdout:1457: expect deleted 2024-03-19T11:31:27.633 INFO:tasks.rados.rados.0.smithi038.stdout:1458: snap_create 2024-03-19T11:31:27.633 INFO:tasks.rados.rados.0.smithi038.stdout:1455: expect (ObjNum 170 snap 17 seq_num 170) 2024-03-19T11:31:27.677 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:27.678 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:31:27.678 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:31:27.678 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:31:27.678 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:27.682 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:27.690+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:27.692 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-19T11:31:27.726 INFO:tasks.rados.rados.0.smithi038.stdout:1456: finishing set_chunk to oid 29 2024-03-19T11:31:27.783 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:27.788+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:27.814 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:27.814 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:31:27.814 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:31:27.814 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:31:27.814 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.814 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:58 3.2 3:5f1d333e:::bfcdf551f6f2543327e9c601a4377e9a5ef396f5b97ae8331892d1bcf6b55a9d:head [read 0~161292 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:31:27.815 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.762024+0000", 2024-03-19T11:31:27.815 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 75.059275080999996, 2024-03-19T11:31:27.815 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013495629, 2024-03-19T11:31:27.815 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:27.815 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:27.815 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:27.815 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.815 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:27.815 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762024+0000", 2024-03-19T11:31:27.815 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.816 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.816 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.816 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:27.816 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762024+0000", 2024-03-19T11:31:27.816 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.816 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.816 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.816 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:27.816 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762024+0000", 2024-03-19T11:31:27.817 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9999999999999997e-07 2024-03-19T11:31:27.817 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.817 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.817 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:27.817 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762027+0000", 2024-03-19T11:31:27.817 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0410000000000002e-06 2024-03-19T11:31:27.817 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.818 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.818 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:27.818 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762037+0000", 2024-03-19T11:31:27.818 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1020000000000003e-06 2024-03-19T11:31:27.818 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.818 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.818 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:27.818 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762043+0000", 2024-03-19T11:31:27.818 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4409999999999998e-06 2024-03-19T11:31:27.818 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.818 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.818 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:27.818 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762198+0000", 2024-03-19T11:31:27.819 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015540000000000001 2024-03-19T11:31:27.819 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.819 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.819 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:31:27.819 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762203+0000", 2024-03-19T11:31:27.819 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5109999999999997e-06 2024-03-19T11:31:27.819 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.819 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.819 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:27.819 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.774710+0000", 2024-03-19T11:31:27.819 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012507452 2024-03-19T11:31:27.819 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.819 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.819 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:27.819 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.774754+0000", 2024-03-19T11:31:27.820 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3869999999999998e-05 2024-03-19T11:31:27.820 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.820 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.820 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:27.820 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.775519+0000", 2024-03-19T11:31:27.820 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00076521200000000003 2024-03-19T11:31:27.820 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.820 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:27.820 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.820 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.820 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.820 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:59 3.2 3:78b4b14e:::e1ef0c0c594c36c534f3829406e07ffb60d57b70360f62977e0d545d8ca2bf18:head [read 0~149977 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:31:27.820 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.762054+0000", 2024-03-19T11:31:27.820 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 75.059244649999997, 2024-03-19T11:31:27.821 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015354483, 2024-03-19T11:31:27.821 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:27.821 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:27.821 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:27.821 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.821 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:27.821 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762054+0000", 2024-03-19T11:31:27.821 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.821 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.821 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.821 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:27.821 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762054+0000", 2024-03-19T11:31:27.821 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.821 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.821 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.822 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:27.822 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762055+0000", 2024-03-19T11:31:27.822 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.32e-07 2024-03-19T11:31:27.822 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.822 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.822 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:27.822 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762058+0000", 2024-03-19T11:31:27.822 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9189999999999999e-06 2024-03-19T11:31:27.822 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.822 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.822 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:27.822 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762065+0000", 2024-03-19T11:31:27.822 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2409999999999998e-06 2024-03-19T11:31:27.822 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.823 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.823 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:27.823 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762070+0000", 2024-03-19T11:31:27.823 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6910000000000002e-06 2024-03-19T11:31:27.823 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.823 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.823 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:27.823 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762227+0000", 2024-03-19T11:31:27.823 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015606999999999999 2024-03-19T11:31:27.823 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.823 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.824 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:31:27.824 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762232+0000", 2024-03-19T11:31:27.824 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3029999999999999e-06 2024-03-19T11:31:27.824 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.824 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.824 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:27.824 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.776984+0000", 2024-03-19T11:31:27.824 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014752487999999999 2024-03-19T11:31:27.824 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.824 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.824 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:27.824 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.777030+0000", 2024-03-19T11:31:27.825 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5359999999999999e-05 2024-03-19T11:31:27.825 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.825 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.825 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:27.825 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.777409+0000", 2024-03-19T11:31:27.825 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037897899999999999 2024-03-19T11:31:27.825 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.825 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:27.825 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.825 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.825 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.825 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1067 4.0 4:01018511:::smithi03829856-19:head [write 477332~766101] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:31:27.826 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.950506+0000", 2024-03-19T11:31:27.826 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 73.870792937999994, 2024-03-19T11:31:27.826 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016279426, 2024-03-19T11:31:27.826 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:27.826 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:27.826 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:27.826 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:27.826 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:27.826 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1067 2024-03-19T11:31:27.826 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.826 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:27.826 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.826 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:27.826 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950506+0000", 2024-03-19T11:31:27.827 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.827 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.827 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.827 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:27.827 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950506+0000", 2024-03-19T11:31:27.827 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.827 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.827 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.827 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:27.827 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950509+0000", 2024-03-19T11:31:27.827 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.021e-06 2024-03-19T11:31:27.827 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.827 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.828 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:27.828 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950892+0000", 2024-03-19T11:31:27.828 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00038263300000000001 2024-03-19T11:31:27.828 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.828 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.828 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:27.828 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950912+0000", 2024-03-19T11:31:27.828 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0539000000000001e-05 2024-03-19T11:31:27.828 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.828 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.828 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:27.828 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950922+0000", 2024-03-19T11:31:27.828 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.577e-06 2024-03-19T11:31:27.828 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.829 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.829 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:27.829 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950982+0000", 2024-03-19T11:31:27.829 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0760000000000001e-05 2024-03-19T11:31:27.829 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.829 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.829 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:27.829 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.951058+0000", 2024-03-19T11:31:27.829 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5078000000000002e-05 2024-03-19T11:31:27.829 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.829 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.829 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:27.829 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.951173+0000", 2024-03-19T11:31:27.829 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011516200000000001 2024-03-19T11:31:27.830 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.830 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.830 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:27.830 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.955064+0000", 2024-03-19T11:31:27.830 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003890896 2024-03-19T11:31:27.830 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.830 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.830 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:27.830 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966670+0000", 2024-03-19T11:31:27.830 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011606795 2024-03-19T11:31:27.830 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.830 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.830 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:27.830 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966731+0000", 2024-03-19T11:31:27.831 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0528000000000001e-05 2024-03-19T11:31:27.831 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.831 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.831 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:27.831 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966785+0000", 2024-03-19T11:31:27.831 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4437000000000001e-05 2024-03-19T11:31:27.831 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.831 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:27.831 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.831 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.831 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.831 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1231 4.0 4:0dc3b91e:::smithi03829856-28:head [write 1808051~636013] snapc 9=[9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:31:27.831 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.173016+0000", 2024-03-19T11:31:27.831 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 69.648282824999995, 2024-03-19T11:31:27.832 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013973719000000001, 2024-03-19T11:31:27.832 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:27.832 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:27.832 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:27.832 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:27.832 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:27.832 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1231 2024-03-19T11:31:27.832 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.832 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:27.832 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.832 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:27.832 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173016+0000", 2024-03-19T11:31:27.832 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.832 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.832 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.833 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:27.833 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173016+0000", 2024-03-19T11:31:27.833 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.833 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.833 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:27.833 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173019+0000", 2024-03-19T11:31:27.833 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.253e-06 2024-03-19T11:31:27.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.833 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.833 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:27.833 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176143+0000", 2024-03-19T11:31:27.833 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031240930000000001 2024-03-19T11:31:27.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.834 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.834 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:27.834 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176177+0000", 2024-03-19T11:31:27.834 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3237e-05 2024-03-19T11:31:27.834 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.834 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.834 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:27.834 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176196+0000", 2024-03-19T11:31:27.834 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9358e-05 2024-03-19T11:31:27.834 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.834 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.834 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:27.834 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176493+0000", 2024-03-19T11:31:27.834 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00029743899999999998 2024-03-19T11:31:27.835 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.835 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.835 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:27.835 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176786+0000", 2024-03-19T11:31:27.835 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00029259000000000002 2024-03-19T11:31:27.835 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.835 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.835 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:27.835 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177250+0000", 2024-03-19T11:31:27.835 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00046376000000000002 2024-03-19T11:31:27.835 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.835 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.835 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:27.835 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.182511+0000", 2024-03-19T11:31:27.835 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0052609019999999996 2024-03-19T11:31:27.836 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.836 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.836 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:27.836 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.186896+0000", 2024-03-19T11:31:27.836 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0043857009999999997 2024-03-19T11:31:27.836 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.836 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.836 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:27.836 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.186948+0000", 2024-03-19T11:31:27.836 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1653000000000002e-05 2024-03-19T11:31:27.836 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.836 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.836 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:27.836 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.186990+0000", 2024-03-19T11:31:27.836 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1733000000000002e-05 2024-03-19T11:31:27.837 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.837 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:27.837 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.837 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.837 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.837 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1409 4.10 4:0dc3b91e:::smithi03829856-28:head [sparse-read 0~1851556] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:31:27.837 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.918248+0000", 2024-03-19T11:31:27.837 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 61.903050536000002, 2024-03-19T11:31:27.837 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013726986, 2024-03-19T11:31:27.837 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:27.837 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:27.837 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:27.837 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:27.837 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:27.838 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1409 2024-03-19T11:31:27.838 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.838 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:27.838 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.838 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:27.838 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918248+0000", 2024-03-19T11:31:27.838 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.838 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.838 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.838 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:27.838 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918248+0000", 2024-03-19T11:31:27.838 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.838 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.838 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.839 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:27.839 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918249+0000", 2024-03-19T11:31:27.839 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4700000000000002e-07 2024-03-19T11:31:27.839 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.839 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.839 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:27.839 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918251+0000", 2024-03-19T11:31:27.839 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.542e-06 2024-03-19T11:31:27.839 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.839 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.839 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:27.839 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918258+0000", 2024-03-19T11:31:27.839 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4629999999999998e-06 2024-03-19T11:31:27.839 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.839 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.840 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:27.840 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918263+0000", 2024-03-19T11:31:27.840 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8049999999999997e-06 2024-03-19T11:31:27.840 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.840 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.840 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:27.840 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.931469+0000", 2024-03-19T11:31:27.840 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013206458000000001 2024-03-19T11:31:27.840 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.840 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.840 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:27.840 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.931586+0000", 2024-03-19T11:31:27.840 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000116655 2024-03-19T11:31:27.840 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.840 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.841 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:27.841 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.931975+0000", 2024-03-19T11:31:27.841 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00038961600000000002 2024-03-19T11:31:27.841 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.841 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:27.841 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.841 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.841 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.841 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(3.1 49/48 [PushOp(3:9205291d:::smithi03829856-50 low_tier:head, version: 28'36, data_included: [393216~851968,1703936~196608], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:9205291d:::smithi03829856-50 low_tier:head@28'36, size: 2823543, copy_subset: [(0, 2823543)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1900544, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:baab691a:::smithi03829856-2 low_tier:head, version: 28'37, data_included: [393216~524288,1507328~509479], data_size: 1033767, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:baab691a:::smithi03829856-2 low_tier:head@28'37, size: 2016807, copy_subset: [(0, 2016807)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2016807, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:98ce8426:::smithi03829856-4 low_tier:head, version: 28'38, data_included: [393216~720896,1638400~327680], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:98ce8426:::smithi03829856-4 low_tier:head@28'38, size: 2050421, copy_subset: [(0, 2050421)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1966080, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:a86be07b:::smithi03829856-44 low_tier:head, version: 28'41, data_included: [720896~655360,1769472~393216], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:a86be07b:::smithi03829856-44 low_tier:head@28'41, size: 2722701, copy_subset: [(0, 2722701)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2162688, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:bb56919e:::smithi03829856-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'44, data_included: [393216~786432,1703936~262144], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:bb56919e:::smithi03829856-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'44, size: 2806736, copy_subset: [(0, 2806736)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1966080, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:88687f71:::smithi03829856-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'45, data_included: [589824~655360,1638400~393216], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:88687f71:::smithi03829856-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'45, size: 2201684, copy_subset: [(0, 2201684)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2031616, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:b1f822cf:::smithi03829856-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'46, data_included: [720896~786432,1966080~262144], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:b1f822cf:::smithi03829856-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'46, size: 2369754, copy_subset: [(0, 2369754)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2228224, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:a5e8313a:::smithi03829856-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'47, data_included: [524288~720896,1769472~327680], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:a5e8313a:::smithi03829856-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'47, size: 2571438, copy_subset: [(0, 2571438)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:bca5ec64:::e3a26bf9230b1b95b8cf1a4cb01909236bf2bf3fd59b083b2291d278ab98264d:head, version: 28'48, data_included: [0~393216], data_size: 393216, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bca5ec64:::e3a26bf9230b1b95b8cf1a4cb01909236bf2bf3fd59b083b2291d278ab98264d:head@28'48, size: 533504, copy_subset: [(0, 533504)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 533504, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:27.841 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:26.948137+0000", 2024-03-19T11:31:27.841 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 60.873161723000003, 2024-03-19T11:31:27.841 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.036085578, 2024-03-19T11:31:27.842 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:27.842 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:27.842 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:27.842 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.842 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:27.842 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.948137+0000", 2024-03-19T11:31:27.842 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.842 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.842 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.842 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:27.843 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.948137+0000", 2024-03-19T11:31:27.843 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.843 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.843 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.843 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:27.843 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.948139+0000", 2024-03-19T11:31:27.843 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1399999999999998e-06 2024-03-19T11:31:27.843 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.843 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.843 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:27.843 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954288+0000", 2024-03-19T11:31:27.843 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0061487429999999999 2024-03-19T11:31:27.843 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.843 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.844 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:27.844 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954388+0000", 2024-03-19T11:31:27.844 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010022 2024-03-19T11:31:27.844 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.844 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.844 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:27.844 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954523+0000", 2024-03-19T11:31:27.844 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000134332 2024-03-19T11:31:27.844 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.844 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.844 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:27.844 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.955120+0000", 2024-03-19T11:31:27.844 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00059695799999999997 2024-03-19T11:31:27.844 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.844 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.845 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:27.845 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.955194+0000", 2024-03-19T11:31:27.845 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4077000000000002e-05 2024-03-19T11:31:27.845 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.845 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.845 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:27.845 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.984223+0000", 2024-03-19T11:31:27.845 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029029108000000001 2024-03-19T11:31:27.845 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.845 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:27.845 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.845 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.845 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.845 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(3.1 49/48 [PushOp(3:99a49d1b:::895a63407b6109b3cca9a6f5ef827134d56679bef42daadc5892269682c428ed:head, version: 32'49, data_included: [393216~388096], data_size: 388096, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:99a49d1b:::895a63407b6109b3cca9a6f5ef827134d56679bef42daadc5892269682c428ed:head@32'49, size: 781312, copy_subset: [(0, 781312)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 781312, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:27.846 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:26.954637+0000", 2024-03-19T11:31:27.846 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 60.866662106, 2024-03-19T11:31:27.846 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.030729057000000001, 2024-03-19T11:31:27.846 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:27.846 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:27.846 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:27.846 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.846 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:27.846 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954637+0000", 2024-03-19T11:31:27.846 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.846 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.846 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.846 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:27.847 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954637+0000", 2024-03-19T11:31:27.847 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.847 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.847 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.847 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:27.847 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954637+0000", 2024-03-19T11:31:27.847 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7299999999999995e-07 2024-03-19T11:31:27.847 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.847 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.847 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:27.847 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954873+0000", 2024-03-19T11:31:27.847 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023520299999999999 2024-03-19T11:31:27.847 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.847 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.848 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:27.848 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954897+0000", 2024-03-19T11:31:27.848 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4110000000000001e-05 2024-03-19T11:31:27.848 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.848 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.848 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:27.848 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954917+0000", 2024-03-19T11:31:27.848 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.067e-05 2024-03-19T11:31:27.848 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.848 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.848 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:27.848 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.984289+0000", 2024-03-19T11:31:27.848 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029371191000000001 2024-03-19T11:31:27.848 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.849 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.849 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:27.849 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.984303+0000", 2024-03-19T11:31:27.849 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4888999999999999e-05 2024-03-19T11:31:27.849 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.849 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.849 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:27.849 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.985366+0000", 2024-03-19T11:31:27.849 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001062321 2024-03-19T11:31:27.849 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.849 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:27.849 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.849 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.849 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.849 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1472 4.10 4:0dc3b91e:::smithi03829856-28:head [read 534528~391680] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-03-19T11:31:27.850 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:27.091748+0000", 2024-03-19T11:31:27.850 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 60.729551217999997, 2024-03-19T11:31:27.850 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.83574118500000005, 2024-03-19T11:31:27.850 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:27.850 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:27.850 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:27.850 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:27.850 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:27.850 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1472 2024-03-19T11:31:27.850 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.851 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:27.851 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.851 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:27.851 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091748+0000", 2024-03-19T11:31:27.851 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.852 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.852 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.852 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:27.853 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091748+0000", 2024-03-19T11:31:27.853 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.853 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.853 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.853 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:27.853 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091750+0000", 2024-03-19T11:31:27.853 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3130000000000001e-06 2024-03-19T11:31:27.853 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.853 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.853 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:27.853 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091760+0000", 2024-03-19T11:31:27.853 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0560999999999999e-05 2024-03-19T11:31:27.853 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.853 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.854 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:27.854 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091784+0000", 2024-03-19T11:31:27.854 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3915000000000001e-05 2024-03-19T11:31:27.854 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.854 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.854 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:27.854 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091797+0000", 2024-03-19T11:31:27.854 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2660000000000001e-05 2024-03-19T11:31:27.854 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.854 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.854 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:27.854 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091874+0000", 2024-03-19T11:31:27.854 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6757999999999999e-05 2024-03-19T11:31:27.854 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.855 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.855 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:27.855 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091882+0000", 2024-03-19T11:31:27.855 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5520000000000005e-06 2024-03-19T11:31:27.855 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.855 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.855 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:27.855 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.927033+0000", 2024-03-19T11:31:27.855 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.83515094099999998 2024-03-19T11:31:27.855 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.855 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.855 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:27.855 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.927250+0000", 2024-03-19T11:31:27.855 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00021675199999999999 2024-03-19T11:31:27.855 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.856 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.856 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:27.856 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.927489+0000", 2024-03-19T11:31:27.856 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023873300000000001 2024-03-19T11:31:27.856 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.856 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:27.856 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.856 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.856 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.856 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1474 4.10 4:0dc3b91e:::smithi03829856-28:head [assert-version v27,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-03-19T11:31:27.856 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:27.934139+0000", 2024-03-19T11:31:27.856 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 59.887159431999997, 2024-03-19T11:31:27.856 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024764884000000001, 2024-03-19T11:31:27.856 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:27.857 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:27.857 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:27.857 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:27.857 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:27.857 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1474 2024-03-19T11:31:27.857 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.857 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:27.857 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.857 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:27.857 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934139+0000", 2024-03-19T11:31:27.857 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.857 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.857 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.857 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:27.858 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934139+0000", 2024-03-19T11:31:27.858 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.858 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.858 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.858 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:27.858 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934142+0000", 2024-03-19T11:31:27.858 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5459999999999998e-06 2024-03-19T11:31:27.858 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.858 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.859 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:27.859 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934152+0000", 2024-03-19T11:31:27.859 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9399999999999997e-06 2024-03-19T11:31:27.859 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.859 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.859 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:27.859 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934173+0000", 2024-03-19T11:31:27.859 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1522000000000002e-05 2024-03-19T11:31:27.859 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.859 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.859 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:27.859 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934185+0000", 2024-03-19T11:31:27.859 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1946999999999999e-05 2024-03-19T11:31:27.859 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.860 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.860 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:27.860 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934282+0000", 2024-03-19T11:31:27.860 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6272000000000003e-05 2024-03-19T11:31:27.860 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.860 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.860 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:27.860 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934379+0000", 2024-03-19T11:31:27.860 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6966999999999995e-05 2024-03-19T11:31:27.860 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.860 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.860 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:27.860 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.955023+0000", 2024-03-19T11:31:27.860 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020644302 2024-03-19T11:31:27.860 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.861 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.861 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:27.861 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.955167+0000", 2024-03-19T11:31:27.861 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014404500000000001 2024-03-19T11:31:27.861 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.861 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.861 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1", 2024-03-19T11:31:27.861 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.957913+0000", 2024-03-19T11:31:27.861 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0027463779999999998 2024-03-19T11:31:27.861 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.861 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.861 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:27.861 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.958548+0000", 2024-03-19T11:31:27.861 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00063495700000000001 2024-03-19T11:31:27.862 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.862 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.862 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:27.862 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.958785+0000", 2024-03-19T11:31:27.862 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000236872 2024-03-19T11:31:27.862 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.862 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.862 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:27.862 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.958854+0000", 2024-03-19T11:31:27.862 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9099999999999999e-05 2024-03-19T11:31:27.862 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.862 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.862 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:27.862 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.958904+0000", 2024-03-19T11:31:27.862 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0036e-05 2024-03-19T11:31:27.863 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.863 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:27.863 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.863 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.863 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.863 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2011 4.0 4:01018511:::smithi03829856-19:c [read 0~3563051,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:31:27.863 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:46.264515+0000", 2024-03-19T11:31:27.863 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 41.556783764999999, 2024-03-19T11:31:27.863 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013774359999999999, 2024-03-19T11:31:27.863 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:27.863 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:27.864 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:27.864 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:27.864 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:27.864 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2011 2024-03-19T11:31:27.864 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.864 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:27.864 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.864 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:27.864 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264515+0000", 2024-03-19T11:31:27.864 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.864 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.864 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.864 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:27.864 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264515+0000", 2024-03-19T11:31:27.865 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.865 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.865 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.865 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:27.865 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264516+0000", 2024-03-19T11:31:27.865 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5899999999999995e-07 2024-03-19T11:31:27.865 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.865 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.865 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:27.865 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264522+0000", 2024-03-19T11:31:27.865 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6019999999999999e-06 2024-03-19T11:31:27.865 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.865 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.865 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:27.865 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264535+0000", 2024-03-19T11:31:27.866 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3207000000000001e-05 2024-03-19T11:31:27.866 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.866 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.866 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:27.866 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264543+0000", 2024-03-19T11:31:27.866 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6270000000000006e-06 2024-03-19T11:31:27.866 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.866 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.866 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:27.866 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.271665+0000", 2024-03-19T11:31:27.866 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.007121881 2024-03-19T11:31:27.866 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.866 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.867 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:27.867 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.271835+0000", 2024-03-19T11:31:27.867 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000169958 2024-03-19T11:31:27.867 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.867 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.867 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:27.867 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.278289+0000", 2024-03-19T11:31:27.867 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0064542260000000004 2024-03-19T11:31:27.867 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.867 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:27.867 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.867 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.867 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.867 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 75/74 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:7, version: 55'82, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@55'82, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 58'86, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@58'86, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [504210~968657,1572695~79919], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:12, version: 66'101, data_included: [440350~816629,1309629~231947], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:12@66'101, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:10, version: 70'112, data_included: [440350~816629,1309629~231947], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:10@70'112, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 70'113, data_included: [440350~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@70'113, size: 2605085, copy_subset: [(0, 2605085)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1488926, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:27.868 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.108941+0000", 2024-03-19T11:31:27.868 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 33.712357695000001, 2024-03-19T11:31:27.868 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.067446694000000001, 2024-03-19T11:31:27.868 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:27.868 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:27.868 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:27.868 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.868 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:27.868 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.108941+0000", 2024-03-19T11:31:27.869 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.869 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.869 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.869 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:27.869 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.108941+0000", 2024-03-19T11:31:27.870 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.870 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.870 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.870 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:27.870 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.108943+0000", 2024-03-19T11:31:27.870 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2259999999999999e-06 2024-03-19T11:31:27.870 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.870 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.870 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:27.870 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123892+0000", 2024-03-19T11:31:27.870 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014948655999999999 2024-03-19T11:31:27.870 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.871 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.871 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:27.871 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123993+0000", 2024-03-19T11:31:27.871 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000101236 2024-03-19T11:31:27.871 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.871 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.871 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:27.871 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124126+0000", 2024-03-19T11:31:27.871 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000133184 2024-03-19T11:31:27.871 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.871 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.872 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:27.872 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124655+0000", 2024-03-19T11:31:27.872 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00052849600000000004 2024-03-19T11:31:27.872 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.872 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.872 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:27.872 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124723+0000", 2024-03-19T11:31:27.872 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7797000000000002e-05 2024-03-19T11:31:27.872 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.872 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.872 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:27.872 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.176388+0000", 2024-03-19T11:31:27.873 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.051665098999999999 2024-03-19T11:31:27.873 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.873 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:27.873 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.874 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.874 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.874 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.21 e75,74 4:8401c1f0:::smithi03829856-43:8,46'66,4:866f3dcc:::smithi03829856-31:2,47'73)", 2024-03-19T11:31:27.874 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.124226+0000", 2024-03-19T11:31:27.875 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 33.697073039000003, 2024-03-19T11:31:27.875 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.052217913999999997, 2024-03-19T11:31:27.875 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:27.875 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:27.875 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:27.875 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.875 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:27.875 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124226+0000", 2024-03-19T11:31:27.875 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.875 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.875 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.875 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:27.876 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124226+0000", 2024-03-19T11:31:27.876 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.876 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.876 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.876 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:27.876 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124227+0000", 2024-03-19T11:31:27.876 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.477e-06 2024-03-19T11:31:27.876 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.876 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.876 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:27.876 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124240+0000", 2024-03-19T11:31:27.877 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2379e-05 2024-03-19T11:31:27.877 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.877 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.877 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:27.877 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124264+0000", 2024-03-19T11:31:27.877 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4224999999999999e-05 2024-03-19T11:31:27.877 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.877 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.877 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:27.877 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124287+0000", 2024-03-19T11:31:27.877 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3557000000000002e-05 2024-03-19T11:31:27.877 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.878 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.878 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:27.878 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.176274+0000", 2024-03-19T11:31:27.878 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.051986248999999998 2024-03-19T11:31:27.878 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.878 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.878 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:27.878 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.176285+0000", 2024-03-19T11:31:27.878 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0818999999999999e-05 2024-03-19T11:31:27.878 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.878 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.878 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:27.879 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.176444+0000", 2024-03-19T11:31:27.879 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015920800000000001 2024-03-19T11:31:27.879 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.879 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:27.879 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.879 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.880 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.880 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 75/74 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:7, version: 55'82, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@55'82, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 58'86, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@58'86, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [1652614~973634,2888511~52681,3182305~22261], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3204566, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:12, version: 66'101, data_included: [1541576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:12@66'101, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:10, version: 70'112, data_included: [1541576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:10@70'112, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 70'113, data_included: [1488926~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@70'113, size: 2605085, copy_subset: [(0, 2605085)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2537502, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1488926, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:27.880 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.195615+0000", 2024-03-19T11:31:27.880 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 33.625684141999997, 2024-03-19T11:31:27.880 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.036930510999999999, 2024-03-19T11:31:27.880 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:27.880 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:27.880 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:27.880 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.880 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:27.880 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.195615+0000", 2024-03-19T11:31:27.880 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.881 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.881 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.881 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:27.881 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.195615+0000", 2024-03-19T11:31:27.881 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.881 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.881 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.881 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:27.881 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.195617+0000", 2024-03-19T11:31:27.881 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0490000000000002e-06 2024-03-19T11:31:27.882 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.882 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.882 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:27.882 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204654+0000", 2024-03-19T11:31:27.882 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0090370339999999993 2024-03-19T11:31:27.882 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.882 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.882 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:27.882 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204730+0000", 2024-03-19T11:31:27.882 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5939000000000006e-05 2024-03-19T11:31:27.882 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.882 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.883 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:27.883 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204844+0000", 2024-03-19T11:31:27.883 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000114066 2024-03-19T11:31:27.883 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.883 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.883 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:27.883 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.205232+0000", 2024-03-19T11:31:27.883 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000388045 2024-03-19T11:31:27.883 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.883 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.883 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:27.884 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.205277+0000", 2024-03-19T11:31:27.884 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5071999999999998e-05 2024-03-19T11:31:27.884 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.884 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.884 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:27.884 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.232545+0000", 2024-03-19T11:31:27.884 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027268305999999999 2024-03-19T11:31:27.884 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.884 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:27.884 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.884 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.884 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.884 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.21 e75,74 4:866f3dcc:::smithi03829856-31:c,70'114)", 2024-03-19T11:31:27.885 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.204941+0000", 2024-03-19T11:31:27.885 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 33.616357516999997, 2024-03-19T11:31:27.885 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.028797390999999999, 2024-03-19T11:31:27.885 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:27.885 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:27.885 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:27.885 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.885 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:27.885 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204941+0000", 2024-03-19T11:31:27.885 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.885 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.885 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.885 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:27.886 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204941+0000", 2024-03-19T11:31:27.886 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.886 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.886 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.886 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:27.886 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204943+0000", 2024-03-19T11:31:27.886 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.35e-06 2024-03-19T11:31:27.886 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.886 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.886 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:27.886 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204950+0000", 2024-03-19T11:31:27.887 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.374e-06 2024-03-19T11:31:27.887 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.887 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.887 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:27.887 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204965+0000", 2024-03-19T11:31:27.887 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5169e-05 2024-03-19T11:31:27.887 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.887 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.887 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:27.887 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204979+0000", 2024-03-19T11:31:27.887 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4260000000000001e-05 2024-03-19T11:31:27.887 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.887 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.888 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:27.888 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.232607+0000", 2024-03-19T11:31:27.888 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027627518 2024-03-19T11:31:27.888 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.888 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.888 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:27.888 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.232626+0000", 2024-03-19T11:31:27.888 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9471999999999999e-05 2024-03-19T11:31:27.888 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.888 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.888 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:27.888 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.233739+0000", 2024-03-19T11:31:27.889 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011122479999999999 2024-03-19T11:31:27.889 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.889 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:27.889 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.889 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.889 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.889 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 75/74 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [2599014~241336], data_size: 241336, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2840350, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [2599014~241336,2948041~446940], data_size: 688276, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3394981, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [3204566~621403], data_size: 621403, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3865577, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3204566, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:12, version: 66'101, data_included: [2590152~250198,2948041~581396], data_size: 831594, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:12@66'101, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3529437, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:10, version: 70'112, data_included: [2590152~250198,2948041~581396], data_size: 831594, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:10@70'112, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3529437, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 70'113, data_included: [2537502~67583], data_size: 67583, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@70'113, size: 2605085, copy_subset: [(0, 2605085)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2605085, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2537502, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:27.889 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.241268+0000", 2024-03-19T11:31:27.889 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 33.580030397000002, 2024-03-19T11:31:27.889 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015726741999999998, 2024-03-19T11:31:27.889 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:27.889 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:27.890 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:27.890 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.890 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:27.890 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.241268+0000", 2024-03-19T11:31:27.890 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.890 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.890 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.890 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:27.890 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.241268+0000", 2024-03-19T11:31:27.890 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.890 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.890 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.890 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:27.891 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.241270+0000", 2024-03-19T11:31:27.891 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5659999999999999e-06 2024-03-19T11:31:27.891 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.891 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.891 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:27.891 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.245791+0000", 2024-03-19T11:31:27.891 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0045205810000000001 2024-03-19T11:31:27.891 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.891 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.891 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:27.891 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.245853+0000", 2024-03-19T11:31:27.891 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2671999999999999e-05 2024-03-19T11:31:27.891 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.892 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.892 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:27.892 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.245916+0000", 2024-03-19T11:31:27.892 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2626000000000007e-05 2024-03-19T11:31:27.892 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.892 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.892 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:27.892 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.246176+0000", 2024-03-19T11:31:27.892 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000260136 2024-03-19T11:31:27.892 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.892 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.892 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:27.892 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.246210+0000", 2024-03-19T11:31:27.893 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4183000000000001e-05 2024-03-19T11:31:27.893 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.893 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.893 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:27.893 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.256995+0000", 2024-03-19T11:31:27.893 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010784978000000001 2024-03-19T11:31:27.893 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.893 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:27.893 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.893 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.893 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.893 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 75/74 [PushOp(4:866f3dcc:::smithi03829856-31:head, version: 70'115, data_included: [406769~773083,1439434~275493], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:head@70'115, size: 2806769, copy_subset: [(0, 2806769)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1714927, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:27.894 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.245999+0000", 2024-03-19T11:31:27.894 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 33.575299811000001, 2024-03-19T11:31:27.894 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014650178999999999, 2024-03-19T11:31:27.894 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:27.894 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:27.894 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:27.894 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.894 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:27.894 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.245999+0000", 2024-03-19T11:31:27.894 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.894 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.894 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.895 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:27.895 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.245999+0000", 2024-03-19T11:31:27.895 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.895 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.895 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.895 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:27.895 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.246000+0000", 2024-03-19T11:31:27.895 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9800000000000003e-07 2024-03-19T11:31:27.895 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.895 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.895 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:27.895 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.246544+0000", 2024-03-19T11:31:27.895 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00054456700000000003 2024-03-19T11:31:27.896 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.896 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.896 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:27.896 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.246578+0000", 2024-03-19T11:31:27.896 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3995999999999998e-05 2024-03-19T11:31:27.896 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.896 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.896 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:27.896 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.246595+0000", 2024-03-19T11:31:27.896 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6558e-05 2024-03-19T11:31:27.896 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.896 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.896 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:27.897 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.257102+0000", 2024-03-19T11:31:27.897 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010506925 2024-03-19T11:31:27.897 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.897 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.897 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:27.897 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.257126+0000", 2024-03-19T11:31:27.897 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4443e-05 2024-03-19T11:31:27.897 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.897 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.897 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:27.897 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.260649+0000", 2024-03-19T11:31:27.897 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003522992 2024-03-19T11:31:27.897 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.898 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:27.898 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.898 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.898 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.898 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2609 4.19 4:9bd2ffb1:::smithi03829856-30:head [read 0~2487403,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e89)", 2024-03-19T11:31:27.898 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:08.055259+0000", 2024-03-19T11:31:27.898 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 19.766040221000001, 2024-03-19T11:31:27.898 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014845676, 2024-03-19T11:31:27.898 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:27.898 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:27.898 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:27.898 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:27.899 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:27.899 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2609 2024-03-19T11:31:27.899 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.899 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:27.899 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.899 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:27.899 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:08.055259+0000", 2024-03-19T11:31:27.899 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.899 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.899 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.899 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:27.900 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:08.055259+0000", 2024-03-19T11:31:27.900 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.900 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.900 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.900 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:27.900 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:08.055259+0000", 2024-03-19T11:31:27.900 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8500000000000001e-07 2024-03-19T11:31:27.900 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.900 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.900 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:27.900 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:08.055265+0000", 2024-03-19T11:31:27.900 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.558e-06 2024-03-19T11:31:27.901 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.901 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.901 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:27.901 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:08.055274+0000", 2024-03-19T11:31:27.901 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0000000000000002e-06 2024-03-19T11:31:27.901 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.901 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.901 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:27.901 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:08.055282+0000", 2024-03-19T11:31:27.901 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.011e-06 2024-03-19T11:31:27.901 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.901 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.902 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:27.902 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:08.055667+0000", 2024-03-19T11:31:27.902 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00038538100000000002 2024-03-19T11:31:27.902 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.902 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.902 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:31:27.902 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:08.055688+0000", 2024-03-19T11:31:27.902 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0729000000000001e-05 2024-03-19T11:31:27.902 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.902 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.902 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:27.902 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:08.066671+0000", 2024-03-19T11:31:27.903 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010983504 2024-03-19T11:31:27.903 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.903 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.903 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:27.903 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:08.066996+0000", 2024-03-19T11:31:27.903 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032403899999999998 2024-03-19T11:31:27.903 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.903 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.903 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:27.903 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:08.070104+0000", 2024-03-19T11:31:27.903 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031087689999999999 2024-03-19T11:31:27.903 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.903 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:27.904 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.904 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.904 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.904 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2780 3.2 3:47f177ee:::0a67e437b20d4ad126b67a2338de84fe4ef684c5c77cee818318d652193ba67c:head [write 0~2533888] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-03-19T11:31:27.904 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:12.308707+0000", 2024-03-19T11:31:27.904 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 15.512591466, 2024-03-19T11:31:27.904 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014502703, 2024-03-19T11:31:27.904 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:27.904 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:27.904 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:27.904 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:27.904 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:27.905 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2780 2024-03-19T11:31:27.905 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.905 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:27.905 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.905 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:27.905 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.308707+0000", 2024-03-19T11:31:27.905 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.905 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.905 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.905 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:27.905 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.308707+0000", 2024-03-19T11:31:27.906 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.906 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.906 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.906 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:27.906 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.308709+0000", 2024-03-19T11:31:27.906 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6050000000000001e-06 2024-03-19T11:31:27.906 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.906 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.906 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:27.906 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.309858+0000", 2024-03-19T11:31:27.906 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011490339999999999 2024-03-19T11:31:27.906 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.907 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.907 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:27.907 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.309879+0000", 2024-03-19T11:31:27.907 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1254e-05 2024-03-19T11:31:27.907 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.907 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.907 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:27.907 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.309888+0000", 2024-03-19T11:31:27.907 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.67e-06 2024-03-19T11:31:27.907 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.907 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.907 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:27.908 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.310032+0000", 2024-03-19T11:31:27.908 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000144499 2024-03-19T11:31:27.908 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.908 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.908 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:27.908 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.310438+0000", 2024-03-19T11:31:27.908 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00040561999999999998 2024-03-19T11:31:27.908 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.908 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.908 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:31:27.908 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.310835+0000", 2024-03-19T11:31:27.909 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00039648400000000001 2024-03-19T11:31:27.909 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.909 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.909 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:27.909 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.322685+0000", 2024-03-19T11:31:27.909 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011850673000000001 2024-03-19T11:31:27.909 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.909 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.909 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:27.910 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.323012+0000", 2024-03-19T11:31:27.910 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000327032 2024-03-19T11:31:27.910 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.910 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.910 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:27.910 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.323108+0000", 2024-03-19T11:31:27.910 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.5570999999999997e-05 2024-03-19T11:31:27.910 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.910 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.910 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:27.910 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.323210+0000", 2024-03-19T11:31:27.911 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000102261 2024-03-19T11:31:27.911 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.911 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:27.911 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.911 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.911 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.911 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2789 3.2 3:48367551:::ca87bd16019841aaa831bb522ee079e5f4e6a1ea67fbcb51e5f89263b6c5d8f6:head [write 0~3280896] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-03-19T11:31:27.911 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:12.340289+0000", 2024-03-19T11:31:27.911 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 15.481009801000001, 2024-03-19T11:31:27.911 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016291298999999999, 2024-03-19T11:31:27.911 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:27.911 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:27.912 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:27.912 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:27.912 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:27.912 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2789 2024-03-19T11:31:27.912 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.912 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:27.912 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.912 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:27.912 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.340289+0000", 2024-03-19T11:31:27.912 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.912 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.912 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.913 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:27.913 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.340289+0000", 2024-03-19T11:31:27.913 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.913 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.913 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.913 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:27.913 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.340291+0000", 2024-03-19T11:31:27.913 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1270000000000001e-06 2024-03-19T11:31:27.913 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.913 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.913 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:27.913 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.341426+0000", 2024-03-19T11:31:27.914 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011348459999999999 2024-03-19T11:31:27.914 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.914 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.914 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:27.914 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.341454+0000", 2024-03-19T11:31:27.914 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8204e-05 2024-03-19T11:31:27.914 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.914 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.914 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:27.914 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.341467+0000", 2024-03-19T11:31:27.914 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2700999999999999e-05 2024-03-19T11:31:27.915 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.915 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.915 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:27.915 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.341623+0000", 2024-03-19T11:31:27.915 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015600100000000001 2024-03-19T11:31:27.915 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.915 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.915 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:27.915 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.342034+0000", 2024-03-19T11:31:27.915 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000411247 2024-03-19T11:31:27.915 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.915 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.916 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:31:27.916 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.342443+0000", 2024-03-19T11:31:27.916 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00040901400000000002 2024-03-19T11:31:27.916 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.916 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.916 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:27.916 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.356159+0000", 2024-03-19T11:31:27.916 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013716058999999999 2024-03-19T11:31:27.916 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.916 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.916 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:27.917 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.356462+0000", 2024-03-19T11:31:27.917 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000302594 2024-03-19T11:31:27.917 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.917 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.917 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:27.917 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.356530+0000", 2024-03-19T11:31:27.917 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7852e-05 2024-03-19T11:31:27.917 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.917 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.917 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:27.917 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.356580+0000", 2024-03-19T11:31:27.917 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0654e-05 2024-03-19T11:31:27.918 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.918 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:27.918 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.918 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.918 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.918 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3046 4.19 4:9c74b127:::smithi03829856-33:head [write 400066~716205,stat] snapc 1f=[1f,1e,1d,1c,1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-03-19T11:31:27.918 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:22.147457+0000", 2024-03-19T11:31:27.918 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.6738415169999996, 2024-03-19T11:31:27.918 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016153588, 2024-03-19T11:31:27.919 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:27.919 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:27.919 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:27.919 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:27.919 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:27.919 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3046 2024-03-19T11:31:27.919 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.919 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:27.919 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.919 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:27.919 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.147457+0000", 2024-03-19T11:31:27.919 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.920 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.920 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.920 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:27.920 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.147457+0000", 2024-03-19T11:31:27.920 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:27.920 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.920 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.920 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:27.920 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.147459+0000", 2024-03-19T11:31:27.920 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1160000000000002e-06 2024-03-19T11:31:27.920 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.921 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.921 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:27.921 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.148083+0000", 2024-03-19T11:31:27.921 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00062361999999999997 2024-03-19T11:31:27.921 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.921 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.921 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:27.921 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.148100+0000", 2024-03-19T11:31:27.921 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6524999999999998e-05 2024-03-19T11:31:27.921 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.921 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.922 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:27.922 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.148110+0000", 2024-03-19T11:31:27.922 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0093e-05 2024-03-19T11:31:27.922 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.922 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.922 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:27.922 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.148197+0000", 2024-03-19T11:31:27.922 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7639000000000006e-05 2024-03-19T11:31:27.922 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.922 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.922 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:27.923 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.148359+0000", 2024-03-19T11:31:27.923 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000161953 2024-03-19T11:31:27.923 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.923 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.923 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 2", 2024-03-19T11:31:27.923 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.149074+0000", 2024-03-19T11:31:27.923 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00071444100000000001 2024-03-19T11:31:27.923 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.923 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.923 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:27.923 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.158856+0000", 2024-03-19T11:31:27.923 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0097818169999999999 2024-03-19T11:31:27.923 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.923 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.924 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:27.924 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.163525+0000", 2024-03-19T11:31:27.924 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046690270000000001 2024-03-19T11:31:27.924 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.924 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.924 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:27.924 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.163569+0000", 2024-03-19T11:31:27.924 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4279000000000001e-05 2024-03-19T11:31:27.924 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:27.925 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:27.925 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:27.925 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.163611+0000", 2024-03-19T11:31:27.925 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2077999999999999e-05 2024-03-19T11:31:27.925 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.925 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:27.925 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.925 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:27.925 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:27.926 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:27.927 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-19T11:31:27.928 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:27.938+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:28.029 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:28.036+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:28.042 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:28.050+0000 7fb17dd72640 -1 osd.4 108 handle_osd_map injecting map crc failure 2024-03-19T11:31:28.046 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:28.052+0000 7f70c6fdf640 -1 osd.0 108 handle_osd_map injecting map crc failure 2024-03-19T11:31:28.056 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-03-19T11:31:28.057 INFO:tasks.rados.rados.0.smithi038.stdout:1455: done (4 left) 2024-03-19T11:31:28.057 INFO:tasks.rados.rados.0.smithi038.stdout:1456: done (3 left) 2024-03-19T11:31:28.057 INFO:tasks.rados.rados.0.smithi038.stdout:1457: done (2 left) 2024-03-19T11:31:28.057 INFO:tasks.rados.rados.0.smithi038.stdout:1458: done (1 left) 2024-03-19T11:31:28.057 INFO:tasks.rados.rados.0.smithi038.stdout:1459: tier_promote oid 10 2024-03-19T11:31:28.057 INFO:tasks.rados.rados.0.smithi038.stdout:1460: write (excl) oid 47 current snap is 32 2024-03-19T11:31:28.057 INFO:tasks.rados.rados.0.smithi038.stdout:1460: seq_num 251 ranges {618557=727148,2104745=113812} 2024-03-19T11:31:28.063 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:28.063 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:31:28.063 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:31:28.063 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:28.066 INFO:tasks.rados.rados.0.smithi038.stdout:1460: writing smithi03829856-47 from 618557 to 1345705 tid 1 2024-03-19T11:31:28.071 INFO:tasks.rados.rados.0.smithi038.stdout:1460: writing smithi03829856-47 from 2104745 to 2218557 tid 2 2024-03-19T11:31:28.071 INFO:tasks.rados.rados.0.smithi038.stdout:1461: read oid 1 snap -1 2024-03-19T11:31:28.072 INFO:tasks.rados.rados.0.smithi038.stdout:1461: expect (ObjNum 226 snap 25 seq_num 226) 2024-03-19T11:31:28.072 INFO:tasks.rados.rados.0.smithi038.stdout:1462: tier_flush oid 30 2024-03-19T11:31:28.072 INFO:tasks.rados.rados.0.smithi038.stdout:1462: tier_flush oid 30 snap -1 2024-03-19T11:31:28.072 INFO:tasks.rados.rados.0.smithi038.stdout:1463: write (excl) oid 44 current snap is 32 2024-03-19T11:31:28.072 INFO:tasks.rados.rados.0.smithi038.stdout:1463: seq_num 252 ranges {635364=402397,1763227=472137} 2024-03-19T11:31:28.078 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-19T11:31:28.078 INFO:tasks.rados.rados.0.smithi038.stdout:1463: writing smithi03829856-44 from 635364 to 1037761 tid 1 2024-03-19T11:31:28.085 INFO:tasks.rados.rados.0.smithi038.stdout:1463: writing smithi03829856-44 from 1763227 to 2235364 tid 2 2024-03-19T11:31:28.085 INFO:tasks.rados.rados.0.smithi038.stdout:1464: write oid 36 current snap is 32 2024-03-19T11:31:28.085 INFO:tasks.rados.rados.0.smithi038.stdout:1464: seq_num 253 ranges {652171=477646,1821709=430462} 2024-03-19T11:31:28.093 INFO:tasks.rados.rados.0.smithi038.stdout:1464: writing smithi03829856-36 from 652171 to 1129817 tid 1 2024-03-19T11:31:28.098 INFO:tasks.rados.rados.0.smithi038.stdout:1464: writing smithi03829856-36 from 1821709 to 2252171 tid 2 2024-03-19T11:31:28.098 INFO:tasks.rados.rados.0.smithi038.stdout:1465: tier_evict oid 29 2024-03-19T11:31:28.099 INFO:tasks.rados.rados.0.smithi038.stdout:1465: tier_evict oid 29 snap -1 2024-03-19T11:31:28.099 INFO:tasks.rados.rados.0.smithi038.stdout:1460: finishing write tid 1 to smithi03829856-47 2024-03-19T11:31:28.099 INFO:tasks.rados.rados.0.smithi038.stdout:1452: expect (ObjNum 206 snap 22 seq_num 206) 2024-03-19T11:31:28.129 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:28.136+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:28.209 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:28.209 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:31:28.209 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:31:28.209 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:31:28.209 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:28.223 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-19T11:31:28.265 INFO:tasks.rados.rados.0.smithi038.stdout:1459: got (0) Success 2024-03-19T11:31:28.265 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 10 v 89 (ObjNum 179 snap 19 seq_num 179) dirty exists 2024-03-19T11:31:28.265 INFO:tasks.rados.rados.0.smithi038.stdout:1460: finishing write tid 2 to smithi03829856-47 2024-03-19T11:31:28.265 INFO:tasks.rados.rados.0.smithi038.stdout:1460: finishing write tid 3 to smithi03829856-47 2024-03-19T11:31:28.265 INFO:tasks.rados.rados.0.smithi038.stdout:1452: done (7 left) 2024-03-19T11:31:28.265 INFO:tasks.rados.rados.0.smithi038.stdout:1459: done (6 left) 2024-03-19T11:31:28.265 INFO:tasks.rados.rados.0.smithi038.stdout:1466: read oid 34 snap -1 2024-03-19T11:31:28.265 INFO:tasks.rados.rados.0.smithi038.stdout:1466: expect (ObjNum 72 snap 5 seq_num 72) 2024-03-19T11:31:28.265 INFO:tasks.rados.rados.0.smithi038.stdout:1467: read oid 16 snap -1 2024-03-19T11:31:28.266 INFO:tasks.rados.rados.0.smithi038.stdout:1467: expect (ObjNum 65 snap 3 seq_num 65) 2024-03-19T11:31:28.266 INFO:tasks.rados.rados.0.smithi038.stdout:1468: read oid 2 snap -1 2024-03-19T11:31:28.266 INFO:tasks.rados.rados.0.smithi038.stdout:1468: expect (ObjNum 206 snap 22 seq_num 206) 2024-03-19T11:31:28.266 INFO:tasks.rados.rados.0.smithi038.stdout:1469: set_chunk oid 17 target oid 2024-03-19T11:31:28.267 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:28.278+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:28.291 INFO:tasks.rados.rados.0.smithi038.stdout:1469: set_chunk oid 17 offset: 283648 length: 2407936 target oid 262a0c0840a3de2f9291cefa7282c21878d49a3f13bcfaffdbeba72545664470 offset: 0 snap -1 2024-03-19T11:31:28.291 INFO:tasks.rados.rados.0.smithi038.stdout:1470: read oid 9 snap 26 2024-03-19T11:31:28.292 INFO:tasks.rados.rados.0.smithi038.stdout:1470: expect (ObjNum 221 snap 24 seq_num 221) 2024-03-19T11:31:28.292 INFO:tasks.rados.rados.0.smithi038.stdout:1471: read oid 8 snap -1 2024-03-19T11:31:28.292 INFO:tasks.rados.rados.0.smithi038.stdout:1471: expect deleted 2024-03-19T11:31:28.292 INFO:tasks.rados.rados.0.smithi038.stdout:1472: read oid 18 snap -1 2024-03-19T11:31:28.292 INFO:tasks.rados.rados.0.smithi038.stdout:1472: expect (ObjNum 199 snap 22 seq_num 199) 2024-03-19T11:31:28.292 INFO:tasks.rados.rados.0.smithi038.stdout:1473: write oid 19 current snap is 32 2024-03-19T11:31:28.292 INFO:tasks.rados.rados.0.smithi038.stdout:1461: expect (ObjNum 226 snap 25 seq_num 226) 2024-03-19T11:31:28.323 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-19T11:31:28.330 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:28.338+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:28.359 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:28.360 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:31:28.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:31:28.360 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:31:28.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.360 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:851 4.9 4:953e720b:::smithi03829856-45:head [write 591613~454162,stat] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:31:28.360 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.763514+0000", 2024-03-19T11:31:28.360 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 79.601959996999994, 2024-03-19T11:31:28.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027825888999999999, 2024-03-19T11:31:28.360 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:28.360 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:28.360 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:28.360 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:28.361 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:28.361 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 851 2024-03-19T11:31:28.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.361 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:28.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:28.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763514+0000", 2024-03-19T11:31:28.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:28.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:28.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763514+0000", 2024-03-19T11:31:28.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:28.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.362 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:28.362 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763517+0000", 2024-03-19T11:31:28.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8499999999999998e-06 2024-03-19T11:31:28.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.362 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:28.362 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.764714+0000", 2024-03-19T11:31:28.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011971270000000001 2024-03-19T11:31:28.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.363 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:28.363 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.764745+0000", 2024-03-19T11:31:28.363 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0546999999999997e-05 2024-03-19T11:31:28.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.363 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.363 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:28.363 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.764767+0000", 2024-03-19T11:31:28.363 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2286999999999999e-05 2024-03-19T11:31:28.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.363 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.363 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:28.363 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.773924+0000", 2024-03-19T11:31:28.363 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0091567360000000004 2024-03-19T11:31:28.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.364 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:28.364 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.774217+0000", 2024-03-19T11:31:28.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029350099999999999 2024-03-19T11:31:28.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.364 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-19T11:31:28.364 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.774770+0000", 2024-03-19T11:31:28.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00055288600000000005 2024-03-19T11:31:28.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.364 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:28.364 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.786413+0000", 2024-03-19T11:31:28.365 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011643229999999999 2024-03-19T11:31:28.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:28.365 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791104+0000", 2024-03-19T11:31:28.365 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0046900830000000003 2024-03-19T11:31:28.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:28.365 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791248+0000", 2024-03-19T11:31:28.365 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014410200000000001 2024-03-19T11:31:28.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:28.366 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791340+0000", 2024-03-19T11:31:28.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2540000000000005e-05 2024-03-19T11:31:28.366 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:28.366 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:28.366 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:28.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.366 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1162 4.1 4:8401c1f0:::smithi03829856-43:head [write 594981~609197] snapc 8=[8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-03-19T11:31:28.366 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.838981+0000", 2024-03-19T11:31:28.366 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 71.526493505999994, 2024-03-19T11:31:28.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027704349999999999, 2024-03-19T11:31:28.366 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:28.366 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:28.366 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:28.366 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:28.367 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:28.367 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1162 2024-03-19T11:31:28.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.367 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:28.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.367 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:28.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.838981+0000", 2024-03-19T11:31:28.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:28.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.367 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:28.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.838981+0000", 2024-03-19T11:31:28.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:28.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.368 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:28.368 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.838983+0000", 2024-03-19T11:31:28.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5399999999999998e-06 2024-03-19T11:31:28.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.368 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:28.368 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841322+0000", 2024-03-19T11:31:28.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002338592 2024-03-19T11:31:28.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.368 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:28.368 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841371+0000", 2024-03-19T11:31:28.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9123999999999999e-05 2024-03-19T11:31:28.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.369 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.369 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:28.369 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841399+0000", 2024-03-19T11:31:28.369 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7718000000000001e-05 2024-03-19T11:31:28.369 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.369 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.369 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:28.369 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841577+0000", 2024-03-19T11:31:28.369 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000178097 2024-03-19T11:31:28.369 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.369 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.369 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:28.369 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841825+0000", 2024-03-19T11:31:28.369 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00024824299999999999 2024-03-19T11:31:28.369 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.370 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.370 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-19T11:31:28.370 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842227+0000", 2024-03-19T11:31:28.370 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00040212199999999998 2024-03-19T11:31:28.370 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.370 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.370 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:28.370 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.865182+0000", 2024-03-19T11:31:28.370 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022954810999999999 2024-03-19T11:31:28.370 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.370 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.370 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:28.370 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.866360+0000", 2024-03-19T11:31:28.370 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001178449 2024-03-19T11:31:28.370 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.371 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.371 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:28.371 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.866538+0000", 2024-03-19T11:31:28.371 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017791000000000001 2024-03-19T11:31:28.371 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.371 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.371 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:28.371 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.866685+0000", 2024-03-19T11:31:28.371 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000146744 2024-03-19T11:31:28.371 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:28.371 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:28.371 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:28.371 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.371 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.372 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1271 4.9 4:953e720b:::smithi03829856-45:2 [read 795136~1008128] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-03-19T11:31:28.372 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:21.868056+0000", 2024-03-19T11:31:28.372 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 66.497418001, 2024-03-19T11:31:28.372 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.047270591000000001, 2024-03-19T11:31:28.372 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:28.372 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:28.372 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:28.372 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:28.372 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:28.372 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1271 2024-03-19T11:31:28.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.372 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:28.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.372 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:28.372 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868056+0000", 2024-03-19T11:31:28.372 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:28.373 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.373 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.373 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:28.373 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868056+0000", 2024-03-19T11:31:28.373 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:28.373 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.373 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.373 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:28.373 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868059+0000", 2024-03-19T11:31:28.373 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0469999999999998e-06 2024-03-19T11:31:28.373 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.373 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.373 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:28.373 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868072+0000", 2024-03-19T11:31:28.374 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2700999999999999e-05 2024-03-19T11:31:28.374 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.374 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:28.374 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868106+0000", 2024-03-19T11:31:28.374 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4152000000000001e-05 2024-03-19T11:31:28.374 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.374 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:28.374 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868126+0000", 2024-03-19T11:31:28.374 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9386999999999999e-05 2024-03-19T11:31:28.374 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.374 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:28.375 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.911447+0000", 2024-03-19T11:31:28.375 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.043321164000000002 2024-03-19T11:31:28.375 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.375 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.375 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:28.375 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.911762+0000", 2024-03-19T11:31:28.375 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00031492400000000001 2024-03-19T11:31:28.375 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.375 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.375 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:28.375 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.915327+0000", 2024-03-19T11:31:28.375 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003565216 2024-03-19T11:31:28.375 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:28.375 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:28.375 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:28.376 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.376 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.376 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(3.1 84/83 [PushOp(3:9205291d:::smithi03829856-50 low_tier:head, version: 28'36, data_included: [393216~851968,1703936~196608], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:9205291d:::smithi03829856-50 low_tier:head@28'36, size: 2823543, copy_subset: [(0, 2823543)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1900544, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:baab691a:::smithi03829856-2 low_tier:head, version: 28'37, data_included: [393216~524288,1507328~509479], data_size: 1033767, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:baab691a:::smithi03829856-2 low_tier:head@28'37, size: 2016807, copy_subset: [(0, 2016807)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2016807, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:98ce8426:::smithi03829856-4 low_tier:head, version: 28'38, data_included: [393216~720896,1638400~327680], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:98ce8426:::smithi03829856-4 low_tier:head@28'38, size: 2050421, copy_subset: [(0, 2050421)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1966080, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:a86be07b:::smithi03829856-44 low_tier:head, version: 28'41, data_included: [720896~655360,1769472~393216], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:a86be07b:::smithi03829856-44 low_tier:head@28'41, size: 2722701, copy_subset: [(0, 2722701)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2162688, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:bb56919e:::smithi03829856-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'44, data_included: [393216~786432,1703936~262144], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:bb56919e:::smithi03829856-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'44, size: 2806736, copy_subset: [(0, 2806736)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1966080, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:b1f822cf:::smithi03829856-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'46, data_included: [720896~786432,1966080~262144], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:b1f822cf:::smithi03829856-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'46, size: 2369754, copy_subset: [(0, 2369754)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2228224, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:a5e8313a:::smithi03829856-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'47, data_included: [524288~720896,1769472~327680], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:a5e8313a:::smithi03829856-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'47, size: 2571438, copy_subset: [(0, 2571438)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:bca5ec64:::e3a26bf9230b1b95b8cf1a4cb01909236bf2bf3fd59b083b2291d278ab98264d:head, version: 28'48, data_included: [0~393216], data_size: 393216, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bca5ec64:::e3a26bf9230b1b95b8cf1a4cb01909236bf2bf3fd59b083b2291d278ab98264d:head@28'48, size: 533504, copy_subset: [(0, 533504)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 533504, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:99a49d1b:::895a63407b6109b3cca9a6f5ef827134d56679bef42daadc5892269682c428ed:head, version: 32'49, data_included: [393216~388096], data_size: 388096, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:99a49d1b:::895a63407b6109b3cca9a6f5ef827134d56679bef42daadc5892269682c428ed:head@32'49, size: 781312, copy_subset: [(0, 781312)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 781312, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:83064589:::a3639ef4c215b429a637f1406db2398074b33464b996fe20e20d8b66c6bd720b:head, version: 33'51, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:83064589:::a3639ef4c215b429a637f1406db2398074b33464b996fe20e20d8b66c6bd720b:head@33'51, size: 1606656, copy_subset: [(0, 1606656)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:28.376 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:03.092965+0000", 2024-03-19T11:31:28.376 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 25.272508958, 2024-03-19T11:31:28.376 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.061194046000000002, 2024-03-19T11:31:28.376 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:28.376 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:28.376 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:28.376 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.376 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:28.376 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.092965+0000", 2024-03-19T11:31:28.376 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:28.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.377 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.377 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:28.377 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.092965+0000", 2024-03-19T11:31:28.377 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:28.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.377 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.377 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:28.377 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.092968+0000", 2024-03-19T11:31:28.377 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8720000000000002e-06 2024-03-19T11:31:28.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.377 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.377 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:28.377 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.106869+0000", 2024-03-19T11:31:28.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013900554000000001 2024-03-19T11:31:28.378 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.378 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.378 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:28.378 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.107077+0000", 2024-03-19T11:31:28.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000208626 2024-03-19T11:31:28.378 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.378 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.378 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:28.378 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.107372+0000", 2024-03-19T11:31:28.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029439199999999998 2024-03-19T11:31:28.378 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.378 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.378 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:28.378 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.108053+0000", 2024-03-19T11:31:28.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00068115899999999997 2024-03-19T11:31:28.379 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.379 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.379 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:28.379 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.108165+0000", 2024-03-19T11:31:28.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011162000000000001 2024-03-19T11:31:28.379 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.379 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.379 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:28.379 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.154159+0000", 2024-03-19T11:31:28.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.045994822999999997 2024-03-19T11:31:28.379 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:28.379 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:28.379 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:28.379 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.380 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.380 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.5 84/83 [PushOp(4:a0a50d04:::smithi03829856-13:b, version: 56'65, data_included: [601684~619341,1669078~429235], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a0a50d04:::smithi03829856-13:b@56'65, size: 2201684, copy_subset: [(0, 2201684)], clone_subset: {}, snapset: 17=[]:{b: [b, 6], 12: [12, d], 14: [13], 17: [17, 16, 15]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2098313, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:a0a50d04:::smithi03829856-13:14, version: 74'84, data_included: [601684~754801,1669078~293775], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a0a50d04:::smithi03829856-13:14@74'84, size: 2722734, copy_subset: [(0, 2722734)], clone_subset: {}, snapset: 17=[]:{b: [b, 6], 12: [12, d], 14: [13], 17: [17, 16, 15]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1962853, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:a0a50d04:::smithi03829856-13:12, version: 81'90, data_included: [601684~619341,1669078~429235], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a0a50d04:::smithi03829856-13:12@81'90, size: 2201684, copy_subset: [(0, 2201684)], clone_subset: {}, snapset: 17=[]:{b: [b, 6], 12: [12, d], 14: [13], 17: [17, 16, 15]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2098313, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:a38b9453:::smithi03829856-37:13, version: 81'92, data_included: [605052~774376,1923587~274200], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a38b9453:::smithi03829856-37:13@81'92, size: 2605052, copy_subset: [(0, 2605052)], clone_subset: {}, snapset: 14=[]:{13: [13, 12, d, b, 6]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2197787, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:a0a50d04:::smithi03829856-13:17, version: 82'94, data_included: [601684~754801,1669078~293775], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a0a50d04:::smithi03829856-13:17@82'94, size: 3025260, copy_subset: [(0, 3025260)], clone_subset: {}, snapset: 17=[]:{b: [b, 6], 12: [12, d], 14: [13], 17: [17, 16, 15]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1962853, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:a0a50d04:::smithi03829856-13:head, version: 82'97, data_included: [601684~619341,1669078~429235], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:a0a50d04:::smithi03829856-13:head@82'97, size: 2201684, copy_subset: [(0, 2201684)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2098313, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:28.380 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:03.103661+0000", 2024-03-19T11:31:28.380 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 25.261812936999998, 2024-03-19T11:31:28.380 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027776637, 2024-03-19T11:31:28.380 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:28.380 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:28.380 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:28.380 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.380 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:28.380 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.103661+0000", 2024-03-19T11:31:28.380 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:28.380 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.381 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.381 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:28.381 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.103661+0000", 2024-03-19T11:31:28.381 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:28.381 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.381 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.381 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:28.381 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.103664+0000", 2024-03-19T11:31:28.381 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.48e-06 2024-03-19T11:31:28.381 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.381 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.381 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:28.381 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.109797+0000", 2024-03-19T11:31:28.381 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061333380000000003 2024-03-19T11:31:28.381 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.382 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:28.382 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.109866+0000", 2024-03-19T11:31:28.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8474999999999997e-05 2024-03-19T11:31:28.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.382 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:28.382 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.109957+0000", 2024-03-19T11:31:28.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1669000000000006e-05 2024-03-19T11:31:28.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.382 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:28.382 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110162+0000", 2024-03-19T11:31:28.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000204417 2024-03-19T11:31:28.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:28.383 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110186+0000", 2024-03-19T11:31:28.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4729e-05 2024-03-19T11:31:28.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:28.383 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.131438+0000", 2024-03-19T11:31:28.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021251529000000002 2024-03-19T11:31:28.383 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:28.383 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:28.383 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:28.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.383 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.5 84/83 [PushOp(4:a38b9453:::smithi03829856-37:head, version: 81'93, data_included: [1775443~781480,2724284~166520], data_size: 948000, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:a38b9453:::smithi03829856-37:head@81'93, size: 2958032, copy_subset: [(0, 2958032)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2958032, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1775443, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:28.384 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:03.110032+0000", 2024-03-19T11:31:28.384 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 25.255442709, 2024-03-19T11:31:28.384 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027756439000000001, 2024-03-19T11:31:28.384 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:28.384 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:28.384 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:28.384 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.384 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:28.384 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110032+0000", 2024-03-19T11:31:28.384 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:28.384 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.384 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.384 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:28.384 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110032+0000", 2024-03-19T11:31:28.384 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:28.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.385 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.385 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:28.385 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110033+0000", 2024-03-19T11:31:28.385 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1510000000000001e-06 2024-03-19T11:31:28.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.385 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.385 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:28.385 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110353+0000", 2024-03-19T11:31:28.385 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00031981199999999999 2024-03-19T11:31:28.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.385 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.385 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:28.385 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110375+0000", 2024-03-19T11:31:28.385 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2772000000000002e-05 2024-03-19T11:31:28.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.386 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:28.386 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110396+0000", 2024-03-19T11:31:28.386 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1066000000000001e-05 2024-03-19T11:31:28.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.386 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:28.386 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.131543+0000", 2024-03-19T11:31:28.386 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021146166000000001 2024-03-19T11:31:28.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.386 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:28.386 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.131563+0000", 2024-03-19T11:31:28.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0750999999999998e-05 2024-03-19T11:31:28.387 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.387 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:28.387 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.137788+0000", 2024-03-19T11:31:28.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062247209999999999 2024-03-19T11:31:28.387 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:28.387 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:28.387 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:28.387 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.387 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.5 e84,83 4:a38b9453:::smithi03829856-37:14,74'82)", 2024-03-19T11:31:28.387 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:03.110423+0000", 2024-03-19T11:31:28.387 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 25.255051423000001, 2024-03-19T11:31:28.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027613453, 2024-03-19T11:31:28.387 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:28.388 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:28.388 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:28.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:28.388 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110423+0000", 2024-03-19T11:31:28.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:28.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:28.388 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110423+0000", 2024-03-19T11:31:28.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:28.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:28.389 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110424+0000", 2024-03-19T11:31:28.389 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.8100000000000002e-07 2024-03-19T11:31:28.389 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.389 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.389 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:28.389 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110429+0000", 2024-03-19T11:31:28.389 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5269999999999998e-06 2024-03-19T11:31:28.389 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.389 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.389 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:28.389 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110440+0000", 2024-03-19T11:31:28.389 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1221e-05 2024-03-19T11:31:28.389 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.389 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.389 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:28.390 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110450+0000", 2024-03-19T11:31:28.390 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5370000000000003e-06 2024-03-19T11:31:28.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.390 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.390 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:28.390 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.137887+0000", 2024-03-19T11:31:28.390 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027436642000000001 2024-03-19T11:31:28.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.390 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.390 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:28.390 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.137898+0000", 2024-03-19T11:31:28.390 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1021e-05 2024-03-19T11:31:28.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.390 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.391 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:28.391 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.138036+0000", 2024-03-19T11:31:28.391 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013872400000000001 2024-03-19T11:31:28.391 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:28.391 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:28.391 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:28.391 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.391 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.391 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.18 85/84 [PushOp(4:1d58db11:::smithi03829856-18:6, version: 55'80, data_included: [759683~639358,1904434~409218], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:6@55'80, size: 3159683, copy_subset: [(0, 3159683)], clone_subset: {}, snapset: 16=[]:{6: [6], b: [b], 15: [15, 13, 12, d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2313652, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1d58db11:::smithi03829856-18:b, version: 58'84, data_included: [443718~955323,1564138~93253], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:b@58'84, size: 3445402, copy_subset: [(0, 3445402)], clone_subset: {}, snapset: 16=[]:{6: [6], b: [b], 15: [15, 13, 12, d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1657391, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:10, version: 66'95, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:10@66'95, size: 3764735, copy_subset: [(0, 3764735)], clone_subset: {}, snapset: 15=[]:{c: [b, 6], 10: [d], 12: [12], 15: [15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:c, version: 70'102, data_included: [416807~576167,1542129~472409], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:c@70'102, size: 2857157, copy_subset: [(0, 2857157)], clone_subset: {}, snapset: 15=[]:{c: [b, 6], 10: [d], 12: [12], 15: [15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2014538, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:15, version: 77'110, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:15@77'110, size: 2621892, copy_subset: [(0, 2621892)], clone_subset: {}, snapset: 15=[]:{c: [b, 6], 10: [d], 12: [12], 15: [15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:28.391 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:04.083107+0000", 2024-03-19T11:31:28.391 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.282367142999998, 2024-03-19T11:31:28.391 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.042109168000000002, 2024-03-19T11:31:28.391 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:28.392 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:28.392 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:28.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.392 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:28.392 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.083107+0000", 2024-03-19T11:31:28.392 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:28.392 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.392 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:28.392 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.083107+0000", 2024-03-19T11:31:28.392 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:28.392 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.392 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:28.393 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.083109+0000", 2024-03-19T11:31:28.393 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7060000000000001e-06 2024-03-19T11:31:28.393 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.393 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:28.393 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092477+0000", 2024-03-19T11:31:28.393 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0093678589999999992 2024-03-19T11:31:28.393 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.393 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:28.393 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092549+0000", 2024-03-19T11:31:28.393 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2313999999999999e-05 2024-03-19T11:31:28.393 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.394 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:28.394 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092640+0000", 2024-03-19T11:31:28.394 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1262999999999997e-05 2024-03-19T11:31:28.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.394 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.394 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:28.394 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.093076+0000", 2024-03-19T11:31:28.394 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00043617800000000001 2024-03-19T11:31:28.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.394 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.394 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:28.394 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.093145+0000", 2024-03-19T11:31:28.394 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8651000000000004e-05 2024-03-19T11:31:28.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.394 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.395 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:28.395 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.125216+0000", 2024-03-19T11:31:28.395 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032071197000000003 2024-03-19T11:31:28.395 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:28.395 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:28.395 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:28.395 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.395 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.395 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.18 e85,84 4:1d58db11:::smithi03829856-18:7,38'48,4:1d58db11:::smithi03829856-18:9,46'64,4:1d58db11:::smithi03829856-18:2,48'76,4:1d58db11:::smithi03829856-18:c,70'104)", 2024-03-19T11:31:28.395 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:04.092733+0000", 2024-03-19T11:31:28.395 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.272740782, 2024-03-19T11:31:28.395 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033436452999999998, 2024-03-19T11:31:28.395 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:28.395 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:28.395 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:28.396 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.396 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:28.396 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092733+0000", 2024-03-19T11:31:28.396 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:28.396 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.396 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.396 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:28.396 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092733+0000", 2024-03-19T11:31:28.396 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:28.396 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.396 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.396 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:28.396 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092734+0000", 2024-03-19T11:31:28.396 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0699999999999996e-07 2024-03-19T11:31:28.397 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.397 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.397 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:28.397 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092745+0000", 2024-03-19T11:31:28.397 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0393e-05 2024-03-19T11:31:28.397 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.397 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.397 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:28.397 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092762+0000", 2024-03-19T11:31:28.397 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7649e-05 2024-03-19T11:31:28.397 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.397 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.397 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:28.397 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092778+0000", 2024-03-19T11:31:28.397 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5625999999999999e-05 2024-03-19T11:31:28.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.398 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.398 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:28.398 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.125342+0000", 2024-03-19T11:31:28.398 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032564348 2024-03-19T11:31:28.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.398 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.398 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:28.398 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.125375+0000", 2024-03-19T11:31:28.398 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2441000000000003e-05 2024-03-19T11:31:28.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.398 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.398 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:28.398 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.126170+0000", 2024-03-19T11:31:28.398 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00079508900000000002 2024-03-19T11:31:28.399 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:28.399 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:28.399 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:28.399 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.399 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.399 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(3.3 85/84 [PushOp(3:fb6cb292:::smithi03829856-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'21, data_included: [655360~851968,1900544~196608], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:fb6cb292:::smithi03829856-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'21, size: 2302526, copy_subset: [(0, 2302526)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:d281c5a5:::smithi03829856-30 low_tier:head, version: 28'42, data_included: [458752~589824,1507328~458752], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:d281c5a5:::smithi03829856-30 low_tier:head@28'42, size: 2487403, copy_subset: [(0, 2487403)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1966080, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:ed957b32:::smithi03829856-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'44, data_included: [524288~786432,1900544~233912], data_size: 1020344, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:ed957b32:::smithi03829856-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'44, size: 2134456, copy_subset: [(0, 2134456)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2134456, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:f23a45c7:::smithi03829856-36 low_tier:head, version: 28'45, data_included: [524288~786432,1835008~262144], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:f23a45c7:::smithi03829856-36 low_tier:head@28'45, size: 2588245, copy_subset: [(0, 2588245)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:edb38de0:::smithi03829856-40 low_tier:head, version: 28'46, data_included: [655360~720896,2097152~327680], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:edb38de0:::smithi03829856-40 low_tier:head@28'46, size: 2655473, copy_subset: [(0, 2655473)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2424832, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:f92b3f40:::b1fb925e39ab40886d36b70ff1c8c4f867ef6cb2ef150fbe90de57d057dfec36:head, version: 29'47, data_included: [262144~720896,1507328~327680], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:f92b3f40:::b1fb925e39ab40886d36b70ff1c8c4f867ef6cb2ef150fbe90de57d057dfec36:head@29'47, size: 2098176, copy_subset: [(0, 2098176)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1835008, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:fc3d03e1:::626f73f06d73fccc02b43398389ff991d56d81381f5e170c36fb9439a43cbbc0:head, version: 32'48, data_included: [0~131072], data_size: 131072, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:fc3d03e1:::626f73f06d73fccc02b43398389ff991d56d81381f5e170c36fb9439a43cbbc0:head@32'48, size: 462848, copy_subset: [(0, 462848)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 462848, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:f8934067:::8892b7b523af2cde6be0d91d0468f8be86a651cb052f82daf3e2c6278805eb6e:head, version: 32'49, data_included: [0~131072], data_size: 131072, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:f8934067:::8892b7b523af2cde6be0d91d0468f8be86a651cb052f82daf3e2c6278805eb6e:head@32'49, size: 202240, copy_subset: [(0, 202240)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 202240, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:28.399 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:04.108513+0000", 2024-03-19T11:31:28.399 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.256961341, 2024-03-19T11:31:28.399 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.036668899999999997, 2024-03-19T11:31:28.399 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:28.399 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:28.399 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:28.399 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.400 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:28.400 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.108513+0000", 2024-03-19T11:31:28.400 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:28.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.400 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.400 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:28.400 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.108513+0000", 2024-03-19T11:31:28.400 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:28.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.400 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.400 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:28.400 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.108515+0000", 2024-03-19T11:31:28.400 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7579999999999999e-06 2024-03-19T11:31:28.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.400 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.401 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:28.401 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115352+0000", 2024-03-19T11:31:28.401 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0068369950000000002 2024-03-19T11:31:28.401 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.401 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.401 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:28.401 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115421+0000", 2024-03-19T11:31:28.401 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9282999999999994e-05 2024-03-19T11:31:28.401 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.401 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.401 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:28.401 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115505+0000", 2024-03-19T11:31:28.401 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3714999999999994e-05 2024-03-19T11:31:28.401 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.401 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.402 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:28.402 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115948+0000", 2024-03-19T11:31:28.402 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000443036 2024-03-19T11:31:28.402 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.402 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.402 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:28.402 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116012+0000", 2024-03-19T11:31:28.402 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3905999999999994e-05 2024-03-19T11:31:28.402 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.402 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.402 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:28.402 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.145182+0000", 2024-03-19T11:31:28.403 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.029170207 2024-03-19T11:31:28.403 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:28.403 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:28.403 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:28.403 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.403 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.403 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.4.13:82 3.3 e85/84 3:f6b6d21a:::smithi03829856-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 85'103, mlcod=0'0)", 2024-03-19T11:31:28.403 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:04.115589+0000", 2024-03-19T11:31:28.403 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.249885034999998, 2024-03-19T11:31:28.403 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035569142999999998, 2024-03-19T11:31:28.403 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:28.403 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:28.403 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:28.403 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.403 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:28.404 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115589+0000", 2024-03-19T11:31:28.404 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:28.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.404 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.404 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:28.404 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115589+0000", 2024-03-19T11:31:28.404 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:28.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.404 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.404 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:28.404 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115591+0000", 2024-03-19T11:31:28.404 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2780000000000001e-06 2024-03-19T11:31:28.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.405 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.405 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:28.405 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115602+0000", 2024-03-19T11:31:28.405 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1797000000000001e-05 2024-03-19T11:31:28.405 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.405 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.405 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:28.405 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115613+0000", 2024-03-19T11:31:28.405 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1025e-05 2024-03-19T11:31:28.405 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.405 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.405 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:28.405 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115619+0000", 2024-03-19T11:31:28.405 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.976e-06 2024-03-19T11:31:28.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.406 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:28.406 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.145267+0000", 2024-03-19T11:31:28.406 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.02964791 2024-03-19T11:31:28.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.406 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:28.406 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.145304+0000", 2024-03-19T11:31:28.406 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6455000000000002e-05 2024-03-19T11:31:28.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.406 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:28.406 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.151114+0000", 2024-03-19T11:31:28.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0058103139999999996 2024-03-19T11:31:28.407 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.407 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.407 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:28.407 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.151158+0000", 2024-03-19T11:31:28.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4388000000000003e-05 2024-03-19T11:31:28.407 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:28.407 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:28.407 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:28.407 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.407 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.407 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.4.13:84 3.3 e85/84 3:ce6951d3:::smithi03829856-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 85'104, mlcod=0'0)", 2024-03-19T11:31:28.407 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:04.115824+0000", 2024-03-19T11:31:28.407 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.249650032000002, 2024-03-19T11:31:28.408 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035373398, 2024-03-19T11:31:28.408 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:28.408 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:28.408 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:28.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.408 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:28.408 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115824+0000", 2024-03-19T11:31:28.408 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:28.408 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.408 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:28.408 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115824+0000", 2024-03-19T11:31:28.408 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:28.408 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.409 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:28.409 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115825+0000", 2024-03-19T11:31:28.409 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3399999999999999e-07 2024-03-19T11:31:28.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.409 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.409 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:28.409 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115861+0000", 2024-03-19T11:31:28.409 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6303000000000001e-05 2024-03-19T11:31:28.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.409 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.409 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:28.409 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115869+0000", 2024-03-19T11:31:28.409 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0350000000000001e-06 2024-03-19T11:31:28.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.410 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:28.410 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115874+0000", 2024-03-19T11:31:28.410 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1599999999999997e-06 2024-03-19T11:31:28.410 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.410 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:28.410 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.145686+0000", 2024-03-19T11:31:28.410 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.029811252 2024-03-19T11:31:28.410 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.410 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:28.410 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.145706+0000", 2024-03-19T11:31:28.410 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0264000000000001e-05 2024-03-19T11:31:28.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.411 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:28.411 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.151161+0000", 2024-03-19T11:31:28.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054555280000000003 2024-03-19T11:31:28.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.411 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:28.411 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.151198+0000", 2024-03-19T11:31:28.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6322000000000002e-05 2024-03-19T11:31:28.411 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:28.411 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:28.411 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:28.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.412 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.4.13:85 3.3 e85/84 3:ce6951d3:::smithi03829856-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 85'105, mlcod=0'0)", 2024-03-19T11:31:28.412 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:04.115887+0000", 2024-03-19T11:31:28.412 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.249587418000001, 2024-03-19T11:31:28.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035346673000000002, 2024-03-19T11:31:28.412 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:28.412 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:28.412 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:28.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:28.412 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115887+0000", 2024-03-19T11:31:28.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:28.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:28.412 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115887+0000", 2024-03-19T11:31:28.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:28.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.413 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:28.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115887+0000", 2024-03-19T11:31:28.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2300000000000002e-07 2024-03-19T11:31:28.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.413 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:28.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115900+0000", 2024-03-19T11:31:28.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3044000000000001e-05 2024-03-19T11:31:28.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.413 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:28.414 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115906+0000", 2024-03-19T11:31:28.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5929999999999996e-06 2024-03-19T11:31:28.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.414 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:28.414 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115910+0000", 2024-03-19T11:31:28.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9269999999999998e-06 2024-03-19T11:31:28.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.414 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:28.414 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.145919+0000", 2024-03-19T11:31:28.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030009650999999998 2024-03-19T11:31:28.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:28.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.145948+0000", 2024-03-19T11:31:28.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9047000000000002e-05 2024-03-19T11:31:28.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:28.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.151200+0000", 2024-03-19T11:31:28.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052512970000000003 2024-03-19T11:31:28.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:28.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.151234+0000", 2024-03-19T11:31:28.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3791000000000002e-05 2024-03-19T11:31:28.415 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:28.416 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:28.416 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:28.416 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.416 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(3.3 85/84 [PushOp(3:fcdd1002:::06a0957e01d2e04d1b09d0c04f111819905dcf8372f2c6bee5014c22d5b575ef:head, version: 34'50, data_included: [0~458752], data_size: 458752, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:fcdd1002:::06a0957e01d2e04d1b09d0c04f111819905dcf8372f2c6bee5014c22d5b575ef:head@34'50, size: 689152, copy_subset: [(0, 689152)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 689152, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:28.416 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:04.115917+0000", 2024-03-19T11:31:28.416 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.249556986999998, 2024-03-19T11:31:28.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033796169000000001, 2024-03-19T11:31:28.417 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:28.418 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:28.418 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:28.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:28.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115917+0000", 2024-03-19T11:31:28.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:28.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:28.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115917+0000", 2024-03-19T11:31:28.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:28.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:28.418 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115918+0000", 2024-03-19T11:31:28.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8000000000000002e-07 2024-03-19T11:31:28.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:28.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116163+0000", 2024-03-19T11:31:28.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00024516400000000003 2024-03-19T11:31:28.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:28.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116181+0000", 2024-03-19T11:31:28.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8372999999999998e-05 2024-03-19T11:31:28.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:28.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116196+0000", 2024-03-19T11:31:28.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5041e-05 2024-03-19T11:31:28.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:28.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.147578+0000", 2024-03-19T11:31:28.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031382284000000003 2024-03-19T11:31:28.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:28.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.147594+0000", 2024-03-19T11:31:28.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5153e-05 2024-03-19T11:31:28.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.421 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.421 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:28.421 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.149713+0000", 2024-03-19T11:31:28.421 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021198739999999999 2024-03-19T11:31:28.421 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:28.421 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:28.421 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:28.421 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.421 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.421 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.4.13:83 3.3 e85/84 3:f6b6d21a:::smithi03829856-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 85'106, mlcod=0'0)", 2024-03-19T11:31:28.421 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:04.116216+0000", 2024-03-19T11:31:28.421 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.249257927999999, 2024-03-19T11:31:28.421 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035049444999999999, 2024-03-19T11:31:28.421 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:28.421 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:28.422 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:28.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.422 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:28.422 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116216+0000", 2024-03-19T11:31:28.422 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:28.422 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.422 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:28.422 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116216+0000", 2024-03-19T11:31:28.422 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:28.422 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.422 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:28.422 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116217+0000", 2024-03-19T11:31:28.422 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.22e-07 2024-03-19T11:31:28.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:28.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116221+0000", 2024-03-19T11:31:28.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5349999999999998e-06 2024-03-19T11:31:28.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:28.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116237+0000", 2024-03-19T11:31:28.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5532e-05 2024-03-19T11:31:28.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:28.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116245+0000", 2024-03-19T11:31:28.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0450000000000001e-06 2024-03-19T11:31:28.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:28.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.146739+0000", 2024-03-19T11:31:28.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030494351999999999 2024-03-19T11:31:28.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:28.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.146776+0000", 2024-03-19T11:31:28.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6894999999999998e-05 2024-03-19T11:31:28.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:28.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.151237+0000", 2024-03-19T11:31:28.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044605160000000003 2024-03-19T11:31:28.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:28.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.151266+0000", 2024-03-19T11:31:28.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9247999999999998e-05 2024-03-19T11:31:28.425 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:28.425 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:28.425 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:28.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.425 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3194 4.d 4:b65e6658:::smithi03829856-9:head [read 0~3714347,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108)", 2024-03-19T11:31:28.425 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.202652+0000", 2024-03-19T11:31:28.425 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.1628223390000001, 2024-03-19T11:31:28.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031511255000000002, 2024-03-19T11:31:28.426 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:28.426 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:28.426 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:28.426 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:28.426 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:28.426 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3194 2024-03-19T11:31:28.426 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.426 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:28.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.426 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:28.426 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202652+0000", 2024-03-19T11:31:28.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:28.426 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.426 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:28.427 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202652+0000", 2024-03-19T11:31:28.427 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:28.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.427 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:28.427 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202653+0000", 2024-03-19T11:31:28.427 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3960000000000001e-06 2024-03-19T11:31:28.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.427 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:28.427 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202661+0000", 2024-03-19T11:31:28.427 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0830000000000004e-06 2024-03-19T11:31:28.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.428 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:28.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202679+0000", 2024-03-19T11:31:28.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7972e-05 2024-03-19T11:31:28.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.428 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:28.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202693+0000", 2024-03-19T11:31:28.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4031e-05 2024-03-19T11:31:28.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.428 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:28.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.203299+0000", 2024-03-19T11:31:28.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00060564400000000002 2024-03-19T11:31:28.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:28.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.203528+0000", 2024-03-19T11:31:28.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000229049 2024-03-19T11:31:28.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:28.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.234163+0000", 2024-03-19T11:31:28.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030635079999999999 2024-03-19T11:31:28.429 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:28.429 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:28.429 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:28.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.430 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3195 4.d 4:b65e6658:::smithi03829856-9:head [read 0~3714347,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108)", 2024-03-19T11:31:28.430 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.202714+0000", 2024-03-19T11:31:28.430 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.162760308, 2024-03-19T11:31:28.430 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.028551382, 2024-03-19T11:31:28.430 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:28.430 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:28.430 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:28.430 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:28.430 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:28.430 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3195 2024-03-19T11:31:28.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.430 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:28.430 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:28.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202714+0000", 2024-03-19T11:31:28.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:28.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:28.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202714+0000", 2024-03-19T11:31:28.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:28.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:28.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202715+0000", 2024-03-19T11:31:28.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.203e-06 2024-03-19T11:31:28.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.432 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:28.432 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202722+0000", 2024-03-19T11:31:28.432 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1289999999999998e-06 2024-03-19T11:31:28.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.432 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:28.432 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202739+0000", 2024-03-19T11:31:28.432 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6929e-05 2024-03-19T11:31:28.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.432 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:28.432 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202754+0000", 2024-03-19T11:31:28.432 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4642e-05 2024-03-19T11:31:28.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.433 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:28.433 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.227218+0000", 2024-03-19T11:31:28.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.024464587999999999 2024-03-19T11:31:28.433 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.433 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:28.433 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.227471+0000", 2024-03-19T11:31:28.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000252747 2024-03-19T11:31:28.433 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.433 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:28.433 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.231265+0000", 2024-03-19T11:31:28.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037941440000000002 2024-03-19T11:31:28.434 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:28.434 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:28.434 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:28.434 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.434 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.434 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3196 4.d 4:b65e6658:::smithi03829856-9:head [sparse-read 0~2135368] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108)", 2024-03-19T11:31:28.434 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.202774+0000", 2024-03-19T11:31:28.434 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.1627005669999999, 2024-03-19T11:31:28.434 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033044519000000001, 2024-03-19T11:31:28.434 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:28.434 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:28.434 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:28.434 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:28.435 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:28.435 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3196 2024-03-19T11:31:28.435 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.435 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:28.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.435 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:28.435 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202774+0000", 2024-03-19T11:31:28.435 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:28.435 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.435 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:28.435 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202774+0000", 2024-03-19T11:31:28.435 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:28.435 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.436 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:28.436 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202775+0000", 2024-03-19T11:31:28.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.423e-06 2024-03-19T11:31:28.436 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.436 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.436 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:28.436 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202782+0000", 2024-03-19T11:31:28.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3660000000000003e-06 2024-03-19T11:31:28.436 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.436 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.436 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:28.436 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202801+0000", 2024-03-19T11:31:28.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8182e-05 2024-03-19T11:31:28.436 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.437 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.437 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:28.437 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202815+0000", 2024-03-19T11:31:28.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4042e-05 2024-03-19T11:31:28.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.437 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.437 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:28.437 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.234294+0000", 2024-03-19T11:31:28.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031479493999999997 2024-03-19T11:31:28.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.437 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.437 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:28.437 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.234670+0000", 2024-03-19T11:31:28.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000375595 2024-03-19T11:31:28.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.438 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:28.438 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.235818+0000", 2024-03-19T11:31:28.438 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011484170000000001 2024-03-19T11:31:28.438 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:28.438 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:28.438 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:28.438 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.438 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(osd.0.13:71 3.3 3:e22267ea:::smithi03829856-38 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e108)", 2024-03-19T11:31:28.438 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.209268+0000", 2024-03-19T11:31:28.438 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.156206316, 2024-03-19T11:31:28.438 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.038534928000000003, 2024-03-19T11:31:28.438 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:28.439 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:28.439 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:28.439 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.439 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:28.439 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.209268+0000", 2024-03-19T11:31:28.439 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:28.439 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.439 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.439 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:28.439 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.209268+0000", 2024-03-19T11:31:28.439 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:28.439 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.439 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.439 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:28.439 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.209272+0000", 2024-03-19T11:31:28.439 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0500000000000002e-06 2024-03-19T11:31:28.440 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.440 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.440 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:28.440 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.209291+0000", 2024-03-19T11:31:28.440 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8734999999999999e-05 2024-03-19T11:31:28.440 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.440 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.440 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:28.440 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.209329+0000", 2024-03-19T11:31:28.440 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8223000000000003e-05 2024-03-19T11:31:28.440 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.440 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.440 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:28.440 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.209354+0000", 2024-03-19T11:31:28.441 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5114e-05 2024-03-19T11:31:28.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.441 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.441 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:28.441 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.209573+0000", 2024-03-19T11:31:28.441 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021842000000000001 2024-03-19T11:31:28.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.441 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.441 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:28.441 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.209814+0000", 2024-03-19T11:31:28.441 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00024179400000000001 2024-03-19T11:31:28.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.441 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.441 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:28.442 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.210284+0000", 2024-03-19T11:31:28.442 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00046991699999999998 2024-03-19T11:31:28.442 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.442 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.442 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:28.442 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.213128+0000", 2024-03-19T11:31:28.442 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002843609 2024-03-19T11:31:28.442 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.442 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.442 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:28.442 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.247417+0000", 2024-03-19T11:31:28.442 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.034289312000000002 2024-03-19T11:31:28.442 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.442 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.442 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:28.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.247692+0000", 2024-03-19T11:31:28.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027449000000000001 2024-03-19T11:31:28.443 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.443 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.443 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:28.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.247803+0000", 2024-03-19T11:31:28.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111264 2024-03-19T11:31:28.443 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:28.443 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:28.443 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:28.443 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.443 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.444 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3198 3.3 3:f117d76c:::3a52856062567f05d8c7df91dad2c51911138f6216e1ec709153bdff536e4abe:head [write 0~927232] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-19T11:31:28.444 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.213434+0000", 2024-03-19T11:31:28.444 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.1520406249999999, 2024-03-19T11:31:28.444 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.039874882, 2024-03-19T11:31:28.444 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:28.444 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:28.444 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:28.444 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:28.444 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:28.444 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3198 2024-03-19T11:31:28.444 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.444 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:28.444 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.444 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:28.445 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.213434+0000", 2024-03-19T11:31:28.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:28.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:28.445 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.213434+0000", 2024-03-19T11:31:28.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:28.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:28.445 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.213436+0000", 2024-03-19T11:31:28.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4430000000000002e-06 2024-03-19T11:31:28.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.446 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:28.446 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.219501+0000", 2024-03-19T11:31:28.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0060649240000000002 2024-03-19T11:31:28.446 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.446 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:28.446 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.219531+0000", 2024-03-19T11:31:28.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0470000000000001e-05 2024-03-19T11:31:28.446 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.446 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:28.446 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.219552+0000", 2024-03-19T11:31:28.447 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1012000000000001e-05 2024-03-19T11:31:28.447 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.447 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.447 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:28.447 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.231849+0000", 2024-03-19T11:31:28.447 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012296105999999999 2024-03-19T11:31:28.447 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.447 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.447 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:28.447 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.232349+0000", 2024-03-19T11:31:28.447 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00050035100000000001 2024-03-19T11:31:28.447 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.447 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.447 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:28.448 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.233102+0000", 2024-03-19T11:31:28.448 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00075262300000000001 2024-03-19T11:31:28.448 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.448 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.448 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:28.448 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.249722+0000", 2024-03-19T11:31:28.448 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016620901 2024-03-19T11:31:28.448 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.448 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.448 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:28.448 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.253185+0000", 2024-03-19T11:31:28.448 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034625340000000002 2024-03-19T11:31:28.448 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.448 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.449 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:28.449 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.253252+0000", 2024-03-19T11:31:28.449 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6538000000000006e-05 2024-03-19T11:31:28.449 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:28.449 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:28.449 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:28.449 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.253309+0000", 2024-03-19T11:31:28.449 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.698e-05 2024-03-19T11:31:28.449 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:28.449 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:28.449 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:28.449 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:28.449 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:28.449 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:28.451 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-19T11:31:28.451 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:28.458+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:28.468 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:31:28.468 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:31:28.485 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:31:28.485 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:31:28.500 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:31:28.501 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:31:28.507 INFO:tasks.rados.rados.0.smithi038.stdout:1460: finishing write tid 4 to smithi03829856-47 2024-03-19T11:31:28.507 INFO:tasks.rados.rados.0.smithi038.stdout:1460: oid 47 updating version 0 to 148 2024-03-19T11:31:28.507 INFO:tasks.rados.rados.0.smithi038.stdout:1460: oid 47 version 148 is already newer than 147 2024-03-19T11:31:28.508 INFO:tasks.rados.rados.0.smithi038.stdout:1460: oid 47 version 148 is already newer than 146 2024-03-19T11:31:28.508 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 47 v 148 (ObjNum 251 snap 32 seq_num 251) dirty exists 2024-03-19T11:31:28.508 INFO:tasks.rados.rados.0.smithi038.stdout:1460: left oid 47 (ObjNum 251 snap 32 seq_num 251) 2024-03-19T11:31:28.508 INFO:tasks.rados.rados.0.smithi038.stdout:1463: finishing write tid 1 to smithi03829856-44 2024-03-19T11:31:28.508 INFO:tasks.rados.rados.0.smithi038.stdout:1473: seq_num 254 ranges {668978=552895,1763838=505140} 2024-03-19T11:31:28.514 INFO:tasks.rados.rados.0.smithi038.stdout:1473: writing smithi03829856-19 from 668978 to 1221873 tid 1 2024-03-19T11:31:28.517 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:31:28.517 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:31:28.521 INFO:tasks.rados.rados.0.smithi038.stdout:1473: writing smithi03829856-19 from 1763838 to 2268978 tid 2 2024-03-19T11:31:28.521 INFO:tasks.rados.rados.0.smithi038.stdout:1460: done (13 left) 2024-03-19T11:31:28.521 INFO:tasks.rados.rados.0.smithi038.stdout:1461: done (12 left) 2024-03-19T11:31:28.521 INFO:tasks.rados.rados.0.smithi038.stdout:1474: read oid 24 snap -1 2024-03-19T11:31:28.521 INFO:tasks.rados.rados.0.smithi038.stdout:1474: expect (ObjNum 23 snap 0 seq_num 23) 2024-03-19T11:31:28.521 INFO:tasks.rados.rados.0.smithi038.stdout:1475: write oid 48 current snap is 32 2024-03-19T11:31:28.521 INFO:tasks.rados.rados.0.smithi038.stdout:1463: finishing write tid 2 to smithi03829856-44 2024-03-19T11:31:28.522 INFO:tasks.rados.rados.0.smithi038.stdout:1463: finishing write tid 3 to smithi03829856-44 2024-03-19T11:31:28.522 INFO:tasks.rados.rados.0.smithi038.stdout:1463: finishing write tid 4 to smithi03829856-44 2024-03-19T11:31:28.522 INFO:tasks.rados.rados.0.smithi038.stdout:1463: oid 44 updating version 0 to 113 2024-03-19T11:31:28.522 INFO:tasks.rados.rados.0.smithi038.stdout:1463: oid 44 version 113 is already newer than 111 2024-03-19T11:31:28.522 INFO:tasks.rados.rados.0.smithi038.stdout:1463: oid 44 version 113 is already newer than 112 2024-03-19T11:31:28.522 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 44 v 113 (ObjNum 252 snap 32 seq_num 252) dirty exists 2024-03-19T11:31:28.522 INFO:tasks.rados.rados.0.smithi038.stdout:1463: left oid 44 (ObjNum 252 snap 32 seq_num 252) 2024-03-19T11:31:28.522 INFO:tasks.rados.rados.0.smithi038.stdout:1464: finishing write tid 1 to smithi03829856-36 2024-03-19T11:31:28.522 INFO:tasks.rados.rados.0.smithi038.stdout:1475: seq_num 255 ranges {685785=628144,1822320=463465} 2024-03-19T11:31:28.529 INFO:tasks.rados.rados.0.smithi038.stdout:1475: writing smithi03829856-48 from 685785 to 1313929 tid 1 2024-03-19T11:31:28.536 INFO:tasks.rados.rados.0.smithi038.stdout:1475: writing smithi03829856-48 from 1822320 to 2285785 tid 2 2024-03-19T11:31:28.536 INFO:tasks.rados.rados.0.smithi038.stdout:1463: done (13 left) 2024-03-19T11:31:28.536 INFO:tasks.rados.rados.0.smithi038.stdout:1476: write oid 38 current snap is 32 2024-03-19T11:31:28.537 INFO:tasks.rados.rados.0.smithi038.stdout:1465: got (22) Invalid argument 2024-03-19T11:31:28.537 INFO:tasks.rados.rados.0.smithi038.stdout:1464: finishing write tid 2 to smithi03829856-36 2024-03-19T11:31:28.537 INFO:tasks.rados.rados.0.smithi038.stdout:1464: finishing write tid 3 to smithi03829856-36 2024-03-19T11:31:28.537 INFO:tasks.rados.rados.0.smithi038.stdout:1464: finishing write tid 4 to smithi03829856-36 2024-03-19T11:31:28.537 INFO:tasks.rados.rados.0.smithi038.stdout:1464: oid 36 updating version 0 to 117 2024-03-19T11:31:28.537 INFO:tasks.rados.rados.0.smithi038.stdout:1464: oid 36 updating version 117 to 118 2024-03-19T11:31:28.537 INFO:tasks.rados.rados.0.smithi038.stdout:1464: oid 36 updating version 118 to 119 2024-03-19T11:31:28.537 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 36 v 119 (ObjNum 253 snap 32 seq_num 253) dirty exists 2024-03-19T11:31:28.537 INFO:tasks.rados.rados.0.smithi038.stdout:1464: left oid 36 (ObjNum 253 snap 32 seq_num 253) 2024-03-19T11:31:28.537 INFO:tasks.rados.rados.0.smithi038.stdout:1462: got (0) Success 2024-03-19T11:31:28.537 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 30 v 0 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-03-19T11:31:28.537 INFO:tasks.rados.rados.0.smithi038.stdout:1468: expect (ObjNum 206 snap 22 seq_num 206) 2024-03-19T11:31:28.544 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:31:28.545 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:31:28.552 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:28.562+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:28.562 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:31:28.562 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:31:28.580 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:31:28.580 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:31:28.598 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:31:28.598 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:31:28.625 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:28.625 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:28.639 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-19T11:31:28.680 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:28.690+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:28.695 INFO:tasks.rados.rados.0.smithi038.stdout:1466: expect (ObjNum 72 snap 5 seq_num 72) 2024-03-19T11:31:28.754 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:28.760+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:28.802 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:28.803 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:28.817 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-19T11:31:28.855 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:28.860+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:28.869 INFO:tasks.rados.rados.0.smithi038.stdout:1467: expect (ObjNum 65 snap 3 seq_num 65) 2024-03-19T11:31:28.939 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:28.939 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:28.953 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-19T11:31:28.956 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:28.960+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:29.008 INFO:tasks.rados.rados.0.smithi038.stdout:1469: finishing set_chunk to oid 17 2024-03-19T11:31:29.008 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 17 v 237 (ObjNum 213 snap 22 seq_num 213) dirty exists 2024-03-19T11:31:29.008 INFO:tasks.rados.rados.0.smithi038.stdout:1472: expect (ObjNum 199 snap 22 seq_num 199) 2024-03-19T11:31:29.057 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:29.064+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:29.071 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:29.072 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:29.086 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-19T11:31:29.157 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:29.166+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:29.206 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:29.207 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:29.222 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-19T11:31:29.258 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:29.264+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:29.308 INFO:tasks.rados.rados.0.smithi038.stdout:1470: expect (ObjNum 221 snap 24 seq_num 221) 2024-03-19T11:31:29.359 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:29.364+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:29.384 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:29.385 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:29.399 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-19T11:31:29.460 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:29.468+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:29.518 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:29.519 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:29.533 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-19T11:31:29.545 INFO:tasks.rados.rados.0.smithi038.stdout:1473: finishing write tid 1 to smithi03829856-19 2024-03-19T11:31:29.546 INFO:tasks.rados.rados.0.smithi038.stdout:1473: finishing write tid 2 to smithi03829856-19 2024-03-19T11:31:29.546 INFO:tasks.rados.rados.0.smithi038.stdout:1474: expect (ObjNum 23 snap 0 seq_num 23) 2024-03-19T11:31:29.561 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:29.568+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:29.621 INFO:tasks.rados.rados.0.smithi038.stdout:1473: finishing write tid 3 to smithi03829856-19 2024-03-19T11:31:29.621 INFO:tasks.rados.rados.0.smithi038.stdout:1473: finishing write tid 4 to smithi03829856-19 2024-03-19T11:31:29.622 INFO:tasks.rados.rados.0.smithi038.stdout:1473: oid 19 updating version 0 to 95 2024-03-19T11:31:29.622 INFO:tasks.rados.rados.0.smithi038.stdout:1473: oid 19 version 95 is already newer than 94 2024-03-19T11:31:29.622 INFO:tasks.rados.rados.0.smithi038.stdout:1473: oid 19 version 95 is already newer than 93 2024-03-19T11:31:29.622 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 19 v 95 (ObjNum 254 snap 32 seq_num 254) dirty exists 2024-03-19T11:31:29.622 INFO:tasks.rados.rados.0.smithi038.stdout:1473: left oid 19 (ObjNum 254 snap 32 seq_num 254) 2024-03-19T11:31:29.622 INFO:tasks.rados.rados.0.smithi038.stdout:1475: finishing write tid 1 to smithi03829856-48 2024-03-19T11:31:29.622 INFO:tasks.rados.rados.0.smithi038.stdout:1475: finishing write tid 2 to smithi03829856-48 2024-03-19T11:31:29.622 INFO:tasks.rados.rados.0.smithi038.stdout:1475: finishing write tid 3 to smithi03829856-48 2024-03-19T11:31:29.622 INFO:tasks.rados.rados.0.smithi038.stdout:1475: finishing write tid 4 to smithi03829856-48 2024-03-19T11:31:29.622 INFO:tasks.rados.rados.0.smithi038.stdout:1475: oid 48 updating version 0 to 131 2024-03-19T11:31:29.623 INFO:tasks.rados.rados.0.smithi038.stdout:1475: oid 48 updating version 131 to 132 2024-03-19T11:31:29.623 INFO:tasks.rados.rados.0.smithi038.stdout:1475: oid 48 updating version 132 to 133 2024-03-19T11:31:29.623 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 48 v 133 (ObjNum 255 snap 32 seq_num 255) dirty exists 2024-03-19T11:31:29.623 INFO:tasks.rados.rados.0.smithi038.stdout:1475: left oid 48 (ObjNum 255 snap 32 seq_num 255) 2024-03-19T11:31:29.623 INFO:tasks.rados.rados.0.smithi038.stdout:1476: seq_num 256 ranges {702592=703393,1880802=421790} 2024-03-19T11:31:29.644 INFO:tasks.rados.rados.0.smithi038.stdout:1476: writing smithi03829856-38 from 702592 to 1405985 tid 1 2024-03-19T11:31:29.649 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:29.649 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:29.661 INFO:tasks.rados.rados.0.smithi038.stdout:1476: writing smithi03829856-38 from 1880802 to 2302592 tid 2 2024-03-19T11:31:29.662 INFO:tasks.rados.rados.0.smithi038.stdout:1462: done (13 left) 2024-03-19T11:31:29.662 INFO:tasks.rados.rados.0.smithi038.stdout:1464: done (12 left) 2024-03-19T11:31:29.662 INFO:tasks.rados.rados.0.smithi038.stdout:1465: done (11 left) 2024-03-19T11:31:29.662 INFO:tasks.rados.rados.0.smithi038.stdout:1466: done (10 left) 2024-03-19T11:31:29.662 INFO:tasks.rados.rados.0.smithi038.stdout:1467: done (9 left) 2024-03-19T11:31:29.663 INFO:tasks.rados.rados.0.smithi038.stdout:1468: done (8 left) 2024-03-19T11:31:29.663 INFO:tasks.rados.rados.0.smithi038.stdout:1469: done (7 left) 2024-03-19T11:31:29.663 INFO:tasks.rados.rados.0.smithi038.stdout:1470: done (6 left) 2024-03-19T11:31:29.663 INFO:tasks.rados.rados.0.smithi038.stdout:1471: done (5 left) 2024-03-19T11:31:29.663 INFO:tasks.rados.rados.0.smithi038.stdout:1472: done (4 left) 2024-03-19T11:31:29.663 INFO:tasks.rados.rados.0.smithi038.stdout:1473: done (3 left) 2024-03-19T11:31:29.663 INFO:tasks.rados.rados.0.smithi038.stdout:1474: done (2 left) 2024-03-19T11:31:29.663 INFO:tasks.rados.rados.0.smithi038.stdout:1475: done (1 left) 2024-03-19T11:31:29.663 INFO:tasks.rados.rados.0.smithi038.stdout:1477: write (excl) oid 34 current snap is 32 2024-03-19T11:31:29.664 INFO:tasks.rados.rados.0.smithi038.stdout:1477: seq_num 257 ranges {719399=778642,1939284=380115} 2024-03-19T11:31:29.666 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-19T11:31:29.667 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:29.674+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:29.683 INFO:tasks.rados.rados.0.smithi038.stdout:1477: writing smithi03829856-34 from 719399 to 1498041 tid 1 2024-03-19T11:31:29.693 INFO:tasks.rados.rados.0.smithi038.stdout:1477: writing smithi03829856-34 from 1939284 to 2319399 tid 2 2024-03-19T11:31:29.693 INFO:tasks.rados.rados.0.smithi038.stdout:1478: set_chunk oid 44 target oid 2024-03-19T11:31:29.702 INFO:tasks.rados.rados.0.smithi038.stdout:1478: set_chunk oid 44 offset: 512 length: 149504 target oid 71ba98cb5315c867d5670e7d214b2369d52235d1182658413d3890cfb982be6e offset: 0 snap -1 2024-03-19T11:31:29.702 INFO:tasks.rados.rados.0.smithi038.stdout:1479: snap_remove snap 30 2024-03-19T11:31:29.766 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:29.772+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:29.829 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:29.829 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:29.843 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-19T11:31:29.892 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:29.900+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:29.968 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:29.978+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:29.976 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:29.977 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:29.991 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-19T11:31:30.076 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:30.084+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:30.094 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-03-19T11:31:30.150 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:30.150 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:30.168 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-19T11:31:30.171 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:30.178+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:30.196 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:30.206+0000 7fb17dd72640 -1 osd.4 109 handle_osd_map injecting map crc failure 2024-03-19T11:31:30.197 INFO:tasks.rados.rados.0.smithi038.stdout:1479: done (3 left) 2024-03-19T11:31:30.197 INFO:tasks.rados.rados.0.smithi038.stdout:1480: read oid 13 snap -1 2024-03-19T11:31:30.197 INFO:tasks.rados.rados.0.smithi038.stdout:1480: expect (ObjNum 12 snap 0 seq_num 12) 2024-03-19T11:31:30.197 INFO:tasks.rados.rados.0.smithi038.stdout:1481: rollback oid 6 current snap is 32 2024-03-19T11:31:30.197 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 6 to 31 2024-03-19T11:31:30.197 INFO:tasks.rados.rados.0.smithi038.stdout:1482: set_chunk oid 48 target oid 2024-03-19T11:31:30.230 INFO:tasks.rados.rados.0.smithi038.stdout:1482: set_chunk oid 48 offset: 928768 length: 707584 target oid 54d153ae84ef6629a1f554cb04ec5ef8a88fe5c9524164f0127f4608a5038e8c offset: 0 snap -1 2024-03-19T11:31:30.230 INFO:tasks.rados.rados.0.smithi038.stdout:1483: set_chunk oid 41 target oid 2024-03-19T11:31:30.242 INFO:tasks.rados.rados.0.smithi038.stdout:1483: set_chunk oid 41 offset: 479232 length: 1731584 target oid 7a6f3f2fde314ad4f256a690128330a02fdb0795ca91ac7cdfdc13f6e073bf4e offset: 0 snap 25 2024-03-19T11:31:30.242 INFO:tasks.rados.rados.0.smithi038.stdout:1484: set_chunk oid 27 target oid 2024-03-19T11:31:30.268 INFO:tasks.rados.rados.0.smithi038.stdout:1484: set_chunk oid 27 offset: 121344 length: 1891328 target oid 82961be9db699200ed6e75cf08a4f73fb747551dc2b7f5e32dff1935cac50d4f offset: 0 snap -1 2024-03-19T11:31:30.268 INFO:tasks.rados.rados.0.smithi038.stdout:1485: write (excl) oid 9 current snap is 32 2024-03-19T11:31:30.268 INFO:tasks.rados.rados.0.smithi038.stdout:1476: finishing write tid 1 to smithi03829856-38 2024-03-19T11:31:30.269 INFO:tasks.rados.rados.0.smithi038.stdout:1485: seq_num 258 ranges {736206=453891,1881413=454793} 2024-03-19T11:31:30.271 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:30.276+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:30.274 INFO:tasks.rados.rados.0.smithi038.stdout:1485: writing smithi03829856-9 from 736206 to 1190097 tid 1 2024-03-19T11:31:30.283 INFO:tasks.rados.rados.0.smithi038.stdout:1485: writing smithi03829856-9 from 1881413 to 2336206 tid 2 2024-03-19T11:31:30.283 INFO:tasks.rados.rados.0.smithi038.stdout:1486: rollback oid 23 current snap is 32 2024-03-19T11:31:30.283 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 23 to 25 2024-03-19T11:31:30.283 INFO:tasks.rados.rados.0.smithi038.stdout:1476: finishing write tid 2 to smithi03829856-38 2024-03-19T11:31:30.283 INFO:tasks.rados.rados.0.smithi038.stdout:1476: finishing write tid 3 to smithi03829856-38 2024-03-19T11:31:30.283 INFO:tasks.rados.rados.0.smithi038.stdout:1476: finishing write tid 4 to smithi03829856-38 2024-03-19T11:31:30.283 INFO:tasks.rados.rados.0.smithi038.stdout:1476: oid 38 updating version 0 to 144 2024-03-19T11:31:30.283 INFO:tasks.rados.rados.0.smithi038.stdout:1476: oid 38 updating version 144 to 145 2024-03-19T11:31:30.283 INFO:tasks.rados.rados.0.smithi038.stdout:1476: oid 38 version 145 is already newer than 143 2024-03-19T11:31:30.283 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 38 v 145 (ObjNum 256 snap 32 seq_num 256) dirty exists 2024-03-19T11:31:30.283 INFO:tasks.rados.rados.0.smithi038.stdout:1476: left oid 38 (ObjNum 256 snap 32 seq_num 256) 2024-03-19T11:31:30.283 INFO:tasks.rados.rados.0.smithi038.stdout:1477: finishing write tid 1 to smithi03829856-34 2024-03-19T11:31:30.283 INFO:tasks.rados.rados.0.smithi038.stdout:1477: finishing write tid 2 to smithi03829856-34 2024-03-19T11:31:30.283 INFO:tasks.rados.rados.0.smithi038.stdout:1477: finishing write tid 3 to smithi03829856-34 2024-03-19T11:31:30.284 INFO:tasks.rados.rados.0.smithi038.stdout:1477: finishing write tid 4 to smithi03829856-34 2024-03-19T11:31:30.284 INFO:tasks.rados.rados.0.smithi038.stdout:1477: oid 34 updating version 0 to 41 2024-03-19T11:31:30.284 INFO:tasks.rados.rados.0.smithi038.stdout:1477: oid 34 version 41 is already newer than 40 2024-03-19T11:31:30.284 INFO:tasks.rados.rados.0.smithi038.stdout:1477: oid 34 version 41 is already newer than 39 2024-03-19T11:31:30.284 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 34 v 41 (ObjNum 257 snap 32 seq_num 257) dirty exists 2024-03-19T11:31:30.284 INFO:tasks.rados.rados.0.smithi038.stdout:1477: left oid 34 (ObjNum 257 snap 32 seq_num 257) 2024-03-19T11:31:30.284 INFO:tasks.rados.rados.0.smithi038.stdout:1478: finishing set_chunk to oid 44 2024-03-19T11:31:30.284 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 44 v 113 (ObjNum 252 snap 32 seq_num 252) dirty exists 2024-03-19T11:31:30.284 INFO:tasks.rados.rados.0.smithi038.stdout:1481: finishing rollback tid 0 to smithi03829856-6 2024-03-19T11:31:30.284 INFO:tasks.rados.rados.0.smithi038.stdout:1481: finishing rollback tid 1 to smithi03829856-6 2024-03-19T11:31:30.284 INFO:tasks.rados.rados.0.smithi038.stdout:1481: finishing rollback tid 2 to smithi03829856-6 2024-03-19T11:31:30.284 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 6 v 123 (ObjNum 195 snap 22 seq_num 195) dirty exists 2024-03-19T11:31:30.284 INFO:tasks.rados.rados.0.smithi038.stdout:1482: finishing set_chunk to oid 48 2024-03-19T11:31:30.284 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 48 v 133 (ObjNum 255 snap 32 seq_num 255) dirty exists 2024-03-19T11:31:30.284 INFO:tasks.rados.rados.0.smithi038.stdout:1483: finishing set_chunk to oid 41 2024-03-19T11:31:30.284 INFO:tasks.rados.rados.0.smithi038.stdout:1484: finishing set_chunk to oid 27 2024-03-19T11:31:30.284 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 27 v 122 (ObjNum 240 snap 30 seq_num 240) dirty exists 2024-03-19T11:31:30.285 INFO:tasks.rados.rados.0.smithi038.stdout:1476: done (9 left) 2024-03-19T11:31:30.285 INFO:tasks.rados.rados.0.smithi038.stdout:1477: done (8 left) 2024-03-19T11:31:30.285 INFO:tasks.rados.rados.0.smithi038.stdout:1478: done (7 left) 2024-03-19T11:31:30.285 INFO:tasks.rados.rados.0.smithi038.stdout:1481: done (6 left) 2024-03-19T11:31:30.285 INFO:tasks.rados.rados.0.smithi038.stdout:1482: done (5 left) 2024-03-19T11:31:30.285 INFO:tasks.rados.rados.0.smithi038.stdout:1483: done (4 left) 2024-03-19T11:31:30.285 INFO:tasks.rados.rados.0.smithi038.stdout:1484: done (3 left) 2024-03-19T11:31:30.285 INFO:tasks.rados.rados.0.smithi038.stdout:1487: delete oid 8 current snap is 32 2024-03-19T11:31:30.295 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:30.295 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:30.297 INFO:tasks.rados.rados.0.smithi038.stdout:1487: done (3 left) 2024-03-19T11:31:30.297 INFO:tasks.rados.rados.0.smithi038.stdout:1488: read oid 41 snap -1 2024-03-19T11:31:30.297 INFO:tasks.rados.rados.0.smithi038.stdout:1488: expect (ObjNum 170 snap 17 seq_num 170) 2024-03-19T11:31:30.297 INFO:tasks.rados.rados.0.smithi038.stdout:1489: tier_promote oid 48 2024-03-19T11:31:30.297 INFO:tasks.rados.rados.0.smithi038.stdout:1490: read oid 35 snap -1 2024-03-19T11:31:30.297 INFO:tasks.rados.rados.0.smithi038.stdout:1490: expect deleted 2024-03-19T11:31:30.297 INFO:tasks.rados.rados.0.smithi038.stdout:1491: read oid 20 snap -1 2024-03-19T11:31:30.297 INFO:tasks.rados.rados.0.smithi038.stdout:1491: expect deleted 2024-03-19T11:31:30.297 INFO:tasks.rados.rados.0.smithi038.stdout:1492: rollback oid 10 current snap is 32 2024-03-19T11:31:30.297 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 10 to 28 2024-03-19T11:31:30.297 INFO:tasks.rados.rados.0.smithi038.stdout:1493: snap_create 2024-03-19T11:31:30.304 INFO:tasks.rados.rados.0.smithi038.stdout:1485: finishing write tid 1 to smithi03829856-9 2024-03-19T11:31:30.304 INFO:tasks.rados.rados.0.smithi038.stdout:1485: finishing write tid 2 to smithi03829856-9 2024-03-19T11:31:30.304 INFO:tasks.rados.rados.0.smithi038.stdout:1485: finishing write tid 3 to smithi03829856-9 2024-03-19T11:31:30.304 INFO:tasks.rados.rados.0.smithi038.stdout:1485: finishing write tid 4 to smithi03829856-9 2024-03-19T11:31:30.304 INFO:tasks.rados.rados.0.smithi038.stdout:1485: oid 9 updating version 0 to 103 2024-03-19T11:31:30.304 INFO:tasks.rados.rados.0.smithi038.stdout:1485: oid 9 version 103 is already newer than 102 2024-03-19T11:31:30.304 INFO:tasks.rados.rados.0.smithi038.stdout:1485: oid 9 updating version 103 to 104 2024-03-19T11:31:30.304 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 9 v 104 (ObjNum 258 snap 32 seq_num 258) dirty exists 2024-03-19T11:31:30.304 INFO:tasks.rados.rados.0.smithi038.stdout:1485: left oid 9 (ObjNum 258 snap 32 seq_num 258) 2024-03-19T11:31:30.304 INFO:tasks.rados.rados.0.smithi038.stdout:1488: expect (ObjNum 170 snap 17 seq_num 170) 2024-03-19T11:31:30.315 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-19T11:31:30.372 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:30.382+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:30.396 INFO:tasks.rados.rados.0.smithi038.stdout:1486: finishing rollback tid 0 to smithi03829856-23 2024-03-19T11:31:30.396 INFO:tasks.rados.rados.0.smithi038.stdout:1486: finishing rollback tid 1 to smithi03829856-23 2024-03-19T11:31:30.396 INFO:tasks.rados.rados.0.smithi038.stdout:1486: finishing rollback tid 2 to smithi03829856-23 2024-03-19T11:31:30.396 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 23 v 131 (ObjNum 224 snap 25 seq_num 224) dirty exists 2024-03-19T11:31:30.396 INFO:tasks.rados.rados.0.smithi038.stdout:1492: finishing rollback tid 0 to smithi03829856-10 2024-03-19T11:31:30.397 INFO:tasks.rados.rados.0.smithi038.stdout:1492: finishing rollback tid 1 to smithi03829856-10 2024-03-19T11:31:30.397 INFO:tasks.rados.rados.0.smithi038.stdout:1492: finishing rollback tid 2 to smithi03829856-10 2024-03-19T11:31:30.397 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 10 v 149 (ObjNum 179 snap 19 seq_num 179) dirty exists 2024-03-19T11:31:30.397 INFO:tasks.rados.rados.0.smithi038.stdout:1489: got (0) Success 2024-03-19T11:31:30.397 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 48 v 133 (ObjNum 255 snap 32 seq_num 255) dirty exists 2024-03-19T11:31:30.473 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:30.482+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:30.477 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:30.477 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:30.491 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-19T11:31:30.574 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:30.582+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:30.615 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-19T11:31:30.652 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:30.652 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:30.666 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-19T11:31:30.712 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:30.720+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:30.751 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:31:30.752 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:31:30.769 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:31:30.770 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:31:30.775 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:30.780+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:30.786 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:31:30.786 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:31:30.802 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:31:30.802 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:31:30.822 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:31:30.823 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:31:30.830 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:30.831 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:31:30.831 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:31:30.831 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:31:30.831 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.831 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1063 4.4 4:214026b3:::smithi03829856-47:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:31:30.831 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.784904+0000", 2024-03-19T11:31:30.831 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 77.055915948999996, 2024-03-19T11:31:30.831 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024425912000000001, 2024-03-19T11:31:30.831 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:30.831 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:30.831 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:30.831 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:30.831 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:30.831 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1063 2024-03-19T11:31:30.832 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.832 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:30.832 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.832 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:30.832 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784904+0000", 2024-03-19T11:31:30.832 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:30.832 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.832 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.832 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:30.832 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784904+0000", 2024-03-19T11:31:30.832 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:30.832 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.832 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.832 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:30.833 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784906+0000", 2024-03-19T11:31:30.833 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1409999999999999e-06 2024-03-19T11:31:30.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.833 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.833 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:30.833 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784915+0000", 2024-03-19T11:31:30.833 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5459999999999992e-06 2024-03-19T11:31:30.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.833 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.833 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:30.833 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784935+0000", 2024-03-19T11:31:30.833 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0310999999999999e-05 2024-03-19T11:31:30.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.833 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.833 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:30.834 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784946+0000", 2024-03-19T11:31:30.834 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0675e-05 2024-03-19T11:31:30.834 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.834 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.834 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:30.834 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.785032+0000", 2024-03-19T11:31:30.834 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6049000000000002e-05 2024-03-19T11:31:30.834 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.834 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.834 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:30.834 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.785136+0000", 2024-03-19T11:31:30.834 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000104209 2024-03-19T11:31:30.834 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.834 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.835 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:30.835 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.809330+0000", 2024-03-19T11:31:30.835 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024193981 2024-03-19T11:31:30.835 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:30.835 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:30.835 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:30.835 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.835 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.835 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1433 4.16 4:6be89c06:::smithi03829856-24:c [sparse-read 0~1100427,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:31:30.835 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.976652+0000", 2024-03-19T11:31:30.835 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 64.864168247999999, 2024-03-19T11:31:30.835 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95684753199999995, 2024-03-19T11:31:30.835 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:30.836 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:30.836 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:30.836 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:30.836 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:30.836 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1433 2024-03-19T11:31:30.836 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.836 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:30.836 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.836 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:30.836 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976652+0000", 2024-03-19T11:31:30.836 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:30.836 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.836 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.836 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:30.836 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976652+0000", 2024-03-19T11:31:30.837 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:30.837 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.837 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.837 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:30.837 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976654+0000", 2024-03-19T11:31:30.837 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6199999999999999e-06 2024-03-19T11:31:30.837 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.837 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.837 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:30.837 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976668+0000", 2024-03-19T11:31:30.837 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3213999999999999e-05 2024-03-19T11:31:30.837 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.837 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.837 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:30.838 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976694+0000", 2024-03-19T11:31:30.838 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6488000000000001e-05 2024-03-19T11:31:30.838 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.838 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.838 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:30.838 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976705+0000", 2024-03-19T11:31:30.838 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1044e-05 2024-03-19T11:31:30.838 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.838 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.838 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:30.838 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976795+0000", 2024-03-19T11:31:30.838 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0283000000000003e-05 2024-03-19T11:31:30.838 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.838 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.838 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:30.839 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976811+0000", 2024-03-19T11:31:30.839 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5262000000000001e-05 2024-03-19T11:31:30.839 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.839 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.839 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:30.839 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.927337+0000", 2024-03-19T11:31:30.839 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95052597100000003 2024-03-19T11:31:30.839 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.839 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.839 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:30.839 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.927495+0000", 2024-03-19T11:31:30.839 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015788600000000001 2024-03-19T11:31:30.839 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.839 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.840 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:30.840 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.933499+0000", 2024-03-19T11:31:30.840 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0060047640000000001 2024-03-19T11:31:30.840 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:30.840 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:30.840 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:30.840 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.840 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.840 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1434 4.16 4:6be89c06:::smithi03829856-24:c [read 0~2386561,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:31:30.840 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.976739+0000", 2024-03-19T11:31:30.840 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 64.864081087000002, 2024-03-19T11:31:30.840 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95781317399999999, 2024-03-19T11:31:30.840 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:30.840 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:30.840 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:30.841 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:30.841 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:30.841 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1434 2024-03-19T11:31:30.842 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.842 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:30.842 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.842 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:30.842 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976739+0000", 2024-03-19T11:31:30.842 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:30.842 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.842 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.842 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:30.842 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976739+0000", 2024-03-19T11:31:30.842 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:30.842 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.842 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.842 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:30.842 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976740+0000", 2024-03-19T11:31:30.843 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8800000000000001e-07 2024-03-19T11:31:30.843 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.843 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.843 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:30.843 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976748+0000", 2024-03-19T11:31:30.843 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8979999999999996e-06 2024-03-19T11:31:30.843 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.843 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.844 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:30.844 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976757+0000", 2024-03-19T11:31:30.844 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9439999999999994e-06 2024-03-19T11:31:30.844 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.844 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.844 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:30.844 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976763+0000", 2024-03-19T11:31:30.844 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5030000000000002e-06 2024-03-19T11:31:30.844 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.844 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.845 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:30.845 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976849+0000", 2024-03-19T11:31:30.845 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5803999999999994e-05 2024-03-19T11:31:30.845 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.845 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.845 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:30.845 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976860+0000", 2024-03-19T11:31:30.845 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0869e-05 2024-03-19T11:31:30.845 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.845 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.845 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:30.845 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.933562+0000", 2024-03-19T11:31:30.845 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95670232300000002 2024-03-19T11:31:30.846 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.846 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.846 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:30.846 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.933693+0000", 2024-03-19T11:31:30.846 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013110699999999999 2024-03-19T11:31:30.846 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.846 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.846 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:30.846 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.934552+0000", 2024-03-19T11:31:30.846 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00085883800000000001 2024-03-19T11:31:30.846 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:30.846 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:30.846 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:30.846 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.847 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.847 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1435 4.16 4:6be89c06:::smithi03829856-24:c [sparse-read 0~1100427] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:31:30.847 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.976775+0000", 2024-03-19T11:31:30.847 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 64.864045415999996, 2024-03-19T11:31:30.847 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95962779499999995, 2024-03-19T11:31:30.847 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:30.847 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:30.847 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:30.847 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:30.847 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:30.847 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1435 2024-03-19T11:31:30.848 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.848 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:30.848 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.848 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:30.848 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976775+0000", 2024-03-19T11:31:30.848 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:30.848 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.848 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.848 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:30.848 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976775+0000", 2024-03-19T11:31:30.848 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:30.848 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.848 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.848 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:30.849 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976775+0000", 2024-03-19T11:31:30.849 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9800000000000003e-07 2024-03-19T11:31:30.849 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.849 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.849 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:30.849 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976779+0000", 2024-03-19T11:31:30.849 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3830000000000001e-06 2024-03-19T11:31:30.849 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.849 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.849 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:30.849 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976785+0000", 2024-03-19T11:31:30.849 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5649999999999998e-06 2024-03-19T11:31:30.849 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.850 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.850 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:30.850 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976790+0000", 2024-03-19T11:31:30.850 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1379999999999998e-06 2024-03-19T11:31:30.850 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.850 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.850 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:30.850 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976889+0000", 2024-03-19T11:31:30.850 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8331000000000004e-05 2024-03-19T11:31:30.850 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.851 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.851 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:30.851 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976894+0000", 2024-03-19T11:31:30.851 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8270000000000003e-06 2024-03-19T11:31:30.851 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.851 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.851 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:30.851 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.935982+0000", 2024-03-19T11:31:30.851 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95908714100000003 2024-03-19T11:31:30.851 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.851 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.852 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:30.852 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.936093+0000", 2024-03-19T11:31:30.852 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000111557 2024-03-19T11:31:30.852 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.852 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.852 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:30.852 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.936402+0000", 2024-03-19T11:31:30.852 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030925499999999998 2024-03-19T11:31:30.852 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:30.852 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:30.852 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:30.853 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.853 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.853 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1442 4.6 4:626acd4d:::smithi03829856-2:head [cache-evict] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:31:30.853 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.978864+0000", 2024-03-19T11:31:30.853 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 64.861955752, 2024-03-19T11:31:30.853 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95708845099999995, 2024-03-19T11:31:30.853 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:30.853 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:30.853 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:30.853 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:30.853 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:30.853 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1442 2024-03-19T11:31:30.854 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.854 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:30.854 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.854 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:30.854 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978864+0000", 2024-03-19T11:31:30.854 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:30.854 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.854 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.854 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:30.854 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978864+0000", 2024-03-19T11:31:30.855 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:30.855 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.855 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.855 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:30.855 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978866+0000", 2024-03-19T11:31:30.855 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0729999999999999e-06 2024-03-19T11:31:30.855 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.855 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.855 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:30.855 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978877+0000", 2024-03-19T11:31:30.856 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0390999999999999e-05 2024-03-19T11:31:30.856 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.856 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.856 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:30.856 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978902+0000", 2024-03-19T11:31:30.856 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5635000000000001e-05 2024-03-19T11:31:30.856 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.856 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.856 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:30.856 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978915+0000", 2024-03-19T11:31:30.856 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2617e-05 2024-03-19T11:31:30.857 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.857 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.857 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:30.857 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.979005+0000", 2024-03-19T11:31:30.857 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0552999999999999e-05 2024-03-19T11:31:30.857 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.857 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.857 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:30.857 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.979021+0000", 2024-03-19T11:31:30.857 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5102999999999999e-05 2024-03-19T11:31:30.857 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.858 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.858 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:30.858 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.926879+0000", 2024-03-19T11:31:30.858 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.94785817900000002 2024-03-19T11:31:30.858 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.858 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.858 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:30.858 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.927277+0000", 2024-03-19T11:31:30.858 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00039819600000000002 2024-03-19T11:31:30.858 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.858 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.859 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:30.859 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.935752+0000", 2024-03-19T11:31:30.859 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0084753420000000003 2024-03-19T11:31:30.859 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.859 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.859 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:30.859 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.935874+0000", 2024-03-19T11:31:30.859 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000121311 2024-03-19T11:31:30.859 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.859 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.859 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:30.860 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.935953+0000", 2024-03-19T11:31:30.860 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9050999999999999e-05 2024-03-19T11:31:30.860 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:30.860 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:30.860 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:30.860 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.860 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.860 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2003 4.16 4:6868af91:::smithi03829856-10:head [tier-promote] snapc 12=[12,11,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:31:30.860 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:45.804046+0000", 2024-03-19T11:31:30.860 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 45.036773838999999, 2024-03-19T11:31:30.860 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018976396, 2024-03-19T11:31:30.861 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:30.861 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:30.861 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:30.861 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:30.861 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:30.861 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2003 2024-03-19T11:31:30.861 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.861 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:30.861 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.861 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:30.861 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804046+0000", 2024-03-19T11:31:30.861 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:30.862 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.862 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.862 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:30.862 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804046+0000", 2024-03-19T11:31:30.862 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:30.862 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.862 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.862 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:30.862 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804052+0000", 2024-03-19T11:31:30.862 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7599999999999999e-06 2024-03-19T11:31:30.863 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.863 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.863 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:30.863 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804090+0000", 2024-03-19T11:31:30.863 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8018000000000001e-05 2024-03-19T11:31:30.863 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.863 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.863 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:30.863 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804256+0000", 2024-03-19T11:31:30.863 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016593599999999999 2024-03-19T11:31:30.863 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.863 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.864 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:30.864 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804300+0000", 2024-03-19T11:31:30.864 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3648e-05 2024-03-19T11:31:30.864 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.864 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.864 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:30.864 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804538+0000", 2024-03-19T11:31:30.864 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000238347 2024-03-19T11:31:30.864 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.864 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.864 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:30.865 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804716+0000", 2024-03-19T11:31:30.865 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000177968 2024-03-19T11:31:30.865 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.865 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.865 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:30.865 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.823023+0000", 2024-03-19T11:31:30.865 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018306718999999999 2024-03-19T11:31:30.865 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:30.865 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:30.865 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:30.865 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.866 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.866 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1d58db11:::smithi03829856-18:6, version: 55'80, data_included: [759683~639358,1904434~409218], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:6@55'80, size: 3159683, copy_subset: [(0, 3159683)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2313652, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1d58db11:::smithi03829856-18:b, version: 58'84, data_included: [443718~955323,1564138~93253], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:b@58'84, size: 3445402, copy_subset: [(0, 3445402)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1657391, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:10, version: 66'95, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:10@66'95, size: 3764735, copy_subset: [(0, 3764735)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:12, version: 66'97, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:12@66'97, size: 2504243, copy_subset: [(0, 2504243)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:c, version: 70'102, data_included: [416807~576167,1542129~472409], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:c@70'102, size: 2857157, copy_subset: [(0, 2857157)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2014538, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:30.866 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.275179+0000", 2024-03-19T11:31:30.866 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 36.565640848999998, 2024-03-19T11:31:30.866 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026559353000000001, 2024-03-19T11:31:30.866 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:30.866 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:30.866 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:30.866 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.866 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:30.866 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.275179+0000", 2024-03-19T11:31:30.867 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:30.867 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.867 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.867 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:30.867 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.275179+0000", 2024-03-19T11:31:30.867 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:30.867 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.867 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.867 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:30.867 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.275196+0000", 2024-03-19T11:31:30.867 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6946999999999999e-05 2024-03-19T11:31:30.867 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.868 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.868 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:30.868 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281456+0000", 2024-03-19T11:31:30.868 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0062600529999999998 2024-03-19T11:31:30.868 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.868 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.868 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:30.868 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281524+0000", 2024-03-19T11:31:30.868 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7971000000000004e-05 2024-03-19T11:31:30.868 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.868 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.868 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:30.869 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281591+0000", 2024-03-19T11:31:30.869 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6704000000000004e-05 2024-03-19T11:31:30.869 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.869 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.869 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:30.869 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281865+0000", 2024-03-19T11:31:30.869 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000274097 2024-03-19T11:31:30.869 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.869 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.869 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:30.869 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281901+0000", 2024-03-19T11:31:30.869 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5777e-05 2024-03-19T11:31:30.869 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.869 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.869 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:30.870 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.301738+0000", 2024-03-19T11:31:30.870 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019837804000000001 2024-03-19T11:31:30.870 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:30.870 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:30.870 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:30.870 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.870 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.870 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.18 e75,74 4:1d58db11:::smithi03829856-18:2,48'76)", 2024-03-19T11:31:30.870 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.281661+0000", 2024-03-19T11:31:30.871 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 36.559159317999999, 2024-03-19T11:31:30.871 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020384337999999998, 2024-03-19T11:31:30.871 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:30.871 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:30.871 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:30.871 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.871 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:30.871 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281661+0000", 2024-03-19T11:31:30.871 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:30.871 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.871 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.871 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:30.871 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281661+0000", 2024-03-19T11:31:30.871 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:30.871 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.872 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.872 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:30.872 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281661+0000", 2024-03-19T11:31:30.872 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9100000000000003e-07 2024-03-19T11:31:30.872 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.872 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.872 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:30.872 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281668+0000", 2024-03-19T11:31:30.872 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8199999999999999e-06 2024-03-19T11:31:30.872 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.872 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.872 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:30.872 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281679+0000", 2024-03-19T11:31:30.872 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1107999999999999e-05 2024-03-19T11:31:30.873 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.873 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.873 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:30.873 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281691+0000", 2024-03-19T11:31:30.873 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1399999999999999e-05 2024-03-19T11:31:30.873 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.873 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.873 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:30.873 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.301804+0000", 2024-03-19T11:31:30.873 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020113669000000001 2024-03-19T11:31:30.873 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.873 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.873 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:30.873 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.301818+0000", 2024-03-19T11:31:30.873 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3718e-05 2024-03-19T11:31:30.874 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.874 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.874 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:30.874 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302045+0000", 2024-03-19T11:31:30.874 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000226832 2024-03-19T11:31:30.874 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:30.874 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:30.874 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:30.874 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.874 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.874 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1dabc0f1:::smithi03829856-1:head, version: 70'103, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:head@70'103, size: 2621892, copy_subset: [(0, 2621892)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:30.874 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.281783+0000", 2024-03-19T11:31:30.874 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 36.559037042, 2024-03-19T11:31:30.874 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026187221, 2024-03-19T11:31:30.875 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:30.875 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:30.875 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:30.875 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.875 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:30.875 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281783+0000", 2024-03-19T11:31:30.875 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:30.875 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.875 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.875 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:30.875 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281783+0000", 2024-03-19T11:31:30.875 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:30.875 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.875 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.875 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:30.875 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281783+0000", 2024-03-19T11:31:30.876 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1699999999999999e-07 2024-03-19T11:31:30.878 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.878 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.878 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:30.878 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.282962+0000", 2024-03-19T11:31:30.878 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001178601 2024-03-19T11:31:30.878 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.878 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.878 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:30.879 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.282987+0000", 2024-03-19T11:31:30.879 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4899e-05 2024-03-19T11:31:30.879 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.879 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.879 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:30.879 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283006+0000", 2024-03-19T11:31:30.879 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9494999999999999e-05 2024-03-19T11:31:30.879 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.879 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.879 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:30.879 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.305757+0000", 2024-03-19T11:31:30.879 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.022750570000000001 2024-03-19T11:31:30.879 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.880 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.880 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:30.880 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.305770+0000", 2024-03-19T11:31:30.880 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2791e-05 2024-03-19T11:31:30.880 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.880 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.880 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:30.880 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.307970+0000", 2024-03-19T11:31:30.880 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022005480000000001 2024-03-19T11:31:30.880 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:30.880 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:30.880 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:30.880 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.880 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.881 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.18 e75,74 4:1d58db11:::smithi03829856-18:c,70'104)", 2024-03-19T11:31:30.881 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.283036+0000", 2024-03-19T11:31:30.881 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 36.557783942, 2024-03-19T11:31:30.881 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019146032, 2024-03-19T11:31:30.881 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:30.881 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:30.881 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:30.881 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.881 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:30.881 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283036+0000", 2024-03-19T11:31:30.881 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:30.881 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.881 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.881 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:30.882 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283036+0000", 2024-03-19T11:31:30.882 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:30.882 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.882 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.882 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:30.882 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283037+0000", 2024-03-19T11:31:30.882 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4899999999999995e-07 2024-03-19T11:31:30.882 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.882 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.882 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:30.882 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283042+0000", 2024-03-19T11:31:30.882 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0930000000000002e-06 2024-03-19T11:31:30.882 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.882 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.883 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:30.883 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283053+0000", 2024-03-19T11:31:30.883 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1083999999999999e-05 2024-03-19T11:31:30.883 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.883 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.883 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:30.883 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283059+0000", 2024-03-19T11:31:30.883 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6329999999999999e-06 2024-03-19T11:31:30.883 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.883 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.883 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:30.883 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302090+0000", 2024-03-19T11:31:30.883 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01903038 2024-03-19T11:31:30.883 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.883 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.884 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:30.884 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302097+0000", 2024-03-19T11:31:30.884 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.221e-06 2024-03-19T11:31:30.884 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.884 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.884 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:30.884 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302182+0000", 2024-03-19T11:31:30.884 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5072000000000001e-05 2024-03-19T11:31:30.884 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:30.884 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:30.884 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:30.884 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.884 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.884 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(osd.5.0:143 4.18 e75/74 4:1d58db11:::smithi03829856-18:head v 75'106, mlcod=0'0)", 2024-03-19T11:31:30.884 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.283083+0000", 2024-03-19T11:31:30.885 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 36.557736615000003, 2024-03-19T11:31:30.885 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025309439, 2024-03-19T11:31:30.885 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:30.885 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:30.885 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:30.885 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.885 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:30.885 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283083+0000", 2024-03-19T11:31:30.885 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:30.885 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.885 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.885 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:30.885 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283083+0000", 2024-03-19T11:31:30.885 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:30.885 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.886 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.886 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:30.886 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283084+0000", 2024-03-19T11:31:30.886 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7800000000000002e-07 2024-03-19T11:31:30.886 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.886 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.886 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:30.886 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.284082+0000", 2024-03-19T11:31:30.886 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0009977880000000001 2024-03-19T11:31:30.886 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.886 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.886 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:30.886 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.284095+0000", 2024-03-19T11:31:30.886 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3256e-05 2024-03-19T11:31:30.887 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.887 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.887 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:30.887 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.284102+0000", 2024-03-19T11:31:30.887 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.605e-06 2024-03-19T11:31:30.887 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.887 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.887 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:30.887 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302285+0000", 2024-03-19T11:31:30.887 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018182423 2024-03-19T11:31:30.887 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.887 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.887 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:30.887 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302341+0000", 2024-03-19T11:31:30.888 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.626e-05 2024-03-19T11:31:30.888 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.888 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.888 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:30.888 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.308338+0000", 2024-03-19T11:31:30.888 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0059968030000000002 2024-03-19T11:31:30.888 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.888 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.888 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:30.888 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.308393+0000", 2024-03-19T11:31:30.888 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4826e-05 2024-03-19T11:31:30.889 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:30.889 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:30.889 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:30.889 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.889 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.889 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "pg_update_log_missing(4.18 epoch 75/74 rep_tid 144 entries 75'107 (0'0) error 4:1d58db11:::smithi03829856-18:head by client.4415.0:2283 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-19T11:31:30.889 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.288252+0000", 2024-03-19T11:31:30.889 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 36.552568065000003, 2024-03-19T11:31:30.889 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020167377, 2024-03-19T11:31:30.889 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:30.889 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-03-19T11:31:30.890 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:30.890 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.890 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:30.890 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288252+0000", 2024-03-19T11:31:30.890 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:30.890 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.890 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.890 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:30.891 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288252+0000", 2024-03-19T11:31:30.891 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:30.891 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.891 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.891 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:30.891 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288253+0000", 2024-03-19T11:31:30.891 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2109999999999999e-06 2024-03-19T11:31:30.891 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.891 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.891 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:30.891 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288268+0000", 2024-03-19T11:31:30.891 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5003e-05 2024-03-19T11:31:30.892 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.892 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.892 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:30.892 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288301+0000", 2024-03-19T11:31:30.892 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2539999999999997e-05 2024-03-19T11:31:30.892 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.892 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.892 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:30.892 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288319+0000", 2024-03-19T11:31:30.892 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8232000000000001e-05 2024-03-19T11:31:30.892 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.893 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.893 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:30.893 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.305490+0000", 2024-03-19T11:31:30.893 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017170857000000001 2024-03-19T11:31:30.893 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.893 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.893 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:30.893 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.308419+0000", 2024-03-19T11:31:30.893 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029295340000000001 2024-03-19T11:31:30.893 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:30.893 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:30.894 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:30.894 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.894 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.894 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1d58db11:::smithi03829856-18:6, version: 55'80, data_included: [2313652~310127,3128657~31026], data_size: 341153, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:6@55'80, size: 3159683, copy_subset: [(0, 3159683)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3159683, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2313652, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:1d58db11:::smithi03829856-18:b, version: 58'84, data_included: [1657391~966388,2633066~82188], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:b@58'84, size: 3445402, copy_subset: [(0, 3445402)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2715254, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1657391, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:10, version: 66'95, data_included: [1829043~668012,3134120~380564], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:10@66'95, size: 3764735, copy_subset: [(0, 3764735)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3514684, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:12, version: 66'97, data_included: [1829043~507467], data_size: 507467, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:12@66'97, size: 2504243, copy_subset: [(0, 2504243)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2504243, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:c, version: 70'102, data_included: [2014538~156250], data_size: 156250, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:c@70'102, size: 2857157, copy_subset: [(0, 2857157)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2857157, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2014538, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:30.894 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.312362+0000", 2024-03-19T11:31:30.894 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 36.528458016000002, 2024-03-19T11:31:30.894 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017968951, 2024-03-19T11:31:30.894 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:30.894 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:30.894 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:30.894 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.895 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:30.895 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.312362+0000", 2024-03-19T11:31:30.895 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:30.895 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.895 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.895 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:30.895 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.312362+0000", 2024-03-19T11:31:30.895 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:30.895 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.895 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.896 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:30.896 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.312363+0000", 2024-03-19T11:31:30.896 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.528e-06 2024-03-19T11:31:30.896 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.896 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.896 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:30.896 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315748+0000", 2024-03-19T11:31:30.896 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033842830000000001 2024-03-19T11:31:30.896 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.896 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.896 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:30.897 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315795+0000", 2024-03-19T11:31:30.897 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7382999999999997e-05 2024-03-19T11:31:30.897 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.897 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.897 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:30.897 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315845+0000", 2024-03-19T11:31:30.897 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0319999999999999e-05 2024-03-19T11:31:30.897 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.897 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.897 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:30.897 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.316123+0000", 2024-03-19T11:31:30.897 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00027747200000000002 2024-03-19T11:31:30.897 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.898 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.898 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:30.898 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.316170+0000", 2024-03-19T11:31:30.898 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7466999999999998e-05 2024-03-19T11:31:30.898 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.898 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.898 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:30.898 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.330331+0000", 2024-03-19T11:31:30.898 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014160498000000001 2024-03-19T11:31:30.898 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:30.899 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:30.899 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:30.899 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.899 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.899 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1dabc0f1:::smithi03829856-1:head, version: 70'103, data_included: [1829043~555153], data_size: 555153, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:head@70'103, size: 2621892, copy_subset: [(0, 2621892)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2621892, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:30.899 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.315898+0000", 2024-03-19T11:31:30.899 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 36.524922029999999, 2024-03-19T11:31:30.899 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016880424000000002, 2024-03-19T11:31:30.899 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:30.899 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:30.899 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:30.899 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.900 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:30.900 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315898+0000", 2024-03-19T11:31:30.900 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:30.900 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.900 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.900 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:30.900 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315898+0000", 2024-03-19T11:31:30.900 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:30.900 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.900 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.901 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:30.901 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315899+0000", 2024-03-19T11:31:30.901 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.003e-06 2024-03-19T11:31:30.901 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.901 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.901 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:30.901 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.316486+0000", 2024-03-19T11:31:30.901 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00058673099999999995 2024-03-19T11:31:30.901 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.901 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.901 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:30.902 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.316519+0000", 2024-03-19T11:31:30.902 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3124000000000003e-05 2024-03-19T11:31:30.902 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.902 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.902 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:30.902 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.316534+0000", 2024-03-19T11:31:30.902 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.485e-05 2024-03-19T11:31:30.902 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.902 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.902 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:30.902 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.330415+0000", 2024-03-19T11:31:30.902 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013880905000000001 2024-03-19T11:31:30.903 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.903 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.903 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:30.903 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.330439+0000", 2024-03-19T11:31:30.903 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4630999999999998e-05 2024-03-19T11:31:30.903 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.903 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.903 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:30.903 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.332778+0000", 2024-03-19T11:31:30.903 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023391800000000002 2024-03-19T11:31:30.903 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:30.904 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:30.904 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:30.904 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.904 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.904 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2387 4.2 4:43b32242:::smithi03829856-22:head [read 596992~684544] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-03-19T11:31:30.904 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:59.319806+0000", 2024-03-19T11:31:30.904 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 31.521014362999999, 2024-03-19T11:31:30.904 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.78144344399999999, 2024-03-19T11:31:30.904 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:30.904 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:30.904 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:30.904 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:30.905 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:30.905 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2387 2024-03-19T11:31:30.905 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.905 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:30.905 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.905 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:30.905 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319806+0000", 2024-03-19T11:31:30.905 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:30.905 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.905 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.905 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:30.905 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319806+0000", 2024-03-19T11:31:30.906 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:30.906 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.906 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.906 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:30.906 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319807+0000", 2024-03-19T11:31:30.906 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3519999999999999e-06 2024-03-19T11:31:30.906 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.906 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.906 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:30.906 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319812+0000", 2024-03-19T11:31:30.906 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3959999999999997e-06 2024-03-19T11:31:30.907 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.907 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.907 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:30.907 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319827+0000", 2024-03-19T11:31:30.907 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4566000000000001e-05 2024-03-19T11:31:30.907 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.907 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.907 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:30.907 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319835+0000", 2024-03-19T11:31:30.907 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.994e-06 2024-03-19T11:31:30.907 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.907 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.907 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:30.908 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.320119+0000", 2024-03-19T11:31:30.908 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00028445199999999998 2024-03-19T11:31:30.908 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.908 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.908 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:30.908 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.320129+0000", 2024-03-19T11:31:30.908 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9580000000000002e-06 2024-03-19T11:31:30.908 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.908 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.908 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:30.908 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.069842+0000", 2024-03-19T11:31:30.908 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.74971218399999995 2024-03-19T11:31:30.908 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.909 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.909 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:31:30.909 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.070010+0000", 2024-03-19T11:31:30.909 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016846600000000001 2024-03-19T11:31:30.909 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.909 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.909 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:30.909 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.090911+0000", 2024-03-19T11:31:30.909 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020901494999999999 2024-03-19T11:31:30.909 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.909 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.909 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:31:30.909 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.091249+0000", 2024-03-19T11:31:30.909 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000337602 2024-03-19T11:31:30.909 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.910 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.910 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:30.910 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.098478+0000", 2024-03-19T11:31:30.910 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0072287549999999999 2024-03-19T11:31:30.910 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.910 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.910 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:30.910 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.098617+0000", 2024-03-19T11:31:30.910 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013871599999999999 2024-03-19T11:31:30.910 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.910 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.910 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:30.910 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.101249+0000", 2024-03-19T11:31:30.910 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002632508 2024-03-19T11:31:30.910 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:30.911 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:30.911 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:30.911 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.911 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.911 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 81/80 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 15=[]:{7: [6], b: [b], 10: [d], 12: [12, 11], 15: [15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:7, version: 55'82, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@55'82, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 16=[]:{7: [6], b: [b], 10: [d], 16: [16, 15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 15=[]:{7: [6], b: [b], 10: [d], 12: [12, 11], 15: [15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 58'86, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@58'86, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 16=[]:{7: [6], b: [b], 10: [d], 16: [16, 15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [504210~968657,1572695~79919], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 16=[]:{7: [6], b: [b], 10: [d], 16: [16, 15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:30.911 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:00.089924+0000", 2024-03-19T11:31:30.911 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 30.750896134000001, 2024-03-19T11:31:30.911 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024686583000000002, 2024-03-19T11:31:30.911 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:30.911 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:30.911 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:30.911 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.911 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:30.912 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.089924+0000", 2024-03-19T11:31:30.912 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:30.912 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.912 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.912 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:30.912 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.089924+0000", 2024-03-19T11:31:30.912 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:30.912 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.912 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.912 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:30.912 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.089926+0000", 2024-03-19T11:31:30.912 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9589999999999998e-06 2024-03-19T11:31:30.912 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.912 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.912 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:30.913 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093311+0000", 2024-03-19T11:31:30.913 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033854409999999999 2024-03-19T11:31:30.913 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.913 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.913 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:30.913 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093365+0000", 2024-03-19T11:31:30.913 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3795000000000002e-05 2024-03-19T11:31:30.913 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.913 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.913 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:30.913 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093438+0000", 2024-03-19T11:31:30.913 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2591999999999999e-05 2024-03-19T11:31:30.913 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.913 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.913 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:30.913 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093690+0000", 2024-03-19T11:31:30.914 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025264999999999999 2024-03-19T11:31:30.914 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.914 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.914 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:30.914 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093731+0000", 2024-03-19T11:31:30.914 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0881999999999999e-05 2024-03-19T11:31:30.914 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.914 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.914 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:30.914 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.114610+0000", 2024-03-19T11:31:30.914 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020879264000000002 2024-03-19T11:31:30.914 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:30.914 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:30.914 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:30.915 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.915 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.915 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.21 e81,80 4:8401c1f0:::smithi03829856-43:8,46'66,4:866f3dcc:::smithi03829856-31:2,47'73)", 2024-03-19T11:31:30.915 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:00.093514+0000", 2024-03-19T11:31:30.915 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 30.747305533999999, 2024-03-19T11:31:30.915 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027375713999999999, 2024-03-19T11:31:30.915 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:30.915 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:30.915 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:30.915 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.915 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:30.915 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093514+0000", 2024-03-19T11:31:30.915 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:30.915 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.915 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.916 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:30.916 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093514+0000", 2024-03-19T11:31:30.916 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:30.916 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.916 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.916 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:30.916 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093516+0000", 2024-03-19T11:31:30.916 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1179999999999999e-06 2024-03-19T11:31:30.916 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.917 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.917 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:30.917 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093523+0000", 2024-03-19T11:31:30.917 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7500000000000003e-06 2024-03-19T11:31:30.917 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.917 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.917 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:30.917 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093534+0000", 2024-03-19T11:31:30.917 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0831e-05 2024-03-19T11:31:30.917 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.917 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.917 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:30.917 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093542+0000", 2024-03-19T11:31:30.917 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0139999999999998e-06 2024-03-19T11:31:30.918 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.918 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.918 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:30.918 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.120682+0000", 2024-03-19T11:31:30.918 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027139838999999999 2024-03-19T11:31:30.918 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.918 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.918 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:30.918 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.120693+0000", 2024-03-19T11:31:30.918 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.136e-05 2024-03-19T11:31:30.918 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.918 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.918 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:30.918 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.120890+0000", 2024-03-19T11:31:30.918 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019680199999999999 2024-03-19T11:31:30.919 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:30.919 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:30.919 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:30.919 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.919 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.919 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2927 4.24 4:266590a5:::smithi03829856-16:head [append 0~0] snapc 1b=[1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-19T11:31:30.919 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:18.054474+0000", 2024-03-19T11:31:30.919 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 12.786346287000001, 2024-03-19T11:31:30.919 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024856390999999999, 2024-03-19T11:31:30.919 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:30.919 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:30.919 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:30.919 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:30.919 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:30.919 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2927 2024-03-19T11:31:30.920 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.920 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:30.920 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.920 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:30.920 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054474+0000", 2024-03-19T11:31:30.920 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:30.920 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.920 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.920 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:30.920 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054474+0000", 2024-03-19T11:31:30.920 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:30.920 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.920 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.920 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:30.920 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054476+0000", 2024-03-19T11:31:30.921 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6469999999999998e-06 2024-03-19T11:31:30.921 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.921 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.921 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:30.921 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054489+0000", 2024-03-19T11:31:30.921 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2612999999999999e-05 2024-03-19T11:31:30.921 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.921 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.921 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:30.921 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054526+0000", 2024-03-19T11:31:30.921 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7481999999999998e-05 2024-03-19T11:31:30.921 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.921 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.921 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:30.921 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054547+0000", 2024-03-19T11:31:30.922 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0820999999999999e-05 2024-03-19T11:31:30.922 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.922 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.922 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:30.922 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054633+0000", 2024-03-19T11:31:30.922 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5556000000000006e-05 2024-03-19T11:31:30.922 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.922 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.922 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-03-19T11:31:30.922 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054639+0000", 2024-03-19T11:31:30.922 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7300000000000002e-06 2024-03-19T11:31:30.923 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.923 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.923 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:30.923 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.061592+0000", 2024-03-19T11:31:30.923 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0069539279999999998 2024-03-19T11:31:30.923 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.923 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.923 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:30.923 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.061746+0000", 2024-03-19T11:31:30.923 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015351699999999999 2024-03-19T11:31:30.923 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.923 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.924 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:30.924 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.062031+0000", 2024-03-19T11:31:30.924 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00028539700000000001 2024-03-19T11:31:30.924 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.924 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.924 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:30.924 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.079164+0000", 2024-03-19T11:31:30.924 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017132865000000001 2024-03-19T11:31:30.924 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.924 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.924 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:30.924 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.079217+0000", 2024-03-19T11:31:30.925 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2373999999999997e-05 2024-03-19T11:31:30.925 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.925 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.925 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:30.925 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.079284+0000", 2024-03-19T11:31:30.925 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7433000000000003e-05 2024-03-19T11:31:30.925 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.925 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.925 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:30.925 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.079330+0000", 2024-03-19T11:31:30.925 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6028e-05 2024-03-19T11:31:30.926 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:30.926 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:30.926 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:30.926 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.926 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.926 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2928 4.24 4:266590a5:::smithi03829856-16:head [rollback b] snapc 1b=[1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-19T11:31:30.926 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:18.054594+0000", 2024-03-19T11:31:30.926 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 12.786226299999999, 2024-03-19T11:31:30.926 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025781921999999999, 2024-03-19T11:31:30.926 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:30.926 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:30.927 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:30.927 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:30.927 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:30.927 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2928 2024-03-19T11:31:30.927 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.927 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:30.927 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.927 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:30.927 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054594+0000", 2024-03-19T11:31:30.927 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:30.927 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.927 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.927 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:30.927 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054594+0000", 2024-03-19T11:31:30.927 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:30.928 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.928 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.928 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:30.928 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054595+0000", 2024-03-19T11:31:30.928 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.018e-06 2024-03-19T11:31:30.928 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.928 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.928 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:30.928 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054601+0000", 2024-03-19T11:31:30.928 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2620000000000004e-06 2024-03-19T11:31:30.928 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.928 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.928 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:30.928 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054618+0000", 2024-03-19T11:31:30.928 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6877000000000001e-05 2024-03-19T11:31:30.929 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.929 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.929 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:30.929 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054632+0000", 2024-03-19T11:31:30.929 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4175000000000001e-05 2024-03-19T11:31:30.929 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.929 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.929 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:30.929 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054676+0000", 2024-03-19T11:31:30.929 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3659000000000003e-05 2024-03-19T11:31:30.929 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.930 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.930 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:31:30.930 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054679+0000", 2024-03-19T11:31:30.930 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0649999999999999e-06 2024-03-19T11:31:30.930 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.930 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.930 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:30.930 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.064021+0000", 2024-03-19T11:31:30.930 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093424210000000001 2024-03-19T11:31:30.930 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.930 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.931 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:30.931 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.064158+0000", 2024-03-19T11:31:30.931 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013652099999999999 2024-03-19T11:31:30.931 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.931 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.931 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:30.931 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.072763+0000", 2024-03-19T11:31:30.931 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.008605784 2024-03-19T11:31:30.931 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.931 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.931 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:30.931 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.079171+0000", 2024-03-19T11:31:30.932 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0064071320000000003 2024-03-19T11:31:30.932 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.932 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.932 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:30.932 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.080257+0000", 2024-03-19T11:31:30.932 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0010866280000000001 2024-03-19T11:31:30.932 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.932 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.932 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:30.932 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.080324+0000", 2024-03-19T11:31:30.932 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6976000000000004e-05 2024-03-19T11:31:30.933 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.933 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.933 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:30.933 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.080376+0000", 2024-03-19T11:31:30.933 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1403999999999998e-05 2024-03-19T11:31:30.933 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:30.933 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:30.933 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:30.933 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.933 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.933 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2929 4.24 4:266590a5:::smithi03829856-16:head [append 0~0] snapc 1b=[1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-19T11:31:30.933 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:18.054649+0000", 2024-03-19T11:31:30.934 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 12.786170833, 2024-03-19T11:31:30.934 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026690519999999999, 2024-03-19T11:31:30.934 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:30.934 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:30.934 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:30.934 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:30.934 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:30.934 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2929 2024-03-19T11:31:30.934 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.934 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:30.934 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.935 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:30.935 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054649+0000", 2024-03-19T11:31:30.935 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:30.935 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.935 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.935 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:30.935 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054649+0000", 2024-03-19T11:31:30.935 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:30.935 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.935 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.935 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:30.935 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054650+0000", 2024-03-19T11:31:30.936 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5199999999999995e-07 2024-03-19T11:31:30.936 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.936 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.936 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:30.936 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054655+0000", 2024-03-19T11:31:30.936 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1529999999999996e-06 2024-03-19T11:31:30.936 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.936 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.936 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:30.936 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054670+0000", 2024-03-19T11:31:30.936 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5029e-05 2024-03-19T11:31:30.937 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.937 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.937 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:30.937 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054683+0000", 2024-03-19T11:31:30.937 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2670000000000001e-05 2024-03-19T11:31:30.937 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.937 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.937 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:30.937 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054756+0000", 2024-03-19T11:31:30.937 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3559000000000005e-05 2024-03-19T11:31:30.937 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.938 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.938 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:31:30.938 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054760+0000", 2024-03-19T11:31:30.938 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1659999999999998e-06 2024-03-19T11:31:30.938 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.938 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.938 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:30.938 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.075329+0000", 2024-03-19T11:31:30.938 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020569918999999999 2024-03-19T11:31:30.938 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.938 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.938 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:30.939 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.075469+0000", 2024-03-19T11:31:30.939 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013990000000000001 2024-03-19T11:31:30.939 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.939 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.939 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:30.939 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.075730+0000", 2024-03-19T11:31:30.939 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00026011300000000002 2024-03-19T11:31:30.939 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.939 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.939 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:30.939 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.079918+0000", 2024-03-19T11:31:30.940 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0041886279999999998 2024-03-19T11:31:30.940 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.940 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.940 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:30.940 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.081271+0000", 2024-03-19T11:31:30.940 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0013529449999999999 2024-03-19T11:31:30.940 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.940 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.940 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:30.940 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.081308+0000", 2024-03-19T11:31:30.940 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6437999999999999e-05 2024-03-19T11:31:30.941 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:30.941 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:30.941 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:30.941 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.081340+0000", 2024-03-19T11:31:30.941 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2147999999999998e-05 2024-03-19T11:31:30.941 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:30.941 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:30.941 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:30.941 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:30.941 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:30.941 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:30.943 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-19T11:31:30.944 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:31:30.944 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:31:30.944 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:31:30.945 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:31:30.945 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:31:30.946 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:31:30.946 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:30.948+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:31.043 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:31.048+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:31.110 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:31.110 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:31:31.110 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:31:31.110 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:31.124 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-19T11:31:31.144 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:31.154+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:31.206 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-03-19T11:31:31.209 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:31.216+0000 7f6712f92640 -1 osd.2 110 handle_osd_map injecting map crc failure 2024-03-19T11:31:31.215 INFO:tasks.rados.rados.0.smithi038.stdout:1485: done (8 left) 2024-03-19T11:31:31.215 INFO:tasks.rados.rados.0.smithi038.stdout:1486: done (7 left) 2024-03-19T11:31:31.215 INFO:tasks.rados.rados.0.smithi038.stdout:1488: done (6 left) 2024-03-19T11:31:31.215 INFO:tasks.rados.rados.0.smithi038.stdout:1489: done (5 left) 2024-03-19T11:31:31.215 INFO:tasks.rados.rados.0.smithi038.stdout:1490: done (4 left) 2024-03-19T11:31:31.215 INFO:tasks.rados.rados.0.smithi038.stdout:1491: done (3 left) 2024-03-19T11:31:31.215 INFO:tasks.rados.rados.0.smithi038.stdout:1492: done (2 left) 2024-03-19T11:31:31.215 INFO:tasks.rados.rados.0.smithi038.stdout:1493: done (1 left) 2024-03-19T11:31:31.216 INFO:tasks.rados.rados.0.smithi038.stdout:1494: read oid 42 snap -1 2024-03-19T11:31:31.216 INFO:tasks.rados.rados.0.smithi038.stdout:1494: expect deleted 2024-03-19T11:31:31.216 INFO:tasks.rados.rados.0.smithi038.stdout:1495: tier_evict oid 6 2024-03-19T11:31:31.216 INFO:tasks.rados.rados.0.smithi038.stdout:1495: tier_evict oid 6 snap -1 2024-03-19T11:31:31.216 INFO:tasks.rados.rados.0.smithi038.stdout:1496: tier_promote oid 18 2024-03-19T11:31:31.216 INFO:tasks.rados.rados.0.smithi038.stdout:1497: read oid 46 snap -1 2024-03-19T11:31:31.216 INFO:tasks.rados.rados.0.smithi038.stdout:1497: expect (ObjNum 143 snap 15 seq_num 143) 2024-03-19T11:31:31.216 INFO:tasks.rados.rados.0.smithi038.stdout:1498: write oid 50 current snap is 33 2024-03-19T11:31:31.216 INFO:tasks.rados.rados.0.smithi038.stdout:1498: seq_num 260 ranges {769820=720742,2114730=255090} 2024-03-19T11:31:31.224 INFO:tasks.rados.rados.0.smithi038.stdout:1498: writing smithi03829856-50 from 769820 to 1490562 tid 1 2024-03-19T11:31:31.231 INFO:tasks.rados.rados.0.smithi038.stdout:1498: writing smithi03829856-50 from 2114730 to 2369820 tid 2 2024-03-19T11:31:31.231 INFO:tasks.rados.rados.0.smithi038.stdout:1499: read oid 24 snap 22 2024-03-19T11:31:31.231 INFO:tasks.rados.rados.0.smithi038.stdout:1499: expect (ObjNum 23 snap 0 seq_num 23) 2024-03-19T11:31:31.231 INFO:tasks.rados.rados.0.smithi038.stdout:1500: read oid 1 snap -1 2024-03-19T11:31:31.231 INFO:tasks.rados.rados.0.smithi038.stdout:1500: expect (ObjNum 226 snap 25 seq_num 226) 2024-03-19T11:31:31.231 INFO:tasks.rados.rados.0.smithi038.stdout:1501: read oid 8 snap -1 2024-03-19T11:31:31.231 INFO:tasks.rados.rados.0.smithi038.stdout:1501: expect deleted 2024-03-19T11:31:31.231 INFO:tasks.rados.rados.0.smithi038.stdout:1502: write oid 4 current snap is 33 2024-03-19T11:31:31.231 INFO:tasks.rados.rados.0.smithi038.stdout:1495: got (22) Invalid argument 2024-03-19T11:31:31.231 INFO:tasks.rados.rados.0.smithi038.stdout:1497: expect (ObjNum 143 snap 15 seq_num 143) 2024-03-19T11:31:31.288 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:31.296+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:31.305 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:31.305 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:31:31.305 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:31:31.305 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:31:31.305 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:31.312 INFO:tasks.rados.rados.0.smithi038.stdout:1498: finishing write tid 1 to smithi03829856-50 2024-03-19T11:31:31.313 INFO:tasks.rados.rados.0.smithi038.stdout:1499: expect (ObjNum 23 snap 0 seq_num 23) 2024-03-19T11:31:31.319 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-19T11:31:31.346 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:31.354+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:31.391 INFO:tasks.rados.rados.0.smithi038.stdout:1498: finishing write tid 2 to smithi03829856-50 2024-03-19T11:31:31.391 INFO:tasks.rados.rados.0.smithi038.stdout:1498: finishing write tid 3 to smithi03829856-50 2024-03-19T11:31:31.391 INFO:tasks.rados.rados.0.smithi038.stdout:1498: finishing write tid 4 to smithi03829856-50 2024-03-19T11:31:31.391 INFO:tasks.rados.rados.0.smithi038.stdout:1498: oid 50 updating version 0 to 165 2024-03-19T11:31:31.391 INFO:tasks.rados.rados.0.smithi038.stdout:1498: oid 50 updating version 165 to 166 2024-03-19T11:31:31.391 INFO:tasks.rados.rados.0.smithi038.stdout:1498: oid 50 updating version 166 to 167 2024-03-19T11:31:31.391 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 50 v 167 (ObjNum 260 snap 33 seq_num 260) dirty exists 2024-03-19T11:31:31.391 INFO:tasks.rados.rados.0.smithi038.stdout:1498: left oid 50 (ObjNum 260 snap 33 seq_num 260) 2024-03-19T11:31:31.391 INFO:tasks.rados.rados.0.smithi038.stdout:1502: seq_num 261 ranges {786627=795991,2173212=213415} 2024-03-19T11:31:31.403 INFO:tasks.rados.rados.0.smithi038.stdout:1502: writing smithi03829856-4 from 786627 to 1582618 tid 1 2024-03-19T11:31:31.410 INFO:tasks.rados.rados.0.smithi038.stdout:1502: writing smithi03829856-4 from 2173212 to 2386627 tid 2 2024-03-19T11:31:31.410 INFO:tasks.rados.rados.0.smithi038.stdout:1494: done (9 left) 2024-03-19T11:31:31.410 INFO:tasks.rados.rados.0.smithi038.stdout:1495: done (8 left) 2024-03-19T11:31:31.410 INFO:tasks.rados.rados.0.smithi038.stdout:1497: done (7 left) 2024-03-19T11:31:31.410 INFO:tasks.rados.rados.0.smithi038.stdout:1498: done (6 left) 2024-03-19T11:31:31.410 INFO:tasks.rados.rados.0.smithi038.stdout:1499: done (5 left) 2024-03-19T11:31:31.410 INFO:tasks.rados.rados.0.smithi038.stdout:1503: read oid 6 snap -1 2024-03-19T11:31:31.411 INFO:tasks.rados.rados.0.smithi038.stdout:1503: expect (ObjNum 195 snap 22 seq_num 195) 2024-03-19T11:31:31.411 INFO:tasks.rados.rados.0.smithi038.stdout:1504: tier_promote oid 31 2024-03-19T11:31:31.411 INFO:tasks.rados.rados.0.smithi038.stdout:1500: expect (ObjNum 226 snap 25 seq_num 226) 2024-03-19T11:31:31.446 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:31.452+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:31.489 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:31.489 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:31:31.489 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:31:31.490 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:31:31.490 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.490 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.16 49/48 [PushOp(4:6868af91:::smithi03829856-10:2, version: 29'22, data_included: [551263~793594,1893632~254982], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:2@29'22, size: 2151263, copy_subset: [(0, 2151263)], clone_subset: {}, snapset: b=[]:{2: [2], 7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2148614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:6d9093b1:::smithi03829856-38:4, version: 33'29, data_included: [621859~449625,1865716~598951], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6d9093b1:::smithi03829856-38:4@33'29, size: 2621859, copy_subset: [(0, 2621859)], clone_subset: {}, snapset: 4=[]:{4: [2]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2464667, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:6d9093b1:::smithi03829856-38:head, version: 33'33, data_included: [621859~568773,1729599~479803], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:6d9093b1:::smithi03829856-38:head@33'33, size: 3142876, copy_subset: [(0, 3142876)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2209402, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:6be89c06:::smithi03829856-24:head, version: 36'39, data_included: [741376~589824,1753088~458752], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:6be89c06:::smithi03829856-24:head@36'39, size: 2386561, copy_subset: [(0, 2386561)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2211840, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:6868af91:::smithi03829856-10:7, version: 38'41, data_included: [524385~820472,1602039~228104], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:7@38'41, size: 2924385, copy_subset: [(0, 2924385)], clone_subset: {}, snapset: b=[]:{2: [2], 7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1830143, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:6868af91:::smithi03829856-10:b, version: 47'48, data_included: [524385~820472,1602039~228104], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:b@47'48, size: 2924385, copy_subset: [(0, 2924385)], clone_subset: {}, snapset: b=[]:{2: [2], 7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1830143, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:6868af91:::smithi03829856-10:head, version: 47'52, data_included: [524385~820472,1602039~228104], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:head@47'52, size: 3747928, copy_subset: [(0, 3747928)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1830143, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:31.490 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:26.954289+0000", 2024-03-19T11:31:31.490 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 64.543382581000003, 2024-03-19T11:31:31.490 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029260781999999999, 2024-03-19T11:31:31.490 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:31.490 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:31.490 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:31.490 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.490 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:31.490 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954289+0000", 2024-03-19T11:31:31.490 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:31.490 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.491 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.491 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:31.491 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954289+0000", 2024-03-19T11:31:31.491 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:31.491 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.491 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.491 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:31.491 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954291+0000", 2024-03-19T11:31:31.491 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8050000000000001e-06 2024-03-19T11:31:31.491 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.491 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.491 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:31.491 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.958775+0000", 2024-03-19T11:31:31.491 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0044845859999999996 2024-03-19T11:31:31.491 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.492 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.492 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:31.492 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.958851+0000", 2024-03-19T11:31:31.492 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6050999999999994e-05 2024-03-19T11:31:31.492 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.492 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.492 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:31.492 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.958931+0000", 2024-03-19T11:31:31.494 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9099999999999998e-05 2024-03-19T11:31:31.494 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.494 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.494 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:31.494 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.959206+0000", 2024-03-19T11:31:31.494 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00027495999999999998 2024-03-19T11:31:31.494 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.494 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.495 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:31.495 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.959246+0000", 2024-03-19T11:31:31.495 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0326e-05 2024-03-19T11:31:31.495 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.495 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.495 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:31.495 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.983550+0000", 2024-03-19T11:31:31.495 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024303953999999999 2024-03-19T11:31:31.495 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:31.495 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:31.495 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:31.495 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.495 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.495 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.16 49/48 [PushOp(4:6868af91:::smithi03829856-10:2, version: 29'22, data_included: [2148614~2649], data_size: 2649, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:2@29'22, size: 2151263, copy_subset: [(0, 2151263)], clone_subset: {}, snapset: b=[]:{2: [2], 7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2151263, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2148614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:6d9093b1:::smithi03829856-38:4, version: 33'29, data_included: [2464667~157192], data_size: 157192, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:6d9093b1:::smithi03829856-38:4@33'29, size: 2621859, copy_subset: [(0, 2621859)], clone_subset: {}, snapset: 4=[]:{4: [2]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2621859, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2464667, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:6d9093b1:::smithi03829856-38:head, version: 33'33, data_included: [2209402~412457,3101234~41642], data_size: 454099, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:6d9093b1:::smithi03829856-38:head@33'33, size: 3142876, copy_subset: [(0, 3142876)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3142876, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2209402, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:6be89c06:::smithi03829856-24:head, version: 36'39, data_included: [2211840~51851], data_size: 51851, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:6be89c06:::smithi03829856-24:head@36'39, size: 2386561, copy_subset: [(0, 2386561)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2386561, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2211840, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:6868af91:::smithi03829856-10:7, version: 38'41, data_included: [1830143~321120,2577441~346944], data_size: 668064, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:7@38'41, size: 2924385, copy_subset: [(0, 2924385)], clone_subset: {}, snapset: b=[]:{2: [2], 7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2924385, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1830143, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:6868af91:::smithi03829856-10:b, version: 47'48, data_included: [1830143~321120,2577441~346944], data_size: 668064, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:b@47'48, size: 2924385, copy_subset: [(0, 2924385)], clone_subset: {}, snapset: b=[]:{2: [2], 7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2924385, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1830143, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:6868af91:::smithi03829856-10:head, version: 47'52, data_included: [1830143~348419,2577441~661310], data_size: 1009729, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:head@47'52, size: 3747928, copy_subset: [(0, 3747928)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3747928, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1830143, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:31.495 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:26.998857+0000", 2024-03-19T11:31:31.496 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 64.498814820999996, 2024-03-19T11:31:31.496 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010149703, 2024-03-19T11:31:31.496 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:31.496 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:31.496 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:31.496 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.496 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:31.496 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.998857+0000", 2024-03-19T11:31:31.496 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:31.496 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.496 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.496 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:31.496 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.998857+0000", 2024-03-19T11:31:31.496 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:31.497 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.497 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.497 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:31.497 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.998858+0000", 2024-03-19T11:31:31.497 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.065e-06 2024-03-19T11:31:31.497 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.497 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.497 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:31.497 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.999629+0000", 2024-03-19T11:31:31.497 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000771132 2024-03-19T11:31:31.497 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.497 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.497 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:31.497 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.999688+0000", 2024-03-19T11:31:31.497 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9147000000000001e-05 2024-03-19T11:31:31.498 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.498 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.498 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:31.498 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.999755+0000", 2024-03-19T11:31:31.498 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6891999999999996e-05 2024-03-19T11:31:31.498 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.499 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.499 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:31.499 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.000047+0000", 2024-03-19T11:31:31.499 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000292176 2024-03-19T11:31:31.499 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.499 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.499 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:31.499 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.000094+0000", 2024-03-19T11:31:31.499 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6485e-05 2024-03-19T11:31:31.499 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.499 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.499 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:31.499 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.009007+0000", 2024-03-19T11:31:31.499 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0089128060000000005 2024-03-19T11:31:31.499 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:31.500 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:31.500 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:31.500 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.500 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.500 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1591 4.4 4:266590a5:::smithi03829856-16:head [write 783226~624677,stat] snapc e=[e,d,c,b,a,6] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-03-19T11:31:31.500 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:32.390032+0000", 2024-03-19T11:31:31.500 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 59.107640035999999, 2024-03-19T11:31:31.500 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0090431220000000007, 2024-03-19T11:31:31.500 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:31.500 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:31.500 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:31.500 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:31.500 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:31.500 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1591 2024-03-19T11:31:31.500 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.501 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:31.501 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.501 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:31.501 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.390032+0000", 2024-03-19T11:31:31.501 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:31.501 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.501 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.501 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:31.501 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.390032+0000", 2024-03-19T11:31:31.501 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:31.501 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.501 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.501 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:31.501 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.390033+0000", 2024-03-19T11:31:31.501 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2839999999999999e-06 2024-03-19T11:31:31.502 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.502 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.502 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:31.502 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.390273+0000", 2024-03-19T11:31:31.502 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00024011099999999999 2024-03-19T11:31:31.502 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.502 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.502 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:31.502 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.390286+0000", 2024-03-19T11:31:31.502 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3314e-05 2024-03-19T11:31:31.505 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.505 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.505 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:31.505 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.390295+0000", 2024-03-19T11:31:31.505 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6330000000000002e-06 2024-03-19T11:31:31.505 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.505 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.505 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:31.505 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.390368+0000", 2024-03-19T11:31:31.506 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3065999999999995e-05 2024-03-19T11:31:31.506 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.506 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.506 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:31.506 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.390461+0000", 2024-03-19T11:31:31.506 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2582000000000002e-05 2024-03-19T11:31:31.506 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.506 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.506 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3", 2024-03-19T11:31:31.506 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.390648+0000", 2024-03-19T11:31:31.506 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00018695300000000001 2024-03-19T11:31:31.507 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.507 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.507 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:31.507 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.398909+0000", 2024-03-19T11:31:31.507 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0082613559999999992 2024-03-19T11:31:31.507 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.507 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.507 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:31.507 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.398954+0000", 2024-03-19T11:31:31.507 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5553999999999998e-05 2024-03-19T11:31:31.507 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.507 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.508 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:31.508 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.399017+0000", 2024-03-19T11:31:31.508 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2849999999999996e-05 2024-03-19T11:31:31.508 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.508 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.508 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:31.508 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:32.399075+0000", 2024-03-19T11:31:31.508 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7419e-05 2024-03-19T11:31:31.508 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:31.508 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:31.508 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:31.508 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.508 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.509 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1630 4.d 4:ba44105a:::smithi03829856-26:head [write 467261~717275] snapc 10=[10,f,e,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-03-19T11:31:31.509 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:35.331434+0000", 2024-03-19T11:31:31.509 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 56.166237776000003, 2024-03-19T11:31:31.509 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012902375000000001, 2024-03-19T11:31:31.509 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:31.509 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:31.509 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:31.509 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:31.509 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:31.509 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1630 2024-03-19T11:31:31.509 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.510 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:31.510 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.510 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:31.510 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.331434+0000", 2024-03-19T11:31:31.510 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:31.510 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.510 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.510 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:31.510 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.331434+0000", 2024-03-19T11:31:31.510 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:31.510 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.510 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.510 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:31.511 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.331437+0000", 2024-03-19T11:31:31.511 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5599999999999998e-06 2024-03-19T11:31:31.511 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.511 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.511 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:31.511 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.331938+0000", 2024-03-19T11:31:31.511 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00050090099999999999 2024-03-19T11:31:31.511 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.511 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.511 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:31.511 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.331976+0000", 2024-03-19T11:31:31.511 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8090999999999998e-05 2024-03-19T11:31:31.511 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.512 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.512 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:31.512 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.331994+0000", 2024-03-19T11:31:31.512 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7566000000000001e-05 2024-03-19T11:31:31.512 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.512 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.512 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:31.512 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.332086+0000", 2024-03-19T11:31:31.512 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2452000000000002e-05 2024-03-19T11:31:31.512 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.512 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.512 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:31.512 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.332205+0000", 2024-03-19T11:31:31.513 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000118966 2024-03-19T11:31:31.513 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.513 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.513 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1", 2024-03-19T11:31:31.513 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.332410+0000", 2024-03-19T11:31:31.513 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020430099999999999 2024-03-19T11:31:31.513 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.513 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.513 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:31.513 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.339226+0000", 2024-03-19T11:31:31.513 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0068165980000000001 2024-03-19T11:31:31.513 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.513 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.513 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:31.514 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.344191+0000", 2024-03-19T11:31:31.514 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.004965143 2024-03-19T11:31:31.514 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.514 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.514 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:31.514 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.344272+0000", 2024-03-19T11:31:31.514 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0265000000000005e-05 2024-03-19T11:31:31.514 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.514 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.514 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:31.514 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.344336+0000", 2024-03-19T11:31:31.514 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4531999999999998e-05 2024-03-19T11:31:31.514 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:31.514 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:31.515 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:31.515 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.515 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.515 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1632 4.d 4:ba44105a:::smithi03829856-26:head [setxattr _header (57) in=7b,truncate 2067261] snapc 10=[10,f,e,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-03-19T11:31:31.515 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:35.336987+0000", 2024-03-19T11:31:31.515 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 56.160685116000003, 2024-03-19T11:31:31.515 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0091098789999999995, 2024-03-19T11:31:31.515 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:31.515 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:31.515 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:31.515 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:31.515 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:31.515 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1632 2024-03-19T11:31:31.515 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.515 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:31.516 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.516 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:31.516 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.336987+0000", 2024-03-19T11:31:31.516 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:31.516 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.516 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.516 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:31.516 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.336987+0000", 2024-03-19T11:31:31.516 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:31.516 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.516 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.516 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:31.516 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.336987+0000", 2024-03-19T11:31:31.517 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4600000000000004e-07 2024-03-19T11:31:31.517 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.517 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.517 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:31.517 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.336992+0000", 2024-03-19T11:31:31.517 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6090000000000003e-06 2024-03-19T11:31:31.517 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.517 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.517 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:31.517 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.337002+0000", 2024-03-19T11:31:31.517 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.006e-05 2024-03-19T11:31:31.517 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.517 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.517 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:31.518 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.337009+0000", 2024-03-19T11:31:31.518 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7229999999999998e-06 2024-03-19T11:31:31.518 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.518 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.518 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:31.518 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.338513+0000", 2024-03-19T11:31:31.518 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001503868 2024-03-19T11:31:31.518 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.518 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.518 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:31.518 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.338566+0000", 2024-03-19T11:31:31.518 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3260999999999997e-05 2024-03-19T11:31:31.518 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.518 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.518 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1", 2024-03-19T11:31:31.519 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.338652+0000", 2024-03-19T11:31:31.519 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5889000000000004e-05 2024-03-19T11:31:31.519 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.519 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.519 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:31.519 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.340450+0000", 2024-03-19T11:31:31.519 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017987490000000001 2024-03-19T11:31:31.519 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.519 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.519 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:31.519 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.345982+0000", 2024-03-19T11:31:31.519 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055312620000000003 2024-03-19T11:31:31.519 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.519 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.519 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:31.519 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.346039+0000", 2024-03-19T11:31:31.520 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6879999999999998e-05 2024-03-19T11:31:31.520 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.520 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.520 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:31.520 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.346096+0000", 2024-03-19T11:31:31.520 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7831999999999998e-05 2024-03-19T11:31:31.520 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:31.520 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:31.520 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:31.520 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.520 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.520 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1633 4.d 4:ba44105a:::smithi03829856-26:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-03-19T11:31:31.521 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:35.337032+0000", 2024-03-19T11:31:31.521 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 56.160639396999997, 2024-03-19T11:31:31.521 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093229679999999992, 2024-03-19T11:31:31.521 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:31.521 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:31.521 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:31.521 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:31.521 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:31.521 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1633 2024-03-19T11:31:31.521 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.521 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:31.521 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.521 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:31.522 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.337032+0000", 2024-03-19T11:31:31.522 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:31.522 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.522 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.522 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:31.522 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.337032+0000", 2024-03-19T11:31:31.522 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:31.522 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.522 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.522 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:31.522 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.337033+0000", 2024-03-19T11:31:31.522 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4899999999999995e-07 2024-03-19T11:31:31.523 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.523 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.523 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:31.523 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.337037+0000", 2024-03-19T11:31:31.523 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7249999999999999e-06 2024-03-19T11:31:31.523 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.523 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.523 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:31.523 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.337045+0000", 2024-03-19T11:31:31.523 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5380000000000002e-06 2024-03-19T11:31:31.523 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.523 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.523 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:31.524 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.337051+0000", 2024-03-19T11:31:31.524 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1689999999999997e-06 2024-03-19T11:31:31.524 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.524 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.524 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:31.524 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.339128+0000", 2024-03-19T11:31:31.524 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002076919 2024-03-19T11:31:31.524 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.524 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.524 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-03-19T11:31:31.524 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.339211+0000", 2024-03-19T11:31:31.524 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2896000000000001e-05 2024-03-19T11:31:31.524 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.524 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.525 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:31.525 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.346152+0000", 2024-03-19T11:31:31.525 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0069404779999999999 2024-03-19T11:31:31.525 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.525 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.525 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:31.525 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.346242+0000", 2024-03-19T11:31:31.525 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0031000000000006e-05 2024-03-19T11:31:31.525 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.525 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.525 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:31.525 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.346355+0000", 2024-03-19T11:31:31.525 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000113663 2024-03-19T11:31:31.526 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:31.526 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:31.526 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:31.526 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.526 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.526 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1845 4.4 4:27909373:::smithi03829856-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-03-19T11:31:31.526 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:41.053790+0000", 2024-03-19T11:31:31.526 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 50.443881316000002, 2024-03-19T11:31:31.526 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0091637609999999994, 2024-03-19T11:31:31.526 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:31.526 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:31.526 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:31.526 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:31.526 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:31.527 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1845 2024-03-19T11:31:31.527 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.527 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:31.527 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.527 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:31.527 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053790+0000", 2024-03-19T11:31:31.527 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:31.528 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.528 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.528 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:31.528 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053790+0000", 2024-03-19T11:31:31.528 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:31.528 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.528 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.528 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:31.529 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053792+0000", 2024-03-19T11:31:31.529 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2860000000000001e-06 2024-03-19T11:31:31.529 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.529 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.529 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:31.529 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053802+0000", 2024-03-19T11:31:31.529 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.064e-05 2024-03-19T11:31:31.529 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.529 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.529 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:31.529 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053824+0000", 2024-03-19T11:31:31.529 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1611999999999999e-05 2024-03-19T11:31:31.530 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.530 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.530 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:31.530 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053839+0000", 2024-03-19T11:31:31.530 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4922e-05 2024-03-19T11:31:31.530 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.530 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.530 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:31.530 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.059104+0000", 2024-03-19T11:31:31.530 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0052656179999999997 2024-03-19T11:31:31.530 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.530 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.530 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-03-19T11:31:31.530 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.059267+0000", 2024-03-19T11:31:31.531 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000162776 2024-03-19T11:31:31.531 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.531 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.531 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:31.531 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.062551+0000", 2024-03-19T11:31:31.531 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003283467 2024-03-19T11:31:31.531 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.531 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.531 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:31.531 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.062738+0000", 2024-03-19T11:31:31.531 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00018781900000000001 2024-03-19T11:31:31.531 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.531 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.532 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:31.532 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.062954+0000", 2024-03-19T11:31:31.532 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00021562099999999999 2024-03-19T11:31:31.532 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:31.532 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:31.532 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:31.532 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.532 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.532 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1889 4.4 4:266590a5:::smithi03829856-16:head [rollback d] snapc 10=[10,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-03-19T11:31:31.532 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:41.697617+0000", 2024-03-19T11:31:31.532 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 49.800054781999997, 2024-03-19T11:31:31.532 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018164059999999999, 2024-03-19T11:31:31.532 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:31.532 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:31.532 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:31.533 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:31.533 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:31.533 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1889 2024-03-19T11:31:31.533 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.533 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:31.533 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.533 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:31.533 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697617+0000", 2024-03-19T11:31:31.533 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:31.533 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.533 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.533 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:31.533 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697617+0000", 2024-03-19T11:31:31.533 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:31.533 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.534 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.534 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:31.534 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697619+0000", 2024-03-19T11:31:31.534 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0059999999999999e-06 2024-03-19T11:31:31.534 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.534 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.534 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:31.534 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697631+0000", 2024-03-19T11:31:31.534 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2033e-05 2024-03-19T11:31:31.534 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.534 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.534 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:31.534 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697659+0000", 2024-03-19T11:31:31.534 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8291999999999999e-05 2024-03-19T11:31:31.534 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.535 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.535 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:31.535 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697682+0000", 2024-03-19T11:31:31.535 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3281999999999998e-05 2024-03-19T11:31:31.535 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.535 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.535 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:31.535 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.701122+0000", 2024-03-19T11:31:31.535 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034396679999999999 2024-03-19T11:31:31.535 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.535 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.535 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:31.535 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.701383+0000", 2024-03-19T11:31:31.535 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00026086399999999997 2024-03-19T11:31:31.536 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.536 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.536 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3", 2024-03-19T11:31:31.536 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.709693+0000", 2024-03-19T11:31:31.536 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0083096139999999999 2024-03-19T11:31:31.536 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.536 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.536 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:31.536 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.715437+0000", 2024-03-19T11:31:31.536 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057442379999999996 2024-03-19T11:31:31.536 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.536 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.536 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:31.536 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.715563+0000", 2024-03-19T11:31:31.536 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000125778 2024-03-19T11:31:31.537 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.537 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.537 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:31.537 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.715677+0000", 2024-03-19T11:31:31.537 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000114728 2024-03-19T11:31:31.537 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.537 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.537 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:31.538 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.715781+0000", 2024-03-19T11:31:31.538 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000103557 2024-03-19T11:31:31.538 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:31.538 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:31.538 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:31.538 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.538 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.538 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1890 4.4 4:266590a5:::smithi03829856-16:head [append 0~0] snapc 10=[10,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-03-19T11:31:31.538 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:41.697709+0000", 2024-03-19T11:31:31.538 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 49.799962219999998, 2024-03-19T11:31:31.538 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019825171999999999, 2024-03-19T11:31:31.538 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:31.538 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:31.538 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:31.539 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:31.539 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:31.539 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1890 2024-03-19T11:31:31.539 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.539 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:31.539 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.539 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:31.539 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697709+0000", 2024-03-19T11:31:31.539 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:31.539 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.539 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.539 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:31.539 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697709+0000", 2024-03-19T11:31:31.539 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:31.539 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.540 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.540 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:31.540 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697711+0000", 2024-03-19T11:31:31.540 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.296e-06 2024-03-19T11:31:31.540 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.540 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.540 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:31.540 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697718+0000", 2024-03-19T11:31:31.540 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.464e-06 2024-03-19T11:31:31.540 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.540 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.540 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:31.540 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697739+0000", 2024-03-19T11:31:31.540 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1005e-05 2024-03-19T11:31:31.540 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.541 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.541 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:31.541 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697754+0000", 2024-03-19T11:31:31.541 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4976e-05 2024-03-19T11:31:31.541 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.541 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.541 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:31.541 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.702341+0000", 2024-03-19T11:31:31.541 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0045870980000000004 2024-03-19T11:31:31.541 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.541 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.541 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for blocked object", 2024-03-19T11:31:31.541 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.702482+0000", 2024-03-19T11:31:31.541 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014048000000000001 2024-03-19T11:31:31.542 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.542 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.542 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:31.542 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.714359+0000", 2024-03-19T11:31:31.542 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011877522999999999 2024-03-19T11:31:31.542 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.542 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.542 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:31.542 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.714550+0000", 2024-03-19T11:31:31.542 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019117 2024-03-19T11:31:31.542 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.543 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.543 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3", 2024-03-19T11:31:31.543 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.714911+0000", 2024-03-19T11:31:31.543 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00036070900000000002 2024-03-19T11:31:31.543 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.543 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.543 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:31.543 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.716211+0000", 2024-03-19T11:31:31.543 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001300329 2024-03-19T11:31:31.543 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.543 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.543 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:31.543 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.717326+0000", 2024-03-19T11:31:31.543 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001114056 2024-03-19T11:31:31.543 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.544 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.544 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:31.544 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.717423+0000", 2024-03-19T11:31:31.544 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7048999999999999e-05 2024-03-19T11:31:31.544 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.544 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.544 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:31.544 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.717535+0000", 2024-03-19T11:31:31.544 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011201699999999999 2024-03-19T11:31:31.544 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:31.544 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:31.545 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:31.545 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.545 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.545 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1972 4.4 4:214026b3:::smithi03829856-47:head [tier-promote] snapc 12=[12,11,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:31:31.545 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:45.312422+0000", 2024-03-19T11:31:31.545 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 46.185249546000001, 2024-03-19T11:31:31.545 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013955089, 2024-03-19T11:31:31.545 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:31.545 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:31.545 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:31.545 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:31.545 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:31.545 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1972 2024-03-19T11:31:31.545 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.546 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:31.546 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.546 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:31.546 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.312422+0000", 2024-03-19T11:31:31.546 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:31.546 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.546 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.546 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:31.546 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.312422+0000", 2024-03-19T11:31:31.546 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:31.546 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.547 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.547 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:31.547 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.312424+0000", 2024-03-19T11:31:31.547 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9810000000000002e-06 2024-03-19T11:31:31.547 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.547 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.547 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:31.547 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.312433+0000", 2024-03-19T11:31:31.547 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4549999999999995e-06 2024-03-19T11:31:31.547 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.547 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.547 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:31.547 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.312452+0000", 2024-03-19T11:31:31.547 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9380000000000001e-05 2024-03-19T11:31:31.547 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.548 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.548 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:31.548 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.312460+0000", 2024-03-19T11:31:31.548 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2560000000000002e-06 2024-03-19T11:31:31.548 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.548 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.548 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:31.548 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.312555+0000", 2024-03-19T11:31:31.548 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.5208000000000001e-05 2024-03-19T11:31:31.548 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.548 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.549 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:31.549 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.312704+0000", 2024-03-19T11:31:31.549 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014843899999999999 2024-03-19T11:31:31.549 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.549 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.549 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:31.549 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.326377+0000", 2024-03-19T11:31:31.549 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013673370000000001 2024-03-19T11:31:31.549 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:31.549 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:31.549 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:31.549 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.549 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.549 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2005 4.4 4:214026b3:::smithi03829856-47:head [read 0~2773122,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:31:31.550 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:45.804325+0000", 2024-03-19T11:31:31.550 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 45.693347086000003, 2024-03-19T11:31:31.550 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010899282, 2024-03-19T11:31:31.550 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:31.550 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:31.550 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:31.550 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:31.550 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:31.550 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2005 2024-03-19T11:31:31.550 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.550 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:31.550 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.550 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:31.550 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804325+0000", 2024-03-19T11:31:31.551 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:31.551 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.551 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.551 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:31.551 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804325+0000", 2024-03-19T11:31:31.551 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:31.551 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.551 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.551 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:31.551 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804326+0000", 2024-03-19T11:31:31.551 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.666e-06 2024-03-19T11:31:31.551 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.551 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.552 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:31.552 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804338+0000", 2024-03-19T11:31:31.552 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2096000000000001e-05 2024-03-19T11:31:31.552 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.552 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.552 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:31.552 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804363+0000", 2024-03-19T11:31:31.552 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4762000000000001e-05 2024-03-19T11:31:31.552 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.552 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.552 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:31.552 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804380+0000", 2024-03-19T11:31:31.552 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.713e-05 2024-03-19T11:31:31.552 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.552 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.553 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:31.553 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.812236+0000", 2024-03-19T11:31:31.553 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0078554690000000003 2024-03-19T11:31:31.553 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.553 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.553 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:31.553 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.812541+0000", 2024-03-19T11:31:31.553 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030564299999999997 2024-03-19T11:31:31.553 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.553 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.553 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:31.553 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.815224+0000", 2024-03-19T11:31:31.553 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0026825159999999998 2024-03-19T11:31:31.553 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:31.554 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:31.554 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:31.554 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.554 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.554 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2006 4.4 4:214026b3:::smithi03829856-47:head [read 0~2773122,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:31:31.554 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:45.804421+0000", 2024-03-19T11:31:31.554 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 45.693250347000003, 2024-03-19T11:31:31.554 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013366463, 2024-03-19T11:31:31.554 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:31.554 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:31.554 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:31.554 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:31.554 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:31.555 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2006 2024-03-19T11:31:31.555 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.555 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:31.555 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.555 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:31.555 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804421+0000", 2024-03-19T11:31:31.555 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:31.555 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.555 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.555 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:31.555 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804421+0000", 2024-03-19T11:31:31.555 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:31.555 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.555 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.556 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:31.556 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804423+0000", 2024-03-19T11:31:31.556 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5600000000000001e-06 2024-03-19T11:31:31.556 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.556 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.556 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:31.556 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804433+0000", 2024-03-19T11:31:31.556 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8719999999999997e-06 2024-03-19T11:31:31.556 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.556 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.556 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:31.556 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804454+0000", 2024-03-19T11:31:31.557 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1759e-05 2024-03-19T11:31:31.557 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.557 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.557 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:31.557 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804470+0000", 2024-03-19T11:31:31.557 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5556000000000001e-05 2024-03-19T11:31:31.557 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.557 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.557 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:31.557 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.815325+0000", 2024-03-19T11:31:31.557 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010855069 2024-03-19T11:31:31.557 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.557 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.558 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:31.558 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.815684+0000", 2024-03-19T11:31:31.558 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00035912699999999999 2024-03-19T11:31:31.558 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.558 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.558 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:31.558 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.817788+0000", 2024-03-19T11:31:31.558 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0021035200000000002 2024-03-19T11:31:31.558 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:31.558 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:31.558 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:31.558 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.558 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.558 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(osd.1.0:198 4.16 e67/48 4:6868af91:::smithi03829856-10:head v 67'79, mlcod=66'76)", 2024-03-19T11:31:31.559 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:45.811039+0000", 2024-03-19T11:31:31.559 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 45.686632985999999, 2024-03-19T11:31:31.559 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01087364, 2024-03-19T11:31:31.559 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:31.559 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:31.559 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:31.559 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.559 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:31.559 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.811039+0000", 2024-03-19T11:31:31.559 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:31.559 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.559 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.559 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:31.560 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.811039+0000", 2024-03-19T11:31:31.560 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:31.560 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.560 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.560 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:31.560 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.811041+0000", 2024-03-19T11:31:31.560 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6400000000000001e-06 2024-03-19T11:31:31.560 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.560 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.560 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:31.560 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.812874+0000", 2024-03-19T11:31:31.560 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0018329959999999999 2024-03-19T11:31:31.560 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.560 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.560 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:31.561 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.812916+0000", 2024-03-19T11:31:31.561 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1875000000000001e-05 2024-03-19T11:31:31.561 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.561 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.561 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:31.561 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.812937+0000", 2024-03-19T11:31:31.561 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0985e-05 2024-03-19T11:31:31.561 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.561 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.561 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:31.561 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.813112+0000", 2024-03-19T11:31:31.561 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000174573 2024-03-19T11:31:31.561 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.562 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.562 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:31.562 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.813170+0000", 2024-03-19T11:31:31.562 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8100999999999998e-05 2024-03-19T11:31:31.562 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.562 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.562 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:31.562 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.821792+0000", 2024-03-19T11:31:31.562 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0086218549999999994 2024-03-19T11:31:31.562 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.562 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.562 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:31.562 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.821912+0000", 2024-03-19T11:31:31.562 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000120615 2024-03-19T11:31:31.563 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:31.563 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:31.563 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:31.563 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.563 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.563 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2027 4.2d 4:b4ef97e0:::smithi03829856-3:head [read 0~3697507,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:31:31.563 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:46.264908+0000", 2024-03-19T11:31:31.563 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 45.232764119999999, 2024-03-19T11:31:31.563 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010951414, 2024-03-19T11:31:31.563 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:31.563 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:31.563 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:31.564 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:31.564 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:31.564 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2027 2024-03-19T11:31:31.564 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.564 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:31.564 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.564 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:31.564 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264908+0000", 2024-03-19T11:31:31.564 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:31.564 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.564 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.564 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:31.565 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264908+0000", 2024-03-19T11:31:31.565 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:31.565 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.565 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.565 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:31.565 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264908+0000", 2024-03-19T11:31:31.565 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5799999999999999e-07 2024-03-19T11:31:31.565 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.565 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.565 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:31.565 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264912+0000", 2024-03-19T11:31:31.565 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2649999999999998e-06 2024-03-19T11:31:31.565 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.565 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.566 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:31.566 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264920+0000", 2024-03-19T11:31:31.566 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.79e-06 2024-03-19T11:31:31.566 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.566 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.566 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:31.566 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264926+0000", 2024-03-19T11:31:31.566 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9989999999999997e-06 2024-03-19T11:31:31.566 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.566 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.566 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:31.566 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.273461+0000", 2024-03-19T11:31:31.567 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0085352680000000004 2024-03-19T11:31:31.567 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.567 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.567 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:31.567 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.273598+0000", 2024-03-19T11:31:31.567 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013622299999999999 2024-03-19T11:31:31.567 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.567 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.567 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:31.567 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.275859+0000", 2024-03-19T11:31:31.567 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022613109999999998 2024-03-19T11:31:31.567 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:31.567 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:31.567 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:31.568 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.568 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.568 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2028 4.2d 4:b4ef97e0:::smithi03829856-3:head [read 0~3697507,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:31:31.568 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:46.264934+0000", 2024-03-19T11:31:31.568 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 45.232737489000002, 2024-03-19T11:31:31.568 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013708481, 2024-03-19T11:31:31.568 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:31.568 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:31.568 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:31.568 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:31.568 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:31.568 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2028 2024-03-19T11:31:31.568 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.568 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:31.569 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.569 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:31.569 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264934+0000", 2024-03-19T11:31:31.569 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:31.569 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.569 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.569 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:31.569 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264934+0000", 2024-03-19T11:31:31.569 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:31.569 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.569 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.569 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:31.569 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264935+0000", 2024-03-19T11:31:31.569 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2599999999999998e-07 2024-03-19T11:31:31.569 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.570 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.570 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:31.570 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264938+0000", 2024-03-19T11:31:31.570 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.163e-06 2024-03-19T11:31:31.570 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.570 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.570 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:31.570 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264945+0000", 2024-03-19T11:31:31.570 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.0369999999999997e-06 2024-03-19T11:31:31.570 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.570 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.570 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:31.570 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264950+0000", 2024-03-19T11:31:31.570 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1170000000000003e-06 2024-03-19T11:31:31.571 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.571 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.571 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:31.571 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.275914+0000", 2024-03-19T11:31:31.571 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010964150000000001 2024-03-19T11:31:31.571 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.571 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.571 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:31.571 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.276037+0000", 2024-03-19T11:31:31.571 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012250100000000001 2024-03-19T11:31:31.571 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.571 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.571 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:31.571 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.278643+0000", 2024-03-19T11:31:31.571 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002606087 2024-03-19T11:31:31.572 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:31.572 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:31.572 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:31.572 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.572 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.572 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2045 4.4 4:21eac9b2:::smithi03829856-4:head [read 0~2537857,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:31:31.572 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:46.265120+0000", 2024-03-19T11:31:31.572 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 45.232551180999998, 2024-03-19T11:31:31.572 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0096591980000000008, 2024-03-19T11:31:31.572 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:31.572 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:31.572 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:31.572 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:31.572 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:31.572 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2045 2024-03-19T11:31:31.573 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.573 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:31.573 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.573 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:31.573 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265120+0000", 2024-03-19T11:31:31.573 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:31.573 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.573 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.573 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:31.573 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265120+0000", 2024-03-19T11:31:31.573 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:31.573 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.573 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.573 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:31.573 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265121+0000", 2024-03-19T11:31:31.574 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2399999999999998e-07 2024-03-19T11:31:31.574 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.574 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.574 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:31.574 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265126+0000", 2024-03-19T11:31:31.574 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4870000000000002e-06 2024-03-19T11:31:31.574 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.574 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.574 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:31.574 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265134+0000", 2024-03-19T11:31:31.574 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9740000000000002e-06 2024-03-19T11:31:31.574 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.574 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.574 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:31.575 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265143+0000", 2024-03-19T11:31:31.575 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7530000000000007e-06 2024-03-19T11:31:31.575 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.575 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.575 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:31.575 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.273765+0000", 2024-03-19T11:31:31.575 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0086213880000000007 2024-03-19T11:31:31.575 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.575 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.575 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:31.575 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.273872+0000", 2024-03-19T11:31:31.575 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000107452 2024-03-19T11:31:31.575 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.575 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.575 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:31.576 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.274780+0000", 2024-03-19T11:31:31.576 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00090762000000000004 2024-03-19T11:31:31.576 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:31.576 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:31.576 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:31.576 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.576 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.576 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2046 4.4 4:21eac9b2:::smithi03829856-4:head [read 0~2537857,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:31:31.576 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:46.265152+0000", 2024-03-19T11:31:31.576 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 45.232519822, 2024-03-19T11:31:31.576 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013074417, 2024-03-19T11:31:31.576 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:31.576 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:31.577 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:31.577 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:31.577 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:31.577 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2046 2024-03-19T11:31:31.577 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.577 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:31.577 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.577 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:31.577 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265152+0000", 2024-03-19T11:31:31.577 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:31.577 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.578 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.578 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:31.578 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265152+0000", 2024-03-19T11:31:31.578 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:31.578 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.578 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.578 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:31.578 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265152+0000", 2024-03-19T11:31:31.578 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8999999999999997e-07 2024-03-19T11:31:31.578 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.578 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.578 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:31.578 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265155+0000", 2024-03-19T11:31:31.579 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0920000000000002e-06 2024-03-19T11:31:31.579 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.579 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.579 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:31.579 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265162+0000", 2024-03-19T11:31:31.579 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6730000000000003e-06 2024-03-19T11:31:31.579 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.579 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.579 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:31.579 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265167+0000", 2024-03-19T11:31:31.580 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1839999999999998e-06 2024-03-19T11:31:31.580 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.580 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.580 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:31.580 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.276218+0000", 2024-03-19T11:31:31.580 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011050559999999999 2024-03-19T11:31:31.580 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.580 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.580 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:31.580 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.276345+0000", 2024-03-19T11:31:31.580 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000127243 2024-03-19T11:31:31.580 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.580 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.580 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:31.580 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.278226+0000", 2024-03-19T11:31:31.581 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0018811749999999999 2024-03-19T11:31:31.581 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:31.581 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:31.581 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:31.581 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.581 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.581 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:3046 4.19 e103/73 4:9c74b127:::smithi03829856-33:head v 103'148, mlcod=99'144)", 2024-03-19T11:31:31.581 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:22.149246+0000", 2024-03-19T11:31:31.581 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 9.3484257540000009, 2024-03-19T11:31:31.581 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014101367, 2024-03-19T11:31:31.582 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:31.582 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:31.582 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:31.582 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.582 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:31.582 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.149246+0000", 2024-03-19T11:31:31.582 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:31.582 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.582 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.582 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:31.582 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.149246+0000", 2024-03-19T11:31:31.582 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:31.582 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.583 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.583 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:31.583 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.149246+0000", 2024-03-19T11:31:31.583 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2300000000000002e-07 2024-03-19T11:31:31.583 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.583 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.583 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:31.583 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.149663+0000", 2024-03-19T11:31:31.583 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00041629599999999997 2024-03-19T11:31:31.583 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.583 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.583 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:31.584 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.149682+0000", 2024-03-19T11:31:31.584 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9500999999999998e-05 2024-03-19T11:31:31.584 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.584 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.584 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:31.584 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.149692+0000", 2024-03-19T11:31:31.584 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.825e-06 2024-03-19T11:31:31.584 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.584 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.584 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:31.584 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.149797+0000", 2024-03-19T11:31:31.584 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000105222 2024-03-19T11:31:31.584 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.584 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.585 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:31.585 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.149835+0000", 2024-03-19T11:31:31.585 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7934e-05 2024-03-19T11:31:31.585 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.585 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.585 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:31.585 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.163304+0000", 2024-03-19T11:31:31.585 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013469122 2024-03-19T11:31:31.585 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.585 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.585 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:31.585 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.163347+0000", 2024-03-19T11:31:31.585 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3143999999999999e-05 2024-03-19T11:31:31.585 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:31.585 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:31.586 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:31.586 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.586 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.586 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:3221 4.4 e109/83 4:214026b3:::smithi03829856-47:head v 109'146, mlcod=108'142)", 2024-03-19T11:31:31.586 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:28.083665+0000", 2024-03-19T11:31:31.586 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.41400623, 2024-03-19T11:31:31.586 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010710984, 2024-03-19T11:31:31.586 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:31.586 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:31.586 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:31.586 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.586 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:31.586 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.083665+0000", 2024-03-19T11:31:31.586 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:31.587 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.587 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.587 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:31.587 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.083665+0000", 2024-03-19T11:31:31.587 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:31.587 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.587 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.587 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:31.587 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.083666+0000", 2024-03-19T11:31:31.587 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5900000000000003e-07 2024-03-19T11:31:31.587 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.587 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.587 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:31.588 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.084140+0000", 2024-03-19T11:31:31.588 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00047418100000000001 2024-03-19T11:31:31.588 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.588 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.588 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:31.588 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.084164+0000", 2024-03-19T11:31:31.588 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3978999999999999e-05 2024-03-19T11:31:31.588 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.588 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.588 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:31.588 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.084174+0000", 2024-03-19T11:31:31.588 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0149999999999999e-05 2024-03-19T11:31:31.588 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.588 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.588 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:31.589 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.084657+0000", 2024-03-19T11:31:31.589 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00048310200000000001 2024-03-19T11:31:31.589 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.589 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.589 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:31.589 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.084720+0000", 2024-03-19T11:31:31.589 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2946000000000003e-05 2024-03-19T11:31:31.589 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.589 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.589 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:31.589 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.094299+0000", 2024-03-19T11:31:31.589 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0095788390000000005 2024-03-19T11:31:31.589 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.589 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.589 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:31.590 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.094376+0000", 2024-03-19T11:31:31.590 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7428000000000005e-05 2024-03-19T11:31:31.590 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:31.590 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:31.590 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:31.590 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.590 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.590 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3240 4.1c 4:3c71e808:::smithi03829856-34:head [read 0~3210104,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e109)", 2024-03-19T11:31:31.590 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:28.277613+0000", 2024-03-19T11:31:31.590 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.2200590650000001, 2024-03-19T11:31:31.590 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011179553, 2024-03-19T11:31:31.590 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:31.590 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:31.590 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:31.590 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:31.591 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:31.591 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3240 2024-03-19T11:31:31.591 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.591 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:31.591 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.591 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:31.591 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.277613+0000", 2024-03-19T11:31:31.591 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:31.591 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.591 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.591 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:31.591 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.277613+0000", 2024-03-19T11:31:31.591 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:31.591 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.591 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.592 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:31.592 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.277614+0000", 2024-03-19T11:31:31.592 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1769999999999999e-06 2024-03-19T11:31:31.592 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.592 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.592 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:31.592 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.277622+0000", 2024-03-19T11:31:31.592 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3590000000000008e-06 2024-03-19T11:31:31.592 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.592 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.592 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:31.592 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.277644+0000", 2024-03-19T11:31:31.592 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1642e-05 2024-03-19T11:31:31.592 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.593 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.593 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:31.593 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.277660+0000", 2024-03-19T11:31:31.593 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6316000000000001e-05 2024-03-19T11:31:31.593 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.593 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.593 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:31.593 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.284699+0000", 2024-03-19T11:31:31.593 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0070386950000000002 2024-03-19T11:31:31.593 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.593 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.593 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:31.593 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.284990+0000", 2024-03-19T11:31:31.594 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00029164099999999998 2024-03-19T11:31:31.594 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:31.594 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:31.594 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:31.594 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.288792+0000", 2024-03-19T11:31:31.594 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038017229999999999 2024-03-19T11:31:31.594 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:31.594 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:31.594 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:31.594 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:31.594 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:31.594 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:31.596 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-19T11:31:31.596 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:31.606+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:31.625 INFO:tasks.rados.rados.0.smithi038.stdout:1496: got (0) Success 2024-03-19T11:31:31.625 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 18 v 124 (ObjNum 199 snap 22 seq_num 199) dirty exists 2024-03-19T11:31:31.626 INFO:tasks.rados.rados.0.smithi038.stdout:1502: finishing write tid 1 to smithi03829856-4 2024-03-19T11:31:31.626 INFO:tasks.rados.rados.0.smithi038.stdout:1504: got (0) Success 2024-03-19T11:31:31.626 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 31 v 140 (ObjNum 232 snap 27 seq_num 232) dirty exists 2024-03-19T11:31:31.626 INFO:tasks.rados.rados.0.smithi038.stdout:1502: finishing write tid 2 to smithi03829856-4 2024-03-19T11:31:31.626 INFO:tasks.rados.rados.0.smithi038.stdout:1502: finishing write tid 3 to smithi03829856-4 2024-03-19T11:31:31.626 INFO:tasks.rados.rados.0.smithi038.stdout:1496: done (6 left) 2024-03-19T11:31:31.626 INFO:tasks.rados.rados.0.smithi038.stdout:1500: done (5 left) 2024-03-19T11:31:31.626 INFO:tasks.rados.rados.0.smithi038.stdout:1501: done (4 left) 2024-03-19T11:31:31.626 INFO:tasks.rados.rados.0.smithi038.stdout:1504: done (3 left) 2024-03-19T11:31:31.626 INFO:tasks.rados.rados.0.smithi038.stdout:1505: read oid 33 snap -1 2024-03-19T11:31:31.626 INFO:tasks.rados.rados.0.smithi038.stdout:1505: expect (ObjNum 248 snap 30 seq_num 248) 2024-03-19T11:31:31.626 INFO:tasks.rados.rados.0.smithi038.stdout:1506: tier_promote oid 25 2024-03-19T11:31:31.626 INFO:tasks.rados.rados.0.smithi038.stdout:1502: finishing write tid 4 to smithi03829856-4 2024-03-19T11:31:31.626 INFO:tasks.rados.rados.0.smithi038.stdout:1502: oid 4 updating version 0 to 155 2024-03-19T11:31:31.627 INFO:tasks.rados.rados.0.smithi038.stdout:1502: oid 4 version 155 is already newer than 154 2024-03-19T11:31:31.627 INFO:tasks.rados.rados.0.smithi038.stdout:1502: oid 4 version 155 is already newer than 153 2024-03-19T11:31:31.627 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 4 v 155 (ObjNum 261 snap 33 seq_num 261) dirty exists 2024-03-19T11:31:31.627 INFO:tasks.rados.rados.0.smithi038.stdout:1502: left oid 4 (ObjNum 261 snap 33 seq_num 261) 2024-03-19T11:31:31.627 INFO:tasks.rados.rados.0.smithi038.stdout:1503: expect (ObjNum 195 snap 22 seq_num 195) 2024-03-19T11:31:31.697 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:31.706+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:31.746 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:31.746 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:31:31.746 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:31:31.746 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:31.759 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-19T11:31:31.804 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:31.812+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:31.816 INFO:tasks.rados.rados.0.smithi038.stdout:1506: got (2) No such file or directory 2024-03-19T11:31:31.816 INFO:tasks.rados.rados.0.smithi038.stdout:1506: got expected ENOENT (src dne) 2024-03-19T11:31:31.816 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 25 v 131 (ObjNum 97 snap 0 seq_num 0) dirty dne 2024-03-19T11:31:31.816 INFO:tasks.rados.rados.0.smithi038.stdout:1505: expect (ObjNum 248 snap 30 seq_num 248) 2024-03-19T11:31:31.872 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:31.873 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:31:31.873 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:31:31.873 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:31:31.873 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:31.886 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-19T11:31:31.899 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:31.910+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:31.939 INFO:tasks.rados.rados.0.smithi038.stdout:1502: done (4 left) 2024-03-19T11:31:31.939 INFO:tasks.rados.rados.0.smithi038.stdout:1503: done (3 left) 2024-03-19T11:31:31.939 INFO:tasks.rados.rados.0.smithi038.stdout:1505: done (2 left) 2024-03-19T11:31:31.939 INFO:tasks.rados.rados.0.smithi038.stdout:1506: done (1 left) 2024-03-19T11:31:31.939 INFO:tasks.rados.rados.0.smithi038.stdout:1507: set_chunk oid 42 target oid 2024-03-19T11:31:31.939 INFO:tasks.rados.rados.0.smithi038.stdout:1507: set_chunk oid 42 offset: 0 length: 4000000 target oid smithi03829856- offset: 0 snap -1 2024-03-19T11:31:31.939 INFO:tasks.rados.rados.0.smithi038.stdout:1508: write (excl) oid 1 current snap is 33 2024-03-19T11:31:31.939 INFO:tasks.rados.rados.0.smithi038.stdout:1508: seq_num 262 ranges {403434=471240,1315341=400756} 2024-03-19T11:31:31.946 INFO:tasks.rados.rados.0.smithi038.stdout:1508: writing smithi03829856-1 from 403434 to 874674 tid 1 2024-03-19T11:31:31.953 INFO:tasks.rados.rados.0.smithi038.stdout:1508: writing smithi03829856-1 from 1315341 to 1716097 tid 2 2024-03-19T11:31:31.953 INFO:tasks.rados.rados.0.smithi038.stdout:1509: snap_create 2024-03-19T11:31:31.953 INFO:tasks.rados.rados.0.smithi038.stdout:1507: finishing set_chunk to oid 42 2024-03-19T11:31:31.953 INFO:tasks.rados.rados.0.smithi038.stdout:1507: got expected ENOENT (src dne) 2024-03-19T11:31:31.962 INFO:tasks.rados.rados.0.smithi038.stdout:1508: finishing write tid 1 to smithi03829856-1 2024-03-19T11:31:31.962 INFO:tasks.rados.rados.0.smithi038.stdout:1508: finishing write tid 2 to smithi03829856-1 2024-03-19T11:31:31.965 INFO:tasks.rados.rados.0.smithi038.stdout:1508: finishing write tid 3 to smithi03829856-1 2024-03-19T11:31:31.966 INFO:tasks.rados.rados.0.smithi038.stdout:1508: finishing write tid 4 to smithi03829856-1 2024-03-19T11:31:31.966 INFO:tasks.rados.rados.0.smithi038.stdout:1508: oid 1 updating version 0 to 156 2024-03-19T11:31:31.966 INFO:tasks.rados.rados.0.smithi038.stdout:1508: oid 1 updating version 156 to 158 2024-03-19T11:31:31.966 INFO:tasks.rados.rados.0.smithi038.stdout:1508: oid 1 version 158 is already newer than 157 2024-03-19T11:31:31.966 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 1 v 158 (ObjNum 262 snap 33 seq_num 262) dirty exists 2024-03-19T11:31:31.966 INFO:tasks.rados.rados.0.smithi038.stdout:1508: left oid 1 (ObjNum 262 snap 33 seq_num 262) 2024-03-19T11:31:32.000 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:32.010+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:32.056 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:32.056 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:31:32.056 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:31:32.056 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:31:32.056 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.056 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.6.13:59 3.2 3:78b4b14e:::e1ef0c0c594c36c534f3829406e07ffb60d57b70360f62977e0d545d8ca2bf18:head [read 0~149977 [fadvise_sequential]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:31:32.056 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:12.762054+0000", 2024-03-19T11:31:32.056 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 79.300517060000004, 2024-03-19T11:31:32.057 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015354483, 2024-03-19T11:31:32.057 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:32.057 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:32.057 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:32.057 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.057 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:32.057 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762054+0000", 2024-03-19T11:31:32.057 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:32.057 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.057 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.057 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:32.057 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762054+0000", 2024-03-19T11:31:32.057 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:32.057 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.058 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.058 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:32.058 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762055+0000", 2024-03-19T11:31:32.058 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.32e-07 2024-03-19T11:31:32.058 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.058 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.058 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:32.058 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762058+0000", 2024-03-19T11:31:32.058 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9189999999999999e-06 2024-03-19T11:31:32.058 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.058 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.058 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:32.058 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762065+0000", 2024-03-19T11:31:32.058 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2409999999999998e-06 2024-03-19T11:31:32.058 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.059 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.059 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:32.059 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762070+0000", 2024-03-19T11:31:32.059 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6910000000000002e-06 2024-03-19T11:31:32.059 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.059 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.059 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:32.059 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762227+0000", 2024-03-19T11:31:32.059 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015606999999999999 2024-03-19T11:31:32.059 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.059 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.059 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:31:32.059 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.762232+0000", 2024-03-19T11:31:32.060 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3029999999999999e-06 2024-03-19T11:31:32.060 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.060 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.060 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:32.060 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.776984+0000", 2024-03-19T11:31:32.061 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014752487999999999 2024-03-19T11:31:32.061 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.061 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.061 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:32.061 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.777030+0000", 2024-03-19T11:31:32.061 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5359999999999999e-05 2024-03-19T11:31:32.061 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.061 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.061 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:32.061 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:12.777409+0000", 2024-03-19T11:31:32.061 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037897899999999999 2024-03-19T11:31:32.061 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:32.061 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:32.062 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:32.062 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.062 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.062 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1067 4.0 4:01018511:::smithi03829856-19:head [write 477332~766101] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:31:32.062 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.950506+0000", 2024-03-19T11:31:32.062 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 78.112065348000002, 2024-03-19T11:31:32.062 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016279426, 2024-03-19T11:31:32.062 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:32.063 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:32.063 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:32.063 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:32.063 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:32.063 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1067 2024-03-19T11:31:32.063 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.063 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:32.063 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.063 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:32.063 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950506+0000", 2024-03-19T11:31:32.063 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:32.063 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.064 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.064 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:32.064 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950506+0000", 2024-03-19T11:31:32.064 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:32.064 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.064 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.064 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:32.064 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950509+0000", 2024-03-19T11:31:32.064 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.021e-06 2024-03-19T11:31:32.064 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.064 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.064 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:32.064 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950892+0000", 2024-03-19T11:31:32.065 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00038263300000000001 2024-03-19T11:31:32.065 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.065 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.065 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:32.065 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950912+0000", 2024-03-19T11:31:32.065 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0539000000000001e-05 2024-03-19T11:31:32.066 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.066 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.066 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:32.066 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950922+0000", 2024-03-19T11:31:32.066 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.577e-06 2024-03-19T11:31:32.066 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.066 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.066 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:32.066 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950982+0000", 2024-03-19T11:31:32.066 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0760000000000001e-05 2024-03-19T11:31:32.067 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.067 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.067 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:32.067 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.951058+0000", 2024-03-19T11:31:32.067 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5078000000000002e-05 2024-03-19T11:31:32.067 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.067 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.067 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:32.068 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.951173+0000", 2024-03-19T11:31:32.068 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011516200000000001 2024-03-19T11:31:32.068 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.068 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.068 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:32.068 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.955064+0000", 2024-03-19T11:31:32.068 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003890896 2024-03-19T11:31:32.068 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.068 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.068 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:32.068 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966670+0000", 2024-03-19T11:31:32.069 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011606795 2024-03-19T11:31:32.069 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.069 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.069 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:32.069 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966731+0000", 2024-03-19T11:31:32.069 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0528000000000001e-05 2024-03-19T11:31:32.069 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.069 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.069 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:32.069 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966785+0000", 2024-03-19T11:31:32.070 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4437000000000001e-05 2024-03-19T11:31:32.070 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:32.070 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:32.070 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:32.070 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.070 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.070 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1231 4.0 4:0dc3b91e:::smithi03829856-28:head [write 1808051~636013] snapc 9=[9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:31:32.071 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.173016+0000", 2024-03-19T11:31:32.071 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 73.889555235000003, 2024-03-19T11:31:32.071 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013973719000000001, 2024-03-19T11:31:32.071 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:32.071 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:32.071 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:32.071 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:32.071 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:32.071 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1231 2024-03-19T11:31:32.071 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.071 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:32.071 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.071 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:32.071 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173016+0000", 2024-03-19T11:31:32.072 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:32.072 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.072 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.072 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:32.072 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173016+0000", 2024-03-19T11:31:32.072 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:32.072 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.072 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.072 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:32.072 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.173019+0000", 2024-03-19T11:31:32.072 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.253e-06 2024-03-19T11:31:32.072 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.072 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.072 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:32.072 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176143+0000", 2024-03-19T11:31:32.073 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031240930000000001 2024-03-19T11:31:32.073 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.073 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.073 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:32.073 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176177+0000", 2024-03-19T11:31:32.073 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3237e-05 2024-03-19T11:31:32.073 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.073 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.073 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:32.073 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176196+0000", 2024-03-19T11:31:32.073 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9358e-05 2024-03-19T11:31:32.073 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.073 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.073 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:32.074 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176493+0000", 2024-03-19T11:31:32.074 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00029743899999999998 2024-03-19T11:31:32.074 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.074 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.074 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:32.074 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.176786+0000", 2024-03-19T11:31:32.074 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00029259000000000002 2024-03-19T11:31:32.074 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.074 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.074 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:32.074 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.177250+0000", 2024-03-19T11:31:32.074 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00046376000000000002 2024-03-19T11:31:32.074 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.074 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.074 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:32.075 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.182511+0000", 2024-03-19T11:31:32.075 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0052609019999999996 2024-03-19T11:31:32.075 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.075 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.075 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:32.075 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.186896+0000", 2024-03-19T11:31:32.075 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0043857009999999997 2024-03-19T11:31:32.075 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.075 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.075 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:32.075 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.186948+0000", 2024-03-19T11:31:32.075 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1653000000000002e-05 2024-03-19T11:31:32.075 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.076 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.076 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:32.076 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.186990+0000", 2024-03-19T11:31:32.076 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1733000000000002e-05 2024-03-19T11:31:32.076 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:32.076 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:32.077 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:32.077 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.077 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.077 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1409 4.10 4:0dc3b91e:::smithi03829856-28:head [sparse-read 0~1851556] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:31:32.077 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.918248+0000", 2024-03-19T11:31:32.077 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 66.144322946000003, 2024-03-19T11:31:32.077 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013726986, 2024-03-19T11:31:32.077 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:32.077 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:32.077 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:32.077 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:32.077 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:32.078 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1409 2024-03-19T11:31:32.078 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.078 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:32.078 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.078 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:32.078 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918248+0000", 2024-03-19T11:31:32.078 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:32.078 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.078 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.078 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:32.078 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918248+0000", 2024-03-19T11:31:32.078 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:32.078 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.078 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.078 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:32.079 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918249+0000", 2024-03-19T11:31:32.079 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4700000000000002e-07 2024-03-19T11:31:32.079 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.079 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.079 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:32.079 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918251+0000", 2024-03-19T11:31:32.079 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.542e-06 2024-03-19T11:31:32.079 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.079 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.079 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:32.079 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918258+0000", 2024-03-19T11:31:32.080 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4629999999999998e-06 2024-03-19T11:31:32.080 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.080 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.080 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:32.080 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.918263+0000", 2024-03-19T11:31:32.080 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8049999999999997e-06 2024-03-19T11:31:32.080 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.080 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.080 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:32.080 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.931469+0000", 2024-03-19T11:31:32.080 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013206458000000001 2024-03-19T11:31:32.080 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.080 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.080 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:32.081 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.931586+0000", 2024-03-19T11:31:32.081 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000116655 2024-03-19T11:31:32.081 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.081 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.081 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:32.081 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.931975+0000", 2024-03-19T11:31:32.081 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00038961600000000002 2024-03-19T11:31:32.081 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:32.081 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:32.081 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:32.082 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.082 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.082 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(3.1 49/48 [PushOp(3:9205291d:::smithi03829856-50 low_tier:head, version: 28'36, data_included: [393216~851968,1703936~196608], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:9205291d:::smithi03829856-50 low_tier:head@28'36, size: 2823543, copy_subset: [(0, 2823543)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1900544, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:baab691a:::smithi03829856-2 low_tier:head, version: 28'37, data_included: [393216~524288,1507328~509479], data_size: 1033767, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:baab691a:::smithi03829856-2 low_tier:head@28'37, size: 2016807, copy_subset: [(0, 2016807)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2016807, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:98ce8426:::smithi03829856-4 low_tier:head, version: 28'38, data_included: [393216~720896,1638400~327680], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:98ce8426:::smithi03829856-4 low_tier:head@28'38, size: 2050421, copy_subset: [(0, 2050421)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1966080, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:a86be07b:::smithi03829856-44 low_tier:head, version: 28'41, data_included: [720896~655360,1769472~393216], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:a86be07b:::smithi03829856-44 low_tier:head@28'41, size: 2722701, copy_subset: [(0, 2722701)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2162688, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:bb56919e:::smithi03829856-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'44, data_included: [393216~786432,1703936~262144], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:bb56919e:::smithi03829856-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'44, size: 2806736, copy_subset: [(0, 2806736)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1966080, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:88687f71:::smithi03829856-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'45, data_included: [589824~655360,1638400~393216], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:88687f71:::smithi03829856-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'45, size: 2201684, copy_subset: [(0, 2201684)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2031616, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:b1f822cf:::smithi03829856-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'46, data_included: [720896~786432,1966080~262144], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:b1f822cf:::smithi03829856-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'46, size: 2369754, copy_subset: [(0, 2369754)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2228224, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:a5e8313a:::smithi03829856-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'47, data_included: [524288~720896,1769472~327680], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:a5e8313a:::smithi03829856-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'47, size: 2571438, copy_subset: [(0, 2571438)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:bca5ec64:::e3a26bf9230b1b95b8cf1a4cb01909236bf2bf3fd59b083b2291d278ab98264d:head, version: 28'48, data_included: [0~393216], data_size: 393216, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bca5ec64:::e3a26bf9230b1b95b8cf1a4cb01909236bf2bf3fd59b083b2291d278ab98264d:head@28'48, size: 533504, copy_subset: [(0, 533504)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 533504, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:32.082 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:26.948137+0000", 2024-03-19T11:31:32.082 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 65.114434133000003, 2024-03-19T11:31:32.082 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.036085578, 2024-03-19T11:31:32.082 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:32.082 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:32.082 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:32.082 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.082 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:32.082 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.948137+0000", 2024-03-19T11:31:32.083 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:32.083 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.083 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.083 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:32.083 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.948137+0000", 2024-03-19T11:31:32.083 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:32.083 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.083 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.083 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:32.083 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.948139+0000", 2024-03-19T11:31:32.083 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1399999999999998e-06 2024-03-19T11:31:32.083 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.084 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.084 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:32.084 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954288+0000", 2024-03-19T11:31:32.084 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0061487429999999999 2024-03-19T11:31:32.084 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.084 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.084 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:32.084 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954388+0000", 2024-03-19T11:31:32.084 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010022 2024-03-19T11:31:32.084 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.084 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.084 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:32.084 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954523+0000", 2024-03-19T11:31:32.085 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000134332 2024-03-19T11:31:32.085 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.085 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.085 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:32.085 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.955120+0000", 2024-03-19T11:31:32.085 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00059695799999999997 2024-03-19T11:31:32.085 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.085 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.085 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:32.085 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.955194+0000", 2024-03-19T11:31:32.085 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4077000000000002e-05 2024-03-19T11:31:32.085 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.085 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.085 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:32.086 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.984223+0000", 2024-03-19T11:31:32.086 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029029108000000001 2024-03-19T11:31:32.086 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:32.086 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:32.086 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:32.086 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.086 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.086 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(3.1 49/48 [PushOp(3:99a49d1b:::895a63407b6109b3cca9a6f5ef827134d56679bef42daadc5892269682c428ed:head, version: 32'49, data_included: [393216~388096], data_size: 388096, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:99a49d1b:::895a63407b6109b3cca9a6f5ef827134d56679bef42daadc5892269682c428ed:head@32'49, size: 781312, copy_subset: [(0, 781312)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 781312, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:32.086 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:26.954637+0000", 2024-03-19T11:31:32.086 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 65.107934516, 2024-03-19T11:31:32.086 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.030729057000000001, 2024-03-19T11:31:32.086 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:32.086 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:32.086 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:32.087 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.087 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:32.087 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954637+0000", 2024-03-19T11:31:32.087 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:32.087 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.087 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.087 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:32.087 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954637+0000", 2024-03-19T11:31:32.087 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:32.087 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.087 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.087 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:32.087 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954637+0000", 2024-03-19T11:31:32.087 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7299999999999995e-07 2024-03-19T11:31:32.087 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.088 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.088 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:32.088 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954873+0000", 2024-03-19T11:31:32.088 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023520299999999999 2024-03-19T11:31:32.088 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.088 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.088 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:32.088 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954897+0000", 2024-03-19T11:31:32.088 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4110000000000001e-05 2024-03-19T11:31:32.088 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.088 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.088 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:32.088 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954917+0000", 2024-03-19T11:31:32.089 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.067e-05 2024-03-19T11:31:32.089 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.089 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.089 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:32.089 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.984289+0000", 2024-03-19T11:31:32.089 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029371191000000001 2024-03-19T11:31:32.089 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.089 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.089 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:32.089 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.984303+0000", 2024-03-19T11:31:32.089 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4888999999999999e-05 2024-03-19T11:31:32.089 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.089 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.089 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:32.090 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.985366+0000", 2024-03-19T11:31:32.090 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001062321 2024-03-19T11:31:32.090 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:32.090 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:32.090 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:32.090 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.090 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.090 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1472 4.10 4:0dc3b91e:::smithi03829856-28:head [read 534528~391680] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-03-19T11:31:32.090 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:27.091748+0000", 2024-03-19T11:31:32.090 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 64.970823628000005, 2024-03-19T11:31:32.090 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.83574118500000005, 2024-03-19T11:31:32.090 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:32.090 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:32.090 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:32.090 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:32.091 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:32.091 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1472 2024-03-19T11:31:32.091 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.091 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:32.091 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.091 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:32.091 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091748+0000", 2024-03-19T11:31:32.091 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:32.091 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.091 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.091 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:32.091 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091748+0000", 2024-03-19T11:31:32.091 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:32.091 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.091 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.092 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:32.092 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091750+0000", 2024-03-19T11:31:32.092 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3130000000000001e-06 2024-03-19T11:31:32.092 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.092 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.092 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:32.092 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091760+0000", 2024-03-19T11:31:32.092 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0560999999999999e-05 2024-03-19T11:31:32.092 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.092 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.092 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:32.092 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091784+0000", 2024-03-19T11:31:32.092 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3915000000000001e-05 2024-03-19T11:31:32.092 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.092 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.093 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:32.093 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091797+0000", 2024-03-19T11:31:32.093 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2660000000000001e-05 2024-03-19T11:31:32.093 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.093 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.093 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:32.093 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091874+0000", 2024-03-19T11:31:32.093 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6757999999999999e-05 2024-03-19T11:31:32.093 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.093 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.093 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:32.093 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091882+0000", 2024-03-19T11:31:32.093 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5520000000000005e-06 2024-03-19T11:31:32.093 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.093 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.094 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:32.094 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.927033+0000", 2024-03-19T11:31:32.094 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.83515094099999998 2024-03-19T11:31:32.094 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.094 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.094 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:32.094 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.927250+0000", 2024-03-19T11:31:32.094 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00021675199999999999 2024-03-19T11:31:32.094 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.094 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.094 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:32.094 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.927489+0000", 2024-03-19T11:31:32.094 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023873300000000001 2024-03-19T11:31:32.095 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:32.095 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:32.095 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:32.095 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.095 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.095 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1474 4.10 4:0dc3b91e:::smithi03829856-28:head [assert-version v27,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-03-19T11:31:32.095 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:27.934139+0000", 2024-03-19T11:31:32.095 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 64.128431841999998, 2024-03-19T11:31:32.095 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024764884000000001, 2024-03-19T11:31:32.095 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:32.095 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:32.095 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:32.095 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:32.095 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:32.095 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1474 2024-03-19T11:31:32.096 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.096 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:32.096 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.096 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:32.096 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934139+0000", 2024-03-19T11:31:32.096 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:32.096 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.096 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.096 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:32.096 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934139+0000", 2024-03-19T11:31:32.096 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:32.096 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.096 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.096 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:32.096 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934142+0000", 2024-03-19T11:31:32.097 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5459999999999998e-06 2024-03-19T11:31:32.097 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.097 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.097 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:32.097 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934152+0000", 2024-03-19T11:31:32.097 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9399999999999997e-06 2024-03-19T11:31:32.097 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.097 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.097 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:32.097 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934173+0000", 2024-03-19T11:31:32.097 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1522000000000002e-05 2024-03-19T11:31:32.097 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.097 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.097 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:32.097 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934185+0000", 2024-03-19T11:31:32.098 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1946999999999999e-05 2024-03-19T11:31:32.098 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.098 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.098 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:32.098 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934282+0000", 2024-03-19T11:31:32.098 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6272000000000003e-05 2024-03-19T11:31:32.098 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.098 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.098 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:32.098 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934379+0000", 2024-03-19T11:31:32.098 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6966999999999995e-05 2024-03-19T11:31:32.098 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.098 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.098 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:32.098 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.955023+0000", 2024-03-19T11:31:32.099 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020644302 2024-03-19T11:31:32.099 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.099 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.099 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:32.099 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.955167+0000", 2024-03-19T11:31:32.099 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014404500000000001 2024-03-19T11:31:32.099 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.099 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.100 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1", 2024-03-19T11:31:32.100 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.957913+0000", 2024-03-19T11:31:32.100 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0027463779999999998 2024-03-19T11:31:32.100 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.100 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.100 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:32.100 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.958548+0000", 2024-03-19T11:31:32.100 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00063495700000000001 2024-03-19T11:31:32.100 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.100 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.100 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:32.100 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.958785+0000", 2024-03-19T11:31:32.100 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000236872 2024-03-19T11:31:32.100 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.101 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.101 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:32.101 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.958854+0000", 2024-03-19T11:31:32.101 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9099999999999999e-05 2024-03-19T11:31:32.101 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.101 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.101 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:32.101 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.958904+0000", 2024-03-19T11:31:32.101 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0036e-05 2024-03-19T11:31:32.101 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:32.101 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:32.101 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:32.101 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.101 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.102 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2011 4.0 4:01018511:::smithi03829856-19:c [read 0~3563051,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:31:32.102 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:46.264515+0000", 2024-03-19T11:31:32.102 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 45.798056174999999, 2024-03-19T11:31:32.102 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013774359999999999, 2024-03-19T11:31:32.102 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:32.102 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:32.102 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:32.102 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:32.102 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:32.102 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2011 2024-03-19T11:31:32.102 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.102 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:32.102 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.102 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:32.103 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264515+0000", 2024-03-19T11:31:32.106 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:32.106 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.106 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.106 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:32.106 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264515+0000", 2024-03-19T11:31:32.106 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:32.106 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.106 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.106 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:32.107 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264516+0000", 2024-03-19T11:31:32.107 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5899999999999995e-07 2024-03-19T11:31:32.107 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.107 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.107 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:32.107 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264522+0000", 2024-03-19T11:31:32.107 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6019999999999999e-06 2024-03-19T11:31:32.107 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.108 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.108 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:32.108 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264535+0000", 2024-03-19T11:31:32.108 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3207000000000001e-05 2024-03-19T11:31:32.108 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.108 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.108 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:32.108 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264543+0000", 2024-03-19T11:31:32.108 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6270000000000006e-06 2024-03-19T11:31:32.108 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.108 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.108 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:32.109 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.271665+0000", 2024-03-19T11:31:32.109 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.007121881 2024-03-19T11:31:32.109 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.109 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.109 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:32.109 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.271835+0000", 2024-03-19T11:31:32.109 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000169958 2024-03-19T11:31:32.109 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.109 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.109 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:32.110 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.278289+0000", 2024-03-19T11:31:32.110 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0064542260000000004 2024-03-19T11:31:32.110 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:32.110 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:32.110 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:32.110 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.110 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.110 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 75/74 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:7, version: 55'82, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@55'82, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 58'86, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@58'86, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [504210~968657,1572695~79919], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:12, version: 66'101, data_included: [440350~816629,1309629~231947], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:12@66'101, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:10, version: 70'112, data_included: [440350~816629,1309629~231947], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:10@70'112, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 70'113, data_included: [440350~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@70'113, size: 2605085, copy_subset: [(0, 2605085)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1488926, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:32.110 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.108941+0000", 2024-03-19T11:31:32.110 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 37.953630105000002, 2024-03-19T11:31:32.110 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.067446694000000001, 2024-03-19T11:31:32.110 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:32.111 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:32.111 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:32.111 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.111 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:32.111 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.108941+0000", 2024-03-19T11:31:32.111 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:32.111 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.111 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.111 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:32.111 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.108941+0000", 2024-03-19T11:31:32.111 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:32.112 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.112 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.112 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:32.112 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.108943+0000", 2024-03-19T11:31:32.112 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2259999999999999e-06 2024-03-19T11:31:32.112 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.112 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.112 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:32.112 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123892+0000", 2024-03-19T11:31:32.112 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014948655999999999 2024-03-19T11:31:32.112 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.112 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.113 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:32.113 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123993+0000", 2024-03-19T11:31:32.113 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000101236 2024-03-19T11:31:32.113 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.113 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.113 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:32.113 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124126+0000", 2024-03-19T11:31:32.113 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000133184 2024-03-19T11:31:32.113 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.113 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.114 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:32.114 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124655+0000", 2024-03-19T11:31:32.114 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00052849600000000004 2024-03-19T11:31:32.114 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.114 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.114 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:32.114 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124723+0000", 2024-03-19T11:31:32.114 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7797000000000002e-05 2024-03-19T11:31:32.114 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.114 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.114 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:32.114 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.176388+0000", 2024-03-19T11:31:32.114 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.051665098999999999 2024-03-19T11:31:32.115 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:32.115 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:32.115 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:32.115 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.115 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.115 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.21 e75,74 4:8401c1f0:::smithi03829856-43:8,46'66,4:866f3dcc:::smithi03829856-31:2,47'73)", 2024-03-19T11:31:32.115 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.124226+0000", 2024-03-19T11:31:32.115 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 37.938345449000003, 2024-03-19T11:31:32.115 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.052217913999999997, 2024-03-19T11:31:32.115 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:32.115 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:32.116 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:32.116 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.116 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:32.116 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124226+0000", 2024-03-19T11:31:32.116 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:32.116 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.116 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.116 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:32.116 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124226+0000", 2024-03-19T11:31:32.116 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:32.116 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.116 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.117 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:32.117 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124227+0000", 2024-03-19T11:31:32.117 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.477e-06 2024-03-19T11:31:32.117 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.117 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.117 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:32.117 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124240+0000", 2024-03-19T11:31:32.117 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2379e-05 2024-03-19T11:31:32.117 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.117 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.117 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:32.118 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124264+0000", 2024-03-19T11:31:32.118 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4224999999999999e-05 2024-03-19T11:31:32.118 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.118 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.118 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:32.118 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124287+0000", 2024-03-19T11:31:32.118 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3557000000000002e-05 2024-03-19T11:31:32.118 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.118 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.118 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:32.118 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.176274+0000", 2024-03-19T11:31:32.119 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.051986248999999998 2024-03-19T11:31:32.119 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.119 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.119 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:32.119 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.176285+0000", 2024-03-19T11:31:32.119 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0818999999999999e-05 2024-03-19T11:31:32.119 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.119 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.119 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:32.119 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.176444+0000", 2024-03-19T11:31:32.119 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015920800000000001 2024-03-19T11:31:32.119 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:32.119 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:32.119 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:32.119 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.120 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.120 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 75/74 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:7, version: 55'82, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@55'82, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 58'86, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@58'86, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [1652614~973634,2888511~52681,3182305~22261], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3204566, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:12, version: 66'101, data_included: [1541576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:12@66'101, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:10, version: 70'112, data_included: [1541576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:10@70'112, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 70'113, data_included: [1488926~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@70'113, size: 2605085, copy_subset: [(0, 2605085)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2537502, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1488926, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:32.120 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.195615+0000", 2024-03-19T11:31:32.120 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 37.866956551999998, 2024-03-19T11:31:32.120 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.036930510999999999, 2024-03-19T11:31:32.120 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:32.120 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:32.120 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:32.120 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.120 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:32.120 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.195615+0000", 2024-03-19T11:31:32.120 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:32.120 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.120 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.121 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:32.121 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.195615+0000", 2024-03-19T11:31:32.121 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:32.121 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.121 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.121 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:32.121 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.195617+0000", 2024-03-19T11:31:32.121 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0490000000000002e-06 2024-03-19T11:31:32.121 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.121 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.121 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:32.121 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204654+0000", 2024-03-19T11:31:32.121 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0090370339999999993 2024-03-19T11:31:32.121 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.121 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.122 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:32.122 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204730+0000", 2024-03-19T11:31:32.122 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5939000000000006e-05 2024-03-19T11:31:32.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.122 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:32.122 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204844+0000", 2024-03-19T11:31:32.122 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000114066 2024-03-19T11:31:32.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.122 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:32.122 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.205232+0000", 2024-03-19T11:31:32.122 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000388045 2024-03-19T11:31:32.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.123 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:32.123 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.205277+0000", 2024-03-19T11:31:32.123 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5071999999999998e-05 2024-03-19T11:31:32.123 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.123 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.123 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:32.123 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.232545+0000", 2024-03-19T11:31:32.123 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027268305999999999 2024-03-19T11:31:32.123 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:32.123 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:32.123 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:32.123 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.123 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.123 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.21 e75,74 4:866f3dcc:::smithi03829856-31:c,70'114)", 2024-03-19T11:31:32.124 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.204941+0000", 2024-03-19T11:31:32.124 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 37.857629926999998, 2024-03-19T11:31:32.124 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.028797390999999999, 2024-03-19T11:31:32.124 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:32.124 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:32.124 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:32.124 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.124 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:32.124 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204941+0000", 2024-03-19T11:31:32.124 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:32.124 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.124 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.124 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:32.124 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204941+0000", 2024-03-19T11:31:32.124 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:32.125 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.125 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.125 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:32.125 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204943+0000", 2024-03-19T11:31:32.125 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.35e-06 2024-03-19T11:31:32.125 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.125 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.125 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:32.125 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204950+0000", 2024-03-19T11:31:32.125 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.374e-06 2024-03-19T11:31:32.125 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.125 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.125 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:32.125 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204965+0000", 2024-03-19T11:31:32.126 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5169e-05 2024-03-19T11:31:32.126 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.126 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.126 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:32.126 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204979+0000", 2024-03-19T11:31:32.126 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4260000000000001e-05 2024-03-19T11:31:32.126 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.126 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.126 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:32.126 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.232607+0000", 2024-03-19T11:31:32.126 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027627518 2024-03-19T11:31:32.126 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.126 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.126 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:32.127 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.232626+0000", 2024-03-19T11:31:32.127 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9471999999999999e-05 2024-03-19T11:31:32.127 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.127 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.127 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:32.127 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.233739+0000", 2024-03-19T11:31:32.127 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011122479999999999 2024-03-19T11:31:32.127 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:32.127 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:32.127 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:32.127 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.127 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.127 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 75/74 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [2599014~241336], data_size: 241336, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2840350, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [2599014~241336,2948041~446940], data_size: 688276, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3394981, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [3204566~621403], data_size: 621403, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3865577, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3204566, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:12, version: 66'101, data_included: [2590152~250198,2948041~581396], data_size: 831594, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:12@66'101, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3529437, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:10, version: 70'112, data_included: [2590152~250198,2948041~581396], data_size: 831594, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:10@70'112, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3529437, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 70'113, data_included: [2537502~67583], data_size: 67583, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@70'113, size: 2605085, copy_subset: [(0, 2605085)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2605085, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2537502, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:32.127 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.241268+0000", 2024-03-19T11:31:32.128 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 37.821302807000002, 2024-03-19T11:31:32.128 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015726741999999998, 2024-03-19T11:31:32.128 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:32.128 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:32.128 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:32.128 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.128 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:32.128 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.241268+0000", 2024-03-19T11:31:32.128 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:32.128 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.128 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.128 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:32.128 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.241268+0000", 2024-03-19T11:31:32.128 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:32.129 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.129 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.129 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:32.129 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.241270+0000", 2024-03-19T11:31:32.129 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5659999999999999e-06 2024-03-19T11:31:32.129 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.129 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.129 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:32.129 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.245791+0000", 2024-03-19T11:31:32.129 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0045205810000000001 2024-03-19T11:31:32.129 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.129 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.129 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:32.129 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.245853+0000", 2024-03-19T11:31:32.130 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2671999999999999e-05 2024-03-19T11:31:32.130 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.130 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.130 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:32.130 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.245916+0000", 2024-03-19T11:31:32.130 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2626000000000007e-05 2024-03-19T11:31:32.130 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.130 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.130 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:32.130 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.246176+0000", 2024-03-19T11:31:32.130 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000260136 2024-03-19T11:31:32.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.131 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.131 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:32.131 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.246210+0000", 2024-03-19T11:31:32.131 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4183000000000001e-05 2024-03-19T11:31:32.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.131 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.131 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:32.131 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.256995+0000", 2024-03-19T11:31:32.131 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010784978000000001 2024-03-19T11:31:32.131 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:32.131 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:32.131 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:32.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.131 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.132 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 75/74 [PushOp(4:866f3dcc:::smithi03829856-31:head, version: 70'115, data_included: [406769~773083,1439434~275493], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:head@70'115, size: 2806769, copy_subset: [(0, 2806769)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1714927, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:32.132 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.245999+0000", 2024-03-19T11:31:32.132 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 37.816572221000001, 2024-03-19T11:31:32.132 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014650178999999999, 2024-03-19T11:31:32.132 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:32.132 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:32.132 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:32.132 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.132 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:32.132 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.245999+0000", 2024-03-19T11:31:32.132 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:32.132 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.132 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.132 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:32.132 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.245999+0000", 2024-03-19T11:31:32.132 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:32.133 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.133 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.133 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:32.133 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.246000+0000", 2024-03-19T11:31:32.133 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9800000000000003e-07 2024-03-19T11:31:32.133 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.133 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.133 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:32.133 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.246544+0000", 2024-03-19T11:31:32.133 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00054456700000000003 2024-03-19T11:31:32.133 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.133 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.134 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:32.134 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.246578+0000", 2024-03-19T11:31:32.134 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3995999999999998e-05 2024-03-19T11:31:32.134 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.134 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.134 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:32.134 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.246595+0000", 2024-03-19T11:31:32.134 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6558e-05 2024-03-19T11:31:32.134 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.134 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.134 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:32.134 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.257102+0000", 2024-03-19T11:31:32.134 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010506925 2024-03-19T11:31:32.134 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.135 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.135 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:32.135 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.257126+0000", 2024-03-19T11:31:32.135 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4443e-05 2024-03-19T11:31:32.135 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.135 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.135 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:32.135 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.260649+0000", 2024-03-19T11:31:32.135 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003522992 2024-03-19T11:31:32.135 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:32.135 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:32.135 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:32.135 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.136 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.136 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2609 4.19 4:9bd2ffb1:::smithi03829856-30:head [read 0~2487403,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e89)", 2024-03-19T11:31:32.136 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:08.055259+0000", 2024-03-19T11:31:32.136 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 24.007312631000001, 2024-03-19T11:31:32.136 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014845676, 2024-03-19T11:31:32.136 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:32.136 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:32.136 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:32.136 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:32.136 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:32.136 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2609 2024-03-19T11:31:32.136 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.136 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:32.136 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.136 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:32.137 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:08.055259+0000", 2024-03-19T11:31:32.137 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:32.137 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.137 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.137 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:32.137 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:08.055259+0000", 2024-03-19T11:31:32.137 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:32.137 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.137 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.137 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:32.137 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:08.055259+0000", 2024-03-19T11:31:32.137 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8500000000000001e-07 2024-03-19T11:31:32.137 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.138 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.138 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:32.138 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:08.055265+0000", 2024-03-19T11:31:32.138 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.558e-06 2024-03-19T11:31:32.138 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.138 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.138 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:32.138 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:08.055274+0000", 2024-03-19T11:31:32.138 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0000000000000002e-06 2024-03-19T11:31:32.138 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.138 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.138 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:32.138 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:08.055282+0000", 2024-03-19T11:31:32.138 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.011e-06 2024-03-19T11:31:32.138 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.139 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.139 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:32.139 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:08.055667+0000", 2024-03-19T11:31:32.139 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00038538100000000002 2024-03-19T11:31:32.139 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.139 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.139 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:31:32.139 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:08.055688+0000", 2024-03-19T11:31:32.139 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0729000000000001e-05 2024-03-19T11:31:32.139 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.139 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.139 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:32.139 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:08.066671+0000", 2024-03-19T11:31:32.139 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010983504 2024-03-19T11:31:32.139 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.140 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.140 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:32.140 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:08.066996+0000", 2024-03-19T11:31:32.140 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032403899999999998 2024-03-19T11:31:32.140 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.140 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.140 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:32.140 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:08.070104+0000", 2024-03-19T11:31:32.140 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031087689999999999 2024-03-19T11:31:32.140 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:32.140 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:32.140 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:32.140 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.140 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.140 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2780 3.2 3:47f177ee:::0a67e437b20d4ad126b67a2338de84fe4ef684c5c77cee818318d652193ba67c:head [write 0~2533888] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-03-19T11:31:32.141 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:12.308707+0000", 2024-03-19T11:31:32.141 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 19.753863876, 2024-03-19T11:31:32.141 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014502703, 2024-03-19T11:31:32.141 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:32.141 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:32.141 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:32.141 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:32.141 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:32.141 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2780 2024-03-19T11:31:32.141 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.141 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:32.141 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.141 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:32.142 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.308707+0000", 2024-03-19T11:31:32.142 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:32.142 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.142 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.142 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:32.142 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.308707+0000", 2024-03-19T11:31:32.142 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:32.142 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.142 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.142 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:32.142 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.308709+0000", 2024-03-19T11:31:32.142 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6050000000000001e-06 2024-03-19T11:31:32.143 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.143 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.143 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:32.143 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.309858+0000", 2024-03-19T11:31:32.143 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011490339999999999 2024-03-19T11:31:32.143 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.143 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.143 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:32.143 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.309879+0000", 2024-03-19T11:31:32.143 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1254e-05 2024-03-19T11:31:32.143 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.143 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.143 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:32.143 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.309888+0000", 2024-03-19T11:31:32.143 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.67e-06 2024-03-19T11:31:32.144 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.144 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.144 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:32.144 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.310032+0000", 2024-03-19T11:31:32.144 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000144499 2024-03-19T11:31:32.144 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.144 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.144 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:32.144 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.310438+0000", 2024-03-19T11:31:32.144 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00040561999999999998 2024-03-19T11:31:32.144 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.144 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.144 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:31:32.144 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.310835+0000", 2024-03-19T11:31:32.144 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00039648400000000001 2024-03-19T11:31:32.145 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.145 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.145 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:32.145 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.322685+0000", 2024-03-19T11:31:32.145 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011850673000000001 2024-03-19T11:31:32.145 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.145 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.145 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:32.145 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.323012+0000", 2024-03-19T11:31:32.145 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000327032 2024-03-19T11:31:32.145 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.145 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.145 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:32.145 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.323108+0000", 2024-03-19T11:31:32.145 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.5570999999999997e-05 2024-03-19T11:31:32.146 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.146 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.146 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:32.146 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.323210+0000", 2024-03-19T11:31:32.146 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000102261 2024-03-19T11:31:32.146 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:32.146 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:32.146 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:32.146 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.146 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.146 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2789 3.2 3:48367551:::ca87bd16019841aaa831bb522ee079e5f4e6a1ea67fbcb51e5f89263b6c5d8f6:head [write 0~3280896] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-03-19T11:31:32.146 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:12.340289+0000", 2024-03-19T11:31:32.146 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 19.722282211, 2024-03-19T11:31:32.146 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016291298999999999, 2024-03-19T11:31:32.146 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:32.147 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:32.147 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:32.147 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:32.147 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:32.147 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2789 2024-03-19T11:31:32.147 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.147 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:32.147 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.147 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:32.147 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.340289+0000", 2024-03-19T11:31:32.147 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:32.147 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.147 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.147 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:32.147 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.340289+0000", 2024-03-19T11:31:32.148 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:32.148 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.148 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.148 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:32.148 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.340291+0000", 2024-03-19T11:31:32.148 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1270000000000001e-06 2024-03-19T11:31:32.148 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.148 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.148 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:32.148 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.341426+0000", 2024-03-19T11:31:32.148 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011348459999999999 2024-03-19T11:31:32.148 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.148 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.148 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:32.149 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.341454+0000", 2024-03-19T11:31:32.149 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8204e-05 2024-03-19T11:31:32.149 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.149 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.149 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:32.149 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.341467+0000", 2024-03-19T11:31:32.149 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2700999999999999e-05 2024-03-19T11:31:32.149 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.149 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.149 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:32.149 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.341623+0000", 2024-03-19T11:31:32.150 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015600100000000001 2024-03-19T11:31:32.150 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.150 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.150 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:32.150 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.342034+0000", 2024-03-19T11:31:32.150 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000411247 2024-03-19T11:31:32.150 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.150 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.150 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:31:32.150 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.342443+0000", 2024-03-19T11:31:32.150 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00040901400000000002 2024-03-19T11:31:32.150 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.150 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.150 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:32.151 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.356159+0000", 2024-03-19T11:31:32.151 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013716058999999999 2024-03-19T11:31:32.151 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.151 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.151 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:32.151 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.356462+0000", 2024-03-19T11:31:32.151 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000302594 2024-03-19T11:31:32.151 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.151 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.151 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:32.151 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.356530+0000", 2024-03-19T11:31:32.151 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7852e-05 2024-03-19T11:31:32.151 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.151 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.151 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:32.152 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.356580+0000", 2024-03-19T11:31:32.152 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0654e-05 2024-03-19T11:31:32.152 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:32.152 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:32.152 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:32.152 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.152 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.152 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3046 4.19 4:9c74b127:::smithi03829856-33:head [write 400066~716205,stat] snapc 1f=[1f,1e,1d,1c,1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-03-19T11:31:32.152 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:22.147457+0000", 2024-03-19T11:31:32.152 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 9.9151139270000002, 2024-03-19T11:31:32.152 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016153588, 2024-03-19T11:31:32.152 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:32.152 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:32.152 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:32.153 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:32.153 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:32.153 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3046 2024-03-19T11:31:32.153 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.153 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:32.153 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.153 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:32.153 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.147457+0000", 2024-03-19T11:31:32.153 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:32.153 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.153 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.153 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:32.153 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.147457+0000", 2024-03-19T11:31:32.154 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:32.154 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.154 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.154 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:32.154 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.147459+0000", 2024-03-19T11:31:32.154 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1160000000000002e-06 2024-03-19T11:31:32.154 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.154 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.154 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:32.154 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.148083+0000", 2024-03-19T11:31:32.154 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00062361999999999997 2024-03-19T11:31:32.154 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.154 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.154 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:32.155 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.148100+0000", 2024-03-19T11:31:32.155 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6524999999999998e-05 2024-03-19T11:31:32.155 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.155 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.155 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:32.155 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.148110+0000", 2024-03-19T11:31:32.155 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0093e-05 2024-03-19T11:31:32.155 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.155 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.155 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:32.155 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.148197+0000", 2024-03-19T11:31:32.155 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7639000000000006e-05 2024-03-19T11:31:32.155 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.155 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.155 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:32.156 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.148359+0000", 2024-03-19T11:31:32.156 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000161953 2024-03-19T11:31:32.156 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.156 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.156 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 2", 2024-03-19T11:31:32.156 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.149074+0000", 2024-03-19T11:31:32.156 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00071444100000000001 2024-03-19T11:31:32.156 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.156 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.156 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:32.156 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.158856+0000", 2024-03-19T11:31:32.156 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0097818169999999999 2024-03-19T11:31:32.156 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.156 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.157 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:32.157 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.163525+0000", 2024-03-19T11:31:32.157 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046690270000000001 2024-03-19T11:31:32.157 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.157 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.157 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:32.157 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.163569+0000", 2024-03-19T11:31:32.157 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4279000000000001e-05 2024-03-19T11:31:32.157 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.157 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.157 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:32.157 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.163611+0000", 2024-03-19T11:31:32.157 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2077999999999999e-05 2024-03-19T11:31:32.157 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:32.157 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:32.158 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:32.158 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.158 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.158 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3228 4.19 4:9bd2ffb1:::smithi03829856-30:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e109)", 2024-03-19T11:31:32.158 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:28.084149+0000", 2024-03-19T11:31:32.158 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.9784219480000003, 2024-03-19T11:31:32.158 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.049907076000000002, 2024-03-19T11:31:32.158 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:32.158 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:32.158 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:32.158 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:32.158 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:32.158 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3228 2024-03-19T11:31:32.158 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.159 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:32.159 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.159 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:32.159 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.084149+0000", 2024-03-19T11:31:32.159 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:32.159 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.159 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.159 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:32.159 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.084149+0000", 2024-03-19T11:31:32.159 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:32.159 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.159 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.159 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:32.159 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.084152+0000", 2024-03-19T11:31:32.159 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2409999999999998e-06 2024-03-19T11:31:32.160 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.160 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.160 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:32.160 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.084161+0000", 2024-03-19T11:31:32.160 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9609999999999994e-06 2024-03-19T11:31:32.160 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.160 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.160 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:32.160 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.084181+0000", 2024-03-19T11:31:32.160 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0786999999999999e-05 2024-03-19T11:31:32.160 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.160 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.160 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:32.160 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.084194+0000", 2024-03-19T11:31:32.160 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2824e-05 2024-03-19T11:31:32.161 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.161 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.161 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:32.161 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.086656+0000", 2024-03-19T11:31:32.161 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0024622480000000002 2024-03-19T11:31:32.161 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.161 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.161 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:32.161 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.086894+0000", 2024-03-19T11:31:32.161 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000237414 2024-03-19T11:31:32.161 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:32.161 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:32.161 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:32.161 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.134056+0000", 2024-03-19T11:31:32.161 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.047162600999999998 2024-03-19T11:31:32.162 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:32.162 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:32.162 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:32.162 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:32.162 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:32.162 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:32.163 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-19T11:31:32.164 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:32.170+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:32.225 INFO:tasks.rados.rados.0.smithi038.stdout:1507: done (3 left) 2024-03-19T11:31:32.226 INFO:tasks.rados.rados.0.smithi038.stdout:1508: done (2 left) 2024-03-19T11:31:32.226 INFO:tasks.rados.rados.0.smithi038.stdout:1509: done (1 left) 2024-03-19T11:31:32.226 INFO:tasks.rados.rados.0.smithi038.stdout:1510: read oid 6 snap -1 2024-03-19T11:31:32.226 INFO:tasks.rados.rados.0.smithi038.stdout:1510: expect (ObjNum 195 snap 22 seq_num 195) 2024-03-19T11:31:32.226 INFO:tasks.rados.rados.0.smithi038.stdout:1511: write (excl) oid 5 current snap is 34 2024-03-19T11:31:32.226 INFO:tasks.rados.rados.0.smithi038.stdout:1511: seq_num 263 ranges {420241=546489,1373823=544414} 2024-03-19T11:31:32.226 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-19T11:31:32.226 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-19T11:31:32.227 INFO:tasks.thrashosds.thrasher:Removing osd 3, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-19T11:31:32.227 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 3 2024-03-19T11:31:32.231 INFO:tasks.rados.rados.0.smithi038.stdout:1511: writing smithi03829856-5 from 420241 to 966730 tid 1 2024-03-19T11:31:32.236 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:32.240+0000 7f70c4fdb640 -1 osd.0 111 handle_osd_map injecting map crc failure 2024-03-19T11:31:32.239 INFO:tasks.rados.rados.0.smithi038.stdout:1511: writing smithi03829856-5 from 1373823 to 1918237 tid 2 2024-03-19T11:31:32.239 INFO:tasks.rados.rados.0.smithi038.stdout:1512: read oid 12 snap -1 2024-03-19T11:31:32.239 INFO:tasks.rados.rados.0.smithi038.stdout:1512: expect (ObjNum 228 snap 26 seq_num 228) 2024-03-19T11:31:32.240 INFO:tasks.rados.rados.0.smithi038.stdout:1513: read oid 11 snap -1 2024-03-19T11:31:32.240 INFO:tasks.rados.rados.0.smithi038.stdout:1513: expect (ObjNum 229 snap 26 seq_num 229) 2024-03-19T11:31:32.240 INFO:tasks.rados.rados.0.smithi038.stdout:1514: delete oid 27 current snap is 34 2024-03-19T11:31:32.260 INFO:tasks.rados.rados.0.smithi038.stdout:1510: expect (ObjNum 195 snap 22 seq_num 195) 2024-03-19T11:31:32.264 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:32.272+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:32.306 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:32.306 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:31:32.306 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:31:32.306 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:32.320 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-19T11:31:32.366 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:32.372+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:32.443 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:32.443 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:31:32.443 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:31:32.443 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:31:32.443 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:32.450 INFO:tasks.rados.rados.0.smithi038.stdout:1511: finishing write tid 1 to smithi03829856-5 2024-03-19T11:31:32.450 INFO:tasks.rados.rados.0.smithi038.stdout:1511: finishing write tid 2 to smithi03829856-5 2024-03-19T11:31:32.450 INFO:tasks.rados.rados.0.smithi038.stdout:1511: finishing write tid 3 to smithi03829856-5 2024-03-19T11:31:32.451 INFO:tasks.rados.rados.0.smithi038.stdout:1511: finishing write tid 4 to smithi03829856-5 2024-03-19T11:31:32.451 INFO:tasks.rados.rados.0.smithi038.stdout:1511: oid 5 updating version 0 to 173 2024-03-19T11:31:32.451 INFO:tasks.rados.rados.0.smithi038.stdout:1511: oid 5 updating version 173 to 174 2024-03-19T11:31:32.451 INFO:tasks.rados.rados.0.smithi038.stdout:1511: oid 5 updating version 174 to 175 2024-03-19T11:31:32.451 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 5 v 175 (ObjNum 263 snap 34 seq_num 263) dirty exists 2024-03-19T11:31:32.451 INFO:tasks.rados.rados.0.smithi038.stdout:1511: left oid 5 (ObjNum 263 snap 34 seq_num 263) 2024-03-19T11:31:32.451 INFO:tasks.rados.rados.0.smithi038.stdout:1513: expect (ObjNum 229 snap 26 seq_num 229) 2024-03-19T11:31:32.457 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-19T11:31:32.504 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:32.514+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:32.536 INFO:tasks.ceph.mon.a.smithi038.stderr:2024-03-19T11:31:32.544+0000 7f3f60a19640 -1 mon.a@0(leader).osd e112 definitely_dead 0 2024-03-19T11:31:32.568 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:32.578+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:32.585 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:32.585 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:31:32.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:31:32.585 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:31:32.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.585 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:851 4.9 4:953e720b:::smithi03829856-45:head [write 591613~454162,stat] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:31:32.585 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.763514+0000", 2024-03-19T11:31:32.585 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 83.828129473000004, 2024-03-19T11:31:32.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027825888999999999, 2024-03-19T11:31:32.585 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:32.585 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:32.585 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:32.586 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:32.586 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:32.586 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 851 2024-03-19T11:31:32.586 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.586 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:32.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.586 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:32.586 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763514+0000", 2024-03-19T11:31:32.586 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:32.586 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.586 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:32.586 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763514+0000", 2024-03-19T11:31:32.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:32.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.587 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:32.587 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763517+0000", 2024-03-19T11:31:32.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8499999999999998e-06 2024-03-19T11:31:32.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.587 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:32.587 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.764714+0000", 2024-03-19T11:31:32.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011971270000000001 2024-03-19T11:31:32.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.587 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:32.588 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.764745+0000", 2024-03-19T11:31:32.588 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0546999999999997e-05 2024-03-19T11:31:32.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.588 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.588 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:32.588 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.764767+0000", 2024-03-19T11:31:32.588 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2286999999999999e-05 2024-03-19T11:31:32.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.588 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.588 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:32.588 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.773924+0000", 2024-03-19T11:31:32.588 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0091567360000000004 2024-03-19T11:31:32.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.588 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.588 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:32.589 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.774217+0000", 2024-03-19T11:31:32.589 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029350099999999999 2024-03-19T11:31:32.589 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.589 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-19T11:31:32.589 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.774770+0000", 2024-03-19T11:31:32.589 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00055288600000000005 2024-03-19T11:31:32.589 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.589 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:32.589 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.786413+0000", 2024-03-19T11:31:32.589 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011643229999999999 2024-03-19T11:31:32.589 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.590 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:32.590 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791104+0000", 2024-03-19T11:31:32.590 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0046900830000000003 2024-03-19T11:31:32.590 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.590 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.590 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:32.590 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791248+0000", 2024-03-19T11:31:32.590 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014410200000000001 2024-03-19T11:31:32.590 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.590 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.590 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:32.590 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791340+0000", 2024-03-19T11:31:32.590 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2540000000000005e-05 2024-03-19T11:31:32.590 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:32.591 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:32.591 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:32.591 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.591 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1162 4.1 4:8401c1f0:::smithi03829856-43:head [write 594981~609197] snapc 8=[8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-03-19T11:31:32.591 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.838981+0000", 2024-03-19T11:31:32.591 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 75.752662982000004, 2024-03-19T11:31:32.591 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027704349999999999, 2024-03-19T11:31:32.591 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:32.591 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:32.591 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:32.591 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:32.591 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:32.591 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1162 2024-03-19T11:31:32.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.592 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:32.592 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.592 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:32.592 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.838981+0000", 2024-03-19T11:31:32.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:32.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.592 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.592 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:32.592 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.838981+0000", 2024-03-19T11:31:32.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:32.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.592 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.593 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:32.593 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.838983+0000", 2024-03-19T11:31:32.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5399999999999998e-06 2024-03-19T11:31:32.593 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.593 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.593 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:32.593 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841322+0000", 2024-03-19T11:31:32.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002338592 2024-03-19T11:31:32.593 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.593 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.593 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:32.593 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841371+0000", 2024-03-19T11:31:32.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9123999999999999e-05 2024-03-19T11:31:32.593 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.594 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.594 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:32.594 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841399+0000", 2024-03-19T11:31:32.594 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7718000000000001e-05 2024-03-19T11:31:32.594 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.594 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.594 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:32.594 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841577+0000", 2024-03-19T11:31:32.594 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000178097 2024-03-19T11:31:32.594 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.594 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.594 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:32.594 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841825+0000", 2024-03-19T11:31:32.595 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00024824299999999999 2024-03-19T11:31:32.595 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.595 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.595 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-19T11:31:32.595 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842227+0000", 2024-03-19T11:31:32.595 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00040212199999999998 2024-03-19T11:31:32.595 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.595 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.595 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:32.595 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.865182+0000", 2024-03-19T11:31:32.595 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022954810999999999 2024-03-19T11:31:32.595 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.595 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:32.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.866360+0000", 2024-03-19T11:31:32.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001178449 2024-03-19T11:31:32.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:32.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.866538+0000", 2024-03-19T11:31:32.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017791000000000001 2024-03-19T11:31:32.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:32.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.866685+0000", 2024-03-19T11:31:32.597 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000146744 2024-03-19T11:31:32.597 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:32.597 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:32.597 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:32.597 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.597 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1271 4.9 4:953e720b:::smithi03829856-45:2 [read 795136~1008128] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-03-19T11:31:32.597 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:21.868056+0000", 2024-03-19T11:31:32.597 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 70.723587476999995, 2024-03-19T11:31:32.597 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.047270591000000001, 2024-03-19T11:31:32.597 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:32.598 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:32.598 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:32.598 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:32.598 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:32.598 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1271 2024-03-19T11:31:32.598 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.598 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:32.598 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.598 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:32.599 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868056+0000", 2024-03-19T11:31:32.599 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:32.599 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.599 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.599 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:32.599 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868056+0000", 2024-03-19T11:31:32.599 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:32.599 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.599 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.599 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:32.599 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868059+0000", 2024-03-19T11:31:32.600 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0469999999999998e-06 2024-03-19T11:31:32.600 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.600 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.600 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:32.600 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868072+0000", 2024-03-19T11:31:32.600 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2700999999999999e-05 2024-03-19T11:31:32.600 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.601 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.601 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:32.601 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868106+0000", 2024-03-19T11:31:32.601 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4152000000000001e-05 2024-03-19T11:31:32.601 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.601 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.601 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:32.601 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868126+0000", 2024-03-19T11:31:32.601 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9386999999999999e-05 2024-03-19T11:31:32.601 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.601 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.602 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:32.602 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.911447+0000", 2024-03-19T11:31:32.602 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.043321164000000002 2024-03-19T11:31:32.602 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.602 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.602 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:32.602 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.911762+0000", 2024-03-19T11:31:32.602 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00031492400000000001 2024-03-19T11:31:32.602 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.602 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.603 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:32.603 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.915327+0000", 2024-03-19T11:31:32.603 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003565216 2024-03-19T11:31:32.603 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:32.603 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:32.603 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:32.603 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.603 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.603 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(3.1 84/83 [PushOp(3:9205291d:::smithi03829856-50 low_tier:head, version: 28'36, data_included: [393216~851968,1703936~196608], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:9205291d:::smithi03829856-50 low_tier:head@28'36, size: 2823543, copy_subset: [(0, 2823543)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1900544, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:baab691a:::smithi03829856-2 low_tier:head, version: 28'37, data_included: [393216~524288,1507328~509479], data_size: 1033767, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:baab691a:::smithi03829856-2 low_tier:head@28'37, size: 2016807, copy_subset: [(0, 2016807)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2016807, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:98ce8426:::smithi03829856-4 low_tier:head, version: 28'38, data_included: [393216~720896,1638400~327680], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:98ce8426:::smithi03829856-4 low_tier:head@28'38, size: 2050421, copy_subset: [(0, 2050421)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1966080, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:a86be07b:::smithi03829856-44 low_tier:head, version: 28'41, data_included: [720896~655360,1769472~393216], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:a86be07b:::smithi03829856-44 low_tier:head@28'41, size: 2722701, copy_subset: [(0, 2722701)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2162688, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:bb56919e:::smithi03829856-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'44, data_included: [393216~786432,1703936~262144], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:bb56919e:::smithi03829856-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'44, size: 2806736, copy_subset: [(0, 2806736)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1966080, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:b1f822cf:::smithi03829856-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'46, data_included: [720896~786432,1966080~262144], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:b1f822cf:::smithi03829856-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'46, size: 2369754, copy_subset: [(0, 2369754)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2228224, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:a5e8313a:::smithi03829856-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'47, data_included: [524288~720896,1769472~327680], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:a5e8313a:::smithi03829856-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'47, size: 2571438, copy_subset: [(0, 2571438)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:bca5ec64:::e3a26bf9230b1b95b8cf1a4cb01909236bf2bf3fd59b083b2291d278ab98264d:head, version: 28'48, data_included: [0~393216], data_size: 393216, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bca5ec64:::e3a26bf9230b1b95b8cf1a4cb01909236bf2bf3fd59b083b2291d278ab98264d:head@28'48, size: 533504, copy_subset: [(0, 533504)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 533504, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:99a49d1b:::895a63407b6109b3cca9a6f5ef827134d56679bef42daadc5892269682c428ed:head, version: 32'49, data_included: [393216~388096], data_size: 388096, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:99a49d1b:::895a63407b6109b3cca9a6f5ef827134d56679bef42daadc5892269682c428ed:head@32'49, size: 781312, copy_subset: [(0, 781312)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 781312, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:83064589:::a3639ef4c215b429a637f1406db2398074b33464b996fe20e20d8b66c6bd720b:head, version: 33'51, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:83064589:::a3639ef4c215b429a637f1406db2398074b33464b996fe20e20d8b66c6bd720b:head@33'51, size: 1606656, copy_subset: [(0, 1606656)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:32.603 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:03.092965+0000", 2024-03-19T11:31:32.604 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.498678433999999, 2024-03-19T11:31:32.604 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.061194046000000002, 2024-03-19T11:31:32.604 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:32.604 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:32.604 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:32.604 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.604 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:32.604 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.092965+0000", 2024-03-19T11:31:32.604 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:32.604 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.605 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:32.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.092965+0000", 2024-03-19T11:31:32.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:32.605 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.605 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:32.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.092968+0000", 2024-03-19T11:31:32.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8720000000000002e-06 2024-03-19T11:31:32.605 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.606 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:32.606 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.106869+0000", 2024-03-19T11:31:32.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013900554000000001 2024-03-19T11:31:32.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.606 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:32.606 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.107077+0000", 2024-03-19T11:31:32.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000208626 2024-03-19T11:31:32.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:32.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.107372+0000", 2024-03-19T11:31:32.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029439199999999998 2024-03-19T11:31:32.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:32.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.108053+0000", 2024-03-19T11:31:32.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00068115899999999997 2024-03-19T11:31:32.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.608 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:32.608 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.108165+0000", 2024-03-19T11:31:32.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011162000000000001 2024-03-19T11:31:32.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.608 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:32.608 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.154159+0000", 2024-03-19T11:31:32.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.045994822999999997 2024-03-19T11:31:32.608 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:32.608 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:32.608 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:32.609 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.609 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.5 84/83 [PushOp(4:a0a50d04:::smithi03829856-13:b, version: 56'65, data_included: [601684~619341,1669078~429235], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a0a50d04:::smithi03829856-13:b@56'65, size: 2201684, copy_subset: [(0, 2201684)], clone_subset: {}, snapset: 17=[]:{b: [b, 6], 12: [12, d], 14: [13], 17: [17, 16, 15]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2098313, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:a0a50d04:::smithi03829856-13:14, version: 74'84, data_included: [601684~754801,1669078~293775], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a0a50d04:::smithi03829856-13:14@74'84, size: 2722734, copy_subset: [(0, 2722734)], clone_subset: {}, snapset: 17=[]:{b: [b, 6], 12: [12, d], 14: [13], 17: [17, 16, 15]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1962853, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:a0a50d04:::smithi03829856-13:12, version: 81'90, data_included: [601684~619341,1669078~429235], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a0a50d04:::smithi03829856-13:12@81'90, size: 2201684, copy_subset: [(0, 2201684)], clone_subset: {}, snapset: 17=[]:{b: [b, 6], 12: [12, d], 14: [13], 17: [17, 16, 15]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2098313, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:a38b9453:::smithi03829856-37:13, version: 81'92, data_included: [605052~774376,1923587~274200], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a38b9453:::smithi03829856-37:13@81'92, size: 2605052, copy_subset: [(0, 2605052)], clone_subset: {}, snapset: 14=[]:{13: [13, 12, d, b, 6]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2197787, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:a0a50d04:::smithi03829856-13:17, version: 82'94, data_included: [601684~754801,1669078~293775], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a0a50d04:::smithi03829856-13:17@82'94, size: 3025260, copy_subset: [(0, 3025260)], clone_subset: {}, snapset: 17=[]:{b: [b, 6], 12: [12, d], 14: [13], 17: [17, 16, 15]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1962853, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:a0a50d04:::smithi03829856-13:head, version: 82'97, data_included: [601684~619341,1669078~429235], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:a0a50d04:::smithi03829856-13:head@82'97, size: 2201684, copy_subset: [(0, 2201684)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2098313, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:32.609 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:03.103661+0000", 2024-03-19T11:31:32.609 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.487982413000001, 2024-03-19T11:31:32.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027776637, 2024-03-19T11:31:32.609 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:32.609 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:32.609 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:32.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.610 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:32.610 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.103661+0000", 2024-03-19T11:31:32.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:32.610 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.610 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:32.610 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.103661+0000", 2024-03-19T11:31:32.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:32.610 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.610 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:32.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.103664+0000", 2024-03-19T11:31:32.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.48e-06 2024-03-19T11:31:32.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:32.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.109797+0000", 2024-03-19T11:31:32.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061333380000000003 2024-03-19T11:31:32.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:32.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.109866+0000", 2024-03-19T11:31:32.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8474999999999997e-05 2024-03-19T11:31:32.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:32.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.109957+0000", 2024-03-19T11:31:32.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1669000000000006e-05 2024-03-19T11:31:32.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:32.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110162+0000", 2024-03-19T11:31:32.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000204417 2024-03-19T11:31:32.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.613 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:32.613 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110186+0000", 2024-03-19T11:31:32.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4729e-05 2024-03-19T11:31:32.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.613 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:32.613 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.131438+0000", 2024-03-19T11:31:32.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021251529000000002 2024-03-19T11:31:32.613 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:32.614 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:32.614 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:32.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.614 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.5 84/83 [PushOp(4:a38b9453:::smithi03829856-37:head, version: 81'93, data_included: [1775443~781480,2724284~166520], data_size: 948000, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:a38b9453:::smithi03829856-37:head@81'93, size: 2958032, copy_subset: [(0, 2958032)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2958032, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1775443, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:32.614 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:03.110032+0000", 2024-03-19T11:31:32.614 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 29.481612184999999, 2024-03-19T11:31:32.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027756439000000001, 2024-03-19T11:31:32.614 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:32.614 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:32.615 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:32.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:32.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110032+0000", 2024-03-19T11:31:32.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:32.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:32.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110032+0000", 2024-03-19T11:31:32.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:32.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:32.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110033+0000", 2024-03-19T11:31:32.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1510000000000001e-06 2024-03-19T11:31:32.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:32.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110353+0000", 2024-03-19T11:31:32.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00031981199999999999 2024-03-19T11:31:32.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:32.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110375+0000", 2024-03-19T11:31:32.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2772000000000002e-05 2024-03-19T11:31:32.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:32.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110396+0000", 2024-03-19T11:31:32.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1066000000000001e-05 2024-03-19T11:31:32.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.617 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:32.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.131543+0000", 2024-03-19T11:31:32.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021146166000000001 2024-03-19T11:31:32.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.617 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:32.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.131563+0000", 2024-03-19T11:31:32.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0750999999999998e-05 2024-03-19T11:31:32.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.617 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:32.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.137788+0000", 2024-03-19T11:31:32.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062247209999999999 2024-03-19T11:31:32.618 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:32.618 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:32.618 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:32.618 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.618 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.18 85/84 [PushOp(4:1d58db11:::smithi03829856-18:6, version: 55'80, data_included: [759683~639358,1904434~409218], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:6@55'80, size: 3159683, copy_subset: [(0, 3159683)], clone_subset: {}, snapset: 16=[]:{6: [6], b: [b], 15: [15, 13, 12, d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2313652, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1d58db11:::smithi03829856-18:b, version: 58'84, data_included: [443718~955323,1564138~93253], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:b@58'84, size: 3445402, copy_subset: [(0, 3445402)], clone_subset: {}, snapset: 16=[]:{6: [6], b: [b], 15: [15, 13, 12, d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1657391, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:10, version: 66'95, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:10@66'95, size: 3764735, copy_subset: [(0, 3764735)], clone_subset: {}, snapset: 15=[]:{c: [b, 6], 10: [d], 12: [12], 15: [15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:c, version: 70'102, data_included: [416807~576167,1542129~472409], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:c@70'102, size: 2857157, copy_subset: [(0, 2857157)], clone_subset: {}, snapset: 15=[]:{c: [b, 6], 10: [d], 12: [12], 15: [15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2014538, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:15, version: 77'110, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:15@77'110, size: 2621892, copy_subset: [(0, 2621892)], clone_subset: {}, snapset: 15=[]:{c: [b, 6], 10: [d], 12: [12], 15: [15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:32.618 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:04.083107+0000", 2024-03-19T11:31:32.618 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.508536619000001, 2024-03-19T11:31:32.618 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.042109168000000002, 2024-03-19T11:31:32.618 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:32.618 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:32.618 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:32.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.618 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:32.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.083107+0000", 2024-03-19T11:31:32.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:32.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:32.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.083107+0000", 2024-03-19T11:31:32.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:32.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:32.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.083109+0000", 2024-03-19T11:31:32.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7060000000000001e-06 2024-03-19T11:31:32.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:32.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092477+0000", 2024-03-19T11:31:32.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0093678589999999992 2024-03-19T11:31:32.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:32.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092549+0000", 2024-03-19T11:31:32.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2313999999999999e-05 2024-03-19T11:31:32.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:32.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092640+0000", 2024-03-19T11:31:32.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1262999999999997e-05 2024-03-19T11:31:32.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:32.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.093076+0000", 2024-03-19T11:31:32.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00043617800000000001 2024-03-19T11:31:32.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:32.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.093145+0000", 2024-03-19T11:31:32.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8651000000000004e-05 2024-03-19T11:31:32.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:32.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.125216+0000", 2024-03-19T11:31:32.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032071197000000003 2024-03-19T11:31:32.621 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:32.622 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:32.622 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:32.622 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.622 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.622 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.18 e85,84 4:1d58db11:::smithi03829856-18:7,38'48,4:1d58db11:::smithi03829856-18:9,46'64,4:1d58db11:::smithi03829856-18:2,48'76,4:1d58db11:::smithi03829856-18:c,70'104)", 2024-03-19T11:31:32.622 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:04.092733+0000", 2024-03-19T11:31:32.622 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.498910257999999, 2024-03-19T11:31:32.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033436452999999998, 2024-03-19T11:31:32.622 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:32.622 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:32.622 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:32.622 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.622 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:32.622 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092733+0000", 2024-03-19T11:31:32.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:32.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:32.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092733+0000", 2024-03-19T11:31:32.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:32.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:32.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092734+0000", 2024-03-19T11:31:32.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0699999999999996e-07 2024-03-19T11:31:32.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:32.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092745+0000", 2024-03-19T11:31:32.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0393e-05 2024-03-19T11:31:32.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:32.624 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092762+0000", 2024-03-19T11:31:32.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7649e-05 2024-03-19T11:31:32.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:32.624 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092778+0000", 2024-03-19T11:31:32.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5625999999999999e-05 2024-03-19T11:31:32.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:32.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.125342+0000", 2024-03-19T11:31:32.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032564348 2024-03-19T11:31:32.625 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:32.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.125375+0000", 2024-03-19T11:31:32.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2441000000000003e-05 2024-03-19T11:31:32.625 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:32.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.126170+0000", 2024-03-19T11:31:32.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00079508900000000002 2024-03-19T11:31:32.625 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:32.625 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:32.625 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:32.626 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.626 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(3.3 85/84 [PushOp(3:fb6cb292:::smithi03829856-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'21, data_included: [655360~851968,1900544~196608], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:fb6cb292:::smithi03829856-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'21, size: 2302526, copy_subset: [(0, 2302526)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:d281c5a5:::smithi03829856-30 low_tier:head, version: 28'42, data_included: [458752~589824,1507328~458752], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:d281c5a5:::smithi03829856-30 low_tier:head@28'42, size: 2487403, copy_subset: [(0, 2487403)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1966080, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:ed957b32:::smithi03829856-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'44, data_included: [524288~786432,1900544~233912], data_size: 1020344, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:ed957b32:::smithi03829856-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'44, size: 2134456, copy_subset: [(0, 2134456)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2134456, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:f23a45c7:::smithi03829856-36 low_tier:head, version: 28'45, data_included: [524288~786432,1835008~262144], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:f23a45c7:::smithi03829856-36 low_tier:head@28'45, size: 2588245, copy_subset: [(0, 2588245)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:edb38de0:::smithi03829856-40 low_tier:head, version: 28'46, data_included: [655360~720896,2097152~327680], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:edb38de0:::smithi03829856-40 low_tier:head@28'46, size: 2655473, copy_subset: [(0, 2655473)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2424832, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:f92b3f40:::b1fb925e39ab40886d36b70ff1c8c4f867ef6cb2ef150fbe90de57d057dfec36:head, version: 29'47, data_included: [262144~720896,1507328~327680], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:f92b3f40:::b1fb925e39ab40886d36b70ff1c8c4f867ef6cb2ef150fbe90de57d057dfec36:head@29'47, size: 2098176, copy_subset: [(0, 2098176)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1835008, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:fc3d03e1:::626f73f06d73fccc02b43398389ff991d56d81381f5e170c36fb9439a43cbbc0:head, version: 32'48, data_included: [0~131072], data_size: 131072, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:fc3d03e1:::626f73f06d73fccc02b43398389ff991d56d81381f5e170c36fb9439a43cbbc0:head@32'48, size: 462848, copy_subset: [(0, 462848)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 462848, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:f8934067:::8892b7b523af2cde6be0d91d0468f8be86a651cb052f82daf3e2c6278805eb6e:head, version: 32'49, data_included: [0~131072], data_size: 131072, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:f8934067:::8892b7b523af2cde6be0d91d0468f8be86a651cb052f82daf3e2c6278805eb6e:head@32'49, size: 202240, copy_subset: [(0, 202240)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 202240, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:32.626 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:04.108513+0000", 2024-03-19T11:31:32.626 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.483130816999999, 2024-03-19T11:31:32.626 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.036668899999999997, 2024-03-19T11:31:32.626 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:32.626 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:32.626 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:32.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.626 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:32.626 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.108513+0000", 2024-03-19T11:31:32.626 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:32.626 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:32.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.108513+0000", 2024-03-19T11:31:32.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:32.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:32.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.108515+0000", 2024-03-19T11:31:32.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7579999999999999e-06 2024-03-19T11:31:32.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:32.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115352+0000", 2024-03-19T11:31:32.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0068369950000000002 2024-03-19T11:31:32.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:32.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115421+0000", 2024-03-19T11:31:32.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9282999999999994e-05 2024-03-19T11:31:32.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:32.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115505+0000", 2024-03-19T11:31:32.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3714999999999994e-05 2024-03-19T11:31:32.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:32.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115948+0000", 2024-03-19T11:31:32.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000443036 2024-03-19T11:31:32.629 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.629 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.629 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:32.629 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116012+0000", 2024-03-19T11:31:32.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3905999999999994e-05 2024-03-19T11:31:32.629 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.629 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.629 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:32.629 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.145182+0000", 2024-03-19T11:31:32.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.029170207 2024-03-19T11:31:32.630 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:32.630 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:32.630 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:32.630 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.630 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.4.13:82 3.3 e85/84 3:f6b6d21a:::smithi03829856-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 85'103, mlcod=0'0)", 2024-03-19T11:31:32.630 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:04.115589+0000", 2024-03-19T11:31:32.630 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.476054511000001, 2024-03-19T11:31:32.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035569142999999998, 2024-03-19T11:31:32.630 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:32.631 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:32.631 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:32.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.631 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:32.631 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115589+0000", 2024-03-19T11:31:32.631 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:32.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.631 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:32.631 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115589+0000", 2024-03-19T11:31:32.631 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:32.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:32.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115591+0000", 2024-03-19T11:31:32.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2780000000000001e-06 2024-03-19T11:31:32.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:32.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115602+0000", 2024-03-19T11:31:32.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1797000000000001e-05 2024-03-19T11:31:32.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:32.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115613+0000", 2024-03-19T11:31:32.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1025e-05 2024-03-19T11:31:32.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:32.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115619+0000", 2024-03-19T11:31:32.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.976e-06 2024-03-19T11:31:32.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.634 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:32.634 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.145267+0000", 2024-03-19T11:31:32.634 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.02964791 2024-03-19T11:31:32.634 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.634 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.634 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:32.634 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.145304+0000", 2024-03-19T11:31:32.634 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6455000000000002e-05 2024-03-19T11:31:32.634 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.634 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.634 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:32.635 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.151114+0000", 2024-03-19T11:31:32.635 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0058103139999999996 2024-03-19T11:31:32.635 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.635 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:32.635 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.151158+0000", 2024-03-19T11:31:32.635 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4388000000000003e-05 2024-03-19T11:31:32.635 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:32.635 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:32.635 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:32.635 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.636 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.4.13:84 3.3 e85/84 3:ce6951d3:::smithi03829856-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 85'104, mlcod=0'0)", 2024-03-19T11:31:32.636 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:04.115824+0000", 2024-03-19T11:31:32.636 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.475819508000001, 2024-03-19T11:31:32.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035373398, 2024-03-19T11:31:32.636 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:32.636 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:32.636 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:32.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:32.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115824+0000", 2024-03-19T11:31:32.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:32.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.637 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:32.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115824+0000", 2024-03-19T11:31:32.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:32.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.637 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:32.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115825+0000", 2024-03-19T11:31:32.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3399999999999999e-07 2024-03-19T11:31:32.638 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.638 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.638 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:32.638 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115861+0000", 2024-03-19T11:31:32.638 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6303000000000001e-05 2024-03-19T11:31:32.638 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.638 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.638 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:32.638 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115869+0000", 2024-03-19T11:31:32.638 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0350000000000001e-06 2024-03-19T11:31:32.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.639 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:32.639 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115874+0000", 2024-03-19T11:31:32.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1599999999999997e-06 2024-03-19T11:31:32.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.639 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:32.639 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.145686+0000", 2024-03-19T11:31:32.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.029811252 2024-03-19T11:31:32.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:32.640 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.145706+0000", 2024-03-19T11:31:32.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0264000000000001e-05 2024-03-19T11:31:32.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:32.640 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.151161+0000", 2024-03-19T11:31:32.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054555280000000003 2024-03-19T11:31:32.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.641 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:32.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.151198+0000", 2024-03-19T11:31:32.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6322000000000002e-05 2024-03-19T11:31:32.641 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:32.641 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:32.641 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:32.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.641 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.4.13:85 3.3 e85/84 3:ce6951d3:::smithi03829856-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 85'105, mlcod=0'0)", 2024-03-19T11:31:32.641 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:04.115887+0000", 2024-03-19T11:31:32.641 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.475756894, 2024-03-19T11:31:32.642 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035346673000000002, 2024-03-19T11:31:32.642 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:32.642 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:32.642 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:32.642 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.642 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:32.642 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115887+0000", 2024-03-19T11:31:32.642 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:32.642 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.642 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.642 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:32.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115887+0000", 2024-03-19T11:31:32.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:32.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.643 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:32.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115887+0000", 2024-03-19T11:31:32.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2300000000000002e-07 2024-03-19T11:31:32.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.643 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:32.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115900+0000", 2024-03-19T11:31:32.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3044000000000001e-05 2024-03-19T11:31:32.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.644 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:32.644 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115906+0000", 2024-03-19T11:31:32.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5929999999999996e-06 2024-03-19T11:31:32.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.644 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:32.644 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115910+0000", 2024-03-19T11:31:32.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9269999999999998e-06 2024-03-19T11:31:32.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:32.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.145919+0000", 2024-03-19T11:31:32.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030009650999999998 2024-03-19T11:31:32.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:32.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.145948+0000", 2024-03-19T11:31:32.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9047000000000002e-05 2024-03-19T11:31:32.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:32.646 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.151200+0000", 2024-03-19T11:31:32.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052512970000000003 2024-03-19T11:31:32.646 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:32.646 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.151234+0000", 2024-03-19T11:31:32.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3791000000000002e-05 2024-03-19T11:31:32.646 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:32.646 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:32.647 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:32.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.647 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(3.3 85/84 [PushOp(3:fcdd1002:::06a0957e01d2e04d1b09d0c04f111819905dcf8372f2c6bee5014c22d5b575ef:head, version: 34'50, data_included: [0~458752], data_size: 458752, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:fcdd1002:::06a0957e01d2e04d1b09d0c04f111819905dcf8372f2c6bee5014c22d5b575ef:head@34'50, size: 689152, copy_subset: [(0, 689152)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 689152, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:32.647 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:04.115917+0000", 2024-03-19T11:31:32.647 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.475726463000001, 2024-03-19T11:31:32.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033796169000000001, 2024-03-19T11:31:32.647 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:32.647 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:32.647 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:32.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.648 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:32.648 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115917+0000", 2024-03-19T11:31:32.648 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:32.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.648 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:32.648 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115917+0000", 2024-03-19T11:31:32.648 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:32.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:32.649 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115918+0000", 2024-03-19T11:31:32.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8000000000000002e-07 2024-03-19T11:31:32.649 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:32.649 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116163+0000", 2024-03-19T11:31:32.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00024516400000000003 2024-03-19T11:31:32.649 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.650 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:32.650 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116181+0000", 2024-03-19T11:31:32.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8372999999999998e-05 2024-03-19T11:31:32.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.650 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.650 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:32.650 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116196+0000", 2024-03-19T11:31:32.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5041e-05 2024-03-19T11:31:32.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.650 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.650 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:32.650 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.147578+0000", 2024-03-19T11:31:32.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031382284000000003 2024-03-19T11:31:32.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.651 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:32.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.147594+0000", 2024-03-19T11:31:32.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5153e-05 2024-03-19T11:31:32.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.651 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:32.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.149713+0000", 2024-03-19T11:31:32.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021198739999999999 2024-03-19T11:31:32.651 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:32.651 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:32.651 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:32.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.652 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.652 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.4.13:83 3.3 e85/84 3:f6b6d21a:::smithi03829856-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 85'106, mlcod=0'0)", 2024-03-19T11:31:32.652 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:04.116216+0000", 2024-03-19T11:31:32.652 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.475427404000001, 2024-03-19T11:31:32.652 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035049444999999999, 2024-03-19T11:31:32.652 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:32.652 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:32.652 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:32.652 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.652 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:32.652 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116216+0000", 2024-03-19T11:31:32.652 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:32.652 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.652 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.653 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:32.653 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116216+0000", 2024-03-19T11:31:32.653 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:32.653 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.653 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.653 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:32.653 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116217+0000", 2024-03-19T11:31:32.653 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.22e-07 2024-03-19T11:31:32.653 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.653 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.653 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:32.653 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116221+0000", 2024-03-19T11:31:32.653 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5349999999999998e-06 2024-03-19T11:31:32.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.654 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:32.654 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116237+0000", 2024-03-19T11:31:32.654 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5532e-05 2024-03-19T11:31:32.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.654 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:32.654 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116245+0000", 2024-03-19T11:31:32.654 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0450000000000001e-06 2024-03-19T11:31:32.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.654 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:32.655 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.146739+0000", 2024-03-19T11:31:32.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030494351999999999 2024-03-19T11:31:32.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:32.655 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.146776+0000", 2024-03-19T11:31:32.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6894999999999998e-05 2024-03-19T11:31:32.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:32.655 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.151237+0000", 2024-03-19T11:31:32.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044605160000000003 2024-03-19T11:31:32.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.656 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:32.656 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.151266+0000", 2024-03-19T11:31:32.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9247999999999998e-05 2024-03-19T11:31:32.656 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:32.656 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:32.656 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:32.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.656 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3194 4.d 4:b65e6658:::smithi03829856-9:head [read 0~3714347,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108)", 2024-03-19T11:31:32.656 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.202652+0000", 2024-03-19T11:31:32.656 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.3889918149999998, 2024-03-19T11:31:32.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031511255000000002, 2024-03-19T11:31:32.656 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:32.656 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:32.657 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:32.657 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:32.657 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:32.657 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3194 2024-03-19T11:31:32.657 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.657 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:32.657 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.657 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:32.657 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202652+0000", 2024-03-19T11:31:32.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:32.657 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.657 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.657 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:32.657 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202652+0000", 2024-03-19T11:31:32.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:32.658 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.658 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:32.658 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202653+0000", 2024-03-19T11:31:32.658 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3960000000000001e-06 2024-03-19T11:31:32.658 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.658 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:32.658 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202661+0000", 2024-03-19T11:31:32.658 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0830000000000004e-06 2024-03-19T11:31:32.658 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.658 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:32.658 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202679+0000", 2024-03-19T11:31:32.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7972e-05 2024-03-19T11:31:32.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:32.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202693+0000", 2024-03-19T11:31:32.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4031e-05 2024-03-19T11:31:32.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:32.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.203299+0000", 2024-03-19T11:31:32.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00060564400000000002 2024-03-19T11:31:32.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:32.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.203528+0000", 2024-03-19T11:31:32.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000229049 2024-03-19T11:31:32.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.660 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:32.660 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.234163+0000", 2024-03-19T11:31:32.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030635079999999999 2024-03-19T11:31:32.660 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:32.660 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:32.660 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:32.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.660 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3195 4.d 4:b65e6658:::smithi03829856-9:head [read 0~3714347,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108)", 2024-03-19T11:31:32.660 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.202714+0000", 2024-03-19T11:31:32.660 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.3889297840000001, 2024-03-19T11:31:32.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.028551382, 2024-03-19T11:31:32.661 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:32.661 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:32.661 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:32.661 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:32.661 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:32.661 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3195 2024-03-19T11:31:32.661 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.661 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:32.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.661 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:32.661 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202714+0000", 2024-03-19T11:31:32.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:32.661 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.661 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:32.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202714+0000", 2024-03-19T11:31:32.662 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:32.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.662 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:32.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202715+0000", 2024-03-19T11:31:32.662 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.203e-06 2024-03-19T11:31:32.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.662 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:32.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202722+0000", 2024-03-19T11:31:32.662 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1289999999999998e-06 2024-03-19T11:31:32.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.663 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:32.663 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202739+0000", 2024-03-19T11:31:32.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6929e-05 2024-03-19T11:31:32.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.663 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:32.663 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202754+0000", 2024-03-19T11:31:32.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4642e-05 2024-03-19T11:31:32.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.663 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:32.663 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.227218+0000", 2024-03-19T11:31:32.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.024464587999999999 2024-03-19T11:31:32.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.664 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:32.664 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.227471+0000", 2024-03-19T11:31:32.664 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000252747 2024-03-19T11:31:32.664 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.664 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:32.664 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.231265+0000", 2024-03-19T11:31:32.664 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037941440000000002 2024-03-19T11:31:32.664 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:32.664 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:32.664 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:32.664 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.665 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3196 4.d 4:b65e6658:::smithi03829856-9:head [sparse-read 0~2135368] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108)", 2024-03-19T11:31:32.665 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.202774+0000", 2024-03-19T11:31:32.665 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.3888700429999998, 2024-03-19T11:31:32.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033044519000000001, 2024-03-19T11:31:32.665 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:32.665 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:32.665 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:32.665 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:32.665 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:32.665 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3196 2024-03-19T11:31:32.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.665 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:32.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.666 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:32.666 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202774+0000", 2024-03-19T11:31:32.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:32.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.666 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:32.666 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202774+0000", 2024-03-19T11:31:32.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:32.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:32.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202775+0000", 2024-03-19T11:31:32.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.423e-06 2024-03-19T11:31:32.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:32.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202782+0000", 2024-03-19T11:31:32.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3660000000000003e-06 2024-03-19T11:31:32.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.668 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:32.668 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202801+0000", 2024-03-19T11:31:32.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8182e-05 2024-03-19T11:31:32.668 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.668 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.668 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:32.668 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202815+0000", 2024-03-19T11:31:32.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4042e-05 2024-03-19T11:31:32.668 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.668 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.669 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:32.669 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.234294+0000", 2024-03-19T11:31:32.669 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031479493999999997 2024-03-19T11:31:32.669 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.669 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.669 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:32.669 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.234670+0000", 2024-03-19T11:31:32.669 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000375595 2024-03-19T11:31:32.669 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.669 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.670 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:32.670 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.235818+0000", 2024-03-19T11:31:32.670 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011484170000000001 2024-03-19T11:31:32.670 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:32.670 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:32.670 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:32.670 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.670 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.670 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(osd.0.13:71 3.3 3:e22267ea:::smithi03829856-38 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e108)", 2024-03-19T11:31:32.670 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.209268+0000", 2024-03-19T11:31:32.671 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.3823757920000004, 2024-03-19T11:31:32.671 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.038534928000000003, 2024-03-19T11:31:32.671 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:32.671 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:32.671 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:32.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.671 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:32.671 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.209268+0000", 2024-03-19T11:31:32.671 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:32.671 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:32.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.209268+0000", 2024-03-19T11:31:32.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:32.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:32.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.209272+0000", 2024-03-19T11:31:32.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0500000000000002e-06 2024-03-19T11:31:32.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:32.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.209291+0000", 2024-03-19T11:31:32.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8734999999999999e-05 2024-03-19T11:31:32.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.673 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:32.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.209329+0000", 2024-03-19T11:31:32.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8223000000000003e-05 2024-03-19T11:31:32.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.673 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:32.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.209354+0000", 2024-03-19T11:31:32.674 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5114e-05 2024-03-19T11:31:32.674 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.674 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.674 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:32.674 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.209573+0000", 2024-03-19T11:31:32.674 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021842000000000001 2024-03-19T11:31:32.674 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.674 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.674 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:32.674 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.209814+0000", 2024-03-19T11:31:32.674 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00024179400000000001 2024-03-19T11:31:32.675 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.675 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.675 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:32.675 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.210284+0000", 2024-03-19T11:31:32.675 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00046991699999999998 2024-03-19T11:31:32.675 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.675 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.675 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:32.675 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.213128+0000", 2024-03-19T11:31:32.675 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002843609 2024-03-19T11:31:32.675 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.675 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.676 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:32.676 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.247417+0000", 2024-03-19T11:31:32.676 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.034289312000000002 2024-03-19T11:31:32.676 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.676 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.676 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:32.676 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.247692+0000", 2024-03-19T11:31:32.676 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027449000000000001 2024-03-19T11:31:32.676 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.676 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.676 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:32.677 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.247803+0000", 2024-03-19T11:31:32.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111264 2024-03-19T11:31:32.677 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:32.677 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:32.677 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:32.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.677 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3198 3.3 3:f117d76c:::3a52856062567f05d8c7df91dad2c51911138f6216e1ec709153bdff536e4abe:head [write 0~927232] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-19T11:31:32.677 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.213434+0000", 2024-03-19T11:31:32.677 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 5.3782101009999996, 2024-03-19T11:31:32.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.039874882, 2024-03-19T11:31:32.678 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:32.678 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:32.678 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:32.678 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:32.678 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:32.678 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3198 2024-03-19T11:31:32.678 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.678 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:32.678 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.678 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:32.678 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.213434+0000", 2024-03-19T11:31:32.678 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:32.679 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.679 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:32.679 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.213434+0000", 2024-03-19T11:31:32.679 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:32.679 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.679 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:32.679 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.213436+0000", 2024-03-19T11:31:32.679 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4430000000000002e-06 2024-03-19T11:31:32.679 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.680 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.680 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:32.680 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.219501+0000", 2024-03-19T11:31:32.680 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0060649240000000002 2024-03-19T11:31:32.680 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.680 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.680 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:32.680 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.219531+0000", 2024-03-19T11:31:32.680 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0470000000000001e-05 2024-03-19T11:31:32.680 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.680 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.681 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:32.681 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.219552+0000", 2024-03-19T11:31:32.681 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1012000000000001e-05 2024-03-19T11:31:32.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.681 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:32.682 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.231849+0000", 2024-03-19T11:31:32.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012296105999999999 2024-03-19T11:31:32.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.682 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.682 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:32.682 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.232349+0000", 2024-03-19T11:31:32.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00050035100000000001 2024-03-19T11:31:32.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.682 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.682 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:32.682 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.233102+0000", 2024-03-19T11:31:32.683 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00075262300000000001 2024-03-19T11:31:32.683 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.683 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.683 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:32.683 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.249722+0000", 2024-03-19T11:31:32.683 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016620901 2024-03-19T11:31:32.683 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.683 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.683 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:32.683 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.253185+0000", 2024-03-19T11:31:32.683 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034625340000000002 2024-03-19T11:31:32.684 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.684 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.684 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:32.684 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.253252+0000", 2024-03-19T11:31:32.684 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6538000000000006e-05 2024-03-19T11:31:32.684 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.684 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.684 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:32.684 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.253309+0000", 2024-03-19T11:31:32.684 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.698e-05 2024-03-19T11:31:32.685 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:32.685 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:32.685 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:32.685 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.685 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.685 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3323 4.18 4:1d58db11:::smithi03829856-18:head [tier-promote] snapc 22=[22,21,1f,1e,1d,1c,1b,1a,18,17] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-19T11:31:32.685 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:31.226946+0000", 2024-03-19T11:31:32.685 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.364697805, 2024-03-19T11:31:32.685 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.037925183000000001, 2024-03-19T11:31:32.685 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:32.685 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:32.685 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:32.686 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:32.686 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:32.686 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3323 2024-03-19T11:31:32.686 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.686 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:32.686 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.686 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:32.686 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.226946+0000", 2024-03-19T11:31:32.686 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:32.686 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.687 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.687 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:32.687 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.226946+0000", 2024-03-19T11:31:32.687 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:32.687 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.687 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.687 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:32.687 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.226947+0000", 2024-03-19T11:31:32.687 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4670000000000001e-06 2024-03-19T11:31:32.687 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.687 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.687 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:32.687 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.226955+0000", 2024-03-19T11:31:32.688 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9130000000000003e-06 2024-03-19T11:31:32.688 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.688 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.688 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:32.688 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.226978+0000", 2024-03-19T11:31:32.688 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2229e-05 2024-03-19T11:31:32.688 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.688 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.688 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:32.688 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.226993+0000", 2024-03-19T11:31:32.688 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5420000000000001e-05 2024-03-19T11:31:32.688 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.688 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.688 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:32.688 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.227146+0000", 2024-03-19T11:31:32.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015327200000000001 2024-03-19T11:31:32.689 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.689 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.689 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:32.689 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.227406+0000", 2024-03-19T11:31:32.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025941799999999998 2024-03-19T11:31:32.689 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:32.689 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:32.689 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:32.689 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.264871+0000", 2024-03-19T11:31:32.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.037465463999999997 2024-03-19T11:31:32.689 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:32.689 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:32.689 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:32.689 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:32.690 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:32.690 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:32.691 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-19T11:31:32.692 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:32.696+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:32.693 INFO:tasks.rados.rados.0.smithi038.stdout:1512: expect (ObjNum 228 snap 26 seq_num 228) 2024-03-19T11:31:32.792 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:32.802+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:32.864 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:33.328 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:31:33.329 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:31:33.329 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:33.329 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-19T11:31:33.330 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-03-19T11:31:33.335 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-19T11:31:33.336 INFO:tasks.rados.rados.0.smithi038.stdout:1510: done (5 left) 2024-03-19T11:31:33.336 INFO:tasks.rados.rados.0.smithi038.stdout:1511: done (4 left) 2024-03-19T11:31:33.337 INFO:tasks.rados.rados.0.smithi038.stdout:1512: done (3 left) 2024-03-19T11:31:33.337 INFO:tasks.rados.rados.0.smithi038.stdout:1513: done (2 left) 2024-03-19T11:31:33.337 INFO:tasks.rados.rados.0.smithi038.stdout:1514: done (1 left) 2024-03-19T11:31:33.337 INFO:tasks.rados.rados.0.smithi038.stdout:1515: write oid 24 current snap is 34 2024-03-19T11:31:33.337 INFO:tasks.rados.rados.0.smithi038.stdout:1515: seq_num 265 ranges {453855=696987,1890787=548083} 2024-03-19T11:31:33.337 INFO:tasks.rados.rados.0.smithi038.stdout:1515: writing smithi03829856-24 from 453855 to 1150842 tid 1 2024-03-19T11:31:33.337 INFO:tasks.rados.rados.0.smithi038.stdout:1515: writing smithi03829856-24 from 1890787 to 2438870 tid 2 2024-03-19T11:31:33.337 INFO:tasks.rados.rados.0.smithi038.stdout:1516: set_chunk oid 9 target oid 2024-03-19T11:31:33.337 INFO:tasks.rados.rados.0.smithi038.stdout:1516: set_chunk oid 9 offset: 43008 length: 1874944 target oid 33a8d345f1c2729481f508082046c4f5945e9985a505cdfadb58a39103d098fa offset: 0 snap 31 2024-03-19T11:31:33.337 INFO:tasks.rados.rados.0.smithi038.stdout:1517: write oid 36 current snap is 34 2024-03-19T11:31:33.337 INFO:tasks.rados.rados.0.smithi038.stdout:1517: seq_num 266 ranges {470662=772236,1832916=408094} 2024-03-19T11:31:33.337 INFO:tasks.rados.rados.0.smithi038.stdout:1517: writing smithi03829856-36 from 470662 to 1242898 tid 1 2024-03-19T11:31:33.337 INFO:tasks.rados.rados.0.smithi038.stdout:1517: writing smithi03829856-36 from 1832916 to 2241010 tid 2 2024-03-19T11:31:33.337 INFO:tasks.rados.rados.0.smithi038.stdout:1518: read oid 29 snap 31 2024-03-19T11:31:33.338 INFO:tasks.rados.rados.0.smithi038.stdout:1518: expect (ObjNum 219 snap 23 seq_num 219) 2024-03-19T11:31:33.338 INFO:tasks.rados.rados.0.smithi038.stdout:1519: read oid 33 snap -1 2024-03-19T11:31:33.338 INFO:tasks.rados.rados.0.smithi038.stdout:1519: expect (ObjNum 248 snap 30 seq_num 248) 2024-03-19T11:31:33.338 INFO:tasks.rados.rados.0.smithi038.stdout:1520: write (excl) oid 26 current snap is 34 2024-03-19T11:31:33.338 INFO:tasks.rados.rados.0.smithi038.stdout:1515: finishing write tid 1 to smithi03829856-24 2024-03-19T11:31:33.338 INFO:tasks.rados.rados.0.smithi038.stdout:1520: seq_num 267 ranges {487469=563838,1607751=551752} 2024-03-19T11:31:33.338 INFO:tasks.rados.rados.0.smithi038.stdout:1520: writing smithi03829856-26 from 487469 to 1051307 tid 1 2024-03-19T11:31:33.338 INFO:tasks.rados.rados.0.smithi038.stdout:1520: writing smithi03829856-26 from 1607751 to 2159503 tid 2 2024-03-19T11:31:33.338 INFO:tasks.rados.rados.0.smithi038.stdout:1521: read oid 27 snap -1 2024-03-19T11:31:33.338 INFO:tasks.rados.rados.0.smithi038.stdout:1521: expect deleted 2024-03-19T11:31:33.338 INFO:tasks.rados.rados.0.smithi038.stdout:1522: read oid 44 snap -1 2024-03-19T11:31:33.338 INFO:tasks.rados.rados.0.smithi038.stdout:1522: expect (ObjNum 252 snap 32 seq_num 252) 2024-03-19T11:31:33.338 INFO:tasks.rados.rados.0.smithi038.stdout:1523: write (excl) oid 31 current snap is 34 2024-03-19T11:31:33.338 INFO:tasks.rados.rados.0.smithi038.stdout:1515: finishing write tid 2 to smithi03829856-24 2024-03-19T11:31:33.338 INFO:tasks.rados.rados.0.smithi038.stdout:1515: finishing write tid 3 to smithi03829856-24 2024-03-19T11:31:33.339 INFO:tasks.rados.rados.0.smithi038.stdout:1515: finishing write tid 4 to smithi03829856-24 2024-03-19T11:31:33.339 INFO:tasks.rados.rados.0.smithi038.stdout:1515: oid 24 updating version 0 to 153 2024-03-19T11:31:33.339 INFO:tasks.rados.rados.0.smithi038.stdout:1515: oid 24 updating version 153 to 154 2024-03-19T11:31:33.339 INFO:tasks.rados.rados.0.smithi038.stdout:1515: oid 24 updating version 154 to 155 2024-03-19T11:31:33.339 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 24 v 155 (ObjNum 265 snap 34 seq_num 265) dirty exists 2024-03-19T11:31:33.339 INFO:tasks.rados.rados.0.smithi038.stdout:1515: left oid 24 (ObjNum 265 snap 34 seq_num 265) 2024-03-19T11:31:33.339 INFO:tasks.rados.rados.0.smithi038.stdout:1516: finishing set_chunk to oid 9 2024-03-19T11:31:33.339 INFO:tasks.rados.rados.0.smithi038.stdout:1523: seq_num 268 ranges {504276=639087,1666233=695410} 2024-03-19T11:31:33.339 INFO:tasks.rados.rados.0.smithi038.stdout:1523: writing smithi03829856-31 from 504276 to 1143363 tid 1 2024-03-19T11:31:33.340 INFO:tasks.rados.rados.0.smithi038.stdout:1523: writing smithi03829856-31 from 1666233 to 2361643 tid 2 2024-03-19T11:31:33.340 INFO:tasks.rados.rados.0.smithi038.stdout:1515: done (9 left) 2024-03-19T11:31:33.340 INFO:tasks.rados.rados.0.smithi038.stdout:1516: done (8 left) 2024-03-19T11:31:33.340 INFO:tasks.rados.rados.0.smithi038.stdout:1524: read oid 50 snap -1 2024-03-19T11:31:33.340 INFO:tasks.rados.rados.0.smithi038.stdout:1524: expect (ObjNum 260 snap 33 seq_num 260) 2024-03-19T11:31:33.340 INFO:tasks.rados.rados.0.smithi038.stdout:1525: snap_remove snap 22 2024-03-19T11:31:33.340 INFO:teuthology.orchestra.run.smithi038.stderr:marked out osd.3. 2024-03-19T11:31:33.340 INFO:tasks.rados.rados.0.smithi038.stdout:1517: finishing write tid 1 to smithi03829856-36 2024-03-19T11:31:33.341 INFO:tasks.rados.rados.0.smithi038.stdout:1517: finishing write tid 2 to smithi03829856-36 2024-03-19T11:31:33.341 INFO:tasks.rados.rados.0.smithi038.stdout:1517: finishing write tid 3 to smithi03829856-36 2024-03-19T11:31:33.341 INFO:tasks.rados.rados.0.smithi038.stdout:1517: finishing write tid 4 to smithi03829856-36 2024-03-19T11:31:33.341 INFO:tasks.rados.rados.0.smithi038.stdout:1517: oid 36 updating version 0 to 127 2024-03-19T11:31:33.341 INFO:tasks.rados.rados.0.smithi038.stdout:1517: oid 36 version 127 is already newer than 125 2024-03-19T11:31:33.341 INFO:tasks.rados.rados.0.smithi038.stdout:1517: oid 36 version 127 is already newer than 126 2024-03-19T11:31:33.341 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 36 v 127 (ObjNum 266 snap 34 seq_num 266) dirty exists 2024-03-19T11:31:33.341 INFO:tasks.rados.rados.0.smithi038.stdout:1517: left oid 36 (ObjNum 266 snap 34 seq_num 266) 2024-03-19T11:31:33.341 INFO:tasks.rados.rados.0.smithi038.stdout:1518: expect (ObjNum 219 snap 23 seq_num 219) 2024-03-19T11:31:33.342 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:33.336+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:33.374 INFO:tasks.rados.rados.0.smithi038.stdout:1519: expect (ObjNum 248 snap 30 seq_num 248) 2024-03-19T11:31:33.431 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:33.436+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:33.499 INFO:tasks.rados.rados.0.smithi038.stdout:1520: finishing write tid 1 to smithi03829856-26 2024-03-19T11:31:33.499 INFO:tasks.rados.rados.0.smithi038.stdout:1520: finishing write tid 2 to smithi03829856-26 2024-03-19T11:31:33.499 INFO:tasks.rados.rados.0.smithi038.stdout:1520: finishing write tid 3 to smithi03829856-26 2024-03-19T11:31:33.499 INFO:tasks.rados.rados.0.smithi038.stdout:1517: done (9 left) 2024-03-19T11:31:33.499 INFO:tasks.rados.rados.0.smithi038.stdout:1518: done (8 left) 2024-03-19T11:31:33.499 INFO:tasks.rados.rados.0.smithi038.stdout:1519: done (7 left) 2024-03-19T11:31:33.499 INFO:tasks.rados.rados.0.smithi038.stdout:1521: done (6 left) 2024-03-19T11:31:33.499 INFO:tasks.rados.rados.0.smithi038.stdout:1525: done (5 left) 2024-03-19T11:31:33.499 INFO:tasks.rados.rados.0.smithi038.stdout:1526: write (excl) oid 9 current snap is 34 2024-03-19T11:31:33.499 INFO:tasks.rados.rados.0.smithi038.stdout:1526: seq_num 269 ranges {521083=714336,1724715=555421} 2024-03-19T11:31:33.505 INFO:tasks.rados.rados.0.smithi038.stdout:1526: writing smithi03829856-9 from 521083 to 1235419 tid 1 2024-03-19T11:31:33.509 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:33.509 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:31:33.510 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:31:33.510 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:31:33.510 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:33.512 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:31:33.512 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:31:33.513 INFO:tasks.rados.rados.0.smithi038.stdout:1526: writing smithi03829856-9 from 1724715 to 2280136 tid 2 2024-03-19T11:31:33.513 INFO:tasks.rados.rados.0.smithi038.stdout:1527: tier_promote oid 10 2024-03-19T11:31:33.513 INFO:tasks.rados.rados.0.smithi038.stdout:1520: finishing write tid 4 to smithi03829856-26 2024-03-19T11:31:33.513 INFO:tasks.rados.rados.0.smithi038.stdout:1520: oid 26 updating version 0 to 76 2024-03-19T11:31:33.513 INFO:tasks.rados.rados.0.smithi038.stdout:1520: oid 26 version 76 is already newer than 74 2024-03-19T11:31:33.513 INFO:tasks.rados.rados.0.smithi038.stdout:1520: oid 26 version 76 is already newer than 75 2024-03-19T11:31:33.513 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 26 v 76 (ObjNum 267 snap 34 seq_num 267) dirty exists 2024-03-19T11:31:33.513 INFO:tasks.rados.rados.0.smithi038.stdout:1520: left oid 26 (ObjNum 267 snap 34 seq_num 267) 2024-03-19T11:31:33.513 INFO:tasks.rados.rados.0.smithi038.stdout:1522: expect (ObjNum 252 snap 32 seq_num 252) 2024-03-19T11:31:33.525 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-19T11:31:33.530 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:31:33.530 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:31:33.546 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:31:33.546 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:31:33.562 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:31:33.563 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:31:33.572 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:33.582+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:33.580 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:31:33.580 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:31:33.600 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:31:33.600 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:31:33.620 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:31:33.621 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:31:33.630 INFO:tasks.rados.rados.0.smithi038.stdout:1523: finishing write tid 1 to smithi03829856-31 2024-03-19T11:31:33.630 INFO:tasks.rados.rados.0.smithi038.stdout:1523: finishing write tid 2 to smithi03829856-31 2024-03-19T11:31:33.630 INFO:tasks.rados.rados.0.smithi038.stdout:1524: expect (ObjNum 260 snap 33 seq_num 260) 2024-03-19T11:31:33.632 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:33.640+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:33.639 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:31:33.639 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:31:33.647 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:33.647 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:33.661 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-19T11:31:33.731 INFO:tasks.rados.rados.0.smithi038.stdout:1480: expect (ObjNum 12 snap 0 seq_num 12) 2024-03-19T11:31:33.752 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:33.762+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:33.799 INFO:tasks.rados.rados.0.smithi038.stdout:1526: finishing write tid 1 to smithi03829856-9 2024-03-19T11:31:33.800 INFO:tasks.rados.rados.0.smithi038.stdout:1526: finishing write tid 2 to smithi03829856-9 2024-03-19T11:31:33.800 INFO:tasks.rados.rados.0.smithi038.stdout:1526: finishing write tid 3 to smithi03829856-9 2024-03-19T11:31:33.800 INFO:tasks.rados.rados.0.smithi038.stdout:1526: finishing write tid 4 to smithi03829856-9 2024-03-19T11:31:33.800 INFO:tasks.rados.rados.0.smithi038.stdout:1526: oid 9 updating version 0 to 111 2024-03-19T11:31:33.800 INFO:tasks.rados.rados.0.smithi038.stdout:1526: oid 9 version 111 is already newer than 110 2024-03-19T11:31:33.800 INFO:tasks.rados.rados.0.smithi038.stdout:1526: oid 9 version 111 is already newer than 109 2024-03-19T11:31:33.800 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 9 v 111 (ObjNum 269 snap 34 seq_num 269) dirty exists 2024-03-19T11:31:33.800 INFO:tasks.rados.rados.0.smithi038.stdout:1526: left oid 9 (ObjNum 269 snap 34 seq_num 269) 2024-03-19T11:31:33.800 INFO:tasks.rados.rados.0.smithi038.stdout:1480: done (6 left) 2024-03-19T11:31:33.800 INFO:tasks.rados.rados.0.smithi038.stdout:1520: done (5 left) 2024-03-19T11:31:33.801 INFO:tasks.rados.rados.0.smithi038.stdout:1522: done (4 left) 2024-03-19T11:31:33.801 INFO:tasks.rados.rados.0.smithi038.stdout:1524: done (3 left) 2024-03-19T11:31:33.801 INFO:tasks.rados.rados.0.smithi038.stdout:1526: done (2 left) 2024-03-19T11:31:33.801 INFO:tasks.rados.rados.0.smithi038.stdout:1528: read oid 37 snap -1 2024-03-19T11:31:33.801 INFO:tasks.rados.rados.0.smithi038.stdout:1528: expect (ObjNum 236 snap 28 seq_num 236) 2024-03-19T11:31:33.801 INFO:tasks.rados.rados.0.smithi038.stdout:1529: read oid 28 snap -1 2024-03-19T11:31:33.801 INFO:tasks.rados.rados.0.smithi038.stdout:1529: expect (ObjNum 87 snap 8 seq_num 87) 2024-03-19T11:31:33.801 INFO:tasks.rados.rados.0.smithi038.stdout:1530: read oid 35 snap -1 2024-03-19T11:31:33.801 INFO:tasks.rados.rados.0.smithi038.stdout:1530: expect deleted 2024-03-19T11:31:33.801 INFO:tasks.rados.rados.0.smithi038.stdout:1531: write oid 4 current snap is 34 2024-03-19T11:31:33.801 INFO:tasks.rados.rados.0.smithi038.stdout:1531: seq_num 270 ranges {537890=789585,1783197=699079} 2024-03-19T11:31:33.820 INFO:tasks.rados.rados.0.smithi038.stdout:1531: writing smithi03829856-4 from 537890 to 1327475 tid 1 2024-03-19T11:31:33.825 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:33.826 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:33.834 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:33.842+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:33.840 INFO:tasks.rados.rados.0.smithi038.stdout:1531: writing smithi03829856-4 from 1783197 to 2482276 tid 2 2024-03-19T11:31:33.840 INFO:tasks.rados.rados.0.smithi038.stdout:1532: read oid 14 snap -1 2024-03-19T11:31:33.840 INFO:tasks.rados.rados.0.smithi038.stdout:1532: expect (ObjNum 241 snap 30 seq_num 241) 2024-03-19T11:31:33.840 INFO:tasks.rados.rados.0.smithi038.stdout:1533: read oid 15 snap -1 2024-03-19T11:31:33.841 INFO:tasks.rados.rados.0.smithi038.stdout:1533: expect (ObjNum 209 snap 22 seq_num 209) 2024-03-19T11:31:33.841 INFO:tasks.rados.rados.0.smithi038.stdout:1534: read oid 7 snap -1 2024-03-19T11:31:33.841 INFO:tasks.rados.rados.0.smithi038.stdout:1534: expect (ObjNum 210 snap 22 seq_num 210) 2024-03-19T11:31:33.841 INFO:tasks.rados.rados.0.smithi038.stdout:1535: read oid 46 snap -1 2024-03-19T11:31:33.841 INFO:tasks.rados.rados.0.smithi038.stdout:1535: expect (ObjNum 143 snap 15 seq_num 143) 2024-03-19T11:31:33.841 INFO:tasks.rados.rados.0.smithi038.stdout:1536: set_chunk oid 5 target oid 2024-03-19T11:31:33.842 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-19T11:31:33.935 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:33.942+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:34.008 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:34.008 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:34.022 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-19T11:31:34.036 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:34.044+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:34.053 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:34.060+0000 7f6a7d9d6640 -1 osd.1 113 handle_osd_map injecting map crc failure 2024-03-19T11:31:34.058 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-03-19T11:31:34.137 INFO:tasks.rados.rados.0.smithi038.stdout:1536: set_chunk oid 5 offset: 554496 length: 739328 target oid 7184943f1222654f40d471efce70e2890c369fe8dcdbce03fbbe7b96c6f4efe8 offset: 0 snap -1 2024-03-19T11:31:34.137 INFO:tasks.rados.rados.0.smithi038.stdout:1537: write (excl) oid 3 current snap is 34 2024-03-19T11:31:34.137 INFO:tasks.rados.rados.0.smithi038.stdout:1537: seq_num 271 ranges {554697=464834,1725326=559090} 2024-03-19T11:31:34.137 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:34.144+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:34.144 INFO:tasks.rados.rados.0.smithi038.stdout:1537: writing smithi03829856-3 from 554697 to 1019531 tid 1 2024-03-19T11:31:34.156 INFO:tasks.rados.rados.0.smithi038.stdout:1537: writing smithi03829856-3 from 1725326 to 2284416 tid 2 2024-03-19T11:31:34.156 INFO:tasks.rados.rados.0.smithi038.stdout:1538: tier_flush oid 25 2024-03-19T11:31:34.156 INFO:tasks.rados.rados.0.smithi038.stdout:1528: expect (ObjNum 236 snap 28 seq_num 236) 2024-03-19T11:31:34.238 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:34.238 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:34.238 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:34.244+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:34.254 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-19T11:31:34.306 INFO:tasks.rados.rados.0.smithi038.stdout:1531: finishing write tid 1 to smithi03829856-4 2024-03-19T11:31:34.306 INFO:tasks.rados.rados.0.smithi038.stdout:1532: expect (ObjNum 241 snap 30 seq_num 241) 2024-03-19T11:31:34.338 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:34.344+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:34.368 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:34.369 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:34.383 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-19T11:31:34.411 INFO:tasks.rados.rados.0.smithi038.stdout:1533: expect (ObjNum 209 snap 22 seq_num 209) 2024-03-19T11:31:34.439 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:34.444+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:34.504 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:34.505 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:34.519 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-19T11:31:34.564 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:34.574+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:34.618 INFO:tasks.rados.rados.0.smithi038.stdout:1531: finishing write tid 2 to smithi03829856-4 2024-03-19T11:31:34.619 INFO:tasks.rados.rados.0.smithi038.stdout:1531: finishing write tid 3 to smithi03829856-4 2024-03-19T11:31:34.619 INFO:tasks.rados.rados.0.smithi038.stdout:1535: expect (ObjNum 143 snap 15 seq_num 143) 2024-03-19T11:31:34.641 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:34.648+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:34.642 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:34.642 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:34.656 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-19T11:31:34.697 INFO:tasks.rados.rados.0.smithi038.stdout:1531: finishing write tid 4 to smithi03829856-4 2024-03-19T11:31:34.697 INFO:tasks.rados.rados.0.smithi038.stdout:1531: oid 4 updating version 0 to 165 2024-03-19T11:31:34.697 INFO:tasks.rados.rados.0.smithi038.stdout:1531: oid 4 version 165 is already newer than 164 2024-03-19T11:31:34.697 INFO:tasks.rados.rados.0.smithi038.stdout:1531: oid 4 version 165 is already newer than 163 2024-03-19T11:31:34.697 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 4 v 165 (ObjNum 270 snap 34 seq_num 270) dirty exists 2024-03-19T11:31:34.697 INFO:tasks.rados.rados.0.smithi038.stdout:1531: left oid 4 (ObjNum 270 snap 34 seq_num 270) 2024-03-19T11:31:34.698 INFO:tasks.rados.rados.0.smithi038.stdout:1534: expect (ObjNum 210 snap 22 seq_num 210) 2024-03-19T11:31:34.742 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:34.748+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:34.776 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:34.776 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:34.790 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-19T11:31:34.844 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:34.854+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:34.898 INFO:tasks.rados.rados.0.smithi038.stdout:1523: finishing write tid 3 to smithi03829856-31 2024-03-19T11:31:34.899 INFO:tasks.rados.rados.0.smithi038.stdout:1523: finishing write tid 4 to smithi03829856-31 2024-03-19T11:31:34.899 INFO:tasks.rados.rados.0.smithi038.stdout:1523: oid 31 updating version 0 to 157 2024-03-19T11:31:34.899 INFO:tasks.rados.rados.0.smithi038.stdout:1523: oid 31 updating version 157 to 159 2024-03-19T11:31:34.899 INFO:tasks.rados.rados.0.smithi038.stdout:1523: oid 31 version 159 is already newer than 158 2024-03-19T11:31:34.899 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 31 v 159 (ObjNum 268 snap 34 seq_num 268) dirty exists 2024-03-19T11:31:34.899 INFO:tasks.rados.rados.0.smithi038.stdout:1523: left oid 31 (ObjNum 268 snap 34 seq_num 268) 2024-03-19T11:31:34.899 INFO:tasks.rados.rados.0.smithi038.stdout:1529: expect (ObjNum 87 snap 8 seq_num 87) 2024-03-19T11:31:34.945 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:34.954+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:34.946 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:34.946 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:34.960 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-19T11:31:35.004 INFO:tasks.rados.rados.0.smithi038.stdout:1527: got (0) Success 2024-03-19T11:31:35.004 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 10 v 149 (ObjNum 179 snap 19 seq_num 179) dirty exists 2024-03-19T11:31:35.005 INFO:tasks.rados.rados.0.smithi038.stdout:1537: finishing write tid 1 to smithi03829856-3 2024-03-19T11:31:35.005 INFO:tasks.rados.rados.0.smithi038.stdout:1538: tier_flush oid 25 snap -1 2024-03-19T11:31:35.005 INFO:tasks.rados.rados.0.smithi038.stdout:1537: finishing write tid 2 to smithi03829856-3 2024-03-19T11:31:35.005 INFO:tasks.rados.rados.0.smithi038.stdout:1537: finishing write tid 3 to smithi03829856-3 2024-03-19T11:31:35.005 INFO:tasks.rados.rados.0.smithi038.stdout:1537: finishing write tid 4 to smithi03829856-3 2024-03-19T11:31:35.005 INFO:tasks.rados.rados.0.smithi038.stdout:1537: oid 3 updating version 0 to 112 2024-03-19T11:31:35.005 INFO:tasks.rados.rados.0.smithi038.stdout:1537: oid 3 updating version 112 to 114 2024-03-19T11:31:35.005 INFO:tasks.rados.rados.0.smithi038.stdout:1537: oid 3 version 114 is already newer than 113 2024-03-19T11:31:35.005 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 3 v 114 (ObjNum 271 snap 34 seq_num 271) dirty exists 2024-03-19T11:31:35.005 INFO:tasks.rados.rados.0.smithi038.stdout:1537: left oid 3 (ObjNum 271 snap 34 seq_num 271) 2024-03-19T11:31:35.005 INFO:tasks.rados.rados.0.smithi038.stdout:1536: finishing set_chunk to oid 5 2024-03-19T11:31:35.005 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 5 v 175 (ObjNum 263 snap 34 seq_num 263) dirty exists 2024-03-19T11:31:35.005 INFO:tasks.rados.rados.0.smithi038.stdout:1523: done (12 left) 2024-03-19T11:31:35.005 INFO:tasks.rados.rados.0.smithi038.stdout:1527: done (11 left) 2024-03-19T11:31:35.006 INFO:tasks.rados.rados.0.smithi038.stdout:1528: done (10 left) 2024-03-19T11:31:35.006 INFO:tasks.rados.rados.0.smithi038.stdout:1529: done (9 left) 2024-03-19T11:31:35.006 INFO:tasks.rados.rados.0.smithi038.stdout:1530: done (8 left) 2024-03-19T11:31:35.006 INFO:tasks.rados.rados.0.smithi038.stdout:1531: done (7 left) 2024-03-19T11:31:35.006 INFO:tasks.rados.rados.0.smithi038.stdout:1532: done (6 left) 2024-03-19T11:31:35.006 INFO:tasks.rados.rados.0.smithi038.stdout:1533: done (5 left) 2024-03-19T11:31:35.006 INFO:tasks.rados.rados.0.smithi038.stdout:1534: done (4 left) 2024-03-19T11:31:35.006 INFO:tasks.rados.rados.0.smithi038.stdout:1535: done (3 left) 2024-03-19T11:31:35.006 INFO:tasks.rados.rados.0.smithi038.stdout:1536: done (2 left) 2024-03-19T11:31:35.006 INFO:tasks.rados.rados.0.smithi038.stdout:1537: done (1 left) 2024-03-19T11:31:35.007 INFO:tasks.rados.rados.0.smithi038.stdout:1539: delete oid 34 current snap is 34 2024-03-19T11:31:35.007 INFO:tasks.rados.rados.0.smithi038.stdout:1538: got (2) No such file or directory 2024-03-19T11:31:35.007 INFO:tasks.rados.rados.0.smithi038.stdout:1538: got expected ENOENT (src dne) 2024-03-19T11:31:35.008 INFO:tasks.rados.rados.0.smithi038.stdout:1538: done (1 left) 2024-03-19T11:31:35.008 INFO:tasks.rados.rados.0.smithi038.stdout:1539: done (0 left) 2024-03-19T11:31:35.008 INFO:tasks.rados.rados.0.smithi038.stdout:1540: tier_evict oid 27 2024-03-19T11:31:35.008 INFO:tasks.rados.rados.0.smithi038.stdout:1540: tier_evict oid 27 snap -1 2024-03-19T11:31:35.008 INFO:tasks.rados.rados.0.smithi038.stdout:1541: read oid 39 snap -1 2024-03-19T11:31:35.008 INFO:tasks.rados.rados.0.smithi038.stdout:1541: expect deleted 2024-03-19T11:31:35.008 INFO:tasks.rados.rados.0.smithi038.stdout:1542: write oid 33 current snap is 34 2024-03-19T11:31:35.008 INFO:tasks.rados.rados.0.smithi038.stdout:1542: seq_num 273 ranges {588311=615332,1842290=562759} 2024-03-19T11:31:35.018 INFO:tasks.rados.rados.0.smithi038.stdout:1542: writing smithi03829856-33 from 588311 to 1203643 tid 1 2024-03-19T11:31:35.028 INFO:tasks.rados.rados.0.smithi038.stdout:1542: writing smithi03829856-33 from 1842290 to 2405049 tid 2 2024-03-19T11:31:35.028 INFO:tasks.rados.rados.0.smithi038.stdout:1543: read oid 29 snap 21 2024-03-19T11:31:35.029 INFO:tasks.rados.rados.0.smithi038.stdout:1543: expect (ObjNum 165 snap 17 seq_num 165) 2024-03-19T11:31:35.029 INFO:tasks.rados.rados.0.smithi038.stdout:1544: write (excl) oid 23 current snap is 34 2024-03-19T11:31:35.029 INFO:tasks.rados.rados.0.smithi038.stdout:1544: seq_num 274 ranges {605118=406934,1617125=706417} 2024-03-19T11:31:35.034 INFO:tasks.rados.rados.0.smithi038.stdout:1544: writing smithi03829856-23 from 605118 to 1012052 tid 1 2024-03-19T11:31:35.042 INFO:tasks.rados.rados.0.smithi038.stdout:1544: writing smithi03829856-23 from 1617125 to 2323542 tid 2 2024-03-19T11:31:35.042 INFO:tasks.rados.rados.0.smithi038.stdout:1545: read oid 48 snap -1 2024-03-19T11:31:35.042 INFO:tasks.rados.rados.0.smithi038.stdout:1545: expect (ObjNum 255 snap 32 seq_num 255) 2024-03-19T11:31:35.042 INFO:tasks.rados.rados.0.smithi038.stdout:1546: write (excl) oid 45 current snap is 34 2024-03-19T11:31:35.042 INFO:tasks.rados.rados.0.smithi038.stdout:1540: got (22) Invalid argument 2024-03-19T11:31:35.043 INFO:tasks.rados.rados.0.smithi038.stdout:1542: finishing write tid 1 to smithi03829856-33 2024-03-19T11:31:35.043 INFO:tasks.rados.rados.0.smithi038.stdout:1546: seq_num 275 ranges {621925=482183,1559254=566428} 2024-03-19T11:31:35.046 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:35.054+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:35.049 INFO:tasks.rados.rados.0.smithi038.stdout:1546: writing smithi03829856-45 from 621925 to 1104108 tid 1 2024-03-19T11:31:35.055 INFO:tasks.rados.rados.0.smithi038.stdout:1546: writing smithi03829856-45 from 1559254 to 2125682 tid 2 2024-03-19T11:31:35.055 INFO:tasks.rados.rados.0.smithi038.stdout:1540: done (6 left) 2024-03-19T11:31:35.055 INFO:tasks.rados.rados.0.smithi038.stdout:1541: done (5 left) 2024-03-19T11:31:35.055 INFO:tasks.rados.rados.0.smithi038.stdout:1547: snap_create 2024-03-19T11:31:35.056 INFO:tasks.rados.rados.0.smithi038.stdout:1542: finishing write tid 2 to smithi03829856-33 2024-03-19T11:31:35.056 INFO:tasks.rados.rados.0.smithi038.stdout:1542: finishing write tid 3 to smithi03829856-33 2024-03-19T11:31:35.056 INFO:tasks.rados.rados.0.smithi038.stdout:1542: finishing write tid 4 to smithi03829856-33 2024-03-19T11:31:35.056 INFO:tasks.rados.rados.0.smithi038.stdout:1542: oid 33 updating version 0 to 181 2024-03-19T11:31:35.056 INFO:tasks.rados.rados.0.smithi038.stdout:1542: oid 33 updating version 181 to 182 2024-03-19T11:31:35.056 INFO:tasks.rados.rados.0.smithi038.stdout:1542: oid 33 version 182 is already newer than 180 2024-03-19T11:31:35.056 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 33 v 182 (ObjNum 273 snap 34 seq_num 273) dirty exists 2024-03-19T11:31:35.056 INFO:tasks.rados.rados.0.smithi038.stdout:1542: left oid 33 (ObjNum 273 snap 34 seq_num 273) 2024-03-19T11:31:35.056 INFO:tasks.rados.rados.0.smithi038.stdout:1543: expect (ObjNum 165 snap 17 seq_num 165) 2024-03-19T11:31:35.060 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:35.064+0000 7f6712f92640 -1 osd.2 114 handle_osd_map injecting map crc failure 2024-03-19T11:31:35.099 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:35.099 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:35.112 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-19T11:31:35.148 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:35.158+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:35.248 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:35.256+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:35.267 INFO:tasks.rados.rados.0.smithi038.stdout:1544: finishing write tid 1 to smithi03829856-23 2024-03-19T11:31:35.267 INFO:tasks.rados.rados.0.smithi038.stdout:1544: finishing write tid 2 to smithi03829856-23 2024-03-19T11:31:35.267 INFO:tasks.rados.rados.0.smithi038.stdout:1545: expect (ObjNum 255 snap 32 seq_num 255) 2024-03-19T11:31:35.280 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:35.280 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:35.294 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-19T11:31:35.380 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:35.388+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:35.417 INFO:tasks.rados.rados.0.smithi038.stdout:1544: finishing write tid 3 to smithi03829856-23 2024-03-19T11:31:35.418 INFO:tasks.rados.rados.0.smithi038.stdout:1544: finishing write tid 4 to smithi03829856-23 2024-03-19T11:31:35.418 INFO:tasks.rados.rados.0.smithi038.stdout:1544: oid 23 updating version 0 to 137 2024-03-19T11:31:35.418 INFO:tasks.rados.rados.0.smithi038.stdout:1544: oid 23 updating version 137 to 138 2024-03-19T11:31:35.418 INFO:tasks.rados.rados.0.smithi038.stdout:1544: oid 23 version 138 is already newer than 136 2024-03-19T11:31:35.418 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 23 v 138 (ObjNum 274 snap 34 seq_num 274) dirty exists 2024-03-19T11:31:35.418 INFO:tasks.rados.rados.0.smithi038.stdout:1544: left oid 23 (ObjNum 274 snap 34 seq_num 274) 2024-03-19T11:31:35.418 INFO:tasks.rados.rados.0.smithi038.stdout:1546: finishing write tid 1 to smithi03829856-45 2024-03-19T11:31:35.418 INFO:tasks.rados.rados.0.smithi038.stdout:1546: finishing write tid 2 to smithi03829856-45 2024-03-19T11:31:35.418 INFO:tasks.rados.rados.0.smithi038.stdout:1546: finishing write tid 3 to smithi03829856-45 2024-03-19T11:31:35.418 INFO:tasks.rados.rados.0.smithi038.stdout:1546: finishing write tid 4 to smithi03829856-45 2024-03-19T11:31:35.418 INFO:tasks.rados.rados.0.smithi038.stdout:1546: oid 45 updating version 0 to 79 2024-03-19T11:31:35.418 INFO:tasks.rados.rados.0.smithi038.stdout:1546: oid 45 version 79 is already newer than 78 2024-03-19T11:31:35.419 INFO:tasks.rados.rados.0.smithi038.stdout:1546: oid 45 version 79 is already newer than 77 2024-03-19T11:31:35.419 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 45 v 79 (ObjNum 275 snap 34 seq_num 275) dirty exists 2024-03-19T11:31:35.419 INFO:tasks.rados.rados.0.smithi038.stdout:1546: left oid 45 (ObjNum 275 snap 34 seq_num 275) 2024-03-19T11:31:35.450 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:35.462+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:35.451 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:35.451 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:35.465 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-19T11:31:35.552 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:35.560+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:35.632 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:35.632 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:35.646 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-19T11:31:35.652 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:35.662+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:35.655 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-19T11:31:35.754 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:35.762+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:35.806 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:35.806 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:35.820 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-19T11:31:35.843 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:31:35.844 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:31:35.854 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:35.860+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:35.860 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:31:35.860 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:31:35.876 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:31:35.877 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:31:35.893 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:31:35.893 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:31:35.913 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:31:35.914 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:31:35.929 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:35.930 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:31:35.930 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:31:35.930 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:31:35.930 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:35.934 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:31:35.934 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:31:35.944 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-19T11:31:35.955 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:31:35.956 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:31:35.956 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:35.964+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:35.976 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:31:35.976 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:31:36.057 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:36.066+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:36.061 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:36.072+0000 7fa5e08db640 -1 osd.3 115 handle_osd_map injecting map crc failure 2024-03-19T11:31:36.068 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:36.076+0000 7fa5de8d7640 -1 osd.3 115 handle_osd_map injecting map crc failure 2024-03-19T11:31:36.070 INFO:tasks.rados.rados.0.smithi038.stdout:1542: done (5 left) 2024-03-19T11:31:36.070 INFO:tasks.rados.rados.0.smithi038.stdout:1543: done (4 left) 2024-03-19T11:31:36.070 INFO:tasks.rados.rados.0.smithi038.stdout:1544: done (3 left) 2024-03-19T11:31:36.070 INFO:tasks.rados.rados.0.smithi038.stdout:1545: done (2 left) 2024-03-19T11:31:36.070 INFO:tasks.rados.rados.0.smithi038.stdout:1546: done (1 left) 2024-03-19T11:31:36.070 INFO:tasks.rados.rados.0.smithi038.stdout:1547: done (0 left) 2024-03-19T11:31:36.070 INFO:tasks.rados.rados.0.smithi038.stdout:1548: read oid 27 snap -1 2024-03-19T11:31:36.070 INFO:tasks.rados.rados.0.smithi038.stdout:1548: expect deleted 2024-03-19T11:31:36.070 INFO:tasks.rados.rados.0.smithi038.stdout:1549: read oid 17 snap -1 2024-03-19T11:31:36.070 INFO:tasks.rados.rados.0.smithi038.stdout:1549: expect (ObjNum 213 snap 22 seq_num 213) 2024-03-19T11:31:36.070 INFO:tasks.rados.rados.0.smithi038.stdout:1550: rollback oid 25 current snap is 35 2024-03-19T11:31:36.070 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 25 to 24 2024-03-19T11:31:36.070 INFO:tasks.rados.rados.0.smithi038.stdout:1551: read oid 2 snap -1 2024-03-19T11:31:36.070 INFO:tasks.rados.rados.0.smithi038.stdout:1551: expect (ObjNum 206 snap 22 seq_num 206) 2024-03-19T11:31:36.071 INFO:tasks.rados.rados.0.smithi038.stdout:1552: snap_create 2024-03-19T11:31:36.072 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:36.072 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:31:36.072 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:31:36.072 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:31:36.072 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.072 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.16 49/48 [PushOp(4:6868af91:::smithi03829856-10:2, version: 29'22, data_included: [551263~793594,1893632~254982], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:2@29'22, size: 2151263, copy_subset: [(0, 2151263)], clone_subset: {}, snapset: b=[]:{2: [2], 7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2148614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:6d9093b1:::smithi03829856-38:4, version: 33'29, data_included: [621859~449625,1865716~598951], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6d9093b1:::smithi03829856-38:4@33'29, size: 2621859, copy_subset: [(0, 2621859)], clone_subset: {}, snapset: 4=[]:{4: [2]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2464667, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:6d9093b1:::smithi03829856-38:head, version: 33'33, data_included: [621859~568773,1729599~479803], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:6d9093b1:::smithi03829856-38:head@33'33, size: 3142876, copy_subset: [(0, 3142876)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2209402, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:6be89c06:::smithi03829856-24:head, version: 36'39, data_included: [741376~589824,1753088~458752], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:6be89c06:::smithi03829856-24:head@36'39, size: 2386561, copy_subset: [(0, 2386561)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2211840, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:6868af91:::smithi03829856-10:7, version: 38'41, data_included: [524385~820472,1602039~228104], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:7@38'41, size: 2924385, copy_subset: [(0, 2924385)], clone_subset: {}, snapset: b=[]:{2: [2], 7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1830143, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:6868af91:::smithi03829856-10:b, version: 47'48, data_included: [524385~820472,1602039~228104], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:b@47'48, size: 2924385, copy_subset: [(0, 2924385)], clone_subset: {}, snapset: b=[]:{2: [2], 7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1830143, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:6868af91:::smithi03829856-10:head, version: 47'52, data_included: [524385~820472,1602039~228104], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:head@47'52, size: 3747928, copy_subset: [(0, 3747928)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1830143, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:36.072 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:26.954289+0000", 2024-03-19T11:31:36.073 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 69.127159535000004, 2024-03-19T11:31:36.073 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029260781999999999, 2024-03-19T11:31:36.073 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:36.073 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:36.073 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:36.073 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.073 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:36.073 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954289+0000", 2024-03-19T11:31:36.073 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.073 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.073 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.073 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:36.073 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954289+0000", 2024-03-19T11:31:36.073 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.074 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.074 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.074 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:36.074 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954291+0000", 2024-03-19T11:31:36.074 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8050000000000001e-06 2024-03-19T11:31:36.074 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.074 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.074 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:36.074 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.958775+0000", 2024-03-19T11:31:36.074 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0044845859999999996 2024-03-19T11:31:36.074 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.074 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.074 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:36.074 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.958851+0000", 2024-03-19T11:31:36.075 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6050999999999994e-05 2024-03-19T11:31:36.075 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.075 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.075 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:36.075 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.958931+0000", 2024-03-19T11:31:36.075 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9099999999999998e-05 2024-03-19T11:31:36.075 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.075 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.075 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:36.075 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.959206+0000", 2024-03-19T11:31:36.075 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00027495999999999998 2024-03-19T11:31:36.075 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.075 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.075 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:36.076 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.959246+0000", 2024-03-19T11:31:36.076 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0326e-05 2024-03-19T11:31:36.076 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.076 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.076 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:36.076 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.983550+0000", 2024-03-19T11:31:36.076 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024303953999999999 2024-03-19T11:31:36.076 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.076 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:36.076 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.076 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.077 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.077 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.16 49/48 [PushOp(4:6868af91:::smithi03829856-10:2, version: 29'22, data_included: [2148614~2649], data_size: 2649, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:2@29'22, size: 2151263, copy_subset: [(0, 2151263)], clone_subset: {}, snapset: b=[]:{2: [2], 7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2151263, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2148614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:6d9093b1:::smithi03829856-38:4, version: 33'29, data_included: [2464667~157192], data_size: 157192, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:6d9093b1:::smithi03829856-38:4@33'29, size: 2621859, copy_subset: [(0, 2621859)], clone_subset: {}, snapset: 4=[]:{4: [2]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2621859, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2464667, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:6d9093b1:::smithi03829856-38:head, version: 33'33, data_included: [2209402~412457,3101234~41642], data_size: 454099, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:6d9093b1:::smithi03829856-38:head@33'33, size: 3142876, copy_subset: [(0, 3142876)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3142876, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2209402, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:6be89c06:::smithi03829856-24:head, version: 36'39, data_included: [2211840~51851], data_size: 51851, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:6be89c06:::smithi03829856-24:head@36'39, size: 2386561, copy_subset: [(0, 2386561)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2386561, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2211840, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:6868af91:::smithi03829856-10:7, version: 38'41, data_included: [1830143~321120,2577441~346944], data_size: 668064, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:7@38'41, size: 2924385, copy_subset: [(0, 2924385)], clone_subset: {}, snapset: b=[]:{2: [2], 7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2924385, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1830143, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:6868af91:::smithi03829856-10:b, version: 47'48, data_included: [1830143~321120,2577441~346944], data_size: 668064, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:b@47'48, size: 2924385, copy_subset: [(0, 2924385)], clone_subset: {}, snapset: b=[]:{2: [2], 7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2924385, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1830143, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:6868af91:::smithi03829856-10:head, version: 47'52, data_included: [1830143~348419,2577441~661310], data_size: 1009729, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:head@47'52, size: 3747928, copy_subset: [(0, 3747928)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3747928, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1830143, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:36.077 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:26.998857+0000", 2024-03-19T11:31:36.077 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 69.082591774999997, 2024-03-19T11:31:36.077 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010149703, 2024-03-19T11:31:36.077 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:36.077 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:36.077 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:36.077 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.077 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:36.077 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.998857+0000", 2024-03-19T11:31:36.077 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.077 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.078 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.078 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:36.078 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.998857+0000", 2024-03-19T11:31:36.078 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.078 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.078 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.078 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:36.078 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.998858+0000", 2024-03-19T11:31:36.078 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.065e-06 2024-03-19T11:31:36.078 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.078 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.078 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:36.078 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.999629+0000", 2024-03-19T11:31:36.078 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000771132 2024-03-19T11:31:36.079 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.079 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.079 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:36.079 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.999688+0000", 2024-03-19T11:31:36.079 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9147000000000001e-05 2024-03-19T11:31:36.079 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.079 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.079 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:36.079 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.999755+0000", 2024-03-19T11:31:36.079 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6891999999999996e-05 2024-03-19T11:31:36.079 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.079 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.079 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:36.079 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.000047+0000", 2024-03-19T11:31:36.080 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000292176 2024-03-19T11:31:36.080 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.080 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.080 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:36.080 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.000094+0000", 2024-03-19T11:31:36.080 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6485e-05 2024-03-19T11:31:36.080 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.080 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.080 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:36.080 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.009007+0000", 2024-03-19T11:31:36.080 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0089128060000000005 2024-03-19T11:31:36.080 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.080 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:36.080 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.080 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.081 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.081 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1630 4.d 4:ba44105a:::smithi03829856-26:head [write 467261~717275] snapc 10=[10,f,e,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-03-19T11:31:36.081 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:35.331434+0000", 2024-03-19T11:31:36.081 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 60.750014729999997, 2024-03-19T11:31:36.081 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012902375000000001, 2024-03-19T11:31:36.081 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:36.081 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:36.081 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:36.081 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:36.081 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:36.081 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1630 2024-03-19T11:31:36.081 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.081 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:36.081 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.082 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:36.082 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.331434+0000", 2024-03-19T11:31:36.082 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.082 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.082 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.082 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:36.082 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.331434+0000", 2024-03-19T11:31:36.082 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.082 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.082 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.082 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:36.082 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.331437+0000", 2024-03-19T11:31:36.082 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5599999999999998e-06 2024-03-19T11:31:36.082 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.083 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.083 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:36.083 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.331938+0000", 2024-03-19T11:31:36.083 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00050090099999999999 2024-03-19T11:31:36.083 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.083 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.083 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:36.083 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.331976+0000", 2024-03-19T11:31:36.083 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8090999999999998e-05 2024-03-19T11:31:36.083 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.083 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.083 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:36.083 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.331994+0000", 2024-03-19T11:31:36.083 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7566000000000001e-05 2024-03-19T11:31:36.083 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.084 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.084 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:36.084 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.332086+0000", 2024-03-19T11:31:36.084 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2452000000000002e-05 2024-03-19T11:31:36.084 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.084 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.084 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:36.084 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.332205+0000", 2024-03-19T11:31:36.084 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000118966 2024-03-19T11:31:36.084 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.084 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.084 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1", 2024-03-19T11:31:36.084 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.332410+0000", 2024-03-19T11:31:36.084 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020430099999999999 2024-03-19T11:31:36.085 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.085 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.085 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:36.085 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.339226+0000", 2024-03-19T11:31:36.085 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0068165980000000001 2024-03-19T11:31:36.085 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.085 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.085 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:36.085 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.344191+0000", 2024-03-19T11:31:36.085 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.004965143 2024-03-19T11:31:36.085 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.085 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.085 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:36.085 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.344272+0000", 2024-03-19T11:31:36.085 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0265000000000005e-05 2024-03-19T11:31:36.086 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.086 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.086 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:36.086 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.344336+0000", 2024-03-19T11:31:36.086 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4531999999999998e-05 2024-03-19T11:31:36.086 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.086 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:36.086 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.086 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.086 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.086 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1633 4.d 4:ba44105a:::smithi03829856-26:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-03-19T11:31:36.086 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:35.337032+0000", 2024-03-19T11:31:36.086 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 60.744416350999998, 2024-03-19T11:31:36.087 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093229679999999992, 2024-03-19T11:31:36.087 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:36.087 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:36.087 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:36.087 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:36.087 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:36.087 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1633 2024-03-19T11:31:36.087 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.087 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:36.087 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.087 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:36.087 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.337032+0000", 2024-03-19T11:31:36.087 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.087 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.087 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.088 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:36.088 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.337032+0000", 2024-03-19T11:31:36.088 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.088 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.088 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.088 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:36.088 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.337033+0000", 2024-03-19T11:31:36.088 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4899999999999995e-07 2024-03-19T11:31:36.088 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.088 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.088 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:36.088 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.337037+0000", 2024-03-19T11:31:36.088 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7249999999999999e-06 2024-03-19T11:31:36.089 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.089 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.089 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:36.089 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.337045+0000", 2024-03-19T11:31:36.089 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5380000000000002e-06 2024-03-19T11:31:36.089 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.089 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.089 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:36.089 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.337051+0000", 2024-03-19T11:31:36.089 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1689999999999997e-06 2024-03-19T11:31:36.089 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.089 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.089 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:36.089 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.339128+0000", 2024-03-19T11:31:36.089 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002076919 2024-03-19T11:31:36.090 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.090 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.090 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-03-19T11:31:36.090 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.339211+0000", 2024-03-19T11:31:36.090 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2896000000000001e-05 2024-03-19T11:31:36.090 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.090 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.090 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:36.090 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.346152+0000", 2024-03-19T11:31:36.090 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0069404779999999999 2024-03-19T11:31:36.090 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.090 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.090 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:36.091 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.346242+0000", 2024-03-19T11:31:36.091 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0031000000000006e-05 2024-03-19T11:31:36.091 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.091 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.091 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:36.091 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.346355+0000", 2024-03-19T11:31:36.091 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000113663 2024-03-19T11:31:36.091 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.091 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:36.091 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.091 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.091 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.091 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1845 4.4 4:27909373:::smithi03829856-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e62)", 2024-03-19T11:31:36.091 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:41.053790+0000", 2024-03-19T11:31:36.092 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 55.027658270000003, 2024-03-19T11:31:36.092 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0091637609999999994, 2024-03-19T11:31:36.092 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:36.092 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:36.092 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:36.092 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:36.092 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:36.092 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1845 2024-03-19T11:31:36.092 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.092 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:36.092 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.092 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:36.092 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053790+0000", 2024-03-19T11:31:36.092 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.092 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.093 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.093 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:36.093 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053790+0000", 2024-03-19T11:31:36.093 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.093 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.093 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.093 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:36.093 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053792+0000", 2024-03-19T11:31:36.093 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2860000000000001e-06 2024-03-19T11:31:36.093 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.093 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.093 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:36.093 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053802+0000", 2024-03-19T11:31:36.093 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.064e-05 2024-03-19T11:31:36.093 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.094 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.094 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:36.094 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053824+0000", 2024-03-19T11:31:36.094 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1611999999999999e-05 2024-03-19T11:31:36.094 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.094 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.094 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:36.094 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.053839+0000", 2024-03-19T11:31:36.094 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4922e-05 2024-03-19T11:31:36.094 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.094 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.094 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:36.094 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.059104+0000", 2024-03-19T11:31:36.094 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0052656179999999997 2024-03-19T11:31:36.095 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.095 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.095 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-03-19T11:31:36.095 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.059267+0000", 2024-03-19T11:31:36.095 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000162776 2024-03-19T11:31:36.095 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.095 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.095 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:36.095 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.062551+0000", 2024-03-19T11:31:36.095 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003283467 2024-03-19T11:31:36.095 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.095 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.095 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:36.095 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.062738+0000", 2024-03-19T11:31:36.095 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00018781900000000001 2024-03-19T11:31:36.096 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.096 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.096 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:36.096 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.062954+0000", 2024-03-19T11:31:36.096 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00021562099999999999 2024-03-19T11:31:36.096 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.096 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:36.096 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.096 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.096 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.096 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1889 4.4 4:266590a5:::smithi03829856-16:head [rollback d] snapc 10=[10,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-03-19T11:31:36.096 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:41.697617+0000", 2024-03-19T11:31:36.096 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 54.383831735999998, 2024-03-19T11:31:36.096 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018164059999999999, 2024-03-19T11:31:36.096 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:36.097 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:36.097 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:36.097 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:36.097 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:36.097 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1889 2024-03-19T11:31:36.097 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.097 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:36.097 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.097 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:36.097 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697617+0000", 2024-03-19T11:31:36.097 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.097 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.097 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.097 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:36.097 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697617+0000", 2024-03-19T11:31:36.098 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.098 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.098 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.098 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:36.098 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697619+0000", 2024-03-19T11:31:36.098 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0059999999999999e-06 2024-03-19T11:31:36.098 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.098 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.098 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:36.098 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697631+0000", 2024-03-19T11:31:36.098 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2033e-05 2024-03-19T11:31:36.098 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.098 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.099 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:36.099 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697659+0000", 2024-03-19T11:31:36.099 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8291999999999999e-05 2024-03-19T11:31:36.099 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.099 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.099 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:36.099 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697682+0000", 2024-03-19T11:31:36.099 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3281999999999998e-05 2024-03-19T11:31:36.099 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.099 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.099 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:36.099 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.701122+0000", 2024-03-19T11:31:36.099 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034396679999999999 2024-03-19T11:31:36.099 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.099 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.100 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:36.100 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.701383+0000", 2024-03-19T11:31:36.100 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00026086399999999997 2024-03-19T11:31:36.100 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.100 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.100 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3", 2024-03-19T11:31:36.100 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.709693+0000", 2024-03-19T11:31:36.100 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0083096139999999999 2024-03-19T11:31:36.100 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.100 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.100 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:36.100 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.715437+0000", 2024-03-19T11:31:36.100 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057442379999999996 2024-03-19T11:31:36.100 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.101 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.101 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:36.101 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.715563+0000", 2024-03-19T11:31:36.101 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000125778 2024-03-19T11:31:36.101 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.101 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.101 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:36.101 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.715677+0000", 2024-03-19T11:31:36.101 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000114728 2024-03-19T11:31:36.101 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.101 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.101 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:36.101 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.715781+0000", 2024-03-19T11:31:36.101 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000103557 2024-03-19T11:31:36.101 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.102 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:36.102 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.102 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.102 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.102 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1890 4.4 4:266590a5:::smithi03829856-16:head [append 0~0] snapc 10=[10,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-03-19T11:31:36.102 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:41.697709+0000", 2024-03-19T11:31:36.102 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 54.383739173999999, 2024-03-19T11:31:36.102 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019825171999999999, 2024-03-19T11:31:36.102 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:36.102 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:36.102 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:36.102 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:36.102 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:36.102 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1890 2024-03-19T11:31:36.102 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.103 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:36.103 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.103 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:36.103 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697709+0000", 2024-03-19T11:31:36.103 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.103 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.103 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.103 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:36.103 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697709+0000", 2024-03-19T11:31:36.103 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.104 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.104 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.104 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:36.104 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697711+0000", 2024-03-19T11:31:36.104 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.296e-06 2024-03-19T11:31:36.104 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.104 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.104 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:36.104 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697718+0000", 2024-03-19T11:31:36.104 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.464e-06 2024-03-19T11:31:36.104 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.104 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.104 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:36.104 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697739+0000", 2024-03-19T11:31:36.104 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1005e-05 2024-03-19T11:31:36.105 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.105 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.105 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:36.105 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697754+0000", 2024-03-19T11:31:36.105 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4976e-05 2024-03-19T11:31:36.105 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.105 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.105 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:36.105 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.702341+0000", 2024-03-19T11:31:36.105 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0045870980000000004 2024-03-19T11:31:36.106 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.106 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.106 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for blocked object", 2024-03-19T11:31:36.106 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.702482+0000", 2024-03-19T11:31:36.106 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014048000000000001 2024-03-19T11:31:36.106 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.106 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.106 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:36.106 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.714359+0000", 2024-03-19T11:31:36.106 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011877522999999999 2024-03-19T11:31:36.106 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.106 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.106 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:36.106 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.714550+0000", 2024-03-19T11:31:36.107 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019117 2024-03-19T11:31:36.107 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.107 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.107 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3", 2024-03-19T11:31:36.107 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.714911+0000", 2024-03-19T11:31:36.107 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00036070900000000002 2024-03-19T11:31:36.107 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.107 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.107 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:36.107 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.716211+0000", 2024-03-19T11:31:36.107 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001300329 2024-03-19T11:31:36.107 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.107 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.107 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:36.107 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.717326+0000", 2024-03-19T11:31:36.108 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001114056 2024-03-19T11:31:36.108 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.108 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.108 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:36.108 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.717423+0000", 2024-03-19T11:31:36.108 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7048999999999999e-05 2024-03-19T11:31:36.108 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.108 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.108 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:36.109 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.717535+0000", 2024-03-19T11:31:36.109 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011201699999999999 2024-03-19T11:31:36.109 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.109 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:36.109 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.109 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.109 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.109 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1972 4.4 4:214026b3:::smithi03829856-47:head [tier-promote] snapc 12=[12,11,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:31:36.109 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:45.312422+0000", 2024-03-19T11:31:36.109 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 50.769026500000002, 2024-03-19T11:31:36.109 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013955089, 2024-03-19T11:31:36.109 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:36.109 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:36.109 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:36.110 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:36.110 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:36.110 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1972 2024-03-19T11:31:36.110 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.110 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:36.110 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.110 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:36.110 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.312422+0000", 2024-03-19T11:31:36.110 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.110 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.110 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.110 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:36.110 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.312422+0000", 2024-03-19T11:31:36.110 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.110 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.111 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.111 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:36.111 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.312424+0000", 2024-03-19T11:31:36.111 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9810000000000002e-06 2024-03-19T11:31:36.111 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.111 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.111 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:36.111 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.312433+0000", 2024-03-19T11:31:36.111 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4549999999999995e-06 2024-03-19T11:31:36.111 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.111 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.111 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:36.111 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.312452+0000", 2024-03-19T11:31:36.111 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9380000000000001e-05 2024-03-19T11:31:36.112 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.112 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.112 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:36.112 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.312460+0000", 2024-03-19T11:31:36.112 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2560000000000002e-06 2024-03-19T11:31:36.112 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.112 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.112 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:36.112 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.312555+0000", 2024-03-19T11:31:36.112 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.5208000000000001e-05 2024-03-19T11:31:36.112 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.112 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.112 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:36.112 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.312704+0000", 2024-03-19T11:31:36.113 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014843899999999999 2024-03-19T11:31:36.113 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.113 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.113 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:36.113 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.326377+0000", 2024-03-19T11:31:36.113 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013673370000000001 2024-03-19T11:31:36.113 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.113 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:36.113 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.113 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.113 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.113 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2005 4.4 4:214026b3:::smithi03829856-47:head [read 0~2773122,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:31:36.113 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:45.804325+0000", 2024-03-19T11:31:36.113 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 50.277124039999997, 2024-03-19T11:31:36.113 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010899282, 2024-03-19T11:31:36.114 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:36.114 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:36.114 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:36.114 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:36.114 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:36.114 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2005 2024-03-19T11:31:36.114 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.114 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:36.114 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.114 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:36.114 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804325+0000", 2024-03-19T11:31:36.114 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.114 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.114 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.115 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:36.115 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804325+0000", 2024-03-19T11:31:36.115 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.115 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.116 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.116 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:36.116 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804326+0000", 2024-03-19T11:31:36.116 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.666e-06 2024-03-19T11:31:36.116 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.116 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.116 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:36.116 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804338+0000", 2024-03-19T11:31:36.116 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2096000000000001e-05 2024-03-19T11:31:36.116 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.116 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.116 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:36.116 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804363+0000", 2024-03-19T11:31:36.116 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4762000000000001e-05 2024-03-19T11:31:36.117 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.117 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.117 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:36.117 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804380+0000", 2024-03-19T11:31:36.117 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.713e-05 2024-03-19T11:31:36.117 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.117 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.117 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:36.117 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.812236+0000", 2024-03-19T11:31:36.117 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0078554690000000003 2024-03-19T11:31:36.117 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.117 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.117 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:36.117 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.812541+0000", 2024-03-19T11:31:36.117 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030564299999999997 2024-03-19T11:31:36.118 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.118 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.118 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:36.118 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.815224+0000", 2024-03-19T11:31:36.119 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0026825159999999998 2024-03-19T11:31:36.119 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.119 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:36.119 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.119 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.119 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.119 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2006 4.4 4:214026b3:::smithi03829856-47:head [read 0~2773122,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:31:36.120 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:45.804421+0000", 2024-03-19T11:31:36.120 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 50.277027300999997, 2024-03-19T11:31:36.120 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013366463, 2024-03-19T11:31:36.120 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:36.120 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:36.120 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:36.120 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:36.120 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:36.120 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2006 2024-03-19T11:31:36.120 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.120 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:36.120 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.120 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:36.121 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804421+0000", 2024-03-19T11:31:36.121 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.121 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.121 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.121 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:36.121 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804421+0000", 2024-03-19T11:31:36.121 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.121 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.121 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.121 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:36.121 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804423+0000", 2024-03-19T11:31:36.121 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5600000000000001e-06 2024-03-19T11:31:36.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.122 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:36.122 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804433+0000", 2024-03-19T11:31:36.122 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8719999999999997e-06 2024-03-19T11:31:36.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.122 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:36.122 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804454+0000", 2024-03-19T11:31:36.122 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1759e-05 2024-03-19T11:31:36.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.122 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:36.123 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804470+0000", 2024-03-19T11:31:36.123 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5556000000000001e-05 2024-03-19T11:31:36.123 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.123 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.123 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:36.123 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.815325+0000", 2024-03-19T11:31:36.123 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010855069 2024-03-19T11:31:36.123 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.123 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.123 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:36.123 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.815684+0000", 2024-03-19T11:31:36.123 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00035912699999999999 2024-03-19T11:31:36.123 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.123 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.123 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:36.124 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.817788+0000", 2024-03-19T11:31:36.124 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0021035200000000002 2024-03-19T11:31:36.124 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.124 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:36.124 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.124 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.124 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.124 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(osd.1.0:198 4.16 e67/48 4:6868af91:::smithi03829856-10:head v 67'79, mlcod=66'76)", 2024-03-19T11:31:36.124 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:45.811039+0000", 2024-03-19T11:31:36.125 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 50.27040994, 2024-03-19T11:31:36.125 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01087364, 2024-03-19T11:31:36.125 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:36.125 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:36.125 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:36.125 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.125 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:36.125 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.811039+0000", 2024-03-19T11:31:36.125 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.125 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.125 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.125 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:36.125 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.811039+0000", 2024-03-19T11:31:36.126 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.126 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.126 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.126 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:36.126 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.811041+0000", 2024-03-19T11:31:36.126 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6400000000000001e-06 2024-03-19T11:31:36.126 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.126 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.126 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:36.126 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.812874+0000", 2024-03-19T11:31:36.126 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0018329959999999999 2024-03-19T11:31:36.126 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.126 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.127 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:36.127 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.812916+0000", 2024-03-19T11:31:36.127 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1875000000000001e-05 2024-03-19T11:31:36.127 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.127 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.127 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:36.127 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.812937+0000", 2024-03-19T11:31:36.127 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0985e-05 2024-03-19T11:31:36.127 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.127 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.127 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:36.127 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.813112+0000", 2024-03-19T11:31:36.127 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000174573 2024-03-19T11:31:36.127 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.128 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.128 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:36.128 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.813170+0000", 2024-03-19T11:31:36.128 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8100999999999998e-05 2024-03-19T11:31:36.128 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.128 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.128 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:36.128 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.821792+0000", 2024-03-19T11:31:36.128 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0086218549999999994 2024-03-19T11:31:36.129 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.129 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.129 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:36.129 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.821912+0000", 2024-03-19T11:31:36.129 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000120615 2024-03-19T11:31:36.129 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.129 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:36.129 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.129 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.129 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.129 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2027 4.2d 4:b4ef97e0:::smithi03829856-3:head [read 0~3697507,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:31:36.129 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:46.264908+0000", 2024-03-19T11:31:36.129 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 49.816541074, 2024-03-19T11:31:36.129 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010951414, 2024-03-19T11:31:36.130 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:36.130 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:36.130 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:36.130 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:36.130 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:36.130 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2027 2024-03-19T11:31:36.130 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.130 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:36.130 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.130 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:36.130 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264908+0000", 2024-03-19T11:31:36.130 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.130 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.130 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.131 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:36.131 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264908+0000", 2024-03-19T11:31:36.131 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.131 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.131 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:36.131 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264908+0000", 2024-03-19T11:31:36.131 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5799999999999999e-07 2024-03-19T11:31:36.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.131 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.131 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:36.131 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264912+0000", 2024-03-19T11:31:36.131 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2649999999999998e-06 2024-03-19T11:31:36.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.131 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.132 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:36.132 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264920+0000", 2024-03-19T11:31:36.132 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.79e-06 2024-03-19T11:31:36.132 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.132 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.132 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:36.132 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264926+0000", 2024-03-19T11:31:36.132 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9989999999999997e-06 2024-03-19T11:31:36.132 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.132 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.132 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:36.132 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.273461+0000", 2024-03-19T11:31:36.132 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0085352680000000004 2024-03-19T11:31:36.132 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.132 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.133 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:36.133 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.273598+0000", 2024-03-19T11:31:36.133 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013622299999999999 2024-03-19T11:31:36.133 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.133 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.133 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:36.133 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.275859+0000", 2024-03-19T11:31:36.133 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022613109999999998 2024-03-19T11:31:36.133 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.134 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:36.134 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.134 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.134 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.134 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2028 4.2d 4:b4ef97e0:::smithi03829856-3:head [read 0~3697507,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:31:36.134 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:46.264934+0000", 2024-03-19T11:31:36.134 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 49.816514443000003, 2024-03-19T11:31:36.134 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013708481, 2024-03-19T11:31:36.134 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:36.134 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:36.134 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:36.135 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:36.135 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:36.135 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2028 2024-03-19T11:31:36.135 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.135 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:36.135 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.135 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:36.135 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264934+0000", 2024-03-19T11:31:36.135 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.135 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.135 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.135 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:36.135 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264934+0000", 2024-03-19T11:31:36.135 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.135 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.136 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.136 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:36.136 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264935+0000", 2024-03-19T11:31:36.136 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2599999999999998e-07 2024-03-19T11:31:36.136 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.136 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.136 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:36.136 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264938+0000", 2024-03-19T11:31:36.136 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.163e-06 2024-03-19T11:31:36.136 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.136 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.136 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:36.136 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264945+0000", 2024-03-19T11:31:36.136 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.0369999999999997e-06 2024-03-19T11:31:36.137 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.137 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.137 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:36.137 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264950+0000", 2024-03-19T11:31:36.137 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1170000000000003e-06 2024-03-19T11:31:36.137 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.137 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.137 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:36.137 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.275914+0000", 2024-03-19T11:31:36.137 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010964150000000001 2024-03-19T11:31:36.137 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.137 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.137 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:36.137 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.276037+0000", 2024-03-19T11:31:36.137 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012250100000000001 2024-03-19T11:31:36.138 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.138 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.138 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:36.138 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.278643+0000", 2024-03-19T11:31:36.138 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002606087 2024-03-19T11:31:36.138 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.138 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:36.138 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.138 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.138 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.138 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2045 4.4 4:21eac9b2:::smithi03829856-4:head [read 0~2537857,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:31:36.138 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:46.265120+0000", 2024-03-19T11:31:36.138 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 49.816328134999999, 2024-03-19T11:31:36.138 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0096591980000000008, 2024-03-19T11:31:36.139 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:36.139 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:36.139 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:36.139 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:36.139 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:36.139 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2045 2024-03-19T11:31:36.139 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.139 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:36.139 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.139 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:36.139 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265120+0000", 2024-03-19T11:31:36.139 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.139 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.139 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.139 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:36.140 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265120+0000", 2024-03-19T11:31:36.140 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.140 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.140 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.140 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:36.140 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265121+0000", 2024-03-19T11:31:36.140 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2399999999999998e-07 2024-03-19T11:31:36.140 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.140 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.140 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:36.140 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265126+0000", 2024-03-19T11:31:36.140 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4870000000000002e-06 2024-03-19T11:31:36.140 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.140 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.140 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:36.141 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265134+0000", 2024-03-19T11:31:36.141 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9740000000000002e-06 2024-03-19T11:31:36.141 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.141 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.141 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:36.141 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265143+0000", 2024-03-19T11:31:36.141 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7530000000000007e-06 2024-03-19T11:31:36.141 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.141 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.141 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:36.141 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.273765+0000", 2024-03-19T11:31:36.141 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0086213880000000007 2024-03-19T11:31:36.141 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.141 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.142 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:36.142 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.273872+0000", 2024-03-19T11:31:36.142 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000107452 2024-03-19T11:31:36.142 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.142 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.142 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:36.142 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.274780+0000", 2024-03-19T11:31:36.142 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00090762000000000004 2024-03-19T11:31:36.142 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.142 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:36.142 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.142 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.142 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.142 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2046 4.4 4:21eac9b2:::smithi03829856-4:head [read 0~2537857,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:31:36.143 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:46.265152+0000", 2024-03-19T11:31:36.143 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 49.816296776000002, 2024-03-19T11:31:36.143 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013074417, 2024-03-19T11:31:36.143 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:36.143 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:36.143 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:36.143 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:36.143 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:36.143 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2046 2024-03-19T11:31:36.143 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.144 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:36.144 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.144 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:36.144 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265152+0000", 2024-03-19T11:31:36.144 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.144 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.144 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.144 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:36.144 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265152+0000", 2024-03-19T11:31:36.144 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.144 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.144 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.144 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:36.145 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265152+0000", 2024-03-19T11:31:36.145 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8999999999999997e-07 2024-03-19T11:31:36.145 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.145 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.145 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:36.145 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265155+0000", 2024-03-19T11:31:36.145 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0920000000000002e-06 2024-03-19T11:31:36.145 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.145 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.145 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:36.145 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265162+0000", 2024-03-19T11:31:36.145 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6730000000000003e-06 2024-03-19T11:31:36.145 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.145 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.146 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:36.146 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265167+0000", 2024-03-19T11:31:36.146 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1839999999999998e-06 2024-03-19T11:31:36.146 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.146 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.146 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:36.146 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.276218+0000", 2024-03-19T11:31:36.146 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011050559999999999 2024-03-19T11:31:36.146 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.146 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.146 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:36.146 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.276345+0000", 2024-03-19T11:31:36.146 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000127243 2024-03-19T11:31:36.146 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.147 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.147 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:36.147 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.278226+0000", 2024-03-19T11:31:36.147 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0018811749999999999 2024-03-19T11:31:36.147 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.147 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:36.147 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.147 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.147 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.147 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:3046 4.19 e103/73 4:9c74b127:::smithi03829856-33:head v 103'148, mlcod=99'144)", 2024-03-19T11:31:36.147 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:22.149246+0000", 2024-03-19T11:31:36.147 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 13.932202708, 2024-03-19T11:31:36.147 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014101367, 2024-03-19T11:31:36.147 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:36.148 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:36.148 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:36.148 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.148 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:36.148 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.149246+0000", 2024-03-19T11:31:36.148 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.148 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.148 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.148 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:36.148 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.149246+0000", 2024-03-19T11:31:36.148 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.148 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.148 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.148 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:36.148 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.149246+0000", 2024-03-19T11:31:36.149 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2300000000000002e-07 2024-03-19T11:31:36.149 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.149 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.149 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:36.149 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.149663+0000", 2024-03-19T11:31:36.149 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00041629599999999997 2024-03-19T11:31:36.149 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.149 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.149 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:36.149 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.149682+0000", 2024-03-19T11:31:36.149 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9500999999999998e-05 2024-03-19T11:31:36.149 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.149 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.149 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:36.150 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.149692+0000", 2024-03-19T11:31:36.150 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.825e-06 2024-03-19T11:31:36.150 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.150 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.150 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:36.150 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.149797+0000", 2024-03-19T11:31:36.150 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000105222 2024-03-19T11:31:36.150 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.150 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.150 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:36.150 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.149835+0000", 2024-03-19T11:31:36.150 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7934e-05 2024-03-19T11:31:36.150 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.150 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.151 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:36.151 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.163304+0000", 2024-03-19T11:31:36.151 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013469122 2024-03-19T11:31:36.151 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.151 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.151 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:36.151 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.163347+0000", 2024-03-19T11:31:36.151 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3143999999999999e-05 2024-03-19T11:31:36.151 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.151 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:36.151 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.151 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.151 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.151 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:3221 4.4 e109/83 4:214026b3:::smithi03829856-47:head v 109'146, mlcod=108'142)", 2024-03-19T11:31:36.151 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:28.083665+0000", 2024-03-19T11:31:36.152 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 7.9977831840000002, 2024-03-19T11:31:36.152 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010710984, 2024-03-19T11:31:36.152 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:36.152 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:36.152 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:36.152 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.152 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:36.152 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.083665+0000", 2024-03-19T11:31:36.152 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.152 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.152 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.152 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:36.152 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.083665+0000", 2024-03-19T11:31:36.152 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.152 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.153 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.153 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:36.153 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.083666+0000", 2024-03-19T11:31:36.153 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5900000000000003e-07 2024-03-19T11:31:36.153 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.153 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.153 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:36.153 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.084140+0000", 2024-03-19T11:31:36.153 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00047418100000000001 2024-03-19T11:31:36.153 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.153 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.153 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:36.153 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.084164+0000", 2024-03-19T11:31:36.153 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3978999999999999e-05 2024-03-19T11:31:36.153 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.154 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.154 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:36.154 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.084174+0000", 2024-03-19T11:31:36.154 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0149999999999999e-05 2024-03-19T11:31:36.154 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.154 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.154 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:36.154 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.084657+0000", 2024-03-19T11:31:36.154 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00048310200000000001 2024-03-19T11:31:36.154 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.154 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.154 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:36.154 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.084720+0000", 2024-03-19T11:31:36.154 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2946000000000003e-05 2024-03-19T11:31:36.155 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.155 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.155 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:36.155 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.094299+0000", 2024-03-19T11:31:36.155 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0095788390000000005 2024-03-19T11:31:36.155 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.155 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.155 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:36.155 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.094376+0000", 2024-03-19T11:31:36.155 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7428000000000005e-05 2024-03-19T11:31:36.155 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.155 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:36.155 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.155 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.155 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.156 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3240 4.1c 4:3c71e808:::smithi03829856-34:head [read 0~3210104,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e109)", 2024-03-19T11:31:36.156 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:28.277613+0000", 2024-03-19T11:31:36.156 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 7.8038360190000002, 2024-03-19T11:31:36.156 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011179553, 2024-03-19T11:31:36.156 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:36.156 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:36.156 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:36.156 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:36.156 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:36.156 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3240 2024-03-19T11:31:36.156 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.156 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:36.156 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.156 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:36.156 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.277613+0000", 2024-03-19T11:31:36.157 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.157 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.157 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.157 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:36.157 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.277613+0000", 2024-03-19T11:31:36.157 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.157 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.157 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.157 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:36.157 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.277614+0000", 2024-03-19T11:31:36.157 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1769999999999999e-06 2024-03-19T11:31:36.157 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.157 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.158 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:36.158 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.277622+0000", 2024-03-19T11:31:36.158 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3590000000000008e-06 2024-03-19T11:31:36.158 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.158 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.158 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:36.158 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.277644+0000", 2024-03-19T11:31:36.158 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1642e-05 2024-03-19T11:31:36.158 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.158 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.158 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:36.158 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.277660+0000", 2024-03-19T11:31:36.158 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6316000000000001e-05 2024-03-19T11:31:36.158 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.159 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.159 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:36.159 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.284699+0000", 2024-03-19T11:31:36.159 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0070386950000000002 2024-03-19T11:31:36.159 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.159 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.159 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:36.159 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.284990+0000", 2024-03-19T11:31:36.159 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00029164099999999998 2024-03-19T11:31:36.159 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.159 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.159 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:36.159 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.288792+0000", 2024-03-19T11:31:36.159 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038017229999999999 2024-03-19T11:31:36.159 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.160 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:36.160 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.160 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.160 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.160 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:3361 4.19 e112/73 4:9ef41243:::smithi03829856-5:head v 112'173, mlcod=111'169)", 2024-03-19T11:31:36.160 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:32.254683+0000", 2024-03-19T11:31:36.160 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.8267651049999998, 2024-03-19T11:31:36.160 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014131003, 2024-03-19T11:31:36.160 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:36.160 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:36.160 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:36.160 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.160 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:36.160 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.254683+0000", 2024-03-19T11:31:36.160 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.161 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.161 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.161 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:36.161 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.254683+0000", 2024-03-19T11:31:36.161 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.161 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.161 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.161 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:36.161 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.254684+0000", 2024-03-19T11:31:36.161 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0900000000000001e-07 2024-03-19T11:31:36.161 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.161 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.161 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:36.161 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.255377+0000", 2024-03-19T11:31:36.162 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00069336899999999999 2024-03-19T11:31:36.162 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.162 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.162 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:36.162 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.255407+0000", 2024-03-19T11:31:36.162 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9113000000000001e-05 2024-03-19T11:31:36.162 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.162 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.162 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:36.162 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.255426+0000", 2024-03-19T11:31:36.162 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9026e-05 2024-03-19T11:31:36.162 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.162 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.162 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:36.163 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.255607+0000", 2024-03-19T11:31:36.163 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00018114199999999999 2024-03-19T11:31:36.163 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.163 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.163 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:36.163 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.255656+0000", 2024-03-19T11:31:36.163 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9679999999999999e-05 2024-03-19T11:31:36.163 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.163 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.163 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:36.163 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.268717+0000", 2024-03-19T11:31:36.163 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013060992 2024-03-19T11:31:36.163 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.163 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.164 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:36.164 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.268814+0000", 2024-03-19T11:31:36.164 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7071999999999995e-05 2024-03-19T11:31:36.164 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.164 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:36.164 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.164 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.164 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.164 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:3407 4.d e113/86 4:b65e6658:::smithi03829856-9:head v 113'109, mlcod=112'105)", 2024-03-19T11:31:36.164 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:33.520349+0000", 2024-03-19T11:31:36.164 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 2.561099837, 2024-03-19T11:31:36.164 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0098278280000000003, 2024-03-19T11:31:36.164 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:36.165 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:36.165 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:36.165 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.165 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:36.165 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.520349+0000", 2024-03-19T11:31:36.165 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.165 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.165 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.165 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:36.165 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.520349+0000", 2024-03-19T11:31:36.165 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.165 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.165 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.165 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:36.166 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.520350+0000", 2024-03-19T11:31:36.166 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.299e-06 2024-03-19T11:31:36.166 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.166 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.166 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:36.166 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.521383+0000", 2024-03-19T11:31:36.166 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001032664 2024-03-19T11:31:36.166 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.166 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.166 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:36.166 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.521397+0000", 2024-03-19T11:31:36.166 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4455e-05 2024-03-19T11:31:36.166 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.167 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.167 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:36.167 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.521408+0000", 2024-03-19T11:31:36.167 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0744000000000001e-05 2024-03-19T11:31:36.167 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.167 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.167 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:36.167 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.521498+0000", 2024-03-19T11:31:36.167 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9711999999999998e-05 2024-03-19T11:31:36.167 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.167 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.167 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:36.167 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.521528+0000", 2024-03-19T11:31:36.167 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0827999999999997e-05 2024-03-19T11:31:36.168 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.168 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.168 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:36.168 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.530137+0000", 2024-03-19T11:31:36.168 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0086083979999999997 2024-03-19T11:31:36.168 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.168 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.168 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:36.168 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.530177+0000", 2024-03-19T11:31:36.168 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.9728000000000003e-05 2024-03-19T11:31:36.168 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.168 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:36.168 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.168 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.168 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:36.169 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:36.169 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-03-19T11:31:36.170 INFO:tasks.rados.rados.0.smithi038.stdout:1550: finishing rollback tid 1 to smithi03829856-25 2024-03-19T11:31:36.170 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 25 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-03-19T11:31:36.170 INFO:tasks.rados.rados.0.smithi038.stdout:1551: expect (ObjNum 206 snap 22 seq_num 206) 2024-03-19T11:31:36.171 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-19T11:31:36.172 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:36.182+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:36.243 INFO:tasks.rados.rados.0.smithi038.stdout:1549: expect (ObjNum 213 snap 22 seq_num 213) 2024-03-19T11:31:36.284 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:36.296+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:36.330 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:36.331 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:31:36.331 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:31:36.331 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:36.345 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-19T11:31:36.389 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:36.400+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:36.473 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:36.482+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:36.496 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:36.496 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:31:36.496 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:31:36.496 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:31:36.496 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:36.508 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-19T11:31:36.574 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:36.582+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:36.640 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:36.640 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:31:36.640 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:31:36.640 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:31:36.640 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.640 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1067 4.0 4:01018511:::smithi03829856-19:head [write 477332~766101] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:31:36.640 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.950506+0000", 2024-03-19T11:31:36.640 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 82.696327095000001, 2024-03-19T11:31:36.640 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016279426, 2024-03-19T11:31:36.641 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:36.641 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:36.641 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:36.641 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:36.641 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:36.641 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1067 2024-03-19T11:31:36.641 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.641 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:36.641 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.641 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:36.641 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950506+0000", 2024-03-19T11:31:36.641 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.641 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.641 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.642 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:36.642 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950506+0000", 2024-03-19T11:31:36.642 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.642 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.642 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.642 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:36.642 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950509+0000", 2024-03-19T11:31:36.642 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.021e-06 2024-03-19T11:31:36.642 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.642 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.642 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:36.642 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950892+0000", 2024-03-19T11:31:36.642 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00038263300000000001 2024-03-19T11:31:36.642 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.643 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.643 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:36.643 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950912+0000", 2024-03-19T11:31:36.644 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0539000000000001e-05 2024-03-19T11:31:36.644 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.644 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.644 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:36.644 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950922+0000", 2024-03-19T11:31:36.644 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.577e-06 2024-03-19T11:31:36.644 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.644 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.644 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:36.644 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950982+0000", 2024-03-19T11:31:36.645 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0760000000000001e-05 2024-03-19T11:31:36.645 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.645 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.645 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:36.645 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.951058+0000", 2024-03-19T11:31:36.645 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5078000000000002e-05 2024-03-19T11:31:36.645 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.645 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.645 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:36.645 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.951173+0000", 2024-03-19T11:31:36.645 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011516200000000001 2024-03-19T11:31:36.645 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.645 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.645 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:36.646 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.955064+0000", 2024-03-19T11:31:36.646 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003890896 2024-03-19T11:31:36.646 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.646 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.646 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:36.646 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966670+0000", 2024-03-19T11:31:36.646 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011606795 2024-03-19T11:31:36.646 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.646 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.646 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:36.646 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966731+0000", 2024-03-19T11:31:36.646 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0528000000000001e-05 2024-03-19T11:31:36.646 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.646 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.647 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:36.647 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966785+0000", 2024-03-19T11:31:36.647 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4437000000000001e-05 2024-03-19T11:31:36.647 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.647 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:36.647 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.647 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.647 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.647 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(3.1 49/48 [PushOp(3:9205291d:::smithi03829856-50 low_tier:head, version: 28'36, data_included: [393216~851968,1703936~196608], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:9205291d:::smithi03829856-50 low_tier:head@28'36, size: 2823543, copy_subset: [(0, 2823543)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1900544, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:baab691a:::smithi03829856-2 low_tier:head, version: 28'37, data_included: [393216~524288,1507328~509479], data_size: 1033767, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:baab691a:::smithi03829856-2 low_tier:head@28'37, size: 2016807, copy_subset: [(0, 2016807)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2016807, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:98ce8426:::smithi03829856-4 low_tier:head, version: 28'38, data_included: [393216~720896,1638400~327680], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:98ce8426:::smithi03829856-4 low_tier:head@28'38, size: 2050421, copy_subset: [(0, 2050421)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1966080, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:a86be07b:::smithi03829856-44 low_tier:head, version: 28'41, data_included: [720896~655360,1769472~393216], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:a86be07b:::smithi03829856-44 low_tier:head@28'41, size: 2722701, copy_subset: [(0, 2722701)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2162688, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:bb56919e:::smithi03829856-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'44, data_included: [393216~786432,1703936~262144], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:bb56919e:::smithi03829856-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'44, size: 2806736, copy_subset: [(0, 2806736)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1966080, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:88687f71:::smithi03829856-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'45, data_included: [589824~655360,1638400~393216], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:88687f71:::smithi03829856-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'45, size: 2201684, copy_subset: [(0, 2201684)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2031616, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:b1f822cf:::smithi03829856-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'46, data_included: [720896~786432,1966080~262144], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:b1f822cf:::smithi03829856-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'46, size: 2369754, copy_subset: [(0, 2369754)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2228224, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:a5e8313a:::smithi03829856-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'47, data_included: [524288~720896,1769472~327680], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:a5e8313a:::smithi03829856-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'47, size: 2571438, copy_subset: [(0, 2571438)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:bca5ec64:::e3a26bf9230b1b95b8cf1a4cb01909236bf2bf3fd59b083b2291d278ab98264d:head, version: 28'48, data_included: [0~393216], data_size: 393216, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bca5ec64:::e3a26bf9230b1b95b8cf1a4cb01909236bf2bf3fd59b083b2291d278ab98264d:head@28'48, size: 533504, copy_subset: [(0, 533504)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 533504, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:36.647 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:26.948137+0000", 2024-03-19T11:31:36.647 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 69.698695880000002, 2024-03-19T11:31:36.647 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.036085578, 2024-03-19T11:31:36.648 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:36.648 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:36.648 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:36.648 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.648 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:36.648 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.948137+0000", 2024-03-19T11:31:36.648 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.648 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.648 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.648 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:36.648 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.948137+0000", 2024-03-19T11:31:36.648 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.648 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.648 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.648 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:36.649 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.948139+0000", 2024-03-19T11:31:36.649 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1399999999999998e-06 2024-03-19T11:31:36.649 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.649 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.649 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:36.649 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954288+0000", 2024-03-19T11:31:36.649 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0061487429999999999 2024-03-19T11:31:36.650 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.650 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.650 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:36.650 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954388+0000", 2024-03-19T11:31:36.650 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010022 2024-03-19T11:31:36.650 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.650 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.650 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:36.650 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954523+0000", 2024-03-19T11:31:36.650 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000134332 2024-03-19T11:31:36.650 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.650 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.650 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:36.651 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.955120+0000", 2024-03-19T11:31:36.651 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00059695799999999997 2024-03-19T11:31:36.651 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.651 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.651 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:36.651 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.955194+0000", 2024-03-19T11:31:36.651 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4077000000000002e-05 2024-03-19T11:31:36.651 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.651 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.651 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:36.651 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.984223+0000", 2024-03-19T11:31:36.651 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029029108000000001 2024-03-19T11:31:36.651 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.652 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:36.652 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.652 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.652 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.652 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(3.1 49/48 [PushOp(3:99a49d1b:::895a63407b6109b3cca9a6f5ef827134d56679bef42daadc5892269682c428ed:head, version: 32'49, data_included: [393216~388096], data_size: 388096, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:99a49d1b:::895a63407b6109b3cca9a6f5ef827134d56679bef42daadc5892269682c428ed:head@32'49, size: 781312, copy_subset: [(0, 781312)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 781312, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:36.652 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:26.954637+0000", 2024-03-19T11:31:36.652 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 69.692196263, 2024-03-19T11:31:36.652 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.030729057000000001, 2024-03-19T11:31:36.652 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:36.652 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:36.652 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:36.652 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.652 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:36.652 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954637+0000", 2024-03-19T11:31:36.653 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.653 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.653 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.653 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:36.653 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954637+0000", 2024-03-19T11:31:36.653 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.653 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.653 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.653 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:36.653 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954637+0000", 2024-03-19T11:31:36.653 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7299999999999995e-07 2024-03-19T11:31:36.654 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.654 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.654 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:36.654 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954873+0000", 2024-03-19T11:31:36.654 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023520299999999999 2024-03-19T11:31:36.654 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.654 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.654 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:36.655 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954897+0000", 2024-03-19T11:31:36.655 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4110000000000001e-05 2024-03-19T11:31:36.655 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.655 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.655 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:36.655 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954917+0000", 2024-03-19T11:31:36.655 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.067e-05 2024-03-19T11:31:36.655 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.655 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.655 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:36.655 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.984289+0000", 2024-03-19T11:31:36.655 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029371191000000001 2024-03-19T11:31:36.655 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.656 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.656 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:36.656 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.984303+0000", 2024-03-19T11:31:36.656 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4888999999999999e-05 2024-03-19T11:31:36.656 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.656 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.656 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:36.656 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.985366+0000", 2024-03-19T11:31:36.656 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001062321 2024-03-19T11:31:36.656 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.656 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:36.656 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.656 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.656 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.657 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1472 4.10 4:0dc3b91e:::smithi03829856-28:head [read 534528~391680] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-03-19T11:31:36.657 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:27.091748+0000", 2024-03-19T11:31:36.657 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 69.555085375000004, 2024-03-19T11:31:36.657 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.83574118500000005, 2024-03-19T11:31:36.657 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:36.657 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:36.657 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:36.657 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:36.657 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:36.657 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1472 2024-03-19T11:31:36.657 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.657 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:36.657 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.657 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:36.658 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091748+0000", 2024-03-19T11:31:36.658 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.658 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.658 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.658 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:36.658 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091748+0000", 2024-03-19T11:31:36.658 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.658 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.658 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.658 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:36.658 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091750+0000", 2024-03-19T11:31:36.659 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3130000000000001e-06 2024-03-19T11:31:36.659 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.659 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.659 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:36.659 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091760+0000", 2024-03-19T11:31:36.660 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0560999999999999e-05 2024-03-19T11:31:36.660 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.660 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.660 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:36.660 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091784+0000", 2024-03-19T11:31:36.660 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3915000000000001e-05 2024-03-19T11:31:36.660 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.660 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.660 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:36.660 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091797+0000", 2024-03-19T11:31:36.660 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2660000000000001e-05 2024-03-19T11:31:36.661 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.661 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.661 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:36.661 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091874+0000", 2024-03-19T11:31:36.661 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6757999999999999e-05 2024-03-19T11:31:36.661 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.661 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.661 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:36.661 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091882+0000", 2024-03-19T11:31:36.661 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5520000000000005e-06 2024-03-19T11:31:36.661 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.661 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.661 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:36.662 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.927033+0000", 2024-03-19T11:31:36.662 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.83515094099999998 2024-03-19T11:31:36.662 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.662 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.662 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:36.662 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.927250+0000", 2024-03-19T11:31:36.662 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00021675199999999999 2024-03-19T11:31:36.662 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.662 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.662 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:36.663 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.927489+0000", 2024-03-19T11:31:36.663 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023873300000000001 2024-03-19T11:31:36.663 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.663 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:36.663 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.663 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.663 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.663 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1474 4.10 4:0dc3b91e:::smithi03829856-28:head [assert-version v27,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-03-19T11:31:36.663 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:27.934139+0000", 2024-03-19T11:31:36.663 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 68.712693588999997, 2024-03-19T11:31:36.663 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024764884000000001, 2024-03-19T11:31:36.664 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:36.664 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:36.664 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:36.664 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:36.664 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:36.664 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1474 2024-03-19T11:31:36.664 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.664 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:36.665 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.665 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:36.665 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934139+0000", 2024-03-19T11:31:36.665 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.665 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.665 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.665 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:36.665 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934139+0000", 2024-03-19T11:31:36.665 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.665 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.665 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.665 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:36.665 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934142+0000", 2024-03-19T11:31:36.665 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5459999999999998e-06 2024-03-19T11:31:36.665 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.666 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.666 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:36.666 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934152+0000", 2024-03-19T11:31:36.666 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9399999999999997e-06 2024-03-19T11:31:36.666 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.666 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.666 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:36.666 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934173+0000", 2024-03-19T11:31:36.666 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1522000000000002e-05 2024-03-19T11:31:36.666 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.666 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.666 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:36.666 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934185+0000", 2024-03-19T11:31:36.666 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1946999999999999e-05 2024-03-19T11:31:36.667 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.667 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.667 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:36.667 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934282+0000", 2024-03-19T11:31:36.667 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6272000000000003e-05 2024-03-19T11:31:36.667 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.667 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.667 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:36.667 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934379+0000", 2024-03-19T11:31:36.667 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6966999999999995e-05 2024-03-19T11:31:36.667 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.667 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.667 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:36.667 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.955023+0000", 2024-03-19T11:31:36.668 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020644302 2024-03-19T11:31:36.668 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.668 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.668 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:36.668 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.955167+0000", 2024-03-19T11:31:36.668 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014404500000000001 2024-03-19T11:31:36.668 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.669 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.669 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1", 2024-03-19T11:31:36.669 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.957913+0000", 2024-03-19T11:31:36.669 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0027463779999999998 2024-03-19T11:31:36.669 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.669 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.669 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:36.669 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.958548+0000", 2024-03-19T11:31:36.669 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00063495700000000001 2024-03-19T11:31:36.669 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.670 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.670 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:36.670 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.958785+0000", 2024-03-19T11:31:36.670 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000236872 2024-03-19T11:31:36.670 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.671 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.671 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:36.671 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.958854+0000", 2024-03-19T11:31:36.671 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9099999999999999e-05 2024-03-19T11:31:36.671 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.671 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.671 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:36.671 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.958904+0000", 2024-03-19T11:31:36.672 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0036e-05 2024-03-19T11:31:36.672 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.672 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:36.672 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.672 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.672 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.672 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 75/74 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:7, version: 55'82, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@55'82, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 58'86, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@58'86, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [504210~968657,1572695~79919], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:12, version: 66'101, data_included: [440350~816629,1309629~231947], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:12@66'101, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:10, version: 70'112, data_included: [440350~816629,1309629~231947], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:10@70'112, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 70'113, data_included: [440350~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@70'113, size: 2605085, copy_subset: [(0, 2605085)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1488926, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:36.673 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.108941+0000", 2024-03-19T11:31:36.673 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 42.537891852000001, 2024-03-19T11:31:36.673 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.067446694000000001, 2024-03-19T11:31:36.673 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:36.673 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:36.673 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:36.673 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.673 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:36.673 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.108941+0000", 2024-03-19T11:31:36.673 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.673 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.673 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.673 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:36.674 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.108941+0000", 2024-03-19T11:31:36.674 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.674 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.674 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.674 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:36.674 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.108943+0000", 2024-03-19T11:31:36.674 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2259999999999999e-06 2024-03-19T11:31:36.675 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.675 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.675 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:36.675 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123892+0000", 2024-03-19T11:31:36.675 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014948655999999999 2024-03-19T11:31:36.676 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.676 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.676 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:36.676 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123993+0000", 2024-03-19T11:31:36.676 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000101236 2024-03-19T11:31:36.676 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.676 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.676 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:36.676 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124126+0000", 2024-03-19T11:31:36.676 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000133184 2024-03-19T11:31:36.677 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.677 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.677 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:36.677 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124655+0000", 2024-03-19T11:31:36.677 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00052849600000000004 2024-03-19T11:31:36.677 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.677 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.677 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:36.677 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124723+0000", 2024-03-19T11:31:36.677 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7797000000000002e-05 2024-03-19T11:31:36.677 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.677 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.677 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:36.678 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.176388+0000", 2024-03-19T11:31:36.678 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.051665098999999999 2024-03-19T11:31:36.678 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.678 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:36.678 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.678 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.678 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.678 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.21 e75,74 4:8401c1f0:::smithi03829856-43:8,46'66,4:866f3dcc:::smithi03829856-31:2,47'73)", 2024-03-19T11:31:36.678 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.124226+0000", 2024-03-19T11:31:36.678 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 42.522607196000003, 2024-03-19T11:31:36.678 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.052217913999999997, 2024-03-19T11:31:36.679 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:36.679 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:36.679 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:36.679 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.679 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:36.679 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124226+0000", 2024-03-19T11:31:36.679 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.679 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.680 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.680 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:36.681 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124226+0000", 2024-03-19T11:31:36.681 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.681 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.681 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.681 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:36.681 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124227+0000", 2024-03-19T11:31:36.681 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.477e-06 2024-03-19T11:31:36.681 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.681 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.681 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:36.681 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124240+0000", 2024-03-19T11:31:36.682 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2379e-05 2024-03-19T11:31:36.682 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.682 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.682 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:36.682 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124264+0000", 2024-03-19T11:31:36.682 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4224999999999999e-05 2024-03-19T11:31:36.682 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.682 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.682 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:36.682 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124287+0000", 2024-03-19T11:31:36.682 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3557000000000002e-05 2024-03-19T11:31:36.682 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.682 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.682 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:36.682 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.176274+0000", 2024-03-19T11:31:36.683 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.051986248999999998 2024-03-19T11:31:36.683 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.683 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.683 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:36.683 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.176285+0000", 2024-03-19T11:31:36.683 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0818999999999999e-05 2024-03-19T11:31:36.683 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.683 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.683 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:36.683 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.176444+0000", 2024-03-19T11:31:36.683 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015920800000000001 2024-03-19T11:31:36.683 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.683 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:36.683 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.683 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.684 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.684 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 75/74 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:7, version: 55'82, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@55'82, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 58'86, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@58'86, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [1652614~973634,2888511~52681,3182305~22261], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3204566, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:12, version: 66'101, data_included: [1541576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:12@66'101, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:10, version: 70'112, data_included: [1541576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:10@70'112, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 70'113, data_included: [1488926~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@70'113, size: 2605085, copy_subset: [(0, 2605085)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2537502, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1488926, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:36.684 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.195615+0000", 2024-03-19T11:31:36.684 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 42.451218298999997, 2024-03-19T11:31:36.684 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.036930510999999999, 2024-03-19T11:31:36.684 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:36.684 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:36.685 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:36.685 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.685 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:36.685 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.195615+0000", 2024-03-19T11:31:36.685 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.685 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.685 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.685 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:36.685 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.195615+0000", 2024-03-19T11:31:36.685 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.685 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.685 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.685 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:36.685 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.195617+0000", 2024-03-19T11:31:36.685 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0490000000000002e-06 2024-03-19T11:31:36.686 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.686 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.686 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:36.686 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204654+0000", 2024-03-19T11:31:36.686 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0090370339999999993 2024-03-19T11:31:36.686 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.686 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.686 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:36.686 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204730+0000", 2024-03-19T11:31:36.686 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5939000000000006e-05 2024-03-19T11:31:36.686 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.686 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.686 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:36.686 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204844+0000", 2024-03-19T11:31:36.687 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000114066 2024-03-19T11:31:36.687 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.687 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.687 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:36.687 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.205232+0000", 2024-03-19T11:31:36.687 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000388045 2024-03-19T11:31:36.687 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.687 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.687 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:36.687 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.205277+0000", 2024-03-19T11:31:36.687 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5071999999999998e-05 2024-03-19T11:31:36.687 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.687 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.687 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:36.687 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.232545+0000", 2024-03-19T11:31:36.688 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027268305999999999 2024-03-19T11:31:36.688 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.688 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:36.688 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.688 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.688 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.688 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.21 e75,74 4:866f3dcc:::smithi03829856-31:c,70'114)", 2024-03-19T11:31:36.688 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.204941+0000", 2024-03-19T11:31:36.688 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 42.441891673999997, 2024-03-19T11:31:36.688 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.028797390999999999, 2024-03-19T11:31:36.688 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:36.688 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:36.688 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:36.688 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.688 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:36.688 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204941+0000", 2024-03-19T11:31:36.689 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.689 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.689 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.689 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:36.689 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204941+0000", 2024-03-19T11:31:36.689 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.689 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.689 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.689 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:36.689 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204943+0000", 2024-03-19T11:31:36.689 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.35e-06 2024-03-19T11:31:36.689 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.689 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.689 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:36.689 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204950+0000", 2024-03-19T11:31:36.690 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.374e-06 2024-03-19T11:31:36.690 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.690 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.690 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:36.690 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204965+0000", 2024-03-19T11:31:36.690 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5169e-05 2024-03-19T11:31:36.690 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.690 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.690 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:36.690 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204979+0000", 2024-03-19T11:31:36.690 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4260000000000001e-05 2024-03-19T11:31:36.690 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.690 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.690 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:36.690 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.232607+0000", 2024-03-19T11:31:36.691 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027627518 2024-03-19T11:31:36.691 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.691 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.691 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:36.691 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.232626+0000", 2024-03-19T11:31:36.691 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9471999999999999e-05 2024-03-19T11:31:36.691 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.691 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.691 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:36.691 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.233739+0000", 2024-03-19T11:31:36.691 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011122479999999999 2024-03-19T11:31:36.691 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.691 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:36.691 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.691 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.692 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.692 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2789 3.2 3:48367551:::ca87bd16019841aaa831bb522ee079e5f4e6a1ea67fbcb51e5f89263b6c5d8f6:head [write 0~3280896] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-03-19T11:31:36.692 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:12.340289+0000", 2024-03-19T11:31:36.692 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 24.306543957999999, 2024-03-19T11:31:36.692 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016291298999999999, 2024-03-19T11:31:36.692 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:36.692 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:36.692 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:36.692 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:36.692 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:36.692 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2789 2024-03-19T11:31:36.692 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.692 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:36.692 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.692 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:36.692 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.340289+0000", 2024-03-19T11:31:36.693 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.693 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.693 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.693 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:36.693 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.340289+0000", 2024-03-19T11:31:36.693 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.693 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.693 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.693 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:36.693 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.340291+0000", 2024-03-19T11:31:36.693 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1270000000000001e-06 2024-03-19T11:31:36.693 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.693 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.693 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:36.694 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.341426+0000", 2024-03-19T11:31:36.694 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011348459999999999 2024-03-19T11:31:36.694 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.694 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.694 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:36.694 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.341454+0000", 2024-03-19T11:31:36.694 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8204e-05 2024-03-19T11:31:36.694 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.694 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.694 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:36.694 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.341467+0000", 2024-03-19T11:31:36.694 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2700999999999999e-05 2024-03-19T11:31:36.694 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.694 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.694 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:36.695 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.341623+0000", 2024-03-19T11:31:36.695 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015600100000000001 2024-03-19T11:31:36.695 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.695 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.695 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:36.695 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.342034+0000", 2024-03-19T11:31:36.695 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000411247 2024-03-19T11:31:36.695 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.695 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.695 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:31:36.695 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.342443+0000", 2024-03-19T11:31:36.695 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00040901400000000002 2024-03-19T11:31:36.695 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.695 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.695 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:36.696 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.356159+0000", 2024-03-19T11:31:36.696 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013716058999999999 2024-03-19T11:31:36.696 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.696 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.696 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:36.696 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.356462+0000", 2024-03-19T11:31:36.696 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000302594 2024-03-19T11:31:36.696 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.696 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.696 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:36.696 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.356530+0000", 2024-03-19T11:31:36.696 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7852e-05 2024-03-19T11:31:36.696 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.696 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.696 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:36.696 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.356580+0000", 2024-03-19T11:31:36.697 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0654e-05 2024-03-19T11:31:36.697 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.697 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:36.697 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.697 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.697 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.697 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3046 4.19 4:9c74b127:::smithi03829856-33:head [write 400066~716205,stat] snapc 1f=[1f,1e,1d,1c,1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-03-19T11:31:36.697 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:22.147457+0000", 2024-03-19T11:31:36.697 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 14.499375673999999, 2024-03-19T11:31:36.697 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016153588, 2024-03-19T11:31:36.697 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:36.697 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:36.697 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:36.698 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:36.698 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:36.698 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3046 2024-03-19T11:31:36.698 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.698 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:36.698 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.698 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:36.698 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.147457+0000", 2024-03-19T11:31:36.698 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.698 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.698 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.698 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:36.698 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.147457+0000", 2024-03-19T11:31:36.698 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.698 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.699 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.699 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:36.699 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.147459+0000", 2024-03-19T11:31:36.699 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1160000000000002e-06 2024-03-19T11:31:36.699 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.699 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.699 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:36.699 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.148083+0000", 2024-03-19T11:31:36.699 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00062361999999999997 2024-03-19T11:31:36.699 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.699 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.699 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:36.699 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.148100+0000", 2024-03-19T11:31:36.699 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6524999999999998e-05 2024-03-19T11:31:36.699 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.700 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.700 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:36.700 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.148110+0000", 2024-03-19T11:31:36.700 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0093e-05 2024-03-19T11:31:36.700 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.700 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.700 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:36.700 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.148197+0000", 2024-03-19T11:31:36.700 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7639000000000006e-05 2024-03-19T11:31:36.700 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.700 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.700 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:36.700 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.148359+0000", 2024-03-19T11:31:36.700 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000161953 2024-03-19T11:31:36.700 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.701 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.701 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 2", 2024-03-19T11:31:36.701 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.149074+0000", 2024-03-19T11:31:36.701 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00071444100000000001 2024-03-19T11:31:36.701 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.701 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.701 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:36.701 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.158856+0000", 2024-03-19T11:31:36.701 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0097818169999999999 2024-03-19T11:31:36.701 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.701 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.701 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:36.701 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.163525+0000", 2024-03-19T11:31:36.701 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046690270000000001 2024-03-19T11:31:36.701 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.702 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.702 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:36.702 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.163569+0000", 2024-03-19T11:31:36.702 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4279000000000001e-05 2024-03-19T11:31:36.702 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.702 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.702 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:36.702 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.163611+0000", 2024-03-19T11:31:36.702 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2077999999999999e-05 2024-03-19T11:31:36.702 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.702 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:36.702 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.702 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.702 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.702 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3228 4.19 4:9bd2ffb1:::smithi03829856-30:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e109)", 2024-03-19T11:31:36.703 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:28.084149+0000", 2024-03-19T11:31:36.703 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.5626836950000005, 2024-03-19T11:31:36.703 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.049907076000000002, 2024-03-19T11:31:36.703 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:36.703 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:36.703 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:36.703 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:36.703 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:36.703 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3228 2024-03-19T11:31:36.703 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.703 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:36.703 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.703 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:36.703 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.084149+0000", 2024-03-19T11:31:36.703 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.704 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.704 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.704 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:36.704 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.084149+0000", 2024-03-19T11:31:36.704 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.704 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.704 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.704 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:36.704 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.084152+0000", 2024-03-19T11:31:36.704 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2409999999999998e-06 2024-03-19T11:31:36.704 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.704 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.704 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:36.704 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.084161+0000", 2024-03-19T11:31:36.704 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9609999999999994e-06 2024-03-19T11:31:36.704 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.705 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.705 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:36.705 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.084181+0000", 2024-03-19T11:31:36.705 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0786999999999999e-05 2024-03-19T11:31:36.705 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.705 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.705 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:36.705 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.084194+0000", 2024-03-19T11:31:36.705 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2824e-05 2024-03-19T11:31:36.705 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.705 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.705 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:36.705 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.086656+0000", 2024-03-19T11:31:36.705 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0024622480000000002 2024-03-19T11:31:36.705 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.706 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.706 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:36.706 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.086894+0000", 2024-03-19T11:31:36.706 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000237414 2024-03-19T11:31:36.706 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.706 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.706 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:36.706 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.134056+0000", 2024-03-19T11:31:36.706 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.047162600999999998 2024-03-19T11:31:36.706 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.706 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:36.706 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.706 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.706 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.706 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3361 4.19 4:9ef41243:::smithi03829856-5:head [write 420241~546489,stat] snapc 23=[23,22,21,1f,1e,1d,1c,1b,1a,18,17] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-03-19T11:31:36.707 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:32.244085+0000", 2024-03-19T11:31:36.707 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.4027482969999996, 2024-03-19T11:31:36.707 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025127221000000002, 2024-03-19T11:31:36.707 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:36.707 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:36.707 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:36.707 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:36.707 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:36.707 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3361 2024-03-19T11:31:36.707 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.707 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:36.707 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.707 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:36.707 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.244085+0000", 2024-03-19T11:31:36.707 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.708 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.708 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.708 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:36.708 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.244085+0000", 2024-03-19T11:31:36.708 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.708 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.708 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.708 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:36.708 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.244088+0000", 2024-03-19T11:31:36.708 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5779999999999999e-06 2024-03-19T11:31:36.708 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.708 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.708 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:36.708 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.244785+0000", 2024-03-19T11:31:36.708 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00069645700000000004 2024-03-19T11:31:36.709 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.709 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.709 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:36.709 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.244833+0000", 2024-03-19T11:31:36.709 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7902000000000003e-05 2024-03-19T11:31:36.709 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.709 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.709 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:36.709 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.244857+0000", 2024-03-19T11:31:36.709 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4301e-05 2024-03-19T11:31:36.709 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.709 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.709 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:36.709 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.245014+0000", 2024-03-19T11:31:36.709 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000157014 2024-03-19T11:31:36.709 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.710 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.710 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-03-19T11:31:36.710 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.245038+0000", 2024-03-19T11:31:36.710 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3655e-05 2024-03-19T11:31:36.710 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.710 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.710 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:36.710 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.251603+0000", 2024-03-19T11:31:36.710 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0065650650000000001 2024-03-19T11:31:36.710 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.710 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.710 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:36.710 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.251796+0000", 2024-03-19T11:31:36.710 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019291100000000001 2024-03-19T11:31:36.710 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.711 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.711 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 2", 2024-03-19T11:31:36.711 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.252201+0000", 2024-03-19T11:31:36.711 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00040590400000000001 2024-03-19T11:31:36.711 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.711 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.711 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:36.711 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.264409+0000", 2024-03-19T11:31:36.711 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012207314 2024-03-19T11:31:36.711 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.711 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.711 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:36.711 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.269042+0000", 2024-03-19T11:31:36.711 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046334949999999996 2024-03-19T11:31:36.711 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.712 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.712 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:36.712 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.269135+0000", 2024-03-19T11:31:36.712 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2310000000000002e-05 2024-03-19T11:31:36.712 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.712 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.712 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:36.712 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.269212+0000", 2024-03-19T11:31:36.712 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7315000000000001e-05 2024-03-19T11:31:36.712 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.712 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:36.712 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.712 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.712 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.712 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3399 4.21 4:866f3dcc:::smithi03829856-31:head [write 504276~639087,stat] snapc 23=[23,22,21,1f,1e,1d,1c,1b,1a,18,17] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-03-19T11:31:36.713 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:33.034450+0000", 2024-03-19T11:31:36.713 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.612382685, 2024-03-19T11:31:36.713 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026707371000000001, 2024-03-19T11:31:36.713 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:36.713 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:36.713 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:36.713 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:36.713 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:36.713 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3399 2024-03-19T11:31:36.713 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.713 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:36.713 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.713 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:36.713 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.034450+0000", 2024-03-19T11:31:36.713 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.714 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.714 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.714 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:36.714 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.034450+0000", 2024-03-19T11:31:36.714 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.714 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.714 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.714 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:36.714 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.034452+0000", 2024-03-19T11:31:36.714 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.877e-06 2024-03-19T11:31:36.714 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.714 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.714 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:36.714 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.034786+0000", 2024-03-19T11:31:36.715 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00033416099999999997 2024-03-19T11:31:36.715 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.715 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.715 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:36.715 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.034813+0000", 2024-03-19T11:31:36.715 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6792e-05 2024-03-19T11:31:36.715 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.715 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.715 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:36.715 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.034828+0000", 2024-03-19T11:31:36.715 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5285999999999999e-05 2024-03-19T11:31:36.715 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.715 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.715 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:36.715 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.034947+0000", 2024-03-19T11:31:36.715 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011857 2024-03-19T11:31:36.716 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.716 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.716 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:36.716 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.035325+0000", 2024-03-19T11:31:36.716 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037801499999999999 2024-03-19T11:31:36.716 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.716 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.716 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1", 2024-03-19T11:31:36.716 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.035660+0000", 2024-03-19T11:31:36.716 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00033508499999999999 2024-03-19T11:31:36.716 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.716 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.716 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:36.716 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.043413+0000", 2024-03-19T11:31:36.716 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077528110000000001 2024-03-19T11:31:36.717 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.717 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.717 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:36.717 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.060284+0000", 2024-03-19T11:31:36.717 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016870823 2024-03-19T11:31:36.717 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.717 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.717 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:36.717 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.061003+0000", 2024-03-19T11:31:36.717 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000719687 2024-03-19T11:31:36.717 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.717 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.717 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:36.717 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.061158+0000", 2024-03-19T11:31:36.717 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015426400000000001 2024-03-19T11:31:36.718 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.718 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:36.718 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.718 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.718 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.718 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3400 4.21 4:866f3dcc:::smithi03829856-31:head [write 1666233~695410] snapc 23=[23,22,21,1f,1e,1d,1c,1b,1a,18,17] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-03-19T11:31:36.718 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:33.043656+0000", 2024-03-19T11:31:36.718 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.6031774040000002, 2024-03-19T11:31:36.718 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018155691000000002, 2024-03-19T11:31:36.718 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:36.718 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:36.718 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:36.718 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:36.718 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:36.718 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3400 2024-03-19T11:31:36.719 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.719 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:36.719 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.719 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:36.719 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.043656+0000", 2024-03-19T11:31:36.719 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.719 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.719 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.719 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:36.719 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.043656+0000", 2024-03-19T11:31:36.719 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.719 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.719 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.719 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:36.719 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.043658+0000", 2024-03-19T11:31:36.720 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5239999999999999e-06 2024-03-19T11:31:36.720 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.720 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.720 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:36.720 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044163+0000", 2024-03-19T11:31:36.720 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00050451399999999996 2024-03-19T11:31:36.720 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.720 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.720 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:36.720 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044205+0000", 2024-03-19T11:31:36.720 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2128000000000001e-05 2024-03-19T11:31:36.720 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.720 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.720 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:36.720 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044229+0000", 2024-03-19T11:31:36.720 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4097999999999999e-05 2024-03-19T11:31:36.721 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.721 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.721 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:36.721 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044414+0000", 2024-03-19T11:31:36.721 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00018484600000000001 2024-03-19T11:31:36.721 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.721 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.721 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:36.721 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044628+0000", 2024-03-19T11:31:36.721 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00021459499999999999 2024-03-19T11:31:36.721 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.721 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.721 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1", 2024-03-19T11:31:36.721 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044909+0000", 2024-03-19T11:31:36.721 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00028030399999999999 2024-03-19T11:31:36.722 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.722 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.722 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:36.722 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.061219+0000", 2024-03-19T11:31:36.722 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016310495000000001 2024-03-19T11:31:36.722 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.722 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.722 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:36.722 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.061362+0000", 2024-03-19T11:31:36.722 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014293500000000001 2024-03-19T11:31:36.722 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.722 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.722 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:36.722 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.061671+0000", 2024-03-19T11:31:36.722 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030884599999999999 2024-03-19T11:31:36.723 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.723 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.723 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:36.723 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.061811+0000", 2024-03-19T11:31:36.723 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014040600000000001 2024-03-19T11:31:36.723 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.723 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:36.723 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.723 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.723 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.723 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3401 4.21 4:866f3dcc:::smithi03829856-31:head [setxattr _header (57) in=7b,truncate 2504276] snapc 23=[23,22,21,1f,1e,1d,1c,1b,1a,18,17] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-03-19T11:31:36.723 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:33.044315+0000", 2024-03-19T11:31:36.723 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.602518458, 2024-03-19T11:31:36.723 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.021591882, 2024-03-19T11:31:36.723 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:36.723 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "waiting for sub ops", 2024-03-19T11:31:36.724 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:36.724 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:36.724 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:36.724 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3401 2024-03-19T11:31:36.724 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.724 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:36.724 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.724 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:36.724 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044315+0000", 2024-03-19T11:31:36.724 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.724 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.724 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.724 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:36.724 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044315+0000", 2024-03-19T11:31:36.725 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.725 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.725 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.725 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:36.725 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044317+0000", 2024-03-19T11:31:36.725 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.097e-06 2024-03-19T11:31:36.725 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.725 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.725 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:36.725 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044328+0000", 2024-03-19T11:31:36.725 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1362000000000001e-05 2024-03-19T11:31:36.725 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.725 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.725 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:36.725 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044354+0000", 2024-03-19T11:31:36.725 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6268999999999999e-05 2024-03-19T11:31:36.726 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.726 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.726 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:36.726 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044371+0000", 2024-03-19T11:31:36.726 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7028999999999999e-05 2024-03-19T11:31:36.726 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.726 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.726 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:36.726 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.052493+0000", 2024-03-19T11:31:36.726 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0081214270000000005 2024-03-19T11:31:36.726 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.726 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.726 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:36.726 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.052874+0000", 2024-03-19T11:31:36.727 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000381665 2024-03-19T11:31:36.727 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.727 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.727 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1", 2024-03-19T11:31:36.727 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.053343+0000", 2024-03-19T11:31:36.727 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00046883 2024-03-19T11:31:36.727 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.727 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.727 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:36.727 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.061993+0000", 2024-03-19T11:31:36.727 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0086501479999999999 2024-03-19T11:31:36.727 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.727 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.727 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:36.727 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.065906+0000", 2024-03-19T11:31:36.728 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0039130550000000004 2024-03-19T11:31:36.728 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.728 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:36.728 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.728 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.728 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.728 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3402 4.21 4:866f3dcc:::smithi03829856-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e112)", 2024-03-19T11:31:36.728 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:33.044411+0000", 2024-03-19T11:31:36.728 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.6024218169999997, 2024-03-19T11:31:36.728 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.034955695000000002, 2024-03-19T11:31:36.728 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:36.728 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-03-19T11:31:36.728 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:36.728 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:36.728 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:36.729 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3402 2024-03-19T11:31:36.729 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.729 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:36.729 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.729 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:36.729 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044411+0000", 2024-03-19T11:31:36.729 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.729 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.729 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.729 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:36.729 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044411+0000", 2024-03-19T11:31:36.729 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.729 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.729 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.729 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:36.730 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044413+0000", 2024-03-19T11:31:36.730 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.517e-06 2024-03-19T11:31:36.730 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.730 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.730 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:36.730 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044420+0000", 2024-03-19T11:31:36.730 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3799999999999996e-06 2024-03-19T11:31:36.730 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.730 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.730 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:36.730 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044466+0000", 2024-03-19T11:31:36.730 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5713000000000001e-05 2024-03-19T11:31:36.730 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.730 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.730 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:36.731 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044484+0000", 2024-03-19T11:31:36.731 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8128999999999999e-05 2024-03-19T11:31:36.731 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.731 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.731 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:36.731 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.055330+0000", 2024-03-19T11:31:36.731 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01084574 2024-03-19T11:31:36.731 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.731 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.731 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-03-19T11:31:36.731 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.055450+0000", 2024-03-19T11:31:36.731 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000120104 2024-03-19T11:31:36.731 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.731 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.731 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:36.732 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.079347+0000", 2024-03-19T11:31:36.732 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.023897049 2024-03-19T11:31:36.732 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.732 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.732 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:36.732 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.079367+0000", 2024-03-19T11:31:36.732 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0063000000000001e-05 2024-03-19T11:31:36.732 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.732 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:36.732 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.732 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.732 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.732 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3403 4.19 4:9b248e68:::smithi03829856-50:head [sparse-read 0~1696039,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e112)", 2024-03-19T11:31:36.732 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:33.044524+0000", 2024-03-19T11:31:36.732 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.6023088329999999, 2024-03-19T11:31:36.732 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016698992999999999, 2024-03-19T11:31:36.733 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:36.733 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:36.733 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:36.733 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:36.733 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:36.733 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3403 2024-03-19T11:31:36.733 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.733 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:36.733 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.733 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:36.733 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044524+0000", 2024-03-19T11:31:36.733 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.733 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.733 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.733 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:36.734 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044524+0000", 2024-03-19T11:31:36.734 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.734 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.734 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.734 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:36.734 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044526+0000", 2024-03-19T11:31:36.734 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4470000000000001e-06 2024-03-19T11:31:36.734 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.734 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.734 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:36.734 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044535+0000", 2024-03-19T11:31:36.734 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7450000000000007e-06 2024-03-19T11:31:36.734 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.734 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.735 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:36.735 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044562+0000", 2024-03-19T11:31:36.735 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6548999999999999e-05 2024-03-19T11:31:36.735 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.735 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.735 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:36.735 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044581+0000", 2024-03-19T11:31:36.735 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9468999999999999e-05 2024-03-19T11:31:36.735 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.735 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.735 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:36.735 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.055503+0000", 2024-03-19T11:31:36.735 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010921461 2024-03-19T11:31:36.735 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.735 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.735 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:36.736 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.055616+0000", 2024-03-19T11:31:36.736 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011270899999999999 2024-03-19T11:31:36.736 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.736 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.736 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:36.736 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.061223+0000", 2024-03-19T11:31:36.736 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005607613 2024-03-19T11:31:36.736 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.736 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:36.736 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.736 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.736 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.736 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3404 4.19 4:9b248e68:::smithi03829856-50:head [sparse-read 0~1696039] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e112)", 2024-03-19T11:31:36.736 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:33.044604+0000", 2024-03-19T11:31:36.736 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.6022285109999999, 2024-03-19T11:31:36.737 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017390282, 2024-03-19T11:31:36.737 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:36.737 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:36.737 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:36.737 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:36.737 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:36.737 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3404 2024-03-19T11:31:36.737 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.737 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:36.737 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.737 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:36.737 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044604+0000", 2024-03-19T11:31:36.737 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.737 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.738 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.738 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:36.738 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044604+0000", 2024-03-19T11:31:36.738 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.738 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.738 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.738 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:36.738 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044606+0000", 2024-03-19T11:31:36.738 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.393e-06 2024-03-19T11:31:36.738 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.738 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.738 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:36.738 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044616+0000", 2024-03-19T11:31:36.738 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0322999999999999e-05 2024-03-19T11:31:36.738 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.739 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.739 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:36.739 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044636+0000", 2024-03-19T11:31:36.739 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9819000000000001e-05 2024-03-19T11:31:36.739 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.739 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.739 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:36.739 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044653+0000", 2024-03-19T11:31:36.739 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7232999999999999e-05 2024-03-19T11:31:36.739 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.739 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.739 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:36.739 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.061515+0000", 2024-03-19T11:31:36.739 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016861322000000002 2024-03-19T11:31:36.739 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.739 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.740 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:36.740 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.061645+0000", 2024-03-19T11:31:36.740 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013014999999999999 2024-03-19T11:31:36.740 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.740 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.740 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:36.740 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.061995+0000", 2024-03-19T11:31:36.740 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00035004200000000001 2024-03-19T11:31:36.740 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.740 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:36.740 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.740 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.740 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.740 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3405 4.19 4:9b248e68:::smithi03829856-50:head [sparse-read 0~1696039] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e112)", 2024-03-19T11:31:36.740 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:33.044707+0000", 2024-03-19T11:31:36.741 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.6021260829999999, 2024-03-19T11:31:36.741 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018552987999999999, 2024-03-19T11:31:36.741 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:36.741 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:36.741 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:36.741 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:36.741 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:36.741 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3405 2024-03-19T11:31:36.741 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.741 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:36.741 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.741 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:36.741 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044707+0000", 2024-03-19T11:31:36.741 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.741 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.741 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.742 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:36.742 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044707+0000", 2024-03-19T11:31:36.742 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:36.742 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.742 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.742 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:36.742 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044708+0000", 2024-03-19T11:31:36.742 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.356e-06 2024-03-19T11:31:36.742 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.742 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.742 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:36.742 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044715+0000", 2024-03-19T11:31:36.742 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6900000000000003e-06 2024-03-19T11:31:36.742 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.742 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.743 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:36.743 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044736+0000", 2024-03-19T11:31:36.743 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0948999999999999e-05 2024-03-19T11:31:36.743 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.743 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.743 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:36.743 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044751+0000", 2024-03-19T11:31:36.743 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5389000000000001e-05 2024-03-19T11:31:36.743 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.743 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.743 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:36.743 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.062070+0000", 2024-03-19T11:31:36.743 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017318291 2024-03-19T11:31:36.743 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.743 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.744 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:36.744 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.062309+0000", 2024-03-19T11:31:36.744 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000239573 2024-03-19T11:31:36.744 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:36.744 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:36.744 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:36.744 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.063260+0000", 2024-03-19T11:31:36.744 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00095074000000000005 2024-03-19T11:31:36.744 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.744 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:36.744 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.744 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:36.744 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:36.744 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:36.746 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-19T11:31:36.746 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:36.754+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:36.856 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:36.866+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:36.884 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:36.884 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:31:36.884 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:31:36.884 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:36.898 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-19T11:31:36.951 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:36.958+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:37.050 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:37.060+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:37.069 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-03-19T11:31:37.081 INFO:tasks.rados.rados.0.smithi038.stdout:1548: done (4 left) 2024-03-19T11:31:37.081 INFO:tasks.rados.rados.0.smithi038.stdout:1549: done (3 left) 2024-03-19T11:31:37.081 INFO:tasks.rados.rados.0.smithi038.stdout:1550: done (2 left) 2024-03-19T11:31:37.081 INFO:tasks.rados.rados.0.smithi038.stdout:1551: done (1 left) 2024-03-19T11:31:37.081 INFO:tasks.rados.rados.0.smithi038.stdout:1552: done (0 left) 2024-03-19T11:31:37.081 INFO:tasks.rados.rados.0.smithi038.stdout:1553: tier_promote oid 36 2024-03-19T11:31:37.081 INFO:tasks.rados.rados.0.smithi038.stdout:1554: read oid 26 snap -1 2024-03-19T11:31:37.081 INFO:tasks.rados.rados.0.smithi038.stdout:1554: expect (ObjNum 267 snap 34 seq_num 267) 2024-03-19T11:31:37.081 INFO:tasks.rados.rados.0.smithi038.stdout:1555: read oid 17 snap -1 2024-03-19T11:31:37.081 INFO:tasks.rados.rados.0.smithi038.stdout:1555: expect (ObjNum 213 snap 22 seq_num 213) 2024-03-19T11:31:37.081 INFO:tasks.rados.rados.0.smithi038.stdout:1556: tier_evict oid 37 2024-03-19T11:31:37.082 INFO:tasks.rados.rados.0.smithi038.stdout:1556: tier_evict oid 37 snap -1 2024-03-19T11:31:37.082 INFO:tasks.rados.rados.0.smithi038.stdout:1557: write (excl) oid 47 current snap is 36 2024-03-19T11:31:37.082 INFO:tasks.rados.rados.0.smithi038.stdout:1557: seq_num 276 ranges {638732=557432,1617736=710086} 2024-03-19T11:31:37.082 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:37.088+0000 7f6a7f9da640 -1 osd.1 116 handle_osd_map injecting map crc failure 2024-03-19T11:31:37.082 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:37.090+0000 7f143d8b8640 -1 osd.5 116 handle_osd_map injecting map crc failure 2024-03-19T11:31:37.083 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:37.084 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:31:37.084 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:31:37.084 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:31:37.084 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:37.092 INFO:tasks.rados.rados.0.smithi038.stdout:1557: writing smithi03829856-47 from 638732 to 1196164 tid 1 2024-03-19T11:31:37.102 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-19T11:31:37.103 INFO:tasks.rados.rados.0.smithi038.stdout:1557: writing smithi03829856-47 from 1617736 to 2327822 tid 2 2024-03-19T11:31:37.103 INFO:tasks.rados.rados.0.smithi038.stdout:1558: read oid 46 snap -1 2024-03-19T11:31:37.103 INFO:tasks.rados.rados.0.smithi038.stdout:1558: expect (ObjNum 143 snap 15 seq_num 143) 2024-03-19T11:31:37.103 INFO:tasks.rados.rados.0.smithi038.stdout:1559: set_chunk oid 18 target oid 2024-03-19T11:31:37.110 INFO:tasks.rados.rados.0.smithi038.stdout:1559: set_chunk oid 18 offset: 2212352 length: 228864 target oid d484036e18b7b9f7ba9cc695d7e16916a79d4e5b91504f668174b2d43fc861fa offset: 0 snap 25 2024-03-19T11:31:37.110 INFO:tasks.rados.rados.0.smithi038.stdout:1560: read oid 28 snap 32 2024-03-19T11:31:37.110 INFO:tasks.rados.rados.0.smithi038.stdout:1560: expect (ObjNum 87 snap 8 seq_num 87) 2024-03-19T11:31:37.110 INFO:tasks.rados.rados.0.smithi038.stdout:1561: read oid 33 snap -1 2024-03-19T11:31:37.110 INFO:tasks.rados.rados.0.smithi038.stdout:1561: expect (ObjNum 273 snap 34 seq_num 273) 2024-03-19T11:31:37.110 INFO:tasks.rados.rados.0.smithi038.stdout:1562: write (excl) oid 5 current snap is 36 2024-03-19T11:31:37.110 INFO:tasks.rados.rados.0.smithi038.stdout:1556: got (22) Invalid argument 2024-03-19T11:31:37.111 INFO:tasks.rados.rados.0.smithi038.stdout:1562: seq_num 277 ranges {655539=632681,2076218=570097} 2024-03-19T11:31:37.119 INFO:tasks.rados.rados.0.smithi038.stdout:1562: writing smithi03829856-5 from 655539 to 1288220 tid 1 2024-03-19T11:31:37.133 INFO:tasks.rados.rados.0.smithi038.stdout:1562: writing smithi03829856-5 from 2076218 to 2646315 tid 2 2024-03-19T11:31:37.133 INFO:tasks.rados.rados.0.smithi038.stdout:1556: done (9 left) 2024-03-19T11:31:37.133 INFO:tasks.rados.rados.0.smithi038.stdout:1563: read oid 15 snap -1 2024-03-19T11:31:37.133 INFO:tasks.rados.rados.0.smithi038.stdout:1563: expect (ObjNum 209 snap 22 seq_num 209) 2024-03-19T11:31:37.133 INFO:tasks.rados.rados.0.smithi038.stdout:1564: snap_create 2024-03-19T11:31:37.133 INFO:tasks.rados.rados.0.smithi038.stdout:1553: got (0) Success 2024-03-19T11:31:37.133 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 36 v 127 (ObjNum 266 snap 34 seq_num 266) dirty exists 2024-03-19T11:31:37.134 INFO:tasks.rados.rados.0.smithi038.stdout:1554: expect (ObjNum 267 snap 34 seq_num 267) 2024-03-19T11:31:37.151 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:37.160+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:37.244 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:37.244 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:31:37.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:31:37.244 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:31:37.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.245 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:851 4.9 4:953e720b:::smithi03829856-45:head [write 591613~454162,stat] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:31:37.245 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.763514+0000", 2024-03-19T11:31:37.245 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 88.486181619000007, 2024-03-19T11:31:37.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027825888999999999, 2024-03-19T11:31:37.245 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:37.245 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:37.245 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:37.246 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:37.246 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:37.246 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 851 2024-03-19T11:31:37.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.246 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:37.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:37.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763514+0000", 2024-03-19T11:31:37.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:37.247 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763514+0000", 2024-03-19T11:31:37.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.247 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:37.247 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763517+0000", 2024-03-19T11:31:37.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8499999999999998e-06 2024-03-19T11:31:37.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.247 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:37.248 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.764714+0000", 2024-03-19T11:31:37.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011971270000000001 2024-03-19T11:31:37.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.248 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:37.248 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.764745+0000", 2024-03-19T11:31:37.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0546999999999997e-05 2024-03-19T11:31:37.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.248 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:37.248 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.764767+0000", 2024-03-19T11:31:37.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2286999999999999e-05 2024-03-19T11:31:37.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:37.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.773924+0000", 2024-03-19T11:31:37.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0091567360000000004 2024-03-19T11:31:37.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:37.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.774217+0000", 2024-03-19T11:31:37.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029350099999999999 2024-03-19T11:31:37.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-19T11:31:37.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.774770+0000", 2024-03-19T11:31:37.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00055288600000000005 2024-03-19T11:31:37.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:37.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.786413+0000", 2024-03-19T11:31:37.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011643229999999999 2024-03-19T11:31:37.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:37.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791104+0000", 2024-03-19T11:31:37.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0046900830000000003 2024-03-19T11:31:37.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:37.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791248+0000", 2024-03-19T11:31:37.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014410200000000001 2024-03-19T11:31:37.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:37.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791340+0000", 2024-03-19T11:31:37.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2540000000000005e-05 2024-03-19T11:31:37.252 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.252 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:37.252 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.252 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1162 4.1 4:8401c1f0:::smithi03829856-43:head [write 594981~609197] snapc 8=[8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-03-19T11:31:37.252 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.838981+0000", 2024-03-19T11:31:37.253 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 80.410715128000007, 2024-03-19T11:31:37.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027704349999999999, 2024-03-19T11:31:37.253 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:37.253 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:37.253 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:37.253 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:37.253 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:37.253 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1162 2024-03-19T11:31:37.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.254 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:37.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.254 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:37.254 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.838981+0000", 2024-03-19T11:31:37.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.254 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:37.254 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.838981+0000", 2024-03-19T11:31:37.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:37.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.838983+0000", 2024-03-19T11:31:37.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5399999999999998e-06 2024-03-19T11:31:37.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:37.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841322+0000", 2024-03-19T11:31:37.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002338592 2024-03-19T11:31:37.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:37.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841371+0000", 2024-03-19T11:31:37.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9123999999999999e-05 2024-03-19T11:31:37.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:37.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841399+0000", 2024-03-19T11:31:37.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7718000000000001e-05 2024-03-19T11:31:37.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:37.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841577+0000", 2024-03-19T11:31:37.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000178097 2024-03-19T11:31:37.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:37.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841825+0000", 2024-03-19T11:31:37.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00024824299999999999 2024-03-19T11:31:37.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-19T11:31:37.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842227+0000", 2024-03-19T11:31:37.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00040212199999999998 2024-03-19T11:31:37.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:37.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.865182+0000", 2024-03-19T11:31:37.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022954810999999999 2024-03-19T11:31:37.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:37.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.866360+0000", 2024-03-19T11:31:37.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001178449 2024-03-19T11:31:37.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:37.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.866538+0000", 2024-03-19T11:31:37.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017791000000000001 2024-03-19T11:31:37.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:37.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.866685+0000", 2024-03-19T11:31:37.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000146744 2024-03-19T11:31:37.259 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.259 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:37.259 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.260 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1271 4.9 4:953e720b:::smithi03829856-45:2 [read 795136~1008128] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-03-19T11:31:37.260 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:21.868056+0000", 2024-03-19T11:31:37.260 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 75.381639622999998, 2024-03-19T11:31:37.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.047270591000000001, 2024-03-19T11:31:37.260 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:37.260 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:37.260 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:37.260 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:37.260 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:37.260 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1271 2024-03-19T11:31:37.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.260 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:37.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.261 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:37.261 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868056+0000", 2024-03-19T11:31:37.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.261 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:37.261 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868056+0000", 2024-03-19T11:31:37.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.262 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:37.262 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868059+0000", 2024-03-19T11:31:37.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0469999999999998e-06 2024-03-19T11:31:37.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.263 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:37.263 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868072+0000", 2024-03-19T11:31:37.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2700999999999999e-05 2024-03-19T11:31:37.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.263 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:37.264 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868106+0000", 2024-03-19T11:31:37.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4152000000000001e-05 2024-03-19T11:31:37.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:37.264 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868126+0000", 2024-03-19T11:31:37.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9386999999999999e-05 2024-03-19T11:31:37.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:37.264 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.911447+0000", 2024-03-19T11:31:37.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.043321164000000002 2024-03-19T11:31:37.265 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.265 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.265 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:37.265 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.911762+0000", 2024-03-19T11:31:37.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00031492400000000001 2024-03-19T11:31:37.265 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.265 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.265 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:37.265 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.915327+0000", 2024-03-19T11:31:37.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003565216 2024-03-19T11:31:37.265 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.265 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:37.265 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.265 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.266 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(3.1 84/83 [PushOp(3:9205291d:::smithi03829856-50 low_tier:head, version: 28'36, data_included: [393216~851968,1703936~196608], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:9205291d:::smithi03829856-50 low_tier:head@28'36, size: 2823543, copy_subset: [(0, 2823543)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1900544, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:baab691a:::smithi03829856-2 low_tier:head, version: 28'37, data_included: [393216~524288,1507328~509479], data_size: 1033767, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:baab691a:::smithi03829856-2 low_tier:head@28'37, size: 2016807, copy_subset: [(0, 2016807)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2016807, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:98ce8426:::smithi03829856-4 low_tier:head, version: 28'38, data_included: [393216~720896,1638400~327680], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:98ce8426:::smithi03829856-4 low_tier:head@28'38, size: 2050421, copy_subset: [(0, 2050421)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1966080, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:a86be07b:::smithi03829856-44 low_tier:head, version: 28'41, data_included: [720896~655360,1769472~393216], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:a86be07b:::smithi03829856-44 low_tier:head@28'41, size: 2722701, copy_subset: [(0, 2722701)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2162688, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:bb56919e:::smithi03829856-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'44, data_included: [393216~786432,1703936~262144], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:bb56919e:::smithi03829856-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'44, size: 2806736, copy_subset: [(0, 2806736)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1966080, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:b1f822cf:::smithi03829856-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'46, data_included: [720896~786432,1966080~262144], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:b1f822cf:::smithi03829856-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'46, size: 2369754, copy_subset: [(0, 2369754)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2228224, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:a5e8313a:::smithi03829856-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'47, data_included: [524288~720896,1769472~327680], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:a5e8313a:::smithi03829856-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'47, size: 2571438, copy_subset: [(0, 2571438)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:bca5ec64:::e3a26bf9230b1b95b8cf1a4cb01909236bf2bf3fd59b083b2291d278ab98264d:head, version: 28'48, data_included: [0~393216], data_size: 393216, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bca5ec64:::e3a26bf9230b1b95b8cf1a4cb01909236bf2bf3fd59b083b2291d278ab98264d:head@28'48, size: 533504, copy_subset: [(0, 533504)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 533504, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:99a49d1b:::895a63407b6109b3cca9a6f5ef827134d56679bef42daadc5892269682c428ed:head, version: 32'49, data_included: [393216~388096], data_size: 388096, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:99a49d1b:::895a63407b6109b3cca9a6f5ef827134d56679bef42daadc5892269682c428ed:head@32'49, size: 781312, copy_subset: [(0, 781312)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 781312, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:83064589:::a3639ef4c215b429a637f1406db2398074b33464b996fe20e20d8b66c6bd720b:head, version: 33'51, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:83064589:::a3639ef4c215b429a637f1406db2398074b33464b996fe20e20d8b66c6bd720b:head@33'51, size: 1606656, copy_subset: [(0, 1606656)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:37.266 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:03.092965+0000", 2024-03-19T11:31:37.266 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.156730580000001, 2024-03-19T11:31:37.266 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.061194046000000002, 2024-03-19T11:31:37.266 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:37.266 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:37.266 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:37.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.266 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:37.266 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.092965+0000", 2024-03-19T11:31:37.266 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.267 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:37.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.092965+0000", 2024-03-19T11:31:37.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.267 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:37.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.092968+0000", 2024-03-19T11:31:37.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8720000000000002e-06 2024-03-19T11:31:37.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.267 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:37.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.106869+0000", 2024-03-19T11:31:37.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013900554000000001 2024-03-19T11:31:37.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.268 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.268 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:37.268 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.107077+0000", 2024-03-19T11:31:37.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000208626 2024-03-19T11:31:37.268 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.268 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.268 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:37.268 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.107372+0000", 2024-03-19T11:31:37.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029439199999999998 2024-03-19T11:31:37.268 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.268 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.268 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:37.268 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.108053+0000", 2024-03-19T11:31:37.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00068115899999999997 2024-03-19T11:31:37.268 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.269 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.269 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:37.269 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.108165+0000", 2024-03-19T11:31:37.269 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011162000000000001 2024-03-19T11:31:37.269 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.269 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.269 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:37.269 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.154159+0000", 2024-03-19T11:31:37.269 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.045994822999999997 2024-03-19T11:31:37.269 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.269 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:37.269 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.269 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.269 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.270 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.5 84/83 [PushOp(4:a0a50d04:::smithi03829856-13:b, version: 56'65, data_included: [601684~619341,1669078~429235], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a0a50d04:::smithi03829856-13:b@56'65, size: 2201684, copy_subset: [(0, 2201684)], clone_subset: {}, snapset: 17=[]:{b: [b, 6], 12: [12, d], 14: [13], 17: [17, 16, 15]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2098313, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:a0a50d04:::smithi03829856-13:14, version: 74'84, data_included: [601684~754801,1669078~293775], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a0a50d04:::smithi03829856-13:14@74'84, size: 2722734, copy_subset: [(0, 2722734)], clone_subset: {}, snapset: 17=[]:{b: [b, 6], 12: [12, d], 14: [13], 17: [17, 16, 15]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1962853, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:a0a50d04:::smithi03829856-13:12, version: 81'90, data_included: [601684~619341,1669078~429235], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a0a50d04:::smithi03829856-13:12@81'90, size: 2201684, copy_subset: [(0, 2201684)], clone_subset: {}, snapset: 17=[]:{b: [b, 6], 12: [12, d], 14: [13], 17: [17, 16, 15]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2098313, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:a38b9453:::smithi03829856-37:13, version: 81'92, data_included: [605052~774376,1923587~274200], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a38b9453:::smithi03829856-37:13@81'92, size: 2605052, copy_subset: [(0, 2605052)], clone_subset: {}, snapset: 14=[]:{13: [13, 12, d, b, 6]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2197787, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:a0a50d04:::smithi03829856-13:17, version: 82'94, data_included: [601684~754801,1669078~293775], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a0a50d04:::smithi03829856-13:17@82'94, size: 3025260, copy_subset: [(0, 3025260)], clone_subset: {}, snapset: 17=[]:{b: [b, 6], 12: [12, d], 14: [13], 17: [17, 16, 15]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1962853, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:a0a50d04:::smithi03829856-13:head, version: 82'97, data_included: [601684~619341,1669078~429235], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:a0a50d04:::smithi03829856-13:head@82'97, size: 2201684, copy_subset: [(0, 2201684)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2098313, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:37.270 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:03.103661+0000", 2024-03-19T11:31:37.270 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.146034559, 2024-03-19T11:31:37.270 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027776637, 2024-03-19T11:31:37.270 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:37.270 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:37.270 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:37.270 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.270 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:37.270 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.103661+0000", 2024-03-19T11:31:37.270 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.270 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.270 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.270 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:37.271 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.103661+0000", 2024-03-19T11:31:37.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.271 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:37.271 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.103664+0000", 2024-03-19T11:31:37.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.48e-06 2024-03-19T11:31:37.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.271 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:37.271 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.109797+0000", 2024-03-19T11:31:37.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061333380000000003 2024-03-19T11:31:37.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:37.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.109866+0000", 2024-03-19T11:31:37.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8474999999999997e-05 2024-03-19T11:31:37.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:37.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.109957+0000", 2024-03-19T11:31:37.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1669000000000006e-05 2024-03-19T11:31:37.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:37.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110162+0000", 2024-03-19T11:31:37.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000204417 2024-03-19T11:31:37.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:37.273 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110186+0000", 2024-03-19T11:31:37.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4729e-05 2024-03-19T11:31:37.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:37.273 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.131438+0000", 2024-03-19T11:31:37.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021251529000000002 2024-03-19T11:31:37.273 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.273 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:37.273 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.273 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.5 84/83 [PushOp(4:a38b9453:::smithi03829856-37:head, version: 81'93, data_included: [1775443~781480,2724284~166520], data_size: 948000, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:a38b9453:::smithi03829856-37:head@81'93, size: 2958032, copy_subset: [(0, 2958032)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2958032, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1775443, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:37.273 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:03.110032+0000", 2024-03-19T11:31:37.274 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 34.139664330999999, 2024-03-19T11:31:37.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027756439000000001, 2024-03-19T11:31:37.274 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:37.274 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:37.274 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:37.274 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.274 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:37.274 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110032+0000", 2024-03-19T11:31:37.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.274 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.274 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.274 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:37.274 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110032+0000", 2024-03-19T11:31:37.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:37.275 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110033+0000", 2024-03-19T11:31:37.275 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1510000000000001e-06 2024-03-19T11:31:37.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:37.275 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110353+0000", 2024-03-19T11:31:37.275 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00031981199999999999 2024-03-19T11:31:37.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:37.275 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110375+0000", 2024-03-19T11:31:37.275 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2772000000000002e-05 2024-03-19T11:31:37.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:37.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110396+0000", 2024-03-19T11:31:37.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1066000000000001e-05 2024-03-19T11:31:37.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:37.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.131543+0000", 2024-03-19T11:31:37.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021146166000000001 2024-03-19T11:31:37.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:37.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.131563+0000", 2024-03-19T11:31:37.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0750999999999998e-05 2024-03-19T11:31:37.277 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.277 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:37.277 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.137788+0000", 2024-03-19T11:31:37.277 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062247209999999999 2024-03-19T11:31:37.277 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.277 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:37.277 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.277 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.277 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.18 85/84 [PushOp(4:1d58db11:::smithi03829856-18:6, version: 55'80, data_included: [759683~639358,1904434~409218], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:6@55'80, size: 3159683, copy_subset: [(0, 3159683)], clone_subset: {}, snapset: 16=[]:{6: [6], b: [b], 15: [15, 13, 12, d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2313652, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1d58db11:::smithi03829856-18:b, version: 58'84, data_included: [443718~955323,1564138~93253], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:b@58'84, size: 3445402, copy_subset: [(0, 3445402)], clone_subset: {}, snapset: 16=[]:{6: [6], b: [b], 15: [15, 13, 12, d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1657391, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:10, version: 66'95, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:10@66'95, size: 3764735, copy_subset: [(0, 3764735)], clone_subset: {}, snapset: 15=[]:{c: [b, 6], 10: [d], 12: [12], 15: [15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:c, version: 70'102, data_included: [416807~576167,1542129~472409], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:c@70'102, size: 2857157, copy_subset: [(0, 2857157)], clone_subset: {}, snapset: 15=[]:{c: [b, 6], 10: [d], 12: [12], 15: [15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2014538, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:15, version: 77'110, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:15@77'110, size: 2621892, copy_subset: [(0, 2621892)], clone_subset: {}, snapset: 15=[]:{c: [b, 6], 10: [d], 12: [12], 15: [15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:37.277 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:04.083107+0000", 2024-03-19T11:31:37.277 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 33.166588765, 2024-03-19T11:31:37.277 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.042109168000000002, 2024-03-19T11:31:37.278 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:37.278 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:37.278 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:37.278 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.278 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:37.278 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.083107+0000", 2024-03-19T11:31:37.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.278 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.278 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.278 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:37.278 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.083107+0000", 2024-03-19T11:31:37.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.278 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.278 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.278 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:37.279 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.083109+0000", 2024-03-19T11:31:37.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7060000000000001e-06 2024-03-19T11:31:37.279 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.279 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:37.279 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092477+0000", 2024-03-19T11:31:37.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0093678589999999992 2024-03-19T11:31:37.279 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.279 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:37.279 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092549+0000", 2024-03-19T11:31:37.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2313999999999999e-05 2024-03-19T11:31:37.279 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.279 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:37.280 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092640+0000", 2024-03-19T11:31:37.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1262999999999997e-05 2024-03-19T11:31:37.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.280 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:37.280 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.093076+0000", 2024-03-19T11:31:37.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00043617800000000001 2024-03-19T11:31:37.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.280 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:37.280 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.093145+0000", 2024-03-19T11:31:37.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8651000000000004e-05 2024-03-19T11:31:37.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.280 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:37.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.125216+0000", 2024-03-19T11:31:37.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032071197000000003 2024-03-19T11:31:37.281 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.281 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:37.281 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.281 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.18 e85,84 4:1d58db11:::smithi03829856-18:7,38'48,4:1d58db11:::smithi03829856-18:9,46'64,4:1d58db11:::smithi03829856-18:2,48'76,4:1d58db11:::smithi03829856-18:c,70'104)", 2024-03-19T11:31:37.281 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:04.092733+0000", 2024-03-19T11:31:37.281 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 33.156962403999998, 2024-03-19T11:31:37.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033436452999999998, 2024-03-19T11:31:37.281 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:37.281 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:37.281 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:37.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:37.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092733+0000", 2024-03-19T11:31:37.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:37.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092733+0000", 2024-03-19T11:31:37.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:37.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092734+0000", 2024-03-19T11:31:37.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0699999999999996e-07 2024-03-19T11:31:37.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:37.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092745+0000", 2024-03-19T11:31:37.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0393e-05 2024-03-19T11:31:37.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:37.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092762+0000", 2024-03-19T11:31:37.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7649e-05 2024-03-19T11:31:37.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:37.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092778+0000", 2024-03-19T11:31:37.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5625999999999999e-05 2024-03-19T11:31:37.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:37.284 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.125342+0000", 2024-03-19T11:31:37.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032564348 2024-03-19T11:31:37.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.284 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:37.284 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.125375+0000", 2024-03-19T11:31:37.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2441000000000003e-05 2024-03-19T11:31:37.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.284 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:37.284 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.126170+0000", 2024-03-19T11:31:37.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00079508900000000002 2024-03-19T11:31:37.284 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.284 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:37.284 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.285 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(3.3 85/84 [PushOp(3:fb6cb292:::smithi03829856-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'21, data_included: [655360~851968,1900544~196608], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:fb6cb292:::smithi03829856-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'21, size: 2302526, copy_subset: [(0, 2302526)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:d281c5a5:::smithi03829856-30 low_tier:head, version: 28'42, data_included: [458752~589824,1507328~458752], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:d281c5a5:::smithi03829856-30 low_tier:head@28'42, size: 2487403, copy_subset: [(0, 2487403)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1966080, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:ed957b32:::smithi03829856-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'44, data_included: [524288~786432,1900544~233912], data_size: 1020344, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:ed957b32:::smithi03829856-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'44, size: 2134456, copy_subset: [(0, 2134456)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2134456, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:f23a45c7:::smithi03829856-36 low_tier:head, version: 28'45, data_included: [524288~786432,1835008~262144], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:f23a45c7:::smithi03829856-36 low_tier:head@28'45, size: 2588245, copy_subset: [(0, 2588245)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:edb38de0:::smithi03829856-40 low_tier:head, version: 28'46, data_included: [655360~720896,2097152~327680], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:edb38de0:::smithi03829856-40 low_tier:head@28'46, size: 2655473, copy_subset: [(0, 2655473)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2424832, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:f92b3f40:::b1fb925e39ab40886d36b70ff1c8c4f867ef6cb2ef150fbe90de57d057dfec36:head, version: 29'47, data_included: [262144~720896,1507328~327680], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:f92b3f40:::b1fb925e39ab40886d36b70ff1c8c4f867ef6cb2ef150fbe90de57d057dfec36:head@29'47, size: 2098176, copy_subset: [(0, 2098176)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1835008, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:fc3d03e1:::626f73f06d73fccc02b43398389ff991d56d81381f5e170c36fb9439a43cbbc0:head, version: 32'48, data_included: [0~131072], data_size: 131072, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:fc3d03e1:::626f73f06d73fccc02b43398389ff991d56d81381f5e170c36fb9439a43cbbc0:head@32'48, size: 462848, copy_subset: [(0, 462848)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 462848, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:f8934067:::8892b7b523af2cde6be0d91d0468f8be86a651cb052f82daf3e2c6278805eb6e:head, version: 32'49, data_included: [0~131072], data_size: 131072, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:f8934067:::8892b7b523af2cde6be0d91d0468f8be86a651cb052f82daf3e2c6278805eb6e:head@32'49, size: 202240, copy_subset: [(0, 202240)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 202240, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:37.285 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:04.108513+0000", 2024-03-19T11:31:37.285 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 33.141182962999999, 2024-03-19T11:31:37.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.036668899999999997, 2024-03-19T11:31:37.285 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:37.285 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:37.285 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:37.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:37.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.108513+0000", 2024-03-19T11:31:37.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:37.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.108513+0000", 2024-03-19T11:31:37.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:37.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.108515+0000", 2024-03-19T11:31:37.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7579999999999999e-06 2024-03-19T11:31:37.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:37.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115352+0000", 2024-03-19T11:31:37.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0068369950000000002 2024-03-19T11:31:37.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.287 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:37.287 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115421+0000", 2024-03-19T11:31:37.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9282999999999994e-05 2024-03-19T11:31:37.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.287 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:37.287 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115505+0000", 2024-03-19T11:31:37.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3714999999999994e-05 2024-03-19T11:31:37.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.287 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:37.287 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115948+0000", 2024-03-19T11:31:37.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000443036 2024-03-19T11:31:37.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.288 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:37.288 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116012+0000", 2024-03-19T11:31:37.288 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3905999999999994e-05 2024-03-19T11:31:37.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.288 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:37.288 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.145182+0000", 2024-03-19T11:31:37.288 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.029170207 2024-03-19T11:31:37.288 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.288 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:37.288 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.288 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.4.13:82 3.3 e85/84 3:f6b6d21a:::smithi03829856-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 85'103, mlcod=0'0)", 2024-03-19T11:31:37.289 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:04.115589+0000", 2024-03-19T11:31:37.289 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 33.134106656999997, 2024-03-19T11:31:37.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035569142999999998, 2024-03-19T11:31:37.289 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:37.289 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:37.289 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:37.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:37.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115589+0000", 2024-03-19T11:31:37.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:37.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115589+0000", 2024-03-19T11:31:37.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:37.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115591+0000", 2024-03-19T11:31:37.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2780000000000001e-06 2024-03-19T11:31:37.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:37.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115602+0000", 2024-03-19T11:31:37.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1797000000000001e-05 2024-03-19T11:31:37.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:37.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115613+0000", 2024-03-19T11:31:37.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1025e-05 2024-03-19T11:31:37.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:37.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115619+0000", 2024-03-19T11:31:37.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.976e-06 2024-03-19T11:31:37.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:37.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.145267+0000", 2024-03-19T11:31:37.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.02964791 2024-03-19T11:31:37.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:37.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.145304+0000", 2024-03-19T11:31:37.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6455000000000002e-05 2024-03-19T11:31:37.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.292 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:37.292 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.151114+0000", 2024-03-19T11:31:37.292 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0058103139999999996 2024-03-19T11:31:37.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.292 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:37.292 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.151158+0000", 2024-03-19T11:31:37.292 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4388000000000003e-05 2024-03-19T11:31:37.292 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.292 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:37.292 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.293 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.4.13:84 3.3 e85/84 3:ce6951d3:::smithi03829856-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 85'104, mlcod=0'0)", 2024-03-19T11:31:37.293 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:04.115824+0000", 2024-03-19T11:31:37.293 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 33.133871654000004, 2024-03-19T11:31:37.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035373398, 2024-03-19T11:31:37.293 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:37.293 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:37.293 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:37.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.293 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:37.293 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115824+0000", 2024-03-19T11:31:37.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.293 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:37.293 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115824+0000", 2024-03-19T11:31:37.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.294 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:37.294 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115825+0000", 2024-03-19T11:31:37.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3399999999999999e-07 2024-03-19T11:31:37.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.294 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:37.294 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115861+0000", 2024-03-19T11:31:37.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6303000000000001e-05 2024-03-19T11:31:37.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.294 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:37.294 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115869+0000", 2024-03-19T11:31:37.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0350000000000001e-06 2024-03-19T11:31:37.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:37.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115874+0000", 2024-03-19T11:31:37.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1599999999999997e-06 2024-03-19T11:31:37.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:37.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.145686+0000", 2024-03-19T11:31:37.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.029811252 2024-03-19T11:31:37.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:37.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.145706+0000", 2024-03-19T11:31:37.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0264000000000001e-05 2024-03-19T11:31:37.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.296 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:37.296 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.151161+0000", 2024-03-19T11:31:37.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054555280000000003 2024-03-19T11:31:37.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.296 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:37.296 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.151198+0000", 2024-03-19T11:31:37.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6322000000000002e-05 2024-03-19T11:31:37.296 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.296 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:37.296 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.297 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.4.13:85 3.3 e85/84 3:ce6951d3:::smithi03829856-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 85'105, mlcod=0'0)", 2024-03-19T11:31:37.297 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:04.115887+0000", 2024-03-19T11:31:37.297 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 33.133809040000003, 2024-03-19T11:31:37.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035346673000000002, 2024-03-19T11:31:37.297 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:37.297 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:37.297 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:37.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:37.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115887+0000", 2024-03-19T11:31:37.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:37.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115887+0000", 2024-03-19T11:31:37.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:37.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115887+0000", 2024-03-19T11:31:37.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2300000000000002e-07 2024-03-19T11:31:37.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:37.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115900+0000", 2024-03-19T11:31:37.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3044000000000001e-05 2024-03-19T11:31:37.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:37.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115906+0000", 2024-03-19T11:31:37.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5929999999999996e-06 2024-03-19T11:31:37.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:37.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115910+0000", 2024-03-19T11:31:37.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9269999999999998e-06 2024-03-19T11:31:37.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:37.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.145919+0000", 2024-03-19T11:31:37.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030009650999999998 2024-03-19T11:31:37.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:37.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.145948+0000", 2024-03-19T11:31:37.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9047000000000002e-05 2024-03-19T11:31:37.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.300 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:37.300 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.151200+0000", 2024-03-19T11:31:37.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052512970000000003 2024-03-19T11:31:37.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.300 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:37.300 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.151234+0000", 2024-03-19T11:31:37.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3791000000000002e-05 2024-03-19T11:31:37.300 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.300 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:37.300 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.301 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(3.3 85/84 [PushOp(3:fcdd1002:::06a0957e01d2e04d1b09d0c04f111819905dcf8372f2c6bee5014c22d5b575ef:head, version: 34'50, data_included: [0~458752], data_size: 458752, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:fcdd1002:::06a0957e01d2e04d1b09d0c04f111819905dcf8372f2c6bee5014c22d5b575ef:head@34'50, size: 689152, copy_subset: [(0, 689152)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 689152, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:37.301 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:04.115917+0000", 2024-03-19T11:31:37.301 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 33.133778608999997, 2024-03-19T11:31:37.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033796169000000001, 2024-03-19T11:31:37.301 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:37.301 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:37.301 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:37.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.301 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:37.301 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115917+0000", 2024-03-19T11:31:37.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.301 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:37.301 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115917+0000", 2024-03-19T11:31:37.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:37.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115918+0000", 2024-03-19T11:31:37.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8000000000000002e-07 2024-03-19T11:31:37.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:37.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116163+0000", 2024-03-19T11:31:37.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00024516400000000003 2024-03-19T11:31:37.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:37.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116181+0000", 2024-03-19T11:31:37.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8372999999999998e-05 2024-03-19T11:31:37.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:37.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116196+0000", 2024-03-19T11:31:37.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5041e-05 2024-03-19T11:31:37.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:37.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.147578+0000", 2024-03-19T11:31:37.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031382284000000003 2024-03-19T11:31:37.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:37.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.147594+0000", 2024-03-19T11:31:37.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5153e-05 2024-03-19T11:31:37.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:37.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.149713+0000", 2024-03-19T11:31:37.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021198739999999999 2024-03-19T11:31:37.304 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.304 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:37.304 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.304 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.4.13:83 3.3 e85/84 3:f6b6d21a:::smithi03829856-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 85'106, mlcod=0'0)", 2024-03-19T11:31:37.304 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:04.116216+0000", 2024-03-19T11:31:37.305 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 33.133479549999997, 2024-03-19T11:31:37.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035049444999999999, 2024-03-19T11:31:37.305 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:37.305 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:37.305 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:37.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.305 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:37.305 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116216+0000", 2024-03-19T11:31:37.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.305 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.305 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:37.305 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116216+0000", 2024-03-19T11:31:37.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.305 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:37.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116217+0000", 2024-03-19T11:31:37.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.22e-07 2024-03-19T11:31:37.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:37.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116221+0000", 2024-03-19T11:31:37.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5349999999999998e-06 2024-03-19T11:31:37.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:37.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116237+0000", 2024-03-19T11:31:37.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5532e-05 2024-03-19T11:31:37.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:37.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116245+0000", 2024-03-19T11:31:37.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0450000000000001e-06 2024-03-19T11:31:37.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:37.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.146739+0000", 2024-03-19T11:31:37.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030494351999999999 2024-03-19T11:31:37.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:37.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.146776+0000", 2024-03-19T11:31:37.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6894999999999998e-05 2024-03-19T11:31:37.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:37.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.151237+0000", 2024-03-19T11:31:37.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044605160000000003 2024-03-19T11:31:37.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:37.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.151266+0000", 2024-03-19T11:31:37.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9247999999999998e-05 2024-03-19T11:31:37.308 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.308 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:37.308 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.309 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3194 4.d 4:b65e6658:::smithi03829856-9:head [read 0~3714347,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108)", 2024-03-19T11:31:37.309 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.202652+0000", 2024-03-19T11:31:37.309 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.047043961, 2024-03-19T11:31:37.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031511255000000002, 2024-03-19T11:31:37.309 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:37.309 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:37.309 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:37.309 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:37.309 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:37.309 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3194 2024-03-19T11:31:37.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.309 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:37.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.309 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:37.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202652+0000", 2024-03-19T11:31:37.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:37.310 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202652+0000", 2024-03-19T11:31:37.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:37.310 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202653+0000", 2024-03-19T11:31:37.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3960000000000001e-06 2024-03-19T11:31:37.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:37.310 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202661+0000", 2024-03-19T11:31:37.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0830000000000004e-06 2024-03-19T11:31:37.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:37.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202679+0000", 2024-03-19T11:31:37.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7972e-05 2024-03-19T11:31:37.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:37.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202693+0000", 2024-03-19T11:31:37.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4031e-05 2024-03-19T11:31:37.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:37.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.203299+0000", 2024-03-19T11:31:37.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00060564400000000002 2024-03-19T11:31:37.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:37.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.203528+0000", 2024-03-19T11:31:37.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000229049 2024-03-19T11:31:37.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:37.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.234163+0000", 2024-03-19T11:31:37.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030635079999999999 2024-03-19T11:31:37.312 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.312 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:37.312 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.313 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3195 4.d 4:b65e6658:::smithi03829856-9:head [read 0~3714347,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108)", 2024-03-19T11:31:37.313 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.202714+0000", 2024-03-19T11:31:37.313 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.046981929999999, 2024-03-19T11:31:37.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.028551382, 2024-03-19T11:31:37.313 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:37.313 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:37.313 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:37.313 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:37.313 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:37.313 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3195 2024-03-19T11:31:37.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.313 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:37.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:37.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202714+0000", 2024-03-19T11:31:37.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.314 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:37.314 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202714+0000", 2024-03-19T11:31:37.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.314 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:37.314 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202715+0000", 2024-03-19T11:31:37.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.203e-06 2024-03-19T11:31:37.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.314 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:37.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202722+0000", 2024-03-19T11:31:37.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1289999999999998e-06 2024-03-19T11:31:37.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:37.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202739+0000", 2024-03-19T11:31:37.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6929e-05 2024-03-19T11:31:37.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:37.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202754+0000", 2024-03-19T11:31:37.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4642e-05 2024-03-19T11:31:37.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:37.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.227218+0000", 2024-03-19T11:31:37.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.024464587999999999 2024-03-19T11:31:37.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:37.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.227471+0000", 2024-03-19T11:31:37.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000252747 2024-03-19T11:31:37.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:37.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.231265+0000", 2024-03-19T11:31:37.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037941440000000002 2024-03-19T11:31:37.316 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.316 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:37.316 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.317 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3196 4.d 4:b65e6658:::smithi03829856-9:head [sparse-read 0~2135368] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108)", 2024-03-19T11:31:37.317 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.202774+0000", 2024-03-19T11:31:37.317 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.046922189, 2024-03-19T11:31:37.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033044519000000001, 2024-03-19T11:31:37.317 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:37.317 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:37.317 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:37.317 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:37.317 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:37.317 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3196 2024-03-19T11:31:37.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.317 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:37.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.318 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:37.318 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202774+0000", 2024-03-19T11:31:37.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.318 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:37.318 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202774+0000", 2024-03-19T11:31:37.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.318 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:37.318 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202775+0000", 2024-03-19T11:31:37.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.423e-06 2024-03-19T11:31:37.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:37.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202782+0000", 2024-03-19T11:31:37.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3660000000000003e-06 2024-03-19T11:31:37.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:37.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202801+0000", 2024-03-19T11:31:37.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8182e-05 2024-03-19T11:31:37.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:37.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202815+0000", 2024-03-19T11:31:37.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4042e-05 2024-03-19T11:31:37.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:37.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.234294+0000", 2024-03-19T11:31:37.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031479493999999997 2024-03-19T11:31:37.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:37.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.234670+0000", 2024-03-19T11:31:37.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000375595 2024-03-19T11:31:37.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:37.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.235818+0000", 2024-03-19T11:31:37.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011484170000000001 2024-03-19T11:31:37.320 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.320 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:37.320 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.321 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(osd.0.13:71 3.3 3:e22267ea:::smithi03829856-38 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e108)", 2024-03-19T11:31:37.321 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.209268+0000", 2024-03-19T11:31:37.321 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.040427938000001, 2024-03-19T11:31:37.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.038534928000000003, 2024-03-19T11:31:37.321 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:37.321 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:37.321 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:37.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:37.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.209268+0000", 2024-03-19T11:31:37.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:37.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.209268+0000", 2024-03-19T11:31:37.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:37.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.209272+0000", 2024-03-19T11:31:37.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0500000000000002e-06 2024-03-19T11:31:37.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:37.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.209291+0000", 2024-03-19T11:31:37.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8734999999999999e-05 2024-03-19T11:31:37.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.323 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:37.323 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.209329+0000", 2024-03-19T11:31:37.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8223000000000003e-05 2024-03-19T11:31:37.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.323 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:37.323 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.209354+0000", 2024-03-19T11:31:37.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5114e-05 2024-03-19T11:31:37.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:37.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.209573+0000", 2024-03-19T11:31:37.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021842000000000001 2024-03-19T11:31:37.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:37.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.209814+0000", 2024-03-19T11:31:37.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00024179400000000001 2024-03-19T11:31:37.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:37.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.210284+0000", 2024-03-19T11:31:37.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00046991699999999998 2024-03-19T11:31:37.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:37.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.213128+0000", 2024-03-19T11:31:37.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002843609 2024-03-19T11:31:37.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:37.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.247417+0000", 2024-03-19T11:31:37.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.034289312000000002 2024-03-19T11:31:37.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:37.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.247692+0000", 2024-03-19T11:31:37.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027449000000000001 2024-03-19T11:31:37.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:37.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.247803+0000", 2024-03-19T11:31:37.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111264 2024-03-19T11:31:37.326 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.326 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:37.326 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.326 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3198 3.3 3:f117d76c:::3a52856062567f05d8c7df91dad2c51911138f6216e1ec709153bdff536e4abe:head [write 0~927232] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-19T11:31:37.326 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.213434+0000", 2024-03-19T11:31:37.326 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.036262247, 2024-03-19T11:31:37.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.039874882, 2024-03-19T11:31:37.326 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:37.326 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:37.326 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:37.327 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:37.327 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:37.327 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3198 2024-03-19T11:31:37.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.327 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:37.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:37.327 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.213434+0000", 2024-03-19T11:31:37.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:37.327 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.213434+0000", 2024-03-19T11:31:37.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:37.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.213436+0000", 2024-03-19T11:31:37.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4430000000000002e-06 2024-03-19T11:31:37.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:37.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.219501+0000", 2024-03-19T11:31:37.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0060649240000000002 2024-03-19T11:31:37.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:37.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.219531+0000", 2024-03-19T11:31:37.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0470000000000001e-05 2024-03-19T11:31:37.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:37.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.219552+0000", 2024-03-19T11:31:37.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1012000000000001e-05 2024-03-19T11:31:37.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:37.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.231849+0000", 2024-03-19T11:31:37.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012296105999999999 2024-03-19T11:31:37.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:37.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.232349+0000", 2024-03-19T11:31:37.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00050035100000000001 2024-03-19T11:31:37.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:37.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.233102+0000", 2024-03-19T11:31:37.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00075262300000000001 2024-03-19T11:31:37.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:37.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.249722+0000", 2024-03-19T11:31:37.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016620901 2024-03-19T11:31:37.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:37.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.253185+0000", 2024-03-19T11:31:37.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034625340000000002 2024-03-19T11:31:37.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.331 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:37.331 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.253252+0000", 2024-03-19T11:31:37.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6538000000000006e-05 2024-03-19T11:31:37.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.331 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:37.331 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.253309+0000", 2024-03-19T11:31:37.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.698e-05 2024-03-19T11:31:37.331 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.331 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:37.331 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.331 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3323 4.18 4:1d58db11:::smithi03829856-18:head [tier-promote] snapc 22=[22,21,1f,1e,1d,1c,1b,1a,18,17] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-19T11:31:37.331 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:31.226946+0000", 2024-03-19T11:31:37.332 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 6.0227499509999998, 2024-03-19T11:31:37.332 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.037925183000000001, 2024-03-19T11:31:37.332 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:37.332 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:37.332 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:37.332 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:37.332 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:37.332 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3323 2024-03-19T11:31:37.332 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.332 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:37.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.332 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:37.332 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.226946+0000", 2024-03-19T11:31:37.332 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.332 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:37.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.226946+0000", 2024-03-19T11:31:37.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:37.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.226947+0000", 2024-03-19T11:31:37.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4670000000000001e-06 2024-03-19T11:31:37.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:37.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.226955+0000", 2024-03-19T11:31:37.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9130000000000003e-06 2024-03-19T11:31:37.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:37.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.226978+0000", 2024-03-19T11:31:37.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2229e-05 2024-03-19T11:31:37.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:37.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.226993+0000", 2024-03-19T11:31:37.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5420000000000001e-05 2024-03-19T11:31:37.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:37.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.227146+0000", 2024-03-19T11:31:37.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015327200000000001 2024-03-19T11:31:37.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.335 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:37.335 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.227406+0000", 2024-03-19T11:31:37.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025941799999999998 2024-03-19T11:31:37.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.335 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:37.335 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.264871+0000", 2024-03-19T11:31:37.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.037465463999999997 2024-03-19T11:31:37.335 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.335 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:37.335 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.335 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.335 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:37.335 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:37.336 INFO:tasks.rados.rados.0.smithi038.stdout:1555: expect (ObjNum 213 snap 22 seq_num 213) 2024-03-19T11:31:37.337 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-19T11:31:37.337 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:37.346+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:37.441 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:37.452+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:37.512 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:37.513 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:31:37.513 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:31:37.513 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:37.526 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-19T11:31:37.539 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:37.548+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:37.560 INFO:tasks.rados.rados.0.smithi038.stdout:1557: finishing write tid 1 to smithi03829856-47 2024-03-19T11:31:37.560 INFO:tasks.rados.rados.0.smithi038.stdout:1557: finishing write tid 2 to smithi03829856-47 2024-03-19T11:31:37.560 INFO:tasks.rados.rados.0.smithi038.stdout:1557: finishing write tid 3 to smithi03829856-47 2024-03-19T11:31:37.560 INFO:tasks.rados.rados.0.smithi038.stdout:1558: expect (ObjNum 143 snap 15 seq_num 143) 2024-03-19T11:31:37.638 INFO:tasks.rados.rados.0.smithi038.stdout:1557: finishing write tid 4 to smithi03829856-47 2024-03-19T11:31:37.638 INFO:tasks.rados.rados.0.smithi038.stdout:1557: oid 47 updating version 0 to 167 2024-03-19T11:31:37.638 INFO:tasks.rados.rados.0.smithi038.stdout:1557: oid 47 updating version 167 to 168 2024-03-19T11:31:37.638 INFO:tasks.rados.rados.0.smithi038.stdout:1557: oid 47 updating version 168 to 169 2024-03-19T11:31:37.638 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 47 v 169 (ObjNum 276 snap 36 seq_num 276) dirty exists 2024-03-19T11:31:37.639 INFO:tasks.rados.rados.0.smithi038.stdout:1557: left oid 47 (ObjNum 276 snap 36 seq_num 276) 2024-03-19T11:31:37.639 INFO:tasks.rados.rados.0.smithi038.stdout:1559: finishing set_chunk to oid 18 2024-03-19T11:31:37.639 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 18 set_chunk returned error code -95 offset: 2212352 length: 228864 2024-03-19T11:31:37.639 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 18 v 0 (ObjNum 199 snap 22 seq_num 199) dirty exists 2024-03-19T11:31:37.639 INFO:tasks.rados.rados.0.smithi038.stdout:1560: expect (ObjNum 87 snap 8 seq_num 87) 2024-03-19T11:31:37.640 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:37.652+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:37.651 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:37.651 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:31:37.651 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:31:37.651 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:31:37.651 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:37.665 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-19T11:31:37.741 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:37.750+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:37.745 INFO:tasks.rados.rados.0.smithi038.stdout:1561: expect (ObjNum 273 snap 34 seq_num 273) 2024-03-19T11:31:37.795 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:37.796 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:31:37.796 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:31:37.797 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:31:37.797 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.797 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1158 4.8 4:15708482:::smithi03829856-23:head [tier-promote] snapc 8=[8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-03-19T11:31:37.797 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.827517+0000", 2024-03-19T11:31:37.797 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 80.974136156, 2024-03-19T11:31:37.797 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.034096983999999997, 2024-03-19T11:31:37.798 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:37.799 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:37.799 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:37.799 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:37.799 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:37.799 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1158 2024-03-19T11:31:37.800 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.800 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:37.801 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.801 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:37.801 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827517+0000", 2024-03-19T11:31:37.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.801 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.801 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.801 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:37.801 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827517+0000", 2024-03-19T11:31:37.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.801 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.801 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.801 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:37.801 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827520+0000", 2024-03-19T11:31:37.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.45e-06 2024-03-19T11:31:37.802 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.802 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.802 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:37.802 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827535+0000", 2024-03-19T11:31:37.802 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.509e-05 2024-03-19T11:31:37.802 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.802 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.802 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:37.802 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827579+0000", 2024-03-19T11:31:37.802 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4211000000000001e-05 2024-03-19T11:31:37.802 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.802 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.802 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:37.802 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827603+0000", 2024-03-19T11:31:37.803 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3586000000000001e-05 2024-03-19T11:31:37.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.803 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:37.803 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827789+0000", 2024-03-19T11:31:37.803 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018580199999999999 2024-03-19T11:31:37.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.803 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:37.803 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.828077+0000", 2024-03-19T11:31:37.803 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028846500000000001 2024-03-19T11:31:37.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.803 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:37.803 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.861614+0000", 2024-03-19T11:31:37.804 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033536379999999998 2024-03-19T11:31:37.804 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.804 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:37.804 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.804 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.804 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.804 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1162 4.1 e39/21 4:8401c1f0:::smithi03829856-43:head v 39'47, mlcod=38'43)", 2024-03-19T11:31:37.804 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.842598+0000", 2024-03-19T11:31:37.804 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 80.959054386000005, 2024-03-19T11:31:37.804 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0233756, 2024-03-19T11:31:37.804 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:37.804 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:37.804 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:37.804 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.805 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:37.805 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842598+0000", 2024-03-19T11:31:37.805 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.805 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.805 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:37.805 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842598+0000", 2024-03-19T11:31:37.805 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.805 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.805 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:37.805 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842599+0000", 2024-03-19T11:31:37.805 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6000000000000003e-07 2024-03-19T11:31:37.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.806 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.806 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:37.806 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.843566+0000", 2024-03-19T11:31:37.806 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00096688700000000004 2024-03-19T11:31:37.806 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.806 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.806 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:37.806 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.843601+0000", 2024-03-19T11:31:37.806 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.502e-05 2024-03-19T11:31:37.806 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.806 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.806 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:37.806 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.843621+0000", 2024-03-19T11:31:37.806 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0305e-05 2024-03-19T11:31:37.807 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.807 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:37.807 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.843882+0000", 2024-03-19T11:31:37.807 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026027100000000002 2024-03-19T11:31:37.807 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.807 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:37.807 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.844022+0000", 2024-03-19T11:31:37.807 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014020100000000001 2024-03-19T11:31:37.807 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.807 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:37.807 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.865862+0000", 2024-03-19T11:31:37.808 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021839725000000001 2024-03-19T11:31:37.808 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.808 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.808 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:37.808 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.865974+0000", 2024-03-19T11:31:37.808 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011243099999999999 2024-03-19T11:31:37.808 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.808 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:37.808 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.808 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.808 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.808 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1209 4.8 4:1d58db11:::smithi03829856-18:head [write 645402~551297,stat] snapc 9=[9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:31:37.808 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:18.121657+0000", 2024-03-19T11:31:37.808 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 79.679995383999994, 2024-03-19T11:31:37.808 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.025074349999999999, 2024-03-19T11:31:37.809 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:37.809 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:37.809 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:37.809 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:37.809 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:37.809 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1209 2024-03-19T11:31:37.809 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.809 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:37.809 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.809 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:37.809 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.121657+0000", 2024-03-19T11:31:37.809 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.809 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.809 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.810 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:37.810 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.121657+0000", 2024-03-19T11:31:37.810 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.810 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.810 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.810 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:37.810 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.121661+0000", 2024-03-19T11:31:37.810 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0489999999999997e-06 2024-03-19T11:31:37.810 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.810 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.810 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:37.810 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123747+0000", 2024-03-19T11:31:37.810 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0020857720000000001 2024-03-19T11:31:37.810 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.811 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:37.811 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123779+0000", 2024-03-19T11:31:37.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2258000000000002e-05 2024-03-19T11:31:37.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.811 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:37.811 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123795+0000", 2024-03-19T11:31:37.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5177000000000001e-05 2024-03-19T11:31:37.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.811 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:37.811 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123949+0000", 2024-03-19T11:31:37.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015403699999999999 2024-03-19T11:31:37.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.812 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.812 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:37.812 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.124134+0000", 2024-03-19T11:31:37.812 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000185484 2024-03-19T11:31:37.812 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.812 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.812 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:37.812 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.124554+0000", 2024-03-19T11:31:37.812 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00042019699999999998 2024-03-19T11:31:37.812 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.812 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.812 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:37.812 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146267+0000", 2024-03-19T11:31:37.812 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021712838000000002 2024-03-19T11:31:37.812 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.812 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.813 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:37.813 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146424+0000", 2024-03-19T11:31:37.813 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015647299999999999 2024-03-19T11:31:37.813 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.813 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.813 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:37.813 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146597+0000", 2024-03-19T11:31:37.813 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017303699999999999 2024-03-19T11:31:37.813 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.813 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.813 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:37.813 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146732+0000", 2024-03-19T11:31:37.813 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013502800000000001 2024-03-19T11:31:37.813 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.814 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:37.814 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.814 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.814 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.814 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1443 4.2 4:43b32242:::smithi03829856-22:head [read 0~2352947,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:31:37.814 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.979417+0000", 2024-03-19T11:31:37.814 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 71.822235711000005, 2024-03-19T11:31:37.814 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.96719988700000004, 2024-03-19T11:31:37.814 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:37.814 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:37.814 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:37.814 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:37.814 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:37.814 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1443 2024-03-19T11:31:37.815 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.815 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:37.815 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.815 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:37.815 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979417+0000", 2024-03-19T11:31:37.815 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.815 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.815 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.815 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:37.815 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979417+0000", 2024-03-19T11:31:37.815 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.815 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.815 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.815 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:37.816 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979420+0000", 2024-03-19T11:31:37.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.366e-06 2024-03-19T11:31:37.816 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:37.816 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979437+0000", 2024-03-19T11:31:37.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6192e-05 2024-03-19T11:31:37.816 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:37.816 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979474+0000", 2024-03-19T11:31:37.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7601000000000001e-05 2024-03-19T11:31:37.816 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:37.817 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979496+0000", 2024-03-19T11:31:37.817 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1506999999999999e-05 2024-03-19T11:31:37.817 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.817 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.817 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:37.817 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979712+0000", 2024-03-19T11:31:37.817 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021641299999999999 2024-03-19T11:31:37.817 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.817 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.817 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:31:37.817 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979746+0000", 2024-03-19T11:31:37.817 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3386000000000002e-05 2024-03-19T11:31:37.817 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.817 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.818 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:37.818 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.931552+0000", 2024-03-19T11:31:37.818 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.95180659899999998 2024-03-19T11:31:37.818 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.818 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.818 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:37.818 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.934681+0000", 2024-03-19T11:31:37.818 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0031292289999999999 2024-03-19T11:31:37.818 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.818 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.818 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:37.818 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.942701+0000", 2024-03-19T11:31:37.818 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0080198809999999995 2024-03-19T11:31:37.818 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.819 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:37.819 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.942839+0000", 2024-03-19T11:31:37.819 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013816999999999999 2024-03-19T11:31:37.819 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.819 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:37.819 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.946617+0000", 2024-03-19T11:31:37.819 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003777543 2024-03-19T11:31:37.819 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.819 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:37.819 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.819 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.819 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.820 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1444 4.2 4:43b32242:::smithi03829856-22:head [read 0~2352947,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:31:37.820 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.979553+0000", 2024-03-19T11:31:37.820 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 71.822100008000007, 2024-03-19T11:31:37.820 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.97015945400000003, 2024-03-19T11:31:37.820 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:37.820 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:37.820 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:37.820 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:37.820 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:37.820 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1444 2024-03-19T11:31:37.820 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.820 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:37.820 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.820 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:37.820 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979553+0000", 2024-03-19T11:31:37.821 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.821 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.821 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.821 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:37.821 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979553+0000", 2024-03-19T11:31:37.821 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.821 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.821 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.821 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:37.821 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979554+0000", 2024-03-19T11:31:37.821 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.403e-06 2024-03-19T11:31:37.821 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.821 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.821 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:37.822 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979564+0000", 2024-03-19T11:31:37.822 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0032e-05 2024-03-19T11:31:37.822 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.822 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.822 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:37.822 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979579+0000", 2024-03-19T11:31:37.822 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4875000000000001e-05 2024-03-19T11:31:37.822 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.822 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.822 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:37.822 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979595+0000", 2024-03-19T11:31:37.822 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6300999999999999e-05 2024-03-19T11:31:37.822 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.822 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.822 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:37.823 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979874+0000", 2024-03-19T11:31:37.823 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000279065 2024-03-19T11:31:37.823 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.823 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.823 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:31:37.823 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979904+0000", 2024-03-19T11:31:37.823 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.94e-05 2024-03-19T11:31:37.823 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.823 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.823 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:37.823 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.934781+0000", 2024-03-19T11:31:37.823 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.95487744900000004 2024-03-19T11:31:37.823 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.823 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.823 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:37.824 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.934946+0000", 2024-03-19T11:31:37.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016485999999999999 2024-03-19T11:31:37.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.824 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:37.824 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.946829+0000", 2024-03-19T11:31:37.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011882482999999999 2024-03-19T11:31:37.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.824 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:37.824 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.946989+0000", 2024-03-19T11:31:37.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000160421 2024-03-19T11:31:37.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.824 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:37.825 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.949712+0000", 2024-03-19T11:31:37.825 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002723165 2024-03-19T11:31:37.825 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.825 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:37.825 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.825 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.825 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.825 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1445 4.2 4:43b32242:::smithi03829856-22:head [read 0~2352947,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:31:37.825 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.979653+0000", 2024-03-19T11:31:37.825 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 71.821999876999996, 2024-03-19T11:31:37.825 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.97249507999999996, 2024-03-19T11:31:37.825 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:37.825 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:37.825 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:37.826 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:37.826 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:37.826 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1445 2024-03-19T11:31:37.826 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.826 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:37.826 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.826 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:37.826 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979653+0000", 2024-03-19T11:31:37.826 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.826 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.826 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.826 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:37.826 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979653+0000", 2024-03-19T11:31:37.826 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.827 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.827 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:37.827 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979654+0000", 2024-03-19T11:31:37.827 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.372e-06 2024-03-19T11:31:37.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.827 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.827 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:37.827 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979662+0000", 2024-03-19T11:31:37.828 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9959999999999993e-06 2024-03-19T11:31:37.828 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.828 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.828 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:37.828 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979698+0000", 2024-03-19T11:31:37.828 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5763000000000003e-05 2024-03-19T11:31:37.828 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.828 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.828 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:37.828 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979714+0000", 2024-03-19T11:31:37.828 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6283999999999999e-05 2024-03-19T11:31:37.828 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.828 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.828 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:37.829 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979990+0000", 2024-03-19T11:31:37.829 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000275829 2024-03-19T11:31:37.829 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.829 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.829 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:31:37.829 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.980018+0000", 2024-03-19T11:31:37.829 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8082999999999999e-05 2024-03-19T11:31:37.829 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.829 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.829 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:37.829 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.935620+0000", 2024-03-19T11:31:37.829 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.95560133800000002 2024-03-19T11:31:37.829 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.829 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.829 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:37.830 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.935795+0000", 2024-03-19T11:31:37.830 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000175656 2024-03-19T11:31:37.830 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.830 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.830 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:37.830 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.949835+0000", 2024-03-19T11:31:37.830 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014040083 2024-03-19T11:31:37.830 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.834 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.835 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:37.835 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.950002+0000", 2024-03-19T11:31:37.835 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016673 2024-03-19T11:31:37.835 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.835 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.835 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:37.835 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.952148+0000", 2024-03-19T11:31:37.835 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021459460000000001 2024-03-19T11:31:37.835 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.835 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:37.835 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.835 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.835 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.836 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2207 4.28 4:15708482:::smithi03829856-23:head [write 541225~644694,stat] snapc 13=[13,12,11,d,b,6] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-03-19T11:31:37.836 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:51.160660+0000", 2024-03-19T11:31:37.836 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 46.640993051000002, 2024-03-19T11:31:37.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.025400267000000001, 2024-03-19T11:31:37.836 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:37.836 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:37.836 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:37.836 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:37.836 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:37.836 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2207 2024-03-19T11:31:37.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.836 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:37.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.836 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:37.837 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.160660+0000", 2024-03-19T11:31:37.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.837 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:37.837 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.160660+0000", 2024-03-19T11:31:37.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.837 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:37.837 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.160688+0000", 2024-03-19T11:31:37.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8058000000000001e-05 2024-03-19T11:31:37.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.838 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:37.838 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.168614+0000", 2024-03-19T11:31:37.838 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079262810000000003 2024-03-19T11:31:37.838 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.838 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.838 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:37.838 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.168662+0000", 2024-03-19T11:31:37.838 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8327e-05 2024-03-19T11:31:37.838 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.838 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.838 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:37.838 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.168679+0000", 2024-03-19T11:31:37.838 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6305999999999999e-05 2024-03-19T11:31:37.838 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.839 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:37.839 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.168830+0000", 2024-03-19T11:31:37.839 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000151714 2024-03-19T11:31:37.839 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.839 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:37.839 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.169107+0000", 2024-03-19T11:31:37.839 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027608299999999999 2024-03-19T11:31:37.839 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.839 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:37.839 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.169605+0000", 2024-03-19T11:31:37.839 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00049874999999999995 2024-03-19T11:31:37.840 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.840 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.840 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:37.840 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.185769+0000", 2024-03-19T11:31:37.840 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016163397999999999 2024-03-19T11:31:37.840 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.840 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.840 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:37.840 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.185809+0000", 2024-03-19T11:31:37.840 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0598e-05 2024-03-19T11:31:37.840 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.840 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.840 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:37.840 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.185898+0000", 2024-03-19T11:31:37.840 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8270999999999996e-05 2024-03-19T11:31:37.841 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.841 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.841 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:37.841 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.186060+0000", 2024-03-19T11:31:37.841 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016248099999999999 2024-03-19T11:31:37.841 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.841 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:37.841 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.841 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.841 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.841 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.15 75/74 [PushOp(4:af9d0d56:::smithi03829856-20:head, version: 35'55, data_included: [655360~589824,1900544~418789], data_size: 1008613, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:af9d0d56:::smithi03829856-20:head@35'55, size: 2319333, copy_subset: [(0, 2319333)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2319333, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:aa3303a5:::smithi03829856-17:b, version: 58'79, data_included: [614400~692224,1903006~356352], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:aa3303a5:::smithi03829856-17:b@58'79, size: 2268912, copy_subset: [(0, 2268912)], clone_subset: {}, snapset: 14=[]:{b: [b, 6], 10: [d], 12: [12, 11], 14: [13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2259358, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ad3ad1b3:::smithi03829856-21:b, version: 58'81, data_included: [736140~537686,2020581~315559], data_size: 853245, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:ad3ad1b3:::smithi03829856-21:b@58'81, size: 2336140, copy_subset: [(0, 2336140)], clone_subset: {}, snapset: 10=[]:{b: [b, 6], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2336140, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:afa9f208:::smithi03829856-35:12, version: 67'105, data_included: [521050~698057,1806623~350519], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:afa9f208:::smithi03829856-35:12@67'105, size: 2521050, copy_subset: [(0, 2521050)], clone_subset: {}, snapset: 12=[]:{11: [11, d, b, 6], 12: [12]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2157142, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:aa3303a5:::smithi03829856-17:12, version: 69'113, data_included: [413472~893152,1808868~155424], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:aa3303a5:::smithi03829856-17:12@69'113, size: 2319366, copy_subset: [(0, 2319366)], clone_subset: {}, snapset: 14=[]:{b: [b, 6], 10: [d], 12: [12, 11], 14: [13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1964292, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ac9cd3a3:::smithi03829856-8:10, version: 70'117, data_included: [517649~620182,1776668~428394], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:ac9cd3a3:::smithi03829856-8:10@70'117, size: 3008420, copy_subset: [(0, 3008420)], clone_subset: {}, snapset: 10=[]:{10: [d, b, 6]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2205062, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ac9cd3a3:::smithi03829856-8:head, version: 70'118, data_included: [517649~620182,1776668~428394], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:ac9cd3a3:::smithi03829856-8:head@70'118, size: 3008420, copy_subset: [(0, 3008420)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2205062, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:aa3303a5:::smithi03829856-17:10, version: 70'119, data_included: [413472~893152,1818036~155424], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:aa3303a5:::smithi03829856-17:10@70'119, size: 3613472, copy_subset: [(0, 3613472)], clone_subset: {}, snapset: 14=[]:{b: [b, 6], 10: [d], 12: [12, 11], 14: [13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1973460, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ad3ad1b3:::smithi03829856-21:10, version: 72'122, data_included: [531121~742705,1711057~305871], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:ad3ad1b3:::smithi03829856-21:10@72'122, size: 3731121, copy_subset: [(0, 3731121)], clone_subset: {}, snapset: 10=[]:{b: [b, 6], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2016928, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:37.841 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.081994+0000", 2024-03-19T11:31:37.841 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 43.719658354000003, 2024-03-19T11:31:37.841 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.040292051000000002, 2024-03-19T11:31:37.842 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:37.842 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:37.842 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:37.842 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.842 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:37.842 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.081994+0000", 2024-03-19T11:31:37.842 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.842 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.842 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.842 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:37.842 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.081994+0000", 2024-03-19T11:31:37.842 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.842 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.842 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.842 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:37.843 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.081996+0000", 2024-03-19T11:31:37.843 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5069999999999999e-06 2024-03-19T11:31:37.843 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.843 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:37.843 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088193+0000", 2024-03-19T11:31:37.843 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061970209999999996 2024-03-19T11:31:37.843 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.843 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:37.843 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088252+0000", 2024-03-19T11:31:37.843 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8838999999999997e-05 2024-03-19T11:31:37.843 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.843 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:37.844 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088322+0000", 2024-03-19T11:31:37.844 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0455000000000002e-05 2024-03-19T11:31:37.844 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.844 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.844 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:37.844 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088660+0000", 2024-03-19T11:31:37.844 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033735299999999998 2024-03-19T11:31:37.844 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.844 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.844 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:37.844 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088727+0000", 2024-03-19T11:31:37.844 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7335999999999994e-05 2024-03-19T11:31:37.844 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.845 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.845 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:37.845 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.122286+0000", 2024-03-19T11:31:37.845 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033559539999999999 2024-03-19T11:31:37.845 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.845 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:37.845 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.845 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.845 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.845 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.15 e75,74 4:ac9cd3a3:::smithi03829856-8:4,49'71)", 2024-03-19T11:31:37.845 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.088380+0000", 2024-03-19T11:31:37.845 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 43.713272492000002, 2024-03-19T11:31:37.845 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.044894303000000003, 2024-03-19T11:31:37.845 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:37.845 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:37.846 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:37.846 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.846 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:37.846 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088380+0000", 2024-03-19T11:31:37.846 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.846 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.846 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.846 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:37.846 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088380+0000", 2024-03-19T11:31:37.846 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.846 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.846 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.846 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:37.846 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088381+0000", 2024-03-19T11:31:37.846 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0500000000000004e-07 2024-03-19T11:31:37.847 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.847 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.847 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:37.847 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088385+0000", 2024-03-19T11:31:37.847 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2370000000000003e-06 2024-03-19T11:31:37.847 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.847 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.847 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:37.847 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088392+0000", 2024-03-19T11:31:37.847 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4240000000000004e-06 2024-03-19T11:31:37.847 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.847 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.847 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:37.847 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088397+0000", 2024-03-19T11:31:37.847 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8829999999999997e-06 2024-03-19T11:31:37.848 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.848 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.848 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:37.848 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.122356+0000", 2024-03-19T11:31:37.848 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033958588999999997 2024-03-19T11:31:37.848 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.848 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.848 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:37.848 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.122364+0000", 2024-03-19T11:31:37.848 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0800000000000006e-06 2024-03-19T11:31:37.849 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.849 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.849 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:37.849 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.133275+0000", 2024-03-19T11:31:37.849 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010910585 2024-03-19T11:31:37.849 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.849 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:37.849 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.849 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.849 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.849 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:2286 4.15 e75/74 4:afa9f208:::smithi03829856-35:head v 75'131, mlcod=33'54)", 2024-03-19T11:31:37.849 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.089414+0000", 2024-03-19T11:31:37.849 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 43.712239252000003, 2024-03-19T11:31:37.849 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.045421256, 2024-03-19T11:31:37.850 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:37.850 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:37.850 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:37.850 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.850 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:37.850 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.089414+0000", 2024-03-19T11:31:37.850 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.850 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.850 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.850 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:37.850 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.089414+0000", 2024-03-19T11:31:37.850 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.850 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.850 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.850 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:37.851 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.089414+0000", 2024-03-19T11:31:37.851 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1100000000000005e-07 2024-03-19T11:31:37.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.851 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.851 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:37.851 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.089423+0000", 2024-03-19T11:31:37.851 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3799999999999994e-06 2024-03-19T11:31:37.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.851 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.851 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:37.851 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.089436+0000", 2024-03-19T11:31:37.851 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2931e-05 2024-03-19T11:31:37.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.851 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.851 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:37.852 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.089447+0000", 2024-03-19T11:31:37.852 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1044e-05 2024-03-19T11:31:37.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.852 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.852 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:37.852 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.133330+0000", 2024-03-19T11:31:37.852 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.043882833000000003 2024-03-19T11:31:37.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.852 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.852 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:37.852 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.133353+0000", 2024-03-19T11:31:37.853 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3629999999999999e-05 2024-03-19T11:31:37.853 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.853 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.853 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:37.853 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.134786+0000", 2024-03-19T11:31:37.853 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0014327980000000001 2024-03-19T11:31:37.853 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.853 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.853 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:37.853 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.134835+0000", 2024-03-19T11:31:37.853 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8829000000000003e-05 2024-03-19T11:31:37.853 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.853 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:37.853 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.853 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.854 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.854 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2802 4.28 4:15708482:::smithi03829856-23:head [write 497540~612664] snapc 19=[18,17,15,b] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-03-19T11:31:37.854 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:14.060002+0000", 2024-03-19T11:31:37.854 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.741651123, 2024-03-19T11:31:37.854 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.028374612, 2024-03-19T11:31:37.854 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:37.854 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:37.854 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:37.854 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:37.854 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:37.854 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2802 2024-03-19T11:31:37.854 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.854 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:37.854 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.855 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:37.855 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.060002+0000", 2024-03-19T11:31:37.855 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.855 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.855 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.855 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:37.855 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.060002+0000", 2024-03-19T11:31:37.855 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.855 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.855 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.855 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:37.855 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.060005+0000", 2024-03-19T11:31:37.855 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5130000000000001e-06 2024-03-19T11:31:37.855 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.855 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.856 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:37.856 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.061352+0000", 2024-03-19T11:31:37.856 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0013468060000000001 2024-03-19T11:31:37.856 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.856 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.856 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:37.856 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.061403+0000", 2024-03-19T11:31:37.856 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1181999999999999e-05 2024-03-19T11:31:37.856 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.856 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.856 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:37.856 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.061428+0000", 2024-03-19T11:31:37.856 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4715999999999998e-05 2024-03-19T11:31:37.856 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.857 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.857 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:37.857 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.061600+0000", 2024-03-19T11:31:37.857 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017224100000000001 2024-03-19T11:31:37.857 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.857 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.857 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op must wait for map", 2024-03-19T11:31:37.857 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.061619+0000", 2024-03-19T11:31:37.857 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9213000000000001e-05 2024-03-19T11:31:37.857 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.857 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.857 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:37.857 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.066953+0000", 2024-03-19T11:31:37.858 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053332149999999997 2024-03-19T11:31:37.858 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.858 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.858 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:37.858 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.067236+0000", 2024-03-19T11:31:37.858 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028381400000000002 2024-03-19T11:31:37.858 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.858 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.858 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:37.858 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.067698+0000", 2024-03-19T11:31:37.858 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00046152200000000002 2024-03-19T11:31:37.858 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.858 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.858 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:37.858 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.082795+0000", 2024-03-19T11:31:37.859 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015096729 2024-03-19T11:31:37.859 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.859 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.859 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:37.859 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.088033+0000", 2024-03-19T11:31:37.859 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052383259999999997 2024-03-19T11:31:37.859 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.859 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.859 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:37.859 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.088209+0000", 2024-03-19T11:31:37.859 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000175611 2024-03-19T11:31:37.859 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.859 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.859 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:37.859 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.088376+0000", 2024-03-19T11:31:37.860 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016772400000000001 2024-03-19T11:31:37.860 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.860 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:37.860 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.860 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.860 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.860 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2803 4.28 4:15708482:::smithi03829856-23:head [write 1840862~440060] snapc 19=[18,17,15,b] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-03-19T11:31:37.860 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:14.066423+0000", 2024-03-19T11:31:37.860 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.73522998, 2024-03-19T11:31:37.860 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022744533000000001, 2024-03-19T11:31:37.860 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:37.860 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:37.860 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:37.860 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:37.860 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:37.861 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2803 2024-03-19T11:31:37.861 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.861 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:37.861 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.861 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:37.861 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.066423+0000", 2024-03-19T11:31:37.861 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.861 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.861 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.861 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:37.861 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.066423+0000", 2024-03-19T11:31:37.861 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.861 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.861 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.862 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:37.862 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.066425+0000", 2024-03-19T11:31:37.862 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2500000000000001e-06 2024-03-19T11:31:37.862 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.862 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.862 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:37.862 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.067351+0000", 2024-03-19T11:31:37.862 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00092597000000000003 2024-03-19T11:31:37.862 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.862 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.862 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:37.862 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.067380+0000", 2024-03-19T11:31:37.862 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9326e-05 2024-03-19T11:31:37.863 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.863 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.863 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:37.863 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.067401+0000", 2024-03-19T11:31:37.863 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0458999999999998e-05 2024-03-19T11:31:37.863 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.863 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.863 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:37.863 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.078507+0000", 2024-03-19T11:31:37.863 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.01110651 2024-03-19T11:31:37.863 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.863 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.863 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:37.864 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.078694+0000", 2024-03-19T11:31:37.864 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000186334 2024-03-19T11:31:37.864 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.864 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.864 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:37.864 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.078997+0000", 2024-03-19T11:31:37.864 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00030362400000000001 2024-03-19T11:31:37.864 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.864 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.864 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:37.864 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.088954+0000", 2024-03-19T11:31:37.864 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0099565639999999993 2024-03-19T11:31:37.864 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.865 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.865 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:37.865 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.088978+0000", 2024-03-19T11:31:37.865 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4355999999999999e-05 2024-03-19T11:31:37.865 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.865 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.865 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:37.865 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.089067+0000", 2024-03-19T11:31:37.865 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8664000000000004e-05 2024-03-19T11:31:37.865 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.865 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.865 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:37.865 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.089167+0000", 2024-03-19T11:31:37.865 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100476 2024-03-19T11:31:37.865 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.866 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:37.866 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.866 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.866 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.866 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3102 4.15 4:afa9f208:::smithi03829856-35:head [write 484101~692450,stat] snapc 1f=[1f,1e,1d,1c,1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-19T11:31:37.866 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.553012+0000", 2024-03-19T11:31:37.866 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.248640822, 2024-03-19T11:31:37.866 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.024256340000000001, 2024-03-19T11:31:37.866 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:37.866 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:37.866 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:37.866 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:37.866 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:37.866 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3102 2024-03-19T11:31:37.867 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.867 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:37.867 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.867 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:37.867 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.553012+0000", 2024-03-19T11:31:37.867 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.867 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.867 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.867 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:37.867 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.553012+0000", 2024-03-19T11:31:37.867 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.867 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.867 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.867 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:37.867 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.553015+0000", 2024-03-19T11:31:37.868 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5200000000000002e-06 2024-03-19T11:31:37.868 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.868 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.868 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:37.868 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.554342+0000", 2024-03-19T11:31:37.868 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0013261760000000001 2024-03-19T11:31:37.868 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.868 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.868 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:37.868 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.554377+0000", 2024-03-19T11:31:37.868 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5299000000000002e-05 2024-03-19T11:31:37.868 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.868 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.868 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:37.869 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.554401+0000", 2024-03-19T11:31:37.869 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4360000000000001e-05 2024-03-19T11:31:37.869 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.869 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.869 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:37.869 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.554595+0000", 2024-03-19T11:31:37.869 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019390499999999999 2024-03-19T11:31:37.869 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.869 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.869 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:37.869 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.554801+0000", 2024-03-19T11:31:37.869 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020614799999999999 2024-03-19T11:31:37.869 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.869 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.869 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:37.870 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.555182+0000", 2024-03-19T11:31:37.870 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00038094700000000002 2024-03-19T11:31:37.870 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.870 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.870 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:37.870 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.569800+0000", 2024-03-19T11:31:37.870 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0146179 2024-03-19T11:31:37.870 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.870 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.870 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:37.870 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.576922+0000", 2024-03-19T11:31:37.870 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0071215339999999997 2024-03-19T11:31:37.870 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.870 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.871 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:37.871 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.577114+0000", 2024-03-19T11:31:37.871 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019218100000000001 2024-03-19T11:31:37.871 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.871 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.871 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:37.871 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.577268+0000", 2024-03-19T11:31:37.871 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015437 2024-03-19T11:31:37.872 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.872 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:37.872 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.872 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.872 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.872 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3103 4.15 4:afa9f208:::smithi03829856-35:head [write 1636889~447212] snapc 1f=[1f,1e,1d,1c,1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-19T11:31:37.872 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.557970+0000", 2024-03-19T11:31:37.872 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.243682435, 2024-03-19T11:31:37.872 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033363528000000003, 2024-03-19T11:31:37.872 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:37.872 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:37.872 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:37.872 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:37.872 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:37.873 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3103 2024-03-19T11:31:37.873 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.873 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:37.873 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.873 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:37.873 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.557970+0000", 2024-03-19T11:31:37.873 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.873 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.874 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.874 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:37.874 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.557970+0000", 2024-03-19T11:31:37.874 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.874 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.875 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.875 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:37.875 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.557973+0000", 2024-03-19T11:31:37.875 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6060000000000001e-06 2024-03-19T11:31:37.875 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.875 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.875 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:37.875 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558806+0000", 2024-03-19T11:31:37.875 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00083305499999999997 2024-03-19T11:31:37.875 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.875 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.875 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:37.875 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558843+0000", 2024-03-19T11:31:37.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6519000000000001e-05 2024-03-19T11:31:37.876 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.876 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.876 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:37.876 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558863+0000", 2024-03-19T11:31:37.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0622999999999999e-05 2024-03-19T11:31:37.876 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.876 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.876 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:37.876 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.562831+0000", 2024-03-19T11:31:37.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039681250000000003 2024-03-19T11:31:37.876 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.877 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.877 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:37.877 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.563003+0000", 2024-03-19T11:31:37.877 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000171638 2024-03-19T11:31:37.877 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.877 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.877 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:37.877 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.563335+0000", 2024-03-19T11:31:37.877 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033165899999999999 2024-03-19T11:31:37.877 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.877 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.877 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:37.877 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.569815+0000", 2024-03-19T11:31:37.877 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00648024 2024-03-19T11:31:37.878 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.878 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.878 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:37.878 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.590982+0000", 2024-03-19T11:31:37.878 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021167235 2024-03-19T11:31:37.878 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.878 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.878 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:37.878 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.591170+0000", 2024-03-19T11:31:37.878 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018796899999999999 2024-03-19T11:31:37.878 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.878 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.878 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:37.878 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.591334+0000", 2024-03-19T11:31:37.879 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000163859 2024-03-19T11:31:37.879 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.879 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:37.879 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.879 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.879 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.879 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3104 4.15 4:afa9f208:::smithi03829856-35:head [setxattr _header (57) in=7b,truncate 2084101] snapc 1f=[1f,1e,1d,1c,1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-19T11:31:37.879 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.558905+0000", 2024-03-19T11:31:37.879 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.242747353, 2024-03-19T11:31:37.879 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032860320999999998, 2024-03-19T11:31:37.879 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:37.879 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:37.879 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:37.879 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:37.880 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:37.880 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3104 2024-03-19T11:31:37.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.880 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:37.880 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.880 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:37.880 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558905+0000", 2024-03-19T11:31:37.880 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.880 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.880 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:37.880 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558905+0000", 2024-03-19T11:31:37.880 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.880 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.881 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:37.881 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558907+0000", 2024-03-19T11:31:37.881 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.547e-06 2024-03-19T11:31:37.881 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.881 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.881 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:37.881 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558916+0000", 2024-03-19T11:31:37.881 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0589999999999992e-06 2024-03-19T11:31:37.881 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.881 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.881 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:37.881 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558931+0000", 2024-03-19T11:31:37.881 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5262999999999999e-05 2024-03-19T11:31:37.881 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.881 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.882 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:37.882 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558947+0000", 2024-03-19T11:31:37.882 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6140000000000001e-05 2024-03-19T11:31:37.882 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.882 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.882 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:37.882 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.567458+0000", 2024-03-19T11:31:37.882 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085106650000000006 2024-03-19T11:31:37.882 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.883 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.883 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:37.883 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.567777+0000", 2024-03-19T11:31:37.883 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00031917399999999998 2024-03-19T11:31:37.883 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.883 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.883 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:37.883 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.568090+0000", 2024-03-19T11:31:37.883 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00031286900000000002 2024-03-19T11:31:37.883 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.883 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.883 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:37.884 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.576738+0000", 2024-03-19T11:31:37.884 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0086476379999999992 2024-03-19T11:31:37.884 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.884 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.884 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:37.884 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.591474+0000", 2024-03-19T11:31:37.884 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014736281 2024-03-19T11:31:37.884 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.884 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.884 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:37.884 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.591576+0000", 2024-03-19T11:31:37.884 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000102303 2024-03-19T11:31:37.884 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.884 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.884 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:37.885 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.591766+0000", 2024-03-19T11:31:37.885 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018938200000000001 2024-03-19T11:31:37.885 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.885 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:37.885 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.885 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.885 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.885 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3105 4.15 4:afa9f208:::smithi03829856-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e104)", 2024-03-19T11:31:37.885 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.558970+0000", 2024-03-19T11:31:37.885 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.242682918, 2024-03-19T11:31:37.885 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033553684, 2024-03-19T11:31:37.885 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:37.885 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:37.885 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:37.886 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:37.886 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:37.886 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3105 2024-03-19T11:31:37.886 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.886 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:37.886 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.886 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:37.886 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558970+0000", 2024-03-19T11:31:37.886 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.886 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.886 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.886 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:37.886 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558970+0000", 2024-03-19T11:31:37.886 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.886 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.887 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.887 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:37.887 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558971+0000", 2024-03-19T11:31:37.887 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4729999999999999e-06 2024-03-19T11:31:37.887 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.887 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.887 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:37.887 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558978+0000", 2024-03-19T11:31:37.887 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1369999999999996e-06 2024-03-19T11:31:37.887 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.887 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.887 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:37.887 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.559003+0000", 2024-03-19T11:31:37.887 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4525999999999999e-05 2024-03-19T11:31:37.887 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.888 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.888 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:37.888 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.559019+0000", 2024-03-19T11:31:37.888 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5999999999999999e-05 2024-03-19T11:31:37.888 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.888 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.888 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:37.888 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.569615+0000", 2024-03-19T11:31:37.888 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010596257 2024-03-19T11:31:37.888 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.888 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.888 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-03-19T11:31:37.888 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.569763+0000", 2024-03-19T11:31:37.888 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000147319 2024-03-19T11:31:37.888 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.889 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.889 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:37.889 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.592019+0000", 2024-03-19T11:31:37.889 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022256597999999999 2024-03-19T11:31:37.889 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.889 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.889 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:37.889 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.592250+0000", 2024-03-19T11:31:37.889 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023081100000000001 2024-03-19T11:31:37.889 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.889 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.889 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:37.889 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.592524+0000", 2024-03-19T11:31:37.889 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000273563 2024-03-19T11:31:37.890 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.890 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:37.890 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.890 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.890 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.890 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3106 4.15 4:ad3ad1b3:::smithi03829856-21:head [write 500908~484052] snapc 1f=[1f,1e,1d,1c,1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-19T11:31:37.890 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.563910+0000", 2024-03-19T11:31:37.890 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.237742967999999, 2024-03-19T11:31:37.890 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.029166066000000001, 2024-03-19T11:31:37.890 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:37.890 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:37.890 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:37.890 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:37.890 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:37.890 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3106 2024-03-19T11:31:37.891 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.891 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:37.891 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.891 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:37.891 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.563910+0000", 2024-03-19T11:31:37.891 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.891 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.891 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.891 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:37.891 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.563910+0000", 2024-03-19T11:31:37.891 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.891 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.891 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.891 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:37.892 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.563913+0000", 2024-03-19T11:31:37.892 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9960000000000002e-06 2024-03-19T11:31:37.892 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.892 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.892 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:37.892 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.564853+0000", 2024-03-19T11:31:37.892 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00093991599999999995 2024-03-19T11:31:37.892 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.892 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.892 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:37.892 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.564886+0000", 2024-03-19T11:31:37.892 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3315999999999997e-05 2024-03-19T11:31:37.892 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.892 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.892 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:37.893 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.564908+0000", 2024-03-19T11:31:37.893 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2262000000000001e-05 2024-03-19T11:31:37.893 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.893 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:37.893 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.569861+0000", 2024-03-19T11:31:37.893 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0049529070000000003 2024-03-19T11:31:37.893 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.893 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:37.893 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.569998+0000", 2024-03-19T11:31:37.893 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000136546 2024-03-19T11:31:37.893 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:37.894 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.570279+0000", 2024-03-19T11:31:37.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028119899999999999 2024-03-19T11:31:37.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:37.894 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.586586+0000", 2024-03-19T11:31:37.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016306621 2024-03-19T11:31:37.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:37.894 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.592743+0000", 2024-03-19T11:31:37.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061571150000000003 2024-03-19T11:31:37.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.895 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:37.895 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.592915+0000", 2024-03-19T11:31:37.895 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017273200000000001 2024-03-19T11:31:37.895 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.895 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.895 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:37.895 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.593076+0000", 2024-03-19T11:31:37.895 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016045600000000001 2024-03-19T11:31:37.895 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.895 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:37.895 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.895 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.895 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.895 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3107 4.15 4:ad3ad1b3:::smithi03829856-21:head [write 1411724~689184] snapc 1f=[1f,1e,1d,1c,1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-19T11:31:37.896 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.570058+0000", 2024-03-19T11:31:37.896 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.23159525, 2024-03-19T11:31:37.896 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023258826, 2024-03-19T11:31:37.896 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:37.896 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:37.896 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:37.896 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:37.896 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:37.896 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3107 2024-03-19T11:31:37.896 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.896 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:37.896 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.896 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:37.896 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.570058+0000", 2024-03-19T11:31:37.896 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.897 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.897 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.897 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:37.897 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.570058+0000", 2024-03-19T11:31:37.897 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.897 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.897 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.897 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:37.897 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.570061+0000", 2024-03-19T11:31:37.897 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0469999999999998e-06 2024-03-19T11:31:37.897 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.897 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.897 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:37.897 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572012+0000", 2024-03-19T11:31:37.898 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0019516760000000001 2024-03-19T11:31:37.898 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.898 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.898 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:37.898 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572052+0000", 2024-03-19T11:31:37.898 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9353000000000001e-05 2024-03-19T11:31:37.898 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.898 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.898 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:37.898 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572076+0000", 2024-03-19T11:31:37.898 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4459000000000001e-05 2024-03-19T11:31:37.898 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.898 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.899 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:37.899 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.577408+0000", 2024-03-19T11:31:37.899 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053316329999999997 2024-03-19T11:31:37.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.899 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.899 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:37.899 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.577697+0000", 2024-03-19T11:31:37.899 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028967499999999998 2024-03-19T11:31:37.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.899 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.899 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:37.899 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.578148+0000", 2024-03-19T11:31:37.899 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00045085900000000001 2024-03-19T11:31:37.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.900 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.900 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:37.900 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.586737+0000", 2024-03-19T11:31:37.900 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085889060000000003 2024-03-19T11:31:37.900 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.900 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.900 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:37.900 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.593175+0000", 2024-03-19T11:31:37.900 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0064375090000000001 2024-03-19T11:31:37.900 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.900 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.900 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:37.900 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.593235+0000", 2024-03-19T11:31:37.900 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0553000000000001e-05 2024-03-19T11:31:37.900 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.901 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.901 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:37.901 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.593316+0000", 2024-03-19T11:31:37.901 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1155999999999993e-05 2024-03-19T11:31:37.901 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.901 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:37.901 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.901 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.901 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.901 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3108 4.15 4:ad3ad1b3:::smithi03829856-21:head [setxattr _header (57) in=7b,truncate 2100908] snapc 1f=[1f,1e,1d,1c,1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-19T11:31:37.901 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.572119+0000", 2024-03-19T11:31:37.901 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.229534065999999, 2024-03-19T11:31:37.901 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021614495000000001, 2024-03-19T11:31:37.901 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:37.902 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:37.902 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:37.902 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:37.902 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:37.902 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3108 2024-03-19T11:31:37.902 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.902 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:37.902 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.902 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:37.902 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572119+0000", 2024-03-19T11:31:37.902 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.902 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.902 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.902 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:37.903 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572119+0000", 2024-03-19T11:31:37.903 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.903 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.903 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.903 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:37.903 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572121+0000", 2024-03-19T11:31:37.903 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.103e-06 2024-03-19T11:31:37.903 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.903 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.903 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:37.903 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572130+0000", 2024-03-19T11:31:37.903 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5230000000000001e-06 2024-03-19T11:31:37.903 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.903 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.903 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:37.904 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572157+0000", 2024-03-19T11:31:37.904 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6775999999999999e-05 2024-03-19T11:31:37.904 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.904 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.904 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:37.904 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572176+0000", 2024-03-19T11:31:37.904 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8516e-05 2024-03-19T11:31:37.904 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.904 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.904 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:37.904 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.584773+0000", 2024-03-19T11:31:37.904 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012596889 2024-03-19T11:31:37.904 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.904 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.905 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:37.905 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.585059+0000", 2024-03-19T11:31:37.905 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028658999999999998 2024-03-19T11:31:37.905 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.905 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.905 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:37.905 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.585591+0000", 2024-03-19T11:31:37.905 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00053221300000000002 2024-03-19T11:31:37.905 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.905 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.905 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:37.905 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.587798+0000", 2024-03-19T11:31:37.905 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0022066469999999999 2024-03-19T11:31:37.905 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.905 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.906 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:37.906 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.593611+0000", 2024-03-19T11:31:37.906 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0058127450000000002 2024-03-19T11:31:37.906 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.906 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.906 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:37.906 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.593671+0000", 2024-03-19T11:31:37.906 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0733000000000003e-05 2024-03-19T11:31:37.906 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.906 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.906 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:37.906 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.593733+0000", 2024-03-19T11:31:37.906 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1760000000000005e-05 2024-03-19T11:31:37.906 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.907 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:37.907 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.907 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.907 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.907 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3109 4.15 4:ad3ad1b3:::smithi03829856-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e104)", 2024-03-19T11:31:37.907 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.572203+0000", 2024-03-19T11:31:37.907 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.229449304999999, 2024-03-19T11:31:37.907 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021786163000000001, 2024-03-19T11:31:37.907 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:37.907 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:37.907 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:37.907 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:37.907 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:37.907 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3109 2024-03-19T11:31:37.907 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.908 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:37.908 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.908 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:37.908 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572203+0000", 2024-03-19T11:31:37.908 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.908 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.908 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.908 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:37.908 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572203+0000", 2024-03-19T11:31:37.908 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:37.908 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.908 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.908 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:37.908 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572205+0000", 2024-03-19T11:31:37.908 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.297e-06 2024-03-19T11:31:37.909 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.909 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.909 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:37.909 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572212+0000", 2024-03-19T11:31:37.909 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9129999999999997e-06 2024-03-19T11:31:37.909 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.909 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.909 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:37.909 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572233+0000", 2024-03-19T11:31:37.909 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1713e-05 2024-03-19T11:31:37.909 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.909 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.909 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:37.909 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572251+0000", 2024-03-19T11:31:37.909 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7112000000000002e-05 2024-03-19T11:31:37.910 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.910 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.910 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:37.910 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.587006+0000", 2024-03-19T11:31:37.910 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014755024 2024-03-19T11:31:37.910 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.910 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.910 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-03-19T11:31:37.910 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.587242+0000", 2024-03-19T11:31:37.910 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023635099999999999 2024-03-19T11:31:37.910 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.910 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.910 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:37.910 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.593797+0000", 2024-03-19T11:31:37.910 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0065553520000000004 2024-03-19T11:31:37.911 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.911 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.911 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:37.911 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.593880+0000", 2024-03-19T11:31:37.911 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2529999999999998e-05 2024-03-19T11:31:37.911 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:37.911 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:37.911 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:37.911 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.593990+0000", 2024-03-19T11:31:37.911 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010987100000000001 2024-03-19T11:31:37.911 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.911 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:37.911 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.911 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:37.911 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:37.912 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:37.913 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-19T11:31:37.913 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:37.922+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:37.921 INFO:tasks.rados.rados.0.smithi038.stdout:1562: finishing write tid 1 to smithi03829856-5 2024-03-19T11:31:37.921 INFO:tasks.rados.rados.0.smithi038.stdout:1562: finishing write tid 2 to smithi03829856-5 2024-03-19T11:31:37.922 INFO:tasks.rados.rados.0.smithi038.stdout:1562: finishing write tid 3 to smithi03829856-5 2024-03-19T11:31:37.922 INFO:tasks.rados.rados.0.smithi038.stdout:1562: finishing write tid 4 to smithi03829856-5 2024-03-19T11:31:37.922 INFO:tasks.rados.rados.0.smithi038.stdout:1562: oid 5 updating version 0 to 189 2024-03-19T11:31:37.922 INFO:tasks.rados.rados.0.smithi038.stdout:1562: oid 5 version 189 is already newer than 188 2024-03-19T11:31:37.922 INFO:tasks.rados.rados.0.smithi038.stdout:1562: oid 5 updating version 189 to 190 2024-03-19T11:31:37.922 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 5 v 190 (ObjNum 277 snap 36 seq_num 277) dirty exists 2024-03-19T11:31:37.922 INFO:tasks.rados.rados.0.smithi038.stdout:1562: left oid 5 (ObjNum 277 snap 36 seq_num 277) 2024-03-19T11:31:37.922 INFO:tasks.rados.rados.0.smithi038.stdout:1563: expect (ObjNum 209 snap 22 seq_num 209) 2024-03-19T11:31:37.994 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-19T11:31:38.014 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:38.022+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:38.094 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:38.094 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:31:38.094 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:31:38.094 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:38.108 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-19T11:31:38.126 INFO:tasks.rados.rados.0.smithi038.stdout:1553: done (10 left) 2024-03-19T11:31:38.126 INFO:tasks.rados.rados.0.smithi038.stdout:1554: done (9 left) 2024-03-19T11:31:38.126 INFO:tasks.rados.rados.0.smithi038.stdout:1555: done (8 left) 2024-03-19T11:31:38.127 INFO:tasks.rados.rados.0.smithi038.stdout:1557: done (7 left) 2024-03-19T11:31:38.127 INFO:tasks.rados.rados.0.smithi038.stdout:1558: done (6 left) 2024-03-19T11:31:38.127 INFO:tasks.rados.rados.0.smithi038.stdout:1559: done (5 left) 2024-03-19T11:31:38.127 INFO:tasks.rados.rados.0.smithi038.stdout:1560: done (4 left) 2024-03-19T11:31:38.127 INFO:tasks.rados.rados.0.smithi038.stdout:1561: done (3 left) 2024-03-19T11:31:38.127 INFO:tasks.rados.rados.0.smithi038.stdout:1562: done (2 left) 2024-03-19T11:31:38.127 INFO:tasks.rados.rados.0.smithi038.stdout:1563: done (1 left) 2024-03-19T11:31:38.127 INFO:tasks.rados.rados.0.smithi038.stdout:1564: done (0 left) 2024-03-19T11:31:38.127 INFO:tasks.rados.rados.0.smithi038.stdout:1565: read oid 39 snap -1 2024-03-19T11:31:38.128 INFO:tasks.rados.rados.0.smithi038.stdout:1565: expect deleted 2024-03-19T11:31:38.128 INFO:tasks.rados.rados.0.smithi038.stdout:1566: read oid 26 snap -1 2024-03-19T11:31:38.128 INFO:tasks.rados.rados.0.smithi038.stdout:1566: expect (ObjNum 267 snap 34 seq_num 267) 2024-03-19T11:31:38.128 INFO:tasks.rados.rados.0.smithi038.stdout:1567: read oid 7 snap -1 2024-03-19T11:31:38.128 INFO:tasks.rados.rados.0.smithi038.stdout:1567: expect (ObjNum 210 snap 22 seq_num 210) 2024-03-19T11:31:38.128 INFO:tasks.rados.rados.0.smithi038.stdout:1568: read oid 33 snap -1 2024-03-19T11:31:38.128 INFO:tasks.rados.rados.0.smithi038.stdout:1568: expect (ObjNum 273 snap 34 seq_num 273) 2024-03-19T11:31:38.128 INFO:tasks.rados.rados.0.smithi038.stdout:1569: rollback oid 2 current snap is 37 2024-03-19T11:31:38.128 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 2 to 26 2024-03-19T11:31:38.128 INFO:tasks.rados.rados.0.smithi038.stdout:1570: write oid 11 current snap is 37 2024-03-19T11:31:38.128 INFO:tasks.rados.rados.0.smithi038.stdout:1570: seq_num 278 ranges {672346=707930,2134700=537646} 2024-03-19T11:31:38.137 INFO:tasks.rados.rados.0.smithi038.stdout:1570: writing smithi03829856-11 from 672346 to 1380276 tid 1 2024-03-19T11:31:38.143 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:38.150+0000 7f1835403640 -1 osd.6 117 handle_osd_map injecting map crc failure 2024-03-19T11:31:38.144 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:38.154+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:38.145 INFO:tasks.rados.rados.0.smithi038.stdout:1570: writing smithi03829856-11 from 2134700 to 2672346 tid 2 2024-03-19T11:31:38.145 INFO:tasks.rados.rados.0.smithi038.stdout:1571: rollback oid 18 current snap is 37 2024-03-19T11:31:38.146 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 18 to 33 2024-03-19T11:31:38.146 INFO:tasks.rados.rados.0.smithi038.stdout:1568: expect (ObjNum 273 snap 34 seq_num 273) 2024-03-19T11:31:38.158 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:31:38.158 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:31:38.174 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:31:38.175 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:31:38.190 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:31:38.190 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:31:38.205 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:31:38.206 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:31:38.217 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:38.226+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:38.224 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:31:38.224 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:31:38.243 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:31:38.243 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:31:38.261 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:31:38.262 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:31:38.277 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:38.277 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:38.280 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:31:38.280 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:31:38.291 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-19T11:31:38.320 INFO:tasks.rados.rados.0.smithi038.stdout:1566: expect (ObjNum 267 snap 34 seq_num 267) 2024-03-19T11:31:38.337 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:38.346+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:38.343 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-19T11:31:38.343 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-19T11:31:38.343 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-03-19T11:31:38.343 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-19T11:31:38.343 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-19T11:31:38.343 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok config set filestore_inject_stall 3 2024-03-19T11:31:38.398 INFO:tasks.rados.rados.0.smithi038.stdout:1569: finishing rollback tid 0 to smithi03829856-2 2024-03-19T11:31:38.398 INFO:tasks.rados.rados.0.smithi038.stdout:1569: finishing rollback tid 1 to smithi03829856-2 2024-03-19T11:31:38.398 INFO:tasks.rados.rados.0.smithi038.stdout:1569: finishing rollback tid 2 to smithi03829856-2 2024-03-19T11:31:38.398 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 2 v 83 (ObjNum 206 snap 22 seq_num 206) dirty exists 2024-03-19T11:31:38.398 INFO:tasks.rados.rados.0.smithi038.stdout:1567: expect (ObjNum 210 snap 22 seq_num 210) 2024-03-19T11:31:38.433 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:38.442+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:38.456 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:38.456 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:38.471 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-19T11:31:38.502 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:38.502 INFO:teuthology.orchestra.run.smithi172.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-19T11:31:38.502 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:38.519 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:38.526+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:38.598 INFO:tasks.rados.rados.0.smithi038.stdout:1570: finishing write tid 1 to smithi03829856-11 2024-03-19T11:31:38.599 INFO:tasks.rados.rados.0.smithi038.stdout:1570: finishing write tid 2 to smithi03829856-11 2024-03-19T11:31:38.599 INFO:tasks.rados.rados.0.smithi038.stdout:1570: finishing write tid 3 to smithi03829856-11 2024-03-19T11:31:38.599 INFO:tasks.rados.rados.0.smithi038.stdout:1570: finishing write tid 4 to smithi03829856-11 2024-03-19T11:31:38.599 INFO:tasks.rados.rados.0.smithi038.stdout:1570: oid 11 updating version 0 to 120 2024-03-19T11:31:38.599 INFO:tasks.rados.rados.0.smithi038.stdout:1570: oid 11 version 120 is already newer than 119 2024-03-19T11:31:38.599 INFO:tasks.rados.rados.0.smithi038.stdout:1570: oid 11 updating version 120 to 121 2024-03-19T11:31:38.599 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 11 v 121 (ObjNum 278 snap 37 seq_num 278) dirty exists 2024-03-19T11:31:38.599 INFO:tasks.rados.rados.0.smithi038.stdout:1570: left oid 11 (ObjNum 278 snap 37 seq_num 278) 2024-03-19T11:31:38.599 INFO:tasks.rados.rados.0.smithi038.stdout:1571: finishing rollback tid 0 to smithi03829856-18 2024-03-19T11:31:38.599 INFO:tasks.rados.rados.0.smithi038.stdout:1571: finishing rollback tid 1 to smithi03829856-18 2024-03-19T11:31:38.599 INFO:tasks.rados.rados.0.smithi038.stdout:1571: finishing rollback tid 2 to smithi03829856-18 2024-03-19T11:31:38.599 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 18 v 164 (ObjNum 199 snap 22 seq_num 199) dirty exists 2024-03-19T11:31:38.599 INFO:tasks.rados.rados.0.smithi038.stdout:1565: done (6 left) 2024-03-19T11:31:38.600 INFO:tasks.rados.rados.0.smithi038.stdout:1566: done (5 left) 2024-03-19T11:31:38.600 INFO:tasks.rados.rados.0.smithi038.stdout:1567: done (4 left) 2024-03-19T11:31:38.600 INFO:tasks.rados.rados.0.smithi038.stdout:1568: done (3 left) 2024-03-19T11:31:38.600 INFO:tasks.rados.rados.0.smithi038.stdout:1569: done (2 left) 2024-03-19T11:31:38.600 INFO:tasks.rados.rados.0.smithi038.stdout:1570: done (1 left) 2024-03-19T11:31:38.600 INFO:tasks.rados.rados.0.smithi038.stdout:1571: done (0 left) 2024-03-19T11:31:38.600 INFO:tasks.rados.rados.0.smithi038.stdout:1572: read oid 40 snap 24 2024-03-19T11:31:38.600 INFO:tasks.rados.rados.0.smithi038.stdout:1572: expect (ObjNum 141 snap 15 seq_num 141) 2024-03-19T11:31:38.600 INFO:tasks.rados.rados.0.smithi038.stdout:1573: read oid 28 snap -1 2024-03-19T11:31:38.600 INFO:tasks.rados.rados.0.smithi038.stdout:1573: expect (ObjNum 87 snap 8 seq_num 87) 2024-03-19T11:31:38.600 INFO:tasks.rados.rados.0.smithi038.stdout:1574: write oid 9 current snap is 37 2024-03-19T11:31:38.600 INFO:tasks.rados.rados.0.smithi038.stdout:1574: seq_num 279 ranges {689153=783179,2076829=457413} 2024-03-19T11:31:38.618 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:38.628+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:38.621 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:38.622 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:38.622 INFO:tasks.rados.rados.0.smithi038.stdout:1574: writing smithi03829856-9 from 689153 to 1472332 tid 1 2024-03-19T11:31:38.638 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-19T11:31:38.642 INFO:tasks.rados.rados.0.smithi038.stdout:1574: writing smithi03829856-9 from 2076829 to 2534242 tid 2 2024-03-19T11:31:38.642 INFO:tasks.rados.rados.0.smithi038.stdout:1575: read oid 23 snap -1 2024-03-19T11:31:38.642 INFO:tasks.rados.rados.0.smithi038.stdout:1575: expect (ObjNum 274 snap 34 seq_num 274) 2024-03-19T11:31:38.642 INFO:tasks.rados.rados.0.smithi038.stdout:1576: set_chunk oid 37 target oid 2024-03-19T11:31:38.652 INFO:tasks.rados.rados.0.smithi038.stdout:1576: set_chunk oid 37 offset: 73728 length: 375808 target oid a5c29f405628e32298944d1643d270bc8133537caee5283271d2fe95475fcba3 offset: 0 snap -1 2024-03-19T11:31:38.652 INFO:tasks.rados.rados.0.smithi038.stdout:1577: read oid 22 snap -1 2024-03-19T11:31:38.652 INFO:tasks.rados.rados.0.smithi038.stdout:1577: expect (ObjNum 21 snap 0 seq_num 21) 2024-03-19T11:31:38.652 INFO:tasks.rados.rados.0.smithi038.stdout:1578: tier_flush oid 43 2024-03-19T11:31:38.652 INFO:tasks.rados.rados.0.smithi038.stdout:1578: tier_flush oid 43 snap -1 2024-03-19T11:31:38.652 INFO:tasks.rados.rados.0.smithi038.stdout:1572: expect (ObjNum 141 snap 15 seq_num 141) 2024-03-19T11:31:38.719 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:38.726+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:38.758 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:38.759 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:38.772 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-19T11:31:38.778 INFO:tasks.rados.rados.0.smithi038.stdout:1573: expect (ObjNum 87 snap 8 seq_num 87) 2024-03-19T11:31:38.860 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:38.870+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:38.883 INFO:tasks.rados.rados.0.smithi038.stdout:1574: finishing write tid 1 to smithi03829856-9 2024-03-19T11:31:38.883 INFO:tasks.rados.rados.0.smithi038.stdout:1574: finishing write tid 2 to smithi03829856-9 2024-03-19T11:31:38.883 INFO:tasks.rados.rados.0.smithi038.stdout:1574: finishing write tid 3 to smithi03829856-9 2024-03-19T11:31:38.883 INFO:tasks.rados.rados.0.smithi038.stdout:1574: finishing write tid 4 to smithi03829856-9 2024-03-19T11:31:38.883 INFO:tasks.rados.rados.0.smithi038.stdout:1574: oid 9 updating version 0 to 118 2024-03-19T11:31:38.883 INFO:tasks.rados.rados.0.smithi038.stdout:1574: oid 9 version 118 is already newer than 116 2024-03-19T11:31:38.883 INFO:tasks.rados.rados.0.smithi038.stdout:1574: oid 9 version 118 is already newer than 117 2024-03-19T11:31:38.884 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 9 v 118 (ObjNum 279 snap 37 seq_num 279) dirty exists 2024-03-19T11:31:38.884 INFO:tasks.rados.rados.0.smithi038.stdout:1574: left oid 9 (ObjNum 279 snap 37 seq_num 279) 2024-03-19T11:31:38.884 INFO:tasks.rados.rados.0.smithi038.stdout:1578: got (16) Device or resource busy 2024-03-19T11:31:38.884 INFO:tasks.rados.rados.0.smithi038.stdout:check_oldest_snap_flushed oid 43 v 146 (ObjNum 237 snap 29 seq_num 237) unflushed 29 2024-03-19T11:31:38.884 INFO:tasks.rados.rados.0.smithi038.stdout:1575: expect (ObjNum 274 snap 34 seq_num 274) 2024-03-19T11:31:38.921 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:38.930+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:38.937 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:38.937 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:38.951 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-19T11:31:39.023 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:39.032+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:39.027 INFO:tasks.rados.rados.0.smithi038.stdout:1576: finishing set_chunk to oid 37 2024-03-19T11:31:39.027 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 37 v 125 (ObjNum 236 snap 28 seq_num 236) dirty exists 2024-03-19T11:31:39.027 INFO:tasks.rados.rados.0.smithi038.stdout:1577: expect (ObjNum 21 snap 0 seq_num 21) 2024-03-19T11:31:39.086 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:39.086 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:39.099 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-19T11:31:39.102 INFO:tasks.rados.rados.0.smithi038.stdout:1572: done (6 left) 2024-03-19T11:31:39.102 INFO:tasks.rados.rados.0.smithi038.stdout:1573: done (5 left) 2024-03-19T11:31:39.102 INFO:tasks.rados.rados.0.smithi038.stdout:1574: done (4 left) 2024-03-19T11:31:39.102 INFO:tasks.rados.rados.0.smithi038.stdout:1575: done (3 left) 2024-03-19T11:31:39.102 INFO:tasks.rados.rados.0.smithi038.stdout:1576: done (2 left) 2024-03-19T11:31:39.102 INFO:tasks.rados.rados.0.smithi038.stdout:1577: done (1 left) 2024-03-19T11:31:39.102 INFO:tasks.rados.rados.0.smithi038.stdout:1578: done (0 left) 2024-03-19T11:31:39.102 INFO:tasks.rados.rados.0.smithi038.stdout:1579: write oid 49 current snap is 37 2024-03-19T11:31:39.102 INFO:tasks.rados.rados.0.smithi038.stdout:1579: seq_num 280 ranges {705960=458428,1735311=717424} 2024-03-19T11:31:39.103 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-03-19T11:31:39.111 INFO:tasks.rados.rados.0.smithi038.stdout:1579: writing smithi03829856-49 from 705960 to 1164388 tid 1 2024-03-19T11:31:39.123 INFO:tasks.rados.rados.0.smithi038.stdout:1579: writing smithi03829856-49 from 1735311 to 2452735 tid 2 2024-03-19T11:31:39.123 INFO:tasks.rados.rados.0.smithi038.stdout:1580: snap_remove snap 25 2024-03-19T11:31:39.124 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:39.134+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:39.217 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:39.218 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:39.224 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:39.234+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:39.232 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-19T11:31:39.327 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:39.336+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:39.388 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:39.388 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:39.401 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-19T11:31:39.446 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:39.456+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:39.529 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:39.540+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:39.569 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:39.569 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:39.584 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-19T11:31:39.630 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:39.638+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:39.730 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:39.740+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:39.748 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:39.748 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:39.763 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-19T11:31:39.849 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:39.860+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:39.930 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:39.931 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:39.932 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:39.941+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:39.944 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-19T11:31:40.033 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:40.041+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:40.062 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-03-19T11:31:40.092 INFO:tasks.rados.rados.0.smithi038.stdout:1580: done (1 left) 2024-03-19T11:31:40.092 INFO:tasks.rados.rados.0.smithi038.stdout:1581: rollback oid 27 current snap is 37 2024-03-19T11:31:40.092 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 27 to 31 2024-03-19T11:31:40.093 INFO:tasks.rados.rados.0.smithi038.stdout:1579: finishing write tid 1 to smithi03829856-49 2024-03-19T11:31:40.093 INFO:tasks.rados.rados.0.smithi038.stdout:1579: finishing write tid 2 to smithi03829856-49 2024-03-19T11:31:40.093 INFO:tasks.rados.rados.0.smithi038.stdout:1579: finishing write tid 3 to smithi03829856-49 2024-03-19T11:31:40.093 INFO:tasks.rados.rados.0.smithi038.stdout:1579: finishing write tid 4 to smithi03829856-49 2024-03-19T11:31:40.093 INFO:tasks.rados.rados.0.smithi038.stdout:1579: oid 49 updating version 0 to 54 2024-03-19T11:31:40.093 INFO:tasks.rados.rados.0.smithi038.stdout:1579: oid 49 version 54 is already newer than 53 2024-03-19T11:31:40.093 INFO:tasks.rados.rados.0.smithi038.stdout:1579: oid 49 updating version 54 to 55 2024-03-19T11:31:40.093 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 49 v 55 (ObjNum 280 snap 37 seq_num 280) dirty exists 2024-03-19T11:31:40.093 INFO:tasks.rados.rados.0.smithi038.stdout:1579: left oid 49 (ObjNum 280 snap 37 seq_num 280) 2024-03-19T11:31:40.093 INFO:tasks.rados.rados.0.smithi038.stdout:1579: done (1 left) 2024-03-19T11:31:40.093 INFO:tasks.rados.rados.0.smithi038.stdout:1582: read oid 20 snap -1 2024-03-19T11:31:40.093 INFO:tasks.rados.rados.0.smithi038.stdout:1582: expect deleted 2024-03-19T11:31:40.093 INFO:tasks.rados.rados.0.smithi038.stdout:1583: write oid 40 current snap is 37 2024-03-19T11:31:40.093 INFO:tasks.rados.rados.0.smithi038.stdout:1583: seq_num 281 ranges {722767=533677,1793793=461082} 2024-03-19T11:31:40.096 INFO:tasks.rados.rados.0.smithi038.stdout:1583: writing smithi03829856-40 from 722767 to 1256444 tid 1 2024-03-19T11:31:40.106 INFO:tasks.rados.rados.0.smithi038.stdout:1583: writing smithi03829856-40 from 1793793 to 2254875 tid 2 2024-03-19T11:31:40.106 INFO:tasks.rados.rados.0.smithi038.stdout:1584: tier_flush oid 10 2024-03-19T11:31:40.106 INFO:tasks.rados.rados.0.smithi038.stdout:1584: tier_flush oid 10 snap -1 2024-03-19T11:31:40.106 INFO:tasks.rados.rados.0.smithi038.stdout:1581: finishing rollback tid 1 to smithi03829856-27 2024-03-19T11:31:40.106 INFO:tasks.rados.rados.0.smithi038.stdout:1582: done (3 left) 2024-03-19T11:31:40.106 INFO:tasks.rados.rados.0.smithi038.stdout:1585: write oid 13 current snap is 37 2024-03-19T11:31:40.106 INFO:tasks.rados.rados.0.smithi038.stdout:1581: finishing rollback tid 2 to smithi03829856-27 2024-03-19T11:31:40.106 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 27 v 147 (ObjNum 240 snap 30 seq_num 240) dirty exists 2024-03-19T11:31:40.106 INFO:tasks.rados.rados.0.smithi038.stdout:1585: seq_num 282 ranges {739574=725279,1968628=721093} 2024-03-19T11:31:40.123 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:40.123 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:40.124 INFO:tasks.rados.rados.0.smithi038.stdout:1585: writing smithi03829856-13 from 739574 to 1464853 tid 1 2024-03-19T11:31:40.133 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:40.144+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:40.136 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-19T11:31:40.139 INFO:tasks.rados.rados.0.smithi038.stdout:1585: writing smithi03829856-13 from 1968628 to 2689721 tid 2 2024-03-19T11:31:40.139 INFO:tasks.rados.rados.0.smithi038.stdout:1581: done (3 left) 2024-03-19T11:31:40.139 INFO:tasks.rados.rados.0.smithi038.stdout:1586: read oid 5 snap -1 2024-03-19T11:31:40.139 INFO:tasks.rados.rados.0.smithi038.stdout:1586: expect (ObjNum 277 snap 36 seq_num 277) 2024-03-19T11:31:40.139 INFO:tasks.rados.rados.0.smithi038.stdout:1587: read oid 9 snap 33 2024-03-19T11:31:40.140 INFO:tasks.rados.rados.0.smithi038.stdout:1587: expect (ObjNum 258 snap 32 seq_num 258) 2024-03-19T11:31:40.140 INFO:tasks.rados.rados.0.smithi038.stdout:1588: read oid 35 snap -1 2024-03-19T11:31:40.140 INFO:tasks.rados.rados.0.smithi038.stdout:1588: expect deleted 2024-03-19T11:31:40.140 INFO:tasks.rados.rados.0.smithi038.stdout:1589: tier_evict oid 47 2024-03-19T11:31:40.140 INFO:tasks.rados.rados.0.smithi038.stdout:1589: tier_evict oid 47 snap -1 2024-03-19T11:31:40.140 INFO:tasks.rados.rados.0.smithi038.stdout:1584: got (16) Device or resource busy 2024-03-19T11:31:40.140 INFO:tasks.rados.rados.0.smithi038.stdout:check_oldest_snap_flushed oid 10 v 149 (ObjNum 179 snap 19 seq_num 179) unflushed 32 2024-03-19T11:31:40.140 INFO:tasks.rados.rados.0.smithi038.stdout:1583: finishing write tid 1 to smithi03829856-40 2024-03-19T11:31:40.140 INFO:tasks.rados.rados.0.smithi038.stdout:1583: finishing write tid 2 to smithi03829856-40 2024-03-19T11:31:40.141 INFO:tasks.rados.rados.0.smithi038.stdout:1584: done (6 left) 2024-03-19T11:31:40.141 INFO:tasks.rados.rados.0.smithi038.stdout:1590: read oid 24 snap -1 2024-03-19T11:31:40.141 INFO:tasks.rados.rados.0.smithi038.stdout:1590: expect (ObjNum 265 snap 34 seq_num 265) 2024-03-19T11:31:40.141 INFO:tasks.rados.rados.0.smithi038.stdout:1583: finishing write tid 3 to smithi03829856-40 2024-03-19T11:31:40.141 INFO:tasks.rados.rados.0.smithi038.stdout:1583: finishing write tid 4 to smithi03829856-40 2024-03-19T11:31:40.141 INFO:tasks.rados.rados.0.smithi038.stdout:1583: oid 40 updating version 0 to 86 2024-03-19T11:31:40.141 INFO:tasks.rados.rados.0.smithi038.stdout:1583: oid 40 version 86 is already newer than 84 2024-03-19T11:31:40.141 INFO:tasks.rados.rados.0.smithi038.stdout:1583: oid 40 version 86 is already newer than 85 2024-03-19T11:31:40.141 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 40 v 86 (ObjNum 281 snap 37 seq_num 281) dirty exists 2024-03-19T11:31:40.141 INFO:tasks.rados.rados.0.smithi038.stdout:1583: left oid 40 (ObjNum 281 snap 37 seq_num 281) 2024-03-19T11:31:40.141 INFO:tasks.rados.rados.0.smithi038.stdout:1591: read oid 49 snap -1 2024-03-19T11:31:40.141 INFO:tasks.rados.rados.0.smithi038.stdout:1591: expect (ObjNum 280 snap 37 seq_num 280) 2024-03-19T11:31:40.141 INFO:tasks.rados.rados.0.smithi038.stdout:1583: done (7 left) 2024-03-19T11:31:40.141 INFO:tasks.rados.rados.0.smithi038.stdout:1592: read oid 32 snap 24 2024-03-19T11:31:40.141 INFO:tasks.rados.rados.0.smithi038.stdout:1592: expect (ObjNum 31 snap 0 seq_num 31) 2024-03-19T11:31:40.142 INFO:tasks.rados.rados.0.smithi038.stdout:1593: write (excl) oid 20 current snap is 37 2024-03-19T11:31:40.142 INFO:tasks.rados.rados.0.smithi038.stdout:1593: seq_num 283 ranges {756381=400528,1910757=464751} 2024-03-19T11:31:40.149 INFO:tasks.rados.rados.0.smithi038.stdout:1593: writing smithi03829856-20 from 756381 to 1156909 tid 1 2024-03-19T11:31:40.163 INFO:tasks.rados.rados.0.smithi038.stdout:1593: writing smithi03829856-20 from 1910757 to 2375508 tid 2 2024-03-19T11:31:40.163 INFO:tasks.rados.rados.0.smithi038.stdout:1594: read oid 45 snap 33 2024-03-19T11:31:40.163 INFO:tasks.rados.rados.0.smithi038.stdout:1594: expect (ObjNum 212 snap 22 seq_num 212) 2024-03-19T11:31:40.163 INFO:tasks.rados.rados.0.smithi038.stdout:1595: read oid 30 snap -1 2024-03-19T11:31:40.163 INFO:tasks.rados.rados.0.smithi038.stdout:1595: expect (ObjNum 29 snap 0 seq_num 29) 2024-03-19T11:31:40.163 INFO:tasks.rados.rados.0.smithi038.stdout:1596: write (excl) oid 29 current snap is 37 2024-03-19T11:31:40.163 INFO:tasks.rados.rados.0.smithi038.stdout:1589: got (22) Invalid argument 2024-03-19T11:31:40.163 INFO:tasks.rados.rados.0.smithi038.stdout:1596: seq_num 284 ranges {773188=475777,1969239=724762} 2024-03-19T11:31:40.172 INFO:tasks.rados.rados.0.smithi038.stdout:1596: writing smithi03829856-29 from 773188 to 1248965 tid 1 2024-03-19T11:31:40.184 INFO:tasks.rados.rados.0.smithi038.stdout:1596: writing smithi03829856-29 from 1969239 to 2694001 tid 2 2024-03-19T11:31:40.184 INFO:tasks.rados.rados.0.smithi038.stdout:1588: done (11 left) 2024-03-19T11:31:40.184 INFO:tasks.rados.rados.0.smithi038.stdout:1589: done (10 left) 2024-03-19T11:31:40.184 INFO:tasks.rados.rados.0.smithi038.stdout:1597: read oid 1 snap 29 2024-03-19T11:31:40.184 INFO:tasks.rados.rados.0.smithi038.stdout:1597: expect (ObjNum 226 snap 25 seq_num 226) 2024-03-19T11:31:40.184 INFO:tasks.rados.rados.0.smithi038.stdout:1598: read oid 48 snap -1 2024-03-19T11:31:40.184 INFO:tasks.rados.rados.0.smithi038.stdout:1598: expect (ObjNum 255 snap 32 seq_num 255) 2024-03-19T11:31:40.184 INFO:tasks.rados.rados.0.smithi038.stdout:1599: read oid 31 snap -1 2024-03-19T11:31:40.184 INFO:tasks.rados.rados.0.smithi038.stdout:1599: expect (ObjNum 268 snap 34 seq_num 268) 2024-03-19T11:31:40.184 INFO:tasks.rados.rados.0.smithi038.stdout:1600: set_chunk oid 43 target oid 2024-03-19T11:31:40.198 INFO:tasks.rados.rados.0.smithi038.stdout:1600: set_chunk oid 43 offset: 2754048 length: 675328 target oid 46057cc02fa533a49f4f2ae1c36d2f3f17eb9ed1eb3ff0aea2b6ea40b8c17cd3 offset: 0 snap 33 2024-03-19T11:31:40.198 INFO:tasks.rados.rados.0.smithi038.stdout:1601: delete oid 47 current snap is 37 2024-03-19T11:31:40.198 INFO:tasks.rados.rados.0.smithi038.stdout:1590: expect (ObjNum 265 snap 34 seq_num 265) 2024-03-19T11:31:40.234 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:40.244+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:40.267 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:40.267 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:40.280 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-19T11:31:40.296 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-19T11:31:40.316 INFO:tasks.rados.rados.0.smithi038.stdout:1586: expect (ObjNum 277 snap 36 seq_num 277) 2024-03-19T11:31:40.360 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:40.372+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:40.437 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:40.437 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:40.437 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:40.445+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:40.451 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-19T11:31:40.464 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:31:40.465 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:31:40.492 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:31:40.492 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:31:40.508 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:31:40.509 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:31:40.520 INFO:tasks.rados.rados.0.smithi038.stdout:1585: finishing write tid 1 to smithi03829856-13 2024-03-19T11:31:40.520 INFO:tasks.rados.rados.0.smithi038.stdout:1585: finishing write tid 2 to smithi03829856-13 2024-03-19T11:31:40.520 INFO:tasks.rados.rados.0.smithi038.stdout:1585: finishing write tid 3 to smithi03829856-13 2024-03-19T11:31:40.520 INFO:tasks.rados.rados.0.smithi038.stdout:1585: finishing write tid 4 to smithi03829856-13 2024-03-19T11:31:40.520 INFO:tasks.rados.rados.0.smithi038.stdout:1585: oid 13 updating version 0 to 134 2024-03-19T11:31:40.520 INFO:tasks.rados.rados.0.smithi038.stdout:1585: oid 13 version 134 is already newer than 132 2024-03-19T11:31:40.521 INFO:tasks.rados.rados.0.smithi038.stdout:1585: oid 13 version 134 is already newer than 133 2024-03-19T11:31:40.521 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 13 v 134 (ObjNum 282 snap 37 seq_num 282) dirty exists 2024-03-19T11:31:40.521 INFO:tasks.rados.rados.0.smithi038.stdout:1585: left oid 13 (ObjNum 282 snap 37 seq_num 282) 2024-03-19T11:31:40.521 INFO:tasks.rados.rados.0.smithi038.stdout:1593: finishing write tid 1 to smithi03829856-20 2024-03-19T11:31:40.521 INFO:tasks.rados.rados.0.smithi038.stdout:1585: done (14 left) 2024-03-19T11:31:40.521 INFO:tasks.rados.rados.0.smithi038.stdout:1586: done (13 left) 2024-03-19T11:31:40.521 INFO:tasks.rados.rados.0.smithi038.stdout:1590: done (12 left) 2024-03-19T11:31:40.521 INFO:tasks.rados.rados.0.smithi038.stdout:1601: done (11 left) 2024-03-19T11:31:40.521 INFO:tasks.rados.rados.0.smithi038.stdout:1602: write oid 15 current snap is 37 2024-03-19T11:31:40.521 INFO:tasks.rados.rados.0.smithi038.stdout:1591: expect (ObjNum 280 snap 37 seq_num 280) 2024-03-19T11:31:40.525 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:31:40.525 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:31:40.541 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:40.548+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:40.545 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:31:40.546 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:31:40.567 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:31:40.567 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:31:40.579 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:40.579 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:31:40.579 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:31:40.579 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:40.588 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:31:40.589 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:31:40.592 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-19T11:31:40.609 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:31:40.609 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:31:40.672 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:40.684+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:40.716 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:40.716 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:31:40.716 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:31:40.716 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:31:40.716 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:40.729 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-19T11:31:40.769 INFO:tasks.rados.rados.0.smithi038.stdout:1595: expect (ObjNum 29 snap 0 seq_num 29) 2024-03-19T11:31:40.769 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:40.780+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:40.842 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:40.852+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:40.849 INFO:tasks.rados.rados.0.smithi038.stdout:1587: expect (ObjNum 258 snap 32 seq_num 258) 2024-03-19T11:31:40.904 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:40.904 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:31:40.905 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:31:40.905 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:31:40.905 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.905 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1067 4.0 4:01018511:::smithi03829856-19:head [write 477332~766101] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:31:40.905 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.950506+0000", 2024-03-19T11:31:40.905 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 86.961251837000006, 2024-03-19T11:31:40.905 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016279426, 2024-03-19T11:31:40.905 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:40.905 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:40.905 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:40.905 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:40.905 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:40.905 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1067 2024-03-19T11:31:40.905 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.906 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:40.906 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.906 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:40.906 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950506+0000", 2024-03-19T11:31:40.906 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:40.906 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.906 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.906 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:40.906 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950506+0000", 2024-03-19T11:31:40.906 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:40.906 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.906 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.906 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:40.906 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950509+0000", 2024-03-19T11:31:40.907 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.021e-06 2024-03-19T11:31:40.907 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.907 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.907 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:40.907 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950892+0000", 2024-03-19T11:31:40.907 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00038263300000000001 2024-03-19T11:31:40.907 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.907 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.907 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:40.907 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950912+0000", 2024-03-19T11:31:40.907 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0539000000000001e-05 2024-03-19T11:31:40.907 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.907 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.907 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:40.908 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950922+0000", 2024-03-19T11:31:40.908 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.577e-06 2024-03-19T11:31:40.908 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.908 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.908 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:40.908 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950982+0000", 2024-03-19T11:31:40.908 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0760000000000001e-05 2024-03-19T11:31:40.908 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.908 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.908 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:40.908 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.951058+0000", 2024-03-19T11:31:40.908 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5078000000000002e-05 2024-03-19T11:31:40.908 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.908 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.908 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:40.909 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.951173+0000", 2024-03-19T11:31:40.909 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011516200000000001 2024-03-19T11:31:40.909 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.909 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.909 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:40.909 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.955064+0000", 2024-03-19T11:31:40.909 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003890896 2024-03-19T11:31:40.909 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.909 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.909 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:40.909 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966670+0000", 2024-03-19T11:31:40.909 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011606795 2024-03-19T11:31:40.909 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.909 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.909 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:40.910 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966731+0000", 2024-03-19T11:31:40.910 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0528000000000001e-05 2024-03-19T11:31:40.910 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.910 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.910 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:40.910 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966785+0000", 2024-03-19T11:31:40.910 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4437000000000001e-05 2024-03-19T11:31:40.910 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:40.910 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:40.910 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:40.910 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.910 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.910 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(3.1 49/48 [PushOp(3:9205291d:::smithi03829856-50 low_tier:head, version: 28'36, data_included: [393216~851968,1703936~196608], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:9205291d:::smithi03829856-50 low_tier:head@28'36, size: 2823543, copy_subset: [(0, 2823543)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1900544, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:baab691a:::smithi03829856-2 low_tier:head, version: 28'37, data_included: [393216~524288,1507328~509479], data_size: 1033767, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:baab691a:::smithi03829856-2 low_tier:head@28'37, size: 2016807, copy_subset: [(0, 2016807)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2016807, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:98ce8426:::smithi03829856-4 low_tier:head, version: 28'38, data_included: [393216~720896,1638400~327680], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:98ce8426:::smithi03829856-4 low_tier:head@28'38, size: 2050421, copy_subset: [(0, 2050421)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1966080, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:a86be07b:::smithi03829856-44 low_tier:head, version: 28'41, data_included: [720896~655360,1769472~393216], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:a86be07b:::smithi03829856-44 low_tier:head@28'41, size: 2722701, copy_subset: [(0, 2722701)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2162688, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:bb56919e:::smithi03829856-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'44, data_included: [393216~786432,1703936~262144], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:bb56919e:::smithi03829856-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'44, size: 2806736, copy_subset: [(0, 2806736)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1966080, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:88687f71:::smithi03829856-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'45, data_included: [589824~655360,1638400~393216], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:88687f71:::smithi03829856-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'45, size: 2201684, copy_subset: [(0, 2201684)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2031616, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:b1f822cf:::smithi03829856-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'46, data_included: [720896~786432,1966080~262144], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:b1f822cf:::smithi03829856-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'46, size: 2369754, copy_subset: [(0, 2369754)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2228224, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:a5e8313a:::smithi03829856-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'47, data_included: [524288~720896,1769472~327680], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:a5e8313a:::smithi03829856-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'47, size: 2571438, copy_subset: [(0, 2571438)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:bca5ec64:::e3a26bf9230b1b95b8cf1a4cb01909236bf2bf3fd59b083b2291d278ab98264d:head, version: 28'48, data_included: [0~393216], data_size: 393216, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bca5ec64:::e3a26bf9230b1b95b8cf1a4cb01909236bf2bf3fd59b083b2291d278ab98264d:head@28'48, size: 533504, copy_subset: [(0, 533504)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 533504, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:40.911 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:26.948137+0000", 2024-03-19T11:31:40.911 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 73.963620621999993, 2024-03-19T11:31:40.911 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.036085578, 2024-03-19T11:31:40.911 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:40.911 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:40.911 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:40.911 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.911 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:40.911 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.948137+0000", 2024-03-19T11:31:40.911 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:40.911 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.911 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.911 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:40.911 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.948137+0000", 2024-03-19T11:31:40.912 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:40.912 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.912 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.912 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:40.912 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.948139+0000", 2024-03-19T11:31:40.912 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1399999999999998e-06 2024-03-19T11:31:40.912 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.912 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.912 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:40.912 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954288+0000", 2024-03-19T11:31:40.912 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0061487429999999999 2024-03-19T11:31:40.912 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.912 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.913 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:40.913 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954388+0000", 2024-03-19T11:31:40.913 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010022 2024-03-19T11:31:40.913 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.913 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.913 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:40.913 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954523+0000", 2024-03-19T11:31:40.913 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000134332 2024-03-19T11:31:40.913 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.913 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.913 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:40.913 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.955120+0000", 2024-03-19T11:31:40.913 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00059695799999999997 2024-03-19T11:31:40.913 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.913 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.914 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:40.914 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.955194+0000", 2024-03-19T11:31:40.914 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4077000000000002e-05 2024-03-19T11:31:40.914 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.914 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.914 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:40.914 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.984223+0000", 2024-03-19T11:31:40.914 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029029108000000001 2024-03-19T11:31:40.914 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:40.914 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:40.914 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:40.914 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.914 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.914 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(3.1 49/48 [PushOp(3:99a49d1b:::895a63407b6109b3cca9a6f5ef827134d56679bef42daadc5892269682c428ed:head, version: 32'49, data_included: [393216~388096], data_size: 388096, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:99a49d1b:::895a63407b6109b3cca9a6f5ef827134d56679bef42daadc5892269682c428ed:head@32'49, size: 781312, copy_subset: [(0, 781312)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 781312, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:40.915 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:26.954637+0000", 2024-03-19T11:31:40.915 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 73.957121005000005, 2024-03-19T11:31:40.915 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.030729057000000001, 2024-03-19T11:31:40.915 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:40.915 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:40.915 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:40.915 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.915 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:40.915 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954637+0000", 2024-03-19T11:31:40.915 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:40.915 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.915 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.915 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:40.915 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954637+0000", 2024-03-19T11:31:40.915 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:40.916 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.916 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.916 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:40.916 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954637+0000", 2024-03-19T11:31:40.916 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7299999999999995e-07 2024-03-19T11:31:40.916 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.916 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.916 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:40.916 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954873+0000", 2024-03-19T11:31:40.916 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023520299999999999 2024-03-19T11:31:40.916 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.916 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.917 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:40.917 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954897+0000", 2024-03-19T11:31:40.917 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4110000000000001e-05 2024-03-19T11:31:40.917 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.917 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.917 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:40.917 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954917+0000", 2024-03-19T11:31:40.917 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.067e-05 2024-03-19T11:31:40.917 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.917 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.917 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:40.917 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.984289+0000", 2024-03-19T11:31:40.917 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029371191000000001 2024-03-19T11:31:40.918 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.918 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.918 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:40.918 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.984303+0000", 2024-03-19T11:31:40.918 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4888999999999999e-05 2024-03-19T11:31:40.918 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.918 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.918 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:40.918 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.985366+0000", 2024-03-19T11:31:40.918 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001062321 2024-03-19T11:31:40.918 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:40.918 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:40.918 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:40.918 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.919 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.919 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1472 4.10 4:0dc3b91e:::smithi03829856-28:head [read 534528~391680] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-03-19T11:31:40.919 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:27.091748+0000", 2024-03-19T11:31:40.919 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 73.820010116999995, 2024-03-19T11:31:40.919 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.83574118500000005, 2024-03-19T11:31:40.919 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:40.919 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:40.919 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:40.919 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:40.919 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:40.919 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1472 2024-03-19T11:31:40.919 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.919 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:40.919 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.920 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:40.920 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091748+0000", 2024-03-19T11:31:40.920 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:40.920 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.920 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.920 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:40.920 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091748+0000", 2024-03-19T11:31:40.920 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:40.920 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.920 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.920 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:40.920 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091750+0000", 2024-03-19T11:31:40.920 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3130000000000001e-06 2024-03-19T11:31:40.920 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.920 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.921 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:40.921 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091760+0000", 2024-03-19T11:31:40.921 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0560999999999999e-05 2024-03-19T11:31:40.921 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.921 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.921 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:40.921 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091784+0000", 2024-03-19T11:31:40.921 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3915000000000001e-05 2024-03-19T11:31:40.921 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.921 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.921 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:40.921 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091797+0000", 2024-03-19T11:31:40.921 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2660000000000001e-05 2024-03-19T11:31:40.921 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.921 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.922 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:40.922 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091874+0000", 2024-03-19T11:31:40.922 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6757999999999999e-05 2024-03-19T11:31:40.922 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.922 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.922 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:40.922 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091882+0000", 2024-03-19T11:31:40.922 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5520000000000005e-06 2024-03-19T11:31:40.922 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.922 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.922 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:40.922 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.927033+0000", 2024-03-19T11:31:40.922 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.83515094099999998 2024-03-19T11:31:40.922 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.923 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.923 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:40.923 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.927250+0000", 2024-03-19T11:31:40.923 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00021675199999999999 2024-03-19T11:31:40.923 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.923 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.923 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:40.923 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.927489+0000", 2024-03-19T11:31:40.923 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023873300000000001 2024-03-19T11:31:40.923 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:40.923 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:40.923 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:40.923 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.923 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.923 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1474 4.10 4:0dc3b91e:::smithi03829856-28:head [assert-version v27,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-03-19T11:31:40.924 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:27.934139+0000", 2024-03-19T11:31:40.924 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 72.977618331000002, 2024-03-19T11:31:40.924 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024764884000000001, 2024-03-19T11:31:40.924 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:40.924 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:40.924 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:40.924 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:40.924 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:40.924 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1474 2024-03-19T11:31:40.924 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.924 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:40.924 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.924 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:40.924 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934139+0000", 2024-03-19T11:31:40.924 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:40.925 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.925 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.925 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:40.925 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934139+0000", 2024-03-19T11:31:40.925 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:40.925 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.925 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.925 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:40.925 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934142+0000", 2024-03-19T11:31:40.925 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5459999999999998e-06 2024-03-19T11:31:40.925 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.925 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.925 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:40.925 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934152+0000", 2024-03-19T11:31:40.925 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9399999999999997e-06 2024-03-19T11:31:40.926 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.926 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.926 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:40.926 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934173+0000", 2024-03-19T11:31:40.926 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1522000000000002e-05 2024-03-19T11:31:40.926 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.926 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.926 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:40.926 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934185+0000", 2024-03-19T11:31:40.926 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1946999999999999e-05 2024-03-19T11:31:40.926 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.926 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.926 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:40.926 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934282+0000", 2024-03-19T11:31:40.927 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6272000000000003e-05 2024-03-19T11:31:40.927 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.927 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.927 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:40.927 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934379+0000", 2024-03-19T11:31:40.927 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6966999999999995e-05 2024-03-19T11:31:40.927 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.927 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.927 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:40.927 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.955023+0000", 2024-03-19T11:31:40.927 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020644302 2024-03-19T11:31:40.927 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.928 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.928 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:40.928 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.955167+0000", 2024-03-19T11:31:40.928 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014404500000000001 2024-03-19T11:31:40.928 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.928 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.928 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1", 2024-03-19T11:31:40.928 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.957913+0000", 2024-03-19T11:31:40.928 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0027463779999999998 2024-03-19T11:31:40.928 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.928 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.928 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:40.928 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.958548+0000", 2024-03-19T11:31:40.928 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00063495700000000001 2024-03-19T11:31:40.929 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.929 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.929 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:40.929 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.958785+0000", 2024-03-19T11:31:40.929 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000236872 2024-03-19T11:31:40.929 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.929 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.929 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:40.929 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.958854+0000", 2024-03-19T11:31:40.929 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9099999999999999e-05 2024-03-19T11:31:40.929 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.929 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.929 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:40.929 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.958904+0000", 2024-03-19T11:31:40.929 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0036e-05 2024-03-19T11:31:40.930 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:40.930 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:40.930 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:40.930 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.930 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.930 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 75/74 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:7, version: 55'82, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@55'82, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 58'86, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@58'86, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [504210~968657,1572695~79919], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:12, version: 66'101, data_included: [440350~816629,1309629~231947], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:12@66'101, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:10, version: 70'112, data_included: [440350~816629,1309629~231947], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:10@70'112, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 70'113, data_included: [440350~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@70'113, size: 2605085, copy_subset: [(0, 2605085)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1488926, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:40.930 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.108941+0000", 2024-03-19T11:31:40.930 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 46.802816593999999, 2024-03-19T11:31:40.930 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.067446694000000001, 2024-03-19T11:31:40.930 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:40.930 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:40.930 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:40.930 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.931 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:40.931 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.108941+0000", 2024-03-19T11:31:40.931 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:40.931 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.931 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.931 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:40.931 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.108941+0000", 2024-03-19T11:31:40.931 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:40.931 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.931 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.931 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:40.931 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.108943+0000", 2024-03-19T11:31:40.931 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2259999999999999e-06 2024-03-19T11:31:40.931 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.932 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.932 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:40.932 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123892+0000", 2024-03-19T11:31:40.932 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014948655999999999 2024-03-19T11:31:40.932 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.932 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.932 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:40.932 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123993+0000", 2024-03-19T11:31:40.932 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000101236 2024-03-19T11:31:40.932 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.932 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.932 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:40.932 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124126+0000", 2024-03-19T11:31:40.932 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000133184 2024-03-19T11:31:40.932 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.933 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.933 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:40.933 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124655+0000", 2024-03-19T11:31:40.933 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00052849600000000004 2024-03-19T11:31:40.933 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.933 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.933 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:40.933 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124723+0000", 2024-03-19T11:31:40.933 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7797000000000002e-05 2024-03-19T11:31:40.933 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.933 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.933 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:40.933 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.176388+0000", 2024-03-19T11:31:40.933 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.051665098999999999 2024-03-19T11:31:40.933 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:40.934 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:40.934 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:40.934 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.934 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.934 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.21 e75,74 4:8401c1f0:::smithi03829856-43:8,46'66,4:866f3dcc:::smithi03829856-31:2,47'73)", 2024-03-19T11:31:40.934 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.124226+0000", 2024-03-19T11:31:40.934 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 46.787531938000001, 2024-03-19T11:31:40.934 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.052217913999999997, 2024-03-19T11:31:40.934 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:40.934 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:40.934 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:40.934 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.934 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:40.934 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124226+0000", 2024-03-19T11:31:40.935 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:40.935 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.935 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.935 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:40.935 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124226+0000", 2024-03-19T11:31:40.935 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:40.935 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.935 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.935 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:40.935 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124227+0000", 2024-03-19T11:31:40.935 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.477e-06 2024-03-19T11:31:40.935 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.935 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.935 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:40.936 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124240+0000", 2024-03-19T11:31:40.936 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2379e-05 2024-03-19T11:31:40.936 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.936 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.936 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:40.936 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124264+0000", 2024-03-19T11:31:40.936 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4224999999999999e-05 2024-03-19T11:31:40.936 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.936 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.936 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:40.936 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124287+0000", 2024-03-19T11:31:40.936 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3557000000000002e-05 2024-03-19T11:31:40.936 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.936 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.937 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:40.937 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.176274+0000", 2024-03-19T11:31:40.937 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.051986248999999998 2024-03-19T11:31:40.937 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.937 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.937 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:40.937 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.176285+0000", 2024-03-19T11:31:40.937 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0818999999999999e-05 2024-03-19T11:31:40.937 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.937 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.937 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:40.937 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.176444+0000", 2024-03-19T11:31:40.937 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015920800000000001 2024-03-19T11:31:40.937 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:40.937 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:40.938 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:40.938 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.938 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.938 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 75/74 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:7, version: 55'82, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@55'82, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 58'86, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@58'86, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [1652614~973634,2888511~52681,3182305~22261], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3204566, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:12, version: 66'101, data_included: [1541576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:12@66'101, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:10, version: 70'112, data_included: [1541576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:10@70'112, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 70'113, data_included: [1488926~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@70'113, size: 2605085, copy_subset: [(0, 2605085)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2537502, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1488926, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:40.938 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.195615+0000", 2024-03-19T11:31:40.938 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 46.716143041000002, 2024-03-19T11:31:40.938 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.036930510999999999, 2024-03-19T11:31:40.938 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:40.938 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:40.938 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:40.938 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.938 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:40.938 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.195615+0000", 2024-03-19T11:31:40.938 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:40.939 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.939 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.939 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:40.939 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.195615+0000", 2024-03-19T11:31:40.939 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:40.939 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.939 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.939 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:40.939 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.195617+0000", 2024-03-19T11:31:40.939 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0490000000000002e-06 2024-03-19T11:31:40.939 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.939 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.939 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:40.939 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204654+0000", 2024-03-19T11:31:40.939 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0090370339999999993 2024-03-19T11:31:40.940 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.940 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.940 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:40.940 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204730+0000", 2024-03-19T11:31:40.940 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5939000000000006e-05 2024-03-19T11:31:40.940 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.940 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.940 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:40.940 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204844+0000", 2024-03-19T11:31:40.940 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000114066 2024-03-19T11:31:40.940 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.940 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.940 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:40.940 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.205232+0000", 2024-03-19T11:31:40.940 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000388045 2024-03-19T11:31:40.941 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.941 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.941 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:40.941 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.205277+0000", 2024-03-19T11:31:40.941 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5071999999999998e-05 2024-03-19T11:31:40.941 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.941 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.941 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:40.941 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.232545+0000", 2024-03-19T11:31:40.941 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027268305999999999 2024-03-19T11:31:40.941 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:40.941 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:40.941 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:40.941 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.941 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.942 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.21 e75,74 4:866f3dcc:::smithi03829856-31:c,70'114)", 2024-03-19T11:31:40.942 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.204941+0000", 2024-03-19T11:31:40.942 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 46.706816416000002, 2024-03-19T11:31:40.942 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.028797390999999999, 2024-03-19T11:31:40.942 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:40.942 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:40.942 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:40.942 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.942 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:40.942 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204941+0000", 2024-03-19T11:31:40.942 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:40.942 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.943 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.943 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:40.943 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204941+0000", 2024-03-19T11:31:40.943 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:40.943 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.943 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.943 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:40.943 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204943+0000", 2024-03-19T11:31:40.943 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.35e-06 2024-03-19T11:31:40.943 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.943 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.943 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:40.943 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204950+0000", 2024-03-19T11:31:40.943 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.374e-06 2024-03-19T11:31:40.943 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.944 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.944 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:40.944 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204965+0000", 2024-03-19T11:31:40.944 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5169e-05 2024-03-19T11:31:40.944 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.944 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.944 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:40.944 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204979+0000", 2024-03-19T11:31:40.944 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4260000000000001e-05 2024-03-19T11:31:40.944 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.944 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.944 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:40.944 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.232607+0000", 2024-03-19T11:31:40.944 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027627518 2024-03-19T11:31:40.944 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.945 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.945 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:40.945 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.232626+0000", 2024-03-19T11:31:40.945 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9471999999999999e-05 2024-03-19T11:31:40.945 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.945 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.945 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:40.945 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.233739+0000", 2024-03-19T11:31:40.945 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011122479999999999 2024-03-19T11:31:40.945 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:40.945 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:40.945 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:40.945 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.945 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.946 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2789 3.2 3:48367551:::ca87bd16019841aaa831bb522ee079e5f4e6a1ea67fbcb51e5f89263b6c5d8f6:head [write 0~3280896] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-03-19T11:31:40.946 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:12.340289+0000", 2024-03-19T11:31:40.946 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 28.5714687, 2024-03-19T11:31:40.946 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016291298999999999, 2024-03-19T11:31:40.946 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:40.946 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:40.946 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:40.946 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:40.946 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:40.946 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2789 2024-03-19T11:31:40.946 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.946 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:40.946 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.946 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:40.947 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.340289+0000", 2024-03-19T11:31:40.947 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:40.947 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.947 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.947 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:40.947 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.340289+0000", 2024-03-19T11:31:40.947 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:40.947 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.947 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.947 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:40.947 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.340291+0000", 2024-03-19T11:31:40.947 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1270000000000001e-06 2024-03-19T11:31:40.947 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.947 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.947 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:40.948 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.341426+0000", 2024-03-19T11:31:40.948 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011348459999999999 2024-03-19T11:31:40.948 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.948 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.948 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:40.948 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.341454+0000", 2024-03-19T11:31:40.948 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8204e-05 2024-03-19T11:31:40.948 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.948 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.948 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:40.948 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.341467+0000", 2024-03-19T11:31:40.948 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2700999999999999e-05 2024-03-19T11:31:40.948 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.948 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.948 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:40.949 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.341623+0000", 2024-03-19T11:31:40.949 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015600100000000001 2024-03-19T11:31:40.949 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.949 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.949 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:40.949 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.342034+0000", 2024-03-19T11:31:40.949 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000411247 2024-03-19T11:31:40.949 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.949 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.949 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:31:40.949 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.342443+0000", 2024-03-19T11:31:40.949 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00040901400000000002 2024-03-19T11:31:40.949 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.949 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.949 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:40.950 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.356159+0000", 2024-03-19T11:31:40.950 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013716058999999999 2024-03-19T11:31:40.950 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.950 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.950 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:40.950 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.356462+0000", 2024-03-19T11:31:40.951 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000302594 2024-03-19T11:31:40.951 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.951 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.951 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:40.951 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.356530+0000", 2024-03-19T11:31:40.951 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7852e-05 2024-03-19T11:31:40.951 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.951 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.951 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:40.951 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.356580+0000", 2024-03-19T11:31:40.951 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0654e-05 2024-03-19T11:31:40.951 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:40.951 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:40.951 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:40.951 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.952 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.952 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3046 4.19 4:9c74b127:::smithi03829856-33:head [write 400066~716205,stat] snapc 1f=[1f,1e,1d,1c,1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-03-19T11:31:40.952 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:22.147457+0000", 2024-03-19T11:31:40.952 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.764300416000001, 2024-03-19T11:31:40.952 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016153588, 2024-03-19T11:31:40.952 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:40.952 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:40.952 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:40.952 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:40.952 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:40.952 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3046 2024-03-19T11:31:40.952 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.952 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:40.952 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.952 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:40.953 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.147457+0000", 2024-03-19T11:31:40.953 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:40.953 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.953 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.953 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:40.953 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.147457+0000", 2024-03-19T11:31:40.953 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:40.953 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.953 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.953 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:40.953 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.147459+0000", 2024-03-19T11:31:40.953 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1160000000000002e-06 2024-03-19T11:31:40.953 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.953 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.953 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:40.954 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.148083+0000", 2024-03-19T11:31:40.954 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00062361999999999997 2024-03-19T11:31:40.954 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.954 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.954 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:40.954 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.148100+0000", 2024-03-19T11:31:40.954 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6524999999999998e-05 2024-03-19T11:31:40.954 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.954 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.954 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:40.954 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.148110+0000", 2024-03-19T11:31:40.954 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0093e-05 2024-03-19T11:31:40.954 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.955 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.955 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:40.955 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.148197+0000", 2024-03-19T11:31:40.955 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7639000000000006e-05 2024-03-19T11:31:40.955 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.955 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.955 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:40.955 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.148359+0000", 2024-03-19T11:31:40.955 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000161953 2024-03-19T11:31:40.955 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.955 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.955 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 2", 2024-03-19T11:31:40.955 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.149074+0000", 2024-03-19T11:31:40.955 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00071444100000000001 2024-03-19T11:31:40.955 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.956 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.956 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:40.956 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.158856+0000", 2024-03-19T11:31:40.956 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0097818169999999999 2024-03-19T11:31:40.956 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.956 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.956 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:40.956 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.163525+0000", 2024-03-19T11:31:40.956 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046690270000000001 2024-03-19T11:31:40.956 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.956 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.956 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:40.956 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.163569+0000", 2024-03-19T11:31:40.956 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4279000000000001e-05 2024-03-19T11:31:40.956 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.956 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.957 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:40.957 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.163611+0000", 2024-03-19T11:31:40.957 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2077999999999999e-05 2024-03-19T11:31:40.957 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:40.957 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:40.957 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:40.957 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.957 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.957 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3228 4.19 4:9bd2ffb1:::smithi03829856-30:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e109)", 2024-03-19T11:31:40.957 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:28.084149+0000", 2024-03-19T11:31:40.957 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 12.827608437, 2024-03-19T11:31:40.957 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.049907076000000002, 2024-03-19T11:31:40.957 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:40.957 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:40.957 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:40.958 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:40.958 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:40.958 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3228 2024-03-19T11:31:40.958 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.958 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:40.958 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.958 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:40.958 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.084149+0000", 2024-03-19T11:31:40.958 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:40.958 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.958 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.958 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:40.958 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.084149+0000", 2024-03-19T11:31:40.958 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:40.959 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.959 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.959 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:40.959 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.084152+0000", 2024-03-19T11:31:40.959 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2409999999999998e-06 2024-03-19T11:31:40.959 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.959 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.959 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:40.959 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.084161+0000", 2024-03-19T11:31:40.959 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9609999999999994e-06 2024-03-19T11:31:40.959 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.959 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.959 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:40.959 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.084181+0000", 2024-03-19T11:31:40.960 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0786999999999999e-05 2024-03-19T11:31:40.960 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.960 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.960 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:40.960 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.084194+0000", 2024-03-19T11:31:40.960 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2824e-05 2024-03-19T11:31:40.960 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.960 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.960 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:40.960 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.086656+0000", 2024-03-19T11:31:40.960 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0024622480000000002 2024-03-19T11:31:40.960 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.960 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.960 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:40.960 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.086894+0000", 2024-03-19T11:31:40.960 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000237414 2024-03-19T11:31:40.961 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.961 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.961 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:40.961 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.134056+0000", 2024-03-19T11:31:40.961 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.047162600999999998 2024-03-19T11:31:40.961 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:40.961 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:40.961 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:40.961 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.961 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.961 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3361 4.19 4:9ef41243:::smithi03829856-5:head [write 420241~546489,stat] snapc 23=[23,22,21,1f,1e,1d,1c,1b,1a,18,17] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-03-19T11:31:40.961 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:32.244085+0000", 2024-03-19T11:31:40.961 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 8.6676730390000003, 2024-03-19T11:31:40.961 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025127221000000002, 2024-03-19T11:31:40.961 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:40.962 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:40.962 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:40.962 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:40.962 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:40.962 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3361 2024-03-19T11:31:40.962 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.962 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:40.962 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.962 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:40.962 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.244085+0000", 2024-03-19T11:31:40.962 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:40.962 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.962 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.962 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:40.963 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.244085+0000", 2024-03-19T11:31:40.963 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:40.963 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.963 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.963 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:40.963 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.244088+0000", 2024-03-19T11:31:40.963 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5779999999999999e-06 2024-03-19T11:31:40.963 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.963 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.963 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:40.963 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.244785+0000", 2024-03-19T11:31:40.963 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00069645700000000004 2024-03-19T11:31:40.963 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.963 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.963 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:40.964 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.244833+0000", 2024-03-19T11:31:40.964 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7902000000000003e-05 2024-03-19T11:31:40.964 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.964 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.964 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:40.964 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.244857+0000", 2024-03-19T11:31:40.964 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4301e-05 2024-03-19T11:31:40.964 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.964 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.964 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:40.964 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.245014+0000", 2024-03-19T11:31:40.964 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000157014 2024-03-19T11:31:40.964 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.964 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.964 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-03-19T11:31:40.965 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.245038+0000", 2024-03-19T11:31:40.965 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3655e-05 2024-03-19T11:31:40.965 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.965 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.965 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:40.965 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.251603+0000", 2024-03-19T11:31:40.965 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0065650650000000001 2024-03-19T11:31:40.965 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.965 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.965 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:40.965 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.251796+0000", 2024-03-19T11:31:40.965 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019291100000000001 2024-03-19T11:31:40.965 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.965 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.965 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 2", 2024-03-19T11:31:40.965 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.252201+0000", 2024-03-19T11:31:40.966 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00040590400000000001 2024-03-19T11:31:40.966 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.966 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.966 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:40.966 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.264409+0000", 2024-03-19T11:31:40.966 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012207314 2024-03-19T11:31:40.966 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.966 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.966 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:40.966 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.269042+0000", 2024-03-19T11:31:40.966 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046334949999999996 2024-03-19T11:31:40.966 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.966 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.966 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:40.967 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.269135+0000", 2024-03-19T11:31:40.967 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2310000000000002e-05 2024-03-19T11:31:40.967 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.967 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.967 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:40.967 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.269212+0000", 2024-03-19T11:31:40.967 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7315000000000001e-05 2024-03-19T11:31:40.967 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:40.967 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:40.967 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:40.967 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.967 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.967 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3399 4.21 4:866f3dcc:::smithi03829856-31:head [write 504276~639087,stat] snapc 23=[23,22,21,1f,1e,1d,1c,1b,1a,18,17] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-03-19T11:31:40.967 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:33.034450+0000", 2024-03-19T11:31:40.967 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 7.8773074269999999, 2024-03-19T11:31:40.968 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026707371000000001, 2024-03-19T11:31:40.968 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:40.968 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:40.968 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:40.968 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:40.968 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:40.968 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3399 2024-03-19T11:31:40.968 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.968 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:40.968 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.968 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:40.968 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.034450+0000", 2024-03-19T11:31:40.968 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:40.968 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.968 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.969 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:40.969 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.034450+0000", 2024-03-19T11:31:40.969 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:40.969 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.969 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.969 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:40.969 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.034452+0000", 2024-03-19T11:31:40.969 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.877e-06 2024-03-19T11:31:40.969 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.969 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.969 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:40.969 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.034786+0000", 2024-03-19T11:31:40.969 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00033416099999999997 2024-03-19T11:31:40.969 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.969 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.970 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:40.970 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.034813+0000", 2024-03-19T11:31:40.970 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6792e-05 2024-03-19T11:31:40.970 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.970 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.970 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:40.970 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.034828+0000", 2024-03-19T11:31:40.970 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5285999999999999e-05 2024-03-19T11:31:40.970 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.970 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.970 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:40.970 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.034947+0000", 2024-03-19T11:31:40.970 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011857 2024-03-19T11:31:40.970 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.970 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.971 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:40.971 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.035325+0000", 2024-03-19T11:31:40.971 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037801499999999999 2024-03-19T11:31:40.971 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.971 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.971 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1", 2024-03-19T11:31:40.971 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.035660+0000", 2024-03-19T11:31:40.971 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00033508499999999999 2024-03-19T11:31:40.971 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.971 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.971 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:40.971 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.043413+0000", 2024-03-19T11:31:40.971 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077528110000000001 2024-03-19T11:31:40.971 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.971 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.972 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:40.972 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.060284+0000", 2024-03-19T11:31:40.972 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016870823 2024-03-19T11:31:40.972 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.972 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.972 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:40.972 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.061003+0000", 2024-03-19T11:31:40.972 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000719687 2024-03-19T11:31:40.972 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.972 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.972 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:40.972 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.061158+0000", 2024-03-19T11:31:40.972 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015426400000000001 2024-03-19T11:31:40.972 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:40.972 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:40.973 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:40.973 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.973 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.973 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3400 4.21 4:866f3dcc:::smithi03829856-31:head [write 1666233~695410] snapc 23=[23,22,21,1f,1e,1d,1c,1b,1a,18,17] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-03-19T11:31:40.973 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:33.043656+0000", 2024-03-19T11:31:40.973 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 7.868102146, 2024-03-19T11:31:40.973 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018155691000000002, 2024-03-19T11:31:40.973 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:40.973 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:40.973 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:40.973 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:40.973 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:40.973 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3400 2024-03-19T11:31:40.973 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.973 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:40.974 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.974 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:40.974 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.043656+0000", 2024-03-19T11:31:40.974 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:40.974 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.974 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.974 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:40.974 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.043656+0000", 2024-03-19T11:31:40.974 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:40.974 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.974 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.974 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:40.974 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.043658+0000", 2024-03-19T11:31:40.974 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5239999999999999e-06 2024-03-19T11:31:40.974 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.975 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.975 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:40.975 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044163+0000", 2024-03-19T11:31:40.975 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00050451399999999996 2024-03-19T11:31:40.975 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.975 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.975 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:40.975 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044205+0000", 2024-03-19T11:31:40.975 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2128000000000001e-05 2024-03-19T11:31:40.975 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.975 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.975 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:40.975 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044229+0000", 2024-03-19T11:31:40.975 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4097999999999999e-05 2024-03-19T11:31:40.976 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.976 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.976 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:40.976 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044414+0000", 2024-03-19T11:31:40.976 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00018484600000000001 2024-03-19T11:31:40.976 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.976 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.976 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:40.976 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044628+0000", 2024-03-19T11:31:40.976 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00021459499999999999 2024-03-19T11:31:40.976 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.976 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.976 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1", 2024-03-19T11:31:40.976 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044909+0000", 2024-03-19T11:31:40.976 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00028030399999999999 2024-03-19T11:31:40.976 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.977 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.977 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:40.977 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.061219+0000", 2024-03-19T11:31:40.977 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016310495000000001 2024-03-19T11:31:40.977 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.977 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.977 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:40.977 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.061362+0000", 2024-03-19T11:31:40.977 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014293500000000001 2024-03-19T11:31:40.977 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.977 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.977 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:40.977 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.061671+0000", 2024-03-19T11:31:40.977 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030884599999999999 2024-03-19T11:31:40.977 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.978 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.978 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:40.978 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.061811+0000", 2024-03-19T11:31:40.978 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014040600000000001 2024-03-19T11:31:40.978 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:40.978 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:40.978 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:40.978 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.978 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.978 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3401 4.21 4:866f3dcc:::smithi03829856-31:head [setxattr _header (57) in=7b,truncate 2504276] snapc 23=[23,22,21,1f,1e,1d,1c,1b,1a,18,17] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-03-19T11:31:40.978 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:33.044315+0000", 2024-03-19T11:31:40.978 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 7.8674432000000003, 2024-03-19T11:31:40.978 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.021591882, 2024-03-19T11:31:40.978 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:40.978 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "waiting for sub ops", 2024-03-19T11:31:40.979 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:40.979 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:40.979 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:40.979 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3401 2024-03-19T11:31:40.979 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.979 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:40.979 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.979 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:40.979 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044315+0000", 2024-03-19T11:31:40.979 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:40.979 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.979 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.979 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:40.979 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044315+0000", 2024-03-19T11:31:40.980 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:40.980 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.980 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.980 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:40.980 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044317+0000", 2024-03-19T11:31:40.980 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.097e-06 2024-03-19T11:31:40.980 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.980 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.980 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:40.980 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044328+0000", 2024-03-19T11:31:40.980 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1362000000000001e-05 2024-03-19T11:31:40.980 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.980 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.980 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:40.981 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044354+0000", 2024-03-19T11:31:40.981 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6268999999999999e-05 2024-03-19T11:31:40.981 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.981 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.981 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:40.981 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044371+0000", 2024-03-19T11:31:40.981 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7028999999999999e-05 2024-03-19T11:31:40.981 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.981 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.981 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:40.981 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.052493+0000", 2024-03-19T11:31:40.981 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0081214270000000005 2024-03-19T11:31:40.981 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.981 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.981 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:40.982 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.052874+0000", 2024-03-19T11:31:40.982 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000381665 2024-03-19T11:31:40.982 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.982 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.982 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1", 2024-03-19T11:31:40.982 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.053343+0000", 2024-03-19T11:31:40.982 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00046883 2024-03-19T11:31:40.982 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.982 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.982 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:40.982 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.061993+0000", 2024-03-19T11:31:40.982 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0086501479999999999 2024-03-19T11:31:40.982 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.982 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.982 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:40.983 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.065906+0000", 2024-03-19T11:31:40.983 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0039130550000000004 2024-03-19T11:31:40.983 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:40.983 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:40.983 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:40.983 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.983 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.983 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3402 4.21 4:866f3dcc:::smithi03829856-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e112)", 2024-03-19T11:31:40.983 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:33.044411+0000", 2024-03-19T11:31:40.983 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 7.8673465589999996, 2024-03-19T11:31:40.983 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.034955695000000002, 2024-03-19T11:31:40.983 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:40.983 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-03-19T11:31:40.983 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:40.983 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:40.984 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:40.984 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3402 2024-03-19T11:31:40.984 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.984 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:40.984 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.984 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:40.984 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044411+0000", 2024-03-19T11:31:40.984 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:40.984 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.984 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.984 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:40.984 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044411+0000", 2024-03-19T11:31:40.984 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:40.984 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.984 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.985 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:40.985 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044413+0000", 2024-03-19T11:31:40.985 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.517e-06 2024-03-19T11:31:40.985 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.985 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.985 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:40.985 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044420+0000", 2024-03-19T11:31:40.985 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3799999999999996e-06 2024-03-19T11:31:40.985 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.985 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.985 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:40.985 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044466+0000", 2024-03-19T11:31:40.985 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5713000000000001e-05 2024-03-19T11:31:40.985 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.985 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.986 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:40.986 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044484+0000", 2024-03-19T11:31:40.986 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8128999999999999e-05 2024-03-19T11:31:40.986 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.986 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.986 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:40.986 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.055330+0000", 2024-03-19T11:31:40.986 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01084574 2024-03-19T11:31:40.986 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.986 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.986 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-03-19T11:31:40.986 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.055450+0000", 2024-03-19T11:31:40.986 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000120104 2024-03-19T11:31:40.986 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.986 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.987 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:40.987 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.079347+0000", 2024-03-19T11:31:40.987 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.023897049 2024-03-19T11:31:40.987 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.987 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.987 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:40.987 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.079367+0000", 2024-03-19T11:31:40.987 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0063000000000001e-05 2024-03-19T11:31:40.987 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:40.987 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:40.987 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:40.987 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.987 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.987 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3403 4.19 4:9b248e68:::smithi03829856-50:head [sparse-read 0~1696039,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e112)", 2024-03-19T11:31:40.987 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:33.044524+0000", 2024-03-19T11:31:40.988 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 7.8672335750000002, 2024-03-19T11:31:40.988 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016698992999999999, 2024-03-19T11:31:40.988 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:40.988 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:40.988 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:40.988 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:40.988 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:40.988 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3403 2024-03-19T11:31:40.988 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.988 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:40.988 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.988 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:40.988 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044524+0000", 2024-03-19T11:31:40.988 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:40.988 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.988 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.989 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:40.989 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044524+0000", 2024-03-19T11:31:40.989 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:40.989 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.989 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.989 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:40.989 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044526+0000", 2024-03-19T11:31:40.989 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4470000000000001e-06 2024-03-19T11:31:40.989 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.989 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.989 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:40.989 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044535+0000", 2024-03-19T11:31:40.989 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7450000000000007e-06 2024-03-19T11:31:40.989 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.989 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.990 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:40.990 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044562+0000", 2024-03-19T11:31:40.990 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6548999999999999e-05 2024-03-19T11:31:40.990 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.990 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.990 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:40.990 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044581+0000", 2024-03-19T11:31:40.990 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9468999999999999e-05 2024-03-19T11:31:40.990 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.990 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.990 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:40.990 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.055503+0000", 2024-03-19T11:31:40.990 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010921461 2024-03-19T11:31:40.990 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.990 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.991 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:40.991 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.055616+0000", 2024-03-19T11:31:40.991 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011270899999999999 2024-03-19T11:31:40.991 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.991 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.991 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:40.991 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.061223+0000", 2024-03-19T11:31:40.991 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005607613 2024-03-19T11:31:40.991 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:40.991 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:40.991 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:40.991 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.991 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.991 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3404 4.19 4:9b248e68:::smithi03829856-50:head [sparse-read 0~1696039] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e112)", 2024-03-19T11:31:40.991 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:33.044604+0000", 2024-03-19T11:31:40.992 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 7.8671532529999997, 2024-03-19T11:31:40.992 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017390282, 2024-03-19T11:31:40.992 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:40.992 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:40.992 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:40.992 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:40.992 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:40.992 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3404 2024-03-19T11:31:40.992 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.992 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:40.992 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.992 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:40.992 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044604+0000", 2024-03-19T11:31:40.992 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:40.992 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.993 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.993 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:40.993 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044604+0000", 2024-03-19T11:31:40.993 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:40.993 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.993 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.993 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:40.993 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044606+0000", 2024-03-19T11:31:40.993 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.393e-06 2024-03-19T11:31:40.993 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.993 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.993 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:40.993 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044616+0000", 2024-03-19T11:31:40.993 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0322999999999999e-05 2024-03-19T11:31:40.993 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.994 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.994 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:40.994 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044636+0000", 2024-03-19T11:31:40.994 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9819000000000001e-05 2024-03-19T11:31:40.994 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.994 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.994 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:40.994 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044653+0000", 2024-03-19T11:31:40.994 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7232999999999999e-05 2024-03-19T11:31:40.994 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.994 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.994 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:40.994 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.061515+0000", 2024-03-19T11:31:40.994 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016861322000000002 2024-03-19T11:31:40.994 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.995 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.995 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:40.995 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.061645+0000", 2024-03-19T11:31:40.995 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013014999999999999 2024-03-19T11:31:40.995 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.995 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.995 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:40.995 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.061995+0000", 2024-03-19T11:31:40.995 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00035004200000000001 2024-03-19T11:31:40.995 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:40.995 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:40.995 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:40.995 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.995 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.995 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3405 4.19 4:9b248e68:::smithi03829856-50:head [sparse-read 0~1696039] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e112)", 2024-03-19T11:31:40.996 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:33.044707+0000", 2024-03-19T11:31:40.996 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 7.8670508249999997, 2024-03-19T11:31:40.996 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018552987999999999, 2024-03-19T11:31:40.996 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:40.996 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:40.996 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:40.996 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:40.996 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:40.996 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3405 2024-03-19T11:31:40.996 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.996 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:40.996 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.996 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:40.996 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044707+0000", 2024-03-19T11:31:40.996 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:40.996 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.997 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.997 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:40.997 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044707+0000", 2024-03-19T11:31:40.997 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:40.997 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.997 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.997 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:40.997 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044708+0000", 2024-03-19T11:31:40.997 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.356e-06 2024-03-19T11:31:40.997 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.997 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.997 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:40.997 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044715+0000", 2024-03-19T11:31:40.997 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6900000000000003e-06 2024-03-19T11:31:40.997 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.998 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.998 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:40.998 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044736+0000", 2024-03-19T11:31:40.998 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0948999999999999e-05 2024-03-19T11:31:40.998 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.998 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.998 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:40.998 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044751+0000", 2024-03-19T11:31:40.998 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5389000000000001e-05 2024-03-19T11:31:40.998 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.998 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.998 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:40.998 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.062070+0000", 2024-03-19T11:31:40.998 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017318291 2024-03-19T11:31:40.998 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.999 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.999 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:40.999 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.062309+0000", 2024-03-19T11:31:40.999 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000239573 2024-03-19T11:31:40.999 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:40.999 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:40.999 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:40.999 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.063260+0000", 2024-03-19T11:31:40.999 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00095074000000000005 2024-03-19T11:31:40.999 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:40.999 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:40.999 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:40.999 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:40.999 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:40.999 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:41.001 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-19T11:31:41.001 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:41.012+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:41.011 INFO:tasks.rados.rados.0.smithi038.stdout:1592: expect (ObjNum 31 snap 0 seq_num 31) 2024-03-19T11:31:41.069 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:41.080+0000 7f6a7f9da640 -1 osd.1 119 handle_osd_map injecting map crc failure 2024-03-19T11:31:41.089 INFO:tasks.rados.rados.0.smithi038.stdout:1593: finishing write tid 2 to smithi03829856-20 2024-03-19T11:31:41.090 INFO:tasks.rados.rados.0.smithi038.stdout:1593: finishing write tid 3 to smithi03829856-20 2024-03-19T11:31:41.090 INFO:tasks.rados.rados.0.smithi038.stdout:1593: finishing write tid 4 to smithi03829856-20 2024-03-19T11:31:41.090 INFO:tasks.rados.rados.0.smithi038.stdout:1593: oid 20 updating version 0 to 262 2024-03-19T11:31:41.090 INFO:tasks.rados.rados.0.smithi038.stdout:1593: oid 20 updating version 262 to 264 2024-03-19T11:31:41.090 INFO:tasks.rados.rados.0.smithi038.stdout:1593: oid 20 version 264 is already newer than 263 2024-03-19T11:31:41.090 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 20 v 264 (ObjNum 283 snap 37 seq_num 283) dirty exists 2024-03-19T11:31:41.090 INFO:tasks.rados.rados.0.smithi038.stdout:1593: left oid 20 (ObjNum 283 snap 37 seq_num 283) 2024-03-19T11:31:41.090 INFO:tasks.rados.rados.0.smithi038.stdout:1594: expect (ObjNum 212 snap 22 seq_num 212) 2024-03-19T11:31:41.102 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:41.109+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:41.136 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:41.136 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:31:41.136 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:31:41.136 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:41.150 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-19T11:31:41.203 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:41.212+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:41.303 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:41.312+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:41.317 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:41.318 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:31:41.318 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:31:41.318 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:31:41.318 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:41.320 INFO:tasks.rados.rados.0.smithi038.stdout:1599: expect (ObjNum 268 snap 34 seq_num 268) 2024-03-19T11:31:41.332 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-19T11:31:41.404 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:41.413+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:41.452 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:41.452 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:31:41.452 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:31:41.452 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:31:41.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.452 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:851 4.9 4:953e720b:::smithi03829856-45:head [write 591613~454162,stat] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:31:41.452 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.763514+0000", 2024-03-19T11:31:41.452 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 92.697253575000005, 2024-03-19T11:31:41.452 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027825888999999999, 2024-03-19T11:31:41.453 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:41.453 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:41.453 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:41.453 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:41.453 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:41.453 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 851 2024-03-19T11:31:41.453 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.453 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:41.453 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.453 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:41.453 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763514+0000", 2024-03-19T11:31:41.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:41.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:41.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763514+0000", 2024-03-19T11:31:41.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:41.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:41.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763517+0000", 2024-03-19T11:31:41.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8499999999999998e-06 2024-03-19T11:31:41.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:41.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.764714+0000", 2024-03-19T11:31:41.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011971270000000001 2024-03-19T11:31:41.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:41.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.764745+0000", 2024-03-19T11:31:41.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0546999999999997e-05 2024-03-19T11:31:41.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:41.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.764767+0000", 2024-03-19T11:31:41.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2286999999999999e-05 2024-03-19T11:31:41.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:41.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.773924+0000", 2024-03-19T11:31:41.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0091567360000000004 2024-03-19T11:31:41.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.456 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:41.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.774217+0000", 2024-03-19T11:31:41.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029350099999999999 2024-03-19T11:31:41.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.456 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-19T11:31:41.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.774770+0000", 2024-03-19T11:31:41.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00055288600000000005 2024-03-19T11:31:41.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.456 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:41.457 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.786413+0000", 2024-03-19T11:31:41.457 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011643229999999999 2024-03-19T11:31:41.457 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.457 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.457 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:41.457 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791104+0000", 2024-03-19T11:31:41.457 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0046900830000000003 2024-03-19T11:31:41.457 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.457 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.457 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:41.457 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791248+0000", 2024-03-19T11:31:41.457 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014410200000000001 2024-03-19T11:31:41.457 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.458 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.458 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:41.458 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791340+0000", 2024-03-19T11:31:41.458 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2540000000000005e-05 2024-03-19T11:31:41.458 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:41.458 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:41.458 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:41.458 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.458 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.458 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1162 4.1 4:8401c1f0:::smithi03829856-43:head [write 594981~609197] snapc 8=[8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-03-19T11:31:41.458 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.838981+0000", 2024-03-19T11:31:41.458 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 84.621787084000005, 2024-03-19T11:31:41.458 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027704349999999999, 2024-03-19T11:31:41.458 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:41.458 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:41.459 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:41.459 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:41.459 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:41.459 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1162 2024-03-19T11:31:41.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.459 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:41.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.459 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:41.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.838981+0000", 2024-03-19T11:31:41.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:41.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.460 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:41.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.838981+0000", 2024-03-19T11:31:41.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:41.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.460 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:41.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.838983+0000", 2024-03-19T11:31:41.461 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5399999999999998e-06 2024-03-19T11:31:41.461 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.461 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:41.461 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841322+0000", 2024-03-19T11:31:41.461 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002338592 2024-03-19T11:31:41.461 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.461 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.461 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:41.461 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841371+0000", 2024-03-19T11:31:41.462 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9123999999999999e-05 2024-03-19T11:31:41.462 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.462 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.462 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:41.462 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841399+0000", 2024-03-19T11:31:41.462 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7718000000000001e-05 2024-03-19T11:31:41.462 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.462 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.462 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:41.462 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841577+0000", 2024-03-19T11:31:41.462 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000178097 2024-03-19T11:31:41.462 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.462 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.462 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:41.463 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841825+0000", 2024-03-19T11:31:41.463 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00024824299999999999 2024-03-19T11:31:41.463 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.463 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.463 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-19T11:31:41.463 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842227+0000", 2024-03-19T11:31:41.463 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00040212199999999998 2024-03-19T11:31:41.463 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.463 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.463 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:41.463 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.865182+0000", 2024-03-19T11:31:41.463 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022954810999999999 2024-03-19T11:31:41.463 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.464 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.464 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:41.464 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.866360+0000", 2024-03-19T11:31:41.464 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001178449 2024-03-19T11:31:41.464 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.464 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.464 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:41.464 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.866538+0000", 2024-03-19T11:31:41.464 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017791000000000001 2024-03-19T11:31:41.464 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.464 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.464 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:41.464 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.866685+0000", 2024-03-19T11:31:41.464 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000146744 2024-03-19T11:31:41.464 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:41.465 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:41.465 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:41.465 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.465 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.465 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1271 4.9 4:953e720b:::smithi03829856-45:2 [read 795136~1008128] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-03-19T11:31:41.465 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:21.868056+0000", 2024-03-19T11:31:41.465 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 79.592711578999996, 2024-03-19T11:31:41.465 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.047270591000000001, 2024-03-19T11:31:41.465 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:41.465 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:41.465 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:41.465 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:41.465 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:41.465 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1271 2024-03-19T11:31:41.466 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.466 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:41.466 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.466 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:41.466 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868056+0000", 2024-03-19T11:31:41.466 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:41.466 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.466 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.466 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:41.466 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868056+0000", 2024-03-19T11:31:41.466 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:41.466 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.466 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.466 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:41.467 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868059+0000", 2024-03-19T11:31:41.467 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0469999999999998e-06 2024-03-19T11:31:41.467 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.467 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.467 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:41.467 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868072+0000", 2024-03-19T11:31:41.467 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2700999999999999e-05 2024-03-19T11:31:41.467 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.467 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.467 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:41.467 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868106+0000", 2024-03-19T11:31:41.467 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4152000000000001e-05 2024-03-19T11:31:41.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.468 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:41.468 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868126+0000", 2024-03-19T11:31:41.468 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9386999999999999e-05 2024-03-19T11:31:41.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.468 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:41.468 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.911447+0000", 2024-03-19T11:31:41.468 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.043321164000000002 2024-03-19T11:31:41.468 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.468 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.468 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:41.468 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.911762+0000", 2024-03-19T11:31:41.469 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00031492400000000001 2024-03-19T11:31:41.469 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.469 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.469 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:41.469 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.915327+0000", 2024-03-19T11:31:41.469 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003565216 2024-03-19T11:31:41.469 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:41.469 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:41.469 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:41.469 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.469 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.470 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(3.1 84/83 [PushOp(3:9205291d:::smithi03829856-50 low_tier:head, version: 28'36, data_included: [393216~851968,1703936~196608], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:9205291d:::smithi03829856-50 low_tier:head@28'36, size: 2823543, copy_subset: [(0, 2823543)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1900544, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:baab691a:::smithi03829856-2 low_tier:head, version: 28'37, data_included: [393216~524288,1507328~509479], data_size: 1033767, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:baab691a:::smithi03829856-2 low_tier:head@28'37, size: 2016807, copy_subset: [(0, 2016807)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2016807, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:98ce8426:::smithi03829856-4 low_tier:head, version: 28'38, data_included: [393216~720896,1638400~327680], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:98ce8426:::smithi03829856-4 low_tier:head@28'38, size: 2050421, copy_subset: [(0, 2050421)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1966080, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:a86be07b:::smithi03829856-44 low_tier:head, version: 28'41, data_included: [720896~655360,1769472~393216], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:a86be07b:::smithi03829856-44 low_tier:head@28'41, size: 2722701, copy_subset: [(0, 2722701)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2162688, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:bb56919e:::smithi03829856-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'44, data_included: [393216~786432,1703936~262144], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:bb56919e:::smithi03829856-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'44, size: 2806736, copy_subset: [(0, 2806736)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1966080, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:b1f822cf:::smithi03829856-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'46, data_included: [720896~786432,1966080~262144], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:b1f822cf:::smithi03829856-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'46, size: 2369754, copy_subset: [(0, 2369754)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2228224, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:a5e8313a:::smithi03829856-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'47, data_included: [524288~720896,1769472~327680], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:a5e8313a:::smithi03829856-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'47, size: 2571438, copy_subset: [(0, 2571438)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:bca5ec64:::e3a26bf9230b1b95b8cf1a4cb01909236bf2bf3fd59b083b2291d278ab98264d:head, version: 28'48, data_included: [0~393216], data_size: 393216, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bca5ec64:::e3a26bf9230b1b95b8cf1a4cb01909236bf2bf3fd59b083b2291d278ab98264d:head@28'48, size: 533504, copy_subset: [(0, 533504)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 533504, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:99a49d1b:::895a63407b6109b3cca9a6f5ef827134d56679bef42daadc5892269682c428ed:head, version: 32'49, data_included: [393216~388096], data_size: 388096, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:99a49d1b:::895a63407b6109b3cca9a6f5ef827134d56679bef42daadc5892269682c428ed:head@32'49, size: 781312, copy_subset: [(0, 781312)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 781312, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:83064589:::a3639ef4c215b429a637f1406db2398074b33464b996fe20e20d8b66c6bd720b:head, version: 33'51, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:83064589:::a3639ef4c215b429a637f1406db2398074b33464b996fe20e20d8b66c6bd720b:head@33'51, size: 1606656, copy_subset: [(0, 1606656)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:41.470 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:03.092965+0000", 2024-03-19T11:31:41.470 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 38.367802535999999, 2024-03-19T11:31:41.470 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.061194046000000002, 2024-03-19T11:31:41.470 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:41.470 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:41.470 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:41.470 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.470 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:41.470 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.092965+0000", 2024-03-19T11:31:41.471 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:41.471 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.471 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.471 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:41.471 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.092965+0000", 2024-03-19T11:31:41.471 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:41.471 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.471 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.471 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:41.471 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.092968+0000", 2024-03-19T11:31:41.471 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8720000000000002e-06 2024-03-19T11:31:41.472 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.472 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.472 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:41.472 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.106869+0000", 2024-03-19T11:31:41.472 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013900554000000001 2024-03-19T11:31:41.472 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.472 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.472 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:41.472 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.107077+0000", 2024-03-19T11:31:41.472 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000208626 2024-03-19T11:31:41.472 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.473 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.473 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:41.473 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.107372+0000", 2024-03-19T11:31:41.473 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029439199999999998 2024-03-19T11:31:41.473 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.473 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.473 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:41.473 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.108053+0000", 2024-03-19T11:31:41.473 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00068115899999999997 2024-03-19T11:31:41.473 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.474 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.474 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:41.474 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.108165+0000", 2024-03-19T11:31:41.474 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011162000000000001 2024-03-19T11:31:41.474 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.474 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.474 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:41.474 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.154159+0000", 2024-03-19T11:31:41.474 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.045994822999999997 2024-03-19T11:31:41.474 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:41.474 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:41.475 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:41.475 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.475 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.475 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.5 84/83 [PushOp(4:a0a50d04:::smithi03829856-13:b, version: 56'65, data_included: [601684~619341,1669078~429235], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a0a50d04:::smithi03829856-13:b@56'65, size: 2201684, copy_subset: [(0, 2201684)], clone_subset: {}, snapset: 17=[]:{b: [b, 6], 12: [12, d], 14: [13], 17: [17, 16, 15]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2098313, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:a0a50d04:::smithi03829856-13:14, version: 74'84, data_included: [601684~754801,1669078~293775], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a0a50d04:::smithi03829856-13:14@74'84, size: 2722734, copy_subset: [(0, 2722734)], clone_subset: {}, snapset: 17=[]:{b: [b, 6], 12: [12, d], 14: [13], 17: [17, 16, 15]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1962853, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:a0a50d04:::smithi03829856-13:12, version: 81'90, data_included: [601684~619341,1669078~429235], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a0a50d04:::smithi03829856-13:12@81'90, size: 2201684, copy_subset: [(0, 2201684)], clone_subset: {}, snapset: 17=[]:{b: [b, 6], 12: [12, d], 14: [13], 17: [17, 16, 15]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2098313, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:a38b9453:::smithi03829856-37:13, version: 81'92, data_included: [605052~774376,1923587~274200], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a38b9453:::smithi03829856-37:13@81'92, size: 2605052, copy_subset: [(0, 2605052)], clone_subset: {}, snapset: 14=[]:{13: [13, 12, d, b, 6]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2197787, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:a0a50d04:::smithi03829856-13:17, version: 82'94, data_included: [601684~754801,1669078~293775], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a0a50d04:::smithi03829856-13:17@82'94, size: 3025260, copy_subset: [(0, 3025260)], clone_subset: {}, snapset: 17=[]:{b: [b, 6], 12: [12, d], 14: [13], 17: [17, 16, 15]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1962853, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:a0a50d04:::smithi03829856-13:head, version: 82'97, data_included: [601684~619341,1669078~429235], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:a0a50d04:::smithi03829856-13:head@82'97, size: 2201684, copy_subset: [(0, 2201684)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2098313, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:41.475 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:03.103661+0000", 2024-03-19T11:31:41.475 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 38.357106514999998, 2024-03-19T11:31:41.475 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027776637, 2024-03-19T11:31:41.475 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:41.475 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:41.475 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:41.476 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.476 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:41.476 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.103661+0000", 2024-03-19T11:31:41.476 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:41.476 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.476 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.476 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:41.476 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.103661+0000", 2024-03-19T11:31:41.476 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:41.476 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.476 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.477 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:41.477 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.103664+0000", 2024-03-19T11:31:41.477 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.48e-06 2024-03-19T11:31:41.477 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.477 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.477 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:41.477 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.109797+0000", 2024-03-19T11:31:41.477 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061333380000000003 2024-03-19T11:31:41.477 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.477 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.477 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:41.478 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.109866+0000", 2024-03-19T11:31:41.478 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8474999999999997e-05 2024-03-19T11:31:41.478 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.478 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.478 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:41.478 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.109957+0000", 2024-03-19T11:31:41.478 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1669000000000006e-05 2024-03-19T11:31:41.478 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.478 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.478 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:41.478 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110162+0000", 2024-03-19T11:31:41.479 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000204417 2024-03-19T11:31:41.479 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.479 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.479 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:41.479 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110186+0000", 2024-03-19T11:31:41.479 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4729e-05 2024-03-19T11:31:41.479 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.479 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.479 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:41.479 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.131438+0000", 2024-03-19T11:31:41.479 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021251529000000002 2024-03-19T11:31:41.479 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:41.480 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:41.480 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:41.480 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.480 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.480 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.5 84/83 [PushOp(4:a38b9453:::smithi03829856-37:head, version: 81'93, data_included: [1775443~781480,2724284~166520], data_size: 948000, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:a38b9453:::smithi03829856-37:head@81'93, size: 2958032, copy_subset: [(0, 2958032)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2958032, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1775443, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:41.480 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:03.110032+0000", 2024-03-19T11:31:41.480 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 38.350736286999997, 2024-03-19T11:31:41.480 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027756439000000001, 2024-03-19T11:31:41.480 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:41.480 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:41.480 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:41.480 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.481 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:41.481 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110032+0000", 2024-03-19T11:31:41.481 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:41.481 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.481 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.481 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:41.481 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110032+0000", 2024-03-19T11:31:41.481 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:41.481 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.481 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.481 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:41.481 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110033+0000", 2024-03-19T11:31:41.482 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1510000000000001e-06 2024-03-19T11:31:41.482 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.482 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.482 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:41.482 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110353+0000", 2024-03-19T11:31:41.482 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00031981199999999999 2024-03-19T11:31:41.482 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.482 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.482 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:41.482 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110375+0000", 2024-03-19T11:31:41.482 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2772000000000002e-05 2024-03-19T11:31:41.482 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.483 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.483 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:41.483 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110396+0000", 2024-03-19T11:31:41.483 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1066000000000001e-05 2024-03-19T11:31:41.483 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.483 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.483 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:41.483 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.131543+0000", 2024-03-19T11:31:41.483 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021146166000000001 2024-03-19T11:31:41.483 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.483 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.483 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:41.484 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.131563+0000", 2024-03-19T11:31:41.484 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0750999999999998e-05 2024-03-19T11:31:41.484 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.484 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.484 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:41.484 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.137788+0000", 2024-03-19T11:31:41.484 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062247209999999999 2024-03-19T11:31:41.484 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:41.484 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:41.484 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:41.484 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.485 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.485 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.18 85/84 [PushOp(4:1d58db11:::smithi03829856-18:6, version: 55'80, data_included: [759683~639358,1904434~409218], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:6@55'80, size: 3159683, copy_subset: [(0, 3159683)], clone_subset: {}, snapset: 16=[]:{6: [6], b: [b], 15: [15, 13, 12, d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2313652, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1d58db11:::smithi03829856-18:b, version: 58'84, data_included: [443718~955323,1564138~93253], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:b@58'84, size: 3445402, copy_subset: [(0, 3445402)], clone_subset: {}, snapset: 16=[]:{6: [6], b: [b], 15: [15, 13, 12, d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1657391, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:10, version: 66'95, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:10@66'95, size: 3764735, copy_subset: [(0, 3764735)], clone_subset: {}, snapset: 15=[]:{c: [b, 6], 10: [d], 12: [12], 15: [15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:c, version: 70'102, data_included: [416807~576167,1542129~472409], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:c@70'102, size: 2857157, copy_subset: [(0, 2857157)], clone_subset: {}, snapset: 15=[]:{c: [b, 6], 10: [d], 12: [12], 15: [15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2014538, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:15, version: 77'110, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:15@77'110, size: 2621892, copy_subset: [(0, 2621892)], clone_subset: {}, snapset: 15=[]:{c: [b, 6], 10: [d], 12: [12], 15: [15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:41.485 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:04.083107+0000", 2024-03-19T11:31:41.485 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 37.377660720999998, 2024-03-19T11:31:41.485 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.042109168000000002, 2024-03-19T11:31:41.485 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:41.485 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:41.485 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:41.485 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.485 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:41.485 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.083107+0000", 2024-03-19T11:31:41.485 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:41.486 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.486 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.486 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:41.486 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.083107+0000", 2024-03-19T11:31:41.486 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:41.486 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.486 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.486 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:41.486 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.083109+0000", 2024-03-19T11:31:41.486 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7060000000000001e-06 2024-03-19T11:31:41.487 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.487 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.487 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:41.487 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092477+0000", 2024-03-19T11:31:41.487 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0093678589999999992 2024-03-19T11:31:41.487 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.487 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.487 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:41.487 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092549+0000", 2024-03-19T11:31:41.487 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2313999999999999e-05 2024-03-19T11:31:41.487 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.488 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.488 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:41.488 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092640+0000", 2024-03-19T11:31:41.488 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1262999999999997e-05 2024-03-19T11:31:41.488 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.488 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.488 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:41.488 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.093076+0000", 2024-03-19T11:31:41.488 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00043617800000000001 2024-03-19T11:31:41.488 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.488 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.488 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:41.489 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.093145+0000", 2024-03-19T11:31:41.489 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8651000000000004e-05 2024-03-19T11:31:41.489 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.489 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.489 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:41.489 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.125216+0000", 2024-03-19T11:31:41.489 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032071197000000003 2024-03-19T11:31:41.489 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:41.489 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:41.489 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:41.489 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.489 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.489 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.18 e85,84 4:1d58db11:::smithi03829856-18:7,38'48,4:1d58db11:::smithi03829856-18:9,46'64,4:1d58db11:::smithi03829856-18:2,48'76,4:1d58db11:::smithi03829856-18:c,70'104)", 2024-03-19T11:31:41.490 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:04.092733+0000", 2024-03-19T11:31:41.490 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 37.368034360000003, 2024-03-19T11:31:41.490 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033436452999999998, 2024-03-19T11:31:41.490 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:41.490 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:41.490 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:41.490 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.490 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:41.490 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092733+0000", 2024-03-19T11:31:41.490 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:41.490 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.490 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.491 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:41.491 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092733+0000", 2024-03-19T11:31:41.491 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:41.491 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.491 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.491 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:41.491 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092734+0000", 2024-03-19T11:31:41.491 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0699999999999996e-07 2024-03-19T11:31:41.491 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.491 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.491 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:41.491 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092745+0000", 2024-03-19T11:31:41.492 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0393e-05 2024-03-19T11:31:41.492 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.492 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.492 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:41.492 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092762+0000", 2024-03-19T11:31:41.492 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7649e-05 2024-03-19T11:31:41.492 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.492 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.492 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:41.492 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092778+0000", 2024-03-19T11:31:41.492 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5625999999999999e-05 2024-03-19T11:31:41.492 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.492 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.492 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:41.493 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.125342+0000", 2024-03-19T11:31:41.493 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032564348 2024-03-19T11:31:41.493 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.493 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.493 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:41.493 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.125375+0000", 2024-03-19T11:31:41.493 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2441000000000003e-05 2024-03-19T11:31:41.493 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.493 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.493 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:41.493 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.126170+0000", 2024-03-19T11:31:41.493 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00079508900000000002 2024-03-19T11:31:41.493 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:41.493 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:41.493 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:41.494 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.494 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.494 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(3.3 85/84 [PushOp(3:fb6cb292:::smithi03829856-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'21, data_included: [655360~851968,1900544~196608], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:fb6cb292:::smithi03829856-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'21, size: 2302526, copy_subset: [(0, 2302526)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:d281c5a5:::smithi03829856-30 low_tier:head, version: 28'42, data_included: [458752~589824,1507328~458752], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:d281c5a5:::smithi03829856-30 low_tier:head@28'42, size: 2487403, copy_subset: [(0, 2487403)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1966080, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:ed957b32:::smithi03829856-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'44, data_included: [524288~786432,1900544~233912], data_size: 1020344, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:ed957b32:::smithi03829856-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'44, size: 2134456, copy_subset: [(0, 2134456)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2134456, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:f23a45c7:::smithi03829856-36 low_tier:head, version: 28'45, data_included: [524288~786432,1835008~262144], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:f23a45c7:::smithi03829856-36 low_tier:head@28'45, size: 2588245, copy_subset: [(0, 2588245)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:edb38de0:::smithi03829856-40 low_tier:head, version: 28'46, data_included: [655360~720896,2097152~327680], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:edb38de0:::smithi03829856-40 low_tier:head@28'46, size: 2655473, copy_subset: [(0, 2655473)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2424832, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:f92b3f40:::b1fb925e39ab40886d36b70ff1c8c4f867ef6cb2ef150fbe90de57d057dfec36:head, version: 29'47, data_included: [262144~720896,1507328~327680], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:f92b3f40:::b1fb925e39ab40886d36b70ff1c8c4f867ef6cb2ef150fbe90de57d057dfec36:head@29'47, size: 2098176, copy_subset: [(0, 2098176)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1835008, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:fc3d03e1:::626f73f06d73fccc02b43398389ff991d56d81381f5e170c36fb9439a43cbbc0:head, version: 32'48, data_included: [0~131072], data_size: 131072, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:fc3d03e1:::626f73f06d73fccc02b43398389ff991d56d81381f5e170c36fb9439a43cbbc0:head@32'48, size: 462848, copy_subset: [(0, 462848)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 462848, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:f8934067:::8892b7b523af2cde6be0d91d0468f8be86a651cb052f82daf3e2c6278805eb6e:head, version: 32'49, data_included: [0~131072], data_size: 131072, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:f8934067:::8892b7b523af2cde6be0d91d0468f8be86a651cb052f82daf3e2c6278805eb6e:head@32'49, size: 202240, copy_subset: [(0, 202240)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 202240, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:41.494 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:04.108513+0000", 2024-03-19T11:31:41.494 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 37.352254919000003, 2024-03-19T11:31:41.494 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.036668899999999997, 2024-03-19T11:31:41.494 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:41.494 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:41.494 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:41.494 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.494 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:41.494 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.108513+0000", 2024-03-19T11:31:41.494 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:41.494 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.495 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.495 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:41.495 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.108513+0000", 2024-03-19T11:31:41.495 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:41.495 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.495 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.495 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:41.495 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.108515+0000", 2024-03-19T11:31:41.495 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7579999999999999e-06 2024-03-19T11:31:41.495 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.495 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.495 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:41.495 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115352+0000", 2024-03-19T11:31:41.495 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0068369950000000002 2024-03-19T11:31:41.495 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.496 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.496 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:41.496 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115421+0000", 2024-03-19T11:31:41.496 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9282999999999994e-05 2024-03-19T11:31:41.496 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.496 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.496 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:41.496 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115505+0000", 2024-03-19T11:31:41.496 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3714999999999994e-05 2024-03-19T11:31:41.496 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.496 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.496 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:41.496 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115948+0000", 2024-03-19T11:31:41.496 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000443036 2024-03-19T11:31:41.496 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.496 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.497 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:41.497 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116012+0000", 2024-03-19T11:31:41.497 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3905999999999994e-05 2024-03-19T11:31:41.497 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.497 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.497 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:41.497 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.145182+0000", 2024-03-19T11:31:41.497 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.029170207 2024-03-19T11:31:41.497 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:41.497 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:41.497 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:41.497 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.497 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.497 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.4.13:82 3.3 e85/84 3:f6b6d21a:::smithi03829856-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 85'103, mlcod=0'0)", 2024-03-19T11:31:41.497 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:04.115589+0000", 2024-03-19T11:31:41.498 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 37.345178613000002, 2024-03-19T11:31:41.498 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035569142999999998, 2024-03-19T11:31:41.498 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:41.498 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:41.498 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:41.498 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.498 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:41.498 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115589+0000", 2024-03-19T11:31:41.498 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:41.498 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.498 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.498 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:41.499 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115589+0000", 2024-03-19T11:31:41.499 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:41.499 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.499 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.499 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:41.499 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115591+0000", 2024-03-19T11:31:41.499 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2780000000000001e-06 2024-03-19T11:31:41.499 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.499 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.499 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:41.499 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115602+0000", 2024-03-19T11:31:41.499 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1797000000000001e-05 2024-03-19T11:31:41.499 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.499 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.499 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:41.500 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115613+0000", 2024-03-19T11:31:41.500 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1025e-05 2024-03-19T11:31:41.500 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.500 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.500 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:41.500 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115619+0000", 2024-03-19T11:31:41.500 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.976e-06 2024-03-19T11:31:41.500 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.500 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.500 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:41.500 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.145267+0000", 2024-03-19T11:31:41.500 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.02964791 2024-03-19T11:31:41.501 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.501 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.501 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:41.501 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.145304+0000", 2024-03-19T11:31:41.501 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6455000000000002e-05 2024-03-19T11:31:41.501 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.501 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.501 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:41.501 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.151114+0000", 2024-03-19T11:31:41.501 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0058103139999999996 2024-03-19T11:31:41.501 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.501 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.501 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:41.501 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.151158+0000", 2024-03-19T11:31:41.501 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4388000000000003e-05 2024-03-19T11:31:41.502 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:41.502 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:41.502 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:41.502 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.502 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.502 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.4.13:84 3.3 e85/84 3:ce6951d3:::smithi03829856-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 85'104, mlcod=0'0)", 2024-03-19T11:31:41.502 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:04.115824+0000", 2024-03-19T11:31:41.502 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 37.344943610000001, 2024-03-19T11:31:41.502 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035373398, 2024-03-19T11:31:41.502 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:41.502 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:41.502 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:41.503 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.503 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:41.503 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115824+0000", 2024-03-19T11:31:41.503 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:41.503 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.503 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.503 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:41.503 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115824+0000", 2024-03-19T11:31:41.503 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:41.503 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.503 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.503 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:41.504 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115825+0000", 2024-03-19T11:31:41.504 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3399999999999999e-07 2024-03-19T11:31:41.504 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.504 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.504 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:41.504 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115861+0000", 2024-03-19T11:31:41.504 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6303000000000001e-05 2024-03-19T11:31:41.504 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.504 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.504 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:41.505 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115869+0000", 2024-03-19T11:31:41.505 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0350000000000001e-06 2024-03-19T11:31:41.505 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.505 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.505 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:41.505 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115874+0000", 2024-03-19T11:31:41.505 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1599999999999997e-06 2024-03-19T11:31:41.505 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.505 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.505 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:41.505 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.145686+0000", 2024-03-19T11:31:41.506 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.029811252 2024-03-19T11:31:41.506 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.506 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.506 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:41.506 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.145706+0000", 2024-03-19T11:31:41.506 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0264000000000001e-05 2024-03-19T11:31:41.506 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.506 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.506 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:41.506 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.151161+0000", 2024-03-19T11:31:41.507 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054555280000000003 2024-03-19T11:31:41.507 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.507 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.507 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:41.507 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.151198+0000", 2024-03-19T11:31:41.507 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6322000000000002e-05 2024-03-19T11:31:41.507 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:41.507 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:41.507 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:41.507 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.507 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.508 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.4.13:85 3.3 e85/84 3:ce6951d3:::smithi03829856-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 85'105, mlcod=0'0)", 2024-03-19T11:31:41.508 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:04.115887+0000", 2024-03-19T11:31:41.508 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 37.344880996000001, 2024-03-19T11:31:41.508 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035346673000000002, 2024-03-19T11:31:41.508 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:41.508 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:41.508 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:41.508 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.508 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:41.508 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115887+0000", 2024-03-19T11:31:41.509 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:41.509 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.509 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.509 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:41.509 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115887+0000", 2024-03-19T11:31:41.509 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:41.509 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.509 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.509 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:41.509 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115887+0000", 2024-03-19T11:31:41.509 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2300000000000002e-07 2024-03-19T11:31:41.509 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.510 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.510 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:41.510 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115900+0000", 2024-03-19T11:31:41.510 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3044000000000001e-05 2024-03-19T11:31:41.510 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.510 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.510 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:41.510 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115906+0000", 2024-03-19T11:31:41.510 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5929999999999996e-06 2024-03-19T11:31:41.510 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.511 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.511 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:41.511 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115910+0000", 2024-03-19T11:31:41.511 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9269999999999998e-06 2024-03-19T11:31:41.511 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.511 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.511 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:41.511 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.145919+0000", 2024-03-19T11:31:41.511 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030009650999999998 2024-03-19T11:31:41.511 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.511 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.511 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:41.512 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.145948+0000", 2024-03-19T11:31:41.512 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9047000000000002e-05 2024-03-19T11:31:41.512 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.512 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.512 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:41.512 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.151200+0000", 2024-03-19T11:31:41.512 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052512970000000003 2024-03-19T11:31:41.512 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.512 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.512 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:41.512 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.151234+0000", 2024-03-19T11:31:41.512 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3791000000000002e-05 2024-03-19T11:31:41.513 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:41.513 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:41.513 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:41.513 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.513 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.513 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(3.3 85/84 [PushOp(3:fcdd1002:::06a0957e01d2e04d1b09d0c04f111819905dcf8372f2c6bee5014c22d5b575ef:head, version: 34'50, data_included: [0~458752], data_size: 458752, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:fcdd1002:::06a0957e01d2e04d1b09d0c04f111819905dcf8372f2c6bee5014c22d5b575ef:head@34'50, size: 689152, copy_subset: [(0, 689152)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 689152, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:41.513 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:04.115917+0000", 2024-03-19T11:31:41.513 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 37.344850565000002, 2024-03-19T11:31:41.513 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033796169000000001, 2024-03-19T11:31:41.513 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:41.513 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:41.514 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:41.514 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.514 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:41.514 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115917+0000", 2024-03-19T11:31:41.514 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:41.514 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.514 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.514 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:41.514 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115917+0000", 2024-03-19T11:31:41.514 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:41.514 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.515 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.515 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:41.515 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115918+0000", 2024-03-19T11:31:41.515 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8000000000000002e-07 2024-03-19T11:31:41.515 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.515 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.515 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:41.515 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116163+0000", 2024-03-19T11:31:41.515 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00024516400000000003 2024-03-19T11:31:41.515 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.516 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.516 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:41.516 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116181+0000", 2024-03-19T11:31:41.516 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8372999999999998e-05 2024-03-19T11:31:41.516 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.516 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.516 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:41.516 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116196+0000", 2024-03-19T11:31:41.516 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5041e-05 2024-03-19T11:31:41.516 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.516 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.516 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:41.517 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.147578+0000", 2024-03-19T11:31:41.517 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031382284000000003 2024-03-19T11:31:41.517 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.517 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.517 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:41.517 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.147594+0000", 2024-03-19T11:31:41.517 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5153e-05 2024-03-19T11:31:41.517 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.517 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.517 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:41.517 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.149713+0000", 2024-03-19T11:31:41.517 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021198739999999999 2024-03-19T11:31:41.518 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:41.518 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:41.518 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:41.518 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.518 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.518 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.4.13:83 3.3 e85/84 3:f6b6d21a:::smithi03829856-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 85'106, mlcod=0'0)", 2024-03-19T11:31:41.518 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:04.116216+0000", 2024-03-19T11:31:41.518 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 37.344551506000002, 2024-03-19T11:31:41.518 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035049444999999999, 2024-03-19T11:31:41.518 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:41.519 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:41.519 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:41.519 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.519 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:41.519 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116216+0000", 2024-03-19T11:31:41.519 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:41.519 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.519 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.519 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:41.520 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116216+0000", 2024-03-19T11:31:41.520 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:41.520 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.520 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.520 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:41.520 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116217+0000", 2024-03-19T11:31:41.520 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.22e-07 2024-03-19T11:31:41.520 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.520 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.520 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:41.520 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116221+0000", 2024-03-19T11:31:41.520 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5349999999999998e-06 2024-03-19T11:31:41.521 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.521 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.521 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:41.521 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116237+0000", 2024-03-19T11:31:41.521 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5532e-05 2024-03-19T11:31:41.521 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.521 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.521 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:41.521 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116245+0000", 2024-03-19T11:31:41.522 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0450000000000001e-06 2024-03-19T11:31:41.522 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.522 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.522 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:41.522 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.146739+0000", 2024-03-19T11:31:41.522 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030494351999999999 2024-03-19T11:31:41.522 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.522 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.522 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:41.522 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.146776+0000", 2024-03-19T11:31:41.522 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6894999999999998e-05 2024-03-19T11:31:41.523 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.523 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.523 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:41.523 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.151237+0000", 2024-03-19T11:31:41.523 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044605160000000003 2024-03-19T11:31:41.523 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.523 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.523 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:41.523 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.151266+0000", 2024-03-19T11:31:41.523 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9247999999999998e-05 2024-03-19T11:31:41.524 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:41.524 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:41.524 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:41.524 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.524 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.524 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3194 4.d 4:b65e6658:::smithi03829856-9:head [read 0~3714347,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108)", 2024-03-19T11:31:41.524 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.202652+0000", 2024-03-19T11:31:41.524 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.258115917, 2024-03-19T11:31:41.524 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031511255000000002, 2024-03-19T11:31:41.524 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:41.524 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:41.524 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:41.524 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:41.524 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:41.525 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3194 2024-03-19T11:31:41.525 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.525 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:41.525 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.525 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:41.525 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202652+0000", 2024-03-19T11:31:41.525 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:41.525 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.525 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.525 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:41.525 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202652+0000", 2024-03-19T11:31:41.525 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:41.526 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.526 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.526 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:41.526 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202653+0000", 2024-03-19T11:31:41.526 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3960000000000001e-06 2024-03-19T11:31:41.526 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.526 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.526 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:41.526 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202661+0000", 2024-03-19T11:31:41.526 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0830000000000004e-06 2024-03-19T11:31:41.526 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.526 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.526 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:41.527 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202679+0000", 2024-03-19T11:31:41.527 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7972e-05 2024-03-19T11:31:41.527 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.527 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.527 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:41.527 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202693+0000", 2024-03-19T11:31:41.527 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4031e-05 2024-03-19T11:31:41.527 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.527 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.527 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:41.527 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.203299+0000", 2024-03-19T11:31:41.527 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00060564400000000002 2024-03-19T11:31:41.527 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.527 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.527 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:41.527 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.203528+0000", 2024-03-19T11:31:41.528 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000229049 2024-03-19T11:31:41.528 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.528 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.528 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:41.528 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.234163+0000", 2024-03-19T11:31:41.528 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030635079999999999 2024-03-19T11:31:41.528 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:41.528 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:41.528 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:41.528 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.528 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.528 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3195 4.d 4:b65e6658:::smithi03829856-9:head [read 0~3714347,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108)", 2024-03-19T11:31:41.528 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.202714+0000", 2024-03-19T11:31:41.528 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.258053886000001, 2024-03-19T11:31:41.528 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.028551382, 2024-03-19T11:31:41.529 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:41.529 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:41.529 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:41.529 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:41.529 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:41.529 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3195 2024-03-19T11:31:41.529 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.529 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:41.529 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.529 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:41.529 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202714+0000", 2024-03-19T11:31:41.529 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:41.529 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.529 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.529 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:41.530 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202714+0000", 2024-03-19T11:31:41.530 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:41.530 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.530 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.530 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:41.530 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202715+0000", 2024-03-19T11:31:41.530 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.203e-06 2024-03-19T11:31:41.530 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.530 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.530 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:41.530 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202722+0000", 2024-03-19T11:31:41.531 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1289999999999998e-06 2024-03-19T11:31:41.531 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.531 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.531 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:41.531 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202739+0000", 2024-03-19T11:31:41.531 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6929e-05 2024-03-19T11:31:41.531 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.531 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.531 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:41.531 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202754+0000", 2024-03-19T11:31:41.531 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4642e-05 2024-03-19T11:31:41.531 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.531 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.531 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:41.532 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.227218+0000", 2024-03-19T11:31:41.532 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.024464587999999999 2024-03-19T11:31:41.532 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.532 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.532 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:41.532 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.227471+0000", 2024-03-19T11:31:41.532 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000252747 2024-03-19T11:31:41.532 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.532 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.532 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:41.532 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.231265+0000", 2024-03-19T11:31:41.532 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037941440000000002 2024-03-19T11:31:41.532 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:41.532 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:41.532 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:41.533 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.533 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.533 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3196 4.d 4:b65e6658:::smithi03829856-9:head [sparse-read 0~2135368] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108)", 2024-03-19T11:31:41.533 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.202774+0000", 2024-03-19T11:31:41.533 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.257994145, 2024-03-19T11:31:41.533 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033044519000000001, 2024-03-19T11:31:41.533 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:41.533 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:41.533 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:41.533 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:41.533 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:41.533 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3196 2024-03-19T11:31:41.534 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.534 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:41.534 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.534 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:41.534 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202774+0000", 2024-03-19T11:31:41.534 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:41.534 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.534 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.534 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:41.534 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202774+0000", 2024-03-19T11:31:41.534 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:41.534 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.534 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.534 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:41.535 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202775+0000", 2024-03-19T11:31:41.535 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.423e-06 2024-03-19T11:31:41.535 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.535 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.535 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:41.535 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202782+0000", 2024-03-19T11:31:41.535 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3660000000000003e-06 2024-03-19T11:31:41.535 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.535 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.535 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:41.535 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202801+0000", 2024-03-19T11:31:41.535 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8182e-05 2024-03-19T11:31:41.535 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.535 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.535 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:41.536 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202815+0000", 2024-03-19T11:31:41.536 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4042e-05 2024-03-19T11:31:41.536 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.536 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.536 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:41.536 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.234294+0000", 2024-03-19T11:31:41.536 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031479493999999997 2024-03-19T11:31:41.536 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.536 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.536 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:41.536 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.234670+0000", 2024-03-19T11:31:41.536 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000375595 2024-03-19T11:31:41.536 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.537 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.537 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:41.537 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.235818+0000", 2024-03-19T11:31:41.537 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011484170000000001 2024-03-19T11:31:41.537 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:41.537 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:41.537 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:41.537 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.537 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.537 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(osd.0.13:71 3.3 3:e22267ea:::smithi03829856-38 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e108)", 2024-03-19T11:31:41.537 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.209268+0000", 2024-03-19T11:31:41.537 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.251499894, 2024-03-19T11:31:41.537 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.038534928000000003, 2024-03-19T11:31:41.537 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:41.538 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:41.538 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:41.538 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.538 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:41.538 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.209268+0000", 2024-03-19T11:31:41.538 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:41.538 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.538 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.538 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:41.538 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.209268+0000", 2024-03-19T11:31:41.539 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:41.539 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.539 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.539 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:41.539 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.209272+0000", 2024-03-19T11:31:41.539 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0500000000000002e-06 2024-03-19T11:31:41.539 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.539 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.539 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:41.539 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.209291+0000", 2024-03-19T11:31:41.539 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8734999999999999e-05 2024-03-19T11:31:41.540 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.540 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.540 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:41.540 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.209329+0000", 2024-03-19T11:31:41.540 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8223000000000003e-05 2024-03-19T11:31:41.540 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.540 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.540 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:41.540 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.209354+0000", 2024-03-19T11:31:41.540 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5114e-05 2024-03-19T11:31:41.540 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.541 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.541 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:41.541 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.209573+0000", 2024-03-19T11:31:41.541 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021842000000000001 2024-03-19T11:31:41.541 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.541 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.541 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:41.541 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.209814+0000", 2024-03-19T11:31:41.541 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00024179400000000001 2024-03-19T11:31:41.541 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.541 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.542 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:41.542 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.210284+0000", 2024-03-19T11:31:41.542 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00046991699999999998 2024-03-19T11:31:41.542 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.542 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.542 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:41.542 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.213128+0000", 2024-03-19T11:31:41.542 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002843609 2024-03-19T11:31:41.542 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.542 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.542 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:41.542 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.247417+0000", 2024-03-19T11:31:41.542 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.034289312000000002 2024-03-19T11:31:41.542 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.542 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.543 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:41.543 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.247692+0000", 2024-03-19T11:31:41.543 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027449000000000001 2024-03-19T11:31:41.543 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.543 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.543 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:41.543 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.247803+0000", 2024-03-19T11:31:41.543 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111264 2024-03-19T11:31:41.543 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:41.543 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:41.543 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:41.543 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.543 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.543 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3198 3.3 3:f117d76c:::3a52856062567f05d8c7df91dad2c51911138f6216e1ec709153bdff536e4abe:head [write 0~927232] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-19T11:31:41.544 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.213434+0000", 2024-03-19T11:31:41.544 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.247334202999999, 2024-03-19T11:31:41.544 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.039874882, 2024-03-19T11:31:41.544 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:41.544 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:41.544 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:41.544 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:41.544 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:41.544 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3198 2024-03-19T11:31:41.544 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.544 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:41.544 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.544 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:41.544 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.213434+0000", 2024-03-19T11:31:41.545 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:41.545 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.545 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.545 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:41.545 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.213434+0000", 2024-03-19T11:31:41.545 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:41.545 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.545 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.545 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:41.545 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.213436+0000", 2024-03-19T11:31:41.546 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4430000000000002e-06 2024-03-19T11:31:41.546 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.546 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.546 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:41.546 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.219501+0000", 2024-03-19T11:31:41.546 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0060649240000000002 2024-03-19T11:31:41.546 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.546 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.546 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:41.546 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.219531+0000", 2024-03-19T11:31:41.546 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0470000000000001e-05 2024-03-19T11:31:41.547 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.547 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.547 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:41.547 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.219552+0000", 2024-03-19T11:31:41.547 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1012000000000001e-05 2024-03-19T11:31:41.547 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.547 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.547 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:41.547 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.231849+0000", 2024-03-19T11:31:41.547 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012296105999999999 2024-03-19T11:31:41.547 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.548 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.548 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:41.548 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.232349+0000", 2024-03-19T11:31:41.548 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00050035100000000001 2024-03-19T11:31:41.548 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.548 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.548 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:41.548 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.233102+0000", 2024-03-19T11:31:41.548 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00075262300000000001 2024-03-19T11:31:41.548 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.548 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.548 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:41.548 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.249722+0000", 2024-03-19T11:31:41.549 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016620901 2024-03-19T11:31:41.549 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.549 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.549 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:41.549 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.253185+0000", 2024-03-19T11:31:41.549 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034625340000000002 2024-03-19T11:31:41.549 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.549 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.549 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:41.549 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.253252+0000", 2024-03-19T11:31:41.549 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6538000000000006e-05 2024-03-19T11:31:41.550 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.550 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.550 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:41.550 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.253309+0000", 2024-03-19T11:31:41.550 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.698e-05 2024-03-19T11:31:41.550 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:41.550 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:41.550 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:41.550 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.550 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.550 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3323 4.18 4:1d58db11:::smithi03829856-18:head [tier-promote] snapc 22=[22,21,1f,1e,1d,1c,1b,1a,18,17] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-19T11:31:41.550 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:31.226946+0000", 2024-03-19T11:31:41.550 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.233821906999999, 2024-03-19T11:31:41.551 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.037925183000000001, 2024-03-19T11:31:41.551 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:41.551 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:41.551 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:41.551 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:41.551 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:41.551 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3323 2024-03-19T11:31:41.551 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.551 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:41.551 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.551 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:41.551 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.226946+0000", 2024-03-19T11:31:41.552 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:41.552 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.552 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.552 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:41.552 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.226946+0000", 2024-03-19T11:31:41.552 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:41.552 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.552 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.552 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:41.552 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.226947+0000", 2024-03-19T11:31:41.552 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4670000000000001e-06 2024-03-19T11:31:41.552 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.553 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.553 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:41.553 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.226955+0000", 2024-03-19T11:31:41.553 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9130000000000003e-06 2024-03-19T11:31:41.553 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.553 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.553 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:41.553 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.226978+0000", 2024-03-19T11:31:41.553 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2229e-05 2024-03-19T11:31:41.553 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.553 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.553 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:41.554 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.226993+0000", 2024-03-19T11:31:41.554 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5420000000000001e-05 2024-03-19T11:31:41.554 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.554 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.554 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:41.554 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.227146+0000", 2024-03-19T11:31:41.554 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015327200000000001 2024-03-19T11:31:41.554 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.554 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.554 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:41.554 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.227406+0000", 2024-03-19T11:31:41.554 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025941799999999998 2024-03-19T11:31:41.555 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:41.555 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:41.555 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:41.555 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.264871+0000", 2024-03-19T11:31:41.555 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.037465463999999997 2024-03-19T11:31:41.555 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:41.555 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:41.555 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:41.555 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:41.555 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:41.555 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:41.558 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-19T11:31:41.559 INFO:tasks.rados.rados.0.smithi038.stdout:1596: finishing write tid 1 to smithi03829856-29 2024-03-19T11:31:41.559 INFO:tasks.rados.rados.0.smithi038.stdout:1596: finishing write tid 2 to smithi03829856-29 2024-03-19T11:31:41.559 INFO:tasks.rados.rados.0.smithi038.stdout:1602: seq_num 286 ranges {406802=626275,1686203=728431} 2024-03-19T11:31:41.559 INFO:tasks.rados.rados.0.smithi038.stdout:1602: writing smithi03829856-15 from 406802 to 1033077 tid 1 2024-03-19T11:31:41.559 INFO:tasks.rados.rados.0.smithi038.stdout:1602: writing smithi03829856-15 from 1686203 to 2414634 tid 2 2024-03-19T11:31:41.559 INFO:tasks.rados.rados.0.smithi038.stdout:1587: done (11 left) 2024-03-19T11:31:41.559 INFO:tasks.rados.rados.0.smithi038.stdout:1591: done (10 left) 2024-03-19T11:31:41.559 INFO:tasks.rados.rados.0.smithi038.stdout:1592: done (9 left) 2024-03-19T11:31:41.559 INFO:tasks.rados.rados.0.smithi038.stdout:1593: done (8 left) 2024-03-19T11:31:41.560 INFO:tasks.rados.rados.0.smithi038.stdout:1594: done (7 left) 2024-03-19T11:31:41.560 INFO:tasks.rados.rados.0.smithi038.stdout:1595: done (6 left) 2024-03-19T11:31:41.560 INFO:tasks.rados.rados.0.smithi038.stdout:1599: done (5 left) 2024-03-19T11:31:41.560 INFO:tasks.rados.rados.0.smithi038.stdout:1603: set_chunk oid 34 target oid 2024-03-19T11:31:41.560 INFO:tasks.rados.rados.0.smithi038.stdout:1603: set_chunk oid 34 offset: 0 length: 4000000 target oid smithi03829856- offset: 0 snap -1 2024-03-19T11:31:41.560 INFO:tasks.rados.rados.0.smithi038.stdout:1604: set_chunk oid 7 target oid 2024-03-19T11:31:41.560 INFO:tasks.rados.rados.0.smithi038.stdout:1604: set_chunk oid 7 offset: 2461184 length: 193024 target oid 137dae16b2f4f8726add3850379e1a5680206af866ef3958b58c6fda3b3fb089 offset: 0 snap -1 2024-03-19T11:31:41.560 INFO:tasks.rados.rados.0.smithi038.stdout:1605: read oid 41 snap -1 2024-03-19T11:31:41.560 INFO:tasks.rados.rados.0.smithi038.stdout:1605: expect (ObjNum 170 snap 17 seq_num 170) 2024-03-19T11:31:41.560 INFO:tasks.rados.rados.0.smithi038.stdout:1606: write (excl) oid 18 current snap is 37 2024-03-19T11:31:41.560 INFO:tasks.rados.rados.0.smithi038.stdout:1596: finishing write tid 3 to smithi03829856-29 2024-03-19T11:31:41.560 INFO:tasks.rados.rados.0.smithi038.stdout:1596: finishing write tid 4 to smithi03829856-29 2024-03-19T11:31:41.561 INFO:tasks.rados.rados.0.smithi038.stdout:1596: oid 29 updating version 0 to 106 2024-03-19T11:31:41.561 INFO:tasks.rados.rados.0.smithi038.stdout:1596: oid 29 updating version 106 to 108 2024-03-19T11:31:41.561 INFO:tasks.rados.rados.0.smithi038.stdout:1596: oid 29 version 108 is already newer than 107 2024-03-19T11:31:41.561 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 29 v 108 (ObjNum 284 snap 37 seq_num 284) dirty exists 2024-03-19T11:31:41.561 INFO:tasks.rados.rados.0.smithi038.stdout:1596: left oid 29 (ObjNum 284 snap 37 seq_num 284) 2024-03-19T11:31:41.561 INFO:tasks.rados.rados.0.smithi038.stdout:1600: finishing set_chunk to oid 43 2024-03-19T11:31:41.561 INFO:tasks.rados.rados.0.smithi038.stdout:1598: expect (ObjNum 255 snap 32 seq_num 255) 2024-03-19T11:31:41.561 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:41.568+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:41.659 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:41.668+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:41.673 INFO:tasks.rados.rados.0.smithi038.stdout:1597: expect (ObjNum 226 snap 25 seq_num 226) 2024-03-19T11:31:41.726 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:41.726 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:31:41.726 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:31:41.726 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:41.740 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-19T11:31:41.784 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:41.793+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:41.857 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:41.857 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:31:41.857 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:31:41.857 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:31:41.857 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:41.862 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:41.869+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:41.872 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-19T11:31:41.884 INFO:tasks.rados.rados.0.smithi038.stdout:1602: finishing write tid 1 to smithi03829856-15 2024-03-19T11:31:41.884 INFO:tasks.rados.rados.0.smithi038.stdout:1603: finishing set_chunk to oid 34 2024-03-19T11:31:41.884 INFO:tasks.rados.rados.0.smithi038.stdout:1603: got expected ERANGE (src dne) 2024-03-19T11:31:41.884 INFO:tasks.rados.rados.0.smithi038.stdout:1602: finishing write tid 2 to smithi03829856-15 2024-03-19T11:31:41.884 INFO:tasks.rados.rados.0.smithi038.stdout:1606: seq_num 287 ranges {423609=701524,1628332=472089,2648987=174622} 2024-03-19T11:31:41.890 INFO:tasks.rados.rados.0.smithi038.stdout:1606: writing smithi03829856-18 from 423609 to 1125133 tid 1 2024-03-19T11:31:41.895 INFO:tasks.rados.rados.0.smithi038.stdout:1606: writing smithi03829856-18 from 1628332 to 2100421 tid 2 2024-03-19T11:31:41.899 INFO:tasks.rados.rados.0.smithi038.stdout:1606: writing smithi03829856-18 from 2648987 to 2823609 tid 3 2024-03-19T11:31:41.900 INFO:tasks.rados.rados.0.smithi038.stdout:1596: done (8 left) 2024-03-19T11:31:41.900 INFO:tasks.rados.rados.0.smithi038.stdout:1597: done (7 left) 2024-03-19T11:31:41.900 INFO:tasks.rados.rados.0.smithi038.stdout:1598: done (6 left) 2024-03-19T11:31:41.900 INFO:tasks.rados.rados.0.smithi038.stdout:1600: done (5 left) 2024-03-19T11:31:41.900 INFO:tasks.rados.rados.0.smithi038.stdout:1603: done (4 left) 2024-03-19T11:31:41.900 INFO:tasks.rados.rados.0.smithi038.stdout:1607: snap_create 2024-03-19T11:31:41.900 INFO:tasks.rados.rados.0.smithi038.stdout:1602: finishing write tid 3 to smithi03829856-15 2024-03-19T11:31:41.900 INFO:tasks.rados.rados.0.smithi038.stdout:1602: finishing write tid 4 to smithi03829856-15 2024-03-19T11:31:41.900 INFO:tasks.rados.rados.0.smithi038.stdout:1602: oid 15 updating version 0 to 52 2024-03-19T11:31:41.900 INFO:tasks.rados.rados.0.smithi038.stdout:1602: oid 15 version 52 is already newer than 50 2024-03-19T11:31:41.900 INFO:tasks.rados.rados.0.smithi038.stdout:1602: oid 15 version 52 is already newer than 51 2024-03-19T11:31:41.900 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 15 v 52 (ObjNum 286 snap 37 seq_num 286) dirty exists 2024-03-19T11:31:41.900 INFO:tasks.rados.rados.0.smithi038.stdout:1602: left oid 15 (ObjNum 286 snap 37 seq_num 286) 2024-03-19T11:31:41.900 INFO:tasks.rados.rados.0.smithi038.stdout:1604: finishing set_chunk to oid 7 2024-03-19T11:31:41.901 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 7 set_chunk returned error code -95 offset: 2461184 length: 193024 2024-03-19T11:31:41.901 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 7 v 0 (ObjNum 210 snap 22 seq_num 210) dirty exists 2024-03-19T11:31:41.901 INFO:tasks.rados.rados.0.smithi038.stdout:1605: expect (ObjNum 170 snap 17 seq_num 170) 2024-03-19T11:31:41.962 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:41.969+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:41.989 INFO:tasks.rados.rados.0.smithi038.stdout:1606: finishing write tid 1 to smithi03829856-18 2024-03-19T11:31:41.989 INFO:tasks.rados.rados.0.smithi038.stdout:1606: finishing write tid 2 to smithi03829856-18 2024-03-19T11:31:41.989 INFO:tasks.rados.rados.0.smithi038.stdout:1606: finishing write tid 3 to smithi03829856-18 2024-03-19T11:31:41.989 INFO:tasks.rados.rados.0.smithi038.stdout:1606: finishing write tid 4 to smithi03829856-18 2024-03-19T11:31:41.989 INFO:tasks.rados.rados.0.smithi038.stdout:1606: finishing write tid 5 to smithi03829856-18 2024-03-19T11:31:41.989 INFO:tasks.rados.rados.0.smithi038.stdout:1606: oid 18 updating version 0 to 170 2024-03-19T11:31:41.989 INFO:tasks.rados.rados.0.smithi038.stdout:1606: oid 18 version 170 is already newer than 169 2024-03-19T11:31:41.990 INFO:tasks.rados.rados.0.smithi038.stdout:1606: oid 18 updating version 170 to 172 2024-03-19T11:31:41.990 INFO:tasks.rados.rados.0.smithi038.stdout:1606: oid 18 version 172 is already newer than 171 2024-03-19T11:31:41.990 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 18 v 172 (ObjNum 287 snap 37 seq_num 287) dirty exists 2024-03-19T11:31:41.990 INFO:tasks.rados.rados.0.smithi038.stdout:1606: left oid 18 (ObjNum 287 snap 37 seq_num 287) 2024-03-19T11:31:42.043 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:42.043 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:31:42.043 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:31:42.043 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:31:42.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.043 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1158 4.8 4:15708482:::smithi03829856-23:head [tier-promote] snapc 8=[8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-03-19T11:31:42.043 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.827517+0000", 2024-03-19T11:31:42.043 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 85.222477685000001, 2024-03-19T11:31:42.043 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.034096983999999997, 2024-03-19T11:31:42.043 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:42.043 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:42.044 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:42.044 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:42.044 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:42.044 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1158 2024-03-19T11:31:42.044 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.044 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:42.044 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.044 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:42.044 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827517+0000", 2024-03-19T11:31:42.044 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.044 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.044 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.044 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:42.044 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827517+0000", 2024-03-19T11:31:42.045 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.045 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.045 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.045 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:42.045 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827520+0000", 2024-03-19T11:31:42.045 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.45e-06 2024-03-19T11:31:42.045 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.045 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.045 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:42.045 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827535+0000", 2024-03-19T11:31:42.045 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.509e-05 2024-03-19T11:31:42.045 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.045 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.045 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:42.046 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827579+0000", 2024-03-19T11:31:42.046 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4211000000000001e-05 2024-03-19T11:31:42.046 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.046 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.046 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:42.046 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827603+0000", 2024-03-19T11:31:42.046 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3586000000000001e-05 2024-03-19T11:31:42.046 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.046 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.046 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.046 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827789+0000", 2024-03-19T11:31:42.046 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018580199999999999 2024-03-19T11:31:42.046 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.047 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.047 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:42.047 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.828077+0000", 2024-03-19T11:31:42.047 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028846500000000001 2024-03-19T11:31:42.047 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.047 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.047 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:42.047 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.861614+0000", 2024-03-19T11:31:42.047 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033536379999999998 2024-03-19T11:31:42.047 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.047 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:42.047 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.048 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.048 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.048 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1162 4.1 e39/21 4:8401c1f0:::smithi03829856-43:head v 39'47, mlcod=38'43)", 2024-03-19T11:31:42.048 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.842598+0000", 2024-03-19T11:31:42.048 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 85.207395915000006, 2024-03-19T11:31:42.048 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0233756, 2024-03-19T11:31:42.048 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:42.048 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:42.048 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:42.048 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.048 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:42.048 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842598+0000", 2024-03-19T11:31:42.048 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.048 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.048 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.049 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:42.049 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842598+0000", 2024-03-19T11:31:42.049 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.049 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.049 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.049 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:42.049 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842599+0000", 2024-03-19T11:31:42.049 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6000000000000003e-07 2024-03-19T11:31:42.049 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.049 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.049 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:42.049 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.843566+0000", 2024-03-19T11:31:42.049 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00096688700000000004 2024-03-19T11:31:42.049 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.050 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.050 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:42.050 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.843601+0000", 2024-03-19T11:31:42.050 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.502e-05 2024-03-19T11:31:42.050 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.050 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.050 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:42.050 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.843621+0000", 2024-03-19T11:31:42.050 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0305e-05 2024-03-19T11:31:42.050 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.050 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.050 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.050 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.843882+0000", 2024-03-19T11:31:42.050 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026027100000000002 2024-03-19T11:31:42.051 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.051 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.051 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:42.051 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.844022+0000", 2024-03-19T11:31:42.051 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014020100000000001 2024-03-19T11:31:42.051 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.051 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.051 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:42.051 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.865862+0000", 2024-03-19T11:31:42.051 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021839725000000001 2024-03-19T11:31:42.051 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.051 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.051 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:42.051 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.865974+0000", 2024-03-19T11:31:42.051 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011243099999999999 2024-03-19T11:31:42.052 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.052 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:42.052 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.052 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.052 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.052 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1209 4.8 4:1d58db11:::smithi03829856-18:head [write 645402~551297,stat] snapc 9=[9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:31:42.052 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:18.121657+0000", 2024-03-19T11:31:42.052 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 83.928336912999995, 2024-03-19T11:31:42.052 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.025074349999999999, 2024-03-19T11:31:42.052 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:42.052 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:42.052 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:42.052 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:42.052 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:42.053 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1209 2024-03-19T11:31:42.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.053 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:42.053 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.053 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:42.053 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.121657+0000", 2024-03-19T11:31:42.053 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.053 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.053 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:42.053 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.121657+0000", 2024-03-19T11:31:42.053 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.053 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.053 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:42.054 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.121661+0000", 2024-03-19T11:31:42.054 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0489999999999997e-06 2024-03-19T11:31:42.054 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.054 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:42.054 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123747+0000", 2024-03-19T11:31:42.054 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0020857720000000001 2024-03-19T11:31:42.054 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.054 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:42.054 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123779+0000", 2024-03-19T11:31:42.054 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2258000000000002e-05 2024-03-19T11:31:42.054 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.054 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:42.055 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123795+0000", 2024-03-19T11:31:42.055 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5177000000000001e-05 2024-03-19T11:31:42.055 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.055 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.055 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.055 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123949+0000", 2024-03-19T11:31:42.055 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015403699999999999 2024-03-19T11:31:42.055 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.055 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.055 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:42.055 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.124134+0000", 2024-03-19T11:31:42.055 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000185484 2024-03-19T11:31:42.055 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.055 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.056 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:42.056 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.124554+0000", 2024-03-19T11:31:42.056 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00042019699999999998 2024-03-19T11:31:42.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.056 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:42.056 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146267+0000", 2024-03-19T11:31:42.056 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021712838000000002 2024-03-19T11:31:42.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.056 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:42.056 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146424+0000", 2024-03-19T11:31:42.056 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015647299999999999 2024-03-19T11:31:42.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.057 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:42.057 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146597+0000", 2024-03-19T11:31:42.057 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017303699999999999 2024-03-19T11:31:42.057 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.057 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:42.057 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146732+0000", 2024-03-19T11:31:42.057 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013502800000000001 2024-03-19T11:31:42.057 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.058 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:42.058 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.058 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.058 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.058 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1443 4.2 4:43b32242:::smithi03829856-22:head [read 0~2352947,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:31:42.058 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.979417+0000", 2024-03-19T11:31:42.058 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 76.070577240000006, 2024-03-19T11:31:42.058 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.96719988700000004, 2024-03-19T11:31:42.058 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:42.058 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:42.059 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:42.059 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:42.059 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:42.059 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1443 2024-03-19T11:31:42.059 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.059 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:42.059 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.059 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:42.059 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979417+0000", 2024-03-19T11:31:42.059 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.059 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.059 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.059 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:42.060 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979417+0000", 2024-03-19T11:31:42.060 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.060 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.060 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.060 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:42.060 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979420+0000", 2024-03-19T11:31:42.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.366e-06 2024-03-19T11:31:42.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.061 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:42.061 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979437+0000", 2024-03-19T11:31:42.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6192e-05 2024-03-19T11:31:42.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.061 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:42.061 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979474+0000", 2024-03-19T11:31:42.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7601000000000001e-05 2024-03-19T11:31:42.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:42.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979496+0000", 2024-03-19T11:31:42.062 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1506999999999999e-05 2024-03-19T11:31:42.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979712+0000", 2024-03-19T11:31:42.062 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021641299999999999 2024-03-19T11:31:42.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:31:42.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979746+0000", 2024-03-19T11:31:42.062 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3386000000000002e-05 2024-03-19T11:31:42.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.063 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.063 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.063 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.931552+0000", 2024-03-19T11:31:42.063 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.95180659899999998 2024-03-19T11:31:42.063 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.063 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.063 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:42.063 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.934681+0000", 2024-03-19T11:31:42.063 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0031292289999999999 2024-03-19T11:31:42.063 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.063 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.064 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.064 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.942701+0000", 2024-03-19T11:31:42.065 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0080198809999999995 2024-03-19T11:31:42.065 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.065 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.065 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:42.065 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.942839+0000", 2024-03-19T11:31:42.065 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013816999999999999 2024-03-19T11:31:42.065 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.065 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.066 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:42.066 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.946617+0000", 2024-03-19T11:31:42.066 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003777543 2024-03-19T11:31:42.066 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.066 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:42.066 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.066 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.066 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1444 4.2 4:43b32242:::smithi03829856-22:head [read 0~2352947,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:31:42.066 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.979553+0000", 2024-03-19T11:31:42.066 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 76.070441536999994, 2024-03-19T11:31:42.067 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.97015945400000003, 2024-03-19T11:31:42.067 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:42.067 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:42.067 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:42.067 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:42.067 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:42.067 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1444 2024-03-19T11:31:42.067 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.067 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:42.067 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.067 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:42.068 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979553+0000", 2024-03-19T11:31:42.068 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.068 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.068 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.068 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:42.068 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979553+0000", 2024-03-19T11:31:42.068 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.068 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.068 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.068 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:42.068 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979554+0000", 2024-03-19T11:31:42.068 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.403e-06 2024-03-19T11:31:42.068 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.069 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.069 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:42.069 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979564+0000", 2024-03-19T11:31:42.069 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0032e-05 2024-03-19T11:31:42.069 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.069 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.069 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:42.069 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979579+0000", 2024-03-19T11:31:42.069 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4875000000000001e-05 2024-03-19T11:31:42.069 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.070 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.070 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:42.070 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979595+0000", 2024-03-19T11:31:42.070 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6300999999999999e-05 2024-03-19T11:31:42.070 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.070 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.070 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.070 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979874+0000", 2024-03-19T11:31:42.070 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000279065 2024-03-19T11:31:42.071 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.071 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.071 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:31:42.071 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979904+0000", 2024-03-19T11:31:42.071 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.94e-05 2024-03-19T11:31:42.071 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.071 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.071 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.071 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.934781+0000", 2024-03-19T11:31:42.071 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.95487744900000004 2024-03-19T11:31:42.072 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.072 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.072 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:42.072 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.934946+0000", 2024-03-19T11:31:42.072 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016485999999999999 2024-03-19T11:31:42.072 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.072 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.072 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.072 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.946829+0000", 2024-03-19T11:31:42.072 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011882482999999999 2024-03-19T11:31:42.072 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:42.073 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.946989+0000", 2024-03-19T11:31:42.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000160421 2024-03-19T11:31:42.073 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:42.073 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.949712+0000", 2024-03-19T11:31:42.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002723165 2024-03-19T11:31:42.073 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.074 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:42.074 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.074 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.074 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.074 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1445 4.2 4:43b32242:::smithi03829856-22:head [read 0~2352947,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:31:42.075 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.979653+0000", 2024-03-19T11:31:42.075 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 76.070341405999997, 2024-03-19T11:31:42.075 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.97249507999999996, 2024-03-19T11:31:42.075 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:42.075 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:42.075 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:42.075 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:42.075 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:42.075 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1445 2024-03-19T11:31:42.075 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.075 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:42.075 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.075 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:42.075 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979653+0000", 2024-03-19T11:31:42.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.076 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:42.076 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979653+0000", 2024-03-19T11:31:42.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.076 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:42.076 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979654+0000", 2024-03-19T11:31:42.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.372e-06 2024-03-19T11:31:42.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.076 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:42.077 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979662+0000", 2024-03-19T11:31:42.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9959999999999993e-06 2024-03-19T11:31:42.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.077 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:42.077 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979698+0000", 2024-03-19T11:31:42.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5763000000000003e-05 2024-03-19T11:31:42.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.077 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:42.077 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979714+0000", 2024-03-19T11:31:42.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6283999999999999e-05 2024-03-19T11:31:42.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.078 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.078 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979990+0000", 2024-03-19T11:31:42.078 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000275829 2024-03-19T11:31:42.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.078 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:31:42.078 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.980018+0000", 2024-03-19T11:31:42.078 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8082999999999999e-05 2024-03-19T11:31:42.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.078 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.935620+0000", 2024-03-19T11:31:42.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.95560133800000002 2024-03-19T11:31:42.079 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.079 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.079 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:42.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.935795+0000", 2024-03-19T11:31:42.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000175656 2024-03-19T11:31:42.079 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.079 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.079 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.949835+0000", 2024-03-19T11:31:42.080 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014040083 2024-03-19T11:31:42.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.080 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:42.080 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.950002+0000", 2024-03-19T11:31:42.080 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016673 2024-03-19T11:31:42.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.080 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:42.080 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.952148+0000", 2024-03-19T11:31:42.081 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021459460000000001 2024-03-19T11:31:42.081 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.081 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:42.081 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.081 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.081 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.081 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2207 4.28 4:15708482:::smithi03829856-23:head [write 541225~644694,stat] snapc 13=[13,12,11,d,b,6] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-03-19T11:31:42.081 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:51.160660+0000", 2024-03-19T11:31:42.081 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 50.889334580000003, 2024-03-19T11:31:42.081 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.025400267000000001, 2024-03-19T11:31:42.082 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:42.082 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:42.082 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:42.082 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:42.082 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:42.082 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2207 2024-03-19T11:31:42.082 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.082 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:42.082 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.082 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:42.083 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.160660+0000", 2024-03-19T11:31:42.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.083 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.083 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:42.083 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.160660+0000", 2024-03-19T11:31:42.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.083 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.084 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.084 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:42.084 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.160688+0000", 2024-03-19T11:31:42.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8058000000000001e-05 2024-03-19T11:31:42.084 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.084 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.085 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:42.085 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.168614+0000", 2024-03-19T11:31:42.085 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079262810000000003 2024-03-19T11:31:42.085 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.085 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:42.085 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.168662+0000", 2024-03-19T11:31:42.085 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8327e-05 2024-03-19T11:31:42.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.086 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.086 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:42.086 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.168679+0000", 2024-03-19T11:31:42.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6305999999999999e-05 2024-03-19T11:31:42.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.086 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.086 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.168830+0000", 2024-03-19T11:31:42.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000151714 2024-03-19T11:31:42.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:42.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.169107+0000", 2024-03-19T11:31:42.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027608299999999999 2024-03-19T11:31:42.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:42.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.169605+0000", 2024-03-19T11:31:42.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00049874999999999995 2024-03-19T11:31:42.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.088 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:42.088 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.185769+0000", 2024-03-19T11:31:42.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016163397999999999 2024-03-19T11:31:42.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.088 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:42.088 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.185809+0000", 2024-03-19T11:31:42.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0598e-05 2024-03-19T11:31:42.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.089 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:42.089 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.185898+0000", 2024-03-19T11:31:42.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8270999999999996e-05 2024-03-19T11:31:42.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.089 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:42.089 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.186060+0000", 2024-03-19T11:31:42.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016248099999999999 2024-03-19T11:31:42.089 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.089 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:42.090 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.090 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.090 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.15 75/74 [PushOp(4:af9d0d56:::smithi03829856-20:head, version: 35'55, data_included: [655360~589824,1900544~418789], data_size: 1008613, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:af9d0d56:::smithi03829856-20:head@35'55, size: 2319333, copy_subset: [(0, 2319333)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2319333, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:aa3303a5:::smithi03829856-17:b, version: 58'79, data_included: [614400~692224,1903006~356352], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:aa3303a5:::smithi03829856-17:b@58'79, size: 2268912, copy_subset: [(0, 2268912)], clone_subset: {}, snapset: 14=[]:{b: [b, 6], 10: [d], 12: [12, 11], 14: [13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2259358, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ad3ad1b3:::smithi03829856-21:b, version: 58'81, data_included: [736140~537686,2020581~315559], data_size: 853245, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:ad3ad1b3:::smithi03829856-21:b@58'81, size: 2336140, copy_subset: [(0, 2336140)], clone_subset: {}, snapset: 10=[]:{b: [b, 6], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2336140, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:afa9f208:::smithi03829856-35:12, version: 67'105, data_included: [521050~698057,1806623~350519], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:afa9f208:::smithi03829856-35:12@67'105, size: 2521050, copy_subset: [(0, 2521050)], clone_subset: {}, snapset: 12=[]:{11: [11, d, b, 6], 12: [12]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2157142, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:aa3303a5:::smithi03829856-17:12, version: 69'113, data_included: [413472~893152,1808868~155424], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:aa3303a5:::smithi03829856-17:12@69'113, size: 2319366, copy_subset: [(0, 2319366)], clone_subset: {}, snapset: 14=[]:{b: [b, 6], 10: [d], 12: [12, 11], 14: [13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1964292, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ac9cd3a3:::smithi03829856-8:10, version: 70'117, data_included: [517649~620182,1776668~428394], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:ac9cd3a3:::smithi03829856-8:10@70'117, size: 3008420, copy_subset: [(0, 3008420)], clone_subset: {}, snapset: 10=[]:{10: [d, b, 6]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2205062, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ac9cd3a3:::smithi03829856-8:head, version: 70'118, data_included: [517649~620182,1776668~428394], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:ac9cd3a3:::smithi03829856-8:head@70'118, size: 3008420, copy_subset: [(0, 3008420)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2205062, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:aa3303a5:::smithi03829856-17:10, version: 70'119, data_included: [413472~893152,1818036~155424], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:aa3303a5:::smithi03829856-17:10@70'119, size: 3613472, copy_subset: [(0, 3613472)], clone_subset: {}, snapset: 14=[]:{b: [b, 6], 10: [d], 12: [12, 11], 14: [13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1973460, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ad3ad1b3:::smithi03829856-21:10, version: 72'122, data_included: [531121~742705,1711057~305871], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:ad3ad1b3:::smithi03829856-21:10@72'122, size: 3731121, copy_subset: [(0, 3731121)], clone_subset: {}, snapset: 10=[]:{b: [b, 6], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2016928, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:42.090 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.081994+0000", 2024-03-19T11:31:42.090 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 47.967999882999997, 2024-03-19T11:31:42.090 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.040292051000000002, 2024-03-19T11:31:42.090 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:42.090 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:42.090 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:42.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.091 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:42.091 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.081994+0000", 2024-03-19T11:31:42.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.091 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:42.091 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.081994+0000", 2024-03-19T11:31:42.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.092 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:42.092 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.081996+0000", 2024-03-19T11:31:42.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5069999999999999e-06 2024-03-19T11:31:42.092 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.092 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:42.092 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088193+0000", 2024-03-19T11:31:42.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061970209999999996 2024-03-19T11:31:42.092 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.093 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:42.093 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088252+0000", 2024-03-19T11:31:42.093 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8838999999999997e-05 2024-03-19T11:31:42.093 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.093 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:42.093 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088322+0000", 2024-03-19T11:31:42.093 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0455000000000002e-05 2024-03-19T11:31:42.093 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.093 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.093 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088660+0000", 2024-03-19T11:31:42.093 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033735299999999998 2024-03-19T11:31:42.094 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.094 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:42.094 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088727+0000", 2024-03-19T11:31:42.094 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7335999999999994e-05 2024-03-19T11:31:42.094 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.094 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:42.094 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.122286+0000", 2024-03-19T11:31:42.094 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033559539999999999 2024-03-19T11:31:42.095 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.095 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:42.095 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.095 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.15 e75,74 4:ac9cd3a3:::smithi03829856-8:4,49'71)", 2024-03-19T11:31:42.095 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.088380+0000", 2024-03-19T11:31:42.095 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 47.961614021000003, 2024-03-19T11:31:42.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.044894303000000003, 2024-03-19T11:31:42.096 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:42.096 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:42.096 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:42.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.096 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:42.096 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088380+0000", 2024-03-19T11:31:42.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.096 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:42.096 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088380+0000", 2024-03-19T11:31:42.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:42.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088381+0000", 2024-03-19T11:31:42.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0500000000000004e-07 2024-03-19T11:31:42.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:42.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088385+0000", 2024-03-19T11:31:42.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2370000000000003e-06 2024-03-19T11:31:42.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:42.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088392+0000", 2024-03-19T11:31:42.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4240000000000004e-06 2024-03-19T11:31:42.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:42.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088397+0000", 2024-03-19T11:31:42.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8829999999999997e-06 2024-03-19T11:31:42.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.122356+0000", 2024-03-19T11:31:42.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033958588999999997 2024-03-19T11:31:42.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.099 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:42.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.122364+0000", 2024-03-19T11:31:42.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0800000000000006e-06 2024-03-19T11:31:42.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.099 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:42.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.133275+0000", 2024-03-19T11:31:42.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010910585 2024-03-19T11:31:42.099 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.099 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:42.100 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.100 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:2286 4.15 e75/74 4:afa9f208:::smithi03829856-35:head v 75'131, mlcod=33'54)", 2024-03-19T11:31:42.100 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.089414+0000", 2024-03-19T11:31:42.100 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 47.960580780999997, 2024-03-19T11:31:42.100 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.045421256, 2024-03-19T11:31:42.100 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:42.100 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:42.100 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:42.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.100 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:42.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.089414+0000", 2024-03-19T11:31:42.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.101 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:42.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.089414+0000", 2024-03-19T11:31:42.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.101 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:42.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.089414+0000", 2024-03-19T11:31:42.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1100000000000005e-07 2024-03-19T11:31:42.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:42.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.089423+0000", 2024-03-19T11:31:42.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3799999999999994e-06 2024-03-19T11:31:42.102 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:42.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.089436+0000", 2024-03-19T11:31:42.103 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2931e-05 2024-03-19T11:31:42.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.103 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:42.103 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.089447+0000", 2024-03-19T11:31:42.103 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1044e-05 2024-03-19T11:31:42.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.103 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.103 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.133330+0000", 2024-03-19T11:31:42.103 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.043882833000000003 2024-03-19T11:31:42.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.103 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:42.104 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.133353+0000", 2024-03-19T11:31:42.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3629999999999999e-05 2024-03-19T11:31:42.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.104 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:42.104 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.134786+0000", 2024-03-19T11:31:42.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0014327980000000001 2024-03-19T11:31:42.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.104 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:42.104 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.134835+0000", 2024-03-19T11:31:42.105 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8829000000000003e-05 2024-03-19T11:31:42.105 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.105 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:42.105 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.105 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2802 4.28 4:15708482:::smithi03829856-23:head [write 497540~612664] snapc 19=[18,17,15,b] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-03-19T11:31:42.105 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:14.060002+0000", 2024-03-19T11:31:42.105 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.989992652000002, 2024-03-19T11:31:42.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.028374612, 2024-03-19T11:31:42.106 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:42.106 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:42.106 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:42.106 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:42.106 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:42.106 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2802 2024-03-19T11:31:42.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.107 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:42.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.107 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:42.107 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.060002+0000", 2024-03-19T11:31:42.107 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.107 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:42.108 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.060002+0000", 2024-03-19T11:31:42.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.108 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.108 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:42.108 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.060005+0000", 2024-03-19T11:31:42.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5130000000000001e-06 2024-03-19T11:31:42.108 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.109 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.109 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:42.109 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.061352+0000", 2024-03-19T11:31:42.109 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0013468060000000001 2024-03-19T11:31:42.109 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.109 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.109 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:42.109 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.061403+0000", 2024-03-19T11:31:42.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1181999999999999e-05 2024-03-19T11:31:42.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.110 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.110 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:42.110 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.061428+0000", 2024-03-19T11:31:42.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4715999999999998e-05 2024-03-19T11:31:42.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.111 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.111 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.111 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.061600+0000", 2024-03-19T11:31:42.111 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017224100000000001 2024-03-19T11:31:42.111 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.111 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.111 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op must wait for map", 2024-03-19T11:31:42.111 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.061619+0000", 2024-03-19T11:31:42.111 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9213000000000001e-05 2024-03-19T11:31:42.111 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.112 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.112 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.112 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.066953+0000", 2024-03-19T11:31:42.112 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053332149999999997 2024-03-19T11:31:42.112 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.112 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.112 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:42.113 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.067236+0000", 2024-03-19T11:31:42.113 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028381400000000002 2024-03-19T11:31:42.113 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.113 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.113 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:42.113 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.067698+0000", 2024-03-19T11:31:42.113 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00046152200000000002 2024-03-19T11:31:42.113 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.113 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.114 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:42.114 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.082795+0000", 2024-03-19T11:31:42.114 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015096729 2024-03-19T11:31:42.114 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.114 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.114 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:42.114 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.088033+0000", 2024-03-19T11:31:42.114 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052383259999999997 2024-03-19T11:31:42.114 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.114 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.114 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:42.115 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.088209+0000", 2024-03-19T11:31:42.115 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000175611 2024-03-19T11:31:42.115 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.115 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.115 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:42.115 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.088376+0000", 2024-03-19T11:31:42.115 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016772400000000001 2024-03-19T11:31:42.115 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.115 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:42.115 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.115 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.116 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.116 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2803 4.28 4:15708482:::smithi03829856-23:head [write 1840862~440060] snapc 19=[18,17,15,b] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-03-19T11:31:42.116 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:14.066423+0000", 2024-03-19T11:31:42.116 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.983571509000001, 2024-03-19T11:31:42.116 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022744533000000001, 2024-03-19T11:31:42.116 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:42.116 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:42.116 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:42.116 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:42.116 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:42.116 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2803 2024-03-19T11:31:42.116 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.116 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:42.117 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.117 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:42.117 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.066423+0000", 2024-03-19T11:31:42.117 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.117 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.117 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.117 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:42.117 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.066423+0000", 2024-03-19T11:31:42.117 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.117 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.117 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.118 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:42.118 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.066425+0000", 2024-03-19T11:31:42.118 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2500000000000001e-06 2024-03-19T11:31:42.118 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.118 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.118 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:42.118 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.067351+0000", 2024-03-19T11:31:42.118 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00092597000000000003 2024-03-19T11:31:42.118 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.118 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.118 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:42.119 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.067380+0000", 2024-03-19T11:31:42.119 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9326e-05 2024-03-19T11:31:42.119 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.119 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.119 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:42.119 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.067401+0000", 2024-03-19T11:31:42.119 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0458999999999998e-05 2024-03-19T11:31:42.119 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.119 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.119 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.119 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.078507+0000", 2024-03-19T11:31:42.120 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.01110651 2024-03-19T11:31:42.120 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.120 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.120 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:42.120 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.078694+0000", 2024-03-19T11:31:42.120 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000186334 2024-03-19T11:31:42.120 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.120 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.120 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:42.120 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.078997+0000", 2024-03-19T11:31:42.121 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00030362400000000001 2024-03-19T11:31:42.121 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.121 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:42.121 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.088954+0000", 2024-03-19T11:31:42.121 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0099565639999999993 2024-03-19T11:31:42.121 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.121 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:42.121 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.088978+0000", 2024-03-19T11:31:42.121 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4355999999999999e-05 2024-03-19T11:31:42.121 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.121 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:42.122 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.089067+0000", 2024-03-19T11:31:42.122 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8664000000000004e-05 2024-03-19T11:31:42.122 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.122 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.122 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:42.122 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.089167+0000", 2024-03-19T11:31:42.122 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100476 2024-03-19T11:31:42.122 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.122 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:42.122 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.123 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.123 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.123 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3102 4.15 4:afa9f208:::smithi03829856-35:head [write 484101~692450,stat] snapc 1f=[1f,1e,1d,1c,1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-19T11:31:42.123 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.553012+0000", 2024-03-19T11:31:42.123 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.496982351, 2024-03-19T11:31:42.123 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.024256340000000001, 2024-03-19T11:31:42.123 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:42.123 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:42.123 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:42.123 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:42.123 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:42.123 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3102 2024-03-19T11:31:42.123 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.123 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:42.123 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.124 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:42.125 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.553012+0000", 2024-03-19T11:31:42.125 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.125 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.125 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.126 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:42.126 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.553012+0000", 2024-03-19T11:31:42.126 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.126 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.126 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.126 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:42.126 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.553015+0000", 2024-03-19T11:31:42.126 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5200000000000002e-06 2024-03-19T11:31:42.126 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.126 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.126 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:42.127 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.554342+0000", 2024-03-19T11:31:42.127 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0013261760000000001 2024-03-19T11:31:42.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.127 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:42.127 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.554377+0000", 2024-03-19T11:31:42.127 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5299000000000002e-05 2024-03-19T11:31:42.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.127 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:42.127 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.554401+0000", 2024-03-19T11:31:42.127 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4360000000000001e-05 2024-03-19T11:31:42.128 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.128 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.128 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.128 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.554595+0000", 2024-03-19T11:31:42.128 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019390499999999999 2024-03-19T11:31:42.128 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.128 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.128 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:42.128 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.554801+0000", 2024-03-19T11:31:42.128 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020614799999999999 2024-03-19T11:31:42.128 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.128 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.128 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:42.129 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.555182+0000", 2024-03-19T11:31:42.129 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00038094700000000002 2024-03-19T11:31:42.129 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.129 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:42.129 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.569800+0000", 2024-03-19T11:31:42.129 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0146179 2024-03-19T11:31:42.129 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.129 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:42.129 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.576922+0000", 2024-03-19T11:31:42.129 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0071215339999999997 2024-03-19T11:31:42.129 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.130 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:42.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.577114+0000", 2024-03-19T11:31:42.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019218100000000001 2024-03-19T11:31:42.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.130 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:42.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.577268+0000", 2024-03-19T11:31:42.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015437 2024-03-19T11:31:42.130 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.130 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:42.130 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.130 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3103 4.15 4:afa9f208:::smithi03829856-35:head [write 1636889~447212] snapc 1f=[1f,1e,1d,1c,1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-19T11:31:42.131 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.557970+0000", 2024-03-19T11:31:42.131 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.492023964000001, 2024-03-19T11:31:42.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033363528000000003, 2024-03-19T11:31:42.131 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:42.131 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:42.131 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:42.131 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:42.131 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:42.131 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3103 2024-03-19T11:31:42.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.131 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:42.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.131 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:42.131 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.557970+0000", 2024-03-19T11:31:42.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.132 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.132 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:42.132 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.557970+0000", 2024-03-19T11:31:42.132 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.132 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.132 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:42.132 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.557973+0000", 2024-03-19T11:31:42.132 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6060000000000001e-06 2024-03-19T11:31:42.132 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.132 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:42.132 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558806+0000", 2024-03-19T11:31:42.133 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00083305499999999997 2024-03-19T11:31:42.133 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.133 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.133 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:42.133 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558843+0000", 2024-03-19T11:31:42.133 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6519000000000001e-05 2024-03-19T11:31:42.133 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.133 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.133 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:42.133 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558863+0000", 2024-03-19T11:31:42.133 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0622999999999999e-05 2024-03-19T11:31:42.133 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.133 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.133 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.133 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.562831+0000", 2024-03-19T11:31:42.134 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039681250000000003 2024-03-19T11:31:42.134 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.134 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.134 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:42.134 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.563003+0000", 2024-03-19T11:31:42.134 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000171638 2024-03-19T11:31:42.134 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.134 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.134 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:42.134 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.563335+0000", 2024-03-19T11:31:42.134 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033165899999999999 2024-03-19T11:31:42.134 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.135 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.135 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:42.135 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.569815+0000", 2024-03-19T11:31:42.135 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00648024 2024-03-19T11:31:42.135 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.135 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.135 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:42.135 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.590982+0000", 2024-03-19T11:31:42.135 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021167235 2024-03-19T11:31:42.135 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.135 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.136 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:42.136 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.591170+0000", 2024-03-19T11:31:42.136 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018796899999999999 2024-03-19T11:31:42.136 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.136 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.136 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:42.136 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.591334+0000", 2024-03-19T11:31:42.136 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000163859 2024-03-19T11:31:42.136 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.136 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:42.136 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.136 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.136 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.136 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3104 4.15 4:afa9f208:::smithi03829856-35:head [setxattr _header (57) in=7b,truncate 2084101] snapc 1f=[1f,1e,1d,1c,1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-19T11:31:42.137 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.558905+0000", 2024-03-19T11:31:42.137 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.491088882, 2024-03-19T11:31:42.137 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032860320999999998, 2024-03-19T11:31:42.137 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:42.137 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:42.137 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:42.137 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:42.137 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:42.137 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3104 2024-03-19T11:31:42.137 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.137 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:42.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.138 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:42.138 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558905+0000", 2024-03-19T11:31:42.138 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.138 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:42.138 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558905+0000", 2024-03-19T11:31:42.139 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.139 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.139 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.139 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:42.139 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558907+0000", 2024-03-19T11:31:42.139 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.547e-06 2024-03-19T11:31:42.139 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.139 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.139 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:42.139 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558916+0000", 2024-03-19T11:31:42.139 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0589999999999992e-06 2024-03-19T11:31:42.139 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.140 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:42.140 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558931+0000", 2024-03-19T11:31:42.140 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5262999999999999e-05 2024-03-19T11:31:42.140 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.140 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:42.140 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558947+0000", 2024-03-19T11:31:42.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6140000000000001e-05 2024-03-19T11:31:42.141 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.141 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.141 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.141 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.567458+0000", 2024-03-19T11:31:42.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085106650000000006 2024-03-19T11:31:42.141 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.141 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.141 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:42.141 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.567777+0000", 2024-03-19T11:31:42.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00031917399999999998 2024-03-19T11:31:42.141 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.142 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:42.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.568090+0000", 2024-03-19T11:31:42.142 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00031286900000000002 2024-03-19T11:31:42.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.142 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:42.143 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.576738+0000", 2024-03-19T11:31:42.143 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0086476379999999992 2024-03-19T11:31:42.143 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.143 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.143 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:42.143 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.591474+0000", 2024-03-19T11:31:42.143 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014736281 2024-03-19T11:31:42.143 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.143 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.143 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:42.143 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.591576+0000", 2024-03-19T11:31:42.144 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000102303 2024-03-19T11:31:42.144 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.144 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.144 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:42.144 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.591766+0000", 2024-03-19T11:31:42.144 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018938200000000001 2024-03-19T11:31:42.144 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.144 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:42.144 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.144 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.145 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.145 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3105 4.15 4:afa9f208:::smithi03829856-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e104)", 2024-03-19T11:31:42.145 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.558970+0000", 2024-03-19T11:31:42.145 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.491024447000001, 2024-03-19T11:31:42.145 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033553684, 2024-03-19T11:31:42.145 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:42.145 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:42.145 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:42.145 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:42.145 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:42.145 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3105 2024-03-19T11:31:42.146 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.146 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:42.146 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.146 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:42.146 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558970+0000", 2024-03-19T11:31:42.146 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.146 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.146 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.146 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:42.146 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558970+0000", 2024-03-19T11:31:42.146 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.146 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.146 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.146 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:42.147 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558971+0000", 2024-03-19T11:31:42.147 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4729999999999999e-06 2024-03-19T11:31:42.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.147 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:42.147 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558978+0000", 2024-03-19T11:31:42.147 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1369999999999996e-06 2024-03-19T11:31:42.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.147 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:42.147 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.559003+0000", 2024-03-19T11:31:42.147 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4525999999999999e-05 2024-03-19T11:31:42.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.148 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.148 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:42.148 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.559019+0000", 2024-03-19T11:31:42.148 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5999999999999999e-05 2024-03-19T11:31:42.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.148 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.148 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.148 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.569615+0000", 2024-03-19T11:31:42.148 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010596257 2024-03-19T11:31:42.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.148 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.148 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-03-19T11:31:42.148 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.569763+0000", 2024-03-19T11:31:42.149 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000147319 2024-03-19T11:31:42.149 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.149 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.149 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.149 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.592019+0000", 2024-03-19T11:31:42.149 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022256597999999999 2024-03-19T11:31:42.149 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.149 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.149 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:42.149 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.592250+0000", 2024-03-19T11:31:42.149 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023081100000000001 2024-03-19T11:31:42.149 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.149 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.150 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:42.150 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.592524+0000", 2024-03-19T11:31:42.150 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000273563 2024-03-19T11:31:42.150 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.150 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:42.150 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.150 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.150 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.150 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3106 4.15 4:ad3ad1b3:::smithi03829856-21:head [write 500908~484052] snapc 1f=[1f,1e,1d,1c,1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-19T11:31:42.150 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.563910+0000", 2024-03-19T11:31:42.150 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.486084497, 2024-03-19T11:31:42.151 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.029166066000000001, 2024-03-19T11:31:42.151 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:42.151 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:42.151 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:42.151 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:42.151 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:42.151 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3106 2024-03-19T11:31:42.151 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.151 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:42.151 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.151 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:42.151 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.563910+0000", 2024-03-19T11:31:42.151 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.151 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.152 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:42.152 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.563910+0000", 2024-03-19T11:31:42.152 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.152 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.152 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:42.152 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.563913+0000", 2024-03-19T11:31:42.152 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9960000000000002e-06 2024-03-19T11:31:42.152 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.152 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:42.152 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.564853+0000", 2024-03-19T11:31:42.153 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00093991599999999995 2024-03-19T11:31:42.153 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.153 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.153 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:42.153 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.564886+0000", 2024-03-19T11:31:42.153 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3315999999999997e-05 2024-03-19T11:31:42.153 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.153 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.153 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:42.153 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.564908+0000", 2024-03-19T11:31:42.153 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2262000000000001e-05 2024-03-19T11:31:42.153 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.153 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.153 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.153 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.569861+0000", 2024-03-19T11:31:42.154 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0049529070000000003 2024-03-19T11:31:42.154 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.154 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.154 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:42.154 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.569998+0000", 2024-03-19T11:31:42.154 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000136546 2024-03-19T11:31:42.154 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.155 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.155 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:42.155 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.570279+0000", 2024-03-19T11:31:42.155 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028119899999999999 2024-03-19T11:31:42.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.155 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.155 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:42.155 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.586586+0000", 2024-03-19T11:31:42.155 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016306621 2024-03-19T11:31:42.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.155 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.155 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:42.155 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.592743+0000", 2024-03-19T11:31:42.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061571150000000003 2024-03-19T11:31:42.156 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.156 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:42.156 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.592915+0000", 2024-03-19T11:31:42.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017273200000000001 2024-03-19T11:31:42.156 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.156 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:42.157 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.593076+0000", 2024-03-19T11:31:42.157 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016045600000000001 2024-03-19T11:31:42.157 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.157 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:42.157 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.157 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.157 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.157 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3107 4.15 4:ad3ad1b3:::smithi03829856-21:head [write 1411724~689184] snapc 1f=[1f,1e,1d,1c,1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-19T11:31:42.157 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.570058+0000", 2024-03-19T11:31:42.157 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.479936778999999, 2024-03-19T11:31:42.157 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023258826, 2024-03-19T11:31:42.157 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:42.157 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:42.157 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:42.158 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:42.158 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:42.158 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3107 2024-03-19T11:31:42.158 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.158 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:42.158 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.158 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:42.158 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.570058+0000", 2024-03-19T11:31:42.158 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.159 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.159 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.159 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:42.159 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.570058+0000", 2024-03-19T11:31:42.159 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.159 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.159 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.159 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:42.159 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.570061+0000", 2024-03-19T11:31:42.159 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0469999999999998e-06 2024-03-19T11:31:42.159 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.159 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.160 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:42.160 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572012+0000", 2024-03-19T11:31:42.160 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0019516760000000001 2024-03-19T11:31:42.160 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.160 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.160 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:42.160 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572052+0000", 2024-03-19T11:31:42.160 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9353000000000001e-05 2024-03-19T11:31:42.160 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.160 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.160 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:42.160 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572076+0000", 2024-03-19T11:31:42.160 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4459000000000001e-05 2024-03-19T11:31:42.161 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.161 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.161 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.161 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.577408+0000", 2024-03-19T11:31:42.161 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053316329999999997 2024-03-19T11:31:42.161 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.161 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.161 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:42.161 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.577697+0000", 2024-03-19T11:31:42.161 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028967499999999998 2024-03-19T11:31:42.162 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.162 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.162 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:42.162 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.578148+0000", 2024-03-19T11:31:42.162 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00045085900000000001 2024-03-19T11:31:42.162 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.162 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.162 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:42.162 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.586737+0000", 2024-03-19T11:31:42.162 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085889060000000003 2024-03-19T11:31:42.162 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.162 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.162 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:42.162 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.593175+0000", 2024-03-19T11:31:42.163 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0064375090000000001 2024-03-19T11:31:42.163 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.163 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.163 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:42.163 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.593235+0000", 2024-03-19T11:31:42.163 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0553000000000001e-05 2024-03-19T11:31:42.163 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.163 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.163 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:42.163 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.593316+0000", 2024-03-19T11:31:42.163 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1155999999999993e-05 2024-03-19T11:31:42.164 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.164 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:42.164 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.164 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.164 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.164 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3108 4.15 4:ad3ad1b3:::smithi03829856-21:head [setxattr _header (57) in=7b,truncate 2100908] snapc 1f=[1f,1e,1d,1c,1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-19T11:31:42.164 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.572119+0000", 2024-03-19T11:31:42.164 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.477875595, 2024-03-19T11:31:42.164 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021614495000000001, 2024-03-19T11:31:42.164 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:42.164 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:42.164 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:42.164 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:42.164 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:42.165 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3108 2024-03-19T11:31:42.165 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.165 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:42.165 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.165 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:42.165 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572119+0000", 2024-03-19T11:31:42.165 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.165 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.165 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.165 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:42.165 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572119+0000", 2024-03-19T11:31:42.165 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.166 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.166 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.166 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:42.166 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572121+0000", 2024-03-19T11:31:42.166 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.103e-06 2024-03-19T11:31:42.166 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.166 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.166 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:42.166 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572130+0000", 2024-03-19T11:31:42.166 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5230000000000001e-06 2024-03-19T11:31:42.166 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.166 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.166 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:42.166 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572157+0000", 2024-03-19T11:31:42.167 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6775999999999999e-05 2024-03-19T11:31:42.167 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.167 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.167 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:42.167 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572176+0000", 2024-03-19T11:31:42.167 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8516e-05 2024-03-19T11:31:42.167 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.167 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.167 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.167 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.584773+0000", 2024-03-19T11:31:42.167 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012596889 2024-03-19T11:31:42.167 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.167 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.167 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:42.168 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.585059+0000", 2024-03-19T11:31:42.168 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028658999999999998 2024-03-19T11:31:42.168 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.168 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.168 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:42.168 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.585591+0000", 2024-03-19T11:31:42.168 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00053221300000000002 2024-03-19T11:31:42.168 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.168 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.168 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:42.168 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.587798+0000", 2024-03-19T11:31:42.168 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0022066469999999999 2024-03-19T11:31:42.168 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.169 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.169 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:42.169 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.593611+0000", 2024-03-19T11:31:42.169 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0058127450000000002 2024-03-19T11:31:42.169 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.169 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.169 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:42.169 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.593671+0000", 2024-03-19T11:31:42.169 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0733000000000003e-05 2024-03-19T11:31:42.169 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.169 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.169 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:42.169 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.593733+0000", 2024-03-19T11:31:42.170 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1760000000000005e-05 2024-03-19T11:31:42.170 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.170 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:42.170 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.170 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.170 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.170 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3109 4.15 4:ad3ad1b3:::smithi03829856-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e104)", 2024-03-19T11:31:42.170 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.572203+0000", 2024-03-19T11:31:42.170 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.477790834, 2024-03-19T11:31:42.170 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021786163000000001, 2024-03-19T11:31:42.170 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:42.170 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:42.170 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:42.170 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:42.171 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:42.171 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3109 2024-03-19T11:31:42.171 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.171 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:42.171 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.171 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:42.171 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572203+0000", 2024-03-19T11:31:42.171 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.171 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.171 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.171 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:42.171 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572203+0000", 2024-03-19T11:31:42.171 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.171 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.172 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.172 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:42.172 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572205+0000", 2024-03-19T11:31:42.172 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.297e-06 2024-03-19T11:31:42.172 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.172 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.172 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:42.172 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572212+0000", 2024-03-19T11:31:42.172 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9129999999999997e-06 2024-03-19T11:31:42.172 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.172 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.172 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:42.172 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572233+0000", 2024-03-19T11:31:42.172 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1713e-05 2024-03-19T11:31:42.172 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.173 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.173 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:42.173 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572251+0000", 2024-03-19T11:31:42.173 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7112000000000002e-05 2024-03-19T11:31:42.173 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.173 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.173 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.173 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.587006+0000", 2024-03-19T11:31:42.173 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014755024 2024-03-19T11:31:42.173 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.173 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.173 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-03-19T11:31:42.173 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.587242+0000", 2024-03-19T11:31:42.173 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023635099999999999 2024-03-19T11:31:42.174 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.174 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.174 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.174 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.593797+0000", 2024-03-19T11:31:42.174 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0065553520000000004 2024-03-19T11:31:42.174 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.174 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.174 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:42.174 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.593880+0000", 2024-03-19T11:31:42.174 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2529999999999998e-05 2024-03-19T11:31:42.174 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.174 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.174 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:42.175 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.593990+0000", 2024-03-19T11:31:42.175 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010987100000000001 2024-03-19T11:31:42.175 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.175 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:42.175 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.175 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.175 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:42.176 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:42.176 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-03-19T11:31:42.177 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:42.088+0000 7f6a7f9da640 -1 osd.1 120 handle_osd_map injecting map crc failure 2024-03-19T11:31:42.179 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-19T11:31:42.180 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:42.081+0000 7f1837407640 -1 osd.6 120 handle_osd_map injecting map crc failure 2024-03-19T11:31:42.180 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:42.185+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:42.181 INFO:tasks.rados.rados.0.smithi038.stdout:1602: done (4 left) 2024-03-19T11:31:42.181 INFO:tasks.rados.rados.0.smithi038.stdout:1604: done (3 left) 2024-03-19T11:31:42.181 INFO:tasks.rados.rados.0.smithi038.stdout:1605: done (2 left) 2024-03-19T11:31:42.181 INFO:tasks.rados.rados.0.smithi038.stdout:1606: done (1 left) 2024-03-19T11:31:42.181 INFO:tasks.rados.rados.0.smithi038.stdout:1607: done (0 left) 2024-03-19T11:31:42.181 INFO:tasks.rados.rados.0.smithi038.stdout:1608: read oid 19 snap -1 2024-03-19T11:31:42.182 INFO:tasks.rados.rados.0.smithi038.stdout:1608: expect (ObjNum 254 snap 32 seq_num 254) 2024-03-19T11:31:42.182 INFO:tasks.rados.rados.0.smithi038.stdout:1609: write (excl) oid 44 current snap is 38 2024-03-19T11:31:42.182 INFO:tasks.rados.rados.0.smithi038.stdout:1609: seq_num 288 ranges {440416=776773,1686814=732100} 2024-03-19T11:31:42.182 INFO:tasks.rados.rados.0.smithi038.stdout:1609: writing smithi03829856-44 from 440416 to 1217189 tid 1 2024-03-19T11:31:42.182 INFO:tasks.rados.rados.0.smithi038.stdout:1609: writing smithi03829856-44 from 1686814 to 2418914 tid 2 2024-03-19T11:31:42.182 INFO:tasks.rados.rados.0.smithi038.stdout:1610: set_chunk oid 40 target oid 2024-03-19T11:31:42.182 INFO:tasks.rados.rados.0.smithi038.stdout:1610: set_chunk oid 40 offset: 11264 length: 2491392 target oid 06ff6785225cc3b4cfdfb553e1752811c747de0144ee0c1f32d8d664fcdf3873 offset: 0 snap -1 2024-03-19T11:31:42.182 INFO:tasks.rados.rados.0.smithi038.stdout:1611: set_chunk oid 6 target oid 2024-03-19T11:31:42.194 INFO:tasks.rados.rados.0.smithi038.stdout:1611: set_chunk oid 6 offset: 28672 length: 1694208 target oid f0b88282dad6d5e863443e1b4716d5ec10811c9297c3e041d7618c576ecc8504 offset: 0 snap -1 2024-03-19T11:31:42.195 INFO:tasks.rados.rados.0.smithi038.stdout:1612: tier_promote oid 22 2024-03-19T11:31:42.195 INFO:tasks.rados.rados.0.smithi038.stdout:1608: expect (ObjNum 254 snap 32 seq_num 254) 2024-03-19T11:31:42.278 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:42.288+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:42.324 INFO:tasks.rados.rados.0.smithi038.stdout:1609: finishing write tid 1 to smithi03829856-44 2024-03-19T11:31:42.324 INFO:tasks.rados.rados.0.smithi038.stdout:1609: finishing write tid 2 to smithi03829856-44 2024-03-19T11:31:42.324 INFO:tasks.rados.rados.0.smithi038.stdout:1609: finishing write tid 3 to smithi03829856-44 2024-03-19T11:31:42.324 INFO:tasks.rados.rados.0.smithi038.stdout:1608: done (4 left) 2024-03-19T11:31:42.324 INFO:tasks.rados.rados.0.smithi038.stdout:1613: write oid 14 current snap is 38 2024-03-19T11:31:42.325 INFO:tasks.rados.rados.0.smithi038.stdout:1613: seq_num 289 ranges {457223=568375,1461649=475758,2420186=437037} 2024-03-19T11:31:42.330 INFO:tasks.rados.rados.0.smithi038.stdout:1613: writing smithi03829856-14 from 457223 to 1025598 tid 1 2024-03-19T11:31:42.335 INFO:tasks.rados.rados.0.smithi038.stdout:1613: writing smithi03829856-14 from 1461649 to 1937407 tid 2 2024-03-19T11:31:42.340 INFO:tasks.rados.rados.0.smithi038.stdout:1613: writing smithi03829856-14 from 2420186 to 2857223 tid 3 2024-03-19T11:31:42.341 INFO:tasks.rados.rados.0.smithi038.stdout:1614: read oid 2 snap -1 2024-03-19T11:31:42.341 INFO:tasks.rados.rados.0.smithi038.stdout:1614: expect (ObjNum 206 snap 22 seq_num 206) 2024-03-19T11:31:42.341 INFO:tasks.rados.rados.0.smithi038.stdout:1615: tier_flush oid 10 2024-03-19T11:31:42.341 INFO:tasks.rados.rados.0.smithi038.stdout:1615: tier_flush oid 10 snap -1 2024-03-19T11:31:42.341 INFO:tasks.rados.rados.0.smithi038.stdout:1609: finishing write tid 4 to smithi03829856-44 2024-03-19T11:31:42.341 INFO:tasks.rados.rados.0.smithi038.stdout:1609: oid 44 updating version 0 to 128 2024-03-19T11:31:42.341 INFO:tasks.rados.rados.0.smithi038.stdout:1609: oid 44 version 128 is already newer than 127 2024-03-19T11:31:42.341 INFO:tasks.rados.rados.0.smithi038.stdout:1609: oid 44 version 128 is already newer than 126 2024-03-19T11:31:42.341 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 44 v 128 (ObjNum 288 snap 38 seq_num 288) dirty exists 2024-03-19T11:31:42.341 INFO:tasks.rados.rados.0.smithi038.stdout:1609: left oid 44 (ObjNum 288 snap 38 seq_num 288) 2024-03-19T11:31:42.341 INFO:tasks.rados.rados.0.smithi038.stdout:1610: finishing set_chunk to oid 40 2024-03-19T11:31:42.341 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 40 v 86 (ObjNum 281 snap 37 seq_num 281) dirty exists 2024-03-19T11:31:42.341 INFO:tasks.rados.rados.0.smithi038.stdout:1609: done (6 left) 2024-03-19T11:31:42.342 INFO:tasks.rados.rados.0.smithi038.stdout:1610: done (5 left) 2024-03-19T11:31:42.342 INFO:tasks.rados.rados.0.smithi038.stdout:1616: write (excl) oid 9 current snap is 38 2024-03-19T11:31:42.342 INFO:tasks.rados.rados.0.smithi038.stdout:1611: finishing set_chunk to oid 6 2024-03-19T11:31:42.342 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 6 set_chunk returned error code -95 offset: 28672 length: 1694208 2024-03-19T11:31:42.342 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 6 v 0 (ObjNum 195 snap 22 seq_num 195) dirty exists 2024-03-19T11:31:42.342 INFO:tasks.rados.rados.0.smithi038.stdout:1612: got (0) Success 2024-03-19T11:31:42.342 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 22 v 23 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-03-19T11:31:42.342 INFO:tasks.rados.rados.0.smithi038.stdout:1613: finishing write tid 1 to smithi03829856-14 2024-03-19T11:31:42.342 INFO:tasks.rados.rados.0.smithi038.stdout:1613: finishing write tid 2 to smithi03829856-14 2024-03-19T11:31:42.342 INFO:tasks.rados.rados.0.smithi038.stdout:1616: seq_num 290 ranges {474030=643624,1520131=735769} 2024-03-19T11:31:42.347 INFO:tasks.rados.rados.0.smithi038.stdout:1616: writing smithi03829856-9 from 474030 to 1117654 tid 1 2024-03-19T11:31:42.354 INFO:tasks.rados.rados.0.smithi038.stdout:1616: writing smithi03829856-9 from 1520131 to 2255900 tid 2 2024-03-19T11:31:42.354 INFO:tasks.rados.rados.0.smithi038.stdout:1611: done (5 left) 2024-03-19T11:31:42.354 INFO:tasks.rados.rados.0.smithi038.stdout:1612: done (4 left) 2024-03-19T11:31:42.354 INFO:tasks.rados.rados.0.smithi038.stdout:1617: read oid 46 snap -1 2024-03-19T11:31:42.354 INFO:tasks.rados.rados.0.smithi038.stdout:1617: expect (ObjNum 143 snap 15 seq_num 143) 2024-03-19T11:31:42.354 INFO:tasks.rados.rados.0.smithi038.stdout:1618: snap_remove snap 26 2024-03-19T11:31:42.358 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:42.358 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:31:42.358 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:31:42.359 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:42.374 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-19T11:31:42.425 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:42.433+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:42.480 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:42.488+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:42.493 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:42.493 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:31:42.493 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:31:42.493 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:31:42.493 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:42.507 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-19T11:31:42.593 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:42.601+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:42.631 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-19T11:31:42.669 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:42.669 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:31:42.669 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:31:42.670 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:31:42.670 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.670 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:30 4.5 4:ac9cd3a3:::smithi03829856-8:head [write 1776668~340981] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:31:42.670 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:04.777922+0000", 2024-03-19T11:31:42.670 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 97.901259827999993, 2024-03-19T11:31:42.670 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015469894, 2024-03-19T11:31:42.670 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:42.670 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:42.670 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:42.670 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:42.670 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:42.670 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 30 2024-03-19T11:31:42.670 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.671 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:42.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.671 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:42.671 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.777922+0000", 2024-03-19T11:31:42.671 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.671 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.671 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:42.671 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.777922+0000", 2024-03-19T11:31:42.671 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.671 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.671 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:42.671 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.777924+0000", 2024-03-19T11:31:42.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3999999999999999e-06 2024-03-19T11:31:42.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:42.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778461+0000", 2024-03-19T11:31:42.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00053698299999999999 2024-03-19T11:31:42.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:42.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778487+0000", 2024-03-19T11:31:42.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5947000000000001e-05 2024-03-19T11:31:42.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:42.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778502+0000", 2024-03-19T11:31:42.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4256e-05 2024-03-19T11:31:42.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.673 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.781978+0000", 2024-03-19T11:31:42.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00347693 2024-03-19T11:31:42.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.673 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:42.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.782196+0000", 2024-03-19T11:31:42.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021726499999999999 2024-03-19T11:31:42.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.674 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:42.674 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.782572+0000", 2024-03-19T11:31:42.674 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00037635000000000001 2024-03-19T11:31:42.674 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.674 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.674 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:42.674 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.786960+0000", 2024-03-19T11:31:42.674 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043882130000000002 2024-03-19T11:31:42.674 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.674 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.674 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:42.674 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793113+0000", 2024-03-19T11:31:42.674 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061530760000000004 2024-03-19T11:31:42.675 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.675 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.675 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:42.675 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793276+0000", 2024-03-19T11:31:42.675 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000162652 2024-03-19T11:31:42.675 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.675 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.675 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:42.675 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793392+0000", 2024-03-19T11:31:42.675 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115822 2024-03-19T11:31:42.675 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.675 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:42.675 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.675 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.676 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.676 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:31 4.5 4:ac9cd3a3:::smithi03829856-8:head [setxattr _header (55) in=7b,truncate 2117649] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:31:42.676 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:04.778533+0000", 2024-03-19T11:31:42.676 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 97.900648692999994, 2024-03-19T11:31:42.676 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015182533, 2024-03-19T11:31:42.676 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:42.676 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:42.676 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:42.676 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:42.676 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:42.677 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 31 2024-03-19T11:31:42.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.677 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:42.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.677 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:42.677 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778533+0000", 2024-03-19T11:31:42.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.677 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:42.677 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778533+0000", 2024-03-19T11:31:42.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.678 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:42.678 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778534+0000", 2024-03-19T11:31:42.678 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1069999999999999e-06 2024-03-19T11:31:42.678 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.678 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.678 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:42.678 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778541+0000", 2024-03-19T11:31:42.678 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2529999999999998e-06 2024-03-19T11:31:42.678 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.678 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.678 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:42.679 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778558+0000", 2024-03-19T11:31:42.679 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6290000000000002e-05 2024-03-19T11:31:42.679 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.679 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:42.679 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778568+0000", 2024-03-19T11:31:42.679 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0360000000000001e-05 2024-03-19T11:31:42.679 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.679 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.679 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.785511+0000", 2024-03-19T11:31:42.679 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00694306 2024-03-19T11:31:42.680 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.680 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.680 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:42.680 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.785710+0000", 2024-03-19T11:31:42.680 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019837199999999999 2024-03-19T11:31:42.680 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.680 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.680 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:42.680 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.785929+0000", 2024-03-19T11:31:42.680 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021960799999999999 2024-03-19T11:31:42.680 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.680 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.680 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:42.681 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.787281+0000", 2024-03-19T11:31:42.681 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001352313 2024-03-19T11:31:42.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.681 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:42.681 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793492+0000", 2024-03-19T11:31:42.681 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062107760000000003 2024-03-19T11:31:42.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.681 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:42.681 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793578+0000", 2024-03-19T11:31:42.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5711999999999995e-05 2024-03-19T11:31:42.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.682 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.682 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:42.682 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793716+0000", 2024-03-19T11:31:42.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000137682 2024-03-19T11:31:42.682 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.682 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:42.682 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.682 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.682 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:32 4.5 4:ac9cd3a3:::smithi03829856-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:31:42.682 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:04.778582+0000", 2024-03-19T11:31:42.683 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 97.900600178999994, 2024-03-19T11:31:42.683 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015761827999999999, 2024-03-19T11:31:42.683 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:42.683 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:42.683 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:42.683 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:42.683 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:42.683 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 32 2024-03-19T11:31:42.683 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.683 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:42.683 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.684 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:42.684 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778582+0000", 2024-03-19T11:31:42.684 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.684 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.684 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.684 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:42.684 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778582+0000", 2024-03-19T11:31:42.684 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.684 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.685 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.685 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:42.685 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778583+0000", 2024-03-19T11:31:42.685 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.026e-06 2024-03-19T11:31:42.685 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.685 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.685 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:42.685 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778587+0000", 2024-03-19T11:31:42.685 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7870000000000001e-06 2024-03-19T11:31:42.686 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.686 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.686 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:42.686 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778599+0000", 2024-03-19T11:31:42.686 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1623e-05 2024-03-19T11:31:42.686 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.686 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.687 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:42.687 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778608+0000", 2024-03-19T11:31:42.687 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7399999999999993e-06 2024-03-19T11:31:42.687 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.687 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.687 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.687 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.787009+0000", 2024-03-19T11:31:42.688 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0084010490000000007 2024-03-19T11:31:42.688 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.688 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.688 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-03-19T11:31:42.688 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.787202+0000", 2024-03-19T11:31:42.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019309500000000001 2024-03-19T11:31:42.689 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.689 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.689 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.689 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793832+0000", 2024-03-19T11:31:42.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.006630052 2024-03-19T11:31:42.689 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.689 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.690 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:42.690 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.794048+0000", 2024-03-19T11:31:42.690 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000215505 2024-03-19T11:31:42.690 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.690 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.690 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:42.690 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.794343+0000", 2024-03-19T11:31:42.690 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029595100000000003 2024-03-19T11:31:42.690 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.690 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:42.690 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.690 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.690 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.690 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:891 4.5 4:af9d0d56:::smithi03829856-20:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:31:42.691 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:09.286410+0000", 2024-03-19T11:31:42.691 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 93.392771847000006, 2024-03-19T11:31:42.691 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035191405000000002, 2024-03-19T11:31:42.691 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:42.691 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:42.691 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:42.691 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:42.691 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:42.691 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 891 2024-03-19T11:31:42.691 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.691 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:42.691 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.691 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:42.691 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286410+0000", 2024-03-19T11:31:42.692 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.692 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.692 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.692 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:42.692 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286410+0000", 2024-03-19T11:31:42.692 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.692 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.692 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.692 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:42.692 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286413+0000", 2024-03-19T11:31:42.692 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2660000000000002e-06 2024-03-19T11:31:42.692 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.692 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.692 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:42.693 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286426+0000", 2024-03-19T11:31:42.693 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.258e-05 2024-03-19T11:31:42.693 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.693 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.693 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:42.693 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286460+0000", 2024-03-19T11:31:42.693 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3943999999999999e-05 2024-03-19T11:31:42.693 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.693 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.693 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:42.693 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286477+0000", 2024-03-19T11:31:42.693 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6799999999999998e-05 2024-03-19T11:31:42.693 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.693 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.693 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.694 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286605+0000", 2024-03-19T11:31:42.694 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000128892 2024-03-19T11:31:42.694 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.694 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.694 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:42.694 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286769+0000", 2024-03-19T11:31:42.694 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016359799999999999 2024-03-19T11:31:42.694 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.694 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.694 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:42.694 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.321601+0000", 2024-03-19T11:31:42.694 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.034832324999999997 2024-03-19T11:31:42.694 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.695 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:42.695 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.695 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.695 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.695 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1015 4.5 4:af9d0d56:::smithi03829856-20:head [tier-promote] snapc 6=[6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:31:42.695 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:12.761855+0000", 2024-03-19T11:31:42.695 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 89.917326321000004, 2024-03-19T11:31:42.695 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026760430000000002, 2024-03-19T11:31:42.695 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:42.695 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:42.695 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:42.695 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:42.695 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:42.695 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1015 2024-03-19T11:31:42.696 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.696 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:42.696 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.696 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:42.696 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761855+0000", 2024-03-19T11:31:42.696 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.696 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.696 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.696 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:42.696 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761855+0000", 2024-03-19T11:31:42.696 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.696 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.696 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.696 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:42.696 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761858+0000", 2024-03-19T11:31:42.697 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8140000000000001e-06 2024-03-19T11:31:42.697 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.697 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.697 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:42.697 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761872+0000", 2024-03-19T11:31:42.697 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3813e-05 2024-03-19T11:31:42.697 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.697 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.697 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:42.697 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761905+0000", 2024-03-19T11:31:42.697 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2642999999999999e-05 2024-03-19T11:31:42.697 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.697 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.697 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:42.697 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761925+0000", 2024-03-19T11:31:42.698 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0489e-05 2024-03-19T11:31:42.698 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.698 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.698 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.698 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.762070+0000", 2024-03-19T11:31:42.698 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000145153 2024-03-19T11:31:42.698 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.698 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.698 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:42.698 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.762365+0000", 2024-03-19T11:31:42.698 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029482100000000002 2024-03-19T11:31:42.698 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.698 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.698 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:42.699 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.788616+0000", 2024-03-19T11:31:42.699 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026250697 2024-03-19T11:31:42.699 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.699 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:42.699 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.699 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.699 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.699 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1121 4.5 4:aa3303a5:::smithi03829856-17:head [tier-promote] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:31:42.699 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:14.164281+0000", 2024-03-19T11:31:42.699 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 88.514901076000001, 2024-03-19T11:31:42.699 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023896435000000001, 2024-03-19T11:31:42.699 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:42.699 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:42.699 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:42.700 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:42.700 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:42.700 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1121 2024-03-19T11:31:42.700 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.700 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:42.700 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.700 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:42.700 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164281+0000", 2024-03-19T11:31:42.700 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.700 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.700 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.700 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:42.700 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164281+0000", 2024-03-19T11:31:42.700 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.701 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.701 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.701 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:42.701 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164283+0000", 2024-03-19T11:31:42.701 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8500000000000001e-06 2024-03-19T11:31:42.701 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.701 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.701 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:42.701 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164291+0000", 2024-03-19T11:31:42.701 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.14e-06 2024-03-19T11:31:42.701 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.701 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.701 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:42.701 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164313+0000", 2024-03-19T11:31:42.702 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.207e-05 2024-03-19T11:31:42.702 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.702 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.702 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:42.702 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164328+0000", 2024-03-19T11:31:42.702 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5403000000000001e-05 2024-03-19T11:31:42.702 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.702 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.702 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.702 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.167981+0000", 2024-03-19T11:31:42.702 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036529879999999998 2024-03-19T11:31:42.702 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.702 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.703 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:42.703 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.168274+0000", 2024-03-19T11:31:42.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029313100000000002 2024-03-19T11:31:42.703 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.703 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.703 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:42.703 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.188177+0000", 2024-03-19T11:31:42.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019902853000000002 2024-03-19T11:31:42.703 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.703 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:42.703 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.703 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.703 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.703 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1302 4.15 4:aa3303a5:::smithi03829856-17:head [write 413472~620140,stat] snapc b=[b,a,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-03-19T11:31:42.704 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:22.966426+0000", 2024-03-19T11:31:42.704 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 79.712755342999998, 2024-03-19T11:31:42.704 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018043513000000001, 2024-03-19T11:31:42.704 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:42.704 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:42.704 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:42.704 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:42.704 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:42.704 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1302 2024-03-19T11:31:42.704 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.704 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:42.704 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.704 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:42.704 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.966426+0000", 2024-03-19T11:31:42.704 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.705 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.705 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.705 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:42.705 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.966426+0000", 2024-03-19T11:31:42.705 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.705 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.705 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.705 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:42.705 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.966429+0000", 2024-03-19T11:31:42.705 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3829999999999999e-06 2024-03-19T11:31:42.705 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.705 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.705 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:42.705 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972117+0000", 2024-03-19T11:31:42.705 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0056885750000000004 2024-03-19T11:31:42.706 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.706 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.706 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:42.706 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972156+0000", 2024-03-19T11:31:42.706 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8878000000000003e-05 2024-03-19T11:31:42.706 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.706 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.706 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:42.706 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972184+0000", 2024-03-19T11:31:42.706 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7509999999999999e-05 2024-03-19T11:31:42.706 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.706 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.706 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.707 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972309+0000", 2024-03-19T11:31:42.707 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012525899999999999 2024-03-19T11:31:42.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.707 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:42.707 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972566+0000", 2024-03-19T11:31:42.707 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025740799999999999 2024-03-19T11:31:42.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.707 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:42.707 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972899+0000", 2024-03-19T11:31:42.707 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033238799999999998 2024-03-19T11:31:42.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.707 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:42.708 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984088+0000", 2024-03-19T11:31:42.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011189466 2024-03-19T11:31:42.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.708 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.708 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:42.708 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984138+0000", 2024-03-19T11:31:42.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9666999999999997e-05 2024-03-19T11:31:42.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.708 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.708 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:42.708 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984329+0000", 2024-03-19T11:31:42.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019086799999999999 2024-03-19T11:31:42.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.708 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.709 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:42.709 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984470+0000", 2024-03-19T11:31:42.709 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000141111 2024-03-19T11:31:42.709 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.709 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:42.709 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.709 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.709 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.709 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.1 50/49 [PushOp(4:866f3dcc:::smithi03829856-31:7, version: 38'44, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@38'44, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 45'62, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@45'62, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 46'68, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@46'68, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:7, version: 47'71, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@47'71, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 47'72, data_included: [440350~816629,1309629~231947], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@47'72, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:42.709 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.959457+0000", 2024-03-19T11:31:42.709 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 74.719724925999998, 2024-03-19T11:31:42.709 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018416666000000002, 2024-03-19T11:31:42.709 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:42.709 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:42.710 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:42.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.710 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:42.710 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.959457+0000", 2024-03-19T11:31:42.710 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.710 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.710 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:42.710 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.959457+0000", 2024-03-19T11:31:42.710 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.710 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.710 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:42.711 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.959458+0000", 2024-03-19T11:31:42.711 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4899999999999999e-06 2024-03-19T11:31:42.711 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.711 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.711 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:42.711 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.962822+0000", 2024-03-19T11:31:42.711 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0033640620000000001 2024-03-19T11:31:42.711 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.711 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.711 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:42.711 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.962893+0000", 2024-03-19T11:31:42.711 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1060000000000001e-05 2024-03-19T11:31:42.711 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.712 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.712 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:42.712 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.962977+0000", 2024-03-19T11:31:42.712 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3311000000000002e-05 2024-03-19T11:31:42.712 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.712 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.712 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.712 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963208+0000", 2024-03-19T11:31:42.713 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023121200000000001 2024-03-19T11:31:42.713 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.713 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.713 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:42.713 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963245+0000", 2024-03-19T11:31:42.713 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6844999999999997e-05 2024-03-19T11:31:42.713 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.713 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.713 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:42.713 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.977874+0000", 2024-03-19T11:31:42.713 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014628686 2024-03-19T11:31:42.713 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.713 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:42.713 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.714 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.714 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.714 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2284 4.15 4:afa9f208:::smithi03829856-35:head [read 1314816~702976] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-03-19T11:31:42.714 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:53.596775+0000", 2024-03-19T11:31:42.714 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 49.082406839999997, 2024-03-19T11:31:42.715 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.47260528299999999, 2024-03-19T11:31:42.715 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:42.715 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:42.715 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:42.715 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:42.715 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:42.715 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2284 2024-03-19T11:31:42.715 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.715 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:42.715 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.715 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:42.715 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596775+0000", 2024-03-19T11:31:42.715 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.715 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.716 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.716 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:42.716 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596775+0000", 2024-03-19T11:31:42.716 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.716 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.716 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.717 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:42.717 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596778+0000", 2024-03-19T11:31:42.717 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.174e-06 2024-03-19T11:31:42.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.717 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.717 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:42.717 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596794+0000", 2024-03-19T11:31:42.717 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5372000000000001e-05 2024-03-19T11:31:42.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.717 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.717 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:42.717 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596828+0000", 2024-03-19T11:31:42.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4107999999999999e-05 2024-03-19T11:31:42.718 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.718 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.718 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:42.718 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596844+0000", 2024-03-19T11:31:42.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6296000000000001e-05 2024-03-19T11:31:42.718 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.719 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.719 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596968+0000", 2024-03-19T11:31:42.719 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012383699999999999 2024-03-19T11:31:42.719 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.719 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:31:42.719 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596988+0000", 2024-03-19T11:31:42.719 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0645e-05 2024-03-19T11:31:42.719 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.720 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.720 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.068916+0000", 2024-03-19T11:31:42.720 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.47192744800000003 2024-03-19T11:31:42.720 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.720 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:42.720 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.069015+0000", 2024-03-19T11:31:42.720 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9500999999999995e-05 2024-03-19T11:31:42.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.721 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.721 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:42.721 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.069380+0000", 2024-03-19T11:31:42.721 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00036490200000000001 2024-03-19T11:31:42.721 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.721 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:42.721 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.722 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.722 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.722 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2286 4.15 4:afa9f208:::smithi03829856-35:head [assert-version v108,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e75)", 2024-03-19T11:31:42.722 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.084826+0000", 2024-03-19T11:31:42.722 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.594355473, 2024-03-19T11:31:42.722 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.055799556, 2024-03-19T11:31:42.722 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:42.723 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:42.723 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:42.723 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:42.723 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:42.723 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2286 2024-03-19T11:31:42.723 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.723 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:42.724 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.724 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:42.724 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084826+0000", 2024-03-19T11:31:42.724 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.724 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.724 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.724 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:42.724 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084826+0000", 2024-03-19T11:31:42.724 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.724 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.724 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.725 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:42.725 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084828+0000", 2024-03-19T11:31:42.725 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1409999999999999e-06 2024-03-19T11:31:42.725 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.725 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.725 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:42.725 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084838+0000", 2024-03-19T11:31:42.725 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3740000000000004e-06 2024-03-19T11:31:42.725 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.725 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.725 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:42.725 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084859+0000", 2024-03-19T11:31:42.725 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1644000000000001e-05 2024-03-19T11:31:42.726 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.726 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.726 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:42.726 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084871+0000", 2024-03-19T11:31:42.726 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1127e-05 2024-03-19T11:31:42.726 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.726 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.726 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.726 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084961+0000", 2024-03-19T11:31:42.726 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0371999999999995e-05 2024-03-19T11:31:42.726 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.726 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.726 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:42.726 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.085130+0000", 2024-03-19T11:31:42.727 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016884800000000001 2024-03-19T11:31:42.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.727 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5,7", 2024-03-19T11:31:42.727 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.089309+0000", 2024-03-19T11:31:42.727 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041788650000000004 2024-03-19T11:31:42.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.727 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:42.727 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.090362+0000", 2024-03-19T11:31:42.727 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001053362 2024-03-19T11:31:42.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.728 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:42.728 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.090511+0000", 2024-03-19T11:31:42.728 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014887200000000001 2024-03-19T11:31:42.728 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.728 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.728 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:42.728 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.140479+0000", 2024-03-19T11:31:42.728 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.049968006000000002 2024-03-19T11:31:42.728 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.728 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.728 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:42.728 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.140534+0000", 2024-03-19T11:31:42.728 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4917000000000003e-05 2024-03-19T11:31:42.729 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.729 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.729 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:42.729 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.140626+0000", 2024-03-19T11:31:42.729 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2027999999999994e-05 2024-03-19T11:31:42.729 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.729 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:42.729 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.729 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.729 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.729 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1d58db11:::smithi03829856-18:6, version: 55'80, data_included: [759683~639358,1904434~409218], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:6@55'80, size: 3159683, copy_subset: [(0, 3159683)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2313652, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1d58db11:::smithi03829856-18:b, version: 58'84, data_included: [443718~955323,1564138~93253], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:b@58'84, size: 3445402, copy_subset: [(0, 3445402)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1657391, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:10, version: 66'95, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:10@66'95, size: 3764735, copy_subset: [(0, 3764735)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:12, version: 66'97, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:12@66'97, size: 2504243, copy_subset: [(0, 2504243)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:c, version: 70'102, data_included: [416807~576167,1542129~472409], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:c@70'102, size: 2857157, copy_subset: [(0, 2857157)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2014538, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:42.729 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.274749+0000", 2024-03-19T11:31:42.729 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.404432542000002, 2024-03-19T11:31:42.729 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017652601, 2024-03-19T11:31:42.730 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:42.730 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:42.730 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:42.730 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.730 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:42.730 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.274749+0000", 2024-03-19T11:31:42.730 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.730 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.730 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.730 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:42.730 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.274749+0000", 2024-03-19T11:31:42.730 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.730 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.730 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.731 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:42.731 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.274751+0000", 2024-03-19T11:31:42.731 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8500000000000001e-06 2024-03-19T11:31:42.731 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.731 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.731 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:42.731 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.278426+0000", 2024-03-19T11:31:42.731 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036747559999999999 2024-03-19T11:31:42.731 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.731 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.731 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:42.731 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.278470+0000", 2024-03-19T11:31:42.731 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4119000000000002e-05 2024-03-19T11:31:42.731 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.731 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.732 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:42.732 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.278506+0000", 2024-03-19T11:31:42.732 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6066000000000003e-05 2024-03-19T11:31:42.732 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.732 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.732 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.732 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.278709+0000", 2024-03-19T11:31:42.732 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000202711 2024-03-19T11:31:42.732 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.732 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.732 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:42.732 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.278731+0000", 2024-03-19T11:31:42.732 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2657e-05 2024-03-19T11:31:42.732 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.732 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.733 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:42.733 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.292402+0000", 2024-03-19T11:31:42.733 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013670442 2024-03-19T11:31:42.733 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.733 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:42.733 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.733 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.733 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.733 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1dabc0f1:::smithi03829856-1:head, version: 70'103, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:head@70'103, size: 2621892, copy_subset: [(0, 2621892)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:42.733 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.280012+0000", 2024-03-19T11:31:42.733 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.399169897, 2024-03-19T11:31:42.733 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018569209, 2024-03-19T11:31:42.733 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:42.733 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:42.734 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:42.734 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.734 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:42.734 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280012+0000", 2024-03-19T11:31:42.734 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.734 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.734 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.734 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:42.734 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280012+0000", 2024-03-19T11:31:42.734 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.734 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.734 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.734 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:42.734 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280013+0000", 2024-03-19T11:31:42.735 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9899999999999999e-07 2024-03-19T11:31:42.735 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.735 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.735 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:42.735 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280584+0000", 2024-03-19T11:31:42.735 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057090299999999999 2024-03-19T11:31:42.735 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.735 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.735 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:42.735 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280612+0000", 2024-03-19T11:31:42.735 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8442999999999999e-05 2024-03-19T11:31:42.735 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.735 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.735 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:42.736 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280631+0000", 2024-03-19T11:31:42.736 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8896999999999999e-05 2024-03-19T11:31:42.736 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.736 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.736 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.736 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.295041+0000", 2024-03-19T11:31:42.736 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014410127999999999 2024-03-19T11:31:42.736 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.736 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.736 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:42.736 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.295050+0000", 2024-03-19T11:31:42.736 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2350000000000005e-06 2024-03-19T11:31:42.736 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.736 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.737 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:42.737 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.298581+0000", 2024-03-19T11:31:42.737 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035308039999999998 2024-03-19T11:31:42.737 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.737 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:42.737 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.737 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.737 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.737 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.18 e75,74 4:1d58db11:::smithi03829856-18:c,70'104)", 2024-03-19T11:31:42.737 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.280680+0000", 2024-03-19T11:31:42.737 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.398501422000002, 2024-03-19T11:31:42.737 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018106229000000001, 2024-03-19T11:31:42.737 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:42.737 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:42.737 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:42.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.738 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:42.738 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280680+0000", 2024-03-19T11:31:42.738 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.738 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.738 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:42.738 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280680+0000", 2024-03-19T11:31:42.738 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.738 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.738 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:42.738 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280681+0000", 2024-03-19T11:31:42.738 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2099999999999995e-07 2024-03-19T11:31:42.739 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.739 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.739 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:42.739 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280686+0000", 2024-03-19T11:31:42.739 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2240000000000003e-06 2024-03-19T11:31:42.739 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.739 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.739 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:42.741 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280698+0000", 2024-03-19T11:31:42.741 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1698999999999999e-05 2024-03-19T11:31:42.741 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.741 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.741 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:42.741 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280705+0000", 2024-03-19T11:31:42.741 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4350000000000002e-06 2024-03-19T11:31:42.741 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.741 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.742 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.742 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.298643+0000", 2024-03-19T11:31:42.742 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017938352000000001 2024-03-19T11:31:42.742 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.742 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.742 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:42.742 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.298656+0000", 2024-03-19T11:31:42.742 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3084999999999999e-05 2024-03-19T11:31:42.742 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.742 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.742 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:42.742 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.298787+0000", 2024-03-19T11:31:42.742 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013061299999999999 2024-03-19T11:31:42.742 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.743 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:42.743 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.743 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.743 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.743 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.5.0:143 4.18 e75/74 4:1d58db11:::smithi03829856-18:head v 75'106, mlcod=0'0)", 2024-03-19T11:31:42.743 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.282327+0000", 2024-03-19T11:31:42.743 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.396855051000003, 2024-03-19T11:31:42.743 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016753244, 2024-03-19T11:31:42.743 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:42.743 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:42.743 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:42.743 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.743 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:42.743 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.282327+0000", 2024-03-19T11:31:42.743 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.744 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.744 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.744 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:42.744 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.282327+0000", 2024-03-19T11:31:42.744 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.744 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.744 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.744 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:42.744 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.282328+0000", 2024-03-19T11:31:42.744 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.488e-06 2024-03-19T11:31:42.744 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.744 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.744 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:42.744 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.283127+0000", 2024-03-19T11:31:42.744 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00079853699999999999 2024-03-19T11:31:42.745 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.745 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.745 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:42.745 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.283152+0000", 2024-03-19T11:31:42.745 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5215000000000002e-05 2024-03-19T11:31:42.745 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.745 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.745 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:42.745 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.283167+0000", 2024-03-19T11:31:42.745 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4635e-05 2024-03-19T11:31:42.745 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.745 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.745 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.745 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.292633+0000", 2024-03-19T11:31:42.745 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0094664339999999993 2024-03-19T11:31:42.746 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.746 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.746 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:42.746 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.292655+0000", 2024-03-19T11:31:42.746 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1934999999999999e-05 2024-03-19T11:31:42.746 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.746 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.746 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:42.746 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.298981+0000", 2024-03-19T11:31:42.746 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0063255129999999996 2024-03-19T11:31:42.746 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.746 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.746 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:42.746 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.299080+0000", 2024-03-19T11:31:42.747 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9487000000000005e-05 2024-03-19T11:31:42.747 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.747 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:42.747 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.747 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.747 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.747 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2441 4.5 4:a38b9453:::smithi03829856-37:head [delete] snapc 17=[17,16,15,13,12,b,6] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e84)", 2024-03-19T11:31:42.747 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:03.060663+0000", 2024-03-19T11:31:42.747 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 39.618519206999999, 2024-03-19T11:31:42.747 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10917726, 2024-03-19T11:31:42.747 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:42.747 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:42.747 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:42.747 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:42.747 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:42.748 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2441 2024-03-19T11:31:42.748 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.748 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:42.748 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.748 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:42.748 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060663+0000", 2024-03-19T11:31:42.748 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.748 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.748 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.748 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:42.748 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060663+0000", 2024-03-19T11:31:42.748 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.748 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.748 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.748 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:42.749 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060666+0000", 2024-03-19T11:31:42.749 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.05e-06 2024-03-19T11:31:42.749 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.749 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.749 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:42.749 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060680+0000", 2024-03-19T11:31:42.749 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4276e-05 2024-03-19T11:31:42.749 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.749 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.749 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:42.749 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060714+0000", 2024-03-19T11:31:42.749 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4325000000000001e-05 2024-03-19T11:31:42.749 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.749 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.750 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:42.750 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060737+0000", 2024-03-19T11:31:42.750 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2959000000000002e-05 2024-03-19T11:31:42.750 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.750 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.750 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.750 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060895+0000", 2024-03-19T11:31:42.750 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000157787 2024-03-19T11:31:42.750 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.750 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.750 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:31:42.750 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060923+0000", 2024-03-19T11:31:42.751 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8019000000000001e-05 2024-03-19T11:31:42.751 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.751 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.751 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.751 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.066222+0000", 2024-03-19T11:31:42.751 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052989270000000002 2024-03-19T11:31:42.751 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.751 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.752 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:42.752 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.070153+0000", 2024-03-19T11:31:42.752 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039315119999999999 2024-03-19T11:31:42.752 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.752 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.752 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.752 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.147930+0000", 2024-03-19T11:31:42.752 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.077776459000000006 2024-03-19T11:31:42.752 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.752 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.752 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:42.752 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.148079+0000", 2024-03-19T11:31:42.752 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014888 2024-03-19T11:31:42.752 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.753 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.753 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:42.753 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.148370+0000", 2024-03-19T11:31:42.753 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029130999999999997 2024-03-19T11:31:42.753 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.753 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.754 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:42.754 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.152531+0000", 2024-03-19T11:31:42.754 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041605399999999999 2024-03-19T11:31:42.754 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.754 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.754 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:42.754 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.169762+0000", 2024-03-19T11:31:42.754 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017231483999999998 2024-03-19T11:31:42.754 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.754 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.754 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:42.754 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.169807+0000", 2024-03-19T11:31:42.754 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5161999999999999e-05 2024-03-19T11:31:42.755 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.755 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.755 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:42.755 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.169840+0000", 2024-03-19T11:31:42.755 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2570000000000002e-05 2024-03-19T11:31:42.755 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.755 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:42.755 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.756 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.756 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.756 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPushReply(4.5 84/83 [PushReplyOp(4:a38b9453:::smithi03829856-37:head)])", 2024-03-19T11:31:42.756 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:03.084896+0000", 2024-03-19T11:31:42.756 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 39.594286072999999, 2024-03-19T11:31:42.756 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.02071104, 2024-03-19T11:31:42.756 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:42.756 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-19T11:31:42.756 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:42.756 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.756 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:42.757 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.084896+0000", 2024-03-19T11:31:42.757 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.757 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.757 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.757 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:42.757 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.084896+0000", 2024-03-19T11:31:42.757 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.757 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.758 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.758 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:42.758 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.084897+0000", 2024-03-19T11:31:42.758 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.686e-06 2024-03-19T11:31:42.758 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.758 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.758 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:42.758 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.084910+0000", 2024-03-19T11:31:42.758 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2948e-05 2024-03-19T11:31:42.759 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.759 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:42.759 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.084936+0000", 2024-03-19T11:31:42.759 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5982e-05 2024-03-19T11:31:42.759 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.759 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:42.759 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.084955+0000", 2024-03-19T11:31:42.759 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8416000000000001e-05 2024-03-19T11:31:42.760 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.760 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.760 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.760 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.102822+0000", 2024-03-19T11:31:42.760 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017866756000000001 2024-03-19T11:31:42.760 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.760 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.760 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:42.760 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.105607+0000", 2024-03-19T11:31:42.761 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027852520000000002 2024-03-19T11:31:42.761 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.761 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:42.761 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.761 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.761 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.761 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.4.0:540 4.18 e111/86 4:1d58db11:::smithi03829856-18:head v 111'153, mlcod=108'150)", 2024-03-19T11:31:42.762 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:31.238185+0000", 2024-03-19T11:31:42.762 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.44099671, 2024-03-19T11:31:42.762 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021779390999999999, 2024-03-19T11:31:42.762 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:42.762 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:42.762 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:42.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.763 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:42.763 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.238185+0000", 2024-03-19T11:31:42.763 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.763 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.763 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:42.763 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.238185+0000", 2024-03-19T11:31:42.763 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.763 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.763 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:42.764 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.238188+0000", 2024-03-19T11:31:42.764 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4739999999999999e-06 2024-03-19T11:31:42.764 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.764 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:42.764 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.240837+0000", 2024-03-19T11:31:42.764 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026495189999999999 2024-03-19T11:31:42.764 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.764 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:42.764 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.240867+0000", 2024-03-19T11:31:42.764 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0363000000000001e-05 2024-03-19T11:31:42.764 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.765 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:42.765 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.240885+0000", 2024-03-19T11:31:42.765 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7153e-05 2024-03-19T11:31:42.765 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.765 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.765 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.765 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.241029+0000", 2024-03-19T11:31:42.765 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014455499999999999 2024-03-19T11:31:42.765 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.765 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.765 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:42.765 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.241069+0000", 2024-03-19T11:31:42.765 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9409000000000002e-05 2024-03-19T11:31:42.765 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.765 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.766 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:42.766 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.259833+0000", 2024-03-19T11:31:42.766 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018764306000000001 2024-03-19T11:31:42.766 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.766 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.766 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:42.766 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.259964+0000", 2024-03-19T11:31:42.766 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000131612 2024-03-19T11:31:42.766 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.766 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:42.766 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.766 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.766 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.766 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3282 4.5 4:a0a50d04:::smithi03829856-13:head [read 0~2201684,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e112)", 2024-03-19T11:31:42.767 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:32.240600+0000", 2024-03-19T11:31:42.767 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.438581506, 2024-03-19T11:31:42.767 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.84623566699999997, 2024-03-19T11:31:42.767 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:42.767 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:42.767 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:42.767 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:42.767 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:42.767 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3282 2024-03-19T11:31:42.767 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.767 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:42.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.767 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:42.767 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240600+0000", 2024-03-19T11:31:42.767 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.768 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.768 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.768 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:42.768 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240600+0000", 2024-03-19T11:31:42.768 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.768 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.768 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.768 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:42.768 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240602+0000", 2024-03-19T11:31:42.768 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0700000000000001e-06 2024-03-19T11:31:42.768 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.768 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.768 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:42.768 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240611+0000", 2024-03-19T11:31:42.768 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.969e-06 2024-03-19T11:31:42.769 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.769 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.769 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:42.769 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240630+0000", 2024-03-19T11:31:42.769 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8253e-05 2024-03-19T11:31:42.769 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.769 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.769 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:42.769 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240642+0000", 2024-03-19T11:31:42.769 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2666e-05 2024-03-19T11:31:42.769 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.769 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.769 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.769 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.241488+0000", 2024-03-19T11:31:42.770 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000845252 2024-03-19T11:31:42.770 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.770 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.770 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:31:42.770 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.241503+0000", 2024-03-19T11:31:42.770 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5797000000000001e-05 2024-03-19T11:31:42.770 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.770 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.770 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.770 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.083186+0000", 2024-03-19T11:31:42.770 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.84168299599999996 2024-03-19T11:31:42.770 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.770 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.770 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:42.771 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.083618+0000", 2024-03-19T11:31:42.771 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000432046 2024-03-19T11:31:42.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.771 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:42.771 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.086836+0000", 2024-03-19T11:31:42.771 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032176180000000002 2024-03-19T11:31:42.771 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.771 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:42.771 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.771 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3283 4.5 4:a0a50d04:::smithi03829856-13:head [sparse-read 0~1151947] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e112)", 2024-03-19T11:31:42.771 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:32.240669+0000", 2024-03-19T11:31:42.772 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.438512506, 2024-03-19T11:31:42.772 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.84791886900000002, 2024-03-19T11:31:42.772 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:42.772 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:42.772 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:42.772 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:42.772 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:42.772 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3283 2024-03-19T11:31:42.772 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.772 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:42.772 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.772 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:42.772 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240669+0000", 2024-03-19T11:31:42.772 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.773 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.773 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.773 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:42.773 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240669+0000", 2024-03-19T11:31:42.773 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.773 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.773 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.773 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:42.773 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240670+0000", 2024-03-19T11:31:42.773 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8199999999999998e-07 2024-03-19T11:31:42.773 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.773 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.773 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:42.773 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240675+0000", 2024-03-19T11:31:42.773 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0849999999999996e-06 2024-03-19T11:31:42.774 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.774 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.774 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:42.774 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240688+0000", 2024-03-19T11:31:42.774 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2901000000000001e-05 2024-03-19T11:31:42.774 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.774 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.774 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:42.774 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240696+0000", 2024-03-19T11:31:42.774 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2910000000000007e-06 2024-03-19T11:31:42.774 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.774 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.774 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.774 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.241573+0000", 2024-03-19T11:31:42.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00087639100000000002 2024-03-19T11:31:42.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:31:42.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.241587+0000", 2024-03-19T11:31:42.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4467999999999999e-05 2024-03-19T11:31:42.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.087901+0000", 2024-03-19T11:31:42.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.84631403500000002 2024-03-19T11:31:42.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:42.776 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.088089+0000", 2024-03-19T11:31:42.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000187171 2024-03-19T11:31:42.776 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.776 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.776 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:42.776 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.088588+0000", 2024-03-19T11:31:42.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00049964499999999995 2024-03-19T11:31:42.776 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.776 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:42.776 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.776 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.776 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.776 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3284 4.5 4:a0a50d04:::smithi03829856-13:head [sparse-read 0~1151947] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e112)", 2024-03-19T11:31:42.776 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:32.240710+0000", 2024-03-19T11:31:42.776 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 10.43847221, 2024-03-19T11:31:42.777 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.84860797899999996, 2024-03-19T11:31:42.777 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:42.777 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:42.777 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:42.777 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:42.777 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:42.777 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3284 2024-03-19T11:31:42.777 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.777 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:42.777 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.777 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:42.777 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240710+0000", 2024-03-19T11:31:42.777 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.777 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.777 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.778 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:42.778 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240710+0000", 2024-03-19T11:31:42.778 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:42.778 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.778 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.778 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:42.778 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240710+0000", 2024-03-19T11:31:42.778 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9599999999999999e-07 2024-03-19T11:31:42.778 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.778 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.778 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:42.778 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240714+0000", 2024-03-19T11:31:42.778 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8480000000000003e-06 2024-03-19T11:31:42.778 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.779 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:42.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240726+0000", 2024-03-19T11:31:42.779 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.186e-05 2024-03-19T11:31:42.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.779 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:42.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240736+0000", 2024-03-19T11:31:42.779 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0040999999999999e-05 2024-03-19T11:31:42.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.779 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.241633+0000", 2024-03-19T11:31:42.779 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00089736299999999998 2024-03-19T11:31:42.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.780 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:31:42.780 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.241645+0000", 2024-03-19T11:31:42.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1600000000000001e-05 2024-03-19T11:31:42.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.780 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:42.780 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.088667+0000", 2024-03-19T11:31:42.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.84702242800000005 2024-03-19T11:31:42.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.780 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:42.780 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.088837+0000", 2024-03-19T11:31:42.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000169949 2024-03-19T11:31:42.781 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:42.781 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:42.781 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:42.781 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.089318+0000", 2024-03-19T11:31:42.781 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00048019399999999998 2024-03-19T11:31:42.781 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.781 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:42.781 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.781 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:42.781 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:42.781 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:42.783 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-19T11:31:42.783 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:42.789+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:42.785 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:31:42.785 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:31:42.801 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:31:42.801 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:31:42.817 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:31:42.818 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:31:42.833 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:31:42.833 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:31:42.851 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:31:42.852 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:31:42.870 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:31:42.870 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:31:42.883 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:42.893+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:42.889 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:31:42.889 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:31:42.908 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:31:42.908 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:31:42.952 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:42.952 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:42.966 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-19T11:31:42.985 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:42.996+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:43.049 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-03-19T11:31:43.056 INFO:tasks.rados.rados.0.smithi038.stdout:1618: done (5 left) 2024-03-19T11:31:43.056 INFO:tasks.rados.rados.0.smithi038.stdout:1619: write (excl) oid 22 current snap is 38 2024-03-19T11:31:43.056 INFO:tasks.rados.rados.0.smithi038.stdout:1619: seq_num 291 ranges {490837=718873,1862260=479427,2875032=15805} 2024-03-19T11:31:43.060 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:43.069+0000 7f1835403640 -1 osd.6 121 handle_osd_map injecting map crc failure 2024-03-19T11:31:43.061 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:43.069+0000 7f923da1c640 -1 osd.7 121 handle_osd_map injecting map crc failure 2024-03-19T11:31:43.062 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:43.069+0000 7f143b8b4640 -1 osd.5 121 handle_osd_map injecting map crc failure 2024-03-19T11:31:43.064 INFO:tasks.rados.rados.0.smithi038.stdout:1619: writing smithi03829856-22 from 490837 to 1209710 tid 1 2024-03-19T11:31:43.071 INFO:tasks.rados.rados.0.smithi038.stdout:1619: writing smithi03829856-22 from 1862260 to 2341687 tid 2 2024-03-19T11:31:43.075 INFO:tasks.rados.rados.0.smithi038.stdout:1619: writing smithi03829856-22 from 2875032 to 2890837 tid 3 2024-03-19T11:31:43.075 INFO:tasks.rados.rados.0.smithi038.stdout:1620: snap_create 2024-03-19T11:31:43.075 INFO:tasks.rados.rados.0.smithi038.stdout:1613: finishing write tid 3 to smithi03829856-14 2024-03-19T11:31:43.075 INFO:tasks.rados.rados.0.smithi038.stdout:1613: finishing write tid 4 to smithi03829856-14 2024-03-19T11:31:43.075 INFO:tasks.rados.rados.0.smithi038.stdout:1613: finishing write tid 5 to smithi03829856-14 2024-03-19T11:31:43.075 INFO:tasks.rados.rados.0.smithi038.stdout:1613: oid 14 updating version 0 to 76 2024-03-19T11:31:43.075 INFO:tasks.rados.rados.0.smithi038.stdout:1613: oid 14 version 76 is already newer than 75 2024-03-19T11:31:43.075 INFO:tasks.rados.rados.0.smithi038.stdout:1613: oid 14 updating version 76 to 77 2024-03-19T11:31:43.075 INFO:tasks.rados.rados.0.smithi038.stdout:1613: oid 14 updating version 77 to 78 2024-03-19T11:31:43.075 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 14 v 78 (ObjNum 289 snap 38 seq_num 289) dirty exists 2024-03-19T11:31:43.075 INFO:tasks.rados.rados.0.smithi038.stdout:1613: left oid 14 (ObjNum 289 snap 38 seq_num 289) 2024-03-19T11:31:43.075 INFO:tasks.rados.rados.0.smithi038.stdout:1615: got (16) Device or resource busy 2024-03-19T11:31:43.076 INFO:tasks.rados.rados.0.smithi038.stdout:check_oldest_snap_flushed oid 10 v 149 (ObjNum 179 snap 19 seq_num 179) unflushed 32 2024-03-19T11:31:43.076 INFO:tasks.rados.rados.0.smithi038.stdout:1614: expect (ObjNum 206 snap 22 seq_num 206) 2024-03-19T11:31:43.125 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:43.136+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:43.147 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:43.147 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:43.161 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-19T11:31:43.186 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:43.196+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:43.266 INFO:tasks.rados.rados.0.smithi038.stdout:1616: finishing write tid 1 to smithi03829856-9 2024-03-19T11:31:43.266 INFO:tasks.rados.rados.0.smithi038.stdout:1617: expect (ObjNum 143 snap 15 seq_num 143) 2024-03-19T11:31:43.283 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:43.284 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:43.287 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:43.296+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:43.297 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-19T11:31:43.344 INFO:tasks.rados.rados.0.smithi038.stdout:1616: finishing write tid 2 to smithi03829856-9 2024-03-19T11:31:43.344 INFO:tasks.rados.rados.0.smithi038.stdout:1616: finishing write tid 3 to smithi03829856-9 2024-03-19T11:31:43.345 INFO:tasks.rados.rados.0.smithi038.stdout:1616: finishing write tid 4 to smithi03829856-9 2024-03-19T11:31:43.345 INFO:tasks.rados.rados.0.smithi038.stdout:1616: oid 9 updating version 0 to 123 2024-03-19T11:31:43.345 INFO:tasks.rados.rados.0.smithi038.stdout:1616: oid 9 version 123 is already newer than 122 2024-03-19T11:31:43.345 INFO:tasks.rados.rados.0.smithi038.stdout:1616: oid 9 updating version 123 to 124 2024-03-19T11:31:43.345 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 9 v 124 (ObjNum 290 snap 38 seq_num 290) dirty exists 2024-03-19T11:31:43.345 INFO:tasks.rados.rados.0.smithi038.stdout:1616: left oid 9 (ObjNum 290 snap 38 seq_num 290) 2024-03-19T11:31:43.346 INFO:tasks.rados.rados.0.smithi038.stdout:1619: finishing write tid 1 to smithi03829856-22 2024-03-19T11:31:43.346 INFO:tasks.rados.rados.0.smithi038.stdout:1619: finishing write tid 2 to smithi03829856-22 2024-03-19T11:31:43.346 INFO:tasks.rados.rados.0.smithi038.stdout:1619: finishing write tid 3 to smithi03829856-22 2024-03-19T11:31:43.346 INFO:tasks.rados.rados.0.smithi038.stdout:1619: finishing write tid 4 to smithi03829856-22 2024-03-19T11:31:43.346 INFO:tasks.rados.rados.0.smithi038.stdout:1619: finishing write tid 5 to smithi03829856-22 2024-03-19T11:31:43.346 INFO:tasks.rados.rados.0.smithi038.stdout:1619: oid 22 updating version 0 to 35 2024-03-19T11:31:43.346 INFO:tasks.rados.rados.0.smithi038.stdout:1619: oid 22 updating version 35 to 36 2024-03-19T11:31:43.346 INFO:tasks.rados.rados.0.smithi038.stdout:1619: oid 22 version 36 is already newer than 33 2024-03-19T11:31:43.346 INFO:tasks.rados.rados.0.smithi038.stdout:1619: oid 22 version 36 is already newer than 34 2024-03-19T11:31:43.346 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 22 v 36 (ObjNum 291 snap 38 seq_num 291) dirty exists 2024-03-19T11:31:43.346 INFO:tasks.rados.rados.0.smithi038.stdout:1619: left oid 22 (ObjNum 291 snap 38 seq_num 291) 2024-03-19T11:31:43.388 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:43.396+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:43.425 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:43.425 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:43.439 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-19T11:31:43.489 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:43.497+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:43.517 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-19T11:31:43.518 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-19T11:31:43.518 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-03-19T11:31:43.518 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-19T11:31:43.518 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-19T11:31:43.518 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set filestore_inject_stall 3 2024-03-19T11:31:43.590 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:43.600+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:43.606 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:43.606 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:43.621 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-19T11:31:43.644 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:43.644 INFO:teuthology.orchestra.run.smithi038.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-19T11:31:43.644 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:43.705 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:43.716+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:43.777 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:43.777 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:43.792 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-19T11:31:43.793 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:43.801+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:43.893 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:43.901+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:43.966 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:43.966 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:43.980 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-19T11:31:44.025 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:44.036+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:44.053 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:44.064+0000 7fa5e08db640 -1 osd.3 122 handle_osd_map injecting map crc failure 2024-03-19T11:31:44.059 INFO:tasks.rados.rados.0.smithi038.stdout:1613: done (6 left) 2024-03-19T11:31:44.059 INFO:tasks.rados.rados.0.smithi038.stdout:1614: done (5 left) 2024-03-19T11:31:44.059 INFO:tasks.rados.rados.0.smithi038.stdout:1615: done (4 left) 2024-03-19T11:31:44.059 INFO:tasks.rados.rados.0.smithi038.stdout:1616: done (3 left) 2024-03-19T11:31:44.059 INFO:tasks.rados.rados.0.smithi038.stdout:1617: done (2 left) 2024-03-19T11:31:44.059 INFO:tasks.rados.rados.0.smithi038.stdout:1619: done (1 left) 2024-03-19T11:31:44.059 INFO:tasks.rados.rados.0.smithi038.stdout:1620: done (0 left) 2024-03-19T11:31:44.059 INFO:tasks.rados.rados.0.smithi038.stdout:1621: tier_evict oid 15 2024-03-19T11:31:44.059 INFO:tasks.rados.rados.0.smithi038.stdout:1621: tier_evict oid 15 snap -1 2024-03-19T11:31:44.060 INFO:tasks.rados.rados.0.smithi038.stdout:1622: read oid 25 snap -1 2024-03-19T11:31:44.060 INFO:tasks.rados.rados.0.smithi038.stdout:1622: expect deleted 2024-03-19T11:31:44.060 INFO:tasks.rados.rados.0.smithi038.stdout:1623: tier_evict oid 11 2024-03-19T11:31:44.060 INFO:tasks.rados.rados.0.smithi038.stdout:1623: tier_evict oid 11 snap -1 2024-03-19T11:31:44.060 INFO:tasks.rados.rados.0.smithi038.stdout:1624: read oid 19 snap -1 2024-03-19T11:31:44.060 INFO:tasks.rados.rados.0.smithi038.stdout:1624: expect (ObjNum 254 snap 32 seq_num 254) 2024-03-19T11:31:44.060 INFO:tasks.rados.rados.0.smithi038.stdout:1625: tier_promote oid 42 2024-03-19T11:31:44.060 INFO:tasks.rados.rados.0.smithi038.stdout:1626: set_chunk oid 36 target oid 2024-03-19T11:31:44.060 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:44.069+0000 7fb17dd72640 -1 osd.4 122 handle_osd_map injecting map crc failure 2024-03-19T11:31:44.083 INFO:tasks.rados.rados.0.smithi038.stdout:1626: set_chunk oid 36 offset: 1892864 length: 354304 target oid 0c67ea802ec2b4caf1543d58a97b3189eb94afab1df43c3c721c44b3f5082632 offset: 0 snap -1 2024-03-19T11:31:44.083 INFO:tasks.rados.rados.0.smithi038.stdout:1627: tier_promote oid 27 2024-03-19T11:31:44.083 INFO:tasks.rados.rados.0.smithi038.stdout:1625: got (2) No such file or directory 2024-03-19T11:31:44.084 INFO:tasks.rados.rados.0.smithi038.stdout:1625: got expected ENOENT (src dne) 2024-03-19T11:31:44.084 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 42 v 54 (ObjNum 97 snap 0 seq_num 1) dirty dne 2024-03-19T11:31:44.084 INFO:tasks.rados.rados.0.smithi038.stdout:1625: done (6 left) 2024-03-19T11:31:44.084 INFO:tasks.rados.rados.0.smithi038.stdout:1628: read oid 44 snap -1 2024-03-19T11:31:44.084 INFO:tasks.rados.rados.0.smithi038.stdout:1628: expect (ObjNum 288 snap 38 seq_num 288) 2024-03-19T11:31:44.084 INFO:tasks.rados.rados.0.smithi038.stdout:1629: tier_promote oid 35 2024-03-19T11:31:44.084 INFO:tasks.rados.rados.0.smithi038.stdout:1623: got (22) Invalid argument 2024-03-19T11:31:44.085 INFO:tasks.rados.rados.0.smithi038.stdout:1621: got (22) Invalid argument 2024-03-19T11:31:44.085 INFO:tasks.rados.rados.0.smithi038.stdout:1624: expect (ObjNum 254 snap 32 seq_num 254) 2024-03-19T11:31:44.095 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:44.101+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:44.116 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:44.116 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:44.133 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-19T11:31:44.196 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:44.205+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:44.220 INFO:tasks.rados.rados.0.smithi038.stdout:1629: got (0) Success 2024-03-19T11:31:44.220 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 35 v 239 (ObjNum 273 snap 0 seq_num 0) clean dne 2024-03-19T11:31:44.220 INFO:tasks.rados.rados.0.smithi038.stdout:1626: finishing set_chunk to oid 36 2024-03-19T11:31:44.220 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 36 v 127 (ObjNum 266 snap 34 seq_num 266) dirty exists 2024-03-19T11:31:44.220 INFO:tasks.rados.rados.0.smithi038.stdout:1628: expect (ObjNum 288 snap 38 seq_num 288) 2024-03-19T11:31:44.298 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:44.308+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:44.301 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:44.301 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:44.315 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-19T11:31:44.398 INFO:tasks.rados.rados.0.smithi038.stdout:1627: got (0) Success 2024-03-19T11:31:44.399 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 27 v 147 (ObjNum 240 snap 30 seq_num 240) dirty exists 2024-03-19T11:31:44.399 INFO:tasks.rados.rados.0.smithi038.stdout:1621: done (7 left) 2024-03-19T11:31:44.399 INFO:tasks.rados.rados.0.smithi038.stdout:1622: done (6 left) 2024-03-19T11:31:44.399 INFO:tasks.rados.rados.0.smithi038.stdout:1623: done (5 left) 2024-03-19T11:31:44.399 INFO:tasks.rados.rados.0.smithi038.stdout:1624: done (4 left) 2024-03-19T11:31:44.399 INFO:tasks.rados.rados.0.smithi038.stdout:1626: done (3 left) 2024-03-19T11:31:44.399 INFO:tasks.rados.rados.0.smithi038.stdout:1627: done (2 left) 2024-03-19T11:31:44.399 INFO:tasks.rados.rados.0.smithi038.stdout:1628: done (1 left) 2024-03-19T11:31:44.399 INFO:tasks.rados.rados.0.smithi038.stdout:1629: done (0 left) 2024-03-19T11:31:44.399 INFO:tasks.rados.rados.0.smithi038.stdout:1630: set_chunk oid 31 target oid 2024-03-19T11:31:44.400 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:44.405+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:44.408 INFO:tasks.rados.rados.0.smithi038.stdout:1630: set_chunk oid 31 offset: 1122816 length: 435712 target oid 1b6c3f4c5f64877316cf30a52a910deb21a940b1ed106f677db17957358e10da offset: 0 snap -1 2024-03-19T11:31:44.408 INFO:tasks.rados.rados.0.smithi038.stdout:1631: read oid 32 snap -1 2024-03-19T11:31:44.408 INFO:tasks.rados.rados.0.smithi038.stdout:1631: expect (ObjNum 245 snap 30 seq_num 245) 2024-03-19T11:31:44.409 INFO:tasks.rados.rados.0.smithi038.stdout:1632: write oid 4 current snap is 39 2024-03-19T11:31:44.409 INFO:tasks.rados.rados.0.smithi038.stdout:1632: seq_num 292 ranges {507644=794122,1920742=623085} 2024-03-19T11:31:44.419 INFO:tasks.rados.rados.0.smithi038.stdout:1632: writing smithi03829856-4 from 507644 to 1301766 tid 1 2024-03-19T11:31:44.431 INFO:tasks.rados.rados.0.smithi038.stdout:1632: writing smithi03829856-4 from 1920742 to 2543827 tid 2 2024-03-19T11:31:44.431 INFO:tasks.rados.rados.0.smithi038.stdout:1633: rollback oid 35 current snap is 39 2024-03-19T11:31:44.431 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 35 to 27 2024-03-19T11:31:44.431 INFO:tasks.rados.rados.0.smithi038.stdout:1630: finishing set_chunk to oid 31 2024-03-19T11:31:44.431 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 31 v 159 (ObjNum 268 snap 34 seq_num 268) dirty exists 2024-03-19T11:31:44.431 INFO:tasks.rados.rados.0.smithi038.stdout:1631: expect (ObjNum 245 snap 30 seq_num 245) 2024-03-19T11:31:44.487 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:44.487 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:44.499 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:44.505+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:44.501 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-19T11:31:44.545 INFO:tasks.rados.rados.0.smithi038.stdout:1632: finishing write tid 1 to smithi03829856-4 2024-03-19T11:31:44.546 INFO:tasks.rados.rados.0.smithi038.stdout:1633: finishing rollback tid 1 to smithi03829856-35 2024-03-19T11:31:44.546 INFO:tasks.rados.rados.0.smithi038.stdout:1633: finishing rollback tid 2 to smithi03829856-35 2024-03-19T11:31:44.546 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 35 v 275 (ObjNum 217 snap 23 seq_num 217) dirty exists 2024-03-19T11:31:44.546 INFO:tasks.rados.rados.0.smithi038.stdout:1632: finishing write tid 2 to smithi03829856-4 2024-03-19T11:31:44.546 INFO:tasks.rados.rados.0.smithi038.stdout:1632: finishing write tid 3 to smithi03829856-4 2024-03-19T11:31:44.546 INFO:tasks.rados.rados.0.smithi038.stdout:1632: finishing write tid 4 to smithi03829856-4 2024-03-19T11:31:44.546 INFO:tasks.rados.rados.0.smithi038.stdout:1632: oid 4 updating version 0 to 185 2024-03-19T11:31:44.546 INFO:tasks.rados.rados.0.smithi038.stdout:1632: oid 4 updating version 185 to 187 2024-03-19T11:31:44.546 INFO:tasks.rados.rados.0.smithi038.stdout:1632: oid 4 version 187 is already newer than 186 2024-03-19T11:31:44.546 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 4 v 187 (ObjNum 292 snap 39 seq_num 292) dirty exists 2024-03-19T11:31:44.547 INFO:tasks.rados.rados.0.smithi038.stdout:1632: left oid 4 (ObjNum 292 snap 39 seq_num 292) 2024-03-19T11:31:44.547 INFO:tasks.rados.rados.0.smithi038.stdout:1630: done (3 left) 2024-03-19T11:31:44.547 INFO:tasks.rados.rados.0.smithi038.stdout:1631: done (2 left) 2024-03-19T11:31:44.547 INFO:tasks.rados.rados.0.smithi038.stdout:1632: done (1 left) 2024-03-19T11:31:44.547 INFO:tasks.rados.rados.0.smithi038.stdout:1633: done (0 left) 2024-03-19T11:31:44.547 INFO:tasks.rados.rados.0.smithi038.stdout:1634: read oid 16 snap 33 2024-03-19T11:31:44.547 INFO:tasks.rados.rados.0.smithi038.stdout:1634: expect (ObjNum 65 snap 3 seq_num 65) 2024-03-19T11:31:44.547 INFO:tasks.rados.rados.0.smithi038.stdout:1635: read oid 9 snap 37 2024-03-19T11:31:44.547 INFO:tasks.rados.rados.0.smithi038.stdout:1635: expect (ObjNum 279 snap 37 seq_num 279) 2024-03-19T11:31:44.547 INFO:tasks.rados.rados.0.smithi038.stdout:1636: set_chunk oid 3 target oid 2024-03-19T11:31:44.569 INFO:tasks.rados.rados.0.smithi038.stdout:1636: set_chunk oid 3 offset: 1208320 length: 1283072 target oid 8ecde0a60bb5e7ef6be1337e3990deb05f80af3d10e60da83d980f35393bfa84 offset: 0 snap -1 2024-03-19T11:31:44.570 INFO:tasks.rados.rados.0.smithi038.stdout:1637: set_chunk oid 38 target oid 2024-03-19T11:31:44.584 INFO:tasks.rados.rados.0.smithi038.stdout:1637: set_chunk oid 38 offset: 308736 length: 1110016 target oid 09cebe7a5e6aff1a81fe6023910843a5c6b064281020f3f9d1190b5c1b392a5e offset: 0 snap -1 2024-03-19T11:31:44.584 INFO:tasks.rados.rados.0.smithi038.stdout:1638: rollback oid 34 current snap is 39 2024-03-19T11:31:44.584 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 34 to 27 2024-03-19T11:31:44.584 INFO:tasks.rados.rados.0.smithi038.stdout:1634: expect (ObjNum 65 snap 3 seq_num 65) 2024-03-19T11:31:44.600 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:44.609+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:44.685 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:44.686 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:44.700 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-19T11:31:44.701 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:44.712+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:44.739 INFO:tasks.rados.rados.0.smithi038.stdout:1635: expect (ObjNum 279 snap 37 seq_num 279) 2024-03-19T11:31:44.801 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:44.812+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:44.858 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:44.859 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:44.873 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-19T11:31:44.902 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:44.909+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:44.914 INFO:tasks.rados.rados.0.smithi038.stdout:1636: finishing set_chunk to oid 3 2024-03-19T11:31:44.914 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 3 v 114 (ObjNum 271 snap 34 seq_num 271) dirty exists 2024-03-19T11:31:44.914 INFO:tasks.rados.rados.0.smithi038.stdout:1634: done (4 left) 2024-03-19T11:31:44.914 INFO:tasks.rados.rados.0.smithi038.stdout:1635: done (3 left) 2024-03-19T11:31:44.914 INFO:tasks.rados.rados.0.smithi038.stdout:1636: done (2 left) 2024-03-19T11:31:44.915 INFO:tasks.rados.rados.0.smithi038.stdout:1639: tier_evict oid 8 2024-03-19T11:31:44.915 INFO:tasks.rados.rados.0.smithi038.stdout:1639: tier_evict oid 8 snap -1 2024-03-19T11:31:44.915 INFO:tasks.rados.rados.0.smithi038.stdout:1637: finishing set_chunk to oid 38 2024-03-19T11:31:44.915 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 38 v 145 (ObjNum 256 snap 32 seq_num 256) dirty exists 2024-03-19T11:31:44.915 INFO:tasks.rados.rados.0.smithi038.stdout:1638: finishing rollback tid 1 to smithi03829856-34 2024-03-19T11:31:44.915 INFO:tasks.rados.rados.0.smithi038.stdout:1638: finishing rollback tid 2 to smithi03829856-34 2024-03-19T11:31:44.915 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 34 v 53 (ObjNum 72 snap 5 seq_num 72) dirty exists 2024-03-19T11:31:44.915 INFO:tasks.rados.rados.0.smithi038.stdout:1637: done (2 left) 2024-03-19T11:31:44.915 INFO:tasks.rados.rados.0.smithi038.stdout:1638: done (1 left) 2024-03-19T11:31:44.916 INFO:tasks.rados.rados.0.smithi038.stdout:1640: set_chunk oid 24 target oid 2024-03-19T11:31:44.920 INFO:tasks.rados.rados.0.smithi038.stdout:1640: set_chunk oid 24 offset: 975872 length: 321024 target oid b3484d14924e3de62478ce3ab4b3a498f3600de7c2bcf93c0208b369cd127221 offset: 0 snap 35 2024-03-19T11:31:44.921 INFO:tasks.rados.rados.0.smithi038.stdout:1641: delete oid 46 current snap is 39 2024-03-19T11:31:44.921 INFO:tasks.rados.rados.0.smithi038.stdout:1639: got (22) Invalid argument 2024-03-19T11:31:44.924 INFO:tasks.rados.rados.0.smithi038.stdout:1639: done (2 left) 2024-03-19T11:31:44.925 INFO:tasks.rados.rados.0.smithi038.stdout:1641: done (1 left) 2024-03-19T11:31:44.925 INFO:tasks.rados.rados.0.smithi038.stdout:1642: read oid 43 snap -1 2024-03-19T11:31:44.925 INFO:tasks.rados.rados.0.smithi038.stdout:1642: expect (ObjNum 237 snap 29 seq_num 237) 2024-03-19T11:31:44.925 INFO:tasks.rados.rados.0.smithi038.stdout:1643: read oid 15 snap -1 2024-03-19T11:31:44.925 INFO:tasks.rados.rados.0.smithi038.stdout:1643: expect (ObjNum 286 snap 37 seq_num 286) 2024-03-19T11:31:44.925 INFO:tasks.rados.rados.0.smithi038.stdout:1644: write (excl) oid 35 current snap is 39 2024-03-19T11:31:44.925 INFO:tasks.rados.rados.0.smithi038.stdout:1644: seq_num 294 ranges {541258=544620,1637706=626754,2840948=100310} 2024-03-19T11:31:44.926 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-19T11:31:44.932 INFO:tasks.rados.rados.0.smithi038.stdout:1644: writing smithi03829856-35 from 541258 to 1085878 tid 1 2024-03-19T11:31:44.939 INFO:tasks.rados.rados.0.smithi038.stdout:1644: writing smithi03829856-35 from 1637706 to 2264460 tid 2 2024-03-19T11:31:44.943 INFO:tasks.rados.rados.0.smithi038.stdout:1644: writing smithi03829856-35 from 2840948 to 2941258 tid 3 2024-03-19T11:31:44.944 INFO:tasks.rados.rados.0.smithi038.stdout:1645: read oid 39 snap -1 2024-03-19T11:31:44.944 INFO:tasks.rados.rados.0.smithi038.stdout:1645: expect deleted 2024-03-19T11:31:44.944 INFO:tasks.rados.rados.0.smithi038.stdout:1646: write oid 22 current snap is 39 2024-03-19T11:31:44.944 INFO:tasks.rados.rados.0.smithi038.stdout:1646: seq_num 295 ranges {558065=619869,1579835=486765,2468371=489694} 2024-03-19T11:31:44.950 INFO:tasks.rados.rados.0.smithi038.stdout:1646: writing smithi03829856-22 from 558065 to 1177934 tid 1 2024-03-19T11:31:44.955 INFO:tasks.rados.rados.0.smithi038.stdout:1646: writing smithi03829856-22 from 1579835 to 2066600 tid 2 2024-03-19T11:31:44.961 INFO:tasks.rados.rados.0.smithi038.stdout:1646: writing smithi03829856-22 from 2468371 to 2958065 tid 3 2024-03-19T11:31:44.961 INFO:tasks.rados.rados.0.smithi038.stdout:1647: read oid 33 snap -1 2024-03-19T11:31:44.961 INFO:tasks.rados.rados.0.smithi038.stdout:1647: expect (ObjNum 273 snap 34 seq_num 273) 2024-03-19T11:31:44.961 INFO:tasks.rados.rados.0.smithi038.stdout:1648: delete oid 17 current snap is 39 2024-03-19T11:31:44.961 INFO:tasks.rados.rados.0.smithi038.stdout:1640: finishing set_chunk to oid 24 2024-03-19T11:31:44.962 INFO:tasks.rados.rados.0.smithi038.stdout:1642: expect (ObjNum 237 snap 29 seq_num 237) 2024-03-19T11:31:45.003 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:45.012+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:45.042 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:45.042 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:45.056 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-19T11:31:45.071 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:45.080+0000 7f6a7d9d6640 -1 osd.1 123 handle_osd_map injecting map crc failure 2024-03-19T11:31:45.084 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-03-19T11:31:45.096 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:31:45.096 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:31:45.104 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:45.113+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:45.113 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:31:45.114 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:31:45.131 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:31:45.131 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:31:45.146 INFO:tasks.rados.rados.0.smithi038.stdout:1643: expect (ObjNum 286 snap 37 seq_num 286) 2024-03-19T11:31:45.149 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:31:45.150 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:31:45.187 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:31:45.187 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:31:45.205 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:45.213+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:45.219 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:45.219 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:31:45.219 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:31:45.219 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:31:45.219 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:45.224 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:31:45.224 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:31:45.235 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-19T11:31:45.245 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:31:45.246 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:31:45.268 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:31:45.268 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:31:45.329 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:45.340+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:45.342 INFO:tasks.rados.rados.0.smithi038.stdout:1644: finishing write tid 1 to smithi03829856-35 2024-03-19T11:31:45.342 INFO:tasks.rados.rados.0.smithi038.stdout:1644: finishing write tid 2 to smithi03829856-35 2024-03-19T11:31:45.342 INFO:tasks.rados.rados.0.smithi038.stdout:1644: finishing write tid 3 to smithi03829856-35 2024-03-19T11:31:45.342 INFO:tasks.rados.rados.0.smithi038.stdout:1644: finishing write tid 4 to smithi03829856-35 2024-03-19T11:31:45.342 INFO:tasks.rados.rados.0.smithi038.stdout:1644: finishing write tid 5 to smithi03829856-35 2024-03-19T11:31:45.342 INFO:tasks.rados.rados.0.smithi038.stdout:1644: oid 35 updating version 0 to 279 2024-03-19T11:31:45.342 INFO:tasks.rados.rados.0.smithi038.stdout:1644: oid 35 version 279 is already newer than 278 2024-03-19T11:31:45.342 INFO:tasks.rados.rados.0.smithi038.stdout:1644: oid 35 version 279 is already newer than 276 2024-03-19T11:31:45.342 INFO:tasks.rados.rados.0.smithi038.stdout:1644: oid 35 version 279 is already newer than 277 2024-03-19T11:31:45.343 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 35 v 279 (ObjNum 294 snap 39 seq_num 294) dirty exists 2024-03-19T11:31:45.343 INFO:tasks.rados.rados.0.smithi038.stdout:1644: left oid 35 (ObjNum 294 snap 39 seq_num 294) 2024-03-19T11:31:45.343 INFO:tasks.rados.rados.0.smithi038.stdout:1640: done (7 left) 2024-03-19T11:31:45.343 INFO:tasks.rados.rados.0.smithi038.stdout:1642: done (6 left) 2024-03-19T11:31:45.343 INFO:tasks.rados.rados.0.smithi038.stdout:1643: done (5 left) 2024-03-19T11:31:45.343 INFO:tasks.rados.rados.0.smithi038.stdout:1644: done (4 left) 2024-03-19T11:31:45.343 INFO:tasks.rados.rados.0.smithi038.stdout:1645: done (3 left) 2024-03-19T11:31:45.343 INFO:tasks.rados.rados.0.smithi038.stdout:1648: done (2 left) 2024-03-19T11:31:45.343 INFO:tasks.rados.rados.0.smithi038.stdout:1649: read oid 32 snap -1 2024-03-19T11:31:45.343 INFO:tasks.rados.rados.0.smithi038.stdout:1649: expect (ObjNum 245 snap 30 seq_num 245) 2024-03-19T11:31:45.343 INFO:tasks.rados.rados.0.smithi038.stdout:1650: write (excl) oid 38 current snap is 39 2024-03-19T11:31:45.343 INFO:tasks.rados.rados.0.smithi038.stdout:1650: seq_num 297 ranges {591679=486720,1813152=490434} 2024-03-19T11:31:45.347 INFO:tasks.rados.rados.0.smithi038.stdout:1650: writing smithi03829856-38 from 591679 to 1078399 tid 1 2024-03-19T11:31:45.355 INFO:tasks.rados.rados.0.smithi038.stdout:1650: writing smithi03829856-38 from 1813152 to 2303586 tid 2 2024-03-19T11:31:45.355 INFO:tasks.rados.rados.0.smithi038.stdout:1651: read oid 12 snap -1 2024-03-19T11:31:45.355 INFO:tasks.rados.rados.0.smithi038.stdout:1651: expect (ObjNum 228 snap 26 seq_num 228) 2024-03-19T11:31:45.356 INFO:tasks.rados.rados.0.smithi038.stdout:1652: write (excl) oid 49 current snap is 39 2024-03-19T11:31:45.356 INFO:tasks.rados.rados.0.smithi038.stdout:1646: finishing write tid 1 to smithi03829856-22 2024-03-19T11:31:45.356 INFO:tasks.rados.rados.0.smithi038.stdout:1646: finishing write tid 2 to smithi03829856-22 2024-03-19T11:31:45.356 INFO:tasks.rados.rados.0.smithi038.stdout:1647: expect (ObjNum 273 snap 34 seq_num 273) 2024-03-19T11:31:45.371 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:45.371 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:31:45.371 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:31:45.371 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:31:45.371 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.371 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1067 4.0 4:01018511:::smithi03829856-19:head [write 477332~766101] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:31:45.371 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.950506+0000", 2024-03-19T11:31:45.371 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 91.425844251000001, 2024-03-19T11:31:45.372 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016279426, 2024-03-19T11:31:45.372 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:45.372 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:45.372 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:45.372 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:45.372 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:45.372 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1067 2024-03-19T11:31:45.372 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.372 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:45.372 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.372 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:45.372 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950506+0000", 2024-03-19T11:31:45.372 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:45.372 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.373 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.373 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:45.373 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950506+0000", 2024-03-19T11:31:45.373 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:45.373 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.373 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.373 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:45.373 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950509+0000", 2024-03-19T11:31:45.373 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.021e-06 2024-03-19T11:31:45.373 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.373 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.373 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:45.373 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950892+0000", 2024-03-19T11:31:45.373 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00038263300000000001 2024-03-19T11:31:45.373 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.374 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.374 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:45.374 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950912+0000", 2024-03-19T11:31:45.374 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0539000000000001e-05 2024-03-19T11:31:45.374 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.374 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.374 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:45.374 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950922+0000", 2024-03-19T11:31:45.374 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.577e-06 2024-03-19T11:31:45.374 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.374 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.374 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:45.374 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.950982+0000", 2024-03-19T11:31:45.374 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0760000000000001e-05 2024-03-19T11:31:45.375 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.375 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.375 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:45.375 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.951058+0000", 2024-03-19T11:31:45.375 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5078000000000002e-05 2024-03-19T11:31:45.375 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.375 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.375 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:45.375 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.951173+0000", 2024-03-19T11:31:45.375 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011516200000000001 2024-03-19T11:31:45.375 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.375 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.375 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:45.375 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.955064+0000", 2024-03-19T11:31:45.376 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003890896 2024-03-19T11:31:45.376 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.376 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.376 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:45.376 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966670+0000", 2024-03-19T11:31:45.376 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011606795 2024-03-19T11:31:45.376 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.376 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.376 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:45.376 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966731+0000", 2024-03-19T11:31:45.376 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0528000000000001e-05 2024-03-19T11:31:45.376 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.376 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.376 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:45.376 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.966785+0000", 2024-03-19T11:31:45.377 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4437000000000001e-05 2024-03-19T11:31:45.377 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:45.377 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:45.377 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:45.377 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.377 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.377 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(3.1 49/48 [PushOp(3:9205291d:::smithi03829856-50 low_tier:head, version: 28'36, data_included: [393216~851968,1703936~196608], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:9205291d:::smithi03829856-50 low_tier:head@28'36, size: 2823543, copy_subset: [(0, 2823543)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1900544, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:baab691a:::smithi03829856-2 low_tier:head, version: 28'37, data_included: [393216~524288,1507328~509479], data_size: 1033767, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:baab691a:::smithi03829856-2 low_tier:head@28'37, size: 2016807, copy_subset: [(0, 2016807)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2016807, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:98ce8426:::smithi03829856-4 low_tier:head, version: 28'38, data_included: [393216~720896,1638400~327680], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:98ce8426:::smithi03829856-4 low_tier:head@28'38, size: 2050421, copy_subset: [(0, 2050421)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1966080, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:a86be07b:::smithi03829856-44 low_tier:head, version: 28'41, data_included: [720896~655360,1769472~393216], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:a86be07b:::smithi03829856-44 low_tier:head@28'41, size: 2722701, copy_subset: [(0, 2722701)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2162688, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:bb56919e:::smithi03829856-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'44, data_included: [393216~786432,1703936~262144], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:bb56919e:::smithi03829856-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'44, size: 2806736, copy_subset: [(0, 2806736)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1966080, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:88687f71:::smithi03829856-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'45, data_included: [589824~655360,1638400~393216], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:88687f71:::smithi03829856-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'45, size: 2201684, copy_subset: [(0, 2201684)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2031616, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:b1f822cf:::smithi03829856-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'46, data_included: [720896~786432,1966080~262144], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:b1f822cf:::smithi03829856-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'46, size: 2369754, copy_subset: [(0, 2369754)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2228224, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:a5e8313a:::smithi03829856-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'47, data_included: [524288~720896,1769472~327680], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:a5e8313a:::smithi03829856-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'47, size: 2571438, copy_subset: [(0, 2571438)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:bca5ec64:::e3a26bf9230b1b95b8cf1a4cb01909236bf2bf3fd59b083b2291d278ab98264d:head, version: 28'48, data_included: [0~393216], data_size: 393216, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bca5ec64:::e3a26bf9230b1b95b8cf1a4cb01909236bf2bf3fd59b083b2291d278ab98264d:head@28'48, size: 533504, copy_subset: [(0, 533504)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 533504, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:45.377 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:26.948137+0000", 2024-03-19T11:31:45.377 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 78.428213036000002, 2024-03-19T11:31:45.377 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.036085578, 2024-03-19T11:31:45.377 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:45.377 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:45.377 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:45.377 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.378 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:45.378 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.948137+0000", 2024-03-19T11:31:45.378 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:45.378 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.378 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.378 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:45.378 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.948137+0000", 2024-03-19T11:31:45.378 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:45.378 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.378 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.378 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:45.378 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.948139+0000", 2024-03-19T11:31:45.378 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1399999999999998e-06 2024-03-19T11:31:45.378 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.379 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.379 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:45.379 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954288+0000", 2024-03-19T11:31:45.379 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0061487429999999999 2024-03-19T11:31:45.379 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.379 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.379 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:45.379 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954388+0000", 2024-03-19T11:31:45.379 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010022 2024-03-19T11:31:45.379 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.379 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.379 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:45.379 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954523+0000", 2024-03-19T11:31:45.379 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000134332 2024-03-19T11:31:45.379 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.380 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.380 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:45.380 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.955120+0000", 2024-03-19T11:31:45.380 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00059695799999999997 2024-03-19T11:31:45.380 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.380 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.380 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:45.380 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.955194+0000", 2024-03-19T11:31:45.380 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4077000000000002e-05 2024-03-19T11:31:45.380 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.380 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.380 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:45.380 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.984223+0000", 2024-03-19T11:31:45.380 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029029108000000001 2024-03-19T11:31:45.380 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:45.381 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:45.381 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:45.381 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.381 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.381 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(3.1 49/48 [PushOp(3:99a49d1b:::895a63407b6109b3cca9a6f5ef827134d56679bef42daadc5892269682c428ed:head, version: 32'49, data_included: [393216~388096], data_size: 388096, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:99a49d1b:::895a63407b6109b3cca9a6f5ef827134d56679bef42daadc5892269682c428ed:head@32'49, size: 781312, copy_subset: [(0, 781312)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 781312, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:45.381 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:26.954637+0000", 2024-03-19T11:31:45.381 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 78.421713419, 2024-03-19T11:31:45.381 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.030729057000000001, 2024-03-19T11:31:45.381 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:45.381 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:45.381 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:45.381 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.381 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:45.381 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954637+0000", 2024-03-19T11:31:45.381 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:45.382 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.382 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.382 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:45.382 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954637+0000", 2024-03-19T11:31:45.382 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:45.382 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.382 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.382 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:45.382 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954637+0000", 2024-03-19T11:31:45.382 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7299999999999995e-07 2024-03-19T11:31:45.382 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.382 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.382 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:45.382 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954873+0000", 2024-03-19T11:31:45.383 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023520299999999999 2024-03-19T11:31:45.383 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.383 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.383 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:45.383 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954897+0000", 2024-03-19T11:31:45.383 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4110000000000001e-05 2024-03-19T11:31:45.383 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.383 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.383 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:45.383 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954917+0000", 2024-03-19T11:31:45.383 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.067e-05 2024-03-19T11:31:45.383 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.383 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.383 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:45.383 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.984289+0000", 2024-03-19T11:31:45.384 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029371191000000001 2024-03-19T11:31:45.384 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.384 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.384 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:45.384 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.984303+0000", 2024-03-19T11:31:45.384 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4888999999999999e-05 2024-03-19T11:31:45.384 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.384 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.384 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:45.384 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.985366+0000", 2024-03-19T11:31:45.384 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001062321 2024-03-19T11:31:45.384 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:45.384 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:45.384 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:45.385 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.385 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.385 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1472 4.10 4:0dc3b91e:::smithi03829856-28:head [read 534528~391680] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-03-19T11:31:45.385 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:27.091748+0000", 2024-03-19T11:31:45.385 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 78.284602531000004, 2024-03-19T11:31:45.385 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.83574118500000005, 2024-03-19T11:31:45.385 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:45.385 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:45.385 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:45.385 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:45.385 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:45.385 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1472 2024-03-19T11:31:45.385 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.385 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:45.386 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.386 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:45.386 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091748+0000", 2024-03-19T11:31:45.386 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:45.386 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.386 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.386 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:45.386 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091748+0000", 2024-03-19T11:31:45.386 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:45.386 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.386 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.386 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:45.386 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091750+0000", 2024-03-19T11:31:45.386 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3130000000000001e-06 2024-03-19T11:31:45.386 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.387 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.387 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:45.387 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091760+0000", 2024-03-19T11:31:45.387 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0560999999999999e-05 2024-03-19T11:31:45.387 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.387 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.387 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:45.387 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091784+0000", 2024-03-19T11:31:45.387 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3915000000000001e-05 2024-03-19T11:31:45.387 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.387 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.387 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:45.388 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091797+0000", 2024-03-19T11:31:45.388 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2660000000000001e-05 2024-03-19T11:31:45.388 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.388 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.388 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:45.388 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091874+0000", 2024-03-19T11:31:45.388 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6757999999999999e-05 2024-03-19T11:31:45.388 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.388 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.388 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:45.388 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.091882+0000", 2024-03-19T11:31:45.388 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5520000000000005e-06 2024-03-19T11:31:45.388 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.388 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.388 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:45.389 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.927033+0000", 2024-03-19T11:31:45.389 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.83515094099999998 2024-03-19T11:31:45.389 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.389 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.389 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:45.389 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.927250+0000", 2024-03-19T11:31:45.389 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00021675199999999999 2024-03-19T11:31:45.389 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.389 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.389 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:45.389 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.927489+0000", 2024-03-19T11:31:45.389 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023873300000000001 2024-03-19T11:31:45.389 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:45.389 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:45.389 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:45.390 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.390 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.390 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1474 4.10 4:0dc3b91e:::smithi03829856-28:head [assert-version v27,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-03-19T11:31:45.390 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:27.934139+0000", 2024-03-19T11:31:45.390 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 77.442210744999997, 2024-03-19T11:31:45.390 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024764884000000001, 2024-03-19T11:31:45.390 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:45.390 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:45.390 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:45.390 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:45.390 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:45.390 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1474 2024-03-19T11:31:45.390 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.390 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:45.391 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.391 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:45.391 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934139+0000", 2024-03-19T11:31:45.391 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:45.391 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.391 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.391 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:45.391 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934139+0000", 2024-03-19T11:31:45.391 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:45.391 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.391 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.391 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:45.391 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934142+0000", 2024-03-19T11:31:45.391 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5459999999999998e-06 2024-03-19T11:31:45.391 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.392 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.392 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:45.392 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934152+0000", 2024-03-19T11:31:45.392 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9399999999999997e-06 2024-03-19T11:31:45.392 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.392 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.392 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:45.392 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934173+0000", 2024-03-19T11:31:45.392 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1522000000000002e-05 2024-03-19T11:31:45.392 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.392 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.392 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:45.392 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934185+0000", 2024-03-19T11:31:45.392 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1946999999999999e-05 2024-03-19T11:31:45.392 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.393 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.393 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:45.393 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934282+0000", 2024-03-19T11:31:45.393 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6272000000000003e-05 2024-03-19T11:31:45.393 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.393 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.393 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:45.393 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.934379+0000", 2024-03-19T11:31:45.393 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6966999999999995e-05 2024-03-19T11:31:45.393 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.393 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.393 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:45.393 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.955023+0000", 2024-03-19T11:31:45.393 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020644302 2024-03-19T11:31:45.394 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.394 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.394 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:45.394 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.955167+0000", 2024-03-19T11:31:45.394 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014404500000000001 2024-03-19T11:31:45.394 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.394 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.394 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1", 2024-03-19T11:31:45.394 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.957913+0000", 2024-03-19T11:31:45.394 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0027463779999999998 2024-03-19T11:31:45.394 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.394 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.394 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:45.394 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.958548+0000", 2024-03-19T11:31:45.394 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00063495700000000001 2024-03-19T11:31:45.395 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.395 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.395 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:45.395 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.958785+0000", 2024-03-19T11:31:45.395 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000236872 2024-03-19T11:31:45.395 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.395 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.395 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:45.395 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.958854+0000", 2024-03-19T11:31:45.395 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9099999999999999e-05 2024-03-19T11:31:45.395 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.395 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.395 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:45.395 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:27.958904+0000", 2024-03-19T11:31:45.396 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0036e-05 2024-03-19T11:31:45.396 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:45.396 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:45.396 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:45.396 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.396 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.396 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 75/74 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:7, version: 55'82, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@55'82, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 58'86, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@58'86, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [504210~968657,1572695~79919], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:12, version: 66'101, data_included: [440350~816629,1309629~231947], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:12@66'101, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:10, version: 70'112, data_included: [440350~816629,1309629~231947], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:10@70'112, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 70'113, data_included: [440350~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@70'113, size: 2605085, copy_subset: [(0, 2605085)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1488926, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:45.396 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.108941+0000", 2024-03-19T11:31:45.396 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 51.267409008000001, 2024-03-19T11:31:45.396 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.067446694000000001, 2024-03-19T11:31:45.396 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:45.396 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:45.396 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:45.396 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.397 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:45.397 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.108941+0000", 2024-03-19T11:31:45.397 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:45.397 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.397 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.397 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:45.397 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.108941+0000", 2024-03-19T11:31:45.397 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:45.397 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.397 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.397 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:45.397 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.108943+0000", 2024-03-19T11:31:45.397 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2259999999999999e-06 2024-03-19T11:31:45.397 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.397 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.398 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:45.398 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123892+0000", 2024-03-19T11:31:45.398 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014948655999999999 2024-03-19T11:31:45.398 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.398 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.398 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:45.398 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123993+0000", 2024-03-19T11:31:45.398 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000101236 2024-03-19T11:31:45.398 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.398 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.398 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:45.398 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124126+0000", 2024-03-19T11:31:45.398 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000133184 2024-03-19T11:31:45.398 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.399 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.399 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:45.399 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124655+0000", 2024-03-19T11:31:45.399 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00052849600000000004 2024-03-19T11:31:45.399 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.399 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.399 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:45.399 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124723+0000", 2024-03-19T11:31:45.399 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7797000000000002e-05 2024-03-19T11:31:45.399 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.399 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.399 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:45.399 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.176388+0000", 2024-03-19T11:31:45.399 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.051665098999999999 2024-03-19T11:31:45.399 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:45.400 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:45.400 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:45.400 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.400 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.400 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.21 e75,74 4:8401c1f0:::smithi03829856-43:8,46'66,4:866f3dcc:::smithi03829856-31:2,47'73)", 2024-03-19T11:31:45.400 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.124226+0000", 2024-03-19T11:31:45.400 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 51.252124352000003, 2024-03-19T11:31:45.400 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.052217913999999997, 2024-03-19T11:31:45.400 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:45.400 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:45.400 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:45.400 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.400 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:45.400 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124226+0000", 2024-03-19T11:31:45.400 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:45.401 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.401 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.401 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:45.401 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124226+0000", 2024-03-19T11:31:45.401 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:45.401 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.401 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.401 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:45.401 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124227+0000", 2024-03-19T11:31:45.401 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.477e-06 2024-03-19T11:31:45.401 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.401 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.401 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:45.401 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124240+0000", 2024-03-19T11:31:45.402 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2379e-05 2024-03-19T11:31:45.402 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.402 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.402 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:45.402 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124264+0000", 2024-03-19T11:31:45.402 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4224999999999999e-05 2024-03-19T11:31:45.402 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.402 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.402 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:45.402 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.124287+0000", 2024-03-19T11:31:45.402 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3557000000000002e-05 2024-03-19T11:31:45.402 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.402 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.402 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:45.403 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.176274+0000", 2024-03-19T11:31:45.403 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.051986248999999998 2024-03-19T11:31:45.403 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.403 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.403 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:45.403 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.176285+0000", 2024-03-19T11:31:45.403 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0818999999999999e-05 2024-03-19T11:31:45.403 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.403 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.403 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:45.403 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.176444+0000", 2024-03-19T11:31:45.403 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015920800000000001 2024-03-19T11:31:45.403 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:45.403 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:45.404 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:45.404 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.404 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.404 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 75/74 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:7, version: 55'82, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@55'82, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 58'86, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@58'86, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [1652614~973634,2888511~52681,3182305~22261], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3204566, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:12, version: 66'101, data_included: [1541576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:12@66'101, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:10, version: 70'112, data_included: [1541576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:10@70'112, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 70'113, data_included: [1488926~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@70'113, size: 2605085, copy_subset: [(0, 2605085)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2537502, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1488926, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:45.404 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.195615+0000", 2024-03-19T11:31:45.404 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 51.180735454999997, 2024-03-19T11:31:45.404 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.036930510999999999, 2024-03-19T11:31:45.404 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:45.404 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:45.404 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:45.404 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.404 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:45.404 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.195615+0000", 2024-03-19T11:31:45.404 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:45.405 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.405 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.405 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:45.405 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.195615+0000", 2024-03-19T11:31:45.405 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:45.405 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.405 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.405 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:45.405 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.195617+0000", 2024-03-19T11:31:45.405 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0490000000000002e-06 2024-03-19T11:31:45.405 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.405 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.405 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:45.405 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204654+0000", 2024-03-19T11:31:45.406 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0090370339999999993 2024-03-19T11:31:45.406 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.406 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.406 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:45.406 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204730+0000", 2024-03-19T11:31:45.406 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5939000000000006e-05 2024-03-19T11:31:45.406 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.406 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.406 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:45.406 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204844+0000", 2024-03-19T11:31:45.406 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000114066 2024-03-19T11:31:45.406 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.406 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.407 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:45.407 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.205232+0000", 2024-03-19T11:31:45.407 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000388045 2024-03-19T11:31:45.407 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.407 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.407 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:45.407 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.205277+0000", 2024-03-19T11:31:45.407 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5071999999999998e-05 2024-03-19T11:31:45.407 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.407 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.407 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:45.407 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.232545+0000", 2024-03-19T11:31:45.407 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027268305999999999 2024-03-19T11:31:45.407 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:45.407 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:45.408 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:45.408 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.408 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.408 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.21 e75,74 4:866f3dcc:::smithi03829856-31:c,70'114)", 2024-03-19T11:31:45.408 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.204941+0000", 2024-03-19T11:31:45.408 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 51.171408829999997, 2024-03-19T11:31:45.408 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.028797390999999999, 2024-03-19T11:31:45.408 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:45.408 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:45.408 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:45.408 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.408 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:45.408 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204941+0000", 2024-03-19T11:31:45.408 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:45.409 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.409 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.409 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:45.409 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204941+0000", 2024-03-19T11:31:45.409 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:45.409 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.409 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.409 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:45.409 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204943+0000", 2024-03-19T11:31:45.409 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.35e-06 2024-03-19T11:31:45.409 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.409 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.409 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:45.409 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204950+0000", 2024-03-19T11:31:45.410 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.374e-06 2024-03-19T11:31:45.410 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.410 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.410 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:45.410 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204965+0000", 2024-03-19T11:31:45.410 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5169e-05 2024-03-19T11:31:45.410 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.410 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.410 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:45.410 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.204979+0000", 2024-03-19T11:31:45.410 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4260000000000001e-05 2024-03-19T11:31:45.410 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.411 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.411 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:45.411 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.232607+0000", 2024-03-19T11:31:45.411 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027627518 2024-03-19T11:31:45.411 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.411 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.411 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:45.411 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.232626+0000", 2024-03-19T11:31:45.411 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9471999999999999e-05 2024-03-19T11:31:45.411 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.411 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.411 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:45.411 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.233739+0000", 2024-03-19T11:31:45.411 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011122479999999999 2024-03-19T11:31:45.412 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:45.412 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:45.412 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:45.412 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.412 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.412 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2789 3.2 3:48367551:::ca87bd16019841aaa831bb522ee079e5f4e6a1ea67fbcb51e5f89263b6c5d8f6:head [write 0~3280896] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-03-19T11:31:45.412 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:12.340289+0000", 2024-03-19T11:31:45.412 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 33.036061113999999, 2024-03-19T11:31:45.412 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016291298999999999, 2024-03-19T11:31:45.412 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:45.412 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:45.412 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:45.412 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:45.412 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:45.413 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2789 2024-03-19T11:31:45.413 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.413 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:45.413 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.413 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:45.413 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.340289+0000", 2024-03-19T11:31:45.413 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:45.413 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.413 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.413 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:45.413 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.340289+0000", 2024-03-19T11:31:45.413 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:45.413 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.413 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.413 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:45.414 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.340291+0000", 2024-03-19T11:31:45.414 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1270000000000001e-06 2024-03-19T11:31:45.414 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.414 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.414 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:45.414 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.341426+0000", 2024-03-19T11:31:45.414 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011348459999999999 2024-03-19T11:31:45.414 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.414 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.414 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:45.414 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.341454+0000", 2024-03-19T11:31:45.414 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8204e-05 2024-03-19T11:31:45.414 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.415 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.415 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:45.415 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.341467+0000", 2024-03-19T11:31:45.415 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2700999999999999e-05 2024-03-19T11:31:45.415 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.415 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.415 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:45.415 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.341623+0000", 2024-03-19T11:31:45.415 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015600100000000001 2024-03-19T11:31:45.415 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.415 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.415 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:45.415 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.342034+0000", 2024-03-19T11:31:45.415 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000411247 2024-03-19T11:31:45.415 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.416 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.416 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5", 2024-03-19T11:31:45.416 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.342443+0000", 2024-03-19T11:31:45.416 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00040901400000000002 2024-03-19T11:31:45.416 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.416 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.416 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:45.416 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.356159+0000", 2024-03-19T11:31:45.416 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013716058999999999 2024-03-19T11:31:45.416 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.416 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.416 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:45.416 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.356462+0000", 2024-03-19T11:31:45.416 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000302594 2024-03-19T11:31:45.417 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.417 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.417 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:45.417 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.356530+0000", 2024-03-19T11:31:45.417 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7852e-05 2024-03-19T11:31:45.417 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.417 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.417 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:45.417 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:12.356580+0000", 2024-03-19T11:31:45.417 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0654e-05 2024-03-19T11:31:45.417 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:45.417 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:45.417 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:45.417 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.417 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.418 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3046 4.19 4:9c74b127:::smithi03829856-33:head [write 400066~716205,stat] snapc 1f=[1f,1e,1d,1c,1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-03-19T11:31:45.418 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:22.147457+0000", 2024-03-19T11:31:45.418 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 23.228892829999999, 2024-03-19T11:31:45.418 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016153588, 2024-03-19T11:31:45.418 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:45.418 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:45.418 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:45.418 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:45.418 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:45.418 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3046 2024-03-19T11:31:45.418 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.418 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:45.418 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.418 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:45.419 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.147457+0000", 2024-03-19T11:31:45.419 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:45.419 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.419 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.419 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:45.419 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.147457+0000", 2024-03-19T11:31:45.419 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:45.419 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.419 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.419 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:45.419 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.147459+0000", 2024-03-19T11:31:45.419 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1160000000000002e-06 2024-03-19T11:31:45.419 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.419 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.419 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:45.420 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.148083+0000", 2024-03-19T11:31:45.420 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00062361999999999997 2024-03-19T11:31:45.420 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.420 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.420 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:45.420 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.148100+0000", 2024-03-19T11:31:45.420 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6524999999999998e-05 2024-03-19T11:31:45.420 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.420 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.420 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:45.420 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.148110+0000", 2024-03-19T11:31:45.420 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0093e-05 2024-03-19T11:31:45.420 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.420 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.421 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:45.421 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.148197+0000", 2024-03-19T11:31:45.421 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7639000000000006e-05 2024-03-19T11:31:45.421 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.421 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.421 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:45.421 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.148359+0000", 2024-03-19T11:31:45.421 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000161953 2024-03-19T11:31:45.421 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.421 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.421 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 2", 2024-03-19T11:31:45.421 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.149074+0000", 2024-03-19T11:31:45.421 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00071444100000000001 2024-03-19T11:31:45.421 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.422 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.422 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:45.422 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.158856+0000", 2024-03-19T11:31:45.422 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0097818169999999999 2024-03-19T11:31:45.422 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.422 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.422 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:45.422 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.163525+0000", 2024-03-19T11:31:45.422 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046690270000000001 2024-03-19T11:31:45.422 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.422 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.422 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:45.422 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.163569+0000", 2024-03-19T11:31:45.422 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4279000000000001e-05 2024-03-19T11:31:45.423 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.423 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.423 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:45.423 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.163611+0000", 2024-03-19T11:31:45.423 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2077999999999999e-05 2024-03-19T11:31:45.423 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:45.423 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:45.423 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:45.423 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.423 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.423 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3228 4.19 4:9bd2ffb1:::smithi03829856-30:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e109)", 2024-03-19T11:31:45.423 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:28.084149+0000", 2024-03-19T11:31:45.423 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 17.292200851, 2024-03-19T11:31:45.423 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.049907076000000002, 2024-03-19T11:31:45.423 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:45.424 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:45.424 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:45.424 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:45.424 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:45.424 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3228 2024-03-19T11:31:45.424 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.424 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:45.424 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.424 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:45.424 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.084149+0000", 2024-03-19T11:31:45.424 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:45.424 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.424 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.424 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:45.425 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.084149+0000", 2024-03-19T11:31:45.425 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:45.425 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.425 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.425 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:45.425 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.084152+0000", 2024-03-19T11:31:45.425 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2409999999999998e-06 2024-03-19T11:31:45.425 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.425 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.425 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:45.425 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.084161+0000", 2024-03-19T11:31:45.425 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9609999999999994e-06 2024-03-19T11:31:45.425 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.425 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.425 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:45.426 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.084181+0000", 2024-03-19T11:31:45.426 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0786999999999999e-05 2024-03-19T11:31:45.426 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.426 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.426 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:45.426 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.084194+0000", 2024-03-19T11:31:45.426 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2824e-05 2024-03-19T11:31:45.426 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.426 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.426 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:45.426 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.086656+0000", 2024-03-19T11:31:45.426 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0024622480000000002 2024-03-19T11:31:45.426 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.426 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.426 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:45.427 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.086894+0000", 2024-03-19T11:31:45.427 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000237414 2024-03-19T11:31:45.427 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.427 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.427 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:45.427 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.134056+0000", 2024-03-19T11:31:45.427 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.047162600999999998 2024-03-19T11:31:45.427 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:45.427 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:45.427 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:45.427 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.427 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.427 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3361 4.19 4:9ef41243:::smithi03829856-5:head [write 420241~546489,stat] snapc 23=[23,22,21,1f,1e,1d,1c,1b,1a,18,17] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-03-19T11:31:45.427 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:32.244085+0000", 2024-03-19T11:31:45.428 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 13.132265453, 2024-03-19T11:31:45.428 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025127221000000002, 2024-03-19T11:31:45.428 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:45.428 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:45.428 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:45.428 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:45.428 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:45.428 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3361 2024-03-19T11:31:45.428 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.428 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:45.428 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.428 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:45.428 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.244085+0000", 2024-03-19T11:31:45.428 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:45.429 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.429 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.429 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:45.429 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.244085+0000", 2024-03-19T11:31:45.429 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:45.429 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.429 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.429 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:45.429 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.244088+0000", 2024-03-19T11:31:45.429 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5779999999999999e-06 2024-03-19T11:31:45.429 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.429 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.429 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:45.429 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.244785+0000", 2024-03-19T11:31:45.429 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00069645700000000004 2024-03-19T11:31:45.430 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.430 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.430 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:45.430 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.244833+0000", 2024-03-19T11:31:45.430 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7902000000000003e-05 2024-03-19T11:31:45.430 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.430 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.430 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:45.430 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.244857+0000", 2024-03-19T11:31:45.430 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4301e-05 2024-03-19T11:31:45.430 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.430 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.430 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:45.430 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.245014+0000", 2024-03-19T11:31:45.430 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000157014 2024-03-19T11:31:45.431 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.431 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.431 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-03-19T11:31:45.431 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.245038+0000", 2024-03-19T11:31:45.431 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3655e-05 2024-03-19T11:31:45.431 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.431 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.431 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:45.431 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.251603+0000", 2024-03-19T11:31:45.431 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0065650650000000001 2024-03-19T11:31:45.431 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.431 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.431 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:45.431 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.251796+0000", 2024-03-19T11:31:45.432 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019291100000000001 2024-03-19T11:31:45.432 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.432 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.432 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 2", 2024-03-19T11:31:45.432 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.252201+0000", 2024-03-19T11:31:45.432 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00040590400000000001 2024-03-19T11:31:45.432 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.432 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.432 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:45.432 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.264409+0000", 2024-03-19T11:31:45.432 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012207314 2024-03-19T11:31:45.432 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.432 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.432 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:45.432 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.269042+0000", 2024-03-19T11:31:45.433 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046334949999999996 2024-03-19T11:31:45.433 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.433 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.433 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:45.433 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.269135+0000", 2024-03-19T11:31:45.433 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2310000000000002e-05 2024-03-19T11:31:45.433 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.433 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.433 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:45.433 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.269212+0000", 2024-03-19T11:31:45.433 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7315000000000001e-05 2024-03-19T11:31:45.433 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:45.433 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:45.433 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:45.434 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.434 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.434 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3399 4.21 4:866f3dcc:::smithi03829856-31:head [write 504276~639087,stat] snapc 23=[23,22,21,1f,1e,1d,1c,1b,1a,18,17] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-03-19T11:31:45.434 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:33.034450+0000", 2024-03-19T11:31:45.434 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 12.341899841, 2024-03-19T11:31:45.434 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026707371000000001, 2024-03-19T11:31:45.434 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:45.434 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:45.434 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:45.434 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:45.434 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:45.434 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3399 2024-03-19T11:31:45.434 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.434 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:45.434 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.435 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:45.435 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.034450+0000", 2024-03-19T11:31:45.435 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:45.435 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.435 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.435 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:45.435 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.034450+0000", 2024-03-19T11:31:45.435 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:45.435 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.435 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.435 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:45.435 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.034452+0000", 2024-03-19T11:31:45.435 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.877e-06 2024-03-19T11:31:45.435 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.436 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.436 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:45.436 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.034786+0000", 2024-03-19T11:31:45.436 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00033416099999999997 2024-03-19T11:31:45.436 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.436 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.436 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:45.436 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.034813+0000", 2024-03-19T11:31:45.436 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6792e-05 2024-03-19T11:31:45.436 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.436 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.436 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:45.436 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.034828+0000", 2024-03-19T11:31:45.436 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5285999999999999e-05 2024-03-19T11:31:45.436 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.437 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.437 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:45.437 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.034947+0000", 2024-03-19T11:31:45.437 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011857 2024-03-19T11:31:45.437 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.437 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.437 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:45.437 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.035325+0000", 2024-03-19T11:31:45.437 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037801499999999999 2024-03-19T11:31:45.437 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.437 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.437 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1", 2024-03-19T11:31:45.437 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.035660+0000", 2024-03-19T11:31:45.437 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00033508499999999999 2024-03-19T11:31:45.437 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.438 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.438 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:45.438 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.043413+0000", 2024-03-19T11:31:45.438 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077528110000000001 2024-03-19T11:31:45.438 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.438 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.438 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:45.438 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.060284+0000", 2024-03-19T11:31:45.438 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016870823 2024-03-19T11:31:45.438 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.438 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.438 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:45.438 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.061003+0000", 2024-03-19T11:31:45.438 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000719687 2024-03-19T11:31:45.439 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.439 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.439 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:45.439 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.061158+0000", 2024-03-19T11:31:45.439 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015426400000000001 2024-03-19T11:31:45.439 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:45.439 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:45.439 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:45.439 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.439 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.439 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3400 4.21 4:866f3dcc:::smithi03829856-31:head [write 1666233~695410] snapc 23=[23,22,21,1f,1e,1d,1c,1b,1a,18,17] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-03-19T11:31:45.439 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:33.043656+0000", 2024-03-19T11:31:45.439 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 12.33269456, 2024-03-19T11:31:45.439 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018155691000000002, 2024-03-19T11:31:45.439 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:45.440 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:45.440 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:45.440 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:45.440 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:45.440 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3400 2024-03-19T11:31:45.440 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.440 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:45.440 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.440 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:45.440 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.043656+0000", 2024-03-19T11:31:45.440 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:45.440 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.440 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.440 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:45.440 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.043656+0000", 2024-03-19T11:31:45.441 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:45.441 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.441 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.441 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:45.441 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.043658+0000", 2024-03-19T11:31:45.441 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5239999999999999e-06 2024-03-19T11:31:45.441 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.441 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.441 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:45.441 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044163+0000", 2024-03-19T11:31:45.441 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00050451399999999996 2024-03-19T11:31:45.441 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.441 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.441 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:45.441 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044205+0000", 2024-03-19T11:31:45.442 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2128000000000001e-05 2024-03-19T11:31:45.442 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.442 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.442 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:45.442 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044229+0000", 2024-03-19T11:31:45.442 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4097999999999999e-05 2024-03-19T11:31:45.442 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.442 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.442 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:45.442 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044414+0000", 2024-03-19T11:31:45.442 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00018484600000000001 2024-03-19T11:31:45.442 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.442 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.442 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:45.443 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044628+0000", 2024-03-19T11:31:45.443 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00021459499999999999 2024-03-19T11:31:45.443 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.443 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.443 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1", 2024-03-19T11:31:45.443 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044909+0000", 2024-03-19T11:31:45.443 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00028030399999999999 2024-03-19T11:31:45.443 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.443 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.443 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:45.443 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.061219+0000", 2024-03-19T11:31:45.443 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016310495000000001 2024-03-19T11:31:45.443 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.443 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.443 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:45.444 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.061362+0000", 2024-03-19T11:31:45.444 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014293500000000001 2024-03-19T11:31:45.444 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.444 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.444 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:45.444 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.061671+0000", 2024-03-19T11:31:45.444 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030884599999999999 2024-03-19T11:31:45.444 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.444 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.444 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:45.444 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.061811+0000", 2024-03-19T11:31:45.444 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014040600000000001 2024-03-19T11:31:45.444 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:45.444 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:45.444 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:45.445 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.445 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.445 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3401 4.21 4:866f3dcc:::smithi03829856-31:head [setxattr _header (57) in=7b,truncate 2504276] snapc 23=[23,22,21,1f,1e,1d,1c,1b,1a,18,17] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-03-19T11:31:45.445 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:33.044315+0000", 2024-03-19T11:31:45.445 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 12.332035614, 2024-03-19T11:31:45.445 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.021591882, 2024-03-19T11:31:45.445 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:45.445 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "waiting for sub ops", 2024-03-19T11:31:45.445 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:45.445 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:45.445 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:45.445 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3401 2024-03-19T11:31:45.445 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.446 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:45.446 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.446 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:45.446 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044315+0000", 2024-03-19T11:31:45.446 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:45.446 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.446 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.446 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:45.446 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044315+0000", 2024-03-19T11:31:45.446 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:45.446 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.446 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.446 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:45.446 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044317+0000", 2024-03-19T11:31:45.447 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.097e-06 2024-03-19T11:31:45.447 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.447 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.447 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:45.447 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044328+0000", 2024-03-19T11:31:45.447 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1362000000000001e-05 2024-03-19T11:31:45.447 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.447 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.447 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:45.447 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044354+0000", 2024-03-19T11:31:45.447 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6268999999999999e-05 2024-03-19T11:31:45.447 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.447 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.447 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:45.447 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044371+0000", 2024-03-19T11:31:45.448 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7028999999999999e-05 2024-03-19T11:31:45.448 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.448 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.448 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:45.448 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.052493+0000", 2024-03-19T11:31:45.448 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0081214270000000005 2024-03-19T11:31:45.448 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.448 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.448 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:45.448 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.052874+0000", 2024-03-19T11:31:45.448 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000381665 2024-03-19T11:31:45.448 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.448 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.448 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1", 2024-03-19T11:31:45.449 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.053343+0000", 2024-03-19T11:31:45.449 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00046883 2024-03-19T11:31:45.449 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.449 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.449 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:45.449 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.061993+0000", 2024-03-19T11:31:45.449 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0086501479999999999 2024-03-19T11:31:45.449 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.449 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.449 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:45.449 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.065906+0000", 2024-03-19T11:31:45.449 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0039130550000000004 2024-03-19T11:31:45.449 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:45.449 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:45.449 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:45.450 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.450 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.450 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3402 4.21 4:866f3dcc:::smithi03829856-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e112)", 2024-03-19T11:31:45.450 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:33.044411+0000", 2024-03-19T11:31:45.450 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 12.331938973, 2024-03-19T11:31:45.450 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.034955695000000002, 2024-03-19T11:31:45.450 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:45.450 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-03-19T11:31:45.450 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:45.450 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:45.450 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:45.451 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3402 2024-03-19T11:31:45.451 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.451 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:45.451 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.451 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:45.451 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044411+0000", 2024-03-19T11:31:45.451 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:45.451 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.451 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.451 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:45.451 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044411+0000", 2024-03-19T11:31:45.451 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:45.451 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.451 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.452 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:45.452 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044413+0000", 2024-03-19T11:31:45.452 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.517e-06 2024-03-19T11:31:45.452 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.452 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.452 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:45.452 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044420+0000", 2024-03-19T11:31:45.452 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3799999999999996e-06 2024-03-19T11:31:45.452 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.452 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.452 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:45.452 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044466+0000", 2024-03-19T11:31:45.452 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5713000000000001e-05 2024-03-19T11:31:45.452 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.453 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.453 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:45.453 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044484+0000", 2024-03-19T11:31:45.453 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8128999999999999e-05 2024-03-19T11:31:45.453 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.453 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.453 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:45.453 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.055330+0000", 2024-03-19T11:31:45.453 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01084574 2024-03-19T11:31:45.453 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.453 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.453 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-03-19T11:31:45.453 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.055450+0000", 2024-03-19T11:31:45.453 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000120104 2024-03-19T11:31:45.454 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.454 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.454 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:45.454 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.079347+0000", 2024-03-19T11:31:45.454 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.023897049 2024-03-19T11:31:45.454 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.454 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.454 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:45.454 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.079367+0000", 2024-03-19T11:31:45.454 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0063000000000001e-05 2024-03-19T11:31:45.454 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:45.454 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:45.454 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:45.454 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.454 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.455 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3403 4.19 4:9b248e68:::smithi03829856-50:head [sparse-read 0~1696039,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e112)", 2024-03-19T11:31:45.455 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:33.044524+0000", 2024-03-19T11:31:45.455 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 12.331825989, 2024-03-19T11:31:45.455 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016698992999999999, 2024-03-19T11:31:45.455 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:45.455 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:45.455 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:45.455 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:45.455 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:45.455 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3403 2024-03-19T11:31:45.455 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.455 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:45.455 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.455 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:45.456 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044524+0000", 2024-03-19T11:31:45.456 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:45.456 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.456 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.456 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:45.456 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044524+0000", 2024-03-19T11:31:45.456 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:45.456 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.456 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.456 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:45.456 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044526+0000", 2024-03-19T11:31:45.456 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4470000000000001e-06 2024-03-19T11:31:45.456 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.456 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.457 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:45.457 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044535+0000", 2024-03-19T11:31:45.457 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7450000000000007e-06 2024-03-19T11:31:45.457 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.457 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.457 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:45.457 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044562+0000", 2024-03-19T11:31:45.457 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6548999999999999e-05 2024-03-19T11:31:45.457 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.457 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.457 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:45.457 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044581+0000", 2024-03-19T11:31:45.457 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9468999999999999e-05 2024-03-19T11:31:45.457 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.457 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.458 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:45.458 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.055503+0000", 2024-03-19T11:31:45.458 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010921461 2024-03-19T11:31:45.458 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.458 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.458 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:45.458 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.055616+0000", 2024-03-19T11:31:45.458 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011270899999999999 2024-03-19T11:31:45.458 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.458 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.458 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:45.458 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.061223+0000", 2024-03-19T11:31:45.458 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005607613 2024-03-19T11:31:45.458 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:45.458 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:45.459 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:45.459 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.459 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.459 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3404 4.19 4:9b248e68:::smithi03829856-50:head [sparse-read 0~1696039] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e112)", 2024-03-19T11:31:45.459 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:33.044604+0000", 2024-03-19T11:31:45.459 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 12.331745667, 2024-03-19T11:31:45.459 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017390282, 2024-03-19T11:31:45.459 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:45.459 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:45.459 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:45.459 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:45.459 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:45.459 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3404 2024-03-19T11:31:45.459 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.460 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:45.460 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.460 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:45.460 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044604+0000", 2024-03-19T11:31:45.460 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:45.460 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.460 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.460 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:45.460 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044604+0000", 2024-03-19T11:31:45.460 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:45.460 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.460 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.460 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:45.460 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044606+0000", 2024-03-19T11:31:45.460 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.393e-06 2024-03-19T11:31:45.461 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.461 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.461 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:45.461 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044616+0000", 2024-03-19T11:31:45.461 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0322999999999999e-05 2024-03-19T11:31:45.461 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.461 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.461 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:45.461 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044636+0000", 2024-03-19T11:31:45.461 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9819000000000001e-05 2024-03-19T11:31:45.461 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.461 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.462 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:45.462 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044653+0000", 2024-03-19T11:31:45.462 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7232999999999999e-05 2024-03-19T11:31:45.462 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.462 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.462 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:45.462 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.061515+0000", 2024-03-19T11:31:45.462 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016861322000000002 2024-03-19T11:31:45.462 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.462 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.462 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:45.462 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.061645+0000", 2024-03-19T11:31:45.463 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013014999999999999 2024-03-19T11:31:45.463 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.463 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.463 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:45.463 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.061995+0000", 2024-03-19T11:31:45.463 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00035004200000000001 2024-03-19T11:31:45.463 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:45.463 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:45.463 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:45.463 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.463 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.463 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3405 4.19 4:9b248e68:::smithi03829856-50:head [sparse-read 0~1696039] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e112)", 2024-03-19T11:31:45.463 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:33.044707+0000", 2024-03-19T11:31:45.463 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 12.331643239, 2024-03-19T11:31:45.464 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018552987999999999, 2024-03-19T11:31:45.464 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:45.464 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:45.464 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:45.464 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:45.464 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:45.464 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3405 2024-03-19T11:31:45.464 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.464 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:45.464 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.464 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:45.464 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044707+0000", 2024-03-19T11:31:45.464 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:45.464 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.464 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.465 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:45.465 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044707+0000", 2024-03-19T11:31:45.465 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:45.465 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.465 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.465 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:45.465 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044708+0000", 2024-03-19T11:31:45.465 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.356e-06 2024-03-19T11:31:45.465 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.465 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.465 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:45.465 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044715+0000", 2024-03-19T11:31:45.465 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6900000000000003e-06 2024-03-19T11:31:45.465 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.465 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.466 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:45.466 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044736+0000", 2024-03-19T11:31:45.466 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0948999999999999e-05 2024-03-19T11:31:45.466 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.466 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.466 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:45.466 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.044751+0000", 2024-03-19T11:31:45.466 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5389000000000001e-05 2024-03-19T11:31:45.466 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.466 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.466 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:45.466 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.062070+0000", 2024-03-19T11:31:45.466 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017318291 2024-03-19T11:31:45.466 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.466 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.467 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:45.467 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.062309+0000", 2024-03-19T11:31:45.467 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000239573 2024-03-19T11:31:45.467 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:45.467 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:45.467 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:45.467 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:33.063260+0000", 2024-03-19T11:31:45.467 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00095074000000000005 2024-03-19T11:31:45.467 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:45.467 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:45.467 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:45.467 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:45.467 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:45.467 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:45.469 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-19T11:31:45.469 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:45.477+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:45.531 INFO:tasks.rados.rados.0.smithi038.stdout:1646: finishing write tid 3 to smithi03829856-22 2024-03-19T11:31:45.531 INFO:tasks.rados.rados.0.smithi038.stdout:1646: finishing write tid 4 to smithi03829856-22 2024-03-19T11:31:45.531 INFO:tasks.rados.rados.0.smithi038.stdout:1646: finishing write tid 5 to smithi03829856-22 2024-03-19T11:31:45.531 INFO:tasks.rados.rados.0.smithi038.stdout:1646: oid 22 updating version 0 to 40 2024-03-19T11:31:45.531 INFO:tasks.rados.rados.0.smithi038.stdout:1646: oid 22 updating version 40 to 41 2024-03-19T11:31:45.531 INFO:tasks.rados.rados.0.smithi038.stdout:1646: oid 22 version 41 is already newer than 39 2024-03-19T11:31:45.531 INFO:tasks.rados.rados.0.smithi038.stdout:1646: oid 22 version 41 is already newer than 38 2024-03-19T11:31:45.531 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 22 v 41 (ObjNum 295 snap 39 seq_num 295) dirty exists 2024-03-19T11:31:45.531 INFO:tasks.rados.rados.0.smithi038.stdout:1646: left oid 22 (ObjNum 295 snap 39 seq_num 295) 2024-03-19T11:31:45.531 INFO:tasks.rados.rados.0.smithi038.stdout:1649: expect (ObjNum 245 snap 30 seq_num 245) 2024-03-19T11:31:45.570 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:45.577+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:45.602 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:45.602 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:31:45.602 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:31:45.602 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:45.615 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-19T11:31:45.625 INFO:tasks.rados.rados.0.smithi038.stdout:1650: finishing write tid 1 to smithi03829856-38 2024-03-19T11:31:45.625 INFO:tasks.rados.rados.0.smithi038.stdout:1650: finishing write tid 2 to smithi03829856-38 2024-03-19T11:31:45.625 INFO:tasks.rados.rados.0.smithi038.stdout:1650: finishing write tid 3 to smithi03829856-38 2024-03-19T11:31:45.625 INFO:tasks.rados.rados.0.smithi038.stdout:1650: finishing write tid 4 to smithi03829856-38 2024-03-19T11:31:45.626 INFO:tasks.rados.rados.0.smithi038.stdout:1650: oid 38 updating version 0 to 179 2024-03-19T11:31:45.626 INFO:tasks.rados.rados.0.smithi038.stdout:1650: oid 38 version 179 is already newer than 178 2024-03-19T11:31:45.626 INFO:tasks.rados.rados.0.smithi038.stdout:1650: oid 38 version 179 is already newer than 177 2024-03-19T11:31:45.626 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 38 v 179 (ObjNum 297 snap 39 seq_num 297) dirty exists 2024-03-19T11:31:45.626 INFO:tasks.rados.rados.0.smithi038.stdout:1650: left oid 38 (ObjNum 297 snap 39 seq_num 297) 2024-03-19T11:31:45.626 INFO:tasks.rados.rados.0.smithi038.stdout:1651: expect (ObjNum 228 snap 26 seq_num 228) 2024-03-19T11:31:45.671 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:45.680+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:45.772 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:45.784+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:45.781 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:45.782 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:31:45.782 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:31:45.782 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:31:45.782 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:45.796 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-19T11:31:45.834 INFO:tasks.rados.rados.0.smithi038.stdout:1652: seq_num 298 ranges {608486=561969,1871634=634092,2950640=57846} 2024-03-19T11:31:45.854 INFO:tasks.rados.rados.0.smithi038.stdout:1652: writing smithi03829856-49 from 608486 to 1170455 tid 1 2024-03-19T11:31:45.873 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:45.884+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:45.875 INFO:tasks.rados.rados.0.smithi038.stdout:1652: writing smithi03829856-49 from 1871634 to 2505726 tid 2 2024-03-19T11:31:45.884 INFO:tasks.rados.rados.0.smithi038.stdout:1652: writing smithi03829856-49 from 2950640 to 3008486 tid 3 2024-03-19T11:31:45.884 INFO:tasks.rados.rados.0.smithi038.stdout:1646: done (5 left) 2024-03-19T11:31:45.884 INFO:tasks.rados.rados.0.smithi038.stdout:1647: done (4 left) 2024-03-19T11:31:45.884 INFO:tasks.rados.rados.0.smithi038.stdout:1649: done (3 left) 2024-03-19T11:31:45.884 INFO:tasks.rados.rados.0.smithi038.stdout:1650: done (2 left) 2024-03-19T11:31:45.884 INFO:tasks.rados.rados.0.smithi038.stdout:1651: done (1 left) 2024-03-19T11:31:45.884 INFO:tasks.rados.rados.0.smithi038.stdout:1653: read oid 16 snap 33 2024-03-19T11:31:45.885 INFO:tasks.rados.rados.0.smithi038.stdout:1653: expect (ObjNum 65 snap 3 seq_num 65) 2024-03-19T11:31:45.885 INFO:tasks.rados.rados.0.smithi038.stdout:1654: write (excl) oid 20 current snap is 39 2024-03-19T11:31:45.885 INFO:tasks.rados.rados.0.smithi038.stdout:1652: finishing write tid 1 to smithi03829856-49 2024-03-19T11:31:45.885 INFO:tasks.rados.rados.0.smithi038.stdout:1652: finishing write tid 2 to smithi03829856-49 2024-03-19T11:31:45.885 INFO:tasks.rados.rados.0.smithi038.stdout:1654: seq_num 299 ranges {625293=637218,1813763=494103,2978063=47230} 2024-03-19T11:31:45.903 INFO:tasks.rados.rados.0.smithi038.stdout:1654: writing smithi03829856-20 from 625293 to 1262511 tid 1 2024-03-19T11:31:45.918 INFO:tasks.rados.rados.0.smithi038.stdout:1654: writing smithi03829856-20 from 1813763 to 2307866 tid 2 2024-03-19T11:31:45.924 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:45.924 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:31:45.924 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:31:45.924 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:31:45.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.924 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:851 4.9 4:953e720b:::smithi03829856-45:head [write 591613~454162,stat] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:31:45.924 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.763514+0000", 2024-03-19T11:31:45.924 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 97.166331323999998, 2024-03-19T11:31:45.924 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027825888999999999, 2024-03-19T11:31:45.924 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:45.925 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:45.925 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:45.925 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:45.925 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:45.925 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 851 2024-03-19T11:31:45.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.925 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:45.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.925 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:45.925 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763514+0000", 2024-03-19T11:31:45.925 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:45.926 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.926 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.926 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:45.926 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763514+0000", 2024-03-19T11:31:45.926 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:45.926 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.926 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.926 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:45.926 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763517+0000", 2024-03-19T11:31:45.926 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8499999999999998e-06 2024-03-19T11:31:45.926 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.926 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.926 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:45.926 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.764714+0000", 2024-03-19T11:31:45.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011971270000000001 2024-03-19T11:31:45.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.927 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:45.927 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.764745+0000", 2024-03-19T11:31:45.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0546999999999997e-05 2024-03-19T11:31:45.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.927 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:45.927 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.764767+0000", 2024-03-19T11:31:45.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2286999999999999e-05 2024-03-19T11:31:45.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.927 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:45.927 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.773924+0000", 2024-03-19T11:31:45.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0091567360000000004 2024-03-19T11:31:45.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.928 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:45.928 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.774217+0000", 2024-03-19T11:31:45.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029350099999999999 2024-03-19T11:31:45.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.928 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-19T11:31:45.928 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.774770+0000", 2024-03-19T11:31:45.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00055288600000000005 2024-03-19T11:31:45.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.928 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:45.928 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.786413+0000", 2024-03-19T11:31:45.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011643229999999999 2024-03-19T11:31:45.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:45.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791104+0000", 2024-03-19T11:31:45.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0046900830000000003 2024-03-19T11:31:45.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:45.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791248+0000", 2024-03-19T11:31:45.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014410200000000001 2024-03-19T11:31:45.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:45.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791340+0000", 2024-03-19T11:31:45.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2540000000000005e-05 2024-03-19T11:31:45.930 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:45.930 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:45.930 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:45.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.930 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1162 4.1 4:8401c1f0:::smithi03829856-43:head [write 594981~609197] snapc 8=[8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-03-19T11:31:45.930 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.838981+0000", 2024-03-19T11:31:45.930 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 89.090864832999998, 2024-03-19T11:31:45.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027704349999999999, 2024-03-19T11:31:45.930 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:45.930 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:45.930 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:45.931 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:45.931 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:45.931 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1162 2024-03-19T11:31:45.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.931 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:45.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.931 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:45.931 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.838981+0000", 2024-03-19T11:31:45.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:45.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.932 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.932 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:45.932 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.838981+0000", 2024-03-19T11:31:45.932 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:45.932 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.932 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.932 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:45.932 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.838983+0000", 2024-03-19T11:31:45.932 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5399999999999998e-06 2024-03-19T11:31:45.932 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.932 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.932 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:45.932 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841322+0000", 2024-03-19T11:31:45.932 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002338592 2024-03-19T11:31:45.933 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.933 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:45.933 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841371+0000", 2024-03-19T11:31:45.933 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9123999999999999e-05 2024-03-19T11:31:45.933 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.933 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:45.933 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841399+0000", 2024-03-19T11:31:45.933 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7718000000000001e-05 2024-03-19T11:31:45.933 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.933 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:45.934 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841577+0000", 2024-03-19T11:31:45.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000178097 2024-03-19T11:31:45.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.934 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:45.934 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841825+0000", 2024-03-19T11:31:45.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00024824299999999999 2024-03-19T11:31:45.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.934 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-19T11:31:45.934 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842227+0000", 2024-03-19T11:31:45.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00040212199999999998 2024-03-19T11:31:45.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.934 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:45.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.865182+0000", 2024-03-19T11:31:45.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022954810999999999 2024-03-19T11:31:45.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:45.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.866360+0000", 2024-03-19T11:31:45.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001178449 2024-03-19T11:31:45.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:45.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.866538+0000", 2024-03-19T11:31:45.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017791000000000001 2024-03-19T11:31:45.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.936 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:45.936 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.866685+0000", 2024-03-19T11:31:45.936 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000146744 2024-03-19T11:31:45.936 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:45.936 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:45.936 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:45.936 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.936 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.936 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1271 4.9 4:953e720b:::smithi03829856-45:2 [read 795136~1008128] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-03-19T11:31:45.936 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:21.868056+0000", 2024-03-19T11:31:45.936 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 84.061789328000003, 2024-03-19T11:31:45.936 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.047270591000000001, 2024-03-19T11:31:45.936 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:45.936 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:45.936 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:45.937 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:45.937 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:45.937 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1271 2024-03-19T11:31:45.937 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.937 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:45.937 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.937 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:45.937 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868056+0000", 2024-03-19T11:31:45.937 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:45.938 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.938 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.938 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:45.938 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868056+0000", 2024-03-19T11:31:45.938 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:45.938 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.938 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.938 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:45.938 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868059+0000", 2024-03-19T11:31:45.938 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0469999999999998e-06 2024-03-19T11:31:45.938 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.938 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.938 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:45.938 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868072+0000", 2024-03-19T11:31:45.939 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2700999999999999e-05 2024-03-19T11:31:45.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.939 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:45.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868106+0000", 2024-03-19T11:31:45.939 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4152000000000001e-05 2024-03-19T11:31:45.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.939 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:45.940 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868126+0000", 2024-03-19T11:31:45.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9386999999999999e-05 2024-03-19T11:31:45.940 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.940 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.940 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:45.940 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.911447+0000", 2024-03-19T11:31:45.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.043321164000000002 2024-03-19T11:31:45.940 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.940 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.940 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:45.940 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.911762+0000", 2024-03-19T11:31:45.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00031492400000000001 2024-03-19T11:31:45.940 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.940 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.940 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:45.941 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.915327+0000", 2024-03-19T11:31:45.941 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003565216 2024-03-19T11:31:45.941 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:45.941 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:45.941 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:45.941 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.941 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.941 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(3.1 84/83 [PushOp(3:9205291d:::smithi03829856-50 low_tier:head, version: 28'36, data_included: [393216~851968,1703936~196608], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:9205291d:::smithi03829856-50 low_tier:head@28'36, size: 2823543, copy_subset: [(0, 2823543)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1900544, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:baab691a:::smithi03829856-2 low_tier:head, version: 28'37, data_included: [393216~524288,1507328~509479], data_size: 1033767, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:baab691a:::smithi03829856-2 low_tier:head@28'37, size: 2016807, copy_subset: [(0, 2016807)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2016807, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:98ce8426:::smithi03829856-4 low_tier:head, version: 28'38, data_included: [393216~720896,1638400~327680], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:98ce8426:::smithi03829856-4 low_tier:head@28'38, size: 2050421, copy_subset: [(0, 2050421)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1966080, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:a86be07b:::smithi03829856-44 low_tier:head, version: 28'41, data_included: [720896~655360,1769472~393216], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:a86be07b:::smithi03829856-44 low_tier:head@28'41, size: 2722701, copy_subset: [(0, 2722701)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2162688, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:bb56919e:::smithi03829856-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'44, data_included: [393216~786432,1703936~262144], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:bb56919e:::smithi03829856-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'44, size: 2806736, copy_subset: [(0, 2806736)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1966080, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:b1f822cf:::smithi03829856-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'46, data_included: [720896~786432,1966080~262144], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:b1f822cf:::smithi03829856-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'46, size: 2369754, copy_subset: [(0, 2369754)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2228224, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:a5e8313a:::smithi03829856-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'47, data_included: [524288~720896,1769472~327680], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:a5e8313a:::smithi03829856-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'47, size: 2571438, copy_subset: [(0, 2571438)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:bca5ec64:::e3a26bf9230b1b95b8cf1a4cb01909236bf2bf3fd59b083b2291d278ab98264d:head, version: 28'48, data_included: [0~393216], data_size: 393216, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bca5ec64:::e3a26bf9230b1b95b8cf1a4cb01909236bf2bf3fd59b083b2291d278ab98264d:head@28'48, size: 533504, copy_subset: [(0, 533504)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 533504, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:99a49d1b:::895a63407b6109b3cca9a6f5ef827134d56679bef42daadc5892269682c428ed:head, version: 32'49, data_included: [393216~388096], data_size: 388096, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:99a49d1b:::895a63407b6109b3cca9a6f5ef827134d56679bef42daadc5892269682c428ed:head@32'49, size: 781312, copy_subset: [(0, 781312)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 781312, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:83064589:::a3639ef4c215b429a637f1406db2398074b33464b996fe20e20d8b66c6bd720b:head, version: 33'51, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:83064589:::a3639ef4c215b429a637f1406db2398074b33464b996fe20e20d8b66c6bd720b:head@33'51, size: 1606656, copy_subset: [(0, 1606656)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:45.941 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:03.092965+0000", 2024-03-19T11:31:45.941 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 42.836880284999999, 2024-03-19T11:31:45.941 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.061194046000000002, 2024-03-19T11:31:45.941 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:45.941 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:45.942 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:45.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.942 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:45.942 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.092965+0000", 2024-03-19T11:31:45.942 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:45.942 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.942 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:45.942 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.092965+0000", 2024-03-19T11:31:45.942 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:45.942 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.942 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:45.942 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.092968+0000", 2024-03-19T11:31:45.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8720000000000002e-06 2024-03-19T11:31:45.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:45.943 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.106869+0000", 2024-03-19T11:31:45.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013900554000000001 2024-03-19T11:31:45.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:45.943 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.107077+0000", 2024-03-19T11:31:45.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000208626 2024-03-19T11:31:45.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:45.943 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.107372+0000", 2024-03-19T11:31:45.944 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029439199999999998 2024-03-19T11:31:45.944 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.944 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:45.944 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.108053+0000", 2024-03-19T11:31:45.944 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00068115899999999997 2024-03-19T11:31:45.944 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.944 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:45.944 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.108165+0000", 2024-03-19T11:31:45.944 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011162000000000001 2024-03-19T11:31:45.944 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.944 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:45.945 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.154159+0000", 2024-03-19T11:31:45.945 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.045994822999999997 2024-03-19T11:31:45.945 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:45.945 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:45.945 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:45.945 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.945 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.5 84/83 [PushOp(4:a0a50d04:::smithi03829856-13:b, version: 56'65, data_included: [601684~619341,1669078~429235], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a0a50d04:::smithi03829856-13:b@56'65, size: 2201684, copy_subset: [(0, 2201684)], clone_subset: {}, snapset: 17=[]:{b: [b, 6], 12: [12, d], 14: [13], 17: [17, 16, 15]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2098313, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:a0a50d04:::smithi03829856-13:14, version: 74'84, data_included: [601684~754801,1669078~293775], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a0a50d04:::smithi03829856-13:14@74'84, size: 2722734, copy_subset: [(0, 2722734)], clone_subset: {}, snapset: 17=[]:{b: [b, 6], 12: [12, d], 14: [13], 17: [17, 16, 15]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1962853, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:a0a50d04:::smithi03829856-13:12, version: 81'90, data_included: [601684~619341,1669078~429235], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a0a50d04:::smithi03829856-13:12@81'90, size: 2201684, copy_subset: [(0, 2201684)], clone_subset: {}, snapset: 17=[]:{b: [b, 6], 12: [12, d], 14: [13], 17: [17, 16, 15]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2098313, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:a38b9453:::smithi03829856-37:13, version: 81'92, data_included: [605052~774376,1923587~274200], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a38b9453:::smithi03829856-37:13@81'92, size: 2605052, copy_subset: [(0, 2605052)], clone_subset: {}, snapset: 14=[]:{13: [13, 12, d, b, 6]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2197787, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:a0a50d04:::smithi03829856-13:17, version: 82'94, data_included: [601684~754801,1669078~293775], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a0a50d04:::smithi03829856-13:17@82'94, size: 3025260, copy_subset: [(0, 3025260)], clone_subset: {}, snapset: 17=[]:{b: [b, 6], 12: [12, d], 14: [13], 17: [17, 16, 15]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1962853, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:a0a50d04:::smithi03829856-13:head, version: 82'97, data_included: [601684~619341,1669078~429235], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:a0a50d04:::smithi03829856-13:head@82'97, size: 2201684, copy_subset: [(0, 2201684)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2098313, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:45.945 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:03.103661+0000", 2024-03-19T11:31:45.945 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 42.826184263999998, 2024-03-19T11:31:45.945 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027776637, 2024-03-19T11:31:45.945 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:45.945 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:45.946 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:45.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.946 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:45.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.103661+0000", 2024-03-19T11:31:45.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:45.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.946 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:45.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.103661+0000", 2024-03-19T11:31:45.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:45.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.946 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:45.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.103664+0000", 2024-03-19T11:31:45.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.48e-06 2024-03-19T11:31:45.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:45.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.109797+0000", 2024-03-19T11:31:45.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061333380000000003 2024-03-19T11:31:45.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:45.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.109866+0000", 2024-03-19T11:31:45.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8474999999999997e-05 2024-03-19T11:31:45.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:45.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.109957+0000", 2024-03-19T11:31:45.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1669000000000006e-05 2024-03-19T11:31:45.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.948 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:45.948 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110162+0000", 2024-03-19T11:31:45.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000204417 2024-03-19T11:31:45.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.948 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:45.948 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110186+0000", 2024-03-19T11:31:45.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4729e-05 2024-03-19T11:31:45.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.948 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:45.949 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.131438+0000", 2024-03-19T11:31:45.949 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021251529000000002 2024-03-19T11:31:45.949 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:45.949 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:45.949 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:45.949 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.949 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.5 84/83 [PushOp(4:a38b9453:::smithi03829856-37:head, version: 81'93, data_included: [1775443~781480,2724284~166520], data_size: 948000, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:a38b9453:::smithi03829856-37:head@81'93, size: 2958032, copy_subset: [(0, 2958032)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2958032, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1775443, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:45.949 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:03.110032+0000", 2024-03-19T11:31:45.949 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 42.819814035999997, 2024-03-19T11:31:45.949 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027756439000000001, 2024-03-19T11:31:45.949 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:45.949 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:45.949 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:45.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.949 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:45.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110032+0000", 2024-03-19T11:31:45.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:45.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:45.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110032+0000", 2024-03-19T11:31:45.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:45.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:45.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110033+0000", 2024-03-19T11:31:45.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1510000000000001e-06 2024-03-19T11:31:45.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:45.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110353+0000", 2024-03-19T11:31:45.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00031981199999999999 2024-03-19T11:31:45.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:45.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110375+0000", 2024-03-19T11:31:45.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2772000000000002e-05 2024-03-19T11:31:45.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:45.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110396+0000", 2024-03-19T11:31:45.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1066000000000001e-05 2024-03-19T11:31:45.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.952 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:45.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.131543+0000", 2024-03-19T11:31:45.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021146166000000001 2024-03-19T11:31:45.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.952 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:45.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.131563+0000", 2024-03-19T11:31:45.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0750999999999998e-05 2024-03-19T11:31:45.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.952 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:45.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.137788+0000", 2024-03-19T11:31:45.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062247209999999999 2024-03-19T11:31:45.952 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:45.952 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:45.953 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:45.953 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.953 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.953 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.18 85/84 [PushOp(4:1d58db11:::smithi03829856-18:6, version: 55'80, data_included: [759683~639358,1904434~409218], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:6@55'80, size: 3159683, copy_subset: [(0, 3159683)], clone_subset: {}, snapset: 16=[]:{6: [6], b: [b], 15: [15, 13, 12, d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2313652, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1d58db11:::smithi03829856-18:b, version: 58'84, data_included: [443718~955323,1564138~93253], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:b@58'84, size: 3445402, copy_subset: [(0, 3445402)], clone_subset: {}, snapset: 16=[]:{6: [6], b: [b], 15: [15, 13, 12, d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1657391, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:10, version: 66'95, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:10@66'95, size: 3764735, copy_subset: [(0, 3764735)], clone_subset: {}, snapset: 15=[]:{c: [b, 6], 10: [d], 12: [12], 15: [15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:c, version: 70'102, data_included: [416807~576167,1542129~472409], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:c@70'102, size: 2857157, copy_subset: [(0, 2857157)], clone_subset: {}, snapset: 15=[]:{c: [b, 6], 10: [d], 12: [12], 15: [15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2014538, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:15, version: 77'110, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:15@77'110, size: 2621892, copy_subset: [(0, 2621892)], clone_subset: {}, snapset: 15=[]:{c: [b, 6], 10: [d], 12: [12], 15: [15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:45.953 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:04.083107+0000", 2024-03-19T11:31:45.953 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 41.846738469999998, 2024-03-19T11:31:45.953 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.042109168000000002, 2024-03-19T11:31:45.953 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:45.953 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:45.953 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:45.953 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.953 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:45.953 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.083107+0000", 2024-03-19T11:31:45.954 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:45.954 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.954 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:45.954 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.083107+0000", 2024-03-19T11:31:45.954 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:45.954 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.954 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:45.954 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.083109+0000", 2024-03-19T11:31:45.954 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7060000000000001e-06 2024-03-19T11:31:45.954 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.954 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:45.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092477+0000", 2024-03-19T11:31:45.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0093678589999999992 2024-03-19T11:31:45.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:45.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092549+0000", 2024-03-19T11:31:45.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2313999999999999e-05 2024-03-19T11:31:45.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:45.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092640+0000", 2024-03-19T11:31:45.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1262999999999997e-05 2024-03-19T11:31:45.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:45.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.093076+0000", 2024-03-19T11:31:45.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00043617800000000001 2024-03-19T11:31:45.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.956 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:45.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.093145+0000", 2024-03-19T11:31:45.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8651000000000004e-05 2024-03-19T11:31:45.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.956 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:45.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.125216+0000", 2024-03-19T11:31:45.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032071197000000003 2024-03-19T11:31:45.956 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:45.956 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:45.957 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:45.957 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.957 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.957 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.18 e85,84 4:1d58db11:::smithi03829856-18:7,38'48,4:1d58db11:::smithi03829856-18:9,46'64,4:1d58db11:::smithi03829856-18:2,48'76,4:1d58db11:::smithi03829856-18:c,70'104)", 2024-03-19T11:31:45.957 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:04.092733+0000", 2024-03-19T11:31:45.957 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 41.837112109000003, 2024-03-19T11:31:45.957 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033436452999999998, 2024-03-19T11:31:45.957 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:45.957 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:45.957 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:45.957 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.957 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:45.957 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092733+0000", 2024-03-19T11:31:45.957 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:45.957 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.958 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.958 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:45.958 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092733+0000", 2024-03-19T11:31:45.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:45.958 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.958 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.958 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:45.958 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092734+0000", 2024-03-19T11:31:45.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0699999999999996e-07 2024-03-19T11:31:45.958 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.958 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.958 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:45.958 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092745+0000", 2024-03-19T11:31:45.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0393e-05 2024-03-19T11:31:45.959 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.959 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.959 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:45.959 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092762+0000", 2024-03-19T11:31:45.959 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7649e-05 2024-03-19T11:31:45.959 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.959 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.959 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:45.959 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092778+0000", 2024-03-19T11:31:45.959 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5625999999999999e-05 2024-03-19T11:31:45.959 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.959 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.959 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:45.959 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.125342+0000", 2024-03-19T11:31:45.960 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032564348 2024-03-19T11:31:45.960 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.960 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.960 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:45.960 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.125375+0000", 2024-03-19T11:31:45.960 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2441000000000003e-05 2024-03-19T11:31:45.960 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.960 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.960 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:45.960 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.126170+0000", 2024-03-19T11:31:45.960 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00079508900000000002 2024-03-19T11:31:45.960 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:45.960 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:45.960 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:45.960 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.961 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.961 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(3.3 85/84 [PushOp(3:fb6cb292:::smithi03829856-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'21, data_included: [655360~851968,1900544~196608], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:fb6cb292:::smithi03829856-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'21, size: 2302526, copy_subset: [(0, 2302526)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:d281c5a5:::smithi03829856-30 low_tier:head, version: 28'42, data_included: [458752~589824,1507328~458752], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:d281c5a5:::smithi03829856-30 low_tier:head@28'42, size: 2487403, copy_subset: [(0, 2487403)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1966080, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:ed957b32:::smithi03829856-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'44, data_included: [524288~786432,1900544~233912], data_size: 1020344, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:ed957b32:::smithi03829856-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'44, size: 2134456, copy_subset: [(0, 2134456)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2134456, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:f23a45c7:::smithi03829856-36 low_tier:head, version: 28'45, data_included: [524288~786432,1835008~262144], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:f23a45c7:::smithi03829856-36 low_tier:head@28'45, size: 2588245, copy_subset: [(0, 2588245)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:edb38de0:::smithi03829856-40 low_tier:head, version: 28'46, data_included: [655360~720896,2097152~327680], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:edb38de0:::smithi03829856-40 low_tier:head@28'46, size: 2655473, copy_subset: [(0, 2655473)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2424832, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:f92b3f40:::b1fb925e39ab40886d36b70ff1c8c4f867ef6cb2ef150fbe90de57d057dfec36:head, version: 29'47, data_included: [262144~720896,1507328~327680], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:f92b3f40:::b1fb925e39ab40886d36b70ff1c8c4f867ef6cb2ef150fbe90de57d057dfec36:head@29'47, size: 2098176, copy_subset: [(0, 2098176)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1835008, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:fc3d03e1:::626f73f06d73fccc02b43398389ff991d56d81381f5e170c36fb9439a43cbbc0:head, version: 32'48, data_included: [0~131072], data_size: 131072, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:fc3d03e1:::626f73f06d73fccc02b43398389ff991d56d81381f5e170c36fb9439a43cbbc0:head@32'48, size: 462848, copy_subset: [(0, 462848)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 462848, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:f8934067:::8892b7b523af2cde6be0d91d0468f8be86a651cb052f82daf3e2c6278805eb6e:head, version: 32'49, data_included: [0~131072], data_size: 131072, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:f8934067:::8892b7b523af2cde6be0d91d0468f8be86a651cb052f82daf3e2c6278805eb6e:head@32'49, size: 202240, copy_subset: [(0, 202240)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 202240, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:45.961 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:04.108513+0000", 2024-03-19T11:31:45.961 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 41.821332667999997, 2024-03-19T11:31:45.961 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.036668899999999997, 2024-03-19T11:31:45.961 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:45.961 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:45.961 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:45.961 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.961 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:45.961 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.108513+0000", 2024-03-19T11:31:45.961 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:45.961 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.962 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.962 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:45.962 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.108513+0000", 2024-03-19T11:31:45.962 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:45.962 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.962 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.962 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:45.962 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.108515+0000", 2024-03-19T11:31:45.962 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7579999999999999e-06 2024-03-19T11:31:45.962 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.962 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.962 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:45.962 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115352+0000", 2024-03-19T11:31:45.962 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0068369950000000002 2024-03-19T11:31:45.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.963 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:45.963 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115421+0000", 2024-03-19T11:31:45.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9282999999999994e-05 2024-03-19T11:31:45.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.963 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:45.963 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115505+0000", 2024-03-19T11:31:45.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3714999999999994e-05 2024-03-19T11:31:45.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.963 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:45.963 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115948+0000", 2024-03-19T11:31:45.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000443036 2024-03-19T11:31:45.964 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.964 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.964 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:45.964 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116012+0000", 2024-03-19T11:31:45.964 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3905999999999994e-05 2024-03-19T11:31:45.964 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.964 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.964 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:45.964 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.145182+0000", 2024-03-19T11:31:45.964 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.029170207 2024-03-19T11:31:45.964 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:45.964 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:45.964 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:45.964 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.965 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.965 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.4.13:82 3.3 e85/84 3:f6b6d21a:::smithi03829856-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 85'103, mlcod=0'0)", 2024-03-19T11:31:45.965 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:04.115589+0000", 2024-03-19T11:31:45.965 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 41.814256362000002, 2024-03-19T11:31:45.965 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035569142999999998, 2024-03-19T11:31:45.965 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:45.965 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:45.965 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:45.965 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.965 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:45.965 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115589+0000", 2024-03-19T11:31:45.965 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:45.965 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.965 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.965 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:45.966 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115589+0000", 2024-03-19T11:31:45.966 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:45.966 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.966 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:45.966 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115591+0000", 2024-03-19T11:31:45.966 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2780000000000001e-06 2024-03-19T11:31:45.966 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.966 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:45.966 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115602+0000", 2024-03-19T11:31:45.966 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1797000000000001e-05 2024-03-19T11:31:45.966 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.967 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:45.967 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115613+0000", 2024-03-19T11:31:45.967 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1025e-05 2024-03-19T11:31:45.967 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.967 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.967 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:45.967 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115619+0000", 2024-03-19T11:31:45.967 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.976e-06 2024-03-19T11:31:45.967 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.967 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.967 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:45.967 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.145267+0000", 2024-03-19T11:31:45.967 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.02964791 2024-03-19T11:31:45.967 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.968 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.968 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:45.968 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.145304+0000", 2024-03-19T11:31:45.968 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6455000000000002e-05 2024-03-19T11:31:45.968 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.968 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.968 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:45.968 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.151114+0000", 2024-03-19T11:31:45.968 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0058103139999999996 2024-03-19T11:31:45.968 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.968 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.968 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:45.968 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.151158+0000", 2024-03-19T11:31:45.968 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4388000000000003e-05 2024-03-19T11:31:45.969 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:45.969 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:45.969 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:45.969 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.969 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.969 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.4.13:84 3.3 e85/84 3:ce6951d3:::smithi03829856-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 85'104, mlcod=0'0)", 2024-03-19T11:31:45.969 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:04.115824+0000", 2024-03-19T11:31:45.969 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 41.814021359000002, 2024-03-19T11:31:45.969 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035373398, 2024-03-19T11:31:45.969 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:45.969 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:45.969 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:45.969 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.969 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:45.970 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115824+0000", 2024-03-19T11:31:45.970 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:45.970 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.970 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.970 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:45.970 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115824+0000", 2024-03-19T11:31:45.970 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:45.970 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.970 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.970 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:45.970 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115825+0000", 2024-03-19T11:31:45.970 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3399999999999999e-07 2024-03-19T11:31:45.970 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.970 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.971 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:45.971 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115861+0000", 2024-03-19T11:31:45.971 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6303000000000001e-05 2024-03-19T11:31:45.971 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.971 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.971 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:45.971 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115869+0000", 2024-03-19T11:31:45.971 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0350000000000001e-06 2024-03-19T11:31:45.971 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.971 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.971 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:45.971 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115874+0000", 2024-03-19T11:31:45.971 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1599999999999997e-06 2024-03-19T11:31:45.972 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.972 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.972 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:45.972 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.145686+0000", 2024-03-19T11:31:45.972 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.029811252 2024-03-19T11:31:45.972 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.972 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.972 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:45.972 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.145706+0000", 2024-03-19T11:31:45.972 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0264000000000001e-05 2024-03-19T11:31:45.972 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.972 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.972 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:45.972 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.151161+0000", 2024-03-19T11:31:45.973 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054555280000000003 2024-03-19T11:31:45.973 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.973 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.973 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:45.973 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.151198+0000", 2024-03-19T11:31:45.973 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6322000000000002e-05 2024-03-19T11:31:45.973 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:45.973 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:45.973 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:45.973 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.973 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.973 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.4.13:85 3.3 e85/84 3:ce6951d3:::smithi03829856-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 85'105, mlcod=0'0)", 2024-03-19T11:31:45.973 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:04.115887+0000", 2024-03-19T11:31:45.973 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 41.813958745000001, 2024-03-19T11:31:45.973 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035346673000000002, 2024-03-19T11:31:45.974 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:45.974 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:45.974 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:45.974 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.974 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:45.974 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115887+0000", 2024-03-19T11:31:45.974 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:45.974 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.974 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.974 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:45.974 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115887+0000", 2024-03-19T11:31:45.974 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:45.974 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.974 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.975 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:45.975 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115887+0000", 2024-03-19T11:31:45.975 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2300000000000002e-07 2024-03-19T11:31:45.975 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.975 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.975 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:45.975 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115900+0000", 2024-03-19T11:31:45.975 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3044000000000001e-05 2024-03-19T11:31:45.975 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.975 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.975 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:45.975 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115906+0000", 2024-03-19T11:31:45.975 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5929999999999996e-06 2024-03-19T11:31:45.975 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.975 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.975 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:45.975 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115910+0000", 2024-03-19T11:31:45.976 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9269999999999998e-06 2024-03-19T11:31:45.976 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.976 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.976 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:45.976 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.145919+0000", 2024-03-19T11:31:45.976 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030009650999999998 2024-03-19T11:31:45.976 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.976 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.976 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:45.976 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.145948+0000", 2024-03-19T11:31:45.976 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9047000000000002e-05 2024-03-19T11:31:45.976 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.976 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.976 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:45.976 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.151200+0000", 2024-03-19T11:31:45.976 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052512970000000003 2024-03-19T11:31:45.977 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.977 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.977 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:45.977 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.151234+0000", 2024-03-19T11:31:45.977 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3791000000000002e-05 2024-03-19T11:31:45.977 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:45.977 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:45.977 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:45.977 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.977 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.977 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(3.3 85/84 [PushOp(3:fcdd1002:::06a0957e01d2e04d1b09d0c04f111819905dcf8372f2c6bee5014c22d5b575ef:head, version: 34'50, data_included: [0~458752], data_size: 458752, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:fcdd1002:::06a0957e01d2e04d1b09d0c04f111819905dcf8372f2c6bee5014c22d5b575ef:head@34'50, size: 689152, copy_subset: [(0, 689152)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 689152, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:45.977 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:04.115917+0000", 2024-03-19T11:31:45.977 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 41.813928314000002, 2024-03-19T11:31:45.977 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033796169000000001, 2024-03-19T11:31:45.978 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:45.978 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:45.978 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:45.978 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.978 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:45.978 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115917+0000", 2024-03-19T11:31:45.978 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:45.978 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.978 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.978 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:45.978 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115917+0000", 2024-03-19T11:31:45.978 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:45.978 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.979 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.979 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:45.979 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115918+0000", 2024-03-19T11:31:45.979 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8000000000000002e-07 2024-03-19T11:31:45.979 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.979 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.979 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:45.979 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116163+0000", 2024-03-19T11:31:45.979 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00024516400000000003 2024-03-19T11:31:45.979 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.979 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.979 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:45.979 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116181+0000", 2024-03-19T11:31:45.979 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8372999999999998e-05 2024-03-19T11:31:45.979 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.980 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.980 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:45.980 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116196+0000", 2024-03-19T11:31:45.980 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5041e-05 2024-03-19T11:31:45.980 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.980 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.980 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:45.980 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.147578+0000", 2024-03-19T11:31:45.980 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031382284000000003 2024-03-19T11:31:45.980 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.980 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.980 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:45.980 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.147594+0000", 2024-03-19T11:31:45.980 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5153e-05 2024-03-19T11:31:45.981 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.981 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.981 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:45.981 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.149713+0000", 2024-03-19T11:31:45.981 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021198739999999999 2024-03-19T11:31:45.981 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:45.981 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:45.981 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:45.981 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.981 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.981 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.4.13:83 3.3 e85/84 3:f6b6d21a:::smithi03829856-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 85'106, mlcod=0'0)", 2024-03-19T11:31:45.981 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:04.116216+0000", 2024-03-19T11:31:45.981 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 41.813629255000002, 2024-03-19T11:31:45.981 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035049444999999999, 2024-03-19T11:31:45.982 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:45.982 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:45.982 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:45.982 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.982 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:45.982 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116216+0000", 2024-03-19T11:31:45.982 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:45.982 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.982 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.982 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:45.982 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116216+0000", 2024-03-19T11:31:45.982 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:45.982 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.982 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.982 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:45.983 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116217+0000", 2024-03-19T11:31:45.983 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.22e-07 2024-03-19T11:31:45.983 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.983 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.983 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:45.983 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116221+0000", 2024-03-19T11:31:45.983 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5349999999999998e-06 2024-03-19T11:31:45.983 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.983 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.983 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:45.983 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116237+0000", 2024-03-19T11:31:45.983 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5532e-05 2024-03-19T11:31:45.983 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.983 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.983 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:45.984 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116245+0000", 2024-03-19T11:31:45.984 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0450000000000001e-06 2024-03-19T11:31:45.984 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.984 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.984 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:45.984 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.146739+0000", 2024-03-19T11:31:45.984 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030494351999999999 2024-03-19T11:31:45.984 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.984 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.984 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:45.984 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.146776+0000", 2024-03-19T11:31:45.984 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6894999999999998e-05 2024-03-19T11:31:45.984 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.984 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.984 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:45.985 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.151237+0000", 2024-03-19T11:31:45.985 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044605160000000003 2024-03-19T11:31:45.985 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.985 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.985 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:45.985 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.151266+0000", 2024-03-19T11:31:45.985 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9247999999999998e-05 2024-03-19T11:31:45.985 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:45.985 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:45.985 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:45.985 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.986 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.986 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3194 4.d 4:b65e6658:::smithi03829856-9:head [read 0~3714347,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108)", 2024-03-19T11:31:45.986 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.202652+0000", 2024-03-19T11:31:45.986 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.727193666000002, 2024-03-19T11:31:45.986 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031511255000000002, 2024-03-19T11:31:45.986 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:45.986 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:45.986 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:45.986 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:45.986 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:45.986 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3194 2024-03-19T11:31:45.986 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.986 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:45.986 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.987 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:45.987 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202652+0000", 2024-03-19T11:31:45.987 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:45.987 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.987 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.987 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:45.987 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202652+0000", 2024-03-19T11:31:45.987 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:45.987 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.987 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.987 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:45.987 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202653+0000", 2024-03-19T11:31:45.987 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3960000000000001e-06 2024-03-19T11:31:45.987 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.987 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.988 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:45.988 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202661+0000", 2024-03-19T11:31:45.988 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0830000000000004e-06 2024-03-19T11:31:45.988 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.988 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.988 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:45.988 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202679+0000", 2024-03-19T11:31:45.988 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7972e-05 2024-03-19T11:31:45.988 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.988 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.988 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:45.988 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202693+0000", 2024-03-19T11:31:45.988 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4031e-05 2024-03-19T11:31:45.988 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.989 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.989 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:45.989 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.203299+0000", 2024-03-19T11:31:45.989 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00060564400000000002 2024-03-19T11:31:45.989 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.989 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.989 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:45.989 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.203528+0000", 2024-03-19T11:31:45.989 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000229049 2024-03-19T11:31:45.989 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.989 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.989 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:45.989 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.234163+0000", 2024-03-19T11:31:45.989 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030635079999999999 2024-03-19T11:31:45.989 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:45.990 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:45.990 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:45.990 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.990 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.990 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3195 4.d 4:b65e6658:::smithi03829856-9:head [read 0~3714347,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108)", 2024-03-19T11:31:45.990 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.202714+0000", 2024-03-19T11:31:45.990 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.727131634999999, 2024-03-19T11:31:45.990 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.028551382, 2024-03-19T11:31:45.990 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:45.990 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:45.990 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:45.990 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:45.990 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:45.990 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3195 2024-03-19T11:31:45.991 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.991 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:45.991 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.991 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:45.991 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202714+0000", 2024-03-19T11:31:45.991 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:45.991 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.991 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.991 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:45.991 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202714+0000", 2024-03-19T11:31:45.991 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:45.991 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.991 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.991 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:45.992 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202715+0000", 2024-03-19T11:31:45.992 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.203e-06 2024-03-19T11:31:45.992 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.992 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.992 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:45.992 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202722+0000", 2024-03-19T11:31:45.992 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1289999999999998e-06 2024-03-19T11:31:45.992 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.992 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.992 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:45.992 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202739+0000", 2024-03-19T11:31:45.992 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6929e-05 2024-03-19T11:31:45.992 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.993 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.993 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:45.993 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202754+0000", 2024-03-19T11:31:45.993 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4642e-05 2024-03-19T11:31:45.993 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.993 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.993 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:45.993 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.227218+0000", 2024-03-19T11:31:45.993 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.024464587999999999 2024-03-19T11:31:45.993 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.993 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.993 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:45.993 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.227471+0000", 2024-03-19T11:31:45.993 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000252747 2024-03-19T11:31:45.994 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.994 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.994 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:45.994 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.231265+0000", 2024-03-19T11:31:45.994 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037941440000000002 2024-03-19T11:31:45.994 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:45.994 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:45.994 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:45.994 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.994 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.994 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3196 4.d 4:b65e6658:::smithi03829856-9:head [sparse-read 0~2135368] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108)", 2024-03-19T11:31:45.994 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.202774+0000", 2024-03-19T11:31:45.994 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.727071894000002, 2024-03-19T11:31:45.995 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033044519000000001, 2024-03-19T11:31:45.995 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:45.995 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:45.995 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:45.995 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:45.995 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:45.995 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3196 2024-03-19T11:31:45.995 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.995 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:45.995 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.995 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:45.995 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202774+0000", 2024-03-19T11:31:45.995 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:45.995 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.996 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.996 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:45.996 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202774+0000", 2024-03-19T11:31:45.996 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:45.996 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.996 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.996 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:45.996 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202775+0000", 2024-03-19T11:31:45.996 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.423e-06 2024-03-19T11:31:45.996 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.996 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.996 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:45.996 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202782+0000", 2024-03-19T11:31:45.996 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3660000000000003e-06 2024-03-19T11:31:45.997 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.997 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.997 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:45.997 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202801+0000", 2024-03-19T11:31:45.997 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8182e-05 2024-03-19T11:31:45.997 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.997 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.997 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:45.997 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202815+0000", 2024-03-19T11:31:45.997 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4042e-05 2024-03-19T11:31:45.997 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.997 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.997 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:45.997 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.234294+0000", 2024-03-19T11:31:45.997 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031479493999999997 2024-03-19T11:31:45.998 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.998 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.998 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:45.998 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.234670+0000", 2024-03-19T11:31:45.998 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000375595 2024-03-19T11:31:45.998 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.998 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.998 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:45.998 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.235818+0000", 2024-03-19T11:31:45.998 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011484170000000001 2024-03-19T11:31:45.998 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:45.998 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:45.998 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:45.998 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:45.999 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.999 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(osd.0.13:71 3.3 3:e22267ea:::smithi03829856-38 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e108)", 2024-03-19T11:31:45.999 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.209268+0000", 2024-03-19T11:31:45.999 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.720577642999999, 2024-03-19T11:31:45.999 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.038534928000000003, 2024-03-19T11:31:45.999 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:45.999 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:45.999 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:45.999 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:45.999 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:45.999 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.209268+0000", 2024-03-19T11:31:45.999 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:45.999 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.000 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.000 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:46.000 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.209268+0000", 2024-03-19T11:31:46.000 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:46.000 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.000 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.000 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:46.000 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.209272+0000", 2024-03-19T11:31:46.000 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0500000000000002e-06 2024-03-19T11:31:46.000 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.000 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.000 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:46.000 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.209291+0000", 2024-03-19T11:31:46.000 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8734999999999999e-05 2024-03-19T11:31:46.001 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.001 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.001 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:46.001 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.209329+0000", 2024-03-19T11:31:46.001 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8223000000000003e-05 2024-03-19T11:31:46.001 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.001 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.001 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:46.001 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.209354+0000", 2024-03-19T11:31:46.001 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5114e-05 2024-03-19T11:31:46.001 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.001 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.001 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:46.001 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.209573+0000", 2024-03-19T11:31:46.001 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021842000000000001 2024-03-19T11:31:46.002 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.002 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.002 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:46.002 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.209814+0000", 2024-03-19T11:31:46.002 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00024179400000000001 2024-03-19T11:31:46.002 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.002 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.002 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:46.002 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.210284+0000", 2024-03-19T11:31:46.002 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00046991699999999998 2024-03-19T11:31:46.002 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.002 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.002 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:46.002 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.213128+0000", 2024-03-19T11:31:46.003 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002843609 2024-03-19T11:31:46.003 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.003 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.003 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:46.003 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.247417+0000", 2024-03-19T11:31:46.003 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.034289312000000002 2024-03-19T11:31:46.003 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.003 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.003 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:46.003 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.247692+0000", 2024-03-19T11:31:46.003 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027449000000000001 2024-03-19T11:31:46.003 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.003 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.003 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:46.004 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.247803+0000", 2024-03-19T11:31:46.004 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111264 2024-03-19T11:31:46.004 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:46.004 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:46.004 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:46.004 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.004 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.004 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3198 3.3 3:f117d76c:::3a52856062567f05d8c7df91dad2c51911138f6216e1ec709153bdff536e4abe:head [write 0~927232] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-19T11:31:46.004 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.213434+0000", 2024-03-19T11:31:46.004 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.716411952000001, 2024-03-19T11:31:46.004 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.039874882, 2024-03-19T11:31:46.004 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:46.004 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:46.004 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:46.004 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:46.005 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:46.005 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3198 2024-03-19T11:31:46.005 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.005 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:46.005 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.005 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:46.005 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.213434+0000", 2024-03-19T11:31:46.005 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:46.005 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.005 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.005 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:46.005 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.213434+0000", 2024-03-19T11:31:46.005 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:46.005 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.006 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.006 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:46.006 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.213436+0000", 2024-03-19T11:31:46.006 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4430000000000002e-06 2024-03-19T11:31:46.006 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.006 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.006 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:46.006 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.219501+0000", 2024-03-19T11:31:46.006 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0060649240000000002 2024-03-19T11:31:46.006 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.006 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.006 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:46.006 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.219531+0000", 2024-03-19T11:31:46.007 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0470000000000001e-05 2024-03-19T11:31:46.007 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.007 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.007 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:46.007 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.219552+0000", 2024-03-19T11:31:46.007 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1012000000000001e-05 2024-03-19T11:31:46.007 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.007 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.007 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:46.007 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.231849+0000", 2024-03-19T11:31:46.007 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012296105999999999 2024-03-19T11:31:46.007 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.007 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.007 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:46.008 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.232349+0000", 2024-03-19T11:31:46.008 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00050035100000000001 2024-03-19T11:31:46.008 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.008 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.008 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:46.008 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.233102+0000", 2024-03-19T11:31:46.008 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00075262300000000001 2024-03-19T11:31:46.008 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.008 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.008 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:46.008 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.249722+0000", 2024-03-19T11:31:46.008 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016620901 2024-03-19T11:31:46.008 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.008 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.009 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:46.009 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.253185+0000", 2024-03-19T11:31:46.009 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034625340000000002 2024-03-19T11:31:46.009 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.009 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.009 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:46.009 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.253252+0000", 2024-03-19T11:31:46.009 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6538000000000006e-05 2024-03-19T11:31:46.009 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.009 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.009 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:46.009 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.253309+0000", 2024-03-19T11:31:46.009 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.698e-05 2024-03-19T11:31:46.009 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:46.009 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:46.010 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:46.010 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.010 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.010 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3323 4.18 4:1d58db11:::smithi03829856-18:head [tier-promote] snapc 22=[22,21,1f,1e,1d,1c,1b,1a,18,17] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-19T11:31:46.010 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:31.226946+0000", 2024-03-19T11:31:46.010 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.702899656, 2024-03-19T11:31:46.010 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.037925183000000001, 2024-03-19T11:31:46.010 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:46.010 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:46.010 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:46.010 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:46.010 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:46.010 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3323 2024-03-19T11:31:46.010 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.011 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:46.011 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.011 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:46.011 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.226946+0000", 2024-03-19T11:31:46.011 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:46.011 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.011 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.011 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:46.011 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.226946+0000", 2024-03-19T11:31:46.011 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:46.011 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.011 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.011 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:46.011 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.226947+0000", 2024-03-19T11:31:46.012 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4670000000000001e-06 2024-03-19T11:31:46.012 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.012 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.012 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:46.012 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.226955+0000", 2024-03-19T11:31:46.012 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9130000000000003e-06 2024-03-19T11:31:46.012 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.012 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.012 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:46.012 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.226978+0000", 2024-03-19T11:31:46.012 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2229e-05 2024-03-19T11:31:46.012 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.012 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.012 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:46.012 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.226993+0000", 2024-03-19T11:31:46.013 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5420000000000001e-05 2024-03-19T11:31:46.013 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.013 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.013 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:46.013 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.227146+0000", 2024-03-19T11:31:46.013 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015327200000000001 2024-03-19T11:31:46.013 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.013 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.013 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:46.013 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.227406+0000", 2024-03-19T11:31:46.013 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025941799999999998 2024-03-19T11:31:46.013 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.013 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.013 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:46.014 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.264871+0000", 2024-03-19T11:31:46.014 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.037465463999999997 2024-03-19T11:31:46.014 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:46.014 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:46.014 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:46.014 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:46.014 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:46.014 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:46.016 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-19T11:31:46.017 INFO:tasks.rados.rados.0.smithi038.stdout:1654: writing smithi03829856-20 from 2978063 to 3025293 tid 3 2024-03-19T11:31:46.017 INFO:tasks.rados.rados.0.smithi038.stdout:1655: read oid 5 snap -1 2024-03-19T11:31:46.017 INFO:tasks.rados.rados.0.smithi038.stdout:1655: expect (ObjNum 277 snap 36 seq_num 277) 2024-03-19T11:31:46.017 INFO:tasks.rados.rados.0.smithi038.stdout:1656: rollback oid 32 current snap is 39 2024-03-19T11:31:46.017 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 32 to 21 2024-03-19T11:31:46.017 INFO:tasks.rados.rados.0.smithi038.stdout:1652: finishing write tid 3 to smithi03829856-49 2024-03-19T11:31:46.018 INFO:tasks.rados.rados.0.smithi038.stdout:1652: finishing write tid 4 to smithi03829856-49 2024-03-19T11:31:46.018 INFO:tasks.rados.rados.0.smithi038.stdout:1657: write (excl) oid 31 current snap is 39 2024-03-19T11:31:46.018 INFO:tasks.rados.rados.0.smithi038.stdout:1657: seq_num 300 ranges {642100=712467,1872245=637761} 2024-03-19T11:31:46.018 INFO:tasks.rados.rados.0.smithi038.stdout:1657: writing smithi03829856-31 from 642100 to 1354567 tid 1 2024-03-19T11:31:46.018 INFO:tasks.rados.rados.0.smithi038.stdout:1657: writing smithi03829856-31 from 1872245 to 2510006 tid 2 2024-03-19T11:31:46.018 INFO:tasks.rados.rados.0.smithi038.stdout:1658: read oid 45 snap -1 2024-03-19T11:31:46.018 INFO:tasks.rados.rados.0.smithi038.stdout:1658: expect (ObjNum 275 snap 34 seq_num 275) 2024-03-19T11:31:46.018 INFO:tasks.rados.rados.0.smithi038.stdout:1659: write oid 25 current snap is 39 2024-03-19T11:31:46.018 INFO:tasks.rados.rados.0.smithi038.stdout:1652: finishing write tid 5 to smithi03829856-49 2024-03-19T11:31:46.018 INFO:tasks.rados.rados.0.smithi038.stdout:1652: oid 49 updating version 0 to 59 2024-03-19T11:31:46.018 INFO:tasks.rados.rados.0.smithi038.stdout:1652: oid 49 updating version 59 to 60 2024-03-19T11:31:46.018 INFO:tasks.rados.rados.0.smithi038.stdout:1652: oid 49 updating version 60 to 62 2024-03-19T11:31:46.018 INFO:tasks.rados.rados.0.smithi038.stdout:1652: oid 49 version 62 is already newer than 61 2024-03-19T11:31:46.018 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 49 v 62 (ObjNum 298 snap 39 seq_num 298) dirty exists 2024-03-19T11:31:46.019 INFO:tasks.rados.rados.0.smithi038.stdout:1652: left oid 49 (ObjNum 298 snap 39 seq_num 298) 2024-03-19T11:31:46.019 INFO:tasks.rados.rados.0.smithi038.stdout:1653: expect (ObjNum 65 snap 3 seq_num 65) 2024-03-19T11:31:46.019 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:46.024+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:46.098 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-03-19T11:31:46.109 INFO:tasks.rados.rados.0.smithi038.stdout:1654: finishing write tid 1 to smithi03829856-20 2024-03-19T11:31:46.109 INFO:tasks.rados.rados.0.smithi038.stdout:1654: finishing write tid 2 to smithi03829856-20 2024-03-19T11:31:46.109 INFO:tasks.rados.rados.0.smithi038.stdout:1656: finishing rollback tid 0 to smithi03829856-32 2024-03-19T11:31:46.109 INFO:tasks.rados.rados.0.smithi038.stdout:1654: finishing write tid 3 to smithi03829856-20 2024-03-19T11:31:46.109 INFO:tasks.rados.rados.0.smithi038.stdout:1654: finishing write tid 4 to smithi03829856-20 2024-03-19T11:31:46.109 INFO:tasks.rados.rados.0.smithi038.stdout:1654: finishing write tid 5 to smithi03829856-20 2024-03-19T11:31:46.110 INFO:tasks.rados.rados.0.smithi038.stdout:1654: oid 20 updating version 0 to 283 2024-03-19T11:31:46.110 INFO:tasks.rados.rados.0.smithi038.stdout:1654: oid 20 updating version 283 to 285 2024-03-19T11:31:46.110 INFO:tasks.rados.rados.0.smithi038.stdout:1654: oid 20 updating version 285 to 286 2024-03-19T11:31:46.110 INFO:tasks.rados.rados.0.smithi038.stdout:1654: oid 20 version 286 is already newer than 284 2024-03-19T11:31:46.110 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 20 v 286 (ObjNum 299 snap 39 seq_num 299) dirty exists 2024-03-19T11:31:46.110 INFO:tasks.rados.rados.0.smithi038.stdout:1654: left oid 20 (ObjNum 299 snap 39 seq_num 299) 2024-03-19T11:31:46.110 INFO:tasks.rados.rados.0.smithi038.stdout:1655: expect (ObjNum 277 snap 36 seq_num 277) 2024-03-19T11:31:46.116 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:46.125+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:46.148 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:46.148 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:31:46.148 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:31:46.148 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:46.162 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-19T11:31:46.248 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:46.257+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:46.292 INFO:tasks.rados.rados.0.smithi038.stdout:1656: finishing rollback tid 1 to smithi03829856-32 2024-03-19T11:31:46.293 INFO:tasks.rados.rados.0.smithi038.stdout:1656: finishing rollback tid 2 to smithi03829856-32 2024-03-19T11:31:46.293 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 32 v 145 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-03-19T11:31:46.293 INFO:tasks.rados.rados.0.smithi038.stdout:1657: finishing write tid 1 to smithi03829856-31 2024-03-19T11:31:46.293 INFO:tasks.rados.rados.0.smithi038.stdout:1657: finishing write tid 2 to smithi03829856-31 2024-03-19T11:31:46.293 INFO:tasks.rados.rados.0.smithi038.stdout:1657: finishing write tid 3 to smithi03829856-31 2024-03-19T11:31:46.293 INFO:tasks.rados.rados.0.smithi038.stdout:1657: finishing write tid 4 to smithi03829856-31 2024-03-19T11:31:46.293 INFO:tasks.rados.rados.0.smithi038.stdout:1657: oid 31 updating version 0 to 175 2024-03-19T11:31:46.293 INFO:tasks.rados.rados.0.smithi038.stdout:1657: oid 31 updating version 175 to 176 2024-03-19T11:31:46.293 INFO:tasks.rados.rados.0.smithi038.stdout:1657: oid 31 updating version 176 to 177 2024-03-19T11:31:46.294 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 31 v 177 (ObjNum 300 snap 39 seq_num 300) dirty exists 2024-03-19T11:31:46.294 INFO:tasks.rados.rados.0.smithi038.stdout:1657: left oid 31 (ObjNum 300 snap 39 seq_num 300) 2024-03-19T11:31:46.294 INFO:tasks.rados.rados.0.smithi038.stdout:1658: expect (ObjNum 275 snap 34 seq_num 275) 2024-03-19T11:31:46.319 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:46.328+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:46.326 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:46.326 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:31:46.326 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:31:46.327 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:31:46.327 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:46.340 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-19T11:31:46.428 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:46.437+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:46.487 INFO:tasks.rados.rados.0.smithi038.stdout:1659: seq_num 301 ranges {658907=787716,1930727=497772,3032909=25998} 2024-03-19T11:31:46.507 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:46.507 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:31:46.507 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:31:46.507 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:31:46.507 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.507 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1158 4.8 4:15708482:::smithi03829856-23:head [tier-promote] snapc 8=[8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-03-19T11:31:46.508 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.827517+0000", 2024-03-19T11:31:46.508 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 89.687122751999993, 2024-03-19T11:31:46.508 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.034096983999999997, 2024-03-19T11:31:46.508 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:46.508 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:46.508 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:46.508 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:46.508 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:46.508 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1158 2024-03-19T11:31:46.508 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.508 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:46.508 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.508 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:46.508 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827517+0000", 2024-03-19T11:31:46.509 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:46.509 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.509 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.509 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:46.509 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827517+0000", 2024-03-19T11:31:46.509 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:46.509 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.509 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.509 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:46.509 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827520+0000", 2024-03-19T11:31:46.509 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.45e-06 2024-03-19T11:31:46.509 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.509 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.509 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:46.510 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827535+0000", 2024-03-19T11:31:46.510 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.509e-05 2024-03-19T11:31:46.510 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.510 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.510 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:46.510 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827579+0000", 2024-03-19T11:31:46.510 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4211000000000001e-05 2024-03-19T11:31:46.510 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.510 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.511 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:46.511 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827603+0000", 2024-03-19T11:31:46.511 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3586000000000001e-05 2024-03-19T11:31:46.511 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.511 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.511 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:46.511 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827789+0000", 2024-03-19T11:31:46.511 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018580199999999999 2024-03-19T11:31:46.511 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.511 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.511 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:46.512 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.828077+0000", 2024-03-19T11:31:46.512 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028846500000000001 2024-03-19T11:31:46.512 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.512 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.512 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:46.512 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.861614+0000", 2024-03-19T11:31:46.512 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033536379999999998 2024-03-19T11:31:46.512 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:46.512 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:46.512 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:46.512 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.512 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.512 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1162 4.1 e39/21 4:8401c1f0:::smithi03829856-43:head v 39'47, mlcod=38'43)", 2024-03-19T11:31:46.513 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.842598+0000", 2024-03-19T11:31:46.513 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 89.672040981999999, 2024-03-19T11:31:46.513 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0233756, 2024-03-19T11:31:46.513 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:46.513 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:46.513 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:46.513 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.513 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:46.513 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842598+0000", 2024-03-19T11:31:46.513 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:46.513 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.514 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.514 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:46.514 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842598+0000", 2024-03-19T11:31:46.514 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:46.515 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.515 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.515 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:46.515 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842599+0000", 2024-03-19T11:31:46.515 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6000000000000003e-07 2024-03-19T11:31:46.515 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.515 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.515 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:46.515 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.843566+0000", 2024-03-19T11:31:46.515 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00096688700000000004 2024-03-19T11:31:46.515 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.515 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.515 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:46.515 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.843601+0000", 2024-03-19T11:31:46.516 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.502e-05 2024-03-19T11:31:46.516 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.516 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.516 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:46.516 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.843621+0000", 2024-03-19T11:31:46.516 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0305e-05 2024-03-19T11:31:46.516 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.516 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.516 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:46.516 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.843882+0000", 2024-03-19T11:31:46.516 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026027100000000002 2024-03-19T11:31:46.516 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.516 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.516 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:46.516 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.844022+0000", 2024-03-19T11:31:46.517 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014020100000000001 2024-03-19T11:31:46.517 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.517 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.517 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:46.517 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.865862+0000", 2024-03-19T11:31:46.517 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021839725000000001 2024-03-19T11:31:46.517 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.517 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.517 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:46.517 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.865974+0000", 2024-03-19T11:31:46.517 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011243099999999999 2024-03-19T11:31:46.517 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:46.517 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:46.517 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:46.517 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.518 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.518 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1209 4.8 4:1d58db11:::smithi03829856-18:head [write 645402~551297,stat] snapc 9=[9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:31:46.518 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:18.121657+0000", 2024-03-19T11:31:46.518 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 88.392981980000002, 2024-03-19T11:31:46.518 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.025074349999999999, 2024-03-19T11:31:46.518 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:46.518 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:46.518 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:46.518 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:46.518 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:46.518 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1209 2024-03-19T11:31:46.518 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.518 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:46.518 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.519 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:46.519 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.121657+0000", 2024-03-19T11:31:46.519 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:46.519 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.519 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.519 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:46.519 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.121657+0000", 2024-03-19T11:31:46.519 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:46.519 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.519 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.519 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:46.519 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.121661+0000", 2024-03-19T11:31:46.519 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0489999999999997e-06 2024-03-19T11:31:46.519 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.519 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.520 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:46.520 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123747+0000", 2024-03-19T11:31:46.520 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0020857720000000001 2024-03-19T11:31:46.520 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.520 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.520 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:46.520 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123779+0000", 2024-03-19T11:31:46.520 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2258000000000002e-05 2024-03-19T11:31:46.520 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.520 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.520 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:46.520 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123795+0000", 2024-03-19T11:31:46.520 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5177000000000001e-05 2024-03-19T11:31:46.520 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.521 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.521 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:46.521 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123949+0000", 2024-03-19T11:31:46.521 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015403699999999999 2024-03-19T11:31:46.521 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.521 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.521 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:46.521 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.124134+0000", 2024-03-19T11:31:46.521 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000185484 2024-03-19T11:31:46.521 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.521 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.521 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:46.521 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.124554+0000", 2024-03-19T11:31:46.521 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00042019699999999998 2024-03-19T11:31:46.521 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.522 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.522 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:46.522 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146267+0000", 2024-03-19T11:31:46.522 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021712838000000002 2024-03-19T11:31:46.522 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.522 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.522 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:46.522 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146424+0000", 2024-03-19T11:31:46.522 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015647299999999999 2024-03-19T11:31:46.522 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.522 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.523 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:46.523 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146597+0000", 2024-03-19T11:31:46.523 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017303699999999999 2024-03-19T11:31:46.523 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.523 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.523 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:46.523 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146732+0000", 2024-03-19T11:31:46.523 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013502800000000001 2024-03-19T11:31:46.523 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:46.523 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:46.523 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:46.523 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.523 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.523 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1443 4.2 4:43b32242:::smithi03829856-22:head [read 0~2352947,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:31:46.523 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.979417+0000", 2024-03-19T11:31:46.524 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 80.535222306999998, 2024-03-19T11:31:46.524 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.96719988700000004, 2024-03-19T11:31:46.524 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:46.524 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:46.524 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:46.524 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:46.524 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:46.524 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1443 2024-03-19T11:31:46.524 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.524 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:46.524 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.524 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:46.524 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979417+0000", 2024-03-19T11:31:46.524 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:46.524 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.525 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.525 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:46.525 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979417+0000", 2024-03-19T11:31:46.525 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:46.525 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.525 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.525 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:46.525 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979420+0000", 2024-03-19T11:31:46.525 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.366e-06 2024-03-19T11:31:46.525 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.525 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.525 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:46.525 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979437+0000", 2024-03-19T11:31:46.525 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6192e-05 2024-03-19T11:31:46.525 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.526 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.526 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:46.526 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979474+0000", 2024-03-19T11:31:46.526 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7601000000000001e-05 2024-03-19T11:31:46.526 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.526 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.526 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:46.526 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979496+0000", 2024-03-19T11:31:46.526 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1506999999999999e-05 2024-03-19T11:31:46.526 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.526 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.526 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:46.526 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979712+0000", 2024-03-19T11:31:46.527 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021641299999999999 2024-03-19T11:31:46.527 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.527 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.527 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:31:46.527 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979746+0000", 2024-03-19T11:31:46.527 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3386000000000002e-05 2024-03-19T11:31:46.527 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.527 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.527 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:46.527 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.931552+0000", 2024-03-19T11:31:46.527 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.95180659899999998 2024-03-19T11:31:46.528 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.528 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.528 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:46.528 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.934681+0000", 2024-03-19T11:31:46.528 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0031292289999999999 2024-03-19T11:31:46.528 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.528 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.528 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:46.528 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.942701+0000", 2024-03-19T11:31:46.528 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0080198809999999995 2024-03-19T11:31:46.528 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.528 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.528 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:46.528 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.942839+0000", 2024-03-19T11:31:46.529 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013816999999999999 2024-03-19T11:31:46.529 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.529 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.529 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:46.529 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.946617+0000", 2024-03-19T11:31:46.529 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003777543 2024-03-19T11:31:46.529 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:46.529 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:46.529 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:46.529 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.529 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.530 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1444 4.2 4:43b32242:::smithi03829856-22:head [read 0~2352947,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:31:46.530 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.979553+0000", 2024-03-19T11:31:46.530 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 80.535086604, 2024-03-19T11:31:46.530 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.97015945400000003, 2024-03-19T11:31:46.530 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:46.530 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:46.530 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:46.531 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:46.531 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:46.531 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1444 2024-03-19T11:31:46.531 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.531 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:46.531 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.531 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:46.531 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979553+0000", 2024-03-19T11:31:46.531 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:46.531 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.531 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.531 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:46.531 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979553+0000", 2024-03-19T11:31:46.531 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:46.532 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.532 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.532 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:46.532 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979554+0000", 2024-03-19T11:31:46.532 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.403e-06 2024-03-19T11:31:46.532 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.532 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.532 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:46.532 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979564+0000", 2024-03-19T11:31:46.532 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0032e-05 2024-03-19T11:31:46.533 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.533 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.533 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:46.533 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979579+0000", 2024-03-19T11:31:46.533 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4875000000000001e-05 2024-03-19T11:31:46.533 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.533 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.533 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:46.533 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979595+0000", 2024-03-19T11:31:46.533 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6300999999999999e-05 2024-03-19T11:31:46.533 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.533 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.534 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:46.534 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979874+0000", 2024-03-19T11:31:46.534 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000279065 2024-03-19T11:31:46.534 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.534 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.534 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:31:46.534 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979904+0000", 2024-03-19T11:31:46.534 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.94e-05 2024-03-19T11:31:46.534 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.534 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.534 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:46.534 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.934781+0000", 2024-03-19T11:31:46.534 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.95487744900000004 2024-03-19T11:31:46.535 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.535 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.535 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:46.535 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.934946+0000", 2024-03-19T11:31:46.535 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016485999999999999 2024-03-19T11:31:46.535 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.535 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.535 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:46.535 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.946829+0000", 2024-03-19T11:31:46.535 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011882482999999999 2024-03-19T11:31:46.535 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.536 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.536 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:46.536 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.946989+0000", 2024-03-19T11:31:46.536 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000160421 2024-03-19T11:31:46.536 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.536 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.536 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:46.536 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.949712+0000", 2024-03-19T11:31:46.536 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002723165 2024-03-19T11:31:46.536 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:46.536 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:46.536 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:46.537 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.537 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.537 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1445 4.2 4:43b32242:::smithi03829856-22:head [read 0~2352947,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:31:46.537 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.979653+0000", 2024-03-19T11:31:46.537 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 80.534986473000004, 2024-03-19T11:31:46.537 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.97249507999999996, 2024-03-19T11:31:46.537 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:46.537 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:46.537 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:46.537 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:46.537 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:46.537 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1445 2024-03-19T11:31:46.538 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.538 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:46.538 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.538 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:46.538 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979653+0000", 2024-03-19T11:31:46.538 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:46.538 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.538 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.539 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:46.539 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979653+0000", 2024-03-19T11:31:46.539 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:46.539 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.539 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.539 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:46.539 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979654+0000", 2024-03-19T11:31:46.539 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.372e-06 2024-03-19T11:31:46.539 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.539 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.539 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:46.539 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979662+0000", 2024-03-19T11:31:46.540 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9959999999999993e-06 2024-03-19T11:31:46.540 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.540 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.540 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:46.540 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979698+0000", 2024-03-19T11:31:46.540 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5763000000000003e-05 2024-03-19T11:31:46.540 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.540 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.540 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:46.540 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979714+0000", 2024-03-19T11:31:46.540 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6283999999999999e-05 2024-03-19T11:31:46.540 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.540 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.540 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:46.540 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979990+0000", 2024-03-19T11:31:46.541 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000275829 2024-03-19T11:31:46.541 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.541 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.541 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:31:46.541 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.980018+0000", 2024-03-19T11:31:46.541 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8082999999999999e-05 2024-03-19T11:31:46.541 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.541 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.541 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:46.541 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.935620+0000", 2024-03-19T11:31:46.541 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.95560133800000002 2024-03-19T11:31:46.541 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.541 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.541 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:46.541 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.935795+0000", 2024-03-19T11:31:46.542 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000175656 2024-03-19T11:31:46.542 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.542 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.542 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:46.542 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.949835+0000", 2024-03-19T11:31:46.542 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014040083 2024-03-19T11:31:46.542 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.542 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.542 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:46.542 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.950002+0000", 2024-03-19T11:31:46.542 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016673 2024-03-19T11:31:46.542 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.542 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.542 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:46.542 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.952148+0000", 2024-03-19T11:31:46.543 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021459460000000001 2024-03-19T11:31:46.543 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:46.543 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:46.543 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:46.543 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.543 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.543 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2207 4.28 4:15708482:::smithi03829856-23:head [write 541225~644694,stat] snapc 13=[13,12,11,d,b,6] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-03-19T11:31:46.543 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:51.160660+0000", 2024-03-19T11:31:46.543 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 55.353979647000003, 2024-03-19T11:31:46.543 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.025400267000000001, 2024-03-19T11:31:46.544 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:46.544 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:46.544 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:46.544 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:46.544 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:46.544 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2207 2024-03-19T11:31:46.544 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.544 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:46.544 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.544 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:46.544 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.160660+0000", 2024-03-19T11:31:46.544 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:46.544 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.545 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.545 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:46.545 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.160660+0000", 2024-03-19T11:31:46.545 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:46.545 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.545 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.545 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:46.545 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.160688+0000", 2024-03-19T11:31:46.545 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8058000000000001e-05 2024-03-19T11:31:46.545 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.545 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.545 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:46.546 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.168614+0000", 2024-03-19T11:31:46.546 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079262810000000003 2024-03-19T11:31:46.546 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.546 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.546 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:46.546 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.168662+0000", 2024-03-19T11:31:46.546 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8327e-05 2024-03-19T11:31:46.546 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.546 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.546 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:46.546 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.168679+0000", 2024-03-19T11:31:46.546 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6305999999999999e-05 2024-03-19T11:31:46.546 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.547 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.547 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:46.547 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.168830+0000", 2024-03-19T11:31:46.547 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000151714 2024-03-19T11:31:46.547 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.547 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.547 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:46.547 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.169107+0000", 2024-03-19T11:31:46.547 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027608299999999999 2024-03-19T11:31:46.547 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.547 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.547 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:46.547 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.169605+0000", 2024-03-19T11:31:46.547 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00049874999999999995 2024-03-19T11:31:46.547 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.548 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.548 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:46.548 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.185769+0000", 2024-03-19T11:31:46.548 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016163397999999999 2024-03-19T11:31:46.548 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.548 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.548 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:46.548 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.185809+0000", 2024-03-19T11:31:46.548 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0598e-05 2024-03-19T11:31:46.548 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.548 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.548 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:46.548 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.185898+0000", 2024-03-19T11:31:46.548 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8270999999999996e-05 2024-03-19T11:31:46.548 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.549 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.549 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:46.549 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.186060+0000", 2024-03-19T11:31:46.549 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016248099999999999 2024-03-19T11:31:46.549 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:46.549 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:46.549 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:46.549 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.549 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.549 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.15 75/74 [PushOp(4:af9d0d56:::smithi03829856-20:head, version: 35'55, data_included: [655360~589824,1900544~418789], data_size: 1008613, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:af9d0d56:::smithi03829856-20:head@35'55, size: 2319333, copy_subset: [(0, 2319333)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2319333, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:aa3303a5:::smithi03829856-17:b, version: 58'79, data_included: [614400~692224,1903006~356352], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:aa3303a5:::smithi03829856-17:b@58'79, size: 2268912, copy_subset: [(0, 2268912)], clone_subset: {}, snapset: 14=[]:{b: [b, 6], 10: [d], 12: [12, 11], 14: [13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2259358, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ad3ad1b3:::smithi03829856-21:b, version: 58'81, data_included: [736140~537686,2020581~315559], data_size: 853245, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:ad3ad1b3:::smithi03829856-21:b@58'81, size: 2336140, copy_subset: [(0, 2336140)], clone_subset: {}, snapset: 10=[]:{b: [b, 6], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2336140, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:afa9f208:::smithi03829856-35:12, version: 67'105, data_included: [521050~698057,1806623~350519], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:afa9f208:::smithi03829856-35:12@67'105, size: 2521050, copy_subset: [(0, 2521050)], clone_subset: {}, snapset: 12=[]:{11: [11, d, b, 6], 12: [12]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2157142, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:aa3303a5:::smithi03829856-17:12, version: 69'113, data_included: [413472~893152,1808868~155424], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:aa3303a5:::smithi03829856-17:12@69'113, size: 2319366, copy_subset: [(0, 2319366)], clone_subset: {}, snapset: 14=[]:{b: [b, 6], 10: [d], 12: [12, 11], 14: [13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1964292, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ac9cd3a3:::smithi03829856-8:10, version: 70'117, data_included: [517649~620182,1776668~428394], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:ac9cd3a3:::smithi03829856-8:10@70'117, size: 3008420, copy_subset: [(0, 3008420)], clone_subset: {}, snapset: 10=[]:{10: [d, b, 6]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2205062, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ac9cd3a3:::smithi03829856-8:head, version: 70'118, data_included: [517649~620182,1776668~428394], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:ac9cd3a3:::smithi03829856-8:head@70'118, size: 3008420, copy_subset: [(0, 3008420)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2205062, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:aa3303a5:::smithi03829856-17:10, version: 70'119, data_included: [413472~893152,1818036~155424], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:aa3303a5:::smithi03829856-17:10@70'119, size: 3613472, copy_subset: [(0, 3613472)], clone_subset: {}, snapset: 14=[]:{b: [b, 6], 10: [d], 12: [12, 11], 14: [13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1973460, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ad3ad1b3:::smithi03829856-21:10, version: 72'122, data_included: [531121~742705,1711057~305871], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:ad3ad1b3:::smithi03829856-21:10@72'122, size: 3731121, copy_subset: [(0, 3731121)], clone_subset: {}, snapset: 10=[]:{b: [b, 6], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2016928, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:46.549 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.081994+0000", 2024-03-19T11:31:46.549 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 52.432644949999997, 2024-03-19T11:31:46.549 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.040292051000000002, 2024-03-19T11:31:46.549 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:46.550 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:46.550 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:46.550 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.550 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:46.550 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.081994+0000", 2024-03-19T11:31:46.550 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:46.550 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.550 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.550 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:46.550 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.081994+0000", 2024-03-19T11:31:46.550 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:46.550 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.550 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.550 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:46.551 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.081996+0000", 2024-03-19T11:31:46.551 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5069999999999999e-06 2024-03-19T11:31:46.551 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.551 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.551 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:46.551 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088193+0000", 2024-03-19T11:31:46.551 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061970209999999996 2024-03-19T11:31:46.551 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.551 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.551 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:46.551 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088252+0000", 2024-03-19T11:31:46.551 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8838999999999997e-05 2024-03-19T11:31:46.551 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.551 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.552 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:46.552 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088322+0000", 2024-03-19T11:31:46.552 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0455000000000002e-05 2024-03-19T11:31:46.552 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.552 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.552 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:46.552 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088660+0000", 2024-03-19T11:31:46.552 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033735299999999998 2024-03-19T11:31:46.552 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.552 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.552 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:46.552 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088727+0000", 2024-03-19T11:31:46.552 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7335999999999994e-05 2024-03-19T11:31:46.552 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.552 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.553 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:46.553 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.122286+0000", 2024-03-19T11:31:46.553 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033559539999999999 2024-03-19T11:31:46.553 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:46.553 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:46.553 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:46.553 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.553 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.553 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.15 e75,74 4:ac9cd3a3:::smithi03829856-8:4,49'71)", 2024-03-19T11:31:46.553 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.088380+0000", 2024-03-19T11:31:46.553 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 52.426259088000002, 2024-03-19T11:31:46.553 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.044894303000000003, 2024-03-19T11:31:46.553 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:46.553 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:46.553 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:46.554 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.554 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:46.554 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088380+0000", 2024-03-19T11:31:46.554 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:46.554 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.554 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.554 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:46.554 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088380+0000", 2024-03-19T11:31:46.554 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:46.554 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.554 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.554 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:46.554 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088381+0000", 2024-03-19T11:31:46.555 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0500000000000004e-07 2024-03-19T11:31:46.555 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.555 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.555 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:46.555 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088385+0000", 2024-03-19T11:31:46.555 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2370000000000003e-06 2024-03-19T11:31:46.555 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.555 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.555 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:46.555 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088392+0000", 2024-03-19T11:31:46.556 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4240000000000004e-06 2024-03-19T11:31:46.556 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.556 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.556 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:46.556 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088397+0000", 2024-03-19T11:31:46.556 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8829999999999997e-06 2024-03-19T11:31:46.556 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.556 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.556 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:46.556 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.122356+0000", 2024-03-19T11:31:46.556 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033958588999999997 2024-03-19T11:31:46.557 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.557 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.557 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:46.557 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.122364+0000", 2024-03-19T11:31:46.557 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0800000000000006e-06 2024-03-19T11:31:46.557 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.557 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.557 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:46.557 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.133275+0000", 2024-03-19T11:31:46.557 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010910585 2024-03-19T11:31:46.557 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:46.557 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:46.557 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:46.557 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.558 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.558 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:2286 4.15 e75/74 4:afa9f208:::smithi03829856-35:head v 75'131, mlcod=33'54)", 2024-03-19T11:31:46.558 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.089414+0000", 2024-03-19T11:31:46.558 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 52.425225847999997, 2024-03-19T11:31:46.558 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.045421256, 2024-03-19T11:31:46.558 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:46.558 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:46.558 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:46.558 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.558 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:46.558 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.089414+0000", 2024-03-19T11:31:46.558 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:46.558 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.558 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.559 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:46.559 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.089414+0000", 2024-03-19T11:31:46.559 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:46.559 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.559 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.559 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:46.560 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.089414+0000", 2024-03-19T11:31:46.560 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1100000000000005e-07 2024-03-19T11:31:46.560 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.560 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.560 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:46.560 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.089423+0000", 2024-03-19T11:31:46.560 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3799999999999994e-06 2024-03-19T11:31:46.560 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.560 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.560 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:46.560 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.089436+0000", 2024-03-19T11:31:46.560 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2931e-05 2024-03-19T11:31:46.560 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.560 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.561 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:46.561 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.089447+0000", 2024-03-19T11:31:46.561 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1044e-05 2024-03-19T11:31:46.561 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.561 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.561 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:46.561 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.133330+0000", 2024-03-19T11:31:46.561 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.043882833000000003 2024-03-19T11:31:46.561 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.561 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.561 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:46.561 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.133353+0000", 2024-03-19T11:31:46.561 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3629999999999999e-05 2024-03-19T11:31:46.561 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.561 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.562 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:46.562 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.134786+0000", 2024-03-19T11:31:46.562 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0014327980000000001 2024-03-19T11:31:46.562 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.562 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.562 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:46.562 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.134835+0000", 2024-03-19T11:31:46.562 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8829000000000003e-05 2024-03-19T11:31:46.562 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:46.562 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:46.562 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:46.562 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.562 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.562 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2802 4.28 4:15708482:::smithi03829856-23:head [write 497540~612664] snapc 19=[18,17,15,b] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-03-19T11:31:46.563 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:14.060002+0000", 2024-03-19T11:31:46.563 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.454637718999997, 2024-03-19T11:31:46.563 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.028374612, 2024-03-19T11:31:46.563 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:46.563 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:46.563 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:46.563 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:46.563 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:46.563 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2802 2024-03-19T11:31:46.563 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.563 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:46.563 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.563 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:46.563 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.060002+0000", 2024-03-19T11:31:46.563 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:46.564 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.564 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.564 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:46.564 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.060002+0000", 2024-03-19T11:31:46.564 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:46.564 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.564 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.564 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:46.564 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.060005+0000", 2024-03-19T11:31:46.564 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5130000000000001e-06 2024-03-19T11:31:46.564 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.564 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.564 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:46.564 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.061352+0000", 2024-03-19T11:31:46.564 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0013468060000000001 2024-03-19T11:31:46.565 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.565 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.565 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:46.565 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.061403+0000", 2024-03-19T11:31:46.565 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1181999999999999e-05 2024-03-19T11:31:46.565 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.565 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.565 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:46.565 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.061428+0000", 2024-03-19T11:31:46.565 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4715999999999998e-05 2024-03-19T11:31:46.565 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.565 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.565 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:46.565 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.061600+0000", 2024-03-19T11:31:46.566 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017224100000000001 2024-03-19T11:31:46.566 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.566 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.566 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op must wait for map", 2024-03-19T11:31:46.566 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.061619+0000", 2024-03-19T11:31:46.566 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9213000000000001e-05 2024-03-19T11:31:46.567 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.567 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.567 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:46.567 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.066953+0000", 2024-03-19T11:31:46.567 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053332149999999997 2024-03-19T11:31:46.567 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.567 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.567 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:46.567 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.067236+0000", 2024-03-19T11:31:46.567 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028381400000000002 2024-03-19T11:31:46.567 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.567 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.567 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:46.567 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.067698+0000", 2024-03-19T11:31:46.567 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00046152200000000002 2024-03-19T11:31:46.568 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.568 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.568 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:46.568 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.082795+0000", 2024-03-19T11:31:46.568 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015096729 2024-03-19T11:31:46.568 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.568 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.568 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:46.569 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.088033+0000", 2024-03-19T11:31:46.569 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052383259999999997 2024-03-19T11:31:46.569 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.569 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.569 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:46.569 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.088209+0000", 2024-03-19T11:31:46.569 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000175611 2024-03-19T11:31:46.569 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.569 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.569 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:46.569 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.088376+0000", 2024-03-19T11:31:46.570 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016772400000000001 2024-03-19T11:31:46.570 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:46.570 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:46.570 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:46.570 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.570 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.570 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2803 4.28 4:15708482:::smithi03829856-23:head [write 1840862~440060] snapc 19=[18,17,15,b] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-03-19T11:31:46.570 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:14.066423+0000", 2024-03-19T11:31:46.570 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.448216576, 2024-03-19T11:31:46.570 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022744533000000001, 2024-03-19T11:31:46.571 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:46.571 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:46.571 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:46.571 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:46.571 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:46.571 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2803 2024-03-19T11:31:46.571 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.571 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:46.571 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.571 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:46.571 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.066423+0000", 2024-03-19T11:31:46.571 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:46.571 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.571 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.572 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:46.572 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.066423+0000", 2024-03-19T11:31:46.572 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:46.572 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.572 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.572 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:46.572 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.066425+0000", 2024-03-19T11:31:46.572 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2500000000000001e-06 2024-03-19T11:31:46.572 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.572 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.573 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:46.573 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.067351+0000", 2024-03-19T11:31:46.573 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00092597000000000003 2024-03-19T11:31:46.573 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.573 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.573 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:46.573 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.067380+0000", 2024-03-19T11:31:46.573 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9326e-05 2024-03-19T11:31:46.573 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.573 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.573 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:46.573 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.067401+0000", 2024-03-19T11:31:46.573 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0458999999999998e-05 2024-03-19T11:31:46.574 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.574 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.574 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:46.574 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.078507+0000", 2024-03-19T11:31:46.574 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.01110651 2024-03-19T11:31:46.574 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.574 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.574 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:46.574 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.078694+0000", 2024-03-19T11:31:46.574 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000186334 2024-03-19T11:31:46.574 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.574 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.575 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:46.575 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.078997+0000", 2024-03-19T11:31:46.575 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00030362400000000001 2024-03-19T11:31:46.575 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.575 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.575 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:46.575 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.088954+0000", 2024-03-19T11:31:46.575 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0099565639999999993 2024-03-19T11:31:46.575 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.575 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.575 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:46.576 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.088978+0000", 2024-03-19T11:31:46.576 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4355999999999999e-05 2024-03-19T11:31:46.576 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.576 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.576 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:46.576 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.089067+0000", 2024-03-19T11:31:46.576 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8664000000000004e-05 2024-03-19T11:31:46.576 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.576 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.576 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:46.577 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.089167+0000", 2024-03-19T11:31:46.577 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100476 2024-03-19T11:31:46.577 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:46.577 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:46.577 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:46.577 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.577 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.577 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3102 4.15 4:afa9f208:::smithi03829856-35:head [write 484101~692450,stat] snapc 1f=[1f,1e,1d,1c,1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-19T11:31:46.577 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.553012+0000", 2024-03-19T11:31:46.578 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.961627417999999, 2024-03-19T11:31:46.578 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.024256340000000001, 2024-03-19T11:31:46.578 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:46.578 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:46.578 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:46.578 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:46.578 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:46.578 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3102 2024-03-19T11:31:46.578 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.578 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:46.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.579 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:46.579 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.553012+0000", 2024-03-19T11:31:46.579 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:46.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.579 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:46.579 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.553012+0000", 2024-03-19T11:31:46.579 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:46.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.579 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:46.579 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.553015+0000", 2024-03-19T11:31:46.579 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5200000000000002e-06 2024-03-19T11:31:46.580 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.580 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.580 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:46.580 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.554342+0000", 2024-03-19T11:31:46.580 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0013261760000000001 2024-03-19T11:31:46.580 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.580 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.580 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:46.580 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.554377+0000", 2024-03-19T11:31:46.580 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5299000000000002e-05 2024-03-19T11:31:46.580 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.580 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.580 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:46.580 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.554401+0000", 2024-03-19T11:31:46.581 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4360000000000001e-05 2024-03-19T11:31:46.581 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.581 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:46.581 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.554595+0000", 2024-03-19T11:31:46.581 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019390499999999999 2024-03-19T11:31:46.581 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.581 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:46.581 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.554801+0000", 2024-03-19T11:31:46.581 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020614799999999999 2024-03-19T11:31:46.581 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.581 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:46.581 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.555182+0000", 2024-03-19T11:31:46.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00038094700000000002 2024-03-19T11:31:46.582 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.582 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:46.582 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.569800+0000", 2024-03-19T11:31:46.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0146179 2024-03-19T11:31:46.582 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.582 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:46.582 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.576922+0000", 2024-03-19T11:31:46.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0071215339999999997 2024-03-19T11:31:46.582 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.582 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:46.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.577114+0000", 2024-03-19T11:31:46.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019218100000000001 2024-03-19T11:31:46.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.583 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:46.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.577268+0000", 2024-03-19T11:31:46.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015437 2024-03-19T11:31:46.583 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:46.583 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:46.583 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:46.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.583 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3103 4.15 4:afa9f208:::smithi03829856-35:head [write 1636889~447212] snapc 1f=[1f,1e,1d,1c,1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-19T11:31:46.583 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.557970+0000", 2024-03-19T11:31:46.584 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.956669031000001, 2024-03-19T11:31:46.584 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033363528000000003, 2024-03-19T11:31:46.584 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:46.584 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:46.584 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:46.584 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:46.584 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:46.584 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3103 2024-03-19T11:31:46.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.584 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:46.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.584 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:46.584 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.557970+0000", 2024-03-19T11:31:46.584 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:46.585 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.585 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:46.585 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.557970+0000", 2024-03-19T11:31:46.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:46.585 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.585 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:46.585 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.557973+0000", 2024-03-19T11:31:46.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6060000000000001e-06 2024-03-19T11:31:46.585 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.585 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:46.585 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558806+0000", 2024-03-19T11:31:46.586 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00083305499999999997 2024-03-19T11:31:46.586 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.586 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:46.586 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558843+0000", 2024-03-19T11:31:46.586 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6519000000000001e-05 2024-03-19T11:31:46.586 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.586 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:46.586 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558863+0000", 2024-03-19T11:31:46.586 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0622999999999999e-05 2024-03-19T11:31:46.586 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.586 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:46.586 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.562831+0000", 2024-03-19T11:31:46.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039681250000000003 2024-03-19T11:31:46.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.587 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:46.587 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.563003+0000", 2024-03-19T11:31:46.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000171638 2024-03-19T11:31:46.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.587 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:46.587 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.563335+0000", 2024-03-19T11:31:46.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033165899999999999 2024-03-19T11:31:46.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.587 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:46.588 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.569815+0000", 2024-03-19T11:31:46.588 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00648024 2024-03-19T11:31:46.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.588 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.588 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:46.588 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.590982+0000", 2024-03-19T11:31:46.588 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021167235 2024-03-19T11:31:46.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.588 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.588 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:46.588 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.591170+0000", 2024-03-19T11:31:46.588 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018796899999999999 2024-03-19T11:31:46.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.588 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.589 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:46.589 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.591334+0000", 2024-03-19T11:31:46.589 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000163859 2024-03-19T11:31:46.589 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:46.589 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:46.589 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:46.589 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.589 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3104 4.15 4:afa9f208:::smithi03829856-35:head [setxattr _header (57) in=7b,truncate 2084101] snapc 1f=[1f,1e,1d,1c,1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-19T11:31:46.589 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.558905+0000", 2024-03-19T11:31:46.589 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.955733948999999, 2024-03-19T11:31:46.589 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032860320999999998, 2024-03-19T11:31:46.589 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:46.589 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:46.590 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:46.590 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:46.590 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:46.590 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3104 2024-03-19T11:31:46.590 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.590 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:46.590 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.590 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:46.590 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558905+0000", 2024-03-19T11:31:46.590 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:46.590 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.590 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.590 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:46.590 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558905+0000", 2024-03-19T11:31:46.590 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:46.591 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.591 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:46.591 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558907+0000", 2024-03-19T11:31:46.591 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.547e-06 2024-03-19T11:31:46.591 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.591 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:46.591 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558916+0000", 2024-03-19T11:31:46.591 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0589999999999992e-06 2024-03-19T11:31:46.591 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.591 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:46.591 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558931+0000", 2024-03-19T11:31:46.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5262999999999999e-05 2024-03-19T11:31:46.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.592 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.592 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:46.592 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558947+0000", 2024-03-19T11:31:46.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6140000000000001e-05 2024-03-19T11:31:46.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.592 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.592 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:46.592 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.567458+0000", 2024-03-19T11:31:46.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085106650000000006 2024-03-19T11:31:46.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.592 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.592 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:46.592 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.567777+0000", 2024-03-19T11:31:46.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00031917399999999998 2024-03-19T11:31:46.593 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.593 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.593 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:46.593 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.568090+0000", 2024-03-19T11:31:46.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00031286900000000002 2024-03-19T11:31:46.593 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.593 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.593 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:46.593 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.576738+0000", 2024-03-19T11:31:46.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0086476379999999992 2024-03-19T11:31:46.593 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.593 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.593 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:46.593 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.591474+0000", 2024-03-19T11:31:46.594 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014736281 2024-03-19T11:31:46.594 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.594 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.594 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:46.594 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.591576+0000", 2024-03-19T11:31:46.594 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000102303 2024-03-19T11:31:46.594 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.594 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.594 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:46.594 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.591766+0000", 2024-03-19T11:31:46.594 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018938200000000001 2024-03-19T11:31:46.594 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:46.594 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:46.594 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:46.595 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.595 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.595 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3105 4.15 4:afa9f208:::smithi03829856-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e104)", 2024-03-19T11:31:46.595 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.558970+0000", 2024-03-19T11:31:46.595 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.955669514, 2024-03-19T11:31:46.595 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033553684, 2024-03-19T11:31:46.595 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:46.595 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:46.595 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:46.595 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:46.595 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:46.595 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3105 2024-03-19T11:31:46.595 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.595 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:46.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:46.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558970+0000", 2024-03-19T11:31:46.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:46.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:46.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558970+0000", 2024-03-19T11:31:46.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:46.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:46.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558971+0000", 2024-03-19T11:31:46.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4729999999999999e-06 2024-03-19T11:31:46.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.597 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:46.597 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558978+0000", 2024-03-19T11:31:46.597 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1369999999999996e-06 2024-03-19T11:31:46.597 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.597 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:46.597 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.559003+0000", 2024-03-19T11:31:46.597 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4525999999999999e-05 2024-03-19T11:31:46.597 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.597 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:46.597 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.559019+0000", 2024-03-19T11:31:46.597 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5999999999999999e-05 2024-03-19T11:31:46.597 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.598 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.598 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:46.598 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.569615+0000", 2024-03-19T11:31:46.598 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010596257 2024-03-19T11:31:46.598 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.598 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.598 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-03-19T11:31:46.598 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.569763+0000", 2024-03-19T11:31:46.598 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000147319 2024-03-19T11:31:46.598 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.598 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.598 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:46.598 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.592019+0000", 2024-03-19T11:31:46.598 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022256597999999999 2024-03-19T11:31:46.598 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.599 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.599 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:46.599 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.592250+0000", 2024-03-19T11:31:46.599 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023081100000000001 2024-03-19T11:31:46.599 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.599 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.599 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:46.599 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.592524+0000", 2024-03-19T11:31:46.599 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000273563 2024-03-19T11:31:46.599 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:46.599 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:46.599 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:46.599 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.599 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.599 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3106 4.15 4:ad3ad1b3:::smithi03829856-21:head [write 500908~484052] snapc 1f=[1f,1e,1d,1c,1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-19T11:31:46.600 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.563910+0000", 2024-03-19T11:31:46.600 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.950729564, 2024-03-19T11:31:46.600 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.029166066000000001, 2024-03-19T11:31:46.600 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:46.600 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:46.600 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:46.600 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:46.600 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:46.600 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3106 2024-03-19T11:31:46.600 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.600 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:46.600 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.600 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:46.600 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.563910+0000", 2024-03-19T11:31:46.601 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:46.601 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.601 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.601 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:46.601 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.563910+0000", 2024-03-19T11:31:46.601 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:46.601 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.601 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.601 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:46.602 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.563913+0000", 2024-03-19T11:31:46.602 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9960000000000002e-06 2024-03-19T11:31:46.602 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.602 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.602 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:46.602 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.564853+0000", 2024-03-19T11:31:46.602 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00093991599999999995 2024-03-19T11:31:46.602 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.602 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.602 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:46.602 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.564886+0000", 2024-03-19T11:31:46.602 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3315999999999997e-05 2024-03-19T11:31:46.602 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.602 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.603 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:46.603 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.564908+0000", 2024-03-19T11:31:46.603 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2262000000000001e-05 2024-03-19T11:31:46.603 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.603 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.603 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:46.603 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.569861+0000", 2024-03-19T11:31:46.603 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0049529070000000003 2024-03-19T11:31:46.603 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.603 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.604 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:46.604 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.569998+0000", 2024-03-19T11:31:46.604 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000136546 2024-03-19T11:31:46.604 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.604 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.604 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:46.604 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.570279+0000", 2024-03-19T11:31:46.604 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028119899999999999 2024-03-19T11:31:46.604 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.604 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.604 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:46.604 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.586586+0000", 2024-03-19T11:31:46.604 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016306621 2024-03-19T11:31:46.604 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.604 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.605 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:46.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.592743+0000", 2024-03-19T11:31:46.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061571150000000003 2024-03-19T11:31:46.605 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.605 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:46.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.592915+0000", 2024-03-19T11:31:46.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017273200000000001 2024-03-19T11:31:46.605 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.606 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:46.606 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.593076+0000", 2024-03-19T11:31:46.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016045600000000001 2024-03-19T11:31:46.606 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:46.606 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:46.606 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:46.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.606 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3107 4.15 4:ad3ad1b3:::smithi03829856-21:head [write 1411724~689184] snapc 1f=[1f,1e,1d,1c,1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-19T11:31:46.606 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.570058+0000", 2024-03-19T11:31:46.606 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.944581845999998, 2024-03-19T11:31:46.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023258826, 2024-03-19T11:31:46.606 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:46.606 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:46.607 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:46.607 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:46.607 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:46.607 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3107 2024-03-19T11:31:46.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.607 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:46.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:46.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.570058+0000", 2024-03-19T11:31:46.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:46.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.608 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:46.608 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.570058+0000", 2024-03-19T11:31:46.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:46.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.608 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:46.608 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.570061+0000", 2024-03-19T11:31:46.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0469999999999998e-06 2024-03-19T11:31:46.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.609 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:46.609 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572012+0000", 2024-03-19T11:31:46.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0019516760000000001 2024-03-19T11:31:46.609 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.609 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:46.609 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572052+0000", 2024-03-19T11:31:46.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9353000000000001e-05 2024-03-19T11:31:46.609 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.609 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:46.609 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572076+0000", 2024-03-19T11:31:46.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4459000000000001e-05 2024-03-19T11:31:46.610 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.610 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:46.610 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.577408+0000", 2024-03-19T11:31:46.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053316329999999997 2024-03-19T11:31:46.610 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.610 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:46.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.577697+0000", 2024-03-19T11:31:46.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028967499999999998 2024-03-19T11:31:46.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:46.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.578148+0000", 2024-03-19T11:31:46.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00045085900000000001 2024-03-19T11:31:46.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:46.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.586737+0000", 2024-03-19T11:31:46.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085889060000000003 2024-03-19T11:31:46.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:46.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.593175+0000", 2024-03-19T11:31:46.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0064375090000000001 2024-03-19T11:31:46.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:46.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.593235+0000", 2024-03-19T11:31:46.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0553000000000001e-05 2024-03-19T11:31:46.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.613 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:46.613 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.593316+0000", 2024-03-19T11:31:46.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1155999999999993e-05 2024-03-19T11:31:46.613 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:46.613 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:46.613 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:46.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.613 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3108 4.15 4:ad3ad1b3:::smithi03829856-21:head [setxattr _header (57) in=7b,truncate 2100908] snapc 1f=[1f,1e,1d,1c,1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-19T11:31:46.614 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.572119+0000", 2024-03-19T11:31:46.614 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.942520662, 2024-03-19T11:31:46.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021614495000000001, 2024-03-19T11:31:46.614 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:46.614 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:46.614 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:46.614 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:46.614 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:46.614 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3108 2024-03-19T11:31:46.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.614 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:46.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.614 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:46.614 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572119+0000", 2024-03-19T11:31:46.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:46.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:46.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572119+0000", 2024-03-19T11:31:46.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:46.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:46.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572121+0000", 2024-03-19T11:31:46.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.103e-06 2024-03-19T11:31:46.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:46.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572130+0000", 2024-03-19T11:31:46.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5230000000000001e-06 2024-03-19T11:31:46.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:46.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572157+0000", 2024-03-19T11:31:46.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6775999999999999e-05 2024-03-19T11:31:46.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:46.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572176+0000", 2024-03-19T11:31:46.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8516e-05 2024-03-19T11:31:46.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:46.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.584773+0000", 2024-03-19T11:31:46.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012596889 2024-03-19T11:31:46.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.617 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:46.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.585059+0000", 2024-03-19T11:31:46.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028658999999999998 2024-03-19T11:31:46.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.617 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:46.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.585591+0000", 2024-03-19T11:31:46.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00053221300000000002 2024-03-19T11:31:46.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.618 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:46.618 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.587798+0000", 2024-03-19T11:31:46.618 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0022066469999999999 2024-03-19T11:31:46.618 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.618 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:46.618 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.593611+0000", 2024-03-19T11:31:46.618 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0058127450000000002 2024-03-19T11:31:46.618 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.618 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:46.618 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.593671+0000", 2024-03-19T11:31:46.618 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0733000000000003e-05 2024-03-19T11:31:46.618 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:46.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.593733+0000", 2024-03-19T11:31:46.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1760000000000005e-05 2024-03-19T11:31:46.619 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:46.619 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:46.619 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:46.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.619 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3109 4.15 4:ad3ad1b3:::smithi03829856-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e104)", 2024-03-19T11:31:46.619 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.572203+0000", 2024-03-19T11:31:46.619 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.942435901, 2024-03-19T11:31:46.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021786163000000001, 2024-03-19T11:31:46.619 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:46.619 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:46.620 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:46.620 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:46.620 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:46.620 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3109 2024-03-19T11:31:46.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.620 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:46.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:46.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572203+0000", 2024-03-19T11:31:46.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:46.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:46.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572203+0000", 2024-03-19T11:31:46.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:46.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:46.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572205+0000", 2024-03-19T11:31:46.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.297e-06 2024-03-19T11:31:46.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:46.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572212+0000", 2024-03-19T11:31:46.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9129999999999997e-06 2024-03-19T11:31:46.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:46.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572233+0000", 2024-03-19T11:31:46.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1713e-05 2024-03-19T11:31:46.622 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.622 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.622 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:46.622 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572251+0000", 2024-03-19T11:31:46.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7112000000000002e-05 2024-03-19T11:31:46.622 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.622 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.622 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:46.622 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.587006+0000", 2024-03-19T11:31:46.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014755024 2024-03-19T11:31:46.622 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.622 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.622 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-03-19T11:31:46.622 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.587242+0000", 2024-03-19T11:31:46.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023635099999999999 2024-03-19T11:31:46.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:46.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.593797+0000", 2024-03-19T11:31:46.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0065553520000000004 2024-03-19T11:31:46.625 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:46.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.593880+0000", 2024-03-19T11:31:46.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2529999999999998e-05 2024-03-19T11:31:46.625 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:46.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:46.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:46.626 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.593990+0000", 2024-03-19T11:31:46.626 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010987100000000001 2024-03-19T11:31:46.626 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:46.626 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:46.626 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:46.626 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:46.626 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:46.626 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:46.629 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-19T11:31:46.629 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:46.637+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:46.631 INFO:tasks.rados.rados.0.smithi038.stdout:1659: writing smithi03829856-25 from 658907 to 1446623 tid 1 2024-03-19T11:31:46.631 INFO:tasks.rados.rados.0.smithi038.stdout:1659: writing smithi03829856-25 from 1930727 to 2428499 tid 2 2024-03-19T11:31:46.631 INFO:tasks.rados.rados.0.smithi038.stdout:1659: writing smithi03829856-25 from 3032909 to 3058907 tid 3 2024-03-19T11:31:46.631 INFO:tasks.rados.rados.0.smithi038.stdout:1652: done (7 left) 2024-03-19T11:31:46.631 INFO:tasks.rados.rados.0.smithi038.stdout:1653: done (6 left) 2024-03-19T11:31:46.632 INFO:tasks.rados.rados.0.smithi038.stdout:1654: done (5 left) 2024-03-19T11:31:46.632 INFO:tasks.rados.rados.0.smithi038.stdout:1655: done (4 left) 2024-03-19T11:31:46.632 INFO:tasks.rados.rados.0.smithi038.stdout:1656: done (3 left) 2024-03-19T11:31:46.632 INFO:tasks.rados.rados.0.smithi038.stdout:1657: done (2 left) 2024-03-19T11:31:46.632 INFO:tasks.rados.rados.0.smithi038.stdout:1658: done (1 left) 2024-03-19T11:31:46.632 INFO:tasks.rados.rados.0.smithi038.stdout:1660: set_chunk oid 46 target oid 2024-03-19T11:31:46.632 INFO:tasks.rados.rados.0.smithi038.stdout:1660: set_chunk oid 46 offset: 0 length: 4000000 target oid smithi03829856- offset: 0 snap -1 2024-03-19T11:31:46.632 INFO:tasks.rados.rados.0.smithi038.stdout:1661: read oid 39 snap 21 2024-03-19T11:31:46.632 INFO:tasks.rados.rados.0.smithi038.stdout:1661: expect (ObjNum 163 snap 17 seq_num 163) 2024-03-19T11:31:46.632 INFO:tasks.rados.rados.0.smithi038.stdout:1662: read oid 24 snap 38 2024-03-19T11:31:46.632 INFO:tasks.rados.rados.0.smithi038.stdout:1662: expect (ObjNum 265 snap 34 seq_num 265) 2024-03-19T11:31:46.632 INFO:tasks.rados.rados.0.smithi038.stdout:1663: write (excl) oid 33 current snap is 39 2024-03-19T11:31:46.632 INFO:tasks.rados.rados.0.smithi038.stdout:1659: finishing write tid 1 to smithi03829856-25 2024-03-19T11:31:46.632 INFO:tasks.rados.rados.0.smithi038.stdout:1659: finishing write tid 2 to smithi03829856-25 2024-03-19T11:31:46.633 INFO:tasks.rados.rados.0.smithi038.stdout:1663: seq_num 302 ranges {675714=462965,1589209=641430,2943979=131735} 2024-03-19T11:31:46.633 INFO:tasks.rados.rados.0.smithi038.stdout:1663: writing smithi03829856-33 from 675714 to 1138679 tid 1 2024-03-19T11:31:46.633 INFO:tasks.rados.rados.0.smithi038.stdout:1663: writing smithi03829856-33 from 1589209 to 2230639 tid 2 2024-03-19T11:31:46.633 INFO:tasks.rados.rados.0.smithi038.stdout:1663: writing smithi03829856-33 from 2943979 to 3075714 tid 3 2024-03-19T11:31:46.633 INFO:tasks.rados.rados.0.smithi038.stdout:1664: write oid 34 current snap is 39 2024-03-19T11:31:46.633 INFO:tasks.rados.rados.0.smithi038.stdout:1660: finishing set_chunk to oid 46 2024-03-19T11:31:46.633 INFO:tasks.rados.rados.0.smithi038.stdout:1660: got expected ERANGE (src dne) 2024-03-19T11:31:46.633 INFO:tasks.rados.rados.0.smithi038.stdout:1659: finishing write tid 3 to smithi03829856-25 2024-03-19T11:31:46.633 INFO:tasks.rados.rados.0.smithi038.stdout:1659: finishing write tid 4 to smithi03829856-25 2024-03-19T11:31:46.633 INFO:tasks.rados.rados.0.smithi038.stdout:1659: finishing write tid 5 to smithi03829856-25 2024-03-19T11:31:46.633 INFO:tasks.rados.rados.0.smithi038.stdout:1659: oid 25 updating version 0 to 144 2024-03-19T11:31:46.633 INFO:tasks.rados.rados.0.smithi038.stdout:1659: oid 25 updating version 144 to 145 2024-03-19T11:31:46.633 INFO:tasks.rados.rados.0.smithi038.stdout:1659: oid 25 version 145 is already newer than 143 2024-03-19T11:31:46.633 INFO:tasks.rados.rados.0.smithi038.stdout:1659: oid 25 updating version 145 to 146 2024-03-19T11:31:46.633 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 25 v 146 (ObjNum 301 snap 39 seq_num 301) dirty exists 2024-03-19T11:31:46.634 INFO:tasks.rados.rados.0.smithi038.stdout:1659: left oid 25 (ObjNum 301 snap 39 seq_num 301) 2024-03-19T11:31:46.634 INFO:tasks.rados.rados.0.smithi038.stdout:1664: seq_num 303 ranges {692521=538214,1931338=785088} 2024-03-19T11:31:46.634 INFO:tasks.rados.rados.0.smithi038.stdout:1664: writing smithi03829856-34 from 692521 to 1230735 tid 1 2024-03-19T11:31:46.634 INFO:tasks.rados.rados.0.smithi038.stdout:1664: writing smithi03829856-34 from 1931338 to 2716426 tid 2 2024-03-19T11:31:46.634 INFO:tasks.rados.rados.0.smithi038.stdout:1659: done (5 left) 2024-03-19T11:31:46.634 INFO:tasks.rados.rados.0.smithi038.stdout:1660: done (4 left) 2024-03-19T11:31:46.634 INFO:tasks.rados.rados.0.smithi038.stdout:1665: set_chunk oid 3 target oid 2024-03-19T11:31:46.634 INFO:tasks.rados.rados.0.smithi038.stdout:1665: set_chunk oid 3 offset: 1548800 length: 631296 target oid 835d9c473c3499fec05bdb392050de709bd9ae19fad07cd5bd41d67d3873714b offset: 0 snap -1 2024-03-19T11:31:46.634 INFO:tasks.rados.rados.0.smithi038.stdout:1666: read oid 44 snap 35 2024-03-19T11:31:46.634 INFO:tasks.rados.rados.0.smithi038.stdout:1666: expect (ObjNum 252 snap 32 seq_num 252) 2024-03-19T11:31:46.634 INFO:tasks.rados.rados.0.smithi038.stdout:1667: read oid 12 snap -1 2024-03-19T11:31:46.634 INFO:tasks.rados.rados.0.smithi038.stdout:1667: expect (ObjNum 228 snap 26 seq_num 228) 2024-03-19T11:31:46.634 INFO:tasks.rados.rados.0.smithi038.stdout:1668: tier_evict oid 6 2024-03-19T11:31:46.634 INFO:tasks.rados.rados.0.smithi038.stdout:1668: tier_evict oid 6 snap -1 2024-03-19T11:31:46.634 INFO:tasks.rados.rados.0.smithi038.stdout:1662: expect (ObjNum 265 snap 34 seq_num 265) 2024-03-19T11:31:46.729 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:46.737+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:46.751 INFO:tasks.rados.rados.0.smithi038.stdout:1661: expect (ObjNum 163 snap 17 seq_num 163) 2024-03-19T11:31:46.804 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:46.804 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:31:46.804 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:31:46.804 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:46.818 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-19T11:31:46.860 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:46.869+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:46.908 INFO:tasks.rados.rados.0.smithi038.stdout:1663: finishing write tid 1 to smithi03829856-33 2024-03-19T11:31:46.908 INFO:tasks.rados.rados.0.smithi038.stdout:1663: finishing write tid 2 to smithi03829856-33 2024-03-19T11:31:46.908 INFO:tasks.rados.rados.0.smithi038.stdout:1663: finishing write tid 3 to smithi03829856-33 2024-03-19T11:31:46.908 INFO:tasks.rados.rados.0.smithi038.stdout:1663: finishing write tid 4 to smithi03829856-33 2024-03-19T11:31:46.908 INFO:tasks.rados.rados.0.smithi038.stdout:1663: finishing write tid 5 to smithi03829856-33 2024-03-19T11:31:46.908 INFO:tasks.rados.rados.0.smithi038.stdout:1663: oid 33 updating version 0 to 207 2024-03-19T11:31:46.908 INFO:tasks.rados.rados.0.smithi038.stdout:1663: oid 33 version 207 is already newer than 205 2024-03-19T11:31:46.908 INFO:tasks.rados.rados.0.smithi038.stdout:1663: oid 33 version 207 is already newer than 206 2024-03-19T11:31:46.908 INFO:tasks.rados.rados.0.smithi038.stdout:1663: oid 33 version 207 is already newer than 204 2024-03-19T11:31:46.908 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 33 v 207 (ObjNum 302 snap 39 seq_num 302) dirty exists 2024-03-19T11:31:46.908 INFO:tasks.rados.rados.0.smithi038.stdout:1663: left oid 33 (ObjNum 302 snap 39 seq_num 302) 2024-03-19T11:31:46.909 INFO:tasks.rados.rados.0.smithi038.stdout:1664: finishing write tid 1 to smithi03829856-34 2024-03-19T11:31:46.909 INFO:tasks.rados.rados.0.smithi038.stdout:1664: finishing write tid 2 to smithi03829856-34 2024-03-19T11:31:46.909 INFO:tasks.rados.rados.0.smithi038.stdout:1664: finishing write tid 3 to smithi03829856-34 2024-03-19T11:31:46.909 INFO:tasks.rados.rados.0.smithi038.stdout:1664: finishing write tid 4 to smithi03829856-34 2024-03-19T11:31:46.909 INFO:tasks.rados.rados.0.smithi038.stdout:1664: oid 34 updating version 0 to 55 2024-03-19T11:31:46.909 INFO:tasks.rados.rados.0.smithi038.stdout:1664: oid 34 version 55 is already newer than 54 2024-03-19T11:31:46.909 INFO:tasks.rados.rados.0.smithi038.stdout:1664: oid 34 updating version 55 to 56 2024-03-19T11:31:46.909 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 34 v 56 (ObjNum 303 snap 39 seq_num 303) dirty exists 2024-03-19T11:31:46.909 INFO:tasks.rados.rados.0.smithi038.stdout:1664: left oid 34 (ObjNum 303 snap 39 seq_num 303) 2024-03-19T11:31:46.909 INFO:tasks.rados.rados.0.smithi038.stdout:1665: finishing set_chunk to oid 3 2024-03-19T11:31:46.909 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 3 set_chunk returned error code -95 offset: 1548800 length: 631296 2024-03-19T11:31:46.909 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 3 v 0 (ObjNum 271 snap 34 seq_num 271) dirty exists 2024-03-19T11:31:46.909 INFO:tasks.rados.rados.0.smithi038.stdout:1661: done (7 left) 2024-03-19T11:31:46.909 INFO:tasks.rados.rados.0.smithi038.stdout:1662: done (6 left) 2024-03-19T11:31:46.910 INFO:tasks.rados.rados.0.smithi038.stdout:1663: done (5 left) 2024-03-19T11:31:46.910 INFO:tasks.rados.rados.0.smithi038.stdout:1664: done (4 left) 2024-03-19T11:31:46.910 INFO:tasks.rados.rados.0.smithi038.stdout:1665: done (3 left) 2024-03-19T11:31:46.910 INFO:tasks.rados.rados.0.smithi038.stdout:1669: read oid 35 snap -1 2024-03-19T11:31:46.910 INFO:tasks.rados.rados.0.smithi038.stdout:1669: expect (ObjNum 294 snap 39 seq_num 294) 2024-03-19T11:31:46.910 INFO:tasks.rados.rados.0.smithi038.stdout:1670: read oid 28 snap -1 2024-03-19T11:31:46.910 INFO:tasks.rados.rados.0.smithi038.stdout:1670: expect (ObjNum 87 snap 8 seq_num 87) 2024-03-19T11:31:46.910 INFO:tasks.rados.rados.0.smithi038.stdout:1671: write (excl) oid 7 current snap is 39 2024-03-19T11:31:46.910 INFO:tasks.rados.rados.0.smithi038.stdout:1668: got (22) Invalid argument 2024-03-19T11:31:46.910 INFO:tasks.rados.rados.0.smithi038.stdout:1666: expect (ObjNum 252 snap 32 seq_num 252) 2024-03-19T11:31:46.931 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:46.937+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:46.987 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:46.987 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:31:46.987 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:31:46.987 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:31:46.987 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:47.001 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-19T11:31:47.028 INFO:tasks.rados.rados.0.smithi038.stdout:1667: expect (ObjNum 228 snap 26 seq_num 228) 2024-03-19T11:31:47.033 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:47.039+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:47.104 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:47.111+0000 7f6a7d9d6640 -1 osd.1 125 handle_osd_map injecting map crc failure 2024-03-19T11:31:47.132 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:47.139+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:47.171 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:47.171 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:31:47.172 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:31:47.172 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:31:47.172 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.172 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:30 4.5 4:ac9cd3a3:::smithi03829856-8:head [write 1776668~340981] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:31:47.172 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:04.777922+0000", 2024-03-19T11:31:47.172 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 102.40124377799999, 2024-03-19T11:31:47.172 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015469894, 2024-03-19T11:31:47.172 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:47.172 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:47.172 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:47.172 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:47.172 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:47.172 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 30 2024-03-19T11:31:47.173 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.173 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:47.173 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.173 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:47.173 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.777922+0000", 2024-03-19T11:31:47.173 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:47.173 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.173 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.173 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:47.173 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.777922+0000", 2024-03-19T11:31:47.173 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:47.173 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.173 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.173 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:47.174 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.777924+0000", 2024-03-19T11:31:47.174 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3999999999999999e-06 2024-03-19T11:31:47.174 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.174 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.174 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:47.174 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778461+0000", 2024-03-19T11:31:47.174 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00053698299999999999 2024-03-19T11:31:47.174 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.174 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.174 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:47.174 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778487+0000", 2024-03-19T11:31:47.174 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5947000000000001e-05 2024-03-19T11:31:47.174 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.174 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.175 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:47.175 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778502+0000", 2024-03-19T11:31:47.175 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4256e-05 2024-03-19T11:31:47.175 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.175 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.175 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:47.175 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.781978+0000", 2024-03-19T11:31:47.175 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00347693 2024-03-19T11:31:47.175 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.175 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.175 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:47.175 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.782196+0000", 2024-03-19T11:31:47.175 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021726499999999999 2024-03-19T11:31:47.175 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.176 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.176 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:47.176 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.782572+0000", 2024-03-19T11:31:47.176 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00037635000000000001 2024-03-19T11:31:47.176 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.176 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.176 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:47.176 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.786960+0000", 2024-03-19T11:31:47.176 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043882130000000002 2024-03-19T11:31:47.176 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.176 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.176 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:47.176 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793113+0000", 2024-03-19T11:31:47.176 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061530760000000004 2024-03-19T11:31:47.176 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.177 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.177 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:47.177 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793276+0000", 2024-03-19T11:31:47.177 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000162652 2024-03-19T11:31:47.177 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.177 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.177 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:47.177 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793392+0000", 2024-03-19T11:31:47.177 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115822 2024-03-19T11:31:47.177 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:47.177 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:47.177 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:47.177 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.177 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.177 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:31 4.5 4:ac9cd3a3:::smithi03829856-8:head [setxattr _header (55) in=7b,truncate 2117649] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:31:47.178 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:04.778533+0000", 2024-03-19T11:31:47.178 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 102.40063264299999, 2024-03-19T11:31:47.178 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015182533, 2024-03-19T11:31:47.178 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:47.178 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:47.178 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:47.178 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:47.178 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:47.178 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 31 2024-03-19T11:31:47.178 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.178 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:47.178 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.178 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:47.178 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778533+0000", 2024-03-19T11:31:47.179 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:47.179 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.179 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.179 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:47.179 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778533+0000", 2024-03-19T11:31:47.179 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:47.179 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.179 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.179 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:47.179 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778534+0000", 2024-03-19T11:31:47.180 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1069999999999999e-06 2024-03-19T11:31:47.180 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.180 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.180 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:47.180 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778541+0000", 2024-03-19T11:31:47.180 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2529999999999998e-06 2024-03-19T11:31:47.180 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.180 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.180 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:47.180 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778558+0000", 2024-03-19T11:31:47.180 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6290000000000002e-05 2024-03-19T11:31:47.181 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.181 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.181 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:47.181 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778568+0000", 2024-03-19T11:31:47.181 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0360000000000001e-05 2024-03-19T11:31:47.181 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.181 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.181 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:47.181 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.785511+0000", 2024-03-19T11:31:47.181 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00694306 2024-03-19T11:31:47.181 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.181 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.181 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:47.182 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.785710+0000", 2024-03-19T11:31:47.182 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019837199999999999 2024-03-19T11:31:47.182 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.182 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.182 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:47.182 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.785929+0000", 2024-03-19T11:31:47.182 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021960799999999999 2024-03-19T11:31:47.182 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.182 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.182 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:47.182 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.787281+0000", 2024-03-19T11:31:47.182 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001352313 2024-03-19T11:31:47.182 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.183 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.183 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:47.183 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793492+0000", 2024-03-19T11:31:47.183 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062107760000000003 2024-03-19T11:31:47.183 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.183 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.183 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:47.183 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793578+0000", 2024-03-19T11:31:47.183 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5711999999999995e-05 2024-03-19T11:31:47.183 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.183 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.183 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:47.183 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793716+0000", 2024-03-19T11:31:47.183 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000137682 2024-03-19T11:31:47.184 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:47.184 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:47.184 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:47.184 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.184 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.184 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:32 4.5 4:ac9cd3a3:::smithi03829856-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:31:47.184 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:04.778582+0000", 2024-03-19T11:31:47.184 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 102.40058412899999, 2024-03-19T11:31:47.184 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015761827999999999, 2024-03-19T11:31:47.184 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:47.184 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:47.184 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:47.184 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:47.184 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:47.185 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 32 2024-03-19T11:31:47.185 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.185 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:47.185 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.185 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:47.185 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778582+0000", 2024-03-19T11:31:47.185 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:47.185 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.185 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.185 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:47.185 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778582+0000", 2024-03-19T11:31:47.185 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:47.185 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.185 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.185 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:47.186 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778583+0000", 2024-03-19T11:31:47.186 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.026e-06 2024-03-19T11:31:47.186 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.186 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.186 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:47.186 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778587+0000", 2024-03-19T11:31:47.186 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7870000000000001e-06 2024-03-19T11:31:47.186 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.186 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.186 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:47.186 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778599+0000", 2024-03-19T11:31:47.186 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1623e-05 2024-03-19T11:31:47.186 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.187 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.187 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:47.187 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778608+0000", 2024-03-19T11:31:47.187 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7399999999999993e-06 2024-03-19T11:31:47.187 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.187 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.187 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:47.187 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.787009+0000", 2024-03-19T11:31:47.187 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0084010490000000007 2024-03-19T11:31:47.187 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.187 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.187 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-03-19T11:31:47.187 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.787202+0000", 2024-03-19T11:31:47.187 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019309500000000001 2024-03-19T11:31:47.187 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.188 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.188 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:47.188 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793832+0000", 2024-03-19T11:31:47.188 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.006630052 2024-03-19T11:31:47.188 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.188 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.188 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:47.188 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.794048+0000", 2024-03-19T11:31:47.188 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000215505 2024-03-19T11:31:47.188 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.188 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.188 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:47.188 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.794343+0000", 2024-03-19T11:31:47.188 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029595100000000003 2024-03-19T11:31:47.189 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:47.189 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:47.189 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:47.189 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.189 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.189 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:891 4.5 4:af9d0d56:::smithi03829856-20:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:31:47.189 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:09.286410+0000", 2024-03-19T11:31:47.189 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 97.892755797000007, 2024-03-19T11:31:47.189 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035191405000000002, 2024-03-19T11:31:47.189 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:47.189 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:47.189 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:47.189 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:47.189 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:47.189 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 891 2024-03-19T11:31:47.190 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.190 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:47.190 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.190 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:47.190 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286410+0000", 2024-03-19T11:31:47.190 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:47.190 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.191 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.191 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:47.191 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286410+0000", 2024-03-19T11:31:47.191 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:47.191 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.191 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.191 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:47.191 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286413+0000", 2024-03-19T11:31:47.192 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2660000000000002e-06 2024-03-19T11:31:47.192 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.192 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.192 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:47.192 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286426+0000", 2024-03-19T11:31:47.192 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.258e-05 2024-03-19T11:31:47.193 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.193 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.193 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:47.193 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286460+0000", 2024-03-19T11:31:47.193 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3943999999999999e-05 2024-03-19T11:31:47.193 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.193 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.194 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:47.194 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286477+0000", 2024-03-19T11:31:47.194 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6799999999999998e-05 2024-03-19T11:31:47.194 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.194 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.194 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:47.194 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286605+0000", 2024-03-19T11:31:47.194 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000128892 2024-03-19T11:31:47.194 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.194 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.194 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:47.195 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286769+0000", 2024-03-19T11:31:47.195 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016359799999999999 2024-03-19T11:31:47.195 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.195 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.195 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:47.195 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.321601+0000", 2024-03-19T11:31:47.195 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.034832324999999997 2024-03-19T11:31:47.195 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:47.195 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:47.195 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:47.195 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.195 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.196 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1015 4.5 4:af9d0d56:::smithi03829856-20:head [tier-promote] snapc 6=[6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:31:47.196 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:12.761855+0000", 2024-03-19T11:31:47.196 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 94.417310271000005, 2024-03-19T11:31:47.196 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026760430000000002, 2024-03-19T11:31:47.196 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:47.196 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:47.196 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:47.196 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:47.196 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:47.196 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1015 2024-03-19T11:31:47.196 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.196 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:47.197 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.197 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:47.197 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761855+0000", 2024-03-19T11:31:47.197 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:47.197 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.197 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.197 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:47.197 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761855+0000", 2024-03-19T11:31:47.197 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:47.198 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.198 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.198 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:47.198 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761858+0000", 2024-03-19T11:31:47.198 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8140000000000001e-06 2024-03-19T11:31:47.198 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.198 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.198 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:47.198 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761872+0000", 2024-03-19T11:31:47.198 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3813e-05 2024-03-19T11:31:47.198 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.199 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:47.199 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761905+0000", 2024-03-19T11:31:47.199 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2642999999999999e-05 2024-03-19T11:31:47.199 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.199 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:47.199 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761925+0000", 2024-03-19T11:31:47.199 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0489e-05 2024-03-19T11:31:47.200 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.200 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.200 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:47.200 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.762070+0000", 2024-03-19T11:31:47.200 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000145153 2024-03-19T11:31:47.200 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.200 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.200 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:47.200 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.762365+0000", 2024-03-19T11:31:47.201 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029482100000000002 2024-03-19T11:31:47.201 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.201 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.201 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:47.201 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.788616+0000", 2024-03-19T11:31:47.201 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026250697 2024-03-19T11:31:47.201 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:47.201 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:47.201 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:47.201 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.201 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.202 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1121 4.5 4:aa3303a5:::smithi03829856-17:head [tier-promote] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:31:47.202 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:14.164281+0000", 2024-03-19T11:31:47.202 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 93.014885026000002, 2024-03-19T11:31:47.202 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023896435000000001, 2024-03-19T11:31:47.202 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:47.202 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:47.202 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:47.202 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:47.202 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:47.203 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1121 2024-03-19T11:31:47.203 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.203 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:47.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.203 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:47.203 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164281+0000", 2024-03-19T11:31:47.203 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:47.203 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.204 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:47.204 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164281+0000", 2024-03-19T11:31:47.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:47.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.204 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:47.204 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164283+0000", 2024-03-19T11:31:47.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8500000000000001e-06 2024-03-19T11:31:47.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.205 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:47.205 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164291+0000", 2024-03-19T11:31:47.205 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.14e-06 2024-03-19T11:31:47.206 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.206 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.206 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:47.206 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164313+0000", 2024-03-19T11:31:47.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.207e-05 2024-03-19T11:31:47.206 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.206 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.206 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:47.206 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164328+0000", 2024-03-19T11:31:47.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5403000000000001e-05 2024-03-19T11:31:47.206 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.206 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.206 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:47.207 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.167981+0000", 2024-03-19T11:31:47.207 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036529879999999998 2024-03-19T11:31:47.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.207 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:47.207 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.168274+0000", 2024-03-19T11:31:47.207 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029313100000000002 2024-03-19T11:31:47.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.207 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:47.207 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.188177+0000", 2024-03-19T11:31:47.207 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019902853000000002 2024-03-19T11:31:47.207 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:47.207 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:47.208 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:47.208 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.208 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.208 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1302 4.15 4:aa3303a5:::smithi03829856-17:head [write 413472~620140,stat] snapc b=[b,a,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-03-19T11:31:47.208 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:22.966426+0000", 2024-03-19T11:31:47.208 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 84.212739292999998, 2024-03-19T11:31:47.208 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018043513000000001, 2024-03-19T11:31:47.208 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:47.208 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:47.208 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:47.208 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:47.208 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:47.208 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1302 2024-03-19T11:31:47.208 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.209 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:47.209 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.209 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:47.209 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.966426+0000", 2024-03-19T11:31:47.209 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:47.209 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.209 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.209 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:47.209 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.966426+0000", 2024-03-19T11:31:47.209 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:47.209 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.209 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.209 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:47.209 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.966429+0000", 2024-03-19T11:31:47.210 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3829999999999999e-06 2024-03-19T11:31:47.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.210 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.210 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:47.210 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972117+0000", 2024-03-19T11:31:47.210 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0056885750000000004 2024-03-19T11:31:47.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.210 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.210 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:47.210 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972156+0000", 2024-03-19T11:31:47.210 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8878000000000003e-05 2024-03-19T11:31:47.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.210 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.210 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:47.211 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972184+0000", 2024-03-19T11:31:47.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7509999999999999e-05 2024-03-19T11:31:47.211 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.211 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:47.211 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972309+0000", 2024-03-19T11:31:47.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012525899999999999 2024-03-19T11:31:47.211 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.211 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:47.211 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972566+0000", 2024-03-19T11:31:47.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025740799999999999 2024-03-19T11:31:47.211 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.212 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:47.212 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972899+0000", 2024-03-19T11:31:47.212 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033238799999999998 2024-03-19T11:31:47.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.212 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:47.212 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984088+0000", 2024-03-19T11:31:47.212 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011189466 2024-03-19T11:31:47.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.212 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:47.212 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984138+0000", 2024-03-19T11:31:47.212 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9666999999999997e-05 2024-03-19T11:31:47.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.213 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:47.213 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984329+0000", 2024-03-19T11:31:47.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019086799999999999 2024-03-19T11:31:47.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.213 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.213 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:47.213 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984470+0000", 2024-03-19T11:31:47.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000141111 2024-03-19T11:31:47.213 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:47.213 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:47.213 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:47.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.213 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.213 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.1 50/49 [PushOp(4:866f3dcc:::smithi03829856-31:7, version: 38'44, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@38'44, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 45'62, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@45'62, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 46'68, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@46'68, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:7, version: 47'71, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@47'71, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 47'72, data_included: [440350~816629,1309629~231947], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@47'72, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:47.214 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.959457+0000", 2024-03-19T11:31:47.214 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 79.219708875999999, 2024-03-19T11:31:47.214 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018416666000000002, 2024-03-19T11:31:47.214 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:47.214 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:47.214 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:47.214 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.214 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:47.214 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.959457+0000", 2024-03-19T11:31:47.214 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:47.214 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.214 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.214 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:47.215 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.959457+0000", 2024-03-19T11:31:47.215 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:47.215 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.215 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.215 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:47.215 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.959458+0000", 2024-03-19T11:31:47.215 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4899999999999999e-06 2024-03-19T11:31:47.215 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.215 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.215 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:47.215 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.962822+0000", 2024-03-19T11:31:47.215 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0033640620000000001 2024-03-19T11:31:47.215 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.216 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:47.216 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.962893+0000", 2024-03-19T11:31:47.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1060000000000001e-05 2024-03-19T11:31:47.216 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.216 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:47.216 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.962977+0000", 2024-03-19T11:31:47.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3311000000000002e-05 2024-03-19T11:31:47.216 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.216 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:47.216 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963208+0000", 2024-03-19T11:31:47.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023121200000000001 2024-03-19T11:31:47.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:47.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963245+0000", 2024-03-19T11:31:47.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6844999999999997e-05 2024-03-19T11:31:47.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:47.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.977874+0000", 2024-03-19T11:31:47.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014628686 2024-03-19T11:31:47.217 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:47.217 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:47.218 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:47.218 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.218 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.218 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2284 4.15 4:afa9f208:::smithi03829856-35:head [read 1314816~702976] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-03-19T11:31:47.218 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:53.596775+0000", 2024-03-19T11:31:47.218 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 53.582390789999998, 2024-03-19T11:31:47.218 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.47260528299999999, 2024-03-19T11:31:47.218 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:47.218 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:47.218 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:47.218 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:47.218 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:47.218 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2284 2024-03-19T11:31:47.219 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.219 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:47.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.219 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:47.219 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596775+0000", 2024-03-19T11:31:47.219 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:47.219 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.219 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:47.219 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596775+0000", 2024-03-19T11:31:47.219 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:47.219 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.219 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:47.219 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596778+0000", 2024-03-19T11:31:47.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.174e-06 2024-03-19T11:31:47.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.220 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:47.220 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596794+0000", 2024-03-19T11:31:47.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5372000000000001e-05 2024-03-19T11:31:47.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.220 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:47.220 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596828+0000", 2024-03-19T11:31:47.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4107999999999999e-05 2024-03-19T11:31:47.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.220 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:47.220 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596844+0000", 2024-03-19T11:31:47.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6296000000000001e-05 2024-03-19T11:31:47.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:47.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596968+0000", 2024-03-19T11:31:47.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012383699999999999 2024-03-19T11:31:47.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:31:47.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596988+0000", 2024-03-19T11:31:47.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0645e-05 2024-03-19T11:31:47.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:47.222 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.068916+0000", 2024-03-19T11:31:47.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.47192744800000003 2024-03-19T11:31:47.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:47.222 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.069015+0000", 2024-03-19T11:31:47.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9500999999999995e-05 2024-03-19T11:31:47.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:47.222 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.069380+0000", 2024-03-19T11:31:47.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00036490200000000001 2024-03-19T11:31:47.222 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:47.222 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:47.223 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:47.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.223 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2286 4.15 4:afa9f208:::smithi03829856-35:head [assert-version v108,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e75)", 2024-03-19T11:31:47.223 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.084826+0000", 2024-03-19T11:31:47.223 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 53.094339423000001, 2024-03-19T11:31:47.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.055799556, 2024-03-19T11:31:47.223 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:47.223 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:47.223 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:47.223 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:47.223 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:47.223 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2286 2024-03-19T11:31:47.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.223 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:47.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.224 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:47.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084826+0000", 2024-03-19T11:31:47.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:47.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.224 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:47.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084826+0000", 2024-03-19T11:31:47.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:47.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.224 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:47.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084828+0000", 2024-03-19T11:31:47.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1409999999999999e-06 2024-03-19T11:31:47.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:47.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084838+0000", 2024-03-19T11:31:47.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3740000000000004e-06 2024-03-19T11:31:47.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:47.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084859+0000", 2024-03-19T11:31:47.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1644000000000001e-05 2024-03-19T11:31:47.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:47.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084871+0000", 2024-03-19T11:31:47.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1127e-05 2024-03-19T11:31:47.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:47.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084961+0000", 2024-03-19T11:31:47.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0371999999999995e-05 2024-03-19T11:31:47.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:47.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.085130+0000", 2024-03-19T11:31:47.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016884800000000001 2024-03-19T11:31:47.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5,7", 2024-03-19T11:31:47.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.089309+0000", 2024-03-19T11:31:47.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041788650000000004 2024-03-19T11:31:47.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.227 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:47.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.090362+0000", 2024-03-19T11:31:47.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001053362 2024-03-19T11:31:47.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.227 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:47.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.090511+0000", 2024-03-19T11:31:47.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014887200000000001 2024-03-19T11:31:47.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.227 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:47.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.140479+0000", 2024-03-19T11:31:47.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.049968006000000002 2024-03-19T11:31:47.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.228 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.228 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:47.228 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.140534+0000", 2024-03-19T11:31:47.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4917000000000003e-05 2024-03-19T11:31:47.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.228 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.228 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:47.228 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.140626+0000", 2024-03-19T11:31:47.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2027999999999994e-05 2024-03-19T11:31:47.228 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:47.228 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:47.228 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:47.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.229 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.229 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1d58db11:::smithi03829856-18:6, version: 55'80, data_included: [759683~639358,1904434~409218], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:6@55'80, size: 3159683, copy_subset: [(0, 3159683)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2313652, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1d58db11:::smithi03829856-18:b, version: 58'84, data_included: [443718~955323,1564138~93253], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:b@58'84, size: 3445402, copy_subset: [(0, 3445402)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1657391, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:10, version: 66'95, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:10@66'95, size: 3764735, copy_subset: [(0, 3764735)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:12, version: 66'97, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:12@66'97, size: 2504243, copy_subset: [(0, 2504243)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:c, version: 70'102, data_included: [416807~576167,1542129~472409], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:c@70'102, size: 2857157, copy_subset: [(0, 2857157)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2014538, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:47.229 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.274749+0000", 2024-03-19T11:31:47.229 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 52.904416492000003, 2024-03-19T11:31:47.229 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017652601, 2024-03-19T11:31:47.229 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:47.229 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:47.229 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:47.229 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.229 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:47.229 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.274749+0000", 2024-03-19T11:31:47.229 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:47.230 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.230 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:47.230 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.274749+0000", 2024-03-19T11:31:47.230 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:47.230 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.230 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:47.230 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.274751+0000", 2024-03-19T11:31:47.230 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8500000000000001e-06 2024-03-19T11:31:47.230 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.230 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:47.230 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.278426+0000", 2024-03-19T11:31:47.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036747559999999999 2024-03-19T11:31:47.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.231 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:47.231 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.278470+0000", 2024-03-19T11:31:47.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4119000000000002e-05 2024-03-19T11:31:47.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.231 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:47.231 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.278506+0000", 2024-03-19T11:31:47.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6066000000000003e-05 2024-03-19T11:31:47.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.231 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:47.232 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.278709+0000", 2024-03-19T11:31:47.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000202711 2024-03-19T11:31:47.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.232 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:47.232 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.278731+0000", 2024-03-19T11:31:47.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2657e-05 2024-03-19T11:31:47.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.232 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:47.232 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.292402+0000", 2024-03-19T11:31:47.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013670442 2024-03-19T11:31:47.232 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:47.232 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:47.232 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:47.233 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.233 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1dabc0f1:::smithi03829856-1:head, version: 70'103, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:head@70'103, size: 2621892, copy_subset: [(0, 2621892)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:47.233 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.280012+0000", 2024-03-19T11:31:47.233 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 52.899153847000001, 2024-03-19T11:31:47.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018569209, 2024-03-19T11:31:47.233 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:47.233 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:47.233 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:47.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.233 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:47.233 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280012+0000", 2024-03-19T11:31:47.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:47.233 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.234 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:47.234 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280012+0000", 2024-03-19T11:31:47.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:47.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.234 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.234 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:47.234 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280013+0000", 2024-03-19T11:31:47.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9899999999999999e-07 2024-03-19T11:31:47.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.234 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.234 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:47.234 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280584+0000", 2024-03-19T11:31:47.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057090299999999999 2024-03-19T11:31:47.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:47.235 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280612+0000", 2024-03-19T11:31:47.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8442999999999999e-05 2024-03-19T11:31:47.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:47.235 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280631+0000", 2024-03-19T11:31:47.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8896999999999999e-05 2024-03-19T11:31:47.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:47.235 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.295041+0000", 2024-03-19T11:31:47.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014410127999999999 2024-03-19T11:31:47.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.236 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:47.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.295050+0000", 2024-03-19T11:31:47.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2350000000000005e-06 2024-03-19T11:31:47.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:47.238 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.298581+0000", 2024-03-19T11:31:47.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035308039999999998 2024-03-19T11:31:47.238 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:47.238 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:47.238 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:47.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.238 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.18 e75,74 4:1d58db11:::smithi03829856-18:c,70'104)", 2024-03-19T11:31:47.238 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.280680+0000", 2024-03-19T11:31:47.238 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 52.898485372000003, 2024-03-19T11:31:47.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018106229000000001, 2024-03-19T11:31:47.238 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:47.239 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:47.239 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:47.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:47.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280680+0000", 2024-03-19T11:31:47.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:47.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:47.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280680+0000", 2024-03-19T11:31:47.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:47.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:47.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280681+0000", 2024-03-19T11:31:47.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2099999999999995e-07 2024-03-19T11:31:47.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:47.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280686+0000", 2024-03-19T11:31:47.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2240000000000003e-06 2024-03-19T11:31:47.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:47.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280698+0000", 2024-03-19T11:31:47.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1698999999999999e-05 2024-03-19T11:31:47.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:47.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280705+0000", 2024-03-19T11:31:47.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4350000000000002e-06 2024-03-19T11:31:47.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:47.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.298643+0000", 2024-03-19T11:31:47.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017938352000000001 2024-03-19T11:31:47.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:47.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.298656+0000", 2024-03-19T11:31:47.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3084999999999999e-05 2024-03-19T11:31:47.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:47.242 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.298787+0000", 2024-03-19T11:31:47.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013061299999999999 2024-03-19T11:31:47.242 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:47.242 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:47.242 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:47.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.242 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.5.0:143 4.18 e75/74 4:1d58db11:::smithi03829856-18:head v 75'106, mlcod=0'0)", 2024-03-19T11:31:47.242 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.282327+0000", 2024-03-19T11:31:47.242 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 52.896839001000004, 2024-03-19T11:31:47.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016753244, 2024-03-19T11:31:47.242 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:47.242 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:47.242 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:47.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.243 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:47.243 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.282327+0000", 2024-03-19T11:31:47.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:47.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.243 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:47.243 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.282327+0000", 2024-03-19T11:31:47.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:47.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.243 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:47.243 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.282328+0000", 2024-03-19T11:31:47.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.488e-06 2024-03-19T11:31:47.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:47.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.283127+0000", 2024-03-19T11:31:47.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00079853699999999999 2024-03-19T11:31:47.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:47.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.283152+0000", 2024-03-19T11:31:47.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5215000000000002e-05 2024-03-19T11:31:47.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:47.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.283167+0000", 2024-03-19T11:31:47.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4635e-05 2024-03-19T11:31:47.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:47.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.292633+0000", 2024-03-19T11:31:47.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0094664339999999993 2024-03-19T11:31:47.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:47.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.292655+0000", 2024-03-19T11:31:47.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1934999999999999e-05 2024-03-19T11:31:47.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:47.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.298981+0000", 2024-03-19T11:31:47.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0063255129999999996 2024-03-19T11:31:47.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:47.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.299080+0000", 2024-03-19T11:31:47.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9487000000000005e-05 2024-03-19T11:31:47.246 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:47.246 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:47.246 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:47.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.246 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2441 4.5 4:a38b9453:::smithi03829856-37:head [delete] snapc 17=[17,16,15,13,12,b,6] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e84)", 2024-03-19T11:31:47.246 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:03.060663+0000", 2024-03-19T11:31:47.246 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 44.118503156999999, 2024-03-19T11:31:47.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10917726, 2024-03-19T11:31:47.247 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:47.247 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:47.247 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:47.247 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:47.247 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:47.247 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2441 2024-03-19T11:31:47.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.247 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:47.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.247 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:47.247 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060663+0000", 2024-03-19T11:31:47.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:47.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.247 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:47.248 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060663+0000", 2024-03-19T11:31:47.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:47.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.248 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:47.248 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060666+0000", 2024-03-19T11:31:47.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.05e-06 2024-03-19T11:31:47.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.248 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:47.248 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060680+0000", 2024-03-19T11:31:47.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4276e-05 2024-03-19T11:31:47.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.248 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:47.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060714+0000", 2024-03-19T11:31:47.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4325000000000001e-05 2024-03-19T11:31:47.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:47.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060737+0000", 2024-03-19T11:31:47.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2959000000000002e-05 2024-03-19T11:31:47.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:47.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060895+0000", 2024-03-19T11:31:47.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000157787 2024-03-19T11:31:47.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:31:47.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060923+0000", 2024-03-19T11:31:47.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8019000000000001e-05 2024-03-19T11:31:47.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:47.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.066222+0000", 2024-03-19T11:31:47.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052989270000000002 2024-03-19T11:31:47.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:47.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.070153+0000", 2024-03-19T11:31:47.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039315119999999999 2024-03-19T11:31:47.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:47.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.147930+0000", 2024-03-19T11:31:47.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.077776459000000006 2024-03-19T11:31:47.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:47.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.148079+0000", 2024-03-19T11:31:47.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014888 2024-03-19T11:31:47.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:47.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.148370+0000", 2024-03-19T11:31:47.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029130999999999997 2024-03-19T11:31:47.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:47.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.152531+0000", 2024-03-19T11:31:47.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041605399999999999 2024-03-19T11:31:47.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:47.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.169762+0000", 2024-03-19T11:31:47.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017231483999999998 2024-03-19T11:31:47.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:47.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.169807+0000", 2024-03-19T11:31:47.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5161999999999999e-05 2024-03-19T11:31:47.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:47.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.169840+0000", 2024-03-19T11:31:47.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2570000000000002e-05 2024-03-19T11:31:47.253 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:47.253 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:47.253 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:47.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.253 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPushReply(4.5 84/83 [PushReplyOp(4:a38b9453:::smithi03829856-37:head)])", 2024-03-19T11:31:47.253 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:03.084896+0000", 2024-03-19T11:31:47.253 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 44.094270023, 2024-03-19T11:31:47.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.02071104, 2024-03-19T11:31:47.253 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:47.254 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-19T11:31:47.254 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:47.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.254 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:47.254 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.084896+0000", 2024-03-19T11:31:47.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:47.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.254 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:47.254 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.084896+0000", 2024-03-19T11:31:47.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:47.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.254 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:47.254 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.084897+0000", 2024-03-19T11:31:47.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.686e-06 2024-03-19T11:31:47.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:47.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.084910+0000", 2024-03-19T11:31:47.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2948e-05 2024-03-19T11:31:47.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:47.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.084936+0000", 2024-03-19T11:31:47.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5982e-05 2024-03-19T11:31:47.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:47.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.084955+0000", 2024-03-19T11:31:47.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8416000000000001e-05 2024-03-19T11:31:47.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:47.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.102822+0000", 2024-03-19T11:31:47.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017866756000000001 2024-03-19T11:31:47.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:47.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.105607+0000", 2024-03-19T11:31:47.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027852520000000002 2024-03-19T11:31:47.256 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:47.256 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:47.257 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:47.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.257 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.4.0:540 4.18 e111/86 4:1d58db11:::smithi03829856-18:head v 111'153, mlcod=108'150)", 2024-03-19T11:31:47.257 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:31.238185+0000", 2024-03-19T11:31:47.257 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 15.940980659999999, 2024-03-19T11:31:47.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021779390999999999, 2024-03-19T11:31:47.257 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:47.257 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:47.257 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:47.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:47.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.238185+0000", 2024-03-19T11:31:47.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:47.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:47.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.238185+0000", 2024-03-19T11:31:47.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:47.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:47.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.238188+0000", 2024-03-19T11:31:47.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4739999999999999e-06 2024-03-19T11:31:47.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:47.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.240837+0000", 2024-03-19T11:31:47.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026495189999999999 2024-03-19T11:31:47.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:47.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.240867+0000", 2024-03-19T11:31:47.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0363000000000001e-05 2024-03-19T11:31:47.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:47.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.240885+0000", 2024-03-19T11:31:47.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7153e-05 2024-03-19T11:31:47.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:47.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.241029+0000", 2024-03-19T11:31:47.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014455499999999999 2024-03-19T11:31:47.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:47.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.241069+0000", 2024-03-19T11:31:47.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9409000000000002e-05 2024-03-19T11:31:47.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:47.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.259833+0000", 2024-03-19T11:31:47.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018764306000000001 2024-03-19T11:31:47.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:47.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.259964+0000", 2024-03-19T11:31:47.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000131612 2024-03-19T11:31:47.261 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:47.261 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:47.261 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:47.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.261 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3282 4.5 4:a0a50d04:::smithi03829856-13:head [read 0~2201684,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e112)", 2024-03-19T11:31:47.261 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:32.240600+0000", 2024-03-19T11:31:47.261 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.938565455999999, 2024-03-19T11:31:47.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.84623566699999997, 2024-03-19T11:31:47.261 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:47.261 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:47.261 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:47.261 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:47.261 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:47.261 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3282 2024-03-19T11:31:47.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.262 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:47.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.262 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:47.262 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240600+0000", 2024-03-19T11:31:47.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:47.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.262 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:47.262 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240600+0000", 2024-03-19T11:31:47.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:47.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.262 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:47.263 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240602+0000", 2024-03-19T11:31:47.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0700000000000001e-06 2024-03-19T11:31:47.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.263 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:47.263 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240611+0000", 2024-03-19T11:31:47.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.969e-06 2024-03-19T11:31:47.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.263 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:47.263 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240630+0000", 2024-03-19T11:31:47.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8253e-05 2024-03-19T11:31:47.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:47.264 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240642+0000", 2024-03-19T11:31:47.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2666e-05 2024-03-19T11:31:47.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.264 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:47.264 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.241488+0000", 2024-03-19T11:31:47.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000845252 2024-03-19T11:31:47.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.265 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:31:47.265 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.241503+0000", 2024-03-19T11:31:47.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5797000000000001e-05 2024-03-19T11:31:47.265 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.265 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.265 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:47.265 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.083186+0000", 2024-03-19T11:31:47.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.84168299599999996 2024-03-19T11:31:47.265 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.265 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.265 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:47.265 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.083618+0000", 2024-03-19T11:31:47.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000432046 2024-03-19T11:31:47.265 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.266 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:47.266 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.086836+0000", 2024-03-19T11:31:47.266 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032176180000000002 2024-03-19T11:31:47.266 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:47.266 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:47.266 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:47.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.266 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3283 4.5 4:a0a50d04:::smithi03829856-13:head [sparse-read 0~1151947] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e112)", 2024-03-19T11:31:47.267 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:32.240669+0000", 2024-03-19T11:31:47.267 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.938496455999999, 2024-03-19T11:31:47.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.84791886900000002, 2024-03-19T11:31:47.267 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:47.267 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:47.267 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:47.267 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:47.267 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:47.267 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3283 2024-03-19T11:31:47.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.267 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:47.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.267 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:47.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240669+0000", 2024-03-19T11:31:47.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:47.268 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.268 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.268 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:47.268 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240669+0000", 2024-03-19T11:31:47.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:47.268 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.268 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.268 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:47.268 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240670+0000", 2024-03-19T11:31:47.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8199999999999998e-07 2024-03-19T11:31:47.268 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.268 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.268 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:47.268 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240675+0000", 2024-03-19T11:31:47.269 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0849999999999996e-06 2024-03-19T11:31:47.269 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.269 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.269 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:47.269 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240688+0000", 2024-03-19T11:31:47.269 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2901000000000001e-05 2024-03-19T11:31:47.269 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.269 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.269 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:47.269 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240696+0000", 2024-03-19T11:31:47.269 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2910000000000007e-06 2024-03-19T11:31:47.269 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.269 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.269 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:47.270 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.241573+0000", 2024-03-19T11:31:47.270 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00087639100000000002 2024-03-19T11:31:47.270 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.270 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.270 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:31:47.270 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.241587+0000", 2024-03-19T11:31:47.270 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4467999999999999e-05 2024-03-19T11:31:47.270 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.270 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.270 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:47.270 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.087901+0000", 2024-03-19T11:31:47.270 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.84631403500000002 2024-03-19T11:31:47.270 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.270 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.271 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:47.271 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.088089+0000", 2024-03-19T11:31:47.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000187171 2024-03-19T11:31:47.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.271 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:47.271 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.088588+0000", 2024-03-19T11:31:47.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00049964499999999995 2024-03-19T11:31:47.271 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:47.271 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:47.271 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:47.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.272 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3284 4.5 4:a0a50d04:::smithi03829856-13:head [sparse-read 0~1151947] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e112)", 2024-03-19T11:31:47.272 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:32.240710+0000", 2024-03-19T11:31:47.272 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 14.938456159999999, 2024-03-19T11:31:47.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.84860797899999996, 2024-03-19T11:31:47.272 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:47.272 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:47.272 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:47.272 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:47.272 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:47.272 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3284 2024-03-19T11:31:47.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.272 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:47.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:47.273 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240710+0000", 2024-03-19T11:31:47.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:47.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:47.273 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240710+0000", 2024-03-19T11:31:47.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:47.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:47.273 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240710+0000", 2024-03-19T11:31:47.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9599999999999999e-07 2024-03-19T11:31:47.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.274 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.274 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:47.274 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240714+0000", 2024-03-19T11:31:47.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8480000000000003e-06 2024-03-19T11:31:47.274 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.274 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.274 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:47.274 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240726+0000", 2024-03-19T11:31:47.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.186e-05 2024-03-19T11:31:47.274 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.274 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.274 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:47.274 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240736+0000", 2024-03-19T11:31:47.275 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0040999999999999e-05 2024-03-19T11:31:47.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:47.275 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.241633+0000", 2024-03-19T11:31:47.275 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00089736299999999998 2024-03-19T11:31:47.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:31:47.275 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.241645+0000", 2024-03-19T11:31:47.275 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1600000000000001e-05 2024-03-19T11:31:47.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:47.275 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.088667+0000", 2024-03-19T11:31:47.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.84702242800000005 2024-03-19T11:31:47.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:47.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.088837+0000", 2024-03-19T11:31:47.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000169949 2024-03-19T11:31:47.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:47.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:47.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:47.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.089318+0000", 2024-03-19T11:31:47.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00048019399999999998 2024-03-19T11:31:47.276 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:47.276 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:47.276 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:47.276 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:47.277 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:47.277 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:47.277 INFO:tasks.rados.rados.0.smithi038.stdout:1670: expect (ObjNum 87 snap 8 seq_num 87) 2024-03-19T11:31:47.278 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-19T11:31:47.279 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:47.287+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:47.286 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-19T11:31:47.354 INFO:tasks.rados.rados.0.smithi038.stdout:1669: expect (ObjNum 294 snap 39 seq_num 294) 2024-03-19T11:31:47.379 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:47.387+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:47.407 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:47.407 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:31:47.407 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:31:47.407 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:47.422 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-19T11:31:47.431 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:31:47.431 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:31:47.448 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:31:47.449 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:31:47.465 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:31:47.465 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:31:47.480 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:47.487+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:47.482 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:31:47.482 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:31:47.500 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:31:47.500 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:31:47.517 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:31:47.517 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:31:47.535 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:31:47.535 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:31:47.546 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:47.546 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:31:47.546 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:31:47.546 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:31:47.546 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:47.555 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:31:47.555 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:31:47.561 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-19T11:31:47.565 INFO:tasks.rados.rados.0.smithi038.stdout:1671: seq_num 304 ranges {709328=613463,1989820=645099} 2024-03-19T11:31:47.574 INFO:tasks.rados.rados.0.smithi038.stdout:1671: writing smithi03829856-7 from 709328 to 1322791 tid 1 2024-03-19T11:31:47.581 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:47.591+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:47.582 INFO:tasks.rados.rados.0.smithi038.stdout:1671: writing smithi03829856-7 from 1989820 to 2634919 tid 2 2024-03-19T11:31:47.582 INFO:tasks.rados.rados.0.smithi038.stdout:1666: done (5 left) 2024-03-19T11:31:47.582 INFO:tasks.rados.rados.0.smithi038.stdout:1667: done (4 left) 2024-03-19T11:31:47.582 INFO:tasks.rados.rados.0.smithi038.stdout:1668: done (3 left) 2024-03-19T11:31:47.583 INFO:tasks.rados.rados.0.smithi038.stdout:1669: done (2 left) 2024-03-19T11:31:47.583 INFO:tasks.rados.rados.0.smithi038.stdout:1670: done (1 left) 2024-03-19T11:31:47.583 INFO:tasks.rados.rados.0.smithi038.stdout:1672: write (excl) oid 46 current snap is 39 2024-03-19T11:31:47.583 INFO:tasks.rados.rados.0.smithi038.stdout:1672: seq_num 305 ranges {726135=405065,1764655=788757,3026248=99887} 2024-03-19T11:31:47.588 INFO:tasks.rados.rados.0.smithi038.stdout:1672: writing smithi03829856-46 from 726135 to 1131200 tid 1 2024-03-19T11:31:47.596 INFO:tasks.rados.rados.0.smithi038.stdout:1672: writing smithi03829856-46 from 1764655 to 2553412 tid 2 2024-03-19T11:31:47.600 INFO:tasks.rados.rados.0.smithi038.stdout:1672: writing smithi03829856-46 from 3026248 to 3126135 tid 3 2024-03-19T11:31:47.600 INFO:tasks.rados.rados.0.smithi038.stdout:1673: set_chunk oid 31 target oid 2024-03-19T11:31:47.605 INFO:tasks.rados.rados.0.smithi038.stdout:1673: set_chunk oid 31 offset: 2121728 length: 173056 target oid 2411c534ee3196b097515d39a91ada69dc63fde41b05b0ab304fab263f60f2ef offset: 0 snap -1 2024-03-19T11:31:47.605 INFO:tasks.rados.rados.0.smithi038.stdout:1674: write (excl) oid 33 current snap is 39 2024-03-19T11:31:47.605 INFO:tasks.rados.rados.0.smithi038.stdout:1674: seq_num 306 ranges {742942=480314,1823137=648768} 2024-03-19T11:31:47.611 INFO:tasks.rados.rados.0.smithi038.stdout:1674: writing smithi03829856-33 from 742942 to 1223256 tid 1 2024-03-19T11:31:47.619 INFO:tasks.rados.rados.0.smithi038.stdout:1674: writing smithi03829856-33 from 1823137 to 2471905 tid 2 2024-03-19T11:31:47.620 INFO:tasks.rados.rados.0.smithi038.stdout:1675: read oid 15 snap -1 2024-03-19T11:31:47.620 INFO:tasks.rados.rados.0.smithi038.stdout:1675: expect (ObjNum 286 snap 37 seq_num 286) 2024-03-19T11:31:47.620 INFO:tasks.rados.rados.0.smithi038.stdout:1676: delete oid 19 current snap is 39 2024-03-19T11:31:47.620 INFO:tasks.rados.rados.0.smithi038.stdout:1671: finishing write tid 1 to smithi03829856-7 2024-03-19T11:31:47.620 INFO:tasks.rados.rados.0.smithi038.stdout:1671: finishing write tid 2 to smithi03829856-7 2024-03-19T11:31:47.620 INFO:tasks.rados.rados.0.smithi038.stdout:1671: finishing write tid 3 to smithi03829856-7 2024-03-19T11:31:47.620 INFO:tasks.rados.rados.0.smithi038.stdout:1671: finishing write tid 4 to smithi03829856-7 2024-03-19T11:31:47.620 INFO:tasks.rados.rados.0.smithi038.stdout:1671: oid 7 updating version 0 to 58 2024-03-19T11:31:47.620 INFO:tasks.rados.rados.0.smithi038.stdout:1671: oid 7 version 58 is already newer than 57 2024-03-19T11:31:47.620 INFO:tasks.rados.rados.0.smithi038.stdout:1671: oid 7 version 58 is already newer than 56 2024-03-19T11:31:47.620 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 7 v 58 (ObjNum 304 snap 39 seq_num 304) dirty exists 2024-03-19T11:31:47.620 INFO:tasks.rados.rados.0.smithi038.stdout:1671: left oid 7 (ObjNum 304 snap 39 seq_num 304) 2024-03-19T11:31:47.621 INFO:tasks.rados.rados.0.smithi038.stdout:1673: finishing set_chunk to oid 31 2024-03-19T11:31:47.621 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 31 v 177 (ObjNum 300 snap 39 seq_num 300) dirty exists 2024-03-19T11:31:47.621 INFO:tasks.rados.rados.0.smithi038.stdout:1674: finishing write tid 1 to smithi03829856-33 2024-03-19T11:31:47.624 INFO:tasks.rados.rados.0.smithi038.stdout:1674: finishing write tid 2 to smithi03829856-33 2024-03-19T11:31:47.625 INFO:tasks.rados.rados.0.smithi038.stdout:1674: finishing write tid 3 to smithi03829856-33 2024-03-19T11:31:47.625 INFO:tasks.rados.rados.0.smithi038.stdout:1674: finishing write tid 4 to smithi03829856-33 2024-03-19T11:31:47.626 INFO:tasks.rados.rados.0.smithi038.stdout:1674: oid 33 updating version 0 to 209 2024-03-19T11:31:47.626 INFO:tasks.rados.rados.0.smithi038.stdout:1674: oid 33 version 209 is already newer than 208 2024-03-19T11:31:47.626 INFO:tasks.rados.rados.0.smithi038.stdout:1674: oid 33 updating version 209 to 210 2024-03-19T11:31:47.626 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 33 v 210 (ObjNum 306 snap 39 seq_num 306) dirty exists 2024-03-19T11:31:47.626 INFO:tasks.rados.rados.0.smithi038.stdout:1674: left oid 33 (ObjNum 306 snap 39 seq_num 306) 2024-03-19T11:31:47.627 INFO:tasks.rados.rados.0.smithi038.stdout:1671: done (5 left) 2024-03-19T11:31:47.627 INFO:tasks.rados.rados.0.smithi038.stdout:1673: done (4 left) 2024-03-19T11:31:47.627 INFO:tasks.rados.rados.0.smithi038.stdout:1674: done (3 left) 2024-03-19T11:31:47.627 INFO:tasks.rados.rados.0.smithi038.stdout:1676: done (2 left) 2024-03-19T11:31:47.627 INFO:tasks.rados.rados.0.smithi038.stdout:1677: tier_evict oid 30 2024-03-19T11:31:47.627 INFO:tasks.rados.rados.0.smithi038.stdout:1677: tier_evict oid 30 snap -1 2024-03-19T11:31:47.627 INFO:tasks.rados.rados.0.smithi038.stdout:1678: set_chunk oid 27 target oid 2024-03-19T11:31:47.639 INFO:tasks.rados.rados.0.smithi038.stdout:1678: set_chunk oid 27 offset: 862720 length: 976896 target oid e5d9d693bcadde5c5a9fc7d16c3e119039994a63455ef1bd1da8cb6ef9094378 offset: 0 snap -1 2024-03-19T11:31:47.639 INFO:tasks.rados.rados.0.smithi038.stdout:1679: set_chunk oid 24 target oid 2024-03-19T11:31:47.649 INFO:tasks.rados.rados.0.smithi038.stdout:1679: set_chunk oid 24 offset: 105472 length: 1308160 target oid b5e900721010c1b2fd758b09bdbd843dc37a0985a42149e933acded10b20a53d offset: 0 snap -1 2024-03-19T11:31:47.649 INFO:tasks.rados.rados.0.smithi038.stdout:1680: write (excl) oid 9 current snap is 39 2024-03-19T11:31:47.650 INFO:tasks.rados.rados.0.smithi038.stdout:1677: got (22) Invalid argument 2024-03-19T11:31:47.650 INFO:tasks.rados.rados.0.smithi038.stdout:1680: seq_num 308 ranges {776556=630812,1823748=652437,3108517=68039} 2024-03-19T11:31:47.661 INFO:tasks.rados.rados.0.smithi038.stdout:1680: writing smithi03829856-9 from 776556 to 1407368 tid 1 2024-03-19T11:31:47.670 INFO:tasks.rados.rados.0.smithi038.stdout:1680: writing smithi03829856-9 from 1823748 to 2476185 tid 2 2024-03-19T11:31:47.676 INFO:tasks.rados.rados.0.smithi038.stdout:1680: writing smithi03829856-9 from 3108517 to 3176556 tid 3 2024-03-19T11:31:47.676 INFO:tasks.rados.rados.0.smithi038.stdout:1677: done (5 left) 2024-03-19T11:31:47.676 INFO:tasks.rados.rados.0.smithi038.stdout:1681: set_chunk oid 28 target oid 2024-03-19T11:31:47.681 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:47.689+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:47.683 INFO:tasks.rados.rados.0.smithi038.stdout:1681: set_chunk oid 28 offset: 3153408 length: 263168 target oid 6cd5a32322d0e7e5229cb4016e5cdce487c2a7887ec765ca69a9bb462a79ef13 offset: 0 snap -1 2024-03-19T11:31:47.683 INFO:tasks.rados.rados.0.smithi038.stdout:1682: read oid 40 snap -1 2024-03-19T11:31:47.683 INFO:tasks.rados.rados.0.smithi038.stdout:1682: expect (ObjNum 281 snap 37 seq_num 281) 2024-03-19T11:31:47.683 INFO:tasks.rados.rados.0.smithi038.stdout:1683: read oid 22 snap -1 2024-03-19T11:31:47.683 INFO:tasks.rados.rados.0.smithi038.stdout:1683: expect (ObjNum 295 snap 39 seq_num 295) 2024-03-19T11:31:47.683 INFO:tasks.rados.rados.0.smithi038.stdout:1684: tier_evict oid 1 2024-03-19T11:31:47.683 INFO:tasks.rados.rados.0.smithi038.stdout:1684: tier_evict oid 1 snap -1 2024-03-19T11:31:47.683 INFO:tasks.rados.rados.0.smithi038.stdout:1685: write (excl) oid 25 current snap is 39 2024-03-19T11:31:47.683 INFO:tasks.rados.rados.0.smithi038.stdout:1685: seq_num 309 ranges {793363=706061,2282230=796095} 2024-03-19T11:31:47.690 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:47.690 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:47.695 INFO:tasks.rados.rados.0.smithi038.stdout:1685: writing smithi03829856-25 from 793363 to 1499424 tid 1 2024-03-19T11:31:47.707 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-19T11:31:47.708 INFO:tasks.rados.rados.0.smithi038.stdout:1685: writing smithi03829856-25 from 2282230 to 3078325 tid 2 2024-03-19T11:31:47.708 INFO:tasks.rados.rados.0.smithi038.stdout:1686: write oid 5 current snap is 39 2024-03-19T11:31:47.708 INFO:tasks.rados.rados.0.smithi038.stdout:1675: expect (ObjNum 286 snap 37 seq_num 286) 2024-03-19T11:31:47.783 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:47.789+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:47.829 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:47.829 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:47.843 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-19T11:31:47.888 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:47.899+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:47.952 INFO:tasks.rados.rados.0.smithi038.stdout:1678: finishing set_chunk to oid 27 2024-03-19T11:31:47.952 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 27 set_chunk returned error code -95 offset: 862720 length: 976896 2024-03-19T11:31:47.952 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 27 v 0 (ObjNum 240 snap 30 seq_num 240) dirty exists 2024-03-19T11:31:47.952 INFO:tasks.rados.rados.0.smithi038.stdout:1679: finishing set_chunk to oid 24 2024-03-19T11:31:47.952 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 24 set_chunk returned error code -95 offset: 105472 length: 1308160 2024-03-19T11:31:47.952 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 24 v 0 (ObjNum 265 snap 34 seq_num 265) dirty exists 2024-03-19T11:31:47.952 INFO:tasks.rados.rados.0.smithi038.stdout:1680: finishing write tid 1 to smithi03829856-9 2024-03-19T11:31:47.952 INFO:tasks.rados.rados.0.smithi038.stdout:1680: finishing write tid 2 to smithi03829856-9 2024-03-19T11:31:47.952 INFO:tasks.rados.rados.0.smithi038.stdout:1680: finishing write tid 3 to smithi03829856-9 2024-03-19T11:31:47.953 INFO:tasks.rados.rados.0.smithi038.stdout:1680: finishing write tid 4 to smithi03829856-9 2024-03-19T11:31:47.953 INFO:tasks.rados.rados.0.smithi038.stdout:1680: finishing write tid 5 to smithi03829856-9 2024-03-19T11:31:47.953 INFO:tasks.rados.rados.0.smithi038.stdout:1680: oid 9 updating version 0 to 131 2024-03-19T11:31:47.953 INFO:tasks.rados.rados.0.smithi038.stdout:1680: oid 9 version 131 is already newer than 130 2024-03-19T11:31:47.953 INFO:tasks.rados.rados.0.smithi038.stdout:1680: oid 9 version 131 is already newer than 129 2024-03-19T11:31:47.953 INFO:tasks.rados.rados.0.smithi038.stdout:1680: oid 9 updating version 131 to 132 2024-03-19T11:31:47.953 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 9 v 132 (ObjNum 308 snap 39 seq_num 308) dirty exists 2024-03-19T11:31:47.953 INFO:tasks.rados.rados.0.smithi038.stdout:1680: left oid 9 (ObjNum 308 snap 39 seq_num 308) 2024-03-19T11:31:47.953 INFO:tasks.rados.rados.0.smithi038.stdout:1684: got (22) Invalid argument 2024-03-19T11:31:47.953 INFO:tasks.rados.rados.0.smithi038.stdout:1681: finishing set_chunk to oid 28 2024-03-19T11:31:47.953 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 28 v 27 (ObjNum 87 snap 8 seq_num 87) dirty exists 2024-03-19T11:31:47.953 INFO:tasks.rados.rados.0.smithi038.stdout:1682: expect (ObjNum 281 snap 37 seq_num 281) 2024-03-19T11:31:47.962 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:47.963 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:47.977 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-19T11:31:48.021 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:48.031+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:48.086 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:48.093+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:48.121 INFO:tasks.rados.rados.0.smithi038.stdout:1683: expect (ObjNum 295 snap 39 seq_num 295) 2024-03-19T11:31:48.122 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-03-19T11:31:48.138 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:48.139 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:48.152 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-19T11:31:48.197 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:48.207+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:48.288 INFO:tasks.rados.rados.0.smithi038.stdout:1685: finishing write tid 1 to smithi03829856-25 2024-03-19T11:31:48.288 INFO:tasks.rados.rados.0.smithi038.stdout:1685: finishing write tid 2 to smithi03829856-25 2024-03-19T11:31:48.288 INFO:tasks.rados.rados.0.smithi038.stdout:1685: finishing write tid 3 to smithi03829856-25 2024-03-19T11:31:48.288 INFO:tasks.rados.rados.0.smithi038.stdout:1685: finishing write tid 4 to smithi03829856-25 2024-03-19T11:31:48.288 INFO:tasks.rados.rados.0.smithi038.stdout:1685: oid 25 updating version 0 to 147 2024-03-19T11:31:48.288 INFO:tasks.rados.rados.0.smithi038.stdout:1685: oid 25 updating version 147 to 148 2024-03-19T11:31:48.288 INFO:tasks.rados.rados.0.smithi038.stdout:1685: oid 25 updating version 148 to 149 2024-03-19T11:31:48.288 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 25 v 149 (ObjNum 309 snap 39 seq_num 309) dirty exists 2024-03-19T11:31:48.288 INFO:tasks.rados.rados.0.smithi038.stdout:1685: left oid 25 (ObjNum 309 snap 39 seq_num 309) 2024-03-19T11:31:48.289 INFO:tasks.rados.rados.0.smithi038.stdout:1686: seq_num 310 ranges {410170=781310,1940712=656106,3163363=46807} 2024-03-19T11:31:48.289 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:48.297+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:48.305 INFO:tasks.rados.rados.0.smithi038.stdout:1686: writing smithi03829856-5 from 410170 to 1191480 tid 1 2024-03-19T11:31:48.314 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:48.314 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:48.326 INFO:tasks.rados.rados.0.smithi038.stdout:1686: writing smithi03829856-5 from 1940712 to 2596818 tid 2 2024-03-19T11:31:48.328 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-19T11:31:48.333 INFO:tasks.rados.rados.0.smithi038.stdout:1686: writing smithi03829856-5 from 3163363 to 3210170 tid 3 2024-03-19T11:31:48.333 INFO:tasks.rados.rados.0.smithi038.stdout:1675: done (10 left) 2024-03-19T11:31:48.333 INFO:tasks.rados.rados.0.smithi038.stdout:1678: done (9 left) 2024-03-19T11:31:48.333 INFO:tasks.rados.rados.0.smithi038.stdout:1679: done (8 left) 2024-03-19T11:31:48.333 INFO:tasks.rados.rados.0.smithi038.stdout:1680: done (7 left) 2024-03-19T11:31:48.333 INFO:tasks.rados.rados.0.smithi038.stdout:1681: done (6 left) 2024-03-19T11:31:48.333 INFO:tasks.rados.rados.0.smithi038.stdout:1682: done (5 left) 2024-03-19T11:31:48.333 INFO:tasks.rados.rados.0.smithi038.stdout:1683: done (4 left) 2024-03-19T11:31:48.333 INFO:tasks.rados.rados.0.smithi038.stdout:1684: done (3 left) 2024-03-19T11:31:48.333 INFO:tasks.rados.rados.0.smithi038.stdout:1685: done (2 left) 2024-03-19T11:31:48.333 INFO:tasks.rados.rados.0.smithi038.stdout:1687: tier_evict oid 19 2024-03-19T11:31:48.333 INFO:tasks.rados.rados.0.smithi038.stdout:1687: tier_evict oid 19 snap -1 2024-03-19T11:31:48.333 INFO:tasks.rados.rados.0.smithi038.stdout:1686: finishing write tid 1 to smithi03829856-5 2024-03-19T11:31:48.334 INFO:tasks.rados.rados.0.smithi038.stdout:1686: finishing write tid 2 to smithi03829856-5 2024-03-19T11:31:48.334 INFO:tasks.rados.rados.0.smithi038.stdout:1688: set_chunk oid 3 target oid 2024-03-19T11:31:48.350 INFO:tasks.rados.rados.0.smithi038.stdout:1688: set_chunk oid 3 offset: 1113088 length: 476672 target oid 59bff1359e5fb6fca79c117a6cf8aa5f12349977bac48c48f13b0eba610d7b37 offset: 0 snap -1 2024-03-19T11:31:48.351 INFO:tasks.rados.rados.0.smithi038.stdout:1689: snap_create 2024-03-19T11:31:48.351 INFO:tasks.rados.rados.0.smithi038.stdout:1687: got (22) Invalid argument 2024-03-19T11:31:48.351 INFO:tasks.rados.rados.0.smithi038.stdout:1686: finishing write tid 3 to smithi03829856-5 2024-03-19T11:31:48.351 INFO:tasks.rados.rados.0.smithi038.stdout:1686: finishing write tid 4 to smithi03829856-5 2024-03-19T11:31:48.351 INFO:tasks.rados.rados.0.smithi038.stdout:1686: finishing write tid 5 to smithi03829856-5 2024-03-19T11:31:48.351 INFO:tasks.rados.rados.0.smithi038.stdout:1686: oid 5 updating version 0 to 215 2024-03-19T11:31:48.351 INFO:tasks.rados.rados.0.smithi038.stdout:1686: oid 5 version 215 is already newer than 213 2024-03-19T11:31:48.351 INFO:tasks.rados.rados.0.smithi038.stdout:1686: oid 5 version 215 is already newer than 214 2024-03-19T11:31:48.351 INFO:tasks.rados.rados.0.smithi038.stdout:1686: oid 5 version 215 is already newer than 212 2024-03-19T11:31:48.351 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 5 v 215 (ObjNum 310 snap 39 seq_num 310) dirty exists 2024-03-19T11:31:48.351 INFO:tasks.rados.rados.0.smithi038.stdout:1686: left oid 5 (ObjNum 310 snap 39 seq_num 310) 2024-03-19T11:31:48.352 INFO:tasks.rados.rados.0.smithi038.stdout:1688: finishing set_chunk to oid 3 2024-03-19T11:31:48.352 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 3 set_chunk returned error code -95 offset: 1113088 length: 476672 2024-03-19T11:31:48.352 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 3 v 0 (ObjNum 271 snap 34 seq_num 271) dirty exists 2024-03-19T11:31:48.389 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:48.399+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:48.440 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:48.440 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:48.453 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-19T11:31:48.491 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:48.497+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:48.591 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:48.601+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:48.620 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:48.620 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:48.634 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-19T11:31:48.660 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-19T11:31:48.660 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-19T11:31:48.660 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-19T11:31:48.660 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-03-19T11:31:48.692 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:48.701+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:48.754 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:48.754 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:48.769 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-19T11:31:48.817 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:48.827+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:48.894 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:48.903+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:48.896 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:48.896 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:48.910 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-19T11:31:48.980 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:31:48.981 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:48.981 INFO:teuthology.orchestra.run.smithi038.stdout: "epoch": 127, 2024-03-19T11:31:48.981 INFO:teuthology.orchestra.run.smithi038.stdout: "fsid": "246bcf68-c431-4aa7-b787-5e9b2d9519d6", 2024-03-19T11:31:48.981 INFO:teuthology.orchestra.run.smithi038.stdout: "created": "2024-03-19T11:29:33.327073+0000", 2024-03-19T11:31:48.981 INFO:teuthology.orchestra.run.smithi038.stdout: "modified": "2024-03-19T11:31:48.055624+0000", 2024-03-19T11:31:48.981 INFO:teuthology.orchestra.run.smithi038.stdout: "last_up_change": "2024-03-19T11:29:43.166618+0000", 2024-03-19T11:31:48.981 INFO:teuthology.orchestra.run.smithi038.stdout: "last_in_change": "2024-03-19T11:31:33.048666+0000", 2024-03-19T11:31:48.981 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-19T11:31:48.981 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_num": 5801984, 2024-03-19T11:31:48.981 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_set": [ 2024-03-19T11:31:48.981 INFO:teuthology.orchestra.run.smithi038.stdout: "noscrub", 2024-03-19T11:31:48.981 INFO:teuthology.orchestra.run.smithi038.stdout: "pglog_hardlimit", 2024-03-19T11:31:48.982 INFO:teuthology.orchestra.run.smithi038.stdout: "purged_snapdirs", 2024-03-19T11:31:48.982 INFO:teuthology.orchestra.run.smithi038.stdout: "recovery_deletes", 2024-03-19T11:31:48.982 INFO:teuthology.orchestra.run.smithi038.stdout: "sortbitwise" 2024-03-19T11:31:48.982 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-19T11:31:48.982 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_version": 5, 2024-03-19T11:31:48.982 INFO:teuthology.orchestra.run.smithi038.stdout: "full_ratio": 0.94999998807907104, 2024-03-19T11:31:48.982 INFO:teuthology.orchestra.run.smithi038.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-19T11:31:48.982 INFO:teuthology.orchestra.run.smithi038.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-19T11:31:48.982 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_snapshot": "", 2024-03-19T11:31:48.982 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_max": 4, 2024-03-19T11:31:48.982 INFO:teuthology.orchestra.run.smithi038.stdout: "max_osd": 8, 2024-03-19T11:31:48.983 INFO:teuthology.orchestra.run.smithi038.stdout: "require_min_compat_client": "reef", 2024-03-19T11:31:48.983 INFO:teuthology.orchestra.run.smithi038.stdout: "min_compat_client": "jewel", 2024-03-19T11:31:48.983 INFO:teuthology.orchestra.run.smithi038.stdout: "require_osd_release": "squid", 2024-03-19T11:31:48.983 INFO:teuthology.orchestra.run.smithi038.stdout: "allow_crimson": false, 2024-03-19T11:31:48.983 INFO:teuthology.orchestra.run.smithi038.stdout: "pools": [ 2024-03-19T11:31:48.983 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:48.983 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 1, 2024-03-19T11:31:48.983 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": ".mgr", 2024-03-19T11:31:48.983 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-19T11:29:42.219654+0000", 2024-03-19T11:31:48.984 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-19T11:31:48.984 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-03-19T11:31:48.984 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-19T11:31:48.984 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 2, 2024-03-19T11:31:48.984 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-19T11:31:48.984 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-19T11:31:48.984 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-19T11:31:48.984 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-19T11:31:48.984 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-19T11:31:48.984 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-19T11:31:48.985 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-19T11:31:48.985 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-19T11:31:48.985 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 1, 2024-03-19T11:31:48.985 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 1, 2024-03-19T11:31:48.985 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 1, 2024-03-19T11:31:48.985 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 1, 2024-03-19T11:31:48.985 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 1, 2024-03-19T11:31:48.985 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-19T11:31:48.986 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-19T11:31:48.986 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-19T11:31:48.986 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-19T11:31:48.986 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-19T11:31:48.986 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-19T11:31:48.986 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-19T11:31:48.986 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:48.986 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "16", 2024-03-19T11:31:48.986 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-19T11:31:48.986 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-19T11:31:48.986 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-19T11:31:48.986 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-19T11:31:48.986 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-19T11:31:48.986 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-03-19T11:31:48.987 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-03-19T11:31:48.987 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-19T11:31:48.987 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-19T11:31:48.987 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-19T11:31:48.987 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-19T11:31:48.987 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-19T11:31:48.987 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-19T11:31:48.987 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-19T11:31:48.987 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-19T11:31:48.987 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-19T11:31:48.987 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-19T11:31:48.987 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-19T11:31:48.987 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-19T11:31:48.987 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-19T11:31:48.988 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-19T11:31:48.988 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-19T11:31:48.988 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-19T11:31:48.988 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-19T11:31:48.988 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-19T11:31:48.988 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-19T11:31:48.988 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:48.988 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-19T11:31:48.988 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-19T11:31:48.988 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-19T11:31:48.988 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-19T11:31:48.988 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-19T11:31:48.988 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-19T11:31:48.988 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-19T11:31:48.988 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-19T11:31:48.989 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-19T11:31:48.989 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-19T11:31:48.989 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-19T11:31:48.989 INFO:teuthology.orchestra.run.smithi038.stdout: "options": { 2024-03-19T11:31:48.989 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_max": 32, 2024-03-19T11:31:48.989 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_min": 1 2024-03-19T11:31:48.989 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:48.989 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-19T11:31:48.989 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr": {} 2024-03-19T11:31:48.989 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:48.989 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-19T11:31:48.989 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-19T11:31:48.990 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 8, 2024-03-19T11:31:48.990 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 8, 2024-03-19T11:31:48.990 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.25, 2024-03-19T11:31:48.990 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 2, 2024-03-19T11:31:48.990 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 2, 2024-03-19T11:31:48.990 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 1, 2024-03-19T11:31:48.990 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 1, 2024-03-19T11:31:48.990 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 1 2024-03-19T11:31:48.990 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:48.990 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:48.990 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:48.990 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 2, 2024-03-19T11:31:48.990 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": "rbd", 2024-03-19T11:31:48.991 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-19T11:29:44.428406+0000", 2024-03-19T11:31:48.991 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-19T11:31:48.991 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-03-19T11:31:48.991 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-19T11:31:48.991 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 2, 2024-03-19T11:31:48.991 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-19T11:31:48.991 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-19T11:31:48.991 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-19T11:31:48.991 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-19T11:31:48.991 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-19T11:31:48.991 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-19T11:31:48.991 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-19T11:31:48.991 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-19T11:31:48.992 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 8, 2024-03-19T11:31:48.992 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 8, 2024-03-19T11:31:48.992 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 8, 2024-03-19T11:31:48.992 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 8, 2024-03-19T11:31:48.992 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 8, 2024-03-19T11:31:48.992 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-19T11:31:48.992 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-19T11:31:48.992 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-19T11:31:48.992 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-19T11:31:48.992 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-19T11:31:48.992 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-19T11:31:48.992 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-19T11:31:48.992 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:48.992 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "18", 2024-03-19T11:31:48.993 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-19T11:31:48.993 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-19T11:31:48.993 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-19T11:31:48.993 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-19T11:31:48.993 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-19T11:31:48.993 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-03-19T11:31:48.993 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-03-19T11:31:48.993 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-19T11:31:48.993 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-19T11:31:48.993 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-19T11:31:48.993 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-19T11:31:48.993 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-19T11:31:48.994 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-19T11:31:48.994 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-19T11:31:48.994 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-19T11:31:48.994 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-19T11:31:48.994 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-19T11:31:48.994 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-19T11:31:48.994 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-19T11:31:48.994 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-19T11:31:48.994 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-19T11:31:48.994 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-19T11:31:48.994 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-19T11:31:48.994 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-19T11:31:48.994 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-19T11:31:48.995 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-19T11:31:48.995 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:48.995 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-19T11:31:48.995 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-19T11:31:48.995 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-19T11:31:48.995 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-19T11:31:48.995 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-19T11:31:48.995 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-19T11:31:48.995 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-19T11:31:48.995 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-19T11:31:48.995 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-19T11:31:48.995 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-19T11:31:48.995 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-19T11:31:48.996 INFO:teuthology.orchestra.run.smithi038.stdout: "options": {}, 2024-03-19T11:31:48.996 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-19T11:31:48.996 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd": {} 2024-03-19T11:31:48.996 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:48.996 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-19T11:31:48.996 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-19T11:31:48.996 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 2, 2024-03-19T11:31:48.996 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 2, 2024-03-19T11:31:48.996 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.75, 2024-03-19T11:31:48.996 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 1.5, 2024-03-19T11:31:48.996 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 1.5, 2024-03-19T11:31:48.997 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 1, 2024-03-19T11:31:48.997 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 1, 2024-03-19T11:31:48.997 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 1 2024-03-19T11:31:48.997 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:48.997 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:48.997 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:48.997 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 3, 2024-03-19T11:31:48.997 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": "low_tier", 2024-03-19T11:31:48.997 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-19T11:29:55.608699+0000", 2024-03-19T11:31:48.997 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-19T11:31:48.997 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-03-19T11:31:48.997 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-19T11:31:48.997 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 2, 2024-03-19T11:31:48.997 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-19T11:31:48.998 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-19T11:31:48.998 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-19T11:31:48.998 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-19T11:31:48.998 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-19T11:31:48.998 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-19T11:31:48.998 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-19T11:31:48.998 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-19T11:31:48.998 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 4, 2024-03-19T11:31:48.999 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 4, 2024-03-19T11:31:48.999 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 4, 2024-03-19T11:31:48.999 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 4, 2024-03-19T11:31:48.999 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 4, 2024-03-19T11:31:48.999 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-19T11:31:48.999 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-19T11:31:48.999 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-19T11:31:48.999 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-19T11:31:48.999 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-19T11:31:48.999 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-19T11:31:48.999 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-19T11:31:48.999 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:48.999 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "22", 2024-03-19T11:31:48.999 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-19T11:31:49.000 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-19T11:31:49.000 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-19T11:31:49.000 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-19T11:31:49.000 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-19T11:31:49.000 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-03-19T11:31:49.000 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-03-19T11:31:49.000 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-19T11:31:49.000 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-19T11:31:49.000 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-19T11:31:49.000 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-19T11:31:49.000 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-19T11:31:49.000 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-19T11:31:49.000 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-19T11:31:49.001 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-19T11:31:49.001 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-19T11:31:49.001 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-19T11:31:49.001 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-19T11:31:49.001 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-19T11:31:49.001 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-19T11:31:49.001 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-19T11:31:49.001 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-19T11:31:49.001 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-19T11:31:49.001 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-19T11:31:49.001 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-19T11:31:49.002 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-19T11:31:49.002 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.002 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-19T11:31:49.002 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-19T11:31:49.002 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-19T11:31:49.002 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-19T11:31:49.002 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-19T11:31:49.002 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-19T11:31:49.002 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-19T11:31:49.002 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-19T11:31:49.002 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-19T11:31:49.002 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-19T11:31:49.002 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-19T11:31:49.003 INFO:teuthology.orchestra.run.smithi038.stdout: "options": {}, 2024-03-19T11:31:49.003 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": {}, 2024-03-19T11:31:49.003 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-19T11:31:49.003 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-19T11:31:49.003 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 2, 2024-03-19T11:31:49.003 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 2, 2024-03-19T11:31:49.003 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.75, 2024-03-19T11:31:49.003 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 1.5, 2024-03-19T11:31:49.003 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 1.5, 2024-03-19T11:31:49.003 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 1, 2024-03-19T11:31:49.003 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 1, 2024-03-19T11:31:49.003 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 1 2024-03-19T11:31:49.003 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:49.003 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.004 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.004 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 4, 2024-03-19T11:31:49.004 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": "unique_pool_0", 2024-03-19T11:31:49.004 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-19T11:29:56.945423+0000", 2024-03-19T11:31:49.004 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 8193, 2024-03-19T11:31:49.004 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-19T11:31:49.004 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-19T11:31:49.004 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 2, 2024-03-19T11:31:49.004 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-19T11:31:49.004 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-19T11:31:49.004 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-19T11:31:49.004 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-19T11:31:49.004 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-19T11:31:49.005 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-19T11:31:49.005 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-19T11:31:49.005 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-19T11:31:49.005 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 37, 2024-03-19T11:31:49.005 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 34, 2024-03-19T11:31:49.005 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 26, 2024-03-19T11:31:49.005 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 26, 2024-03-19T11:31:49.005 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 36, 2024-03-19T11:31:49.005 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-19T11:31:49.005 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "4.25", 2024-03-19T11:31:49.005 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 112, 2024-03-19T11:31:49.005 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 111, 2024-03-19T11:31:49.005 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 111, 2024-03-19T11:31:49.005 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "56'66", 2024-03-19T11:31:49.005 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "108'129" 2024-03-19T11:31:49.006 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.006 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "126", 2024-03-19T11:31:49.006 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-19T11:31:49.006 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "126", 2024-03-19T11:31:49.006 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "126", 2024-03-19T11:31:49.006 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-19T11:31:49.006 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-19T11:31:49.006 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 40, 2024-03-19T11:31:49.006 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 123, 2024-03-19T11:31:49.006 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-19T11:31:49.006 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-19T11:31:49.006 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-19T11:31:49.006 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-19T11:31:49.006 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-19T11:31:49.007 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-19T11:31:49.007 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-19T11:31:49.007 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-19T11:31:49.007 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-19T11:31:49.007 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-19T11:31:49.007 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-19T11:31:49.007 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-19T11:31:49.007 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-19T11:31:49.007 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-19T11:31:49.007 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-19T11:31:49.007 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-19T11:31:49.007 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-19T11:31:49.007 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-19T11:31:49.008 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-19T11:31:49.008 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.008 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-19T11:31:49.008 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-19T11:31:49.008 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-19T11:31:49.008 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-19T11:31:49.008 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-19T11:31:49.008 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-19T11:31:49.008 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-19T11:31:49.008 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-19T11:31:49.008 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-19T11:31:49.008 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-19T11:31:49.008 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-19T11:31:49.008 INFO:teuthology.orchestra.run.smithi038.stdout: "options": { 2024-03-19T11:31:49.008 INFO:teuthology.orchestra.run.smithi038.stdout: "dedup_cdc_chunk_size": 131072, 2024-03-19T11:31:49.009 INFO:teuthology.orchestra.run.smithi038.stdout: "dedup_chunk_algorithm": "fastcdc", 2024-03-19T11:31:49.009 INFO:teuthology.orchestra.run.smithi038.stdout: "dedup_tier": 3, 2024-03-19T11:31:49.009 INFO:teuthology.orchestra.run.smithi038.stdout: "fingerprint_algorithm": "sha256" 2024-03-19T11:31:49.009 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.009 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-19T11:31:49.009 INFO:teuthology.orchestra.run.smithi038.stdout: "rados": {} 2024-03-19T11:31:49.009 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.009 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-19T11:31:49.009 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-19T11:31:49.009 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 2.3599998950958252, 2024-03-19T11:31:49.009 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 2.3599998950958252, 2024-03-19T11:31:49.009 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.87999999523162842, 2024-03-19T11:31:49.009 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 2.0799999237060547, 2024-03-19T11:31:49.010 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 2.0799999237060547, 2024-03-19T11:31:49.010 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 1, 2024-03-19T11:31:49.010 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 1, 2024-03-19T11:31:49.010 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 1 2024-03-19T11:31:49.010 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:49.010 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:49.010 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-19T11:31:49.010 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-03-19T11:31:49.010 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.010 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 0, 2024-03-19T11:31:49.011 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "3bb2d2be-5253-4da0-acf3-85a114dc3801", 2024-03-19T11:31:49.011 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-19T11:31:49.011 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-19T11:31:49.011 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-19T11:31:49.011 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-19T11:31:49.011 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-19T11:31:49.011 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-19T11:31:49.011 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 13, 2024-03-19T11:31:49.011 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 122, 2024-03-19T11:31:49.011 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-19T11:31:49.011 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-19T11:31:49.011 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-19T11:31:49.011 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:49.011 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.011 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:49.012 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6808", 2024-03-19T11:31:49.012 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3148574891 2024-03-19T11:31:49.012 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.012 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.012 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:49.012 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6811", 2024-03-19T11:31:49.012 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3148574891 2024-03-19T11:31:49.012 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:49.012 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:49.012 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.012 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-19T11:31:49.012 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:49.012 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.012 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:49.012 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6813", 2024-03-19T11:31:49.013 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3148574891 2024-03-19T11:31:49.013 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.013 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.013 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:49.013 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6815", 2024-03-19T11:31:49.013 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3148574891 2024-03-19T11:31:49.013 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:49.013 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:49.013 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.013 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-19T11:31:49.013 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:49.013 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.013 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:49.013 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6822", 2024-03-19T11:31:49.014 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3148574891 2024-03-19T11:31:49.014 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.014 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.014 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:49.014 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6823", 2024-03-19T11:31:49.014 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3148574891 2024-03-19T11:31:49.014 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:49.014 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:49.014 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.014 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-19T11:31:49.014 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:49.014 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.014 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:49.015 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6818", 2024-03-19T11:31:49.015 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3148574891 2024-03-19T11:31:49.015 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.015 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.015 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:49.015 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6821", 2024-03-19T11:31:49.015 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3148574891 2024-03-19T11:31:49.015 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:49.015 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:49.015 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.016 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6811/3148574891", 2024-03-19T11:31:49.016 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6815/3148574891", 2024-03-19T11:31:49.016 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6823/3148574891", 2024-03-19T11:31:49.016 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6821/3148574891", 2024-03-19T11:31:49.016 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-19T11:31:49.016 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-19T11:31:49.016 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-19T11:31:49.016 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:49.016 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.016 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.016 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 1, 2024-03-19T11:31:49.016 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "e1c926b2-e0be-4ccc-b8a5-d80d0b584ead", 2024-03-19T11:31:49.016 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-19T11:31:49.017 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-19T11:31:49.017 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-19T11:31:49.017 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-19T11:31:49.017 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-19T11:31:49.017 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-19T11:31:49.017 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 14, 2024-03-19T11:31:49.017 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 126, 2024-03-19T11:31:49.017 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-19T11:31:49.017 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-19T11:31:49.017 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-19T11:31:49.018 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:49.018 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.018 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:49.018 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6824", 2024-03-19T11:31:49.018 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 675027622 2024-03-19T11:31:49.018 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.018 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.018 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:49.019 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6825", 2024-03-19T11:31:49.019 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 675027622 2024-03-19T11:31:49.019 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:49.019 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:49.019 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.019 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-19T11:31:49.019 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:49.019 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.019 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:49.019 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6826", 2024-03-19T11:31:49.019 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 675027622 2024-03-19T11:31:49.019 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.020 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.020 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:49.020 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6827", 2024-03-19T11:31:49.020 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 675027622 2024-03-19T11:31:49.020 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:49.020 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:49.020 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.020 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-19T11:31:49.020 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:49.020 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.020 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:49.020 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6830", 2024-03-19T11:31:49.020 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 675027622 2024-03-19T11:31:49.020 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.021 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.021 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:49.021 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6831", 2024-03-19T11:31:49.021 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 675027622 2024-03-19T11:31:49.021 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:49.021 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:49.021 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.021 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-19T11:31:49.021 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:49.021 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.021 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:49.021 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6828", 2024-03-19T11:31:49.021 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 675027622 2024-03-19T11:31:49.021 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.021 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.022 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:49.022 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6829", 2024-03-19T11:31:49.022 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 675027622 2024-03-19T11:31:49.022 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:49.022 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:49.022 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.022 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6825/675027622", 2024-03-19T11:31:49.022 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6827/675027622", 2024-03-19T11:31:49.022 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6831/675027622", 2024-03-19T11:31:49.022 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6829/675027622", 2024-03-19T11:31:49.022 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-19T11:31:49.022 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-19T11:31:49.022 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-19T11:31:49.022 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:49.022 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.023 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.023 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 2, 2024-03-19T11:31:49.023 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "61262219-4f8a-4733-af9d-b5208a17c0ea", 2024-03-19T11:31:49.023 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-19T11:31:49.023 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-19T11:31:49.023 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-19T11:31:49.023 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-19T11:31:49.023 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-19T11:31:49.023 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-19T11:31:49.023 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 14, 2024-03-19T11:31:49.023 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 113, 2024-03-19T11:31:49.023 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-19T11:31:49.023 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-19T11:31:49.023 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-19T11:31:49.023 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:49.024 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.024 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:49.024 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6800", 2024-03-19T11:31:49.024 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 58279616 2024-03-19T11:31:49.024 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.024 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.024 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:49.024 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6801", 2024-03-19T11:31:49.024 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 58279616 2024-03-19T11:31:49.024 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:49.024 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:49.024 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.024 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-19T11:31:49.024 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:49.024 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.025 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:49.025 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6802", 2024-03-19T11:31:49.025 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 58279616 2024-03-19T11:31:49.025 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.025 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.025 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:49.025 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6804", 2024-03-19T11:31:49.025 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 58279616 2024-03-19T11:31:49.025 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:49.025 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:49.025 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.025 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-19T11:31:49.025 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:49.026 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.026 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:49.026 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6814", 2024-03-19T11:31:49.026 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 58279616 2024-03-19T11:31:49.026 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.026 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.026 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:49.026 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6817", 2024-03-19T11:31:49.026 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 58279616 2024-03-19T11:31:49.026 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:49.026 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:49.026 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.026 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-19T11:31:49.026 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:49.027 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.027 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:49.027 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6807", 2024-03-19T11:31:49.027 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 58279616 2024-03-19T11:31:49.027 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.027 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.027 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:49.027 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6810", 2024-03-19T11:31:49.027 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 58279616 2024-03-19T11:31:49.027 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:49.027 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:49.027 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.027 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6801/58279616", 2024-03-19T11:31:49.027 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6804/58279616", 2024-03-19T11:31:49.027 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6817/58279616", 2024-03-19T11:31:49.028 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6810/58279616", 2024-03-19T11:31:49.028 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-19T11:31:49.028 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-19T11:31:49.028 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-19T11:31:49.028 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:49.028 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.028 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.028 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 3, 2024-03-19T11:31:49.028 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "a31f8d1b-4e38-4ca3-b251-0877002d7235", 2024-03-19T11:31:49.028 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-19T11:31:49.028 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 0, 2024-03-19T11:31:49.028 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 0, 2024-03-19T11:31:49.028 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-19T11:31:49.028 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-19T11:31:49.029 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-19T11:31:49.029 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 14, 2024-03-19T11:31:49.029 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 82, 2024-03-19T11:31:49.029 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-19T11:31:49.029 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-19T11:31:49.029 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-19T11:31:49.029 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:49.029 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.029 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:49.029 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6803", 2024-03-19T11:31:49.029 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2942163952 2024-03-19T11:31:49.029 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.029 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.029 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:49.030 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6805", 2024-03-19T11:31:49.030 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2942163952 2024-03-19T11:31:49.030 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:49.030 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:49.030 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.030 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-19T11:31:49.030 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:49.030 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.030 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:49.030 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6806", 2024-03-19T11:31:49.030 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2942163952 2024-03-19T11:31:49.030 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.030 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.030 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:49.030 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6809", 2024-03-19T11:31:49.031 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2942163952 2024-03-19T11:31:49.031 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:49.031 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:49.031 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.031 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-19T11:31:49.031 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:49.031 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.031 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:49.031 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6819", 2024-03-19T11:31:49.031 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2942163952 2024-03-19T11:31:49.031 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.031 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.031 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:49.031 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6820", 2024-03-19T11:31:49.031 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2942163952 2024-03-19T11:31:49.032 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:49.032 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:49.032 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.032 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-19T11:31:49.032 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:49.032 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.032 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:49.032 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6812", 2024-03-19T11:31:49.032 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2942163952 2024-03-19T11:31:49.032 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.032 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.032 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:49.032 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6816", 2024-03-19T11:31:49.032 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2942163952 2024-03-19T11:31:49.032 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:49.033 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:49.033 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.033 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6805/2942163952", 2024-03-19T11:31:49.033 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6809/2942163952", 2024-03-19T11:31:49.033 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6820/2942163952", 2024-03-19T11:31:49.033 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6816/2942163952", 2024-03-19T11:31:49.033 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-19T11:31:49.033 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-19T11:31:49.033 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-19T11:31:49.033 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:49.033 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.033 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.033 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 4, 2024-03-19T11:31:49.033 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "6746723d-f59c-40ba-857e-d8c954d2675f", 2024-03-19T11:31:49.034 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-19T11:31:49.034 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-19T11:31:49.034 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-19T11:31:49.034 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-19T11:31:49.034 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-19T11:31:49.034 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-19T11:31:49.034 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 13, 2024-03-19T11:31:49.034 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 116, 2024-03-19T11:31:49.034 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-19T11:31:49.034 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-19T11:31:49.034 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-19T11:31:49.034 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:49.034 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.034 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:49.034 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6808", 2024-03-19T11:31:49.035 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1353919713 2024-03-19T11:31:49.035 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.035 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.035 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:49.035 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6810", 2024-03-19T11:31:49.035 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1353919713 2024-03-19T11:31:49.035 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:49.035 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:49.035 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.035 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-19T11:31:49.035 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:49.035 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.036 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:49.036 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6812", 2024-03-19T11:31:49.036 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1353919713 2024-03-19T11:31:49.036 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.036 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.036 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:49.036 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6814", 2024-03-19T11:31:49.036 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1353919713 2024-03-19T11:31:49.036 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:49.036 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:49.036 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.036 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-19T11:31:49.036 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:49.036 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.037 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:49.037 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6819", 2024-03-19T11:31:49.037 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1353919713 2024-03-19T11:31:49.037 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.037 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.037 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:49.037 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6821", 2024-03-19T11:31:49.037 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1353919713 2024-03-19T11:31:49.037 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:49.037 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:49.037 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.037 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-19T11:31:49.037 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:49.037 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.037 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:49.038 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6816", 2024-03-19T11:31:49.038 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1353919713 2024-03-19T11:31:49.038 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.038 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.038 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:49.038 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6818", 2024-03-19T11:31:49.038 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1353919713 2024-03-19T11:31:49.038 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:49.038 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:49.038 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.038 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.172:6810/1353919713", 2024-03-19T11:31:49.038 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.172:6814/1353919713", 2024-03-19T11:31:49.038 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.172:6821/1353919713", 2024-03-19T11:31:49.039 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.172:6818/1353919713", 2024-03-19T11:31:49.039 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-19T11:31:49.039 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-19T11:31:49.039 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-19T11:31:49.039 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:49.039 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.039 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.039 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 5, 2024-03-19T11:31:49.039 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "f59afdf8-2277-4583-af68-78136f1c5d62", 2024-03-19T11:31:49.039 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-19T11:31:49.039 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-19T11:31:49.039 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-19T11:31:49.039 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-19T11:31:49.039 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-19T11:31:49.039 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-19T11:31:49.040 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 14, 2024-03-19T11:31:49.040 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 113, 2024-03-19T11:31:49.040 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-19T11:31:49.040 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-19T11:31:49.040 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-19T11:31:49.040 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:49.040 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.040 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:49.040 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6809", 2024-03-19T11:31:49.040 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2733060387 2024-03-19T11:31:49.040 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.040 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.040 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:49.040 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6811", 2024-03-19T11:31:49.041 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2733060387 2024-03-19T11:31:49.041 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:49.041 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:49.041 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.041 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-19T11:31:49.041 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:49.041 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.041 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:49.041 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6813", 2024-03-19T11:31:49.041 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2733060387 2024-03-19T11:31:49.041 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.041 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.041 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:49.041 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6815", 2024-03-19T11:31:49.041 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2733060387 2024-03-19T11:31:49.042 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:49.042 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:49.042 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.042 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-19T11:31:49.042 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:49.042 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.042 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:49.042 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6822", 2024-03-19T11:31:49.042 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2733060387 2024-03-19T11:31:49.042 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.042 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.042 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:49.042 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6823", 2024-03-19T11:31:49.042 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2733060387 2024-03-19T11:31:49.042 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:49.043 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:49.043 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.043 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-19T11:31:49.043 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:49.043 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.043 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:49.043 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6817", 2024-03-19T11:31:49.043 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2733060387 2024-03-19T11:31:49.043 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.043 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.043 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:49.043 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6820", 2024-03-19T11:31:49.043 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2733060387 2024-03-19T11:31:49.043 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:49.043 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:49.044 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.044 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.172:6811/2733060387", 2024-03-19T11:31:49.044 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.172:6815/2733060387", 2024-03-19T11:31:49.044 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.172:6823/2733060387", 2024-03-19T11:31:49.044 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.172:6820/2733060387", 2024-03-19T11:31:49.044 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-19T11:31:49.044 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-19T11:31:49.044 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-19T11:31:49.044 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:49.044 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.044 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.044 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 6, 2024-03-19T11:31:49.044 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "f91d8c47-4e10-40c6-a02d-66d6b72a657b", 2024-03-19T11:31:49.044 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-19T11:31:49.044 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-19T11:31:49.045 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-19T11:31:49.045 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-19T11:31:49.045 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-19T11:31:49.045 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-19T11:31:49.045 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 13, 2024-03-19T11:31:49.045 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 112, 2024-03-19T11:31:49.045 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-19T11:31:49.045 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-19T11:31:49.045 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-19T11:31:49.045 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:49.045 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.045 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:49.045 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6800", 2024-03-19T11:31:49.045 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 337743699 2024-03-19T11:31:49.046 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.046 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.046 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:49.046 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6801", 2024-03-19T11:31:49.046 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 337743699 2024-03-19T11:31:49.046 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:49.046 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:49.046 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.046 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-19T11:31:49.046 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:49.046 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.046 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:49.046 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6802", 2024-03-19T11:31:49.046 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 337743699 2024-03-19T11:31:49.047 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.047 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.047 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:49.047 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6803", 2024-03-19T11:31:49.047 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 337743699 2024-03-19T11:31:49.047 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:49.047 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:49.047 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.047 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-19T11:31:49.047 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:49.047 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.047 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:49.047 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6806", 2024-03-19T11:31:49.047 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 337743699 2024-03-19T11:31:49.047 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.048 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.048 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:49.048 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6807", 2024-03-19T11:31:49.048 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 337743699 2024-03-19T11:31:49.048 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:49.048 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:49.048 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.048 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-19T11:31:49.048 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:49.048 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.048 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:49.048 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6804", 2024-03-19T11:31:49.048 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 337743699 2024-03-19T11:31:49.048 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.048 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.049 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:49.049 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6805", 2024-03-19T11:31:49.049 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 337743699 2024-03-19T11:31:49.049 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:49.049 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:49.049 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.049 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.172:6801/337743699", 2024-03-19T11:31:49.049 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.172:6803/337743699", 2024-03-19T11:31:49.049 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.172:6807/337743699", 2024-03-19T11:31:49.049 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.172:6805/337743699", 2024-03-19T11:31:49.049 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-19T11:31:49.049 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-19T11:31:49.049 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-19T11:31:49.049 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:49.049 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.050 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.050 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 7, 2024-03-19T11:31:49.050 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "46fc052a-5708-42ff-9d5a-bed7a0fc457a", 2024-03-19T11:31:49.050 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-19T11:31:49.050 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-19T11:31:49.050 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-19T11:31:49.050 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-19T11:31:49.050 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-19T11:31:49.050 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-19T11:31:49.050 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 14, 2024-03-19T11:31:49.050 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 113, 2024-03-19T11:31:49.050 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-19T11:31:49.050 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-19T11:31:49.051 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-19T11:31:49.051 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:49.051 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.051 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:49.051 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6824", 2024-03-19T11:31:49.051 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 4145361100 2024-03-19T11:31:49.051 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.051 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.051 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:49.051 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6825", 2024-03-19T11:31:49.051 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 4145361100 2024-03-19T11:31:49.051 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:49.051 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:49.051 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.051 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-19T11:31:49.052 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:49.052 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.052 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:49.052 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6826", 2024-03-19T11:31:49.052 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 4145361100 2024-03-19T11:31:49.052 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.052 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.052 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:49.052 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6827", 2024-03-19T11:31:49.052 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 4145361100 2024-03-19T11:31:49.052 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:49.052 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:49.052 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.052 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-19T11:31:49.052 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:49.053 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.053 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:49.053 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6830", 2024-03-19T11:31:49.053 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 4145361100 2024-03-19T11:31:49.053 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.053 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.053 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:49.053 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6831", 2024-03-19T11:31:49.053 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 4145361100 2024-03-19T11:31:49.053 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:49.053 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:49.053 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.053 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-19T11:31:49.053 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-19T11:31:49.054 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.054 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-19T11:31:49.054 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6828", 2024-03-19T11:31:49.054 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 4145361100 2024-03-19T11:31:49.054 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.054 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.054 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-19T11:31:49.054 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.172:6829", 2024-03-19T11:31:49.054 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 4145361100 2024-03-19T11:31:49.054 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:49.054 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:49.054 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.054 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.172:6825/4145361100", 2024-03-19T11:31:49.054 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.172:6827/4145361100", 2024-03-19T11:31:49.055 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.172:6831/4145361100", 2024-03-19T11:31:49.055 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.172:6829/4145361100", 2024-03-19T11:31:49.055 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-19T11:31:49.055 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-19T11:31:49.055 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-19T11:31:49.055 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:49.055 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:49.055 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-19T11:31:49.055 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_xinfo": [ 2024-03-19T11:31:49.055 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.055 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 0, 2024-03-19T11:31:49.055 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-19T11:31:49.055 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-19T11:31:49.055 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-19T11:31:49.056 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-19T11:31:49.056 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-19T11:31:49.056 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-19T11:31:01.531687+0000", 2024-03-19T11:31:49.056 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-19T11:31:49.056 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.056 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.056 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 1, 2024-03-19T11:31:49.056 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-19T11:31:49.056 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-19T11:31:49.056 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-19T11:31:49.056 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-19T11:31:49.056 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-19T11:31:49.056 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-19T11:30:52.791546+0000", 2024-03-19T11:31:49.056 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-19T11:31:49.056 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.056 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.057 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 2, 2024-03-19T11:31:49.057 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-19T11:31:49.057 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-19T11:31:49.057 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-19T11:31:49.057 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-19T11:31:49.057 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-19T11:31:49.057 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-19T11:30:51.216737+0000", 2024-03-19T11:31:49.057 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-19T11:31:49.057 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.057 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.057 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 3, 2024-03-19T11:31:49.057 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-19T11:31:49.057 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-19T11:31:49.057 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-19T11:31:49.058 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-19T11:31:49.058 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 65536, 2024-03-19T11:31:49.058 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-19T11:31:06.780394+0000", 2024-03-19T11:31:49.058 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-19T11:31:49.058 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.058 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.058 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 4, 2024-03-19T11:31:49.058 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-19T11:31:49.058 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-19T11:31:49.058 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-19T11:31:49.058 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-19T11:31:49.058 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-19T11:31:49.058 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-19T11:31:06.311323+0000", 2024-03-19T11:31:49.058 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-19T11:31:49.059 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.059 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.059 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 5, 2024-03-19T11:31:49.059 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-19T11:31:49.059 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-19T11:31:49.059 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-19T11:31:49.059 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-19T11:31:49.059 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-19T11:31:49.059 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-19T11:30:59.034487+0000", 2024-03-19T11:31:49.059 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-19T11:31:49.059 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.060 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.060 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 6, 2024-03-19T11:31:49.060 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-19T11:31:49.060 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-19T11:31:49.060 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-19T11:31:49.060 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-19T11:31:49.060 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-19T11:31:49.060 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-19T11:31:04.628602+0000", 2024-03-19T11:31:49.060 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-19T11:31:49.060 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.060 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.060 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 7, 2024-03-19T11:31:49.061 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-19T11:31:49.061 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-19T11:31:49.061 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-19T11:31:49.061 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-19T11:31:49.061 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-19T11:31:49.061 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-19T11:31:09.716969+0000", 2024-03-19T11:31:49.061 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-19T11:31:49.061 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:49.061 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-19T11:31:49.061 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_upmap": [], 2024-03-19T11:31:49.061 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_upmap_items": [], 2024-03-19T11:31:49.061 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_upmap_primaries": [], 2024-03-19T11:31:49.061 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_temp": [], 2024-03-19T11:31:49.061 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_temp": [], 2024-03-19T11:31:49.061 INFO:teuthology.orchestra.run.smithi038.stdout: "blocklist": {}, 2024-03-19T11:31:49.062 INFO:teuthology.orchestra.run.smithi038.stdout: "range_blocklist": {}, 2024-03-19T11:31:49.062 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profiles": { 2024-03-19T11:31:49.062 INFO:teuthology.orchestra.run.smithi038.stdout: "default": { 2024-03-19T11:31:49.062 INFO:teuthology.orchestra.run.smithi038.stdout: "crush-failure-domain": "osd", 2024-03-19T11:31:49.062 INFO:teuthology.orchestra.run.smithi038.stdout: "k": "2", 2024-03-19T11:31:49.062 INFO:teuthology.orchestra.run.smithi038.stdout: "m": "1", 2024-03-19T11:31:49.062 INFO:teuthology.orchestra.run.smithi038.stdout: "plugin": "jerasure", 2024-03-19T11:31:49.062 INFO:teuthology.orchestra.run.smithi038.stdout: "technique": "reed_sol_van" 2024-03-19T11:31:49.062 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:49.062 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.063 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps_queue": [ 2024-03-19T11:31:49.063 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.063 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 4, 2024-03-19T11:31:49.063 INFO:teuthology.orchestra.run.smithi038.stdout: "snaps": [ 2024-03-19T11:31:49.063 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.063 INFO:teuthology.orchestra.run.smithi038.stdout: "begin": 27, 2024-03-19T11:31:49.063 INFO:teuthology.orchestra.run.smithi038.stdout: "length": 2 2024-03-19T11:31:49.063 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:49.063 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:49.063 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:49.063 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-19T11:31:49.063 INFO:teuthology.orchestra.run.smithi038.stdout: "new_removed_snaps": [], 2024-03-19T11:31:49.063 INFO:teuthology.orchestra.run.smithi038.stdout: "new_purged_snaps": [ 2024-03-19T11:31:49.063 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.063 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 4, 2024-03-19T11:31:49.064 INFO:teuthology.orchestra.run.smithi038.stdout: "snaps": [ 2024-03-19T11:31:49.064 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.064 INFO:teuthology.orchestra.run.smithi038.stdout: "begin": 24, 2024-03-19T11:31:49.064 INFO:teuthology.orchestra.run.smithi038.stdout: "length": 1 2024-03-19T11:31:49.064 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:49.064 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:49.064 INFO:teuthology.orchestra.run.smithi038.stdout: "begin": 32, 2024-03-19T11:31:49.064 INFO:teuthology.orchestra.run.smithi038.stdout: "length": 1 2024-03-19T11:31:49.064 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:49.064 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:49.064 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:49.064 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-19T11:31:49.064 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_node_flags": {}, 2024-03-19T11:31:49.064 INFO:teuthology.orchestra.run.smithi038.stdout: "device_class_flags": {}, 2024-03-19T11:31:49.064 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode": { 2024-03-19T11:31:49.065 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode_enabled": false, 2024-03-19T11:31:49.065 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_bucket_count": 0, 2024-03-19T11:31:49.065 INFO:teuthology.orchestra.run.smithi038.stdout: "degraded_stretch_mode": 0, 2024-03-19T11:31:49.065 INFO:teuthology.orchestra.run.smithi038.stdout: "recovering_stretch_mode": 0, 2024-03-19T11:31:49.065 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode_bucket": 0 2024-03-19T11:31:49.065 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:49.065 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:49.066 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:49.073+0000 7f143d8b8640 -1 osd.5 127 handle_osd_map injecting map crc failure 2024-03-19T11:31:49.068 INFO:tasks.thrashosds.thrasher:j is {'epoch': 127, 'fsid': '246bcf68-c431-4aa7-b787-5e9b2d9519d6', 'created': '2024-03-19T11:29:33.327073+0000', 'modified': '2024-03-19T11:31:48.055624+0000', 'last_up_change': '2024-03-19T11:29:43.166618+0000', 'last_in_change': '2024-03-19T11:31:33.048666+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 8, 'require_min_compat_client': 'reef', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T11:29:42.219654+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-03-19T11:29:44.428406+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 0.75, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'low_tier', 'create_time': '2024-03-19T11:29:55.608699+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 0.75, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'unique_pool_0', 'create_time': '2024-03-19T11:29:56.945423+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 37, 'pg_placement_num': 34, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 36, 'last_pg_merge_meta': {'source_pgid': '4.25', 'ready_epoch': 112, 'last_epoch_started': 111, 'last_epoch_clean': 111, 'source_version': "56'66", 'target_version': "108'129"}, 'last_change': '126', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '126', 'last_force_op_resend_preluminous': '126', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 40, 'snap_epoch': 123, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'dedup_cdc_chunk_size': 131072, 'dedup_chunk_algorithm': 'fastcdc', 'dedup_tier': 3, 'fingerprint_algorithm': 'sha256'}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.359999895095825, 'score_stable': 2.359999895095825, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.0799999237060547, 'raw_score_stable': 2.0799999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '3bb2d2be-5253-4da0-acf3-85a114dc3801', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 122, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6808', 'nonce': 3148574891}, {'type': 'v1', 'addr': '172.21.15.38:6811', 'nonce': 3148574891}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6813', 'nonce': 3148574891}, {'type': 'v1', 'addr': '172.21.15.38:6815', 'nonce': 3148574891}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6822', 'nonce': 3148574891}, {'type': 'v1', 'addr': '172.21.15.38:6823', 'nonce': 3148574891}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6818', 'nonce': 3148574891}, {'type': 'v1', 'addr': '172.21.15.38:6821', 'nonce': 3148574891}]}, 'public_addr': '172.21.15.38:6811/3148574891', 'cluster_addr': '172.21.15.38:6815/3148574891', 'heartbeat_back_addr': '172.21.15.38:6823/3148574891', 'heartbeat_front_addr': '172.21.15.38:6821/3148574891', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'e1c926b2-e0be-4ccc-b8a5-d80d0b584ead', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 126, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6824', 'nonce': 675027622}, {'type': 'v1', 'addr': '172.21.15.38:6825', 'nonce': 675027622}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6826', 'nonce': 675027622}, {'type': 'v1', 'addr': '172.21.15.38:6827', 'nonce': 675027622}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6830', 'nonce': 675027622}, {'type': 'v1', 'addr': '172.21.15.38:6831', 'nonce': 675027622}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6828', 'nonce': 675027622}, {'type': 'v1', 'addr': '172.21.15.38:6829', 'nonce': 675027622}]}, 'public_addr': '172.21.15.38:6825/675027622', 'cluster_addr': '172.21.15.38:6827/675027622', 'heartbeat_back_addr': '172.21.15.38:6831/675027622', 'heartbeat_front_addr': '172.21.15.38:6829/675027622', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '61262219-4f8a-4733-af9d-b5208a17c0ea', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 113, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6800', 'nonce': 58279616}, {'type': 'v1', 'addr': '172.21.15.38:6801', 'nonce': 58279616}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6802', 'nonce': 58279616}, {'type': 'v1', 'addr': '172.21.15.38:6804', 'nonce': 58279616}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6814', 'nonce': 58279616}, {'type': 'v1', 'addr': '172.21.15.38:6817', 'nonce': 58279616}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6807', 'nonce': 58279616}, {'type': 'v1', 'addr': '172.21.15.38:6810', 'nonce': 58279616}]}, 'public_addr': '172.21.15.38:6801/58279616', 'cluster_addr': '172.21.15.38:6804/58279616', 'heartbeat_back_addr': '172.21.15.38:6817/58279616', 'heartbeat_front_addr': '172.21.15.38:6810/58279616', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'a31f8d1b-4e38-4ca3-b251-0877002d7235', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 82, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6803', 'nonce': 2942163952}, {'type': 'v1', 'addr': '172.21.15.38:6805', 'nonce': 2942163952}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6806', 'nonce': 2942163952}, {'type': 'v1', 'addr': '172.21.15.38:6809', 'nonce': 2942163952}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6819', 'nonce': 2942163952}, {'type': 'v1', 'addr': '172.21.15.38:6820', 'nonce': 2942163952}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6812', 'nonce': 2942163952}, {'type': 'v1', 'addr': '172.21.15.38:6816', 'nonce': 2942163952}]}, 'public_addr': '172.21.15.38:6805/2942163952', 'cluster_addr': '172.21.15.38:6809/2942163952', 'heartbeat_back_addr': '172.21.15.38:6820/2942163952', 'heartbeat_front_addr': '172.21.15.38:6816/2942163952', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '6746723d-f59c-40ba-857e-d8c954d2675f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 116, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6808', 'nonce': 1353919713}, {'type': 'v1', 'addr': '172.21.15.172:6810', 'nonce': 1353919713}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6812', 'nonce': 1353919713}, {'type': 'v1', 'addr': '172.21.15.172:6814', 'nonce': 1353919713}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6819', 'nonce': 1353919713}, {'type': 'v1', 'addr': '172.21.15.172:6821', 'nonce': 1353919713}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6816', 'nonce': 1353919713}, {'type': 'v1', 'addr': '172.21.15.172:6818', 'nonce': 1353919713}]}, 'public_addr': '172.21.15.172:6810/1353919713', 'cluster_addr': '172.21.15.172:6814/1353919713', 'heartbeat_back_addr': '172.21.15.172:6821/1353919713', 'heartbeat_front_addr': '172.21.15.172:6818/1353919713', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'f59afdf8-2277-4583-af68-78136f1c5d62', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 113, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6809', 'nonce': 2733060387}, {'type': 'v1', 'addr': '172.21.15.172:6811', 'nonce': 2733060387}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6813', 'nonce': 2733060387}, {'type': 'v1', 'addr': '172.21.15.172:6815', 'nonce': 2733060387}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6822', 'nonce': 2733060387}, {'type': 'v1', 'addr': '172.21.15.172:6823', 'nonce': 2733060387}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6817', 'nonce': 2733060387}, {'type': 'v1', 'addr': '172.21.15.172:6820', 'nonce': 2733060387}]}, 'public_addr': '172.21.15.172:6811/2733060387', 'cluster_addr': '172.21.15.172:6815/2733060387', 'heartbeat_back_addr': '172.21.15.172:6823/2733060387', 'heartbeat_front_addr': '172.21.15.172:6820/2733060387', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'f91d8c47-4e10-40c6-a02d-66d6b72a657b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 112, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6800', 'nonce': 337743699}, {'type': 'v1', 'addr': '172.21.15.172:6801', 'nonce': 337743699}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6802', 'nonce': 337743699}, {'type': 'v1', 'addr': '172.21.15.172:6803', 'nonce': 337743699}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6806', 'nonce': 337743699}, {'type': 'v1', 'addr': '172.21.15.172:6807', 'nonce': 337743699}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6804', 'nonce': 337743699}, {'type': 'v1', 'addr': '172.21.15.172:6805', 'nonce': 337743699}]}, 'public_addr': '172.21.15.172:6801/337743699', 'cluster_addr': '172.21.15.172:6803/337743699', 'heartbeat_back_addr': '172.21.15.172:6807/337743699', 'heartbeat_front_addr': '172.21.15.172:6805/337743699', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '46fc052a-5708-42ff-9d5a-bed7a0fc457a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 14, 'up_thru': 113, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6824', 'nonce': 4145361100}, {'type': 'v1', 'addr': '172.21.15.172:6825', 'nonce': 4145361100}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6826', 'nonce': 4145361100}, {'type': 'v1', 'addr': '172.21.15.172:6827', 'nonce': 4145361100}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6830', 'nonce': 4145361100}, {'type': 'v1', 'addr': '172.21.15.172:6831', 'nonce': 4145361100}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.172:6828', 'nonce': 4145361100}, {'type': 'v1', 'addr': '172.21.15.172:6829', 'nonce': 4145361100}]}, 'public_addr': '172.21.15.172:6825/4145361100', 'cluster_addr': '172.21.15.172:6827/4145361100', 'heartbeat_back_addr': '172.21.15.172:6831/4145361100', 'heartbeat_front_addr': '172.21.15.172:6829/4145361100', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-19T11:31:01.531687+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-19T11:30:52.791546+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-19T11:30:51.216737+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-19T11:31:06.780394+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-19T11:31:06.311323+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-19T11:30:59.034487+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-19T11:31:04.628602+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-19T11:31:09.716969+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 4, 'snaps': [{'begin': 27, 'length': 2}]}], 'new_removed_snaps': [], 'new_purged_snaps': [{'pool': 4, 'snaps': [{'begin': 24, 'length': 1}, {'begin': 32, 'length': 1}]}], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-19T11:31:49.068 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-03-19T11:31:49.068 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-03-19T11:31:49.069 INFO:tasks.rados.rados.0.smithi038.stdout:1686: done (4 left) 2024-03-19T11:31:49.069 INFO:tasks.rados.rados.0.smithi038.stdout:1687: done (3 left) 2024-03-19T11:31:49.069 INFO:tasks.rados.rados.0.smithi038.stdout:1688: done (2 left) 2024-03-19T11:31:49.069 INFO:tasks.rados.rados.0.smithi038.stdout:1689: done (1 left) 2024-03-19T11:31:49.069 INFO:tasks.rados.rados.0.smithi038.stdout:1690: read oid 9 snap -1 2024-03-19T11:31:49.069 INFO:tasks.rados.rados.0.smithi038.stdout:1690: expect (ObjNum 308 snap 39 seq_num 308) 2024-03-19T11:31:49.069 INFO:tasks.rados.rados.0.smithi038.stdout:1691: tier_evict oid 20 2024-03-19T11:31:49.069 INFO:tasks.rados.rados.0.smithi038.stdout:1691: tier_evict oid 20 snap -1 2024-03-19T11:31:49.070 INFO:tasks.rados.rados.0.smithi038.stdout:1692: read oid 3 snap -1 2024-03-19T11:31:49.070 INFO:tasks.rados.rados.0.smithi038.stdout:1692: expect (ObjNum 271 snap 34 seq_num 271) 2024-03-19T11:31:49.070 INFO:tasks.rados.rados.0.smithi038.stdout:1693: write oid 41 current snap is 40 2024-03-19T11:31:49.070 INFO:tasks.rados.rados.0.smithi038.stdout:1693: seq_num 311 ranges {426977=456559,1482841=799764,2958080=268897} 2024-03-19T11:31:49.070 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:49.075+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:49.070 INFO:tasks.rados.rados.0.smithi038.stdout:1693: writing smithi03829856-41 from 426977 to 883536 tid 1 2024-03-19T11:31:49.076 INFO:tasks.rados.rados.0.smithi038.stdout:1693: writing smithi03829856-41 from 1482841 to 2282605 tid 2 2024-03-19T11:31:49.080 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:49.080 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:49.082 INFO:tasks.rados.rados.0.smithi038.stdout:1693: writing smithi03829856-41 from 2958080 to 3226977 tid 3 2024-03-19T11:31:49.083 INFO:tasks.rados.rados.0.smithi038.stdout:1694: write oid 43 current snap is 40 2024-03-19T11:31:49.083 INFO:tasks.rados.rados.0.smithi038.stdout:1694: seq_num 312 ranges {443784=648161,1657676=659775,2818209=425575} 2024-03-19T11:31:49.089 INFO:tasks.rados.rados.0.smithi038.stdout:1694: writing smithi03829856-43 from 443784 to 1091945 tid 1 2024-03-19T11:31:49.098 INFO:tasks.rados.rados.0.smithi038.stdout:1694: writing smithi03829856-43 from 1657676 to 2317451 tid 2 2024-03-19T11:31:49.100 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-19T11:31:49.103 INFO:tasks.rados.rados.0.smithi038.stdout:1694: writing smithi03829856-43 from 2818209 to 3243784 tid 3 2024-03-19T11:31:49.103 INFO:tasks.rados.rados.0.smithi038.stdout:1695: write (excl) oid 4 current snap is 40 2024-03-19T11:31:49.104 INFO:tasks.rados.rados.0.smithi038.stdout:1695: seq_num 313 ranges {460591=723410,1716158=403433,2729279=440140} 2024-03-19T11:31:49.110 INFO:tasks.rados.rados.0.smithi038.stdout:1695: writing smithi03829856-4 from 460591 to 1184001 tid 1 2024-03-19T11:31:49.115 INFO:tasks.rados.rados.0.smithi038.stdout:1695: writing smithi03829856-4 from 1716158 to 2119591 tid 2 2024-03-19T11:31:49.121 INFO:tasks.rados.rados.0.smithi038.stdout:1695: writing smithi03829856-4 from 2729279 to 3169419 tid 3 2024-03-19T11:31:49.121 INFO:tasks.rados.rados.0.smithi038.stdout:1696: snap_remove snap 29 2024-03-19T11:31:49.168 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:49.177+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:49.216 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:49.216 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:49.230 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-19T11:31:49.269 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:49.277+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:49.342 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:49.349+0000 7f923fa20640 -1 osd.7 127 handle_osd_map injecting map crc failure 2024-03-19T11:31:49.370 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:49.379+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:49.398 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:49.399 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:49.411 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-19T11:31:49.497 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:49.507+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:49.572 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:49.579+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:49.572 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-19T11:31:49.578 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:49.578 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:49.593 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-19T11:31:49.673 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:49.683+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:49.725 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:49.725 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:49.732 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:31:49.732 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:31:49.739 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-19T11:31:49.749 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:31:49.749 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:31:49.766 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:31:49.766 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:31:49.774 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:49.783+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:49.782 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:31:49.783 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:31:49.802 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:31:49.803 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:31:49.822 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:31:49.823 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:31:49.844 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:31:49.844 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:31:49.858 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:49.858 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:31:49.858 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:31:49.858 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:31:49.858 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:49.864 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:31:49.864 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:31:49.872 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-19T11:31:49.874 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:49.883+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:49.975 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:49.983+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:49.993 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:49.994 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:31:49.994 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:31:49.994 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:31:49.994 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:49.994 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:851 4.9 4:953e720b:::smithi03829856-45:head [write 591613~454162,stat] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:31:49.994 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:08.763514+0000", 2024-03-19T11:31:49.995 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 101.23830166899999, 2024-03-19T11:31:49.995 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027825888999999999, 2024-03-19T11:31:49.995 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:49.995 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:49.995 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:49.995 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:49.996 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:49.996 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 851 2024-03-19T11:31:49.996 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:49.996 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:49.996 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:49.996 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:49.996 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763514+0000", 2024-03-19T11:31:49.996 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:49.996 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:49.996 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:49.996 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:49.996 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763514+0000", 2024-03-19T11:31:49.997 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:49.997 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:49.997 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:49.997 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:49.997 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.763517+0000", 2024-03-19T11:31:49.997 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8499999999999998e-06 2024-03-19T11:31:49.997 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:49.997 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:49.997 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:49.997 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.764714+0000", 2024-03-19T11:31:49.998 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011971270000000001 2024-03-19T11:31:49.998 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:49.998 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:49.998 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:49.998 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.764745+0000", 2024-03-19T11:31:49.998 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0546999999999997e-05 2024-03-19T11:31:49.998 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:49.998 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:49.998 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:49.999 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.764767+0000", 2024-03-19T11:31:49.999 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2286999999999999e-05 2024-03-19T11:31:49.999 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:49.999 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:49.999 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:49.999 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.773924+0000", 2024-03-19T11:31:49.999 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0091567360000000004 2024-03-19T11:31:49.999 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:49.999 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:49.999 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:49.999 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.774217+0000", 2024-03-19T11:31:49.999 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029350099999999999 2024-03-19T11:31:49.999 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.000 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.000 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 1", 2024-03-19T11:31:50.000 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.774770+0000", 2024-03-19T11:31:50.000 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00055288600000000005 2024-03-19T11:31:50.000 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.000 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.000 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:50.000 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.786413+0000", 2024-03-19T11:31:50.000 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011643229999999999 2024-03-19T11:31:50.000 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.000 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.000 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:50.000 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791104+0000", 2024-03-19T11:31:50.000 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0046900830000000003 2024-03-19T11:31:50.000 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.001 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.001 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:50.001 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791248+0000", 2024-03-19T11:31:50.001 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014410200000000001 2024-03-19T11:31:50.001 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.001 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.001 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:50.001 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:08.791340+0000", 2024-03-19T11:31:50.001 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2540000000000005e-05 2024-03-19T11:31:50.001 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.001 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:50.001 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.001 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.002 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.002 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1162 4.1 4:8401c1f0:::smithi03829856-43:head [write 594981~609197] snapc 8=[8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-03-19T11:31:50.002 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.838981+0000", 2024-03-19T11:31:50.002 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 93.162835177999995, 2024-03-19T11:31:50.002 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027704349999999999, 2024-03-19T11:31:50.002 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:50.002 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:50.002 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:50.002 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:50.002 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:50.002 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1162 2024-03-19T11:31:50.002 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.002 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:50.002 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.003 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:50.003 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.838981+0000", 2024-03-19T11:31:50.003 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.003 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.003 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.003 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:50.003 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.838981+0000", 2024-03-19T11:31:50.003 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.003 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.003 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.003 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:50.003 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.838983+0000", 2024-03-19T11:31:50.003 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5399999999999998e-06 2024-03-19T11:31:50.004 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.004 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.004 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:50.004 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841322+0000", 2024-03-19T11:31:50.004 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002338592 2024-03-19T11:31:50.004 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.004 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.004 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:50.004 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841371+0000", 2024-03-19T11:31:50.004 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9123999999999999e-05 2024-03-19T11:31:50.004 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.004 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.004 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:50.004 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841399+0000", 2024-03-19T11:31:50.005 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7718000000000001e-05 2024-03-19T11:31:50.005 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.005 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.005 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:50.005 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841577+0000", 2024-03-19T11:31:50.005 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000178097 2024-03-19T11:31:50.005 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.005 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.005 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:50.005 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.841825+0000", 2024-03-19T11:31:50.005 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00024824299999999999 2024-03-19T11:31:50.005 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.005 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.005 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5", 2024-03-19T11:31:50.006 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842227+0000", 2024-03-19T11:31:50.006 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00040212199999999998 2024-03-19T11:31:50.006 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.006 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.006 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:50.006 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.865182+0000", 2024-03-19T11:31:50.006 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022954810999999999 2024-03-19T11:31:50.006 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.006 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.006 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:50.006 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.866360+0000", 2024-03-19T11:31:50.006 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001178449 2024-03-19T11:31:50.006 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.006 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.007 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:50.007 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.866538+0000", 2024-03-19T11:31:50.007 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017791000000000001 2024-03-19T11:31:50.007 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.007 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.007 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:50.007 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.866685+0000", 2024-03-19T11:31:50.007 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000146744 2024-03-19T11:31:50.007 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.007 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:50.007 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.007 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.007 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.008 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1271 4.9 4:953e720b:::smithi03829856-45:2 [read 795136~1008128] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-03-19T11:31:50.008 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:21.868056+0000", 2024-03-19T11:31:50.008 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 88.133759673, 2024-03-19T11:31:50.008 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.047270591000000001, 2024-03-19T11:31:50.008 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:50.008 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:50.008 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:50.009 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:50.009 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:50.009 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1271 2024-03-19T11:31:50.009 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.009 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:50.009 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.009 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:50.009 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868056+0000", 2024-03-19T11:31:50.009 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.009 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.009 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.010 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:50.010 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868056+0000", 2024-03-19T11:31:50.010 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.010 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.010 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.010 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:50.010 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868059+0000", 2024-03-19T11:31:50.010 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0469999999999998e-06 2024-03-19T11:31:50.011 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.011 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.011 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:50.011 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868072+0000", 2024-03-19T11:31:50.011 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2700999999999999e-05 2024-03-19T11:31:50.011 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.011 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.011 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:50.011 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868106+0000", 2024-03-19T11:31:50.012 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4152000000000001e-05 2024-03-19T11:31:50.012 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.012 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.012 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:50.012 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.868126+0000", 2024-03-19T11:31:50.012 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9386999999999999e-05 2024-03-19T11:31:50.012 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.012 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.012 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:50.013 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.911447+0000", 2024-03-19T11:31:50.013 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.043321164000000002 2024-03-19T11:31:50.013 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.013 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.013 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:50.013 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.911762+0000", 2024-03-19T11:31:50.013 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00031492400000000001 2024-03-19T11:31:50.013 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.013 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.013 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:50.014 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:21.915327+0000", 2024-03-19T11:31:50.014 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003565216 2024-03-19T11:31:50.014 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.014 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:50.014 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.014 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.014 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.014 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(3.1 84/83 [PushOp(3:9205291d:::smithi03829856-50 low_tier:head, version: 28'36, data_included: [393216~851968,1703936~196608], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:9205291d:::smithi03829856-50 low_tier:head@28'36, size: 2823543, copy_subset: [(0, 2823543)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1900544, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:baab691a:::smithi03829856-2 low_tier:head, version: 28'37, data_included: [393216~524288,1507328~509479], data_size: 1033767, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:baab691a:::smithi03829856-2 low_tier:head@28'37, size: 2016807, copy_subset: [(0, 2016807)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2016807, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:98ce8426:::smithi03829856-4 low_tier:head, version: 28'38, data_included: [393216~720896,1638400~327680], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:98ce8426:::smithi03829856-4 low_tier:head@28'38, size: 2050421, copy_subset: [(0, 2050421)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1966080, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:a86be07b:::smithi03829856-44 low_tier:head, version: 28'41, data_included: [720896~655360,1769472~393216], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:a86be07b:::smithi03829856-44 low_tier:head@28'41, size: 2722701, copy_subset: [(0, 2722701)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2162688, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:bb56919e:::smithi03829856-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'44, data_included: [393216~786432,1703936~262144], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:bb56919e:::smithi03829856-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'44, size: 2806736, copy_subset: [(0, 2806736)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1966080, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:b1f822cf:::smithi03829856-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'46, data_included: [720896~786432,1966080~262144], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:b1f822cf:::smithi03829856-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'46, size: 2369754, copy_subset: [(0, 2369754)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2228224, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:a5e8313a:::smithi03829856-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'47, data_included: [524288~720896,1769472~327680], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:a5e8313a:::smithi03829856-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'47, size: 2571438, copy_subset: [(0, 2571438)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:bca5ec64:::e3a26bf9230b1b95b8cf1a4cb01909236bf2bf3fd59b083b2291d278ab98264d:head, version: 28'48, data_included: [0~393216], data_size: 393216, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:bca5ec64:::e3a26bf9230b1b95b8cf1a4cb01909236bf2bf3fd59b083b2291d278ab98264d:head@28'48, size: 533504, copy_subset: [(0, 533504)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 533504, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:99a49d1b:::895a63407b6109b3cca9a6f5ef827134d56679bef42daadc5892269682c428ed:head, version: 32'49, data_included: [393216~388096], data_size: 388096, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:99a49d1b:::895a63407b6109b3cca9a6f5ef827134d56679bef42daadc5892269682c428ed:head@32'49, size: 781312, copy_subset: [(0, 781312)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 781312, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:83064589:::a3639ef4c215b429a637f1406db2398074b33464b996fe20e20d8b66c6bd720b:head, version: 33'51, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:83064589:::a3639ef4c215b429a637f1406db2398074b33464b996fe20e20d8b66c6bd720b:head@33'51, size: 1606656, copy_subset: [(0, 1606656)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:50.014 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:03.092965+0000", 2024-03-19T11:31:50.014 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 46.908850630000003, 2024-03-19T11:31:50.014 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.061194046000000002, 2024-03-19T11:31:50.014 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:50.015 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:50.015 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:50.015 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.015 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:50.015 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.092965+0000", 2024-03-19T11:31:50.015 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.015 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.015 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.015 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:50.015 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.092965+0000", 2024-03-19T11:31:50.015 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.015 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.015 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.016 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:50.016 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.092968+0000", 2024-03-19T11:31:50.016 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8720000000000002e-06 2024-03-19T11:31:50.016 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.016 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.016 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:50.016 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.106869+0000", 2024-03-19T11:31:50.016 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013900554000000001 2024-03-19T11:31:50.016 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.016 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.016 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:50.016 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.107077+0000", 2024-03-19T11:31:50.016 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000208626 2024-03-19T11:31:50.017 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.017 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.017 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:50.017 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.107372+0000", 2024-03-19T11:31:50.017 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029439199999999998 2024-03-19T11:31:50.017 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.017 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.017 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:50.017 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.108053+0000", 2024-03-19T11:31:50.017 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00068115899999999997 2024-03-19T11:31:50.018 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.018 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.018 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:50.018 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.108165+0000", 2024-03-19T11:31:50.018 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011162000000000001 2024-03-19T11:31:50.018 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.018 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.018 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:50.018 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.154159+0000", 2024-03-19T11:31:50.018 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.045994822999999997 2024-03-19T11:31:50.018 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.018 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:50.018 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.018 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.019 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.019 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.5 84/83 [PushOp(4:a0a50d04:::smithi03829856-13:b, version: 56'65, data_included: [601684~619341,1669078~429235], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a0a50d04:::smithi03829856-13:b@56'65, size: 2201684, copy_subset: [(0, 2201684)], clone_subset: {}, snapset: 17=[]:{b: [b, 6], 12: [12, d], 14: [13], 17: [17, 16, 15]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2098313, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:a0a50d04:::smithi03829856-13:14, version: 74'84, data_included: [601684~754801,1669078~293775], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a0a50d04:::smithi03829856-13:14@74'84, size: 2722734, copy_subset: [(0, 2722734)], clone_subset: {}, snapset: 17=[]:{b: [b, 6], 12: [12, d], 14: [13], 17: [17, 16, 15]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1962853, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:a0a50d04:::smithi03829856-13:12, version: 81'90, data_included: [601684~619341,1669078~429235], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a0a50d04:::smithi03829856-13:12@81'90, size: 2201684, copy_subset: [(0, 2201684)], clone_subset: {}, snapset: 17=[]:{b: [b, 6], 12: [12, d], 14: [13], 17: [17, 16, 15]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2098313, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:a38b9453:::smithi03829856-37:13, version: 81'92, data_included: [605052~774376,1923587~274200], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a38b9453:::smithi03829856-37:13@81'92, size: 2605052, copy_subset: [(0, 2605052)], clone_subset: {}, snapset: 14=[]:{13: [13, 12, d, b, 6]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2197787, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:a0a50d04:::smithi03829856-13:17, version: 82'94, data_included: [601684~754801,1669078~293775], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a0a50d04:::smithi03829856-13:17@82'94, size: 3025260, copy_subset: [(0, 3025260)], clone_subset: {}, snapset: 17=[]:{b: [b, 6], 12: [12, d], 14: [13], 17: [17, 16, 15]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1962853, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:a0a50d04:::smithi03829856-13:head, version: 82'97, data_included: [601684~619341,1669078~429235], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:a0a50d04:::smithi03829856-13:head@82'97, size: 2201684, copy_subset: [(0, 2201684)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2098313, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:50.019 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:03.103661+0000", 2024-03-19T11:31:50.019 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 46.898154609000002, 2024-03-19T11:31:50.019 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027776637, 2024-03-19T11:31:50.019 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:50.019 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:50.020 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:50.020 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.020 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:50.020 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.103661+0000", 2024-03-19T11:31:50.020 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.020 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.020 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.020 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:50.020 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.103661+0000", 2024-03-19T11:31:50.020 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.020 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.020 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.020 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:50.020 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.103664+0000", 2024-03-19T11:31:50.021 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.48e-06 2024-03-19T11:31:50.021 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.021 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.021 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:50.021 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.109797+0000", 2024-03-19T11:31:50.021 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061333380000000003 2024-03-19T11:31:50.021 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.021 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.021 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:50.021 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.109866+0000", 2024-03-19T11:31:50.021 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8474999999999997e-05 2024-03-19T11:31:50.021 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.022 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.022 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:50.022 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.109957+0000", 2024-03-19T11:31:50.022 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1669000000000006e-05 2024-03-19T11:31:50.022 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.022 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.022 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:50.022 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110162+0000", 2024-03-19T11:31:50.022 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000204417 2024-03-19T11:31:50.022 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.022 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.022 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:50.022 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110186+0000", 2024-03-19T11:31:50.023 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4729e-05 2024-03-19T11:31:50.023 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.023 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.023 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:50.023 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.131438+0000", 2024-03-19T11:31:50.023 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021251529000000002 2024-03-19T11:31:50.023 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.023 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:50.023 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.023 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.023 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.023 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.5 84/83 [PushOp(4:a38b9453:::smithi03829856-37:head, version: 81'93, data_included: [1775443~781480,2724284~166520], data_size: 948000, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:a38b9453:::smithi03829856-37:head@81'93, size: 2958032, copy_subset: [(0, 2958032)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2958032, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1775443, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:50.023 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:03.110032+0000", 2024-03-19T11:31:50.024 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 46.891784381000001, 2024-03-19T11:31:50.024 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027756439000000001, 2024-03-19T11:31:50.024 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:50.024 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:50.024 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:50.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.024 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:50.024 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110032+0000", 2024-03-19T11:31:50.024 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.024 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.024 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.024 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:50.024 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110032+0000", 2024-03-19T11:31:50.025 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.025 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.025 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.025 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:50.025 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110033+0000", 2024-03-19T11:31:50.025 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1510000000000001e-06 2024-03-19T11:31:50.025 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.025 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.025 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:50.026 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110353+0000", 2024-03-19T11:31:50.026 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00031981199999999999 2024-03-19T11:31:50.026 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.026 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.026 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:50.026 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110375+0000", 2024-03-19T11:31:50.026 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2772000000000002e-05 2024-03-19T11:31:50.026 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.026 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.027 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:50.027 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.110396+0000", 2024-03-19T11:31:50.027 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1066000000000001e-05 2024-03-19T11:31:50.027 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.027 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.027 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:50.027 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.131543+0000", 2024-03-19T11:31:50.027 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021146166000000001 2024-03-19T11:31:50.027 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.027 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.027 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:50.027 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.131563+0000", 2024-03-19T11:31:50.027 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0750999999999998e-05 2024-03-19T11:31:50.028 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.028 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:50.028 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.137788+0000", 2024-03-19T11:31:50.028 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062247209999999999 2024-03-19T11:31:50.028 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.028 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:50.028 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.028 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.028 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.18 85/84 [PushOp(4:1d58db11:::smithi03829856-18:6, version: 55'80, data_included: [759683~639358,1904434~409218], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:6@55'80, size: 3159683, copy_subset: [(0, 3159683)], clone_subset: {}, snapset: 16=[]:{6: [6], b: [b], 15: [15, 13, 12, d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2313652, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1d58db11:::smithi03829856-18:b, version: 58'84, data_included: [443718~955323,1564138~93253], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:b@58'84, size: 3445402, copy_subset: [(0, 3445402)], clone_subset: {}, snapset: 16=[]:{6: [6], b: [b], 15: [15, 13, 12, d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1657391, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:10, version: 66'95, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:10@66'95, size: 3764735, copy_subset: [(0, 3764735)], clone_subset: {}, snapset: 15=[]:{c: [b, 6], 10: [d], 12: [12], 15: [15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:c, version: 70'102, data_included: [416807~576167,1542129~472409], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:c@70'102, size: 2857157, copy_subset: [(0, 2857157)], clone_subset: {}, snapset: 15=[]:{c: [b, 6], 10: [d], 12: [12], 15: [15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2014538, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:15, version: 77'110, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:15@77'110, size: 2621892, copy_subset: [(0, 2621892)], clone_subset: {}, snapset: 15=[]:{c: [b, 6], 10: [d], 12: [12], 15: [15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:50.029 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:04.083107+0000", 2024-03-19T11:31:50.029 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 45.918708815000002, 2024-03-19T11:31:50.029 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.042109168000000002, 2024-03-19T11:31:50.029 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:50.029 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:50.029 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:50.029 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.029 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:50.029 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.083107+0000", 2024-03-19T11:31:50.029 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.029 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.029 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.029 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:50.030 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.083107+0000", 2024-03-19T11:31:50.030 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.030 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.030 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.030 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:50.030 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.083109+0000", 2024-03-19T11:31:50.030 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7060000000000001e-06 2024-03-19T11:31:50.030 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.031 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.031 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:50.031 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092477+0000", 2024-03-19T11:31:50.031 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0093678589999999992 2024-03-19T11:31:50.031 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.031 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.031 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:50.031 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092549+0000", 2024-03-19T11:31:50.031 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2313999999999999e-05 2024-03-19T11:31:50.031 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.031 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.032 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:50.032 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092640+0000", 2024-03-19T11:31:50.032 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1262999999999997e-05 2024-03-19T11:31:50.032 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.032 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.032 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:50.032 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.093076+0000", 2024-03-19T11:31:50.032 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00043617800000000001 2024-03-19T11:31:50.033 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.033 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.033 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:50.033 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.093145+0000", 2024-03-19T11:31:50.033 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.8651000000000004e-05 2024-03-19T11:31:50.033 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.033 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.033 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:50.033 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.125216+0000", 2024-03-19T11:31:50.033 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032071197000000003 2024-03-19T11:31:50.033 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.033 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:50.033 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.033 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.034 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.034 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.18 e85,84 4:1d58db11:::smithi03829856-18:7,38'48,4:1d58db11:::smithi03829856-18:9,46'64,4:1d58db11:::smithi03829856-18:2,48'76,4:1d58db11:::smithi03829856-18:c,70'104)", 2024-03-19T11:31:50.034 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:04.092733+0000", 2024-03-19T11:31:50.034 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 45.909082454, 2024-03-19T11:31:50.034 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033436452999999998, 2024-03-19T11:31:50.034 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:50.034 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:50.034 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:50.034 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.034 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:50.035 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092733+0000", 2024-03-19T11:31:50.035 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.035 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.035 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.035 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:50.035 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092733+0000", 2024-03-19T11:31:50.035 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.036 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.036 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:50.036 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092734+0000", 2024-03-19T11:31:50.036 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0699999999999996e-07 2024-03-19T11:31:50.036 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.036 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:50.036 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092745+0000", 2024-03-19T11:31:50.036 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0393e-05 2024-03-19T11:31:50.036 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.037 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.037 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:50.037 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092762+0000", 2024-03-19T11:31:50.037 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7649e-05 2024-03-19T11:31:50.037 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.038 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.038 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:50.038 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.092778+0000", 2024-03-19T11:31:50.038 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5625999999999999e-05 2024-03-19T11:31:50.038 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.038 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.039 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:50.039 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.125342+0000", 2024-03-19T11:31:50.039 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032564348 2024-03-19T11:31:50.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.039 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:50.040 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.125375+0000", 2024-03-19T11:31:50.040 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2441000000000003e-05 2024-03-19T11:31:50.040 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.040 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.040 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:50.040 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.126170+0000", 2024-03-19T11:31:50.041 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00079508900000000002 2024-03-19T11:31:50.041 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.041 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:50.041 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.041 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.041 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.041 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(3.3 85/84 [PushOp(3:fb6cb292:::smithi03829856-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'21, data_included: [655360~851968,1900544~196608], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:fb6cb292:::smithi03829856-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'21, size: 2302526, copy_subset: [(0, 2302526)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:d281c5a5:::smithi03829856-30 low_tier:head, version: 28'42, data_included: [458752~589824,1507328~458752], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:d281c5a5:::smithi03829856-30 low_tier:head@28'42, size: 2487403, copy_subset: [(0, 2487403)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1966080, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:ed957b32:::smithi03829856-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head, version: 28'44, data_included: [524288~786432,1900544~233912], data_size: 1020344, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:ed957b32:::smithi03829856-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head@28'44, size: 2134456, copy_subset: [(0, 2134456)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2134456, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:f23a45c7:::smithi03829856-36 low_tier:head, version: 28'45, data_included: [524288~786432,1835008~262144], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:f23a45c7:::smithi03829856-36 low_tier:head@28'45, size: 2588245, copy_subset: [(0, 2588245)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:edb38de0:::smithi03829856-40 low_tier:head, version: 28'46, data_included: [655360~720896,2097152~327680], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:edb38de0:::smithi03829856-40 low_tier:head@28'46, size: 2655473, copy_subset: [(0, 2655473)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2424832, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:f92b3f40:::b1fb925e39ab40886d36b70ff1c8c4f867ef6cb2ef150fbe90de57d057dfec36:head, version: 29'47, data_included: [262144~720896,1507328~327680], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:f92b3f40:::b1fb925e39ab40886d36b70ff1c8c4f867ef6cb2ef150fbe90de57d057dfec36:head@29'47, size: 2098176, copy_subset: [(0, 2098176)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1835008, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:fc3d03e1:::626f73f06d73fccc02b43398389ff991d56d81381f5e170c36fb9439a43cbbc0:head, version: 32'48, data_included: [0~131072], data_size: 131072, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:fc3d03e1:::626f73f06d73fccc02b43398389ff991d56d81381f5e170c36fb9439a43cbbc0:head@32'48, size: 462848, copy_subset: [(0, 462848)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 462848, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(3:f8934067:::8892b7b523af2cde6be0d91d0468f8be86a651cb052f82daf3e2c6278805eb6e:head, version: 32'49, data_included: [0~131072], data_size: 131072, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:f8934067:::8892b7b523af2cde6be0d91d0468f8be86a651cb052f82daf3e2c6278805eb6e:head@32'49, size: 202240, copy_subset: [(0, 202240)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 202240, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:50.041 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:04.108513+0000", 2024-03-19T11:31:50.041 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 45.893303013000001, 2024-03-19T11:31:50.041 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.036668899999999997, 2024-03-19T11:31:50.042 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:50.042 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:50.042 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:50.042 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.042 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:50.042 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.108513+0000", 2024-03-19T11:31:50.042 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.042 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.042 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.042 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:50.042 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.108513+0000", 2024-03-19T11:31:50.042 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.042 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.042 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.043 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:50.043 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.108515+0000", 2024-03-19T11:31:50.043 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7579999999999999e-06 2024-03-19T11:31:50.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.043 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:50.043 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115352+0000", 2024-03-19T11:31:50.043 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0068369950000000002 2024-03-19T11:31:50.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.043 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:50.043 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115421+0000", 2024-03-19T11:31:50.044 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9282999999999994e-05 2024-03-19T11:31:50.044 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.044 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.044 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:50.044 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115505+0000", 2024-03-19T11:31:50.044 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3714999999999994e-05 2024-03-19T11:31:50.044 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.044 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.044 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:50.044 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115948+0000", 2024-03-19T11:31:50.044 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000443036 2024-03-19T11:31:50.044 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.044 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.044 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:50.045 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116012+0000", 2024-03-19T11:31:50.045 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3905999999999994e-05 2024-03-19T11:31:50.045 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.045 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.045 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:50.045 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.145182+0000", 2024-03-19T11:31:50.045 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.029170207 2024-03-19T11:31:50.045 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.045 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:50.045 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.045 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.046 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.046 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.4.13:82 3.3 e85/84 3:f6b6d21a:::smithi03829856-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 85'103, mlcod=0'0)", 2024-03-19T11:31:50.046 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:04.115589+0000", 2024-03-19T11:31:50.046 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 45.886226706999999, 2024-03-19T11:31:50.046 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035569142999999998, 2024-03-19T11:31:50.046 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:50.047 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:50.047 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:50.047 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.047 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:50.047 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115589+0000", 2024-03-19T11:31:50.047 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.048 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.048 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.048 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:50.048 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115589+0000", 2024-03-19T11:31:50.048 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.049 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.049 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.049 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:50.049 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115591+0000", 2024-03-19T11:31:50.049 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2780000000000001e-06 2024-03-19T11:31:50.049 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.050 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.050 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:50.050 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115602+0000", 2024-03-19T11:31:50.050 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1797000000000001e-05 2024-03-19T11:31:50.050 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.050 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.051 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:50.051 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115613+0000", 2024-03-19T11:31:50.051 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1025e-05 2024-03-19T11:31:50.051 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.051 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.052 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:50.052 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115619+0000", 2024-03-19T11:31:50.052 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.976e-06 2024-03-19T11:31:50.052 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.052 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.052 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:50.053 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.145267+0000", 2024-03-19T11:31:50.053 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.02964791 2024-03-19T11:31:50.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.053 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.053 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:50.053 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.145304+0000", 2024-03-19T11:31:50.054 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6455000000000002e-05 2024-03-19T11:31:50.054 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.054 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:50.054 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.151114+0000", 2024-03-19T11:31:50.054 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0058103139999999996 2024-03-19T11:31:50.054 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.054 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:50.055 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.151158+0000", 2024-03-19T11:31:50.055 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4388000000000003e-05 2024-03-19T11:31:50.055 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.055 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:50.055 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.055 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.055 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.055 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.4.13:84 3.3 e85/84 3:ce6951d3:::smithi03829856-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 85'104, mlcod=0'0)", 2024-03-19T11:31:50.055 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:04.115824+0000", 2024-03-19T11:31:50.055 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 45.885991703999998, 2024-03-19T11:31:50.055 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035373398, 2024-03-19T11:31:50.056 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:50.056 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:50.056 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:50.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.056 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:50.056 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115824+0000", 2024-03-19T11:31:50.056 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.057 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:50.057 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115824+0000", 2024-03-19T11:31:50.057 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.057 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.057 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:50.057 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115825+0000", 2024-03-19T11:31:50.057 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.3399999999999999e-07 2024-03-19T11:31:50.058 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.058 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.058 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:50.058 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115861+0000", 2024-03-19T11:31:50.058 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6303000000000001e-05 2024-03-19T11:31:50.058 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.058 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.058 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:50.058 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115869+0000", 2024-03-19T11:31:50.058 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0350000000000001e-06 2024-03-19T11:31:50.058 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.058 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.058 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:50.058 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115874+0000", 2024-03-19T11:31:50.059 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1599999999999997e-06 2024-03-19T11:31:50.059 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.059 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.059 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:50.059 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.145686+0000", 2024-03-19T11:31:50.059 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.029811252 2024-03-19T11:31:50.059 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.059 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.059 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:50.059 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.145706+0000", 2024-03-19T11:31:50.059 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0264000000000001e-05 2024-03-19T11:31:50.059 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.059 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.059 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:50.060 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.151161+0000", 2024-03-19T11:31:50.060 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0054555280000000003 2024-03-19T11:31:50.060 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.060 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.060 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:50.060 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.151198+0000", 2024-03-19T11:31:50.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6322000000000002e-05 2024-03-19T11:31:50.061 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.061 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:50.061 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.061 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.4.13:85 3.3 e85/84 3:ce6951d3:::smithi03829856-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 85'105, mlcod=0'0)", 2024-03-19T11:31:50.061 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:04.115887+0000", 2024-03-19T11:31:50.061 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 45.885929089999998, 2024-03-19T11:31:50.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035346673000000002, 2024-03-19T11:31:50.061 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:50.062 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:50.062 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:50.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:50.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115887+0000", 2024-03-19T11:31:50.063 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.063 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.063 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.063 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:50.063 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115887+0000", 2024-03-19T11:31:50.063 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.063 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.063 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.063 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:50.063 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115887+0000", 2024-03-19T11:31:50.064 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2300000000000002e-07 2024-03-19T11:31:50.064 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.064 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.064 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:50.064 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115900+0000", 2024-03-19T11:31:50.064 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3044000000000001e-05 2024-03-19T11:31:50.064 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.064 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.064 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:50.064 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115906+0000", 2024-03-19T11:31:50.064 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.5929999999999996e-06 2024-03-19T11:31:50.064 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.064 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.065 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:50.065 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115910+0000", 2024-03-19T11:31:50.065 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9269999999999998e-06 2024-03-19T11:31:50.065 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.065 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.065 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:50.065 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.145919+0000", 2024-03-19T11:31:50.065 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030009650999999998 2024-03-19T11:31:50.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.066 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.066 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:50.066 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.145948+0000", 2024-03-19T11:31:50.066 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9047000000000002e-05 2024-03-19T11:31:50.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.066 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.066 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:50.066 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.151200+0000", 2024-03-19T11:31:50.066 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052512970000000003 2024-03-19T11:31:50.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.067 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.067 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:50.067 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.151234+0000", 2024-03-19T11:31:50.067 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3791000000000002e-05 2024-03-19T11:31:50.067 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.067 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:50.067 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.067 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.067 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.067 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(3.3 85/84 [PushOp(3:fcdd1002:::06a0957e01d2e04d1b09d0c04f111819905dcf8372f2c6bee5014c22d5b575ef:head, version: 34'50, data_included: [0~458752], data_size: 458752, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:fcdd1002:::06a0957e01d2e04d1b09d0c04f111819905dcf8372f2c6bee5014c22d5b575ef:head@34'50, size: 689152, copy_subset: [(0, 689152)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 689152, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:50.067 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:04.115917+0000", 2024-03-19T11:31:50.067 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 45.885898658999999, 2024-03-19T11:31:50.068 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033796169000000001, 2024-03-19T11:31:50.068 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:50.068 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:50.068 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:50.068 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.068 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:50.068 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115917+0000", 2024-03-19T11:31:50.068 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.068 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.069 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.069 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:50.069 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115917+0000", 2024-03-19T11:31:50.069 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.069 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.069 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.069 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:50.069 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.115918+0000", 2024-03-19T11:31:50.069 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8000000000000002e-07 2024-03-19T11:31:50.069 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.069 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.070 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:50.070 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116163+0000", 2024-03-19T11:31:50.070 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00024516400000000003 2024-03-19T11:31:50.070 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.070 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.070 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:50.070 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116181+0000", 2024-03-19T11:31:50.070 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8372999999999998e-05 2024-03-19T11:31:50.070 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.070 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.070 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:50.070 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116196+0000", 2024-03-19T11:31:50.071 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5041e-05 2024-03-19T11:31:50.071 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.071 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.071 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:50.071 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.147578+0000", 2024-03-19T11:31:50.071 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031382284000000003 2024-03-19T11:31:50.071 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.071 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.071 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:50.071 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.147594+0000", 2024-03-19T11:31:50.072 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5153e-05 2024-03-19T11:31:50.072 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.072 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.072 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:50.072 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.149713+0000", 2024-03-19T11:31:50.072 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021198739999999999 2024-03-19T11:31:50.072 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.072 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:50.072 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.072 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.072 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.072 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.4.13:83 3.3 e85/84 3:f6b6d21a:::smithi03829856-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 85'106, mlcod=0'0)", 2024-03-19T11:31:50.072 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:04.116216+0000", 2024-03-19T11:31:50.072 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 45.885599599999999, 2024-03-19T11:31:50.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035049444999999999, 2024-03-19T11:31:50.073 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:50.073 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:50.073 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:50.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:50.073 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116216+0000", 2024-03-19T11:31:50.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.073 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:50.073 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116216+0000", 2024-03-19T11:31:50.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.073 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.074 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.074 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:50.074 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116217+0000", 2024-03-19T11:31:50.074 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.22e-07 2024-03-19T11:31:50.074 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.074 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.074 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:50.074 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116221+0000", 2024-03-19T11:31:50.074 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5349999999999998e-06 2024-03-19T11:31:50.074 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.074 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.074 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:50.074 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116237+0000", 2024-03-19T11:31:50.075 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5532e-05 2024-03-19T11:31:50.075 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.075 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.075 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:50.075 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.116245+0000", 2024-03-19T11:31:50.075 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0450000000000001e-06 2024-03-19T11:31:50.075 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.075 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.075 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:50.075 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.146739+0000", 2024-03-19T11:31:50.075 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030494351999999999 2024-03-19T11:31:50.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.076 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:50.076 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.146776+0000", 2024-03-19T11:31:50.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6894999999999998e-05 2024-03-19T11:31:50.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.076 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:50.076 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.151237+0000", 2024-03-19T11:31:50.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0044605160000000003 2024-03-19T11:31:50.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.076 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:50.076 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:04.151266+0000", 2024-03-19T11:31:50.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9247999999999998e-05 2024-03-19T11:31:50.077 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.077 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:50.077 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.077 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3194 4.d 4:b65e6658:::smithi03829856-9:head [read 0~3714347,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108)", 2024-03-19T11:31:50.077 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.202652+0000", 2024-03-19T11:31:50.077 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.799164010999998, 2024-03-19T11:31:50.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031511255000000002, 2024-03-19T11:31:50.078 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:50.078 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:50.078 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:50.078 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:50.078 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:50.078 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3194 2024-03-19T11:31:50.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.078 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:50.079 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.079 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:50.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202652+0000", 2024-03-19T11:31:50.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.079 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.079 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.079 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:50.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202652+0000", 2024-03-19T11:31:50.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.079 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.080 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:50.080 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202653+0000", 2024-03-19T11:31:50.080 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3960000000000001e-06 2024-03-19T11:31:50.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.080 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:50.080 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202661+0000", 2024-03-19T11:31:50.080 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0830000000000004e-06 2024-03-19T11:31:50.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.080 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:50.080 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202679+0000", 2024-03-19T11:31:50.081 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7972e-05 2024-03-19T11:31:50.081 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.081 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.081 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:50.081 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202693+0000", 2024-03-19T11:31:50.081 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4031e-05 2024-03-19T11:31:50.081 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.081 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.082 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:50.082 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.203299+0000", 2024-03-19T11:31:50.082 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00060564400000000002 2024-03-19T11:31:50.082 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.082 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.082 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:50.082 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.203528+0000", 2024-03-19T11:31:50.082 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000229049 2024-03-19T11:31:50.082 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.082 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.082 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:50.082 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.234163+0000", 2024-03-19T11:31:50.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030635079999999999 2024-03-19T11:31:50.083 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.083 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:50.083 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.083 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.083 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3195 4.d 4:b65e6658:::smithi03829856-9:head [read 0~3714347,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108)", 2024-03-19T11:31:50.083 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.202714+0000", 2024-03-19T11:31:50.084 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.79910198, 2024-03-19T11:31:50.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.028551382, 2024-03-19T11:31:50.084 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:50.084 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:50.084 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:50.084 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:50.084 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:50.084 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3195 2024-03-19T11:31:50.084 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.084 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:50.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.085 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:50.085 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202714+0000", 2024-03-19T11:31:50.085 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.085 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.085 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:50.085 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202714+0000", 2024-03-19T11:31:50.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.086 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.086 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:50.086 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202715+0000", 2024-03-19T11:31:50.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.203e-06 2024-03-19T11:31:50.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:50.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202722+0000", 2024-03-19T11:31:50.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1289999999999998e-06 2024-03-19T11:31:50.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:50.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202739+0000", 2024-03-19T11:31:50.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6929e-05 2024-03-19T11:31:50.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.088 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:50.088 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202754+0000", 2024-03-19T11:31:50.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4642e-05 2024-03-19T11:31:50.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.088 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:50.088 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.227218+0000", 2024-03-19T11:31:50.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.024464587999999999 2024-03-19T11:31:50.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.089 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:50.089 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.227471+0000", 2024-03-19T11:31:50.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000252747 2024-03-19T11:31:50.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.089 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:50.089 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.231265+0000", 2024-03-19T11:31:50.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0037941440000000002 2024-03-19T11:31:50.090 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.090 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:50.090 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.090 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.090 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3196 4.d 4:b65e6658:::smithi03829856-9:head [sparse-read 0~2135368] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108)", 2024-03-19T11:31:50.090 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.202774+0000", 2024-03-19T11:31:50.090 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.799042238999998, 2024-03-19T11:31:50.090 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033044519000000001, 2024-03-19T11:31:50.090 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:50.090 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:50.090 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:50.091 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:50.091 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:50.091 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3196 2024-03-19T11:31:50.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.091 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:50.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.091 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:50.091 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202774+0000", 2024-03-19T11:31:50.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.091 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:50.091 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202774+0000", 2024-03-19T11:31:50.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.092 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.092 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:50.092 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202775+0000", 2024-03-19T11:31:50.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.423e-06 2024-03-19T11:31:50.092 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.093 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:50.093 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202782+0000", 2024-03-19T11:31:50.093 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.3660000000000003e-06 2024-03-19T11:31:50.093 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.093 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:50.093 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202801+0000", 2024-03-19T11:31:50.093 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8182e-05 2024-03-19T11:31:50.093 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.093 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:50.094 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.202815+0000", 2024-03-19T11:31:50.094 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4042e-05 2024-03-19T11:31:50.094 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.094 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:50.094 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.234294+0000", 2024-03-19T11:31:50.094 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.031479493999999997 2024-03-19T11:31:50.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.095 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:50.095 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.234670+0000", 2024-03-19T11:31:50.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000375595 2024-03-19T11:31:50.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.095 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:50.095 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.235818+0000", 2024-03-19T11:31:50.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011484170000000001 2024-03-19T11:31:50.095 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.095 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:50.095 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.096 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(osd.0.13:71 3.3 3:e22267ea:::smithi03829856-38 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e108)", 2024-03-19T11:31:50.096 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.209268+0000", 2024-03-19T11:31:50.096 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.792547987999999, 2024-03-19T11:31:50.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.038534928000000003, 2024-03-19T11:31:50.096 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:50.096 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:50.096 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:50.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:50.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.209268+0000", 2024-03-19T11:31:50.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:50.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.209268+0000", 2024-03-19T11:31:50.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:50.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.209272+0000", 2024-03-19T11:31:50.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0500000000000002e-06 2024-03-19T11:31:50.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:50.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.209291+0000", 2024-03-19T11:31:50.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8734999999999999e-05 2024-03-19T11:31:50.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:50.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.209329+0000", 2024-03-19T11:31:50.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8223000000000003e-05 2024-03-19T11:31:50.098 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.098 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.098 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:50.098 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.209354+0000", 2024-03-19T11:31:50.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5114e-05 2024-03-19T11:31:50.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.099 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:50.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.209573+0000", 2024-03-19T11:31:50.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021842000000000001 2024-03-19T11:31:50.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.099 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:50.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.209814+0000", 2024-03-19T11:31:50.099 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00024179400000000001 2024-03-19T11:31:50.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.099 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.099 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:50.099 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.210284+0000", 2024-03-19T11:31:50.100 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00046991699999999998 2024-03-19T11:31:50.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.100 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:50.100 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.213128+0000", 2024-03-19T11:31:50.100 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002843609 2024-03-19T11:31:50.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.100 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:50.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.247417+0000", 2024-03-19T11:31:50.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.034289312000000002 2024-03-19T11:31:50.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.101 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:50.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.247692+0000", 2024-03-19T11:31:50.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027449000000000001 2024-03-19T11:31:50.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:50.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.247803+0000", 2024-03-19T11:31:50.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000111264 2024-03-19T11:31:50.102 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.102 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:50.102 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.102 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.102 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3198 3.3 3:f117d76c:::3a52856062567f05d8c7df91dad2c51911138f6216e1ec709153bdff536e4abe:head [write 0~927232] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-19T11:31:50.102 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.213434+0000", 2024-03-19T11:31:50.102 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 22.788382296999998, 2024-03-19T11:31:50.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.039874882, 2024-03-19T11:31:50.102 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:50.103 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:50.103 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:50.103 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:50.103 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:50.103 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3198 2024-03-19T11:31:50.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.103 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:50.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.103 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:50.103 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.213434+0000", 2024-03-19T11:31:50.103 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.103 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:50.103 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.213434+0000", 2024-03-19T11:31:50.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.104 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:50.104 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.213436+0000", 2024-03-19T11:31:50.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4430000000000002e-06 2024-03-19T11:31:50.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.104 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:50.104 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.219501+0000", 2024-03-19T11:31:50.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0060649240000000002 2024-03-19T11:31:50.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.104 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:50.105 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.219531+0000", 2024-03-19T11:31:50.105 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0470000000000001e-05 2024-03-19T11:31:50.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.105 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:50.105 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.219552+0000", 2024-03-19T11:31:50.105 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1012000000000001e-05 2024-03-19T11:31:50.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.105 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:50.105 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.231849+0000", 2024-03-19T11:31:50.105 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012296105999999999 2024-03-19T11:31:50.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.105 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:50.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.232349+0000", 2024-03-19T11:31:50.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00050035100000000001 2024-03-19T11:31:50.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:50.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.233102+0000", 2024-03-19T11:31:50.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00075262300000000001 2024-03-19T11:31:50.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:50.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.249722+0000", 2024-03-19T11:31:50.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016620901 2024-03-19T11:31:50.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.107 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:50.107 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.253185+0000", 2024-03-19T11:31:50.107 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0034625340000000002 2024-03-19T11:31:50.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.107 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:50.107 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.253252+0000", 2024-03-19T11:31:50.107 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6538000000000006e-05 2024-03-19T11:31:50.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.107 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:50.107 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.253309+0000", 2024-03-19T11:31:50.107 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.698e-05 2024-03-19T11:31:50.107 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.107 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:50.108 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.108 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.108 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3323 4.18 4:1d58db11:::smithi03829856-18:head [tier-promote] snapc 22=[22,21,1f,1e,1d,1c,1b,1a,18,17] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-19T11:31:50.108 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:31.226946+0000", 2024-03-19T11:31:50.108 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 18.774870001, 2024-03-19T11:31:50.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.037925183000000001, 2024-03-19T11:31:50.108 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:50.108 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:50.108 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:50.108 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:50.108 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:50.108 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3323 2024-03-19T11:31:50.108 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.108 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:50.109 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.109 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:50.109 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.226946+0000", 2024-03-19T11:31:50.109 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.109 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.109 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.109 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:50.109 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.226946+0000", 2024-03-19T11:31:50.109 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.109 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.109 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.109 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:50.109 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.226947+0000", 2024-03-19T11:31:50.109 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4670000000000001e-06 2024-03-19T11:31:50.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.110 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.110 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:50.110 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.226955+0000", 2024-03-19T11:31:50.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9130000000000003e-06 2024-03-19T11:31:50.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.110 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.110 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:50.110 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.226978+0000", 2024-03-19T11:31:50.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2229e-05 2024-03-19T11:31:50.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.110 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.110 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:50.110 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.226993+0000", 2024-03-19T11:31:50.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5420000000000001e-05 2024-03-19T11:31:50.111 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.111 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.111 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:50.111 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.227146+0000", 2024-03-19T11:31:50.111 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015327200000000001 2024-03-19T11:31:50.111 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.111 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.111 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:50.111 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.227406+0000", 2024-03-19T11:31:50.111 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025941799999999998 2024-03-19T11:31:50.111 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.111 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.111 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:50.111 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.264871+0000", 2024-03-19T11:31:50.111 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.037465463999999997 2024-03-19T11:31:50.112 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.112 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:50.112 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.112 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.112 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:50.112 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:50.113 INFO:tasks.rados.rados.0.smithi038.stdout:1696: done (7 left) 2024-03-19T11:31:50.113 INFO:tasks.rados.rados.0.smithi038.stdout:1697: snap_remove snap 39 2024-03-19T11:31:50.114 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-19T11:31:50.114 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:50.065+0000 7fb17dd72640 -1 osd.4 128 handle_osd_map injecting map crc failure 2024-03-19T11:31:50.114 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:50.081+0000 7f143b8b4640 -1 osd.5 128 handle_osd_map injecting map crc failure 2024-03-19T11:31:50.115 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:50.121+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:50.214 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:50.221+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:50.278 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:50.278 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:31:50.278 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:31:50.278 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:50.292 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-19T11:31:50.340 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:50.349+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:50.416 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:50.423+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:50.463 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:50.463 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:31:50.463 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:31:50.463 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:31:50.463 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:50.479 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-19T11:31:50.524 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:50.533+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:50.605 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:50.606 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:31:50.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:31:50.606 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:31:50.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.606 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1158 4.8 4:15708482:::smithi03829856-23:head [tier-promote] snapc 8=[8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-03-19T11:31:50.606 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.827517+0000", 2024-03-19T11:31:50.606 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 93.785413220999999, 2024-03-19T11:31:50.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.034096983999999997, 2024-03-19T11:31:50.606 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:50.607 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:50.607 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:50.607 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:50.607 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:50.607 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1158 2024-03-19T11:31:50.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.607 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:50.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:50.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827517+0000", 2024-03-19T11:31:50.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:50.608 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827517+0000", 2024-03-19T11:31:50.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.608 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:50.608 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827520+0000", 2024-03-19T11:31:50.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.45e-06 2024-03-19T11:31:50.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.608 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:50.608 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827535+0000", 2024-03-19T11:31:50.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.509e-05 2024-03-19T11:31:50.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.609 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:50.609 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827579+0000", 2024-03-19T11:31:50.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4211000000000001e-05 2024-03-19T11:31:50.609 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.609 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:50.609 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827603+0000", 2024-03-19T11:31:50.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3586000000000001e-05 2024-03-19T11:31:50.610 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.610 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:50.610 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827789+0000", 2024-03-19T11:31:50.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018580199999999999 2024-03-19T11:31:50.610 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.610 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:50.610 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.828077+0000", 2024-03-19T11:31:50.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028846500000000001 2024-03-19T11:31:50.610 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:50.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.861614+0000", 2024-03-19T11:31:50.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033536379999999998 2024-03-19T11:31:50.611 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.611 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:50.611 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.611 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:1162 4.1 e39/21 4:8401c1f0:::smithi03829856-43:head v 39'47, mlcod=38'43)", 2024-03-19T11:31:50.611 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.842598+0000", 2024-03-19T11:31:50.611 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 93.770331451000004, 2024-03-19T11:31:50.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0233756, 2024-03-19T11:31:50.611 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:50.611 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:50.611 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:50.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:50.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842598+0000", 2024-03-19T11:31:50.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:50.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842598+0000", 2024-03-19T11:31:50.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:50.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.842599+0000", 2024-03-19T11:31:50.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6000000000000003e-07 2024-03-19T11:31:50.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.613 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:50.613 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.843566+0000", 2024-03-19T11:31:50.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00096688700000000004 2024-03-19T11:31:50.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.613 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:50.613 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.843601+0000", 2024-03-19T11:31:50.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.502e-05 2024-03-19T11:31:50.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.614 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:50.614 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.843621+0000", 2024-03-19T11:31:50.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0305e-05 2024-03-19T11:31:50.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:50.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.843882+0000", 2024-03-19T11:31:50.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026027100000000002 2024-03-19T11:31:50.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:50.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.844022+0000", 2024-03-19T11:31:50.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014020100000000001 2024-03-19T11:31:50.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:50.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.865862+0000", 2024-03-19T11:31:50.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021839725000000001 2024-03-19T11:31:50.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.617 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:50.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.865974+0000", 2024-03-19T11:31:50.618 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00011243099999999999 2024-03-19T11:31:50.618 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.618 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:50.618 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.618 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.619 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1209 4.8 4:1d58db11:::smithi03829856-18:head [write 645402~551297,stat] snapc 9=[9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:31:50.619 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:18.121657+0000", 2024-03-19T11:31:50.619 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 92.491272448999993, 2024-03-19T11:31:50.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.025074349999999999, 2024-03-19T11:31:50.619 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:50.619 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:50.619 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:50.619 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:50.619 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:50.619 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1209 2024-03-19T11:31:50.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.619 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:50.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:50.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.121657+0000", 2024-03-19T11:31:50.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:50.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.121657+0000", 2024-03-19T11:31:50.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:50.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.121661+0000", 2024-03-19T11:31:50.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0489999999999997e-06 2024-03-19T11:31:50.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:50.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123747+0000", 2024-03-19T11:31:50.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0020857720000000001 2024-03-19T11:31:50.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:50.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123779+0000", 2024-03-19T11:31:50.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2258000000000002e-05 2024-03-19T11:31:50.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:50.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123795+0000", 2024-03-19T11:31:50.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5177000000000001e-05 2024-03-19T11:31:50.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.622 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.622 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:50.622 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123949+0000", 2024-03-19T11:31:50.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015403699999999999 2024-03-19T11:31:50.622 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.622 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.622 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:50.622 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.124134+0000", 2024-03-19T11:31:50.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000185484 2024-03-19T11:31:50.622 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.622 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.622 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:50.622 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.124554+0000", 2024-03-19T11:31:50.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00042019699999999998 2024-03-19T11:31:50.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:50.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146267+0000", 2024-03-19T11:31:50.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021712838000000002 2024-03-19T11:31:50.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:50.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146424+0000", 2024-03-19T11:31:50.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015647299999999999 2024-03-19T11:31:50.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:50.624 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146597+0000", 2024-03-19T11:31:50.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017303699999999999 2024-03-19T11:31:50.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:50.624 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146732+0000", 2024-03-19T11:31:50.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013502800000000001 2024-03-19T11:31:50.624 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.624 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:50.624 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.625 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1443 4.2 4:43b32242:::smithi03829856-22:head [read 0~2352947,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:31:50.625 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.979417+0000", 2024-03-19T11:31:50.625 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 84.633512776000003, 2024-03-19T11:31:50.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.96719988700000004, 2024-03-19T11:31:50.625 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:50.625 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:50.625 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:50.625 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:50.625 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:50.625 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1443 2024-03-19T11:31:50.625 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.625 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:50.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.626 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:50.626 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979417+0000", 2024-03-19T11:31:50.626 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.626 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.626 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:50.626 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979417+0000", 2024-03-19T11:31:50.626 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.626 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.626 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:50.626 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979420+0000", 2024-03-19T11:31:50.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.366e-06 2024-03-19T11:31:50.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:50.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979437+0000", 2024-03-19T11:31:50.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6192e-05 2024-03-19T11:31:50.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:50.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979474+0000", 2024-03-19T11:31:50.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7601000000000001e-05 2024-03-19T11:31:50.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:50.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979496+0000", 2024-03-19T11:31:50.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1506999999999999e-05 2024-03-19T11:31:50.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:50.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979712+0000", 2024-03-19T11:31:50.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021641299999999999 2024-03-19T11:31:50.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:31:50.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979746+0000", 2024-03-19T11:31:50.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3386000000000002e-05 2024-03-19T11:31:50.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.629 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.629 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:50.629 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.931552+0000", 2024-03-19T11:31:50.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.95180659899999998 2024-03-19T11:31:50.629 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.629 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.629 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:50.629 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.934681+0000", 2024-03-19T11:31:50.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0031292289999999999 2024-03-19T11:31:50.630 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.630 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:50.630 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.942701+0000", 2024-03-19T11:31:50.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0080198809999999995 2024-03-19T11:31:50.630 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.630 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:50.630 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.942839+0000", 2024-03-19T11:31:50.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013816999999999999 2024-03-19T11:31:50.630 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.631 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:50.631 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.946617+0000", 2024-03-19T11:31:50.631 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003777543 2024-03-19T11:31:50.631 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.631 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:50.631 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.631 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1444 4.2 4:43b32242:::smithi03829856-22:head [read 0~2352947,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:31:50.631 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.979553+0000", 2024-03-19T11:31:50.631 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 84.633377073000005, 2024-03-19T11:31:50.631 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.97015945400000003, 2024-03-19T11:31:50.631 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:50.631 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:50.631 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:50.632 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:50.632 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:50.632 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1444 2024-03-19T11:31:50.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.632 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:50.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:50.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979553+0000", 2024-03-19T11:31:50.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:50.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979553+0000", 2024-03-19T11:31:50.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:50.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979554+0000", 2024-03-19T11:31:50.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.403e-06 2024-03-19T11:31:50.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:50.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979564+0000", 2024-03-19T11:31:50.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0032e-05 2024-03-19T11:31:50.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.634 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:50.634 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979579+0000", 2024-03-19T11:31:50.634 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4875000000000001e-05 2024-03-19T11:31:50.634 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.634 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.634 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:50.634 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979595+0000", 2024-03-19T11:31:50.634 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6300999999999999e-05 2024-03-19T11:31:50.634 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.634 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.634 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:50.635 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979874+0000", 2024-03-19T11:31:50.635 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000279065 2024-03-19T11:31:50.635 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.635 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:31:50.635 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979904+0000", 2024-03-19T11:31:50.635 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.94e-05 2024-03-19T11:31:50.635 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.635 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:50.635 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.934781+0000", 2024-03-19T11:31:50.635 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.95487744900000004 2024-03-19T11:31:50.635 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.635 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:50.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.934946+0000", 2024-03-19T11:31:50.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016485999999999999 2024-03-19T11:31:50.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:50.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.946829+0000", 2024-03-19T11:31:50.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011882482999999999 2024-03-19T11:31:50.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:50.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.946989+0000", 2024-03-19T11:31:50.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000160421 2024-03-19T11:31:50.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.637 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:50.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.949712+0000", 2024-03-19T11:31:50.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002723165 2024-03-19T11:31:50.637 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.637 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:50.637 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.637 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1445 4.2 4:43b32242:::smithi03829856-22:head [read 0~2352947,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:31:50.637 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.979653+0000", 2024-03-19T11:31:50.637 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 84.633276941999995, 2024-03-19T11:31:50.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.97249507999999996, 2024-03-19T11:31:50.637 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:50.638 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:50.638 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:50.638 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:50.638 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:50.638 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1445 2024-03-19T11:31:50.638 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.638 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:50.638 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.638 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:50.638 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979653+0000", 2024-03-19T11:31:50.638 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.638 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.638 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.638 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:50.639 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979653+0000", 2024-03-19T11:31:50.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.639 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:50.639 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979654+0000", 2024-03-19T11:31:50.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.372e-06 2024-03-19T11:31:50.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.639 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:50.639 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979662+0000", 2024-03-19T11:31:50.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9959999999999993e-06 2024-03-19T11:31:50.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.639 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:50.639 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979698+0000", 2024-03-19T11:31:50.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5763000000000003e-05 2024-03-19T11:31:50.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:50.640 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979714+0000", 2024-03-19T11:31:50.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6283999999999999e-05 2024-03-19T11:31:50.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:50.640 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979990+0000", 2024-03-19T11:31:50.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000275829 2024-03-19T11:31:50.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.641 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:31:50.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.980018+0000", 2024-03-19T11:31:50.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8082999999999999e-05 2024-03-19T11:31:50.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.641 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:50.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.935620+0000", 2024-03-19T11:31:50.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.95560133800000002 2024-03-19T11:31:50.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.641 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:50.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.935795+0000", 2024-03-19T11:31:50.642 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000175656 2024-03-19T11:31:50.642 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.642 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.642 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:50.642 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.949835+0000", 2024-03-19T11:31:50.642 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014040083 2024-03-19T11:31:50.642 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.642 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.642 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:50.642 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.950002+0000", 2024-03-19T11:31:50.642 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016673 2024-03-19T11:31:50.642 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.642 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.642 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:50.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.952148+0000", 2024-03-19T11:31:50.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021459460000000001 2024-03-19T11:31:50.643 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.643 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:50.643 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.643 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2207 4.28 4:15708482:::smithi03829856-23:head [write 541225~644694,stat] snapc 13=[13,12,11,d,b,6] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-03-19T11:31:50.643 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:51.160660+0000", 2024-03-19T11:31:50.643 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 59.452270116000001, 2024-03-19T11:31:50.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.025400267000000001, 2024-03-19T11:31:50.643 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:50.643 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:50.643 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:50.643 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:50.644 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:50.644 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2207 2024-03-19T11:31:50.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.644 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:50.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.644 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:50.644 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.160660+0000", 2024-03-19T11:31:50.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.644 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:50.644 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.160660+0000", 2024-03-19T11:31:50.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:50.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.160688+0000", 2024-03-19T11:31:50.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8058000000000001e-05 2024-03-19T11:31:50.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:50.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.168614+0000", 2024-03-19T11:31:50.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079262810000000003 2024-03-19T11:31:50.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:50.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.168662+0000", 2024-03-19T11:31:50.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8327e-05 2024-03-19T11:31:50.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:50.646 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.168679+0000", 2024-03-19T11:31:50.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6305999999999999e-05 2024-03-19T11:31:50.646 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:50.646 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.168830+0000", 2024-03-19T11:31:50.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000151714 2024-03-19T11:31:50.646 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:50.646 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.169107+0000", 2024-03-19T11:31:50.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027608299999999999 2024-03-19T11:31:50.646 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.647 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:50.647 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.169605+0000", 2024-03-19T11:31:50.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00049874999999999995 2024-03-19T11:31:50.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.647 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:50.647 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.185769+0000", 2024-03-19T11:31:50.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016163397999999999 2024-03-19T11:31:50.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.647 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:50.647 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.185809+0000", 2024-03-19T11:31:50.648 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0598e-05 2024-03-19T11:31:50.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.648 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:50.648 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.185898+0000", 2024-03-19T11:31:50.648 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8270999999999996e-05 2024-03-19T11:31:50.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:50.649 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.186060+0000", 2024-03-19T11:31:50.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016248099999999999 2024-03-19T11:31:50.649 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.649 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:50.649 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.650 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.650 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.15 75/74 [PushOp(4:af9d0d56:::smithi03829856-20:head, version: 35'55, data_included: [655360~589824,1900544~418789], data_size: 1008613, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:af9d0d56:::smithi03829856-20:head@35'55, size: 2319333, copy_subset: [(0, 2319333)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2319333, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:aa3303a5:::smithi03829856-17:b, version: 58'79, data_included: [614400~692224,1903006~356352], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:aa3303a5:::smithi03829856-17:b@58'79, size: 2268912, copy_subset: [(0, 2268912)], clone_subset: {}, snapset: 14=[]:{b: [b, 6], 10: [d], 12: [12, 11], 14: [13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2259358, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ad3ad1b3:::smithi03829856-21:b, version: 58'81, data_included: [736140~537686,2020581~315559], data_size: 853245, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:ad3ad1b3:::smithi03829856-21:b@58'81, size: 2336140, copy_subset: [(0, 2336140)], clone_subset: {}, snapset: 10=[]:{b: [b, 6], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2336140, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:afa9f208:::smithi03829856-35:12, version: 67'105, data_included: [521050~698057,1806623~350519], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:afa9f208:::smithi03829856-35:12@67'105, size: 2521050, copy_subset: [(0, 2521050)], clone_subset: {}, snapset: 12=[]:{11: [11, d, b, 6], 12: [12]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2157142, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:aa3303a5:::smithi03829856-17:12, version: 69'113, data_included: [413472~893152,1808868~155424], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:aa3303a5:::smithi03829856-17:12@69'113, size: 2319366, copy_subset: [(0, 2319366)], clone_subset: {}, snapset: 14=[]:{b: [b, 6], 10: [d], 12: [12, 11], 14: [13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1964292, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ac9cd3a3:::smithi03829856-8:10, version: 70'117, data_included: [517649~620182,1776668~428394], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:ac9cd3a3:::smithi03829856-8:10@70'117, size: 3008420, copy_subset: [(0, 3008420)], clone_subset: {}, snapset: 10=[]:{10: [d, b, 6]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2205062, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ac9cd3a3:::smithi03829856-8:head, version: 70'118, data_included: [517649~620182,1776668~428394], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:ac9cd3a3:::smithi03829856-8:head@70'118, size: 3008420, copy_subset: [(0, 3008420)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2205062, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:aa3303a5:::smithi03829856-17:10, version: 70'119, data_included: [413472~893152,1818036~155424], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:aa3303a5:::smithi03829856-17:10@70'119, size: 3613472, copy_subset: [(0, 3613472)], clone_subset: {}, snapset: 14=[]:{b: [b, 6], 10: [d], 12: [12, 11], 14: [13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1973460, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ad3ad1b3:::smithi03829856-21:10, version: 72'122, data_included: [531121~742705,1711057~305871], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:ad3ad1b3:::smithi03829856-21:10@72'122, size: 3731121, copy_subset: [(0, 3731121)], clone_subset: {}, snapset: 10=[]:{b: [b, 6], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2016928, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:50.650 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.081994+0000", 2024-03-19T11:31:50.650 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 56.530935419000002, 2024-03-19T11:31:50.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.040292051000000002, 2024-03-19T11:31:50.650 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:50.650 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:50.651 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:50.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.651 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:50.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.081994+0000", 2024-03-19T11:31:50.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.651 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:50.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.081994+0000", 2024-03-19T11:31:50.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.651 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:50.652 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.081996+0000", 2024-03-19T11:31:50.652 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5069999999999999e-06 2024-03-19T11:31:50.652 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.652 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.652 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:50.652 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088193+0000", 2024-03-19T11:31:50.652 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061970209999999996 2024-03-19T11:31:50.652 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.652 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.652 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:50.652 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088252+0000", 2024-03-19T11:31:50.653 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8838999999999997e-05 2024-03-19T11:31:50.653 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.653 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.653 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:50.653 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088322+0000", 2024-03-19T11:31:50.653 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0455000000000002e-05 2024-03-19T11:31:50.653 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.653 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.653 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:50.653 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088660+0000", 2024-03-19T11:31:50.653 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033735299999999998 2024-03-19T11:31:50.653 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.653 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.653 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:50.653 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088727+0000", 2024-03-19T11:31:50.654 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7335999999999994e-05 2024-03-19T11:31:50.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.654 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:50.654 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.122286+0000", 2024-03-19T11:31:50.654 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033559539999999999 2024-03-19T11:31:50.654 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.654 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:50.654 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.654 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.15 e75,74 4:ac9cd3a3:::smithi03829856-8:4,49'71)", 2024-03-19T11:31:50.654 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.088380+0000", 2024-03-19T11:31:50.654 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 56.524549557, 2024-03-19T11:31:50.654 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.044894303000000003, 2024-03-19T11:31:50.655 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:50.655 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:50.655 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:50.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:50.655 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088380+0000", 2024-03-19T11:31:50.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:50.655 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088380+0000", 2024-03-19T11:31:50.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:50.656 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088381+0000", 2024-03-19T11:31:50.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0500000000000004e-07 2024-03-19T11:31:50.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.656 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:50.656 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088385+0000", 2024-03-19T11:31:50.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2370000000000003e-06 2024-03-19T11:31:50.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.656 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:50.656 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088392+0000", 2024-03-19T11:31:50.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4240000000000004e-06 2024-03-19T11:31:50.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.656 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:50.657 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088397+0000", 2024-03-19T11:31:50.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8829999999999997e-06 2024-03-19T11:31:50.657 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.657 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.657 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:50.657 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.122356+0000", 2024-03-19T11:31:50.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033958588999999997 2024-03-19T11:31:50.657 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.657 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.657 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:50.657 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.122364+0000", 2024-03-19T11:31:50.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0800000000000006e-06 2024-03-19T11:31:50.657 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.658 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:50.658 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.133275+0000", 2024-03-19T11:31:50.658 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010910585 2024-03-19T11:31:50.658 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.658 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:50.658 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.658 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.658 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:2286 4.15 e75/74 4:afa9f208:::smithi03829856-35:head v 75'131, mlcod=33'54)", 2024-03-19T11:31:50.658 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.089414+0000", 2024-03-19T11:31:50.658 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 56.523516317000002, 2024-03-19T11:31:50.658 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.045421256, 2024-03-19T11:31:50.659 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:50.659 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:50.659 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:50.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:50.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.089414+0000", 2024-03-19T11:31:50.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:50.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.089414+0000", 2024-03-19T11:31:50.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:50.660 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.089414+0000", 2024-03-19T11:31:50.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1100000000000005e-07 2024-03-19T11:31:50.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.660 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:50.660 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.089423+0000", 2024-03-19T11:31:50.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3799999999999994e-06 2024-03-19T11:31:50.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.660 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:50.661 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.089436+0000", 2024-03-19T11:31:50.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2931e-05 2024-03-19T11:31:50.661 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.661 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:50.661 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.089447+0000", 2024-03-19T11:31:50.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1044e-05 2024-03-19T11:31:50.661 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.661 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:50.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.133330+0000", 2024-03-19T11:31:50.662 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.043882833000000003 2024-03-19T11:31:50.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.662 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:50.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.133353+0000", 2024-03-19T11:31:50.662 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3629999999999999e-05 2024-03-19T11:31:50.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.662 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:50.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.134786+0000", 2024-03-19T11:31:50.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0014327980000000001 2024-03-19T11:31:50.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.663 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:50.663 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.134835+0000", 2024-03-19T11:31:50.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8829000000000003e-05 2024-03-19T11:31:50.663 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.663 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:50.663 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.663 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2802 4.28 4:15708482:::smithi03829856-23:head [write 497540~612664] snapc 19=[18,17,15,b] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-03-19T11:31:50.663 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:14.060002+0000", 2024-03-19T11:31:50.663 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 36.552928188000003, 2024-03-19T11:31:50.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.028374612, 2024-03-19T11:31:50.664 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:50.664 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:50.664 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:50.664 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:50.664 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:50.664 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2802 2024-03-19T11:31:50.664 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.664 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:50.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.664 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:50.664 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.060002+0000", 2024-03-19T11:31:50.664 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.664 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.665 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:50.665 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.060002+0000", 2024-03-19T11:31:50.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.665 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:50.665 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.060005+0000", 2024-03-19T11:31:50.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5130000000000001e-06 2024-03-19T11:31:50.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.665 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:50.665 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.061352+0000", 2024-03-19T11:31:50.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0013468060000000001 2024-03-19T11:31:50.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.665 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:50.666 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.061403+0000", 2024-03-19T11:31:50.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1181999999999999e-05 2024-03-19T11:31:50.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.666 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:50.666 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.061428+0000", 2024-03-19T11:31:50.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4715999999999998e-05 2024-03-19T11:31:50.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.666 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:50.666 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.061600+0000", 2024-03-19T11:31:50.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017224100000000001 2024-03-19T11:31:50.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op must wait for map", 2024-03-19T11:31:50.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.061619+0000", 2024-03-19T11:31:50.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9213000000000001e-05 2024-03-19T11:31:50.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:50.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.066953+0000", 2024-03-19T11:31:50.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053332149999999997 2024-03-19T11:31:50.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:50.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.067236+0000", 2024-03-19T11:31:50.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028381400000000002 2024-03-19T11:31:50.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:50.668 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.067698+0000", 2024-03-19T11:31:50.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00046152200000000002 2024-03-19T11:31:50.668 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.668 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.668 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:50.668 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.082795+0000", 2024-03-19T11:31:50.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015096729 2024-03-19T11:31:50.668 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.668 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.668 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:50.668 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.088033+0000", 2024-03-19T11:31:50.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052383259999999997 2024-03-19T11:31:50.668 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.668 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.668 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:50.669 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.088209+0000", 2024-03-19T11:31:50.669 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000175611 2024-03-19T11:31:50.669 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.669 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.669 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:50.669 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.088376+0000", 2024-03-19T11:31:50.669 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016772400000000001 2024-03-19T11:31:50.669 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.669 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:50.669 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.669 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.670 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.670 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2803 4.28 4:15708482:::smithi03829856-23:head [write 1840862~440060] snapc 19=[18,17,15,b] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-03-19T11:31:50.670 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:14.066423+0000", 2024-03-19T11:31:50.670 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 36.546507044999998, 2024-03-19T11:31:50.670 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022744533000000001, 2024-03-19T11:31:50.670 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:50.670 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:50.670 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:50.670 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:50.670 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:50.670 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2803 2024-03-19T11:31:50.671 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.671 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:50.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.671 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:50.671 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.066423+0000", 2024-03-19T11:31:50.671 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.671 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.671 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:50.671 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.066423+0000", 2024-03-19T11:31:50.671 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:50.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.066425+0000", 2024-03-19T11:31:50.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2500000000000001e-06 2024-03-19T11:31:50.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:50.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.067351+0000", 2024-03-19T11:31:50.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00092597000000000003 2024-03-19T11:31:50.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.673 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:50.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.067380+0000", 2024-03-19T11:31:50.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9326e-05 2024-03-19T11:31:50.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.673 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:50.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.067401+0000", 2024-03-19T11:31:50.674 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0458999999999998e-05 2024-03-19T11:31:50.674 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.674 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.674 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:50.674 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.078507+0000", 2024-03-19T11:31:50.674 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.01110651 2024-03-19T11:31:50.674 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.674 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.674 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:50.674 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.078694+0000", 2024-03-19T11:31:50.674 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000186334 2024-03-19T11:31:50.674 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.674 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.674 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:50.675 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.078997+0000", 2024-03-19T11:31:50.675 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00030362400000000001 2024-03-19T11:31:50.675 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.675 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.675 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:50.675 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.088954+0000", 2024-03-19T11:31:50.675 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0099565639999999993 2024-03-19T11:31:50.675 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.675 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.675 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:50.675 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.088978+0000", 2024-03-19T11:31:50.675 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4355999999999999e-05 2024-03-19T11:31:50.675 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.675 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.675 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:50.676 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.089067+0000", 2024-03-19T11:31:50.676 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8664000000000004e-05 2024-03-19T11:31:50.676 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.676 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.676 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:50.676 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.089167+0000", 2024-03-19T11:31:50.676 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000100476 2024-03-19T11:31:50.676 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.676 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:50.676 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.676 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.676 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.676 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3102 4.15 4:afa9f208:::smithi03829856-35:head [write 484101~692450,stat] snapc 1f=[1f,1e,1d,1c,1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-19T11:31:50.676 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.553012+0000", 2024-03-19T11:31:50.676 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.059917887000001, 2024-03-19T11:31:50.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.024256340000000001, 2024-03-19T11:31:50.677 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:50.677 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:50.677 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:50.677 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:50.677 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:50.677 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3102 2024-03-19T11:31:50.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.677 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:50.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.677 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:50.677 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.553012+0000", 2024-03-19T11:31:50.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.678 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.678 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.678 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:50.678 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.553012+0000", 2024-03-19T11:31:50.678 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.678 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.678 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.678 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:50.678 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.553015+0000", 2024-03-19T11:31:50.678 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5200000000000002e-06 2024-03-19T11:31:50.678 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.678 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.678 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:50.678 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.554342+0000", 2024-03-19T11:31:50.679 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0013261760000000001 2024-03-19T11:31:50.679 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.679 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:50.679 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.554377+0000", 2024-03-19T11:31:50.679 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5299000000000002e-05 2024-03-19T11:31:50.679 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.679 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:50.679 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.554401+0000", 2024-03-19T11:31:50.679 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4360000000000001e-05 2024-03-19T11:31:50.679 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.679 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:50.679 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.554595+0000", 2024-03-19T11:31:50.680 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019390499999999999 2024-03-19T11:31:50.680 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.680 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.680 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:50.680 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.554801+0000", 2024-03-19T11:31:50.680 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020614799999999999 2024-03-19T11:31:50.680 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.680 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.680 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:50.680 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.555182+0000", 2024-03-19T11:31:50.680 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00038094700000000002 2024-03-19T11:31:50.680 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.680 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.680 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:50.680 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.569800+0000", 2024-03-19T11:31:50.681 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0146179 2024-03-19T11:31:50.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.681 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:50.681 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.576922+0000", 2024-03-19T11:31:50.681 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0071215339999999997 2024-03-19T11:31:50.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.681 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:50.681 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.577114+0000", 2024-03-19T11:31:50.681 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019218100000000001 2024-03-19T11:31:50.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.681 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:50.681 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.577268+0000", 2024-03-19T11:31:50.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015437 2024-03-19T11:31:50.682 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.682 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:50.682 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.682 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.682 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3103 4.15 4:afa9f208:::smithi03829856-35:head [write 1636889~447212] snapc 1f=[1f,1e,1d,1c,1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-19T11:31:50.682 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.557970+0000", 2024-03-19T11:31:50.682 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.054959499999999, 2024-03-19T11:31:50.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033363528000000003, 2024-03-19T11:31:50.682 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:50.682 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:50.682 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:50.682 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:50.682 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:50.683 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3103 2024-03-19T11:31:50.683 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.683 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:50.683 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.683 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:50.683 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.557970+0000", 2024-03-19T11:31:50.683 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.683 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.683 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.683 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:50.683 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.557970+0000", 2024-03-19T11:31:50.683 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.683 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.683 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.683 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:50.684 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.557973+0000", 2024-03-19T11:31:50.684 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6060000000000001e-06 2024-03-19T11:31:50.684 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.684 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.684 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:50.684 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558806+0000", 2024-03-19T11:31:50.684 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00083305499999999997 2024-03-19T11:31:50.684 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.684 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.684 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:50.684 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558843+0000", 2024-03-19T11:31:50.684 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6519000000000001e-05 2024-03-19T11:31:50.684 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.684 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.684 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:50.685 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558863+0000", 2024-03-19T11:31:50.685 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0622999999999999e-05 2024-03-19T11:31:50.685 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.685 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.685 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:50.685 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.562831+0000", 2024-03-19T11:31:50.685 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039681250000000003 2024-03-19T11:31:50.685 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.685 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.685 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:50.685 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.563003+0000", 2024-03-19T11:31:50.685 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000171638 2024-03-19T11:31:50.685 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.685 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.685 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:50.686 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.563335+0000", 2024-03-19T11:31:50.686 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033165899999999999 2024-03-19T11:31:50.686 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.686 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.686 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:50.686 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.569815+0000", 2024-03-19T11:31:50.686 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00648024 2024-03-19T11:31:50.686 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.686 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.686 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:50.686 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.590982+0000", 2024-03-19T11:31:50.686 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021167235 2024-03-19T11:31:50.686 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.687 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.687 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:50.687 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.591170+0000", 2024-03-19T11:31:50.687 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018796899999999999 2024-03-19T11:31:50.687 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.687 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.687 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:50.687 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.591334+0000", 2024-03-19T11:31:50.687 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000163859 2024-03-19T11:31:50.687 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.687 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:50.687 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.687 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.687 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.687 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3104 4.15 4:afa9f208:::smithi03829856-35:head [setxattr _header (57) in=7b,truncate 2084101] snapc 1f=[1f,1e,1d,1c,1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-19T11:31:50.688 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.558905+0000", 2024-03-19T11:31:50.688 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.054024418000001, 2024-03-19T11:31:50.688 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032860320999999998, 2024-03-19T11:31:50.688 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:50.688 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:50.688 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:50.688 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:50.688 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:50.688 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3104 2024-03-19T11:31:50.688 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.688 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:50.688 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.688 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:50.688 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558905+0000", 2024-03-19T11:31:50.688 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.689 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.689 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.689 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:50.689 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558905+0000", 2024-03-19T11:31:50.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.689 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.689 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.689 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:50.689 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558907+0000", 2024-03-19T11:31:50.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.547e-06 2024-03-19T11:31:50.689 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.689 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.689 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:50.689 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558916+0000", 2024-03-19T11:31:50.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0589999999999992e-06 2024-03-19T11:31:50.689 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.690 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.690 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:50.690 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558931+0000", 2024-03-19T11:31:50.690 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5262999999999999e-05 2024-03-19T11:31:50.690 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.690 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.690 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:50.690 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558947+0000", 2024-03-19T11:31:50.690 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6140000000000001e-05 2024-03-19T11:31:50.690 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.690 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.690 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:50.690 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.567458+0000", 2024-03-19T11:31:50.690 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085106650000000006 2024-03-19T11:31:50.691 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.691 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.691 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:50.691 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.567777+0000", 2024-03-19T11:31:50.691 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00031917399999999998 2024-03-19T11:31:50.691 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.691 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.691 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:50.691 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.568090+0000", 2024-03-19T11:31:50.691 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00031286900000000002 2024-03-19T11:31:50.691 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.691 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.691 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:50.691 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.576738+0000", 2024-03-19T11:31:50.691 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0086476379999999992 2024-03-19T11:31:50.692 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.692 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.692 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:50.692 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.591474+0000", 2024-03-19T11:31:50.692 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014736281 2024-03-19T11:31:50.692 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.692 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.692 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:50.692 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.591576+0000", 2024-03-19T11:31:50.692 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000102303 2024-03-19T11:31:50.692 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.692 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.692 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:50.692 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.591766+0000", 2024-03-19T11:31:50.692 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018938200000000001 2024-03-19T11:31:50.693 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.693 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:50.693 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.693 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.693 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.693 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3105 4.15 4:afa9f208:::smithi03829856-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e104)", 2024-03-19T11:31:50.693 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.558970+0000", 2024-03-19T11:31:50.693 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.053959982999999, 2024-03-19T11:31:50.693 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033553684, 2024-03-19T11:31:50.693 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:50.693 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:50.693 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:50.693 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:50.693 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:50.693 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3105 2024-03-19T11:31:50.694 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.694 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:50.694 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.694 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:50.694 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558970+0000", 2024-03-19T11:31:50.694 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.694 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.694 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.694 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:50.694 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558970+0000", 2024-03-19T11:31:50.694 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.694 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.694 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.694 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:50.694 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558971+0000", 2024-03-19T11:31:50.695 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4729999999999999e-06 2024-03-19T11:31:50.695 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.695 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.695 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:50.695 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558978+0000", 2024-03-19T11:31:50.695 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1369999999999996e-06 2024-03-19T11:31:50.695 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.695 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.695 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:50.695 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.559003+0000", 2024-03-19T11:31:50.695 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4525999999999999e-05 2024-03-19T11:31:50.695 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.695 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.695 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:50.695 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.559019+0000", 2024-03-19T11:31:50.696 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5999999999999999e-05 2024-03-19T11:31:50.696 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.696 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.696 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:50.696 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.569615+0000", 2024-03-19T11:31:50.696 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010596257 2024-03-19T11:31:50.696 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.696 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.696 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-03-19T11:31:50.696 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.569763+0000", 2024-03-19T11:31:50.696 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000147319 2024-03-19T11:31:50.696 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.696 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.696 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:50.696 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.592019+0000", 2024-03-19T11:31:50.697 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022256597999999999 2024-03-19T11:31:50.697 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.697 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.697 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:50.697 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.592250+0000", 2024-03-19T11:31:50.697 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023081100000000001 2024-03-19T11:31:50.697 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.697 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.697 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:50.697 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.592524+0000", 2024-03-19T11:31:50.697 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000273563 2024-03-19T11:31:50.697 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.697 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:50.697 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.697 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.698 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.698 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3106 4.15 4:ad3ad1b3:::smithi03829856-21:head [write 500908~484052] snapc 1f=[1f,1e,1d,1c,1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-19T11:31:50.698 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.563910+0000", 2024-03-19T11:31:50.698 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.049020033000001, 2024-03-19T11:31:50.698 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.029166066000000001, 2024-03-19T11:31:50.698 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:50.698 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:50.698 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:50.698 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:50.698 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:50.698 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3106 2024-03-19T11:31:50.698 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.698 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:50.698 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.699 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:50.699 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.563910+0000", 2024-03-19T11:31:50.699 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.699 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.699 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.699 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:50.699 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.563910+0000", 2024-03-19T11:31:50.699 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.699 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.699 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.699 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:50.699 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.563913+0000", 2024-03-19T11:31:50.699 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9960000000000002e-06 2024-03-19T11:31:50.699 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.700 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.700 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:50.700 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.564853+0000", 2024-03-19T11:31:50.700 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00093991599999999995 2024-03-19T11:31:50.700 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.700 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.700 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:50.700 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.564886+0000", 2024-03-19T11:31:50.700 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3315999999999997e-05 2024-03-19T11:31:50.700 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.700 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.700 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:50.700 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.564908+0000", 2024-03-19T11:31:50.700 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2262000000000001e-05 2024-03-19T11:31:50.700 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.701 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.701 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:50.701 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.569861+0000", 2024-03-19T11:31:50.701 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0049529070000000003 2024-03-19T11:31:50.701 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.701 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.701 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:50.701 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.569998+0000", 2024-03-19T11:31:50.701 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000136546 2024-03-19T11:31:50.701 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.701 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.701 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:50.701 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.570279+0000", 2024-03-19T11:31:50.701 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028119899999999999 2024-03-19T11:31:50.701 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.702 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.702 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:50.702 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.586586+0000", 2024-03-19T11:31:50.702 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016306621 2024-03-19T11:31:50.702 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.702 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.702 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:50.702 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.592743+0000", 2024-03-19T11:31:50.702 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061571150000000003 2024-03-19T11:31:50.702 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.703 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.703 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:50.703 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.592915+0000", 2024-03-19T11:31:50.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017273200000000001 2024-03-19T11:31:50.703 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.703 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.703 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:50.703 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.593076+0000", 2024-03-19T11:31:50.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016045600000000001 2024-03-19T11:31:50.703 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.703 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:50.703 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.703 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.703 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.704 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3107 4.15 4:ad3ad1b3:::smithi03829856-21:head [write 1411724~689184] snapc 1f=[1f,1e,1d,1c,1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-19T11:31:50.704 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.570058+0000", 2024-03-19T11:31:50.704 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.042872315, 2024-03-19T11:31:50.704 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023258826, 2024-03-19T11:31:50.704 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:50.704 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:50.704 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:50.704 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:50.704 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:50.704 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3107 2024-03-19T11:31:50.705 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.705 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:50.705 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.705 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:50.705 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.570058+0000", 2024-03-19T11:31:50.705 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.705 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.705 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.705 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:50.706 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.570058+0000", 2024-03-19T11:31:50.706 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.706 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.706 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.706 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:50.706 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.570061+0000", 2024-03-19T11:31:50.706 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0469999999999998e-06 2024-03-19T11:31:50.706 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.706 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.706 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:50.707 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572012+0000", 2024-03-19T11:31:50.707 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0019516760000000001 2024-03-19T11:31:50.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.707 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:50.707 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572052+0000", 2024-03-19T11:31:50.707 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9353000000000001e-05 2024-03-19T11:31:50.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.707 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:50.708 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572076+0000", 2024-03-19T11:31:50.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4459000000000001e-05 2024-03-19T11:31:50.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.708 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.708 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:50.708 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.577408+0000", 2024-03-19T11:31:50.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053316329999999997 2024-03-19T11:31:50.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.708 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.708 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:50.708 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.577697+0000", 2024-03-19T11:31:50.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028967499999999998 2024-03-19T11:31:50.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.709 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.709 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:50.709 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.578148+0000", 2024-03-19T11:31:50.709 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00045085900000000001 2024-03-19T11:31:50.709 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.709 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.709 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:50.709 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.586737+0000", 2024-03-19T11:31:50.709 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085889060000000003 2024-03-19T11:31:50.709 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.709 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.709 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:50.709 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.593175+0000", 2024-03-19T11:31:50.709 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0064375090000000001 2024-03-19T11:31:50.709 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.710 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:50.710 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.593235+0000", 2024-03-19T11:31:50.710 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0553000000000001e-05 2024-03-19T11:31:50.710 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.710 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:50.710 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.593316+0000", 2024-03-19T11:31:50.710 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1155999999999993e-05 2024-03-19T11:31:50.710 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.710 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:50.710 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.710 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.711 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3108 4.15 4:ad3ad1b3:::smithi03829856-21:head [setxattr _header (57) in=7b,truncate 2100908] snapc 1f=[1f,1e,1d,1c,1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-19T11:31:50.711 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.572119+0000", 2024-03-19T11:31:50.711 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.040811131000002, 2024-03-19T11:31:50.711 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021614495000000001, 2024-03-19T11:31:50.711 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:50.711 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:50.711 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:50.711 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:50.711 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:50.711 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3108 2024-03-19T11:31:50.711 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.711 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:50.711 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.711 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:50.711 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572119+0000", 2024-03-19T11:31:50.712 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.712 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.712 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.712 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:50.712 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572119+0000", 2024-03-19T11:31:50.712 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.712 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.712 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.712 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:50.712 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572121+0000", 2024-03-19T11:31:50.712 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.103e-06 2024-03-19T11:31:50.712 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.712 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.712 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:50.712 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572130+0000", 2024-03-19T11:31:50.712 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.5230000000000001e-06 2024-03-19T11:31:50.713 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.713 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.713 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:50.713 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572157+0000", 2024-03-19T11:31:50.713 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6775999999999999e-05 2024-03-19T11:31:50.713 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.713 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.713 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:50.713 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572176+0000", 2024-03-19T11:31:50.713 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8516e-05 2024-03-19T11:31:50.713 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.713 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.713 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:50.713 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.584773+0000", 2024-03-19T11:31:50.714 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012596889 2024-03-19T11:31:50.714 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.714 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.714 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:50.714 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.585059+0000", 2024-03-19T11:31:50.714 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028658999999999998 2024-03-19T11:31:50.714 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.714 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.714 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:50.714 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.585591+0000", 2024-03-19T11:31:50.714 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00053221300000000002 2024-03-19T11:31:50.714 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.714 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.714 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:50.714 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.587798+0000", 2024-03-19T11:31:50.715 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0022066469999999999 2024-03-19T11:31:50.715 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.715 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.715 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:50.715 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.593611+0000", 2024-03-19T11:31:50.715 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0058127450000000002 2024-03-19T11:31:50.715 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.715 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.715 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:50.715 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.593671+0000", 2024-03-19T11:31:50.715 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0733000000000003e-05 2024-03-19T11:31:50.715 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.715 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.715 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:50.715 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.593733+0000", 2024-03-19T11:31:50.715 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1760000000000005e-05 2024-03-19T11:31:50.716 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.716 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:50.716 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.716 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.716 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.716 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3109 4.15 4:ad3ad1b3:::smithi03829856-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e104)", 2024-03-19T11:31:50.716 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.572203+0000", 2024-03-19T11:31:50.716 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.040726370000002, 2024-03-19T11:31:50.716 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021786163000000001, 2024-03-19T11:31:50.716 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:50.716 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:50.716 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:50.716 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:50.716 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:50.717 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3109 2024-03-19T11:31:50.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.717 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:50.717 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.717 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:50.717 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572203+0000", 2024-03-19T11:31:50.717 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.717 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.717 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:50.717 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572203+0000", 2024-03-19T11:31:50.717 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:50.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.717 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.717 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:50.718 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572205+0000", 2024-03-19T11:31:50.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.297e-06 2024-03-19T11:31:50.718 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.718 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.718 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:50.718 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572212+0000", 2024-03-19T11:31:50.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9129999999999997e-06 2024-03-19T11:31:50.718 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.718 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.718 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:50.718 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572233+0000", 2024-03-19T11:31:50.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1713e-05 2024-03-19T11:31:50.718 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.718 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.718 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:50.719 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.572251+0000", 2024-03-19T11:31:50.719 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7112000000000002e-05 2024-03-19T11:31:50.719 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.719 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:50.719 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.587006+0000", 2024-03-19T11:31:50.719 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014755024 2024-03-19T11:31:50.719 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.719 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-03-19T11:31:50.719 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.587242+0000", 2024-03-19T11:31:50.719 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023635099999999999 2024-03-19T11:31:50.719 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.719 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:50.720 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.593797+0000", 2024-03-19T11:31:50.720 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0065553520000000004 2024-03-19T11:31:50.720 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.720 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:50.720 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.593880+0000", 2024-03-19T11:31:50.721 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2529999999999998e-05 2024-03-19T11:31:50.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:50.721 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:50.721 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:50.721 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.593990+0000", 2024-03-19T11:31:50.721 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010987100000000001 2024-03-19T11:31:50.721 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.721 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:50.721 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.721 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:50.721 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:50.721 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:50.723 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-19T11:31:50.725 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:50.731+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:50.824 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:50.833+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:50.893 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:50.893 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:31:50.893 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:31:50.893 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:50.906 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-19T11:31:50.948 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:50.957+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:51.026 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:51.033+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:51.066 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:51.075+0000 7f70c4fdb640 -1 osd.0 129 handle_osd_map injecting map crc failure 2024-03-19T11:31:51.069 INFO:tasks.rados.rados.0.smithi038.stdout:1697: done (7 left) 2024-03-19T11:31:51.069 INFO:tasks.rados.rados.0.smithi038.stdout:1698: read oid 37 snap -1 2024-03-19T11:31:51.069 INFO:tasks.rados.rados.0.smithi038.stdout:1698: expect (ObjNum 236 snap 28 seq_num 236) 2024-03-19T11:31:51.069 INFO:tasks.rados.rados.0.smithi038.stdout:1699: set_chunk oid 12 target oid 2024-03-19T11:31:51.073 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:51.073 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:31:51.074 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:31:51.074 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:31:51.074 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:51.089 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-03-19T11:31:51.096 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-19T11:31:51.100 INFO:tasks.rados.rados.0.smithi038.stdout:1699: set_chunk oid 12 offset: 194048 length: 1670144 target oid 3559813227633c24d5fd86fc360a065e38f9f53f0bc9c926810ae8e314feec90 offset: 0 snap -1 2024-03-19T11:31:51.100 INFO:tasks.rados.rados.0.smithi038.stdout:1700: read oid 38 snap 37 2024-03-19T11:31:51.100 INFO:tasks.rados.rados.0.smithi038.stdout:1700: expect (ObjNum 256 snap 32 seq_num 256) 2024-03-19T11:31:51.100 INFO:tasks.rados.rados.0.smithi038.stdout:1701: set_chunk oid 1 target oid 2024-03-19T11:31:51.119 INFO:tasks.rados.rados.0.smithi038.stdout:1701: set_chunk oid 1 offset: 732672 length: 947712 target oid d2066257da49ede0b899178e5ef10e29ff81c3161d48b8bdd7a81e6bf2f992f1 offset: 0 snap -1 2024-03-19T11:31:51.119 INFO:tasks.rados.rados.0.smithi038.stdout:1702: write (excl) oid 45 current snap is 40 2024-03-19T11:31:51.119 INFO:tasks.rados.rados.0.smithi038.stdout:1691: got (22) Invalid argument 2024-03-19T11:31:51.119 INFO:tasks.rados.rados.0.smithi038.stdout:1702: seq_num 314 ranges {477398=798659,1774640=663444,2873055=404343} 2024-03-19T11:31:51.126 INFO:tasks.rados.rados.0.smithi038.stdout:1702: writing smithi03829856-45 from 477398 to 1276057 tid 1 2024-03-19T11:31:51.126 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:51.135+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:51.134 INFO:tasks.rados.rados.0.smithi038.stdout:1702: writing smithi03829856-45 from 1774640 to 2438084 tid 2 2024-03-19T11:31:51.142 INFO:tasks.rados.rados.0.smithi038.stdout:1702: writing smithi03829856-45 from 2873055 to 3277398 tid 3 2024-03-19T11:31:51.142 INFO:tasks.rados.rados.0.smithi038.stdout:1691: done (11 left) 2024-03-19T11:31:51.142 INFO:tasks.rados.rados.0.smithi038.stdout:1703: read oid 39 snap 36 2024-03-19T11:31:51.142 INFO:tasks.rados.rados.0.smithi038.stdout:1703: expect deleted 2024-03-19T11:31:51.142 INFO:tasks.rados.rados.0.smithi038.stdout:1704: set_chunk oid 35 target oid 2024-03-19T11:31:51.152 INFO:tasks.rados.rados.0.smithi038.stdout:1704: set_chunk oid 35 offset: 1194496 length: 1122304 target oid 98ce3b6006d6f2df0b19db7df1b66473ebe2ba56613bdbeb4a5bf25f32426ad6 offset: 0 snap -1 2024-03-19T11:31:51.152 INFO:tasks.rados.rados.0.smithi038.stdout:1705: read oid 16 snap -1 2024-03-19T11:31:51.152 INFO:tasks.rados.rados.0.smithi038.stdout:1705: expect (ObjNum 65 snap 3 seq_num 65) 2024-03-19T11:31:51.152 INFO:tasks.rados.rados.0.smithi038.stdout:1706: read oid 17 snap 24 2024-03-19T11:31:51.152 INFO:tasks.rados.rados.0.smithi038.stdout:1706: expect (ObjNum 213 snap 22 seq_num 213) 2024-03-19T11:31:51.152 INFO:tasks.rados.rados.0.smithi038.stdout:1707: write oid 29 current snap is 40 2024-03-19T11:31:51.152 INFO:tasks.rados.rados.0.smithi038.stdout:1690: expect (ObjNum 308 snap 39 seq_num 308) 2024-03-19T11:31:51.227 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:51.233+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:51.278 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:51.278 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:31:51.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:31:51.278 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:31:51.278 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.278 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:30 4.5 4:ac9cd3a3:::smithi03829856-8:head [write 1776668~340981] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:31:51.278 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:04.777922+0000", 2024-03-19T11:31:51.278 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 106.50799088700001, 2024-03-19T11:31:51.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015469894, 2024-03-19T11:31:51.279 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:51.279 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:51.279 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:51.279 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:51.279 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:51.279 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 30 2024-03-19T11:31:51.279 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.279 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:51.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.279 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:51.279 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.777922+0000", 2024-03-19T11:31:51.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.279 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.280 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:51.280 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.777922+0000", 2024-03-19T11:31:51.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.280 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:51.280 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.777924+0000", 2024-03-19T11:31:51.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3999999999999999e-06 2024-03-19T11:31:51.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.280 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:51.280 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778461+0000", 2024-03-19T11:31:51.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00053698299999999999 2024-03-19T11:31:51.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:51.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778487+0000", 2024-03-19T11:31:51.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5947000000000001e-05 2024-03-19T11:31:51.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:51.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778502+0000", 2024-03-19T11:31:51.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4256e-05 2024-03-19T11:31:51.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:51.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.781978+0000", 2024-03-19T11:31:51.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00347693 2024-03-19T11:31:51.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:51.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.782196+0000", 2024-03-19T11:31:51.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021726499999999999 2024-03-19T11:31:51.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:51.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.782572+0000", 2024-03-19T11:31:51.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00037635000000000001 2024-03-19T11:31:51.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:51.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.786960+0000", 2024-03-19T11:31:51.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043882130000000002 2024-03-19T11:31:51.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:51.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793113+0000", 2024-03-19T11:31:51.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061530760000000004 2024-03-19T11:31:51.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:51.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793276+0000", 2024-03-19T11:31:51.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000162652 2024-03-19T11:31:51.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:51.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793392+0000", 2024-03-19T11:31:51.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115822 2024-03-19T11:31:51.284 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.284 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:51.284 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.284 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:31 4.5 4:ac9cd3a3:::smithi03829856-8:head [setxattr _header (55) in=7b,truncate 2117649] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:31:51.284 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:04.778533+0000", 2024-03-19T11:31:51.284 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 106.50737975200001, 2024-03-19T11:31:51.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015182533, 2024-03-19T11:31:51.284 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:51.284 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:51.284 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:51.284 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:51.284 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:51.285 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 31 2024-03-19T11:31:51.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.285 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:51.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:51.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778533+0000", 2024-03-19T11:31:51.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:51.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778533+0000", 2024-03-19T11:31:51.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:51.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778534+0000", 2024-03-19T11:31:51.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1069999999999999e-06 2024-03-19T11:31:51.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:51.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778541+0000", 2024-03-19T11:31:51.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2529999999999998e-06 2024-03-19T11:31:51.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:51.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778558+0000", 2024-03-19T11:31:51.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6290000000000002e-05 2024-03-19T11:31:51.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.287 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:51.287 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778568+0000", 2024-03-19T11:31:51.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0360000000000001e-05 2024-03-19T11:31:51.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.287 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:51.287 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.785511+0000", 2024-03-19T11:31:51.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00694306 2024-03-19T11:31:51.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.287 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:51.287 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.785710+0000", 2024-03-19T11:31:51.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019837199999999999 2024-03-19T11:31:51.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.288 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:51.288 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.785929+0000", 2024-03-19T11:31:51.288 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021960799999999999 2024-03-19T11:31:51.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.288 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:51.288 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.787281+0000", 2024-03-19T11:31:51.288 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001352313 2024-03-19T11:31:51.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.288 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:51.288 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793492+0000", 2024-03-19T11:31:51.288 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0062107760000000003 2024-03-19T11:31:51.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:51.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793578+0000", 2024-03-19T11:31:51.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.5711999999999995e-05 2024-03-19T11:31:51.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:51.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793716+0000", 2024-03-19T11:31:51.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000137682 2024-03-19T11:31:51.289 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.289 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:51.289 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.290 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:32 4.5 4:ac9cd3a3:::smithi03829856-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:31:51.290 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:04.778582+0000", 2024-03-19T11:31:51.290 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 106.50733123800001, 2024-03-19T11:31:51.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015761827999999999, 2024-03-19T11:31:51.290 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:51.290 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:51.290 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:51.290 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:51.290 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:51.290 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 32 2024-03-19T11:31:51.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.290 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:51.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:51.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778582+0000", 2024-03-19T11:31:51.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:51.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778582+0000", 2024-03-19T11:31:51.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:51.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778583+0000", 2024-03-19T11:31:51.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.026e-06 2024-03-19T11:31:51.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.292 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:51.292 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778587+0000", 2024-03-19T11:31:51.292 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7870000000000001e-06 2024-03-19T11:31:51.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.292 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:51.292 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778599+0000", 2024-03-19T11:31:51.292 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1623e-05 2024-03-19T11:31:51.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.292 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:51.292 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778608+0000", 2024-03-19T11:31:51.292 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7399999999999993e-06 2024-03-19T11:31:51.292 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.293 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:51.293 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.787009+0000", 2024-03-19T11:31:51.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0084010490000000007 2024-03-19T11:31:51.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.293 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-03-19T11:31:51.293 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.787202+0000", 2024-03-19T11:31:51.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019309500000000001 2024-03-19T11:31:51.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.293 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:51.293 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793832+0000", 2024-03-19T11:31:51.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.006630052 2024-03-19T11:31:51.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.294 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:51.294 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.794048+0000", 2024-03-19T11:31:51.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000215505 2024-03-19T11:31:51.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.294 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:51.294 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.794343+0000", 2024-03-19T11:31:51.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029595100000000003 2024-03-19T11:31:51.294 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.294 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:51.294 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.295 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:891 4.5 4:af9d0d56:::smithi03829856-20:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:31:51.295 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:09.286410+0000", 2024-03-19T11:31:51.295 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 101.999502906, 2024-03-19T11:31:51.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035191405000000002, 2024-03-19T11:31:51.295 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:51.295 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:51.295 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:51.295 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:51.295 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:51.295 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 891 2024-03-19T11:31:51.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.295 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:51.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:51.296 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286410+0000", 2024-03-19T11:31:51.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.296 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:51.296 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286410+0000", 2024-03-19T11:31:51.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.296 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:51.296 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286413+0000", 2024-03-19T11:31:51.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2660000000000002e-06 2024-03-19T11:31:51.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:51.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286426+0000", 2024-03-19T11:31:51.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.258e-05 2024-03-19T11:31:51.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:51.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286460+0000", 2024-03-19T11:31:51.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3943999999999999e-05 2024-03-19T11:31:51.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:51.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286477+0000", 2024-03-19T11:31:51.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6799999999999998e-05 2024-03-19T11:31:51.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:51.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286605+0000", 2024-03-19T11:31:51.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000128892 2024-03-19T11:31:51.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:51.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286769+0000", 2024-03-19T11:31:51.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016359799999999999 2024-03-19T11:31:51.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:51.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.321601+0000", 2024-03-19T11:31:51.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.034832324999999997 2024-03-19T11:31:51.299 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.299 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:51.299 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.299 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1015 4.5 4:af9d0d56:::smithi03829856-20:head [tier-promote] snapc 6=[6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:31:51.299 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:12.761855+0000", 2024-03-19T11:31:51.299 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 98.524057380000002, 2024-03-19T11:31:51.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026760430000000002, 2024-03-19T11:31:51.299 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:51.299 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:51.299 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:51.300 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:51.300 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:51.300 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1015 2024-03-19T11:31:51.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.300 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:51.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.300 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:51.300 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761855+0000", 2024-03-19T11:31:51.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.300 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:51.300 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761855+0000", 2024-03-19T11:31:51.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.301 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:51.301 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761858+0000", 2024-03-19T11:31:51.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8140000000000001e-06 2024-03-19T11:31:51.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:51.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761872+0000", 2024-03-19T11:31:51.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3813e-05 2024-03-19T11:31:51.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:51.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761905+0000", 2024-03-19T11:31:51.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2642999999999999e-05 2024-03-19T11:31:51.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:51.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761925+0000", 2024-03-19T11:31:51.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0489e-05 2024-03-19T11:31:51.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:51.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.762070+0000", 2024-03-19T11:31:51.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000145153 2024-03-19T11:31:51.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:51.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.762365+0000", 2024-03-19T11:31:51.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029482100000000002 2024-03-19T11:31:51.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:51.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.788616+0000", 2024-03-19T11:31:51.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026250697 2024-03-19T11:31:51.304 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.304 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:51.304 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.304 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1121 4.5 4:aa3303a5:::smithi03829856-17:head [tier-promote] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:31:51.304 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:14.164281+0000", 2024-03-19T11:31:51.304 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 97.121632134999999, 2024-03-19T11:31:51.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023896435000000001, 2024-03-19T11:31:51.305 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:51.305 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:51.305 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:51.305 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:51.305 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:51.305 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1121 2024-03-19T11:31:51.305 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.305 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:51.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.305 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:51.305 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164281+0000", 2024-03-19T11:31:51.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:51.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164281+0000", 2024-03-19T11:31:51.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:51.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164283+0000", 2024-03-19T11:31:51.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8500000000000001e-06 2024-03-19T11:31:51.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:51.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164291+0000", 2024-03-19T11:31:51.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.14e-06 2024-03-19T11:31:51.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:51.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164313+0000", 2024-03-19T11:31:51.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.207e-05 2024-03-19T11:31:51.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:51.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164328+0000", 2024-03-19T11:31:51.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5403000000000001e-05 2024-03-19T11:31:51.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:51.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.167981+0000", 2024-03-19T11:31:51.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036529879999999998 2024-03-19T11:31:51.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:51.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.168274+0000", 2024-03-19T11:31:51.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029313100000000002 2024-03-19T11:31:51.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.309 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:51.309 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.188177+0000", 2024-03-19T11:31:51.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019902853000000002 2024-03-19T11:31:51.309 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.309 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:51.309 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.309 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1302 4.15 4:aa3303a5:::smithi03829856-17:head [write 413472~620140,stat] snapc b=[b,a,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-03-19T11:31:51.309 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:22.966426+0000", 2024-03-19T11:31:51.309 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 88.319486401999995, 2024-03-19T11:31:51.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018043513000000001, 2024-03-19T11:31:51.309 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:51.310 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:51.310 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:51.310 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:51.310 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:51.310 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1302 2024-03-19T11:31:51.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.310 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:51.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:51.310 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.966426+0000", 2024-03-19T11:31:51.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:51.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.966426+0000", 2024-03-19T11:31:51.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:51.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.966429+0000", 2024-03-19T11:31:51.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3829999999999999e-06 2024-03-19T11:31:51.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:51.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972117+0000", 2024-03-19T11:31:51.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0056885750000000004 2024-03-19T11:31:51.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:51.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972156+0000", 2024-03-19T11:31:51.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8878000000000003e-05 2024-03-19T11:31:51.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:51.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972184+0000", 2024-03-19T11:31:51.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7509999999999999e-05 2024-03-19T11:31:51.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:51.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972309+0000", 2024-03-19T11:31:51.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012525899999999999 2024-03-19T11:31:51.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:51.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972566+0000", 2024-03-19T11:31:51.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025740799999999999 2024-03-19T11:31:51.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:51.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972899+0000", 2024-03-19T11:31:51.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033238799999999998 2024-03-19T11:31:51.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:51.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984088+0000", 2024-03-19T11:31:51.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011189466 2024-03-19T11:31:51.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.314 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:51.314 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984138+0000", 2024-03-19T11:31:51.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9666999999999997e-05 2024-03-19T11:31:51.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.314 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:51.314 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984329+0000", 2024-03-19T11:31:51.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019086799999999999 2024-03-19T11:31:51.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.314 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:51.314 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984470+0000", 2024-03-19T11:31:51.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000141111 2024-03-19T11:31:51.315 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.315 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:51.315 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.315 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.1 50/49 [PushOp(4:866f3dcc:::smithi03829856-31:7, version: 38'44, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@38'44, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 45'62, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@45'62, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 46'68, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@46'68, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:7, version: 47'71, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@47'71, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 47'72, data_included: [440350~816629,1309629~231947], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@47'72, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:51.315 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.959457+0000", 2024-03-19T11:31:51.315 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 83.326455984999996, 2024-03-19T11:31:51.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018416666000000002, 2024-03-19T11:31:51.315 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:51.315 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:51.316 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:51.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:51.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.959457+0000", 2024-03-19T11:31:51.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:51.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.959457+0000", 2024-03-19T11:31:51.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:51.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.959458+0000", 2024-03-19T11:31:51.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4899999999999999e-06 2024-03-19T11:31:51.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:51.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.962822+0000", 2024-03-19T11:31:51.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0033640620000000001 2024-03-19T11:31:51.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:51.318 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.962893+0000", 2024-03-19T11:31:51.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1060000000000001e-05 2024-03-19T11:31:51.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.318 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:51.318 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.962977+0000", 2024-03-19T11:31:51.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3311000000000002e-05 2024-03-19T11:31:51.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.318 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:51.318 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963208+0000", 2024-03-19T11:31:51.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023121200000000001 2024-03-19T11:31:51.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:51.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963245+0000", 2024-03-19T11:31:51.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6844999999999997e-05 2024-03-19T11:31:51.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:51.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.977874+0000", 2024-03-19T11:31:51.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014628686 2024-03-19T11:31:51.319 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.319 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:51.319 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.320 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2284 4.15 4:afa9f208:::smithi03829856-35:head [read 1314816~702976] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-03-19T11:31:51.320 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:53.596775+0000", 2024-03-19T11:31:51.320 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.689137899000002, 2024-03-19T11:31:51.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.47260528299999999, 2024-03-19T11:31:51.320 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:51.320 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:51.320 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:51.320 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:51.320 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:51.320 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2284 2024-03-19T11:31:51.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.320 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:51.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:51.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596775+0000", 2024-03-19T11:31:51.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:51.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596775+0000", 2024-03-19T11:31:51.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:51.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596778+0000", 2024-03-19T11:31:51.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.174e-06 2024-03-19T11:31:51.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:51.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596794+0000", 2024-03-19T11:31:51.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5372000000000001e-05 2024-03-19T11:31:51.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:51.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596828+0000", 2024-03-19T11:31:51.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4107999999999999e-05 2024-03-19T11:31:51.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:51.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596844+0000", 2024-03-19T11:31:51.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6296000000000001e-05 2024-03-19T11:31:51.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.323 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:51.323 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596968+0000", 2024-03-19T11:31:51.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012383699999999999 2024-03-19T11:31:51.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.323 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:31:51.323 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596988+0000", 2024-03-19T11:31:51.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0645e-05 2024-03-19T11:31:51.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.323 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:51.323 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.068916+0000", 2024-03-19T11:31:51.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.47192744800000003 2024-03-19T11:31:51.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:51.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.069015+0000", 2024-03-19T11:31:51.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9500999999999995e-05 2024-03-19T11:31:51.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:51.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.069380+0000", 2024-03-19T11:31:51.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00036490200000000001 2024-03-19T11:31:51.324 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.324 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:51.324 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.324 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2286 4.15 4:afa9f208:::smithi03829856-35:head [assert-version v108,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e75)", 2024-03-19T11:31:51.325 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.084826+0000", 2024-03-19T11:31:51.325 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.201086531999998, 2024-03-19T11:31:51.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.055799556, 2024-03-19T11:31:51.325 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:51.325 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:51.325 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:51.325 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:51.325 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:51.325 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2286 2024-03-19T11:31:51.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.325 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:51.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:51.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084826+0000", 2024-03-19T11:31:51.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:51.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084826+0000", 2024-03-19T11:31:51.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:51.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084828+0000", 2024-03-19T11:31:51.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1409999999999999e-06 2024-03-19T11:31:51.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:51.327 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084838+0000", 2024-03-19T11:31:51.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3740000000000004e-06 2024-03-19T11:31:51.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:51.327 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084859+0000", 2024-03-19T11:31:51.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1644000000000001e-05 2024-03-19T11:31:51.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:51.327 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084871+0000", 2024-03-19T11:31:51.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1127e-05 2024-03-19T11:31:51.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:51.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084961+0000", 2024-03-19T11:31:51.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0371999999999995e-05 2024-03-19T11:31:51.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:51.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.085130+0000", 2024-03-19T11:31:51.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016884800000000001 2024-03-19T11:31:51.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5,7", 2024-03-19T11:31:51.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.089309+0000", 2024-03-19T11:31:51.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041788650000000004 2024-03-19T11:31:51.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:51.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.090362+0000", 2024-03-19T11:31:51.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001053362 2024-03-19T11:31:51.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:51.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.090511+0000", 2024-03-19T11:31:51.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014887200000000001 2024-03-19T11:31:51.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:51.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.140479+0000", 2024-03-19T11:31:51.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.049968006000000002 2024-03-19T11:31:51.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:51.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.140534+0000", 2024-03-19T11:31:51.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4917000000000003e-05 2024-03-19T11:31:51.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:51.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.140626+0000", 2024-03-19T11:31:51.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2027999999999994e-05 2024-03-19T11:31:51.330 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.330 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:51.330 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.331 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1d58db11:::smithi03829856-18:6, version: 55'80, data_included: [759683~639358,1904434~409218], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:6@55'80, size: 3159683, copy_subset: [(0, 3159683)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2313652, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1d58db11:::smithi03829856-18:b, version: 58'84, data_included: [443718~955323,1564138~93253], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:b@58'84, size: 3445402, copy_subset: [(0, 3445402)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1657391, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:10, version: 66'95, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:10@66'95, size: 3764735, copy_subset: [(0, 3764735)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:12, version: 66'97, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:12@66'97, size: 2504243, copy_subset: [(0, 2504243)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:c, version: 70'102, data_included: [416807~576167,1542129~472409], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:c@70'102, size: 2857157, copy_subset: [(0, 2857157)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2014538, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:51.331 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.274749+0000", 2024-03-19T11:31:51.331 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.011163601, 2024-03-19T11:31:51.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017652601, 2024-03-19T11:31:51.331 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:51.331 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:51.331 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:51.331 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.331 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:51.331 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.274749+0000", 2024-03-19T11:31:51.331 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.332 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.332 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:51.332 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.274749+0000", 2024-03-19T11:31:51.332 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.332 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.332 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:51.332 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.274751+0000", 2024-03-19T11:31:51.332 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8500000000000001e-06 2024-03-19T11:31:51.332 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.332 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.332 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:51.332 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.278426+0000", 2024-03-19T11:31:51.332 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036747559999999999 2024-03-19T11:31:51.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:51.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.278470+0000", 2024-03-19T11:31:51.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4119000000000002e-05 2024-03-19T11:31:51.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:51.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.278506+0000", 2024-03-19T11:31:51.333 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6066000000000003e-05 2024-03-19T11:31:51.333 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.333 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.333 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:51.333 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.278709+0000", 2024-03-19T11:31:51.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000202711 2024-03-19T11:31:51.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:51.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.278731+0000", 2024-03-19T11:31:51.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2657e-05 2024-03-19T11:31:51.334 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.334 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.334 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:51.334 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.292402+0000", 2024-03-19T11:31:51.334 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013670442 2024-03-19T11:31:51.334 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.334 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:51.335 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.335 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1dabc0f1:::smithi03829856-1:head, version: 70'103, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:head@70'103, size: 2621892, copy_subset: [(0, 2621892)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:51.335 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.280012+0000", 2024-03-19T11:31:51.335 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.005900955999998, 2024-03-19T11:31:51.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018569209, 2024-03-19T11:31:51.335 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:51.335 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:51.335 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:51.335 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.335 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:51.335 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280012+0000", 2024-03-19T11:31:51.335 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.335 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.336 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:51.336 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280012+0000", 2024-03-19T11:31:51.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.336 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:51.336 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280013+0000", 2024-03-19T11:31:51.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9899999999999999e-07 2024-03-19T11:31:51.336 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.336 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.336 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:51.336 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280584+0000", 2024-03-19T11:31:51.336 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057090299999999999 2024-03-19T11:31:51.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:51.337 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280612+0000", 2024-03-19T11:31:51.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8442999999999999e-05 2024-03-19T11:31:51.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:51.337 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280631+0000", 2024-03-19T11:31:51.337 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8896999999999999e-05 2024-03-19T11:31:51.337 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.337 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.337 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:51.337 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.295041+0000", 2024-03-19T11:31:51.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014410127999999999 2024-03-19T11:31:51.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.338 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.338 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:51.338 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.295050+0000", 2024-03-19T11:31:51.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2350000000000005e-06 2024-03-19T11:31:51.338 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.338 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.338 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:51.338 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.298581+0000", 2024-03-19T11:31:51.338 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035308039999999998 2024-03-19T11:31:51.338 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.338 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:51.338 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.339 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.18 e75,74 4:1d58db11:::smithi03829856-18:c,70'104)", 2024-03-19T11:31:51.339 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.280680+0000", 2024-03-19T11:31:51.339 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.005232481, 2024-03-19T11:31:51.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018106229000000001, 2024-03-19T11:31:51.339 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:51.339 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:51.339 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:51.339 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.339 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:51.339 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280680+0000", 2024-03-19T11:31:51.339 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.339 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.340 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:51.340 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280680+0000", 2024-03-19T11:31:51.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.340 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:51.340 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280681+0000", 2024-03-19T11:31:51.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2099999999999995e-07 2024-03-19T11:31:51.340 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.340 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.340 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:51.340 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280686+0000", 2024-03-19T11:31:51.340 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2240000000000003e-06 2024-03-19T11:31:51.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.341 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:51.341 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280698+0000", 2024-03-19T11:31:51.341 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1698999999999999e-05 2024-03-19T11:31:51.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.341 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:51.341 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280705+0000", 2024-03-19T11:31:51.341 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4350000000000002e-06 2024-03-19T11:31:51.341 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.341 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.341 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:51.341 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.298643+0000", 2024-03-19T11:31:51.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017938352000000001 2024-03-19T11:31:51.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.342 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:51.342 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.298656+0000", 2024-03-19T11:31:51.342 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3084999999999999e-05 2024-03-19T11:31:51.342 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.342 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.342 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:51.342 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.298787+0000", 2024-03-19T11:31:51.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013061299999999999 2024-03-19T11:31:51.343 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.343 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:51.343 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.343 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.343 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.5.0:143 4.18 e75/74 4:1d58db11:::smithi03829856-18:head v 75'106, mlcod=0'0)", 2024-03-19T11:31:51.343 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.282327+0000", 2024-03-19T11:31:51.343 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.003586110000001, 2024-03-19T11:31:51.343 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016753244, 2024-03-19T11:31:51.343 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:51.343 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:51.343 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:51.343 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.344 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:51.344 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.282327+0000", 2024-03-19T11:31:51.344 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.344 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.344 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.345 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:51.345 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.282327+0000", 2024-03-19T11:31:51.345 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:51.346 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.282328+0000", 2024-03-19T11:31:51.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.488e-06 2024-03-19T11:31:51.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:51.346 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.283127+0000", 2024-03-19T11:31:51.346 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00079853699999999999 2024-03-19T11:31:51.346 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.346 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.346 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:51.346 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.283152+0000", 2024-03-19T11:31:51.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5215000000000002e-05 2024-03-19T11:31:51.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.347 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.347 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:51.347 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.283167+0000", 2024-03-19T11:31:51.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4635e-05 2024-03-19T11:31:51.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.347 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.347 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:51.347 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.292633+0000", 2024-03-19T11:31:51.347 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0094664339999999993 2024-03-19T11:31:51.347 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.347 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.347 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:51.347 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.292655+0000", 2024-03-19T11:31:51.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1934999999999999e-05 2024-03-19T11:31:51.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:51.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.298981+0000", 2024-03-19T11:31:51.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0063255129999999996 2024-03-19T11:31:51.348 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.348 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.348 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:51.348 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.299080+0000", 2024-03-19T11:31:51.348 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9487000000000005e-05 2024-03-19T11:31:51.348 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.348 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:51.348 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.349 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.349 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2441 4.5 4:a38b9453:::smithi03829856-37:head [delete] snapc 17=[17,16,15,13,12,b,6] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e84)", 2024-03-19T11:31:51.349 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:03.060663+0000", 2024-03-19T11:31:51.349 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.225250266000003, 2024-03-19T11:31:51.349 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10917726, 2024-03-19T11:31:51.349 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:51.349 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:51.349 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:51.349 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:51.349 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:51.349 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2441 2024-03-19T11:31:51.349 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.349 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:51.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.350 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:51.350 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060663+0000", 2024-03-19T11:31:51.350 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.350 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:51.350 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060663+0000", 2024-03-19T11:31:51.350 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.350 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.350 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:51.350 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060666+0000", 2024-03-19T11:31:51.350 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.05e-06 2024-03-19T11:31:51.350 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:51.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060680+0000", 2024-03-19T11:31:51.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4276e-05 2024-03-19T11:31:51.351 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:51.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060714+0000", 2024-03-19T11:31:51.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4325000000000001e-05 2024-03-19T11:31:51.351 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.351 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.351 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:51.351 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060737+0000", 2024-03-19T11:31:51.351 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2959000000000002e-05 2024-03-19T11:31:51.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.352 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.352 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:51.352 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060895+0000", 2024-03-19T11:31:51.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000157787 2024-03-19T11:31:51.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.352 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.352 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:31:51.352 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060923+0000", 2024-03-19T11:31:51.352 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8019000000000001e-05 2024-03-19T11:31:51.352 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.352 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:51.353 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.066222+0000", 2024-03-19T11:31:51.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052989270000000002 2024-03-19T11:31:51.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:51.353 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.070153+0000", 2024-03-19T11:31:51.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039315119999999999 2024-03-19T11:31:51.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.353 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.353 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:51.353 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.147930+0000", 2024-03-19T11:31:51.353 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.077776459000000006 2024-03-19T11:31:51.353 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.354 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:51.354 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.148079+0000", 2024-03-19T11:31:51.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014888 2024-03-19T11:31:51.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.354 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:51.354 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.148370+0000", 2024-03-19T11:31:51.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029130999999999997 2024-03-19T11:31:51.354 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.354 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.354 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:51.354 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.152531+0000", 2024-03-19T11:31:51.354 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041605399999999999 2024-03-19T11:31:51.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.355 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:51.355 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.169762+0000", 2024-03-19T11:31:51.355 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017231483999999998 2024-03-19T11:31:51.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.355 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:51.355 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.169807+0000", 2024-03-19T11:31:51.355 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5161999999999999e-05 2024-03-19T11:31:51.355 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.355 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.355 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:51.355 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.169840+0000", 2024-03-19T11:31:51.356 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2570000000000002e-05 2024-03-19T11:31:51.356 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.356 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:51.356 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.356 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.356 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPushReply(4.5 84/83 [PushReplyOp(4:a38b9453:::smithi03829856-37:head)])", 2024-03-19T11:31:51.356 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:03.084896+0000", 2024-03-19T11:31:51.356 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.201017131999997, 2024-03-19T11:31:51.356 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.02071104, 2024-03-19T11:31:51.356 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:51.356 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-19T11:31:51.356 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:51.356 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.356 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:51.357 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.084896+0000", 2024-03-19T11:31:51.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.357 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.357 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:51.357 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.084896+0000", 2024-03-19T11:31:51.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.357 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.357 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:51.357 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.084897+0000", 2024-03-19T11:31:51.357 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.686e-06 2024-03-19T11:31:51.357 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.357 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.358 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:51.358 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.084910+0000", 2024-03-19T11:31:51.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2948e-05 2024-03-19T11:31:51.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.358 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:51.358 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.084936+0000", 2024-03-19T11:31:51.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5982e-05 2024-03-19T11:31:51.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.358 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.358 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:51.358 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.084955+0000", 2024-03-19T11:31:51.358 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8416000000000001e-05 2024-03-19T11:31:51.358 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:51.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.102822+0000", 2024-03-19T11:31:51.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017866756000000001 2024-03-19T11:31:51.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.359 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:51.359 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.105607+0000", 2024-03-19T11:31:51.359 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027852520000000002 2024-03-19T11:31:51.359 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.359 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:51.359 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.359 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.359 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.359 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.4.0:540 4.18 e111/86 4:1d58db11:::smithi03829856-18:head v 111'153, mlcod=108'150)", 2024-03-19T11:31:51.360 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:31.238185+0000", 2024-03-19T11:31:51.360 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 20.047727769000002, 2024-03-19T11:31:51.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021779390999999999, 2024-03-19T11:31:51.360 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:51.360 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:51.360 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:51.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:51.360 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.238185+0000", 2024-03-19T11:31:51.360 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.360 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.360 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.360 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:51.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.238185+0000", 2024-03-19T11:31:51.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:51.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.238188+0000", 2024-03-19T11:31:51.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4739999999999999e-06 2024-03-19T11:31:51.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.361 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:51.361 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.240837+0000", 2024-03-19T11:31:51.361 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026495189999999999 2024-03-19T11:31:51.361 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.361 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.362 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:51.362 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.240867+0000", 2024-03-19T11:31:51.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0363000000000001e-05 2024-03-19T11:31:51.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.362 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:51.362 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.240885+0000", 2024-03-19T11:31:51.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7153e-05 2024-03-19T11:31:51.362 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.362 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.362 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:51.362 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.241029+0000", 2024-03-19T11:31:51.362 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014455499999999999 2024-03-19T11:31:51.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.363 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.363 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:51.363 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.241069+0000", 2024-03-19T11:31:51.363 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9409000000000002e-05 2024-03-19T11:31:51.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.363 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.363 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:51.363 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.259833+0000", 2024-03-19T11:31:51.363 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018764306000000001 2024-03-19T11:31:51.363 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.363 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.364 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:51.364 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.259964+0000", 2024-03-19T11:31:51.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000131612 2024-03-19T11:31:51.364 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.364 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:51.364 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.364 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.364 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.364 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3282 4.5 4:a0a50d04:::smithi03829856-13:head [read 0~2201684,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e112)", 2024-03-19T11:31:51.364 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:32.240600+0000", 2024-03-19T11:31:51.364 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.045312565, 2024-03-19T11:31:51.364 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.84623566699999997, 2024-03-19T11:31:51.364 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:51.365 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:51.365 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:51.365 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:51.365 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:51.365 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3282 2024-03-19T11:31:51.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.365 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:51.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.365 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:51.365 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240600+0000", 2024-03-19T11:31:51.365 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.365 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.366 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:51.366 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240600+0000", 2024-03-19T11:31:51.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.366 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:51.366 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240602+0000", 2024-03-19T11:31:51.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0700000000000001e-06 2024-03-19T11:31:51.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.366 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:51.366 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240611+0000", 2024-03-19T11:31:51.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.969e-06 2024-03-19T11:31:51.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.367 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:51.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240630+0000", 2024-03-19T11:31:51.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8253e-05 2024-03-19T11:31:51.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.367 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:51.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240642+0000", 2024-03-19T11:31:51.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2666e-05 2024-03-19T11:31:51.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.367 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:51.368 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.241488+0000", 2024-03-19T11:31:51.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000845252 2024-03-19T11:31:51.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.368 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:31:51.368 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.241503+0000", 2024-03-19T11:31:51.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5797000000000001e-05 2024-03-19T11:31:51.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.368 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:51.368 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.083186+0000", 2024-03-19T11:31:51.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.84168299599999996 2024-03-19T11:31:51.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.369 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.369 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:51.369 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.083618+0000", 2024-03-19T11:31:51.369 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000432046 2024-03-19T11:31:51.369 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.369 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.369 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:51.369 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.086836+0000", 2024-03-19T11:31:51.369 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032176180000000002 2024-03-19T11:31:51.369 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.369 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:51.369 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.369 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.369 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.369 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3283 4.5 4:a0a50d04:::smithi03829856-13:head [sparse-read 0~1151947] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e112)", 2024-03-19T11:31:51.370 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:32.240669+0000", 2024-03-19T11:31:51.370 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.045243565, 2024-03-19T11:31:51.370 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.84791886900000002, 2024-03-19T11:31:51.370 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:51.370 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:51.370 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:51.370 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:51.370 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:51.370 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3283 2024-03-19T11:31:51.370 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.370 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:51.370 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.370 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:51.370 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240669+0000", 2024-03-19T11:31:51.371 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.371 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.371 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.371 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:51.371 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240669+0000", 2024-03-19T11:31:51.371 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.371 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.371 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.371 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:51.371 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240670+0000", 2024-03-19T11:31:51.371 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8199999999999998e-07 2024-03-19T11:31:51.371 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.371 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.371 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:51.372 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240675+0000", 2024-03-19T11:31:51.372 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0849999999999996e-06 2024-03-19T11:31:51.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.372 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:51.372 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240688+0000", 2024-03-19T11:31:51.372 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2901000000000001e-05 2024-03-19T11:31:51.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.372 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:51.372 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240696+0000", 2024-03-19T11:31:51.372 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2910000000000007e-06 2024-03-19T11:31:51.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.373 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:51.373 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.241573+0000", 2024-03-19T11:31:51.373 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00087639100000000002 2024-03-19T11:31:51.373 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.373 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.373 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:31:51.373 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.241587+0000", 2024-03-19T11:31:51.373 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4467999999999999e-05 2024-03-19T11:31:51.373 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.373 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.373 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:51.373 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.087901+0000", 2024-03-19T11:31:51.373 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.84631403500000002 2024-03-19T11:31:51.373 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.373 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.374 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:51.374 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.088089+0000", 2024-03-19T11:31:51.374 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000187171 2024-03-19T11:31:51.374 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.374 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:51.374 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.088588+0000", 2024-03-19T11:31:51.374 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00049964499999999995 2024-03-19T11:31:51.374 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.374 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:51.374 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.374 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.374 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3284 4.5 4:a0a50d04:::smithi03829856-13:head [sparse-read 0~1151947] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e112)", 2024-03-19T11:31:51.375 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:32.240710+0000", 2024-03-19T11:31:51.375 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 19.045203269000002, 2024-03-19T11:31:51.375 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.84860797899999996, 2024-03-19T11:31:51.375 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:51.375 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:51.375 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:51.375 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:51.375 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:51.375 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3284 2024-03-19T11:31:51.375 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.375 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:51.375 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.375 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:51.375 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240710+0000", 2024-03-19T11:31:51.375 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.376 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.376 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.376 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:51.376 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240710+0000", 2024-03-19T11:31:51.376 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.376 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.376 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.376 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:51.376 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240710+0000", 2024-03-19T11:31:51.376 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9599999999999999e-07 2024-03-19T11:31:51.376 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.376 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.376 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:51.376 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240714+0000", 2024-03-19T11:31:51.377 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8480000000000003e-06 2024-03-19T11:31:51.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.377 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.377 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:51.377 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240726+0000", 2024-03-19T11:31:51.377 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.186e-05 2024-03-19T11:31:51.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.377 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.377 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:51.377 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240736+0000", 2024-03-19T11:31:51.377 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0040999999999999e-05 2024-03-19T11:31:51.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.377 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.377 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:51.378 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.241633+0000", 2024-03-19T11:31:51.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00089736299999999998 2024-03-19T11:31:51.378 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.378 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.378 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:31:51.378 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.241645+0000", 2024-03-19T11:31:51.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1600000000000001e-05 2024-03-19T11:31:51.378 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.378 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.378 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:51.378 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.088667+0000", 2024-03-19T11:31:51.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.84702242800000005 2024-03-19T11:31:51.378 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.379 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.379 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:51.379 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.088837+0000", 2024-03-19T11:31:51.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000169949 2024-03-19T11:31:51.379 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.379 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.379 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:51.379 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.089318+0000", 2024-03-19T11:31:51.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00048019399999999998 2024-03-19T11:31:51.379 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.379 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:51.379 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.379 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.380 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:51.380 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:51.382 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-19T11:31:51.382 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:51.389+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:51.468 INFO:tasks.rados.rados.0.smithi038.stdout:1692: expect (ObjNum 271 snap 34 seq_num 271) 2024-03-19T11:31:51.482 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:51.489+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:51.552 INFO:tasks.rados.rados.0.smithi038.stdout:1672: finishing write tid 1 to smithi03829856-46 2024-03-19T11:31:51.553 INFO:tasks.rados.rados.0.smithi038.stdout:1672: finishing write tid 2 to smithi03829856-46 2024-03-19T11:31:51.553 INFO:tasks.rados.rados.0.smithi038.stdout:1672: finishing write tid 3 to smithi03829856-46 2024-03-19T11:31:51.553 INFO:tasks.rados.rados.0.smithi038.stdout:1672: finishing write tid 4 to smithi03829856-46 2024-03-19T11:31:51.553 INFO:tasks.rados.rados.0.smithi038.stdout:1672: finishing write tid 5 to smithi03829856-46 2024-03-19T11:31:51.553 INFO:tasks.rados.rados.0.smithi038.stdout:1672: oid 46 updating version 0 to 189 2024-03-19T11:31:51.553 INFO:tasks.rados.rados.0.smithi038.stdout:1672: oid 46 version 189 is already newer than 188 2024-03-19T11:31:51.553 INFO:tasks.rados.rados.0.smithi038.stdout:1672: oid 46 updating version 189 to 190 2024-03-19T11:31:51.553 INFO:tasks.rados.rados.0.smithi038.stdout:1672: oid 46 updating version 190 to 191 2024-03-19T11:31:51.553 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 46 v 191 (ObjNum 305 snap 39 seq_num 305) dirty exists 2024-03-19T11:31:51.553 INFO:tasks.rados.rados.0.smithi038.stdout:1672: left oid 46 (ObjNum 305 snap 39 seq_num 305) 2024-03-19T11:31:51.554 INFO:tasks.rados.rados.0.smithi038.stdout:1693: finishing write tid 1 to smithi03829856-41 2024-03-19T11:31:51.554 INFO:tasks.rados.rados.0.smithi038.stdout:1693: finishing write tid 2 to smithi03829856-41 2024-03-19T11:31:51.554 INFO:tasks.rados.rados.0.smithi038.stdout:1693: finishing write tid 3 to smithi03829856-41 2024-03-19T11:31:51.554 INFO:tasks.rados.rados.0.smithi038.stdout:1693: finishing write tid 4 to smithi03829856-41 2024-03-19T11:31:51.554 INFO:tasks.rados.rados.0.smithi038.stdout:1695: finishing write tid 1 to smithi03829856-4 2024-03-19T11:31:51.554 INFO:tasks.rados.rados.0.smithi038.stdout:1693: finishing write tid 5 to smithi03829856-41 2024-03-19T11:31:51.554 INFO:tasks.rados.rados.0.smithi038.stdout:1693: oid 41 updating version 0 to 195 2024-03-19T11:31:51.554 INFO:tasks.rados.rados.0.smithi038.stdout:1693: oid 41 version 195 is already newer than 193 2024-03-19T11:31:51.554 INFO:tasks.rados.rados.0.smithi038.stdout:1693: oid 41 version 195 is already newer than 194 2024-03-19T11:31:51.554 INFO:tasks.rados.rados.0.smithi038.stdout:1693: oid 41 updating version 195 to 196 2024-03-19T11:31:51.554 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 41 v 196 (ObjNum 311 snap 40 seq_num 311) dirty exists 2024-03-19T11:31:51.554 INFO:tasks.rados.rados.0.smithi038.stdout:1693: left oid 41 (ObjNum 311 snap 40 seq_num 311) 2024-03-19T11:31:51.554 INFO:tasks.rados.rados.0.smithi038.stdout:1695: finishing write tid 2 to smithi03829856-4 2024-03-19T11:31:51.555 INFO:tasks.rados.rados.0.smithi038.stdout:1695: finishing write tid 3 to smithi03829856-4 2024-03-19T11:31:51.555 INFO:tasks.rados.rados.0.smithi038.stdout:1695: finishing write tid 4 to smithi03829856-4 2024-03-19T11:31:51.555 INFO:tasks.rados.rados.0.smithi038.stdout:1695: finishing write tid 5 to smithi03829856-4 2024-03-19T11:31:51.555 INFO:tasks.rados.rados.0.smithi038.stdout:1695: oid 4 updating version 0 to 200 2024-03-19T11:31:51.555 INFO:tasks.rados.rados.0.smithi038.stdout:1695: oid 4 version 200 is already newer than 199 2024-03-19T11:31:51.555 INFO:tasks.rados.rados.0.smithi038.stdout:1695: oid 4 version 200 is already newer than 198 2024-03-19T11:31:51.555 INFO:tasks.rados.rados.0.smithi038.stdout:1695: oid 4 updating version 200 to 201 2024-03-19T11:31:51.555 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 4 v 201 (ObjNum 313 snap 40 seq_num 313) dirty exists 2024-03-19T11:31:51.555 INFO:tasks.rados.rados.0.smithi038.stdout:1695: left oid 4 (ObjNum 313 snap 40 seq_num 313) 2024-03-19T11:31:51.555 INFO:tasks.rados.rados.0.smithi038.stdout:1698: expect (ObjNum 236 snap 28 seq_num 236) 2024-03-19T11:31:51.556 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:51.556 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:31:51.556 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:31:51.556 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:51.566 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-19T11:31:51.608 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:51.617+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:51.680 INFO:tasks.rados.rados.0.smithi038.stdout:1700: expect (ObjNum 256 snap 32 seq_num 256) 2024-03-19T11:31:51.683 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:51.691+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:51.732 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:51.732 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:31:51.732 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:31:51.732 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:31:51.732 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:51.746 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-19T11:31:51.792 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:51.801+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:51.809 INFO:tasks.rados.rados.0.smithi038.stdout:1699: finishing set_chunk to oid 12 2024-03-19T11:31:51.810 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 12 v 105 (ObjNum 228 snap 26 seq_num 228) dirty exists 2024-03-19T11:31:51.810 INFO:tasks.rados.rados.0.smithi038.stdout:1701: finishing set_chunk to oid 1 2024-03-19T11:31:51.810 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 1 v 158 (ObjNum 262 snap 33 seq_num 262) dirty exists 2024-03-19T11:31:51.810 INFO:tasks.rados.rados.0.smithi038.stdout:1702: finishing write tid 1 to smithi03829856-45 2024-03-19T11:31:51.810 INFO:tasks.rados.rados.0.smithi038.stdout:1702: finishing write tid 2 to smithi03829856-45 2024-03-19T11:31:51.810 INFO:tasks.rados.rados.0.smithi038.stdout:1702: finishing write tid 3 to smithi03829856-45 2024-03-19T11:31:51.810 INFO:tasks.rados.rados.0.smithi038.stdout:1702: finishing write tid 4 to smithi03829856-45 2024-03-19T11:31:51.810 INFO:tasks.rados.rados.0.smithi038.stdout:1702: finishing write tid 5 to smithi03829856-45 2024-03-19T11:31:51.810 INFO:tasks.rados.rados.0.smithi038.stdout:1702: oid 45 updating version 0 to 88 2024-03-19T11:31:51.810 INFO:tasks.rados.rados.0.smithi038.stdout:1702: oid 45 updating version 88 to 90 2024-03-19T11:31:51.810 INFO:tasks.rados.rados.0.smithi038.stdout:1702: oid 45 version 90 is already newer than 87 2024-03-19T11:31:51.810 INFO:tasks.rados.rados.0.smithi038.stdout:1702: oid 45 version 90 is already newer than 89 2024-03-19T11:31:51.811 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 45 v 90 (ObjNum 314 snap 40 seq_num 314) dirty exists 2024-03-19T11:31:51.811 INFO:tasks.rados.rados.0.smithi038.stdout:1702: left oid 45 (ObjNum 314 snap 40 seq_num 314) 2024-03-19T11:31:51.811 INFO:tasks.rados.rados.0.smithi038.stdout:1705: expect (ObjNum 65 snap 3 seq_num 65) 2024-03-19T11:31:51.866 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:51.866 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:31:51.866 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:31:51.866 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:31:51.866 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.866 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:31:51.866 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:44.200920+0000", 2024-03-19T11:31:51.866 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 127.673005188, 2024-03-19T11:31:51.867 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.80881896499999995, 2024-03-19T11:31:51.867 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:51.867 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:51.867 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:51.867 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:31:51.867 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:31:51.867 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1 2024-03-19T11:31:51.867 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.867 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:51.867 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.867 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:51.867 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200920+0000", 2024-03-19T11:31:51.867 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.867 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.868 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.868 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:51.868 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200920+0000", 2024-03-19T11:31:51.868 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.868 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.868 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.868 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:51.868 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200923+0000", 2024-03-19T11:31:51.868 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6469999999999998e-06 2024-03-19T11:31:51.868 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.868 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.868 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:51.868 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200937+0000", 2024-03-19T11:31:51.868 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.399e-05 2024-03-19T11:31:51.869 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.869 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.869 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:51.869 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200960+0000", 2024-03-19T11:31:51.869 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2603e-05 2024-03-19T11:31:51.869 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.869 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.869 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:51.869 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200978+0000", 2024-03-19T11:31:51.869 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.791e-05 2024-03-19T11:31:51.869 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.869 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.869 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:51.869 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.201081+0000", 2024-03-19T11:31:51.869 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010344900000000001 2024-03-19T11:31:51.870 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.870 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.870 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:31:51.870 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.201096+0000", 2024-03-19T11:31:51.870 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4443e-05 2024-03-19T11:31:51.870 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.870 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.870 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:51.870 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.005978+0000", 2024-03-19T11:31:51.870 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.80488281500000003 2024-03-19T11:31:51.870 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.870 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.870 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:51.871 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.006267+0000", 2024-03-19T11:31:51.871 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000288688 2024-03-19T11:31:51.871 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.871 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.871 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:51.871 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.006534+0000", 2024-03-19T11:31:51.871 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026725799999999998 2024-03-19T11:31:51.871 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.871 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.871 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:51.871 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.008457+0000", 2024-03-19T11:31:51.871 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0019227820000000001 2024-03-19T11:31:51.871 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.871 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.871 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:51.872 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.009570+0000", 2024-03-19T11:31:51.872 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011131750000000001 2024-03-19T11:31:51.872 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.872 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.872 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:51.872 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.009670+0000", 2024-03-19T11:31:51.872 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9458999999999998e-05 2024-03-19T11:31:51.872 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.872 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.872 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:51.872 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.009739+0000", 2024-03-19T11:31:51.872 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9746000000000006e-05 2024-03-19T11:31:51.872 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.872 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:51.872 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.873 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.873 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.873 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:31:51.873 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.339898+0000", 2024-03-19T11:31:51.873 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 126.534028081, 2024-03-19T11:31:51.873 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019501793, 2024-03-19T11:31:51.873 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:51.873 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:51.873 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:51.873 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:31:51.873 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:31:51.873 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113 2024-03-19T11:31:51.873 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.873 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:51.873 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.874 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:51.874 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339898+0000", 2024-03-19T11:31:51.874 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.874 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.874 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.874 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:51.874 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339898+0000", 2024-03-19T11:31:51.874 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.874 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.874 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.874 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:51.874 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339899+0000", 2024-03-19T11:31:51.874 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.353e-06 2024-03-19T11:31:51.875 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.875 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.875 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:51.875 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339907+0000", 2024-03-19T11:31:51.875 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3000000000000002e-06 2024-03-19T11:31:51.875 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.875 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.875 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:51.875 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339924+0000", 2024-03-19T11:31:51.875 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6906999999999999e-05 2024-03-19T11:31:51.875 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.875 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.875 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:51.875 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339933+0000", 2024-03-19T11:31:51.875 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8969999999999997e-06 2024-03-19T11:31:51.876 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.876 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.876 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:51.876 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352239+0000", 2024-03-19T11:31:51.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012305814999999999 2024-03-19T11:31:51.876 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.876 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.876 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:51.876 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352466+0000", 2024-03-19T11:31:51.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022684100000000001 2024-03-19T11:31:51.876 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.876 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.876 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:51.876 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352816+0000", 2024-03-19T11:31:51.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00035030499999999997 2024-03-19T11:31:51.877 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.877 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.877 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:51.877 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.358956+0000", 2024-03-19T11:31:51.877 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061400079999999998 2024-03-19T11:31:51.877 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.877 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.877 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:51.877 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359118+0000", 2024-03-19T11:31:51.877 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016237599999999999 2024-03-19T11:31:51.877 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.877 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.877 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:51.877 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359287+0000", 2024-03-19T11:31:51.877 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000168488 2024-03-19T11:31:51.878 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.878 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.878 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:51.878 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359399+0000", 2024-03-19T11:31:51.878 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000112503 2024-03-19T11:31:51.878 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.878 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:51.878 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.878 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.878 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.878 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:31:51.878 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.339947+0000", 2024-03-19T11:31:51.878 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 126.533978314, 2024-03-19T11:31:51.878 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021923069999999999, 2024-03-19T11:31:51.879 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:51.879 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:51.879 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:51.879 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:31:51.879 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:31:51.879 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 114 2024-03-19T11:31:51.879 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.879 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:51.879 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.879 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:51.879 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339947+0000", 2024-03-19T11:31:51.879 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.879 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.879 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.880 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:51.880 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339947+0000", 2024-03-19T11:31:51.880 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.880 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.880 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:51.880 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339948+0000", 2024-03-19T11:31:51.880 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6000000000000003e-07 2024-03-19T11:31:51.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.880 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.880 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:51.880 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339956+0000", 2024-03-19T11:31:51.880 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1559999999999995e-06 2024-03-19T11:31:51.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.881 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.881 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:51.881 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339967+0000", 2024-03-19T11:31:51.881 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0669999999999999e-05 2024-03-19T11:31:51.881 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.881 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.881 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:51.881 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339975+0000", 2024-03-19T11:31:51.881 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0099999999999995e-06 2024-03-19T11:31:51.881 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.881 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.881 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:51.881 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.353672+0000", 2024-03-19T11:31:51.881 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013696583999999999 2024-03-19T11:31:51.881 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.882 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.882 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:51.882 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.353808+0000", 2024-03-19T11:31:51.882 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013652299999999999 2024-03-19T11:31:51.882 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.882 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.882 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:51.882 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.354036+0000", 2024-03-19T11:31:51.882 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022744799999999999 2024-03-19T11:31:51.882 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.882 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.882 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:51.882 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.360480+0000", 2024-03-19T11:31:51.882 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0064441630000000001 2024-03-19T11:31:51.882 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.883 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.883 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:51.883 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361722+0000", 2024-03-19T11:31:51.883 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001242754 2024-03-19T11:31:51.883 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.883 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.883 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:51.883 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361807+0000", 2024-03-19T11:31:51.883 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4605999999999996e-05 2024-03-19T11:31:51.883 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.883 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.883 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:51.883 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361870+0000", 2024-03-19T11:31:51.883 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3396000000000001e-05 2024-03-19T11:31:51.883 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.884 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:51.884 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.884 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.884 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.884 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:31:51.884 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.340017+0000", 2024-03-19T11:31:51.884 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 126.533908998, 2024-03-19T11:31:51.884 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022169355000000002, 2024-03-19T11:31:51.884 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:51.884 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:51.884 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:51.884 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:31:51.884 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:31:51.884 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 115 2024-03-19T11:31:51.884 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.885 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:51.885 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.885 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:51.885 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340017+0000", 2024-03-19T11:31:51.885 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.885 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.885 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.885 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:51.885 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340017+0000", 2024-03-19T11:31:51.885 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.885 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.885 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.885 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:51.885 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340018+0000", 2024-03-19T11:31:51.885 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.043e-06 2024-03-19T11:31:51.886 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.886 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.886 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:51.886 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340139+0000", 2024-03-19T11:31:51.886 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121502 2024-03-19T11:31:51.886 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.886 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.886 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:51.886 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340157+0000", 2024-03-19T11:31:51.886 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7382999999999999e-05 2024-03-19T11:31:51.886 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.886 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.886 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:51.886 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340166+0000", 2024-03-19T11:31:51.887 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2469999999999997e-06 2024-03-19T11:31:51.887 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.887 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.887 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:51.887 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.357470+0000", 2024-03-19T11:31:51.887 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017304119999999999 2024-03-19T11:31:51.887 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.887 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.887 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:51.887 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.357671+0000", 2024-03-19T11:31:51.887 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000200798 2024-03-19T11:31:51.887 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.887 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.887 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:51.887 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.358002+0000", 2024-03-19T11:31:51.888 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033090099999999998 2024-03-19T11:31:51.888 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.888 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.888 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:51.888 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361892+0000", 2024-03-19T11:31:51.888 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00389018 2024-03-19T11:31:51.888 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.888 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.888 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:51.888 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361983+0000", 2024-03-19T11:31:51.888 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0976000000000004e-05 2024-03-19T11:31:51.888 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.888 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.888 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:51.888 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362071+0000", 2024-03-19T11:31:51.889 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8022999999999994e-05 2024-03-19T11:31:51.889 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.889 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.889 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:51.889 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362186+0000", 2024-03-19T11:31:51.889 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115182 2024-03-19T11:31:51.889 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.889 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:51.889 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.889 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.889 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.889 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:31:51.889 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.340183+0000", 2024-03-19T11:31:51.889 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 126.533742166, 2024-03-19T11:31:51.889 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022279841000000002, 2024-03-19T11:31:51.889 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:51.890 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:51.890 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:51.890 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:31:51.890 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:31:51.890 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 116 2024-03-19T11:31:51.890 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.890 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:51.890 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.890 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:51.890 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340183+0000", 2024-03-19T11:31:51.890 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.890 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.890 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.890 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:51.891 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340183+0000", 2024-03-19T11:31:51.891 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.891 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.891 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.891 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:51.891 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340184+0000", 2024-03-19T11:31:51.891 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9000000000000005e-07 2024-03-19T11:31:51.891 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.891 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.891 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:51.891 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340191+0000", 2024-03-19T11:31:51.891 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0360000000000003e-06 2024-03-19T11:31:51.891 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.891 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.891 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:51.892 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340201+0000", 2024-03-19T11:31:51.892 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0844e-05 2024-03-19T11:31:51.892 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.892 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.892 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:51.892 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340209+0000", 2024-03-19T11:31:51.892 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7360000000000001e-06 2024-03-19T11:31:51.892 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.892 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.892 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:51.892 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359535+0000", 2024-03-19T11:31:51.892 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019325546999999998 2024-03-19T11:31:51.892 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.892 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.892 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:51.893 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359672+0000", 2024-03-19T11:31:51.893 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013706600000000001 2024-03-19T11:31:51.893 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.893 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:51.893 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359905+0000", 2024-03-19T11:31:51.893 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023334500000000001 2024-03-19T11:31:51.893 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.893 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:51.893 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362306+0000", 2024-03-19T11:31:51.893 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002401041 2024-03-19T11:31:51.893 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:51.894 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362329+0000", 2024-03-19T11:31:51.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3283999999999999e-05 2024-03-19T11:31:51.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:51.894 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362402+0000", 2024-03-19T11:31:51.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2459000000000005e-05 2024-03-19T11:31:51.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:51.894 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362463+0000", 2024-03-19T11:31:51.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1493000000000002e-05 2024-03-19T11:31:51.894 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.894 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:51.895 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.895 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.895 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.895 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:31:51.895 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.340222+0000", 2024-03-19T11:31:51.895 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 126.533703583, 2024-03-19T11:31:51.895 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022500218999999998, 2024-03-19T11:31:51.895 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:51.895 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:51.895 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:51.895 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:31:51.895 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:31:51.895 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 117 2024-03-19T11:31:51.895 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.895 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:51.896 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.896 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:51.896 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340222+0000", 2024-03-19T11:31:51.896 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.896 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.896 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.896 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:51.896 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340222+0000", 2024-03-19T11:31:51.896 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.896 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.896 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.896 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:51.896 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340223+0000", 2024-03-19T11:31:51.896 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7999999999999995e-07 2024-03-19T11:31:51.896 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.897 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.897 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:51.897 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340229+0000", 2024-03-19T11:31:51.897 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0399999999999998e-06 2024-03-19T11:31:51.897 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.897 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.897 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:51.897 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340239+0000", 2024-03-19T11:31:51.897 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0190000000000001e-05 2024-03-19T11:31:51.897 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.897 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.897 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:51.897 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340250+0000", 2024-03-19T11:31:51.897 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0720000000000001e-05 2024-03-19T11:31:51.898 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.898 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.898 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:51.898 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.360587+0000", 2024-03-19T11:31:51.898 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020337542 2024-03-19T11:31:51.898 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.898 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.898 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:51.898 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.360818+0000", 2024-03-19T11:31:51.898 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023120500000000001 2024-03-19T11:31:51.898 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.898 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.898 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:51.898 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361218+0000", 2024-03-19T11:31:51.898 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000399637 2024-03-19T11:31:51.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.899 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.899 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:51.899 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362483+0000", 2024-03-19T11:31:51.899 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012649200000000001 2024-03-19T11:31:51.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.899 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.899 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:51.899 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362538+0000", 2024-03-19T11:31:51.899 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4490000000000002e-05 2024-03-19T11:31:51.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.899 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.899 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:51.899 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362617+0000", 2024-03-19T11:31:51.899 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9839000000000006e-05 2024-03-19T11:31:51.900 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.900 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.900 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:51.900 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362722+0000", 2024-03-19T11:31:51.900 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104956 2024-03-19T11:31:51.900 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.900 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:51.900 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.900 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.900 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.900 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.15 51/50 [PushOp(4:afa9f208:::smithi03829856-35:head, version: 28'46, data_included: [571438~623878,1806623~424698], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:afa9f208:::smithi03829856-35:head@28'46, size: 2571438, copy_subset: [(0, 2571438)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2231321, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:af9d0d56:::smithi03829856-20:head, version: 35'55, data_included: [655360~589824,1900544~418789], data_size: 1008613, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:af9d0d56:::smithi03829856-20:head@35'55, size: 2319333, copy_subset: [(0, 2319333)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2319333, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ad3ad1b3:::smithi03829856-21:b, version: 47'66, data_included: [736140~537686,2020581~315559], data_size: 853245, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:ad3ad1b3:::smithi03829856-21:b@47'66, size: 2336140, copy_subset: [(0, 2336140)], clone_subset: {}, snapset: b=[]:{b: [b, a, 6, 5]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2336140, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ad3ad1b3:::smithi03829856-21:head, version: 47'70, data_included: [531121~742705,1711057~305871], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:ad3ad1b3:::smithi03829856-21:head@47'70, size: 3731121, copy_subset: [(0, 3731121)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2016928, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ac9cd3a3:::smithi03829856-8:head, version: 49'72, data_included: [517649~620182,1776668~428394], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:ac9cd3a3:::smithi03829856-8:head@49'72, size: 3008420, copy_subset: [(0, 3008420)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2205062, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:aa3303a5:::smithi03829856-17:b, version: 49'73, data_included: [614400~692224,1903006~356352], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:aa3303a5:::smithi03829856-17:b@49'73, size: 2268912, copy_subset: [(0, 2268912)], clone_subset: {}, snapset: b=[]:{b: [b, a, 6, 5]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2259358, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:aa3303a5:::smithi03829856-17:head, version: 49'74, data_included: [413472~893152,1818036~155424], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:aa3303a5:::smithi03829856-17:head@49'74, size: 3613472, copy_subset: [(0, 3613472)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1973460, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:51.900 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:28.058610+0000", 2024-03-19T11:31:51.900 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 83.815315411, 2024-03-19T11:31:51.900 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027100032, 2024-03-19T11:31:51.901 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:51.901 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:51.901 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:51.901 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.901 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:51.901 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.058610+0000", 2024-03-19T11:31:51.901 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.901 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.901 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.901 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:51.901 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.058610+0000", 2024-03-19T11:31:51.901 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.901 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.901 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.901 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:51.902 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.058611+0000", 2024-03-19T11:31:51.902 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1599999999999999e-06 2024-03-19T11:31:51.902 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.902 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.902 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:51.902 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.063874+0000", 2024-03-19T11:31:51.902 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052625270000000004 2024-03-19T11:31:51.902 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.902 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.902 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:51.902 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.063935+0000", 2024-03-19T11:31:51.902 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0594000000000003e-05 2024-03-19T11:31:51.902 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.902 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.902 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:51.903 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064024+0000", 2024-03-19T11:31:51.903 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9273999999999999e-05 2024-03-19T11:31:51.903 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.903 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.903 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:51.903 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064385+0000", 2024-03-19T11:31:51.903 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00036129100000000002 2024-03-19T11:31:51.903 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.903 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.903 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:51.903 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064437+0000", 2024-03-19T11:31:51.903 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2346000000000003e-05 2024-03-19T11:31:51.903 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.903 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.903 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:51.904 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085710+0000", 2024-03-19T11:31:51.904 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021272840000000001 2024-03-19T11:31:51.904 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.904 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:51.904 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.904 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.904 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.904 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.15 e51,50 4:ac9cd3a3:::smithi03829856-8:4,49'71)", 2024-03-19T11:31:51.904 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:28.064112+0000", 2024-03-19T11:31:51.904 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 83.809813585000001, 2024-03-19T11:31:51.904 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026480417999999999, 2024-03-19T11:31:51.904 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:51.904 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:51.904 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:51.904 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.905 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:51.905 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064112+0000", 2024-03-19T11:31:51.905 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.905 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.905 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.905 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:51.905 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064112+0000", 2024-03-19T11:31:51.905 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.905 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.905 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.905 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:51.905 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064113+0000", 2024-03-19T11:31:51.905 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3600000000000003e-07 2024-03-19T11:31:51.905 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.905 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.906 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:51.906 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064118+0000", 2024-03-19T11:31:51.906 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0549999999999999e-06 2024-03-19T11:31:51.906 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.906 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.906 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:51.906 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064128+0000", 2024-03-19T11:31:51.906 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0349e-05 2024-03-19T11:31:51.906 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.906 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.906 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:51.906 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064136+0000", 2024-03-19T11:31:51.906 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6249999999999998e-06 2024-03-19T11:31:51.906 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.907 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.907 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:51.907 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085773+0000", 2024-03-19T11:31:51.907 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021636829999999999 2024-03-19T11:31:51.907 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.907 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.907 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:51.907 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085788+0000", 2024-03-19T11:31:51.907 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5512999999999999e-05 2024-03-19T11:31:51.907 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.908 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.908 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:51.908 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.090592+0000", 2024-03-19T11:31:51.908 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0048044100000000003 2024-03-19T11:31:51.908 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.908 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:51.908 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.908 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.908 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.908 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.5 51/50 [PushOp(4:a38b9453:::smithi03829856-37:head, version: 28'45, data_included: [605052~774376,1923587~274200], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:a38b9453:::smithi03829856-37:head@28'45, size: 2605052, copy_subset: [(0, 2605052)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2197787, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:a0a50d04:::smithi03829856-13:b, version: 47'63, data_included: [601684~619341,1669078~429235], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a0a50d04:::smithi03829856-13:b@47'63, size: 2201684, copy_subset: [(0, 2201684)], clone_subset: {}, snapset: b=[]:{b: [b, a, 6, 5]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2098313, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:51.908 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:28.064324+0000", 2024-03-19T11:31:51.908 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 83.809601740999994, 2024-03-19T11:31:51.908 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027179020000000002, 2024-03-19T11:31:51.908 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:51.908 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:51.909 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:51.909 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.909 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:51.909 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064324+0000", 2024-03-19T11:31:51.909 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.909 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.909 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.909 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:51.909 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064324+0000", 2024-03-19T11:31:51.909 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.909 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.909 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.909 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:51.909 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064324+0000", 2024-03-19T11:31:51.910 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1199999999999998e-07 2024-03-19T11:31:51.910 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.910 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.910 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:51.910 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.065527+0000", 2024-03-19T11:31:51.910 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012028939999999999 2024-03-19T11:31:51.910 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.910 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.910 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:51.910 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.065552+0000", 2024-03-19T11:31:51.910 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4440999999999999e-05 2024-03-19T11:31:51.910 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.910 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.910 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:51.911 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.065568+0000", 2024-03-19T11:31:51.911 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6498e-05 2024-03-19T11:31:51.911 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.911 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.911 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:51.911 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085812+0000", 2024-03-19T11:31:51.911 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020243954000000002 2024-03-19T11:31:51.911 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.911 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.911 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:51.911 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085825+0000", 2024-03-19T11:31:51.911 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3298999999999999e-05 2024-03-19T11:31:51.911 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.911 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.911 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:51.912 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.091503+0000", 2024-03-19T11:31:51.912 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.005677522 2024-03-19T11:31:51.912 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.912 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:51.912 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.912 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.912 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.912 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.24 75/74 [PushOp(4:27909373:::smithi03829856-46:8, version: 46'59, data_included: [756315~767970,1933572~280606], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:8@46'59, size: 2756315, copy_subset: [(0, 2756315)], clone_subset: {}, snapset: 10=[]:{8: [6], c: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2214178, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:266590a5:::smithi03829856-16:e, version: 70'107, data_included: [652105~662359,1728171~386217], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:e@70'107, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 10=[]:{e: [d, b, 6]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:266590a5:::smithi03829856-16:head, version: 70'108, data_included: [652105~662359,1728171~386217], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:head@70'108, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:27909373:::smithi03829856-46:c, version: 70'109, data_included: [628595~895690,1807440~152886], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:c@70'109, size: 3428595, copy_subset: [(0, 3428595)], clone_subset: {}, snapset: 10=[]:{8: [6], c: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1960326, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:27909373:::smithi03829856-46:head, version: 70'110, data_included: [403401~454961,1584925~506039], data_size: 961000, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:head@70'110, size: 2403401, copy_subset: [(0, 2403401)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2403401, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:51.912 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.099172+0000", 2024-03-19T11:31:51.912 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.774753306999997, 2024-03-19T11:31:51.912 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.024528580000000001, 2024-03-19T11:31:51.912 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:51.912 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:51.912 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:51.913 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.913 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:51.913 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.099172+0000", 2024-03-19T11:31:51.913 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.913 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.913 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.913 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:51.913 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.099172+0000", 2024-03-19T11:31:51.913 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.913 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.913 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.913 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:51.913 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.099175+0000", 2024-03-19T11:31:51.913 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7750000000000001e-06 2024-03-19T11:31:51.913 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.913 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.914 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:51.914 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105219+0000", 2024-03-19T11:31:51.914 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0060440789999999999 2024-03-19T11:31:51.914 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.914 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.914 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:51.914 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105285+0000", 2024-03-19T11:31:51.914 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6260000000000006e-05 2024-03-19T11:31:51.914 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.914 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.914 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:51.914 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105366+0000", 2024-03-19T11:31:51.914 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.038e-05 2024-03-19T11:31:51.914 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.915 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.915 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:51.915 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105637+0000", 2024-03-19T11:31:51.915 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027085199999999997 2024-03-19T11:31:51.915 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.915 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.915 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:51.915 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105679+0000", 2024-03-19T11:31:51.915 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2088000000000001e-05 2024-03-19T11:31:51.915 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.915 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.915 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:51.915 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.123701+0000", 2024-03-19T11:31:51.915 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018022145999999999 2024-03-19T11:31:51.915 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.916 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:51.916 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.916 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.916 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.916 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.24 e75,74 4:266590a5:::smithi03829856-16:4,47'63,4:27909373:::smithi03829856-46:5,55'79,4:266590a5:::smithi03829856-16:10,67'105)", 2024-03-19T11:31:51.916 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.105437+0000", 2024-03-19T11:31:51.916 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 57.768488304000002, 2024-03-19T11:31:51.916 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.01860726, 2024-03-19T11:31:51.916 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:51.916 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:51.916 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:51.916 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.916 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:51.916 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105437+0000", 2024-03-19T11:31:51.916 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.917 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.917 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.917 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:51.917 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105437+0000", 2024-03-19T11:31:51.917 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.917 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.917 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.917 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:51.917 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105438+0000", 2024-03-19T11:31:51.917 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6799999999999999e-07 2024-03-19T11:31:51.917 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.917 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.917 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:51.917 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105445+0000", 2024-03-19T11:31:51.917 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3749999999999999e-06 2024-03-19T11:31:51.918 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.918 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.918 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:51.918 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105454+0000", 2024-03-19T11:31:51.918 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7909999999999999e-06 2024-03-19T11:31:51.918 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.918 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.918 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:51.918 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105465+0000", 2024-03-19T11:31:51.918 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.045e-05 2024-03-19T11:31:51.918 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.918 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.918 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:51.918 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.123753+0000", 2024-03-19T11:31:51.918 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0182884 2024-03-19T11:31:51.919 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.919 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.919 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:51.919 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.123765+0000", 2024-03-19T11:31:51.919 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2143e-05 2024-03-19T11:31:51.919 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.919 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.919 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:51.919 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.124045+0000", 2024-03-19T11:31:51.919 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027923299999999999 2024-03-19T11:31:51.919 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.919 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:51.919 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.919 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.919 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.920 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:3103 4.15 e104/76 4:afa9f208:::smithi03829856-35:head v 104'221, mlcod=101'216)", 2024-03-19T11:31:51.920 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.563593+0000", 2024-03-19T11:31:51.920 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.310333025999999, 2024-03-19T11:31:51.920 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026983566, 2024-03-19T11:31:51.920 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:51.920 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:51.920 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:51.920 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.920 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:51.920 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.563593+0000", 2024-03-19T11:31:51.920 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.920 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.920 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.920 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:51.920 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.563593+0000", 2024-03-19T11:31:51.921 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.921 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.921 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.921 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:51.921 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.563595+0000", 2024-03-19T11:31:51.921 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2859999999999998e-06 2024-03-19T11:31:51.921 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.921 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.921 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:51.921 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.564339+0000", 2024-03-19T11:31:51.921 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00074400099999999999 2024-03-19T11:31:51.921 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.921 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.921 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:51.921 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.564376+0000", 2024-03-19T11:31:51.922 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6628999999999998e-05 2024-03-19T11:31:51.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.922 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:51.922 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.564396+0000", 2024-03-19T11:31:51.922 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0596000000000001e-05 2024-03-19T11:31:51.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.922 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:51.922 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.566600+0000", 2024-03-19T11:31:51.922 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002204021 2024-03-19T11:31:51.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.923 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:51.923 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.566642+0000", 2024-03-19T11:31:51.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1902e-05 2024-03-19T11:31:51.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.923 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:51.923 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.590494+0000", 2024-03-19T11:31:51.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023851816000000001 2024-03-19T11:31:51.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.923 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:51.923 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.590576+0000", 2024-03-19T11:31:51.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2315e-05 2024-03-19T11:31:51.923 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.924 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:51.924 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.924 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.924 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:3104 4.15 e104/76 4:afa9f208:::smithi03829856-35:head v 104'222, mlcod=101'216)", 2024-03-19T11:31:51.924 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.568280+0000", 2024-03-19T11:31:51.924 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.305645182999999, 2024-03-19T11:31:51.924 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022345304999999999, 2024-03-19T11:31:51.924 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:51.924 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:51.924 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:51.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.924 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:51.924 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.568280+0000", 2024-03-19T11:31:51.924 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.925 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:51.925 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.568280+0000", 2024-03-19T11:31:51.925 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.925 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:51.925 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.568282+0000", 2024-03-19T11:31:51.925 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0090000000000002e-06 2024-03-19T11:31:51.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.925 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:51.925 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.568299+0000", 2024-03-19T11:31:51.925 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6237000000000001e-05 2024-03-19T11:31:51.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.926 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.926 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:51.926 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.568331+0000", 2024-03-19T11:31:51.926 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2601999999999997e-05 2024-03-19T11:31:51.926 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.926 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.926 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:51.926 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.568350+0000", 2024-03-19T11:31:51.926 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8280000000000001e-05 2024-03-19T11:31:51.926 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.926 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.926 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:51.926 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.571311+0000", 2024-03-19T11:31:51.926 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029617770000000001 2024-03-19T11:31:51.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.927 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:51.927 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.571379+0000", 2024-03-19T11:31:51.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7331999999999998e-05 2024-03-19T11:31:51.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.927 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:51.927 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.590583+0000", 2024-03-19T11:31:51.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019204649000000001 2024-03-19T11:31:51.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.927 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:51.927 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.590626+0000", 2024-03-19T11:31:51.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2419000000000002e-05 2024-03-19T11:31:51.928 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.928 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:51.928 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.928 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:3106 4.15 e104/76 4:ad3ad1b3:::smithi03829856-21:head v 104'224, mlcod=101'216)", 2024-03-19T11:31:51.928 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.571056+0000", 2024-03-19T11:31:51.928 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.302869725000001, 2024-03-19T11:31:51.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019636355000000001, 2024-03-19T11:31:51.928 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:51.928 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:51.928 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:51.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.928 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:51.928 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.571056+0000", 2024-03-19T11:31:51.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:51.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.571056+0000", 2024-03-19T11:31:51.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:51.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.571058+0000", 2024-03-19T11:31:51.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2400000000000002e-06 2024-03-19T11:31:51.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:51.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.571586+0000", 2024-03-19T11:31:51.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00052782299999999996 2024-03-19T11:31:51.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:51.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.571618+0000", 2024-03-19T11:31:51.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2453000000000003e-05 2024-03-19T11:31:51.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:51.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.571638+0000", 2024-03-19T11:31:51.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9176000000000001e-05 2024-03-19T11:31:51.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:51.931 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.573958+0000", 2024-03-19T11:31:51.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0023201699999999999 2024-03-19T11:31:51.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.931 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:51.931 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.574023+0000", 2024-03-19T11:31:51.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5425000000000005e-05 2024-03-19T11:31:51.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.931 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:51.931 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.590646+0000", 2024-03-19T11:31:51.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016623189 2024-03-19T11:31:51.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.931 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:51.932 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.590692+0000", 2024-03-19T11:31:51.932 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5878999999999999e-05 2024-03-19T11:31:51.932 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.932 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:51.932 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.932 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.932 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.932 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.3.0:309 4.0 e108/60 4:01018511:::smithi03829856-19:b v 108'91, mlcod=100'87)", 2024-03-19T11:31:51.932 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.205898+0000", 2024-03-19T11:31:51.932 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.668027930000001, 2024-03-19T11:31:51.932 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.034464328000000002, 2024-03-19T11:31:51.932 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:51.932 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:51.932 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:51.932 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.933 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:51.933 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.205898+0000", 2024-03-19T11:31:51.933 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.933 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.933 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:51.933 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.205898+0000", 2024-03-19T11:31:51.933 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.933 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.933 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:51.933 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.205899+0000", 2024-03-19T11:31:51.933 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.16e-07 2024-03-19T11:31:51.933 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.934 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:51.934 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.205916+0000", 2024-03-19T11:31:51.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7229000000000001e-05 2024-03-19T11:31:51.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.934 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:51.934 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.205946+0000", 2024-03-19T11:31:51.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0233999999999999e-05 2024-03-19T11:31:51.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.934 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:51.934 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.205966+0000", 2024-03-19T11:31:51.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9434999999999999e-05 2024-03-19T11:31:51.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:51.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.206162+0000", 2024-03-19T11:31:51.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000196918 2024-03-19T11:31:51.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:51.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.206227+0000", 2024-03-19T11:31:51.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4381000000000006e-05 2024-03-19T11:31:51.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:51.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.240212+0000", 2024-03-19T11:31:51.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033984852000000003 2024-03-19T11:31:51.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.936 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.936 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:51.936 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.240362+0000", 2024-03-19T11:31:51.936 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015036300000000001 2024-03-19T11:31:51.936 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.936 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:51.936 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.936 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.936 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.936 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.5.0:238 4.15 e108/76 4:ac9cd3a3:::smithi03829856-8:10 v 108'231, mlcod=104'228)", 2024-03-19T11:31:51.936 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.215160+0000", 2024-03-19T11:31:51.936 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.658765521999999, 2024-03-19T11:31:51.936 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.024600749000000002, 2024-03-19T11:31:51.936 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:51.936 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:51.937 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:51.937 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.937 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:51.937 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215160+0000", 2024-03-19T11:31:51.937 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.937 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.937 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.937 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:51.937 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215160+0000", 2024-03-19T11:31:51.937 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.937 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.937 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.937 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:51.937 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215161+0000", 2024-03-19T11:31:51.937 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6000000000000002e-07 2024-03-19T11:31:51.937 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.938 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.938 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:51.938 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215172+0000", 2024-03-19T11:31:51.938 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1042e-05 2024-03-19T11:31:51.938 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.938 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.938 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:51.938 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215213+0000", 2024-03-19T11:31:51.938 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1167000000000001e-05 2024-03-19T11:31:51.938 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.938 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.938 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:51.938 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215234+0000", 2024-03-19T11:31:51.938 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0534e-05 2024-03-19T11:31:51.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.939 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:51.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215668+0000", 2024-03-19T11:31:51.939 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000434074 2024-03-19T11:31:51.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.939 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:51.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215749+0000", 2024-03-19T11:31:51.939 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0702000000000001e-05 2024-03-19T11:31:51.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.939 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:51.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.239629+0000", 2024-03-19T11:31:51.939 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023880855999999999 2024-03-19T11:31:51.940 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.940 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.940 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:51.940 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.239761+0000", 2024-03-19T11:31:51.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000131514 2024-03-19T11:31:51.940 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.940 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:51.940 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.940 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.940 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.940 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.5.0:239 4.8 e108/100 4:14611006:::smithi03829856-25:c v 108'125, mlcod=103'122)", 2024-03-19T11:31:51.940 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.215292+0000", 2024-03-19T11:31:51.940 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.658633338000001, 2024-03-19T11:31:51.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.025163089999999999, 2024-03-19T11:31:51.940 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:51.940 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:51.941 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:51.941 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.941 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:51.941 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215292+0000", 2024-03-19T11:31:51.941 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.941 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.941 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.941 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:51.941 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215292+0000", 2024-03-19T11:31:51.941 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.941 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.941 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.941 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:51.941 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215293+0000", 2024-03-19T11:31:51.942 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.47e-07 2024-03-19T11:31:51.942 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.942 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:51.942 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215315+0000", 2024-03-19T11:31:51.942 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1691e-05 2024-03-19T11:31:51.942 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.942 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:51.942 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215348+0000", 2024-03-19T11:31:51.942 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2676999999999999e-05 2024-03-19T11:31:51.942 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.942 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:51.943 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215362+0000", 2024-03-19T11:31:51.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3978999999999999e-05 2024-03-19T11:31:51.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:51.943 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215727+0000", 2024-03-19T11:31:51.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000365583 2024-03-19T11:31:51.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:51.943 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215804+0000", 2024-03-19T11:31:51.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7392999999999996e-05 2024-03-19T11:31:51.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:51.944 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.240368+0000", 2024-03-19T11:31:51.944 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.024563807999999999 2024-03-19T11:31:51.944 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.944 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:51.944 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.240455+0000", 2024-03-19T11:31:51.944 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7112000000000002e-05 2024-03-19T11:31:51.944 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.944 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:51.944 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.944 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.944 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.5.0:240 4.15 e108/76 4:aa3303a5:::smithi03829856-17:b v 108'233, mlcod=104'228)", 2024-03-19T11:31:51.945 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.215400+0000", 2024-03-19T11:31:51.945 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.658525608000001, 2024-03-19T11:31:51.945 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026645319000000001, 2024-03-19T11:31:51.945 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:51.945 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:51.945 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:51.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.945 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:51.945 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215400+0000", 2024-03-19T11:31:51.945 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.945 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.945 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:51.945 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215400+0000", 2024-03-19T11:31:51.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.946 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:51.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215401+0000", 2024-03-19T11:31:51.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3e-07 2024-03-19T11:31:51.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.946 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:51.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215417+0000", 2024-03-19T11:31:51.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5821999999999999e-05 2024-03-19T11:31:51.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.946 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:51.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215443+0000", 2024-03-19T11:31:51.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6971e-05 2024-03-19T11:31:51.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:51.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215457+0000", 2024-03-19T11:31:51.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3139000000000001e-05 2024-03-19T11:31:51.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:51.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.234414+0000", 2024-03-19T11:31:51.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018957108 2024-03-19T11:31:51.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:51.948 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.234470+0000", 2024-03-19T11:31:51.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6274999999999999e-05 2024-03-19T11:31:51.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.948 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:51.948 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.241943+0000", 2024-03-19T11:31:51.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0074728770000000002 2024-03-19T11:31:51.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.948 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:51.948 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.242045+0000", 2024-03-19T11:31:51.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000102497 2024-03-19T11:31:51.948 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.948 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:51.948 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.949 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.949 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.5.0:241 4.8 e108/100 4:15708482:::smithi03829856-23:10 v 108'127, mlcod=103'122)", 2024-03-19T11:31:51.949 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.217180+0000", 2024-03-19T11:31:51.949 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.656745340000001, 2024-03-19T11:31:51.949 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.025026023000000001, 2024-03-19T11:31:51.949 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:51.949 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:51.949 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:51.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.949 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:51.949 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.217180+0000", 2024-03-19T11:31:51.949 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.949 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:51.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.217180+0000", 2024-03-19T11:31:51.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:51.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.217182+0000", 2024-03-19T11:31:51.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.12e-06 2024-03-19T11:31:51.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:51.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.217200+0000", 2024-03-19T11:31:51.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7677999999999999e-05 2024-03-19T11:31:51.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:51.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.217246+0000", 2024-03-19T11:31:51.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6053000000000001e-05 2024-03-19T11:31:51.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:51.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.217266+0000", 2024-03-19T11:31:51.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0085000000000002e-05 2024-03-19T11:31:51.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:51.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.226876+0000", 2024-03-19T11:31:51.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0096094389999999991 2024-03-19T11:31:51.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.952 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:51.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.226932+0000", 2024-03-19T11:31:51.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6234999999999999e-05 2024-03-19T11:31:51.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.952 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:51.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.242063+0000", 2024-03-19T11:31:51.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015131083 2024-03-19T11:31:51.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.952 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:51.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.242206+0000", 2024-03-19T11:31:51.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014333 2024-03-19T11:31:51.952 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.953 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:51.953 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.953 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.953 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.953 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.1.0:279 4.24 e108/83 4:27909373:::smithi03829856-46:c v 108'129, mlcod=100'124)", 2024-03-19T11:31:51.953 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.223035+0000", 2024-03-19T11:31:51.953 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.650890999000001, 2024-03-19T11:31:51.953 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019488583, 2024-03-19T11:31:51.953 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:51.953 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:51.953 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:51.953 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.953 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:51.953 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.223035+0000", 2024-03-19T11:31:51.953 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.954 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.954 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:51.954 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.223035+0000", 2024-03-19T11:31:51.954 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:51.954 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.954 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:51.954 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.223037+0000", 2024-03-19T11:31:51.954 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3829999999999999e-06 2024-03-19T11:31:51.954 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.954 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:51.954 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.223073+0000", 2024-03-19T11:31:51.954 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6306000000000001e-05 2024-03-19T11:31:51.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:51.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.223095+0000", 2024-03-19T11:31:51.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1738000000000001e-05 2024-03-19T11:31:51.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:51.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.223107+0000", 2024-03-19T11:31:51.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2082000000000001e-05 2024-03-19T11:31:51.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:51.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.226145+0000", 2024-03-19T11:31:51.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003038022 2024-03-19T11:31:51.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.956 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:51.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.226201+0000", 2024-03-19T11:31:51.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6107999999999999e-05 2024-03-19T11:31:51.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.956 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:51.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.242322+0000", 2024-03-19T11:31:51.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016120196 2024-03-19T11:31:51.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:51.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:51.956 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:51.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.242523+0000", 2024-03-19T11:31:51.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020174799999999999 2024-03-19T11:31:51.957 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.957 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:51.957 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.957 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:51.957 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:51.957 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:51.957 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-19T11:31:51.958 INFO:tasks.rados.rados.0.smithi038.stdout:1706: expect (ObjNum 213 snap 22 seq_num 213) 2024-03-19T11:31:51.959 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:51.969+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:51.961 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-19T11:31:52.060 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:52.067+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:52.074 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:52.083+0000 7f6710f8e640 -1 osd.2 130 handle_osd_map injecting map crc failure 2024-03-19T11:31:52.078 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-03-19T11:31:52.081 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:52.089+0000 7f923da1c640 -1 osd.7 130 handle_osd_map injecting map crc failure 2024-03-19T11:31:52.083 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:52.084 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:31:52.084 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:52.084 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop_reply(osd.0.0:399 4.16 e130/113 ondisk, result = 0)", 2024-03-19T11:31:52.084 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:52.095139+0000", 2024-03-19T11:31:52.084 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.00013467700000000001, 2024-03-19T11:31:52.084 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000155513, 2024-03-19T11:31:52.084 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:52.084 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:52.084 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:52.084 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:52.085 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:52.085 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:52.095139+0000", 2024-03-19T11:31:52.085 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:52.085 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:52.085 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:52.085 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:52.085 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:52.095139+0000", 2024-03-19T11:31:52.085 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:52.085 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:52.085 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:52.085 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:52.085 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:52.095145+0000", 2024-03-19T11:31:52.085 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1850000000000001e-06 2024-03-19T11:31:52.085 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:52.085 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:52.086 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:52.086 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:52.095153+0000", 2024-03-19T11:31:52.086 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1720000000000007e-06 2024-03-19T11:31:52.086 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:52.086 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:52.086 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:52.086 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:52.095170+0000", 2024-03-19T11:31:52.086 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6931000000000001e-05 2024-03-19T11:31:52.086 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:52.086 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:52.086 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:52.086 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:52.095180+0000", 2024-03-19T11:31:52.086 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0271999999999999e-05 2024-03-19T11:31:52.086 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:52.087 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:52.087 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:52.087 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:52.095234+0000", 2024-03-19T11:31:52.087 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3140999999999998e-05 2024-03-19T11:31:52.087 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:52.087 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:52.087 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:52.087 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:52.095243+0000", 2024-03-19T11:31:52.087 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7450000000000007e-06 2024-03-19T11:31:52.087 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:52.087 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:52.087 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:52.088 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:52.088 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-19T11:31:52.088 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 1 2024-03-19T11:31:52.088 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:52.101 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-19T11:31:52.119 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:31:52.119 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:31:52.136 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:31:52.136 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:31:52.153 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:31:52.153 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:31:52.160 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:52.171+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:52.169 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:31:52.170 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:31:52.190 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:31:52.191 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:31:52.211 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:31:52.211 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:31:52.230 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:52.230 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:52.232 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:31:52.232 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:31:52.243 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-19T11:31:52.254 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:31:52.255 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:31:52.262 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:52.269+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:52.263 INFO:tasks.rados.rados.0.smithi038.stdout:1704: finishing set_chunk to oid 35 2024-03-19T11:31:52.263 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 35 v 279 (ObjNum 294 snap 39 seq_num 294) dirty exists 2024-03-19T11:31:52.263 INFO:tasks.rados.rados.0.smithi038.stdout:1694: finishing write tid 1 to smithi03829856-43 2024-03-19T11:31:52.263 INFO:tasks.rados.rados.0.smithi038.stdout:1694: finishing write tid 2 to smithi03829856-43 2024-03-19T11:31:52.263 INFO:tasks.rados.rados.0.smithi038.stdout:1707: seq_num 315 ranges {494205=473908,1433122=407102,2384125=462684} 2024-03-19T11:31:52.268 INFO:tasks.rados.rados.0.smithi038.stdout:1707: writing smithi03829856-29 from 494205 to 968113 tid 1 2024-03-19T11:31:52.273 INFO:tasks.rados.rados.0.smithi038.stdout:1707: writing smithi03829856-29 from 1433122 to 1840224 tid 2 2024-03-19T11:31:52.280 INFO:tasks.rados.rados.0.smithi038.stdout:1707: writing smithi03829856-29 from 2384125 to 2846809 tid 3 2024-03-19T11:31:52.280 INFO:tasks.rados.rados.0.smithi038.stdout:1672: done (15 left) 2024-03-19T11:31:52.280 INFO:tasks.rados.rados.0.smithi038.stdout:1690: done (14 left) 2024-03-19T11:31:52.280 INFO:tasks.rados.rados.0.smithi038.stdout:1692: done (13 left) 2024-03-19T11:31:52.281 INFO:tasks.rados.rados.0.smithi038.stdout:1693: done (12 left) 2024-03-19T11:31:52.281 INFO:tasks.rados.rados.0.smithi038.stdout:1695: done (11 left) 2024-03-19T11:31:52.281 INFO:tasks.rados.rados.0.smithi038.stdout:1698: done (10 left) 2024-03-19T11:31:52.281 INFO:tasks.rados.rados.0.smithi038.stdout:1699: done (9 left) 2024-03-19T11:31:52.281 INFO:tasks.rados.rados.0.smithi038.stdout:1700: done (8 left) 2024-03-19T11:31:52.281 INFO:tasks.rados.rados.0.smithi038.stdout:1701: done (7 left) 2024-03-19T11:31:52.281 INFO:tasks.rados.rados.0.smithi038.stdout:1702: done (6 left) 2024-03-19T11:31:52.281 INFO:tasks.rados.rados.0.smithi038.stdout:1703: done (5 left) 2024-03-19T11:31:52.281 INFO:tasks.rados.rados.0.smithi038.stdout:1704: done (4 left) 2024-03-19T11:31:52.281 INFO:tasks.rados.rados.0.smithi038.stdout:1705: done (3 left) 2024-03-19T11:31:52.281 INFO:tasks.rados.rados.0.smithi038.stdout:1706: done (2 left) 2024-03-19T11:31:52.281 INFO:tasks.rados.rados.0.smithi038.stdout:1708: read oid 6 snap -1 2024-03-19T11:31:52.281 INFO:tasks.rados.rados.0.smithi038.stdout:1708: expect (ObjNum 195 snap 22 seq_num 195) 2024-03-19T11:31:52.281 INFO:tasks.rados.rados.0.smithi038.stdout:1709: rollback oid 49 current snap is 40 2024-03-19T11:31:52.281 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 49 to 28 2024-03-19T11:31:52.282 INFO:tasks.rados.rados.0.smithi038.stdout:1694: finishing write tid 3 to smithi03829856-43 2024-03-19T11:31:52.282 INFO:tasks.rados.rados.0.smithi038.stdout:1694: finishing write tid 4 to smithi03829856-43 2024-03-19T11:31:52.282 INFO:tasks.rados.rados.0.smithi038.stdout:1694: finishing write tid 5 to smithi03829856-43 2024-03-19T11:31:52.282 INFO:tasks.rados.rados.0.smithi038.stdout:1694: oid 43 updating version 0 to 182 2024-03-19T11:31:52.282 INFO:tasks.rados.rados.0.smithi038.stdout:1694: oid 43 version 182 is already newer than 181 2024-03-19T11:31:52.282 INFO:tasks.rados.rados.0.smithi038.stdout:1694: oid 43 version 182 is already newer than 180 2024-03-19T11:31:52.282 INFO:tasks.rados.rados.0.smithi038.stdout:1694: oid 43 updating version 182 to 183 2024-03-19T11:31:52.282 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 43 v 183 (ObjNum 312 snap 40 seq_num 312) dirty exists 2024-03-19T11:31:52.282 INFO:tasks.rados.rados.0.smithi038.stdout:1694: left oid 43 (ObjNum 312 snap 40 seq_num 312) 2024-03-19T11:31:52.282 INFO:tasks.rados.rados.0.smithi038.stdout:1694: done (3 left) 2024-03-19T11:31:52.282 INFO:tasks.rados.rados.0.smithi038.stdout:1710: tier_promote oid 5 2024-03-19T11:31:52.282 INFO:tasks.rados.rados.0.smithi038.stdout:1711: rollback oid 1 current snap is 40 2024-03-19T11:31:52.282 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 1 to 21 2024-03-19T11:31:52.282 INFO:tasks.rados.rados.0.smithi038.stdout:1712: tier_evict oid 8 2024-03-19T11:31:52.283 INFO:tasks.rados.rados.0.smithi038.stdout:1712: tier_evict oid 8 snap -1 2024-03-19T11:31:52.283 INFO:tasks.rados.rados.0.smithi038.stdout:1713: write (excl) oid 23 current snap is 40 2024-03-19T11:31:52.283 INFO:tasks.rados.rados.0.smithi038.stdout:1713: seq_num 316 ranges {511012=549157,1775251=550760,3095195=215817} 2024-03-19T11:31:52.285 INFO:tasks.rados.rados.0.smithi038.stdout:1713: writing smithi03829856-23 from 511012 to 1060169 tid 1 2024-03-19T11:31:52.292 INFO:tasks.rados.rados.0.smithi038.stdout:1713: writing smithi03829856-23 from 1775251 to 2326011 tid 2 2024-03-19T11:31:52.300 INFO:tasks.rados.rados.0.smithi038.stdout:1713: writing smithi03829856-23 from 3095195 to 3311012 tid 3 2024-03-19T11:31:52.300 INFO:tasks.rados.rados.0.smithi038.stdout:1714: read oid 46 snap -1 2024-03-19T11:31:52.300 INFO:tasks.rados.rados.0.smithi038.stdout:1714: expect (ObjNum 305 snap 39 seq_num 305) 2024-03-19T11:31:52.300 INFO:tasks.rados.rados.0.smithi038.stdout:1715: read oid 11 snap 33 2024-03-19T11:31:52.300 INFO:tasks.rados.rados.0.smithi038.stdout:1715: expect (ObjNum 229 snap 26 seq_num 229) 2024-03-19T11:31:52.301 INFO:tasks.rados.rados.0.smithi038.stdout:1716: read oid 37 snap -1 2024-03-19T11:31:52.301 INFO:tasks.rados.rados.0.smithi038.stdout:1716: expect (ObjNum 236 snap 28 seq_num 236) 2024-03-19T11:31:52.301 INFO:tasks.rados.rados.0.smithi038.stdout:1717: read oid 35 snap 27 2024-03-19T11:31:52.301 INFO:tasks.rados.rados.0.smithi038.stdout:1717: expect (ObjNum 217 snap 23 seq_num 217) 2024-03-19T11:31:52.301 INFO:tasks.rados.rados.0.smithi038.stdout:1718: write oid 32 current snap is 40 2024-03-19T11:31:52.301 INFO:tasks.rados.rados.0.smithi038.stdout:1712: got (22) Invalid argument 2024-03-19T11:31:52.301 INFO:tasks.rados.rados.0.smithi038.stdout:1710: got (0) Success 2024-03-19T11:31:52.301 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 5 v 215 (ObjNum 310 snap 39 seq_num 310) dirty exists 2024-03-19T11:31:52.301 INFO:tasks.rados.rados.0.smithi038.stdout:1707: finishing write tid 1 to smithi03829856-29 2024-03-19T11:31:52.301 INFO:tasks.rados.rados.0.smithi038.stdout:1707: finishing write tid 2 to smithi03829856-29 2024-03-19T11:31:52.301 INFO:tasks.rados.rados.0.smithi038.stdout:1718: seq_num 317 ranges {527819=624406,1833733=410771,2722618=601581} 2024-03-19T11:31:52.306 INFO:tasks.rados.rados.0.smithi038.stdout:1718: writing smithi03829856-32 from 527819 to 1152225 tid 1 2024-03-19T11:31:52.313 INFO:tasks.rados.rados.0.smithi038.stdout:1718: writing smithi03829856-32 from 1833733 to 2244504 tid 2 2024-03-19T11:31:52.319 INFO:tasks.rados.rados.0.smithi038.stdout:1718: writing smithi03829856-32 from 2722618 to 3324199 tid 3 2024-03-19T11:31:52.319 INFO:tasks.rados.rados.0.smithi038.stdout:1710: done (11 left) 2024-03-19T11:31:52.319 INFO:tasks.rados.rados.0.smithi038.stdout:1712: done (10 left) 2024-03-19T11:31:52.319 INFO:tasks.rados.rados.0.smithi038.stdout:1719: write oid 19 current snap is 40 2024-03-19T11:31:52.319 INFO:tasks.rados.rados.0.smithi038.stdout:1707: finishing write tid 3 to smithi03829856-29 2024-03-19T11:31:52.319 INFO:tasks.rados.rados.0.smithi038.stdout:1711: finishing rollback tid 0 to smithi03829856-1 2024-03-19T11:31:52.319 INFO:tasks.rados.rados.0.smithi038.stdout:1707: finishing write tid 4 to smithi03829856-29 2024-03-19T11:31:52.319 INFO:tasks.rados.rados.0.smithi038.stdout:1707: finishing write tid 5 to smithi03829856-29 2024-03-19T11:31:52.319 INFO:tasks.rados.rados.0.smithi038.stdout:1707: oid 29 updating version 0 to 115 2024-03-19T11:31:52.320 INFO:tasks.rados.rados.0.smithi038.stdout:1707: oid 29 version 115 is already newer than 114 2024-03-19T11:31:52.320 INFO:tasks.rados.rados.0.smithi038.stdout:1707: oid 29 updating version 115 to 116 2024-03-19T11:31:52.320 INFO:tasks.rados.rados.0.smithi038.stdout:1707: oid 29 updating version 116 to 117 2024-03-19T11:31:52.320 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 29 v 117 (ObjNum 315 snap 40 seq_num 315) dirty exists 2024-03-19T11:31:52.320 INFO:tasks.rados.rados.0.smithi038.stdout:1707: left oid 29 (ObjNum 315 snap 40 seq_num 315) 2024-03-19T11:31:52.320 INFO:tasks.rados.rados.0.smithi038.stdout:1719: seq_num 318 ranges {544626=699655,1892215=554429,3150041=194585} 2024-03-19T11:31:52.325 INFO:tasks.rados.rados.0.smithi038.stdout:1719: writing smithi03829856-19 from 544626 to 1244281 tid 1 2024-03-19T11:31:52.333 INFO:tasks.rados.rados.0.smithi038.stdout:1719: writing smithi03829856-19 from 1892215 to 2446644 tid 2 2024-03-19T11:31:52.338 INFO:tasks.rados.rados.0.smithi038.stdout:1719: writing smithi03829856-19 from 3150041 to 3344626 tid 3 2024-03-19T11:31:52.338 INFO:tasks.rados.rados.0.smithi038.stdout:1707: done (10 left) 2024-03-19T11:31:52.339 INFO:tasks.rados.rados.0.smithi038.stdout:1720: read oid 29 snap -1 2024-03-19T11:31:52.339 INFO:tasks.rados.rados.0.smithi038.stdout:1720: expect (ObjNum 315 snap 40 seq_num 315) 2024-03-19T11:31:52.339 INFO:tasks.rados.rados.0.smithi038.stdout:1721: set_chunk oid 7 target oid 2024-03-19T11:31:52.342 INFO:tasks.rados.rados.0.smithi038.stdout:1721: set_chunk oid 7 offset: 2453504 length: 221184 target oid 6877dc0fba79e392de2321aec9e81fbb16dcf971b0ce0be23908056325971e47 offset: 0 snap -1 2024-03-19T11:31:52.342 INFO:tasks.rados.rados.0.smithi038.stdout:1722: write (excl) oid 18 current snap is 40 2024-03-19T11:31:52.343 INFO:tasks.rados.rados.0.smithi038.stdout:1722: seq_num 319 ranges {561433=774904,1950697=414440,2777464=583969} 2024-03-19T11:31:52.350 INFO:tasks.rados.rados.0.smithi038.stdout:1722: writing smithi03829856-18 from 561433 to 1336337 tid 1 2024-03-19T11:31:52.355 INFO:tasks.rados.rados.0.smithi038.stdout:1722: writing smithi03829856-18 from 1950697 to 2365137 tid 2 2024-03-19T11:31:52.362 INFO:tasks.rados.rados.0.smithi038.stdout:1722: writing smithi03829856-18 from 2777464 to 3361433 tid 3 2024-03-19T11:31:52.362 INFO:tasks.rados.rados.0.smithi038.stdout:1723: read oid 47 snap 38 2024-03-19T11:31:52.362 INFO:tasks.rados.rados.0.smithi038.stdout:1723: expect deleted 2024-03-19T11:31:52.362 INFO:tasks.rados.rados.0.smithi038.stdout:1724: tier_evict oid 14 2024-03-19T11:31:52.362 INFO:tasks.rados.rados.0.smithi038.stdout:1724: tier_evict oid 14 snap -1 2024-03-19T11:31:52.362 INFO:tasks.rados.rados.0.smithi038.stdout:1708: expect (ObjNum 195 snap 22 seq_num 195) 2024-03-19T11:31:52.363 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:52.371+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:52.377 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:52.377 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:52.390 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-19T11:31:52.477 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:52.487+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:52.543 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:52.543 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:52.557 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-19T11:31:52.563 INFO:tasks.rados.rados.0.smithi038.stdout:1714: expect (ObjNum 305 snap 39 seq_num 305) 2024-03-19T11:31:52.565 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:52.573+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:52.663 INFO:tasks.rados.rados.0.smithi038.stdout:1711: finishing rollback tid 1 to smithi03829856-1 2024-03-19T11:31:52.663 INFO:tasks.rados.rados.0.smithi038.stdout:1711: finishing rollback tid 2 to smithi03829856-1 2024-03-19T11:31:52.663 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 1 v 185 (ObjNum 51 snap 0 seq_num 51) dirty exists 2024-03-19T11:31:52.663 INFO:tasks.rados.rados.0.smithi038.stdout:1713: finishing write tid 1 to smithi03829856-23 2024-03-19T11:31:52.663 INFO:tasks.rados.rados.0.smithi038.stdout:1716: expect (ObjNum 236 snap 28 seq_num 236) 2024-03-19T11:31:52.666 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:52.675+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:52.683 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:52.683 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:52.697 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-19T11:31:52.767 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:52.777+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:52.788 INFO:tasks.rados.rados.0.smithi038.stdout:1713: finishing write tid 2 to smithi03829856-23 2024-03-19T11:31:52.788 INFO:tasks.rados.rados.0.smithi038.stdout:1715: expect (ObjNum 229 snap 26 seq_num 229) 2024-03-19T11:31:52.860 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:52.860 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:52.868 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:52.875+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:52.873 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-19T11:31:52.969 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:52.977+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:53.023 INFO:tasks.rados.rados.0.smithi038.stdout:1718: finishing write tid 1 to smithi03829856-32 2024-03-19T11:31:53.024 INFO:tasks.rados.rados.0.smithi038.stdout:1713: finishing write tid 3 to smithi03829856-23 2024-03-19T11:31:53.024 INFO:tasks.rados.rados.0.smithi038.stdout:1713: finishing write tid 4 to smithi03829856-23 2024-03-19T11:31:53.024 INFO:tasks.rados.rados.0.smithi038.stdout:1713: finishing write tid 5 to smithi03829856-23 2024-03-19T11:31:53.024 INFO:tasks.rados.rados.0.smithi038.stdout:1713: oid 23 updating version 0 to 155 2024-03-19T11:31:53.024 INFO:tasks.rados.rados.0.smithi038.stdout:1713: oid 23 updating version 155 to 156 2024-03-19T11:31:53.024 INFO:tasks.rados.rados.0.smithi038.stdout:1713: oid 23 version 156 is already newer than 153 2024-03-19T11:31:53.024 INFO:tasks.rados.rados.0.smithi038.stdout:1713: oid 23 version 156 is already newer than 154 2024-03-19T11:31:53.024 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 23 v 156 (ObjNum 316 snap 40 seq_num 316) dirty exists 2024-03-19T11:31:53.024 INFO:tasks.rados.rados.0.smithi038.stdout:1713: left oid 23 (ObjNum 316 snap 40 seq_num 316) 2024-03-19T11:31:53.024 INFO:tasks.rados.rados.0.smithi038.stdout:1718: finishing write tid 2 to smithi03829856-32 2024-03-19T11:31:53.024 INFO:tasks.rados.rados.0.smithi038.stdout:1718: finishing write tid 3 to smithi03829856-32 2024-03-19T11:31:53.024 INFO:tasks.rados.rados.0.smithi038.stdout:1718: finishing write tid 4 to smithi03829856-32 2024-03-19T11:31:53.025 INFO:tasks.rados.rados.0.smithi038.stdout:1718: finishing write tid 5 to smithi03829856-32 2024-03-19T11:31:53.025 INFO:tasks.rados.rados.0.smithi038.stdout:1718: oid 32 updating version 0 to 150 2024-03-19T11:31:53.025 INFO:tasks.rados.rados.0.smithi038.stdout:1718: oid 32 updating version 150 to 152 2024-03-19T11:31:53.025 INFO:tasks.rados.rados.0.smithi038.stdout:1718: oid 32 version 152 is already newer than 151 2024-03-19T11:31:53.025 INFO:tasks.rados.rados.0.smithi038.stdout:1718: oid 32 updating version 152 to 153 2024-03-19T11:31:53.025 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 32 v 153 (ObjNum 317 snap 40 seq_num 317) dirty exists 2024-03-19T11:31:53.025 INFO:tasks.rados.rados.0.smithi038.stdout:1718: left oid 32 (ObjNum 317 snap 40 seq_num 317) 2024-03-19T11:31:53.025 INFO:tasks.rados.rados.0.smithi038.stdout:1717: expect (ObjNum 217 snap 23 seq_num 217) 2024-03-19T11:31:53.031 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:53.032 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:53.043 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-19T11:31:53.069 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:53.077+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:53.157 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:53.157 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:53.170 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-19T11:31:53.170 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:53.177+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:53.271 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:53.279+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:53.275 INFO:tasks.rados.rados.0.smithi038.stdout:1719: finishing write tid 1 to smithi03829856-19 2024-03-19T11:31:53.276 INFO:tasks.rados.rados.0.smithi038.stdout:1719: finishing write tid 2 to smithi03829856-19 2024-03-19T11:31:53.276 INFO:tasks.rados.rados.0.smithi038.stdout:1719: finishing write tid 3 to smithi03829856-19 2024-03-19T11:31:53.276 INFO:tasks.rados.rados.0.smithi038.stdout:1719: finishing write tid 4 to smithi03829856-19 2024-03-19T11:31:53.276 INFO:tasks.rados.rados.0.smithi038.stdout:1719: finishing write tid 5 to smithi03829856-19 2024-03-19T11:31:53.276 INFO:tasks.rados.rados.0.smithi038.stdout:1719: oid 19 updating version 0 to 109 2024-03-19T11:31:53.276 INFO:tasks.rados.rados.0.smithi038.stdout:1719: oid 19 version 109 is already newer than 108 2024-03-19T11:31:53.277 INFO:tasks.rados.rados.0.smithi038.stdout:1719: oid 19 updating version 109 to 110 2024-03-19T11:31:53.277 INFO:tasks.rados.rados.0.smithi038.stdout:1719: oid 19 updating version 110 to 111 2024-03-19T11:31:53.277 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 19 v 111 (ObjNum 318 snap 40 seq_num 318) dirty exists 2024-03-19T11:31:53.277 INFO:tasks.rados.rados.0.smithi038.stdout:1719: left oid 19 (ObjNum 318 snap 40 seq_num 318) 2024-03-19T11:31:53.277 INFO:tasks.rados.rados.0.smithi038.stdout:1721: finishing set_chunk to oid 7 2024-03-19T11:31:53.277 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 7 v 58 (ObjNum 304 snap 39 seq_num 304) dirty exists 2024-03-19T11:31:53.278 INFO:tasks.rados.rados.0.smithi038.stdout:1720: expect (ObjNum 315 snap 40 seq_num 315) 2024-03-19T11:31:53.287 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:53.287 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:53.301 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-19T11:31:53.372 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:53.381+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:53.467 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:53.467 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:53.473 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:53.479+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:53.480 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-19T11:31:53.516 INFO:tasks.rados.rados.0.smithi038.stdout:1724: got (22) Invalid argument 2024-03-19T11:31:53.516 INFO:tasks.rados.rados.0.smithi038.stdout:1722: finishing write tid 1 to smithi03829856-18 2024-03-19T11:31:53.516 INFO:tasks.rados.rados.0.smithi038.stdout:1722: finishing write tid 2 to smithi03829856-18 2024-03-19T11:31:53.516 INFO:tasks.rados.rados.0.smithi038.stdout:1722: finishing write tid 3 to smithi03829856-18 2024-03-19T11:31:53.516 INFO:tasks.rados.rados.0.smithi038.stdout:1722: finishing write tid 4 to smithi03829856-18 2024-03-19T11:31:53.516 INFO:tasks.rados.rados.0.smithi038.stdout:1722: finishing write tid 5 to smithi03829856-18 2024-03-19T11:31:53.516 INFO:tasks.rados.rados.0.smithi038.stdout:1722: oid 18 updating version 0 to 190 2024-03-19T11:31:53.516 INFO:tasks.rados.rados.0.smithi038.stdout:1722: oid 18 version 190 is already newer than 189 2024-03-19T11:31:53.516 INFO:tasks.rados.rados.0.smithi038.stdout:1722: oid 18 version 190 is already newer than 188 2024-03-19T11:31:53.516 INFO:tasks.rados.rados.0.smithi038.stdout:1722: oid 18 version 190 is already newer than 187 2024-03-19T11:31:53.516 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 18 v 190 (ObjNum 319 snap 40 seq_num 319) dirty exists 2024-03-19T11:31:53.516 INFO:tasks.rados.rados.0.smithi038.stdout:1722: left oid 18 (ObjNum 319 snap 40 seq_num 319) 2024-03-19T11:31:53.516 INFO:tasks.rados.rados.0.smithi038.stdout:1708: done (14 left) 2024-03-19T11:31:53.517 INFO:tasks.rados.rados.0.smithi038.stdout:1711: done (13 left) 2024-03-19T11:31:53.517 INFO:tasks.rados.rados.0.smithi038.stdout:1713: done (12 left) 2024-03-19T11:31:53.517 INFO:tasks.rados.rados.0.smithi038.stdout:1714: done (11 left) 2024-03-19T11:31:53.517 INFO:tasks.rados.rados.0.smithi038.stdout:1715: done (10 left) 2024-03-19T11:31:53.517 INFO:tasks.rados.rados.0.smithi038.stdout:1716: done (9 left) 2024-03-19T11:31:53.517 INFO:tasks.rados.rados.0.smithi038.stdout:1717: done (8 left) 2024-03-19T11:31:53.517 INFO:tasks.rados.rados.0.smithi038.stdout:1718: done (7 left) 2024-03-19T11:31:53.517 INFO:tasks.rados.rados.0.smithi038.stdout:1719: done (6 left) 2024-03-19T11:31:53.517 INFO:tasks.rados.rados.0.smithi038.stdout:1720: done (5 left) 2024-03-19T11:31:53.517 INFO:tasks.rados.rados.0.smithi038.stdout:1721: done (4 left) 2024-03-19T11:31:53.517 INFO:tasks.rados.rados.0.smithi038.stdout:1722: done (3 left) 2024-03-19T11:31:53.517 INFO:tasks.rados.rados.0.smithi038.stdout:1723: done (2 left) 2024-03-19T11:31:53.517 INFO:tasks.rados.rados.0.smithi038.stdout:1724: done (1 left) 2024-03-19T11:31:53.517 INFO:tasks.rados.rados.0.smithi038.stdout:1725: read oid 13 snap 33 2024-03-19T11:31:53.517 INFO:tasks.rados.rados.0.smithi038.stdout:1725: expect (ObjNum 12 snap 0 seq_num 12) 2024-03-19T11:31:53.518 INFO:tasks.rados.rados.0.smithi038.stdout:1726: read oid 48 snap -1 2024-03-19T11:31:53.518 INFO:tasks.rados.rados.0.smithi038.stdout:1726: expect (ObjNum 255 snap 32 seq_num 255) 2024-03-19T11:31:53.518 INFO:tasks.rados.rados.0.smithi038.stdout:1727: snap_remove snap 27 2024-03-19T11:31:53.544 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:53.551+0000 7f6a7d9d6640 -1 osd.1 131 handle_osd_map injecting map crc failure 2024-03-19T11:31:53.544 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:53.555+0000 7f6a7d9d6640 -1 osd.1 131 handle_osd_map injecting map crc failure 2024-03-19T11:31:53.573 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:53.581+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:53.631 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:53.631 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:53.645 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-19T11:31:53.689 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:53.699+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:53.776 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:53.783+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:53.811 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:53.811 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:53.824 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-19T11:31:53.876 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:31:53.885+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:31:53.957 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:53.958 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:53.972 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-19T11:31:53.978 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:53.987+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:54.062 INFO:tasks.rados.rados.0.smithi038.stdout:1727: done (3 left) 2024-03-19T11:31:54.063 INFO:tasks.rados.rados.0.smithi038.stdout:1728: write oid 10 current snap is 40 2024-03-19T11:31:54.063 INFO:tasks.rados.rados.0.smithi038.stdout:1728: seq_num 320 ranges {578240=566506,1609179=558098,2804887=551750} 2024-03-19T11:31:54.070 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-19T11:31:54.070 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-19T11:31:54.070 INFO:tasks.thrashosds.thrasher:Adding osd 3 2024-03-19T11:31:54.071 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 3 2024-03-19T11:31:54.085 INFO:tasks.rados.rados.0.smithi038.stdout:1728: writing smithi03829856-10 from 578240 to 1144746 tid 1 2024-03-19T11:31:54.086 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:54.095+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:54.094 INFO:tasks.rados.rados.0.smithi038.stdout:1728: writing smithi03829856-10 from 1609179 to 2167277 tid 2 2024-03-19T11:31:54.103 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-03-19T11:31:54.104 INFO:tasks.rados.rados.0.smithi038.stdout:1728: writing smithi03829856-10 from 2804887 to 3356637 tid 3 2024-03-19T11:31:54.104 INFO:tasks.rados.rados.0.smithi038.stdout:1729: rollback oid 18 current snap is 40 2024-03-19T11:31:54.104 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 18 to 35 2024-03-19T11:31:54.104 INFO:tasks.rados.rados.0.smithi038.stdout:1726: expect (ObjNum 255 snap 32 seq_num 255) 2024-03-19T11:31:54.157 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:54.157 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:54.172 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-19T11:31:54.179 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:54.187+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:54.209 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:54.219+0000 7fa5de8d7640 -1 osd.3 132 handle_osd_map injecting map crc failure 2024-03-19T11:31:54.260 INFO:tasks.rados.rados.0.smithi038.stdout:1725: expect (ObjNum 12 snap 0 seq_num 12) 2024-03-19T11:31:54.273 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-19T11:31:54.291 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:54.291 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:54.305 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-19T11:31:54.317 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:54.327+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:54.329 INFO:tasks.rados.rados.0.smithi038.stdout:1709: finishing rollback tid 0 to smithi03829856-49 2024-03-19T11:31:54.329 INFO:tasks.rados.rados.0.smithi038.stdout:1709: finishing rollback tid 1 to smithi03829856-49 2024-03-19T11:31:54.329 INFO:tasks.rados.rados.0.smithi038.stdout:1709: finishing rollback tid 2 to smithi03829856-49 2024-03-19T11:31:54.329 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 49 v 67 (ObjNum 234 snap 28 seq_num 234) dirty exists 2024-03-19T11:31:54.329 INFO:tasks.rados.rados.0.smithi038.stdout:1728: finishing write tid 1 to smithi03829856-10 2024-03-19T11:31:54.329 INFO:tasks.rados.rados.0.smithi038.stdout:1728: finishing write tid 2 to smithi03829856-10 2024-03-19T11:31:54.329 INFO:tasks.rados.rados.0.smithi038.stdout:1709: done (4 left) 2024-03-19T11:31:54.329 INFO:tasks.rados.rados.0.smithi038.stdout:1725: done (3 left) 2024-03-19T11:31:54.330 INFO:tasks.rados.rados.0.smithi038.stdout:1726: done (2 left) 2024-03-19T11:31:54.330 INFO:tasks.rados.rados.0.smithi038.stdout:1730: read oid 45 snap -1 2024-03-19T11:31:54.330 INFO:tasks.rados.rados.0.smithi038.stdout:1730: expect (ObjNum 314 snap 40 seq_num 314) 2024-03-19T11:31:54.330 INFO:tasks.rados.rados.0.smithi038.stdout:1731: rollback oid 35 current snap is 40 2024-03-19T11:31:54.330 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 35 to 32 2024-03-19T11:31:54.330 INFO:tasks.rados.rados.0.smithi038.stdout:1729: finishing rollback tid 0 to smithi03829856-18 2024-03-19T11:31:54.330 INFO:tasks.rados.rados.0.smithi038.stdout:1728: finishing write tid 3 to smithi03829856-10 2024-03-19T11:31:54.330 INFO:tasks.rados.rados.0.smithi038.stdout:1728: finishing write tid 4 to smithi03829856-10 2024-03-19T11:31:54.330 INFO:tasks.rados.rados.0.smithi038.stdout:1728: finishing write tid 5 to smithi03829856-10 2024-03-19T11:31:54.330 INFO:tasks.rados.rados.0.smithi038.stdout:1728: oid 10 updating version 0 to 191 2024-03-19T11:31:54.330 INFO:tasks.rados.rados.0.smithi038.stdout:1728: oid 10 version 191 is already newer than 189 2024-03-19T11:31:54.330 INFO:tasks.rados.rados.0.smithi038.stdout:1728: oid 10 version 191 is already newer than 190 2024-03-19T11:31:54.330 INFO:tasks.rados.rados.0.smithi038.stdout:1728: oid 10 updating version 191 to 192 2024-03-19T11:31:54.330 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 10 v 192 (ObjNum 320 snap 40 seq_num 320) dirty exists 2024-03-19T11:31:54.331 INFO:tasks.rados.rados.0.smithi038.stdout:1728: left oid 10 (ObjNum 320 snap 40 seq_num 320) 2024-03-19T11:31:54.331 INFO:tasks.rados.rados.0.smithi038.stdout:1729: finishing rollback tid 1 to smithi03829856-18 2024-03-19T11:31:54.331 INFO:tasks.rados.rados.0.smithi038.stdout:1729: finishing rollback tid 2 to smithi03829856-18 2024-03-19T11:31:54.331 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 18 v 197 (ObjNum 199 snap 22 seq_num 199) dirty exists 2024-03-19T11:31:54.331 INFO:tasks.rados.rados.0.smithi038.stdout:1728: done (3 left) 2024-03-19T11:31:54.331 INFO:tasks.rados.rados.0.smithi038.stdout:1729: done (2 left) 2024-03-19T11:31:54.331 INFO:tasks.rados.rados.0.smithi038.stdout:1732: read oid 19 snap -1 2024-03-19T11:31:54.331 INFO:tasks.rados.rados.0.smithi038.stdout:1732: expect (ObjNum 318 snap 40 seq_num 318) 2024-03-19T11:31:54.331 INFO:tasks.rados.rados.0.smithi038.stdout:1733: read oid 4 snap 34 2024-03-19T11:31:54.331 INFO:tasks.rados.rados.0.smithi038.stdout:1733: expect (ObjNum 270 snap 34 seq_num 270) 2024-03-19T11:31:54.331 INFO:tasks.rados.rados.0.smithi038.stdout:1734: read oid 6 snap -1 2024-03-19T11:31:54.331 INFO:tasks.rados.rados.0.smithi038.stdout:1734: expect (ObjNum 195 snap 22 seq_num 195) 2024-03-19T11:31:54.331 INFO:tasks.rados.rados.0.smithi038.stdout:1735: read oid 49 snap -1 2024-03-19T11:31:54.331 INFO:tasks.rados.rados.0.smithi038.stdout:1735: expect (ObjNum 234 snap 28 seq_num 234) 2024-03-19T11:31:54.331 INFO:tasks.rados.rados.0.smithi038.stdout:1736: read oid 7 snap 24 2024-03-19T11:31:54.332 INFO:tasks.rados.rados.0.smithi038.stdout:1736: expect (ObjNum 210 snap 22 seq_num 210) 2024-03-19T11:31:54.332 INFO:tasks.rados.rados.0.smithi038.stdout:1737: delete oid 11 current snap is 40 2024-03-19T11:31:54.338 INFO:tasks.rados.rados.0.smithi038.stdout:1735: expect (ObjNum 234 snap 28 seq_num 234) 2024-03-19T11:31:54.382 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:54.389+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:54.416 INFO:tasks.ceph.mon.a.smithi038.stderr:2024-03-19T11:31:54.423+0000 7f3f60a19640 -1 mon.a@0(leader).osd e133 definitely_dead 0 2024-03-19T11:31:54.417 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:31:54.417 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:31:54.434 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:31:54.435 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:54.435 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:54.435 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:31:54.449 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-19T11:31:54.450 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:31:54.451 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:31:54.466 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:31:54.466 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:31:54.484 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:31:54.484 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:31:54.492 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:54.501+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:54.502 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:31:54.502 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:31:54.520 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:31:54.520 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:31:54.538 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:31:54.538 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:31:54.597 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:54.607+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:54.615 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:54.616 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:31:54.616 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:31:54.616 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:31:54.616 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:54.631 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-19T11:31:54.663 INFO:tasks.rados.rados.0.smithi038.stdout:1733: expect (ObjNum 270 snap 34 seq_num 270) 2024-03-19T11:31:54.684 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:54.691+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:54.754 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:54.754 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:31:54.754 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:31:54.754 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:31:54.754 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.754 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1158 4.8 4:15708482:::smithi03829856-23:head [tier-promote] snapc 8=[8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-03-19T11:31:54.754 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.827517+0000", 2024-03-19T11:31:54.755 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 97.936068047000006, 2024-03-19T11:31:54.755 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.034096983999999997, 2024-03-19T11:31:54.755 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:54.755 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:54.755 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:54.755 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:54.755 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:54.755 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1158 2024-03-19T11:31:54.755 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.755 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:54.755 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.755 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:54.755 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827517+0000", 2024-03-19T11:31:54.755 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:54.756 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.756 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.756 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:54.756 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827517+0000", 2024-03-19T11:31:54.756 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:54.756 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.756 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.756 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:54.756 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827520+0000", 2024-03-19T11:31:54.756 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.45e-06 2024-03-19T11:31:54.756 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.756 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.756 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:54.756 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827535+0000", 2024-03-19T11:31:54.757 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.509e-05 2024-03-19T11:31:54.757 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.757 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.757 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:54.757 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827579+0000", 2024-03-19T11:31:54.757 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4211000000000001e-05 2024-03-19T11:31:54.757 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.757 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.757 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:54.757 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827603+0000", 2024-03-19T11:31:54.757 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3586000000000001e-05 2024-03-19T11:31:54.757 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.757 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.757 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:54.758 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827789+0000", 2024-03-19T11:31:54.758 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018580199999999999 2024-03-19T11:31:54.758 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.758 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.758 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:54.758 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.828077+0000", 2024-03-19T11:31:54.758 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028846500000000001 2024-03-19T11:31:54.758 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.758 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.758 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:54.758 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.861614+0000", 2024-03-19T11:31:54.758 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033536379999999998 2024-03-19T11:31:54.758 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:54.758 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:54.759 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:54.759 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.759 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1209 4.8 4:1d58db11:::smithi03829856-18:head [write 645402~551297,stat] snapc 9=[9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:31:54.759 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:18.121657+0000", 2024-03-19T11:31:54.759 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 96.641927275, 2024-03-19T11:31:54.759 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.025074349999999999, 2024-03-19T11:31:54.759 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:54.759 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:54.759 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:54.759 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:54.759 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:54.759 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1209 2024-03-19T11:31:54.759 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.760 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:54.760 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.760 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:54.760 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.121657+0000", 2024-03-19T11:31:54.760 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:54.760 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.760 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.760 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:54.760 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.121657+0000", 2024-03-19T11:31:54.760 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:54.760 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.760 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.760 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:54.760 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.121661+0000", 2024-03-19T11:31:54.760 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0489999999999997e-06 2024-03-19T11:31:54.761 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.761 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.761 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:54.761 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123747+0000", 2024-03-19T11:31:54.761 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0020857720000000001 2024-03-19T11:31:54.761 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.761 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.761 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:54.761 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123779+0000", 2024-03-19T11:31:54.761 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2258000000000002e-05 2024-03-19T11:31:54.761 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.761 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.761 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:54.761 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123795+0000", 2024-03-19T11:31:54.762 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5177000000000001e-05 2024-03-19T11:31:54.762 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.762 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.762 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:54.762 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123949+0000", 2024-03-19T11:31:54.762 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015403699999999999 2024-03-19T11:31:54.762 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.762 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.762 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:54.762 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.124134+0000", 2024-03-19T11:31:54.762 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000185484 2024-03-19T11:31:54.762 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.763 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:54.763 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.124554+0000", 2024-03-19T11:31:54.763 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00042019699999999998 2024-03-19T11:31:54.763 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.763 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:54.763 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146267+0000", 2024-03-19T11:31:54.764 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021712838000000002 2024-03-19T11:31:54.764 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.764 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:54.764 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146424+0000", 2024-03-19T11:31:54.764 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015647299999999999 2024-03-19T11:31:54.764 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.764 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:54.764 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146597+0000", 2024-03-19T11:31:54.764 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017303699999999999 2024-03-19T11:31:54.764 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.764 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:54.765 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146732+0000", 2024-03-19T11:31:54.765 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013502800000000001 2024-03-19T11:31:54.765 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:54.765 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:54.765 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:54.765 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.765 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.765 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1443 4.2 4:43b32242:::smithi03829856-22:head [read 0~2352947,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:31:54.765 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.979417+0000", 2024-03-19T11:31:54.765 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 88.784167601999997, 2024-03-19T11:31:54.765 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.96719988700000004, 2024-03-19T11:31:54.766 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:54.766 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:54.766 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:54.766 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:54.766 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:54.766 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1443 2024-03-19T11:31:54.766 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.766 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:54.766 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.766 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:54.766 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979417+0000", 2024-03-19T11:31:54.767 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:54.767 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.767 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:54.767 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979417+0000", 2024-03-19T11:31:54.767 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:54.767 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.767 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:54.767 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979420+0000", 2024-03-19T11:31:54.767 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.366e-06 2024-03-19T11:31:54.767 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.767 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.767 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:54.768 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979437+0000", 2024-03-19T11:31:54.768 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6192e-05 2024-03-19T11:31:54.768 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.768 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.768 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:54.768 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979474+0000", 2024-03-19T11:31:54.768 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7601000000000001e-05 2024-03-19T11:31:54.768 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.768 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.768 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:54.769 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979496+0000", 2024-03-19T11:31:54.769 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1506999999999999e-05 2024-03-19T11:31:54.769 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.769 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.769 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:54.769 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979712+0000", 2024-03-19T11:31:54.769 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021641299999999999 2024-03-19T11:31:54.769 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.769 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.769 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:31:54.769 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979746+0000", 2024-03-19T11:31:54.769 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3386000000000002e-05 2024-03-19T11:31:54.770 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.770 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.770 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:54.770 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.931552+0000", 2024-03-19T11:31:54.770 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.95180659899999998 2024-03-19T11:31:54.770 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.770 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.770 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:54.770 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.934681+0000", 2024-03-19T11:31:54.770 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0031292289999999999 2024-03-19T11:31:54.770 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.770 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.771 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:54.771 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.942701+0000", 2024-03-19T11:31:54.771 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0080198809999999995 2024-03-19T11:31:54.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.771 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:54.771 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.942839+0000", 2024-03-19T11:31:54.771 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013816999999999999 2024-03-19T11:31:54.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.771 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:54.771 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.946617+0000", 2024-03-19T11:31:54.771 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003777543 2024-03-19T11:31:54.771 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:54.772 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:54.772 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:54.772 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.772 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.772 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1444 4.2 4:43b32242:::smithi03829856-22:head [read 0~2352947,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:31:54.772 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.979553+0000", 2024-03-19T11:31:54.772 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 88.784031898999999, 2024-03-19T11:31:54.772 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.97015945400000003, 2024-03-19T11:31:54.772 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:54.772 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:54.772 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:54.772 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:54.772 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:54.772 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1444 2024-03-19T11:31:54.773 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.773 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:54.773 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.773 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:54.773 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979553+0000", 2024-03-19T11:31:54.773 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:54.773 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.773 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.773 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:54.773 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979553+0000", 2024-03-19T11:31:54.773 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:54.773 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.773 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.773 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:54.773 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979554+0000", 2024-03-19T11:31:54.774 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.403e-06 2024-03-19T11:31:54.774 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.774 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.774 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:54.774 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979564+0000", 2024-03-19T11:31:54.774 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0032e-05 2024-03-19T11:31:54.774 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.774 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.774 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:54.774 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979579+0000", 2024-03-19T11:31:54.774 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4875000000000001e-05 2024-03-19T11:31:54.774 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.774 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:54.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979595+0000", 2024-03-19T11:31:54.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6300999999999999e-05 2024-03-19T11:31:54.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:54.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979874+0000", 2024-03-19T11:31:54.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000279065 2024-03-19T11:31:54.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:31:54.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979904+0000", 2024-03-19T11:31:54.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.94e-05 2024-03-19T11:31:54.776 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.776 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.776 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:54.776 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.934781+0000", 2024-03-19T11:31:54.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.95487744900000004 2024-03-19T11:31:54.776 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.776 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.776 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:54.776 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.934946+0000", 2024-03-19T11:31:54.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016485999999999999 2024-03-19T11:31:54.776 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.776 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.776 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:54.777 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.946829+0000", 2024-03-19T11:31:54.777 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011882482999999999 2024-03-19T11:31:54.777 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.777 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.777 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:54.777 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.946989+0000", 2024-03-19T11:31:54.777 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000160421 2024-03-19T11:31:54.777 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.777 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.777 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:54.777 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.949712+0000", 2024-03-19T11:31:54.778 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002723165 2024-03-19T11:31:54.778 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:54.778 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:54.778 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:54.778 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.778 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.778 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1445 4.2 4:43b32242:::smithi03829856-22:head [read 0~2352947,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:31:54.778 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.979653+0000", 2024-03-19T11:31:54.778 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 88.783931768000002, 2024-03-19T11:31:54.778 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.97249507999999996, 2024-03-19T11:31:54.778 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:54.778 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:54.779 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:54.779 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:54.779 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:54.779 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1445 2024-03-19T11:31:54.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.779 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:54.779 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:54.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979653+0000", 2024-03-19T11:31:54.779 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:54.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.779 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:54.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979653+0000", 2024-03-19T11:31:54.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:54.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.780 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:54.780 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979654+0000", 2024-03-19T11:31:54.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.372e-06 2024-03-19T11:31:54.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.780 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:54.780 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979662+0000", 2024-03-19T11:31:54.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9959999999999993e-06 2024-03-19T11:31:54.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.780 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:54.781 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979698+0000", 2024-03-19T11:31:54.781 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5763000000000003e-05 2024-03-19T11:31:54.781 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.781 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.781 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:54.781 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979714+0000", 2024-03-19T11:31:54.781 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6283999999999999e-05 2024-03-19T11:31:54.781 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.781 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.781 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:54.781 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979990+0000", 2024-03-19T11:31:54.781 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000275829 2024-03-19T11:31:54.782 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.782 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.782 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:31:54.782 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.980018+0000", 2024-03-19T11:31:54.782 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8082999999999999e-05 2024-03-19T11:31:54.782 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.782 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.782 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:54.782 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.935620+0000", 2024-03-19T11:31:54.782 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.95560133800000002 2024-03-19T11:31:54.782 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.782 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.783 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:54.783 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.935795+0000", 2024-03-19T11:31:54.783 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000175656 2024-03-19T11:31:54.783 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.783 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:54.783 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.949835+0000", 2024-03-19T11:31:54.783 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014040083 2024-03-19T11:31:54.783 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.783 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:54.783 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.950002+0000", 2024-03-19T11:31:54.783 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016673 2024-03-19T11:31:54.783 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.783 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.784 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:54.784 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.952148+0000", 2024-03-19T11:31:54.784 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021459460000000001 2024-03-19T11:31:54.784 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:54.784 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:54.784 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:54.784 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.784 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.784 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2207 4.28 4:15708482:::smithi03829856-23:head [write 541225~644694,stat] snapc 13=[13,12,11,d,b,6] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-03-19T11:31:54.785 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:51.160660+0000", 2024-03-19T11:31:54.785 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 63.602924942000001, 2024-03-19T11:31:54.785 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.025400267000000001, 2024-03-19T11:31:54.785 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:54.785 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:54.785 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:54.785 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:54.785 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:54.785 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2207 2024-03-19T11:31:54.785 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.785 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:54.785 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.785 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:54.785 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.160660+0000", 2024-03-19T11:31:54.786 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:54.786 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.786 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.786 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:54.786 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.160660+0000", 2024-03-19T11:31:54.786 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:54.786 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.786 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.786 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:54.786 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.160688+0000", 2024-03-19T11:31:54.786 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8058000000000001e-05 2024-03-19T11:31:54.787 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.787 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.787 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:54.787 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.168614+0000", 2024-03-19T11:31:54.787 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079262810000000003 2024-03-19T11:31:54.787 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.787 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.787 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:54.787 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.168662+0000", 2024-03-19T11:31:54.787 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8327e-05 2024-03-19T11:31:54.787 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.787 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.787 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:54.787 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.168679+0000", 2024-03-19T11:31:54.788 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6305999999999999e-05 2024-03-19T11:31:54.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.788 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:54.788 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.168830+0000", 2024-03-19T11:31:54.788 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000151714 2024-03-19T11:31:54.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.788 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.788 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:54.788 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.169107+0000", 2024-03-19T11:31:54.788 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027608299999999999 2024-03-19T11:31:54.788 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.789 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.789 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:54.789 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.169605+0000", 2024-03-19T11:31:54.789 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00049874999999999995 2024-03-19T11:31:54.789 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.789 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.789 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:54.789 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.185769+0000", 2024-03-19T11:31:54.789 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016163397999999999 2024-03-19T11:31:54.789 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.789 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.789 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:54.789 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.185809+0000", 2024-03-19T11:31:54.789 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0598e-05 2024-03-19T11:31:54.790 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.790 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.790 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:54.790 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.185898+0000", 2024-03-19T11:31:54.790 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8270999999999996e-05 2024-03-19T11:31:54.790 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.790 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.790 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:54.790 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.186060+0000", 2024-03-19T11:31:54.790 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016248099999999999 2024-03-19T11:31:54.790 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:54.790 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:54.790 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:54.790 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.791 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.791 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.15 75/74 [PushOp(4:af9d0d56:::smithi03829856-20:head, version: 35'55, data_included: [655360~589824,1900544~418789], data_size: 1008613, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:af9d0d56:::smithi03829856-20:head@35'55, size: 2319333, copy_subset: [(0, 2319333)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2319333, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:aa3303a5:::smithi03829856-17:b, version: 58'79, data_included: [614400~692224,1903006~356352], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:aa3303a5:::smithi03829856-17:b@58'79, size: 2268912, copy_subset: [(0, 2268912)], clone_subset: {}, snapset: 14=[]:{b: [b, 6], 10: [d], 12: [12, 11], 14: [13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2259358, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ad3ad1b3:::smithi03829856-21:b, version: 58'81, data_included: [736140~537686,2020581~315559], data_size: 853245, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:ad3ad1b3:::smithi03829856-21:b@58'81, size: 2336140, copy_subset: [(0, 2336140)], clone_subset: {}, snapset: 10=[]:{b: [b, 6], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2336140, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:afa9f208:::smithi03829856-35:12, version: 67'105, data_included: [521050~698057,1806623~350519], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:afa9f208:::smithi03829856-35:12@67'105, size: 2521050, copy_subset: [(0, 2521050)], clone_subset: {}, snapset: 12=[]:{11: [11, d, b, 6], 12: [12]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2157142, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:aa3303a5:::smithi03829856-17:12, version: 69'113, data_included: [413472~893152,1808868~155424], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:aa3303a5:::smithi03829856-17:12@69'113, size: 2319366, copy_subset: [(0, 2319366)], clone_subset: {}, snapset: 14=[]:{b: [b, 6], 10: [d], 12: [12, 11], 14: [13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1964292, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ac9cd3a3:::smithi03829856-8:10, version: 70'117, data_included: [517649~620182,1776668~428394], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:ac9cd3a3:::smithi03829856-8:10@70'117, size: 3008420, copy_subset: [(0, 3008420)], clone_subset: {}, snapset: 10=[]:{10: [d, b, 6]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2205062, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ac9cd3a3:::smithi03829856-8:head, version: 70'118, data_included: [517649~620182,1776668~428394], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:ac9cd3a3:::smithi03829856-8:head@70'118, size: 3008420, copy_subset: [(0, 3008420)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2205062, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:aa3303a5:::smithi03829856-17:10, version: 70'119, data_included: [413472~893152,1818036~155424], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:aa3303a5:::smithi03829856-17:10@70'119, size: 3613472, copy_subset: [(0, 3613472)], clone_subset: {}, snapset: 14=[]:{b: [b, 6], 10: [d], 12: [12, 11], 14: [13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1973460, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ad3ad1b3:::smithi03829856-21:10, version: 72'122, data_included: [531121~742705,1711057~305871], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:ad3ad1b3:::smithi03829856-21:10@72'122, size: 3731121, copy_subset: [(0, 3731121)], clone_subset: {}, snapset: 10=[]:{b: [b, 6], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2016928, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:54.791 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.081994+0000", 2024-03-19T11:31:54.791 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 60.681590245000002, 2024-03-19T11:31:54.791 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.040292051000000002, 2024-03-19T11:31:54.791 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:54.791 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:54.791 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:54.791 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.791 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:54.791 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.081994+0000", 2024-03-19T11:31:54.791 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:54.791 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.792 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.792 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:54.792 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.081994+0000", 2024-03-19T11:31:54.792 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:54.792 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.792 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.792 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:54.792 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.081996+0000", 2024-03-19T11:31:54.792 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5069999999999999e-06 2024-03-19T11:31:54.792 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.792 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.792 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:54.792 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088193+0000", 2024-03-19T11:31:54.792 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061970209999999996 2024-03-19T11:31:54.792 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.793 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:54.793 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088252+0000", 2024-03-19T11:31:54.793 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8838999999999997e-05 2024-03-19T11:31:54.793 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.793 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:54.793 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088322+0000", 2024-03-19T11:31:54.793 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0455000000000002e-05 2024-03-19T11:31:54.793 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.793 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.793 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:54.793 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088660+0000", 2024-03-19T11:31:54.793 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033735299999999998 2024-03-19T11:31:54.793 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.794 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.794 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:54.794 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088727+0000", 2024-03-19T11:31:54.794 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7335999999999994e-05 2024-03-19T11:31:54.794 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.794 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.794 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:54.794 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.122286+0000", 2024-03-19T11:31:54.794 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033559539999999999 2024-03-19T11:31:54.794 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:54.794 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:54.794 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:54.794 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.794 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.795 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.15 e75,74 4:ac9cd3a3:::smithi03829856-8:4,49'71)", 2024-03-19T11:31:54.795 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.088380+0000", 2024-03-19T11:31:54.795 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 60.675204383000001, 2024-03-19T11:31:54.795 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.044894303000000003, 2024-03-19T11:31:54.795 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:54.795 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:54.795 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:54.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.795 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:54.795 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088380+0000", 2024-03-19T11:31:54.795 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:54.795 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.795 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.795 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:54.795 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088380+0000", 2024-03-19T11:31:54.796 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:54.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.796 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.796 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:54.796 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088381+0000", 2024-03-19T11:31:54.796 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0500000000000004e-07 2024-03-19T11:31:54.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.796 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.796 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:54.796 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088385+0000", 2024-03-19T11:31:54.796 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2370000000000003e-06 2024-03-19T11:31:54.796 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.796 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.796 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:54.797 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088392+0000", 2024-03-19T11:31:54.797 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4240000000000004e-06 2024-03-19T11:31:54.797 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.797 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.797 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:54.797 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088397+0000", 2024-03-19T11:31:54.797 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8829999999999997e-06 2024-03-19T11:31:54.797 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.797 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.797 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:54.797 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.122356+0000", 2024-03-19T11:31:54.797 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033958588999999997 2024-03-19T11:31:54.797 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.797 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.797 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:54.798 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.122364+0000", 2024-03-19T11:31:54.798 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0800000000000006e-06 2024-03-19T11:31:54.798 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.798 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.798 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:54.798 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.133275+0000", 2024-03-19T11:31:54.798 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010910585 2024-03-19T11:31:54.798 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:54.798 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:54.798 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:54.798 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.798 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.798 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:2286 4.15 e75/74 4:afa9f208:::smithi03829856-35:head v 75'131, mlcod=33'54)", 2024-03-19T11:31:54.798 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.089414+0000", 2024-03-19T11:31:54.798 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 60.674171143000002, 2024-03-19T11:31:54.799 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.045421256, 2024-03-19T11:31:54.799 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:54.799 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:54.799 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:54.799 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.799 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:54.799 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.089414+0000", 2024-03-19T11:31:54.799 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:54.799 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.799 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.799 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:54.799 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.089414+0000", 2024-03-19T11:31:54.800 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:54.800 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.800 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:54.800 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.089414+0000", 2024-03-19T11:31:54.800 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1100000000000005e-07 2024-03-19T11:31:54.800 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.800 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.800 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:54.800 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.089423+0000", 2024-03-19T11:31:54.800 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3799999999999994e-06 2024-03-19T11:31:54.801 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.801 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.801 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:54.801 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.089436+0000", 2024-03-19T11:31:54.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2931e-05 2024-03-19T11:31:54.801 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.801 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.801 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:54.801 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.089447+0000", 2024-03-19T11:31:54.801 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1044e-05 2024-03-19T11:31:54.801 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.801 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.801 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:54.801 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.133330+0000", 2024-03-19T11:31:54.802 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.043882833000000003 2024-03-19T11:31:54.802 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.802 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.802 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:54.802 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.133353+0000", 2024-03-19T11:31:54.802 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3629999999999999e-05 2024-03-19T11:31:54.802 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.802 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.802 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:54.803 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.134786+0000", 2024-03-19T11:31:54.803 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0014327980000000001 2024-03-19T11:31:54.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.803 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:54.803 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.134835+0000", 2024-03-19T11:31:54.803 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8829000000000003e-05 2024-03-19T11:31:54.803 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:54.803 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:54.803 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:54.803 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.803 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.803 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2802 4.28 4:15708482:::smithi03829856-23:head [write 497540~612664] snapc 19=[18,17,15,b] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-03-19T11:31:54.803 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:14.060002+0000", 2024-03-19T11:31:54.803 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 40.703583014000003, 2024-03-19T11:31:54.804 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.028374612, 2024-03-19T11:31:54.804 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:54.804 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:54.804 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:54.804 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:54.804 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:54.804 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2802 2024-03-19T11:31:54.804 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.804 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:54.804 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.805 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:54.805 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.060002+0000", 2024-03-19T11:31:54.805 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:54.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.805 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.805 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:54.805 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.060002+0000", 2024-03-19T11:31:54.805 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:54.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.805 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.805 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:54.805 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.060005+0000", 2024-03-19T11:31:54.805 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5130000000000001e-06 2024-03-19T11:31:54.805 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.806 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.806 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:54.806 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.061352+0000", 2024-03-19T11:31:54.806 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0013468060000000001 2024-03-19T11:31:54.806 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.806 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.806 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:54.806 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.061403+0000", 2024-03-19T11:31:54.806 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1181999999999999e-05 2024-03-19T11:31:54.806 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.806 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.807 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:54.807 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.061428+0000", 2024-03-19T11:31:54.807 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4715999999999998e-05 2024-03-19T11:31:54.807 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.807 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:54.807 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.061600+0000", 2024-03-19T11:31:54.807 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017224100000000001 2024-03-19T11:31:54.807 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.807 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.807 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op must wait for map", 2024-03-19T11:31:54.807 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.061619+0000", 2024-03-19T11:31:54.807 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9213000000000001e-05 2024-03-19T11:31:54.807 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.808 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.808 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:54.808 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.066953+0000", 2024-03-19T11:31:54.808 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053332149999999997 2024-03-19T11:31:54.808 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.808 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.808 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:54.808 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.067236+0000", 2024-03-19T11:31:54.808 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028381400000000002 2024-03-19T11:31:54.808 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.808 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.808 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:54.808 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.067698+0000", 2024-03-19T11:31:54.808 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00046152200000000002 2024-03-19T11:31:54.809 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.809 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.809 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:54.809 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.082795+0000", 2024-03-19T11:31:54.809 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015096729 2024-03-19T11:31:54.809 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.809 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.809 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:54.809 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.088033+0000", 2024-03-19T11:31:54.809 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052383259999999997 2024-03-19T11:31:54.809 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.809 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.809 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:54.809 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.088209+0000", 2024-03-19T11:31:54.809 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000175611 2024-03-19T11:31:54.810 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.810 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.810 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:54.810 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.088376+0000", 2024-03-19T11:31:54.810 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016772400000000001 2024-03-19T11:31:54.810 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:54.810 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:54.810 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:54.810 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.810 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.810 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3103 4.15 4:afa9f208:::smithi03829856-35:head [write 1636889~447212] snapc 1f=[1f,1e,1d,1c,1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-19T11:31:54.810 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.557970+0000", 2024-03-19T11:31:54.810 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 31.205614325999999, 2024-03-19T11:31:54.810 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033363528000000003, 2024-03-19T11:31:54.810 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:54.811 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:54.811 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:54.811 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:54.811 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:54.811 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3103 2024-03-19T11:31:54.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.811 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:54.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.811 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:54.811 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.557970+0000", 2024-03-19T11:31:54.811 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:54.811 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.811 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.811 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:54.811 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.557970+0000", 2024-03-19T11:31:54.812 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:54.812 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.812 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.812 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:54.813 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.557973+0000", 2024-03-19T11:31:54.813 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6060000000000001e-06 2024-03-19T11:31:54.813 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.813 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.813 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:54.814 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558806+0000", 2024-03-19T11:31:54.814 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00083305499999999997 2024-03-19T11:31:54.814 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.814 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.814 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:54.814 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558843+0000", 2024-03-19T11:31:54.814 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6519000000000001e-05 2024-03-19T11:31:54.814 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.814 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.814 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:54.814 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558863+0000", 2024-03-19T11:31:54.814 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0622999999999999e-05 2024-03-19T11:31:54.814 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.814 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.815 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:54.815 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.562831+0000", 2024-03-19T11:31:54.815 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039681250000000003 2024-03-19T11:31:54.815 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.815 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.815 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:54.815 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.563003+0000", 2024-03-19T11:31:54.815 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000171638 2024-03-19T11:31:54.815 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.815 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.815 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:54.815 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.563335+0000", 2024-03-19T11:31:54.815 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033165899999999999 2024-03-19T11:31:54.815 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.815 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:54.816 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.569815+0000", 2024-03-19T11:31:54.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00648024 2024-03-19T11:31:54.816 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:54.816 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.590982+0000", 2024-03-19T11:31:54.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021167235 2024-03-19T11:31:54.816 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.816 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:54.816 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.591170+0000", 2024-03-19T11:31:54.816 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018796899999999999 2024-03-19T11:31:54.816 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.816 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.817 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:54.817 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.591334+0000", 2024-03-19T11:31:54.817 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000163859 2024-03-19T11:31:54.817 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:54.817 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:54.817 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:54.817 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.817 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.817 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3104 4.15 4:afa9f208:::smithi03829856-35:head [setxattr _header (57) in=7b,truncate 2084101] snapc 1f=[1f,1e,1d,1c,1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-19T11:31:54.817 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.558905+0000", 2024-03-19T11:31:54.817 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 31.204679244000001, 2024-03-19T11:31:54.818 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032860320999999998, 2024-03-19T11:31:54.818 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:54.818 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:54.818 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:54.818 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:54.818 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:54.818 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3104 2024-03-19T11:31:54.818 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.818 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:54.818 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.818 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:54.818 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558905+0000", 2024-03-19T11:31:54.818 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:54.818 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.819 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:54.819 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558905+0000", 2024-03-19T11:31:54.819 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:54.819 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.819 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:54.819 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558907+0000", 2024-03-19T11:31:54.819 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.547e-06 2024-03-19T11:31:54.819 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.819 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.819 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:54.820 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558916+0000", 2024-03-19T11:31:54.820 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0589999999999992e-06 2024-03-19T11:31:54.820 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.820 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.820 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:54.820 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558931+0000", 2024-03-19T11:31:54.820 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5262999999999999e-05 2024-03-19T11:31:54.820 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.820 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.820 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:54.820 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558947+0000", 2024-03-19T11:31:54.820 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6140000000000001e-05 2024-03-19T11:31:54.820 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.821 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.821 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:54.821 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.567458+0000", 2024-03-19T11:31:54.821 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085106650000000006 2024-03-19T11:31:54.821 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.821 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.821 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:54.821 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.567777+0000", 2024-03-19T11:31:54.821 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00031917399999999998 2024-03-19T11:31:54.821 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.821 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.821 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:54.822 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.568090+0000", 2024-03-19T11:31:54.822 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00031286900000000002 2024-03-19T11:31:54.822 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.822 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.822 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:54.822 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.576738+0000", 2024-03-19T11:31:54.822 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0086476379999999992 2024-03-19T11:31:54.822 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.822 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.822 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:54.822 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.591474+0000", 2024-03-19T11:31:54.822 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014736281 2024-03-19T11:31:54.822 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.822 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.823 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:54.823 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.591576+0000", 2024-03-19T11:31:54.823 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000102303 2024-03-19T11:31:54.823 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.823 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.823 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:54.823 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.591766+0000", 2024-03-19T11:31:54.823 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018938200000000001 2024-03-19T11:31:54.823 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:54.823 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:54.823 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:54.823 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.824 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.824 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3105 4.15 4:afa9f208:::smithi03829856-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e104)", 2024-03-19T11:31:54.824 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.558970+0000", 2024-03-19T11:31:54.824 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 31.204614808999999, 2024-03-19T11:31:54.824 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033553684, 2024-03-19T11:31:54.824 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:54.824 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:54.824 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:54.824 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:54.824 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:54.824 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3105 2024-03-19T11:31:54.824 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.825 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:54.825 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.825 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:54.825 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558970+0000", 2024-03-19T11:31:54.825 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:54.825 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.825 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.825 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:54.825 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558970+0000", 2024-03-19T11:31:54.825 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:54.825 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.825 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.826 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:54.826 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558971+0000", 2024-03-19T11:31:54.826 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4729999999999999e-06 2024-03-19T11:31:54.826 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.826 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.826 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:54.826 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558978+0000", 2024-03-19T11:31:54.826 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1369999999999996e-06 2024-03-19T11:31:54.826 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.826 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.826 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:54.826 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.559003+0000", 2024-03-19T11:31:54.826 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4525999999999999e-05 2024-03-19T11:31:54.826 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.827 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.827 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:54.827 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.559019+0000", 2024-03-19T11:31:54.827 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5999999999999999e-05 2024-03-19T11:31:54.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.827 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.827 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:54.827 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.569615+0000", 2024-03-19T11:31:54.827 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010596257 2024-03-19T11:31:54.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.827 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.827 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-03-19T11:31:54.827 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.569763+0000", 2024-03-19T11:31:54.827 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000147319 2024-03-19T11:31:54.827 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.828 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.828 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:54.828 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.592019+0000", 2024-03-19T11:31:54.828 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022256597999999999 2024-03-19T11:31:54.828 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.828 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.828 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:54.828 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.592250+0000", 2024-03-19T11:31:54.828 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023081100000000001 2024-03-19T11:31:54.828 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.828 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.828 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:54.829 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.592524+0000", 2024-03-19T11:31:54.829 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000273563 2024-03-19T11:31:54.829 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:54.829 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:54.829 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:54.829 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.829 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.829 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3106 4.15 4:ad3ad1b3:::smithi03829856-21:head [write 500908~484052] snapc 1f=[1f,1e,1d,1c,1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-19T11:31:54.829 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.563910+0000", 2024-03-19T11:31:54.829 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 31.199674859000002, 2024-03-19T11:31:54.829 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.029166066000000001, 2024-03-19T11:31:54.829 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:54.829 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:54.829 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:54.830 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:54.830 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:54.830 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3106 2024-03-19T11:31:54.830 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.830 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:54.830 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.830 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:54.830 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.563910+0000", 2024-03-19T11:31:54.830 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:54.830 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.830 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.830 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:54.830 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.563910+0000", 2024-03-19T11:31:54.830 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:54.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.831 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:54.831 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.563913+0000", 2024-03-19T11:31:54.831 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9960000000000002e-06 2024-03-19T11:31:54.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.831 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:54.831 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.564853+0000", 2024-03-19T11:31:54.831 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00093991599999999995 2024-03-19T11:31:54.831 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.831 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.832 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:54.832 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.564886+0000", 2024-03-19T11:31:54.832 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3315999999999997e-05 2024-03-19T11:31:54.832 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.832 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:54.832 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.564908+0000", 2024-03-19T11:31:54.832 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2262000000000001e-05 2024-03-19T11:31:54.832 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.832 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.832 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:54.832 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.569861+0000", 2024-03-19T11:31:54.833 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0049529070000000003 2024-03-19T11:31:54.833 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.833 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.833 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:54.833 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.569998+0000", 2024-03-19T11:31:54.833 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000136546 2024-03-19T11:31:54.833 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.833 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.833 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:31:54.833 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.570279+0000", 2024-03-19T11:31:54.833 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028119899999999999 2024-03-19T11:31:54.833 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.833 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.833 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:54.834 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.586586+0000", 2024-03-19T11:31:54.834 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016306621 2024-03-19T11:31:54.834 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.834 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.834 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:54.834 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.592743+0000", 2024-03-19T11:31:54.834 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061571150000000003 2024-03-19T11:31:54.834 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.834 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.834 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:54.834 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.592915+0000", 2024-03-19T11:31:54.834 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017273200000000001 2024-03-19T11:31:54.834 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.834 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.835 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:54.835 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.593076+0000", 2024-03-19T11:31:54.835 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016045600000000001 2024-03-19T11:31:54.835 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:54.835 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:54.835 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:54.835 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.835 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.835 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.21 130/129 [PushOp(4:866f3dcc:::smithi03829856-31:23, version: 112'156, data_included: [406769~892231,1439434~156345], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:23@112'156, size: 3899224, copy_subset: [(0, 3899224)], clone_subset: {}, snapset: 28=[]:{1c: [1a, 17], 23: [23, 22, 21, 1f, 1e, 1d], 28: [28, 27, 26, 25, 24]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1595779, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:1e, version: 122'169, data_included: [527786~724480,1703317~324096], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:1e@122'169, size: 3327786, copy_subset: [(0, 3327786)], clone_subset: {}, snapset: 1e=[]:{1e: [1e, 1d, 1a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2027413, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:1c, version: 122'171, data_included: [406769~892231,1439434~156345], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:1c@122'171, size: 3159716, copy_subset: [(0, 3159716)], clone_subset: {}, snapset: 28=[]:{1c: [1a, 17], 23: [23, 22, 21, 1f, 1e, 1d], 28: [28, 27, 26, 25, 24]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1595779, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:28, version: 124'174, data_included: [406769~892231,1439434~156345], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:28@124'174, size: 2504276, copy_subset: [(0, 2504276)], clone_subset: {}, snapset: 28=[]:{1c: [1a, 17], 23: [23, 22, 21, 1f, 1e, 1d], 28: [28, 27, 26, 25, 24]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1595779, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:head, version: 126'178, data_included: [406769~947798,1439434~100778], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:head@126'178, size: 3042100, copy_subset: [(0, 3042100)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1540212, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:54.835 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:51.141672+0000", 2024-03-19T11:31:54.835 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 3.6219128280000001, 2024-03-19T11:31:54.835 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.038330659000000003, 2024-03-19T11:31:54.835 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:54.835 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:54.835 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:54.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.836 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:54.836 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.141672+0000", 2024-03-19T11:31:54.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:54.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.836 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:54.836 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.141672+0000", 2024-03-19T11:31:54.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:54.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.836 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.836 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:54.836 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.141673+0000", 2024-03-19T11:31:54.836 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.361e-06 2024-03-19T11:31:54.836 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.837 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:54.837 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.149477+0000", 2024-03-19T11:31:54.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0078036019999999998 2024-03-19T11:31:54.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.837 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:54.837 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.149526+0000", 2024-03-19T11:31:54.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8730000000000003e-05 2024-03-19T11:31:54.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.837 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.837 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:54.837 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.149570+0000", 2024-03-19T11:31:54.837 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4493000000000002e-05 2024-03-19T11:31:54.837 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.838 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.838 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:54.838 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.149925+0000", 2024-03-19T11:31:54.838 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00035484699999999999 2024-03-19T11:31:54.838 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.838 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.838 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:54.838 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.149987+0000", 2024-03-19T11:31:54.838 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2182999999999997e-05 2024-03-19T11:31:54.838 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.838 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.838 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:54.838 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.180003+0000", 2024-03-19T11:31:54.838 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030015442999999999 2024-03-19T11:31:54.839 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:54.839 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:54.839 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:54.839 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.839 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:3885 4.21 e130/129 4:8401c1f0:::smithi03829856-43:head v 130'180, mlcod=0'0)", 2024-03-19T11:31:54.839 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:51.149617+0000", 2024-03-19T11:31:54.839 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 3.613967368, 2024-03-19T11:31:54.839 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.040106843000000003, 2024-03-19T11:31:54.839 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:54.839 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:54.839 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:54.839 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.839 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:54.839 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.149617+0000", 2024-03-19T11:31:54.840 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:54.840 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.840 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.840 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:54.840 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.149617+0000", 2024-03-19T11:31:54.840 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:54.840 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.840 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.840 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:54.840 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.149618+0000", 2024-03-19T11:31:54.840 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2e-07 2024-03-19T11:31:54.840 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.840 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.840 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:54.840 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.150382+0000", 2024-03-19T11:31:54.841 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00076398999999999998 2024-03-19T11:31:54.841 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.841 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.841 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:54.841 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.150391+0000", 2024-03-19T11:31:54.841 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6409999999999997e-06 2024-03-19T11:31:54.841 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.841 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.841 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:54.841 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.150398+0000", 2024-03-19T11:31:54.841 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2299999999999996e-06 2024-03-19T11:31:54.842 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.842 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.842 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:54.842 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.180043+0000", 2024-03-19T11:31:54.842 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.029645663999999999 2024-03-19T11:31:54.842 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.842 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.842 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:54.842 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.180068+0000", 2024-03-19T11:31:54.842 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4195000000000001e-05 2024-03-19T11:31:54.842 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.843 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:54.843 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.189686+0000", 2024-03-19T11:31:54.843 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0096187340000000003 2024-03-19T11:31:54.843 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.843 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:54.843 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.189724+0000", 2024-03-19T11:31:54.843 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7868999999999999e-05 2024-03-19T11:31:54.843 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:54.843 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:54.843 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:54.843 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.843 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.843 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.21 e130,129 4:8401c1f0:::smithi03829856-43:b,109'151,4:866f3dcc:::smithi03829856-31:b,109'153)", 2024-03-19T11:31:54.844 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:51.150411+0000", 2024-03-19T11:31:54.844 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 3.6131735300000001, 2024-03-19T11:31:54.844 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.039270521000000003, 2024-03-19T11:31:54.844 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:54.844 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:54.844 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:54.844 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.844 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:54.844 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.150411+0000", 2024-03-19T11:31:54.845 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:54.845 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.845 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.845 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:54.845 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.150411+0000", 2024-03-19T11:31:54.845 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:54.845 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.845 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.845 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:54.845 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.150412+0000", 2024-03-19T11:31:54.845 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9000000000000002e-07 2024-03-19T11:31:54.845 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.845 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.845 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:54.846 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.150416+0000", 2024-03-19T11:31:54.846 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4479999999999996e-06 2024-03-19T11:31:54.846 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.846 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.846 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:54.846 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.150433+0000", 2024-03-19T11:31:54.846 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6549999999999999e-05 2024-03-19T11:31:54.846 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.846 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.846 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:54.846 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.150439+0000", 2024-03-19T11:31:54.847 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1410000000000002e-06 2024-03-19T11:31:54.847 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.847 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.847 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:54.847 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.189519+0000", 2024-03-19T11:31:54.847 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.039080239000000003 2024-03-19T11:31:54.847 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.847 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.847 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:54.847 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.189528+0000", 2024-03-19T11:31:54.847 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2580000000000001e-06 2024-03-19T11:31:54.847 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.847 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.847 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:54.848 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.189682+0000", 2024-03-19T11:31:54.848 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015349499999999999 2024-03-19T11:31:54.848 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:54.848 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:54.848 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:54.848 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.848 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.848 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:3886 4.21 e130/129 4:8401c1f0:::smithi03829856-43:head v 130'181, mlcod=0'0)", 2024-03-19T11:31:54.848 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:51.153376+0000", 2024-03-19T11:31:54.848 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 3.6102088229999998, 2024-03-19T11:31:54.848 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.036393429999999997, 2024-03-19T11:31:54.848 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:54.848 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:54.848 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:54.849 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.849 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:54.849 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.153376+0000", 2024-03-19T11:31:54.849 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:54.849 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.849 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.849 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:54.849 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.153376+0000", 2024-03-19T11:31:54.849 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:54.849 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.849 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.850 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:54.850 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.153377+0000", 2024-03-19T11:31:54.850 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9299999999999997e-07 2024-03-19T11:31:54.850 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.850 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.850 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:54.850 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.153879+0000", 2024-03-19T11:31:54.850 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00050270000000000002 2024-03-19T11:31:54.850 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.850 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.850 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:54.850 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.153897+0000", 2024-03-19T11:31:54.851 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7676000000000002e-05 2024-03-19T11:31:54.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.851 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.851 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:54.851 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.153906+0000", 2024-03-19T11:31:54.851 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9390000000000003e-06 2024-03-19T11:31:54.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.851 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.851 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:54.851 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.184615+0000", 2024-03-19T11:31:54.851 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030708566 2024-03-19T11:31:54.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.852 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.852 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:54.852 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.184660+0000", 2024-03-19T11:31:54.852 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5933999999999998e-05 2024-03-19T11:31:54.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.852 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.852 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:54.852 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.189731+0000", 2024-03-19T11:31:54.852 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050710190000000004 2024-03-19T11:31:54.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.852 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.853 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:54.853 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.189769+0000", 2024-03-19T11:31:54.853 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7802999999999997e-05 2024-03-19T11:31:54.853 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:54.853 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:54.853 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:54.853 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.853 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.853 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:3887 4.21 e130/129 4:8401c1f0:::smithi03829856-43:head v 130'182, mlcod=0'0)", 2024-03-19T11:31:54.853 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:51.158488+0000", 2024-03-19T11:31:54.853 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 3.6050969820000001, 2024-03-19T11:31:54.853 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.034422726000000001, 2024-03-19T11:31:54.853 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:54.853 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:54.854 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:54.854 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.854 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:54.854 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.158488+0000", 2024-03-19T11:31:54.854 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:54.854 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.854 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.854 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:54.854 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.158488+0000", 2024-03-19T11:31:54.854 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:54.854 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.854 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.854 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:54.854 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.158491+0000", 2024-03-19T11:31:54.854 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8490000000000002e-06 2024-03-19T11:31:54.855 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.855 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.855 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:54.855 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.159149+0000", 2024-03-19T11:31:54.855 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00065869500000000005 2024-03-19T11:31:54.855 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.855 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.855 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:54.855 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.159186+0000", 2024-03-19T11:31:54.855 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6607999999999999e-05 2024-03-19T11:31:54.855 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.855 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.855 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:54.855 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.159204+0000", 2024-03-19T11:31:54.855 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8099999999999999e-05 2024-03-19T11:31:54.856 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.856 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.856 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:54.856 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.187524+0000", 2024-03-19T11:31:54.856 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.028320271000000001 2024-03-19T11:31:54.856 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.856 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.856 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:54.856 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.187558+0000", 2024-03-19T11:31:54.856 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4078999999999997e-05 2024-03-19T11:31:54.857 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.857 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.857 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:54.857 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.192869+0000", 2024-03-19T11:31:54.857 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053108269999999997 2024-03-19T11:31:54.857 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.857 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.857 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:54.857 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.192910+0000", 2024-03-19T11:31:54.857 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1297000000000001e-05 2024-03-19T11:31:54.857 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:54.857 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:54.858 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:54.858 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.858 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.858 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:3888 4.21 e130/129 4:8401c1f0:::smithi03829856-43:head v 130'183, mlcod=0'0)", 2024-03-19T11:31:54.858 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:51.160972+0000", 2024-03-19T11:31:54.858 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 3.60261224, 2024-03-19T11:31:54.858 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032362711000000002, 2024-03-19T11:31:54.858 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:54.858 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:54.858 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:54.858 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.858 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:54.858 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.160972+0000", 2024-03-19T11:31:54.858 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:54.859 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.859 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.859 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:54.859 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.160972+0000", 2024-03-19T11:31:54.859 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:54.859 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.859 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.859 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:54.859 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.160974+0000", 2024-03-19T11:31:54.860 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.677e-06 2024-03-19T11:31:54.860 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.860 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.860 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:54.860 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.160987+0000", 2024-03-19T11:31:54.860 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3076e-05 2024-03-19T11:31:54.860 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.860 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.860 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:54.860 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.161009+0000", 2024-03-19T11:31:54.860 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.154e-05 2024-03-19T11:31:54.860 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.860 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.860 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:54.860 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.161021+0000", 2024-03-19T11:31:54.861 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2559e-05 2024-03-19T11:31:54.861 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.861 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.861 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:54.861 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.188979+0000", 2024-03-19T11:31:54.861 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027958061999999999 2024-03-19T11:31:54.861 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.861 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.861 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:54.861 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.189014+0000", 2024-03-19T11:31:54.861 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4644999999999998e-05 2024-03-19T11:31:54.861 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.861 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.862 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:54.862 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.193300+0000", 2024-03-19T11:31:54.862 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0042857939999999999 2024-03-19T11:31:54.862 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:54.862 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:54.862 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:54.862 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.193335+0000", 2024-03-19T11:31:54.862 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5358000000000003e-05 2024-03-19T11:31:54.862 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:54.862 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:54.862 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:54.862 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:54.862 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:54.862 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:54.864 INFO:tasks.rados.rados.0.smithi038.stdout:1731: finishing rollback tid 0 to smithi03829856-35 2024-03-19T11:31:54.864 INFO:tasks.rados.rados.0.smithi038.stdout:1731: finishing rollback tid 1 to smithi03829856-35 2024-03-19T11:31:54.864 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 35 v 303 (ObjNum 97 snap 0 seq_num 1) dirty dne 2024-03-19T11:31:54.864 INFO:tasks.rados.rados.0.smithi038.stdout:1732: expect (ObjNum 318 snap 40 seq_num 318) 2024-03-19T11:31:54.865 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-19T11:31:54.865 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:54.871+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:54.916 INFO:tasks.rados.rados.0.smithi038.stdout:1734: expect (ObjNum 195 snap 22 seq_num 195) 2024-03-19T11:31:54.965 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:54.973+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:54.991 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:54.991 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:31:54.992 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:31:54.992 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:55.005 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-19T11:31:55.066 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:55.075+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:55.106 INFO:tasks.rados.rados.0.smithi038.stdout:1730: expect (ObjNum 314 snap 40 seq_num 314) 2024-03-19T11:31:55.147 INFO:teuthology.orchestra.run.smithi038.stderr:marked in osd.3. 2024-03-19T11:31:55.162 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-03-19T11:31:55.163 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:55.171+0000 7f70c4fdb640 -1 osd.0 133 handle_osd_map injecting map crc failure 2024-03-19T11:31:55.166 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:55.173+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:55.175 INFO:tasks.thrashosds.thrasher:Added osd 3 2024-03-19T11:31:55.182 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:55.183 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:31:55.183 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:31:55.183 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:31:55.183 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:55.197 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-19T11:31:55.268 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:55.275+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:55.349 INFO:tasks.rados.rados.0.smithi038.stdout:1736: expect (ObjNum 210 snap 22 seq_num 210) 2024-03-19T11:31:55.364 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:55.365 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:31:55.365 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:31:55.365 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:31:55.365 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.365 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:30 4.5 4:ac9cd3a3:::smithi03829856-8:head [write 1776668~340981] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:31:55.365 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:04.777922+0000", 2024-03-19T11:31:55.365 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 110.59477610899999, 2024-03-19T11:31:55.365 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015469894, 2024-03-19T11:31:55.365 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:55.365 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:55.365 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:55.365 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:55.365 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:55.366 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 30 2024-03-19T11:31:55.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.366 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:55.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.366 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:55.366 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.777922+0000", 2024-03-19T11:31:55.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.366 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:55.366 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.777922+0000", 2024-03-19T11:31:55.366 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.366 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.366 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.367 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:55.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.777924+0000", 2024-03-19T11:31:55.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3999999999999999e-06 2024-03-19T11:31:55.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.367 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:55.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778461+0000", 2024-03-19T11:31:55.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00053698299999999999 2024-03-19T11:31:55.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.367 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:55.367 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778487+0000", 2024-03-19T11:31:55.367 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5947000000000001e-05 2024-03-19T11:31:55.367 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.367 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.368 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:55.368 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778502+0000", 2024-03-19T11:31:55.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4256e-05 2024-03-19T11:31:55.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.368 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:55.368 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.781978+0000", 2024-03-19T11:31:55.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00347693 2024-03-19T11:31:55.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.368 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:55.368 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.782196+0000", 2024-03-19T11:31:55.368 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021726499999999999 2024-03-19T11:31:55.368 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.368 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.369 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:55.369 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.782572+0000", 2024-03-19T11:31:55.369 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00037635000000000001 2024-03-19T11:31:55.369 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.369 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.369 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:55.369 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.786960+0000", 2024-03-19T11:31:55.369 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043882130000000002 2024-03-19T11:31:55.369 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.369 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.369 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:55.369 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793113+0000", 2024-03-19T11:31:55.369 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061530760000000004 2024-03-19T11:31:55.369 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.369 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.370 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:55.370 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793276+0000", 2024-03-19T11:31:55.370 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000162652 2024-03-19T11:31:55.370 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.370 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.370 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:55.370 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793392+0000", 2024-03-19T11:31:55.370 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115822 2024-03-19T11:31:55.370 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.370 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:55.370 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.370 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.370 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.370 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:32 4.5 4:ac9cd3a3:::smithi03829856-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:31:55.371 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:04.778582+0000", 2024-03-19T11:31:55.371 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 110.59411646, 2024-03-19T11:31:55.371 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015761827999999999, 2024-03-19T11:31:55.371 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:55.371 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:55.371 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:55.371 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:55.371 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:55.371 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 32 2024-03-19T11:31:55.371 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.371 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:55.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.372 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:55.372 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778582+0000", 2024-03-19T11:31:55.372 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.372 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:55.372 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778582+0000", 2024-03-19T11:31:55.372 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.372 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.372 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.373 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:55.373 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778583+0000", 2024-03-19T11:31:55.373 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.026e-06 2024-03-19T11:31:55.373 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.373 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.373 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:55.373 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778587+0000", 2024-03-19T11:31:55.373 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7870000000000001e-06 2024-03-19T11:31:55.373 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.373 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.373 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:55.373 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778599+0000", 2024-03-19T11:31:55.373 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1623e-05 2024-03-19T11:31:55.373 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.374 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:55.374 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778608+0000", 2024-03-19T11:31:55.374 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7399999999999993e-06 2024-03-19T11:31:55.374 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.374 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:55.374 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.787009+0000", 2024-03-19T11:31:55.374 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0084010490000000007 2024-03-19T11:31:55.374 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.374 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.374 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-03-19T11:31:55.374 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.787202+0000", 2024-03-19T11:31:55.374 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019309500000000001 2024-03-19T11:31:55.375 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.375 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.375 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:55.375 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793832+0000", 2024-03-19T11:31:55.375 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.006630052 2024-03-19T11:31:55.375 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.375 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.375 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:55.375 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.794048+0000", 2024-03-19T11:31:55.375 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000215505 2024-03-19T11:31:55.375 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.375 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.375 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:55.375 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.794343+0000", 2024-03-19T11:31:55.375 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029595100000000003 2024-03-19T11:31:55.376 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.376 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:55.376 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.376 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.376 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.376 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:891 4.5 4:af9d0d56:::smithi03829856-20:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:31:55.376 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:09.286410+0000", 2024-03-19T11:31:55.376 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 106.08628812800001, 2024-03-19T11:31:55.376 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035191405000000002, 2024-03-19T11:31:55.376 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:55.376 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:55.376 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:55.376 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:55.376 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:55.376 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 891 2024-03-19T11:31:55.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.377 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:55.377 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.377 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:55.377 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286410+0000", 2024-03-19T11:31:55.377 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.377 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.377 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:55.377 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286410+0000", 2024-03-19T11:31:55.377 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.377 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.377 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.377 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:55.377 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286413+0000", 2024-03-19T11:31:55.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2660000000000002e-06 2024-03-19T11:31:55.378 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.378 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.378 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:55.378 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286426+0000", 2024-03-19T11:31:55.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.258e-05 2024-03-19T11:31:55.378 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.378 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.378 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:55.378 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286460+0000", 2024-03-19T11:31:55.378 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3943999999999999e-05 2024-03-19T11:31:55.378 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.378 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.378 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:55.379 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286477+0000", 2024-03-19T11:31:55.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6799999999999998e-05 2024-03-19T11:31:55.379 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.379 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.379 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:55.379 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286605+0000", 2024-03-19T11:31:55.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000128892 2024-03-19T11:31:55.379 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.379 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.379 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:55.379 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286769+0000", 2024-03-19T11:31:55.379 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016359799999999999 2024-03-19T11:31:55.379 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.379 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.379 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:55.380 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.321601+0000", 2024-03-19T11:31:55.380 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.034832324999999997 2024-03-19T11:31:55.380 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.380 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:55.380 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.380 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.380 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.380 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1015 4.5 4:af9d0d56:::smithi03829856-20:head [tier-promote] snapc 6=[6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:31:55.380 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:12.761855+0000", 2024-03-19T11:31:55.380 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 102.61084260200001, 2024-03-19T11:31:55.380 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026760430000000002, 2024-03-19T11:31:55.380 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:55.380 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:55.381 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:55.381 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:55.381 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:55.381 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1015 2024-03-19T11:31:55.381 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.381 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:55.381 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.381 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:55.381 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761855+0000", 2024-03-19T11:31:55.381 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.381 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.381 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.381 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:55.381 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761855+0000", 2024-03-19T11:31:55.381 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.382 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:55.382 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761858+0000", 2024-03-19T11:31:55.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8140000000000001e-06 2024-03-19T11:31:55.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.382 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:55.382 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761872+0000", 2024-03-19T11:31:55.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3813e-05 2024-03-19T11:31:55.382 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.382 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.382 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:55.382 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761905+0000", 2024-03-19T11:31:55.382 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2642999999999999e-05 2024-03-19T11:31:55.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:55.383 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761925+0000", 2024-03-19T11:31:55.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0489e-05 2024-03-19T11:31:55.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:55.383 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.762070+0000", 2024-03-19T11:31:55.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000145153 2024-03-19T11:31:55.383 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.383 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.383 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:55.383 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.762365+0000", 2024-03-19T11:31:55.383 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029482100000000002 2024-03-19T11:31:55.384 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.384 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.384 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:55.384 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.788616+0000", 2024-03-19T11:31:55.384 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026250697 2024-03-19T11:31:55.384 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.384 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:55.384 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.384 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.384 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.384 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1121 4.5 4:aa3303a5:::smithi03829856-17:head [tier-promote] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:31:55.384 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:14.164281+0000", 2024-03-19T11:31:55.384 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 101.208417357, 2024-03-19T11:31:55.384 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023896435000000001, 2024-03-19T11:31:55.384 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:55.385 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:55.385 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:55.385 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:55.385 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:55.385 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1121 2024-03-19T11:31:55.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.385 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:55.385 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.385 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:55.385 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164281+0000", 2024-03-19T11:31:55.385 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.385 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.385 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.385 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:55.385 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164281+0000", 2024-03-19T11:31:55.386 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.386 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:55.386 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164283+0000", 2024-03-19T11:31:55.386 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8500000000000001e-06 2024-03-19T11:31:55.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.386 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:55.386 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164291+0000", 2024-03-19T11:31:55.386 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.14e-06 2024-03-19T11:31:55.386 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.386 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.386 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:55.387 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164313+0000", 2024-03-19T11:31:55.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.207e-05 2024-03-19T11:31:55.387 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.387 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:55.387 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164328+0000", 2024-03-19T11:31:55.387 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5403000000000001e-05 2024-03-19T11:31:55.387 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.387 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.387 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:55.387 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.167981+0000", 2024-03-19T11:31:55.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036529879999999998 2024-03-19T11:31:55.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:55.388 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.168274+0000", 2024-03-19T11:31:55.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029313100000000002 2024-03-19T11:31:55.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.388 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.388 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:55.388 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.188177+0000", 2024-03-19T11:31:55.388 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019902853000000002 2024-03-19T11:31:55.388 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.388 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:55.388 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.388 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.389 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.389 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1302 4.15 4:aa3303a5:::smithi03829856-17:head [write 413472~620140,stat] snapc b=[b,a,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-03-19T11:31:55.389 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:22.966426+0000", 2024-03-19T11:31:55.389 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 92.406271623999999, 2024-03-19T11:31:55.389 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018043513000000001, 2024-03-19T11:31:55.389 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:55.389 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:55.389 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:55.389 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:55.389 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:55.389 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1302 2024-03-19T11:31:55.389 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.389 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:55.389 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.389 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:55.390 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.966426+0000", 2024-03-19T11:31:55.390 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.390 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.390 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:55.390 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.966426+0000", 2024-03-19T11:31:55.390 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.390 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.390 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:55.390 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.966429+0000", 2024-03-19T11:31:55.390 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3829999999999999e-06 2024-03-19T11:31:55.390 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.391 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.391 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:55.391 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972117+0000", 2024-03-19T11:31:55.391 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0056885750000000004 2024-03-19T11:31:55.391 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.391 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.391 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:55.391 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972156+0000", 2024-03-19T11:31:55.391 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8878000000000003e-05 2024-03-19T11:31:55.391 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.391 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.391 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:55.391 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972184+0000", 2024-03-19T11:31:55.391 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7509999999999999e-05 2024-03-19T11:31:55.391 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.392 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:55.392 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972309+0000", 2024-03-19T11:31:55.392 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012525899999999999 2024-03-19T11:31:55.392 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.392 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:55.392 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972566+0000", 2024-03-19T11:31:55.392 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025740799999999999 2024-03-19T11:31:55.392 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.392 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.392 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:55.392 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972899+0000", 2024-03-19T11:31:55.392 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033238799999999998 2024-03-19T11:31:55.393 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.393 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:55.393 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984088+0000", 2024-03-19T11:31:55.393 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011189466 2024-03-19T11:31:55.393 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.393 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:55.393 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984138+0000", 2024-03-19T11:31:55.393 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9666999999999997e-05 2024-03-19T11:31:55.393 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.393 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.393 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:55.393 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984329+0000", 2024-03-19T11:31:55.393 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019086799999999999 2024-03-19T11:31:55.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.394 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.394 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:55.394 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984470+0000", 2024-03-19T11:31:55.394 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000141111 2024-03-19T11:31:55.394 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.394 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:55.394 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.394 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.394 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.1 50/49 [PushOp(4:866f3dcc:::smithi03829856-31:7, version: 38'44, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@38'44, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 45'62, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@45'62, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 46'68, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@46'68, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:7, version: 47'71, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@47'71, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 47'72, data_included: [440350~816629,1309629~231947], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@47'72, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:55.394 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.959457+0000", 2024-03-19T11:31:55.394 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 87.413241206999999, 2024-03-19T11:31:55.394 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018416666000000002, 2024-03-19T11:31:55.395 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:55.395 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:55.395 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:55.395 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.395 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:55.395 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.959457+0000", 2024-03-19T11:31:55.395 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.395 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.395 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.395 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:55.395 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.959457+0000", 2024-03-19T11:31:55.396 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.396 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.396 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.396 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:55.396 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.959458+0000", 2024-03-19T11:31:55.396 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4899999999999999e-06 2024-03-19T11:31:55.396 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.396 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.396 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:55.396 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.962822+0000", 2024-03-19T11:31:55.396 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0033640620000000001 2024-03-19T11:31:55.397 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.397 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.397 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:55.397 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.962893+0000", 2024-03-19T11:31:55.397 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1060000000000001e-05 2024-03-19T11:31:55.397 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.397 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.397 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:55.397 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.962977+0000", 2024-03-19T11:31:55.397 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3311000000000002e-05 2024-03-19T11:31:55.397 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.397 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.397 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:55.397 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963208+0000", 2024-03-19T11:31:55.397 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023121200000000001 2024-03-19T11:31:55.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.398 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.398 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:55.398 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963245+0000", 2024-03-19T11:31:55.398 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6844999999999997e-05 2024-03-19T11:31:55.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.398 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.398 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:55.398 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.977874+0000", 2024-03-19T11:31:55.398 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014628686 2024-03-19T11:31:55.398 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.398 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:55.398 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.398 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.399 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.399 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2284 4.15 4:afa9f208:::smithi03829856-35:head [read 1314816~702976] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-03-19T11:31:55.399 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:53.596775+0000", 2024-03-19T11:31:55.399 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 61.775923120999998, 2024-03-19T11:31:55.399 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.47260528299999999, 2024-03-19T11:31:55.399 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:55.399 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:55.399 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:55.399 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:55.399 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:55.399 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2284 2024-03-19T11:31:55.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.400 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:55.400 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.400 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:55.400 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596775+0000", 2024-03-19T11:31:55.400 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.400 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.400 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:55.400 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596775+0000", 2024-03-19T11:31:55.400 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.400 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.400 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.400 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:55.401 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596778+0000", 2024-03-19T11:31:55.401 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.174e-06 2024-03-19T11:31:55.401 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.401 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.401 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:55.401 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596794+0000", 2024-03-19T11:31:55.401 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5372000000000001e-05 2024-03-19T11:31:55.401 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.401 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.401 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:55.401 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596828+0000", 2024-03-19T11:31:55.401 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4107999999999999e-05 2024-03-19T11:31:55.401 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.401 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.401 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:55.402 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596844+0000", 2024-03-19T11:31:55.402 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6296000000000001e-05 2024-03-19T11:31:55.402 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.402 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.402 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:55.402 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596968+0000", 2024-03-19T11:31:55.402 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012383699999999999 2024-03-19T11:31:55.402 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.402 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.402 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:31:55.402 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596988+0000", 2024-03-19T11:31:55.402 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0645e-05 2024-03-19T11:31:55.402 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.402 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.403 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:55.403 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.068916+0000", 2024-03-19T11:31:55.403 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.47192744800000003 2024-03-19T11:31:55.403 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.403 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.403 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:55.403 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.069015+0000", 2024-03-19T11:31:55.403 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9500999999999995e-05 2024-03-19T11:31:55.403 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.403 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.403 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:55.403 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.069380+0000", 2024-03-19T11:31:55.403 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00036490200000000001 2024-03-19T11:31:55.403 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.403 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:55.404 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.404 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.404 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2286 4.15 4:afa9f208:::smithi03829856-35:head [assert-version v108,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e75)", 2024-03-19T11:31:55.404 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.084826+0000", 2024-03-19T11:31:55.404 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 61.287871754000001, 2024-03-19T11:31:55.404 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.055799556, 2024-03-19T11:31:55.404 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:55.404 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:55.404 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:55.404 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:55.404 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:55.404 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2286 2024-03-19T11:31:55.404 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.405 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:55.405 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.405 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:55.405 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084826+0000", 2024-03-19T11:31:55.405 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.405 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.405 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.405 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:55.405 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084826+0000", 2024-03-19T11:31:55.405 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.405 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.405 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.405 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:55.405 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084828+0000", 2024-03-19T11:31:55.405 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1409999999999999e-06 2024-03-19T11:31:55.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.406 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:55.406 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084838+0000", 2024-03-19T11:31:55.406 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3740000000000004e-06 2024-03-19T11:31:55.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.406 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:55.406 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084859+0000", 2024-03-19T11:31:55.406 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1644000000000001e-05 2024-03-19T11:31:55.406 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.406 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.406 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:55.406 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084871+0000", 2024-03-19T11:31:55.406 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1127e-05 2024-03-19T11:31:55.407 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.407 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.407 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:55.407 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084961+0000", 2024-03-19T11:31:55.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0371999999999995e-05 2024-03-19T11:31:55.407 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.407 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.407 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:55.407 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.085130+0000", 2024-03-19T11:31:55.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016884800000000001 2024-03-19T11:31:55.407 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.407 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.407 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5,7", 2024-03-19T11:31:55.407 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.089309+0000", 2024-03-19T11:31:55.407 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041788650000000004 2024-03-19T11:31:55.408 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.408 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:55.408 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.090362+0000", 2024-03-19T11:31:55.408 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001053362 2024-03-19T11:31:55.408 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.408 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:55.408 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.090511+0000", 2024-03-19T11:31:55.408 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014887200000000001 2024-03-19T11:31:55.408 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.408 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.408 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:55.408 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.140479+0000", 2024-03-19T11:31:55.409 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.049968006000000002 2024-03-19T11:31:55.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.409 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.409 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:55.409 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.140534+0000", 2024-03-19T11:31:55.409 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4917000000000003e-05 2024-03-19T11:31:55.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.409 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.409 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:55.409 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.140626+0000", 2024-03-19T11:31:55.409 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2027999999999994e-05 2024-03-19T11:31:55.409 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.409 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:55.409 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.409 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.410 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1d58db11:::smithi03829856-18:6, version: 55'80, data_included: [759683~639358,1904434~409218], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:6@55'80, size: 3159683, copy_subset: [(0, 3159683)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2313652, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1d58db11:::smithi03829856-18:b, version: 58'84, data_included: [443718~955323,1564138~93253], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:b@58'84, size: 3445402, copy_subset: [(0, 3445402)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1657391, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:10, version: 66'95, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:10@66'95, size: 3764735, copy_subset: [(0, 3764735)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:12, version: 66'97, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:12@66'97, size: 2504243, copy_subset: [(0, 2504243)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:c, version: 70'102, data_included: [416807~576167,1542129~472409], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:c@70'102, size: 2857157, copy_subset: [(0, 2857157)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2014538, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:55.410 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.274749+0000", 2024-03-19T11:31:55.410 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 61.097948823000003, 2024-03-19T11:31:55.410 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017652601, 2024-03-19T11:31:55.410 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:55.410 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:55.410 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:55.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.410 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:55.410 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.274749+0000", 2024-03-19T11:31:55.410 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.410 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.410 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.411 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:55.411 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.274749+0000", 2024-03-19T11:31:55.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.411 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:55.411 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.274751+0000", 2024-03-19T11:31:55.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8500000000000001e-06 2024-03-19T11:31:55.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.411 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:55.411 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.278426+0000", 2024-03-19T11:31:55.411 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036747559999999999 2024-03-19T11:31:55.411 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.411 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:55.412 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.278470+0000", 2024-03-19T11:31:55.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4119000000000002e-05 2024-03-19T11:31:55.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:55.412 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.278506+0000", 2024-03-19T11:31:55.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6066000000000003e-05 2024-03-19T11:31:55.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.412 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:55.412 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.278709+0000", 2024-03-19T11:31:55.412 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000202711 2024-03-19T11:31:55.412 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.412 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.413 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:55.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.278731+0000", 2024-03-19T11:31:55.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2657e-05 2024-03-19T11:31:55.413 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.413 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.413 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:55.413 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.292402+0000", 2024-03-19T11:31:55.413 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013670442 2024-03-19T11:31:55.413 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.414 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:55.414 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.414 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.414 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1dabc0f1:::smithi03829856-1:head, version: 70'103, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:head@70'103, size: 2621892, copy_subset: [(0, 2621892)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:55.414 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.280012+0000", 2024-03-19T11:31:55.414 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 61.092686178000001, 2024-03-19T11:31:55.414 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018569209, 2024-03-19T11:31:55.414 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:55.414 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:55.414 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:55.414 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.414 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:55.414 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280012+0000", 2024-03-19T11:31:55.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:55.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280012+0000", 2024-03-19T11:31:55.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:55.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280013+0000", 2024-03-19T11:31:55.415 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9899999999999999e-07 2024-03-19T11:31:55.415 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.415 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.415 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:55.415 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280584+0000", 2024-03-19T11:31:55.416 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057090299999999999 2024-03-19T11:31:55.416 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.416 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:55.416 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280612+0000", 2024-03-19T11:31:55.416 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8442999999999999e-05 2024-03-19T11:31:55.416 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.416 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:55.416 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280631+0000", 2024-03-19T11:31:55.416 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8896999999999999e-05 2024-03-19T11:31:55.416 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.416 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.416 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:55.416 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.295041+0000", 2024-03-19T11:31:55.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014410127999999999 2024-03-19T11:31:55.417 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.417 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:55.417 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.295050+0000", 2024-03-19T11:31:55.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2350000000000005e-06 2024-03-19T11:31:55.417 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.417 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.417 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:55.417 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.298581+0000", 2024-03-19T11:31:55.417 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035308039999999998 2024-03-19T11:31:55.418 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.418 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:55.418 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.418 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.418 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.18 e75,74 4:1d58db11:::smithi03829856-18:c,70'104)", 2024-03-19T11:31:55.418 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.280680+0000", 2024-03-19T11:31:55.418 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 61.092017703000003, 2024-03-19T11:31:55.418 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018106229000000001, 2024-03-19T11:31:55.418 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:55.418 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:55.418 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:55.418 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.418 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:55.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280680+0000", 2024-03-19T11:31:55.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.419 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:55.419 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280680+0000", 2024-03-19T11:31:55.419 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.419 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.419 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:55.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280681+0000", 2024-03-19T11:31:55.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2099999999999995e-07 2024-03-19T11:31:55.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:55.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280686+0000", 2024-03-19T11:31:55.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2240000000000003e-06 2024-03-19T11:31:55.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.420 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:55.420 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280698+0000", 2024-03-19T11:31:55.420 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1698999999999999e-05 2024-03-19T11:31:55.420 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.420 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.421 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:55.421 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280705+0000", 2024-03-19T11:31:55.421 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4350000000000002e-06 2024-03-19T11:31:55.421 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.421 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.421 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:55.421 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.298643+0000", 2024-03-19T11:31:55.421 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017938352000000001 2024-03-19T11:31:55.421 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.421 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.421 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:55.421 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.298656+0000", 2024-03-19T11:31:55.421 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3084999999999999e-05 2024-03-19T11:31:55.421 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.421 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.422 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:55.422 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.298787+0000", 2024-03-19T11:31:55.422 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013061299999999999 2024-03-19T11:31:55.422 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.422 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:55.422 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.422 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.422 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.422 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.5.0:143 4.18 e75/74 4:1d58db11:::smithi03829856-18:head v 75'106, mlcod=0'0)", 2024-03-19T11:31:55.422 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.282327+0000", 2024-03-19T11:31:55.422 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 61.090371331999997, 2024-03-19T11:31:55.422 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016753244, 2024-03-19T11:31:55.422 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:55.423 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:55.423 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:55.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:55.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.282327+0000", 2024-03-19T11:31:55.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:55.423 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.282327+0000", 2024-03-19T11:31:55.423 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.423 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.423 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.423 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:55.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.282328+0000", 2024-03-19T11:31:55.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.488e-06 2024-03-19T11:31:55.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:55.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.283127+0000", 2024-03-19T11:31:55.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00079853699999999999 2024-03-19T11:31:55.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:55.424 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.283152+0000", 2024-03-19T11:31:55.424 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5215000000000002e-05 2024-03-19T11:31:55.424 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.424 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.424 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:55.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.283167+0000", 2024-03-19T11:31:55.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4635e-05 2024-03-19T11:31:55.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:55.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.292633+0000", 2024-03-19T11:31:55.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0094664339999999993 2024-03-19T11:31:55.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.425 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.425 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:55.425 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.292655+0000", 2024-03-19T11:31:55.425 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1934999999999999e-05 2024-03-19T11:31:55.425 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.426 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:55.426 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.298981+0000", 2024-03-19T11:31:55.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0063255129999999996 2024-03-19T11:31:55.426 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.426 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:55.426 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.299080+0000", 2024-03-19T11:31:55.426 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9487000000000005e-05 2024-03-19T11:31:55.426 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.426 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:55.426 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.426 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.426 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.427 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2441 4.5 4:a38b9453:::smithi03829856-37:head [delete] snapc 17=[17,16,15,13,12,b,6] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e84)", 2024-03-19T11:31:55.427 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:03.060663+0000", 2024-03-19T11:31:55.427 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 52.312035487999999, 2024-03-19T11:31:55.427 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10917726, 2024-03-19T11:31:55.427 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:55.427 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:55.427 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:55.427 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:55.427 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:55.427 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2441 2024-03-19T11:31:55.427 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.427 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:55.427 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.427 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:55.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060663+0000", 2024-03-19T11:31:55.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.428 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:55.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060663+0000", 2024-03-19T11:31:55.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.428 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:55.428 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060666+0000", 2024-03-19T11:31:55.428 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.05e-06 2024-03-19T11:31:55.428 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.428 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:55.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060680+0000", 2024-03-19T11:31:55.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4276e-05 2024-03-19T11:31:55.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:55.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060714+0000", 2024-03-19T11:31:55.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4325000000000001e-05 2024-03-19T11:31:55.429 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.429 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.429 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:55.429 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060737+0000", 2024-03-19T11:31:55.429 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2959000000000002e-05 2024-03-19T11:31:55.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.430 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.430 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:55.430 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060895+0000", 2024-03-19T11:31:55.430 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000157787 2024-03-19T11:31:55.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.430 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.430 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:31:55.430 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060923+0000", 2024-03-19T11:31:55.430 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8019000000000001e-05 2024-03-19T11:31:55.430 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.430 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.430 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:55.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.066222+0000", 2024-03-19T11:31:55.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052989270000000002 2024-03-19T11:31:55.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:55.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.070153+0000", 2024-03-19T11:31:55.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039315119999999999 2024-03-19T11:31:55.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:55.431 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.147930+0000", 2024-03-19T11:31:55.431 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.077776459000000006 2024-03-19T11:31:55.431 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.431 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.431 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:55.432 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.148079+0000", 2024-03-19T11:31:55.432 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014888 2024-03-19T11:31:55.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.432 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:55.432 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.148370+0000", 2024-03-19T11:31:55.432 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029130999999999997 2024-03-19T11:31:55.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.432 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:55.432 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.152531+0000", 2024-03-19T11:31:55.432 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041605399999999999 2024-03-19T11:31:55.432 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.432 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.432 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:55.433 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.169762+0000", 2024-03-19T11:31:55.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017231483999999998 2024-03-19T11:31:55.433 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.433 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:55.433 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.169807+0000", 2024-03-19T11:31:55.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5161999999999999e-05 2024-03-19T11:31:55.433 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.433 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.433 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:55.433 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.169840+0000", 2024-03-19T11:31:55.433 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2570000000000002e-05 2024-03-19T11:31:55.433 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.433 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:55.433 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.434 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.434 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.434 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPushReply(4.5 84/83 [PushReplyOp(4:a38b9453:::smithi03829856-37:head)])", 2024-03-19T11:31:55.434 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:03.084896+0000", 2024-03-19T11:31:55.434 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 52.287802354, 2024-03-19T11:31:55.434 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.02071104, 2024-03-19T11:31:55.434 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:55.434 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-19T11:31:55.434 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:55.434 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.434 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:55.434 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.084896+0000", 2024-03-19T11:31:55.434 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.434 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.435 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:55.435 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.084896+0000", 2024-03-19T11:31:55.435 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.435 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.435 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:55.435 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.084897+0000", 2024-03-19T11:31:55.435 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.686e-06 2024-03-19T11:31:55.435 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.435 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.435 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:55.435 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.084910+0000", 2024-03-19T11:31:55.435 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2948e-05 2024-03-19T11:31:55.435 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.436 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.436 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:55.436 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.084936+0000", 2024-03-19T11:31:55.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5982e-05 2024-03-19T11:31:55.436 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.436 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.436 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:55.436 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.084955+0000", 2024-03-19T11:31:55.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8416000000000001e-05 2024-03-19T11:31:55.436 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.436 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.436 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:55.436 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.102822+0000", 2024-03-19T11:31:55.436 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017866756000000001 2024-03-19T11:31:55.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.437 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.437 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:55.437 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.105607+0000", 2024-03-19T11:31:55.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027852520000000002 2024-03-19T11:31:55.437 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.437 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:55.437 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.437 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.437 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.437 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.4.0:540 4.18 e111/86 4:1d58db11:::smithi03829856-18:head v 111'153, mlcod=108'150)", 2024-03-19T11:31:55.437 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:31.238185+0000", 2024-03-19T11:31:55.437 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 24.134512991000001, 2024-03-19T11:31:55.437 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021779390999999999, 2024-03-19T11:31:55.438 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:55.438 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:55.438 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:55.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.438 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:55.438 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.238185+0000", 2024-03-19T11:31:55.438 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.438 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.438 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.439 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:55.439 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.238185+0000", 2024-03-19T11:31:55.439 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.439 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.439 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.439 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:55.439 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.238188+0000", 2024-03-19T11:31:55.439 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4739999999999999e-06 2024-03-19T11:31:55.439 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.439 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.439 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:55.439 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.240837+0000", 2024-03-19T11:31:55.439 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026495189999999999 2024-03-19T11:31:55.439 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.439 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.440 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:55.440 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.240867+0000", 2024-03-19T11:31:55.440 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0363000000000001e-05 2024-03-19T11:31:55.440 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.440 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.440 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:55.440 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.240885+0000", 2024-03-19T11:31:55.440 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7153e-05 2024-03-19T11:31:55.440 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.440 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.440 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:55.440 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.241029+0000", 2024-03-19T11:31:55.440 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014455499999999999 2024-03-19T11:31:55.440 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.440 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.441 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:55.441 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.241069+0000", 2024-03-19T11:31:55.441 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9409000000000002e-05 2024-03-19T11:31:55.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.441 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.441 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:55.441 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.259833+0000", 2024-03-19T11:31:55.441 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018764306000000001 2024-03-19T11:31:55.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.441 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.441 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:55.441 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.259964+0000", 2024-03-19T11:31:55.441 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000131612 2024-03-19T11:31:55.441 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.442 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:55.442 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.442 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.442 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.442 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3282 4.5 4:a0a50d04:::smithi03829856-13:head [read 0~2201684,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e112)", 2024-03-19T11:31:55.442 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:32.240600+0000", 2024-03-19T11:31:55.442 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.132097786999999, 2024-03-19T11:31:55.442 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.84623566699999997, 2024-03-19T11:31:55.442 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:55.442 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:55.442 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:55.442 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:55.442 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:55.442 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3282 2024-03-19T11:31:55.442 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.443 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:55.443 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.443 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:55.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240600+0000", 2024-03-19T11:31:55.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.443 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.443 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.443 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:55.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240600+0000", 2024-03-19T11:31:55.443 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.443 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.443 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.443 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:55.443 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240602+0000", 2024-03-19T11:31:55.444 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0700000000000001e-06 2024-03-19T11:31:55.444 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.444 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.444 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:55.444 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240611+0000", 2024-03-19T11:31:55.444 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.969e-06 2024-03-19T11:31:55.444 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.444 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.444 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:55.444 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240630+0000", 2024-03-19T11:31:55.444 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8253e-05 2024-03-19T11:31:55.444 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.444 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.444 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:55.444 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240642+0000", 2024-03-19T11:31:55.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2666e-05 2024-03-19T11:31:55.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:55.445 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.241488+0000", 2024-03-19T11:31:55.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000845252 2024-03-19T11:31:55.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:31:55.445 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.241503+0000", 2024-03-19T11:31:55.445 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5797000000000001e-05 2024-03-19T11:31:55.445 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.445 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.445 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:55.446 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.083186+0000", 2024-03-19T11:31:55.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.84168299599999996 2024-03-19T11:31:55.446 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.446 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:55.446 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.083618+0000", 2024-03-19T11:31:55.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000432046 2024-03-19T11:31:55.446 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.446 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.446 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:55.446 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.086836+0000", 2024-03-19T11:31:55.446 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032176180000000002 2024-03-19T11:31:55.446 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.446 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:55.447 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.447 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.447 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.447 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3283 4.5 4:a0a50d04:::smithi03829856-13:head [sparse-read 0~1151947] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e112)", 2024-03-19T11:31:55.447 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:32.240669+0000", 2024-03-19T11:31:55.447 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.132028786999999, 2024-03-19T11:31:55.447 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.84791886900000002, 2024-03-19T11:31:55.447 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:55.447 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:55.447 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:55.447 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:55.447 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:55.447 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3283 2024-03-19T11:31:55.447 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.447 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:55.447 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.448 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:55.448 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240669+0000", 2024-03-19T11:31:55.448 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.448 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.448 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.448 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:55.448 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240669+0000", 2024-03-19T11:31:55.448 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.448 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.448 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.448 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:55.448 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240670+0000", 2024-03-19T11:31:55.448 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8199999999999998e-07 2024-03-19T11:31:55.448 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.448 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.449 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:55.449 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240675+0000", 2024-03-19T11:31:55.449 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0849999999999996e-06 2024-03-19T11:31:55.449 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.449 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.449 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:55.449 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240688+0000", 2024-03-19T11:31:55.449 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2901000000000001e-05 2024-03-19T11:31:55.449 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.449 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.449 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:55.449 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240696+0000", 2024-03-19T11:31:55.449 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2910000000000007e-06 2024-03-19T11:31:55.449 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.449 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.450 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:55.450 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.241573+0000", 2024-03-19T11:31:55.450 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00087639100000000002 2024-03-19T11:31:55.450 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.450 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.450 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:31:55.450 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.241587+0000", 2024-03-19T11:31:55.450 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4467999999999999e-05 2024-03-19T11:31:55.450 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.450 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.450 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:55.450 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.087901+0000", 2024-03-19T11:31:55.450 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.84631403500000002 2024-03-19T11:31:55.451 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.451 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.451 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:55.451 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.088089+0000", 2024-03-19T11:31:55.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000187171 2024-03-19T11:31:55.451 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.451 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.451 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:55.451 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.088588+0000", 2024-03-19T11:31:55.451 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00049964499999999995 2024-03-19T11:31:55.451 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.451 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:55.451 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.451 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.452 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3284 4.5 4:a0a50d04:::smithi03829856-13:head [sparse-read 0~1151947] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e112)", 2024-03-19T11:31:55.452 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:32.240710+0000", 2024-03-19T11:31:55.452 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 23.131988491000001, 2024-03-19T11:31:55.452 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.84860797899999996, 2024-03-19T11:31:55.452 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:55.452 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:55.452 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:55.452 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:55.452 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:55.452 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3284 2024-03-19T11:31:55.452 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.452 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:55.452 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.452 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:55.453 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240710+0000", 2024-03-19T11:31:55.453 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.453 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.453 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.453 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:55.453 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240710+0000", 2024-03-19T11:31:55.453 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.453 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.453 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.453 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:55.453 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240710+0000", 2024-03-19T11:31:55.453 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9599999999999999e-07 2024-03-19T11:31:55.453 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.453 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.453 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:55.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240714+0000", 2024-03-19T11:31:55.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8480000000000003e-06 2024-03-19T11:31:55.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:55.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240726+0000", 2024-03-19T11:31:55.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.186e-05 2024-03-19T11:31:55.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:55.454 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240736+0000", 2024-03-19T11:31:55.454 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0040999999999999e-05 2024-03-19T11:31:55.454 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.454 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.454 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:55.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.241633+0000", 2024-03-19T11:31:55.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00089736299999999998 2024-03-19T11:31:55.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:31:55.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.241645+0000", 2024-03-19T11:31:55.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1600000000000001e-05 2024-03-19T11:31:55.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.455 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:55.455 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.088667+0000", 2024-03-19T11:31:55.455 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.84702242800000005 2024-03-19T11:31:55.455 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.455 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.456 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:55.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.088837+0000", 2024-03-19T11:31:55.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000169949 2024-03-19T11:31:55.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.456 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:55.456 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.089318+0000", 2024-03-19T11:31:55.456 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00048019399999999998 2024-03-19T11:31:55.456 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.456 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:55.456 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.456 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.456 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.456 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3899 4.5 4:a38b9453:::smithi03829856-37:head [read 0~3966452,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e130)", 2024-03-19T11:31:55.456 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:51.081535+0000", 2024-03-19T11:31:55.457 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 4.2911632539999998, 2024-03-19T11:31:55.457 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017507140000000001, 2024-03-19T11:31:55.457 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:55.457 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:55.457 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:55.457 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:55.457 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:55.457 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3899 2024-03-19T11:31:55.457 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.457 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:55.457 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.457 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:55.457 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.081535+0000", 2024-03-19T11:31:55.457 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.457 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.458 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.458 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:55.458 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.081535+0000", 2024-03-19T11:31:55.458 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.458 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.458 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.458 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:55.458 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.081536+0000", 2024-03-19T11:31:55.458 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5999999999999999e-06 2024-03-19T11:31:55.458 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.458 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.458 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:55.458 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.081548+0000", 2024-03-19T11:31:55.458 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1892000000000001e-05 2024-03-19T11:31:55.458 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.459 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:55.459 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.081567+0000", 2024-03-19T11:31:55.459 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9023000000000001e-05 2024-03-19T11:31:55.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.459 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:55.459 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.081581+0000", 2024-03-19T11:31:55.459 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.364e-05 2024-03-19T11:31:55.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.459 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.459 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:55.459 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.081744+0000", 2024-03-19T11:31:55.459 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000163287 2024-03-19T11:31:55.459 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.460 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:31:55.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.081760+0000", 2024-03-19T11:31:55.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5892999999999998e-05 2024-03-19T11:31:55.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.460 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:55.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.095401+0000", 2024-03-19T11:31:55.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013640470999999999 2024-03-19T11:31:55.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.460 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:55.460 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.095542+0000", 2024-03-19T11:31:55.460 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000141213 2024-03-19T11:31:55.460 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.460 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.461 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:55.461 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.099042+0000", 2024-03-19T11:31:55.461 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035001210000000001 2024-03-19T11:31:55.461 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.461 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:55.461 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.461 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.461 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:55.461 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:55.463 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-19T11:31:55.463 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:55.469+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:55.550 INFO:tasks.rados.rados.0.smithi038.stdout:1730: done (7 left) 2024-03-19T11:31:55.550 INFO:tasks.rados.rados.0.smithi038.stdout:1731: done (6 left) 2024-03-19T11:31:55.550 INFO:tasks.rados.rados.0.smithi038.stdout:1732: done (5 left) 2024-03-19T11:31:55.550 INFO:tasks.rados.rados.0.smithi038.stdout:1733: done (4 left) 2024-03-19T11:31:55.550 INFO:tasks.rados.rados.0.smithi038.stdout:1734: done (3 left) 2024-03-19T11:31:55.550 INFO:tasks.rados.rados.0.smithi038.stdout:1735: done (2 left) 2024-03-19T11:31:55.550 INFO:tasks.rados.rados.0.smithi038.stdout:1736: done (1 left) 2024-03-19T11:31:55.551 INFO:tasks.rados.rados.0.smithi038.stdout:1737: done (0 left) 2024-03-19T11:31:55.551 INFO:tasks.rados.rados.0.smithi038.stdout:1738: snap_remove snap 36 2024-03-19T11:31:55.563 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:55.571+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:55.633 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:55.633 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:31:55.633 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:31:55.633 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:55.648 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-19T11:31:55.664 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:55.675+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:55.765 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:55.775+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:55.769 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:55.769 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:31:55.769 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:31:55.769 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:31:55.769 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:55.783 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-19T11:31:55.867 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:55.873+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:55.905 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:55.905 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:31:55.905 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:31:55.906 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:31:55.906 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.906 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:31:55.906 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:44.200920+0000", 2024-03-19T11:31:55.906 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 131.712513831, 2024-03-19T11:31:55.906 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.80881896499999995, 2024-03-19T11:31:55.906 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:55.906 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:55.906 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:55.906 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:31:55.906 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:31:55.906 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1 2024-03-19T11:31:55.906 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.907 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:55.907 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.907 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:55.907 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200920+0000", 2024-03-19T11:31:55.907 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.907 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.907 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.907 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:55.907 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200920+0000", 2024-03-19T11:31:55.907 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.907 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.907 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.907 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:55.907 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200923+0000", 2024-03-19T11:31:55.908 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6469999999999998e-06 2024-03-19T11:31:55.908 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.908 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.908 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:55.908 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200937+0000", 2024-03-19T11:31:55.908 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.399e-05 2024-03-19T11:31:55.908 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.908 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.908 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:55.908 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200960+0000", 2024-03-19T11:31:55.908 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2603e-05 2024-03-19T11:31:55.908 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.908 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.908 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:55.908 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200978+0000", 2024-03-19T11:31:55.909 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.791e-05 2024-03-19T11:31:55.909 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.909 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.909 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:55.909 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.201081+0000", 2024-03-19T11:31:55.909 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010344900000000001 2024-03-19T11:31:55.909 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.909 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.909 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:31:55.909 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.201096+0000", 2024-03-19T11:31:55.909 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4443e-05 2024-03-19T11:31:55.909 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.909 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.909 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:55.909 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.005978+0000", 2024-03-19T11:31:55.910 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.80488281500000003 2024-03-19T11:31:55.910 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.910 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.910 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:55.910 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.006267+0000", 2024-03-19T11:31:55.910 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000288688 2024-03-19T11:31:55.910 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.910 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.910 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:55.910 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.006534+0000", 2024-03-19T11:31:55.910 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026725799999999998 2024-03-19T11:31:55.911 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.911 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.911 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:55.911 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.008457+0000", 2024-03-19T11:31:55.911 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0019227820000000001 2024-03-19T11:31:55.911 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.911 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.911 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:55.911 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.009570+0000", 2024-03-19T11:31:55.911 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011131750000000001 2024-03-19T11:31:55.911 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.911 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.911 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:55.911 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.009670+0000", 2024-03-19T11:31:55.912 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9458999999999998e-05 2024-03-19T11:31:55.912 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.912 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.912 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:55.912 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.009739+0000", 2024-03-19T11:31:55.912 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9746000000000006e-05 2024-03-19T11:31:55.912 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.912 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:55.912 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.912 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.912 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.912 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:31:55.913 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.339898+0000", 2024-03-19T11:31:55.913 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 130.57353672400001, 2024-03-19T11:31:55.913 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019501793, 2024-03-19T11:31:55.913 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:55.914 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:55.914 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:55.914 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:31:55.914 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:31:55.914 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113 2024-03-19T11:31:55.914 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.914 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:55.914 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.914 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:55.914 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339898+0000", 2024-03-19T11:31:55.914 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.915 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.915 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.915 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:55.915 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339898+0000", 2024-03-19T11:31:55.915 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.915 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.915 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.915 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:55.915 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339899+0000", 2024-03-19T11:31:55.915 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.353e-06 2024-03-19T11:31:55.915 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.915 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.915 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:55.916 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339907+0000", 2024-03-19T11:31:55.916 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3000000000000002e-06 2024-03-19T11:31:55.916 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.916 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.916 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:55.916 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339924+0000", 2024-03-19T11:31:55.916 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6906999999999999e-05 2024-03-19T11:31:55.916 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.916 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.916 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:55.916 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339933+0000", 2024-03-19T11:31:55.916 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8969999999999997e-06 2024-03-19T11:31:55.916 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.916 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.917 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:55.917 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352239+0000", 2024-03-19T11:31:55.917 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012305814999999999 2024-03-19T11:31:55.917 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.917 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.917 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:55.917 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352466+0000", 2024-03-19T11:31:55.917 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022684100000000001 2024-03-19T11:31:55.917 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.917 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.917 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:55.917 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352816+0000", 2024-03-19T11:31:55.917 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00035030499999999997 2024-03-19T11:31:55.917 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.918 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.918 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:55.918 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.358956+0000", 2024-03-19T11:31:55.918 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061400079999999998 2024-03-19T11:31:55.918 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.918 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.918 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:55.918 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359118+0000", 2024-03-19T11:31:55.918 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016237599999999999 2024-03-19T11:31:55.918 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.918 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.918 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:55.918 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359287+0000", 2024-03-19T11:31:55.919 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000168488 2024-03-19T11:31:55.919 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.919 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.919 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:55.919 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359399+0000", 2024-03-19T11:31:55.919 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000112503 2024-03-19T11:31:55.919 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.919 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:55.919 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.919 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.919 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.919 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:31:55.919 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.339947+0000", 2024-03-19T11:31:55.919 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 130.573486957, 2024-03-19T11:31:55.919 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021923069999999999, 2024-03-19T11:31:55.920 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:55.920 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:55.920 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:55.920 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:31:55.920 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:31:55.920 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 114 2024-03-19T11:31:55.920 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.920 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:55.920 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.920 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:55.920 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339947+0000", 2024-03-19T11:31:55.920 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.920 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.920 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.921 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:55.921 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339947+0000", 2024-03-19T11:31:55.921 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.921 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.921 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.921 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:55.921 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339948+0000", 2024-03-19T11:31:55.921 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6000000000000003e-07 2024-03-19T11:31:55.921 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.921 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.921 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:55.921 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339956+0000", 2024-03-19T11:31:55.921 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1559999999999995e-06 2024-03-19T11:31:55.921 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.922 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:55.922 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339967+0000", 2024-03-19T11:31:55.922 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0669999999999999e-05 2024-03-19T11:31:55.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.922 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:55.922 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339975+0000", 2024-03-19T11:31:55.922 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0099999999999995e-06 2024-03-19T11:31:55.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.922 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:55.922 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.353672+0000", 2024-03-19T11:31:55.922 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013696583999999999 2024-03-19T11:31:55.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.923 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:55.923 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.353808+0000", 2024-03-19T11:31:55.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013652299999999999 2024-03-19T11:31:55.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.923 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:55.923 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.354036+0000", 2024-03-19T11:31:55.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022744799999999999 2024-03-19T11:31:55.924 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.924 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:55.924 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.360480+0000", 2024-03-19T11:31:55.924 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0064441630000000001 2024-03-19T11:31:55.924 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.924 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:55.924 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361722+0000", 2024-03-19T11:31:55.924 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001242754 2024-03-19T11:31:55.924 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.925 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:55.925 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361807+0000", 2024-03-19T11:31:55.925 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4605999999999996e-05 2024-03-19T11:31:55.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.925 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:55.925 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361870+0000", 2024-03-19T11:31:55.925 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3396000000000001e-05 2024-03-19T11:31:55.925 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.925 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:55.925 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.925 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:31:55.926 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.340017+0000", 2024-03-19T11:31:55.926 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 130.57341764099999, 2024-03-19T11:31:55.926 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022169355000000002, 2024-03-19T11:31:55.926 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:55.926 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:55.926 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:55.926 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:31:55.926 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:31:55.926 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 115 2024-03-19T11:31:55.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.927 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:55.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.927 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:55.927 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340017+0000", 2024-03-19T11:31:55.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.927 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:55.927 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340017+0000", 2024-03-19T11:31:55.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.927 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:55.928 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340018+0000", 2024-03-19T11:31:55.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.043e-06 2024-03-19T11:31:55.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.928 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:55.928 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340139+0000", 2024-03-19T11:31:55.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121502 2024-03-19T11:31:55.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.928 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:55.928 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340157+0000", 2024-03-19T11:31:55.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7382999999999999e-05 2024-03-19T11:31:55.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:55.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340166+0000", 2024-03-19T11:31:55.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2469999999999997e-06 2024-03-19T11:31:55.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:55.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.357470+0000", 2024-03-19T11:31:55.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017304119999999999 2024-03-19T11:31:55.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:55.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.357671+0000", 2024-03-19T11:31:55.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000200798 2024-03-19T11:31:55.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:55.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.358002+0000", 2024-03-19T11:31:55.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033090099999999998 2024-03-19T11:31:55.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:55.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361892+0000", 2024-03-19T11:31:55.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00389018 2024-03-19T11:31:55.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:55.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361983+0000", 2024-03-19T11:31:55.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0976000000000004e-05 2024-03-19T11:31:55.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.931 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:55.931 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362071+0000", 2024-03-19T11:31:55.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8022999999999994e-05 2024-03-19T11:31:55.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.931 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:55.931 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362186+0000", 2024-03-19T11:31:55.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115182 2024-03-19T11:31:55.931 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.931 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:55.932 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.932 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.932 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.932 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:31:55.932 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.340183+0000", 2024-03-19T11:31:55.932 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 130.573250809, 2024-03-19T11:31:55.932 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022279841000000002, 2024-03-19T11:31:55.932 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:55.932 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:55.932 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:55.932 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:31:55.932 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:31:55.932 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 116 2024-03-19T11:31:55.932 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.932 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:55.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.933 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:55.933 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340183+0000", 2024-03-19T11:31:55.933 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.933 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.933 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:55.933 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340183+0000", 2024-03-19T11:31:55.933 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.933 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.933 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:55.933 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340184+0000", 2024-03-19T11:31:55.933 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9000000000000005e-07 2024-03-19T11:31:55.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.934 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:55.934 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340191+0000", 2024-03-19T11:31:55.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0360000000000003e-06 2024-03-19T11:31:55.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.934 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:55.934 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340201+0000", 2024-03-19T11:31:55.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0844e-05 2024-03-19T11:31:55.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:55.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340209+0000", 2024-03-19T11:31:55.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7360000000000001e-06 2024-03-19T11:31:55.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:55.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359535+0000", 2024-03-19T11:31:55.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019325546999999998 2024-03-19T11:31:55.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.936 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.936 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:55.936 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359672+0000", 2024-03-19T11:31:55.936 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013706600000000001 2024-03-19T11:31:55.936 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.936 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.936 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:55.936 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359905+0000", 2024-03-19T11:31:55.936 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023334500000000001 2024-03-19T11:31:55.936 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.937 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.937 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:55.937 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362306+0000", 2024-03-19T11:31:55.937 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002401041 2024-03-19T11:31:55.937 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.937 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.937 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:55.937 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362329+0000", 2024-03-19T11:31:55.937 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3283999999999999e-05 2024-03-19T11:31:55.937 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.937 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.937 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:55.937 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362402+0000", 2024-03-19T11:31:55.937 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2459000000000005e-05 2024-03-19T11:31:55.938 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.938 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.938 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:55.938 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362463+0000", 2024-03-19T11:31:55.938 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1493000000000002e-05 2024-03-19T11:31:55.938 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.938 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:55.938 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.938 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.938 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.938 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:31:55.938 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.340222+0000", 2024-03-19T11:31:55.939 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 130.57321222600001, 2024-03-19T11:31:55.939 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022500218999999998, 2024-03-19T11:31:55.939 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:55.939 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:55.939 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:55.939 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:31:55.939 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:31:55.939 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 117 2024-03-19T11:31:55.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.939 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:55.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.939 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:55.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340222+0000", 2024-03-19T11:31:55.939 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.940 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.940 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:55.940 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340222+0000", 2024-03-19T11:31:55.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.940 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.940 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.940 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:55.940 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340223+0000", 2024-03-19T11:31:55.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7999999999999995e-07 2024-03-19T11:31:55.940 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.940 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.940 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:55.940 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340229+0000", 2024-03-19T11:31:55.941 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0399999999999998e-06 2024-03-19T11:31:55.941 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.941 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.941 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:55.941 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340239+0000", 2024-03-19T11:31:55.941 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0190000000000001e-05 2024-03-19T11:31:55.941 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.941 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.941 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:55.941 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340250+0000", 2024-03-19T11:31:55.941 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0720000000000001e-05 2024-03-19T11:31:55.941 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.941 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.941 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:55.941 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.360587+0000", 2024-03-19T11:31:55.942 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020337542 2024-03-19T11:31:55.942 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.942 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:55.942 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.360818+0000", 2024-03-19T11:31:55.942 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023120500000000001 2024-03-19T11:31:55.942 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.942 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:31:55.942 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361218+0000", 2024-03-19T11:31:55.942 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000399637 2024-03-19T11:31:55.942 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:55.943 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362483+0000", 2024-03-19T11:31:55.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012649200000000001 2024-03-19T11:31:55.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:55.943 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362538+0000", 2024-03-19T11:31:55.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4490000000000002e-05 2024-03-19T11:31:55.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:55.943 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362617+0000", 2024-03-19T11:31:55.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9839000000000006e-05 2024-03-19T11:31:55.944 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.944 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:55.944 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362722+0000", 2024-03-19T11:31:55.944 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104956 2024-03-19T11:31:55.944 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.944 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:55.944 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.945 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.945 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.15 51/50 [PushOp(4:afa9f208:::smithi03829856-35:head, version: 28'46, data_included: [571438~623878,1806623~424698], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:afa9f208:::smithi03829856-35:head@28'46, size: 2571438, copy_subset: [(0, 2571438)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2231321, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:af9d0d56:::smithi03829856-20:head, version: 35'55, data_included: [655360~589824,1900544~418789], data_size: 1008613, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:af9d0d56:::smithi03829856-20:head@35'55, size: 2319333, copy_subset: [(0, 2319333)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2319333, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ad3ad1b3:::smithi03829856-21:b, version: 47'66, data_included: [736140~537686,2020581~315559], data_size: 853245, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:ad3ad1b3:::smithi03829856-21:b@47'66, size: 2336140, copy_subset: [(0, 2336140)], clone_subset: {}, snapset: b=[]:{b: [b, a, 6, 5]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2336140, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ad3ad1b3:::smithi03829856-21:head, version: 47'70, data_included: [531121~742705,1711057~305871], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:ad3ad1b3:::smithi03829856-21:head@47'70, size: 3731121, copy_subset: [(0, 3731121)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2016928, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ac9cd3a3:::smithi03829856-8:head, version: 49'72, data_included: [517649~620182,1776668~428394], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:ac9cd3a3:::smithi03829856-8:head@49'72, size: 3008420, copy_subset: [(0, 3008420)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2205062, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:aa3303a5:::smithi03829856-17:b, version: 49'73, data_included: [614400~692224,1903006~356352], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:aa3303a5:::smithi03829856-17:b@49'73, size: 2268912, copy_subset: [(0, 2268912)], clone_subset: {}, snapset: b=[]:{b: [b, a, 6, 5]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2259358, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:aa3303a5:::smithi03829856-17:head, version: 49'74, data_included: [413472~893152,1818036~155424], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:aa3303a5:::smithi03829856-17:head@49'74, size: 3613472, copy_subset: [(0, 3613472)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1973460, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:55.945 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:28.058610+0000", 2024-03-19T11:31:55.945 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 87.854824054000005, 2024-03-19T11:31:55.945 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027100032, 2024-03-19T11:31:55.945 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:55.945 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:55.945 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:55.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.945 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:55.945 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.058610+0000", 2024-03-19T11:31:55.945 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.945 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.946 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:55.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.058610+0000", 2024-03-19T11:31:55.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.946 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:55.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.058611+0000", 2024-03-19T11:31:55.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1599999999999999e-06 2024-03-19T11:31:55.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.946 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:55.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.063874+0000", 2024-03-19T11:31:55.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052625270000000004 2024-03-19T11:31:55.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:55.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.063935+0000", 2024-03-19T11:31:55.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0594000000000003e-05 2024-03-19T11:31:55.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:55.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064024+0000", 2024-03-19T11:31:55.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9273999999999999e-05 2024-03-19T11:31:55.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:55.948 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064385+0000", 2024-03-19T11:31:55.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00036129100000000002 2024-03-19T11:31:55.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.948 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:55.948 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064437+0000", 2024-03-19T11:31:55.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2346000000000003e-05 2024-03-19T11:31:55.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.948 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:55.948 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085710+0000", 2024-03-19T11:31:55.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021272840000000001 2024-03-19T11:31:55.948 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.949 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:55.949 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.949 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.949 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.15 e51,50 4:ac9cd3a3:::smithi03829856-8:4,49'71)", 2024-03-19T11:31:55.949 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:28.064112+0000", 2024-03-19T11:31:55.949 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 87.849322228000005, 2024-03-19T11:31:55.949 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026480417999999999, 2024-03-19T11:31:55.949 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:55.949 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:55.949 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:55.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.949 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:55.949 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064112+0000", 2024-03-19T11:31:55.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:55.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064112+0000", 2024-03-19T11:31:55.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:55.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064113+0000", 2024-03-19T11:31:55.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3600000000000003e-07 2024-03-19T11:31:55.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:55.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064118+0000", 2024-03-19T11:31:55.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0549999999999999e-06 2024-03-19T11:31:55.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:55.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064128+0000", 2024-03-19T11:31:55.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0349e-05 2024-03-19T11:31:55.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:55.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064136+0000", 2024-03-19T11:31:55.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6249999999999998e-06 2024-03-19T11:31:55.951 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.951 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.951 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:55.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085773+0000", 2024-03-19T11:31:55.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021636829999999999 2024-03-19T11:31:55.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.952 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:55.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085788+0000", 2024-03-19T11:31:55.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5512999999999999e-05 2024-03-19T11:31:55.952 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.952 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.952 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:55.952 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.090592+0000", 2024-03-19T11:31:55.952 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0048044100000000003 2024-03-19T11:31:55.952 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.952 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:55.953 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.953 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.953 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.953 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.5 51/50 [PushOp(4:a38b9453:::smithi03829856-37:head, version: 28'45, data_included: [605052~774376,1923587~274200], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:a38b9453:::smithi03829856-37:head@28'45, size: 2605052, copy_subset: [(0, 2605052)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2197787, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:a0a50d04:::smithi03829856-13:b, version: 47'63, data_included: [601684~619341,1669078~429235], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a0a50d04:::smithi03829856-13:b@47'63, size: 2201684, copy_subset: [(0, 2201684)], clone_subset: {}, snapset: b=[]:{b: [b, a, 6, 5]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2098313, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:55.953 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:28.064324+0000", 2024-03-19T11:31:55.953 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 87.849110383999999, 2024-03-19T11:31:55.953 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027179020000000002, 2024-03-19T11:31:55.953 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:55.953 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:55.953 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:55.953 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.953 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:55.953 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064324+0000", 2024-03-19T11:31:55.953 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.954 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.954 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:55.954 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064324+0000", 2024-03-19T11:31:55.954 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.954 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.954 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:55.954 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064324+0000", 2024-03-19T11:31:55.954 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1199999999999998e-07 2024-03-19T11:31:55.954 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.954 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.954 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:55.954 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.065527+0000", 2024-03-19T11:31:55.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012028939999999999 2024-03-19T11:31:55.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:55.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.065552+0000", 2024-03-19T11:31:55.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4440999999999999e-05 2024-03-19T11:31:55.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:55.955 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.065568+0000", 2024-03-19T11:31:55.955 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6498e-05 2024-03-19T11:31:55.955 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.955 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.955 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:55.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085812+0000", 2024-03-19T11:31:55.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020243954000000002 2024-03-19T11:31:55.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.956 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:55.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085825+0000", 2024-03-19T11:31:55.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3298999999999999e-05 2024-03-19T11:31:55.956 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.956 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.956 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:55.956 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.091503+0000", 2024-03-19T11:31:55.956 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.005677522 2024-03-19T11:31:55.956 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.956 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:55.957 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.957 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.957 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.957 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.24 75/74 [PushOp(4:27909373:::smithi03829856-46:8, version: 46'59, data_included: [756315~767970,1933572~280606], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:8@46'59, size: 2756315, copy_subset: [(0, 2756315)], clone_subset: {}, snapset: 10=[]:{8: [6], c: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2214178, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:266590a5:::smithi03829856-16:e, version: 70'107, data_included: [652105~662359,1728171~386217], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:e@70'107, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 10=[]:{e: [d, b, 6]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:266590a5:::smithi03829856-16:head, version: 70'108, data_included: [652105~662359,1728171~386217], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:head@70'108, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:27909373:::smithi03829856-46:c, version: 70'109, data_included: [628595~895690,1807440~152886], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:c@70'109, size: 3428595, copy_subset: [(0, 3428595)], clone_subset: {}, snapset: 10=[]:{8: [6], c: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1960326, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:27909373:::smithi03829856-46:head, version: 70'110, data_included: [403401~454961,1584925~506039], data_size: 961000, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:head@70'110, size: 2403401, copy_subset: [(0, 2403401)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2403401, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:55.957 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.099172+0000", 2024-03-19T11:31:55.957 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 61.814261950000002, 2024-03-19T11:31:55.957 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.024528580000000001, 2024-03-19T11:31:55.957 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:55.957 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:55.957 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:55.957 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.957 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:55.957 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.099172+0000", 2024-03-19T11:31:55.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.958 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.958 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.958 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:55.958 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.099172+0000", 2024-03-19T11:31:55.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.958 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.958 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.958 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:55.958 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.099175+0000", 2024-03-19T11:31:55.958 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7750000000000001e-06 2024-03-19T11:31:55.958 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.958 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.958 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:55.959 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105219+0000", 2024-03-19T11:31:55.959 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0060440789999999999 2024-03-19T11:31:55.959 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.959 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.959 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:55.959 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105285+0000", 2024-03-19T11:31:55.959 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6260000000000006e-05 2024-03-19T11:31:55.959 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.959 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.959 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:55.959 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105366+0000", 2024-03-19T11:31:55.959 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.038e-05 2024-03-19T11:31:55.959 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.959 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.960 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:55.960 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105637+0000", 2024-03-19T11:31:55.960 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027085199999999997 2024-03-19T11:31:55.960 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.960 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.960 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:55.960 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105679+0000", 2024-03-19T11:31:55.960 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2088000000000001e-05 2024-03-19T11:31:55.960 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.960 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.960 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:55.960 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.123701+0000", 2024-03-19T11:31:55.960 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018022145999999999 2024-03-19T11:31:55.960 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.961 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:55.961 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.961 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.961 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.961 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.24 e75,74 4:266590a5:::smithi03829856-16:4,47'63,4:27909373:::smithi03829856-46:5,55'79,4:266590a5:::smithi03829856-16:10,67'105)", 2024-03-19T11:31:55.961 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.105437+0000", 2024-03-19T11:31:55.961 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 61.807996946999999, 2024-03-19T11:31:55.961 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.01860726, 2024-03-19T11:31:55.961 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:55.961 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:55.961 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:55.961 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.961 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:55.961 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105437+0000", 2024-03-19T11:31:55.961 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.962 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.962 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.962 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:55.962 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105437+0000", 2024-03-19T11:31:55.962 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.962 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.962 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.962 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:55.962 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105438+0000", 2024-03-19T11:31:55.962 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6799999999999999e-07 2024-03-19T11:31:55.962 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.962 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.962 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:55.963 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105445+0000", 2024-03-19T11:31:55.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3749999999999999e-06 2024-03-19T11:31:55.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.963 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:55.963 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105454+0000", 2024-03-19T11:31:55.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7909999999999999e-06 2024-03-19T11:31:55.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.963 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:55.963 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105465+0000", 2024-03-19T11:31:55.963 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.045e-05 2024-03-19T11:31:55.963 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.963 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.963 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:55.964 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.123753+0000", 2024-03-19T11:31:55.964 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0182884 2024-03-19T11:31:55.964 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.964 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.964 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:55.964 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.123765+0000", 2024-03-19T11:31:55.964 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2143e-05 2024-03-19T11:31:55.964 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.964 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.964 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:55.964 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.124045+0000", 2024-03-19T11:31:55.964 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027923299999999999 2024-03-19T11:31:55.964 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.964 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:55.965 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.965 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.965 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.965 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:3103 4.15 e104/76 4:afa9f208:::smithi03829856-35:head v 104'221, mlcod=101'216)", 2024-03-19T11:31:55.965 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.563593+0000", 2024-03-19T11:31:55.965 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.349841669, 2024-03-19T11:31:55.965 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026983566, 2024-03-19T11:31:55.965 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:55.965 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:55.965 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:55.965 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.965 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:55.965 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.563593+0000", 2024-03-19T11:31:55.965 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.966 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.966 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:55.966 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.563593+0000", 2024-03-19T11:31:55.966 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.966 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.966 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:55.966 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.563595+0000", 2024-03-19T11:31:55.966 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2859999999999998e-06 2024-03-19T11:31:55.966 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.966 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.966 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:55.966 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.564339+0000", 2024-03-19T11:31:55.966 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00074400099999999999 2024-03-19T11:31:55.967 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.967 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.967 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:55.967 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.564376+0000", 2024-03-19T11:31:55.967 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6628999999999998e-05 2024-03-19T11:31:55.967 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.967 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.967 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:55.967 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.564396+0000", 2024-03-19T11:31:55.967 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0596000000000001e-05 2024-03-19T11:31:55.967 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.967 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.967 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:55.968 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.566600+0000", 2024-03-19T11:31:55.968 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002204021 2024-03-19T11:31:55.968 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.968 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.968 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:55.968 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.566642+0000", 2024-03-19T11:31:55.968 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1902e-05 2024-03-19T11:31:55.968 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.968 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.968 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:55.968 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.590494+0000", 2024-03-19T11:31:55.968 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023851816000000001 2024-03-19T11:31:55.968 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.968 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.968 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:55.969 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.590576+0000", 2024-03-19T11:31:55.969 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2315e-05 2024-03-19T11:31:55.969 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.969 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:55.969 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.969 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.969 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.969 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:3104 4.15 e104/76 4:afa9f208:::smithi03829856-35:head v 104'222, mlcod=101'216)", 2024-03-19T11:31:55.969 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.568280+0000", 2024-03-19T11:31:55.969 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.345153826000001, 2024-03-19T11:31:55.969 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022345304999999999, 2024-03-19T11:31:55.969 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:55.969 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:55.969 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:55.970 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.970 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:55.970 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.568280+0000", 2024-03-19T11:31:55.970 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.970 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.970 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.970 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:55.970 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.568280+0000", 2024-03-19T11:31:55.970 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.970 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.970 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.970 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:55.970 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.568282+0000", 2024-03-19T11:31:55.970 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0090000000000002e-06 2024-03-19T11:31:55.970 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.971 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.971 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:55.971 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.568299+0000", 2024-03-19T11:31:55.971 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6237000000000001e-05 2024-03-19T11:31:55.971 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.971 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.971 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:55.971 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.568331+0000", 2024-03-19T11:31:55.971 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2601999999999997e-05 2024-03-19T11:31:55.971 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.971 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.971 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:55.971 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.568350+0000", 2024-03-19T11:31:55.971 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8280000000000001e-05 2024-03-19T11:31:55.972 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.972 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.972 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:55.972 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.571311+0000", 2024-03-19T11:31:55.972 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029617770000000001 2024-03-19T11:31:55.972 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.972 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.972 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:55.972 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.571379+0000", 2024-03-19T11:31:55.972 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7331999999999998e-05 2024-03-19T11:31:55.972 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.972 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.972 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:55.972 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.590583+0000", 2024-03-19T11:31:55.973 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019204649000000001 2024-03-19T11:31:55.973 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.973 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.973 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:55.973 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.590626+0000", 2024-03-19T11:31:55.973 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2419000000000002e-05 2024-03-19T11:31:55.973 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.973 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:55.973 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.973 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.973 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.973 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:3106 4.15 e104/76 4:ad3ad1b3:::smithi03829856-21:head v 104'224, mlcod=101'216)", 2024-03-19T11:31:55.973 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.571056+0000", 2024-03-19T11:31:55.973 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.342378367999999, 2024-03-19T11:31:55.973 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019636355000000001, 2024-03-19T11:31:55.974 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:55.974 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:55.974 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:55.974 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.974 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:55.974 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.571056+0000", 2024-03-19T11:31:55.974 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.974 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.974 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.974 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:55.974 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.571056+0000", 2024-03-19T11:31:55.974 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.974 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.974 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.975 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:55.975 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.571058+0000", 2024-03-19T11:31:55.975 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2400000000000002e-06 2024-03-19T11:31:55.975 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.975 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.975 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:55.975 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.571586+0000", 2024-03-19T11:31:55.975 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00052782299999999996 2024-03-19T11:31:55.975 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.975 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.975 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:55.975 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.571618+0000", 2024-03-19T11:31:55.975 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2453000000000003e-05 2024-03-19T11:31:55.975 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.976 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.976 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:55.976 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.571638+0000", 2024-03-19T11:31:55.976 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9176000000000001e-05 2024-03-19T11:31:55.976 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.976 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.976 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:55.976 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.573958+0000", 2024-03-19T11:31:55.976 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0023201699999999999 2024-03-19T11:31:55.976 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.976 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.976 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:55.976 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.574023+0000", 2024-03-19T11:31:55.976 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5425000000000005e-05 2024-03-19T11:31:55.977 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.977 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.977 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:55.977 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.590646+0000", 2024-03-19T11:31:55.977 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016623189 2024-03-19T11:31:55.977 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.977 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.977 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:55.977 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.590692+0000", 2024-03-19T11:31:55.977 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5878999999999999e-05 2024-03-19T11:31:55.977 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.977 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:55.977 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.977 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.978 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.978 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.3.0:309 4.0 e108/60 4:01018511:::smithi03829856-19:b v 108'91, mlcod=100'87)", 2024-03-19T11:31:55.978 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.205898+0000", 2024-03-19T11:31:55.978 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.707536572999999, 2024-03-19T11:31:55.978 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.034464328000000002, 2024-03-19T11:31:55.978 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:55.978 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:55.978 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:55.978 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.978 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:55.978 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.205898+0000", 2024-03-19T11:31:55.978 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.979 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.979 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.979 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:55.979 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.205898+0000", 2024-03-19T11:31:55.979 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.979 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.979 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.979 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:55.979 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.205899+0000", 2024-03-19T11:31:55.979 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.16e-07 2024-03-19T11:31:55.979 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.979 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.979 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:55.979 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.205916+0000", 2024-03-19T11:31:55.979 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7229000000000001e-05 2024-03-19T11:31:55.980 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.980 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.980 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:55.980 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.205946+0000", 2024-03-19T11:31:55.980 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0233999999999999e-05 2024-03-19T11:31:55.980 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.980 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.980 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:55.980 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.205966+0000", 2024-03-19T11:31:55.980 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9434999999999999e-05 2024-03-19T11:31:55.980 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.980 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.980 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:55.980 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.206162+0000", 2024-03-19T11:31:55.980 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000196918 2024-03-19T11:31:55.981 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.981 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.981 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:55.981 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.206227+0000", 2024-03-19T11:31:55.981 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4381000000000006e-05 2024-03-19T11:31:55.981 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.981 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.981 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:55.981 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.240212+0000", 2024-03-19T11:31:55.981 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033984852000000003 2024-03-19T11:31:55.981 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.981 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.981 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:55.981 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.240362+0000", 2024-03-19T11:31:55.982 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015036300000000001 2024-03-19T11:31:55.982 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.982 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:55.982 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.982 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.982 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.982 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.5.0:238 4.15 e108/76 4:ac9cd3a3:::smithi03829856-8:10 v 108'231, mlcod=104'228)", 2024-03-19T11:31:55.982 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.215160+0000", 2024-03-19T11:31:55.982 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.698274165000001, 2024-03-19T11:31:55.982 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.024600749000000002, 2024-03-19T11:31:55.982 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:55.982 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:55.982 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:55.982 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.983 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:55.983 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215160+0000", 2024-03-19T11:31:55.983 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.983 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.983 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.983 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:55.983 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215160+0000", 2024-03-19T11:31:55.983 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.983 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.983 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.983 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:55.983 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215161+0000", 2024-03-19T11:31:55.983 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6000000000000002e-07 2024-03-19T11:31:55.983 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.984 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.984 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:55.984 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215172+0000", 2024-03-19T11:31:55.984 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1042e-05 2024-03-19T11:31:55.984 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.984 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.984 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:55.984 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215213+0000", 2024-03-19T11:31:55.984 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1167000000000001e-05 2024-03-19T11:31:55.984 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.984 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.984 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:55.985 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215234+0000", 2024-03-19T11:31:55.985 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0534e-05 2024-03-19T11:31:55.985 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.985 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.985 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:55.985 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215668+0000", 2024-03-19T11:31:55.985 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000434074 2024-03-19T11:31:55.985 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.985 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.985 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:55.985 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215749+0000", 2024-03-19T11:31:55.985 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0702000000000001e-05 2024-03-19T11:31:55.985 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.986 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.986 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:55.986 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.239629+0000", 2024-03-19T11:31:55.986 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023880855999999999 2024-03-19T11:31:55.986 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.986 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.986 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:55.986 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.239761+0000", 2024-03-19T11:31:55.986 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000131514 2024-03-19T11:31:55.986 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.986 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:55.986 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.987 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.987 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.987 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.5.0:239 4.8 e108/100 4:14611006:::smithi03829856-25:c v 108'125, mlcod=103'122)", 2024-03-19T11:31:55.987 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.215292+0000", 2024-03-19T11:31:55.987 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.698141980999999, 2024-03-19T11:31:55.987 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.025163089999999999, 2024-03-19T11:31:55.987 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:55.987 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:55.987 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:55.987 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.987 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:55.987 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215292+0000", 2024-03-19T11:31:55.988 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.988 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.988 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.988 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:55.988 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215292+0000", 2024-03-19T11:31:55.988 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.988 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.988 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.988 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:55.988 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215293+0000", 2024-03-19T11:31:55.988 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.47e-07 2024-03-19T11:31:55.988 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.988 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.989 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:55.989 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215315+0000", 2024-03-19T11:31:55.989 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1691e-05 2024-03-19T11:31:55.989 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.989 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.989 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:55.989 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215348+0000", 2024-03-19T11:31:55.989 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2676999999999999e-05 2024-03-19T11:31:55.989 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.989 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.989 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:55.989 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215362+0000", 2024-03-19T11:31:55.989 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3978999999999999e-05 2024-03-19T11:31:55.990 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.990 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.990 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:55.990 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215727+0000", 2024-03-19T11:31:55.990 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000365583 2024-03-19T11:31:55.990 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.990 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.990 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:55.990 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215804+0000", 2024-03-19T11:31:55.990 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7392999999999996e-05 2024-03-19T11:31:55.990 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.990 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.991 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:55.991 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.240368+0000", 2024-03-19T11:31:55.991 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.024563807999999999 2024-03-19T11:31:55.991 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.991 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.991 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:55.991 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.240455+0000", 2024-03-19T11:31:55.991 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7112000000000002e-05 2024-03-19T11:31:55.991 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.991 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:55.991 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.991 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.991 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.992 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.5.0:240 4.15 e108/76 4:aa3303a5:::smithi03829856-17:b v 108'233, mlcod=104'228)", 2024-03-19T11:31:55.992 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.215400+0000", 2024-03-19T11:31:55.992 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.698034250999999, 2024-03-19T11:31:55.992 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026645319000000001, 2024-03-19T11:31:55.992 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:55.992 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:55.992 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:55.992 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.992 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:55.992 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215400+0000", 2024-03-19T11:31:55.992 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.992 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.992 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.992 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:55.993 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215400+0000", 2024-03-19T11:31:55.993 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.993 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.993 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.993 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:55.993 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215401+0000", 2024-03-19T11:31:55.993 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3e-07 2024-03-19T11:31:55.993 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.993 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.993 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:55.993 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215417+0000", 2024-03-19T11:31:55.993 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5821999999999999e-05 2024-03-19T11:31:55.993 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.994 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.994 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:55.994 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215443+0000", 2024-03-19T11:31:55.994 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6971e-05 2024-03-19T11:31:55.994 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.994 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.994 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:55.994 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215457+0000", 2024-03-19T11:31:55.994 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3139000000000001e-05 2024-03-19T11:31:55.994 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.994 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.994 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:55.994 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.234414+0000", 2024-03-19T11:31:55.994 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018957108 2024-03-19T11:31:55.995 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.995 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.995 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:55.995 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.234470+0000", 2024-03-19T11:31:55.995 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6274999999999999e-05 2024-03-19T11:31:55.995 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.995 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.995 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:55.995 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.241943+0000", 2024-03-19T11:31:55.995 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0074728770000000002 2024-03-19T11:31:55.995 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.995 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.995 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:55.996 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.242045+0000", 2024-03-19T11:31:55.996 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000102497 2024-03-19T11:31:55.996 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.996 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:55.996 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:55.996 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.996 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.996 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.5.0:241 4.8 e108/100 4:15708482:::smithi03829856-23:10 v 108'127, mlcod=103'122)", 2024-03-19T11:31:55.996 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.217180+0000", 2024-03-19T11:31:55.996 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.696253982999998, 2024-03-19T11:31:55.996 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.025026023000000001, 2024-03-19T11:31:55.996 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:55.996 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:55.996 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:55.997 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.997 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:55.997 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.217180+0000", 2024-03-19T11:31:55.997 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.997 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.997 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.997 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:55.997 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.217180+0000", 2024-03-19T11:31:55.997 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:55.997 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.997 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.997 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:55.998 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.217182+0000", 2024-03-19T11:31:55.998 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.12e-06 2024-03-19T11:31:55.998 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.998 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.998 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:55.998 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.217200+0000", 2024-03-19T11:31:55.998 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7677999999999999e-05 2024-03-19T11:31:55.998 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.998 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.998 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:55.998 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.217246+0000", 2024-03-19T11:31:55.998 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6053000000000001e-05 2024-03-19T11:31:55.998 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.999 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.999 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:55.999 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.217266+0000", 2024-03-19T11:31:55.999 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0085000000000002e-05 2024-03-19T11:31:55.999 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.999 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.999 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:55.999 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.226876+0000", 2024-03-19T11:31:55.999 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0096094389999999991 2024-03-19T11:31:55.999 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:55.999 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:55.999 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:55.999 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.226932+0000", 2024-03-19T11:31:55.999 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6234999999999999e-05 2024-03-19T11:31:56.000 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:56.000 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:56.000 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:56.000 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.242063+0000", 2024-03-19T11:31:56.000 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015131083 2024-03-19T11:31:56.000 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:56.000 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:56.000 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:56.000 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.242206+0000", 2024-03-19T11:31:56.000 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014333 2024-03-19T11:31:56.000 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:56.000 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:56.001 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:56.001 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:56.001 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:56.001 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.1.0:279 4.24 e108/83 4:27909373:::smithi03829856-46:c v 108'129, mlcod=100'124)", 2024-03-19T11:31:56.001 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.223035+0000", 2024-03-19T11:31:56.001 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.690399641999999, 2024-03-19T11:31:56.001 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019488583, 2024-03-19T11:31:56.001 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:56.001 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:56.001 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:56.001 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:56.001 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:56.002 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.223035+0000", 2024-03-19T11:31:56.002 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:56.002 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:56.002 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:56.002 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:56.002 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.223035+0000", 2024-03-19T11:31:56.002 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:56.002 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:56.002 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:56.002 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:56.002 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.223037+0000", 2024-03-19T11:31:56.002 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3829999999999999e-06 2024-03-19T11:31:56.002 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:56.002 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:56.003 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:56.003 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.223073+0000", 2024-03-19T11:31:56.003 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6306000000000001e-05 2024-03-19T11:31:56.003 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:56.003 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:56.003 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:56.003 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.223095+0000", 2024-03-19T11:31:56.003 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1738000000000001e-05 2024-03-19T11:31:56.003 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:56.003 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:56.003 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:56.003 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.223107+0000", 2024-03-19T11:31:56.003 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2082000000000001e-05 2024-03-19T11:31:56.003 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:56.004 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:56.004 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:56.004 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.226145+0000", 2024-03-19T11:31:56.004 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003038022 2024-03-19T11:31:56.004 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:56.004 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:56.004 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:56.004 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.226201+0000", 2024-03-19T11:31:56.004 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6107999999999999e-05 2024-03-19T11:31:56.004 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:56.004 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:56.004 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:56.004 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.242322+0000", 2024-03-19T11:31:56.004 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016120196 2024-03-19T11:31:56.004 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:56.005 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:56.005 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:56.005 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.242523+0000", 2024-03-19T11:31:56.005 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020174799999999999 2024-03-19T11:31:56.005 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:56.005 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:56.005 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:56.005 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:56.005 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:56.005 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:56.007 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-19T11:31:56.007 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:56.015+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:56.108 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:56.115+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:56.147 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:56.147 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:31:56.147 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:31:56.147 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:56.164 INFO:tasks.rados.rados.0.smithi038.stdout:1738: done (0 left) 2024-03-19T11:31:56.164 INFO:tasks.rados.rados.0.smithi038.stdout:1739: read oid 18 snap 24 2024-03-19T11:31:56.164 INFO:tasks.rados.rados.0.smithi038.stdout:1739: expect (ObjNum 199 snap 22 seq_num 199) 2024-03-19T11:31:56.164 INFO:tasks.rados.rados.0.smithi038.stdout:1740: read oid 33 snap -1 2024-03-19T11:31:56.165 INFO:tasks.rados.rados.0.smithi038.stdout:1740: expect (ObjNum 306 snap 39 seq_num 306) 2024-03-19T11:31:56.165 INFO:tasks.rados.rados.0.smithi038.stdout:1741: set_chunk oid 30 target oid 2024-03-19T11:31:56.172 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-19T11:31:56.217 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:56.227+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:56.310 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:56.319+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:56.350 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:56.350 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:31:56.350 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:31:56.350 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:31:56.350 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:56.363 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-19T11:31:56.451 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:56.459+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:56.512 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:56.521+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:56.537 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:56.537 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:31:56.537 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:31:56.537 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:31:56.538 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.538 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1217 4.3 4:c8b09e87:::smithi03829856-15:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:31:56.538 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.155966+0000", 2024-03-19T11:31:56.538 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 98.387479986000002, 2024-03-19T11:31:56.538 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.023984647000000001, 2024-03-19T11:31:56.538 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:56.538 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:56.538 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:56.538 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:56.538 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:56.538 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1217 2024-03-19T11:31:56.538 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.538 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:56.538 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.539 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:56.539 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155966+0000", 2024-03-19T11:31:56.539 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:56.539 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.539 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.539 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:56.539 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155966+0000", 2024-03-19T11:31:56.539 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:56.539 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.539 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.539 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:56.539 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155968+0000", 2024-03-19T11:31:56.539 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.795e-06 2024-03-19T11:31:56.539 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.539 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.540 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:56.540 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155978+0000", 2024-03-19T11:31:56.540 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0574000000000001e-05 2024-03-19T11:31:56.540 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.540 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.540 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:56.540 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.156000+0000", 2024-03-19T11:31:56.540 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1250000000000002e-05 2024-03-19T11:31:56.540 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.540 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.540 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:56.541 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.156010+0000", 2024-03-19T11:31:56.541 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0168e-05 2024-03-19T11:31:56.541 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.541 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.541 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.541 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.156119+0000", 2024-03-19T11:31:56.541 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010950699999999999 2024-03-19T11:31:56.542 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.542 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.542 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:56.542 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.156231+0000", 2024-03-19T11:31:56.542 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000111445 2024-03-19T11:31:56.542 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.542 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.542 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:56.543 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.179951+0000", 2024-03-19T11:31:56.543 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.023719908000000001 2024-03-19T11:31:56.543 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:56.543 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:56.543 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:56.543 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.543 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.544 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.24 75/74 [PushOp(4:27909373:::smithi03829856-46:8, version: 46'59, data_included: [756315~767970,1933572~280606], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:8@46'59, size: 2756315, copy_subset: [(0, 2756315)], clone_subset: {}, snapset: 10=[]:{8: [6], c: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2214178, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:266590a5:::smithi03829856-16:e, version: 70'107, data_included: [652105~662359,1728171~386217], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:e@70'107, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 10=[]:{e: [d, b, 6]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:266590a5:::smithi03829856-16:head, version: 70'108, data_included: [652105~662359,1728171~386217], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:head@70'108, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:27909373:::smithi03829856-46:c, version: 70'109, data_included: [628595~895690,1807440~152886], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:c@70'109, size: 3428595, copy_subset: [(0, 3428595)], clone_subset: {}, snapset: 10=[]:{8: [6], c: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1960326, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:27909373:::smithi03829856-46:head, version: 70'110, data_included: [403401~454961,1584925~506039], data_size: 961000, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:head@70'110, size: 2403401, copy_subset: [(0, 2403401)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2403401, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:56.544 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.099395+0000", 2024-03-19T11:31:56.544 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 62.444050824999998, 2024-03-19T11:31:56.544 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.045391157000000001, 2024-03-19T11:31:56.544 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:56.544 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:56.544 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:56.545 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.545 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:56.545 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.099395+0000", 2024-03-19T11:31:56.545 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:56.545 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.545 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.545 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:56.545 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.099395+0000", 2024-03-19T11:31:56.545 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:56.545 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.546 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.546 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:56.546 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.099397+0000", 2024-03-19T11:31:56.546 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.691e-06 2024-03-19T11:31:56.546 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.546 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.546 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:56.546 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105009+0000", 2024-03-19T11:31:56.546 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0056121349999999999 2024-03-19T11:31:56.546 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.546 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.546 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:56.546 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105077+0000", 2024-03-19T11:31:56.546 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8561999999999998e-05 2024-03-19T11:31:56.546 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.547 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.547 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:56.547 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105133+0000", 2024-03-19T11:31:56.547 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5590000000000001e-05 2024-03-19T11:31:56.547 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.547 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.547 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.547 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105531+0000", 2024-03-19T11:31:56.547 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00039756200000000001 2024-03-19T11:31:56.547 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.547 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.547 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:56.548 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105579+0000", 2024-03-19T11:31:56.548 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8028999999999997e-05 2024-03-19T11:31:56.548 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.548 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.548 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:56.548 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.144786+0000", 2024-03-19T11:31:56.548 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.039207588000000002 2024-03-19T11:31:56.548 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:56.548 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:56.548 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:56.548 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.548 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.550 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.24 e75,74 4:266590a5:::smithi03829856-16:4,47'63,4:27909373:::smithi03829856-46:5,55'79,4:266590a5:::smithi03829856-16:10,67'105)", 2024-03-19T11:31:56.550 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.105191+0000", 2024-03-19T11:31:56.550 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 62.438254725, 2024-03-19T11:31:56.550 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.039580830999999997, 2024-03-19T11:31:56.550 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:56.550 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:56.551 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:56.551 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.551 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:56.551 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105191+0000", 2024-03-19T11:31:56.551 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:56.551 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.551 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.551 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:56.551 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105191+0000", 2024-03-19T11:31:56.551 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:56.551 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.551 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.551 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:56.551 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105192+0000", 2024-03-19T11:31:56.552 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.099e-06 2024-03-19T11:31:56.552 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.552 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.552 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:56.552 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105200+0000", 2024-03-19T11:31:56.552 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5279999999999998e-06 2024-03-19T11:31:56.552 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.552 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.552 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:56.552 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105211+0000", 2024-03-19T11:31:56.552 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1471000000000001e-05 2024-03-19T11:31:56.552 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.552 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.552 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:56.553 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105220+0000", 2024-03-19T11:31:56.553 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3469999999999999e-06 2024-03-19T11:31:56.553 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.553 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.553 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.553 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.144219+0000", 2024-03-19T11:31:56.553 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.038999024 2024-03-19T11:31:56.553 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.553 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.553 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:56.553 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.144257+0000", 2024-03-19T11:31:56.553 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8501999999999998e-05 2024-03-19T11:31:56.553 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.553 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.553 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:56.554 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.144772+0000", 2024-03-19T11:31:56.554 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00051486000000000001 2024-03-19T11:31:56.554 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:56.554 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:56.554 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:56.554 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.554 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.554 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 75/74 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:7, version: 55'82, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@55'82, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 58'86, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@58'86, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [504210~968657,1572695~79919], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:12, version: 66'101, data_included: [440350~816629,1309629~231947], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:12@66'101, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:10, version: 70'112, data_included: [440350~816629,1309629~231947], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:10@70'112, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 70'113, data_included: [440350~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@70'113, size: 2605085, copy_subset: [(0, 2605085)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1488926, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:56.554 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.107266+0000", 2024-03-19T11:31:56.554 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 62.436180208000003, 2024-03-19T11:31:56.554 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.048592669999999998, 2024-03-19T11:31:56.554 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:56.554 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:56.555 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:56.555 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.555 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:56.555 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.107266+0000", 2024-03-19T11:31:56.555 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:56.555 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.555 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.555 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:56.555 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.107266+0000", 2024-03-19T11:31:56.555 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:56.555 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.555 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.555 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:56.555 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.107267+0000", 2024-03-19T11:31:56.555 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1459999999999999e-06 2024-03-19T11:31:56.556 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.556 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.556 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:56.556 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.122861+0000", 2024-03-19T11:31:56.556 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01559372 2024-03-19T11:31:56.556 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.556 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.556 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:56.556 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.122932+0000", 2024-03-19T11:31:56.556 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1477999999999996e-05 2024-03-19T11:31:56.556 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.556 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.556 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:56.556 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.122999+0000", 2024-03-19T11:31:56.556 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6832e-05 2024-03-19T11:31:56.557 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.557 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.557 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.557 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123332+0000", 2024-03-19T11:31:56.557 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000333271 2024-03-19T11:31:56.557 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.557 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.557 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:56.557 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123371+0000", 2024-03-19T11:31:56.557 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8525999999999997e-05 2024-03-19T11:31:56.557 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.557 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.557 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:56.557 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.155858+0000", 2024-03-19T11:31:56.557 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.032487697000000003 2024-03-19T11:31:56.558 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:56.558 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:56.558 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:56.558 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.558 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.558 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.21 e75,74 4:8401c1f0:::smithi03829856-43:8,46'66,4:866f3dcc:::smithi03829856-31:2,47'73)", 2024-03-19T11:31:56.558 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.123075+0000", 2024-03-19T11:31:56.558 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 62.420370988999998, 2024-03-19T11:31:56.558 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.033023045000000001, 2024-03-19T11:31:56.558 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:56.558 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:56.558 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:56.558 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.558 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:56.559 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123075+0000", 2024-03-19T11:31:56.559 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:56.559 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.559 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.559 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:56.559 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123075+0000", 2024-03-19T11:31:56.559 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:56.559 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.559 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.559 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:56.559 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123076+0000", 2024-03-19T11:31:56.559 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5e-06 2024-03-19T11:31:56.559 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.559 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.559 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:56.560 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123086+0000", 2024-03-19T11:31:56.560 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8060000000000006e-06 2024-03-19T11:31:56.560 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.560 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.560 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:56.560 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123099+0000", 2024-03-19T11:31:56.560 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3251000000000001e-05 2024-03-19T11:31:56.560 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.560 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.560 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:56.560 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123112+0000", 2024-03-19T11:31:56.560 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2774e-05 2024-03-19T11:31:56.560 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.560 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.560 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.561 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.155902+0000", 2024-03-19T11:31:56.561 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.032790225999999999 2024-03-19T11:31:56.561 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.561 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.561 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:56.561 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.155914+0000", 2024-03-19T11:31:56.561 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1476e-05 2024-03-19T11:31:56.561 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.561 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.561 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:56.561 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.156098+0000", 2024-03-19T11:31:56.561 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000184012 2024-03-19T11:31:56.561 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:56.561 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:56.561 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:56.562 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.562 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.562 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 75/74 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:7, version: 55'82, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@55'82, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 58'86, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@58'86, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [1652614~973634,2888511~52681,3182305~22261], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3204566, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:12, version: 66'101, data_included: [1541576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:12@66'101, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:10, version: 70'112, data_included: [1541576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:10@70'112, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 70'113, data_included: [1488926~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@70'113, size: 2605085, copy_subset: [(0, 2605085)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2537502, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1488926, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:31:56.562 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.174030+0000", 2024-03-19T11:31:56.562 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 62.369415785000001, 2024-03-19T11:31:56.562 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.040799275000000003, 2024-03-19T11:31:56.562 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:56.562 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:56.562 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:56.562 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.562 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:56.562 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.174030+0000", 2024-03-19T11:31:56.562 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:56.563 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.563 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.563 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:56.563 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.174030+0000", 2024-03-19T11:31:56.563 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:56.563 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.563 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.563 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:56.563 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.174031+0000", 2024-03-19T11:31:56.563 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.322e-06 2024-03-19T11:31:56.563 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.563 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.563 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:56.563 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.184771+0000", 2024-03-19T11:31:56.563 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010739227 2024-03-19T11:31:56.563 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.564 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.564 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:56.564 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.184872+0000", 2024-03-19T11:31:56.564 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000101234 2024-03-19T11:31:56.564 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.564 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.564 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:56.564 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.184985+0000", 2024-03-19T11:31:56.564 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000113545 2024-03-19T11:31:56.564 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.564 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.564 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.564 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.185463+0000", 2024-03-19T11:31:56.564 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00047749899999999999 2024-03-19T11:31:56.564 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.565 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.565 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:56.565 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.185534+0000", 2024-03-19T11:31:56.565 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1007999999999995e-05 2024-03-19T11:31:56.565 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.565 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.565 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:56.565 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.214829+0000", 2024-03-19T11:31:56.565 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029295439999999999 2024-03-19T11:31:56.565 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:56.565 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:56.565 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:56.565 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.565 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.565 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2372 4.16 4:6868af91:::smithi03829856-10:b [read 832512~818688] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-03-19T11:31:56.566 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:58.455732+0000", 2024-03-19T11:31:56.566 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 58.087714069999997, 2024-03-19T11:31:56.566 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.69150541200000004, 2024-03-19T11:31:56.566 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:56.566 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:56.566 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:56.566 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:56.566 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:56.566 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2372 2024-03-19T11:31:56.566 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.566 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:56.566 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.566 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:56.566 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455732+0000", 2024-03-19T11:31:56.567 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:56.567 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.567 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.567 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:56.567 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455732+0000", 2024-03-19T11:31:56.567 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:56.567 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.567 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.567 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:56.567 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455735+0000", 2024-03-19T11:31:56.567 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.756e-06 2024-03-19T11:31:56.567 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.567 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.567 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:56.567 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455744+0000", 2024-03-19T11:31:56.568 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1880000000000008e-06 2024-03-19T11:31:56.568 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.568 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.568 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:56.568 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455764+0000", 2024-03-19T11:31:56.568 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0044999999999999e-05 2024-03-19T11:31:56.568 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.568 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.568 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:56.568 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455774+0000", 2024-03-19T11:31:56.568 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8770000000000005e-06 2024-03-19T11:31:56.568 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.568 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.568 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.568 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455837+0000", 2024-03-19T11:31:56.569 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3588000000000002e-05 2024-03-19T11:31:56.569 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.569 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.569 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:56.569 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455846+0000", 2024-03-19T11:31:56.569 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6030000000000005e-06 2024-03-19T11:31:56.569 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.569 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.569 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.569 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.076525+0000", 2024-03-19T11:31:56.569 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.62067875900000002 2024-03-19T11:31:56.569 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.569 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.569 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:31:56.569 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.076981+0000", 2024-03-19T11:31:56.569 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00045633300000000001 2024-03-19T11:31:56.570 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.570 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.570 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.570 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.104422+0000", 2024-03-19T11:31:56.570 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027440665999999999 2024-03-19T11:31:56.570 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.570 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.570 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:31:56.570 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.104786+0000", 2024-03-19T11:31:56.570 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00036471499999999999 2024-03-19T11:31:56.570 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.570 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.570 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.570 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.140725+0000", 2024-03-19T11:31:56.571 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.035939093999999998 2024-03-19T11:31:56.571 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.571 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.571 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:31:56.571 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.140841+0000", 2024-03-19T11:31:56.571 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000115207 2024-03-19T11:31:56.571 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.571 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.571 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.571 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.145559+0000", 2024-03-19T11:31:56.571 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.004718486 2024-03-19T11:31:56.571 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.571 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.571 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:56.571 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.145760+0000", 2024-03-19T11:31:56.572 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000201058 2024-03-19T11:31:56.572 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.572 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.572 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:56.572 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.147237+0000", 2024-03-19T11:31:56.572 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014770370000000001 2024-03-19T11:31:56.572 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:56.572 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:56.572 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:56.572 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.572 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.572 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.4.13:82 3.3 3:f6b6d21a:::smithi03829856-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-03-19T11:31:56.572 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:04.069448+0000", 2024-03-19T11:31:56.572 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 52.473998285, 2024-03-19T11:31:56.573 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.086427773999999999, 2024-03-19T11:31:56.573 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:56.573 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:56.573 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:56.573 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.573 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:56.573 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069448+0000", 2024-03-19T11:31:56.573 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:56.573 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.573 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.573 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:56.573 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069448+0000", 2024-03-19T11:31:56.573 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:56.573 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.573 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.574 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:56.574 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069449+0000", 2024-03-19T11:31:56.574 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8160000000000001e-06 2024-03-19T11:31:56.574 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.574 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.574 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:56.574 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069458+0000", 2024-03-19T11:31:56.574 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2900000000000002e-06 2024-03-19T11:31:56.574 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.574 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.574 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:56.574 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069491+0000", 2024-03-19T11:31:56.574 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3229000000000003e-05 2024-03-19T11:31:56.574 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.575 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.575 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:56.575 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069501+0000", 2024-03-19T11:31:56.575 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7629999999999995e-06 2024-03-19T11:31:56.575 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.575 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.575 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.575 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069573+0000", 2024-03-19T11:31:56.575 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2349999999999997e-05 2024-03-19T11:31:56.575 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.575 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.575 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:56.575 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069581+0000", 2024-03-19T11:31:56.575 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.388e-06 2024-03-19T11:31:56.575 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.575 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.576 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.576 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.075451+0000", 2024-03-19T11:31:56.576 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005869956 2024-03-19T11:31:56.576 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.576 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.576 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:56.576 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.077511+0000", 2024-03-19T11:31:56.576 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0020592610000000002 2024-03-19T11:31:56.576 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.576 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.576 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.576 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.108781+0000", 2024-03-19T11:31:56.576 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.031270344999999998 2024-03-19T11:31:56.576 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.576 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.577 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:56.577 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.108895+0000", 2024-03-19T11:31:56.577 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000114059 2024-03-19T11:31:56.577 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.577 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.577 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:56.577 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.109065+0000", 2024-03-19T11:31:56.577 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016979899999999999 2024-03-19T11:31:56.577 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.577 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.577 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:56.577 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.111730+0000", 2024-03-19T11:31:56.577 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0026651410000000002 2024-03-19T11:31:56.577 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.577 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.578 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:56.578 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.155616+0000", 2024-03-19T11:31:56.578 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.043885986000000002 2024-03-19T11:31:56.578 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.578 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.578 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:56.578 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.155745+0000", 2024-03-19T11:31:56.578 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012948899999999999 2024-03-19T11:31:56.578 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.578 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.578 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:56.578 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.155875+0000", 2024-03-19T11:31:56.578 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012990199999999999 2024-03-19T11:31:56.579 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:56.579 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:56.579 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:56.579 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.579 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.579 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.4.13:83 3.3 3:f6b6d21a:::smithi03829856-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-03-19T11:31:56.579 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:04.069526+0000", 2024-03-19T11:31:56.579 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 52.473919504999998, 2024-03-19T11:31:56.579 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.087140880000000004, 2024-03-19T11:31:56.579 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:56.579 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:56.579 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:56.579 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.579 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:56.580 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069526+0000", 2024-03-19T11:31:56.580 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:56.580 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.580 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.580 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:56.580 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069526+0000", 2024-03-19T11:31:56.580 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:56.580 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.580 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.580 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:56.580 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069527+0000", 2024-03-19T11:31:56.580 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2799999999999996e-07 2024-03-19T11:31:56.580 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.580 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.580 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:56.580 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069531+0000", 2024-03-19T11:31:56.581 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1489999999999996e-06 2024-03-19T11:31:56.581 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.581 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.581 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:56.581 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069539+0000", 2024-03-19T11:31:56.581 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3189999999999995e-06 2024-03-19T11:31:56.581 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.581 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.581 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:56.581 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069545+0000", 2024-03-19T11:31:56.581 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.592e-06 2024-03-19T11:31:56.581 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.581 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.581 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.581 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069616+0000", 2024-03-19T11:31:56.582 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1130000000000005e-05 2024-03-19T11:31:56.582 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.582 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.582 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:56.582 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069628+0000", 2024-03-19T11:31:56.582 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1697e-05 2024-03-19T11:31:56.582 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.582 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.582 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.582 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.077594+0000", 2024-03-19T11:31:56.582 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0079656989999999997 2024-03-19T11:31:56.582 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.582 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.582 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:56.583 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.077674+0000", 2024-03-19T11:31:56.583 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0338000000000002e-05 2024-03-19T11:31:56.583 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.583 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.583 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.583 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.112771+0000", 2024-03-19T11:31:56.583 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.035097325999999998 2024-03-19T11:31:56.583 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.583 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.583 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:56.583 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.112882+0000", 2024-03-19T11:31:56.583 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000110949 2024-03-19T11:31:56.583 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.583 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.583 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:56.584 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.113031+0000", 2024-03-19T11:31:56.584 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000148355 2024-03-19T11:31:56.584 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.584 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.584 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:56.584 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.114076+0000", 2024-03-19T11:31:56.584 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001045009 2024-03-19T11:31:56.584 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.584 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.584 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:56.584 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156482+0000", 2024-03-19T11:31:56.584 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.042406354 2024-03-19T11:31:56.584 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.584 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.584 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:56.585 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156549+0000", 2024-03-19T11:31:56.585 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7229000000000003e-05 2024-03-19T11:31:56.585 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.585 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.585 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:56.585 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156667+0000", 2024-03-19T11:31:56.585 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000118106 2024-03-19T11:31:56.585 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:56.585 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:56.585 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:56.585 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.585 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.585 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.4.13:84 3.3 3:ce6951d3:::smithi03829856-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-03-19T11:31:56.585 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:04.071995+0000", 2024-03-19T11:31:56.585 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 52.471450730000001, 2024-03-19T11:31:56.586 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.084172497999999998, 2024-03-19T11:31:56.586 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:56.586 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:56.586 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:56.586 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.586 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:56.586 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.071995+0000", 2024-03-19T11:31:56.586 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:56.586 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.586 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.586 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:56.586 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.071995+0000", 2024-03-19T11:31:56.586 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:56.586 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.587 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.587 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:56.587 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.071997+0000", 2024-03-19T11:31:56.587 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6029999999999999e-06 2024-03-19T11:31:56.587 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.587 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.587 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:56.587 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072004+0000", 2024-03-19T11:31:56.587 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8600000000000004e-06 2024-03-19T11:31:56.587 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.587 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.587 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:56.587 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072033+0000", 2024-03-19T11:31:56.587 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9836999999999999e-05 2024-03-19T11:31:56.587 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.588 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.588 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:56.588 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072042+0000", 2024-03-19T11:31:56.588 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9279999999999999e-06 2024-03-19T11:31:56.588 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.588 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.588 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.588 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072123+0000", 2024-03-19T11:31:56.588 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0630000000000006e-05 2024-03-19T11:31:56.588 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.588 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.588 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:56.588 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072132+0000", 2024-03-19T11:31:56.588 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5560000000000008e-06 2024-03-19T11:31:56.588 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.589 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.589 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.589 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.077712+0000", 2024-03-19T11:31:56.589 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055804970000000002 2024-03-19T11:31:56.589 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.589 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.589 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:56.589 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.079495+0000", 2024-03-19T11:31:56.589 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017831850000000001 2024-03-19T11:31:56.589 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.589 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.589 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.589 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.109825+0000", 2024-03-19T11:31:56.589 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.030329819000000001 2024-03-19T11:31:56.589 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.590 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.590 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:56.590 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.109904+0000", 2024-03-19T11:31:56.590 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9269000000000003e-05 2024-03-19T11:31:56.590 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.590 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.590 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:56.590 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.110085+0000", 2024-03-19T11:31:56.590 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00018018600000000001 2024-03-19T11:31:56.590 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.590 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.590 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:56.590 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.113581+0000", 2024-03-19T11:31:56.590 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034962919999999998 2024-03-19T11:31:56.591 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.591 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.591 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:56.591 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.155925+0000", 2024-03-19T11:31:56.591 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.042344404000000002 2024-03-19T11:31:56.591 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.591 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.591 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:56.591 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156026+0000", 2024-03-19T11:31:56.591 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000100599 2024-03-19T11:31:56.591 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.591 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.591 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:56.591 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156168+0000", 2024-03-19T11:31:56.591 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014183300000000001 2024-03-19T11:31:56.592 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:56.592 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:56.592 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:56.592 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.592 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.592 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.4.13:85 3.3 3:ce6951d3:::smithi03829856-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-03-19T11:31:56.592 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:04.072068+0000", 2024-03-19T11:31:56.592 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 52.471378094000002, 2024-03-19T11:31:56.592 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.084368466000000003, 2024-03-19T11:31:56.592 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:56.592 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:56.592 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:56.592 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.592 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:56.592 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072068+0000", 2024-03-19T11:31:56.593 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:56.593 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.593 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.593 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:56.593 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072068+0000", 2024-03-19T11:31:56.593 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:56.593 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.593 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.593 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:56.593 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072068+0000", 2024-03-19T11:31:56.593 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9699999999999996e-07 2024-03-19T11:31:56.593 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.593 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.593 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:56.593 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072074+0000", 2024-03-19T11:31:56.594 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2279999999999998e-06 2024-03-19T11:31:56.594 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.594 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.594 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:56.594 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072140+0000", 2024-03-19T11:31:56.594 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6638999999999997e-05 2024-03-19T11:31:56.594 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.594 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.594 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:56.594 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072147+0000", 2024-03-19T11:31:56.594 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8750000000000002e-06 2024-03-19T11:31:56.594 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.594 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.594 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.594 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072201+0000", 2024-03-19T11:31:56.595 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3634000000000002e-05 2024-03-19T11:31:56.595 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.595 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.595 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:56.595 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072211+0000", 2024-03-19T11:31:56.595 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0589e-05 2024-03-19T11:31:56.595 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.595 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.595 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.595 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.079687+0000", 2024-03-19T11:31:56.595 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074757219999999997 2024-03-19T11:31:56.595 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.595 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.595 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:56.595 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.079769+0000", 2024-03-19T11:31:56.596 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2251999999999998e-05 2024-03-19T11:31:56.596 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.596 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.596 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.596 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.111798+0000", 2024-03-19T11:31:56.596 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.032028889999999997 2024-03-19T11:31:56.596 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.596 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.596 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:56.596 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.111902+0000", 2024-03-19T11:31:56.596 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000103773 2024-03-19T11:31:56.596 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.596 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.596 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:56.596 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.112073+0000", 2024-03-19T11:31:56.597 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017102799999999999 2024-03-19T11:31:56.597 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.597 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.597 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:56.597 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.113591+0000", 2024-03-19T11:31:56.597 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0015179569999999999 2024-03-19T11:31:56.597 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.597 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.597 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:56.597 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156223+0000", 2024-03-19T11:31:56.597 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.042632153999999998 2024-03-19T11:31:56.597 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.597 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.597 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:56.597 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156321+0000", 2024-03-19T11:31:56.598 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8246999999999996e-05 2024-03-19T11:31:56.598 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.598 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.598 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:56.598 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156436+0000", 2024-03-19T11:31:56.598 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000114881 2024-03-19T11:31:56.598 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:56.598 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:56.598 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:56.598 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.598 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.598 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3203 4.6 4:626acd4d:::smithi03829856-2:1f [sparse-read 0~1665607,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108)", 2024-03-19T11:31:56.598 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:27.253945+0000", 2024-03-19T11:31:56.598 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 29.289501027, 2024-03-19T11:31:56.599 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.83307735599999999, 2024-03-19T11:31:56.599 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:56.599 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:56.599 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:56.599 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:56.599 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:56.599 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3203 2024-03-19T11:31:56.599 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.599 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:56.599 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.599 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:56.599 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.253945+0000", 2024-03-19T11:31:56.599 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:56.599 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.599 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.600 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:56.600 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.253945+0000", 2024-03-19T11:31:56.600 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:56.600 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.600 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.600 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:56.600 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.253947+0000", 2024-03-19T11:31:56.600 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5979999999999999e-06 2024-03-19T11:31:56.600 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.600 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.600 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:56.600 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.253957+0000", 2024-03-19T11:31:56.600 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7699999999999996e-06 2024-03-19T11:31:56.600 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.601 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.601 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:56.601 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.253980+0000", 2024-03-19T11:31:56.601 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2466000000000001e-05 2024-03-19T11:31:56.601 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.601 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.601 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:56.601 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.253992+0000", 2024-03-19T11:31:56.601 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1985e-05 2024-03-19T11:31:56.601 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.601 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.601 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.601 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254071+0000", 2024-03-19T11:31:56.601 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8869000000000007e-05 2024-03-19T11:31:56.601 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.602 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.602 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:56.602 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254084+0000", 2024-03-19T11:31:56.602 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3716999999999999e-05 2024-03-19T11:31:56.602 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.602 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.602 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.602 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.086473+0000", 2024-03-19T11:31:56.602 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.83238890300000001 2024-03-19T11:31:56.602 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.602 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.602 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:56.602 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.086618+0000", 2024-03-19T11:31:56.602 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014479200000000001 2024-03-19T11:31:56.603 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.603 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.603 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:56.603 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.087022+0000", 2024-03-19T11:31:56.603 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00040425599999999997 2024-03-19T11:31:56.603 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:56.603 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:56.603 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:56.603 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.603 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.603 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3204 4.6 4:626acd4d:::smithi03829856-2:1f [read 0~3462242,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108)", 2024-03-19T11:31:56.603 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:27.254015+0000", 2024-03-19T11:31:56.603 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 29.289431245999999, 2024-03-19T11:31:56.603 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.83578705200000003, 2024-03-19T11:31:56.603 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:56.604 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:56.604 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:56.604 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:56.604 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:56.604 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3204 2024-03-19T11:31:56.604 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.604 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:56.604 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.604 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:56.604 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254015+0000", 2024-03-19T11:31:56.604 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:56.604 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.604 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.604 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:56.604 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254015+0000", 2024-03-19T11:31:56.605 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:56.605 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.605 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.605 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:56.605 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254015+0000", 2024-03-19T11:31:56.605 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8599999999999997e-07 2024-03-19T11:31:56.605 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.605 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.605 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:56.605 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254022+0000", 2024-03-19T11:31:56.605 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2870000000000001e-06 2024-03-19T11:31:56.605 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.605 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.605 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:56.605 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254031+0000", 2024-03-19T11:31:56.606 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0960000000000007e-06 2024-03-19T11:31:56.606 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.606 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.606 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:56.606 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254036+0000", 2024-03-19T11:31:56.606 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3510000000000001e-06 2024-03-19T11:31:56.606 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.606 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.606 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.606 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254137+0000", 2024-03-19T11:31:56.606 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000101065 2024-03-19T11:31:56.606 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.606 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.606 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:56.607 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254147+0000", 2024-03-19T11:31:56.607 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3929999999999997e-06 2024-03-19T11:31:56.607 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.607 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.607 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.607 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.087055+0000", 2024-03-19T11:31:56.607 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.83290843299999995 2024-03-19T11:31:56.607 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.607 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.607 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:56.607 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.087142+0000", 2024-03-19T11:31:56.607 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7151999999999995e-05 2024-03-19T11:31:56.607 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.607 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.607 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:56.607 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.089802+0000", 2024-03-19T11:31:56.608 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0026594890000000001 2024-03-19T11:31:56.608 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:56.608 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:56.608 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:56.608 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.608 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.608 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3205 4.6 4:626acd4d:::smithi03829856-2:1f [sparse-read 0~1665607] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108)", 2024-03-19T11:31:56.608 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:27.254049+0000", 2024-03-19T11:31:56.608 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 29.289397228999999, 2024-03-19T11:31:56.608 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.83625012399999998, 2024-03-19T11:31:56.608 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:56.608 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:56.608 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:56.608 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:56.609 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:56.609 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3205 2024-03-19T11:31:56.609 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.609 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:56.609 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.609 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:56.609 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254049+0000", 2024-03-19T11:31:56.609 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:56.609 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.609 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.609 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:56.609 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254049+0000", 2024-03-19T11:31:56.609 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:56.609 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.609 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.610 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:56.610 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254049+0000", 2024-03-19T11:31:56.610 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0699999999999997e-07 2024-03-19T11:31:56.610 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.610 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.610 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:56.610 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254053+0000", 2024-03-19T11:31:56.610 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.546e-06 2024-03-19T11:31:56.610 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.610 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.610 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:56.610 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254063+0000", 2024-03-19T11:31:56.610 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7939999999999997e-06 2024-03-19T11:31:56.610 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.611 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.611 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:56.611 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254072+0000", 2024-03-19T11:31:56.611 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9250000000000001e-06 2024-03-19T11:31:56.611 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.611 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.611 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.611 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254186+0000", 2024-03-19T11:31:56.611 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011446300000000001 2024-03-19T11:31:56.611 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.611 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.611 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:56.611 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254198+0000", 2024-03-19T11:31:56.611 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1933000000000001e-05 2024-03-19T11:31:56.611 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.612 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.612 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.612 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.089838+0000", 2024-03-19T11:31:56.612 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.83563974200000002 2024-03-19T11:31:56.612 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.612 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.612 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:56.612 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.089941+0000", 2024-03-19T11:31:56.612 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000103052 2024-03-19T11:31:56.612 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.612 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.612 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:56.612 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.090299+0000", 2024-03-19T11:31:56.612 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00035806199999999997 2024-03-19T11:31:56.612 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:56.613 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:56.613 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:56.613 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.613 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.613 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3220 4.16 4:6868af91:::smithi03829856-10:head [tier-promote] snapc 21=[21,20,1f,1e,1d,1c,1b,1a,18,17] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-19T11:31:56.613 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:28.068879+0000", 2024-03-19T11:31:56.613 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 28.474567022999999, 2024-03-19T11:31:56.613 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025399244000000001, 2024-03-19T11:31:56.613 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:56.613 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:56.613 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:56.613 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:56.613 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:56.613 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3220 2024-03-19T11:31:56.613 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.614 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:56.614 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.614 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:56.614 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.068879+0000", 2024-03-19T11:31:56.614 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:56.614 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.614 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.614 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:56.614 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.068879+0000", 2024-03-19T11:31:56.614 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:56.614 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.614 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.614 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:56.614 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.068881+0000", 2024-03-19T11:31:56.615 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3879999999999998e-06 2024-03-19T11:31:56.615 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.615 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.615 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:56.615 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.068892+0000", 2024-03-19T11:31:56.615 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0672999999999999e-05 2024-03-19T11:31:56.615 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.615 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.615 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:56.615 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.068917+0000", 2024-03-19T11:31:56.615 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5367e-05 2024-03-19T11:31:56.615 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.615 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.615 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:56.615 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.068935+0000", 2024-03-19T11:31:56.616 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7832000000000001e-05 2024-03-19T11:31:56.616 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.616 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.616 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.616 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.069075+0000", 2024-03-19T11:31:56.616 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013980899999999999 2024-03-19T11:31:56.616 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.616 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.616 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-03-19T11:31:56.616 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.069091+0000", 2024-03-19T11:31:56.616 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6325e-05 2024-03-19T11:31:56.616 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.616 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.616 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.616 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.071382+0000", 2024-03-19T11:31:56.617 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022910109999999999 2024-03-19T11:31:56.617 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.617 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.617 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:56.617 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.071538+0000", 2024-03-19T11:31:56.617 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015600999999999999 2024-03-19T11:31:56.617 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.617 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.617 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:56.617 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.094278+0000", 2024-03-19T11:31:56.617 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.022739829 2024-03-19T11:31:56.617 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:56.617 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:56.617 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:56.617 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.617 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.618 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3885 4.21 4:8401c1f0:::smithi03829856-43:head [write 443784~648161] snapc 29=[29,28,27,26,25,24,23,22,21,1f,1e,1d,1a,17] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e129)", 2024-03-19T11:31:56.618 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:50.078273+0000", 2024-03-19T11:31:56.618 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 6.4651732019999999, 2024-03-19T11:31:56.618 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1185664280000001, 2024-03-19T11:31:56.618 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:56.618 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:56.618 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:56.618 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:56.618 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:56.618 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3885 2024-03-19T11:31:56.618 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.618 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:56.618 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.618 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:56.619 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.078273+0000", 2024-03-19T11:31:56.619 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:56.619 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.619 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.619 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:56.619 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.078273+0000", 2024-03-19T11:31:56.619 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:56.619 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.619 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.619 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:56.619 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.078276+0000", 2024-03-19T11:31:56.619 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.942e-06 2024-03-19T11:31:56.619 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.619 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.619 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:56.620 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.078790+0000", 2024-03-19T11:31:56.620 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00051449799999999995 2024-03-19T11:31:56.620 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.620 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.620 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:56.620 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.078809+0000", 2024-03-19T11:31:56.620 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8980000000000001e-05 2024-03-19T11:31:56.620 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.620 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.620 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:56.620 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.078821+0000", 2024-03-19T11:31:56.620 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1432e-05 2024-03-19T11:31:56.620 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.620 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.620 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.621 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.079070+0000", 2024-03-19T11:31:56.621 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00024937100000000002 2024-03-19T11:31:56.621 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.621 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.621 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-03-19T11:31:56.621 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.079079+0000", 2024-03-19T11:31:56.621 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2949999999999999e-06 2024-03-19T11:31:56.621 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.621 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.621 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.621 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.083821+0000", 2024-03-19T11:31:56.621 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0047417129999999998 2024-03-19T11:31:56.621 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.621 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.622 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:56.622 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.083836+0000", 2024-03-19T11:31:56.622 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4591e-05 2024-03-19T11:31:56.622 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.622 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.622 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.622 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.101489+0000", 2024-03-19T11:31:56.622 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0176535019999999 2024-03-19T11:31:56.622 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.623 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.623 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:56.623 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.104918+0000", 2024-03-19T11:31:56.623 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003429353 2024-03-19T11:31:56.623 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.623 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.623 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.623 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.142247+0000", 2024-03-19T11:31:56.623 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.037329108 2024-03-19T11:31:56.623 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.623 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.623 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:56.623 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.142409+0000", 2024-03-19T11:31:56.623 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016169700000000001 2024-03-19T11:31:56.623 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.624 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.624 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4,5", 2024-03-19T11:31:56.624 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.142754+0000", 2024-03-19T11:31:56.624 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000345025 2024-03-19T11:31:56.624 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.624 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.624 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:56.624 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.161867+0000", 2024-03-19T11:31:56.624 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019112371 2024-03-19T11:31:56.624 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.624 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.624 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:56.624 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.166589+0000", 2024-03-19T11:31:56.624 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0047220270000000002 2024-03-19T11:31:56.624 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.625 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.625 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:56.625 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.196673+0000", 2024-03-19T11:31:56.625 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.030084422999999999 2024-03-19T11:31:56.625 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.625 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.625 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:56.625 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.196730+0000", 2024-03-19T11:31:56.625 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7027000000000001e-05 2024-03-19T11:31:56.625 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.625 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.625 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:56.625 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.196839+0000", 2024-03-19T11:31:56.625 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000109073 2024-03-19T11:31:56.626 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:56.626 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:56.626 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:56.626 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.626 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.626 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3886 4.21 4:8401c1f0:::smithi03829856-43:head [write 1657676~659775] snapc 29=[29,28,27,26,25,24,23,22,21,1f,1e,1d,1a,17] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e129)", 2024-03-19T11:31:56.626 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:50.078842+0000", 2024-03-19T11:31:56.626 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 6.4646041639999998, 2024-03-19T11:31:56.626 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1181774870000001, 2024-03-19T11:31:56.626 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:56.626 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:56.626 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:56.626 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:56.626 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:56.626 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3886 2024-03-19T11:31:56.627 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.627 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:56.627 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.627 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:56.627 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.078842+0000", 2024-03-19T11:31:56.627 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:56.627 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.627 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.627 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:56.627 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.078842+0000", 2024-03-19T11:31:56.627 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:56.627 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.627 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.627 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:56.627 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.078843+0000", 2024-03-19T11:31:56.628 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7400000000000002e-07 2024-03-19T11:31:56.628 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.628 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.628 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:56.628 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.079370+0000", 2024-03-19T11:31:56.628 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00052713200000000003 2024-03-19T11:31:56.628 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.628 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.628 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:56.628 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.079383+0000", 2024-03-19T11:31:56.628 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3131e-05 2024-03-19T11:31:56.628 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.628 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.628 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:56.629 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.079393+0000", 2024-03-19T11:31:56.629 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0114e-05 2024-03-19T11:31:56.629 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.629 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.629 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.629 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082249+0000", 2024-03-19T11:31:56.629 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0028558260000000001 2024-03-19T11:31:56.629 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.629 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.629 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:31:56.629 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082258+0000", 2024-03-19T11:31:56.629 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.5740000000000002e-06 2024-03-19T11:31:56.629 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.629 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.630 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.630 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.084372+0000", 2024-03-19T11:31:56.630 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0021131909999999999 2024-03-19T11:31:56.630 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.630 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.630 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:56.630 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.084383+0000", 2024-03-19T11:31:56.630 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1099e-05 2024-03-19T11:31:56.630 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.630 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.630 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.630 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.105453+0000", 2024-03-19T11:31:56.630 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.021070116 2024-03-19T11:31:56.630 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.630 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.631 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:56.631 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.105574+0000", 2024-03-19T11:31:56.631 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012102199999999999 2024-03-19T11:31:56.631 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.631 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.631 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.631 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.151364+0000", 2024-03-19T11:31:56.631 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.045790294000000002 2024-03-19T11:31:56.631 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.631 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.631 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:56.631 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.151579+0000", 2024-03-19T11:31:56.631 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00021460399999999999 2024-03-19T11:31:56.631 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.631 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.632 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4,5", 2024-03-19T11:31:56.632 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.151927+0000", 2024-03-19T11:31:56.632 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00034839700000000002 2024-03-19T11:31:56.632 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.632 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.632 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:56.632 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.161878+0000", 2024-03-19T11:31:56.632 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0099512430000000002 2024-03-19T11:31:56.632 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.632 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.632 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:56.632 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.169256+0000", 2024-03-19T11:31:56.632 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0073777440000000003 2024-03-19T11:31:56.632 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.632 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.633 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:56.633 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.196889+0000", 2024-03-19T11:31:56.633 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027633179000000001 2024-03-19T11:31:56.633 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.633 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.633 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:56.633 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.196937+0000", 2024-03-19T11:31:56.633 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7966999999999997e-05 2024-03-19T11:31:56.633 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.633 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.633 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:56.633 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.197019+0000", 2024-03-19T11:31:56.633 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1979999999999998e-05 2024-03-19T11:31:56.633 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:56.634 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:56.634 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:56.634 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.634 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.634 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3887 4.21 4:8401c1f0:::smithi03829856-43:head [write 2818209~425575] snapc 29=[29,28,27,26,25,24,23,22,21,1f,1e,1d,1a,17] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e129)", 2024-03-19T11:31:56.634 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:50.082016+0000", 2024-03-19T11:31:56.634 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 6.4614303240000002, 2024-03-19T11:31:56.634 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.115265664, 2024-03-19T11:31:56.634 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:56.634 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:56.634 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:56.634 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:56.634 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:56.634 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3887 2024-03-19T11:31:56.635 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.635 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:56.635 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.635 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:56.635 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082016+0000", 2024-03-19T11:31:56.635 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:56.635 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.635 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.635 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:56.635 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082016+0000", 2024-03-19T11:31:56.635 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:56.635 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.635 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.635 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:56.636 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082017+0000", 2024-03-19T11:31:56.636 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.587e-06 2024-03-19T11:31:56.636 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.636 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.636 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:56.636 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082203+0000", 2024-03-19T11:31:56.636 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000185613 2024-03-19T11:31:56.636 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.636 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.636 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:56.636 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082214+0000", 2024-03-19T11:31:56.636 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1357e-05 2024-03-19T11:31:56.636 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.636 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.636 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:56.636 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082221+0000", 2024-03-19T11:31:56.637 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6290000000000004e-06 2024-03-19T11:31:56.637 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.637 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.637 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.637 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082300+0000", 2024-03-19T11:31:56.637 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9425999999999995e-05 2024-03-19T11:31:56.637 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.637 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.637 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:31:56.637 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082306+0000", 2024-03-19T11:31:56.637 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1419999999999998e-06 2024-03-19T11:31:56.637 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.637 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.637 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.638 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.084417+0000", 2024-03-19T11:31:56.638 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0021107589999999998 2024-03-19T11:31:56.638 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.638 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.638 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:56.638 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.084426+0000", 2024-03-19T11:31:56.638 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8410000000000006e-06 2024-03-19T11:31:56.638 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.638 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.638 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.638 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.105770+0000", 2024-03-19T11:31:56.638 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0213442100000001 2024-03-19T11:31:56.638 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.638 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.638 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:56.639 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.105890+0000", 2024-03-19T11:31:56.639 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000120349 2024-03-19T11:31:56.639 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.639 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.639 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.639 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.157275+0000", 2024-03-19T11:31:56.639 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.051384678000000003 2024-03-19T11:31:56.639 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.639 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.639 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:56.639 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.157445+0000", 2024-03-19T11:31:56.639 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000169746 2024-03-19T11:31:56.639 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.639 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.639 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4,5", 2024-03-19T11:31:56.640 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.157790+0000", 2024-03-19T11:31:56.640 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00034491799999999999 2024-03-19T11:31:56.640 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.640 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.640 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:56.640 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.161935+0000", 2024-03-19T11:31:56.640 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0041450269999999999 2024-03-19T11:31:56.640 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.640 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.640 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:56.640 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.189666+0000", 2024-03-19T11:31:56.640 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027731062000000001 2024-03-19T11:31:56.640 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.640 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.641 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:56.641 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.197073+0000", 2024-03-19T11:31:56.641 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074070270000000001 2024-03-19T11:31:56.641 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.641 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.641 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:56.641 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.197116+0000", 2024-03-19T11:31:56.641 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2758e-05 2024-03-19T11:31:56.641 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.641 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.641 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:56.641 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.197281+0000", 2024-03-19T11:31:56.641 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016553499999999999 2024-03-19T11:31:56.641 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:56.641 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:56.642 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:56.642 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.642 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.642 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3888 4.21 4:8401c1f0:::smithi03829856-43:head [setxattr _header (57) in=7b,truncate 3243784] snapc 29=[29,28,27,26,25,24,23,22,21,1f,1e,1d,1a,17] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e129)", 2024-03-19T11:31:56.642 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:50.082238+0000", 2024-03-19T11:31:56.642 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 6.461207441, 2024-03-19T11:31:56.642 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.115518945, 2024-03-19T11:31:56.642 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:56.642 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:56.642 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:56.642 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:56.642 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:56.642 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3888 2024-03-19T11:31:56.642 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.642 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:56.643 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.643 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:56.643 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082238+0000", 2024-03-19T11:31:56.643 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:56.643 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.643 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.643 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:56.643 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082238+0000", 2024-03-19T11:31:56.643 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:56.643 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.643 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.643 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:56.643 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082239+0000", 2024-03-19T11:31:56.643 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6400000000000001e-07 2024-03-19T11:31:56.643 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.644 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.644 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:56.644 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082249+0000", 2024-03-19T11:31:56.644 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9490000000000008e-06 2024-03-19T11:31:56.644 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.644 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.644 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:56.644 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082260+0000", 2024-03-19T11:31:56.644 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0499000000000001e-05 2024-03-19T11:31:56.644 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.644 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.644 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:56.644 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082267+0000", 2024-03-19T11:31:56.644 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9029999999999998e-06 2024-03-19T11:31:56.644 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.645 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.645 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.645 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082341+0000", 2024-03-19T11:31:56.645 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4776000000000004e-05 2024-03-19T11:31:56.645 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.645 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.645 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:31:56.645 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082350+0000", 2024-03-19T11:31:56.645 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2279999999999998e-06 2024-03-19T11:31:56.645 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.645 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.645 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.645 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.084459+0000", 2024-03-19T11:31:56.645 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0021094579999999998 2024-03-19T11:31:56.645 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.646 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.646 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:56.646 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.084468+0000", 2024-03-19T11:31:56.646 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2e-06 2024-03-19T11:31:56.646 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.646 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.646 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.646 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.105958+0000", 2024-03-19T11:31:56.646 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0214894450000001 2024-03-19T11:31:56.646 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.646 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.646 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:56.646 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.106112+0000", 2024-03-19T11:31:56.646 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000154022 2024-03-19T11:31:56.646 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.647 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.647 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.647 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.160042+0000", 2024-03-19T11:31:56.647 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.053929935999999998 2024-03-19T11:31:56.647 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.647 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.647 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:56.647 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.160205+0000", 2024-03-19T11:31:56.647 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016317600000000001 2024-03-19T11:31:56.647 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.647 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.647 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4,5", 2024-03-19T11:31:56.647 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.160566+0000", 2024-03-19T11:31:56.647 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000360918 2024-03-19T11:31:56.648 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.648 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.648 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:31:56.648 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.162064+0000", 2024-03-19T11:31:56.648 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014983380000000001 2024-03-19T11:31:56.648 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.648 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.648 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:56.648 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.196608+0000", 2024-03-19T11:31:56.648 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.034544115 2024-03-19T11:31:56.648 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.648 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.648 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:56.648 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.197420+0000", 2024-03-19T11:31:56.648 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00081147299999999995 2024-03-19T11:31:56.648 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.649 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.649 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:31:56.649 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.197519+0000", 2024-03-19T11:31:56.649 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8932999999999996e-05 2024-03-19T11:31:56.649 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.649 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.649 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:56.649 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.197757+0000", 2024-03-19T11:31:56.649 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000238812 2024-03-19T11:31:56.649 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:56.649 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:56.649 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:56.649 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.649 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.649 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3889 4.21 4:8401c1f0:::smithi03829856-43:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e129)", 2024-03-19T11:31:56.650 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:50.082302+0000", 2024-03-19T11:31:56.650 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 6.4611442920000002, 2024-03-19T11:31:56.650 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.116076547, 2024-03-19T11:31:56.650 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:31:56.650 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:31:56.650 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:31:56.650 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:31:56.650 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:56.650 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3889 2024-03-19T11:31:56.650 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.650 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:31:56.650 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.650 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:31:56.650 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082302+0000", 2024-03-19T11:31:56.651 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:56.651 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.651 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.651 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:31:56.651 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082302+0000", 2024-03-19T11:31:56.651 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:31:56.651 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.651 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.651 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:31:56.651 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082303+0000", 2024-03-19T11:31:56.651 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2720000000000001e-06 2024-03-19T11:31:56.651 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.651 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.651 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:31:56.651 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082314+0000", 2024-03-19T11:31:56.651 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0864e-05 2024-03-19T11:31:56.652 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.652 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.652 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:31:56.652 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082327+0000", 2024-03-19T11:31:56.652 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2972e-05 2024-03-19T11:31:56.652 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.652 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.652 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:31:56.652 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082339+0000", 2024-03-19T11:31:56.652 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2458e-05 2024-03-19T11:31:56.652 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.652 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.652 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.652 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.085086+0000", 2024-03-19T11:31:56.652 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0027472859999999998 2024-03-19T11:31:56.653 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.653 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.653 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:31:56.653 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.085100+0000", 2024-03-19T11:31:56.653 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3339000000000001e-05 2024-03-19T11:31:56.653 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.653 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.653 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.653 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.106149+0000", 2024-03-19T11:31:56.653 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0210496010000001 2024-03-19T11:31:56.653 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.653 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.653 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:56.653 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.106232+0000", 2024-03-19T11:31:56.653 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2923999999999994e-05 2024-03-19T11:31:56.654 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.654 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.654 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.654 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.161418+0000", 2024-03-19T11:31:56.654 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.055186156 2024-03-19T11:31:56.654 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.654 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.654 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-03-19T11:31:56.654 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.161535+0000", 2024-03-19T11:31:56.654 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000117018 2024-03-19T11:31:56.654 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.654 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.654 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:31:56.654 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.197872+0000", 2024-03-19T11:31:56.655 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.036336161999999998 2024-03-19T11:31:56.655 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.655 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.655 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:31:56.655 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.198037+0000", 2024-03-19T11:31:56.655 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000164924 2024-03-19T11:31:56.655 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:31:56.655 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:31:56.655 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:31:56.655 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.198378+0000", 2024-03-19T11:31:56.655 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00034157099999999999 2024-03-19T11:31:56.655 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:56.655 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:56.655 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:56.655 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:31:56.656 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:31:56.656 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:56.656 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-19T11:31:56.658 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-19T11:31:56.658 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:56.667+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:56.758 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:56.767+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:56.823 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:31:56.823 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:31:56.823 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:31:56.823 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:31:56.839 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-19T11:31:56.849 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:31:56.850 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:31:56.866 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:31:56.867 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:31:56.867 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:56.875+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:56.884 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:31:56.885 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:31:56.901 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:31:56.901 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:31:56.921 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:31:56.922 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:31:56.942 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:31:56.942 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:31:56.960 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:56.969+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:56.963 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:31:56.963 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:31:56.964 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:56.964 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:56.979 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-19T11:31:56.984 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:31:56.984 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:31:57.061 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:57.071+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:57.101 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:57.101 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:57.115 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-19T11:31:57.163 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:57.171+0000 7fa5e08db640 -1 osd.3 135 handle_osd_map injecting map crc failure 2024-03-19T11:31:57.163 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:57.171+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:57.167 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:57.177+0000 7f923da1c640 -1 osd.7 135 handle_osd_map injecting map crc failure 2024-03-19T11:31:57.186 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-03-19T11:31:57.212 INFO:tasks.rados.rados.0.smithi038.stdout:1741: set_chunk oid 30 offset: 223744 length: 921600 target oid 0c6d0baa205b4f6f261e0515a98bd6a31a7c2d7d7d254c0037ac9b079073e9f2 offset: 0 snap -1 2024-03-19T11:31:57.212 INFO:tasks.rados.rados.0.smithi038.stdout:1742: read oid 41 snap -1 2024-03-19T11:31:57.212 INFO:tasks.rados.rados.0.smithi038.stdout:1742: expect (ObjNum 311 snap 40 seq_num 311) 2024-03-19T11:31:57.212 INFO:tasks.rados.rados.0.smithi038.stdout:1743: set_chunk oid 11 target oid 2024-03-19T11:31:57.212 INFO:tasks.rados.rados.0.smithi038.stdout:1743: set_chunk oid 11 offset: 0 length: 4000000 target oid smithi03829856- offset: 0 snap -1 2024-03-19T11:31:57.213 INFO:tasks.rados.rados.0.smithi038.stdout:1744: read oid 31 snap -1 2024-03-19T11:31:57.213 INFO:tasks.rados.rados.0.smithi038.stdout:1744: expect (ObjNum 300 snap 39 seq_num 300) 2024-03-19T11:31:57.213 INFO:tasks.rados.rados.0.smithi038.stdout:1745: write oid 38 current snap is 40 2024-03-19T11:31:57.213 INFO:tasks.rados.rados.0.smithi038.stdout:1745: seq_num 322 ranges {611854=717004,2126143=561767,3259733=152121} 2024-03-19T11:31:57.221 INFO:tasks.rados.rados.0.smithi038.stdout:1745: writing smithi03829856-38 from 611854 to 1328858 tid 1 2024-03-19T11:31:57.233 INFO:tasks.rados.rados.0.smithi038.stdout:1745: writing smithi03829856-38 from 2126143 to 2687910 tid 2 2024-03-19T11:31:57.238 INFO:tasks.rados.rados.0.smithi038.stdout:1745: writing smithi03829856-38 from 3259733 to 3411854 tid 3 2024-03-19T11:31:57.238 INFO:tasks.rados.rados.0.smithi038.stdout:1746: read oid 22 snap -1 2024-03-19T11:31:57.238 INFO:tasks.rados.rados.0.smithi038.stdout:1746: expect (ObjNum 295 snap 39 seq_num 295) 2024-03-19T11:31:57.252 INFO:tasks.rados.rados.0.smithi038.stdout:1739: expect (ObjNum 199 snap 22 seq_num 199) 2024-03-19T11:31:57.267 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:57.277+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:57.335 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:57.336 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:57.359 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-19T11:31:57.405 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:57.415+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:57.468 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:57.475+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:57.512 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:57.512 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:57.524 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-19T11:31:57.569 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:57.579+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:57.573 INFO:tasks.rados.rados.0.smithi038.stdout:1740: expect (ObjNum 306 snap 39 seq_num 306) 2024-03-19T11:31:57.670 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:57.677+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:57.681 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:57.681 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:57.695 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-19T11:31:57.781 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:57.791+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:57.850 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:57.850 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:57.855 INFO:tasks.rados.rados.0.smithi038.stdout:1743: finishing set_chunk to oid 11 2024-03-19T11:31:57.855 INFO:tasks.rados.rados.0.smithi038.stdout:1743: got expected ERANGE (src dne) 2024-03-19T11:31:57.855 INFO:tasks.rados.rados.0.smithi038.stdout:1741: finishing set_chunk to oid 30 2024-03-19T11:31:57.855 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 30 set_chunk returned error code -95 offset: 223744 length: 921600 2024-03-19T11:31:57.855 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 30 v 0 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-03-19T11:31:57.855 INFO:tasks.rados.rados.0.smithi038.stdout:1742: expect (ObjNum 311 snap 40 seq_num 311) 2024-03-19T11:31:57.863 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-19T11:31:57.905 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:57.915+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:31:57.972 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:57.979+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:57.999 INFO:tasks.rados.rados.0.smithi038.stdout:1745: finishing write tid 1 to smithi03829856-38 2024-03-19T11:31:57.999 INFO:tasks.rados.rados.0.smithi038.stdout:1744: expect (ObjNum 300 snap 39 seq_num 300) 2024-03-19T11:31:58.025 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:58.025 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:58.039 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-19T11:31:58.060 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:58.067+0000 7f6a7f9da640 -1 osd.1 136 handle_osd_map injecting map crc failure 2024-03-19T11:31:58.061 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:31:58.067+0000 7f70c6fdf640 -1 osd.0 136 handle_osd_map injecting map crc failure 2024-03-19T11:31:58.061 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:58.069+0000 7f143d8b8640 -1 osd.5 136 handle_osd_map injecting map crc failure 2024-03-19T11:31:58.062 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-03-19T11:31:58.073 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:58.081+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:58.166 INFO:tasks.rados.rados.0.smithi038.stdout:1745: finishing write tid 2 to smithi03829856-38 2024-03-19T11:31:58.166 INFO:tasks.rados.rados.0.smithi038.stdout:1745: finishing write tid 3 to smithi03829856-38 2024-03-19T11:31:58.166 INFO:tasks.rados.rados.0.smithi038.stdout:1739: done (7 left) 2024-03-19T11:31:58.166 INFO:tasks.rados.rados.0.smithi038.stdout:1740: done (6 left) 2024-03-19T11:31:58.166 INFO:tasks.rados.rados.0.smithi038.stdout:1741: done (5 left) 2024-03-19T11:31:58.166 INFO:tasks.rados.rados.0.smithi038.stdout:1742: done (4 left) 2024-03-19T11:31:58.166 INFO:tasks.rados.rados.0.smithi038.stdout:1743: done (3 left) 2024-03-19T11:31:58.166 INFO:tasks.rados.rados.0.smithi038.stdout:1744: done (2 left) 2024-03-19T11:31:58.166 INFO:tasks.rados.rados.0.smithi038.stdout:1747: read oid 29 snap 21 2024-03-19T11:31:58.166 INFO:tasks.rados.rados.0.smithi038.stdout:1747: expect (ObjNum 165 snap 17 seq_num 165) 2024-03-19T11:31:58.167 INFO:tasks.rados.rados.0.smithi038.stdout:1748: rollback oid 3 current snap is 40 2024-03-19T11:31:58.167 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 3 to 28 2024-03-19T11:31:58.167 INFO:tasks.rados.rados.0.smithi038.stdout:1749: read oid 24 snap -1 2024-03-19T11:31:58.167 INFO:tasks.rados.rados.0.smithi038.stdout:1749: expect (ObjNum 265 snap 34 seq_num 265) 2024-03-19T11:31:58.167 INFO:tasks.rados.rados.0.smithi038.stdout:1750: read oid 11 snap -1 2024-03-19T11:31:58.167 INFO:tasks.rados.rados.0.smithi038.stdout:1750: expect deleted 2024-03-19T11:31:58.167 INFO:tasks.rados.rados.0.smithi038.stdout:1751: snap_remove snap 24 2024-03-19T11:31:58.173 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:58.183+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:58.192 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:58.192 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:58.206 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-19T11:31:58.274 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:58.281+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:58.338 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:58.338 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:58.352 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-19T11:31:58.375 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:58.383+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:58.476 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:58.483+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:58.520 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:58.521 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:58.534 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-19T11:31:58.576 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:58.583+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:58.650 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:58.650 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:58.664 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-19T11:31:58.712 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:58.721+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:58.779 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:58.785+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:31:58.781 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:58.781 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:58.795 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-19T11:31:58.880 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:31:58.887+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:31:58.959 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:58.959 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:58.973 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-19T11:31:58.981 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:58.991+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:59.003 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-19T11:31:59.061 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:59.069+0000 7f143b8b4640 -1 osd.5 137 handle_osd_map injecting map crc failure 2024-03-19T11:31:59.065 INFO:tasks.rados.rados.0.smithi038.stdout:1751: done (6 left) 2024-03-19T11:31:59.065 INFO:tasks.rados.rados.0.smithi038.stdout:1752: read oid 50 snap -1 2024-03-19T11:31:59.065 INFO:tasks.rados.rados.0.smithi038.stdout:1752: expect (ObjNum 260 snap 33 seq_num 260) 2024-03-19T11:31:59.067 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:31:59.073+0000 7f923da1c640 -1 osd.7 137 handle_osd_map injecting map crc failure 2024-03-19T11:31:59.067 INFO:tasks.rados.rados.0.smithi038.stdout:1753: read oid 45 snap 21 2024-03-19T11:31:59.068 INFO:tasks.rados.rados.0.smithi038.stdout:1753: expect (ObjNum 148 snap 15 seq_num 148) 2024-03-19T11:31:59.068 INFO:tasks.rados.rados.0.smithi038.stdout:1754: write oid 32 current snap is 40 2024-03-19T11:31:59.068 INFO:tasks.rados.rados.0.smithi038.stdout:1754: seq_num 323 ranges {628661=792253,2184625=421778,3403509=25152} 2024-03-19T11:31:59.074 INFO:tasks.rados.rados.0.smithi038.stdout:1754: writing smithi03829856-32 from 628661 to 1420914 tid 1 2024-03-19T11:31:59.082 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:59.091+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:31:59.082 INFO:tasks.rados.rados.0.smithi038.stdout:1754: writing smithi03829856-32 from 2184625 to 2606403 tid 2 2024-03-19T11:31:59.088 INFO:tasks.rados.rados.0.smithi038.stdout:1754: writing smithi03829856-32 from 3403509 to 3428661 tid 3 2024-03-19T11:31:59.088 INFO:tasks.rados.rados.0.smithi038.stdout:1745: finishing write tid 4 to smithi03829856-38 2024-03-19T11:31:59.088 INFO:tasks.rados.rados.0.smithi038.stdout:1745: finishing write tid 5 to smithi03829856-38 2024-03-19T11:31:59.088 INFO:tasks.rados.rados.0.smithi038.stdout:1745: oid 38 updating version 0 to 193 2024-03-19T11:31:59.088 INFO:tasks.rados.rados.0.smithi038.stdout:1745: oid 38 updating version 193 to 195 2024-03-19T11:31:59.089 INFO:tasks.rados.rados.0.smithi038.stdout:1745: oid 38 updating version 195 to 196 2024-03-19T11:31:59.089 INFO:tasks.rados.rados.0.smithi038.stdout:1745: oid 38 version 196 is already newer than 194 2024-03-19T11:31:59.089 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 38 v 196 (ObjNum 322 snap 40 seq_num 322) dirty exists 2024-03-19T11:31:59.089 INFO:tasks.rados.rados.0.smithi038.stdout:1745: left oid 38 (ObjNum 322 snap 40 seq_num 322) 2024-03-19T11:31:59.089 INFO:tasks.rados.rados.0.smithi038.stdout:1746: expect (ObjNum 295 snap 39 seq_num 295) 2024-03-19T11:31:59.106 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:59.106 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:59.122 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-19T11:31:59.179 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:31:59.179 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:31:59.182 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:59.191+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:59.195 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:31:59.196 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:31:59.211 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:31:59.212 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:31:59.229 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:31:59.229 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:31:59.239 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:31:59.239 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:31:59.249 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:31:59.250 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:31:59.254 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-19T11:31:59.263 INFO:tasks.rados.rados.0.smithi038.stdout:1748: finishing rollback tid 0 to smithi03829856-3 2024-03-19T11:31:59.263 INFO:tasks.rados.rados.0.smithi038.stdout:1748: finishing rollback tid 1 to smithi03829856-3 2024-03-19T11:31:59.263 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 3 v 141 (ObjNum 97 snap 0 seq_num 1) dirty dne 2024-03-19T11:31:59.263 INFO:tasks.rados.rados.0.smithi038.stdout:1749: expect (ObjNum 265 snap 34 seq_num 265) 2024-03-19T11:31:59.268 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:31:59.268 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:31:59.283 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:31:59.291+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:31:59.287 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:31:59.287 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:31:59.306 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:31:59.306 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:31:59.378 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:59.378 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:31:59.378 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:31:59.378 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:31:59.378 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:59.379 INFO:tasks.rados.rados.0.smithi038.stdout:1752: expect (ObjNum 260 snap 33 seq_num 260) 2024-03-19T11:31:59.384 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:59.393+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:59.392 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-19T11:31:59.482 INFO:tasks.rados.rados.0.smithi038.stdout:1747: expect (ObjNum 165 snap 17 seq_num 165) 2024-03-19T11:31:59.484 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:59.493+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:59.565 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:59.565 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:31:59.565 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:31:59.565 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:31:59.565 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.565 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:30 4.5 4:ac9cd3a3:::smithi03829856-8:head [write 1776668~340981] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:31:59.565 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:04.777922+0000", 2024-03-19T11:31:59.565 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 114.794770333, 2024-03-19T11:31:59.565 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015469894, 2024-03-19T11:31:59.565 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:59.565 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:59.565 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:59.566 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:59.566 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:59.566 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 30 2024-03-19T11:31:59.566 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.566 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:59.566 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.566 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:59.566 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.777922+0000", 2024-03-19T11:31:59.566 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:59.566 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.566 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.566 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:59.566 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.777922+0000", 2024-03-19T11:31:59.567 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:59.567 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.567 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.567 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:59.567 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.777924+0000", 2024-03-19T11:31:59.567 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3999999999999999e-06 2024-03-19T11:31:59.567 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.567 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.567 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:59.567 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778461+0000", 2024-03-19T11:31:59.567 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00053698299999999999 2024-03-19T11:31:59.567 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.567 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.567 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:59.568 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778487+0000", 2024-03-19T11:31:59.568 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5947000000000001e-05 2024-03-19T11:31:59.568 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.568 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.568 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:59.568 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778502+0000", 2024-03-19T11:31:59.568 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4256e-05 2024-03-19T11:31:59.568 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.568 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.568 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:59.568 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.781978+0000", 2024-03-19T11:31:59.568 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00347693 2024-03-19T11:31:59.568 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.568 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.568 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:59.569 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.782196+0000", 2024-03-19T11:31:59.569 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021726499999999999 2024-03-19T11:31:59.569 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.569 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.569 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:59.569 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.782572+0000", 2024-03-19T11:31:59.569 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00037635000000000001 2024-03-19T11:31:59.569 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.569 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.569 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:59.569 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.786960+0000", 2024-03-19T11:31:59.569 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0043882130000000002 2024-03-19T11:31:59.569 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.569 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.570 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:59.570 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793113+0000", 2024-03-19T11:31:59.570 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061530760000000004 2024-03-19T11:31:59.570 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.570 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.570 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:59.570 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793276+0000", 2024-03-19T11:31:59.570 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000162652 2024-03-19T11:31:59.570 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.570 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.570 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:59.570 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793392+0000", 2024-03-19T11:31:59.570 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115822 2024-03-19T11:31:59.571 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:59.571 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:59.571 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:59.571 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.571 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.571 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:32 4.5 4:ac9cd3a3:::smithi03829856-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e28)", 2024-03-19T11:31:59.571 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:04.778582+0000", 2024-03-19T11:31:59.571 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 114.794110684, 2024-03-19T11:31:59.571 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015761827999999999, 2024-03-19T11:31:59.571 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:59.571 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:59.571 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:59.571 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:59.571 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:59.572 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 32 2024-03-19T11:31:59.572 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.572 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:59.572 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.572 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:59.572 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778582+0000", 2024-03-19T11:31:59.572 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:59.572 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.572 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.572 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:59.572 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778582+0000", 2024-03-19T11:31:59.572 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:59.572 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.572 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.573 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:59.573 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778583+0000", 2024-03-19T11:31:59.573 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.026e-06 2024-03-19T11:31:59.573 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.573 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.573 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:59.573 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778587+0000", 2024-03-19T11:31:59.573 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.7870000000000001e-06 2024-03-19T11:31:59.573 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.573 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.573 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:59.573 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778599+0000", 2024-03-19T11:31:59.573 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1623e-05 2024-03-19T11:31:59.573 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.574 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.574 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:59.574 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.778608+0000", 2024-03-19T11:31:59.574 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7399999999999993e-06 2024-03-19T11:31:59.574 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.574 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.574 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:59.574 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.787009+0000", 2024-03-19T11:31:59.574 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0084010490000000007 2024-03-19T11:31:59.574 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.574 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.574 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-03-19T11:31:59.574 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.787202+0000", 2024-03-19T11:31:59.574 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019309500000000001 2024-03-19T11:31:59.575 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.575 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.575 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:59.575 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.793832+0000", 2024-03-19T11:31:59.575 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.006630052 2024-03-19T11:31:59.575 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.575 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.575 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:59.575 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.794048+0000", 2024-03-19T11:31:59.575 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000215505 2024-03-19T11:31:59.575 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.575 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.575 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:59.575 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:04.794343+0000", 2024-03-19T11:31:59.575 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029595100000000003 2024-03-19T11:31:59.576 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:59.576 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:59.576 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:59.576 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.576 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.576 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:891 4.5 4:af9d0d56:::smithi03829856-20:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:31:59.576 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:09.286410+0000", 2024-03-19T11:31:59.576 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 110.286282352, 2024-03-19T11:31:59.576 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035191405000000002, 2024-03-19T11:31:59.576 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:59.576 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:59.576 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:59.576 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:59.576 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:59.577 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 891 2024-03-19T11:31:59.577 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.577 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:59.577 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.577 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:59.577 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286410+0000", 2024-03-19T11:31:59.577 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:59.577 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.577 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.577 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:59.577 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286410+0000", 2024-03-19T11:31:59.577 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:59.577 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.577 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.577 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:59.578 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286413+0000", 2024-03-19T11:31:59.578 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2660000000000002e-06 2024-03-19T11:31:59.578 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.578 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.578 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:59.578 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286426+0000", 2024-03-19T11:31:59.578 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.258e-05 2024-03-19T11:31:59.578 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.578 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.578 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:59.578 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286460+0000", 2024-03-19T11:31:59.578 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3943999999999999e-05 2024-03-19T11:31:59.578 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.578 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.579 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:59.579 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286477+0000", 2024-03-19T11:31:59.579 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6799999999999998e-05 2024-03-19T11:31:59.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.579 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:59.579 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286605+0000", 2024-03-19T11:31:59.579 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000128892 2024-03-19T11:31:59.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.579 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.579 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:59.579 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286769+0000", 2024-03-19T11:31:59.579 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016359799999999999 2024-03-19T11:31:59.579 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.580 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.580 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:59.580 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.321601+0000", 2024-03-19T11:31:59.580 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.034832324999999997 2024-03-19T11:31:59.580 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:59.580 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:59.580 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:59.580 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.580 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.580 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1015 4.5 4:af9d0d56:::smithi03829856-20:head [tier-promote] snapc 6=[6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:31:59.580 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:12.761855+0000", 2024-03-19T11:31:59.580 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 106.810836826, 2024-03-19T11:31:59.580 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026760430000000002, 2024-03-19T11:31:59.580 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:59.581 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:59.581 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:59.581 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:59.581 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:59.581 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1015 2024-03-19T11:31:59.581 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.581 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:59.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.581 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:59.581 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761855+0000", 2024-03-19T11:31:59.581 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:59.581 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.581 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.581 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:59.581 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761855+0000", 2024-03-19T11:31:59.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:59.582 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.582 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:59.582 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761858+0000", 2024-03-19T11:31:59.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8140000000000001e-06 2024-03-19T11:31:59.582 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.582 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:59.582 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761872+0000", 2024-03-19T11:31:59.582 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3813e-05 2024-03-19T11:31:59.582 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.582 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.582 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:59.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761905+0000", 2024-03-19T11:31:59.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2642999999999999e-05 2024-03-19T11:31:59.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.583 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:59.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761925+0000", 2024-03-19T11:31:59.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0489e-05 2024-03-19T11:31:59.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.583 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:59.583 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.762070+0000", 2024-03-19T11:31:59.583 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000145153 2024-03-19T11:31:59.583 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.583 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.584 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:59.584 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.762365+0000", 2024-03-19T11:31:59.584 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029482100000000002 2024-03-19T11:31:59.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.584 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:59.584 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.788616+0000", 2024-03-19T11:31:59.584 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026250697 2024-03-19T11:31:59.584 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:59.584 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:59.584 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:59.584 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.584 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.584 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1121 4.5 4:aa3303a5:::smithi03829856-17:head [tier-promote] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:31:59.584 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:14.164281+0000", 2024-03-19T11:31:59.585 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 105.408411581, 2024-03-19T11:31:59.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023896435000000001, 2024-03-19T11:31:59.585 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:59.585 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:59.585 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:59.585 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:59.585 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:59.585 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1121 2024-03-19T11:31:59.585 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.585 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:59.585 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.585 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:59.585 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164281+0000", 2024-03-19T11:31:59.585 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:59.586 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.586 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:59.586 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164281+0000", 2024-03-19T11:31:59.586 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:59.586 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.586 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:59.586 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164283+0000", 2024-03-19T11:31:59.586 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8500000000000001e-06 2024-03-19T11:31:59.586 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.586 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.586 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:59.586 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164291+0000", 2024-03-19T11:31:59.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.14e-06 2024-03-19T11:31:59.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.587 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:59.587 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164313+0000", 2024-03-19T11:31:59.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.207e-05 2024-03-19T11:31:59.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.587 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:59.587 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164328+0000", 2024-03-19T11:31:59.587 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5403000000000001e-05 2024-03-19T11:31:59.587 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.587 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.587 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:59.587 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.167981+0000", 2024-03-19T11:31:59.588 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036529879999999998 2024-03-19T11:31:59.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.588 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.588 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:59.588 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.168274+0000", 2024-03-19T11:31:59.588 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029313100000000002 2024-03-19T11:31:59.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.588 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.588 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:59.588 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.188177+0000", 2024-03-19T11:31:59.588 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019902853000000002 2024-03-19T11:31:59.588 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:59.588 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:59.588 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:59.588 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.589 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1302 4.15 4:aa3303a5:::smithi03829856-17:head [write 413472~620140,stat] snapc b=[b,a,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-03-19T11:31:59.589 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:22.966426+0000", 2024-03-19T11:31:59.589 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 96.606265848000007, 2024-03-19T11:31:59.589 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018043513000000001, 2024-03-19T11:31:59.589 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:59.589 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:59.589 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:59.589 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:59.589 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:59.589 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1302 2024-03-19T11:31:59.589 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.589 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:59.589 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.590 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:59.590 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.966426+0000", 2024-03-19T11:31:59.590 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:59.590 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.590 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.590 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:59.590 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.966426+0000", 2024-03-19T11:31:59.590 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:59.590 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.590 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.590 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:59.590 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.966429+0000", 2024-03-19T11:31:59.590 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3829999999999999e-06 2024-03-19T11:31:59.590 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.590 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.591 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:59.591 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972117+0000", 2024-03-19T11:31:59.591 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0056885750000000004 2024-03-19T11:31:59.591 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.591 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:59.591 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972156+0000", 2024-03-19T11:31:59.591 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8878000000000003e-05 2024-03-19T11:31:59.591 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.591 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:59.591 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972184+0000", 2024-03-19T11:31:59.591 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7509999999999999e-05 2024-03-19T11:31:59.591 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.591 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.592 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:59.592 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972309+0000", 2024-03-19T11:31:59.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012525899999999999 2024-03-19T11:31:59.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.592 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.592 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:59.592 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972566+0000", 2024-03-19T11:31:59.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025740799999999999 2024-03-19T11:31:59.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.592 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.592 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:59.592 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972899+0000", 2024-03-19T11:31:59.592 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033238799999999998 2024-03-19T11:31:59.592 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.593 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.593 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:59.593 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984088+0000", 2024-03-19T11:31:59.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011189466 2024-03-19T11:31:59.593 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.593 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.593 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:59.593 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984138+0000", 2024-03-19T11:31:59.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9666999999999997e-05 2024-03-19T11:31:59.593 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.593 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.593 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:59.593 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984329+0000", 2024-03-19T11:31:59.593 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019086799999999999 2024-03-19T11:31:59.594 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.594 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.594 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:59.594 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984470+0000", 2024-03-19T11:31:59.594 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000141111 2024-03-19T11:31:59.594 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:59.594 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:59.594 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:59.594 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.594 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.594 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.1 50/49 [PushOp(4:866f3dcc:::smithi03829856-31:7, version: 38'44, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@38'44, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 45'62, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@45'62, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 46'68, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@46'68, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:7, version: 47'71, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@47'71, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 47'72, data_included: [440350~816629,1309629~231947], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@47'72, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:59.594 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.959457+0000", 2024-03-19T11:31:59.594 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 91.613235431000007, 2024-03-19T11:31:59.594 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018416666000000002, 2024-03-19T11:31:59.595 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:59.595 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:59.595 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:59.595 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.595 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:59.595 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.959457+0000", 2024-03-19T11:31:59.595 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:59.595 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.595 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.595 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:59.595 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.959457+0000", 2024-03-19T11:31:59.595 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:59.595 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.595 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.595 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:59.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.959458+0000", 2024-03-19T11:31:59.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4899999999999999e-06 2024-03-19T11:31:59.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:59.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.962822+0000", 2024-03-19T11:31:59.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0033640620000000001 2024-03-19T11:31:59.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.596 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:59.596 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.962893+0000", 2024-03-19T11:31:59.596 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1060000000000001e-05 2024-03-19T11:31:59.596 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.596 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.597 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:59.597 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.962977+0000", 2024-03-19T11:31:59.597 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3311000000000002e-05 2024-03-19T11:31:59.597 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.597 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:59.597 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963208+0000", 2024-03-19T11:31:59.597 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023121200000000001 2024-03-19T11:31:59.597 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.597 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:59.597 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963245+0000", 2024-03-19T11:31:59.597 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6844999999999997e-05 2024-03-19T11:31:59.597 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.597 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.598 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:59.598 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.977874+0000", 2024-03-19T11:31:59.598 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014628686 2024-03-19T11:31:59.598 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:59.598 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:59.598 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:59.598 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.598 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.598 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2284 4.15 4:afa9f208:::smithi03829856-35:head [read 1314816~702976] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-03-19T11:31:59.598 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:53.596775+0000", 2024-03-19T11:31:59.598 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 65.975917344999999, 2024-03-19T11:31:59.598 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.47260528299999999, 2024-03-19T11:31:59.598 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:59.599 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:59.599 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:59.599 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:59.599 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:59.599 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2284 2024-03-19T11:31:59.599 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.599 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:59.599 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.599 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:59.599 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596775+0000", 2024-03-19T11:31:59.599 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:59.599 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.599 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.600 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:59.600 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596775+0000", 2024-03-19T11:31:59.600 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:59.600 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.600 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.600 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:59.600 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596778+0000", 2024-03-19T11:31:59.600 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.174e-06 2024-03-19T11:31:59.600 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.600 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.600 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:59.600 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596794+0000", 2024-03-19T11:31:59.600 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5372000000000001e-05 2024-03-19T11:31:59.600 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.600 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.601 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:59.601 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596828+0000", 2024-03-19T11:31:59.601 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4107999999999999e-05 2024-03-19T11:31:59.601 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.601 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.601 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:59.601 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596844+0000", 2024-03-19T11:31:59.601 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6296000000000001e-05 2024-03-19T11:31:59.601 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.601 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.601 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:59.601 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596968+0000", 2024-03-19T11:31:59.601 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012383699999999999 2024-03-19T11:31:59.601 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.602 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.602 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:31:59.602 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596988+0000", 2024-03-19T11:31:59.602 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0645e-05 2024-03-19T11:31:59.602 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.602 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.602 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:59.602 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.068916+0000", 2024-03-19T11:31:59.602 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.47192744800000003 2024-03-19T11:31:59.602 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.602 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.602 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:59.602 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.069015+0000", 2024-03-19T11:31:59.602 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9500999999999995e-05 2024-03-19T11:31:59.603 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.603 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.603 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:59.603 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.069380+0000", 2024-03-19T11:31:59.603 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00036490200000000001 2024-03-19T11:31:59.603 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:59.603 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:59.603 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:59.603 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.603 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.603 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2286 4.15 4:afa9f208:::smithi03829856-35:head [assert-version v108,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e75)", 2024-03-19T11:31:59.603 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.084826+0000", 2024-03-19T11:31:59.603 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 65.487865978000002, 2024-03-19T11:31:59.603 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.055799556, 2024-03-19T11:31:59.604 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:59.604 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:59.604 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:59.604 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:59.604 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:59.604 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2286 2024-03-19T11:31:59.604 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.604 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:59.604 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.604 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:59.604 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084826+0000", 2024-03-19T11:31:59.604 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:59.604 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.604 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.605 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:59.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084826+0000", 2024-03-19T11:31:59.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:59.605 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.605 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:59.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084828+0000", 2024-03-19T11:31:59.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1409999999999999e-06 2024-03-19T11:31:59.605 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.605 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:59.605 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084838+0000", 2024-03-19T11:31:59.605 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3740000000000004e-06 2024-03-19T11:31:59.605 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.605 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.606 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:59.606 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084859+0000", 2024-03-19T11:31:59.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1644000000000001e-05 2024-03-19T11:31:59.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.606 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:59.606 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084871+0000", 2024-03-19T11:31:59.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1127e-05 2024-03-19T11:31:59.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.606 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.606 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:59.606 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084961+0000", 2024-03-19T11:31:59.606 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0371999999999995e-05 2024-03-19T11:31:59.606 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:59.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.085130+0000", 2024-03-19T11:31:59.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016884800000000001 2024-03-19T11:31:59.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5,7", 2024-03-19T11:31:59.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.089309+0000", 2024-03-19T11:31:59.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041788650000000004 2024-03-19T11:31:59.607 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.607 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.607 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:59.607 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.090362+0000", 2024-03-19T11:31:59.607 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001053362 2024-03-19T11:31:59.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.608 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:59.608 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.090511+0000", 2024-03-19T11:31:59.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014887200000000001 2024-03-19T11:31:59.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.608 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:59.608 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.140479+0000", 2024-03-19T11:31:59.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.049968006000000002 2024-03-19T11:31:59.608 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.608 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.608 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:59.608 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.140534+0000", 2024-03-19T11:31:59.608 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4917000000000003e-05 2024-03-19T11:31:59.609 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.609 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:59.609 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.140626+0000", 2024-03-19T11:31:59.609 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2027999999999994e-05 2024-03-19T11:31:59.609 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:59.609 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:59.609 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:59.609 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.609 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.609 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1d58db11:::smithi03829856-18:6, version: 55'80, data_included: [759683~639358,1904434~409218], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:6@55'80, size: 3159683, copy_subset: [(0, 3159683)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2313652, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1d58db11:::smithi03829856-18:b, version: 58'84, data_included: [443718~955323,1564138~93253], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:b@58'84, size: 3445402, copy_subset: [(0, 3445402)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1657391, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:10, version: 66'95, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:10@66'95, size: 3764735, copy_subset: [(0, 3764735)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:12, version: 66'97, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:12@66'97, size: 2504243, copy_subset: [(0, 2504243)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:c, version: 70'102, data_included: [416807~576167,1542129~472409], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:c@70'102, size: 2857157, copy_subset: [(0, 2857157)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2014538, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:59.609 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.274749+0000", 2024-03-19T11:31:59.609 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 65.297943047000004, 2024-03-19T11:31:59.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017652601, 2024-03-19T11:31:59.610 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:59.610 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:59.610 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:59.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.610 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:59.610 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.274749+0000", 2024-03-19T11:31:59.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:59.610 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.610 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.610 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:59.610 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.274749+0000", 2024-03-19T11:31:59.610 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:59.610 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:59.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.274751+0000", 2024-03-19T11:31:59.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8500000000000001e-06 2024-03-19T11:31:59.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:59.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.278426+0000", 2024-03-19T11:31:59.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036747559999999999 2024-03-19T11:31:59.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.611 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.611 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:59.611 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.278470+0000", 2024-03-19T11:31:59.611 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4119000000000002e-05 2024-03-19T11:31:59.611 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:59.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.278506+0000", 2024-03-19T11:31:59.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6066000000000003e-05 2024-03-19T11:31:59.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:59.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.278709+0000", 2024-03-19T11:31:59.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000202711 2024-03-19T11:31:59.612 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.612 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.612 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:59.612 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.278731+0000", 2024-03-19T11:31:59.612 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2657e-05 2024-03-19T11:31:59.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.613 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:59.613 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.292402+0000", 2024-03-19T11:31:59.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013670442 2024-03-19T11:31:59.613 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:59.613 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:59.613 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:59.613 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.613 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.613 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1dabc0f1:::smithi03829856-1:head, version: 70'103, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:head@70'103, size: 2621892, copy_subset: [(0, 2621892)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:31:59.613 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.280012+0000", 2024-03-19T11:31:59.613 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 65.292680402000002, 2024-03-19T11:31:59.613 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018569209, 2024-03-19T11:31:59.614 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:59.614 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:59.614 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:59.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.614 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:59.614 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280012+0000", 2024-03-19T11:31:59.614 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:59.614 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.614 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:59.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280012+0000", 2024-03-19T11:31:59.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:59.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:59.615 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280013+0000", 2024-03-19T11:31:59.615 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9899999999999999e-07 2024-03-19T11:31:59.615 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.615 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.615 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:59.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280584+0000", 2024-03-19T11:31:59.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057090299999999999 2024-03-19T11:31:59.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:59.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280612+0000", 2024-03-19T11:31:59.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8442999999999999e-05 2024-03-19T11:31:59.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.616 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:59.616 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280631+0000", 2024-03-19T11:31:59.616 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8896999999999999e-05 2024-03-19T11:31:59.616 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.616 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.617 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:59.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.295041+0000", 2024-03-19T11:31:59.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014410127999999999 2024-03-19T11:31:59.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.617 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:59.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.295050+0000", 2024-03-19T11:31:59.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2350000000000005e-06 2024-03-19T11:31:59.617 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.617 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.617 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:59.617 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.298581+0000", 2024-03-19T11:31:59.617 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035308039999999998 2024-03-19T11:31:59.617 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:59.618 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:59.618 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:59.618 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.618 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.618 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.18 e75,74 4:1d58db11:::smithi03829856-18:c,70'104)", 2024-03-19T11:31:59.618 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.280680+0000", 2024-03-19T11:31:59.618 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 65.292011927000004, 2024-03-19T11:31:59.618 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018106229000000001, 2024-03-19T11:31:59.618 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:59.618 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:59.619 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:59.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:59.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280680+0000", 2024-03-19T11:31:59.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:59.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:59.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280680+0000", 2024-03-19T11:31:59.619 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:59.619 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.619 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.619 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:59.619 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280681+0000", 2024-03-19T11:31:59.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2099999999999995e-07 2024-03-19T11:31:59.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:59.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280686+0000", 2024-03-19T11:31:59.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2240000000000003e-06 2024-03-19T11:31:59.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:59.620 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280698+0000", 2024-03-19T11:31:59.620 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1698999999999999e-05 2024-03-19T11:31:59.620 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.620 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.620 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:59.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280705+0000", 2024-03-19T11:31:59.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4350000000000002e-06 2024-03-19T11:31:59.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:59.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.298643+0000", 2024-03-19T11:31:59.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017938352000000001 2024-03-19T11:31:59.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.621 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.621 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:59.621 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.298656+0000", 2024-03-19T11:31:59.621 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3084999999999999e-05 2024-03-19T11:31:59.621 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.622 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.622 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:59.622 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.298787+0000", 2024-03-19T11:31:59.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013061299999999999 2024-03-19T11:31:59.622 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:59.622 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:59.622 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:59.622 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.622 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.622 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.5.0:143 4.18 e75/74 4:1d58db11:::smithi03829856-18:head v 75'106, mlcod=0'0)", 2024-03-19T11:31:59.622 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.282327+0000", 2024-03-19T11:31:59.622 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 65.290365555999998, 2024-03-19T11:31:59.622 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016753244, 2024-03-19T11:31:59.622 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:59.623 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:59.623 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:59.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:59.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.282327+0000", 2024-03-19T11:31:59.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:59.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:59.623 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.282327+0000", 2024-03-19T11:31:59.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:59.623 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.623 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.623 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:59.624 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.282328+0000", 2024-03-19T11:31:59.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.488e-06 2024-03-19T11:31:59.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:59.624 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.283127+0000", 2024-03-19T11:31:59.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00079853699999999999 2024-03-19T11:31:59.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.624 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:59.624 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.283152+0000", 2024-03-19T11:31:59.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5215000000000002e-05 2024-03-19T11:31:59.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:59.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.283167+0000", 2024-03-19T11:31:59.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4635e-05 2024-03-19T11:31:59.625 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:59.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.292633+0000", 2024-03-19T11:31:59.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0094664339999999993 2024-03-19T11:31:59.625 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.626 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:59.626 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.292655+0000", 2024-03-19T11:31:59.626 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1934999999999999e-05 2024-03-19T11:31:59.626 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.626 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:59.626 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.298981+0000", 2024-03-19T11:31:59.626 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0063255129999999996 2024-03-19T11:31:59.626 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.626 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:59.626 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.299080+0000", 2024-03-19T11:31:59.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9487000000000005e-05 2024-03-19T11:31:59.627 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:59.627 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:59.627 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:59.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.627 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2441 4.5 4:a38b9453:::smithi03829856-37:head [delete] snapc 17=[17,16,15,13,12,b,6] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e84)", 2024-03-19T11:31:59.627 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:03.060663+0000", 2024-03-19T11:31:59.627 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 56.512029712, 2024-03-19T11:31:59.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10917726, 2024-03-19T11:31:59.627 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:59.627 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:59.627 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:59.627 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:59.628 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:59.628 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2441 2024-03-19T11:31:59.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.628 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:59.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:59.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060663+0000", 2024-03-19T11:31:59.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:59.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:59.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060663+0000", 2024-03-19T11:31:59.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:59.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.629 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.629 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:59.629 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060666+0000", 2024-03-19T11:31:59.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.05e-06 2024-03-19T11:31:59.629 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.629 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.629 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:59.629 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060680+0000", 2024-03-19T11:31:59.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4276e-05 2024-03-19T11:31:59.629 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.629 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.629 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:59.629 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060714+0000", 2024-03-19T11:31:59.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4325000000000001e-05 2024-03-19T11:31:59.629 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.630 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:59.630 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060737+0000", 2024-03-19T11:31:59.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2959000000000002e-05 2024-03-19T11:31:59.630 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.630 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:59.630 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060895+0000", 2024-03-19T11:31:59.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000157787 2024-03-19T11:31:59.630 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.630 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:31:59.630 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060923+0000", 2024-03-19T11:31:59.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8019000000000001e-05 2024-03-19T11:31:59.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.631 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:59.631 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.066222+0000", 2024-03-19T11:31:59.631 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052989270000000002 2024-03-19T11:31:59.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.631 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-03-19T11:31:59.631 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.070153+0000", 2024-03-19T11:31:59.631 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039315119999999999 2024-03-19T11:31:59.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.631 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:59.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.147930+0000", 2024-03-19T11:31:59.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.077776459000000006 2024-03-19T11:31:59.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:59.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.148079+0000", 2024-03-19T11:31:59.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014888 2024-03-19T11:31:59.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:31:59.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.148370+0000", 2024-03-19T11:31:59.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029130999999999997 2024-03-19T11:31:59.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:31:59.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.152531+0000", 2024-03-19T11:31:59.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041605399999999999 2024-03-19T11:31:59.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:31:59.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.169762+0000", 2024-03-19T11:31:59.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017231483999999998 2024-03-19T11:31:59.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:59.633 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.169807+0000", 2024-03-19T11:31:59.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5161999999999999e-05 2024-03-19T11:31:59.633 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.633 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:59.634 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.169840+0000", 2024-03-19T11:31:59.634 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2570000000000002e-05 2024-03-19T11:31:59.634 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:59.634 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:59.634 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:59.634 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.634 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.634 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPushReply(4.5 84/83 [PushReplyOp(4:a38b9453:::smithi03829856-37:head)])", 2024-03-19T11:31:59.634 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:03.084896+0000", 2024-03-19T11:31:59.634 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 56.487796578000001, 2024-03-19T11:31:59.634 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.02071104, 2024-03-19T11:31:59.634 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:59.634 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-19T11:31:59.634 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:59.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.635 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:59.635 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.084896+0000", 2024-03-19T11:31:59.635 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:59.635 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.635 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:59.635 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.084896+0000", 2024-03-19T11:31:59.635 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:59.635 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.635 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:59.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.084897+0000", 2024-03-19T11:31:59.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.686e-06 2024-03-19T11:31:59.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:59.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.084910+0000", 2024-03-19T11:31:59.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2948e-05 2024-03-19T11:31:59.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:59.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.084936+0000", 2024-03-19T11:31:59.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5982e-05 2024-03-19T11:31:59.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:59.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.084955+0000", 2024-03-19T11:31:59.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8416000000000001e-05 2024-03-19T11:31:59.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.637 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:59.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.102822+0000", 2024-03-19T11:31:59.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017866756000000001 2024-03-19T11:31:59.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.637 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:59.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.105607+0000", 2024-03-19T11:31:59.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027852520000000002 2024-03-19T11:31:59.637 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:59.638 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:59.638 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:59.638 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.638 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.638 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.4.0:540 4.18 e111/86 4:1d58db11:::smithi03829856-18:head v 111'153, mlcod=108'150)", 2024-03-19T11:31:59.638 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:31.238185+0000", 2024-03-19T11:31:59.638 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 28.334507214999999, 2024-03-19T11:31:59.638 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021779390999999999, 2024-03-19T11:31:59.638 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:59.638 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:31:59.638 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:59.638 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.638 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:59.638 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.238185+0000", 2024-03-19T11:31:59.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:59.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.639 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:59.639 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.238185+0000", 2024-03-19T11:31:59.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:59.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.639 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:59.639 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.238188+0000", 2024-03-19T11:31:59.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4739999999999999e-06 2024-03-19T11:31:59.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.639 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:59.639 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.240837+0000", 2024-03-19T11:31:59.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026495189999999999 2024-03-19T11:31:59.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:59.640 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.240867+0000", 2024-03-19T11:31:59.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0363000000000001e-05 2024-03-19T11:31:59.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:59.640 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.240885+0000", 2024-03-19T11:31:59.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7153e-05 2024-03-19T11:31:59.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:59.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.241029+0000", 2024-03-19T11:31:59.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014455499999999999 2024-03-19T11:31:59.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.641 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:59.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.241069+0000", 2024-03-19T11:31:59.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9409000000000002e-05 2024-03-19T11:31:59.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.641 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:31:59.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.259833+0000", 2024-03-19T11:31:59.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018764306000000001 2024-03-19T11:31:59.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.642 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:59.642 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.259964+0000", 2024-03-19T11:31:59.642 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000131612 2024-03-19T11:31:59.642 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:59.642 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:59.642 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:59.642 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.642 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.642 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3282 4.5 4:a0a50d04:::smithi03829856-13:head [read 0~2201684,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e112)", 2024-03-19T11:31:59.642 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:32.240600+0000", 2024-03-19T11:31:59.642 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.332092011, 2024-03-19T11:31:59.642 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.84623566699999997, 2024-03-19T11:31:59.642 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:59.643 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:59.643 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:59.643 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:59.643 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:59.643 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3282 2024-03-19T11:31:59.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.643 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:59.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.643 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:59.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240600+0000", 2024-03-19T11:31:59.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:59.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.643 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:59.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240600+0000", 2024-03-19T11:31:59.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:59.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.644 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:59.644 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240602+0000", 2024-03-19T11:31:59.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0700000000000001e-06 2024-03-19T11:31:59.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.644 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:59.644 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240611+0000", 2024-03-19T11:31:59.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.969e-06 2024-03-19T11:31:59.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.644 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:59.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240630+0000", 2024-03-19T11:31:59.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8253e-05 2024-03-19T11:31:59.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:59.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240642+0000", 2024-03-19T11:31:59.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2666e-05 2024-03-19T11:31:59.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:59.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.241488+0000", 2024-03-19T11:31:59.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000845252 2024-03-19T11:31:59.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:31:59.646 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.241503+0000", 2024-03-19T11:31:59.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5797000000000001e-05 2024-03-19T11:31:59.646 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:59.646 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.083186+0000", 2024-03-19T11:31:59.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.84168299599999996 2024-03-19T11:31:59.646 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:59.646 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.083618+0000", 2024-03-19T11:31:59.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000432046 2024-03-19T11:31:59.646 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.647 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:59.647 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.086836+0000", 2024-03-19T11:31:59.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032176180000000002 2024-03-19T11:31:59.647 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:59.647 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:59.647 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:59.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.647 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3283 4.5 4:a0a50d04:::smithi03829856-13:head [sparse-read 0~1151947] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e112)", 2024-03-19T11:31:59.647 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:32.240669+0000", 2024-03-19T11:31:59.647 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.332023011, 2024-03-19T11:31:59.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.84791886900000002, 2024-03-19T11:31:59.647 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:59.647 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:59.648 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:59.648 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:59.648 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:59.648 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3283 2024-03-19T11:31:59.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.648 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:59.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.648 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:59.648 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240669+0000", 2024-03-19T11:31:59.648 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:59.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.648 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:59.648 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240669+0000", 2024-03-19T11:31:59.648 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:59.649 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:59.649 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240670+0000", 2024-03-19T11:31:59.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8199999999999998e-07 2024-03-19T11:31:59.649 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:59.649 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240675+0000", 2024-03-19T11:31:59.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0849999999999996e-06 2024-03-19T11:31:59.649 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:59.649 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240688+0000", 2024-03-19T11:31:59.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2901000000000001e-05 2024-03-19T11:31:59.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.650 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.650 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:59.650 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240696+0000", 2024-03-19T11:31:59.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2910000000000007e-06 2024-03-19T11:31:59.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.650 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.650 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:59.650 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.241573+0000", 2024-03-19T11:31:59.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00087639100000000002 2024-03-19T11:31:59.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.650 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.651 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:31:59.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.241587+0000", 2024-03-19T11:31:59.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4467999999999999e-05 2024-03-19T11:31:59.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.651 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:59.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.087901+0000", 2024-03-19T11:31:59.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.84631403500000002 2024-03-19T11:31:59.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.651 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:59.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.088089+0000", 2024-03-19T11:31:59.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000187171 2024-03-19T11:31:59.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.652 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.652 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:59.652 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.088588+0000", 2024-03-19T11:31:59.652 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00049964499999999995 2024-03-19T11:31:59.652 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:59.652 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:59.652 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:59.652 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.652 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.652 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3284 4.5 4:a0a50d04:::smithi03829856-13:head [sparse-read 0~1151947] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e112)", 2024-03-19T11:31:59.652 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:32.240710+0000", 2024-03-19T11:31:59.652 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 27.331982714999999, 2024-03-19T11:31:59.652 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.84860797899999996, 2024-03-19T11:31:59.652 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:59.653 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:59.653 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:59.653 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:59.653 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:59.653 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3284 2024-03-19T11:31:59.653 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.653 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:59.653 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.653 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:59.653 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240710+0000", 2024-03-19T11:31:59.653 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:59.653 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.653 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.653 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:59.653 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240710+0000", 2024-03-19T11:31:59.654 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:59.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.654 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:59.654 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240710+0000", 2024-03-19T11:31:59.654 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9599999999999999e-07 2024-03-19T11:31:59.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.654 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:59.654 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240714+0000", 2024-03-19T11:31:59.654 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8480000000000003e-06 2024-03-19T11:31:59.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.654 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:59.655 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240726+0000", 2024-03-19T11:31:59.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.186e-05 2024-03-19T11:31:59.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:59.655 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240736+0000", 2024-03-19T11:31:59.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0040999999999999e-05 2024-03-19T11:31:59.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:59.655 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.241633+0000", 2024-03-19T11:31:59.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00089736299999999998 2024-03-19T11:31:59.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:31:59.656 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.241645+0000", 2024-03-19T11:31:59.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1600000000000001e-05 2024-03-19T11:31:59.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.656 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:59.656 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.088667+0000", 2024-03-19T11:31:59.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.84702242800000005 2024-03-19T11:31:59.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.656 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:59.656 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.088837+0000", 2024-03-19T11:31:59.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000169949 2024-03-19T11:31:59.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.656 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:59.657 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.089318+0000", 2024-03-19T11:31:59.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00048019399999999998 2024-03-19T11:31:59.657 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:59.657 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:59.657 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:59.657 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.657 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.657 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3899 4.5 4:a38b9453:::smithi03829856-37:head [read 0~3966452,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e130)", 2024-03-19T11:31:59.657 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:51.081535+0000", 2024-03-19T11:31:59.657 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 8.4911574779999999, 2024-03-19T11:31:59.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017507140000000001, 2024-03-19T11:31:59.657 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:31:59.657 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:31:59.658 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:31:59.658 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:31:59.658 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:31:59.658 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3899 2024-03-19T11:31:59.658 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.658 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:31:59.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.658 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:31:59.658 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.081535+0000", 2024-03-19T11:31:59.658 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:59.658 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.658 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:31:59.658 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.081535+0000", 2024-03-19T11:31:59.658 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:31:59.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:31:59.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.081536+0000", 2024-03-19T11:31:59.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5999999999999999e-06 2024-03-19T11:31:59.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:31:59.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.081548+0000", 2024-03-19T11:31:59.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1892000000000001e-05 2024-03-19T11:31:59.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:31:59.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.081567+0000", 2024-03-19T11:31:59.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9023000000000001e-05 2024-03-19T11:31:59.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.660 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:31:59.660 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.081581+0000", 2024-03-19T11:31:59.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.364e-05 2024-03-19T11:31:59.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.660 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:59.660 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.081744+0000", 2024-03-19T11:31:59.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000163287 2024-03-19T11:31:59.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.660 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:31:59.661 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.081760+0000", 2024-03-19T11:31:59.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5892999999999998e-05 2024-03-19T11:31:59.661 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.661 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:31:59.661 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.095401+0000", 2024-03-19T11:31:59.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013640470999999999 2024-03-19T11:31:59.661 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.661 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:31:59.661 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.095542+0000", 2024-03-19T11:31:59.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000141213 2024-03-19T11:31:59.661 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:31:59.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:31:59.661 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:31:59.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.099042+0000", 2024-03-19T11:31:59.662 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035001210000000001 2024-03-19T11:31:59.662 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:59.662 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:59.662 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:59.662 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:31:59.662 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:31:59.662 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:59.664 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-19T11:31:59.664 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:31:59.673+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:31:59.684 INFO:tasks.rados.rados.0.smithi038.stdout:1754: finishing write tid 1 to smithi03829856-32 2024-03-19T11:31:59.684 INFO:tasks.rados.rados.0.smithi038.stdout:1753: expect (ObjNum 148 snap 15 seq_num 148) 2024-03-19T11:31:59.765 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:59.773+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:59.825 INFO:tasks.rados.rados.0.smithi038.stdout:1754: finishing write tid 2 to smithi03829856-32 2024-03-19T11:31:59.825 INFO:tasks.rados.rados.0.smithi038.stdout:1754: finishing write tid 3 to smithi03829856-32 2024-03-19T11:31:59.825 INFO:tasks.rados.rados.0.smithi038.stdout:1754: finishing write tid 4 to smithi03829856-32 2024-03-19T11:31:59.825 INFO:tasks.rados.rados.0.smithi038.stdout:1754: finishing write tid 5 to smithi03829856-32 2024-03-19T11:31:59.825 INFO:tasks.rados.rados.0.smithi038.stdout:1754: oid 32 updating version 0 to 161 2024-03-19T11:31:59.825 INFO:tasks.rados.rados.0.smithi038.stdout:1754: oid 32 updating version 161 to 162 2024-03-19T11:31:59.826 INFO:tasks.rados.rados.0.smithi038.stdout:1754: oid 32 updating version 162 to 163 2024-03-19T11:31:59.826 INFO:tasks.rados.rados.0.smithi038.stdout:1754: oid 32 version 163 is already newer than 160 2024-03-19T11:31:59.826 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 32 v 163 (ObjNum 323 snap 40 seq_num 323) dirty exists 2024-03-19T11:31:59.826 INFO:tasks.rados.rados.0.smithi038.stdout:1754: left oid 32 (ObjNum 323 snap 40 seq_num 323) 2024-03-19T11:31:59.826 INFO:tasks.rados.rados.0.smithi038.stdout:1745: done (8 left) 2024-03-19T11:31:59.826 INFO:tasks.rados.rados.0.smithi038.stdout:1746: done (7 left) 2024-03-19T11:31:59.826 INFO:tasks.rados.rados.0.smithi038.stdout:1747: done (6 left) 2024-03-19T11:31:59.826 INFO:tasks.rados.rados.0.smithi038.stdout:1748: done (5 left) 2024-03-19T11:31:59.826 INFO:tasks.rados.rados.0.smithi038.stdout:1749: done (4 left) 2024-03-19T11:31:59.826 INFO:tasks.rados.rados.0.smithi038.stdout:1750: done (3 left) 2024-03-19T11:31:59.826 INFO:tasks.rados.rados.0.smithi038.stdout:1752: done (2 left) 2024-03-19T11:31:59.826 INFO:tasks.rados.rados.0.smithi038.stdout:1753: done (1 left) 2024-03-19T11:31:59.826 INFO:tasks.rados.rados.0.smithi038.stdout:1754: done (0 left) 2024-03-19T11:31:59.827 INFO:tasks.rados.rados.0.smithi038.stdout:1755: tier_evict oid 47 2024-03-19T11:31:59.827 INFO:tasks.rados.rados.0.smithi038.stdout:1755: tier_evict oid 47 snap -1 2024-03-19T11:31:59.827 INFO:tasks.rados.rados.0.smithi038.stdout:1756: read oid 36 snap -1 2024-03-19T11:31:59.827 INFO:tasks.rados.rados.0.smithi038.stdout:1756: expect (ObjNum 266 snap 34 seq_num 266) 2024-03-19T11:31:59.827 INFO:tasks.rados.rados.0.smithi038.stdout:1757: delete oid 19 current snap is 40 2024-03-19T11:31:59.827 INFO:tasks.rados.rados.0.smithi038.stdout:1755: got (22) Invalid argument 2024-03-19T11:31:59.829 INFO:tasks.rados.rados.0.smithi038.stdout:1755: done (2 left) 2024-03-19T11:31:59.829 INFO:tasks.rados.rados.0.smithi038.stdout:1757: done (1 left) 2024-03-19T11:31:59.829 INFO:tasks.rados.rados.0.smithi038.stdout:1758: tier_promote oid 32 2024-03-19T11:31:59.829 INFO:tasks.rados.rados.0.smithi038.stdout:1759: write oid 12 current snap is 40 2024-03-19T11:31:59.830 INFO:tasks.rados.rados.0.smithi038.stdout:1759: seq_num 325 ranges {662275=542751,1785236=425447,2942002=520273} 2024-03-19T11:31:59.835 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:31:59.835 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:31:59.835 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:31:59.835 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:31:59.849 INFO:tasks.rados.rados.0.smithi038.stdout:1759: writing smithi03829856-12 from 662275 to 1205026 tid 1 2024-03-19T11:31:59.850 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-19T11:31:59.865 INFO:tasks.rados.rados.0.smithi038.stdout:1759: writing smithi03829856-12 from 1785236 to 2210683 tid 2 2024-03-19T11:31:59.885 INFO:tasks.rados.rados.0.smithi038.stdout:1759: writing smithi03829856-12 from 2942002 to 3462275 tid 3 2024-03-19T11:31:59.885 INFO:tasks.rados.rados.0.smithi038.stdout:1760: delete oid 10 current snap is 40 2024-03-19T11:31:59.885 INFO:tasks.rados.rados.0.smithi038.stdout:1756: expect (ObjNum 266 snap 34 seq_num 266) 2024-03-19T11:31:59.892 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:31:59.901+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:31:59.967 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:31:59.975+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:32:00.002 INFO:tasks.rados.rados.0.smithi038.stdout:1758: got (0) Success 2024-03-19T11:32:00.003 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 32 v 163 (ObjNum 323 snap 40 seq_num 323) dirty exists 2024-03-19T11:32:00.003 INFO:tasks.rados.rados.0.smithi038.stdout:1759: finishing write tid 1 to smithi03829856-12 2024-03-19T11:32:00.003 INFO:tasks.rados.rados.0.smithi038.stdout:1759: finishing write tid 2 to smithi03829856-12 2024-03-19T11:32:00.003 INFO:tasks.rados.rados.0.smithi038.stdout:1759: finishing write tid 3 to smithi03829856-12 2024-03-19T11:32:00.003 INFO:tasks.rados.rados.0.smithi038.stdout:1759: finishing write tid 4 to smithi03829856-12 2024-03-19T11:32:00.003 INFO:tasks.rados.rados.0.smithi038.stdout:1759: finishing write tid 5 to smithi03829856-12 2024-03-19T11:32:00.003 INFO:tasks.rados.rados.0.smithi038.stdout:1759: oid 12 updating version 0 to 145 2024-03-19T11:32:00.003 INFO:tasks.rados.rados.0.smithi038.stdout:1759: oid 12 updating version 145 to 146 2024-03-19T11:32:00.003 INFO:tasks.rados.rados.0.smithi038.stdout:1759: oid 12 version 146 is already newer than 144 2024-03-19T11:32:00.003 INFO:tasks.rados.rados.0.smithi038.stdout:1759: oid 12 version 146 is already newer than 143 2024-03-19T11:32:00.003 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 12 v 146 (ObjNum 325 snap 40 seq_num 325) dirty exists 2024-03-19T11:32:00.003 INFO:tasks.rados.rados.0.smithi038.stdout:1759: left oid 12 (ObjNum 325 snap 40 seq_num 325) 2024-03-19T11:32:00.003 INFO:tasks.rados.rados.0.smithi038.stdout:1756: done (3 left) 2024-03-19T11:32:00.003 INFO:tasks.rados.rados.0.smithi038.stdout:1758: done (2 left) 2024-03-19T11:32:00.003 INFO:tasks.rados.rados.0.smithi038.stdout:1759: done (1 left) 2024-03-19T11:32:00.004 INFO:tasks.rados.rados.0.smithi038.stdout:1760: done (0 left) 2024-03-19T11:32:00.004 INFO:tasks.rados.rados.0.smithi038.stdout:1761: snap_remove snap 33 2024-03-19T11:32:00.018 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:32:00.018 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:32:00.019 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:32:00.019 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:32:00.019 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:32:00.033 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-19T11:32:00.068 INFO:tasks.rados.rados.0.smithi038.stdout:1761: done (0 left) 2024-03-19T11:32:00.068 INFO:tasks.rados.rados.0.smithi038.stdout:1762: write oid 45 current snap is 40 2024-03-19T11:32:00.068 INFO:tasks.rados.rados.0.smithi038.stdout:1762: seq_num 327 ranges {695889=409602,1618553=712763,2996848=499041} 2024-03-19T11:32:00.068 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:32:00.075+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:32:00.088 INFO:tasks.rados.rados.0.smithi038.stdout:1762: writing smithi03829856-45 from 695889 to 1105491 tid 1 2024-03-19T11:32:00.099 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-03-19T11:32:00.109 INFO:tasks.rados.rados.0.smithi038.stdout:1762: writing smithi03829856-45 from 1618553 to 2331316 tid 2 2024-03-19T11:32:00.127 INFO:tasks.rados.rados.0.smithi038.stdout:1762: writing smithi03829856-45 from 2996848 to 3495889 tid 3 2024-03-19T11:32:00.127 INFO:tasks.rados.rados.0.smithi038.stdout:1763: read oid 20 snap 28 2024-03-19T11:32:00.127 INFO:tasks.rados.rados.0.smithi038.stdout:1763: expect deleted 2024-03-19T11:32:00.127 INFO:tasks.rados.rados.0.smithi038.stdout:1764: read oid 27 snap -1 2024-03-19T11:32:00.127 INFO:tasks.rados.rados.0.smithi038.stdout:1764: expect (ObjNum 240 snap 30 seq_num 240) 2024-03-19T11:32:00.127 INFO:tasks.rados.rados.0.smithi038.stdout:1765: tier_promote oid 8 2024-03-19T11:32:00.127 INFO:tasks.rados.rados.0.smithi038.stdout:1762: finishing write tid 1 to smithi03829856-45 2024-03-19T11:32:00.127 INFO:tasks.rados.rados.0.smithi038.stdout:1766: read oid 39 snap -1 2024-03-19T11:32:00.127 INFO:tasks.rados.rados.0.smithi038.stdout:1766: expect deleted 2024-03-19T11:32:00.128 INFO:tasks.rados.rados.0.smithi038.stdout:1767: read oid 2 snap -1 2024-03-19T11:32:00.128 INFO:tasks.rados.rados.0.smithi038.stdout:1767: expect (ObjNum 206 snap 22 seq_num 206) 2024-03-19T11:32:00.128 INFO:tasks.rados.rados.0.smithi038.stdout:1768: read oid 31 snap -1 2024-03-19T11:32:00.128 INFO:tasks.rados.rados.0.smithi038.stdout:1768: expect (ObjNum 300 snap 39 seq_num 300) 2024-03-19T11:32:00.128 INFO:tasks.rados.rados.0.smithi038.stdout:1769: read oid 22 snap -1 2024-03-19T11:32:00.128 INFO:tasks.rados.rados.0.smithi038.stdout:1769: expect (ObjNum 295 snap 39 seq_num 295) 2024-03-19T11:32:00.128 INFO:tasks.rados.rados.0.smithi038.stdout:1770: write oid 26 current snap is 40 2024-03-19T11:32:00.128 INFO:tasks.rados.rados.0.smithi038.stdout:1762: finishing write tid 2 to smithi03829856-45 2024-03-19T11:32:00.128 INFO:tasks.rados.rados.0.smithi038.stdout:1770: seq_num 328 ranges {712696=484851,1960682=572774,3307918=204778} 2024-03-19T11:32:00.140 INFO:tasks.rados.rados.0.smithi038.stdout:1770: writing smithi03829856-26 from 712696 to 1197547 tid 1 2024-03-19T11:32:00.154 INFO:tasks.rados.rados.0.smithi038.stdout:1770: writing smithi03829856-26 from 1960682 to 2533456 tid 2 2024-03-19T11:32:00.165 INFO:tasks.rados.rados.0.smithi038.stdout:1770: writing smithi03829856-26 from 3307918 to 3512696 tid 3 2024-03-19T11:32:00.165 INFO:tasks.rados.rados.0.smithi038.stdout:1771: set_chunk oid 4 target oid 2024-03-19T11:32:00.165 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:32:00.165 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:32:00.165 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:32:00.165 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:32:00.166 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.166 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:32:00.166 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:44.200920+0000", 2024-03-19T11:32:00.166 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 135.97216196599999, 2024-03-19T11:32:00.166 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.80881896499999995, 2024-03-19T11:32:00.166 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:00.166 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:00.166 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:00.166 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:32:00.166 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:32:00.166 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1 2024-03-19T11:32:00.166 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.166 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:00.166 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.167 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:00.167 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200920+0000", 2024-03-19T11:32:00.167 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:00.167 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.167 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.167 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:00.167 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200920+0000", 2024-03-19T11:32:00.167 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:00.167 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.167 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.167 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:00.167 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200923+0000", 2024-03-19T11:32:00.167 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6469999999999998e-06 2024-03-19T11:32:00.168 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.168 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.168 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:00.168 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200937+0000", 2024-03-19T11:32:00.168 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.399e-05 2024-03-19T11:32:00.168 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.168 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.168 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:00.168 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200960+0000", 2024-03-19T11:32:00.168 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2603e-05 2024-03-19T11:32:00.168 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.168 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.168 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:00.168 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200978+0000", 2024-03-19T11:32:00.169 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.791e-05 2024-03-19T11:32:00.169 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.169 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.169 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:00.169 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.201081+0000", 2024-03-19T11:32:00.169 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010344900000000001 2024-03-19T11:32:00.169 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.169 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.169 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:32:00.169 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.201096+0000", 2024-03-19T11:32:00.169 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4443e-05 2024-03-19T11:32:00.169 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.169 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.169 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:00.170 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.005978+0000", 2024-03-19T11:32:00.170 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.80488281500000003 2024-03-19T11:32:00.170 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.170 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.170 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:00.170 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.006267+0000", 2024-03-19T11:32:00.170 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000288688 2024-03-19T11:32:00.170 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.170 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.170 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:32:00.170 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.006534+0000", 2024-03-19T11:32:00.170 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026725799999999998 2024-03-19T11:32:00.170 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.170 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.171 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:32:00.171 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.008457+0000", 2024-03-19T11:32:00.171 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0019227820000000001 2024-03-19T11:32:00.171 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.171 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.171 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:00.171 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.009570+0000", 2024-03-19T11:32:00.171 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011131750000000001 2024-03-19T11:32:00.171 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.171 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.171 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:00.171 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.009670+0000", 2024-03-19T11:32:00.171 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9458999999999998e-05 2024-03-19T11:32:00.171 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.172 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.172 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:00.172 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.009739+0000", 2024-03-19T11:32:00.172 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9746000000000006e-05 2024-03-19T11:32:00.172 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:00.172 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:00.172 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:00.172 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.172 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.172 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:32:00.172 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.339898+0000", 2024-03-19T11:32:00.172 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 134.833184859, 2024-03-19T11:32:00.172 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019501793, 2024-03-19T11:32:00.172 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:00.172 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:00.173 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:00.173 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:32:00.173 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:32:00.173 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113 2024-03-19T11:32:00.173 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.173 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:00.173 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.173 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:00.173 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339898+0000", 2024-03-19T11:32:00.173 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:00.173 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.173 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.173 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:00.173 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339898+0000", 2024-03-19T11:32:00.174 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:00.174 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.174 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.174 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:00.174 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339899+0000", 2024-03-19T11:32:00.174 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.353e-06 2024-03-19T11:32:00.174 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.174 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.174 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:00.174 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339907+0000", 2024-03-19T11:32:00.174 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3000000000000002e-06 2024-03-19T11:32:00.174 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.174 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.174 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:00.174 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339924+0000", 2024-03-19T11:32:00.175 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6906999999999999e-05 2024-03-19T11:32:00.175 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.175 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.175 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:00.175 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339933+0000", 2024-03-19T11:32:00.175 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8969999999999997e-06 2024-03-19T11:32:00.175 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.175 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.175 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:00.175 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352239+0000", 2024-03-19T11:32:00.175 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012305814999999999 2024-03-19T11:32:00.175 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.175 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.175 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:00.175 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352466+0000", 2024-03-19T11:32:00.176 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022684100000000001 2024-03-19T11:32:00.176 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.176 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.176 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:32:00.176 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352816+0000", 2024-03-19T11:32:00.176 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00035030499999999997 2024-03-19T11:32:00.176 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.176 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.176 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:32:00.176 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.358956+0000", 2024-03-19T11:32:00.176 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061400079999999998 2024-03-19T11:32:00.176 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.176 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.176 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:00.176 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359118+0000", 2024-03-19T11:32:00.177 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016237599999999999 2024-03-19T11:32:00.177 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.177 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.177 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:00.177 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359287+0000", 2024-03-19T11:32:00.177 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000168488 2024-03-19T11:32:00.177 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.177 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.177 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:00.177 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359399+0000", 2024-03-19T11:32:00.177 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000112503 2024-03-19T11:32:00.177 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:00.177 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:00.177 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:00.177 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.178 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.178 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:32:00.178 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.339947+0000", 2024-03-19T11:32:00.178 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 134.83313509199999, 2024-03-19T11:32:00.178 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021923069999999999, 2024-03-19T11:32:00.178 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:00.178 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:00.178 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:00.178 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:32:00.178 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:32:00.178 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 114 2024-03-19T11:32:00.178 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.178 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:00.179 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.179 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:00.179 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339947+0000", 2024-03-19T11:32:00.179 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:00.179 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.179 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.179 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:00.179 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339947+0000", 2024-03-19T11:32:00.179 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:00.179 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.179 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.179 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:00.179 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339948+0000", 2024-03-19T11:32:00.180 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6000000000000003e-07 2024-03-19T11:32:00.180 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.180 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.180 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:00.180 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339956+0000", 2024-03-19T11:32:00.180 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1559999999999995e-06 2024-03-19T11:32:00.180 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.180 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.180 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:00.180 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339967+0000", 2024-03-19T11:32:00.180 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0669999999999999e-05 2024-03-19T11:32:00.180 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.180 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.180 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:00.181 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339975+0000", 2024-03-19T11:32:00.181 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0099999999999995e-06 2024-03-19T11:32:00.181 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.181 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.181 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:00.181 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.353672+0000", 2024-03-19T11:32:00.181 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013696583999999999 2024-03-19T11:32:00.181 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.181 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.181 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:00.181 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.353808+0000", 2024-03-19T11:32:00.181 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013652299999999999 2024-03-19T11:32:00.181 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.181 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.182 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:32:00.182 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.354036+0000", 2024-03-19T11:32:00.182 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022744799999999999 2024-03-19T11:32:00.182 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.182 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.182 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:32:00.182 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.360480+0000", 2024-03-19T11:32:00.182 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0064441630000000001 2024-03-19T11:32:00.182 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.182 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.182 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:00.182 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361722+0000", 2024-03-19T11:32:00.182 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001242754 2024-03-19T11:32:00.182 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.183 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.183 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:00.183 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361807+0000", 2024-03-19T11:32:00.183 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4605999999999996e-05 2024-03-19T11:32:00.183 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.183 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.183 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:00.183 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361870+0000", 2024-03-19T11:32:00.183 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3396000000000001e-05 2024-03-19T11:32:00.183 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:00.183 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:00.183 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:00.183 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.184 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.184 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:32:00.184 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.340017+0000", 2024-03-19T11:32:00.184 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 134.83306577600001, 2024-03-19T11:32:00.184 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022169355000000002, 2024-03-19T11:32:00.184 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:00.184 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:00.184 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:00.184 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:32:00.184 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:32:00.184 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 115 2024-03-19T11:32:00.184 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.184 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:00.184 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.185 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:00.185 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340017+0000", 2024-03-19T11:32:00.185 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:00.185 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.185 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.185 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:00.185 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340017+0000", 2024-03-19T11:32:00.185 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:00.185 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.185 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.185 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:00.185 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340018+0000", 2024-03-19T11:32:00.185 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.043e-06 2024-03-19T11:32:00.185 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.185 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.186 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:00.186 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340139+0000", 2024-03-19T11:32:00.186 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121502 2024-03-19T11:32:00.186 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.186 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.186 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:00.186 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340157+0000", 2024-03-19T11:32:00.186 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7382999999999999e-05 2024-03-19T11:32:00.186 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.186 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.186 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:00.186 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340166+0000", 2024-03-19T11:32:00.186 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2469999999999997e-06 2024-03-19T11:32:00.186 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.187 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.187 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:00.187 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.357470+0000", 2024-03-19T11:32:00.187 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017304119999999999 2024-03-19T11:32:00.187 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.187 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.187 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:00.187 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.357671+0000", 2024-03-19T11:32:00.187 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000200798 2024-03-19T11:32:00.187 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.187 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.187 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:32:00.187 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.358002+0000", 2024-03-19T11:32:00.187 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033090099999999998 2024-03-19T11:32:00.187 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.188 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.188 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:32:00.188 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361892+0000", 2024-03-19T11:32:00.188 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00389018 2024-03-19T11:32:00.188 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.188 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.188 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:00.188 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361983+0000", 2024-03-19T11:32:00.188 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0976000000000004e-05 2024-03-19T11:32:00.188 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.188 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.188 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:00.188 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362071+0000", 2024-03-19T11:32:00.188 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8022999999999994e-05 2024-03-19T11:32:00.188 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.189 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.189 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:00.189 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362186+0000", 2024-03-19T11:32:00.189 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115182 2024-03-19T11:32:00.189 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:00.189 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:00.189 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:00.189 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.189 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.189 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:32:00.189 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.340183+0000", 2024-03-19T11:32:00.189 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 134.83289894399999, 2024-03-19T11:32:00.189 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022279841000000002, 2024-03-19T11:32:00.189 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:00.189 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:00.190 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:00.190 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:32:00.190 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:32:00.190 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 116 2024-03-19T11:32:00.190 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.190 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:00.190 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.190 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:00.190 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340183+0000", 2024-03-19T11:32:00.190 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:00.190 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.190 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.190 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:00.190 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340183+0000", 2024-03-19T11:32:00.191 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:00.191 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.191 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.191 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:00.191 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340184+0000", 2024-03-19T11:32:00.191 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9000000000000005e-07 2024-03-19T11:32:00.191 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.191 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.191 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:00.191 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340191+0000", 2024-03-19T11:32:00.191 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0360000000000003e-06 2024-03-19T11:32:00.191 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.191 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.191 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:00.191 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340201+0000", 2024-03-19T11:32:00.192 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0844e-05 2024-03-19T11:32:00.192 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.192 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.192 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:00.192 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340209+0000", 2024-03-19T11:32:00.192 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7360000000000001e-06 2024-03-19T11:32:00.192 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.192 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.192 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:00.192 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359535+0000", 2024-03-19T11:32:00.192 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019325546999999998 2024-03-19T11:32:00.192 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.192 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.192 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:00.193 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359672+0000", 2024-03-19T11:32:00.193 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013706600000000001 2024-03-19T11:32:00.193 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.193 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.193 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:32:00.193 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359905+0000", 2024-03-19T11:32:00.193 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023334500000000001 2024-03-19T11:32:00.193 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.193 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.193 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:00.193 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362306+0000", 2024-03-19T11:32:00.193 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002401041 2024-03-19T11:32:00.193 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.193 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.193 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:32:00.194 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362329+0000", 2024-03-19T11:32:00.194 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3283999999999999e-05 2024-03-19T11:32:00.194 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.194 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.194 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:00.194 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362402+0000", 2024-03-19T11:32:00.194 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2459000000000005e-05 2024-03-19T11:32:00.194 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.194 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.194 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:00.195 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362463+0000", 2024-03-19T11:32:00.195 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1493000000000002e-05 2024-03-19T11:32:00.195 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:00.195 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:00.195 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:00.195 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.195 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.195 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:32:00.195 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.340222+0000", 2024-03-19T11:32:00.195 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 134.832860361, 2024-03-19T11:32:00.195 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022500218999999998, 2024-03-19T11:32:00.195 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:00.195 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:00.195 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:00.195 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:32:00.196 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:32:00.196 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 117 2024-03-19T11:32:00.196 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.196 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:00.196 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.196 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:00.196 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340222+0000", 2024-03-19T11:32:00.196 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:00.196 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.196 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.196 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:00.196 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340222+0000", 2024-03-19T11:32:00.196 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:00.196 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.197 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.197 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:00.197 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340223+0000", 2024-03-19T11:32:00.197 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7999999999999995e-07 2024-03-19T11:32:00.197 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.197 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.197 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:00.197 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340229+0000", 2024-03-19T11:32:00.197 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0399999999999998e-06 2024-03-19T11:32:00.197 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.197 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.198 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:00.198 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340239+0000", 2024-03-19T11:32:00.198 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0190000000000001e-05 2024-03-19T11:32:00.198 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.198 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.198 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:00.198 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340250+0000", 2024-03-19T11:32:00.198 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0720000000000001e-05 2024-03-19T11:32:00.198 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.198 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.198 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:00.198 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.360587+0000", 2024-03-19T11:32:00.198 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020337542 2024-03-19T11:32:00.198 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.199 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:00.199 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.360818+0000", 2024-03-19T11:32:00.199 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023120500000000001 2024-03-19T11:32:00.199 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.199 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:32:00.199 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361218+0000", 2024-03-19T11:32:00.199 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000399637 2024-03-19T11:32:00.199 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.199 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:32:00.199 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362483+0000", 2024-03-19T11:32:00.199 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012649200000000001 2024-03-19T11:32:00.200 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.200 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.200 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:00.200 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362538+0000", 2024-03-19T11:32:00.200 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4490000000000002e-05 2024-03-19T11:32:00.200 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.200 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.200 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:00.200 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362617+0000", 2024-03-19T11:32:00.200 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9839000000000006e-05 2024-03-19T11:32:00.200 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.200 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.200 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:00.200 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362722+0000", 2024-03-19T11:32:00.200 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104956 2024-03-19T11:32:00.201 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:00.201 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:00.201 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:00.201 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.201 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.201 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.15 51/50 [PushOp(4:afa9f208:::smithi03829856-35:head, version: 28'46, data_included: [571438~623878,1806623~424698], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:afa9f208:::smithi03829856-35:head@28'46, size: 2571438, copy_subset: [(0, 2571438)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2231321, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:af9d0d56:::smithi03829856-20:head, version: 35'55, data_included: [655360~589824,1900544~418789], data_size: 1008613, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:af9d0d56:::smithi03829856-20:head@35'55, size: 2319333, copy_subset: [(0, 2319333)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2319333, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ad3ad1b3:::smithi03829856-21:b, version: 47'66, data_included: [736140~537686,2020581~315559], data_size: 853245, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:ad3ad1b3:::smithi03829856-21:b@47'66, size: 2336140, copy_subset: [(0, 2336140)], clone_subset: {}, snapset: b=[]:{b: [b, a, 6, 5]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2336140, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ad3ad1b3:::smithi03829856-21:head, version: 47'70, data_included: [531121~742705,1711057~305871], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:ad3ad1b3:::smithi03829856-21:head@47'70, size: 3731121, copy_subset: [(0, 3731121)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2016928, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ac9cd3a3:::smithi03829856-8:head, version: 49'72, data_included: [517649~620182,1776668~428394], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:ac9cd3a3:::smithi03829856-8:head@49'72, size: 3008420, copy_subset: [(0, 3008420)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2205062, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:aa3303a5:::smithi03829856-17:b, version: 49'73, data_included: [614400~692224,1903006~356352], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:aa3303a5:::smithi03829856-17:b@49'73, size: 2268912, copy_subset: [(0, 2268912)], clone_subset: {}, snapset: b=[]:{b: [b, a, 6, 5]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2259358, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:aa3303a5:::smithi03829856-17:head, version: 49'74, data_included: [413472~893152,1818036~155424], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:aa3303a5:::smithi03829856-17:head@49'74, size: 3613472, copy_subset: [(0, 3613472)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1973460, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:32:00.201 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:28.058610+0000", 2024-03-19T11:32:00.201 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 92.114472188999997, 2024-03-19T11:32:00.201 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027100032, 2024-03-19T11:32:00.201 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:00.201 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:32:00.201 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:00.201 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.201 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:00.202 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.058610+0000", 2024-03-19T11:32:00.202 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:00.202 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.202 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.202 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:00.202 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.058610+0000", 2024-03-19T11:32:00.202 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:00.202 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.202 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.202 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:00.202 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.058611+0000", 2024-03-19T11:32:00.202 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1599999999999999e-06 2024-03-19T11:32:00.202 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.202 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.202 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:00.203 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.063874+0000", 2024-03-19T11:32:00.203 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052625270000000004 2024-03-19T11:32:00.203 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.203 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:00.203 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.063935+0000", 2024-03-19T11:32:00.203 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0594000000000003e-05 2024-03-19T11:32:00.203 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.203 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:00.203 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064024+0000", 2024-03-19T11:32:00.203 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9273999999999999e-05 2024-03-19T11:32:00.203 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.203 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:00.204 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064385+0000", 2024-03-19T11:32:00.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00036129100000000002 2024-03-19T11:32:00.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.204 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:00.204 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064437+0000", 2024-03-19T11:32:00.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2346000000000003e-05 2024-03-19T11:32:00.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.204 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:00.204 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085710+0000", 2024-03-19T11:32:00.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021272840000000001 2024-03-19T11:32:00.204 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:00.204 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:00.204 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:00.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.205 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.15 e51,50 4:ac9cd3a3:::smithi03829856-8:4,49'71)", 2024-03-19T11:32:00.205 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:28.064112+0000", 2024-03-19T11:32:00.205 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 92.108970362999997, 2024-03-19T11:32:00.205 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026480417999999999, 2024-03-19T11:32:00.205 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:00.205 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:32:00.205 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:00.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.205 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:00.205 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064112+0000", 2024-03-19T11:32:00.205 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:00.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.206 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:00.206 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064112+0000", 2024-03-19T11:32:00.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:00.206 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.206 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.206 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:00.206 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064113+0000", 2024-03-19T11:32:00.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3600000000000003e-07 2024-03-19T11:32:00.206 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.206 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.206 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:00.206 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064118+0000", 2024-03-19T11:32:00.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0549999999999999e-06 2024-03-19T11:32:00.206 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.207 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:00.207 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064128+0000", 2024-03-19T11:32:00.207 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0349e-05 2024-03-19T11:32:00.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.207 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:00.207 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064136+0000", 2024-03-19T11:32:00.207 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6249999999999998e-06 2024-03-19T11:32:00.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.207 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:00.207 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085773+0000", 2024-03-19T11:32:00.207 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021636829999999999 2024-03-19T11:32:00.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.208 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.208 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:00.208 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085788+0000", 2024-03-19T11:32:00.208 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5512999999999999e-05 2024-03-19T11:32:00.208 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.208 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.208 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:00.208 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.090592+0000", 2024-03-19T11:32:00.208 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0048044100000000003 2024-03-19T11:32:00.208 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:00.208 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:00.208 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:00.208 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.208 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.208 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.5 51/50 [PushOp(4:a38b9453:::smithi03829856-37:head, version: 28'45, data_included: [605052~774376,1923587~274200], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:a38b9453:::smithi03829856-37:head@28'45, size: 2605052, copy_subset: [(0, 2605052)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2197787, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:a0a50d04:::smithi03829856-13:b, version: 47'63, data_included: [601684~619341,1669078~429235], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a0a50d04:::smithi03829856-13:b@47'63, size: 2201684, copy_subset: [(0, 2201684)], clone_subset: {}, snapset: b=[]:{b: [b, a, 6, 5]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2098313, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:32:00.209 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:28.064324+0000", 2024-03-19T11:32:00.209 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 92.108758519000006, 2024-03-19T11:32:00.209 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027179020000000002, 2024-03-19T11:32:00.209 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:00.209 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:32:00.209 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:00.209 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.209 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:00.209 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064324+0000", 2024-03-19T11:32:00.209 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:00.209 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.209 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.209 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:00.209 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064324+0000", 2024-03-19T11:32:00.210 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:00.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.210 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.210 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:00.210 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064324+0000", 2024-03-19T11:32:00.210 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1199999999999998e-07 2024-03-19T11:32:00.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.210 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.210 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:00.210 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.065527+0000", 2024-03-19T11:32:00.210 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012028939999999999 2024-03-19T11:32:00.210 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.210 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.210 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:00.211 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.065552+0000", 2024-03-19T11:32:00.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4440999999999999e-05 2024-03-19T11:32:00.211 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.211 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:00.211 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.065568+0000", 2024-03-19T11:32:00.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6498e-05 2024-03-19T11:32:00.211 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.211 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:00.211 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085812+0000", 2024-03-19T11:32:00.211 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020243954000000002 2024-03-19T11:32:00.211 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.211 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.211 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:00.212 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085825+0000", 2024-03-19T11:32:00.212 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3298999999999999e-05 2024-03-19T11:32:00.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.212 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:00.212 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.091503+0000", 2024-03-19T11:32:00.212 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.005677522 2024-03-19T11:32:00.212 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:00.212 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:00.212 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:00.212 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.212 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.212 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.24 75/74 [PushOp(4:27909373:::smithi03829856-46:8, version: 46'59, data_included: [756315~767970,1933572~280606], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:8@46'59, size: 2756315, copy_subset: [(0, 2756315)], clone_subset: {}, snapset: 10=[]:{8: [6], c: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2214178, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:266590a5:::smithi03829856-16:e, version: 70'107, data_included: [652105~662359,1728171~386217], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:e@70'107, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 10=[]:{e: [d, b, 6]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:266590a5:::smithi03829856-16:head, version: 70'108, data_included: [652105~662359,1728171~386217], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:head@70'108, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:27909373:::smithi03829856-46:c, version: 70'109, data_included: [628595~895690,1807440~152886], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:c@70'109, size: 3428595, copy_subset: [(0, 3428595)], clone_subset: {}, snapset: 10=[]:{8: [6], c: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1960326, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:27909373:::smithi03829856-46:head, version: 70'110, data_included: [403401~454961,1584925~506039], data_size: 961000, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:head@70'110, size: 2403401, copy_subset: [(0, 2403401)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2403401, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:32:00.212 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.099172+0000", 2024-03-19T11:32:00.213 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 66.073910084999994, 2024-03-19T11:32:00.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.024528580000000001, 2024-03-19T11:32:00.213 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:00.213 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:32:00.213 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:00.213 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.213 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:00.213 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.099172+0000", 2024-03-19T11:32:00.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:00.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.213 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.213 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:00.213 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.099172+0000", 2024-03-19T11:32:00.213 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:00.213 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.214 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.214 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:00.214 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.099175+0000", 2024-03-19T11:32:00.214 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7750000000000001e-06 2024-03-19T11:32:00.214 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.214 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.214 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:00.214 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105219+0000", 2024-03-19T11:32:00.214 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0060440789999999999 2024-03-19T11:32:00.214 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.214 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.214 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:00.214 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105285+0000", 2024-03-19T11:32:00.214 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6260000000000006e-05 2024-03-19T11:32:00.214 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.215 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.215 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:00.215 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105366+0000", 2024-03-19T11:32:00.215 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.038e-05 2024-03-19T11:32:00.215 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.215 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.215 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:00.215 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105637+0000", 2024-03-19T11:32:00.215 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027085199999999997 2024-03-19T11:32:00.215 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.215 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.215 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:00.215 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105679+0000", 2024-03-19T11:32:00.215 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2088000000000001e-05 2024-03-19T11:32:00.215 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.216 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:00.216 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.123701+0000", 2024-03-19T11:32:00.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018022145999999999 2024-03-19T11:32:00.216 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:00.216 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:00.216 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:00.216 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.216 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.216 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.24 e75,74 4:266590a5:::smithi03829856-16:4,47'63,4:27909373:::smithi03829856-46:5,55'79,4:266590a5:::smithi03829856-16:10,67'105)", 2024-03-19T11:32:00.216 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.105437+0000", 2024-03-19T11:32:00.216 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 66.067645081999999, 2024-03-19T11:32:00.216 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.01860726, 2024-03-19T11:32:00.216 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:00.216 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:32:00.217 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:00.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:00.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105437+0000", 2024-03-19T11:32:00.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:00.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:00.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105437+0000", 2024-03-19T11:32:00.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:00.217 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.217 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.217 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:00.217 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105438+0000", 2024-03-19T11:32:00.217 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6799999999999999e-07 2024-03-19T11:32:00.218 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.218 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.218 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:00.218 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105445+0000", 2024-03-19T11:32:00.218 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3749999999999999e-06 2024-03-19T11:32:00.218 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.218 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.218 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:00.218 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105454+0000", 2024-03-19T11:32:00.218 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7909999999999999e-06 2024-03-19T11:32:00.218 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.218 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.218 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:00.218 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105465+0000", 2024-03-19T11:32:00.219 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.045e-05 2024-03-19T11:32:00.219 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.219 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:00.219 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.123753+0000", 2024-03-19T11:32:00.219 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0182884 2024-03-19T11:32:00.219 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.219 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:00.219 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.123765+0000", 2024-03-19T11:32:00.219 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2143e-05 2024-03-19T11:32:00.219 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.219 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.219 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:00.219 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.124045+0000", 2024-03-19T11:32:00.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027923299999999999 2024-03-19T11:32:00.220 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:00.220 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:00.220 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:00.220 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.220 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:3103 4.15 e104/76 4:afa9f208:::smithi03829856-35:head v 104'221, mlcod=101'216)", 2024-03-19T11:32:00.220 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.563593+0000", 2024-03-19T11:32:00.220 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 36.609489803999999, 2024-03-19T11:32:00.220 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026983566, 2024-03-19T11:32:00.220 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:00.220 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:00.220 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:00.220 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.220 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:00.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.563593+0000", 2024-03-19T11:32:00.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:00.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:00.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.563593+0000", 2024-03-19T11:32:00.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:00.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.221 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:00.221 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.563595+0000", 2024-03-19T11:32:00.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2859999999999998e-06 2024-03-19T11:32:00.221 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:00.222 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.564339+0000", 2024-03-19T11:32:00.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00074400099999999999 2024-03-19T11:32:00.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:00.222 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.564376+0000", 2024-03-19T11:32:00.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6628999999999998e-05 2024-03-19T11:32:00.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.222 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.222 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:00.222 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.564396+0000", 2024-03-19T11:32:00.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0596000000000001e-05 2024-03-19T11:32:00.222 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.223 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:00.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.566600+0000", 2024-03-19T11:32:00.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002204021 2024-03-19T11:32:00.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.223 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:00.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.566642+0000", 2024-03-19T11:32:00.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1902e-05 2024-03-19T11:32:00.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.223 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:00.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.590494+0000", 2024-03-19T11:32:00.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023851816000000001 2024-03-19T11:32:00.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.224 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:00.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.590576+0000", 2024-03-19T11:32:00.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2315e-05 2024-03-19T11:32:00.224 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:00.224 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:00.224 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:00.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.224 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:3104 4.15 e104/76 4:afa9f208:::smithi03829856-35:head v 104'222, mlcod=101'216)", 2024-03-19T11:32:00.225 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.568280+0000", 2024-03-19T11:32:00.225 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 36.604801961, 2024-03-19T11:32:00.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022345304999999999, 2024-03-19T11:32:00.225 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:00.225 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:00.225 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:00.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:00.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.568280+0000", 2024-03-19T11:32:00.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:00.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:00.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.568280+0000", 2024-03-19T11:32:00.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:00.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:00.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.568282+0000", 2024-03-19T11:32:00.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0090000000000002e-06 2024-03-19T11:32:00.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:00.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.568299+0000", 2024-03-19T11:32:00.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6237000000000001e-05 2024-03-19T11:32:00.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:00.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.568331+0000", 2024-03-19T11:32:00.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2601999999999997e-05 2024-03-19T11:32:00.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.227 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:00.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.568350+0000", 2024-03-19T11:32:00.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8280000000000001e-05 2024-03-19T11:32:00.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.227 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:00.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.571311+0000", 2024-03-19T11:32:00.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029617770000000001 2024-03-19T11:32:00.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.228 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:00.228 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.571379+0000", 2024-03-19T11:32:00.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7331999999999998e-05 2024-03-19T11:32:00.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.228 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.228 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:00.228 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.590583+0000", 2024-03-19T11:32:00.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019204649000000001 2024-03-19T11:32:00.228 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.228 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.228 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:00.228 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.590626+0000", 2024-03-19T11:32:00.228 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2419000000000002e-05 2024-03-19T11:32:00.228 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:00.229 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:00.229 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:00.229 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.229 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.229 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:3106 4.15 e104/76 4:ad3ad1b3:::smithi03829856-21:head v 104'224, mlcod=101'216)", 2024-03-19T11:32:00.229 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.571056+0000", 2024-03-19T11:32:00.229 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 36.602026502999998, 2024-03-19T11:32:00.229 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019636355000000001, 2024-03-19T11:32:00.229 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:00.229 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:00.229 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:00.229 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.229 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:00.229 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.571056+0000", 2024-03-19T11:32:00.230 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:00.230 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.230 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:00.230 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.571056+0000", 2024-03-19T11:32:00.230 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:00.230 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.230 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:00.230 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.571058+0000", 2024-03-19T11:32:00.230 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2400000000000002e-06 2024-03-19T11:32:00.230 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.230 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.230 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:00.230 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.571586+0000", 2024-03-19T11:32:00.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00052782299999999996 2024-03-19T11:32:00.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.231 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:00.231 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.571618+0000", 2024-03-19T11:32:00.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2453000000000003e-05 2024-03-19T11:32:00.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.231 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:00.231 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.571638+0000", 2024-03-19T11:32:00.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9176000000000001e-05 2024-03-19T11:32:00.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.231 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:00.232 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.573958+0000", 2024-03-19T11:32:00.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0023201699999999999 2024-03-19T11:32:00.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.232 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:00.232 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.574023+0000", 2024-03-19T11:32:00.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5425000000000005e-05 2024-03-19T11:32:00.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.232 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:00.232 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.590646+0000", 2024-03-19T11:32:00.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016623189 2024-03-19T11:32:00.232 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.233 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:00.233 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.590692+0000", 2024-03-19T11:32:00.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5878999999999999e-05 2024-03-19T11:32:00.233 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:00.233 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:00.233 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:00.233 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.233 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.3.0:309 4.0 e108/60 4:01018511:::smithi03829856-19:b v 108'91, mlcod=100'87)", 2024-03-19T11:32:00.233 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.205898+0000", 2024-03-19T11:32:00.233 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.967184707999998, 2024-03-19T11:32:00.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.034464328000000002, 2024-03-19T11:32:00.233 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:00.233 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:00.234 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:00.234 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.234 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:00.234 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.205898+0000", 2024-03-19T11:32:00.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:00.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.234 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.234 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:00.234 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.205898+0000", 2024-03-19T11:32:00.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:00.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.234 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.234 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:00.234 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.205899+0000", 2024-03-19T11:32:00.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.16e-07 2024-03-19T11:32:00.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:00.235 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.205916+0000", 2024-03-19T11:32:00.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7229000000000001e-05 2024-03-19T11:32:00.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:00.235 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.205946+0000", 2024-03-19T11:32:00.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0233999999999999e-05 2024-03-19T11:32:00.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:00.235 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.205966+0000", 2024-03-19T11:32:00.236 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9434999999999999e-05 2024-03-19T11:32:00.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.236 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:00.236 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.206162+0000", 2024-03-19T11:32:00.236 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000196918 2024-03-19T11:32:00.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.236 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:00.236 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.206227+0000", 2024-03-19T11:32:00.236 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4381000000000006e-05 2024-03-19T11:32:00.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.236 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:00.236 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.240212+0000", 2024-03-19T11:32:00.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033984852000000003 2024-03-19T11:32:00.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:00.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.240362+0000", 2024-03-19T11:32:00.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015036300000000001 2024-03-19T11:32:00.237 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:00.237 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:00.237 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:00.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.237 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.5.0:238 4.15 e108/76 4:ac9cd3a3:::smithi03829856-8:10 v 108'231, mlcod=104'228)", 2024-03-19T11:32:00.237 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.215160+0000", 2024-03-19T11:32:00.237 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.9579223, 2024-03-19T11:32:00.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.024600749000000002, 2024-03-19T11:32:00.238 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:00.238 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:00.238 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:00.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:00.238 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215160+0000", 2024-03-19T11:32:00.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:00.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:00.238 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215160+0000", 2024-03-19T11:32:00.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:00.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:00.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215161+0000", 2024-03-19T11:32:00.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6000000000000002e-07 2024-03-19T11:32:00.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:00.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215172+0000", 2024-03-19T11:32:00.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1042e-05 2024-03-19T11:32:00.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:00.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215213+0000", 2024-03-19T11:32:00.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1167000000000001e-05 2024-03-19T11:32:00.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:00.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215234+0000", 2024-03-19T11:32:00.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0534e-05 2024-03-19T11:32:00.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:00.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215668+0000", 2024-03-19T11:32:00.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000434074 2024-03-19T11:32:00.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.240 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.240 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:00.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215749+0000", 2024-03-19T11:32:00.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0702000000000001e-05 2024-03-19T11:32:00.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:00.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.239629+0000", 2024-03-19T11:32:00.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023880855999999999 2024-03-19T11:32:00.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:00.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.239761+0000", 2024-03-19T11:32:00.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000131514 2024-03-19T11:32:00.242 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:00.242 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:00.242 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:00.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.242 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.5.0:239 4.8 e108/100 4:14611006:::smithi03829856-25:c v 108'125, mlcod=103'122)", 2024-03-19T11:32:00.242 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.215292+0000", 2024-03-19T11:32:00.242 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.957790115999998, 2024-03-19T11:32:00.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.025163089999999999, 2024-03-19T11:32:00.242 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:00.243 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:00.243 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:00.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.243 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:00.243 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215292+0000", 2024-03-19T11:32:00.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:00.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.243 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:00.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215292+0000", 2024-03-19T11:32:00.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:00.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:00.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215293+0000", 2024-03-19T11:32:00.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.47e-07 2024-03-19T11:32:00.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:00.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215315+0000", 2024-03-19T11:32:00.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1691e-05 2024-03-19T11:32:00.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:00.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215348+0000", 2024-03-19T11:32:00.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2676999999999999e-05 2024-03-19T11:32:00.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:00.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215362+0000", 2024-03-19T11:32:00.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3978999999999999e-05 2024-03-19T11:32:00.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.245 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:00.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215727+0000", 2024-03-19T11:32:00.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000365583 2024-03-19T11:32:00.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:00.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215804+0000", 2024-03-19T11:32:00.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7392999999999996e-05 2024-03-19T11:32:00.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:00.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.240368+0000", 2024-03-19T11:32:00.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.024563807999999999 2024-03-19T11:32:00.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:00.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.240455+0000", 2024-03-19T11:32:00.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7112000000000002e-05 2024-03-19T11:32:00.247 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:00.247 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:00.247 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:00.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.247 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.5.0:240 4.15 e108/76 4:aa3303a5:::smithi03829856-17:b v 108'233, mlcod=104'228)", 2024-03-19T11:32:00.247 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.215400+0000", 2024-03-19T11:32:00.247 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.957682386000002, 2024-03-19T11:32:00.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026645319000000001, 2024-03-19T11:32:00.247 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:00.248 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:00.248 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:00.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.248 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:00.248 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215400+0000", 2024-03-19T11:32:00.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:00.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.248 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:00.248 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215400+0000", 2024-03-19T11:32:00.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:00.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.248 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:00.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215401+0000", 2024-03-19T11:32:00.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3e-07 2024-03-19T11:32:00.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:00.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215417+0000", 2024-03-19T11:32:00.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5821999999999999e-05 2024-03-19T11:32:00.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:00.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215443+0000", 2024-03-19T11:32:00.249 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6971e-05 2024-03-19T11:32:00.249 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.249 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:00.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215457+0000", 2024-03-19T11:32:00.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3139000000000001e-05 2024-03-19T11:32:00.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:00.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.234414+0000", 2024-03-19T11:32:00.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018957108 2024-03-19T11:32:00.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:00.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.234470+0000", 2024-03-19T11:32:00.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6274999999999999e-05 2024-03-19T11:32:00.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:00.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.241943+0000", 2024-03-19T11:32:00.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0074728770000000002 2024-03-19T11:32:00.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.251 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:00.251 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.242045+0000", 2024-03-19T11:32:00.251 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000102497 2024-03-19T11:32:00.251 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:00.251 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:00.251 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:00.251 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.252 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.5.0:241 4.8 e108/100 4:15708482:::smithi03829856-23:10 v 108'127, mlcod=103'122)", 2024-03-19T11:32:00.252 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.217180+0000", 2024-03-19T11:32:00.252 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.955902117999997, 2024-03-19T11:32:00.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.025026023000000001, 2024-03-19T11:32:00.252 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:00.252 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:00.252 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:00.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.252 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:00.252 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.217180+0000", 2024-03-19T11:32:00.252 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:00.252 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.252 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:00.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.217180+0000", 2024-03-19T11:32:00.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:00.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:00.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.217182+0000", 2024-03-19T11:32:00.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.12e-06 2024-03-19T11:32:00.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:00.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.217200+0000", 2024-03-19T11:32:00.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7677999999999999e-05 2024-03-19T11:32:00.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.254 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:00.254 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.217246+0000", 2024-03-19T11:32:00.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6053000000000001e-05 2024-03-19T11:32:00.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.254 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:00.254 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.217266+0000", 2024-03-19T11:32:00.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0085000000000002e-05 2024-03-19T11:32:00.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.254 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:00.254 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.226876+0000", 2024-03-19T11:32:00.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0096094389999999991 2024-03-19T11:32:00.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:00.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.226932+0000", 2024-03-19T11:32:00.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6234999999999999e-05 2024-03-19T11:32:00.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:00.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.242063+0000", 2024-03-19T11:32:00.255 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015131083 2024-03-19T11:32:00.255 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:00.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.242206+0000", 2024-03-19T11:32:00.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014333 2024-03-19T11:32:00.256 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:00.256 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:00.256 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:00.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.256 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.1.0:279 4.24 e108/83 4:27909373:::smithi03829856-46:c v 108'129, mlcod=100'124)", 2024-03-19T11:32:00.256 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.223035+0000", 2024-03-19T11:32:00.256 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.950047777000002, 2024-03-19T11:32:00.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019488583, 2024-03-19T11:32:00.256 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:00.256 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:00.256 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:00.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:00.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.223035+0000", 2024-03-19T11:32:00.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:00.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:00.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.223035+0000", 2024-03-19T11:32:00.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:00.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:00.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.223037+0000", 2024-03-19T11:32:00.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3829999999999999e-06 2024-03-19T11:32:00.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:00.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.223073+0000", 2024-03-19T11:32:00.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6306000000000001e-05 2024-03-19T11:32:00.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:00.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.223095+0000", 2024-03-19T11:32:00.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1738000000000001e-05 2024-03-19T11:32:00.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.258 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:00.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.223107+0000", 2024-03-19T11:32:00.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2082000000000001e-05 2024-03-19T11:32:00.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:00.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.226145+0000", 2024-03-19T11:32:00.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003038022 2024-03-19T11:32:00.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:00.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.226201+0000", 2024-03-19T11:32:00.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6107999999999999e-05 2024-03-19T11:32:00.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:00.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.242322+0000", 2024-03-19T11:32:00.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016120196 2024-03-19T11:32:00.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:00.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:00.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:00.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.242523+0000", 2024-03-19T11:32:00.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020174799999999999 2024-03-19T11:32:00.260 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:00.260 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:00.260 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:00.260 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:00.260 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:00.260 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:32:00.261 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-19T11:32:00.261 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-19T11:32:00.261 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_0 2024-03-19T11:32:00.261 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T11:32:00.263 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:32:00.269+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:32:00.263 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-19T11:32:00.264 INFO:tasks.rados.rados.0.smithi038.stdout:1771: set_chunk oid 4 offset: 132608 length: 901632 target oid 22fc9f7876f62fbaf034981208f738a1842c01abf2ea866893032a4063e6a98c offset: 0 snap -1 2024-03-19T11:32:00.264 INFO:tasks.rados.rados.0.smithi038.stdout:1772: set_chunk oid 28 target oid 2024-03-19T11:32:00.264 INFO:tasks.rados.rados.0.smithi038.stdout:1772: set_chunk oid 28 offset: 2083328 length: 1216000 target oid cd5ae45fc9ef1adb1ff090fcaa9e2c778c77ab41c97a45bf056a655f4d4118b1 offset: 0 snap -1 2024-03-19T11:32:00.265 INFO:tasks.rados.rados.0.smithi038.stdout:1773: read oid 33 snap -1 2024-03-19T11:32:00.265 INFO:tasks.rados.rados.0.smithi038.stdout:1773: expect (ObjNum 306 snap 39 seq_num 306) 2024-03-19T11:32:00.265 INFO:tasks.rados.rados.0.smithi038.stdout:1774: write oid 14 current snap is 40 2024-03-19T11:32:00.265 INFO:tasks.rados.rados.0.smithi038.stdout:1774: seq_num 329 ranges {729503=560100,2019164=716432,3335341=194162} 2024-03-19T11:32:00.265 INFO:tasks.rados.rados.0.smithi038.stdout:1774: writing smithi03829856-14 from 729503 to 1289603 tid 1 2024-03-19T11:32:00.265 INFO:tasks.rados.rados.0.smithi038.stdout:1774: writing smithi03829856-14 from 2019164 to 2735596 tid 2 2024-03-19T11:32:00.265 INFO:tasks.rados.rados.0.smithi038.stdout:1774: writing smithi03829856-14 from 3335341 to 3529503 tid 3 2024-03-19T11:32:00.265 INFO:tasks.rados.rados.0.smithi038.stdout:1775: snap_remove snap 38 2024-03-19T11:32:00.363 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:32:00.371+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:32:00.380 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:32:00.380 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:32:00.380 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:32:00.380 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:32:00.396 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-19T11:32:00.464 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:32:00.473+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:32:00.547 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:32:00.547 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:32:00.547 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:32:00.547 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:32:00.547 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:32:00.557 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:32:00.558 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-03-19T11:32:00.560 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-19T11:32:00.565 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:32:00.573+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:32:00.574 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":201,"stamp":"2024-03-19T11:32:00.173128+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":492593815,"num_objects":350,"num_object_clones":116,"num_object_copies":700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":346,"num_whiteouts":7,"num_read":3180,"num_read_kb":3233177,"num_write":1537,"num_write_kb":461781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":529,"num_bytes_recovered":504527041,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":94,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3386,"ondisk_log_size":3386,"up":94,"acting":94,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":986272,"kb_used_data":728480,"kb_used_omap":12,"kb_used_meta":257651,"kb_avail":753988448,"statfs":{"total":773094113280,"available":772084170752,"internally_reserved":0,"allocated":745963520,"data_stored":888986829,"data_compressed":19218963,"data_compressed_allocated":19148800,"data_compressed_original":44665119,"omap_allocated":12751,"internal_metadata":263835185},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,4],"upper_bound":2048},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6518919,"num_objects":6,"num_object_clones":5,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":3,"num_read":152,"num_read_kb":214981,"num_write":41,"num_write_kb":9481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":110073960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001817"},"pg_stats":[{"pgid":"4.21","version":"135'193","reported_seq":868,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-19T11:31:58.070010+0000","last_change":"2024-03-19T11:31:56.183778+0000","last_active":"2024-03-19T11:31:58.070010+0000","last_peered":"2024-03-19T11:31:58.070010+0000","last_clean":"2024-03-19T11:31:58.070010+0000","last_became_active":"2024-03-19T11:31:53.079052+0000","last_became_peered":"2024-03-19T11:31:53.079052+0000","last_unstale":"2024-03-19T11:31:58.070010+0000","last_undegraded":"2024-03-19T11:31:58.070010+0000","last_fullsized":"2024-03-19T11:31:58.070010+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":6,"last_scrub":"62'96","last_scrub_stamp":"2024-03-19T11:30:43.400885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:43.400885+0000","objects_scrubbed":9,"log_size":193,"log_dups_size":0,"ondisk_log_size":193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.014439463,"stat_sum":{"num_bytes":15416396,"num_objects":7,"num_object_clones":5,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":128,"num_read_kb":172690,"num_write":47,"num_write_kb":14031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":35884101,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1d","length":"1"},{"start":"1f","length":"1"},{"start":"26","length":"1"},{"start":"29","length":"1"}]},{"pgid":"4.20","version":"56'46","reported_seq":266,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-19T11:31:58.075630+0000","last_change":"2024-03-19T11:31:56.190772+0000","last_active":"2024-03-19T11:31:58.075630+0000","last_peered":"2024-03-19T11:31:58.075630+0000","last_clean":"2024-03-19T11:31:58.075630+0000","last_became_active":"2024-03-19T11:30:53.067251+0000","last_became_peered":"2024-03-19T11:30:53.067251+0000","last_unstale":"2024-03-19T11:31:58.075630+0000","last_undegraded":"2024-03-19T11:31:58.075630+0000","last_fullsized":"2024-03-19T11:31:58.075630+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":6,"last_scrub":"56'46","last_scrub_stamp":"2024-03-19T11:31:54.326784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:31:54.326784+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:33:04.593418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0074969650000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1d","length":"1"},{"start":"1f","length":"1"},{"start":"26","length":"1"},{"start":"29","length":"1"}]},{"pgid":"4.1f","version":"0'0","reported_seq":271,"reported_epoch":137,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:58.076413+0000","last_change":"2024-03-19T11:31:56.184478+0000","last_active":"2024-03-19T11:31:58.076413+0000","last_peered":"2024-03-19T11:31:58.076413+0000","last_clean":"2024-03-19T11:31:58.076413+0000","last_became_active":"2024-03-19T11:30:53.087211+0000","last_became_peered":"2024-03-19T11:30:53.087211+0000","last_unstale":"2024-03-19T11:31:58.076413+0000","last_undegraded":"2024-03-19T11:31:58.076413+0000","last_fullsized":"2024-03-19T11:31:58.076413+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:31:44.200984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:31:44.200984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:57.288918+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.90932528000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1d","length":"1"},{"start":"1f","length":"1"},{"start":"29","length":"1"}]},{"pgid":"4.1e","version":"134'95","reported_seq":313,"reported_epoch":136,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:57.181317+0000","last_change":"2024-03-19T11:31:56.100290+0000","last_active":"2024-03-19T11:31:57.181317+0000","last_peered":"2024-03-19T11:31:57.181317+0000","last_clean":"2024-03-19T11:31:57.181317+0000","last_became_active":"2024-03-19T11:30:53.084452+0000","last_became_peered":"2024-03-19T11:30:53.084452+0000","last_unstale":"2024-03-19T11:31:57.181317+0000","last_undegraded":"2024-03-19T11:31:57.181317+0000","last_fullsized":"2024-03-19T11:31:57.181317+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"64'62","last_scrub_stamp":"2024-03-19T11:30:42.778159+0000","last_deep_scrub":"64'62","last_deep_scrub_stamp":"2024-03-19T11:30:42.778159+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.778159+0000","objects_scrubbed":4,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:07.898333+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0.046533468000000001,"stat_sum":{"num_bytes":5901612,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":76,"num_read_kb":100235,"num_write":9,"num_write_kb":2564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3779426,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1c","length":"1"},{"start":"1f","length":"1"}]},{"pgid":"4.1d","version":"112'76","reported_seq":44,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-19T11:31:59.079009+0000","last_change":"2024-03-19T11:31:59.079009+0000","last_active":"2024-03-19T11:31:59.079009+0000","last_peered":"2024-03-19T11:31:59.079009+0000","last_clean":"2024-03-19T11:31:59.079009+0000","last_became_active":"2024-03-19T11:31:56.186117+0000","last_became_peered":"2024-03-19T11:31:56.186117+0000","last_unstale":"2024-03-19T11:31:59.079009+0000","last_undegraded":"2024-03-19T11:31:59.079009+0000","last_fullsized":"2024-03-19T11:31:59.079009+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":5,"last_scrub":"112'76","last_scrub_stamp":"2024-03-19T11:31:57.951264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:31:57.951264+0000","objects_scrubbed":2,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:33:08.630609+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00145835,"stat_sum":{"num_bytes":4554730,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":31,"num_read_kb":30797,"num_write":9,"num_write_kb":2431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":7152062,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1a","length":"1"},{"start":"1d","length":"1"},{"start":"1f","length":"1"},{"start":"26","length":"1"},{"start":"29","length":"1"}]},{"pgid":"4.1c","version":"138'62","reported_seq":319,"reported_epoch":138,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:59.074721+0000","last_change":"2024-03-19T11:31:59.074721+0000","last_active":"2024-03-19T11:31:59.074721+0000","last_peered":"2024-03-19T11:31:59.074721+0000","last_clean":"2024-03-19T11:31:59.074721+0000","last_became_active":"2024-03-19T11:30:53.082655+0000","last_became_peered":"2024-03-19T11:30:53.082655+0000","last_unstale":"2024-03-19T11:31:59.074721+0000","last_undegraded":"2024-03-19T11:31:59.074721+0000","last_fullsized":"2024-03-19T11:31:59.074721+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":5,"last_scrub":"62'31","last_scrub_stamp":"2024-03-19T11:30:42.771396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.771396+0000","objects_scrubbed":3,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:00.683895+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001886071,"stat_sum":{"num_bytes":7461382,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":72,"num_read_kb":100297,"num_write":10,"num_write_kb":2426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":5034309,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1d","length":"1"},{"start":"1f","length":"1"},{"start":"26","length":"1"},{"start":"29","length":"1"}]},{"pgid":"4.1b","version":"135'159","reported_seq":411,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-19T11:31:58.070329+0000","last_change":"2024-03-19T11:31:56.297242+0000","last_active":"2024-03-19T11:31:58.070329+0000","last_peered":"2024-03-19T11:31:58.070329+0000","last_clean":"2024-03-19T11:31:58.070329+0000","last_became_active":"2024-03-19T11:31:56.194970+0000","last_became_peered":"2024-03-19T11:31:56.194970+0000","last_unstale":"2024-03-19T11:31:58.070329+0000","last_undegraded":"2024-03-19T11:31:58.070329+0000","last_fullsized":"2024-03-19T11:31:58.070329+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":5,"last_scrub":"62'62","last_scrub_stamp":"2024-03-19T11:30:42.181424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:42.181424+0000","objects_scrubbed":5,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:01.543249+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0045640000000000003,"stat_sum":{"num_bytes":16692235,"num_objects":9,"num_object_clones":6,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":130,"num_read_kb":141794,"num_write":32,"num_write_kb":7536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":42417899,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":6,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1d","length":"1"},{"start":"1f","length":"1"},{"start":"26","length":"1"},{"start":"29","length":"1"}]},{"pgid":"4.1a","version":"133'165","reported_seq":406,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-19T11:31:58.069984+0000","last_change":"2024-03-19T11:31:56.340995+0000","last_active":"2024-03-19T11:31:58.069984+0000","last_peered":"2024-03-19T11:31:58.069984+0000","last_clean":"2024-03-19T11:31:58.069984+0000","last_became_active":"2024-03-19T11:31:56.197984+0000","last_became_peered":"2024-03-19T11:31:56.197984+0000","last_unstale":"2024-03-19T11:31:58.069984+0000","last_undegraded":"2024-03-19T11:31:58.069984+0000","last_fullsized":"2024-03-19T11:31:58.069984+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":5,"last_scrub":"61'76","last_scrub_stamp":"2024-03-19T11:30:41.433501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:41.433501+0000","objects_scrubbed":5,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:01.565589+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0012302249999999999,"stat_sum":{"num_bytes":14909314,"num_objects":8,"num_object_clones":6,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":101,"num_read_kb":110139,"num_write":39,"num_write_kb":12385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":41536658,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1d","length":"1"},{"start":"1f","length":"1"},{"start":"26","length":"1"},{"start":"29","length":"1"}]},{"pgid":"4.19","version":"138'230","reported_seq":880,"reported_epoch":138,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:59.099522+0000","last_change":"2024-03-19T11:31:59.088039+0000","last_active":"2024-03-19T11:31:59.099522+0000","last_peered":"2024-03-19T11:31:59.099522+0000","last_clean":"2024-03-19T11:31:59.099522+0000","last_became_active":"2024-03-19T11:31:59.081198+0000","last_became_peered":"2024-03-19T11:31:59.081198+0000","last_unstale":"2024-03-19T11:31:59.099522+0000","last_undegraded":"2024-03-19T11:31:59.099522+0000","last_fullsized":"2024-03-19T11:31:59.099522+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":5,"last_scrub":"41'37","last_scrub_stamp":"2024-03-19T11:30:27.705159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:27.705159+0000","objects_scrubbed":4,"log_size":230,"log_dups_size":0,"ondisk_log_size":230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.020915878999999998,"stat_sum":{"num_bytes":29412989,"num_objects":14,"num_object_clones":10,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":258,"num_read_kb":275961,"num_write":98,"num_write_kb":22219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":64849907,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":7,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1f","length":"1"}]},{"pgid":"4.18","version":"135'201","reported_seq":711,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-19T11:31:58.069950+0000","last_change":"2024-03-19T11:31:56.212678+0000","last_active":"2024-03-19T11:31:58.069950+0000","last_peered":"2024-03-19T11:31:58.069950+0000","last_clean":"2024-03-19T11:31:58.069950+0000","last_became_active":"2024-03-19T11:31:06.066324+0000","last_became_peered":"2024-03-19T11:31:06.066324+0000","last_unstale":"2024-03-19T11:31:58.069950+0000","last_undegraded":"2024-03-19T11:31:58.069950+0000","last_fullsized":"2024-03-19T11:31:58.069950+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":5,"last_scrub":"131'190","last_scrub_stamp":"2024-03-19T11:31:53.489126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:31:53.489126+0000","objects_scrubbed":8,"log_size":201,"log_dups_size":0,"ondisk_log_size":201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:53.924761+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.043438013999999997,"stat_sum":{"num_bytes":17443326,"num_objects":8,"num_object_clones":6,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":133,"num_read_kb":167938,"num_write":48,"num_write_kb":14099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":19971477,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":7,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1d","length":"1"},{"start":"1f","length":"1"},{"start":"26","length":"1"},{"start":"29","length":"1"}]},{"pgid":"4.17","version":"41'24","reported_seq":34,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-19T11:31:59.079684+0000","last_change":"2024-03-19T11:31:59.079684+0000","last_active":"2024-03-19T11:31:59.079684+0000","last_peered":"2024-03-19T11:31:59.079684+0000","last_clean":"2024-03-19T11:31:59.079684+0000","last_became_active":"2024-03-19T11:31:56.188311+0000","last_became_peered":"2024-03-19T11:31:56.188311+0000","last_unstale":"2024-03-19T11:31:59.079684+0000","last_undegraded":"2024-03-19T11:31:59.079684+0000","last_fullsized":"2024-03-19T11:31:59.079684+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":5,"last_scrub":"41'24","last_scrub_stamp":"2024-03-19T11:31:56.997750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:31:56.997750+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:33:20.831798+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0015831319999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1a","length":"1"},{"start":"1d","length":"1"},{"start":"1f","length":"1"},{"start":"26","length":"1"},{"start":"29","length":"1"}]},{"pgid":"4.16","version":"133'192","reported_seq":459,"reported_epoch":136,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:57.180712+0000","last_change":"2024-03-19T11:31:56.310719+0000","last_active":"2024-03-19T11:31:57.180712+0000","last_peered":"2024-03-19T11:31:57.180712+0000","last_clean":"2024-03-19T11:31:57.180712+0000","last_became_active":"2024-03-19T11:31:56.193239+0000","last_became_peered":"2024-03-19T11:31:56.193239+0000","last_unstale":"2024-03-19T11:31:57.180712+0000","last_undegraded":"2024-03-19T11:31:57.180712+0000","last_fullsized":"2024-03-19T11:31:57.180712+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":5,"last_scrub":"49'56","last_scrub_stamp":"2024-03-19T11:30:29.212766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:29.212766+0000","objects_scrubbed":5,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.023633399999999999,"stat_sum":{"num_bytes":17227443,"num_objects":11,"num_object_clones":8,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":200,"num_read_kb":227440,"num_write":54,"num_write_kb":14599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":54759035,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":9,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1c","length":"1"},{"start":"1f","length":"1"}]},{"pgid":"4.15","version":"137'307","reported_seq":929,"reported_epoch":138,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:59.082995+0000","last_change":"2024-03-19T11:31:58.123657+0000","last_active":"2024-03-19T11:31:59.082995+0000","last_peered":"2024-03-19T11:31:59.082995+0000","last_clean":"2024-03-19T11:31:59.082995+0000","last_became_active":"2024-03-19T11:30:56.063100+0000","last_became_peered":"2024-03-19T11:30:56.063100+0000","last_unstale":"2024-03-19T11:31:59.082995+0000","last_undegraded":"2024-03-19T11:31:59.082995+0000","last_fullsized":"2024-03-19T11:31:59.082995+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":5,"last_scrub":"49'74","last_scrub_stamp":"2024-03-19T11:30:29.457542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:29.457542+0000","objects_scrubbed":7,"log_size":307,"log_dups_size":0,"ondisk_log_size":307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.038863723000000003,"stat_sum":{"num_bytes":26587769,"num_objects":16,"num_object_clones":11,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":3,"num_read":231,"num_read_kb":264670,"num_write":99,"num_write_kb":23318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":18744415,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":8,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1d","length":"1"},{"start":"1f","length":"1"}]},{"pgid":"4.14","version":"40'56","reported_seq":26,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-19T11:31:59.078946+0000","last_change":"2024-03-19T11:31:59.078946+0000","last_active":"2024-03-19T11:31:59.078946+0000","last_peered":"2024-03-19T11:31:59.078946+0000","last_clean":"2024-03-19T11:31:59.078946+0000","last_became_active":"2024-03-19T11:31:56.186206+0000","last_became_peered":"2024-03-19T11:31:56.186206+0000","last_unstale":"2024-03-19T11:31:59.078946+0000","last_undegraded":"2024-03-19T11:31:59.078946+0000","last_fullsized":"2024-03-19T11:31:59.078946+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":5,"last_scrub":"40'56","last_scrub_stamp":"2024-03-19T11:30:54.998282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:54.998282+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:10.857984+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001349729,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1a","length":"1"},{"start":"1d","length":"1"},{"start":"1f","length":"1"},{"start":"26","length":"1"},{"start":"29","length":"1"}]},{"pgid":"4.13","version":"135'71","reported_seq":388,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-19T11:31:58.071059+0000","last_change":"2024-03-19T11:31:56.213785+0000","last_active":"2024-03-19T11:31:58.071059+0000","last_peered":"2024-03-19T11:31:58.071059+0000","last_clean":"2024-03-19T11:31:58.071059+0000","last_became_active":"2024-03-19T11:30:45.245367+0000","last_became_peered":"2024-03-19T11:30:45.245367+0000","last_unstale":"2024-03-19T11:31:58.071059+0000","last_undegraded":"2024-03-19T11:31:58.071059+0000","last_fullsized":"2024-03-19T11:31:58.071059+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":5,"last_scrub":"131'63","last_scrub_stamp":"2024-03-19T11:31:52.522191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:31:52.522191+0000","objects_scrubbed":6,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:58.525303+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.041379873999999997,"stat_sum":{"num_bytes":12019598,"num_objects":6,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":127,"num_read_kb":174115,"num_write":25,"num_write_kb":7688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":3001789,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":6,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1d","length":"1"},{"start":"1f","length":"1"},{"start":"26","length":"1"},{"start":"29","length":"1"}]},{"pgid":"4.12","version":"28'6","reported_seq":252,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-19T11:31:59.085023+0000","last_change":"2024-03-19T11:31:59.085811+0000","last_active":"2024-03-19T11:31:59.085023+0000","last_peered":"2024-03-19T11:31:59.085023+0000","last_clean":"2024-03-19T11:31:59.085023+0000","last_became_active":"2024-03-19T11:31:00.064380+0000","last_became_peered":"2024-03-19T11:31:00.064380+0000","last_unstale":"2024-03-19T11:31:59.085023+0000","last_undegraded":"2024-03-19T11:31:59.085023+0000","last_fullsized":"2024-03-19T11:31:59.085023+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":5,"last_scrub":"28'6","last_scrub_stamp":"2024-03-19T11:31:04.263428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:31:04.263428+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:27.105760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01096308,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1a","length":"1"},{"start":"1d","length":"1"},{"start":"1f","length":"1"},{"start":"26","length":"1"},{"start":"29","length":"1"}]},{"pgid":"4.11","version":"138'121","reported_seq":459,"reported_epoch":138,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:59.108087+0000","last_change":"2024-03-19T11:31:59.091819+0000","last_active":"2024-03-19T11:31:59.108087+0000","last_peered":"2024-03-19T11:31:59.108087+0000","last_clean":"2024-03-19T11:31:59.108087+0000","last_became_active":"2024-03-19T11:31:59.090904+0000","last_became_peered":"2024-03-19T11:31:59.090904+0000","last_unstale":"2024-03-19T11:31:59.108087+0000","last_undegraded":"2024-03-19T11:31:59.108087+0000","last_fullsized":"2024-03-19T11:31:59.108087+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":5,"last_scrub":"119'108","last_scrub_stamp":"2024-03-19T11:31:41.414347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:31:41.414347+0000","objects_scrubbed":4,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:33:06.575242+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00051386100000000001,"stat_sum":{"num_bytes":8393046,"num_objects":5,"num_object_clones":4,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":69,"num_read_kb":88779,"num_write":30,"num_write_kb":8245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":22501418,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1d","length":"1"},{"start":"1f","length":"1"},{"start":"29","length":"1"}]},{"pgid":"4.10","version":"126'35","reported_seq":34,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-19T11:31:59.078016+0000","last_change":"2024-03-19T11:31:59.078016+0000","last_active":"2024-03-19T11:31:59.078016+0000","last_peered":"2024-03-19T11:31:59.078016+0000","last_clean":"2024-03-19T11:31:59.078016+0000","last_became_active":"2024-03-19T11:31:56.186088+0000","last_became_peered":"2024-03-19T11:31:56.186088+0000","last_unstale":"2024-03-19T11:31:59.078016+0000","last_undegraded":"2024-03-19T11:31:59.078016+0000","last_fullsized":"2024-03-19T11:31:59.078016+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":5,"last_scrub":"126'35","last_scrub_stamp":"2024-03-19T11:31:54.903318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:31:54.903318+0000","objects_scrubbed":1,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:33:12.338813+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00084686699999999995,"stat_sum":{"num_bytes":3462209,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":51,"num_read_kb":72483,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":10805708,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1a","length":"1"},{"start":"1d","length":"1"},{"start":"1f","length":"1"},{"start":"26","length":"1"},{"start":"29","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":66,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-19T11:31:58.075957+0000","last_change":"2024-03-19T11:31:28.155731+0000","last_active":"2024-03-19T11:31:58.075957+0000","last_peered":"2024-03-19T11:31:58.075957+0000","last_clean":"2024-03-19T11:31:58.075957+0000","last_became_active":"2024-03-19T11:31:28.155376+0000","last_became_peered":"2024-03-19T11:31:28.155376+0000","last_unstale":"2024-03-19T11:31:58.075957+0000","last_undegraded":"2024-03-19T11:31:58.075957+0000","last_fullsized":"2024-03-19T11:31:58.075957+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:31:08.381559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:31:08.381559+0000","last_clean_scrub_stamp":"2024-03-19T11:31:08.381559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:34.700388+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"59'92","reported_seq":225,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-19T11:31:58.070189+0000","last_change":"2024-03-19T11:31:56.187557+0000","last_active":"2024-03-19T11:31:58.070189+0000","last_peered":"2024-03-19T11:31:58.070189+0000","last_clean":"2024-03-19T11:31:58.070189+0000","last_became_active":"2024-03-19T11:31:03.052657+0000","last_became_peered":"2024-03-19T11:31:03.052657+0000","last_unstale":"2024-03-19T11:31:58.070189+0000","last_undegraded":"2024-03-19T11:31:58.070189+0000","last_fullsized":"2024-03-19T11:31:58.070189+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'92","last_scrub_stamp":"2024-03-19T11:30:41.351520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:41.351520+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.015637529000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1d","length":"1"},{"start":"1f","length":"1"},{"start":"26","length":"1"},{"start":"29","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":132,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-19T11:31:59.073730+0000","last_change":"2024-03-19T11:31:19.162627+0000","last_active":"2024-03-19T11:31:59.073730+0000","last_peered":"2024-03-19T11:31:59.073730+0000","last_clean":"2024-03-19T11:31:59.073730+0000","last_became_active":"2024-03-19T11:31:00.067099+0000","last_became_peered":"2024-03-19T11:31:00.067099+0000","last_unstale":"2024-03-19T11:31:59.073730+0000","last_undegraded":"2024-03-19T11:31:59.073730+0000","last_fullsized":"2024-03-19T11:31:59.073730+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:31:19.162377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:31:19.162377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:22.623613+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"133'113","reported_seq":626,"reported_epoch":138,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:59.093358+0000","last_change":"2024-03-19T11:31:59.091698+0000","last_active":"2024-03-19T11:31:59.093358+0000","last_peered":"2024-03-19T11:31:59.093358+0000","last_clean":"2024-03-19T11:31:59.093358+0000","last_became_active":"2024-03-19T11:31:59.090908+0000","last_became_peered":"2024-03-19T11:31:59.090908+0000","last_unstale":"2024-03-19T11:31:59.093358+0000","last_undegraded":"2024-03-19T11:31:59.093358+0000","last_fullsized":"2024-03-19T11:31:59.093358+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'57","last_scrub_stamp":"2024-03-19T11:30:43.792326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:43.792326+0000","objects_scrubbed":6,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00072825199999999998,"stat_sum":{"num_bytes":10107726,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":46,"num_read_kb":55657,"num_write":26,"num_write_kb":7803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":16468451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1d","length":"1"},{"start":"1f","length":"1"},{"start":"29","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":275,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-19T11:31:59.084310+0000","last_change":"2024-03-19T11:31:18.554690+0000","last_active":"2024-03-19T11:31:59.084310+0000","last_peered":"2024-03-19T11:31:59.084310+0000","last_clean":"2024-03-19T11:31:59.084310+0000","last_became_active":"2024-03-19T11:30:59.071925+0000","last_became_peered":"2024-03-19T11:30:59.071925+0000","last_unstale":"2024-03-19T11:31:59.084310+0000","last_undegraded":"2024-03-19T11:31:59.084310+0000","last_fullsized":"2024-03-19T11:31:59.084310+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:31:18.554377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:31:18.554377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:39.778063+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"134'71","reported_seq":527,"reported_epoch":136,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:57.180603+0000","last_change":"2024-03-19T11:31:56.122866+0000","last_active":"2024-03-19T11:31:57.180603+0000","last_peered":"2024-03-19T11:31:57.180603+0000","last_clean":"2024-03-19T11:31:57.180603+0000","last_became_active":"2024-03-19T11:31:54.075787+0000","last_became_peered":"2024-03-19T11:31:54.075787+0000","last_unstale":"2024-03-19T11:31:57.180603+0000","last_undegraded":"2024-03-19T11:31:57.180603+0000","last_fullsized":"2024-03-19T11:31:57.180603+0000","mapping_epoch":21,"log_start":"126'64","ondisk_log_start":"126'64","created":21,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"41'19","last_scrub_stamp":"2024-03-19T11:31:17.108816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:31:17.108816+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":51,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:18.681757+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.94360694499999997,"stat_sum":{"num_bytes":10334380,"num_objects":5,"num_object_clones":4,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":58950,"num_write":26,"num_write_kb":6942,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":18724730,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1c","length":"2"}]},{"pgid":"2.4","version":"0'0","reported_seq":260,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-19T11:31:59.074391+0000","last_change":"2024-03-19T11:30:55.366872+0000","last_active":"2024-03-19T11:31:59.074391+0000","last_peered":"2024-03-19T11:31:59.074391+0000","last_clean":"2024-03-19T11:31:59.074391+0000","last_became_active":"2024-03-19T11:29:46.540939+0000","last_became_peered":"2024-03-19T11:29:46.540939+0000","last_unstale":"2024-03-19T11:31:59.074391+0000","last_undegraded":"2024-03-19T11:31:59.074391+0000","last_fullsized":"2024-03-19T11:31:59.074391+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:30:55.366631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:30:55.366631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:10.640223+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"138'49","reported_seq":325,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-19T11:31:59.081141+0000","last_change":"2024-03-19T11:31:59.082430+0000","last_active":"2024-03-19T11:31:59.081141+0000","last_peered":"2024-03-19T11:31:59.081141+0000","last_clean":"2024-03-19T11:31:59.081141+0000","last_became_active":"2024-03-19T11:31:58.089321+0000","last_became_peered":"2024-03-19T11:31:58.089321+0000","last_unstale":"2024-03-19T11:31:59.081141+0000","last_undegraded":"2024-03-19T11:31:59.081141+0000","last_fullsized":"2024-03-19T11:31:59.081141+0000","mapping_epoch":80,"log_start":"133'45","ondisk_log_start":"133'45","created":21,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-03-19T11:31:05.279955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:31:05.279955+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":45,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:23.658123+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0084113230000000001,"stat_sum":{"num_bytes":7251417,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":58,"num_read_kb":79646,"num_write":15,"num_write_kb":4421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1a","length":"1"},{"start":"1d","length":"1"},{"start":"1f","length":"1"},{"start":"26","length":"1"},{"start":"29","length":"1"}]},{"pgid":"3.3","version":"133'147","reported_seq":507,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-19T11:31:58.069936+0000","last_change":"2024-03-19T11:31:16.639529+0000","last_active":"2024-03-19T11:31:58.069936+0000","last_peered":"2024-03-19T11:31:58.069936+0000","last_clean":"2024-03-19T11:31:58.069936+0000","last_became_active":"2024-03-19T11:31:06.066161+0000","last_became_peered":"2024-03-19T11:31:06.066161+0000","last_unstale":"2024-03-19T11:31:58.069936+0000","last_undegraded":"2024-03-19T11:31:58.069936+0000","last_fullsized":"2024-03-19T11:31:58.069936+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'109","last_scrub_stamp":"2024-03-19T11:31:16.639357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:31:16.639357+0000","objects_scrubbed":47,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:31.688365+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47972149,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":114,"num_read_kb":8539,"num_write":157,"num_write_kb":71520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":25111105,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":259,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-19T11:31:59.093524+0000","last_change":"2024-03-19T11:31:17.684706+0000","last_active":"2024-03-19T11:31:59.093524+0000","last_peered":"2024-03-19T11:31:59.093524+0000","last_clean":"2024-03-19T11:31:59.093524+0000","last_became_active":"2024-03-19T11:29:46.320679+0000","last_became_peered":"2024-03-19T11:29:46.320679+0000","last_unstale":"2024-03-19T11:31:59.093524+0000","last_undegraded":"2024-03-19T11:31:59.093524+0000","last_fullsized":"2024-03-19T11:31:59.093524+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:31:17.684492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:31:17.684492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:21.475454+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"138'237","reported_seq":879,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-19T11:31:59.097397+0000","last_change":"2024-03-19T11:31:59.098052+0000","last_active":"2024-03-19T11:31:59.097397+0000","last_peered":"2024-03-19T11:31:59.097397+0000","last_clean":"2024-03-19T11:31:59.097397+0000","last_became_active":"2024-03-19T11:31:50.083216+0000","last_became_peered":"2024-03-19T11:31:50.083216+0000","last_unstale":"2024-03-19T11:31:59.097397+0000","last_undegraded":"2024-03-19T11:31:59.097397+0000","last_fullsized":"2024-03-19T11:31:59.097397+0000","mapping_epoch":83,"log_start":"123'187","ondisk_log_start":"123'187","created":21,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'119","last_scrub_stamp":"2024-03-19T11:30:52.512203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:52.512203+0000","objects_scrubbed":7,"log_size":50,"log_dups_size":220,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:09.111167+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.023880894999999999,"stat_sum":{"num_bytes":25374094,"num_objects":14,"num_object_clones":9,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":275,"num_read_kb":344748,"num_write":43,"num_write_kb":10176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":6111103,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":9,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1a","length":"1"},{"start":"1d","length":"1"},{"start":"1f","length":"1"},{"start":"26","length":"1"},{"start":"29","length":"1"}]},{"pgid":"3.0","version":"136'173","reported_seq":438,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-19T11:31:59.073576+0000","last_change":"2024-03-19T11:31:32.260387+0000","last_active":"2024-03-19T11:31:59.073576+0000","last_peered":"2024-03-19T11:31:59.073576+0000","last_clean":"2024-03-19T11:31:59.073576+0000","last_became_active":"2024-03-19T11:29:57.549928+0000","last_became_peered":"2024-03-19T11:29:57.549928+0000","last_unstale":"2024-03-19T11:31:59.073576+0000","last_undegraded":"2024-03-19T11:31:59.073576+0000","last_fullsized":"2024-03-19T11:31:59.073576+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'152","last_scrub_stamp":"2024-03-19T11:31:32.260193+0000","last_deep_scrub":"112'152","last_deep_scrub_stamp":"2024-03-19T11:31:32.260193+0000","last_clean_scrub_stamp":"2024-03-19T11:31:32.260193+0000","objects_scrubbed":33,"log_size":173,"log_dups_size":0,"ondisk_log_size":173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:33.590837+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33630026,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":113,"num_read_kb":3273,"num_write":156,"num_write_kb":57847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":271,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-19T11:31:59.073941+0000","last_change":"2024-03-19T11:31:56.189583+0000","last_active":"2024-03-19T11:31:59.073941+0000","last_peered":"2024-03-19T11:31:59.073941+0000","last_clean":"2024-03-19T11:31:59.073941+0000","last_became_active":"2024-03-19T11:31:56.189192+0000","last_became_peered":"2024-03-19T11:31:56.189192+0000","last_unstale":"2024-03-19T11:31:59.073941+0000","last_undegraded":"2024-03-19T11:31:59.073941+0000","last_fullsized":"2024-03-19T11:31:59.073941+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:31:28.553022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:31:28.553022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:36.687185+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"138'88","reported_seq":279,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-19T11:31:59.081108+0000","last_change":"2024-03-19T11:31:59.081677+0000","last_active":"2024-03-19T11:31:59.081108+0000","last_peered":"2024-03-19T11:31:59.081108+0000","last_clean":"2024-03-19T11:31:59.081108+0000","last_became_active":"2024-03-19T11:31:24.063043+0000","last_became_peered":"2024-03-19T11:31:24.063043+0000","last_unstale":"2024-03-19T11:31:59.081108+0000","last_undegraded":"2024-03-19T11:31:59.081108+0000","last_fullsized":"2024-03-19T11:31:59.081108+0000","mapping_epoch":80,"log_start":"99'63","ondisk_log_start":"99'63","created":21,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'40","last_scrub_stamp":"2024-03-19T11:30:56.505586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:56.505586+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":63,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:11.588649+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0076167279999999997,"stat_sum":{"num_bytes":8284354,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":28,"num_read_kb":33089,"num_write":14,"num_write_kb":4191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":7070511,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1a","length":"1"},{"start":"1d","length":"1"},{"start":"1f","length":"1"},{"start":"26","length":"1"},{"start":"29","length":"1"}]},{"pgid":"3.1","version":"126'136","reported_seq":532,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-03-19T11:31:57.181696+0000","last_change":"2024-03-19T11:31:31.021284+0000","last_active":"2024-03-19T11:31:57.181696+0000","last_peered":"2024-03-19T11:31:57.181696+0000","last_clean":"2024-03-19T11:31:57.181696+0000","last_became_active":"2024-03-19T11:31:03.077272+0000","last_became_peered":"2024-03-19T11:31:03.077272+0000","last_unstale":"2024-03-19T11:31:57.181696+0000","last_undegraded":"2024-03-19T11:31:57.181696+0000","last_fullsized":"2024-03-19T11:31:57.181696+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'122","last_scrub_stamp":"2024-03-19T11:31:31.021132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:31:31.021132+0000","objects_scrubbed":44,"log_size":136,"log_dups_size":0,"ondisk_log_size":136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:31.636517+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47435916,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":106,"num_read_kb":6676,"num_write":143,"num_write_kb":65676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":42087532,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":270,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-19T11:31:59.084651+0000","last_change":"2024-03-19T11:31:21.625119+0000","last_active":"2024-03-19T11:31:59.084651+0000","last_peered":"2024-03-19T11:31:59.084651+0000","last_clean":"2024-03-19T11:31:59.084651+0000","last_became_active":"2024-03-19T11:31:00.068989+0000","last_became_peered":"2024-03-19T11:31:00.068989+0000","last_unstale":"2024-03-19T11:31:59.084651+0000","last_undegraded":"2024-03-19T11:31:59.084651+0000","last_fullsized":"2024-03-19T11:31:59.084651+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:31:21.624883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:31:21.624883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:33.811304+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"133'91","reported_seq":302,"reported_epoch":136,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:57.181038+0000","last_change":"2024-03-19T11:31:56.175425+0000","last_active":"2024-03-19T11:31:57.181038+0000","last_peered":"2024-03-19T11:31:57.181038+0000","last_clean":"2024-03-19T11:31:57.181038+0000","last_became_active":"2024-03-19T11:31:28.085766+0000","last_became_peered":"2024-03-19T11:31:28.085766+0000","last_unstale":"2024-03-19T11:31:57.181038+0000","last_undegraded":"2024-03-19T11:31:57.181038+0000","last_fullsized":"2024-03-19T11:31:57.181038+0000","mapping_epoch":79,"log_start":"89'77","ondisk_log_start":"89'77","created":21,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'65","last_scrub_stamp":"2024-03-19T11:30:53.385414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:53.385414+0000","objects_scrubbed":3,"log_size":14,"log_dups_size":41,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:07.324011+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.94513740400000001,"stat_sum":{"num_bytes":4635797,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":90,"num_read_kb":137573,"num_write":11,"num_write_kb":2729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4209633,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1c","length":"2"},{"start":"1f","length":"1"},{"start":"29","length":"1"}]},{"pgid":"3.2","version":"131'134","reported_seq":511,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-19T11:31:59.103962+0000","last_change":"2024-03-19T11:31:59.102989+0000","last_active":"2024-03-19T11:31:59.103962+0000","last_peered":"2024-03-19T11:31:59.103962+0000","last_clean":"2024-03-19T11:31:59.103962+0000","last_became_active":"2024-03-19T11:31:59.102401+0000","last_became_peered":"2024-03-19T11:31:59.102401+0000","last_unstale":"2024-03-19T11:31:59.103962+0000","last_undegraded":"2024-03-19T11:31:59.103962+0000","last_fullsized":"2024-03-19T11:31:59.103962+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'98","last_scrub_stamp":"2024-03-19T11:31:18.907500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:56.503774+0000","last_clean_scrub_stamp":"2024-03-19T11:31:18.907500+0000","objects_scrubbed":39,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:22.495854+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46303452,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":119,"num_read_kb":10556,"num_write":149,"num_write_kb":59101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":28699794,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":259,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-19T11:31:59.082555+0000","last_change":"2024-03-19T11:31:04.588296+0000","last_active":"2024-03-19T11:31:59.082555+0000","last_peered":"2024-03-19T11:31:59.082555+0000","last_clean":"2024-03-19T11:31:59.082555+0000","last_became_active":"2024-03-19T11:29:46.321760+0000","last_became_peered":"2024-03-19T11:29:46.321760+0000","last_unstale":"2024-03-19T11:31:59.082555+0000","last_undegraded":"2024-03-19T11:31:59.082555+0000","last_fullsized":"2024-03-19T11:31:59.082555+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:31:04.588071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:45.187160+0000","last_clean_scrub_stamp":"2024-03-19T11:31:04.588071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:11.976969+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":436,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-19T11:31:59.084775+0000","last_change":"2024-03-19T11:31:03.367805+0000","last_active":"2024-03-19T11:31:59.084775+0000","last_peered":"2024-03-19T11:31:59.084775+0000","last_clean":"2024-03-19T11:31:59.084775+0000","last_became_active":"2024-03-19T11:30:59.073709+0000","last_became_peered":"2024-03-19T11:30:59.073709+0000","last_unstale":"2024-03-19T11:31:59.084775+0000","last_undegraded":"2024-03-19T11:31:59.084775+0000","last_fullsized":"2024-03-19T11:31:59.084775+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2024-03-19T11:31:03.367669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:43.166618+0000","last_clean_scrub_stamp":"2024-03-19T11:31:03.367669+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:29.977573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"135'142","reported_seq":768,"reported_epoch":137,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:58.075830+0000","last_change":"2024-03-19T11:31:56.186228+0000","last_active":"2024-03-19T11:31:58.075830+0000","last_peered":"2024-03-19T11:31:58.075830+0000","last_clean":"2024-03-19T11:31:58.075830+0000","last_became_active":"2024-03-19T11:31:33.081858+0000","last_became_peered":"2024-03-19T11:31:33.081858+0000","last_unstale":"2024-03-19T11:31:58.075830+0000","last_undegraded":"2024-03-19T11:31:58.075830+0000","last_fullsized":"2024-03-19T11:31:58.075830+0000","mapping_epoch":83,"log_start":"108'129","ondisk_log_start":"108'129","created":21,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'81","last_scrub_stamp":"2024-03-19T11:30:52.360396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:52.360396+0000","objects_scrubbed":6,"log_size":13,"log_dups_size":82,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:04.828041+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.90835851899999998,"stat_sum":{"num_bytes":12695531,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":97,"num_read_kb":120837,"num_write":17,"num_write_kb":4280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":10558635,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1d","length":"1"},{"start":"1f","length":"1"},{"start":"29","length":"1"}]},{"pgid":"4.8","version":"138'162","reported_seq":625,"reported_epoch":138,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:59.083091+0000","last_change":"2024-03-19T11:31:59.083091+0000","last_active":"2024-03-19T11:31:59.083091+0000","last_peered":"2024-03-19T11:31:59.083091+0000","last_clean":"2024-03-19T11:31:59.083091+0000","last_became_active":"2024-03-19T11:31:20.080111+0000","last_became_peered":"2024-03-19T11:31:20.080111+0000","last_unstale":"2024-03-19T11:31:59.083091+0000","last_undegraded":"2024-03-19T11:31:59.083091+0000","last_fullsized":"2024-03-19T11:31:59.083091+0000","mapping_epoch":21,"log_start":"97'116","ondisk_log_start":"97'116","created":21,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'71","last_scrub_stamp":"2024-03-19T11:30:53.381386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:53.381386+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":80,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:04.288706+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.109067308,"stat_sum":{"num_bytes":15539898,"num_objects":7,"num_object_clones":5,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":38,"num_read_kb":30885,"num_write":39,"num_write_kb":10510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":6,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1d","length":"1"},{"start":"1f","length":"1"},{"start":"26","length":"1"},{"start":"29","length":"1"}]},{"pgid":"4.9","version":"135'94","reported_seq":644,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-19T11:31:58.070818+0000","last_change":"2024-03-19T11:31:56.202159+0000","last_active":"2024-03-19T11:31:58.070818+0000","last_peered":"2024-03-19T11:31:58.070818+0000","last_clean":"2024-03-19T11:31:58.070818+0000","last_became_active":"2024-03-19T11:31:16.087517+0000","last_became_peered":"2024-03-19T11:31:16.087517+0000","last_unstale":"2024-03-19T11:31:58.070818+0000","last_undegraded":"2024-03-19T11:31:58.070818+0000","last_fullsized":"2024-03-19T11:31:58.070818+0000","mapping_epoch":80,"log_start":"90'70","ondisk_log_start":"90'70","created":21,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'55","last_scrub_stamp":"2024-03-19T11:30:54.706235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:54.706235+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":46,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:03.301897+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.032118059999999997,"stat_sum":{"num_bytes":7965327,"num_objects":4,"num_object_clones":3,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":75,"num_read_kb":92778,"num_write":14,"num_write_kb":4587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":5842489,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1d","length":"1"},{"start":"1f","length":"1"},{"start":"26","length":"1"},{"start":"29","length":"1"}]},{"pgid":"4.a","version":"135'137","reported_seq":655,"reported_epoch":137,"state":"active+clean+snaptrim","last_fresh":"2024-03-19T11:31:58.076599+0000","last_change":"2024-03-19T11:31:56.192510+0000","last_active":"2024-03-19T11:31:58.076599+0000","last_peered":"2024-03-19T11:31:58.076599+0000","last_clean":"2024-03-19T11:31:58.076599+0000","last_became_active":"2024-03-19T11:31:12.075247+0000","last_became_peered":"2024-03-19T11:31:12.075247+0000","last_unstale":"2024-03-19T11:31:58.076599+0000","last_undegraded":"2024-03-19T11:31:58.076599+0000","last_fullsized":"2024-03-19T11:31:58.076599+0000","mapping_epoch":21,"log_start":"87'100","ondisk_log_start":"87'100","created":21,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'77","last_scrub_stamp":"2024-03-19T11:30:55.291823+0000","last_deep_scrub":"62'77","last_deep_scrub_stamp":"2024-03-19T11:30:55.291823+0000","last_clean_scrub_stamp":"2024-03-19T11:30:55.291823+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":59,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:24.826010+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.91028832100000001,"stat_sum":{"num_bytes":11437806,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":94,"num_read_kb":130857,"num_write":13,"num_write_kb":3949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1d","length":"1"},{"start":"1f","length":"1"},{"start":"29","length":"1"}]},{"pgid":"4.b","version":"123'70","reported_seq":603,"reported_epoch":136,"state":"active+clean+snaptrim_wait","last_fresh":"2024-03-19T11:31:57.180362+0000","last_change":"2024-03-19T11:31:56.177623+0000","last_active":"2024-03-19T11:31:57.180362+0000","last_peered":"2024-03-19T11:31:57.180362+0000","last_clean":"2024-03-19T11:31:57.180362+0000","last_became_active":"2024-03-19T11:31:08.066470+0000","last_became_peered":"2024-03-19T11:31:08.066470+0000","last_unstale":"2024-03-19T11:31:57.180362+0000","last_undegraded":"2024-03-19T11:31:57.180362+0000","last_fullsized":"2024-03-19T11:31:57.180362+0000","mapping_epoch":21,"log_start":"85'64","ondisk_log_start":"85'64","created":21,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'62","last_scrub_stamp":"2024-03-19T11:30:56.095039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:56.095039+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":30,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:03.683317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.92171077599999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1c","length":"2"},{"start":"1f","length":"1"},{"start":"29","length":"1"}]},{"pgid":"4.c","version":"56'29","reported_seq":237,"reported_epoch":137,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-19T11:31:58.070270+0000","last_change":"2024-03-19T11:31:57.195416+0000","last_active":"2024-03-19T11:31:58.070270+0000","last_peered":"2024-03-19T11:31:58.070270+0000","last_clean":"2024-03-19T11:31:54.083555+0000","last_became_active":"2024-03-19T11:31:57.192944+0000","last_became_peered":"2024-03-19T11:31:57.192944+0000","last_unstale":"2024-03-19T11:31:58.070270+0000","last_undegraded":"2024-03-19T11:31:58.070270+0000","last_fullsized":"2024-03-19T11:31:58.070270+0000","mapping_epoch":135,"log_start":"56'29","ondisk_log_start":"56'29","created":21,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'29","last_scrub_stamp":"2024-03-19T11:30:53.743385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:53.743385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011354115,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,6],"avail_no_missing":["4","3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1d","length":"1"},{"start":"1f","length":"1"},{"start":"29","length":"1"}]},{"pgid":"4.d","version":"137'141","reported_seq":689,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-19T11:31:58.184190+0000","last_change":"2024-03-19T11:31:56.205554+0000","last_active":"2024-03-19T11:31:58.184190+0000","last_peered":"2024-03-19T11:31:58.184190+0000","last_clean":"2024-03-19T11:31:58.184190+0000","last_became_active":"2024-03-19T11:31:06.068526+0000","last_became_peered":"2024-03-19T11:31:06.068526+0000","last_unstale":"2024-03-19T11:31:58.184190+0000","last_undegraded":"2024-03-19T11:31:58.184190+0000","last_fullsized":"2024-03-19T11:31:58.184190+0000","mapping_epoch":86,"log_start":"70'71","ondisk_log_start":"70'71","created":21,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'73","last_scrub_stamp":"2024-03-19T11:30:53.519178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:53.519178+0000","objects_scrubbed":6,"log_size":70,"log_dups_size":50,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:11.945845+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.034326192999999998,"stat_sum":{"num_bytes":14282247,"num_objects":9,"num_object_clones":7,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":86,"num_read_kb":99466,"num_write":32,"num_write_kb":8752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":8146998,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1d","length":"1"},{"start":"1f","length":"1"},{"start":"26","length":"1"},{"start":"29","length":"1"}]},{"pgid":"4.e","version":"135'139","reported_seq":632,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-19T11:31:58.071245+0000","last_change":"2024-03-19T11:31:56.197090+0000","last_active":"2024-03-19T11:31:58.071245+0000","last_peered":"2024-03-19T11:31:58.071245+0000","last_clean":"2024-03-19T11:31:58.071245+0000","last_became_active":"2024-03-19T11:30:38.060487+0000","last_became_peered":"2024-03-19T11:30:38.060487+0000","last_unstale":"2024-03-19T11:31:58.071245+0000","last_undegraded":"2024-03-19T11:31:58.071245+0000","last_fullsized":"2024-03-19T11:31:58.071245+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'65","last_scrub_stamp":"2024-03-19T11:30:43.824620+0000","last_deep_scrub":"65'65","last_deep_scrub_stamp":"2024-03-19T11:30:43.824620+0000","last_clean_scrub_stamp":"2024-03-19T11:30:43.824620+0000","objects_scrubbed":4,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:05.376518+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0.027103848999999999,"stat_sum":{"num_bytes":9271278,"num_objects":8,"num_object_clones":6,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":2,"num_read":77,"num_read_kb":92209,"num_write":23,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1d","length":"1"},{"start":"1f","length":"1"},{"start":"26","length":"1"},{"start":"29","length":"1"}]},{"pgid":"4.f","version":"0'0","reported_seq":474,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-03-19T11:31:59.082836+0000","last_change":"2024-03-19T11:31:59.084894+0000","last_active":"2024-03-19T11:31:59.082836+0000","last_peered":"2024-03-19T11:31:59.082836+0000","last_clean":"2024-03-19T11:31:59.082836+0000","last_became_active":"2024-03-19T11:31:56.190300+0000","last_became_peered":"2024-03-19T11:31:56.190300+0000","last_unstale":"2024-03-19T11:31:59.082836+0000","last_undegraded":"2024-03-19T11:31:59.082836+0000","last_fullsized":"2024-03-19T11:31:59.082836+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T11:30:43.411893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T11:29:57.528796+0000","last_clean_scrub_stamp":"2024-03-19T11:30:43.411893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:32:05.297978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01047794,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1a","length":"1"},{"start":"1d","length":"1"},{"start":"1f","length":"1"},{"start":"26","length":"1"},{"start":"29","length":"1"}]}],"pool_stats":[{"poolid":4,"num_pg":34,"stat_sum":{"num_bytes":316661904,"num_objects":165,"num_object_clones":116,"num_object_copies":330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":7,"num_read":2612,"num_read_kb":3204033,"num_write":773,"num_write_kb":205221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":381,"num_bytes_recovered":407570754,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":94,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":501661696,"data_stored":627953853,"data_compressed":3563344,"data_compressed_allocated":2351104,"data_compressed_original":8362271,"omap_allocated":0,"internal_metadata":0},"log_size":2679,"ondisk_log_size":2679,"up":68,"acting":68,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":175341543,"num_objects":183,"num_object_clones":0,"num_object_copies":366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":183,"num_whiteouts":0,"num_read":452,"num_read_kb":29044,"num_write":605,"num_write_kb":254144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":95898431,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210178048,"data_stored":226509170,"data_compressed":14452290,"data_compressed_allocated":15441920,"data_compressed_original":32243712,"omap_allocated":0,"internal_metadata":0},"log_size":590,"ondisk_log_size":590,"up":8,"acting":8,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":1586784,"data_compressed":19832,"data_compressed_allocated":98304,"data_compressed_original":1560576,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542175,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76788,"kb_used_data":45604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94295052,"statfs":{"total":96636764160,"available":96558133248,"internally_reserved":0,"allocated":46698496,"data_stored":64699099,"data_compressed":50665,"data_compressed_allocated":81920,"data_compressed_original":774144,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542175,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":120232,"kb_used_data":89044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94251608,"statfs":{"total":96636764160,"available":96513646592,"internally_reserved":0,"allocated":91181056,"data_stored":109224774,"data_compressed":1990609,"data_compressed_allocated":2039808,"data_compressed_original":4579328,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542175,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":147836,"kb_used_data":116656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94224004,"statfs":{"total":96636764160,"available":96485380096,"internally_reserved":0,"allocated":119455744,"data_stored":126390520,"data_compressed":2543178,"data_compressed_allocated":2617344,"data_compressed_original":5961299,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542174,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":101440,"kb_used_data":70252,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94270400,"statfs":{"total":96636764160,"available":96532889600,"internally_reserved":0,"allocated":71938048,"data_stored":85605133,"data_compressed":1857338,"data_compressed_allocated":1679360,"data_compressed_original":4339543,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542174,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":106432,"kb_used_data":71092,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94265408,"statfs":{"total":96636764160,"available":96527777792,"internally_reserved":0,"allocated":72798208,"data_stored":87773115,"data_compressed":1792009,"data_compressed_allocated":1613824,"data_compressed_original":4011863,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574878,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":110460,"kb_used_data":79276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94261380,"statfs":{"total":96636764160,"available":96523653120,"internally_reserved":0,"allocated":81178624,"data_stored":101553237,"data_compressed":2045893,"data_compressed_allocated":2068480,"data_compressed_original":4673536,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574878,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":178832,"kb_used_data":143484,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94193008,"statfs":{"total":96636764160,"available":96453640192,"internally_reserved":0,"allocated":146927616,"data_stored":177954349,"data_compressed":4404517,"data_compressed_allocated":4472832,"data_compressed_original":9745438,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574878,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":144252,"kb_used_data":113072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94227588,"statfs":{"total":96636764160,"available":96489050112,"internally_reserved":0,"allocated":115785728,"data_stored":135786602,"data_compressed":4534754,"data_compressed_allocated":4575232,"data_compressed_original":10579968,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,4],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6884,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":528928,"data_compressed":6884,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":6064,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":56348672,"data_stored":60669701,"data_compressed":3937417,"data_compressed_allocated":4186112,"data_compressed_original":8646656,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":18419712,"data_stored":19947342,"data_compressed":1240233,"data_compressed_allocated":1335296,"data_compressed_original":2904064,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":18419712,"data_stored":19947342,"data_compressed":1240233,"data_compressed_allocated":1335296,"data_compressed_original":2904064,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":26349568,"data_stored":28463762,"data_compressed":1762457,"data_compressed_allocated":1896448,"data_compressed_original":4055040,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":56348672,"data_stored":60669701,"data_compressed":3937417,"data_compressed_allocated":4186112,"data_compressed_original":8646656,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":26349568,"data_stored":28463762,"data_compressed":1762457,"data_compressed_allocated":1896448,"data_compressed_original":4055040,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":7942144,"data_stored":8347560,"data_compressed":572076,"data_compressed_allocated":606208,"data_compressed_original":1032192,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57851904,"data_stored":73761391,"data_compressed":590453,"data_compressed_allocated":356352,"data_compressed_original":1409024,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":52846592,"data_stored":66991901,"data_compressed":551776,"data_compressed_allocated":278528,"data_compressed_original":1107799,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":51982336,"data_stored":64822859,"data_compressed":617105,"data_compressed_allocated":344064,"data_compressed_original":1435479,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90734592,"data_stored":94932233,"data_compressed":780721,"data_compressed_allocated":720896,"data_compressed_original":1906259,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":89047040,"data_stored":116453964,"data_compressed":467100,"data_compressed_allocated":286720,"data_compressed_original":1098782,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":62042112,"data_stored":77660560,"data_compressed":228152,"data_compressed_allocated":143360,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53276672,"data_stored":72257669,"data_compressed":283436,"data_compressed_allocated":172032,"data_compressed_original":618496,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":43880448,"data_stored":61073276,"data_compressed":44601,"data_compressed_allocated":49152,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T11:32:00.577 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pgp_num 26 2024-03-19T11:32:00.665 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:32:00.673+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:32:00.675 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:32:00.675 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:32:00.675 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:32:00.675 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:32:00.675 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.675 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1217 4.3 4:c8b09e87:::smithi03829856-15:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:32:00.675 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.155966+0000", 2024-03-19T11:32:00.675 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 102.526790381, 2024-03-19T11:32:00.675 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.023984647000000001, 2024-03-19T11:32:00.676 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:00.676 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:00.676 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:00.676 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:00.676 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:00.676 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1217 2024-03-19T11:32:00.676 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.676 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:00.676 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.676 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:00.676 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155966+0000", 2024-03-19T11:32:00.676 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:00.676 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.677 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.677 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:00.677 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155966+0000", 2024-03-19T11:32:00.677 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:00.677 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.677 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.677 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:00.677 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155968+0000", 2024-03-19T11:32:00.677 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.795e-06 2024-03-19T11:32:00.677 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.677 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.677 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:00.678 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155978+0000", 2024-03-19T11:32:00.678 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0574000000000001e-05 2024-03-19T11:32:00.678 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.678 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.678 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:00.678 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.156000+0000", 2024-03-19T11:32:00.678 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1250000000000002e-05 2024-03-19T11:32:00.678 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.678 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.678 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:00.678 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.156010+0000", 2024-03-19T11:32:00.679 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0168e-05 2024-03-19T11:32:00.679 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.679 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.679 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.679 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.156119+0000", 2024-03-19T11:32:00.679 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010950699999999999 2024-03-19T11:32:00.679 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.679 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.679 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:00.679 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.156231+0000", 2024-03-19T11:32:00.679 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000111445 2024-03-19T11:32:00.679 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.679 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.680 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:00.680 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.179951+0000", 2024-03-19T11:32:00.680 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.023719908000000001 2024-03-19T11:32:00.680 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:00.680 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:00.680 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:00.680 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.680 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.680 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.24 75/74 [PushOp(4:27909373:::smithi03829856-46:8, version: 46'59, data_included: [756315~767970,1933572~280606], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:8@46'59, size: 2756315, copy_subset: [(0, 2756315)], clone_subset: {}, snapset: 10=[]:{8: [6], c: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2214178, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:266590a5:::smithi03829856-16:e, version: 70'107, data_included: [652105~662359,1728171~386217], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:e@70'107, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 10=[]:{e: [d, b, 6]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:266590a5:::smithi03829856-16:head, version: 70'108, data_included: [652105~662359,1728171~386217], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:head@70'108, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:27909373:::smithi03829856-46:c, version: 70'109, data_included: [628595~895690,1807440~152886], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:c@70'109, size: 3428595, copy_subset: [(0, 3428595)], clone_subset: {}, snapset: 10=[]:{8: [6], c: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1960326, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:27909373:::smithi03829856-46:head, version: 70'110, data_included: [403401~454961,1584925~506039], data_size: 961000, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:head@70'110, size: 2403401, copy_subset: [(0, 2403401)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2403401, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:32:00.680 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.099395+0000", 2024-03-19T11:32:00.680 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 66.58336122, 2024-03-19T11:32:00.680 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.045391157000000001, 2024-03-19T11:32:00.681 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:00.681 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:00.681 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:00.681 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.681 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:00.681 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.099395+0000", 2024-03-19T11:32:00.681 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:00.681 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.681 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.681 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:00.681 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.099395+0000", 2024-03-19T11:32:00.681 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:00.681 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.681 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.682 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:00.682 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.099397+0000", 2024-03-19T11:32:00.682 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.691e-06 2024-03-19T11:32:00.682 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.682 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.682 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:00.682 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105009+0000", 2024-03-19T11:32:00.682 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0056121349999999999 2024-03-19T11:32:00.682 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.682 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.682 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:00.683 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105077+0000", 2024-03-19T11:32:00.683 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8561999999999998e-05 2024-03-19T11:32:00.683 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.683 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.683 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:00.683 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105133+0000", 2024-03-19T11:32:00.683 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5590000000000001e-05 2024-03-19T11:32:00.683 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.683 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.683 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.683 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105531+0000", 2024-03-19T11:32:00.683 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00039756200000000001 2024-03-19T11:32:00.683 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.684 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.684 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:00.684 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105579+0000", 2024-03-19T11:32:00.684 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8028999999999997e-05 2024-03-19T11:32:00.684 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.684 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.684 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:00.684 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.144786+0000", 2024-03-19T11:32:00.684 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.039207588000000002 2024-03-19T11:32:00.684 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:00.684 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:00.684 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:00.685 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.685 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.685 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.24 e75,74 4:266590a5:::smithi03829856-16:4,47'63,4:27909373:::smithi03829856-46:5,55'79,4:266590a5:::smithi03829856-16:10,67'105)", 2024-03-19T11:32:00.685 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.105191+0000", 2024-03-19T11:32:00.685 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 66.577565120000003, 2024-03-19T11:32:00.685 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.039580830999999997, 2024-03-19T11:32:00.685 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:00.685 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:00.685 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:00.685 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.685 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:00.685 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105191+0000", 2024-03-19T11:32:00.685 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:00.686 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.686 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.686 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:00.686 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105191+0000", 2024-03-19T11:32:00.686 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:00.686 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.686 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.686 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:00.686 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105192+0000", 2024-03-19T11:32:00.686 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.099e-06 2024-03-19T11:32:00.686 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.686 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.686 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:00.687 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105200+0000", 2024-03-19T11:32:00.687 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5279999999999998e-06 2024-03-19T11:32:00.687 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.687 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.687 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:00.687 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105211+0000", 2024-03-19T11:32:00.687 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1471000000000001e-05 2024-03-19T11:32:00.687 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.687 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.687 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:00.687 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105220+0000", 2024-03-19T11:32:00.687 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3469999999999999e-06 2024-03-19T11:32:00.687 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.688 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.688 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.688 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.144219+0000", 2024-03-19T11:32:00.688 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.038999024 2024-03-19T11:32:00.688 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.688 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.688 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:00.688 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.144257+0000", 2024-03-19T11:32:00.688 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8501999999999998e-05 2024-03-19T11:32:00.688 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.688 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.688 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:00.688 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.144772+0000", 2024-03-19T11:32:00.689 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00051486000000000001 2024-03-19T11:32:00.689 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:00.689 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:00.689 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:00.689 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.689 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.689 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 75/74 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:7, version: 55'82, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@55'82, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 58'86, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@58'86, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [504210~968657,1572695~79919], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:12, version: 66'101, data_included: [440350~816629,1309629~231947], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:12@66'101, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:10, version: 70'112, data_included: [440350~816629,1309629~231947], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:10@70'112, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 70'113, data_included: [440350~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@70'113, size: 2605085, copy_subset: [(0, 2605085)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1488926, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:32:00.689 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.107266+0000", 2024-03-19T11:32:00.689 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 66.575490603000006, 2024-03-19T11:32:00.689 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.048592669999999998, 2024-03-19T11:32:00.689 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:00.689 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:00.689 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:00.690 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.690 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:00.690 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.107266+0000", 2024-03-19T11:32:00.690 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:00.690 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.690 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.690 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:00.690 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.107266+0000", 2024-03-19T11:32:00.690 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:00.690 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.690 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.690 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:00.690 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.107267+0000", 2024-03-19T11:32:00.690 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1459999999999999e-06 2024-03-19T11:32:00.691 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.691 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.691 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:00.691 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.122861+0000", 2024-03-19T11:32:00.691 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01559372 2024-03-19T11:32:00.691 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.691 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.691 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:00.691 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.122932+0000", 2024-03-19T11:32:00.691 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1477999999999996e-05 2024-03-19T11:32:00.691 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.691 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.691 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:00.691 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.122999+0000", 2024-03-19T11:32:00.691 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6832e-05 2024-03-19T11:32:00.692 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.692 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.692 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.692 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123332+0000", 2024-03-19T11:32:00.692 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000333271 2024-03-19T11:32:00.692 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.692 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.692 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:00.692 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123371+0000", 2024-03-19T11:32:00.692 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8525999999999997e-05 2024-03-19T11:32:00.692 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.692 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.692 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:00.692 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.155858+0000", 2024-03-19T11:32:00.693 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.032487697000000003 2024-03-19T11:32:00.693 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:00.693 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:00.693 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:00.693 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.693 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.693 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.21 e75,74 4:8401c1f0:::smithi03829856-43:8,46'66,4:866f3dcc:::smithi03829856-31:2,47'73)", 2024-03-19T11:32:00.693 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.123075+0000", 2024-03-19T11:32:00.693 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 66.559681384000001, 2024-03-19T11:32:00.693 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.033023045000000001, 2024-03-19T11:32:00.693 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:00.693 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:00.693 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:00.693 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.694 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:00.694 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123075+0000", 2024-03-19T11:32:00.694 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:00.694 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.694 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.694 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:00.694 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123075+0000", 2024-03-19T11:32:00.694 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:00.695 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.695 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.695 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:00.695 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123076+0000", 2024-03-19T11:32:00.695 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5e-06 2024-03-19T11:32:00.695 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.695 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.695 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:00.696 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123086+0000", 2024-03-19T11:32:00.696 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8060000000000006e-06 2024-03-19T11:32:00.696 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.696 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.696 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:00.696 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123099+0000", 2024-03-19T11:32:00.696 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3251000000000001e-05 2024-03-19T11:32:00.696 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.696 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.696 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:00.696 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123112+0000", 2024-03-19T11:32:00.696 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2774e-05 2024-03-19T11:32:00.697 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.697 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.697 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.697 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.155902+0000", 2024-03-19T11:32:00.697 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.032790225999999999 2024-03-19T11:32:00.697 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.697 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.697 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:00.697 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.155914+0000", 2024-03-19T11:32:00.697 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1476e-05 2024-03-19T11:32:00.698 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.698 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.698 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:00.698 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.156098+0000", 2024-03-19T11:32:00.698 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000184012 2024-03-19T11:32:00.698 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:00.698 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:00.698 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:00.698 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.698 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.698 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 75/74 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:7, version: 55'82, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@55'82, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 58'86, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@58'86, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [1652614~973634,2888511~52681,3182305~22261], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3204566, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:12, version: 66'101, data_included: [1541576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:12@66'101, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:10, version: 70'112, data_included: [1541576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:10@70'112, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 70'113, data_included: [1488926~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@70'113, size: 2605085, copy_subset: [(0, 2605085)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2537502, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1488926, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:32:00.698 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.174030+0000", 2024-03-19T11:32:00.699 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 66.508726179999996, 2024-03-19T11:32:00.699 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.040799275000000003, 2024-03-19T11:32:00.699 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:00.699 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:00.699 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:00.699 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.699 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:00.699 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.174030+0000", 2024-03-19T11:32:00.699 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:00.699 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.699 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.700 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:00.700 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.174030+0000", 2024-03-19T11:32:00.700 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:00.700 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.700 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.700 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:00.700 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.174031+0000", 2024-03-19T11:32:00.700 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.322e-06 2024-03-19T11:32:00.700 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.700 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.700 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:00.701 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.184771+0000", 2024-03-19T11:32:00.701 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010739227 2024-03-19T11:32:00.701 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.701 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.701 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:00.701 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.184872+0000", 2024-03-19T11:32:00.701 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000101234 2024-03-19T11:32:00.701 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.701 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.701 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:00.702 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.184985+0000", 2024-03-19T11:32:00.702 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000113545 2024-03-19T11:32:00.702 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.702 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.702 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.702 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.185463+0000", 2024-03-19T11:32:00.702 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00047749899999999999 2024-03-19T11:32:00.702 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.702 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.702 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:00.702 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.185534+0000", 2024-03-19T11:32:00.702 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1007999999999995e-05 2024-03-19T11:32:00.702 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.703 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.703 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:00.703 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.214829+0000", 2024-03-19T11:32:00.703 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029295439999999999 2024-03-19T11:32:00.703 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:00.703 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:00.703 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:00.703 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.703 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.703 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2372 4.16 4:6868af91:::smithi03829856-10:b [read 832512~818688] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-03-19T11:32:00.703 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:58.455732+0000", 2024-03-19T11:32:00.703 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 62.227024465, 2024-03-19T11:32:00.703 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.69150541200000004, 2024-03-19T11:32:00.703 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:00.704 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:00.704 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:00.704 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:00.704 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:00.704 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2372 2024-03-19T11:32:00.704 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.704 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:00.704 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.704 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:00.704 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455732+0000", 2024-03-19T11:32:00.704 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:00.704 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.704 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.704 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:00.704 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455732+0000", 2024-03-19T11:32:00.705 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:00.705 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.705 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.705 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:00.705 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455735+0000", 2024-03-19T11:32:00.705 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.756e-06 2024-03-19T11:32:00.705 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.705 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.705 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:00.705 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455744+0000", 2024-03-19T11:32:00.705 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1880000000000008e-06 2024-03-19T11:32:00.705 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.705 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.705 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:00.705 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455764+0000", 2024-03-19T11:32:00.706 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0044999999999999e-05 2024-03-19T11:32:00.706 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.706 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.706 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:00.706 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455774+0000", 2024-03-19T11:32:00.706 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8770000000000005e-06 2024-03-19T11:32:00.706 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.706 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.706 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.706 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455837+0000", 2024-03-19T11:32:00.706 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3588000000000002e-05 2024-03-19T11:32:00.706 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.706 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.706 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:00.707 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455846+0000", 2024-03-19T11:32:00.707 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6030000000000005e-06 2024-03-19T11:32:00.707 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.707 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.707 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.707 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.076525+0000", 2024-03-19T11:32:00.707 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.62067875900000002 2024-03-19T11:32:00.707 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.707 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.707 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:32:00.707 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.076981+0000", 2024-03-19T11:32:00.707 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00045633300000000001 2024-03-19T11:32:00.707 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.707 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.707 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.708 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.104422+0000", 2024-03-19T11:32:00.708 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027440665999999999 2024-03-19T11:32:00.708 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.708 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.708 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:32:00.708 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.104786+0000", 2024-03-19T11:32:00.708 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00036471499999999999 2024-03-19T11:32:00.708 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.708 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.708 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.708 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.140725+0000", 2024-03-19T11:32:00.708 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.035939093999999998 2024-03-19T11:32:00.709 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.709 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.709 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:32:00.709 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.140841+0000", 2024-03-19T11:32:00.709 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000115207 2024-03-19T11:32:00.709 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.709 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.709 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.709 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.145559+0000", 2024-03-19T11:32:00.709 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.004718486 2024-03-19T11:32:00.709 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.709 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.709 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:00.709 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.145760+0000", 2024-03-19T11:32:00.710 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000201058 2024-03-19T11:32:00.710 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.710 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.710 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:00.710 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.147237+0000", 2024-03-19T11:32:00.710 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014770370000000001 2024-03-19T11:32:00.710 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:00.710 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:00.710 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:00.710 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.710 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.710 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.4.13:82 3.3 3:f6b6d21a:::smithi03829856-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-03-19T11:32:00.711 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:04.069448+0000", 2024-03-19T11:32:00.711 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 56.613308680000003, 2024-03-19T11:32:00.711 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.086427773999999999, 2024-03-19T11:32:00.711 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:00.711 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:00.711 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:00.711 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.711 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:00.711 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069448+0000", 2024-03-19T11:32:00.711 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:00.711 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.711 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.711 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:00.712 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069448+0000", 2024-03-19T11:32:00.712 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:00.712 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.712 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.712 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:00.712 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069449+0000", 2024-03-19T11:32:00.712 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8160000000000001e-06 2024-03-19T11:32:00.712 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.712 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.713 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:00.713 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069458+0000", 2024-03-19T11:32:00.713 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2900000000000002e-06 2024-03-19T11:32:00.713 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.713 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.713 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:00.713 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069491+0000", 2024-03-19T11:32:00.713 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3229000000000003e-05 2024-03-19T11:32:00.713 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.713 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.713 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:00.714 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069501+0000", 2024-03-19T11:32:00.714 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7629999999999995e-06 2024-03-19T11:32:00.714 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.714 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.714 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.714 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069573+0000", 2024-03-19T11:32:00.714 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2349999999999997e-05 2024-03-19T11:32:00.714 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.714 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.715 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:00.715 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069581+0000", 2024-03-19T11:32:00.715 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.388e-06 2024-03-19T11:32:00.715 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.715 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.715 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.715 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.075451+0000", 2024-03-19T11:32:00.715 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005869956 2024-03-19T11:32:00.715 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.715 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.715 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:32:00.715 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.077511+0000", 2024-03-19T11:32:00.716 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0020592610000000002 2024-03-19T11:32:00.716 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.716 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.716 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.716 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.108781+0000", 2024-03-19T11:32:00.716 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.031270344999999998 2024-03-19T11:32:00.716 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.716 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.716 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:00.717 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.108895+0000", 2024-03-19T11:32:00.717 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000114059 2024-03-19T11:32:00.717 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.717 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.717 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4", 2024-03-19T11:32:00.717 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.109065+0000", 2024-03-19T11:32:00.717 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016979899999999999 2024-03-19T11:32:00.717 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.717 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.717 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:32:00.717 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.111730+0000", 2024-03-19T11:32:00.717 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0026651410000000002 2024-03-19T11:32:00.718 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.718 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.718 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:00.718 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.155616+0000", 2024-03-19T11:32:00.718 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.043885986000000002 2024-03-19T11:32:00.718 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.718 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.718 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:32:00.718 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.155745+0000", 2024-03-19T11:32:00.718 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012948899999999999 2024-03-19T11:32:00.719 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.719 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.719 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:00.719 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.155875+0000", 2024-03-19T11:32:00.719 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012990199999999999 2024-03-19T11:32:00.719 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:00.719 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:00.719 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:00.719 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.720 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.720 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.4.13:83 3.3 3:f6b6d21a:::smithi03829856-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-03-19T11:32:00.720 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:04.069526+0000", 2024-03-19T11:32:00.720 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 56.6132299, 2024-03-19T11:32:00.720 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.087140880000000004, 2024-03-19T11:32:00.720 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:00.720 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:00.720 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:00.720 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.720 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:00.721 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069526+0000", 2024-03-19T11:32:00.721 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:00.721 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.721 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.721 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:00.721 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069526+0000", 2024-03-19T11:32:00.721 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:00.721 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.721 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.721 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:00.722 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069527+0000", 2024-03-19T11:32:00.722 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2799999999999996e-07 2024-03-19T11:32:00.722 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.722 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.722 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:00.722 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069531+0000", 2024-03-19T11:32:00.722 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1489999999999996e-06 2024-03-19T11:32:00.722 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.722 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.723 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:00.723 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069539+0000", 2024-03-19T11:32:00.723 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3189999999999995e-06 2024-03-19T11:32:00.723 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.723 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.723 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:00.723 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069545+0000", 2024-03-19T11:32:00.723 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.592e-06 2024-03-19T11:32:00.723 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.724 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.724 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.724 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069616+0000", 2024-03-19T11:32:00.724 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1130000000000005e-05 2024-03-19T11:32:00.724 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.724 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.724 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:00.724 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069628+0000", 2024-03-19T11:32:00.724 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1697e-05 2024-03-19T11:32:00.724 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.724 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.724 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.725 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.077594+0000", 2024-03-19T11:32:00.725 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0079656989999999997 2024-03-19T11:32:00.725 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.725 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.725 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:32:00.725 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.077674+0000", 2024-03-19T11:32:00.725 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0338000000000002e-05 2024-03-19T11:32:00.725 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.725 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.725 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.725 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.112771+0000", 2024-03-19T11:32:00.725 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.035097325999999998 2024-03-19T11:32:00.725 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.725 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.725 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:00.726 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.112882+0000", 2024-03-19T11:32:00.726 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000110949 2024-03-19T11:32:00.726 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.726 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.726 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4", 2024-03-19T11:32:00.726 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.113031+0000", 2024-03-19T11:32:00.726 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000148355 2024-03-19T11:32:00.726 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.726 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.726 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:32:00.726 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.114076+0000", 2024-03-19T11:32:00.726 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001045009 2024-03-19T11:32:00.726 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.727 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.727 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:00.727 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156482+0000", 2024-03-19T11:32:00.727 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.042406354 2024-03-19T11:32:00.727 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.727 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.727 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:32:00.727 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156549+0000", 2024-03-19T11:32:00.727 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7229000000000003e-05 2024-03-19T11:32:00.727 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.727 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.727 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:00.727 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156667+0000", 2024-03-19T11:32:00.727 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000118106 2024-03-19T11:32:00.727 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:00.728 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:00.728 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:00.728 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.728 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.728 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.4.13:84 3.3 3:ce6951d3:::smithi03829856-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-03-19T11:32:00.728 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:04.071995+0000", 2024-03-19T11:32:00.728 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 56.610761125000003, 2024-03-19T11:32:00.728 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.084172497999999998, 2024-03-19T11:32:00.728 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:00.728 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:00.728 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:00.729 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.729 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:00.729 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.071995+0000", 2024-03-19T11:32:00.729 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:00.729 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.729 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.729 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:00.729 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.071995+0000", 2024-03-19T11:32:00.729 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:00.729 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.729 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.729 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:00.729 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.071997+0000", 2024-03-19T11:32:00.729 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6029999999999999e-06 2024-03-19T11:32:00.730 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.730 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.730 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:00.730 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072004+0000", 2024-03-19T11:32:00.730 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8600000000000004e-06 2024-03-19T11:32:00.730 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.730 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.730 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:00.730 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072033+0000", 2024-03-19T11:32:00.730 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9836999999999999e-05 2024-03-19T11:32:00.730 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.730 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.730 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:00.731 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072042+0000", 2024-03-19T11:32:00.731 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9279999999999999e-06 2024-03-19T11:32:00.731 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.731 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.731 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.731 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072123+0000", 2024-03-19T11:32:00.731 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0630000000000006e-05 2024-03-19T11:32:00.731 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.731 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.731 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:00.732 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072132+0000", 2024-03-19T11:32:00.732 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5560000000000008e-06 2024-03-19T11:32:00.732 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.732 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.732 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.732 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.077712+0000", 2024-03-19T11:32:00.732 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055804970000000002 2024-03-19T11:32:00.732 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.732 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.732 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:32:00.732 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.079495+0000", 2024-03-19T11:32:00.732 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017831850000000001 2024-03-19T11:32:00.732 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.732 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.732 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.733 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.109825+0000", 2024-03-19T11:32:00.733 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.030329819000000001 2024-03-19T11:32:00.733 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.733 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.733 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:00.733 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.109904+0000", 2024-03-19T11:32:00.733 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9269000000000003e-05 2024-03-19T11:32:00.734 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.734 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.734 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4", 2024-03-19T11:32:00.734 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.110085+0000", 2024-03-19T11:32:00.734 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00018018600000000001 2024-03-19T11:32:00.734 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.734 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.734 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:32:00.734 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.113581+0000", 2024-03-19T11:32:00.734 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034962919999999998 2024-03-19T11:32:00.734 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.734 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.734 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:00.735 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.155925+0000", 2024-03-19T11:32:00.735 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.042344404000000002 2024-03-19T11:32:00.735 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.735 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.735 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:32:00.735 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156026+0000", 2024-03-19T11:32:00.735 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000100599 2024-03-19T11:32:00.735 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.735 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.736 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:00.736 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156168+0000", 2024-03-19T11:32:00.736 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014183300000000001 2024-03-19T11:32:00.736 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:00.736 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:00.736 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:00.736 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.736 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.736 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.4.13:85 3.3 3:ce6951d3:::smithi03829856-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-03-19T11:32:00.736 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:04.072068+0000", 2024-03-19T11:32:00.736 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 56.610688488999998, 2024-03-19T11:32:00.736 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.084368466000000003, 2024-03-19T11:32:00.736 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:00.736 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:00.737 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:00.737 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.737 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:00.737 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072068+0000", 2024-03-19T11:32:00.737 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:00.737 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.737 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.737 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:00.737 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072068+0000", 2024-03-19T11:32:00.737 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:00.737 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.738 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.738 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:00.738 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072068+0000", 2024-03-19T11:32:00.738 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9699999999999996e-07 2024-03-19T11:32:00.738 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.738 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.738 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:00.738 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072074+0000", 2024-03-19T11:32:00.738 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2279999999999998e-06 2024-03-19T11:32:00.738 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.738 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.738 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:00.738 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072140+0000", 2024-03-19T11:32:00.739 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6638999999999997e-05 2024-03-19T11:32:00.739 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.739 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.739 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:00.739 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072147+0000", 2024-03-19T11:32:00.739 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8750000000000002e-06 2024-03-19T11:32:00.739 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.739 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.740 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.740 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072201+0000", 2024-03-19T11:32:00.740 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3634000000000002e-05 2024-03-19T11:32:00.740 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.740 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.740 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:00.740 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072211+0000", 2024-03-19T11:32:00.740 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0589e-05 2024-03-19T11:32:00.740 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.740 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.740 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.740 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.079687+0000", 2024-03-19T11:32:00.740 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074757219999999997 2024-03-19T11:32:00.740 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.741 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.741 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:32:00.741 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.079769+0000", 2024-03-19T11:32:00.741 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2251999999999998e-05 2024-03-19T11:32:00.741 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.741 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.741 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.741 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.111798+0000", 2024-03-19T11:32:00.741 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.032028889999999997 2024-03-19T11:32:00.741 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.741 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.741 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:00.741 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.111902+0000", 2024-03-19T11:32:00.741 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000103773 2024-03-19T11:32:00.742 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.742 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.742 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4", 2024-03-19T11:32:00.742 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.112073+0000", 2024-03-19T11:32:00.742 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017102799999999999 2024-03-19T11:32:00.742 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.742 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.742 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:32:00.742 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.113591+0000", 2024-03-19T11:32:00.742 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0015179569999999999 2024-03-19T11:32:00.742 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.742 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.743 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:00.743 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156223+0000", 2024-03-19T11:32:00.743 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.042632153999999998 2024-03-19T11:32:00.743 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.743 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.743 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:32:00.743 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156321+0000", 2024-03-19T11:32:00.743 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8246999999999996e-05 2024-03-19T11:32:00.743 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.743 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.743 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:00.743 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156436+0000", 2024-03-19T11:32:00.743 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000114881 2024-03-19T11:32:00.743 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:00.744 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:00.744 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:00.744 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.744 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.744 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3203 4.6 4:626acd4d:::smithi03829856-2:1f [sparse-read 0~1665607,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108)", 2024-03-19T11:32:00.744 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:27.253945+0000", 2024-03-19T11:32:00.744 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 33.428811422000003, 2024-03-19T11:32:00.744 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.83307735599999999, 2024-03-19T11:32:00.744 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:00.744 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:00.744 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:00.745 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:00.745 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:00.745 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3203 2024-03-19T11:32:00.745 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.745 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:00.745 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.745 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:00.745 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.253945+0000", 2024-03-19T11:32:00.745 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:00.745 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.745 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.745 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:00.745 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.253945+0000", 2024-03-19T11:32:00.746 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:00.746 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.746 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.746 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:00.746 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.253947+0000", 2024-03-19T11:32:00.746 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5979999999999999e-06 2024-03-19T11:32:00.746 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.746 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.746 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:00.746 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.253957+0000", 2024-03-19T11:32:00.747 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7699999999999996e-06 2024-03-19T11:32:00.747 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.747 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.747 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:00.747 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.253980+0000", 2024-03-19T11:32:00.747 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2466000000000001e-05 2024-03-19T11:32:00.747 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.747 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.747 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:00.747 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.253992+0000", 2024-03-19T11:32:00.747 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1985e-05 2024-03-19T11:32:00.747 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.747 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.747 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.747 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254071+0000", 2024-03-19T11:32:00.748 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8869000000000007e-05 2024-03-19T11:32:00.748 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.748 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.748 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:00.748 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254084+0000", 2024-03-19T11:32:00.748 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3716999999999999e-05 2024-03-19T11:32:00.748 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.748 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.749 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.749 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.086473+0000", 2024-03-19T11:32:00.749 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.83238890300000001 2024-03-19T11:32:00.749 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.749 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.749 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:00.749 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.086618+0000", 2024-03-19T11:32:00.749 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014479200000000001 2024-03-19T11:32:00.749 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.749 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.749 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:00.750 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.087022+0000", 2024-03-19T11:32:00.750 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00040425599999999997 2024-03-19T11:32:00.750 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:00.750 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:00.750 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:00.750 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.750 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.750 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3204 4.6 4:626acd4d:::smithi03829856-2:1f [read 0~3462242,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108)", 2024-03-19T11:32:00.751 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:27.254015+0000", 2024-03-19T11:32:00.751 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 33.428741641000002, 2024-03-19T11:32:00.751 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.83578705200000003, 2024-03-19T11:32:00.751 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:00.751 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:00.751 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:00.751 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:00.751 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:00.751 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3204 2024-03-19T11:32:00.751 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.751 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:00.751 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.751 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:00.752 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254015+0000", 2024-03-19T11:32:00.752 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:00.752 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.752 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.752 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:00.752 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254015+0000", 2024-03-19T11:32:00.752 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:00.752 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.752 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.752 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:00.752 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254015+0000", 2024-03-19T11:32:00.752 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8599999999999997e-07 2024-03-19T11:32:00.753 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.753 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.753 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:00.753 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254022+0000", 2024-03-19T11:32:00.753 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2870000000000001e-06 2024-03-19T11:32:00.753 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.753 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.753 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:00.753 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254031+0000", 2024-03-19T11:32:00.753 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0960000000000007e-06 2024-03-19T11:32:00.754 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.754 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.754 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:00.754 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254036+0000", 2024-03-19T11:32:00.754 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3510000000000001e-06 2024-03-19T11:32:00.754 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.754 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.754 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.754 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254137+0000", 2024-03-19T11:32:00.754 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000101065 2024-03-19T11:32:00.754 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.754 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.754 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:00.755 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254147+0000", 2024-03-19T11:32:00.755 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3929999999999997e-06 2024-03-19T11:32:00.755 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.755 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.755 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.755 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.087055+0000", 2024-03-19T11:32:00.755 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.83290843299999995 2024-03-19T11:32:00.755 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.755 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.755 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:00.755 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.087142+0000", 2024-03-19T11:32:00.756 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7151999999999995e-05 2024-03-19T11:32:00.756 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.756 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.756 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:00.756 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.089802+0000", 2024-03-19T11:32:00.756 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0026594890000000001 2024-03-19T11:32:00.756 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:00.756 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:00.756 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:00.756 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.757 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.757 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3205 4.6 4:626acd4d:::smithi03829856-2:1f [sparse-read 0~1665607] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108)", 2024-03-19T11:32:00.757 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:27.254049+0000", 2024-03-19T11:32:00.757 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 33.428707623999998, 2024-03-19T11:32:00.757 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.83625012399999998, 2024-03-19T11:32:00.757 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:00.757 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:00.757 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:00.757 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:00.757 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:00.757 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3205 2024-03-19T11:32:00.757 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.758 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:00.758 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.758 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:00.758 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254049+0000", 2024-03-19T11:32:00.758 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:00.758 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.758 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.758 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:00.758 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254049+0000", 2024-03-19T11:32:00.758 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:00.759 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.759 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.759 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:00.759 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254049+0000", 2024-03-19T11:32:00.759 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0699999999999997e-07 2024-03-19T11:32:00.759 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.759 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.759 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:00.759 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254053+0000", 2024-03-19T11:32:00.759 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.546e-06 2024-03-19T11:32:00.759 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.760 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.760 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:00.760 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254063+0000", 2024-03-19T11:32:00.760 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7939999999999997e-06 2024-03-19T11:32:00.760 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.760 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.760 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:00.760 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254072+0000", 2024-03-19T11:32:00.760 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9250000000000001e-06 2024-03-19T11:32:00.760 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.760 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.761 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.761 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254186+0000", 2024-03-19T11:32:00.761 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011446300000000001 2024-03-19T11:32:00.761 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.761 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.761 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:00.761 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254198+0000", 2024-03-19T11:32:00.761 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1933000000000001e-05 2024-03-19T11:32:00.761 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.762 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.762 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.762 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.089838+0000", 2024-03-19T11:32:00.762 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.83563974200000002 2024-03-19T11:32:00.762 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.762 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.762 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:00.762 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.089941+0000", 2024-03-19T11:32:00.762 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000103052 2024-03-19T11:32:00.763 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.763 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.763 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:00.763 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.090299+0000", 2024-03-19T11:32:00.763 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00035806199999999997 2024-03-19T11:32:00.763 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:00.763 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:00.763 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:00.763 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.763 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.763 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3220 4.16 4:6868af91:::smithi03829856-10:head [tier-promote] snapc 21=[21,20,1f,1e,1d,1c,1b,1a,18,17] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-19T11:32:00.763 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:28.068879+0000", 2024-03-19T11:32:00.763 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 32.613877418000001, 2024-03-19T11:32:00.763 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025399244000000001, 2024-03-19T11:32:00.764 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:00.764 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:00.764 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:00.764 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:00.764 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:00.764 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3220 2024-03-19T11:32:00.764 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.764 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:00.764 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.764 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:00.764 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.068879+0000", 2024-03-19T11:32:00.764 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:00.764 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.764 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.764 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:00.765 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.068879+0000", 2024-03-19T11:32:00.765 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:00.765 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.765 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.765 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:00.765 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.068881+0000", 2024-03-19T11:32:00.765 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3879999999999998e-06 2024-03-19T11:32:00.765 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.765 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.765 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:00.765 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.068892+0000", 2024-03-19T11:32:00.765 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0672999999999999e-05 2024-03-19T11:32:00.765 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.765 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.766 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:00.766 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.068917+0000", 2024-03-19T11:32:00.766 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5367e-05 2024-03-19T11:32:00.766 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.766 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.766 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:00.766 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.068935+0000", 2024-03-19T11:32:00.766 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7832000000000001e-05 2024-03-19T11:32:00.766 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.766 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.766 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.766 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.069075+0000", 2024-03-19T11:32:00.766 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013980899999999999 2024-03-19T11:32:00.766 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.767 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.767 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-03-19T11:32:00.767 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.069091+0000", 2024-03-19T11:32:00.767 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6325e-05 2024-03-19T11:32:00.767 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.767 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.767 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.767 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.071382+0000", 2024-03-19T11:32:00.767 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022910109999999999 2024-03-19T11:32:00.767 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.767 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.767 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:00.767 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.071538+0000", 2024-03-19T11:32:00.767 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015600999999999999 2024-03-19T11:32:00.767 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.768 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.768 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:00.768 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.094278+0000", 2024-03-19T11:32:00.768 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.022739829 2024-03-19T11:32:00.768 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:00.768 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:00.768 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:00.768 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.768 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.768 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3885 4.21 4:8401c1f0:::smithi03829856-43:head [write 443784~648161] snapc 29=[29,28,27,26,25,24,23,22,21,1f,1e,1d,1a,17] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e129)", 2024-03-19T11:32:00.768 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:50.078273+0000", 2024-03-19T11:32:00.768 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 10.604483597, 2024-03-19T11:32:00.769 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1185664280000001, 2024-03-19T11:32:00.769 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:00.769 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:00.769 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:00.769 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:00.769 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:00.769 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3885 2024-03-19T11:32:00.769 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.769 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:00.769 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.769 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:00.769 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.078273+0000", 2024-03-19T11:32:00.770 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:00.770 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.770 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.770 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:00.770 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.078273+0000", 2024-03-19T11:32:00.770 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:00.770 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.770 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.770 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:00.770 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.078276+0000", 2024-03-19T11:32:00.770 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.942e-06 2024-03-19T11:32:00.770 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.770 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.770 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:00.771 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.078790+0000", 2024-03-19T11:32:00.771 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00051449799999999995 2024-03-19T11:32:00.771 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.771 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.771 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:00.771 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.078809+0000", 2024-03-19T11:32:00.771 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8980000000000001e-05 2024-03-19T11:32:00.771 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.771 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.771 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:00.771 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.078821+0000", 2024-03-19T11:32:00.772 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1432e-05 2024-03-19T11:32:00.772 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.772 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.772 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.772 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.079070+0000", 2024-03-19T11:32:00.772 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00024937100000000002 2024-03-19T11:32:00.772 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.772 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.772 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-03-19T11:32:00.772 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.079079+0000", 2024-03-19T11:32:00.772 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2949999999999999e-06 2024-03-19T11:32:00.772 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.772 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.772 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.773 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.083821+0000", 2024-03-19T11:32:00.773 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0047417129999999998 2024-03-19T11:32:00.773 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.773 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.773 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:00.773 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.083836+0000", 2024-03-19T11:32:00.773 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4591e-05 2024-03-19T11:32:00.773 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.773 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.773 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.773 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.101489+0000", 2024-03-19T11:32:00.773 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0176535019999999 2024-03-19T11:32:00.774 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.774 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.774 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:32:00.774 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.104918+0000", 2024-03-19T11:32:00.774 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003429353 2024-03-19T11:32:00.774 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.774 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.774 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.774 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.142247+0000", 2024-03-19T11:32:00.774 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.037329108 2024-03-19T11:32:00.775 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.775 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.775 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:00.775 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.142409+0000", 2024-03-19T11:32:00.775 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016169700000000001 2024-03-19T11:32:00.775 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.775 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.775 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4,5", 2024-03-19T11:32:00.775 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.142754+0000", 2024-03-19T11:32:00.775 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000345025 2024-03-19T11:32:00.775 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.775 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.775 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:32:00.776 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.161867+0000", 2024-03-19T11:32:00.776 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019112371 2024-03-19T11:32:00.776 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.776 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.776 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:00.776 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.166589+0000", 2024-03-19T11:32:00.776 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0047220270000000002 2024-03-19T11:32:00.776 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.776 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.776 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:00.776 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.196673+0000", 2024-03-19T11:32:00.777 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.030084422999999999 2024-03-19T11:32:00.777 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.777 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.777 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:32:00.777 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.196730+0000", 2024-03-19T11:32:00.777 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7027000000000001e-05 2024-03-19T11:32:00.777 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.777 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.777 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:00.777 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.196839+0000", 2024-03-19T11:32:00.777 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000109073 2024-03-19T11:32:00.778 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:00.778 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:00.778 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:00.778 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.778 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.778 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3886 4.21 4:8401c1f0:::smithi03829856-43:head [write 1657676~659775] snapc 29=[29,28,27,26,25,24,23,22,21,1f,1e,1d,1a,17] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e129)", 2024-03-19T11:32:00.778 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:50.078842+0000", 2024-03-19T11:32:00.778 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 10.603914559, 2024-03-19T11:32:00.778 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1181774870000001, 2024-03-19T11:32:00.778 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:00.778 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:00.778 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:00.778 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:00.779 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:00.779 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3886 2024-03-19T11:32:00.779 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.779 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:00.779 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.779 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:00.779 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.078842+0000", 2024-03-19T11:32:00.779 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:00.779 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.779 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.779 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:00.780 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.078842+0000", 2024-03-19T11:32:00.780 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:00.780 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.780 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.780 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:00.780 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.078843+0000", 2024-03-19T11:32:00.780 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7400000000000002e-07 2024-03-19T11:32:00.780 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.780 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.780 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:00.780 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.079370+0000", 2024-03-19T11:32:00.780 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00052713200000000003 2024-03-19T11:32:00.780 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.780 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.780 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:00.781 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.079383+0000", 2024-03-19T11:32:00.781 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3131e-05 2024-03-19T11:32:00.781 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.781 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.781 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:00.781 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.079393+0000", 2024-03-19T11:32:00.781 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0114e-05 2024-03-19T11:32:00.781 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.781 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.781 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.781 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082249+0000", 2024-03-19T11:32:00.781 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0028558260000000001 2024-03-19T11:32:00.781 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.781 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.782 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:32:00.782 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082258+0000", 2024-03-19T11:32:00.782 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.5740000000000002e-06 2024-03-19T11:32:00.782 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.782 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.782 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.782 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.084372+0000", 2024-03-19T11:32:00.782 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0021131909999999999 2024-03-19T11:32:00.782 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.782 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.782 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:00.782 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.084383+0000", 2024-03-19T11:32:00.782 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1099e-05 2024-03-19T11:32:00.783 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.783 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.783 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.783 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.105453+0000", 2024-03-19T11:32:00.783 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.021070116 2024-03-19T11:32:00.783 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.783 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.783 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:32:00.783 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.105574+0000", 2024-03-19T11:32:00.783 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012102199999999999 2024-03-19T11:32:00.783 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.783 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.784 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.784 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.151364+0000", 2024-03-19T11:32:00.784 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.045790294000000002 2024-03-19T11:32:00.784 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.784 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.784 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:00.784 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.151579+0000", 2024-03-19T11:32:00.784 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00021460399999999999 2024-03-19T11:32:00.784 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.784 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.784 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4,5", 2024-03-19T11:32:00.784 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.151927+0000", 2024-03-19T11:32:00.785 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00034839700000000002 2024-03-19T11:32:00.785 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.785 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.786 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:32:00.786 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.161878+0000", 2024-03-19T11:32:00.786 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0099512430000000002 2024-03-19T11:32:00.786 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.786 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.786 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:00.786 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.169256+0000", 2024-03-19T11:32:00.786 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0073777440000000003 2024-03-19T11:32:00.786 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.786 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.787 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:00.787 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.196889+0000", 2024-03-19T11:32:00.787 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027633179000000001 2024-03-19T11:32:00.787 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.787 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.787 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:32:00.788 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.196937+0000", 2024-03-19T11:32:00.788 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7966999999999997e-05 2024-03-19T11:32:00.788 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.788 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.788 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:00.788 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.197019+0000", 2024-03-19T11:32:00.788 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1979999999999998e-05 2024-03-19T11:32:00.788 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:00.788 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:00.788 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:00.788 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.788 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.788 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3887 4.21 4:8401c1f0:::smithi03829856-43:head [write 2818209~425575] snapc 29=[29,28,27,26,25,24,23,22,21,1f,1e,1d,1a,17] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e129)", 2024-03-19T11:32:00.789 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:50.082016+0000", 2024-03-19T11:32:00.789 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 10.600740719000001, 2024-03-19T11:32:00.789 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.115265664, 2024-03-19T11:32:00.789 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:00.789 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:00.789 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:00.789 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:00.789 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:00.789 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3887 2024-03-19T11:32:00.789 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.790 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:00.790 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.790 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:00.790 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082016+0000", 2024-03-19T11:32:00.790 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:00.790 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.790 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.790 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:00.790 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082016+0000", 2024-03-19T11:32:00.790 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:00.790 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.791 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.791 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:00.791 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082017+0000", 2024-03-19T11:32:00.791 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.587e-06 2024-03-19T11:32:00.791 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.791 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.791 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:00.791 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082203+0000", 2024-03-19T11:32:00.791 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000185613 2024-03-19T11:32:00.791 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.791 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.791 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:00.792 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082214+0000", 2024-03-19T11:32:00.792 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1357e-05 2024-03-19T11:32:00.792 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.792 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.792 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:00.792 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082221+0000", 2024-03-19T11:32:00.792 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6290000000000004e-06 2024-03-19T11:32:00.792 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.792 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.792 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.792 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082300+0000", 2024-03-19T11:32:00.792 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9425999999999995e-05 2024-03-19T11:32:00.792 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.793 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.793 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:32:00.793 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082306+0000", 2024-03-19T11:32:00.793 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1419999999999998e-06 2024-03-19T11:32:00.793 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.793 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.793 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.793 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.084417+0000", 2024-03-19T11:32:00.793 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0021107589999999998 2024-03-19T11:32:00.794 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.794 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.794 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:00.794 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.084426+0000", 2024-03-19T11:32:00.794 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8410000000000006e-06 2024-03-19T11:32:00.794 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.794 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.794 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.794 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.105770+0000", 2024-03-19T11:32:00.794 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0213442100000001 2024-03-19T11:32:00.794 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.795 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.795 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:32:00.795 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.105890+0000", 2024-03-19T11:32:00.795 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000120349 2024-03-19T11:32:00.795 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.795 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.795 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.795 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.157275+0000", 2024-03-19T11:32:00.795 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.051384678000000003 2024-03-19T11:32:00.796 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.796 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.796 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:00.796 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.157445+0000", 2024-03-19T11:32:00.796 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000169746 2024-03-19T11:32:00.796 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.796 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.796 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4,5", 2024-03-19T11:32:00.796 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.157790+0000", 2024-03-19T11:32:00.797 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00034491799999999999 2024-03-19T11:32:00.797 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.797 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.797 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:32:00.797 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.161935+0000", 2024-03-19T11:32:00.797 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0041450269999999999 2024-03-19T11:32:00.797 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.797 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.797 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:00.797 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.189666+0000", 2024-03-19T11:32:00.797 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027731062000000001 2024-03-19T11:32:00.797 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.797 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.798 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:00.798 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.197073+0000", 2024-03-19T11:32:00.798 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074070270000000001 2024-03-19T11:32:00.798 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.798 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.798 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:32:00.798 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.197116+0000", 2024-03-19T11:32:00.798 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2758e-05 2024-03-19T11:32:00.798 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.798 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.798 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:00.798 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.197281+0000", 2024-03-19T11:32:00.799 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016553499999999999 2024-03-19T11:32:00.799 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:00.799 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:00.799 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:00.799 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.799 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.799 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3888 4.21 4:8401c1f0:::smithi03829856-43:head [setxattr _header (57) in=7b,truncate 3243784] snapc 29=[29,28,27,26,25,24,23,22,21,1f,1e,1d,1a,17] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e129)", 2024-03-19T11:32:00.799 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:50.082238+0000", 2024-03-19T11:32:00.799 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 10.600517836, 2024-03-19T11:32:00.799 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.115518945, 2024-03-19T11:32:00.799 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:00.799 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:00.799 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:00.799 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:00.800 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:00.800 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3888 2024-03-19T11:32:00.800 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.800 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:00.800 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.800 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:00.800 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082238+0000", 2024-03-19T11:32:00.800 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:00.800 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.800 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.800 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:00.800 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082238+0000", 2024-03-19T11:32:00.800 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:00.800 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.801 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.801 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:00.801 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082239+0000", 2024-03-19T11:32:00.801 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6400000000000001e-07 2024-03-19T11:32:00.801 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.801 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.801 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:00.801 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082249+0000", 2024-03-19T11:32:00.801 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9490000000000008e-06 2024-03-19T11:32:00.801 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.801 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.801 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:00.801 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082260+0000", 2024-03-19T11:32:00.801 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0499000000000001e-05 2024-03-19T11:32:00.801 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.802 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.802 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:00.802 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082267+0000", 2024-03-19T11:32:00.802 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9029999999999998e-06 2024-03-19T11:32:00.802 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.802 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.802 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.802 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082341+0000", 2024-03-19T11:32:00.802 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4776000000000004e-05 2024-03-19T11:32:00.802 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.802 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.802 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:32:00.802 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082350+0000", 2024-03-19T11:32:00.802 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2279999999999998e-06 2024-03-19T11:32:00.802 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.803 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.803 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.803 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.084459+0000", 2024-03-19T11:32:00.803 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0021094579999999998 2024-03-19T11:32:00.803 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.803 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.803 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:00.803 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.084468+0000", 2024-03-19T11:32:00.803 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2e-06 2024-03-19T11:32:00.803 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.803 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.803 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.803 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.105958+0000", 2024-03-19T11:32:00.803 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0214894450000001 2024-03-19T11:32:00.804 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.804 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.804 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:32:00.804 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.106112+0000", 2024-03-19T11:32:00.804 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000154022 2024-03-19T11:32:00.804 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.804 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.804 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.804 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.160042+0000", 2024-03-19T11:32:00.804 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.053929935999999998 2024-03-19T11:32:00.804 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.805 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.805 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:00.805 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.160205+0000", 2024-03-19T11:32:00.805 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016317600000000001 2024-03-19T11:32:00.805 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.805 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.805 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4,5", 2024-03-19T11:32:00.805 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.160566+0000", 2024-03-19T11:32:00.805 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000360918 2024-03-19T11:32:00.805 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.805 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.805 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:32:00.805 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.162064+0000", 2024-03-19T11:32:00.805 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014983380000000001 2024-03-19T11:32:00.806 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.806 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.806 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:00.806 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.196608+0000", 2024-03-19T11:32:00.806 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.034544115 2024-03-19T11:32:00.806 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.806 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.806 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:00.806 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.197420+0000", 2024-03-19T11:32:00.806 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00081147299999999995 2024-03-19T11:32:00.806 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.806 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.806 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:32:00.806 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.197519+0000", 2024-03-19T11:32:00.806 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8932999999999996e-05 2024-03-19T11:32:00.807 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.807 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.807 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:00.807 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.197757+0000", 2024-03-19T11:32:00.807 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000238812 2024-03-19T11:32:00.807 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:00.807 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:00.807 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:00.807 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.807 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.807 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3889 4.21 4:8401c1f0:::smithi03829856-43:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e129)", 2024-03-19T11:32:00.807 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:50.082302+0000", 2024-03-19T11:32:00.807 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 10.600454686999999, 2024-03-19T11:32:00.807 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.116076547, 2024-03-19T11:32:00.808 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:00.808 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:00.808 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:00.808 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:00.808 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:00.808 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3889 2024-03-19T11:32:00.808 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.808 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:00.808 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.808 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:00.808 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082302+0000", 2024-03-19T11:32:00.808 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:00.808 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.808 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.809 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:00.809 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082302+0000", 2024-03-19T11:32:00.809 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:00.809 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.809 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.809 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:00.809 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082303+0000", 2024-03-19T11:32:00.809 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2720000000000001e-06 2024-03-19T11:32:00.809 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.809 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.809 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:00.809 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082314+0000", 2024-03-19T11:32:00.809 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0864e-05 2024-03-19T11:32:00.809 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.810 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.810 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:00.810 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082327+0000", 2024-03-19T11:32:00.810 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2972e-05 2024-03-19T11:32:00.810 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.810 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.810 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:00.810 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082339+0000", 2024-03-19T11:32:00.810 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2458e-05 2024-03-19T11:32:00.810 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.810 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.810 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.810 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.085086+0000", 2024-03-19T11:32:00.810 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0027472859999999998 2024-03-19T11:32:00.811 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.811 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.811 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:00.811 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.085100+0000", 2024-03-19T11:32:00.811 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3339000000000001e-05 2024-03-19T11:32:00.811 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.811 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.811 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.811 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.106149+0000", 2024-03-19T11:32:00.811 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0210496010000001 2024-03-19T11:32:00.812 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.812 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.812 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:32:00.812 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.106232+0000", 2024-03-19T11:32:00.812 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2923999999999994e-05 2024-03-19T11:32:00.812 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.812 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.812 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.812 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.161418+0000", 2024-03-19T11:32:00.812 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.055186156 2024-03-19T11:32:00.812 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.812 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.812 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-03-19T11:32:00.812 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.161535+0000", 2024-03-19T11:32:00.813 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000117018 2024-03-19T11:32:00.813 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.813 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.813 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:00.813 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.197872+0000", 2024-03-19T11:32:00.813 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.036336161999999998 2024-03-19T11:32:00.813 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.813 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.813 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:00.813 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.198037+0000", 2024-03-19T11:32:00.813 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000164924 2024-03-19T11:32:00.813 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:00.813 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:00.814 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:00.814 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.198378+0000", 2024-03-19T11:32:00.814 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00034157099999999999 2024-03-19T11:32:00.814 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:00.814 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:00.814 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:00.814 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:00.814 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:00.814 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:32:00.816 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-19T11:32:00.817 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:32:00.823+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:32:00.916 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:32:00.923+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:32:00.997 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:32:00.997 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:32:00.997 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:32:00.997 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:32:01.011 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-19T11:32:01.053 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:32:01.063+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:32:01.072 INFO:teuthology.orchestra.run.smithi038.stderr:set pool 4 pgp_num to 26 2024-03-19T11:32:01.077 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-03-19T11:32:01.078 INFO:tasks.rados.rados.0.smithi038.stdout:1775: done (13 left) 2024-03-19T11:32:01.078 INFO:tasks.rados.rados.0.smithi038.stdout:1776: write (excl) oid 37 current snap is 40 2024-03-19T11:32:01.078 INFO:tasks.rados.rados.0.smithi038.stdout:1776: seq_num 330 ranges {746310=635349,2077646=576443,3362764=183546} 2024-03-19T11:32:01.086 INFO:tasks.rados.rados.0.smithi038.stdout:1776: writing smithi03829856-37 from 746310 to 1381659 tid 1 2024-03-19T11:32:01.094 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:32:01.097+0000 7f143b8b4640 -1 osd.5 139 handle_osd_map injecting map crc failure 2024-03-19T11:32:01.094 INFO:tasks.rados.rados.0.smithi038.stdout:1776: writing smithi03829856-37 from 2077646 to 2654089 tid 2 2024-03-19T11:32:01.099 INFO:tasks.rados.rados.0.smithi038.stdout:1776: writing smithi03829856-37 from 3362764 to 3546310 tid 3 2024-03-19T11:32:01.100 INFO:tasks.rados.rados.0.smithi038.stdout:1777: delete oid 41 current snap is 40 2024-03-19T11:32:01.100 INFO:tasks.rados.rados.0.smithi038.stdout:1765: got (0) Success 2024-03-19T11:32:01.100 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 8 v 211 (ObjNum 49 snap 0 seq_num 1022187120) clean dne 2024-03-19T11:32:01.100 INFO:tasks.rados.rados.0.smithi038.stdout:1767: expect (ObjNum 206 snap 22 seq_num 206) 2024-03-19T11:32:01.157 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:32:01.167+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:32:01.190 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:32:01.190 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:32:01.190 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:32:01.190 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:32:01.190 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:32:01.205 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-19T11:32:01.220 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:32:01.229+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:32:01.266 INFO:tasks.rados.rados.0.smithi038.stdout:1764: expect (ObjNum 240 snap 30 seq_num 240) 2024-03-19T11:32:01.321 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:32:01.329+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:32:01.324 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-19T11:32:01.365 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:32:01.366 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:32:01.366 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:32:01.366 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:32:01.366 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.366 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1063 4.4 4:214026b3:::smithi03829856-47:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:32:01.366 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.784904+0000", 2024-03-19T11:32:01.366 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 107.58928397699999, 2024-03-19T11:32:01.366 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024425912000000001, 2024-03-19T11:32:01.366 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:01.366 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:01.366 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:01.366 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:01.366 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:01.367 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1063 2024-03-19T11:32:01.367 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.367 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:01.367 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.367 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:01.367 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784904+0000", 2024-03-19T11:32:01.367 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:01.367 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.367 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.367 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:01.367 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784904+0000", 2024-03-19T11:32:01.367 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:01.367 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.367 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.367 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:01.368 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784906+0000", 2024-03-19T11:32:01.368 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1409999999999999e-06 2024-03-19T11:32:01.368 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.368 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.368 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:01.368 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784915+0000", 2024-03-19T11:32:01.368 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5459999999999992e-06 2024-03-19T11:32:01.368 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.368 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.368 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:01.368 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784935+0000", 2024-03-19T11:32:01.368 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0310999999999999e-05 2024-03-19T11:32:01.368 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.368 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.368 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:01.369 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784946+0000", 2024-03-19T11:32:01.369 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0675e-05 2024-03-19T11:32:01.369 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.369 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.369 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:01.369 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.785032+0000", 2024-03-19T11:32:01.369 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6049000000000002e-05 2024-03-19T11:32:01.369 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.369 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.369 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:01.369 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.785136+0000", 2024-03-19T11:32:01.369 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000104209 2024-03-19T11:32:01.369 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.369 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.369 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:01.370 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.809330+0000", 2024-03-19T11:32:01.370 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024193981 2024-03-19T11:32:01.370 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:01.370 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:01.370 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:01.370 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.370 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.370 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1433 4.16 4:6be89c06:::smithi03829856-24:c [sparse-read 0~1100427,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:32:01.370 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.976652+0000", 2024-03-19T11:32:01.370 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 95.397536275999997, 2024-03-19T11:32:01.370 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95684753199999995, 2024-03-19T11:32:01.370 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:01.370 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:01.371 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:01.371 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:01.371 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:01.371 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1433 2024-03-19T11:32:01.371 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.371 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:01.371 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.371 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:01.371 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976652+0000", 2024-03-19T11:32:01.371 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:01.371 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.371 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.371 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:01.371 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976652+0000", 2024-03-19T11:32:01.372 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:01.372 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.372 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.372 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:01.372 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976654+0000", 2024-03-19T11:32:01.372 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6199999999999999e-06 2024-03-19T11:32:01.372 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.372 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.372 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:01.372 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976668+0000", 2024-03-19T11:32:01.372 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3213999999999999e-05 2024-03-19T11:32:01.372 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.372 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.373 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:01.373 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976694+0000", 2024-03-19T11:32:01.373 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6488000000000001e-05 2024-03-19T11:32:01.373 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.373 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.373 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:01.373 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976705+0000", 2024-03-19T11:32:01.373 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1044e-05 2024-03-19T11:32:01.373 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.373 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.373 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:01.373 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976795+0000", 2024-03-19T11:32:01.373 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0283000000000003e-05 2024-03-19T11:32:01.373 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.374 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.374 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:01.374 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976811+0000", 2024-03-19T11:32:01.374 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5262000000000001e-05 2024-03-19T11:32:01.374 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.374 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.374 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:01.374 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.927337+0000", 2024-03-19T11:32:01.374 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95052597100000003 2024-03-19T11:32:01.374 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.374 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.374 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:01.374 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.927495+0000", 2024-03-19T11:32:01.374 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015788600000000001 2024-03-19T11:32:01.375 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.375 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.375 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:01.375 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.933499+0000", 2024-03-19T11:32:01.375 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0060047640000000001 2024-03-19T11:32:01.375 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:01.375 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:01.375 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:01.375 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.375 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.375 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1434 4.16 4:6be89c06:::smithi03829856-24:c [read 0~2386561,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:32:01.375 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.976739+0000", 2024-03-19T11:32:01.375 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 95.397449115000001, 2024-03-19T11:32:01.375 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95781317399999999, 2024-03-19T11:32:01.375 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:01.376 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:01.376 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:01.376 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:01.376 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:01.376 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1434 2024-03-19T11:32:01.376 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.376 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:01.376 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.376 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:01.376 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976739+0000", 2024-03-19T11:32:01.376 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:01.376 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.376 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.376 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:01.376 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976739+0000", 2024-03-19T11:32:01.377 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:01.377 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.377 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.377 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:01.377 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976740+0000", 2024-03-19T11:32:01.377 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8800000000000001e-07 2024-03-19T11:32:01.377 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.377 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.377 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:01.377 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976748+0000", 2024-03-19T11:32:01.377 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8979999999999996e-06 2024-03-19T11:32:01.377 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.377 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.377 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:01.377 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976757+0000", 2024-03-19T11:32:01.378 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9439999999999994e-06 2024-03-19T11:32:01.378 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.378 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.378 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:01.378 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976763+0000", 2024-03-19T11:32:01.378 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5030000000000002e-06 2024-03-19T11:32:01.378 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.378 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.378 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:01.378 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976849+0000", 2024-03-19T11:32:01.378 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5803999999999994e-05 2024-03-19T11:32:01.378 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.378 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.378 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:01.379 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976860+0000", 2024-03-19T11:32:01.379 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0869e-05 2024-03-19T11:32:01.379 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.379 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.379 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:01.379 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.933562+0000", 2024-03-19T11:32:01.379 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95670232300000002 2024-03-19T11:32:01.379 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.379 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.379 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:01.379 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.933693+0000", 2024-03-19T11:32:01.379 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013110699999999999 2024-03-19T11:32:01.379 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.379 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.380 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:01.380 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.934552+0000", 2024-03-19T11:32:01.380 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00085883800000000001 2024-03-19T11:32:01.380 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:01.380 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:01.380 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:01.380 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.380 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.380 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1435 4.16 4:6be89c06:::smithi03829856-24:c [sparse-read 0~1100427] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:32:01.380 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.976775+0000", 2024-03-19T11:32:01.381 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 95.397413443999994, 2024-03-19T11:32:01.381 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95962779499999995, 2024-03-19T11:32:01.381 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:01.381 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:01.381 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:01.381 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:01.381 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:01.381 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1435 2024-03-19T11:32:01.381 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.381 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:01.382 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.382 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:01.382 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976775+0000", 2024-03-19T11:32:01.382 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:01.382 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.382 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.382 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:01.382 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976775+0000", 2024-03-19T11:32:01.382 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:01.382 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.382 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.382 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:01.382 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976775+0000", 2024-03-19T11:32:01.382 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9800000000000003e-07 2024-03-19T11:32:01.383 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.383 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.383 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:01.383 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976779+0000", 2024-03-19T11:32:01.383 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3830000000000001e-06 2024-03-19T11:32:01.383 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.383 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.383 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:01.383 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976785+0000", 2024-03-19T11:32:01.383 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5649999999999998e-06 2024-03-19T11:32:01.383 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.383 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.383 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:01.383 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976790+0000", 2024-03-19T11:32:01.383 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1379999999999998e-06 2024-03-19T11:32:01.384 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.384 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.384 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:01.384 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976889+0000", 2024-03-19T11:32:01.384 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8331000000000004e-05 2024-03-19T11:32:01.384 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.384 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.384 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:01.384 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976894+0000", 2024-03-19T11:32:01.384 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8270000000000003e-06 2024-03-19T11:32:01.384 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.384 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.384 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:01.384 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.935982+0000", 2024-03-19T11:32:01.384 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95908714100000003 2024-03-19T11:32:01.385 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.385 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.385 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:01.385 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.936093+0000", 2024-03-19T11:32:01.385 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000111557 2024-03-19T11:32:01.385 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.385 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.385 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:01.385 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.936402+0000", 2024-03-19T11:32:01.385 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030925499999999998 2024-03-19T11:32:01.385 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:01.385 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:01.385 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:01.385 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.386 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.386 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1442 4.6 4:626acd4d:::smithi03829856-2:head [cache-evict] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:32:01.386 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.978864+0000", 2024-03-19T11:32:01.386 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 95.395323779999998, 2024-03-19T11:32:01.386 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95708845099999995, 2024-03-19T11:32:01.386 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:01.386 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:01.386 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:01.386 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:01.386 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:01.386 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1442 2024-03-19T11:32:01.386 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.386 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:01.386 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.386 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:01.387 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978864+0000", 2024-03-19T11:32:01.387 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:01.387 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.387 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.387 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:01.387 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978864+0000", 2024-03-19T11:32:01.387 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:01.387 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.387 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.387 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:01.387 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978866+0000", 2024-03-19T11:32:01.387 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0729999999999999e-06 2024-03-19T11:32:01.387 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.387 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.387 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:01.388 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978877+0000", 2024-03-19T11:32:01.388 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0390999999999999e-05 2024-03-19T11:32:01.388 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.388 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.388 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:01.388 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978902+0000", 2024-03-19T11:32:01.388 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5635000000000001e-05 2024-03-19T11:32:01.388 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.388 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.388 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:01.388 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978915+0000", 2024-03-19T11:32:01.388 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2617e-05 2024-03-19T11:32:01.388 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.388 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.389 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:01.389 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.979005+0000", 2024-03-19T11:32:01.389 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0552999999999999e-05 2024-03-19T11:32:01.389 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.389 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.389 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:01.389 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.979021+0000", 2024-03-19T11:32:01.389 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5102999999999999e-05 2024-03-19T11:32:01.389 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.389 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.389 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:01.389 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.926879+0000", 2024-03-19T11:32:01.389 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.94785817900000002 2024-03-19T11:32:01.389 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.389 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.390 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:32:01.390 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.927277+0000", 2024-03-19T11:32:01.390 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00039819600000000002 2024-03-19T11:32:01.390 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.391 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.391 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:01.391 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.935752+0000", 2024-03-19T11:32:01.391 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0084753420000000003 2024-03-19T11:32:01.391 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.391 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.391 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:01.391 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.935874+0000", 2024-03-19T11:32:01.391 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000121311 2024-03-19T11:32:01.391 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.391 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.391 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:01.392 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.935953+0000", 2024-03-19T11:32:01.392 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9050999999999999e-05 2024-03-19T11:32:01.392 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:01.392 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:01.392 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:01.392 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.392 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.392 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2003 4.16 4:6868af91:::smithi03829856-10:head [tier-promote] snapc 12=[12,11,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:32:01.392 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:45.804046+0000", 2024-03-19T11:32:01.392 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 75.570141867000004, 2024-03-19T11:32:01.392 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018976396, 2024-03-19T11:32:01.392 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:01.393 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:01.393 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:01.393 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:01.393 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:01.393 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2003 2024-03-19T11:32:01.393 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.393 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:01.393 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.393 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:01.393 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804046+0000", 2024-03-19T11:32:01.393 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:01.393 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.394 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.394 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:01.394 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804046+0000", 2024-03-19T11:32:01.394 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:01.394 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.394 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.394 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:01.394 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804052+0000", 2024-03-19T11:32:01.395 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7599999999999999e-06 2024-03-19T11:32:01.395 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.395 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.395 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:01.395 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804090+0000", 2024-03-19T11:32:01.395 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8018000000000001e-05 2024-03-19T11:32:01.395 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.396 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.396 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:01.396 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804256+0000", 2024-03-19T11:32:01.396 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016593599999999999 2024-03-19T11:32:01.396 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.396 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.396 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:01.397 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804300+0000", 2024-03-19T11:32:01.397 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3648e-05 2024-03-19T11:32:01.397 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.397 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.397 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:01.397 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804538+0000", 2024-03-19T11:32:01.397 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000238347 2024-03-19T11:32:01.397 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.397 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.398 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:01.398 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804716+0000", 2024-03-19T11:32:01.398 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000177968 2024-03-19T11:32:01.398 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.398 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.398 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:01.398 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.823023+0000", 2024-03-19T11:32:01.398 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018306718999999999 2024-03-19T11:32:01.398 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:01.398 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:01.398 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:01.398 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.399 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.399 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1d58db11:::smithi03829856-18:6, version: 55'80, data_included: [759683~639358,1904434~409218], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:6@55'80, size: 3159683, copy_subset: [(0, 3159683)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2313652, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1d58db11:::smithi03829856-18:b, version: 58'84, data_included: [443718~955323,1564138~93253], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:b@58'84, size: 3445402, copy_subset: [(0, 3445402)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1657391, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:10, version: 66'95, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:10@66'95, size: 3764735, copy_subset: [(0, 3764735)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:12, version: 66'97, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:12@66'97, size: 2504243, copy_subset: [(0, 2504243)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:c, version: 70'102, data_included: [416807~576167,1542129~472409], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:c@70'102, size: 2857157, copy_subset: [(0, 2857157)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2014538, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:32:01.399 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.275179+0000", 2024-03-19T11:32:01.399 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 67.099008877000003, 2024-03-19T11:32:01.399 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026559353000000001, 2024-03-19T11:32:01.399 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:01.399 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:01.399 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:01.399 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.399 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:01.399 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.275179+0000", 2024-03-19T11:32:01.400 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:01.400 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.400 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.401 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:01.401 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.275179+0000", 2024-03-19T11:32:01.401 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:01.401 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.401 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.401 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:01.401 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.275196+0000", 2024-03-19T11:32:01.401 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6946999999999999e-05 2024-03-19T11:32:01.401 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.401 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.401 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:01.401 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281456+0000", 2024-03-19T11:32:01.401 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0062600529999999998 2024-03-19T11:32:01.402 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.402 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.402 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:01.402 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281524+0000", 2024-03-19T11:32:01.402 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7971000000000004e-05 2024-03-19T11:32:01.402 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.402 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.402 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:01.402 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281591+0000", 2024-03-19T11:32:01.402 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6704000000000004e-05 2024-03-19T11:32:01.402 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.402 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.403 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:01.403 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281865+0000", 2024-03-19T11:32:01.403 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000274097 2024-03-19T11:32:01.403 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.403 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.403 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:01.403 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281901+0000", 2024-03-19T11:32:01.403 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5777e-05 2024-03-19T11:32:01.403 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.403 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.403 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:01.404 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.301738+0000", 2024-03-19T11:32:01.404 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019837804000000001 2024-03-19T11:32:01.404 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:01.404 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:01.404 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:01.405 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.405 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.405 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.18 e75,74 4:1d58db11:::smithi03829856-18:2,48'76)", 2024-03-19T11:32:01.405 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.281661+0000", 2024-03-19T11:32:01.405 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 67.092527345999997, 2024-03-19T11:32:01.405 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020384337999999998, 2024-03-19T11:32:01.405 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:01.405 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:01.405 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:01.405 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.405 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:01.405 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281661+0000", 2024-03-19T11:32:01.406 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:01.406 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.406 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.406 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:01.406 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281661+0000", 2024-03-19T11:32:01.406 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:01.406 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.406 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.406 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:01.406 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281661+0000", 2024-03-19T11:32:01.406 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9100000000000003e-07 2024-03-19T11:32:01.407 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.407 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.407 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:01.407 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281668+0000", 2024-03-19T11:32:01.407 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8199999999999999e-06 2024-03-19T11:32:01.407 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.408 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.408 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:01.408 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281679+0000", 2024-03-19T11:32:01.408 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1107999999999999e-05 2024-03-19T11:32:01.408 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.408 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.408 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:01.408 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281691+0000", 2024-03-19T11:32:01.408 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1399999999999999e-05 2024-03-19T11:32:01.408 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.408 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.409 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:01.409 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.301804+0000", 2024-03-19T11:32:01.409 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020113669000000001 2024-03-19T11:32:01.409 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.409 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.409 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:01.409 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.301818+0000", 2024-03-19T11:32:01.409 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3718e-05 2024-03-19T11:32:01.409 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.410 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.410 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:01.410 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302045+0000", 2024-03-19T11:32:01.410 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000226832 2024-03-19T11:32:01.410 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:01.410 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:01.411 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:01.411 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.411 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.411 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1dabc0f1:::smithi03829856-1:head, version: 70'103, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:head@70'103, size: 2621892, copy_subset: [(0, 2621892)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:32:01.411 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.281783+0000", 2024-03-19T11:32:01.411 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 67.092405069999998, 2024-03-19T11:32:01.411 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026187221, 2024-03-19T11:32:01.411 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:01.411 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:01.411 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:01.411 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.411 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:01.411 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281783+0000", 2024-03-19T11:32:01.411 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:01.411 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.412 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.412 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:01.412 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281783+0000", 2024-03-19T11:32:01.412 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:01.412 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.412 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.412 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:01.412 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281783+0000", 2024-03-19T11:32:01.412 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1699999999999999e-07 2024-03-19T11:32:01.412 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.412 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.412 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:01.412 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.282962+0000", 2024-03-19T11:32:01.412 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001178601 2024-03-19T11:32:01.412 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.413 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.413 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:01.413 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.282987+0000", 2024-03-19T11:32:01.413 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4899e-05 2024-03-19T11:32:01.413 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.413 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.413 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:01.413 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283006+0000", 2024-03-19T11:32:01.413 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9494999999999999e-05 2024-03-19T11:32:01.413 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.413 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.413 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:01.413 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.305757+0000", 2024-03-19T11:32:01.413 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.022750570000000001 2024-03-19T11:32:01.413 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.414 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.414 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:01.414 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.305770+0000", 2024-03-19T11:32:01.414 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2791e-05 2024-03-19T11:32:01.414 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.414 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.414 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:01.414 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.307970+0000", 2024-03-19T11:32:01.414 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022005480000000001 2024-03-19T11:32:01.414 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:01.414 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:01.414 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:01.414 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.414 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.415 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.18 e75,74 4:1d58db11:::smithi03829856-18:c,70'104)", 2024-03-19T11:32:01.415 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.283036+0000", 2024-03-19T11:32:01.415 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 67.091151969999999, 2024-03-19T11:32:01.415 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019146032, 2024-03-19T11:32:01.415 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:01.415 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:01.415 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:01.415 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.415 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:01.415 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283036+0000", 2024-03-19T11:32:01.415 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:01.415 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.415 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.415 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:01.415 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283036+0000", 2024-03-19T11:32:01.416 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:01.416 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.416 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.416 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:01.416 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283037+0000", 2024-03-19T11:32:01.416 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4899999999999995e-07 2024-03-19T11:32:01.416 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.416 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.416 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:01.416 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283042+0000", 2024-03-19T11:32:01.416 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0930000000000002e-06 2024-03-19T11:32:01.416 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.416 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.416 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:01.417 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283053+0000", 2024-03-19T11:32:01.417 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1083999999999999e-05 2024-03-19T11:32:01.417 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.417 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.417 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:01.417 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283059+0000", 2024-03-19T11:32:01.417 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6329999999999999e-06 2024-03-19T11:32:01.417 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.417 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.417 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:01.417 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302090+0000", 2024-03-19T11:32:01.417 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01903038 2024-03-19T11:32:01.417 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.417 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.417 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:01.418 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302097+0000", 2024-03-19T11:32:01.418 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.221e-06 2024-03-19T11:32:01.418 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.418 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.418 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:01.418 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302182+0000", 2024-03-19T11:32:01.418 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5072000000000001e-05 2024-03-19T11:32:01.418 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:01.418 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:01.418 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:01.418 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.418 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.418 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(osd.5.0:143 4.18 e75/74 4:1d58db11:::smithi03829856-18:head v 75'106, mlcod=0'0)", 2024-03-19T11:32:01.418 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.283083+0000", 2024-03-19T11:32:01.419 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 67.091104642999994, 2024-03-19T11:32:01.419 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025309439, 2024-03-19T11:32:01.419 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:01.419 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:01.419 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:01.419 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.419 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:01.419 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283083+0000", 2024-03-19T11:32:01.419 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:01.419 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.419 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.419 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:01.419 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283083+0000", 2024-03-19T11:32:01.419 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:01.419 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.420 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.420 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:01.420 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283084+0000", 2024-03-19T11:32:01.420 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7800000000000002e-07 2024-03-19T11:32:01.420 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.420 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.420 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:01.420 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.284082+0000", 2024-03-19T11:32:01.420 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0009977880000000001 2024-03-19T11:32:01.420 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.420 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.420 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:01.420 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.284095+0000", 2024-03-19T11:32:01.420 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3256e-05 2024-03-19T11:32:01.420 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.421 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.421 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:01.421 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.284102+0000", 2024-03-19T11:32:01.421 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.605e-06 2024-03-19T11:32:01.421 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.421 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.421 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:01.421 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302285+0000", 2024-03-19T11:32:01.421 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018182423 2024-03-19T11:32:01.421 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.421 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.421 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:01.421 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302341+0000", 2024-03-19T11:32:01.421 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.626e-05 2024-03-19T11:32:01.422 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.422 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.422 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:32:01.422 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.308338+0000", 2024-03-19T11:32:01.422 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0059968030000000002 2024-03-19T11:32:01.422 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.422 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.422 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:01.422 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.308393+0000", 2024-03-19T11:32:01.422 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4826e-05 2024-03-19T11:32:01.422 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:01.422 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:01.422 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:01.422 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.422 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.423 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "pg_update_log_missing(4.18 epoch 75/74 rep_tid 144 entries 75'107 (0'0) error 4:1d58db11:::smithi03829856-18:head by client.4415.0:2283 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-19T11:32:01.423 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.288252+0000", 2024-03-19T11:32:01.423 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 67.085936093000001, 2024-03-19T11:32:01.423 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020167377, 2024-03-19T11:32:01.423 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:01.423 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-03-19T11:32:01.423 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:01.423 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.423 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:01.423 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288252+0000", 2024-03-19T11:32:01.423 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:01.423 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.423 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.423 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:01.424 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288252+0000", 2024-03-19T11:32:01.424 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:01.424 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.424 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.424 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:01.424 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288253+0000", 2024-03-19T11:32:01.424 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2109999999999999e-06 2024-03-19T11:32:01.424 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.424 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.424 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:01.424 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288268+0000", 2024-03-19T11:32:01.424 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5003e-05 2024-03-19T11:32:01.424 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.424 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.424 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:01.425 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288301+0000", 2024-03-19T11:32:01.425 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2539999999999997e-05 2024-03-19T11:32:01.425 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.425 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.425 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:01.425 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288319+0000", 2024-03-19T11:32:01.425 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8232000000000001e-05 2024-03-19T11:32:01.425 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.425 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.425 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:01.425 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.305490+0000", 2024-03-19T11:32:01.425 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017170857000000001 2024-03-19T11:32:01.425 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.425 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.425 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:01.426 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.308419+0000", 2024-03-19T11:32:01.426 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029295340000000001 2024-03-19T11:32:01.426 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:01.426 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:01.426 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:01.426 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.426 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.426 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1d58db11:::smithi03829856-18:6, version: 55'80, data_included: [2313652~310127,3128657~31026], data_size: 341153, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:6@55'80, size: 3159683, copy_subset: [(0, 3159683)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3159683, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2313652, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:1d58db11:::smithi03829856-18:b, version: 58'84, data_included: [1657391~966388,2633066~82188], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:b@58'84, size: 3445402, copy_subset: [(0, 3445402)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2715254, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1657391, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:10, version: 66'95, data_included: [1829043~668012,3134120~380564], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:10@66'95, size: 3764735, copy_subset: [(0, 3764735)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3514684, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:12, version: 66'97, data_included: [1829043~507467], data_size: 507467, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:12@66'97, size: 2504243, copy_subset: [(0, 2504243)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2504243, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:c, version: 70'102, data_included: [2014538~156250], data_size: 156250, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:c@70'102, size: 2857157, copy_subset: [(0, 2857157)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2857157, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2014538, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:32:01.426 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.312362+0000", 2024-03-19T11:32:01.426 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 67.061826044, 2024-03-19T11:32:01.426 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017968951, 2024-03-19T11:32:01.426 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:01.426 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:01.426 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:01.427 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.427 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:01.427 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.312362+0000", 2024-03-19T11:32:01.427 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:01.427 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.427 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.427 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:01.427 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.312362+0000", 2024-03-19T11:32:01.427 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:01.427 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.427 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.427 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:01.427 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.312363+0000", 2024-03-19T11:32:01.427 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.528e-06 2024-03-19T11:32:01.427 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.427 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.428 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:01.428 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315748+0000", 2024-03-19T11:32:01.428 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033842830000000001 2024-03-19T11:32:01.428 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.428 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.428 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:01.428 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315795+0000", 2024-03-19T11:32:01.428 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7382999999999997e-05 2024-03-19T11:32:01.428 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.428 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.428 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:01.428 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315845+0000", 2024-03-19T11:32:01.428 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0319999999999999e-05 2024-03-19T11:32:01.428 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.428 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.429 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:01.429 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.316123+0000", 2024-03-19T11:32:01.429 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00027747200000000002 2024-03-19T11:32:01.429 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.429 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.429 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:01.429 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.316170+0000", 2024-03-19T11:32:01.429 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7466999999999998e-05 2024-03-19T11:32:01.429 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.429 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.429 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:01.429 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.330331+0000", 2024-03-19T11:32:01.429 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014160498000000001 2024-03-19T11:32:01.429 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:01.430 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:01.430 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:01.430 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.430 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.430 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1dabc0f1:::smithi03829856-1:head, version: 70'103, data_included: [1829043~555153], data_size: 555153, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:head@70'103, size: 2621892, copy_subset: [(0, 2621892)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2621892, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:32:01.431 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.315898+0000", 2024-03-19T11:32:01.431 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 67.058290057999997, 2024-03-19T11:32:01.431 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016880424000000002, 2024-03-19T11:32:01.431 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:01.431 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:01.431 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:01.431 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.431 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:01.431 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315898+0000", 2024-03-19T11:32:01.432 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:01.432 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.432 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.432 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:01.432 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315898+0000", 2024-03-19T11:32:01.432 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:01.432 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.432 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.432 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:01.432 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315899+0000", 2024-03-19T11:32:01.432 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.003e-06 2024-03-19T11:32:01.432 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.432 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.432 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:01.433 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.316486+0000", 2024-03-19T11:32:01.433 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00058673099999999995 2024-03-19T11:32:01.433 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.433 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.433 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:01.433 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.316519+0000", 2024-03-19T11:32:01.433 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3124000000000003e-05 2024-03-19T11:32:01.433 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.433 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.433 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:01.433 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.316534+0000", 2024-03-19T11:32:01.433 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.485e-05 2024-03-19T11:32:01.433 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.433 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.433 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:01.434 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.330415+0000", 2024-03-19T11:32:01.434 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013880905000000001 2024-03-19T11:32:01.434 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.434 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.434 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:01.434 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.330439+0000", 2024-03-19T11:32:01.434 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4630999999999998e-05 2024-03-19T11:32:01.434 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.434 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.434 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:01.434 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.332778+0000", 2024-03-19T11:32:01.434 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023391800000000002 2024-03-19T11:32:01.434 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:01.434 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:01.434 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:01.435 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.435 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.435 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2387 4.2 4:43b32242:::smithi03829856-22:head [read 596992~684544] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-03-19T11:32:01.435 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:59.319806+0000", 2024-03-19T11:32:01.435 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 62.054382390999997, 2024-03-19T11:32:01.435 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.78144344399999999, 2024-03-19T11:32:01.435 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:01.435 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:01.435 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:01.435 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:01.435 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:01.435 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2387 2024-03-19T11:32:01.435 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.435 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:01.436 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.436 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:01.436 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319806+0000", 2024-03-19T11:32:01.436 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:01.436 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.436 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.436 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:01.436 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319806+0000", 2024-03-19T11:32:01.436 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:01.436 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.436 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.436 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:01.436 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319807+0000", 2024-03-19T11:32:01.436 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3519999999999999e-06 2024-03-19T11:32:01.436 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.437 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.437 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:01.437 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319812+0000", 2024-03-19T11:32:01.437 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3959999999999997e-06 2024-03-19T11:32:01.437 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.437 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.437 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:01.437 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319827+0000", 2024-03-19T11:32:01.437 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4566000000000001e-05 2024-03-19T11:32:01.437 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.437 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.437 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:01.437 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319835+0000", 2024-03-19T11:32:01.437 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.994e-06 2024-03-19T11:32:01.437 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.438 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.438 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:01.438 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.320119+0000", 2024-03-19T11:32:01.438 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00028445199999999998 2024-03-19T11:32:01.438 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.438 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.438 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:01.438 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.320129+0000", 2024-03-19T11:32:01.438 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9580000000000002e-06 2024-03-19T11:32:01.438 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.438 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.438 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:01.438 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.069842+0000", 2024-03-19T11:32:01.438 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.74971218399999995 2024-03-19T11:32:01.439 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.439 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.439 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:32:01.439 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.070010+0000", 2024-03-19T11:32:01.439 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016846600000000001 2024-03-19T11:32:01.439 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.439 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.439 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:01.439 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.090911+0000", 2024-03-19T11:32:01.439 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020901494999999999 2024-03-19T11:32:01.439 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.439 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.439 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:32:01.439 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.091249+0000", 2024-03-19T11:32:01.440 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000337602 2024-03-19T11:32:01.440 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.440 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.440 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:01.440 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.098478+0000", 2024-03-19T11:32:01.440 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0072287549999999999 2024-03-19T11:32:01.440 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.440 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.440 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:01.440 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.098617+0000", 2024-03-19T11:32:01.440 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013871599999999999 2024-03-19T11:32:01.440 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.440 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.440 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:01.440 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.101249+0000", 2024-03-19T11:32:01.441 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002632508 2024-03-19T11:32:01.441 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:01.441 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:01.441 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:01.441 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.441 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.441 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 81/80 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 15=[]:{7: [6], b: [b], 10: [d], 12: [12, 11], 15: [15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:7, version: 55'82, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@55'82, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 16=[]:{7: [6], b: [b], 10: [d], 16: [16, 15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 15=[]:{7: [6], b: [b], 10: [d], 12: [12, 11], 15: [15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 58'86, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@58'86, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 16=[]:{7: [6], b: [b], 10: [d], 16: [16, 15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [504210~968657,1572695~79919], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 16=[]:{7: [6], b: [b], 10: [d], 16: [16, 15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:32:01.441 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:00.089924+0000", 2024-03-19T11:32:01.441 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 61.284264161999999, 2024-03-19T11:32:01.441 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024686583000000002, 2024-03-19T11:32:01.441 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:01.441 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:01.441 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:01.441 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.442 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:01.442 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.089924+0000", 2024-03-19T11:32:01.442 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:01.442 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.442 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.442 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:01.442 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.089924+0000", 2024-03-19T11:32:01.442 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:01.442 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.442 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.442 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:01.442 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.089926+0000", 2024-03-19T11:32:01.442 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9589999999999998e-06 2024-03-19T11:32:01.442 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.442 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.443 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:01.443 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093311+0000", 2024-03-19T11:32:01.443 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033854409999999999 2024-03-19T11:32:01.443 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.443 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.443 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:01.443 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093365+0000", 2024-03-19T11:32:01.443 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3795000000000002e-05 2024-03-19T11:32:01.443 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.443 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.443 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:01.443 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093438+0000", 2024-03-19T11:32:01.443 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2591999999999999e-05 2024-03-19T11:32:01.443 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.443 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.444 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:01.444 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093690+0000", 2024-03-19T11:32:01.444 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025264999999999999 2024-03-19T11:32:01.444 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.444 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.444 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:01.444 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093731+0000", 2024-03-19T11:32:01.444 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0881999999999999e-05 2024-03-19T11:32:01.444 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.444 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.444 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:01.444 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.114610+0000", 2024-03-19T11:32:01.444 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020879264000000002 2024-03-19T11:32:01.444 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:01.445 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:01.445 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:01.445 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.445 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.445 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.21 e81,80 4:8401c1f0:::smithi03829856-43:8,46'66,4:866f3dcc:::smithi03829856-31:2,47'73)", 2024-03-19T11:32:01.445 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:00.093514+0000", 2024-03-19T11:32:01.445 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 61.280673561999997, 2024-03-19T11:32:01.445 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027375713999999999, 2024-03-19T11:32:01.445 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:01.445 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:01.445 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:01.445 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.445 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:01.445 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093514+0000", 2024-03-19T11:32:01.445 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:01.445 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.446 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.446 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:01.446 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093514+0000", 2024-03-19T11:32:01.446 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:01.446 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.446 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.446 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:01.446 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093516+0000", 2024-03-19T11:32:01.446 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1179999999999999e-06 2024-03-19T11:32:01.446 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.446 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.446 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:01.446 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093523+0000", 2024-03-19T11:32:01.446 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7500000000000003e-06 2024-03-19T11:32:01.447 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.447 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.447 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:01.447 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093534+0000", 2024-03-19T11:32:01.447 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0831e-05 2024-03-19T11:32:01.447 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.447 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.447 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:01.447 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093542+0000", 2024-03-19T11:32:01.447 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0139999999999998e-06 2024-03-19T11:32:01.447 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.447 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.447 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:01.447 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.120682+0000", 2024-03-19T11:32:01.447 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027139838999999999 2024-03-19T11:32:01.448 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.448 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.448 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:01.448 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.120693+0000", 2024-03-19T11:32:01.448 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.136e-05 2024-03-19T11:32:01.448 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.448 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.448 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:01.448 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.120890+0000", 2024-03-19T11:32:01.448 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019680199999999999 2024-03-19T11:32:01.448 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:01.448 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:01.448 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:01.448 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.449 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.449 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2927 4.24 4:266590a5:::smithi03829856-16:head [append 0~0] snapc 1b=[1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-19T11:32:01.449 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:18.054474+0000", 2024-03-19T11:32:01.449 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 43.319714314999999, 2024-03-19T11:32:01.449 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024856390999999999, 2024-03-19T11:32:01.449 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:01.449 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:01.449 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:01.449 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:01.449 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:01.449 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2927 2024-03-19T11:32:01.449 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.449 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:01.449 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.449 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:01.450 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054474+0000", 2024-03-19T11:32:01.450 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:01.450 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.450 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.450 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:01.450 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054474+0000", 2024-03-19T11:32:01.450 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:01.450 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.450 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.450 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:01.450 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054476+0000", 2024-03-19T11:32:01.450 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6469999999999998e-06 2024-03-19T11:32:01.450 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.450 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.451 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:01.451 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054489+0000", 2024-03-19T11:32:01.451 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2612999999999999e-05 2024-03-19T11:32:01.451 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.451 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.451 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:01.451 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054526+0000", 2024-03-19T11:32:01.451 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7481999999999998e-05 2024-03-19T11:32:01.451 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.451 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.451 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:01.451 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054547+0000", 2024-03-19T11:32:01.451 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0820999999999999e-05 2024-03-19T11:32:01.451 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.452 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.452 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:01.452 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054633+0000", 2024-03-19T11:32:01.452 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5556000000000006e-05 2024-03-19T11:32:01.452 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.452 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.452 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-03-19T11:32:01.452 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054639+0000", 2024-03-19T11:32:01.452 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7300000000000002e-06 2024-03-19T11:32:01.452 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.452 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.452 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:01.452 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.061592+0000", 2024-03-19T11:32:01.452 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0069539279999999998 2024-03-19T11:32:01.452 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.453 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.453 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:01.453 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.061746+0000", 2024-03-19T11:32:01.453 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015351699999999999 2024-03-19T11:32:01.453 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.453 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.453 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:32:01.453 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.062031+0000", 2024-03-19T11:32:01.453 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00028539700000000001 2024-03-19T11:32:01.453 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.453 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.453 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:32:01.453 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.079164+0000", 2024-03-19T11:32:01.453 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017132865000000001 2024-03-19T11:32:01.453 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.454 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.454 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:01.454 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.079217+0000", 2024-03-19T11:32:01.454 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2373999999999997e-05 2024-03-19T11:32:01.454 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.454 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.454 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:32:01.454 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.079284+0000", 2024-03-19T11:32:01.454 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7433000000000003e-05 2024-03-19T11:32:01.454 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.454 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.454 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:01.454 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.079330+0000", 2024-03-19T11:32:01.454 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6028e-05 2024-03-19T11:32:01.454 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:01.455 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:01.455 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:01.455 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.455 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.455 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2928 4.24 4:266590a5:::smithi03829856-16:head [rollback b] snapc 1b=[1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-19T11:32:01.455 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:18.054594+0000", 2024-03-19T11:32:01.455 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 43.319594328000001, 2024-03-19T11:32:01.455 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025781921999999999, 2024-03-19T11:32:01.455 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:01.455 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:01.455 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:01.455 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:01.455 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:01.455 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2928 2024-03-19T11:32:01.456 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.456 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:01.456 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.456 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:01.456 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054594+0000", 2024-03-19T11:32:01.456 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:01.456 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.456 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.456 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:01.456 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054594+0000", 2024-03-19T11:32:01.456 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:01.456 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.456 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.456 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:01.456 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054595+0000", 2024-03-19T11:32:01.457 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.018e-06 2024-03-19T11:32:01.457 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.457 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.457 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:01.457 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054601+0000", 2024-03-19T11:32:01.457 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2620000000000004e-06 2024-03-19T11:32:01.457 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.457 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.457 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:01.457 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054618+0000", 2024-03-19T11:32:01.457 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6877000000000001e-05 2024-03-19T11:32:01.457 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.457 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.457 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:01.457 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054632+0000", 2024-03-19T11:32:01.458 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4175000000000001e-05 2024-03-19T11:32:01.458 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.458 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.458 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:01.458 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054676+0000", 2024-03-19T11:32:01.458 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3659000000000003e-05 2024-03-19T11:32:01.458 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.458 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.458 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:32:01.458 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054679+0000", 2024-03-19T11:32:01.458 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0649999999999999e-06 2024-03-19T11:32:01.458 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.458 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.458 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:01.459 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.064021+0000", 2024-03-19T11:32:01.459 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093424210000000001 2024-03-19T11:32:01.459 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.459 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.459 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:01.459 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.064158+0000", 2024-03-19T11:32:01.459 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013652099999999999 2024-03-19T11:32:01.459 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.459 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.459 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:32:01.459 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.072763+0000", 2024-03-19T11:32:01.459 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.008605784 2024-03-19T11:32:01.459 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.459 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.460 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:32:01.460 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.079171+0000", 2024-03-19T11:32:01.460 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0064071320000000003 2024-03-19T11:32:01.460 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.460 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.460 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:01.460 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.080257+0000", 2024-03-19T11:32:01.460 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0010866280000000001 2024-03-19T11:32:01.460 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.460 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.460 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:32:01.460 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.080324+0000", 2024-03-19T11:32:01.460 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6976000000000004e-05 2024-03-19T11:32:01.460 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.461 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.461 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:01.461 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.080376+0000", 2024-03-19T11:32:01.461 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1403999999999998e-05 2024-03-19T11:32:01.461 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:01.461 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:01.461 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:01.461 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.461 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.461 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2929 4.24 4:266590a5:::smithi03829856-16:head [append 0~0] snapc 1b=[1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-19T11:32:01.461 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:18.054649+0000", 2024-03-19T11:32:01.461 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 43.319538860999998, 2024-03-19T11:32:01.461 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026690519999999999, 2024-03-19T11:32:01.461 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:01.461 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:01.462 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:01.462 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:01.462 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:01.462 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2929 2024-03-19T11:32:01.462 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.462 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:01.462 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.462 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:01.462 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054649+0000", 2024-03-19T11:32:01.462 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:01.462 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.462 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.462 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:01.462 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054649+0000", 2024-03-19T11:32:01.463 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:01.463 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.463 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.463 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:01.463 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054650+0000", 2024-03-19T11:32:01.463 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5199999999999995e-07 2024-03-19T11:32:01.463 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.463 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.463 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:01.463 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054655+0000", 2024-03-19T11:32:01.463 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1529999999999996e-06 2024-03-19T11:32:01.463 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.463 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.463 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:01.464 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054670+0000", 2024-03-19T11:32:01.464 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5029e-05 2024-03-19T11:32:01.464 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.464 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.464 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:01.464 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054683+0000", 2024-03-19T11:32:01.464 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2670000000000001e-05 2024-03-19T11:32:01.464 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.464 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.464 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:01.464 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054756+0000", 2024-03-19T11:32:01.464 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3559000000000005e-05 2024-03-19T11:32:01.464 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.464 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.465 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:32:01.465 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054760+0000", 2024-03-19T11:32:01.465 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1659999999999998e-06 2024-03-19T11:32:01.465 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.465 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.465 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:01.465 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.075329+0000", 2024-03-19T11:32:01.465 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020569918999999999 2024-03-19T11:32:01.465 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.465 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.465 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:01.465 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.075469+0000", 2024-03-19T11:32:01.465 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013990000000000001 2024-03-19T11:32:01.465 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.465 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.466 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:32:01.466 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.075730+0000", 2024-03-19T11:32:01.466 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00026011300000000002 2024-03-19T11:32:01.466 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.466 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.466 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:32:01.466 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.079918+0000", 2024-03-19T11:32:01.466 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0041886279999999998 2024-03-19T11:32:01.466 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.466 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.466 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:01.466 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.081271+0000", 2024-03-19T11:32:01.466 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0013529449999999999 2024-03-19T11:32:01.467 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.467 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.467 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:32:01.467 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.081308+0000", 2024-03-19T11:32:01.467 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6437999999999999e-05 2024-03-19T11:32:01.467 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:01.467 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:01.467 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:01.467 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.081340+0000", 2024-03-19T11:32:01.467 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2147999999999998e-05 2024-03-19T11:32:01.467 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:01.467 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:01.467 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:01.467 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:01.467 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:01.468 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:32:01.468 INFO:tasks.rados.rados.0.smithi038.stdout:1769: expect (ObjNum 295 snap 39 seq_num 295) 2024-03-19T11:32:01.469 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-19T11:32:01.470 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:32:01.470 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:32:01.470 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:32:01.477+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:32:01.483 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:32:01.484 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:32:01.501 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:32:01.501 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:32:01.518 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:32:01.518 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:32:01.537 INFO:tasks.rados.rados.0.smithi038.stdout:1762: finishing write tid 3 to smithi03829856-45 2024-03-19T11:32:01.538 INFO:tasks.rados.rados.0.smithi038.stdout:1762: finishing write tid 4 to smithi03829856-45 2024-03-19T11:32:01.538 INFO:tasks.rados.rados.0.smithi038.stdout:1762: finishing write tid 5 to smithi03829856-45 2024-03-19T11:32:01.538 INFO:tasks.rados.rados.0.smithi038.stdout:1762: oid 45 updating version 0 to 102 2024-03-19T11:32:01.538 INFO:tasks.rados.rados.0.smithi038.stdout:1762: oid 45 version 102 is already newer than 100 2024-03-19T11:32:01.538 INFO:tasks.rados.rados.0.smithi038.stdout:1762: oid 45 version 102 is already newer than 101 2024-03-19T11:32:01.538 INFO:tasks.rados.rados.0.smithi038.stdout:1762: oid 45 version 102 is already newer than 99 2024-03-19T11:32:01.538 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 45 v 102 (ObjNum 327 snap 40 seq_num 327) dirty exists 2024-03-19T11:32:01.538 INFO:tasks.rados.rados.0.smithi038.stdout:1762: left oid 45 (ObjNum 327 snap 40 seq_num 327) 2024-03-19T11:32:01.539 INFO:tasks.rados.rados.0.smithi038.stdout:1770: finishing write tid 1 to smithi03829856-26 2024-03-19T11:32:01.539 INFO:tasks.rados.rados.0.smithi038.stdout:1770: finishing write tid 2 to smithi03829856-26 2024-03-19T11:32:01.539 INFO:tasks.rados.rados.0.smithi038.stdout:1762: done (14 left) 2024-03-19T11:32:01.539 INFO:tasks.rados.rados.0.smithi038.stdout:1763: done (13 left) 2024-03-19T11:32:01.539 INFO:tasks.rados.rados.0.smithi038.stdout:1764: done (12 left) 2024-03-19T11:32:01.539 INFO:tasks.rados.rados.0.smithi038.stdout:1765: done (11 left) 2024-03-19T11:32:01.539 INFO:tasks.rados.rados.0.smithi038.stdout:1766: done (10 left) 2024-03-19T11:32:01.539 INFO:tasks.rados.rados.0.smithi038.stdout:1767: done (9 left) 2024-03-19T11:32:01.539 INFO:tasks.rados.rados.0.smithi038.stdout:1769: done (8 left) 2024-03-19T11:32:01.539 INFO:tasks.rados.rados.0.smithi038.stdout:1777: done (7 left) 2024-03-19T11:32:01.539 INFO:tasks.rados.rados.0.smithi038.stdout:1778: read oid 39 snap -1 2024-03-19T11:32:01.539 INFO:tasks.rados.rados.0.smithi038.stdout:1778: expect deleted 2024-03-19T11:32:01.539 INFO:tasks.rados.rados.0.smithi038.stdout:1779: tier_promote oid 12 2024-03-19T11:32:01.539 INFO:tasks.rados.rados.0.smithi038.stdout:1770: finishing write tid 3 to smithi03829856-26 2024-03-19T11:32:01.539 INFO:tasks.rados.rados.0.smithi038.stdout:1770: finishing write tid 4 to smithi03829856-26 2024-03-19T11:32:01.540 INFO:tasks.rados.rados.0.smithi038.stdout:1770: finishing write tid 5 to smithi03829856-26 2024-03-19T11:32:01.540 INFO:tasks.rados.rados.0.smithi038.stdout:1770: oid 26 updating version 0 to 81 2024-03-19T11:32:01.540 INFO:tasks.rados.rados.0.smithi038.stdout:1770: oid 26 version 81 is already newer than 80 2024-03-19T11:32:01.540 INFO:tasks.rados.rados.0.smithi038.stdout:1770: oid 26 version 81 is already newer than 78 2024-03-19T11:32:01.540 INFO:tasks.rados.rados.0.smithi038.stdout:1770: oid 26 version 81 is already newer than 79 2024-03-19T11:32:01.540 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 26 v 81 (ObjNum 328 snap 40 seq_num 328) dirty exists 2024-03-19T11:32:01.540 INFO:tasks.rados.rados.0.smithi038.stdout:1770: left oid 26 (ObjNum 328 snap 40 seq_num 328) 2024-03-19T11:32:01.540 INFO:tasks.rados.rados.0.smithi038.stdout:1771: finishing set_chunk to oid 4 2024-03-19T11:32:01.540 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 4 v 201 (ObjNum 313 snap 40 seq_num 313) dirty exists 2024-03-19T11:32:01.540 INFO:tasks.rados.rados.0.smithi038.stdout:1770: done (8 left) 2024-03-19T11:32:01.540 INFO:tasks.rados.rados.0.smithi038.stdout:1771: done (7 left) 2024-03-19T11:32:01.540 INFO:tasks.rados.rados.0.smithi038.stdout:1780: read oid 5 snap -1 2024-03-19T11:32:01.540 INFO:tasks.rados.rados.0.smithi038.stdout:1780: expect (ObjNum 310 snap 39 seq_num 310) 2024-03-19T11:32:01.540 INFO:tasks.rados.rados.0.smithi038.stdout:1781: read oid 2 snap -1 2024-03-19T11:32:01.540 INFO:tasks.rados.rados.0.smithi038.stdout:1781: expect (ObjNum 206 snap 22 seq_num 206) 2024-03-19T11:32:01.541 INFO:tasks.rados.rados.0.smithi038.stdout:1782: read oid 13 snap -1 2024-03-19T11:32:01.541 INFO:tasks.rados.rados.0.smithi038.stdout:1782: expect (ObjNum 282 snap 37 seq_num 282) 2024-03-19T11:32:01.541 INFO:tasks.rados.rados.0.smithi038.stdout:1783: read oid 41 snap -1 2024-03-19T11:32:01.541 INFO:tasks.rados.rados.0.smithi038.stdout:1783: expect deleted 2024-03-19T11:32:01.541 INFO:tasks.rados.rados.0.smithi038.stdout:1784: set_chunk oid 24 target oid 2024-03-19T11:32:01.542 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:32:01.542 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:32:01.548 INFO:tasks.rados.rados.0.smithi038.stdout:1784: set_chunk oid 24 offset: 595968 length: 707072 target oid b9a9d44d1b933ccf11217fc0037f822a9e5bd67a495395fc5cbfdd094484691f offset: 0 snap -1 2024-03-19T11:32:01.548 INFO:tasks.rados.rados.0.smithi038.stdout:1785: set_chunk oid 10 target oid 2024-03-19T11:32:01.548 INFO:tasks.rados.rados.0.smithi038.stdout:1785: set_chunk oid 10 offset: 0 length: 4000000 target oid smithi03829856- offset: 0 snap -1 2024-03-19T11:32:01.548 INFO:tasks.rados.rados.0.smithi038.stdout:1786: set_chunk oid 25 target oid 2024-03-19T11:32:01.548 INFO:tasks.rados.rados.0.smithi038.stdout:1772: finishing set_chunk to oid 28 2024-03-19T11:32:01.548 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 28 set_chunk returned error code -95 offset: 2083328 length: 1216000 2024-03-19T11:32:01.548 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 28 v 0 (ObjNum 87 snap 8 seq_num 87) dirty exists 2024-03-19T11:32:01.562 INFO:tasks.rados.rados.0.smithi038.stdout:1786: set_chunk oid 25 offset: 291328 length: 1269248 target oid 60a6ef3660aee1e5dc7e8a3e84e014a19b1d1e09864bc23b8365611be9831bc5 offset: 0 snap -1 2024-03-19T11:32:01.563 INFO:tasks.rados.rados.0.smithi038.stdout:1772: done (13 left) 2024-03-19T11:32:01.563 INFO:tasks.rados.rados.0.smithi038.stdout:1787: write oid 17 current snap is 40 2024-03-19T11:32:01.563 INFO:tasks.rados.rados.0.smithi038.stdout:1787: seq_num 332 ranges {779924=785847,2078257=580112,3301257=278667} 2024-03-19T11:32:01.563 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:32:01.564 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:32:01.570 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:32:01.577+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:32:01.571 INFO:tasks.rados.rados.0.smithi038.stdout:1787: writing smithi03829856-17 from 779924 to 1565771 tid 1 2024-03-19T11:32:01.578 INFO:tasks.rados.rados.0.smithi038.stdout:1787: writing smithi03829856-17 from 2078257 to 2658369 tid 2 2024-03-19T11:32:01.585 INFO:tasks.rados.rados.0.smithi038.stdout:1787: writing smithi03829856-17 from 3301257 to 3579924 tid 3 2024-03-19T11:32:01.585 INFO:tasks.rados.rados.0.smithi038.stdout:1788: rollback oid 29 current snap is 40 2024-03-19T11:32:01.585 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 29 to 37 2024-03-19T11:32:01.585 INFO:tasks.rados.rados.0.smithi038.stdout:1773: expect (ObjNum 306 snap 39 seq_num 306) 2024-03-19T11:32:01.594 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:32:01.594 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:32:01.601 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:01.601 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:01.615 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-19T11:32:01.615 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:32:01.615 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:32:01.673 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:32:01.683+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:32:01.771 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:32:01.779+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:32:01.784 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:01.784 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:01.796 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-19T11:32:01.867 INFO:tasks.rados.rados.0.smithi038.stdout:1774: finishing write tid 1 to smithi03829856-14 2024-03-19T11:32:01.867 INFO:tasks.rados.rados.0.smithi038.stdout:1774: finishing write tid 2 to smithi03829856-14 2024-03-19T11:32:01.867 INFO:tasks.rados.rados.0.smithi038.stdout:1774: finishing write tid 3 to smithi03829856-14 2024-03-19T11:32:01.868 INFO:tasks.rados.rados.0.smithi038.stdout:1774: finishing write tid 4 to smithi03829856-14 2024-03-19T11:32:01.868 INFO:tasks.rados.rados.0.smithi038.stdout:1774: finishing write tid 5 to smithi03829856-14 2024-03-19T11:32:01.868 INFO:tasks.rados.rados.0.smithi038.stdout:1774: oid 14 updating version 0 to 95 2024-03-19T11:32:01.868 INFO:tasks.rados.rados.0.smithi038.stdout:1774: oid 14 version 95 is already newer than 92 2024-03-19T11:32:01.868 INFO:tasks.rados.rados.0.smithi038.stdout:1774: oid 14 version 95 is already newer than 93 2024-03-19T11:32:01.868 INFO:tasks.rados.rados.0.smithi038.stdout:1774: oid 14 version 95 is already newer than 94 2024-03-19T11:32:01.868 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 14 v 95 (ObjNum 329 snap 40 seq_num 329) dirty exists 2024-03-19T11:32:01.868 INFO:tasks.rados.rados.0.smithi038.stdout:1774: left oid 14 (ObjNum 329 snap 40 seq_num 329) 2024-03-19T11:32:01.868 INFO:tasks.rados.rados.0.smithi038.stdout:1776: finishing write tid 1 to smithi03829856-37 2024-03-19T11:32:01.868 INFO:tasks.rados.rados.0.smithi038.stdout:1776: finishing write tid 2 to smithi03829856-37 2024-03-19T11:32:01.868 INFO:tasks.rados.rados.0.smithi038.stdout:1776: finishing write tid 3 to smithi03829856-37 2024-03-19T11:32:01.868 INFO:tasks.rados.rados.0.smithi038.stdout:1776: finishing write tid 4 to smithi03829856-37 2024-03-19T11:32:01.868 INFO:tasks.rados.rados.0.smithi038.stdout:1776: finishing write tid 5 to smithi03829856-37 2024-03-19T11:32:01.868 INFO:tasks.rados.rados.0.smithi038.stdout:1776: oid 37 updating version 0 to 151 2024-03-19T11:32:01.868 INFO:tasks.rados.rados.0.smithi038.stdout:1776: oid 37 version 151 is already newer than 146 2024-03-19T11:32:01.869 INFO:tasks.rados.rados.0.smithi038.stdout:1776: oid 37 version 151 is already newer than 147 2024-03-19T11:32:01.869 INFO:tasks.rados.rados.0.smithi038.stdout:1776: oid 37 version 151 is already newer than 148 2024-03-19T11:32:01.869 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 37 v 151 (ObjNum 330 snap 40 seq_num 330) dirty exists 2024-03-19T11:32:01.869 INFO:tasks.rados.rados.0.smithi038.stdout:1776: left oid 37 (ObjNum 330 snap 40 seq_num 330) 2024-03-19T11:32:01.869 INFO:tasks.rados.rados.0.smithi038.stdout:1773: done (14 left) 2024-03-19T11:32:01.869 INFO:tasks.rados.rados.0.smithi038.stdout:1774: done (13 left) 2024-03-19T11:32:01.869 INFO:tasks.rados.rados.0.smithi038.stdout:1776: done (12 left) 2024-03-19T11:32:01.869 INFO:tasks.rados.rados.0.smithi038.stdout:1789: delete oid 14 current snap is 40 2024-03-19T11:32:01.869 INFO:tasks.rados.rados.0.smithi038.stdout:1779: got (0) Success 2024-03-19T11:32:01.869 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 12 v 146 (ObjNum 325 snap 40 seq_num 325) dirty exists 2024-03-19T11:32:01.869 INFO:tasks.rados.rados.0.smithi038.stdout:1781: expect (ObjNum 206 snap 22 seq_num 206) 2024-03-19T11:32:01.871 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:32:01.879+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:32:01.956 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:01.956 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:01.970 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-19T11:32:02.013 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:32:02.023+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:32:02.024 INFO:tasks.rados.rados.0.smithi038.stdout:1780: expect (ObjNum 310 snap 39 seq_num 310) 2024-03-19T11:32:02.074 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:32:02.081+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:32:02.112 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:32:02.121+0000 7f1835403640 -1 osd.6 140 handle_osd_map injecting map crc failure 2024-03-19T11:32:02.148 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:02.148 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:02.161 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-19T11:32:02.175 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:32:02.181+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:32:02.239 INFO:tasks.rados.rados.0.smithi038.stdout:1784: finishing set_chunk to oid 24 2024-03-19T11:32:02.240 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 24 set_chunk returned error code -95 offset: 595968 length: 707072 2024-03-19T11:32:02.240 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 24 v 0 (ObjNum 265 snap 34 seq_num 265) dirty exists 2024-03-19T11:32:02.240 INFO:tasks.rados.rados.0.smithi038.stdout:1785: finishing set_chunk to oid 10 2024-03-19T11:32:02.240 INFO:tasks.rados.rados.0.smithi038.stdout:1785: got expected ERANGE (src dne) 2024-03-19T11:32:02.240 INFO:tasks.rados.rados.0.smithi038.stdout:1782: expect (ObjNum 282 snap 37 seq_num 282) 2024-03-19T11:32:02.276 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:32:02.285+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:32:02.318 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:02.318 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:02.331 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-19T11:32:02.362 INFO:tasks.rados.rados.0.smithi038.stdout:1786: finishing set_chunk to oid 25 2024-03-19T11:32:02.363 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 25 v 149 (ObjNum 309 snap 39 seq_num 309) dirty exists 2024-03-19T11:32:02.363 INFO:tasks.rados.rados.0.smithi038.stdout:1787: finishing write tid 1 to smithi03829856-17 2024-03-19T11:32:02.363 INFO:tasks.rados.rados.0.smithi038.stdout:1787: finishing write tid 2 to smithi03829856-17 2024-03-19T11:32:02.363 INFO:tasks.rados.rados.0.smithi038.stdout:1787: finishing write tid 3 to smithi03829856-17 2024-03-19T11:32:02.363 INFO:tasks.rados.rados.0.smithi038.stdout:1788: finishing rollback tid 0 to smithi03829856-29 2024-03-19T11:32:02.363 INFO:tasks.rados.rados.0.smithi038.stdout:1787: finishing write tid 4 to smithi03829856-17 2024-03-19T11:32:02.363 INFO:tasks.rados.rados.0.smithi038.stdout:1787: finishing write tid 5 to smithi03829856-17 2024-03-19T11:32:02.363 INFO:tasks.rados.rados.0.smithi038.stdout:1787: oid 17 updating version 0 to 315 2024-03-19T11:32:02.363 INFO:tasks.rados.rados.0.smithi038.stdout:1787: oid 17 version 315 is already newer than 313 2024-03-19T11:32:02.363 INFO:tasks.rados.rados.0.smithi038.stdout:1787: oid 17 version 315 is already newer than 314 2024-03-19T11:32:02.363 INFO:tasks.rados.rados.0.smithi038.stdout:1787: oid 17 updating version 315 to 316 2024-03-19T11:32:02.363 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 17 v 316 (ObjNum 332 snap 40 seq_num 332) dirty exists 2024-03-19T11:32:02.363 INFO:tasks.rados.rados.0.smithi038.stdout:1787: left oid 17 (ObjNum 332 snap 40 seq_num 332) 2024-03-19T11:32:02.363 INFO:tasks.rados.rados.0.smithi038.stdout:1788: finishing rollback tid 1 to smithi03829856-29 2024-03-19T11:32:02.364 INFO:tasks.rados.rados.0.smithi038.stdout:1788: finishing rollback tid 2 to smithi03829856-29 2024-03-19T11:32:02.364 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 29 v 124 (ObjNum 284 snap 37 seq_num 284) dirty exists 2024-03-19T11:32:02.364 INFO:tasks.rados.rados.0.smithi038.stdout:1778: done (12 left) 2024-03-19T11:32:02.364 INFO:tasks.rados.rados.0.smithi038.stdout:1779: done (11 left) 2024-03-19T11:32:02.364 INFO:tasks.rados.rados.0.smithi038.stdout:1780: done (10 left) 2024-03-19T11:32:02.364 INFO:tasks.rados.rados.0.smithi038.stdout:1781: done (9 left) 2024-03-19T11:32:02.364 INFO:tasks.rados.rados.0.smithi038.stdout:1782: done (8 left) 2024-03-19T11:32:02.364 INFO:tasks.rados.rados.0.smithi038.stdout:1783: done (7 left) 2024-03-19T11:32:02.364 INFO:tasks.rados.rados.0.smithi038.stdout:1784: done (6 left) 2024-03-19T11:32:02.364 INFO:tasks.rados.rados.0.smithi038.stdout:1785: done (5 left) 2024-03-19T11:32:02.364 INFO:tasks.rados.rados.0.smithi038.stdout:1786: done (4 left) 2024-03-19T11:32:02.364 INFO:tasks.rados.rados.0.smithi038.stdout:1787: done (3 left) 2024-03-19T11:32:02.364 INFO:tasks.rados.rados.0.smithi038.stdout:1788: done (2 left) 2024-03-19T11:32:02.364 INFO:tasks.rados.rados.0.smithi038.stdout:1789: done (1 left) 2024-03-19T11:32:02.364 INFO:tasks.rados.rados.0.smithi038.stdout:1790: read oid 42 snap -1 2024-03-19T11:32:02.365 INFO:tasks.rados.rados.0.smithi038.stdout:1790: expect deleted 2024-03-19T11:32:02.365 INFO:tasks.rados.rados.0.smithi038.stdout:1791: read oid 38 snap -1 2024-03-19T11:32:02.365 INFO:tasks.rados.rados.0.smithi038.stdout:1791: expect (ObjNum 322 snap 40 seq_num 322) 2024-03-19T11:32:02.365 INFO:tasks.rados.rados.0.smithi038.stdout:1792: tier_promote oid 44 2024-03-19T11:32:02.365 INFO:tasks.rados.rados.0.smithi038.stdout:1768: expect (ObjNum 300 snap 39 seq_num 300) 2024-03-19T11:32:02.409 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:32:02.419+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:32:02.480 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:32:02.489+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:32:02.480 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:02.480 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:02.493 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-19T11:32:02.519 INFO:tasks.rados.rados.0.smithi038.stdout:1792: got (0) Success 2024-03-19T11:32:02.519 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 44 v 128 (ObjNum 288 snap 38 seq_num 288) dirty exists 2024-03-19T11:32:02.519 INFO:tasks.rados.rados.0.smithi038.stdout:1791: expect (ObjNum 322 snap 40 seq_num 322) 2024-03-19T11:32:02.581 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:32:02.589+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:32:02.626 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:02.626 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:02.640 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-19T11:32:02.681 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:32:02.691+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:32:02.750 INFO:tasks.rados.rados.0.smithi038.stdout:1768: done (3 left) 2024-03-19T11:32:02.750 INFO:tasks.rados.rados.0.smithi038.stdout:1790: done (2 left) 2024-03-19T11:32:02.750 INFO:tasks.rados.rados.0.smithi038.stdout:1791: done (1 left) 2024-03-19T11:32:02.750 INFO:tasks.rados.rados.0.smithi038.stdout:1792: done (0 left) 2024-03-19T11:32:02.751 INFO:tasks.rados.rados.0.smithi038.stdout:1793: write oid 49 current snap is 40 2024-03-19T11:32:02.751 INFO:tasks.rados.rados.0.smithi038.stdout:1793: seq_num 334 ranges {413538=536345,1395221=583781,2556103=658617} 2024-03-19T11:32:02.766 INFO:tasks.rados.rados.0.smithi038.stdout:1793: writing smithi03829856-49 from 413538 to 949883 tid 1 2024-03-19T11:32:02.782 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:32:02.791+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:32:02.782 INFO:tasks.rados.rados.0.smithi038.stdout:1793: writing smithi03829856-49 from 1395221 to 1979002 tid 2 2024-03-19T11:32:02.802 INFO:tasks.rados.rados.0.smithi038.stdout:1793: writing smithi03829856-49 from 2556103 to 3214720 tid 3 2024-03-19T11:32:02.802 INFO:tasks.rados.rados.0.smithi038.stdout:1794: write (excl) oid 32 current snap is 40 2024-03-19T11:32:02.802 INFO:tasks.rados.rados.0.smithi038.stdout:1794: seq_num 335 ranges {430345=727947,1570056=727439,2699879=469889} 2024-03-19T11:32:02.811 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:02.811 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:02.817 INFO:tasks.rados.rados.0.smithi038.stdout:1794: writing smithi03829856-32 from 430345 to 1158292 tid 1 2024-03-19T11:32:02.827 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-19T11:32:02.828 INFO:tasks.rados.rados.0.smithi038.stdout:1794: writing smithi03829856-32 from 1570056 to 2297495 tid 2 2024-03-19T11:32:02.837 INFO:tasks.rados.rados.0.smithi038.stdout:1794: writing smithi03829856-32 from 2699879 to 3169768 tid 3 2024-03-19T11:32:02.837 INFO:tasks.rados.rados.0.smithi038.stdout:1795: write oid 37 current snap is 40 2024-03-19T11:32:02.837 INFO:tasks.rados.rados.0.smithi038.stdout:1795: seq_num 336 ranges {447152=403196,1512185=587450,2727302=681161} 2024-03-19T11:32:02.844 INFO:tasks.rados.rados.0.smithi038.stdout:1795: writing smithi03829856-37 from 447152 to 850348 tid 1 2024-03-19T11:32:02.854 INFO:tasks.rados.rados.0.smithi038.stdout:1795: writing smithi03829856-37 from 1512185 to 2099635 tid 2 2024-03-19T11:32:02.863 INFO:tasks.rados.rados.0.smithi038.stdout:1795: writing smithi03829856-37 from 2727302 to 3408463 tid 3 2024-03-19T11:32:02.863 INFO:tasks.rados.rados.0.smithi038.stdout:1796: read oid 25 snap 28 2024-03-19T11:32:02.863 INFO:tasks.rados.rados.0.smithi038.stdout:1796: expect deleted 2024-03-19T11:32:02.863 INFO:tasks.rados.rados.0.smithi038.stdout:1797: snap_remove snap 34 2024-03-19T11:32:02.883 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:32:02.893+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:32:02.962 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:02.962 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:02.975 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-19T11:32:03.020 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:32:03.029+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:32:03.058 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:32:03.065+0000 7f1837407640 -1 osd.6 141 handle_osd_map injecting map crc failure 2024-03-19T11:32:03.069 INFO:tasks.rados.rados.0.smithi038.stdout:1797: done (4 left) 2024-03-19T11:32:03.069 INFO:tasks.rados.rados.0.smithi038.stdout:1798: set_chunk oid 21 target oid 2024-03-19T11:32:03.069 INFO:tasks.rados.rados.0.smithi038.stdout:1793: finishing write tid 1 to smithi03829856-49 2024-03-19T11:32:03.069 INFO:tasks.rados.rados.0.smithi038.stdout:1793: finishing write tid 2 to smithi03829856-49 2024-03-19T11:32:03.070 INFO:tasks.rados.rados.0.smithi038.stdout:1793: finishing write tid 3 to smithi03829856-49 2024-03-19T11:32:03.070 INFO:tasks.rados.rados.0.smithi038.stdout:1793: finishing write tid 4 to smithi03829856-49 2024-03-19T11:32:03.070 INFO:tasks.rados.rados.0.smithi038.stdout:1793: finishing write tid 5 to smithi03829856-49 2024-03-19T11:32:03.070 INFO:tasks.rados.rados.0.smithi038.stdout:1793: oid 49 updating version 0 to 77 2024-03-19T11:32:03.070 INFO:tasks.rados.rados.0.smithi038.stdout:1793: oid 49 version 77 is already newer than 76 2024-03-19T11:32:03.070 INFO:tasks.rados.rados.0.smithi038.stdout:1793: oid 49 version 77 is already newer than 74 2024-03-19T11:32:03.070 INFO:tasks.rados.rados.0.smithi038.stdout:1793: oid 49 version 77 is already newer than 75 2024-03-19T11:32:03.070 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 49 v 77 (ObjNum 334 snap 40 seq_num 334) dirty exists 2024-03-19T11:32:03.070 INFO:tasks.rados.rados.0.smithi038.stdout:1793: left oid 49 (ObjNum 334 snap 40 seq_num 334) 2024-03-19T11:32:03.070 INFO:tasks.rados.rados.0.smithi038.stdout:1794: finishing write tid 1 to smithi03829856-32 2024-03-19T11:32:03.070 INFO:tasks.rados.rados.0.smithi038.stdout:1794: finishing write tid 2 to smithi03829856-32 2024-03-19T11:32:03.070 INFO:tasks.rados.rados.0.smithi038.stdout:1794: finishing write tid 3 to smithi03829856-32 2024-03-19T11:32:03.070 INFO:tasks.rados.rados.0.smithi038.stdout:1794: finishing write tid 4 to smithi03829856-32 2024-03-19T11:32:03.070 INFO:tasks.rados.rados.0.smithi038.stdout:1794: finishing write tid 5 to smithi03829856-32 2024-03-19T11:32:03.071 INFO:tasks.rados.rados.0.smithi038.stdout:1794: oid 32 updating version 0 to 178 2024-03-19T11:32:03.071 INFO:tasks.rados.rados.0.smithi038.stdout:1794: oid 32 version 178 is already newer than 175 2024-03-19T11:32:03.071 INFO:tasks.rados.rados.0.smithi038.stdout:1794: oid 32 version 178 is already newer than 177 2024-03-19T11:32:03.071 INFO:tasks.rados.rados.0.smithi038.stdout:1794: oid 32 version 178 is already newer than 176 2024-03-19T11:32:03.071 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 32 v 178 (ObjNum 335 snap 40 seq_num 335) dirty exists 2024-03-19T11:32:03.071 INFO:tasks.rados.rados.0.smithi038.stdout:1794: left oid 32 (ObjNum 335 snap 40 seq_num 335) 2024-03-19T11:32:03.071 INFO:tasks.rados.rados.0.smithi038.stdout:1795: finishing write tid 1 to smithi03829856-37 2024-03-19T11:32:03.071 INFO:tasks.rados.rados.0.smithi038.stdout:1795: finishing write tid 2 to smithi03829856-37 2024-03-19T11:32:03.071 INFO:tasks.rados.rados.0.smithi038.stdout:1795: finishing write tid 3 to smithi03829856-37 2024-03-19T11:32:03.071 INFO:tasks.rados.rados.0.smithi038.stdout:1795: finishing write tid 4 to smithi03829856-37 2024-03-19T11:32:03.071 INFO:tasks.rados.rados.0.smithi038.stdout:1795: finishing write tid 5 to smithi03829856-37 2024-03-19T11:32:03.071 INFO:tasks.rados.rados.0.smithi038.stdout:1795: oid 37 updating version 0 to 155 2024-03-19T11:32:03.071 INFO:tasks.rados.rados.0.smithi038.stdout:1795: oid 37 version 155 is already newer than 153 2024-03-19T11:32:03.071 INFO:tasks.rados.rados.0.smithi038.stdout:1795: oid 37 version 155 is already newer than 152 2024-03-19T11:32:03.071 INFO:tasks.rados.rados.0.smithi038.stdout:1795: oid 37 version 155 is already newer than 154 2024-03-19T11:32:03.072 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 37 v 155 (ObjNum 336 snap 40 seq_num 336) dirty exists 2024-03-19T11:32:03.072 INFO:tasks.rados.rados.0.smithi038.stdout:1795: left oid 37 (ObjNum 336 snap 40 seq_num 336) 2024-03-19T11:32:03.085 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:32:03.095+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:32:03.086 INFO:tasks.rados.rados.0.smithi038.stdout:1798: set_chunk oid 21 offset: 369664 length: 407040 target oid 250dc83186939742a493cff93a71040b177ae7822ba30fccd3c14c7c1fa65494 offset: 0 snap -1 2024-03-19T11:32:03.086 INFO:tasks.rados.rados.0.smithi038.stdout:1793: done (4 left) 2024-03-19T11:32:03.086 INFO:tasks.rados.rados.0.smithi038.stdout:1794: done (3 left) 2024-03-19T11:32:03.086 INFO:tasks.rados.rados.0.smithi038.stdout:1795: done (2 left) 2024-03-19T11:32:03.086 INFO:tasks.rados.rados.0.smithi038.stdout:1796: done (1 left) 2024-03-19T11:32:03.087 INFO:tasks.rados.rados.0.smithi038.stdout:1799: tier_promote oid 10 2024-03-19T11:32:03.087 INFO:tasks.rados.rados.0.smithi038.stdout:1800: set_chunk oid 43 target oid 2024-03-19T11:32:03.087 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:32:03.095+0000 7f70c6fdf640 -1 osd.0 141 handle_osd_map injecting map crc failure 2024-03-19T11:32:03.094 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:32:03.101+0000 7f923da1c640 -1 osd.7 141 handle_osd_map injecting map crc failure 2024-03-19T11:32:03.109 INFO:tasks.rados.rados.0.smithi038.stdout:1800: set_chunk oid 43 offset: 367104 length: 581120 target oid 931a6d14a094fed3838c3389c11bca366d48b4dcc0e01838a370f27bca697931 offset: 0 snap -1 2024-03-19T11:32:03.109 INFO:tasks.rados.rados.0.smithi038.stdout:1801: read oid 11 snap -1 2024-03-19T11:32:03.109 INFO:tasks.rados.rados.0.smithi038.stdout:1801: expect deleted 2024-03-19T11:32:03.109 INFO:tasks.rados.rados.0.smithi038.stdout:1802: read oid 32 snap -1 2024-03-19T11:32:03.110 INFO:tasks.rados.rados.0.smithi038.stdout:1802: expect (ObjNum 335 snap 40 seq_num 335) 2024-03-19T11:32:03.110 INFO:tasks.rados.rados.0.smithi038.stdout:1803: set_chunk oid 25 target oid 2024-03-19T11:32:03.114 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-03-19T11:32:03.120 INFO:tasks.rados.rados.0.smithi038.stdout:1803: set_chunk oid 25 offset: 778752 length: 343552 target oid 24f60ca521d10192f29380bfee98e6ae2a281639962b8631dca6cb9cac7694c1 offset: 0 snap -1 2024-03-19T11:32:03.121 INFO:tasks.rados.rados.0.smithi038.stdout:1804: read oid 9 snap -1 2024-03-19T11:32:03.121 INFO:tasks.rados.rados.0.smithi038.stdout:1804: expect (ObjNum 308 snap 39 seq_num 308) 2024-03-19T11:32:03.121 INFO:tasks.rados.rados.0.smithi038.stdout:1805: read oid 7 snap -1 2024-03-19T11:32:03.121 INFO:tasks.rados.rados.0.smithi038.stdout:1805: expect (ObjNum 304 snap 39 seq_num 304) 2024-03-19T11:32:03.121 INFO:tasks.rados.rados.0.smithi038.stdout:1806: write oid 16 current snap is 40 2024-03-19T11:32:03.121 INFO:tasks.rados.rados.0.smithi038.stdout:1798: finishing set_chunk to oid 21 2024-03-19T11:32:03.121 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 21 v 226 (ObjNum 244 snap 30 seq_num 244) dirty exists 2024-03-19T11:32:03.121 INFO:tasks.rados.rados.0.smithi038.stdout:1806: seq_num 337 ranges {463959=478445,1570667=731108,3038372=492433} 2024-03-19T11:32:03.136 INFO:tasks.rados.rados.0.smithi038.stdout:1806: writing smithi03829856-16 from 463959 to 942404 tid 1 2024-03-19T11:32:03.152 INFO:tasks.rados.rados.0.smithi038.stdout:1806: writing smithi03829856-16 from 1570667 to 2301775 tid 2 2024-03-19T11:32:03.158 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:03.158 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:03.164 INFO:tasks.rados.rados.0.smithi038.stdout:1806: writing smithi03829856-16 from 3038372 to 3530805 tid 3 2024-03-19T11:32:03.165 INFO:tasks.rados.rados.0.smithi038.stdout:1798: done (8 left) 2024-03-19T11:32:03.165 INFO:tasks.rados.rados.0.smithi038.stdout:1807: rollback oid 17 current snap is 40 2024-03-19T11:32:03.165 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 17 to 21 2024-03-19T11:32:03.165 INFO:tasks.rados.rados.0.smithi038.stdout:1799: got (0) Success 2024-03-19T11:32:03.165 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 10 v 201 (ObjNum 289 snap 0 seq_num 0) clean dne 2024-03-19T11:32:03.165 INFO:tasks.rados.rados.0.smithi038.stdout:1803: finishing set_chunk to oid 25 2024-03-19T11:32:03.165 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 25 set_chunk returned error code -95 offset: 778752 length: 343552 2024-03-19T11:32:03.165 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 25 v 0 (ObjNum 309 snap 39 seq_num 309) dirty exists 2024-03-19T11:32:03.165 INFO:tasks.rados.rados.0.smithi038.stdout:1802: expect (ObjNum 335 snap 40 seq_num 335) 2024-03-19T11:32:03.174 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-19T11:32:03.186 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:32:03.195+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:32:03.287 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:32:03.293+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:32:03.339 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:03.339 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:03.352 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-19T11:32:03.396 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:32:03.405+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:32:03.442 INFO:tasks.rados.rados.0.smithi038.stdout:1800: finishing set_chunk to oid 43 2024-03-19T11:32:03.442 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 43 v 183 (ObjNum 312 snap 40 seq_num 312) dirty exists 2024-03-19T11:32:03.442 INFO:tasks.rados.rados.0.smithi038.stdout:1806: finishing write tid 1 to smithi03829856-16 2024-03-19T11:32:03.442 INFO:tasks.rados.rados.0.smithi038.stdout:1804: expect (ObjNum 308 snap 39 seq_num 308) 2024-03-19T11:32:03.489 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:32:03.497+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:32:03.520 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:03.520 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:03.534 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-19T11:32:03.590 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:32:03.599+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:32:03.631 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-19T11:32:03.691 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:32:03.701+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:32:03.702 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:03.702 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:03.716 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-19T11:32:03.723 INFO:tasks.rados.rados.0.smithi038.stdout:1805: expect (ObjNum 304 snap 39 seq_num 304) 2024-03-19T11:32:03.781 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:32:03.781 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:32:03.792 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:32:03.803+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:32:03.797 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:32:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:32:03.814 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:32:03.814 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:32:03.831 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:32:03.831 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:32:03.850 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:32:03.851 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:32:03.871 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:32:03.872 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:32:03.883 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:03.883 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:03.891 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:32:03.891 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:32:03.893 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:32:03.903+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:32:03.898 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-19T11:32:03.911 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:32:03.912 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:32:03.956 INFO:tasks.rados.rados.0.smithi038.stdout:1806: finishing write tid 2 to smithi03829856-16 2024-03-19T11:32:03.956 INFO:tasks.rados.rados.0.smithi038.stdout:1807: finishing rollback tid 0 to smithi03829856-17 2024-03-19T11:32:03.957 INFO:tasks.rados.rados.0.smithi038.stdout:1806: finishing write tid 3 to smithi03829856-16 2024-03-19T11:32:03.957 INFO:tasks.rados.rados.0.smithi038.stdout:1806: finishing write tid 4 to smithi03829856-16 2024-03-19T11:32:03.957 INFO:tasks.rados.rados.0.smithi038.stdout:1806: finishing write tid 5 to smithi03829856-16 2024-03-19T11:32:03.957 INFO:tasks.rados.rados.0.smithi038.stdout:1806: oid 16 updating version 0 to 266 2024-03-19T11:32:03.957 INFO:tasks.rados.rados.0.smithi038.stdout:1806: oid 16 version 266 is already newer than 263 2024-03-19T11:32:03.957 INFO:tasks.rados.rados.0.smithi038.stdout:1806: oid 16 version 266 is already newer than 265 2024-03-19T11:32:03.957 INFO:tasks.rados.rados.0.smithi038.stdout:1806: oid 16 version 266 is already newer than 264 2024-03-19T11:32:03.957 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 16 v 266 (ObjNum 337 snap 40 seq_num 337) dirty exists 2024-03-19T11:32:03.957 INFO:tasks.rados.rados.0.smithi038.stdout:1806: left oid 16 (ObjNum 337 snap 40 seq_num 337) 2024-03-19T11:32:03.958 INFO:tasks.rados.rados.0.smithi038.stdout:1807: finishing rollback tid 1 to smithi03829856-17 2024-03-19T11:32:03.958 INFO:tasks.rados.rados.0.smithi038.stdout:1807: finishing rollback tid 2 to smithi03829856-17 2024-03-19T11:32:03.958 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 17 v 324 (ObjNum 183 snap 20 seq_num 183) dirty exists 2024-03-19T11:32:03.958 INFO:tasks.rados.rados.0.smithi038.stdout:1799: done (8 left) 2024-03-19T11:32:03.958 INFO:tasks.rados.rados.0.smithi038.stdout:1800: done (7 left) 2024-03-19T11:32:03.958 INFO:tasks.rados.rados.0.smithi038.stdout:1801: done (6 left) 2024-03-19T11:32:03.958 INFO:tasks.rados.rados.0.smithi038.stdout:1802: done (5 left) 2024-03-19T11:32:03.958 INFO:tasks.rados.rados.0.smithi038.stdout:1803: done (4 left) 2024-03-19T11:32:03.958 INFO:tasks.rados.rados.0.smithi038.stdout:1804: done (3 left) 2024-03-19T11:32:03.958 INFO:tasks.rados.rados.0.smithi038.stdout:1805: done (2 left) 2024-03-19T11:32:03.958 INFO:tasks.rados.rados.0.smithi038.stdout:1806: done (1 left) 2024-03-19T11:32:03.958 INFO:tasks.rados.rados.0.smithi038.stdout:1807: done (0 left) 2024-03-19T11:32:03.958 INFO:tasks.rados.rados.0.smithi038.stdout:1808: write oid 26 current snap is 40 2024-03-19T11:32:03.959 INFO:tasks.rados.rados.0.smithi038.stdout:1808: seq_num 338 ranges {480766=553694,1629149=591119,2782148=420058} 2024-03-19T11:32:03.963 INFO:tasks.rados.rados.0.smithi038.stdout:1808: writing smithi03829856-26 from 480766 to 1034460 tid 1 2024-03-19T11:32:03.973 INFO:tasks.rados.rados.0.smithi038.stdout:1808: writing smithi03829856-26 from 1629149 to 2220268 tid 2 2024-03-19T11:32:03.981 INFO:tasks.rados.rados.0.smithi038.stdout:1808: writing smithi03829856-26 from 2782148 to 3202206 tid 3 2024-03-19T11:32:03.981 INFO:tasks.rados.rados.0.smithi038.stdout:1809: read oid 8 snap 21 2024-03-19T11:32:03.981 INFO:tasks.rados.rados.0.smithi038.stdout:1809: expect (ObjNum 60 snap 3 seq_num 60) 2024-03-19T11:32:03.982 INFO:tasks.rados.rados.0.smithi038.stdout:1810: tier_evict oid 32 2024-03-19T11:32:03.982 INFO:tasks.rados.rados.0.smithi038.stdout:1810: tier_evict oid 32 snap -1 2024-03-19T11:32:03.982 INFO:tasks.rados.rados.0.smithi038.stdout:1808: finishing write tid 1 to smithi03829856-26 2024-03-19T11:32:03.982 INFO:tasks.rados.rados.0.smithi038.stdout:1808: finishing write tid 2 to smithi03829856-26 2024-03-19T11:32:03.982 INFO:tasks.rados.rados.0.smithi038.stdout:1811: set_chunk oid 1 target oid 2024-03-19T11:32:03.985 INFO:tasks.rados.rados.0.smithi038.stdout:1811: set_chunk oid 1 offset: 107008 length: 27136 target oid aebd0cf1b8db8538a15698326819d852d1d41199a7471dfc22cb1543179627bf offset: 0 snap -1 2024-03-19T11:32:03.985 INFO:tasks.rados.rados.0.smithi038.stdout:1812: set_chunk oid 17 target oid 2024-03-19T11:32:03.995 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:32:04.001+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:32:04.004 INFO:tasks.rados.rados.0.smithi038.stdout:1812: set_chunk oid 17 offset: 2255360 length: 404480 target oid d74b06c5ebf876545fbe8e193958a836a3baddca44c791e0c79f2fa9a075c35e offset: 0 snap -1 2024-03-19T11:32:04.004 INFO:tasks.rados.rados.0.smithi038.stdout:1813: write (excl) oid 21 current snap is 40 2024-03-19T11:32:04.004 INFO:tasks.rados.rados.0.smithi038.stdout:1810: got (22) Invalid argument 2024-03-19T11:32:04.004 INFO:tasks.rados.rados.0.smithi038.stdout:1813: seq_num 339 ranges {497573=628943,1687631=734777,3093218=604355} 2024-03-19T11:32:04.016 INFO:tasks.rados.rados.0.smithi038.stdout:1813: writing smithi03829856-21 from 497573 to 1126516 tid 1 2024-03-19T11:32:04.026 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:32:04.026 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:32:04.026 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:32:04.026 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:32:04.026 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.027 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:891 4.5 4:af9d0d56:::smithi03829856-20:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:32:04.027 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:09.286410+0000", 2024-03-19T11:32:04.027 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 114.74890030100001, 2024-03-19T11:32:04.027 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035191405000000002, 2024-03-19T11:32:04.027 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:04.027 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:32:04.027 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:04.027 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:32:04.027 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:04.028 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 891 2024-03-19T11:32:04.028 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.028 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:04.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.028 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:04.028 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286410+0000", 2024-03-19T11:32:04.028 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.028 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.028 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:04.028 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286410+0000", 2024-03-19T11:32:04.028 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.028 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.028 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.029 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:04.029 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286413+0000", 2024-03-19T11:32:04.029 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2660000000000002e-06 2024-03-19T11:32:04.029 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.029 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.029 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:04.029 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286426+0000", 2024-03-19T11:32:04.029 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.258e-05 2024-03-19T11:32:04.029 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.029 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.029 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:04.029 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286460+0000", 2024-03-19T11:32:04.029 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3943999999999999e-05 2024-03-19T11:32:04.029 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.030 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.030 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:04.030 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286477+0000", 2024-03-19T11:32:04.030 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6799999999999998e-05 2024-03-19T11:32:04.030 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.030 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.030 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:04.030 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286605+0000", 2024-03-19T11:32:04.030 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000128892 2024-03-19T11:32:04.030 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.030 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.030 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:04.030 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286769+0000", 2024-03-19T11:32:04.031 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016359799999999999 2024-03-19T11:32:04.031 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.031 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.031 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:04.031 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.321601+0000", 2024-03-19T11:32:04.031 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.034832324999999997 2024-03-19T11:32:04.031 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.031 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:04.031 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.031 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.031 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.031 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1015 4.5 4:af9d0d56:::smithi03829856-20:head [tier-promote] snapc 6=[6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:32:04.031 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:12.761855+0000", 2024-03-19T11:32:04.031 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 111.273454775, 2024-03-19T11:32:04.031 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026760430000000002, 2024-03-19T11:32:04.032 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:04.032 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:32:04.032 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:04.032 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:32:04.032 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:04.032 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1015 2024-03-19T11:32:04.032 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.032 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:04.033 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.033 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:04.033 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761855+0000", 2024-03-19T11:32:04.033 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.033 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.033 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.033 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:04.034 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761855+0000", 2024-03-19T11:32:04.034 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.034 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.034 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.034 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:04.034 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761858+0000", 2024-03-19T11:32:04.034 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8140000000000001e-06 2024-03-19T11:32:04.034 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.034 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.034 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:04.034 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761872+0000", 2024-03-19T11:32:04.034 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3813e-05 2024-03-19T11:32:04.034 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.035 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.035 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:04.035 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761905+0000", 2024-03-19T11:32:04.035 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2642999999999999e-05 2024-03-19T11:32:04.035 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.035 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.035 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:04.035 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761925+0000", 2024-03-19T11:32:04.035 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0489e-05 2024-03-19T11:32:04.035 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.035 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.035 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:04.035 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.762070+0000", 2024-03-19T11:32:04.035 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000145153 2024-03-19T11:32:04.035 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.036 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:04.036 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.762365+0000", 2024-03-19T11:32:04.036 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029482100000000002 2024-03-19T11:32:04.036 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.036 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:04.036 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.788616+0000", 2024-03-19T11:32:04.036 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026250697 2024-03-19T11:32:04.036 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.036 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:04.036 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.036 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.036 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.037 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1121 4.5 4:aa3303a5:::smithi03829856-17:head [tier-promote] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:32:04.037 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:14.164281+0000", 2024-03-19T11:32:04.037 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 109.87102953, 2024-03-19T11:32:04.037 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023896435000000001, 2024-03-19T11:32:04.037 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:04.037 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:32:04.037 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:04.037 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:32:04.037 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:04.037 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1121 2024-03-19T11:32:04.038 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.038 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:04.038 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.038 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:04.038 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164281+0000", 2024-03-19T11:32:04.038 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.038 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.038 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.038 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:04.038 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164281+0000", 2024-03-19T11:32:04.038 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.038 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.038 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.039 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:04.039 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164283+0000", 2024-03-19T11:32:04.039 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8500000000000001e-06 2024-03-19T11:32:04.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.039 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:04.039 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164291+0000", 2024-03-19T11:32:04.039 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.14e-06 2024-03-19T11:32:04.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.039 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.039 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:04.039 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164313+0000", 2024-03-19T11:32:04.039 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.207e-05 2024-03-19T11:32:04.039 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.040 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.040 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:04.040 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164328+0000", 2024-03-19T11:32:04.040 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5403000000000001e-05 2024-03-19T11:32:04.040 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.040 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.040 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:04.040 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.167981+0000", 2024-03-19T11:32:04.040 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036529879999999998 2024-03-19T11:32:04.040 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.040 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.040 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:04.040 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.168274+0000", 2024-03-19T11:32:04.040 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029313100000000002 2024-03-19T11:32:04.041 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.041 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.041 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:04.041 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.188177+0000", 2024-03-19T11:32:04.041 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019902853000000002 2024-03-19T11:32:04.041 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.041 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:04.041 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.041 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.041 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.041 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1302 4.15 4:aa3303a5:::smithi03829856-17:head [write 413472~620140,stat] snapc b=[b,a,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-03-19T11:32:04.041 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:22.966426+0000", 2024-03-19T11:32:04.041 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 101.068883797, 2024-03-19T11:32:04.041 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018043513000000001, 2024-03-19T11:32:04.042 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:04.042 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:04.042 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:04.042 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:32:04.042 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:04.042 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1302 2024-03-19T11:32:04.042 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.042 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:04.042 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.042 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:04.042 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.966426+0000", 2024-03-19T11:32:04.042 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.043 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:04.043 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.966426+0000", 2024-03-19T11:32:04.043 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.043 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.043 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.043 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:04.043 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.966429+0000", 2024-03-19T11:32:04.044 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3829999999999999e-06 2024-03-19T11:32:04.044 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.044 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.044 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:04.044 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972117+0000", 2024-03-19T11:32:04.044 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0056885750000000004 2024-03-19T11:32:04.044 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.044 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.044 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:04.045 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972156+0000", 2024-03-19T11:32:04.045 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8878000000000003e-05 2024-03-19T11:32:04.045 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.045 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.045 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:04.045 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972184+0000", 2024-03-19T11:32:04.045 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7509999999999999e-05 2024-03-19T11:32:04.045 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.045 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.045 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:04.045 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972309+0000", 2024-03-19T11:32:04.045 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012525899999999999 2024-03-19T11:32:04.045 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.046 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.046 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:04.046 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972566+0000", 2024-03-19T11:32:04.046 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00025740799999999999 2024-03-19T11:32:04.046 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.046 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.046 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:32:04.046 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.972899+0000", 2024-03-19T11:32:04.046 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033238799999999998 2024-03-19T11:32:04.046 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.046 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.046 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:04.046 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984088+0000", 2024-03-19T11:32:04.047 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011189466 2024-03-19T11:32:04.047 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.047 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.047 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:32:04.047 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984138+0000", 2024-03-19T11:32:04.047 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.9666999999999997e-05 2024-03-19T11:32:04.047 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.047 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.047 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:04.047 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984329+0000", 2024-03-19T11:32:04.047 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00019086799999999999 2024-03-19T11:32:04.047 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.047 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.047 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:04.047 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:22.984470+0000", 2024-03-19T11:32:04.048 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000141111 2024-03-19T11:32:04.048 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.048 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:04.048 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.048 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.048 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.048 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.1 50/49 [PushOp(4:866f3dcc:::smithi03829856-31:7, version: 38'44, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@38'44, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 45'62, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@45'62, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 46'68, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@46'68, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:7, version: 47'71, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@47'71, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: b=[]:{7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 47'72, data_included: [440350~816629,1309629~231947], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@47'72, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:32:04.048 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:27.959457+0000", 2024-03-19T11:32:04.048 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 96.075853379999998, 2024-03-19T11:32:04.049 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018416666000000002, 2024-03-19T11:32:04.049 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:04.049 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:32:04.049 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:04.049 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.049 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:04.049 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.959457+0000", 2024-03-19T11:32:04.049 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.049 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.049 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.049 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:04.049 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.959457+0000", 2024-03-19T11:32:04.049 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.050 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.050 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.050 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:04.050 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.959458+0000", 2024-03-19T11:32:04.050 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4899999999999999e-06 2024-03-19T11:32:04.050 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.050 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.050 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:04.050 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.962822+0000", 2024-03-19T11:32:04.050 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0033640620000000001 2024-03-19T11:32:04.050 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.050 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.050 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:04.051 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.962893+0000", 2024-03-19T11:32:04.051 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1060000000000001e-05 2024-03-19T11:32:04.051 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.051 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.051 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:04.051 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.962977+0000", 2024-03-19T11:32:04.051 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3311000000000002e-05 2024-03-19T11:32:04.051 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.051 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.051 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:04.051 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963208+0000", 2024-03-19T11:32:04.052 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023121200000000001 2024-03-19T11:32:04.052 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.052 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.052 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:04.052 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.963245+0000", 2024-03-19T11:32:04.052 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6844999999999997e-05 2024-03-19T11:32:04.052 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.052 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.052 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:04.052 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:27.977874+0000", 2024-03-19T11:32:04.052 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014628686 2024-03-19T11:32:04.052 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.052 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:04.052 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.053 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.053 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2284 4.15 4:afa9f208:::smithi03829856-35:head [read 1314816~702976] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-03-19T11:32:04.053 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:53.596775+0000", 2024-03-19T11:32:04.053 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 70.438535294000005, 2024-03-19T11:32:04.053 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.47260528299999999, 2024-03-19T11:32:04.053 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:04.053 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:32:04.053 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:04.053 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:32:04.053 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:04.053 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2284 2024-03-19T11:32:04.053 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.053 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:04.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.054 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:04.054 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596775+0000", 2024-03-19T11:32:04.054 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.054 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.054 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.054 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:04.055 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596775+0000", 2024-03-19T11:32:04.055 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.055 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.055 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.055 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:04.055 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596778+0000", 2024-03-19T11:32:04.055 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.174e-06 2024-03-19T11:32:04.055 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.055 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.055 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:04.055 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596794+0000", 2024-03-19T11:32:04.056 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5372000000000001e-05 2024-03-19T11:32:04.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.056 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:04.056 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596828+0000", 2024-03-19T11:32:04.056 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4107999999999999e-05 2024-03-19T11:32:04.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.056 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:04.056 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596844+0000", 2024-03-19T11:32:04.056 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6296000000000001e-05 2024-03-19T11:32:04.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.056 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.056 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:04.057 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596968+0000", 2024-03-19T11:32:04.057 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012383699999999999 2024-03-19T11:32:04.057 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.057 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:32:04.057 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596988+0000", 2024-03-19T11:32:04.057 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0645e-05 2024-03-19T11:32:04.057 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.057 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.057 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:04.057 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.068916+0000", 2024-03-19T11:32:04.058 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.47192744800000003 2024-03-19T11:32:04.058 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.058 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.058 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:04.058 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.069015+0000", 2024-03-19T11:32:04.058 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9500999999999995e-05 2024-03-19T11:32:04.058 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.058 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.058 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:04.058 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.069380+0000", 2024-03-19T11:32:04.059 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00036490200000000001 2024-03-19T11:32:04.059 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.059 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:04.059 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.059 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.059 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.059 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2286 4.15 4:afa9f208:::smithi03829856-35:head [assert-version v108,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e75)", 2024-03-19T11:32:04.059 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.084826+0000", 2024-03-19T11:32:04.059 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 69.950483926999993, 2024-03-19T11:32:04.059 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.055799556, 2024-03-19T11:32:04.059 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:04.060 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:04.060 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:04.060 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:32:04.060 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:04.060 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2286 2024-03-19T11:32:04.060 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.060 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:04.060 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.060 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:04.060 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084826+0000", 2024-03-19T11:32:04.060 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.060 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.060 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.060 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:04.061 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084826+0000", 2024-03-19T11:32:04.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.061 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:04.061 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084828+0000", 2024-03-19T11:32:04.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1409999999999999e-06 2024-03-19T11:32:04.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.061 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:04.061 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084838+0000", 2024-03-19T11:32:04.061 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3740000000000004e-06 2024-03-19T11:32:04.061 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.061 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.061 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:04.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084859+0000", 2024-03-19T11:32:04.062 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1644000000000001e-05 2024-03-19T11:32:04.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:04.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084871+0000", 2024-03-19T11:32:04.062 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1127e-05 2024-03-19T11:32:04.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.062 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:04.062 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084961+0000", 2024-03-19T11:32:04.062 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0371999999999995e-05 2024-03-19T11:32:04.062 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.062 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.063 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:04.063 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.085130+0000", 2024-03-19T11:32:04.063 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016884800000000001 2024-03-19T11:32:04.063 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.063 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.063 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5,7", 2024-03-19T11:32:04.063 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.089309+0000", 2024-03-19T11:32:04.063 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041788650000000004 2024-03-19T11:32:04.063 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.063 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.063 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:32:04.063 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.090362+0000", 2024-03-19T11:32:04.063 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001053362 2024-03-19T11:32:04.063 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.063 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.064 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:04.064 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.090511+0000", 2024-03-19T11:32:04.064 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014887200000000001 2024-03-19T11:32:04.064 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.064 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.064 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:04.064 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.140479+0000", 2024-03-19T11:32:04.064 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.049968006000000002 2024-03-19T11:32:04.064 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.064 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.064 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:04.064 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.140534+0000", 2024-03-19T11:32:04.065 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4917000000000003e-05 2024-03-19T11:32:04.065 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.065 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.065 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:04.065 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.140626+0000", 2024-03-19T11:32:04.065 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2027999999999994e-05 2024-03-19T11:32:04.065 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.066 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:04.066 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.066 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.066 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.066 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1dabc0f1:::smithi03829856-1:head, version: 70'103, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:head@70'103, size: 2621892, copy_subset: [(0, 2621892)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:32:04.066 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.280012+0000", 2024-03-19T11:32:04.066 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 69.755298350999993, 2024-03-19T11:32:04.066 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018569209, 2024-03-19T11:32:04.066 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:04.066 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:32:04.066 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:04.066 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.066 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:04.066 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280012+0000", 2024-03-19T11:32:04.067 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.067 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.067 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.067 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:04.067 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280012+0000", 2024-03-19T11:32:04.067 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.067 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.067 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.067 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:04.067 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280013+0000", 2024-03-19T11:32:04.067 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9899999999999999e-07 2024-03-19T11:32:04.067 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.067 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.067 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:04.068 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280584+0000", 2024-03-19T11:32:04.068 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00057090299999999999 2024-03-19T11:32:04.068 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.068 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.068 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:04.068 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280612+0000", 2024-03-19T11:32:04.068 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8442999999999999e-05 2024-03-19T11:32:04.068 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.068 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.068 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:04.069 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280631+0000", 2024-03-19T11:32:04.069 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8896999999999999e-05 2024-03-19T11:32:04.069 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.069 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.069 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:04.069 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.295041+0000", 2024-03-19T11:32:04.069 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014410127999999999 2024-03-19T11:32:04.069 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.069 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.069 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:04.069 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.295050+0000", 2024-03-19T11:32:04.070 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2350000000000005e-06 2024-03-19T11:32:04.070 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.070 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.070 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:04.070 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.298581+0000", 2024-03-19T11:32:04.070 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0035308039999999998 2024-03-19T11:32:04.070 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.070 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:04.070 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.070 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.070 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.070 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.18 e75,74 4:1d58db11:::smithi03829856-18:c,70'104)", 2024-03-19T11:32:04.070 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.280680+0000", 2024-03-19T11:32:04.071 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 69.754629875999996, 2024-03-19T11:32:04.071 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018106229000000001, 2024-03-19T11:32:04.071 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:04.071 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:32:04.071 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:04.071 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.071 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:04.071 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280680+0000", 2024-03-19T11:32:04.071 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.071 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.071 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.071 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:04.071 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280680+0000", 2024-03-19T11:32:04.071 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.071 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.072 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.072 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:04.072 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280681+0000", 2024-03-19T11:32:04.072 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2099999999999995e-07 2024-03-19T11:32:04.072 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.072 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.072 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:04.072 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280686+0000", 2024-03-19T11:32:04.072 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2240000000000003e-06 2024-03-19T11:32:04.072 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.072 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.072 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:04.072 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280698+0000", 2024-03-19T11:32:04.072 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1698999999999999e-05 2024-03-19T11:32:04.072 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:04.073 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.280705+0000", 2024-03-19T11:32:04.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4350000000000002e-06 2024-03-19T11:32:04.073 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:04.073 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.298643+0000", 2024-03-19T11:32:04.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017938352000000001 2024-03-19T11:32:04.073 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.073 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.073 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:04.073 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.298656+0000", 2024-03-19T11:32:04.073 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3084999999999999e-05 2024-03-19T11:32:04.073 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.074 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.074 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:04.074 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.298787+0000", 2024-03-19T11:32:04.074 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013061299999999999 2024-03-19T11:32:04.074 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.074 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:04.074 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.074 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.074 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.074 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2441 4.5 4:a38b9453:::smithi03829856-37:head [delete] snapc 17=[17,16,15,13,12,b,6] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e84)", 2024-03-19T11:32:04.074 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:03.060663+0000", 2024-03-19T11:32:04.074 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 60.974647660999999, 2024-03-19T11:32:04.074 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10917726, 2024-03-19T11:32:04.074 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:04.075 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:04.075 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:04.075 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:32:04.075 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:04.075 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2441 2024-03-19T11:32:04.075 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.075 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:04.075 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.075 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:04.076 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060663+0000", 2024-03-19T11:32:04.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.076 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:04.076 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060663+0000", 2024-03-19T11:32:04.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.076 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:04.076 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060666+0000", 2024-03-19T11:32:04.076 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.05e-06 2024-03-19T11:32:04.076 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.076 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.076 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:04.077 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060680+0000", 2024-03-19T11:32:04.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4276e-05 2024-03-19T11:32:04.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.077 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:04.077 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060714+0000", 2024-03-19T11:32:04.077 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4325000000000001e-05 2024-03-19T11:32:04.077 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.077 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.077 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:04.078 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060737+0000", 2024-03-19T11:32:04.078 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2959000000000002e-05 2024-03-19T11:32:04.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.078 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:04.078 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060895+0000", 2024-03-19T11:32:04.078 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000157787 2024-03-19T11:32:04.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.078 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:32:04.078 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060923+0000", 2024-03-19T11:32:04.078 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8019000000000001e-05 2024-03-19T11:32:04.078 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.078 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.079 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:04.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.066222+0000", 2024-03-19T11:32:04.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052989270000000002 2024-03-19T11:32:04.079 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.079 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.079 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-03-19T11:32:04.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.070153+0000", 2024-03-19T11:32:04.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039315119999999999 2024-03-19T11:32:04.079 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.079 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.079 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:04.079 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.147930+0000", 2024-03-19T11:32:04.079 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.077776459000000006 2024-03-19T11:32:04.079 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.080 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:04.080 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.148079+0000", 2024-03-19T11:32:04.080 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014888 2024-03-19T11:32:04.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.080 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.080 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:32:04.080 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.148370+0000", 2024-03-19T11:32:04.080 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029130999999999997 2024-03-19T11:32:04.080 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.081 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.081 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:32:04.081 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.152531+0000", 2024-03-19T11:32:04.081 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041605399999999999 2024-03-19T11:32:04.081 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.081 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.081 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:04.081 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.169762+0000", 2024-03-19T11:32:04.081 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017231483999999998 2024-03-19T11:32:04.081 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.081 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.081 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:04.081 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.169807+0000", 2024-03-19T11:32:04.082 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5161999999999999e-05 2024-03-19T11:32:04.082 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.082 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.082 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:04.082 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.169840+0000", 2024-03-19T11:32:04.082 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2570000000000002e-05 2024-03-19T11:32:04.082 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.082 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:04.082 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.082 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.083 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPushReply(4.5 84/83 [PushReplyOp(4:a38b9453:::smithi03829856-37:head)])", 2024-03-19T11:32:04.083 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:03.084896+0000", 2024-03-19T11:32:04.083 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 60.950414527, 2024-03-19T11:32:04.083 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.02071104, 2024-03-19T11:32:04.083 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:04.083 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-19T11:32:04.083 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:04.083 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.083 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:04.084 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.084896+0000", 2024-03-19T11:32:04.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.084 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.084 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.084 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:04.084 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.084896+0000", 2024-03-19T11:32:04.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.084 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.084 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.084 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:04.084 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.084897+0000", 2024-03-19T11:32:04.084 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.686e-06 2024-03-19T11:32:04.085 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.085 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:04.085 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.084910+0000", 2024-03-19T11:32:04.085 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2948e-05 2024-03-19T11:32:04.085 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.085 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:04.085 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.084936+0000", 2024-03-19T11:32:04.085 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5982e-05 2024-03-19T11:32:04.085 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.085 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.085 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:04.085 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.084955+0000", 2024-03-19T11:32:04.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8416000000000001e-05 2024-03-19T11:32:04.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.086 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.086 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:04.086 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.102822+0000", 2024-03-19T11:32:04.086 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017866756000000001 2024-03-19T11:32:04.086 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.087 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:04.087 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.105607+0000", 2024-03-19T11:32:04.087 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027852520000000002 2024-03-19T11:32:04.087 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.087 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:04.087 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.087 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.087 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.087 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.4.0:540 4.18 e111/86 4:1d58db11:::smithi03829856-18:head v 111'153, mlcod=108'150)", 2024-03-19T11:32:04.087 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:31.238185+0000", 2024-03-19T11:32:04.087 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 32.797125164000001, 2024-03-19T11:32:04.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021779390999999999, 2024-03-19T11:32:04.088 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:04.088 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:04.088 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:04.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.088 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:04.088 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.238185+0000", 2024-03-19T11:32:04.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.088 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:04.088 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.238185+0000", 2024-03-19T11:32:04.088 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.088 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.088 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.089 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:04.089 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.238188+0000", 2024-03-19T11:32:04.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4739999999999999e-06 2024-03-19T11:32:04.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.089 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:04.089 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.240837+0000", 2024-03-19T11:32:04.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026495189999999999 2024-03-19T11:32:04.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.089 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.089 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:04.089 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.240867+0000", 2024-03-19T11:32:04.089 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0363000000000001e-05 2024-03-19T11:32:04.089 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:04.090 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.240885+0000", 2024-03-19T11:32:04.090 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7153e-05 2024-03-19T11:32:04.090 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:04.090 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.241029+0000", 2024-03-19T11:32:04.090 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014455499999999999 2024-03-19T11:32:04.090 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.090 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.090 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:04.090 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.241069+0000", 2024-03-19T11:32:04.090 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9409000000000002e-05 2024-03-19T11:32:04.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.091 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:04.091 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.259833+0000", 2024-03-19T11:32:04.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018764306000000001 2024-03-19T11:32:04.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.091 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.091 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:04.091 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.259964+0000", 2024-03-19T11:32:04.091 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000131612 2024-03-19T11:32:04.091 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.091 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:04.091 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.091 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.092 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3282 4.5 4:a0a50d04:::smithi03829856-13:head [read 0~2201684,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e112)", 2024-03-19T11:32:04.092 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:32.240600+0000", 2024-03-19T11:32:04.092 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 31.794709959999999, 2024-03-19T11:32:04.092 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.84623566699999997, 2024-03-19T11:32:04.092 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:04.092 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:32:04.092 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:04.092 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:32:04.092 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:04.092 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3282 2024-03-19T11:32:04.092 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.092 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:04.092 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.093 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:04.093 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240600+0000", 2024-03-19T11:32:04.093 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.093 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.093 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:04.093 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240600+0000", 2024-03-19T11:32:04.093 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.093 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.093 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:04.093 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240602+0000", 2024-03-19T11:32:04.093 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0700000000000001e-06 2024-03-19T11:32:04.093 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.093 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.094 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:04.094 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240611+0000", 2024-03-19T11:32:04.094 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.969e-06 2024-03-19T11:32:04.094 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.094 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:04.094 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240630+0000", 2024-03-19T11:32:04.094 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8253e-05 2024-03-19T11:32:04.094 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.094 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.094 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:04.095 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240642+0000", 2024-03-19T11:32:04.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2666e-05 2024-03-19T11:32:04.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.095 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:04.095 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.241488+0000", 2024-03-19T11:32:04.095 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000845252 2024-03-19T11:32:04.095 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.095 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.095 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:32:04.096 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.241503+0000", 2024-03-19T11:32:04.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5797000000000001e-05 2024-03-19T11:32:04.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.096 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:04.096 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.083186+0000", 2024-03-19T11:32:04.096 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.84168299599999996 2024-03-19T11:32:04.096 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.096 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.096 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:04.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.083618+0000", 2024-03-19T11:32:04.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000432046 2024-03-19T11:32:04.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.097 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:04.097 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.086836+0000", 2024-03-19T11:32:04.097 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032176180000000002 2024-03-19T11:32:04.097 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.097 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:04.097 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.097 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.097 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.098 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3283 4.5 4:a0a50d04:::smithi03829856-13:head [sparse-read 0~1151947] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e112)", 2024-03-19T11:32:04.098 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:32.240669+0000", 2024-03-19T11:32:04.098 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 31.794640959999999, 2024-03-19T11:32:04.098 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.84791886900000002, 2024-03-19T11:32:04.098 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:04.098 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:32:04.099 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:04.099 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:32:04.099 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:04.099 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3283 2024-03-19T11:32:04.099 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.099 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:04.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.100 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:04.100 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240669+0000", 2024-03-19T11:32:04.100 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.100 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.100 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.100 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:04.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240669+0000", 2024-03-19T11:32:04.101 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.101 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.101 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.101 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:04.101 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240670+0000", 2024-03-19T11:32:04.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8199999999999998e-07 2024-03-19T11:32:04.102 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.102 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:04.102 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240675+0000", 2024-03-19T11:32:04.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0849999999999996e-06 2024-03-19T11:32:04.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.103 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:04.103 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240688+0000", 2024-03-19T11:32:04.103 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2901000000000001e-05 2024-03-19T11:32:04.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.104 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:04.104 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240696+0000", 2024-03-19T11:32:04.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2910000000000007e-06 2024-03-19T11:32:04.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.105 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:04.105 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.241573+0000", 2024-03-19T11:32:04.105 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00087639100000000002 2024-03-19T11:32:04.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.105 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:32:04.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.241587+0000", 2024-03-19T11:32:04.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4467999999999999e-05 2024-03-19T11:32:04.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:04.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.087901+0000", 2024-03-19T11:32:04.107 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.84631403500000002 2024-03-19T11:32:04.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.107 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:04.107 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.088089+0000", 2024-03-19T11:32:04.107 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000187171 2024-03-19T11:32:04.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.108 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:04.108 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.088588+0000", 2024-03-19T11:32:04.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00049964499999999995 2024-03-19T11:32:04.108 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.108 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:04.109 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.109 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.109 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.109 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3284 4.5 4:a0a50d04:::smithi03829856-13:head [sparse-read 0~1151947] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e112)", 2024-03-19T11:32:04.109 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:32.240710+0000", 2024-03-19T11:32:04.109 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 31.794600664000001, 2024-03-19T11:32:04.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.84860797899999996, 2024-03-19T11:32:04.110 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:04.110 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:32:04.110 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:04.110 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:32:04.110 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:04.111 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3284 2024-03-19T11:32:04.111 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.111 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:04.111 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.111 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:04.111 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240710+0000", 2024-03-19T11:32:04.111 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.112 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.112 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.112 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:04.112 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240710+0000", 2024-03-19T11:32:04.112 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.112 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.112 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.113 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:04.113 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240710+0000", 2024-03-19T11:32:04.113 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9599999999999999e-07 2024-03-19T11:32:04.113 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.113 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.113 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:04.113 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240714+0000", 2024-03-19T11:32:04.113 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8480000000000003e-06 2024-03-19T11:32:04.114 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.114 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.114 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:04.114 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240726+0000", 2024-03-19T11:32:04.114 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.186e-05 2024-03-19T11:32:04.114 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.114 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.115 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:04.115 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240736+0000", 2024-03-19T11:32:04.115 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0040999999999999e-05 2024-03-19T11:32:04.116 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.116 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.116 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:04.116 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.241633+0000", 2024-03-19T11:32:04.116 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00089736299999999998 2024-03-19T11:32:04.116 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.116 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.116 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:32:04.116 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.241645+0000", 2024-03-19T11:32:04.116 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1600000000000001e-05 2024-03-19T11:32:04.116 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.116 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.116 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:04.116 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.088667+0000", 2024-03-19T11:32:04.116 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.84702242800000005 2024-03-19T11:32:04.117 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.117 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.117 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:04.117 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.088837+0000", 2024-03-19T11:32:04.117 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000169949 2024-03-19T11:32:04.117 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.117 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.117 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:04.117 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.089318+0000", 2024-03-19T11:32:04.117 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00048019399999999998 2024-03-19T11:32:04.117 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.117 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:04.117 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.118 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.118 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.118 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:4125 4.5 4:a38b9453:::smithi03829856-37:head [write 746310~635349,stat] snapc 29=[27,25,24,22,21,1e,17] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-03-19T11:32:04.118 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:32:01.099163+0000", 2024-03-19T11:32:04.118 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 2.9361470039999999, 2024-03-19T11:32:04.118 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030764071, 2024-03-19T11:32:04.118 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:04.118 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:04.118 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:04.118 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:32:04.118 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:04.118 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 4125 2024-03-19T11:32:04.118 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.119 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:04.119 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.119 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:04.119 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.099163+0000", 2024-03-19T11:32:04.119 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.119 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.119 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.119 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:04.119 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.099163+0000", 2024-03-19T11:32:04.119 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.119 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.119 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.119 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:04.119 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.099166+0000", 2024-03-19T11:32:04.119 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9629999999999998e-06 2024-03-19T11:32:04.120 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.120 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.120 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:04.120 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.100575+0000", 2024-03-19T11:32:04.120 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001408982 2024-03-19T11:32:04.120 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.120 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.120 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:04.120 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.100610+0000", 2024-03-19T11:32:04.121 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4609e-05 2024-03-19T11:32:04.121 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.121 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:04.121 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.100625+0000", 2024-03-19T11:32:04.121 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4826e-05 2024-03-19T11:32:04.121 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.121 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:04.121 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.100759+0000", 2024-03-19T11:32:04.122 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013436500000000001 2024-03-19T11:32:04.122 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.122 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.122 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:04.122 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.100947+0000", 2024-03-19T11:32:04.122 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018814299999999999 2024-03-19T11:32:04.122 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.122 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.122 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:32:04.122 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.101457+0000", 2024-03-19T11:32:04.122 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00051012699999999998 2024-03-19T11:32:04.123 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.124 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.124 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:04.124 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.125933+0000", 2024-03-19T11:32:04.124 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.024475851 2024-03-19T11:32:04.124 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.124 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.124 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:32:04.124 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.129604+0000", 2024-03-19T11:32:04.124 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036707139999999998 2024-03-19T11:32:04.125 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.125 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.125 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:04.125 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.129774+0000", 2024-03-19T11:32:04.125 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017009100000000001 2024-03-19T11:32:04.125 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.126 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.126 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:04.126 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.129927+0000", 2024-03-19T11:32:04.126 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015339999999999999 2024-03-19T11:32:04.126 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.126 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:04.126 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.126 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.126 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.127 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:4126 4.5 4:a38b9453:::smithi03829856-37:head [write 2077646~576443] snapc 29=[27,25,24,22,21,1e,17] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-03-19T11:32:04.127 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:32:01.106778+0000", 2024-03-19T11:32:04.127 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 2.9285323989999998, 2024-03-19T11:32:04.127 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.024531504999999999, 2024-03-19T11:32:04.127 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:04.127 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:04.127 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:04.127 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:32:04.127 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:04.127 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 4126 2024-03-19T11:32:04.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.127 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:04.128 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.128 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:04.128 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.106778+0000", 2024-03-19T11:32:04.128 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.128 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.128 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.129 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:04.129 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.106778+0000", 2024-03-19T11:32:04.129 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.129 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.129 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:04.129 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.106781+0000", 2024-03-19T11:32:04.129 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.456e-06 2024-03-19T11:32:04.129 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.129 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:04.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.107491+0000", 2024-03-19T11:32:04.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00070938200000000005 2024-03-19T11:32:04.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.130 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:04.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.107529+0000", 2024-03-19T11:32:04.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8039e-05 2024-03-19T11:32:04.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.130 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:04.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.107551+0000", 2024-03-19T11:32:04.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2753000000000001e-05 2024-03-19T11:32:04.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.131 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:04.131 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.114973+0000", 2024-03-19T11:32:04.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0074218569999999996 2024-03-19T11:32:04.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.131 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:04.131 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.115239+0000", 2024-03-19T11:32:04.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026613800000000002 2024-03-19T11:32:04.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.132 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:32:04.132 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.115718+0000", 2024-03-19T11:32:04.132 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000478708 2024-03-19T11:32:04.132 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.132 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:32:04.133 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.129957+0000", 2024-03-19T11:32:04.133 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014239069 2024-03-19T11:32:04.133 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.134 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.134 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:04.134 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.131069+0000", 2024-03-19T11:32:04.134 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001111534 2024-03-19T11:32:04.134 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.135 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.135 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:04.135 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.131203+0000", 2024-03-19T11:32:04.135 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013444100000000001 2024-03-19T11:32:04.135 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.135 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.135 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:04.135 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.131309+0000", 2024-03-19T11:32:04.135 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000106128 2024-03-19T11:32:04.135 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.136 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:04.136 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.136 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.136 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.136 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:4127 4.5 4:a38b9453:::smithi03829856-37:head [write 3362764~183546] snapc 29=[27,25,24,22,21,1e,17] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-03-19T11:32:04.136 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:32:01.112007+0000", 2024-03-19T11:32:04.136 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 2.9233035539999999, 2024-03-19T11:32:04.136 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019799431999999999, 2024-03-19T11:32:04.136 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:04.137 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:04.137 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:04.137 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:32:04.137 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:04.137 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 4127 2024-03-19T11:32:04.137 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.137 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:04.137 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.137 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:04.137 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.112007+0000", 2024-03-19T11:32:04.137 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.137 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.137 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.137 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:04.137 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.112007+0000", 2024-03-19T11:32:04.138 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.138 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:04.138 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.112010+0000", 2024-03-19T11:32:04.138 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0400000000000001e-06 2024-03-19T11:32:04.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.138 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:04.138 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.112358+0000", 2024-03-19T11:32:04.138 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000347762 2024-03-19T11:32:04.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.139 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.139 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:04.139 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.112389+0000", 2024-03-19T11:32:04.139 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.1347999999999999e-05 2024-03-19T11:32:04.139 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.139 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.139 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:04.139 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.112413+0000", 2024-03-19T11:32:04.139 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4145999999999999e-05 2024-03-19T11:32:04.139 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.139 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.140 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:04.140 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.122440+0000", 2024-03-19T11:32:04.140 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010026782 2024-03-19T11:32:04.140 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.140 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:04.140 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.122650+0000", 2024-03-19T11:32:04.140 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000209922 2024-03-19T11:32:04.140 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.140 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:32:04.140 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.123104+0000", 2024-03-19T11:32:04.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000454363 2024-03-19T11:32:04.141 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.141 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.141 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:32:04.141 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.129980+0000", 2024-03-19T11:32:04.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0068763410000000002 2024-03-19T11:32:04.141 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.141 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.141 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:04.141 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.131498+0000", 2024-03-19T11:32:04.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001517434 2024-03-19T11:32:04.141 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.141 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.142 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:04.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.131670+0000", 2024-03-19T11:32:04.142 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017248799999999999 2024-03-19T11:32:04.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.142 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:04.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.131806+0000", 2024-03-19T11:32:04.142 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013580600000000001 2024-03-19T11:32:04.142 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.142 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:04.142 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.143 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.143 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:4128 4.5 4:a38b9453:::smithi03829856-37:head [setxattr _header (57) in=7b,truncate 3546310] snapc 29=[27,25,24,22,21,1e,17] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-03-19T11:32:04.143 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:32:01.112446+0000", 2024-03-19T11:32:04.143 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 2.922864122, 2024-03-19T11:32:04.143 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019934741999999998, 2024-03-19T11:32:04.143 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:04.143 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:04.143 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:04.143 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:32:04.144 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:04.144 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 4128 2024-03-19T11:32:04.144 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.144 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:04.144 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.144 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:04.144 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.112446+0000", 2024-03-19T11:32:04.144 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.144 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.144 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.144 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:04.144 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.112446+0000", 2024-03-19T11:32:04.144 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.144 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.144 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.145 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:04.145 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.112448+0000", 2024-03-19T11:32:04.145 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5829999999999999e-06 2024-03-19T11:32:04.145 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.145 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.145 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:04.145 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.112457+0000", 2024-03-19T11:32:04.145 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6369999999999994e-06 2024-03-19T11:32:04.145 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.145 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.145 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:04.145 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.112480+0000", 2024-03-19T11:32:04.145 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3014e-05 2024-03-19T11:32:04.146 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.146 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.146 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:04.146 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.112495+0000", 2024-03-19T11:32:04.146 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4596999999999999e-05 2024-03-19T11:32:04.146 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.146 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.146 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:04.147 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.127994+0000", 2024-03-19T11:32:04.147 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015499092000000001 2024-03-19T11:32:04.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.147 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:04.147 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.128282+0000", 2024-03-19T11:32:04.147 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028747199999999999 2024-03-19T11:32:04.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.147 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:32:04.147 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.128807+0000", 2024-03-19T11:32:04.147 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00052564099999999998 2024-03-19T11:32:04.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.147 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:32:04.148 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.130396+0000", 2024-03-19T11:32:04.148 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001588922 2024-03-19T11:32:04.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.148 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.148 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:04.148 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.132196+0000", 2024-03-19T11:32:04.148 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00179981 2024-03-19T11:32:04.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.148 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.148 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:04.148 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.132283+0000", 2024-03-19T11:32:04.148 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6966999999999996e-05 2024-03-19T11:32:04.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.148 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.148 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:04.149 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.132381+0000", 2024-03-19T11:32:04.149 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8006999999999999e-05 2024-03-19T11:32:04.149 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.149 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:04.149 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.149 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.149 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.149 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:4129 4.5 4:a38b9453:::smithi03829856-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e140)", 2024-03-19T11:32:04.149 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:32:01.112516+0000", 2024-03-19T11:32:04.149 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 2.9227942339999999, 2024-03-19T11:32:04.149 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020215219, 2024-03-19T11:32:04.149 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:04.149 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:32:04.149 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:04.149 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:32:04.150 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:04.150 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 4129 2024-03-19T11:32:04.150 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.150 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:04.150 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.150 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:04.150 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.112516+0000", 2024-03-19T11:32:04.150 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.150 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.150 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.151 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:04.151 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.112516+0000", 2024-03-19T11:32:04.151 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.151 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.151 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.151 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:04.151 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.112517+0000", 2024-03-19T11:32:04.151 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.342e-06 2024-03-19T11:32:04.151 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.152 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:04.152 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.112524+0000", 2024-03-19T11:32:04.153 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0230000000000004e-06 2024-03-19T11:32:04.153 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.153 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.153 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:04.153 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.112543+0000", 2024-03-19T11:32:04.153 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9026e-05 2024-03-19T11:32:04.153 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.153 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.153 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:04.153 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.112557+0000", 2024-03-19T11:32:04.155 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4011e-05 2024-03-19T11:32:04.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.155 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.155 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:04.155 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.130091+0000", 2024-03-19T11:32:04.155 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017533863 2024-03-19T11:32:04.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.155 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.156 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-03-19T11:32:04.156 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.130332+0000", 2024-03-19T11:32:04.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00024066199999999999 2024-03-19T11:32:04.156 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.156 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:04.156 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.132445+0000", 2024-03-19T11:32:04.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021125570000000001 2024-03-19T11:32:04.156 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.156 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:04.156 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.132570+0000", 2024-03-19T11:32:04.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012532900000000001 2024-03-19T11:32:04.156 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.157 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.157 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:04.157 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.132731+0000", 2024-03-19T11:32:04.157 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016140600000000001 2024-03-19T11:32:04.157 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.157 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:04.157 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.157 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.157 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:04.157 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:32:04.159 INFO:tasks.rados.rados.0.smithi038.stdout:1813: writing smithi03829856-21 from 1687631 to 2422408 tid 2 2024-03-19T11:32:04.159 INFO:tasks.rados.rados.0.smithi038.stdout:1813: writing smithi03829856-21 from 3093218 to 3697573 tid 3 2024-03-19T11:32:04.160 INFO:tasks.rados.rados.0.smithi038.stdout:1810: done (5 left) 2024-03-19T11:32:04.160 INFO:tasks.rados.rados.0.smithi038.stdout:1814: tier_flush oid 9 2024-03-19T11:32:04.160 INFO:tasks.rados.rados.0.smithi038.stdout:1814: tier_flush oid 9 snap -1 2024-03-19T11:32:04.160 INFO:tasks.rados.rados.0.smithi038.stdout:1808: finishing write tid 3 to smithi03829856-26 2024-03-19T11:32:04.160 INFO:tasks.rados.rados.0.smithi038.stdout:1808: finishing write tid 4 to smithi03829856-26 2024-03-19T11:32:04.160 INFO:tasks.rados.rados.0.smithi038.stdout:1815: write oid 40 current snap is 40 2024-03-19T11:32:04.160 INFO:tasks.rados.rados.0.smithi038.stdout:1808: finishing write tid 5 to smithi03829856-26 2024-03-19T11:32:04.160 INFO:tasks.rados.rados.0.smithi038.stdout:1808: oid 26 updating version 0 to 85 2024-03-19T11:32:04.161 INFO:tasks.rados.rados.0.smithi038.stdout:1808: oid 26 updating version 85 to 87 2024-03-19T11:32:04.161 INFO:tasks.rados.rados.0.smithi038.stdout:1808: oid 26 version 87 is already newer than 86 2024-03-19T11:32:04.161 INFO:tasks.rados.rados.0.smithi038.stdout:1808: oid 26 version 87 is already newer than 84 2024-03-19T11:32:04.161 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 26 v 87 (ObjNum 338 snap 40 seq_num 338) dirty exists 2024-03-19T11:32:04.161 INFO:tasks.rados.rados.0.smithi038.stdout:1808: left oid 26 (ObjNum 338 snap 40 seq_num 338) 2024-03-19T11:32:04.161 INFO:tasks.rados.rados.0.smithi038.stdout:1811: finishing set_chunk to oid 1 2024-03-19T11:32:04.161 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 1 v 185 (ObjNum 51 snap 0 seq_num 51) dirty exists 2024-03-19T11:32:04.161 INFO:tasks.rados.rados.0.smithi038.stdout:1809: expect (ObjNum 60 snap 3 seq_num 60) 2024-03-19T11:32:04.162 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-19T11:32:04.162 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-03-19T11:32:04.163 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:32:04.167+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:32:04.190 INFO:tasks.rados.rados.0.smithi038.stdout:1812: finishing set_chunk to oid 17 2024-03-19T11:32:04.190 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 17 v 324 (ObjNum 183 snap 20 seq_num 183) dirty exists 2024-03-19T11:32:04.190 INFO:tasks.rados.rados.0.smithi038.stdout:1813: finishing write tid 1 to smithi03829856-21 2024-03-19T11:32:04.190 INFO:tasks.rados.rados.0.smithi038.stdout:1813: finishing write tid 2 to smithi03829856-21 2024-03-19T11:32:04.191 INFO:tasks.rados.rados.0.smithi038.stdout:1815: seq_num 340 ranges {514380=704192,1629760=478435,2604288=442602} 2024-03-19T11:32:04.196 INFO:tasks.rados.rados.0.smithi038.stdout:1815: writing smithi03829856-40 from 514380 to 1218572 tid 1 2024-03-19T11:32:04.201 INFO:tasks.rados.rados.0.smithi038.stdout:1815: writing smithi03829856-40 from 1629760 to 2108195 tid 2 2024-03-19T11:32:04.208 INFO:tasks.rados.rados.0.smithi038.stdout:1815: writing smithi03829856-40 from 2604288 to 3046890 tid 3 2024-03-19T11:32:04.208 INFO:tasks.rados.rados.0.smithi038.stdout:1808: done (6 left) 2024-03-19T11:32:04.208 INFO:tasks.rados.rados.0.smithi038.stdout:1809: done (5 left) 2024-03-19T11:32:04.208 INFO:tasks.rados.rados.0.smithi038.stdout:1811: done (4 left) 2024-03-19T11:32:04.208 INFO:tasks.rados.rados.0.smithi038.stdout:1812: done (3 left) 2024-03-19T11:32:04.208 INFO:tasks.rados.rados.0.smithi038.stdout:1816: write oid 15 current snap is 40 2024-03-19T11:32:04.208 INFO:tasks.rados.rados.0.smithi038.stdout:1816: seq_num 341 ranges {531187=779441,2088242=738446,3431711=299476} 2024-03-19T11:32:04.219 INFO:tasks.rados.rados.0.smithi038.stdout:1816: writing smithi03829856-15 from 531187 to 1310628 tid 1 2024-03-19T11:32:04.228 INFO:tasks.rados.rados.0.smithi038.stdout:1816: writing smithi03829856-15 from 2088242 to 2826688 tid 2 2024-03-19T11:32:04.235 INFO:tasks.rados.rados.0.smithi038.stdout:1816: writing smithi03829856-15 from 3431711 to 3731187 tid 3 2024-03-19T11:32:04.235 INFO:tasks.rados.rados.0.smithi038.stdout:1817: snap_remove snap 32 2024-03-19T11:32:04.260 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:32:04.269+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:32:04.335 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:32:04.336 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:32:04.336 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:32:04.336 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:32:04.350 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-19T11:32:04.362 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:32:04.371+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:32:04.462 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:32:04.471+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:32:04.517 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:32:04.518 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:32:04.518 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:32:04.518 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:32:04.518 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:32:04.532 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-19T11:32:04.580 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:32:04.589+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:32:04.655 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:32:04.655 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:32:04.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:32:04.655 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:32:04.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.655 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:32:04.655 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:44.200920+0000", 2024-03-19T11:32:04.655 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 140.46204636100001, 2024-03-19T11:32:04.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.80881896499999995, 2024-03-19T11:32:04.656 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:04.656 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:04.656 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:04.656 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:32:04.656 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:32:04.656 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1 2024-03-19T11:32:04.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.656 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:04.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.656 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:04.657 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200920+0000", 2024-03-19T11:32:04.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.657 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.657 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.657 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:04.657 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200920+0000", 2024-03-19T11:32:04.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.658 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.658 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:04.658 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200923+0000", 2024-03-19T11:32:04.658 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6469999999999998e-06 2024-03-19T11:32:04.658 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.658 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:04.658 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200937+0000", 2024-03-19T11:32:04.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.399e-05 2024-03-19T11:32:04.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:04.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200960+0000", 2024-03-19T11:32:04.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2603e-05 2024-03-19T11:32:04.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:04.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200978+0000", 2024-03-19T11:32:04.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.791e-05 2024-03-19T11:32:04.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.660 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:04.660 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.201081+0000", 2024-03-19T11:32:04.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010344900000000001 2024-03-19T11:32:04.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.660 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:32:04.660 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.201096+0000", 2024-03-19T11:32:04.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4443e-05 2024-03-19T11:32:04.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.660 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:04.660 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.005978+0000", 2024-03-19T11:32:04.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.80488281500000003 2024-03-19T11:32:04.661 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.661 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:04.661 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.006267+0000", 2024-03-19T11:32:04.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000288688 2024-03-19T11:32:04.661 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.661 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:32:04.661 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.006534+0000", 2024-03-19T11:32:04.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026725799999999998 2024-03-19T11:32:04.661 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.662 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:32:04.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.008457+0000", 2024-03-19T11:32:04.662 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0019227820000000001 2024-03-19T11:32:04.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.662 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:04.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.009570+0000", 2024-03-19T11:32:04.662 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011131750000000001 2024-03-19T11:32:04.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.662 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:04.663 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.009670+0000", 2024-03-19T11:32:04.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9458999999999998e-05 2024-03-19T11:32:04.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.663 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:04.663 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.009739+0000", 2024-03-19T11:32:04.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9746000000000006e-05 2024-03-19T11:32:04.663 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.663 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:04.663 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.663 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:32:04.663 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.339898+0000", 2024-03-19T11:32:04.664 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 139.32306925399999, 2024-03-19T11:32:04.664 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019501793, 2024-03-19T11:32:04.664 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:04.664 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:04.664 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:04.664 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:32:04.664 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:32:04.664 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113 2024-03-19T11:32:04.664 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.664 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:04.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.664 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:04.664 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339898+0000", 2024-03-19T11:32:04.664 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.665 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:04.665 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339898+0000", 2024-03-19T11:32:04.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.665 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:04.665 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339899+0000", 2024-03-19T11:32:04.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.353e-06 2024-03-19T11:32:04.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.665 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:04.665 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339907+0000", 2024-03-19T11:32:04.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3000000000000002e-06 2024-03-19T11:32:04.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.666 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:04.666 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339924+0000", 2024-03-19T11:32:04.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6906999999999999e-05 2024-03-19T11:32:04.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.666 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:04.666 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339933+0000", 2024-03-19T11:32:04.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8969999999999997e-06 2024-03-19T11:32:04.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.666 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:04.666 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352239+0000", 2024-03-19T11:32:04.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012305814999999999 2024-03-19T11:32:04.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:04.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352466+0000", 2024-03-19T11:32:04.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022684100000000001 2024-03-19T11:32:04.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:32:04.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352816+0000", 2024-03-19T11:32:04.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00035030499999999997 2024-03-19T11:32:04.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:32:04.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.358956+0000", 2024-03-19T11:32:04.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061400079999999998 2024-03-19T11:32:04.668 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.668 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.668 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:04.668 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359118+0000", 2024-03-19T11:32:04.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016237599999999999 2024-03-19T11:32:04.668 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.668 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.668 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:04.668 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359287+0000", 2024-03-19T11:32:04.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000168488 2024-03-19T11:32:04.668 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.668 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.668 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:04.668 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359399+0000", 2024-03-19T11:32:04.669 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000112503 2024-03-19T11:32:04.669 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.669 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:04.669 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.669 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.669 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.669 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:32:04.669 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.339947+0000", 2024-03-19T11:32:04.669 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 139.32301948700001, 2024-03-19T11:32:04.669 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021923069999999999, 2024-03-19T11:32:04.669 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:04.669 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:04.669 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:04.669 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:32:04.670 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:32:04.670 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 114 2024-03-19T11:32:04.670 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.670 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:04.670 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.670 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:04.670 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339947+0000", 2024-03-19T11:32:04.670 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.670 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.670 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.671 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:04.671 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339947+0000", 2024-03-19T11:32:04.671 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.671 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.671 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:04.671 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339948+0000", 2024-03-19T11:32:04.671 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6000000000000003e-07 2024-03-19T11:32:04.671 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.671 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:04.671 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339956+0000", 2024-03-19T11:32:04.671 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1559999999999995e-06 2024-03-19T11:32:04.671 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:04.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339967+0000", 2024-03-19T11:32:04.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0669999999999999e-05 2024-03-19T11:32:04.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:04.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339975+0000", 2024-03-19T11:32:04.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0099999999999995e-06 2024-03-19T11:32:04.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:04.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.353672+0000", 2024-03-19T11:32:04.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013696583999999999 2024-03-19T11:32:04.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.673 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:04.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.353808+0000", 2024-03-19T11:32:04.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013652299999999999 2024-03-19T11:32:04.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.673 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:32:04.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.354036+0000", 2024-03-19T11:32:04.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022744799999999999 2024-03-19T11:32:04.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.673 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:32:04.674 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.360480+0000", 2024-03-19T11:32:04.674 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0064441630000000001 2024-03-19T11:32:04.674 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.674 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.674 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:04.674 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361722+0000", 2024-03-19T11:32:04.674 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001242754 2024-03-19T11:32:04.674 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.674 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.674 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:04.674 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361807+0000", 2024-03-19T11:32:04.674 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4605999999999996e-05 2024-03-19T11:32:04.674 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.674 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.675 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:04.675 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361870+0000", 2024-03-19T11:32:04.675 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3396000000000001e-05 2024-03-19T11:32:04.675 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.675 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:04.675 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.675 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.675 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.675 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:32:04.675 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.340017+0000", 2024-03-19T11:32:04.675 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 139.322950171, 2024-03-19T11:32:04.675 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022169355000000002, 2024-03-19T11:32:04.676 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:04.676 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:04.676 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:04.676 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:32:04.676 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:32:04.676 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 115 2024-03-19T11:32:04.676 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.676 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:04.676 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.676 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:04.676 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340017+0000", 2024-03-19T11:32:04.676 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.677 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:04.677 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340017+0000", 2024-03-19T11:32:04.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.678 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:04.678 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340018+0000", 2024-03-19T11:32:04.678 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.043e-06 2024-03-19T11:32:04.678 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.678 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.678 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:04.679 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340139+0000", 2024-03-19T11:32:04.679 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121502 2024-03-19T11:32:04.679 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.679 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:04.679 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340157+0000", 2024-03-19T11:32:04.679 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7382999999999999e-05 2024-03-19T11:32:04.679 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.679 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:04.679 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340166+0000", 2024-03-19T11:32:04.679 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2469999999999997e-06 2024-03-19T11:32:04.680 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.680 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.680 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:04.680 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.357470+0000", 2024-03-19T11:32:04.680 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017304119999999999 2024-03-19T11:32:04.680 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.680 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.680 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:04.680 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.357671+0000", 2024-03-19T11:32:04.680 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000200798 2024-03-19T11:32:04.680 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.681 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:32:04.681 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.358002+0000", 2024-03-19T11:32:04.681 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033090099999999998 2024-03-19T11:32:04.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.681 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:32:04.681 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361892+0000", 2024-03-19T11:32:04.681 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00389018 2024-03-19T11:32:04.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.681 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:04.681 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361983+0000", 2024-03-19T11:32:04.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0976000000000004e-05 2024-03-19T11:32:04.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.682 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.682 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:04.682 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362071+0000", 2024-03-19T11:32:04.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8022999999999994e-05 2024-03-19T11:32:04.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.682 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.682 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:04.682 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362186+0000", 2024-03-19T11:32:04.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115182 2024-03-19T11:32:04.682 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.682 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:04.682 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.683 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.683 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:32:04.683 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.340183+0000", 2024-03-19T11:32:04.683 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 139.32278333900001, 2024-03-19T11:32:04.683 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022279841000000002, 2024-03-19T11:32:04.683 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:04.683 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:04.683 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:04.683 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:32:04.683 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:32:04.683 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 116 2024-03-19T11:32:04.683 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.684 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:04.684 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.684 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:04.684 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340183+0000", 2024-03-19T11:32:04.684 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.684 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.684 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.684 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:04.684 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340183+0000", 2024-03-19T11:32:04.684 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.684 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.684 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.684 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:04.684 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340184+0000", 2024-03-19T11:32:04.684 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9000000000000005e-07 2024-03-19T11:32:04.685 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.685 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.685 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:04.685 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340191+0000", 2024-03-19T11:32:04.685 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0360000000000003e-06 2024-03-19T11:32:04.685 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.685 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.685 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:04.685 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340201+0000", 2024-03-19T11:32:04.685 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0844e-05 2024-03-19T11:32:04.685 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.685 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.685 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:04.685 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340209+0000", 2024-03-19T11:32:04.686 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7360000000000001e-06 2024-03-19T11:32:04.686 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.686 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.686 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:04.686 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359535+0000", 2024-03-19T11:32:04.686 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019325546999999998 2024-03-19T11:32:04.686 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.686 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.686 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:04.686 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359672+0000", 2024-03-19T11:32:04.686 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013706600000000001 2024-03-19T11:32:04.686 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.686 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.686 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:32:04.687 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359905+0000", 2024-03-19T11:32:04.687 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023334500000000001 2024-03-19T11:32:04.687 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.687 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.687 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:04.687 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362306+0000", 2024-03-19T11:32:04.687 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002401041 2024-03-19T11:32:04.687 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.687 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.687 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:32:04.687 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362329+0000", 2024-03-19T11:32:04.687 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3283999999999999e-05 2024-03-19T11:32:04.687 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.687 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.687 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:04.688 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362402+0000", 2024-03-19T11:32:04.688 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2459000000000005e-05 2024-03-19T11:32:04.688 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.688 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.688 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:04.688 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362463+0000", 2024-03-19T11:32:04.688 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1493000000000002e-05 2024-03-19T11:32:04.688 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.688 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:04.688 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.688 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.688 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.688 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:32:04.689 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.340222+0000", 2024-03-19T11:32:04.689 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 139.32274475599999, 2024-03-19T11:32:04.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022500218999999998, 2024-03-19T11:32:04.689 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:04.689 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:04.689 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:04.689 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:32:04.689 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:32:04.689 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 117 2024-03-19T11:32:04.689 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.689 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:04.689 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.689 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:04.689 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340222+0000", 2024-03-19T11:32:04.690 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.690 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.690 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.690 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:04.690 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340222+0000", 2024-03-19T11:32:04.690 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.690 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.690 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.690 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:04.690 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340223+0000", 2024-03-19T11:32:04.690 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7999999999999995e-07 2024-03-19T11:32:04.690 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.690 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.691 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:04.691 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340229+0000", 2024-03-19T11:32:04.691 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0399999999999998e-06 2024-03-19T11:32:04.691 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.691 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.691 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:04.691 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340239+0000", 2024-03-19T11:32:04.691 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0190000000000001e-05 2024-03-19T11:32:04.691 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.691 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.691 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:04.692 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340250+0000", 2024-03-19T11:32:04.692 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0720000000000001e-05 2024-03-19T11:32:04.692 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.692 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.692 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:04.692 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.360587+0000", 2024-03-19T11:32:04.693 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020337542 2024-03-19T11:32:04.693 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.693 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.693 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:04.693 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.360818+0000", 2024-03-19T11:32:04.693 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023120500000000001 2024-03-19T11:32:04.694 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.694 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.694 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:32:04.694 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361218+0000", 2024-03-19T11:32:04.694 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000399637 2024-03-19T11:32:04.694 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.694 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.694 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:32:04.695 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362483+0000", 2024-03-19T11:32:04.695 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012649200000000001 2024-03-19T11:32:04.695 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.695 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.695 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:04.695 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362538+0000", 2024-03-19T11:32:04.695 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4490000000000002e-05 2024-03-19T11:32:04.695 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.695 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.695 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:04.695 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362617+0000", 2024-03-19T11:32:04.695 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9839000000000006e-05 2024-03-19T11:32:04.695 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.695 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.696 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:04.696 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362722+0000", 2024-03-19T11:32:04.696 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104956 2024-03-19T11:32:04.696 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.696 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:04.696 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.696 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.696 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.696 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.15 51/50 [PushOp(4:afa9f208:::smithi03829856-35:head, version: 28'46, data_included: [571438~623878,1806623~424698], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:afa9f208:::smithi03829856-35:head@28'46, size: 2571438, copy_subset: [(0, 2571438)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2231321, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:af9d0d56:::smithi03829856-20:head, version: 35'55, data_included: [655360~589824,1900544~418789], data_size: 1008613, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:af9d0d56:::smithi03829856-20:head@35'55, size: 2319333, copy_subset: [(0, 2319333)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2319333, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ad3ad1b3:::smithi03829856-21:b, version: 47'66, data_included: [736140~537686,2020581~315559], data_size: 853245, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:ad3ad1b3:::smithi03829856-21:b@47'66, size: 2336140, copy_subset: [(0, 2336140)], clone_subset: {}, snapset: b=[]:{b: [b, a, 6, 5]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2336140, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ad3ad1b3:::smithi03829856-21:head, version: 47'70, data_included: [531121~742705,1711057~305871], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:ad3ad1b3:::smithi03829856-21:head@47'70, size: 3731121, copy_subset: [(0, 3731121)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2016928, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ac9cd3a3:::smithi03829856-8:head, version: 49'72, data_included: [517649~620182,1776668~428394], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:ac9cd3a3:::smithi03829856-8:head@49'72, size: 3008420, copy_subset: [(0, 3008420)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2205062, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:aa3303a5:::smithi03829856-17:b, version: 49'73, data_included: [614400~692224,1903006~356352], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:aa3303a5:::smithi03829856-17:b@49'73, size: 2268912, copy_subset: [(0, 2268912)], clone_subset: {}, snapset: b=[]:{b: [b, a, 6, 5]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2259358, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:aa3303a5:::smithi03829856-17:head, version: 49'74, data_included: [413472~893152,1818036~155424], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:aa3303a5:::smithi03829856-17:head@49'74, size: 3613472, copy_subset: [(0, 3613472)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1973460, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:32:04.697 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:28.058610+0000", 2024-03-19T11:32:04.697 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 96.604356584000001, 2024-03-19T11:32:04.697 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027100032, 2024-03-19T11:32:04.697 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:04.697 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:32:04.697 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:04.697 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.697 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:04.697 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.058610+0000", 2024-03-19T11:32:04.697 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.697 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.697 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.697 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:04.697 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.058610+0000", 2024-03-19T11:32:04.697 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.698 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.698 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.698 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:04.698 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.058611+0000", 2024-03-19T11:32:04.698 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1599999999999999e-06 2024-03-19T11:32:04.698 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.698 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.698 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:04.698 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.063874+0000", 2024-03-19T11:32:04.698 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052625270000000004 2024-03-19T11:32:04.698 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.698 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.698 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:04.699 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.063935+0000", 2024-03-19T11:32:04.699 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0594000000000003e-05 2024-03-19T11:32:04.699 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.699 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.699 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:04.699 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064024+0000", 2024-03-19T11:32:04.699 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9273999999999999e-05 2024-03-19T11:32:04.699 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.699 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.699 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:04.700 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064385+0000", 2024-03-19T11:32:04.700 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00036129100000000002 2024-03-19T11:32:04.700 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.700 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.700 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:04.700 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064437+0000", 2024-03-19T11:32:04.700 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2346000000000003e-05 2024-03-19T11:32:04.701 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.701 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.701 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:04.701 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085710+0000", 2024-03-19T11:32:04.701 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021272840000000001 2024-03-19T11:32:04.701 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.701 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:04.701 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.701 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.701 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.701 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.15 e51,50 4:ac9cd3a3:::smithi03829856-8:4,49'71)", 2024-03-19T11:32:04.701 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:28.064112+0000", 2024-03-19T11:32:04.701 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 96.598854758000002, 2024-03-19T11:32:04.701 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026480417999999999, 2024-03-19T11:32:04.702 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:04.702 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:32:04.702 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:04.702 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.702 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:04.702 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064112+0000", 2024-03-19T11:32:04.702 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.702 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.702 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.702 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:04.702 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064112+0000", 2024-03-19T11:32:04.702 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.702 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.703 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.703 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:04.703 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064113+0000", 2024-03-19T11:32:04.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3600000000000003e-07 2024-03-19T11:32:04.703 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.703 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.703 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:04.703 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064118+0000", 2024-03-19T11:32:04.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0549999999999999e-06 2024-03-19T11:32:04.703 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.703 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.703 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:04.703 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064128+0000", 2024-03-19T11:32:04.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0349e-05 2024-03-19T11:32:04.703 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.704 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.704 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:04.704 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064136+0000", 2024-03-19T11:32:04.704 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6249999999999998e-06 2024-03-19T11:32:04.704 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.704 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.704 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:04.704 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085773+0000", 2024-03-19T11:32:04.704 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021636829999999999 2024-03-19T11:32:04.704 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.704 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.704 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:04.704 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085788+0000", 2024-03-19T11:32:04.704 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5512999999999999e-05 2024-03-19T11:32:04.704 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.705 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.705 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:04.705 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.090592+0000", 2024-03-19T11:32:04.705 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0048044100000000003 2024-03-19T11:32:04.705 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.705 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:04.705 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.705 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.705 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.705 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.5 51/50 [PushOp(4:a38b9453:::smithi03829856-37:head, version: 28'45, data_included: [605052~774376,1923587~274200], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:a38b9453:::smithi03829856-37:head@28'45, size: 2605052, copy_subset: [(0, 2605052)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2197787, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:a0a50d04:::smithi03829856-13:b, version: 47'63, data_included: [601684~619341,1669078~429235], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a0a50d04:::smithi03829856-13:b@47'63, size: 2201684, copy_subset: [(0, 2201684)], clone_subset: {}, snapset: b=[]:{b: [b, a, 6, 5]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2098313, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:32:04.705 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:28.064324+0000", 2024-03-19T11:32:04.705 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 96.598642913999996, 2024-03-19T11:32:04.705 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027179020000000002, 2024-03-19T11:32:04.705 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:04.705 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:32:04.706 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:04.706 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.706 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:04.706 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064324+0000", 2024-03-19T11:32:04.706 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.706 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.706 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.706 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:04.706 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064324+0000", 2024-03-19T11:32:04.706 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.706 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.706 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.706 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:04.706 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064324+0000", 2024-03-19T11:32:04.707 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1199999999999998e-07 2024-03-19T11:32:04.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.707 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:04.707 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.065527+0000", 2024-03-19T11:32:04.707 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012028939999999999 2024-03-19T11:32:04.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.707 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:04.707 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.065552+0000", 2024-03-19T11:32:04.707 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4440999999999999e-05 2024-03-19T11:32:04.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.707 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:04.707 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.065568+0000", 2024-03-19T11:32:04.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6498e-05 2024-03-19T11:32:04.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.708 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.708 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:04.708 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085812+0000", 2024-03-19T11:32:04.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020243954000000002 2024-03-19T11:32:04.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.708 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.708 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:04.708 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085825+0000", 2024-03-19T11:32:04.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3298999999999999e-05 2024-03-19T11:32:04.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.708 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.708 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:04.708 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.091503+0000", 2024-03-19T11:32:04.709 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.005677522 2024-03-19T11:32:04.709 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.709 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:04.709 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.709 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.709 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.709 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.24 75/74 [PushOp(4:27909373:::smithi03829856-46:8, version: 46'59, data_included: [756315~767970,1933572~280606], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:8@46'59, size: 2756315, copy_subset: [(0, 2756315)], clone_subset: {}, snapset: 10=[]:{8: [6], c: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2214178, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:266590a5:::smithi03829856-16:e, version: 70'107, data_included: [652105~662359,1728171~386217], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:e@70'107, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 10=[]:{e: [d, b, 6]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:266590a5:::smithi03829856-16:head, version: 70'108, data_included: [652105~662359,1728171~386217], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:head@70'108, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:27909373:::smithi03829856-46:c, version: 70'109, data_included: [628595~895690,1807440~152886], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:c@70'109, size: 3428595, copy_subset: [(0, 3428595)], clone_subset: {}, snapset: 10=[]:{8: [6], c: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1960326, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:27909373:::smithi03829856-46:head, version: 70'110, data_included: [403401~454961,1584925~506039], data_size: 961000, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:head@70'110, size: 2403401, copy_subset: [(0, 2403401)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2403401, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:32:04.709 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.099172+0000", 2024-03-19T11:32:04.709 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 70.563794479999999, 2024-03-19T11:32:04.709 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.024528580000000001, 2024-03-19T11:32:04.709 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:04.709 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:32:04.709 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:04.709 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.710 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:04.710 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.099172+0000", 2024-03-19T11:32:04.710 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.710 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.710 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:04.710 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.099172+0000", 2024-03-19T11:32:04.710 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.710 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.710 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:04.710 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.099175+0000", 2024-03-19T11:32:04.710 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7750000000000001e-06 2024-03-19T11:32:04.710 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.711 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:04.711 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105219+0000", 2024-03-19T11:32:04.711 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0060440789999999999 2024-03-19T11:32:04.711 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.711 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.711 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:04.711 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105285+0000", 2024-03-19T11:32:04.711 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6260000000000006e-05 2024-03-19T11:32:04.712 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.712 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.712 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:04.712 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105366+0000", 2024-03-19T11:32:04.712 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.038e-05 2024-03-19T11:32:04.712 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.713 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.713 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:04.713 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105637+0000", 2024-03-19T11:32:04.713 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027085199999999997 2024-03-19T11:32:04.713 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.713 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.713 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:04.713 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105679+0000", 2024-03-19T11:32:04.713 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2088000000000001e-05 2024-03-19T11:32:04.713 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.713 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.713 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:04.714 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.123701+0000", 2024-03-19T11:32:04.714 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018022145999999999 2024-03-19T11:32:04.714 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.714 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:04.714 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.714 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.714 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.714 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.24 e75,74 4:266590a5:::smithi03829856-16:4,47'63,4:27909373:::smithi03829856-46:5,55'79,4:266590a5:::smithi03829856-16:10,67'105)", 2024-03-19T11:32:04.714 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.105437+0000", 2024-03-19T11:32:04.714 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 70.557529477000003, 2024-03-19T11:32:04.714 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.01860726, 2024-03-19T11:32:04.714 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:04.714 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:32:04.715 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:04.715 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.715 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:04.715 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105437+0000", 2024-03-19T11:32:04.715 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.715 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.715 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.715 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:04.715 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105437+0000", 2024-03-19T11:32:04.716 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.716 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.716 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.716 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:04.716 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105438+0000", 2024-03-19T11:32:04.716 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6799999999999999e-07 2024-03-19T11:32:04.716 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.717 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.717 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:04.717 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105445+0000", 2024-03-19T11:32:04.717 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3749999999999999e-06 2024-03-19T11:32:04.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.717 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.717 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:04.717 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105454+0000", 2024-03-19T11:32:04.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7909999999999999e-06 2024-03-19T11:32:04.718 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.718 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.718 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:04.718 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105465+0000", 2024-03-19T11:32:04.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.045e-05 2024-03-19T11:32:04.718 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.718 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.719 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:04.719 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.123753+0000", 2024-03-19T11:32:04.719 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0182884 2024-03-19T11:32:04.719 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.719 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:04.719 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.123765+0000", 2024-03-19T11:32:04.719 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2143e-05 2024-03-19T11:32:04.719 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.719 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:04.719 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.124045+0000", 2024-03-19T11:32:04.719 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027923299999999999 2024-03-19T11:32:04.719 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.720 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:04.720 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.720 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.720 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:3103 4.15 e104/76 4:afa9f208:::smithi03829856-35:head v 104'221, mlcod=101'216)", 2024-03-19T11:32:04.720 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.563593+0000", 2024-03-19T11:32:04.720 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 41.099374199000003, 2024-03-19T11:32:04.720 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026983566, 2024-03-19T11:32:04.720 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:04.720 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:04.720 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:04.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.720 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:04.720 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.563593+0000", 2024-03-19T11:32:04.720 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.721 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.721 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:04.721 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.563593+0000", 2024-03-19T11:32:04.721 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.721 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.721 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:04.721 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.563595+0000", 2024-03-19T11:32:04.721 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2859999999999998e-06 2024-03-19T11:32:04.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.721 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.721 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:04.721 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.564339+0000", 2024-03-19T11:32:04.721 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00074400099999999999 2024-03-19T11:32:04.722 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.722 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.722 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:04.722 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.564376+0000", 2024-03-19T11:32:04.722 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6628999999999998e-05 2024-03-19T11:32:04.722 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.722 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.722 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:04.722 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.564396+0000", 2024-03-19T11:32:04.722 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0596000000000001e-05 2024-03-19T11:32:04.722 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.722 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.722 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:04.722 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.566600+0000", 2024-03-19T11:32:04.723 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002204021 2024-03-19T11:32:04.723 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.723 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.723 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:04.723 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.566642+0000", 2024-03-19T11:32:04.723 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1902e-05 2024-03-19T11:32:04.723 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.723 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.723 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:04.723 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.590494+0000", 2024-03-19T11:32:04.723 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023851816000000001 2024-03-19T11:32:04.723 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.723 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.723 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:04.724 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.590576+0000", 2024-03-19T11:32:04.724 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2315e-05 2024-03-19T11:32:04.724 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.724 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:04.724 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.724 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.724 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.725 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:3104 4.15 e104/76 4:afa9f208:::smithi03829856-35:head v 104'222, mlcod=101'216)", 2024-03-19T11:32:04.725 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.568280+0000", 2024-03-19T11:32:04.725 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 41.094686355999997, 2024-03-19T11:32:04.725 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022345304999999999, 2024-03-19T11:32:04.725 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:04.725 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:04.725 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:04.725 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.725 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:04.726 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.568280+0000", 2024-03-19T11:32:04.726 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.726 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.726 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.726 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:04.726 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.568280+0000", 2024-03-19T11:32:04.726 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.726 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.727 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:04.727 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.568282+0000", 2024-03-19T11:32:04.727 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0090000000000002e-06 2024-03-19T11:32:04.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.727 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:04.727 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.568299+0000", 2024-03-19T11:32:04.727 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6237000000000001e-05 2024-03-19T11:32:04.728 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.728 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.728 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:04.728 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.568331+0000", 2024-03-19T11:32:04.728 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2601999999999997e-05 2024-03-19T11:32:04.728 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.728 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.728 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:04.728 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.568350+0000", 2024-03-19T11:32:04.729 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8280000000000001e-05 2024-03-19T11:32:04.729 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.729 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.729 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:04.729 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.571311+0000", 2024-03-19T11:32:04.729 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029617770000000001 2024-03-19T11:32:04.729 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.729 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.729 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:04.730 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.571379+0000", 2024-03-19T11:32:04.730 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7331999999999998e-05 2024-03-19T11:32:04.730 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.730 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.730 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:04.730 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.590583+0000", 2024-03-19T11:32:04.730 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019204649000000001 2024-03-19T11:32:04.730 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.731 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.731 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:04.731 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.590626+0000", 2024-03-19T11:32:04.731 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2419000000000002e-05 2024-03-19T11:32:04.731 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.731 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:04.731 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.731 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.731 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.732 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:3106 4.15 e104/76 4:ad3ad1b3:::smithi03829856-21:head v 104'224, mlcod=101'216)", 2024-03-19T11:32:04.732 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.571056+0000", 2024-03-19T11:32:04.732 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 41.091910898000002, 2024-03-19T11:32:04.732 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019636355000000001, 2024-03-19T11:32:04.732 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:04.732 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:04.732 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:04.732 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.732 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:04.733 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.571056+0000", 2024-03-19T11:32:04.733 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.733 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.733 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.733 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:04.733 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.571056+0000", 2024-03-19T11:32:04.733 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.733 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.733 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.734 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:04.734 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.571058+0000", 2024-03-19T11:32:04.734 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2400000000000002e-06 2024-03-19T11:32:04.734 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.734 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.734 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:04.734 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.571586+0000", 2024-03-19T11:32:04.734 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00052782299999999996 2024-03-19T11:32:04.734 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.735 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.735 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:04.735 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.571618+0000", 2024-03-19T11:32:04.735 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2453000000000003e-05 2024-03-19T11:32:04.735 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.735 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.735 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:04.735 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.571638+0000", 2024-03-19T11:32:04.735 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9176000000000001e-05 2024-03-19T11:32:04.736 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.736 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.736 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:04.736 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.573958+0000", 2024-03-19T11:32:04.736 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0023201699999999999 2024-03-19T11:32:04.736 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.736 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.736 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:04.736 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.574023+0000", 2024-03-19T11:32:04.737 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5425000000000005e-05 2024-03-19T11:32:04.737 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.737 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.737 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:04.737 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.590646+0000", 2024-03-19T11:32:04.737 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016623189 2024-03-19T11:32:04.737 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.737 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.738 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:04.738 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.590692+0000", 2024-03-19T11:32:04.738 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5878999999999999e-05 2024-03-19T11:32:04.738 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.738 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:04.738 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.738 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.738 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.738 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.3.0:309 4.0 e108/60 4:01018511:::smithi03829856-19:b v 108'91, mlcod=100'87)", 2024-03-19T11:32:04.738 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.205898+0000", 2024-03-19T11:32:04.738 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 37.457069103000002, 2024-03-19T11:32:04.738 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.034464328000000002, 2024-03-19T11:32:04.738 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:04.738 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:04.739 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:04.739 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.739 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:04.739 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.205898+0000", 2024-03-19T11:32:04.739 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.739 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.739 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.739 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:04.739 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.205898+0000", 2024-03-19T11:32:04.739 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.739 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.739 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.740 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:04.740 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.205899+0000", 2024-03-19T11:32:04.740 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.16e-07 2024-03-19T11:32:04.740 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.740 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.740 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:04.740 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.205916+0000", 2024-03-19T11:32:04.740 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7229000000000001e-05 2024-03-19T11:32:04.740 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.740 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.740 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:04.740 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.205946+0000", 2024-03-19T11:32:04.740 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0233999999999999e-05 2024-03-19T11:32:04.740 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.740 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.741 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:04.741 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.205966+0000", 2024-03-19T11:32:04.741 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9434999999999999e-05 2024-03-19T11:32:04.741 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.741 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.741 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:04.741 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.206162+0000", 2024-03-19T11:32:04.741 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000196918 2024-03-19T11:32:04.741 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.741 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.741 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:04.741 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.206227+0000", 2024-03-19T11:32:04.741 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4381000000000006e-05 2024-03-19T11:32:04.741 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.742 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.742 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:04.742 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.240212+0000", 2024-03-19T11:32:04.742 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033984852000000003 2024-03-19T11:32:04.742 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.742 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.742 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:04.742 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.240362+0000", 2024-03-19T11:32:04.742 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015036300000000001 2024-03-19T11:32:04.742 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.742 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:04.742 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.742 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.742 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.743 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.5.0:238 4.15 e108/76 4:ac9cd3a3:::smithi03829856-8:10 v 108'231, mlcod=104'228)", 2024-03-19T11:32:04.743 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.215160+0000", 2024-03-19T11:32:04.743 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 37.447806694999997, 2024-03-19T11:32:04.743 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.024600749000000002, 2024-03-19T11:32:04.743 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:04.743 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:04.743 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:04.743 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.743 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:04.743 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215160+0000", 2024-03-19T11:32:04.743 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.743 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.743 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.743 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:04.744 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215160+0000", 2024-03-19T11:32:04.744 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.744 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.744 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.744 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:04.744 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215161+0000", 2024-03-19T11:32:04.744 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6000000000000002e-07 2024-03-19T11:32:04.744 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.744 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.744 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:04.744 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215172+0000", 2024-03-19T11:32:04.744 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1042e-05 2024-03-19T11:32:04.744 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.744 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.744 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:04.745 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215213+0000", 2024-03-19T11:32:04.745 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1167000000000001e-05 2024-03-19T11:32:04.745 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.745 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.745 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:04.745 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215234+0000", 2024-03-19T11:32:04.745 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0534e-05 2024-03-19T11:32:04.745 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.745 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.745 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:04.745 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215668+0000", 2024-03-19T11:32:04.745 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000434074 2024-03-19T11:32:04.746 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.746 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.746 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:04.746 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215749+0000", 2024-03-19T11:32:04.746 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0702000000000001e-05 2024-03-19T11:32:04.746 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.746 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.747 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:04.747 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.239629+0000", 2024-03-19T11:32:04.747 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023880855999999999 2024-03-19T11:32:04.747 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.747 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.747 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:04.747 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.239761+0000", 2024-03-19T11:32:04.748 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000131514 2024-03-19T11:32:04.748 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.748 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:04.748 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.748 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.748 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.749 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.5.0:239 4.8 e108/100 4:14611006:::smithi03829856-25:c v 108'125, mlcod=103'122)", 2024-03-19T11:32:04.749 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.215292+0000", 2024-03-19T11:32:04.749 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 37.447674511000002, 2024-03-19T11:32:04.749 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.025163089999999999, 2024-03-19T11:32:04.749 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:04.749 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:04.749 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:04.750 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.750 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:04.750 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215292+0000", 2024-03-19T11:32:04.750 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.750 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.750 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.751 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:04.751 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215292+0000", 2024-03-19T11:32:04.751 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.751 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.751 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.751 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:04.751 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215293+0000", 2024-03-19T11:32:04.752 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.47e-07 2024-03-19T11:32:04.752 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.752 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.752 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:04.752 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215315+0000", 2024-03-19T11:32:04.752 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1691e-05 2024-03-19T11:32:04.753 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.753 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.753 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:04.753 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215348+0000", 2024-03-19T11:32:04.753 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2676999999999999e-05 2024-03-19T11:32:04.753 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.753 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.754 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:04.754 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215362+0000", 2024-03-19T11:32:04.754 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3978999999999999e-05 2024-03-19T11:32:04.754 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.754 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.754 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:04.755 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215727+0000", 2024-03-19T11:32:04.755 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000365583 2024-03-19T11:32:04.755 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.755 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.755 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:04.755 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215804+0000", 2024-03-19T11:32:04.755 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7392999999999996e-05 2024-03-19T11:32:04.756 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.756 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.756 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:04.756 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.240368+0000", 2024-03-19T11:32:04.756 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.024563807999999999 2024-03-19T11:32:04.757 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.757 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.757 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:04.757 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.240455+0000", 2024-03-19T11:32:04.757 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7112000000000002e-05 2024-03-19T11:32:04.757 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.757 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:04.758 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.758 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.758 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.758 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.5.0:240 4.15 e108/76 4:aa3303a5:::smithi03829856-17:b v 108'233, mlcod=104'228)", 2024-03-19T11:32:04.758 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.215400+0000", 2024-03-19T11:32:04.758 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 37.447566780999999, 2024-03-19T11:32:04.759 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026645319000000001, 2024-03-19T11:32:04.759 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:04.759 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:04.759 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:04.759 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.759 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:04.760 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215400+0000", 2024-03-19T11:32:04.760 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.760 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.760 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.760 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:04.760 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215400+0000", 2024-03-19T11:32:04.761 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.761 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.761 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.761 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:04.761 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215401+0000", 2024-03-19T11:32:04.761 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3e-07 2024-03-19T11:32:04.761 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.762 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.762 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:04.762 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215417+0000", 2024-03-19T11:32:04.762 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5821999999999999e-05 2024-03-19T11:32:04.762 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.762 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.763 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:04.763 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215443+0000", 2024-03-19T11:32:04.763 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6971e-05 2024-03-19T11:32:04.763 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.763 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.763 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:04.763 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215457+0000", 2024-03-19T11:32:04.764 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3139000000000001e-05 2024-03-19T11:32:04.764 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.764 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.764 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:04.764 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.234414+0000", 2024-03-19T11:32:04.764 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018957108 2024-03-19T11:32:04.765 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.765 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.765 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:04.765 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.234470+0000", 2024-03-19T11:32:04.765 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6274999999999999e-05 2024-03-19T11:32:04.765 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.766 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.766 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:04.766 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.241943+0000", 2024-03-19T11:32:04.766 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0074728770000000002 2024-03-19T11:32:04.766 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.766 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.767 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:04.767 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.242045+0000", 2024-03-19T11:32:04.767 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000102497 2024-03-19T11:32:04.767 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.767 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:04.767 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.768 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.768 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.768 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.5.0:241 4.8 e108/100 4:15708482:::smithi03829856-23:10 v 108'127, mlcod=103'122)", 2024-03-19T11:32:04.768 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.217180+0000", 2024-03-19T11:32:04.768 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 37.445786513000002, 2024-03-19T11:32:04.768 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.025026023000000001, 2024-03-19T11:32:04.768 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:04.769 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:04.769 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:04.769 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.769 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:04.769 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.217180+0000", 2024-03-19T11:32:04.769 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.770 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.770 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.770 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:04.770 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.217180+0000", 2024-03-19T11:32:04.770 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.770 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.771 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:04.771 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.217182+0000", 2024-03-19T11:32:04.771 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.12e-06 2024-03-19T11:32:04.771 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.771 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.772 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:04.772 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.217200+0000", 2024-03-19T11:32:04.772 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7677999999999999e-05 2024-03-19T11:32:04.772 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.772 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.772 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:04.772 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.217246+0000", 2024-03-19T11:32:04.773 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6053000000000001e-05 2024-03-19T11:32:04.773 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.773 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.773 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:04.773 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.217266+0000", 2024-03-19T11:32:04.773 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0085000000000002e-05 2024-03-19T11:32:04.774 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.774 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.774 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:04.774 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.226876+0000", 2024-03-19T11:32:04.774 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0096094389999999991 2024-03-19T11:32:04.774 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.775 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:04.775 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.226932+0000", 2024-03-19T11:32:04.775 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6234999999999999e-05 2024-03-19T11:32:04.775 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.775 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.776 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:04.776 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.242063+0000", 2024-03-19T11:32:04.776 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015131083 2024-03-19T11:32:04.776 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.776 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.776 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:04.777 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.242206+0000", 2024-03-19T11:32:04.777 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014333 2024-03-19T11:32:04.777 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.777 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:04.777 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.777 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.777 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.778 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.1.0:279 4.24 e108/83 4:27909373:::smithi03829856-46:c v 108'129, mlcod=100'124)", 2024-03-19T11:32:04.778 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.223035+0000", 2024-03-19T11:32:04.778 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 37.439932171999999, 2024-03-19T11:32:04.778 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019488583, 2024-03-19T11:32:04.778 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:04.778 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:04.778 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:04.778 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.778 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:04.778 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.223035+0000", 2024-03-19T11:32:04.779 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.779 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:04.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.223035+0000", 2024-03-19T11:32:04.779 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:04.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.779 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:04.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.223037+0000", 2024-03-19T11:32:04.779 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3829999999999999e-06 2024-03-19T11:32:04.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.779 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.779 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:04.779 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.223073+0000", 2024-03-19T11:32:04.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6306000000000001e-05 2024-03-19T11:32:04.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.780 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:04.780 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.223095+0000", 2024-03-19T11:32:04.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1738000000000001e-05 2024-03-19T11:32:04.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.780 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:04.780 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.223107+0000", 2024-03-19T11:32:04.780 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2082000000000001e-05 2024-03-19T11:32:04.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.780 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.780 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:04.780 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.226145+0000", 2024-03-19T11:32:04.781 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003038022 2024-03-19T11:32:04.781 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.781 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.781 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:04.781 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.226201+0000", 2024-03-19T11:32:04.781 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6107999999999999e-05 2024-03-19T11:32:04.781 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.781 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.781 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:04.781 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.242322+0000", 2024-03-19T11:32:04.781 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016120196 2024-03-19T11:32:04.781 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:04.781 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:04.781 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:04.781 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.242523+0000", 2024-03-19T11:32:04.782 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020174799999999999 2024-03-19T11:32:04.782 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.782 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:04.782 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.782 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:04.782 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:04.782 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:32:04.783 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-19T11:32:04.784 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:32:04.791+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:32:04.884 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:32:04.891+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:32:04.921 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:32:04.921 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:32:04.921 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:32:04.922 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:32:04.936 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-19T11:32:04.985 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:32:04.993+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:32:05.081 INFO:tasks.rados.rados.0.smithi038.stdout:1817: done (4 left) 2024-03-19T11:32:05.081 INFO:tasks.rados.rados.0.smithi038.stdout:1818: write (excl) oid 32 current snap is 40 2024-03-19T11:32:05.082 INFO:tasks.rados.rados.0.smithi038.stdout:1814: got (16) Device or resource busy 2024-03-19T11:32:05.082 INFO:tasks.rados.rados.0.smithi038.stdout:check_oldest_snap_flushed oid 9 v 132 (ObjNum 308 snap 39 seq_num 308) unflushed 40 2024-03-19T11:32:05.082 INFO:tasks.rados.rados.0.smithi038.stdout:1813: finishing write tid 3 to smithi03829856-21 2024-03-19T11:32:05.082 INFO:tasks.rados.rados.0.smithi038.stdout:1813: finishing write tid 4 to smithi03829856-21 2024-03-19T11:32:05.082 INFO:tasks.rados.rados.0.smithi038.stdout:1818: seq_num 342 ranges {547994=454690,1746724=482104,2659134=465146,3605976=142018} 2024-03-19T11:32:05.084 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:32:05.091+0000 7f6a7d9d6640 -1 osd.1 143 handle_osd_map injecting map crc failure 2024-03-19T11:32:05.086 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:32:05.095+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:32:05.094 INFO:tasks.rados.rados.0.smithi038.stdout:1818: writing smithi03829856-32 from 547994 to 1002684 tid 1 2024-03-19T11:32:05.102 INFO:tasks.rados.rados.0.smithi038.stdout:1818: writing smithi03829856-32 from 1746724 to 2228828 tid 2 2024-03-19T11:32:05.109 INFO:tasks.rados.rados.0.smithi038.stdout:1818: writing smithi03829856-32 from 2659134 to 3124280 tid 3 2024-03-19T11:32:05.113 INFO:tasks.rados.rados.0.smithi038.stdout:1818: writing smithi03829856-32 from 3605976 to 3747994 tid 4 2024-03-19T11:32:05.113 INFO:tasks.rados.rados.0.smithi038.stdout:1814: done (4 left) 2024-03-19T11:32:05.113 INFO:tasks.rados.rados.0.smithi038.stdout:1819: write (excl) oid 46 current snap is 40 2024-03-19T11:32:05.113 INFO:tasks.rados.rados.0.smithi038.stdout:1819: seq_num 343 ranges {564801=646292,1921559=742115,3202910=561891} 2024-03-19T11:32:05.119 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:32:05.119 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:32:05.119 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:32:05.119 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:32:05.119 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:32:05.121 INFO:tasks.rados.rados.0.smithi038.stdout:1819: writing smithi03829856-46 from 564801 to 1211093 tid 1 2024-03-19T11:32:05.130 INFO:tasks.rados.rados.0.smithi038.stdout:1819: writing smithi03829856-46 from 1921559 to 2663674 tid 2 2024-03-19T11:32:05.138 INFO:tasks.rados.rados.0.smithi038.stdout:1819: writing smithi03829856-46 from 3202910 to 3764801 tid 3 2024-03-19T11:32:05.138 INFO:tasks.rados.rados.0.smithi038.stdout:1820: write (excl) oid 19 current snap is 40 2024-03-19T11:32:05.138 INFO:tasks.rados.rados.0.smithi038.stdout:1813: finishing write tid 5 to smithi03829856-21 2024-03-19T11:32:05.138 INFO:tasks.rados.rados.0.smithi038.stdout:1813: oid 21 updating version 0 to 330 2024-03-19T11:32:05.139 INFO:tasks.rados.rados.0.smithi038.stdout:1813: oid 21 version 330 is already newer than 329 2024-03-19T11:32:05.139 INFO:tasks.rados.rados.0.smithi038.stdout:1813: oid 21 version 330 is already newer than 327 2024-03-19T11:32:05.139 INFO:tasks.rados.rados.0.smithi038.stdout:1813: oid 21 version 330 is already newer than 328 2024-03-19T11:32:05.139 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 21 v 330 (ObjNum 339 snap 40 seq_num 339) dirty exists 2024-03-19T11:32:05.139 INFO:tasks.rados.rados.0.smithi038.stdout:1813: left oid 21 (ObjNum 339 snap 40 seq_num 339) 2024-03-19T11:32:05.139 INFO:tasks.rados.rados.0.smithi038.stdout:1815: finishing write tid 1 to smithi03829856-40 2024-03-19T11:32:05.139 INFO:tasks.rados.rados.0.smithi038.stdout:1815: finishing write tid 2 to smithi03829856-40 2024-03-19T11:32:05.139 INFO:tasks.rados.rados.0.smithi038.stdout:1815: finishing write tid 3 to smithi03829856-40 2024-03-19T11:32:05.139 INFO:tasks.rados.rados.0.smithi038.stdout:1815: finishing write tid 4 to smithi03829856-40 2024-03-19T11:32:05.139 INFO:tasks.rados.rados.0.smithi038.stdout:1815: finishing write tid 5 to smithi03829856-40 2024-03-19T11:32:05.139 INFO:tasks.rados.rados.0.smithi038.stdout:1815: oid 40 updating version 0 to 101 2024-03-19T11:32:05.139 INFO:tasks.rados.rados.0.smithi038.stdout:1815: oid 40 updating version 101 to 103 2024-03-19T11:32:05.139 INFO:tasks.rados.rados.0.smithi038.stdout:1815: oid 40 version 103 is already newer than 102 2024-03-19T11:32:05.139 INFO:tasks.rados.rados.0.smithi038.stdout:1815: oid 40 updating version 103 to 104 2024-03-19T11:32:05.140 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 40 v 104 (ObjNum 340 snap 40 seq_num 340) dirty exists 2024-03-19T11:32:05.140 INFO:tasks.rados.rados.0.smithi038.stdout:1815: left oid 40 (ObjNum 340 snap 40 seq_num 340) 2024-03-19T11:32:05.140 INFO:tasks.rados.rados.0.smithi038.stdout:1820: seq_num 344 ranges {581608=721541,1863688=485773,3113980=604043} 2024-03-19T11:32:05.141 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-19T11:32:05.145 INFO:tasks.rados.rados.0.smithi038.stdout:1820: writing smithi03829856-19 from 581608 to 1303149 tid 1 2024-03-19T11:32:05.150 INFO:tasks.rados.rados.0.smithi038.stdout:1820: writing smithi03829856-19 from 1863688 to 2349461 tid 2 2024-03-19T11:32:05.158 INFO:tasks.rados.rados.0.smithi038.stdout:1820: writing smithi03829856-19 from 3113980 to 3718023 tid 3 2024-03-19T11:32:05.158 INFO:tasks.rados.rados.0.smithi038.stdout:1813: done (5 left) 2024-03-19T11:32:05.158 INFO:tasks.rados.rados.0.smithi038.stdout:1815: done (4 left) 2024-03-19T11:32:05.158 INFO:tasks.rados.rados.0.smithi038.stdout:1821: set_chunk oid 23 target oid 2024-03-19T11:32:05.182 INFO:tasks.rados.rados.0.smithi038.stdout:1821: set_chunk oid 23 offset: 174080 length: 1800192 target oid 32887202f721310eb181669a200ec19eb31345a839889e9cc986bf6c72e76d13 offset: 0 snap -1 2024-03-19T11:32:05.182 INFO:tasks.rados.rados.0.smithi038.stdout:1822: read oid 20 snap -1 2024-03-19T11:32:05.182 INFO:tasks.rados.rados.0.smithi038.stdout:1822: expect (ObjNum 299 snap 39 seq_num 299) 2024-03-19T11:32:05.182 INFO:tasks.rados.rados.0.smithi038.stdout:1823: snap_remove snap 35 2024-03-19T11:32:05.187 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:32:05.193+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:32:05.277 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:32:05.277 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:32:05.277 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:32:05.278 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:32:05.278 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.278 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1217 4.3 4:c8b09e87:::smithi03829856-15:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:32:05.278 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.155966+0000", 2024-03-19T11:32:05.278 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 107.127753665, 2024-03-19T11:32:05.278 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.023984647000000001, 2024-03-19T11:32:05.278 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:05.278 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:05.278 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:05.278 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:05.278 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:05.278 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1217 2024-03-19T11:32:05.278 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.278 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:05.279 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.279 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:05.279 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155966+0000", 2024-03-19T11:32:05.279 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.279 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.279 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.279 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:05.279 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155966+0000", 2024-03-19T11:32:05.279 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.279 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.279 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.279 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:05.279 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155968+0000", 2024-03-19T11:32:05.279 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.795e-06 2024-03-19T11:32:05.280 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.280 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.280 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:05.280 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155978+0000", 2024-03-19T11:32:05.280 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0574000000000001e-05 2024-03-19T11:32:05.280 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.280 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.280 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:05.280 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.156000+0000", 2024-03-19T11:32:05.280 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1250000000000002e-05 2024-03-19T11:32:05.280 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.280 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.280 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:05.280 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.156010+0000", 2024-03-19T11:32:05.281 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0168e-05 2024-03-19T11:32:05.281 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.281 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.281 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.281 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.156119+0000", 2024-03-19T11:32:05.281 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010950699999999999 2024-03-19T11:32:05.281 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.281 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.281 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:05.281 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.156231+0000", 2024-03-19T11:32:05.281 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000111445 2024-03-19T11:32:05.281 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.281 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.281 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:05.283 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.179951+0000", 2024-03-19T11:32:05.283 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.023719908000000001 2024-03-19T11:32:05.283 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.283 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:05.283 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.284 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.284 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.284 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.24 75/74 [PushOp(4:27909373:::smithi03829856-46:8, version: 46'59, data_included: [756315~767970,1933572~280606], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:8@46'59, size: 2756315, copy_subset: [(0, 2756315)], clone_subset: {}, snapset: 10=[]:{8: [6], c: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2214178, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:266590a5:::smithi03829856-16:e, version: 70'107, data_included: [652105~662359,1728171~386217], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:e@70'107, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 10=[]:{e: [d, b, 6]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:266590a5:::smithi03829856-16:head, version: 70'108, data_included: [652105~662359,1728171~386217], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:head@70'108, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:27909373:::smithi03829856-46:c, version: 70'109, data_included: [628595~895690,1807440~152886], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:c@70'109, size: 3428595, copy_subset: [(0, 3428595)], clone_subset: {}, snapset: 10=[]:{8: [6], c: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1960326, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:27909373:::smithi03829856-46:head, version: 70'110, data_included: [403401~454961,1584925~506039], data_size: 961000, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:head@70'110, size: 2403401, copy_subset: [(0, 2403401)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2403401, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:32:05.284 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.099395+0000", 2024-03-19T11:32:05.284 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 71.184324504000003, 2024-03-19T11:32:05.284 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.045391157000000001, 2024-03-19T11:32:05.284 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:05.284 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:05.284 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:05.284 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.284 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:05.284 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.099395+0000", 2024-03-19T11:32:05.284 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.284 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.285 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.285 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:05.285 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.099395+0000", 2024-03-19T11:32:05.285 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.285 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.285 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.285 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:05.285 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.099397+0000", 2024-03-19T11:32:05.285 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.691e-06 2024-03-19T11:32:05.285 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.285 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.285 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:05.285 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105009+0000", 2024-03-19T11:32:05.285 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0056121349999999999 2024-03-19T11:32:05.285 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.286 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.286 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:05.286 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105077+0000", 2024-03-19T11:32:05.286 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8561999999999998e-05 2024-03-19T11:32:05.286 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.286 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.286 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:05.286 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105133+0000", 2024-03-19T11:32:05.286 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5590000000000001e-05 2024-03-19T11:32:05.286 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.286 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.286 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.286 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105531+0000", 2024-03-19T11:32:05.286 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00039756200000000001 2024-03-19T11:32:05.287 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.287 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.287 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:05.287 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105579+0000", 2024-03-19T11:32:05.287 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8028999999999997e-05 2024-03-19T11:32:05.287 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.287 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.287 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:05.287 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.144786+0000", 2024-03-19T11:32:05.287 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.039207588000000002 2024-03-19T11:32:05.287 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.287 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:05.287 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.287 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.288 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.288 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.24 e75,74 4:266590a5:::smithi03829856-16:4,47'63,4:27909373:::smithi03829856-46:5,55'79,4:266590a5:::smithi03829856-16:10,67'105)", 2024-03-19T11:32:05.288 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.105191+0000", 2024-03-19T11:32:05.288 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 71.178528404000005, 2024-03-19T11:32:05.288 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.039580830999999997, 2024-03-19T11:32:05.288 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:05.288 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:05.288 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:05.288 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.288 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:05.288 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105191+0000", 2024-03-19T11:32:05.288 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.288 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.288 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.288 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:05.289 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105191+0000", 2024-03-19T11:32:05.289 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.289 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.289 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.289 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:05.289 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105192+0000", 2024-03-19T11:32:05.289 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.099e-06 2024-03-19T11:32:05.289 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.289 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.289 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:05.289 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105200+0000", 2024-03-19T11:32:05.289 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5279999999999998e-06 2024-03-19T11:32:05.289 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.289 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.289 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:05.290 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105211+0000", 2024-03-19T11:32:05.290 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1471000000000001e-05 2024-03-19T11:32:05.290 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.290 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.290 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:05.290 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105220+0000", 2024-03-19T11:32:05.290 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3469999999999999e-06 2024-03-19T11:32:05.290 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.290 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.290 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.290 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.144219+0000", 2024-03-19T11:32:05.290 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.038999024 2024-03-19T11:32:05.290 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.290 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.290 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:05.291 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.144257+0000", 2024-03-19T11:32:05.291 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8501999999999998e-05 2024-03-19T11:32:05.291 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.291 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.291 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:05.291 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.144772+0000", 2024-03-19T11:32:05.291 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00051486000000000001 2024-03-19T11:32:05.291 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.291 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:05.291 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.291 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.291 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.291 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 75/74 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:7, version: 55'82, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@55'82, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 58'86, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@58'86, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [504210~968657,1572695~79919], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:12, version: 66'101, data_included: [440350~816629,1309629~231947], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:12@66'101, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:10, version: 70'112, data_included: [440350~816629,1309629~231947], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:10@70'112, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 70'113, data_included: [440350~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@70'113, size: 2605085, copy_subset: [(0, 2605085)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1488926, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:32:05.291 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.107266+0000", 2024-03-19T11:32:05.292 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 71.176453886999994, 2024-03-19T11:32:05.292 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.048592669999999998, 2024-03-19T11:32:05.292 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:05.292 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:05.292 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:05.292 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.292 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:05.292 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.107266+0000", 2024-03-19T11:32:05.292 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.292 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.292 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.292 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:05.292 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.107266+0000", 2024-03-19T11:32:05.292 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.293 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.293 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.293 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:05.293 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.107267+0000", 2024-03-19T11:32:05.293 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1459999999999999e-06 2024-03-19T11:32:05.293 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.293 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.293 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:05.293 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.122861+0000", 2024-03-19T11:32:05.294 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01559372 2024-03-19T11:32:05.294 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.294 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.294 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:05.294 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.122932+0000", 2024-03-19T11:32:05.294 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1477999999999996e-05 2024-03-19T11:32:05.294 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.294 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.294 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:05.294 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.122999+0000", 2024-03-19T11:32:05.294 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6832e-05 2024-03-19T11:32:05.294 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.294 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.294 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.295 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123332+0000", 2024-03-19T11:32:05.295 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000333271 2024-03-19T11:32:05.295 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.295 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.295 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:05.295 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123371+0000", 2024-03-19T11:32:05.295 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8525999999999997e-05 2024-03-19T11:32:05.295 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.295 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.295 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:05.295 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.155858+0000", 2024-03-19T11:32:05.295 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.032487697000000003 2024-03-19T11:32:05.295 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.295 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:05.295 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.296 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.296 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.296 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.21 e75,74 4:8401c1f0:::smithi03829856-43:8,46'66,4:866f3dcc:::smithi03829856-31:2,47'73)", 2024-03-19T11:32:05.296 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.123075+0000", 2024-03-19T11:32:05.296 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 71.160644668000003, 2024-03-19T11:32:05.296 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.033023045000000001, 2024-03-19T11:32:05.296 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:05.296 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:05.296 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:05.296 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.296 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:05.296 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123075+0000", 2024-03-19T11:32:05.296 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.296 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.296 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.297 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:05.297 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123075+0000", 2024-03-19T11:32:05.297 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.297 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.297 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.297 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:05.297 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123076+0000", 2024-03-19T11:32:05.297 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5e-06 2024-03-19T11:32:05.297 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.297 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.297 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:05.298 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123086+0000", 2024-03-19T11:32:05.298 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8060000000000006e-06 2024-03-19T11:32:05.298 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.298 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.298 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:05.298 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123099+0000", 2024-03-19T11:32:05.298 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3251000000000001e-05 2024-03-19T11:32:05.298 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.298 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.298 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:05.298 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123112+0000", 2024-03-19T11:32:05.298 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2774e-05 2024-03-19T11:32:05.298 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.298 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.298 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.299 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.155902+0000", 2024-03-19T11:32:05.299 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.032790225999999999 2024-03-19T11:32:05.299 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.299 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.299 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:05.299 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.155914+0000", 2024-03-19T11:32:05.299 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1476e-05 2024-03-19T11:32:05.299 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.299 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.299 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:05.299 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.156098+0000", 2024-03-19T11:32:05.299 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000184012 2024-03-19T11:32:05.299 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.299 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:05.299 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.300 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.300 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.300 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 75/74 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:7, version: 55'82, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@55'82, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 58'86, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@58'86, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [1652614~973634,2888511~52681,3182305~22261], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3204566, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:12, version: 66'101, data_included: [1541576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:12@66'101, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:10, version: 70'112, data_included: [1541576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:10@70'112, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 70'113, data_included: [1488926~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@70'113, size: 2605085, copy_subset: [(0, 2605085)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2537502, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1488926, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:32:05.300 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.174030+0000", 2024-03-19T11:32:05.300 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 71.109689463999999, 2024-03-19T11:32:05.300 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.040799275000000003, 2024-03-19T11:32:05.300 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:05.300 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:05.300 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:05.300 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.300 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:05.300 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.174030+0000", 2024-03-19T11:32:05.301 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.301 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.301 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.301 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:05.301 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.174030+0000", 2024-03-19T11:32:05.301 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.301 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.301 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.301 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:05.301 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.174031+0000", 2024-03-19T11:32:05.302 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.322e-06 2024-03-19T11:32:05.302 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.302 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.302 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:05.302 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.184771+0000", 2024-03-19T11:32:05.302 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010739227 2024-03-19T11:32:05.302 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.302 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.302 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:05.302 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.184872+0000", 2024-03-19T11:32:05.302 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000101234 2024-03-19T11:32:05.302 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.302 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.302 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:05.302 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.184985+0000", 2024-03-19T11:32:05.303 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000113545 2024-03-19T11:32:05.303 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.303 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.303 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.303 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.185463+0000", 2024-03-19T11:32:05.303 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00047749899999999999 2024-03-19T11:32:05.303 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.303 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.303 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:05.303 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.185534+0000", 2024-03-19T11:32:05.303 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1007999999999995e-05 2024-03-19T11:32:05.303 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.303 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.303 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:05.304 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.214829+0000", 2024-03-19T11:32:05.304 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029295439999999999 2024-03-19T11:32:05.304 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.304 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:05.304 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.304 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.304 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.305 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2372 4.16 4:6868af91:::smithi03829856-10:b [read 832512~818688] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-03-19T11:32:05.305 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:58.455732+0000", 2024-03-19T11:32:05.305 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 66.827987749000002, 2024-03-19T11:32:05.305 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.69150541200000004, 2024-03-19T11:32:05.305 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:05.305 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:05.305 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:05.305 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:05.305 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:05.305 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2372 2024-03-19T11:32:05.305 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.305 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:05.305 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.305 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:05.305 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455732+0000", 2024-03-19T11:32:05.306 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.306 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.306 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.306 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:05.306 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455732+0000", 2024-03-19T11:32:05.306 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.306 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.306 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.306 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:05.306 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455735+0000", 2024-03-19T11:32:05.306 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.756e-06 2024-03-19T11:32:05.306 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.306 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.306 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:05.306 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455744+0000", 2024-03-19T11:32:05.307 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1880000000000008e-06 2024-03-19T11:32:05.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.307 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.307 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:05.307 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455764+0000", 2024-03-19T11:32:05.307 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0044999999999999e-05 2024-03-19T11:32:05.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.307 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.307 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:05.307 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455774+0000", 2024-03-19T11:32:05.307 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8770000000000005e-06 2024-03-19T11:32:05.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.307 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.307 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.307 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455837+0000", 2024-03-19T11:32:05.308 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3588000000000002e-05 2024-03-19T11:32:05.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.308 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.308 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:05.308 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455846+0000", 2024-03-19T11:32:05.308 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6030000000000005e-06 2024-03-19T11:32:05.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.308 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.308 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.308 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.076525+0000", 2024-03-19T11:32:05.309 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.62067875900000002 2024-03-19T11:32:05.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.309 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.309 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:32:05.309 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.076981+0000", 2024-03-19T11:32:05.309 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00045633300000000001 2024-03-19T11:32:05.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.309 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.309 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.309 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.104422+0000", 2024-03-19T11:32:05.309 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027440665999999999 2024-03-19T11:32:05.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.310 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.310 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:32:05.310 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.104786+0000", 2024-03-19T11:32:05.310 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00036471499999999999 2024-03-19T11:32:05.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.311 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.311 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.311 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.140725+0000", 2024-03-19T11:32:05.311 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.035939093999999998 2024-03-19T11:32:05.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.311 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.311 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:32:05.312 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.140841+0000", 2024-03-19T11:32:05.312 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000115207 2024-03-19T11:32:05.312 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.312 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.312 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.312 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.145559+0000", 2024-03-19T11:32:05.312 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.004718486 2024-03-19T11:32:05.312 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.312 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.312 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:05.313 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.145760+0000", 2024-03-19T11:32:05.313 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000201058 2024-03-19T11:32:05.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.313 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.313 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:05.313 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.147237+0000", 2024-03-19T11:32:05.313 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014770370000000001 2024-03-19T11:32:05.313 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.313 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:05.313 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.313 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.313 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.4.13:82 3.3 3:f6b6d21a:::smithi03829856-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-03-19T11:32:05.313 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:04.069448+0000", 2024-03-19T11:32:05.314 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 61.214271963999998, 2024-03-19T11:32:05.314 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.086427773999999999, 2024-03-19T11:32:05.314 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:05.314 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:05.314 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:05.314 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.314 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:05.314 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069448+0000", 2024-03-19T11:32:05.314 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.314 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.314 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.314 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:05.314 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069448+0000", 2024-03-19T11:32:05.314 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.315 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.315 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.315 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:05.315 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069449+0000", 2024-03-19T11:32:05.315 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8160000000000001e-06 2024-03-19T11:32:05.315 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.315 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.315 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:05.315 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069458+0000", 2024-03-19T11:32:05.315 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2900000000000002e-06 2024-03-19T11:32:05.315 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.315 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.315 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:05.315 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069491+0000", 2024-03-19T11:32:05.316 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3229000000000003e-05 2024-03-19T11:32:05.316 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.316 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.316 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:05.316 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069501+0000", 2024-03-19T11:32:05.316 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7629999999999995e-06 2024-03-19T11:32:05.316 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.316 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.316 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.316 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069573+0000", 2024-03-19T11:32:05.316 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2349999999999997e-05 2024-03-19T11:32:05.316 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.316 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.316 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:05.317 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069581+0000", 2024-03-19T11:32:05.317 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.388e-06 2024-03-19T11:32:05.317 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.317 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.317 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.317 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.075451+0000", 2024-03-19T11:32:05.317 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005869956 2024-03-19T11:32:05.317 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.317 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.317 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:32:05.317 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.077511+0000", 2024-03-19T11:32:05.317 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0020592610000000002 2024-03-19T11:32:05.317 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.317 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.317 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.318 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.108781+0000", 2024-03-19T11:32:05.318 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.031270344999999998 2024-03-19T11:32:05.318 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.318 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.318 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:05.318 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.108895+0000", 2024-03-19T11:32:05.318 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000114059 2024-03-19T11:32:05.318 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.318 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.318 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4", 2024-03-19T11:32:05.318 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.109065+0000", 2024-03-19T11:32:05.318 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016979899999999999 2024-03-19T11:32:05.318 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.319 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.319 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:32:05.319 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.111730+0000", 2024-03-19T11:32:05.319 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0026651410000000002 2024-03-19T11:32:05.319 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.319 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.319 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:05.319 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.155616+0000", 2024-03-19T11:32:05.319 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.043885986000000002 2024-03-19T11:32:05.319 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.319 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.320 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:32:05.320 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.155745+0000", 2024-03-19T11:32:05.320 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012948899999999999 2024-03-19T11:32:05.320 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.320 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.320 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:05.320 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.155875+0000", 2024-03-19T11:32:05.320 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012990199999999999 2024-03-19T11:32:05.321 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.321 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:05.321 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.321 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.321 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.321 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.4.13:83 3.3 3:f6b6d21a:::smithi03829856-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-03-19T11:32:05.321 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:04.069526+0000", 2024-03-19T11:32:05.321 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 61.214193184000003, 2024-03-19T11:32:05.322 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.087140880000000004, 2024-03-19T11:32:05.322 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:05.322 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:05.322 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:05.322 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.322 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:05.322 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069526+0000", 2024-03-19T11:32:05.323 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.323 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.323 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.323 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:05.323 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069526+0000", 2024-03-19T11:32:05.323 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.324 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.324 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.324 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:05.324 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069527+0000", 2024-03-19T11:32:05.324 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2799999999999996e-07 2024-03-19T11:32:05.324 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.324 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.325 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:05.325 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069531+0000", 2024-03-19T11:32:05.325 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1489999999999996e-06 2024-03-19T11:32:05.325 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.325 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.325 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:05.325 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069539+0000", 2024-03-19T11:32:05.326 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3189999999999995e-06 2024-03-19T11:32:05.326 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.326 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.326 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:05.326 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069545+0000", 2024-03-19T11:32:05.326 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.592e-06 2024-03-19T11:32:05.326 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.327 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.327 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.327 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069616+0000", 2024-03-19T11:32:05.327 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1130000000000005e-05 2024-03-19T11:32:05.327 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.327 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.327 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:05.327 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069628+0000", 2024-03-19T11:32:05.328 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1697e-05 2024-03-19T11:32:05.328 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.328 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.328 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.328 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.077594+0000", 2024-03-19T11:32:05.328 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0079656989999999997 2024-03-19T11:32:05.328 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.329 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.329 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:32:05.329 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.077674+0000", 2024-03-19T11:32:05.329 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0338000000000002e-05 2024-03-19T11:32:05.329 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.329 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.329 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.330 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.112771+0000", 2024-03-19T11:32:05.330 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.035097325999999998 2024-03-19T11:32:05.330 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.330 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.330 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:05.330 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.112882+0000", 2024-03-19T11:32:05.331 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000110949 2024-03-19T11:32:05.331 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.331 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.331 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4", 2024-03-19T11:32:05.331 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.113031+0000", 2024-03-19T11:32:05.331 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000148355 2024-03-19T11:32:05.331 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.331 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.331 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:32:05.331 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.114076+0000", 2024-03-19T11:32:05.332 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001045009 2024-03-19T11:32:05.332 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.332 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.332 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:05.332 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156482+0000", 2024-03-19T11:32:05.332 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.042406354 2024-03-19T11:32:05.332 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.332 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.332 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:32:05.332 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156549+0000", 2024-03-19T11:32:05.332 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7229000000000003e-05 2024-03-19T11:32:05.333 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.333 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.333 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:05.333 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156667+0000", 2024-03-19T11:32:05.333 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000118106 2024-03-19T11:32:05.333 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.333 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:05.333 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.333 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.333 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.333 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.4.13:84 3.3 3:ce6951d3:::smithi03829856-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-03-19T11:32:05.333 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:04.071995+0000", 2024-03-19T11:32:05.333 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 61.211724408999999, 2024-03-19T11:32:05.333 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.084172497999999998, 2024-03-19T11:32:05.334 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:05.334 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:05.334 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:05.334 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.334 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:05.334 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.071995+0000", 2024-03-19T11:32:05.334 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.334 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.334 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.334 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:05.334 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.071995+0000", 2024-03-19T11:32:05.334 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.334 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.335 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.335 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:05.335 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.071997+0000", 2024-03-19T11:32:05.335 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6029999999999999e-06 2024-03-19T11:32:05.335 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.335 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.335 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:05.335 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072004+0000", 2024-03-19T11:32:05.335 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8600000000000004e-06 2024-03-19T11:32:05.335 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.335 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.335 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:05.335 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072033+0000", 2024-03-19T11:32:05.335 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9836999999999999e-05 2024-03-19T11:32:05.335 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.336 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.336 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:05.336 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072042+0000", 2024-03-19T11:32:05.336 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9279999999999999e-06 2024-03-19T11:32:05.336 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.336 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.336 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.336 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072123+0000", 2024-03-19T11:32:05.336 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0630000000000006e-05 2024-03-19T11:32:05.336 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.336 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.336 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:05.337 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072132+0000", 2024-03-19T11:32:05.337 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5560000000000008e-06 2024-03-19T11:32:05.337 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.337 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.337 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.337 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.077712+0000", 2024-03-19T11:32:05.337 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055804970000000002 2024-03-19T11:32:05.337 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.338 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.338 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:32:05.338 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.079495+0000", 2024-03-19T11:32:05.338 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017831850000000001 2024-03-19T11:32:05.338 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.338 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.338 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.339 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.109825+0000", 2024-03-19T11:32:05.339 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.030329819000000001 2024-03-19T11:32:05.339 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.339 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.339 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:05.339 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.109904+0000", 2024-03-19T11:32:05.340 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9269000000000003e-05 2024-03-19T11:32:05.340 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.340 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.340 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4", 2024-03-19T11:32:05.340 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.110085+0000", 2024-03-19T11:32:05.340 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00018018600000000001 2024-03-19T11:32:05.340 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.341 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.341 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:32:05.341 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.113581+0000", 2024-03-19T11:32:05.341 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034962919999999998 2024-03-19T11:32:05.341 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.341 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.341 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:05.341 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.155925+0000", 2024-03-19T11:32:05.341 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.042344404000000002 2024-03-19T11:32:05.342 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.342 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.342 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:32:05.342 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156026+0000", 2024-03-19T11:32:05.342 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000100599 2024-03-19T11:32:05.342 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.342 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.342 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:05.343 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156168+0000", 2024-03-19T11:32:05.343 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014183300000000001 2024-03-19T11:32:05.343 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.343 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:05.343 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.343 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.343 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.343 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.4.13:85 3.3 3:ce6951d3:::smithi03829856-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-03-19T11:32:05.343 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:04.072068+0000", 2024-03-19T11:32:05.343 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 61.211651773, 2024-03-19T11:32:05.343 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.084368466000000003, 2024-03-19T11:32:05.344 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:05.344 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:05.344 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:05.344 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.344 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:05.344 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072068+0000", 2024-03-19T11:32:05.344 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.344 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.344 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.344 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:05.344 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072068+0000", 2024-03-19T11:32:05.344 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.344 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.344 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.344 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:05.345 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072068+0000", 2024-03-19T11:32:05.345 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9699999999999996e-07 2024-03-19T11:32:05.345 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.345 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.345 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:05.345 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072074+0000", 2024-03-19T11:32:05.345 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2279999999999998e-06 2024-03-19T11:32:05.345 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.345 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.345 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:05.345 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072140+0000", 2024-03-19T11:32:05.345 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6638999999999997e-05 2024-03-19T11:32:05.345 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.345 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.345 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:05.346 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072147+0000", 2024-03-19T11:32:05.346 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8750000000000002e-06 2024-03-19T11:32:05.346 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.346 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.346 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.346 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072201+0000", 2024-03-19T11:32:05.346 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3634000000000002e-05 2024-03-19T11:32:05.346 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.346 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.346 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:05.346 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072211+0000", 2024-03-19T11:32:05.346 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0589e-05 2024-03-19T11:32:05.346 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.347 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.347 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.347 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.079687+0000", 2024-03-19T11:32:05.347 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074757219999999997 2024-03-19T11:32:05.347 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.347 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.347 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:32:05.347 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.079769+0000", 2024-03-19T11:32:05.347 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2251999999999998e-05 2024-03-19T11:32:05.347 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.347 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.347 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.347 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.111798+0000", 2024-03-19T11:32:05.347 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.032028889999999997 2024-03-19T11:32:05.347 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.348 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.348 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:05.348 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.111902+0000", 2024-03-19T11:32:05.348 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000103773 2024-03-19T11:32:05.348 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.348 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.348 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4", 2024-03-19T11:32:05.348 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.112073+0000", 2024-03-19T11:32:05.348 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017102799999999999 2024-03-19T11:32:05.348 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.348 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.348 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:32:05.348 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.113591+0000", 2024-03-19T11:32:05.348 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0015179569999999999 2024-03-19T11:32:05.349 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.349 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.349 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:05.349 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156223+0000", 2024-03-19T11:32:05.349 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.042632153999999998 2024-03-19T11:32:05.349 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.349 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.349 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:32:05.349 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156321+0000", 2024-03-19T11:32:05.349 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8246999999999996e-05 2024-03-19T11:32:05.349 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.349 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.349 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:05.349 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156436+0000", 2024-03-19T11:32:05.350 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000114881 2024-03-19T11:32:05.350 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.350 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:05.350 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.350 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.350 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.350 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3203 4.6 4:626acd4d:::smithi03829856-2:1f [sparse-read 0~1665607,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108)", 2024-03-19T11:32:05.350 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:27.253945+0000", 2024-03-19T11:32:05.350 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 38.029774705999998, 2024-03-19T11:32:05.350 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.83307735599999999, 2024-03-19T11:32:05.350 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:05.350 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:05.350 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:05.350 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:05.350 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:05.351 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3203 2024-03-19T11:32:05.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.351 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:05.351 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.351 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:05.351 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.253945+0000", 2024-03-19T11:32:05.351 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.351 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.351 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:05.351 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.253945+0000", 2024-03-19T11:32:05.351 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.351 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.352 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:05.352 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.253947+0000", 2024-03-19T11:32:05.352 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5979999999999999e-06 2024-03-19T11:32:05.352 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.352 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.352 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:05.352 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.253957+0000", 2024-03-19T11:32:05.352 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7699999999999996e-06 2024-03-19T11:32:05.352 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.352 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.352 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:05.352 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.253980+0000", 2024-03-19T11:32:05.352 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2466000000000001e-05 2024-03-19T11:32:05.352 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.352 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.353 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:05.353 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.253992+0000", 2024-03-19T11:32:05.353 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1985e-05 2024-03-19T11:32:05.353 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.353 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.353 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.353 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254071+0000", 2024-03-19T11:32:05.353 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8869000000000007e-05 2024-03-19T11:32:05.353 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.353 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.353 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:05.353 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254084+0000", 2024-03-19T11:32:05.353 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3716999999999999e-05 2024-03-19T11:32:05.353 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.353 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.354 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.354 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.086473+0000", 2024-03-19T11:32:05.354 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.83238890300000001 2024-03-19T11:32:05.354 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.354 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.354 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:05.354 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.086618+0000", 2024-03-19T11:32:05.354 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014479200000000001 2024-03-19T11:32:05.354 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.354 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.354 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:05.354 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.087022+0000", 2024-03-19T11:32:05.354 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00040425599999999997 2024-03-19T11:32:05.354 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.354 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:05.355 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.355 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.355 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.355 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3204 4.6 4:626acd4d:::smithi03829856-2:1f [read 0~3462242,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108)", 2024-03-19T11:32:05.355 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:27.254015+0000", 2024-03-19T11:32:05.355 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 38.029704924999997, 2024-03-19T11:32:05.355 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.83578705200000003, 2024-03-19T11:32:05.355 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:05.355 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:05.355 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:05.355 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:05.355 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:05.355 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3204 2024-03-19T11:32:05.355 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.355 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:05.356 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.356 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:05.356 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254015+0000", 2024-03-19T11:32:05.356 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.356 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.356 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.356 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:05.356 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254015+0000", 2024-03-19T11:32:05.356 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.356 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.356 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.356 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:05.356 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254015+0000", 2024-03-19T11:32:05.356 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8599999999999997e-07 2024-03-19T11:32:05.356 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.357 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.357 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:05.357 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254022+0000", 2024-03-19T11:32:05.357 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2870000000000001e-06 2024-03-19T11:32:05.357 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.357 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.357 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:05.357 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254031+0000", 2024-03-19T11:32:05.357 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0960000000000007e-06 2024-03-19T11:32:05.357 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.357 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.357 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:05.357 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254036+0000", 2024-03-19T11:32:05.357 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3510000000000001e-06 2024-03-19T11:32:05.357 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.358 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.358 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.358 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254137+0000", 2024-03-19T11:32:05.358 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000101065 2024-03-19T11:32:05.358 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.358 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.358 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:05.358 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254147+0000", 2024-03-19T11:32:05.358 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3929999999999997e-06 2024-03-19T11:32:05.358 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.358 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.358 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.358 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.087055+0000", 2024-03-19T11:32:05.358 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.83290843299999995 2024-03-19T11:32:05.359 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.359 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.359 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:05.359 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.087142+0000", 2024-03-19T11:32:05.359 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7151999999999995e-05 2024-03-19T11:32:05.359 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.359 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.359 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:05.359 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.089802+0000", 2024-03-19T11:32:05.359 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0026594890000000001 2024-03-19T11:32:05.359 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.359 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:05.359 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.359 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.359 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.360 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3205 4.6 4:626acd4d:::smithi03829856-2:1f [sparse-read 0~1665607] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108)", 2024-03-19T11:32:05.360 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:27.254049+0000", 2024-03-19T11:32:05.360 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 38.029670908, 2024-03-19T11:32:05.360 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.83625012399999998, 2024-03-19T11:32:05.360 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:05.360 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:05.360 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:05.360 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:05.360 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:05.360 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3205 2024-03-19T11:32:05.360 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.360 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:05.360 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.360 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:05.361 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254049+0000", 2024-03-19T11:32:05.361 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.361 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.361 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.361 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:05.361 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254049+0000", 2024-03-19T11:32:05.361 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.361 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.361 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.361 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:05.361 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254049+0000", 2024-03-19T11:32:05.361 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0699999999999997e-07 2024-03-19T11:32:05.361 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.361 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.362 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:05.362 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254053+0000", 2024-03-19T11:32:05.362 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.546e-06 2024-03-19T11:32:05.362 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.362 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.362 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:05.362 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254063+0000", 2024-03-19T11:32:05.362 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7939999999999997e-06 2024-03-19T11:32:05.362 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.362 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.362 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:05.362 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254072+0000", 2024-03-19T11:32:05.362 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9250000000000001e-06 2024-03-19T11:32:05.362 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.362 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.363 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.363 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254186+0000", 2024-03-19T11:32:05.363 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011446300000000001 2024-03-19T11:32:05.363 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.363 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.363 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:05.363 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254198+0000", 2024-03-19T11:32:05.363 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1933000000000001e-05 2024-03-19T11:32:05.363 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.363 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.363 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.363 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.089838+0000", 2024-03-19T11:32:05.363 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.83563974200000002 2024-03-19T11:32:05.363 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.364 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.364 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:05.364 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.089941+0000", 2024-03-19T11:32:05.364 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000103052 2024-03-19T11:32:05.364 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.364 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.364 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:05.364 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.090299+0000", 2024-03-19T11:32:05.364 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00035806199999999997 2024-03-19T11:32:05.364 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.364 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:05.364 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.364 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.364 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.364 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3220 4.16 4:6868af91:::smithi03829856-10:head [tier-promote] snapc 21=[21,20,1f,1e,1d,1c,1b,1a,18,17] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-19T11:32:05.365 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:28.068879+0000", 2024-03-19T11:32:05.365 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 37.214840701999996, 2024-03-19T11:32:05.365 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025399244000000001, 2024-03-19T11:32:05.365 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:05.365 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:05.365 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:05.365 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:05.365 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:05.365 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3220 2024-03-19T11:32:05.365 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.365 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:05.365 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.365 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:05.365 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.068879+0000", 2024-03-19T11:32:05.366 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.366 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.366 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.366 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:05.366 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.068879+0000", 2024-03-19T11:32:05.366 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.366 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.366 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.366 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:05.366 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.068881+0000", 2024-03-19T11:32:05.366 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3879999999999998e-06 2024-03-19T11:32:05.366 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.366 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.366 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:05.367 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.068892+0000", 2024-03-19T11:32:05.367 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0672999999999999e-05 2024-03-19T11:32:05.367 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.367 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.367 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:05.367 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.068917+0000", 2024-03-19T11:32:05.367 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5367e-05 2024-03-19T11:32:05.367 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.367 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.367 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:05.367 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.068935+0000", 2024-03-19T11:32:05.367 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7832000000000001e-05 2024-03-19T11:32:05.367 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.367 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.367 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.368 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.069075+0000", 2024-03-19T11:32:05.368 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013980899999999999 2024-03-19T11:32:05.368 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.368 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.368 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-03-19T11:32:05.368 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.069091+0000", 2024-03-19T11:32:05.368 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6325e-05 2024-03-19T11:32:05.368 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.368 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.368 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.368 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.071382+0000", 2024-03-19T11:32:05.368 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022910109999999999 2024-03-19T11:32:05.368 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.368 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.368 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:05.369 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.071538+0000", 2024-03-19T11:32:05.369 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015600999999999999 2024-03-19T11:32:05.369 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.369 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.369 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:05.369 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.094278+0000", 2024-03-19T11:32:05.369 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.022739829 2024-03-19T11:32:05.369 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.369 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:05.369 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.369 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.369 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.369 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3885 4.21 4:8401c1f0:::smithi03829856-43:head [write 443784~648161] snapc 29=[29,28,27,26,25,24,23,22,21,1f,1e,1d,1a,17] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e129)", 2024-03-19T11:32:05.369 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:50.078273+0000", 2024-03-19T11:32:05.370 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 15.205446881, 2024-03-19T11:32:05.370 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1185664280000001, 2024-03-19T11:32:05.370 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:05.370 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:05.370 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:05.370 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:05.370 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:05.370 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3885 2024-03-19T11:32:05.370 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.370 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:05.370 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.370 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:05.370 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.078273+0000", 2024-03-19T11:32:05.370 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.370 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.371 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.371 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:05.371 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.078273+0000", 2024-03-19T11:32:05.371 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.371 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.371 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.371 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:05.371 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.078276+0000", 2024-03-19T11:32:05.371 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.942e-06 2024-03-19T11:32:05.371 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.371 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.371 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:05.371 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.078790+0000", 2024-03-19T11:32:05.371 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00051449799999999995 2024-03-19T11:32:05.372 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.372 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.372 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:05.372 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.078809+0000", 2024-03-19T11:32:05.372 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8980000000000001e-05 2024-03-19T11:32:05.372 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.372 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.372 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:05.372 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.078821+0000", 2024-03-19T11:32:05.372 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1432e-05 2024-03-19T11:32:05.372 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.372 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.372 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.372 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.079070+0000", 2024-03-19T11:32:05.372 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00024937100000000002 2024-03-19T11:32:05.373 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.373 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.373 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-03-19T11:32:05.373 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.079079+0000", 2024-03-19T11:32:05.373 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2949999999999999e-06 2024-03-19T11:32:05.373 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.373 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.373 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.373 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.083821+0000", 2024-03-19T11:32:05.373 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0047417129999999998 2024-03-19T11:32:05.373 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.373 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.373 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:05.373 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.083836+0000", 2024-03-19T11:32:05.374 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4591e-05 2024-03-19T11:32:05.374 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.374 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.374 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.374 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.101489+0000", 2024-03-19T11:32:05.374 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0176535019999999 2024-03-19T11:32:05.374 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.374 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.374 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:32:05.374 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.104918+0000", 2024-03-19T11:32:05.374 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003429353 2024-03-19T11:32:05.374 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.374 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.375 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.375 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.142247+0000", 2024-03-19T11:32:05.375 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.037329108 2024-03-19T11:32:05.375 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.375 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.375 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:05.375 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.142409+0000", 2024-03-19T11:32:05.375 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016169700000000001 2024-03-19T11:32:05.375 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.375 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.375 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4,5", 2024-03-19T11:32:05.375 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.142754+0000", 2024-03-19T11:32:05.375 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000345025 2024-03-19T11:32:05.375 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.375 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.376 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:32:05.376 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.161867+0000", 2024-03-19T11:32:05.376 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019112371 2024-03-19T11:32:05.376 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.376 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.376 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:05.376 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.166589+0000", 2024-03-19T11:32:05.376 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0047220270000000002 2024-03-19T11:32:05.376 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.376 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.376 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:05.376 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.196673+0000", 2024-03-19T11:32:05.376 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.030084422999999999 2024-03-19T11:32:05.376 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.377 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.377 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:32:05.377 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.196730+0000", 2024-03-19T11:32:05.377 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7027000000000001e-05 2024-03-19T11:32:05.377 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.377 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.377 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:05.377 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.196839+0000", 2024-03-19T11:32:05.377 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000109073 2024-03-19T11:32:05.377 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.377 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:05.377 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.377 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.377 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.377 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3886 4.21 4:8401c1f0:::smithi03829856-43:head [write 1657676~659775] snapc 29=[29,28,27,26,25,24,23,22,21,1f,1e,1d,1a,17] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e129)", 2024-03-19T11:32:05.378 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:50.078842+0000", 2024-03-19T11:32:05.378 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 15.204877843, 2024-03-19T11:32:05.378 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1181774870000001, 2024-03-19T11:32:05.378 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:05.378 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:05.378 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:05.378 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:05.378 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:05.378 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3886 2024-03-19T11:32:05.378 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.378 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:05.378 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.378 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:05.378 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.078842+0000", 2024-03-19T11:32:05.379 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.379 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.379 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.379 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:05.379 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.078842+0000", 2024-03-19T11:32:05.379 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.379 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.379 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.379 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:05.379 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.078843+0000", 2024-03-19T11:32:05.379 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7400000000000002e-07 2024-03-19T11:32:05.379 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.379 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.379 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:05.380 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.079370+0000", 2024-03-19T11:32:05.380 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00052713200000000003 2024-03-19T11:32:05.380 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.380 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.380 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:05.380 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.079383+0000", 2024-03-19T11:32:05.380 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3131e-05 2024-03-19T11:32:05.380 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.380 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.380 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:05.381 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.079393+0000", 2024-03-19T11:32:05.381 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0114e-05 2024-03-19T11:32:05.381 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.381 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.381 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.381 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082249+0000", 2024-03-19T11:32:05.381 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0028558260000000001 2024-03-19T11:32:05.381 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.381 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.381 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:32:05.381 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082258+0000", 2024-03-19T11:32:05.381 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.5740000000000002e-06 2024-03-19T11:32:05.381 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.381 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.382 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.382 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.084372+0000", 2024-03-19T11:32:05.382 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0021131909999999999 2024-03-19T11:32:05.382 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.382 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.382 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:05.382 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.084383+0000", 2024-03-19T11:32:05.382 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1099e-05 2024-03-19T11:32:05.382 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.382 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.382 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.382 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.105453+0000", 2024-03-19T11:32:05.382 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.021070116 2024-03-19T11:32:05.382 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.382 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.383 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:32:05.383 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.105574+0000", 2024-03-19T11:32:05.383 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012102199999999999 2024-03-19T11:32:05.383 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.383 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.383 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.383 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.151364+0000", 2024-03-19T11:32:05.383 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.045790294000000002 2024-03-19T11:32:05.383 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.383 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.383 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:05.383 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.151579+0000", 2024-03-19T11:32:05.383 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00021460399999999999 2024-03-19T11:32:05.383 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.383 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.384 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4,5", 2024-03-19T11:32:05.384 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.151927+0000", 2024-03-19T11:32:05.384 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00034839700000000002 2024-03-19T11:32:05.384 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.384 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.384 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:32:05.384 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.161878+0000", 2024-03-19T11:32:05.384 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0099512430000000002 2024-03-19T11:32:05.384 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.384 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.384 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:05.384 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.169256+0000", 2024-03-19T11:32:05.384 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0073777440000000003 2024-03-19T11:32:05.384 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.385 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.385 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:05.385 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.196889+0000", 2024-03-19T11:32:05.385 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027633179000000001 2024-03-19T11:32:05.385 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.385 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.385 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:32:05.385 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.196937+0000", 2024-03-19T11:32:05.385 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7966999999999997e-05 2024-03-19T11:32:05.385 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.385 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.385 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:05.385 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.197019+0000", 2024-03-19T11:32:05.385 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1979999999999998e-05 2024-03-19T11:32:05.385 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.386 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:05.386 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.386 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.386 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.386 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3887 4.21 4:8401c1f0:::smithi03829856-43:head [write 2818209~425575] snapc 29=[29,28,27,26,25,24,23,22,21,1f,1e,1d,1a,17] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e129)", 2024-03-19T11:32:05.386 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:50.082016+0000", 2024-03-19T11:32:05.386 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 15.201704003, 2024-03-19T11:32:05.386 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.115265664, 2024-03-19T11:32:05.386 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:05.386 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:05.386 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:05.386 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:05.386 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:05.386 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3887 2024-03-19T11:32:05.387 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.387 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:05.387 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.387 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:05.387 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082016+0000", 2024-03-19T11:32:05.387 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.387 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.387 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.387 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:05.387 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082016+0000", 2024-03-19T11:32:05.387 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.387 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.387 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.387 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:05.387 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082017+0000", 2024-03-19T11:32:05.388 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.587e-06 2024-03-19T11:32:05.388 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.388 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.388 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:05.388 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082203+0000", 2024-03-19T11:32:05.388 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000185613 2024-03-19T11:32:05.388 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.388 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.388 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:05.388 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082214+0000", 2024-03-19T11:32:05.388 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1357e-05 2024-03-19T11:32:05.388 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.388 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.388 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:05.389 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082221+0000", 2024-03-19T11:32:05.389 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6290000000000004e-06 2024-03-19T11:32:05.389 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.389 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.389 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.389 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082300+0000", 2024-03-19T11:32:05.389 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9425999999999995e-05 2024-03-19T11:32:05.389 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.389 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.389 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:32:05.389 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082306+0000", 2024-03-19T11:32:05.389 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1419999999999998e-06 2024-03-19T11:32:05.389 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.389 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.390 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.390 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.084417+0000", 2024-03-19T11:32:05.390 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0021107589999999998 2024-03-19T11:32:05.390 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.390 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.390 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:05.390 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.084426+0000", 2024-03-19T11:32:05.390 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8410000000000006e-06 2024-03-19T11:32:05.390 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.390 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.390 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.390 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.105770+0000", 2024-03-19T11:32:05.390 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0213442100000001 2024-03-19T11:32:05.390 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.390 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.391 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:32:05.391 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.105890+0000", 2024-03-19T11:32:05.391 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000120349 2024-03-19T11:32:05.391 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.391 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.391 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.391 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.157275+0000", 2024-03-19T11:32:05.391 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.051384678000000003 2024-03-19T11:32:05.391 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.391 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.391 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:05.391 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.157445+0000", 2024-03-19T11:32:05.391 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000169746 2024-03-19T11:32:05.391 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.392 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.392 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4,5", 2024-03-19T11:32:05.392 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.157790+0000", 2024-03-19T11:32:05.392 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00034491799999999999 2024-03-19T11:32:05.392 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.392 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.392 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:32:05.392 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.161935+0000", 2024-03-19T11:32:05.392 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0041450269999999999 2024-03-19T11:32:05.392 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.392 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.392 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:05.393 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.189666+0000", 2024-03-19T11:32:05.393 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027731062000000001 2024-03-19T11:32:05.393 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.393 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.393 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:05.393 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.197073+0000", 2024-03-19T11:32:05.393 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074070270000000001 2024-03-19T11:32:05.393 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.393 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.393 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:32:05.393 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.197116+0000", 2024-03-19T11:32:05.393 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2758e-05 2024-03-19T11:32:05.393 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.393 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.393 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:05.394 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.197281+0000", 2024-03-19T11:32:05.394 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016553499999999999 2024-03-19T11:32:05.394 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.394 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:05.394 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.394 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.394 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.394 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3888 4.21 4:8401c1f0:::smithi03829856-43:head [setxattr _header (57) in=7b,truncate 3243784] snapc 29=[29,28,27,26,25,24,23,22,21,1f,1e,1d,1a,17] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e129)", 2024-03-19T11:32:05.394 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:50.082238+0000", 2024-03-19T11:32:05.394 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 15.20148112, 2024-03-19T11:32:05.394 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.115518945, 2024-03-19T11:32:05.394 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:05.394 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:05.394 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:05.395 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:05.395 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:05.395 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3888 2024-03-19T11:32:05.395 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.395 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:05.395 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.395 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:05.395 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082238+0000", 2024-03-19T11:32:05.395 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.395 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.395 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.395 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:05.395 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082238+0000", 2024-03-19T11:32:05.395 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.396 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.396 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.396 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:05.396 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082239+0000", 2024-03-19T11:32:05.396 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6400000000000001e-07 2024-03-19T11:32:05.396 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.396 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.396 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:05.396 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082249+0000", 2024-03-19T11:32:05.396 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9490000000000008e-06 2024-03-19T11:32:05.396 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.396 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.396 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:05.396 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082260+0000", 2024-03-19T11:32:05.396 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0499000000000001e-05 2024-03-19T11:32:05.397 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.397 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.397 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:05.397 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082267+0000", 2024-03-19T11:32:05.397 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9029999999999998e-06 2024-03-19T11:32:05.397 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.397 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.397 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.397 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082341+0000", 2024-03-19T11:32:05.397 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4776000000000004e-05 2024-03-19T11:32:05.397 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.397 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.397 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:32:05.397 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082350+0000", 2024-03-19T11:32:05.397 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2279999999999998e-06 2024-03-19T11:32:05.398 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.398 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.398 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.398 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.084459+0000", 2024-03-19T11:32:05.398 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0021094579999999998 2024-03-19T11:32:05.398 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.398 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.398 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:05.398 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.084468+0000", 2024-03-19T11:32:05.398 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2e-06 2024-03-19T11:32:05.398 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.398 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.398 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.398 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.105958+0000", 2024-03-19T11:32:05.399 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0214894450000001 2024-03-19T11:32:05.399 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.399 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.399 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:32:05.399 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.106112+0000", 2024-03-19T11:32:05.399 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000154022 2024-03-19T11:32:05.399 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.399 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.399 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.399 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.160042+0000", 2024-03-19T11:32:05.399 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.053929935999999998 2024-03-19T11:32:05.399 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.399 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.399 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:05.399 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.160205+0000", 2024-03-19T11:32:05.400 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016317600000000001 2024-03-19T11:32:05.400 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.400 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.400 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4,5", 2024-03-19T11:32:05.400 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.160566+0000", 2024-03-19T11:32:05.400 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000360918 2024-03-19T11:32:05.400 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.400 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.400 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:32:05.400 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.162064+0000", 2024-03-19T11:32:05.400 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014983380000000001 2024-03-19T11:32:05.400 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.400 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.400 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:05.401 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.196608+0000", 2024-03-19T11:32:05.401 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.034544115 2024-03-19T11:32:05.401 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.401 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.401 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:05.401 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.197420+0000", 2024-03-19T11:32:05.401 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00081147299999999995 2024-03-19T11:32:05.401 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.401 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.401 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:32:05.401 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.197519+0000", 2024-03-19T11:32:05.401 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8932999999999996e-05 2024-03-19T11:32:05.401 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.401 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.402 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:05.402 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.197757+0000", 2024-03-19T11:32:05.402 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000238812 2024-03-19T11:32:05.402 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.402 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:05.402 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.402 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.402 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.402 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3889 4.21 4:8401c1f0:::smithi03829856-43:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e129)", 2024-03-19T11:32:05.402 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:50.082302+0000", 2024-03-19T11:32:05.402 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 15.201417971, 2024-03-19T11:32:05.402 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.116076547, 2024-03-19T11:32:05.402 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:05.402 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:05.402 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:05.403 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:05.403 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:05.403 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3889 2024-03-19T11:32:05.403 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.403 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:05.403 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.403 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:05.403 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082302+0000", 2024-03-19T11:32:05.403 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.403 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.403 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.404 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:05.404 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082302+0000", 2024-03-19T11:32:05.404 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.404 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.404 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.404 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:05.404 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082303+0000", 2024-03-19T11:32:05.404 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2720000000000001e-06 2024-03-19T11:32:05.404 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.404 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.404 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:05.404 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082314+0000", 2024-03-19T11:32:05.404 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0864e-05 2024-03-19T11:32:05.404 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.405 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.405 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:05.405 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082327+0000", 2024-03-19T11:32:05.405 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2972e-05 2024-03-19T11:32:05.405 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.405 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.405 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:05.405 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082339+0000", 2024-03-19T11:32:05.405 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2458e-05 2024-03-19T11:32:05.405 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.405 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.405 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.405 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.085086+0000", 2024-03-19T11:32:05.405 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0027472859999999998 2024-03-19T11:32:05.406 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.406 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.406 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:05.406 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.085100+0000", 2024-03-19T11:32:05.406 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3339000000000001e-05 2024-03-19T11:32:05.406 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.406 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.406 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.406 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.106149+0000", 2024-03-19T11:32:05.406 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0210496010000001 2024-03-19T11:32:05.406 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.406 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.406 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:32:05.406 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.106232+0000", 2024-03-19T11:32:05.407 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2923999999999994e-05 2024-03-19T11:32:05.407 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.407 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.407 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.407 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.161418+0000", 2024-03-19T11:32:05.407 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.055186156 2024-03-19T11:32:05.407 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.407 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.407 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-03-19T11:32:05.407 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.161535+0000", 2024-03-19T11:32:05.407 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000117018 2024-03-19T11:32:05.407 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.407 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.407 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.407 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.197872+0000", 2024-03-19T11:32:05.408 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.036336161999999998 2024-03-19T11:32:05.408 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.408 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.408 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:05.408 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.198037+0000", 2024-03-19T11:32:05.408 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000164924 2024-03-19T11:32:05.408 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.408 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.408 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:05.408 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.198378+0000", 2024-03-19T11:32:05.408 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00034157099999999999 2024-03-19T11:32:05.408 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.408 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:05.409 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.409 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.409 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:05.409 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:32:05.410 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-19T11:32:05.411 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:32:05.417+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:32:05.511 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:32:05.521+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:32:05.584 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:32:05.584 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:32:05.584 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:32:05.584 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:32:05.599 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-19T11:32:05.612 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:32:05.621+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:32:05.713 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:32:05.721+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:32:05.718 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:32:05.718 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:32:05.718 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:32:05.718 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:32:05.718 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:32:05.733 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-19T11:32:05.814 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:32:05.823+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:32:05.856 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:32:05.856 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:32:05.857 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:32:05.857 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:32:05.857 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.857 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1063 4.4 4:214026b3:::smithi03829856-47:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:32:05.857 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.784904+0000", 2024-03-19T11:32:05.857 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 112.07846175100001, 2024-03-19T11:32:05.857 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024425912000000001, 2024-03-19T11:32:05.857 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:05.857 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:05.857 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:05.857 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:05.857 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:05.857 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1063 2024-03-19T11:32:05.858 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.858 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:05.858 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.858 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:05.858 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784904+0000", 2024-03-19T11:32:05.858 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.858 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.858 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.858 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:05.858 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784904+0000", 2024-03-19T11:32:05.858 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.858 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.858 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.859 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:05.859 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784906+0000", 2024-03-19T11:32:05.859 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1409999999999999e-06 2024-03-19T11:32:05.859 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.859 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.859 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:05.859 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784915+0000", 2024-03-19T11:32:05.859 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5459999999999992e-06 2024-03-19T11:32:05.859 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.859 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.859 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:05.859 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784935+0000", 2024-03-19T11:32:05.859 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0310999999999999e-05 2024-03-19T11:32:05.859 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.859 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.860 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:05.860 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784946+0000", 2024-03-19T11:32:05.860 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0675e-05 2024-03-19T11:32:05.860 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.860 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.860 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.860 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.785032+0000", 2024-03-19T11:32:05.861 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6049000000000002e-05 2024-03-19T11:32:05.861 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.861 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.861 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:05.861 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.785136+0000", 2024-03-19T11:32:05.861 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000104209 2024-03-19T11:32:05.861 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.861 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.861 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:05.861 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.809330+0000", 2024-03-19T11:32:05.861 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024193981 2024-03-19T11:32:05.861 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.861 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:05.862 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.862 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.862 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.862 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1433 4.16 4:6be89c06:::smithi03829856-24:c [sparse-read 0~1100427,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:32:05.862 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.976652+0000", 2024-03-19T11:32:05.862 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 99.886714049999995, 2024-03-19T11:32:05.862 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95684753199999995, 2024-03-19T11:32:05.862 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:05.862 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:05.862 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:05.862 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:05.862 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:05.862 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1433 2024-03-19T11:32:05.862 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.863 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:05.863 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.863 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:05.863 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976652+0000", 2024-03-19T11:32:05.863 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.863 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.863 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.863 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:05.863 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976652+0000", 2024-03-19T11:32:05.863 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.863 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.863 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.863 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:05.863 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976654+0000", 2024-03-19T11:32:05.863 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6199999999999999e-06 2024-03-19T11:32:05.864 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.864 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.864 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:05.864 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976668+0000", 2024-03-19T11:32:05.864 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3213999999999999e-05 2024-03-19T11:32:05.864 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.864 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.864 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:05.864 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976694+0000", 2024-03-19T11:32:05.864 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6488000000000001e-05 2024-03-19T11:32:05.864 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.864 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.864 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:05.864 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976705+0000", 2024-03-19T11:32:05.864 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1044e-05 2024-03-19T11:32:05.865 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.865 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.865 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.865 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976795+0000", 2024-03-19T11:32:05.865 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0283000000000003e-05 2024-03-19T11:32:05.865 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.865 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.865 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:05.865 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976811+0000", 2024-03-19T11:32:05.865 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5262000000000001e-05 2024-03-19T11:32:05.865 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.865 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.865 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.865 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.927337+0000", 2024-03-19T11:32:05.865 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95052597100000003 2024-03-19T11:32:05.866 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.866 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.866 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:05.866 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.927495+0000", 2024-03-19T11:32:05.866 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015788600000000001 2024-03-19T11:32:05.866 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.866 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.866 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:05.866 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.933499+0000", 2024-03-19T11:32:05.866 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0060047640000000001 2024-03-19T11:32:05.866 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.866 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:05.866 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.866 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.866 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.867 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1434 4.16 4:6be89c06:::smithi03829856-24:c [read 0~2386561,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:32:05.867 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.976739+0000", 2024-03-19T11:32:05.867 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 99.886626888999999, 2024-03-19T11:32:05.867 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95781317399999999, 2024-03-19T11:32:05.867 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:05.867 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:05.867 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:05.867 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:05.867 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:05.867 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1434 2024-03-19T11:32:05.867 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.867 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:05.867 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.867 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:05.867 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976739+0000", 2024-03-19T11:32:05.868 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.868 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.868 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.868 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:05.868 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976739+0000", 2024-03-19T11:32:05.868 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.868 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.868 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.868 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:05.868 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976740+0000", 2024-03-19T11:32:05.868 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8800000000000001e-07 2024-03-19T11:32:05.868 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.868 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.868 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:05.869 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976748+0000", 2024-03-19T11:32:05.869 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8979999999999996e-06 2024-03-19T11:32:05.869 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.869 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.869 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:05.869 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976757+0000", 2024-03-19T11:32:05.869 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9439999999999994e-06 2024-03-19T11:32:05.869 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.869 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.869 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:05.869 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976763+0000", 2024-03-19T11:32:05.869 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5030000000000002e-06 2024-03-19T11:32:05.869 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.869 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.869 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.870 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976849+0000", 2024-03-19T11:32:05.870 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5803999999999994e-05 2024-03-19T11:32:05.870 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.870 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.870 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:05.870 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976860+0000", 2024-03-19T11:32:05.870 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0869e-05 2024-03-19T11:32:05.870 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.870 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.870 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.870 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.933562+0000", 2024-03-19T11:32:05.870 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95670232300000002 2024-03-19T11:32:05.870 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.870 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.871 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:05.871 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.933693+0000", 2024-03-19T11:32:05.871 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013110699999999999 2024-03-19T11:32:05.871 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.871 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.871 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:05.871 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.934552+0000", 2024-03-19T11:32:05.871 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00085883800000000001 2024-03-19T11:32:05.871 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.871 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:05.871 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.871 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.871 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.871 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1435 4.16 4:6be89c06:::smithi03829856-24:c [sparse-read 0~1100427] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:32:05.872 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.976775+0000", 2024-03-19T11:32:05.872 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 99.886591218000007, 2024-03-19T11:32:05.872 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95962779499999995, 2024-03-19T11:32:05.872 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:05.872 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:05.872 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:05.872 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:05.872 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:05.872 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1435 2024-03-19T11:32:05.872 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.872 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:05.872 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.872 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:05.872 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976775+0000", 2024-03-19T11:32:05.873 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.873 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.873 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.873 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:05.873 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976775+0000", 2024-03-19T11:32:05.873 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.873 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.873 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.873 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:05.873 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976775+0000", 2024-03-19T11:32:05.874 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9800000000000003e-07 2024-03-19T11:32:05.874 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.874 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.874 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:05.874 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976779+0000", 2024-03-19T11:32:05.874 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3830000000000001e-06 2024-03-19T11:32:05.874 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.874 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.874 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:05.874 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976785+0000", 2024-03-19T11:32:05.874 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5649999999999998e-06 2024-03-19T11:32:05.874 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.874 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.875 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:05.875 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976790+0000", 2024-03-19T11:32:05.875 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1379999999999998e-06 2024-03-19T11:32:05.875 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.875 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.875 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.875 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976889+0000", 2024-03-19T11:32:05.875 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8331000000000004e-05 2024-03-19T11:32:05.875 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.875 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.875 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:05.876 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976894+0000", 2024-03-19T11:32:05.876 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8270000000000003e-06 2024-03-19T11:32:05.876 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.876 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.876 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.876 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.935982+0000", 2024-03-19T11:32:05.876 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95908714100000003 2024-03-19T11:32:05.876 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.876 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.876 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:05.876 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.936093+0000", 2024-03-19T11:32:05.877 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000111557 2024-03-19T11:32:05.877 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.877 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.877 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:05.877 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.936402+0000", 2024-03-19T11:32:05.877 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030925499999999998 2024-03-19T11:32:05.877 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.877 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:05.877 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.877 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.877 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.877 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1442 4.6 4:626acd4d:::smithi03829856-2:head [cache-evict] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:32:05.878 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.978864+0000", 2024-03-19T11:32:05.878 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 99.884501553999996, 2024-03-19T11:32:05.878 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95708845099999995, 2024-03-19T11:32:05.878 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:05.878 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:05.878 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:05.878 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:05.878 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:05.878 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1442 2024-03-19T11:32:05.878 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.879 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:05.879 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.879 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:05.879 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978864+0000", 2024-03-19T11:32:05.879 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.879 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.879 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.879 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:05.879 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978864+0000", 2024-03-19T11:32:05.879 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.880 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.880 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.880 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:05.880 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978866+0000", 2024-03-19T11:32:05.880 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0729999999999999e-06 2024-03-19T11:32:05.880 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.880 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.880 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:05.880 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978877+0000", 2024-03-19T11:32:05.880 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0390999999999999e-05 2024-03-19T11:32:05.880 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.880 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.880 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:05.880 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978902+0000", 2024-03-19T11:32:05.881 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5635000000000001e-05 2024-03-19T11:32:05.881 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.881 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.881 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:05.881 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978915+0000", 2024-03-19T11:32:05.881 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2617e-05 2024-03-19T11:32:05.881 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.881 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.881 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.881 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.979005+0000", 2024-03-19T11:32:05.881 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0552999999999999e-05 2024-03-19T11:32:05.881 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.881 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.881 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:05.882 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.979021+0000", 2024-03-19T11:32:05.882 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5102999999999999e-05 2024-03-19T11:32:05.882 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.882 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.882 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.882 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.926879+0000", 2024-03-19T11:32:05.882 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.94785817900000002 2024-03-19T11:32:05.882 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.882 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.882 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:32:05.882 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.927277+0000", 2024-03-19T11:32:05.882 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00039819600000000002 2024-03-19T11:32:05.882 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.883 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.883 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.883 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.935752+0000", 2024-03-19T11:32:05.883 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0084753420000000003 2024-03-19T11:32:05.883 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.883 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.883 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:05.883 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.935874+0000", 2024-03-19T11:32:05.883 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000121311 2024-03-19T11:32:05.883 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.883 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.883 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:05.883 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.935953+0000", 2024-03-19T11:32:05.884 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9050999999999999e-05 2024-03-19T11:32:05.884 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.884 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:05.884 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.884 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.884 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.884 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2003 4.16 4:6868af91:::smithi03829856-10:head [tier-promote] snapc 12=[12,11,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:32:05.884 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:45.804046+0000", 2024-03-19T11:32:05.885 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 80.059319641000002, 2024-03-19T11:32:05.885 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018976396, 2024-03-19T11:32:05.885 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:05.885 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:05.885 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:05.885 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:05.885 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:05.885 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2003 2024-03-19T11:32:05.885 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.885 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:05.885 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.885 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:05.885 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804046+0000", 2024-03-19T11:32:05.885 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.885 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.886 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.886 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:05.886 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804046+0000", 2024-03-19T11:32:05.886 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.886 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.886 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.886 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:05.886 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804052+0000", 2024-03-19T11:32:05.886 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7599999999999999e-06 2024-03-19T11:32:05.886 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.886 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.886 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:05.886 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804090+0000", 2024-03-19T11:32:05.886 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8018000000000001e-05 2024-03-19T11:32:05.887 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.887 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.887 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:05.887 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804256+0000", 2024-03-19T11:32:05.887 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016593599999999999 2024-03-19T11:32:05.887 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.887 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.887 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:05.887 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804300+0000", 2024-03-19T11:32:05.887 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3648e-05 2024-03-19T11:32:05.887 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.887 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.887 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.887 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804538+0000", 2024-03-19T11:32:05.887 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000238347 2024-03-19T11:32:05.888 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.888 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.888 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:05.888 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804716+0000", 2024-03-19T11:32:05.888 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000177968 2024-03-19T11:32:05.888 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.888 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.888 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:05.888 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.823023+0000", 2024-03-19T11:32:05.888 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018306718999999999 2024-03-19T11:32:05.888 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.888 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:05.888 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.888 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.889 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.889 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1d58db11:::smithi03829856-18:6, version: 55'80, data_included: [759683~639358,1904434~409218], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:6@55'80, size: 3159683, copy_subset: [(0, 3159683)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2313652, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1d58db11:::smithi03829856-18:b, version: 58'84, data_included: [443718~955323,1564138~93253], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:b@58'84, size: 3445402, copy_subset: [(0, 3445402)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1657391, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:10, version: 66'95, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:10@66'95, size: 3764735, copy_subset: [(0, 3764735)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:12, version: 66'97, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:12@66'97, size: 2504243, copy_subset: [(0, 2504243)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:c, version: 70'102, data_included: [416807~576167,1542129~472409], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:c@70'102, size: 2857157, copy_subset: [(0, 2857157)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2014538, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:32:05.889 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.275179+0000", 2024-03-19T11:32:05.889 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 71.588186651000001, 2024-03-19T11:32:05.889 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026559353000000001, 2024-03-19T11:32:05.889 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:05.889 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:05.889 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:05.889 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.889 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:05.890 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.275179+0000", 2024-03-19T11:32:05.890 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.890 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.890 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.890 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:05.890 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.275179+0000", 2024-03-19T11:32:05.890 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.890 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.890 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.890 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:05.890 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.275196+0000", 2024-03-19T11:32:05.891 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6946999999999999e-05 2024-03-19T11:32:05.891 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.891 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.891 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:05.891 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281456+0000", 2024-03-19T11:32:05.891 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0062600529999999998 2024-03-19T11:32:05.891 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.891 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.891 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:05.891 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281524+0000", 2024-03-19T11:32:05.891 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7971000000000004e-05 2024-03-19T11:32:05.891 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.891 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.891 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:05.891 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281591+0000", 2024-03-19T11:32:05.892 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6704000000000004e-05 2024-03-19T11:32:05.892 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.892 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.892 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.892 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281865+0000", 2024-03-19T11:32:05.892 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000274097 2024-03-19T11:32:05.892 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.892 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.892 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:05.892 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281901+0000", 2024-03-19T11:32:05.892 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5777e-05 2024-03-19T11:32:05.892 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.892 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.892 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:05.892 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.301738+0000", 2024-03-19T11:32:05.893 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019837804000000001 2024-03-19T11:32:05.893 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.893 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:05.893 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.893 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.893 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.893 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.18 e75,74 4:1d58db11:::smithi03829856-18:2,48'76)", 2024-03-19T11:32:05.893 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.281661+0000", 2024-03-19T11:32:05.893 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 71.581705119999995, 2024-03-19T11:32:05.893 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020384337999999998, 2024-03-19T11:32:05.893 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:05.893 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:05.894 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:05.894 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.894 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:05.894 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281661+0000", 2024-03-19T11:32:05.894 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.894 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.894 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.894 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:05.894 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281661+0000", 2024-03-19T11:32:05.894 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.894 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.895 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.895 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:05.895 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281661+0000", 2024-03-19T11:32:05.895 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9100000000000003e-07 2024-03-19T11:32:05.895 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.895 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.895 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:05.895 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281668+0000", 2024-03-19T11:32:05.895 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8199999999999999e-06 2024-03-19T11:32:05.895 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.895 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.895 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:05.895 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281679+0000", 2024-03-19T11:32:05.895 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1107999999999999e-05 2024-03-19T11:32:05.896 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.896 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.896 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:05.896 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281691+0000", 2024-03-19T11:32:05.896 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1399999999999999e-05 2024-03-19T11:32:05.896 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.896 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.896 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.896 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.301804+0000", 2024-03-19T11:32:05.896 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020113669000000001 2024-03-19T11:32:05.896 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.896 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.896 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:05.896 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.301818+0000", 2024-03-19T11:32:05.896 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3718e-05 2024-03-19T11:32:05.897 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.897 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.897 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:05.897 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302045+0000", 2024-03-19T11:32:05.897 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000226832 2024-03-19T11:32:05.897 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.897 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:05.897 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.897 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.897 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.897 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1dabc0f1:::smithi03829856-1:head, version: 70'103, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:head@70'103, size: 2621892, copy_subset: [(0, 2621892)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:32:05.897 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.281783+0000", 2024-03-19T11:32:05.897 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 71.581582843999996, 2024-03-19T11:32:05.897 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026187221, 2024-03-19T11:32:05.897 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:05.898 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:05.898 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:05.898 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.898 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:05.898 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281783+0000", 2024-03-19T11:32:05.898 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.898 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.898 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.898 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:05.898 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281783+0000", 2024-03-19T11:32:05.898 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.898 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.898 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.898 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:05.898 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281783+0000", 2024-03-19T11:32:05.899 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1699999999999999e-07 2024-03-19T11:32:05.899 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.899 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.899 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:05.899 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.282962+0000", 2024-03-19T11:32:05.899 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001178601 2024-03-19T11:32:05.899 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.899 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.899 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:05.899 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.282987+0000", 2024-03-19T11:32:05.899 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4899e-05 2024-03-19T11:32:05.899 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.900 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.900 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:05.900 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283006+0000", 2024-03-19T11:32:05.900 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9494999999999999e-05 2024-03-19T11:32:05.900 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.900 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.900 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.900 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.305757+0000", 2024-03-19T11:32:05.900 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.022750570000000001 2024-03-19T11:32:05.900 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.900 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.900 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:05.900 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.305770+0000", 2024-03-19T11:32:05.900 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2791e-05 2024-03-19T11:32:05.901 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.901 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.901 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:05.901 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.307970+0000", 2024-03-19T11:32:05.901 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022005480000000001 2024-03-19T11:32:05.901 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.901 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:05.901 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.901 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.901 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.901 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.18 e75,74 4:1d58db11:::smithi03829856-18:c,70'104)", 2024-03-19T11:32:05.902 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.283036+0000", 2024-03-19T11:32:05.902 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 71.580329743999997, 2024-03-19T11:32:05.902 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019146032, 2024-03-19T11:32:05.902 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:05.902 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:05.902 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:05.902 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.902 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:05.902 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283036+0000", 2024-03-19T11:32:05.902 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.902 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.902 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.903 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:05.903 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283036+0000", 2024-03-19T11:32:05.903 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.903 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.903 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.903 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:05.903 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283037+0000", 2024-03-19T11:32:05.903 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4899999999999995e-07 2024-03-19T11:32:05.903 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.903 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.903 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:05.903 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283042+0000", 2024-03-19T11:32:05.903 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0930000000000002e-06 2024-03-19T11:32:05.903 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.904 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.904 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:05.904 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283053+0000", 2024-03-19T11:32:05.904 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1083999999999999e-05 2024-03-19T11:32:05.904 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.904 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.904 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:05.904 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283059+0000", 2024-03-19T11:32:05.904 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6329999999999999e-06 2024-03-19T11:32:05.904 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.904 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.904 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.904 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302090+0000", 2024-03-19T11:32:05.904 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01903038 2024-03-19T11:32:05.904 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.905 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.905 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:05.905 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302097+0000", 2024-03-19T11:32:05.905 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.221e-06 2024-03-19T11:32:05.905 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.905 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.905 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:05.905 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302182+0000", 2024-03-19T11:32:05.905 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5072000000000001e-05 2024-03-19T11:32:05.905 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.905 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:05.905 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.905 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.905 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.905 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(osd.5.0:143 4.18 e75/74 4:1d58db11:::smithi03829856-18:head v 75'106, mlcod=0'0)", 2024-03-19T11:32:05.906 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.283083+0000", 2024-03-19T11:32:05.906 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 71.580282417000006, 2024-03-19T11:32:05.906 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025309439, 2024-03-19T11:32:05.906 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:05.906 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:05.906 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:05.906 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.906 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:05.906 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283083+0000", 2024-03-19T11:32:05.906 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.906 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.906 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.906 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:05.906 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283083+0000", 2024-03-19T11:32:05.907 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.907 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.907 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.907 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:05.907 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283084+0000", 2024-03-19T11:32:05.907 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7800000000000002e-07 2024-03-19T11:32:05.907 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.907 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.907 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:05.907 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.284082+0000", 2024-03-19T11:32:05.907 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0009977880000000001 2024-03-19T11:32:05.907 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.907 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.907 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:05.907 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.284095+0000", 2024-03-19T11:32:05.908 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3256e-05 2024-03-19T11:32:05.908 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.908 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.908 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:05.908 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.284102+0000", 2024-03-19T11:32:05.908 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.605e-06 2024-03-19T11:32:05.908 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.908 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.908 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.908 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302285+0000", 2024-03-19T11:32:05.908 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018182423 2024-03-19T11:32:05.908 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.908 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.908 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:05.908 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302341+0000", 2024-03-19T11:32:05.909 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.626e-05 2024-03-19T11:32:05.909 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.909 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.909 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:32:05.909 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.308338+0000", 2024-03-19T11:32:05.909 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0059968030000000002 2024-03-19T11:32:05.909 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.909 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.909 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:05.909 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.308393+0000", 2024-03-19T11:32:05.909 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4826e-05 2024-03-19T11:32:05.909 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.909 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:05.909 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.909 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.910 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.910 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "pg_update_log_missing(4.18 epoch 75/74 rep_tid 144 entries 75'107 (0'0) error 4:1d58db11:::smithi03829856-18:head by client.4415.0:2283 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-19T11:32:05.910 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.288252+0000", 2024-03-19T11:32:05.910 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 71.575113866999999, 2024-03-19T11:32:05.910 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020167377, 2024-03-19T11:32:05.910 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:05.910 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-03-19T11:32:05.910 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:05.910 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.910 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:05.910 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288252+0000", 2024-03-19T11:32:05.910 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.910 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.911 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.911 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:05.911 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288252+0000", 2024-03-19T11:32:05.911 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.911 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.911 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.911 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:05.911 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288253+0000", 2024-03-19T11:32:05.911 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2109999999999999e-06 2024-03-19T11:32:05.911 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.911 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.911 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:05.911 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288268+0000", 2024-03-19T11:32:05.911 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5003e-05 2024-03-19T11:32:05.911 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.911 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.912 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:05.912 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288301+0000", 2024-03-19T11:32:05.912 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2539999999999997e-05 2024-03-19T11:32:05.912 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.912 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.912 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:05.912 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288319+0000", 2024-03-19T11:32:05.912 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8232000000000001e-05 2024-03-19T11:32:05.912 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.912 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.912 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.912 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.305490+0000", 2024-03-19T11:32:05.912 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017170857000000001 2024-03-19T11:32:05.912 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.912 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.913 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:05.913 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.308419+0000", 2024-03-19T11:32:05.913 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029295340000000001 2024-03-19T11:32:05.913 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.913 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:05.913 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.913 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.913 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.913 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1d58db11:::smithi03829856-18:6, version: 55'80, data_included: [2313652~310127,3128657~31026], data_size: 341153, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:6@55'80, size: 3159683, copy_subset: [(0, 3159683)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3159683, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2313652, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:1d58db11:::smithi03829856-18:b, version: 58'84, data_included: [1657391~966388,2633066~82188], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:b@58'84, size: 3445402, copy_subset: [(0, 3445402)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2715254, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1657391, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:10, version: 66'95, data_included: [1829043~668012,3134120~380564], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:10@66'95, size: 3764735, copy_subset: [(0, 3764735)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3514684, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:12, version: 66'97, data_included: [1829043~507467], data_size: 507467, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:12@66'97, size: 2504243, copy_subset: [(0, 2504243)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2504243, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:c, version: 70'102, data_included: [2014538~156250], data_size: 156250, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:c@70'102, size: 2857157, copy_subset: [(0, 2857157)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2857157, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2014538, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:32:05.913 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.312362+0000", 2024-03-19T11:32:05.913 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 71.551003817999998, 2024-03-19T11:32:05.913 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017968951, 2024-03-19T11:32:05.913 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:05.913 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:05.914 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:05.914 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.914 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:05.914 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.312362+0000", 2024-03-19T11:32:05.914 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.914 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.914 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.914 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:05.914 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.312362+0000", 2024-03-19T11:32:05.914 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.914 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.914 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.914 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:05.914 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.312363+0000", 2024-03-19T11:32:05.914 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.528e-06 2024-03-19T11:32:05.915 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.915 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.915 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:05.915 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315748+0000", 2024-03-19T11:32:05.915 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033842830000000001 2024-03-19T11:32:05.915 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.915 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.915 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:05.915 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315795+0000", 2024-03-19T11:32:05.915 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7382999999999997e-05 2024-03-19T11:32:05.915 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.915 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.915 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:05.915 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315845+0000", 2024-03-19T11:32:05.916 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0319999999999999e-05 2024-03-19T11:32:05.916 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.916 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.916 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.916 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.316123+0000", 2024-03-19T11:32:05.916 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00027747200000000002 2024-03-19T11:32:05.916 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.916 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.916 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:05.916 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.316170+0000", 2024-03-19T11:32:05.916 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7466999999999998e-05 2024-03-19T11:32:05.916 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.916 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.916 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:05.917 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.330331+0000", 2024-03-19T11:32:05.917 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014160498000000001 2024-03-19T11:32:05.917 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.917 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:05.917 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.917 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.917 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.917 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1dabc0f1:::smithi03829856-1:head, version: 70'103, data_included: [1829043~555153], data_size: 555153, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:head@70'103, size: 2621892, copy_subset: [(0, 2621892)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2621892, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:32:05.917 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.315898+0000", 2024-03-19T11:32:05.917 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 71.547467831999995, 2024-03-19T11:32:05.917 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016880424000000002, 2024-03-19T11:32:05.917 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:05.917 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:05.917 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:05.917 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.917 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:05.918 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315898+0000", 2024-03-19T11:32:05.918 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.918 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.918 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.918 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:05.918 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315898+0000", 2024-03-19T11:32:05.918 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.918 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.918 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.918 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:05.918 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315899+0000", 2024-03-19T11:32:05.918 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.003e-06 2024-03-19T11:32:05.918 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.918 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.919 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:05.919 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.316486+0000", 2024-03-19T11:32:05.919 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00058673099999999995 2024-03-19T11:32:05.919 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.919 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.919 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:05.919 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.316519+0000", 2024-03-19T11:32:05.919 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3124000000000003e-05 2024-03-19T11:32:05.919 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.919 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.919 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:05.919 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.316534+0000", 2024-03-19T11:32:05.919 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.485e-05 2024-03-19T11:32:05.919 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.919 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.920 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.920 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.330415+0000", 2024-03-19T11:32:05.920 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013880905000000001 2024-03-19T11:32:05.920 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.920 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.920 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:05.920 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.330439+0000", 2024-03-19T11:32:05.920 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4630999999999998e-05 2024-03-19T11:32:05.920 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.920 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.920 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:05.920 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.332778+0000", 2024-03-19T11:32:05.920 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023391800000000002 2024-03-19T11:32:05.920 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.920 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:05.921 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.921 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.921 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.921 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2387 4.2 4:43b32242:::smithi03829856-22:head [read 596992~684544] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-03-19T11:32:05.921 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:59.319806+0000", 2024-03-19T11:32:05.921 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 66.543560165000002, 2024-03-19T11:32:05.921 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.78144344399999999, 2024-03-19T11:32:05.921 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:05.921 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:05.921 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:05.921 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:05.921 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:05.921 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2387 2024-03-19T11:32:05.921 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.921 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:05.921 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.922 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:05.922 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319806+0000", 2024-03-19T11:32:05.922 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.922 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.922 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.922 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:05.922 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319806+0000", 2024-03-19T11:32:05.922 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.922 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.922 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.922 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:05.922 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319807+0000", 2024-03-19T11:32:05.922 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3519999999999999e-06 2024-03-19T11:32:05.922 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.923 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.923 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:05.923 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319812+0000", 2024-03-19T11:32:05.923 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3959999999999997e-06 2024-03-19T11:32:05.923 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.923 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.923 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:05.923 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319827+0000", 2024-03-19T11:32:05.923 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4566000000000001e-05 2024-03-19T11:32:05.923 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.923 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.923 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:05.923 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319835+0000", 2024-03-19T11:32:05.923 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.994e-06 2024-03-19T11:32:05.923 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.924 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.924 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.924 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.320119+0000", 2024-03-19T11:32:05.924 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00028445199999999998 2024-03-19T11:32:05.924 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.924 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.924 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:05.924 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.320129+0000", 2024-03-19T11:32:05.924 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9580000000000002e-06 2024-03-19T11:32:05.924 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.924 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.924 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.924 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.069842+0000", 2024-03-19T11:32:05.924 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.74971218399999995 2024-03-19T11:32:05.924 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.925 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.925 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:32:05.925 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.070010+0000", 2024-03-19T11:32:05.925 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016846600000000001 2024-03-19T11:32:05.925 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.925 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.925 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.925 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.090911+0000", 2024-03-19T11:32:05.925 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020901494999999999 2024-03-19T11:32:05.925 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.925 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.925 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:32:05.925 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.091249+0000", 2024-03-19T11:32:05.925 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000337602 2024-03-19T11:32:05.925 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.926 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.926 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.926 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.098478+0000", 2024-03-19T11:32:05.926 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0072287549999999999 2024-03-19T11:32:05.926 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.926 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.926 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:05.926 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.098617+0000", 2024-03-19T11:32:05.926 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013871599999999999 2024-03-19T11:32:05.926 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.926 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.926 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:05.926 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.101249+0000", 2024-03-19T11:32:05.926 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002632508 2024-03-19T11:32:05.926 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.927 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:05.927 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.927 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.927 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.927 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 81/80 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 15=[]:{7: [6], b: [b], 10: [d], 12: [12, 11], 15: [15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:7, version: 55'82, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@55'82, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 16=[]:{7: [6], b: [b], 10: [d], 16: [16, 15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 15=[]:{7: [6], b: [b], 10: [d], 12: [12, 11], 15: [15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 58'86, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@58'86, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 16=[]:{7: [6], b: [b], 10: [d], 16: [16, 15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [504210~968657,1572695~79919], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 16=[]:{7: [6], b: [b], 10: [d], 16: [16, 15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:32:05.927 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:00.089924+0000", 2024-03-19T11:32:05.927 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 65.773441935999998, 2024-03-19T11:32:05.927 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024686583000000002, 2024-03-19T11:32:05.927 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:05.927 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:05.927 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:05.927 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.927 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:05.927 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.089924+0000", 2024-03-19T11:32:05.928 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.928 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.928 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.928 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:05.928 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.089924+0000", 2024-03-19T11:32:05.928 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.928 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.928 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.928 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:05.928 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.089926+0000", 2024-03-19T11:32:05.928 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9589999999999998e-06 2024-03-19T11:32:05.928 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.928 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.928 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:05.928 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093311+0000", 2024-03-19T11:32:05.929 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033854409999999999 2024-03-19T11:32:05.929 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.929 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.929 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:05.929 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093365+0000", 2024-03-19T11:32:05.929 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3795000000000002e-05 2024-03-19T11:32:05.929 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.929 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.929 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:05.929 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093438+0000", 2024-03-19T11:32:05.929 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2591999999999999e-05 2024-03-19T11:32:05.930 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.930 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.930 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.930 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093690+0000", 2024-03-19T11:32:05.930 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025264999999999999 2024-03-19T11:32:05.930 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.930 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.930 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:05.930 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093731+0000", 2024-03-19T11:32:05.930 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0881999999999999e-05 2024-03-19T11:32:05.931 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.931 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.931 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:05.931 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.114610+0000", 2024-03-19T11:32:05.931 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020879264000000002 2024-03-19T11:32:05.931 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.931 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:05.931 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.931 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.931 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.931 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.21 e81,80 4:8401c1f0:::smithi03829856-43:8,46'66,4:866f3dcc:::smithi03829856-31:2,47'73)", 2024-03-19T11:32:05.931 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:00.093514+0000", 2024-03-19T11:32:05.931 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 65.769851336000002, 2024-03-19T11:32:05.931 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027375713999999999, 2024-03-19T11:32:05.931 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:05.932 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:05.932 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:05.932 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.932 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:05.932 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093514+0000", 2024-03-19T11:32:05.932 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.932 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.932 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.932 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:05.932 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093514+0000", 2024-03-19T11:32:05.932 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.932 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.932 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.932 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:05.932 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093516+0000", 2024-03-19T11:32:05.933 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1179999999999999e-06 2024-03-19T11:32:05.933 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.933 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.933 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:05.933 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093523+0000", 2024-03-19T11:32:05.933 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7500000000000003e-06 2024-03-19T11:32:05.933 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.933 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.933 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:05.933 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093534+0000", 2024-03-19T11:32:05.933 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0831e-05 2024-03-19T11:32:05.933 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.933 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.933 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:05.933 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093542+0000", 2024-03-19T11:32:05.934 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0139999999999998e-06 2024-03-19T11:32:05.934 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.934 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.934 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.934 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.120682+0000", 2024-03-19T11:32:05.934 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027139838999999999 2024-03-19T11:32:05.934 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.934 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.934 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:05.934 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.120693+0000", 2024-03-19T11:32:05.934 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.136e-05 2024-03-19T11:32:05.934 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.934 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.934 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:05.934 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.120890+0000", 2024-03-19T11:32:05.935 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019680199999999999 2024-03-19T11:32:05.935 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.935 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:05.935 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.935 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.935 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.935 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2927 4.24 4:266590a5:::smithi03829856-16:head [append 0~0] snapc 1b=[1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-19T11:32:05.935 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:18.054474+0000", 2024-03-19T11:32:05.935 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 47.808892088999997, 2024-03-19T11:32:05.935 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024856390999999999, 2024-03-19T11:32:05.935 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:05.935 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:05.935 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:05.935 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:05.935 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:05.936 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2927 2024-03-19T11:32:05.936 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.936 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:05.936 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.936 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:05.936 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054474+0000", 2024-03-19T11:32:05.936 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.936 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.936 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.936 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:05.936 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054474+0000", 2024-03-19T11:32:05.936 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.936 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.936 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.936 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:05.937 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054476+0000", 2024-03-19T11:32:05.937 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6469999999999998e-06 2024-03-19T11:32:05.937 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.937 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.937 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:05.937 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054489+0000", 2024-03-19T11:32:05.937 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2612999999999999e-05 2024-03-19T11:32:05.937 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.937 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.937 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:05.937 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054526+0000", 2024-03-19T11:32:05.937 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7481999999999998e-05 2024-03-19T11:32:05.937 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.937 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.938 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:05.938 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054547+0000", 2024-03-19T11:32:05.938 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0820999999999999e-05 2024-03-19T11:32:05.938 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.938 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.938 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.938 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054633+0000", 2024-03-19T11:32:05.938 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5556000000000006e-05 2024-03-19T11:32:05.938 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.938 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.938 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-03-19T11:32:05.938 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054639+0000", 2024-03-19T11:32:05.938 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7300000000000002e-06 2024-03-19T11:32:05.938 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.938 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.939 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.939 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.061592+0000", 2024-03-19T11:32:05.939 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0069539279999999998 2024-03-19T11:32:05.939 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.939 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.939 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:05.939 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.061746+0000", 2024-03-19T11:32:05.939 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015351699999999999 2024-03-19T11:32:05.939 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.939 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.939 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:32:05.939 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.062031+0000", 2024-03-19T11:32:05.939 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00028539700000000001 2024-03-19T11:32:05.939 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.940 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.940 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:32:05.940 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.079164+0000", 2024-03-19T11:32:05.940 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017132865000000001 2024-03-19T11:32:05.940 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.940 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.940 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:05.940 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.079217+0000", 2024-03-19T11:32:05.940 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2373999999999997e-05 2024-03-19T11:32:05.940 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.940 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.940 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:32:05.940 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.079284+0000", 2024-03-19T11:32:05.940 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7433000000000003e-05 2024-03-19T11:32:05.940 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.940 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.941 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:05.941 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.079330+0000", 2024-03-19T11:32:05.941 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6028e-05 2024-03-19T11:32:05.941 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.941 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:05.941 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.941 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.941 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.941 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2928 4.24 4:266590a5:::smithi03829856-16:head [rollback b] snapc 1b=[1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-19T11:32:05.941 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:18.054594+0000", 2024-03-19T11:32:05.941 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 47.808772101999999, 2024-03-19T11:32:05.941 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025781921999999999, 2024-03-19T11:32:05.941 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:05.941 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:05.941 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:05.942 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:05.942 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:05.942 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2928 2024-03-19T11:32:05.942 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.942 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:05.942 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.942 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:05.942 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054594+0000", 2024-03-19T11:32:05.942 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.942 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.942 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.942 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:05.942 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054594+0000", 2024-03-19T11:32:05.942 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.942 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.943 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.943 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:05.943 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054595+0000", 2024-03-19T11:32:05.943 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.018e-06 2024-03-19T11:32:05.943 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.943 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.943 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:05.943 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054601+0000", 2024-03-19T11:32:05.943 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2620000000000004e-06 2024-03-19T11:32:05.943 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.943 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.944 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:05.944 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054618+0000", 2024-03-19T11:32:05.944 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6877000000000001e-05 2024-03-19T11:32:05.944 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.944 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.944 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:05.944 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054632+0000", 2024-03-19T11:32:05.944 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4175000000000001e-05 2024-03-19T11:32:05.944 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.944 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.944 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.944 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054676+0000", 2024-03-19T11:32:05.944 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3659000000000003e-05 2024-03-19T11:32:05.944 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.944 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.944 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:32:05.945 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054679+0000", 2024-03-19T11:32:05.945 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0649999999999999e-06 2024-03-19T11:32:05.945 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.945 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.945 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.945 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.064021+0000", 2024-03-19T11:32:05.945 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093424210000000001 2024-03-19T11:32:05.945 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.945 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.945 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:05.945 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.064158+0000", 2024-03-19T11:32:05.945 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013652099999999999 2024-03-19T11:32:05.945 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.945 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.945 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:32:05.946 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.072763+0000", 2024-03-19T11:32:05.946 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.008605784 2024-03-19T11:32:05.946 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.946 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.946 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:32:05.946 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.079171+0000", 2024-03-19T11:32:05.946 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0064071320000000003 2024-03-19T11:32:05.946 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.946 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.946 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:05.946 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.080257+0000", 2024-03-19T11:32:05.946 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0010866280000000001 2024-03-19T11:32:05.946 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.946 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.947 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:32:05.947 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.080324+0000", 2024-03-19T11:32:05.947 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6976000000000004e-05 2024-03-19T11:32:05.947 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.947 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.947 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:05.947 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.080376+0000", 2024-03-19T11:32:05.947 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1403999999999998e-05 2024-03-19T11:32:05.947 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.947 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:05.947 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.947 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.947 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.947 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2929 4.24 4:266590a5:::smithi03829856-16:head [append 0~0] snapc 1b=[1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-19T11:32:05.947 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:18.054649+0000", 2024-03-19T11:32:05.948 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 47.808716635000003, 2024-03-19T11:32:05.948 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026690519999999999, 2024-03-19T11:32:05.948 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:05.948 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:05.948 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:05.948 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:05.948 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:05.948 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2929 2024-03-19T11:32:05.948 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.948 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:05.948 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.948 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:05.948 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054649+0000", 2024-03-19T11:32:05.948 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.948 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.948 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.949 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:05.949 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054649+0000", 2024-03-19T11:32:05.949 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:05.949 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.949 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.949 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:05.949 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054650+0000", 2024-03-19T11:32:05.949 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5199999999999995e-07 2024-03-19T11:32:05.949 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.949 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.949 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:05.949 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054655+0000", 2024-03-19T11:32:05.949 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1529999999999996e-06 2024-03-19T11:32:05.949 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.950 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.950 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:05.950 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054670+0000", 2024-03-19T11:32:05.950 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5029e-05 2024-03-19T11:32:05.950 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.950 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.950 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:05.950 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054683+0000", 2024-03-19T11:32:05.950 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2670000000000001e-05 2024-03-19T11:32:05.950 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.950 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.950 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.950 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054756+0000", 2024-03-19T11:32:05.950 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3559000000000005e-05 2024-03-19T11:32:05.951 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.951 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.951 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:32:05.951 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054760+0000", 2024-03-19T11:32:05.951 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1659999999999998e-06 2024-03-19T11:32:05.951 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.951 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.951 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:05.951 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.075329+0000", 2024-03-19T11:32:05.951 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020569918999999999 2024-03-19T11:32:05.951 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.951 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.951 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:05.951 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.075469+0000", 2024-03-19T11:32:05.951 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013990000000000001 2024-03-19T11:32:05.952 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.952 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.952 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:32:05.952 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.075730+0000", 2024-03-19T11:32:05.952 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00026011300000000002 2024-03-19T11:32:05.952 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.952 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.952 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:32:05.952 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.079918+0000", 2024-03-19T11:32:05.952 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0041886279999999998 2024-03-19T11:32:05.952 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.952 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.952 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:05.952 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.081271+0000", 2024-03-19T11:32:05.952 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0013529449999999999 2024-03-19T11:32:05.953 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.953 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.953 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:32:05.953 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.081308+0000", 2024-03-19T11:32:05.953 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6437999999999999e-05 2024-03-19T11:32:05.953 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:05.953 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:05.953 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:05.953 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.081340+0000", 2024-03-19T11:32:05.953 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2147999999999998e-05 2024-03-19T11:32:05.953 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.953 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:05.953 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.953 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:05.953 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:05.954 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:32:05.954 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-19T11:32:05.955 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:32:05.965+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:32:05.956 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-19T11:32:06.060 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:32:06.067+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:32:06.074 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:32:06.081+0000 7f1837407640 -1 osd.6 144 handle_osd_map injecting map crc failure 2024-03-19T11:32:06.075 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:32:06.083+0000 7f6712f92640 -1 osd.2 144 handle_osd_map injecting map crc failure 2024-03-19T11:32:06.077 INFO:tasks.rados.rados.0.smithi038.stdout:1823: done (6 left) 2024-03-19T11:32:06.077 INFO:tasks.rados.rados.0.smithi038.stdout:1824: snap_remove snap 28 2024-03-19T11:32:06.086 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:32:06.093+0000 7f923da1c640 -1 osd.7 144 handle_osd_map injecting map crc failure 2024-03-19T11:32:06.113 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-19T11:32:06.113 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-19T11:32:06.113 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-19T11:32:06.113 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 0.000000 2024-03-19T11:32:06.113 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 0 2024-03-19T11:32:06.137 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:32:06.138 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:32:06.138 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:32:06.138 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:32:06.154 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-19T11:32:06.161 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:32:06.169+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:32:06.163 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-03-19T11:32:06.171 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:32:06.172 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:32:06.192 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:32:06.192 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:32:06.209 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:32:06.209 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:32:06.227 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:32:06.227 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:32:06.248 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:32:06.250 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:32:06.261 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:32:06.271+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:32:06.286 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:32:06.286 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:32:06.293 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:06.293 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:06.309 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:32:06.309 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:32:06.310 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-19T11:32:06.345 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:32:06.345 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:32:06.365 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:32:06.371+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:32:06.463 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:06.464 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:06.464 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:32:06.473+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:32:06.478 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-19T11:32:06.565 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:32:06.573+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:32:06.598 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:06.598 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:06.613 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-19T11:32:06.666 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:32:06.675+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:32:06.729 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:06.730 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:06.744 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-19T11:32:06.766 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:32:06.777+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:32:06.868 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:32:06.877+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:32:06.917 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:06.917 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:06.932 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-19T11:32:06.968 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:32:06.975+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:32:07.064 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:07.064 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:07.069 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:32:07.077+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:32:07.078 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-19T11:32:07.085 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-03-19T11:32:07.094 INFO:tasks.rados.rados.0.smithi038.stdout:1824: done (6 left) 2024-03-19T11:32:07.095 INFO:tasks.rados.rados.0.smithi038.stdout:1825: tier_evict oid 34 2024-03-19T11:32:07.095 INFO:tasks.rados.rados.0.smithi038.stdout:1825: tier_evict oid 34 snap -1 2024-03-19T11:32:07.095 INFO:tasks.rados.rados.0.smithi038.stdout:1826: read oid 39 snap -1 2024-03-19T11:32:07.095 INFO:tasks.rados.rados.0.smithi038.stdout:1826: expect deleted 2024-03-19T11:32:07.095 INFO:tasks.rados.rados.0.smithi038.stdout:1827: read oid 24 snap 21 2024-03-19T11:32:07.095 INFO:tasks.rados.rados.0.smithi038.stdout:1827: expect (ObjNum 23 snap 0 seq_num 23) 2024-03-19T11:32:07.095 INFO:tasks.rados.rados.0.smithi038.stdout:1828: set_chunk oid 11 target oid 2024-03-19T11:32:07.095 INFO:tasks.rados.rados.0.smithi038.stdout:1828: set_chunk oid 11 offset: 0 length: 4000000 target oid smithi03829856- offset: 0 snap -1 2024-03-19T11:32:07.095 INFO:tasks.rados.rados.0.smithi038.stdout:1829: read oid 7 snap -1 2024-03-19T11:32:07.095 INFO:tasks.rados.rados.0.smithi038.stdout:1829: expect (ObjNum 304 snap 39 seq_num 304) 2024-03-19T11:32:07.095 INFO:tasks.rados.rados.0.smithi038.stdout:1830: tier_promote oid 3 2024-03-19T11:32:07.095 INFO:tasks.rados.rados.0.smithi038.stdout:1831: rollback oid 26 current snap is 40 2024-03-19T11:32:07.095 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 26 to 21 2024-03-19T11:32:07.095 INFO:tasks.rados.rados.0.smithi038.stdout:1832: tier_flush oid 47 2024-03-19T11:32:07.096 INFO:tasks.rados.rados.0.smithi038.stdout:1832: tier_flush oid 47 snap -1 2024-03-19T11:32:07.096 INFO:tasks.rados.rados.0.smithi038.stdout:1833: snap_create 2024-03-19T11:32:07.096 INFO:tasks.rados.rados.0.smithi038.stdout:1816: finishing write tid 1 to smithi03829856-15 2024-03-19T11:32:07.096 INFO:tasks.rados.rados.0.smithi038.stdout:1816: finishing write tid 2 to smithi03829856-15 2024-03-19T11:32:07.096 INFO:tasks.rados.rados.0.smithi038.stdout:1816: finishing write tid 3 to smithi03829856-15 2024-03-19T11:32:07.096 INFO:tasks.rados.rados.0.smithi038.stdout:1816: finishing write tid 4 to smithi03829856-15 2024-03-19T11:32:07.096 INFO:tasks.rados.rados.0.smithi038.stdout:1816: finishing write tid 5 to smithi03829856-15 2024-03-19T11:32:07.096 INFO:tasks.rados.rados.0.smithi038.stdout:1816: oid 15 updating version 0 to 87 2024-03-19T11:32:07.096 INFO:tasks.rados.rados.0.smithi038.stdout:1816: oid 15 updating version 87 to 90 2024-03-19T11:32:07.096 INFO:tasks.rados.rados.0.smithi038.stdout:1816: oid 15 version 90 is already newer than 88 2024-03-19T11:32:07.096 INFO:tasks.rados.rados.0.smithi038.stdout:1816: oid 15 version 90 is already newer than 89 2024-03-19T11:32:07.096 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 15 v 90 (ObjNum 341 snap 40 seq_num 341) dirty exists 2024-03-19T11:32:07.096 INFO:tasks.rados.rados.0.smithi038.stdout:1816: left oid 15 (ObjNum 341 snap 40 seq_num 341) 2024-03-19T11:32:07.096 INFO:tasks.rados.rados.0.smithi038.stdout:1818: finishing write tid 1 to smithi03829856-32 2024-03-19T11:32:07.096 INFO:tasks.rados.rados.0.smithi038.stdout:1818: finishing write tid 2 to smithi03829856-32 2024-03-19T11:32:07.096 INFO:tasks.rados.rados.0.smithi038.stdout:1818: finishing write tid 3 to smithi03829856-32 2024-03-19T11:32:07.097 INFO:tasks.rados.rados.0.smithi038.stdout:1818: finishing write tid 4 to smithi03829856-32 2024-03-19T11:32:07.097 INFO:tasks.rados.rados.0.smithi038.stdout:1818: finishing write tid 5 to smithi03829856-32 2024-03-19T11:32:07.097 INFO:tasks.rados.rados.0.smithi038.stdout:1818: finishing write tid 6 to smithi03829856-32 2024-03-19T11:32:07.097 INFO:tasks.rados.rados.0.smithi038.stdout:1818: oid 32 updating version 0 to 189 2024-03-19T11:32:07.097 INFO:tasks.rados.rados.0.smithi038.stdout:1818: oid 32 version 189 is already newer than 181 2024-03-19T11:32:07.097 INFO:tasks.rados.rados.0.smithi038.stdout:1818: oid 32 version 189 is already newer than 186 2024-03-19T11:32:07.097 INFO:tasks.rados.rados.0.smithi038.stdout:1818: oid 32 version 189 is already newer than 188 2024-03-19T11:32:07.097 INFO:tasks.rados.rados.0.smithi038.stdout:1818: oid 32 version 189 is already newer than 187 2024-03-19T11:32:07.097 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 32 v 189 (ObjNum 342 snap 40 seq_num 342) dirty exists 2024-03-19T11:32:07.097 INFO:tasks.rados.rados.0.smithi038.stdout:1818: left oid 32 (ObjNum 342 snap 40 seq_num 342) 2024-03-19T11:32:07.097 INFO:tasks.rados.rados.0.smithi038.stdout:1819: finishing write tid 1 to smithi03829856-46 2024-03-19T11:32:07.097 INFO:tasks.rados.rados.0.smithi038.stdout:1819: finishing write tid 2 to smithi03829856-46 2024-03-19T11:32:07.097 INFO:tasks.rados.rados.0.smithi038.stdout:1819: finishing write tid 3 to smithi03829856-46 2024-03-19T11:32:07.097 INFO:tasks.rados.rados.0.smithi038.stdout:1819: finishing write tid 4 to smithi03829856-46 2024-03-19T11:32:07.097 INFO:tasks.rados.rados.0.smithi038.stdout:1819: finishing write tid 5 to smithi03829856-46 2024-03-19T11:32:07.098 INFO:tasks.rados.rados.0.smithi038.stdout:1819: oid 46 updating version 0 to 280 2024-03-19T11:32:07.098 INFO:tasks.rados.rados.0.smithi038.stdout:1819: oid 46 version 280 is already newer than 277 2024-03-19T11:32:07.098 INFO:tasks.rados.rados.0.smithi038.stdout:1819: oid 46 version 280 is already newer than 278 2024-03-19T11:32:07.098 INFO:tasks.rados.rados.0.smithi038.stdout:1819: oid 46 version 280 is already newer than 279 2024-03-19T11:32:07.098 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 46 v 280 (ObjNum 343 snap 40 seq_num 343) dirty exists 2024-03-19T11:32:07.098 INFO:tasks.rados.rados.0.smithi038.stdout:1819: left oid 46 (ObjNum 343 snap 40 seq_num 343) 2024-03-19T11:32:07.098 INFO:tasks.rados.rados.0.smithi038.stdout:1820: finishing write tid 1 to smithi03829856-19 2024-03-19T11:32:07.098 INFO:tasks.rados.rados.0.smithi038.stdout:1820: finishing write tid 2 to smithi03829856-19 2024-03-19T11:32:07.098 INFO:tasks.rados.rados.0.smithi038.stdout:1820: finishing write tid 3 to smithi03829856-19 2024-03-19T11:32:07.098 INFO:tasks.rados.rados.0.smithi038.stdout:1820: finishing write tid 4 to smithi03829856-19 2024-03-19T11:32:07.099 INFO:tasks.rados.rados.0.smithi038.stdout:1820: finishing write tid 5 to smithi03829856-19 2024-03-19T11:32:07.099 INFO:tasks.rados.rados.0.smithi038.stdout:1820: oid 19 updating version 0 to 121 2024-03-19T11:32:07.099 INFO:tasks.rados.rados.0.smithi038.stdout:1820: oid 19 version 121 is already newer than 119 2024-03-19T11:32:07.099 INFO:tasks.rados.rados.0.smithi038.stdout:1820: oid 19 version 121 is already newer than 120 2024-03-19T11:32:07.099 INFO:tasks.rados.rados.0.smithi038.stdout:1820: oid 19 updating version 121 to 122 2024-03-19T11:32:07.099 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 19 v 122 (ObjNum 344 snap 40 seq_num 344) dirty exists 2024-03-19T11:32:07.099 INFO:tasks.rados.rados.0.smithi038.stdout:1820: left oid 19 (ObjNum 344 snap 40 seq_num 344) 2024-03-19T11:32:07.099 INFO:tasks.rados.rados.0.smithi038.stdout:1821: finishing set_chunk to oid 23 2024-03-19T11:32:07.099 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 23 set_chunk returned error code -95 offset: 174080 length: 1800192 2024-03-19T11:32:07.099 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 23 v 0 (ObjNum 316 snap 40 seq_num 316) dirty exists 2024-03-19T11:32:07.099 INFO:tasks.rados.rados.0.smithi038.stdout:1822: expect (ObjNum 299 snap 39 seq_num 299) 2024-03-19T11:32:07.099 INFO:teuthology.orchestra.run.smithi038.stderr:set osd.2 primary-affinity to 0 (802) 2024-03-19T11:32:07.100 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:32:07.101+0000 7fb17dd72640 -1 osd.4 145 handle_osd_map injecting map crc failure 2024-03-19T11:32:07.193 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:32:07.203+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:32:07.242 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:07.242 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:07.256 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-19T11:32:07.256 INFO:tasks.rados.rados.0.smithi038.stdout:1828: finishing set_chunk to oid 11 2024-03-19T11:32:07.257 INFO:tasks.rados.rados.0.smithi038.stdout:1828: got expected ERANGE (src dne) 2024-03-19T11:32:07.257 INFO:tasks.rados.rados.0.smithi038.stdout:1831: finishing rollback tid 0 to smithi03829856-26 2024-03-19T11:32:07.257 INFO:tasks.rados.rados.0.smithi038.stdout:1832: got (0) Success 2024-03-19T11:32:07.257 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 47 v 177 (ObjNum 273 snap 0 seq_num 0) clean dne 2024-03-19T11:32:07.257 INFO:tasks.rados.rados.0.smithi038.stdout:1830: got (0) Success 2024-03-19T11:32:07.257 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 3 v 141 (ObjNum 97 snap 0 seq_num 1) dirty dne 2024-03-19T11:32:07.257 INFO:tasks.rados.rados.0.smithi038.stdout:1827: expect (ObjNum 23 snap 0 seq_num 23) 2024-03-19T11:32:07.270 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:32:07.279+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:32:07.333 INFO:tasks.rados.rados.0.smithi038.stdout:1829: expect (ObjNum 304 snap 39 seq_num 304) 2024-03-19T11:32:07.371 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:32:07.379+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:32:07.385 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:07.385 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:07.399 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-19T11:32:07.472 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:32:07.479+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:32:07.521 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:07.522 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:07.536 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-19T11:32:07.567 INFO:tasks.rados.rados.0.smithi038.stdout:1831: finishing rollback tid 1 to smithi03829856-26 2024-03-19T11:32:07.567 INFO:tasks.rados.rados.0.smithi038.stdout:1831: finishing rollback tid 2 to smithi03829856-26 2024-03-19T11:32:07.567 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 26 v 90 (ObjNum 123 snap 15 seq_num 123) dirty exists 2024-03-19T11:32:07.583 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:32:07.593+0000 7fb18b3b0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25234) UID: 0 2024-03-19T11:32:07.659 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:07.659 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:07.674 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-19T11:32:07.674 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:32:07.683+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:32:07.775 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:32:07.783+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:32:07.837 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:07.838 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:07.853 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-19T11:32:07.921 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:32:07.929+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:32:08.022 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:32:08.029+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:32:08.039 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:08.040 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:08.054 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-19T11:32:08.064 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:32:08.071+0000 7f70c4fdb640 -1 osd.0 146 handle_osd_map injecting map crc failure 2024-03-19T11:32:08.085 INFO:tasks.rados.rados.0.smithi038.stdout:1816: done (14 left) 2024-03-19T11:32:08.086 INFO:tasks.rados.rados.0.smithi038.stdout:1818: done (13 left) 2024-03-19T11:32:08.086 INFO:tasks.rados.rados.0.smithi038.stdout:1819: done (12 left) 2024-03-19T11:32:08.086 INFO:tasks.rados.rados.0.smithi038.stdout:1820: done (11 left) 2024-03-19T11:32:08.086 INFO:tasks.rados.rados.0.smithi038.stdout:1821: done (10 left) 2024-03-19T11:32:08.086 INFO:tasks.rados.rados.0.smithi038.stdout:1822: done (9 left) 2024-03-19T11:32:08.086 INFO:tasks.rados.rados.0.smithi038.stdout:1826: done (8 left) 2024-03-19T11:32:08.087 INFO:tasks.rados.rados.0.smithi038.stdout:1827: done (7 left) 2024-03-19T11:32:08.087 INFO:tasks.rados.rados.0.smithi038.stdout:1828: done (6 left) 2024-03-19T11:32:08.087 INFO:tasks.rados.rados.0.smithi038.stdout:1829: done (5 left) 2024-03-19T11:32:08.087 INFO:tasks.rados.rados.0.smithi038.stdout:1830: done (4 left) 2024-03-19T11:32:08.087 INFO:tasks.rados.rados.0.smithi038.stdout:1831: done (3 left) 2024-03-19T11:32:08.087 INFO:tasks.rados.rados.0.smithi038.stdout:1832: done (2 left) 2024-03-19T11:32:08.087 INFO:tasks.rados.rados.0.smithi038.stdout:1833: done (1 left) 2024-03-19T11:32:08.087 INFO:tasks.rados.rados.0.smithi038.stdout:1834: set_chunk oid 38 target oid 2024-03-19T11:32:08.101 INFO:tasks.rados.rados.0.smithi038.stdout:1834: set_chunk oid 38 offset: 525312 length: 594944 target oid 72f81af5065bc3d7543d146edb85a3a40345768fd47243009c19b24bdf0f6608 offset: 0 snap 37 2024-03-19T11:32:08.102 INFO:tasks.rados.rados.0.smithi038.stdout:1835: write (excl) oid 5 current snap is 41 2024-03-19T11:32:08.102 INFO:tasks.rados.rados.0.smithi038.stdout:1825: got (22) Invalid argument 2024-03-19T11:32:08.102 INFO:tasks.rados.rados.0.smithi038.stdout:1835: seq_num 345 ranges {598415=796790,1922170=745784,3141403=415315} 2024-03-19T11:32:08.109 INFO:tasks.rados.rados.0.smithi038.stdout:1835: writing smithi03829856-5 from 598415 to 1395205 tid 1 2024-03-19T11:32:08.118 INFO:tasks.rados.rados.0.smithi038.stdout:1835: writing smithi03829856-5 from 1922170 to 2667954 tid 2 2024-03-19T11:32:08.123 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:32:08.131+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:32:08.125 INFO:tasks.rados.rados.0.smithi038.stdout:1835: writing smithi03829856-5 from 3141403 to 3556718 tid 3 2024-03-19T11:32:08.125 INFO:tasks.rados.rados.0.smithi038.stdout:1825: done (2 left) 2024-03-19T11:32:08.125 INFO:tasks.rados.rados.0.smithi038.stdout:1836: read oid 28 snap -1 2024-03-19T11:32:08.125 INFO:tasks.rados.rados.0.smithi038.stdout:1836: expect (ObjNum 87 snap 8 seq_num 87) 2024-03-19T11:32:08.126 INFO:tasks.rados.rados.0.smithi038.stdout:1837: tier_evict oid 2 2024-03-19T11:32:08.126 INFO:tasks.rados.rados.0.smithi038.stdout:1837: tier_evict oid 2 snap -1 2024-03-19T11:32:08.126 INFO:tasks.rados.rados.0.smithi038.stdout:1834: finishing set_chunk to oid 38 2024-03-19T11:32:08.126 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 38 set_chunk returned error code -95 offset: 525312 length: 594944 2024-03-19T11:32:08.126 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 38 v 0 (ObjNum 322 snap 40 seq_num 322) dirty exists 2024-03-19T11:32:08.126 INFO:tasks.rados.rados.0.smithi038.stdout:1834: done (3 left) 2024-03-19T11:32:08.127 INFO:tasks.rados.rados.0.smithi038.stdout:1838: snap_remove snap 21 2024-03-19T11:32:08.221 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:08.221 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:08.225 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:32:08.235+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:32:08.235 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-19T11:32:08.327 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:32:08.335+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:32:08.352 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:08.352 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:08.365 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-19T11:32:08.367 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-19T11:32:08.429 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:32:08.437+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:32:08.484 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:08.484 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:08.498 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-19T11:32:08.525 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:32:08.525 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:32:08.531 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:32:08.539+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:32:08.542 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:32:08.542 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:32:08.559 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:32:08.560 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:32:08.576 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:32:08.576 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:32:08.594 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-19T11:32:08.594 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-19T11:32:08.615 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:32:08.615 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:32:08.633 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:32:08.641+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:32:08.636 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:32:08.636 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:32:08.658 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:32:08.658 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:32:08.667 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:32:08.667 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:32:08.667 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:32:08.667 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:32:08.668 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:32:08.681 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-19T11:32:08.768 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:32:08.777+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:32:08.835 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:32:08.843+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:32:08.850 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:32:08.850 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:32:08.850 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:32:08.850 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:32:08.850 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.850 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:32:08.850 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:44.200920+0000", 2024-03-19T11:32:08.850 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 144.65704946, 2024-03-19T11:32:08.850 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.80881896499999995, 2024-03-19T11:32:08.850 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:08.851 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:08.851 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:08.851 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:32:08.851 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:32:08.851 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1 2024-03-19T11:32:08.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.851 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:08.851 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.851 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:08.851 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200920+0000", 2024-03-19T11:32:08.851 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:08.851 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.851 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.851 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:08.851 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200920+0000", 2024-03-19T11:32:08.852 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:08.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.852 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.852 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:08.852 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200923+0000", 2024-03-19T11:32:08.852 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6469999999999998e-06 2024-03-19T11:32:08.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.852 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.852 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:08.852 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200937+0000", 2024-03-19T11:32:08.852 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.399e-05 2024-03-19T11:32:08.852 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.852 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.852 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:08.852 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200960+0000", 2024-03-19T11:32:08.853 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2603e-05 2024-03-19T11:32:08.853 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.853 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.853 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:08.853 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200978+0000", 2024-03-19T11:32:08.853 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.791e-05 2024-03-19T11:32:08.853 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.853 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.853 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:08.853 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.201081+0000", 2024-03-19T11:32:08.853 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010344900000000001 2024-03-19T11:32:08.853 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.853 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.853 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:32:08.853 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.201096+0000", 2024-03-19T11:32:08.854 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4443e-05 2024-03-19T11:32:08.854 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.854 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.854 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:08.854 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.005978+0000", 2024-03-19T11:32:08.854 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.80488281500000003 2024-03-19T11:32:08.854 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.854 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.854 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:08.854 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.006267+0000", 2024-03-19T11:32:08.854 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000288688 2024-03-19T11:32:08.854 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.854 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.854 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:32:08.854 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.006534+0000", 2024-03-19T11:32:08.855 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026725799999999998 2024-03-19T11:32:08.855 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.855 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.855 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:32:08.855 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.008457+0000", 2024-03-19T11:32:08.855 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0019227820000000001 2024-03-19T11:32:08.855 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.855 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.855 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:08.855 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.009570+0000", 2024-03-19T11:32:08.855 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011131750000000001 2024-03-19T11:32:08.855 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.855 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.855 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:08.855 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.009670+0000", 2024-03-19T11:32:08.856 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9458999999999998e-05 2024-03-19T11:32:08.856 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.856 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.856 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:08.856 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.009739+0000", 2024-03-19T11:32:08.856 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9746000000000006e-05 2024-03-19T11:32:08.856 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:08.856 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:08.856 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:08.856 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.856 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.856 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:32:08.856 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.339898+0000", 2024-03-19T11:32:08.856 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 143.51807235300001, 2024-03-19T11:32:08.857 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019501793, 2024-03-19T11:32:08.857 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:08.857 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:08.857 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:08.857 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:32:08.857 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:32:08.857 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113 2024-03-19T11:32:08.857 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.857 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:08.857 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.857 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:08.857 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339898+0000", 2024-03-19T11:32:08.857 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:08.857 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.857 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.858 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:08.858 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339898+0000", 2024-03-19T11:32:08.858 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:08.858 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.858 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.858 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:08.858 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339899+0000", 2024-03-19T11:32:08.858 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.353e-06 2024-03-19T11:32:08.858 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.858 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.858 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:08.858 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339907+0000", 2024-03-19T11:32:08.858 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3000000000000002e-06 2024-03-19T11:32:08.858 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.858 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.859 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:08.859 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339924+0000", 2024-03-19T11:32:08.859 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6906999999999999e-05 2024-03-19T11:32:08.859 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.859 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.859 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:08.859 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339933+0000", 2024-03-19T11:32:08.859 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8969999999999997e-06 2024-03-19T11:32:08.859 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.859 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.859 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:08.859 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352239+0000", 2024-03-19T11:32:08.859 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012305814999999999 2024-03-19T11:32:08.859 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.859 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.860 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:08.860 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352466+0000", 2024-03-19T11:32:08.860 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022684100000000001 2024-03-19T11:32:08.860 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.860 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.860 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:32:08.860 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352816+0000", 2024-03-19T11:32:08.860 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00035030499999999997 2024-03-19T11:32:08.860 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.860 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.860 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:32:08.860 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.358956+0000", 2024-03-19T11:32:08.860 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061400079999999998 2024-03-19T11:32:08.860 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.860 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.861 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:08.861 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359118+0000", 2024-03-19T11:32:08.861 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016237599999999999 2024-03-19T11:32:08.861 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.861 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.861 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:08.861 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359287+0000", 2024-03-19T11:32:08.861 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000168488 2024-03-19T11:32:08.861 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.861 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.861 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:08.861 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359399+0000", 2024-03-19T11:32:08.861 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000112503 2024-03-19T11:32:08.861 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:08.861 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:08.862 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:08.862 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.862 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.862 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:32:08.862 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.339947+0000", 2024-03-19T11:32:08.862 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 143.518022586, 2024-03-19T11:32:08.862 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021923069999999999, 2024-03-19T11:32:08.862 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:08.862 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:08.862 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:08.862 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:32:08.862 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:32:08.862 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 114 2024-03-19T11:32:08.862 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.863 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:08.863 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.863 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:08.863 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339947+0000", 2024-03-19T11:32:08.863 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:08.863 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.863 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.863 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:08.863 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339947+0000", 2024-03-19T11:32:08.863 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:08.863 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.863 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.863 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:08.863 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339948+0000", 2024-03-19T11:32:08.863 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6000000000000003e-07 2024-03-19T11:32:08.864 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.864 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.864 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:08.864 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339956+0000", 2024-03-19T11:32:08.864 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1559999999999995e-06 2024-03-19T11:32:08.864 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.865 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.865 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:08.865 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339967+0000", 2024-03-19T11:32:08.865 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0669999999999999e-05 2024-03-19T11:32:08.865 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.865 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.865 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:08.865 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339975+0000", 2024-03-19T11:32:08.866 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0099999999999995e-06 2024-03-19T11:32:08.866 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.866 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.867 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:08.867 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.353672+0000", 2024-03-19T11:32:08.867 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013696583999999999 2024-03-19T11:32:08.867 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.867 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.867 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:08.867 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.353808+0000", 2024-03-19T11:32:08.867 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013652299999999999 2024-03-19T11:32:08.867 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.868 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.868 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:32:08.868 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.354036+0000", 2024-03-19T11:32:08.868 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022744799999999999 2024-03-19T11:32:08.868 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.868 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.868 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:32:08.868 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.360480+0000", 2024-03-19T11:32:08.869 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0064441630000000001 2024-03-19T11:32:08.869 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.869 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.869 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:08.869 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361722+0000", 2024-03-19T11:32:08.869 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001242754 2024-03-19T11:32:08.869 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.869 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.869 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:08.870 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361807+0000", 2024-03-19T11:32:08.870 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4605999999999996e-05 2024-03-19T11:32:08.870 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.870 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.870 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:08.870 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361870+0000", 2024-03-19T11:32:08.870 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3396000000000001e-05 2024-03-19T11:32:08.870 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:08.870 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:08.870 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:08.870 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.870 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.870 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:32:08.871 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.340017+0000", 2024-03-19T11:32:08.871 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 143.51795326999999, 2024-03-19T11:32:08.871 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022169355000000002, 2024-03-19T11:32:08.871 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:08.871 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:08.871 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:08.871 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:32:08.871 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:32:08.871 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 115 2024-03-19T11:32:08.871 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.871 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:08.871 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.871 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:08.872 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340017+0000", 2024-03-19T11:32:08.872 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:08.872 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.872 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.872 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:08.872 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340017+0000", 2024-03-19T11:32:08.872 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:08.872 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.872 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.872 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:08.872 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340018+0000", 2024-03-19T11:32:08.872 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.043e-06 2024-03-19T11:32:08.872 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.872 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.872 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:08.873 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340139+0000", 2024-03-19T11:32:08.873 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121502 2024-03-19T11:32:08.873 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.873 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.873 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:08.873 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340157+0000", 2024-03-19T11:32:08.873 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7382999999999999e-05 2024-03-19T11:32:08.873 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.873 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.873 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:08.873 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340166+0000", 2024-03-19T11:32:08.873 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2469999999999997e-06 2024-03-19T11:32:08.873 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.873 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.874 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:08.874 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.357470+0000", 2024-03-19T11:32:08.874 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017304119999999999 2024-03-19T11:32:08.874 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.874 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.874 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:08.874 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.357671+0000", 2024-03-19T11:32:08.874 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000200798 2024-03-19T11:32:08.874 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.874 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.874 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:32:08.874 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.358002+0000", 2024-03-19T11:32:08.874 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033090099999999998 2024-03-19T11:32:08.874 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.875 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.875 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:32:08.875 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361892+0000", 2024-03-19T11:32:08.875 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00389018 2024-03-19T11:32:08.875 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.875 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.875 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:08.875 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361983+0000", 2024-03-19T11:32:08.875 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0976000000000004e-05 2024-03-19T11:32:08.875 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.875 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.875 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:08.875 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362071+0000", 2024-03-19T11:32:08.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8022999999999994e-05 2024-03-19T11:32:08.876 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.876 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.876 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:08.876 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362186+0000", 2024-03-19T11:32:08.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115182 2024-03-19T11:32:08.876 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:08.876 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:08.876 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:08.876 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.876 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.876 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:32:08.876 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.340183+0000", 2024-03-19T11:32:08.876 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 143.517786438, 2024-03-19T11:32:08.876 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022279841000000002, 2024-03-19T11:32:08.877 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:08.877 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:08.877 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:08.877 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:32:08.877 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:32:08.877 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 116 2024-03-19T11:32:08.877 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.877 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:08.877 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.877 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:08.877 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340183+0000", 2024-03-19T11:32:08.877 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:08.877 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.877 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.878 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:08.878 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340183+0000", 2024-03-19T11:32:08.878 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:08.878 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.878 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.878 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:08.878 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340184+0000", 2024-03-19T11:32:08.878 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9000000000000005e-07 2024-03-19T11:32:08.878 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.878 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.878 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:08.878 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340191+0000", 2024-03-19T11:32:08.878 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0360000000000003e-06 2024-03-19T11:32:08.878 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.879 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.879 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:08.879 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340201+0000", 2024-03-19T11:32:08.879 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0844e-05 2024-03-19T11:32:08.879 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.879 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.879 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:08.879 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340209+0000", 2024-03-19T11:32:08.879 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7360000000000001e-06 2024-03-19T11:32:08.879 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.879 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.879 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:08.879 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359535+0000", 2024-03-19T11:32:08.880 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019325546999999998 2024-03-19T11:32:08.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.880 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.880 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:08.880 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359672+0000", 2024-03-19T11:32:08.880 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013706600000000001 2024-03-19T11:32:08.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.880 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.880 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:32:08.881 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359905+0000", 2024-03-19T11:32:08.881 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023334500000000001 2024-03-19T11:32:08.881 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.881 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.881 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:08.881 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362306+0000", 2024-03-19T11:32:08.881 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002401041 2024-03-19T11:32:08.881 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.881 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.881 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:32:08.881 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362329+0000", 2024-03-19T11:32:08.881 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3283999999999999e-05 2024-03-19T11:32:08.881 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.882 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.882 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:08.882 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362402+0000", 2024-03-19T11:32:08.882 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2459000000000005e-05 2024-03-19T11:32:08.882 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.882 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.882 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:08.882 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362463+0000", 2024-03-19T11:32:08.882 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1493000000000002e-05 2024-03-19T11:32:08.882 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:08.882 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:08.882 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:08.882 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.882 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.883 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:32:08.883 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.340222+0000", 2024-03-19T11:32:08.883 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 143.51774785500001, 2024-03-19T11:32:08.883 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022500218999999998, 2024-03-19T11:32:08.883 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:08.883 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:08.883 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:08.883 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:32:08.883 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:32:08.883 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 117 2024-03-19T11:32:08.883 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.883 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:08.883 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.883 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:08.884 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340222+0000", 2024-03-19T11:32:08.884 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:08.884 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.884 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.884 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:08.884 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340222+0000", 2024-03-19T11:32:08.884 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:08.884 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.884 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.884 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:08.884 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340223+0000", 2024-03-19T11:32:08.884 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7999999999999995e-07 2024-03-19T11:32:08.884 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.884 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.885 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:08.885 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340229+0000", 2024-03-19T11:32:08.885 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0399999999999998e-06 2024-03-19T11:32:08.885 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.885 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.885 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:08.885 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340239+0000", 2024-03-19T11:32:08.885 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0190000000000001e-05 2024-03-19T11:32:08.885 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.885 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.885 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:08.885 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340250+0000", 2024-03-19T11:32:08.885 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0720000000000001e-05 2024-03-19T11:32:08.885 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.886 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.886 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:08.886 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.360587+0000", 2024-03-19T11:32:08.886 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020337542 2024-03-19T11:32:08.886 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.886 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.886 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:08.886 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.360818+0000", 2024-03-19T11:32:08.886 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023120500000000001 2024-03-19T11:32:08.886 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.886 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.886 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:32:08.886 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361218+0000", 2024-03-19T11:32:08.887 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000399637 2024-03-19T11:32:08.887 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.887 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.887 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:32:08.887 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362483+0000", 2024-03-19T11:32:08.887 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012649200000000001 2024-03-19T11:32:08.887 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.887 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.887 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:08.887 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362538+0000", 2024-03-19T11:32:08.887 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4490000000000002e-05 2024-03-19T11:32:08.887 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.887 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.887 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:08.888 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362617+0000", 2024-03-19T11:32:08.888 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9839000000000006e-05 2024-03-19T11:32:08.888 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.888 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.888 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:08.888 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362722+0000", 2024-03-19T11:32:08.888 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104956 2024-03-19T11:32:08.888 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:08.888 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:08.888 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:08.888 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.888 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.888 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.15 51/50 [PushOp(4:afa9f208:::smithi03829856-35:head, version: 28'46, data_included: [571438~623878,1806623~424698], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:afa9f208:::smithi03829856-35:head@28'46, size: 2571438, copy_subset: [(0, 2571438)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2231321, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:af9d0d56:::smithi03829856-20:head, version: 35'55, data_included: [655360~589824,1900544~418789], data_size: 1008613, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:af9d0d56:::smithi03829856-20:head@35'55, size: 2319333, copy_subset: [(0, 2319333)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2319333, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ad3ad1b3:::smithi03829856-21:b, version: 47'66, data_included: [736140~537686,2020581~315559], data_size: 853245, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:ad3ad1b3:::smithi03829856-21:b@47'66, size: 2336140, copy_subset: [(0, 2336140)], clone_subset: {}, snapset: b=[]:{b: [b, a, 6, 5]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2336140, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ad3ad1b3:::smithi03829856-21:head, version: 47'70, data_included: [531121~742705,1711057~305871], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:ad3ad1b3:::smithi03829856-21:head@47'70, size: 3731121, copy_subset: [(0, 3731121)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2016928, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ac9cd3a3:::smithi03829856-8:head, version: 49'72, data_included: [517649~620182,1776668~428394], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:ac9cd3a3:::smithi03829856-8:head@49'72, size: 3008420, copy_subset: [(0, 3008420)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2205062, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:aa3303a5:::smithi03829856-17:b, version: 49'73, data_included: [614400~692224,1903006~356352], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:aa3303a5:::smithi03829856-17:b@49'73, size: 2268912, copy_subset: [(0, 2268912)], clone_subset: {}, snapset: b=[]:{b: [b, a, 6, 5]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2259358, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:aa3303a5:::smithi03829856-17:head, version: 49'74, data_included: [413472~893152,1818036~155424], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:aa3303a5:::smithi03829856-17:head@49'74, size: 3613472, copy_subset: [(0, 3613472)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1973460, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:32:08.889 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:28.058610+0000", 2024-03-19T11:32:08.889 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 100.79935968300001, 2024-03-19T11:32:08.889 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027100032, 2024-03-19T11:32:08.889 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:08.889 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:32:08.889 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:08.889 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.889 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:08.889 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.058610+0000", 2024-03-19T11:32:08.889 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:08.889 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.889 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.889 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:08.890 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.058610+0000", 2024-03-19T11:32:08.890 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:08.890 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.890 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.890 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:08.890 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.058611+0000", 2024-03-19T11:32:08.890 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1599999999999999e-06 2024-03-19T11:32:08.890 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.890 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.890 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:08.890 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.063874+0000", 2024-03-19T11:32:08.890 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052625270000000004 2024-03-19T11:32:08.890 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.890 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.891 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:08.891 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.063935+0000", 2024-03-19T11:32:08.891 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0594000000000003e-05 2024-03-19T11:32:08.891 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.891 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.891 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:08.891 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064024+0000", 2024-03-19T11:32:08.891 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9273999999999999e-05 2024-03-19T11:32:08.891 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.891 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.891 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:08.891 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064385+0000", 2024-03-19T11:32:08.891 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00036129100000000002 2024-03-19T11:32:08.892 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.892 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.892 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:08.892 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064437+0000", 2024-03-19T11:32:08.892 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2346000000000003e-05 2024-03-19T11:32:08.892 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.892 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.892 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:08.892 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085710+0000", 2024-03-19T11:32:08.892 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021272840000000001 2024-03-19T11:32:08.892 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:08.892 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:08.892 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:08.892 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.893 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.15 e51,50 4:ac9cd3a3:::smithi03829856-8:4,49'71)", 2024-03-19T11:32:08.893 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:28.064112+0000", 2024-03-19T11:32:08.893 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 100.79385785700001, 2024-03-19T11:32:08.893 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026480417999999999, 2024-03-19T11:32:08.893 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:08.893 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:32:08.893 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:08.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.893 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:08.893 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064112+0000", 2024-03-19T11:32:08.893 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:08.893 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.893 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:08.894 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064112+0000", 2024-03-19T11:32:08.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:08.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:08.894 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064113+0000", 2024-03-19T11:32:08.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3600000000000003e-07 2024-03-19T11:32:08.894 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.894 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.894 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:08.894 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064118+0000", 2024-03-19T11:32:08.894 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0549999999999999e-06 2024-03-19T11:32:08.895 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.895 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.895 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:08.895 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064128+0000", 2024-03-19T11:32:08.895 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0349e-05 2024-03-19T11:32:08.895 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.895 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.895 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:08.895 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064136+0000", 2024-03-19T11:32:08.895 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6249999999999998e-06 2024-03-19T11:32:08.895 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.895 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.895 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:08.895 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085773+0000", 2024-03-19T11:32:08.895 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021636829999999999 2024-03-19T11:32:08.896 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.896 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.896 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:08.896 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085788+0000", 2024-03-19T11:32:08.896 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5512999999999999e-05 2024-03-19T11:32:08.896 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.896 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.896 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:08.896 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.090592+0000", 2024-03-19T11:32:08.896 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0048044100000000003 2024-03-19T11:32:08.896 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:08.896 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:08.896 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:08.896 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.897 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.897 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.5 51/50 [PushOp(4:a38b9453:::smithi03829856-37:head, version: 28'45, data_included: [605052~774376,1923587~274200], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:a38b9453:::smithi03829856-37:head@28'45, size: 2605052, copy_subset: [(0, 2605052)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2197787, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:a0a50d04:::smithi03829856-13:b, version: 47'63, data_included: [601684~619341,1669078~429235], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a0a50d04:::smithi03829856-13:b@47'63, size: 2201684, copy_subset: [(0, 2201684)], clone_subset: {}, snapset: b=[]:{b: [b, a, 6, 5]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2098313, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:32:08.897 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:28.064324+0000", 2024-03-19T11:32:08.897 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 100.793646013, 2024-03-19T11:32:08.897 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027179020000000002, 2024-03-19T11:32:08.897 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:08.897 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:32:08.897 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:08.897 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.897 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:08.897 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064324+0000", 2024-03-19T11:32:08.897 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:08.897 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.897 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.898 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:08.898 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064324+0000", 2024-03-19T11:32:08.898 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:08.898 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.898 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.898 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:08.898 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064324+0000", 2024-03-19T11:32:08.898 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1199999999999998e-07 2024-03-19T11:32:08.898 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.898 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.898 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:08.898 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.065527+0000", 2024-03-19T11:32:08.898 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012028939999999999 2024-03-19T11:32:08.898 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.899 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.899 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:08.899 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.065552+0000", 2024-03-19T11:32:08.899 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4440999999999999e-05 2024-03-19T11:32:08.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.899 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.899 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:08.899 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.065568+0000", 2024-03-19T11:32:08.899 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6498e-05 2024-03-19T11:32:08.899 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.899 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.899 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:08.899 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085812+0000", 2024-03-19T11:32:08.899 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020243954000000002 2024-03-19T11:32:08.900 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.900 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.900 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:08.900 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085825+0000", 2024-03-19T11:32:08.900 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3298999999999999e-05 2024-03-19T11:32:08.900 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.900 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.900 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:08.900 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.091503+0000", 2024-03-19T11:32:08.900 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.005677522 2024-03-19T11:32:08.900 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:08.900 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:08.900 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:08.900 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.901 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.901 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.24 75/74 [PushOp(4:27909373:::smithi03829856-46:8, version: 46'59, data_included: [756315~767970,1933572~280606], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:8@46'59, size: 2756315, copy_subset: [(0, 2756315)], clone_subset: {}, snapset: 10=[]:{8: [6], c: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2214178, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:266590a5:::smithi03829856-16:e, version: 70'107, data_included: [652105~662359,1728171~386217], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:e@70'107, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 10=[]:{e: [d, b, 6]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:266590a5:::smithi03829856-16:head, version: 70'108, data_included: [652105~662359,1728171~386217], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:head@70'108, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:27909373:::smithi03829856-46:c, version: 70'109, data_included: [628595~895690,1807440~152886], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:c@70'109, size: 3428595, copy_subset: [(0, 3428595)], clone_subset: {}, snapset: 10=[]:{8: [6], c: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1960326, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:27909373:::smithi03829856-46:head, version: 70'110, data_included: [403401~454961,1584925~506039], data_size: 961000, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:head@70'110, size: 2403401, copy_subset: [(0, 2403401)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2403401, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:32:08.901 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.099172+0000", 2024-03-19T11:32:08.901 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 74.758797579000003, 2024-03-19T11:32:08.901 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.024528580000000001, 2024-03-19T11:32:08.901 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:08.901 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:32:08.901 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:08.901 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.901 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:08.901 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.099172+0000", 2024-03-19T11:32:08.901 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:08.901 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.901 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.901 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:08.902 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.099172+0000", 2024-03-19T11:32:08.902 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:08.902 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.902 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.902 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:08.902 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.099175+0000", 2024-03-19T11:32:08.902 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7750000000000001e-06 2024-03-19T11:32:08.902 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.902 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.902 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:08.902 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105219+0000", 2024-03-19T11:32:08.902 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0060440789999999999 2024-03-19T11:32:08.902 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.902 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.903 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:08.903 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105285+0000", 2024-03-19T11:32:08.903 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6260000000000006e-05 2024-03-19T11:32:08.903 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.903 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.903 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:08.903 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105366+0000", 2024-03-19T11:32:08.903 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.038e-05 2024-03-19T11:32:08.903 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.903 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.903 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:08.903 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105637+0000", 2024-03-19T11:32:08.903 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027085199999999997 2024-03-19T11:32:08.903 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.903 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.904 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:08.904 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105679+0000", 2024-03-19T11:32:08.904 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2088000000000001e-05 2024-03-19T11:32:08.904 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.904 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.904 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:08.904 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.123701+0000", 2024-03-19T11:32:08.904 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018022145999999999 2024-03-19T11:32:08.904 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:08.904 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:08.904 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:08.904 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.904 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.904 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.24 e75,74 4:266590a5:::smithi03829856-16:4,47'63,4:27909373:::smithi03829856-46:5,55'79,4:266590a5:::smithi03829856-16:10,67'105)", 2024-03-19T11:32:08.905 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.105437+0000", 2024-03-19T11:32:08.905 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 74.752532575999993, 2024-03-19T11:32:08.905 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.01860726, 2024-03-19T11:32:08.905 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:08.905 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:32:08.905 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:08.905 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.905 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:08.905 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105437+0000", 2024-03-19T11:32:08.905 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:08.905 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.905 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.905 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:08.905 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105437+0000", 2024-03-19T11:32:08.905 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:08.906 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.906 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.906 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:08.906 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105438+0000", 2024-03-19T11:32:08.906 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6799999999999999e-07 2024-03-19T11:32:08.906 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.906 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.906 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:08.906 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105445+0000", 2024-03-19T11:32:08.906 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3749999999999999e-06 2024-03-19T11:32:08.906 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.906 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.906 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:08.907 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105454+0000", 2024-03-19T11:32:08.907 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7909999999999999e-06 2024-03-19T11:32:08.907 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.907 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.907 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:08.907 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105465+0000", 2024-03-19T11:32:08.907 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.045e-05 2024-03-19T11:32:08.907 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.907 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.907 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:08.907 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.123753+0000", 2024-03-19T11:32:08.907 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0182884 2024-03-19T11:32:08.907 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.907 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.907 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:08.908 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.123765+0000", 2024-03-19T11:32:08.908 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2143e-05 2024-03-19T11:32:08.908 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.908 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.908 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:08.908 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.124045+0000", 2024-03-19T11:32:08.908 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027923299999999999 2024-03-19T11:32:08.908 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:08.908 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:08.908 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:08.908 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.908 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.908 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:3103 4.15 e104/76 4:afa9f208:::smithi03829856-35:head v 104'221, mlcod=101'216)", 2024-03-19T11:32:08.908 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.563593+0000", 2024-03-19T11:32:08.909 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 45.294377298000001, 2024-03-19T11:32:08.909 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026983566, 2024-03-19T11:32:08.909 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:08.909 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:08.909 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:08.909 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.909 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:08.909 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.563593+0000", 2024-03-19T11:32:08.909 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:08.909 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.909 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.909 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:08.909 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.563593+0000", 2024-03-19T11:32:08.909 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:08.910 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.910 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.910 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:08.910 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.563595+0000", 2024-03-19T11:32:08.910 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2859999999999998e-06 2024-03-19T11:32:08.910 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.910 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.910 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:08.910 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.564339+0000", 2024-03-19T11:32:08.910 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00074400099999999999 2024-03-19T11:32:08.910 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.910 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.910 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:08.910 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.564376+0000", 2024-03-19T11:32:08.911 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6628999999999998e-05 2024-03-19T11:32:08.911 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.911 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.911 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:08.911 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.564396+0000", 2024-03-19T11:32:08.911 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0596000000000001e-05 2024-03-19T11:32:08.911 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.911 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.911 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:08.911 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.566600+0000", 2024-03-19T11:32:08.911 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002204021 2024-03-19T11:32:08.911 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.911 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.911 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:08.912 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.566642+0000", 2024-03-19T11:32:08.912 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1902e-05 2024-03-19T11:32:08.912 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.912 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.912 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:08.912 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.590494+0000", 2024-03-19T11:32:08.912 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023851816000000001 2024-03-19T11:32:08.912 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.912 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.912 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:08.912 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.590576+0000", 2024-03-19T11:32:08.912 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2315e-05 2024-03-19T11:32:08.912 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:08.912 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:08.913 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:08.913 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.913 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.913 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:3104 4.15 e104/76 4:afa9f208:::smithi03829856-35:head v 104'222, mlcod=101'216)", 2024-03-19T11:32:08.913 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.568280+0000", 2024-03-19T11:32:08.913 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 45.289689455000001, 2024-03-19T11:32:08.913 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022345304999999999, 2024-03-19T11:32:08.913 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:08.913 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:08.913 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:08.913 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.913 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:08.913 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.568280+0000", 2024-03-19T11:32:08.913 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:08.914 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.914 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.914 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:08.914 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.568280+0000", 2024-03-19T11:32:08.914 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:08.914 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.914 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.914 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:08.914 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.568282+0000", 2024-03-19T11:32:08.914 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0090000000000002e-06 2024-03-19T11:32:08.914 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.914 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.914 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:08.914 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.568299+0000", 2024-03-19T11:32:08.915 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6237000000000001e-05 2024-03-19T11:32:08.915 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.915 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.915 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:08.915 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.568331+0000", 2024-03-19T11:32:08.915 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2601999999999997e-05 2024-03-19T11:32:08.915 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.915 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.915 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:08.915 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.568350+0000", 2024-03-19T11:32:08.915 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8280000000000001e-05 2024-03-19T11:32:08.915 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.915 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.915 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:08.915 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.571311+0000", 2024-03-19T11:32:08.916 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029617770000000001 2024-03-19T11:32:08.916 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.916 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.916 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:08.916 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.571379+0000", 2024-03-19T11:32:08.916 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7331999999999998e-05 2024-03-19T11:32:08.916 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.916 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.916 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:08.916 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.590583+0000", 2024-03-19T11:32:08.916 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019204649000000001 2024-03-19T11:32:08.916 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.916 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.916 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:08.917 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.590626+0000", 2024-03-19T11:32:08.917 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2419000000000002e-05 2024-03-19T11:32:08.917 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:08.917 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:08.917 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:08.917 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.917 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.917 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:3106 4.15 e104/76 4:ad3ad1b3:::smithi03829856-21:head v 104'224, mlcod=101'216)", 2024-03-19T11:32:08.917 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.571056+0000", 2024-03-19T11:32:08.917 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 45.286913996999999, 2024-03-19T11:32:08.917 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019636355000000001, 2024-03-19T11:32:08.917 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:08.917 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:08.917 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:08.918 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.918 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:08.918 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.571056+0000", 2024-03-19T11:32:08.918 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:08.918 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.918 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.918 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:08.918 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.571056+0000", 2024-03-19T11:32:08.918 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:08.918 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.918 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.918 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:08.918 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.571058+0000", 2024-03-19T11:32:08.918 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2400000000000002e-06 2024-03-19T11:32:08.919 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.919 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.919 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:08.919 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.571586+0000", 2024-03-19T11:32:08.919 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00052782299999999996 2024-03-19T11:32:08.920 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.920 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.920 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:08.920 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.571618+0000", 2024-03-19T11:32:08.920 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2453000000000003e-05 2024-03-19T11:32:08.920 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.920 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.920 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:08.921 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.571638+0000", 2024-03-19T11:32:08.921 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9176000000000001e-05 2024-03-19T11:32:08.921 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.921 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.921 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:08.921 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.573958+0000", 2024-03-19T11:32:08.921 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0023201699999999999 2024-03-19T11:32:08.921 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.921 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.921 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:08.921 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.574023+0000", 2024-03-19T11:32:08.921 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5425000000000005e-05 2024-03-19T11:32:08.921 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.921 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.922 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:08.922 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.590646+0000", 2024-03-19T11:32:08.922 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016623189 2024-03-19T11:32:08.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.922 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:08.922 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.590692+0000", 2024-03-19T11:32:08.922 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5878999999999999e-05 2024-03-19T11:32:08.922 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:08.922 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:08.922 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:08.922 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.922 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.923 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.3.0:309 4.0 e108/60 4:01018511:::smithi03829856-19:b v 108'91, mlcod=100'87)", 2024-03-19T11:32:08.923 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.205898+0000", 2024-03-19T11:32:08.923 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 41.652072201999999, 2024-03-19T11:32:08.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.034464328000000002, 2024-03-19T11:32:08.923 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:08.923 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:08.923 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:08.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.923 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:08.923 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.205898+0000", 2024-03-19T11:32:08.923 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:08.923 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.923 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.923 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:08.924 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.205898+0000", 2024-03-19T11:32:08.924 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:08.924 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.924 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:08.924 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.205899+0000", 2024-03-19T11:32:08.924 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.16e-07 2024-03-19T11:32:08.924 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.924 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:08.924 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.205916+0000", 2024-03-19T11:32:08.924 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7229000000000001e-05 2024-03-19T11:32:08.924 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.924 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.925 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:08.925 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.205946+0000", 2024-03-19T11:32:08.925 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0233999999999999e-05 2024-03-19T11:32:08.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.925 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.925 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:08.925 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.205966+0000", 2024-03-19T11:32:08.925 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9434999999999999e-05 2024-03-19T11:32:08.925 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.926 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.926 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:08.926 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.206162+0000", 2024-03-19T11:32:08.926 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000196918 2024-03-19T11:32:08.926 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.926 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.926 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:08.926 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.206227+0000", 2024-03-19T11:32:08.926 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4381000000000006e-05 2024-03-19T11:32:08.926 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.926 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.926 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:08.926 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.240212+0000", 2024-03-19T11:32:08.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033984852000000003 2024-03-19T11:32:08.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.927 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:08.927 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.240362+0000", 2024-03-19T11:32:08.927 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015036300000000001 2024-03-19T11:32:08.927 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:08.927 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:08.927 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:08.927 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.927 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.927 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.5.0:238 4.15 e108/76 4:ac9cd3a3:::smithi03829856-8:10 v 108'231, mlcod=104'228)", 2024-03-19T11:32:08.927 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.215160+0000", 2024-03-19T11:32:08.927 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 41.642809794000001, 2024-03-19T11:32:08.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.024600749000000002, 2024-03-19T11:32:08.928 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:08.928 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:08.928 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:08.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.928 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:08.928 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215160+0000", 2024-03-19T11:32:08.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:08.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.928 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.928 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:08.928 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215160+0000", 2024-03-19T11:32:08.928 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:08.928 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:08.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215161+0000", 2024-03-19T11:32:08.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6000000000000002e-07 2024-03-19T11:32:08.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:08.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215172+0000", 2024-03-19T11:32:08.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1042e-05 2024-03-19T11:32:08.929 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.929 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.929 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:08.929 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215213+0000", 2024-03-19T11:32:08.929 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1167000000000001e-05 2024-03-19T11:32:08.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:08.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215234+0000", 2024-03-19T11:32:08.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0534e-05 2024-03-19T11:32:08.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:08.930 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215668+0000", 2024-03-19T11:32:08.930 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000434074 2024-03-19T11:32:08.930 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.930 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.930 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:08.931 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215749+0000", 2024-03-19T11:32:08.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0702000000000001e-05 2024-03-19T11:32:08.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.931 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:08.931 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.239629+0000", 2024-03-19T11:32:08.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023880855999999999 2024-03-19T11:32:08.931 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.931 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.931 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:08.931 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.239761+0000", 2024-03-19T11:32:08.931 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000131514 2024-03-19T11:32:08.931 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:08.931 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:08.932 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:08.932 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.932 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.932 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.5.0:239 4.8 e108/100 4:14611006:::smithi03829856-25:c v 108'125, mlcod=103'122)", 2024-03-19T11:32:08.932 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.215292+0000", 2024-03-19T11:32:08.932 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 41.64267761, 2024-03-19T11:32:08.932 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.025163089999999999, 2024-03-19T11:32:08.932 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:08.932 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:08.932 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:08.932 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.932 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:08.932 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215292+0000", 2024-03-19T11:32:08.933 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:08.933 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.933 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:08.933 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215292+0000", 2024-03-19T11:32:08.933 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:08.933 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.933 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:08.933 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215293+0000", 2024-03-19T11:32:08.933 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.47e-07 2024-03-19T11:32:08.933 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.933 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.933 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:08.934 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215315+0000", 2024-03-19T11:32:08.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1691e-05 2024-03-19T11:32:08.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.934 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:08.934 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215348+0000", 2024-03-19T11:32:08.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2676999999999999e-05 2024-03-19T11:32:08.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.934 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:08.934 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215362+0000", 2024-03-19T11:32:08.934 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3978999999999999e-05 2024-03-19T11:32:08.934 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.934 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:08.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215727+0000", 2024-03-19T11:32:08.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000365583 2024-03-19T11:32:08.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:08.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215804+0000", 2024-03-19T11:32:08.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7392999999999996e-05 2024-03-19T11:32:08.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.935 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.935 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:08.935 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.240368+0000", 2024-03-19T11:32:08.935 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.024563807999999999 2024-03-19T11:32:08.935 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.936 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.936 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:08.936 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.240455+0000", 2024-03-19T11:32:08.938 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7112000000000002e-05 2024-03-19T11:32:08.938 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:08.938 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:08.938 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:08.938 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.938 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.938 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.5.0:240 4.15 e108/76 4:aa3303a5:::smithi03829856-17:b v 108'233, mlcod=104'228)", 2024-03-19T11:32:08.938 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.215400+0000", 2024-03-19T11:32:08.938 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 41.642569880000003, 2024-03-19T11:32:08.939 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026645319000000001, 2024-03-19T11:32:08.939 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:08.939 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:08.939 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:08.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.939 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:08.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215400+0000", 2024-03-19T11:32:08.939 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:08.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.939 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:08.939 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215400+0000", 2024-03-19T11:32:08.939 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:08.939 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.939 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.940 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:08.940 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215401+0000", 2024-03-19T11:32:08.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3e-07 2024-03-19T11:32:08.940 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.940 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.940 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:08.940 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215417+0000", 2024-03-19T11:32:08.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5821999999999999e-05 2024-03-19T11:32:08.940 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.940 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.940 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:08.940 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215443+0000", 2024-03-19T11:32:08.940 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6971e-05 2024-03-19T11:32:08.940 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.941 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.941 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:08.941 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215457+0000", 2024-03-19T11:32:08.941 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3139000000000001e-05 2024-03-19T11:32:08.941 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.941 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.941 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:08.941 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.234414+0000", 2024-03-19T11:32:08.941 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018957108 2024-03-19T11:32:08.941 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.941 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.941 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:08.941 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.234470+0000", 2024-03-19T11:32:08.941 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6274999999999999e-05 2024-03-19T11:32:08.941 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.942 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:08.942 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.241943+0000", 2024-03-19T11:32:08.942 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0074728770000000002 2024-03-19T11:32:08.942 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.942 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:08.942 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.242045+0000", 2024-03-19T11:32:08.942 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000102497 2024-03-19T11:32:08.942 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:08.942 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:08.942 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:08.942 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.942 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.943 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.5.0:241 4.8 e108/100 4:15708482:::smithi03829856-23:10 v 108'127, mlcod=103'122)", 2024-03-19T11:32:08.943 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.217180+0000", 2024-03-19T11:32:08.943 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 41.640789611999999, 2024-03-19T11:32:08.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.025026023000000001, 2024-03-19T11:32:08.943 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:08.943 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:08.943 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:08.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:08.943 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.217180+0000", 2024-03-19T11:32:08.943 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:08.943 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.943 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.943 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:08.944 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.217180+0000", 2024-03-19T11:32:08.944 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:08.944 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.944 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:08.944 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.217182+0000", 2024-03-19T11:32:08.944 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.12e-06 2024-03-19T11:32:08.944 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.944 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:08.944 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.217200+0000", 2024-03-19T11:32:08.944 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7677999999999999e-05 2024-03-19T11:32:08.944 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.944 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.944 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:08.945 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.217246+0000", 2024-03-19T11:32:08.945 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6053000000000001e-05 2024-03-19T11:32:08.945 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.945 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:08.945 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.217266+0000", 2024-03-19T11:32:08.945 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0085000000000002e-05 2024-03-19T11:32:08.945 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.945 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:08.945 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.226876+0000", 2024-03-19T11:32:08.945 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0096094389999999991 2024-03-19T11:32:08.945 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.945 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.946 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:08.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.226932+0000", 2024-03-19T11:32:08.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6234999999999999e-05 2024-03-19T11:32:08.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.946 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:08.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.242063+0000", 2024-03-19T11:32:08.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015131083 2024-03-19T11:32:08.946 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.946 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.946 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:08.946 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.242206+0000", 2024-03-19T11:32:08.946 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014333 2024-03-19T11:32:08.946 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:08.947 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:08.947 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:08.947 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.947 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.1.0:279 4.24 e108/83 4:27909373:::smithi03829856-46:c v 108'129, mlcod=100'124)", 2024-03-19T11:32:08.947 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.223035+0000", 2024-03-19T11:32:08.947 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 41.634935271000003, 2024-03-19T11:32:08.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019488583, 2024-03-19T11:32:08.947 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:08.947 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:08.947 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:08.947 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.947 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:08.947 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.223035+0000", 2024-03-19T11:32:08.947 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:08.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.948 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:08.948 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.223035+0000", 2024-03-19T11:32:08.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:08.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.948 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:08.948 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.223037+0000", 2024-03-19T11:32:08.948 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3829999999999999e-06 2024-03-19T11:32:08.948 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.948 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.948 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:08.948 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.223073+0000", 2024-03-19T11:32:08.949 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6306000000000001e-05 2024-03-19T11:32:08.949 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.949 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:08.949 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.223095+0000", 2024-03-19T11:32:08.949 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1738000000000001e-05 2024-03-19T11:32:08.949 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.949 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:08.949 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.223107+0000", 2024-03-19T11:32:08.949 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2082000000000001e-05 2024-03-19T11:32:08.949 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.949 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.949 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:08.949 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.226145+0000", 2024-03-19T11:32:08.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003038022 2024-03-19T11:32:08.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:08.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.226201+0000", 2024-03-19T11:32:08.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6107999999999999e-05 2024-03-19T11:32:08.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:08.950 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.242322+0000", 2024-03-19T11:32:08.950 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016120196 2024-03-19T11:32:08.950 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:08.950 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:08.950 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:08.951 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.242523+0000", 2024-03-19T11:32:08.951 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020174799999999999 2024-03-19T11:32:08.951 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:08.951 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:08.951 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:08.951 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:08.951 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:08.951 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:32:08.952 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:32:08.961+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:32:08.953 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-19T11:32:09.055 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:32:09.061+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:32:09.081 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:32:09.091+0000 7fa5e08db640 -1 osd.3 147 handle_osd_map injecting map crc failure 2024-03-19T11:32:09.083 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:32:09.089+0000 7fb17dd72640 -1 osd.4 147 handle_osd_map injecting map crc failure 2024-03-19T11:32:09.086 INFO:tasks.rados.rados.0.smithi038.stdout:1838: done (3 left) 2024-03-19T11:32:09.086 INFO:tasks.rados.rados.0.smithi038.stdout:1839: rollback oid 26 current snap is 41 2024-03-19T11:32:09.087 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 26 to 37 2024-03-19T11:32:09.087 INFO:tasks.rados.rados.0.smithi038.stdout:1835: finishing write tid 1 to smithi03829856-5 2024-03-19T11:32:09.087 INFO:tasks.rados.rados.0.smithi038.stdout:1837: got (22) Invalid argument 2024-03-19T11:32:09.087 INFO:tasks.rados.rados.0.smithi038.stdout:1836: expect (ObjNum 87 snap 8 seq_num 87) 2024-03-19T11:32:09.096 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:32:09.096 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:32:09.096 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:32:09.096 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:32:09.118 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-19T11:32:09.149 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-03-19T11:32:09.169 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:32:09.179+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:32:09.244 INFO:tasks.rados.rados.0.smithi038.stdout:1835: finishing write tid 2 to smithi03829856-5 2024-03-19T11:32:09.244 INFO:tasks.rados.rados.0.smithi038.stdout:1835: finishing write tid 3 to smithi03829856-5 2024-03-19T11:32:09.244 INFO:tasks.rados.rados.0.smithi038.stdout:1835: finishing write tid 4 to smithi03829856-5 2024-03-19T11:32:09.244 INFO:tasks.rados.rados.0.smithi038.stdout:1835: finishing write tid 5 to smithi03829856-5 2024-03-19T11:32:09.244 INFO:tasks.rados.rados.0.smithi038.stdout:1835: oid 5 updating version 0 to 243 2024-03-19T11:32:09.244 INFO:tasks.rados.rados.0.smithi038.stdout:1835: oid 5 version 243 is already newer than 242 2024-03-19T11:32:09.244 INFO:tasks.rados.rados.0.smithi038.stdout:1835: oid 5 version 243 is already newer than 241 2024-03-19T11:32:09.244 INFO:tasks.rados.rados.0.smithi038.stdout:1835: oid 5 version 243 is already newer than 240 2024-03-19T11:32:09.244 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 5 v 243 (ObjNum 345 snap 41 seq_num 345) dirty exists 2024-03-19T11:32:09.244 INFO:tasks.rados.rados.0.smithi038.stdout:1835: left oid 5 (ObjNum 345 snap 41 seq_num 345) 2024-03-19T11:32:09.244 INFO:tasks.rados.rados.0.smithi038.stdout:1835: done (3 left) 2024-03-19T11:32:09.244 INFO:tasks.rados.rados.0.smithi038.stdout:1836: done (2 left) 2024-03-19T11:32:09.245 INFO:tasks.rados.rados.0.smithi038.stdout:1837: done (1 left) 2024-03-19T11:32:09.245 INFO:tasks.rados.rados.0.smithi038.stdout:1840: delete oid 9 current snap is 41 2024-03-19T11:32:09.250 INFO:tasks.rados.rados.0.smithi038.stdout:1840: done (1 left) 2024-03-19T11:32:09.250 INFO:tasks.rados.rados.0.smithi038.stdout:1841: set_chunk oid 30 target oid 2024-03-19T11:32:09.254 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:32:09.263+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:32:09.258 INFO:tasks.rados.rados.0.smithi038.stdout:1841: set_chunk oid 30 offset: 938496 length: 320000 target oid b009208b9d448f5d2e901cec7e10447d73717505b75670db95cbadcf40c41026 offset: 0 snap -1 2024-03-19T11:32:09.258 INFO:tasks.rados.rados.0.smithi038.stdout:1842: snap_create 2024-03-19T11:32:09.259 INFO:tasks.rados.rados.0.smithi038.stdout:1841: finishing set_chunk to oid 30 2024-03-19T11:32:09.259 INFO:tasks.rados.rados.0.smithi038.stdout:Range is overlapped: oid 30 set_chunk returned error code -95 offset: 938496 length: 320000 2024-03-19T11:32:09.259 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 30 v 0 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-03-19T11:32:09.273 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:32:09.273 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:32:09.273 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:32:09.273 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:32:09.273 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:32:09.288 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-19T11:32:09.373 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:32:09.383+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:32:09.457 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:32:09.457 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:32:09.457 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:32:09.457 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:32:09.457 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.457 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1217 4.3 4:c8b09e87:::smithi03829856-15:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:32:09.457 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:18.155966+0000", 2024-03-19T11:32:09.457 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 111.30724609400001, 2024-03-19T11:32:09.458 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.023984647000000001, 2024-03-19T11:32:09.458 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:09.458 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:09.458 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:09.458 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:09.458 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:09.458 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1217 2024-03-19T11:32:09.458 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.458 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:09.458 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.458 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:09.458 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155966+0000", 2024-03-19T11:32:09.458 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:09.458 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.458 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.459 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:09.459 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155966+0000", 2024-03-19T11:32:09.459 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:09.459 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.459 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.459 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:09.459 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155968+0000", 2024-03-19T11:32:09.459 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.795e-06 2024-03-19T11:32:09.459 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.459 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.459 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:09.459 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.155978+0000", 2024-03-19T11:32:09.459 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0574000000000001e-05 2024-03-19T11:32:09.459 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.459 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.460 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:09.460 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.156000+0000", 2024-03-19T11:32:09.460 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1250000000000002e-05 2024-03-19T11:32:09.460 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.460 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.460 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:09.460 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.156010+0000", 2024-03-19T11:32:09.460 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0168e-05 2024-03-19T11:32:09.460 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.460 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.460 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.460 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.156119+0000", 2024-03-19T11:32:09.460 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010950699999999999 2024-03-19T11:32:09.460 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.460 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.461 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:09.461 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.156231+0000", 2024-03-19T11:32:09.461 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000111445 2024-03-19T11:32:09.461 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.461 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.461 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:09.461 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:18.179951+0000", 2024-03-19T11:32:09.461 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.023719908000000001 2024-03-19T11:32:09.461 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:09.461 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:09.461 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:09.461 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.461 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.462 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.24 75/74 [PushOp(4:27909373:::smithi03829856-46:8, version: 46'59, data_included: [756315~767970,1933572~280606], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:8@46'59, size: 2756315, copy_subset: [(0, 2756315)], clone_subset: {}, snapset: 10=[]:{8: [6], c: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2214178, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:266590a5:::smithi03829856-16:e, version: 70'107, data_included: [652105~662359,1728171~386217], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:e@70'107, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 10=[]:{e: [d, b, 6]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:266590a5:::smithi03829856-16:head, version: 70'108, data_included: [652105~662359,1728171~386217], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:head@70'108, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:27909373:::smithi03829856-46:c, version: 70'109, data_included: [628595~895690,1807440~152886], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:c@70'109, size: 3428595, copy_subset: [(0, 3428595)], clone_subset: {}, snapset: 10=[]:{8: [6], c: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1960326, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:27909373:::smithi03829856-46:head, version: 70'110, data_included: [403401~454961,1584925~506039], data_size: 961000, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:head@70'110, size: 2403401, copy_subset: [(0, 2403401)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2403401, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:32:09.462 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.099395+0000", 2024-03-19T11:32:09.462 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 75.363816932999995, 2024-03-19T11:32:09.462 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.045391157000000001, 2024-03-19T11:32:09.462 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:09.462 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:09.462 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:09.462 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.462 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:09.462 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.099395+0000", 2024-03-19T11:32:09.462 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:09.462 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.463 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.463 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:09.463 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.099395+0000", 2024-03-19T11:32:09.463 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:09.463 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.463 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.463 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:09.463 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.099397+0000", 2024-03-19T11:32:09.463 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.691e-06 2024-03-19T11:32:09.463 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.463 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.463 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:09.463 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105009+0000", 2024-03-19T11:32:09.463 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0056121349999999999 2024-03-19T11:32:09.463 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.464 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.464 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:09.464 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105077+0000", 2024-03-19T11:32:09.464 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8561999999999998e-05 2024-03-19T11:32:09.464 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.464 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.464 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:09.464 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105133+0000", 2024-03-19T11:32:09.464 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5590000000000001e-05 2024-03-19T11:32:09.464 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.464 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.464 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.464 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105531+0000", 2024-03-19T11:32:09.464 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00039756200000000001 2024-03-19T11:32:09.464 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.465 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.465 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:09.465 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105579+0000", 2024-03-19T11:32:09.465 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8028999999999997e-05 2024-03-19T11:32:09.465 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.465 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.465 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:09.465 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.144786+0000", 2024-03-19T11:32:09.465 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.039207588000000002 2024-03-19T11:32:09.465 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:09.465 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:09.466 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:09.466 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.466 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.466 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.24 e75,74 4:266590a5:::smithi03829856-16:4,47'63,4:27909373:::smithi03829856-46:5,55'79,4:266590a5:::smithi03829856-16:10,67'105)", 2024-03-19T11:32:09.466 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.105191+0000", 2024-03-19T11:32:09.466 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 75.358020832999998, 2024-03-19T11:32:09.466 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.039580830999999997, 2024-03-19T11:32:09.466 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:09.466 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:09.466 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:09.467 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.467 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:09.467 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105191+0000", 2024-03-19T11:32:09.467 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:09.467 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.467 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.467 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:09.467 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105191+0000", 2024-03-19T11:32:09.467 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:09.467 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.467 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.467 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:09.467 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105192+0000", 2024-03-19T11:32:09.467 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.099e-06 2024-03-19T11:32:09.468 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.468 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.468 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:09.468 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105200+0000", 2024-03-19T11:32:09.468 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5279999999999998e-06 2024-03-19T11:32:09.468 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.468 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.468 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:09.468 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105211+0000", 2024-03-19T11:32:09.468 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1471000000000001e-05 2024-03-19T11:32:09.468 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.468 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.468 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:09.468 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.105220+0000", 2024-03-19T11:32:09.468 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3469999999999999e-06 2024-03-19T11:32:09.468 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.469 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.469 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.469 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.144219+0000", 2024-03-19T11:32:09.469 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.038999024 2024-03-19T11:32:09.469 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.469 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.469 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:09.469 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.144257+0000", 2024-03-19T11:32:09.469 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8501999999999998e-05 2024-03-19T11:32:09.469 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.469 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.469 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:09.469 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.144772+0000", 2024-03-19T11:32:09.469 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00051486000000000001 2024-03-19T11:32:09.469 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:09.470 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:09.470 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:09.470 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.470 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.470 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 75/74 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:7, version: 55'82, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@55'82, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 58'86, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@58'86, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [504210~968657,1572695~79919], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:12, version: 66'101, data_included: [440350~816629,1309629~231947], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:12@66'101, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:10, version: 70'112, data_included: [440350~816629,1309629~231947], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:10@70'112, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 70'113, data_included: [440350~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@70'113, size: 2605085, copy_subset: [(0, 2605085)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1488926, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:32:09.470 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.107266+0000", 2024-03-19T11:32:09.470 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 75.355946316000001, 2024-03-19T11:32:09.470 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.048592669999999998, 2024-03-19T11:32:09.470 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:09.470 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:09.470 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:09.470 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.470 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:09.471 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.107266+0000", 2024-03-19T11:32:09.471 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:09.471 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.471 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.471 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:09.471 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.107266+0000", 2024-03-19T11:32:09.471 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:09.471 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.471 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.471 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:09.471 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.107267+0000", 2024-03-19T11:32:09.471 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1459999999999999e-06 2024-03-19T11:32:09.471 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.472 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.473 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:09.473 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.122861+0000", 2024-03-19T11:32:09.473 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01559372 2024-03-19T11:32:09.476 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.476 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.476 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:09.476 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.122932+0000", 2024-03-19T11:32:09.476 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1477999999999996e-05 2024-03-19T11:32:09.476 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.476 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.476 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:09.476 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.122999+0000", 2024-03-19T11:32:09.476 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6832e-05 2024-03-19T11:32:09.476 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.477 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.477 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.477 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123332+0000", 2024-03-19T11:32:09.477 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000333271 2024-03-19T11:32:09.477 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.477 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.477 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:09.477 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123371+0000", 2024-03-19T11:32:09.477 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8525999999999997e-05 2024-03-19T11:32:09.477 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.477 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.477 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:09.477 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.155858+0000", 2024-03-19T11:32:09.477 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.032487697000000003 2024-03-19T11:32:09.477 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:09.478 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:09.478 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:09.478 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.478 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.478 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.21 e75,74 4:8401c1f0:::smithi03829856-43:8,46'66,4:866f3dcc:::smithi03829856-31:2,47'73)", 2024-03-19T11:32:09.478 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.123075+0000", 2024-03-19T11:32:09.478 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 75.340137096999996, 2024-03-19T11:32:09.478 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.033023045000000001, 2024-03-19T11:32:09.478 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:09.478 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:09.478 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:09.478 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.478 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:09.478 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123075+0000", 2024-03-19T11:32:09.479 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:09.479 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.479 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.479 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:09.479 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123075+0000", 2024-03-19T11:32:09.479 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:09.479 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.479 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.479 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:09.479 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123076+0000", 2024-03-19T11:32:09.479 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5e-06 2024-03-19T11:32:09.479 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.479 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.479 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:09.479 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123086+0000", 2024-03-19T11:32:09.480 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8060000000000006e-06 2024-03-19T11:32:09.480 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.480 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.480 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:09.480 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123099+0000", 2024-03-19T11:32:09.480 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3251000000000001e-05 2024-03-19T11:32:09.480 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.480 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.480 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:09.480 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.123112+0000", 2024-03-19T11:32:09.480 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2774e-05 2024-03-19T11:32:09.480 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.480 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.480 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.480 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.155902+0000", 2024-03-19T11:32:09.481 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.032790225999999999 2024-03-19T11:32:09.481 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.481 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.481 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:09.481 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.155914+0000", 2024-03-19T11:32:09.481 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1476e-05 2024-03-19T11:32:09.482 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.482 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.482 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:09.482 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.156098+0000", 2024-03-19T11:32:09.482 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000184012 2024-03-19T11:32:09.482 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:09.482 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:09.482 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:09.482 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.483 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.483 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 75/74 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:7, version: 55'82, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@55'82, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [1550438~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2599014, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 58'86, data_included: [1955874~258320,2888511~52681], data_size: 311001, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@58'86, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2941192, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [1652614~973634,2888511~52681,3182305~22261], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3204566, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:12, version: 66'101, data_included: [1541576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:12@66'101, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:10, version: 70'112, data_included: [1541576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:10@70'112, size: 3529437, copy_subset: [(0, 3529437)], clone_subset: {}, snapset: 12=[]:{7: [6], b: [b], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2590152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1541576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:head, version: 70'113, data_included: [1488926~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:head@70'113, size: 2605085, copy_subset: [(0, 2605085)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2537502, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1488926, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:32:09.483 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.174030+0000", 2024-03-19T11:32:09.483 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 75.289181893000006, 2024-03-19T11:32:09.483 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.040799275000000003, 2024-03-19T11:32:09.483 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:09.483 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:09.483 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:09.483 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.484 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:09.484 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.174030+0000", 2024-03-19T11:32:09.484 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:09.484 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.484 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.484 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:09.484 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.174030+0000", 2024-03-19T11:32:09.484 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:09.484 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.484 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.484 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:09.484 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.174031+0000", 2024-03-19T11:32:09.484 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.322e-06 2024-03-19T11:32:09.484 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.484 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.485 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:09.485 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.184771+0000", 2024-03-19T11:32:09.485 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010739227 2024-03-19T11:32:09.485 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.485 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.485 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:09.485 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.184872+0000", 2024-03-19T11:32:09.485 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000101234 2024-03-19T11:32:09.485 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.485 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.485 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:09.485 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.184985+0000", 2024-03-19T11:32:09.485 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000113545 2024-03-19T11:32:09.485 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.485 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.486 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.486 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.185463+0000", 2024-03-19T11:32:09.486 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00047749899999999999 2024-03-19T11:32:09.486 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.486 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.486 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:09.486 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.185534+0000", 2024-03-19T11:32:09.486 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1007999999999995e-05 2024-03-19T11:32:09.486 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.486 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.486 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:09.486 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.214829+0000", 2024-03-19T11:32:09.486 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029295439999999999 2024-03-19T11:32:09.486 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:09.486 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:09.487 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:09.487 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.487 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.487 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2372 4.16 4:6868af91:::smithi03829856-10:b [read 832512~818688] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-03-19T11:32:09.487 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:58.455732+0000", 2024-03-19T11:32:09.487 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 71.007480177999994, 2024-03-19T11:32:09.487 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.69150541200000004, 2024-03-19T11:32:09.487 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:09.487 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:09.487 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:09.487 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:09.487 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:09.487 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2372 2024-03-19T11:32:09.487 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.487 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:09.488 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.488 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:09.488 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455732+0000", 2024-03-19T11:32:09.488 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:09.488 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.488 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.488 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:09.488 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455732+0000", 2024-03-19T11:32:09.488 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:09.488 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.488 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.488 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:09.488 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455735+0000", 2024-03-19T11:32:09.488 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.756e-06 2024-03-19T11:32:09.489 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.489 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.489 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:09.489 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455744+0000", 2024-03-19T11:32:09.489 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1880000000000008e-06 2024-03-19T11:32:09.489 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.489 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.489 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:09.489 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455764+0000", 2024-03-19T11:32:09.489 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0044999999999999e-05 2024-03-19T11:32:09.489 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.489 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.489 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:09.489 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455774+0000", 2024-03-19T11:32:09.489 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8770000000000005e-06 2024-03-19T11:32:09.489 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.490 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.490 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.490 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455837+0000", 2024-03-19T11:32:09.490 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3588000000000002e-05 2024-03-19T11:32:09.490 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.490 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.490 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:09.490 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:58.455846+0000", 2024-03-19T11:32:09.490 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6030000000000005e-06 2024-03-19T11:32:09.490 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.490 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.490 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.490 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.076525+0000", 2024-03-19T11:32:09.490 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.62067875900000002 2024-03-19T11:32:09.491 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.491 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.491 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:32:09.491 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.076981+0000", 2024-03-19T11:32:09.491 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00045633300000000001 2024-03-19T11:32:09.491 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.491 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.491 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.491 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.104422+0000", 2024-03-19T11:32:09.491 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027440665999999999 2024-03-19T11:32:09.491 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.491 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.491 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:32:09.492 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.104786+0000", 2024-03-19T11:32:09.492 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00036471499999999999 2024-03-19T11:32:09.492 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.492 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.492 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.492 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.140725+0000", 2024-03-19T11:32:09.492 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.035939093999999998 2024-03-19T11:32:09.492 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.492 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.492 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:32:09.492 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.140841+0000", 2024-03-19T11:32:09.492 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000115207 2024-03-19T11:32:09.492 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.492 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.492 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.492 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.145559+0000", 2024-03-19T11:32:09.493 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.004718486 2024-03-19T11:32:09.493 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.493 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.493 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:09.493 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.145760+0000", 2024-03-19T11:32:09.493 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000201058 2024-03-19T11:32:09.493 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.493 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.493 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:09.493 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.147237+0000", 2024-03-19T11:32:09.493 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014770370000000001 2024-03-19T11:32:09.493 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:09.493 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:09.493 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:09.493 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.494 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.494 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.4.13:82 3.3 3:f6b6d21a:::smithi03829856-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-03-19T11:32:09.494 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:04.069448+0000", 2024-03-19T11:32:09.494 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 65.393764392999998, 2024-03-19T11:32:09.494 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.086427773999999999, 2024-03-19T11:32:09.494 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:09.494 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:09.494 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:09.494 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.494 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:09.494 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069448+0000", 2024-03-19T11:32:09.494 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:09.494 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.494 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.495 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:09.495 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069448+0000", 2024-03-19T11:32:09.495 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:09.495 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.495 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.495 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:09.495 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069449+0000", 2024-03-19T11:32:09.495 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8160000000000001e-06 2024-03-19T11:32:09.495 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.495 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.495 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:09.495 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069458+0000", 2024-03-19T11:32:09.495 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2900000000000002e-06 2024-03-19T11:32:09.495 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.495 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.496 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:09.496 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069491+0000", 2024-03-19T11:32:09.496 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3229000000000003e-05 2024-03-19T11:32:09.496 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.496 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.496 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:09.496 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069501+0000", 2024-03-19T11:32:09.496 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7629999999999995e-06 2024-03-19T11:32:09.496 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.496 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.496 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.496 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069573+0000", 2024-03-19T11:32:09.496 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2349999999999997e-05 2024-03-19T11:32:09.496 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.496 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.497 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:09.497 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069581+0000", 2024-03-19T11:32:09.497 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.388e-06 2024-03-19T11:32:09.497 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.497 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.497 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.497 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.075451+0000", 2024-03-19T11:32:09.497 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005869956 2024-03-19T11:32:09.497 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.497 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.497 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:32:09.497 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.077511+0000", 2024-03-19T11:32:09.497 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0020592610000000002 2024-03-19T11:32:09.497 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.497 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.498 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.498 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.108781+0000", 2024-03-19T11:32:09.498 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.031270344999999998 2024-03-19T11:32:09.498 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.498 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.498 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:09.498 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.108895+0000", 2024-03-19T11:32:09.498 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000114059 2024-03-19T11:32:09.498 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.498 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.498 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4", 2024-03-19T11:32:09.498 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.109065+0000", 2024-03-19T11:32:09.498 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016979899999999999 2024-03-19T11:32:09.498 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.498 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.499 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:32:09.499 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.111730+0000", 2024-03-19T11:32:09.499 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0026651410000000002 2024-03-19T11:32:09.499 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.499 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.499 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:09.499 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.155616+0000", 2024-03-19T11:32:09.499 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.043885986000000002 2024-03-19T11:32:09.499 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.499 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.499 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:32:09.499 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.155745+0000", 2024-03-19T11:32:09.499 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012948899999999999 2024-03-19T11:32:09.499 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.499 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.500 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:09.500 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.155875+0000", 2024-03-19T11:32:09.500 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012990199999999999 2024-03-19T11:32:09.500 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:09.500 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:09.500 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:09.500 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.500 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.500 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.4.13:83 3.3 3:f6b6d21a:::smithi03829856-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-03-19T11:32:09.500 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:04.069526+0000", 2024-03-19T11:32:09.500 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 65.393685613000002, 2024-03-19T11:32:09.500 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.087140880000000004, 2024-03-19T11:32:09.500 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:09.500 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:09.501 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:09.501 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.501 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:09.501 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069526+0000", 2024-03-19T11:32:09.501 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:09.501 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.501 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.501 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:09.501 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069526+0000", 2024-03-19T11:32:09.501 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:09.501 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.502 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.502 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:09.502 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069527+0000", 2024-03-19T11:32:09.502 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2799999999999996e-07 2024-03-19T11:32:09.502 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.502 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.502 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:09.502 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069531+0000", 2024-03-19T11:32:09.502 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1489999999999996e-06 2024-03-19T11:32:09.502 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.503 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.503 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:09.503 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069539+0000", 2024-03-19T11:32:09.503 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3189999999999995e-06 2024-03-19T11:32:09.503 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.503 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.503 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:09.503 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069545+0000", 2024-03-19T11:32:09.503 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.592e-06 2024-03-19T11:32:09.503 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.503 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.503 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.504 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069616+0000", 2024-03-19T11:32:09.504 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1130000000000005e-05 2024-03-19T11:32:09.504 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.504 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.504 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:09.504 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.069628+0000", 2024-03-19T11:32:09.504 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1697e-05 2024-03-19T11:32:09.504 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.504 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.504 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.504 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.077594+0000", 2024-03-19T11:32:09.504 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0079656989999999997 2024-03-19T11:32:09.505 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.505 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.505 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:32:09.505 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.077674+0000", 2024-03-19T11:32:09.505 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0338000000000002e-05 2024-03-19T11:32:09.505 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.505 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.505 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.505 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.112771+0000", 2024-03-19T11:32:09.505 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.035097325999999998 2024-03-19T11:32:09.505 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.505 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.505 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:09.506 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.112882+0000", 2024-03-19T11:32:09.506 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000110949 2024-03-19T11:32:09.506 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.506 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.506 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4", 2024-03-19T11:32:09.506 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.113031+0000", 2024-03-19T11:32:09.506 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000148355 2024-03-19T11:32:09.506 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.506 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.506 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:32:09.507 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.114076+0000", 2024-03-19T11:32:09.507 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001045009 2024-03-19T11:32:09.507 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.507 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.507 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:09.507 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156482+0000", 2024-03-19T11:32:09.507 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.042406354 2024-03-19T11:32:09.507 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.507 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.507 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:32:09.507 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156549+0000", 2024-03-19T11:32:09.507 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7229000000000003e-05 2024-03-19T11:32:09.507 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.507 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.507 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:09.508 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156667+0000", 2024-03-19T11:32:09.508 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000118106 2024-03-19T11:32:09.508 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:09.508 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:09.508 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:09.508 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.508 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.508 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.4.13:84 3.3 3:ce6951d3:::smithi03829856-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-03-19T11:32:09.508 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:04.071995+0000", 2024-03-19T11:32:09.508 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 65.391216838000005, 2024-03-19T11:32:09.508 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.084172497999999998, 2024-03-19T11:32:09.508 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:09.508 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:09.508 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:09.508 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.509 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:09.509 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.071995+0000", 2024-03-19T11:32:09.509 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:09.509 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.509 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.509 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:09.509 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.071995+0000", 2024-03-19T11:32:09.509 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:09.509 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.509 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.509 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:09.509 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.071997+0000", 2024-03-19T11:32:09.509 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6029999999999999e-06 2024-03-19T11:32:09.509 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.509 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.509 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:09.510 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072004+0000", 2024-03-19T11:32:09.510 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8600000000000004e-06 2024-03-19T11:32:09.510 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.510 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.510 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:09.510 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072033+0000", 2024-03-19T11:32:09.510 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9836999999999999e-05 2024-03-19T11:32:09.510 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.510 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.510 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:09.510 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072042+0000", 2024-03-19T11:32:09.510 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9279999999999999e-06 2024-03-19T11:32:09.510 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.510 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.511 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.511 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072123+0000", 2024-03-19T11:32:09.511 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0630000000000006e-05 2024-03-19T11:32:09.511 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.511 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.511 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:09.511 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072132+0000", 2024-03-19T11:32:09.511 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5560000000000008e-06 2024-03-19T11:32:09.511 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.511 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.511 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.511 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.077712+0000", 2024-03-19T11:32:09.511 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055804970000000002 2024-03-19T11:32:09.511 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.512 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.512 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:32:09.512 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.079495+0000", 2024-03-19T11:32:09.512 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017831850000000001 2024-03-19T11:32:09.512 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.512 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.512 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.512 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.109825+0000", 2024-03-19T11:32:09.512 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.030329819000000001 2024-03-19T11:32:09.512 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.512 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.512 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:09.512 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.109904+0000", 2024-03-19T11:32:09.512 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9269000000000003e-05 2024-03-19T11:32:09.512 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.513 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.513 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4", 2024-03-19T11:32:09.513 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.110085+0000", 2024-03-19T11:32:09.513 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00018018600000000001 2024-03-19T11:32:09.513 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.513 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.513 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:32:09.513 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.113581+0000", 2024-03-19T11:32:09.513 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034962919999999998 2024-03-19T11:32:09.513 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.513 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.513 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:09.513 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.155925+0000", 2024-03-19T11:32:09.513 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.042344404000000002 2024-03-19T11:32:09.513 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.514 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.514 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:32:09.514 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156026+0000", 2024-03-19T11:32:09.514 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000100599 2024-03-19T11:32:09.514 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.514 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.514 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:09.514 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156168+0000", 2024-03-19T11:32:09.514 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014183300000000001 2024-03-19T11:32:09.514 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:09.514 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:09.514 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:09.514 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.514 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.515 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(osd.4.13:85 3.3 3:ce6951d3:::smithi03829856-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-03-19T11:32:09.515 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:04.072068+0000", 2024-03-19T11:32:09.515 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 65.391144202000007, 2024-03-19T11:32:09.515 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.084368466000000003, 2024-03-19T11:32:09.515 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:09.515 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:09.515 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:09.515 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.515 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:09.515 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072068+0000", 2024-03-19T11:32:09.515 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:09.515 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.515 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.515 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:09.515 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072068+0000", 2024-03-19T11:32:09.516 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:09.516 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.516 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.516 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:09.516 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072068+0000", 2024-03-19T11:32:09.516 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9699999999999996e-07 2024-03-19T11:32:09.516 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.516 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.516 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:09.516 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072074+0000", 2024-03-19T11:32:09.516 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2279999999999998e-06 2024-03-19T11:32:09.516 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.516 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.516 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:09.516 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072140+0000", 2024-03-19T11:32:09.517 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6638999999999997e-05 2024-03-19T11:32:09.517 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.517 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.517 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:09.517 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072147+0000", 2024-03-19T11:32:09.519 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8750000000000002e-06 2024-03-19T11:32:09.519 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.519 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.520 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.521 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072201+0000", 2024-03-19T11:32:09.521 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3634000000000002e-05 2024-03-19T11:32:09.521 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.521 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.521 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:09.521 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.072211+0000", 2024-03-19T11:32:09.521 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0589e-05 2024-03-19T11:32:09.521 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.521 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.521 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.522 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.079687+0000", 2024-03-19T11:32:09.522 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074757219999999997 2024-03-19T11:32:09.522 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.522 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.522 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:32:09.522 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.079769+0000", 2024-03-19T11:32:09.523 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2251999999999998e-05 2024-03-19T11:32:09.523 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.523 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.523 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.523 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.111798+0000", 2024-03-19T11:32:09.523 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.032028889999999997 2024-03-19T11:32:09.523 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.523 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.523 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:09.523 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.111902+0000", 2024-03-19T11:32:09.523 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000103773 2024-03-19T11:32:09.523 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.524 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.524 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4", 2024-03-19T11:32:09.524 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.112073+0000", 2024-03-19T11:32:09.524 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017102799999999999 2024-03-19T11:32:09.524 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.524 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.524 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:32:09.524 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.113591+0000", 2024-03-19T11:32:09.524 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0015179569999999999 2024-03-19T11:32:09.524 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.524 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.524 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:09.524 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156223+0000", 2024-03-19T11:32:09.524 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.042632153999999998 2024-03-19T11:32:09.524 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.525 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.525 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:32:09.525 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156321+0000", 2024-03-19T11:32:09.525 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8246999999999996e-05 2024-03-19T11:32:09.525 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.525 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.525 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:09.525 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:04.156436+0000", 2024-03-19T11:32:09.525 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000114881 2024-03-19T11:32:09.525 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:09.525 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:09.525 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:09.525 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.525 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.526 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3203 4.6 4:626acd4d:::smithi03829856-2:1f [sparse-read 0~1665607,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108)", 2024-03-19T11:32:09.526 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:27.253945+0000", 2024-03-19T11:32:09.526 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 42.209267134999997, 2024-03-19T11:32:09.526 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.83307735599999999, 2024-03-19T11:32:09.526 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:09.526 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:09.526 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:09.526 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:09.526 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:09.526 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3203 2024-03-19T11:32:09.526 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.526 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:09.526 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.526 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:09.526 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.253945+0000", 2024-03-19T11:32:09.527 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:09.527 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.527 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.527 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:09.527 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.253945+0000", 2024-03-19T11:32:09.527 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:09.527 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.527 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.527 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:09.527 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.253947+0000", 2024-03-19T11:32:09.527 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5979999999999999e-06 2024-03-19T11:32:09.528 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.528 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.528 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:09.528 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.253957+0000", 2024-03-19T11:32:09.528 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7699999999999996e-06 2024-03-19T11:32:09.528 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.528 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.528 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:09.528 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.253980+0000", 2024-03-19T11:32:09.528 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2466000000000001e-05 2024-03-19T11:32:09.529 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.529 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.529 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:09.529 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.253992+0000", 2024-03-19T11:32:09.529 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1985e-05 2024-03-19T11:32:09.529 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.529 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.529 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.529 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254071+0000", 2024-03-19T11:32:09.529 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8869000000000007e-05 2024-03-19T11:32:09.529 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.529 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.529 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:09.529 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254084+0000", 2024-03-19T11:32:09.529 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3716999999999999e-05 2024-03-19T11:32:09.530 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.530 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.530 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.530 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.086473+0000", 2024-03-19T11:32:09.530 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.83238890300000001 2024-03-19T11:32:09.530 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.530 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.530 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:09.530 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.086618+0000", 2024-03-19T11:32:09.531 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014479200000000001 2024-03-19T11:32:09.531 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.531 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.531 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:09.531 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.087022+0000", 2024-03-19T11:32:09.531 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00040425599999999997 2024-03-19T11:32:09.531 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:09.531 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:09.531 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:09.531 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.531 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.532 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3204 4.6 4:626acd4d:::smithi03829856-2:1f [read 0~3462242,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108)", 2024-03-19T11:32:09.532 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:27.254015+0000", 2024-03-19T11:32:09.532 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 42.209197353999997, 2024-03-19T11:32:09.532 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.83578705200000003, 2024-03-19T11:32:09.532 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:09.532 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:09.532 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:09.532 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:09.532 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:09.532 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3204 2024-03-19T11:32:09.533 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.533 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:09.533 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.533 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:09.533 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254015+0000", 2024-03-19T11:32:09.533 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:09.533 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.533 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.533 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:09.533 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254015+0000", 2024-03-19T11:32:09.533 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:09.533 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.533 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.533 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:09.533 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254015+0000", 2024-03-19T11:32:09.534 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8599999999999997e-07 2024-03-19T11:32:09.534 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.534 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.534 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:09.534 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254022+0000", 2024-03-19T11:32:09.534 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2870000000000001e-06 2024-03-19T11:32:09.534 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.534 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.534 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:09.534 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254031+0000", 2024-03-19T11:32:09.535 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0960000000000007e-06 2024-03-19T11:32:09.535 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.535 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.535 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:09.535 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254036+0000", 2024-03-19T11:32:09.535 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3510000000000001e-06 2024-03-19T11:32:09.535 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.535 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.535 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.535 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254137+0000", 2024-03-19T11:32:09.535 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000101065 2024-03-19T11:32:09.535 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.535 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.535 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:09.535 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254147+0000", 2024-03-19T11:32:09.536 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3929999999999997e-06 2024-03-19T11:32:09.536 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.536 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.536 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.536 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.087055+0000", 2024-03-19T11:32:09.536 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.83290843299999995 2024-03-19T11:32:09.536 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.536 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.536 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:09.536 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.087142+0000", 2024-03-19T11:32:09.536 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7151999999999995e-05 2024-03-19T11:32:09.536 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.537 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.537 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:09.537 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.089802+0000", 2024-03-19T11:32:09.537 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0026594890000000001 2024-03-19T11:32:09.537 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:09.537 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:09.537 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:09.537 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.537 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.537 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3205 4.6 4:626acd4d:::smithi03829856-2:1f [sparse-read 0~1665607] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108)", 2024-03-19T11:32:09.537 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:27.254049+0000", 2024-03-19T11:32:09.537 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 42.209163337, 2024-03-19T11:32:09.537 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.83625012399999998, 2024-03-19T11:32:09.537 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:09.537 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:09.538 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:09.538 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:09.538 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:09.538 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3205 2024-03-19T11:32:09.538 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.538 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:09.538 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.538 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:09.538 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254049+0000", 2024-03-19T11:32:09.538 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:09.538 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.538 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.538 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:09.538 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254049+0000", 2024-03-19T11:32:09.539 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:09.539 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.539 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.539 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:09.539 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254049+0000", 2024-03-19T11:32:09.539 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0699999999999997e-07 2024-03-19T11:32:09.539 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.539 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.539 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:09.539 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254053+0000", 2024-03-19T11:32:09.539 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.546e-06 2024-03-19T11:32:09.539 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.539 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.539 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:09.539 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254063+0000", 2024-03-19T11:32:09.540 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7939999999999997e-06 2024-03-19T11:32:09.540 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.540 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.540 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:09.540 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254072+0000", 2024-03-19T11:32:09.540 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9250000000000001e-06 2024-03-19T11:32:09.540 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.540 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.540 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.540 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254186+0000", 2024-03-19T11:32:09.540 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011446300000000001 2024-03-19T11:32:09.540 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.540 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.540 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:09.540 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:27.254198+0000", 2024-03-19T11:32:09.541 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1933000000000001e-05 2024-03-19T11:32:09.541 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.541 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.541 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.541 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.089838+0000", 2024-03-19T11:32:09.541 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.83563974200000002 2024-03-19T11:32:09.541 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.541 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.541 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:09.541 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.089941+0000", 2024-03-19T11:32:09.541 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000103052 2024-03-19T11:32:09.541 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.542 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.542 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:09.542 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.090299+0000", 2024-03-19T11:32:09.542 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00035806199999999997 2024-03-19T11:32:09.542 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:09.542 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:09.542 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:09.542 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.542 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.542 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3220 4.16 4:6868af91:::smithi03829856-10:head [tier-promote] snapc 21=[21,20,1f,1e,1d,1c,1b,1a,18,17] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-19T11:32:09.542 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:28.068879+0000", 2024-03-19T11:32:09.542 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 41.394333131000003, 2024-03-19T11:32:09.542 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025399244000000001, 2024-03-19T11:32:09.542 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:09.543 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:09.543 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:09.543 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:09.543 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:09.543 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3220 2024-03-19T11:32:09.543 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.543 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:09.543 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.543 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:09.543 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.068879+0000", 2024-03-19T11:32:09.543 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:09.543 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.544 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.544 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:09.544 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.068879+0000", 2024-03-19T11:32:09.544 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:09.544 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.544 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.544 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:09.544 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.068881+0000", 2024-03-19T11:32:09.544 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3879999999999998e-06 2024-03-19T11:32:09.544 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.544 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.544 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:09.544 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.068892+0000", 2024-03-19T11:32:09.544 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0672999999999999e-05 2024-03-19T11:32:09.545 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.545 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.545 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:09.545 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.068917+0000", 2024-03-19T11:32:09.545 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5367e-05 2024-03-19T11:32:09.545 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.545 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.545 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:09.545 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.068935+0000", 2024-03-19T11:32:09.545 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7832000000000001e-05 2024-03-19T11:32:09.545 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.545 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.546 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.546 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.069075+0000", 2024-03-19T11:32:09.546 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013980899999999999 2024-03-19T11:32:09.546 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.546 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.546 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-03-19T11:32:09.546 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.069091+0000", 2024-03-19T11:32:09.546 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6325e-05 2024-03-19T11:32:09.546 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.546 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.546 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.546 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.071382+0000", 2024-03-19T11:32:09.547 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022910109999999999 2024-03-19T11:32:09.547 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.547 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.547 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:09.547 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.071538+0000", 2024-03-19T11:32:09.547 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015600999999999999 2024-03-19T11:32:09.547 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.547 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.547 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:09.547 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.094278+0000", 2024-03-19T11:32:09.547 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.022739829 2024-03-19T11:32:09.547 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:09.547 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:09.547 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:09.548 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.548 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.548 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3885 4.21 4:8401c1f0:::smithi03829856-43:head [write 443784~648161] snapc 29=[29,28,27,26,25,24,23,22,21,1f,1e,1d,1a,17] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e129)", 2024-03-19T11:32:09.548 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:50.078273+0000", 2024-03-19T11:32:09.548 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 19.38493931, 2024-03-19T11:32:09.548 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1185664280000001, 2024-03-19T11:32:09.548 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:09.548 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:09.548 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:09.548 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:09.548 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:09.548 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3885 2024-03-19T11:32:09.548 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.549 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:09.549 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.549 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:09.549 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.078273+0000", 2024-03-19T11:32:09.549 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:09.549 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.549 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.549 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:09.549 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.078273+0000", 2024-03-19T11:32:09.549 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:09.549 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.549 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.549 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:09.549 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.078276+0000", 2024-03-19T11:32:09.549 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.942e-06 2024-03-19T11:32:09.550 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.550 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.550 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:09.550 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.078790+0000", 2024-03-19T11:32:09.550 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00051449799999999995 2024-03-19T11:32:09.550 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.550 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.550 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:09.550 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.078809+0000", 2024-03-19T11:32:09.550 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8980000000000001e-05 2024-03-19T11:32:09.550 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.550 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.550 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:09.551 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.078821+0000", 2024-03-19T11:32:09.551 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1432e-05 2024-03-19T11:32:09.551 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.551 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.551 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.554 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.079070+0000", 2024-03-19T11:32:09.554 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00024937100000000002 2024-03-19T11:32:09.554 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.555 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.555 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-03-19T11:32:09.555 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.079079+0000", 2024-03-19T11:32:09.555 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2949999999999999e-06 2024-03-19T11:32:09.555 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.555 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.555 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.555 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.083821+0000", 2024-03-19T11:32:09.555 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0047417129999999998 2024-03-19T11:32:09.555 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.555 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.555 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:09.555 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.083836+0000", 2024-03-19T11:32:09.555 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4591e-05 2024-03-19T11:32:09.555 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.556 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.556 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.556 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.101489+0000", 2024-03-19T11:32:09.556 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0176535019999999 2024-03-19T11:32:09.556 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.556 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.556 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:32:09.556 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.104918+0000", 2024-03-19T11:32:09.556 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003429353 2024-03-19T11:32:09.556 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.556 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.556 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.556 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.142247+0000", 2024-03-19T11:32:09.556 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.037329108 2024-03-19T11:32:09.557 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.557 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.557 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:09.557 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.142409+0000", 2024-03-19T11:32:09.557 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016169700000000001 2024-03-19T11:32:09.557 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.557 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.557 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4,5", 2024-03-19T11:32:09.557 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.142754+0000", 2024-03-19T11:32:09.557 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000345025 2024-03-19T11:32:09.557 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.557 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.558 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:32:09.558 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.161867+0000", 2024-03-19T11:32:09.558 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019112371 2024-03-19T11:32:09.558 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.558 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.558 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:09.558 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.166589+0000", 2024-03-19T11:32:09.558 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0047220270000000002 2024-03-19T11:32:09.558 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.558 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.558 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:09.558 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.196673+0000", 2024-03-19T11:32:09.558 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.030084422999999999 2024-03-19T11:32:09.558 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.558 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.559 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:32:09.559 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.196730+0000", 2024-03-19T11:32:09.559 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7027000000000001e-05 2024-03-19T11:32:09.559 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.559 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.559 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:09.559 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.196839+0000", 2024-03-19T11:32:09.559 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000109073 2024-03-19T11:32:09.559 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:09.559 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:09.559 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:09.559 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.559 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.559 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3886 4.21 4:8401c1f0:::smithi03829856-43:head [write 1657676~659775] snapc 29=[29,28,27,26,25,24,23,22,21,1f,1e,1d,1a,17] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e129)", 2024-03-19T11:32:09.559 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:50.078842+0000", 2024-03-19T11:32:09.560 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 19.384370272000002, 2024-03-19T11:32:09.560 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1181774870000001, 2024-03-19T11:32:09.560 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:09.560 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:09.560 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:09.560 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:09.560 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:09.560 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3886 2024-03-19T11:32:09.560 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.560 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:09.560 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.560 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:09.560 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.078842+0000", 2024-03-19T11:32:09.560 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:09.560 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.561 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.561 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:09.561 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.078842+0000", 2024-03-19T11:32:09.561 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:09.561 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.561 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.561 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:09.561 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.078843+0000", 2024-03-19T11:32:09.561 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7400000000000002e-07 2024-03-19T11:32:09.561 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.561 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.561 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:09.561 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.079370+0000", 2024-03-19T11:32:09.561 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00052713200000000003 2024-03-19T11:32:09.561 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.562 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.562 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:09.562 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.079383+0000", 2024-03-19T11:32:09.562 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3131e-05 2024-03-19T11:32:09.562 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.562 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.562 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:09.562 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.079393+0000", 2024-03-19T11:32:09.562 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0114e-05 2024-03-19T11:32:09.562 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.562 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.562 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.562 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082249+0000", 2024-03-19T11:32:09.562 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0028558260000000001 2024-03-19T11:32:09.563 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.563 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.563 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:32:09.563 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082258+0000", 2024-03-19T11:32:09.563 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.5740000000000002e-06 2024-03-19T11:32:09.563 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.563 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.563 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.563 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.084372+0000", 2024-03-19T11:32:09.563 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0021131909999999999 2024-03-19T11:32:09.563 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.563 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.563 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:09.563 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.084383+0000", 2024-03-19T11:32:09.564 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1099e-05 2024-03-19T11:32:09.564 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.564 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.564 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.564 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.105453+0000", 2024-03-19T11:32:09.564 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.021070116 2024-03-19T11:32:09.564 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.564 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.564 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:32:09.564 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.105574+0000", 2024-03-19T11:32:09.564 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012102199999999999 2024-03-19T11:32:09.565 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.565 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.565 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.565 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.151364+0000", 2024-03-19T11:32:09.565 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.045790294000000002 2024-03-19T11:32:09.565 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.565 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.565 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:09.565 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.151579+0000", 2024-03-19T11:32:09.565 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00021460399999999999 2024-03-19T11:32:09.565 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.565 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.565 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4,5", 2024-03-19T11:32:09.565 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.151927+0000", 2024-03-19T11:32:09.565 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00034839700000000002 2024-03-19T11:32:09.566 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.566 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.566 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:32:09.566 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.161878+0000", 2024-03-19T11:32:09.566 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0099512430000000002 2024-03-19T11:32:09.566 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.566 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.566 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:09.566 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.169256+0000", 2024-03-19T11:32:09.566 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0073777440000000003 2024-03-19T11:32:09.566 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.567 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.567 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:09.567 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.196889+0000", 2024-03-19T11:32:09.567 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027633179000000001 2024-03-19T11:32:09.567 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.567 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.567 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:32:09.567 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.196937+0000", 2024-03-19T11:32:09.567 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7966999999999997e-05 2024-03-19T11:32:09.567 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.567 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.567 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:09.567 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.197019+0000", 2024-03-19T11:32:09.567 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1979999999999998e-05 2024-03-19T11:32:09.568 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:09.568 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:09.568 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:09.568 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.568 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.568 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3887 4.21 4:8401c1f0:::smithi03829856-43:head [write 2818209~425575] snapc 29=[29,28,27,26,25,24,23,22,21,1f,1e,1d,1a,17] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e129)", 2024-03-19T11:32:09.568 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:50.082016+0000", 2024-03-19T11:32:09.568 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 19.381196431999999, 2024-03-19T11:32:09.568 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.115265664, 2024-03-19T11:32:09.568 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:09.568 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:09.568 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:09.568 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:09.568 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:09.568 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3887 2024-03-19T11:32:09.569 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.569 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:09.569 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.569 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:09.569 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082016+0000", 2024-03-19T11:32:09.569 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:09.569 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.569 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.569 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:09.569 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082016+0000", 2024-03-19T11:32:09.569 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:09.569 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.569 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.569 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:09.570 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082017+0000", 2024-03-19T11:32:09.570 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.587e-06 2024-03-19T11:32:09.570 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.570 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.570 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:09.570 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082203+0000", 2024-03-19T11:32:09.570 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000185613 2024-03-19T11:32:09.570 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.570 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.570 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:09.570 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082214+0000", 2024-03-19T11:32:09.570 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1357e-05 2024-03-19T11:32:09.570 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.570 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.570 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:09.571 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082221+0000", 2024-03-19T11:32:09.571 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6290000000000004e-06 2024-03-19T11:32:09.571 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.571 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.571 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.571 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082300+0000", 2024-03-19T11:32:09.571 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9425999999999995e-05 2024-03-19T11:32:09.571 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.571 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.571 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:32:09.571 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082306+0000", 2024-03-19T11:32:09.571 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1419999999999998e-06 2024-03-19T11:32:09.571 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.571 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.572 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.572 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.084417+0000", 2024-03-19T11:32:09.572 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0021107589999999998 2024-03-19T11:32:09.572 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.572 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.572 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:09.572 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.084426+0000", 2024-03-19T11:32:09.572 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8410000000000006e-06 2024-03-19T11:32:09.572 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.572 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.572 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.572 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.105770+0000", 2024-03-19T11:32:09.572 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0213442100000001 2024-03-19T11:32:09.572 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.572 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.573 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:32:09.573 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.105890+0000", 2024-03-19T11:32:09.573 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000120349 2024-03-19T11:32:09.573 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.573 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.573 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.573 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.157275+0000", 2024-03-19T11:32:09.573 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.051384678000000003 2024-03-19T11:32:09.573 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.574 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.574 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:09.574 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.157445+0000", 2024-03-19T11:32:09.574 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000169746 2024-03-19T11:32:09.574 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.574 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.574 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4,5", 2024-03-19T11:32:09.574 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.157790+0000", 2024-03-19T11:32:09.574 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00034491799999999999 2024-03-19T11:32:09.574 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.574 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.574 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:32:09.574 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.161935+0000", 2024-03-19T11:32:09.575 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0041450269999999999 2024-03-19T11:32:09.575 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.575 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.575 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:09.575 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.189666+0000", 2024-03-19T11:32:09.575 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027731062000000001 2024-03-19T11:32:09.575 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.575 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.575 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:09.575 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.197073+0000", 2024-03-19T11:32:09.575 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074070270000000001 2024-03-19T11:32:09.575 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.575 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.575 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:32:09.576 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.197116+0000", 2024-03-19T11:32:09.576 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2758e-05 2024-03-19T11:32:09.576 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.576 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.576 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:09.576 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.197281+0000", 2024-03-19T11:32:09.576 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016553499999999999 2024-03-19T11:32:09.576 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:09.576 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:09.576 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:09.576 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.576 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.576 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3888 4.21 4:8401c1f0:::smithi03829856-43:head [setxattr _header (57) in=7b,truncate 3243784] snapc 29=[29,28,27,26,25,24,23,22,21,1f,1e,1d,1a,17] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e129)", 2024-03-19T11:32:09.576 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:50.082238+0000", 2024-03-19T11:32:09.577 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 19.380973549, 2024-03-19T11:32:09.577 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.115518945, 2024-03-19T11:32:09.577 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:09.577 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:09.577 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:09.577 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:09.577 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:09.578 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3888 2024-03-19T11:32:09.578 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.578 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:09.578 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.578 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:09.578 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082238+0000", 2024-03-19T11:32:09.578 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:09.578 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.578 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.578 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:09.579 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082238+0000", 2024-03-19T11:32:09.579 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:09.579 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.579 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.579 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:09.579 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082239+0000", 2024-03-19T11:32:09.579 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6400000000000001e-07 2024-03-19T11:32:09.579 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.579 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.579 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:09.579 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082249+0000", 2024-03-19T11:32:09.579 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9490000000000008e-06 2024-03-19T11:32:09.579 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.579 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.579 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:09.580 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082260+0000", 2024-03-19T11:32:09.580 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0499000000000001e-05 2024-03-19T11:32:09.580 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.580 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.580 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:09.580 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082267+0000", 2024-03-19T11:32:09.580 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9029999999999998e-06 2024-03-19T11:32:09.580 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.580 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.580 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.580 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082341+0000", 2024-03-19T11:32:09.580 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4776000000000004e-05 2024-03-19T11:32:09.580 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.580 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.581 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:32:09.581 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082350+0000", 2024-03-19T11:32:09.581 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2279999999999998e-06 2024-03-19T11:32:09.581 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.581 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.581 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.581 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.084459+0000", 2024-03-19T11:32:09.581 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0021094579999999998 2024-03-19T11:32:09.581 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.581 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.581 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:09.581 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.084468+0000", 2024-03-19T11:32:09.581 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2e-06 2024-03-19T11:32:09.581 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.582 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.582 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.582 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.105958+0000", 2024-03-19T11:32:09.582 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0214894450000001 2024-03-19T11:32:09.582 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.582 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.582 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:32:09.582 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.106112+0000", 2024-03-19T11:32:09.582 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000154022 2024-03-19T11:32:09.582 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.582 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.582 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.582 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.160042+0000", 2024-03-19T11:32:09.582 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.053929935999999998 2024-03-19T11:32:09.583 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.583 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.583 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:09.583 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.160205+0000", 2024-03-19T11:32:09.583 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016317600000000001 2024-03-19T11:32:09.583 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.583 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.583 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 4,5", 2024-03-19T11:32:09.583 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.160566+0000", 2024-03-19T11:32:09.583 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000360918 2024-03-19T11:32:09.583 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.583 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.584 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:32:09.584 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.162064+0000", 2024-03-19T11:32:09.584 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014983380000000001 2024-03-19T11:32:09.584 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.584 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.584 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:09.584 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.196608+0000", 2024-03-19T11:32:09.584 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.034544115 2024-03-19T11:32:09.584 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.584 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.584 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:09.584 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.197420+0000", 2024-03-19T11:32:09.585 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00081147299999999995 2024-03-19T11:32:09.585 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.585 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.585 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:32:09.585 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.197519+0000", 2024-03-19T11:32:09.585 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8932999999999996e-05 2024-03-19T11:32:09.585 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.585 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.585 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:09.585 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.197757+0000", 2024-03-19T11:32:09.585 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000238812 2024-03-19T11:32:09.585 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:09.585 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:09.586 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:09.586 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.586 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.586 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3889 4.21 4:8401c1f0:::smithi03829856-43:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e129)", 2024-03-19T11:32:09.586 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:50.082302+0000", 2024-03-19T11:32:09.586 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 19.380910400000001, 2024-03-19T11:32:09.586 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.116076547, 2024-03-19T11:32:09.586 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:09.586 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:09.586 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:09.586 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:09.586 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:09.586 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3889 2024-03-19T11:32:09.587 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.587 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:09.587 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.587 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:09.587 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082302+0000", 2024-03-19T11:32:09.587 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:09.587 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.587 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.587 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:09.587 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082302+0000", 2024-03-19T11:32:09.587 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:09.587 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.587 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.588 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:09.588 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082303+0000", 2024-03-19T11:32:09.588 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2720000000000001e-06 2024-03-19T11:32:09.588 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.588 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.588 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:09.588 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082314+0000", 2024-03-19T11:32:09.588 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0864e-05 2024-03-19T11:32:09.588 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.588 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.588 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:09.589 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082327+0000", 2024-03-19T11:32:09.589 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2972e-05 2024-03-19T11:32:09.589 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.589 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.589 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:09.589 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.082339+0000", 2024-03-19T11:32:09.589 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2458e-05 2024-03-19T11:32:09.589 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.589 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.589 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.589 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.085086+0000", 2024-03-19T11:32:09.589 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0027472859999999998 2024-03-19T11:32:09.589 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.589 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.590 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:09.590 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:50.085100+0000", 2024-03-19T11:32:09.590 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3339000000000001e-05 2024-03-19T11:32:09.590 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.590 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.590 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.590 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.106149+0000", 2024-03-19T11:32:09.590 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0210496010000001 2024-03-19T11:32:09.590 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.590 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.590 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:32:09.590 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.106232+0000", 2024-03-19T11:32:09.590 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2923999999999994e-05 2024-03-19T11:32:09.590 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.591 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.591 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.591 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.161418+0000", 2024-03-19T11:32:09.591 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.055186156 2024-03-19T11:32:09.591 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.591 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.591 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for rw locks", 2024-03-19T11:32:09.591 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.161535+0000", 2024-03-19T11:32:09.591 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000117018 2024-03-19T11:32:09.591 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.591 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.591 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:09.591 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.197872+0000", 2024-03-19T11:32:09.592 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.036336161999999998 2024-03-19T11:32:09.592 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.592 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.592 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:09.592 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.198037+0000", 2024-03-19T11:32:09.592 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000164924 2024-03-19T11:32:09.592 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:09.592 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:09.592 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:09.592 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:51.198378+0000", 2024-03-19T11:32:09.592 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00034157099999999999 2024-03-19T11:32:09.592 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:09.592 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:09.592 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:09.593 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:09.593 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:09.593 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:32:09.594 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:32:09.465+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:32:09.595 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-19T11:32:09.595 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:32:09.603+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:32:09.695 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:32:09.701+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:32:09.768 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:32:09.768 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:32:09.768 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:32:09.768 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:32:09.782 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-19T11:32:09.829 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:32:09.839+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:32:09.897 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:32:09.907+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:32:09.900 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:32:09.900 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:32:09.900 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:32:09.900 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:32:09.900 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:32:09.915 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-19T11:32:10.001 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:32:10.011+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:32:10.086 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:32:10.093+0000 7f143d8b8640 -1 osd.5 148 handle_osd_map injecting map crc failure 2024-03-19T11:32:10.087 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-03-19T11:32:10.094 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:32:10.094 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:32:10.095 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:32:10.095 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:32:10.095 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.095 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1063 4.4 4:214026b3:::smithi03829856-47:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:32:10.095 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:13.784904+0000", 2024-03-19T11:32:10.095 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 116.3137748, 2024-03-19T11:32:10.095 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024425912000000001, 2024-03-19T11:32:10.095 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:10.095 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:10.096 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:10.096 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:10.096 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:10.096 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1063 2024-03-19T11:32:10.096 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.096 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:10.096 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.096 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:10.096 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784904+0000", 2024-03-19T11:32:10.096 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.097 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.097 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.097 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:10.097 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784904+0000", 2024-03-19T11:32:10.097 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.097 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.097 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.097 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:10.097 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784906+0000", 2024-03-19T11:32:10.097 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1409999999999999e-06 2024-03-19T11:32:10.097 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.098 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.098 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:10.098 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784915+0000", 2024-03-19T11:32:10.098 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5459999999999992e-06 2024-03-19T11:32:10.098 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.098 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.098 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:10.098 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784935+0000", 2024-03-19T11:32:10.098 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0310999999999999e-05 2024-03-19T11:32:10.098 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.098 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.098 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:10.098 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.784946+0000", 2024-03-19T11:32:10.098 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0675e-05 2024-03-19T11:32:10.099 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.099 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.099 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.099 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.785032+0000", 2024-03-19T11:32:10.099 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6049000000000002e-05 2024-03-19T11:32:10.099 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.099 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.099 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:10.099 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.785136+0000", 2024-03-19T11:32:10.099 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000104209 2024-03-19T11:32:10.099 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.100 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.100 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:10.100 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:13.809330+0000", 2024-03-19T11:32:10.100 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024193981 2024-03-19T11:32:10.100 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.100 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:10.100 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.100 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.101 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.101 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1433 4.16 4:6be89c06:::smithi03829856-24:c [sparse-read 0~1100427,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:32:10.101 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.976652+0000", 2024-03-19T11:32:10.101 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 104.12202709899999, 2024-03-19T11:32:10.101 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95684753199999995, 2024-03-19T11:32:10.101 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:10.101 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:10.101 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:10.101 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:10.101 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:10.101 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1433 2024-03-19T11:32:10.102 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.102 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:10.102 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.102 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:10.102 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976652+0000", 2024-03-19T11:32:10.103 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.103 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.103 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.103 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:10.103 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976652+0000", 2024-03-19T11:32:10.103 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.103 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.103 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.103 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:10.103 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976654+0000", 2024-03-19T11:32:10.103 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6199999999999999e-06 2024-03-19T11:32:10.103 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.104 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.104 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:10.104 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976668+0000", 2024-03-19T11:32:10.104 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3213999999999999e-05 2024-03-19T11:32:10.104 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.104 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.104 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:10.104 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976694+0000", 2024-03-19T11:32:10.104 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6488000000000001e-05 2024-03-19T11:32:10.104 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.104 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.104 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:10.104 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976705+0000", 2024-03-19T11:32:10.104 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1044e-05 2024-03-19T11:32:10.105 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.105 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.105 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.105 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976795+0000", 2024-03-19T11:32:10.105 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0283000000000003e-05 2024-03-19T11:32:10.105 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.105 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.105 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:10.105 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976811+0000", 2024-03-19T11:32:10.105 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5262000000000001e-05 2024-03-19T11:32:10.105 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.105 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.106 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.106 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.927337+0000", 2024-03-19T11:32:10.106 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95052597100000003 2024-03-19T11:32:10.106 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.106 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.106 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:10.106 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.927495+0000", 2024-03-19T11:32:10.106 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015788600000000001 2024-03-19T11:32:10.106 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.106 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.106 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:10.106 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.933499+0000", 2024-03-19T11:32:10.106 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0060047640000000001 2024-03-19T11:32:10.106 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.106 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:10.107 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.107 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.107 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.107 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1434 4.16 4:6be89c06:::smithi03829856-24:c [read 0~2386561,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:32:10.107 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.976739+0000", 2024-03-19T11:32:10.107 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 104.121939938, 2024-03-19T11:32:10.107 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95781317399999999, 2024-03-19T11:32:10.107 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:10.107 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:10.108 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:10.108 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:10.108 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:10.108 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1434 2024-03-19T11:32:10.108 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.108 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:10.108 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.108 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:10.108 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976739+0000", 2024-03-19T11:32:10.109 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.109 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.109 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.109 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:10.109 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976739+0000", 2024-03-19T11:32:10.109 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.109 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.109 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.109 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:10.109 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976740+0000", 2024-03-19T11:32:10.109 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8800000000000001e-07 2024-03-19T11:32:10.109 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.109 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.109 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:10.110 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976748+0000", 2024-03-19T11:32:10.110 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8979999999999996e-06 2024-03-19T11:32:10.110 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.110 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.110 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:10.110 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976757+0000", 2024-03-19T11:32:10.110 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9439999999999994e-06 2024-03-19T11:32:10.110 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.111 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.111 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:10.111 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976763+0000", 2024-03-19T11:32:10.111 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5030000000000002e-06 2024-03-19T11:32:10.111 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.111 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.111 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.111 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976849+0000", 2024-03-19T11:32:10.111 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5803999999999994e-05 2024-03-19T11:32:10.111 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.111 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.111 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:10.111 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976860+0000", 2024-03-19T11:32:10.111 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0869e-05 2024-03-19T11:32:10.111 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.112 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.112 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.112 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.933562+0000", 2024-03-19T11:32:10.112 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95670232300000002 2024-03-19T11:32:10.112 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.112 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.112 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:10.112 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.933693+0000", 2024-03-19T11:32:10.112 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013110699999999999 2024-03-19T11:32:10.113 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.113 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.113 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:10.113 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.934552+0000", 2024-03-19T11:32:10.113 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00085883800000000001 2024-03-19T11:32:10.113 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.113 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:10.113 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.113 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.113 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.113 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1435 4.16 4:6be89c06:::smithi03829856-24:c [sparse-read 0~1100427] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:32:10.113 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.976775+0000", 2024-03-19T11:32:10.114 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 104.12190426700001, 2024-03-19T11:32:10.114 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95962779499999995, 2024-03-19T11:32:10.114 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:10.114 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:10.114 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:10.114 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:10.114 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:10.114 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1435 2024-03-19T11:32:10.114 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.114 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:10.114 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.114 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:10.114 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976775+0000", 2024-03-19T11:32:10.114 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.115 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.115 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.115 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:10.115 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976775+0000", 2024-03-19T11:32:10.115 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.115 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.115 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.115 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:10.115 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976775+0000", 2024-03-19T11:32:10.115 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9800000000000003e-07 2024-03-19T11:32:10.115 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.115 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.115 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:10.115 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976779+0000", 2024-03-19T11:32:10.116 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3830000000000001e-06 2024-03-19T11:32:10.116 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.116 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.116 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:10.116 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976785+0000", 2024-03-19T11:32:10.116 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5649999999999998e-06 2024-03-19T11:32:10.116 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.116 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.116 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:10.116 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976790+0000", 2024-03-19T11:32:10.116 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1379999999999998e-06 2024-03-19T11:32:10.116 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.116 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.116 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.116 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976889+0000", 2024-03-19T11:32:10.117 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8331000000000004e-05 2024-03-19T11:32:10.117 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.117 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.117 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:10.117 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.976894+0000", 2024-03-19T11:32:10.117 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8270000000000003e-06 2024-03-19T11:32:10.117 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.117 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.117 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.117 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.935982+0000", 2024-03-19T11:32:10.117 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95908714100000003 2024-03-19T11:32:10.117 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.117 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.117 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:10.117 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.936093+0000", 2024-03-19T11:32:10.117 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000111557 2024-03-19T11:32:10.118 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.118 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.118 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:10.118 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.936402+0000", 2024-03-19T11:32:10.118 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030925499999999998 2024-03-19T11:32:10.118 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.118 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:10.118 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.118 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.118 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.118 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1442 4.6 4:626acd4d:::smithi03829856-2:head [cache-evict] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:32:10.118 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:25.978864+0000", 2024-03-19T11:32:10.118 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 104.11981460299999, 2024-03-19T11:32:10.118 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95708845099999995, 2024-03-19T11:32:10.119 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:10.119 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:10.119 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:10.119 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:10.119 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:10.119 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1442 2024-03-19T11:32:10.119 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.119 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:10.119 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.119 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:10.119 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978864+0000", 2024-03-19T11:32:10.119 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.119 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.119 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.119 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:10.120 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978864+0000", 2024-03-19T11:32:10.120 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.120 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.120 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.120 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:10.120 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978866+0000", 2024-03-19T11:32:10.120 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0729999999999999e-06 2024-03-19T11:32:10.120 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.120 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.120 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:10.120 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978877+0000", 2024-03-19T11:32:10.120 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0390999999999999e-05 2024-03-19T11:32:10.120 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.120 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.121 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:10.121 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978902+0000", 2024-03-19T11:32:10.121 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5635000000000001e-05 2024-03-19T11:32:10.121 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.121 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.121 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:10.121 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.978915+0000", 2024-03-19T11:32:10.121 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2617e-05 2024-03-19T11:32:10.121 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.121 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.121 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.121 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.979005+0000", 2024-03-19T11:32:10.121 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0552999999999999e-05 2024-03-19T11:32:10.121 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.121 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.121 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:10.122 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:25.979021+0000", 2024-03-19T11:32:10.122 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5102999999999999e-05 2024-03-19T11:32:10.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.122 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.122 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.926879+0000", 2024-03-19T11:32:10.122 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.94785817900000002 2024-03-19T11:32:10.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.122 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-19T11:32:10.122 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.927277+0000", 2024-03-19T11:32:10.122 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00039819600000000002 2024-03-19T11:32:10.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.123 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.123 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.935752+0000", 2024-03-19T11:32:10.123 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0084753420000000003 2024-03-19T11:32:10.123 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.123 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.123 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:10.123 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.935874+0000", 2024-03-19T11:32:10.123 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000121311 2024-03-19T11:32:10.124 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.124 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.124 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:10.124 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.935953+0000", 2024-03-19T11:32:10.124 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9050999999999999e-05 2024-03-19T11:32:10.124 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.124 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:10.124 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.124 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.124 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.124 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2003 4.16 4:6868af91:::smithi03829856-10:head [tier-promote] snapc 12=[12,11,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:32:10.124 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:45.804046+0000", 2024-03-19T11:32:10.124 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 84.29463269, 2024-03-19T11:32:10.124 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018976396, 2024-03-19T11:32:10.124 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:10.125 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:10.125 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:10.125 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:10.125 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:10.125 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2003 2024-03-19T11:32:10.125 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.125 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:10.125 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.125 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:10.125 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804046+0000", 2024-03-19T11:32:10.125 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.125 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.125 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.125 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:10.125 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804046+0000", 2024-03-19T11:32:10.126 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.126 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.126 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.126 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:10.126 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804052+0000", 2024-03-19T11:32:10.126 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7599999999999999e-06 2024-03-19T11:32:10.126 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.126 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.126 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:10.126 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804090+0000", 2024-03-19T11:32:10.126 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8018000000000001e-05 2024-03-19T11:32:10.126 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.126 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.127 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:10.127 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804256+0000", 2024-03-19T11:32:10.127 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016593599999999999 2024-03-19T11:32:10.127 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.127 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.127 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:10.127 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804300+0000", 2024-03-19T11:32:10.127 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3648e-05 2024-03-19T11:32:10.127 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.127 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.127 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.127 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804538+0000", 2024-03-19T11:32:10.127 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000238347 2024-03-19T11:32:10.127 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.127 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.128 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:10.128 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804716+0000", 2024-03-19T11:32:10.128 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000177968 2024-03-19T11:32:10.128 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.128 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.128 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:10.128 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.823023+0000", 2024-03-19T11:32:10.128 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018306718999999999 2024-03-19T11:32:10.128 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.128 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:10.128 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.128 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.128 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.128 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1d58db11:::smithi03829856-18:6, version: 55'80, data_included: [759683~639358,1904434~409218], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:6@55'80, size: 3159683, copy_subset: [(0, 3159683)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2313652, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1d58db11:::smithi03829856-18:b, version: 58'84, data_included: [443718~955323,1564138~93253], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:b@58'84, size: 3445402, copy_subset: [(0, 3445402)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1657391, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:10, version: 66'95, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:10@66'95, size: 3764735, copy_subset: [(0, 3764735)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:12, version: 66'97, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:12@66'97, size: 2504243, copy_subset: [(0, 2504243)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:c, version: 70'102, data_included: [416807~576167,1542129~472409], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:c@70'102, size: 2857157, copy_subset: [(0, 2857157)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2014538, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:32:10.129 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.275179+0000", 2024-03-19T11:32:10.129 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 75.823499699999999, 2024-03-19T11:32:10.129 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026559353000000001, 2024-03-19T11:32:10.129 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:10.129 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:10.129 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:10.129 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.129 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:10.129 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.275179+0000", 2024-03-19T11:32:10.129 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.129 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.129 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.129 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:10.129 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.275179+0000", 2024-03-19T11:32:10.129 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.130 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.130 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.130 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:10.130 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.275196+0000", 2024-03-19T11:32:10.130 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6946999999999999e-05 2024-03-19T11:32:10.130 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.130 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.130 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:10.130 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281456+0000", 2024-03-19T11:32:10.130 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0062600529999999998 2024-03-19T11:32:10.130 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.130 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.130 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:10.130 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281524+0000", 2024-03-19T11:32:10.130 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7971000000000004e-05 2024-03-19T11:32:10.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.131 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.131 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:10.131 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281591+0000", 2024-03-19T11:32:10.131 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6704000000000004e-05 2024-03-19T11:32:10.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.131 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.131 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.131 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281865+0000", 2024-03-19T11:32:10.131 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000274097 2024-03-19T11:32:10.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.131 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.131 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:10.131 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281901+0000", 2024-03-19T11:32:10.131 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5777e-05 2024-03-19T11:32:10.132 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.132 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.132 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:10.132 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.301738+0000", 2024-03-19T11:32:10.132 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019837804000000001 2024-03-19T11:32:10.132 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.132 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:10.132 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.132 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.132 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.132 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.18 e75,74 4:1d58db11:::smithi03829856-18:2,48'76)", 2024-03-19T11:32:10.132 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.281661+0000", 2024-03-19T11:32:10.132 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 75.817018168999994, 2024-03-19T11:32:10.132 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020384337999999998, 2024-03-19T11:32:10.132 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:10.133 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:10.133 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:10.133 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.133 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:10.133 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281661+0000", 2024-03-19T11:32:10.133 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.133 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.133 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.134 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:10.134 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281661+0000", 2024-03-19T11:32:10.134 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.134 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.134 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.134 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:10.134 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281661+0000", 2024-03-19T11:32:10.134 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9100000000000003e-07 2024-03-19T11:32:10.134 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.134 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.134 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:10.134 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281668+0000", 2024-03-19T11:32:10.134 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8199999999999999e-06 2024-03-19T11:32:10.134 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.135 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.135 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:10.135 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281679+0000", 2024-03-19T11:32:10.135 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1107999999999999e-05 2024-03-19T11:32:10.135 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.135 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.135 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:10.135 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281691+0000", 2024-03-19T11:32:10.135 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1399999999999999e-05 2024-03-19T11:32:10.135 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.135 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.135 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.135 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.301804+0000", 2024-03-19T11:32:10.135 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020113669000000001 2024-03-19T11:32:10.136 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.136 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.136 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:10.136 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.301818+0000", 2024-03-19T11:32:10.136 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3718e-05 2024-03-19T11:32:10.136 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.136 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.136 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:10.136 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302045+0000", 2024-03-19T11:32:10.136 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000226832 2024-03-19T11:32:10.136 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.136 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:10.137 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.137 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.137 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.137 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1dabc0f1:::smithi03829856-1:head, version: 70'103, data_included: [416807~761662,1542129~286914], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:head@70'103, size: 2621892, copy_subset: [(0, 2621892)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:32:10.137 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.281783+0000", 2024-03-19T11:32:10.137 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 75.816895892999995, 2024-03-19T11:32:10.137 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026187221, 2024-03-19T11:32:10.137 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:10.137 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:10.137 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:10.137 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.137 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:10.137 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281783+0000", 2024-03-19T11:32:10.137 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.138 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.138 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.138 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:10.138 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281783+0000", 2024-03-19T11:32:10.138 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.138 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.138 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.138 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:10.138 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.281783+0000", 2024-03-19T11:32:10.138 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1699999999999999e-07 2024-03-19T11:32:10.138 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.138 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.138 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:10.138 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.282962+0000", 2024-03-19T11:32:10.138 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001178601 2024-03-19T11:32:10.139 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.139 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.139 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:10.139 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.282987+0000", 2024-03-19T11:32:10.139 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4899e-05 2024-03-19T11:32:10.139 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.139 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.139 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:10.139 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283006+0000", 2024-03-19T11:32:10.139 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9494999999999999e-05 2024-03-19T11:32:10.139 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.139 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.139 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.139 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.305757+0000", 2024-03-19T11:32:10.139 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.022750570000000001 2024-03-19T11:32:10.140 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.140 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.140 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:10.140 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.305770+0000", 2024-03-19T11:32:10.140 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2791e-05 2024-03-19T11:32:10.140 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.140 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.140 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:10.140 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.307970+0000", 2024-03-19T11:32:10.140 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022005480000000001 2024-03-19T11:32:10.140 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.140 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:10.140 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.140 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.140 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.141 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.18 e75,74 4:1d58db11:::smithi03829856-18:c,70'104)", 2024-03-19T11:32:10.141 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.283036+0000", 2024-03-19T11:32:10.141 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 75.815642792999995, 2024-03-19T11:32:10.141 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019146032, 2024-03-19T11:32:10.141 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:10.141 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:10.141 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:10.141 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.141 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:10.141 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283036+0000", 2024-03-19T11:32:10.141 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.141 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.141 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.141 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:10.141 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283036+0000", 2024-03-19T11:32:10.142 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.142 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.142 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.142 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:10.142 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283037+0000", 2024-03-19T11:32:10.142 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4899999999999995e-07 2024-03-19T11:32:10.142 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.142 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.142 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:10.142 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283042+0000", 2024-03-19T11:32:10.142 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0930000000000002e-06 2024-03-19T11:32:10.142 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.142 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.142 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:10.143 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283053+0000", 2024-03-19T11:32:10.143 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1083999999999999e-05 2024-03-19T11:32:10.143 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.143 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.143 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:10.143 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283059+0000", 2024-03-19T11:32:10.143 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6329999999999999e-06 2024-03-19T11:32:10.143 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.143 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.143 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.143 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302090+0000", 2024-03-19T11:32:10.143 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01903038 2024-03-19T11:32:10.143 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.143 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.144 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:10.144 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302097+0000", 2024-03-19T11:32:10.144 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.221e-06 2024-03-19T11:32:10.144 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.144 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.144 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:10.144 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302182+0000", 2024-03-19T11:32:10.144 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5072000000000001e-05 2024-03-19T11:32:10.144 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.145 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:10.145 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.145 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.145 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.145 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(osd.5.0:143 4.18 e75/74 4:1d58db11:::smithi03829856-18:head v 75'106, mlcod=0'0)", 2024-03-19T11:32:10.145 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.283083+0000", 2024-03-19T11:32:10.145 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 75.815595466000005, 2024-03-19T11:32:10.145 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025309439, 2024-03-19T11:32:10.145 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:10.145 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:10.145 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:10.145 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.145 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:10.145 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283083+0000", 2024-03-19T11:32:10.145 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.146 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.146 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.146 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:10.146 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283083+0000", 2024-03-19T11:32:10.146 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.146 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.146 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.146 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:10.146 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.283084+0000", 2024-03-19T11:32:10.146 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7800000000000002e-07 2024-03-19T11:32:10.146 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.146 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.146 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:10.146 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.284082+0000", 2024-03-19T11:32:10.147 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0009977880000000001 2024-03-19T11:32:10.147 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.147 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.147 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:10.147 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.284095+0000", 2024-03-19T11:32:10.147 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3256e-05 2024-03-19T11:32:10.147 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.147 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.147 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:10.147 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.284102+0000", 2024-03-19T11:32:10.147 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.605e-06 2024-03-19T11:32:10.147 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.147 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.147 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.147 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302285+0000", 2024-03-19T11:32:10.148 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018182423 2024-03-19T11:32:10.148 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.148 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.148 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:10.148 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.302341+0000", 2024-03-19T11:32:10.148 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.626e-05 2024-03-19T11:32:10.148 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.148 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.148 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:32:10.148 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.308338+0000", 2024-03-19T11:32:10.148 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0059968030000000002 2024-03-19T11:32:10.148 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.148 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.148 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:10.149 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.308393+0000", 2024-03-19T11:32:10.149 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4826e-05 2024-03-19T11:32:10.149 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.149 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:10.149 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.149 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.149 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.149 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "pg_update_log_missing(4.18 epoch 75/74 rep_tid 144 entries 75'107 (0'0) error 4:1d58db11:::smithi03829856-18:head by client.4415.0:2283 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-19T11:32:10.149 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.288252+0000", 2024-03-19T11:32:10.149 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 75.810426915999997, 2024-03-19T11:32:10.149 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020167377, 2024-03-19T11:32:10.149 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:10.149 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-03-19T11:32:10.149 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:10.149 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.150 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:10.150 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288252+0000", 2024-03-19T11:32:10.150 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.150 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.150 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.150 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:10.150 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288252+0000", 2024-03-19T11:32:10.150 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.150 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.150 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.150 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:10.150 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288253+0000", 2024-03-19T11:32:10.150 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2109999999999999e-06 2024-03-19T11:32:10.150 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.151 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.151 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:10.151 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288268+0000", 2024-03-19T11:32:10.151 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5003e-05 2024-03-19T11:32:10.151 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.151 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.151 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:10.151 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288301+0000", 2024-03-19T11:32:10.151 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2539999999999997e-05 2024-03-19T11:32:10.151 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.151 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.151 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:10.151 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.288319+0000", 2024-03-19T11:32:10.151 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8232000000000001e-05 2024-03-19T11:32:10.151 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.152 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.152 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.152 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.305490+0000", 2024-03-19T11:32:10.152 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017170857000000001 2024-03-19T11:32:10.152 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.152 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.152 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:10.152 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.308419+0000", 2024-03-19T11:32:10.152 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029295340000000001 2024-03-19T11:32:10.152 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.152 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:10.152 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.152 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.152 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.152 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1d58db11:::smithi03829856-18:6, version: 55'80, data_included: [2313652~310127,3128657~31026], data_size: 341153, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:6@55'80, size: 3159683, copy_subset: [(0, 3159683)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3159683, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2313652, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:1d58db11:::smithi03829856-18:b, version: 58'84, data_included: [1657391~966388,2633066~82188], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1d58db11:::smithi03829856-18:b@58'84, size: 3445402, copy_subset: [(0, 3445402)], clone_subset: {}, snapset: c=[]:{6: [6], b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2715254, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1657391, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:10, version: 66'95, data_included: [1829043~668012,3134120~380564], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:10@66'95, size: 3764735, copy_subset: [(0, 3764735)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3514684, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:12, version: 66'97, data_included: [1829043~507467], data_size: 507467, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:12@66'97, size: 2504243, copy_subset: [(0, 2504243)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2504243, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:1dabc0f1:::smithi03829856-1:c, version: 70'102, data_included: [2014538~156250], data_size: 156250, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:c@70'102, size: 2857157, copy_subset: [(0, 2857157)], clone_subset: {}, snapset: 12=[]:{c: [b, 6], 10: [d], 12: [12, 11]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2857157, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2014538, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:32:10.153 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.312362+0000", 2024-03-19T11:32:10.153 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 75.786316866999996, 2024-03-19T11:32:10.153 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017968951, 2024-03-19T11:32:10.153 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:10.153 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:10.153 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:10.153 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.153 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:10.153 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.312362+0000", 2024-03-19T11:32:10.153 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.153 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.153 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.153 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:10.153 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.312362+0000", 2024-03-19T11:32:10.153 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.154 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.154 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.154 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:10.154 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.312363+0000", 2024-03-19T11:32:10.154 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.528e-06 2024-03-19T11:32:10.154 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.154 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.154 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:10.154 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315748+0000", 2024-03-19T11:32:10.154 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033842830000000001 2024-03-19T11:32:10.155 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.155 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.155 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:10.155 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315795+0000", 2024-03-19T11:32:10.155 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7382999999999997e-05 2024-03-19T11:32:10.155 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.155 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.155 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:10.155 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315845+0000", 2024-03-19T11:32:10.155 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0319999999999999e-05 2024-03-19T11:32:10.155 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.155 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.155 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.155 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.316123+0000", 2024-03-19T11:32:10.155 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00027747200000000002 2024-03-19T11:32:10.156 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.156 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.156 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:10.156 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.316170+0000", 2024-03-19T11:32:10.156 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7466999999999998e-05 2024-03-19T11:32:10.156 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.156 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.156 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:10.156 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.330331+0000", 2024-03-19T11:32:10.156 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014160498000000001 2024-03-19T11:32:10.156 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.156 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:10.156 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.156 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.157 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.157 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.18 75/74 [PushOp(4:1dabc0f1:::smithi03829856-1:head, version: 70'103, data_included: [1829043~555153], data_size: 555153, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:1dabc0f1:::smithi03829856-1:head@70'103, size: 2621892, copy_subset: [(0, 2621892)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2621892, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1829043, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:32:10.157 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:54.315898+0000", 2024-03-19T11:32:10.157 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 75.782780880999994, 2024-03-19T11:32:10.157 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016880424000000002, 2024-03-19T11:32:10.157 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:10.157 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:10.157 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:10.157 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.157 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:10.157 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315898+0000", 2024-03-19T11:32:10.157 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.157 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.157 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.157 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:10.158 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315898+0000", 2024-03-19T11:32:10.158 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.158 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.158 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.158 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:10.158 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.315899+0000", 2024-03-19T11:32:10.158 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.003e-06 2024-03-19T11:32:10.158 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.158 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.158 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:10.158 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.316486+0000", 2024-03-19T11:32:10.158 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00058673099999999995 2024-03-19T11:32:10.158 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.158 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.159 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:10.159 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.316519+0000", 2024-03-19T11:32:10.159 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3124000000000003e-05 2024-03-19T11:32:10.159 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.159 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.159 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:10.159 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.316534+0000", 2024-03-19T11:32:10.159 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.485e-05 2024-03-19T11:32:10.159 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.159 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.159 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.159 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.330415+0000", 2024-03-19T11:32:10.159 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013880905000000001 2024-03-19T11:32:10.159 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.159 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.160 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:10.160 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.330439+0000", 2024-03-19T11:32:10.160 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4630999999999998e-05 2024-03-19T11:32:10.160 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.160 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.160 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:10.160 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:54.332778+0000", 2024-03-19T11:32:10.160 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023391800000000002 2024-03-19T11:32:10.160 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.160 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:10.160 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.160 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.160 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.160 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2387 4.2 4:43b32242:::smithi03829856-22:head [read 596992~684544] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-03-19T11:32:10.161 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:59.319806+0000", 2024-03-19T11:32:10.161 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 70.778873214000001, 2024-03-19T11:32:10.161 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.78144344399999999, 2024-03-19T11:32:10.161 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:10.161 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:10.161 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:10.161 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:10.161 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:10.161 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2387 2024-03-19T11:32:10.161 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.161 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:10.161 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.161 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:10.161 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319806+0000", 2024-03-19T11:32:10.161 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.162 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.162 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.162 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:10.162 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319806+0000", 2024-03-19T11:32:10.162 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.162 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.162 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.162 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:10.162 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319807+0000", 2024-03-19T11:32:10.162 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3519999999999999e-06 2024-03-19T11:32:10.162 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.162 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.162 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:10.162 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319812+0000", 2024-03-19T11:32:10.163 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3959999999999997e-06 2024-03-19T11:32:10.163 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.163 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.163 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:10.163 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319827+0000", 2024-03-19T11:32:10.163 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4566000000000001e-05 2024-03-19T11:32:10.163 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.163 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.163 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:10.163 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.319835+0000", 2024-03-19T11:32:10.163 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.994e-06 2024-03-19T11:32:10.163 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.163 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.163 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.163 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.320119+0000", 2024-03-19T11:32:10.164 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00028445199999999998 2024-03-19T11:32:10.164 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.164 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.164 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-19T11:32:10.164 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:59.320129+0000", 2024-03-19T11:32:10.164 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9580000000000002e-06 2024-03-19T11:32:10.164 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.164 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.164 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.164 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.069842+0000", 2024-03-19T11:32:10.164 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.74971218399999995 2024-03-19T11:32:10.164 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.164 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.164 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:32:10.164 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.070010+0000", 2024-03-19T11:32:10.165 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016846600000000001 2024-03-19T11:32:10.165 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.165 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.165 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.165 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.090911+0000", 2024-03-19T11:32:10.165 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020901494999999999 2024-03-19T11:32:10.165 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.165 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.165 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-19T11:32:10.165 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.091249+0000", 2024-03-19T11:32:10.165 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000337602 2024-03-19T11:32:10.165 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.165 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.165 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.165 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.098478+0000", 2024-03-19T11:32:10.166 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0072287549999999999 2024-03-19T11:32:10.166 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.166 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.166 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:10.166 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.098617+0000", 2024-03-19T11:32:10.166 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013871599999999999 2024-03-19T11:32:10.167 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.167 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.167 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:10.167 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.101249+0000", 2024-03-19T11:32:10.167 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002632508 2024-03-19T11:32:10.167 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.167 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:10.167 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.167 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.167 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.167 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.21 81/80 [PushOp(4:8401c1f0:::smithi03829856-43:7, version: 55'80, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:7@55'80, size: 2840350, copy_subset: [(0, 2840350)], clone_subset: {}, snapset: 15=[]:{7: [6], b: [b], 10: [d], 12: [12, 11], 15: [15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:7, version: 55'82, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:7@55'82, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 16=[]:{7: [6], b: [b], 10: [d], 16: [16, 15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:b, version: 58'84, data_included: [440350~807767,1309629~240809], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:b@58'84, size: 3394981, copy_subset: [(0, 3394981)], clone_subset: {}, snapset: 15=[]:{7: [6], b: [b], 10: [d], 12: [12, 11], 15: [15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1550438, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:b, version: 58'86, data_included: [504210~665397,1572695~383179], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:b@58'86, size: 2941192, copy_subset: [(0, 2941192)], clone_subset: {}, snapset: 16=[]:{7: [6], b: [b], 10: [d], 16: [16, 15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1955874, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:10, version: 66'99, data_included: [504210~968657,1572695~79919], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:10@66'99, size: 3865577, copy_subset: [(0, 3865577)], clone_subset: {}, snapset: 16=[]:{7: [6], b: [b], 10: [d], 16: [16, 15, 13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1652614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:32:10.167 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:00.089924+0000", 2024-03-19T11:32:10.167 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 70.008754984999996, 2024-03-19T11:32:10.167 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024686583000000002, 2024-03-19T11:32:10.168 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:10.168 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:10.168 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:10.168 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.168 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:10.168 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.089924+0000", 2024-03-19T11:32:10.168 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.168 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.168 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.168 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:10.168 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.089924+0000", 2024-03-19T11:32:10.168 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.168 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.168 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.168 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:10.169 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.089926+0000", 2024-03-19T11:32:10.169 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9589999999999998e-06 2024-03-19T11:32:10.169 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.169 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.169 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:10.169 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093311+0000", 2024-03-19T11:32:10.169 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033854409999999999 2024-03-19T11:32:10.169 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.169 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.169 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:10.169 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093365+0000", 2024-03-19T11:32:10.170 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3795000000000002e-05 2024-03-19T11:32:10.170 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.170 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.170 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:10.170 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093438+0000", 2024-03-19T11:32:10.170 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2591999999999999e-05 2024-03-19T11:32:10.170 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.170 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.170 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.170 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093690+0000", 2024-03-19T11:32:10.170 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025264999999999999 2024-03-19T11:32:10.170 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.171 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.171 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:10.171 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093731+0000", 2024-03-19T11:32:10.171 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0881999999999999e-05 2024-03-19T11:32:10.171 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.171 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.171 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:10.171 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.114610+0000", 2024-03-19T11:32:10.171 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020879264000000002 2024-03-19T11:32:10.171 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.171 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:10.171 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.171 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.171 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.171 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.21 e81,80 4:8401c1f0:::smithi03829856-43:8,46'66,4:866f3dcc:::smithi03829856-31:2,47'73)", 2024-03-19T11:32:10.172 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:00.093514+0000", 2024-03-19T11:32:10.172 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 70.005164385, 2024-03-19T11:32:10.172 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027375713999999999, 2024-03-19T11:32:10.172 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:10.172 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:10.172 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:10.172 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.172 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:10.172 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093514+0000", 2024-03-19T11:32:10.172 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.172 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.172 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.172 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:10.172 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093514+0000", 2024-03-19T11:32:10.172 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.173 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.173 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.173 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:10.173 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093516+0000", 2024-03-19T11:32:10.173 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1179999999999999e-06 2024-03-19T11:32:10.173 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.173 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.173 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:10.173 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093523+0000", 2024-03-19T11:32:10.173 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7500000000000003e-06 2024-03-19T11:32:10.173 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.173 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.173 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:10.173 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093534+0000", 2024-03-19T11:32:10.173 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0831e-05 2024-03-19T11:32:10.174 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.174 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.174 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:10.174 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.093542+0000", 2024-03-19T11:32:10.174 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0139999999999998e-06 2024-03-19T11:32:10.174 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.174 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.174 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.174 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.120682+0000", 2024-03-19T11:32:10.174 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027139838999999999 2024-03-19T11:32:10.174 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.174 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.174 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:10.174 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.120693+0000", 2024-03-19T11:32:10.174 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.136e-05 2024-03-19T11:32:10.175 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.175 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.175 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:10.175 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:00.120890+0000", 2024-03-19T11:32:10.175 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019680199999999999 2024-03-19T11:32:10.175 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.175 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:10.175 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.175 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.175 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.175 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2927 4.24 4:266590a5:::smithi03829856-16:head [append 0~0] snapc 1b=[1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-19T11:32:10.176 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:18.054474+0000", 2024-03-19T11:32:10.176 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 52.044205138000002, 2024-03-19T11:32:10.176 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024856390999999999, 2024-03-19T11:32:10.176 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:10.176 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:10.176 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:10.176 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:10.176 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:10.176 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2927 2024-03-19T11:32:10.176 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.176 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:10.176 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.176 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:10.176 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054474+0000", 2024-03-19T11:32:10.176 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.177 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.177 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.177 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:10.177 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054474+0000", 2024-03-19T11:32:10.177 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.177 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.177 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.177 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:10.177 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054476+0000", 2024-03-19T11:32:10.177 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6469999999999998e-06 2024-03-19T11:32:10.177 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.177 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.177 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:10.177 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054489+0000", 2024-03-19T11:32:10.178 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2612999999999999e-05 2024-03-19T11:32:10.178 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.178 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.178 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:10.178 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054526+0000", 2024-03-19T11:32:10.178 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7481999999999998e-05 2024-03-19T11:32:10.178 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.178 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.178 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:10.178 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054547+0000", 2024-03-19T11:32:10.178 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0820999999999999e-05 2024-03-19T11:32:10.178 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.178 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.178 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.179 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054633+0000", 2024-03-19T11:32:10.179 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5556000000000006e-05 2024-03-19T11:32:10.179 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.179 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.179 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-03-19T11:32:10.179 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054639+0000", 2024-03-19T11:32:10.179 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7300000000000002e-06 2024-03-19T11:32:10.179 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.179 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.179 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.179 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.061592+0000", 2024-03-19T11:32:10.179 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0069539279999999998 2024-03-19T11:32:10.179 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.179 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.179 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:10.180 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.061746+0000", 2024-03-19T11:32:10.180 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015351699999999999 2024-03-19T11:32:10.180 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.180 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.180 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:32:10.180 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.062031+0000", 2024-03-19T11:32:10.180 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00028539700000000001 2024-03-19T11:32:10.180 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.180 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.180 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:32:10.180 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.079164+0000", 2024-03-19T11:32:10.180 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017132865000000001 2024-03-19T11:32:10.180 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.180 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.180 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:10.181 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.079217+0000", 2024-03-19T11:32:10.181 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2373999999999997e-05 2024-03-19T11:32:10.181 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.181 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.181 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:32:10.181 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.079284+0000", 2024-03-19T11:32:10.181 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7433000000000003e-05 2024-03-19T11:32:10.181 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.181 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.181 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:10.181 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.079330+0000", 2024-03-19T11:32:10.181 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6028e-05 2024-03-19T11:32:10.181 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.181 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:10.181 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.182 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.182 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.182 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2928 4.24 4:266590a5:::smithi03829856-16:head [rollback b] snapc 1b=[1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-19T11:32:10.182 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:18.054594+0000", 2024-03-19T11:32:10.182 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 52.044085150999997, 2024-03-19T11:32:10.182 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025781921999999999, 2024-03-19T11:32:10.182 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:10.182 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:10.182 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:10.182 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:10.182 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:10.182 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2928 2024-03-19T11:32:10.182 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.182 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:10.182 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.183 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:10.183 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054594+0000", 2024-03-19T11:32:10.183 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.183 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.183 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.183 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:10.183 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054594+0000", 2024-03-19T11:32:10.183 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.183 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.183 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.183 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:10.183 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054595+0000", 2024-03-19T11:32:10.183 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.018e-06 2024-03-19T11:32:10.183 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.183 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.184 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:10.184 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054601+0000", 2024-03-19T11:32:10.184 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2620000000000004e-06 2024-03-19T11:32:10.184 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.184 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.184 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:10.184 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054618+0000", 2024-03-19T11:32:10.184 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6877000000000001e-05 2024-03-19T11:32:10.184 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.184 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.184 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:10.184 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054632+0000", 2024-03-19T11:32:10.184 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4175000000000001e-05 2024-03-19T11:32:10.184 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.184 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.185 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.185 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054676+0000", 2024-03-19T11:32:10.185 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3659000000000003e-05 2024-03-19T11:32:10.185 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.185 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.185 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:32:10.185 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054679+0000", 2024-03-19T11:32:10.185 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0649999999999999e-06 2024-03-19T11:32:10.185 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.185 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.185 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.185 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.064021+0000", 2024-03-19T11:32:10.185 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093424210000000001 2024-03-19T11:32:10.185 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.185 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.186 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:10.186 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.064158+0000", 2024-03-19T11:32:10.186 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013652099999999999 2024-03-19T11:32:10.186 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.186 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.186 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:32:10.186 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.072763+0000", 2024-03-19T11:32:10.186 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.008605784 2024-03-19T11:32:10.186 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.186 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.186 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:32:10.186 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.079171+0000", 2024-03-19T11:32:10.186 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0064071320000000003 2024-03-19T11:32:10.186 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.186 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.187 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:10.187 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.080257+0000", 2024-03-19T11:32:10.187 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0010866280000000001 2024-03-19T11:32:10.187 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.187 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.187 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:32:10.187 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.080324+0000", 2024-03-19T11:32:10.187 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6976000000000004e-05 2024-03-19T11:32:10.187 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.187 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.187 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:10.187 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.080376+0000", 2024-03-19T11:32:10.187 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1403999999999998e-05 2024-03-19T11:32:10.187 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.187 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:10.188 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.188 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.188 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.188 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2929 4.24 4:266590a5:::smithi03829856-16:head [append 0~0] snapc 1b=[1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-19T11:32:10.188 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:18.054649+0000", 2024-03-19T11:32:10.188 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 52.044029684000002, 2024-03-19T11:32:10.188 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026690519999999999, 2024-03-19T11:32:10.188 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:10.188 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:10.188 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:10.188 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:10.188 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:10.188 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2929 2024-03-19T11:32:10.188 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.188 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:10.189 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.189 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:10.189 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054649+0000", 2024-03-19T11:32:10.189 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.189 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.189 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.189 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:10.189 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054649+0000", 2024-03-19T11:32:10.189 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.189 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.189 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.189 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:10.189 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054650+0000", 2024-03-19T11:32:10.189 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5199999999999995e-07 2024-03-19T11:32:10.189 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.190 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.190 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:10.190 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054655+0000", 2024-03-19T11:32:10.190 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1529999999999996e-06 2024-03-19T11:32:10.190 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.190 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.190 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:10.190 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054670+0000", 2024-03-19T11:32:10.190 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5029e-05 2024-03-19T11:32:10.190 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.190 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.190 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:10.190 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054683+0000", 2024-03-19T11:32:10.190 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2670000000000001e-05 2024-03-19T11:32:10.190 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.191 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.191 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.191 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054756+0000", 2024-03-19T11:32:10.191 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3559000000000005e-05 2024-03-19T11:32:10.191 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.191 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.191 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-19T11:32:10.191 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.054760+0000", 2024-03-19T11:32:10.191 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1659999999999998e-06 2024-03-19T11:32:10.191 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.191 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.191 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.191 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.075329+0000", 2024-03-19T11:32:10.191 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.020569918999999999 2024-03-19T11:32:10.191 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.192 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.192 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:10.192 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.075469+0000", 2024-03-19T11:32:10.192 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013990000000000001 2024-03-19T11:32:10.192 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.192 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.192 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 7", 2024-03-19T11:32:10.192 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.075730+0000", 2024-03-19T11:32:10.192 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00026011300000000002 2024-03-19T11:32:10.192 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.192 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.192 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:32:10.192 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.079918+0000", 2024-03-19T11:32:10.192 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0041886279999999998 2024-03-19T11:32:10.192 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.193 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.193 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:10.193 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.081271+0000", 2024-03-19T11:32:10.193 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0013529449999999999 2024-03-19T11:32:10.193 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.193 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.193 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:32:10.193 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.081308+0000", 2024-03-19T11:32:10.193 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6437999999999999e-05 2024-03-19T11:32:10.193 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.193 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.193 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:10.193 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:18.081340+0000", 2024-03-19T11:32:10.193 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2147999999999998e-05 2024-03-19T11:32:10.193 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.194 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:10.194 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.194 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.194 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:10.194 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:32:10.196 INFO:tasks.rados.rados.0.smithi038.stdout:1841: done (2 left) 2024-03-19T11:32:10.196 INFO:tasks.rados.rados.0.smithi038.stdout:1842: done (1 left) 2024-03-19T11:32:10.196 INFO:tasks.rados.rados.0.smithi038.stdout:1843: read oid 35 snap -1 2024-03-19T11:32:10.196 INFO:tasks.rados.rados.0.smithi038.stdout:1843: expect deleted 2024-03-19T11:32:10.196 INFO:tasks.rados.rados.0.smithi038.stdout:1844: read oid 11 snap -1 2024-03-19T11:32:10.196 INFO:tasks.rados.rados.0.smithi038.stdout:1844: expect deleted 2024-03-19T11:32:10.196 INFO:tasks.rados.rados.0.smithi038.stdout:1845: write oid 13 current snap is 42 2024-03-19T11:32:10.196 INFO:tasks.rados.rados.0.smithi038.stdout:1845: seq_num 347 ranges {632029=547288,1639134=749453,2796249=437859,3653524=178505} 2024-03-19T11:32:10.196 INFO:tasks.rados.rados.0.smithi038.stdout:1845: writing smithi03829856-13 from 632029 to 1179317 tid 1 2024-03-19T11:32:10.196 INFO:tasks.rados.rados.0.smithi038.stdout:1845: writing smithi03829856-13 from 1639134 to 2388587 tid 2 2024-03-19T11:32:10.197 INFO:tasks.rados.rados.0.smithi038.stdout:1845: writing smithi03829856-13 from 2796249 to 3234108 tid 3 2024-03-19T11:32:10.197 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:32:10.181+0000 7fb16a0ef640 -1 osd.4 pg_epoch: 149 pg[4.1d( v 149'92 (0'0,149'92] local-lis/les=148/149 n=3 ec=60/21 lis/c=148/134 les/c/f=149/135/0 sis=148) [4,2] r=0 lpr=148 pi=[134,148)/1 luod=146'90 crt=149'92 lcod 146'90 mlcod 146'90 active+recovering rops=1 mbc={255={(1+2)=1}} trimq=[17~1,1e~1,22~1,25~1,28~1] ps=[1a~1,23~2,26~1,29~1]] bad op order, already applied 4307 > this 4305 2024-03-19T11:32:10.197 INFO:tasks.ceph.osd.4.smithi172.stderr:./src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7fb16a0ef640 time 2024-03-19T11:32:10.185005+0000 2024-03-19T11:32:10.197 INFO:tasks.ceph.osd.4.smithi172.stderr:./src/osd/PrimaryLogPG.cc: 4300: ceph_abort_msg("out of order op") 2024-03-19T11:32:10.197 INFO:tasks.ceph.osd.4.smithi172.stderr: ceph version 19.0.0-2159-g9ef43722 (9ef43722ad94ea688fa7f70daab6dd41fac6ee53) squid (dev) 2024-03-19T11:32:10.197 INFO:tasks.ceph.osd.4.smithi172.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x55e7c20bad09] 2024-03-19T11:32:10.197 INFO:tasks.ceph.osd.4.smithi172.stderr: 2: ceph-osd(+0x39907a) [0x55e7c205b07a] 2024-03-19T11:32:10.197 INFO:tasks.ceph.osd.4.smithi172.stderr: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x30e9) [0x55e7c230f509] 2024-03-19T11:32:10.197 INFO:tasks.ceph.osd.4.smithi172.stderr: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55e7c22472c7] 2024-03-19T11:32:10.197 INFO:tasks.ceph.osd.4.smithi172.stderr: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55e7c248fdb9] 2024-03-19T11:32:10.197 INFO:tasks.ceph.osd.4.smithi172.stderr: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55e7c2251f83] 2024-03-19T11:32:10.197 INFO:tasks.ceph.osd.4.smithi172.stderr: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x55e7c274e913] 2024-03-19T11:32:10.197 INFO:tasks.ceph.osd.4.smithi172.stderr: 8: ceph-osd(+0xa8ce74) [0x55e7c274ee74] 2024-03-19T11:32:10.198 INFO:tasks.ceph.osd.4.smithi172.stderr: 9: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fb18f00fb43] 2024-03-19T11:32:10.198 INFO:tasks.ceph.osd.4.smithi172.stderr: 10: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fb18f0a1a00] 2024-03-19T11:32:10.198 INFO:tasks.ceph.osd.4.smithi172.stderr:*** Caught signal (Aborted) ** 2024-03-19T11:32:10.198 INFO:tasks.ceph.osd.4.smithi172.stderr: in thread 7fb16a0ef640 thread_name:tp_osd_tp 2024-03-19T11:32:10.198 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:32:10.185+0000 7fb16a0ef640 -1 ./src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7fb16a0ef640 time 2024-03-19T11:32:10.185005+0000 2024-03-19T11:32:10.198 INFO:tasks.ceph.osd.4.smithi172.stderr:./src/osd/PrimaryLogPG.cc: 4300: ceph_abort_msg("out of order op") 2024-03-19T11:32:10.198 INFO:tasks.ceph.osd.4.smithi172.stderr: 2024-03-19T11:32:10.198 INFO:tasks.ceph.osd.4.smithi172.stderr: ceph version 19.0.0-2159-g9ef43722 (9ef43722ad94ea688fa7f70daab6dd41fac6ee53) squid (dev) 2024-03-19T11:32:10.198 INFO:tasks.ceph.osd.4.smithi172.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x55e7c20bad09] 2024-03-19T11:32:10.198 INFO:tasks.ceph.osd.4.smithi172.stderr: 2: ceph-osd(+0x39907a) [0x55e7c205b07a] 2024-03-19T11:32:10.198 INFO:tasks.ceph.osd.4.smithi172.stderr: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x30e9) [0x55e7c230f509] 2024-03-19T11:32:10.198 INFO:tasks.ceph.osd.4.smithi172.stderr: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55e7c22472c7] 2024-03-19T11:32:10.198 INFO:tasks.ceph.osd.4.smithi172.stderr: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55e7c248fdb9] 2024-03-19T11:32:10.198 INFO:tasks.ceph.osd.4.smithi172.stderr: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55e7c2251f83] 2024-03-19T11:32:10.198 INFO:tasks.ceph.osd.4.smithi172.stderr: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x55e7c274e913] 2024-03-19T11:32:10.199 INFO:tasks.ceph.osd.4.smithi172.stderr: 8: ceph-osd(+0xa8ce74) [0x55e7c274ee74] 2024-03-19T11:32:10.199 INFO:tasks.ceph.osd.4.smithi172.stderr: 9: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fb18f00fb43] 2024-03-19T11:32:10.199 INFO:tasks.ceph.osd.4.smithi172.stderr: 10: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fb18f0a1a00] 2024-03-19T11:32:10.199 INFO:tasks.ceph.osd.4.smithi172.stderr: 2024-03-19T11:32:10.199 INFO:tasks.ceph.osd.4.smithi172.stderr: ceph version 19.0.0-2159-g9ef43722 (9ef43722ad94ea688fa7f70daab6dd41fac6ee53) squid (dev) 2024-03-19T11:32:10.199 INFO:tasks.ceph.osd.4.smithi172.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fb18efbd520] 2024-03-19T11:32:10.199 INFO:tasks.ceph.osd.4.smithi172.stderr: 2: pthread_kill() 2024-03-19T11:32:10.199 INFO:tasks.ceph.osd.4.smithi172.stderr: 3: raise() 2024-03-19T11:32:10.199 INFO:tasks.ceph.osd.4.smithi172.stderr: 4: abort() 2024-03-19T11:32:10.199 INFO:tasks.ceph.osd.4.smithi172.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x17b) [0x55e7c20badbe] 2024-03-19T11:32:10.200 INFO:tasks.ceph.osd.4.smithi172.stderr: 6: ceph-osd(+0x39907a) [0x55e7c205b07a] 2024-03-19T11:32:10.200 INFO:tasks.ceph.osd.4.smithi172.stderr: 7: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x30e9) [0x55e7c230f509] 2024-03-19T11:32:10.200 INFO:tasks.ceph.osd.4.smithi172.stderr: 8: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55e7c22472c7] 2024-03-19T11:32:10.200 INFO:tasks.ceph.osd.4.smithi172.stderr: 9: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55e7c248fdb9] 2024-03-19T11:32:10.200 INFO:tasks.ceph.osd.4.smithi172.stderr: 10: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55e7c2251f83] 2024-03-19T11:32:10.200 INFO:tasks.ceph.osd.4.smithi172.stderr: 11: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x55e7c274e913] 2024-03-19T11:32:10.200 INFO:tasks.ceph.osd.4.smithi172.stderr: 12: ceph-osd(+0xa8ce74) [0x55e7c274ee74] 2024-03-19T11:32:10.200 INFO:tasks.ceph.osd.4.smithi172.stderr: 13: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fb18f00fb43] 2024-03-19T11:32:10.200 INFO:tasks.ceph.osd.4.smithi172.stderr: 14: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fb18f0a1a00] 2024-03-19T11:32:10.200 INFO:tasks.ceph.osd.4.smithi172.stderr:2024-03-19T11:32:10.189+0000 7fb16a0ef640 -1 *** Caught signal (Aborted) ** 2024-03-19T11:32:10.200 INFO:tasks.ceph.osd.4.smithi172.stderr: in thread 7fb16a0ef640 thread_name:tp_osd_tp 2024-03-19T11:32:10.200 INFO:tasks.ceph.osd.4.smithi172.stderr: 2024-03-19T11:32:10.200 INFO:tasks.ceph.osd.4.smithi172.stderr: ceph version 19.0.0-2159-g9ef43722 (9ef43722ad94ea688fa7f70daab6dd41fac6ee53) squid (dev) 2024-03-19T11:32:10.200 INFO:tasks.ceph.osd.4.smithi172.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fb18efbd520] 2024-03-19T11:32:10.201 INFO:tasks.ceph.osd.4.smithi172.stderr: 2: pthread_kill() 2024-03-19T11:32:10.201 INFO:tasks.ceph.osd.4.smithi172.stderr: 3: raise() 2024-03-19T11:32:10.201 INFO:tasks.ceph.osd.4.smithi172.stderr: 4: abort() 2024-03-19T11:32:10.201 INFO:tasks.ceph.osd.4.smithi172.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x17b) [0x55e7c20badbe] 2024-03-19T11:32:10.201 INFO:tasks.ceph.osd.4.smithi172.stderr: 6: ceph-osd(+0x39907a) [0x55e7c205b07a] 2024-03-19T11:32:10.201 INFO:tasks.ceph.osd.4.smithi172.stderr: 7: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x30e9) [0x55e7c230f509] 2024-03-19T11:32:10.201 INFO:tasks.ceph.osd.4.smithi172.stderr: 8: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55e7c22472c7] 2024-03-19T11:32:10.201 INFO:tasks.ceph.osd.4.smithi172.stderr: 9: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55e7c248fdb9] 2024-03-19T11:32:10.201 INFO:tasks.ceph.osd.4.smithi172.stderr: 10: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55e7c2251f83] 2024-03-19T11:32:10.201 INFO:tasks.ceph.osd.4.smithi172.stderr: 11: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x55e7c274e913] 2024-03-19T11:32:10.201 INFO:tasks.ceph.osd.4.smithi172.stderr: 12: ceph-osd(+0xa8ce74) [0x55e7c274ee74] 2024-03-19T11:32:10.201 INFO:tasks.ceph.osd.4.smithi172.stderr: 13: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fb18f00fb43] 2024-03-19T11:32:10.201 INFO:tasks.ceph.osd.4.smithi172.stderr: 14: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fb18f0a1a00] 2024-03-19T11:32:10.201 INFO:tasks.ceph.osd.4.smithi172.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-19T11:32:10.202 INFO:tasks.ceph.osd.4.smithi172.stderr: 2024-03-19T11:32:10.203 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-19T11:32:10.204 INFO:tasks.rados.rados.0.smithi038.stdout:1845: writing smithi03829856-13 from 3653524 to 3832029 tid 4 2024-03-19T11:32:10.204 INFO:tasks.rados.rados.0.smithi038.stdout:1846: read oid 46 snap -1 2024-03-19T11:32:10.205 INFO:tasks.rados.rados.0.smithi038.stdout:1846: expect (ObjNum 343 snap 40 seq_num 343) 2024-03-19T11:32:10.205 INFO:tasks.rados.rados.0.smithi038.stdout:1847: read oid 12 snap -1 2024-03-19T11:32:10.205 INFO:tasks.rados.rados.0.smithi038.stdout:1847: expect (ObjNum 325 snap 40 seq_num 325) 2024-03-19T11:32:10.205 INFO:tasks.rados.rados.0.smithi038.stdout:1848: tier_promote oid 45 2024-03-19T11:32:10.205 INFO:tasks.rados.rados.0.smithi038.stdout:1843: done (6 left) 2024-03-19T11:32:10.205 INFO:tasks.rados.rados.0.smithi038.stdout:1844: done (5 left) 2024-03-19T11:32:10.205 INFO:tasks.rados.rados.0.smithi038.stdout:1849: snap_create 2024-03-19T11:32:10.205 INFO:tasks.rados.rados.0.smithi038.stdout:1846: expect (ObjNum 343 snap 40 seq_num 343) 2024-03-19T11:32:10.205 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:32:10.203+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:32:10.212 INFO:tasks.ceph.osd.4.smithi172.stderr: -2> 2024-03-19T11:32:10.181+0000 7fb16a0ef640 -1 osd.4 pg_epoch: 149 pg[4.1d( v 149'92 (0'0,149'92] local-lis/les=148/149 n=3 ec=60/21 lis/c=148/134 les/c/f=149/135/0 sis=148) [4,2] r=0 lpr=148 pi=[134,148)/1 luod=146'90 crt=149'92 lcod 146'90 mlcod 146'90 active+recovering rops=1 mbc={255={(1+2)=1}} trimq=[17~1,1e~1,22~1,25~1,28~1] ps=[1a~1,23~2,26~1,29~1]] bad op order, already applied 4307 > this 4305 2024-03-19T11:32:10.212 INFO:tasks.ceph.osd.4.smithi172.stderr: -1> 2024-03-19T11:32:10.185+0000 7fb16a0ef640 -1 ./src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7fb16a0ef640 time 2024-03-19T11:32:10.185005+0000 2024-03-19T11:32:10.212 INFO:tasks.ceph.osd.4.smithi172.stderr:./src/osd/PrimaryLogPG.cc: 4300: ceph_abort_msg("out of order op") 2024-03-19T11:32:10.212 INFO:tasks.ceph.osd.4.smithi172.stderr: 2024-03-19T11:32:10.212 INFO:tasks.ceph.osd.4.smithi172.stderr: ceph version 19.0.0-2159-g9ef43722 (9ef43722ad94ea688fa7f70daab6dd41fac6ee53) squid (dev) 2024-03-19T11:32:10.212 INFO:tasks.ceph.osd.4.smithi172.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x55e7c20bad09] 2024-03-19T11:32:10.212 INFO:tasks.ceph.osd.4.smithi172.stderr: 2: ceph-osd(+0x39907a) [0x55e7c205b07a] 2024-03-19T11:32:10.212 INFO:tasks.ceph.osd.4.smithi172.stderr: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x30e9) [0x55e7c230f509] 2024-03-19T11:32:10.212 INFO:tasks.ceph.osd.4.smithi172.stderr: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55e7c22472c7] 2024-03-19T11:32:10.212 INFO:tasks.ceph.osd.4.smithi172.stderr: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55e7c248fdb9] 2024-03-19T11:32:10.212 INFO:tasks.ceph.osd.4.smithi172.stderr: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55e7c2251f83] 2024-03-19T11:32:10.212 INFO:tasks.ceph.osd.4.smithi172.stderr: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x55e7c274e913] 2024-03-19T11:32:10.213 INFO:tasks.ceph.osd.4.smithi172.stderr: 8: ceph-osd(+0xa8ce74) [0x55e7c274ee74] 2024-03-19T11:32:10.213 INFO:tasks.ceph.osd.4.smithi172.stderr: 9: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fb18f00fb43] 2024-03-19T11:32:10.213 INFO:tasks.ceph.osd.4.smithi172.stderr: 10: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fb18f0a1a00] 2024-03-19T11:32:10.213 INFO:tasks.ceph.osd.4.smithi172.stderr: 2024-03-19T11:32:10.213 INFO:tasks.ceph.osd.4.smithi172.stderr: 0> 2024-03-19T11:32:10.189+0000 7fb16a0ef640 -1 *** Caught signal (Aborted) ** 2024-03-19T11:32:10.213 INFO:tasks.ceph.osd.4.smithi172.stderr: in thread 7fb16a0ef640 thread_name:tp_osd_tp 2024-03-19T11:32:10.213 INFO:tasks.ceph.osd.4.smithi172.stderr: 2024-03-19T11:32:10.213 INFO:tasks.ceph.osd.4.smithi172.stderr: ceph version 19.0.0-2159-g9ef43722 (9ef43722ad94ea688fa7f70daab6dd41fac6ee53) squid (dev) 2024-03-19T11:32:10.213 INFO:tasks.ceph.osd.4.smithi172.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fb18efbd520] 2024-03-19T11:32:10.213 INFO:tasks.ceph.osd.4.smithi172.stderr: 2: pthread_kill() 2024-03-19T11:32:10.213 INFO:tasks.ceph.osd.4.smithi172.stderr: 3: raise() 2024-03-19T11:32:10.213 INFO:tasks.ceph.osd.4.smithi172.stderr: 4: abort() 2024-03-19T11:32:10.213 INFO:tasks.ceph.osd.4.smithi172.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x17b) [0x55e7c20badbe] 2024-03-19T11:32:10.213 INFO:tasks.ceph.osd.4.smithi172.stderr: 6: ceph-osd(+0x39907a) [0x55e7c205b07a] 2024-03-19T11:32:10.213 INFO:tasks.ceph.osd.4.smithi172.stderr: 7: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x30e9) [0x55e7c230f509] 2024-03-19T11:32:10.214 INFO:tasks.ceph.osd.4.smithi172.stderr: 8: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55e7c22472c7] 2024-03-19T11:32:10.214 INFO:tasks.ceph.osd.4.smithi172.stderr: 9: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55e7c248fdb9] 2024-03-19T11:32:10.214 INFO:tasks.ceph.osd.4.smithi172.stderr: 10: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55e7c2251f83] 2024-03-19T11:32:10.214 INFO:tasks.ceph.osd.4.smithi172.stderr: 11: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x55e7c274e913] 2024-03-19T11:32:10.214 INFO:tasks.ceph.osd.4.smithi172.stderr: 12: ceph-osd(+0xa8ce74) [0x55e7c274ee74] 2024-03-19T11:32:10.214 INFO:tasks.ceph.osd.4.smithi172.stderr: 13: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fb18f00fb43] 2024-03-19T11:32:10.214 INFO:tasks.ceph.osd.4.smithi172.stderr: 14: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fb18f0a1a00] 2024-03-19T11:32:10.214 INFO:tasks.ceph.osd.4.smithi172.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-19T11:32:10.214 INFO:tasks.ceph.osd.4.smithi172.stderr: 2024-03-19T11:32:10.228 INFO:tasks.ceph.osd.4.smithi172.stderr: -6> 2024-03-19T11:32:10.181+0000 7fb16a0ef640 -1 osd.4 pg_epoch: 149 pg[4.1d( v 149'92 (0'0,149'92] local-lis/les=148/149 n=3 ec=60/21 lis/c=148/134 les/c/f=149/135/0 sis=148) [4,2] r=0 lpr=148 pi=[134,148)/1 luod=146'90 crt=149'92 lcod 146'90 mlcod 146'90 active+recovering rops=1 mbc={255={(1+2)=1}} trimq=[17~1,1e~1,22~1,25~1,28~1] ps=[1a~1,23~2,26~1,29~1]] bad op order, already applied 4307 > this 4305 2024-03-19T11:32:10.229 INFO:tasks.ceph.osd.4.smithi172.stderr: -5> 2024-03-19T11:32:10.185+0000 7fb16a0ef640 -1 ./src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7fb16a0ef640 time 2024-03-19T11:32:10.185005+0000 2024-03-19T11:32:10.229 INFO:tasks.ceph.osd.4.smithi172.stderr:./src/osd/PrimaryLogPG.cc: 4300: ceph_abort_msg("out of order op") 2024-03-19T11:32:10.229 INFO:tasks.ceph.osd.4.smithi172.stderr: 2024-03-19T11:32:10.229 INFO:tasks.ceph.osd.4.smithi172.stderr: ceph version 19.0.0-2159-g9ef43722 (9ef43722ad94ea688fa7f70daab6dd41fac6ee53) squid (dev) 2024-03-19T11:32:10.229 INFO:tasks.ceph.osd.4.smithi172.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x55e7c20bad09] 2024-03-19T11:32:10.229 INFO:tasks.ceph.osd.4.smithi172.stderr: 2: ceph-osd(+0x39907a) [0x55e7c205b07a] 2024-03-19T11:32:10.229 INFO:tasks.ceph.osd.4.smithi172.stderr: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x30e9) [0x55e7c230f509] 2024-03-19T11:32:10.229 INFO:tasks.ceph.osd.4.smithi172.stderr: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55e7c22472c7] 2024-03-19T11:32:10.229 INFO:tasks.ceph.osd.4.smithi172.stderr: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55e7c248fdb9] 2024-03-19T11:32:10.229 INFO:tasks.ceph.osd.4.smithi172.stderr: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55e7c2251f83] 2024-03-19T11:32:10.229 INFO:tasks.ceph.osd.4.smithi172.stderr: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x55e7c274e913] 2024-03-19T11:32:10.229 INFO:tasks.ceph.osd.4.smithi172.stderr: 8: ceph-osd(+0xa8ce74) [0x55e7c274ee74] 2024-03-19T11:32:10.229 INFO:tasks.ceph.osd.4.smithi172.stderr: 9: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fb18f00fb43] 2024-03-19T11:32:10.230 INFO:tasks.ceph.osd.4.smithi172.stderr: 10: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fb18f0a1a00] 2024-03-19T11:32:10.230 INFO:tasks.ceph.osd.4.smithi172.stderr: 2024-03-19T11:32:10.230 INFO:tasks.ceph.osd.4.smithi172.stderr: -4> 2024-03-19T11:32:10.189+0000 7fb16a0ef640 -1 *** Caught signal (Aborted) ** 2024-03-19T11:32:10.230 INFO:tasks.ceph.osd.4.smithi172.stderr: in thread 7fb16a0ef640 thread_name:tp_osd_tp 2024-03-19T11:32:10.230 INFO:tasks.ceph.osd.4.smithi172.stderr: 2024-03-19T11:32:10.230 INFO:tasks.ceph.osd.4.smithi172.stderr: ceph version 19.0.0-2159-g9ef43722 (9ef43722ad94ea688fa7f70daab6dd41fac6ee53) squid (dev) 2024-03-19T11:32:10.230 INFO:tasks.ceph.osd.4.smithi172.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fb18efbd520] 2024-03-19T11:32:10.230 INFO:tasks.ceph.osd.4.smithi172.stderr: 2: pthread_kill() 2024-03-19T11:32:10.230 INFO:tasks.ceph.osd.4.smithi172.stderr: 3: raise() 2024-03-19T11:32:10.230 INFO:tasks.ceph.osd.4.smithi172.stderr: 4: abort() 2024-03-19T11:32:10.230 INFO:tasks.ceph.osd.4.smithi172.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x17b) [0x55e7c20badbe] 2024-03-19T11:32:10.230 INFO:tasks.ceph.osd.4.smithi172.stderr: 6: ceph-osd(+0x39907a) [0x55e7c205b07a] 2024-03-19T11:32:10.230 INFO:tasks.ceph.osd.4.smithi172.stderr: 7: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x30e9) [0x55e7c230f509] 2024-03-19T11:32:10.230 INFO:tasks.ceph.osd.4.smithi172.stderr: 8: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x55e7c22472c7] 2024-03-19T11:32:10.231 INFO:tasks.ceph.osd.4.smithi172.stderr: 9: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55e7c248fdb9] 2024-03-19T11:32:10.231 INFO:tasks.ceph.osd.4.smithi172.stderr: 10: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55e7c2251f83] 2024-03-19T11:32:10.231 INFO:tasks.ceph.osd.4.smithi172.stderr: 11: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x55e7c274e913] 2024-03-19T11:32:10.231 INFO:tasks.ceph.osd.4.smithi172.stderr: 12: ceph-osd(+0xa8ce74) [0x55e7c274ee74] 2024-03-19T11:32:10.231 INFO:tasks.ceph.osd.4.smithi172.stderr: 13: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fb18f00fb43] 2024-03-19T11:32:10.231 INFO:tasks.ceph.osd.4.smithi172.stderr: 14: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fb18f0a1a00] 2024-03-19T11:32:10.231 INFO:tasks.ceph.osd.4.smithi172.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-19T11:32:10.231 INFO:tasks.ceph.osd.4.smithi172.stderr: 2024-03-19T11:32:10.296 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:32:10.303+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:32:10.299 INFO:tasks.rados.rados.0.smithi038.stdout:1848: got (0) Success 2024-03-19T11:32:10.299 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 45 v 102 (ObjNum 327 snap 40 seq_num 327) dirty exists 2024-03-19T11:32:10.299 INFO:tasks.rados.rados.0.smithi038.stdout:1845: finishing write tid 1 to smithi03829856-13 2024-03-19T11:32:10.299 INFO:tasks.rados.rados.0.smithi038.stdout:1847: expect (ObjNum 325 snap 40 seq_num 325) 2024-03-19T11:32:10.366 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:32:10.366 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:32:10.366 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-19T11:32:10.366 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:32:10.381 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-19T11:32:10.425 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:32:10.435+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:32:10.498 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:32:10.505+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:32:10.534 INFO:tasks.rados.rados.0.smithi038.stdout:1845: finishing write tid 2 to smithi03829856-13 2024-03-19T11:32:10.534 INFO:tasks.rados.rados.0.smithi038.stdout:1845: finishing write tid 3 to smithi03829856-13 2024-03-19T11:32:10.534 INFO:tasks.rados.rados.0.smithi038.stdout:1845: finishing write tid 4 to smithi03829856-13 2024-03-19T11:32:10.534 INFO:tasks.rados.rados.0.smithi038.stdout:1845: finishing write tid 5 to smithi03829856-13 2024-03-19T11:32:10.534 INFO:tasks.rados.rados.0.smithi038.stdout:1845: finishing write tid 6 to smithi03829856-13 2024-03-19T11:32:10.535 INFO:tasks.rados.rados.0.smithi038.stdout:1845: oid 13 updating version 0 to 177 2024-03-19T11:32:10.535 INFO:tasks.rados.rados.0.smithi038.stdout:1845: oid 13 version 177 is already newer than 173 2024-03-19T11:32:10.535 INFO:tasks.rados.rados.0.smithi038.stdout:1845: oid 13 version 177 is already newer than 176 2024-03-19T11:32:10.535 INFO:tasks.rados.rados.0.smithi038.stdout:1845: oid 13 version 177 is already newer than 174 2024-03-19T11:32:10.535 INFO:tasks.rados.rados.0.smithi038.stdout:1845: oid 13 version 177 is already newer than 175 2024-03-19T11:32:10.535 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 13 v 177 (ObjNum 347 snap 42 seq_num 347) dirty exists 2024-03-19T11:32:10.535 INFO:tasks.rados.rados.0.smithi038.stdout:1845: left oid 13 (ObjNum 347 snap 42 seq_num 347) 2024-03-19T11:32:10.548 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:32:10.548 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-19T11:32:10.548 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-19T11:32:10.548 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-19T11:32:10.548 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:32:10.563 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-19T11:32:10.677 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-19T11:32:10.692 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-19T11:32:10.692 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-19T11:32:10.692 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-19T11:32:10.692 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-19T11:32:10.693 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.693 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.16 49/48 [PushOp(4:6868af91:::smithi03829856-10:2, version: 29'22, data_included: [551263~793594,1893632~254982], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:2@29'22, size: 2151263, copy_subset: [(0, 2151263)], clone_subset: {}, snapset: b=[]:{2: [2], 7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2148614, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:6d9093b1:::smithi03829856-38:4, version: 33'29, data_included: [621859~449625,1865716~598951], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6d9093b1:::smithi03829856-38:4@33'29, size: 2621859, copy_subset: [(0, 2621859)], clone_subset: {}, snapset: 4=[]:{4: [2]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2464667, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:6d9093b1:::smithi03829856-38:head, version: 33'33, data_included: [621859~568773,1729599~479803], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:6d9093b1:::smithi03829856-38:head@33'33, size: 3142876, copy_subset: [(0, 3142876)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2209402, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:6be89c06:::smithi03829856-24:head, version: 36'39, data_included: [741376~589824,1753088~458752], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:6be89c06:::smithi03829856-24:head@36'39, size: 2386561, copy_subset: [(0, 2386561)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2211840, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:6868af91:::smithi03829856-10:7, version: 38'41, data_included: [524385~820472,1602039~228104], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:7@38'41, size: 2924385, copy_subset: [(0, 2924385)], clone_subset: {}, snapset: b=[]:{2: [2], 7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1830143, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:6868af91:::smithi03829856-10:b, version: 47'48, data_included: [524385~820472,1602039~228104], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:b@47'48, size: 2924385, copy_subset: [(0, 2924385)], clone_subset: {}, snapset: b=[]:{2: [2], 7: [6, 5], b: [b, a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1830143, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:6868af91:::smithi03829856-10:head, version: 47'52, data_included: [524385~820472,1602039~228104], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:6868af91:::smithi03829856-10:head@47'52, size: 3747928, copy_subset: [(0, 3747928)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1830143, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:32:10.693 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:26.954289+0000", 2024-03-19T11:32:10.693 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 103.74749011500001, 2024-03-19T11:32:10.693 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.029260781999999999, 2024-03-19T11:32:10.693 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:10.693 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:10.693 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:10.693 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.693 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:10.693 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954289+0000", 2024-03-19T11:32:10.693 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.693 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.693 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.694 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:10.694 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954289+0000", 2024-03-19T11:32:10.694 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.694 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.694 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.694 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:10.694 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.954291+0000", 2024-03-19T11:32:10.694 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8050000000000001e-06 2024-03-19T11:32:10.694 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.694 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.694 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:10.694 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.958775+0000", 2024-03-19T11:32:10.694 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0044845859999999996 2024-03-19T11:32:10.694 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.695 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.695 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:10.695 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.958851+0000", 2024-03-19T11:32:10.695 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6050999999999994e-05 2024-03-19T11:32:10.695 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.695 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.695 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:10.695 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.958931+0000", 2024-03-19T11:32:10.695 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9099999999999998e-05 2024-03-19T11:32:10.695 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.696 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.696 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.696 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.959206+0000", 2024-03-19T11:32:10.696 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00027495999999999998 2024-03-19T11:32:10.696 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.696 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.696 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:10.696 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.959246+0000", 2024-03-19T11:32:10.696 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0326e-05 2024-03-19T11:32:10.696 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.697 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.697 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:10.697 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:26.983550+0000", 2024-03-19T11:32:10.697 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024303953999999999 2024-03-19T11:32:10.697 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.697 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:10.697 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.697 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.697 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.697 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1630 4.d 4:ba44105a:::smithi03829856-26:head [write 467261~717275] snapc 10=[10,f,e,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-03-19T11:32:10.697 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:35.331434+0000", 2024-03-19T11:32:10.697 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 95.370345310000005, 2024-03-19T11:32:10.697 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012902375000000001, 2024-03-19T11:32:10.697 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:10.698 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:10.698 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:10.698 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:10.698 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:10.698 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1630 2024-03-19T11:32:10.698 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.698 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:10.698 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.698 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:10.698 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.331434+0000", 2024-03-19T11:32:10.698 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.698 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.698 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.698 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:10.699 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.331434+0000", 2024-03-19T11:32:10.699 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.699 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.699 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.699 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:10.699 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.331437+0000", 2024-03-19T11:32:10.700 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5599999999999998e-06 2024-03-19T11:32:10.700 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.700 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.700 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:10.700 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.331938+0000", 2024-03-19T11:32:10.700 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00050090099999999999 2024-03-19T11:32:10.701 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.701 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.701 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:10.701 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.331976+0000", 2024-03-19T11:32:10.701 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8090999999999998e-05 2024-03-19T11:32:10.701 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.701 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.701 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:10.701 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.331994+0000", 2024-03-19T11:32:10.702 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7566000000000001e-05 2024-03-19T11:32:10.702 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.702 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.702 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.702 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.332086+0000", 2024-03-19T11:32:10.702 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2452000000000002e-05 2024-03-19T11:32:10.702 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.702 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.702 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:10.702 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.332205+0000", 2024-03-19T11:32:10.702 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000118966 2024-03-19T11:32:10.702 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.703 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.703 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1", 2024-03-19T11:32:10.703 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.332410+0000", 2024-03-19T11:32:10.703 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020430099999999999 2024-03-19T11:32:10.703 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.703 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.703 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:32:10.703 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.339226+0000", 2024-03-19T11:32:10.703 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0068165980000000001 2024-03-19T11:32:10.704 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.704 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.704 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:10.704 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.344191+0000", 2024-03-19T11:32:10.704 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.004965143 2024-03-19T11:32:10.704 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.704 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.704 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:32:10.704 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.344272+0000", 2024-03-19T11:32:10.704 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0265000000000005e-05 2024-03-19T11:32:10.705 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.705 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.705 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:10.705 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:35.344336+0000", 2024-03-19T11:32:10.705 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4531999999999998e-05 2024-03-19T11:32:10.705 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.705 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:10.705 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.705 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.705 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.706 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1889 4.4 4:266590a5:::smithi03829856-16:head [rollback d] snapc 10=[10,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-03-19T11:32:10.706 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:41.697617+0000", 2024-03-19T11:32:10.706 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 89.004162316000006, 2024-03-19T11:32:10.706 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018164059999999999, 2024-03-19T11:32:10.706 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:10.706 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:10.706 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:10.706 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:10.706 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:10.706 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1889 2024-03-19T11:32:10.707 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.707 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:10.707 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.707 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:10.707 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697617+0000", 2024-03-19T11:32:10.707 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.707 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.707 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.707 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:10.708 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697617+0000", 2024-03-19T11:32:10.708 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.708 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.708 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.708 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:10.708 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697619+0000", 2024-03-19T11:32:10.708 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0059999999999999e-06 2024-03-19T11:32:10.708 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.708 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.708 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:10.708 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697631+0000", 2024-03-19T11:32:10.708 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2033e-05 2024-03-19T11:32:10.709 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.709 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.709 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:10.709 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697659+0000", 2024-03-19T11:32:10.709 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8291999999999999e-05 2024-03-19T11:32:10.709 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.709 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.709 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:10.709 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697682+0000", 2024-03-19T11:32:10.709 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3281999999999998e-05 2024-03-19T11:32:10.709 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.709 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.709 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.710 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.701122+0000", 2024-03-19T11:32:10.710 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034396679999999999 2024-03-19T11:32:10.710 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.710 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.710 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:10.710 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.701383+0000", 2024-03-19T11:32:10.710 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00026086399999999997 2024-03-19T11:32:10.710 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.710 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.710 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3", 2024-03-19T11:32:10.710 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.709693+0000", 2024-03-19T11:32:10.710 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0083096139999999999 2024-03-19T11:32:10.710 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.711 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.711 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:10.711 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.715437+0000", 2024-03-19T11:32:10.711 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057442379999999996 2024-03-19T11:32:10.711 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.711 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.711 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:32:10.711 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.715563+0000", 2024-03-19T11:32:10.711 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000125778 2024-03-19T11:32:10.711 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.711 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.711 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:32:10.711 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.715677+0000", 2024-03-19T11:32:10.712 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000114728 2024-03-19T11:32:10.712 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.712 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.713 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:10.713 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.715781+0000", 2024-03-19T11:32:10.713 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000103557 2024-03-19T11:32:10.713 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.713 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:10.713 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.714 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.714 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.714 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1890 4.4 4:266590a5:::smithi03829856-16:head [append 0~0] snapc 10=[10,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-03-19T11:32:10.714 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:41.697709+0000", 2024-03-19T11:32:10.714 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 89.004069754, 2024-03-19T11:32:10.714 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019825171999999999, 2024-03-19T11:32:10.714 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:10.715 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:10.715 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:10.715 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:10.715 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:10.715 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1890 2024-03-19T11:32:10.715 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.715 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:10.715 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.715 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:10.716 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697709+0000", 2024-03-19T11:32:10.716 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.716 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.716 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.716 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:10.716 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697709+0000", 2024-03-19T11:32:10.716 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.716 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.716 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.717 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:10.717 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697711+0000", 2024-03-19T11:32:10.717 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.296e-06 2024-03-19T11:32:10.717 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.717 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.717 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:10.717 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697718+0000", 2024-03-19T11:32:10.717 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.464e-06 2024-03-19T11:32:10.717 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.718 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.718 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:10.718 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697739+0000", 2024-03-19T11:32:10.718 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1005e-05 2024-03-19T11:32:10.718 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.718 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.718 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:10.718 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.697754+0000", 2024-03-19T11:32:10.719 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4976e-05 2024-03-19T11:32:10.719 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.719 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.719 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.719 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.702341+0000", 2024-03-19T11:32:10.719 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0045870980000000004 2024-03-19T11:32:10.719 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.719 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.719 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for blocked object", 2024-03-19T11:32:10.719 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.702482+0000", 2024-03-19T11:32:10.720 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014048000000000001 2024-03-19T11:32:10.720 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.720 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.720 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.720 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.714359+0000", 2024-03-19T11:32:10.720 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011877522999999999 2024-03-19T11:32:10.720 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.720 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.720 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:10.720 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.714550+0000", 2024-03-19T11:32:10.721 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019117 2024-03-19T11:32:10.721 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.721 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.721 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3", 2024-03-19T11:32:10.721 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.714911+0000", 2024-03-19T11:32:10.721 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00036070900000000002 2024-03-19T11:32:10.721 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.721 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.721 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:32:10.722 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.716211+0000", 2024-03-19T11:32:10.722 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001300329 2024-03-19T11:32:10.722 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.722 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.722 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:10.722 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.717326+0000", 2024-03-19T11:32:10.722 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001114056 2024-03-19T11:32:10.723 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.723 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.723 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:32:10.723 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.717423+0000", 2024-03-19T11:32:10.723 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7048999999999999e-05 2024-03-19T11:32:10.723 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.723 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.723 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:10.724 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:41.717535+0000", 2024-03-19T11:32:10.724 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011201699999999999 2024-03-19T11:32:10.724 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.724 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:10.724 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.724 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.724 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.724 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:1972 4.4 4:214026b3:::smithi03829856-47:head [tier-promote] snapc 12=[12,11,d,c,b,6] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:32:10.724 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:45.312422+0000", 2024-03-19T11:32:10.724 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 85.389357079999996, 2024-03-19T11:32:10.724 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013955089, 2024-03-19T11:32:10.724 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:10.724 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:10.725 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:10.725 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:10.725 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:10.725 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 1972 2024-03-19T11:32:10.725 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.725 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:10.725 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.725 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:10.725 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.312422+0000", 2024-03-19T11:32:10.725 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.726 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.726 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.726 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:10.726 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.312422+0000", 2024-03-19T11:32:10.726 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.726 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.726 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.726 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:10.726 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.312424+0000", 2024-03-19T11:32:10.726 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9810000000000002e-06 2024-03-19T11:32:10.726 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.726 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.726 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:10.727 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.312433+0000", 2024-03-19T11:32:10.727 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4549999999999995e-06 2024-03-19T11:32:10.727 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.727 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.727 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:10.727 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.312452+0000", 2024-03-19T11:32:10.727 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9380000000000001e-05 2024-03-19T11:32:10.727 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.727 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.727 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:10.727 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.312460+0000", 2024-03-19T11:32:10.727 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2560000000000002e-06 2024-03-19T11:32:10.728 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.728 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.728 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.728 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.312555+0000", 2024-03-19T11:32:10.728 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.5208000000000001e-05 2024-03-19T11:32:10.728 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.728 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.728 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:10.728 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.312704+0000", 2024-03-19T11:32:10.728 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014843899999999999 2024-03-19T11:32:10.728 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.728 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.728 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:10.729 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.326377+0000", 2024-03-19T11:32:10.729 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013673370000000001 2024-03-19T11:32:10.729 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.729 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:10.729 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.729 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.729 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.729 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2005 4.4 4:214026b3:::smithi03829856-47:head [read 0~2773122,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:32:10.729 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:45.804325+0000", 2024-03-19T11:32:10.729 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 84.897454620000005, 2024-03-19T11:32:10.729 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010899282, 2024-03-19T11:32:10.730 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:10.730 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:10.730 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:10.730 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:10.730 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:10.730 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2005 2024-03-19T11:32:10.730 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.730 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:10.730 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.730 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:10.730 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804325+0000", 2024-03-19T11:32:10.730 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.730 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.730 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.731 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:10.731 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804325+0000", 2024-03-19T11:32:10.731 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.731 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.731 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.731 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:10.731 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804326+0000", 2024-03-19T11:32:10.731 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.666e-06 2024-03-19T11:32:10.731 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.731 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.731 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:10.731 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804338+0000", 2024-03-19T11:32:10.732 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2096000000000001e-05 2024-03-19T11:32:10.732 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.732 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.732 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:10.732 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804363+0000", 2024-03-19T11:32:10.732 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4762000000000001e-05 2024-03-19T11:32:10.732 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.732 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.732 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:10.732 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804380+0000", 2024-03-19T11:32:10.732 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.713e-05 2024-03-19T11:32:10.732 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.732 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.732 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.733 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.812236+0000", 2024-03-19T11:32:10.733 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0078554690000000003 2024-03-19T11:32:10.733 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.733 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.733 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:10.733 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.812541+0000", 2024-03-19T11:32:10.733 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030564299999999997 2024-03-19T11:32:10.733 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.733 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.733 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:10.733 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.815224+0000", 2024-03-19T11:32:10.733 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0026825159999999998 2024-03-19T11:32:10.733 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.733 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:10.733 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.734 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.734 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.734 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2006 4.4 4:214026b3:::smithi03829856-47:head [read 0~2773122,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:32:10.734 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:45.804421+0000", 2024-03-19T11:32:10.734 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 84.897357881000005, 2024-03-19T11:32:10.734 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013366463, 2024-03-19T11:32:10.734 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:10.734 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:10.734 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:10.734 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:10.734 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:10.734 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2006 2024-03-19T11:32:10.734 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.735 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:10.735 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.735 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:10.735 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804421+0000", 2024-03-19T11:32:10.735 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.735 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.735 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.735 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:10.735 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804421+0000", 2024-03-19T11:32:10.735 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.735 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.735 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.735 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:10.735 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804423+0000", 2024-03-19T11:32:10.735 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5600000000000001e-06 2024-03-19T11:32:10.736 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.736 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.736 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:10.736 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804433+0000", 2024-03-19T11:32:10.736 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8719999999999997e-06 2024-03-19T11:32:10.736 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.736 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.736 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:10.736 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804454+0000", 2024-03-19T11:32:10.736 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1759e-05 2024-03-19T11:32:10.736 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.737 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.737 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:10.737 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.804470+0000", 2024-03-19T11:32:10.737 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5556000000000001e-05 2024-03-19T11:32:10.737 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.737 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.737 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.737 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.815325+0000", 2024-03-19T11:32:10.737 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010855069 2024-03-19T11:32:10.737 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.737 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.737 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:10.738 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.815684+0000", 2024-03-19T11:32:10.738 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00035912699999999999 2024-03-19T11:32:10.738 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.738 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.738 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:10.738 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.817788+0000", 2024-03-19T11:32:10.738 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0021035200000000002 2024-03-19T11:32:10.738 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.738 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:10.738 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.738 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.739 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.739 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(osd.1.0:198 4.16 e67/48 4:6868af91:::smithi03829856-10:head v 67'79, mlcod=66'76)", 2024-03-19T11:32:10.739 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:45.811039+0000", 2024-03-19T11:32:10.739 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 84.890740519999994, 2024-03-19T11:32:10.739 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.01087364, 2024-03-19T11:32:10.739 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:10.739 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:10.739 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:10.739 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.739 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:10.740 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.811039+0000", 2024-03-19T11:32:10.740 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.740 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.740 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.740 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:10.740 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.811039+0000", 2024-03-19T11:32:10.740 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.740 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.740 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.740 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:10.740 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.811041+0000", 2024-03-19T11:32:10.740 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6400000000000001e-06 2024-03-19T11:32:10.740 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.741 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.741 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:10.741 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.812874+0000", 2024-03-19T11:32:10.741 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0018329959999999999 2024-03-19T11:32:10.741 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.741 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.741 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:10.741 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.812916+0000", 2024-03-19T11:32:10.741 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1875000000000001e-05 2024-03-19T11:32:10.741 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.741 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.742 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:10.742 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.812937+0000", 2024-03-19T11:32:10.742 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0985e-05 2024-03-19T11:32:10.742 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.742 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.742 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.742 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.813112+0000", 2024-03-19T11:32:10.742 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000174573 2024-03-19T11:32:10.742 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.742 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.742 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:10.743 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.813170+0000", 2024-03-19T11:32:10.743 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8100999999999998e-05 2024-03-19T11:32:10.743 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.743 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.743 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:32:10.743 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.821792+0000", 2024-03-19T11:32:10.743 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0086218549999999994 2024-03-19T11:32:10.743 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.743 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.743 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:10.743 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:45.821912+0000", 2024-03-19T11:32:10.744 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000120615 2024-03-19T11:32:10.744 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.744 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:10.744 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.744 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.744 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.744 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2027 4.2d 4:b4ef97e0:::smithi03829856-3:head [read 0~3697507,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:32:10.744 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:46.264908+0000", 2024-03-19T11:32:10.744 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 84.436871654000001, 2024-03-19T11:32:10.745 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010951414, 2024-03-19T11:32:10.745 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:10.745 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:10.745 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:10.745 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:10.745 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:10.745 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2027 2024-03-19T11:32:10.745 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.745 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:10.745 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.745 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:10.745 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264908+0000", 2024-03-19T11:32:10.745 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.746 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.746 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.746 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:10.746 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264908+0000", 2024-03-19T11:32:10.746 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.746 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.746 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.746 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:10.746 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264908+0000", 2024-03-19T11:32:10.746 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5799999999999999e-07 2024-03-19T11:32:10.747 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.747 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.747 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:10.747 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264912+0000", 2024-03-19T11:32:10.747 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2649999999999998e-06 2024-03-19T11:32:10.747 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.747 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.747 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:10.747 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264920+0000", 2024-03-19T11:32:10.747 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.79e-06 2024-03-19T11:32:10.748 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.748 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.748 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:10.748 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264926+0000", 2024-03-19T11:32:10.748 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9989999999999997e-06 2024-03-19T11:32:10.748 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.748 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.748 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.748 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.273461+0000", 2024-03-19T11:32:10.748 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0085352680000000004 2024-03-19T11:32:10.749 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.749 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.749 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:10.749 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.273598+0000", 2024-03-19T11:32:10.749 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013622299999999999 2024-03-19T11:32:10.749 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.749 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.750 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:10.750 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.275859+0000", 2024-03-19T11:32:10.750 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022613109999999998 2024-03-19T11:32:10.750 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.750 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:10.750 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.750 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.750 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.750 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2028 4.2d 4:b4ef97e0:::smithi03829856-3:head [read 0~3697507,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:32:10.750 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:46.264934+0000", 2024-03-19T11:32:10.751 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 84.436845023000004, 2024-03-19T11:32:10.751 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013708481, 2024-03-19T11:32:10.751 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:10.751 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:10.751 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:10.751 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:10.751 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:10.751 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2028 2024-03-19T11:32:10.751 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.751 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:10.751 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.751 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:10.751 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264934+0000", 2024-03-19T11:32:10.752 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.752 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.752 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.752 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:10.752 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264934+0000", 2024-03-19T11:32:10.752 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.752 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.752 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.752 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:10.752 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264935+0000", 2024-03-19T11:32:10.752 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2599999999999998e-07 2024-03-19T11:32:10.752 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.752 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.752 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:10.753 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264938+0000", 2024-03-19T11:32:10.753 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.163e-06 2024-03-19T11:32:10.753 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.753 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.753 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:10.753 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264945+0000", 2024-03-19T11:32:10.753 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.0369999999999997e-06 2024-03-19T11:32:10.753 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.753 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.753 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:10.753 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.264950+0000", 2024-03-19T11:32:10.753 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1170000000000003e-06 2024-03-19T11:32:10.754 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.754 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.754 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.754 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.275914+0000", 2024-03-19T11:32:10.754 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010964150000000001 2024-03-19T11:32:10.754 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.754 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.755 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:10.755 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.276037+0000", 2024-03-19T11:32:10.755 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012250100000000001 2024-03-19T11:32:10.755 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.755 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.755 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:10.755 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.278643+0000", 2024-03-19T11:32:10.755 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002606087 2024-03-19T11:32:10.755 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.755 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:10.756 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.756 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.756 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.756 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:2046 4.4 4:21eac9b2:::smithi03829856-4:head [read 0~2537857,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-03-19T11:32:10.756 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:30:46.265152+0000", 2024-03-19T11:32:10.756 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 84.436627356000002, 2024-03-19T11:32:10.756 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013074417, 2024-03-19T11:32:10.756 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:10.756 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:10.756 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:10.756 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:10.756 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:10.756 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 2046 2024-03-19T11:32:10.756 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.757 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:10.757 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.757 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:10.757 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265152+0000", 2024-03-19T11:32:10.757 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.757 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.757 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.757 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:10.757 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265152+0000", 2024-03-19T11:32:10.757 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.758 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.758 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.758 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:10.758 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265152+0000", 2024-03-19T11:32:10.758 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8999999999999997e-07 2024-03-19T11:32:10.758 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.758 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.758 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:10.758 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265155+0000", 2024-03-19T11:32:10.758 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0920000000000002e-06 2024-03-19T11:32:10.758 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.758 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.759 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:10.759 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265162+0000", 2024-03-19T11:32:10.759 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6730000000000003e-06 2024-03-19T11:32:10.759 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.759 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.759 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:10.759 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.265167+0000", 2024-03-19T11:32:10.759 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1839999999999998e-06 2024-03-19T11:32:10.759 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.759 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.759 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.759 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.276218+0000", 2024-03-19T11:32:10.759 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011050559999999999 2024-03-19T11:32:10.760 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.760 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.760 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:10.760 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.276345+0000", 2024-03-19T11:32:10.760 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000127243 2024-03-19T11:32:10.760 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.760 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.760 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:10.760 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:30:46.278226+0000", 2024-03-19T11:32:10.760 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0018811749999999999 2024-03-19T11:32:10.760 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.760 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:10.761 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.761 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.761 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.761 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:3046 4.19 e103/73 4:9c74b127:::smithi03829856-33:head v 103'148, mlcod=99'144)", 2024-03-19T11:32:10.761 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:22.149246+0000", 2024-03-19T11:32:10.761 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 48.552533287999999, 2024-03-19T11:32:10.761 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014101367, 2024-03-19T11:32:10.761 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:10.761 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:10.761 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:10.761 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.761 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:10.762 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.149246+0000", 2024-03-19T11:32:10.762 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.762 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.762 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.762 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:10.762 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.149246+0000", 2024-03-19T11:32:10.762 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.762 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.762 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.762 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:10.762 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.149246+0000", 2024-03-19T11:32:10.762 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2300000000000002e-07 2024-03-19T11:32:10.762 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.763 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.763 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:10.763 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.149663+0000", 2024-03-19T11:32:10.763 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00041629599999999997 2024-03-19T11:32:10.763 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.763 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.763 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:10.763 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.149682+0000", 2024-03-19T11:32:10.763 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9500999999999998e-05 2024-03-19T11:32:10.763 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.763 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.763 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:10.763 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.149692+0000", 2024-03-19T11:32:10.763 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.825e-06 2024-03-19T11:32:10.764 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.764 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.764 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.764 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.149797+0000", 2024-03-19T11:32:10.764 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000105222 2024-03-19T11:32:10.764 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.764 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.764 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:10.764 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.149835+0000", 2024-03-19T11:32:10.764 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7934e-05 2024-03-19T11:32:10.764 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.764 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.765 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:32:10.765 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.163304+0000", 2024-03-19T11:32:10.765 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013469122 2024-03-19T11:32:10.765 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.765 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.765 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:10.765 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:22.163347+0000", 2024-03-19T11:32:10.765 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3143999999999999e-05 2024-03-19T11:32:10.765 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.765 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:10.766 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.766 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.766 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.766 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:3221 4.4 e109/83 4:214026b3:::smithi03829856-47:head v 109'146, mlcod=108'142)", 2024-03-19T11:32:10.766 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:28.083665+0000", 2024-03-19T11:32:10.766 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 42.618113764, 2024-03-19T11:32:10.766 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010710984, 2024-03-19T11:32:10.766 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:10.766 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:10.766 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:10.766 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.766 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:10.766 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.083665+0000", 2024-03-19T11:32:10.767 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.767 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.767 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.767 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:10.767 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.083665+0000", 2024-03-19T11:32:10.767 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.767 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.767 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.767 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:10.767 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.083666+0000", 2024-03-19T11:32:10.767 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5900000000000003e-07 2024-03-19T11:32:10.767 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.767 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.767 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:10.768 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.084140+0000", 2024-03-19T11:32:10.768 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00047418100000000001 2024-03-19T11:32:10.768 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.768 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.768 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:10.768 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.084164+0000", 2024-03-19T11:32:10.768 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3978999999999999e-05 2024-03-19T11:32:10.768 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.768 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.768 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:10.768 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.084174+0000", 2024-03-19T11:32:10.769 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0149999999999999e-05 2024-03-19T11:32:10.769 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.769 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.769 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.769 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.084657+0000", 2024-03-19T11:32:10.769 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00048310200000000001 2024-03-19T11:32:10.769 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.769 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.769 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:10.769 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.084720+0000", 2024-03-19T11:32:10.769 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2946000000000003e-05 2024-03-19T11:32:10.769 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.769 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.770 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:32:10.770 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.094299+0000", 2024-03-19T11:32:10.770 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0095788390000000005 2024-03-19T11:32:10.770 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.770 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.770 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:10.770 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.094376+0000", 2024-03-19T11:32:10.770 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7428000000000005e-05 2024-03-19T11:32:10.770 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.770 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:10.770 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.770 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.770 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.771 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3240 4.1c 4:3c71e808:::smithi03829856-34:head [read 0~3210104,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e109)", 2024-03-19T11:32:10.771 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:28.277613+0000", 2024-03-19T11:32:10.771 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 42.424166599000003, 2024-03-19T11:32:10.771 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011179553, 2024-03-19T11:32:10.771 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:10.771 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:10.771 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:10.771 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:10.772 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:10.772 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3240 2024-03-19T11:32:10.772 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.772 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:10.772 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.772 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:10.772 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.277613+0000", 2024-03-19T11:32:10.772 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.772 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.773 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.773 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:10.773 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.277613+0000", 2024-03-19T11:32:10.773 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.773 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.773 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.773 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:10.773 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.277614+0000", 2024-03-19T11:32:10.773 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1769999999999999e-06 2024-03-19T11:32:10.773 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.773 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.774 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:10.774 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.277622+0000", 2024-03-19T11:32:10.774 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3590000000000008e-06 2024-03-19T11:32:10.774 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.774 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.774 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:10.774 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.277644+0000", 2024-03-19T11:32:10.774 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1642e-05 2024-03-19T11:32:10.774 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.774 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.774 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:10.775 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.277660+0000", 2024-03-19T11:32:10.775 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6316000000000001e-05 2024-03-19T11:32:10.775 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.775 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.775 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.775 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.284699+0000", 2024-03-19T11:32:10.775 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0070386950000000002 2024-03-19T11:32:10.775 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.775 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.775 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:10.775 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.284990+0000", 2024-03-19T11:32:10.776 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00029164099999999998 2024-03-19T11:32:10.776 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.776 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.776 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:10.776 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:28.288792+0000", 2024-03-19T11:32:10.776 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038017229999999999 2024-03-19T11:32:10.776 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.776 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:10.776 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.776 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.776 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.776 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:3361 4.19 e112/73 4:9ef41243:::smithi03829856-5:head v 112'173, mlcod=111'169)", 2024-03-19T11:32:10.776 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:32.254683+0000", 2024-03-19T11:32:10.777 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 38.447095685000001, 2024-03-19T11:32:10.777 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014131003, 2024-03-19T11:32:10.777 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:10.777 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:10.777 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:10.777 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.777 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:10.777 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.254683+0000", 2024-03-19T11:32:10.777 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.778 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.778 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.778 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:10.778 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.254683+0000", 2024-03-19T11:32:10.778 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.778 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.781 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.782 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:10.782 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.254684+0000", 2024-03-19T11:32:10.782 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0900000000000001e-07 2024-03-19T11:32:10.782 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.782 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.782 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:10.782 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.255377+0000", 2024-03-19T11:32:10.782 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00069336899999999999 2024-03-19T11:32:10.782 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.783 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.783 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:10.783 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.255407+0000", 2024-03-19T11:32:10.783 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9113000000000001e-05 2024-03-19T11:32:10.783 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.783 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.783 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:10.783 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.255426+0000", 2024-03-19T11:32:10.783 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9026e-05 2024-03-19T11:32:10.783 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.784 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.784 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.784 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.255607+0000", 2024-03-19T11:32:10.784 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00018114199999999999 2024-03-19T11:32:10.784 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.784 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.784 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:10.784 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.255656+0000", 2024-03-19T11:32:10.784 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9679999999999999e-05 2024-03-19T11:32:10.784 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.784 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.784 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:32:10.784 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.268717+0000", 2024-03-19T11:32:10.785 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013060992 2024-03-19T11:32:10.785 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.785 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.785 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:10.785 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:32.268814+0000", 2024-03-19T11:32:10.785 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7071999999999995e-05 2024-03-19T11:32:10.785 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.785 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:10.785 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.785 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.786 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.786 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.24 121/120 [PushOp(4:266590a5:::smithi03829856-16:1c, version: 100'122, data_included: [652105~662359,1728171~386217], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:1c@100'122, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 1c=[]:{1b: [1a, 17], 1c: [1c]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:27909373:::smithi03829856-46:head, version: 108'129, data_included: [403401~454961,1584925~506039], data_size: 961000, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:head@108'129, size: 2403401, copy_subset: [(0, 2403401)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2403401, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:266590a5:::smithi03829856-16:1b, version: 119'132, data_included: [652105~662359,1728171~386217], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:1b@119'132, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 1c=[]:{1b: [1a, 17], 1c: [1c]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:266590a5:::smithi03829856-16:head, version: 119'133, data_included: [652105~662359,1728171~386217], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:head@119'133, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:32:10.786 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:42.122262+0000", 2024-03-19T11:32:10.786 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 28.579516939000001, 2024-03-19T11:32:10.786 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.036216470000000001, 2024-03-19T11:32:10.786 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:10.786 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:10.786 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:10.786 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.786 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:10.787 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:42.122262+0000", 2024-03-19T11:32:10.787 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.787 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.787 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.787 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:10.787 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:42.122262+0000", 2024-03-19T11:32:10.787 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.787 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.787 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.787 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:10.787 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:42.122264+0000", 2024-03-19T11:32:10.787 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.243e-06 2024-03-19T11:32:10.787 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.788 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.788 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:10.788 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:42.126418+0000", 2024-03-19T11:32:10.788 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0041535330000000001 2024-03-19T11:32:10.788 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.788 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.788 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:10.789 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:42.126502+0000", 2024-03-19T11:32:10.789 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4226000000000003e-05 2024-03-19T11:32:10.789 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.789 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.789 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:10.789 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:42.126612+0000", 2024-03-19T11:32:10.789 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011014900000000001 2024-03-19T11:32:10.789 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.789 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.789 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.789 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:42.127119+0000", 2024-03-19T11:32:10.790 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00050673999999999999 2024-03-19T11:32:10.790 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.790 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.790 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:10.790 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:42.127200+0000", 2024-03-19T11:32:10.790 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0548000000000002e-05 2024-03-19T11:32:10.790 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.790 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.790 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:10.790 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:42.158479+0000", 2024-03-19T11:32:10.791 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.031279030999999999 2024-03-19T11:32:10.791 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.791 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:10.791 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.791 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.791 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.791 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGRecoveryDelete(4.24 e121,120 4:266590a5:::smithi03829856-16:4,47'63,4:27909373:::smithi03829856-46:5,55'79,4:266590a5:::smithi03829856-16:10,67'105,4:27909373:::smithi03829856-46:8,85'116,4:266590a5:::smithi03829856-16:e,108'126,4:27909373:::smithi03829856-46:c,108'128)", 2024-03-19T11:32:10.791 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:42.126728+0000", 2024-03-19T11:32:10.791 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 28.575051432999999, 2024-03-19T11:32:10.791 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.032572336, 2024-03-19T11:32:10.791 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:10.791 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:10.791 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:10.791 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.791 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:10.792 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:42.126728+0000", 2024-03-19T11:32:10.792 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.792 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.792 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.792 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:10.792 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:42.126728+0000", 2024-03-19T11:32:10.792 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.792 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.792 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.792 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:10.792 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:42.126729+0000", 2024-03-19T11:32:10.792 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4500000000000001e-06 2024-03-19T11:32:10.793 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.793 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.793 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:10.793 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:42.126750+0000", 2024-03-19T11:32:10.793 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0336e-05 2024-03-19T11:32:10.793 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.793 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.793 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:10.793 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:42.126787+0000", 2024-03-19T11:32:10.793 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7431000000000001e-05 2024-03-19T11:32:10.793 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.793 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.793 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:10.793 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:42.126825+0000", 2024-03-19T11:32:10.794 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8244999999999997e-05 2024-03-19T11:32:10.794 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.794 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.794 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.794 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:42.158572+0000", 2024-03-19T11:32:10.794 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.031746586 2024-03-19T11:32:10.794 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.794 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.794 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:10.794 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:42.158593+0000", 2024-03-19T11:32:10.794 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0985e-05 2024-03-19T11:32:10.794 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.794 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.794 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:10.795 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:42.159300+0000", 2024-03-19T11:32:10.795 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00070730299999999995 2024-03-19T11:32:10.795 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.795 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:10.795 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.795 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.795 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.795 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "MOSDPGPush(4.24 121/120 [PushOp(4:266590a5:::smithi03829856-16:1c, version: 100'122, data_included: [2114388~668125], data_size: 668125, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:1c@100'122, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 1c=[]:{1b: [1a, 17], 1c: [1c]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3092455, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:266590a5:::smithi03829856-16:1b, version: 119'132, data_included: [2114388~668125], data_size: 668125, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:1b@119'132, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 1c=[]:{1b: [1a, 17], 1c: [1c]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3092455, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false)),PushOp(4:266590a5:::smithi03829856-16:head, version: 119'133, data_included: [2114388~668125], data_size: 668125, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:head@119'133, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3092455, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-03-19T11:32:10.795 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:42.165711+0000", 2024-03-19T11:32:10.795 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 28.536068437000001, 2024-03-19T11:32:10.795 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013947197999999999, 2024-03-19T11:32:10.795 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:10.795 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-19T11:32:10.795 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:10.796 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.796 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:10.796 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:42.165711+0000", 2024-03-19T11:32:10.796 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.796 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.796 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.796 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:10.796 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:42.165711+0000", 2024-03-19T11:32:10.796 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.796 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.796 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.796 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:10.796 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:42.165712+0000", 2024-03-19T11:32:10.796 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.663e-06 2024-03-19T11:32:10.797 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.797 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.797 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:10.797 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:42.166863+0000", 2024-03-19T11:32:10.797 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001150826 2024-03-19T11:32:10.797 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.797 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.797 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:10.797 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:42.166974+0000", 2024-03-19T11:32:10.797 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000110345 2024-03-19T11:32:10.797 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.797 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.798 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:10.798 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:42.167078+0000", 2024-03-19T11:32:10.798 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000104065 2024-03-19T11:32:10.798 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.798 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.798 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.798 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:42.167464+0000", 2024-03-19T11:32:10.798 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00038676299999999998 2024-03-19T11:32:10.798 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.798 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.798 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:10.798 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:42.167500+0000", 2024-03-19T11:32:10.799 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5682000000000002e-05 2024-03-19T11:32:10.799 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.799 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.799 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:10.799 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:42.179658+0000", 2024-03-19T11:32:10.799 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012157853999999999 2024-03-19T11:32:10.799 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.799 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:10.799 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.799 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.799 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.799 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.4415.0:3644 3.0 e121/20 3:26365b39:::06ff6785225cc3b4cfdfb553e1752811c747de0144ee0c1f32d8d664fcdf3873:head v 121'160, mlcod=119'158)", 2024-03-19T11:32:10.799 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:42.167963+0000", 2024-03-19T11:32:10.800 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 28.533815800999999, 2024-03-19T11:32:10.800 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010323956, 2024-03-19T11:32:10.800 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:10.800 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:10.800 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:10.800 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.800 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:10.800 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:42.167963+0000", 2024-03-19T11:32:10.800 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.800 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.800 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.800 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:10.800 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:42.167963+0000", 2024-03-19T11:32:10.800 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.800 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.801 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.801 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:10.801 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:42.167965+0000", 2024-03-19T11:32:10.801 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8330000000000001e-06 2024-03-19T11:32:10.801 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.801 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.801 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:10.801 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:42.170973+0000", 2024-03-19T11:32:10.801 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030075050000000002 2024-03-19T11:32:10.801 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.801 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.801 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:10.801 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:42.171001+0000", 2024-03-19T11:32:10.801 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.7818e-05 2024-03-19T11:32:10.801 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.802 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.802 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:10.802 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:42.171018+0000", 2024-03-19T11:32:10.802 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7246000000000001e-05 2024-03-19T11:32:10.802 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.802 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.802 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.802 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:42.171144+0000", 2024-03-19T11:32:10.802 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000126415 2024-03-19T11:32:10.802 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.802 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.803 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:10.803 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:42.171178+0000", 2024-03-19T11:32:10.803 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3380999999999998e-05 2024-03-19T11:32:10.803 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.803 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.803 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:32:10.803 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:42.178203+0000", 2024-03-19T11:32:10.803 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0070249759999999996 2024-03-19T11:32:10.803 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.803 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.803 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:10.803 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:42.178287+0000", 2024-03-19T11:32:10.803 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4782000000000002e-05 2024-03-19T11:32:10.803 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.804 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:10.804 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.804 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.804 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.804 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.4415.0:3792 4.19 4:9c74b127:::smithi03829856-33:head [write 675714~462965,stat] snapc 28=[28,27,26,25,24,23,22,21,1f,1e,1d,1a,17] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-03-19T11:32:10.804 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-19T11:31:46.560439+0000", 2024-03-19T11:32:10.804 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 24.141339879, 2024-03-19T11:32:10.804 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011671071, 2024-03-19T11:32:10.804 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-19T11:32:10.804 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:10.804 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-19T11:32:10.804 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.4415", 2024-03-19T11:32:10.804 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:10.804 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 3792 2024-03-19T11:32:10.805 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.805 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-19T11:32:10.805 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.805 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-19T11:32:10.805 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:46.560439+0000", 2024-03-19T11:32:10.805 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.805 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.805 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.805 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-19T11:32:10.805 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:46.560439+0000", 2024-03-19T11:32:10.805 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-19T11:32:10.805 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.805 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.806 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-19T11:32:10.806 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:46.560443+0000", 2024-03-19T11:32:10.806 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7100000000000001e-06 2024-03-19T11:32:10.806 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.806 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.806 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-19T11:32:10.806 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:46.560993+0000", 2024-03-19T11:32:10.806 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00055005699999999995 2024-03-19T11:32:10.806 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.806 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.806 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-19T11:32:10.806 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:46.561039+0000", 2024-03-19T11:32:10.806 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6354999999999999e-05 2024-03-19T11:32:10.806 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.807 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.807 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-19T11:32:10.807 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:46.561065+0000", 2024-03-19T11:32:10.807 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5718999999999999e-05 2024-03-19T11:32:10.807 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.807 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.807 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-19T11:32:10.807 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:46.561212+0000", 2024-03-19T11:32:10.807 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014678799999999999 2024-03-19T11:32:10.807 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.807 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.807 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-19T11:32:10.807 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:46.561408+0000", 2024-03-19T11:32:10.807 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019614599999999999 2024-03-19T11:32:10.807 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.808 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.808 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0", 2024-03-19T11:32:10.808 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:46.561814+0000", 2024-03-19T11:32:10.808 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00040588799999999997 2024-03-19T11:32:10.808 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.808 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.808 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-19T11:32:10.808 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:46.569354+0000", 2024-03-19T11:32:10.808 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0075403680000000004 2024-03-19T11:32:10.808 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.808 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.808 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:10.808 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:46.571912+0000", 2024-03-19T11:32:10.808 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025574740000000001 2024-03-19T11:32:10.809 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.809 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.809 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-19T11:32:10.809 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:46.572017+0000", 2024-03-19T11:32:10.809 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010535900000000001 2024-03-19T11:32:10.809 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-19T11:32:10.809 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-19T11:32:10.809 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-19T11:32:10.809 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-19T11:31:46.572110+0000", 2024-03-19T11:32:10.809 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3207000000000004e-05 2024-03-19T11:32:10.809 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.809 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:10.809 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.809 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-19T11:32:10.809 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-19T11:32:10.810 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-19T11:32:10.811 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-19T11:32:10.812 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:32:10.819+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:32:10.814 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-19T11:32:10.814 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-19T11:32:10.830 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-19T11:32:10.830 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-19T11:32:10.845 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-19T11:32:10.846 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-19T11:32:10.861 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-19T11:32:10.862 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-19T11:32:10.912 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:32:10.921+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:32:10.924 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:10.924 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:10.938 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-19T11:32:11.013 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:32:11.025+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:32:11.109 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:11.109 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:11.114 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:32:11.121+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:32:11.122 INFO:tasks.rados.rados.0.smithi038.stdout:1845: done (5 left) 2024-03-19T11:32:11.122 INFO:tasks.rados.rados.0.smithi038.stdout:1846: done (4 left) 2024-03-19T11:32:11.123 INFO:tasks.rados.rados.0.smithi038.stdout:1847: done (3 left) 2024-03-19T11:32:11.123 INFO:tasks.rados.rados.0.smithi038.stdout:1848: done (2 left) 2024-03-19T11:32:11.123 INFO:tasks.rados.rados.0.smithi038.stdout:1849: done (1 left) 2024-03-19T11:32:11.123 INFO:tasks.rados.rados.0.smithi038.stdout:1850: read oid 14 snap -1 2024-03-19T11:32:11.123 INFO:tasks.rados.rados.0.smithi038.stdout:1850: expect deleted 2024-03-19T11:32:11.123 INFO:tasks.rados.rados.0.smithi038.stdout:1851: read oid 37 snap 40 2024-03-19T11:32:11.123 INFO:tasks.rados.rados.0.smithi038.stdout:1851: expect (ObjNum 336 snap 40 seq_num 336) 2024-03-19T11:32:11.123 INFO:tasks.rados.rados.0.smithi038.stdout:1852: read oid 21 snap -1 2024-03-19T11:32:11.123 INFO:tasks.rados.rados.0.smithi038.stdout:1852: expect (ObjNum 339 snap 40 seq_num 339) 2024-03-19T11:32:11.123 INFO:tasks.rados.rados.0.smithi038.stdout:1853: read oid 2 snap -1 2024-03-19T11:32:11.123 INFO:tasks.rados.rados.0.smithi038.stdout:1853: expect (ObjNum 206 snap 22 seq_num 206) 2024-03-19T11:32:11.123 INFO:tasks.rados.rados.0.smithi038.stdout:1854: write (excl) oid 24 current snap is 43 2024-03-19T11:32:11.123 INFO:tasks.rados.rados.0.smithi038.stdout:1854: seq_num 348 ranges {648836=622537,1981263=493111,3107319=741517} 2024-03-19T11:32:11.130 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-19T11:32:11.138 INFO:tasks.rados.rados.0.smithi038.stdout:1854: writing smithi03829856-24 from 648836 to 1271373 tid 1 2024-03-19T11:32:11.149 INFO:tasks.rados.rados.0.smithi038.stdout:1854: writing smithi03829856-24 from 1981263 to 2474374 tid 2 2024-03-19T11:32:11.161 INFO:tasks.rados.rados.0.smithi038.stdout:1854: writing smithi03829856-24 from 3107319 to 3848836 tid 3 2024-03-19T11:32:11.162 INFO:tasks.rados.rados.0.smithi038.stdout:1855: snap_create 2024-03-19T11:32:11.162 INFO:tasks.rados.rados.0.smithi038.stdout:1853: expect (ObjNum 206 snap 22 seq_num 206) 2024-03-19T11:32:11.215 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:32:11.223+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:32:11.258 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:11.258 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:11.271 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-19T11:32:11.320 INFO:tasks.rados.rados.0.smithi038.stdout:1852: expect (ObjNum 339 snap 40 seq_num 339) 2024-03-19T11:32:11.387 INFO:teuthology.orchestra.run.smithi172.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-19T11:32:11.390 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-19T11:32:11.394 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-19T11:32:11.444 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:32:11.453+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:32:11.488 INFO:tasks.rados.rados.0.smithi038.stdout:1854: finishing write tid 1 to smithi03829856-24 2024-03-19T11:32:11.489 INFO:tasks.rados.rados.0.smithi038.stdout:1854: finishing write tid 2 to smithi03829856-24 2024-03-19T11:32:11.489 INFO:tasks.rados.rados.0.smithi038.stdout:1854: finishing write tid 3 to smithi03829856-24 2024-03-19T11:32:11.489 INFO:tasks.rados.rados.0.smithi038.stdout:1854: finishing write tid 4 to smithi03829856-24 2024-03-19T11:32:11.489 INFO:tasks.rados.rados.0.smithi038.stdout:1854: finishing write tid 5 to smithi03829856-24 2024-03-19T11:32:11.489 INFO:tasks.rados.rados.0.smithi038.stdout:1854: oid 24 updating version 0 to 216 2024-03-19T11:32:11.489 INFO:tasks.rados.rados.0.smithi038.stdout:1854: oid 24 updating version 216 to 218 2024-03-19T11:32:11.489 INFO:tasks.rados.rados.0.smithi038.stdout:1854: oid 24 version 218 is already newer than 217 2024-03-19T11:32:11.489 INFO:tasks.rados.rados.0.smithi038.stdout:1854: oid 24 updating version 218 to 219 2024-03-19T11:32:11.489 INFO:tasks.rados.rados.0.smithi038.stdout:update_object_version oid 24 v 219 (ObjNum 348 snap 43 seq_num 348) dirty exists 2024-03-19T11:32:11.489 INFO:tasks.rados.rados.0.smithi038.stdout:1854: left oid 24 (ObjNum 348 snap 43 seq_num 348) 2024-03-19T11:32:11.503 INFO:teuthology.orchestra.run.smithi172.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-19T11:32:11.506 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-19T11:32:11.507 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-19T11:32:11.519 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:32:11.527+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:32:11.620 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:32:11.627+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:32:11.649 INFO:teuthology.orchestra.run.smithi172.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-19T11:32:11.652 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-19T11:32:11.653 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-19T11:32:11.721 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:32:11.731+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:32:11.776 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:32:11.777 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:32:11.777 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:32:11.777 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:32:11.791 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-19T11:32:11.822 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:32:11.831+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:32:11.914 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:32:11.914 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:32:11.914 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:32:11.915 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:32:11.915 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:32:11.917 INFO:tasks.ceph.osd.4.smithi172.stderr:daemon-helper: command crashed with signal 6 2024-03-19T11:32:11.929 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-19T11:32:12.024 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:32:12.031+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:32:12.102 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:32:12.102 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:32:12.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:32:12.102 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:32:12.102 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.102 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1158 4.8 4:15708482:::smithi03829856-23:head [tier-promote] snapc 8=[8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-03-19T11:32:12.102 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:16.827517+0000", 2024-03-19T11:32:12.102 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 115.282025103, 2024-03-19T11:32:12.102 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.034096983999999997, 2024-03-19T11:32:12.102 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:12.103 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:32:12.103 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:12.103 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:32:12.103 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:12.103 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1158 2024-03-19T11:32:12.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.103 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:12.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.103 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:12.103 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827517+0000", 2024-03-19T11:32:12.103 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.103 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.103 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.103 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:12.104 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827517+0000", 2024-03-19T11:32:12.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.104 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:12.104 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827520+0000", 2024-03-19T11:32:12.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.45e-06 2024-03-19T11:32:12.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.104 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:12.104 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827535+0000", 2024-03-19T11:32:12.104 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.509e-05 2024-03-19T11:32:12.104 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.104 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.104 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:12.105 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827579+0000", 2024-03-19T11:32:12.105 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4211000000000001e-05 2024-03-19T11:32:12.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.105 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:12.105 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827603+0000", 2024-03-19T11:32:12.105 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3586000000000001e-05 2024-03-19T11:32:12.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.105 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.105 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.105 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.827789+0000", 2024-03-19T11:32:12.105 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018580199999999999 2024-03-19T11:32:12.105 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:12.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.828077+0000", 2024-03-19T11:32:12.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028846500000000001 2024-03-19T11:32:12.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.106 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.106 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:12.106 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:16.861614+0000", 2024-03-19T11:32:12.106 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033536379999999998 2024-03-19T11:32:12.106 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.106 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:12.106 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.106 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.107 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.107 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1209 4.8 4:1d58db11:::smithi03829856-18:head [write 645402~551297,stat] snapc 9=[9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2024-03-19T11:32:12.107 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:18.121657+0000", 2024-03-19T11:32:12.107 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 113.987884331, 2024-03-19T11:32:12.107 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.025074349999999999, 2024-03-19T11:32:12.107 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:12.107 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:12.107 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:12.107 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:32:12.107 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:12.107 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1209 2024-03-19T11:32:12.107 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.108 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:12.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.108 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:12.108 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.121657+0000", 2024-03-19T11:32:12.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.108 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.108 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:12.108 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.121657+0000", 2024-03-19T11:32:12.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.108 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.108 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.108 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:12.108 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.121661+0000", 2024-03-19T11:32:12.108 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0489999999999997e-06 2024-03-19T11:32:12.109 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.109 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.109 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:12.109 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123747+0000", 2024-03-19T11:32:12.109 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0020857720000000001 2024-03-19T11:32:12.109 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.109 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.109 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:12.109 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123779+0000", 2024-03-19T11:32:12.109 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2258000000000002e-05 2024-03-19T11:32:12.109 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.109 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.109 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:12.109 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123795+0000", 2024-03-19T11:32:12.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5177000000000001e-05 2024-03-19T11:32:12.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.110 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.110 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.110 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.123949+0000", 2024-03-19T11:32:12.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015403699999999999 2024-03-19T11:32:12.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.110 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.110 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:12.110 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.124134+0000", 2024-03-19T11:32:12.110 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000185484 2024-03-19T11:32:12.110 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.110 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.111 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:32:12.111 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.124554+0000", 2024-03-19T11:32:12.111 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00042019699999999998 2024-03-19T11:32:12.111 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.111 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.111 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:32:12.111 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146267+0000", 2024-03-19T11:32:12.111 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021712838000000002 2024-03-19T11:32:12.111 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.111 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.111 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:12.111 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146424+0000", 2024-03-19T11:32:12.111 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015647299999999999 2024-03-19T11:32:12.111 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.112 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.112 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:12.112 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146597+0000", 2024-03-19T11:32:12.112 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017303699999999999 2024-03-19T11:32:12.112 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.112 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.112 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:12.112 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:18.146732+0000", 2024-03-19T11:32:12.112 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013502800000000001 2024-03-19T11:32:12.112 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.112 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:12.112 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.112 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.112 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.113 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1443 4.2 4:43b32242:::smithi03829856-22:head [read 0~2352947,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:32:12.113 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.979417+0000", 2024-03-19T11:32:12.113 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 106.130124658, 2024-03-19T11:32:12.113 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.96719988700000004, 2024-03-19T11:32:12.113 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:12.113 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:32:12.113 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:12.113 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:32:12.113 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:12.113 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1443 2024-03-19T11:32:12.113 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.113 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:12.113 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.113 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:12.114 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979417+0000", 2024-03-19T11:32:12.114 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.114 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.114 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.114 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:12.114 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979417+0000", 2024-03-19T11:32:12.114 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.114 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.114 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.114 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:12.114 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979420+0000", 2024-03-19T11:32:12.115 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.366e-06 2024-03-19T11:32:12.115 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.115 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.115 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:12.115 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979437+0000", 2024-03-19T11:32:12.115 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6192e-05 2024-03-19T11:32:12.115 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.115 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.115 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:12.115 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979474+0000", 2024-03-19T11:32:12.115 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7601000000000001e-05 2024-03-19T11:32:12.116 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.116 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.116 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:12.116 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979496+0000", 2024-03-19T11:32:12.116 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1506999999999999e-05 2024-03-19T11:32:12.116 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.116 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.116 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.116 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979712+0000", 2024-03-19T11:32:12.116 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00021641299999999999 2024-03-19T11:32:12.116 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.116 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.117 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:32:12.117 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979746+0000", 2024-03-19T11:32:12.117 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3386000000000002e-05 2024-03-19T11:32:12.117 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.117 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.117 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.117 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.931552+0000", 2024-03-19T11:32:12.117 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.95180659899999998 2024-03-19T11:32:12.117 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.117 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.117 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-03-19T11:32:12.117 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.934681+0000", 2024-03-19T11:32:12.117 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0031292289999999999 2024-03-19T11:32:12.117 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.118 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.118 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.118 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.942701+0000", 2024-03-19T11:32:12.118 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0080198809999999995 2024-03-19T11:32:12.118 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.118 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.118 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:12.118 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.942839+0000", 2024-03-19T11:32:12.118 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013816999999999999 2024-03-19T11:32:12.118 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.118 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.118 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:12.118 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.946617+0000", 2024-03-19T11:32:12.119 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003777543 2024-03-19T11:32:12.119 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.119 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:12.119 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.119 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.119 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.119 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1444 4.2 4:43b32242:::smithi03829856-22:head [read 0~2352947,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:32:12.119 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.979553+0000", 2024-03-19T11:32:12.119 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 106.129988955, 2024-03-19T11:32:12.119 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.97015945400000003, 2024-03-19T11:32:12.119 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:12.119 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:32:12.120 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:12.120 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:32:12.120 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:12.120 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1444 2024-03-19T11:32:12.120 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.120 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:12.120 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.120 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:12.120 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979553+0000", 2024-03-19T11:32:12.120 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.120 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.121 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:12.121 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979553+0000", 2024-03-19T11:32:12.121 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.121 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.121 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:12.121 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979554+0000", 2024-03-19T11:32:12.121 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.403e-06 2024-03-19T11:32:12.121 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.121 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.122 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:12.122 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979564+0000", 2024-03-19T11:32:12.122 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0032e-05 2024-03-19T11:32:12.122 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.122 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.122 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:12.122 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979579+0000", 2024-03-19T11:32:12.122 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4875000000000001e-05 2024-03-19T11:32:12.122 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.122 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.123 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:12.123 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979595+0000", 2024-03-19T11:32:12.123 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6300999999999999e-05 2024-03-19T11:32:12.123 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.123 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.123 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.123 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979874+0000", 2024-03-19T11:32:12.123 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000279065 2024-03-19T11:32:12.123 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.123 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.123 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:32:12.123 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979904+0000", 2024-03-19T11:32:12.123 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.94e-05 2024-03-19T11:32:12.124 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.124 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.124 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.124 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.934781+0000", 2024-03-19T11:32:12.124 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.95487744900000004 2024-03-19T11:32:12.124 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.124 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.124 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-03-19T11:32:12.124 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.934946+0000", 2024-03-19T11:32:12.124 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016485999999999999 2024-03-19T11:32:12.124 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.124 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.124 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.124 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.946829+0000", 2024-03-19T11:32:12.125 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.011882482999999999 2024-03-19T11:32:12.125 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.125 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.125 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:12.125 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.946989+0000", 2024-03-19T11:32:12.125 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000160421 2024-03-19T11:32:12.125 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.125 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.125 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:12.125 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.949712+0000", 2024-03-19T11:32:12.125 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002723165 2024-03-19T11:32:12.125 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.126 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:12.126 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.126 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.126 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.126 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1445 4.2 4:43b32242:::smithi03829856-22:head [read 0~2352947,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48)", 2024-03-19T11:32:12.126 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:25.979653+0000", 2024-03-19T11:32:12.126 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 106.12988882400001, 2024-03-19T11:32:12.126 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.97249507999999996, 2024-03-19T11:32:12.126 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:12.126 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:32:12.126 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:12.126 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:32:12.127 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:12.127 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1445 2024-03-19T11:32:12.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.127 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:12.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.127 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:12.127 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979653+0000", 2024-03-19T11:32:12.127 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.127 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.127 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:12.127 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979653+0000", 2024-03-19T11:32:12.127 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.127 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.128 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.128 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:12.128 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979654+0000", 2024-03-19T11:32:12.128 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.372e-06 2024-03-19T11:32:12.128 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.128 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.128 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:12.128 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979662+0000", 2024-03-19T11:32:12.128 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9959999999999993e-06 2024-03-19T11:32:12.128 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.128 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.128 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:12.128 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979698+0000", 2024-03-19T11:32:12.129 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5763000000000003e-05 2024-03-19T11:32:12.129 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.129 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:12.129 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979714+0000", 2024-03-19T11:32:12.129 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6283999999999999e-05 2024-03-19T11:32:12.129 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.129 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.129 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.979990+0000", 2024-03-19T11:32:12.129 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000275829 2024-03-19T11:32:12.129 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.129 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.129 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:32:12.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:25.980018+0000", 2024-03-19T11:32:12.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8082999999999999e-05 2024-03-19T11:32:12.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.130 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.935620+0000", 2024-03-19T11:32:12.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.95560133800000002 2024-03-19T11:32:12.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.130 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.130 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-03-19T11:32:12.130 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.935795+0000", 2024-03-19T11:32:12.130 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000175656 2024-03-19T11:32:12.130 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.131 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.131 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.949835+0000", 2024-03-19T11:32:12.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014040083 2024-03-19T11:32:12.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.131 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:12.131 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.950002+0000", 2024-03-19T11:32:12.131 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016673 2024-03-19T11:32:12.131 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.131 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.131 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:12.131 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:26.952148+0000", 2024-03-19T11:32:12.132 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021459460000000001 2024-03-19T11:32:12.132 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.132 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:12.132 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.132 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.132 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.132 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2207 4.28 4:15708482:::smithi03829856-23:head [write 541225~644694,stat] snapc 13=[13,12,11,d,b,6] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-03-19T11:32:12.132 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:51.160660+0000", 2024-03-19T11:32:12.132 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 80.948881998000005, 2024-03-19T11:32:12.132 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.025400267000000001, 2024-03-19T11:32:12.132 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:12.132 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:12.132 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:12.133 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:32:12.133 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:12.133 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2207 2024-03-19T11:32:12.133 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.133 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:12.133 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.133 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:12.133 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.160660+0000", 2024-03-19T11:32:12.133 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.133 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.133 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.133 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:12.133 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.160660+0000", 2024-03-19T11:32:12.133 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.134 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.134 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.134 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:12.134 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.160688+0000", 2024-03-19T11:32:12.134 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8058000000000001e-05 2024-03-19T11:32:12.134 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.134 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.134 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:12.134 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.168614+0000", 2024-03-19T11:32:12.134 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0079262810000000003 2024-03-19T11:32:12.134 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.134 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.134 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:12.135 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.168662+0000", 2024-03-19T11:32:12.135 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8327e-05 2024-03-19T11:32:12.135 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.135 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.135 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:12.135 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.168679+0000", 2024-03-19T11:32:12.135 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6305999999999999e-05 2024-03-19T11:32:12.135 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.135 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.135 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.136 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.168830+0000", 2024-03-19T11:32:12.136 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000151714 2024-03-19T11:32:12.136 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.136 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.136 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:12.136 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.169107+0000", 2024-03-19T11:32:12.136 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027608299999999999 2024-03-19T11:32:12.136 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.136 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.136 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:32:12.136 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.169605+0000", 2024-03-19T11:32:12.136 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00049874999999999995 2024-03-19T11:32:12.136 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.136 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.137 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:12.137 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.185769+0000", 2024-03-19T11:32:12.137 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016163397999999999 2024-03-19T11:32:12.137 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.137 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.137 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:32:12.137 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.185809+0000", 2024-03-19T11:32:12.137 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0598e-05 2024-03-19T11:32:12.137 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.137 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.137 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:12.137 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.185898+0000", 2024-03-19T11:32:12.137 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8270999999999996e-05 2024-03-19T11:32:12.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.138 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:12.138 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:51.186060+0000", 2024-03-19T11:32:12.138 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016248099999999999 2024-03-19T11:32:12.138 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.138 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:12.138 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.138 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.138 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.138 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.15 75/74 [PushOp(4:af9d0d56:::smithi03829856-20:head, version: 35'55, data_included: [655360~589824,1900544~418789], data_size: 1008613, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:af9d0d56:::smithi03829856-20:head@35'55, size: 2319333, copy_subset: [(0, 2319333)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2319333, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:aa3303a5:::smithi03829856-17:b, version: 58'79, data_included: [614400~692224,1903006~356352], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:aa3303a5:::smithi03829856-17:b@58'79, size: 2268912, copy_subset: [(0, 2268912)], clone_subset: {}, snapset: 14=[]:{b: [b, 6], 10: [d], 12: [12, 11], 14: [13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2259358, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ad3ad1b3:::smithi03829856-21:b, version: 58'81, data_included: [736140~537686,2020581~315559], data_size: 853245, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:ad3ad1b3:::smithi03829856-21:b@58'81, size: 2336140, copy_subset: [(0, 2336140)], clone_subset: {}, snapset: 10=[]:{b: [b, 6], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2336140, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:afa9f208:::smithi03829856-35:12, version: 67'105, data_included: [521050~698057,1806623~350519], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:afa9f208:::smithi03829856-35:12@67'105, size: 2521050, copy_subset: [(0, 2521050)], clone_subset: {}, snapset: 12=[]:{11: [11, d, b, 6], 12: [12]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2157142, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:aa3303a5:::smithi03829856-17:12, version: 69'113, data_included: [413472~893152,1808868~155424], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:aa3303a5:::smithi03829856-17:12@69'113, size: 2319366, copy_subset: [(0, 2319366)], clone_subset: {}, snapset: 14=[]:{b: [b, 6], 10: [d], 12: [12, 11], 14: [13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1964292, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ac9cd3a3:::smithi03829856-8:10, version: 70'117, data_included: [517649~620182,1776668~428394], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:ac9cd3a3:::smithi03829856-8:10@70'117, size: 3008420, copy_subset: [(0, 3008420)], clone_subset: {}, snapset: 10=[]:{10: [d, b, 6]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2205062, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ac9cd3a3:::smithi03829856-8:head, version: 70'118, data_included: [517649~620182,1776668~428394], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:ac9cd3a3:::smithi03829856-8:head@70'118, size: 3008420, copy_subset: [(0, 3008420)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2205062, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:aa3303a5:::smithi03829856-17:10, version: 70'119, data_included: [413472~893152,1818036~155424], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:aa3303a5:::smithi03829856-17:10@70'119, size: 3613472, copy_subset: [(0, 3613472)], clone_subset: {}, snapset: 14=[]:{b: [b, 6], 10: [d], 12: [12, 11], 14: [13]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1973460, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ad3ad1b3:::smithi03829856-21:10, version: 72'122, data_included: [531121~742705,1711057~305871], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:ad3ad1b3:::smithi03829856-21:10@72'122, size: 3731121, copy_subset: [(0, 3731121)], clone_subset: {}, snapset: 10=[]:{b: [b, 6], 10: [d]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2016928, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:32:12.138 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.081994+0000", 2024-03-19T11:32:12.138 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 78.027547300999998, 2024-03-19T11:32:12.139 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.040292051000000002, 2024-03-19T11:32:12.139 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:12.139 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:32:12.139 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:12.139 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.139 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:12.139 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.081994+0000", 2024-03-19T11:32:12.139 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.139 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.139 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.139 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:12.139 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.081994+0000", 2024-03-19T11:32:12.139 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.140 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.140 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:12.140 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.081996+0000", 2024-03-19T11:32:12.140 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5069999999999999e-06 2024-03-19T11:32:12.140 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.140 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:12.140 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088193+0000", 2024-03-19T11:32:12.140 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061970209999999996 2024-03-19T11:32:12.140 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.140 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.140 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:12.140 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088252+0000", 2024-03-19T11:32:12.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.8838999999999997e-05 2024-03-19T11:32:12.141 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.141 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.141 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:12.141 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088322+0000", 2024-03-19T11:32:12.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0455000000000002e-05 2024-03-19T11:32:12.141 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.141 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.141 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.141 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088660+0000", 2024-03-19T11:32:12.141 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033735299999999998 2024-03-19T11:32:12.141 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.141 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.142 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:12.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088727+0000", 2024-03-19T11:32:12.142 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7335999999999994e-05 2024-03-19T11:32:12.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.142 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:12.142 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.122286+0000", 2024-03-19T11:32:12.142 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033559539999999999 2024-03-19T11:32:12.142 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.142 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:12.142 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.142 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.142 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.142 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.15 e75,74 4:ac9cd3a3:::smithi03829856-8:4,49'71)", 2024-03-19T11:32:12.143 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.088380+0000", 2024-03-19T11:32:12.143 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 78.021161438999997, 2024-03-19T11:32:12.143 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.044894303000000003, 2024-03-19T11:32:12.143 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:12.143 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:32:12.143 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:12.143 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.143 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:12.143 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088380+0000", 2024-03-19T11:32:12.143 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.143 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.143 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.143 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:12.143 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088380+0000", 2024-03-19T11:32:12.144 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.144 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.144 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.144 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:12.144 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088381+0000", 2024-03-19T11:32:12.144 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0500000000000004e-07 2024-03-19T11:32:12.144 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.144 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.144 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:12.144 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088385+0000", 2024-03-19T11:32:12.144 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2370000000000003e-06 2024-03-19T11:32:12.144 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.144 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.144 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:12.145 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088392+0000", 2024-03-19T11:32:12.145 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.4240000000000004e-06 2024-03-19T11:32:12.145 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.145 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.145 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:12.145 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.088397+0000", 2024-03-19T11:32:12.145 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8829999999999997e-06 2024-03-19T11:32:12.145 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.145 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.145 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.145 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.122356+0000", 2024-03-19T11:32:12.145 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033958588999999997 2024-03-19T11:32:12.145 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.146 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.146 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:12.146 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.122364+0000", 2024-03-19T11:32:12.146 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0800000000000006e-06 2024-03-19T11:32:12.146 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.146 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.146 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:12.146 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.133275+0000", 2024-03-19T11:32:12.146 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010910585 2024-03-19T11:32:12.146 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.146 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:12.146 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.146 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.146 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.147 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:2286 4.15 e75/74 4:afa9f208:::smithi03829856-35:head v 75'131, mlcod=33'54)", 2024-03-19T11:32:12.147 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.089414+0000", 2024-03-19T11:32:12.147 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 78.020128198999998, 2024-03-19T11:32:12.147 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.045421256, 2024-03-19T11:32:12.147 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:12.147 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:12.147 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:12.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.147 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:12.147 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.089414+0000", 2024-03-19T11:32:12.147 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.147 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.147 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.147 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:12.147 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.089414+0000", 2024-03-19T11:32:12.148 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.148 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.148 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:12.148 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.089414+0000", 2024-03-19T11:32:12.148 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1100000000000005e-07 2024-03-19T11:32:12.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.148 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.148 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:12.148 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.089423+0000", 2024-03-19T11:32:12.148 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3799999999999994e-06 2024-03-19T11:32:12.148 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.148 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.148 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:12.149 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.089436+0000", 2024-03-19T11:32:12.149 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2931e-05 2024-03-19T11:32:12.149 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.149 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.149 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:12.149 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.089447+0000", 2024-03-19T11:32:12.149 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1044e-05 2024-03-19T11:32:12.149 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.149 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.149 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.149 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.133330+0000", 2024-03-19T11:32:12.149 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.043882833000000003 2024-03-19T11:32:12.149 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.150 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.150 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:12.150 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.133353+0000", 2024-03-19T11:32:12.150 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3629999999999999e-05 2024-03-19T11:32:12.150 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.150 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.150 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:12.150 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.134786+0000", 2024-03-19T11:32:12.150 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0014327980000000001 2024-03-19T11:32:12.150 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.150 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.151 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:12.151 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.134835+0000", 2024-03-19T11:32:12.151 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8829000000000003e-05 2024-03-19T11:32:12.151 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.151 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:12.151 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.151 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.151 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.151 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2802 4.28 4:15708482:::smithi03829856-23:head [write 497540~612664] snapc 19=[18,17,15,b] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-03-19T11:32:12.151 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:14.060002+0000", 2024-03-19T11:32:12.152 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 58.049540069999999, 2024-03-19T11:32:12.152 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.028374612, 2024-03-19T11:32:12.152 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:12.152 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:12.152 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:12.152 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:32:12.152 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:12.152 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2802 2024-03-19T11:32:12.152 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.152 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:12.152 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.152 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:12.152 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.060002+0000", 2024-03-19T11:32:12.152 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.153 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.153 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.153 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:12.153 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.060002+0000", 2024-03-19T11:32:12.153 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.153 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.153 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.153 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:12.153 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.060005+0000", 2024-03-19T11:32:12.153 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5130000000000001e-06 2024-03-19T11:32:12.153 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.154 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.154 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:12.154 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.061352+0000", 2024-03-19T11:32:12.154 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0013468060000000001 2024-03-19T11:32:12.154 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.154 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.154 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:12.154 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.061403+0000", 2024-03-19T11:32:12.154 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.1181999999999999e-05 2024-03-19T11:32:12.154 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.154 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.154 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:12.155 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.061428+0000", 2024-03-19T11:32:12.155 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4715999999999998e-05 2024-03-19T11:32:12.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.155 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.155 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.155 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.061600+0000", 2024-03-19T11:32:12.155 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017224100000000001 2024-03-19T11:32:12.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.155 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.155 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op must wait for map", 2024-03-19T11:32:12.155 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.061619+0000", 2024-03-19T11:32:12.155 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9213000000000001e-05 2024-03-19T11:32:12.155 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.155 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.156 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.156 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.066953+0000", 2024-03-19T11:32:12.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053332149999999997 2024-03-19T11:32:12.156 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.156 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:12.156 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.067236+0000", 2024-03-19T11:32:12.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028381400000000002 2024-03-19T11:32:12.156 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.156 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.156 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:32:12.156 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.067698+0000", 2024-03-19T11:32:12.156 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00046152200000000002 2024-03-19T11:32:12.156 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.157 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.157 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:12.157 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.082795+0000", 2024-03-19T11:32:12.157 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015096729 2024-03-19T11:32:12.157 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.157 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.157 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:32:12.157 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.088033+0000", 2024-03-19T11:32:12.157 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052383259999999997 2024-03-19T11:32:12.157 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.157 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.157 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:12.157 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.088209+0000", 2024-03-19T11:32:12.157 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000175611 2024-03-19T11:32:12.157 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.158 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.158 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:12.158 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:14.088376+0000", 2024-03-19T11:32:12.158 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016772400000000001 2024-03-19T11:32:12.158 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.158 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:12.158 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.158 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.158 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.158 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3103 4.15 4:afa9f208:::smithi03829856-35:head [write 1636889~447212] snapc 1f=[1f,1e,1d,1c,1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-19T11:32:12.158 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.557970+0000", 2024-03-19T11:32:12.158 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.551571381999999, 2024-03-19T11:32:12.158 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033363528000000003, 2024-03-19T11:32:12.159 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:12.159 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:12.159 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:12.159 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:32:12.159 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:12.159 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3103 2024-03-19T11:32:12.159 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.159 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:12.159 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.159 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:12.159 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.557970+0000", 2024-03-19T11:32:12.159 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.160 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.160 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.160 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:12.160 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.557970+0000", 2024-03-19T11:32:12.160 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.160 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.160 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.160 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:12.160 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.557973+0000", 2024-03-19T11:32:12.160 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6060000000000001e-06 2024-03-19T11:32:12.160 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.160 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.160 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:12.161 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558806+0000", 2024-03-19T11:32:12.161 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00083305499999999997 2024-03-19T11:32:12.161 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.161 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.161 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:12.161 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558843+0000", 2024-03-19T11:32:12.161 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6519000000000001e-05 2024-03-19T11:32:12.161 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.161 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.161 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:12.161 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558863+0000", 2024-03-19T11:32:12.161 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0622999999999999e-05 2024-03-19T11:32:12.161 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.161 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.162 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.162 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.562831+0000", 2024-03-19T11:32:12.162 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039681250000000003 2024-03-19T11:32:12.162 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.162 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.162 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:12.162 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.563003+0000", 2024-03-19T11:32:12.162 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000171638 2024-03-19T11:32:12.163 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.163 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.163 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:32:12.163 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.563335+0000", 2024-03-19T11:32:12.163 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033165899999999999 2024-03-19T11:32:12.163 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.163 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.163 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:32:12.164 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.569815+0000", 2024-03-19T11:32:12.164 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00648024 2024-03-19T11:32:12.164 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.164 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.164 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:12.164 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.590982+0000", 2024-03-19T11:32:12.164 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021167235 2024-03-19T11:32:12.164 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.164 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.165 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:12.165 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.591170+0000", 2024-03-19T11:32:12.165 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018796899999999999 2024-03-19T11:32:12.165 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.165 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.165 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:12.165 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.591334+0000", 2024-03-19T11:32:12.165 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000163859 2024-03-19T11:32:12.165 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.165 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:12.165 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.165 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.165 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.166 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3104 4.15 4:afa9f208:::smithi03829856-35:head [setxattr _header (57) in=7b,truncate 2084101] snapc 1f=[1f,1e,1d,1c,1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-19T11:32:12.166 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.558905+0000", 2024-03-19T11:32:12.166 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.550636300000001, 2024-03-19T11:32:12.166 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032860320999999998, 2024-03-19T11:32:12.166 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:12.166 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:12.166 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:12.166 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:32:12.166 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:12.166 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3104 2024-03-19T11:32:12.166 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.166 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:12.166 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.167 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:12.167 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558905+0000", 2024-03-19T11:32:12.167 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.167 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.167 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.167 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:12.167 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558905+0000", 2024-03-19T11:32:12.167 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.167 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.167 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.167 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:12.167 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558907+0000", 2024-03-19T11:32:12.167 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.547e-06 2024-03-19T11:32:12.168 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.168 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.168 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:12.168 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558916+0000", 2024-03-19T11:32:12.168 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0589999999999992e-06 2024-03-19T11:32:12.168 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.168 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.168 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:12.168 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558931+0000", 2024-03-19T11:32:12.168 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5262999999999999e-05 2024-03-19T11:32:12.168 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.168 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.169 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:12.169 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558947+0000", 2024-03-19T11:32:12.169 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6140000000000001e-05 2024-03-19T11:32:12.169 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.169 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.169 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.169 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.567458+0000", 2024-03-19T11:32:12.169 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0085106650000000006 2024-03-19T11:32:12.169 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.169 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.169 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:12.169 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.567777+0000", 2024-03-19T11:32:12.170 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00031917399999999998 2024-03-19T11:32:12.170 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.170 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.170 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:32:12.170 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.568090+0000", 2024-03-19T11:32:12.170 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00031286900000000002 2024-03-19T11:32:12.170 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.170 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.170 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:32:12.170 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.576738+0000", 2024-03-19T11:32:12.170 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0086476379999999992 2024-03-19T11:32:12.170 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.170 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.171 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:12.171 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.591474+0000", 2024-03-19T11:32:12.171 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014736281 2024-03-19T11:32:12.171 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.171 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.171 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:12.171 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.591576+0000", 2024-03-19T11:32:12.172 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000102303 2024-03-19T11:32:12.172 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.172 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.172 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:12.172 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.591766+0000", 2024-03-19T11:32:12.172 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018938200000000001 2024-03-19T11:32:12.172 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.172 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:12.172 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.172 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.172 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.173 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3105 4.15 4:afa9f208:::smithi03829856-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e104)", 2024-03-19T11:32:12.173 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.558970+0000", 2024-03-19T11:32:12.173 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.550571865000002, 2024-03-19T11:32:12.173 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033553684, 2024-03-19T11:32:12.173 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:12.173 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:32:12.173 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:12.173 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:32:12.173 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:12.173 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3105 2024-03-19T11:32:12.173 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.173 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:12.173 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.173 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:12.173 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558970+0000", 2024-03-19T11:32:12.174 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.174 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.174 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.174 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:12.174 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558970+0000", 2024-03-19T11:32:12.174 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.174 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.174 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.174 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:12.174 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558971+0000", 2024-03-19T11:32:12.174 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4729999999999999e-06 2024-03-19T11:32:12.174 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.174 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.174 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:12.175 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.558978+0000", 2024-03-19T11:32:12.175 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.1369999999999996e-06 2024-03-19T11:32:12.175 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.175 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.175 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:12.175 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.559003+0000", 2024-03-19T11:32:12.175 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4525999999999999e-05 2024-03-19T11:32:12.175 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.175 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.175 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:12.175 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.559019+0000", 2024-03-19T11:32:12.175 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5999999999999999e-05 2024-03-19T11:32:12.175 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.175 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.175 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.176 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.569615+0000", 2024-03-19T11:32:12.176 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.010596257 2024-03-19T11:32:12.176 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.176 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.176 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-03-19T11:32:12.176 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.569763+0000", 2024-03-19T11:32:12.176 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000147319 2024-03-19T11:32:12.176 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.176 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.176 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.176 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.592019+0000", 2024-03-19T11:32:12.176 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022256597999999999 2024-03-19T11:32:12.176 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.176 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.177 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:12.177 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.592250+0000", 2024-03-19T11:32:12.177 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023081100000000001 2024-03-19T11:32:12.177 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.177 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.177 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:12.177 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.592524+0000", 2024-03-19T11:32:12.177 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000273563 2024-03-19T11:32:12.177 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.177 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:12.177 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.177 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.177 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.177 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3106 4.15 4:ad3ad1b3:::smithi03829856-21:head [write 500908~484052] snapc 1f=[1f,1e,1d,1c,1b,1a,18,17,b] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-19T11:32:12.177 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.563910+0000", 2024-03-19T11:32:12.178 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 48.545631915000001, 2024-03-19T11:32:12.178 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.029166066000000001, 2024-03-19T11:32:12.178 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:12.178 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:12.178 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:12.178 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:32:12.178 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:12.178 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3106 2024-03-19T11:32:12.178 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.178 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:12.178 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.178 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:12.178 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.563910+0000", 2024-03-19T11:32:12.178 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.179 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.179 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.179 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:12.179 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.563910+0000", 2024-03-19T11:32:12.179 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.179 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.179 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.179 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:12.179 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.563913+0000", 2024-03-19T11:32:12.179 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9960000000000002e-06 2024-03-19T11:32:12.179 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.179 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.179 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:12.179 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.564853+0000", 2024-03-19T11:32:12.179 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00093991599999999995 2024-03-19T11:32:12.180 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.180 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.180 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:12.180 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.564886+0000", 2024-03-19T11:32:12.180 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3315999999999997e-05 2024-03-19T11:32:12.180 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.180 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.180 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:12.180 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.564908+0000", 2024-03-19T11:32:12.180 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2262000000000001e-05 2024-03-19T11:32:12.180 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.180 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.180 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.180 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.569861+0000", 2024-03-19T11:32:12.181 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0049529070000000003 2024-03-19T11:32:12.181 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.181 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.181 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:12.181 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.569998+0000", 2024-03-19T11:32:12.181 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000136546 2024-03-19T11:32:12.181 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.181 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.181 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 7", 2024-03-19T11:32:12.181 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.570279+0000", 2024-03-19T11:32:12.181 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028119899999999999 2024-03-19T11:32:12.181 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.181 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.181 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:32:12.182 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.586586+0000", 2024-03-19T11:32:12.182 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016306621 2024-03-19T11:32:12.182 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.182 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.182 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:12.182 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.592743+0000", 2024-03-19T11:32:12.182 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061571150000000003 2024-03-19T11:32:12.182 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.182 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.182 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:12.183 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.592915+0000", 2024-03-19T11:32:12.183 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017273200000000001 2024-03-19T11:32:12.183 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.183 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.183 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:12.183 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.593076+0000", 2024-03-19T11:32:12.183 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016045600000000001 2024-03-19T11:32:12.183 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.183 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:12.183 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.183 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.183 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.183 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.21 130/129 [PushOp(4:866f3dcc:::smithi03829856-31:23, version: 112'156, data_included: [406769~892231,1439434~156345], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:23@112'156, size: 3899224, copy_subset: [(0, 3899224)], clone_subset: {}, snapset: 28=[]:{1c: [1a, 17], 23: [23, 22, 21, 1f, 1e, 1d], 28: [28, 27, 26, 25, 24]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1595779, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:8401c1f0:::smithi03829856-43:1e, version: 122'169, data_included: [527786~724480,1703317~324096], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8401c1f0:::smithi03829856-43:1e@122'169, size: 3327786, copy_subset: [(0, 3327786)], clone_subset: {}, snapset: 1e=[]:{1e: [1e, 1d, 1a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2027413, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:1c, version: 122'171, data_included: [406769~892231,1439434~156345], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:1c@122'171, size: 3159716, copy_subset: [(0, 3159716)], clone_subset: {}, snapset: 28=[]:{1c: [1a, 17], 23: [23, 22, 21, 1f, 1e, 1d], 28: [28, 27, 26, 25, 24]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1595779, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:28, version: 124'174, data_included: [406769~892231,1439434~156345], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:28@124'174, size: 2504276, copy_subset: [(0, 2504276)], clone_subset: {}, snapset: 28=[]:{1c: [1a, 17], 23: [23, 22, 21, 1f, 1e, 1d], 28: [28, 27, 26, 25, 24]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1595779, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:866f3dcc:::smithi03829856-31:head, version: 126'178, data_included: [406769~947798,1439434~100778], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:866f3dcc:::smithi03829856-31:head@126'178, size: 3042100, copy_subset: [(0, 3042100)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1540212, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:32:12.183 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:51.141672+0000", 2024-03-19T11:32:12.184 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 20.967869883999999, 2024-03-19T11:32:12.184 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.038330659000000003, 2024-03-19T11:32:12.184 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:12.184 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:32:12.184 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:12.184 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.184 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:12.184 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.141672+0000", 2024-03-19T11:32:12.184 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.184 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.184 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.184 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:12.184 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.141672+0000", 2024-03-19T11:32:12.184 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.184 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.185 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.185 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:12.185 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.141673+0000", 2024-03-19T11:32:12.185 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.361e-06 2024-03-19T11:32:12.185 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.185 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.185 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:12.185 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.149477+0000", 2024-03-19T11:32:12.185 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0078036019999999998 2024-03-19T11:32:12.185 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.185 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.185 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:12.185 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.149526+0000", 2024-03-19T11:32:12.185 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.8730000000000003e-05 2024-03-19T11:32:12.186 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.186 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.186 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:12.186 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.149570+0000", 2024-03-19T11:32:12.186 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4493000000000002e-05 2024-03-19T11:32:12.186 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.186 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.186 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.186 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.149925+0000", 2024-03-19T11:32:12.186 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00035484699999999999 2024-03-19T11:32:12.186 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.187 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.187 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:12.187 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.149987+0000", 2024-03-19T11:32:12.187 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2182999999999997e-05 2024-03-19T11:32:12.187 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.187 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.187 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:12.187 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.180003+0000", 2024-03-19T11:32:12.187 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030015442999999999 2024-03-19T11:32:12.187 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.187 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:12.187 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.187 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.187 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.188 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:3885 4.21 e130/129 4:8401c1f0:::smithi03829856-43:head v 130'180, mlcod=0'0)", 2024-03-19T11:32:12.188 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:51.149617+0000", 2024-03-19T11:32:12.188 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 20.959924424, 2024-03-19T11:32:12.188 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.040106843000000003, 2024-03-19T11:32:12.188 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:12.188 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:12.188 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:12.188 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.188 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:12.188 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.149617+0000", 2024-03-19T11:32:12.188 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.188 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.188 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.188 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:12.188 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.149617+0000", 2024-03-19T11:32:12.189 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.189 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.189 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.189 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:12.189 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.149618+0000", 2024-03-19T11:32:12.189 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2e-07 2024-03-19T11:32:12.189 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.189 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.189 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:12.189 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.150382+0000", 2024-03-19T11:32:12.189 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00076398999999999998 2024-03-19T11:32:12.189 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.189 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.189 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:12.190 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.150391+0000", 2024-03-19T11:32:12.190 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6409999999999997e-06 2024-03-19T11:32:12.190 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.190 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.190 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:12.190 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.150398+0000", 2024-03-19T11:32:12.190 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.2299999999999996e-06 2024-03-19T11:32:12.190 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.190 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.190 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.190 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.180043+0000", 2024-03-19T11:32:12.190 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.029645663999999999 2024-03-19T11:32:12.190 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.190 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.191 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:12.191 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.180068+0000", 2024-03-19T11:32:12.191 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4195000000000001e-05 2024-03-19T11:32:12.191 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.191 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.191 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:12.191 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.189686+0000", 2024-03-19T11:32:12.191 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0096187340000000003 2024-03-19T11:32:12.191 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.191 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.191 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:12.191 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.189724+0000", 2024-03-19T11:32:12.191 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7868999999999999e-05 2024-03-19T11:32:12.191 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.192 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:12.192 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.192 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.192 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.192 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.21 e130,129 4:8401c1f0:::smithi03829856-43:b,109'151,4:866f3dcc:::smithi03829856-31:b,109'153)", 2024-03-19T11:32:12.192 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:51.150411+0000", 2024-03-19T11:32:12.192 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 20.959130586000001, 2024-03-19T11:32:12.192 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.039270521000000003, 2024-03-19T11:32:12.192 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:12.192 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:32:12.192 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:12.192 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.192 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:12.192 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.150411+0000", 2024-03-19T11:32:12.192 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.193 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.193 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.193 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:12.193 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.150411+0000", 2024-03-19T11:32:12.193 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.193 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.193 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.193 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:12.193 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.150412+0000", 2024-03-19T11:32:12.193 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9000000000000002e-07 2024-03-19T11:32:12.193 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.193 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.193 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:12.193 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.150416+0000", 2024-03-19T11:32:12.193 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.4479999999999996e-06 2024-03-19T11:32:12.194 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.194 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.194 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:12.194 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.150433+0000", 2024-03-19T11:32:12.194 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6549999999999999e-05 2024-03-19T11:32:12.194 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.194 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.194 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:12.194 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.150439+0000", 2024-03-19T11:32:12.194 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1410000000000002e-06 2024-03-19T11:32:12.194 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.194 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.194 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.194 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.189519+0000", 2024-03-19T11:32:12.195 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.039080239000000003 2024-03-19T11:32:12.195 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.195 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.195 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:12.195 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.189528+0000", 2024-03-19T11:32:12.195 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2580000000000001e-06 2024-03-19T11:32:12.195 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.195 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.195 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:12.195 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.189682+0000", 2024-03-19T11:32:12.195 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015349499999999999 2024-03-19T11:32:12.195 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.195 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:12.195 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.195 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.196 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.196 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:3886 4.21 e130/129 4:8401c1f0:::smithi03829856-43:head v 130'181, mlcod=0'0)", 2024-03-19T11:32:12.196 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:51.153376+0000", 2024-03-19T11:32:12.196 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 20.956165879, 2024-03-19T11:32:12.196 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.036393429999999997, 2024-03-19T11:32:12.196 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:12.196 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:12.196 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:12.196 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.196 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:12.196 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.153376+0000", 2024-03-19T11:32:12.196 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.196 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.196 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.197 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:12.197 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.153376+0000", 2024-03-19T11:32:12.197 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.197 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.197 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.197 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:12.197 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.153377+0000", 2024-03-19T11:32:12.197 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9299999999999997e-07 2024-03-19T11:32:12.197 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.197 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.197 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:12.197 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.153879+0000", 2024-03-19T11:32:12.197 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00050270000000000002 2024-03-19T11:32:12.197 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.197 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.198 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:12.198 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.153897+0000", 2024-03-19T11:32:12.198 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7676000000000002e-05 2024-03-19T11:32:12.198 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.198 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.198 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:12.198 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.153906+0000", 2024-03-19T11:32:12.198 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9390000000000003e-06 2024-03-19T11:32:12.198 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.198 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.198 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.198 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.184615+0000", 2024-03-19T11:32:12.198 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030708566 2024-03-19T11:32:12.198 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.199 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:12.199 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.184660+0000", 2024-03-19T11:32:12.199 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5933999999999998e-05 2024-03-19T11:32:12.199 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.199 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:12.199 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.189731+0000", 2024-03-19T11:32:12.199 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0050710190000000004 2024-03-19T11:32:12.199 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.199 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.199 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:12.199 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.189769+0000", 2024-03-19T11:32:12.199 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7802999999999997e-05 2024-03-19T11:32:12.199 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.200 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:12.200 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.200 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.200 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.200 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:3887 4.21 e130/129 4:8401c1f0:::smithi03829856-43:head v 130'182, mlcod=0'0)", 2024-03-19T11:32:12.200 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:51.158488+0000", 2024-03-19T11:32:12.200 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 20.951054037999999, 2024-03-19T11:32:12.200 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.034422726000000001, 2024-03-19T11:32:12.200 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:12.200 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:12.200 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:12.200 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.200 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:12.200 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.158488+0000", 2024-03-19T11:32:12.200 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.201 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.201 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.201 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:12.201 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.158488+0000", 2024-03-19T11:32:12.201 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.201 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.201 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.201 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:12.201 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.158491+0000", 2024-03-19T11:32:12.201 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8490000000000002e-06 2024-03-19T11:32:12.201 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.201 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.201 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:12.201 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.159149+0000", 2024-03-19T11:32:12.202 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00065869500000000005 2024-03-19T11:32:12.202 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.202 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.202 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:12.202 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.159186+0000", 2024-03-19T11:32:12.202 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6607999999999999e-05 2024-03-19T11:32:12.202 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.202 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.202 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:12.202 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.159204+0000", 2024-03-19T11:32:12.202 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8099999999999999e-05 2024-03-19T11:32:12.202 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.202 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.202 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.203 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.187524+0000", 2024-03-19T11:32:12.203 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.028320271000000001 2024-03-19T11:32:12.203 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.203 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:12.203 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.187558+0000", 2024-03-19T11:32:12.203 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4078999999999997e-05 2024-03-19T11:32:12.203 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.203 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:12.203 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.192869+0000", 2024-03-19T11:32:12.203 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0053108269999999997 2024-03-19T11:32:12.203 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.203 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.203 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:12.204 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.192910+0000", 2024-03-19T11:32:12.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1297000000000001e-05 2024-03-19T11:32:12.204 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.204 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:12.204 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.204 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.204 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:3888 4.21 e130/129 4:8401c1f0:::smithi03829856-43:head v 130'183, mlcod=0'0)", 2024-03-19T11:32:12.204 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:51.160972+0000", 2024-03-19T11:32:12.204 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 20.948569295999999, 2024-03-19T11:32:12.204 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.032362711000000002, 2024-03-19T11:32:12.204 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:12.204 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:12.204 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:12.204 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.205 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:12.205 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.160972+0000", 2024-03-19T11:32:12.205 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.205 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:12.205 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.160972+0000", 2024-03-19T11:32:12.205 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.205 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:12.205 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.160974+0000", 2024-03-19T11:32:12.205 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.677e-06 2024-03-19T11:32:12.205 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.205 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.206 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:12.206 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.160987+0000", 2024-03-19T11:32:12.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3076e-05 2024-03-19T11:32:12.206 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.206 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.206 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:12.206 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.161009+0000", 2024-03-19T11:32:12.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.154e-05 2024-03-19T11:32:12.206 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.206 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.206 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:12.206 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.161021+0000", 2024-03-19T11:32:12.206 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2559e-05 2024-03-19T11:32:12.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.207 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.207 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.188979+0000", 2024-03-19T11:32:12.207 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027958061999999999 2024-03-19T11:32:12.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.207 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:12.207 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.189014+0000", 2024-03-19T11:32:12.207 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4644999999999998e-05 2024-03-19T11:32:12.207 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.207 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.207 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:12.207 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.193300+0000", 2024-03-19T11:32:12.207 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0042857939999999999 2024-03-19T11:32:12.208 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.208 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.208 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:12.208 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:51.193335+0000", 2024-03-19T11:32:12.210 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5358000000000003e-05 2024-03-19T11:32:12.210 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.210 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:12.210 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.210 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.210 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:12.210 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:32:12.211 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-19T11:32:12.211 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-19T11:32:12.211 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [0, 1, 2, 4, 5, 6, 7, 3] 2024-03-19T11:32:12.211 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 7 2024-03-19T11:32:12.212 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-03-19T11:32:12.212 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:32:12.123+0000 7f70c6fdf640 -1 osd.0 150 handle_osd_map injecting map crc failure 2024-03-19T11:32:12.213 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:32:12.127+0000 7f6a7f9da640 -1 osd.1 150 handle_osd_map injecting map crc failure 2024-03-19T11:32:12.214 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-19T11:32:12.215 INFO:tasks.rados.rados.0.smithi038.stdout:1850: done (6 left) 2024-03-19T11:32:12.215 INFO:tasks.rados.rados.0.smithi038.stdout:1852: done (5 left) 2024-03-19T11:32:12.215 INFO:tasks.rados.rados.0.smithi038.stdout:1853: done (4 left) 2024-03-19T11:32:12.215 INFO:tasks.rados.rados.0.smithi038.stdout:1854: done (3 left) 2024-03-19T11:32:12.215 INFO:tasks.rados.rados.0.smithi038.stdout:1855: done (2 left) 2024-03-19T11:32:12.215 INFO:tasks.rados.rados.0.smithi038.stdout:1856: tier_evict oid 28 2024-03-19T11:32:12.215 INFO:tasks.rados.rados.0.smithi038.stdout:1856: tier_evict oid 28 snap -1 2024-03-19T11:32:12.215 INFO:tasks.rados.rados.0.smithi038.stdout:1857: read oid 47 snap 31 2024-03-19T11:32:12.215 INFO:tasks.rados.rados.0.smithi038.stdout:1857: expect (ObjNum 46 snap 0 seq_num 46) 2024-03-19T11:32:12.215 INFO:tasks.rados.rados.0.smithi038.stdout:1858: tier_evict oid 9 2024-03-19T11:32:12.215 INFO:tasks.rados.rados.0.smithi038.stdout:1858: tier_evict oid 9 snap -1 2024-03-19T11:32:12.215 INFO:tasks.rados.rados.0.smithi038.stdout:1859: read oid 42 snap -1 2024-03-19T11:32:12.216 INFO:tasks.rados.rados.0.smithi038.stdout:1859: expect deleted 2024-03-19T11:32:12.216 INFO:tasks.rados.rados.0.smithi038.stdout:1860: snap_create 2024-03-19T11:32:12.216 INFO:tasks.rados.rados.0.smithi038.stdout:1856: got (22) Invalid argument 2024-03-19T11:32:12.216 INFO:tasks.rados.rados.0.smithi038.stdout:1857: expect (ObjNum 46 snap 0 seq_num 46) 2024-03-19T11:32:12.216 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:32:12.219+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:32:12.313 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:32:12.323+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:32:12.344 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:32:12.344 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:32:12.344 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:32:12.344 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:32:12.358 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-19T11:32:12.413 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:32:12.423+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:32:12.482 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:32:12.482 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:32:12.482 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:32:12.482 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:32:12.482 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:32:12.496 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-19T11:32:12.514 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:32:12.523+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:32:12.520 INFO:tasks.ceph.mon.a.smithi038.stderr:2024-03-19T11:32:12.527+0000 7f3f60a19640 -1 mon.a@0(leader).osd e151 definitely_dead 0 2024-03-19T11:32:12.616 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:32:12.625+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:32:12.623 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:32:12.623 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:32:12.623 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:32:12.624 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:32:12.624 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.624 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:891 4.5 4:af9d0d56:::smithi03829856-20:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e32)", 2024-03-19T11:32:12.624 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:09.286410+0000", 2024-03-19T11:32:12.624 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 123.34546619, 2024-03-19T11:32:12.624 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.035191405000000002, 2024-03-19T11:32:12.624 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:12.624 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:32:12.624 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:12.624 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:32:12.624 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:12.624 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 891 2024-03-19T11:32:12.624 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.624 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:12.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:12.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286410+0000", 2024-03-19T11:32:12.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.625 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:12.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286410+0000", 2024-03-19T11:32:12.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.625 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.625 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.625 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:12.625 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286413+0000", 2024-03-19T11:32:12.625 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2660000000000002e-06 2024-03-19T11:32:12.626 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.626 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:12.626 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286426+0000", 2024-03-19T11:32:12.626 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.258e-05 2024-03-19T11:32:12.626 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.626 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:12.626 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286460+0000", 2024-03-19T11:32:12.626 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.3943999999999999e-05 2024-03-19T11:32:12.626 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.626 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.626 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:12.626 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286477+0000", 2024-03-19T11:32:12.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6799999999999998e-05 2024-03-19T11:32:12.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286605+0000", 2024-03-19T11:32:12.627 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000128892 2024-03-19T11:32:12.627 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.627 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.627 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:12.627 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.286769+0000", 2024-03-19T11:32:12.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016359799999999999 2024-03-19T11:32:12.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.628 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:12.628 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:09.321601+0000", 2024-03-19T11:32:12.628 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.034832324999999997 2024-03-19T11:32:12.628 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.628 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:12.628 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.628 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.628 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.628 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1015 4.5 4:af9d0d56:::smithi03829856-20:head [tier-promote] snapc 6=[6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-03-19T11:32:12.628 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:12.761855+0000", 2024-03-19T11:32:12.629 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 119.87002066399999, 2024-03-19T11:32:12.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026760430000000002, 2024-03-19T11:32:12.629 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:12.629 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:32:12.629 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:12.629 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:32:12.629 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:12.629 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1015 2024-03-19T11:32:12.629 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.629 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:12.629 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.629 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:12.629 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761855+0000", 2024-03-19T11:32:12.629 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.630 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.630 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:12.630 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761855+0000", 2024-03-19T11:32:12.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.630 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.630 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:12.630 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761858+0000", 2024-03-19T11:32:12.630 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8140000000000001e-06 2024-03-19T11:32:12.630 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.630 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.630 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:12.630 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761872+0000", 2024-03-19T11:32:12.631 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3813e-05 2024-03-19T11:32:12.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.631 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:12.631 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761905+0000", 2024-03-19T11:32:12.631 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2642999999999999e-05 2024-03-19T11:32:12.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.631 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:12.631 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.761925+0000", 2024-03-19T11:32:12.631 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0489e-05 2024-03-19T11:32:12.631 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.631 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.631 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.631 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.762070+0000", 2024-03-19T11:32:12.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000145153 2024-03-19T11:32:12.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:12.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.762365+0000", 2024-03-19T11:32:12.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029482100000000002 2024-03-19T11:32:12.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.632 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.632 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:12.632 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:12.788616+0000", 2024-03-19T11:32:12.632 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026250697 2024-03-19T11:32:12.632 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.632 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:12.632 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.632 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.633 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.633 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:1121 4.5 4:aa3303a5:::smithi03829856-17:head [tier-promote] snapc 7=[7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e36)", 2024-03-19T11:32:12.633 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:14.164281+0000", 2024-03-19T11:32:12.633 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 118.46759541900001, 2024-03-19T11:32:12.633 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023896435000000001, 2024-03-19T11:32:12.633 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:12.633 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:32:12.633 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:12.633 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:32:12.633 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:12.633 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1121 2024-03-19T11:32:12.634 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.634 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:12.634 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.634 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:12.634 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164281+0000", 2024-03-19T11:32:12.634 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.634 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.634 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.634 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:12.634 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164281+0000", 2024-03-19T11:32:12.634 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.634 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.635 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:12.635 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164283+0000", 2024-03-19T11:32:12.635 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8500000000000001e-06 2024-03-19T11:32:12.635 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.635 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:12.635 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164291+0000", 2024-03-19T11:32:12.635 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.14e-06 2024-03-19T11:32:12.635 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.635 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.635 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:12.635 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164313+0000", 2024-03-19T11:32:12.635 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.207e-05 2024-03-19T11:32:12.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:12.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.164328+0000", 2024-03-19T11:32:12.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5403000000000001e-05 2024-03-19T11:32:12.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.167981+0000", 2024-03-19T11:32:12.636 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036529879999999998 2024-03-19T11:32:12.636 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.636 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.636 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:12.636 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.168274+0000", 2024-03-19T11:32:12.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029313100000000002 2024-03-19T11:32:12.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.637 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:12.637 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:14.188177+0000", 2024-03-19T11:32:12.637 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019902853000000002 2024-03-19T11:32:12.637 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.637 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:12.637 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.637 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.637 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.637 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2284 4.15 4:afa9f208:::smithi03829856-35:head [read 1314816~702976] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-03-19T11:32:12.637 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:53.596775+0000", 2024-03-19T11:32:12.637 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 79.035101182999995, 2024-03-19T11:32:12.638 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.47260528299999999, 2024-03-19T11:32:12.638 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:12.638 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:32:12.638 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:12.638 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:32:12.638 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:12.638 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2284 2024-03-19T11:32:12.638 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.638 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:12.638 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.638 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:12.638 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596775+0000", 2024-03-19T11:32:12.638 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.639 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:12.639 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596775+0000", 2024-03-19T11:32:12.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.639 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:12.639 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596778+0000", 2024-03-19T11:32:12.639 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.174e-06 2024-03-19T11:32:12.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.639 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.639 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:12.639 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596794+0000", 2024-03-19T11:32:12.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5372000000000001e-05 2024-03-19T11:32:12.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:12.640 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596828+0000", 2024-03-19T11:32:12.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4107999999999999e-05 2024-03-19T11:32:12.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:12.640 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596844+0000", 2024-03-19T11:32:12.640 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6296000000000001e-05 2024-03-19T11:32:12.640 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.640 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.640 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596968+0000", 2024-03-19T11:32:12.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012383699999999999 2024-03-19T11:32:12.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.641 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:32:12.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:53.596988+0000", 2024-03-19T11:32:12.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0645e-05 2024-03-19T11:32:12.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.641 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.641 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.068916+0000", 2024-03-19T11:32:12.641 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.47192744800000003 2024-03-19T11:32:12.641 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.641 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.642 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:12.642 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.069015+0000", 2024-03-19T11:32:12.642 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9500999999999995e-05 2024-03-19T11:32:12.642 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.642 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.642 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:12.642 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.069380+0000", 2024-03-19T11:32:12.642 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00036490200000000001 2024-03-19T11:32:12.642 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.642 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:12.642 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.642 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.642 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.642 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2286 4.15 4:afa9f208:::smithi03829856-35:head [assert-version v108,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e75)", 2024-03-19T11:32:12.643 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.084826+0000", 2024-03-19T11:32:12.643 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 78.547049815999998, 2024-03-19T11:32:12.643 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.055799556, 2024-03-19T11:32:12.643 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:12.643 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:12.643 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:12.643 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:32:12.643 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:12.643 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2286 2024-03-19T11:32:12.643 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.643 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:12.643 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.643 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:12.643 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084826+0000", 2024-03-19T11:32:12.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.644 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:12.644 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084826+0000", 2024-03-19T11:32:12.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.644 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:12.644 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084828+0000", 2024-03-19T11:32:12.644 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1409999999999999e-06 2024-03-19T11:32:12.644 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.644 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.644 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:12.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084838+0000", 2024-03-19T11:32:12.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.3740000000000004e-06 2024-03-19T11:32:12.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:12.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084859+0000", 2024-03-19T11:32:12.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1644000000000001e-05 2024-03-19T11:32:12.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.645 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:12.645 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084871+0000", 2024-03-19T11:32:12.645 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1127e-05 2024-03-19T11:32:12.645 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.645 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.646 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.084961+0000", 2024-03-19T11:32:12.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0371999999999995e-05 2024-03-19T11:32:12.646 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:12.646 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.085130+0000", 2024-03-19T11:32:12.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016884800000000001 2024-03-19T11:32:12.646 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.646 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 5,7", 2024-03-19T11:32:12.646 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.089309+0000", 2024-03-19T11:32:12.646 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041788650000000004 2024-03-19T11:32:12.646 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.646 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.647 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:32:12.647 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.090362+0000", 2024-03-19T11:32:12.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001053362 2024-03-19T11:32:12.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.647 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:12.647 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.090511+0000", 2024-03-19T11:32:12.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014887200000000001 2024-03-19T11:32:12.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.647 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.647 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:12.647 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.140479+0000", 2024-03-19T11:32:12.647 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.049968006000000002 2024-03-19T11:32:12.647 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.648 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:12.648 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.140534+0000", 2024-03-19T11:32:12.648 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4917000000000003e-05 2024-03-19T11:32:12.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.648 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:12.648 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.140626+0000", 2024-03-19T11:32:12.648 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2027999999999994e-05 2024-03-19T11:32:12.648 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.648 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:12.648 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.648 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.648 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.648 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:2441 4.5 4:a38b9453:::smithi03829856-37:head [delete] snapc 17=[17,16,15,13,12,b,6] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e84)", 2024-03-19T11:32:12.649 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:03.060663+0000", 2024-03-19T11:32:12.649 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 69.571213549999996, 2024-03-19T11:32:12.649 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.10917726, 2024-03-19T11:32:12.649 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:12.649 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:12.649 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:12.649 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:32:12.649 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:12.649 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 2441 2024-03-19T11:32:12.649 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.649 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:12.649 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.649 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:12.649 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060663+0000", 2024-03-19T11:32:12.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.650 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.650 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:12.650 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060663+0000", 2024-03-19T11:32:12.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.650 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.650 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:12.650 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060666+0000", 2024-03-19T11:32:12.650 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.05e-06 2024-03-19T11:32:12.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.650 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.650 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:12.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060680+0000", 2024-03-19T11:32:12.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4276e-05 2024-03-19T11:32:12.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.651 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:12.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060714+0000", 2024-03-19T11:32:12.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4325000000000001e-05 2024-03-19T11:32:12.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.651 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:12.651 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060737+0000", 2024-03-19T11:32:12.651 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2959000000000002e-05 2024-03-19T11:32:12.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.651 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.652 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.652 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060895+0000", 2024-03-19T11:32:12.652 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000157787 2024-03-19T11:32:12.652 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.652 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.652 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:32:12.652 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.060923+0000", 2024-03-19T11:32:12.652 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8019000000000001e-05 2024-03-19T11:32:12.652 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.652 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.652 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.652 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.066222+0000", 2024-03-19T11:32:12.652 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052989270000000002 2024-03-19T11:32:12.652 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.653 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.653 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for degraded object", 2024-03-19T11:32:12.653 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.070153+0000", 2024-03-19T11:32:12.653 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0039315119999999999 2024-03-19T11:32:12.653 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.653 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.653 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.653 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.147930+0000", 2024-03-19T11:32:12.653 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.077776459000000006 2024-03-19T11:32:12.653 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.653 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.653 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:12.653 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.148079+0000", 2024-03-19T11:32:12.653 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014888 2024-03-19T11:32:12.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.654 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:32:12.654 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.148370+0000", 2024-03-19T11:32:12.654 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00029130999999999997 2024-03-19T11:32:12.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.654 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:32:12.654 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.152531+0000", 2024-03-19T11:32:12.654 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0041605399999999999 2024-03-19T11:32:12.654 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.654 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.654 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:12.654 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.169762+0000", 2024-03-19T11:32:12.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017231483999999998 2024-03-19T11:32:12.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:12.655 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.169807+0000", 2024-03-19T11:32:12.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5161999999999999e-05 2024-03-19T11:32:12.655 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.655 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.655 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:12.655 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.169840+0000", 2024-03-19T11:32:12.655 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2570000000000002e-05 2024-03-19T11:32:12.655 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.655 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:12.655 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.656 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPushReply(4.5 84/83 [PushReplyOp(4:a38b9453:::smithi03829856-37:head)])", 2024-03-19T11:32:12.656 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:03.084896+0000", 2024-03-19T11:32:12.656 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 69.546980415999997, 2024-03-19T11:32:12.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.02071104, 2024-03-19T11:32:12.656 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:12.656 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-19T11:32:12.656 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:12.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.656 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:12.656 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.084896+0000", 2024-03-19T11:32:12.656 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.656 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.656 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.657 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:12.657 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.084896+0000", 2024-03-19T11:32:12.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.657 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.657 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.657 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:12.657 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.084897+0000", 2024-03-19T11:32:12.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.686e-06 2024-03-19T11:32:12.657 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.657 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.657 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:12.657 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.084910+0000", 2024-03-19T11:32:12.657 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2948e-05 2024-03-19T11:32:12.657 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.658 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:12.658 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.084936+0000", 2024-03-19T11:32:12.658 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.5982e-05 2024-03-19T11:32:12.658 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.658 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:12.658 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.084955+0000", 2024-03-19T11:32:12.658 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8416000000000001e-05 2024-03-19T11:32:12.658 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.658 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.658 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.658 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.102822+0000", 2024-03-19T11:32:12.658 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017866756000000001 2024-03-19T11:32:12.658 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.659 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:12.659 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:03.105607+0000", 2024-03-19T11:32:12.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0027852520000000002 2024-03-19T11:32:12.659 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.659 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:12.659 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.659 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.659 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.659 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.4.0:540 4.18 e111/86 4:1d58db11:::smithi03829856-18:head v 111'153, mlcod=108'150)", 2024-03-19T11:32:12.659 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:31.238185+0000", 2024-03-19T11:32:12.659 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 41.393691052999998, 2024-03-19T11:32:12.659 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021779390999999999, 2024-03-19T11:32:12.659 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:12.660 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:12.660 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:12.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.660 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:12.660 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.238185+0000", 2024-03-19T11:32:12.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.660 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:12.660 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.238185+0000", 2024-03-19T11:32:12.660 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.660 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.660 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.660 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:12.661 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.238188+0000", 2024-03-19T11:32:12.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4739999999999999e-06 2024-03-19T11:32:12.661 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.661 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:12.661 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.240837+0000", 2024-03-19T11:32:12.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0026495189999999999 2024-03-19T11:32:12.661 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.661 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:12.661 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.240867+0000", 2024-03-19T11:32:12.661 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0363000000000001e-05 2024-03-19T11:32:12.661 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.661 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.661 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:12.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.240885+0000", 2024-03-19T11:32:12.662 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7153e-05 2024-03-19T11:32:12.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.662 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.241029+0000", 2024-03-19T11:32:12.662 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014455499999999999 2024-03-19T11:32:12.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.662 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:12.662 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.241069+0000", 2024-03-19T11:32:12.662 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9409000000000002e-05 2024-03-19T11:32:12.662 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.662 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.663 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:12.663 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.259833+0000", 2024-03-19T11:32:12.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018764306000000001 2024-03-19T11:32:12.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.663 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:12.663 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:31.259964+0000", 2024-03-19T11:32:12.663 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000131612 2024-03-19T11:32:12.663 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.663 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:12.663 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.663 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.663 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.663 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3282 4.5 4:a0a50d04:::smithi03829856-13:head [read 0~2201684,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e112)", 2024-03-19T11:32:12.663 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:32.240600+0000", 2024-03-19T11:32:12.664 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 40.391275849000003, 2024-03-19T11:32:12.664 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.84623566699999997, 2024-03-19T11:32:12.664 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:12.664 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:32:12.664 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:12.664 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:32:12.664 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:12.664 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3282 2024-03-19T11:32:12.664 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.664 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:12.664 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.664 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:12.664 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240600+0000", 2024-03-19T11:32:12.664 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.665 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:12.665 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240600+0000", 2024-03-19T11:32:12.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.665 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:12.665 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240602+0000", 2024-03-19T11:32:12.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0700000000000001e-06 2024-03-19T11:32:12.665 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.665 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.665 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:12.665 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240611+0000", 2024-03-19T11:32:12.665 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.969e-06 2024-03-19T11:32:12.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.666 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:12.666 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240630+0000", 2024-03-19T11:32:12.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8253e-05 2024-03-19T11:32:12.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.666 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:12.666 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240642+0000", 2024-03-19T11:32:12.666 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2666e-05 2024-03-19T11:32:12.666 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.666 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.666 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.241488+0000", 2024-03-19T11:32:12.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000845252 2024-03-19T11:32:12.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:32:12.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.241503+0000", 2024-03-19T11:32:12.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5797000000000001e-05 2024-03-19T11:32:12.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.667 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.667 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.083186+0000", 2024-03-19T11:32:12.667 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.84168299599999996 2024-03-19T11:32:12.667 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.667 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.668 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:12.668 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.083618+0000", 2024-03-19T11:32:12.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000432046 2024-03-19T11:32:12.668 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.668 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.668 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:12.668 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.086836+0000", 2024-03-19T11:32:12.668 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0032176180000000002 2024-03-19T11:32:12.668 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.668 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:12.668 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.668 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.668 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.668 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3283 4.5 4:a0a50d04:::smithi03829856-13:head [sparse-read 0~1151947] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e112)", 2024-03-19T11:32:12.669 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:32.240669+0000", 2024-03-19T11:32:12.669 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 40.391206849, 2024-03-19T11:32:12.669 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.84791886900000002, 2024-03-19T11:32:12.669 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:12.669 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:32:12.669 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:12.669 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:32:12.669 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:12.669 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3283 2024-03-19T11:32:12.669 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.669 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:12.669 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.669 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:12.669 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240669+0000", 2024-03-19T11:32:12.669 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.670 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.670 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.670 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:12.670 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240669+0000", 2024-03-19T11:32:12.670 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.670 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.670 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.670 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:12.670 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240670+0000", 2024-03-19T11:32:12.670 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8199999999999998e-07 2024-03-19T11:32:12.670 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.670 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.670 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:12.670 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240675+0000", 2024-03-19T11:32:12.671 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0849999999999996e-06 2024-03-19T11:32:12.671 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.671 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:12.671 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240688+0000", 2024-03-19T11:32:12.671 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2901000000000001e-05 2024-03-19T11:32:12.671 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.671 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:12.671 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240696+0000", 2024-03-19T11:32:12.671 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2910000000000007e-06 2024-03-19T11:32:12.671 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.671 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.671 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.241573+0000", 2024-03-19T11:32:12.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00087639100000000002 2024-03-19T11:32:12.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:32:12.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.241587+0000", 2024-03-19T11:32:12.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4467999999999999e-05 2024-03-19T11:32:12.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.672 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.087901+0000", 2024-03-19T11:32:12.672 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.84631403500000002 2024-03-19T11:32:12.672 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.672 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.672 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:12.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.088089+0000", 2024-03-19T11:32:12.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000187171 2024-03-19T11:32:12.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.673 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:12.673 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.088588+0000", 2024-03-19T11:32:12.673 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00049964499999999995 2024-03-19T11:32:12.673 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.673 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:12.673 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.673 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.673 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.673 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:3284 4.5 4:a0a50d04:::smithi03829856-13:head [sparse-read 0~1151947] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e112)", 2024-03-19T11:32:12.673 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:32.240710+0000", 2024-03-19T11:32:12.674 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 40.391166552999998, 2024-03-19T11:32:12.674 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.84860797899999996, 2024-03-19T11:32:12.674 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:12.674 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:32:12.674 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:12.674 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:32:12.674 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:12.674 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 3284 2024-03-19T11:32:12.674 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.674 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:12.674 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.674 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:12.674 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240710+0000", 2024-03-19T11:32:12.674 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.674 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.675 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.675 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:12.675 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240710+0000", 2024-03-19T11:32:12.675 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.675 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.675 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.675 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:12.675 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240710+0000", 2024-03-19T11:32:12.675 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9599999999999999e-07 2024-03-19T11:32:12.675 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.675 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.675 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:12.675 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240714+0000", 2024-03-19T11:32:12.675 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8480000000000003e-06 2024-03-19T11:32:12.676 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.676 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.676 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:12.676 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240726+0000", 2024-03-19T11:32:12.676 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.186e-05 2024-03-19T11:32:12.676 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.676 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.676 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:12.676 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.240736+0000", 2024-03-19T11:32:12.676 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0040999999999999e-05 2024-03-19T11:32:12.676 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.676 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.676 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.676 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.241633+0000", 2024-03-19T11:32:12.676 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00089736299999999998 2024-03-19T11:32:12.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.677 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:32:12.677 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:32.241645+0000", 2024-03-19T11:32:12.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1600000000000001e-05 2024-03-19T11:32:12.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.677 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.677 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.088667+0000", 2024-03-19T11:32:12.677 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.84702242800000005 2024-03-19T11:32:12.677 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.677 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.677 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:12.677 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.088837+0000", 2024-03-19T11:32:12.678 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000169949 2024-03-19T11:32:12.678 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.678 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.678 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:12.678 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:33.089318+0000", 2024-03-19T11:32:12.678 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00048019399999999998 2024-03-19T11:32:12.678 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.678 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:12.678 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.678 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.678 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.678 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:4125 4.5 4:a38b9453:::smithi03829856-37:head [write 746310~635349,stat] snapc 29=[27,25,24,22,21,1e,17] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-03-19T11:32:12.678 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:32:01.099163+0000", 2024-03-19T11:32:12.678 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.532712892999999, 2024-03-19T11:32:12.679 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.030764071, 2024-03-19T11:32:12.679 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:12.679 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:12.679 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:12.679 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:32:12.679 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:12.679 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 4125 2024-03-19T11:32:12.679 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.679 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:12.679 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.679 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:12.679 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.099163+0000", 2024-03-19T11:32:12.679 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.679 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.680 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.680 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:12.680 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.099163+0000", 2024-03-19T11:32:12.680 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.680 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.680 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.680 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:12.680 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.099166+0000", 2024-03-19T11:32:12.680 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.9629999999999998e-06 2024-03-19T11:32:12.680 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.680 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.680 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:12.680 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.100575+0000", 2024-03-19T11:32:12.681 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001408982 2024-03-19T11:32:12.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.681 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:12.681 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.100610+0000", 2024-03-19T11:32:12.681 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.4609e-05 2024-03-19T11:32:12.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.681 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:12.681 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.100625+0000", 2024-03-19T11:32:12.681 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4826e-05 2024-03-19T11:32:12.681 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.681 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.682 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.682 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.100759+0000", 2024-03-19T11:32:12.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013436500000000001 2024-03-19T11:32:12.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.682 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.682 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:12.682 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.100947+0000", 2024-03-19T11:32:12.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00018814299999999999 2024-03-19T11:32:12.682 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.682 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.682 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:32:12.682 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.101457+0000", 2024-03-19T11:32:12.682 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00051012699999999998 2024-03-19T11:32:12.683 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.683 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.683 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:12.683 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.125933+0000", 2024-03-19T11:32:12.683 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.024475851 2024-03-19T11:32:12.683 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.683 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.683 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:32:12.683 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.129604+0000", 2024-03-19T11:32:12.683 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036707139999999998 2024-03-19T11:32:12.683 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.683 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.683 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:12.683 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.129774+0000", 2024-03-19T11:32:12.684 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00017009100000000001 2024-03-19T11:32:12.684 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.684 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.684 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:12.684 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.129927+0000", 2024-03-19T11:32:12.684 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015339999999999999 2024-03-19T11:32:12.684 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.684 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:12.684 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.684 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.684 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.684 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:4126 4.5 4:a38b9453:::smithi03829856-37:head [write 2077646~576443] snapc 29=[27,25,24,22,21,1e,17] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-03-19T11:32:12.684 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:32:01.106778+0000", 2024-03-19T11:32:12.685 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.525098288000001, 2024-03-19T11:32:12.685 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.024531504999999999, 2024-03-19T11:32:12.685 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:12.685 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:12.685 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:12.685 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:32:12.685 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:12.685 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 4126 2024-03-19T11:32:12.685 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.685 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:12.685 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.685 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:12.685 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.106778+0000", 2024-03-19T11:32:12.685 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.686 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.686 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.686 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:12.686 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.106778+0000", 2024-03-19T11:32:12.686 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.686 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.686 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.686 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:12.686 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.106781+0000", 2024-03-19T11:32:12.686 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.456e-06 2024-03-19T11:32:12.686 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.686 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.686 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:12.686 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.107491+0000", 2024-03-19T11:32:12.687 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00070938200000000005 2024-03-19T11:32:12.687 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.687 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.687 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:12.687 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.107529+0000", 2024-03-19T11:32:12.687 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.8039e-05 2024-03-19T11:32:12.687 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.687 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.687 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:12.687 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.107551+0000", 2024-03-19T11:32:12.687 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2753000000000001e-05 2024-03-19T11:32:12.687 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.687 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.687 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.688 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.114973+0000", 2024-03-19T11:32:12.688 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0074218569999999996 2024-03-19T11:32:12.688 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.688 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.688 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:12.688 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.115239+0000", 2024-03-19T11:32:12.688 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026613800000000002 2024-03-19T11:32:12.688 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.688 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.688 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:32:12.688 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.115718+0000", 2024-03-19T11:32:12.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000478708 2024-03-19T11:32:12.689 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.689 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.689 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:32:12.689 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.129957+0000", 2024-03-19T11:32:12.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.014239069 2024-03-19T11:32:12.689 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.689 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.689 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:12.689 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.131069+0000", 2024-03-19T11:32:12.689 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001111534 2024-03-19T11:32:12.689 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.689 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.690 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:12.690 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.131203+0000", 2024-03-19T11:32:12.690 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013444100000000001 2024-03-19T11:32:12.690 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.690 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.690 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:12.690 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.131309+0000", 2024-03-19T11:32:12.690 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000106128 2024-03-19T11:32:12.690 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.690 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:12.690 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.690 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.691 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.691 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:4128 4.5 4:a38b9453:::smithi03829856-37:head [setxattr _header (57) in=7b,truncate 3546310] snapc 29=[27,25,24,22,21,1e,17] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-03-19T11:32:12.691 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:32:01.112446+0000", 2024-03-19T11:32:12.691 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.519430011000001, 2024-03-19T11:32:12.691 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019934741999999998, 2024-03-19T11:32:12.691 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:12.691 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:12.691 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:12.691 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:32:12.691 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:12.691 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 4128 2024-03-19T11:32:12.691 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.691 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:12.691 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.691 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:12.692 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.112446+0000", 2024-03-19T11:32:12.692 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.692 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.692 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.692 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:12.692 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.112446+0000", 2024-03-19T11:32:12.692 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.692 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.692 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.692 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:12.692 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.112448+0000", 2024-03-19T11:32:12.692 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5829999999999999e-06 2024-03-19T11:32:12.692 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.692 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.692 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:12.693 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.112457+0000", 2024-03-19T11:32:12.693 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.6369999999999994e-06 2024-03-19T11:32:12.693 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.693 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.693 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:12.693 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.112480+0000", 2024-03-19T11:32:12.693 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3014e-05 2024-03-19T11:32:12.693 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.693 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.693 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:12.693 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.112495+0000", 2024-03-19T11:32:12.693 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4596999999999999e-05 2024-03-19T11:32:12.693 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.694 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.694 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.694 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.127994+0000", 2024-03-19T11:32:12.694 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015499092000000001 2024-03-19T11:32:12.694 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.694 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.694 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:12.694 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.128282+0000", 2024-03-19T11:32:12.694 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00028747199999999999 2024-03-19T11:32:12.694 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.694 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.694 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:32:12.694 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.128807+0000", 2024-03-19T11:32:12.694 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00052564099999999998 2024-03-19T11:32:12.695 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.695 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.695 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:32:12.695 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.130396+0000", 2024-03-19T11:32:12.695 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001588922 2024-03-19T11:32:12.695 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.695 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.695 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:12.695 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.132196+0000", 2024-03-19T11:32:12.695 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00179981 2024-03-19T11:32:12.695 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.695 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.695 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:12.695 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.132283+0000", 2024-03-19T11:32:12.695 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6966999999999996e-05 2024-03-19T11:32:12.696 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.696 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.696 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:12.696 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.132381+0000", 2024-03-19T11:32:12.696 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.8006999999999999e-05 2024-03-19T11:32:12.696 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.696 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:12.696 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.696 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.696 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.696 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:4129 4.5 4:a38b9453:::smithi03829856-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e140)", 2024-03-19T11:32:12.696 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:32:01.112516+0000", 2024-03-19T11:32:12.696 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 11.519360123, 2024-03-19T11:32:12.696 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020215219, 2024-03-19T11:32:12.697 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:12.697 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:32:12.697 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:12.697 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:32:12.697 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:12.697 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 4129 2024-03-19T11:32:12.697 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.697 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:12.697 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.697 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:12.697 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.112516+0000", 2024-03-19T11:32:12.697 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.697 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.697 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.697 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:12.698 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.112516+0000", 2024-03-19T11:32:12.698 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.698 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.698 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.698 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:12.698 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.112517+0000", 2024-03-19T11:32:12.698 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.342e-06 2024-03-19T11:32:12.698 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.698 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.698 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:12.698 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.112524+0000", 2024-03-19T11:32:12.698 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.0230000000000004e-06 2024-03-19T11:32:12.698 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.698 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.699 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:12.699 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.112543+0000", 2024-03-19T11:32:12.699 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9026e-05 2024-03-19T11:32:12.699 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.699 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.699 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:12.699 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.112557+0000", 2024-03-19T11:32:12.699 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4011e-05 2024-03-19T11:32:12.699 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.699 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.699 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.699 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.130091+0000", 2024-03-19T11:32:12.699 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017533863 2024-03-19T11:32:12.699 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.700 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.700 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for rw locks", 2024-03-19T11:32:12.700 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.130332+0000", 2024-03-19T11:32:12.700 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00024066199999999999 2024-03-19T11:32:12.700 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.700 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.700 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.700 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.132445+0000", 2024-03-19T11:32:12.700 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0021125570000000001 2024-03-19T11:32:12.700 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.700 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.700 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:12.700 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.132570+0000", 2024-03-19T11:32:12.700 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012532900000000001 2024-03-19T11:32:12.700 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.701 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.701 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:12.701 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:01.132731+0000", 2024-03-19T11:32:12.701 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016140600000000001 2024-03-19T11:32:12.701 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.701 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:12.701 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.701 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.701 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.701 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:4319 4.5 4:a0a50d04:::smithi03829856-13:head [write 632029~547288] snapc 2b=[2b,2a,27,21] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-03-19T11:32:12.701 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:32:10.114755+0000", 2024-03-19T11:32:12.701 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 2.5171212069999997, 2024-03-19T11:32:12.701 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.028558534, 2024-03-19T11:32:12.701 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:12.701 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:12.702 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:12.702 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:32:12.702 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:12.702 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 4319 2024-03-19T11:32:12.702 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.702 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:12.702 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.702 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:12.702 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:10.114755+0000", 2024-03-19T11:32:12.702 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.702 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.702 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.702 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:12.702 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:10.114755+0000", 2024-03-19T11:32:12.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.703 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.703 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.703 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:12.703 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:10.114759+0000", 2024-03-19T11:32:12.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.5899999999999999e-06 2024-03-19T11:32:12.703 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.703 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.703 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:12.703 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:10.115817+0000", 2024-03-19T11:32:12.703 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001058051 2024-03-19T11:32:12.703 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.703 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.703 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:12.704 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:10.115856+0000", 2024-03-19T11:32:12.704 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9771999999999998e-05 2024-03-19T11:32:12.704 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.704 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.704 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:12.704 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:10.115877+0000", 2024-03-19T11:32:12.704 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0503999999999999e-05 2024-03-19T11:32:12.704 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.704 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.704 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.704 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:10.116081+0000", 2024-03-19T11:32:12.704 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020387000000000001 2024-03-19T11:32:12.704 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.704 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.705 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op must wait for map", 2024-03-19T11:32:12.705 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:10.116104+0000", 2024-03-19T11:32:12.705 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3e-05 2024-03-19T11:32:12.705 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.705 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.705 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.705 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:10.120181+0000", 2024-03-19T11:32:12.705 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.004077397 2024-03-19T11:32:12.705 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.705 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.705 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:12.705 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:10.120338+0000", 2024-03-19T11:32:12.705 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000156451 2024-03-19T11:32:12.705 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.706 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.706 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:32:12.706 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:10.120673+0000", 2024-03-19T11:32:12.706 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033567100000000001 2024-03-19T11:32:12.706 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.706 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.706 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:12.706 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:10.139010+0000", 2024-03-19T11:32:12.706 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018336837000000002 2024-03-19T11:32:12.706 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.706 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.706 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:32:12.706 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:10.142616+0000", 2024-03-19T11:32:12.707 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0036059439999999998 2024-03-19T11:32:12.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.707 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:12.707 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:10.142878+0000", 2024-03-19T11:32:12.707 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026202300000000001 2024-03-19T11:32:12.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.707 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.707 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:12.707 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:10.143313+0000", 2024-03-19T11:32:12.707 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00043542400000000001 2024-03-19T11:32:12.707 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.707 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:12.707 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.708 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.708 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:4320 4.5 4:a0a50d04:::smithi03829856-13:head [write 1639134~749453] snapc 2b=[2b,2a,27,21] ondisk+write+known_if_redirected+supports_pool_eio e149)", 2024-03-19T11:32:12.708 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:32:10.121685+0000", 2024-03-19T11:32:12.708 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 2.5101908420000001, 2024-03-19T11:32:12.708 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023934581, 2024-03-19T11:32:12.708 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:12.708 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:12.708 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:12.708 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:32:12.708 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:12.708 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 4320 2024-03-19T11:32:12.708 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.708 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:12.709 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.709 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:12.709 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:10.121685+0000", 2024-03-19T11:32:12.709 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.709 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.709 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.709 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:12.709 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:10.121685+0000", 2024-03-19T11:32:12.709 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.709 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.709 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.709 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:12.709 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:10.121689+0000", 2024-03-19T11:32:12.709 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.9670000000000003e-06 2024-03-19T11:32:12.710 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.710 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:12.710 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:10.122806+0000", 2024-03-19T11:32:12.710 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001116286 2024-03-19T11:32:12.710 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.710 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:12.710 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:10.122843+0000", 2024-03-19T11:32:12.710 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.7382999999999998e-05 2024-03-19T11:32:12.710 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.710 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.710 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:12.710 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:10.122866+0000", 2024-03-19T11:32:12.711 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3295999999999999e-05 2024-03-19T11:32:12.711 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.711 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.711 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.711 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:10.128533+0000", 2024-03-19T11:32:12.711 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0056666370000000004 2024-03-19T11:32:12.711 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.711 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.711 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:12.711 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:10.128860+0000", 2024-03-19T11:32:12.711 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00032726499999999998 2024-03-19T11:32:12.711 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.712 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.712 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 4", 2024-03-19T11:32:12.712 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:10.129355+0000", 2024-03-19T11:32:12.712 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00049517300000000001 2024-03-19T11:32:12.712 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.712 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.713 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:32:12.713 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:10.144712+0000", 2024-03-19T11:32:12.713 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015356724 2024-03-19T11:32:12.713 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.713 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.713 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:12.713 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:10.145307+0000", 2024-03-19T11:32:12.713 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00059544100000000005 2024-03-19T11:32:12.713 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.713 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.713 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:12.713 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:10.145477+0000", 2024-03-19T11:32:12.713 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016946800000000001 2024-03-19T11:32:12.713 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.714 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.714 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:12.714 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:10.145620+0000", 2024-03-19T11:32:12.714 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000142941 2024-03-19T11:32:12.714 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.714 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:12.714 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.714 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.714 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.714 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:4336 4.5 4.ca29d1c5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e150)", 2024-03-19T11:32:12.714 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:32:11.134695+0000", 2024-03-19T11:32:12.714 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.497181235, 2024-03-19T11:32:12.714 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0029021659999999, 2024-03-19T11:32:12.714 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:12.715 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-19T11:32:12.715 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:12.715 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:32:12.715 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:12.715 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 4336 2024-03-19T11:32:12.715 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.715 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:12.715 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.715 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:12.715 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:11.134695+0000", 2024-03-19T11:32:12.715 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.715 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.715 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.715 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:12.715 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:11.134695+0000", 2024-03-19T11:32:12.716 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.716 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.716 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.716 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:12.716 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:11.134698+0000", 2024-03-19T11:32:12.716 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.8030000000000001e-06 2024-03-19T11:32:12.716 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.716 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.716 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:12.716 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:11.134713+0000", 2024-03-19T11:32:12.716 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4980000000000001e-05 2024-03-19T11:32:12.716 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.716 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.716 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:12.717 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:11.134745+0000", 2024-03-19T11:32:12.717 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2276000000000001e-05 2024-03-19T11:32:12.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.717 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.717 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:12.717 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:11.134762+0000", 2024-03-19T11:32:12.717 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6592999999999999e-05 2024-03-19T11:32:12.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.717 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.717 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.717 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:11.134868+0000", 2024-03-19T11:32:12.717 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000105989 2024-03-19T11:32:12.717 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.717 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.718 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:32:12.718 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:11.134884+0000", 2024-03-19T11:32:12.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6172999999999999e-05 2024-03-19T11:32:12.718 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.718 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.718 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.718 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:12.137557+0000", 2024-03-19T11:32:12.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0026728570000001 2024-03-19T11:32:12.718 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.718 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.718 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:12.718 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:12.137597+0000", 2024-03-19T11:32:12.718 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.0494999999999998e-05 2024-03-19T11:32:12.718 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.719 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:12.719 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.719 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.719 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.719 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:4337 4.5 4.ca29d1c5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e150)", 2024-03-19T11:32:12.719 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:32:11.134801+0000", 2024-03-19T11:32:12.719 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.4970751359999999, 2024-03-19T11:32:12.719 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0028507799999999, 2024-03-19T11:32:12.719 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:12.719 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-19T11:32:12.719 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:12.719 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:32:12.719 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:12.719 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 4337 2024-03-19T11:32:12.720 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.720 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:12.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.720 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:12.720 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:11.134801+0000", 2024-03-19T11:32:12.720 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.720 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.720 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:12.720 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:11.134801+0000", 2024-03-19T11:32:12.720 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.720 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.720 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.720 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:12.721 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:11.134802+0000", 2024-03-19T11:32:12.721 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.423e-06 2024-03-19T11:32:12.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.721 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.721 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:12.721 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:11.134812+0000", 2024-03-19T11:32:12.721 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2730000000000008e-06 2024-03-19T11:32:12.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.721 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.721 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:12.721 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:11.134833+0000", 2024-03-19T11:32:12.721 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1443000000000001e-05 2024-03-19T11:32:12.721 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.722 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.722 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:12.722 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:11.134848+0000", 2024-03-19T11:32:12.722 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5075999999999999e-05 2024-03-19T11:32:12.722 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.722 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.722 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.722 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:11.134970+0000", 2024-03-19T11:32:12.722 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00012152200000000001 2024-03-19T11:32:12.722 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.722 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.722 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:32:12.722 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:11.134988+0000", 2024-03-19T11:32:12.722 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8525999999999998e-05 2024-03-19T11:32:12.723 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.723 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.723 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.723 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:12.137640+0000", 2024-03-19T11:32:12.723 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.00265216 2024-03-19T11:32:12.723 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.723 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.723 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:12.723 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:12.137652+0000", 2024-03-19T11:32:12.723 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1357e-05 2024-03-19T11:32:12.723 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.723 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:12.723 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.723 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.723 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.724 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4415.0:4338 4.5 4.ca29d1c5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e150)", 2024-03-19T11:32:12.724 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:32:11.134875+0000", 2024-03-19T11:32:12.724 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 1.4970016180000001, 2024-03-19T11:32:12.724 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.002823945, 2024-03-19T11:32:12.724 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:12.724 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "reached pg", 2024-03-19T11:32:12.724 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:12.724 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4415", 2024-03-19T11:32:12.724 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.38:0/306659979", 2024-03-19T11:32:12.724 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 4338 2024-03-19T11:32:12.724 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.724 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:12.724 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.724 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:12.724 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:11.134875+0000", 2024-03-19T11:32:12.725 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.725 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.725 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.725 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:12.725 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:11.134875+0000", 2024-03-19T11:32:12.725 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:12.725 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.725 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.725 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:12.725 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:11.134876+0000", 2024-03-19T11:32:12.725 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4100000000000001e-06 2024-03-19T11:32:12.725 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.725 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.725 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:12.725 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:11.134885+0000", 2024-03-19T11:32:12.726 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9770000000000006e-06 2024-03-19T11:32:12.726 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.726 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.726 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:12.726 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:11.134903+0000", 2024-03-19T11:32:12.726 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8499e-05 2024-03-19T11:32:12.726 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.726 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.726 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:12.726 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:11.134919+0000", 2024-03-19T11:32:12.726 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5270000000000001e-05 2024-03-19T11:32:12.726 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.726 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.726 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.727 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:11.135041+0000", 2024-03-19T11:32:12.727 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000122402 2024-03-19T11:32:12.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.727 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:32:12.727 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:11.135062+0000", 2024-03-19T11:32:12.727 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0472999999999999e-05 2024-03-19T11:32:12.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.727 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:12.727 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:12.137689+0000", 2024-03-19T11:32:12.727 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.002627741 2024-03-19T11:32:12.727 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:12.727 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:12.728 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:12.728 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:32:12.137698+0000", 2024-03-19T11:32:12.728 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.1730000000000001e-06 2024-03-19T11:32:12.728 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.728 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:12.728 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.728 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:12.728 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:12.728 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:32:12.730 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-19T11:32:12.730 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:32:12.739+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:32:12.830 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 389, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 507, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1362, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 844, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 799, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1196, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2024-03-19T11:32:12.898 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:32:12.898 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:32:12.898 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:32:12.898 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:32:12.912 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-19T11:32:12.956 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:32:12.965+0000 7f1446eee640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:32:13.029 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:32:13.029 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:32:13.029 INFO:teuthology.orchestra.run.smithi172.stdout: "complaint_time": 30, 2024-03-19T11:32:13.029 INFO:teuthology.orchestra.run.smithi172.stdout: "num_blocked_ops": 0 2024-03-19T11:32:13.029 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:32:13.034 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:32:13.041+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:32:13.043 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-19T11:32:13.058 INFO:teuthology.orchestra.run.smithi038.stderr:marked out osd.7. 2024-03-19T11:32:13.059 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-03-19T11:32:13.073 INFO:tasks.rados.rados.0.smithi038.stdout:1856: done (6 left) 2024-03-19T11:32:13.073 INFO:tasks.rados.rados.0.smithi038.stdout:1857: done (5 left) 2024-03-19T11:32:13.074 INFO:tasks.rados.rados.0.smithi038.stdout:1859: done (4 left) 2024-03-19T11:32:13.074 INFO:tasks.rados.rados.0.smithi038.stdout:1860: done (3 left) 2024-03-19T11:32:13.074 INFO:tasks.rados.rados.0.smithi038.stdout:1861: read oid 12 snap -1 2024-03-19T11:32:13.074 INFO:tasks.rados.rados.0.smithi038.stdout:1861: expect (ObjNum 325 snap 40 seq_num 325) 2024-03-19T11:32:13.074 INFO:tasks.rados.rados.0.smithi038.stdout:1862: read oid 49 snap 41 2024-03-19T11:32:13.075 INFO:tasks.rados.rados.0.smithi038.stdout:1862: expect (ObjNum 334 snap 40 seq_num 334) 2024-03-19T11:32:13.075 INFO:tasks.rados.rados.0.smithi038.stdout:1863: read oid 47 snap -1 2024-03-19T11:32:13.075 INFO:tasks.rados.rados.0.smithi038.stdout:1863: expect deleted 2024-03-19T11:32:13.075 INFO:tasks.rados.rados.0.smithi038.stdout:1864: rollback oid 20 current snap is 45 2024-03-19T11:32:13.075 INFO:tasks.rados.rados.0.smithi038.stdout:rollback oid 20 to 40 2024-03-19T11:32:13.075 INFO:tasks.rados.rados.0.smithi038.stdout:1865: set_chunk oid 5 target oid 2024-03-19T11:32:13.097 INFO:tasks.rados.rados.0.smithi038.stdout:1865: set_chunk oid 5 offset: 1286144 length: 2026496 target oid ad8aba8dae359c866029127c4b6f72215b909f96cae5cb3b2c3af04f48aefc6b offset: 0 snap -1 2024-03-19T11:32:13.098 INFO:tasks.rados.rados.0.smithi038.stdout:1866: read oid 32 snap -1 2024-03-19T11:32:13.098 INFO:tasks.rados.rados.0.smithi038.stdout:1866: expect (ObjNum 342 snap 40 seq_num 342) 2024-03-19T11:32:13.098 INFO:tasks.rados.rados.0.smithi038.stdout:1862: expect (ObjNum 334 snap 40 seq_num 334) 2024-03-19T11:32:13.149 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:32:13.159+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:32:13.221 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:32:13.221 INFO:teuthology.orchestra.run.smithi172.stdout: "size": 20, 2024-03-19T11:32:13.221 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 600, 2024-03-19T11:32:13.221 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [ 2024-03-19T11:32:13.221 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.221 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-03-19T11:32:13.221 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:44.200920+0000", 2024-03-19T11:32:13.222 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 149.02757994000001, 2024-03-19T11:32:13.222 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.80881896499999995, 2024-03-19T11:32:13.222 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:13.222 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:13.222 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:13.222 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:32:13.223 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:32:13.223 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 1 2024-03-19T11:32:13.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.223 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:13.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.223 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:13.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200920+0000", 2024-03-19T11:32:13.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:13.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.223 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.223 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:13.223 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200920+0000", 2024-03-19T11:32:13.223 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:13.223 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.224 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:13.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200923+0000", 2024-03-19T11:32:13.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6469999999999998e-06 2024-03-19T11:32:13.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.224 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:13.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200937+0000", 2024-03-19T11:32:13.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.399e-05 2024-03-19T11:32:13.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.224 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.224 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:13.224 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200960+0000", 2024-03-19T11:32:13.224 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2603e-05 2024-03-19T11:32:13.224 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:13.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.200978+0000", 2024-03-19T11:32:13.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.791e-05 2024-03-19T11:32:13.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:13.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.201081+0000", 2024-03-19T11:32:13.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00010344900000000001 2024-03-19T11:32:13.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.225 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.225 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for peered", 2024-03-19T11:32:13.225 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:44.201096+0000", 2024-03-19T11:32:13.225 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.4443e-05 2024-03-19T11:32:13.225 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:13.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.005978+0000", 2024-03-19T11:32:13.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.80488281500000003 2024-03-19T11:32:13.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:13.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.006267+0000", 2024-03-19T11:32:13.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000288688 2024-03-19T11:32:13.226 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.226 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.226 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:32:13.226 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.006534+0000", 2024-03-19T11:32:13.226 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00026725799999999998 2024-03-19T11:32:13.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.227 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:32:13.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.008457+0000", 2024-03-19T11:32:13.227 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0019227820000000001 2024-03-19T11:32:13.227 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.227 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.227 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:13.227 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.009570+0000", 2024-03-19T11:32:13.230 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0011131750000000001 2024-03-19T11:32:13.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.231 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:13.231 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.009670+0000", 2024-03-19T11:32:13.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9458999999999998e-05 2024-03-19T11:32:13.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.231 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.231 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:13.231 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.009739+0000", 2024-03-19T11:32:13.231 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.9746000000000006e-05 2024-03-19T11:32:13.231 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:13.231 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:13.231 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:13.231 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.232 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.232 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:32:13.232 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.339898+0000", 2024-03-19T11:32:13.232 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 147.88860283299999, 2024-03-19T11:32:13.232 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019501793, 2024-03-19T11:32:13.232 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:13.232 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:13.232 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:13.232 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:32:13.232 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:32:13.232 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 113 2024-03-19T11:32:13.233 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.233 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:13.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.233 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:13.233 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339898+0000", 2024-03-19T11:32:13.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:13.233 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.233 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:13.233 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339898+0000", 2024-03-19T11:32:13.233 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:13.233 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.233 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.233 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:13.233 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339899+0000", 2024-03-19T11:32:13.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.353e-06 2024-03-19T11:32:13.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.234 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.234 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:13.234 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339907+0000", 2024-03-19T11:32:13.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.3000000000000002e-06 2024-03-19T11:32:13.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.234 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.234 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:13.234 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339924+0000", 2024-03-19T11:32:13.234 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6906999999999999e-05 2024-03-19T11:32:13.234 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:13.235 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339933+0000", 2024-03-19T11:32:13.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8969999999999997e-06 2024-03-19T11:32:13.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:13.235 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352239+0000", 2024-03-19T11:32:13.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.012305814999999999 2024-03-19T11:32:13.235 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.235 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:13.235 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352466+0000", 2024-03-19T11:32:13.235 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022684100000000001 2024-03-19T11:32:13.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.236 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:32:13.236 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.352816+0000", 2024-03-19T11:32:13.236 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00035030499999999997 2024-03-19T11:32:13.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.236 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:32:13.236 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.358956+0000", 2024-03-19T11:32:13.236 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0061400079999999998 2024-03-19T11:32:13.236 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.236 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:13.236 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359118+0000", 2024-03-19T11:32:13.236 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00016237599999999999 2024-03-19T11:32:13.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:13.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359287+0000", 2024-03-19T11:32:13.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000168488 2024-03-19T11:32:13.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.237 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:13.237 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359399+0000", 2024-03-19T11:32:13.237 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000112503 2024-03-19T11:32:13.237 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:13.237 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:13.237 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:13.237 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.238 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:32:13.238 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.339947+0000", 2024-03-19T11:32:13.238 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 147.88855306599999, 2024-03-19T11:32:13.238 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021923069999999999, 2024-03-19T11:32:13.238 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:13.238 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:13.238 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:13.238 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:32:13.238 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:32:13.238 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 114 2024-03-19T11:32:13.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.238 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:13.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.238 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:13.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339947+0000", 2024-03-19T11:32:13.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:13.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:13.239 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339947+0000", 2024-03-19T11:32:13.239 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:13.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.239 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.239 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:13.240 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339948+0000", 2024-03-19T11:32:13.240 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6000000000000003e-07 2024-03-19T11:32:13.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:13.241 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339956+0000", 2024-03-19T11:32:13.241 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.1559999999999995e-06 2024-03-19T11:32:13.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.241 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.241 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:13.242 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339967+0000", 2024-03-19T11:32:13.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0669999999999999e-05 2024-03-19T11:32:13.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.242 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:13.242 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.339975+0000", 2024-03-19T11:32:13.242 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0099999999999995e-06 2024-03-19T11:32:13.242 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.242 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.242 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:13.242 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.353672+0000", 2024-03-19T11:32:13.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.013696583999999999 2024-03-19T11:32:13.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.243 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:13.243 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.353808+0000", 2024-03-19T11:32:13.243 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013652299999999999 2024-03-19T11:32:13.243 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.243 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.243 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:32:13.243 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.354036+0000", 2024-03-19T11:32:13.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00022744799999999999 2024-03-19T11:32:13.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:32:13.244 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.360480+0000", 2024-03-19T11:32:13.244 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0064441630000000001 2024-03-19T11:32:13.244 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.244 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.244 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:13.245 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361722+0000", 2024-03-19T11:32:13.245 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.001242754 2024-03-19T11:32:13.245 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.245 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:13.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361807+0000", 2024-03-19T11:32:13.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.4605999999999996e-05 2024-03-19T11:32:13.246 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.246 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.246 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:13.246 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361870+0000", 2024-03-19T11:32:13.246 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3396000000000001e-05 2024-03-19T11:32:13.246 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:13.246 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:13.246 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:13.247 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.247 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.247 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:32:13.247 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.340017+0000", 2024-03-19T11:32:13.247 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 147.88848375000001, 2024-03-19T11:32:13.247 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022169355000000002, 2024-03-19T11:32:13.247 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:13.247 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:13.247 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:13.247 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:32:13.247 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:32:13.248 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 115 2024-03-19T11:32:13.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.248 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:13.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.248 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:13.248 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340017+0000", 2024-03-19T11:32:13.248 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:13.248 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.248 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.249 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:13.249 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340017+0000", 2024-03-19T11:32:13.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:13.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.250 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:13.250 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340018+0000", 2024-03-19T11:32:13.250 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.043e-06 2024-03-19T11:32:13.250 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.250 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.251 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:13.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340139+0000", 2024-03-19T11:32:13.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000121502 2024-03-19T11:32:13.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.253 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:13.253 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340157+0000", 2024-03-19T11:32:13.253 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7382999999999999e-05 2024-03-19T11:32:13.253 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.253 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.254 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:13.254 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340166+0000", 2024-03-19T11:32:13.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.2469999999999997e-06 2024-03-19T11:32:13.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.254 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:13.254 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.357470+0000", 2024-03-19T11:32:13.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.017304119999999999 2024-03-19T11:32:13.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.254 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.254 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:13.254 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.357671+0000", 2024-03-19T11:32:13.254 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000200798 2024-03-19T11:32:13.254 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.255 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.255 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:32:13.255 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.358002+0000", 2024-03-19T11:32:13.256 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00033090099999999998 2024-03-19T11:32:13.256 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.256 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.256 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:32:13.256 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361892+0000", 2024-03-19T11:32:13.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00389018 2024-03-19T11:32:13.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:13.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361983+0000", 2024-03-19T11:32:13.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.0976000000000004e-05 2024-03-19T11:32:13.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:13.257 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362071+0000", 2024-03-19T11:32:13.257 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.8022999999999994e-05 2024-03-19T11:32:13.257 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.257 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.257 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:13.258 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362186+0000", 2024-03-19T11:32:13.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000115182 2024-03-19T11:32:13.258 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:13.258 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:13.258 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:13.258 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.258 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.258 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:32:13.258 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.340183+0000", 2024-03-19T11:32:13.258 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 147.88831691799999, 2024-03-19T11:32:13.258 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022279841000000002, 2024-03-19T11:32:13.258 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:13.258 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:13.258 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:13.259 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:32:13.259 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:32:13.259 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 116 2024-03-19T11:32:13.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.259 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:13.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:13.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340183+0000", 2024-03-19T11:32:13.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:13.259 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.259 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.259 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:13.259 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340183+0000", 2024-03-19T11:32:13.259 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:13.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:13.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340184+0000", 2024-03-19T11:32:13.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.9000000000000005e-07 2024-03-19T11:32:13.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:13.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340191+0000", 2024-03-19T11:32:13.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0360000000000003e-06 2024-03-19T11:32:13.260 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.260 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.260 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:13.260 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340201+0000", 2024-03-19T11:32:13.260 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0844e-05 2024-03-19T11:32:13.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.261 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:13.261 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340209+0000", 2024-03-19T11:32:13.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7360000000000001e-06 2024-03-19T11:32:13.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.261 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:13.261 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359535+0000", 2024-03-19T11:32:13.261 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019325546999999998 2024-03-19T11:32:13.261 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.261 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.262 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:13.262 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359672+0000", 2024-03-19T11:32:13.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00013706600000000001 2024-03-19T11:32:13.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.262 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:32:13.262 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.359905+0000", 2024-03-19T11:32:13.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023334500000000001 2024-03-19T11:32:13.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.262 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.262 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:13.262 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362306+0000", 2024-03-19T11:32:13.262 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002401041 2024-03-19T11:32:13.262 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.263 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:32:13.263 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362329+0000", 2024-03-19T11:32:13.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3283999999999999e-05 2024-03-19T11:32:13.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.263 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:13.263 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362402+0000", 2024-03-19T11:32:13.263 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.2459000000000005e-05 2024-03-19T11:32:13.263 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.263 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.263 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:13.263 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362463+0000", 2024-03-19T11:32:13.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.1493000000000002e-05 2024-03-19T11:32:13.264 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:13.264 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:13.264 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:13.264 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.264 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.264 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_op(client.4201.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-03-19T11:32:13.264 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:29:45.340222+0000", 2024-03-19T11:32:13.264 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 147.888278335, 2024-03-19T11:32:13.264 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022500218999999998, 2024-03-19T11:32:13.264 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:13.264 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:13.264 INFO:teuthology.orchestra.run.smithi172.stdout: "client_info": { 2024-03-19T11:32:13.264 INFO:teuthology.orchestra.run.smithi172.stdout: "client": "client.4201", 2024-03-19T11:32:13.265 INFO:teuthology.orchestra.run.smithi172.stdout: "client_addr": "172.21.15.172:0/3321583434", 2024-03-19T11:32:13.265 INFO:teuthology.orchestra.run.smithi172.stdout: "tid": 117 2024-03-19T11:32:13.265 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.265 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:13.265 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.265 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:13.265 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340222+0000", 2024-03-19T11:32:13.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:13.265 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.265 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.265 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:13.265 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340222+0000", 2024-03-19T11:32:13.265 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:13.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.266 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:13.266 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340223+0000", 2024-03-19T11:32:13.266 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7999999999999995e-07 2024-03-19T11:32:13.266 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.266 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.266 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:13.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340229+0000", 2024-03-19T11:32:13.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0399999999999998e-06 2024-03-19T11:32:13.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.267 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:13.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340239+0000", 2024-03-19T11:32:13.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0190000000000001e-05 2024-03-19T11:32:13.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.267 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.267 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:13.267 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.340250+0000", 2024-03-19T11:32:13.267 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0720000000000001e-05 2024-03-19T11:32:13.267 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.268 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.268 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:13.268 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.360587+0000", 2024-03-19T11:32:13.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020337542 2024-03-19T11:32:13.268 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.268 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.268 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:13.268 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.360818+0000", 2024-03-19T11:32:13.268 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00023120500000000001 2024-03-19T11:32:13.268 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.268 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.269 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "waiting for subops from 0", 2024-03-19T11:32:13.269 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.361218+0000", 2024-03-19T11:32:13.269 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000399637 2024-03-19T11:32:13.269 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.269 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.269 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "op_commit", 2024-03-19T11:32:13.269 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362483+0000", 2024-03-19T11:32:13.269 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012649200000000001 2024-03-19T11:32:13.269 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.269 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.269 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "sub_op_commit_rec", 2024-03-19T11:32:13.269 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362538+0000", 2024-03-19T11:32:13.269 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.4490000000000002e-05 2024-03-19T11:32:13.269 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.270 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.270 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:13.270 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362617+0000", 2024-03-19T11:32:13.270 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.9839000000000006e-05 2024-03-19T11:32:13.270 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.270 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.270 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:13.270 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:29:45.362722+0000", 2024-03-19T11:32:13.270 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000104956 2024-03-19T11:32:13.270 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:13.270 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:13.271 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:13.271 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.271 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.271 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.15 51/50 [PushOp(4:afa9f208:::smithi03829856-35:head, version: 28'46, data_included: [571438~623878,1806623~424698], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:afa9f208:::smithi03829856-35:head@28'46, size: 2571438, copy_subset: [(0, 2571438)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2231321, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:af9d0d56:::smithi03829856-20:head, version: 35'55, data_included: [655360~589824,1900544~418789], data_size: 1008613, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:af9d0d56:::smithi03829856-20:head@35'55, size: 2319333, copy_subset: [(0, 2319333)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2319333, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ad3ad1b3:::smithi03829856-21:b, version: 47'66, data_included: [736140~537686,2020581~315559], data_size: 853245, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:ad3ad1b3:::smithi03829856-21:b@47'66, size: 2336140, copy_subset: [(0, 2336140)], clone_subset: {}, snapset: b=[]:{b: [b, a, 6, 5]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2336140, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ad3ad1b3:::smithi03829856-21:head, version: 47'70, data_included: [531121~742705,1711057~305871], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:ad3ad1b3:::smithi03829856-21:head@47'70, size: 3731121, copy_subset: [(0, 3731121)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2016928, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:ac9cd3a3:::smithi03829856-8:head, version: 49'72, data_included: [517649~620182,1776668~428394], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:ac9cd3a3:::smithi03829856-8:head@49'72, size: 3008420, copy_subset: [(0, 3008420)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2205062, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:aa3303a5:::smithi03829856-17:b, version: 49'73, data_included: [614400~692224,1903006~356352], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:aa3303a5:::smithi03829856-17:b@49'73, size: 2268912, copy_subset: [(0, 2268912)], clone_subset: {}, snapset: b=[]:{b: [b, a, 6, 5]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2259358, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:aa3303a5:::smithi03829856-17:head, version: 49'74, data_included: [413472~893152,1818036~155424], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:aa3303a5:::smithi03829856-17:head@49'74, size: 3613472, copy_subset: [(0, 3613472)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1973460, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:32:13.271 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:28.058610+0000", 2024-03-19T11:32:13.271 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 105.16989016300001, 2024-03-19T11:32:13.271 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027100032, 2024-03-19T11:32:13.272 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:13.272 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:32:13.272 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:13.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:13.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.058610+0000", 2024-03-19T11:32:13.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:13.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.272 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:13.272 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.058610+0000", 2024-03-19T11:32:13.272 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:13.272 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.272 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:13.273 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.058611+0000", 2024-03-19T11:32:13.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1599999999999999e-06 2024-03-19T11:32:13.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:13.273 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.063874+0000", 2024-03-19T11:32:13.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0052625270000000004 2024-03-19T11:32:13.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.273 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:13.273 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.063935+0000", 2024-03-19T11:32:13.273 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.0594000000000003e-05 2024-03-19T11:32:13.273 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.273 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.274 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:13.274 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064024+0000", 2024-03-19T11:32:13.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.9273999999999999e-05 2024-03-19T11:32:13.274 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.274 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.274 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:13.274 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064385+0000", 2024-03-19T11:32:13.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00036129100000000002 2024-03-19T11:32:13.274 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.274 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.274 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:13.274 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064437+0000", 2024-03-19T11:32:13.274 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.2346000000000003e-05 2024-03-19T11:32:13.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.275 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:13.275 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085710+0000", 2024-03-19T11:32:13.275 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021272840000000001 2024-03-19T11:32:13.275 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:13.275 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:13.275 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:13.275 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.275 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.275 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.15 e51,50 4:ac9cd3a3:::smithi03829856-8:4,49'71)", 2024-03-19T11:32:13.275 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:28.064112+0000", 2024-03-19T11:32:13.275 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 105.16438833700001, 2024-03-19T11:32:13.275 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026480417999999999, 2024-03-19T11:32:13.276 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:13.276 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:32:13.276 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:13.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:13.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064112+0000", 2024-03-19T11:32:13.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:13.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:13.276 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064112+0000", 2024-03-19T11:32:13.276 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:13.276 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.276 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.276 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:13.277 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064113+0000", 2024-03-19T11:32:13.277 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3600000000000003e-07 2024-03-19T11:32:13.277 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.277 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:13.277 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064118+0000", 2024-03-19T11:32:13.277 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.0549999999999999e-06 2024-03-19T11:32:13.277 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.277 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.277 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:13.277 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064128+0000", 2024-03-19T11:32:13.277 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.0349e-05 2024-03-19T11:32:13.277 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.278 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.278 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:13.278 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064136+0000", 2024-03-19T11:32:13.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.6249999999999998e-06 2024-03-19T11:32:13.278 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.278 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.278 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:13.278 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085773+0000", 2024-03-19T11:32:13.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.021636829999999999 2024-03-19T11:32:13.278 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.278 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.278 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:13.278 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085788+0000", 2024-03-19T11:32:13.278 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5512999999999999e-05 2024-03-19T11:32:13.279 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.279 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:13.279 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.090592+0000", 2024-03-19T11:32:13.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0048044100000000003 2024-03-19T11:32:13.279 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:13.279 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:13.279 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:13.279 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.279 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.279 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.5 51/50 [PushOp(4:a38b9453:::smithi03829856-37:head, version: 28'45, data_included: [605052~774376,1923587~274200], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:a38b9453:::smithi03829856-37:head@28'45, size: 2605052, copy_subset: [(0, 2605052)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2197787, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:a0a50d04:::smithi03829856-13:b, version: 47'63, data_included: [601684~619341,1669078~429235], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:a0a50d04:::smithi03829856-13:b@47'63, size: 2201684, copy_subset: [(0, 2201684)], clone_subset: {}, snapset: b=[]:{b: [b, a, 6, 5]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2098313, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:32:13.279 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:28.064324+0000", 2024-03-19T11:32:13.279 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 105.164176493, 2024-03-19T11:32:13.279 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.027179020000000002, 2024-03-19T11:32:13.280 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:13.280 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:32:13.280 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:13.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.280 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:13.280 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064324+0000", 2024-03-19T11:32:13.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:13.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.280 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:13.280 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064324+0000", 2024-03-19T11:32:13.280 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:13.280 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.280 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:13.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.064324+0000", 2024-03-19T11:32:13.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1199999999999998e-07 2024-03-19T11:32:13.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:13.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.065527+0000", 2024-03-19T11:32:13.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0012028939999999999 2024-03-19T11:32:13.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.281 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.281 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:13.281 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.065552+0000", 2024-03-19T11:32:13.281 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.4440999999999999e-05 2024-03-19T11:32:13.281 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:13.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.065568+0000", 2024-03-19T11:32:13.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6498e-05 2024-03-19T11:32:13.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:13.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085812+0000", 2024-03-19T11:32:13.282 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.020243954000000002 2024-03-19T11:32:13.282 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.282 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.282 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:13.282 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.085825+0000", 2024-03-19T11:32:13.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3298999999999999e-05 2024-03-19T11:32:13.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.283 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:13.283 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:28.091503+0000", 2024-03-19T11:32:13.283 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.005677522 2024-03-19T11:32:13.283 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:13.283 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:13.283 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:13.283 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.283 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.283 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGPush(4.24 75/74 [PushOp(4:27909373:::smithi03829856-46:8, version: 46'59, data_included: [756315~767970,1933572~280606], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:8@46'59, size: 2756315, copy_subset: [(0, 2756315)], clone_subset: {}, snapset: 10=[]:{8: [6], c: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2214178, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:266590a5:::smithi03829856-16:e, version: 70'107, data_included: [652105~662359,1728171~386217], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:e@70'107, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 10=[]:{e: [d, b, 6]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:266590a5:::smithi03829856-16:head, version: 70'108, data_included: [652105~662359,1728171~386217], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:266590a5:::smithi03829856-16:head@70'108, size: 3092455, copy_subset: [(0, 3092455)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2114388, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:27909373:::smithi03829856-46:c, version: 70'109, data_included: [628595~895690,1807440~152886], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:c@70'109, size: 3428595, copy_subset: [(0, 3428595)], clone_subset: {}, snapset: 10=[]:{8: [6], c: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1960326, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false)),PushOp(4:27909373:::smithi03829856-46:head, version: 70'110, data_included: [403401~454961,1584925~506039], data_size: 961000, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:27909373:::smithi03829856-46:head@70'110, size: 2403401, copy_subset: [(0, 2403401)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2403401, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-19T11:32:13.283 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.099172+0000", 2024-03-19T11:32:13.283 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 79.129328059000002, 2024-03-19T11:32:13.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.024528580000000001, 2024-03-19T11:32:13.284 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:13.284 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:32:13.284 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:13.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.284 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:13.284 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.099172+0000", 2024-03-19T11:32:13.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:13.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.284 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.284 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:13.284 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.099172+0000", 2024-03-19T11:32:13.284 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:13.284 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:13.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.099175+0000", 2024-03-19T11:32:13.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.7750000000000001e-06 2024-03-19T11:32:13.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:13.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105219+0000", 2024-03-19T11:32:13.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0060440789999999999 2024-03-19T11:32:13.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.285 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.285 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:13.285 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105285+0000", 2024-03-19T11:32:13.285 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.6260000000000006e-05 2024-03-19T11:32:13.285 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:13.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105366+0000", 2024-03-19T11:32:13.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.038e-05 2024-03-19T11:32:13.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:13.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105637+0000", 2024-03-19T11:32:13.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027085199999999997 2024-03-19T11:32:13.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.286 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.286 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:13.286 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105679+0000", 2024-03-19T11:32:13.286 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2088000000000001e-05 2024-03-19T11:32:13.286 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.287 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:13.287 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.123701+0000", 2024-03-19T11:32:13.287 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018022145999999999 2024-03-19T11:32:13.287 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:13.287 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:13.287 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:13.287 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.287 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.287 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "MOSDPGRecoveryDelete(4.24 e75,74 4:266590a5:::smithi03829856-16:4,47'63,4:27909373:::smithi03829856-46:5,55'79,4:266590a5:::smithi03829856-16:10,67'105)", 2024-03-19T11:32:13.288 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:30:54.105437+0000", 2024-03-19T11:32:13.288 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 79.123063056000007, 2024-03-19T11:32:13.288 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.01860726, 2024-03-19T11:32:13.288 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:13.288 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "started", 2024-03-19T11:32:13.288 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:13.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.288 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:13.288 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105437+0000", 2024-03-19T11:32:13.288 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:13.288 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.288 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.288 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:13.288 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105437+0000", 2024-03-19T11:32:13.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:13.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:13.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105438+0000", 2024-03-19T11:32:13.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6799999999999999e-07 2024-03-19T11:32:13.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:13.289 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105445+0000", 2024-03-19T11:32:13.289 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3749999999999999e-06 2024-03-19T11:32:13.289 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.289 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.289 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:13.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105454+0000", 2024-03-19T11:32:13.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.7909999999999999e-06 2024-03-19T11:32:13.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:13.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.105465+0000", 2024-03-19T11:32:13.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.045e-05 2024-03-19T11:32:13.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:13.290 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.123753+0000", 2024-03-19T11:32:13.290 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0182884 2024-03-19T11:32:13.290 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.290 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.290 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:13.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.123765+0000", 2024-03-19T11:32:13.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2143e-05 2024-03-19T11:32:13.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.291 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:13.291 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:30:54.124045+0000", 2024-03-19T11:32:13.291 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00027923299999999999 2024-03-19T11:32:13.291 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:13.291 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:13.291 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:13.291 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.291 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.291 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:3103 4.15 e104/76 4:afa9f208:::smithi03829856-35:head v 104'221, mlcod=101'216)", 2024-03-19T11:32:13.292 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.563593+0000", 2024-03-19T11:32:13.292 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 49.664907778, 2024-03-19T11:32:13.292 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026983566, 2024-03-19T11:32:13.292 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:13.292 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:13.292 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:13.292 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.292 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:13.292 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.563593+0000", 2024-03-19T11:32:13.292 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:13.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.293 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:13.293 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.563593+0000", 2024-03-19T11:32:13.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:13.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.293 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:13.293 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.563595+0000", 2024-03-19T11:32:13.293 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2859999999999998e-06 2024-03-19T11:32:13.293 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.293 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.293 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:13.293 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.564339+0000", 2024-03-19T11:32:13.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00074400099999999999 2024-03-19T11:32:13.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.294 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:13.294 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.564376+0000", 2024-03-19T11:32:13.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6628999999999998e-05 2024-03-19T11:32:13.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.294 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:13.294 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.564396+0000", 2024-03-19T11:32:13.294 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0596000000000001e-05 2024-03-19T11:32:13.294 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.294 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.294 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:13.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.566600+0000", 2024-03-19T11:32:13.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.002204021 2024-03-19T11:32:13.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:13.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.566642+0000", 2024-03-19T11:32:13.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1902e-05 2024-03-19T11:32:13.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:13.295 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.590494+0000", 2024-03-19T11:32:13.295 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023851816000000001 2024-03-19T11:32:13.295 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.295 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.295 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:13.296 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.590576+0000", 2024-03-19T11:32:13.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.2315e-05 2024-03-19T11:32:13.296 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:13.296 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:13.296 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:13.296 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.296 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.296 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:3104 4.15 e104/76 4:afa9f208:::smithi03829856-35:head v 104'222, mlcod=101'216)", 2024-03-19T11:32:13.296 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.568280+0000", 2024-03-19T11:32:13.296 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 49.660219935000001, 2024-03-19T11:32:13.296 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.022345304999999999, 2024-03-19T11:32:13.296 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:13.296 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:13.296 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:13.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:13.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.568280+0000", 2024-03-19T11:32:13.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:13.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:13.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.568280+0000", 2024-03-19T11:32:13.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:13.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.297 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.297 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:13.297 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.568282+0000", 2024-03-19T11:32:13.297 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0090000000000002e-06 2024-03-19T11:32:13.297 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:13.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.568299+0000", 2024-03-19T11:32:13.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.6237000000000001e-05 2024-03-19T11:32:13.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:13.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.568331+0000", 2024-03-19T11:32:13.298 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2601999999999997e-05 2024-03-19T11:32:13.298 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.298 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.298 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:13.298 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.568350+0000", 2024-03-19T11:32:13.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.8280000000000001e-05 2024-03-19T11:32:13.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:13.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.571311+0000", 2024-03-19T11:32:13.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0029617770000000001 2024-03-19T11:32:13.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:13.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.571379+0000", 2024-03-19T11:32:13.299 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.7331999999999998e-05 2024-03-19T11:32:13.299 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.299 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.299 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:13.299 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.590583+0000", 2024-03-19T11:32:13.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019204649000000001 2024-03-19T11:32:13.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.300 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:13.300 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.590626+0000", 2024-03-19T11:32:13.300 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.2419000000000002e-05 2024-03-19T11:32:13.300 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:13.300 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:13.300 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:13.300 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.300 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.300 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(client.4415.0:3106 4.15 e104/76 4:ad3ad1b3:::smithi03829856-21:head v 104'224, mlcod=101'216)", 2024-03-19T11:32:13.300 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:23.571056+0000", 2024-03-19T11:32:13.300 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 49.657444476999999, 2024-03-19T11:32:13.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019636355000000001, 2024-03-19T11:32:13.301 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:13.301 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:13.301 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:13.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.301 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:13.301 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.571056+0000", 2024-03-19T11:32:13.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:13.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.301 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.301 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:13.301 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.571056+0000", 2024-03-19T11:32:13.301 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:13.301 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:13.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.571058+0000", 2024-03-19T11:32:13.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.2400000000000002e-06 2024-03-19T11:32:13.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:13.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.571586+0000", 2024-03-19T11:32:13.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00052782299999999996 2024-03-19T11:32:13.302 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.302 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.302 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:13.302 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.571618+0000", 2024-03-19T11:32:13.302 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2453000000000003e-05 2024-03-19T11:32:13.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:13.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.571638+0000", 2024-03-19T11:32:13.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9176000000000001e-05 2024-03-19T11:32:13.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:13.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.573958+0000", 2024-03-19T11:32:13.303 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0023201699999999999 2024-03-19T11:32:13.303 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.303 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.303 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:13.303 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.574023+0000", 2024-03-19T11:32:13.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.5425000000000005e-05 2024-03-19T11:32:13.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:13.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.590646+0000", 2024-03-19T11:32:13.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016623189 2024-03-19T11:32:13.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.304 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.304 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:13.304 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:23.590692+0000", 2024-03-19T11:32:13.304 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.5878999999999999e-05 2024-03-19T11:32:13.304 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:13.304 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:13.304 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:13.304 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.305 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.3.0:309 4.0 e108/60 4:01018511:::smithi03829856-19:b v 108'91, mlcod=100'87)", 2024-03-19T11:32:13.305 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.205898+0000", 2024-03-19T11:32:13.305 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 46.022602681999999, 2024-03-19T11:32:13.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.034464328000000002, 2024-03-19T11:32:13.305 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:13.305 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:13.305 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:13.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.305 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:13.305 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.205898+0000", 2024-03-19T11:32:13.305 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:13.305 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.305 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:13.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.205898+0000", 2024-03-19T11:32:13.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:13.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:13.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.205899+0000", 2024-03-19T11:32:13.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 9.16e-07 2024-03-19T11:32:13.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.306 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.306 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:13.306 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.205916+0000", 2024-03-19T11:32:13.306 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7229000000000001e-05 2024-03-19T11:32:13.306 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:13.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.205946+0000", 2024-03-19T11:32:13.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.0233999999999999e-05 2024-03-19T11:32:13.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:13.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.205966+0000", 2024-03-19T11:32:13.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.9434999999999999e-05 2024-03-19T11:32:13.307 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.307 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.307 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:13.307 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.206162+0000", 2024-03-19T11:32:13.307 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000196918 2024-03-19T11:32:13.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:13.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.206227+0000", 2024-03-19T11:32:13.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.4381000000000006e-05 2024-03-19T11:32:13.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:13.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.240212+0000", 2024-03-19T11:32:13.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.033984852000000003 2024-03-19T11:32:13.308 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.308 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.308 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:13.308 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.240362+0000", 2024-03-19T11:32:13.308 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00015036300000000001 2024-03-19T11:32:13.309 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:13.309 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:13.309 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:13.309 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.309 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.5.0:238 4.15 e108/76 4:ac9cd3a3:::smithi03829856-8:10 v 108'231, mlcod=104'228)", 2024-03-19T11:32:13.309 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.215160+0000", 2024-03-19T11:32:13.309 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 46.013340274000001, 2024-03-19T11:32:13.309 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.024600749000000002, 2024-03-19T11:32:13.309 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:13.309 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:13.309 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:13.309 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.309 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:13.310 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215160+0000", 2024-03-19T11:32:13.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:13.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:13.310 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215160+0000", 2024-03-19T11:32:13.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:13.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.310 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:13.310 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215161+0000", 2024-03-19T11:32:13.310 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.6000000000000002e-07 2024-03-19T11:32:13.310 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.310 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:13.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215172+0000", 2024-03-19T11:32:13.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.1042e-05 2024-03-19T11:32:13.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:13.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215213+0000", 2024-03-19T11:32:13.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.1167000000000001e-05 2024-03-19T11:32:13.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.311 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.311 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:13.311 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215234+0000", 2024-03-19T11:32:13.311 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0534e-05 2024-03-19T11:32:13.311 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:13.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215668+0000", 2024-03-19T11:32:13.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000434074 2024-03-19T11:32:13.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:13.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215749+0000", 2024-03-19T11:32:13.312 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.0702000000000001e-05 2024-03-19T11:32:13.312 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.312 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.312 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:13.312 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.239629+0000", 2024-03-19T11:32:13.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.023880855999999999 2024-03-19T11:32:13.313 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.313 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.313 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:13.313 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.239761+0000", 2024-03-19T11:32:13.313 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000131514 2024-03-19T11:32:13.313 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:13.313 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:13.313 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:13.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.314 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.5.0:239 4.8 e108/100 4:14611006:::smithi03829856-25:c v 108'125, mlcod=103'122)", 2024-03-19T11:32:13.314 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.215292+0000", 2024-03-19T11:32:13.314 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 46.013208089999999, 2024-03-19T11:32:13.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.025163089999999999, 2024-03-19T11:32:13.314 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:13.314 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:13.314 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:13.314 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.314 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:13.314 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215292+0000", 2024-03-19T11:32:13.314 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:13.314 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:13.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215292+0000", 2024-03-19T11:32:13.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:13.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:13.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215293+0000", 2024-03-19T11:32:13.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.47e-07 2024-03-19T11:32:13.315 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.315 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.315 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:13.315 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215315+0000", 2024-03-19T11:32:13.315 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1691e-05 2024-03-19T11:32:13.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:13.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215348+0000", 2024-03-19T11:32:13.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.2676999999999999e-05 2024-03-19T11:32:13.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:13.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215362+0000", 2024-03-19T11:32:13.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3978999999999999e-05 2024-03-19T11:32:13.316 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.316 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.316 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:13.316 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215727+0000", 2024-03-19T11:32:13.316 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000365583 2024-03-19T11:32:13.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:13.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215804+0000", 2024-03-19T11:32:13.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 7.7392999999999996e-05 2024-03-19T11:32:13.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:13.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.240368+0000", 2024-03-19T11:32:13.317 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.024563807999999999 2024-03-19T11:32:13.317 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.317 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.317 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:13.317 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.240455+0000", 2024-03-19T11:32:13.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 8.7112000000000002e-05 2024-03-19T11:32:13.318 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:13.318 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:13.318 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:13.318 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.318 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.318 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.5.0:240 4.15 e108/76 4:aa3303a5:::smithi03829856-17:b v 108'233, mlcod=104'228)", 2024-03-19T11:32:13.318 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.215400+0000", 2024-03-19T11:32:13.318 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 46.013100360000003, 2024-03-19T11:32:13.318 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.026645319000000001, 2024-03-19T11:32:13.318 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:13.318 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:13.318 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:13.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:13.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215400+0000", 2024-03-19T11:32:13.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:13.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:13.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215400+0000", 2024-03-19T11:32:13.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:13.319 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.319 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.319 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:13.319 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215401+0000", 2024-03-19T11:32:13.319 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 6.3e-07 2024-03-19T11:32:13.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:13.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215417+0000", 2024-03-19T11:32:13.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.5821999999999999e-05 2024-03-19T11:32:13.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:13.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215443+0000", 2024-03-19T11:32:13.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.6971e-05 2024-03-19T11:32:13.320 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.320 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.320 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:13.320 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.215457+0000", 2024-03-19T11:32:13.320 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.3139000000000001e-05 2024-03-19T11:32:13.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:13.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.234414+0000", 2024-03-19T11:32:13.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.018957108 2024-03-19T11:32:13.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:13.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.234470+0000", 2024-03-19T11:32:13.321 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6274999999999999e-05 2024-03-19T11:32:13.321 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.321 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.321 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:13.321 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.241943+0000", 2024-03-19T11:32:13.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0074728770000000002 2024-03-19T11:32:13.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.322 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:13.322 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.242045+0000", 2024-03-19T11:32:13.322 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.000102497 2024-03-19T11:32:13.322 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:13.322 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:13.322 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:13.322 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.322 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.322 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.5.0:241 4.8 e108/100 4:15708482:::smithi03829856-23:10 v 108'127, mlcod=103'122)", 2024-03-19T11:32:13.322 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.217180+0000", 2024-03-19T11:32:13.322 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 46.011320091999998, 2024-03-19T11:32:13.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.025026023000000001, 2024-03-19T11:32:13.323 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:13.323 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:13.323 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:13.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.323 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:13.323 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.217180+0000", 2024-03-19T11:32:13.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:13.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.323 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.323 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:13.323 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.217180+0000", 2024-03-19T11:32:13.323 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:13.323 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:13.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.217182+0000", 2024-03-19T11:32:13.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.12e-06 2024-03-19T11:32:13.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:13.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.217200+0000", 2024-03-19T11:32:13.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.7677999999999999e-05 2024-03-19T11:32:13.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.324 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.324 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:13.324 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.217246+0000", 2024-03-19T11:32:13.324 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 4.6053000000000001e-05 2024-03-19T11:32:13.324 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:13.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.217266+0000", 2024-03-19T11:32:13.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.0085000000000002e-05 2024-03-19T11:32:13.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:13.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.226876+0000", 2024-03-19T11:32:13.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.0096094389999999991 2024-03-19T11:32:13.325 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.325 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.325 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:13.325 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.226932+0000", 2024-03-19T11:32:13.325 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6234999999999999e-05 2024-03-19T11:32:13.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:13.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.242063+0000", 2024-03-19T11:32:13.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.015131083 2024-03-19T11:32:13.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.326 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.326 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:13.326 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.242206+0000", 2024-03-19T11:32:13.326 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00014333 2024-03-19T11:32:13.326 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:13.326 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:13.326 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:13.326 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.327 INFO:teuthology.orchestra.run.smithi172.stdout: "description": "osd_repop(osd.1.0:279 4.24 e108/83 4:27909373:::smithi03829856-46:c v 108'129, mlcod=100'124)", 2024-03-19T11:32:13.327 INFO:teuthology.orchestra.run.smithi172.stdout: "initiated_at": "2024-03-19T11:31:27.223035+0000", 2024-03-19T11:32:13.327 INFO:teuthology.orchestra.run.smithi172.stdout: "age": 46.005465751000003, 2024-03-19T11:32:13.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.019488583, 2024-03-19T11:32:13.327 INFO:teuthology.orchestra.run.smithi172.stdout: "type_data": { 2024-03-19T11:32:13.327 INFO:teuthology.orchestra.run.smithi172.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-19T11:32:13.327 INFO:teuthology.orchestra.run.smithi172.stdout: "events": [ 2024-03-19T11:32:13.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "initiated", 2024-03-19T11:32:13.327 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.223035+0000", 2024-03-19T11:32:13.327 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:13.327 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.327 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.327 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "header_read", 2024-03-19T11:32:13.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.223035+0000", 2024-03-19T11:32:13.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0 2024-03-19T11:32:13.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "throttled", 2024-03-19T11:32:13.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.223037+0000", 2024-03-19T11:32:13.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.3829999999999999e-06 2024-03-19T11:32:13.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.328 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "all_read", 2024-03-19T11:32:13.328 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.223073+0000", 2024-03-19T11:32:13.328 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 3.6306000000000001e-05 2024-03-19T11:32:13.328 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.328 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "dispatched", 2024-03-19T11:32:13.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.223095+0000", 2024-03-19T11:32:13.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 2.1738000000000001e-05 2024-03-19T11:32:13.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "queued_for_pg", 2024-03-19T11:32:13.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.223107+0000", 2024-03-19T11:32:13.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 1.2082000000000001e-05 2024-03-19T11:32:13.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.329 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "reached_pg", 2024-03-19T11:32:13.329 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.226145+0000", 2024-03-19T11:32:13.329 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.003038022 2024-03-19T11:32:13.329 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.329 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "started", 2024-03-19T11:32:13.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.226201+0000", 2024-03-19T11:32:13.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 5.6107999999999999e-05 2024-03-19T11:32:13.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "commit_sent", 2024-03-19T11:32:13.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.242322+0000", 2024-03-19T11:32:13.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.016120196 2024-03-19T11:32:13.330 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-19T11:32:13.330 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-19T11:32:13.330 INFO:teuthology.orchestra.run.smithi172.stdout: "event": "done", 2024-03-19T11:32:13.330 INFO:teuthology.orchestra.run.smithi172.stdout: "time": "2024-03-19T11:31:27.242523+0000", 2024-03-19T11:32:13.330 INFO:teuthology.orchestra.run.smithi172.stdout: "duration": 0.00020174799999999999 2024-03-19T11:32:13.330 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:13.331 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:13.331 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:13.331 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-19T11:32:13.331 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-19T11:32:13.331 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:32:13.331 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 389, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 507, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1362, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 844, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 799, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1196, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2024-03-19T11:32:13.332 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-19T11:32:13.353 INFO:tasks.rados.rados.0.smithi038.stdout:1861: expect (ObjNum 325 snap 40 seq_num 325) 2024-03-19T11:32:13.434 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:32:13.443+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:32:13.463 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:13.463 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:13.477 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-19T11:32:13.578 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:32:13.583+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:32:13.587 INFO:tasks.rados.rados.0.smithi038.stdout:1839: finishing rollback tid 1 to smithi03829856-26 2024-03-19T11:32:13.587 INFO:tasks.rados.rados.0.smithi038.stdout:1839: finishing rollback tid 2 to smithi03829856-26 2024-03-19T11:32:13.587 INFO:tasks.rados.rados.0.smithi038.stdout:1839: finishing rollback tid 0 to smithi03829856-26 2024-03-19T11:32:13.588 INFO:tasks.rados.rados.0.smithi038.stderr:./src/test/osd/RadosModel.h: In function 'virtual void RollbackOp::_finish(TestOp::CallbackInfo*)' thread 7f7706ffd640 time 2024-03-19T11:32:13.598720+0000 2024-03-19T11:32:13.588 INFO:tasks.rados.rados.0.smithi038.stderr:./src/test/osd/RadosModel.h: 1971: FAILED ceph_assert((int)(info->id) > last_finished) 2024-03-19T11:32:13.589 INFO:tasks.rados.rados.0.smithi038.stderr: ceph version 19.0.0-2159-g9ef43722 (9ef43722ad94ea688fa7f70daab6dd41fac6ee53) squid (dev) 2024-03-19T11:32:13.589 INFO:tasks.rados.rados.0.smithi038.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7f77105f5dc1] 2024-03-19T11:32:13.589 INFO:tasks.rados.rados.0.smithi038.stderr: 2: /usr/lib/ceph/libceph-common.so.2(+0x181f75) [0x7f77105f5f75] 2024-03-19T11:32:13.589 INFO:tasks.rados.rados.0.smithi038.stderr: 3: ceph_test_rados(+0x28324) [0x56423cea5324] 2024-03-19T11:32:13.589 INFO:tasks.rados.rados.0.smithi038.stderr: 4: (write_callback(void*, void*)+0x1e) [0x56423ceb841e] 2024-03-19T11:32:13.589 INFO:tasks.rados.rados.0.smithi038.stderr: 5: /lib/librados.so.2(+0xaf7b9) [0x7f7710e997b9] 2024-03-19T11:32:13.590 INFO:tasks.rados.rados.0.smithi038.stderr: 6: /lib/librados.so.2(+0xb084e) [0x7f7710e9a84e] 2024-03-19T11:32:13.590 INFO:tasks.rados.rados.0.smithi038.stderr: 7: /lib/librados.so.2(+0xb0aa3) [0x7f7710e9aaa3] 2024-03-19T11:32:13.590 INFO:tasks.rados.rados.0.smithi038.stderr: 8: /lib/librados.so.2(+0x11999e) [0x7f7710f0399e] 2024-03-19T11:32:13.590 INFO:tasks.rados.rados.0.smithi038.stderr: 9: /lib/librados.so.2(+0xc5d8f) [0x7f7710eafd8f] 2024-03-19T11:32:13.590 INFO:tasks.rados.rados.0.smithi038.stderr: 10: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc2b3) [0x7f770fee42b3] 2024-03-19T11:32:13.590 INFO:tasks.rados.rados.0.smithi038.stderr: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f770fc52b43] 2024-03-19T11:32:13.590 INFO:tasks.rados.rados.0.smithi038.stderr: 12: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f770fce4a00] 2024-03-19T11:32:13.638 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:32:13.649+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:32:13.647 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:13.647 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:13.664 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-19T11:32:13.740 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:32:13.747+0000 7f6a8d018640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:32:13.819 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:13.819 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:13.832 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-19T11:32:13.880 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:32:13.887+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:32:13.940 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:32:13.947+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:32:13.991 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:13.992 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:14.005 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-19T11:32:14.042 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:32:14.049+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:32:14.142 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:32:14.151+0000 7fa5e9f11640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:32:14.185 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:14.185 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:14.198 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-19T11:32:14.242 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 108, in signal self.proc.stdin.write(struct.pack('!b', sig)) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 389, in write self._write_all(data) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 507, in _write_all count = self._write(data) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1362, in _write self.channel.sendall(data) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 844, in sendall sent = self.send(s) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 799, in send return self._send(s, m) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1196, in _send raise socket.error("Socket is closed") OSError: Socket is closed 2024-03-19T11:32:14.345 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:32:14.353+0000 7f1840a3d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:32:14.353 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:14.353 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:14.366 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-19T11:32:14.445 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:32:14.455+0000 7f671c5c8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:32:14.480 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:14.480 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:14.493 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-19T11:32:14.581 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:32:14.591+0000 7f70d0615640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:32:14.637 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2024-03-19T11:32:14.647 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:32:14.657+0000 7f924d05e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:32:14.655 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:14.655 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:14.656 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T11:32:14.656 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/rados.py", line 280, in thread run.wait(tests.values()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 179, in _raise_for_status raise CommandCrashedError(command=self.command) teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_chunk --enable_dedup --low_tier_pool low_tier --dedup_chunk_size 131072 --dedup_chunk_algo fastcdc --max-ops 1500 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 25 --op delete 10 --op snap_create 10 --op snap_remove 10 --op rollback 10 --op set_chunk 30 --op tier_promote 10 --op tier_evict 10 --op tier_flush 5 --op write_excl 25 --pool unique_pool_0' 2024-03-19T11:32:14.657 ERROR:teuthology.run_tasks:Manager failed: rados Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/rados.py", line 299, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/rados.py", line 280, in thread run.wait(tests.values()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 179, in _raise_for_status raise CommandCrashedError(command=self.command) teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_chunk --enable_dedup --low_tier_pool low_tier --dedup_chunk_size 131072 --dedup_chunk_algo fastcdc --max-ops 1500 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 25 --op delete 10 --op snap_create 10 --op snap_remove 10 --op rollback 10 --op set_chunk 30 --op tier_promote 10 --op tier_evict 10 --op tier_flush 5 --op write_excl 25 --pool unique_pool_0' 2024-03-19T11:32:14.672 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-03-19T11:32:14.682 INFO:tasks.thrashosds:joining thrashosds 2024-03-19T11:32:14.683 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-19T11:32:14.818 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:14.818 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:14.831 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-19T11:32:14.952 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:14.952 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:14.967 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-19T11:32:15.096 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:15.096 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:15.100 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-03-19T11:32:15.113 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-19T11:32:15.270 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:15.270 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:15.283 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-19T11:32:15.400 INFO:teuthology.orchestra.run.smithi172.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-19T11:32:15.403 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-19T11:32:15.403 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-19T11:32:15.506 INFO:teuthology.orchestra.run.smithi172.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-19T11:32:15.509 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-19T11:32:15.509 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-19T11:32:15.617 INFO:teuthology.orchestra.run.smithi172.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-19T11:32:15.620 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-19T11:32:15.621 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-19T11:32:15.738 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-19T11:32:15.739 INFO:teuthology.orchestra.run.smithi172.stdout: "ops": [], 2024-03-19T11:32:15.739 INFO:teuthology.orchestra.run.smithi172.stdout: "num_ops": 0 2024-03-19T11:32:15.739 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-19T11:32:15.753 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-19T11:32:15.798 INFO:teuthology.orchestra.run.smithi038.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-03-19T11:32:15.824 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-19T11:32:15.824 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-19T11:32:15.848 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-19T11:32:15.849 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-19T11:32:15.871 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:15.871 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:15.881 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-19T11:32:15.882 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-19T11:32:15.885 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-19T11:32:15.886 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-19T11:32:15.887 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-19T11:32:16.009 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:16.009 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:16.023 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-19T11:32:16.080 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-03-19T11:32:16.113 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-03-19T11:32:16.150 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:16.150 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:16.163 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-19T11:32:16.285 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:16.285 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:16.299 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-19T11:32:16.417 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:16.417 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:16.430 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-19T11:32:16.538 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:16.539 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:16.551 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-19T11:32:16.673 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:16.673 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:16.687 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-19T11:32:16.807 INFO:teuthology.orchestra.run.smithi172.stdout:ERROR: (22) Invalid argument 2024-03-19T11:32:16.807 INFO:teuthology.orchestra.run.smithi172.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-19T11:32:17.084 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-03-19T11:32:17.094 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:32:17.103+0000 7f70c4fdb640 -1 osd.0 155 handle_osd_map injecting map crc failure 2024-03-19T11:32:17.105 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:32:17.095+0000 7fa5e08db640 -1 osd.3 155 handle_osd_map injecting map crc failure 2024-03-19T11:32:18.108 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-19T11:32:18.108 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-03-19T11:32:18.108 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 7 2024-03-19T11:32:18.433 INFO:tasks.ceph.mon.a.smithi038.stderr:2024-03-19T11:32:18.443+0000 7f3f60a19640 -1 mon.a@0(leader).osd e157 definitely_dead 0 2024-03-19T11:32:19.134 INFO:teuthology.orchestra.run.smithi038.stderr:marked in osd.7. 2024-03-19T11:32:19.140 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:32:19.147+0000 7fa5de8d7640 -1 osd.3 157 handle_osd_map injecting map crc failure 2024-03-19T11:32:19.172 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-03-19T11:32:19.172 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-03-19T11:32:19.316 INFO:teuthology.orchestra.run.smithi038.stdout:mon.a: {} 2024-03-19T11:32:19.317 INFO:teuthology.orchestra.run.smithi038.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-19T11:32:19.330 INFO:teuthology.orchestra.run.smithi038.stdout:mon.b: {} 2024-03-19T11:32:19.330 INFO:teuthology.orchestra.run.smithi038.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-19T11:32:19.339 INFO:teuthology.orchestra.run.smithi038.stdout:mon.c: {} 2024-03-19T11:32:19.340 INFO:teuthology.orchestra.run.smithi038.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-19T11:32:19.357 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-19T11:32:20.161 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.0 to 1 (10000) 2024-03-19T11:32:20.187 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-03-19T11:32:20.848 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~6s 2024-03-19T11:32:21.176 INFO:teuthology.orchestra.run.smithi038.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-19T11:32:21.202 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-19T11:32:22.175 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.1 to 1 (10000) 2024-03-19T11:32:22.179 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:32:22.187+0000 7f70c4fdb640 -1 osd.0 160 handle_osd_map injecting map crc failure 2024-03-19T11:32:22.203 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-03-19T11:32:23.073 INFO:teuthology.orchestra.run.smithi038.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-19T11:32:23.101 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-19T11:32:24.201 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.2 to 1 (10000) 2024-03-19T11:32:24.226 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-03-19T11:32:25.197 INFO:teuthology.orchestra.run.smithi038.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-19T11:32:25.225 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-19T11:32:26.227 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.3 to 1 (10000) 2024-03-19T11:32:26.256 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-03-19T11:32:27.059 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~12s 2024-03-19T11:32:27.274 INFO:teuthology.orchestra.run.smithi038.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-19T11:32:27.302 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-03-19T11:32:27.734 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:32:27.741+0000 7f923da1c640 -1 osd.7 165 handle_osd_map injecting map crc failure 2024-03-19T11:32:28.070 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:32:28.077+0000 7f1837407640 -1 osd.6 166 handle_osd_map injecting map crc failure 2024-03-19T11:32:28.082 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.4 to 1 (10000) 2024-03-19T11:32:28.111 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-03-19T11:32:29.291 INFO:teuthology.orchestra.run.smithi038.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-19T11:32:29.317 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-03-19T11:32:30.307 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.5 to 1 (10000) 2024-03-19T11:32:30.336 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-03-19T11:32:31.316 INFO:teuthology.orchestra.run.smithi038.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-19T11:32:31.344 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-03-19T11:32:32.008 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:32:32.014+0000 7fa5de8d7640 -1 osd.3 169 handle_osd_map injecting map crc failure 2024-03-19T11:32:32.345 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:32:32.353+0000 7f1837407640 -1 osd.6 170 handle_osd_map injecting map crc failure 2024-03-19T11:32:32.346 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:32:32.354+0000 7fa5e08db640 -1 osd.3 170 handle_osd_map injecting map crc failure 2024-03-19T11:32:32.353 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.6 to 1 (10000) 2024-03-19T11:32:32.377 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-03-19T11:32:33.078 INFO:teuthology.orchestra.run.smithi038.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-19T11:32:33.104 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-03-19T11:32:33.269 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~19s 2024-03-19T11:32:34.364 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.7 to 1 (10000) 2024-03-19T11:32:34.390 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-03-19T11:32:35.073 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:32:35.082+0000 7f70c4fdb640 -1 osd.0 172 handle_osd_map injecting map crc failure 2024-03-19T11:32:35.391 INFO:teuthology.orchestra.run.smithi038.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-19T11:32:35.415 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-03-19T11:32:35.415 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-03-19T11:32:35.416 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-03-19T11:32:35.416 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-03-19T11:32:35.416 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-19T11:32:35.416 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:32:35.721 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:32:35.722 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":174,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:32:35.375781+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:07.288159+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:19.897514+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:19.226814+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:13.838782+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:26.849963+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:13.752415+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:09.716969+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:32:38.741 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:32:39.057 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:32:39.057 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":176,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:32:38.077910+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:07.288159+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:19.897514+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:19.226814+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:13.838782+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:26.849963+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:13.752415+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:36.044317+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:32:39.480 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~25s 2024-03-19T11:32:42.077 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:32:42.390 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:32:42.390 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":176,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:32:38.077910+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:07.288159+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:19.897514+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:19.226814+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:13.838782+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:26.849963+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:13.752415+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:36.044317+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:32:45.409 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:32:45.690 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~31s 2024-03-19T11:32:45.725 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:32:45.725 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":176,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:32:38.077910+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:07.288159+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:19.897514+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:19.226814+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:13.838782+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:26.849963+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:13.752415+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:36.044317+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:32:48.748 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:32:49.063 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:32:49.063 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":176,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:32:38.077910+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:07.288159+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:19.897514+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:19.226814+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:13.838782+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:26.849963+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:13.752415+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:36.044317+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:32:51.901 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~37s 2024-03-19T11:32:52.085 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:32:52.402 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:32:52.402 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":176,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:32:38.077910+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:07.288159+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:19.897514+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:19.226814+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:13.838782+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:26.849963+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:13.752415+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:36.044317+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:32:55.423 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:32:55.738 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:32:55.738 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":176,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:32:38.077910+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:07.288159+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:19.897514+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:19.226814+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:13.838782+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:26.849963+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:13.752415+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:36.044317+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:32:58.113 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~43s 2024-03-19T11:32:58.758 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:32:59.072 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:32:59.072 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":176,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:32:38.077910+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:07.288159+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:19.897514+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:19.226814+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:13.838782+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:26.849963+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:13.752415+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:36.044317+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:33:02.095 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:33:02.409 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:33:02.409 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":176,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:32:38.077910+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:07.288159+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:19.897514+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:19.226814+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:13.838782+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:26.849963+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:13.752415+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:36.044317+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:33:04.323 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~50s 2024-03-19T11:33:05.430 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:33:05.745 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:33:05.746 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":176,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:32:38.077910+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:07.288159+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:19.897514+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:19.226814+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:13.838782+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:26.849963+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:13.752415+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:36.044317+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:33:08.285 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:33:08.292+0000 7f923fa20640 -1 osd.7 176 handle_osd_map injecting map crc failure 2024-03-19T11:33:08.766 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:33:09.079 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:33:09.079 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":177,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:33:08.283893+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:08.267743+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:19.897514+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:19.226814+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:13.838782+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:26.849963+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:13.752415+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:36.044317+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:33:10.533 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~56s 2024-03-19T11:33:12.100 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:33:12.412 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:33:12.413 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":177,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:33:08.283893+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:08.267743+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:19.897514+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:19.226814+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:13.838782+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:26.849963+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:13.752415+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:36.044317+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:33:13.086 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:33:13.094+0000 7fa5e08db640 -1 osd.3 177 handle_osd_map injecting map crc failure 2024-03-19T11:33:13.092 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:33:13.100+0000 7f143d8b8640 -1 osd.5 177 handle_osd_map injecting map crc failure 2024-03-19T11:33:14.324 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:33:14.329+0000 7f6a7d9d6640 -1 osd.1 177 handle_osd_map injecting map crc failure 2024-03-19T11:33:15.431 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:33:15.744 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:33:15.745 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":178,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:33:13.090553+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:08.267743+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:19.897514+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:19.226814+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:13.838782+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:26.849963+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:13.752415+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:36.044317+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:33:16.744 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~62s 2024-03-19T11:33:18.765 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:33:19.076 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:33:19.076 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":178,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:33:13.090553+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:08.267743+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:19.897514+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:19.226814+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:13.838782+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:26.849963+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:13.752415+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:36.044317+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:33:22.096 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:33:22.416 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:33:22.417 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":178,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:33:13.090553+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:08.267743+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:19.897514+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:19.226814+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:13.838782+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:26.849963+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:13.752415+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:36.044317+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:33:22.954 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~68s 2024-03-19T11:33:23.291 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:33:23.299+0000 7f923fa20640 -1 osd.7 178 handle_osd_map injecting map crc failure 2024-03-19T11:33:25.437 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:33:25.752 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:33:25.752 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":179,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:33:23.278767+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:08.267743+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:19.897514+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:22.331867+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:22.725683+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:26.849963+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:13.752415+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:36.044317+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:33:28.772 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:33:29.089 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:33:29.089 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":180,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:33:28.095743+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:08.267743+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:27.955033+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:22.331867+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:22.725683+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:26.849963+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:13.752415+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:36.044317+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:33:29.164 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~75s 2024-03-19T11:33:32.111 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:33:32.425 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:33:32.426 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":180,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:33:28.095743+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:08.267743+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:27.955033+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:22.331867+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:22.725683+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:26.849963+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:13.752415+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:36.044317+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:33:35.373 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~81s 2024-03-19T11:33:35.446 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:33:35.760 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:33:35.760 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":180,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:33:28.095743+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:08.267743+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:27.955033+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:22.331867+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:22.725683+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:26.849963+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:13.752415+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:36.044317+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:33:38.091 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:33:38.099+0000 7f143d8b8640 -1 osd.5 181 handle_osd_map injecting map crc failure 2024-03-19T11:33:38.781 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:33:39.096 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:33:39.096 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":182,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:33:38.099202+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:08.267743+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:27.955033+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:22.331867+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:22.725683+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:26.849963+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:35.871017+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:36.044317+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:33:41.584 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~87s 2024-03-19T11:33:42.116 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:33:42.433 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:33:42.433 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":182,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:33:38.099202+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:08.267743+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:27.955033+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:22.331867+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:22.725683+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:26.849963+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:35.871017+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:36.044317+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:33:45.454 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:33:45.769 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:33:45.770 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":183,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:33:45.260038+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:08.267743+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:27.955033+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:22.331867+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:22.725683+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:26.849963+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:35.871017+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:44.260147+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:33:47.793 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~93s 2024-03-19T11:33:48.103 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:33:48.109+0000 7fa5e08db640 -1 osd.3 183 handle_osd_map injecting map crc failure 2024-03-19T11:33:48.790 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:33:49.115 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:33:49.115 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":184,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:33:48.102920+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:08.267743+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:27.955033+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:22.331867+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:22.725683+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:32:26.849963+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:35.871017+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:44.260147+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:33:50.371 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:33:50.377+0000 7fa5e08db640 -1 osd.3 184 handle_osd_map injecting map crc failure 2024-03-19T11:33:52.137 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:33:52.453 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:33:52.453 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":185,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:33:49.110530+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:08.267743+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:27.955033+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:22.331867+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:22.725683+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:48.818651+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:35.871017+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:44.260147+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:33:53.097 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:33:53.105+0000 7f70c6fdf640 -1 osd.0 185 handle_osd_map injecting map crc failure 2024-03-19T11:33:53.097 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:33:53.105+0000 7fa5e08db640 -1 osd.3 185 handle_osd_map injecting map crc failure 2024-03-19T11:33:54.002 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~99s 2024-03-19T11:33:55.474 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:33:55.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-19T11:33:55.825+0000 7ffacb7fe640 0 --2- 172.21.15.38:0/3474877026 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffacc071cc0 0x7ffacc070cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-03-19T11:33:55.997 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:33:55.997 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":186,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:33:53.105535+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:08.267743+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:27.955033+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:22.331867+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:22.725683+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:48.818651+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:35.871017+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:44.260147+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:33:59.018 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:33:59.333 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:33:59.333 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":186,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:33:53.105535+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:08.267743+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:27.955033+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:22.331867+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:22.725683+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:48.818651+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:35.871017+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:44.260147+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:34:00.212 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~106s 2024-03-19T11:34:02.353 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:34:02.669 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:34:02.669 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":186,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:33:53.105535+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:08.267743+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:27.955033+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:22.331867+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:22.725683+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:48.818651+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:35.871017+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:44.260147+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:34:05.689 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:34:06.007 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:34:06.008 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":186,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:33:53.105535+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:08.267743+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:27.955033+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:22.331867+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:22.725683+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:48.818651+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:35.871017+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:44.260147+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:34:06.422 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~112s 2024-03-19T11:34:09.028 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:34:09.346 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:34:09.346 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":186,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:33:53.105535+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:08.267743+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:27.955033+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:22.331867+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:22.725683+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:48.818651+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:35.871017+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:44.260147+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:34:12.366 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:34:12.632 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~118s 2024-03-19T11:34:12.676 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:34:12.676 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":186,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:33:53.105535+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:08.267743+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:27.955033+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:22.331867+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:22.725683+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:48.818651+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:35.871017+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:44.260147+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:34:15.696 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:34:16.009 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:34:16.010 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":186,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:33:53.105535+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:08.267743+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:27.955033+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:22.331867+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:22.725683+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:48.818651+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:35.871017+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:44.260147+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:34:18.399 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:34:18.405+0000 7f70c6fdf640 -1 osd.0 186 handle_osd_map injecting map crc failure 2024-03-19T11:34:18.843 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~124s 2024-03-19T11:34:19.030 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:34:19.347 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:34:19.347 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":187,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:34:18.404768+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:18.109913+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:27.955033+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:22.331867+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:22.725683+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:48.818651+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:35.871017+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:44.260147+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:34:22.367 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:34:22.682 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:34:22.683 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":187,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:34:18.404768+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:18.109913+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:27.955033+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:22.331867+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:22.725683+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:48.818651+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:35.871017+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:44.260147+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:34:25.052 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~130s 2024-03-19T11:34:25.703 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:34:26.020 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:34:26.020 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":188,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:34:23.114595+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:18.109913+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:27.955033+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:22.331867+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:22.725683+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:48.818651+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:35.871017+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:44.260147+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:34:29.040 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:34:29.357 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:34:29.357 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":188,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:34:23.114595+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:18.109913+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:27.955033+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:22.331867+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:22.725683+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:48.818651+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:35.871017+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:44.260147+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:34:31.261 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~137s 2024-03-19T11:34:32.361 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:34:32.369+0000 7f6a7f9da640 -1 osd.1 188 handle_osd_map injecting map crc failure 2024-03-19T11:34:32.378 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:34:32.692 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:34:32.692 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":189,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:34:32.363230+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:18.109913+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:27.955033+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:31.412318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:22.725683+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:48.818651+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:35.871017+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:44.260147+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:34:35.712 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:34:36.025 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:34:36.025 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":190,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:34:33.117801+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:18.109913+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:27.955033+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:31.412318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:22.725683+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:48.818651+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:35.871017+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:44.260147+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:34:37.472 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~143s 2024-03-19T11:34:39.046 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:34:39.357 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:34:39.357 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":190,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:34:33.117801+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:18.109913+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:27.955033+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:31.412318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:22.725683+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:48.818651+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:35.871017+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:44.260147+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:34:42.379 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:34:42.696 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:34:42.696 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":190,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:34:33.117801+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:18.109913+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:27.955033+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:31.412318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:22.725683+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:48.818651+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:35.871017+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:44.260147+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:34:43.637 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:34:43.645+0000 7f6710f8e640 -1 osd.2 190 handle_osd_map injecting map crc failure 2024-03-19T11:34:43.683 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~149s 2024-03-19T11:34:45.716 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:34:46.031 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:34:46.031 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":191,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:34:43.160394+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:18.109913+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:27.955033+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:31.412318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:22.725683+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:48.818651+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:43.128765+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:44.260147+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:34:49.052 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:34:49.370 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:34:49.370 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":192,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:34:46.185756+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:18.109913+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:27.955033+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:31.412318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:46.079289+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:48.818651+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:43.128765+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:44.260147+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:34:49.893 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~155s 2024-03-19T11:34:52.389 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:34:52.707 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:34:52.707 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":193,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:34:50.224178+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:18.109913+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:33:27.955033+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:31.412318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:46.079289+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:49.609602+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:43.128765+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:49.391562+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:34:54.267 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:34:54.277+0000 7f923fa20640 -1 osd.7 194 handle_osd_map injecting map crc failure 2024-03-19T11:34:54.268 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:34:54.277+0000 7f70c6fdf640 -1 osd.0 194 handle_osd_map injecting map crc failure 2024-03-19T11:34:55.728 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:34:56.048 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:34:56.048 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":195,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:34:54.269044+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:18.109913+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:53.282078+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:31.412318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:46.079289+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:49.609602+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:43.128765+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:49.391562+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:34:56.101 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~161s 2024-03-19T11:34:59.070 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:34:59.390 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:34:59.390 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":196,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:34:58.126437+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:18.109913+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:53.282078+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:31.412318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:46.079289+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:49.609602+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:43.128765+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:49.391562+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:35:02.310 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~168s 2024-03-19T11:35:02.411 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:35:02.725 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:35:02.725 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":196,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:34:58.126437+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:18.109913+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:53.282078+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:31.412318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:46.079289+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:49.609602+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:43.128765+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:49.391562+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:35:05.746 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:35:06.057 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:35:06.058 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":196,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:34:58.126437+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:18.109913+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:53.282078+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:31.412318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:46.079289+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:49.609602+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:43.128765+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:49.391562+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:35:08.521 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~174s 2024-03-19T11:35:09.079 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:35:09.391 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:35:09.392 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":196,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:34:58.126437+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:18.109913+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:53.282078+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:31.412318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:46.079289+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:49.609602+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:43.128765+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:49.391562+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:35:12.412 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:35:12.727 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:35:12.727 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":196,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:34:58.126437+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:18.109913+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:53.282078+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:31.412318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:46.079289+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:49.609602+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:43.128765+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:49.391562+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:35:14.731 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~180s 2024-03-19T11:35:15.748 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:35:16.057 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:35:16.057 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":196,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:34:58.126437+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:18.109913+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:53.282078+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:31.412318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:46.079289+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:49.609602+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:43.128765+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:49.391562+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:35:19.077 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:35:19.396 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:35:19.396 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":196,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:34:58.126437+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:18.109913+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:53.282078+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:31.412318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:46.079289+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:49.609602+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:43.128765+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:49.391562+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:35:20.940 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~186s 2024-03-19T11:35:22.417 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:35:22.731 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:35:22.731 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":196,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:34:58.126437+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:18.109913+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:53.282078+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:31.412318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:46.079289+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:49.609602+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:43.128765+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:49.391562+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:35:23.332 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:35:23.341+0000 7f1837407640 -1 osd.6 196 handle_osd_map injecting map crc failure 2024-03-19T11:35:23.501 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:35:23.509+0000 7f923da1c640 -1 osd.7 196 handle_osd_map injecting map crc failure 2024-03-19T11:35:25.751 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:35:26.068 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:35:26.069 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":197,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:35:23.338756+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:23.006215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:53.282078+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:31.412318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:46.079289+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:49.609602+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:43.128765+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:49.391562+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:35:27.149 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~193s 2024-03-19T11:35:29.089 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:35:29.405 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:35:29.406 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":198,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:35:28.135557+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:23.006215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:53.282078+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:31.412318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:46.079289+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:49.609602+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:43.128765+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:49.391562+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:35:32.427 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:35:32.735 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:35:32.735 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":198,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:35:28.135557+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:23.006215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:53.282078+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:31.412318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:46.079289+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:49.609602+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:43.128765+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:49.391562+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:35:33.356 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~199s 2024-03-19T11:35:35.754 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:35:36.068 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:35:36.069 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":198,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:35:28.135557+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:23.006215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:53.282078+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:31.412318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:46.079289+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:49.609602+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:43.128765+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:49.391562+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:35:39.087 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:35:39.402 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:35:39.403 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":198,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:35:28.135557+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:23.006215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:53.282078+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:31.412318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:46.079289+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:49.609602+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:43.128765+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:49.391562+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:35:39.566 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~205s 2024-03-19T11:35:42.423 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:35:42.735 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:35:42.735 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":198,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:35:28.135557+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:23.006215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:53.282078+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:31.412318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:46.079289+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:49.609602+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:43.128765+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:49.391562+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:35:45.754 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:35:45.776 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~211s 2024-03-19T11:35:46.067 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:35:46.067 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":198,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:35:28.135557+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:23.006215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:53.282078+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:31.412318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:46.079289+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:49.609602+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:43.128765+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:49.391562+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:35:49.087 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:35:49.397 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:35:49.397 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":198,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:35:28.135557+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:23.006215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:53.282078+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:31.412318+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:46.079289+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:49.609602+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:43.128765+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:49.391562+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:35:51.986 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~217s 2024-03-19T11:35:52.416 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:35:52.729 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:35:52.729 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":199,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:35:52.416310+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:23.006215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:53.282078+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:52.352520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:46.079289+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:49.609602+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:43.128765+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:49.391562+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:35:53.578 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:35:53.588+0000 7f6a7d9d6640 -1 osd.1 198 handle_osd_map injecting map crc failure 2024-03-19T11:35:55.749 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:35:56.066 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:35:56.067 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":200,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:35:53.143645+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:23.006215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:53.282078+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:52.352520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:46.079289+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:49.609602+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:43.128765+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:49.391562+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:35:58.138 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:35:58.148+0000 7f923fa20640 -1 osd.7 201 handle_osd_map injecting map crc failure 2024-03-19T11:35:58.139 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:35:58.152+0000 7f70c6fdf640 -1 osd.0 201 handle_osd_map injecting map crc failure 2024-03-19T11:35:58.197 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~224s 2024-03-19T11:35:59.087 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:35:59.402 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:35:59.402 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":202,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:35:58.145966+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:23.006215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:53.282078+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:52.352520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:46.079289+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:55.662809+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:43.128765+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:49.391562+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:36:02.422 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:36:02.734 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:36:02.734 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":203,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:36:00.168362+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:23.006215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:59.277893+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:52.352520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:46.079289+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:55.662809+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:43.128765+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:49.391562+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:36:04.408 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~230s 2024-03-19T11:36:05.754 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:36:06.065 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:36:06.066 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":204,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:36:03.148592+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:23.006215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:59.277893+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:52.352520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:46.079289+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:55.662809+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:43.128765+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:49.391562+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:36:09.086 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:36:09.401 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:36:09.401 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":204,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:36:03.148592+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:23.006215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:59.277893+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:52.352520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:46.079289+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:55.662809+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:43.128765+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:49.391562+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:36:10.427 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:36:10.436+0000 7f143d8b8640 -1 osd.5 204 handle_osd_map injecting map crc failure 2024-03-19T11:36:10.617 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~236s 2024-03-19T11:36:12.422 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:36:12.726 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:36:12.726 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":205,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:36:10.427615+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:23.006215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:59.277893+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:52.352520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:46.079289+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:55.662809+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:09.980553+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:34:49.391562+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:36:15.744 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:36:16.059 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:36:16.060 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":207,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:36:15.468139+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:23.006215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:59.277893+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:52.352520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:12.949262+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:55.662809+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:09.980553+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:14.577539+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:36:16.826 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~242s 2024-03-19T11:36:19.080 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:36:19.400 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:36:19.400 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":208,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:36:18.154212+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:23.006215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:59.277893+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:52.352520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:12.949262+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:55.662809+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:09.980553+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:14.577539+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:36:22.421 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:36:22.732 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:36:22.732 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":208,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:36:18.154212+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:23.006215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:59.277893+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:52.352520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:12.949262+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:55.662809+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:09.980553+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:14.577539+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:36:23.035 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~248s 2024-03-19T11:36:25.752 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:36:26.069 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:36:26.069 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":208,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:36:18.154212+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:23.006215+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:59.277893+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:52.352520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:12.949262+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:55.662809+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:09.980553+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:14.577539+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:36:29.089 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:36:29.246 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~255s 2024-03-19T11:36:29.350 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:36:29.360+0000 7f1837407640 -1 osd.6 208 handle_osd_map injecting map crc failure 2024-03-19T11:36:29.351 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:36:29.364+0000 7fa5e08db640 -1 osd.3 208 handle_osd_map injecting map crc failure 2024-03-19T11:36:29.408 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:36:29.408 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":209,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:36:29.358861+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:29.014501+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:59.277893+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:52.352520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:12.949262+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:55.662809+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:09.980553+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:14.577539+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:36:32.429 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:36:32.747 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:36:32.747 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":209,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:36:29.358861+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:29.014501+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:59.277893+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:52.352520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:12.949262+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:55.662809+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:09.980553+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:14.577539+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:36:34.181 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:36:34.192+0000 7f923da1c640 -1 osd.7 209 handle_osd_map injecting map crc failure 2024-03-19T11:36:35.455 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~261s 2024-03-19T11:36:35.768 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:36:36.082 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:36:36.082 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":210,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:36:33.159749+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:29.014501+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:59.277893+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:52.352520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:12.949262+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:55.662809+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:09.980553+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:14.577539+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:36:39.102 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:36:39.416 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:36:39.417 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":210,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:36:33.159749+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:29.014501+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:59.277893+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:52.352520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:12.949262+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:55.662809+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:09.980553+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:14.577539+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:36:41.665 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~267s 2024-03-19T11:36:42.437 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:36:42.750 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:36:42.751 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":210,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:36:33.159749+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:29.014501+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:59.277893+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:52.352520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:12.949262+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:55.662809+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:09.980553+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:14.577539+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:36:45.769 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:36:46.082 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:36:46.082 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":210,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:36:33.159749+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:29.014501+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:59.277893+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:52.352520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:12.949262+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:55.662809+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:09.980553+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:14.577539+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:36:47.874 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~273s 2024-03-19T11:36:49.100 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:36:49.415 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:36:49.416 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":210,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:36:33.159749+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:29.014501+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:59.277893+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:52.352520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:12.949262+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:55.662809+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:09.980553+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:14.577539+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:36:52.435 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:36:52.760 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:36:52.760 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":210,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:36:33.159749+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:29.014501+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:59.277893+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:52.352520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:12.949262+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:55.662809+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:09.980553+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:14.577539+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:36:54.084 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~279s 2024-03-19T11:36:55.780 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:36:56.092 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:36:56.092 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":210,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:36:33.159749+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:29.014501+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:59.277893+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:52.352520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:12.949262+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:55.662809+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:09.980553+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:14.577539+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:36:59.111 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:36:59.428 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:36:59.428 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":210,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:36:33.159749+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:29.014501+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:59.277893+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:52.352520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:12.949262+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:55.662809+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:09.980553+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:14.577539+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:37:00.294 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~286s 2024-03-19T11:37:02.451 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:37:02.763 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:37:02.763 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":210,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:36:33.159749+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:29.014501+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:59.277893+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:52.352520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:12.949262+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:55.662809+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:09.980553+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:14.577539+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:37:05.782 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:37:06.098 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:37:06.098 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":210,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:36:33.159749+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:29.014501+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:59.277893+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:52.352520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:12.949262+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:55.662809+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:09.980553+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:14.577539+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:37:06.503 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~292s 2024-03-19T11:37:09.117 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:37:09.428 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:37:09.428 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":210,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:36:33.159749+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:29.014501+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:59.277893+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:52.352520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:12.949262+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:55.662809+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:09.980553+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:14.577539+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:37:12.447 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:37:12.712 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~298s 2024-03-19T11:37:12.761 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:37:12.761 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":211,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:37:12.455245+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:29.014501+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:59.277893+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:35:52.352520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:12.949262+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:37:11.978985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:09.980553+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:14.577539+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:37:15.781 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:37:16.101 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:37:16.102 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":213,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:37:15.487021+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:29.014501+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:37:15.281094+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:37:12.778408+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:12.949262+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:37:11.978985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:09.980553+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:14.577539+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:37:18.299 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:37:18.312+0000 7f923da1c640 -1 osd.7 213 handle_osd_map injecting map crc failure 2024-03-19T11:37:18.922 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~304s 2024-03-19T11:37:18.922 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2024-03-19T11:37:19.122 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:37:19.437 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-19T11:37:19.438 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":214,"fsid":"246bcf68-c431-4aa7-b787-5e9b2d9519d6","created":"2024-03-19T11:29:33.327073+0000","modified":"2024-03-19T11:37:18.173673+0000","last_up_change":"2024-03-19T11:32:11.094368+0000","last_in_change":"2024-03-19T11:32:19.124662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T11:29:42.219654+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-19T11:29:44.428406+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-03-19T11:29:55.608699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9700000286102295,"score_stable":3.9700000286102295,"optimal_score":0.62999999523162842,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-03-19T11:29:56.945423+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":29,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"4.1f","ready_epoch":152,"last_epoch_started":151,"last_epoch_clean":151,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"152","last_force_op_resend_preluminous":"150","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.309999942779541,"score_stable":2.309999942779541,"optimal_score":0.87999999523162842,"raw_score_acting":2.0299999713897705,"raw_score_stable":2.0299999713897705,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3bb2d2be-5253-4da0-acf3-85a114dc3801","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6811","nonce":3148574891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6813","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6815","nonce":3148574891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6823","nonce":3148574891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":3148574891},{"type":"v1","addr":"172.21.15.38:6821","nonce":3148574891}]},"public_addr":"172.21.15.38:6811/3148574891","cluster_addr":"172.21.15.38:6815/3148574891","heartbeat_back_addr":"172.21.15.38:6823/3148574891","heartbeat_front_addr":"172.21.15.38:6821/3148574891","state":["exists","up"]},{"osd":1,"uuid":"e1c926b2-e0be-4ccc-b8a5-d80d0b584ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6825","nonce":675027622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6827","nonce":675027622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6831","nonce":675027622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":675027622},{"type":"v1","addr":"172.21.15.38:6829","nonce":675027622}]},"public_addr":"172.21.15.38:6825/675027622","cluster_addr":"172.21.15.38:6827/675027622","heartbeat_back_addr":"172.21.15.38:6831/675027622","heartbeat_front_addr":"172.21.15.38:6829/675027622","state":["exists","up"]},{"osd":2,"uuid":"61262219-4f8a-4733-af9d-b5208a17c0ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6801","nonce":58279616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6804","nonce":58279616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6817","nonce":58279616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6807","nonce":58279616},{"type":"v1","addr":"172.21.15.38:6810","nonce":58279616}]},"public_addr":"172.21.15.38:6801/58279616","cluster_addr":"172.21.15.38:6804/58279616","heartbeat_back_addr":"172.21.15.38:6817/58279616","heartbeat_front_addr":"172.21.15.38:6810/58279616","state":["exists","up"]},{"osd":3,"uuid":"a31f8d1b-4e38-4ca3-b251-0877002d7235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6803","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6805","nonce":2942163952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6809","nonce":2942163952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6819","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6820","nonce":2942163952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2942163952},{"type":"v1","addr":"172.21.15.38:6816","nonce":2942163952}]},"public_addr":"172.21.15.38:6805/2942163952","cluster_addr":"172.21.15.38:6809/2942163952","heartbeat_back_addr":"172.21.15.38:6820/2942163952","heartbeat_front_addr":"172.21.15.38:6816/2942163952","state":["exists","up"]},{"osd":4,"uuid":"6746723d-f59c-40ba-857e-d8c954d2675f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":150,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6810","nonce":1353919713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6814","nonce":1353919713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6819","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6821","nonce":1353919713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":1353919713},{"type":"v1","addr":"172.21.15.172:6818","nonce":1353919713}]},"public_addr":"172.21.15.172:6810/1353919713","cluster_addr":"172.21.15.172:6814/1353919713","heartbeat_back_addr":"172.21.15.172:6821/1353919713","heartbeat_front_addr":"172.21.15.172:6818/1353919713","state":["exists"]},{"osd":5,"uuid":"f59afdf8-2277-4583-af68-78136f1c5d62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6809","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6811","nonce":2733060387}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6813","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6815","nonce":2733060387}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6823","nonce":2733060387}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6817","nonce":2733060387},{"type":"v1","addr":"172.21.15.172:6820","nonce":2733060387}]},"public_addr":"172.21.15.172:6811/2733060387","cluster_addr":"172.21.15.172:6815/2733060387","heartbeat_back_addr":"172.21.15.172:6823/2733060387","heartbeat_front_addr":"172.21.15.172:6820/2733060387","state":["exists","up"]},{"osd":6,"uuid":"f91d8c47-4e10-40c6-a02d-66d6b72a657b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6801","nonce":337743699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6803","nonce":337743699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6807","nonce":337743699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":337743699},{"type":"v1","addr":"172.21.15.172:6805","nonce":337743699}]},"public_addr":"172.21.15.172:6801/337743699","cluster_addr":"172.21.15.172:6803/337743699","heartbeat_back_addr":"172.21.15.172:6807/337743699","heartbeat_front_addr":"172.21.15.172:6805/337743699","state":["exists","up"]},{"osd":7,"uuid":"46fc052a-5708-42ff-9d5a-bed7a0fc457a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6825","nonce":4145361100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6827","nonce":4145361100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6831","nonce":4145361100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":4145361100},{"type":"v1","addr":"172.21.15.172:6829","nonce":4145361100}]},"public_addr":"172.21.15.172:6825/4145361100","cluster_addr":"172.21.15.172:6827/4145361100","heartbeat_back_addr":"172.21.15.172:6831/4145361100","heartbeat_front_addr":"172.21.15.172:6829/4145361100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:29.014501+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:37:15.281094+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:37:12.778408+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:12.949262+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:31:06.311323+0000","dead_epoch":150},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:37:11.978985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:09.980553+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T11:36:14.577539+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"4.c","osds":[3,6]},{"pgid":"4.1d","osds":[2,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":4,"snaps":[{"begin":23,"length":1},{"begin":37,"length":1},{"begin":40,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T11:37:20.130 INFO:tasks.ceph.osd.0:Sent signal 15 2024-03-19T11:37:20.131 INFO:tasks.ceph.osd.1:Sent signal 15 2024-03-19T11:37:20.131 INFO:tasks.ceph.osd.2:Sent signal 15 2024-03-19T11:37:20.131 INFO:tasks.ceph.osd.3:Sent signal 15 2024-03-19T11:37:20.131 INFO:tasks.ceph.osd.5:Sent signal 15 2024-03-19T11:37:20.132 INFO:tasks.ceph.osd.6:Sent signal 15 2024-03-19T11:37:20.132 INFO:tasks.ceph.osd.7:Sent signal 15 2024-03-19T11:37:20.132 INFO:tasks.ceph.mon.a:Sent signal 15 2024-03-19T11:37:20.132 INFO:tasks.ceph.mon.c:Sent signal 15 2024-03-19T11:37:20.133 INFO:tasks.ceph.mon.b:Sent signal 15 2024-03-19T11:37:20.133 INFO:tasks.ceph.mgr.y:Sent signal 15 2024-03-19T11:37:20.133 INFO:tasks.ceph.mgr.x:Sent signal 15 2024-03-19T11:37:20.134 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:37:20.144+0000 7f1446eee640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25237) UID: 0 2024-03-19T11:37:20.134 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:37:20.144+0000 7f1446eee640 -1 osd.5 214 *** Got signal Terminated *** 2024-03-19T11:37:20.134 INFO:tasks.ceph.osd.5.smithi172.stderr:2024-03-19T11:37:20.144+0000 7f1446eee640 -1 osd.5 214 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T11:37:20.135 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:37:20.143+0000 7f70d0615640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25050) UID: 0 2024-03-19T11:37:20.135 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:37:20.143+0000 7f70d0615640 -1 osd.0 214 *** Got signal Terminated *** 2024-03-19T11:37:20.135 INFO:tasks.ceph.osd.0.smithi038.stderr:2024-03-19T11:37:20.143+0000 7f70d0615640 -1 osd.0 214 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T11:37:20.135 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:37:20.144+0000 7f1840a3d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25239) UID: 0 2024-03-19T11:37:20.135 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:37:20.144+0000 7f1840a3d640 -1 osd.6 214 *** Got signal Terminated *** 2024-03-19T11:37:20.135 INFO:tasks.ceph.osd.6.smithi172.stderr:2024-03-19T11:37:20.144+0000 7f1840a3d640 -1 osd.6 214 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T11:37:20.136 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:37:20.143+0000 7f6a8d018640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25053) UID: 0 2024-03-19T11:37:20.136 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:37:20.143+0000 7f6a8d018640 -1 osd.1 214 *** Got signal Terminated *** 2024-03-19T11:37:20.136 INFO:tasks.ceph.osd.1.smithi038.stderr:2024-03-19T11:37:20.143+0000 7f6a8d018640 -1 osd.1 214 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T11:37:20.136 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:37:20.144+0000 7f924d05e640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25241) UID: 0 2024-03-19T11:37:20.136 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:37:20.144+0000 7f924d05e640 -1 osd.7 214 *** Got signal Terminated *** 2024-03-19T11:37:20.136 INFO:tasks.ceph.osd.7.smithi172.stderr:2024-03-19T11:37:20.144+0000 7f924d05e640 -1 osd.7 214 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T11:37:20.137 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:37:20.143+0000 7f671c5c8640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25055) UID: 0 2024-03-19T11:37:20.137 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:37:20.143+0000 7f671c5c8640 -1 osd.2 214 *** Got signal Terminated *** 2024-03-19T11:37:20.137 INFO:tasks.ceph.osd.2.smithi038.stderr:2024-03-19T11:37:20.143+0000 7f671c5c8640 -1 osd.2 214 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T11:37:20.137 INFO:tasks.ceph.mon.b.smithi172.stderr:2024-03-19T11:37:20.144+0000 7f19560f4640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mon -f --cluster ceph -i b (PID: 24963) UID: 0 2024-03-19T11:37:20.137 INFO:tasks.ceph.mon.b.smithi172.stderr:2024-03-19T11:37:20.144+0000 7f19560f4640 -1 mon.b@1(peon) e1 *** Got Signal Terminated *** 2024-03-19T11:37:20.138 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:37:20.143+0000 7fa5e9f11640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25057) UID: 0 2024-03-19T11:37:20.138 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:37:20.143+0000 7fa5e9f11640 -1 osd.3 214 *** Got signal Terminated *** 2024-03-19T11:37:20.138 INFO:tasks.ceph.osd.3.smithi038.stderr:2024-03-19T11:37:20.143+0000 7fa5e9f11640 -1 osd.3 214 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T11:37:20.138 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-19T11:37:20.144+0000 7f10c3beb640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mgr -f --cluster ceph -i x (PID: 24965) UID: 0 2024-03-19T11:37:20.138 INFO:tasks.ceph.mgr.x.smithi172.stderr:2024-03-19T11:37:20.144+0000 7f10c3beb640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-19T11:37:20.138 INFO:tasks.ceph.mon.a.smithi038.stderr:2024-03-19T11:37:20.147+0000 7f3f66224640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mon -f --cluster ceph -i a (PID: 24905) UID: 0 2024-03-19T11:37:20.139 INFO:tasks.ceph.mon.a.smithi038.stderr:2024-03-19T11:37:20.147+0000 7f3f66224640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-19T11:37:20.139 INFO:tasks.ceph.mon.c.smithi038.stderr:2024-03-19T11:37:20.147+0000 7fa800708640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mon -f --cluster ceph -i c (PID: 24907) UID: 0 2024-03-19T11:37:20.139 INFO:tasks.ceph.mon.c.smithi038.stderr:2024-03-19T11:37:20.147+0000 7fa800708640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-03-19T11:37:20.335 INFO:tasks.ceph.mgr.y.smithi038.stderr:daemon-helper: command crashed with signal 15 2024-03-19T11:37:22.457 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T11:39:22.479 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-19T11:39:22.481 ERROR:teuthology.run_tasks:Manager failed: thrashosds Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/thrashosds.py", line 214, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/rados.py", line 299, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/rados.py", line 280, in thread run.wait(tests.values()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 179, in _raise_for_status raise CommandCrashedError(command=self.command) teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_chunk --enable_dedup --low_tier_pool low_tier --dedup_chunk_size 131072 --dedup_chunk_algo fastcdc --max-ops 1500 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 25 --op delete 10 --op snap_create 10 --op snap_remove 10 --op rollback 10 --op set_chunk 30 --op tier_promote 10 --op tier_evict 10 --op tier_flush 5 --op write_excl 25 --pool unique_pool_0' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/thrashosds.py", line 218, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/ceph_manager.py", line 2819, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/ceph_manager.py", line 2809, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/ceph_manager.py", line 2572, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/ceph_manager.py", line 2564, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/ceph_manager.py", line 1638, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi038 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' 2024-03-19T11:39:22.483 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-19T11:39:22.493 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-19T11:39:22.494 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T11:41:22.517 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-19T11:41:22.519 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/rados.py", line 299, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/rados.py", line 280, in thread run.wait(tests.values()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 179, in _raise_for_status raise CommandCrashedError(command=self.command) teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_chunk --enable_dedup --low_tier_pool low_tier --dedup_chunk_size 131072 --dedup_chunk_algo fastcdc --max-ops 1500 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 25 --op delete 10 --op snap_create 10 --op snap_remove 10 --op rollback 10 --op set_chunk 30 --op tier_promote 10 --op tier_evict 10 --op tier_flush 5 --op write_excl 25 --pool unique_pool_0' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/ceph_manager.py", line 2785, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/ceph_manager.py", line 2617, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/ceph_manager.py", line 2383, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/ceph_manager.py", line 1638, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi038 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-03-19T11:41:22.521 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-19T11:41:22.521 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-19T11:41:22.521 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-19T11:41:22.522 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T11:41:22.522 INFO:tasks.ceph.osd.0:Stopped 2024-03-19T11:41:22.522 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-19T11:41:22.522 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T11:41:22.522 INFO:tasks.ceph.osd.1:Stopped 2024-03-19T11:41:22.522 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-19T11:41:22.522 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T11:41:22.523 INFO:tasks.ceph.osd.2:Stopped 2024-03-19T11:41:22.523 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-19T11:41:22.523 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T11:41:22.523 INFO:tasks.ceph.osd.3:Stopped 2024-03-19T11:41:22.523 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-19T11:41:22.523 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T11:41:22.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T11:41:22.523 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/rados.py", line 299, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/rados.py", line 280, in thread run.wait(tests.values()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 179, in _raise_for_status raise CommandCrashedError(command=self.command) teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_chunk --enable_dedup --low_tier_pool low_tier --dedup_chunk_size 131072 --dedup_chunk_algo fastcdc --max-ops 1500 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 25 --op delete 10 --op snap_create 10 --op snap_remove 10 --op rollback 10 --op set_chunk 30 --op tier_promote 10 --op tier_evict 10 --op tier_flush 5 --op write_excl 25 --pool unique_pool_0' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/ceph_manager.py", line 2785, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/ceph_manager.py", line 2617, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/ceph_manager.py", line 2383, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/ceph_manager.py", line 1638, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi038 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi172 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4' 2024-03-19T11:41:22.524 INFO:tasks.ceph.osd.4:Stopped 2024-03-19T11:41:22.524 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-19T11:41:22.524 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T11:41:22.524 INFO:tasks.ceph.osd.5:Stopped 2024-03-19T11:41:22.524 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-19T11:41:22.524 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T11:41:22.525 INFO:tasks.ceph.osd.6:Stopped 2024-03-19T11:41:22.525 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-19T11:41:22.525 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T11:41:22.525 INFO:tasks.ceph.osd.7:Stopped 2024-03-19T11:41:22.525 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-19T11:41:22.525 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-19T11:41:22.525 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T11:41:22.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T11:41:22.525 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/rados.py", line 299, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/rados.py", line 280, in thread run.wait(tests.values()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 179, in _raise_for_status raise CommandCrashedError(command=self.command) teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_chunk --enable_dedup --low_tier_pool low_tier --dedup_chunk_size 131072 --dedup_chunk_algo fastcdc --max-ops 1500 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 25 --op delete 10 --op snap_create 10 --op snap_remove 10 --op rollback 10 --op set_chunk 30 --op tier_promote 10 --op tier_evict 10 --op tier_flush 5 --op write_excl 25 --pool unique_pool_0' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/ceph_manager.py", line 2785, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/ceph_manager.py", line 2617, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/ceph_manager.py", line 2383, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/ceph_manager.py", line 1638, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi038 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi038 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y' 2024-03-19T11:41:22.526 INFO:tasks.ceph.mgr.y:Stopped 2024-03-19T11:41:22.526 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-19T11:41:22.526 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T11:41:22.526 INFO:tasks.ceph.mgr.x:Stopped 2024-03-19T11:41:22.527 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-19T11:41:22.527 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-19T11:41:22.527 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T11:41:22.527 INFO:tasks.ceph.mon.a:Stopped 2024-03-19T11:41:22.527 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-19T11:41:22.527 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T11:41:22.527 INFO:tasks.ceph.mon.c:Stopped 2024-03-19T11:41:22.527 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-19T11:41:22.528 INFO:teuthology.orchestra.run:waiting for 300 2024-03-19T11:41:22.528 INFO:tasks.ceph.mon.b:Stopped 2024-03-19T11:41:22.528 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-19T11:41:22.528 DEBUG:teuthology.orchestra.run.smithi038:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(REQUEST_STUCK\)' | egrep -v 'failed to encode map' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-19T11:41:22.547 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi038.front.sepia.ceph.com 2024-03-19T11:41:22.547 DEBUG:teuthology.orchestra.run.smithi038:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-19T11:41:22.719 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi038.front.sepia.ceph.com 2024-03-19T11:41:22.719 DEBUG:teuthology.orchestra.run.smithi038:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-19T11:41:22.898 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi038.front.sepia.ceph.com 2024-03-19T11:41:22.899 DEBUG:teuthology.orchestra.run.smithi038:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-19T11:41:23.038 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi038.front.sepia.ceph.com 2024-03-19T11:41:23.039 DEBUG:teuthology.orchestra.run.smithi038:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-19T11:41:23.191 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi172.front.sepia.ceph.com 2024-03-19T11:41:23.191 DEBUG:teuthology.orchestra.run.smithi172:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-19T11:41:23.300 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi172.front.sepia.ceph.com 2024-03-19T11:41:23.300 DEBUG:teuthology.orchestra.run.smithi172:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-19T11:41:23.353 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi172.front.sepia.ceph.com 2024-03-19T11:41:23.353 DEBUG:teuthology.orchestra.run.smithi172:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-19T11:41:23.481 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi172.front.sepia.ceph.com 2024-03-19T11:41:23.481 DEBUG:teuthology.orchestra.run.smithi172:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-19T11:41:23.631 INFO:tasks.ceph:Archiving mon data... 2024-03-19T11:41:23.633 DEBUG:teuthology.misc:Transferring archived files from smithi038:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/rfriedma-2024-03-19_08:22:49-rados:thrash-wip-rf-nextobj-distro-default-smithi/7610739/data/mon.a.tgz 2024-03-19T11:41:23.633 DEBUG:teuthology.orchestra.run.smithi038:> mktemp 2024-03-19T11:41:23.639 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/tmp.WTkci1QAgi 2024-03-19T11:41:23.640 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.WTkci1QAgi 2024-03-19T11:41:23.786 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 0666 /tmp/tmp.WTkci1QAgi 2024-03-19T11:41:23.860 DEBUG:teuthology.orchestra.remote:smithi038:/tmp/tmp.WTkci1QAgi is 402KB 2024-03-19T11:41:24.080 DEBUG:teuthology.orchestra.run.smithi038:> rm -fr /tmp/tmp.WTkci1QAgi 2024-03-19T11:41:24.086 DEBUG:teuthology.misc:Transferring archived files from smithi038:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/rfriedma-2024-03-19_08:22:49-rados:thrash-wip-rf-nextobj-distro-default-smithi/7610739/data/mon.c.tgz 2024-03-19T11:41:24.086 DEBUG:teuthology.orchestra.run.smithi038:> mktemp 2024-03-19T11:41:24.135 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/tmp.IohEzKJZND 2024-03-19T11:41:24.136 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.IohEzKJZND 2024-03-19T11:41:24.277 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 0666 /tmp/tmp.IohEzKJZND 2024-03-19T11:41:24.339 DEBUG:teuthology.orchestra.remote:smithi038:/tmp/tmp.IohEzKJZND is 406KB 2024-03-19T11:41:24.356 DEBUG:teuthology.orchestra.run.smithi038:> rm -fr /tmp/tmp.IohEzKJZND 2024-03-19T11:41:24.361 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/rfriedma-2024-03-19_08:22:49-rados:thrash-wip-rf-nextobj-distro-default-smithi/7610739/data/mon.b.tgz 2024-03-19T11:41:24.362 DEBUG:teuthology.orchestra.run.smithi172:> mktemp 2024-03-19T11:41:24.367 INFO:teuthology.orchestra.run.smithi172.stdout:/tmp/tmp.2BSnhDDRrJ 2024-03-19T11:41:24.368 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.2BSnhDDRrJ 2024-03-19T11:41:24.508 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0666 /tmp/tmp.2BSnhDDRrJ 2024-03-19T11:41:24.577 DEBUG:teuthology.orchestra.remote:smithi172:/tmp/tmp.2BSnhDDRrJ is 406KB 2024-03-19T11:41:24.639 DEBUG:teuthology.orchestra.run.smithi172:> rm -fr /tmp/tmp.2BSnhDDRrJ 2024-03-19T11:41:24.644 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-19T11:41:24.644 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-19T11:41:24.646 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-19T11:41:24.707 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-19T11:41:24.745 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-19T11:41:24.745 INFO:tasks.ceph:Archiving crash dumps... 2024-03-19T11:41:24.747 DEBUG:teuthology.misc:Transferring archived files from smithi038:/var/lib/ceph/crash to /home/teuthworker/archive/rfriedma-2024-03-19_08:22:49-rados:thrash-wip-rf-nextobj-distro-default-smithi/7610739/remote/smithi038/crash 2024-03-19T11:41:24.748 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-19T11:41:24.763 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/lib/ceph/crash to /home/teuthworker/archive/rfriedma-2024-03-19_08:22:49-rados:thrash-wip-rf-nextobj-distro-default-smithi/7610739/remote/smithi172/crash 2024-03-19T11:41:24.764 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-19T11:41:24.844 INFO:tasks.ceph:Compressing logs... 2024-03-19T11:41:24.845 DEBUG:teuthology.orchestra.run.smithi038:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-19T11:41:24.847 DEBUG:teuthology.orchestra.run.smithi172:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-19T11:41:24.872 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28834.log 2024-03-19T11:41:24.874 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32024.log 2024-03-19T11:41:24.874 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.28834.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28834.log.gz 2024-03-19T11:41:24.874 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32671.log 2024-03-19T11:41:24.874 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.32024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36174.log 2024-03-19T11:41:24.874 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32024.log.gz 2024-03-19T11:41:24.874 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.32671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37099.log 2024-03-19T11:41:24.874 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32671.log.gz 2024-03-19T11:41:24.878 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.36174.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36174.log.gz 2024-03-19T11:41:24.878 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34091.log 2024-03-19T11:41:24.878 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.37099.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37099.log.gz 2024-03-19T11:41:24.878 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30220.log 2024-03-19T11:41:24.878 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.34091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38067.log 2024-03-19T11:41:24.878 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34091.log.gz 2024-03-19T11:41:24.879 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.30220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31437.log 2024-03-19T11:41:24.879 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30220.log.gz 2024-03-19T11:41:24.879 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25062.log 2024-03-19T11:41:24.879 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38067.log.gz 2024-03-19T11:41:24.879 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.31437.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-19T11:41:24.879 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31437.log.gz 2024-03-19T11:41:24.879 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.25062.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25062.log.gz -5 2024-03-19T11:41:24.879 INFO:teuthology.orchestra.run.smithi038.stderr: --verbose -- /var/log/ceph/ceph-client.admin.37648.log 2024-03-19T11:41:24.879 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30431.log 2024-03-19T11:41:24.879 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.37648.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37648.log.gz 2024-03-19T11:41:24.879 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37386.log 2024-03-19T11:41:24.882 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.30431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32869.log 2024-03-19T11:41:24.882 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30431.log.gz 2024-03-19T11:41:24.883 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.37386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35582.log 0.0% 2024-03-19T11:41:24.883 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37386.log.gz 2024-03-19T11:41:24.883 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.32869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35060.log 2024-03-19T11:41:24.883 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.35582.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32869.log.gz 2024-03-19T11:41:24.883 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35582.log.gz 2024-03-19T11:41:24.883 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38431.log 2024-03-19T11:41:24.883 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28824.log 2024-03-19T11:41:24.883 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.35060.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35060.log.gz 2024-03-19T11:41:24.883 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29040.log 2024-03-19T11:41:24.883 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38431.log.gz 2024-03-19T11:41:24.883 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.28824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36081.log 2024-03-19T11:41:24.883 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28824.log.gz 2024-03-19T11:41:24.884 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.29040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36787.log 2024-03-19T11:41:24.884 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29040.log.gz 2024-03-19T11:41:24.884 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.36081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36293.log 2024-03-19T11:41:24.884 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36081.log.gz 2024-03-19T11:41:24.884 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.36787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32298.log 2024-03-19T11:41:24.884 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36787.log.gz 2024-03-19T11:41:24.889 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.36293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36605.log 2024-03-19T11:41:24.889 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36293.log.gz 2024-03-19T11:41:24.890 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.32298.log: 93.9% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-19T11:41:24.890 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-19T11:41:24.890 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32298.log.gz 2024-03-19T11:41:24.890 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.36605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38665.log 2024-03-19T11:41:24.890 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36605.log.gz 2024-03-19T11:41:24.890 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35468.log 2024-03-19T11:41:24.890 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31880.log 2024-03-19T11:41:24.890 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38665.log.gz 2024-03-19T11:41:24.890 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.35468.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.36371.log 2024-03-19T11:41:24.890 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35468.log.gz 2024-03-19T11:41:24.890 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.31880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33588.log 2024-03-19T11:41:24.890 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31880.log.gz 2024-03-19T11:41:24.890 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.36371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29758.log 2024-03-19T11:41:24.891 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36371.log.gz 2024-03-19T11:41:24.891 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.33588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.29856.log 2024-03-19T11:41:24.891 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33588.log.gz 2024-03-19T11:41:24.891 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.29758.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29758.log.gz 2024-03-19T11:41:24.891 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29223.log 2024-03-19T11:41:24.891 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.0.29856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29275.log 2024-03-19T11:41:24.891 INFO:teuthology.orchestra.run.smithi038.stderr: 84.2% -- replaced with /var/log/ceph/ceph-client.0.29856.log.gz 2024-03-19T11:41:24.891 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.29223.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29223.log.gz 2024-03-19T11:41:24.891 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30645.log 2024-03-19T11:41:24.891 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.29275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37937.log 2024-03-19T11:41:24.891 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29275.log.gz 2024-03-19T11:41:24.891 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34768.log 2024-03-19T11:41:24.891 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.30645.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30645.log.gz 2024-03-19T11:41:24.891 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.37937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32776.log 2024-03-19T11:41:24.892 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37937.log.gz 2024-03-19T11:41:24.892 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.34768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37073.log 2024-03-19T11:41:24.892 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34768.log.gz 2024-03-19T11:41:24.892 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.32776.log: /var/log/ceph/ceph-client.admin.29395.log 2024-03-19T11:41:24.892 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32776.log.gz 2024-03-19T11:41:24.898 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.37073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38171.log 2024-03-19T11:41:24.898 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37073.log.gz 2024-03-19T11:41:24.898 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.29395.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29395.log.gz 2024-03-19T11:41:24.898 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35930.log 2024-03-19T11:41:24.899 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28626.log 2024-03-19T11:41:24.899 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38171.log.gz 2024-03-19T11:41:24.899 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.35930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33616.log 2024-03-19T11:41:24.899 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35930.log.gz 2024-03-19T11:41:24.899 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.28626.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28626.log.gz -5 2024-03-19T11:41:24.899 INFO:teuthology.orchestra.run.smithi038.stderr: --verbose -- /var/log/ceph/ceph-client.admin.30794.log 2024-03-19T11:41:24.899 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.33616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32962.log 2024-03-19T11:41:24.899 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33616.log.gz 2024-03-19T11:41:24.899 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35747.log 2024-03-19T11:41:24.899 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.30794.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30794.log.gz 2024-03-19T11:41:24.899 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.32962.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.32962.log.gz -5 2024-03-19T11:41:24.899 INFO:teuthology.orchestra.run.smithi038.stderr: --verbose -- /var/log/ceph/ceph-client.admin.36995.log 2024-03-19T11:41:24.899 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.35747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33727.log 2024-03-19T11:41:24.900 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35747.log.gz 2024-03-19T11:41:24.900 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33837.log 2024-03-19T11:41:24.900 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.36995.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36995.log.gz 2024-03-19T11:41:24.900 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.33727.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.33727.log.gz 2024-03-19T11:41:24.900 INFO:teuthology.orchestra.run.smithi038.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.29023.log 2024-03-19T11:41:24.900 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.33837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28821.log 2024-03-19T11:41:24.900 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33837.log.gz 2024-03-19T11:41:24.900 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.29023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38197.log 2024-03-19T11:41:24.900 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29023.log.gz 2024-03-19T11:41:24.900 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.28821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28619.log 2024-03-19T11:41:24.900 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28821.log.gz 2024-03-19T11:41:24.900 INFO:teuthology.orchestra.run.smithi038.stderr:gzip/var/log/ceph/ceph-client.admin.38197.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.28625.log 2024-03-19T11:41:24.900 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38197.log.gz 2024-03-19T11:41:24.900 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.28619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38821.log 2024-03-19T11:41:24.901 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28619.log.gz 2024-03-19T11:41:24.901 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.28625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33319.log 2024-03-19T11:41:24.901 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28625.log.gz 2024-03-19T11:41:24.901 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36345.log 2024-03-19T11:41:24.901 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38821.log.gz 2024-03-19T11:41:24.901 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.33319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37700.log 2024-03-19T11:41:24.901 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33319.log.gz 2024-03-19T11:41:24.901 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.36345.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.36345.log.gz -- 2024-03-19T11:41:24.901 INFO:teuthology.orchestra.run.smithi038.stderr: /var/log/ceph/ceph-client.admin.37360.log 2024-03-19T11:41:24.901 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.37700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32086.log 2024-03-19T11:41:24.901 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37700.log.gz 2024-03-19T11:41:24.901 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.37360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29037.log 2024-03-19T11:41:24.901 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37360.log.gz 2024-03-19T11:41:24.901 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.32086.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32086.log.gz 2024-03-19T11:41:24.902 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32919.log 2024-03-19T11:41:24.913 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25064.log 2024-03-19T11:41:24.913 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25208.log 2024-03-19T11:41:24.913 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.25064.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25064.log.gz 2024-03-19T11:41:24.913 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25184.log 2024-03-19T11:41:24.913 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.25208.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25208.log.gz 2024-03-19T11:41:24.913 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-19T11:41:24.913 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.25184.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25184.log.gz 2024-03-19T11:41:24.913 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25160.log 2024-03-19T11:41:24.913 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-19T11:41:24.914 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-osd.7.log: /var/log/ceph/ceph-client.admin.25160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25112.log 2024-03-19T11:41:24.914 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25160.log.gz 2024-03-19T11:41:24.914 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-19T11:41:24.914 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.25112.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-19T11:41:24.914 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25112.log.gz 2024-03-19T11:41:24.914 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-19T11:41:24.914 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-19T11:41:24.914 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25136.log 2024-03-19T11:41:24.914 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-19T11:41:24.914 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.25136.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25136.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25088.log 2024-03-19T11:41:24.914 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-19T11:41:24.914 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-19T11:41:24.914 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.25088.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25088.log.gz 2024-03-19T11:41:24.915 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25040.log 2024-03-19T11:41:24.915 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-osd.4.log: 89.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-19T11:41:24.915 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph-client.admin.25040.log: 86.9% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25040.log.gz 2024-03-19T11:41:24.915 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/ceph.log.gz 2024-03-19T11:41:24.935 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.29037.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-19T11:41:24.936 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29037.log.gz 2024-03-19T11:41:24.936 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.32919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30131.log 2024-03-19T11:41:24.936 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32919.log.gz 2024-03-19T11:41:24.936 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35148.log 2024-03-19T11:41:24.936 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.30131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35304.log 2024-03-19T11:41:24.936 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30131.log.gz 2024-03-19T11:41:24.936 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.35148.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35148.log.gz 2024-03-19T11:41:24.936 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30888.log 2024-03-19T11:41:24.936 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.35304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30401.log 2024-03-19T11:41:24.936 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35304.log.gz 2024-03-19T11:41:24.936 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.30888.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30888.log.gz 2024-03-19T11:41:24.936 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37464.log 2024-03-19T11:41:24.937 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.30401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38561.log 2024-03-19T11:41:24.937 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30401.log.gz 2024-03-19T11:41:24.937 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.37464.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37464.log.gz -5 2024-03-19T11:41:24.937 INFO:teuthology.orchestra.run.smithi038.stderr: --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-19T11:41:24.937 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38405.log 2024-03-19T11:41:24.937 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38561.log.gz 2024-03-19T11:41:24.937 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36210.log 2024-03-19T11:41:24.937 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38405.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38405.log.gz 2024-03-19T11:41:24.937 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38951.log 2024-03-19T11:41:24.937 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.36210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31671.log 2024-03-19T11:41:24.937 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36210.log.gz 2024-03-19T11:41:24.937 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33280.log 2024-03-19T11:41:24.937 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38951.log.gz 2024-03-19T11:41:24.937 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38145.log 2024-03-19T11:41:24.938 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.31671.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31671.log.gz 2024-03-19T11:41:24.938 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.33280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34246.log 2024-03-19T11:41:24.938 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33280.log.gz 2024-03-19T11:41:24.938 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31528.log 2024-03-19T11:41:24.938 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38145.log.gz 2024-03-19T11:41:24.938 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.34246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28827.log 2024-03-19T11:41:24.938 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34246.log.gz 2024-03-19T11:41:24.938 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.31528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29022.log 2024-03-19T11:41:24.938 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31528.log.gz 2024-03-19T11:41:24.938 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5/var/log/ceph/ceph-client.admin.28827.log: --verbose -- /var/log/ceph/ceph-client.admin.31854.log 2024-03-19T11:41:24.938 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28827.log.gz 2024-03-19T11:41:24.938 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.29022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30569.log 2024-03-19T11:41:24.938 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29022.log.gz 2024-03-19T11:41:24.939 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.31854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36033.log 2024-03-19T11:41:24.939 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31854.log.gz 2024-03-19T11:41:24.939 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.30569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31189.log 2024-03-19T11:41:24.939 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30569.log.gz 2024-03-19T11:41:24.939 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.36033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36449.log 2024-03-19T11:41:24.939 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36033.log.gz 2024-03-19T11:41:24.939 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.31189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33075.log 2024-03-19T11:41:24.939 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31189.log.gz 2024-03-19T11:41:24.939 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.36449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31905.log 2024-03-19T11:41:24.939 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36449.log.gz 2024-03-19T11:41:24.939 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.33075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35414.log 2024-03-19T11:41:24.939 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33075.log.gz 2024-03-19T11:41:24.939 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.31905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31462.log 2024-03-19T11:41:24.939 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31905.log.gz 2024-03-19T11:41:24.940 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.35414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38119.log 2024-03-19T11:41:24.940 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35414.log.gz 2024-03-19T11:41:24.940 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.31462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28623.log 2024-03-19T11:41:24.940 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31462.log.gz 2024-03-19T11:41:24.940 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36475.log 2024-03-19T11:41:24.940 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38119.log.gz 2024-03-19T11:41:24.940 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.28623.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.28623.log.gz --verbose 2024-03-19T11:41:24.940 INFO:teuthology.orchestra.run.smithi038.stderr: -- /var/log/ceph/ceph-client.admin.33657.log 2024-03-19T11:41:24.940 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31595.log 2024-03-19T11:41:24.940 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.36475.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36475.log.gz 2024-03-19T11:41:24.940 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.33657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30824.log 2024-03-19T11:41:24.940 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33657.log.gz 2024-03-19T11:41:24.940 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37542.log 2024-03-19T11:41:24.940 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.31595.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31595.log.gz 2024-03-19T11:41:24.941 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.30824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36813.log 2024-03-19T11:41:24.941 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30824.log.gz 2024-03-19T11:41:24.941 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.37542.log: gzip -5 --verbose 0.0% -- /var/log/ceph/ceph-client.admin.37830.log -- replaced with /var/log/ceph/ceph-client.admin.37542.log.gz 2024-03-19T11:41:24.941 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-03-19T11:41:24.941 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.36813.log: -- /var/log/ceph/ceph-client.admin.38353.log 2024-03-19T11:41:24.941 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36813.log.gz 2024-03-19T11:41:24.941 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.37830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38613.log 2024-03-19T11:41:24.941 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37830.log.gz 2024-03-19T11:41:24.941 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29419.log 2024-03-19T11:41:24.941 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38353.log.gz 2024-03-19T11:41:24.941 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37125.log 2024-03-19T11:41:24.941 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38613.log.gz 2024-03-19T11:41:24.941 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.29419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33433.log 2024-03-19T11:41:24.942 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29419.log.gz 2024-03-19T11:41:24.942 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.37125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34665.log 2024-03-19T11:41:24.942 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37125.log.gz 2024-03-19T11:41:24.942 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.33433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28573.log 2024-03-19T11:41:24.942 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33433.log.gz 2024-03-19T11:41:24.942 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.34665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37989.log 2024-03-19T11:41:24.942 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34665.log.gz 2024-03-19T11:41:24.942 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.28573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35216.log 2024-03-19T11:41:24.942 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28573.log.gz 2024-03-19T11:41:24.942 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.37989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32679.log 2024-03-19T11:41:24.942 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37989.log.gz 2024-03-19T11:41:24.942 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.35216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35352.log 2024-03-19T11:41:24.942 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35216.log.gz 2024-03-19T11:41:24.943 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.32679.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32679.log.gz 2024-03-19T11:41:24.943 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37674.log 2024-03-19T11:41:24.943 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38379.log 2024-03-19T11:41:24.943 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.35352.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35352.log.gz 2024-03-19T11:41:24.943 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.37674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38509.log 2024-03-19T11:41:24.943 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37674.log.gz 2024-03-19T11:41:24.943 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38967.log 2024-03-19T11:41:24.943 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38379.log.gz 2024-03-19T11:41:24.943 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37622.log 2024-03-19T11:41:24.943 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38509.log.gz 2024-03-19T11:41:24.943 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34284.log 2024-03-19T11:41:24.943 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38967.log.gz 2024-03-19T11:41:24.943 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.37622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35851.log 2024-03-19T11:41:24.943 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37622.log.gz 2024-03-19T11:41:24.944 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35555.log 2024-03-19T11:41:24.944 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.34284.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34284.log.gz 2024-03-19T11:41:24.944 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37568.log 2024-03-19T11:41:24.944 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.35851.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35851.log.gz 2024-03-19T11:41:24.944 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.35555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30031.log 2024-03-19T11:41:24.944 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35555.log.gz 2024-03-19T11:41:24.944 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.37568.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.37568.log.gz 2024-03-19T11:41:24.944 INFO:teuthology.orchestra.run.smithi038.stderr: /var/log/ceph/ceph-client.admin.38795.log 2024-03-19T11:41:24.944 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.30031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38327.log 2024-03-19T11:41:24.944 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30031.log.gz 2024-03-19T11:41:24.944 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36891.log 2024-03-19T11:41:24.944 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38795.log.gz 2024-03-19T11:41:24.944 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38327.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38327.log.gzgzip 2024-03-19T11:41:24.945 INFO:teuthology.orchestra.run.smithi038.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.34050.log 2024-03-19T11:41:24.945 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.36891.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-19T11:41:24.945 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36891.log.gz 2024-03-19T11:41:24.945 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.34050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33000.log 2024-03-19T11:41:24.945 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34050.log.gz 2024-03-19T11:41:24.945 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28433.log 2024-03-19T11:41:24.945 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.33000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38457.log 2024-03-19T11:41:24.945 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33000.log.gz 2024-03-19T11:41:24.945 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.28433.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28433.log.gz 2024-03-19T11:41:24.945 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36631.log 2024-03-19T11:41:24.945 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38249.log 2024-03-19T11:41:24.945 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38457.log.gz 2024-03-19T11:41:24.945 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.36631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29589.log 2024-03-19T11:41:24.945 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36631.log.gz 2024-03-19T11:41:24.946 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35308.log 2024-03-19T11:41:24.946 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38249.log.gz 2024-03-19T11:41:24.946 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.29589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34333.log 2024-03-19T11:41:24.946 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29589.log.gz 2024-03-19T11:41:24.946 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.35308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31779.log 2024-03-19T11:41:24.946 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35308.log.gz 2024-03-19T11:41:24.946 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.34333.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.34333.log.gz /var/log/ceph/ceph-client.admin.37963.log 2024-03-19T11:41:24.946 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-03-19T11:41:24.946 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.31779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29299.log 2024-03-19T11:41:24.946 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31779.log.gz 2024-03-19T11:41:24.946 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.37963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32578.log 2024-03-19T11:41:24.946 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37963.log.gz 2024-03-19T11:41:24.947 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.29299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37911.log 2024-03-19T11:41:24.947 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29299.log.gz 2024-03-19T11:41:24.947 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.32578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32835.log 2024-03-19T11:41:24.947 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32578.log.gz 2024-03-19T11:41:24.947 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.37911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29371.log 2024-03-19T11:41:24.947 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37911.log.gz 2024-03-19T11:41:24.947 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.32835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29495.log 2024-03-19T11:41:24.947 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32835.log.gz 2024-03-19T11:41:24.947 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.29371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30478.log 2024-03-19T11:41:24.947 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29371.log.gz 2024-03-19T11:41:24.947 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.29495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36397.log 2024-03-19T11:41:24.947 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29495.log.gz 2024-03-19T11:41:24.947 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.30478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36865.log 2024-03-19T11:41:24.948 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30478.log.gz 2024-03-19T11:41:24.948 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.36397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30989.log 2024-03-19T11:41:24.948 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36397.log.gz 2024-03-19T11:41:24.948 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.36865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38587.log 2024-03-19T11:41:24.948 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36865.log.gz 2024-03-19T11:41:24.948 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.30989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30269.log 2024-03-19T11:41:24.948 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30989.log.gz 2024-03-19T11:41:24.948 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38587.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-19T11:41:24.948 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38587.log.gz 2024-03-19T11:41:24.948 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33909.log 2024-03-19T11:41:24.948 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.30269.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30269.log.gz 2024-03-19T11:41:24.948 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29713.log 2024-03-19T11:41:24.979 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.33909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28622.log 2024-03-19T11:41:24.979 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33909.log.gz 2024-03-19T11:41:24.979 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.29713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36319.log 2024-03-19T11:41:24.979 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29713.log.gz 2024-03-19T11:41:24.979 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36969.log 2024-03-19T11:41:24.979 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.28622.log: /var/log/ceph/ceph-client.admin.36319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28819.log 2024-03-19T11:41:24.979 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36319.log.gz 2024-03-19T11:41:24.979 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28622.log.gz 2024-03-19T11:41:24.980 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.36969.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36969.log.gz 2024-03-19T11:41:24.980 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34821.log 2024-03-19T11:41:24.980 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.28819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34714.log 2024-03-19T11:41:24.980 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28819.log.gz 2024-03-19T11:41:24.980 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.34821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36501.log 2024-03-19T11:41:24.980 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34821.log.gz 2024-03-19T11:41:24.980 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.34714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33233.log 2024-03-19T11:41:24.980 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34714.log.gz 2024-03-19T11:41:24.980 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.36501.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36501.log.gz 2024-03-19T11:41:24.980 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34846.log 2024-03-19T11:41:24.980 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.33233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31729.log 2024-03-19T11:41:24.980 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33233.log.gz 2024-03-19T11:41:24.980 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.34846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36423.log 2024-03-19T11:41:24.980 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34846.log.gz 2024-03-19T11:41:24.981 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.31729.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31729.log.gz 2024-03-19T11:41:24.981 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34630.log 2024-03-19T11:41:24.981 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.36423.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36423.log.gz 2024-03-19T11:41:24.981 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29443.log 2024-03-19T11:41:24.981 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.34630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38925.log 2024-03-19T11:41:24.981 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34630.log.gz 2024-03-19T11:41:24.981 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.29443.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29551.log 2024-03-19T11:41:24.981 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29443.log.gz 2024-03-19T11:41:24.981 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30757.log 2024-03-19T11:41:24.981 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38925.log.gz 2024-03-19T11:41:24.981 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.29551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37596.log 2024-03-19T11:41:24.981 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29551.log.gz 2024-03-19T11:41:24.981 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29043.log 2024-03-19T11:41:24.982 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.30757.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30757.log.gz 2024-03-19T11:41:24.982 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.37596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33502.log 2024-03-19T11:41:24.982 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37596.log.gz 2024-03-19T11:41:24.982 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33467.log 2024-03-19T11:41:24.982 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.29043.log: /var/log/ceph/ceph-client.admin.33502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37516.log 2024-03-19T11:41:24.982 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29043.log.gz 2024-03-19T11:41:24.982 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33502.log.gz 2024-03-19T11:41:24.982 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.33467.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38639.log 2024-03-19T11:41:24.982 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33467.log.gz 2024-03-19T11:41:24.982 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.37516.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37516.log.gz 2024-03-19T11:41:24.982 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32270.log 2024-03-19T11:41:24.982 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31828.log 2024-03-19T11:41:24.982 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38639.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38639.log.gz 2024-03-19T11:41:24.983 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30057.log 2024-03-19T11:41:24.983 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.32270.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32270.log.gz 2024-03-19T11:41:24.983 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.31828.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31828.log.gz 2024-03-19T11:41:24.983 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30698.log 2024-03-19T11:41:24.983 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.30057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31228.log 2024-03-19T11:41:24.983 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30057.log.gz 2024-03-19T11:41:24.983 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.30698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31321.log 2024-03-19T11:41:24.983 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30698.log.gz 2024-03-19T11:41:24.983 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.31228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28613.log 2024-03-19T11:41:24.983 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31228.log.gz 2024-03-19T11:41:24.983 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.31321.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31321.log.gz 2024-03-19T11:41:24.983 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34963.log 2024-03-19T11:41:24.983 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.28613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34370.log 2024-03-19T11:41:24.984 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28613.log.gz 2024-03-19T11:41:24.984 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.34963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35979.log 2024-03-19T11:41:24.984 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34963.log.gz 2024-03-19T11:41:24.984 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.34370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35514.log 2024-03-19T11:41:24.984 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34370.log.gz 2024-03-19T11:41:24.984 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.35979.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35979.log.gz 2024-03-19T11:41:24.984 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37047.log 2024-03-19T11:41:24.984 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37177.log 2024-03-19T11:41:24.984 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.35514.log: /var/log/ceph/ceph-client.admin.37047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29521.log 2024-03-19T11:41:24.984 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35514.log.gz 2024-03-19T11:41:24.984 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37047.log.gz 2024-03-19T11:41:24.984 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.37177.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37177.log.gz 2024-03-19T11:41:24.984 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-19T11:41:24.985 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33041.log 2024-03-19T11:41:24.985 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.29521.log: /var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32528.log 2024-03-19T11:41:24.985 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29521.log.gz 2024-03-19T11:41:24.985 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.33041.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33041.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29928.log 2024-03-19T11:41:24.985 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-03-19T11:41:24.985 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.32528.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32528.log.gz 2024-03-19T11:41:24.985 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37021.log 2024-03-19T11:41:24.985 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24915.log 2024-03-19T11:41:24.985 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.29928.log: /var/log/ceph/ceph-client.admin.37021.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29928.log.gz 2024-03-19T11:41:24.985 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37021.log.gz 2024-03-19T11:41:24.985 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36553.log 2024-03-19T11:41:24.985 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32229.log 2024-03-19T11:41:24.986 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.24915.log: /var/log/ceph/ceph-client.admin.36553.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24915.log.gz 2024-03-19T11:41:24.986 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36553.log.gz 2024-03-19T11:41:24.986 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37752.log 2024-03-19T11:41:24.986 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34160.log 2024-03-19T11:41:24.986 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.37752.log: /var/log/ceph/ceph-client.admin.32229.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32229.log.gz 2024-03-19T11:41:24.986 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37752.log.gz 2024-03-19T11:41:24.986 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35101.log 2024-03-19T11:41:24.986 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33081.log 2024-03-19T11:41:24.986 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.34160.log: 92.6%/var/log/ceph/ceph-client.admin.35101.log: -- replaced with /var/log/ceph/ceph.log.gz 2024-03-19T11:41:24.986 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34160.log.gz 2024-03-19T11:41:24.986 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35101.log.gz 2024-03-19T11:41:24.986 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32121.log 2024-03-19T11:41:24.987 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38301.log 2024-03-19T11:41:24.987 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.33081.log: /var/log/ceph/ceph-client.admin.32121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32744.log 2024-03-19T11:41:24.987 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33081.log.gz 2024-03-19T11:41:24.987 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32121.log.gz 2024-03-19T11:41:24.987 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38301.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38301.log.gz 2024-03-19T11:41:24.987 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37229.log 2024-03-19T11:41:24.987 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29594.log 2024-03-19T11:41:24.987 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.32744.log: /var/log/ceph/ceph-client.admin.37229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30850.log 2024-03-19T11:41:24.987 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32744.log.gz 2024-03-19T11:41:24.987 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37229.log.gz 2024-03-19T11:41:24.987 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.29594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28515.log 2024-03-19T11:41:24.987 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29594.log.gz 2024-03-19T11:41:24.987 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.30850.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30850.log.gz 2024-03-19T11:41:24.988 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29885.log 2024-03-19T11:41:24.988 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32603.log 2024-03-19T11:41:24.988 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.28515.log: /var/log/ceph/ceph-client.admin.29885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30597.log 2024-03-19T11:41:24.988 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28515.log.gz 2024-03-19T11:41:24.988 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29885.log.gz 2024-03-19T11:41:24.988 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.32603.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32603.log.gz 2024-03-19T11:41:24.988 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36709.log 2024-03-19T11:41:24.988 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37726.log 2024-03-19T11:41:24.988 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.30597.log: /var/log/ceph/ceph-client.admin.36709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37778.log 2024-03-19T11:41:24.988 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36709.log.gz 2024-03-19T11:41:24.988 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30597.log.gz 2024-03-19T11:41:24.988 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.37726.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37726.log.gz 2024-03-19T11:41:24.988 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29591.log 2024-03-19T11:41:24.988 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34932.log 2024-03-19T11:41:24.989 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.37778.log: gzip -5/var/log/ceph/ceph-client.admin.29591.log: --verbose -- /var/log/ceph/ceph-client.admin.29347.log 2024-03-19T11:41:24.989 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37778.log.gz 2024-03-19T11:41:24.989 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29591.log.gz 2024-03-19T11:41:24.989 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.34932.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34932.log.gz 2024-03-19T11:41:24.989 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28624.log 2024-03-19T11:41:24.989 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37308.log 2024-03-19T11:41:24.989 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.29347.log: /var/log/ceph/ceph-client.admin.28624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36657.log 2024-03-19T11:41:24.989 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29347.log.gz 2024-03-19T11:41:24.989 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28624.log.gz 2024-03-19T11:41:24.989 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.37308.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.37308.log.gz 2024-03-19T11:41:24.989 INFO:teuthology.orchestra.run.smithi038.stderr: --verbose -- /var/log/ceph/ceph-client.admin.36579.log 2024-03-19T11:41:24.989 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32478.log 2024-03-19T11:41:24.989 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.36657.log: /var/log/ceph/ceph-client.admin.36579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37151.log 2024-03-19T11:41:24.990 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36657.log.gz 2024-03-19T11:41:24.990 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36579.log.gz 2024-03-19T11:41:24.990 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.32478.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32478.log.gzgzip 2024-03-19T11:41:24.990 INFO:teuthology.orchestra.run.smithi038.stderr: -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-19T11:41:24.990 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35722.log 2024-03-19T11:41:24.990 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.37151.log: /var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28460.log 2024-03-19T11:41:24.990 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37151.log.gz 2024-03-19T11:41:24.990 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.35722.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35722.log.gz 2024-03-19T11:41:24.990 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29041.log 2024-03-19T11:41:24.990 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28820.log 2024-03-19T11:41:24.990 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.28460.log: /var/log/ceph/ceph-client.admin.29041.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28460.log.gz 2024-03-19T11:41:24.990 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29041.log.gz 2024-03-19T11:41:24.990 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33934.log 2024-03-19T11:41:24.990 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29249.log 2024-03-19T11:41:24.991 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.28820.log: /var/log/ceph/ceph-client.admin.33934.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28820.log.gz 2024-03-19T11:41:24.991 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33934.log.gz 2024-03-19T11:41:24.991 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36527.log 2024-03-19T11:41:24.991 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32062.log 2024-03-19T11:41:24.991 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.29249.log: /var/log/ceph/ceph-client.admin.36527.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29249.log.gz 2024-03-19T11:41:24.991 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36527.log.gz 2024-03-19T11:41:24.991 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37203.log 2024-03-19T11:41:24.991 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37490.log 2024-03-19T11:41:24.991 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.32062.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32062.log.gz 2024-03-19T11:41:24.991 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.37203.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33701.log 2024-03-19T11:41:24.991 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37203.log.gz 2024-03-19T11:41:24.991 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.37490.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37490.log.gz 2024-03-19T11:41:24.991 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29821.log 2024-03-19T11:41:24.991 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36761.log 2024-03-19T11:41:24.991 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.33701.log: /var/log/ceph/ceph-client.admin.29821.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33701.log.gz 2024-03-19T11:41:24.992 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29821.log.gz 2024-03-19T11:41:24.992 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34129.log 2024-03-19T11:41:24.992 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29323.log 2024-03-19T11:41:24.992 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.36761.log: /var/log/ceph/ceph-client.admin.34129.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36761.log.gz 2024-03-19T11:41:24.992 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34129.log.gz 2024-03-19T11:41:24.992 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33554.log 2024-03-19T11:41:24.992 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.29323.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29323.log.gz 2024-03-19T11:41:24.992 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37412.log 2024-03-19T11:41:25.013 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-19T11:41:25.013 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.33554.log: /var/log/ceph/ceph-client.admin.37412.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33554.log.gz 2024-03-19T11:41:25.013 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37412.log.gz 2024-03-19T11:41:25.013 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31562.log 2024-03-19T11:41:25.013 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33398.log 2024-03-19T11:41:25.013 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.31562.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31562.log.gz 2024-03-19T11:41:25.013 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35007.log 2024-03-19T11:41:25.013 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.33398.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33398.log.gz 2024-03-19T11:41:25.013 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33795.log 2024-03-19T11:41:25.013 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.35007.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35007.log.gz 2024-03-19T11:41:25.014 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38717.log 2024-03-19T11:41:25.014 INFO:teuthology.orchestra.run.smithi038.stderr: 94.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-19T11:41:25.014 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.33795.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33795.log.gz 2024-03-19T11:41:25.014 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29015.log 2024-03-19T11:41:25.014 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35828.log 2024-03-19T11:41:25.014 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38717.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38717.log.gz 2024-03-19T11:41:25.014 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31376.log 2024-03-19T11:41:25.014 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.29015.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29015.log.gz 2024-03-19T11:41:25.014 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.35828.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35828.log.gz 2024-03-19T11:41:25.014 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36839.log 2024-03-19T11:41:25.014 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29042.log 2024-03-19T11:41:25.014 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.31376.log: /var/log/ceph/ceph-client.admin.36839.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31376.log.gz 2024-03-19T11:41:25.014 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36839.log.gz 2024-03-19T11:41:25.015 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38041.log 2024-03-19T11:41:25.015 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38535.log 2024-03-19T11:41:25.015 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.29042.log: /var/log/ceph/ceph-client.admin.38041.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29042.log.gz 2024-03-19T11:41:25.015 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38041.log.gz 2024-03-19T11:41:25.015 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36082.log 2024-03-19T11:41:25.015 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36917.log 2024-03-19T11:41:25.015 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.36082.log: /var/log/ceph/ceph-client.admin.38535.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38535.log.gz 2024-03-19T11:41:25.015 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36082.log.gz 2024-03-19T11:41:25.015 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38275.log 2024-03-19T11:41:25.015 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32873.log 2024-03-19T11:41:25.015 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.36917.log: /var/log/ceph/ceph-client.admin.38275.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36917.log.gz 2024-03-19T11:41:25.015 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38275.log.gz 2024-03-19T11:41:25.015 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30965.log 2024-03-19T11:41:25.016 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31281.log 2024-03-19T11:41:25.016 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.32873.log: /var/log/ceph/ceph-client.admin.30965.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32873.log.gz 2024-03-19T11:41:25.016 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30965.log.gz 2024-03-19T11:41:25.016 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28835.log 2024-03-19T11:41:25.016 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33323.log 2024-03-19T11:41:25.016 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.31281.log: /var/log/ceph/ceph-client.admin.28835.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31281.log.gz 2024-03-19T11:41:25.016 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28835.log.gz 2024-03-19T11:41:25.016 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31134.log 2024-03-19T11:41:25.016 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30376.log 2024-03-19T11:41:25.016 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.33323.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33323.log.gz 2024-03-19T11:41:25.016 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.31134.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31134.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30139.log 2024-03-19T11:41:25.016 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-03-19T11:41:25.016 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.30376.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30376.log.gz 2024-03-19T11:41:25.016 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38769.log 2024-03-19T11:41:25.017 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34440.log 2024-03-19T11:41:25.017 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.30139.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30139.log.gz 2024-03-19T11:41:25.017 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38769.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38769.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31253.log 2024-03-19T11:41:25.017 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-03-19T11:41:25.017 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.34440.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34440.log.gz 2024-03-19T11:41:25.017 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37334.log 2024-03-19T11:41:25.017 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38743.log 2024-03-19T11:41:25.017 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.31253.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31253.log.gz 2024-03-19T11:41:25.017 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.37334.log: 28.8% -- replaced with /var/log/ceph/ceph-client.admin.37334.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36943.log 2024-03-19T11:41:25.017 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-03-19T11:41:25.017 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38743.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38743.log.gz 2024-03-19T11:41:25.017 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38015.log 2024-03-19T11:41:25.018 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28408.log 2024-03-19T11:41:25.018 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.36943.log: /var/log/ceph/ceph-client.admin.38015.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36943.log.gz 2024-03-19T11:41:25.018 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38015.log.gz 2024-03-19T11:41:25.018 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35816.log 2024-03-19T11:41:25.018 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37255.log 2024-03-19T11:41:25.018 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.28408.log: /var/log/ceph/ceph-client.admin.35816.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28408.log.gz 2024-03-19T11:41:25.018 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35816.log.gz 2024-03-19T11:41:25.018 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32396.log 2024-03-19T11:41:25.018 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28547.log 2024-03-19T11:41:25.018 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.37255.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37255.log.gz 2024-03-19T11:41:25.018 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.32396.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32396.log.gz 2024-03-19T11:41:25.018 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38483.log 2024-03-19T11:41:25.018 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.28547.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28547.log.gz 2024-03-19T11:41:25.019 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36236.log 2024-03-19T11:41:25.019 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36735.log 2024-03-19T11:41:25.019 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38483.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38483.log.gz 2024-03-19T11:41:25.019 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.36236.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36236.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29757.log 2024-03-19T11:41:25.019 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-03-19T11:41:25.019 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.36735.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36735.log.gz 2024-03-19T11:41:25.019 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31040.log 2024-03-19T11:41:25.019 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34208.log 2024-03-19T11:41:25.019 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.29757.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29757.log.gz 2024-03-19T11:41:25.019 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.31040.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38847.log 2024-03-19T11:41:25.019 INFO:teuthology.orchestra.run.smithi038.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31040.log.gz 2024-03-19T11:41:25.019 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.34208.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34208.log.gz 2024-03-19T11:41:25.020 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32400.log 2024-03-19T11:41:25.020 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28627.log 2024-03-19T11:41:25.020 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38847.log: /var/log/ceph/ceph-client.admin.32400.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38847.log.gz 2024-03-19T11:41:25.020 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32400.log.gz 2024-03-19T11:41:25.020 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36683.log 2024-03-19T11:41:25.020 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31644.log 2024-03-19T11:41:25.020 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.28627.log: /var/log/ceph/ceph-client.admin.36683.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28627.log.gz 2024-03-19T11:41:25.020 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36683.log.gz 2024-03-19T11:41:25.020 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29968.log 2024-03-19T11:41:25.020 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37859.log 2024-03-19T11:41:25.020 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.31644.log: /var/log/ceph/ceph-client.admin.29968.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31644.log.gz 2024-03-19T11:41:25.020 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29968.log.gz 2024-03-19T11:41:25.020 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38873.log 2024-03-19T11:41:25.021 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30323.log 2024-03-19T11:41:25.021 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.37859.log: /var/log/ceph/ceph-client.admin.38873.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37859.log.gz 2024-03-19T11:41:25.021 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38873.log.gz 2024-03-19T11:41:25.021 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33207.log 2024-03-19T11:41:25.021 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34470.log 2024-03-19T11:41:25.021 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.30323.log: /var/log/ceph/ceph-client.admin.33207.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30323.log.gz 2024-03-19T11:41:25.021 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33207.log.gz 2024-03-19T11:41:25.021 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37282.log 2024-03-19T11:41:25.033 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32609.log 2024-03-19T11:41:25.033 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.34470.log: /var/log/ceph/ceph-client.admin.37282.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34470.log.gz 2024-03-19T11:41:25.033 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37282.log.gz 2024-03-19T11:41:25.033 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33883.log 2024-03-19T11:41:25.033 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33180.log 2024-03-19T11:41:25.033 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.32609.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32609.log.gz 2024-03-19T11:41:25.033 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.33883.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33883.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37438.log 2024-03-19T11:41:25.033 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-03-19T11:41:25.033 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.33180.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33180.log.gz 2024-03-19T11:41:25.033 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30002.log 2024-03-19T11:41:25.033 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28811.log 2024-03-19T11:41:25.033 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.37438.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37438.log.gz 2024-03-19T11:41:25.033 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.30002.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30002.log.gzgzip 2024-03-19T11:41:25.034 INFO:teuthology.orchestra.run.smithi038.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.34577.log 2024-03-19T11:41:25.034 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.28811.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28811.log.gz 2024-03-19T11:41:25.034 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30511.log 2024-03-19T11:41:25.034 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34469.log 2024-03-19T11:41:25.034 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.34577.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34577.log.gz 2024-03-19T11:41:25.034 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.30511.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30511.log.gzgzip 2024-03-19T11:41:25.034 INFO:teuthology.orchestra.run.smithi038.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38223.log 2024-03-19T11:41:25.034 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.34469.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34469.log.gz 2024-03-19T11:41:25.034 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33146.log 2024-03-19T11:41:25.034 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38093.log 2024-03-19T11:41:25.034 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38223.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38223.log.gz 2024-03-19T11:41:25.034 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.33146.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33146.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38899.log 2024-03-19T11:41:25.034 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-03-19T11:41:25.035 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38093.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38093.log.gz 2024-03-19T11:41:25.035 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31077.log 2024-03-19T11:41:25.035 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35189.log 2024-03-19T11:41:25.035 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38899.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38899.log.gz 2024-03-19T11:41:25.035 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.31077.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31077.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37804.log 2024-03-19T11:41:25.035 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-03-19T11:41:25.035 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.35189.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35189.log.gz 2024-03-19T11:41:25.035 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25172.log 2024-03-19T11:41:25.035 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38691.log 2024-03-19T11:41:25.035 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.37804.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37804.log.gz 2024-03-19T11:41:25.035 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.25172.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25172.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35618.log 2024-03-19T11:41:25.035 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-03-19T11:41:25.035 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.38691.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38691.log.gz 2024-03-19T11:41:25.035 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31991.log 2024-03-19T11:41:25.036 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35657.log 2024-03-19T11:41:25.036 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.35618.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35618.log.gz 2024-03-19T11:41:25.036 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.31991.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31991.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31127.log 2024-03-19T11:41:25.036 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-03-19T11:41:25.036 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.35657.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35657.log.gz 2024-03-19T11:41:25.036 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36262.log 2024-03-19T11:41:25.036 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.20137.log 2024-03-19T11:41:25.036 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.31127.log: /var/log/ceph/ceph-client.admin.36262.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31127.log.gz 2024-03-19T11:41:25.036 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36262.log.gz 2024-03-19T11:41:25.036 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33963.log 2024-03-19T11:41:25.036 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29469.log 2024-03-19T11:41:25.036 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph.tmp-client.admin.20137.log: /var/log/ceph/ceph-client.admin.33963.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.20137.log.gz 2024-03-19T11:41:25.036 INFO:teuthology.orchestra.run.smithi038.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33963.log.gz 2024-03-19T11:41:25.036 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32173.log 2024-03-19T11:41:25.036 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37885.log 2024-03-19T11:41:25.037 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.29469.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29469.log.gz 2024-03-19T11:41:25.037 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.32173.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32173.log.gz 2024-03-19T11:41:25.037 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/ceph-client.admin.37885.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37885.log.gz 2024-03-19T11:41:25.037 INFO:teuthology.orchestra.run.smithi172.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-19T11:41:25.129 INFO:teuthology.orchestra.run.smithi172.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-19T11:41:25.260 INFO:teuthology.orchestra.run.smithi038.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-19T11:41:25.451 INFO:teuthology.orchestra.run.smithi038.stderr: 90.0% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-19T11:41:25.896 INFO:teuthology.orchestra.run.smithi038.stderr: 91.7% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-19T11:41:26.136 INFO:teuthology.orchestra.run.smithi172.stderr: 92.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-19T11:41:26.360 INFO:teuthology.orchestra.run.smithi172.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-19T11:41:26.474 INFO:teuthology.orchestra.run.smithi038.stderr: 92.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-19T11:41:26.733 INFO:teuthology.orchestra.run.smithi172.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-19T11:41:26.798 INFO:teuthology.orchestra.run.smithi172.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-19T11:41:26.799 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-19T11:41:26.799 INFO:teuthology.orchestra.run.smithi172.stderr:real 0m1.923s 2024-03-19T11:41:26.799 INFO:teuthology.orchestra.run.smithi172.stderr:user 0m6.563s 2024-03-19T11:41:26.799 INFO:teuthology.orchestra.run.smithi172.stderr:sys 0m0.252s 2024-03-19T11:41:27.137 INFO:teuthology.orchestra.run.smithi038.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-19T11:41:27.292 INFO:teuthology.orchestra.run.smithi038.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-19T11:41:27.294 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-03-19T11:41:27.294 INFO:teuthology.orchestra.run.smithi038.stderr:real 0m2.445s 2024-03-19T11:41:27.295 INFO:teuthology.orchestra.run.smithi038.stderr:user 0m7.992s 2024-03-19T11:41:27.295 INFO:teuthology.orchestra.run.smithi038.stderr:sys 0m0.355s 2024-03-19T11:41:27.295 INFO:tasks.ceph:Archiving logs... 2024-03-19T11:41:27.295 DEBUG:teuthology.misc:Transferring archived files from smithi038:/var/log/ceph to /home/teuthworker/archive/rfriedma-2024-03-19_08:22:49-rados:thrash-wip-rf-nextobj-distro-default-smithi/7610739/remote/smithi038/log 2024-03-19T11:41:27.296 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-19T11:41:28.572 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/log/ceph to /home/teuthworker/archive/rfriedma-2024-03-19_08:22:49-rados:thrash-wip-rf-nextobj-distro-default-smithi/7610739/remote/smithi172/log 2024-03-19T11:41:28.573 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-19T11:41:29.449 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/rados.py", line 299, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/rados.py", line 280, in thread run.wait(tests.values()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 179, in _raise_for_status raise CommandCrashedError(command=self.command) teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_chunk --enable_dedup --low_tier_pool low_tier --dedup_chunk_size 131072 --dedup_chunk_algo fastcdc --max-ops 1500 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 25 --op delete 10 --op snap_create 10 --op snap_remove 10 --op rollback 10 --op set_chunk 30 --op tier_promote 10 --op tier_evict 10 --op tier_flush 5 --op write_excl 25 --pool unique_pool_0' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/ceph_manager.py", line 2785, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/ceph_manager.py", line 2617, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/ceph_manager.py", line 2383, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/ceph_manager.py", line 1638, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi038 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-03-19T11:41:29.450 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-19T11:41:29.460 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/rados.py", line 299, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/rados.py", line 280, in thread run.wait(tests.values()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 179, in _raise_for_status raise CommandCrashedError(command=self.command) teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_chunk --enable_dedup --low_tier_pool low_tier --dedup_chunk_size 131072 --dedup_chunk_algo fastcdc --max-ops 1500 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 25 --op delete 10 --op snap_create 10 --op snap_remove 10 --op rollback 10 --op set_chunk 30 --op tier_promote 10 --op tier_evict 10 --op tier_flush 5 --op write_excl 25 --pool unique_pool_0' 2024-03-19T11:41:29.461 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-19T11:41:29.461 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-19T11:41:29.464 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-19T11:41:29.544 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-19T11:41:29.554 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-19T11:41:29.554 DEBUG:teuthology.orchestra.run.smithi038:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-19T11:41:29.557 DEBUG:teuthology.orchestra.run.smithi172:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-19T11:41:29.568 INFO:teuthology.orchestra.run.smithi038.stdout: remote refid st t when poll reach delay offset jitter 2024-03-19T11:41:29.568 INFO:teuthology.orchestra.run.smithi038.stdout:============================================================================== 2024-03-19T11:41:29.568 INFO:teuthology.orchestra.run.smithi038.stdout:*hv01.front.sepi 44.4.53.2 2 u 29 64 377 0.104 +0.380 2.857 2024-03-19T11:41:29.568 INFO:teuthology.orchestra.run.smithi038.stdout:+hv02.front.sepi 63.231.80.2 3 u 15 64 377 0.096 -1.514 1.670 2024-03-19T11:41:29.568 INFO:teuthology.orchestra.run.smithi038.stdout:+hv03.front.sepi 74.6.168.72 3 u 17 64 377 0.108 -4.391 1.575 2024-03-19T11:41:29.568 INFO:teuthology.orchestra.run.smithi038.stdout: hv04.front.sepi .INIT. 16 u - 256 0 0.000 +0.000 0.000 2024-03-19T11:41:29.570 INFO:teuthology.orchestra.run.smithi172.stdout: remote refid st t when poll reach delay offset jitter 2024-03-19T11:41:29.570 INFO:teuthology.orchestra.run.smithi172.stdout:============================================================================== 2024-03-19T11:41:29.570 INFO:teuthology.orchestra.run.smithi172.stdout:*hv01.front.sepi 44.4.53.2 2 u 15 64 377 0.064 -3.988 2.302 2024-03-19T11:41:29.570 INFO:teuthology.orchestra.run.smithi172.stdout:+hv02.front.sepi 63.231.80.2 3 u 16 64 377 0.101 +0.143 2.834 2024-03-19T11:41:29.571 INFO:teuthology.orchestra.run.smithi172.stdout:+hv03.front.sepi 74.6.168.72 3 u 25 64 377 0.092 -2.818 2.663 2024-03-19T11:41:29.571 INFO:teuthology.orchestra.run.smithi172.stdout: hv04.front.sepi .INIT. 16 u - 256 0 0.000 +0.000 0.000 2024-03-19T11:41:29.571 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-19T11:41:29.603 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-19T11:41:29.604 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-19T11:41:29.636 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-19T11:41:29.671 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-19T11:41:29.703 INFO:teuthology.task.internal:Duration was 1302.533252 seconds 2024-03-19T11:41:29.703 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-19T11:41:29.737 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-19T11:41:29.737 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-19T11:41:29.740 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-19T11:41:29.778 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-19T11:41:29.778 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi038.front.sepia.ceph.com 2024-03-19T11:41:29.778 DEBUG:teuthology.orchestra.run.smithi038:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-19T11:41:29.831 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi172.front.sepia.ceph.com 2024-03-19T11:41:29.831 DEBUG:teuthology.orchestra.run.smithi172:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-19T11:41:29.840 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-19T11:41:29.840 DEBUG:teuthology.orchestra.run.smithi038:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-19T11:41:29.875 DEBUG:teuthology.orchestra.run.smithi172:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-19T11:41:29.910 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-19T11:41:29.911 DEBUG:teuthology.orchestra.run.smithi038:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-19T11:41:29.947 DEBUG:teuthology.orchestra.run.smithi172:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-19T11:41:30.017 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-19T11:41:30.028 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-19T11:41:30.028 DEBUG:teuthology.orchestra.run.smithi038:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-19T11:41:30.038 DEBUG:teuthology.orchestra.run.smithi172:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-19T11:41:30.050 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-19T11:41:30.061 DEBUG:teuthology.orchestra.run.smithi038:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-19T11:41:30.087 DEBUG:teuthology.orchestra.run.smithi172:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-19T11:41:30.096 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern = core 2024-03-19T11:41:30.102 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern = core 2024-03-19T11:41:30.134 DEBUG:teuthology.orchestra.run.smithi038:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-19T11:41:30.178 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi038.front.sepia.ceph.com, flagging run as failed 2024-03-19T11:41:30.178 DEBUG:teuthology.orchestra.run.smithi172:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-19T11:41:30.183 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi172.front.sepia.ceph.com, flagging run as failed 2024-03-19T11:41:30.183 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-19T11:41:30.194 INFO:teuthology.task.internal:Transferring archived files... 2024-03-19T11:41:30.194 DEBUG:teuthology.misc:Transferring archived files from smithi038:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rfriedma-2024-03-19_08:22:49-rados:thrash-wip-rf-nextobj-distro-default-smithi/7610739/remote/smithi038 2024-03-19T11:41:30.195 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-19T11:41:42.238 INFO:teuthology.task.internal:Transferring binaries for coredumps... 2024-03-19T11:41:42.536 INFO:teuthology.task.internal: core looks like: /home/teuthworker/archive/rfriedma-2024-03-19_08:22:49-rados:thrash-wip-rf-nextobj-distro-default-smithi/7610739/remote/smithi038/coredump/1710847933.29856.core.gz: gzip compressed data, was "1710847933.29856.core", last modified: Tue Mar 19 11:41:30 2024, max compression, original size modulo 2^32 291106816 2024-03-19T11:41:42.536 INFO:teuthology.task.internal:core is compressed, try accessing gzip file ... 2024-03-19T11:41:43.659 INFO:teuthology.task.internal: core looks like: /tmp/tmp18u75gg9: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from 'ceph_test_rados --set_chunk --enable_dedup --low_tier_pool low_tier --dedup_chu', real uid: 1000, effective uid: 1000, real gid: 1271, effective gid: 1271, execfn: '/usr/bin/ceph_test_rados', platform: 'x86_64' 2024-03-19T11:41:43.690 INFO:teuthology.task.internal: dump_program: ceph_test_rados --set_chunk --enable_dedup --low_tier_pool low_tier --dedup_chu 2024-03-19T11:41:43.690 DEBUG:teuthology.orchestra.run.smithi038:> which 'ceph_test_rados --set_chunk --enable_dedup --low_tier_pool low_tier --dedup_chu' 2024-03-19T11:41:43.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T11:41:43.698 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/rados.py", line 299, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/rados.py", line 280, in thread run.wait(tests.values()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 179, in _raise_for_status raise CommandCrashedError(command=self.command) teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_chunk --enable_dedup --low_tier_pool low_tier --dedup_chunk_size 131072 --dedup_chunk_algo fastcdc --max-ops 1500 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 25 --op delete 10 --op snap_create 10 --op snap_remove 10 --op rollback 10 --op set_chunk 30 --op tier_promote 10 --op tier_evict 10 --op tier_flush 5 --op write_excl 25 --pool unique_pool_0' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 420, in archive fetch_binaries_for_coredumps(path, rem) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 345, in fetch_binaries_for_coredumps remote_path = remote.sh(['which', dump_program]).rstrip() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi038 with status 1: "which 'ceph_test_rados --set_chunk --enable_dedup --low_tier_pool low_tier --dedup_chu'" 2024-03-19T11:41:43.699 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-19T11:41:43.710 INFO:teuthology.task.internal:Not uploading archives. 2024-03-19T11:41:43.711 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-19T11:41:43.728 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-19T11:41:43.728 DEBUG:teuthology.orchestra.run.smithi038:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-19T11:41:43.743 DEBUG:teuthology.orchestra.run.smithi172:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-19T11:41:43.747 INFO:teuthology.orchestra.run.smithi038.stdout: 658625 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 19 11:41 /home/ubuntu/cephtest 2024-03-19T11:41:43.747 INFO:teuthology.orchestra.run.smithi038.stdout: 658626 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 19 11:19 /home/ubuntu/cephtest/archive 2024-03-19T11:41:43.747 INFO:teuthology.orchestra.run.smithi038.stdout: 658628 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 19 11:41 /home/ubuntu/cephtest/archive/syslog 2024-03-19T11:41:43.747 INFO:teuthology.orchestra.run.smithi038.stdout: 658630 4 -rw-rw-rw- 1 ubuntu ubuntu 806 Mar 19 11:41 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2024-03-19T11:41:43.747 INFO:teuthology.orchestra.run.smithi038.stdout: 658629 184 -rw-rw-r-- 1 ubuntu ubuntu 187957 Mar 19 11:41 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-19T11:41:43.748 INFO:teuthology.orchestra.run.smithi038.stdout: 663760 180 -rw-rw-rw- 1 ubuntu ubuntu 181889 Mar 19 11:41 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2024-03-19T11:41:43.748 INFO:teuthology.orchestra.run.smithi038.stdout: 658627 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 19 11:32 /home/ubuntu/cephtest/archive/coredump 2024-03-19T11:41:43.748 INFO:teuthology.orchestra.run.smithi038.stdout: 674785 176904 -rw------- 1 ubuntu ubuntu 291106816 Mar 19 11:32 /home/ubuntu/cephtest/archive/coredump/1710847933.29856.core 2024-03-19T11:41:43.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T11:41:43.749 INFO:teuthology.orchestra.run.smithi038.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-03-19T11:41:43.749 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/rados.py", line 299, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_ceph-c_9ef43722ad94ea688fa7f70daab6dd41fac6ee53/qa/tasks/rados.py", line 280, in thread run.wait(tests.values()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 179, in _raise_for_status raise CommandCrashedError(command=self.command) teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_chunk --enable_dedup --low_tier_pool low_tier --dedup_chunk_size 131072 --dedup_chunk_algo fastcdc --max-ops 1500 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 25 --op delete 10 --op snap_create 10 --op snap_remove 10 --op rollback 10 --op set_chunk 30 --op tier_promote 10 --op tier_evict 10 --op tier_flush 5 --op write_excl 25 --pool unique_pool_0' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi038 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-03-19T11:41:43.749 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-19T11:41:43.760 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-19T11:41:43.781 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_chunk --enable_dedup --low_tier_pool low_tier --dedup_chunk_size 131072 --dedup_chunk_algo fastcdc --max-ops 1500 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 25 --op delete 10 --op snap_create 10 --op snap_remove 10 --op rollback 10 --op set_chunk 30 --op tier_promote 10 --op tier_evict 10 --op tier_flush 5 --op write_excl 25 --pool unique_pool_0' 2024-03-19T11:41:43.781 INFO:teuthology.nuke:Checking targets against current locks 2024-03-19T11:41:43.785 INFO:teuthology.orchestra.run.smithi172.stdout: 658624 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 19 11:41 /home/ubuntu/cephtest 2024-03-19T11:41:43.785 INFO:teuthology.orchestra.run.smithi172.stdout: 658625 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 19 11:19 /home/ubuntu/cephtest/archive 2024-03-19T11:41:43.786 INFO:teuthology.orchestra.run.smithi172.stdout: 658627 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 19 11:41 /home/ubuntu/cephtest/archive/syslog 2024-03-19T11:41:43.786 INFO:teuthology.orchestra.run.smithi172.stdout: 658629 4 -rw-rw-rw- 1 ubuntu ubuntu 818 Mar 19 11:41 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2024-03-19T11:41:43.786 INFO:teuthology.orchestra.run.smithi172.stdout: 658628 180 -rw-rw-r-- 1 ubuntu ubuntu 182234 Mar 19 11:41 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-19T11:41:43.786 INFO:teuthology.orchestra.run.smithi172.stdout: 663757 172 -rw-rw-rw- 1 ubuntu ubuntu 173409 Mar 19 11:41 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2024-03-19T11:41:43.786 INFO:teuthology.orchestra.run.smithi172.stdout: 658626 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 19 11:32 /home/ubuntu/cephtest/archive/coredump 2024-03-19T11:41:43.786 INFO:teuthology.orchestra.run.smithi172.stdout: 674583 176852 -rw------- 1 root root 838701056 Mar 19 11:32 /home/ubuntu/cephtest/archive/coredump/1710847930.25245.core 2024-03-19T11:41:43.787 INFO:teuthology.orchestra.run.smithi172.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-03-19T11:41:43.805 DEBUG:teuthology.nuke:shortname: smithi038 2024-03-19T11:41:43.805 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-19T11:41:43.825 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi038.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rfriedma-2024-03-19_08:22:49-rados:thrash-wip-rf-nextobj-distro-default-smithi/7610739', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-19 11:10:37.930425', 'locked_by': 'scheduled_rfriedma@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN12b76+6OZUqzRu5kZgY1R3xXUct6XvKeRMHOD6LkRGOzsW2C5qGHLKLNc/hTgPYE3wvjLymEenOFRKGhA/ouk='} 2024-03-19T11:41:43.828 DEBUG:teuthology.nuke:shortname: smithi172 2024-03-19T11:41:43.828 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-19T11:41:43.852 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi172.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rfriedma-2024-03-19_08:22:49-rados:thrash-wip-rf-nextobj-distro-default-smithi/7610739', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-19 11:10:37.928766', 'locked_by': 'scheduled_rfriedma@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOkhYhhIXg4DTniTbJjE1J7at0/UswGbMkAbBGurn57KL5GO5auYOgugBa786l9o0o5uMhI6NPM/z44KCkJT+3g='} 2024-03-19T11:41:43.875 INFO:teuthology.orchestra.console.smithi038:Power off 2024-03-19T11:41:43.876 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-19T11:41:43.899 DEBUG:teuthology.orchestra.console.smithi038:power off output: Chassis Power Control: Down/Off 2024-03-19T11:41:43.899 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T11:41:43.913 INFO:teuthology.orchestra.console.smithi172:Power off 2024-03-19T11:41:43.914 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-19T11:41:43.922 DEBUG:teuthology.orchestra.console.smithi038:check power output: Chassis Power is on 2024-03-19T11:41:43.936 DEBUG:teuthology.orchestra.console.smithi172:power off output: Chassis Power Control: Down/Off 2024-03-19T11:41:43.936 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T11:41:43.959 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is on 2024-03-19T11:41:47.924 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T11:41:47.960 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T11:41:48.036 DEBUG:teuthology.orchestra.console.smithi038:check power output: Chassis Power is on 2024-03-19T11:41:48.070 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is on 2024-03-19T11:41:52.038 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T11:41:52.071 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T11:41:52.150 DEBUG:teuthology.orchestra.console.smithi038:check power output: Chassis Power is off 2024-03-19T11:41:52.181 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is on 2024-03-19T11:41:52.252 INFO:teuthology.orchestra.console.smithi038:Power off completed 2024-03-19T11:41:56.183 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T11:41:56.294 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is off 2024-03-19T11:41:56.395 INFO:teuthology.orchestra.console.smithi172:Power off completed 2024-03-19T11:41:56.504 INFO:teuthology.run:Summary data: description: rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/dedup-io-snaps} duration: 1302.5332520008087 failure_reason: 'Command crashed: ''CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_chunk --enable_dedup --low_tier_pool low_tier --dedup_chunk_size 131072 --dedup_chunk_algo fastcdc --max-ops 1500 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 25 --op delete 10 --op snap_create 10 --op snap_remove 10 --op rollback 10 --op set_chunk 30 --op tier_promote 10 --op tier_evict 10 --op tier_flush 5 --op write_excl 25 --pool unique_pool_0''' flavor: default owner: scheduled_rfriedma@teuthology status: fail success: false 2024-03-19T11:41:56.505 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-19T11:41:56.579 INFO:teuthology.run:FAIL